[ { "path": ".circleci/README.md", "content": "# Multi Model Server CircleCI build\nModel Server uses CircleCI for builds. This folder contains the config and scripts that are needed for CircleCI.\n\n## config.yml\n_config.yml_ contains MMS build logic which will be used by CircleCI.\n\n## Workflows and Jobs\nCurrently, following _workflows_ are available -\n1. smoke\n2. nightly\n3. weekly\n\nFollowing _jobs_ are executed under each workflow -\n1. **build** : Builds _frontend/model-server.jar_ and executes tests from gradle\n2. **modelarchiver** : Builds and tests modelarchiver module\n3. **python-tests** : Executes pytests from _mms/tests/unit_tests/_\n4. **benchmark** : Executes latency benchmark using resnet-18 model\n5. (NEW!) **api-tests** : Executes newman test suite for API testing\n\nFollowing _executors_ are available for job execution -\n1. py27\n2. py36\n\n> Please check the _workflows_, _jobs_ and _executors_ section in _config.yml_ for an up to date list\n\n## scripts\nInstead of using inline commands inside _config.yml_, job steps are configured as shell scripts. \nThis is easier for maintenance and reduces chances of error in config.yml\n\n## images\nMMS uses customized docker image for its CircleCI build. \nTo make sure MMS is compatible with both Python2 and Python3, we use two build projects. \nWe have published two docker images on docker hub for code build\n* prashantsail/mms-build:python2.7\n* prashantsail/mms-build:python3.6\n\nFollowing files in the _images_ folder are used to create the docker images\n* Dockerfile.python2.7 - Dockerfile for prashantsail/mms-build:python2.7\n* Dockerfile.python3.6 - Dockerfile for prashantsail/mms-build:python3.6\n\n## Local CircleCI cli\nTo make it easy for developers to debug build issues locally, MMS supports CircleCI cli for running a job in a container on your machine.\n\n#### Dependencies\n1. CircleCI cli ([Quick Install](https://circleci.com/docs/2.0/local-cli/#quick-installation))\n2. PyYAML (pip install PyYaml)\n3. docker (installed and running)\n\n#### Command\nDevelopers can use the following command to build MMS locally: \n**./run_circleci_tests.py -j -e **\n\n- _workflow_name_ \nThis is a madatory parameter\n\n- _-j, --job job_name_ \nIf specified, executes only the specified job name (along with the required parents). \nIf not specified, all jobs in the workflow are executed sequentially. \n\n- _-e, --executor executor_name_ \nIf specified, job is executed only on the specified executor(docker image). \nIf not specified, job is executed on all the available executors. \n\n```bash\n$ cd multi-model-server\n$ ./run_circleci_tests.py smoke\n$ ./run_circleci_tests.py smoke -j modelarchiver\n$ ./run_circleci_tests.py smoke -e py36\n$ ./run_circleci_tests.py smoke -j modelarchiver -e py36\n```\n\n###### Checklist\n> 1. Make sure docker is running before you start local execution. \n> 2. Docker containers to have **at least 4GB RAM, 2 CPU**. \n> 3. If you are on a network with low bandwidth, we advise you to explicitly pull the docker images - \n> docker pull prashantsail/mms-build:python2.7 \n> docker pull prashantsail/mms-build:python3.6 \n\n`To avoid Pull Request build failures on github, developers should always make sure that their local builds pass.`\n" }, { "path": ".circleci/config.yml", "content": "version: 2.1\n\n\nexecutors:\n py36:\n docker:\n - image: prashantsail/mms-build:python3.6\n environment:\n _JAVA_OPTIONS: \"-Xmx2048m\"\n\n py27:\n docker:\n - image: prashantsail/mms-build:python2.7\n environment:\n _JAVA_OPTIONS: \"-Xmx2048m\"\n\n\ncommands:\n attach-mms-workspace:\n description: \"Attach the MMS directory which was saved into workspace\"\n steps:\n - attach_workspace:\n at: .\n\n install-mms-server:\n description: \"Install MMS server from a wheel\"\n steps:\n - run:\n name: Install MMS\n command: pip install dist/*.whl\n\n exeucute-api-tests:\n description: \"Execute API tests from a collection\"\n parameters:\n collection:\n type: enum\n enum: [management, inference, https]\n default: management\n steps:\n - run:\n name: Start MMS, Execute << parameters.collection >> API Tests, Stop MMS\n command: .circleci/scripts/linux_test_api.sh << parameters.collection >>\n - store_artifacts:\n name: Store server logs from << parameters.collection >> API tests\n path: mms_<< parameters.collection >>.log\n - store_artifacts:\n name: Store << parameters.collection >> API test results\n path: test/<< parameters.collection >>-api-report.html\n\n\njobs:\n build:\n parameters:\n executor:\n type: executor\n executor: << parameters.executor >>\n steps:\n - checkout\n - run:\n name: Build frontend\n command: .circleci/scripts/linux_build.sh\n - store_artifacts:\n name: Store gradle testng results\n path: frontend/server/build/reports/tests/test\n - persist_to_workspace:\n root: .\n paths:\n - .\n\n python-tests:\n parameters:\n executor:\n type: executor\n executor: << parameters.executor >>\n steps:\n - attach-mms-workspace\n - run:\n name: Execute python unit tests\n command: .circleci/scripts/linux_test_python.sh\n - store_artifacts:\n name: Store python Test results\n path: htmlcov\n\n api-tests:\n parameters:\n executor:\n type: executor\n executor: << parameters.executor >>\n steps:\n - attach-mms-workspace\n - install-mms-server\n - exeucute-api-tests:\n collection: management\n - exeucute-api-tests:\n collection: inference\n - exeucute-api-tests:\n collection: https\n\n benchmark:\n parameters:\n executor:\n type: executor\n executor: << parameters.executor >>\n steps:\n - attach-mms-workspace\n - install-mms-server\n - run:\n name: Start MMS, Execute benchmark tests, Stop MMS\n command: .circleci/scripts/linux_test_benchmark.sh\n - store_artifacts:\n name: Store server logs from benchmark tests\n path: mms.log\n - store_artifacts:\n name: Store Benchmark Latency resnet-18 results\n path: /tmp/MMSBenchmark/out/latency/resnet-18/report/\n destination: benchmark-latency-resnet-18\n\n modelarchiver:\n parameters:\n executor:\n type: executor\n executor: << parameters.executor >>\n steps:\n - checkout\n - run:\n name: Execute lint, unit and integration tests\n command: .circleci/scripts/linux_test_modelarchiver.sh\n - store_artifacts:\n name: Store unit tests results from model archiver tests\n path: model-archiver/results_units\n destination: units\n\n\nworkflows:\n version: 2\n\n smoke:\n jobs:\n - &build\n build:\n name: build-<< matrix.executor >>\n matrix: &matrix\n parameters:\n executor: [\"py27\", \"py36\"]\n - &modelarchiver\n modelarchiver:\n name: modelarchiver-<< matrix.executor >>\n matrix: *matrix\n - &python-tests\n python-tests:\n name: python-tests-<< matrix.executor >>\n requires:\n - build-<< matrix.executor >>\n matrix: *matrix\n\n nightly:\n triggers:\n - schedule:\n cron: \"0 0 * * *\"\n filters:\n branches:\n only:\n - master\n jobs:\n - *build\n - *modelarchiver\n - *python-tests\n - &api-tests\n api-tests:\n name: api-tests-<< matrix.executor >>\n requires:\n - build-<< matrix.executor >>\n matrix: *matrix\n\n weekly:\n triggers:\n - schedule:\n cron: \"0 0 * * 0\"\n filters:\n branches:\n only:\n - master\n jobs:\n - *build\n - benchmark:\n name: benchmark-<< matrix.executor >>\n requires:\n - build-<< matrix.executor >>\n matrix: *matrix\n" }, { "path": ".circleci/images/Dockerfile.python2.7", "content": "# Copyright 2018 Amazon.com, Inc. or its affiliates. All Rights Reserved.\n# Licensed under the Apache License, Version 2.0 (the \"License\").\n# You may not use this file except in compliance with the License.\n# A copy of the License is located at\n# http://www.apache.org/licenses/LICENSE-2.0\n# or in the \"license\" file accompanying this file. This file is distributed\n# on an \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either\n# express or implied. See the License for the specific language governing\n# permissions and limitations under the License.\n#\n\nFROM awsdeeplearningteam/mms-build:python2.7@sha256:2b743d6724dead806873cce1330f7b8a0197399a35af47dfd7035251fdade122\n\n# 2020 - Updated Build and Test dependencies\n\n# Python packages for MMS Server\nRUN pip install psutil \\\n && pip install future \\\n && pip install Pillow \\\n && pip install wheel \\\n && pip install twine \\\n && pip install requests \\\n && pip install mock \\\n && pip install numpy \\\n && pip install Image \\\n && pip install mxnet==1.5.0 \\\n && pip install enum34\n\n# Python packages for pytests\nRUN pip install pytest==4.0.0 \\\n && pip install pytest-cov \\\n && pip install pytest-mock\n\n# Python packages for benchmark\nRUN pip install pandas\n\n# Install NodeJS and packages for API tests\nRUN curl -sL https://deb.nodesource.com/setup_14.x | sudo -E bash - \\\n && sudo apt-get install -y nodejs \\\n && sudo npm install -g newman newman-reporter-html\n\n# Install jmeter for benchmark\n# ToDo: Remove --no-check-certificate; temporarily added to bypass jmeter-plugins.org's expired certificate\nRUN cd /opt \\\n && wget https://archive.apache.org/dist/jmeter/binaries/apache-jmeter-5.3.tgz \\\n && tar -xzf apache-jmeter-5.3.tgz \\\n && cd apache-jmeter-5.3 \\\n && ln -s /opt/apache-jmeter-5.3/bin/jmeter /usr/local/bin/jmeter \\\n && wget --no-check-certificate https://jmeter-plugins.org/get/ -O lib/ext/jmeter-plugins-manager-1.4.jar \\\n && wget http://search.maven.org/remotecontent?filepath=kg/apc/cmdrunner/2.2/cmdrunner-2.2.jar -O lib/cmdrunner-2.2.jar \\\n && java -cp lib/ext/jmeter-plugins-manager-1.4.jar org.jmeterplugins.repository.PluginManagerCMDInstaller \\\n && bin/PluginsManagerCMD.sh install jpgc-synthesis=2.1,jpgc-filterresults=2.1,jpgc-mergeresults=2.1,jpgc-cmd=2.1,jpgc-perfmon=2.1\n\n# bzt is used for performance regression test suite\n# bzt requires python 3.6 runtime.\n# Download pyenv, use pyenv to download python 3.6.5.\n# The downloaded python 3.6.5 is isolated and doesn't interfere with default python(2.7)\n# Only before starting the performance regression suite, py 3.6.5 is local installed(pyenv local 3.6.5) in test dir\n# !! MMS server will continue using Python 2.7 !!\nRUN curl https://pyenv.run | bash \\\n && $HOME/.pyenv/bin/pyenv install 3.6.5" }, { "path": ".circleci/images/Dockerfile.python3.6", "content": "# Copyright 2018 Amazon.com, Inc. or its affiliates. All Rights Reserved.\n# Licensed under the Apache License, Version 2.0 (the \"License\").\n# You may not use this file except in compliance with the License.\n# A copy of the License is located at\n# http://www.apache.org/licenses/LICENSE-2.0\n# or in the \"license\" file accompanying this file. This file is distributed\n# on an \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either\n# express or implied. See the License for the specific language governing\n# permissions and limitations under the License.\n#\n\nFROM awsdeeplearningteam/mms-build:python3.6@sha256:2c1afa8834907ceec641d254dffbf4bcc659ca2d00fd6f2872d7521f32c9fa2e\n\n# 2020 - Updated Build and Test dependencies\n\n# Python packages for MMS Server\nRUN pip install psutil \\\n && pip install future \\\n && pip install Pillow \\\n && pip install wheel \\\n && pip install twine \\\n && pip install requests \\\n && pip install mock \\\n && pip install numpy \\\n && pip install Image \\\n && pip install mxnet==1.5.0\n\n# Python packages for pytests\nRUN pip install pytest==4.0.0 \\\n && pip install pytest-cov \\\n && pip install pytest-mock\n\n# Python packages for benchmark\nRUN pip install pandas\n\n# Install NodeJS and packages for API tests\nRUN curl -sL https://deb.nodesource.com/setup_14.x | sudo -E bash - \\\n && sudo apt-get install -y nodejs \\\n && sudo npm install -g newman newman-reporter-html\n\n# Install jmeter for benchmark\n# ToDo: Remove --no-check-certificate; temporarily added to bypass jmeter-plugins.org's expired certificate\nRUN cd /opt \\\n && wget https://archive.apache.org/dist/jmeter/binaries/apache-jmeter-5.3.tgz \\\n && tar -xzf apache-jmeter-5.3.tgz \\\n && cd apache-jmeter-5.3 \\\n && ln -s /opt/apache-jmeter-5.3/bin/jmeter /usr/local/bin/jmeter \\\n && wget --no-check-certificate https://jmeter-plugins.org/get/ -O lib/ext/jmeter-plugins-manager-1.4.jar \\\n && wget http://search.maven.org/remotecontent?filepath=kg/apc/cmdrunner/2.2/cmdrunner-2.2.jar -O lib/cmdrunner-2.2.jar \\\n && java -cp lib/ext/jmeter-plugins-manager-1.4.jar org.jmeterplugins.repository.PluginManagerCMDInstaller \\\n && bin/PluginsManagerCMD.sh install jpgc-synthesis=2.1,jpgc-filterresults=2.1,jpgc-mergeresults=2.1,jpgc-cmd=2.1,jpgc-perfmon=2.1" }, { "path": ".circleci/scripts/linux_build.sh", "content": "#!/bin/bash\n\npython setup.py bdist_wheel --universal" }, { "path": ".circleci/scripts/linux_test_api.sh", "content": "#!/bin/bash\n\nMODEL_STORE_DIR='test/model_store'\n\nMMS_LOG_FILE_MANAGEMENT='mms_management.log'\nMMS_LOG_FILE_INFERENCE='mms_inference.log'\nMMS_LOG_FILE_HTTPS='mms_https.log'\nMMS_CONFIG_FILE_HTTPS='test/resources/config.properties'\n\nPOSTMAN_ENV_FILE='test/postman/environment.json'\nPOSTMAN_COLLECTION_MANAGEMENT='test/postman/management_api_test_collection.json'\nPOSTMAN_COLLECTION_INFERENCE='test/postman/inference_api_test_collection.json'\nPOSTMAN_COLLECTION_HTTPS='test/postman/https_test_collection.json'\nPOSTMAN_DATA_FILE_INFERENCE='test/postman/inference_data.json'\n\nREPORT_FILE_MANAGEMENT='test/management-api-report.html'\nREPORT_FILE_INFERENCE='test/inference-api-report.html'\nREPORT_FILE_HTTPS='test/https-api-report.html'\n\nstart_mms_server() {\n multi-model-server --start --model-store $1 >> $2 2>&1\n sleep 10\n}\n\nstart_mms_secure_server() {\n multi-model-server --start --mms-config $MMS_CONFIG_FILE_HTTPS --model-store $1 >> $2 2>&1\n sleep 10\n}\n\nstop_mms_server() {\n multi-model-server --stop\n}\n\ntrigger_management_tests(){\n start_mms_server $MODEL_STORE_DIR $MMS_LOG_FILE_MANAGEMENT\n newman run -e $POSTMAN_ENV_FILE $POSTMAN_COLLECTION_MANAGEMENT \\\n -r cli,html --reporter-html-export $REPORT_FILE_MANAGEMENT --verbose\n stop_mms_server\n}\n\ntrigger_inference_tests(){\n start_mms_server $MODEL_STORE_DIR $MMS_LOG_FILE_INFERENCE\n newman run -e $POSTMAN_ENV_FILE $POSTMAN_COLLECTION_INFERENCE -d $POSTMAN_DATA_FILE_INFERENCE \\\n -r cli,html --reporter-html-export $REPORT_FILE_INFERENCE --verbose\n stop_mms_server\n}\n\ntrigger_https_tests(){\n start_mms_secure_server $MODEL_STORE_DIR $MMS_LOG_FILE_HTTPS\n newman run --insecure -e $POSTMAN_ENV_FILE $POSTMAN_COLLECTION_HTTPS \\\n -r cli,html --reporter-html-export $REPORT_FILE_HTTPS --verbose\n stop_mms_server\n}\n\nmkdir -p $MODEL_STORE_DIR\n\ncase $1 in\n 'management')\n trigger_management_tests\n ;;\n 'inference')\n trigger_inference_tests\n ;;\n 'https')\n trigger_https_tests\n ;;\n 'ALL')\n trigger_management_tests\n trigger_inference_tests\n trigger_https_tests\n ;;\n *)\n echo $1 'Invalid'\n echo 'Please specify any one of - management | inference | https | ALL'\n exit 1\n ;;\nesac" }, { "path": ".circleci/scripts/linux_test_benchmark.sh", "content": "#!/bin/bash\n\n# Hack needed to make it work with existing benchmark.py\n# benchmark.py expects jmeter to be present at a very specific location\nmkdir -p /home/ubuntu/.linuxbrew/Cellar/jmeter/5.3/libexec/bin/\nln -s /opt/apache-jmeter-5.3/bin/jmeter /home/ubuntu/.linuxbrew/Cellar/jmeter/5.3/libexec/bin/jmeter\n\nmulti-model-server --start >> mms.log 2>&1\nsleep 30\n\ncd benchmarks\npython benchmark.py latency\n\nmulti-model-server --stop" }, { "path": ".circleci/scripts/linux_test_modelarchiver.sh", "content": "#!/bin/bash\n\ncd model-archiver/\n\n# Lint test\npylint -rn --rcfile=./model_archiver/tests/pylintrc model_archiver/.\n\n# Execute python unit tests\npython -m pytest --cov-report html:results_units --cov=./ model_archiver/tests/unit_tests\n\n\n# Install model archiver module\npip install .\n\n# Execute integration tests\npython -m pytest model_archiver/tests/integ_tests\n# ToDo - Report for Integration tests ?" }, { "path": ".circleci/scripts/linux_test_perf_regression.sh", "content": "#!/bin/bash\n\nmulti-model-server --start \\\n --models squeezenet=https://s3.amazonaws.com/model-server/model_archive_1.0/squeezenet_v1.1.mar \\\n >> mms.log 2>&1\nsleep 90\n\ncd performance_regression\n\n# Only on a python 2 environment -\nPY_MAJOR_VER=$(python -c 'import sys; major = sys.version_info.major; print(major);')\nif [ $PY_MAJOR_VER -eq 2 ]; then\n # Hack to use python 3.6.5 for bzt installation and execution\n export PATH=\"/root/.pyenv/bin:/root/.pyenv/shims:$PATH\"\n pyenv local 3.6.5\nfi\n\n# Install dependencies\npip install bzt\n\ncurl -O https://s3.amazonaws.com/model-server/inputs/kitten.jpg\nbzt -o modules.jmeter.path=/opt/apache-jmeter-5.3/bin/jmeter \\\n -o settings.artifacts-dir=/tmp/mms-performance-regression/ \\\n -o modules.console.disable=true \\\n imageInputModelPlan.jmx.yaml \\\n -report\n\nmulti-model-server --stop" }, { "path": ".circleci/scripts/linux_test_python.sh", "content": "#!/bin/bash\n\n# Lint Test\npylint -rn --rcfile=./mms/tests/pylintrc mms/.\n\n# Execute python tests\npython -m pytest --cov-report html:htmlcov --cov=mms/ mms/tests/unit_tests/" }, { "path": ".coveragerc", "content": "# Copyright 2018 Amazon.com, Inc. or its affiliates. All Rights Reserved.\n# Licensed under the Apache License, Version 2.0 (the \"License\").\n# You may not use this file except in compliance with the License.\n# A copy of the License is located at\n# http://www.apache.org/licenses/LICENSE-2.0\n# or in the \"license\" file accompanying this file. This file is distributed\n# on an \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either\n# express or implied. See the License for the specific language governing\n# permissions and limitations under the License.\n\n[report]\nexclude_lines =\n pragma: no cover\n if __name__ == .__main__.:\n if __name__ == \"__main__\" :\n\n[run]\nbranch = True\nomit = */__init__.py\n mms/tests/*\n mms/utils/model_server_error_codes.py\n mms/utils/timeit_decorator.py\n mms/storage.py\n mms/metrics/system_metrics.py\n mms/utils/mxnet/*\n mms/examples/metric_push_example.py\n mms/model_service/*\n" }, { "path": ".github/PULL_REQUEST_TEMPLATE.md", "content": "Before or while filing an issue please feel free to join our [ slack channel](https://join.slack.com/t/mms-awslabs/shared_invite/enQtNDk4MTgzNDc5NzE4LTBkYTAwMjBjMTVmZTdkODRmYTZkNjdjZGYxZDI0ODhiZDdlM2Y0ZGJiZTczMGY3Njc4MmM3OTQ0OWI2ZDMyNGQ) to get in touch with development team, ask questions, find out what's cooking and more!\n\n## Issue #, if available:\n\n## Description of changes:\n\n## Testing done:\n\n**To run CI tests on your changes refer [README.md](https://github.com/awslabs/multi-model-server/blob/master/ci/README.md)**\n\nBy submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.\n" }, { "path": ".gitignore", "content": ".gradle\n\n# Byte-compiled / optimized / DLL files\n__pycache__/\n*.py[cod]\n*$py.class\n\n# C extensions\n*.so\n\n# Distribution / packaging\n.Python\nenv/\nbuild/\ndevelop-eggs/\ndist/\ndownloads/\neggs/\n.eggs/\nlib/\nlib64/\nparts/\nsdist/\nvar/\nwheels/\n*.egg-info/\n.installed.cfg\n*.egg\n\n# PyInstaller\n# Usually these files are written by a python script from a template\n# before PyInstaller builds the exe, so as to inject date/other infos into it.\n*.manifest\n*.spec\n\n# Installer logs\npip-log.txt\npip-delete-this-directory.txt\n\n# Unit test / coverage reports\nhtmlcov/\n.tox/\n.coverage\n.coverage.*\n.cache\nnosetests.xml\ncoverage.xml\n*.cover\n.hypothesis/\n.pytest_cache/\n\n# Translations\n*.mo\n*.pot\n\n# Django stuff:\n*.log\nlocal_settings.py\n\n# Flask stuff:\ninstance/\n.webassets-cache\n\n# Scrapy stuff:\n.scrapy\n\n# Sphinx documentation\ndocs/_build/\n\n# PyBuilder\ntarget/\n\n# Jupyter Notebook\n.ipynb_checkpoints\n\n# pyenv\n.python-version\n\n# celery beat schedule file\ncelerybeat-schedule\n\n# SageMath parsed files\n*.sage.py\n\n# dotenv\n.env\n\n# virtualenv\n.venv\nvenv/\nENV/\n\n# Spyder project settings\n.spyderproject\n.spyproject\n\n# Rope project settings\n.ropeproject\n\n# mkdocs documentation\n/site\n\n# mypy\n.mypy_cache/\n\n# mac\n.DS_Store\n\n# PyCharm\n.idea/\n\n# Log\n*.log.*\n\n# Model\n*.model\n\n# Pictures\n*.jpg\n\n# Prop file in benchmark\nbenchmarks/*.properties\n\n# intellij files\n*.iml\n\n# MMS files\nmms/frontend\nmms/plugins" }, { "path": "LICENSE", "content": " Apache License\n Version 2.0, January 2004\n http://www.apache.org/licenses/\n\n TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION\n\n 1. Definitions.\n\n \"License\" shall mean the terms and conditions for use, reproduction,\n and distribution as defined by Sections 1 through 9 of this document.\n\n \"Licensor\" shall mean the copyright owner or entity authorized by\n the copyright owner that is granting the License.\n\n \"Legal Entity\" shall mean the union of the acting entity and all\n other entities that control, are controlled by, or are under common\n control with that entity. For the purposes of this definition,\n \"control\" means (i) the power, direct or indirect, to cause the\n direction or management of such entity, whether by contract or\n otherwise, or (ii) ownership of fifty percent (50%) or more of the\n outstanding shares, or (iii) beneficial ownership of such entity.\n\n \"You\" (or \"Your\") shall mean an individual or Legal Entity\n exercising permissions granted by this License.\n\n \"Source\" form shall mean the preferred form for making modifications,\n including but not limited to software source code, documentation\n source, and configuration files.\n\n \"Object\" form shall mean any form resulting from mechanical\n transformation or translation of a Source form, including but\n not limited to compiled object code, generated documentation,\n and conversions to other media types.\n\n \"Work\" shall mean the work of authorship, whether in Source or\n Object form, made available under the License, as indicated by a\n copyright notice that is included in or attached to the work\n (an example is provided in the Appendix below).\n\n \"Derivative Works\" shall mean any work, whether in Source or Object\n form, that is based on (or derived from) the Work and for which the\n editorial revisions, annotations, elaborations, or other modifications\n represent, as a whole, an original work of authorship. For the purposes\n of this License, Derivative Works shall not include works that remain\n separable from, or merely link (or bind by name) to the interfaces of,\n the Work and Derivative Works thereof.\n\n \"Contribution\" shall mean any work of authorship, including\n the original version of the Work and any modifications or additions\n to that Work or Derivative Works thereof, that is intentionally\n submitted to Licensor for inclusion in the Work by the copyright owner\n or by an individual or Legal Entity authorized to submit on behalf of\n the copyright owner. For the purposes of this definition, \"submitted\"\n means any form of electronic, verbal, or written communication sent\n to the Licensor or its representatives, including but not limited to\n communication on electronic mailing lists, source code control systems,\n and issue tracking systems that are managed by, or on behalf of, the\n Licensor for the purpose of discussing and improving the Work, but\n excluding communication that is conspicuously marked or otherwise\n designated in writing by the copyright owner as \"Not a Contribution.\"\n\n \"Contributor\" shall mean Licensor and any individual or Legal Entity\n on behalf of whom a Contribution has been received by Licensor and\n subsequently incorporated within the Work.\n\n 2. Grant of Copyright License. Subject to the terms and conditions of\n this License, each Contributor hereby grants to You a perpetual,\n worldwide, non-exclusive, no-charge, royalty-free, irrevocable\n copyright license to reproduce, prepare Derivative Works of,\n publicly display, publicly perform, sublicense, and distribute the\n Work and such Derivative Works in Source or Object form.\n\n 3. Grant of Patent License. Subject to the terms and conditions of\n this License, each Contributor hereby grants to You a perpetual,\n worldwide, non-exclusive, no-charge, royalty-free, irrevocable\n (except as stated in this section) patent license to make, have made,\n use, offer to sell, sell, import, and otherwise transfer the Work,\n where such license applies only to those patent claims licensable\n by such Contributor that are necessarily infringed by their\n Contribution(s) alone or by combination of their Contribution(s)\n with the Work to which such Contribution(s) was submitted. If You\n institute patent litigation against any entity (including a\n cross-claim or counterclaim in a lawsuit) alleging that the Work\n or a Contribution incorporated within the Work constitutes direct\n or contributory patent infringement, then any patent licenses\n granted to You under this License for that Work shall terminate\n as of the date such litigation is filed.\n\n 4. Redistribution. You may reproduce and distribute copies of the\n Work or Derivative Works thereof in any medium, with or without\n modifications, and in Source or Object form, provided that You\n meet the following conditions:\n\n (a) You must give any other recipients of the Work or\n Derivative Works a copy of this License; and\n\n (b) You must cause any modified files to carry prominent notices\n stating that You changed the files; and\n\n (c) You must retain, in the Source form of any Derivative Works\n that You distribute, all copyright, patent, trademark, and\n attribution notices from the Source form of the Work,\n excluding those notices that do not pertain to any part of\n the Derivative Works; and\n\n (d) If the Work includes a \"NOTICE\" text file as part of its\n distribution, then any Derivative Works that You distribute must\n include a readable copy of the attribution notices contained\n within such NOTICE file, excluding those notices that do not\n pertain to any part of the Derivative Works, in at least one\n of the following places: within a NOTICE text file distributed\n as part of the Derivative Works; within the Source form or\n documentation, if provided along with the Derivative Works; or,\n within a display generated by the Derivative Works, if and\n wherever such third-party notices normally appear. The contents\n of the NOTICE file are for informational purposes only and\n do not modify the License. You may add Your own attribution\n notices within Derivative Works that You distribute, alongside\n or as an addendum to the NOTICE text from the Work, provided\n that such additional attribution notices cannot be construed\n as modifying the License.\n\n You may add Your own copyright statement to Your modifications and\n may provide additional or different license terms and conditions\n for use, reproduction, or distribution of Your modifications, or\n for any such Derivative Works as a whole, provided Your use,\n reproduction, and distribution of the Work otherwise complies with\n the conditions stated in this License.\n\n 5. Submission of Contributions. Unless You explicitly state otherwise,\n any Contribution intentionally submitted for inclusion in the Work\n by You to the Licensor shall be under the terms and conditions of\n this License, without any additional terms or conditions.\n Notwithstanding the above, nothing herein shall supersede or modify\n the terms of any separate license agreement you may have executed\n with Licensor regarding such Contributions.\n\n 6. Trademarks. This License does not grant permission to use the trade\n names, trademarks, service marks, or product names of the Licensor,\n except as required for reasonable and customary use in describing the\n origin of the Work and reproducing the content of the NOTICE file.\n\n 7. Disclaimer of Warranty. Unless required by applicable law or\n agreed to in writing, Licensor provides the Work (and each\n Contributor provides its Contributions) on an \"AS IS\" BASIS,\n WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or\n implied, including, without limitation, any warranties or conditions\n of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A\n PARTICULAR PURPOSE. You are solely responsible for determining the\n appropriateness of using or redistributing the Work and assume any\n risks associated with Your exercise of permissions under this License.\n\n 8. Limitation of Liability. In no event and under no legal theory,\n whether in tort (including negligence), contract, or otherwise,\n unless required by applicable law (such as deliberate and grossly\n negligent acts) or agreed to in writing, shall any Contributor be\n liable to You for damages, including any direct, indirect, special,\n incidental, or consequential damages of any character arising as a\n result of this License or out of the use or inability to use the\n Work (including but not limited to damages for loss of goodwill,\n work stoppage, computer failure or malfunction, or any and all\n other commercial damages or losses), even if such Contributor\n has been advised of the possibility of such damages.\n\n 9. Accepting Warranty or Additional Liability. While redistributing\n the Work or Derivative Works thereof, You may choose to offer,\n and charge a fee for, acceptance of support, warranty, indemnity,\n or other liability obligations and/or rights consistent with this\n License. However, in accepting such obligations, You may act only\n on Your own behalf and on Your sole responsibility, not on behalf\n of any other Contributor, and only if You agree to indemnify,\n defend, and hold each Contributor harmless for any liability\n incurred by, or claims asserted against, such Contributor by reason\n of your accepting any such warranty or additional liability.\n\n END OF TERMS AND CONDITIONS\n\n APPENDIX: How to apply the Apache License to your work.\n\n To apply the Apache License to your work, attach the following\n boilerplate notice, with the fields enclosed by brackets \"{}\"\n replaced with your own identifying information. (Don't include\n the brackets!) The text should be enclosed in the appropriate\n comment syntax for the file format. We also recommend that a\n file or class name and description of purpose be included on the\n same \"printed page\" as the copyright notice for easier\n identification within third-party archives.\n\n Copyright {yyyy} {name of copyright owner}\n\n Licensed under the Apache License, Version 2.0 (the \"License\");\n you may not use this file except in compliance with the License.\n You may obtain a copy of the License at\n\n http://www.apache.org/licenses/LICENSE-2.0\n\n Unless required by applicable law or agreed to in writing, software\n distributed under the License is distributed on an \"AS IS\" BASIS,\n WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n See the License for the specific language governing permissions and\n limitations under the License.\n" }, { "path": "LICENSE.txt", "content": " Apache License\n Version 2.0, January 2004\n http://www.apache.org/licenses/\n\n TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION\n\n 1. Definitions.\n\n \"License\" shall mean the terms and conditions for use, reproduction,\n and distribution as defined by Sections 1 through 9 of this document.\n\n \"Licensor\" shall mean the copyright owner or entity authorized by\n the copyright owner that is granting the License.\n\n \"Legal Entity\" shall mean the union of the acting entity and all\n other entities that control, are controlled by, or are under common\n control with that entity. For the purposes of this definition,\n \"control\" means (i) the power, direct or indirect, to cause the\n direction or management of such entity, whether by contract or\n otherwise, or (ii) ownership of fifty percent (50%) or more of the\n outstanding shares, or (iii) beneficial ownership of such entity.\n\n \"You\" (or \"Your\") shall mean an individual or Legal Entity\n exercising permissions granted by this License.\n\n \"Source\" form shall mean the preferred form for making modifications,\n including but not limited to software source code, documentation\n source, and configuration files.\n\n \"Object\" form shall mean any form resulting from mechanical\n transformation or translation of a Source form, including but\n not limited to compiled object code, generated documentation,\n and conversions to other media types.\n\n \"Work\" shall mean the work of authorship, whether in Source or\n Object form, made available under the License, as indicated by a\n copyright notice that is included in or attached to the work\n (an example is provided in the Appendix below).\n\n \"Derivative Works\" shall mean any work, whether in Source or Object\n form, that is based on (or derived from) the Work and for which the\n editorial revisions, annotations, elaborations, or other modifications\n represent, as a whole, an original work of authorship. For the purposes\n of this License, Derivative Works shall not include works that remain\n separable from, or merely link (or bind by name) to the interfaces of,\n the Work and Derivative Works thereof.\n\n \"Contribution\" shall mean any work of authorship, including\n the original version of the Work and any modifications or additions\n to that Work or Derivative Works thereof, that is intentionally\n submitted to Licensor for inclusion in the Work by the copyright owner\n or by an individual or Legal Entity authorized to submit on behalf of\n the copyright owner. For the purposes of this definition, \"submitted\"\n means any form of electronic, verbal, or written communication sent\n to the Licensor or its representatives, including but not limited to\n communication on electronic mailing lists, source code control systems,\n and issue tracking systems that are managed by, or on behalf of, the\n Licensor for the purpose of discussing and improving the Work, but\n excluding communication that is conspicuously marked or otherwise\n designated in writing by the copyright owner as \"Not a Contribution.\"\n\n \"Contributor\" shall mean Licensor and any individual or Legal Entity\n on behalf of whom a Contribution has been received by Licensor and\n subsequently incorporated within the Work.\n\n 2. Grant of Copyright License. Subject to the terms and conditions of\n this License, each Contributor hereby grants to You a perpetual,\n worldwide, non-exclusive, no-charge, royalty-free, irrevocable\n copyright license to reproduce, prepare Derivative Works of,\n publicly display, publicly perform, sublicense, and distribute the\n Work and such Derivative Works in Source or Object form.\n\n 3. Grant of Patent License. Subject to the terms and conditions of\n this License, each Contributor hereby grants to You a perpetual,\n worldwide, non-exclusive, no-charge, royalty-free, irrevocable\n (except as stated in this section) patent license to make, have made,\n use, offer to sell, sell, import, and otherwise transfer the Work,\n where such license applies only to those patent claims licensable\n by such Contributor that are necessarily infringed by their\n Contribution(s) alone or by combination of their Contribution(s)\n with the Work to which such Contribution(s) was submitted. If You\n institute patent litigation against any entity (including a\n cross-claim or counterclaim in a lawsuit) alleging that the Work\n or a Contribution incorporated within the Work constitutes direct\n or contributory patent infringement, then any patent licenses\n granted to You under this License for that Work shall terminate\n as of the date such litigation is filed.\n\n 4. Redistribution. You may reproduce and distribute copies of the\n Work or Derivative Works thereof in any medium, with or without\n modifications, and in Source or Object form, provided that You\n meet the following conditions:\n\n (a) You must give any other recipients of the Work or\n Derivative Works a copy of this License; and\n\n (b) You must cause any modified files to carry prominent notices\n stating that You changed the files; and\n\n (c) You must retain, in the Source form of any Derivative Works\n that You distribute, all copyright, patent, trademark, and\n attribution notices from the Source form of the Work,\n excluding those notices that do not pertain to any part of\n the Derivative Works; and\n\n (d) If the Work includes a \"NOTICE\" text file as part of its\n distribution, then any Derivative Works that You distribute must\n include a readable copy of the attribution notices contained\n within such NOTICE file, excluding those notices that do not\n pertain to any part of the Derivative Works, in at least one\n of the following places: within a NOTICE text file distributed\n as part of the Derivative Works; within the Source form or\n documentation, if provided along with the Derivative Works; or,\n within a display generated by the Derivative Works, if and\n wherever such third-party notices normally appear. The contents\n of the NOTICE file are for informational purposes only and\n do not modify the License. You may add Your own attribution\n notices within Derivative Works that You distribute, alongside\n or as an addendum to the NOTICE text from the Work, provided\n that such additional attribution notices cannot be construed\n as modifying the License.\n\n You may add Your own copyright statement to Your modifications and\n may provide additional or different license terms and conditions\n for use, reproduction, or distribution of Your modifications, or\n for any such Derivative Works as a whole, provided Your use,\n reproduction, and distribution of the Work otherwise complies with\n the conditions stated in this License.\n\n 5. Submission of Contributions. Unless You explicitly state otherwise,\n any Contribution intentionally submitted for inclusion in the Work\n by You to the Licensor shall be under the terms and conditions of\n this License, without any additional terms or conditions.\n Notwithstanding the above, nothing herein shall supersede or modify\n the terms of any separate license agreement you may have executed\n with Licensor regarding such Contributions.\n\n 6. Trademarks. This License does not grant permission to use the trade\n names, trademarks, service marks, or product names of the Licensor,\n except as required for reasonable and customary use in describing the\n origin of the Work and reproducing the content of the NOTICE file.\n\n 7. Disclaimer of Warranty. Unless required by applicable law or\n agreed to in writing, Licensor provides the Work (and each\n Contributor provides its Contributions) on an \"AS IS\" BASIS,\n WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or\n implied, including, without limitation, any warranties or conditions\n of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A\n PARTICULAR PURPOSE. You are solely responsible for determining the\n appropriateness of using or redistributing the Work and assume any\n risks associated with Your exercise of permissions under this License.\n\n 8. Limitation of Liability. In no event and under no legal theory,\n whether in tort (including negligence), contract, or otherwise,\n unless required by applicable law (such as deliberate and grossly\n negligent acts) or agreed to in writing, shall any Contributor be\n liable to You for damages, including any direct, indirect, special,\n incidental, or consequential damages of any character arising as a\n result of this License or out of the use or inability to use the\n Work (including but not limited to damages for loss of goodwill,\n work stoppage, computer failure or malfunction, or any and all\n other commercial damages or losses), even if such Contributor\n has been advised of the possibility of such damages.\n\n 9. Accepting Warranty or Additional Liability. While redistributing\n the Work or Derivative Works thereof, You may choose to offer,\n and charge a fee for, acceptance of support, warranty, indemnity,\n or other liability obligations and/or rights consistent with this\n License. However, in accepting such obligations, You may act only\n on Your own behalf and on Your sole responsibility, not on behalf\n of any other Contributor, and only if You agree to indemnify,\n defend, and hold each Contributor harmless for any liability\n incurred by, or claims asserted against, such Contributor by reason\n of your accepting any such warranty or additional liability.\n\n END OF TERMS AND CONDITIONS\n\n APPENDIX: How to apply the Apache License to your work.\n\n To apply the Apache License to your work, attach the following\n boilerplate notice, with the fields enclosed by brackets \"{}\"\n replaced with your own identifying information. (Don't include\n the brackets!) The text should be enclosed in the appropriate\n comment syntax for the file format. We also recommend that a\n file or class name and description of purpose be included on the\n same \"printed page\" as the copyright notice for easier\n identification within third-party archives.\n\n Copyright {yyyy} {name of copyright owner}\n\n Licensed under the Apache License, Version 2.0 (the \"License\");\n you may not use this file except in compliance with the License.\n You may obtain a copy of the License at\n\n http://www.apache.org/licenses/LICENSE-2.0\n\n Unless required by applicable law or agreed to in writing, software\n distributed under the License is distributed on an \"AS IS\" BASIS,\n WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n See the License for the specific language governing permissions and\n limitations under the License.\n" }, { "path": "MANIFEST.in", "content": "include mms/frontend/model-server.jar\ninclude PyPiDescription.rst\ninclude mms/configs/*\n" }, { "path": "PyPiDescription.rst", "content": "Project Description\n===================\n\nMulti Model Server (MMS) is a flexible and easy to use tool for\nserving deep learning models exported from `MXNet `__\nor the Open Neural Network Exchange (`ONNX `__).\n\nUse the MMS Server CLI, or the pre-configured Docker images, to start a\nservice that sets up HTTP endpoints to handle model inference requests.\n\nDetailed documentation and examples are provided in the `docs\nfolder `__.\n\nPrerequisites\n-------------\n\n* **java 8**: Required. MMS use java to serve HTTP requests. You must install java 8 (or later) and make sure java is on available in $PATH environment variable *before* installing MMS. If you have multiple java installed, you can use $JAVA_HOME environment vairable to control which java to use.\n* **mxnet**: `mxnet` will not be installed by default with MMS 1.0 any more. You have to install it manually if you use MxNet.\n\nFor ubuntu:\n::\n\n sudo apt-get install openjdk-8-jre-headless\n\n\nFor centos\n::\n\n sudo yum install java-1.8.0-openjdk\n\n\nFor Mac:\n::\n\n brew tap caskroom/versions\n brew update\n brew cask install java8\n\n\nInstall MxNet:\n::\n\n pip install mxnet\n\nMXNet offers MKL pip packages that will be much faster when running on Intel hardware.\nTo install mkl package for CPU:\n::\n\n pip install mxnet-mkl\n\nor for GPU instance:\n\n::\n\n pip install mxnet-cu92mkl\n\n\nInstallation\n------------\n\n::\n\n pip install multi-model-server\n\nDevelopment\n-----------\n\nWe welcome new contributors of all experience levels. For information on\nhow to install MMS for development, refer to the `MMS\ndocs `__.\n\nImportant links\n---------------\n\n- `Official source code\n repo `__\n- `Download\n releases `__\n- `Issue\n tracker `__\n\nSource code\n-----------\n\nYou can check the latest source code as follows:\n\n::\n\n git clone https://github.com/awslabs/multi-model-server.git\n\nTesting\n-------\n\nAfter installation, try out the MMS Quickstart for\n\n- `Serving a Model `__\n- `Create a Model Archive `__.\n\nHelp and Support\n----------------\n\n- `Documentation `__\n- `Forum `__\n\nCitation\n--------\n\nIf you use MMS in a publication or project, please cite MMS:\nhttps://github.com/awslabs/multi-model-server\n" }, { "path": "README.md", "content": "Multi Model Server\n=======\n\n| ubuntu/python-2.7 | ubuntu/python-3.6 |\n|---------|---------|\n| ![Python3 Build Status](https://codebuild.us-east-1.amazonaws.com/badges?uuid=eyJlbmNyeXB0ZWREYXRhIjoicGZ6dXFmMU54UGxDaGsxUDhXclJLcFpHTnFMNld6cW5POVpNclc4Vm9BUWJNamZKMGdzbk1lOU92Z0VWQVZJTThsRUttOW8rUzgxZ2F0Ull1U1VkSHo0PSIsIml2UGFyYW1ldGVyU3BlYyI6IkJJaFc1QTEwRGhwUXY1dDgiLCJtYXRlcmlhbFNldFNlcmlhbCI6MX0%3D&branch=master) | ![Python2 Build Status](https://codebuild.us-east-1.amazonaws.com/badges?uuid=eyJlbmNyeXB0ZWREYXRhIjoiYVdIajEwVW9uZ3cvWkZqaHlaRGNUU2M0clE2aUVjelJranJoYTI3S1lHT3R5THJXdklzejU2UVM5NWlUTWdwaVVJalRwYi9GTnJ1aUxiRXIvTGhuQ2g0PSIsIml2UGFyYW1ldGVyU3BlYyI6IjArcHVCaFgvR1pTN1JoSG4iLCJtYXRlcmlhbFNldFNlcmlhbCI6MX0%3D&branch=master) |\n\nMulti Model Server (MMS) is a flexible and easy to use tool for serving deep learning models trained using any ML/DL framework.\n\nUse the MMS Server CLI, or the pre-configured Docker images, to start a service that sets up HTTP endpoints to handle model inference requests.\n\nA quick overview and examples for both serving and packaging are provided below. Detailed documentation and examples are provided in the [docs folder](docs/README.md).\n\nJoin our [ slack channel](https://join.slack.com/t/mms-awslabs/shared_invite/zt-6cv1kx46-MBTOPLNDwmyBynEvFBsNkQ) to get in touch with development team, ask questions, find out what's cooking and more!\n\n## Contents of this Document\n* [Quick Start](#quick-start)\n* [Serve a Model](#serve-a-model)\n* [Other Features](#other-features)\n* [External demos powered by MMS](#external-demos-powered-by-mms)\n* [Contributing](#contributing)\n\n\n## Other Relevant Documents\n* [Latest Version Docs](docs/README.md)\n* [v0.4.0 Docs](https://github.com/awslabs/multi-model-server/blob/v0.4.0/docs/README.md)\n* [Migrating from v0.4.0 to v1.0.0](docs/migration.md)\n\n## Quick Start\n### Prerequisites\nBefore proceeding further with this document, make sure you have the following prerequisites.\n1. Ubuntu, CentOS, or macOS. Windows support is experimental. The following instructions will focus on Linux and macOS only.\n1. Python - Multi Model Server requires python to run the workers.\n1. pip - Pip is a python package management system.\n1. Java 8 - Multi Model Server requires Java 8 to start. You have the following options for installing Java 8:\n\n For Ubuntu:\n ```bash\n sudo apt-get install openjdk-8-jre-headless\n ```\n\n For CentOS:\n ```bash\n sudo yum install java-1.8.0-openjdk\n ```\n\n For macOS:\n ```bash\n brew tap homebrew/cask-versions\n brew update\n brew cask install adoptopenjdk8\n ```\n\n### Installing Multi Model Server with pip\n\n#### Setup\n\n**Step 1:** Setup a Virtual Environment\n\nWe recommend installing and running Multi Model Server in a virtual environment. It's a good practice to run and install all of the Python dependencies in virtual environments. This will provide isolation of the dependencies and ease dependency management.\n\nOne option is to use Virtualenv. This is used to create virtual Python environments. You may install and activate a virtualenv for Python 2.7 as follows:\n\n```bash\npip install virtualenv\n```\n\nThen create a virtual environment:\n```bash\n# Assuming we want to run python2.7 in /usr/local/bin/python2.7\nvirtualenv -p /usr/local/bin/python2.7 /tmp/pyenv2\n# Enter this virtual environment as follows\nsource /tmp/pyenv2/bin/activate\n```\n\nRefer to the [Virtualenv documentation](https://virtualenv.pypa.io/en/stable/) for further information.\n\n**Step 2:** Install MXNet\nMMS won't install the MXNet engine by default. If it isn't already installed in your virtual environment, you must install one of the MXNet pip packages.\n\nFor CPU inference, `mxnet-mkl` is recommended. Install it as follows:\n\n```bash\n# Recommended for running Multi Model Server on CPU hosts\npip install mxnet-mkl\n```\n\nFor GPU inference, `mxnet-cu92mkl` is recommended. Install it as follows:\n\n```bash\n# Recommended for running Multi Model Server on GPU hosts\npip install mxnet-cu92mkl\n```\n\n**Step 3:** Install or Upgrade MMS as follows:\n\n```bash\n# Install latest released version of multi-model-server \npip install multi-model-server\n```\n\nTo upgrade from a previous version of `multi-model-server`, please refer [migration reference](docs/migration.md) document.\n\n**Notes:**\n* A minimal version of `model-archiver` will be installed with MMS as dependency. See [model-archiver](model-archiver/README.md) for more options and details.\n* See the [advanced installation](docs/install.md) page for more options and troubleshooting.\n\n### Serve a Model\n\nOnce installed, you can get MMS model server up and running very quickly. Try out `--help` to see all the CLI options available.\n\n```bash\nmulti-model-server --help\n```\n\nFor this quick start, we'll skip over most of the features, but be sure to take a look at the [full server docs](docs/server.md) when you're ready.\n\nHere is an easy example for serving an object classification model:\n```bash\nmulti-model-server --start --models squeezenet=https://s3.amazonaws.com/model-server/model_archive_1.0/squeezenet_v1.1.mar\n```\n\nWith the command above executed, you have MMS running on your host, listening for inference requests. **Please note, that if you specify model(s) during MMS start - it will automatically scale backend workers to the number equal to available vCPUs (if you run on CPU instance) or to the number of available GPUs (if you run on GPU instance). In case of powerful hosts with a lot of compute resoures (vCPUs or GPUs) this start up and autoscaling process might take considerable time. If you would like to minimize MMS start up time you can try to avoid registering and scaling up model during start up time and move that to a later point by using corresponding [Management API](docs/management_api.md#register-a-model) calls (this allows finer grain control to how much resources are allocated for any particular model).**\n\nTo test it out, you can open a new terminal window next to the one running MMS. Then you can use `curl` to download one of these [cute pictures of a kitten](https://www.google.com/search?q=cute+kitten&tbm=isch&hl=en&cr=&safe=images) and curl's `-o` flag will name it `kitten.jpg` for you. Then you will `curl` a `POST` to the MMS predict endpoint with the kitten's image.\n\n![kitten](docs/images/kitten_small.jpg)\n\nIn the example below, we provide a shortcut for these steps.\n\n```bash\ncurl -O https://s3.amazonaws.com/model-server/inputs/kitten.jpg\ncurl -X POST http://127.0.0.1:8080/predictions/squeezenet -T kitten.jpg\n```\n\nThe predict endpoint will return a prediction response in JSON. It will look something like the following result:\n\n```json\n[\n {\n \"probability\": 0.8582232594490051,\n \"class\": \"n02124075 Egyptian cat\"\n },\n {\n \"probability\": 0.09159987419843674,\n \"class\": \"n02123045 tabby, tabby cat\"\n },\n {\n \"probability\": 0.0374876894056797,\n \"class\": \"n02123159 tiger cat\"\n },\n {\n \"probability\": 0.006165083032101393,\n \"class\": \"n02128385 leopard, Panthera pardus\"\n },\n {\n \"probability\": 0.0031716004014015198,\n \"class\": \"n02127052 lynx, catamount\"\n }\n]\n```\n\nYou will see this result in the response to your `curl` call to the predict endpoint, and in the server logs in the terminal window running MMS. It's also being [logged locally with metrics](docs/metrics.md).\n\nOther models can be downloaded from the [model zoo](docs/model_zoo.md), so try out some of those as well.\n\nNow you've seen how easy it can be to serve a deep learning model with MMS! [Would you like to know more?](docs/server.md)\n\n### Stopping the running model server\nTo stop the current running model-server instance, run the following command:\n```bash\n$ multi-model-server --stop\n```\nYou would see output specifying that multi-model-server has stopped.\n\n### Create a Model Archive\n\nMMS enables you to package up all of your model artifacts into a single model archive. This makes it easy to share and deploy your models.\nTo package a model, check out [model archiver documentation](model-archiver/README.md)\n\n## Recommended production deployments\n\n* MMS doesn't provide authentication. You have to have your own authentication proxy in front of MMS.\n* MMS doesn't provide throttling, it's vulnerable to DDoS attack. It's recommended to running MMS behind a firewall.\n* MMS only allows localhost access by default, see [Network configuration](docs/configuration.md#configure-mms-listening-port) for detail.\n* SSL is not enabled by default, see [Enable SSL](docs/configuration.md#enable-ssl) for detail.\n* MMS use a config.properties file to configure MMS's behavior, see [Manage MMS](docs/configuration.md) page for detail of how to configure MMS.\n* For better security, we recommend running MMS inside docker container. This project includes Dockerfiles to build containers recommended for production deployments. These containers demonstrate how to customize your own production MMS deployment. The basic usage can be found on the [Docker readme](docker/README.md).\n\n## Other Features\n\nBrowse over to the [Docs readme](docs/README.md) for the full index of documentation. This includes more examples, how to customize the API service, API endpoint details, and more.\n\n## External demos powered by MMS\n\nHere are some example demos of deep learning applications, powered by MMS:\n\n | | |\n|:------:|:-----------:|\n| [Product Review Classification](https://thomasdelteil.github.io/TextClassificationCNNs_MXNet/) \"demo4\" |[Visual Search](https://thomasdelteil.github.io/VisualSearch_MXNet/) \"demo1\"|\n| [Facial Emotion Recognition](https://thomasdelteil.github.io/FacialEmotionRecognition_MXNet/) \"demo2\" |[Neural Style Transfer](https://thomasdelteil.github.io/NeuralStyleTransfer_MXNet/) \"demo3\" |\n\n## Contributing\n\nWe welcome all contributions!\n\nTo file a bug or request a feature, please file a GitHub issue. Pull requests are welcome.\n" }, { "path": "_config.yml", "content": "theme: jekyll-theme-cayman" }, { "path": "benchmarks/README.md", "content": "# Multi Model Server Benchmarking\n\nThe benchmarks measure the performance of MMS on various models and benchmarks. It supports either a number of built-in models or a custom model passed in as a path or URL to the .model file. It also runs various benchmarks using these models (see benchmarks section below). The benchmarks are run through a python3 script on the user machine through jmeter. MMS is run on the same machine in a docker instance to avoid network latencies. The benchmark must be run from within the context of the full MMS repo because it executes the local code as the version of MMS (and it is recompiled between runs) for ease of development.\n\n## Installation\n\n### Ubuntu\n\nThe script is mainly intended to run on a Ubuntu EC2 instance. For this reason, we have provided an `install_dependencies.sh` script to install everything needed to execute the benchmark on this environment. All you need to do is run this file and clone the MMS repo.\n\n### MacOS\n\nFor mac, you should have python3 and java installed. If you wish to run the default benchmarks featuring a docker-based instance of MMS, you will need to install docker as well. Finally, you will need to install jmeter with plugins which can be accomplished by running `mac_install_dependencies.sh`.\n\n### Other\n\nFor other environments, manual installation is necessary. The list of dependencies to be installed can be found below or by reading the ubuntu installation script.\n\nThe benchmarking script requires the following to run:\n- python3\n- A JDK and JRE\n- jmeter installed through homebrew or linuxbrew with the plugin manager and the following plugins: jpgc-synthesis=2.1,jpgc-filterresults=2.1,jpgc-mergeresults=2.1,jpgc-cmd=2.1,jpgc-perfmon=2.1\n- Docker-ce with the current user added to the docker group\n- Nvidia-docker (for GPU)\n\n\n## Models\n\nThe pre-loaded models for the benchmark can be mostly found in the [MMS model zoo](https://github.com/awslabs/multi-model-server/blob/master/docs/model_zoo.md). We currently support the following:\n- [resnet: ResNet-18 (Default)](https://github.com/awslabs/multi-model-server/blob/master/docs/model_zoo.md#resnet-18)\n- [squeezenet: SqueezeNet V1.1](https://github.com/awslabs/multi-model-server/blob/master/docs/model_zoo.md#squeezenet_v1.1)\n- [lstm: lstm-ptb](https://github.com/awslabs/multi-model-server/blob/master/docs/model_zoo.md#lstm-ptb)\n- [noop: noop-v1.0](https://s3.amazonaws.com/model-server/models/noop/noop-v1.0.model) Simple Noop model which returns \"Hello world\" to any input specified.\n- [noop_echo: noop_echo-v1.0](https://s3.amazonaws.com/model-server/models/noop/noop_echo-v1.0.model) Simple Noop model which returns whatever input is given to it.\n\n## Benchmarks\n\nWe support several basic benchmarks:\n- throughput: Run inference with enough threads to occupy all workers and ensure full saturation of resources to find the throughput. The number of threads defaults to 100.\n- latency: Run inference with a single thread to determine the latency\n- ping: Test the throughput of pinging against the frontend\n- load: Loads the same model many times in parallel. The number of loads is given by the \"count\" option and defaults to 16.\n- repeated_scale_calls: Will scale the model up to \"scale_up_workers\"=16 then down to \"scale_down_workers\"=1 then up and down repeatedly.\n- multiple_models: Loads and scales up three models (1. noop, 2. lstm, and 3. resnet), at the same time, runs inferences on them, and then scales them down. Use the options \"urlN\", \"modelN_name\", \"dataN\" to specify the model url, model name, and the data to pass to the model respectively. data1 and data2 are of the format \"'Some garbage data being passed here'\" and data3 is the filesystem path to a file to upload.\n\nWe also support compound benchmarks:\n- concurrent_inference: Runs the basic benchmark with different numbers of threads\n\n\n## Examples\n\nRun basic latency test on default resnet-18 model\\\n```./benchmark.py latency```\n\n\nRun basic throughput test on default resnet-18 model.\\\n```./benchmark.py throughput```\n\n\nRun all benchmarks\\\n```./benchmark.py --all```\n\n\nRun using the noop-v1.0 model\\\n```./benchmark.py latency -m noop_v1.0```\n\n\nRun on GPU (4 gpus)\\\n```./benchmark.py latency -g 4```\n\n\nRun with a custom image\\\n```./benchmark.py latency -i {imageFilePath}```\n\n\nRun with a custom model (works only for CNN based models, which accept image as an input for now. We will add support for more input types in future to this command. )\\\n```./benchmark.py latency -c {modelUrl} -i {imageFilePath}```\n\n\nRun with custom options\\\n```./benchmark.py repeated_scale_calls --options scale_up_workers 100 scale_down_workers 10```\n\n\nRun against an already running instance of MMS\\\n```./benchmark.py latency --mms 127.0.0.1``` (defaults to http, port 80, management port = port + 1)\\\n```./benchmark.py latency --mms 127.0.0.1:8080 --management-port 8081```\\\n```./benchmark.py latency --mms https://127.0.0.1:8443```\n\n\nRun verbose with only a single loop\\\n```./benchmark.py latency -v -l 1```\n\n\n## Benchmark options\n\nThe full list of options can be found by running with the -h or --help flags.\n\n\n## Profiling\n\n### Frontend\n\nThe benchmarks can be used in conjunction with standard profiling tools such as JProfiler to analyze the system performance. JProfiler can be downloaded from their [website](https://www.ej-technologies.com/products/jprofiler/overview.html). Once downloaded, open up JProfiler and follow these steps:\n\n1. Run MMS directly through gradle (do not use docker). This can be done either on your machine or on a remote machine accessible through SSH.\n2. In JProfiler, select \"Attach\" from the ribbon and attach to the ModelServer. The process name in the attach window should be \"com.amazonaws.ml.mms.ModelServer\". If it is on a remote machine, select \"On another computer\" in the attach window and enter the SSH details. For the session startup settings, you can leave it with the defaults. At this point, you should see live CPU and Memory Usage data on JProfiler's Telemetries section.\n3. Select Start Recordings in JProfiler's ribbon\n4. Run the Benchmark script targeting your running MMS instance. It might run something like `./benchmark.py throughput --mms https://127.0.0.1:8443`. It can be run on either your local machine or a remote machine (if you are running remote), but we recommend running the benchmark on the same machine as the model server to avoid confounding network latencies.\n5. Once the benchmark script has finished running, select Stop Recordings in JProfiler's ribbon\n\nOnce you have stopped recording, you should be able to analyze the data. One useful section to examine is CPU views > Call Tree and CPU views > Hot Spots to see where the processor time is going.\n\n### Backend\n\nThe benchmarks can also be used to analyze the backend performance using cProfile. It does not require any additional packages to run the benchmark, but viewing the logs does require an additional package. Run `pip install snakeviz` to install this. To run the python profiling, follow these steps:\n\n1. In the file `mms/model_service_worker.py`, set the constant BENCHMARK to true at the top to enable benchmarking.\n2. Run the benchmark and MMS. They can either be done automatically inside the docker container or separately with the \"--mms\" flag.\n3. Run MMS directly through gradle (do not use docker). This can be done either on your machine or on a remote machine accessible through SSH.\n4. Run the Benchmark script targeting your running MMS instance. It might run something like `./benchmark.py throughput --mms https://127.0.0.1:8443`. It can be run on either your local machine or a remote machine (if you are running remote), but we recommend running the benchmark on the same machine as the model server to avoid confounding network latencies.\n5. Run `snakeviz /tmp/mmsPythonProfile.prof` to view the profiling data. It should start up a web server on your machine and automatically open the page.\n6. Don't forget to set BENCHMARK = False in the model_service_worker.py file after you are finished.\n" }, { "path": "benchmarks/benchmark.py", "content": "#!/usr/bin/env python3\n\n# Copyright 2018 Amazon.com, Inc. or its affiliates. All Rights Reserved.\n# Licensed under the Apache License, Version 2.0 (the \"License\").\n# You may not use this file except in compliance with the License.\n# A copy of the License is located at\n# http://www.apache.org/licenses/LICENSE-2.0\n# or in the \"license\" file accompanying this file. This file is distributed\n# on an \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either\n# express or implied. See the License for the specific language governing\n# permissions and limitations under the License.\n\n\"\"\"\nExecute the MMS Benchmark. For instructions, run with the --help flag\n\"\"\"\n\n# pylint: disable=redefined-builtin\n\nimport argparse\nimport itertools\nimport multiprocessing\nimport os\nimport pprint\nimport shutil\nimport subprocess\nimport sys\nimport time\nimport traceback\nfrom functools import reduce\nfrom urllib.request import urlretrieve\n\nimport pandas as pd\n\nBENCHMARK_DIR = \"/tmp/MMSBenchmark/\"\n\nOUT_DIR = os.path.join(BENCHMARK_DIR, 'out/')\nRESOURCE_DIR = os.path.join(BENCHMARK_DIR, 'resource/')\n\nRESOURCE_MAP = {\n 'kitten.jpg': 'https://s3.amazonaws.com/model-server/inputs/kitten.jpg'\n}\n\n# Listing out all the JMX files\nJMX_IMAGE_INPUT_MODEL_PLAN = 'imageInputModelPlan.jmx'\nJMX_TEXT_INPUT_MODEL_PLAN = 'textInputModelPlan.jmx'\nJMX_PING_PLAN = 'pingPlan.jmx'\nJMX_CONCURRENT_LOAD_PLAN = 'concurrentLoadPlan.jmx'\nJMX_CONCURRENT_SCALE_CALLS = 'concurrentScaleCalls.jmx'\nJMX_MULTIPLE_MODELS_LOAD_PLAN = 'multipleModelsLoadPlan.jmx'\nJMX_GRAPHS_GENERATOR_PLAN = 'graphsGenerator.jmx'\n\n# Listing out the models tested\nMODEL_RESNET_18 = 'resnet-18'\nMODEL_SQUEEZE_NET = 'squeezenet'\nMODEL_LSTM_PTB = 'lstm_ptb'\nMODEL_NOOP = 'noop-v1.0'\n\n\nMODEL_MAP = {\n MODEL_SQUEEZE_NET: (JMX_IMAGE_INPUT_MODEL_PLAN, {'url': 'https://s3.amazonaws.com/model-server/models/squeezenet_v1.1/squeezenet_v1.1.model', 'model_name': MODEL_SQUEEZE_NET, 'input_filepath': 'kitten.jpg'}),\n MODEL_RESNET_18: (JMX_IMAGE_INPUT_MODEL_PLAN, {'url': 'https://s3.amazonaws.com/model-server/models/resnet-18/resnet-18.model', 'model_name': MODEL_RESNET_18, 'input_filepath': 'kitten.jpg'}),\n MODEL_LSTM_PTB: (JMX_TEXT_INPUT_MODEL_PLAN, {'url': 'https://s3.amazonaws.com/model-server/models/lstm_ptb/lstm_ptb.model', 'model_name': MODEL_LSTM_PTB, 'data': 'lstm_ip.json'}),\n MODEL_NOOP: (JMX_TEXT_INPUT_MODEL_PLAN, {'url': 'https://s3.amazonaws.com/model-server/models/noop/noop-v1.0.mar', 'model_name': MODEL_NOOP, 'data': 'noop_ip.txt'})\n}\n\n\n# Mapping of which row is relevant for a given JMX Test Plan\nEXPERIMENT_RESULTS_MAP = {\n JMX_IMAGE_INPUT_MODEL_PLAN: ['Inference Request'],\n JMX_TEXT_INPUT_MODEL_PLAN: ['Inference Request'],\n JMX_PING_PLAN: ['Ping Request'],\n JMX_CONCURRENT_LOAD_PLAN: ['Load Model Request'],\n JMX_CONCURRENT_SCALE_CALLS: ['Scale Up Model', 'Scale Down Model'],\n JMX_MULTIPLE_MODELS_LOAD_PLAN: ['Inference Request']\n}\n\n\nJMETER_RESULT_SETTINGS = {\n 'jmeter.reportgenerator.overall_granularity': 1000,\n # 'jmeter.reportgenerator.report_title': '\"MMS Benchmark Report Dashboard\"',\n 'aggregate_rpt_pct1': 50,\n 'aggregate_rpt_pct2': 90,\n 'aggregate_rpt_pct3': 99,\n}\n\n# Dictionary of what's present in the output csv generated v/s what we want to change the column name to for readability\nAGGREGATE_REPORT_CSV_LABELS_MAP = {\n 'aggregate_report_rate': 'Throughput',\n 'average': 'Average',\n 'aggregate_report_median': 'Median',\n 'aggregate_report_90%_line': 'aggregate_report_90_line',\n 'aggregate_report_99%_line': 'aggregate_report_99_line',\n 'aggregate_report_error%': 'aggregate_report_error'\n}\n\n\nCELLAR = '/home/ubuntu/.linuxbrew/Cellar/jmeter' if 'linux' in sys.platform else '/usr/local/Cellar/jmeter'\nJMETER_VERSION = os.listdir(CELLAR)[0]\nCMDRUNNER = '{}/{}/libexec/lib/ext/CMDRunner.jar'.format(CELLAR, JMETER_VERSION)\nJMETER = '{}/{}/libexec/bin/jmeter'.format(CELLAR, JMETER_VERSION)\nMMS_BASE = reduce(lambda val,func: func(val), (os.path.abspath(__file__),) + (os.path.dirname,) * 2)\nJMX_BASE = os.path.join(os.path.dirname(os.path.abspath(__file__)), 'jmx')\nCONFIG_PROP = os.path.join(MMS_BASE, 'benchmarks', 'config.properties')\nCONFIG_PROP_TEMPLATE = os.path.join(MMS_BASE, 'benchmarks', 'config_template.properties')\n\nDOCKER_MMS_BASE = \"/multi-model-server\"\nDOCKER_CONFIG_PROP = os.path.join(DOCKER_MMS_BASE, 'benchmarks', 'config.properties')\n\n# Commenting our NOOPs for now since there's a bug on MMS model loading for .mar files\nALL_BENCHMARKS = list(itertools.product(('latency', 'throughput'), (MODEL_RESNET_18,MODEL_NOOP, MODEL_LSTM_PTB)))\n # + [('multiple_models', MODEL_NOOP)]\n # + list(itertools.product(('load', 'repeated_scale_calls'), (MODEL_RESNET_18,))) \\ To Add once\n # repeated_scale_calls is fixed\n\n\nBENCHMARK_NAMES = ['latency', 'throughput']\n\nclass ChDir:\n def __init__(self, path):\n self.curPath = os.getcwd()\n self.path = path\n\n def __enter__(self):\n os.chdir(self.path)\n\n def __exit__(self, *args):\n os.chdir(self.curPath)\n\n\ndef basename(path):\n return os.path.splitext(os.path.basename(path))[0]\n\n\ndef get_resource(name):\n url = RESOURCE_MAP[name]\n path = os.path.join(RESOURCE_DIR, name)\n if not os.path.exists(path):\n directory = os.path.dirname(path)\n if not os.path.exists(directory):\n os.makedirs(directory)\n urlretrieve(url, path)\n return path\n\n\ndef run_process(cmd, wait=True, **kwargs):\n output = None if pargs.verbose else subprocess.DEVNULL\n if pargs.verbose:\n print(' '.join(cmd) if isinstance(cmd, list) else cmd)\n if not kwargs.get('shell') and isinstance(cmd, str):\n cmd = cmd.split(' ')\n if 'stdout' not in kwargs:\n kwargs['stdout'] = output\n if 'stderr' not in kwargs:\n kwargs['stderr'] = output\n p = subprocess.Popen(cmd, **kwargs)\n if wait:\n p.wait()\n return p\n\n\ndef run_single_benchmark(jmx, jmeter_args=dict(), threads=100, out_dir=None):\n if out_dir is None:\n out_dir = os.path.join(OUT_DIR, benchmark_name, basename(benchmark_model))\n if os.path.exists(out_dir):\n shutil.rmtree(out_dir)\n os.makedirs(out_dir)\n\n protocol = 'http'\n hostname = '127.0.0.1'\n port = 8080\n threads = pargs.threads[0] if pargs.threads else threads\n workers = pargs.workers[0] if pargs.workers else (\n pargs.gpus[0] if pargs.gpus else multiprocessing.cpu_count()\n )\n\n if pargs.mms:\n url = pargs.mms[0]\n if '://' in url:\n protocol, url = url.split('://')\n if ':' in url:\n hostname, port = url.split(':')\n port = int(port)\n else:\n hostname = url\n port = 80\n else:\n # Start MMS\n docker = 'nvidia-docker' if pargs.gpus else 'docker'\n container = 'mms_benchmark_gpu' if pargs.gpus else 'mms_benchmark_cpu'\n docker_path = 'awsdeeplearningteam/multi-model-server:nightly-mxnet-gpu' \\\n if pargs.gpus else 'awsdeeplearningteam/multi-model-server:nightly-mxnet-cpu'\n if pargs.docker:\n container = 'mms_benchmark_{}'.format(pargs.docker[0].split('/')[1])\n docker_path = pargs.docker[0]\n run_process(\"{} rm -f {}\".format(docker, container))\n docker_run_call = \"{} run --name {} -p 8080:8080 -p 8081:8081 -itd {}\".format(docker, container, docker_path)\n run_process(docker_run_call)\n\n management_port = int(pargs.management[0]) if pargs.management else port + 1\n time.sleep(300)\n\n try:\n # temp files\n tmpfile = os.path.join(out_dir, 'output.jtl')\n logfile = os.path.join(out_dir, 'jmeter.log')\n outfile = os.path.join(out_dir, 'out.csv')\n perfmon_file = os.path.join(out_dir, 'perfmon.csv')\n graphsDir = os.path.join(out_dir, 'graphs')\n reportDir = os.path.join(out_dir, 'report')\n\n # run jmeter\n run_jmeter_args = {\n 'hostname': hostname,\n 'port': port,\n 'management_port': management_port,\n 'protocol': protocol,\n 'min_workers': workers,\n 'rampup': 5,\n 'threads': threads,\n 'loops': int(pargs.loops[0]),\n 'perfmon_file': perfmon_file\n }\n run_jmeter_args.update(JMETER_RESULT_SETTINGS)\n run_jmeter_args.update(jmeter_args)\n run_jmeter_args.update(dict(zip(pargs.options[::2], pargs.options[1::2])))\n abs_jmx = jmx if os.path.isabs(jmx) else os.path.join(JMX_BASE, jmx)\n jmeter_args_str = ' '.join(sorted(['-J{}={}'.format(key, val) for key, val in run_jmeter_args.items()]))\n jmeter_call = '{} -n -t {} {} -l {} -j {} -e -o {}'.format(JMETER, abs_jmx, jmeter_args_str, tmpfile, logfile, reportDir)\n run_process(jmeter_call)\n\n time.sleep(30)\n # run AggregateReport\n ag_call = 'java -jar {} --tool Reporter --generate-csv {} --input-jtl {} --plugin-type AggregateReport'.format(CMDRUNNER, outfile, tmpfile)\n run_process(ag_call)\n\n # Generate output graphs\n gLogfile = os.path.join(out_dir, 'graph_jmeter.log')\n graphing_args = {\n 'raw_output': graphsDir,\n 'jtl_input': tmpfile\n }\n graphing_args.update(JMETER_RESULT_SETTINGS)\n gjmx = os.path.join(JMX_BASE, JMX_GRAPHS_GENERATOR_PLAN)\n graphing_args_str = ' '.join(['-J{}={}'.format(key, val) for key, val in graphing_args.items()])\n graphing_call = '{} -n -t {} {} -j {}'.format(JMETER, gjmx, graphing_args_str, gLogfile)\n run_process(graphing_call)\n\n print(\"Output available at {}\".format(out_dir))\n print(\"Report generated at {}\".format(os.path.join(reportDir, 'index.html')))\n\n data_frame = pd.read_csv(outfile, index_col=0)\n report = list()\n for val in EXPERIMENT_RESULTS_MAP[jmx]:\n for full_val in [fv for fv in data_frame.index if val in fv]:\n report.append(decorate_metrics(data_frame, full_val))\n\n return report\n\n except Exception: # pylint: disable=broad-except\n traceback.print_exc()\n\n\ndef run_multi_benchmark(key, xs, *args, **kwargs):\n out_dir = os.path.join(OUT_DIR, benchmark_name, basename(benchmark_model))\n if os.path.exists(out_dir):\n shutil.rmtree(out_dir)\n os.makedirs(out_dir)\n\n reports = dict()\n out_dirs = []\n for i, x in enumerate(xs):\n print(\"Running value {}={} (value {}/{})\".format(key, x, i+1, len(xs)))\n kwargs[key] = x\n sub_out_dir = os.path.join(out_dir, str(i+1))\n out_dirs.append(sub_out_dir)\n report = run_single_benchmark(*args, out_dir=sub_out_dir, **kwargs)\n reports[x] = report\n\n # files\n merge_results = os.path.join(out_dir, 'merge-results.properties')\n joined = os.path.join(out_dir, 'joined.csv')\n reportDir = os.path.join(out_dir, 'report')\n\n # merge runs together\n inputJtls = [os.path.join(out_dirs[i], 'output.jtl') for i in range(len(xs))]\n prefixes = [\"{} {}: \".format(key, x) for x in xs]\n baseJtl = inputJtls[0]\n basePrefix = prefixes[0]\n for i in range(1, len(xs), 3): # MergeResults only joins up to 4 at a time\n with open(merge_results, 'w') as f:\n curInputJtls = [baseJtl] + inputJtls[i:i+3]\n curPrefixes = [basePrefix] + prefixes[i:i+3]\n for j, (jtl, p) in enumerate(zip(curInputJtls, curPrefixes)):\n f.write(\"inputJtl{}={}\\n\".format(j+1, jtl))\n f.write(\"prefixLabel{}={}\\n\".format(j+1, p))\n f.write(\"\\n\")\n merge_call = 'java -jar {} --tool Reporter --generate-csv joined.csv --input-jtl {} --plugin-type MergeResults'.format(CMDRUNNER, merge_results)\n time.sleep(30)\n run_process(merge_call)\n shutil.move('joined.csv', joined) # MergeResults ignores path given and puts result into cwd\n baseJtl = joined\n basePrefix = \"\"\n\n # build report\n time.sleep(30)\n run_process('{} -g {} -o {}'.format(JMETER, joined, reportDir))\n\n print(\"Merged output available at {}\".format(out_dir))\n print(\"Merged report generated at {}\".format(os.path.join(reportDir, 'index.html')))\n\n return reports\n\n\ndef parseModel():\n if benchmark_model in MODEL_MAP:\n plan, jmeter_args = MODEL_MAP[benchmark_model]\n for k, v in jmeter_args.items():\n if v in RESOURCE_MAP:\n jmeter_args[k] = get_resource(v)\n if k == 'data':\n jmeter_args[k] = os.path.join(MMS_BASE, 'benchmarks', v)\n if pargs.input:\n jmeter_args['input_filepath'] = pargs.input[0]\n else:\n plan = JMX_IMAGE_INPUT_MODEL_PLAN\n jmeter_args = {\n 'url': benchmark_model,\n 'model_name': basename(benchmark_model),\n 'input_filepath': pargs.input[0]\n }\n return plan, jmeter_args\n\n\ndef decorate_metrics(data_frame, row_to_read):\n temp_dict = data_frame.loc[row_to_read].to_dict()\n result = dict()\n row_name = row_to_read.replace(' ', '_')\n for key, value in temp_dict.items():\n if key in AGGREGATE_REPORT_CSV_LABELS_MAP:\n new_key = '{}_{}_{}_{}'.format(benchmark_name, benchmark_model, row_name, AGGREGATE_REPORT_CSV_LABELS_MAP[key])\n result[new_key] = value\n return result\n\n\nclass Benchmarks:\n \"\"\"\n Contains benchmarks to run\n \"\"\"\n\n @staticmethod\n def throughput():\n \"\"\"\n Performs a simple single benchmark that measures the model throughput on inference tasks\n \"\"\"\n plan, jmeter_args = parseModel()\n return run_single_benchmark(plan, jmeter_args)\n\n @staticmethod\n def latency():\n \"\"\"\n Performs a simple single benchmark that measures the model latency on inference tasks\n \"\"\"\n plan, jmeter_args = parseModel()\n return run_single_benchmark(plan, jmeter_args, threads=1)\n\n @staticmethod\n def ping():\n \"\"\"\n Performs a simple ping benchmark that measures the throughput for a ping request to the frontend\n \"\"\"\n return run_single_benchmark(JMX_PING_PLAN, dict(), threads=5000)\n\n @staticmethod\n def load():\n \"\"\"\n Benchmarks number of concurrent inference requests\n \"\"\"\n plan, jmeter_args = parseModel()\n plan = JMX_CONCURRENT_LOAD_PLAN\n jmeter_args['count'] = 8\n return run_single_benchmark(plan, jmeter_args)\n\n @staticmethod\n def repeated_scale_calls():\n \"\"\"\n Benchmarks number of concurrent inference requests\n \"\"\"\n plan, jmeter_args = parseModel()\n plan = JMX_CONCURRENT_SCALE_CALLS\n jmeter_args['scale_up_workers'] = 16\n jmeter_args['scale_down_workers'] = 2\n return run_single_benchmark(plan, jmeter_args)\n\n @staticmethod\n def multiple_models():\n \"\"\"\n Tests with 3 models\n \"\"\"\n plan = JMX_MULTIPLE_MODELS_LOAD_PLAN\n jmeter_args = {\n 'url1': MODEL_MAP[MODEL_NOOP][1]['url'],\n 'url2': MODEL_MAP[MODEL_LSTM_PTB][1]['url'],\n 'url3': MODEL_MAP[MODEL_RESNET_18][1]['url'],\n 'model1_name': MODEL_MAP[MODEL_NOOP][1]['model_name'],\n 'model2_name': MODEL_MAP[MODEL_LSTM_PTB][1]['model_name'],\n 'model3_name': MODEL_MAP[MODEL_RESNET_18][1]['model_name'],\n 'data3': get_resource('kitten.jpg')\n }\n return run_single_benchmark(plan, jmeter_args)\n\n @staticmethod\n def concurrent_inference():\n \"\"\"\n Benchmarks number of concurrent inference requests\n \"\"\"\n plan, jmeter_args = parseModel()\n return run_multi_benchmark('threads', range(1, 3*5+1, 3), plan, jmeter_args)\n\n\ndef run_benchmark():\n if hasattr(Benchmarks, benchmark_name):\n print(\"Running benchmark {} with model {}\".format(benchmark_name, benchmark_model))\n res = getattr(Benchmarks, benchmark_name)()\n pprint.pprint(res)\n print('\\n')\n else:\n raise Exception(\"No benchmark benchmark_named {}\".format(benchmark_name))\n\n\ndef modify_config_props_for_mms(pargs):\n shutil.copyfile(CONFIG_PROP_TEMPLATE, CONFIG_PROP)\n with open(CONFIG_PROP, 'a') as f:\n f.write('\\nnumber_of_netty_threads=32')\n f.write('\\njob_queue_size=1000')\n if pargs.gpus:\n f.write('\\nnumber_of_gpu={}'.format(pargs.gpus[0]))\n\n\nif __name__ == '__main__':\n benchmark_name_options = [f for f in dir(Benchmarks) if callable(getattr(Benchmarks, f)) and f[0] != '_']\n parser = argparse.ArgumentParser(prog='multi-model-server-benchmarks', description='Benchmark Multi Model Server')\n\n target = parser.add_mutually_exclusive_group(required=True)\n target.add_argument('name', nargs='?', type=str, choices=benchmark_name_options, help='The name of the benchmark to run')\n target.add_argument('-a', '--all', action='store_true', help='Run all benchmarks')\n target.add_argument('-s', '--suite', action='store_true', help='Run throughput and latency on a supplied model')\n\n model = parser.add_mutually_exclusive_group()\n model.add_argument('-m', '--model', nargs=1, type=str, dest='model', default=[MODEL_RESNET_18], choices=MODEL_MAP.keys(), help='A preloaded model to run. It defaults to {}'.format(MODEL_RESNET_18))\n model.add_argument('-c', '--custom-model', nargs=1, type=str, dest='model', help='The path to a custom model to run. The input argument must also be passed. Currently broken')\n\n parser.add_argument('-d', '--docker', nargs=1, type=str, default=None, help='Docker hub path to use')\n parser.add_argument('-i', '--input', nargs=1, type=str, default=None, help='The input to feed to the test')\n parser.add_argument('-g', '--gpus', nargs=1, type=int, default=None, help='Number of gpus. Leave empty to run CPU only')\n\n parser.add_argument('-l', '--loops', nargs=1, type=int, default=[10], help='Number of loops to run')\n parser.add_argument('-t', '--threads', nargs=1, type=int, default=None, help='Number of jmeter threads to run')\n parser.add_argument('-w', '--workers', nargs=1, type=int, default=None, help='Number of MMS backend workers to use')\n\n parser.add_argument('--mms', nargs=1, type=str, help='Target an already running instance of MMS instead of spinning up a docker container of MMS. Specify the target with the format address:port (for http) or protocol://address:port')\n parser.add_argument('--management-port', dest='management', nargs=1, type=str, help='When targeting a running MMS instance, specify the management port')\n parser.add_argument('-v', '--verbose', action='store_true', help='Display all output')\n parser.add_argument('--options', nargs='*', default=[], help='Additional jmeter arguments. It should follow the format of --options argname1 argval1 argname2 argval2 ...')\n pargs = parser.parse_args()\n\n if os.path.exists(OUT_DIR):\n if pargs.all:\n shutil.rmtree(OUT_DIR)\n os.makedirs(OUT_DIR)\n else:\n os.makedirs(OUT_DIR)\n\n modify_config_props_for_mms(pargs)\n\n if pargs.suite:\n benchmark_model = pargs.model[0].lower()\n for benchmark_name in BENCHMARK_NAMES:\n run_benchmark()\n if not os.path.isdir(os.path.join(OUT_DIR, benchmark_name, basename(benchmark_model), 'report')):\n run_benchmark()\n\n elif pargs.all:\n for benchmark_name, benchmark_model in ALL_BENCHMARKS:\n run_benchmark()\n else:\n benchmark_name = pargs.name.lower()\n benchmark_model = pargs.model[0].lower()\n run_benchmark()\n" }, { "path": "benchmarks/install_dependencies.sh", "content": "#!/bin/bash\n\n# Copyright 2018 Amazon.com, Inc. or its affiliates. All Rights Reserved.\n# Licensed under the Apache License, Version 2.0 (the \"License\").\n# You may not use this file except in compliance with the License.\n# A copy of the License is located at\n# http://www.apache.org/licenses/LICENSE-2.0\n# or in the \"license\" file accompanying this file. This file is distributed\n# on an \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either\n# express or implied. See the License for the specific language governing\n# permissions and limitations under the License.\n\n# This file contains the installation setup for running benchmarks on EC2 isntance.\n# To run on a machine with GPU : ./install_dependencies True\n# To run on a machine with CPU : ./install_dependencies False\nset -ex\n\nsudo apt-get update\nsudo apt-get -y upgrade\necho \"Setting up your Ubuntu machine to load test MMS\"\nsudo apt-get install -y \\\n python \\\n python-pip \\\n python3-pip \\\n python3-tk \\\n python-psutil \\\n default-jre \\\n default-jdk \\\n linuxbrew-wrapper \\\n build-essential\n\nif [[ $1 = True ]]\nthen\n echo \"Installing pip packages for GPU\"\n sudo apt install -y nvidia-cuda-toolkit\n pip install future psutil mxnet-cu92 pillow --user\nelse\n echo \"Installing pip packages for CPU\"\n pip install future psutil mxnet pillow --user\n\nfi\n\npip3 install pandas\n\necho \"Installing JMeter through Brew\"\n# Script would end on errors, but everything works fine\n{\n yes '' | brew update\n} || {\n true\n}\n{\n brew install jmeter --with-plugins\n} || {\n true\n}\n\nwget https://jmeter-plugins.org/get/ -O /home/ubuntu/.linuxbrew/Cellar/jmeter/5.0/libexec/lib/ext/jmeter-plugins-manager-1.3.jar\nwget http://search.maven.org/remotecontent?filepath=kg/apc/cmdrunner/2.2/cmdrunner-2.2.jar -O /home/ubuntu/.linuxbrew/Cellar/jmeter/5.0/libexec/lib/cmdrunner-2.2.jar\njava -cp /home/ubuntu/.linuxbrew/Cellar/jmeter/5.0/libexec/lib/ext/jmeter-plugins-manager-1.3.jar org.jmeterplugins.repository.PluginManagerCMDInstaller\n/home/ubuntu/.linuxbrew/Cellar/jmeter/5.0/libexec/bin/PluginsManagerCMD.sh install jpgc-synthesis=2.1,jpgc-filterresults=2.1,jpgc-mergeresults=2.1,jpgc-cmd=2.1,jpgc-perfmon=2.1\n\necho \"Install docker\"\nsudo apt-get remove docker docker-engine docker.io\nsudo apt-get install -y \\\n apt-transport-https \\\n ca-certificates \\\n curl \\\n software-properties-common\ncurl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo apt-key add -\nsudo add-apt-repository \\\n \"deb [arch=amd64] https://download.docker.com/linux/ubuntu \\\n $(lsb_release -cs) \\\n stable\"\nsudo apt-get update\nsudo apt-get install -y docker-ce\n{\n sudo groupadd docker || {true}\n} || {\n true\n}\n{\n gpasswd -a $USER docker\n} || {\n true\n}\n\n\nif [[ $1 = True ]]\nthen\n echo \"Installing nvidia-docker\"\n # If you have nvidia-docker 1.0 installed: we need to remove it and all existing GPU containers\n {\n docker volume ls -q -f driver=nvidia-docker | xargs -r -I{} -n1 docker ps -q -a -f volume={} | xargs -r docker rm -f\n } || {\n true\n }\n {\n sudo apt-get purge -y nvidia-docker\n } || {\n true\n }\n\n # Add the package repositories\n curl -s -L https://nvidia.github.io/nvidia-docker/gpgkey | \\\n sudo apt-key add -\n distribution=$(. /etc/os-release;echo $ID$VERSION_ID)\n curl -s -L https://nvidia.github.io/nvidia-docker/$distribution/nvidia-docker.list | \\\n sudo tee /etc/apt/sources.list.d/nvidia-docker.list\n sudo apt-get update\n\n # Install nvidia-docker2 and reload the Docker daemon configuration\n sudo apt-get install -y nvidia-docker2\n sudo pkill -SIGHUP dockerd\n\n # Test nvidia-smi with the latest official CUDA image\n docker run --runtime=nvidia --rm nvidia/cuda nvidia-smi\nfi\n" }, { "path": "benchmarks/jmx/concurrentLoadPlan.jmx", "content": "\n\n \n \n \n false\n true\n false\n \n \n \n \n \n \n \n \n \n model_url\n ${__P(url, https://s3.amazonaws.com/model-server/models/resnet-18/resnet-18.model)}\n =\n The url from where to fetch the models from\n \n \n model\n ${__P(model_name,resnet-18)}\n =\n \n \n count\n ${__P(count,10)}\n =\n \n \n \n \n \n \n \n \n ${__P(hostname,127.0.0.1)}\n ${__P(management_port,8444)}\n ${__P(protocol,https)}\n \n \n 6\n \n \n \n \n \n continue\n \n false\n ${__P(loops,1)}\n \n ${count}\n 1\n false\n \n \n \n \n \n \n \n \n false\n ${ctr}\n =\n true\n ctr\n \n \n \n \n \n \n \n /models?url=${model}-${__counter(FALSE, )}&model_name=${model}-${__counter(FALSE,)}\n POST\n true\n false\n true\n false\n \n \n \n \n \n \n ${count}\n 0\n \n \n \n \n \n\n" }, { "path": "benchmarks/jmx/concurrentScaleCalls.jmx", "content": "\n\n \n \n \n false\n true\n false\n \n \n \n \n \n \n \n \n \n model_url\n ${__P(url, https://s3.amazonaws.com/model-server/models/resnet-18/resnet-18.model)}\n =\n The url from where to fetch the models from\n \n \n model\n ${__P(model_name,resnet-18)}\n =\n Name of the model to run the tests on\n \n \n count\n ${__P(count,1)}\n =\n \n \n scale_up_workers\n ${__P(scale_up_workers,1)}\n =\n The number of workers to scale model to\n \n \n scale_down_workers\n ${__P(scale_down_workers,1)}\n Scale down the workers\n =\n \n \n \n \n \n \n \n \n ${__P(hostname,127.0.0.1)}\n ${__P(management_port,8444)}\n ${__P(protocol,https)}\n \n \n 6\n \n \n \n \n \n continue\n \n false\n 1\n \n 1\n 1\n false\n \n \n \n \n \n \n \n \n \n \n \n \n /models?url=${model_url}\n POST\n true\n false\n true\n false\n \n \n \n \n \n \n \n continue\n \n false\n ${__P(loops,10)}\n \n ${__P(threads,2)}\n ${__P(rampup,5)}\n false\n \n \n \n \n \n \n \n \n \n \n \n \n /models/${model}?min_worker=${scale_up_workers}\n PUT\n true\n false\n true\n false\n \n \n \n \n \n \n \n \n \n \n \n \n \n /models/${model}?min_worker=${scale_down_workers}\n PUT\n true\n false\n true\n false\n \n \n \n \n \n \n ${count}\n 0\n \n \n \n \n continue\n \n false\n 1\n \n 1\n 1\n false\n \n \n \n \n \n \n \n \n \n \n \n \n /models/${model}?min_worker=0\n DELETE\n true\n false\n true\n false\n \n \n \n \n \n \n \n \n\n" }, { "path": "benchmarks/jmx/graphsGenerator.jmx", "content": "\n\n \n \n \n false\n true\n false\n \n \n \n \n \n \n \n continue\n \n false\n 1\n \n 1\n 1\n false\n \n \n \n \n \n false\n false\n \n \n false\n 0\n ${__P(graph_prefix,g_)}\n \n 1000\n 600\n 800\n Inference Request\n false\n 150\n \n \n ${__P(raw_output)}\n true\n true\n false\n true\n ${__P(jtl_input)}\n \n \n \n \n \n \n \n\n" }, { "path": "benchmarks/jmx/imageInputModelPlan.jmx", "content": "\n\n \n \n \n false\n true\n false\n \n \n \n \n \n \n \n \n \n cnn_url\n ${__P(url, https://s3.amazonaws.com/model-server/models/squeezenet_v1.1/squeezenet_v1.1.model)}\n =\n The url from where to fetch noop model from\n \n \n scale_up_workers\n ${__P(min_workers,1)}\n =\n The workers to scale No op model to\n \n \n scale_down_workers\n 0\n Offload the No-Op Model\n =\n \n \n model\n ${__P(model_name,squeezenet_v1.1)}\n =\n \n \n \n \n \n \n \n \n ${__P(hostname,127.0.0.1)}\n ${__P(port,8443)}\n ${__P(protocol,https)}\n \n \n 6\n \n \n \n \n \n continue\n \n false\n 1\n \n 1\n 1\n false\n \n \n \n \n \n \n \n \n \n ${__P(management_port,8444)}\n \n \n /models?url=${cnn_url}\n POST\n true\n false\n true\n false\n \n \n \n \n \n \n \n \n \n \n ${__P(management_port,8444)}\n \n \n /models/${model}?min_worker=${scale_up_workers}\n PUT\n true\n false\n true\n false\n \n \n \n \n \n \n \n continue\n \n false\n ${__P(loops,200)}\n \n ${__P(threads,20)}\n ${__P(rampup,5)}\n false\n \n \n \n \n \n \n \n \n ${__P(input_filepath)}\n data\n image/jpeg\n \n \n \n \n \n \n \n \n \n \n /predictions/${model}\n POST\n true\n false\n true\n true\n \n \n \n \n \n \n \n continue\n \n false\n 1\n \n 1\n 1\n false\n \n \n \n \n \n \n \n \n \n ${__P(management_port,8444)}\n \n \n /models/${model}?min_worker=${scale_down_workers}\n DELETE\n true\n false\n true\n false\n \n \n \n \n \n \n \n \n\n" }, { "path": "benchmarks/jmx/multipleModelsLoadPlan.jmx", "content": "\n\n \n \n \n false\n true\n false\n \n \n \n \n \n \n \n \n \n url1\n ${__P(url1,noop.model)}\n =\n The url from where to fetch noop model from\n \n \n url2\n ${__P(url2,lstm_ptb)}\n =\n \n \n url3\n ${__P(url3,https://s3.amazonaws.com/model-server/models/resnet-18/resnet-18.model)}\n =\n \n \n model1\n ${__P(model1_name,noop)}\n =\n \n \n model2\n ${__P(model2_name,lstm_ptb)}\n =\n \n \n model3\n ${__P(model3_name,resnet-18)}\n =\n \n \n scale_up_workers\n ${__P(min_workers,1)}\n =\n \n \n scale_down_workers\n ${__P(scale_down_workers, 0)}\n =\n \n \n \n \n \n \n \n \n ${__P(hostname,127.0.0.1)}\n ${__P(port,8443)}\n ${__P(protocol,https)}\n \n \n 6\n \n \n \n \n \n continue\n \n false\n 1\n \n 1\n 1\n false\n \n \n \n \n \n \n \n \n \n ${__P(management_port,8444)}\n \n \n /models?url=${url1}\n POST\n true\n false\n true\n false\n \n \n \n \n \n \n true\n \n \n \n false\n \n =\n \n \n \n \n ${__P(management_port,8444)}\n \n \n /models/${model1}?min_worker=${scale_up_workers}\n PUT\n true\n false\n true\n false\n \n \n \n \n \n \n \n \n \n \n ${__P(management_port,8444)}\n \n \n /models?url=${url2}\n POST\n true\n false\n true\n false\n \n \n \n \n \n \n true\n \n \n \n false\n \n =\n \n \n \n \n ${__P(management_port,8444)}\n \n \n /models/${model2}?min_worker=${scale_up_workers}\n PUT\n true\n false\n true\n false\n \n \n \n \n \n \n \n \n \n \n ${__P(management_port,8444)}\n \n \n /models?url=${url3}\n POST\n true\n false\n true\n false\n \n \n \n \n \n \n \n \n \n \n ${__P(management_port,8444)}\n \n \n /models/${model3}?min_worker=${scale_up_workers}\n PUT\n true\n false\n true\n false\n \n \n \n \n \n \n \n continue\n \n false\n ${__P(loops,2)}\n \n ${__P(threads,2)}\n ${__P(rampup,5)}\n false\n \n \n \n \n \n \n \n \n \n \n false\n ${__P(data1,'Some garbage data being passed here')}\n =\n true\n data\n \n \n \n \n \n \n \n /predictions/${model1}\n POST\n true\n false\n true\n false\n \n \n \n \n \n \n \n \n \n false\n ${__P(data2,'Some garbage data being passed here')}\n =\n true\n data\n \n \n \n \n \n \n \n /predictions/${model2}\n POST\n true\n false\n true\n false\n \n \n \n \n \n \n \n \n \n ${__P(data3)}\n data\n image/jpeg\n \n \n \n \n \n \n \n \n \n \n /predictions/${model3}\n POST\n true\n false\n true\n true\n \n \n \n \n \n \n \n \n continue\n \n false\n 1\n \n 1\n 1\n false\n \n \n \n \n \n \n \n \n \n ${__P(management_port,8444)}\n \n \n /models/${model1}?min_worker=${scale_down_workers}\n DELETE\n true\n false\n true\n false\n \n \n \n \n \n \n \n \n \n \n ${__P(management_port,8444)}\n \n \n /models/${model2}?min_worker=${scale_down_workers}\n DELETE\n true\n false\n true\n false\n \n \n \n \n \n \n \n \n \n \n ${__P(management_port,8444)}\n \n \n /models/${model3}?min_worker=${scale_down_workers}\n DELETE\n true\n false\n true\n false\n \n \n \n \n \n \n \n \n\n" }, { "path": "benchmarks/jmx/pingPlan.jmx", "content": "\n\n \n \n \n false\n true\n false\n \n \n \n \n \n \n \n \n \n \n ${__P(hostname,127.0.0.1)}\n ${__P(port,8443)}\n ${__P(protocol,https)}\n \n \n 6\n \n \n \n \n \n continue\n \n false\n ${__P(loops,200)}\n \n ${__P(threads,200)}\n ${__P(rampup,5)}\n false\n \n \n \n \n \n \n \n \n \n \n \n \n /ping\n GET\n true\n false\n true\n false\n \n \n \n \n \n \n \n \n\n" }, { "path": "benchmarks/jmx/textInputModelPlan.jmx", "content": "\n\n \n \n \n false\n true\n false\n \n \n \n \n \n \n \n \n \n noop_url\n ${__P(url,noop.model)}\n =\n The url from where to fetch noop model from\n \n \n scale_up_workers\n ${__P(min_workers,1)}\n =\n The workers to scale No op model to\n \n \n scale_down_workers\n 0\n Offload the No-Op Model\n =\n \n \n model\n ${__P(model_name,noop)}\n Model name\n =\n \n \n \n \n \n \n \n \n ${__P(hostname,127.0.0.1)}\n ${__P(port,8443)}\n ${__P(protocol,https)}\n \n \n 6\n \n \n \n \n \n continue\n \n false\n 1\n \n 1\n 1\n false\n \n \n \n \n \n \n \n \n \n ${__P(management_port,8444)}\n \n \n /models?url=${noop_url}\n POST\n true\n false\n true\n false\n \n \n \n \n \n \n \n \n \n \n ${__P(management_port,8444)}\n \n \n /models/${model}?min_worker=${scale_up_workers}\n PUT\n true\n false\n true\n false\n \n \n \n \n \n \n \n continue\n \n false\n ${__P(loops,200)}\n \n ${__P(threads,200)}\n ${__P(rampup,5)}\n false\n \n \n \n \n \n \n \n \n ${__P(data,)}\n data\n application/json\n \n \n \n \n \n \n \n \n \n \n /predictions/${model}\n POST\n true\n false\n true\n true\n \n \n \n \n \n \n \n continue\n \n false\n 1\n \n 1\n 1\n false\n \n \n \n \n \n \n \n \n \n ${__P(management_port,8444)}\n \n \n /models/${model}?min_worker=${scale_down_workers}\n DELETE\n true\n false\n true\n false\n \n \n \n \n \n \n \n \n\n" }, { "path": "benchmarks/lstm_ip.json", "content": "[{\"input_sentence\": \"on the exchange floor as soon as ual stopped trading we for a panic said one top floor trader\"}]" }, { "path": "benchmarks/mac_install_dependencies.sh", "content": "#!/bin/bash\n\n# Copyright 2018 Amazon.com, Inc. or its affiliates. All Rights Reserved.\n# Licensed under the Apache License, Version 2.0 (the \"License\").\n# You may not use this file except in compliance with the License.\n# A copy of the License is located at\n# http://www.apache.org/licenses/LICENSE-2.0\n# or in the \"license\" file accompanying this file. This file is distributed\n# on an \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either\n# express or implied. See the License for the specific language governing\n# permissions and limitations under the License.\n\n# This file contains the installation setup for running benchmarks on EC2 isntance.\n# To run on a machine with GPU : ./install_dependencies True\n# To run on a machine with CPU : ./install_dependencies False\nset -ex\n\necho \"Installing JMeter through Brew\"\n# Script would end on errors, but everything works fine\nbrew update || {\n}\nbrew install jmeter --with-plugins || {\n}\n\nwget https://jmeter-plugins.org/get/ -O /usr/local/Cellar/jmeter/4.0/libexec/lib/ext/jmeter-plugins-manager-1.3.jar\nwget http://search.maven.org/remotecontent?filepath=kg/apc/cmdrunner/2.2/cmdrunner-2.2.jar -O /usr/local/Cellar/jmeter/4.0/libexec/lib/cmdrunner-2.2.jar\njava -cp /usr/local/Cellar/jmeter/4.0/libexec/lib/ext/jmeter-plugins-manager-1.3.jar org.jmeterplugins.repository.PluginManagerCMDInstaller\n/usr/local/Cellar/jmeter/4.0/libexec/bin/PluginsManagerCMD.sh install jpgc-synthesis=2.1,jpgc-filterresults=2.1,jpgc-mergeresults=2.1,jpgc-cmd=2.1,jpgc-perfmon=2.1\n" }, { "path": "benchmarks/noop_ip.txt", "content": "\"[{\\\"input_sentence\\\": \\\"Hello World\\\"}]\"" }, { "path": "benchmarks/upload_results_to_s3.sh", "content": "#!/usr/bin/env bash\n\n# Copyright 2018 Amazon.com, Inc. or its affiliates. All Rights Reserved.\n# Licensed under the Apache License, Version 2.0 (the \"License\").\n# You may not use this file except in compliance with the License.\n# A copy of the License is located at\n# http://www.apache.org/licenses/LICENSE-2.0\n# or in the \"license\" file accompanying this file. This file is distributed\n# on an \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either\n# express or implied. See the License for the specific language governing\n# permissions and limitations under the License.\n\n#Author: Piyush Ghai\n\nset -ex\n\necho \"uploading result files to s3\"\n\nhw_type=cpu\n\nif [ \"$1\" = \"True\" ]\nthen\n hw_type=gpu\nfi\n\necho `pwd`\ncd /tmp/MMSBenchmark/out\necho `pwd`\n\ntoday=`date +\"%m-%d-%y\"`\necho \"Saving on S3 bucket on s3://benchmarkai-metrics-prod/daily/mms/$hw_type/$today\"\n\nfor dir in $(ls `pwd`/)\ndo\n echo $dir\n aws s3 cp $dir/ s3://benchmarkai-metrics-prod/daily/mms/$hw_type/$today/$dir/ --recursive\ndone\n\necho \"Files uploaded\"\n" }, { "path": "ci/Dockerfile.python2.7", "content": "# Copyright 2018 Amazon.com, Inc. or its affiliates. All Rights Reserved.\n# Licensed under the Apache License, Version 2.0 (the \"License\").\n# You may not use this file except in compliance with the License.\n# A copy of the License is located at\n# http://www.apache.org/licenses/LICENSE-2.0\n# or in the \"license\" file accompanying this file. This file is distributed\n# on an \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either\n# express or implied. See the License for the specific language governing\n# permissions and limitations under the License.\n#\n\nFROM ubuntu:14.04.5\n\nENV LANG=\"C.UTF-8\"\n\nENV DOCKER_BUCKET=\"download.docker.com\" \\\n DOCKER_VERSION=\"17.09.0-ce\" \\\n DOCKER_CHANNEL=\"stable\" \\\n DOCKER_SHA256=\"a9e90a73c3cdfbf238f148e1ec0eaff5eb181f92f35bdd938fd7dab18e1c4647\" \\\n DIND_COMMIT=\"3b5fac462d21ca164b3778647420016315289034\" \\\n DOCKER_COMPOSE_VERSION=\"1.16.1\" \\\n GITVERSION_VERSION=\"3.6.5\"\n\n# Install git\nRUN set -ex \\\n && apt-get update \\\n && apt-get install software-properties-common -y --no-install-recommends\\\n && apt-add-repository ppa:git-core/ppa \\\n && apt-get update \\\n && apt-get install git -y --no-install-recommends\\\n && git version\n\nRUN set -ex \\\n && echo 'Acquire::CompressionTypes::Order:: \"gz\";' > /etc/apt/apt.conf.d/99use-gzip-compression \\\n && apt-get update \\\n && apt-get install -y --no-install-recommends wget=1.15-* fakeroot=1.20-* ca-certificates \\\n autoconf=2.69-* automake=1:1.14.* less=458-* groff=1.22.* \\\n bzip2=1.0.* file=1:5.14-* g++=4:4.8.* gcc=4:4.8.* imagemagick=8:6.7.* \\\n libbz2-dev=1.0.* libc6-dev=2.19-* libcurl4-openssl-dev=7.35.* curl=7.35.* \\\n libdb-dev=1:5.3.* libevent-dev=2.0.* libffi-dev=3.1~* \\\n libgeoip-dev=1.6.* libglib2.0-dev=2.40.* libjpeg-dev=8c-* \\\n libkrb5-dev=1.12+* liblzma-dev=5.1.* libmagickcore-dev=8:6.7.* \\\n libmagickwand-dev=8:6.7.* libmysqlclient-dev=5.5.* libncurses5-dev=5.9+* \\\n libpng12-dev=1.2.* libpq-dev=9.3.* libreadline-dev=6.3-* libsqlite3-dev=3.8.* \\\n libssl-dev=1.0.* libtool=2.4.* libwebp-dev=0.4.* libxml2-dev=2.9.* \\\n libxslt1-dev=1.1.* libyaml-dev=0.1.* make=3.81-* patch=2.7.* xz-utils=5.1.* \\\n zlib1g-dev=1:1.2.* tcl=8.6.* tk=8.6.* \\\n e2fsprogs=1.42.* iptables=1.4.* xfsprogs=3.1.* xz-utils=5.1.* \\\n mono-mcs=3.2.* libcurl4-openssl-dev=7.35.* liberror-perl=0.17-* unzip=6.0-*\\\n && rm -rf /var/lib/apt/lists/* \\\n && apt-get clean\n\n# Download and set up GitVersion\nRUN set -ex \\\n && wget \"https://github.com/GitTools/GitVersion/releases/download/v${GITVERSION_VERSION}/GitVersion_${GITVERSION_VERSION}.zip\" -O /tmp/GitVersion_${GITVERSION_VERSION}.zip \\\n && mkdir -p /usr/local/GitVersion_${GITVERSION_VERSION} \\\n && unzip /tmp/GitVersion_${GITVERSION_VERSION}.zip -d /usr/local/GitVersion_${GITVERSION_VERSION} \\\n && rm /tmp/GitVersion_${GITVERSION_VERSION}.zip \\\n && echo \"mono /usr/local/GitVersion_${GITVERSION_VERSION}/GitVersion.exe /output json /showvariable \\$1\" >> /usr/local/bin/gitversion \\\n && chmod +x /usr/local/bin/gitversion\n# Install Docker\nRUN set -ex \\\n && curl -fSL \"https://${DOCKER_BUCKET}/linux/static/${DOCKER_CHANNEL}/x86_64/docker-${DOCKER_VERSION}.tgz\" -o docker.tgz \\\n && echo \"${DOCKER_SHA256} *docker.tgz\" | sha256sum -c - \\\n && tar --extract --file docker.tgz --strip-components 1 --directory /usr/local/bin/ \\\n && rm docker.tgz \\\n && docker -v \\\n# set up subuid/subgid so that \"--userns-remap=default\" works out-of-the-box\n && addgroup dockremap \\\n && useradd -g dockremap dockremap \\\n && echo 'dockremap:165536:65536' >> /etc/subuid \\\n && echo 'dockremap:165536:65536' >> /etc/subgid \\\n && wget \"https://raw.githubusercontent.com/docker/docker/${DIND_COMMIT}/hack/dind\" -O /usr/local/bin/dind \\\n && curl -L https://github.com/docker/compose/releases/download/${DOCKER_COMPOSE_VERSION}/docker-compose-Linux-x86_64 > /usr/local/bin/docker-compose \\\n && chmod +x /usr/local/bin/dind /usr/local/bin/docker-compose \\\n# Ensure docker-compose works\n && docker-compose version\n\nVOLUME /var/lib/docker\n\nCOPY dockerd-entrypoint.sh /usr/local/bin/\n\nENV PATH=\"/usr/local/bin:$PATH\" \\\n GPG_KEY=\"C01E1CAD5EA2C4F0B8E3571504C367C218ADD4FF\" \\\n PYTHON_VERSION=\"2.7.12\" \\\n PYTHON_PIP_VERSION=\"8.1.2\"\n\nRUN set -ex \\\n && apt-get update \\\n && apt-get install -y --no-install-recommends tcl-dev tk-dev \\\n && rm -rf /var/lib/apt/lists/* \\\n\t\\\n\t&& wget -O python.tar.xz \"https://www.python.org/ftp/python/${PYTHON_VERSION%%[a-z]*}/Python-$PYTHON_VERSION.tar.xz\" \\\n\t&& wget -O python.tar.xz.asc \"https://www.python.org/ftp/python/${PYTHON_VERSION%%[a-z]*}/Python-$PYTHON_VERSION.tar.xz.asc\" \\\n\t&& export GNUPGHOME=\"$(mktemp -d)\" \\\n\t&& (gpg --keyserver hkp://p80.pool.sks-keyservers.net:80 --recv-keys \"$GPG_KEY\" \\\n || gpg --keyserver pgp.mit.edu --recv-keys \"$GPG_KEY\" \\\n || gpg --keyserver keyserver.ubuntu.com --recv-keys \"$GPG_KEY\") \\\n\t&& gpg --batch --verify python.tar.xz.asc python.tar.xz \\\n\t&& rm -r \"$GNUPGHOME\" python.tar.xz.asc \\\n\t&& mkdir -p /usr/src/python \\\n\t&& tar -xJC /usr/src/python --strip-components=1 -f python.tar.xz \\\n\t&& rm python.tar.xz \\\n\t\\\n\t&& cd /usr/src/python \\\n\t&& ./configure \\\n\t\t--enable-shared \\\n\t\t--enable-unicode=ucs4 \\\n\t&& make -j$(nproc) \\\n\t&& make install \\\n\t&& ldconfig \\\n\t\\\n\t\t&& wget -O /tmp/get-pip.py 'https://bootstrap.pypa.io/get-pip.py' \\\n\t\t&& python2 /tmp/get-pip.py \"pip==$PYTHON_PIP_VERSION\" \\\n\t\t&& rm /tmp/get-pip.py \\\n# we use \"--force-reinstall\" for the case where the version of pip we're trying to install is the same as the version bundled with Python\n# (\"Requirement already up-to-date: pip==8.1.2 in /usr/local/lib/python3.6/site-packages\")\n# https://github.com/docker-library/python/pull/143#issuecomment-241032683\n\t&& pip install --no-cache-dir --upgrade --force-reinstall \"pip==$PYTHON_PIP_VERSION\" \\\n && pip install awscli==1.* --no-cache-dir \\\n# then we use \"pip list\" to ensure we don't have more than one pip version installed\n# https://github.com/docker-library/python/pull/100\n\t&& [ \"$(pip list |tac|tac| awk -F '[ ()]+' '$1 == \"pip\" { print $2; exit }')\" = \"$PYTHON_PIP_VERSION\" ] \\\n\t\\\n\t&& find /usr/local -depth \\\n\t\t\\( \\\n\t\t\t\\( -type d -a -name test -o -name tests \\) \\\n\t\t\t-o \\\n\t\t\t\\( -type f -a -name '*.pyc' -o -name '*.pyo' \\) \\\n\t\t\\) -exec rm -rf '{}' + \\\n\t&& apt-get purge -y --auto-remove tcl-dev tk-dev \\\n && rm -rf /usr/src/python ~/.cache\n\nENV JAVA_VERSION=8 \\\n JAVA_HOME=\"/usr/lib/jvm/java-8-openjdk-amd64\" \\\n JDK_VERSION=\"8u171-b11-2~14.04\" \\\n JDK_HOME=\"/usr/lib/jvm/java-8-openjdk-amd64\" \\\n JRE_HOME=\"/usr/lib/jvm/java-8-openjdk-amd64/jre\" \\\n ANT_VERSION=1.9.6 \\\n MAVEN_VERSION=3.3.3 \\\n MAVEN_HOME=\"/usr/share/maven\" \\\n MAVEN_CONFIG=\"/root/.m2\" \\\n GRADLE_VERSION=2.7 \\\n PROPERTIES_COMMON_VERSIION=0.92.37.8 \\\n PYTHON_TOOL_VERSION=\"3.3-*\"\n\n# Install Java\nRUN set -ex \\\n && apt-get update \\\n && apt-get install -y software-properties-common=$PROPERTIES_COMMON_VERSIION \\\n && add-apt-repository ppa:openjdk-r/ppa \\\n && apt-get update \\\n && apt-get -y install python-setuptools=$PYTHON_TOOL_VERSION \\\n && apt-get -y install openjdk-$JAVA_VERSION-jdk=$JDK_VERSION \\\n && apt-get clean \\\n # Ensure Java cacerts symlink points to valid location\n && update-ca-certificates -f \\\n && mkdir -p /usr/src/ant \\\n && wget \"http://archive.apache.org/dist/ant/binaries/apache-ant-$ANT_VERSION-bin.tar.gz\" -O /usr/src/ant/apache-ant-$ANT_VERSION-bin.tar.gz \\\n && tar -xzf /usr/src/ant/apache-ant-$ANT_VERSION-bin.tar.gz -C /usr/local \\\n && ln -s /usr/local/apache-ant-$ANT_VERSION/bin/ant /usr/bin/ant \\\n && rm -rf /usr/src/ant \\\n && mkdir -p /usr/share/maven /usr/share/maven/ref $MAVEN_CONFIG \\\n && curl -fsSL \"https://archive.apache.org/dist/maven/maven-3/$MAVEN_VERSION/binaries/apache-maven-$MAVEN_VERSION-bin.tar.gz\" \\\n | tar -xzC /usr/share/maven --strip-components=1 \\\n && ln -s /usr/share/maven/bin/mvn /usr/bin/mvn \\\n && mkdir -p /usr/src/gradle \\\n && wget \"https://services.gradle.org/distributions/gradle-$GRADLE_VERSION-bin.zip\" -O /usr/src/gradle/gradle-$GRADLE_VERSION-bin.zip \\\n && unzip /usr/src/gradle/gradle-$GRADLE_VERSION-bin.zip -d /usr/local \\\n && ln -s /usr/local/gradle-$GRADLE_VERSION/bin/gradle /usr/bin/gradle \\\n && rm -rf /usr/src/gradle \\\n && rm -fr /var/lib/apt/lists/* /tmp/* /var/tmp/*\n\nCOPY m2-settings.xml $MAVEN_CONFIG/settings.xml\n\n# MMS build environment\nRUN set -ex \\\n && apt-get update \\\n && pip install retrying \\\n && pip install mock \\\n && pip install pytest -U \\\n && pip install pylint\n\n# Install protobuf\nRUN wget https://github.com/google/protobuf/archive/v3.4.1.zip \\\n && unzip v3.4.1.zip && rm v3.4.1.zip \\\n && cd protobuf-3.4.1 && ./autogen.sh && ./configure --prefix=/usr && make && make install && cd .. \\\n && rm -r protobuf-3.4.1\n" }, { "path": "ci/Dockerfile.python3.6", "content": "# Copyright 2018 Amazon.com, Inc. or its affiliates. All Rights Reserved.\n# Licensed under the Apache License, Version 2.0 (the \"License\").\n# You may not use this file except in compliance with the License.\n# A copy of the License is located at\n# http://www.apache.org/licenses/LICENSE-2.0\n# or in the \"license\" file accompanying this file. This file is distributed\n# on an \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either\n# express or implied. See the License for the specific language governing\n# permissions and limitations under the License.\n#\n\nFROM ubuntu:14.04.5\n\nENV LANG=\"C.UTF-8\"\n\nENV DOCKER_BUCKET=\"download.docker.com\" \\\n DOCKER_VERSION=\"17.09.0-ce\" \\\n DOCKER_CHANNEL=\"stable\" \\\n DOCKER_SHA256=\"a9e90a73c3cdfbf238f148e1ec0eaff5eb181f92f35bdd938fd7dab18e1c4647\" \\\n DIND_COMMIT=\"3b5fac462d21ca164b3778647420016315289034\" \\\n DOCKER_COMPOSE_VERSION=\"1.16.1\" \\\n GITVERSION_VERSION=\"3.6.5\"\n\n# Install git\nRUN set -ex \\\n && apt-get update \\\n && apt-get install software-properties-common -y --no-install-recommends\\\n && apt-add-repository ppa:git-core/ppa \\\n && apt-get update \\\n && apt-get install git -y --no-install-recommends\\\n && git version\n\nRUN set -ex \\\n && echo 'Acquire::CompressionTypes::Order:: \"gz\";' > /etc/apt/apt.conf.d/99use-gzip-compression \\\n && apt-get update \\\n && apt-get install -y --no-install-recommends wget=1.15-* fakeroot=1.20-* ca-certificates \\\n autoconf=2.69-* automake=1:1.14.* less=458-* groff=1.22.* \\\n bzip2=1.0.* file=1:5.14-* g++=4:4.8.* gcc=4:4.8.* imagemagick=8:6.7.* \\\n libbz2-dev=1.0.* libc6-dev=2.19-* libcurl4-openssl-dev=7.35.* curl=7.35.* \\\n libdb-dev=1:5.3.* libevent-dev=2.0.* libffi-dev=3.1~* \\\n libgeoip-dev=1.6.* libglib2.0-dev=2.40.* libjpeg-dev=8c-* \\\n libkrb5-dev=1.12+* liblzma-dev=5.1.* libmagickcore-dev=8:6.7.* \\\n libmagickwand-dev=8:6.7.* libmysqlclient-dev=5.5.* libncurses5-dev=5.9+* \\\n libpng12-dev=1.2.* libpq-dev=9.3.* libreadline-dev=6.3-* libsqlite3-dev=3.8.* \\\n libssl-dev=1.0.* libtool=2.4.* libwebp-dev=0.4.* libxml2-dev=2.9.* \\\n libxslt1-dev=1.1.* libyaml-dev=0.1.* make=3.81-* patch=2.7.* xz-utils=5.1.* \\\n zlib1g-dev=1:1.2.* tcl=8.6.* tk=8.6.* \\\n e2fsprogs=1.42.* iptables=1.4.* xfsprogs=3.1.* xz-utils=5.1.* \\\n mono-mcs=3.2.* libcurl4-openssl-dev=7.35.* liberror-perl=0.17-* unzip=6.0-*\\\n && rm -rf /var/lib/apt/lists/* \\\n && apt-get clean\n\n# Download and set up GitVersion\nRUN set -ex \\\n && wget \"https://github.com/GitTools/GitVersion/releases/download/v${GITVERSION_VERSION}/GitVersion_${GITVERSION_VERSION}.zip\" -O /tmp/GitVersion_${GITVERSION_VERSION}.zip \\\n && mkdir -p /usr/local/GitVersion_${GITVERSION_VERSION} \\\n && unzip /tmp/GitVersion_${GITVERSION_VERSION}.zip -d /usr/local/GitVersion_${GITVERSION_VERSION} \\\n && rm /tmp/GitVersion_${GITVERSION_VERSION}.zip \\\n && echo \"mono /usr/local/GitVersion_${GITVERSION_VERSION}/GitVersion.exe /output json /showvariable \\$1\" >> /usr/local/bin/gitversion \\\n && chmod +x /usr/local/bin/gitversion\n# Install Docker\nRUN set -ex \\\n && curl -fSL \"https://${DOCKER_BUCKET}/linux/static/${DOCKER_CHANNEL}/x86_64/docker-${DOCKER_VERSION}.tgz\" -o docker.tgz \\\n && echo \"${DOCKER_SHA256} *docker.tgz\" | sha256sum -c - \\\n && tar --extract --file docker.tgz --strip-components 1 --directory /usr/local/bin/ \\\n && rm docker.tgz \\\n && docker -v \\\n# set up subuid/subgid so that \"--userns-remap=default\" works out-of-the-box\n && addgroup dockremap \\\n && useradd -g dockremap dockremap \\\n && echo 'dockremap:165536:65536' >> /etc/subuid \\\n && echo 'dockremap:165536:65536' >> /etc/subgid \\\n && wget \"https://raw.githubusercontent.com/docker/docker/${DIND_COMMIT}/hack/dind\" -O /usr/local/bin/dind \\\n && curl -L https://github.com/docker/compose/releases/download/${DOCKER_COMPOSE_VERSION}/docker-compose-Linux-x86_64 > /usr/local/bin/docker-compose \\\n && chmod +x /usr/local/bin/dind /usr/local/bin/docker-compose \\\n# Ensure docker-compose works\n && docker-compose version\n\nVOLUME /var/lib/docker\n\nCOPY dockerd-entrypoint.sh /usr/local/bin/\n\nENV PATH=\"/usr/local/bin:$PATH\" \\\n GPG_KEY=\"0D96DF4D4110E5C43FBFB17F2D347EA6AA65421D\" \\\n PYTHON_VERSION=\"3.6.5\" \\\n PYTHON_PIP_VERSION=\"10.0.0\" \\\n LC_ALL=C.UTF-8 \\\n LANG=C.UTF-8\n\nRUN apt-get update && apt-get install -y --no-install-recommends \\\n tcl-dev tk-dev \\\n && rm -rf /var/lib/apt/lists/* \\\n \\\n && wget -O python.tar.xz \"https://www.python.org/ftp/python/${PYTHON_VERSION%%[a-z]*}/Python-$PYTHON_VERSION.tar.xz\" \\\n\t&& wget -O python.tar.xz.asc \"https://www.python.org/ftp/python/${PYTHON_VERSION%%[a-z]*}/Python-$PYTHON_VERSION.tar.xz.asc\" \\\n\t&& export GNUPGHOME=\"$(mktemp -d)\" \\\n\t&& (gpg --keyserver hkp://p80.pool.sks-keyservers.net:80 --recv-keys \"$GPG_KEY\" \\\n || gpg --keyserver pgp.mit.edu --recv-keys \"$GPG_KEY\" \\\n || gpg --keyserver keyserver.ubuntu.com --recv-keys \"$GPG_KEY\") \\\n\t&& gpg --batch --verify python.tar.xz.asc python.tar.xz \\\n\t&& rm -r \"$GNUPGHOME\" python.tar.xz.asc \\\n\t&& mkdir -p /usr/src/python \\\n\t&& tar -xJC /usr/src/python --strip-components=1 -f python.tar.xz \\\n\t&& rm python.tar.xz \\\n\t\\\n\t&& cd /usr/src/python \\\n\t&& ./configure \\\n\t\t--enable-loadable-sqlite-extensions \\\n\t\t--enable-shared \\\n\t&& make -j$(nproc) \\\n\t&& make install \\\n\t&& ldconfig \\\n\t\\\n# explicit path to \"pip3\" to ensure distribution-provided \"pip3\" cannot interfere\n\t&& if [ ! -e /usr/local/bin/pip3 ]; then : \\\n\t\t&& wget -O /tmp/get-pip.py 'https://bootstrap.pypa.io/get-pip.py' \\\n\t\t&& python3 /tmp/get-pip.py \"pip==$PYTHON_PIP_VERSION\" \\\n\t\t&& rm /tmp/get-pip.py \\\n\t; fi \\\n# we use \"--force-reinstall\" for the case where the version of pip we're trying to install is the same as the version bundled with Python\n# (\"Requirement already up-to-date: pip==8.1.2 in /usr/local/lib/python3.6/site-packages\")\n# https://github.com/docker-library/python/pull/143#issuecomment-241032683\n\t&& pip3 install --no-cache-dir --upgrade --force-reinstall \"pip==$PYTHON_PIP_VERSION\" \\\n && pip install awscli==1.* boto3 pipenv virtualenv --no-cache-dir \\\n# then we use \"pip list\" to ensure we don't have more than one pip version installed\n# https://github.com/docker-library/python/pull/100\n\t&& [ \"$(pip list |tac|tac| awk -F '[ ()]+' '$1 == \"pip\" { print $2; exit }')\" = \"$PYTHON_PIP_VERSION\" ] \\\n\t\\\n\t&& find /usr/local -depth \\\n\t\t\\( \\\n\t\t\t\\( -type d -a -name test -o -name tests \\) \\\n\t\t\t-o \\\n\t\t\t\\( -type f -a -name '*.pyc' -o -name '*.pyo' \\) \\\n\t\t\\) -exec rm -rf '{}' + \\\n\t&& apt-get purge -y --auto-remove tcl-dev tk-dev \\\n\t&& rm -rf /usr/src/python ~/.cache \\\n\t&& cd /usr/local/bin \\\n\t&& { [ -e easy_install ] || ln -s easy_install-* easy_install; } \\\n\t&& ln -s idle3 idle \\\n\t&& ln -s pydoc3 pydoc \\\n\t&& ln -s python3 python \\\n\t&& ln -s python3-config python-config \\\n && rm -fr /var/lib/apt/lists/* /tmp/* /var/tmp/*\n\nENV JAVA_VERSION=8 \\\n JAVA_HOME=\"/usr/lib/jvm/java-8-openjdk-amd64\" \\\n JDK_VERSION=\"8u171-b11-2~14.04\" \\\n JDK_HOME=\"/usr/lib/jvm/java-8-openjdk-amd64\" \\\n JRE_HOME=\"/usr/lib/jvm/java-8-openjdk-amd64/jre\" \\\n ANT_VERSION=1.9.6 \\\n MAVEN_VERSION=3.3.3 \\\n MAVEN_HOME=\"/usr/share/maven\" \\\n MAVEN_CONFIG=\"/root/.m2\" \\\n GRADLE_VERSION=2.7 \\\n PROPERTIES_COMMON_VERSIION=0.92.37.8 \\\n PYTHON_TOOL_VERSION=\"3.3-*\"\n\n# Install Java\nRUN set -ex \\\n && apt-get update \\\n && apt-get install -y software-properties-common=$PROPERTIES_COMMON_VERSIION \\\n && add-apt-repository ppa:openjdk-r/ppa \\\n && apt-get update \\\n && apt-get -y install python-setuptools=$PYTHON_TOOL_VERSION \\\n && apt-get -y install openjdk-$JAVA_VERSION-jdk=$JDK_VERSION \\\n && apt-get clean \\\n # Ensure Java cacerts symlink points to valid location\n && update-ca-certificates -f \\\n && mkdir -p /usr/src/ant \\\n && wget \"http://archive.apache.org/dist/ant/binaries/apache-ant-$ANT_VERSION-bin.tar.gz\" -O /usr/src/ant/apache-ant-$ANT_VERSION-bin.tar.gz \\\n && tar -xzf /usr/src/ant/apache-ant-$ANT_VERSION-bin.tar.gz -C /usr/local \\\n && ln -s /usr/local/apache-ant-$ANT_VERSION/bin/ant /usr/bin/ant \\\n && rm -rf /usr/src/ant \\\n && mkdir -p /usr/share/maven /usr/share/maven/ref $MAVEN_CONFIG \\\n && curl -fsSL \"https://archive.apache.org/dist/maven/maven-3/$MAVEN_VERSION/binaries/apache-maven-$MAVEN_VERSION-bin.tar.gz\" \\\n | tar -xzC /usr/share/maven --strip-components=1 \\\n && ln -s /usr/share/maven/bin/mvn /usr/bin/mvn \\\n && mkdir -p /usr/src/gradle \\\n && wget \"https://services.gradle.org/distributions/gradle-$GRADLE_VERSION-bin.zip\" -O /usr/src/gradle/gradle-$GRADLE_VERSION-bin.zip \\\n && unzip /usr/src/gradle/gradle-$GRADLE_VERSION-bin.zip -d /usr/local \\\n && ln -s /usr/local/gradle-$GRADLE_VERSION/bin/gradle /usr/bin/gradle \\\n && rm -rf /usr/src/gradle \\\n && rm -fr /var/lib/apt/lists/* /tmp/* /var/tmp/*\n\nCOPY m2-settings.xml $MAVEN_CONFIG/settings.xml\n\n# MMS build environment\nRUN set -ex \\\n && apt-get update \\\n && pip install retrying \\\n && pip install mock \\\n && pip install pytest -U \\\n && pip install pylint\n\n# Install protobuf\nRUN wget https://github.com/google/protobuf/archive/v3.4.1.zip \\\n && unzip v3.4.1.zip && rm v3.4.1.zip \\\n && cd protobuf-3.4.1 && ./autogen.sh && ./configure --prefix=/usr && make && make install && cd .. \\\n && rm -r protobuf-3.4.1\n" }, { "path": "ci/README.md", "content": "# Model Server CI build\n\nModel Server us AWS codebuild for its CI build. This folder contains scripts that needed for AWS codebuild.\n\n## buildspec.yml\nbuildspec.yml contains MMS build logic which will be used by AWS codebuild.\n\n## Docker images\nMMS use customized docker image for its AWS codebuild. To make sure MMS is compatible with\n both Python2 and Python3, we use two build projects. We published two codebuild docker\n images on docker hub:\n* awsdeeplearningteam/mms-build:python2.7\n* awsdeeplearningteam/mms-build:python3.6\n\nFollowing files in this folder is used to create the docker images\n* Dockerfile.python2.7 - Dockerfile for awsdeeplearningteam/mms-build:python2.7\n* Dockerfile.python3.6 - Dockerfile for awsdeeplearningteam/mms-build:python3.6\n* dockerd-entrypoint.sh - AWS codebuild entrypoint script, required by AWS codebuild\n* m2-settings.xml - Limit with repository can be used by maven/gradle in docker container, provided by AWS codebuild.\n\n## AWS codebuild local\nTo make it easy for developer debug build issue locally, MMS support AWS codebuild local.\nDeveloper can use following command to build MMS locally:\n```bash\n$ cd multi-model-server\n$ ./run_ci_tests.sh\n```\n\nTo avoid Pull Request build failure on github, developer should always make sure local build can pass.\n" }, { "path": "ci/buildspec.yml", "content": "# Build Spec for AWS CodeBuild CI\n\nversion: 0.2\n\nphases:\n install:\n commands:\n - apt-get update\n - apt-get install -y curl\n - pip install pip -U\n - pip install future\n - pip install Pillow\n - pip install pytest==4.0.0\n - pip install wheel\n - pip install twine\n - pip install pytest-mock -U\n - pip install requests\n - pip install -U -e .\n - pip install mxnet==1.5.0\n - cd model-archiver/ && pip install -U -e . && cd ../\n\n build:\n commands:\n - frontend/gradlew -p frontend build\n - python -m pytest mms/tests/unit_tests\n - cd model-archiver/ && python -m pytest model_archiver/tests/unit_tests && cd ../\n - cd model-archiver/ && python -m pytest model_archiver/tests/integ_tests && cd ../\n - cd serving-sdk/ && mvn clean deploy && cd ../\n # integration test is broken: https://github.com/awslabs/multi-model-server/issues/437\n #- python -m pytest mms/tests/integration_tests\n - pylint -rn --rcfile=./mms/tests/pylintrc mms/.\n - cd model-archiver/ && pylint -rn --rcfile=./model_archiver/tests/pylintrc model_archiver/. && cd ../\n - $NIGHTLYBUILD\n - eval $NIGHTLYUPLOAD\n\nartifacts:\n files:\n - dist/*.whl\n - model_archiver/dist/*.whl\n - frontend/server/build/reports/**/*\n - frontend/modelarchive/build/reports/**/*\n - frontend/cts/build/reports/**/*\n" }, { "path": "ci/dockerd-entrypoint.sh", "content": "#!/bin/sh\nset -e\n\n/usr/local/bin/dockerd \\\n\t--host=unix:///var/run/docker.sock \\\n\t--host=tcp://127.0.0.1:2375 \\\n\t--storage-driver=overlay &>/var/log/docker.log &\n\n\ntries=0\nd_timeout=60\nuntil docker info >/dev/null 2>&1\ndo\n\tif [ \"$tries\" -gt \"$d_timeout\" ]; then\n cat /var/log/docker.log\n\t\techo 'Timed out trying to connect to internal docker host.' >&2\n\t\texit 1\n\tfi\n tries=$(( $tries + 1 ))\n\tsleep 1\ndone\n\neval \"$@\"\n" }, { "path": "ci/m2-settings.xml", "content": "\n \n \n securecentral\n \n true\n \n \n \n \n central\n https://repo1.maven.org/maven2\n \n true\n \n \n \n \n \n central\n https://repo1.maven.org/maven2\n \n true\n \n \n \n \n \n\n" }, { "path": "docker/Dockerfile.cpu", "content": "FROM ubuntu:18.04\n\nENV PYTHONUNBUFFERED TRUE\n\nRUN apt-get update && \\\n DEBIAN_FRONTEND=noninteractive apt-get install --no-install-recommends -y \\\n fakeroot \\\n ca-certificates \\\n dpkg-dev \\\n g++ \\\n python3-dev \\\n openjdk-8-jdk-headless \\\n curl \\\n vim \\\n && rm -rf /var/lib/apt/lists/* \\\n && cd /tmp \\\n && curl -O https://bootstrap.pypa.io/get-pip.py \\\n && python3 get-pip.py\n\nRUN update-alternatives --install /usr/bin/python python /usr/bin/python3 1\nRUN update-alternatives --install /usr/local/bin/pip pip /usr/local/bin/pip3 1\n\nRUN pip install --no-cache-dir multi-model-server \\\n && pip install --no-cache-dir mxnet-mkl==1.4.0\n\nRUN useradd -m model-server \\\n && mkdir -p /home/model-server/tmp\n\nCOPY dockerd-entrypoint.sh /usr/local/bin/dockerd-entrypoint.sh\nCOPY config.properties /home/model-server\n\nRUN chmod +x /usr/local/bin/dockerd-entrypoint.sh \\\n && chown -R model-server /home/model-server\n\nEXPOSE 8080 8081\n\nUSER model-server\nWORKDIR /home/model-server\nENV TEMP=/home/model-server/tmp\nENTRYPOINT [\"/usr/local/bin/dockerd-entrypoint.sh\"]\nCMD [\"serve\"]\n\nLABEL maintainer=\"dantu@amazon.com, rakvas@amazon.com, lufen@amazon.com, dden@amazon.com\"\n" }, { "path": "docker/Dockerfile.gpu", "content": "FROM nvidia/cuda:9.2-cudnn7-runtime-ubuntu18.04\n\nENV PYTHONUNBUFFERED TRUE\n\nRUN apt-get update && \\\n DEBIAN_FRONTEND=noninteractive apt-get install --no-install-recommends -y \\\n fakeroot \\\n ca-certificates \\\n dpkg-dev \\\n g++ \\\n python3-dev \\\n openjdk-8-jdk-headless \\\n curl \\\n vim \\\n && rm -rf /var/lib/apt/lists/* \\\n && cd /tmp \\\n && curl -O https://bootstrap.pypa.io/get-pip.py \\\n && python3 get-pip.py\n\nRUN update-alternatives --install /usr/bin/python python /usr/bin/python3 1\nRUN update-alternatives --install /usr/local/bin/pip pip /usr/local/bin/pip3 1\n\nRUN pip install --no-cache-dir multi-model-server \\\n && pip install --no-cache-dir mxnet-cu92mkl==1.4.0\n\nRUN useradd -m model-server \\\n && mkdir -p /home/model-server/tmp\n\nCOPY dockerd-entrypoint.sh /usr/local/bin/dockerd-entrypoint.sh\nCOPY config.properties /home/model-server\n\nRUN chmod +x /usr/local/bin/dockerd-entrypoint.sh \\\n && chown -R model-server /home/model-server\n\nEXPOSE 8080 8081\n\nUSER model-server\nWORKDIR /home/model-server\nENV TEMP=/home/model-server/tmp\nENTRYPOINT [\"/usr/local/bin/dockerd-entrypoint.sh\"]\nCMD [\"serve\"]\n\nLABEL maintainer=\"dantu@amazon.com, rakvas@amazon.com, lufen@amazon.com, dden@amazon.com\"\n" }, { "path": "docker/Dockerfile.nightly-cpu", "content": "FROM ubuntu:18.04\n\nENV PYTHONUNBUFFERED TRUE\n\nRUN apt-get update && \\\n DEBIAN_FRONTEND=noninteractive apt-get install --no-install-recommends -y \\\n fakeroot \\\n ca-certificates \\\n dpkg-dev \\\n g++ \\\n python3-dev \\\n openjdk-8-jdk-headless \\\n curl \\\n vim \\\n && rm -rf /var/lib/apt/lists/* \\\n && cd /tmp \\\n && curl -O https://bootstrap.pypa.io/get-pip.py \\\n && python3 get-pip.py\n\nRUN update-alternatives --install /usr/bin/python python /usr/bin/python3 1\nRUN update-alternatives --install /usr/local/bin/pip pip /usr/local/bin/pip3 1\n\nRUN pip install --no-cache-dir --pre multi-model-server \\\n && pip install --no-cache-dir mxnet-mkl\n\nRUN useradd -m model-server \\\n && mkdir -p /home/model-server/tmp\n\nCOPY dockerd-entrypoint.sh /usr/local/bin/dockerd-entrypoint.sh\nCOPY config.properties /home/model-server\n\nRUN chmod +x /usr/local/bin/dockerd-entrypoint.sh \\\n && chown -R model-server /home/model-server\n\nEXPOSE 8080 8081\n\nUSER model-server\nWORKDIR /home/model-server\nENV TEMP=/home/model-server/tmp\nENTRYPOINT [\"/usr/local/bin/dockerd-entrypoint.sh\"]\nCMD [\"serve\"]\n\nLABEL maintainer=\"dantu@amazon.com, rakvas@amazon.com, lufen@amazon.com, dden@amazon.com\"\n" }, { "path": "docker/Dockerfile.nightly-gpu", "content": "FROM nvidia/cuda:9.2-cudnn7-runtime-ubuntu18.04\n\nENV PYTHONUNBUFFERED TRUE\n\nRUN apt-get update && \\\n DEBIAN_FRONTEND=noninteractive apt-get install --no-install-recommends -y \\\n fakeroot \\\n ca-certificates \\\n dpkg-dev \\\n g++ \\\n python3-dev \\\n openjdk-8-jdk-headless \\\n curl \\\n vim \\\n && rm -rf /var/lib/apt/lists/* \\\n && cd /tmp \\\n && curl -O https://bootstrap.pypa.io/get-pip.py \\\n && python3 get-pip.py\n\nRUN update-alternatives --install /usr/bin/python python /usr/bin/python3 1\nRUN update-alternatives --install /usr/local/bin/pip pip /usr/local/bin/pip3 1\n\nRUN pip install --no-cache-dir --pre multi-model-server \\\n && pip install --no-cache-dir mxnet-cu92mkl\n\nRUN useradd -m model-server \\\n && mkdir -p /home/model-server/tmp\n\nCOPY dockerd-entrypoint.sh /usr/local/bin/dockerd-entrypoint.sh\nCOPY config.properties /home/model-server\n\nRUN chmod +x /usr/local/bin/dockerd-entrypoint.sh \\\n && chown -R model-server /home/model-server\n\nEXPOSE 8080 8081\n\nUSER model-server\nWORKDIR /home/model-server\nENV TEMP=/home/model-server/tmp\nENTRYPOINT [\"/usr/local/bin/dockerd-entrypoint.sh\"]\nCMD [\"serve\"]\n\nLABEL maintainer=\"dantu@amazon.com, rakvas@amazon.com, lufen@amazon.com, dden@amazon.com\"\n" }, { "path": "docker/README.md", "content": "[//]: # \"All the references in this file should be actual links because this file would be used by docker hub. DO NOT use relative links or section tagging.\"\n\n# Using Containers with Multi Model Server\n\nMulti Model Server (MMS) can be used with any container service. In this guide, you will learn how to run MMS with Docker.\n\n## Contents of this Document\n* [Quickstart](https://github.com/awslabs/multi-model-server/blob/master/docker/README.md#quickstart)\n* [Available pre-built containers](https://github.com/awslabs/multi-model-server/blob/master/docker/README.md#available-pre-built-containers)\n* [Configuring MMS with Docker](https://github.com/awslabs/multi-model-server/blob/master/docker/README.md#configuring-mms-with-docker)\n\n\n## Other Relevant Documents\n* [Advanced Settings](https://github.com/awslabs/multi-model-server/blob/master/docker/advanced_settings.md)\n * [GPU Inference](https://github.com/awslabs/multi-model-server/blob/master/docker/advanced_settings.md#gpu-inference)\n * [Reference Commands](https://github.com/awslabs/multi-model-server/blob/master/docker/advanced_settings.md#reference-commands)\n * [Docker Details](https://github.com/awslabs/multi-model-server/blob/master/docker/advanced_settings.md#docker-details)\n * [Description of Config File Settings](https://github.com/awslabs/multi-model-server/blob/master/docker/advanced_settings.md#description-of-config-file-settings)\n * [Configuring SSL](https://github.com/awslabs/multi-model-server/blob/master/docker/advanced_settings.md#configuring-ssl)\n* [Launch MMS as a managed inference service on AWS Fargate](https://github.com/awslabs/multi-model-server/blob/master/docs/mms_on_fargate.md)\n * [Introduction to published containers](https://github.com/awslabs/multi-model-server/blob/master/docs/mms_on_fargate.md#familiarize-yourself-with-our-containers)\n * [Creating a AWS Fargate task to server SqueezeNet V1.1](https://github.com/awslabs/multi-model-server/blob/master/docs/mms_on_fargate.md#create-a-aws-faragte-task-to-serve-squeezenet-model)\n * [Creating an Load Balancer](https://github.com/awslabs/multi-model-server/blob/master/docs/mms_on_fargate.md#create-a-load-balancer)\n * [Creating an AWS ECS Service](https://github.com/awslabs/multi-model-server/blob/master/docs/mms_on_fargate.md#creating-an-ecs-service-to-launch-our-aws-fargate-task)\n * [Testing your service](https://github.com/awslabs/multi-model-server/blob/master/docs/mms_on_fargate.md#test-your-service)\n * [Build custom MMS containers images to serve your Deep learning models](https://github.com/awslabs/multi-model-server/blob/master/docs/mms_on_fargate.md#customize-the-containers-to-server-your-custom-deep-learning-models)\n\n## Quickstart\nRunning Multi Model Server with Docker in two steps:\n\n**Step 1: Run the Docker image.**\n\nThis will download the MMS Docker image and run its default configuration, serving a SqueezeNet model.\n\n```bash\ndocker run -itd --name mms -p 80:8080 -p 8081:8081 awsdeeplearningteam/multi-model-server multi-model-server --start --models squeezenet=https://s3.amazonaws.com/model-server/model_archive_1.0/squeezenet_v1.1.mar\n```\n\nWith the `-p` flag, we're setting it up so you can run the Predict API on your host computer's port `80`. This maps to the Docker image's port `8080`.\nIt will run the Management API on your host computer's port `8081`. This maps to the Docker image's port `8081`.\n\n**Step 2: Test inference.**\n\n```bash\ncurl -O https://s3.amazonaws.com/model-server/inputs/kitten.jpg\ncurl -X POST http://127.0.0.1/predictions/squeezenet -T kitten.jpg\n```\n\nAfter fetching this image of a kitten and posting it to the `predict` endpoint, you should see a response similar to the following:\n\n```\n{\n \"prediction\": [\n [\n {\n \"class\": \"n02124075 Egyptian cat\",\n \"probability\": 0.9408261179924011\n...\n```\n\n### Cleaning Up\n\nNow that you have tested it out, you may stop the Docker container. The following command will stop the server and delete the container. It will retain the Docker image for trying out other models and configurations later.\n\n```bash\ndocker rm -f mms\n```\n\n## Available pre-built containers\nWe have following container tags available on [Docker Hub](https://hub.docker.com/r/awsdeeplearningteam/multi-model-server/).\n1. *latest*: This is the latest officially released MMS CPU container. This is based on the latest [Dockerfile.cpu](https://github.com/awslabs/multi-model-server/blob/master/docker/Dockerfile.cpu).\n2. *latest-gpu*: This is the latest officially released MMS GPU container. This is based on the latest [Dockerfile.gpu](https://github.com/awslabs/multi-model-server/blob/master/docker/Dockerfile.gpu).\n3. *(MMS Release Tag)-mxnet-cpu*: Each released version since MMS 1.0.0 has an individual release tagged CPU MXNet container. These containers are based on [Dockerfile.cpu](https://github.com/awslabs/multi-model-server/blob/master/docker/Dockerfile.cpu), in that MMS release.\n4. *(MMS Release Tag)-mxnet-cpu*: Each released version since MMS 1.0.0 has an individual release tagged GPU MXNet container. These containers are based on [Dockerfile.gpu](https://github.com/awslabs/multi-model-server/blob/master/docker/Dockerfile.gpu), in that MMS release.\n5. *nightly-mxnet-cpu*: This is the official CPU container which is built based on the nightly release of MMS pip package. This is built from [Dockerfile.nightly-cpu](https://github.com/awslabs/multi-model-server/blob/master/docker/Dockerfile.nightly-cpu).\n6. *nightly-mxnet-gpu*: This is the official GPU container which is built based on the nightly release of MMS pip package. This is built from [Dockerfile.nightly-gpu](https://github.com/awslabs/multi-model-server/blob/master/docker/Dockerfile.nightly-gpu).\n7. *base-cpu-py2.7*: This is the official Base Python 2.7 CPU container which contains only MMS and python. This is built from [Dockerfile.nightly-gpu](https://github.com/awslabs/multi-model-server/blob/master/docker/advanced-dockerfiles/Dockerfile.base.ubuntu_16_04.py2_7). Please note, this container doesn't have any DL/ML engine installed by definition, it is meant to be used for cases where you would like to bring your own engine/framework into container. **WARNING: Python 2.x will be deprecated from Jan 1 2020.**\n8. *base-cpu-py3.6*: This is the official Base Python 3.6 CPU container which contains only MMS and python. This is built from [Dockerfile.nightly-gpu](https://github.com/awslabs/multi-model-server/blob/master/docker/advanced-dockerfiles/Dockerfile.base.ubuntu_16_04.py3_6). Please note, this container doesn't have any DL/ML engine installed by definition, it is meant to be used for cases where you would like to bring your own engine/framework into container.\n9. *base-gpu-py2.7*: This is the official Base Python 2.7 GPU container which contains only MMS and python. This is built from [Dockerfile.nightly-gpu](https://github.com/awslabs/multi-model-server/blob/master/docker/advanced-dockerfiles/Dockerfile.base.nvidia_cu92_ubuntu_16_04.py2_7). Please note, this container doesn't have any DL/ML engine installed by definition, it is meant to be used for cases where you would like to bring your own engine/framework into container. **WARNING: Python 2.x will be deprecated from Jan 1 2020.**\n10. *base-gpu-py3.6*: This is the official Base Python 3.6 GPU container which contains only MMS and python. This is built from [Dockerfile.nightly-gpu](https://github.com/awslabs/multi-model-server/blob/master/docker/advanced-dockerfiles/Dockerfile.base.nvidia_cu92_ubuntu_16_04.py3_6). Please note, this container doesn't have any DL/ML engine installed by definition, it is meant to be used for cases where you would like to bring your own engine/framework into container.\n11. *nightly-base-cpu-py2.7*: This is the official Nightly Base Python 2.7 CPU container which contains only MMS and python. This is built from [Dockerfile.nightly-gpu](https://github.com/awslabs/multi-model-server/blob/master/docker/advanced-dockerfiles/Dockerfile.base.ubuntu_16_04.py2_7). Please note, this container doesn't have any DL/ML engine installed by definition, it is meant to be used for cases where you would like to bring your own engine/framework into container. **WARNING: Python 2.x will be deprecated from Jan 1 2020.**\n12. *nightly-base-cpu-py3.6*: This is the official Nightly Base Python 3.6 CPU container which contains only MMS and python. This is built from [Dockerfile.nightly-gpu](https://github.com/awslabs/multi-model-server/blob/master/docker/advanced-dockerfiles/Dockerfile.base.ubuntu_16_04.py3_6). Please note, this container doesn't have any DL/ML engine installed by definition, it is meant to be used for cases where you would like to bring your own engine/framework into container.\n13. *nightly-base-gpu-py2.7*: This is the official Nightly Base Python 2.7 GPU container which contains only MMS and python. This is built from [Dockerfile.nightly-gpu](https://github.com/awslabs/multi-model-server/blob/master/docker/advanced-dockerfiles/Dockerfile.base.nvidia_cu92_ubuntu_16_04.py2_7). Please note, this container doesn't have any DL/ML engine installed by definition, it is meant to be used for cases where you would like to bring your own engine/framework into container. **WARNING: Python 2.x will be deprecated from Jan 1 2020.**\n14. *nightly-base-gpu-py3.6*: This is the official Nightly Base Python 3.6 GPU container which contains only MMS and python. This is built from [Dockerfile.nightly-gpu](https://github.com/awslabs/multi-model-server/blob/master/docker/advanced-dockerfiles/Dockerfile.base.nvidia_cu92_ubuntu_16_04.py3_6). Please note, this container doesn't have any DL/ML engine installed by definition, it is meant to be used for cases where you would like to bring your own engine/framework into container.\n\nTo pull the a particular container, run the following command\n\n#### Pulling the latest CPU container:\nDocker pull by default pulls the latest tag. This tag is associated with latest released MMS CPU container. This tag isn't available until after an official release. \n```bash\ndocker pull awsdeeplearningteam/multi-model-server \n``` \n\n#### Pulling the latest GPU container:\nTo pull a official latest released MMS GPU container run the following command. This tag isn't available until after an official release. \n```bash\ndocker pull awsdeeplearningteam/multi-model-server:latest-gpu\n``` \n\n#### Pulling the `nightly-mxnet-cpu` tag:\nTo pull a latest nigthtly MMS CPU container run the following command. This track the pre-release version of MMS.\nWe do not recommend running this container in production setup.\n```bash\ndocker pull awsdeeplearningteam/multi-model-server:nightly-mxnet-cpu\n``` \n\n#### Pulling the `nightly-mxnet-gpu` tag:\nTo pull a latest nigthtly MMS CPU container run the following command. This track the pre-release version of MMS.\nWe do not recommend running this container in production setup.\n```bash\ndocker pull awsdeeplearningteam/multi-model-server:nightly-mxnet-gpu\n``` \n\n## Configuring MMS with Docker\n\nIn the Quickstart section, you launched a Docker image with MMS serving the SqueezeNet model.\nNow you will learn how to configure MMS with Docker to run other models.\nYou will also learn how to collect MMS logs, and optimize MMS with Docker images.\n\n### Using MMS and Docker with a Shared Volume\n\nYou may sometimes want to load different models with a different configuration.\nSetting up a shared volume with the Docker image is the recommended way to handle this.\n\n**Step 1: Create a folder to share with the Docker container.**\n\nCreate a directory for `models`. This will also provide a place for log files to be written.\n\n```bash\nmkdir /tmp/models\n```\n\n**Step 2: Download the configuration template.**\n\nDownload the template `config.properties` and place it in the `models` folder you just created:\n* [config.properties](https://github.com/awslabs/multi-model-server/blob/master/docker/config.properties)\n\n**Step 3: Modify the configuration template.**\n\nEdit the file you downloaded, `config.properties`.\n\n```properties\nvmargs=-Xmx128m -XX:-UseLargePages -XX:+UseG1GC -XX:MaxMetaspaceSize=32M -XX:MaxDirectMemorySize=10m -XX:+ExitOnOutOfMemoryError\nmodel_store=/opt/ml/model\nload_models=ALL\ninference_address=http://0.0.0.0:8080\nmanagement_address=http://0.0.0.0:8081\n# management_address=unix:/tmp/management.sock\n# number_of_netty_threads=0\n# netty_client_threads=0\n# default_response_timeout=120\n# unregister_model_timeout=120\n# default_workers_per_model=0\n# job_queue_size=100\n# async_logging=false\n# number_of_gpu=1\n# cors_allowed_origin\n# cors_allowed_methods\n# cors_allowed_headers\n# keystore=src/test/resources/keystore.p12\n# keystore_pass=changeit\n# keystore_type=PKCS12\n# private_key_file=src/test/resources/key.pem\n# certificate_file=src/test/resources/certs.pem\n# blacklist_env_vars=\n```\n\nModify the configuration file to suite your configuration needs before running the model server.\n\nSave the file.\n\n**Step 4: Run MMS with Docker using a shared volume.**\n\nWhen you run the following command, the `-v` argument and path values of `/tmp/models/:/models` will map the Docker image's `models` folder to your local `/tmp/models` folder.\nMMS will then be able to use the local model file.\n\n```bash\ndocker run -itd --name mms -p 80:8080 -p 8081:8081 -v /tmp/models/:/models awsdeeplearningteam/multi-model-server multi-model-server --start --models squeezenet=https://s3.amazonaws.com/model-server/model_archive_1.0/resnet-18.mar\n```\n\n**NOTE**: If you modify the inference_address or the management_address in the configuration file,\nyou must modify the ports exposed by Docker as well.\n\n**Step 5: Test inference.**\n\nYou will upload the same kitten image as before, but this time you will request the `predictions/resnet` API endpoint.\n\n```bash\ncurl -X POST http://127.0.0.1/predictions/resnet -T @kitten.jpg\n```\n\nGiven that this is a different model, the same image yields a different inference result which is something similar to the following:\n\n```\n{\n \"prediction\": [\n [\n {\n \"class\": \"n02123159 tiger cat\",\n \"probability\": 0.3630334138870239\n },\n...\n```\n\n## Conclusion\n\nYou have tried the default Predictions API settings using a SqueezeNet model. \nYou then configured your Predictions API endpoints to also serve a ResNet-18 model.\nNow you are ready to try some other more **advanced settings** such as:\n\n* GPU inference\n* MMS settings\n\nNext Step: [Advanced Settings](https://github.com/awslabs/multi-model-server/blob/master/docker/advanced_settings.md)\n" }, { "path": "docker/advanced-dockerfiles/Dockerfile.base.nvidia_cu92_ubuntu_16_04.py2_7", "content": "FROM nvidia/cuda:9.2-cudnn7-runtime-ubuntu16.04\n\nENV PYTHONUNBUFFERED TRUE\n\nRUN apt-get update && \\\n DEBIAN_FRONTEND=noninteractive apt-get install --no-install-recommends -y \\\n fakeroot \\\n ca-certificates \\\n dpkg-dev \\\n g++ \\\n python-dev \\\n openjdk-8-jdk-headless \\\n curl \\\n vim \\\n && rm -rf /var/lib/apt/lists/* \\\n && cd /tmp \\\n && curl -O https://bootstrap.pypa.io/get-pip.py \\\n && python get-pip.py\n\nRUN update-alternatives --install /usr/bin/python python /usr/bin/python2.7 1\n\nRUN pip install --no-cache-dir multi-model-server\n\nRUN mkdir -p /home/model-server/tmp\n\nCOPY dockerd-entrypoint.sh /usr/local/bin/dockerd-entrypoint.sh\nCOPY config.properties /home/model-server\n\nRUN chmod +x /usr/local/bin/dockerd-entrypoint.sh\n\nEXPOSE 8080 8081\n\nWORKDIR /home/model-server\nENV TEMP=/home/model-server/tmp\nENTRYPOINT [\"/usr/local/bin/dockerd-entrypoint.sh\"]\nCMD [\"serve\"]\n\nLABEL maintainer=\"dantu@amazon.com, rakvas@amazon.com, lufen@amazon.com, dden@amazon.com\"\n" }, { "path": "docker/advanced-dockerfiles/Dockerfile.base.nvidia_cu92_ubuntu_16_04.py2_7.nightly", "content": "FROM nvidia/cuda:9.2-cudnn7-runtime-ubuntu16.04\n\nENV PYTHONUNBUFFERED TRUE\n\nRUN apt-get update && \\\n DEBIAN_FRONTEND=noninteractive apt-get install --no-install-recommends -y \\\n fakeroot \\\n ca-certificates \\\n dpkg-dev \\\n g++ \\\n python-dev \\\n openjdk-8-jdk-headless \\\n curl \\\n vim \\\n && rm -rf /var/lib/apt/lists/* \\\n && cd /tmp \\\n && curl -O https://bootstrap.pypa.io/get-pip.py \\\n && python get-pip.py\n\nRUN update-alternatives --install /usr/bin/python python /usr/bin/python2.7 1\n\nRUN pip install --no-cache-dir multi-model-server --pre\n\nRUN mkdir -p /home/model-server/tmp\n\nCOPY dockerd-entrypoint.sh /usr/local/bin/dockerd-entrypoint.sh\nCOPY config.properties /home/model-server\n\nRUN chmod +x /usr/local/bin/dockerd-entrypoint.sh\n\nEXPOSE 8080 8081\n\nWORKDIR /home/model-server\nENV TEMP=/home/model-server/tmp\nENTRYPOINT [\"/usr/local/bin/dockerd-entrypoint.sh\"]\nCMD [\"serve\"]\n\nLABEL maintainer=\"dantu@amazon.com, rakvas@amazon.com, lufen@amazon.com, dden@amazon.com\"\n" }, { "path": "docker/advanced-dockerfiles/Dockerfile.base.nvidia_cu92_ubuntu_16_04.py3_6", "content": "FROM nvidia/cuda:9.2-cudnn7-runtime-ubuntu16.04\n\nENV PYTHONUNBUFFERED TRUE\n\n# Install python3.6 and pip3\nENV PATH /usr/local/bin:$PATH\nENV LANG C.UTF-8\nRUN apt-get update && apt-get install -y --no-install-recommends \\\n\t\ttk-dev\n\nRUN set -ex \\\n && echo 'Acquire::CompressionTypes::Order:: \"gz\";' > /etc/apt/apt.conf.d/99use-gzip-compression \\\n && apt-get update \\\n && DEBIAN_FRONTEND=noninteractive apt-get install -y --no-install-recommends wget ca-certificates vim\\\n autoconf automake less groff dpkg-dev \\\n bzip2=1.0.* file g++ gcc imagemagick \\\n libbz2-dev libc6-dev curl \\\n libdb-dev libevent-dev libffi-dev \\\n libgeoip-dev libglib2.0-dev libjpeg-dev \\\n libkrb5-dev liblzma-dev libmagickcore-dev \\\n libmagickwand-dev libmysqlclient-dev libncurses5-dev \\\n libpng12-dev libpq-dev libreadline-dev libsqlite3-dev \\\n libssl-dev libtool libwebp-dev libxml2-dev \\\n libxslt1-dev libyaml-dev make patch xz-utils \\\n zlib1g-dev tcl tk \\\n e2fsprogs iptables xfsprogs xz-utils openjdk-8-jdk-headless fakeroot \\\n mono-mcs libcurl4-openssl-dev liberror-perl unzip\n\n# Install python 3.6\nENV GPG_KEY 0D96DF4D4110E5C43FBFB17F2D347EA6AA65421D\nENV PYTHON_VERSION 3.6.8\n\nRUN set -ex \\\n\t\\\n\t&& wget -O python.tar.xz \"https://www.python.org/ftp/python/${PYTHON_VERSION%%[a-z]*}/Python-$PYTHON_VERSION.tar.xz\" \\\n\t&& wget -O python.tar.xz.asc \"https://www.python.org/ftp/python/${PYTHON_VERSION%%[a-z]*}/Python-$PYTHON_VERSION.tar.xz.asc\" \\\n\t&& export GNUPGHOME=\"$(mktemp -d)\" \\\n\t&& gpg --batch --keyserver hkp://p80.pool.sks-keyservers.net:80 --recv-keys \"$GPG_KEY\" \\\n\t&& gpg --batch --verify python.tar.xz.asc python.tar.xz \\\n\t&& { command -v gpgconf > /dev/null && gpgconf --kill all || :; } \\\n\t&& rm -rf \"$GNUPGHOME\" python.tar.xz.asc \\\n\t&& mkdir -p /usr/src/python \\\n\t&& tar -xJC /usr/src/python --strip-components=1 -f python.tar.xz \\\n\t&& rm python.tar.xz \\\n\t&& cd /usr/src/python \\\n\t&& gnuArch=\"$(dpkg-architecture --query DEB_BUILD_GNU_TYPE)\" \\\n\t&& ./configure \\\n\t\t--build=\"$gnuArch\" \\\n\t\t--enable-loadable-sqlite-extensions \\\n\t\t--enable-shared \\\n\t\t--with-system-expat \\\n\t\t--with-system-ffi \\\n\t\t--without-ensurepip \\\n\t&& make -j \"$(nproc)\" \\\n\t&& make install \\\n\t&& ldconfig \\\n\t\\\n\t&& find /usr/local -depth \\\n\t\t\\( \\\n\t\t\t\\( -type d -a \\( -name test -o -name tests \\) \\) \\\n\t\t\t-o \\\n\t\t\t\\( -type f -a \\( -name '*.pyc' -o -name '*.pyo' \\) \\) \\\n\t\t\\) -exec rm -rf '{}' + \\\n\t&& rm -rf /usr/src/python \\\n\t\\\n\t&& python3 --version\n\n# make some useful symlinks that are expected to exist\nRUN cd /usr/local/bin \\\n\t&& ln -s idle3 idle \\\n\t&& ln -s pydoc3 pydoc \\\n\t&& ln -s python3 python \\\n\t&& ln -s python3-config python-config\n\n# if this is called \"PIP_VERSION\", pip explodes with \"ValueError: invalid truth value ''\"\nENV PYTHON_PIP_VERSION 19.0.3\n\nRUN set -ex; \\\n\t\\\n\twget -O get-pip.py 'https://bootstrap.pypa.io/get-pip.py'; \\\n\t\\\n\tpython get-pip.py \\\n\t\t--disable-pip-version-check \\\n\t\t--no-cache-dir \\\n\t\t\"pip==$PYTHON_PIP_VERSION\" \\\n\t; \\\n\tpip --version; \\\n\t\\\n\tfind /usr/local -depth \\\n\t\t\\( \\\n\t\t\t\\( -type d -a \\( -name test -o -name tests \\) \\) \\\n\t\t\t-o \\\n\t\t\t\\( -type f -a \\( -name '*.pyc' -o -name '*.pyo' \\) \\) \\\n\t\t\\) -exec rm -rf '{}' +; \\\n\trm -f get-pip.py\n\nRUN pip install --no-cache-dir multi-model-server\n\nRUN mkdir -p /home/model-server/tmp\n\nCOPY dockerd-entrypoint.sh /usr/local/bin/dockerd-entrypoint.sh\nCOPY config.properties /home/model-server\n\nRUN chmod +x /usr/local/bin/dockerd-entrypoint.sh\n\nEXPOSE 8080 8081\n\nWORKDIR /home/model-server\nENV TEMP=/home/model-server/tmp\nENTRYPOINT [\"/usr/local/bin/dockerd-entrypoint.sh\"]\nCMD [\"serve\"]\n\nLABEL maintainer=\"dantu@amazon.com, rakvas@amazon.com, lufen@amazon.com, dden@amazon.com\"" }, { "path": "docker/advanced-dockerfiles/Dockerfile.base.nvidia_cu92_ubuntu_16_04.py3_6.nightly", "content": "FROM nvidia/cuda:9.2-cudnn7-runtime-ubuntu16.04\n\nENV PYTHONUNBUFFERED TRUE\n\n# Install python3.6 and pip3\nENV PATH /usr/local/bin:$PATH\nENV LANG C.UTF-8\nRUN apt-get update && apt-get install -y --no-install-recommends \\\n\t\ttk-dev\n\nRUN set -ex \\\n && echo 'Acquire::CompressionTypes::Order:: \"gz\";' > /etc/apt/apt.conf.d/99use-gzip-compression \\\n && apt-get update \\\n && DEBIAN_FRONTEND=noninteractive apt-get install -y --no-install-recommends wget ca-certificates vim\\\n autoconf automake less groff dpkg-dev \\\n bzip2=1.0.* file g++ gcc imagemagick \\\n libbz2-dev libc6-dev curl \\\n libdb-dev libevent-dev libffi-dev \\\n libgeoip-dev libglib2.0-dev libjpeg-dev \\\n libkrb5-dev liblzma-dev libmagickcore-dev \\\n libmagickwand-dev libmysqlclient-dev libncurses5-dev \\\n libpng12-dev libpq-dev libreadline-dev libsqlite3-dev \\\n libssl-dev libtool libwebp-dev libxml2-dev \\\n libxslt1-dev libyaml-dev make patch xz-utils \\\n zlib1g-dev tcl tk \\\n e2fsprogs iptables xfsprogs xz-utils openjdk-8-jdk-headless fakeroot \\\n mono-mcs libcurl4-openssl-dev liberror-perl unzip\n\n# Install python 3.6\nENV GPG_KEY 0D96DF4D4110E5C43FBFB17F2D347EA6AA65421D\nENV PYTHON_VERSION 3.6.8\n\nRUN set -ex \\\n\t\\\n\t&& wget -O python.tar.xz \"https://www.python.org/ftp/python/${PYTHON_VERSION%%[a-z]*}/Python-$PYTHON_VERSION.tar.xz\" \\\n\t&& wget -O python.tar.xz.asc \"https://www.python.org/ftp/python/${PYTHON_VERSION%%[a-z]*}/Python-$PYTHON_VERSION.tar.xz.asc\" \\\n\t&& export GNUPGHOME=\"$(mktemp -d)\" \\\n\t&& gpg --batch --keyserver hkp://p80.pool.sks-keyservers.net:80 --recv-keys \"$GPG_KEY\" \\\n\t&& gpg --batch --verify python.tar.xz.asc python.tar.xz \\\n\t&& { command -v gpgconf > /dev/null && gpgconf --kill all || :; } \\\n\t&& rm -rf \"$GNUPGHOME\" python.tar.xz.asc \\\n\t&& mkdir -p /usr/src/python \\\n\t&& tar -xJC /usr/src/python --strip-components=1 -f python.tar.xz \\\n\t&& rm python.tar.xz \\\n\t&& cd /usr/src/python \\\n\t&& gnuArch=\"$(dpkg-architecture --query DEB_BUILD_GNU_TYPE)\" \\\n\t&& ./configure \\\n\t\t--build=\"$gnuArch\" \\\n\t\t--enable-loadable-sqlite-extensions \\\n\t\t--enable-shared \\\n\t\t--with-system-expat \\\n\t\t--with-system-ffi \\\n\t\t--without-ensurepip \\\n\t&& make -j \"$(nproc)\" \\\n\t&& make install \\\n\t&& ldconfig \\\n\t\\\n\t&& find /usr/local -depth \\\n\t\t\\( \\\n\t\t\t\\( -type d -a \\( -name test -o -name tests \\) \\) \\\n\t\t\t-o \\\n\t\t\t\\( -type f -a \\( -name '*.pyc' -o -name '*.pyo' \\) \\) \\\n\t\t\\) -exec rm -rf '{}' + \\\n\t&& rm -rf /usr/src/python \\\n\t\\\n\t&& python3 --version\n\n# make some useful symlinks that are expected to exist\nRUN cd /usr/local/bin \\\n\t&& ln -s idle3 idle \\\n\t&& ln -s pydoc3 pydoc \\\n\t&& ln -s python3 python \\\n\t&& ln -s python3-config python-config\n\n# if this is called \"PIP_VERSION\", pip explodes with \"ValueError: invalid truth value ''\"\nENV PYTHON_PIP_VERSION 19.0.3\n\nRUN set -ex; \\\n\t\\\n\twget -O get-pip.py 'https://bootstrap.pypa.io/get-pip.py'; \\\n\t\\\n\tpython get-pip.py \\\n\t\t--disable-pip-version-check \\\n\t\t--no-cache-dir \\\n\t\t\"pip==$PYTHON_PIP_VERSION\" \\\n\t; \\\n\tpip --version; \\\n\t\\\n\tfind /usr/local -depth \\\n\t\t\\( \\\n\t\t\t\\( -type d -a \\( -name test -o -name tests \\) \\) \\\n\t\t\t-o \\\n\t\t\t\\( -type f -a \\( -name '*.pyc' -o -name '*.pyo' \\) \\) \\\n\t\t\\) -exec rm -rf '{}' +; \\\n\trm -f get-pip.py\n\nRUN pip install --no-cache-dir multi-model-server --pre\n\nRUN mkdir -p /home/model-server/tmp\n\nCOPY dockerd-entrypoint.sh /usr/local/bin/dockerd-entrypoint.sh\nCOPY config.properties /home/model-server\n\nRUN chmod +x /usr/local/bin/dockerd-entrypoint.sh\n\nEXPOSE 8080 8081\n\nWORKDIR /home/model-server\nENV TEMP=/home/model-server/tmp\nENTRYPOINT [\"/usr/local/bin/dockerd-entrypoint.sh\"]\nCMD [\"serve\"]\n\nLABEL maintainer=\"dantu@amazon.com, rakvas@amazon.com, lufen@amazon.com, dden@amazon.com\"" }, { "path": "docker/advanced-dockerfiles/Dockerfile.base.ubuntu_16_04.py2_7", "content": "FROM ubuntu:16.04\n\nENV PYTHONUNBUFFERED TRUE\n\nRUN apt-get update && \\\n DEBIAN_FRONTEND=noninteractive apt-get install --no-install-recommends -y \\\n fakeroot \\\n ca-certificates \\\n dpkg-dev \\\n g++ \\\n python-dev \\\n openjdk-8-jdk-headless \\\n curl \\\n vim \\\n && rm -rf /var/lib/apt/lists/* \\\n && cd /tmp \\\n && curl -O https://bootstrap.pypa.io/get-pip.py \\\n && python get-pip.py\n\nRUN update-alternatives --install /usr/bin/python python /usr/bin/python2.7 1\n\nRUN pip install --no-cache-dir multi-model-server\n\nRUN mkdir -p /home/model-server/tmp\n\nCOPY dockerd-entrypoint.sh /usr/local/bin/dockerd-entrypoint.sh\nCOPY config.properties /home/model-server\n\nRUN chmod +x /usr/local/bin/dockerd-entrypoint.sh\n\nEXPOSE 8080 8081\n\nWORKDIR /home/model-server\nENV TEMP=/home/model-server/tmp\nENTRYPOINT [\"/usr/local/bin/dockerd-entrypoint.sh\"]\nCMD [\"serve\"]\n\nLABEL maintainer=\"dantu@amazon.com, rakvas@amazon.com, lufen@amazon.com, dden@amazon.com\"\n" }, { "path": "docker/advanced-dockerfiles/Dockerfile.base.ubuntu_16_04.py2_7.nightly", "content": "FROM ubuntu:16.04\n\nENV PYTHONUNBUFFERED TRUE\n\nRUN apt-get update && \\\n DEBIAN_FRONTEND=noninteractive apt-get install --no-install-recommends -y \\\n fakeroot \\\n ca-certificates \\\n dpkg-dev \\\n g++ \\\n python-dev \\\n openjdk-8-jdk-headless \\\n curl \\\n vim \\\n && rm -rf /var/lib/apt/lists/* \\\n && cd /tmp \\\n && curl -O https://bootstrap.pypa.io/get-pip.py \\\n && python get-pip.py\n\nRUN update-alternatives --install /usr/bin/python python /usr/bin/python2.7 1\n\nRUN pip install --no-cache-dir multi-model-server --pre\n\nRUN mkdir -p /home/model-server/tmp\n\nCOPY dockerd-entrypoint.sh /usr/local/bin/dockerd-entrypoint.sh\nCOPY config.properties /home/model-server\n\nRUN chmod +x /usr/local/bin/dockerd-entrypoint.sh\n\nEXPOSE 8080 8081\n\nWORKDIR /home/model-server\nENV TEMP=/home/model-server/tmp\nENTRYPOINT [\"/usr/local/bin/dockerd-entrypoint.sh\"]\nCMD [\"serve\"]\n\nLABEL maintainer=\"dantu@amazon.com, rakvas@amazon.com, lufen@amazon.com, dden@amazon.com\"\n" }, { "path": "docker/advanced-dockerfiles/Dockerfile.base.ubuntu_16_04.py3_6", "content": "FROM ubuntu:16.04\n\nENV PYTHONUNBUFFERED TRUE\n\n# Install python3.6 and pip3\nENV PATH /usr/local/bin:$PATH\nENV LANG C.UTF-8\nRUN apt-get update && apt-get install -y --no-install-recommends \\\n\t\ttk-dev\n\nRUN set -ex \\\n && echo 'Acquire::CompressionTypes::Order:: \"gz\";' > /etc/apt/apt.conf.d/99use-gzip-compression \\\n && apt-get update \\\n && DEBIAN_FRONTEND=noninteractive apt-get install -y --no-install-recommends wget ca-certificates vim\\\n autoconf automake less groff dpkg-dev \\\n bzip2=1.0.* file g++ gcc imagemagick \\\n libbz2-dev libc6-dev curl \\\n libdb-dev libevent-dev libffi-dev \\\n libgeoip-dev libglib2.0-dev libjpeg-dev \\\n libkrb5-dev liblzma-dev libmagickcore-dev \\\n libmagickwand-dev libmysqlclient-dev libncurses5-dev \\\n libpng12-dev libpq-dev libreadline-dev libsqlite3-dev \\\n libssl-dev libtool libwebp-dev libxml2-dev \\\n libxslt1-dev libyaml-dev make patch xz-utils \\\n zlib1g-dev tcl tk \\\n e2fsprogs iptables xfsprogs xz-utils openjdk-8-jdk-headless fakeroot \\\n mono-mcs libcurl4-openssl-dev liberror-perl unzip\n\n# Install python 3.6\nENV GPG_KEY 0D96DF4D4110E5C43FBFB17F2D347EA6AA65421D\nENV PYTHON_VERSION 3.6.8\n\nRUN set -ex \\\n\t\\\n\t&& wget -O python.tar.xz \"https://www.python.org/ftp/python/${PYTHON_VERSION%%[a-z]*}/Python-$PYTHON_VERSION.tar.xz\" \\\n\t&& wget -O python.tar.xz.asc \"https://www.python.org/ftp/python/${PYTHON_VERSION%%[a-z]*}/Python-$PYTHON_VERSION.tar.xz.asc\" \\\n\t&& export GNUPGHOME=\"$(mktemp -d)\" \\\n\t&& gpg --batch --keyserver hkp://p80.pool.sks-keyservers.net:80 --recv-keys \"$GPG_KEY\" \\\n\t&& gpg --batch --verify python.tar.xz.asc python.tar.xz \\\n\t&& { command -v gpgconf > /dev/null && gpgconf --kill all || :; } \\\n\t&& rm -rf \"$GNUPGHOME\" python.tar.xz.asc \\\n\t&& mkdir -p /usr/src/python \\\n\t&& tar -xJC /usr/src/python --strip-components=1 -f python.tar.xz \\\n\t&& rm python.tar.xz \\\n\t&& cd /usr/src/python \\\n\t&& gnuArch=\"$(dpkg-architecture --query DEB_BUILD_GNU_TYPE)\" \\\n\t&& ./configure \\\n\t\t--build=\"$gnuArch\" \\\n\t\t--enable-loadable-sqlite-extensions \\\n\t\t--enable-shared \\\n\t\t--with-system-expat \\\n\t\t--with-system-ffi \\\n\t\t--without-ensurepip \\\n\t&& make -j \"$(nproc)\" \\\n\t&& make install \\\n\t&& ldconfig \\\n\t\\\n\t&& find /usr/local -depth \\\n\t\t\\( \\\n\t\t\t\\( -type d -a \\( -name test -o -name tests \\) \\) \\\n\t\t\t-o \\\n\t\t\t\\( -type f -a \\( -name '*.pyc' -o -name '*.pyo' \\) \\) \\\n\t\t\\) -exec rm -rf '{}' + \\\n\t&& rm -rf /usr/src/python \\\n\t\\\n\t&& python3 --version\n\n# make some useful symlinks that are expected to exist\nRUN cd /usr/local/bin \\\n\t&& ln -s idle3 idle \\\n\t&& ln -s pydoc3 pydoc \\\n\t&& ln -s python3 python \\\n\t&& ln -s python3-config python-config\n\n# if this is called \"PIP_VERSION\", pip explodes with \"ValueError: invalid truth value ''\"\nENV PYTHON_PIP_VERSION 19.0.3\n\nRUN set -ex; \\\n\t\\\n\twget -O get-pip.py 'https://bootstrap.pypa.io/get-pip.py'; \\\n\t\\\n\tpython get-pip.py \\\n\t\t--disable-pip-version-check \\\n\t\t--no-cache-dir \\\n\t\t\"pip==$PYTHON_PIP_VERSION\" \\\n\t; \\\n\tpip --version; \\\n\t\\\n\tfind /usr/local -depth \\\n\t\t\\( \\\n\t\t\t\\( -type d -a \\( -name test -o -name tests \\) \\) \\\n\t\t\t-o \\\n\t\t\t\\( -type f -a \\( -name '*.pyc' -o -name '*.pyo' \\) \\) \\\n\t\t\\) -exec rm -rf '{}' +; \\\n\trm -f get-pip.py\n\nRUN pip install --no-cache-dir multi-model-server\n\nRUN mkdir -p /home/model-server/tmp\n\nCOPY dockerd-entrypoint.sh /usr/local/bin/dockerd-entrypoint.sh\nCOPY config.properties /home/model-server\n\nRUN chmod +x /usr/local/bin/dockerd-entrypoint.sh\n\nEXPOSE 8080 8081\n\nWORKDIR /home/model-server\nENV TEMP=/home/model-server/tmp\nENTRYPOINT [\"/usr/local/bin/dockerd-entrypoint.sh\"]\nCMD [\"serve\"]\n\nLABEL maintainer=\"dantu@amazon.com, rakvas@amazon.com, lufen@amazon.com, dden@amazon.com\"" }, { "path": "docker/advanced-dockerfiles/Dockerfile.base.ubuntu_16_04.py3_6.nightly", "content": "FROM ubuntu:16.04\n\nENV PYTHONUNBUFFERED TRUE\n\n# Install python3.6 and pip3\nENV PATH /usr/local/bin:$PATH\nENV LANG C.UTF-8\nRUN apt-get update && apt-get install -y --no-install-recommends \\\n\t\ttk-dev\n\nRUN set -ex \\\n && echo 'Acquire::CompressionTypes::Order:: \"gz\";' > /etc/apt/apt.conf.d/99use-gzip-compression \\\n && apt-get update \\\n && DEBIAN_FRONTEND=noninteractive apt-get install -y --no-install-recommends wget ca-certificates vim\\\n autoconf automake less groff dpkg-dev \\\n bzip2=1.0.* file g++ gcc imagemagick \\\n libbz2-dev libc6-dev curl \\\n libdb-dev libevent-dev libffi-dev \\\n libgeoip-dev libglib2.0-dev libjpeg-dev \\\n libkrb5-dev liblzma-dev libmagickcore-dev \\\n libmagickwand-dev libmysqlclient-dev libncurses5-dev \\\n libpng12-dev libpq-dev libreadline-dev libsqlite3-dev \\\n libssl-dev libtool libwebp-dev libxml2-dev \\\n libxslt1-dev libyaml-dev make patch xz-utils \\\n zlib1g-dev tcl tk \\\n e2fsprogs iptables xfsprogs xz-utils openjdk-8-jdk-headless fakeroot \\\n mono-mcs libcurl4-openssl-dev liberror-perl unzip\n\n# Install python 3.6\nENV GPG_KEY 0D96DF4D4110E5C43FBFB17F2D347EA6AA65421D\nENV PYTHON_VERSION 3.6.8\n\nRUN set -ex \\\n\t\\\n\t&& wget -O python.tar.xz \"https://www.python.org/ftp/python/${PYTHON_VERSION%%[a-z]*}/Python-$PYTHON_VERSION.tar.xz\" \\\n\t&& wget -O python.tar.xz.asc \"https://www.python.org/ftp/python/${PYTHON_VERSION%%[a-z]*}/Python-$PYTHON_VERSION.tar.xz.asc\" \\\n\t&& export GNUPGHOME=\"$(mktemp -d)\" \\\n\t&& gpg --batch --keyserver hkp://p80.pool.sks-keyservers.net:80 --recv-keys \"$GPG_KEY\" \\\n\t&& gpg --batch --verify python.tar.xz.asc python.tar.xz \\\n\t&& { command -v gpgconf > /dev/null && gpgconf --kill all || :; } \\\n\t&& rm -rf \"$GNUPGHOME\" python.tar.xz.asc \\\n\t&& mkdir -p /usr/src/python \\\n\t&& tar -xJC /usr/src/python --strip-components=1 -f python.tar.xz \\\n\t&& rm python.tar.xz \\\n\t&& cd /usr/src/python \\\n\t&& gnuArch=\"$(dpkg-architecture --query DEB_BUILD_GNU_TYPE)\" \\\n\t&& ./configure \\\n\t\t--build=\"$gnuArch\" \\\n\t\t--enable-loadable-sqlite-extensions \\\n\t\t--enable-shared \\\n\t\t--with-system-expat \\\n\t\t--with-system-ffi \\\n\t\t--without-ensurepip \\\n\t&& make -j \"$(nproc)\" \\\n\t&& make install \\\n\t&& ldconfig \\\n\t\\\n\t&& find /usr/local -depth \\\n\t\t\\( \\\n\t\t\t\\( -type d -a \\( -name test -o -name tests \\) \\) \\\n\t\t\t-o \\\n\t\t\t\\( -type f -a \\( -name '*.pyc' -o -name '*.pyo' \\) \\) \\\n\t\t\\) -exec rm -rf '{}' + \\\n\t&& rm -rf /usr/src/python \\\n\t\\\n\t&& python3 --version\n\n# make some useful symlinks that are expected to exist\nRUN cd /usr/local/bin \\\n\t&& ln -s idle3 idle \\\n\t&& ln -s pydoc3 pydoc \\\n\t&& ln -s python3 python \\\n\t&& ln -s python3-config python-config\n\n# if this is called \"PIP_VERSION\", pip explodes with \"ValueError: invalid truth value ''\"\nENV PYTHON_PIP_VERSION 19.0.3\n\nRUN set -ex; \\\n\t\\\n\twget -O get-pip.py 'https://bootstrap.pypa.io/get-pip.py'; \\\n\t\\\n\tpython get-pip.py \\\n\t\t--disable-pip-version-check \\\n\t\t--no-cache-dir \\\n\t\t\"pip==$PYTHON_PIP_VERSION\" \\\n\t; \\\n\tpip --version; \\\n\t\\\n\tfind /usr/local -depth \\\n\t\t\\( \\\n\t\t\t\\( -type d -a \\( -name test -o -name tests \\) \\) \\\n\t\t\t-o \\\n\t\t\t\\( -type f -a \\( -name '*.pyc' -o -name '*.pyo' \\) \\) \\\n\t\t\\) -exec rm -rf '{}' +; \\\n\trm -f get-pip.py\n\nRUN pip install --no-cache-dir multi-model-server --pre\n\nRUN mkdir -p /home/model-server/tmp\n\nCOPY dockerd-entrypoint.sh /usr/local/bin/dockerd-entrypoint.sh\nCOPY config.properties /home/model-server\n\nRUN chmod +x /usr/local/bin/dockerd-entrypoint.sh\n\nEXPOSE 8080 8081\n\nWORKDIR /home/model-server\nENV TEMP=/home/model-server/tmp\nENTRYPOINT [\"/usr/local/bin/dockerd-entrypoint.sh\"]\nCMD [\"serve\"]\n\nLABEL maintainer=\"dantu@amazon.com, rakvas@amazon.com, lufen@amazon.com, dden@amazon.com\"" }, { "path": "docker/advanced-dockerfiles/config.properties", "content": "# vmargs=-Xmx128m -XX:-UseLargePages -XX:+UseG1GC -XX:MaxMetaspaceSize=32M -XX:MaxDirectMemorySize=10m -XX:+ExitOnOutOfMemoryError\nmodel_store=/opt/ml/model\nload_models=ALL\ninference_address=http://0.0.0.0:8080\nmanagement_address=http://0.0.0.0:8081\n# management_address=unix:/tmp/management.sock\n# number_of_netty_threads=0\n# netty_client_threads=0\n# default_response_timeout=120\n# unregister_model_timeout=120\n# default_workers_per_model=0\n# job_queue_size=100\n# async_logging=false\n# number_of_gpu=1\n# cors_allowed_origin\n# cors_allowed_methods\n# cors_allowed_headers\n# keystore=src/test/resources/keystore.p12\n# keystore_pass=changeit\n# keystore_type=PKCS12\n# private_key_file=src/test/resources/key.pem\n# certificate_file=src/test/resources/certs.pem\n# max_response_size=6553500\n# max_request_size=6553500\n# blacklist_env_vars=\ndecode_input_request=false\n# enable_envvars_config=false" }, { "path": "docker/advanced-dockerfiles/dockerd-entrypoint.sh", "content": "#!/bin/bash\nset -e\n\nif [[ \"$1\" = \"serve\" ]]; then\n shift 1\n multi-model-server --start --mms-config config.properties\nelse\n eval \"$@\"\nfi\n\n# prevent docker exit\ntail -f /dev/null\n" }, { "path": "docker/advanced_settings.md", "content": "# Advanced Settings\n\n## Contents of this Document\n* [GPU Inference](advanced_settings.md#gpu-inference)\n* [Reference Commands](advanced_settings.md#reference-commands)\n* [Docker Details](advanced_settings.md#docker-details)\n* [Description of Config File Settings](advanced_settings.md#description-of-config-file-settings)\n* [Configuring SSL](advanced_settings.md#configuring-ssl)\n\n\n## Other Relevant Documents\n* [Quickstart](README.md#quickstart)\n* [Configuring MMS with Docker](README.md#configuring-mms-with-docker)\n\n\n\n## GPU Inference\n\n**Step 1: Install nvidia-docker.**\n\n`nvidia-docker` is NVIDIA's customized version of Docker that makes accessing your host's GPU resources from Docker a seamless experience. All of your regular Docker commands work the same way.\n\nFollow the [instructions for installing nvidia-docker](https://github.com/NVIDIA/nvidia-docker#quickstart). Return here and follow the next step when the installation completes.\n\n**Step 2: Download the GPU configuration template.**\n\nA GPU configuration template is provided for your use.\nDownload the template a GPU config and place it in the `/tmp/models` folder you just created:\n* [config.properties](config.properties)\n\n**Step 3: Modify the configuration template.**\n\nEdit the file you downloaded, `config.properties` to configure the model-server.\n\nSave the file.\n\n**Step 4: Run MMS with Docker using a shared volume.**\n\nWhen you run the following command, the `-v` argument and path values of `/tmp/models/:/models` will map the `models` folder you created (assuming it was in ) with a folder inside the Docker container. MMS will then be able to use the local model file.\n\n```bash\nnvidia-docker run -itd --name mms -p 80:8080 -p 8081:8081 -v /tmp/models/:/models awsdeeplearningteam/multi-model-server:latest-gpu multi-model-server --start --mms-config /models/config.properties --models squeezenet=https://s3.amazonaws.com/model-server/models/squeezenet_v1.1/squeezenet_v1.1.model\n```\n\n**Step 5: Test inference.**\n\nThis configuration file is using the default Squeezenet model, so you will request the `predictions/squeezenet` API endpoint.\n\n```bash\ncurl -X POST http://127.0.0.1/predictions/squeezenet -F \"data=@kitten.jpg\"\n```\n\nGiven that this is a different model, the same image yields a different inference result which will be something similar to the following:\n\n```\n{\n \"prediction\": [\n [\n {\n \"class\": \"n02123159 tiger cat\",\n \"probability\": 0.3630334138870239\n },\n...\n```\n\n## Reference Commands\n\nManually pull the MMS Docker CPU image:\n```bash\ndocker pull awsdeeplearningteam/multi-model-server\n```\n\nManually pull the MMS Docker GPU image:\n```bash\ndocker pull awsdeeplearningteam/multi-model-server:latest-gpu\n```\n\nList your Docker images:\n```bash\ndocker images\n```\n\nVerify the Docker container is running:\n```bash\ndocker ps -a\n```\n\nStop the Docker container from running:\n```bash\ndocker rm -f mms\n```\n\nDelete the MMS Docker GPU image:\n```bash\ndocker rmi awsdeeplearningteam/multi-model-server:latest-gpu\n```\n\nDelete the MMS Docker GPU image:\n```bash\ndocker rmi awsdeeplearningteam/multi-model-server:latest\n```\n\nOutput the recent logs to console.\n```bash\ndocker logs mms\n```\n\nInteract with the container. This will open a shell prompt inside the container. Use `$ Ctrl-p-Ctrl-q` to detach again.\n```bash\ndocker attach mms\n```\n\nRun the MMS Docker image without starting the Model Server:\n```bash\ndocker run -itd --name mms -p 80:8080 -p 8081:8081 awsdeeplearningteam/multi-model-server /bin/bash\n```\n\nStart MMS in the Docker container (CPU config):\n```bash\ndocker exec mms multi-model-server --start --mms-config /home/model-server/config.properties\n```\n\nStart MMS in the Docker container using nvidia-docker command as follows. :\n```bash\nnvidia-docker exec multi-model-server --start --mms-config /home/model-server/config.properties\n```\n\n**Note**: To use GPU configuration, modify the config.properties to reflect that the model-server should use GPUs.\n\n```properties\ninference_address=http://0.0.0.0:8080\nmanagement_address=http://0.0.0.0:8081\n...\nnumber_of_gpu=8\n...\n```\n\nStop MMS.\n```bash\ndocker exec mms multi-model-server --stop\n```\n\nGet MMS help.\n```bash\ndocker exec mms multi-model-server --help\n```\n\nRefer [Docker CLI](https://docs.docker.com/engine/reference/commandline/run/) to understand each parameter.\n\n\n## Docker Details\n\n### Docker Hub\n\nDocker images are available on [Docker Hub](https://hub.docker.com/r/awsdeeplearningteam):\n* [CPU](https://hub.docker.com/r/awsdeeplearningteam/multi-model-server/tags)\n* [GPU](https://hub.docker.com/r/awsdeeplearningteam/multi-model-server/tags)\n### Building a MMS Docker Image from Scratch\nThe following are the steps to build a container image from scratch.\n\n#### Prerequisites\nIn order to build the Docker image yourself you need the following:\n\n* Install Docker\n* Clone the MMS repo\n\n#### Docker Installation\n\nFor macOS, you have the option of [Docker's Mac installer](https://docs.docker.com/docker-for-mac/install/) or you can simply use `brew`:\n\n```bash\nbrew install docker\n```\n\nFor Windows, you should use [their Windows installer](https://docs.docker.com/docker-for-windows/install/).\n\nFor Linux, check your favorite package manager if brew is available, otherwise use their installation instructions for [Ubuntu](https://docs.docker.com/engine/installation/linux/ubuntu/) or [CentOS](https://docs.docker.com/engine/installation/linux/centos/).\n\n#### Verify Docker\n\nWhen you've competed the installation, verify that Docker is running by running `docker images` in your terminal. If this works, you are ready to continue.\n\n#### Clone the MMS Repo\n\nIf you haven't already, clone the MMS repo and go into the `docker` folder.\n\n```bash\ngit clone https://github.com/awslabs/multi-model-server.git && cd multi-model-server/docker\n```\n\n### Building the Container Image\n\n#### Configuring the Docker Build for Use on EC2\n\nNow you can examine how to build a Docker image with MMS and establish a public accessible endpoint on EC2 instance. You should be able to adapt this information for any cloud provider. This Docker image can be used in other production environments as well. Skip this section if you're building for local use.\n\nThe first step is to create an [EC2 instance](https://aws.amazon.com/ec2/).\n\n### Build Step for CPU container image\n\nThere are separate `Dockerfile` configuration files for CPU and GPU. They are named `Dockerfile.cpu` and `Dockerfile.gpu` respectively.\n\nThe container image consists of MXNet, Java, MMS and all related python libraries.\n\nWe can build the Multi Model Server image based on the Dockerfile as follows: \n\n```bash\n# Building the MMS CPU image\ndocker build -f Dockerfile.cpu -t mms_image .\n```\n\nOnce this completes, run `docker images` from your terminal. You should see the Docker image listed with the tag, `mms_image:latest`.\n\n### Build Step for GPU\n\nIf your host machine has at least one GPU installed, you can use a GPU Docker image to benefit from improved inference performance.\n\nYou need to install [nvidia-docker plugin](https://github.com/NVIDIA/nvidia-docker) before you can use a NVIDIA GPU with Docker.\n\nOnce you install `nvidia-docker`, run following commands (for info modifying the tag, see the CPU section above):\n\n```bash\n# Building the MMS GPU image\ndocker build -f Dockerfile.gpu -t mms_image_gpu .\n```\n\n#### Running the MMS GPU Docker\n\n```bash\nnvidia-docker run -itd -p 80:8080 8081:8081 --name mms -v /home/user/models/:/models mms_image_gpu /bin/bash\n```\n\nThis command starts the Docker instance in a detached mode and mounts `/home/user/models` of the host system into `/models` directory inside the Docker instance.\nConsidering that you modified and copied `config.properties` file into the models directory, before you ran the \nabove `nvidia-docker` command, you would have this configuration file ready to use in the Docker instance.\n\n```bash\nnvidia-docker exec mms multi-model-server --start --mms-config /models/config.properties\n```\n\n### Testing the MMS Docker\n\nNow you can send a request to your server's [api-description endpoint](http://localhost/api-description) to see the \nlist of MMS endpoints or [ping endpoint](http://localhost/ping) to check the health status of the MMS API. \nRemember to add the port if you used a custom one or the IP or DNS of your server if you configured it for that instead \nof localhost. Here are some handy test links for common configurations:\n\n* [http://localhost/api-description](http://localhost/api-description)\n* [http://localhost/ping](http://localhost/ping)\n\nIf `config.properties` file is used as is, the following commands can be run to verify that the Multi Model Server is running.\n\n```bash\ncurl -O https://s3.amazonaws.com/model-server/inputs/kitten.jpg\ncurl -X POST http://127.0.0.1/squeezenet/predict -F \"data=@kitten.jpg\"\n```\n\nThe predict endpoint will return a prediction response in JSON. It will look something like the following result:\n\n```json\n{\n \"prediction\": [\n [\n {\n \"class\": \"n02124075 Egyptian cat\",\n \"probability\": 0.9408261179924011\n },\n {\n \"class\": \"n02127052 lynx, catamount\",\n \"probability\": 0.055966004729270935\n },\n {\n \"class\": \"n02123045 tabby, tabby cat\",\n \"probability\": 0.0025502564385533333\n },\n {\n \"class\": \"n02123159 tiger cat\",\n \"probability\": 0.00034320182749070227\n },\n {\n \"class\": \"n02123394 Persian cat\",\n \"probability\": 0.00026897044153884053\n }\n ]\n ]\n}\n```\n\n## Description of Config File Settings\n\n**For config.properties:**\n\nThe system settings are stored in [config.properties](config.properties). You can modify these settings to use different models, or to apply other customized settings. \n\nNotes on a couple of the parameters:\n\n* **model_store** - The directory on the local host where models must reside for serving.\n* **management_address** - The address:port value on which the model server would serve control plane APIs such as \"GET\", \"PUT\", \"DELETE\" of \"models\"\n* **inference_address** - The address:port value on which the model server would serve data plane APIs such as predictions, ping and api-description\n* **load_models** - List of all the models in the `model_store` which should be loaded on startup\n* **number_of_netty_threads** - Number of threads present to handle the incoming requests. \n* **max_workers** - \n* **job_queue_size** - Number of requests that can be queued. This queue is shared across models.\n* **number_of_gpu** - Number of GPUs available for model server when serving inferences on GPU hosts\n* **keystore** - SSL Key Store\n* **keystore_pass** - SSL password\n* **keystore_type** - Store of cryptographic keys and certificates\n* **private_key_file** - Location of the private key file\n* **certificate_file** - Location of the certificate file\n* **max_response_size** - The maximum buffer size the frontend allocates for a worker response, in bytes.\n* **max_request_size** - The maximum allowable request size that the MMS accepts.\n\nin the range of 0 .. (num-gpu-1) in a round-robin fashion. **By default MMS uses all the available GPUs but this parameter can be configured if user want to use only few of them**.\n\n```properties\n# vmargs=-Xmx1g -XX:MaxDirectMemorySize=512m -Dlog4j.configurationFile=file:///opt/ml/conf/log4j2.xml\n# model_store=/opt/ml/model\n# load_models=ALL\n# inference_address=http://0.0.0.0:8080\n# management_address=http://0.0.0.0:8081\n# number_of_netty_threads=0\n# max_workers=0\n# job_queue_size=1000\n# number_of_gpu=1\n# keystore=src/test/resources/keystore.p12\n# keystore_pass=changeit\n# keystore_type=PKCS12\n# private_key_file=src/test/resources/key.pem\n# certificate_file=src/test/resources/certs.pem\n# max_response_size=6553500\n# max_request_size=6553500\n```\n" }, { "path": "docker/config.properties", "content": "vmargs=-Xmx128m -XX:-UseLargePages -XX:+UseG1GC -XX:MaxMetaspaceSize=32M -XX:MaxDirectMemorySize=10m -XX:+ExitOnOutOfMemoryError\nmodel_store=/opt/ml/model\nload_models=ALL\ninference_address=http://0.0.0.0:8080\nmanagement_address=http://0.0.0.0:8081\npreload_model=false\n# management_address=unix:/tmp/management.sock\n# number_of_netty_threads=0\n# netty_client_threads=0\n# default_response_timeout=120\n# unregister_model_timeout=120\n# default_workers_per_model=0\n# job_queue_size=100\n# async_logging=false\n# number_of_gpu=1\n# cors_allowed_origin\n# cors_allowed_methods\n# cors_allowed_headers\n# keystore=src/test/resources/keystore.p12\n# keystore_pass=changeit\n# keystore_type=PKCS12\n# private_key_file=src/test/resources/key.pem\n# certificate_file=src/test/resources/certs.pem\n# max_response_size=6553500\n# max_request_size=6553500\n# blacklist_env_vars=\n# decode_input_request=true\n# enable_envvars_config=false" }, { "path": "docker/dockerd-entrypoint.sh", "content": "#!/bin/bash\nset -e\n\nif [[ \"$1\" = \"serve\" ]]; then\n shift 1\n multi-model-server --start --mms-config config.properties\nelse\n eval \"$@\"\nfi\n\n# prevent docker exit\ntail -f /dev/null\n" }, { "path": "docs/README.md", "content": "# Multi Model Server Documentation\n\n## Basic Features\n* [Serving Quick Start](../README.md#serve-a-model) - Basic server usage tutorial\n* [Model Archive Quick Start](../model-archiver#creating-a-model-archive) - Tutorial that shows you how to package a model archive file.\n* [Installation](install.md) - Installation procedures and troubleshooting\n* [Serving Models](server.md) - Explains how to use `multi-model-server`.\n * [REST API](rest_api.md) - Specification on the API endpoint for MMS\n * [Model Zoo](model_zoo.md) - A collection of MMS model archive (.mar) files that you can use with MMS.\n* [Packaging Model Archive](../model-archiver/README.md) - Explains how to package model archive file, use `model-archiver`.\n* [Docker](../docker/README.md) - How to use MMS with Docker and cloud services\n* [Logging](logging.md) - How to configure logging\n* [Metrics](metrics.md) - How to configure metrics\n\n## Advanced Features\n* [Advanced settings](configuration.md) - Describes advanced MMS configurations.\n* [Custom Model Service](custom_service.md) - Describes how to develop custom inference services.\n* [Unit Tests](../mms/tests/README.md) - Housekeeping unit tests for MMS.\n* [Benchmark](../benchmarks/README.md) - Use JMeter to run MMS through the paces and collect benchmark data.\n* [Model Serving with Amazon Elastic Inference](elastic_inference.md) - Run Model server on Elastic Inference enabled EC2 instances. \n\n## Example Projects\n* [MMS on Fargate, Serverless Inference](mms_on_fargate.md) - The project which illustrates the step-by-step process to launch MMS as a managed inference production service, on ECS Fargate.\n* [MXNet Vision Service](../examples/mxnet_vision/README.md) - An example MMS project for a MXNet Image Classification model. The project takes JPEG image as input for inference.\n* [LSTM](../examples/lstm_ptb/README.md) - An example MMS project for a recurrent neural network (RNN) using long short-term memory (LSTM). The project takes JSON inputs for inference against a model trained with a specific vocabulary.\n* [Object Detection](../examples/ssd/README.md) - An example MMS project that uses a pretrained Single Shot Multi Object Detection (SSD) model that takes image inputs and infers the types and locations of several classes of objects.\n" }, { "path": "docs/batch_inference_with_mms.md", "content": "# Batch Inference with Model Server\n\n## Contents of this Document\n* [Introduction](#introduction)\n* [Batching example](#batch-inference-with-mms-using-resnet-152-model)\n * [Model Handler Code](#model-handler-a.k.a.-entry-point)\n * [Initialization logic](#initialization-logic)\n * [Preprocess logic](#preprocess-logic)\n * [Inference logic](#inference-logic)\n * [Postprocess logic](#postprocess-logic)\n * [MMS Model Configuration](#mms-model-configuration)\n* [Demo](#demo-to-configure-mms-with-batch-supported-model)\n * [Prerequisites](#pre-requisites)\n * [Running MMS with batch inference](#loading-resnet-152-which-handles-batch-inferences)\n* [Performance results](#performance-benchmarking)\n* [Conclusion](#conclusion) \n\n## Introduction\n\nBatching in the Machine-Learning/Deep-Learning is a process of aggregating inference-requests and sending this aggregated requests through the ML/DL framework for inference at once.\nModel Server (MMS) was designed to natively support batching of incoming inference requests. This functionality provides customer using MMS to optimally utilize their host resources, because most ML/DL frameworks\nare optimized for batch requests. This optimal utilization of host resources in turn reduces the operational expense of hosting an inference service using MMS. In this document we will go through an example of how this is done\nand compare the performance of running a batched inference against running single inference.\n\n## Prerequisites:\nBefore jumping into this document, please go over the following docs\n1. [What is MMS?](../README.md)\n1. [What is custom service code?](custom_service.md)\n\n## Batch Inference with MMS using ResNet-152 model\nTo support batching of inference requests, MMS needs the following:\n1. MMS Model Configuration: MMS provides means to configure \"Max Batch Size\" and \"Max Batch Delay\" through \"POST /models\" API. \n MMS needs to know the maximum batch size that the model can handle and the maximum delay that MMS should wait for, to form this request-batch. \n2. Model Handler code: MMS requires the Model Handler to handle the batch of inference requests. \n\nIn this section we will go over the configuration of MMS to handle batching and the actual code changes required at the model level to handle batching. \n\nLets begin with the \"Model Handler Code\" component and see how we can convert an existing Resnet-152 model from the [MMS Model Zoo](https://github.com/awslabs/multi-model-server/blob/master/docs/model_zoo.md#resnet-152)\ninto a model which can process a batch of requests. For a full working code, refer to [mxnet_vision_batching.py](https://github.com/awslabs/multi-model-server/blob/master/examples/model_service_template/mxnet_vision_batching.py)\n\n### Model Handler a.k.a. entry-point\n\nThe handler method is the entrypoint to the model. When MMS receives a request for a model, MMS forwards this request to the `handler` method associated with the model.\nA model's `handler` entry-point is expected to have the following logic:\n1. Loading Model: The handler code should have logic to load the model-artifacts onto the DL/ML Framework, as a part of initialization. In our example, the DL Framework is MXNet\n1. Once initialized, this handler method is given requests for processing. So, the handler is expected to have the logic for: \n 1. Logic for `preprocess`ing request: The handler converts the incoming request to a format understandable by the ML/DL framework. In our example, that is an NDArray.\n 1. Logic for `inference`: The handler should take the preprocessed data and pass it through through the DL Framework for inference.\n 1. Logic for `postprocess`ing: The handler should take the output of the inference-logic and `postprocess` this data. This is then sent back the clients as the final result.\n\nLet's look into how we define the `initialize`, `preprocess`, `inference` and `postprocess` logic in detail.\n\n#### Initialization logic\nAs a part of the handler's parameters, MMS sends a `context` object to the handler. Besides everything else, this `context` object also contains the batch-size that this model was configured to handle. \nThe configuration for the model comes from the `POST /models` call and this is explained in detail in the below section [MMS Model Configuration](#mms-model-configuration). Lets look at a code snippet,\n\n```python\nclass MXNetVisionServiceBatching(object):\n...\n def initialize(self, context):\n ...\n self._batch_size = context.system_properties[\"batch_size\"]\n data_shapes = []\n # Read the data shape\n data_shapes[0] = self._batch_size\n ...\n # Bind the MXNet module with the data shape NCHW where N is the batch size\n self.mx_model.bind(for_training=False, data_shapes=data_shapes)\n ...\n... \n```\nThis initialization logic binds the MXNet module with a batch-size that the model is configured with. For example, if the incoming requests are images with RGB layers and the input to the network is\nor size [3,224, 224], the batch adds an additional dimension to this data. If the batch size configured with `POST /models` API is 8, the input to the DL Framework becomes [8, 3, 224, 224]. \n\n#### Preprocess logic\nOnce a model is initialized, the DL framework (in our case MXNet) expects a `batch_size` number of requests. Hence, the preprocessing logic should make sure that no matter how many requests\ncome into the handler, it always returns the `batch_size` number of NDArray elements to be used in the inference logic. Lets look at a sample `preprocess` logic below \n\n```python\nimport mxnet as mx\n\nclass MXNetVisionServiceBatching(object):\n...\n def initialize(self, context):\n ...\n...\n def preprocess(self, request):\n img_list = []\n param_name = self.signature['inputs'][0]['data_name']\n input_shape = self.signature['inputs'][0]['data_shape']\n # We are assuming input shape is NCHW\n [c, h, w] = input_shape[1:]\n\n for idx, data in enumerate(request):\n img = data.get(param_name)\n if img is None:\n img = data.get(\"body\")\n\n if img is None:\n img = data.get(\"data\")\n\n if img is None or len(img) == 0:\n logging.error(\"Error processing request\")\n self.erroneous_reqs.add(idx)\n continue\n\n try:\n img_arr = mx.image.imdecode(img, 1, True, None)\n except Exception as e:\n logging.error(e, exc_info=True)\n self.erroneous_reqs.add(idx)\n continue\n\n img_arr = mx.image.imresize(img_arr, w, h, 2)\n img_arr = mx.nd.transpose(img_arr, (2, 0, 1))\n self._num_requests = idx + 1\n img_list.append(img_arr)\n \n logging.debug(\"Worker :{} received {} requests\".format(os.getpid(), self._num_requests))\n reqs = mx.nd.stack(*img_list)\n reqs = reqs.as_in_context(self.mxnet_ctx)\n\n if (self._batch_size - self._num_requests) != 0:\n padding = mx.nd.zeros((self._batch_size - self._num_requests, c, h, w), self.mxnet_ctx, 'uint8')\n reqs = mx.nd.concat(reqs, padding, dim=0)\n\n return reqs\n```\n**NOTE: The above code handles the case where the `handler` doesn't receive a `batch_size` number of requests. This is because MMS waits for a `max_batch_delay` amount of time to receive \n`batch_size` number of requests. If the `max_batch_delay` timer times out before receiving `batch_size` number of requests, MMS bundles what ever requests it received\nand sends it to the handler for processing.**\n\n#### Inference logic\nThe inference logic is similar to the inference logic of processing single requests. Since this isn't as interesting, we will skip explaining this in detail. Sample logic is shown below for\ncompleteness of this document.\n\n```python\nimport mxnet as mx\nfrom collections import namedtuple\n\nclass MXNetVisionServiceBatching(object):\n...\n def initialize(self, context):\n ...\n...\n def preprocess(self, request):\n ...\n...\n def inference(self, model_input):\n batch = namedtuple('Batch', ['data'])\n if self.error is not None:\n return None\n\n self.mx_model.forward(batch([model_input]), is_train=False)\n outputs = self.mx_model.get_outputs()\n res = mx.ndarray.split(outputs[0], axis=0, num_outputs=outputs[0].shape[0])\n res = [res] if not isinstance(res, list) else res\n return res\n... \n``` \n\nLets move onto post-processing. \n\n \n#### Postprocess logic\nThe output of inference logic is fed to the post-processing logic. As we saw before, during preprocessing, if there aren't `batch_size` number of requests, preprocessing logic\npads the difference with 0's. These are artificially created requests sent to the module for inference. **In post-process we should ignore these artificially padded requests**. \nLets look at the sample logic\n\n```python\nimport mxnet as mx\nfrom collections import namedtuple\n\nclass MXNetVisionServiceBatching(object):\n...\n def initialize(self, context):\n ...\n...\n def preprocess(self, request):\n ...\n...\n def inference(self, model_input):\n ...\n def postprocess(self, data):\n res = []\n for idx, resp in data[:self._num_requests]:\n if idx not in self.erroneous_reqs:\n res.append(self.top_probability(resp, self.labels, top=5))\n else:\n res.append(\"This request was not processed successfully. Refer to mms.log for additional information\")\n return res\n...\n```\n\nIn the above code, we iterate only until **self._num_requests**. This variable is assigned a value during the preprocessing step. This logic ensures that the postprocess logic is run only\nfor the actual requests coming from external clients.\n\n\n### MMS Model Configuration\nTo configure MMS to use the batching feature, you would have to provide the batch configuration information through [**POST /models** API](https://github.com/awslabs/multi-model-server/blob/master/docs/management_api.md#register-a-model).\nThe configuration that we are interested in is the following: \n1. `batch_size`: This is the maximum batch size that a model is expected to handle. \n2. `max_batch_delay`: This is the maximum batch delay time MMS waits to receive `batch_size` number of requests. If MMS doesn't receive `batch_size` number of requests\nbefore this timer time's out, it sends what ever requests that were received to the model `handler`.\n\nLet's look at an example using this configuration\n```bash\n# The following command will register a model \"resnet-152.mar\" and configure MMS to use a batch_size of 8 and a max batch delay of 50 milli seconds. \ncurl -X POST \"localhost:8081/models?url=resnet-152.mar&batch_size=8&max_batch_delay=50\"\n```\n \nThese configurations are used both in MMS and in the model's custom-service-code (a.k.a the handler code).\n1. MMS: MMS associates the batch related configuration with each model. The frontend then tries to aggregate the batch-size number of requests and send it to the backend.\n2. Model Custom Handler Code: The handler code is given the information about the batch-size. The handler then uses this information to tell the DL framework about the expected batch size.\n\n\n## Demo to configure MMS with batch-supported model\nIn this section lets bring up model server and launch Resnet-152 model, which has been built to handle a batch of request. \n\n### Pre-requisites\nFollow the main [Readme](https://github.com/awslabs/multi-model-server/blob/master/README.md) and install all the required packages including \"multi-model-server\"\n\n### Loading Resnet-152 which handles batch inferences\n* Start the model server. In this example, we are starting the model server to run on inference port 8080 and management port 8081.\n```text\n$ cat config.properties\n...\ninference_address=http://0.0.0.0:8080\nmanagement_address=http://0.0.0.0:8081\n...\n$ multi-model-server --start\n```\n\n* Verify that the MMS is up and running\n```text\n$ curl localhost:8080/ping\n{\n \"status\": \"Healthy\"\n}\n```\n\n* Now lets launch resnet-152 model, which we have built to handle batch inference. Since this is an example, we are going to launch 1 worker which handles a batch size of 8\nwith a max-batch-delay of 10ms. \n```text\n$ curl -X POST \"localhost:8081/models?url=https://s3.amazonaws.com/model-server/model_archive_1.0/examples/resnet-152-batching/resnet-152.mar&batch_size=8&max_batch_delay=10&initial_workers=1\"\n{\n \"status\": \"Processing worker updates...\"\n}\n```\n\n* Verify that the workers were started properly\n```text\n$ curl localhost:8081/models/resnet-152\n{\n \"modelName\": \"resnet-152\",\n \"modelUrl\": \"https://s3.amazonaws.com/model-server/model_archive_1.0/examples/resnet-152-batching/resnet-152.mar\",\n \"runtime\": \"python\",\n \"minWorkers\": 1,\n \"maxWorkers\": 1,\n \"batchSize\": 8,\n \"maxBatchDelay\": 10,\n \"workers\": [\n {\n \"id\": \"9008\",\n \"startTime\": \"2019-02-19T23:56:33.907Z\",\n \"status\": \"READY\",\n \"gpu\": false,\n \"memoryUsage\": 607715328\n }\n ]\n}\n```\n\n* Now let's test this service. \n * Get an image to test this service\n ```text\n $ curl -O https://s3.amazonaws.com/model-server/inputs/kitten.jpg\n ``` \n * Run inference to test the model\n ```text\n $ curl -X POST localhost/predictions/resnet-152 -T kitten.jpg\n {\n \"probability\": 0.7148938179016113,\n \"class\": \"n02123045 tabby, tabby cat\"\n },\n {\n \"probability\": 0.22877725958824158,\n \"class\": \"n02123159 tiger cat\"\n },\n {\n \"probability\": 0.04032370448112488,\n \"class\": \"n02124075 Egyptian cat\"\n },\n {\n \"probability\": 0.00837081391364336,\n \"class\": \"n02127052 lynx, catamount\"\n },\n {\n \"probability\": 0.0006728120497427881,\n \"class\": \"n02129604 tiger, Panthera tigris\"\n }\n ```\n \n* Now that we have the service up and running, we could run performance tests with the same kitten image as follows. There are multiple tools to measure performance of web-servers. We will use \n[apache-bench](https://httpd.apache.org/docs/2.4/programs/ab.html) to run our performance tests. We chose `apache-bench` for our tests because of the ease of installation and ease of running tests.\nBefore running this test, we need to first install `apache-bench` on our System. Since we were running this on a ubuntu host, we installed apache-bench as follows\n```bash\n$ sudo apt-get udpate && sudo apt-get install apache2-utils\n``` \nNow that installation is done, we can run performance benchmark test as follows. \n```text\n$ ab -k -l -n 10000 -c 1000 -T \"image/jpeg\" -p kitten.jpg localhost:8080/predictions/resnet-152\n```\nThe above test simulates MMS receiving 1000 concurrent requests at once and a total of 10,000 requests. All of these requests are directed to the endpoint \"localhost:8080/predictions/resnet-152\", which assumes\nthat resnet-152 is already registered and scaled-up on MMS. We had done this registration and scaling up in the above steps.\n\n## Performance benchmarking\nWe benchmarked MMS with batch-inference enabled Resnet-152 on a *P3.8xlarge* instance, which is a AWS provided GPU EC2 instance. \nWe ran MMS in our [GPU container](https://hub.docker.com/r/awsdeeplearningteam/multi-model-server/tags) which hosted the above resnet-152 model. \nWe ran the tests for batch sizes 1, 8 and 16 and captured the results. \nWe saw a significant gain in throughput and also saw that the GPU resources were utilized more optimally. Attached is the graph showing the throughput gains. The \nexperiment was done with the following configuration. To understand the details of this configuration please refer the [Configuration document](configuration.md)\n\n```bash\n# MMS configuration\n$ cat config.properties\nmodel_store=/opt/ml/model\ninference_address=http://0.0.0.0:8080\nmanagement_address=http://0.0.0.0:8081\nnumber_of_netty_threads=32\njob_queue_size=1000\nasync_logging=true\n```\n\n```bash\n# To load the model run the following command\n$ curl -X POST \"localhost:81/models?url=https://s3.amazonaws.com/model-server/model_archive_1.0/examples/resnet-152-batching/resnet-152.mar&batch_size=8&max_batch_delay=50&initial_workers=8\"\n\n```\nAs seen from the above command, the number of workers was set at 8 (2 per GPU), and `max_batch_delay` was set at 50 ms. \n\n```bash\n# Apache bench performance test command\n$ ab -k -l -n 10000 -c 1000 -T \"image/jpeg\" -p kitten.jpg localhost:8080/predictions/resnet-152\n```\nWe set the `batch_size` in the above `curl` command at 1, 8 and 16 and captured the results of `ab` command by setting the `-c` option, or concurrency, at 10, 50, 100, 500 and 1000 for each of the `batch_size`s.\n\n![Graph](images/throughput.png)\n\nAs we can see from the above diagram, batching is not a one-size-fits-all solution. For example, when the rate of requests received at MMS is lower, look at 10 concurrent clients,\nthe batch size of 1 out performs batch size of 8 or 16. This is because, some requests would wait until the 50ms timeout occured before getting scheduled to be processed \nby the model handlers. Whereas, by definition, batch size of 1 wouldn't wait for this timeout to occur and the model handler is given the request to handle as and when\nthe requests come in.\n\nWe tried to show in an intuitive way as to why we get lower TPS when batch size is 8 or 16 as compared to batch size of 1, when the number of concurrent requests sent to MMS was set at 10.\nFor this, we kept the batch-size at 8, number of backend workers at 8 and kept the number of concurrent requests coming in at 10 and varied the `max_batch_delay` and ran `ab`. As we can see from the graph below, as\nthe max_batch_delay was set closer to 0, the TPS increased and gets closer to being similar to `batch_size` 1.\n\n![tps_vs_batch_delay](images/tps_vs_delay.png) \n\n## Conclusion\nThe take away from the experiments is that batching is a very useful feature. In cases where the services receive heavy load of requests or each request has high I/O, its advantageous\nto batch the requests. This allows for maximally utilizing the compute resources, especially GPU compute which are also more often than not more expensive. But customers should\ndo their due diligence and perform enough tests to find optimal batch size depending on the number of GPUs available and number of models loaded per GPU. Customers should also\nanalyze their traffic patterns before enabling the batch-inference. As shown in the above experiments, services receiving TPS lesser than the batch size would lead to consistent\n\"batch delay\" timeouts and cause the response latency per request to spike. As any cutting edge technology, batch-inference is definitely a double edged sword. \n\n \n" }, { "path": "docs/configuration.md", "content": "# Advanced configuration\n\nOne of design goal of MMS 1.0 is easy to use. The default settings form MMS should be sufficient for most of use cases. This document describe advanced configurations that allows user to deep customize MMS's behavior.\n\n## Environment variables\n\nUser can set environment variables to change MMS behavior, following is a list of variables that user can set for MMS:\n* JAVA_HOME\n* PYTHONPATH\n* MMS_CONFIG_FILE\n* LOG_LOCATION\n* METRICS_LOCATION\n\n**Note:** environment variable has higher priority that command line or config.properties. It will override other property values.\n\n## Command line parameters\n\nUser can following parameters to start MMS, those parameters will override default MMS behavior:\n\n* **--mms-config** MMS will load specified configuration file if MMS_CONFIG_FILE is not set.\n* **--model-store** This parameter will override `model_store` property in config.properties file.\n* **--models** This parameter will override `load_models' property in config.properties.\n* **--log-config** This parameter will override default log4j2.xml\n* **--foreground** This parameter will run the model server in foreground. If this option is\n disabled, the model server will run in the background.\n\nSee [Running the Model Server](server.md) for detail.\n\n## config.properties file\n\nMMS use a `config.properties` file to store configurations. MMS use following order to locate this `config.properties` file:\n1. if `MMS_CONFIG_FILE` environment variable is set, MMS will load the configuration from the environment variable.\n2. if `--mms-config` parameter is passed to `multi-model-server`, MMS will load the configuration from the parameter.\n3. if there is a `config.properties` in current folder where user start the `multi-model-server`, MMS will load the `config.properties` file form current working directory.\n4. If none of above is specified, MMS will load built-in configuration with default values.\n\n**Note:** Docker image that MMS provided has slightly different default value.\n\n### Customize JVM options\n\nThe restrict MMS frontend memory footprint, certain JVM options is set via **vmargs** property in `config.properties` file\n\n* default: N/A, use JVM default options\n* docker default: -Xmx128m -XX:-UseLargePages -XX:+UseG1GC -XX:MaxMetaspaceSize=32M -XX:MaxDirectMemorySize=10m -XX:OnOutOfMemoryError='kill -9 %p'\n\nUser can adjust those JVM options for fit their memory requirement if needed.\n\n### Load models at startup\n\nUser can configure load models while MMS startup. MMS can load models from `model_store` or from HTTP(s) URL.\n\n* model_store\n\t* standalone: default: N/A, load models from local disk is disabled.\n\t* docker: default model_store location is set to: /opt/ml/model\n\n* load_models\n\t* standalone: default: N/A, no models will be load on startup.\n\t* docker: default: ALL, all model archives in /opt/ml/model will be loadded on startup.\n\n**Note:** `model_store` and `load_models` property can be override by command line parameters.\n\n### Configure MMS listening port\n\nMMS doesn't support authentication natively. To avoid unauthorized access, MMS only allows localhost access by default. Inference API is listening on 8080 port and accepting HTTP request. Management API is listening on 8081 port and accepting HTTP request. See [Enable SSL](#enable-ssl) for configuring HTTPS.\n\n* inference_address: inference API binding address, default: http://127.0.0.1:8080\n* management_address: management API binding address, default: http://127.0.0.1:8081\n\nHere are a couple of examples:\n```properties\n# bind inference API to all network interfaces with SSL enabled\ninference_address=https://0.0.0.0:8443\n\n# bind inference API to private network interfaces\ninference_address=https://172.16.1.10:8080\n```\n\n### Enable SSL\n\nFor users who want to enable HTTPs, you can change `inference_address` or `management_addrss` protocol from http to https, for example: `inference_addrss=https://127.0.0.1`. This will make MMS listening on localhost 443 port to accepting https request.\n\nUser also must provide certificate and private keys to enable SSL. MMS support two ways to configure SSL:\n1. Use keystore\n\t* keystore: Keystore file location, if multiple private key entry in the keystore, first one will be picked. \n\t* keystore_pass: keystore password, key password (if applicable) MUST be the same as keystore password.\n * keystore_type: type of keystore, default: PKCS12\n\n2. Use private-key/certificate files\n\t* private_key_file: private key file location, support both PKCS8 and OpenSSL private key.\n\t* certificate_file: X509 certificate chain file location.\n\n#### Self-signed certificate example\n\nThis is a quick example to enable SSL with self-signed certificate\n\n1. User java keytool to create keystore\n```bash\nkeytool -genkey -keyalg RSA -alias mms -keystore keystore.p12 -storepass changeit -storetype PKCS12 -validity 3600 -keysize 2048 -dname \"CN=www.MY_MMS.com, OU=Cloud Service, O=model server, L=Palo Alto, ST=California, C=US\"\n```\n\nConfig following property in config.properties:\n\n```properties\ninference_address=https://127.0.0.1:8443\nmanagement_address=https://127.0.0.1:8444\nkeystore=keystore.p12\nkeystore_pass=changeit\nkeystore_type=PKCS12\n```\n\n2. User OpenSSL to create private key and certificate\n```bash\n```\n\n\nConfig following property in config.properties:\n\n```properties\ninference_address=https://127.0.0.1:8443\nmanagement_address=https://127.0.0.1:8444\nkeystore=keystore.p12\nkeystore_pass=changeit\nkeystore_type=PKCS12\n```\n\n### Configure Cross-Origin Resource Sharing (CORS)\nCORS is a mechanism that uses additional HTTP headers to tell a browser to let a\nweb application running at one origin (domain) have permission to access selected\nresources from a server at a different origin.\n\nCORS is disabled by default. Configure following properties in config.properties file to enable CORS:\n\n```properties\n# cors_allowed_origin is required to enable CORS, use '*' or your domain name \ncors_allowed_origin=https://yourdomain.com\n# required if you want to use preflight request \ncors_allowed_methods=GET, POST, PUT, OPTIONS\n# required if the request has an Access-Control-Request-Headers header \ncors_allowed_headers=X-Custom-Header\n```\n\n### Preloading a model\nThe model server gives users an option to take advantage of fork() sematics, ie., copy-on-write, on linux based systems. In order to load a model before spinning up the model workers, use `preload_model` option. Model server upon seeing this option set, will load the model just before scaling the first model worker. All the other workers will share the same\ninstance of the loaded model. This way only the memory locations in the loaded model which are touch will be copied over to the individual model-workers process memory space.\n\n```properties\npreload_model=true\n```\n\n### Prefer direct buffer\nConfiguration parameter prefer_direct_buffer controls if the model server will be using direct memory specified by -XX:MaxDirectMemorySize. This parameter is for model server only and doesn't affect other packages' usage of direct memory buffer. Default: false\n\n```properties\nprefer_direct_buffer=true\n```\n\n### Restrict backend worker to access environment variable\n\nEnvironment variable may contains sensitive information like AWS credentials. Backend worker will execute arbitrary model's custom code, which may expose security risk. MMS provides a `blacklist_env_vars` property which allows user to restrict which environment variable can be accessed by backend worker.\n\n* blacklist_env_vars: a regular expression to filter out environment variable names, default: all environment variable will be visible to backend worker.\n\n### Limit GPU usage\nBy default, MMS will use all available GPUs for inference, you use `number_of_gpu` to limit the usage of GPUs.\n\n* number_of_gpu: max number of GPUs that MMS can use for inference, default: available GPUs in system.\n\n### Other properties\n\nMost of those properties are designed for performance tuning. Adjusting those numbers will impact scalability and throughput.\n\n* enable_envvars_config: Enable configuring MMS through environment variables. When this option is set to \"true\", all the static configurations of MMS can come through environment variables as well. default: false\n* number_of_netty_threads: number frontend netty thread, default: number of logical processors available to the JVM.\n* netty_client_threads: number of backend netty thread, default: number of logical processors available to the JVM.\n* default_workers_per_model: number of workers to create for each model that loaded at startup time, default: available GPUs in system or number of logical processors available to the JVM.\n* job_queue_size: number inference jobs that frontend will queue before backend can serve, default 100. Useful in cases where certain requests take predictably longer to complete.\n* async_logging: enable asynchronous logging for higher throughput, log output may be delayed if this is enabled, default: false.\n* default_response_timeout: Timeout, in seconds, used for model's backend workers before they are deemed unresponsive and rebooted. default: 120 seconds.\n* unregister_model_timeout: Timeout, in seconds, used when handling an unregister model request when cleaning a process before it is deemed unresponsive and an error response is sent. default: 120 seconds.\n* decode_input_request: Configuration to let backend workers to decode requests, when the content type is known. \nIf this is set to \"true\", backend workers do \"Bytearray to JSON object\" conversion when the content type is \"application/json\" and \nthe backend workers convert \"Bytearray to utf-8 string\" when the Content-Type of the request is set to \"text*\". default: true \n\n### config.properties Example\n\nSee [config.properties for docker](https://github.com/awslabs/multi-model-server/blob/master/docker/config.properties)\n\n" }, { "path": "docs/custom_service.md", "content": "# Custom Service\n\n## Contents of this Document\n* [Introduction](#introduction)\n* [Requirements for custom service file](#requirements-for-custom-service-file)\n* [Example Custom Service file](#example-custom-service-file)\n* [Creating model archive with entry point](#creating-model-archive-with-entry-point)\n\n## Introduction\n\nA custom service , is the code that is packaged into model archive, that is executed by Multi Model Server (MMS). \nThe custom service is responsible for handling incoming data and passing on to engine for inference. The output of the custom service is returned back as response by MMS.\n\n## Requirements for custom service file\n\nThe custom service file should define a method that acts as an entry point for execution, this function will be invoked by MMS on a inference request. \nThe function can have any name, not necessarily handle, however this function should accept, the following parameters\n \n* **data** - The input data from the incoming request\n* **context** - Is the MMS [context](https://github.com/awslabs/multi-model-server/blob/master/mms/context.py) information passed for use with the custom service if required. \n\n\nThe signature of a entry point function is:\n\n```python\ndef function_name(data,context):\n \"\"\"\n Works on data and context passed\n \"\"\"\n # Use parameters passed\n```\nThe next section, showcases an example custom service.\n\n## Example Custom Service file\n\n```python\n# custom service file\n\n# model_handler.py\n\n\"\"\"\nModelHandler defines a base model handler.\n\"\"\"\nimport logging\n\n\nclass ModelHandler(object):\n \"\"\"\n A base Model handler implementation.\n \"\"\"\n\n def __init__(self):\n self.error = None\n self._context = None\n self._batch_size = 0\n self.initialized = False\n\n def initialize(self, context):\n \"\"\"\n Initialize model. This will be called during model loading time\n :param context: Initial context contains model server system properties.\n :return:\n \"\"\"\n self._context = context\n self._batch_size = context.system_properties[\"batch_size\"]\n self.initialized = True\n\n def preprocess(self, batch):\n \"\"\"\n Transform raw input into model input data.\n :param batch: list of raw requests, should match batch size\n :return: list of preprocessed model input data\n \"\"\"\n # Take the input data and pre-process it make it inference ready\n assert self._batch_size == len(batch), \"Invalid input batch size: {}\".format(len(batch))\n return None\n\n def inference(self, model_input):\n \"\"\"\n Internal inference methods\n :param model_input: transformed model input data\n :return: list of inference output in NDArray\n \"\"\"\n # Do some inference call to engine here and return output\n return None\n\n def postprocess(self, inference_output):\n \"\"\"\n Return predict result in batch.\n :param inference_output: list of inference output\n :return: list of predict results\n \"\"\"\n # Take output from network and post-process to desired format\n return [\"OK\"] * self._batch_size\n \n def handle(self, data, context):\n \"\"\"\n Call preprocess, inference and post-process functions\n :param data: input data\n :param context: mms context\n \"\"\"\n \n model_input = self.preprocess(data)\n model_out = self.inference(model_input)\n return self.postprocess(model_out)\n\n_service = ModelHandler()\n\n\ndef handle(data, context):\n if not _service.initialized:\n _service.initialize(context)\n\n if data is None:\n return None\n\n return _service.handle(data, context)\n\n```\nHere the ``` handle()``` method is our entry point that will be invoked by MMS, with the parameters data and context, it in turn can pass this information to an actual inference class object or handle all the processing in the \n```handle()``` method itself. The ```initialize()``` method is used to initialize the model at load time, so after first time, the service need not be re-initialized in the the life cycle of the relevant worker.\n We recommend using a ```initialize()``` method, avoid initialization at prediction time.\n \n This entry point is engaged in two cases: (1) when MMS is asked to scale a model up, to increase the number of backend workers (it is done either via a ```PUT /models/{model_name}``` request or a ```POST /models``` request with `initial-workers` option or during MMS startup when you use `--models` option (```multi-model-server --start --models {model_name=model.mar}```), ie., you provide model(s) to load) or (2) when MMS gets a ```POST /predictions/{model_name}``` request. (1) is used to scale-up or scale-down workers for a model. (2) is used as a standard way to run inference against a model. (1) is also known as model load time, and that is where you would normally want to put code for model initialization. You can find out more about these and other MMS APIs in [MMS Management API](./management_api.md) and [MMS Inference API](./inference_api.md)\n\n\n### Returning custom error codes \n\nTo return a custom error code back to the user use the `PredictionException` in the `mms.service` module.\n\n```python\nfrom mms.service import PredictionException\ndef handler(data, context):\n # Some unexpected error - returning error code 513\n raise PredictionException(\"Some Prediction Error\", 513)\n```\n\n## Creating model archive with entry point \n\nMMS, identifies the entry point to the custom service, from the manifest file. Thus file creating the model archive, one needs to mention the entry point using the ```--handler``` option. \n\nThe [model-archiver](https://github.com/awslabs/multi-model-server/blob/master/model-archiver/README.md) tool enables the create to an archive understood by MMS.\n\n```python\nmodel-archiver --model-name --handler model_handler:handle --export-path --model-path --runtime python3\n```\n\nThis will create file ```.mar``` in the directory ``````\n\nThis will create a model archive with the custom handler, for python3 runtime. The ```--runtime``` parameter enables usage of specific python version at runtime, by default it uses the default python distribution of the system.\n" }, { "path": "docs/elastic_inference.md", "content": "# Model Serving with Amazon Elastic Inference \n\n## Contents of this Document\n* [Introduction](#introduction)\n* [Custom Service](#custom-service)\n* [Creating a EC2 instance with EIA support](#creating-a-ec2-instance-with-eia-support)\n* [Custom Service file with EIA](#custom-service-file-with-eia)\n* [Running elastic inference on a resnet-152](#running-elastic-inference-on-a-resnet-152)\n\n## Introduction\n\nAmazon Elastic Inference (EI) is a service that allows you to attach low-cost GPU-powered acceleration to Amazon EC2 and Amazon SageMaker \ninstances to reduce the cost of running deep learning inference by up to 75%. With MMS it is easy to deploy a MXNet based model, \ntaking advantage of the attachable hardware accelerator called Elastic Inference Accelerator (EIA).\nIn this document, we explore using EIA attached to a Compute Optimized EC2 instance.\n\n## Custom Service\n\nThe capability to run model inference with the EIA can be achieved by building a custom service to use the EIA context rather than a GPU or CPU context. An MXNet version with support for EIA is required.\n\nTo understand the basics of writing a custom service file refer to the [Custom Service Documentation](https://github.com/awslabs/multi-model-server/blob/master/docs/custom_service.md).\n\n## Creating a EC2 instance with EIA support \n\nTo Create an EC2 instance with EIA support there are few pre-requisites. These include:\n1. [Configuring a Security Group for Amazon EI](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/working-with-ei.html#ei-security).\n2. [Configure AWS PrivateLink Endpoint Services](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/working-with-ei.html#eia-privatelink).\n3. [Creating a IAM Role with EI instance policy](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/working-with-ei.html#ei-role-policy).\n \nThe above steps are explored in detail in [AWS Elastic Inference documentation](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/working-with-ei.html) \n\nOn completing the above steps, following steps need to be followed in launching an instance with EIA support\n\n1. Open the Amazon EC2 console at https://console.aws.amazon.com/ec2/.\n\n2. Choose Launch Instance.\n\n3. Choose one of the Deep Learning AMIs, we recommend Deep Learning AMI v20 or later. This is required to use MXNet with EIA.\n\n4. Choose an Instance Type, we recommend a compute optimized EC2 instance such as c5.2xlarge.\n\n5. Choose Next: Configure Instance Details.\n\n6. Under Configure Instance Details, check the configuration settings. Ensure that you are using the VPC with the security groups for the instance and the Amazon EI accelerator that was set up earlier. For more information, see Configuring Your Security Groups for Amazon EI.\n\n7. For IAM role, select the role that you created in the Configuring an Instance Role with an Amazon EI Policy procedure explained in the above documentation.\n\n8. Select Add an Amazon EI accelerator.\n\n9. Select the size of the Amazon EI accelerator. Your options are eia1.medium, eia1.large, and eia1.xlarge. We recommend selecting the instance size, based on the model size. For larger models, larger instances offer better performance gains. \n\n10. (Optional) You can choose to add storage and tags by choosing Next at the bottom of the page. Or, you can let the instance wizard complete the remaining configuration steps for you.\n\n11. Review the configuration of your instance and choose Launch.\n\n12. You are prompted to choose an existing key pair for your instance or to create a new key pair. \n\n**WARNING: Do NOT select the Proceed without a key pair option. If you launch your instance without a key pair, then you can’t connect to it.**\n\nAfter making your key pair selection, choose Launch Instances.\n\nIt can take a few minutes for the instance to be ready so that you can connect to it. Check that your instance has passed its status checks. You can view this information in the Status Checks column.\n\n\n## Custom Service file with EIA \n\nYou use two different processing contexts with MXNet and EIA\n 1. mxnet.cpu() - used for loading up input data\n 2. mxnet.eia() - used for binding network symbols and params to an attached EIA instance\n\n\n \nWe modify the [base model service template](https://github.com/awslabs/multi-model-server/blob/master/examples/model_service_template/mxnet_model_service.py) to support EIA.\n\n\n```python\n def initialize(self, context):\n # NOT COMPLETE CODE, refer template above for it.\n #....\n #....\n #.... \n # Load MXNet module\n # Symbol Context set to eia\n self.mxnet_ctx = mx.eia()\n self.data_ctx = mx.cpu()\n sym, arg_params, aux_params = mx.model.load_checkpoint(checkpoint_prefix, self.epoch)\n\n # noinspection PyTypeChecker\n self.mx_model = mx.mod.Module(symbol=sym, context=self.mxnet_ctx,\n data_names=data_names, label_names=None)\n self.mx_model.bind(for_training=False, data_shapes=data_shapes)\n self.mx_model.set_params(arg_params, aux_params, allow_missing=True, allow_extra=True)\n \n def inference(self, model_input):\n # NOT COMPLETE CODE, refer template above for it.\n #....\n #....\n #.... \n model_input = [item.as_in_context(self.data_ctx) for item in model_input]\n```\n\nThe above code shows initialization of two contexts, one for data (on CPU) and other for symbols (on EIA).\n\nOnce we have the code ready. We can build a model archive consumable by MMS, using the model-archiver. The [model-archiver](https://github.com/awslabs/multi-model-server/blob/master/model-archiver/README.md) tool enables to build to an archive understood by MMS.\n\n```bash\nmodel-archiver --model-name --handler model_service:handle --export-path --model-path --runtime python\n```\n\nThis will create file ```.mar``` in the directory ``````.\n\n## Running elastic inference on a resnet-152\nA pre-built ResNet-152 model archive that uses Amazon Elastic Inference can be downloaded using the following command:\n```bash\n$ wget https://s3.amazonaws.com/model-server/model_archive_1.0/resnet-152-eia.mar\n```\n**NOTE:** The above archive will only work on EIA-enabled instances.\n\nStart the EIA-enabled EC2 instance. If using a Deep Learning AMI, there are two Conda environments (one for Python 2 and one for Python 3), both of which come with MXNet built will EI support and Multi Model Server. Either of the two can be used.\n```bash\n# python 3 \n$ source activate amazonei_mxnet_p36\n\n# python 2 \n$ source activate amazonei_mxnet_p27\n```\n\nAfter entering one of the Conda environments, we start MMS, with Resnet-152 EIA model:\n```bash\n# Start MMS\n$ multi-model-server --start --models resnet-152=https://s3.amazonaws.com/model-server/model_archive_1.0/resnet-152-eia.mar\n```\n\nNow the model is ready for some inference requests. Let us download a kitten image for classification:\n```bash\n$ curl -O https://s3.amazonaws.com/model-server/inputs/kitten.jpg\n$ curl -X POST http://127.0.0.1:8080/predictions/resnet-152 -T kitten.jpg\n```\n\nThe predict endpoint will return a prediction response in JSON. It will look something like the following result:\n\n```json\n[\n {\n \"probability\": 0.7148934602737427,\n \"class\": \"n02123045 tabby, tabby cat\"\n },\n {\n \"probability\": 0.22877734899520874,\n \"class\": \"n02123159 tiger cat\"\n },\n {\n \"probability\": 0.04032360762357712,\n \"class\": \"n02124075 Egyptian cat\"\n },\n {\n \"probability\": 0.008370809257030487,\n \"class\": \"n02127052 lynx, catamount\"\n },\n {\n \"probability\": 0.0006728142034262419,\n \"class\": \"n02129604 tiger, Panthera tigris\"\n }\n]\n```\n\nResNet-152 identified the tabby cat using Elastic Inference Accelerator while being hosted on MMS. Serving on Amazon EI instances reduces inference costs while benefiting from the performance of a GPU for inference tasks." }, { "path": "docs/images/helpers/plugins_sdk_class_uml_diagrams.puml", "content": "\n@startuml\nContext \"1\" *-- \"many\" Model : contains\nModel \"1\" *-- \"many\" Worker : contains\nEndpoint o-- EndpointTypes\n\nModelServerEndpoint <.. Endpoint\nModelServerEndpoint <.. Context\nModelServerEndpoint <.. Request\nModelServerEndpoint <.. Response\n\ninterface Context {\n +Properties getConfig()\n +Map getModels()\n}\n\ninterface Request {\n +List getHeaderNames()\n +String getRequestURI()\n +Map> getParameterMap()\n +List getParameter(String k)\n +String getContentType()\n +InputStream getInputStream()\n}\n\ninterface Response {\n +void setStatus(int sc)\n +void setStatus(int sc, String phrase)\n +void setHeader(String k, String v)\n +void addHeader(String k, String v)\n +void setContentType(String ct)\n +OutputStream getOutputStream()\n}\n\ninterface Model {\n +String getModelName()\n +String getModelUrl()\n +String getModelHandler()\n +List getModelWorkers()\n}\n\ninterface Worker {\n +boolean isRunning()\n +long getWorkerMemory()\n}\n\ninterface ModelServerEndpoint {\n +void doGet(Request req, Response rsp, Context ctx)\n +void doPost(Request req, Response rsp, Context ctx)\n +void doDelete(Request req, Response rsp, Context ctx)\n +void doPut(Request req, Response rsp, Context ctx)\n}\n\nannotation Endpoint\nEndpoint : +String urlPattern()\nEndpoint : +EndpointTypes endpointType()\nEndpoint : +String description()\n\n\nenum EndpointTypes {\n NONE, \n INFERENCE, \n MANAGEMENT\n}\n\n@enduml" }, { "path": "docs/inference_api.md", "content": "# Inference API\n\nInference API is listening on port 8080 and only accessible from localhost by default. To change the default setting, see [MMS Configuration](configuration.md).\n\nThere are three type of APIs:\n\n1. [API description](#api-description) - Describe MMS inference APIs with OpenAPI 3.0 specification\n2. [Health check API](#health-check-api) - Check MMS health status\n3. [Predictions API](#predictions-api) - Make predictions API call to MMS\n\n## API Description\n\nTo view a full list of inference API, you can use following command:\n\n```bash\ncurl -X OPTIONS http://localhost:8443\n```\n\nThe out is OpenAPI 3.0.1 json format. You can use it to generate client code, see [swagger codegen](https://swagger.io/swagger-codegen/) for detail.\n\n* [Inference API description output](../frontend/server/src/test/resources/inference_open_api.json)\n\n## Health check API\n\nMMS support a `ping` API that user can check MMS health status:\n\n```bash\ncurl http://localhost:8080/ping\n```\n\nYour response, if the server is running should be:\n\n```json\n{\n \"health\": \"healthy!\"\n}\n```\n\n## Predictions API\n\nMMS 1.0 support 0.4 style API calls, those APIs are deprecated, they will be removed in future release. See [Deprecated APIs](#deprecated-api) for detail.\n\nFor each loaded model, user can make REST call to URI: /predictions/{model_name}\n\n* POST /predictions/{model_name}\n\n**curl Example**\n\n```bash\ncurl -O https://s3.amazonaws.com/model-server/inputs/kitten.jpg\n\ncurl -X POST http://localhost:8080/predictions/resnet-18 -T kitten.jpg\n\nor:\n\ncurl -X POST http://localhost:8080/predictions/resnet-18 -F \"data=@kitten.jpg\"\n```\n\nThe result was some JSON that told us our image likely held a tabby cat. The highest prediction was:\n\n```json\n{\n \"class\": \"n02123045 tabby, tabby cat\",\n \"probability\": 0.42514491081237793,\n ...\n}\n```\n\n## Deprecated API\n\nMMS 0.4 style predict API is kept for backward compatible purpose, and will be removed in future release.\n\n* POST /{model_name}/predict\n\n**curl Example**\n\n```bash\ncurl -O https://s3.amazonaws.com/model-server/inputs/kitten.jpg\n\ncurl -X POST http://localhost:8080/resnet-18/predict -F \"data=@kitten.jpg\"\n```\n" }, { "path": "docs/install.md", "content": "\n# Install MMS\n\n## Prerequisites\n\n* **Python**: Required. Multi Model Server (MMS) works with Python 2 or 3. When installing MMS, we recommend that you use a Python and Conda environment to avoid conflicts with your other Apache MXNet or Open Neural Network Exchange (ONNX) installations.\n\n* **java 8**: Required. MMS use java to serve HTTP requests. You must install java 8 (or later) and make sure java is on available in $PATH environment variable *before* installing MMS. If you have multiple java installed, you can use $JAVA_HOME environment vairable to control which java to use.\n\nFor ubuntu:\n```bash\nsudo apt-get install openjdk-8-jre-headless\n```\n\nFor centos\n```bash\nsudo yum install java-1.8.0-openjdk\n```\n\nFor Mac:\n```bash\nbrew tap caskroom/versions\nbrew update\nbrew cask install java8\n```\n\nYou can also download and install [Oracle JDK](https://www.oracle.com/technetwork/java/javase/overview/index.html) manually if you have trouble with above commands.\n\n* **MXNet**: Recommended. MMS won't install `mxnet` by default. MXNet is required for most of examples in this project. MMS won't install mxnet engine by default, you can install mxnet-mkl or mxnet-cu90mkl based on your need. And you can also choose specific version of mxnet if you want.\n\n```bash\npip install mxnet-mkl\n```\n\nor for GPU instance:\n\n```bash\npip install mxnet-cu90mkl\n```\n\n\n* **Curl**: Optional. Curl is used in all of the examples. Install it with your preferred package manager.\n\n* **Unzip**: Optional. Unzip allows you to easily extract model files and inspect their content. If you choose to use it, associate it with `.mar` extensions.\n\n## Install MMS with pip\n\nTo install MMS for the first time, install Python, then run the following command:\n\n```bash\npip install multi-model-server\n```\n\nTo upgrade from a previous version of MMS, run:\n\n```bash\npip install -U multi-model-server\n```\n\n## Install MMS from Source Code\n\n\n\nIf you prefer, you can clone MMS from source code. First, run the following command:\n\n\n\n```bash\ngit clone https://github.com/awslabs/multi-model-server.git && cd multi-model-server\n```\n\nTo install MMS, run:\n\n\n```bash\npip install .\n```\n\nTo upgrade MMS, run:\n\n\n```bash\npip install -U .\n```\n\n\n\n\n## Install MMS for Development\n\nIf you plan to develop with MMS and change some of the source code, install it from source code and make your changes executable with this command:\n\n\n\n```bash\npip install -e .\n```\n\nTo upgrade MMS from source code and make changes executable, run:\n\n\n```bash\npip install -U -e .\n```\n\n## Troubleshooting Installation\n\n\n| Issue | Solution |\n|---|---|\n|java not found, please make sure JAVA_HOME is set properly. | Make sure java is installed. java is on the $PATH or $JAVA_HOME is set properly. |\n|Your PYTHONPATH points to a site-packages dir for Python 3.x but you are running Python 2.x! | You do one of following:
  • use virtualenv
  • unset PYTHONPATH
  • set PYTHONPATH properly
|\n" }, { "path": "docs/logging.md", "content": "# Logging on Multi Model Server\n\nIn this document we will go through logging mechanism in Multi Model Server. We will also go over how to modify the behavior of logging in model-server. Logging in Multi Model Server also covers\nmetrics, as metrics are logged into a file. To further understand how to customize metrics or define custom logging layouts, refer to the [metrics document](metrics.md)\n\n# Pre-requisites\nBefore getting into this tutorials, you must familiarize yourself with log4j2 configuration. Refer to this online [document](https://logging.apache.org/log4j/2.x/manual/configuration.html) on how to configure the log4j2 parameters. Similarly, familiarize yourself with the default [log4j2.xml](../frontend/server/src/main/resources/log4j2.xml) used by Multi Model Server.\n\n# Types of logs\nMulti Model Server currently provides three types of logs.\n1. Access Logs.\n1. Model Server Logs.\n\n## Access Logs:\nThese logs collect the access pattern to Multi Model Server. The configuration pertaining to access logs are as follows,\n```xml\n \n \n \n \n \n \n \n \n```\n\nAs defined in the properties file, the access logs are collected in {LOG_LOCATION}/access_log.log file. When we load the model server\nwith a model and run inference against the server, the following logs are collected into the access_log.log\n```text\n2018-10-15 13:56:18,976 [INFO ] BackendWorker-9000 ACCESS_LOG - /127.0.0.1:64003 \"POST /predictions/resnet-18 HTTP/1.1\" 200 118\n```\nThe above log tells us that a successful `POST` call to `/predictions/resnet-18` was made by remote host `127.0.0.1:64003` it took `118`ms to complete this request.\n\nThese logs are useful to determine the current performance of the model-server as well as understand the requests received by model-server.\n\n## Model Server Logs\nThese logs collect all the logs from Model Server and from the backend workers (the custom model code).\nThe default configuration pertaining to mms logs are as follows:\n```xml\n \n \n \n \n \n \n \n \n```\n\nThis configuration by default dumps all the logs above `DEBUG` level. \n\n### Generating and logging custom logs\nAs a user of Multi Model Server(MMS), you might want to log custom logs into the log files. This could be for debug purposes or\nto log any errors. To accomplish this, simply print the required logs to `stdout/stderr`. MMS will capture the logs generated by the \nbackend workers and log it into the log file. Some examples of logs are as follows\n\n1. Messages printed to stderr \n```text\n2018-10-14 16:46:51,656 [WARN ] W-9000-stderr com.amazonaws.ml.mms.wlm.WorkerLifeCycle - [16:46:51] src/nnvm/legacy_json_util.cc:209: Loading symbol saved by previous version v0.8.0. Attempting to upgrad\\\ne...\n2018-10-14 16:46:51,657 [WARN ] W-9000-stderr com.amazonaws.ml.mms.wlm.WorkerLifeCycle - [16:46:51] src/nnvm/legacy_json_util.cc:217: Symbol successfully upgraded!\n```\n \n1. Messages printed to stdout \n```text\n2018-10-14 16:59:59,926 [INFO ] W-9000-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - preprocess time: 3.60\n2018-10-14 16:59:59,926 [INFO ] W-9000-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - inference time: 117.31\n2018-10-14 16:59:59,926 [INFO ] W-9000-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - postprocess time: 8.52\n```\n\n# Modifying the behavior of the logs\nIn order to modify the default behavior of the logging, you could define `log4j2.xml` file. There are two ways of starting\nmodel server with custom logs\n\n### Provide with config.properties\n Once you define custom `log4j2.xml`, add this to the `config.properties` file as follows\n\n```properties\nvmargs=-Dlog4j.configurationFile=file:///path/to/custom/log4j2.xml\n```\nThen start the model server as follows\n```bash\n$ multi-model-server --start --mms-config /path/to/config.properties\n```\n### Provide with CLI \nAlternatively, you could start the model server with the following command as well\n\n```bash\n$ multi-model-server --start --log-config /path/to/custom/log4j2.xml\n```\n\n# Enable asynchronous logging\nIf your model is super lightweight and seeking for high throughput, you can consider enable asynchronous logging.\nNote that log output maybe delayed and latest log might be lost if MMS is terminated unexpectedly.\nasynchronous logging is disabled by default.\nTo enable asynchronous logging, add following property in `config.properties`:\n \n```properties\nasync_logging=true\n```\n" }, { "path": "docs/management_api.md", "content": "# Management API\n\nMMS provides a set of API allow user to manage models at runtime:\n1. [Register a model](#register-a-model)\n2. [Increase/decrease number of workers for specific model](#scale-workers)\n3. [Describe a model's status](#describe-model)\n4. [Unregister a model](#unregister-a-model)\n5. [List registered models](#list-models)\n\nManagement API is listening on port 8081 and only accessible from localhost by default. To change the default setting, see [MMS Configuration](configuration.md).\n\nSimilar as [Inference API](inference_api.md), Management API also provide a [API description](#api-description) to describe management APIs with OpenAPI 3.0 specification.\n\n## Management APIs\n\n### Register a model\n\n`POST /models`\n* url - Model archive download url. Supports the following locations:\n * a local model archive (.mar); the file must be directly in model_store folder.\n * a local model directory; the directory must be directly in model_store folder. This option can avoid MMS extracting .mar file to temporary folder, which will improve load time and reduce disk space usage.\n * a URI using the HTTP(s) protocol. MMS can download .mar files from the Internet.\n* model_name - the name of the model; this name will be used as {model_name} in other API as path. If this parameter is not present, modelName in MANIFEST.json will be used.\n* handler - the inference handler entry-point. This value will override `handler` in MANIFEST.json if present. **NOTE: Make sure that the given `handler` is in the `PYTHONPATH`. The format of handler is `module_name:method_name`.**\n* runtime - the runtime for the model custom service code. This value will override runtime in MANIFEST.json if present. The default value is `PYTHON`.\n* batch_size - the inference batch size. The default value is `1`.\n* max_batch_delay - the maximum delay for batch aggregation. The default value is 100 milliseconds.\n* initial_workers - the number of initial workers to create. The default value is `0`. MMS will not run inference until there is at least one work assigned.\n* synchronous - whether or not the creation of worker is synchronous. The default value is false. MMS will create new workers without waiting for acknowledgement that the previous worker is online.\n* response_timeout - If the model's backend worker doesn't respond with inference response within this timeout period, the worker will be deemed unresponsive and rebooted. The units is seconds. The default value is 120 seconds.\n\n```bash\ncurl -X POST \"http://localhost:8081/models?url=https%3A%2F%2Fs3.amazonaws.com%2Fmodel-server%2Fmodel_archive_1.0%2Fsqueezenet_v1.1.mar\"\n\n{\n \"status\": \"Model \\\"squeezenet_v1.1\\\" registered\"\n}\n```\n\nUser may want to create workers while register, creating initial workers may take some time, user can choose between synchronous or synchronous call to make sure initial workers are created properly.\n\nThe asynchronous call will return before trying to create workers with HTTP code 202:\n\n```bash\ncurl -v -X POST \"http://localhost:8081/models?initial_workers=1&synchronous=false&url=https%3A%2F%2Fs3.amazonaws.com%2Fmodel-server%2Fmodel_archive_1.0%2Fsqueezenet_v1.1.mar\"\n\n< HTTP/1.1 202 Accepted\n< content-type: application/json\n< x-request-id: 29cde8a4-898e-48df-afef-f1a827a3cbc2\n< content-length: 33\n< connection: keep-alive\n< \n{\n \"status\": \"Worker updated\"\n}\n```\n\nThe synchronous call will return after all workers has be adjusted with HTTP code 200.\n\n```bash\ncurl -v -X POST \"http://localhost:8081/models?initial_workers=1&synchronous=true&url=https%3A%2F%2Fs3.amazonaws.com%2Fmodel-server%2Fmodel_archive_1.0%2Fsqueezenet_v1.1.mar\"\n\n< HTTP/1.1 200 OK\n< content-type: application/json\n< x-request-id: c4b2804e-42b1-4d6f-9e8f-1e8901fc2c6c\n< content-length: 32\n< connection: keep-alive\n< \n{\n \"status\": \"Worker scaled\"\n}\n```\n\n\n### Scale workers\n\n`PUT /models/{model_name}`\n* min_worker - (optional) the minimum number of worker processes. MMS will try to maintain this minimum for specified model. The default value is `1`.\n* max_worker - (optional) the maximum number of worker processes. MMS will make no more that this number of workers for the specified model. The default is the same as the setting for `min_worker`.\n* number_gpu - (optional) the number of GPU worker processes to create. The default value is `0`. If number_gpu exceeds the number of available GPUs, the rest of workers will run on CPU.\n* synchronous - whether or not the call is synchronous. The default value is `false`.\n* timeout - the specified wait time for a worker to complete all pending requests. If exceeded, the work process will be terminated. Use `0` to terminate the backend worker process immediately. Use `-1` to wait infinitely. The default value is `-1`. \n**Note:** not implemented yet.\n\nUse the Scale Worker API to dynamically adjust the number of workers to better serve different inference request loads.\n\nThere are two different flavour of this API, synchronous vs asynchronous.\n\nThe asynchronous call will return immediately with HTTP code 202:\n\n```bash\ncurl -v -X PUT \"http://localhost:8081/models/noop?min_worker=3\"\n\n< HTTP/1.1 202 Accepted\n< content-type: application/json\n< x-request-id: 74b65aab-dea8-470c-bb7a-5a186c7ddee6\n< content-length: 33\n< connection: keep-alive\n< \n{\n \"status\": \"Worker updated\"\n}\n```\n\nThe synchronous call will return after all workers has be adjusted with HTTP code 200.\n\n```bash\ncurl -v -X PUT \"http://localhost:8081/models/noop?min_worker=3&synchronous=true\"\n\n< HTTP/1.1 200 OK\n< content-type: application/json\n< x-request-id: c4b2804e-42b1-4d6f-9e8f-1e8901fc2c6c\n< content-length: 32\n< connection: keep-alive\n< \n{\n \"status\": \"Worker scaled\"\n}\n```\n\n### Describe model\n\n`GET /models/{model_name}`\n\nUse the Describe Model API to get detail runtime status of a model:\n\n```bash\ncurl http://localhost:8081/models/noop\n\n{\n \"modelName\": \"noop\",\n \"modelVersion\": \"snapshot\",\n \"modelUrl\": \"noop.mar\",\n \"engine\": \"MXNet\",\n \"runtime\": \"python\",\n \"minWorkers\": 1,\n \"maxWorkers\": 1,\n \"batchSize\": 1,\n \"maxBatchDelay\": 100,\n \"workers\": [\n {\n \"id\": \"9000\",\n \"startTime\": \"2018-10-02T13:44:53.034Z\",\n \"status\": \"READY\",\n \"gpu\": false,\n \"memoryUsage\": 89247744\n }\n ]\n}\n```\n\n### Unregister a model\n\n`DELETE /models/{model_name}`\n\nUse the Unregister Model API to free up system resources:\n\n```bash\ncurl -X DELETE http://localhost:8081/models/noop\n\n{\n \"status\": \"Model \\\"noop\\\" unregistered\"\n}\n```\n\n### List models\n\n`GET /models`\n* limit - (optional) the maximum number of items to return. It is passed as a query parameter. The default value is `100`.\n* next_page_token - (optional) queries for next page. It is passed as a query parameter. This value is return by a previous API call.\n\nUse the Models API to query current registered models:\n\n```bash\ncurl \"http://localhost:8081/models\"\n```\n\nThis API supports pagination:\n\n```bash\ncurl \"http://localhost:8081/models?limit=2&next_page_token=2\"\n\n{\n \"nextPageToken\": \"4\",\n \"models\": [\n {\n \"modelName\": \"noop\",\n \"modelUrl\": \"noop-v1.0\"\n },\n {\n \"modelName\": \"noop_v0.1\",\n \"modelUrl\": \"noop-v0.1\"\n }\n ]\n}\n```\n\n\n## API Description\n\n`OPTIONS /`\n\nTo view a full list of inference and management APIs, you can use following command:\n\n```bash\n# To view all inference APIs:\ncurl -X OPTIONS http://localhost:8080\n\n# To view all management APIs:\ncurl -X OPTIONS http://localhost:8081\n```\n\nThe out is OpenAPI 3.0.1 json format. You use it to generate client code, see [swagger codegen](https://swagger.io/swagger-codegen/) for detail.\n\nExample outputs of the Inference and Management APIs:\n* [Inference API description output](../frontend/server/src/test/resources/inference_open_api.json)\n* [Management API description output](../frontend/server/src/test/resources/management_open_api.json)\n" }, { "path": "docs/metrics.md", "content": "# Metrics on Model Server\n\n## Contents of this Document\n* [Introduction](#introduction)\n* [System metrics](#system-metrics)\n* [Formatting](#formatting)\n* [Custom Metrics API](#custom-metrics-api)\n\n## Introduction\nMMS collects system level metrics in regular intervals, and also provides an API for custom metrics to be collected. Metrics collected by metrics are logged and can be aggregated by metric agents.\nThe system level metrics are collected every minute. Metrics defined by the custom service code, can be collected per request or a batch of requests. MMS logs these two sets of metrics to different log files.\nMetrics are collected by default at:\n* System metrics - log_directory/mms_metrics.log\n* Custom metrics - log directory/model_metrics.log\n\nThe location of log files and metric files can be configured at [log4j2.xml](https://github.com/awslabs/multi-model-server/blob/master/frontend/server/src/main/resources/log4j2.xml) file.\n\n\n## System Metrics\n\n|\tMetric Name\t|\tDimension\t|\tUnit\t|\tSemantics\t|\n|---|---|---|---|\n|\tCPUUtilization\t|\thost\t|\tpercentage\t|\tcpu utillization on host\t|\n|\tDiskAvailable\t|\thost\t|\tGB\t|\tdisk available on host\t|\n|\tDiskUsed\t|\thost\t|\tGB\t|\tdisk used on host\t|\n|\tDiskUtilization\t|\thost\t|\tpercentage\t|\tdisk used on host\t|\n|\tMemoryAvailable\t|\thost\t|\tMB\t|\tmemory available on host\t|\n|\tMemoryUsed\t|\thost\t|\tMB\t|\tmemory used on host\t|\n|\tMemoryUtilization\t|\thost\t|\tpercentage\t|\tmemory used on host\t|\n|\tRequests2XX\t|\thost\t|\tcount\t|\ttotal number of requests that responded in 200-300 range\t|\n|\tRequests4XX\t|\thost\t|\tcount\t|\ttotal number of requests that responded in 400-500 range |\n|\tRequests5XX\t|\thost\t|\tcount\t|\ttotal number of requests that responded above 500 |\n\n\n## Formatting\n\nThe metrics emitted into log files by default, is in a [StatsD](https://github.com/etsy/statsd) like format.\n\n```bash\nCPUUtilization.Percent:0.0|#Level:Host|#hostname:my_machine_name\nMemoryUsed.Megabytes:13840.328125|#Level:Host|#hostname:my_machine_name \n```\n\nTo enable metric logging in JSON format, we can modify the log formatter in [log4j2.xml](https://github.com/awslabs/multi-model-server/blob/master/frontend/server/src/main/resources/log4j2.xml), This is explained in the logging [document](https://github.com/awslabs/multi-model-server/blob/master/docs/logging.md).\n\nOnce enabled the format emitted to logs, will look as follows\n\n```json\n{ \n \"MetricName\": \"DiskAvailable\",\n \"Value\": \"108.15547180175781\",\n \"Unit\": \"Gigabytes\",\n \"Dimensions\": [\n { \n \"Name\": \"Level\",\n \"Value\": \"Host\"\n }\n ],\n \"HostName\": \"my_machine_name\"\n}\n{ \n \"MetricName\": \"DiskUsage\",\n \"Value\": \"124.13163757324219\",\n \"Unit\": \"Gigabytes\",\n \"Dimensions\": [\n {\n \"Name\": \"Level\",\n \"Value\": \"Host\"\n }\n ],\n \"HostName\": \"my_machine_name\"\n}\n\n```\n\n## Custom Metrics API\n\nMMS enables the custom service code to emit metrics, that are then logged by the system\n\nThe custom service code is provided with a [context](https://github.com/awslabs/multi-model-server/blob/master/mms/context.py) of the current request.\n\nWhich has metrics object.\n\n```python\n# Access context metrics as follows\nmetrics = context.metrics\n```\nAll metrics collected with in the context \n\n### Creating dimension object(s)\n\nDimensions for metrics can be defined as objects\n\n```python\nfrom mms.metrics import dimension\n\n# Dimensions are name value pairs\ndim1 = Dimension(name, value)\ndim2 = Dimension(some_name, some_value)\n.\n.\n.\ndimN= Dimension(name_n, value_n)\n\n```\n\n**NOTE:** Metric functions below accept a list of dimensions\n\n### Add generic metrics\n\nOne can add metrics with generic units using the following function.\n\nFunction API\n```python\n def add_metric(name, value, idx=None, unit=None, dimensions=None):\n \"\"\"\n Add a metric which is generic with custom metrics\n\n Parameters\n ----------\n name : str\n metric name\n value: int, float\n value of metric\n idx: int\n request_id index in batch\n unit: str\n unit of metric\n dimensions: list\n list of dimensions for the metric\n \"\"\"\n```\n\n```python\n# Add Distance as a metric\n# dimensions = [dim1, dim2, dim3, ..., dimN]\n# Assuming batch size is 1 for example\nmetrics.add_metric('DistanceInKM', distance, 'km', dimensions)\n```\n\n\n### Add Time based metrics\nTime based metrics can be added by invoking the following method\n\nFunction API\n```python\n def add_time(name, value, idx=None, unit='ms', dimensions=None):\n \"\"\"\n Add a time based metric like latency, default unit is 'ms'\n\n Parameters\n ----------\n name : str\n metric name\n value: int\n value of metric\n idx: int\n request_id index in batch\n unit: str\n unit of metric, default here is ms, s is also accepted\n dimensions: list\n list of dimensions for the metric\n \"\"\"\n```\n\nNote that the default unit in this case is 'ms'\n\n**Supported units**: ['ms', 's']\n\nTo add custom time based metrics\n\n```python\n# Add inference time\n# dimensions = [dim1, dim2, dim3, ..., dimN]\n# Assuming batch size is 1 for example\nmetrics.add_time('InferenceTime', end_time-start_time, None, 'ms', dimensions)\n```\n\n### Add Size based metrics\nSize based metrics can be added by invoking the following method\n\nFunction API\n```python\n def add_size(name, value, idx=None, unit='MB', dimensions=None):\n \"\"\"\n Add a size based metric\n\n Parameters\n ----------\n name : str\n metric name\n value: int, float\n value of metric\n idx: int\n request_id index in batch\n unit: str\n unit of metric, default here is 'MB', 'kB', 'GB' also supported\n dimensions: list\n list of dimensions for the metric\n \"\"\"\n```\n\nNote that the default unit in this case is 'ms'\n\n**Supported units**: ['MB', 'kB', 'GB']\n\nTo add custom size based metrics\n\n```python\n# Add Image size as a metric\n# dimensions = [dim1, dim2, dim3, ..., dimN]\n# Assuming batch size is 1 for example\nmetrics.add_size('SizeOfImage', img_size, None, 'MB', dimensions)\n```\n\n### Add Percentage based metrics\n\nPercentage based metrics can be added by invoking the following method\n\nFunction API\n```python\n def add_percent(name, value, idx=None, dimensions=None):\n \"\"\"\n Add a percentage based metric\n\n Parameters\n ----------\n name : str\n metric name\n value: int, float\n value of metric\n idx: int\n request_id index in batch\n dimensions: list\n list of dimensions for the metric\n \"\"\"\n```\n\nTo add custom percentage based metrics\n\n```python\n# Add MemoryUtilization as a metric\n# dimensions = [dim1, dim2, dim3, ..., dimN]\n# Assuming batch size is 1 for example\nmetrics.add_percent('MemoryUtilization', utilization_percent, None, dimensions)\n```\n\n### Add Counter based metrics\n\nPercentage based metrics can be added by invoking the following method\n\nFunction API\n```python\n def add_counter(name, value, idx=None, dimensions=None):\n \"\"\"\n Add a counter metric or increment an existing counter metric\n\n Parameters\n ----------\n name : str\n metric name\n value: int\n value of metric\n idx: int\n request_id index in batch\n dimensions: list\n list of dimensions for the metric\n \"\"\"\n```\n\nTo create , increment and decrement counter based metrics we can use the following calls\n```python\n# Add Loop Count as a metric\n# dimensions = [dim1, dim2, dim3, ..., dimN]\n# Assuming batch size is 1 for example\n\n# Create a counter with name 'LoopCount' and dimensions, initial value\nmetrics.add_counter('LoopCount', 1, None, dimensions)\n\n# Increment counter by 2 \nmetrics.add_counter('LoopCount', 2 , None, dimensions)\n\n# Decrement counter by 1\nmetrics.add_counter('LoopCount', -1, None, dimensions)\n\n# Final counter value in this case is 2\n\n```\n" }, { "path": "docs/migration.md", "content": "# Migration from MMS 0.4\n\nMMS 1.0 is a major release that contains significant architecture improvement based on MMS 0.4.\n\nMMS 1.0 adopted micro-services based architecture, the frontend request handler is separated from backend inference worker. The frontend is a java based web service which provide REST API, and backend is python based worker which execute custom service code (Other language worker support is also planed).\n\n## Table of Content\n\n* [Installation](#installation)\n* [Command line interface](#command-line-interface)\n* [API](#api)\n* [Model archive](#model-archive)\n* [Docker container](#docker-container)\n* [Logging](#logging)\n* [Metrics](#metrics)\n* [Configuration](#configuration)\n * [SSL](#ssl)\n\n## Installation\n\nMMS 1.0 made following changes for pip installation package:\n\n* **java 8**: java is required for MMS 1.0. You must install java 8 (or later) and make sure java is on available in $PATH environment variable *before* installing MMS. If you have multiple java installed, you can use $JAVA_HOME environment vairable to control which java to use.\n* **mxnet**: `mxnet` will not be installed by default with MMS 1.0 any more. You have to install it manually.\n\nSee more detail: [Install MMS](install.md)\n\n## Command line interface\nMMS 1.0 made some parameter changes in `mxnet-model-server` command line tool. The following command line parameters from previous versions will no longer function:\n\n* --service, See [Register model](management_api.md#register-a-model) for how to override service entry-point.\n* --gen-api, See [API description](inference_api#api-description) for how to generate your swagger client code.\n* --port, See [Configure MMS listening port](configuration.md#configure-mms-listening-port) for how to configure MMS listening ports.\n* --host, See [Configure MMS listening port](configuration.md#configure-mms-listening-port) for how to bind to specific network interface.\n* --gpu, See [Config properties](configuration.md#other-properties) for how to limit the number of GPUs.\n* --log-file, See [Logging](#logging) for how to specify a log file.\n* --log-rotation-time, See [Logging](#logging) for how to configure log rotation.\n* --log-level, See [Logging](#logging) for how to configure log level.\n* --metrics-write-to, See [Metrics](#metrics) for how to configure metrics.\n\nFor further information on the parameters' updates, please see the [Command Line Interface](server.md#command-line-interface) section of the server documentation.\n\n## API\nYou can continue to use MMS 0.4 inference API in MMS 1.0. However they are deprecated. Please migrate to new [inference API](inference_api.md)\n\n## Model archive\nYou can continue to use your existing MMS 0.4 model archive (`.model` file). We stronger recommend you to migrate to new Model archive (`.mar`) format.\n\nPlease refer to following documents:\n* [Custom service code](custom_service.md)\n* [model-archiver tool](../model-archiver/README.md)\n* [Create model archive example](../examples/mxnet_vision/README.md)\n\n### model-archiver\n`mxnet-model-export` is no longer supported. Instead we release a `model-archiver` CLI tool. `model-archiver` now can be installed standalone:\n\n```bash\npip install model-archiver\n```\nSee [model-archiver](../model-archiver/README.md) for more detail.\n\n## Docker container\n\nMMS docker image makes it easier for you to serve a model. In 0.4 release, MMS require a configuration file (mms_app_cpu.conf or mms_app_gpu.conf) to start MMS in docker container. The old conf file format is no longer supported. To make it simple, MMS no longer requires the --mms-config parameter, the default configuration should work for most of use cases. MMS will start automatically while docker container starts:\n\n```bash\ndocker run -itd --name mms -p 80:8080 -p 8081:8081 awsdeeplearningteam/multi-model-server\n```\n\nAfter docker container started, you can use [Management API](management_api.md) to load models for inference.\n\nSee [Docker Image](../docker/README.md) for detail.\n\n## Logging\n\nMMS 1.0 provides highly customizable logging feature. MMS 0.4 logging parameter (--log-file, , --log-rotation-time and --log-level) in command line is not supported.\n\nFor more detail see [Logging configuration](logging.md)\n\n## Metrics\n\nMMS 1.0 redesigned metrics feature:\n* The old --metrics-write-to parameter is not supported, instead a rich configuration is provided.\n* The built-in ClouldWatch integration is removed, instead MMS 1.0 provide a template allows user to integrated with any metrics server.\n\nSee [Metrics](metrics.md) for more detail.\n\n## Configuration\n\nMMS 1.0 provide a rich set of configuration parameters allow advanced user to customize/tune MMS. A completely new set of parameters are introduced in new config.properties file. The MMS 0.4 format of configuration file is not supported any more.\n\nSee [Advanced configuration](configuration.md) for more detail.\n\n### SSL\n\nMMS 0.4 support SSL via nginx, now MMS 1.0 provide native SSL support. See [Enable SSL](configuration.md#enable-ssl) for detail.\n" }, { "path": "docs/mms_endpoint_plugins.md", "content": "# Introduction \nIn this document, we will go over how to build and load custom endpoints for MMS. We will go over the plugins based\narchitecture for MMS and user experience.\n\n# Plugins SDK\nMMS currently provides an SDK for customers to develop their custom URL endpoints and drop those endpoints into\nMMS for custom URL handling. The SDK is currently published through Maven Central. Let's go over what is \navailable in the SDK and how we could use this SDK to build our custom Endpoint.\n\n# Plugins SDK\nThe Model Server plugins SDK is distributed through Maven Central. \nFind it on [Nexus Repository Manager](https://oss.sonatype.org/#nexus-search;quick~software.amazon.ai)\nThe plugins SDK has multiple components. The following are the main classes:\n1. **ModelServerEndpoint** - This is the main class used to create a custom endpoint.\n2. **Context** - This contains the context for model-server. Context object contains methods to read and modify the \nbehavior of model-server.\n3. **Worker** - This object contains all the information pertaining to a worker\n\n# Plugins architecture\n### MMS plugins loading at startup\nAt startup, MMS reads the configuration file to get the plugins directory. This plugins directory\ncontains endpoint jars. MMS loads all the jars implementing endpoints.\n\n![](images/mms_plugins_startup.jpg)\n\nAs seen in the above diagram, MMS loads all the endpoint \"jars\" implementing \"ModelServerEndpoint\"\nand registers them to \"Management\" or \"Inference\" channels. \n\n### MMS plugins at runtime\nWhen the client invokes \"endpoint_1\", which was loaded at startup time above, MMS looks up the registered\nendpoints and validates if the request is valid. If the request is valid, the custom endpoint\nis scheduled in a separate thread-pool. Once the custom endpoint finishes running, the output\nis sent back to the client by the thread pool.\n\n![](images/mms_plugins_runtime.jpg)\n\n## Class diagram of the MMS custom-endpoint SDK:\n![](images/plugins_sdk_class_diagram.png)\n\n# Writing your own custom endpoint\nIn this section we will go cover how we could develop our own endpoint and test the endpoint with MMS.\nWe will be developing an endpoint called \"GET /execution-parameters\" endpoint, which will return a set of \nconfiguration parameters such as \"MAX_CONCURRENT_TRANSFORMS\", \"BATCH_STRATEGY\", \"MAX_PAYLOAD_IN_MB\",\n\"BATCH\".\n\n## Include Maven dependencies for your project\n\nIn the Maven dependency section, include the following to get the plugins-sdk for MMS.\n\n```xml\n\n software.amazon.ai\n mms-plugins-sdk\n 1.0.1\n\n```\n\nThis will bring in Plugins SDK for MMS into your project.\n\nNow we are ready to build our custom endpoint!\n\n## Build endpoint and register the endpoint as service\nWe create a project with a java file called \"ExecutionParameters\" and also \ncreate a \"META-INF/services/software.amazon.ai.mms.servingsdk.ModelServerEndpoint\" file which acts as registration \nfor this service. Lets look into what goes into each of these folders.\n\n### Project structure\n![](images/project_structure.png)\n\n### ExecutionParameters \n\nHere we define the the behavior of the endpoint as following\n```java\n@Endpoint(\n urlPattern = \"execution-parameters\",\n endpointType = EndpointTypes.INFERENCE,\n description = \"Execution parameters endpoint\")\npublic class ExecutionParameters extends ModelServerEndpoint {\n @Override\n public void doGet(Request req, Response rsp, Context ctx) throws IOException {\n Properties prop = ctx.getConfig();\n HashMap r = new HashMap<>();\n r.put(\"MAX_CONCURRENT_TRANSFORMS\", prop.getProperty(\"NUM_WORKERS\", \"1\"));\n r.put(\"BATCH_STRATEGY\", \"SINGLE_RECORD\");\n r.put(\"MAX_PAYLOAD_IN_MB\", prop.getProperty(\"max_request_size\"));\n r.put(\"BATCH\", \"true\");\n rsp.getOutputStream()\n .write(\n new GsonBuilder()\n .setPrettyPrinting()\n .create()\n .toJson(r)\n .getBytes(StandardCharsets.UTF_8));\n }\n}\n```\n\nHere we have annotated the class with the name of the URL as \"execution-parameters\" and the type of the \nendpoint as \"INFERENCE\". This endpoint only implements a \"doGet\" method. In other words, this endpoint only\nsupports \"GET\" method. The endpoint also returns the output by writing it to the output stream. \n\n### Service file software.amazon.ai.mms.servingsdk.ModelServerEndpoint\nThe service file is used for service loading on the MMS. The contents of this file are as follows\n\n```text\nsoftware.amazon.ai.mms.plugins.endpoint.ExecutionParameters\n``` \n \nAs the contents show, when loading this JAR, MMS loads the \"ExecutionParameters\" class, which in turn defines\nthe functionality of \"execution-parameters\" endpoint.\n\n\n### Getting the endpoint jar\nOnce the project is built, the generated JAR is the custom endpoint jar. The project in this example builds\n\"**execution-parameters.jar**\" . \n\n# Loading the custom endpoint on MMS\n* Lets place this JAR in \"/tmp/plugins\" directory\n```bash\n$ ls /tmp/plugins\nexecution-parameters.jar\n``` \n* Configure MMS to load all the endpoints in plugins directory.\nContents of **config.properties** are:\n```properties\nplugins_path=/tmp/plugins\n```\n* Start model server with this configuration file\n```bash\nmulti-model-server --start --mms-config config.properties\n```\n* MMS will load the endpoints configured in this directory and be ready to serve requests to this endpoint.\n\n* Test the endpoint\n```bash\ncurl -v 4 localhost:8080/execution-parameters\n* TCP_NODELAY set\n* Connected to localhost (::1) port 8080 (#1)\n> GET /execution-parameters HTTP/1.1\n> Host: localhost:8080\n> User-Agent: curl/7.54.0\n> Accept: */*\n>\n< HTTP/1.1 200 OK\n< x-request-id: de5b2255-33ff-4d75-bed4-b24eb7820dec\n< Pragma: no-cache\n< Cache-Control: no-cache; no-store, must-revalidate, private\n< Expires: Thu, 01 Jan 1970 00:00:00 UTC\n< content-length: 94\n< connection: keep-alive\n<\n{\n \"BATCH_STRATEGY\": \"SINGLE_RECORD\",\n \"MAX_CONCURRENT_TRANSFORMS\": \"1\",\n \"BATCH\": \"true\"\n}\n* Connection #1 to host localhost left intact\n```\n\n## Conclusion\n\nFor more implementations of the custom endpoints, please refer the [plugins](../plugins) repository. " }, { "path": "docs/mms_on_fargate.md", "content": "# Serverless Inference with MMS on FARGATE\n\nThis is self-contained step by step guide that shows how to create launch and server your deep learning models with MMS \nin a production setup. \nIn this document you will learn how to launch MMS with AWS Fargate, in order to achieve a serverless inference.\n\n## Prerequisites\n\nEven though it is fully self-contained we do expect the reader to have some knowledge about the following topics:\n\n* [MMS](https://github.com/awslabs/multi-model-server)\n* [What is Amazon Elastic Container Service (ECS)](https://aws.amazon.com/ecs)\n* [What is Fargate](https://aws.amazon.com/fargate)\n* [What is Docker](https://www.docker.com/) and how to use containers\n\nSince we are doing inference, we need to have a pre-trained model that we can use to run inference. \nFor the sake of this article, we will be using \n[SqueezeNet model](https://github.com/awslabs/multi-model-server/blob/master/docs/model_zoo.md#squeezenet_v1.1). \nIn short, SqueezeNet is a model that allows you to recognize objects in a picture. \n\nNow that we have the model chosen, let's discuss at a high level what our pure-container based solution will look like:\n\n![architecture](https://s3.amazonaws.com/multi-model-server/mms-github-docs/MMS+with+Fargate+Article/AWS+Fargate+MMS.jpg)\n\nIn this document we are going to walk you through all the steps of setting up MMS 1.0 on Amazon Fargate services.\nThe steps in this process are as follows:\n\n1. Familiarize yourself with MMS containers\n2. Create a SqueezeNet task definition (with the docker container of MMS) \n3. Create AWS Fargate cluster\n4. Create Application Load Balancer\n5. Create Squeezenet Fargate service on the cluster\n6. Profit!\n\nLet the show begin...\n\n## Familiarize Yourself With Our Containers \n\nWith the current release of [MMS, 1.0](https://github.com/awslabs/multi-model-server/releases/tag/v1.0.0), \nOfficial pre-configured, optimized container images of MMS are provided on [Docker hub](https://hub.docker.com).\n\n* [awsdeeplearningteam/multi-model-server](https://hub.docker.com/r/awsdeeplearningteam/multi-model-server)\n\n```bash\ndocker pull awsdeeplearningteam/multi-model-server\n\n# for gpu image use following command:\ndocker pull awsdeeplearningteam/multi-model-server:latest-gpu\n```\nIn our article we are going to use the official CPU container image.\n\nOne major constraint for using Fargate service is that there is currently no support for GPU on Fargate.\n\nThe model-server container comes with a configuration file pre-baked inside the container.\nIt is highly recommended that you understand all the parameters of the MMS configuration file.\nFamiliarize yourself with the \n[MMS configuration](https://github.com/awslabs/multi-model-server/blob/master/docs/configuration.md) and \n[configuring MMS Container docs](https://github.com/awslabs/multi-model-server/blob/master/docker/README.md).\nWhen you want to launch and host your custom model, you will have to update this configuration. \n\nIn this tutorial, we will be use the squeezenet model from the following S3 link.\n\n```\nhttps://s3.amazonaws.com/model-server/model_archive_1.0/squeezenet_v1.1.mar\n```\n\nSince MMS can consume model files from S3 buckets, we wouldn't need to bake the containers with the actual model files.\n\nThe last question that we need to address: how we should be starting our MMS within our container. \nAnd the answer is very simple, you just need to set the following \n[ENTRYPOINT](https://docs.docker.com/engine/reference/builder/#entrypoint):\n\n```bash\nmulti-model-server --start --models https://s3.amazonaws.com/model-server/model_archive_1.0/squeezenet_v1.1.mar\n```\n\nYou will now have a running container serving squeezenet model.\n\nAt this point, you are ready to start creating actual task definition.\n\n**Note**: To start multiple models with the model-server, you could run the following command with multiple model names\n```bash\n# Example, following command starts model server with Resnet-18 and Squeezenet V1 models\n$ multi-model-server --start --models https://s3.amazonaws.com/model-server/model_archive_1.0/squeezenet_v1.1.mar https://s3.amazonaws.com/model-server/model_archive_1.0/resnet-18.mar\n\n```\n\n\n## Create an AWS Fargate task to serve SqueezeNet model\n\nThis is the first step towards getting your own \"inference service\" up and running in a production setup. \n\n1. Login to the AWS console and go to the Elastic Cloud Service -> Task Definitions and Click “Create new Task Definition”:\n\n![task def](https://s3.amazonaws.com/mms-github-assets/MMS+with+Fargate+Article/1_Create_task_definition.png)\n\n2. Now you need to specify the type of the task, you will be using the Fargate task:\n\n![](https://s3.amazonaws.com/mms-github-assets/MMS+with+Fargate+Article/2_Select_Fargate.png)\n\n3. The task requires some configuration, let's look at it step by step. First set the name:\n\n![](https://s3.amazonaws.com/mms-github-assets/MMS+with+Fargate+Article/3_Config_1.png)\n\nNow is important part, you need to create a [IAM role](https://aws.amazon.com/iam) that will be used to publish metrics to CloudWatch:\n\n![](https://s3.amazonaws.com/mms-github-assets/MMS+with+Fargate+Article/Task+Execution+IAM+Role+.png)\n\nThe containers are optimized for 8 vCPUs, however in this example you are going to use slightly smaller task with 4 vCPUs and 8 GB of RAM:\n\n![](https://s3.amazonaws.com/mms-github-assets/MMS+with+Fargate+Article/cpu+and+ram.png)\n\n4. Now it is time to configure the actual container that the task should be executing.\n\n![](https://s3.amazonaws.com/mms-github-assets/MMS+with+Fargate+Article/container+step+1.png)\n

\n*Note:* If you are using a [custom container](https://github.com/awslabs/multi-model-server/blob/master/docs/mms_on_fargate.md#customize-the-containers-to-serve-your-custom-deep-learning-models), make sure to first upload your container to Amazon ECR or Dockerhub and replace the link in this step with the link to your uploaded container.\n\n5. The next task is to specify the port mapping. You need to expose container port 8080. \nThis is the port that the MMS application inside the container is listening on. \nIf needed it can be configured via the config [here](https://github.com/awslabs/multi-model-server/blob/master/docker/mms_app_cpu.conf#L40).\n\n![](https://s3.amazonaws.com/mms-github-assets/MMS+with+Fargate+Article/port+8080.png)\n\nNext, you will have to configure the health-checks. This is the command that ECS should run to find out whether MMS is running within the container or not. MMS has a pre-configured endpoint `/ping`\nthat can be used for health checks. Configure ECS to reach that endpoint at `http://127.0.0.1:8080/ping` using the `curl` command as shown below:\n\n```bash\ncurl, http://127.0.0.1:8080/ping\n```\n\nThe healthcheck portion of your container configuration should look like the image below:\n\n![](https://s3.amazonaws.com/multi-model-server/mms-github-docs/MMS+with+Fargate+Article/add+container+healthcheck.png)\n\nAfter configuring the health-checks, you can go onto configuring the environment, with the entry point that we have discussed earlier:\n\n![](https://s3.amazonaws.com/multi-model-server/mms-github-docs/MMS+with+Fargate+Article/environtment.png)\n\nEverything else can be left as default. So feel free to click `Create` to create your very first AWS Fargate-task. \nIf everything is ok, you should now be able to see your task in the list of task definitions.\n\nIn ECS, `Services` are created to run Tasks. A service is in charge of \nrunning multiple tasks and making sure the that required number of tasks are always running, \nrestarting un-healthy tasks, adding more tasks when needed. \n \n To have your `inference service` accessible over the Internet, you would need to configure a load-balancer (LB). This LB will be \n in charge of serving the traffic from the Internet and redirecting it to these newly created tasks. \n Let's create an Application Load Balancer now:\n\n## Create a Load Balancer\n\nAWS supports several different types of Load Balancers:\n\n* Application Load Balancer: works on the level 7 of the OSI model (effectively with the HTTP/HTTPS protocols)\n* TCP Load Balancer \n\nFor your cluster you are going to use application load balancer.\n1. Login to the EC2 Console.\n2. Go to the “Load balancers” section.\n3. Click on Create new Load Balancer.\n\n![](https://s3.amazonaws.com/mms-github-assets/MMS+with+Fargate+Article/1__Create_Load_Balancer.png)\n\n5. Choose Application Load Balancer.\n\n![](https://s3.amazonaws.com/mms-github-assets/MMS+with+Fargate+Article/2__HTTP_HTTPS+.png)\n\n6. Set all the required details. **Make a note of the VPC of the LB**. This is important since the LB's VPC and the ECS\ncluster's VPC need to be same for them to communicate with each other.\n\n![](https://s3.amazonaws.com/mms-github-assets/MMS+with+Fargate+Article/3_2_Listeners_and_AZ+.png)\n\n7. Next is configuring the security group. This is also important. Your security group should:\n\n* Allow inbound connections for port 80 (since this is the port on which LB will be listening on)\n* LBs security group needs to be added to the AWS Fargate service's security group, so that all the traffic from LB is accepted\nby your \"inference service\". \n\n![](https://s3.amazonaws.com/mms-github-assets/MMS+with+Fargate+Article/4.+Configure+Security+groups.png)\n\n8. Routing configuration is simple. Here you need to create a “target group”. \nBut, in your case the AWS Fargate service, that you will create later, will automatically create a target group. \nTherefore you will create dummy “target group” that you will delete after the creation of the LB. \n\n![](https://s3.amazonaws.com/mms-github-assets/MMS+with+Fargate+Article/5.+Configure+Routing+(DUmmy).png)\n\n9. Nothing needs to be done for the last two steps. `Finish` the creation and ...\n10. Now you are ready to remove dummy listener and target group\n\n![](https://s3.amazonaws.com/mms-github-assets/MMS+with+Fargate+Article/8__Delete_the_dummy_listener.png)\n\nNow that you are `done-done-done` with the Load Balancer creation, lets move onto creating our Serverless inference service.\n\n## Creating an ECS Service to launch our AWS Fargate task\n\n1. Go to Elastic Container Service → Task Definitions and select the task definitions name. Click on actions and select create service.\n\n![](https://s3.amazonaws.com/mms-github-assets/MMS+with+Fargate+Article/1.+Go+to+task+definitions.png)\n\n2. There are two important things on the first step (apart from naming):\n\n* Platform version: It should be set to 1.1.0 .\n* Number of tasks that the service should maintain as healthy all of the time, for this example you will set this to 3.\n\n![](https://s3.amazonaws.com/mms-github-assets/MMS+with+Fargate+Article/number+of+tasks.png)\n\n3. Now it is time to configure the VPC and the security group. **You should use the same VPC that was used for the LB (and same subnets!).**\n\n![](https://s3.amazonaws.com/mms-github-assets/MMS+with+Fargate+Article/3.2.1+Use+the+existing+VPC+Edit+sg.png)\n\n4. As for the security group, it should be either the same security group as you had for the LB, or the one that accepts traffic from the LBs security group.\n\n![](https://s3.amazonaws.com/mms-github-assets/MMS+with+Fargate+Article/3.2.2+SG+Use+existing.png)\n\n5. Now you can connect your service to the LB that was created in the previous section. Select the \"Application Load Balancer\" and set the LB name:\n\n![](https://s3.amazonaws.com/mms-github-assets/MMS+with+Fargate+Article/3.2.3+Add+load+balancing.png)\n\n6. Now you need to specify which port on the LB our service should be listening on:\n\n![](https://s3.amazonaws.com/mms-github-assets/MMS+with+Fargate+Article/3.2.4+Configure+load+blancer.png)\n\n7. You are not going to use service discovery now, so uncheck it:\n\n![](https://s3.amazonaws.com/mms-github-assets/MMS+with+Fargate+Article/3.2.5+Next.png)\n\n8. In this document, we are not using auto-scaling options. For an actual production system, it is advisable to have this configuration setup.\n\n![](https://s3.amazonaws.com/mms-github-assets/MMS+with+Fargate+Article/3.3+Auto+scaling.png)\n\n9. Now you are `done-done-done` creating a running service. You can move to the final chapter of the journey, which is testing the service you created. \n\n## Test your service\n\nFirst find the DNS name of your LB. It should be in `AWS Console -> Service -> EC2 -> Load Balancers` and click on the LB that you created.\n\n![](https://s3.amazonaws.com/mms-github-assets/MMS+with+Fargate+Article/lb_dns.png)\n\nNow you can run the health checks using this load-balancer public DNS name, to verify that your newly created service is working:\n\n```bash\ncurl InfraLb-1624382880.us-east-1.elb.amazonaws.com/ping \n```\n\n```text\nhttp://infralb-1624382880.us-east-1.elb.amazonaws.com/ping\n{\n \"status\": \"Healthy!\"\n}\n```\n\nAnd now you are finally ready to run our inference! Let's download an example image:\n```bash\ncurl -O https://s3.amazonaws.com/model-server/inputs/kitten.jpg\n```\n\nThe image:\n\n![](https://s3.amazonaws.com/mms-github-assets/MMS+with+Fargate+Article/kitten.jpg)\n\nThe output of this query would be as follows,\n\n```bash\ncurl -X POST InfraLb-1624382880.us-east-1.elb.amazonaws.com/predictions/squeezenet_v1.1 -F \"data=@kitten.jpg\"\n```\n\n```text\n{\n \"prediction\": [\n [\n {\n \"class\": \"n02124075 Egyptian cat\",\n \"probability\": 0.8515275120735168\n },\n {\n \"class\": \"n02123045 tabby, tabby cat\",\n \"probability\": 0.09674164652824402\n },\n {\n \"class\": \"n02123159 tiger cat\",\n \"probability\": 0.03909163549542427\n },\n {\n \"class\": \"n02128385 leopard, Panthera pardus\",\n \"probability\": 0.006105933338403702\n },\n {\n \"class\": \"n02127052 lynx, catamount\",\n \"probability\": 0.003104303264990449\n }\n ]\n ]\n}\n```\n\n## Instead of a Conclusion\n\nThere are a few things that we have not covered here and which are very useful, such as:\n\n* How to configure auto-scaling on our ECS cluster.\n* Running A/B testing of different versions of the model with the Fargate Deployment concepts.\n\nEach of the above topics require their own articles, so stay tuned!!\n\n## Authors\n\n* Aaron Markham\n* Vamshidhar Dantu \n* Viacheslav Kovalevskyi (@b0noi) \n" }, { "path": "docs/model_zoo.md", "content": "# Model Zoo\n\nThis page lists model archives that are pre-trained and pre-packaged, ready to be served for inference with MMS.\nTo propose a model for inclusion, please submit a [pull request](https://github.com/awslabs/multi-model-server/pulls).\n\n*Special thanks to the [Apache MXNet](https://mxnet.incubator.apache.org) community whose Model Zoo and Model Examples were used in generating these model archives.*\n\n\n| Model File | Type | Dataset | Source | Size | Download |\n| --- | --- | --- | --- | --- | --- |\n| [AlexNet](#alexnet) | Image Classification | ImageNet | ONNX | 233 MB | [.mar](https://s3.amazonaws.com/model-server/model_archive_1.0/alexnet.mar) |\n| [ArcFace-ResNet100](#arcface-resnet100_onnx) | Face Recognition | Refined MS-Celeb1M | ONNX | 236.4 MB | [.mar](https://s3.amazonaws.com/model-server/model_archive_1.0/onnx-arcface-resnet100.mar) |\n| [Character-level Convolutional Networks for Text Classification](#crepe) | Text Classification | [Amazon Product Data](http://jmcauley.ucsd.edu/data/amazon/) | Gluon | 40 MB | [.mar](https://s3.amazonaws.com/model-server/model_archive_1.0/crepe.mar) |\n| [CaffeNet](#caffenet) | Image Classification | ImageNet | MXNet | 216 MB | [.mar](https://s3.amazonaws.com/model-server/model_archive_1.0/caffenet/caffenet.mar) |\n| [FERPlus](#ferplus_onnx) | Emotion Detection | FER2013 | ONNX | 35MB | [.mar](https://s3.amazonaws.com/model-server/model_archive_1.0/FERPlus.mar) |\n| [Inception v1](#inception_v1) | Image Classification | ImageNet | ONNX | 27 MB | [.mar](https://s3.amazonaws.com/model-server/model_archive_1.0/onnx-inception_v1.mar) |\n| [Inception v3 w/BatchNorm](#inception_v3) | Image Classification | ImageNet | MXNet | 45 MB | [.mar](https://s3.amazonaws.com/model-server/model_archive_1.0/inception-bn.mar) |\n| [LSTM PTB](#lstm-ptb) | Language Modeling | PennTreeBank | MXNet | 16 MB | [.mar](https://s3.amazonaws.com/model-server/model_archive_1.0/lstm_ptb.mar) |\n| [MobileNetv2-1.0](#mobilenetv2-1.0_onnx) | Image Classification | ImageNet | ONNX | 13.7 MB | [.mar](https://s3.amazonaws.com/model-server/model_archive_1.0/onnx-mobilenet.mar) |\n| [Network in Network (NiN)](#nin) | Image Classification | ImageNet | MXNet | 30 MB | [.mar](https://s3.amazonaws.com/model-server/model_archive_1.0/nin.mar) |\n| [ResNet-152](#resnet-152) | Image Classification | ImageNet | MXNet | 241 MB | [.mar](https://s3.amazonaws.com/model-server/model_archive_1.0/resnet-152.mar) |\n| [ResNet-18](#resnet-18) | Image Classification | ImageNet | MXNet | 43 MB | [.mar](https://s3.amazonaws.com/model-server/model_archive_1.0/resnet-18.mar) |\n| [ResNet50-SSD](#resnet50-ssd) | SSD (Single Shot MultiBox Detector) | ImageNet | MXNet | 124 MB | [.mar](https://s3.amazonaws.com/model-server/model_archive_1.0/resnet50_ssd.mar) |\n| [ResNext101-64x4d](#resnext101) | Image Classification | ImageNet | MXNet | 334 MB | [.mar](https://s3.amazonaws.com/model-server/model_archive_1.0/resnext-101-64x4d.mar) |\n| [ResNet-18v1](#resnet-18v1) | Image Classification | ImageNet | ONNX | 45 MB | [.mar](https://s3.amazonaws.com/model-server/model_archive_1.0/onnx-resnet18v1.mar) |\n| [ResNet-34v1](#resnet-34v1) | Image Classification | ImageNet | ONNX | 83 MB | [.mar](https://s3.amazonaws.com/model-server/model_archive_1.0/onnx-resnet34v1.mar) |\n| [ResNet-50v1](#resnet-50v1) | Image Classification | ImageNet | ONNX | 98 MB | [.mar](https://s3.amazonaws.com/model-server/model_archive_1.0/onnx-resnet50v1.mar) |\n| [ResNet-101v1](#resnet-101v1) | Image Classification | ImageNet | ONNX | 171 MB | [.mar](https://s3.amazonaws.com/model-server/model_archive_1.0/onnx-resnet101v1.mar) |\n| [ResNet-152v1](#resnet-152v1) | Image Classification | ImageNet | ONNX | 231 MB | [.mar](https://s3.amazonaws.com/model-server/model_archive_1.0/onnx-resnet152v1.mar) |\n| [ResNet-18v2](#resnet-18v2) | Image Classification | ImageNet | ONNX | 45 MB | [.mar](https://s3.amazonaws.com/model-server/model_archive_1.0/onnx-resnet18v2.mar) |\n| [ResNet-34v2](#resnet-34v2) | Image Classification | ImageNet | ONNX | 83 MB | [.mar](https://s3.amazonaws.com/model-server/model_archive_1.0/onnx-resnet34v2.mar) |\n| [ResNet-50v2](#resnet-50v2) | Image Classification | ImageNet | ONNX | 98 MB | [.mar](https://s3.amazonaws.com/model-server/model_archive_1.0/onnx-resnet50v2.mar) |\n| [ResNet-101v2](#resnet-101v2) | Image Classification | ImageNet | ONNX | 171 MB | [.mar](https://s3.amazonaws.com/model-server/model_archive_1.0/onnx-resnet101v2.mar) |\n| [ResNet-152v2](#resnet-152v2) | Image Classification | ImageNet | ONNX | 231 MB | [.mar](https://s3.amazonaws.com/model-server/model_archive_1.0/onnx-resnet152v2.mar) |\n| [Shufflenet](#shufflenet) | Image Classification | ImageNet | ONNX | 8.1 MB | [.mar](https://s3.amazonaws.com/model-server/model_archive_1.0/shufflenet.mar) |\n| [SqueezeNet_v1.1](#squeezenet_v1.1_onnx) | Image Classification | ImageNet | ONNX | 5 MB | [.mar](https://s3.amazonaws.com/model-server/model_archive_1.0/onnx-squeezenet.mar) |\n| [SqueezeNet v1.1](#squeezenet_v1.1) | Image Classification | ImageNet | MXNet | 5 MB | [.mar](https://s3.amazonaws.com/model-server/model_archive_1.0/squeezenet_v1.1.mar) |\n| [VGG16](#vgg16) | Image Classification | ImageNet | MXNet | 490 MB | [.mar](https://s3.amazonaws.com/model-server/model_archive_1.0/vgg16.mar) |\n| [VGG16](#vgg16_onnx) | Image Classification | ImageNet | ONNX | 527 MB | [.mar](https://s3.amazonaws.com/model-server/model_archive_1.0/onnx-vgg16.mar) |\n| [VGG16_bn](#vgg16_bn_onnx) | Image Classification | ImageNet | ONNX | 527 MB | [.mar](https://s3.amazonaws.com/model-server/model_archive_1.0/onnx-vgg16_bn.mar) |\n| [VGG19](#vgg19) | Image Classification | ImageNet | MXNet | 509 MB | [.mar](https://s3.amazonaws.com/model-server/model_archive_1.0/vgg19.mar) |\n| [VGG19](#vgg19_onnx) | Image Classification | ImageNet | ONNX | 548 MB | [.mar](https://s3.amazonaws.com/model-server/model_archive_1.0/onnx-vgg19.mar) |\n| [VGG19_bn](#vgg19_bn_onnx) | Image Classification | ImageNet | ONNX | 548 MB | [.mar](https://s3.amazonaws.com/model-server/model_archive_1.0/onnx-vgg19_bn.mar) |\n\n\n## Details on Each Model\nEach model below comes with a basic description, and where available, a link to a scholarly article about the model.\n\nMany of these models use a kitten image to test inference. Use the following to get one that will work:\n```bash\ncurl -O https://s3.amazonaws.com/model-server/inputs/kitten.jpg\n```\n\n\n## AlexNet\n* **Type**: Image classification trained on ImageNet\n\n* **Reference**: [Krizhevsky, et al.](http://papers.nips.cc/paper/4824-imagenet-classification-with-deep-convolutional-neural-networks)\n\n* **Model Service**: [mxnet_vision_service.py](https://github.com/awslabs/multi-model-server/blob/master/examples/mxnet_vision/mxnet_vision_service.py)\n\n* **Start Server**:\n```bash\nmulti-model-server --start --models alexnet=https://s3.amazonaws.com/model-server/model_archive_1.0/alexnet.mar\n```\n\n* **Run Prediction**:\n```bash\ncurl -X POST http://127.0.0.1:8080/predictions/alexnet -T kitten.jpg\n```\n\n## ArcFace-ResNet100 (from ONNX model zoo)\n* **Type**: Face Recognition model trained on refined MS-Celeb1M dataset (model imported from ONNX)\n\n* **Reference**: [Deng et al.](https://arxiv.org/abs/1801.07698)\n\n* **Model Service**:\n * [arcface_service.py](https://s3.amazonaws.com/model-server/model_archive_1.0/examples/onnx-arcface-resnet100/arcface_service.py)\n * [mtcnn_detector.py](https://s3.amazonaws.com/model-server/model_archive_1.0/examples/onnx-arcface-resnet100/mtcnn_detector.py)\n\n* **Install dependencies**:\n```bash\npip install opencv-python\npip install scikit-learn\npip install easydict\npip install scikit-image\npip install numpy\n```\n\n* **Start Server**:\n```bash\nmulti-model-server --start --models arcface=https://s3.amazonaws.com/model-server/model_archive_1.0/onnx-arcface-resnet100.mar\n```\n\n* **Get two test images**:\n```bash\ncurl -O https://s3.amazonaws.com/model-server/inputs/arcface-input1.jpg\n\ncurl -O https://s3.amazonaws.com/model-server/inputs/arcface-input2.jpg\n```\n\n\n* **Run Prediction**:\n```bash\ncurl -X POST http://127.0.0.1:8080/predictions/arcface -F \"img1=@arcface-input1.jpg\" -F \"img2=@arcface-input2.jpg\"\n```\n\n## CaffeNet\n* **Type**: Image classification trained on ImageNet\n\n* **Reference**: [Krizhevsky, et al.](http://papers.nips.cc/paper/4824-imagenet-classification-with-deep-convolutional-neural-networks)\n\n* **Model Service**: [mxnet_vision_service.py](https://github.com/awslabs/multi-model-server/blob/master/examples/mxnet_vision/mxnet_vision_service.py)\n\n* **Start Server**:\n```bash\nmulti-model-server --start --models caffenet=https://s3.amazonaws.com/model-server/model_archive_1.0/caffenet.mar\n```\n\n* **Run Prediction**:\n```bash\ncurl -X POST http://127.0.0.1:8080/predictions/caffenet -T kitten.jpg\n```\n\n## Character-level Convolutional Networks for text Classification\n* **Type**: Character-level Convolutional network for text classification trained on [Amazon Product Data](http://jmcauley.ucsd.edu/data/amazon/).\n\n* **Reference**: [R. He, J. McAuley et al.](https://arxiv.org/abs/1602.01585), [J. McAuley, C. Targett, J. Shi, A. van den Hengel et al.](https://arxiv.org/abs/1506.04757) \n\n* **Model Service**: [gluon_crepe.py](https://github.com/awslabs/multi-model-server/blob/master/examples/gluon_character_cnn/gluon_crepe.py)\n\n* **Start Server**:\n```bash\nmulti-model-server --start --models crepe=https://s3.amazonaws.com/model-server/model_archive_1.0/crepe.mar\n```\n\n* **Run Prediction**:\n```bash\ncurl -X POST http://127.0.0.1:8080/predictions/crepe -F \"data=[{\\\"review_title\\\":\\\"Inception is the best\\\",\\\"review\\\": \\\"great direction and story\\\"}]\"\n```\n\n## DUC-ResNet101 (from ONNX model zoo)\n* **Type**: Semantic Segmentation model trained on the Cityscapes dataset (model imported from ONNX)\n\n* **Reference**: [Wang et al.](https://arxiv.org/abs/1702.08502)\n\n* **Model Service**:\n * [duc_service.py](https://s3.amazonaws.com/model-server/model_archive_1.0/examples/onnx-duc/duc_service.py)\n * [cityscapes_labels.py](https://s3.amazonaws.com/model-server/model_archive_1.0/examples/onnx-duc/cityscapes_labels.py)\n\n* **Install dependencies**:\n```bash\npip install opencv-python\npip install pillow\n```\n\n* **Start Server**:\n```bash\nmulti-model-server --models duc=https://s3.amazonaws.com/model-server/model_archive_1.0/onnx-duc.mar\n```\n\n* **Get the test image**:\n```bash\ncurl -O https://s3.amazonaws.com/multi-model-server/onnx-duc/city1.jpg\n```\n\n* **Download inference script**:\n\nThe script makes an inference call to the server using the test image, displays the colorized segmentation map and prints the confidence score.\n```bash\ncurl -O https://s3.amazonaws.com/multi-model-server/onnx-duc/duc-inference.py\n```\n\n* **Run Prediction**:\n```bash\npython duc-inference.py city1.jpg\n```\n\n## FERPlus\n* **Type**: Emotion detection trained on FER2013 dataset (model imported from ONNX)\n\n* **Reference**: [Barsoum et al.](https://arxiv.org/abs/1608.01041)\n\n* **Model Service**: [emotion_detection_service.py](https://s3.amazonaws.com/model-server/model_archive_1.0/examples/FERPlus/emotion_detection_service.py)\n\n* **Start Server**:\n```bash\nmulti-model-server --start --models FERPlus=https://s3.amazonaws.com/model-server/model_archive_1.0/FERPlus.mar\n```\n\n* **Get a test image**:\n```bash\ncurl -O https://s3.amazonaws.com/model-server/inputs/ferplus-input.jpg\n```\n\n* **Run Prediction**:\n```bash\ncurl -X POST http://127.0.0.1:8080/predictions/FERPlus -T ferplus-input.jpg\n```\n\n\n## Inception v1\n* **Type**: Image classification trained on ImageNet\n\n* **Reference**: [Szegedy, et al., Google](https://arxiv.org/pdf/1512.00567.pdf)\n\n* **Model Service**: [mxnet_vision_service.py](https://github.com/awslabs/multi-model-server/blob/master/examples/mxnet_vision/mxnet_vision_service.py)\n\n* **Start Server**:\n```bash\nmulti-model-server --start --models onnx-inception-v1=https://s3.amazonaws.com/model-server/model_archive_1.0/onnx-inception_v1.mar\n```\n\n* **Run Prediction**:\n```bash\ncurl -X POST http://127.0.0.1:8080/predictions/onnx-inception-v1 -T kitten.jpg\n```\n\n\n## Inception v3\n* **Type**: Image classification trained on ImageNet\n\n* **Reference**: [Szegedy, et al., Google](https://arxiv.org/pdf/1512.00567.pdf)\n\n* **Model Service**: [mxnet_vision_service.py](https://github.com/awslabs/multi-model-server/blob/master/examples/mxnet_vision/mxnet_vision_service.py)\n\n* **Start Server**:\n```bash\nmulti-model-server --start --models inception-bn=https://s3.amazonaws.com/model-server/model_archive_1.0/inception-bn.mar\n```\n\n* **Run Prediction**:\n```bash\ncurl -X POST http://127.0.0.1:8080/predictions/inception-bn -T kitten.jpg\n```\n\n\n## LSTM PTB\nLong short-term memory network trained on the PennTreeBank dataset.\n\n* **Reference**: [Hochreiter, et al.](http://www.bioinf.jku.at/publications/older/2604.pdf)\n\n* **Model Service**: [lstm_ptb_service.py](https://github.com/awslabs/multi-model-server/blob/master/examples/lstm_ptb/lstm_ptb_service.py)\n\n* **Start Server**:\n```bash\nmulti-model-server --start --models lstm_ptb=https://s3.amazonaws.com/model-server/model_archive_1.0/lstm_ptb.mar\n```\n\n* **Run Prediction**:\n```bash\ncurl -X POST http://127.0.0.1:8080/predictions/lstm_ptb -H \"Content-Type: application/json\" -d '[{\"input_sentence\": \"on the exchange floor as soon as ual stopped trading we for a panic said one top floor trader\"}]'\n```\n\n## MobileNetv2-1.0 (from ONNX model zoo)\n* **Type**: Image classification trained on ImageNet (imported from ONNX)\n\n* **Reference**: [Sandler et al.](https://arxiv.org/abs/1801.04381)\n\n* **Model Service**: [mxnet_vision_service.py](https://s3.amazonaws.com/model-server/model_archive_1.0/mxnet_vision_service.py)\n\n* **Start Server**:\n```bash\nmulti-model-server --start --models mobilenet=https://s3.amazonaws.com/model-server/model_archive_1.0/onnx-mobilenet.mar\n```\n\n* **Run Prediction**:\n```bash\ncurl -X POST http://127.0.0.1:8080/predictions/mobilenet -T kitten.jpg\n```\n\n\n## Network in Network\n* **Type**: Image classification trained on ImageNet\n\n* **Reference**: [Lin, et al.](https://arxiv.org/pdf/1312.4400v3.pdf)\n\n* **Model Service**: [mxnet_vision_service.py](https://github.com/awslabs/multi-model-server/blob/master/examples/mxnet_vision/mxnet_vision_service.py)\n\n* **Start Server**:\n```bash\nmulti-model-server --start --models nin=https://s3.amazonaws.com/model-server/model_archive_1.0/nin.mar\n```\n\n* **Run Prediction**:\n```bash\ncurl -X POST http://127.0.0.1:8080/predictions/nin -T kitten.jpg\n```\n\n\n## ResNet-152\n* **Type**: Image classification trained on ImageNet\n\n* **Reference**: [Lin, et al.](https://arxiv.org/pdf/1312.4400v3.pdf)\n\n* **Model Service**: [mxnet_vision_service.py](https://github.com/awslabs/multi-model-server/blob/master/examples/mxnet_vision/mxnet_vision_service.py)\n\n* **Start Server**:\n```bash\nmulti-model-server --start --models resnet-152=https://s3.amazonaws.com/model-server/model_archive_1.0/resnet-152.mar\n```\n\n* **Run Prediction**:\n```bash\ncurl -X POST http://127.0.0.1:8080/predictions/resnet-152 -T kitten.jpg\n```\n\n\n## ResNet-18\n* **Type**: Image classification trained on ImageNet\n\n* **Reference**: [He, et al.](https://arxiv.org/pdf/1512.03385v1.pdf)\n\n* **Model Service**: [mxnet_vision_service.py](https://github.com/awslabs/multi-model-server/blob/master/examples/mxnet_vision/mxnet_vision_service.py)\n\n* **Start Server**:\n```bash\nmulti-model-server --start --models resnet-18=https://s3.amazonaws.com/model-server/model_archive_1.0/resnet-18.mar\n```\n\n* **Run Prediction**:\n```bash\ncurl -X POST http://127.0.0.1:8080/predictions/resnet-18 -T kitten.jpg\n```\n\n\n## ResNet50-SSD\n* **Type**: Image classification trained on ImageNet\n\n* **Reference**: [Liu, et al.](https://arxiv.org/pdf/1512.02325v4.pdf)\n\n* **Model Service**: [ssd_service.py](https://github.com/awslabs/multi-model-server/blob/master/examples/ssd/ssd_service.py)\n\n* **Start Server**:\n```bash\nmulti-model-server --start --models SSD=https://s3.amazonaws.com/model-server/model_archive_1.0/resnet50_ssd.mar\n```\n\n* **Run Prediction**:\n```bash\ncurl -O https://www.dphotographer.co.uk/users/21963/thm1024/1337890426_Img_8133.jpg\n\ncurl -X POST http://127.0.0.1:8080/predictions/SSD -T 1337890426_Img_8133.jpg\n```\n\n\n## ResNext101-64x4d\n* **Type**: Image classification trained on ImageNet\n\n* **Reference**: [Xie, et al.](https://arxiv.org/pdf/1611.05431.pdf)\n\n* **Model Service**: [mxnet_vision_service.py](https://github.com/awslabs/multi-model-server/blob/master/examples/mxnet_vision/mxnet_vision_service.py)\n\n* **Start Server**:\n```bash\nmulti-model-server --start --models resnext101=https://s3.amazonaws.com/model-server/model_archive_1.0/resnext-101-64x4d.mar\n```\n\n* **Run Prediction**:\n```bash\ncurl -X POST http://127.0.0.1:8080/predictions/resnext101 -T kitten.jpg\n```\n\n## ResNet (from ONNX model zoo)\n\n### ResNet18-v1\n* **Type**: Image classification trained on ImageNet (imported from ONNX)\n\n* **Reference**: [He, et al.](https://arxiv.org/abs/1512.03385)\n\n* **Model Service**: [mxnet_vision_service.py](https://s3.amazonaws.com/model-server/model_archive_1.0/mxnet_vision_service.py)\n\n* **Start Server**:\n```bash\nmulti-model-server --start --models resnet18-v1=https://s3.amazonaws.com/model-server/model_archive_1.0/onnx-resnet18v1.mar\n```\n\n* **Run Prediction**:\n```bash\ncurl -X POST http://127.0.0.1:8080/predictions/resnet18-v1 -T kitten.jpg\n```\n\n### ResNet34-v1\n* **Type**: Image classification trained on ImageNet (imported from ONNX)\n\n* **Reference**: [He, et al.](https://arxiv.org/abs/1512.03385)\n\n* **Model Service**: [mxnet_vision_service.py](https://s3.amazonaws.com/model-server/model_archive_1.0/mxnet_vision_service.py)\n\n* **Start Server**:\n```bash\nmulti-model-server --start --models resnet34-v1=https://s3.amazonaws.com/model-server/model_archive_1.0/onnx-resnet34v1.mar\n```\n\n* **Run Prediction**:\n```bash\ncurl -X POST http://127.0.0.1:8080/predictions/resnet34-v1 -T kitten.jpg\n```\n\n### ResNet50-v1\n* **Type**: Image classification trained on ImageNet (imported from ONNX)\n\n* **Reference**: [He, et al.](https://arxiv.org/abs/1512.03385)\n\n* **Model Service**: [mxnet_vision_service.py](https://s3.amazonaws.com/model-server/model_archive_1.0/mxnet_vision_service.py)\n\n* **Start Server**:\n```bash\nmulti-model-server --start --models resnet50-v1=https://s3.amazonaws.com/model-server/model_archive_1.0/onnx-resnet50v1.mar\n```\n\n* **Run Prediction**:\n```bash\ncurl -X POST http://127.0.0.1:8080/predictions/resnet50-v1 -T kitten.jpg\n```\n\n### ResNet101-v1\n* **Type**: Image classification trained on ImageNet (imported from ONNX)\n\n* **Reference**: [He, et al.](https://arxiv.org/abs/1512.03385)\n\n* **Model Service**: [mxnet_vision_service.py](https://s3.amazonaws.com/model-server/model_archive_1.0/mxnet_vision_service.py)\n\n* **Start Server**:\n```bash\nmulti-model-server --start --models resnet101-v1=https://s3.amazonaws.com/model-server/model_archive_1.0/onnx-resnet101v1.mar\n```\n\n* **Run Prediction**:\n```bash\ncurl -X POST http://127.0.0.1:8080/predictions/resnet101-v1 -T kitten.jpg\n```\n\n### ResNet152-v1\n* **Type**: Image classification trained on ImageNet (imported from ONNX)\n\n* **Reference**: [He, et al.](https://arxiv.org/abs/1512.03385)\n\n* **Model Service**: [mxnet_vision_service.py](https://s3.amazonaws.com/model-server/model_archive_1.0/mxnet_vision_service.py)\n\n* **Start Server**:\n```bash\nmulti-model-server --start --models resnet152-v1=https://s3.amazonaws.com/model-server/model_archive_1.0/onnx-resnet152v1.mar\n```\n\n* **Run Prediction**:\n```bash\ncurl -X POST http://127.0.0.1:8080/predictions/resnet152-v1 -T kitten.jpg\n```\n\n### ResNet18-v2\n* **Type**: Image classification trained on ImageNet (imported from ONNX)\n\n* **Reference**: [He, et al.](https://arxiv.org/abs/1603.05027)\n\n* **Model Service**: [mxnet_vision_service.py](https://s3.amazonaws.com/model-server/model_archive_1.0/mxnet_vision_service.py)\n\n* **Start Server**:\n```bash\nmulti-model-server --start --models resnet18-v2=https://s3.amazonaws.com/model-server/model_archive_1.0/onnx-resnet18v2.mar\n```\n\n* **Run Prediction**:\n```bash\ncurl -X POST http://127.0.0.1:8080/predictions/resnet18-v2 -T kitten.jpg\n```\n\n### ResNet34-v2\n* **Type**: Image classification trained on ImageNet (imported from ONNX)\n\n* **Reference**: [He, et al.](https://arxiv.org/abs/1603.05027)\n\n* **Model Service**: [mxnet_vision_service.py](https://s3.amazonaws.com/model-server/model_archive_1.0/mxnet_vision_service.py)\n\n* **Start Server**:\n```bash\nmulti-model-server --start --models resnet34-v2=https://s3.amazonaws.com/model-server/model_archive_1.0/onnx-resnet34v2.mar\n```\n\n* **Run Prediction**:\n```bash\ncurl -X POST http://127.0.0.1:8080/predictions/resnet34-v2 -T kitten.jpg\n```\n\n### ResNet50-v2\n* **Type**: Image classification trained on ImageNet (imported from ONNX)\n\n* **Reference**: [He, et al.](https://arxiv.org/abs/1603.05027)\n\n* **Model Service**: [mxnet_vision_service.py](https://s3.amazonaws.com/model-server/model_archive_1.0/mxnet_vision_service.py)\n\n* **Start Server**:\n```bash\nmulti-model-server --start --models resnet50-v2=https://s3.amazonaws.com/model-server/model_archive_1.0/onnx-resnet50v2.mar\n```\n\n* **Run Prediction**:\n```bash\ncurl -X POST http://127.0.0.1:8080/predictions/resnet50-v2 -T kitten.jpg\n```\n\n### ResNet101-v2\n* **Type**: Image classification trained on ImageNet (imported from ONNX)\n\n* **Reference**: [He, et al.](https://arxiv.org/abs/1603.05027)\n\n* **Model Service**: [mxnet_vision_service.py](https://s3.amazonaws.com/model-server/model_archive_1.0/mxnet_vision_service.py)\n\n* **Start Server**:\n```bash\nmulti-model-server --start --models resnet101-v2=https://s3.amazonaws.com/model-server/model_archive_1.0/onnx-resnet101v2.mar\n```\n\n* **Run Prediction**:\n```bash\ncurl -X POST http://127.0.0.1:8080/predictions/resnet101-v2 -T kitten.jpg\n```\n\n### ResNet152-v2\n* **Type**: Image classification trained on ImageNet (imported from ONNX)\n\n* **Reference**: [He, et al.](https://arxiv.org/abs/1603.05027)\n\n* **Model Service**: [mxnet_vision_service.py](https://s3.amazonaws.com/model-server/model_archive_1.0/mxnet_vision_service.py)\n\n* **Start Server**:\n```bash\nmulti-model-server --start --models resnet152-v2=https://s3.amazonaws.com/model-server/model_archive_1.0/onnx-resnet152v2.mar\n```\n\n* **Run Prediction**:\n```bash\ncurl -X POST http://127.0.0.1:8080/predictions/resnet152-v2 -T kitten.jpg\n```\n\n## Shufflenet_v2\n* **Type**: Image classification trained on ImageNet\n\n* **Reference**: [Zhang, et al.](https://arxiv.org/abs/1707.01083)\n\n* **Model Service**: [mxnet_vision_service.py](https://github.com/awslabs/multi-model-server/blob/master/examples/mxnet_vision/mxnet_vision_service.py)\n\n* **Start Server**:\n```bash\nmulti-model-server --start --models shufflenet=https://s3.amazonaws.com/model-server/model_archive_1.0/shufflenet.mar\n```\n\n* **Run Prediction**:\n```bash\ncurl -X POST http://127.0.0.1:8080/predictions/shufflenet -T kitten.jpg\n```\n\n## SqueezeNet v1.1\n* **Type**: Image classification trained on ImageNet\n\n* **Reference**: [Iandola, et al.](https://arxiv.org/pdf/1602.07360v4.pdf)\n\n* **Model Service**: [mxnet_vision_service.py](https://github.com/awslabs/multi-model-server/blob/master/examples/mxnet_vision/mxnet_vision_service.py)\n\n* **Start Server**:\n```bash\nmulti-model-server --start --models squeezenet_v1.1=https://s3.amazonaws.com/model-server/model_archive_1.0/squeezenet_v1.1.mar\n```\n\n* **Run Prediction**:\n```bash\ncurl -X POST http://127.0.0.1:8080/predictions/squeezenet_v1.1 -T kitten.jpg\n```\n\n## SqueezeNet v1.1 (from ONNX model zoo)\n* **Type**: Image classification trained on ImageNet (imported from ONNX)\n\n* **Reference**: [Iandola, et al.](https://arxiv.org/pdf/1602.07360v4.pdf)\n\n* **Model Service**: [mxnet_vision_service.py](https://github.com/awslabs/multi-model-server/blob/master/examples/mxnet_vision/mxnet_vision_service.py)\n\n* **Start Server**:\n```bash\nmulti-model-server --start --models onnx-squeezenet=https://s3.amazonaws.com/model-server/model_archive_1.0/onnx-squeezenet.mar\n```\n\n* **Run Prediction**:\n```bash\ncurl -X POST http://127.0.0.1:8080/predictions/onnx-squeezenet -T kitten.jpg\n```\n\n\n## VGG16\n* **Type**: Image classification trained on ImageNet\n\n* **Reference**: [Simonyan, et al.](https://arxiv.org/pdf/1409.1556v6.pdf)\n\n* **Model Service**: [mxnet_vision_service.py](https://github.com/awslabs/multi-model-server/blob/master/examples/mxnet_vision/mxnet_vision_service.py)\n\n* **Start Server**:\n```bash\nmulti-model-server --start --models vgg16=https://s3.amazonaws.com/model-server/model_archive_1.0/vgg16.mar\n```\n\n* **Run Prediction**:\n```bash\ncurl -X POST http://127.0.0.1:8080/predictions/vgg16 -T kitten.jpg\n```\n\n## VGG19\n* **Type**: Image classification trained on ImageNet\n\n* **Reference**: [Simonyan, et al.](https://arxiv.org/pdf/1409.1556v6.pdf)\n\n* **Model Service**: [mxnet_vision_service.py](https://github.com/awslabs/multi-model-server/blob/master/examples/mxnet_vision/mxnet_vision_service.py)\n\n* **Start Server**:\n```bash\nmulti-model-server --start --models vgg19=https://s3.amazonaws.com/model-server/model_archive_1.0/vgg19.mar\n```\n\n* **Run Prediction**:\n```bash\ncurl -X POST http://127.0.0.1:8080/predictions/vgg19 -T kitten.jpg\n```\n\n## VGG (from ONNX model zoo)\n\n### VGG16\n* **Type**: Image classification trained on ImageNet (imported from ONNX)\n\n* **Reference**: [Simonyan, et al.](https://arxiv.org/pdf/1409.1556v6.pdf)\n\n* **Model Service**: [mxnet_vision_service.py](https://s3.amazonaws.com/model-server/model_archive_1.0/mxnet_vision_service.py)\n\n* **Start Server**:\n```bash\nmulti-model-server --start --models onnx-vgg16=https://s3.amazonaws.com/model-server/model_archive_1.0/onnx-vgg16.mar\n```\n\n* **Run Prediction**:\n```bash\ncurl -X POST http://127.0.0.1:8080/predictions/onnx-vgg16 -T kitten.jpg\n```\n\n### VGG16_bn\n* **Type**: Image classification trained on ImageNet (imported from ONNX)\n\n* **Reference**: [Simonyan, et al.](https://arxiv.org/pdf/1409.1556v6.pdf) (Batch normalization applied after each conv layer of VGG16)\n\n* **Model Service**: [mxnet_vision_service.py](https://s3.amazonaws.com/model-server/model_archive_1.0/mxnet_vision_service.py)\n\n* **Start Server**:\n```bash\nmulti-model-server --start --models onnx-vgg16_bn=https://s3.amazonaws.com/model-server/model_archive_1.0/onnx-vgg16_bn.mar\n```\n\n* **Run Prediction**:\n```bash\ncurl -X POST http://127.0.0.1:8080/predictions/onnx-vgg16_bn -T kitten.jpg\n```\n\n### VGG19\n* **Type**: Image classification trained on ImageNet (imported from ONNX)\n\n* **Reference**: [Simonyan, et al.](https://arxiv.org/pdf/1409.1556v6.pdf)\n\n* **Model Service**: [mxnet_vision_service.py](https://s3.amazonaws.com/model-server/model_archive_1.0/mxnet_vision_service.py)\n\n* **Start Server**:\n```bash\nmulti-model-server --start --models onnx-vgg19=https://s3.amazonaws.com/model-server/model_archive_1.0/onnx-vgg19.mar\n```\n\n* **Run Prediction**:\n```bash\ncurl -X POST http://127.0.0.1:8080/predictions/onnx-vgg19 -T kitten.jpg\n```\n\n### VGG19_bn\n* **Type**: Image classification trained on ImageNet (imported from ONNX)\n\n* **Reference**: [Simonyan, et al.](https://arxiv.org/pdf/1409.1556v6.pdf) (Batch normalization applied after each conv layer of VGG19)\n\n* **Model Service**: [mxnet_vision_service.py](https://s3.amazonaws.com/model-server/model_archive_1.0/mxnet_vision_service.py)\n\n* **Start Server**:\n```bash\nmulti-model-server --start --models onnx-vgg19_bn=https://s3.amazonaws.com/model-server/model_archive_1.0/onnx-vgg19_bn.mar\n```\n\n* **Run Prediction**:\n```bash\ncurl -X POST http://127.0.0.1:8080/predictions/onnx-vgg19_bn -T kitten.jpg\n```\n" }, { "path": "docs/rest_api.md", "content": "# MMS REST API\n\nMMS use RESTful API for both inference and management calls. The API is compliance with [OpenAPI specification 3.0](https://swagger.io/specification/). User can easily generate client side code for Java, Scala, C#, Javascript use [swagger codegen](https://swagger.io/swagger-codegen/).\n\nWhen MMS startup, it start two web services:\n* [Inference API](inference_api.md)\n* [Management API](management_api.md)\n\nBy default, MMS listening on 8080 port for Inference API and 8081 on Management API.\nBoth API is only accessible from localhost. Please see [MMS Configuration](configuration.md) for how to enable access from remote host. \n" }, { "path": "docs/server.md", "content": "# Running the Model Server\n\n## Contents of this Document\n* [Overview](#overview)\n* [Technical Details](#technical-details)\n* [Model Files](#model-files)\n* [Command Line Interface](#command-line-interface)\n* [Advanced Features](#advanced-features)\n\n## Overview\n\nMulti Model Server can be used for many types of inference in production settings. It provides an easy-to-use command line interface and utilizes [REST based APIs](rest_api.md) handle state prediction requests. Support for models from a wide range of deep learning frameworks is achieved through its [ONNX model](https://onnx.ai) export feature.\n\nFor example, you want to make an app that lets your users snap a picture, and it will tell them what objects were detected in the scene and predictions on what the objects might be. You can use MMS to serve a prediction endpoint for a object detection and identification model that intakes images, then returns predictions. You can also modify MMS behavior with custom services and run multiple models. There are examples of custom services in the [examples](../examples) folder.\n\n## Technical Details\n\nNow that you have a high level view of MMS, let's get a little into the weeds. MMS takes a deep learning model and it wraps it in a set of REST APIs. Currently it comes with a built-in web server that you run from command line. This command line call takes in the single or multiple models you want to serve, along with additional optional parameters controlling the port, host, and logging. MMS supports running custom services to handle the specific inference handling logic. These are covered in more detail in the [custom service](custom_service.md) documentation.\n\nTo try out MMS serving now, you can load the SqueezeNet model, which is under 5 MB, with this example:\n\n```bash\nmulti-model-server --start --models squeezenet=https://s3.amazonaws.com/model-server/model_archive_1.0/squeezenet_v1.1.mar\n```\n\nWith the command above executed, you have MMS running on your host, listening for inference requests.\n\nTo test it out, you will need to open a new terminal window next to the one running MMS. Then we will use `curl` to download one of these [cute pictures of a kitten](https://www.google.com/search?q=cute+kitten&tbm=isch&hl=en&cr=&safe=images) and curl's `-o` flag will name it `kitten.jpg` for us. Then we will `curl` a `POST` to the MMS predictions endpoint with the kitten's image. In the example below, both of these steps are provided.\n\n```bash\ncurl -o kitten.jpg \\\n https://upload.wikimedia.org/wikipedia/commons/8/8f/Cute-kittens-12929201-1600-1200.jpg\ncurl -X POST http://127.0.0.1:8080/predictions/squeezenet -T kitten.jpg\n```\n\n![kitten](https://upload.wikimedia.org/wikipedia/commons/8/8f/Cute-kittens-12929201-1600-1200.jpg)\n\nThe predict endpoint will return a prediction response in JSON. Each of the probabilities are percentages, and the classes are coming from a `synset` file included inside the model archive which holds the thousand ImageNet classes this model is matching against. It will look something like the following result, where the 0.94 result is a 94% probable match with an Egyptian cat:\n\n\n```json\n[\n {\n \"probability\": 0.8582232594490051,\n \"class\": \"n02124075 Egyptian cat\"\n },\n {\n \"probability\": 0.09159987419843674,\n \"class\": \"n02123045 tabby, tabby cat\"\n },\n {\n \"probability\": 0.0374876894056797,\n \"class\": \"n02123159 tiger cat\"\n },\n {\n \"probability\": 0.006165083032101393,\n \"class\": \"n02128385 leopard, Panthera pardus\"\n },\n {\n \"probability\": 0.0031716004014015198,\n \"class\": \"n02127052 lynx, catamount\"\n }\n]\n```\nYou will see this result in the response to your `curl` call to the predict endpoint, in the terminal window running MMS and log files.\n\nAfter this deep dive, you might also be interested in:\n* [Logging](logging.md): logging options that are available\n\n* [Metrics](metrics.md): details on metrics collection \n\n* [REST API Description](rest_api.md): more detail about the server's endpoints\n\n* [Model Zoo](model_zoo.md): try serving different models\n\n* [Custom Services](custom_service.md): learn about serving different kinds of model and inference types\n\n\n## Model Files\n\nThe rest of this topic focus on serving of model files without much discussion on the model files themselves, where they come from, and how they're made. Long story short: it's a zip archive with the parameters, weights, and metadata that define a model that has been trained already. If you want to know more about the model files, take a look at the [model-archiver documentation](../model-archiver/README.md).\n\n## Command Line Interface\n\n```bash\n$ multi-model-server --help\nusage: multi-model-server [-h] [--start]\n [--stop]\n [--mms-config MMS_CONFIG]\n [--model-store MODEL_STORE]\n [--models MODEL_PATH1 MODEL_NAME=MODEL_PATH2... [MODEL_PATH1 MODEL_NAME=MODEL_PATH2... ...]]\n [--log-config LOG_CONFIG]\n\nMulti Model Server\n\noptional arguments:\n -h, --help show this help message and exit\n --start Start the model-server\n --stop Stop the model-server\n --mms-config MMS_CONFIG\n Configuration file for model server\n --model-store MODEL_STORE\n Model store location where models can be loaded\n --models MODEL_PATH1 MODEL_NAME=MODEL_PATH2... [MODEL_PATH1 MODEL_NAME=MODEL_PATH2... ...]\n Models to be loaded using [model_name=]model_location\n format. Location can be a HTTP URL, a model archive\n file or directory contains model archive files in\n MODEL_STORE.\n --log-config LOG_CONFIG\n Log4j configuration file for model server\n```\n\n#### Arguments:\nExample where no models are loaded at start time:\n\n```bash\nmulti-model-server\n```\n\nThere are no default required arguments to start the server\n\n1. **models**: required, = pairs.\n\n a) Model path can be a local file path or URI (s3 link, or http link).\n local file path: path/to/local/model/file or file://root/path/to/model/file\n s3 link: s3://S3_endpoint[:port]/...\n http link: http://hostname/path/to/resource\n\n b) The model file has .mar extension, it is actually a zip file with a .mar extension packing trained models and model signature files. \n\n c) Multiple models loading are also supported by specifying multiple name path pairs.\n1. **model-store**: optional, A location where models are stored by default, all models in this location are loaded, the model name is same as archive or folder name.\n1. **mms-config**: optional, provide a [configuration](configuration.md) file in config.properties format.\n1. **log-config**: optional, This parameter will override default log4j2.xml, present within the server.\n1. **start**: optional, A more descriptive way to start the server.\n1. **stop**: optional, Stop the server if it is already running.\n\n## Advanced Features\n\n### Custom Services\n\nThis topic is covered in much more detail on the [custom service documentation page](custom_service.md), but let's talk about how you start up your MMS server using a custom service and why you might want one.\nLet's say you have a model named `super-fancy-net.mar` in `/models` folder, which can detect a lot of things, but you want an API endpoint that detects only hotdogs. You would use a name that makes sense for it, such as the \"not-hot-dog\" API. In this case we might invoke MMS like this:\n\n```bash\nmulti-model-server --start --model-store /models --models not-hot-dog=super-fancy-net.mar\n```\n\nThis would serve a prediction endpoint at `predictions/not-hot-dog/` and run your custom service code in the archive, the manifest in archive would point to the entry point.\n\n### Serving Multiple Models with MMS\n\nExample multiple model usage:\n\n```bash\nmulti-model-server --start --model-store /models --models name=model_location name2=model_location2\n```\n\nHere's an example for running the resnet-18 and the vgg16 models using local model files.\n\n```bash\nmulti-model-server --start --model-store /models --models resnet-18=resnet-18.mar squeezenet=squeezenet_v1.1.mar\n```\n\nIf you don't have the model files locally, then you can call MMS using URLs to the model files.\n\n```bash\nmulti-model-server --models resnet=https://s3.amazonaws.com/model-server/model_archive_1.0/resnet-18.mar squeezenet=https://s3.amazonaws.com/model-server/model_archive_1.0/squeezenet_v1.1.mar\n```\n\nThis will setup a local host serving resnet-18 model and squeezenet model on the same port, using the default 8080. Check http://127.0.0.1:8081/models to see that each model has an endpoint for prediction. In this case you would see `predictions/resnet` and `predictions/squeezenet`\n\n\n### Logging and Metrics\n\nFor details on logging see the [logging documentation](logging.md). For details on metrics see the [metrics documentation](metrics.md).\n\n\n" }, { "path": "examples/README.md", "content": "# MMS Examples\n\nThe following are examples on how to create and serve model archives with MMS.\n\n* Gluon Models\n * [AlexNet](gluon_alexnet) - train a custom Gluon model, use model archiver, then serve the model archive with MMS\n * [Character CNN](gluon_character_cnn) - download a pre-trained model that classifies product reviews, use model archiver, then serve the model archive with MMS\n\n* Symbolic Models \n * [Image Classification with SqueezeNet](mxnet_vision) - download a pre-trained computer vision model, use model archiver, then serve the model archive with MMS\n * [LSTM with PTB](lstm_ptb) - download a pre-trained LSTM model that generates text, use model archiver, then serve the model archive with MMS\n\n* [How to setup metrics collection with AWS CloudWatch](metrics_cloudwatch)\n\n* [DenseNet PyTorch](densenet_pytorch)\n" }, { "path": "examples/densenet_pytorch/Dockerfile", "content": "FROM awsdeeplearningteam/multi-model-server:base-cpu-py3.6\n\n# Add PyTorch\nRUN pip install --no-cache-dir torch torchvision\n\n" }, { "path": "examples/densenet_pytorch/README.md", "content": "# PyTorch serving \nThis example shows how to serve PyTorch trained models for flower species recognition..\nThe custom handler is implemented in `densenet_service.py`.\nFor simplicity, we'll use a pre-trained model. For simplicity we will use docker container to run Model Server.\n\n## Getting Started With Docker\nBuild the docker image with pytorch as backend engine:\n```bash\n cd examples/densenet_pytorch/\n docker build . -t mms_with_pytorch\n```\n\nRun the container that you have built in previous step.\n```bash\n docker run -it --entrypoint bash mms_with_pytorch\n```\n\nStart the server from inside the container:\n```bash\n multi-model-server --models densenet161_pytorch=https://s3.amazonaws.com/model-server/model_archive_1.0/examples/PyTorch+models/densenet/densenet161_pytorch.mar\n```\n\nNow we can download a sample flower's image\n```bash\n curl -O https://s3.amazonaws.com/model-server/inputs/flower.jpg\n```\nGet the status of the model with the following:\n```bash\n curl -X POST http://127.0.0.1:8080/predictions/densenet161_pytorch -T flower.jpg\n```\n```json\n[\n {\n \"canna lily\": 0.01565943844616413\n },\n {\n \"water lily\": 0.015515935607254505\n },\n {\n \"purple coneflower\": 0.014358781278133392\n },\n {\n \"globe thistle\": 0.014226051047444344\n },\n {\n \"ruby-lipped cattleya\": 0.014212552458047867\n }\n ]\n```\n\nFor more information on MAR files and the built-in REST APIs, see:\n* https://github.com/awslabs/multi-model-server/tree/master/docs\n" }, { "path": "examples/densenet_pytorch/densenet_service.py", "content": "import os\nimport io\nimport json\nimport numpy as np\nfrom PIL import Image\nimport torch\nfrom torch.autograd import Variable\nfrom torchvision import transforms\nimport torch.nn.functional as F\n\n\nclass PyTorchImageClassifier():\n \"\"\"\n PyTorchImageClassifier service class. This service takes a flower \n image and returns the name of that flower. \n \"\"\"\n\n def __init__(self):\n\n self.checkpoint_file_path = None\n self.model = None\n self.mapping = None\n self.device = \"cpu\"\n self.initialized = False\n\n def initialize(self, context):\n \"\"\"\n Load the model and mapping file to perform infernece.\n \"\"\"\n\n properties = context.system_properties\n model_dir = properties.get(\"model_dir\")\n\n # Read checkpoint file\n checkpoint_file_path = os.path.join(model_dir, \"model.pth\")\n if not os.path.isfile(checkpoint_file_path):\n raise RuntimeError(\"Missing model.pth file.\") \n\n # Prepare the model \n checkpoint = torch.load(checkpoint_file_path, map_location='cpu')\n model = checkpoint['model']\n model.classifier = checkpoint['classifier']\n model.load_state_dict(checkpoint['state_dict'])\n model.class_to_idx = checkpoint['class_to_idx']\n\n for param in model.parameters():\n param.requires_grad = False\n \n self.model = model \n \n # Read the mapping file, index to flower\n mapping_file_path = os.path.join(model_dir, \"index_to_name.json\")\n if not os.path.isfile(mapping_file_path):\n raise RuntimeError(\"Missing the mapping file\")\n with open(mapping_file_path) as f:\n self.mapping = json.load(f)\n\n self.initialized = True\n\n def preprocess(self, data):\n \"\"\"\n Scales, crops, and normalizes a PIL image for a PyTorch model,\n returns an Numpy array\n \"\"\"\n image = data[0].get(\"data\")\n if image is None:\n image = data[0].get(\"body\")\n\n my_preprocess = transforms.Compose([\n transforms.Resize(256),\n transforms.CenterCrop(224),\n transforms.ToTensor(),\n transforms.Normalize(mean=[0.485, 0.456, 0.406], \n std=[0.229, 0.224, 0.225])\n ])\n image = Image.open(io.BytesIO(image))\n image = my_preprocess(image)\n return image\n\n def inference(self, img, topk=5):\n ''' Predict the class (or classes) of an image using a trained deep learning model.\n '''\n # Convert 2D image to 1D vector\n img = np.expand_dims(img, 0)\n \n img = torch.from_numpy(img)\n \n self.model.eval()\n inputs = Variable(img).to(self.device)\n logits = self.model.forward(inputs)\n \n ps = F.softmax(logits,dim=1)\n topk = ps.cpu().topk(topk)\n \n probs, classes = (e.data.numpy().squeeze().tolist() for e in topk)\n \n results = []\n for i in range(len(probs)):\n tmp = dict()\n tmp[self.mapping[str(classes[i])]] = probs[i]\n results.append(tmp)\n return [results] \n\n def postprocess(self, inference_output):\n return inference_output\n\n\n# Following code is not necessary if your service class contains `handle(self, data, context)` function\n_service = PyTorchImageClassifier()\n\n\ndef handle(data, context):\n if not _service.initialized:\n _service.initialize(context)\n\n if data is None:\n return None\n\n data = _service.preprocess(data)\n data = _service.inference(data)\n data = _service.postprocess(data)\n\n return data\n" }, { "path": "examples/densenet_pytorch/index_to_name.json", "content": "{\"21\": \"fire lily\", \"3\": \"canterbury bells\", \"45\": \"bolero deep blue\", \"1\": \"pink primrose\", \"34\": \"mexican aster\", \"27\": \"prince of wales feathers\", \"7\": \"moon orchid\", \"16\": \"globe-flower\", \"25\": \"grape hyacinth\", \"26\": \"corn poppy\", \"79\": \"toad lily\", \"39\": \"siam tulip\", \"24\": \"red ginger\", \"67\": \"spring crocus\", \"35\": \"alpine sea holly\", \"32\": \"garden phlox\", \"10\": \"globe thistle\", \"6\": \"tiger lily\", \"93\": \"ball moss\", \"33\": \"love in the mist\", \"9\": \"monkshood\", \"102\": \"blackberry lily\", \"14\": \"spear thistle\", \"19\": \"balloon flower\", \"100\": \"blanket flower\", \"13\": \"king protea\", \"49\": \"oxeye daisy\", \"15\": \"yellow iris\", \"61\": \"cautleya spicata\", \"31\": \"carnation\", \"64\": \"silverbush\", \"68\": \"bearded iris\", \"63\": \"black-eyed susan\", \"69\": \"windflower\", \"62\": \"japanese anemone\", \"20\": \"giant white arum lily\", \"38\": \"great masterwort\", \"4\": \"sweet pea\", \"86\": \"tree mallow\", \"101\": \"trumpet creeper\", \"42\": \"daffodil\", \"22\": \"pincushion flower\", \"2\": \"hard-leaved pocket orchid\", \"54\": \"sunflower\", \"66\": \"osteospermum\", \"70\": \"tree poppy\", \"85\": \"desert-rose\", \"99\": \"bromelia\", \"87\": \"magnolia\", \"5\": \"english marigold\", \"92\": \"bee balm\", \"28\": \"stemless gentian\", \"97\": \"mallow\", \"57\": \"gaura\", \"40\": \"lenten rose\", \"47\": \"marigold\", \"59\": \"orange dahlia\", \"48\": \"buttercup\", \"55\": \"pelargonium\", \"36\": \"ruby-lipped cattleya\", \"91\": \"hippeastrum\", \"29\": \"artichoke\", \"71\": \"gazania\", \"90\": \"canna lily\", \"18\": \"peruvian lily\", \"98\": \"mexican petunia\", \"8\": \"bird of paradise\", \"30\": \"sweet william\", \"17\": \"purple coneflower\", \"52\": \"wild pansy\", \"84\": \"columbine\", \"12\": \"colt's foot\", \"11\": \"snapdragon\", \"96\": \"camellia\", \"23\": \"fritillary\", \"50\": \"common dandelion\", \"44\": \"poinsettia\", \"53\": \"primula\", \"72\": \"azalea\", \"65\": \"californian poppy\", \"80\": \"anthurium\", \"76\": \"morning glory\", \"37\": \"cape flower\", \"56\": \"bishop of llandaff\", \"60\": \"pink-yellow dahlia\", \"82\": \"clematis\", \"58\": \"geranium\", \"75\": \"thorn apple\", \"41\": \"barbeton daisy\", \"95\": \"bougainvillea\", \"43\": \"sword lily\", \"83\": \"hibiscus\", \"78\": \"lotus lotus\", \"88\": \"cyclamen\", \"94\": \"foxglove\", \"81\": \"frangipani\", \"74\": \"rose\", \"89\": \"watercress\", \"73\": \"water lily\", \"46\": \"wallflower\", \"77\": \"passion flower\", \"51\": \"petunia\"}" }, { "path": "examples/gluon_alexnet/README.md", "content": "# Loading and serving Gluon models on Multi Model Server (MMS)\nMulti Model Server (MMS) supports loading and serving MXNet Imperative and Hybrid Gluon models.\nThis is a short tutorial on how to write a custom Gluon model, and then serve it with MMS.\n\nThis tutorial covers the following:\n1. [Prerequisites](#prerequisites)\n2. [Serve a Gluon model](#load-and-serve-a-gluon-model)\n * [Load and serve a pre-trained Gluon model](#load-and-serve-a-pre-trained-gluon-model)\n * [Load and serve a custom Gluon model](#load-and-serve-a-custom-gluon-imperative-model)\n * [Load and serve a custom hybrid Gluon model](#load-and-serve-a-hybrid-gluon-model)\n3. [Conclusion](#conclusion)\n\n## Prerequisites\n* **Basic Gluon knowledge**. If you are using Gluon for the first\ntime, but are familiar with creating a neural network with MXNet or another framework, you may refer this 10 min Gluon crash-course: [Predict with a pre-trained model](http://gluon-crash-course.mxnet.io/predict.html).\n* **Gluon naming**. Fine-tuning pre-trained Gluon models requires some understanding of how the naming conventions work. Take a look at the [Naming of Gluon Parameter and Blocks](https://mxnet.incubator.apache.org/tutorials/gluon/naming.html) tutorial for more information.\n* **Basic MMS knowledge**. If you are using MMS for the first time, you should take advantage of the [MMS QuickStart tutorial](https://github.com/awslabs/multi-model-server#quick-start).\n* **MMS installed**. If you haven't already, [install MMS with pip](https://github.com/awslabs/multi-model-server/blob/master/docs/install.md#install-mms-with-pip) or [install MMS from source](https://github.com/awslabs/multi-model-server/blob/master/docs/install.md#install-mms-from-source-code). Either installation will also install MXNet.\n\nRefer to the [MXNet model zoo](https://mxnet.incubator.apache.org/api/python/gluon/model_zoo.html) documentation for examples of accessing other models.\n\n## Load and serve a Gluon model\nThere are three scenarios for serving a Gluon model with MMS:\n\n1. Load and serve a pre-trained Gluon model.\n2. Load and serve a custom imperative Gluon model.\n3. Load and serve a custom hybrid Gluon model.\n\nTo learn more about the differences between gluon and hybrid gluon models refer to [the following document](http://gluon.mxnet.io/chapter07_distributed-learning/hybridize.html)\n\n### Load and serve a pre-trained Gluon model\nLoading and serving a pre-trained Gluon model is the simplest of the three scenarios. These models don't require you to provide `symbols` and `params` files.\n\nIt is easy to access a model with a couple of lines of code. The following code snippet shows how to load and serve a pretrained Gluon model.\n\n```python\nclass PretrainedAlexnetService(GluonBaseService):\n \"\"\"\n Pretrained alexnet Service\n \"\"\"\n def initialize(self, params):\n self.net = mxnet.gluon.model_zoo.vision.alexnet(pretrained=True)\n self.param_filename = \"alexnet.params\"\n super(PretrainedAlexnetService, self).initialize(params)\n\n def postprocess(self, data):\n idx = data.topk(k=5)[0]\n return [[{'class': (self.labels[int(i.asscalar())]).split()[1], 'probability':\n float(data[0, int(i.asscalar())].asscalar())} for i in idx]]\n\n\nsvc = PretrainedAlexnetService()\n\n\ndef pretrained_gluon_alexnet(data, context):\n res = None\n if not svc.initialized:\n svc.initialize(context)\n\n if data is not None:\n res = svc.predict(data)\n\n return res\n```\n\nFor an actual code implementation, refer to the custom-service code which uses the [pre-trained Alexnet](https://github.com/awslabs/multi-model-server/blob/master/examples/gluon_alexnet/gluon_pretrained_alexnet.py)\n\n### Serve pre-trained model with MMS\nTo serve pre-trained models with MMS we would need to create an model archive file. Follow the below steps to get the example custom service\nloaded and served with MMS.\n\n1. Create a `models` directory\n```bash\nmkdir /tmp/models\n```\n2. Copy the [example code](https://github.com/awslabs/multi-model-server/blob/master/examples/gluon_alexnet/gluon_pretrained_alexnet.py)\nand other required artifacts to this folder\n```bash\ncp ../model_service_template/gluon_base_service.py ../model_service_template/mxnet_utils/ndarray.py gluon_pretrained_alexnet.py synset.txt signature.json /tmp/models/.\n```\n3. Run the model-export tool on this folder.\n```bash\nmodel-archiver --model-name alexnet --model-path /tmp/models --handler gluon_pretrained_alexnet:pretrained_gluon_alexnet --runtime python --export-path /tmp\n```\nThis creates a model-archive file `/tmp/alexnet.mar`.\n\n4. You could run the server with this model file to serve the pre-trained alexnet.\n```bash\nmulti-model-server --start --models alexnet.mar --model-store /tmp\n```\n5. Test your service\n```bash\ncurl -O https://s3.amazonaws.com/model-server/inputs/kitten.jpg\ncurl -X POST http://127.0.0.1:8080/alexnet/predict -F \"data=@kitten.jpg\"\n```\n\n## Load and serve a custom Gluon imperative model\nTo load an imperative model for use with MMS, you must activate the network in a MMS custom service. Once activated, MMS\ncan load the pre-trained parameters and start serving the imperative model. You also need to handle pre-processing and\npost-processing of the image input.\n\nWe created a custom imperative model using Gluon. Refer to\n[custom service code](https://github.com/awslabs/multi-model-server/examples/gluon_alexnet/examples/gluon_alexnet/gluon_alexnet.py)\nThe network definition, which is defined in the example, is as follows\n\n```python\nclass GluonImperativeAlexNet(gluon.Block):\n \"\"\"\n Fully imperative gluon Alexnet model\n \"\"\"\n def __init__(self, classes=1000, **kwargs):\n super(GluonImperativeAlexNet, self).__init__(**kwargs)\n with self.name_scope():\n self.features = nn.Sequential(prefix='')\n with self.features.name_scope():\n self.features.add(nn.Conv2D(64, kernel_size=11, strides=4,\n padding=2, activation='relu'))\n self.features.add(nn.MaxPool2D(pool_size=3, strides=2))\n self.features.add(nn.Conv2D(192, kernel_size=5, padding=2,\n activation='relu'))\n self.features.add(nn.MaxPool2D(pool_size=3, strides=2))\n self.features.add(nn.Conv2D(384, kernel_size=3, padding=1,\n activation='relu'))\n self.features.add(nn.Conv2D(256, kernel_size=3, padding=1,\n activation='relu'))\n self.features.add(nn.Conv2D(256, kernel_size=3, padding=1,\n activation='relu'))\n self.features.add(nn.MaxPool2D(pool_size=3, strides=2))\n self.features.add(nn.Flatten())\n self.features.add(nn.Dense(4096, activation='relu'))\n self.features.add(nn.Dropout(0.5))\n self.features.add(nn.Dense(4096, activation='relu'))\n self.features.add(nn.Dropout(0.5))\n self.output = nn.Dense(classes)\n\n def forward(self, x):\n\tx = self.features(x)\n x = self.output(x)\n return x\n```\n\nThe pre-process, inference and post-process steps are similar to the service code that we saw in the [above section](#load-and-serve-a-pre-trained-gluon-model).\n```python\nclass ImperativeAlexnetService(GluonBaseService):\n \"\"\"\n Gluon alexnet Service\n \"\"\"\n\n def initialize(self, params):\n self.net = GluonImperativeAlexNet()\n self.param_filename = \"alexnet.params\"\n super(ImperativeAlexnetService, self).initialize(params)\n\n def postprocess(self, data):\n idx = data.topk(k=5)[0]\n return [[{'class': (self.labels[int(i.asscalar())]).split()[1], 'probability':\n float(data[0, int(i.asscalar())].asscalar())} for i in idx]]\n\n\nsvc = ImperativeAlexnetService()\n\n\ndef imperative_gluon_alexnet_inf(data, context):\n res = None\n if not svc.initialized:\n svc.initialize(context)\n\n if data is not None:\n res = svc.predict(data)\n\n return res\n```\n\n### Test your imperative Gluon model service\nTo serve imperative Gluon models with MMS we would need to create an model archive file.\nFollow the below steps to get the example custom service loaded and served with MMS.\n\n1. Create a `models` directory\n```bash\nmkdir /tmp/models\n```\n2. Copy the [example code](https://github.com/awslabs/multi-model-server/examples/gluon_alexnet/gluon_imperative_alexnet.py)\nand other required artifacts to this folder\n```bash\ncp ../model_service_template/gluon_base_service.py ../model_service_template/mxnet_utils/ndarray.py gluon_imperative_alexnet.py synset.txt signature.json /tmp/models/.\n```\n3. Download/copy the parameters to this `/tmp/models` directory. For this example, we have the parameters file in a S3 bucket.\n```bash\nwget https://s3.amazonaws.com/gluon-mms-model-files/alexnet.params\nmv alexnet.params /tmp/models\n```\n4. Run the model-export tool on this folder.\n```bash\nmodel-archiver --model-name alexnet --model-path /tmp/models --handler gluon_imperative_alexnet:imperative_gluon_alexnet_inf --runtime python --export-path /tmp\n```\nThis creates a model-archive file `/tmp/alexnet.mar`.\n\n5. You could run the server with this model file to serve the pre-trained alexnet.\n```bash\nmulti-model-server --start --models alexnet.mar --model-store /tmp\n```\n6. Test your service\n```bash\ncurl -O https://s3.amazonaws.com/model-server/inputs/kitten.jpg\ncurl -X POST http://127.0.0.1:8080/alexnet/predict -F \"data=@kitten.jpg\"\n```\n\nThe output should be close to the following:\n\n```json\n{\"prediction\":[{\"class\":\"lynx,\",\"probability\":0.9411474466323853},{\"class\":\"leopard,\",\"probability\":0.016749195754528046},{\"class\":\"tabby,\",\"probability\":0.012754007242619991},{\"class\":\"Egyptian\",\"probability\":0.011728651821613312},{\"class\":\"tiger\",\"probability\":0.008974711410701275}]}\n```\n\n## Load and serve a hybrid Gluon model\nTo serve hybrid Gluon models with MMS, let's consider `gluon_imperative_alexnet.py` in `multi-model-server/examples/gluon_alexnet` folder.\nWe first convert the model to a `Gluon` hybrid block.\nFor additional background on using `HybridBlocks` and the need to `hybridize` refer to this Gluon [hybridize](http://gluon.mxnet.io/chapter07_distributed-learning/hybridize.html#) tutorial.\n\nThe above network, after this conversion, would look as follows:\n```python\nclass GluonHybridAlexNet(HybridBlock):\n \"\"\"\n Hybrid Block gluon model\n \"\"\"\n def __init__(self, classes=1000, **kwargs):\n super(GluonHybridAlexNet, self).__init__(**kwargs)\n with self.name_scope():\n self.features = nn.HybridSequential(prefix='')\n with self.features.name_scope():\n self.features.add(nn.Conv2D(64, kernel_size=11, strides=4,\n padding=2, activation='relu'))\n self.features.add(nn.MaxPool2D(pool_size=3, strides=2))\n self.features.add(nn.Conv2D(192, kernel_size=5, padding=2,\n activation='relu'))\n self.features.add(nn.MaxPool2D(pool_size=3, strides=2))\n self.features.add(nn.Conv2D(384, kernel_size=3, padding=1,\n activation='relu'))\n self.features.add(nn.Conv2D(256, kernel_size=3, padding=1,\n activation='relu'))\n self.features.add(nn.Conv2D(256, kernel_size=3, padding=1,\n activation='relu'))\n self.features.add(nn.MaxPool2D(pool_size=3, strides=2))\n self.features.add(nn.Flatten())\n self.features.add(nn.Dense(4096, activation='relu'))\n self.features.add(nn.Dropout(0.5))\n self.features.add(nn.Dense(4096, activation='relu'))\n self.features.add(nn.Dropout(0.5))\n\n self.output = nn.Dense(classes)\n\n def hybrid_forward(self, F, x):\n x = self.features(x)\n x = self.output(x)\n return x\n```\nWe could use the same custom service code as in the above section,\n\n```python\nclass HybridAlexnetService(GluonBaseService):\n \"\"\"\n Gluon alexnet Service\n \"\"\"\n def initialize(self, params):\n self.net = GluonHybridAlexNet()\n self.param_filename = \"alexnet.params\"\n super(HybridAlexnetService, self).initialize(params)\n self.net.hybridize()\n\n def postprocess(self, data):\n idx = data.topk(k=5)[0]\n return [[{'class': (self.labels[int(i.asscalar())]).split()[1], 'probability':\n float(data[0, int(i.asscalar())].asscalar())} for i in idx]]\n\n\nsvc = HybridAlexnetService()\n\n\ndef hybrid_gluon_alexnet_inf(data, context):\n res = None\n if not svc.initialized:\n svc.initialize(context)\n\n if data is not None:\n res = svc.predict(data)\n\n return res\n```\nSimilar to imperative models, this model doesn't require `Symbols` as the call to `.hybridize()` compiles the neural net.\nThis would store the `symbols` implicitly.\n\n### Test your hybrid Gluon model service\nTo serve Hybrid Gluon models with MMS we would need to create an model archive file.\nFollow the below steps to get the example custom service loaded and served with MMS.\n\n1. Create a `models` directory\n```bash\nmkdir /tmp/models\n```\n2. Copy the [example code](https://github.com/awslabs/multi-model-server/examples/gluon_alexnet/gluon_imperative_alexnet.py)\nand other required artifacts to this folder\n```bash\ncp ../model_service_template/gluon_base_service.py ../model_service_template/mxnet_utils/ndarray.py gluon_hybrid_alexnet.py synset.txt signature.json /tmp/models/.\n```\n3. Download/copy the parameters to this `/tmp/models` directory. For this example, we have the parameters file in a S3 bucket.\n```bash\nwget https://s3.amazonaws.com/gluon-mms-model-files/alexnet.params\nmv alexnet.params /tmp/models\n```\n4. Run the model-export tool on this folder.\n```bash\nmodel-archiver --model-name alexnet --model-path /tmp/models --handler gluon_hybrid_alexnet:hybrid_gluon_alexnet_inf --runtime python --export-path /tmp\n```\nThis creates a model-archive file `/tmp/alexnet.mar`.\n\n5. You could run the server with this model file to serve the pre-trained alexnet.\n```bash\nmulti-model-server --start --models alexnet.mar --model-store /tmp\n```\n6. Test your service\n```bash\ncurl -O https://s3.amazonaws.com/model-server/inputs/kitten.jpg\ncurl -X POST http://127.0.0.1:8080/alexnet/predict -F \"data=@kitten.jpg\"\n```\n\nThe output should be close to the following:\n\n```json\n{\"prediction\":[{\"class\":\"lynx,\",\"probability\":0.9411474466323853},{\"class\":\"leopard,\",\"probability\":0.016749195754528046},{\"class\":\"tabby,\",\"probability\":0.012754007242619991},{\"class\":\"Egyptian\",\"probability\":0.011728651821613312},{\"class\":\"tiger\",\"probability\":0.008974711410701275}]}\n```\n\n## Conclusion\nIn this tutorial you learned how to serve Gluon models in three unique scenarios: a pre-trained imperative model directly from the model zoo, a custom imperative model, and a hybrid model. For further examples of customizing gluon models, try the Gluon tutorial for [Transferring knowledge through fine-tuning](http://gluon.mxnet.io/chapter08_computer-vision/fine-tuning.html). For an advanced custom service example, try the MMS [SSD example](https://github.com/awslabs/multi-model-server/tree/master/examples/ssd).\n" }, { "path": "examples/gluon_alexnet/gluon_hybrid_alexnet.py", "content": "# Copyright 2018 Amazon.com, Inc. or its affiliates. All Rights Reserved.\n# Licensed under the Apache License, Version 2.0 (the \"License\").\n# You may not use this file except in compliance with the License.\n# A copy of the License is located at\n# http://www.apache.org/licenses/LICENSE-2.0\n# or in the \"license\" file accompanying this file. This file is distributed\n# on an \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either\n# express or implied. See the License for the specific language governing\n# permissions and limitations under the License.\n\nfrom mxnet.gluon import nn\nfrom mxnet.gluon.block import HybridBlock\nfrom gluon_base_service import GluonBaseService\n\n\"\"\"\nMMS examples for loading Gluon Hybrid models\n\"\"\"\n\n\nclass GluonHybridAlexNet(HybridBlock):\n \"\"\"\n Hybrid Block gluon model\n \"\"\"\n def __init__(self, classes=1000, **kwargs):\n \"\"\"\n This is the network definition of Gluon Hybrid Alexnet\n :param classes:\n :param kwargs:\n \"\"\"\n super(GluonHybridAlexNet, self).__init__(**kwargs)\n with self.name_scope():\n self.features = nn.HybridSequential(prefix='')\n with self.features.name_scope():\n self.features.add(nn.Conv2D(64, kernel_size=11, strides=4,\n padding=2, activation='relu'))\n self.features.add(nn.MaxPool2D(pool_size=3, strides=2))\n self.features.add(nn.Conv2D(192, kernel_size=5, padding=2,\n activation='relu'))\n self.features.add(nn.MaxPool2D(pool_size=3, strides=2))\n self.features.add(nn.Conv2D(384, kernel_size=3, padding=1,\n activation='relu'))\n self.features.add(nn.Conv2D(256, kernel_size=3, padding=1,\n activation='relu'))\n self.features.add(nn.Conv2D(256, kernel_size=3, padding=1,\n activation='relu'))\n self.features.add(nn.MaxPool2D(pool_size=3, strides=2))\n self.features.add(nn.Flatten())\n self.features.add(nn.Dense(4096, activation='relu'))\n self.features.add(nn.Dropout(0.5))\n self.features.add(nn.Dense(4096, activation='relu'))\n self.features.add(nn.Dropout(0.5))\n\n self.output = nn.Dense(classes)\n\n def hybrid_forward(self, F, x):\n x = self.features(x)\n x = self.output(x)\n return x\n\n\nclass HybridAlexnetService(GluonBaseService):\n \"\"\"\n Gluon alexnet Service\n \"\"\"\n def initialize(self, params):\n self.net = GluonHybridAlexNet()\n self.param_filename = \"alexnet.params\"\n super(HybridAlexnetService, self).initialize(params)\n self.net.hybridize()\n\n def postprocess(self, data):\n idx = data.topk(k=5)[0]\n return [[{'class': (self.labels[int(i.asscalar())]).split()[1], 'probability':\n float(data[0, int(i.asscalar())].asscalar())} for i in idx]]\n\n\nsvc = HybridAlexnetService()\n\n\ndef hybrid_gluon_alexnet_inf(data, context):\n res = None\n if not svc.initialized:\n svc.initialize(context)\n\n if data is not None:\n res = svc.predict(data)\n\n return res\n" }, { "path": "examples/gluon_alexnet/gluon_imperative_alexnet.py", "content": "# Copyright 2018 Amazon.com, Inc. or its affiliates. All Rights Reserved.\n# Licensed under the Apache License, Version 2.0 (the \"License\").\n# You may not use this file except in compliance with the License.\n# A copy of the License is located at\n# http://www.apache.org/licenses/LICENSE-2.0\n# or in the \"license\" file accompanying this file. This file is distributed\n# on an \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either\n# express or implied. See the License for the specific language governing\n# permissions and limitations under the License.\n\nfrom mxnet import gluon\nfrom mxnet.gluon import nn\nfrom gluon_base_service import GluonBaseService\n\n\"\"\"\nMMS examples for loading Gluon Imperative models\n\"\"\"\n\n\nclass GluonImperativeAlexNet(gluon.Block):\n \"\"\"\n Fully imperative gluon Alexnet model\n \"\"\"\n def __init__(self, classes=1000, **kwargs):\n \"\"\"\n This is the network definition of Imperative Alexnet\n :param classes:\n :param kwargs:\n \"\"\"\n super(GluonImperativeAlexNet, self).__init__(**kwargs)\n with self.name_scope():\n self.features = nn.Sequential(prefix='')\n with self.features.name_scope():\n self.features.add(nn.Conv2D(64, kernel_size=11, strides=4,\n padding=2, activation='relu'))\n self.features.add(nn.MaxPool2D(pool_size=3, strides=2))\n self.features.add(nn.Conv2D(192, kernel_size=5, padding=2,\n activation='relu'))\n self.features.add(nn.MaxPool2D(pool_size=3, strides=2))\n self.features.add(nn.Conv2D(384, kernel_size=3, padding=1,\n activation='relu'))\n self.features.add(nn.Conv2D(256, kernel_size=3, padding=1,\n activation='relu'))\n self.features.add(nn.Conv2D(256, kernel_size=3, padding=1,\n activation='relu'))\n self.features.add(nn.MaxPool2D(pool_size=3, strides=2))\n self.features.add(nn.Flatten())\n self.features.add(nn.Dense(4096, activation='relu'))\n self.features.add(nn.Dropout(0.5))\n self.features.add(nn.Dense(4096, activation='relu'))\n self.features.add(nn.Dropout(0.5))\n self.output = nn.Dense(classes)\n\n def forward(self, x):\n x = self.features(x)\n x = self.output(x)\n return x\n\n\nclass ImperativeAlexnetService(GluonBaseService):\n \"\"\"\n Gluon alexnet Service\n \"\"\"\n\n def initialize(self, params):\n self.net = GluonImperativeAlexNet()\n self.param_filename = \"alexnet.params\"\n super(ImperativeAlexnetService, self).initialize(params)\n\n def postprocess(self, data):\n idx = data.topk(k=5)[0]\n return [[{'class': (self.labels[int(i.asscalar())]).split()[1], 'probability':\n float(data[0, int(i.asscalar())].asscalar())} for i in idx]]\n\n\nsvc = ImperativeAlexnetService()\n\n\ndef imperative_gluon_alexnet_inf(data, context):\n \"\"\"\n Handler registered for inference\n :param data:\n :param context:\n :return:\n \"\"\"\n res = None\n if not svc.initialized:\n svc.initialize(context)\n\n if data is not None:\n res = svc.predict(data)\n\n return res\n" }, { "path": "examples/gluon_alexnet/gluon_pretrained_alexnet.py", "content": "# Copyright 2018 Amazon.com, Inc. or its affiliates. All Rights Reserved.\n# Licensed under the Apache License, Version 2.0 (the \"License\").\n# You may not use this file except in compliance with the License.\n# A copy of the License is located at\n# http://www.apache.org/licenses/LICENSE-2.0\n# or in the \"license\" file accompanying this file. This file is distributed\n# on an \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either\n# express or implied. See the License for the specific language governing\n# permissions and limitations under the License.\n\nimport mxnet\nfrom gluon_base_service import GluonBaseService\n\n\"\"\"\nGluon Pretrained Alexnet model\n\"\"\"\n\n\nclass PretrainedAlexnetService(GluonBaseService):\n \"\"\"\n Pretrained alexnet Service\n \"\"\"\n def initialize(self, params):\n \"\"\"\n Initialize the model\n :param params: This is the same as the Context object\n :return:\n \"\"\"\n self.net = mxnet.gluon.model_zoo.vision.alexnet(pretrained=True)\n super(PretrainedAlexnetService, self).initialize(params)\n\n def postprocess(self, data):\n \"\"\"\n Post process for the Gluon Alexnet model\n :param data:\n :return:\n \"\"\"\n idx = data.topk(k=5)[0]\n return [[{'class': (self.labels[int(i.asscalar())]).split()[1], 'probability':\n float(data[0, int(i.asscalar())].asscalar())} for i in idx]]\n\n\nsvc = PretrainedAlexnetService()\n\n\ndef pretrained_gluon_alexnet(data, context):\n \"\"\"\n This is the handler that needs to be registerd in the model-archive.\n :param data:\n :param context:\n :return:\n \"\"\"\n res = None\n if not svc.initialized:\n svc.initialize(context)\n\n if data is not None:\n res = svc.predict(data)\n\n return res\n" }, { "path": "examples/gluon_alexnet/signature.json", "content": "{\n \"inputs\": [\n {\n \"data_name\": \"data\",\n \"data_shape\": [0, 3, 224, 224]\n }\n ],\n \"input_type\": \"image/jpeg\",\n \"outputs\": [\n {\n \"data_name\": \"softmax\",\n \"data_shape\": [0, 1000]\n }\n ],\n \"output_type\": \"application/json\"\n}" }, { "path": "examples/gluon_alexnet/synset.txt", "content": "n01440764 tench, Tinca tinca\nn01443537 goldfish, Carassius auratus\nn01484850 great white shark, white shark, man-eater, man-eating shark, Carcharodon carcharias\nn01491361 tiger shark, Galeocerdo cuvieri\nn01494475 hammerhead, hammerhead shark\nn01496331 electric ray, crampfish, numbfish, torpedo\nn01498041 stingray\nn01514668 cock\nn01514859 hen\nn01518878 ostrich, Struthio camelus\nn01530575 brambling, Fringilla montifringilla\nn01531178 goldfinch, Carduelis carduelis\nn01532829 house finch, linnet, Carpodacus mexicanus\nn01534433 junco, snowbird\nn01537544 indigo bunting, indigo finch, indigo bird, Passerina cyanea\nn01558993 robin, American robin, Turdus migratorius\nn01560419 bulbul\nn01580077 jay\nn01582220 magpie\nn01592084 chickadee\nn01601694 water ouzel, dipper\nn01608432 kite\nn01614925 bald eagle, American eagle, Haliaeetus leucocephalus\nn01616318 vulture\nn01622779 great grey owl, great gray owl, Strix nebulosa\nn01629819 European fire salamander, Salamandra salamandra\nn01630670 common newt, Triturus vulgaris\nn01631663 eft\nn01632458 spotted salamander, Ambystoma maculatum\nn01632777 axolotl, mud puppy, Ambystoma mexicanum\nn01641577 bullfrog, Rana catesbeiana\nn01644373 tree frog, tree-frog\nn01644900 tailed frog, bell toad, ribbed toad, tailed toad, Ascaphus trui\nn01664065 loggerhead, loggerhead turtle, Caretta caretta\nn01665541 leatherback turtle, leatherback, leathery turtle, Dermochelys coriacea\nn01667114 mud turtle\nn01667778 terrapin\nn01669191 box turtle, box tortoise\nn01675722 banded gecko\nn01677366 common iguana, iguana, Iguana iguana\nn01682714 American chameleon, anole, Anolis carolinensis\nn01685808 whiptail, whiptail lizard\nn01687978 agama\nn01688243 frilled lizard, Chlamydosaurus kingi\nn01689811 alligator lizard\nn01692333 Gila monster, Heloderma suspectum\nn01693334 green lizard, Lacerta viridis\nn01694178 African chameleon, Chamaeleo chamaeleon\nn01695060 Komodo dragon, Komodo lizard, dragon lizard, giant lizard, Varanus komodoensis\nn01697457 African crocodile, Nile crocodile, Crocodylus niloticus\nn01698640 American alligator, Alligator mississipiensis\nn01704323 triceratops\nn01728572 thunder snake, worm snake, Carphophis amoenus\nn01728920 ringneck snake, ring-necked snake, ring snake\nn01729322 hognose snake, puff adder, sand viper\nn01729977 green snake, grass snake\nn01734418 king snake, kingsnake\nn01735189 garter snake, grass snake\nn01737021 water snake\nn01739381 vine snake\nn01740131 night snake, Hypsiglena torquata\nn01742172 boa constrictor, Constrictor constrictor\nn01744401 rock python, rock snake, Python sebae\nn01748264 Indian cobra, Naja naja\nn01749939 green mamba\nn01751748 sea snake\nn01753488 horned viper, cerastes, sand viper, horned asp, Cerastes cornutus\nn01755581 diamondback, diamondback rattlesnake, Crotalus adamanteus\nn01756291 sidewinder, horned rattlesnake, Crotalus cerastes\nn01768244 trilobite\nn01770081 harvestman, daddy longlegs, Phalangium opilio\nn01770393 scorpion\nn01773157 black and gold garden spider, Argiope aurantia\nn01773549 barn spider, Araneus cavaticus\nn01773797 garden spider, Aranea diademata\nn01774384 black widow, Latrodectus mactans\nn01774750 tarantula\nn01775062 wolf spider, hunting spider\nn01776313 tick\nn01784675 centipede\nn01795545 black grouse\nn01796340 ptarmigan\nn01797886 ruffed grouse, partridge, Bonasa umbellus\nn01798484 prairie chicken, prairie grouse, prairie fowl\nn01806143 peacock\nn01806567 quail\nn01807496 partridge\nn01817953 African grey, African gray, Psittacus erithacus\nn01818515 macaw\nn01819313 sulphur-crested cockatoo, Kakatoe galerita, Cacatua galerita\nn01820546 lorikeet\nn01824575 coucal\nn01828970 bee eater\nn01829413 hornbill\nn01833805 hummingbird\nn01843065 jacamar\nn01843383 toucan\nn01847000 drake\nn01855032 red-breasted merganser, Mergus serrator\nn01855672 goose\nn01860187 black swan, Cygnus atratus\nn01871265 tusker\nn01872401 echidna, spiny anteater, anteater\nn01873310 platypus, duckbill, duckbilled platypus, duck-billed platypus, Ornithorhynchus anatinus\nn01877812 wallaby, brush kangaroo\nn01882714 koala, koala bear, kangaroo bear, native bear, Phascolarctos cinereus\nn01883070 wombat\nn01910747 jellyfish\nn01914609 sea anemone, anemone\nn01917289 brain coral\nn01924916 flatworm, platyhelminth\nn01930112 nematode, nematode worm, roundworm\nn01943899 conch\nn01944390 snail\nn01945685 slug\nn01950731 sea slug, nudibranch\nn01955084 chiton, coat-of-mail shell, sea cradle, polyplacophore\nn01968897 chambered nautilus, pearly nautilus, nautilus\nn01978287 Dungeness crab, Cancer magister\nn01978455 rock crab, Cancer irroratus\nn01980166 fiddler crab\nn01981276 king crab, Alaska crab, Alaskan king crab, Alaska king crab, Paralithodes camtschatica\nn01983481 American lobster, Northern lobster, Maine lobster, Homarus americanus\nn01984695 spiny lobster, langouste, rock lobster, crawfish, crayfish, sea crawfish\nn01985128 crayfish, crawfish, crawdad, crawdaddy\nn01986214 hermit crab\nn01990800 isopod\nn02002556 white stork, Ciconia ciconia\nn02002724 black stork, Ciconia nigra\nn02006656 spoonbill\nn02007558 flamingo\nn02009229 little blue heron, Egretta caerulea\nn02009912 American egret, great white heron, Egretta albus\nn02011460 bittern\nn02012849 crane\nn02013706 limpkin, Aramus pictus\nn02017213 European gallinule, Porphyrio porphyrio\nn02018207 American coot, marsh hen, mud hen, water hen, Fulica americana\nn02018795 bustard\nn02025239 ruddy turnstone, Arenaria interpres\nn02027492 red-backed sandpiper, dunlin, Erolia alpina\nn02028035 redshank, Tringa totanus\nn02033041 dowitcher\nn02037110 oystercatcher, oyster catcher\nn02051845 pelican\nn02056570 king penguin, Aptenodytes patagonica\nn02058221 albatross, mollymawk\nn02066245 grey whale, gray whale, devilfish, Eschrichtius gibbosus, Eschrichtius robustus\nn02071294 killer whale, killer, orca, grampus, sea wolf, Orcinus orca\nn02074367 dugong, Dugong dugon\nn02077923 sea lion\nn02085620 Chihuahua\nn02085782 Japanese spaniel\nn02085936 Maltese dog, Maltese terrier, Maltese\nn02086079 Pekinese, Pekingese, Peke\nn02086240 Shih-Tzu\nn02086646 Blenheim spaniel\nn02086910 papillon\nn02087046 toy terrier\nn02087394 Rhodesian ridgeback\nn02088094 Afghan hound, Afghan\nn02088238 basset, basset hound\nn02088364 beagle\nn02088466 bloodhound, sleuthhound\nn02088632 bluetick\nn02089078 black-and-tan coonhound\nn02089867 Walker hound, Walker foxhound\nn02089973 English foxhound\nn02090379 redbone\nn02090622 borzoi, Russian wolfhound\nn02090721 Irish wolfhound\nn02091032 Italian greyhound\nn02091134 whippet\nn02091244 Ibizan hound, Ibizan Podenco\nn02091467 Norwegian elkhound, elkhound\nn02091635 otterhound, otter hound\nn02091831 Saluki, gazelle hound\nn02092002 Scottish deerhound, deerhound\nn02092339 Weimaraner\nn02093256 Staffordshire bullterrier, Staffordshire bull terrier\nn02093428 American Staffordshire terrier, Staffordshire terrier, American pit bull terrier, pit bull terrier\nn02093647 Bedlington terrier\nn02093754 Border terrier\nn02093859 Kerry blue terrier\nn02093991 Irish terrier\nn02094114 Norfolk terrier\nn02094258 Norwich terrier\nn02094433 Yorkshire terrier\nn02095314 wire-haired fox terrier\nn02095570 Lakeland terrier\nn02095889 Sealyham terrier, Sealyham\nn02096051 Airedale, Airedale terrier\nn02096177 cairn, cairn terrier\nn02096294 Australian terrier\nn02096437 Dandie Dinmont, Dandie Dinmont terrier\nn02096585 Boston bull, Boston terrier\nn02097047 miniature schnauzer\nn02097130 giant schnauzer\nn02097209 standard schnauzer\nn02097298 Scotch terrier, Scottish terrier, Scottie\nn02097474 Tibetan terrier, chrysanthemum dog\nn02097658 silky terrier, Sydney silky\nn02098105 soft-coated wheaten terrier\nn02098286 West Highland white terrier\nn02098413 Lhasa, Lhasa apso\nn02099267 flat-coated retriever\nn02099429 curly-coated retriever\nn02099601 golden retriever\nn02099712 Labrador retriever\nn02099849 Chesapeake Bay retriever\nn02100236 German short-haired pointer\nn02100583 vizsla, Hungarian pointer\nn02100735 English setter\nn02100877 Irish setter, red setter\nn02101006 Gordon setter\nn02101388 Brittany spaniel\nn02101556 clumber, clumber spaniel\nn02102040 English springer, English springer spaniel\nn02102177 Welsh springer spaniel\nn02102318 cocker spaniel, English cocker spaniel, cocker\nn02102480 Sussex spaniel\nn02102973 Irish water spaniel\nn02104029 kuvasz\nn02104365 schipperke\nn02105056 groenendael\nn02105162 malinois\nn02105251 briard\nn02105412 kelpie\nn02105505 komondor\nn02105641 Old English sheepdog, bobtail\nn02105855 Shetland sheepdog, Shetland sheep dog, Shetland\nn02106030 collie\nn02106166 Border collie\nn02106382 Bouvier des Flandres, Bouviers des Flandres\nn02106550 Rottweiler\nn02106662 German shepherd, German shepherd dog, German police dog, alsatian\nn02107142 Doberman, Doberman pinscher\nn02107312 miniature pinscher\nn02107574 Greater Swiss Mountain dog\nn02107683 Bernese mountain dog\nn02107908 Appenzeller\nn02108000 EntleBucher\nn02108089 boxer\nn02108422 bull mastiff\nn02108551 Tibetan mastiff\nn02108915 French bulldog\nn02109047 Great Dane\nn02109525 Saint Bernard, St Bernard\nn02109961 Eskimo dog, husky\nn02110063 malamute, malemute, Alaskan malamute\nn02110185 Siberian husky\nn02110341 dalmatian, coach dog, carriage dog\nn02110627 affenpinscher, monkey pinscher, monkey dog\nn02110806 basenji\nn02110958 pug, pug-dog\nn02111129 Leonberg\nn02111277 Newfoundland, Newfoundland dog\nn02111500 Great Pyrenees\nn02111889 Samoyed, Samoyede\nn02112018 Pomeranian\nn02112137 chow, chow chow\nn02112350 keeshond\nn02112706 Brabancon griffon\nn02113023 Pembroke, Pembroke Welsh corgi\nn02113186 Cardigan, Cardigan Welsh corgi\nn02113624 toy poodle\nn02113712 miniature poodle\nn02113799 standard poodle\nn02113978 Mexican hairless\nn02114367 timber wolf, grey wolf, gray wolf, Canis lupus\nn02114548 white wolf, Arctic wolf, Canis lupus tundrarum\nn02114712 red wolf, maned wolf, Canis rufus, Canis niger\nn02114855 coyote, prairie wolf, brush wolf, Canis latrans\nn02115641 dingo, warrigal, warragal, Canis dingo\nn02115913 dhole, Cuon alpinus\nn02116738 African hunting dog, hyena dog, Cape hunting dog, Lycaon pictus\nn02117135 hyena, hyaena\nn02119022 red fox, Vulpes vulpes\nn02119789 kit fox, Vulpes macrotis\nn02120079 Arctic fox, white fox, Alopex lagopus\nn02120505 grey fox, gray fox, Urocyon cinereoargenteus\nn02123045 tabby, tabby cat\nn02123159 tiger cat\nn02123394 Persian cat\nn02123597 Siamese cat, Siamese\nn02124075 Egyptian cat\nn02125311 cougar, puma, catamount, mountain lion, painter, panther, Felis concolor\nn02127052 lynx, catamount\nn02128385 leopard, Panthera pardus\nn02128757 snow leopard, ounce, Panthera uncia\nn02128925 jaguar, panther, Panthera onca, Felis onca\nn02129165 lion, king of beasts, Panthera leo\nn02129604 tiger, Panthera tigris\nn02130308 cheetah, chetah, Acinonyx jubatus\nn02132136 brown bear, bruin, Ursus arctos\nn02133161 American black bear, black bear, Ursus americanus, Euarctos americanus\nn02134084 ice bear, polar bear, Ursus Maritimus, Thalarctos maritimus\nn02134418 sloth bear, Melursus ursinus, Ursus ursinus\nn02137549 mongoose\nn02138441 meerkat, mierkat\nn02165105 tiger beetle\nn02165456 ladybug, ladybeetle, lady beetle, ladybird, ladybird beetle\nn02167151 ground beetle, carabid beetle\nn02168699 long-horned beetle, longicorn, longicorn beetle\nn02169497 leaf beetle, chrysomelid\nn02172182 dung beetle\nn02174001 rhinoceros beetle\nn02177972 weevil\nn02190166 fly\nn02206856 bee\nn02219486 ant, emmet, pismire\nn02226429 grasshopper, hopper\nn02229544 cricket\nn02231487 walking stick, walkingstick, stick insect\nn02233338 cockroach, roach\nn02236044 mantis, mantid\nn02256656 cicada, cicala\nn02259212 leafhopper\nn02264363 lacewing, lacewing fly\nn02268443 dragonfly, darning needle, devil's darning needle, sewing needle, snake feeder, snake doctor, mosquito hawk, skeeter hawk\nn02268853 damselfly\nn02276258 admiral\nn02277742 ringlet, ringlet butterfly\nn02279972 monarch, monarch butterfly, milkweed butterfly, Danaus plexippus\nn02280649 cabbage butterfly\nn02281406 sulphur butterfly, sulfur butterfly\nn02281787 lycaenid, lycaenid butterfly\nn02317335 starfish, sea star\nn02319095 sea urchin\nn02321529 sea cucumber, holothurian\nn02325366 wood rabbit, cottontail, cottontail rabbit\nn02326432 hare\nn02328150 Angora, Angora rabbit\nn02342885 hamster\nn02346627 porcupine, hedgehog\nn02356798 fox squirrel, eastern fox squirrel, Sciurus niger\nn02361337 marmot\nn02363005 beaver\nn02364673 guinea pig, Cavia cobaya\nn02389026 sorrel\nn02391049 zebra\nn02395406 hog, pig, grunter, squealer, Sus scrofa\nn02396427 wild boar, boar, Sus scrofa\nn02397096 warthog\nn02398521 hippopotamus, hippo, river horse, Hippopotamus amphibius\nn02403003 ox\nn02408429 water buffalo, water ox, Asiatic buffalo, Bubalus bubalis\nn02410509 bison\nn02412080 ram, tup\nn02415577 bighorn, bighorn sheep, cimarron, Rocky Mountain bighorn, Rocky Mountain sheep, Ovis canadensis\nn02417914 ibex, Capra ibex\nn02422106 hartebeest\nn02422699 impala, Aepyceros melampus\nn02423022 gazelle\nn02437312 Arabian camel, dromedary, Camelus dromedarius\nn02437616 llama\nn02441942 weasel\nn02442845 mink\nn02443114 polecat, fitch, foulmart, foumart, Mustela putorius\nn02443484 black-footed ferret, ferret, Mustela nigripes\nn02444819 otter\nn02445715 skunk, polecat, wood pussy\nn02447366 badger\nn02454379 armadillo\nn02457408 three-toed sloth, ai, Bradypus tridactylus\nn02480495 orangutan, orang, orangutang, Pongo pygmaeus\nn02480855 gorilla, Gorilla gorilla\nn02481823 chimpanzee, chimp, Pan troglodytes\nn02483362 gibbon, Hylobates lar\nn02483708 siamang, Hylobates syndactylus, Symphalangus syndactylus\nn02484975 guenon, guenon monkey\nn02486261 patas, hussar monkey, Erythrocebus patas\nn02486410 baboon\nn02487347 macaque\nn02488291 langur\nn02488702 colobus, colobus monkey\nn02489166 proboscis monkey, Nasalis larvatus\nn02490219 marmoset\nn02492035 capuchin, ringtail, Cebus capucinus\nn02492660 howler monkey, howler\nn02493509 titi, titi monkey\nn02493793 spider monkey, Ateles geoffroyi\nn02494079 squirrel monkey, Saimiri sciureus\nn02497673 Madagascar cat, ring-tailed lemur, Lemur catta\nn02500267 indri, indris, Indri indri, Indri brevicaudatus\nn02504013 Indian elephant, Elephas maximus\nn02504458 African elephant, Loxodonta africana\nn02509815 lesser panda, red panda, panda, bear cat, cat bear, Ailurus fulgens\nn02510455 giant panda, panda, panda bear, coon bear, Ailuropoda melanoleuca\nn02514041 barracouta, snoek\nn02526121 eel\nn02536864 coho, cohoe, coho salmon, blue jack, silver salmon, Oncorhynchus kisutch\nn02606052 rock beauty, Holocanthus tricolor\nn02607072 anemone fish\nn02640242 sturgeon\nn02641379 gar, garfish, garpike, billfish, Lepisosteus osseus\nn02643566 lionfish\nn02655020 puffer, pufferfish, blowfish, globefish\nn02666196 abacus\nn02667093 abaya\nn02669723 academic gown, academic robe, judge's robe\nn02672831 accordion, piano accordion, squeeze box\nn02676566 acoustic guitar\nn02687172 aircraft carrier, carrier, flattop, attack aircraft carrier\nn02690373 airliner\nn02692877 airship, dirigible\nn02699494 altar\nn02701002 ambulance\nn02704792 amphibian, amphibious vehicle\nn02708093 analog clock\nn02727426 apiary, bee house\nn02730930 apron\nn02747177 ashcan, trash can, garbage can, wastebin, ash bin, ash-bin, ashbin, dustbin, trash barrel, trash bin\nn02749479 assault rifle, assault gun\nn02769748 backpack, back pack, knapsack, packsack, rucksack, haversack\nn02776631 bakery, bakeshop, bakehouse\nn02777292 balance beam, beam\nn02782093 balloon\nn02783161 ballpoint, ballpoint pen, ballpen, Biro\nn02786058 Band Aid\nn02787622 banjo\nn02788148 bannister, banister, balustrade, balusters, handrail\nn02790996 barbell\nn02791124 barber chair\nn02791270 barbershop\nn02793495 barn\nn02794156 barometer\nn02795169 barrel, cask\nn02797295 barrow, garden cart, lawn cart, wheelbarrow\nn02799071 baseball\nn02802426 basketball\nn02804414 bassinet\nn02804610 bassoon\nn02807133 bathing cap, swimming cap\nn02808304 bath towel\nn02808440 bathtub, bathing tub, bath, tub\nn02814533 beach wagon, station wagon, wagon, estate car, beach waggon, station waggon, waggon\nn02814860 beacon, lighthouse, beacon light, pharos\nn02815834 beaker\nn02817516 bearskin, busby, shako\nn02823428 beer bottle\nn02823750 beer glass\nn02825657 bell cote, bell cot\nn02834397 bib\nn02835271 bicycle-built-for-two, tandem bicycle, tandem\nn02837789 bikini, two-piece\nn02840245 binder, ring-binder\nn02841315 binoculars, field glasses, opera glasses\nn02843684 birdhouse\nn02859443 boathouse\nn02860847 bobsled, bobsleigh, bob\nn02865351 bolo tie, bolo, bola tie, bola\nn02869837 bonnet, poke bonnet\nn02870880 bookcase\nn02871525 bookshop, bookstore, bookstall\nn02877765 bottlecap\nn02879718 bow\nn02883205 bow tie, bow-tie, bowtie\nn02892201 brass, memorial tablet, plaque\nn02892767 brassiere, bra, bandeau\nn02894605 breakwater, groin, groyne, mole, bulwark, seawall, jetty\nn02895154 breastplate, aegis, egis\nn02906734 broom\nn02909870 bucket, pail\nn02910353 buckle\nn02916936 bulletproof vest\nn02917067 bullet train, bullet\nn02927161 butcher shop, meat market\nn02930766 cab, hack, taxi, taxicab\nn02939185 caldron, cauldron\nn02948072 candle, taper, wax light\nn02950826 cannon\nn02951358 canoe\nn02951585 can opener, tin opener\nn02963159 cardigan\nn02965783 car mirror\nn02966193 carousel, carrousel, merry-go-round, roundabout, whirligig\nn02966687 carpenter's kit, tool kit\nn02971356 carton\nn02974003 car wheel\nn02977058 cash machine, cash dispenser, automated teller machine, automatic teller machine, automated teller, automatic teller, ATM\nn02978881 cassette\nn02979186 cassette player\nn02980441 castle\nn02981792 catamaran\nn02988304 CD player\nn02992211 cello, violoncello\nn02992529 cellular telephone, cellular phone, cellphone, cell, mobile phone\nn02999410 chain\nn03000134 chainlink fence\nn03000247 chain mail, ring mail, mail, chain armor, chain armour, ring armor, ring armour\nn03000684 chain saw, chainsaw\nn03014705 chest\nn03016953 chiffonier, commode\nn03017168 chime, bell, gong\nn03018349 china cabinet, china closet\nn03026506 Christmas stocking\nn03028079 church, church building\nn03032252 cinema, movie theater, movie theatre, movie house, picture palace\nn03041632 cleaver, meat cleaver, chopper\nn03042490 cliff dwelling\nn03045698 cloak\nn03047690 clog, geta, patten, sabot\nn03062245 cocktail shaker\nn03063599 coffee mug\nn03063689 coffeepot\nn03065424 coil, spiral, volute, whorl, helix\nn03075370 combination lock\nn03085013 computer keyboard, keypad\nn03089624 confectionery, confectionary, candy store\nn03095699 container ship, containership, container vessel\nn03100240 convertible\nn03109150 corkscrew, bottle screw\nn03110669 cornet, horn, trumpet, trump\nn03124043 cowboy boot\nn03124170 cowboy hat, ten-gallon hat\nn03125729 cradle\nn03126707 crane\nn03127747 crash helmet\nn03127925 crate\nn03131574 crib, cot\nn03133878 Crock Pot\nn03134739 croquet ball\nn03141823 crutch\nn03146219 cuirass\nn03160309 dam, dike, dyke\nn03179701 desk\nn03180011 desktop computer\nn03187595 dial telephone, dial phone\nn03188531 diaper, nappy, napkin\nn03196217 digital clock\nn03197337 digital watch\nn03201208 dining table, board\nn03207743 dishrag, dishcloth\nn03207941 dishwasher, dish washer, dishwashing machine\nn03208938 disk brake, disc brake\nn03216828 dock, dockage, docking facility\nn03218198 dogsled, dog sled, dog sleigh\nn03220513 dome\nn03223299 doormat, welcome mat\nn03240683 drilling platform, offshore rig\nn03249569 drum, membranophone, tympan\nn03250847 drumstick\nn03255030 dumbbell\nn03259280 Dutch oven\nn03271574 electric fan, blower\nn03272010 electric guitar\nn03272562 electric locomotive\nn03290653 entertainment center\nn03291819 envelope\nn03297495 espresso maker\nn03314780 face powder\nn03325584 feather boa, boa\nn03337140 file, file cabinet, filing cabinet\nn03344393 fireboat\nn03345487 fire engine, fire truck\nn03347037 fire screen, fireguard\nn03355925 flagpole, flagstaff\nn03372029 flute, transverse flute\nn03376595 folding chair\nn03379051 football helmet\nn03384352 forklift\nn03388043 fountain\nn03388183 fountain pen\nn03388549 four-poster\nn03393912 freight car\nn03394916 French horn, horn\nn03400231 frying pan, frypan, skillet\nn03404251 fur coat\nn03417042 garbage truck, dustcart\nn03424325 gasmask, respirator, gas helmet\nn03425413 gas pump, gasoline pump, petrol pump, island dispenser\nn03443371 goblet\nn03444034 go-kart\nn03445777 golf ball\nn03445924 golfcart, golf cart\nn03447447 gondola\nn03447721 gong, tam-tam\nn03450230 gown\nn03452741 grand piano, grand\nn03457902 greenhouse, nursery, glasshouse\nn03459775 grille, radiator grille\nn03461385 grocery store, grocery, food market, market\nn03467068 guillotine\nn03476684 hair slide\nn03476991 hair spray\nn03478589 half track\nn03481172 hammer\nn03482405 hamper\nn03483316 hand blower, blow dryer, blow drier, hair dryer, hair drier\nn03485407 hand-held computer, hand-held microcomputer\nn03485794 handkerchief, hankie, hanky, hankey\nn03492542 hard disc, hard disk, fixed disk\nn03494278 harmonica, mouth organ, harp, mouth harp\nn03495258 harp\nn03496892 harvester, reaper\nn03498962 hatchet\nn03527444 holster\nn03529860 home theater, home theatre\nn03530642 honeycomb\nn03532672 hook, claw\nn03534580 hoopskirt, crinoline\nn03535780 horizontal bar, high bar\nn03538406 horse cart, horse-cart\nn03544143 hourglass\nn03584254 iPod\nn03584829 iron, smoothing iron\nn03590841 jack-o'-lantern\nn03594734 jean, blue jean, denim\nn03594945 jeep, landrover\nn03595614 jersey, T-shirt, tee shirt\nn03598930 jigsaw puzzle\nn03599486 jinrikisha, ricksha, rickshaw\nn03602883 joystick\nn03617480 kimono\nn03623198 knee pad\nn03627232 knot\nn03630383 lab coat, laboratory coat\nn03633091 ladle\nn03637318 lampshade, lamp shade\nn03642806 laptop, laptop computer\nn03649909 lawn mower, mower\nn03657121 lens cap, lens cover\nn03658185 letter opener, paper knife, paperknife\nn03661043 library\nn03662601 lifeboat\nn03666591 lighter, light, igniter, ignitor\nn03670208 limousine, limo\nn03673027 liner, ocean liner\nn03676483 lipstick, lip rouge\nn03680355 Loafer\nn03690938 lotion\nn03691459 loudspeaker, speaker, speaker unit, loudspeaker system, speaker system\nn03692522 loupe, jeweler's loupe\nn03697007 lumbermill, sawmill\nn03706229 magnetic compass\nn03709823 mailbag, postbag\nn03710193 mailbox, letter box\nn03710637 maillot\nn03710721 maillot, tank suit\nn03717622 manhole cover\nn03720891 maraca\nn03721384 marimba, xylophone\nn03724870 mask\nn03729826 matchstick\nn03733131 maypole\nn03733281 maze, labyrinth\nn03733805 measuring cup\nn03742115 medicine chest, medicine cabinet\nn03743016 megalith, megalithic structure\nn03759954 microphone, mike\nn03761084 microwave, microwave oven\nn03763968 military uniform\nn03764736 milk can\nn03769881 minibus\nn03770439 miniskirt, mini\nn03770679 minivan\nn03773504 missile\nn03775071 mitten\nn03775546 mixing bowl\nn03776460 mobile home, manufactured home\nn03777568 Model T\nn03777754 modem\nn03781244 monastery\nn03782006 monitor\nn03785016 moped\nn03786901 mortar\nn03787032 mortarboard\nn03788195 mosque\nn03788365 mosquito net\nn03791053 motor scooter, scooter\nn03792782 mountain bike, all-terrain bike, off-roader\nn03792972 mountain tent\nn03793489 mouse, computer mouse\nn03794056 mousetrap\nn03796401 moving van\nn03803284 muzzle\nn03804744 nail\nn03814639 neck brace\nn03814906 necklace\nn03825788 nipple\nn03832673 notebook, notebook computer\nn03837869 obelisk\nn03838899 oboe, hautboy, hautbois\nn03840681 ocarina, sweet potato\nn03841143 odometer, hodometer, mileometer, milometer\nn03843555 oil filter\nn03854065 organ, pipe organ\nn03857828 oscilloscope, scope, cathode-ray oscilloscope, CRO\nn03866082 overskirt\nn03868242 oxcart\nn03868863 oxygen mask\nn03871628 packet\nn03873416 paddle, boat paddle\nn03874293 paddlewheel, paddle wheel\nn03874599 padlock\nn03876231 paintbrush\nn03877472 pajama, pyjama, pj's, jammies\nn03877845 palace\nn03884397 panpipe, pandean pipe, syrinx\nn03887697 paper towel\nn03888257 parachute, chute\nn03888605 parallel bars, bars\nn03891251 park bench\nn03891332 parking meter\nn03895866 passenger car, coach, carriage\nn03899768 patio, terrace\nn03902125 pay-phone, pay-station\nn03903868 pedestal, plinth, footstall\nn03908618 pencil box, pencil case\nn03908714 pencil sharpener\nn03916031 perfume, essence\nn03920288 Petri dish\nn03924679 photocopier\nn03929660 pick, plectrum, plectron\nn03929855 pickelhaube\nn03930313 picket fence, paling\nn03930630 pickup, pickup truck\nn03933933 pier\nn03935335 piggy bank, penny bank\nn03937543 pill bottle\nn03938244 pillow\nn03942813 ping-pong ball\nn03944341 pinwheel\nn03947888 pirate, pirate ship\nn03950228 pitcher, ewer\nn03954731 plane, carpenter's plane, woodworking plane\nn03956157 planetarium\nn03958227 plastic bag\nn03961711 plate rack\nn03967562 plow, plough\nn03970156 plunger, plumber's helper\nn03976467 Polaroid camera, Polaroid Land camera\nn03976657 pole\nn03977966 police van, police wagon, paddy wagon, patrol wagon, wagon, black Maria\nn03980874 poncho\nn03982430 pool table, billiard table, snooker table\nn03983396 pop bottle, soda bottle\nn03991062 pot, flowerpot\nn03992509 potter's wheel\nn03995372 power drill\nn03998194 prayer rug, prayer mat\nn04004767 printer\nn04005630 prison, prison house\nn04008634 projectile, missile\nn04009552 projector\nn04019541 puck, hockey puck\nn04023962 punching bag, punch bag, punching ball, punchball\nn04026417 purse\nn04033901 quill, quill pen\nn04033995 quilt, comforter, comfort, puff\nn04037443 racer, race car, racing car\nn04039381 racket, racquet\nn04040759 radiator\nn04041544 radio, wireless\nn04044716 radio telescope, radio reflector\nn04049303 rain barrel\nn04065272 recreational vehicle, RV, R.V.\nn04067472 reel\nn04069434 reflex camera\nn04070727 refrigerator, icebox\nn04074963 remote control, remote\nn04081281 restaurant, eating house, eating place, eatery\nn04086273 revolver, six-gun, six-shooter\nn04090263 rifle\nn04099969 rocking chair, rocker\nn04111531 rotisserie\nn04116512 rubber eraser, rubber, pencil eraser\nn04118538 rugby ball\nn04118776 rule, ruler\nn04120489 running shoe\nn04125021 safe\nn04127249 safety pin\nn04131690 saltshaker, salt shaker\nn04133789 sandal\nn04136333 sarong\nn04141076 sax, saxophone\nn04141327 scabbard\nn04141975 scale, weighing machine\nn04146614 school bus\nn04147183 schooner\nn04149813 scoreboard\nn04152593 screen, CRT screen\nn04153751 screw\nn04154565 screwdriver\nn04162706 seat belt, seatbelt\nn04179913 sewing machine\nn04192698 shield, buckler\nn04200800 shoe shop, shoe-shop, shoe store\nn04201297 shoji\nn04204238 shopping basket\nn04204347 shopping cart\nn04208210 shovel\nn04209133 shower cap\nn04209239 shower curtain\nn04228054 ski\nn04229816 ski mask\nn04235860 sleeping bag\nn04238763 slide rule, slipstick\nn04239074 sliding door\nn04243546 slot, one-armed bandit\nn04251144 snorkel\nn04252077 snowmobile\nn04252225 snowplow, snowplough\nn04254120 soap dispenser\nn04254680 soccer ball\nn04254777 sock\nn04258138 solar dish, solar collector, solar furnace\nn04259630 sombrero\nn04263257 soup bowl\nn04264628 space bar\nn04265275 space heater\nn04266014 space shuttle\nn04270147 spatula\nn04273569 speedboat\nn04275548 spider web, spider's web\nn04277352 spindle\nn04285008 sports car, sport car\nn04286575 spotlight, spot\nn04296562 stage\nn04310018 steam locomotive\nn04311004 steel arch bridge\nn04311174 steel drum\nn04317175 stethoscope\nn04325704 stole\nn04326547 stone wall\nn04328186 stopwatch, stop watch\nn04330267 stove\nn04332243 strainer\nn04335435 streetcar, tram, tramcar, trolley, trolley car\nn04336792 stretcher\nn04344873 studio couch, day bed\nn04346328 stupa, tope\nn04347754 submarine, pigboat, sub, U-boat\nn04350905 suit, suit of clothes\nn04355338 sundial\nn04355933 sunglass\nn04356056 sunglasses, dark glasses, shades\nn04357314 sunscreen, sunblock, sun blocker\nn04366367 suspension bridge\nn04367480 swab, swob, mop\nn04370456 sweatshirt\nn04371430 swimming trunks, bathing trunks\nn04371774 swing\nn04372370 switch, electric switch, electrical switch\nn04376876 syringe\nn04380533 table lamp\nn04389033 tank, army tank, armored combat vehicle, armoured combat vehicle\nn04392985 tape player\nn04398044 teapot\nn04399382 teddy, teddy bear\nn04404412 television, television system\nn04409515 tennis ball\nn04417672 thatch, thatched roof\nn04418357 theater curtain, theatre curtain\nn04423845 thimble\nn04428191 thresher, thrasher, threshing machine\nn04429376 throne\nn04435653 tile roof\nn04442312 toaster\nn04443257 tobacco shop, tobacconist shop, tobacconist\nn04447861 toilet seat\nn04456115 torch\nn04458633 totem pole\nn04461696 tow truck, tow car, wrecker\nn04462240 toyshop\nn04465501 tractor\nn04467665 trailer truck, tractor trailer, trucking rig, rig, articulated lorry, semi\nn04476259 tray\nn04479046 trench coat\nn04482393 tricycle, trike, velocipede\nn04483307 trimaran\nn04485082 tripod\nn04486054 triumphal arch\nn04487081 trolleybus, trolley coach, trackless trolley\nn04487394 trombone\nn04493381 tub, vat\nn04501370 turnstile\nn04505470 typewriter keyboard\nn04507155 umbrella\nn04509417 unicycle, monocycle\nn04515003 upright, upright piano\nn04517823 vacuum, vacuum cleaner\nn04522168 vase\nn04523525 vault\nn04525038 velvet\nn04525305 vending machine\nn04532106 vestment\nn04532670 viaduct\nn04536866 violin, fiddle\nn04540053 volleyball\nn04542943 waffle iron\nn04548280 wall clock\nn04548362 wallet, billfold, notecase, pocketbook\nn04550184 wardrobe, closet, press\nn04552348 warplane, military plane\nn04553703 washbasin, handbasin, washbowl, lavabo, wash-hand basin\nn04554684 washer, automatic washer, washing machine\nn04557648 water bottle\nn04560804 water jug\nn04562935 water tower\nn04579145 whiskey jug\nn04579432 whistle\nn04584207 wig\nn04589890 window screen\nn04590129 window shade\nn04591157 Windsor tie\nn04591713 wine bottle\nn04592741 wing\nn04596742 wok\nn04597913 wooden spoon\nn04599235 wool, woolen, woollen\nn04604644 worm fence, snake fence, snake-rail fence, Virginia fence\nn04606251 wreck\nn04612504 yawl\nn04613696 yurt\nn06359193 web site, website, internet site, site\nn06596364 comic book\nn06785654 crossword puzzle, crossword\nn06794110 street sign\nn06874185 traffic light, traffic signal, stoplight\nn07248320 book jacket, dust cover, dust jacket, dust wrapper\nn07565083 menu\nn07579787 plate\nn07583066 guacamole\nn07584110 consomme\nn07590611 hot pot, hotpot\nn07613480 trifle\nn07614500 ice cream, icecream\nn07615774 ice lolly, lolly, lollipop, popsicle\nn07684084 French loaf\nn07693725 bagel, beigel\nn07695742 pretzel\nn07697313 cheeseburger\nn07697537 hotdog, hot dog, red hot\nn07711569 mashed potato\nn07714571 head cabbage\nn07714990 broccoli\nn07715103 cauliflower\nn07716358 zucchini, courgette\nn07716906 spaghetti squash\nn07717410 acorn squash\nn07717556 butternut squash\nn07718472 cucumber, cuke\nn07718747 artichoke, globe artichoke\nn07720875 bell pepper\nn07730033 cardoon\nn07734744 mushroom\nn07742313 Granny Smith\nn07745940 strawberry\nn07747607 orange\nn07749582 lemon\nn07753113 fig\nn07753275 pineapple, ananas\nn07753592 banana\nn07754684 jackfruit, jak, jack\nn07760859 custard apple\nn07768694 pomegranate\nn07802026 hay\nn07831146 carbonara\nn07836838 chocolate sauce, chocolate syrup\nn07860988 dough\nn07871810 meat loaf, meatloaf\nn07873807 pizza, pizza pie\nn07875152 potpie\nn07880968 burrito\nn07892512 red wine\nn07920052 espresso\nn07930864 cup\nn07932039 eggnog\nn09193705 alp\nn09229709 bubble\nn09246464 cliff, drop, drop-off\nn09256479 coral reef\nn09288635 geyser\nn09332890 lakeside, lakeshore\nn09399592 promontory, headland, head, foreland\nn09421951 sandbar, sand bar\nn09428293 seashore, coast, seacoast, sea-coast\nn09468604 valley, vale\nn09472597 volcano\nn09835506 ballplayer, baseball player\nn10148035 groom, bridegroom\nn10565667 scuba diver\nn11879895 rapeseed\nn11939491 daisy\nn12057211 yellow lady's slipper, yellow lady-slipper, Cypripedium calceolus, Cypripedium parviflorum\nn12144580 corn\nn12267677 acorn\nn12620546 hip, rose hip, rosehip\nn12768682 buckeye, horse chestnut, conker\nn12985857 coral fungus\nn12998815 agaric\nn13037406 gyromitra\nn13040303 stinkhorn, carrion fungus\nn13044778 earthstar\nn13052670 hen-of-the-woods, hen of the woods, Polyporus frondosus, Grifola frondosa\nn13054560 bolete\nn13133613 ear, spike, capitulum\nn15075141 toilet tissue, toilet paper, bathroom tissue\n" }, { "path": "examples/gluon_character_cnn/README.md", "content": "# Character-level CNN Model in Gluon trained using Amazon Product Dataset\n\nIn this example, we show how to create a service which classifies a review into product type using [Character-level Convolutional Network Model (CNN) model](https://papers.nips.cc/paper/5782-character-level-convolutional-networks-for-text-classification.pdf) model by Yann LeCunn. This model is trained on [Amazon product data](http://jmcauley.ucsd.edu/data/amazon/) and training detail can be found in a detailed tutorial from Thomas Delteil on [Character CNN training.](https://github.com/ThomasDelteil/CNN_NLP_MXNet).\n\n\n# Step by step to create service\n\n## Step 1 - Download the Gluon Char CNN model file, model parameter and classification labels file to \"/tmp/crepe\"\n\n```bash\n# Create a model directory\nmkdir /tmp/crepe\n\n# Download the model file\ncurl -O https://raw.githubusercontent.com/awslabs/multi-model-server/master/examples/gluon_character_cnn/gluon_crepe.py\n\n# Download the parameters\ncurl -O https://s3.amazonaws.com/model-server/model_archive_1.0/examples/mms-char-cnn-files/crepe_gluon_epoch6.params\n\n# Download classification labels file\ncurl -O https://s3.amazonaws.com/model-server/model_archive_1.0/examples/mms-char-cnn-files/synset.txt\n\n# Move required files to the following folder\nmv crepe_gluon_epoch6.params gluon_crepe.py synset.txt /tmp/crepe\n```\n\n## Step 2 - Look at the Gluon model/service file\n\nFor Gluon models on MMS, the models are defined, within the MMS service file, the skeletal structure of the file looks like follows.\n\n```python\nclass GluonCrepe(HybridBlock):\n \"\"\"\n Hybrid Block gluon Crepe model\n \"\"\"\n def __init__(self, classes=7, **kwargs):\n ## Define model below\n pass\n\nclass CharacterCNNService(object):\n \"\"\"\n Gluon Character-level Convolution Service\n \"\"\"\n\n def __init__(self):\n # The 69 characters as specified in the paper\n self.ALPHABET = list(\"abcdefghijklmnopqrstuvwxyz0123456789-,;.!?:'\\\"/\\\\|_@#$%^&*~`+ =<>()[]{}\")\n # Map Alphabets to index\n self.ALPHABET_INDEX = {letter: index for index, letter in enumerate(self.ALPHABET)}\n # max-length in characters for one document\n self.FEATURE_LEN = 1014\n self.initialized = False\n\n def initialize(self, params):\n self.net = GluonCrepe()\n self.param_filename = \"crepe_gluon_epoch6.params\"\n self.model_name = params.manifest[\"model\"][\"modelName\"]\n\n gpu_id = params.system_properties.get(\"gpu_id\")\n model_dir = params.system_properties.get(\"model_dir\")\n\n synset_file = os.path.join(model_dir, \"synset.txt\")\n param_file_path = os.path.join(model_dir, self.param_filename)\n if not os.path.isfile(param_file_path):\n raise OSError(\"Parameter file not found {}\".format(param_file_path))\n if not os.path.isfile(synset_file):\n raise OSError(\"synset file not available {}\".format(synset_file))\n\n self.ctx = mx.cpu() if gpu_id is None else mx.gpu(gpu_id)\n\n self.net.load_parameters(param_file_path, self.ctx)\n\n self.labels = [line.strip() for line in open(synset_file).readlines()]\n self.initialized = True\n self.net.hybridize(static_shape=True, static_alloc=True)\n\n # define preprocess, inference and postprocess methods\n```\n\nAs shown, the Gluon model derives from the basic gluon hybrid block. Gluon hybrid blocks, provide performance of a symbolic model with a imperative model. More on Gluon, hybrid blocks [here](https://gluon.mxnet.io/chapter07_distributed-learning/hybridize.html).\nThe fully defined service file can be found under [gluon_crepe.py](gluon_crepe.py), we define `preprocess`, `inference`, `postprocess` methods in this file.\n\nThe input size is, limited to 1014, characters as mentioned in the paper. The output is of shape [0,7] as we classify the reviews into seven product categories. Both the input and output are passed on as 'application/json' based text content.\n\n# Step 3 - Prepare synset.txt with list of class names\n\n[synset.txt](synset.txt) is where we define list of all classes detected by the model. The pre-trained Character-level CNN model used in the example is trained to detect 7 classes including Books, CDs_and_Vinyl, Movies_and_TV and more. See synset.txt file for list of all classes.\n\nThe list of classes in synset.txt will be loaded by MMS as list of labels in inference logic.\n\n\n## Step 4 - Export model files with mxnet-model-export CLI tool\n\nWith model file together with signature and files in the model folder, we are ready to export them to MMS model file.\n\n```bash\nmodel-archiver --model-name crepe -f --model-path /tmp/crepe/ --handler gluon_crepe:crepe_inference --runtime python --export-path /tmp\n```\n\nA packaged model can be downloaded from [here.](https://s3.amazonaws.com/model-server/model_archive_1.0/examples/mms-char-cnn-files/crepe.mar)\n\n## Step 5 - Establish inference service\n\n`crepe.mar` file is created by exporting model files. We also defined custom service under gluon_crepe.py. We are ready to establish the Character-level CNN inference service:\n\n```bash\nmulti-model-server --models crepe.mar --model-store /tmp\n```\n\nThe endpoint is on localhost and port 8080. You can change them by passing --host and --port when establishing the service.\n\n## Test inference service\n\nNow we can send post requests to the endpoint we just established.\n\n\nThe key values of application/json input are 'review_title', 'review'. This can be a different value or combined to a single input , to achieve this preprocess method in gluon_crepe.py needs to be modified.\n\nLet's take up a movie, review\n\n```bash\ncurl -X POST http://127.0.0.1:8080/predictions/crepe -F \"data=[{\\\"review_title\\\":\\\"Inception is the best\\\",\\\"review\\\": \\\"great direction and story\\\"}]\"\n```\nPrediction result will be:\n\n```json\n{\n \"confidence\": {\n \"Clothing_Shoes_and_Jewelry\": 0.004,\n \"Home_and_Kitchen\": 0.001,\n \"Sports_and_Outdoors\": 0.001,\n \"CDs_and_Vinyl\": 0.038,\n \"Movies_and_TV\": 0.59,\n \"Cell_Phones_and_Accessories\": 0.0,\n \"Books\": 0.362\n },\n \"predicted\": \"Movies_and_TV\"\n }\n```\n\nLet's try another review, this time for a music album.\n\n```bash\ncurl -X POST http://127.0.0.1:8080/predictions/crepe -F \"data=[{\\\"review_title\\\":\\\"fantastic quality\\\",\\\"review\\\": \\\"quality sound playback\\\"}]\"\n```\n\nPrediction result will be:\n\n```json\n{\n \"confidence\": {\n \"Clothing_Shoes_and_Jewelry\": 0.028,\n \"Home_and_Kitchen\": 0.012,\n \"Sports_and_Outdoors\": 0.028,\n \"CDs_and_Vinyl\": 0.727,\n \"Movies_and_TV\": 0.118,\n \"Cell_Phones_and_Accessories\": 0.068,\n \"Books\": 0.015\n },\n \"predicted\": \"CDs_and_Vinyl\"\n}\n```\n\nReferences\n1. [Character-level CNN](https://papers.nips.cc/paper/5782-character-level-convolutional-networks-for-text-classification.pdf)\n2. [How to train Character-level CNN on gluon](https://github.com/ThomasDelteil/TextClassificationCNNs_MXNet)\n3. [Web Demo of Character-level CNN on gluon](https://thomasdelteil.github.io/TextClassificationCNNs_MXNet/)\n" }, { "path": "examples/gluon_character_cnn/gluon_crepe.py", "content": "# Copyright 2018 Amazon.com, Inc. or its affiliates. All Rights Reserved.\n# Licensed under the Apache License, Version 2.0 (the \"License\").\n# You may not use this file except in compliance with the License.\n# A copy of the License is located at\n# http://www.apache.org/licenses/LICENSE-2.0\n# or in the \"license\" file accompanying this file. This file is distributed\n# on an \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either\n# express or implied. See the License for the specific language governing\n# permissions and limitations under the License.\n\nimport ast\nimport os\n\nimport mxnet as mx\nfrom mxnet import gluon, nd\nfrom mxnet.gluon import nn\nfrom mxnet.gluon.block import HybridBlock\nimport numpy as np\n\n\nclass GluonCrepe(HybridBlock):\n \"\"\"\n Hybrid Block gluon Crepe model\n \"\"\"\n\n def __init__(self, classes=7, **kwargs):\n super(GluonCrepe, self).__init__(**kwargs)\n self.NUM_FILTERS = 256 # number of convolutional filters per convolutional layer\n self.NUM_OUTPUTS = classes # number of classes\n self.FULLY_CONNECTED = 1024 # number of unit in the fully connected dense layer\n self.features = nn.HybridSequential()\n with self.name_scope():\n self.features.add(\n nn.Conv1D(channels=self.NUM_FILTERS, kernel_size=7, activation='relu'),\n nn.MaxPool1D(pool_size=3, strides=3),\n nn.Conv1D(channels=self.NUM_FILTERS, kernel_size=7, activation='relu'),\n nn.MaxPool1D(pool_size=3, strides=3),\n nn.Conv1D(channels=self.NUM_FILTERS, kernel_size=3, activation='relu'),\n nn.Conv1D(channels=self.NUM_FILTERS, kernel_size=3, activation='relu'),\n nn.Conv1D(channels=self.NUM_FILTERS, kernel_size=3, activation='relu'),\n nn.Conv1D(channels=self.NUM_FILTERS, kernel_size=3, activation='relu'),\n nn.MaxPool1D(pool_size=3, strides=3),\n nn.Flatten(),\n nn.Dense(self.FULLY_CONNECTED, activation='relu'),\n nn.Dense(self.FULLY_CONNECTED, activation='relu'),\n )\n self.output = nn.Dense(self.NUM_OUTPUTS)\n\n def hybrid_forward(self, F, x):\n x = self.features(x)\n x = self.output(x)\n return x\n\n\nclass CharacterCNNService(object):\n \"\"\"\n Gluon Character-level Convolution Service\n \"\"\"\n\n def __init__(self):\n # The 69 characters as specified in the paper\n self.ALPHABET = list(\"abcdefghijklmnopqrstuvwxyz0123456789-,;.!?:'\\\"/\\\\|_@#$%^&*~`+ =<>()[]{}\")\n # Map Alphabets to index\n self.ALPHABET_INDEX = {letter: index for index, letter in enumerate(self.ALPHABET)}\n # max-length in characters for one document\n self.FEATURE_LEN = 1014\n self.initialized = False\n\n def initialize(self, params):\n self.net = GluonCrepe()\n self.param_filename = \"crepe_gluon_epoch6.params\"\n self.model_name = params.manifest[\"model\"][\"modelName\"]\n\n gpu_id = params.system_properties.get(\"gpu_id\")\n model_dir = params.system_properties.get(\"model_dir\")\n\n synset_file = os.path.join(model_dir, \"synset.txt\")\n param_file_path = os.path.join(model_dir, self.param_filename)\n if not os.path.isfile(param_file_path):\n raise OSError(\"Parameter file not found {}\".format(param_file_path))\n if not os.path.isfile(synset_file):\n raise OSError(\"synset file not available {}\".format(synset_file))\n\n self.ctx = mx.cpu() if gpu_id is None else mx.gpu(gpu_id)\n\n self.net.load_parameters(param_file_path, self.ctx)\n\n self.labels = [line.strip() for line in open(synset_file).readlines()]\n self.initialized = True\n self.net.hybridize(static_shape=True, static_alloc=True)\n\n def preprocess(self, data):\n \"\"\"\n Pre-process text to a encode it to a form, that gives spatial information to the CNN\n \"\"\"\n # build the text from the request\n if data[0].get('data') is not None:\n data = ast.literal_eval(data[0].get('data').decode('utf-8'))\n text = '{}|{}'.format(data[0].get('review_title'), data[0].get('review'))\n\n encoded = np.zeros([len(self.ALPHABET), self.FEATURE_LEN], dtype='float32')\n review = text.lower()[:self.FEATURE_LEN - 1:-1]\n i = 0\n for letter in text:\n if i >= self.FEATURE_LEN:\n break;\n if letter in self.ALPHABET_INDEX:\n encoded[self.ALPHABET_INDEX[letter]][i] = 1\n i += 1\n return nd.array([encoded], ctx=self.ctx)\n\n def inference(self, data):\n # Call forward/hybrid_forward\n output = self.net(data)\n return output.softmax()\n\n def postprocess(self, data):\n # Post process and output the most likely category\n data = data[0]\n values = {val: float(int(data[i].asnumpy() * 1000) / 1000.0) for i, val in enumerate(self.labels)}\n index = int(nd.argmax(data, axis=0).asnumpy()[0])\n predicted = self.labels[index]\n return [{'predicted': predicted, 'confidence': values}]\n\n def predict(self, data):\n data = self.preprocess(data)\n data = self.inference(data)\n return self.postprocess(data)\n\n\nsvc = CharacterCNNService()\n\n\ndef crepe_inference(data, context):\n res = \"\"\n if not svc.initialized:\n svc.initialize(context)\n\n if data is not None:\n res = svc.predict(data)\n\n return res\n" }, { "path": "examples/gluon_character_cnn/signature.json", "content": "{\n \"inputs\": [\n {\n \"data_name\": \"data\",\n \"data_shape\": [1,1014]\n }\n ],\n \"input_type\": \"application/json\",\n \"outputs\": [\n {\n \"data_name\": \"softmax\",\n \"data_shape\": [0, 7]\n }\n ],\n \"output_type\": \"application/json\"\n}\n" }, { "path": "examples/gluon_character_cnn/synset.txt", "content": "Home_and_Kitchen\nBooks\nCDs_and_Vinyl\nMovies_and_TV\nCell_Phones_and_Accessories\nSports_and_Outdoors\nClothing_Shoes_and_Jewelry\n" }, { "path": "examples/lstm_ptb/README.md", "content": "# Sequence to Sequence inference with LSTM network trained on PenTreeBank data set\n\nIn this example, we show how to create a service which generates sentences with a pre-trained LSTM model with deep model server. This model is trained on [PenTreeBank data](https://catalog.ldc.upenn.edu/ldc99t42) and training detail can be found in [MXNet example](https://github.com/apache/incubator-mxnet/tree/master/example/rnn).\n\nThis model uses [MXNet Bucketing Module](https://mxnet.incubator.apache.org/how_to/bucketing.html) to deal with variable length input sentences and generates output sentences with the same length as inputs.\n\n# Step by step to create service\n\n## Step 1 - Download the pre-trained LSTM model files, signature file and vocabulary dictionary file\n\n```bash\ncd multi-model-server/examples/lstm_ptb\n\ncurl -O https://s3.amazonaws.com/model-server/models/lstm_ptb/lstm_ptb-symbol.json\ncurl -O https://s3.amazonaws.com/model-server/models/lstm_ptb/lstm_ptb-0100.params\ncurl -O https://s3.amazonaws.com/model-server/models/lstm_ptb/vocab_dict.txt\ncurl -O https://s3.amazonaws.com/model-server/models/lstm_ptb/signature.json\n```\n\n## Step 2 - Verify signature file\n\nIn this example, provided mxnet_vision_service.py template assume there is a `signature.json` file that describes input parameter and shape.\n\nAfter [Step 1](#step-1---download-the-pre-trained-lstm-model-files,-signature-file-and-vocabulary-dictionary-file) there should be a signature file in the lstm_ptb folder. Verify that this file exists before proceeding further.\n\nThe signature file looks as follows.\n\n```json\n{\n \"inputs\": [\n {\n \"data_name\": \"data\",\n \"data_shape\": [\n 1,\n 60\n ],\n ...\n }\n ]\n}\n```\nInput data shape is (1, 60). For sequence to sequence models, the inputs can be variable length sequences. In the signature file the input shape should be set to the maximum length of the input sequence, which is the default bucket key. The bucket sizes are defined when training the model. In this example valid bucket sizes are 10, 20, 30, 40, 50 and 60. Default bucket key is the maximum value which is 60. \nCheck [bucketing module tutorials](https://mxnet.incubator.apache.org/faq/bucketing.html) if you want to know more about the bucketing module in MXNet.\n\n## Step 3 - Check vocabulary dictionary file\n\n[vocab_dict.txt](https://s3.amazonaws.com/model-server/models/lstm_ptb/vocab_dict.txt) is to store word to integer indexing information. In this example, each line in the text file represents a (word, index) pair. This file can be in different format and requires different customized parsing methods respectively.\n\n## Step 4 - Create custom service class\n\nWe provide custom service class template code in [model_service_template](../model_service_template) folder:\n1. [model_handler.py](../model_service_template/model_handler.py) - A generic based service class.\n2. [mxnet_utils](../model_service_template/mxnet_utils) - A python package that contains utility classes.\n\n```bash\ncd multi-model-server/examples\n\ncp model_service_template/model_handler.py lstm_ptb/\ncp -r model_service_template/mxnet_utils lstm_ptb/\n```\n\nIn this example, we need to implement `preprocess`, `inference` and `postprocess` methods in a custom service class. Implementation details are in [lstm_ptb_service.py](lstm_ptb_service.py).\n\n## Step 5 - Package the model with `model-archiver` CLI utility\n\nIn this step, we package the following:\n1. pre-trained MXNet Model we downloaded in Step 1.\n2. '[signature.json](signature.json)' file we prepared in step 2.\n3. '[vocab_dict.txt](vocab_dict.txt)' file we prepared in step 3.\n4. custom model service files we prepared in step 4.\n\nWe use `model-archiver` command line utility (CLI) provided by MMS.\nInstall `model-archiver` in case you have not:\n\n```bash\npip install model-archiver\n```\n\nThis tool creates a .mar file that will be provided to MMS for serving inference requests. In following command line, we specify 'lstm_ptb_service:handle' as model archive entry point.\n\n```bash\ncd multi-model-server/examples\nmodel-archiver --model-name lstm_ptb --model-path lstm_ptb --handler lstm_ptb_service:handle\n```\n\n## Step 6 - Start the Inference Service\n\nStart the inference service by providing the 'lstm_ptb.mar' file we created in Step 5.\n\nBy default, the server is started on the localhost at port 8080.\n\n```bash\ncd multi-model-server\n\nmulti-model-server --start --model-store examples --models lstm_ptb.mar\n```\n\n## Test inference service\n\nNow we can send post requests to the endpoint we just established.\n\nSince the entire range of vocabularies in the training set is only 10,000, you may not get very good results with arbitrary test sentences. Instead, we recommend that you test with sentences from the [PTB test data set](https://raw.githubusercontent.com/dmlc/web-data/master/mxnet/ptb/ptb.test.txt). That being said, if you try some random text you should know that any word that isn't in that 10k vocabulary is encoded with an \"invalid label\" of 0. This will create a prediction result of '\\n'. Note that in PTB data set, person name is represented by ``.\n\nThe key value of application/json input is 'input_sentence'. This can be a different value and preprocess method in lstm_ptb_service.py needs to be modified respectively. \n\n```bash\ncurl -X POST http://127.0.0.1:8080/predictions/lstm_ptb -H \"Content-Type: application/json\" -d '[{\"input_sentence\": \"on the exchange floor as soon as ual stopped trading we for a panic said one top floor trader\"}]'\n```\n\nPrediction result will be:\n\n```json\n{\n \"prediction\": \"the 's the the as the 's the the 're to a analyst company trading at \"\n}\n```\n\nLet's try another sentence:\n\n```bash\ncurl -X POST http://127.0.0.1:8080/predictions/lstm_ptb -H \"Content-Type: application/json\" -d '[{\"input_sentence\": \"while friday '\\''s debacle involved mainly professional traders rather than investors it left the market vulnerable to continued selling this morning traders said \"}]'\n```\n\nPrediction result will be:\n\n```json\n{\n \"prediction\": \"the 's stock were in say than were will to to to the the week \\n \\n \\n \\n \\n \\n \\n \\n \\n \\n \"\n}\n```\n\nReferences\n1. [How to use MXNet bucketing module](https://mxnet.incubator.apache.org/how_to/bucketing.html)\n2. [LSTM trained with PennTreeBank data set](https://github.com/apache/incubator-mxnet/tree/master/example/rnn)\n" }, { "path": "examples/lstm_ptb/lstm_ptb_service.py", "content": "import json\nimport os\n\nimport mxnet as mx\n\nfrom mxnet_utils import nlp\nfrom model_handler import ModelHandler\n\n\nclass MXNetLSTMService(ModelHandler):\n \"\"\"\n MXNetLSTMService service class. This service consumes a sentence\n from length 0 to 60 and generates a sentence with the same size.\n \"\"\"\n\n def __init__(self):\n super(MXNetLSTMService, self).__init__()\n self.mxnet_ctx = None\n self.mx_model = None\n self.labels = None\n self.signature = None\n self.data_names = None\n self.data_shapes = None\n self.epoch = 100\n\n self.buckets = [10, 20, 30, 40, 50, 60]\n self.start_label = 1\n self.invalid_key = \"\\n\"\n self.invalid_label = 0\n self.layout = \"NT\"\n self.vocab = {}\n self.idx2word = {}\n\n def initialize(self, context):\n super(MXNetLSTMService, self).initialize(context)\n\n properties = context.system_properties\n model_dir = properties.get(\"model_dir\")\n gpu_id = properties.get(\"gpu_id\")\n batch_size = properties.get(\"batch_size\")\n if batch_size > 1:\n raise ValueError(\"Batch is not supported.\")\n\n # reading signature.json file\n signature_file_path = os.path.join(model_dir, \"signature.json\")\n if not os.path.isfile(signature_file_path):\n raise RuntimeError(\"Missing signature.json file.\")\n\n with open(signature_file_path) as f:\n self.signature = json.load(f)\n\n self.data_names = []\n self.data_shapes = []\n for input_data in self.signature[\"inputs\"]:\n self.data_names.append(input_data[\"data_name\"])\n self.data_shapes.append((input_data['data_name'], tuple(input_data['data_shape'])))\n\n # reading vocab_dict.txt file\n vocab_dict_file = os.path.join(model_dir, \"vocab_dict.txt\")\n with open(vocab_dict_file, 'r') as vocab_file:\n self.vocab[self.invalid_key] = self.invalid_label\n for line in vocab_file:\n word_index = line.split(' ')\n if len(word_index) < 2 or word_index[0] == '':\n continue\n self.vocab[word_index[0]] = int(word_index[1].rstrip())\n for key, val in self.vocab.items():\n self.idx2word[val] = key\n\n # Load pre-trained lstm bucketing module\n num_layers = 2\n num_hidden = 200\n num_embed = 200\n\n stack = mx.rnn.FusedRNNCell(num_hidden, num_layers=num_layers, mode=\"lstm\").unfuse()\n\n # Define symbol generation function for bucket module\n def sym_gen(seq_len):\n data = mx.sym.Variable(\"data\")\n embed = mx.sym.Embedding(data=data, input_dim=len(self.vocab),\n output_dim=num_embed, name=\"embed\")\n\n stack.reset()\n outputs, _ = stack.unroll(seq_len, inputs=embed, merge_outputs=True)\n\n pred = mx.sym.Reshape(outputs, shape=(-1, num_hidden))\n pred = mx.sym.FullyConnected(data=pred, num_hidden=len(self.vocab), name=\"pred\")\n pred = mx.sym.softmax(pred, name='softmax')\n\n return pred, ('data',), None\n\n self.mxnet_ctx = mx.cpu() if gpu_id is None else mx.gpu(gpu_id)\n\n # Create bucketing module and load weights\n self.mx_model = mx.mod.BucketingModule(\n sym_gen=sym_gen,\n default_bucket_key=max(self.buckets),\n context=self.mxnet_ctx)\n\n checkpoint_prefix = \"{}/{}\".format(model_dir, \"lstm_ptb\")\n\n self.mx_model.bind(data_shapes=self.data_shapes, for_training=False)\n\n _, arg_params, aux_params = mx.rnn.load_rnn_checkpoint(stack, checkpoint_prefix, self.epoch)\n self.mx_model.set_params(arg_params, aux_params)\n\n def preprocess(self, data):\n \"\"\"\n This service doesn't support batch, always get data from first item.\n\n :param data:\n :return:\n \"\"\"\n input_data = data[0].get(\"data\")\n if input_data is None:\n input_data = data[0].get(\"body\")\n\n # Convert a string of sentence to a list of string\n sent = input_data[0][\"input_sentence\"].lower().split(\" \")\n assert len(sent) <= self.buckets[-1], \"Sentence length must be no greater than %d.\" % (self.buckets[-1])\n # Encode sentence to a list of int\n res, _ = nlp.encode_sentences(\n [sent], vocab=self.vocab, start_label=self.start_label,\n invalid_label=self.invalid_label)\n\n return res\n\n def inference(self, data):\n data_batch = nlp.pad_sentence(\n data[0], self.buckets, invalid_label=self.invalid_label,\n data_name=self.data_names[0], layout=self.layout)\n self.mx_model.forward(data_batch)\n return self.mx_model.get_outputs()\n\n def postprocess(self, data):\n # Generate predicted sentences\n word_idx = mx.nd.argmax(data[0], axis=1).asnumpy()\n res = \"\"\n for idx in word_idx:\n res += self.idx2word[idx] + \" \"\n\n ret = {\"prediction\": res}\n return [ret]\n\n\n# Following code is not necessary if your service class contains `handle(self, data, context)` function\n_service = MXNetLSTMService()\n\n\ndef handle(data, context):\n if not _service.initialized:\n _service.initialize(context)\n\n if data is None:\n return None\n\n return _service.handle(data, context)\n" }, { "path": "examples/metrics_cloudwatch/__init__.py", "content": "" }, { "path": "examples/metrics_cloudwatch/metric_push_example.py", "content": "# Copyright 2018 Amazon.com, Inc. or its affiliates. All Rights Reserved.\n# Licensed under the Apache License, Version 2.0 (the \"License\").\n# You may not use this file except in compliance with the License.\n# A copy of the License is located at\n# http://www.apache.org/licenses/LICENSE-2.0\n# or in the \"license\" file accompanying this file. This file is distributed\n# on an \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either\n# express or implied. See the License for the specific language governing\n# permissions and limitations under the License.\n\n\"\"\"\nExamples for pushing a log to boto client to cloudwatch\n\"\"\"\nimport types\nimport json\n\nimport boto3 as boto\n\nfrom mms.metrics import system_metrics as sys_metric\nfrom mms.metrics.metric_encoder import MetricEncoder\n\n\ndef generate_system_metrics(mod):\n \"\"\"\n Function acting as a stub for reading a log file, produces similar result\n :param mod:\n :return:\n \"\"\"\n members = dir(mod)\n for i in members:\n value = getattr(mod, i)\n if isinstance(value, types.FunctionType) and value.__name__ != 'collect_all':\n value()\n\n return json.dumps(sys_metric.system_metrics, indent=4, separators=(',', ':'), cls=MetricEncoder)\n\n\ndef push_cloudwatch(metric_json, client):\n \"\"\"\n push metric to cloud watch, do some processing.\n :param metric_json:\n :param client:\n :return:\n \"\"\"\n metrics = json.loads(metric_json)\n cloud_metrics = []\n for metric in metrics:\n cloud_metric = {}\n for key in metric.keys():\n if key != 'RequestId' or key != 'HostName':\n cloud_metric[key] = metric[key]\n cloud_metrics.append(cloud_metric)\n client.put_metric_data(\n Namespace='MXNetModelServer',\n MetricData=cloud_metrics\n )\n\n\ndef connect_cloudwatch():\n client = None\n try:\n client = boto.client('cloudwatch')\n except Exception as e: # pylint: disable=broad-except\n print(str(e))\n\n return client\n\n\nif __name__ == '__main__':\n # Replace this with a log reader\n json_val = generate_system_metrics(sys_metric)\n cloud_client = connect_cloudwatch()\n if cloud_client is not None:\n push_cloudwatch(json_val, cloud_client)\n" }, { "path": "examples/model_service_template/gluon_base_service.py", "content": "# Copyright 2018 Amazon.com, Inc. or its affiliates. All Rights Reserved.\n# Licensed under the Apache License, Version 2.0 (the \"License\").\n# You may not use this file except in compliance with the License.\n# A copy of the License is located at\n# http://www.apache.org/licenses/LICENSE-2.0\n# or in the \"license\" file accompanying this file. This file is distributed\n# on an \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either\n# express or implied. See the License for the specific language governing\n# permissions and limitations under the License.\n\n\"\"\"\nGluon Base service defines a Gluon base service for generic CNN\n\"\"\"\nimport mxnet as mx\nimport numpy as np\nimport os\nimport json\nimport ndarray\n\n\nclass GluonBaseService(object):\n \"\"\"GluonBaseService defines a fundamental service for image classification task.\n In preprocess, input image buffer is read to NDArray and resized respect to input\n shape in signature.\n In post process, top-5 labels are returned.\n \"\"\"\n\n def __init__(self):\n self.param_filename = None\n self.model_name = None\n self.initialized = False\n self.ctx = None\n self.net = None\n self._signature = None\n self.labels = None\n self.signature = None\n\n def initialize(self, params):\n \"\"\"\n Initialization of the network\n :param params: This is the :func `Context` object\n :return:\n \"\"\"\n if self.net is None:\n raise NotImplementedError(\"Gluon network not defined\")\n sys_prop = params.system_properties\n gpu_id = sys_prop.get(\"gpu_id\")\n model_dir = sys_prop.get(\"model_dir\")\n self.model_name = params.manifest[\"model\"][\"modelName\"]\n self.ctx = mx.cpu() if gpu_id is None else mx.gpu(gpu_id)\n\n if self.param_filename is not None:\n param_file_path = os.path.join(model_dir, self.param_filename)\n if not os.path.isfile(param_file_path):\n raise OSError(\"Parameter file not found {}\".format(param_file_path))\n self.net.load_parameters(param_file_path, self.ctx)\n\n synset_file = os.path.join(model_dir, \"synset.txt\")\n signature_file_path = os.path.join(model_dir, \"signature.json\")\n\n if not os.path.isfile(signature_file_path):\n raise OSError(\"Signature file not found {}\".format(signature_file_path))\n\n if not os.path.isfile(synset_file):\n raise OSError(\"synset file not available {}\".format(synset_file))\n\n with open(signature_file_path) as sig_file:\n self.signature = json.load(sig_file)\n \n self.labels = [line.strip() for line in open(synset_file).readlines()]\n self.initialized = True\n\n def preprocess(self, data):\n \"\"\"\n This method considers only one input data\n\n :param data: Data is list of map\n format is\n [\n {\n \"parameterName\": name\n \"parameterValue\": data\n },\n {...}\n ]\n :return:\n \"\"\"\n\n param_name = self.signature['inputs'][0]['data_name']\n input_shape = self.signature['inputs'][0]['data_shape']\n\n img = data[0].get(param_name)\n\n if img is None:\n raise IOError(\"Invalid parameter given\")\n\n # We are assuming input shape is NCHW\n [h, w] = input_shape[2:]\n img_arr = mx.img.imdecode(img)\n img_arr = mx.image.imresize(img_arr, w, h)\n img_arr = img_arr.astype(np.float32)\n img_arr /= 255\n img_arr = mx.image.color_normalize(img_arr,\n mean=mx.nd.array([0.485, 0.456, 0.406]),\n std=mx.nd.array([0.229, 0.224, 0.225]))\n img_arr = mx.nd.transpose(img_arr, (2, 0, 1))\n img_arr = img_arr.expand_dims(axis=0)\n return img_arr\n\n def inference(self, data):\n \"\"\"\n Internal inference methods for MMS service. Run forward computation and\n return output.\n\n Parameters\n ----------\n data : list of NDArray\n Preprocessed inputs in NDArray format.\n\n Returns\n -------\n list of NDArray\n Inference output.\n \"\"\"\n model_input = data.as_in_context(self.ctx)\n output = self.net(model_input)\n return output.softmax()\n\n def postprocess(self, data):\n assert hasattr(self, 'labels'), \\\n \"Can't find labels attribute. Did you put synset.txt file into \" \\\n \"model archive or manually load class label file in __init__?\"\n return [[ndarray.top_probability(d, self.labels, top=5) for d in data]]\n\n def predict(self, data):\n data = self.preprocess(data)\n data = self.inference(data)\n return self.postprocess(data)\n" }, { "path": "examples/model_service_template/model_handler.py", "content": "# Copyright 2018 Amazon.com, Inc. or its affiliates. All Rights Reserved.\n# Licensed under the Apache License, Version 2.0 (the \"License\").\n# You may not use this file except in compliance with the License.\n# A copy of the License is located at\n# http://www.apache.org/licenses/LICENSE-2.0\n# or in the \"license\" file accompanying this file. This file is distributed\n# on an \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either\n# express or implied. See the License for the specific language governing\n# permissions and limitations under the License.\n\n\"\"\"\nModelHandler defines a base model handler.\n\"\"\"\nimport logging\nimport time\n\n\nclass ModelHandler(object):\n \"\"\"\n A base Model handler implementation.\n \"\"\"\n\n def __init__(self):\n self.error = None\n self._context = None\n self._batch_size = 0\n self.initialized = False\n\n def initialize(self, context):\n \"\"\"\n Initialize model. This will be called during model loading time\n\n :param context: Initial context contains model server system properties.\n :return:\n \"\"\"\n self._context = context\n self._batch_size = context.system_properties[\"batch_size\"]\n self.initialized = True\n\n def preprocess(self, batch):\n \"\"\"\n Transform raw input into model input data.\n\n :param batch: list of raw requests, should match batch size\n :return: list of preprocessed model input data\n \"\"\"\n assert self._batch_size == len(batch), \"Invalid input batch size: {}\".format(len(batch))\n return None\n\n def inference(self, model_input):\n \"\"\"\n Internal inference methods\n\n :param model_input: transformed model input data\n :return: list of inference output in NDArray\n \"\"\"\n return None\n\n def postprocess(self, inference_output):\n \"\"\"\n Return predict result in batch.\n\n :param inference_output: list of inference output\n :return: list of predict results\n \"\"\"\n return [\"OK\"] * self._batch_size\n\n def handle(self, data, context):\n \"\"\"\n Custom service entry point function.\n\n :param data: list of objects, raw input from request\n :param context: model server context\n :return: list of outputs to be send back to client\n \"\"\"\n self.error = None # reset earlier errors\n\n try:\n preprocess_start = time.time()\n data = self.preprocess(data)\n inference_start = time.time()\n data = self.inference(data)\n postprocess_start = time.time()\n data = self.postprocess(data)\n end_time = time.time()\n\n metrics = context.metrics\n metrics.add_time(\"PreprocessTime\", round((inference_start - preprocess_start) * 1000, 2))\n metrics.add_time(\"InferenceTime\", round((postprocess_start - inference_start) * 1000, 2))\n metrics.add_time(\"PostprocessTime\", round((end_time - postprocess_start) * 1000, 2))\n\n return data\n except Exception as e:\n logging.error(e, exc_info=True)\n request_processor = context.request_processor\n request_processor.report_status(500, \"Unknown inference error\")\n return [str(e)] * self._batch_size\n" }, { "path": "examples/model_service_template/mxnet_model_service.py", "content": "# Copyright 2018 Amazon.com, Inc. or its affiliates. All Rights Reserved.\n# Licensed under the Apache License, Version 2.0 (the \"License\").\n# You may not use this file except in compliance with the License.\n# A copy of the License is located at\n# http://www.apache.org/licenses/LICENSE-2.0\n# or in the \"license\" file accompanying this file. This file is distributed\n# on an \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either\n# express or implied. See the License for the specific language governing\n# permissions and limitations under the License.\n\n\"\"\"\nMXNetModelService defines an API for MXNet service.\n\"\"\"\nimport json\nimport os\n\nimport mxnet as mx\nfrom mxnet.io import DataBatch\n\nfrom model_handler import ModelHandler\n\n\nclass MXNetModelService(ModelHandler):\n \"\"\"\n MXNetBaseService defines the fundamental loading model and inference\n operations when serving MXNet model. This is a base class and needs to be\n inherited.\n \"\"\"\n\n def __init__(self):\n super(MXNetModelService, self).__init__()\n self.mxnet_ctx = None\n self.mx_model = None\n self.labels = None\n self.signature = None\n self.epoch = 0\n\n # noinspection PyMethodMayBeStatic\n def get_model_files_prefix(self, context):\n return context.manifest[\"model\"][\"modelName\"]\n\n def initialize(self, context):\n \"\"\"\n Initialize model. This will be called during model loading time\n\n :param context: Initial context contains model server system properties.\n :return:\n \"\"\"\n super(MXNetModelService, self).initialize(context)\n\n assert self._batch_size == 1, \"Batch is not supported.\"\n\n properties = context.system_properties\n model_dir = properties.get(\"model_dir\")\n gpu_id = properties.get(\"gpu_id\")\n\n signature_file_path = os.path.join(model_dir, \"signature.json\")\n if not os.path.isfile(signature_file_path):\n raise RuntimeError(\"Missing signature.json file.\")\n\n with open(signature_file_path) as f:\n self.signature = json.load(f)\n\n model_files_prefix = self.get_model_files_prefix(context)\n archive_synset = os.path.join(model_dir, \"synset.txt\")\n if os.path.isfile(archive_synset):\n synset = archive_synset\n self.labels = [line.strip() for line in open(synset).readlines()]\n\n data_names = []\n data_shapes = []\n for input_data in self.signature[\"inputs\"]:\n data_name = input_data[\"data_name\"]\n data_shape = input_data[\"data_shape\"]\n\n # Set batch size\n data_shape[0] = self._batch_size\n\n # Replace 0 entry in data shape with 1 for binding executor.\n for idx in range(len(data_shape)):\n if data_shape[idx] == 0:\n data_shape[idx] = 1\n\n data_names.append(data_name)\n data_shapes.append((data_name, tuple(data_shape)))\n\n checkpoint_prefix = \"{}/{}\".format(model_dir, model_files_prefix)\n\n # Load MXNet module\n self.mxnet_ctx = mx.cpu() if gpu_id is None else mx.gpu(gpu_id)\n sym, arg_params, aux_params = mx.model.load_checkpoint(checkpoint_prefix, self.epoch)\n\n # noinspection PyTypeChecker\n self.mx_model = mx.mod.Module(symbol=sym, context=self.mxnet_ctx,\n data_names=data_names, label_names=None)\n self.mx_model.bind(for_training=False, data_shapes=data_shapes)\n self.mx_model.set_params(arg_params, aux_params, allow_missing=True, allow_extra=True)\n\n def preprocess(self, batch):\n \"\"\"\n Transform raw input into model input data.\n\n :param batch: list of raw requests, should match batch size\n :return: list of preprocessed model input data\n \"\"\"\n assert self._batch_size == len(batch), \"Invalid input batch size: {}\".format(len(batch))\n\n ret = []\n param_name = self.signature['inputs'][0]['data_name']\n\n for idx, request in enumerate(batch):\n data = request.get(param_name)\n if data is None:\n data = request.get(\"body\")\n\n if data is None:\n data = request.get(\"data\")\n\n ret.append(map(mx.nd.array, data))\n\n return ret\n\n def inference(self, model_input):\n \"\"\"\n Internal inference methods for MXNet. Run forward computation and\n return output.\n\n :param model_input: list of NDArray\n Preprocessed inputs in NDArray format.\n :return: list of NDArray\n Inference output.\n \"\"\"\n if self.error is not None:\n return None\n\n # Check input shape\n check_input_shape(model_input, self.signature)\n model_input = [item.as_in_context(self.mxnet_ctx) for item in model_input]\n self.mx_model.forward(DataBatch(model_input))\n model_input = self.mx_model.get_outputs()\n # by pass lazy evaluation get_outputs either returns a list of nd arrays\n # a list of list of NDArray\n for d in model_input:\n if isinstance(d, list):\n for n in model_input:\n if isinstance(n, mx.ndarray.ndarray.NDArray):\n n.wait_to_read()\n elif isinstance(d, mx.ndarray.ndarray.NDArray):\n d.wait_to_read()\n return model_input\n\n def postprocess(self, inference_output):\n if self.error is not None:\n return [self.error] * self._batch_size\n\n return [str(d.asnumpy().tolist()) for d in inference_output]\n\n\ndef check_input_shape(inputs, signature):\n \"\"\"\n Check input data shape consistency with signature.\n\n Parameters\n ----------\n inputs : List of NDArray\n Input data in NDArray format.\n signature : dict\n Dictionary containing model signature.\n \"\"\"\n assert isinstance(inputs, list), 'Input data must be a list.'\n assert len(inputs) == len(signature['inputs']), \\\n \"Input number mismatches with \" \\\n \"signature. %d expected but got %d.\" \\\n % (len(signature['inputs']), len(inputs))\n for input_data, sig_input in zip(inputs, signature[\"inputs\"]):\n assert isinstance(input_data, mx.nd.NDArray), 'Each input must be NDArray.'\n assert len(input_data.shape) == len(sig_input[\"data_shape\"]), \\\n 'Shape dimension of input %s mismatches with ' \\\n 'signature. %d expected but got %d.' \\\n % (sig_input['data_name'],\n len(sig_input['data_shape']),\n len(input_data.shape))\n for idx in range(len(input_data.shape)):\n if idx != 0 and sig_input['data_shape'][idx] != 0:\n assert sig_input['data_shape'][idx] == input_data.shape[idx], \\\n 'Input %s has different shape with ' \\\n 'signature. %s expected but got %s.' \\\n % (sig_input['data_name'], sig_input['data_shape'],\n input_data.shape)\n" }, { "path": "examples/model_service_template/mxnet_utils/__init__.py", "content": "# Copyright 2017 Amazon.com, Inc. or its affiliates. All Rights Reserved.\n# Licensed under the Apache License, Version 2.0 (the \"License\").\n# You may not use this file except in compliance with the License.\n# A copy of the License is located at\n# http://www.apache.org/licenses/LICENSE-2.0\n# or in the \"license\" file accompanying this file. This file is distributed\n# on an \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either\n# express or implied. See the License for the specific language governing\n# permissions and limitations under the License.\n\n\"\"\"\nMXNet Utils\n\"\"\"\n" }, { "path": "examples/model_service_template/mxnet_utils/image.py", "content": "# Copyright 2017 Amazon.com, Inc. or its affiliates. All Rights Reserved.\n# Licensed under the Apache License, Version 2.0 (the \"License\").\n# You may not use this file except in compliance with the License.\n# A copy of the License is located at\n# http://www.apache.org/licenses/LICENSE-2.0\n# or in the \"license\" file accompanying this file. This file is distributed\n# on an \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either\n# express or implied. See the License for the specific language governing\n# permissions and limitations under the License.\n\n\"\"\"\nImage utils\n\"\"\"\nimport base64\nimport sys\nfrom io import BytesIO\n\nimport mxnet as mx\nimport numpy as np\nfrom PIL import Image\nfrom mxnet import image as img\n\n\ndef transform_shape(img_arr, dim_order='NCHW'):\n \"\"\"\n Rearrange image NDArray shape to 'NCHW' or 'NHWC' which\n is valid for MXNet model input.\n Input image NDArray should has dim_order of 'HWC'.\n\n :param img_arr: NDArray\n Image in NDArray format with shape (channel, width, height)\n :param dim_order: str\n Output image dimension order. Valid values are 'NCHW' and 'NHWC'\n\n :return: NDArray\n Image in NDArray format with dim_order shape\n \"\"\"\n assert dim_order in 'NCHW' or dim_order in 'NHWC', \"dim_order must be 'NCHW' or 'NHWC'.\"\n if dim_order == 'NCHW':\n img_arr = mx.nd.transpose(img_arr, (2, 0, 1))\n output = mx.nd.expand_dims(img_arr, axis=0)\n return output\n\n\ndef read(buf, flag=1, to_rgb=True, out=None):\n \"\"\"\n Read and decode an image to an NDArray.\n Input image NDArray should has dim_order of 'HWC'.\n\n Note: `imread` uses OpenCV (not the CV2 Python library).\n MXNet must have been built with USE_OPENCV=1 for `imdecode` to work.\n\n :param buf: str/bytes or numpy.ndarray\n Binary image data as string or numpy ndarray.\n :param flag: {0, 1}, default 1\n 1 for three channel color output. 0 for grayscale output.\n :param to_rgb: bool, default True\n True for RGB formatted output (MXNet default).\n False for BGR formatted output (OpenCV default).\n :param out: NDArray, optional\n Output buffer. Use `None` for automatic allocation.\n :return: NDArray\n An `NDArray` containing the image.\n\n Example\n -------\n >>> buf = open(\"flower.jpg\", 'rb').read()\n >>> image.read(buf)\n \n \"\"\"\n return img.imdecode(buf, flag, to_rgb, out)\n\n\ndef write(img_arr, flag=1, output_format='jpeg', dim_order='CHW'):\n \"\"\"\n Write an NDArray to a base64 string.\n\n :param img_arr: NDArray\n Image in NDArray format with shape (channel, width, height).\n :param flag: {0, 1}, default 1\n 1 for three channel color output. 0 for grayscale output.\n :param output_format: str\n Output image format.\n :param dim_order: str\n Input image dimension order. Valid values are 'CHW' and 'HWC'\n :return: str\n Image in base64 string format\n \"\"\"\n assert dim_order in 'CHW' or dim_order in 'HWC', \"dim_order must be 'CHW' or 'HWC'.\"\n if dim_order == 'CHW':\n img_arr = mx.nd.transpose(img_arr, (1, 2, 0))\n if flag == 1:\n mode = 'RGB'\n else:\n mode = 'L'\n img_arr = mx.nd.reshape(img_arr, (img_arr.shape[0], img_arr.shape[1]))\n img_arr = img_arr.astype(np.uint8).asnumpy()\n image = Image.fromarray(img_arr, mode)\n output = BytesIO()\n image.save(output, format=output_format)\n output.seek(0)\n if sys.version_info[0] < 3:\n return base64.b64encode(output.getvalue())\n else:\n return base64.b64encode(output.getvalue()).decode(\"utf-8\")\n\n\ndef resize(src, new_width, new_height, interp=2):\n \"\"\"\n Resizes image to new_width and new_height.\n Input image NDArray should has dim_order of 'HWC'.\n\n :param src: NDArray\n Source image in NDArray format\n :param new_width: int\n Width in pixel for resized image\n :param new_height: int\n Height in pixel for resized image\n :param interp: int\n interpolation method for all resizing operations\n\n Possible values:\n 0: Nearest Neighbors Interpolation.\n 1: Bilinear interpolation.\n 2: Area-based (resampling using pixel area relation). It may be a\n preferred method for image decimation, as it gives moire-free\n results. But when the image is zoomed, it is similar to the Nearest\n Neighbors method. (used by default).\n 3: Bicubic interpolation over 4x4 pixel neighborhood.\n 4: Lanczos interpolation over 8x8 pixel neighborhood.\n 9: Cubic for enlarge, area for shrink, bilinear for others\n 10: Random select from interpolation method metioned above.\n Note:\n When shrinking an image, it will generally look best with AREA-based\n interpolation, whereas, when enlarging an image, it will generally look best\n with Bicubic (slow) or Bilinear (faster but still looks OK).\n More details can be found in the documentation of OpenCV, please refer to\n http://docs.opencv.org/master/da/d54/group__imgproc__transform.html.\n :return: NDArray\n An `NDArray` containing the resized image.\n \"\"\"\n return img.imresize(src, new_width, new_height, interp)\n\n\ndef fixed_crop(src, x0, y0, w, h, size=None, interp=2):\n \"\"\"\n Crop src at fixed location, and (optionally) resize it to size.\n Input image NDArray should has dim_order of 'HWC'.\n\n :param src: NDArray\n Input image\n :param x0: int\n Left boundary of the cropping area\n :param y0 : int\n Top boundary of the cropping area\n :param w : int\n Width of the cropping area\n :param h : int\n Height of the cropping area\n :param size : tuple of (w, h)\n Optional, resize to new size after cropping\n :param interp : int, optional, default=2\n Interpolation method. See resize for details.\n :return: NDArray\n An `NDArray` containing the cropped image.\n \"\"\"\n return img.fixed_crop(src, x0, y0, w, h, size, interp)\n\n\ndef color_normalize(src, mean, std=None):\n \"\"\"\n Normalize src with mean and std.\n\n :param src : NDArray\n Input image\n :param mean : NDArray\n RGB mean to be subtracted\n :param std : NDArray\n RGB standard deviation to be divided\n :return: NDArray\n An `NDArray` containing the normalized image.\n \"\"\"\n src = src.astype(np.float32)\n return img.color_normalize(src, mean, std)\n" }, { "path": "examples/model_service_template/mxnet_utils/ndarray.py", "content": "# Copyright 2017 Amazon.com, Inc. or its affiliates. All Rights Reserved.\n# Licensed under the Apache License, Version 2.0 (the \"License\").\n# You may not use this file except in compliance with the License.\n# A copy of the License is located at\n# http://www.apache.org/licenses/LICENSE-2.0\n# or in the \"license\" file accompanying this file. This file is distributed\n# on an \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either\n# express or implied. See the License for the specific language governing\n# permissions and limitations under the License.\n\n\"\"\"\nNDArray utils\n\"\"\"\nimport mxnet as mx\nimport numpy as np\n\n\ndef top_probability(data, labels, top=5):\n \"\"\"\n Get top probability prediction from NDArray.\n\n :param data: NDArray\n Data to be predicted\n :param labels: List\n List of class labels\n :param top:\n :return: List\n List of probability: class pairs in sorted order\n \"\"\"\n dim = len(data.shape)\n if dim > 2:\n data = mx.nd.array(\n np.squeeze(data.asnumpy(), axis=tuple(range(dim)[2:])))\n sorted_prob = mx.nd.argsort(data[0], is_ascend=False)\n # pylint: disable=deprecated-lambda\n top_prob = map(lambda x: int(x.asscalar()), sorted_prob[0:top])\n return [{'probability': float(data[0, i].asscalar()), 'class': labels[i]}\n for i in top_prob]\n" }, { "path": "examples/model_service_template/mxnet_utils/nlp.py", "content": "# Copyright 2017 Amazon.com, Inc. or its affiliates. All Rights Reserved.\n# Licensed under the Apache License, Version 2.0 (the \"License\").\n# You may not use this file except in compliance with the License.\n# A copy of the License is located at\n# http://www.apache.org/licenses/LICENSE-2.0\n# or in the \"license\" file accompanying this file. This file is distributed\n# on an \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either\n# express or implied. See the License for the specific language governing\n# permissions and limitations under the License.\n\n\"\"\"\nNLP utils\n\"\"\"\nimport bisect\n\nimport mxnet as mx\nimport numpy as np\n\n\ndef encode_sentences(sentences, vocab=None, invalid_label=-1, invalid_key='\\n', start_label=0):\n \"\"\"\n Encode sentences and (optionally) build a mapping\n from string tokens to integer indices. Unknown keys\n will be added to vocabulary.\n\n :param sentences: list of list of str\n A list of sentences to encode. Each sentence\n should be a list of string tokens.\n :param vocab: None or dict of str -> int\n Optional input Vocabulary\n :param invalid_label: int, default -1\n Index for invalid token, like \n :param invalid_key: str, default '\\\\n'\n Key for invalid token. Use '\\\\n' for end\n of sentence by default.\n :param start_label: int\n lowest index.\n\n :returns:\n res : list of list of int\n encoded sentences\n vocab : dict of str -> int\n result vocabulary\n \"\"\"\n idx = start_label\n if vocab is None:\n vocab = {invalid_key: invalid_label}\n new_vocab = True\n else:\n new_vocab = False\n res = []\n for sent in sentences:\n coded = []\n for word in sent:\n if word not in vocab:\n if not new_vocab:\n coded.append(invalid_label)\n continue\n else:\n if idx == invalid_label:\n idx += 1\n vocab[word] = idx\n idx += 1\n coded.append(vocab[word])\n res.append(coded)\n\n return res, vocab\n\n\ndef pad_sentence(sentence, buckets, invalid_label=-1, data_name='data', layout='NT'):\n \"\"\"\n Pad a sentence to closest length in provided buckets.\n\n :param sentence: list of int\n A list of integer representing an encoded sentence.\n :param buckets: list of int\n Size of the data buckets.\n :param invalid_label: int, optional\n Index for invalid token, like .\n :param data_name: str, optional\n Input data name.\n :param layout: str, optional\n Format of data and label. 'NT' means (batch_size, length)\n and 'TN' means (length, batch_size).\n\n :return: mx.io.DataBatch\n DataBatch contains sentence.\n \"\"\"\n buck = bisect.bisect_left(buckets, len(sentence))\n buff = np.full((buckets[buck],), invalid_label, dtype='float32')\n buff[:len(sentence)] = sentence\n sent_bucket = buckets[buck]\n pad_sent = mx.nd.array([buff], dtype='float32')\n shape = (1, sent_bucket) if layout == 'NT' else (sent_bucket, 1)\n return mx.io.DataBatch([pad_sent], pad=0, bucket_key=sent_bucket,\n provide_data=[mx.io.DataDesc(\n name=data_name,\n shape=shape,\n layout=layout)])\n" }, { "path": "examples/model_service_template/mxnet_vision_batching.py", "content": "# Copyright 2019 Amazon.com, Inc. or its affiliates. All Rights Reserved.\n# Licensed under the Apache License, Version 2.0 (the \"License\").\n# You may not use this file except in compliance with the License.\n# A copy of the License is located at\n# http://www.apache.org/licenses/LICENSE-2.0\n# or in the \"license\" file accompanying this file. This file is distributed\n# on an \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either\n# express or implied. See the License for the specific language governing\n# permissions and limitations under the License.\n\nimport mxnet as mx\nimport json\nimport os\nimport numpy as np\nfrom collections import namedtuple\nimport logging\n\n\nclass MXNetVisionServiceBatching(object):\n def __init__(self):\n \"\"\"\n Initialization for MXNet Vision Service supporting batch inference\n \"\"\"\n self.mxnet_ctx = None\n self.mx_model = None\n self.labels = None\n self.signature = None\n self.epoch = 0\n self._context = None\n self._batch_size = 0\n self.initialized = False\n self.erroneous_reqs = set()\n\n def top_probability(self, data, labels, top=5):\n \"\"\"\n Get top probability prediction from NDArray.\n\n :param data: NDArray\n Data to be predicted\n :param labels: List\n List of class labels\n :param top:\n :return: List\n List of probability: class pairs in sorted order\n \"\"\"\n dim = len(data.shape)\n if dim > 2:\n data = mx.nd.array(\n np.squeeze(data.asnumpy(), axis=tuple(range(dim)[2:])))\n sorted_prob = mx.nd.argsort(data[0], is_ascend=False)\n top_prob = map(lambda x: int(x.asscalar()), sorted_prob[0:top])\n return [{'probability': float(data[0, i].asscalar()), 'class': labels[i]}\n for i in top_prob]\n\n def initialize(self, context):\n \"\"\"\n Initialize model. This will be called during model loading time\n\n :param context: Initial context contains model server system properties.\n :return:\n \"\"\"\n self._context = context\n self._batch_size = context.system_properties[\"batch_size\"]\n self.initialized = True\n\n properties = context.system_properties\n model_dir = properties.get(\"model_dir\")\n gpu_id = properties.get(\"gpu_id\")\n\n signature_file_path = os.path.join(model_dir, \"signature.json\")\n if not os.path.isfile(signature_file_path):\n raise RuntimeError(\"Missing signature.json file.\")\n\n with open(signature_file_path) as f:\n self.signature = json.load(f)\n\n model_files_prefix = context.manifest[\"model\"][\"modelName\"]\n archive_synset = os.path.join(model_dir, \"synset.txt\")\n if os.path.isfile(archive_synset):\n synset = archive_synset\n self.labels = [line.strip() for line in open(synset).readlines()]\n\n data_names = []\n data_shapes = []\n for input_data in self.signature[\"inputs\"]:\n data_name = input_data[\"data_name\"]\n data_shape = input_data[\"data_shape\"]\n\n # Set batch size\n data_shape[0] = self._batch_size\n\n # Replace 0 entry in data shape with 1 for binding executor.\n for idx in range(len(data_shape)):\n if data_shape[idx] == 0:\n data_shape[idx] = 1\n\n data_names.append(data_name)\n data_shapes.append((data_name, tuple(data_shape)))\n\n checkpoint_prefix = \"{}/{}\".format(model_dir, model_files_prefix)\n\n # Load MXNet module\n self.mxnet_ctx = mx.cpu() if gpu_id is None else mx.gpu(gpu_id)\n sym, arg_params, aux_params = mx.model.load_checkpoint(checkpoint_prefix, self.epoch)\n\n self.mx_model = mx.mod.Module(symbol=sym, context=self.mxnet_ctx,\n data_names=data_names, label_names=None)\n self.mx_model.bind(for_training=False, data_shapes=data_shapes)\n self.mx_model.set_params(arg_params, aux_params, allow_missing=True, allow_extra=True)\n\n def inference(self, model_input):\n \"\"\"\n Internal inference methods for MXNet. Run forward computation and\n return output.\n\n :param model_input: list of NDArray\n Preprocessed inputs in NDArray format.\n :return: list of NDArray\n Inference output.\n \"\"\"\n batch = namedtuple('Batch', ['data'])\n\n self.mx_model.forward(batch([model_input]), is_train=False)\n outputs = self.mx_model.get_outputs()\n res = mx.ndarray.split(outputs[0], axis=0, num_outputs=outputs[0].shape[0])\n res = [res] if not isinstance(res, list) else res\n return res\n\n def preprocess(self, request):\n \"\"\"\n Decode all input images into ndarray.\n\n Note: This implementation doesn't properly handle error cases in batch mode,\n If one of the input images is corrupted, all requests in the batch will fail.\n\n :param request:\n :return:\n \"\"\"\n img_list = []\n param_name = self.signature['inputs'][0]['data_name']\n input_shape = self.signature['inputs'][0]['data_shape']\n # We are assuming input shape is NCHW\n [c, h, w] = input_shape[1:]\n\n # Clear error requests set.\n self.erroneous_reqs.clear()\n\n for idx, data in enumerate(request):\n img = data.get(param_name)\n if img is None:\n img = data.get(\"body\")\n\n if img is None:\n img = data.get(\"data\")\n\n if img is None or len(img) == 0:\n logging.error(\"Error processing request\")\n self.erroneous_reqs.add(idx)\n continue\n\n try:\n img_arr = mx.image.imdecode(img, 1, True, None)\n except Exception as e:\n logging.error(e, exc_info=True)\n self.erroneous_reqs.add(idx)\n continue\n\n img_arr = mx.image.imresize(img_arr, w, h, 2)\n img_arr = mx.nd.transpose(img_arr, (2, 0, 1))\n self._num_requests = idx + 1\n img_list.append(img_arr)\n \n logging.debug(\"Worker :{} received {} requests\".format(os.getpid(), self._num_requests))\n reqs = mx.nd.stack(*img_list)\n reqs = reqs.as_in_context(self.mxnet_ctx)\n\n if (self._batch_size - self._num_requests) != 0:\n padding = mx.nd.zeros((self._batch_size - self._num_requests, c, h, w), self.mxnet_ctx, 'uint8')\n reqs = mx.nd.concat(reqs, padding, dim=0)\n\n return reqs\n\n def postprocess(self, data):\n res = []\n for idx, resp in enumerate(data[:self._num_requests]):\n if idx not in self.erroneous_reqs:\n res.append(self.top_probability(resp, self.labels, top=5))\n else:\n res.append(\"This request was not processed successfully. Refer to mms.log for additional information\")\n return res\n\n\n_service = MXNetVisionServiceBatching()\n\n\ndef handle(data, context):\n if not _service.initialized:\n _service.initialize(context)\n\n if data is None:\n return None\n\n try:\n data = _service.preprocess(data)\n data = _service.inference(data)\n data = _service.postprocess(data)\n\n return data\n except Exception as e:\n logging.error(e, exc_info=True)\n request_processor = context.request_processor\n request_processor.report_status(500, \"Unknown inference error\")\n return [str(e)] * _service._batch_size\n" }, { "path": "examples/model_service_template/mxnet_vision_service.py", "content": "# Copyright 2018 Amazon.com, Inc. or its affiliates. All Rights Reserved.\n# Licensed under the Apache License, Version 2.0 (the \"License\").\n# You may not use this file except in compliance with the License.\n# A copy of the License is located at\n# http://www.apache.org/licenses/LICENSE-2.0\n# or in the \"license\" file accompanying this file. This file is distributed\n# on an \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either\n# express or implied. See the License for the specific language governing\n# permissions and limitations under the License.\n\n\"\"\"\nMXNetVisionService defines a MXNet base vision service\n\"\"\"\nimport logging\n\nfrom mxnet_model_service import MXNetModelService\nfrom mxnet_utils import image, ndarray\n\n\nclass MXNetVisionService(MXNetModelService):\n \"\"\"\n MXNetVisionService defines a fundamental service for image classification task.\n In preprocess, input image buffer is read to NDArray and resized respect to input\n shape in signature.\n In post process, top-5 labels are returned.\n \"\"\"\n\n def preprocess(self, request):\n \"\"\"\n Decode all input images into ndarray.\n\n Note: This implementation doesn't properly handle error cases in batch mode,\n If one of the input images is corrupted, all requests in the batch will fail.\n\n :param request:\n :return:\n \"\"\"\n img_list = []\n param_name = self.signature['inputs'][0]['data_name']\n input_shape = self.signature['inputs'][0]['data_shape']\n\n for idx, data in enumerate(request):\n img = data.get(param_name)\n if img is None:\n img = data.get(\"body\")\n\n if img is None:\n img = data.get(\"data\")\n\n if img is None or len(img) == 0:\n self.error = \"Empty image input\"\n return None\n\n # We are assuming input shape is NCHW\n [h, w] = input_shape[2:]\n\n try:\n img_arr = image.read(img)\n except Exception as e:\n logging.warn(e, exc_info=True)\n self.error = \"Corrupted image input\"\n return None\n\n img_arr = image.resize(img_arr, w, h)\n img_arr = image.transform_shape(img_arr)\n img_list.append(img_arr)\n return img_list\n\n def postprocess(self, data):\n if self.error is not None:\n return [self.error] * self._batch_size\n\n assert hasattr(self, 'labels'), \\\n \"Can't find labels attribute. Did you put synset.txt file into \" \\\n \"model archive or manually load class label file in __init__?\"\n return [ndarray.top_probability(d, self.labels, top=5) for d in data]\n\n\n_service = MXNetVisionService()\n\n\ndef handle(data, context):\n if not _service.initialized:\n _service.initialize(context)\n\n if data is None:\n return None\n\n return _service.handle(data, context)\n" }, { "path": "examples/mxnet_vision/README.md", "content": "# MXNet Vision Service\n\nIn this example, we show how to use a pre-trained MXNet model to performing real time Image Classification with MMS\n\nWe choose squeezenet in this example: [Iandola, et al.](https://arxiv.org/pdf/1602.07360v4.pdf). But the same should work for other MXNet Image Classification models.\n\nThe inference service would return the response in the json format.\n\n# Objective\n\n1. Demonstrate how to package a pre-trained squeezenet into model archive (.mar) file\n2. Demonstrate how to create model service code based on provided service template\n3. Demonstrate how to load model archive (.mar) file into MMS and run inference.\n\n## Step 1 - Download the pre-trained squeezenet Model\n\nYou will need the model files in this example. Check this example's directory in case they're already downloaded. Otherwise, you can `curl` the files or download them via your browser:\n\n```bash\ncd multi-model-server/examples/mxnet_vision\ncurl -O https://s3.amazonaws.com/model-server/model_archive_1.0/examples/squeezenet_v1.1/squeezenet_v1.1-symbol.json\ncurl -O https://s3.amazonaws.com/model-server/model_archive_1.0/examples/squeezenet_v1.1/squeezenet_v1.1-0000.params\n```\n\nAlternatively, use these links to download the Symbol and Params files via your browser:\n1. squeezenet_v1.1-symbol.json\n2. squeezenet_v1.1-0000.params\n\n## Step 2 - Prepare the signature file\n\nDefine Input and Output name, type and shape in `signature.json` file. The signature for this example looks like below:\n\n```json\n{\n \"inputs\": [\n {\n \"data_name\": \"data\",\n \"data_shape\": [\n 0,\n 3,\n 224,\n 224\n ]\n }\n ]\n}\n```\n\nIn this pre-trained model, input name is 'data' and shape is '(1,3,224,224)'. Where, the expected input is a color image (3 channels - RGB) of shape 224*224. We also expect input type is a binary JPEG images. In provided mxnet_vision_service.py, you will see the code that take care of converting binary images to tensor NDArray used by MXNet.\n\n*Note:* Typically, if you train your own model, you define the Input and Output Layer name and shape when defining the Neural Network. If you are using a pre-trained MXNet model, to get these Input and Output name and dimensions, you can load the Model and extract the Input and Output layer details. Unfortunately, there are no APIs or easy way to extract the Input shape. Example code below:\n\n```python\n>>> import mxnet as mx\n>>> load_symbol, args, auxs = mx.model.load_checkpoint(\"squeezenet_v1.1\", 0)\n>>> mod = mx.mod.Module(load_symbol, label_names=None, data_names=['data'], context=mx.cpu())\n>>> mod.data_names\n['data']\n>>> mod.bind(data_shapes=[('data', (1, 3, 224, 224))])\n>>> mod.set_params(args, auxs)\n>>> print(mod.data_names)\n>>> print(mod.data_shapes)\n>>> print(mod.output_names)\n>>> print(mod.output_shapes)\n['data']\n[DataDesc[data,(1, 3, 224, 224),,NCHW]]\n['detection_output']\n[('detection_output', (1, 6132, 6))]\n```\n\n## Step 3 - Prepare synset.txt with list of class names\n\n[synset.txt](https://s3.amazonaws.com/model-server/model_archive_1.0/examples/squeezenet_v1.1/synset.txt) is where we define list of all classes detected by the model. The list of classes in synset.txt will be loaded by MMS as list of labels in inference logic.\n\nYou can use `curl` to download it.\n```bash\ncd multi-model-server/examples/mxnet_vision\n\ncurl -O https://s3.amazonaws.com/model-server/model_archive_1.0/examples/squeezenet_v1.1/synset.txt\n```\n\nAlternatively, use following link to download:\nsynset.txt\n\n## Step 4 - Create custom service class\n\nWe provided custom service class template code in [model_service_template](../model_service_template) folder:\n1. [model_handler.py](../model_service_template/model_handler.py) - A generic based service class.\n2. [mxnet_model_service.py](../model_service_template/mxnet_model_service.py) - A MXNet base service class.\n3. [mxnet_vision_service.py](../model_service_template/mxnet_vision_service.py) - A MXNet Vision service class.\n4. [mxnet_utils](../model_service_template/mxnet_utils) - A python package that contains utility classes.\n\nIn this example, you can simple copy them into mxnet_vision folder, as use provided mxnet_vision_service.py as user model archive entry point.\n\n```bash\ncd multi-model-server/examples\ncp -r model_service_template/* mxnet_vision/\n```\n\n## Step 5 - Package the model with `model-archiver` CLI utility\n\nIn this step, we package the following:\n1. pre-trained MXNet Model we downloaded in Step 1.\n2. '[signature.json](https://s3.amazonaws.com/model-server/model_archive_1.0/examples/squeezenet_v1.1/signature.json)' file we prepared in step 2.\n3. '[synset.txt](https://s3.amazonaws.com/model-server/model_archive_1.0/examples/squeezenet_v1.1/synset.txt)' file we prepared in step 3.\n4. custom model service files we prepared in step 4.\n\nWe use `model-archiver` command line utility (CLI) provided by MMS.\nInstall `model-archiver` in case you have not:\n\n```bash\npip install model-archiver\n```\n\nThis tool create a .mar file that will be provided to MMS for serving inference requests. In following command line, we specify 'mxnet_model_service:handle' as model archive entry point.\n\n```bash\ncd multi-model-server/examples\nmodel-archiver --model-name squeezenet_v1.1 --model-path mxnet_vision --handler mxnet_vision_service:handle\n```\n\n## Step 6 - Start the Inference Service\n\nStart the inference service by providing the 'squeezenet_v1.1.mar' file we created in Step 5.\n\nBy default, the server is started on the localhost at port 8080.\n\n```bash\ncd multi-model-server\nmulti-model-server --start --model-store examples --models squeezenet_v1.1.mar\n```\n\nAwesome! we have successfully packaged a pre-trained MXNet model and started a inference service.\n\n`Note:` In this example, MMS loads the .mar file from the local file system. However, you can also store the model archive (.mar file) over a network-accessible storage such as AWS S3, and use a URL such as http:// or https:// to indicate the model location. MMS is capable of loading the model archive over such URLs as well.\n\n## Step 7 - Test sample inference\n\nLet us try the inference server we just started. Use curl to make a prediction call by passing a JPEG image as input to the prediction request.\n\n```bash\ncd multi-model-server\ncurl -X POST http://127.0.0.1:8080/predictions/squeezenet_v1.1 -T docs/images/kitten_small.jpg\n```\n\nYou can expect the response similar to below. The output format is in json.\n\n```json\n[\n {\n \"class\": \"n02127052 lynx, catamount\", \n \"probability\": 0.5721369385719299\n }, \n {\n \"class\": \"n02124075 Egyptian cat\", \n \"probability\": 0.4079437255859375\n }, \n {\n \"class\": \"n02123045 tabby, tabby cat\", \n \"probability\": 0.013694713823497295\n }, \n {\n \"class\": \"n02123394 Persian cat\", \n \"probability\": 0.004954110365360975\n }, \n {\n \"class\": \"n02123159 tiger cat\", \n \"probability\": 0.0012674571480602026\n }\n]\n```\n\nA consumer application can use this response to identify the objects in the input image and their bounding boxes.\n\n## Step 8 - Clean up and stop MMS\n\nMMS will keep running in background. And .mar file will be extracted to system temp directory.\nYou can clean up temp directory by unregister model and use CLI to stop MMS\n\n```bash\ncurl -X DELETE http://127.0.0.1:8081/models/squeezenet_v1.1\n\nmulti-model-server --stop\n```\n" }, { "path": "examples/sockeye_translate/Dockerfile", "content": "FROM nvidia/cuda:9.2-cudnn7-runtime-ubuntu18.04\n\nENV PYTHONUNBUFFERED TRUE\n\nRUN useradd -m model-server && \\\n mkdir -p /home/model-server/tmp\n\nWORKDIR /home/model-server\nENV TEMP=/home/model-server/tmp\n\nRUN apt-get update && \\\n DEBIAN_FRONTEND=noninteractive apt-get install -y \\\n build-essential \\\n python3-dev \\\n python3-venv \\\n openjdk-8-jdk-headless \\\n curl \\\n vim && \\\n rm -rf /var/lib/apt/lists/*\n\nCOPY requirements/ requirements/\nRUN python3 -m venv venv && \\\n . venv/bin/activate && \\\n pip install --upgrade pip setuptools wheel && \\\n pip install sockeye --no-deps -r requirements/sockeye/requirements.gpu-cu92.txt && \\\n pip install --no-cache-dir multi-model-server && \\\n pip install -r requirements/sockeye-serving/requirements.txt\n\nCOPY config/config.properties /home/model-server\nCOPY scripts/mms/dockerd-entrypoint.sh /usr/local/bin/dockerd-entrypoint.sh\n\nRUN chmod +x /usr/local/bin/dockerd-entrypoint.sh && \\\n chown -R model-server /home/model-server\n\nEXPOSE 8080 8081\n\nUSER model-server\nENTRYPOINT [\"/usr/local/bin/dockerd-entrypoint.sh\"]\nCMD [\"serve\"]\n\nLABEL maintainer=\"james.e.woo@gmail.com\"\n" }, { "path": "examples/sockeye_translate/README.md", "content": "# sockeye-serving\nThis example shows how to serve Sockeye models for machine translation.\nThe custom handler is implemented in `sockeye_service.py`.\nSince Sockeye has many dependencies, it's convenient to use Docker.\nFor simplicity, we'll use a pre-trained model and make some assumptions about how we preprocess the data.\n\n## Getting Started With Docker\nPull the latest Docker image:\n```bash\ndocker pull jwoo11/sockeye-serving\n```\n\nDownload the example model archive file (MAR).\nThis is a ZIP archive containing the parameter files and scripts needed to run translation for a particular language:\n* https://www.dropbox.com/s/pk7hmp7a5zjcfcj/zh.mar?dl=0\n\nExtract the MAR file to `/tmp/models`.\n We'll use this directory as a bind mount for Docker:\n```bash\nunzip -d /tmp/models/zh zh.mar\n```\n\nStart the server:\n```bash\ndocker run -itd --name mms -p 8080:8080 -p 8081:8081 -v /tmp/models:/opt/ml/model jwoo11/sockeye-serving serve\n```\n\nNow we can load the model using the management API provided by `multi-model-server`:\n```bash\ncurl -X POST \"http://localhost:8081/models?synchronous=true&initial_workers=1&url=zh\"\n```\nGet the status of the model with the following:\n```bash\ncurl -X GET \"http://localhost:8081/models/zh\"\n```\n```json\n{\n \"modelName\": \"zh\",\n \"modelUrl\": \"zh\",\n \"runtime\": \"python3\",\n \"minWorkers\": 1,\n \"maxWorkers\": 1,\n \"batchSize\": 1,\n \"maxBatchDelay\": 100,\n \"workers\": [\n {\n \"id\": \"9000\",\n \"startTime\": \"2019-01-26T00:49:10.431Z\",\n \"status\": \"READY\",\n \"gpu\": false,\n \"memoryUsage\": 601395200\n }\n ]\n}\n```\n\nTo translate text, use the inference API. Notice that the port is different from above. \n```bash\ncurl -X POST \"http://localhost:8080/predictions/zh\" -H \"Content-Type: application/json\" \\\n -d '{ \"text\": \"我的世界是一款開放世界遊戲，玩家沒有具體要完成的目標，即玩家有超高的自由度選擇如何玩遊戲\" }'\n```\n\nThe translation quality depends on the model. Apparently, this one needs more training:\n```json\n{\n \"translation\": \"in my life was a life of a life of a public public, and a public, a time, a video, a play, which, it was a time of a time of a time.\"\n}\n```\n\nFor more information on MAR files and the built-in REST APIs, see:\n* https://github.com/awslabs/multi-model-server/tree/master/docs\n" }, { "path": "examples/sockeye_translate/config/config.properties", "content": "vmargs=-Xmx128m -XX:-UseLargePages -XX:+UseG1GC -XX:MaxMetaspaceSize=32M -XX:MaxDirectMemorySize=10m -XX:+ExitOnOutOfMemoryError\nmodel_store=/opt/ml/model\n# load_models=ALL\ninference_address=http://0.0.0.0:8080\nmanagement_address=http://0.0.0.0:8081\n# management_address=unix:/tmp/management.sock\n# number_of_netty_threads=0\n# netty_client_threads=0\n# default_workers_per_model=0\n# job_queue_size=100\n# async_logging=false\n# number_of_gpu=1\n# cors_allowed_origin\n# cors_allowed_methods\n# cors_allowed_headers\n# keystore=src/test/resources/keystore.p12\n# keystore_pass=changeit\n# keystore_type=PKCS12\n# private_key_file=src/test/resources/key.pem\n# certificate_file=src/test/resources/certs.pem\n# blacklist_env_vars=\n" }, { "path": "examples/sockeye_translate/model_handler.py", "content": "# Copyright 2018 Amazon.com, Inc. or its affiliates. All Rights Reserved.\n# Licensed under the Apache License, Version 2.0 (the \"License\").\n# You may not use this file except in compliance with the License.\n# A copy of the License is located at\n# http://www.apache.org/licenses/LICENSE-2.0\n# or in the \"license\" file accompanying this file. This file is distributed\n# on an \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either\n# express or implied. See the License for the specific language governing\n# permissions and limitations under the License.\n\n\"\"\"\nModelHandler defines a base model handler.\n\"\"\"\nimport logging\n\n\nclass ModelHandler(object):\n \"\"\"\n A base Model handler implementation.\n \"\"\"\n\n def __init__(self):\n self.error = None\n self._context = None\n self._batch_size = 0\n self.initialized = False\n\n def initialize(self, context):\n \"\"\"\n Initialize model. This will be called during model loading time\n\n :param context: Initial context contains model server system properties.\n :return:\n \"\"\"\n self._context = context\n self._batch_size = context.system_properties[\"batch_size\"]\n self.initialized = True\n\n def preprocess(self, batch):\n \"\"\"\n Transform raw input into model input data.\n\n :param batch: list of raw requests, should match batch size\n :return: list of preprocessed model input data\n \"\"\"\n assert self._batch_size == len(batch), \"Invalid input batch size: {}\".format(len(batch))\n return None\n\n def inference(self, model_input):\n \"\"\"\n Internal inference methods\n\n :param model_input: transformed model input data\n :return: list of inference output in NDArray\n \"\"\"\n return None\n\n def postprocess(self, inference_output):\n \"\"\"\n Return predict result in batch.\n\n :param inference_output: list of inference output\n :return: list of predict results\n \"\"\"\n return [\"OK\"] * self._batch_size\n\n def handle(self, data, context):\n \"\"\"\n Custom service entry point function.\n\n :param data: list of objects, raw input from request\n :param context: model server context\n :return: list of outputs to be send back to client\n \"\"\"\n\n try:\n data = self.preprocess(data)\n data = self.inference(data)\n data = self.postprocess(data)\n\n return data\n except Exception as e:\n logging.error(e, exc_info=True)\n request_processor = context.request_processor\n request_processor.report_status(500, \"Unknown inference error\")\n return [str(e)] * self._batch_size\n" }, { "path": "examples/sockeye_translate/preprocessor.py", "content": "import html\nimport logging\nimport os\nimport subprocess\nfrom html.entities import html5, name2codepoint\n\nimport regex as re\nfrom subword_nmt.apply_bpe import BPE\n\n\nclass Preprocessor(object):\n def __init__(self, bpe_code_file):\n super(Preprocessor, self).__init__()\n\n symbols = ''\n symbol_set = set({})\n\n for k in name2codepoint.keys():\n symbol_set.add(k)\n\n for k in html5.keys():\n symbol_set.add(k.strip(';'))\n\n for s in symbol_set:\n symbols += '|' + s\n\n symbols = symbols.strip('|')\n\n self.single = re.compile('&[ ]?(' + symbols + ')[ ]?;', re.IGNORECASE)\n self.double = re.compile('&[ ]?amp[ ]?;[ ]?(' + symbols + ')[ ]?;', re.IGNORECASE)\n\n self.singleNum = re.compile('&[ ]?#[ ]?([0-9]+)[ ]?;', re.IGNORECASE)\n self.doubleNum = re.compile('&[ ]?amp[ ]?;[ ]?#[ ]?([0-9]+)[ ]?;', re.IGNORECASE)\n\n self.singleXNum = re.compile('&[ ]?#[ ]?x[ ]?([a-f0-9]+)[ ]?;', re.IGNORECASE)\n self.doubleXNum = re.compile('&[ ]?amp[ ]?;[ ]?#[ ]?x[ ]?([a-f0-9]+)[ ]?;', re.IGNORECASE)\n\n self.nbsp = re.compile('(&[ ]?x?[ ]?n[]?b[ ]?([a-z][ ]?){0,6}[ ]?;)|(&[ ]?o[ ]?s[ ]?p[ ]?;)', re.IGNORECASE)\n\n self.shy = re.compile('[ ]?&[ ]?s[ ]?h[ ]?y[ ]?;[ ]?', re.IGNORECASE)\n\n self.bpe = None\n if bpe_code_file:\n with open(bpe_code_file, mode='r', encoding='utf-8') as f:\n self.bpe = BPE(f)\n else:\n logging.error('No BPE code file specified')\n\n def unescape(self, line):\n # put html-escaped (or double escaped) codes back into canonical format\n line = re.sub(self.double, r'&\\1;', line)\n line = re.sub(self.doubleNum, r'&#\\1;', line)\n line = re.sub(self.doubleXNum, r'&#x\\1;', line)\n line = re.sub(self.single, r'&\\1;', line)\n line = re.sub(self.singleNum, r'&#\\1;', line)\n line = re.sub(self.singleXNum, r'&#x\\1;', line)\n\n # get rid of this tag\n # alphabetic characters -- need only get rid of space around their canonical escaped forms\n line = re.sub(self.shy, '', line)\n\n # unescape\n line = html.unescape(line)\n\n # clean up weird errors in the escaping of the non-breaking space\n line = re.sub(self.nbsp, ' ', line)\n return line\n\n def bpe_encode(self, text):\n return self.bpe.process_line(text).strip()\n\n\nclass JoshuaPreprocessor(Preprocessor):\n def __init__(self, bpe_code_file, joshua_path, moses_path, lang):\n super(JoshuaPreprocessor, self).__init__(bpe_code_file)\n\n self.lang = lang\n self.normalizer = os.path.join(joshua_path, 'normalize.pl')\n self.tokenizer = os.path.join(moses_path, 'tokenizer.perl')\n self.cleaner = os.path.join(moses_path, 'remove-non-printing-char.perl')\n\n for f in [self.normalizer, self.tokenizer, self.cleaner]:\n os.chmod(f, 0o755)\n\n def run(self, text):\n text = self.unescape(text)\n\n # normalize, remove non-printing characters, and tokenize\n popen = subprocess.run(\n [self.normalizer, self.lang, '|', self.cleaner, '|', self.tokenizer, '-l', self.lang, '-no-escape', '-q'],\n input=text, encoding='utf-8', stdout=subprocess.PIPE)\n result = popen.stdout.strip()\n\n return self.bpe_encode(result)\n\n\nclass ChineseCharPreprocessor(JoshuaPreprocessor):\n def __init__(self, bpe_code_file, joshua_path, moses_path):\n super(ChineseCharPreprocessor, self).__init__(bpe_code_file, joshua_path, moses_path, 'zh')\n\n self.pattern = re.compile(\n '([\\p{IsHan}\\p{InCJK_Symbols_and_Punctuation}\\p{InCJK_Radicals_Supplement}\\p{InCJK_Compatibility}])',\n re.UNICODE)\n\n def run(self, text):\n text = self.unescape(text)\n\n # normalize and remove non-printing characters\n popen = subprocess.run([self.normalizer, self.lang, '|', self.cleaner], input=text, encoding='utf-8',\n stdout=subprocess.PIPE)\n text = popen.stdout.strip()\n\n # tokenize by separating all ZH characters with a space\n text = self.pattern.sub(r' \\1 ', text).strip()\n\n # tokenize other characters using Moses\n popen = subprocess.run([self.tokenizer, '-l', self.lang, '-no-escape', '-q'], input=text, encoding='utf-8',\n stdout=subprocess.PIPE)\n result = popen.stdout.strip()\n\n return self.bpe_encode(result)\n\n\nclass Detokenizer():\n def __init__(self, path):\n self.de_bpe = re.compile('@@( |$)', re.IGNORECASE)\n self.de_tok = path\n\n os.chmod(self.de_tok, 0o755)\n\n def run(self, text):\n bpe_removed = re.sub(self.de_bpe, '', text.translation.strip())\n popen = subprocess.run([self.de_tok, '-l', 'en'], input=bpe_removed, encoding='utf-8', stdout=subprocess.PIPE,\n env=os.environ)\n return popen.stdout.strip()\n" }, { "path": "examples/sockeye_translate/sockeye_service.py", "content": "import logging\nimport os\nimport re\nfrom contextlib import ExitStack\nfrom sockeye import arguments\nfrom sockeye import constants as const\nfrom sockeye import inference\nfrom sockeye.lexicon import TopKLexicon\nfrom sockeye.output_handler import get_output_handler\nfrom sockeye.utils import check_condition, log_basic_info, determine_context\n\nfrom .model_handler import ModelHandler\nfrom .preprocessor import ChineseCharPreprocessor, Detokenizer\n\n\ndef decode_bytes(data):\n \"\"\"\n Decodes a bytes array from a file upload\n\n :param data: a UTF-8 encoded byte array\n :return: a cleaned string\n \"\"\"\n pattern = re.compile('\\r', re.UNICODE)\n res = data.decode('utf-8', 'ignore')\n res = pattern.sub('', res).strip()\n return res\n\n\ndef get_text(req):\n \"\"\"\n Returns the text string, if any, in the request\n\n :param req: a JSON request\n :return: a text string\n \"\"\"\n for field in ['body']:\n if field in req:\n data = req[field]\n if isinstance(data, str):\n return data\n elif isinstance(data, dict) and 'text' in data:\n return data['text']\n return None\n\n\ndef get_file_data(req):\n \"\"\"\n Returns the file data, if any, in the request\n\n :param req: a JSON request\n :return: a byte array\n \"\"\"\n for field in ['body', 'file', 'data']:\n if field in req:\n data = req[field]\n if isinstance(data, bytearray):\n return data\n return None\n\n\ndef read_sockeye_args(params_path):\n \"\"\"\n Reads command line arguments stored in a file\n\n :param params_path: path to the parameters file\n :return: a list of command line arguments\n \"\"\"\n with open(params_path) as f:\n content = f.readlines()\n\n res = []\n for line in content:\n res += line.split()\n return res\n\n\nclass SockeyeService(ModelHandler):\n \"\"\"\n Consumes text of arbitrary length and returns its translation.\n \"\"\"\n\n def __init__(self):\n super(SockeyeService, self).__init__()\n self.basedir = None\n self.device_ids = []\n self.postprocessor = None\n self.preprocessor = None\n self.sentence_id = 0\n self.translator = None\n\n def initialize(self, context):\n super(SockeyeService, self).initialize(context)\n\n self.basedir = context.system_properties.get('model_dir')\n self.preprocessor = ChineseCharPreprocessor(os.path.join(self.basedir, 'bpe.codes.zh-en'),\n os.path.join(self.basedir, 'scripts'),\n os.path.join(self.basedir, 'scripts'))\n self.postprocessor = Detokenizer(os.path.join(self.basedir, 'scripts', 'detokenize.pl'))\n\n params = arguments.ConfigArgumentParser(description='Translate CLI')\n arguments.add_translate_cli_args(params)\n\n sockeye_args_path = os.path.join(self.basedir, 'sockeye-args.txt')\n sockeye_args = params.parse_args(read_sockeye_args(sockeye_args_path))\n # override models directory\n sockeye_args.models = [self.basedir]\n\n if 'gpu_id' in context.system_properties:\n self.device_ids.append(context.system_properties['gpu_id'])\n else:\n logging.warning('No gpu_id found in context')\n self.device_ids.append(0)\n\n if sockeye_args.checkpoints is not None:\n check_condition(len(sockeye_args.checkpoints) == len(sockeye_args.models),\n 'must provide checkpoints for each model')\n\n if sockeye_args.skip_topk:\n check_condition(sockeye_args.beam_size == 1, '--skip-topk has no effect if beam size is larger than 1')\n check_condition(len(sockeye_args.models) == 1,\n '--skip-topk has no effect for decoding with more than 1 model')\n\n if sockeye_args.nbest_size > 1:\n check_condition(sockeye_args.beam_size >= sockeye_args.nbest_size,\n 'Size of nbest list (--nbest-size) must be smaller or equal to beam size (--beam-size).')\n check_condition(sockeye_args.beam_search_drop == const.BEAM_SEARCH_STOP_ALL,\n '--nbest-size > 1 requires beam search to only stop after all hypotheses are finished '\n '(--beam-search-stop all)')\n if sockeye_args.output_type != const.OUTPUT_HANDLER_NBEST:\n logging.warning('For nbest translation, output handler must be \"%s\", overriding option --output-type.',\n const.OUTPUT_HANDLER_NBEST)\n sockeye_args.output_type = const.OUTPUT_HANDLER_NBEST\n\n log_basic_info(sockeye_args)\n\n output_handler = get_output_handler(sockeye_args.output_type,\n sockeye_args.output,\n sockeye_args.sure_align_threshold)\n\n with ExitStack() as exit_stack:\n check_condition(len(self.device_ids) == 1, 'translate only supports single device for now')\n translator_ctx = determine_context(device_ids=self.device_ids,\n use_cpu=sockeye_args.use_cpu,\n disable_device_locking=sockeye_args.disable_device_locking,\n lock_dir=sockeye_args.lock_dir,\n exit_stack=exit_stack)[0]\n logging.info('Translate Device: %s', translator_ctx)\n\n if sockeye_args.override_dtype == const.DTYPE_FP16:\n logging.warning('Experimental feature \\'--override-dtype float16\\' has been used. '\n 'This feature may be removed or change its behavior in the future. '\n 'DO NOT USE IT IN PRODUCTION')\n\n models, source_vocabs, target_vocab = inference.load_models(\n context=translator_ctx,\n max_input_len=sockeye_args.max_input_len,\n beam_size=sockeye_args.beam_size,\n batch_size=sockeye_args.batch_size,\n model_folders=sockeye_args.models,\n checkpoints=sockeye_args.checkpoints,\n softmax_temperature=sockeye_args.softmax_temperature,\n max_output_length_num_stds=sockeye_args.max_output_length_num_stds,\n decoder_return_logit_inputs=sockeye_args.restrict_lexicon is not None,\n cache_output_layer_w_b=sockeye_args.restrict_lexicon is not None,\n override_dtype=sockeye_args.override_dtype,\n output_scores=output_handler.reports_score())\n restrict_lexicon = None\n if sockeye_args.restrict_lexicon:\n restrict_lexicon = TopKLexicon(source_vocabs[0], target_vocab)\n restrict_lexicon.load(sockeye_args.restrict_lexicon, k=sockeye_args.restrict_lexicon_topk)\n store_beam = sockeye_args.output_type == const.OUTPUT_HANDLER_BEAM_STORE\n self.translator = inference.Translator(context=translator_ctx,\n ensemble_mode=sockeye_args.ensemble_mode,\n bucket_source_width=sockeye_args.bucket_width,\n length_penalty=inference.LengthPenalty(\n sockeye_args.length_penalty_alpha,\n sockeye_args.length_penalty_beta),\n beam_prune=sockeye_args.beam_prune,\n beam_search_stop=sockeye_args.beam_search_stop,\n nbest_size=sockeye_args.nbest_size,\n models=models,\n source_vocabs=source_vocabs,\n target_vocab=target_vocab,\n restrict_lexicon=restrict_lexicon,\n avoid_list=sockeye_args.avoid_list,\n store_beam=store_beam,\n strip_unknown_words=sockeye_args.strip_unknown_words,\n skip_topk=sockeye_args.skip_topk)\n\n def preprocess(self, batch):\n \"\"\"\n Preprocesses a JSON request for translation.\n\n :param batch: a list of JSON requests of the form { 'text': input_string } or { 'file': file_data }\n :return: a list of input strings to translate\n \"\"\"\n logging.info('preprocess grabbed: %s' % batch)\n\n texts = []\n for req in batch:\n data = get_file_data(req)\n\n if data:\n text = decode_bytes(data)\n else:\n text = get_text(req)\n\n if text:\n bpe = self.preprocessor.run(text)\n texts.append(bpe)\n\n return texts\n\n def inference(self, texts):\n \"\"\"\n Translates the input data.\n\n :param texts: a list of strings to translate\n :return: a list of translation objects from Sockeye\n \"\"\"\n logging.info('inference grabbed: %s' % texts)\n\n if texts:\n trans_inputs = []\n for t in texts:\n _input = inference.make_input_from_plain_string(self.sentence_id, t)\n trans_inputs.append(_input)\n outputs = self.translator.translate(trans_inputs)\n\n if len(outputs) != len(trans_inputs):\n logging.warning(\"Number of translation outputs doesn't match the number of inputs\")\n\n self.sentence_id += len(trans_inputs)\n return outputs\n else:\n self.error = 'Input to inference is empty'\n return []\n\n def postprocess(self, outputs):\n \"\"\"\n Converts the translations into a list of JSON responses.\n\n :param outputs: a list of translation objects from Sockeye\n :return: a list of translations of the form: { 'translation': output_string }\n \"\"\"\n logging.info('postprocess grabbed: %s' % outputs)\n\n res = []\n for t in outputs:\n output = self.postprocessor.run(t)\n res.append({'translation': output})\n return res\n\n\n_service = SockeyeService()\n\n\ndef handle(data, context):\n if not _service.initialized:\n _service.initialize(context)\n\n if data is None:\n return None\n\n return _service.handle(data, context)\n" }, { "path": "examples/ssd/README.md", "content": "# Single Shot Multi Object Detection Inference Service\n\nIn this example, we show how to use a pre-trained Single Shot Multi Object Detection (SSD) Multi model for performing real time inference using MMS\n\nThe pre-trained model is trained on the [Pascal VOC 2012 dataset](http://host.robots.ox.ac.uk/pascal/VOC/voc2012/index.html) The network is a SSD model built on Resnet50 as base network to extract image features. The model is trained to detect the following entities (classes): ['aeroplane', 'bicycle', 'bird', 'boat', 'bottle', 'bus', 'car', 'cat', 'chair', 'cow', 'diningtable', 'dog', 'horse', 'motorbike', 'person', 'pottedplant', 'sheep', 'sofa', 'train', 'tvmonitor']. For more details about the model, you can refer [here](https://github.com/apache/incubator-mxnet/tree/master/example/ssd).\n\nThe inference service would return the response in the format - '[(object_class, xmin, ymin, xmax, ymax)]. Where, xmin, ymin, xmax and ymax are the bounding box coordinates of the detected object.\n\n# Objective\n\n1. Demonstrate how to package a a pre-trained MXNet model in MMS\n2. Demonstrate how to create custom service with pre-processing and post-processing\n\n## Step 1 - Download the pre-trained SSD Model\n\nYou will need the model files to use for the export. Check this example's directory in case they're already downloaded. Otherwise, you can `curl` the files or download them via your browser:\n\n```bash\ncd multi-model-server/examples/ssd\n\ncurl -O https://s3.amazonaws.com/model-server/model_archive_1.0/examples/ssd/resnet50_ssd_model-symbol.json\ncurl -O https://s3.amazonaws.com/model-server/model_archive_1.0/examples/ssd/resnet50_ssd_model-0000.params\n```\n\nAlternatively, use these links to download the Symbol and Params files via your browser:\n1. resnet50_ssd_model-symbol.json\n2. resnet50_ssd_model-0000.params\n\n**Note** params file is around 125 MB.\n\n## Step 2 - Prepare the signature file\n\nDefine model input name and shape in `signature.json` file. The signature for this example looks like below:\n\n```json\n{\n \"inputs\": [\n {\n \"data_name\": \"data\",\n \"data_shape\": [\n 1,\n 3,\n 512,\n 512\n ]\n }\n ]\n}\n```\n\nIn the pre-trained model, input name is 'data' and shape is '(1,3,512,512)'. Where, the expected input is a color image (3 channels - RGB) of shape 512*512. We also expect input type is a binary JPEG images. In provided mxnet_vision_service.py, you will see the code that take care of converting binary images to tensor NDArray used by MXNet.\n\n*Note:* Typically, if you train your own model, you define the Input and Output Layer name and shape when defining the Neural Network. If you are using a pre-trained MXNet model, to get these Input and Output name and dimensions, you can load the Model and extract the Input and Output layer details. Unfortunately, there are no APIs or easy way to extract the Input shape. Example code below:\n\n```python\n>>> import mxnet as mx\n>>> load_symbol, args, auxs = mx.model.load_checkpoint(\"resnet50_ssd_model\", 000)\n>>> mod = mx.mod.Module(load_symbol, label_names=None, context=mx.cpu())\n>>> mod.data_names\n['data']\n>>> mod.bind(data_shapes=[('data', (1, 3, 512, 512))])\n>>> mod.set_params(args, auxs)\n>>> print(mod.data_names)\n>>> print(mod.data_shapes)\n>>> print(mod.output_names)\n>>> print(mod.output_shapes)\n['data']\n[DataDesc[data,(1, 3, 512, 512),,NCHW]]\n['detection_output']\n[('detection_output', (1, 6132, 6))]\n```\n\n*Note:* The network generates 6132 detections because we use MXNet's [MultiboxPrior](https://mxnet.incubator.apache.org/api/python/symbol.html#mxnet.contrib.symbol.MultiBoxPrior) to generate the anchor boxes with the following 'Ratios and 'Sizes':\n\n```python\n sizes = [[.1, .141], [.2,.272], [.37, .447], [.54, .619], [.71, .79], [.88, .961]]\n ratios = [[1,2,.5], [1,2,.5,3,1./3], [1,2,.5,3,1./3], [1,2,.5,3,1./3], \\\n [1,2,.5], [1,2,.5]]\n```\n\nTo understand more about the MultiboxPrior, anchor boxes, sizes and ratios, please read [this tutorial](http://gluon.mxnet.io/chapter08_computer-vision/object-detection.html)\n\n## Step 3 - Prepare synset.txt with list of class names\n\n`synset.txt` is where we define list of all classes detected by the model. The pre-trained SSD model used in the example is trained to detect 20 classes - person, car, aeroplane, bicycle and more. See synset.txt file for list of all classes.\n\nThe list of classes in synset.txt will be loaded by MMS as list of labels in inference logic.\n\nYou can use `curl` to download it.\n```bash\ncd multi-model-server/examples/ssd\n\ncurl -O https://s3.amazonaws.com/model-server/model_archive_1.0/examples/ssd/synset.txt\n```\n\nAlternatively, use following link to download:\nsynset.txt\n\n\n## Step 4 - Create custom service class\n\nWe provided custom service class template code in [template](../template) folder:\n1. [model_handler.py](../model_service_template/model_handler.py) - A generic based service class.\n2. [mxnet_model_service.py](../model_service_template/mxnet_model_service.py) - A MXNet base service class.\n3. [mxnet_vision_service.py](../model_service_template/mxnet_vision_service.py) - A MXNet Vision service class.\n4. [mxnet_utils](../model_service_template/mxnet_utils) - A python package that contains utility classes.\n\nIn this example, you can simple copy them into ssd folder, as use provided mxnet_vision_service.py as user model archive entry point.\n\n```bash\ncd multi-model-server/examples\ncp -r model_service_template/* ssd/\n```\n\nIn this example, we extend `MXNetVisionService`, provided by MMS for vision inference use-cases, and reuse its input image preprocess functionality to resize and transform the image shape. We only add custom pre-processing and post-processing steps. See [ssd_service.py](ssd_service.py) for more details on how to extend the base service and add custom pre-processing and post-processing.\n\n## Step 5 - Package the model with `model-archiver` CLI utility\n\nIn this step, we package the following:\n1. pre-trained MXNet Model we downloaded in Step 1.\n2. '[signature.json](https://s3.amazonaws.com/model-server/model_archive_1.0/examples/ssd/signature.json)' file we prepared in step 2.\n3. '[synset.txt](https://s3.amazonaws.com/model-server/model_archive_1.0/examples/ssd/synset.txt)' file we prepared in step 3.\n4. custom model service files we prepared in step 4.\n\nWe use `model-archiver` command line utility (CLI) provided by MMS.\nInstall `model-archiver` in case you have not:\n\n```bash\npip install model-archiver\n```\n\nThis tool create a .mar file that will be provided to MMS for serving inference requests. In following command line, we specify 'ssd_service:handle' as model archive entry point.\n\n```bash\ncd multi-model-server/examples\nmodel-archiver --model-name resnet50_ssd_model --model-path ssd --handler ssd_service:handle\n```\n\n## Step 6 - Start the Inference Service\n\nStart the inference service by providing the 'resnet50_ssd_model.mar' file we created in Step 5.\n\nMMS then extracts the resources (signature, synset, model symbol and params) we have packaged into .mar file and uses the extended custom service, to start the inference server.\n\nBy default, the server is started on the localhost at port 8080.\n\n```bash\ncd multi-model-server\nmulti-model-server --start --model-store examples --models ssd=resnet50_ssd_model.mar\n```\n\nAwesome! we have successfully exported a pre-trained MXNet model, extended MMS with custom preprocess/postprocess and started a inference service.\n\n**Note**: In this example, MMS loads the .mar file from the local file system. However, you can also store the archive (.mar file) over a network-accessible storage such as AWS S3, and use a URL such as http:// or https:// to indicate the model archive location. MMS is capable of loading the model archive over such URLs as well.\n\n## Step 7 - Test sample inference\n\nLet us try the inference server we just started. Open another terminal on the same host. Download a sample image, or try any jpeg image that contains the one or more of the object classes mentioned earlier: 'aeroplane', 'bicycle', 'bird', 'boat', etc...\n\nYou can also use this image of three dogs on a beach.\n![3 dogs on beach](../../docs/images/3dogs.jpg)\n\nUse curl to make a prediction call by passing the downloaded image as input to the prediction request.\n\n```bash\ncd multi-model-server\ncurl -X POST http://127.0.0.1:8080/predictions/ssd -T docs/images/3dogs.jpg\n```\n\nYou can expect the response similar to below. The output format is `[(object_class, xmin, ymin, xmax, ymax)]`.\nWhere, xmin, ymin, xmax and ymax are the bounding box coordinates of the detected object.\n\n```json\n[\n [\n \"dog\", \n 399, \n 128, \n 570, \n 290\n ], \n [\n \"dog\", \n 278, \n 196, \n 417, \n 286\n ], \n [\n \"cow\", \n 205, \n 116, \n 297, \n 272\n ]\n]\n```\n\nA consumer application can use this response to identify the objects in the input image and their bounding boxes.\n\nFor better visualization on the input and how we can use the inference output, see below:\n\nInput Image\n\n![Street Input Image](../../docs/images/dogs-before.jpg)\n\nOutput Image\n\n![Street Output Image](../../docs/images/dogs-after.jpg)\n\n\nSee [More example outputs](example_outputs.md)\n\n# References\n1. Adapted code and pre-trained model from - https://github.com/apache/incubator-mxnet/tree/master/example/ssd\n2. Learn more about SSD in this tutorial - http://gluon.mxnet.io/chapter08_computer-vision/object-detection.html\n" }, { "path": "examples/ssd/example_outputs.md", "content": "# SSD Example Outputs\n\n### Dog Beach\n\n![dog beach](https://farm9.staticflickr.com/8184/8081332083_3a5c242b8b_z_d.jpg)\n```bash\ncurl -o dogbeach.jpg https://farm9.staticflickr.com/8184/8081332083_3a5c242b8b_z_d.jpg\ncurl -X POST http://127.0.0.1:8080/ssd/predict -F \"data=@dogbeach.jpg\"\n{\n \"prediction\": [\n [\n \"person\",\n 203,\n 213,\n 248,\n 347\n ],\n [\n \"dog\",\n 334,\n 175,\n 403,\n 235\n ],\n [\n \"person\",\n 109,\n 211,\n 144,\n 291\n ],\n [\n \"person\",\n 529,\n 31,\n 562,\n 103\n ],\n [\n \"person\",\n 155,\n 12,\n 189,\n 98\n ],\n [\n \"horse\",\n 465,\n 3,\n 527,\n 40\n ],\n [\n \"person\",\n 51,\n 372,\n 96,\n 427\n ],\n [\n \"dog\",\n 80,\n 56,\n 131,\n 96\n ],\n [\n \"person\",\n 70,\n 89,\n 96,\n 155\n ],\n [\n \"cow\",\n 292,\n 188,\n 344,\n 231\n ],\n [\n \"dog\",\n 294,\n 186,\n 349,\n 231\n ]\n ]\n}\n```\n\n### 3 Dogs on Beach\n![3 dogs on beach](https://farm9.staticflickr.com/8051/8081326814_64756479c6_z_d.jpg)\n```bash\ncurl -o 3dogs.jpg https://farm9.staticflickr.com/8051/8081326814_64756479c6_z_d.jpg\ncurl -X POST http://127.0.0.1:8080/ssd/predict -F \"data=@3dogs.jpg\"\n{\n \"prediction\": [\n [\n \"dog\",\n 399,\n 128,\n 570,\n 290\n ],\n [\n \"dog\",\n 278,\n 196,\n 417,\n 286\n ],\n [\n \"cow\",\n 205,\n 116,\n 297,\n 272\n ]\n ]\n}\n```\n### Sailboat\n![sailboat](https://farm9.staticflickr.com/8316/7990362092_84a688a089_z_d.jpg)\n```bash\ncurl -o sailboat.jpg https://farm9.staticflickr.com/8316/7990362092_84a688a089_z_d.jpg\ncurl -X POST http://127.0.0.1:8080/ssd/predict -F \"data=@sailboat.jpg\"\n{\n \"prediction\": [\n [\n \"boat\",\n 160,\n 87,\n 249,\n 318\n ]\n ]\n}\n```\n" }, { "path": "examples/ssd/ssd_service.py", "content": "# Copyright 2017 Amazon.com, Inc. or its affiliates. All Rights Reserved.\n# Licensed under the Apache License, Version 2.0 (the \"License\").\n# You may not use this file except in compliance with the License.\n# A copy of the License is located at\n# http://www.apache.org/licenses/LICENSE-2.0\n# or in the \"license\" file accompanying this file. This file is distributed\n# on an \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either\n# express or implied. See the License for the specific language governing\n# permissions and limitations under the License.\n\nimport numpy as np\n\nfrom mxnet_utils import image\nfrom mxnet_vision_service import MXNetVisionService\n\n\nclass SSDService(MXNetVisionService):\n \"\"\"\n SSD Service to perform real time multi-object detection using pre-trained MXNet SSD model.\n This class extends MXNetVisionService to add custom preprocessing of input\n and preparing the output.\n Reuses input image transformation functionality of MXNetVisionService.\n \"\"\"\n def __init__(self):\n super(SSDService, self).__init__()\n\n # Threshold is used to pick the detection boxes with score > threshold.\n # The detections from this network will be of the format - [[class_id, score, x1, y1, x2, y2]].\n # We pick all detections where 'score > threshold'.\n # You can experiment with different threshold to see the best threshold for the use-case.\n self.threshold = 0.2\n\n # This is used to save the original input image shape.\n # This is required for preparing the bounding box of the detected object \"relative to\n # original input\"\n self.input_width = None\n self.input_height = None\n\n def preprocess(self, batch):\n \"\"\"\n Input image buffer from data is read into NDArray. Then, resized to\n expected shape. Swaps axes to convert image from BGR format to RGB.\n Returns the preprocessed NDArray as a list for next step, Inference.\n \"\"\"\n\n # Read input\n img = batch[0].get(\"data\")\n if img is None:\n img = batch[0].get(\"body\")\n\n input_image = image.read(img)\n\n # Save original input image shape.\n # This is required for preparing the bounding box of the detected object relative to\n # original input\n self.input_height = input_image.shape[0]\n self.input_width = input_image.shape[1]\n\n # Transform input image - resize, BGR to RGB.\n # Reuse MXNetVisionService preprocess to achieve above transformations.\n return super(SSDService, self).preprocess(batch)\n\n def postprocess(self, data):\n \"\"\"\n From the detections, prepares the output in the format of list of\n [(object_class, xmin, ymin, xmax, ymax)]\n object_class is name of the object detected. xmin, ymin, xmax, ymax\n provides the bounding box coordinates.\n\n Example: [(person, 555, 175, 581, 242), (dog, 306, 446, 468, 530)]\n \"\"\"\n\n # Read the detections output after forward pass (inference)\n detections = data[0].asnumpy()\n result = []\n for i in range(detections.shape[0]):\n det = detections[i, :, :]\n res = det[np.where(det[:, 0] >= 0)[0]]\n result.append(res)\n\n # Prepare the output\n dets = result[0]\n classes = self.labels\n width = self.input_width # original input image width\n height = self.input_height # original input image height\n response = []\n for i in range(dets.shape[0]):\n cls_id = int(dets[i, 0])\n if cls_id >= 0:\n score = dets[i, 1]\n if score > self.threshold:\n xmin = int(dets[i, 2] * width)\n ymin = int(dets[i, 3] * height)\n xmax = int(dets[i, 4] * width)\n ymax = int(dets[i, 5] * height)\n class_name = str(cls_id)\n if classes and len(classes) > cls_id:\n class_name = classes[cls_id]\n response.append((class_name, xmin, ymin, xmax, ymax))\n return [response]\n\n\n_service = SSDService()\n\n\ndef handle(data, context):\n if not _service.initialized:\n _service.initialize(context)\n\n if data is None:\n return None\n\n return _service.handle(data, context)\n" }, { "path": "frontend/.gitignore", "content": ".gradle\n.DS_Store\n.idea\n*.iml\nbuild\nlibs\n" }, { "path": "frontend/README.md", "content": "Model Server REST API endpoint\n==============================\n\n## Quick Start\n\n### Building frontend\n\nYou can build frontend using gradle:\n\n```sh\n$ cd frontend\n$ ./gradlew build\n```\n\nYou will find a jar file in frontend/server/build/libs file.\n\n### Starting frontend\n\nFrontend web server using a configuration file to control the behavior of the frontend web server.\nAn sample config.properties can be found in frontend/server/src/test/resources/config.properties.\nThis configure will load a noop model by default. The noop model file is located in frontend/modelarchive/src/test/resources/model/noop-v0.1.model.\n\n#### Start Query service:\n\n```sh\ncd frontend/server\n../gradlew startServer\n```\n\n#### Stop Query service:\n```sh\ncd frontend/server\n../gradlew killServer\n```\n" }, { "path": "frontend/build.gradle", "content": "allprojects {\n version = '1.0'\n\n repositories {\n jcenter()\n }\n\n apply plugin: 'idea'\n idea {\n module {\n outputDir = file('build/classes/java/main')\n testOutputDir = file('build/classes/java/test')\n }\n }\n}\n\ndef javaProjects() {\n return subprojects.findAll();\n}\n\nconfigure(javaProjects()) {\n apply plugin: 'java'\n sourceCompatibility = 1.8\n targetCompatibility = 1.8\n\n defaultTasks 'jar'\n\n\tapply from: file(\"${rootProject.projectDir}/tools/gradle/formatter.gradle\")\n\tapply from: file(\"${rootProject.projectDir}/tools/gradle/check.gradle\")\n\n test {\n useTestNG() {\n // suiteXmlFiles << new File(rootDir, \"testng.xml\") //This is how to add custom testng.xml\n }\n\n testLogging {\n showStandardStreams = true\n events \"passed\", \"skipped\", \"failed\", \"standardOut\", \"standardError\"\n }\n }\n test.finalizedBy(project.tasks.jacocoTestReport)\n\n compileJava {\n options.compilerArgs << \"-proc:none\" << \"-Xlint:all,-options,-static\" << \"-Werror\"\n }\n\n jacocoTestCoverageVerification {\n violationRules {\n rule {\n limit {\n minimum = 0.75\n }\n }\n }\n }\n}\n" }, { "path": "frontend/cts/build.gradle", "content": "dependencies {\n compile project(\":server\")\n}\n\njar {\n manifest {\n attributes 'Main-Class': 'com.amazonaws.ml.mms.cts.Cts'\n }\n includeEmptyDirs = false\n from configurations.runtime.collect { it.isDirectory() ? it : zipTree(it) }\n\n exclude \"META-INF/maven/**\"\n exclude \"META-INF/INDEX.LIST\"\n exclude \"META-INF/MANIFEST*\"\n exclude \"META-INF//LICENSE*\"\n exclude \"META-INF//NOTICE*\"\n}\n" }, { "path": "frontend/cts/src/main/java/com/amazonaws/ml/mms/cts/Cts.java", "content": "/*\n * Copyright 2018 Amazon.com, Inc. or its affiliates. All Rights Reserved.\n *\n * Licensed under the Apache License, Version 2.0 (the \"License\"). You may not use this file except in compliance\n * with the License. A copy of the License is located at\n *\n * http://aws.amazon.com/apache2.0/\n *\n * or in the \"license\" file accompanying this file. This file is distributed on an \"AS IS\" BASIS, WITHOUT WARRANTIES\n * OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions\n * and limitations under the License.\n */\npackage com.amazonaws.ml.mms.cts;\n\nimport com.amazonaws.ml.mms.ModelServer;\nimport com.amazonaws.ml.mms.util.ConfigManager;\nimport io.netty.buffer.Unpooled;\nimport io.netty.handler.codec.http.DefaultFullHttpRequest;\nimport io.netty.handler.codec.http.HttpMethod;\nimport io.netty.handler.codec.http.HttpVersion;\nimport io.netty.handler.codec.http.multipart.HttpPostRequestEncoder;\nimport io.netty.handler.codec.http.multipart.MemoryFileUpload;\nimport java.io.File;\nimport java.io.IOException;\nimport java.net.URL;\nimport java.nio.charset.StandardCharsets;\nimport java.util.ArrayList;\nimport java.util.List;\nimport org.apache.commons.io.FileUtils;\nimport org.apache.commons.io.IOUtils;\nimport org.slf4j.Logger;\nimport org.slf4j.LoggerFactory;\n\npublic final class Cts {\n\n private byte[] kitten;\n private byte[] player1;\n private byte[] player2;\n private List failedModels;\n\n private Cts() {\n failedModels = new ArrayList<>();\n }\n\n public static void main(String[] args) {\n updateLog4jConfiguration();\n\n Cts cts = new Cts();\n cts.startTest();\n }\n\n private void startTest() {\n ConfigManager.init(new ConfigManager.Arguments());\n ConfigManager configManager = ConfigManager.getInstance();\n ModelServer server = new ModelServer(configManager);\n\n Logger logger = LoggerFactory.getLogger(Cts.class);\n try {\n server.start();\n\n kitten =\n loadImage(\n \"https://s3.amazonaws.com/model-server/inputs/kitten.jpg\",\n \"kitten.jpg\");\n player1 =\n loadImage(\n \"https://s3.amazonaws.com/multi-model-server/onnx-arcface/input1.jpg\",\n \"player1.jpg\");\n player2 =\n loadImage(\n \"https://s3.amazonaws.com/multi-model-server/onnx-arcface/input2.jpg\",\n \"player1.jpg\");\n\n HttpClient client = new HttpClient(8081, 8080);\n\n for (ModelInfo info : ModelInfo.MODEL_ARCHIVE_1) {\n runTest(client, info, logger);\n }\n\n for (ModelInfo info : ModelInfo.MODEL_ARCHIVE_04) {\n runTest(client, info, logger);\n }\n } catch (Exception e) {\n logger.error(\"\", e);\n } finally {\n try {\n server.stop();\n } catch (Exception e) {\n logger.error(\"\", e);\n }\n }\n if (failedModels.isEmpty()) {\n logger.info(\"All models passed CTS.\");\n System.exit(0);\n } else {\n logger.info(\"Following models failed CTS:\");\n for (String model : failedModels) {\n logger.info(model);\n }\n System.exit(1);\n }\n }\n\n private void runTest(HttpClient client, ModelInfo info, Logger logger)\n throws HttpPostRequestEncoder.ErrorDataEncoderException, InterruptedException,\n IOException {\n String modelName = info.getModelName();\n String url = info.getUrl();\n int type = info.getType();\n\n logger.info(\"Testing model: {}={}\", modelName, url);\n\n if (!client.registerModel(modelName, url)) {\n failedModels.add(url);\n return;\n }\n\n try {\n if (!predict(client, type, modelName)) {\n failedModels.add(url);\n }\n } finally {\n if (!client.unregisterModel(modelName)) {\n failedModels.add(url);\n }\n }\n }\n\n private boolean predict(HttpClient client, int type, String modelName)\n throws HttpPostRequestEncoder.ErrorDataEncoderException, InterruptedException,\n IOException {\n switch (type) {\n case ModelInfo.FACE_RECOGNITION:\n // arcface\n DefaultFullHttpRequest req =\n new DefaultFullHttpRequest(HttpVersion.HTTP_1_1, HttpMethod.POST, \"/\");\n HttpPostRequestEncoder encoder = new HttpPostRequestEncoder(req, true);\n MemoryFileUpload body =\n new MemoryFileUpload(\n \"img1\", \"img1.jpg\", \"images/jpeg\", null, null, player1.length);\n body.setContent(Unpooled.copiedBuffer(player1));\n encoder.addBodyHttpData(body);\n body =\n new MemoryFileUpload(\n \"img2\", \"img2.jpg\", \"images/jpeg\", null, null, player2.length);\n body.setContent(Unpooled.copiedBuffer(player2));\n encoder.addBodyHttpData(body);\n return client.predict(modelName, req, encoder);\n case ModelInfo.SEMANTIC_SEGMENTATION:\n // duc\n return client.predict(modelName, kitten, \"image/jpeg\");\n case ModelInfo.LANGUAGE_MODELING:\n // lstm\n byte[] json =\n (\"[{'input_sentence': 'on the exchange floor as soon\"\n + \" as ual stopped trading we for a panic\"\n + \" said one top floor trader'}]\")\n .getBytes(StandardCharsets.UTF_8);\n return client.predict(modelName, json, \"application/json\");\n case ModelInfo.IMAGE_CLASSIFICATION:\n case ModelInfo.EMOTION_DETECTION:\n default:\n return client.predict(modelName, kitten, \"image/jpeg\");\n }\n }\n\n private byte[] loadImage(String path, String fileName) throws IOException {\n File file = new File(System.getProperty(\"java.io.tmpdir\"), fileName);\n if (file.exists()) {\n return FileUtils.readFileToByteArray(file);\n }\n byte[] buf = IOUtils.toByteArray(new URL(path));\n FileUtils.writeByteArrayToFile(file, buf);\n return buf;\n }\n\n private static void updateLog4jConfiguration() {\n System.setProperty(\"LOG_LOCATION\", \"logs\");\n System.setProperty(\"METRICS_LOCATION\", \"logs\");\n }\n}\n" }, { "path": "frontend/cts/src/main/java/com/amazonaws/ml/mms/cts/HttpClient.java", "content": "/*\n * Copyright 2018 Amazon.com, Inc. or its affiliates. All Rights Reserved.\n *\n * Licensed under the Apache License, Version 2.0 (the \"License\"). You may not use this file except in compliance\n * with the License. A copy of the License is located at\n *\n * http://aws.amazon.com/apache2.0/\n *\n * or in the \"license\" file accompanying this file. This file is distributed on an \"AS IS\" BASIS, WITHOUT WARRANTIES\n * OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions\n * and limitations under the License.\n */\npackage com.amazonaws.ml.mms.cts;\n\nimport io.netty.bootstrap.Bootstrap;\nimport io.netty.channel.Channel;\nimport io.netty.channel.ChannelHandler;\nimport io.netty.channel.ChannelHandlerContext;\nimport io.netty.channel.ChannelInitializer;\nimport io.netty.channel.ChannelOption;\nimport io.netty.channel.ChannelPipeline;\nimport io.netty.channel.SimpleChannelInboundHandler;\nimport io.netty.channel.nio.NioEventLoopGroup;\nimport io.netty.channel.socket.nio.NioSocketChannel;\nimport io.netty.handler.codec.http.DefaultFullHttpRequest;\nimport io.netty.handler.codec.http.FullHttpResponse;\nimport io.netty.handler.codec.http.HttpClientCodec;\nimport io.netty.handler.codec.http.HttpContentDecompressor;\nimport io.netty.handler.codec.http.HttpHeaderNames;\nimport io.netty.handler.codec.http.HttpMethod;\nimport io.netty.handler.codec.http.HttpObjectAggregator;\nimport io.netty.handler.codec.http.HttpRequest;\nimport io.netty.handler.codec.http.HttpUtil;\nimport io.netty.handler.codec.http.HttpVersion;\nimport io.netty.handler.codec.http.multipart.HttpPostRequestEncoder;\nimport io.netty.handler.stream.ChunkedWriteHandler;\nimport io.netty.handler.timeout.ReadTimeoutException;\nimport io.netty.handler.timeout.ReadTimeoutHandler;\nimport java.io.IOException;\nimport java.net.InetSocketAddress;\nimport java.net.SocketAddress;\nimport java.net.URLEncoder;\nimport java.nio.charset.StandardCharsets;\nimport org.slf4j.Logger;\nimport org.slf4j.LoggerFactory;\n\npublic class HttpClient {\n\n static final Logger logger = LoggerFactory.getLogger(HttpClient.class);\n\n private int managementPort;\n private int inferencePort;\n\n private Bootstrap bootstrap;\n private ClientHandler handler;\n\n public HttpClient(int managementPort, int inferencePort) {\n this.managementPort = managementPort;\n this.inferencePort = inferencePort;\n handler = new ClientHandler();\n bootstrap = bootstrap(handler);\n }\n\n public boolean registerModel(String modelName, String modelUrl)\n throws InterruptedException, IOException {\n Channel channel = connect(bootstrap, managementPort);\n\n String uri =\n \"/models?url=\"\n + URLEncoder.encode(modelUrl, StandardCharsets.UTF_8.name())\n + \"&model_name=\"\n + modelName\n + \"&initial_workers=1&synchronous=true\";\n\n HttpRequest req = new DefaultFullHttpRequest(HttpVersion.HTTP_1_1, HttpMethod.POST, uri);\n channel.writeAndFlush(req).sync();\n\n channel.closeFuture().sync();\n\n int statusCode = handler.getStatusCode();\n String ret = handler.getContent();\n if (statusCode == 200) {\n logger.info(\"registerModel: {} success.\", modelName);\n logger.trace(ret);\n return true;\n }\n logger.warn(\"registerModel: {} failed: {}\", modelUrl, ret);\n return false;\n }\n\n public boolean unregisterModel(String modelName) throws InterruptedException, IOException {\n Channel channel = connect(bootstrap, managementPort);\n\n DefaultFullHttpRequest req =\n new DefaultFullHttpRequest(\n HttpVersion.HTTP_1_1,\n HttpMethod.DELETE,\n \"/models/\" + URLEncoder.encode(modelName, StandardCharsets.UTF_8.name()));\n channel.writeAndFlush(req).sync();\n\n channel.closeFuture().sync();\n\n int statusCode = handler.getStatusCode();\n String ret = handler.getContent();\n if (statusCode == 200) {\n logger.info(\"unregisterModel: {} success.\", modelName);\n logger.trace(ret);\n return true;\n }\n logger.warn(\"unregisterModel: {} failed: {}\", modelName, ret);\n return false;\n }\n\n public boolean predict(String modelName, byte[] content, CharSequence contentType)\n throws InterruptedException, IOException {\n Channel channel = connect(bootstrap, inferencePort);\n\n DefaultFullHttpRequest req =\n new DefaultFullHttpRequest(\n HttpVersion.HTTP_1_1,\n HttpMethod.POST,\n \"/predictions/\"\n + URLEncoder.encode(modelName, StandardCharsets.UTF_8.name()));\n req.content().writeBytes(content);\n HttpUtil.setContentLength(req, content.length);\n req.headers().set(HttpHeaderNames.CONTENT_TYPE, contentType);\n channel.writeAndFlush(req).sync();\n\n channel.closeFuture().sync();\n\n int statusCode = handler.getStatusCode();\n String ret = handler.getContent();\n if (statusCode == 200) {\n logger.info(\"predict: {} success.\", modelName);\n logger.trace(ret);\n return true;\n }\n logger.warn(\"predict: {} failed: {}\", modelName, ret);\n return false;\n }\n\n public boolean predict(\n String modelName, DefaultFullHttpRequest req, HttpPostRequestEncoder requestEncoder)\n throws InterruptedException, HttpPostRequestEncoder.ErrorDataEncoderException,\n IOException {\n Channel channel = connect(bootstrap, inferencePort);\n\n req.setUri(\"/predictions/\" + URLEncoder.encode(modelName, StandardCharsets.UTF_8.name()));\n channel.writeAndFlush(requestEncoder.finalizeRequest());\n if (requestEncoder.isChunked()) {\n channel.writeAndFlush(requestEncoder).sync();\n }\n\n channel.closeFuture().sync();\n\n int statusCode = handler.getStatusCode();\n String ret = handler.getContent();\n if (statusCode == 200) {\n logger.info(\"predict: {} success.\", modelName);\n logger.trace(ret);\n return true;\n }\n logger.warn(\"predict: {} failed: {}\", modelName, ret);\n return false;\n }\n\n private Bootstrap bootstrap(ClientHandler handler) {\n Bootstrap b = new Bootstrap();\n b.group(new NioEventLoopGroup(1))\n .channel(NioSocketChannel.class)\n .option(ChannelOption.CONNECT_TIMEOUT_MILLIS, 10 * 1000)\n .handler(\n new ChannelInitializer() {\n @Override\n public void initChannel(Channel ch) {\n ChannelPipeline p = ch.pipeline();\n p.addLast(new ReadTimeoutHandler(10 * 60 * 1000));\n p.addLast(new HttpClientCodec());\n p.addLast(new HttpContentDecompressor());\n p.addLast(new ChunkedWriteHandler());\n p.addLast(new HttpObjectAggregator(6553600));\n p.addLast(handler);\n }\n });\n return b;\n }\n\n private Channel connect(Bootstrap b, int port) throws InterruptedException {\n SocketAddress address = new InetSocketAddress(\"127.0.0.1\", port);\n return b.connect(address).sync().channel();\n }\n\n @ChannelHandler.Sharable\n private static final class ClientHandler extends SimpleChannelInboundHandler {\n\n private int statusCode;\n private String content;\n\n public ClientHandler() {}\n\n public int getStatusCode() {\n return statusCode;\n }\n\n public String getContent() {\n return content;\n }\n\n @Override\n public void channelRead0(ChannelHandlerContext ctx, FullHttpResponse msg) {\n statusCode = msg.status().code();\n content = msg.content().toString(StandardCharsets.UTF_8);\n ctx.close();\n }\n\n @Override\n public void exceptionCaught(ChannelHandlerContext ctx, Throwable cause) {\n if (cause instanceof IOException) {\n content = \"Failed to connect to MMS\";\n } else if (cause instanceof ReadTimeoutException) {\n content = \"Request to MMS timeout.\";\n } else {\n content = cause.getMessage();\n if (content == null) {\n content = \"NullPointException\";\n }\n logger.error(\"Unknown exception\", cause);\n }\n statusCode = 500;\n ctx.close();\n }\n }\n}\n" }, { "path": "frontend/cts/src/main/java/com/amazonaws/ml/mms/cts/ModelInfo.java", "content": "/*\n * Copyright 2018 Amazon.com, Inc. or its affiliates. All Rights Reserved.\n *\n * Licensed under the Apache License, Version 2.0 (the \"License\"). You may not use this file except in compliance\n * with the License. A copy of the License is located at\n *\n * http://aws.amazon.com/apache2.0/\n *\n * or in the \"license\" file accompanying this file. This file is distributed on an \"AS IS\" BASIS, WITHOUT WARRANTIES\n * OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions\n * and limitations under the License.\n */\npackage com.amazonaws.ml.mms.cts;\n\npublic class ModelInfo {\n\n public static final int IMAGE_CLASSIFICATION = 1;\n public static final int FACE_RECOGNITION = 2;\n public static final int SEMANTIC_SEGMENTATION = 3;\n public static final int EMOTION_DETECTION = 4;\n public static final int LANGUAGE_MODELING = 5;\n\n private static final String S3_PREFIX =\n \"https://s3.amazonaws.com/model-server/model_archive_1.0/\";\n private static final String S3_PREFIX_LEGACY = \"https://s3.amazonaws.com/model-server/models/\";\n\n static final ModelInfo[] MODEL_ARCHIVE_1 = {\n new ModelInfo(\"FERPlus\", ModelInfo.EMOTION_DETECTION),\n new ModelInfo(\"caffenet\"),\n new ModelInfo(\"inception-bn\"),\n new ModelInfo(\"lstm_ptb\", ModelInfo.LANGUAGE_MODELING),\n new ModelInfo(\"nin\"),\n new ModelInfo(\"onnx-arcface-resnet100\", ModelInfo.FACE_RECOGNITION),\n new ModelInfo(\"onnx-duc\", ModelInfo.SEMANTIC_SEGMENTATION),\n new ModelInfo(\"onnx-inception_v1\"),\n new ModelInfo(\"onnx-mobilenet\"),\n new ModelInfo(\"onnx-resnet101v1\"),\n new ModelInfo(\"onnx-resnet101v2\"),\n new ModelInfo(\"onnx-resnet152v1\"),\n new ModelInfo(\"onnx-resnet152v2\"),\n new ModelInfo(\"onnx-resnet18v1\"),\n new ModelInfo(\"onnx-resnet18v2\"),\n new ModelInfo(\"onnx-resnet34v1\"),\n new ModelInfo(\"onnx-resnet34v2\"),\n new ModelInfo(\"onnx-resnet50v1\"),\n new ModelInfo(\"onnx-resnet50v2\"),\n new ModelInfo(\"onnx-squeezenet\"),\n new ModelInfo(\"onnx-vgg16\"),\n new ModelInfo(\"onnx-vgg16_bn\"),\n new ModelInfo(\"onnx-vgg19\"),\n new ModelInfo(\"onnx-vgg19_bn\"),\n new ModelInfo(\"resnet-152\"),\n new ModelInfo(\"resnet-18\"),\n new ModelInfo(\"resnet50_ssd\"),\n new ModelInfo(\"resnext-101-64x4d\"),\n new ModelInfo(\"squeezenet_v1.1\"),\n new ModelInfo(\"squeezenet_v1.2\"),\n new ModelInfo(\"vgg16\"),\n new ModelInfo(\"vgg19\")\n };\n\n static final ModelInfo[] MODEL_ARCHIVE_04 = {\n new ModelInfo(\n \"FERPlus\",\n \"https://s3.amazonaws.com/model-server/models/FERPlus/ferplus.model\",\n EMOTION_DETECTION),\n new ModelInfo(true, \"caffenet\"),\n new ModelInfo(\n \"inception-bn\",\n \"https://s3.amazonaws.com/model-server/models/inception-bn/Inception-BN.model\"),\n new ModelInfo(true, \"lstm_ptb\", LANGUAGE_MODELING),\n new ModelInfo(true, \"nin\"),\n new ModelInfo(\n \"onnx-arcface-resnet100\",\n \"https://s3.amazonaws.com/multi-model-server/onnx-arcface/arcface-resnet100.model\"),\n new ModelInfo(\n \"onnx-duc\",\n \"https://s3.amazonaws.com/multi-model-server/onnx-duc/ResNet_DUC_HDC.model\"),\n new ModelInfo(\n \"onnx-inception_v1\",\n \"https://s3.amazonaws.com/model-server/models/onnx-inception_v1/inception_v1.model\"),\n new ModelInfo(\n \"onnx-mobilenet\",\n \"https://s3.amazonaws.com/multi-model-server/onnx-mobilenet/mobilenetv2-1.0.model\"),\n new ModelInfo(\n \"onnx-resnet101v1\",\n \"https://s3.amazonaws.com/multi-model-server/onnx-resnetv1/resnet101v1.model\"),\n new ModelInfo(\n \"onnx-resnet101v2\",\n \"https://s3.amazonaws.com/multi-model-server/onnx-resnetv2/resnet101v2.model\"),\n new ModelInfo(\n \"onnx-resnet152v1\",\n \"https://s3.amazonaws.com/multi-model-server/onnx-resnetv1/resnet152v1.model\"),\n new ModelInfo(\n \"onnx-resnet152v2\",\n \"https://s3.amazonaws.com/multi-model-server/onnx-resnetv2/resnet152v2.model\"),\n new ModelInfo(\n \"onnx-resnet18v1\",\n \"https://s3.amazonaws.com/multi-model-server/onnx-resnetv1/resnet18v1.model\"),\n new ModelInfo(\n \"onnx-resnet18v2\",\n \"https://s3.amazonaws.com/multi-model-server/onnx-resnetv2/resnet18v2.model\"),\n new ModelInfo(\n \"onnx-resnet34v1\",\n \"https://s3.amazonaws.com/multi-model-server/onnx-resnetv1/resnet34v1.model\"),\n new ModelInfo(\n \"onnx-resnet34v2\",\n \"https://s3.amazonaws.com/multi-model-server/onnx-resnetv2/resnet34v2.model\"),\n new ModelInfo(\n \"onnx-resnet50v1\",\n \"https://s3.amazonaws.com/multi-model-server/onnx-resnetv1/resnet50v1.model\"),\n new ModelInfo(\n \"onnx-resnet50v2\",\n \"https://s3.amazonaws.com/multi-model-server/onnx-resnetv2/resnet50v2.model\"),\n new ModelInfo(\n \"onnx-squeezenet\",\n \"https://s3.amazonaws.com/model-server/models/onnx-squeezenet/squeezenet.model\"),\n new ModelInfo(\n \"onnx-vgg16\", \"https://s3.amazonaws.com/multi-model-server/onnx-vgg16/vgg16.model\"),\n new ModelInfo(\n \"onnx-vgg16_bn\",\n \"https://s3.amazonaws.com/multi-model-server/onnx-vgg16_bn/vgg16_bn.model\"),\n new ModelInfo(\n \"onnx-vgg19\",\n \"https://s3.amazonaws.com/model-server/models/onnx-vgg19/vgg19.model\"),\n new ModelInfo(\n \"onnx-vgg19_bn\",\n \"https://s3.amazonaws.com/multi-model-server/onnx-vgg19_bn/vgg19_bn.model\"),\n new ModelInfo(true, \"resnet-152\"),\n new ModelInfo(true, \"resnet-18\"),\n new ModelInfo(\n \"resnet50_ssd\",\n \"https://s3.amazonaws.com/model-server/models/resnet50_ssd/resnet50_ssd_model.model\"),\n new ModelInfo(true, \"resnext-101-64x4d\"),\n new ModelInfo(true, \"squeezenet_v1.1\"),\n new ModelInfo(true, \"vgg16\"),\n new ModelInfo(true, \"vgg19\")\n };\n\n private String modelName;\n private String url;\n private int type;\n\n public ModelInfo(String modelName) {\n this(false, modelName, IMAGE_CLASSIFICATION);\n }\n\n public ModelInfo(String modelName, int type) {\n this(false, modelName, type);\n }\n\n public ModelInfo(boolean legacy, String modelName) {\n this(legacy, modelName, IMAGE_CLASSIFICATION);\n }\n\n public ModelInfo(boolean legacy, String modelName, int type) {\n this.modelName = modelName;\n if (legacy) {\n url = S3_PREFIX_LEGACY + modelName + '/' + modelName + \".model\";\n } else {\n url = S3_PREFIX + modelName + \".mar\";\n }\n this.type = type;\n }\n\n public ModelInfo(String modelName, String url) {\n this(modelName, url, IMAGE_CLASSIFICATION);\n }\n\n public ModelInfo(String modelName, String url, int type) {\n this.modelName = modelName;\n this.url = url;\n this.type = type;\n }\n\n public String getModelName() {\n return modelName;\n }\n\n public String getUrl() {\n return url;\n }\n\n public int getType() {\n return type;\n }\n}\n" }, { "path": "frontend/cts/src/main/resources/log4j2.xml", "content": "\n\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n\n" }, { "path": "frontend/gradle/wrapper/gradle-wrapper.properties", "content": "#Thu Apr 13 16:20:04 PDT 2017\ndistributionBase=GRADLE_USER_HOME\ndistributionPath=wrapper/dists\nzipStoreBase=GRADLE_USER_HOME\nzipStorePath=wrapper/dists\ndistributionUrl=https\\://services.gradle.org/distributions/gradle-4.9-bin.zip\n" }, { "path": "frontend/gradle.properties", "content": "org.gradle.daemon=true\norg.gradle.jvmargs=-Xmx1024M\nnetty_version=4.1.109.Final\nslf4j_api_version=1.7.32\nslf4j_log4j_version=2.17.1\ngson_version=2.8.9\ncommons_cli_version=1.3.1\ntestng_version=6.8.1\nmms_server_sdk_version=1.0.1\nlmax_disruptor_version=3.4.4\n" }, { "path": "frontend/gradlew", "content": "#!/usr/bin/env bash\n\n##############################################################################\n##\n## Gradle start up script for UN*X\n##\n##############################################################################\n\n# Attempt to set APP_HOME\n# Resolve links: $0 may be a link\nPRG=\"$0\"\n# Need this for relative symlinks.\nwhile [ -h \"$PRG\" ] ; do\n ls=`ls -ld \"$PRG\"`\n link=`expr \"$ls\" : '.*-> \\(.*\\)$'`\n if expr \"$link\" : '/.*' > /dev/null; then\n PRG=\"$link\"\n else\n PRG=`dirname \"$PRG\"`\"/$link\"\n fi\ndone\nSAVED=\"`pwd`\"\ncd \"`dirname \\\"$PRG\\\"`/\" >/dev/null\nAPP_HOME=\"`pwd -P`\"\ncd \"$SAVED\" >/dev/null\n\nAPP_NAME=\"Gradle\"\nAPP_BASE_NAME=`basename \"$0\"`\n\n# Add default JVM options here. You can also use JAVA_OPTS and GRADLE_OPTS to pass JVM options to this script.\nDEFAULT_JVM_OPTS=\"\"\n\n# Use the maximum available, or set MAX_FD != -1 to use that value.\nMAX_FD=\"maximum\"\n\nwarn ( ) {\n echo \"$*\"\n}\n\ndie ( ) {\n echo\n echo \"$*\"\n echo\n exit 1\n}\n\n# OS specific support (must be 'true' or 'false').\ncygwin=false\nmsys=false\ndarwin=false\nnonstop=false\ncase \"`uname`\" in\n CYGWIN* )\n cygwin=true\n ;;\n Darwin* )\n darwin=true\n ;;\n MINGW* )\n msys=true\n ;;\n NONSTOP* )\n nonstop=true\n ;;\nesac\n\nCLASSPATH=$APP_HOME/gradle/wrapper/gradle-wrapper.jar\n\n# Determine the Java command to use to start the JVM.\nif [ -n \"$JAVA_HOME\" ] ; then\n if [ -x \"$JAVA_HOME/jre/sh/java\" ] ; then\n # IBM's JDK on AIX uses strange locations for the executables\n JAVACMD=\"$JAVA_HOME/jre/sh/java\"\n else\n JAVACMD=\"$JAVA_HOME/bin/java\"\n fi\n if [ ! -x \"$JAVACMD\" ] ; then\n die \"ERROR: JAVA_HOME is set to an invalid directory: $JAVA_HOME\n\nPlease set the JAVA_HOME variable in your environment to match the\nlocation of your Java installation.\"\n fi\nelse\n JAVACMD=\"java\"\n which java >/dev/null 2>&1 || die \"ERROR: JAVA_HOME is not set and no 'java' command could be found in your PATH.\n\nPlease set the JAVA_HOME variable in your environment to match the\nlocation of your Java installation.\"\nfi\n\n# Increase the maximum file descriptors if we can.\nif [ \"$cygwin\" = \"false\" -a \"$darwin\" = \"false\" -a \"$nonstop\" = \"false\" ] ; then\n MAX_FD_LIMIT=`ulimit -H -n`\n if [ $? -eq 0 ] ; then\n if [ \"$MAX_FD\" = \"maximum\" -o \"$MAX_FD\" = \"max\" ] ; then\n MAX_FD=\"$MAX_FD_LIMIT\"\n fi\n ulimit -n $MAX_FD\n if [ $? -ne 0 ] ; then\n warn \"Could not set maximum file descriptor limit: $MAX_FD\"\n fi\n else\n warn \"Could not query maximum file descriptor limit: $MAX_FD_LIMIT\"\n fi\nfi\n\n# For Darwin, add options to specify how the application appears in the dock\nif $darwin; then\n GRADLE_OPTS=\"$GRADLE_OPTS \\\"-Xdock:name=$APP_NAME\\\" \\\"-Xdock:icon=$APP_HOME/media/gradle.icns\\\"\"\nfi\n\n# For Cygwin, switch paths to Windows format before running java\nif $cygwin ; then\n APP_HOME=`cygpath --path --mixed \"$APP_HOME\"`\n CLASSPATH=`cygpath --path --mixed \"$CLASSPATH\"`\n JAVACMD=`cygpath --unix \"$JAVACMD\"`\n\n # We build the pattern for arguments to be converted via cygpath\n ROOTDIRSRAW=`find -L / -maxdepth 1 -mindepth 1 -type d 2>/dev/null`\n SEP=\"\"\n for dir in $ROOTDIRSRAW ; do\n ROOTDIRS=\"$ROOTDIRS$SEP$dir\"\n SEP=\"|\"\n done\n OURCYGPATTERN=\"(^($ROOTDIRS))\"\n # Add a user-defined pattern to the cygpath arguments\n if [ \"$GRADLE_CYGPATTERN\" != \"\" ] ; then\n OURCYGPATTERN=\"$OURCYGPATTERN|($GRADLE_CYGPATTERN)\"\n fi\n # Now convert the arguments - kludge to limit ourselves to /bin/sh\n i=0\n for arg in \"$@\" ; do\n CHECK=`echo \"$arg\"|egrep -c \"$OURCYGPATTERN\" -`\n CHECK2=`echo \"$arg\"|egrep -c \"^-\"` ### Determine if an option\n\n if [ $CHECK -ne 0 ] && [ $CHECK2 -eq 0 ] ; then ### Added a condition\n eval `echo args$i`=`cygpath --path --ignore --mixed \"$arg\"`\n else\n eval `echo args$i`=\"\\\"$arg\\\"\"\n fi\n i=$((i+1))\n done\n case $i in\n (0) set -- ;;\n (1) set -- \"$args0\" ;;\n (2) set -- \"$args0\" \"$args1\" ;;\n (3) set -- \"$args0\" \"$args1\" \"$args2\" ;;\n (4) set -- \"$args0\" \"$args1\" \"$args2\" \"$args3\" ;;\n (5) set -- \"$args0\" \"$args1\" \"$args2\" \"$args3\" \"$args4\" ;;\n (6) set -- \"$args0\" \"$args1\" \"$args2\" \"$args3\" \"$args4\" \"$args5\" ;;\n (7) set -- \"$args0\" \"$args1\" \"$args2\" \"$args3\" \"$args4\" \"$args5\" \"$args6\" ;;\n (8) set -- \"$args0\" \"$args1\" \"$args2\" \"$args3\" \"$args4\" \"$args5\" \"$args6\" \"$args7\" ;;\n (9) set -- \"$args0\" \"$args1\" \"$args2\" \"$args3\" \"$args4\" \"$args5\" \"$args6\" \"$args7\" \"$args8\" ;;\n esac\nfi\n\n# Split up the JVM_OPTS And GRADLE_OPTS values into an array, following the shell quoting and substitution rules\nfunction splitJvmOpts() {\n JVM_OPTS=(\"$@\")\n}\neval splitJvmOpts $DEFAULT_JVM_OPTS $JAVA_OPTS $GRADLE_OPTS\nJVM_OPTS[${#JVM_OPTS[*]}]=\"-Dorg.gradle.appname=$APP_BASE_NAME\"\n\n# by default we should be in the correct project dir, but when run from Finder on Mac, the cwd is wrong\nif [[ \"$(uname)\" == \"Darwin\" ]] && [[ \"$HOME\" == \"$PWD\" ]]; then\n cd \"$(dirname \"$0\")\"\nfi\n\nexec \"$JAVACMD\" \"${JVM_OPTS[@]}\" -classpath \"$CLASSPATH\" org.gradle.wrapper.GradleWrapperMain \"$@\"\n" }, { "path": "frontend/gradlew.bat", "content": "@if \"%DEBUG%\" == \"\" @echo off\r\n@rem ##########################################################################\r\n@rem\r\n@rem Gradle startup script for Windows\r\n@rem\r\n@rem ##########################################################################\r\n\r\n@rem Set local scope for the variables with windows NT shell\r\nif \"%OS%\"==\"Windows_NT\" setlocal\r\n\r\nset DIRNAME=%~dp0\r\nif \"%DIRNAME%\" == \"\" set DIRNAME=.\r\nset APP_BASE_NAME=%~n0\r\nset APP_HOME=%DIRNAME%\r\n\r\n@rem Add default JVM options here. You can also use JAVA_OPTS and GRADLE_OPTS to pass JVM options to this script.\r\nset DEFAULT_JVM_OPTS=\r\n\r\n@rem Find java.exe\r\nif defined JAVA_HOME goto findJavaFromJavaHome\r\n\r\nset JAVA_EXE=java.exe\r\n%JAVA_EXE% -version >NUL 2>&1\r\nif \"%ERRORLEVEL%\" == \"0\" goto init\r\n\r\necho.\r\necho ERROR: JAVA_HOME is not set and no 'java' command could be found in your PATH.\r\necho.\r\necho Please set the JAVA_HOME variable in your environment to match the\r\necho location of your Java installation.\r\n\r\ngoto fail\r\n\r\n:findJavaFromJavaHome\r\nset JAVA_HOME=%JAVA_HOME:\"=%\r\nset JAVA_EXE=%JAVA_HOME%/bin/java.exe\r\n\r\nif exist \"%JAVA_EXE%\" goto init\r\n\r\necho.\r\necho ERROR: JAVA_HOME is set to an invalid directory: %JAVA_HOME%\r\necho.\r\necho Please set the JAVA_HOME variable in your environment to match the\r\necho location of your Java installation.\r\n\r\ngoto fail\r\n\r\n:init\r\n@rem Get command-line arguments, handling Windows variants\r\n\r\nif not \"%OS%\" == \"Windows_NT\" goto win9xME_args\r\n\r\n:win9xME_args\r\n@rem Slurp the command line arguments.\r\nset CMD_LINE_ARGS=\r\nset _SKIP=2\r\n\r\n:win9xME_args_slurp\r\nif \"x%~1\" == \"x\" goto execute\r\n\r\nset CMD_LINE_ARGS=%*\r\n\r\n:execute\r\n@rem Setup the command line\r\n\r\nset CLASSPATH=%APP_HOME%\\gradle\\wrapper\\gradle-wrapper.jar\r\n\r\n@rem Execute Gradle\r\n\"%JAVA_EXE%\" %DEFAULT_JVM_OPTS% %JAVA_OPTS% %GRADLE_OPTS% \"-Dorg.gradle.appname=%APP_BASE_NAME%\" -classpath \"%CLASSPATH%\" org.gradle.wrapper.GradleWrapperMain %CMD_LINE_ARGS%\r\n\r\n:end\r\n@rem End local scope for the variables with windows NT shell\r\nif \"%ERRORLEVEL%\"==\"0\" goto mainEnd\r\n\r\n:fail\r\nrem Set variable GRADLE_EXIT_CONSOLE if you need the _script_ return code instead of\r\nrem the _cmd.exe /c_ return code!\r\nif not \"\" == \"%GRADLE_EXIT_CONSOLE%\" exit 1\r\nexit /b 1\r\n\r\n:mainEnd\r\nif \"%OS%\"==\"Windows_NT\" endlocal\r\n\r\n:omega\r\n" }, { "path": "frontend/modelarchive/build.gradle", "content": "dependencies {\n compile \"commons-io:commons-io:2.6\"\n compile \"org.slf4j:slf4j-api:${slf4j_api_version}\"\n compile \"org.apache.logging.log4j:log4j-slf4j-impl:${slf4j_log4j_version}\"\n compile \"com.google.code.gson:gson:${gson_version}\"\n\n testCompile \"commons-cli:commons-cli:${commons_cli_version}\"\n testCompile \"org.testng:testng:${testng_version}\"\n}\n" }, { "path": "frontend/modelarchive/src/main/java/com/amazonaws/ml/mms/archive/DownloadModelException.java", "content": "/*\n * Copyright 2018 Amazon.com, Inc. or its affiliates. All Rights Reserved.\n *\n * Licensed under the Apache License, Version 2.0 (the \"License\"). You may not use this file except in compliance\n * with the License. A copy of the License is located at\n *\n * http://aws.amazon.com/apache2.0/\n *\n * or in the \"license\" file accompanying this file. This file is distributed on an \"AS IS\" BASIS, WITHOUT WARRANTIES\n * OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions\n * and limitations under the License.\n */\npackage com.amazonaws.ml.mms.archive;\n\npublic class DownloadModelException extends ModelException {\n\n static final long serialVersionUID = 1L;\n\n /**\n * Constructs an {@code DownloadModelException} with the specified detail message.\n *\n * @param message The detail message (which is saved for later retrieval by the {@link\n * #getMessage()} method)\n */\n public DownloadModelException(String message) {\n super(message);\n }\n\n /**\n * Constructs an {@code DownloadModelException} with the specified detail message and cause.\n *\n *

Note that the detail message associated with {@code cause} is not automatically\n * incorporated into this exception's detail message.\n *\n * @param message The detail message (which is saved for later retrieval by the {@link\n * #getMessage()} method)\n * @param cause The cause (which is saved for later retrieval by the {@link #getCause()}\n * method). (A null value is permitted, and indicates that the cause is nonexistent or\n * unknown.)\n */\n public DownloadModelException(String message, Throwable cause) {\n super(message, cause);\n }\n}\n" }, { "path": "frontend/modelarchive/src/main/java/com/amazonaws/ml/mms/archive/Hex.java", "content": "/*\n * Copyright 2018 Amazon.com, Inc. or its affiliates. All Rights Reserved.\n *\n * Licensed under the Apache License, Version 2.0 (the \"License\"). You may not use this file except in compliance\n * with the License. A copy of the License is located at\n *\n * http://aws.amazon.com/apache2.0/\n *\n * or in the \"license\" file accompanying this file. This file is distributed on an \"AS IS\" BASIS, WITHOUT WARRANTIES\n * OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions\n * and limitations under the License.\n */\npackage com.amazonaws.ml.mms.archive;\n\npublic final class Hex {\n\n private static final char[] HEX_CHARS = {\n '0', '1', '2', '3', '4', '5', '6', '7', '8', '9', 'a', 'b', 'c', 'd', 'e', 'f'\n };\n\n private Hex() {}\n\n public static String toHexString(byte[] block) {\n return toHexString(block, 0, block.length);\n }\n\n public static String toHexString(byte[] block, int offset, int len) {\n if (block == null) {\n return null;\n }\n if (offset < 0 || offset + len > block.length) {\n throw new IllegalArgumentException(\"Invalid offset or length.\");\n }\n\n StringBuilder buf = new StringBuilder();\n for (int i = offset, size = offset + len; i < size; i++) {\n int high = (block[i] & 0xf0) >> 4;\n int low = block[i] & 0x0f;\n\n buf.append(HEX_CHARS[high]);\n buf.append(HEX_CHARS[low]);\n }\n return buf.toString();\n }\n}\n" }, { "path": "frontend/modelarchive/src/main/java/com/amazonaws/ml/mms/archive/InvalidModelException.java", "content": "/*\n * Copyright 2018 Amazon.com, Inc. or its affiliates. All Rights Reserved.\n *\n * Licensed under the Apache License, Version 2.0 (the \"License\"). You may not use this file except in compliance\n * with the License. A copy of the License is located at\n *\n * http://aws.amazon.com/apache2.0/\n *\n * or in the \"license\" file accompanying this file. This file is distributed on an \"AS IS\" BASIS, WITHOUT WARRANTIES\n * OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions\n * and limitations under the License.\n */\npackage com.amazonaws.ml.mms.archive;\n\npublic class InvalidModelException extends ModelException {\n\n static final long serialVersionUID = 1L;\n\n /**\n * Constructs an {@code InvalidModelException} with the specified detail message.\n *\n * @param message The detail message (which is saved for later retrieval by the {@link\n * #getMessage()} method)\n */\n public InvalidModelException(String message) {\n super(message);\n }\n\n /**\n * Constructs an {@code InvalidModelException} with the specified detail message and cause.\n *\n *

Note that the detail message associated with {@code cause} is not automatically\n * incorporated into this exception's detail message.\n *\n * @param message The detail message (which is saved for later retrieval by the {@link\n * #getMessage()} method)\n * @param cause The cause (which is saved for later retrieval by the {@link #getCause()}\n * method). (A null value is permitted, and indicates that the cause is nonexistent or\n * unknown.)\n */\n public InvalidModelException(String message, Throwable cause) {\n super(message, cause);\n }\n}\n" }, { "path": "frontend/modelarchive/src/main/java/com/amazonaws/ml/mms/archive/LegacyManifest.java", "content": "/*\n * Copyright 2018 Amazon.com, Inc. or its affiliates. All Rights Reserved.\n *\n * Licensed under the Apache License, Version 2.0 (the \"License\"). You may not use this file except in compliance\n * with the License. A copy of the License is located at\n *\n * http://aws.amazon.com/apache2.0/\n *\n * or in the \"license\" file accompanying this file. This file is distributed on an \"AS IS\" BASIS, WITHOUT WARRANTIES\n * OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions\n * and limitations under the License.\n */\npackage com.amazonaws.ml.mms.archive;\n\nimport com.google.gson.annotations.SerializedName;\nimport java.util.Map;\n\npublic class LegacyManifest {\n\n @SerializedName(\"Engine\")\n private Map engine;\n\n @SerializedName(\"Model-Archive-Description\")\n private String description;\n\n @SerializedName(\"License\")\n private String license;\n\n @SerializedName(\"Model-Archive-Version\")\n private String version;\n\n @SerializedName(\"Model-Server\")\n private String serverVersion;\n\n @SerializedName(\"Model\")\n private ModelInfo modelInfo;\n\n @SerializedName(\"Created-By\")\n private CreatedBy createdBy;\n\n public LegacyManifest() {}\n\n public Map getEngine() {\n return engine;\n }\n\n public void setEngine(Map engine) {\n this.engine = engine;\n }\n\n public String getDescription() {\n return description;\n }\n\n public void setDescription(String description) {\n this.description = description;\n }\n\n public String getLicense() {\n return license;\n }\n\n public void setLicense(String license) {\n this.license = license;\n }\n\n public String getVersion() {\n return version;\n }\n\n public void setVersion(String version) {\n this.version = version;\n }\n\n public String getServerVersion() {\n return serverVersion;\n }\n\n public void setServerVersion(String serverVersion) {\n this.serverVersion = serverVersion;\n }\n\n public ModelInfo getModelInfo() {\n return modelInfo;\n }\n\n public void setModelInfo(ModelInfo modelInfo) {\n this.modelInfo = modelInfo;\n }\n\n public CreatedBy getCreatedBy() {\n return createdBy;\n }\n\n public void setCreatedBy(CreatedBy createdBy) {\n this.createdBy = createdBy;\n }\n\n public Manifest migrate() throws InvalidModelException {\n Manifest manifest = new Manifest();\n manifest.setDescription(description);\n manifest.setLicense(license);\n manifest.setSpecificationVersion(\"0.1\");\n\n if (createdBy != null) {\n Manifest.Publisher publisher = new Manifest.Publisher();\n publisher.setAuthor(createdBy.getAuthor());\n publisher.setEmail(createdBy.getEmail());\n manifest.setPublisher(publisher);\n }\n\n if (engine != null) {\n Object engineVersion = engine.get(\"MXNet\");\n if (engineVersion instanceof Number) {\n Manifest.Engine eng = new Manifest.Engine();\n eng.setEngineName(\"MXNet\");\n eng.setEngineVersion(engineVersion.toString());\n manifest.setEngine(eng);\n }\n }\n\n Manifest.Model model = new Manifest.Model();\n model.setModelName(modelInfo.getModelName());\n model.setDescription(modelInfo.getDescription());\n model.setHandler(modelInfo.getService());\n model.setModelVersion(\"snapshot\");\n model.addExtension(\"parametersFile\", modelInfo.getParameters());\n model.addExtension(\"symbolFile\", modelInfo.getSymbol());\n manifest.setModel(model);\n\n if (model.getHandler() == null) {\n throw new InvalidModelException(\"Missing Service entry in MANIFEST.json\");\n }\n\n return manifest;\n }\n\n public static final class CreatedBy {\n\n @SerializedName(\"Author\")\n private String author;\n\n @SerializedName(\"Author-Email\")\n private String email;\n\n public CreatedBy() {}\n\n public String getAuthor() {\n return author;\n }\n\n public void setAuthor(String author) {\n this.author = author;\n }\n\n public String getEmail() {\n return email;\n }\n\n public void setEmail(String email) {\n this.email = email;\n }\n }\n\n public static final class ModelInfo {\n\n @SerializedName(\"Parameters\")\n private String parameters;\n\n @SerializedName(\"Symbol\")\n private String symbol;\n\n @SerializedName(\"Description\")\n private String description;\n\n @SerializedName(\"Model-Name\")\n private String modelName;\n\n @SerializedName(\"Service\")\n private String service;\n\n public ModelInfo() {}\n\n public String getParameters() {\n return parameters;\n }\n\n public void setParameters(String parameters) {\n this.parameters = parameters;\n }\n\n public String getSymbol() {\n return symbol;\n }\n\n public void setSymbol(String symbol) {\n this.symbol = symbol;\n }\n\n public String getDescription() {\n return description;\n }\n\n public void setDescription(String description) {\n this.description = description;\n }\n\n public String getModelName() {\n return modelName;\n }\n\n public void setModelName(String modelName) {\n this.modelName = modelName;\n }\n\n public String getService() {\n return service;\n }\n\n public void setService(String service) {\n this.service = service;\n }\n }\n}\n" }, { "path": "frontend/modelarchive/src/main/java/com/amazonaws/ml/mms/archive/Manifest.java", "content": "/*\n * Copyright 2018 Amazon.com, Inc. or its affiliates. All Rights Reserved.\n *\n * Licensed under the Apache License, Version 2.0 (the \"License\"). You may not use this file except in compliance\n * with the License. A copy of the License is located at\n *\n * http://aws.amazon.com/apache2.0/\n *\n * or in the \"license\" file accompanying this file. This file is distributed on an \"AS IS\" BASIS, WITHOUT WARRANTIES\n * OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions\n * and limitations under the License.\n */\npackage com.amazonaws.ml.mms.archive;\n\nimport com.google.gson.annotations.SerializedName;\nimport java.util.LinkedHashMap;\nimport java.util.Map;\n\npublic class Manifest {\n\n private String specificationVersion;\n private String implementationVersion;\n private String description;\n private String modelServerVersion;\n private String license;\n private RuntimeType runtime;\n private Engine engine;\n private Model model;\n private Publisher publisher;\n\n public Manifest() {\n specificationVersion = \"1.0\";\n implementationVersion = \"1.0\";\n modelServerVersion = \"1.0\";\n license = \"Apache 2.0\";\n runtime = RuntimeType.PYTHON;\n model = new Model();\n }\n\n public String getSpecificationVersion() {\n return specificationVersion;\n }\n\n public void setSpecificationVersion(String specificationVersion) {\n this.specificationVersion = specificationVersion;\n }\n\n public String getImplementationVersion() {\n return implementationVersion;\n }\n\n public void setImplementationVersion(String implementationVersion) {\n this.implementationVersion = implementationVersion;\n }\n\n public String getDescription() {\n return description;\n }\n\n public void setDescription(String description) {\n this.description = description;\n }\n\n public String getModelServerVersion() {\n return modelServerVersion;\n }\n\n public void setModelServerVersion(String modelServerVersion) {\n this.modelServerVersion = modelServerVersion;\n }\n\n public String getLicense() {\n return license;\n }\n\n public void setLicense(String license) {\n this.license = license;\n }\n\n public RuntimeType getRuntime() {\n return runtime;\n }\n\n public void setRuntime(RuntimeType runtime) {\n this.runtime = runtime;\n }\n\n public Engine getEngine() {\n return engine;\n }\n\n public void setEngine(Engine engine) {\n this.engine = engine;\n }\n\n public Model getModel() {\n return model;\n }\n\n public void setModel(Model model) {\n this.model = model;\n }\n\n public Publisher getPublisher() {\n return publisher;\n }\n\n public void setPublisher(Publisher publisher) {\n this.publisher = publisher;\n }\n\n public static final class Publisher {\n\n private String author;\n private String email;\n\n public Publisher() {}\n\n public String getAuthor() {\n return author;\n }\n\n public void setAuthor(String author) {\n this.author = author;\n }\n\n public String getEmail() {\n return email;\n }\n\n public void setEmail(String email) {\n this.email = email;\n }\n }\n\n public static final class Engine {\n\n private String engineName;\n private String engineVersion;\n\n public Engine() {}\n\n public String getEngineName() {\n return engineName;\n }\n\n public void setEngineName(String engineName) {\n this.engineName = engineName;\n }\n\n public String getEngineVersion() {\n return engineVersion;\n }\n\n public void setEngineVersion(String engineVersion) {\n this.engineVersion = engineVersion;\n }\n }\n\n public static final class Model {\n\n private String modelName;\n private String description;\n private String modelVersion;\n private Map extensions;\n private String handler;\n\n public Model() {}\n\n public String getModelName() {\n return modelName;\n }\n\n public void setModelName(String modelName) {\n this.modelName = modelName;\n }\n\n public String getDescription() {\n return description;\n }\n\n public void setDescription(String description) {\n this.description = description;\n }\n\n public String getModelVersion() {\n return modelVersion;\n }\n\n public void setModelVersion(String modelVersion) {\n this.modelVersion = modelVersion;\n }\n\n public Map getExtensions() {\n return extensions;\n }\n\n public void setExtensions(Map extensions) {\n this.extensions = extensions;\n }\n\n public void addExtension(String key, Object value) {\n if (extensions == null) {\n extensions = new LinkedHashMap<>();\n }\n extensions.put(key, value);\n }\n\n public String getHandler() {\n return handler;\n }\n\n public void setHandler(String handler) {\n this.handler = handler;\n }\n }\n\n public enum RuntimeType {\n @SerializedName(\"python\")\n PYTHON(\"python\"),\n @SerializedName(\"python2\")\n PYTHON2(\"python2\"),\n @SerializedName(\"python3\")\n PYTHON3(\"python3\");\n\n String value;\n\n RuntimeType(String value) {\n this.value = value;\n }\n\n public String getValue() {\n return value;\n }\n\n public static RuntimeType fromValue(String value) {\n for (RuntimeType runtime : values()) {\n if (runtime.value.equals(value)) {\n return runtime;\n }\n }\n throw new IllegalArgumentException(\"Invalid RuntimeType value: \" + value);\n }\n }\n}\n" }, { "path": "frontend/modelarchive/src/main/java/com/amazonaws/ml/mms/archive/ModelArchive.java", "content": "/*\n * Copyright 2018 Amazon.com, Inc. or its affiliates. All Rights Reserved.\n *\n * Licensed under the Apache License, Version 2.0 (the \"License\"). You may not use this file except in compliance\n * with the License. A copy of the License is located at\n *\n * http://aws.amazon.com/apache2.0/\n *\n * or in the \"license\" file accompanying this file. This file is distributed on an \"AS IS\" BASIS, WITHOUT WARRANTIES\n * OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions\n * and limitations under the License.\n */\npackage com.amazonaws.ml.mms.archive;\n\nimport com.google.gson.Gson;\nimport com.google.gson.GsonBuilder;\nimport com.google.gson.JsonObject;\nimport com.google.gson.JsonParseException;\nimport com.google.gson.JsonParser;\nimport com.google.gson.JsonPrimitive;\nimport java.io.File;\nimport java.io.FileInputStream;\nimport java.io.FileOutputStream;\nimport java.io.IOException;\nimport java.io.InputStream;\nimport java.io.InputStreamReader;\nimport java.io.Reader;\nimport java.net.HttpURLConnection;\nimport java.net.MalformedURLException;\nimport java.net.SocketTimeoutException;\nimport java.net.URL;\nimport java.nio.charset.StandardCharsets;\nimport java.security.DigestInputStream;\nimport java.security.MessageDigest;\nimport java.security.NoSuchAlgorithmException;\nimport java.util.Enumeration;\nimport java.util.regex.Pattern;\nimport java.util.zip.ZipEntry;\nimport java.util.zip.ZipFile;\nimport java.util.zip.ZipOutputStream;\nimport org.apache.commons.io.FileUtils;\nimport org.apache.commons.io.IOUtils;\nimport org.slf4j.Logger;\nimport org.slf4j.LoggerFactory;\n\npublic class ModelArchive {\n\n private static final Logger logger = LoggerFactory.getLogger(ModelArchive.class);\n\n public static final Gson GSON = new GsonBuilder().setPrettyPrinting().create();\n\n private static final Pattern URL_PATTERN =\n Pattern.compile(\"http(s)?://.*\", Pattern.CASE_INSENSITIVE);\n\n private static final String MANIFEST_FILE = \"MANIFEST.json\";\n\n private Manifest manifest;\n private String url;\n private File modelDir;\n private boolean extracted;\n\n public ModelArchive(Manifest manifest, String url, File modelDir, boolean extracted) {\n this.manifest = manifest;\n this.url = url;\n this.modelDir = modelDir;\n this.extracted = extracted;\n }\n\n public static ModelArchive downloadModel(String modelStore, String url)\n throws ModelException, IOException {\n if (URL_PATTERN.matcher(url).matches()) {\n File modelDir = download(url);\n return load(url, modelDir, true);\n }\n\n if (url.contains(\"..\")) {\n throw new ModelNotFoundException(\"Relative path is not allowed in url: \" + url);\n }\n\n if (modelStore == null) {\n throw new ModelNotFoundException(\"Model store has not been configured.\");\n }\n\n File modelLocation = new File(modelStore, url);\n if (!modelLocation.exists()) {\n throw new ModelNotFoundException(\"Model not found in model store: \" + url);\n }\n if (modelLocation.isFile()) {\n try (InputStream is = new FileInputStream(modelLocation)) {\n File unzipDir = unzip(is, null);\n return load(url, unzipDir, true);\n }\n }\n return load(url, modelLocation, false);\n }\n\n public static void migrate(File legacyModelFile, File destination)\n throws InvalidModelException, IOException {\n boolean failed = true;\n try (ZipFile zip = new ZipFile(legacyModelFile);\n ZipOutputStream zos = new ZipOutputStream(new FileOutputStream(destination))) {\n\n ZipEntry manifestEntry = zip.getEntry(MANIFEST_FILE);\n if (manifestEntry == null) {\n throw new InvalidModelException(\"Missing manifest file in model archive.\");\n }\n\n InputStream is = zip.getInputStream(manifestEntry);\n Reader reader = new InputStreamReader(is, StandardCharsets.UTF_8);\n\n JsonObject json = JsonParser.parseReader(reader).getAsJsonObject();\n JsonPrimitive version = json.getAsJsonPrimitive(\"specificationVersion\");\n Manifest manifest;\n if (version != null && \"1.0\".equals(version.getAsString())) {\n throw new InvalidModelException(\"model archive is already in 1.0 version.\");\n }\n\n LegacyManifest legacyManifest = GSON.fromJson(json, LegacyManifest.class);\n manifest = legacyManifest.migrate();\n\n zos.putNextEntry(new ZipEntry(\"MAR-INF/\"));\n zos.putNextEntry(new ZipEntry(\"MAR-INF/\" + MANIFEST_FILE));\n zos.write(GSON.toJson(manifest).getBytes(StandardCharsets.UTF_8));\n\n Enumeration en = zip.entries();\n while (en.hasMoreElements()) {\n ZipEntry entry = en.nextElement();\n String name = entry.getName();\n if (MANIFEST_FILE.equalsIgnoreCase(name) || name.startsWith(\".\")) {\n continue;\n }\n zos.putNextEntry(new ZipEntry(name));\n if (!entry.isDirectory()) {\n IOUtils.copy(zip.getInputStream(entry), zos);\n }\n }\n failed = false;\n } finally {\n if (failed) {\n FileUtils.deleteQuietly(destination);\n }\n }\n }\n\n private static File download(String path) throws ModelException, IOException {\n HttpURLConnection conn;\n try {\n URL url = new URL(path);\n conn = (HttpURLConnection) url.openConnection();\n if (conn.getResponseCode() != HttpURLConnection.HTTP_OK) {\n throw new DownloadModelException(\n \"Failed to download model from: \"\n + path\n + \", code: \"\n + conn.getResponseCode());\n }\n } catch (MalformedURLException | RuntimeException e) {\n // URLConnection may throw raw RuntimeException if port is out of range.\n throw new ModelNotFoundException(\"Invalid model url: \" + path, e);\n } catch (IOException e) {\n throw new DownloadModelException(\"Failed to download model from: \" + path, e);\n }\n\n try {\n String eTag = conn.getHeaderField(\"ETag\");\n File tmpDir = new File(System.getProperty(\"java.io.tmpdir\"));\n File modelDir = new File(tmpDir, \"models\");\n FileUtils.forceMkdir(modelDir);\n if (eTag != null) {\n if (eTag.startsWith(\"\\\"\") && eTag.endsWith(\"\\\"\") && eTag.length() > 2) {\n eTag = eTag.substring(1, eTag.length() - 1);\n }\n File dir = new File(modelDir, eTag);\n if (dir.exists()) {\n logger.info(\"model folder already exists: {}\", eTag);\n return dir;\n }\n }\n\n return unzip(conn.getInputStream(), eTag);\n } catch (SocketTimeoutException e) {\n throw new DownloadModelException(\"Download model timeout: \" + path, e);\n }\n }\n\n private static ModelArchive load(String url, File dir, boolean extracted)\n throws InvalidModelException, IOException {\n boolean failed = true;\n try {\n File manifestFile = new File(dir, \"MAR-INF/\" + MANIFEST_FILE);\n Manifest manifest;\n if (manifestFile.exists()) {\n // Must be MMS 1.0 or later\n manifest = readFile(manifestFile, Manifest.class);\n } else {\n manifestFile = new File(dir, MANIFEST_FILE);\n boolean nested = false;\n if (!manifestFile.exists()) {\n // Found MANIFEST.json in top level;\n manifestFile = findFile(dir, MANIFEST_FILE, true); // for 0.1 model archive\n nested = true;\n }\n\n if (manifestFile == null) {\n // Must be 1.0\n manifest = new Manifest();\n } else {\n // 0.1 model may have extra parent directory\n LegacyManifest legacyManifest = readFile(manifestFile, LegacyManifest.class);\n manifest = legacyManifest.migrate();\n File modelDir = manifestFile.getParentFile();\n if (extracted && nested) {\n // Move all file to top level, so we can clean up properly.\n moveToTopLevel(modelDir, dir);\n }\n }\n }\n\n failed = false;\n return new ModelArchive(manifest, url, dir, extracted);\n } finally {\n if (extracted && failed) {\n FileUtils.deleteQuietly(dir);\n }\n }\n }\n\n private static T readFile(File file, Class type)\n throws InvalidModelException, IOException {\n try (Reader r = new InputStreamReader(new FileInputStream(file), StandardCharsets.UTF_8)) {\n return GSON.fromJson(r, type);\n } catch (JsonParseException e) {\n throw new InvalidModelException(\"Failed to parse signature.json.\", e);\n }\n }\n\n private static File findFile(File dir, String fileName, boolean recursive) {\n File[] list = dir.listFiles();\n if (list == null) {\n return null;\n }\n for (File file : list) {\n if (recursive && file.isDirectory()) {\n File f = findFile(file, fileName, false);\n if (f != null) {\n return f;\n }\n } else if (file.getName().equalsIgnoreCase(fileName)) {\n return file;\n }\n }\n return null;\n }\n\n private static void moveToTopLevel(File from, File to) throws IOException {\n File[] list = from.listFiles();\n if (list != null) {\n for (File file : list) {\n if (file.isDirectory()) {\n FileUtils.moveDirectoryToDirectory(file, to, false);\n } else {\n FileUtils.moveFileToDirectory(file, to, false);\n }\n }\n }\n }\n\n public static File unzip(InputStream is, String eTag) throws IOException {\n File tmpDir = FileUtils.getTempDirectory();\n File modelDir = new File(tmpDir, \"models\");\n FileUtils.forceMkdir(modelDir);\n\n File tmp = File.createTempFile(\"model\", \".download\");\n FileUtils.forceDelete(tmp);\n FileUtils.forceMkdir(tmp);\n\n MessageDigest md;\n try {\n md = MessageDigest.getInstance(\"SHA1\");\n } catch (NoSuchAlgorithmException e) {\n throw new AssertionError(e);\n }\n ZipUtils.unzip(new DigestInputStream(is, md), tmp);\n if (eTag == null) {\n eTag = Hex.toHexString(md.digest());\n }\n File dir = new File(modelDir, eTag);\n if (dir.exists()) {\n FileUtils.deleteDirectory(tmp);\n logger.info(\"model folder already exists: {}\", eTag);\n return dir;\n }\n\n FileUtils.moveDirectory(tmp, dir);\n\n return dir;\n }\n\n public void validate() throws InvalidModelException {\n Manifest.Model model = manifest.getModel();\n try {\n if (model == null) {\n throw new InvalidModelException(\"Missing Model entry in manifest file.\");\n }\n\n if (model.getModelName() == null) {\n throw new InvalidModelException(\"Model name is not defined.\");\n }\n\n if (model.getHandler() == null) {\n throw new InvalidModelException(\"Model handler is not defined.\");\n }\n\n if (manifest.getRuntime() == null) {\n throw new InvalidModelException(\"Runtime is not defined or invalid.\");\n }\n\n if (manifest.getEngine() != null && manifest.getEngine().getEngineName() == null) {\n throw new InvalidModelException(\"engineName is required in .\");\n }\n } catch (InvalidModelException e) {\n clean();\n throw e;\n }\n }\n\n public String getHandler() {\n return manifest.getModel().getHandler();\n }\n\n public Manifest getManifest() {\n return manifest;\n }\n\n public String getUrl() {\n return url;\n }\n\n public File getModelDir() {\n return modelDir;\n }\n\n public String getModelName() {\n return manifest.getModel().getModelName();\n }\n\n public void clean() {\n if (url != null && extracted) {\n FileUtils.deleteQuietly(modelDir);\n }\n }\n}\n" }, { "path": "frontend/modelarchive/src/main/java/com/amazonaws/ml/mms/archive/ModelException.java", "content": "/*\n * Copyright 2018 Amazon.com, Inc. or its affiliates. All Rights Reserved.\n *\n * Licensed under the Apache License, Version 2.0 (the \"License\"). You may not use this file except in compliance\n * with the License. A copy of the License is located at\n *\n * http://aws.amazon.com/apache2.0/\n *\n * or in the \"license\" file accompanying this file. This file is distributed on an \"AS IS\" BASIS, WITHOUT WARRANTIES\n * OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions\n * and limitations under the License.\n */\npackage com.amazonaws.ml.mms.archive;\n\npublic class ModelException extends Exception {\n\n static final long serialVersionUID = 1L;\n\n /**\n * Constructs an {@code ModelException} with the specified detail message.\n *\n * @param message The detail message (which is saved for later retrieval by the {@link\n * #getMessage()} method)\n */\n public ModelException(String message) {\n super(message);\n }\n\n /**\n * Constructs an {@code ModelException} with the specified detail message and cause.\n *\n *

Note that the detail message associated with {@code cause} is not automatically\n * incorporated into this exception's detail message.\n *\n * @param message The detail message (which is saved for later retrieval by the {@link\n * #getMessage()} method)\n * @param cause The cause (which is saved for later retrieval by the {@link #getCause()}\n * method). (A null value is permitted, and indicates that the cause is nonexistent or\n * unknown.)\n */\n public ModelException(String message, Throwable cause) {\n super(message, cause);\n }\n}\n" }, { "path": "frontend/modelarchive/src/main/java/com/amazonaws/ml/mms/archive/ModelNotFoundException.java", "content": "/*\n * Copyright 2018 Amazon.com, Inc. or its affiliates. All Rights Reserved.\n *\n * Licensed under the Apache License, Version 2.0 (the \"License\"). You may not use this file except in compliance\n * with the License. A copy of the License is located at\n *\n * http://aws.amazon.com/apache2.0/\n *\n * or in the \"license\" file accompanying this file. This file is distributed on an \"AS IS\" BASIS, WITHOUT WARRANTIES\n * OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions\n * and limitations under the License.\n */\npackage com.amazonaws.ml.mms.archive;\n\npublic class ModelNotFoundException extends ModelException {\n\n static final long serialVersionUID = 1L;\n\n /**\n * Constructs an {@code ModelNotFoundException} with the specified detail message.\n *\n * @param message The detail message (which is saved for later retrieval by the {@link\n * #getMessage()} method)\n */\n public ModelNotFoundException(String message) {\n super(message);\n }\n\n /**\n * Constructs an {@code ModelNotFoundException} with the specified detail message and cause.\n *\n *

Note that the detail message associated with {@code cause} is not automatically\n * incorporated into this exception's detail message.\n *\n * @param message The detail message (which is saved for later retrieval by the {@link\n * #getMessage()} method)\n * @param cause The cause (which is saved for later retrieval by the {@link #getCause()}\n * method). (A null value is permitted, and indicates that the cause is nonexistent or\n * unknown.)\n */\n public ModelNotFoundException(String message, Throwable cause) {\n super(message, cause);\n }\n}\n" }, { "path": "frontend/modelarchive/src/main/java/com/amazonaws/ml/mms/archive/ZipUtils.java", "content": "/*\n * Copyright 2018 Amazon.com, Inc. or its affiliates. All Rights Reserved.\n *\n * Licensed under the Apache License, Version 2.0 (the \"License\"). You may not use this file except in compliance\n * with the License. A copy of the License is located at\n *\n * http://aws.amazon.com/apache2.0/\n *\n * or in the \"license\" file accompanying this file. This file is distributed on an \"AS IS\" BASIS, WITHOUT WARRANTIES\n * OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions\n * and limitations under the License.\n */\npackage com.amazonaws.ml.mms.archive;\n\nimport java.io.File;\nimport java.io.FileFilter;\nimport java.io.FileInputStream;\nimport java.io.FileOutputStream;\nimport java.io.IOException;\nimport java.io.InputStream;\nimport java.io.OutputStream;\nimport java.util.zip.ZipEntry;\nimport java.util.zip.ZipInputStream;\nimport java.util.zip.ZipOutputStream;\nimport org.apache.commons.io.FileUtils;\nimport org.apache.commons.io.IOUtils;\n\npublic final class ZipUtils {\n\n private ZipUtils() {}\n\n public static void zip(File src, File dest, boolean includeRootDir) throws IOException {\n int prefix = src.getCanonicalPath().length();\n if (includeRootDir) {\n prefix -= src.getName().length();\n }\n try (ZipOutputStream zos = new ZipOutputStream(new FileOutputStream(dest))) {\n addToZip(prefix, src, null, zos);\n }\n }\n\n public static void unzip(File src, File dest) throws IOException {\n unzip(new FileInputStream(src), dest);\n }\n\n public static void unzip(InputStream is, File dest) throws IOException {\n try (ZipInputStream zis = new ZipInputStream(is)) {\n ZipEntry entry;\n while ((entry = zis.getNextEntry()) != null) {\n String name = entry.getName();\n File file = new File(dest, name);\n if (entry.isDirectory()) {\n FileUtils.forceMkdir(file);\n } else {\n File parentFile = file.getParentFile();\n FileUtils.forceMkdir(parentFile);\n try (OutputStream os = new FileOutputStream(file)) {\n IOUtils.copy(zis, os);\n }\n }\n }\n }\n }\n\n public static void addToZip(int prefix, File file, FileFilter filter, ZipOutputStream zos)\n throws IOException {\n String name = file.getCanonicalPath().substring(prefix);\n if (name.startsWith(\"/\")) {\n name = name.substring(1);\n }\n if (file.isDirectory()) {\n if (!name.isEmpty()) {\n ZipEntry entry = new ZipEntry(name + '/');\n zos.putNextEntry(entry);\n }\n File[] files = file.listFiles(filter);\n if (files != null) {\n for (File f : files) {\n addToZip(prefix, f, filter, zos);\n }\n }\n } else if (file.isFile()) {\n ZipEntry entry = new ZipEntry(name);\n zos.putNextEntry(entry);\n try (FileInputStream fis = new FileInputStream(file)) {\n IOUtils.copy(fis, zos);\n }\n }\n }\n}\n" }, { "path": "frontend/modelarchive/src/test/java/com/amazonaws/ml/mms/archive/CoverageTest.java", "content": "/*\n * Copyright 2018 Amazon.com, Inc. or its affiliates. All Rights Reserved.\n *\n * Licensed under the Apache License, Version 2.0 (the \"License\"). You may not use this file except in compliance\n * with the License. A copy of the License is located at\n *\n * http://aws.amazon.com/apache2.0/\n *\n * or in the \"license\" file accompanying this file. This file is distributed on an \"AS IS\" BASIS, WITHOUT WARRANTIES\n * OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions\n * and limitations under the License.\n */\npackage com.amazonaws.ml.mms.archive;\n\nimport com.amazonaws.ml.mms.test.TestHelper;\nimport java.io.IOException;\nimport org.testng.annotations.Test;\n\npublic class CoverageTest {\n\n @Test\n public void test() throws IOException, ClassNotFoundException {\n TestHelper.testGetterSetters(ModelArchive.class);\n }\n}\n" }, { "path": "frontend/modelarchive/src/test/java/com/amazonaws/ml/mms/archive/Exporter.java", "content": "/*\n * Copyright 2018 Amazon.com, Inc. or its affiliates. All Rights Reserved.\n *\n * Licensed under the Apache License, Version 2.0 (the \"License\"). You may not use this file except in compliance\n * with the License. A copy of the License is located at\n *\n * http://aws.amazon.com/apache2.0/\n *\n * or in the \"license\" file accompanying this file. This file is distributed on an \"AS IS\" BASIS, WITHOUT WARRANTIES\n * OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions\n * and limitations under the License.\n */\npackage com.amazonaws.ml.mms.archive;\n\nimport com.google.gson.Gson;\nimport com.google.gson.GsonBuilder;\nimport java.io.File;\nimport java.io.FileFilter;\nimport java.io.FileOutputStream;\nimport java.io.IOException;\nimport java.net.URL;\nimport java.nio.charset.StandardCharsets;\nimport java.util.List;\nimport java.util.zip.ZipEntry;\nimport java.util.zip.ZipOutputStream;\nimport org.apache.commons.cli.CommandLine;\nimport org.apache.commons.cli.DefaultParser;\nimport org.apache.commons.cli.HelpFormatter;\nimport org.apache.commons.cli.Option;\nimport org.apache.commons.cli.Options;\nimport org.apache.commons.cli.ParseException;\n\npublic final class Exporter {\n\n private static final Gson GSON = new GsonBuilder().setPrettyPrinting().create();\n\n private Exporter() {}\n\n public static void main(String[] args) {\n String jarName = getJarName();\n\n Options options = Config.getOptions();\n DefaultParser parser = new DefaultParser();\n try {\n if (args.length == 0\n || args[0].equalsIgnoreCase(\"-h\")\n || args[0].equalsIgnoreCase(\"--help\")) {\n printHelp(\"java -jar \" + jarName + \" \", options);\n return;\n }\n\n CommandLine cmd = parser.parse(options, args, null, false);\n List cmdArgs = cmd.getArgList();\n if (cmdArgs.isEmpty()) {\n printHelp(\"java -jar \" + jarName + \" \", options);\n return;\n }\n Config config = new Config(cmd);\n String action = cmdArgs.get(0);\n if (!\"export\".equalsIgnoreCase(action)) {\n printHelp(\"java -jar \" + jarName + \" \", options);\n return;\n }\n\n String modelName = config.getModelName();\n if (!modelName.matches(\"[A-Za-z][A-Za-z0-9_\\\\-.]+\")) {\n System.err.println(\n \"model-name must starts with letter and only allows alphanumeric characters, hyphens, underscore or dot.\");\n return;\n }\n\n File modelPath = new File(config.getModelPath()).getCanonicalFile();\n if (!modelPath.exists()) {\n System.err.println(\"model-path not found: \" + modelName);\n return;\n }\n String output = config.getOutputFile();\n File outputFile;\n if (output == null) {\n outputFile = new File(modelPath.getParentFile(), modelName + \".mar\");\n } else {\n outputFile = new File(output);\n }\n\n final String fileName = modelPath.getName();\n if (modelPath.isFile() && fileName.endsWith(\".model\") || fileName.endsWith(\".mar\")) {\n ModelArchive.migrate(modelPath, outputFile);\n return;\n }\n\n if (!modelPath.isDirectory()) {\n System.err.println(\"model-path should be a directory or model archive file.\");\n return;\n }\n\n File[] files = modelPath.listFiles();\n if (files == null) {\n throw new AssertionError(\n \"Failed list files in folder: \" + modelPath.getAbsolutePath());\n }\n\n Manifest manifest = new Manifest();\n Manifest.Model model = new Manifest.Model();\n manifest.setModel(model);\n\n String runtime = config.getRuntime();\n if (runtime != null) {\n manifest.setRuntime(Manifest.RuntimeType.fromValue(runtime));\n }\n\n File symbolFile = findUniqueFile(files, \"-symbol.json\");\n if (symbolFile != null) {\n model.addExtension(\"symbolFile\", symbolFile.getName());\n }\n\n File paramsFile = findUniqueFile(files, \".params\");\n if (paramsFile != null) {\n model.addExtension(\"parametersFile\", paramsFile.getName());\n }\n\n String handler = config.getHandler();\n if (handler == null) {\n File serviceFile = findUniqueFile(files, \"_service.py\");\n if (serviceFile != null) {\n model.setHandler(serviceFile.getName());\n }\n } else {\n Manifest.RuntimeType runtimeType = manifest.getRuntime();\n if (runtimeType == Manifest.RuntimeType.PYTHON\n || runtimeType == Manifest.RuntimeType.PYTHON2\n || runtimeType == Manifest.RuntimeType.PYTHON3) {\n String[] tokens = handler.split(\":\");\n File serviceFile = new File(modelPath, tokens[0]);\n if (serviceFile.exists()) {\n System.err.println(\"handler file is not found in: \" + modelPath);\n return;\n }\n }\n model.setHandler(handler);\n }\n\n try (ZipOutputStream zos = new ZipOutputStream(new FileOutputStream(outputFile))) {\n zos.putNextEntry(new ZipEntry(\"MANIFEST.json\"));\n zos.write(GSON.toJson(manifest).getBytes(StandardCharsets.UTF_8));\n\n int prefix = modelPath.getCanonicalPath().length();\n\n FileFilter filter =\n pathname -> {\n if (pathname.isHidden()) {\n return false;\n }\n String name = pathname.getName();\n return !\"MANIFEST.json\".equalsIgnoreCase(name);\n };\n\n for (File file : files) {\n if (filter.accept(file)) {\n ZipUtils.addToZip(prefix, file, filter, zos);\n }\n }\n } catch (IOException e) {\n e.printStackTrace();\n if (!outputFile.delete()) {\n outputFile.deleteOnExit();\n }\n }\n } catch (InvalidModelException | IOException e) {\n System.err.println(e.getMessage());\n } catch (ParseException e) {\n System.err.println(e.getMessage());\n printHelp(\"java -jar \" + jarName + \" \", options);\n }\n }\n\n private static void printHelp(String message, Options options) {\n HelpFormatter formatter = new HelpFormatter();\n formatter.setLeftPadding(1);\n formatter.setWidth(120);\n formatter.printHelp(message, options);\n }\n\n private static String getJarName() {\n URL url = Exporter.class.getProtectionDomain().getCodeSource().getLocation();\n String path = url.getPath();\n if (\"file\".equalsIgnoreCase(url.getProtocol())) {\n File file = new File(path);\n if (path.toLowerCase().endsWith(\".jar\")) { // we only support jar file for now\n return file.getName();\n }\n }\n return null;\n }\n\n private static File findUniqueFile(File[] list, String extension) throws InvalidModelException {\n File ret = null;\n for (File file : list) {\n if (file.getName().endsWith(extension)) {\n if (ret != null) {\n throw new InvalidModelException(\n \"Multiple \" + extension + \" file found in the path.\");\n }\n ret = file;\n }\n }\n return ret;\n }\n\n private static final class Config {\n\n private String modelName;\n private String modelPath;\n private String handler;\n private String runtime;\n private String outputFile;\n\n public Config(CommandLine cmd) {\n modelName = cmd.getOptionValue(\"model-name\");\n modelPath = cmd.getOptionValue(\"model-path\");\n handler = cmd.getOptionValue(\"handler\");\n runtime = cmd.getOptionValue(\"runtime\");\n handler = cmd.getOptionValue(\"handler\");\n outputFile = cmd.getOptionValue(\"output-file\");\n }\n\n public static Options getOptions() {\n Options options = new Options();\n options.addOption(\n Option.builder(\"n\")\n .longOpt(\"model-name\")\n .hasArg()\n .required()\n .argName(\"MODEL_NAME\")\n .desc(\n \"Exported model name. Exported file will be named as model-name.model and saved in current working directory.\")\n .build());\n options.addOption(\n Option.builder(\"p\")\n .longOpt(\"model-path\")\n .hasArg()\n .required()\n .argName(\"MODEL_PATH\")\n .desc(\n \"Path to the folder containing model related files or legacy model archive. Signature file is required.\")\n .build());\n options.addOption(\n Option.builder(\"r\")\n .longOpt(\"runtime\")\n .hasArg()\n .argName(\"RUNTIME\")\n .desc(\n \"The runtime environment for the MMS to execute your model custom code, default python2.7\")\n .build());\n options.addOption(\n Option.builder(\"e\")\n .longOpt(\"engine\")\n .hasArg()\n .argName(\"engine\")\n .desc(\"The ML framework for your model, default MXNet\")\n .build());\n options.addOption(\n Option.builder(\"s\")\n .longOpt(\"handler\")\n .hasArg()\n .argName(\"HANDLER\")\n .desc(\n \"The entry-point within your code that MMS can call to begin execution.\")\n .build());\n options.addOption(\n Option.builder(\"o\")\n .longOpt(\"output-file\")\n .hasArg()\n .argName(\"OUTPUT_FILE\")\n .desc(\"Output model archive file path.\")\n .build());\n return options;\n }\n\n public String getModelName() {\n return modelName;\n }\n\n public void setModelName(String modelName) {\n this.modelName = modelName;\n }\n\n public String getModelPath() {\n return modelPath;\n }\n\n public void setModelPath(String modelPath) {\n this.modelPath = modelPath;\n }\n\n public String getHandler() {\n return handler;\n }\n\n public void setHandler(String handler) {\n this.handler = handler;\n }\n\n public String getOutputFile() {\n return outputFile;\n }\n\n public void setOutputFile(String outputFile) {\n this.outputFile = outputFile;\n }\n\n public String getRuntime() {\n return runtime;\n }\n\n public void setRuntime(String runtime) {\n this.runtime = runtime;\n }\n }\n}\n" }, { "path": "frontend/modelarchive/src/test/java/com/amazonaws/ml/mms/archive/ModelArchiveTest.java", "content": "/*\n * Copyright 2018 Amazon.com, Inc. or its affiliates. All Rights Reserved.\n *\n * Licensed under the Apache License, Version 2.0 (the \"License\"). You may not use this file except in compliance\n * with the License. A copy of the License is located at\n *\n * http://aws.amazon.com/apache2.0/\n *\n * or in the \"license\" file accompanying this file. This file is distributed on an \"AS IS\" BASIS, WITHOUT WARRANTIES\n * OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions\n * and limitations under the License.\n */\npackage com.amazonaws.ml.mms.archive;\n\nimport java.io.File;\nimport java.io.IOException;\nimport org.apache.commons.io.FileUtils;\nimport org.testng.Assert;\nimport org.testng.annotations.BeforeTest;\nimport org.testng.annotations.Test;\n\npublic class ModelArchiveTest {\n\n private File output;\n\n @BeforeTest\n public void beforeTest() {\n output = new File(\"build/tmp/test/noop.mar\");\n FileUtils.deleteQuietly(output);\n FileUtils.deleteQuietly(new File(\"build/tmp/test/noop\"));\n FileUtils.deleteQuietly(new File(\"build/tmp/test/noop-v0.1.mar\"));\n File tmp = FileUtils.getTempDirectory();\n FileUtils.deleteQuietly(new File(tmp, \"models\"));\n }\n\n @Test\n public void test() throws ModelException, IOException {\n String modelStore = \"src/test/resources/models\";\n\n // load 0.1 model from model folder\n ModelArchive archive = ModelArchive.downloadModel(modelStore, \"noop-v0.1\");\n Assert.assertEquals(archive.getModelName(), \"noop_v0.1\");\n\n // load 0.1 model from model archive\n File src = new File(modelStore, \"noop-v0.1\");\n File target = new File(\"build/tmp/test\", \"noop-v0.1.mar\");\n ZipUtils.zip(src, target, false);\n archive = ModelArchive.downloadModel(\"build/tmp/test\", \"noop-v0.1.mar\");\n Assert.assertEquals(archive.getModelName(), \"noop_v0.1\");\n\n // load model for s3\n archive =\n ModelArchive.downloadModel(\n modelStore,\n \"https://s3.amazonaws.com/model-server/models/squeezenet_v1.1/squeezenet_v1.1.model\");\n Assert.assertEquals(archive.getModelName(), \"squeezenet_v1.1\");\n\n // test export\n String[] args = new String[4];\n args[0] = \"export\";\n args[1] = \"--model-name=noop\";\n args[2] = \"--model-path=\" + archive.getModelDir();\n args[3] = \"--output-file=\" + output.getAbsolutePath();\n\n Exporter.main(args);\n Assert.assertTrue(output.exists());\n\n FileUtils.forceDelete(output);\n\n ModelArchive.migrate(target, output);\n Assert.assertTrue(output.exists());\n\n // load 1.0 model\n archive = ModelArchive.downloadModel(modelStore, \"noop-v1.0\");\n Assert.assertEquals(archive.getModelName(), \"noop\");\n }\n}\n" }, { "path": "frontend/modelarchive/src/test/java/com/amazonaws/ml/mms/test/TestHelper.java", "content": "/*\n * Copyright 2018 Amazon.com, Inc. or its affiliates. All Rights Reserved.\n *\n * Licensed under the Apache License, Version 2.0 (the \"License\"). You may not use this file except in compliance\n * with the License. A copy of the License is located at\n *\n * http://aws.amazon.com/apache2.0/\n *\n * or in the \"license\" file accompanying this file. This file is distributed on an \"AS IS\" BASIS, WITHOUT WARRANTIES\n * OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions\n * and limitations under the License.\n */\npackage com.amazonaws.ml.mms.test;\n\nimport java.io.File;\nimport java.io.IOException;\nimport java.lang.reflect.Constructor;\nimport java.lang.reflect.Method;\nimport java.net.URL;\nimport java.util.ArrayList;\nimport java.util.Collection;\nimport java.util.Collections;\nimport java.util.Enumeration;\nimport java.util.List;\nimport java.util.jar.JarEntry;\nimport java.util.jar.JarFile;\nimport org.apache.commons.io.FileUtils;\n\npublic final class TestHelper {\n\n private TestHelper() {}\n\n public static void testGetterSetters(Class baseClass)\n throws IOException, ClassNotFoundException {\n List> list = getClasses(baseClass);\n for (Class clazz : list) {\n Constructor[] constructors = clazz.getConstructors();\n Object obj = null;\n for (Constructor con : constructors) {\n try {\n Class[] types = con.getParameterTypes();\n Object[] args = new Object[types.length];\n for (int i = 0; i < args.length; ++i) {\n args[i] = getMockValue(types[i]);\n }\n\n obj = con.newInstance(args);\n } catch (ReflectiveOperationException ignore) {\n // ignore\n }\n }\n if (obj == null) {\n continue;\n }\n\n Method[] methods = clazz.getMethods();\n for (Method method : methods) {\n String methodName = method.getName();\n int parameterCount = method.getParameterCount();\n try {\n if (parameterCount == 0 && methodName.startsWith(\"get\")\n || methodName.startsWith(\"is\")) {\n method.invoke(obj);\n } else if (methodName.startsWith(\"set\") && parameterCount == 1) {\n Class type = method.getParameterTypes()[0];\n method.invoke(obj, getMockValue(type));\n }\n } catch (ReflectiveOperationException ignore) {\n // ignore\n }\n }\n }\n }\n\n private static List> getClasses(Class clazz)\n throws IOException, ClassNotFoundException {\n URL url = clazz.getProtectionDomain().getCodeSource().getLocation();\n String path = url.getPath();\n\n if (!\"file\".equalsIgnoreCase(url.getProtocol())) {\n return Collections.emptyList();\n }\n\n List> classList = new ArrayList<>();\n\n File classPath = new File(path);\n if (classPath.isDirectory()) {\n String rootPath = classPath.getCanonicalPath();\n String[] filters = new String[] {\"class\"};\n Collection files = FileUtils.listFiles(classPath, filters, true);\n for (File file : files) {\n String fileName = file.getCanonicalPath();\n fileName = fileName.substring(rootPath.length() + 1);\n fileName = fileName.substring(0, fileName.lastIndexOf(\".\"));\n fileName = fileName.replace(File.separatorChar, '.');\n\n classList.add(Class.forName(fileName));\n }\n } else if (path.toLowerCase().endsWith(\".jar\")) {\n try (JarFile jarFile = new JarFile(path)) {\n Enumeration en = jarFile.entries();\n while (en.hasMoreElements()) {\n JarEntry entry = en.nextElement();\n String fileName = entry.getName();\n if (fileName.endsWith(\".class\")) {\n fileName = fileName.substring(0, fileName.lastIndexOf(\".\"));\n fileName = fileName.replace('/', '.');\n classList.add(Class.forName(fileName));\n }\n }\n }\n }\n\n return classList;\n }\n\n private static Object getMockValue(Class type) {\n if (type.isPrimitive()) {\n if (type == Boolean.TYPE) {\n return Boolean.TRUE;\n }\n if (type == Character.TYPE) {\n return '0';\n }\n if (type == Byte.TYPE) {\n return (byte) 0;\n }\n if (type == Short.TYPE) {\n return (short) 0;\n }\n if (type == Integer.TYPE) {\n return 0;\n }\n if (type == Long.TYPE) {\n return 0L;\n }\n if (type == Float.TYPE) {\n return 0f;\n }\n if (type == Double.TYPE) {\n return 0d;\n }\n }\n return null;\n }\n}\n" }, { "path": "frontend/modelarchive/src/test/resources/models/custom-return-code/MAR-INF/MANIFEST.json", "content": "{\n \"specificationVersion\": \"1.0\",\n \"implementationVersion\": \"1.0\",\n \"description\": \"noop v1.0\",\n \"modelServerVersion\": \"1.0\",\n \"license\": \"Apache 2.0\",\n \"runtime\": \"python\",\n \"model\": {\n \"modelName\": \"pred-custom-return-code\",\n \"description\": \"Tests for custom return code\",\n \"modelVersion\": \"1.0\",\n \"handler\": \"service:handle\"\n },\n \"publisher\": {\n \"author\": \"MXNet SDK team\",\n \"email\": \"noreply@amazon.com\"\n }\n}\n" }, { "path": "frontend/modelarchive/src/test/resources/models/custom-return-code/service.py", "content": "# Copyright 2018 Amazon.com, Inc. or its affiliates. All Rights Reserved.\n# Licensed under the Apache License, Version 2.0 (the \"License\").\n# You may not use this file except in compliance with the License.\n# A copy of the License is located at\n# http://www.apache.org/licenses/LICENSE-2.0\n# or in the \"license\" file accompanying this file. This file is distributed\n# on an \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either\n# express or implied. See the License for the specific language governing\n# permissions and limitations under the License.\n\nfrom mms.service import PredictionException\n\ndef handle(data, ctx):\n # Data is not none in prediction request\n # Python raises PredictionException with custom error code\n if data is not None:\n raise PredictionException(\"Some Prediction Error\", 599)\n return [\"OK\"]\n" }, { "path": "frontend/modelarchive/src/test/resources/models/error_batch/MAR-INF/MANIFEST.json", "content": "{\n \"specificationVersion\": \"1.0\",\n \"implementationVersion\": \"1.0\",\n \"description\": \"invalid model\",\n \"modelServerVersion\": \"1.0\",\n \"license\": \"Apache 2.0\",\n \"runtime\": \"python\",\n \"model\": {\n \"modelName\": \"err_batch\",\n \"description\": \"batch error model\",\n \"modelVersion\": \"1.0\",\n \"handler\": \"service:handle\"\n },\n \"publisher\": {\n \"author\": \"MXNet SDK team\",\n \"email\": \"noreply@amazon.com\"\n }\n}" }, { "path": "frontend/modelarchive/src/test/resources/models/error_batch/service.py", "content": "# Copyright 2018 Amazon.com, Inc. or its affiliates. All Rights Reserved.\n# Licensed under the Apache License, Version 2.0 (the \"License\").\n# You may not use this file except in compliance with the License.\n# A copy of the License is located at\n# http://www.apache.org/licenses/LICENSE-2.0\n# or in the \"license\" file accompanying this file. This file is distributed\n# on an \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either\n# express or implied. See the License for the specific language governing\n# permissions and limitations under the License.\n\n\"\"\"\nInvalidService defines a invalid model handler for testing purpose.\n\"\"\"\n\n\ndef handle(data, context):\n # This model is created to test reporting of an error in a batch of requests\n if data:\n context.set_response_status(code=507, idx=0)\n return [\"Invalid response\"]\n" }, { "path": "frontend/modelarchive/src/test/resources/models/init-error/MAR-INF/MANIFEST.json", "content": "{\n \"specificationVersion\": \"1.0\",\n \"implementationVersion\": \"1.0\",\n \"description\": \"init error model\",\n \"modelServerVersion\": \"1.0\",\n \"license\": \"Apache 2.0\",\n \"runtime\": \"python\",\n \"model\": {\n \"modelName\": \"init-error\",\n \"description\": \"invalid model that cannot init\",\n \"modelVersion\": \"1.0\",\n \"handler\": \"invalid_service:handle\"\n },\n \"publisher\": {\n \"author\": \"MXNet SDK team\",\n \"email\": \"noreply@amazon.com\"\n }\n}\n" }, { "path": "frontend/modelarchive/src/test/resources/models/init-error/invalid_service.py", "content": "# Copyright 2018 Amazon.com, Inc. or its affiliates. All Rights Reserved.\n# Licensed under the Apache License, Version 2.0 (the \"License\").\n# You may not use this file except in compliance with the License.\n# A copy of the License is located at\n# http://www.apache.org/licenses/LICENSE-2.0\n# or in the \"license\" file accompanying this file. This file is distributed\n# on an \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either\n# express or implied. See the License for the specific language governing\n# permissions and limitations under the License.\n\n\"\"\"\nInvalidService defines a invalid model handler for testing purpose.\n\"\"\"\n\ndef handle(data, context):\n raise RuntimeError(\"Initialize failure.\")\n" }, { "path": "frontend/modelarchive/src/test/resources/models/invalid/MAR-INF/MANIFEST.json", "content": "{\n \"specificationVersion\": \"1.0\",\n \"implementationVersion\": \"1.0\",\n \"description\": \"invalid model\",\n \"modelServerVersion\": \"1.0\",\n \"license\": \"Apache 2.0\",\n \"runtime\": \"python\",\n \"model\": {\n \"modelName\": \"invalid\",\n \"description\": \"invalid model\",\n \"modelVersion\": \"1.0\",\n \"handler\": \"invalid_service:handle\"\n },\n \"publisher\": {\n \"author\": \"MXNet SDK team\",\n \"email\": \"noreply@amazon.com\"\n }\n}" }, { "path": "frontend/modelarchive/src/test/resources/models/invalid/invalid_service.py", "content": "# Copyright 2018 Amazon.com, Inc. or its affiliates. All Rights Reserved.\n# Licensed under the Apache License, Version 2.0 (the \"License\").\n# You may not use this file except in compliance with the License.\n# A copy of the License is located at\n# http://www.apache.org/licenses/LICENSE-2.0\n# or in the \"license\" file accompanying this file. This file is distributed\n# on an \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either\n# express or implied. See the License for the specific language governing\n# permissions and limitations under the License.\n\n\"\"\"\nInvalidService defines a invalid model handler for testing purpose.\n\"\"\"\n\ndef handle(data, context):\n return \"Invalid response\"\n" }, { "path": "frontend/modelarchive/src/test/resources/models/loading-memory-error/MAR-INF/MANIFEST.json", "content": "{\n \"specificationVersion\": \"1.0\",\n \"implementationVersion\": \"1.0\",\n \"description\": \"noop v1.0\",\n \"modelServerVersion\": \"1.0\",\n \"license\": \"Apache 2.0\",\n \"runtime\": \"python\",\n \"model\": {\n \"modelName\": \"mem-err\",\n \"description\": \"Tests for memory error\",\n \"modelVersion\": \"1.0\",\n \"handler\": \"service:handle\"\n },\n \"publisher\": {\n \"author\": \"MXNet SDK team\",\n \"email\": \"noreply@amazon.com\"\n }\n}" }, { "path": "frontend/modelarchive/src/test/resources/models/loading-memory-error/service.py", "content": "# Copyright 2018 Amazon.com, Inc. or its affiliates. All Rights Reserved.\n# Licensed under the Apache License, Version 2.0 (the \"License\").\n# You may not use this file except in compliance with the License.\n# A copy of the License is located at\n# http://www.apache.org/licenses/LICENSE-2.0\n# or in the \"license\" file accompanying this file. This file is distributed\n# on an \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either\n# express or implied. See the License for the specific language governing\n# permissions and limitations under the License.\n\n\ndef handle(ctx, data):\n # Python raises MemoryError when the python program goes out of memory. MMS expects this error from the handler\n # if the handlers can not allocate any further memory.\n raise MemoryError(\"Throwing memory error\")\n" }, { "path": "frontend/modelarchive/src/test/resources/models/logging/MAR-INF/MANIFEST.json", "content": "{\n \"specificationVersion\": \"1.0\",\n \"implementationVersion\": \"1.0\",\n \"description\": \"logging v1.0\",\n \"modelServerVersion\": \"1.0\",\n \"license\": \"Apache 2.0\",\n \"runtime\": \"python\",\n \"model\": {\n \"modelName\": \"logging\",\n \"description\": \"logging test model\",\n \"modelVersion\": \"1.0\",\n \"handler\": \"service:handle\"\n },\n \"publisher\": {\n \"author\": \"MXNet SDK team\",\n \"email\": \"noreply@amazon.com\"\n }\n}\n" }, { "path": "frontend/modelarchive/src/test/resources/models/logging/service.py", "content": "# Copyright 2019 Amazon.com, Inc. or its affiliates. All Rights Reserved.\n# Licensed under the Apache License, Version 2.0 (the \"License\").\n# You may not use this file except in compliance with the License.\n# A copy of the License is located at\n# http://www.apache.org/licenses/LICENSE-2.0\n# or in the \"license\" file accompanying this file. This file is distributed\n# on an \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either\n# express or implied. See the License for the specific language governing\n# permissions and limitations under the License.\n\n\"\"\"\nLoggingService defines a no operational model handler.\n\"\"\"\nimport logging\nimport time\nimport os\n\nclass LoggingService(object):\n \"\"\"\n Logging Model handler implementation.\n\n Extend from BaseModelHandler is optional\n \"\"\"\n\n def __init__(self):\n logging.info(\"LoggingService init\")\n self._context = None\n self.initialized = False\n\n def __del__(self):\n logging.info(\"LoggingService exit\")\n\n def initialize(self, context):\n \"\"\"\n Initialize model. This will be called during model loading time\n\n :param context: model server context\n :return:\n \"\"\"\n self.initialized = True\n self._context = context\n\n @staticmethod\n def inference(model_input):\n \"\"\"\n Internal inference methods\n\n :param model_input: transformed model input data\n :return: inference results\n \"\"\"\n time.sleep(0.01)\n logging.info(\"LoggingService inference [PID]: %d\", os.getpid())\n return [\"{} OK\\n\".format(os.getpid())] * len(model_input)\n\n def handle(self, data, context):\n \"\"\"\n Custom service entry point function.\n\n :param context: model server context\n :param data: list of objects, raw input from request\n :return: list of outputs to be send back to client\n \"\"\"\n # Add your initialization code here\n properties = context.system_properties\n try:\n start_time = time.time()\n data = self.inference(data)\n end_time = time.time()\n\n context.set_response_content_type(0, \"text/plain\")\n content_type = context.request_processor[0].get_request_property(\"Content-Type\")\n return data\n except Exception as e:\n logging.error(e, exc_info=True)\n context.request_processor[0].report_status(500, \"Unknown inference error.\")\n return [\"Error {}\".format(str(e))] * len(data)\n\n\n_service = LoggingService()\n\ndef handle(data, context):\n if not _service.initialized:\n _service.initialize(context)\n\n if data is None:\n return None\n\n return _service.handle(data, context)\n" }, { "path": "frontend/modelarchive/src/test/resources/models/noop-no-manifest/service.py", "content": "# Copyright 2018 Amazon.com, Inc. or its affiliates. All Rights Reserved.\n# Licensed under the Apache License, Version 2.0 (the \"License\").\n# You may not use this file except in compliance with the License.\n# A copy of the License is located at\n# http://www.apache.org/licenses/LICENSE-2.0\n# or in the \"license\" file accompanying this file. This file is distributed\n# on an \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either\n# express or implied. See the License for the specific language governing\n# permissions and limitations under the License.\n\n\"\"\"\nNoopService defines a no operational model handler.\n\"\"\"\nimport logging\nimport time\n\n\nclass NoopService(object):\n \"\"\"\n Noop Model handler implementation.\n\n Extend from BaseModelHandler is optional\n \"\"\"\n\n def __init__(self):\n self._context = None\n self.initialized = False\n\n def initialize(self, context):\n \"\"\"\n Initialize model. This will be called during model loading time\n\n :param context: model server context\n :return:\n \"\"\"\n self.initialized = True\n self._context = context\n\n @staticmethod\n def preprocess(data):\n \"\"\"\n Transform raw input into model input data.\n\n :param data: list of objects, raw input from request\n :return: list of model input data\n \"\"\"\n return data\n\n @staticmethod\n def inference(model_input):\n \"\"\"\n Internal inference methods\n\n :param model_input: transformed model input data\n :return: inference results\n \"\"\"\n return model_input\n\n @staticmethod\n def postprocess(model_output):\n return [\"OK\"] * len(model_output)\n\n def handle(self, data, context):\n \"\"\"\n Custom service entry point function.\n\n :param context: model server context\n :param data: list of objects, raw input from request\n :return: list of outputs to be send back to client\n \"\"\"\n # Add your initialization code here\n request_processor = context.request_processor\n try:\n data = self.preprocess(data)\n data = self.inference(data)\n data = self.postprocess(data)\n\n context.set_response_content_type(0, \"text/plain\")\n\n return data\n except Exception as e:\n logging.error(e, exc_info=True)\n request_processor[0].report_status(500, \"Unknown inference error.\")\n return [\"Error {}\".format(str(e))] * len(data)\n\n\n_service = NoopService()\n\n\ndef handle(data, context):\n if not _service.initialized:\n _service.initialize(context)\n\n if data is None:\n return None\n\n return _service.handle(data, context)\n" }, { "path": "frontend/modelarchive/src/test/resources/models/noop-v0.1/MANIFEST.json", "content": "{\n\t\"Created-By\": {\n\t\t\"Author\": \"MXNet SDK team\",\n\t\t\"Author-Email\": \"noreply@amazon.com\"\n\t},\n \"Engine\": {\n \"MXNet\": 0.12\n }, \n \"Model-Archive-Description\": \"noop v0.1\",\n \"License\": \"Apache 2.0\", \n \"Model-Archive-Version\": 0.1, \n \"Model-Server\": 0.4,\n \"Model\": {\n \"Description\": \"no operation model\", \n \"Service\": \"noop_service.py\",\n \"Symbol\": \"noop-symbol.json\",\n \"Parameters\": \"noop-0000.params\",\n \"Signature\": \"signature.json\",\n \"Model-Name\": \"noop_v0.1\",\n \"Model-Format\": \"MXNet-Symbolic\"\n }\n}\n" }, { "path": "frontend/modelarchive/src/test/resources/models/noop-v0.1/noop_service.py", "content": "# Copyright 2017 Amazon.com, Inc. or its affiliates. All Rights Reserved.\n# Licensed under the Apache License, Version 2.0 (the \"License\").\n# You may not use this file except in compliance with the License.\n# A copy of the License is located at\n# http://www.apache.org/licenses/LICENSE-2.0\n# or in the \"license\" file accompanying this file. This file is distributed\n# on an \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either\n# express or implied. See the License for the specific language governing\n# permissions and limitations under the License.\n\n\"\"\"\nNoopService defines a noop service\n\"\"\"\n\nfrom mms.model_service.mxnet_model_service import SingleNodeService\n\n\nclass NoopService(SingleNodeService):\n \"\"\"\n NoopService defines a noop service.\n \"\"\"\n\n def _inference(self, data):\n return \"OK\"\n\n def ping(self):\n return None\n" }, { "path": "frontend/modelarchive/src/test/resources/models/noop-v0.1/signature.json", "content": "{\n \"inputs\": [\n {\n \"data_name\": \"data\",\n \"shape\": [\n 0,\n 3,\n 224,\n 224\n ]\n }\n ],\n \"input_type\": \"image/jpeg\",\n \"outputs\": [\n {\n \"data_name\": \"data\",\n \"shape\": [\n 0,\n 3,\n 224,\n 224\n ]\n }\n ],\n \"output_type\": \"application/json\"\n}\n" }, { "path": "frontend/modelarchive/src/test/resources/models/noop-v1.0/MAR-INF/MANIFEST.json", "content": "{\n \"specificationVersion\": \"1.0\",\n \"implementationVersion\": \"1.0\",\n \"description\": \"noop v1.0\",\n \"modelServerVersion\": \"1.0\",\n \"license\": \"Apache 2.0\",\n \"runtime\": \"python\",\n \"model\": {\n \"modelName\": \"noop\",\n \"description\": \"no operation model\",\n \"modelVersion\": \"1.0\",\n \"handler\": \"service:handle\"\n },\n \"publisher\": {\n \"author\": \"MXNet SDK team\",\n \"email\": \"noreply@amazon.com\"\n }\n}" }, { "path": "frontend/modelarchive/src/test/resources/models/noop-v1.0/service.py", "content": "# Copyright 2018 Amazon.com, Inc. or its affiliates. All Rights Reserved.\n# Licensed under the Apache License, Version 2.0 (the \"License\").\n# You may not use this file except in compliance with the License.\n# A copy of the License is located at\n# http://www.apache.org/licenses/LICENSE-2.0\n# or in the \"license\" file accompanying this file. This file is distributed\n# on an \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either\n# express or implied. See the License for the specific language governing\n# permissions and limitations under the License.\n\n\"\"\"\nNoopService defines a no operational model handler.\n\"\"\"\nimport logging\nimport time\n\n\nclass NoopService(object):\n \"\"\"\n Noop Model handler implementation.\n\n Extend from BaseModelHandler is optional\n \"\"\"\n\n def __init__(self):\n self._context = None\n self.initialized = False\n\n def initialize(self, context):\n \"\"\"\n Initialize model. This will be called during model loading time\n\n :param context: model server context\n :return:\n \"\"\"\n self.initialized = True\n self._context = context\n\n @staticmethod\n def preprocess(data):\n \"\"\"\n Transform raw input into model input data.\n\n :param data: list of objects, raw input from request\n :return: list of model input data\n \"\"\"\n return data\n\n @staticmethod\n def inference(model_input):\n \"\"\"\n Internal inference methods\n\n :param model_input: transformed model input data\n :return: inference results\n \"\"\"\n return model_input\n\n @staticmethod\n def postprocess(model_output):\n return [\"OK\"] * len(model_output)\n\n def handle(self, data, context):\n \"\"\"\n Custom service entry point function.\n\n :param context: model server context\n :param data: list of objects, raw input from request\n :return: list of outputs to be send back to client\n \"\"\"\n # Add your initialization code here\n properties = context.system_properties\n server_name = properties.get(\"server_name\")\n server_version = properties.get(\"server_version\")\n model_dir = properties.get(\"model_dir\")\n gpu_id = properties.get(\"gpu_id\")\n batch_size = properties.get(\"batch_size\")\n\n logging.debug(\"server_name: {}\".format(server_name))\n logging.debug(\"server_version: {}\".format(server_version))\n logging.debug(\"model_dir: {}\".format(model_dir))\n logging.debug(\"gpu_id: {}\".format(gpu_id))\n logging.debug(\"batch_size: {}\".format(batch_size))\n try:\n preprocess_start = time.time()\n data = self.preprocess(data)\n inference_start = time.time()\n data = self.inference(data)\n postprocess_start = time.time()\n data = self.postprocess(data)\n end_time = time.time()\n\n context.set_response_content_type(0, \"text/plain\")\n\n content_type = context.request_processor[0].get_request_property(\"Content-Type\")\n logging.debug(\"content_type: {}\".format(content_type))\n\n metrics = context.metrics\n metrics.add_time(\"PreprocessTime\", round((inference_start - preprocess_start) * 1000, 2))\n metrics.add_time(\"InferenceTime\", round((postprocess_start - inference_start) * 1000, 2))\n metrics.add_time(\"PostprocessTime\", round((end_time - postprocess_start) * 1000, 2))\n return data\n except Exception as e:\n logging.error(e, exc_info=True)\n context.request_processor[0].report_status(500, \"Unknown inference error.\")\n return [\"Error {}\".format(str(e))] * len(data)\n\n\n_service = NoopService()\n\n\ndef handle(data, context):\n if not _service.initialized:\n _service.initialize(context)\n\n if data is None:\n return None\n\n return _service.handle(data, context)\n" }, { "path": "frontend/modelarchive/src/test/resources/models/noop-v1.0-config-tests/MAR-INF/MANIFEST.json", "content": "{\n \"specificationVersion\": \"1.0\",\n \"implementationVersion\": \"1.0\",\n \"description\": \"noop v1.0\",\n \"modelServerVersion\": \"1.0\",\n \"license\": \"Apache 2.0\",\n \"runtime\": \"python\",\n \"model\": {\n \"modelName\": \"noop-config\",\n \"description\": \"no operation model\",\n \"modelVersion\": \"1.0\",\n \"handler\": \"service:handle\"\n },\n \"publisher\": {\n \"author\": \"MXNet SDK team\",\n \"email\": \"noreply@amazon.com\"\n }\n}" }, { "path": "frontend/modelarchive/src/test/resources/models/noop-v1.0-config-tests/service.py", "content": "# Copyright 2018 Amazon.com, Inc. or its affiliates. All Rights Reserved.\n# Licensed under the Apache License, Version 2.0 (the \"License\").\n# You may not use this file except in compliance with the License.\n# A copy of the License is located at\n# http://www.apache.org/licenses/LICENSE-2.0\n# or in the \"license\" file accompanying this file. This file is distributed\n# on an \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either\n# express or implied. See the License for the specific language governing\n# permissions and limitations under the License.\n\n\"\"\"\nNoopService defines a no operational model handler.\n\"\"\"\nimport logging\nimport time\n\n\nclass NoopService(object):\n \"\"\"\n Noop Model handler implementation.\n\n Extend from BaseModelHandler is optional\n \"\"\"\n\n def __init__(self):\n self._context = None\n self.initialized = False\n\n def initialize(self, context):\n \"\"\"\n Initialize model. This will be called during model loading time\n\n :param context: model server context\n :return:\n \"\"\"\n self.initialized = True\n self._context = context\n\n @staticmethod\n def preprocess(data):\n \"\"\"\n Transform raw input into model input data.\n\n :param data: list of objects, raw input from request\n :return: list of model input data\n \"\"\"\n return data\n\n @staticmethod\n def inference(model_input):\n \"\"\"\n Internal inference methods\n\n :param model_input: transformed model input data\n :return: inference results\n \"\"\"\n return model_input\n\n @staticmethod\n def postprocess(model_output):\n return [str(model_output)]\n\n def handle(self, data, context):\n \"\"\"\n Custom service entry point function.\n\n :param context: model server context\n :param data: list of objects, raw input from request\n :return: list of outputs to be send back to client\n \"\"\"\n # Add your initialization code here\n properties = context.system_properties\n server_name = properties.get(\"server_name\")\n server_version = properties.get(\"server_version\")\n model_dir = properties.get(\"model_dir\")\n gpu_id = properties.get(\"gpu_id\")\n batch_size = properties.get(\"batch_size\")\n\n logging.debug(\"server_name: {}\".format(server_name))\n logging.debug(\"server_version: {}\".format(server_version))\n logging.debug(\"model_dir: {}\".format(model_dir))\n logging.debug(\"gpu_id: {}\".format(gpu_id))\n logging.debug(\"batch_size: {}\".format(batch_size))\n try:\n preprocess_start = time.time()\n data = self.preprocess(data)\n inference_start = time.time()\n data = self.inference(data)\n postprocess_start = time.time()\n data = self.postprocess(data)\n end_time = time.time()\n\n context.set_response_content_type(0, \"text/plain\")\n\n content_type = context.get_request_header(0, \"Content-Type\")\n logging.debug(\"content_type: {}\".format(content_type))\n\n metrics = context.metrics\n metrics.add_time(\"PreprocessTime\", round((inference_start - preprocess_start) * 1000, 2))\n metrics.add_time(\"InferenceTime\", round((postprocess_start - inference_start) * 1000, 2))\n metrics.add_time(\"PostprocessTime\", round((end_time - postprocess_start) * 1000, 2))\n return data\n except Exception as e:\n logging.error(e, exc_info=True)\n context.request_processor.report_status(500, \"Unknown inference error.\")\n return [\"Error {}\".format(str(e))] * len(data)\n\n\n_service = NoopService()\n\n\ndef handle(data, context):\n if not _service.initialized:\n _service.initialize(context)\n\n if data is None:\n return None\n\n return _service.handle(data, context)\n" }, { "path": "frontend/modelarchive/src/test/resources/models/prediction-memory-error/MAR-INF/MANIFEST.json", "content": "{\n \"specificationVersion\": \"1.0\",\n \"implementationVersion\": \"1.0\",\n \"description\": \"noop v1.0\",\n \"modelServerVersion\": \"1.0\",\n \"license\": \"Apache 2.0\",\n \"runtime\": \"python\",\n \"model\": {\n \"modelName\": \"pred-mem-err\",\n \"description\": \"Tests for memory error\",\n \"modelVersion\": \"1.0\",\n \"handler\": \"service:handle\"\n },\n \"publisher\": {\n \"author\": \"MXNet SDK team\",\n \"email\": \"noreply@amazon.com\"\n }\n}" }, { "path": "frontend/modelarchive/src/test/resources/models/prediction-memory-error/service.py", "content": "# Copyright 2018 Amazon.com, Inc. or its affiliates. All Rights Reserved.\n# Licensed under the Apache License, Version 2.0 (the \"License\").\n# You may not use this file except in compliance with the License.\n# A copy of the License is located at\n# http://www.apache.org/licenses/LICENSE-2.0\n# or in the \"license\" file accompanying this file. This file is distributed\n# on an \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either\n# express or implied. See the License for the specific language governing\n# permissions and limitations under the License.\n\n\ndef handle(data, ctx):\n # Data is not none in prediction request\n # Python raises MemoryError when the python program goes out of memory. MMS expects this error from the handler\n # if the handlers can not allocate any further memory.\n if data is not None:\n raise MemoryError(\"Some Memory Error\")\n return [\"OK\"]\n" }, { "path": "frontend/modelarchive/src/test/resources/models/respheader-test/MAR-INF/MANIFEST.json", "content": "{\n \"specificationVersion\": \"1.0\",\n \"implementationVersion\": \"1.0\",\n \"description\": \"noop v1.0\",\n \"modelServerVersion\": \"1.0\",\n \"license\": \"Apache 2.0\",\n \"runtime\": \"python\",\n \"model\": {\n \"modelName\": \"respheader\",\n \"description\": \"Tests for response headers\",\n \"modelVersion\": \"1.0\",\n \"handler\": \"service:handle\"\n },\n \"publisher\": {\n \"author\": \"MXNet SDK team\",\n \"email\": \"noreply@amazon.com\"\n }\n}" }, { "path": "frontend/modelarchive/src/test/resources/models/respheader-test/service.py", "content": "# Copyright 2018 Amazon.com, Inc. or its affiliates. All Rights Reserved.\n# Licensed under the Apache License, Version 2.0 (the \"License\").\n# You may not use this file except in compliance with the License.\n# A copy of the License is located at\n# http://www.apache.org/licenses/LICENSE-2.0\n# or in the \"license\" file accompanying this file. This file is distributed\n# on an \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either\n# express or implied. See the License for the specific language governing\n# permissions and limitations under the License.\n\n\"\"\"\nNoopService defines a no operational model handler.\n\"\"\"\nimport logging\nimport time\n\n\nclass NoopService(object):\n \"\"\"\n Noop Model handler implementation.\n\n Extend from BaseModelHandler is optional\n \"\"\"\n\n def __init__(self):\n self._context = None\n self.initialized = False\n\n def initialize(self, context):\n \"\"\"\n Initialize model. This will be called during model loading time\n\n :param context: model server context\n :return:\n \"\"\"\n self.initialized = True\n self._context = context\n\n @staticmethod\n def preprocess(data):\n \"\"\"\n Transform raw input into model input data.\n\n :param data: list of objects, raw input from request\n :return: list of model input data\n \"\"\"\n return data\n\n @staticmethod\n def inference(model_input):\n \"\"\"\n Internal inference methods\n\n :param model_input: transformed model input data\n :return: inference results\n \"\"\"\n return model_input\n\n @staticmethod\n def postprocess(model_output):\n return [str(model_output)]\n\n def handle(self, data, context):\n \"\"\"\n Custom service entry point function.\n\n :param context: model server context\n :param data: list of objects, raw input from request\n :return: list of outputs to be send back to client\n \"\"\"\n # Add your initialization code here\n properties = context.system_properties\n server_name = properties.get(\"server_name\")\n server_version = properties.get(\"server_version\")\n model_dir = properties.get(\"model_dir\")\n gpu_id = properties.get(\"gpu_id\")\n batch_size = properties.get(\"batch_size\")\n\n logging.debug(\"server_name: {}\".format(server_name))\n logging.debug(\"server_version: {}\".format(server_version))\n logging.debug(\"model_dir: {}\".format(model_dir))\n logging.debug(\"gpu_id: {}\".format(gpu_id))\n logging.debug(\"batch_size: {}\".format(batch_size))\n request_processor = context.request_processor\n try:\n data = self.preprocess(data)\n data = self.inference(data)\n data = self.postprocess(data)\n context.set_response_content_type(0, \"text/plain\")\n\n context.set_response_header(0, \"dummy\", \"1\")\n return data\n except Exception as e:\n logging.error(e, exc_info=True)\n request_processor.report_status(500, \"Unknown inference error.\")\n return [\"Error {}\".format(str(e))] * len(data)\n\n\n_service = NoopService()\n\n\ndef handle(data, context):\n if not _service.initialized:\n _service.initialize(context)\n\n if data is None:\n return None\n\n return _service.handle(data, context)\n" }, { "path": "frontend/server/build.gradle", "content": "dependencies {\n compile \"io.netty:netty-all:${netty_version}\"\n compile project(\":modelarchive\")\n compile \"commons-cli:commons-cli:${commons_cli_version}\"\n compile \"software.amazon.ai:mms-plugins-sdk:${mms_server_sdk_version}\"\n compile \"com.lmax:disruptor:${lmax_disruptor_version}\"\n testCompile \"org.testng:testng:${testng_version}\"\n}\n\napply from: file(\"${project.rootProject.projectDir}/tools/gradle/launcher.gradle\")\n\njar {\n manifest {\n attributes 'Main-Class': 'com.amazonaws.ml.mms.ModelServer'\n }\n includeEmptyDirs = false\n from configurations.runtime.collect { it.isDirectory() ? it : zipTree(it) }\n\n exclude \"META-INF/maven/**\"\n exclude \"META-INF/INDEX.LIST\"\n exclude \"META-INF/MANIFEST*\"\n exclude \"META-INF//LICENSE*\"\n exclude \"META-INF//NOTICE*\"\n}\n\ntest {\n doFirst {\n systemProperty \"mmsConfigFile\", 'src/test/resources/config.properties'\n }\n environment \"METRICS_LOCATION\",\"build/logs\"\n environment \"LOG_LOCATION\",\"build/logs\"\n}\n" }, { "path": "frontend/server/src/main/java/com/amazonaws/ml/mms/ModelServer.java", "content": "/*\n * Copyright 2018 Amazon.com, Inc. or its affiliates. All Rights Reserved.\n *\n * Licensed under the Apache License, Version 2.0 (the \"License\"). You may not use this file except in compliance\n * with the License. A copy of the License is located at\n *\n * http://aws.amazon.com/apache2.0/\n *\n * or in the \"license\" file accompanying this file. This file is distributed on an \"AS IS\" BASIS, WITHOUT WARRANTIES\n * OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions\n * and limitations under the License.\n */\npackage com.amazonaws.ml.mms;\n\nimport com.amazonaws.ml.mms.archive.ModelArchive;\nimport com.amazonaws.ml.mms.archive.ModelException;\nimport com.amazonaws.ml.mms.metrics.MetricManager;\nimport com.amazonaws.ml.mms.servingsdk.impl.PluginsManager;\nimport com.amazonaws.ml.mms.util.ConfigManager;\nimport com.amazonaws.ml.mms.util.Connector;\nimport com.amazonaws.ml.mms.util.ConnectorType;\nimport com.amazonaws.ml.mms.util.ServerGroups;\nimport com.amazonaws.ml.mms.wlm.ModelManager;\nimport com.amazonaws.ml.mms.wlm.WorkLoadManager;\nimport io.netty.bootstrap.ServerBootstrap;\nimport io.netty.channel.ChannelFuture;\nimport io.netty.channel.ChannelFutureListener;\nimport io.netty.channel.ChannelOption;\nimport io.netty.channel.EventLoopGroup;\nimport io.netty.channel.FixedRecvByteBufAllocator;\nimport io.netty.channel.ServerChannel;\nimport io.netty.handler.ssl.SslContext;\nimport io.netty.util.internal.logging.InternalLoggerFactory;\nimport io.netty.util.internal.logging.Slf4JLoggerFactory;\nimport java.io.File;\nimport java.io.IOException;\nimport java.lang.annotation.Annotation;\nimport java.security.GeneralSecurityException;\nimport java.util.ArrayList;\nimport java.util.HashMap;\nimport java.util.InvalidPropertiesFormatException;\nimport java.util.List;\nimport java.util.ServiceLoader;\nimport java.util.Set;\nimport java.util.concurrent.ExecutionException;\nimport java.util.concurrent.TimeoutException;\nimport java.util.concurrent.atomic.AtomicBoolean;\nimport org.apache.commons.cli.CommandLine;\nimport org.apache.commons.cli.DefaultParser;\nimport org.apache.commons.cli.HelpFormatter;\nimport org.apache.commons.cli.Options;\nimport org.apache.commons.cli.ParseException;\nimport org.slf4j.Logger;\nimport org.slf4j.LoggerFactory;\nimport software.amazon.ai.mms.servingsdk.ModelServerEndpoint;\nimport software.amazon.ai.mms.servingsdk.annotations.Endpoint;\nimport software.amazon.ai.mms.servingsdk.annotations.helpers.EndpointTypes;\n\npublic class ModelServer {\n\n private Logger logger = LoggerFactory.getLogger(ModelServer.class);\n\n private ServerGroups serverGroups;\n private List futures = new ArrayList<>(2);\n private AtomicBoolean stopped = new AtomicBoolean(false);\n private ConfigManager configManager;\n public static final int MAX_RCVBUF_SIZE = 4096;\n\n /** Creates a new {@code ModelServer} instance. */\n public ModelServer(ConfigManager configManager) {\n this.configManager = configManager;\n serverGroups = new ServerGroups(configManager);\n }\n\n public static void main(String[] args) {\n Options options = ConfigManager.Arguments.getOptions();\n try {\n DefaultParser parser = new DefaultParser();\n CommandLine cmd = parser.parse(options, args, null, false);\n ConfigManager.Arguments arguments = new ConfigManager.Arguments(cmd);\n ConfigManager.init(arguments);\n\n ConfigManager configManager = ConfigManager.getInstance();\n PluginsManager.getInstance().initialize();\n InternalLoggerFactory.setDefaultFactory(Slf4JLoggerFactory.INSTANCE);\n new ModelServer(configManager).startAndWait();\n } catch (IllegalArgumentException e) {\n System.out.println(\"Invalid configuration: \" + e.getMessage()); // NOPMD\n } catch (ParseException e) {\n HelpFormatter formatter = new HelpFormatter();\n formatter.setLeftPadding(1);\n formatter.setWidth(120);\n formatter.printHelp(e.getMessage(), options);\n } catch (Throwable t) {\n t.printStackTrace(); // NOPMD\n } finally {\n System.exit(1); // NOPMD\n }\n }\n\n public void startAndWait() throws InterruptedException, IOException, GeneralSecurityException {\n try {\n List channelFutures = start();\n // Create and schedule metrics manager\n MetricManager.scheduleMetrics(configManager);\n System.out.println(\"Model server started.\"); // NOPMD\n channelFutures.get(0).sync();\n } catch (InvalidPropertiesFormatException e) {\n logger.error(\"Invalid configuration\", e);\n } finally {\n serverGroups.shutdown(true);\n logger.info(\"Model server stopped.\");\n }\n }\n\n private String getDefaultModelName(String name) {\n if (name.contains(\".model\") || name.contains(\".mar\")) {\n return name.substring(name.lastIndexOf('/') + 1, name.lastIndexOf('.'))\n .replaceAll(\"(\\\\W|^_)\", \"_\");\n } else {\n return name.substring(name.lastIndexOf('/') + 1).replaceAll(\"(\\\\W|^_)\", \"_\");\n }\n }\n\n private void initModelStore() {\n WorkLoadManager wlm = new WorkLoadManager(configManager, serverGroups.getBackendGroup());\n ModelManager.init(configManager, wlm);\n Set startupModels = ModelManager.getInstance().getStartupModels();\n String defaultModelName;\n String loadModels = configManager.getLoadModels();\n if (loadModels == null || loadModels.isEmpty()) {\n return;\n }\n\n ModelManager modelManager = ModelManager.getInstance();\n int workers = configManager.getDefaultWorkers();\n if (\"ALL\".equalsIgnoreCase(loadModels)) {\n String modelStore = configManager.getModelStore();\n if (modelStore == null) {\n logger.warn(\"Model store is not configured.\");\n return;\n }\n\n File modelStoreDir = new File(modelStore);\n if (!modelStoreDir.exists()) {\n logger.warn(\"Model store path is not found: {}\", modelStore);\n return;\n }\n\n // Check folders to see if they can be models as well\n File[] files = modelStoreDir.listFiles();\n if (files != null) {\n for (File file : files) {\n if (file.isHidden()) {\n continue;\n }\n String fileName = file.getName();\n if (file.isFile()\n && !fileName.endsWith(\".mar\")\n && !fileName.endsWith(\".model\")) {\n continue;\n }\n try {\n logger.debug(\n \"Loading models from model store: {} preload_model: {}\",\n file.getName(),\n configManager.getPreloadModel());\n defaultModelName = getDefaultModelName(fileName);\n\n ModelArchive archive =\n modelManager.registerModel(\n file.getName(),\n defaultModelName,\n configManager.getPreloadModel());\n modelManager.updateModel(archive.getModelName(), workers, workers);\n startupModels.add(archive.getModelName());\n } catch (ModelException\n | IOException\n | InterruptedException\n | ExecutionException\n | TimeoutException e) {\n logger.warn(\"Failed to load model: \" + file.getAbsolutePath(), e);\n }\n }\n }\n return;\n }\n\n String[] models = loadModels.split(\",\");\n for (String model : models) {\n String[] pair = model.split(\"=\", 2);\n String modelName = null;\n String url;\n if (pair.length == 1) {\n url = pair[0];\n } else {\n modelName = pair[0];\n url = pair[1];\n }\n if (url.isEmpty()) {\n continue;\n }\n\n try {\n logger.info(\n \"Loading initial models: {} preload_model: {}\",\n url,\n configManager.getPreloadModel());\n defaultModelName = getDefaultModelName(url);\n\n ModelArchive archive =\n modelManager.registerModel(\n url,\n modelName,\n null,\n null,\n 1,\n 100,\n configManager.getDefaultResponseTimeoutSeconds(),\n defaultModelName,\n configManager.getPreloadModel());\n modelManager.updateModel(archive.getModelName(), workers, workers);\n startupModels.add(archive.getModelName());\n } catch (ModelException\n | IOException\n | InterruptedException\n | ExecutionException\n | TimeoutException e) {\n logger.warn(\"Failed to load model: \" + url, e);\n }\n }\n }\n\n private void exitModelStore() {\n for (String modelName : ModelManager.getInstance().getModels().keySet()) {\n ModelManager.getInstance().unregisterModel(modelName);\n }\n }\n\n public ChannelFuture initializeServer(\n Connector connector,\n EventLoopGroup serverGroup,\n EventLoopGroup workerGroup,\n ConnectorType type)\n throws InterruptedException, IOException, GeneralSecurityException {\n final String purpose = connector.getPurpose();\n Class channelClass = connector.getServerChannel();\n logger.info(\"Initialize {} server with: {}.\", purpose, channelClass.getSimpleName());\n ServerBootstrap b = new ServerBootstrap();\n b.option(ChannelOption.SO_BACKLOG, 1024)\n .channel(channelClass)\n .childOption(ChannelOption.SO_LINGER, 0)\n .childOption(ChannelOption.SO_REUSEADDR, true)\n .childOption(ChannelOption.SO_KEEPALIVE, true)\n .childOption(\n ChannelOption.RCVBUF_ALLOCATOR,\n new FixedRecvByteBufAllocator(MAX_RCVBUF_SIZE));\n b.group(serverGroup, workerGroup);\n\n SslContext sslCtx = null;\n if (connector.isSsl()) {\n sslCtx = configManager.getSslContext();\n }\n b.childHandler(new ServerInitializer(sslCtx, type));\n\n ChannelFuture future;\n try {\n future = b.bind(connector.getSocketAddress()).sync();\n } catch (Exception e) {\n // https://github.com/netty/netty/issues/2597\n if (e instanceof IOException) {\n throw new IOException(\"Failed to bind to address: \" + connector, e);\n }\n throw e;\n }\n future.addListener(\n (ChannelFutureListener)\n f -> {\n if (!f.isSuccess()) {\n try {\n f.get();\n } catch (InterruptedException | ExecutionException e) {\n logger.error(\"\", e);\n }\n System.exit(-1); // NO PMD\n }\n serverGroups.registerChannel(f.channel());\n });\n\n future.sync();\n\n ChannelFuture f = future.channel().closeFuture();\n f.addListener(\n (ChannelFutureListener)\n listener -> logger.info(\"{} model server stopped.\", purpose));\n\n logger.info(\"{} API bind to: {}\", purpose, connector);\n return f;\n }\n\n /**\n * Main Method that prepares the future for the channel and sets up the ServerBootstrap.\n *\n * @return A ChannelFuture object\n * @throws InterruptedException if interrupted\n */\n public List start()\n throws InterruptedException, IOException, GeneralSecurityException {\n stopped.set(false);\n\n configManager.validateConfigurations();\n\n logger.info(configManager.dumpConfigurations());\n\n initModelStore();\n\n Connector inferenceConnector = configManager.getListener(false);\n Connector managementConnector = configManager.getListener(true);\n\n inferenceConnector.clean();\n managementConnector.clean();\n\n EventLoopGroup serverGroup = serverGroups.getServerGroup();\n EventLoopGroup workerGroup = serverGroups.getChildGroup();\n\n futures.clear();\n\n if (!inferenceConnector.equals(managementConnector)) {\n futures.add(\n initializeServer(\n inferenceConnector,\n serverGroup,\n workerGroup,\n ConnectorType.INFERENCE_CONNECTOR));\n futures.add(\n initializeServer(\n managementConnector,\n serverGroup,\n workerGroup,\n ConnectorType.MANAGEMENT_CONNECTOR));\n } else {\n futures.add(\n initializeServer(\n inferenceConnector, serverGroup, workerGroup, ConnectorType.BOTH));\n }\n\n return futures;\n }\n\n private boolean validEndpoint(Annotation a, EndpointTypes type) {\n return a instanceof Endpoint\n && !((Endpoint) a).urlPattern().isEmpty()\n && ((Endpoint) a).endpointType().equals(type);\n }\n\n private HashMap registerEndpoints(EndpointTypes type) {\n ServiceLoader loader = ServiceLoader.load(ModelServerEndpoint.class);\n HashMap ep = new HashMap<>();\n for (ModelServerEndpoint mep : loader) {\n Class modelServerEndpointClassObj = mep.getClass();\n Annotation[] annotations = modelServerEndpointClassObj.getAnnotations();\n for (Annotation a : annotations) {\n if (validEndpoint(a, type)) {\n ep.put(((Endpoint) a).urlPattern(), mep);\n }\n }\n }\n return ep;\n }\n\n public boolean isRunning() {\n return !stopped.get();\n }\n\n public void stop() {\n if (stopped.get()) {\n return;\n }\n\n stopped.set(true);\n for (ChannelFuture future : futures) {\n future.channel().close();\n }\n serverGroups.shutdown(true);\n serverGroups.init();\n\n exitModelStore();\n }\n}\n" }, { "path": "frontend/server/src/main/java/com/amazonaws/ml/mms/ServerInitializer.java", "content": "/*\n * Copyright 2018 Amazon.com, Inc. or its affiliates. All Rights Reserved.\n *\n * Licensed under the Apache License, Version 2.0 (the \"License\"). You may not use this file except in compliance\n * with the License. A copy of the License is located at\n *\n * http://aws.amazon.com/apache2.0/\n *\n * or in the \"license\" file accompanying this file. This file is distributed on an \"AS IS\" BASIS, WITHOUT WARRANTIES\n * OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions\n * and limitations under the License.\n */\npackage com.amazonaws.ml.mms;\n\nimport com.amazonaws.ml.mms.http.ApiDescriptionRequestHandler;\nimport com.amazonaws.ml.mms.http.HttpRequestHandler;\nimport com.amazonaws.ml.mms.http.HttpRequestHandlerChain;\nimport com.amazonaws.ml.mms.http.InferenceRequestHandler;\nimport com.amazonaws.ml.mms.http.InvalidRequestHandler;\nimport com.amazonaws.ml.mms.http.ManagementRequestHandler;\nimport com.amazonaws.ml.mms.servingsdk.impl.PluginsManager;\nimport com.amazonaws.ml.mms.util.ConfigManager;\nimport com.amazonaws.ml.mms.util.ConnectorType;\nimport io.netty.channel.Channel;\nimport io.netty.channel.ChannelInitializer;\nimport io.netty.channel.ChannelPipeline;\nimport io.netty.handler.codec.http.HttpObjectAggregator;\nimport io.netty.handler.codec.http.HttpServerCodec;\nimport io.netty.handler.ssl.SslContext;\n\n/**\n * A special {@link io.netty.channel.ChannelInboundHandler} which offers an easy way to initialize a\n * {@link io.netty.channel.Channel} once it was registered to its {@link\n * io.netty.channel.EventLoop}.\n */\npublic class ServerInitializer extends ChannelInitializer {\n\n private ConnectorType connectorType;\n private SslContext sslCtx;\n\n /**\n * Creates a new {@code HttpRequestHandler} instance.\n *\n * @param sslCtx null if SSL is not enabled\n * @param type true to initialize a management server instead of an API Server\n */\n public ServerInitializer(SslContext sslCtx, ConnectorType type) {\n this.sslCtx = sslCtx;\n this.connectorType = type;\n }\n\n /** {@inheritDoc} */\n @Override\n public void initChannel(Channel ch) {\n ChannelPipeline pipeline = ch.pipeline();\n HttpRequestHandlerChain apiDescriptionRequestHandler =\n new ApiDescriptionRequestHandler(connectorType);\n HttpRequestHandlerChain invalidRequestHandler = new InvalidRequestHandler();\n\n int maxRequestSize = ConfigManager.getInstance().getMaxRequestSize();\n if (sslCtx != null) {\n pipeline.addLast(\"ssl\", sslCtx.newHandler(ch.alloc()));\n }\n pipeline.addLast(\"http\", new HttpServerCodec());\n pipeline.addLast(\"aggregator\", new HttpObjectAggregator(maxRequestSize));\n\n HttpRequestHandlerChain httpRequestHandlerChain = apiDescriptionRequestHandler;\n if (ConnectorType.BOTH.equals(connectorType)\n || ConnectorType.INFERENCE_CONNECTOR.equals(connectorType)) {\n httpRequestHandlerChain =\n httpRequestHandlerChain.setNextHandler(\n new InferenceRequestHandler(\n PluginsManager.getInstance().getInferenceEndpoints()));\n }\n if (ConnectorType.BOTH.equals(connectorType)\n || ConnectorType.MANAGEMENT_CONNECTOR.equals(connectorType)) {\n httpRequestHandlerChain =\n httpRequestHandlerChain.setNextHandler(\n new ManagementRequestHandler(\n PluginsManager.getInstance().getManagementEndpoints()));\n }\n httpRequestHandlerChain.setNextHandler(invalidRequestHandler);\n pipeline.addLast(\"handler\", new HttpRequestHandler(apiDescriptionRequestHandler));\n }\n}\n" }, { "path": "frontend/server/src/main/java/com/amazonaws/ml/mms/http/ApiDescriptionRequestHandler.java", "content": "package com.amazonaws.ml.mms.http;\n\nimport com.amazonaws.ml.mms.archive.ModelException;\nimport com.amazonaws.ml.mms.openapi.OpenApiUtils;\nimport com.amazonaws.ml.mms.util.ConnectorType;\nimport com.amazonaws.ml.mms.util.NettyUtils;\nimport io.netty.channel.ChannelHandlerContext;\nimport io.netty.handler.codec.http.FullHttpRequest;\nimport io.netty.handler.codec.http.HttpMethod;\nimport io.netty.handler.codec.http.QueryStringDecoder;\n\npublic class ApiDescriptionRequestHandler extends HttpRequestHandlerChain {\n\n private ConnectorType connectorType;\n\n public ApiDescriptionRequestHandler(ConnectorType type) {\n connectorType = type;\n }\n\n @Override\n protected void handleRequest(\n ChannelHandlerContext ctx,\n FullHttpRequest req,\n QueryStringDecoder decoder,\n String[] segments)\n throws ModelException {\n\n if (isApiDescription(segments)) {\n String path = decoder.path();\n if ((\"/\".equals(path) && HttpMethod.OPTIONS.equals(req.method()))\n || (segments.length == 2 && segments[1].equals(\"api-description\"))) {\n handleApiDescription(ctx);\n return;\n }\n throw new MethodNotAllowedException();\n } else {\n chain.handleRequest(ctx, req, decoder, segments);\n }\n }\n\n private boolean isApiDescription(String[] segments) {\n return segments.length == 0 || segments[1].equals(\"api-description\");\n }\n\n private void handleApiDescription(ChannelHandlerContext ctx) {\n NettyUtils.sendJsonResponse(ctx, OpenApiUtils.listApis(connectorType));\n }\n}\n" }, { "path": "frontend/server/src/main/java/com/amazonaws/ml/mms/http/BadRequestException.java", "content": "/*\n * Copyright 2018 Amazon.com, Inc. or its affiliates. All Rights Reserved.\n *\n * Licensed under the Apache License, Version 2.0 (the \"License\"). You may not use this file except in compliance\n * with the License. A copy of the License is located at\n *\n * http://aws.amazon.com/apache2.0/\n *\n * or in the \"license\" file accompanying this file. This file is distributed on an \"AS IS\" BASIS, WITHOUT WARRANTIES\n * OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions\n * and limitations under the License.\n */\npackage com.amazonaws.ml.mms.http;\n\npublic class BadRequestException extends IllegalArgumentException {\n\n private static final long serialVersionUID = 1L;\n\n /**\n * Constructs an {@code BadRequestException} with the specified detail message.\n *\n * @param message The detail message (which is saved for later retrieval by the {@link\n * #getMessage()} method)\n */\n public BadRequestException(String message) {\n super(message);\n }\n\n /**\n * Constructs an {@code BadRequestException} with the specified detail message and cause.\n *\n *

Note that the detail message associated with {@code cause} is not automatically\n * incorporated into this exception's detail message.\n *\n * @param cause The cause (which is saved for later retrieval by the {@link #getCause()}\n * method). (A null value is permitted, and indicates that the cause is nonexistent or\n * unknown.)\n */\n public BadRequestException(Throwable cause) {\n super(cause.getMessage(), cause);\n }\n}\n" }, { "path": "frontend/server/src/main/java/com/amazonaws/ml/mms/http/ConflictStatusException.java", "content": "/*\n * Copyright 2019 Amazon.com, Inc. or its affiliates. All Rights Reserved.\n *\n * Licensed under the Apache License, Version 2.0 (the \"License\"). You may not use this file except in compliance\n * with the License. A copy of the License is located at\n *\n * http://aws.amazon.com/apache2.0/\n *\n * or in the \"license\" file accompanying this file. This file is distributed on an \"AS IS\" BASIS, WITHOUT WARRANTIES\n * OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions\n * and limitations under the License.\n */\npackage com.amazonaws.ml.mms.http;\n\npublic class ConflictStatusException extends IllegalArgumentException {\n\n private static final long serialVersionUID = 1L;\n\n /**\n * Constructs an {@code ConflictStatusException} with the specified detail message.\n *\n * @param message The detail message (which is saved for later retrieval by the {@link\n * #getMessage()} method)\n */\n public ConflictStatusException(String message) {\n super(message);\n }\n\n /**\n * Constructs an {@code ConflictStatusException} with the specified detail message and cause.\n *\n *

Note that the detail message associated with {@code cause} is not automatically\n * incorporated into this exception's detail message.\n *\n * @param cause The cause (which is saved for later retrieval by the {@link #getCause()}\n * method). (A null value is permitted, and indicates that the cause is nonexistent or\n * unknown.)\n */\n public ConflictStatusException(Throwable cause) {\n super(cause.getMessage(), cause);\n }\n}\n" }, { "path": "frontend/server/src/main/java/com/amazonaws/ml/mms/http/DescribeModelResponse.java", "content": "/*\n * Copyright 2018 Amazon.com, Inc. or its affiliates. All Rights Reserved.\n *\n * Licensed under the Apache License, Version 2.0 (the \"License\"). You may not use this file except in compliance\n * with the License. A copy of the License is located at\n *\n * http://aws.amazon.com/apache2.0/\n *\n * or in the \"license\" file accompanying this file. This file is distributed on an \"AS IS\" BASIS, WITHOUT WARRANTIES\n * OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions\n * and limitations under the License.\n */\npackage com.amazonaws.ml.mms.http;\n\nimport java.util.ArrayList;\nimport java.util.Date;\nimport java.util.List;\n\npublic class DescribeModelResponse {\n\n private String modelName;\n private String modelVersion;\n private String modelUrl;\n private String engine;\n private String runtime;\n private int minWorkers;\n private int maxWorkers;\n private int batchSize;\n private int maxBatchDelay;\n private String status;\n private boolean loadedAtStartup;\n\n private List workers;\n private Metrics metrics;\n\n public DescribeModelResponse() {\n workers = new ArrayList<>();\n }\n\n public String getModelName() {\n return modelName;\n }\n\n public void setModelName(String modelName) {\n this.modelName = modelName;\n }\n\n public boolean getLoadedAtStartup() {\n return loadedAtStartup;\n }\n\n public void setLoadedAtStartup(boolean loadedAtStartup) {\n this.loadedAtStartup = loadedAtStartup;\n }\n\n public String getModelVersion() {\n return modelVersion;\n }\n\n public void setModelVersion(String modelVersion) {\n this.modelVersion = modelVersion;\n }\n\n public String getModelUrl() {\n return modelUrl;\n }\n\n public void setModelUrl(String modelUrl) {\n this.modelUrl = modelUrl;\n }\n\n public String getEngine() {\n return engine;\n }\n\n public void setEngine(String engine) {\n this.engine = engine;\n }\n\n public String getRuntime() {\n return runtime;\n }\n\n public void setRuntime(String runtime) {\n this.runtime = runtime;\n }\n\n public int getMinWorkers() {\n return minWorkers;\n }\n\n public void setMinWorkers(int minWorkers) {\n this.minWorkers = minWorkers;\n }\n\n public int getMaxWorkers() {\n return maxWorkers;\n }\n\n public void setMaxWorkers(int maxWorkers) {\n this.maxWorkers = maxWorkers;\n }\n\n public int getBatchSize() {\n return batchSize;\n }\n\n public void setBatchSize(int batchSize) {\n this.batchSize = batchSize;\n }\n\n public int getMaxBatchDelay() {\n return maxBatchDelay;\n }\n\n public void setMaxBatchDelay(int maxBatchDelay) {\n this.maxBatchDelay = maxBatchDelay;\n }\n\n public String getStatus() {\n return status;\n }\n\n public void setStatus(String status) {\n this.status = status;\n }\n\n public List getWorkers() {\n return workers;\n }\n\n public void setWorkers(List workers) {\n this.workers = workers;\n }\n\n public void addWorker(\n String id, long startTime, boolean isRunning, int gpuId, long memoryUsage) {\n Worker worker = new Worker();\n worker.setId(id);\n worker.setStartTime(new Date(startTime));\n worker.setStatus(isRunning ? \"READY\" : \"UNLOADING\");\n worker.setGpu(gpuId >= 0);\n worker.setMemoryUsage(memoryUsage);\n workers.add(worker);\n }\n\n public Metrics getMetrics() {\n return metrics;\n }\n\n public void setMetrics(Metrics metrics) {\n this.metrics = metrics;\n }\n\n public static final class Worker {\n\n private String id;\n private Date startTime;\n private String status;\n private boolean gpu;\n private long memoryUsage;\n\n public Worker() {}\n\n public String getId() {\n return id;\n }\n\n public void setId(String id) {\n this.id = id;\n }\n\n public Date getStartTime() {\n return startTime;\n }\n\n public void setStartTime(Date startTime) {\n this.startTime = startTime;\n }\n\n public String getStatus() {\n return status;\n }\n\n public void setStatus(String status) {\n this.status = status;\n }\n\n public boolean isGpu() {\n return gpu;\n }\n\n public void setGpu(boolean gpu) {\n this.gpu = gpu;\n }\n\n public long getMemoryUsage() {\n return memoryUsage;\n }\n\n public void setMemoryUsage(long memoryUsage) {\n this.memoryUsage = memoryUsage;\n }\n }\n\n public static final class Metrics {\n\n private int rejectedRequests;\n private int waitingQueueSize;\n private int requests;\n\n public int getRejectedRequests() {\n return rejectedRequests;\n }\n\n public void setRejectedRequests(int rejectedRequests) {\n this.rejectedRequests = rejectedRequests;\n }\n\n public int getWaitingQueueSize() {\n return waitingQueueSize;\n }\n\n public void setWaitingQueueSize(int waitingQueueSize) {\n this.waitingQueueSize = waitingQueueSize;\n }\n\n public int getRequests() {\n return requests;\n }\n\n public void setRequests(int requests) {\n this.requests = requests;\n }\n }\n}\n" }, { "path": "frontend/server/src/main/java/com/amazonaws/ml/mms/http/ErrorResponse.java", "content": "/*\n * Copyright 2018 Amazon.com, Inc. or its affiliates. All Rights Reserved.\n *\n * Licensed under the Apache License, Version 2.0 (the \"License\"). You may not use this file except in compliance\n * with the License. A copy of the License is located at\n *\n * http://aws.amazon.com/apache2.0/\n *\n * or in the \"license\" file accompanying this file. This file is distributed on an \"AS IS\" BASIS, WITHOUT WARRANTIES\n * OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions\n * and limitations under the License.\n */\npackage com.amazonaws.ml.mms.http;\n\npublic class ErrorResponse {\n\n private int code;\n private String type;\n private String message;\n\n public ErrorResponse() {}\n\n public ErrorResponse(int code, String message) {\n this.code = code;\n this.message = message;\n }\n\n public ErrorResponse(int code, String type, String message) {\n this.code = code;\n this.type = type;\n this.message = message;\n }\n\n public int getCode() {\n return code;\n }\n\n public String getType() {\n return type;\n }\n\n public String getMessage() {\n return message;\n }\n}\n" }, { "path": "frontend/server/src/main/java/com/amazonaws/ml/mms/http/HttpRequestHandler.java", "content": "/*\n * Copyright 2018 Amazon.com, Inc. or its affiliates. All Rights Reserved.\n *\n * Licensed under the Apache License, Version 2.0 (the \"License\"). You may not use this file except in compliance\n * with the License. A copy of the License is located at\n *\n * http://aws.amazon.com/apache2.0/\n *\n * or in the \"license\" file accompanying this file. This file is distributed on an \"AS IS\" BASIS, WITHOUT WARRANTIES\n * OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions\n * and limitations under the License.\n */\npackage com.amazonaws.ml.mms.http;\n\nimport com.amazonaws.ml.mms.archive.ModelException;\nimport com.amazonaws.ml.mms.archive.ModelNotFoundException;\nimport com.amazonaws.ml.mms.util.NettyUtils;\nimport io.netty.channel.ChannelHandlerContext;\nimport io.netty.channel.SimpleChannelInboundHandler;\nimport io.netty.handler.codec.http.FullHttpRequest;\nimport io.netty.handler.codec.http.HttpResponseStatus;\nimport io.netty.handler.codec.http.QueryStringDecoder;\nimport org.slf4j.Logger;\nimport org.slf4j.LoggerFactory;\n\n/**\n * A class handling inbound HTTP requests.\n *\n *

This class\n */\npublic class HttpRequestHandler extends SimpleChannelInboundHandler {\n\n private static final Logger logger = LoggerFactory.getLogger(HttpRequestHandler.class);\n HttpRequestHandlerChain handlerChain;\n /** Creates a new {@code HttpRequestHandler} instance. */\n public HttpRequestHandler() {}\n\n public HttpRequestHandler(HttpRequestHandlerChain chain) {\n handlerChain = chain;\n }\n\n /** {@inheritDoc} */\n @Override\n protected void channelRead0(ChannelHandlerContext ctx, FullHttpRequest req) {\n try {\n NettyUtils.requestReceived(ctx.channel(), req);\n if (!req.decoderResult().isSuccess()) {\n throw new BadRequestException(\"Invalid HTTP message.\");\n }\n QueryStringDecoder decoder = new QueryStringDecoder(req.uri());\n String path = decoder.path();\n\n String[] segments = path.split(\"/\");\n handlerChain.handleRequest(ctx, req, decoder, segments);\n } catch (ResourceNotFoundException | ModelNotFoundException e) {\n logger.trace(\"\", e);\n NettyUtils.sendError(ctx, HttpResponseStatus.NOT_FOUND, e);\n } catch (BadRequestException | ModelException e) {\n logger.trace(\"\", e);\n NettyUtils.sendError(ctx, HttpResponseStatus.BAD_REQUEST, e);\n } catch (ConflictStatusException e) {\n logger.trace(\"\", e);\n NettyUtils.sendError(ctx, HttpResponseStatus.CONFLICT, e);\n } catch (RequestTimeoutException e) {\n logger.trace(\"\", e);\n NettyUtils.sendError(ctx, HttpResponseStatus.REQUEST_TIMEOUT, e);\n } catch (MethodNotAllowedException e) {\n logger.trace(\"\", e);\n NettyUtils.sendError(ctx, HttpResponseStatus.METHOD_NOT_ALLOWED, e);\n } catch (ServiceUnavailableException e) {\n logger.trace(\"\", e);\n NettyUtils.sendError(ctx, HttpResponseStatus.SERVICE_UNAVAILABLE, e);\n } catch (OutOfMemoryError e) {\n logger.trace(\"\", e);\n NettyUtils.sendError(ctx, HttpResponseStatus.INSUFFICIENT_STORAGE, e);\n } catch (Throwable t) {\n logger.error(\"\", t);\n NettyUtils.sendError(ctx, HttpResponseStatus.INTERNAL_SERVER_ERROR, t);\n }\n }\n\n /** {@inheritDoc} */\n @Override\n public void exceptionCaught(ChannelHandlerContext ctx, Throwable cause) {\n logger.error(\"\", cause);\n if (cause instanceof OutOfMemoryError) {\n NettyUtils.sendError(ctx, HttpResponseStatus.INSUFFICIENT_STORAGE, cause);\n }\n ctx.close();\n }\n}\n" }, { "path": "frontend/server/src/main/java/com/amazonaws/ml/mms/http/HttpRequestHandlerChain.java", "content": "package com.amazonaws.ml.mms.http;\n\nimport com.amazonaws.ml.mms.archive.ModelException;\nimport com.amazonaws.ml.mms.archive.ModelNotFoundException;\nimport com.amazonaws.ml.mms.servingsdk.impl.ModelServerContext;\nimport com.amazonaws.ml.mms.servingsdk.impl.ModelServerRequest;\nimport com.amazonaws.ml.mms.servingsdk.impl.ModelServerResponse;\nimport com.amazonaws.ml.mms.util.NettyUtils;\nimport com.amazonaws.ml.mms.wlm.ModelManager;\nimport io.netty.buffer.Unpooled;\nimport io.netty.channel.ChannelHandlerContext;\nimport io.netty.handler.codec.http.DefaultFullHttpResponse;\nimport io.netty.handler.codec.http.DefaultHttpHeadersFactory;\nimport io.netty.handler.codec.http.FullHttpRequest;\nimport io.netty.handler.codec.http.FullHttpResponse;\nimport io.netty.handler.codec.http.HttpResponseStatus;\nimport io.netty.handler.codec.http.HttpVersion;\nimport io.netty.handler.codec.http.QueryStringDecoder;\nimport java.io.IOException;\nimport java.util.Map;\nimport org.slf4j.Logger;\nimport org.slf4j.LoggerFactory;\nimport software.amazon.ai.mms.servingsdk.ModelServerEndpoint;\nimport software.amazon.ai.mms.servingsdk.ModelServerEndpointException;\n\npublic abstract class HttpRequestHandlerChain {\n private static final Logger logger = LoggerFactory.getLogger(HttpRequestHandler.class);\n protected Map endpointMap;\n protected HttpRequestHandlerChain chain;\n\n public HttpRequestHandlerChain() {}\n\n public HttpRequestHandlerChain(Map map) {\n endpointMap = map;\n }\n\n public HttpRequestHandlerChain setNextHandler(HttpRequestHandlerChain nextHandler) {\n chain = nextHandler;\n return chain;\n }\n\n protected abstract void handleRequest(\n ChannelHandlerContext ctx,\n FullHttpRequest req,\n QueryStringDecoder decoder,\n String[] segments)\n throws ModelNotFoundException, ModelException;\n\n private void run(\n ModelServerEndpoint endpoint,\n FullHttpRequest req,\n FullHttpResponse rsp,\n QueryStringDecoder decoder,\n String method)\n throws IOException {\n switch (method) {\n case \"GET\":\n endpoint.doGet(\n new ModelServerRequest(req, decoder),\n new ModelServerResponse(rsp),\n new ModelServerContext());\n break;\n case \"PUT\":\n endpoint.doPut(\n new ModelServerRequest(req, decoder),\n new ModelServerResponse(rsp),\n new ModelServerContext());\n break;\n case \"DELETE\":\n endpoint.doDelete(\n new ModelServerRequest(req, decoder),\n new ModelServerResponse(rsp),\n new ModelServerContext());\n break;\n case \"POST\":\n endpoint.doPost(\n new ModelServerRequest(req, decoder),\n new ModelServerResponse(rsp),\n new ModelServerContext());\n break;\n default:\n throw new ServiceUnavailableException(\"Invalid HTTP method received\");\n }\n }\n\n protected void handleCustomEndpoint(\n ChannelHandlerContext ctx,\n FullHttpRequest req,\n String[] segments,\n QueryStringDecoder decoder) {\n ModelServerEndpoint endpoint = endpointMap.get(segments[1]);\n Runnable r =\n () -> {\n Long start = System.currentTimeMillis();\n FullHttpResponse rsp =\n new DefaultFullHttpResponse(\n HttpVersion.HTTP_1_1,\n HttpResponseStatus.OK,\n Unpooled.directBuffer(),\n DefaultHttpHeadersFactory.headersFactory(),\n DefaultHttpHeadersFactory.trailersFactory());\n try {\n run(endpoint, req, rsp, decoder, req.method().toString());\n NettyUtils.sendHttpResponse(ctx, rsp, true);\n logger.info(\n \"Running \\\"{}\\\" endpoint took {} ms\",\n segments[0],\n System.currentTimeMillis() - start);\n } catch (ModelServerEndpointException me) {\n NettyUtils.sendError(ctx, HttpResponseStatus.INTERNAL_SERVER_ERROR, me);\n logger.error(\"Error thrown by the model endpoint plugin.\", me);\n } catch (OutOfMemoryError oom) {\n NettyUtils.sendError(\n ctx, HttpResponseStatus.INSUFFICIENT_STORAGE, oom, \"Out of memory\");\n } catch (IOException ioe) {\n NettyUtils.sendError(\n ctx,\n HttpResponseStatus.INTERNAL_SERVER_ERROR,\n ioe,\n \"I/O error while running the custom endpoint\");\n logger.error(\"I/O error while running the custom endpoint.\", ioe);\n } catch (Throwable e) {\n NettyUtils.sendError(\n ctx,\n HttpResponseStatus.INTERNAL_SERVER_ERROR,\n e,\n \"Unknown exception\");\n logger.error(\"Unknown exception\", e);\n }\n };\n ModelManager.getInstance().submitTask(r);\n }\n}\n" }, { "path": "frontend/server/src/main/java/com/amazonaws/ml/mms/http/InferenceRequestHandler.java", "content": "/*\n * Copyright 2018 Amazon.com, Inc. or its affiliates. All Rights Reserved.\n *\n * Licensed under the Apache License, Version 2.0 (the \"License\"). You may not use this file except in compliance\n * with the License. A copy of the License is located at\n *\n * http://aws.amazon.com/apache2.0/\n *\n * or in the \"license\" file accompanying this file. This file is distributed on an \"AS IS\" BASIS, WITHOUT WARRANTIES\n * OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions\n * and limitations under the License.\n */\npackage com.amazonaws.ml.mms.http;\n\nimport com.amazonaws.ml.mms.archive.ModelException;\nimport com.amazonaws.ml.mms.archive.ModelNotFoundException;\nimport com.amazonaws.ml.mms.openapi.OpenApiUtils;\nimport com.amazonaws.ml.mms.util.NettyUtils;\nimport com.amazonaws.ml.mms.util.messages.InputParameter;\nimport com.amazonaws.ml.mms.util.messages.RequestInput;\nimport com.amazonaws.ml.mms.util.messages.WorkerCommands;\nimport com.amazonaws.ml.mms.wlm.Job;\nimport com.amazonaws.ml.mms.wlm.Model;\nimport com.amazonaws.ml.mms.wlm.ModelManager;\nimport io.netty.channel.ChannelHandlerContext;\nimport io.netty.handler.codec.http.FullHttpRequest;\nimport io.netty.handler.codec.http.HttpHeaderValues;\nimport io.netty.handler.codec.http.HttpMethod;\nimport io.netty.handler.codec.http.HttpUtil;\nimport io.netty.handler.codec.http.QueryStringDecoder;\nimport io.netty.handler.codec.http.multipart.DefaultHttpDataFactory;\nimport io.netty.handler.codec.http.multipart.HttpDataFactory;\nimport io.netty.handler.codec.http.multipart.HttpPostRequestDecoder;\nimport java.util.List;\nimport java.util.Map;\nimport org.slf4j.Logger;\nimport org.slf4j.LoggerFactory;\nimport software.amazon.ai.mms.servingsdk.ModelServerEndpoint;\n\n/**\n * A class handling inbound HTTP requests to the management API.\n *\n *

This class\n */\npublic class InferenceRequestHandler extends HttpRequestHandlerChain {\n\n private static final Logger logger = LoggerFactory.getLogger(InferenceRequestHandler.class);\n\n /** Creates a new {@code InferenceRequestHandler} instance. */\n public InferenceRequestHandler(Map ep) {\n endpointMap = ep;\n }\n\n @Override\n protected void handleRequest(\n ChannelHandlerContext ctx,\n FullHttpRequest req,\n QueryStringDecoder decoder,\n String[] segments)\n throws ModelException {\n if (isInferenceReq(segments)) {\n if (endpointMap.getOrDefault(segments[1], null) != null) {\n handleCustomEndpoint(ctx, req, segments, decoder);\n } else {\n switch (segments[1]) {\n case \"ping\":\n ModelManager.getInstance().workerStatus(ctx);\n break;\n case \"models\":\n case \"invocations\":\n validatePredictionsEndpoint(segments);\n handleInvocations(ctx, req, decoder, segments);\n break;\n case \"predictions\":\n handlePredictions(ctx, req, segments);\n break;\n default:\n handleLegacyPredict(ctx, req, decoder, segments);\n break;\n }\n }\n } else {\n chain.handleRequest(ctx, req, decoder, segments);\n }\n }\n\n private boolean isInferenceReq(String[] segments) {\n return segments.length == 0\n || segments[1].equals(\"ping\")\n || (segments.length == 4 && segments[1].equals(\"models\"))\n || segments[1].equals(\"predictions\")\n || segments[1].equals(\"api-description\")\n || segments[1].equals(\"invocations\")\n || (segments.length == 3 && segments[2].equals(\"predict\"))\n || endpointMap.containsKey(segments[1]);\n }\n\n private void validatePredictionsEndpoint(String[] segments) {\n if (segments.length == 2 && \"invocations\".equals(segments[1])) {\n return;\n } else if (segments.length == 4\n && \"models\".equals(segments[1])\n && \"invoke\".equals(segments[3])) {\n return;\n }\n\n throw new ResourceNotFoundException();\n }\n\n private void handlePredictions(\n ChannelHandlerContext ctx, FullHttpRequest req, String[] segments)\n throws ModelNotFoundException {\n if (segments.length < 3) {\n throw new ResourceNotFoundException();\n }\n predict(ctx, req, null, segments[2]);\n }\n\n private void handleInvocations(\n ChannelHandlerContext ctx,\n FullHttpRequest req,\n QueryStringDecoder decoder,\n String[] segments)\n throws ModelNotFoundException {\n String modelName =\n (\"invocations\".equals(segments[1]))\n ? NettyUtils.getParameter(decoder, \"model_name\", null)\n : segments[2];\n if (modelName == null || modelName.isEmpty()) {\n if (ModelManager.getInstance().getStartupModels().size() == 1) {\n modelName = ModelManager.getInstance().getStartupModels().iterator().next();\n }\n }\n predict(ctx, req, decoder, modelName);\n }\n\n private void handleLegacyPredict(\n ChannelHandlerContext ctx,\n FullHttpRequest req,\n QueryStringDecoder decoder,\n String[] segments)\n throws ModelNotFoundException {\n if (segments.length < 3 || !\"predict\".equals(segments[2])) {\n throw new ResourceNotFoundException();\n }\n\n predict(ctx, req, decoder, segments[1]);\n }\n\n private void predict(\n ChannelHandlerContext ctx,\n FullHttpRequest req,\n QueryStringDecoder decoder,\n String modelName)\n throws ModelNotFoundException, BadRequestException {\n RequestInput input = parseRequest(ctx, req, decoder);\n if (modelName == null) {\n throw new BadRequestException(\"Parameter model_name is required.\");\n }\n\n if (HttpMethod.OPTIONS.equals(req.method())) {\n ModelManager modelManager = ModelManager.getInstance();\n Model model = modelManager.getModels().get(modelName);\n if (model == null) {\n throw new ModelNotFoundException(\"Model not found: \" + modelName);\n }\n\n String resp = OpenApiUtils.getModelApi(model);\n NettyUtils.sendJsonResponse(ctx, resp);\n return;\n }\n\n Job job = new Job(ctx, modelName, WorkerCommands.PREDICT, input);\n if (!ModelManager.getInstance().addJob(job)) {\n throw new ServiceUnavailableException(\n \"No worker is available to serve request for model: \"\n + modelName\n + \". Consider increasing job queue size.\");\n }\n }\n\n private static RequestInput parseRequest(\n ChannelHandlerContext ctx, FullHttpRequest req, QueryStringDecoder decoder) {\n String requestId = NettyUtils.getRequestId(ctx.channel());\n RequestInput inputData = new RequestInput(requestId);\n if (decoder != null) {\n for (Map.Entry> entry : decoder.parameters().entrySet()) {\n String key = entry.getKey();\n for (String value : entry.getValue()) {\n inputData.addParameter(new InputParameter(key, value));\n }\n }\n }\n\n CharSequence contentType = HttpUtil.getMimeType(req);\n for (Map.Entry entry : req.headers().entries()) {\n inputData.updateHeaders(entry.getKey(), entry.getValue());\n }\n\n if (HttpPostRequestDecoder.isMultipart(req)\n || HttpHeaderValues.APPLICATION_X_WWW_FORM_URLENCODED.contentEqualsIgnoreCase(\n contentType)) {\n HttpDataFactory factory = new DefaultHttpDataFactory(6553500);\n HttpPostRequestDecoder form = new HttpPostRequestDecoder(factory, req);\n try {\n while (form.hasNext()) {\n inputData.addParameter(NettyUtils.getFormData(form.next()));\n }\n } catch (HttpPostRequestDecoder.EndOfDataDecoderException ignore) {\n logger.trace(\"End of multipart items.\");\n } finally {\n form.cleanFiles();\n form.destroy();\n }\n } else {\n byte[] content = NettyUtils.getBytes(req.content());\n inputData.addParameter(new InputParameter(\"body\", content, contentType));\n }\n return inputData;\n }\n}\n" }, { "path": "frontend/server/src/main/java/com/amazonaws/ml/mms/http/InternalServerException.java", "content": "/*\n * Copyright 2018 Amazon.com, Inc. or its affiliates. All Rights Reserved.\n *\n * Licensed under the Apache License, Version 2.0 (the \"License\"). You may not use this file except in compliance\n * with the License. A copy of the License is located at\n *\n * http://aws.amazon.com/apache2.0/\n *\n * or in the \"license\" file accompanying this file. This file is distributed on an \"AS IS\" BASIS, WITHOUT WARRANTIES\n * OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions\n * and limitations under the License.\n */\npackage com.amazonaws.ml.mms.http;\n\npublic class InternalServerException extends RuntimeException {\n\n static final long serialVersionUID = 1L;\n\n /**\n * Constructs an {@code InternalServerException} with the specified detail message.\n *\n * @param message The detail message (which is saved for later retrieval by the {@link\n * #getMessage()} method)\n */\n public InternalServerException(String message) {\n super(message);\n }\n\n /**\n * Constructs an {@code BadRequestException} with the specified detail message and cause.\n *\n *

Note that the detail message associated with {@code cause} is not automatically\n * incorporated into this exception's detail message.\n *\n * @param message The detail message (which is saved for later retrieval by the {@link\n * #getMessage()} method)\n * @param cause The cause (which is saved for later retrieval by the {@link #getCause()}\n * method). (A null value is permitted, and indicates that the cause is nonexistent or\n * unknown.)\n */\n public InternalServerException(String message, Throwable cause) {\n super(message, cause);\n }\n}\n" }, { "path": "frontend/server/src/main/java/com/amazonaws/ml/mms/http/InvalidPluginException.java", "content": "/*\n * Copyright 2019 Amazon.com, Inc. or its affiliates. All Rights Reserved.\n *\n * Licensed under the Apache License, Version 2.0 (the \"License\"). You may not use this file except in compliance\n * with the License. A copy of the License is located at\n *\n * http://aws.amazon.com/apache2.0/\n *\n * or in the \"license\" file accompanying this file. This file is distributed on an \"AS IS\" BASIS, WITHOUT WARRANTIES\n * OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions\n * and limitations under the License.\n */\npackage com.amazonaws.ml.mms.http;\n\n/** InvaliPluginException is thrown when there is an error while handling a Model Server plugin */\npublic class InvalidPluginException extends RuntimeException {\n\n static final long serialVersionUID = 1L;\n /**\n * Constructs an {@code InvalidPluginException} with {@code null} as its error detail message.\n */\n public InvalidPluginException() {\n super(\"Registered plugin is invalid. Please re-check the configuration and the plugins.\");\n }\n\n /**\n * Constructs an {@code InvalidPluginException} with {@code msg} as its error detail message\n *\n * @param msg : This is the error detail message\n */\n public InvalidPluginException(String msg) {\n super(msg);\n }\n}\n" }, { "path": "frontend/server/src/main/java/com/amazonaws/ml/mms/http/InvalidRequestHandler.java", "content": "package com.amazonaws.ml.mms.http;\n\nimport com.amazonaws.ml.mms.archive.ModelException;\nimport io.netty.channel.ChannelHandlerContext;\nimport io.netty.handler.codec.http.FullHttpRequest;\nimport io.netty.handler.codec.http.QueryStringDecoder;\n\npublic class InvalidRequestHandler extends HttpRequestHandlerChain {\n public InvalidRequestHandler() {}\n\n @Override\n protected void handleRequest(\n ChannelHandlerContext ctx,\n FullHttpRequest req,\n QueryStringDecoder decoder,\n String[] segments)\n throws ModelException {\n throw new ResourceNotFoundException();\n }\n}\n" }, { "path": "frontend/server/src/main/java/com/amazonaws/ml/mms/http/ListModelsResponse.java", "content": "/*\n * Copyright 2018 Amazon.com, Inc. or its affiliates. All Rights Reserved.\n *\n * Licensed under the Apache License, Version 2.0 (the \"License\"). You may not use this file except in compliance\n * with the License. A copy of the License is located at\n *\n * http://aws.amazon.com/apache2.0/\n *\n * or in the \"license\" file accompanying this file. This file is distributed on an \"AS IS\" BASIS, WITHOUT WARRANTIES\n * OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions\n * and limitations under the License.\n */\npackage com.amazonaws.ml.mms.http;\n\nimport java.util.ArrayList;\nimport java.util.List;\n\npublic class ListModelsResponse {\n\n private String nextPageToken;\n private List models;\n\n public ListModelsResponse() {\n models = new ArrayList<>();\n }\n\n public String getNextPageToken() {\n return nextPageToken;\n }\n\n public void setNextPageToken(String nextPageToken) {\n this.nextPageToken = nextPageToken;\n }\n\n public List getModels() {\n return models;\n }\n\n public void setModels(List models) {\n this.models = models;\n }\n\n public void addModel(String modelName, String modelUrl) {\n models.add(new ModelItem(modelName, modelUrl));\n }\n\n public static final class ModelItem {\n\n private String modelName;\n private String modelUrl;\n\n public ModelItem() {}\n\n public ModelItem(String modelName, String modelUrl) {\n this.modelName = modelName;\n this.modelUrl = modelUrl;\n }\n\n public String getModelName() {\n return modelName;\n }\n\n public void setModelName(String modelName) {\n this.modelName = modelName;\n }\n\n public String getModelUrl() {\n return modelUrl;\n }\n\n public void setModelUrl(String modelUrl) {\n this.modelUrl = modelUrl;\n }\n }\n}\n" }, { "path": "frontend/server/src/main/java/com/amazonaws/ml/mms/http/ManagementRequestHandler.java", "content": "/*\n * Copyright 2018 Amazon.com, Inc. or its affiliates. All Rights Reserved.\n *\n * Licensed under the Apache License, Version 2.0 (the \"License\"). You may not use this file except in compliance\n * with the License. A copy of the License is located at\n *\n * http://aws.amazon.com/apache2.0/\n *\n * or in the \"license\" file accompanying this file. This file is distributed on an \"AS IS\" BASIS, WITHOUT WARRANTIES\n * OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions\n * and limitations under the License.\n */\npackage com.amazonaws.ml.mms.http;\n\nimport com.amazonaws.ml.mms.archive.Manifest;\nimport com.amazonaws.ml.mms.archive.ModelArchive;\nimport com.amazonaws.ml.mms.archive.ModelException;\nimport com.amazonaws.ml.mms.archive.ModelNotFoundException;\nimport com.amazonaws.ml.mms.http.messages.RegisterModelRequest;\nimport com.amazonaws.ml.mms.util.ConfigManager;\nimport com.amazonaws.ml.mms.util.JsonUtils;\nimport com.amazonaws.ml.mms.util.NettyUtils;\nimport com.amazonaws.ml.mms.wlm.Model;\nimport com.amazonaws.ml.mms.wlm.ModelManager;\nimport com.amazonaws.ml.mms.wlm.WorkerThread;\nimport io.netty.channel.ChannelHandlerContext;\nimport io.netty.handler.codec.http.FullHttpRequest;\nimport io.netty.handler.codec.http.HttpHeaderValues;\nimport io.netty.handler.codec.http.HttpMethod;\nimport io.netty.handler.codec.http.HttpResponseStatus;\nimport io.netty.handler.codec.http.HttpUtil;\nimport io.netty.handler.codec.http.QueryStringDecoder;\nimport io.netty.util.CharsetUtil;\nimport java.io.IOException;\nimport java.util.ArrayList;\nimport java.util.Collections;\nimport java.util.List;\nimport java.util.Map;\nimport java.util.concurrent.CompletableFuture;\nimport java.util.concurrent.ExecutionException;\nimport java.util.concurrent.TimeoutException;\nimport java.util.function.Function;\nimport software.amazon.ai.mms.servingsdk.ModelServerEndpoint;\n\n/**\n * A class handling inbound HTTP requests to the management API.\n *\n *

This class\n */\npublic class ManagementRequestHandler extends HttpRequestHandlerChain {\n\n /** Creates a new {@code ManagementRequestHandler} instance. */\n public ManagementRequestHandler(Map ep) {\n endpointMap = ep;\n }\n\n @Override\n protected void handleRequest(\n ChannelHandlerContext ctx,\n FullHttpRequest req,\n QueryStringDecoder decoder,\n String[] segments)\n throws ModelException {\n if (isManagementReq(segments)) {\n if (endpointMap.getOrDefault(segments[1], null) != null) {\n handleCustomEndpoint(ctx, req, segments, decoder);\n } else {\n if (!\"models\".equals(segments[1])) {\n throw new ResourceNotFoundException();\n }\n\n HttpMethod method = req.method();\n if (segments.length < 3) {\n if (HttpMethod.GET.equals(method)) {\n handleListModels(ctx, decoder);\n return;\n } else if (HttpMethod.POST.equals(method)) {\n handleRegisterModel(ctx, decoder, req);\n return;\n }\n throw new MethodNotAllowedException();\n }\n\n if (HttpMethod.GET.equals(method)) {\n handleDescribeModel(ctx, segments[2]);\n } else if (HttpMethod.PUT.equals(method)) {\n handleScaleModel(ctx, decoder, segments[2]);\n } else if (HttpMethod.DELETE.equals(method)) {\n handleUnregisterModel(ctx, segments[2]);\n } else {\n throw new MethodNotAllowedException();\n }\n }\n } else {\n chain.handleRequest(ctx, req, decoder, segments);\n }\n }\n\n private boolean isManagementReq(String[] segments) {\n return segments.length == 0\n || ((segments.length == 2 || segments.length == 3) && segments[1].equals(\"models\"))\n || endpointMap.containsKey(segments[1]);\n }\n\n private void handleListModels(ChannelHandlerContext ctx, QueryStringDecoder decoder) {\n int limit = NettyUtils.getIntParameter(decoder, \"limit\", 100);\n int pageToken = NettyUtils.getIntParameter(decoder, \"next_page_token\", 0);\n if (limit > 100 || limit < 0) {\n limit = 100;\n }\n if (pageToken < 0) {\n pageToken = 0;\n }\n\n ModelManager modelManager = ModelManager.getInstance();\n Map models = modelManager.getModels();\n\n List keys = new ArrayList<>(models.keySet());\n Collections.sort(keys);\n ListModelsResponse list = new ListModelsResponse();\n\n int last = pageToken + limit;\n if (last > keys.size()) {\n last = keys.size();\n } else {\n list.setNextPageToken(String.valueOf(last));\n }\n\n for (int i = pageToken; i < last; ++i) {\n String modelName = keys.get(i);\n Model model = models.get(modelName);\n list.addModel(modelName, model.getModelUrl());\n }\n\n NettyUtils.sendJsonResponse(ctx, list);\n }\n\n private void handleDescribeModel(ChannelHandlerContext ctx, String modelName)\n throws ModelNotFoundException {\n ModelManager modelManager = ModelManager.getInstance();\n Model model = modelManager.getModels().get(modelName);\n if (model == null) {\n throw new ModelNotFoundException(\"Model not found: \" + modelName);\n }\n\n DescribeModelResponse resp = new DescribeModelResponse();\n resp.setModelName(modelName);\n resp.setModelUrl(model.getModelUrl());\n resp.setBatchSize(model.getBatchSize());\n resp.setMaxBatchDelay(model.getMaxBatchDelay());\n resp.setMaxWorkers(model.getMaxWorkers());\n resp.setMinWorkers(model.getMinWorkers());\n resp.setLoadedAtStartup(modelManager.getStartupModels().contains(modelName));\n Manifest manifest = model.getModelArchive().getManifest();\n Manifest.Engine engine = manifest.getEngine();\n if (engine != null) {\n resp.setEngine(engine.getEngineName());\n }\n resp.setModelVersion(manifest.getModel().getModelVersion());\n resp.setRuntime(manifest.getRuntime().getValue());\n\n List workers = modelManager.getWorkers(modelName);\n for (WorkerThread worker : workers) {\n String workerId = worker.getWorkerId();\n long startTime = worker.getStartTime();\n boolean isRunning = worker.isRunning();\n int gpuId = worker.getGpuId();\n long memory = worker.getMemory();\n resp.addWorker(workerId, startTime, isRunning, gpuId, memory);\n }\n\n NettyUtils.sendJsonResponse(ctx, resp);\n }\n\n private void handleRegisterModel(\n ChannelHandlerContext ctx, QueryStringDecoder decoder, FullHttpRequest req)\n throws ModelException {\n RegisterModelRequest registerModelRequest = parseRequest(req, decoder);\n String modelUrl = registerModelRequest.getModelUrl();\n if (modelUrl == null) {\n throw new BadRequestException(\"Parameter url is required.\");\n }\n\n String modelName = registerModelRequest.getModelName();\n String runtime = registerModelRequest.getRuntime();\n String handler = registerModelRequest.getHandler();\n int batchSize = registerModelRequest.getBatchSize();\n int maxBatchDelay = registerModelRequest.getMaxBatchDelay();\n int initialWorkers = registerModelRequest.getInitialWorkers();\n boolean synchronous = registerModelRequest.isSynchronous();\n int responseTimeoutSeconds = registerModelRequest.getResponseTimeoutSeconds();\n String preloadModel = registerModelRequest.getPreloadModel();\n if (preloadModel == null) {\n preloadModel = ConfigManager.getInstance().getPreloadModel();\n }\n if (responseTimeoutSeconds == -1) {\n responseTimeoutSeconds = ConfigManager.getInstance().getDefaultResponseTimeoutSeconds();\n }\n Manifest.RuntimeType runtimeType = null;\n if (runtime != null) {\n try {\n runtimeType = Manifest.RuntimeType.fromValue(runtime);\n } catch (IllegalArgumentException e) {\n throw new BadRequestException(e);\n }\n }\n\n ModelManager modelManager = ModelManager.getInstance();\n final ModelArchive archive;\n try {\n\n archive =\n modelManager.registerModel(\n modelUrl,\n modelName,\n runtimeType,\n handler,\n batchSize,\n maxBatchDelay,\n responseTimeoutSeconds,\n null,\n preloadModel);\n } catch (IOException | InterruptedException | ExecutionException | TimeoutException e) {\n throw new InternalServerException(\"Failed to save model: \" + modelUrl, e);\n }\n\n modelName = archive.getModelName();\n\n final String msg = \"Model \\\"\" + modelName + \"\\\" registered\";\n if (initialWorkers <= 0) {\n NettyUtils.sendJsonResponse(ctx, new StatusResponse(msg));\n return;\n }\n\n updateModelWorkers(\n ctx,\n modelName,\n initialWorkers,\n initialWorkers,\n synchronous,\n f -> {\n modelManager.unregisterModel(archive.getModelName());\n return null;\n });\n }\n\n private void handleUnregisterModel(ChannelHandlerContext ctx, String modelName)\n throws ModelNotFoundException, InternalServerException, RequestTimeoutException {\n ModelManager modelManager = ModelManager.getInstance();\n HttpResponseStatus httpResponseStatus = modelManager.unregisterModel(modelName);\n if (httpResponseStatus == HttpResponseStatus.NOT_FOUND) {\n throw new ModelNotFoundException(\"Model not found: \" + modelName);\n } else if (httpResponseStatus == HttpResponseStatus.INTERNAL_SERVER_ERROR) {\n throw new InternalServerException(\"Interrupted while cleaning resources: \" + modelName);\n } else if (httpResponseStatus == HttpResponseStatus.REQUEST_TIMEOUT) {\n throw new RequestTimeoutException(\"Timed out while cleaning resources: \" + modelName);\n }\n String msg = \"Model \\\"\" + modelName + \"\\\" unregistered\";\n NettyUtils.sendJsonResponse(ctx, new StatusResponse(msg));\n }\n\n private void handleScaleModel(\n ChannelHandlerContext ctx, QueryStringDecoder decoder, String modelName)\n throws ModelNotFoundException {\n int minWorkers = NettyUtils.getIntParameter(decoder, \"min_worker\", 1);\n int maxWorkers = NettyUtils.getIntParameter(decoder, \"max_worker\", minWorkers);\n if (maxWorkers < minWorkers) {\n throw new BadRequestException(\"max_worker cannot be less than min_worker.\");\n }\n boolean synchronous =\n Boolean.parseBoolean(NettyUtils.getParameter(decoder, \"synchronous\", null));\n\n ModelManager modelManager = ModelManager.getInstance();\n if (!modelManager.getModels().containsKey(modelName)) {\n throw new ModelNotFoundException(\"Model not found: \" + modelName);\n }\n updateModelWorkers(ctx, modelName, minWorkers, maxWorkers, synchronous, null);\n }\n\n private void updateModelWorkers(\n final ChannelHandlerContext ctx,\n final String modelName,\n int minWorkers,\n int maxWorkers,\n boolean synchronous,\n final Function onError) {\n\n ModelManager modelManager = ModelManager.getInstance();\n CompletableFuture future =\n modelManager.updateModel(modelName, minWorkers, maxWorkers);\n if (!synchronous) {\n NettyUtils.sendJsonResponse(\n ctx,\n new StatusResponse(\"Processing worker updates...\"),\n HttpResponseStatus.ACCEPTED);\n return;\n }\n future.thenApply(\n v -> {\n boolean status = modelManager.scaleRequestStatus(modelName);\n if (HttpResponseStatus.OK.equals(v)) {\n if (status) {\n NettyUtils.sendJsonResponse(\n ctx, new StatusResponse(\"Workers scaled\"), v);\n } else {\n NettyUtils.sendJsonResponse(\n ctx,\n new StatusResponse(\"Workers scaling in progress...\"),\n new HttpResponseStatus(210, \"Partial Success\"));\n }\n } else {\n NettyUtils.sendError(\n ctx,\n v,\n new InternalServerException(\"Failed to start workers\"));\n if (onError != null) {\n onError.apply(null);\n }\n }\n return v;\n })\n .exceptionally(\n (e) -> {\n if (onError != null) {\n onError.apply(null);\n }\n NettyUtils.sendError(ctx, HttpResponseStatus.INTERNAL_SERVER_ERROR, e);\n return null;\n });\n }\n\n private RegisterModelRequest parseRequest(FullHttpRequest req, QueryStringDecoder decoder) {\n RegisterModelRequest in;\n CharSequence mime = HttpUtil.getMimeType(req);\n if (HttpHeaderValues.APPLICATION_JSON.contentEqualsIgnoreCase(mime)) {\n in =\n JsonUtils.GSON.fromJson(\n req.content().toString(CharsetUtil.UTF_8), RegisterModelRequest.class);\n } else {\n in = new RegisterModelRequest(decoder);\n }\n return in;\n }\n}\n" }, { "path": "frontend/server/src/main/java/com/amazonaws/ml/mms/http/MethodNotAllowedException.java", "content": "/*\n * Copyright 2018 Amazon.com, Inc. or its affiliates. All Rights Reserved.\n *\n * Licensed under the Apache License, Version 2.0 (the \"License\"). You may not use this file except in compliance\n * with the License. A copy of the License is located at\n *\n * http://aws.amazon.com/apache2.0/\n *\n * or in the \"license\" file accompanying this file. This file is distributed on an \"AS IS\" BASIS, WITHOUT WARRANTIES\n * OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions\n * and limitations under the License.\n */\npackage com.amazonaws.ml.mms.http;\n\npublic class MethodNotAllowedException extends RuntimeException {\n\n static final long serialVersionUID = 1L;\n\n /**\n * Constructs an {@code MethodNotAllowedException} with {@code null} as its error detail\n * message.\n */\n public MethodNotAllowedException() {\n super(\"Requested method is not allowed, please refer to API document.\");\n }\n}\n" }, { "path": "frontend/server/src/main/java/com/amazonaws/ml/mms/http/RequestTimeoutException.java", "content": "/*\n * Copyright 2018 Amazon.com, Inc. or its affiliates. All Rights Reserved.\n *\n * Licensed under the Apache License, Version 2.0 (the \"License\"). You may not use this file except in compliance\n * with the License. A copy of the License is located at\n *\n * http://aws.amazon.com/apache2.0/\n *\n * or in the \"license\" file accompanying this file. This file is distributed on an \"AS IS\" BASIS, WITHOUT WARRANTIES\n * OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions\n * and limitations under the License.\n */\npackage com.amazonaws.ml.mms.http;\n\npublic class RequestTimeoutException extends RuntimeException {\n\n static final long serialVersionUID = 1L;\n\n /**\n * Constructs an {@code RequestTimeoutException} with the specified detail message.\n *\n * @param message The detail message (which is saved for later retrieval by the {@link\n * #getMessage()} method)\n */\n public RequestTimeoutException(String message) {\n super(message);\n }\n\n /**\n * Constructs an {@code RequestTimeoutException} with the specified detail message and cause.\n *\n *

Note that the detail message associated with {@code cause} is not automatically\n * incorporated into this exception's detail message.\n *\n * @param message The detail message (which is saved for later retrieval by the {@link\n * #getMessage()} method)\n * @param cause The cause (which is saved for later retrieval by the {@link #getCause()}\n * method). (A null value is permitted, and indicates that the cause is nonexistent or\n * unknown.)\n */\n public RequestTimeoutException(String message, Throwable cause) {\n super(message, cause);\n }\n}\n" }, { "path": "frontend/server/src/main/java/com/amazonaws/ml/mms/http/ResourceNotFoundException.java", "content": "/*\n * Copyright 2018 Amazon.com, Inc. or its affiliates. All Rights Reserved.\n *\n * Licensed under the Apache License, Version 2.0 (the \"License\"). You may not use this file except in compliance\n * with the License. A copy of the License is located at\n *\n * http://aws.amazon.com/apache2.0/\n *\n * or in the \"license\" file accompanying this file. This file is distributed on an \"AS IS\" BASIS, WITHOUT WARRANTIES\n * OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions\n * and limitations under the License.\n */\npackage com.amazonaws.ml.mms.http;\n\npublic class ResourceNotFoundException extends RuntimeException {\n\n static final long serialVersionUID = 1L;\n\n /**\n * Constructs an {@code ResourceNotFoundException} with {@code null} as its error detail\n * message.\n */\n public ResourceNotFoundException() {\n super(\"Requested resource is not found, please refer to API document.\");\n }\n}\n" }, { "path": "frontend/server/src/main/java/com/amazonaws/ml/mms/http/ServiceUnavailableException.java", "content": "package com.amazonaws.ml.mms.http;\n\npublic class ServiceUnavailableException extends RuntimeException {\n\n static final long serialVersionUID = 1L;\n\n /**\n * Constructs an {@code ServiceUnavailableException} with the specified detail message.\n *\n * @param message The detail message (which is saved for later retrieval by the {@link\n * #getMessage()} method)\n */\n public ServiceUnavailableException(String message) {\n super(message);\n }\n}\n" }, { "path": "frontend/server/src/main/java/com/amazonaws/ml/mms/http/Session.java", "content": "/*\n * Copyright 2018 Amazon.com, Inc. or its affiliates. All Rights Reserved.\n *\n * Licensed under the Apache License, Version 2.0 (the \"License\"). You may not use this file except in compliance\n * with the License. A copy of the License is located at\n *\n * http://aws.amazon.com/apache2.0/\n *\n * or in the \"license\" file accompanying this file. This file is distributed on an \"AS IS\" BASIS, WITHOUT WARRANTIES\n * OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions\n * and limitations under the License.\n */\npackage com.amazonaws.ml.mms.http;\n\nimport io.netty.handler.codec.http.HttpRequest;\nimport java.util.UUID;\n\npublic class Session {\n\n private String requestId;\n private String remoteIp;\n private String method;\n private String uri;\n private String protocol;\n private int code;\n private long startTime;\n\n public Session(String remoteIp, HttpRequest request) {\n this.remoteIp = remoteIp;\n this.uri = request.uri();\n if (request.decoderResult().isSuccess()) {\n method = request.method().name();\n protocol = request.protocolVersion().text();\n } else {\n method = \"GET\";\n protocol = \"HTTP/1.1\";\n }\n requestId = UUID.randomUUID().toString();\n startTime = System.currentTimeMillis();\n }\n\n public String getRequestId() {\n return requestId;\n }\n\n public void setCode(int code) {\n this.code = code;\n }\n\n @Override\n public String toString() {\n long duration = System.currentTimeMillis() - startTime;\n return remoteIp + \" \\\"\" + method + \" \" + uri + ' ' + protocol + \"\\\" \" + code + ' '\n + duration;\n }\n}\n" }, { "path": "frontend/server/src/main/java/com/amazonaws/ml/mms/http/StatusResponse.java", "content": "/*\n * Copyright 2018 Amazon.com, Inc. or its affiliates. All Rights Reserved.\n *\n * Licensed under the Apache License, Version 2.0 (the \"License\"). You may not use this file except in compliance\n * with the License. A copy of the License is located at\n *\n * http://aws.amazon.com/apache2.0/\n *\n * or in the \"license\" file accompanying this file. This file is distributed on an \"AS IS\" BASIS, WITHOUT WARRANTIES\n * OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions\n * and limitations under the License.\n */\npackage com.amazonaws.ml.mms.http;\n\npublic class StatusResponse {\n\n private String status;\n\n public StatusResponse() {}\n\n public StatusResponse(String status) {\n this.status = status;\n }\n\n public String getStatus() {\n return status;\n }\n\n public void setStatus(String status) {\n this.status = status;\n }\n}\n" }, { "path": "frontend/server/src/main/java/com/amazonaws/ml/mms/http/messages/RegisterModelRequest.java", "content": "/*\n * Copyright 2019 Amazon.com, Inc. or its affiliates. All Rights Reserved.\n *\n * Licensed under the Apache License, Version 2.0 (the \"License\"). You may not use this file except in compliance\n * with the License. A copy of the License is located at\n *\n * http://aws.amazon.com/apache2.0/\n *\n * or in the \"license\" file accompanying this file. This file is distributed on an \"AS IS\" BASIS, WITHOUT WARRANTIES\n * OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions\n * and limitations under the License.\n */\npackage com.amazonaws.ml.mms.http.messages;\n\nimport com.amazonaws.ml.mms.util.ConfigManager;\nimport com.amazonaws.ml.mms.util.NettyUtils;\nimport com.google.gson.annotations.SerializedName;\nimport io.netty.handler.codec.http.QueryStringDecoder;\n\n/** Register Model Request for Model server */\npublic class RegisterModelRequest {\n @SerializedName(\"model_name\")\n private String modelName;\n\n @SerializedName(\"runtime\")\n private String runtime;\n\n @SerializedName(\"handler\")\n private String handler;\n\n @SerializedName(\"batch_size\")\n private int batchSize;\n\n @SerializedName(\"max_batch_delay\")\n private int maxBatchDelay;\n\n @SerializedName(\"initial_workers\")\n private int initialWorkers;\n\n @SerializedName(\"synchronous\")\n private boolean synchronous;\n\n @SerializedName(\"response_timeout\")\n private int responseTimeoutSeconds;\n\n @SerializedName(\"url\")\n private String modelUrl;\n\n @SerializedName(\"preload_model\")\n private String preloadModel;\n\n public RegisterModelRequest(QueryStringDecoder decoder) {\n modelName = NettyUtils.getParameter(decoder, \"model_name\", null);\n runtime = NettyUtils.getParameter(decoder, \"runtime\", null);\n handler = NettyUtils.getParameter(decoder, \"handler\", null);\n batchSize = NettyUtils.getIntParameter(decoder, \"batch_size\", 1);\n maxBatchDelay = NettyUtils.getIntParameter(decoder, \"max_batch_delay\", 100);\n initialWorkers =\n NettyUtils.getIntParameter(\n decoder,\n \"initial_workers\",\n ConfigManager.getInstance().getConfiguredDefaultWorkersPerModel());\n synchronous = Boolean.parseBoolean(NettyUtils.getParameter(decoder, \"synchronous\", \"true\"));\n\n // TODO Fix this so it matches the documentation, where timeouts are specified in seconds.\n // For now, we're being extra careful about backwards compatibility.\n // So, assume parameter is in minutes, and convert to seconds internally.\n responseTimeoutSeconds = 60 * NettyUtils.getIntParameter(decoder, \"response_timeout\", -1);\n if (responseTimeoutSeconds < 0) {\n responseTimeoutSeconds = -1;\n }\n\n modelUrl = NettyUtils.getParameter(decoder, \"url\", null);\n preloadModel = NettyUtils.getParameter(decoder, \"preload_model\", null);\n }\n\n public RegisterModelRequest() {\n batchSize = 1;\n maxBatchDelay = 100;\n synchronous = true;\n initialWorkers = ConfigManager.getInstance().getConfiguredDefaultWorkersPerModel();\n responseTimeoutSeconds = -1;\n preloadModel = null;\n }\n\n public String getModelName() {\n return modelName;\n }\n\n public String getRuntime() {\n return runtime;\n }\n\n public String getHandler() {\n return handler;\n }\n\n public Integer getBatchSize() {\n return batchSize;\n }\n\n public Integer getMaxBatchDelay() {\n return maxBatchDelay;\n }\n\n public Integer getInitialWorkers() {\n return initialWorkers;\n }\n\n public Boolean isSynchronous() {\n return synchronous;\n }\n\n public Integer getResponseTimeoutSeconds() {\n return responseTimeoutSeconds;\n }\n\n public String getModelUrl() {\n return modelUrl;\n }\n\n public String getPreloadModel() {\n return preloadModel;\n }\n}\n" }, { "path": "frontend/server/src/main/java/com/amazonaws/ml/mms/metrics/Dimension.java", "content": "/*\n * Copyright 2018 Amazon.com, Inc. or its affiliates. All Rights Reserved.\n *\n * Licensed under the Apache License, Version 2.0 (the \"License\"). You may not use this file except in compliance\n * with the License. A copy of the License is located at\n *\n * http://aws.amazon.com/apache2.0/\n *\n * or in the \"license\" file accompanying this file. This file is distributed on an \"AS IS\" BASIS, WITHOUT WARRANTIES\n * OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions\n * and limitations under the License.\n */\npackage com.amazonaws.ml.mms.metrics;\n\nimport com.google.gson.annotations.SerializedName;\n\npublic class Dimension {\n\n @SerializedName(\"Name\")\n private String name;\n\n @SerializedName(\"Value\")\n private String value;\n\n public Dimension() {}\n\n public Dimension(String name, String value) {\n this.name = name;\n this.value = value;\n }\n\n public String getName() {\n return name;\n }\n\n public void setName(String name) {\n this.name = name;\n }\n\n public String getValue() {\n return value;\n }\n\n public void setValue(String value) {\n this.value = value;\n }\n}\n" }, { "path": "frontend/server/src/main/java/com/amazonaws/ml/mms/metrics/Metric.java", "content": "/*\n * Copyright 2018 Amazon.com, Inc. or its affiliates. All Rights Reserved.\n *\n * Licensed under the Apache License, Version 2.0 (the \"License\"). You may not use this file except in compliance\n * with the License. A copy of the License is located at\n *\n * http://aws.amazon.com/apache2.0/\n *\n * or in the \"license\" file accompanying this file. This file is distributed on an \"AS IS\" BASIS, WITHOUT WARRANTIES\n * OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions\n * and limitations under the License.\n */\npackage com.amazonaws.ml.mms.metrics;\n\nimport com.google.gson.annotations.SerializedName;\nimport java.util.ArrayList;\nimport java.util.Arrays;\nimport java.util.List;\nimport java.util.regex.Matcher;\nimport java.util.regex.Pattern;\n\npublic class Metric {\n\n private static final Pattern PATTERN =\n Pattern.compile(\n \"\\\\s*(\\\\w+)\\\\.(\\\\w+):([0-9\\\\-,.e]+)\\\\|#([^|]*)\\\\|#hostname:([^,]+),([^,]+)(,(.*))?\");\n\n @SerializedName(\"MetricName\")\n private String metricName;\n\n @SerializedName(\"Value\")\n private String value;\n\n @SerializedName(\"Unit\")\n private String unit;\n\n @SerializedName(\"Dimensions\")\n private List dimensions;\n\n @SerializedName(\"Timestamp\")\n private String timestamp;\n\n @SerializedName(\"RequestId\")\n private String requestId;\n\n @SerializedName(\"HostName\")\n private String hostName;\n\n public Metric() {}\n\n public Metric(\n String metricName,\n String value,\n String unit,\n String hostName,\n Dimension... dimensions) {\n this.metricName = metricName;\n this.value = value;\n this.unit = unit;\n this.hostName = hostName;\n this.dimensions = Arrays.asList(dimensions);\n }\n\n public String getHostName() {\n return hostName;\n }\n\n public void setHostName(String hostName) {\n this.hostName = hostName;\n }\n\n public String getRequestId() {\n return requestId;\n }\n\n public void setRequestId(String requestId) {\n this.requestId = requestId;\n }\n\n public String getMetricName() {\n return metricName;\n }\n\n public void setMetricName(String metricName) {\n this.metricName = metricName;\n }\n\n public String getValue() {\n return value;\n }\n\n public void setValue(String value) {\n this.value = value;\n }\n\n public String getUnit() {\n return unit;\n }\n\n public void setUnit(String unit) {\n this.unit = unit;\n }\n\n public List getDimensions() {\n return dimensions;\n }\n\n public void setDimensions(List dimensions) {\n this.dimensions = dimensions;\n }\n\n public String getTimestamp() {\n return timestamp;\n }\n\n public void setTimestamp(String timestamp) {\n this.timestamp = timestamp;\n }\n\n public static Metric parse(String line) {\n // DiskAvailable.Gigabytes:311|#Level:Host,hostname:localhost\n Matcher matcher = PATTERN.matcher(line);\n if (!matcher.matches()) {\n return null;\n }\n\n Metric metric = new Metric();\n metric.setMetricName(matcher.group(1));\n metric.setUnit(matcher.group(2));\n metric.setValue(matcher.group(3));\n String dimensions = matcher.group(4);\n metric.setHostName(matcher.group(5));\n metric.setTimestamp(matcher.group(6));\n metric.setRequestId(matcher.group(8));\n\n if (dimensions != null) {\n String[] dimension = dimensions.split(\",\");\n List list = new ArrayList<>(dimension.length);\n for (String dime : dimension) {\n String[] pair = dime.split(\":\");\n if (pair.length == 2) {\n list.add(new Dimension(pair[0], pair[1]));\n }\n }\n metric.setDimensions(list);\n }\n\n return metric;\n }\n\n @Override\n public String toString() {\n StringBuilder sb = new StringBuilder(128);\n sb.append(metricName).append('.').append(unit).append(':').append(getValue()).append(\"|#\");\n boolean first = true;\n for (Dimension dimension : getDimensions()) {\n if (first) {\n first = false;\n } else {\n sb.append(',');\n }\n sb.append(dimension.getName()).append(':').append(dimension.getValue());\n }\n sb.append(\"|#hostname:\").append(hostName);\n if (requestId != null) {\n sb.append(\",requestID:\").append(requestId);\n }\n sb.append(\",timestamp:\").append(timestamp);\n return sb.toString();\n }\n}\n" }, { "path": "frontend/server/src/main/java/com/amazonaws/ml/mms/metrics/MetricCollector.java", "content": "/*\n * Copyright 2018 Amazon.com, Inc. or its affiliates. All Rights Reserved.\n *\n * Licensed under the Apache License, Version 2.0 (the \"License\"). You may not use this file except in compliance\n * with the License. A copy of the License is located at\n *\n * http://aws.amazon.com/apache2.0/\n *\n * or in the \"license\" file accompanying this file. This file is distributed on an \"AS IS\" BASIS, WITHOUT WARRANTIES\n * OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions\n * and limitations under the License.\n */\npackage com.amazonaws.ml.mms.metrics;\n\nimport com.amazonaws.ml.mms.util.ConfigManager;\nimport com.amazonaws.ml.mms.wlm.ModelManager;\nimport com.amazonaws.ml.mms.wlm.WorkerThread;\nimport java.io.BufferedReader;\nimport java.io.File;\nimport java.io.IOException;\nimport java.io.InputStreamReader;\nimport java.io.OutputStream;\nimport java.nio.charset.StandardCharsets;\nimport java.util.ArrayList;\nimport java.util.List;\nimport java.util.Map;\nimport org.apache.commons.io.IOUtils;\nimport org.slf4j.Logger;\nimport org.slf4j.LoggerFactory;\n\npublic class MetricCollector implements Runnable {\n\n static final Logger logger = LoggerFactory.getLogger(MetricCollector.class);\n private static final Logger loggerMetrics =\n LoggerFactory.getLogger(ConfigManager.MODEL_SERVER_METRICS_LOGGER);\n private ConfigManager configManager;\n\n public MetricCollector(ConfigManager configManager) {\n this.configManager = configManager;\n }\n\n @Override\n public void run() {\n try {\n // Collect System level Metrics\n String[] args = new String[2];\n args[0] = configManager.getPythonExecutable();\n args[1] = \"mms/metrics/metric_collector.py\";\n File workingDir = new File(configManager.getModelServerHome());\n\n String pythonPath = System.getenv(\"PYTHONPATH\");\n String pythonEnv;\n if ((pythonPath == null || pythonPath.isEmpty())\n && (!workingDir.getAbsolutePath().contains(\"site-package\"))) {\n pythonEnv = \"PYTHONPATH=\" + workingDir.getAbsolutePath();\n } else {\n pythonEnv = \"PYTHONPATH=\" + pythonPath;\n if (!workingDir.getAbsolutePath().contains(\"site-package\")) {\n pythonEnv += File.pathSeparatorChar + workingDir.getAbsolutePath(); // NOPMD\n }\n }\n // sbin added for macs for python sysctl pythonpath\n StringBuilder path = new StringBuilder();\n path.append(\"PATH=\").append(System.getenv(\"PATH\"));\n String osName = System.getProperty(\"os.name\");\n if (osName.startsWith(\"Mac OS X\")) {\n path.append(File.pathSeparatorChar).append(\"/sbin/\");\n }\n String[] env = {pythonEnv, path.toString()};\n final Process p = Runtime.getRuntime().exec(args, env, workingDir);\n\n ModelManager modelManager = ModelManager.getInstance();\n Map workerMap = modelManager.getWorkers();\n try (OutputStream os = p.getOutputStream()) {\n writeWorkerPids(workerMap, os);\n }\n\n new Thread(\n () -> {\n try {\n String error =\n IOUtils.toString(\n p.getErrorStream(), StandardCharsets.UTF_8);\n if (!error.isEmpty()) {\n logger.error(error);\n }\n } catch (IOException e) {\n logger.error(\"\", e);\n }\n })\n .start();\n\n MetricManager metricManager = MetricManager.getInstance();\n try (BufferedReader reader =\n new BufferedReader(\n new InputStreamReader(p.getInputStream(), StandardCharsets.UTF_8))) {\n List metricsSystem = new ArrayList<>();\n metricManager.setMetrics(metricsSystem);\n\n String line;\n while ((line = reader.readLine()) != null) {\n if (line.isEmpty()) {\n break;\n }\n Metric metric = Metric.parse(line);\n if (metric == null) {\n logger.warn(\"Parse metrics failed: \" + line);\n } else {\n loggerMetrics.info(\"{}\", metric);\n metricsSystem.add(metric);\n }\n }\n\n // Collect process level metrics\n while ((line = reader.readLine()) != null) {\n String[] tokens = line.split(\":\");\n if (tokens.length != 2) {\n continue;\n }\n try {\n Integer pid = Integer.valueOf(tokens[0]);\n WorkerThread worker = workerMap.get(pid);\n worker.setMemory(Long.parseLong(tokens[1]));\n } catch (NumberFormatException e) {\n logger.warn(\"Failed to parse memory utilization metrics: \" + line);\n continue;\n }\n }\n }\n } catch (IOException e) {\n logger.error(\"\", e);\n }\n }\n\n private void writeWorkerPids(Map workerMap, OutputStream os)\n throws IOException {\n boolean first = true;\n for (Integer pid : workerMap.keySet()) {\n if (pid < 0) {\n logger.warn(\"worker pid is not available yet.\");\n continue;\n }\n if (first) {\n first = false;\n } else {\n IOUtils.write(\",\", os, StandardCharsets.UTF_8);\n }\n IOUtils.write(pid.toString(), os, StandardCharsets.UTF_8);\n }\n os.write('\\n');\n }\n}\n" }, { "path": "frontend/server/src/main/java/com/amazonaws/ml/mms/metrics/MetricManager.java", "content": "/*\n * Copyright 2018 Amazon.com, Inc. or its affiliates. All Rights Reserved.\n *\n * Licensed under the Apache License, Version 2.0 (the \"License\"). You may not use this file except in compliance\n * with the License. A copy of the License is located at\n *\n * http://aws.amazon.com/apache2.0/\n *\n * or in the \"license\" file accompanying this file. This file is distributed on an \"AS IS\" BASIS, WITHOUT WARRANTIES\n * OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions\n * and limitations under the License.\n */\npackage com.amazonaws.ml.mms.metrics;\n\nimport com.amazonaws.ml.mms.util.ConfigManager;\nimport com.amazonaws.ml.mms.wlm.ModelManager;\nimport java.util.Collections;\nimport java.util.List;\nimport java.util.concurrent.TimeUnit;\n\npublic final class MetricManager {\n\n private static final MetricManager METRIC_MANAGER = new MetricManager();\n private List metrics;\n\n private MetricManager() {\n metrics = Collections.emptyList();\n }\n\n public static MetricManager getInstance() {\n return METRIC_MANAGER;\n }\n\n public static void scheduleMetrics(ConfigManager configManager) {\n MetricCollector metricCollector = new MetricCollector(configManager);\n ModelManager.getInstance()\n .getScheduler()\n .scheduleAtFixedRate(\n metricCollector,\n 0,\n configManager.getMetricTimeInterval(),\n TimeUnit.SECONDS);\n }\n\n public synchronized List getMetrics() {\n return metrics;\n }\n\n public synchronized void setMetrics(List metrics) {\n this.metrics = metrics;\n }\n}\n" }, { "path": "frontend/server/src/main/java/com/amazonaws/ml/mms/openapi/Encoding.java", "content": "/*\n * Copyright 2018 Amazon.com, Inc. or its affiliates. All Rights Reserved.\n *\n * Licensed under the Apache License, Version 2.0 (the \"License\"). You may not use this file except in compliance\n * with the License. A copy of the License is located at\n *\n * http://aws.amazon.com/apache2.0/\n *\n * or in the \"license\" file accompanying this file. This file is distributed on an \"AS IS\" BASIS, WITHOUT WARRANTIES\n * OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions\n * and limitations under the License.\n */\npackage com.amazonaws.ml.mms.openapi;\n\npublic class Encoding {\n\n private String contentType;\n private String style;\n private boolean explode;\n private boolean allowReserved;\n\n public Encoding() {}\n\n public Encoding(String contentType) {\n this.contentType = contentType;\n }\n\n public String getContentType() {\n return contentType;\n }\n\n public void setContentType(String contentType) {\n this.contentType = contentType;\n }\n\n public boolean isAllowReserved() {\n return allowReserved;\n }\n\n public void setAllowReserved(boolean allowReserved) {\n this.allowReserved = allowReserved;\n }\n\n public String getStyle() {\n return style;\n }\n\n public void setStyle(String style) {\n this.style = style;\n }\n\n public boolean isExplode() {\n return explode;\n }\n\n public void setExplode(boolean explode) {\n this.explode = explode;\n }\n}\n" }, { "path": "frontend/server/src/main/java/com/amazonaws/ml/mms/openapi/Info.java", "content": "/*\n * Copyright 2018 Amazon.com, Inc. or its affiliates. All Rights Reserved.\n *\n * Licensed under the Apache License, Version 2.0 (the \"License\"). You may not use this file except in compliance\n * with the License. A copy of the License is located at\n *\n * http://aws.amazon.com/apache2.0/\n *\n * or in the \"license\" file accompanying this file. This file is distributed on an \"AS IS\" BASIS, WITHOUT WARRANTIES\n * OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions\n * and limitations under the License.\n */\npackage com.amazonaws.ml.mms.openapi;\n\npublic class Info {\n\n private String title;\n private String description;\n private String termsOfService;\n private String version;\n\n public Info() {}\n\n public String getTitle() {\n return title;\n }\n\n public void setTitle(String title) {\n this.title = title;\n }\n\n public String getDescription() {\n return description;\n }\n\n public void setDescription(String description) {\n this.description = description;\n }\n\n public String getTermsOfService() {\n return termsOfService;\n }\n\n public void setTermsOfService(String termsOfService) {\n this.termsOfService = termsOfService;\n }\n\n public String getVersion() {\n return version;\n }\n\n public void setVersion(String version) {\n this.version = version;\n }\n}\n" }, { "path": "frontend/server/src/main/java/com/amazonaws/ml/mms/openapi/MediaType.java", "content": "/*\n * Copyright 2018 Amazon.com, Inc. or its affiliates. All Rights Reserved.\n *\n * Licensed under the Apache License, Version 2.0 (the \"License\"). You may not use this file except in compliance\n * with the License. A copy of the License is located at\n *\n * http://aws.amazon.com/apache2.0/\n *\n * or in the \"license\" file accompanying this file. This file is distributed on an \"AS IS\" BASIS, WITHOUT WARRANTIES\n * OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions\n * and limitations under the License.\n */\npackage com.amazonaws.ml.mms.openapi;\n\nimport java.util.LinkedHashMap;\nimport java.util.Map;\n\npublic class MediaType {\n\n private transient String contentType;\n private Schema schema;\n private Map encoding;\n\n public MediaType() {}\n\n public MediaType(String contentType, Schema schema) {\n this.contentType = contentType;\n this.schema = schema;\n }\n\n public String getContentType() {\n return contentType;\n }\n\n public void setContentType(String contentType) {\n this.contentType = contentType;\n }\n\n public Schema getSchema() {\n return schema;\n }\n\n public void setSchema(Schema schema) {\n this.schema = schema;\n }\n\n public Map getEncoding() {\n return encoding;\n }\n\n public void setEncoding(Map encoding) {\n this.encoding = encoding;\n }\n\n public void addEncoding(String contentType, Encoding encoding) {\n if (this.encoding == null) {\n this.encoding = new LinkedHashMap<>();\n }\n this.encoding.put(contentType, encoding);\n }\n}\n" }, { "path": "frontend/server/src/main/java/com/amazonaws/ml/mms/openapi/OpenApi.java", "content": "/*\n * Copyright 2018 Amazon.com, Inc. or its affiliates. All Rights Reserved.\n *\n * Licensed under the Apache License, Version 2.0 (the \"License\"). You may not use this file except in compliance\n * with the License. A copy of the License is located at\n *\n * http://aws.amazon.com/apache2.0/\n *\n * or in the \"license\" file accompanying this file. This file is distributed on an \"AS IS\" BASIS, WITHOUT WARRANTIES\n * OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions\n * and limitations under the License.\n */\npackage com.amazonaws.ml.mms.openapi;\n\nimport java.util.LinkedHashMap;\nimport java.util.Map;\n\npublic class OpenApi {\n\n private String openapi = \"3.0.1\";\n private Info info;\n private Map paths;\n\n public OpenApi() {}\n\n public String getOpenapi() {\n return openapi;\n }\n\n public void setOpenapi(String openapi) {\n this.openapi = openapi;\n }\n\n public Info getInfo() {\n return info;\n }\n\n public void setInfo(Info info) {\n this.info = info;\n }\n\n public Map getPaths() {\n return paths;\n }\n\n public void setPaths(Map paths) {\n this.paths = paths;\n }\n\n public void addPath(String url, Path path) {\n if (paths == null) {\n paths = new LinkedHashMap<>();\n }\n paths.put(url, path);\n }\n}\n" }, { "path": "frontend/server/src/main/java/com/amazonaws/ml/mms/openapi/OpenApiUtils.java", "content": "/*\n * Copyright 2018 Amazon.com, Inc. or its affiliates. All Rights Reserved.\n *\n * Licensed under the Apache License, Version 2.0 (the \"License\"). You may not use this file except in compliance\n * with the License. A copy of the License is located at\n *\n * http://aws.amazon.com/apache2.0/\n *\n * or in the \"license\" file accompanying this file. This file is distributed on an \"AS IS\" BASIS, WITHOUT WARRANTIES\n * OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions\n * and limitations under the License.\n */\npackage com.amazonaws.ml.mms.openapi;\n\nimport com.amazonaws.ml.mms.archive.Manifest;\nimport com.amazonaws.ml.mms.util.ConnectorType;\nimport com.amazonaws.ml.mms.util.JsonUtils;\nimport com.amazonaws.ml.mms.wlm.Model;\nimport io.netty.handler.codec.http.HttpHeaderValues;\nimport java.util.ArrayList;\nimport java.util.List;\n\npublic final class OpenApiUtils {\n\n private OpenApiUtils() {}\n\n public static String listApis(ConnectorType type) {\n OpenApi openApi = new OpenApi();\n Info info = new Info();\n info.setTitle(\"Model Server APIs\");\n info.setDescription(\n \"Model Server is a flexible and easy to use tool for serving deep learning models\");\n info.setVersion(\"1.0.0\");\n openApi.setInfo(info);\n\n if (ConnectorType.BOTH.equals(type) || ConnectorType.INFERENCE_CONNECTOR.equals(type)) {\n listInferenceApis(openApi);\n }\n if (ConnectorType.BOTH.equals(type) || ConnectorType.MANAGEMENT_CONNECTOR.equals(type)) {\n listManagementApis(openApi);\n }\n return JsonUtils.GSON_PRETTY.toJson(openApi);\n }\n\n static void listInferenceApis(OpenApi openApi) {\n openApi.addPath(\"/\", getApiDescriptionPath(false));\n openApi.addPath(\"/{model_name}/predict\", getLegacyPredictPath());\n openApi.addPath(\"/ping\", getPingPath());\n openApi.addPath(\"/predictions/{model_name}\", getPredictionsPath());\n openApi.addPath(\"/api-description\", getApiDescriptionPath(true));\n openApi.addPath(\"/invocations\", getInvocationsPath());\n openApi.addPath(\"/models/{model_name}/invoke\", getInvocationsPath());\n }\n\n static void listManagementApis(OpenApi openApi) {\n openApi.addPath(\"/\", getApiDescriptionPath(false));\n openApi.addPath(\"/models\", getModelsPath());\n openApi.addPath(\"/models/{model_name}\", getModelManagerPath());\n }\n\n public static String getModelApi(Model model) {\n String modelName = model.getModelName();\n OpenApi openApi = new OpenApi();\n Info info = new Info();\n info.setTitle(\"RESTful API for: \" + modelName);\n info.setVersion(\"1.0.0\");\n openApi.setInfo(info);\n\n openApi.addPath(\"/prediction/\" + modelName, getModelPath(modelName));\n\n return JsonUtils.GSON_PRETTY.toJson(openApi);\n }\n\n private static Path getApiDescriptionPath(boolean legacy) {\n Schema schema = new Schema(\"object\");\n schema.addProperty(\"openapi\", new Schema(\"string\"), true);\n schema.addProperty(\"info\", new Schema(\"object\"), true);\n schema.addProperty(\"paths\", new Schema(\"object\"), true);\n MediaType mediaType = new MediaType(HttpHeaderValues.APPLICATION_JSON.toString(), schema);\n\n Operation operation = new Operation(\"apiDescription\");\n operation.addResponse(new Response(\"200\", \"A openapi 3.0.1 descriptor\", mediaType));\n operation.addResponse(new Response(\"500\", \"Internal Server Error\", getErrorResponse()));\n\n Path path = new Path();\n if (legacy) {\n operation.setDeprecated(true);\n path.setGet(operation);\n } else {\n path.setOptions(operation);\n }\n return path;\n }\n\n private static Path getPingPath() {\n Schema schema = new Schema(\"object\");\n schema.addProperty(\n \"status\", new Schema(\"string\", \"Overall status of the Model Server.\"), true);\n MediaType mediaType = new MediaType(HttpHeaderValues.APPLICATION_JSON.toString(), schema);\n\n Operation operation = new Operation(\"ping\");\n operation.addResponse(new Response(\"200\", \"Model server status\", mediaType));\n operation.addResponse(new Response(\"500\", \"Internal Server Error\", getErrorResponse()));\n\n Path path = new Path();\n path.setGet(operation);\n return path;\n }\n\n private static Path getInvocationsPath() {\n Schema schema = new Schema();\n schema.addProperty(\"model_name\", new Schema(\"string\", \"Name of model\"), false);\n\n Schema dataProp = new Schema(\"string\", \"Inference input data\");\n dataProp.setFormat(\"binary\");\n schema.addProperty(\"data\", dataProp, true);\n\n MediaType multipart =\n new MediaType(HttpHeaderValues.MULTIPART_FORM_DATA.toString(), schema);\n\n RequestBody requestBody = new RequestBody();\n requestBody.setRequired(true);\n requestBody.addContent(multipart);\n\n Operation operation =\n new Operation(\"invocations\", \"A generic invocation entry point for all models.\");\n operation.setRequestBody(requestBody);\n operation.addParameter(new QueryParameter(\"model_name\", \"Name of model\"));\n\n MediaType error = getErrorResponse();\n MediaType mediaType = new MediaType(\"*/*\", schema);\n operation.addResponse(new Response(\"200\", \"Model specific output data format\", mediaType));\n operation.addResponse(new Response(\"400\", \"Missing model_name parameter\", error));\n operation.addResponse(new Response(\"404\", \"Model not found\", error));\n operation.addResponse(new Response(\"500\", \"Internal Server Error\", error));\n operation.addResponse(\n new Response(\"503\", \"No worker is available to serve request\", error));\n\n Path path = new Path();\n path.setPost(operation);\n return path;\n }\n\n private static Path getPredictionsPath() {\n Operation post =\n new Operation(\n \"predictions\",\n \"Predictions entry point for each model.\"\n + \" Use OPTIONS method to get detailed model API input and output description.\");\n post.addParameter(new PathParameter(\"model_name\", \"Name of model.\"));\n\n Schema schema = new Schema(\"string\");\n schema.setFormat(\"binary\");\n MediaType mediaType = new MediaType(\"*/*\", schema);\n RequestBody requestBody = new RequestBody();\n requestBody.setDescription(\n \"Input data format is defined by each model. Use OPTIONS method to get details for model input format.\");\n requestBody.setRequired(true);\n requestBody.addContent(mediaType);\n\n post.setRequestBody(requestBody);\n\n schema = new Schema(\"string\");\n schema.setFormat(\"binary\");\n mediaType = new MediaType(\"*/*\", schema);\n\n Response resp =\n new Response(\n \"200\",\n \"Output data format is defined by each model. Use OPTIONS method to get details for model output and output format.\",\n mediaType);\n post.addResponse(resp);\n\n MediaType error = getErrorResponse();\n post.addResponse(new Response(\"404\", \"Model not found\", error));\n post.addResponse(new Response(\"500\", \"Internal Server Error\", error));\n post.addResponse(new Response(\"503\", \"No worker is available to serve request\", error));\n\n Operation options =\n new Operation(\"predictionsApi\", \"Display details of per model input and output.\");\n options.addParameter(new PathParameter(\"model_name\", \"Name of model.\"));\n\n mediaType = new MediaType(\"application/json\", new Schema(\"object\"));\n options.addResponse(new Response(\"200\", \"OK\", mediaType));\n post.addResponse(new Response(\"500\", \"Internal Server Error\", error));\n\n Path path = new Path();\n path.setPost(post);\n path.setOptions(options);\n return path;\n }\n\n private static Path getLegacyPredictPath() {\n Operation operation =\n new Operation(\"predict\", \"A legacy predict entry point for each model.\");\n operation.addParameter(new PathParameter(\"model_name\", \"Name of model to unregister.\"));\n\n Schema schema = new Schema(\"string\");\n schema.setFormat(\"binary\");\n MediaType mediaType = new MediaType(\"*/*\", schema);\n RequestBody requestBody = new RequestBody();\n requestBody.setRequired(true);\n requestBody.setDescription(\"Input data format is defined by each model.\");\n requestBody.addContent(mediaType);\n\n operation.setRequestBody(requestBody);\n\n schema = new Schema(\"string\");\n schema.setFormat(\"binary\");\n\n mediaType = new MediaType(\"*/*\", schema);\n MediaType error = getErrorResponse();\n operation.addResponse(new Response(\"200\", \"Model specific output data format\", mediaType));\n operation.addResponse(new Response(\"404\", \"Model not found\", error));\n operation.addResponse(new Response(\"500\", \"Internal Server Error\", error));\n operation.addResponse(\n new Response(\"503\", \"No worker is available to serve request\", error));\n operation.setDeprecated(true);\n\n Path path = new Path();\n path.setPost(operation);\n return path;\n }\n\n private static Path getModelsPath() {\n Path path = new Path();\n path.setGet(getListModelsOperation());\n path.setPost(getRegisterOperation());\n return path;\n }\n\n private static Path getModelManagerPath() {\n Path path = new Path();\n path.setGet(getDescribeModelOperation());\n path.setPut(getScaleOperation());\n path.setDelete(getUnRegisterOperation());\n return path;\n }\n\n private static Operation getListModelsOperation() {\n Operation operation =\n new Operation(\"listModels\", \"List registered models in Model Server.\");\n\n operation.addParameter(\n new QueryParameter(\n \"limit\",\n \"integer\",\n \"100\",\n \"Use this parameter to specify the maximum number of items to return. When\"\n + \" this value is present, Model Server does not return more than the specified\"\n + \" number of items, but it might return fewer. This value is optional. If you\"\n + \" include a value, it must be between 1 and 1000, inclusive. If you do not\"\n + \" include a value, it defaults to 100.\"));\n operation.addParameter(\n new QueryParameter(\n \"next_page_token\",\n \"The token to retrieve the next set of results. Model Server provides the\"\n + \" token when the response from a previous call has more results than the\"\n + \" maximum page size.\"));\n operation.addParameter(\n new QueryParameter(\n \"model_name_pattern\", \"A model name filter to list only matching models.\"));\n\n Schema schema = new Schema(\"object\");\n schema.addProperty(\n \"nextPageToken\",\n new Schema(\n \"string\",\n \"Use this parameter in a subsequent request after you receive a response\"\n + \" with truncated results. Set it to the value of NextMarker from the\"\n + \" truncated response you just received.\"),\n false);\n\n Schema modelProp = new Schema(\"object\");\n modelProp.addProperty(\"modelName\", new Schema(\"string\", \"Name of the model.\"), true);\n modelProp.addProperty(\"modelUrl\", new Schema(\"string\", \"URL of the model.\"), true);\n Schema modelsProp = new Schema(\"array\", \"A list of registered models.\");\n modelsProp.setItems(modelProp);\n schema.addProperty(\"models\", modelsProp, true);\n MediaType json = new MediaType(HttpHeaderValues.APPLICATION_JSON.toString(), schema);\n\n operation.addResponse(new Response(\"200\", \"OK\", json));\n operation.addResponse(new Response(\"500\", \"Internal Server Error\", getErrorResponse()));\n return operation;\n }\n\n private static Operation getRegisterOperation() {\n Operation operation =\n new Operation(\"registerModel\", \"Register a new model in Model Server.\");\n\n operation.addParameter(\n new QueryParameter(\n \"model_url\",\n \"string\",\n null,\n true,\n \"Model archive download url, support local file or HTTP(s) protocol.\"\n + \" For S3, consider use pre-signed url.\"));\n operation.addParameter(\n new QueryParameter(\n \"model_name\",\n \"Name of model. This value will override modelName in MANIFEST.json if present.\"));\n operation.addParameter(\n new QueryParameter(\n \"handler\",\n \"Inference handler entry-point. This value will override handler in MANIFEST.json if present.\"));\n\n Parameter runtime =\n new QueryParameter(\n \"runtime\",\n \"Runtime for the model custom service code. This value will override runtime in MANIFEST.json if present.\");\n operation.addParameter(runtime);\n operation.addParameter(\n new QueryParameter(\n \"batch_size\", \"integer\", \"1\", \"Inference batch size, default: 1.\"));\n operation.addParameter(\n new QueryParameter(\n \"max_batch_delay\",\n \"integer\",\n \"100\",\n \"Maximum delay for batch aggregation, default: 100.\"));\n operation.addParameter(\n new QueryParameter(\n \"response_timeout\",\n \"integer\",\n \"2\",\n \"Maximum time, in seconds, the Model Server waits for a response from the model inference code, default: 120.\"));\n operation.addParameter(\n new QueryParameter(\n \"initial_workers\",\n \"integer\",\n \"0\",\n \"Number of initial workers, default: 0.\"));\n operation.addParameter(\n new QueryParameter(\n \"synchronous\",\n \"boolean\",\n \"false\",\n \"Decides whether creation of worker synchronous or not, default: false.\"));\n operation.addParameter(\n new QueryParameter(\n \"preload_model\",\n \"boolean\",\n \"false\",\n \"Decides if model should be preloaded, default: false.\"));\n\n Manifest.RuntimeType[] types = Manifest.RuntimeType.values();\n List runtimeTypes = new ArrayList<>(types.length);\n for (Manifest.RuntimeType type : types) {\n runtimeTypes.add(type.toString());\n }\n runtime.getSchema().setEnumeration(runtimeTypes);\n\n MediaType status = getStatusResponse();\n MediaType error = getErrorResponse();\n\n operation.addResponse(new Response(\"200\", \"Model registered\", status));\n operation.addResponse(new Response(\"202\", \"Accepted\", status));\n operation.addResponse(new Response(\"210\", \"Partial Success\", status));\n operation.addResponse(new Response(\"400\", \"Bad request\", error));\n operation.addResponse(new Response(\"404\", \"Model not found\", error));\n operation.addResponse(new Response(\"409\", \"Model already registered\", error));\n operation.addResponse(new Response(\"500\", \"Internal Server Error\", error));\n\n return operation;\n }\n\n private static Operation getUnRegisterOperation() {\n Operation operation =\n new Operation(\n \"unregisterModel\",\n \"Unregister a model from Model Server. This is an asynchronous call by default.\"\n + \" Caller can call listModels to confirm if all the works has be terminated.\");\n\n operation.addParameter(new PathParameter(\"model_name\", \"Name of model to unregister.\"));\n operation.addParameter(\n new QueryParameter(\n \"synchronous\",\n \"boolean\",\n \"false\",\n \"Decides whether the call is synchronous or not, default: false.\"));\n operation.addParameter(\n new QueryParameter(\n \"timeout\",\n \"integer\",\n \"-1\",\n \"Waiting up to the specified wait time if necessary for\"\n + \" a worker to complete all pending requests. Use 0 to terminate backend\"\n + \" worker process immediately. Use -1 for wait infinitely.\"));\n\n MediaType status = getStatusResponse();\n MediaType error = getErrorResponse();\n\n operation.addResponse(new Response(\"200\", \"Model unregistered\", status));\n operation.addResponse(new Response(\"202\", \"Accepted\", status));\n operation.addResponse(new Response(\"404\", \"Model not found\", error));\n operation.addResponse(new Response(\"408\", \"Request Timeout Error\", error));\n operation.addResponse(new Response(\"500\", \"Internal Server Error\", error));\n\n return operation;\n }\n\n private static Operation getDescribeModelOperation() {\n Operation operation =\n new Operation(\n \"describeModel\",\n \"Provides detailed information about the specified model.\");\n\n operation.addParameter(new PathParameter(\"model_name\", \"Name of model to describe.\"));\n\n Schema schema = new Schema(\"object\");\n schema.addProperty(\"modelName\", new Schema(\"string\", \"Name of the model.\"), true);\n schema.addProperty(\"modelVersion\", new Schema(\"string\", \"Version of the model.\"), true);\n schema.addProperty(\"modelUrl\", new Schema(\"string\", \"URL of the model.\"), true);\n schema.addProperty(\n \"minWorkers\", new Schema(\"integer\", \"Configured minimum number of worker.\"), true);\n schema.addProperty(\n \"maxWorkers\", new Schema(\"integer\", \"Configured maximum number of worker.\"), true);\n schema.addProperty(\"batchSize\", new Schema(\"integer\", \"Configured batch size.\"), false);\n schema.addProperty(\n \"maxBatchDelay\",\n new Schema(\"integer\", \"Configured maximum batch delay in ms.\"),\n false);\n schema.addProperty(\n \"status\", new Schema(\"string\", \"Overall health status of the model\"), true);\n\n Schema workers = new Schema(\"array\", \"A list of active backend workers.\");\n Schema worker = new Schema(\"object\");\n worker.addProperty(\"id\", new Schema(\"string\", \"Worker id\"), true);\n worker.addProperty(\"startTime\", new Schema(\"string\", \"Worker start time\"), true);\n worker.addProperty(\"gpu\", new Schema(\"boolean\", \"If running on GPU\"), false);\n Schema workerStatus = new Schema(\"string\", \"Worker status\");\n List status = new ArrayList<>();\n status.add(\"READY\");\n status.add(\"LOADING\");\n status.add(\"UNLOADING\");\n workerStatus.setEnumeration(status);\n worker.addProperty(\"status\", workerStatus, true);\n workers.setItems(worker);\n\n schema.addProperty(\"workers\", workers, true);\n Schema metrics = new Schema(\"object\");\n metrics.addProperty(\n \"rejectedRequests\",\n new Schema(\"integer\", \"Number requests has been rejected in last 10 minutes.\"),\n true);\n metrics.addProperty(\n \"waitingQueueSize\",\n new Schema(\"integer\", \"Number requests waiting in the queue.\"),\n true);\n metrics.addProperty(\n \"requests\",\n new Schema(\"integer\", \"Number requests processed in last 10 minutes.\"),\n true);\n schema.addProperty(\"metrics\", metrics, true);\n\n MediaType mediaType = new MediaType(HttpHeaderValues.APPLICATION_JSON.toString(), schema);\n\n operation.addResponse(new Response(\"200\", \"OK\", mediaType));\n operation.addResponse(new Response(\"500\", \"Internal Server Error\", getErrorResponse()));\n\n return operation;\n }\n\n private static Operation getScaleOperation() {\n Operation operation =\n new Operation(\n \"setAutoScale\",\n \"Configure number of workers for a model, This is a asynchronous call by default.\"\n + \" Caller need to call describeModel check if the model workers has been changed.\");\n operation.addParameter(new PathParameter(\"model_name\", \"Name of model to describe.\"));\n operation.addParameter(\n new QueryParameter(\n \"min_worker\", \"integer\", \"1\", \"Minimum number of worker processes.\"));\n operation.addParameter(\n new QueryParameter(\n \"max_worker\", \"integer\", \"1\", \"Maximum number of worker processes.\"));\n operation.addParameter(\n new QueryParameter(\n \"number_gpu\", \"integer\", \"0\", \"Number of GPU worker processes to create.\"));\n operation.addParameter(\n new QueryParameter(\n \"synchronous\",\n \"boolean\",\n \"false\",\n \"Decides whether the call is synchronous or not, default: false.\"));\n operation.addParameter(\n new QueryParameter(\n \"timeout\",\n \"integer\",\n \"-1\",\n \"Waiting up to the specified wait time if necessary for\"\n + \" a worker to complete all pending requests. Use 0 to terminate backend\"\n + \" worker process immediately. Use -1 for wait infinitely.\"));\n\n MediaType status = getStatusResponse();\n MediaType error = getErrorResponse();\n\n operation.addResponse(new Response(\"200\", \"Model workers updated\", status));\n operation.addResponse(new Response(\"202\", \"Accepted\", status));\n operation.addResponse(new Response(\"210\", \"Partial Success\", status));\n operation.addResponse(new Response(\"400\", \"Bad request\", error));\n operation.addResponse(new Response(\"404\", \"Model not found\", error));\n operation.addResponse(new Response(\"500\", \"Internal Server Error\", error));\n\n return operation;\n }\n\n private static Path getModelPath(String modelName) {\n Operation operation =\n new Operation(modelName, \"A predict entry point for model: \" + modelName + '.');\n operation.addResponse(new Response(\"200\", \"OK\"));\n operation.addResponse(new Response(\"500\", \"Internal Server Error\", getErrorResponse()));\n\n Path path = new Path();\n path.setPost(operation);\n return path;\n }\n\n private static MediaType getErrorResponse() {\n Schema schema = new Schema(\"object\");\n schema.addProperty(\"code\", new Schema(\"integer\", \"Error code.\"), true);\n schema.addProperty(\"type\", new Schema(\"string\", \"Error type.\"), true);\n schema.addProperty(\"message\", new Schema(\"string\", \"Error message.\"), true);\n\n return new MediaType(HttpHeaderValues.APPLICATION_JSON.toString(), schema);\n }\n\n private static MediaType getStatusResponse() {\n Schema schema = new Schema(\"object\");\n schema.addProperty(\"status\", new Schema(\"string\", \"Error type.\"), true);\n return new MediaType(HttpHeaderValues.APPLICATION_JSON.toString(), schema);\n }\n}\n" }, { "path": "frontend/server/src/main/java/com/amazonaws/ml/mms/openapi/Operation.java", "content": "/*\n * Copyright 2018 Amazon.com, Inc. or its affiliates. All Rights Reserved.\n *\n * Licensed under the Apache License, Version 2.0 (the \"License\"). You may not use this file except in compliance\n * with the License. A copy of the License is located at\n *\n * http://aws.amazon.com/apache2.0/\n *\n * or in the \"license\" file accompanying this file. This file is distributed on an \"AS IS\" BASIS, WITHOUT WARRANTIES\n * OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions\n * and limitations under the License.\n */\npackage com.amazonaws.ml.mms.openapi;\n\nimport java.util.ArrayList;\nimport java.util.LinkedHashMap;\nimport java.util.List;\nimport java.util.Map;\n\npublic class Operation {\n\n private String summary;\n private String description;\n private String operationId;\n private List parameters = new ArrayList<>();\n private RequestBody requestBody;\n private Map responses;\n private Boolean deprecated;\n\n public Operation() {}\n\n public Operation(String operationId) {\n this(operationId, null);\n }\n\n public Operation(String operationId, String description) {\n this.operationId = operationId;\n this.description = description;\n }\n\n public String getSummary() {\n return summary;\n }\n\n public void setSummary(String summary) {\n this.summary = summary;\n }\n\n public String getDescription() {\n return description;\n }\n\n public void setDescription(String description) {\n this.description = description;\n }\n\n public String getOperationId() {\n return operationId;\n }\n\n public void setOperationId(String operationId) {\n this.operationId = operationId;\n }\n\n public List getParameters() {\n return parameters;\n }\n\n public void setParameters(List parameters) {\n this.parameters = parameters;\n }\n\n public void addParameter(Parameter parameter) {\n if (parameters == null) {\n parameters = new ArrayList<>();\n }\n parameters.add(parameter);\n }\n\n public RequestBody getRequestBody() {\n return requestBody;\n }\n\n public void setRequestBody(RequestBody requestBody) {\n this.requestBody = requestBody;\n }\n\n public Map getResponses() {\n return responses;\n }\n\n public void setResponses(Map responses) {\n this.responses = responses;\n }\n\n public void addResponse(Response response) {\n if (responses == null) {\n responses = new LinkedHashMap<>();\n }\n responses.put(response.getCode(), response);\n }\n\n public Boolean getDeprecated() {\n return deprecated;\n }\n\n public void setDeprecated(Boolean deprecated) {\n this.deprecated = deprecated;\n }\n}\n" }, { "path": "frontend/server/src/main/java/com/amazonaws/ml/mms/openapi/Parameter.java", "content": "/*\n * Copyright 2018 Amazon.com, Inc. or its affiliates. All Rights Reserved.\n *\n * Licensed under the Apache License, Version 2.0 (the \"License\"). You may not use this file except in compliance\n * with the License. A copy of the License is located at\n *\n * http://aws.amazon.com/apache2.0/\n *\n * or in the \"license\" file accompanying this file. This file is distributed on an \"AS IS\" BASIS, WITHOUT WARRANTIES\n * OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions\n * and limitations under the License.\n */\npackage com.amazonaws.ml.mms.openapi;\n\n@SuppressWarnings(\"PMD.AbstractClassWithoutAbstractMethod\")\npublic abstract class Parameter {\n\n protected String type;\n protected String in;\n protected String name;\n protected String description;\n protected boolean required;\n protected Boolean deprecated;\n protected Boolean allowEmptyValue;\n protected String style;\n protected Boolean explode;\n protected Schema schema;\n\n public void setType(String type) {\n this.type = type;\n }\n\n public String getType() {\n return type;\n }\n\n public String getName() {\n return name;\n }\n\n public void setName(String name) {\n this.name = name;\n }\n\n public String getIn() {\n return in;\n }\n\n public void setIn(String in) {\n this.in = in;\n }\n\n public String getDescription() {\n return description;\n }\n\n public void setDescription(String description) {\n this.description = description;\n }\n\n public boolean isRequired() {\n return required;\n }\n\n public void setRequired(boolean required) {\n this.required = required;\n }\n\n public Boolean getDeprecated() {\n return deprecated;\n }\n\n public void setDeprecated(Boolean deprecated) {\n this.deprecated = deprecated;\n }\n\n public Boolean getAllowEmptyValue() {\n return allowEmptyValue;\n }\n\n public void setAllowEmptyValue(Boolean allowEmptyValue) {\n this.allowEmptyValue = allowEmptyValue;\n }\n\n public String getStyle() {\n return style;\n }\n\n public void setStyle(String style) {\n this.style = style;\n }\n\n public Boolean getExplode() {\n return explode;\n }\n\n public void setExplode(Boolean explode) {\n this.explode = explode;\n }\n\n public Schema getSchema() {\n return schema;\n }\n\n public void setSchema(Schema schema) {\n this.schema = schema;\n }\n}\n" }, { "path": "frontend/server/src/main/java/com/amazonaws/ml/mms/openapi/Path.java", "content": "/*\n * Copyright 2018 Amazon.com, Inc. or its affiliates. All Rights Reserved.\n *\n * Licensed under the Apache License, Version 2.0 (the \"License\"). You may not use this file except in compliance\n * with the License. A copy of the License is located at\n *\n * http://aws.amazon.com/apache2.0/\n *\n * or in the \"license\" file accompanying this file. This file is distributed on an \"AS IS\" BASIS, WITHOUT WARRANTIES\n * OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions\n * and limitations under the License.\n */\npackage com.amazonaws.ml.mms.openapi;\n\nimport java.util.List;\n\npublic class Path {\n\n private Operation get;\n private Operation put;\n private Operation post;\n private Operation head;\n private Operation delete;\n private Operation patch;\n private Operation options;\n private List parameters;\n\n public Operation getGet() {\n return get;\n }\n\n public void setGet(Operation get) {\n this.get = get;\n }\n\n public Operation getPut() {\n return put;\n }\n\n public void setPut(Operation put) {\n this.put = put;\n }\n\n public Operation getPost() {\n return post;\n }\n\n public void setPost(Operation post) {\n this.post = post;\n }\n\n public Operation getHead() {\n return head;\n }\n\n public void setHead(Operation head) {\n this.head = head;\n }\n\n public Operation getDelete() {\n return delete;\n }\n\n public void setDelete(Operation delete) {\n this.delete = delete;\n }\n\n public Operation getPatch() {\n return patch;\n }\n\n public void setPatch(Operation patch) {\n this.patch = patch;\n }\n\n public Operation getOptions() {\n return options;\n }\n\n public void setOptions(Operation options) {\n this.options = options;\n }\n\n public List getParameters() {\n return parameters;\n }\n\n public void setParameters(List parameters) {\n this.parameters = parameters;\n }\n}\n" }, { "path": "frontend/server/src/main/java/com/amazonaws/ml/mms/openapi/PathParameter.java", "content": "/*\n * Copyright 2018 Amazon.com, Inc. or its affiliates. All Rights Reserved.\n *\n * Licensed under the Apache License, Version 2.0 (the \"License\"). You may not use this file except in compliance\n * with the License. A copy of the License is located at\n *\n * http://aws.amazon.com/apache2.0/\n *\n * or in the \"license\" file accompanying this file. This file is distributed on an \"AS IS\" BASIS, WITHOUT WARRANTIES\n * OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions\n * and limitations under the License.\n */\npackage com.amazonaws.ml.mms.openapi;\n\npublic class PathParameter extends Parameter {\n\n public PathParameter() {\n this(null, \"string\", null, null);\n }\n\n public PathParameter(String name, String description) {\n this(name, \"string\", null, description);\n }\n\n public PathParameter(String name, String type, String defaultValue, String description) {\n this.name = name;\n this.description = description;\n in = \"path\";\n required = true;\n schema = new Schema(type, null, defaultValue);\n }\n}\n" }, { "path": "frontend/server/src/main/java/com/amazonaws/ml/mms/openapi/QueryParameter.java", "content": "/*\n * Copyright 2018 Amazon.com, Inc. or its affiliates. All Rights Reserved.\n *\n * Licensed under the Apache License, Version 2.0 (the \"License\"). You may not use this file except in compliance\n * with the License. A copy of the License is located at\n *\n * http://aws.amazon.com/apache2.0/\n *\n * or in the \"license\" file accompanying this file. This file is distributed on an \"AS IS\" BASIS, WITHOUT WARRANTIES\n * OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions\n * and limitations under the License.\n */\npackage com.amazonaws.ml.mms.openapi;\n\npublic class QueryParameter extends Parameter {\n\n public QueryParameter() {\n this(null, \"string\", null, false, null);\n }\n\n public QueryParameter(String name, String description) {\n this(name, \"string\", null, false, description);\n }\n\n public QueryParameter(String name, String type, String description) {\n this(name, type, null, false, description);\n }\n\n public QueryParameter(String name, String type, String defaultValue, String description) {\n this(name, type, defaultValue, false, description);\n }\n\n public QueryParameter(\n String name, String type, String defaultValue, boolean required, String description) {\n this.name = name;\n this.description = description;\n in = \"query\";\n this.required = required;\n schema = new Schema(type, null, defaultValue);\n }\n}\n" }, { "path": "frontend/server/src/main/java/com/amazonaws/ml/mms/openapi/RequestBody.java", "content": "/*\n * Copyright 2018 Amazon.com, Inc. or its affiliates. All Rights Reserved.\n *\n * Licensed under the Apache License, Version 2.0 (the \"License\"). You may not use this file except in compliance\n * with the License. A copy of the License is located at\n *\n * http://aws.amazon.com/apache2.0/\n *\n * or in the \"license\" file accompanying this file. This file is distributed on an \"AS IS\" BASIS, WITHOUT WARRANTIES\n * OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions\n * and limitations under the License.\n */\npackage com.amazonaws.ml.mms.openapi;\n\nimport java.util.LinkedHashMap;\nimport java.util.Map;\n\npublic class RequestBody {\n\n private String description;\n private Map content;\n private boolean required;\n\n public RequestBody() {}\n\n public String getDescription() {\n return description;\n }\n\n public void setDescription(String description) {\n this.description = description;\n }\n\n public Map getContent() {\n return content;\n }\n\n public void setContent(Map content) {\n this.content = content;\n }\n\n public void addContent(MediaType mediaType) {\n if (content == null) {\n content = new LinkedHashMap<>();\n }\n content.put(mediaType.getContentType(), mediaType);\n }\n\n public boolean isRequired() {\n return required;\n }\n\n public void setRequired(boolean required) {\n this.required = required;\n }\n}\n" }, { "path": "frontend/server/src/main/java/com/amazonaws/ml/mms/openapi/Response.java", "content": "/*\n * Copyright 2018 Amazon.com, Inc. or its affiliates. All Rights Reserved.\n *\n * Licensed under the Apache License, Version 2.0 (the \"License\"). You may not use this file except in compliance\n * with the License. A copy of the License is located at\n *\n * http://aws.amazon.com/apache2.0/\n *\n * or in the \"license\" file accompanying this file. This file is distributed on an \"AS IS\" BASIS, WITHOUT WARRANTIES\n * OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions\n * and limitations under the License.\n */\npackage com.amazonaws.ml.mms.openapi;\n\nimport java.util.LinkedHashMap;\nimport java.util.Map;\n\npublic class Response {\n\n private transient String code;\n private String description;\n private Map content;\n\n public Response() {}\n\n public Response(String code, String description) {\n this.code = code;\n this.description = description;\n }\n\n public Response(String code, String description, MediaType mediaType) {\n this.code = code;\n this.description = description;\n content = new LinkedHashMap<>();\n content.put(mediaType.getContentType(), mediaType);\n }\n\n public String getCode() {\n return code;\n }\n\n public String getDescription() {\n return description;\n }\n\n public void setDescription(String description) {\n this.description = description;\n }\n\n public Map getContent() {\n return content;\n }\n\n public void setContent(Map content) {\n this.content = content;\n }\n\n public void addContent(MediaType mediaType) {\n if (content == null) {\n content = new LinkedHashMap<>();\n }\n content.put(mediaType.getContentType(), mediaType);\n }\n}\n" }, { "path": "frontend/server/src/main/java/com/amazonaws/ml/mms/openapi/Schema.java", "content": "/*\n * Copyright 2018 Amazon.com, Inc. or its affiliates. All Rights Reserved.\n *\n * Licensed under the Apache License, Version 2.0 (the \"License\"). You may not use this file except in compliance\n * with the License. A copy of the License is located at\n *\n * http://aws.amazon.com/apache2.0/\n *\n * or in the \"license\" file accompanying this file. This file is distributed on an \"AS IS\" BASIS, WITHOUT WARRANTIES\n * OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions\n * and limitations under the License.\n */\npackage com.amazonaws.ml.mms.openapi;\n\nimport com.google.gson.annotations.SerializedName;\nimport java.util.ArrayList;\nimport java.util.LinkedHashMap;\nimport java.util.List;\nimport java.util.Map;\n\npublic class Schema {\n\n private String type;\n private String format;\n private String name;\n private List required;\n private Map properties;\n private Schema items;\n private String description;\n private Object example;\n private Schema additionalProperties;\n private String discriminator;\n\n @SerializedName(\"enum\")\n private List enumeration;\n\n @SerializedName(\"default\")\n private String defaultValue;\n\n public Schema() {}\n\n public Schema(String type) {\n this(type, null, null);\n }\n\n public Schema(String type, String description) {\n this(type, description, null);\n }\n\n public Schema(String type, String description, String defaultValue) {\n this.type = type;\n this.description = description;\n this.defaultValue = defaultValue;\n }\n\n public String getType() {\n return type;\n }\n\n public void setType(String type) {\n this.type = type;\n }\n\n public String getFormat() {\n return format;\n }\n\n public void setFormat(String format) {\n this.format = format;\n }\n\n public String getName() {\n return name;\n }\n\n public void setName(String name) {\n this.name = name;\n }\n\n public List getRequired() {\n return required;\n }\n\n public void setRequired(List required) {\n this.required = required;\n }\n\n public Map getProperties() {\n return properties;\n }\n\n public void setProperties(Map properties) {\n this.properties = properties;\n }\n\n public void addProperty(String key, Schema schema, boolean requiredProperty) {\n if (properties == null) {\n properties = new LinkedHashMap<>();\n }\n properties.put(key, schema);\n if (requiredProperty) {\n if (required == null) {\n required = new ArrayList<>();\n }\n required.add(key);\n }\n }\n\n public Schema getItems() {\n return items;\n }\n\n public void setItems(Schema items) {\n this.items = items;\n }\n\n public String getDescription() {\n return description;\n }\n\n public void setDescription(String description) {\n this.description = description;\n }\n\n public Object getExample() {\n return example;\n }\n\n public void setExample(Object example) {\n this.example = example;\n }\n\n public Schema getAdditionalProperties() {\n return additionalProperties;\n }\n\n public void setAdditionalProperties(Schema additionalProperties) {\n this.additionalProperties = additionalProperties;\n }\n\n public String getDiscriminator() {\n return discriminator;\n }\n\n public void setDiscriminator(String discriminator) {\n this.discriminator = discriminator;\n }\n\n public List getEnumeration() {\n return enumeration;\n }\n\n public void setEnumeration(List enumeration) {\n this.enumeration = enumeration;\n }\n\n public String getDefaultValue() {\n return defaultValue;\n }\n\n public void setDefaultValue(String defaultValue) {\n this.defaultValue = defaultValue;\n }\n}\n" }, { "path": "frontend/server/src/main/java/com/amazonaws/ml/mms/servingsdk/impl/ModelServerContext.java", "content": "/*\n * Copyright 2019 Amazon.com, Inc. or its affiliates. All Rights Reserved.\n *\n * Licensed under the Apache License, Version 2.0 (the \"License\"). You may not use this file except in compliance\n * with the License. A copy of the License is located at\n *\n * http://aws.amazon.com/apache2.0/\n *\n * or in the \"license\" file accompanying this file. This file is distributed on an \"AS IS\" BASIS, WITHOUT WARRANTIES\n * OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions\n * and limitations under the License.\n */\n\npackage com.amazonaws.ml.mms.servingsdk.impl;\n\nimport com.amazonaws.ml.mms.util.ConfigManager;\nimport com.amazonaws.ml.mms.wlm.ModelManager;\nimport java.util.HashMap;\nimport java.util.Map;\nimport java.util.Properties;\nimport software.amazon.ai.mms.servingsdk.Context;\nimport software.amazon.ai.mms.servingsdk.Model;\n\npublic class ModelServerContext implements Context {\n @Override\n public Properties getConfig() {\n return ConfigManager.getInstance().getConfiguration();\n }\n\n @Override\n public Map getModels() {\n HashMap r = new HashMap<>();\n ModelManager.getInstance().getModels().forEach((k, v) -> r.put(k, new ModelServerModel(v)));\n return r;\n }\n}\n" }, { "path": "frontend/server/src/main/java/com/amazonaws/ml/mms/servingsdk/impl/ModelServerModel.java", "content": "/*\n * Copyright 2019 Amazon.com, Inc. or its affiliates. All Rights Reserved.\n *\n * Licensed under the Apache License, Version 2.0 (the \"License\"). You may not use this file except in compliance\n * with the License. A copy of the License is located at\n *\n * http://aws.amazon.com/apache2.0/\n *\n * or in the \"license\" file accompanying this file. This file is distributed on an \"AS IS\" BASIS, WITHOUT WARRANTIES\n * OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions\n * and limitations under the License.\n */\n\npackage com.amazonaws.ml.mms.servingsdk.impl;\n\nimport com.amazonaws.ml.mms.wlm.ModelManager;\nimport java.util.ArrayList;\nimport java.util.List;\nimport software.amazon.ai.mms.servingsdk.Model;\nimport software.amazon.ai.mms.servingsdk.Worker;\n\npublic class ModelServerModel implements Model {\n private final com.amazonaws.ml.mms.wlm.Model model;\n\n public ModelServerModel(com.amazonaws.ml.mms.wlm.Model m) {\n model = m;\n }\n\n @Override\n public String getModelName() {\n return model.getModelName();\n }\n\n @Override\n public String getModelUrl() {\n return model.getModelUrl();\n }\n\n @Override\n public String getModelHandler() {\n return model.getModelArchive().getHandler();\n }\n\n @Override\n public List getModelWorkers() {\n ArrayList list = new ArrayList<>();\n ModelManager.getInstance()\n .getWorkers(model.getModelName())\n .forEach(r -> list.add(new ModelWorker(r)));\n return list;\n }\n}\n" }, { "path": "frontend/server/src/main/java/com/amazonaws/ml/mms/servingsdk/impl/ModelServerRequest.java", "content": "/*\n * Copyright 2019 Amazon.com, Inc. or its affiliates. All Rights Reserved.\n *\n * Licensed under the Apache License, Version 2.0 (the \"License\"). You may not use this file except in compliance\n * with the License. A copy of the License is located at\n *\n * http://aws.amazon.com/apache2.0/\n *\n * or in the \"license\" file accompanying this file. This file is distributed on an \"AS IS\" BASIS, WITHOUT WARRANTIES\n * OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions\n * and limitations under the License.\n */\n\npackage com.amazonaws.ml.mms.servingsdk.impl;\n\nimport io.netty.handler.codec.http.FullHttpRequest;\nimport io.netty.handler.codec.http.HttpUtil;\nimport io.netty.handler.codec.http.QueryStringDecoder;\nimport java.io.ByteArrayInputStream;\nimport java.util.ArrayList;\nimport java.util.List;\nimport java.util.Map;\nimport software.amazon.ai.mms.servingsdk.http.Request;\n\npublic class ModelServerRequest implements Request {\n private FullHttpRequest req;\n private QueryStringDecoder decoder;\n\n public ModelServerRequest(FullHttpRequest r, QueryStringDecoder d) {\n req = r;\n decoder = d;\n }\n\n @Override\n public List getHeaderNames() {\n return new ArrayList<>(req.headers().names());\n }\n\n @Override\n public String getRequestURI() {\n return req.uri();\n }\n\n @Override\n public Map> getParameterMap() {\n return decoder.parameters();\n }\n\n @Override\n public List getParameter(String k) {\n return decoder.parameters().get(k);\n }\n\n @Override\n public String getContentType() {\n return HttpUtil.getMimeType(req).toString();\n }\n\n @Override\n public ByteArrayInputStream getInputStream() {\n return new ByteArrayInputStream(req.content().array());\n }\n}\n" }, { "path": "frontend/server/src/main/java/com/amazonaws/ml/mms/servingsdk/impl/ModelServerResponse.java", "content": "/*\n * Copyright 2019 Amazon.com, Inc. or its affiliates. All Rights Reserved.\n *\n * Licensed under the Apache License, Version 2.0 (the \"License\"). You may not use this file except in compliance\n * with the License. A copy of the License is located at\n *\n * http://aws.amazon.com/apache2.0/\n *\n * or in the \"license\" file accompanying this file. This file is distributed on an \"AS IS\" BASIS, WITHOUT WARRANTIES\n * OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions\n * and limitations under the License.\n */\n\npackage com.amazonaws.ml.mms.servingsdk.impl;\n\nimport io.netty.buffer.ByteBufOutputStream;\nimport io.netty.handler.codec.http.FullHttpResponse;\nimport io.netty.handler.codec.http.HttpHeaderNames;\nimport io.netty.handler.codec.http.HttpResponseStatus;\nimport java.io.OutputStream;\nimport software.amazon.ai.mms.servingsdk.http.Response;\n\npublic class ModelServerResponse implements Response {\n\n private FullHttpResponse response;\n\n public ModelServerResponse(FullHttpResponse rsp) {\n response = rsp;\n }\n\n @Override\n public void setStatus(int i) {\n response.setStatus(HttpResponseStatus.valueOf(i));\n }\n\n @Override\n public void setStatus(int i, String s) {\n response.setStatus(HttpResponseStatus.valueOf(i, s));\n }\n\n @Override\n public void setHeader(String k, String v) {\n response.headers().set(k, v);\n }\n\n @Override\n public void addHeader(String k, String v) {\n response.headers().add(k, v);\n }\n\n @Override\n public void setContentType(String contentType) {\n response.headers().set(HttpHeaderNames.CONTENT_TYPE, contentType);\n }\n\n @Override\n public OutputStream getOutputStream() {\n return new ByteBufOutputStream(response.content());\n }\n}\n" }, { "path": "frontend/server/src/main/java/com/amazonaws/ml/mms/servingsdk/impl/ModelWorker.java", "content": "/*\n * Copyright 2019 Amazon.com, Inc. or its affiliates. All Rights Reserved.\n *\n * Licensed under the Apache License, Version 2.0 (the \"License\"). You may not use this file except in compliance\n * with the License. A copy of the License is located at\n *\n * http://aws.amazon.com/apache2.0/\n *\n * or in the \"license\" file accompanying this file. This file is distributed on an \"AS IS\" BASIS, WITHOUT WARRANTIES\n * OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions\n * and limitations under the License.\n */\n\npackage com.amazonaws.ml.mms.servingsdk.impl;\n\nimport com.amazonaws.ml.mms.wlm.WorkerState;\nimport com.amazonaws.ml.mms.wlm.WorkerThread;\nimport software.amazon.ai.mms.servingsdk.Worker;\n\npublic class ModelWorker implements Worker {\n boolean running;\n long memory;\n\n public ModelWorker(WorkerThread t) {\n running = t.getState() == WorkerState.WORKER_MODEL_LOADED;\n memory = t.getMemory();\n }\n\n @Override\n public boolean isRunning() {\n return running;\n }\n\n @Override\n public long getWorkerMemory() {\n return memory;\n }\n}\n" }, { "path": "frontend/server/src/main/java/com/amazonaws/ml/mms/servingsdk/impl/PluginsManager.java", "content": "/*\n * Copyright 2019 Amazon.com, Inc. or its affiliates. All Rights Reserved.\n *\n * Licensed under the Apache License, Version 2.0 (the \"License\"). You may not use this file except in compliance\n * with the License. A copy of the License is located at\n *\n * http://aws.amazon.com/apache2.0/\n *\n * or in the \"license\" file accompanying this file. This file is distributed on an \"AS IS\" BASIS, WITHOUT WARRANTIES\n * OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions\n * and limitations under the License.\n */\n\npackage com.amazonaws.ml.mms.servingsdk.impl;\n\nimport com.amazonaws.ml.mms.http.InvalidPluginException;\nimport java.lang.annotation.Annotation;\nimport java.util.HashMap;\nimport java.util.Map;\nimport java.util.ServiceLoader;\nimport org.slf4j.Logger;\nimport org.slf4j.LoggerFactory;\nimport software.amazon.ai.mms.servingsdk.ModelServerEndpoint;\nimport software.amazon.ai.mms.servingsdk.annotations.Endpoint;\nimport software.amazon.ai.mms.servingsdk.annotations.helpers.EndpointTypes;\n\npublic final class PluginsManager {\n\n private static final PluginsManager INSTANCE = new PluginsManager();\n private Logger logger = LoggerFactory.getLogger(PluginsManager.class);\n\n private Map inferenceEndpoints;\n private Map managementEndpoints;\n\n private PluginsManager() {}\n\n public static PluginsManager getInstance() {\n return INSTANCE;\n }\n\n public void initialize() {\n inferenceEndpoints = initInferenceEndpoints();\n managementEndpoints = initManagementEndpoints();\n }\n\n private boolean validateEndpointPlugin(Annotation a, EndpointTypes type) {\n return a instanceof Endpoint\n && !((Endpoint) a).urlPattern().isEmpty()\n && ((Endpoint) a).endpointType().equals(type);\n }\n\n private HashMap getEndpoints(EndpointTypes type)\n throws InvalidPluginException {\n ServiceLoader loader = ServiceLoader.load(ModelServerEndpoint.class);\n HashMap ep = new HashMap<>();\n for (ModelServerEndpoint mep : loader) {\n Class modelServerEndpointClassObj = mep.getClass();\n Annotation[] annotations = modelServerEndpointClassObj.getAnnotations();\n for (Annotation a : annotations) {\n if (validateEndpointPlugin(a, type)) {\n if (ep.get(((Endpoint) a).urlPattern()) != null) {\n throw new InvalidPluginException(\n \"Multiple plugins found for endpoint \"\n + \"\\\"\"\n + ((Endpoint) a).urlPattern()\n + \"\\\"\");\n }\n logger.info(\"Loading plugin for endpoint {}\", ((Endpoint) a).urlPattern());\n ep.put(((Endpoint) a).urlPattern(), mep);\n }\n }\n }\n return ep;\n }\n\n private HashMap initInferenceEndpoints() {\n return getEndpoints(EndpointTypes.INFERENCE);\n }\n\n private HashMap initManagementEndpoints() {\n return getEndpoints(EndpointTypes.MANAGEMENT);\n }\n\n public Map getInferenceEndpoints() {\n return inferenceEndpoints;\n }\n\n public Map getManagementEndpoints() {\n return managementEndpoints;\n }\n}\n" }, { "path": "frontend/server/src/main/java/com/amazonaws/ml/mms/util/ConfigManager.java", "content": "/*\n * Copyright 2018 Amazon.com, Inc. or its affiliates. All Rights Reserved.\n *\n * Licensed under the Apache License, Version 2.0 (the \"License\"). You may not use this file except in compliance\n * with the License. A copy of the License is located at\n *\n * http://aws.amazon.com/apache2.0/\n *\n * or in the \"license\" file accompanying this file. This file is distributed on an \"AS IS\" BASIS, WITHOUT WARRANTIES\n * OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions\n * and limitations under the License.\n */\npackage com.amazonaws.ml.mms.util;\n\nimport io.netty.handler.ssl.SslContext;\nimport io.netty.handler.ssl.SslContextBuilder;\nimport io.netty.handler.ssl.util.SelfSignedCertificate;\nimport java.io.File;\nimport java.io.FileInputStream;\nimport java.io.IOException;\nimport java.io.InputStream;\nimport java.lang.reflect.Field;\nimport java.net.InetAddress;\nimport java.net.UnknownHostException;\nimport java.nio.charset.StandardCharsets;\nimport java.security.GeneralSecurityException;\nimport java.security.KeyException;\nimport java.security.KeyFactory;\nimport java.security.KeyStore;\nimport java.security.PrivateKey;\nimport java.security.cert.Certificate;\nimport java.security.cert.CertificateFactory;\nimport java.security.cert.X509Certificate;\nimport java.security.spec.InvalidKeySpecException;\nimport java.security.spec.PKCS8EncodedKeySpec;\nimport java.util.Arrays;\nimport java.util.Base64;\nimport java.util.Collection;\nimport java.util.Enumeration;\nimport java.util.HashMap;\nimport java.util.InvalidPropertiesFormatException;\nimport java.util.List;\nimport java.util.Properties;\nimport java.util.Set;\nimport java.util.regex.Matcher;\nimport java.util.regex.Pattern;\nimport java.util.regex.PatternSyntaxException;\nimport org.apache.commons.cli.CommandLine;\nimport org.apache.commons.cli.Option;\nimport org.apache.commons.cli.Options;\nimport org.apache.commons.io.IOUtils;\n\npublic final class ConfigManager {\n // Variables that can be configured through config.properties and Environment Variables\n // NOTE: Variables which can be configured through environment variables **SHOULD** have a\n // \"MMS_\" prefix\n\n private static final String MMS_DEBUG = \"debug\";\n private static final String MMS_INFERENCE_ADDRESS = \"inference_address\";\n private static final String MMS_MANAGEMENT_ADDRESS = \"management_address\";\n private static final String MMS_LOAD_MODELS = \"load_models\";\n private static final String MMS_BLACKLIST_ENV_VARS = \"blacklist_env_vars\";\n private static final String MMS_DEFAULT_WORKERS_PER_MODEL = \"default_workers_per_model\";\n\n private static final String MMS_DEFAULT_RESPONSE_TIMEOUT = \"default_response_timeout\";\n private static final String MMS_DEFAULT_RESPONSE_TIMEOUT_SECONDS =\n \"default_response_timeout_seconds\";\n\n private static final String MMS_UNREGISTER_MODEL_TIMEOUT = \"unregister_model_timeout\";\n private static final String MMS_NUMBER_OF_NETTY_THREADS = \"number_of_netty_threads\";\n private static final String MMS_NETTY_CLIENT_THREADS = \"netty_client_threads\";\n private static final String MMS_JOB_QUEUE_SIZE = \"job_queue_size\";\n private static final String MMS_NUMBER_OF_GPU = \"number_of_gpu\";\n private static final String MMS_ASYNC_LOGGING = \"async_logging\";\n private static final String MMS_CORS_ALLOWED_ORIGIN = \"cors_allowed_origin\";\n private static final String MMS_CORS_ALLOWED_METHODS = \"cors_allowed_methods\";\n private static final String MMS_CORS_ALLOWED_HEADERS = \"cors_allowed_headers\";\n private static final String MMS_DECODE_INPUT_REQUEST = \"decode_input_request\";\n private static final String MMS_KEYSTORE = \"keystore\";\n private static final String MMS_KEYSTORE_PASS = \"keystore_pass\";\n private static final String MMS_KEYSTORE_TYPE = \"keystore_type\";\n private static final String MMS_CERTIFICATE_FILE = \"certificate_file\";\n private static final String MMS_PRIVATE_KEY_FILE = \"private_key_file\";\n private static final String MMS_MAX_REQUEST_SIZE = \"max_request_size\";\n private static final String MMS_MAX_RESPONSE_SIZE = \"max_response_size\";\n private static final String MMS_DEFAULT_SERVICE_HANDLER = \"default_service_handler\";\n private static final String MMS_PRELOAD_MODEL = \"preload_model\";\n private static final String MODEL_SERVER_HOME = \"model_server_home\";\n private static final String MMS_MODEL_STORE = \"model_store\";\n private static final String MMS_PREFER_DIRECT_BUFFER = \"prefer_direct_buffer\";\n\n // Configuration which are not documented or enabled through environment variables\n private static final String USE_NATIVE_IO = \"use_native_io\";\n private static final String IO_RATIO = \"io_ratio\";\n private static final String METRIC_TIME_INTERVAL = \"metric_time_interval\";\n private static final String ENABLE_ENVVARS_CONFIG = \"enable_envvars_config\";\n\n // Variables which are local\n public static final String MODEL_METRICS_LOGGER = \"MODEL_METRICS\";\n public static final String MODEL_LOGGER = \"MODEL_LOG\";\n public static final String MODEL_SERVER_METRICS_LOGGER = \"MMS_METRICS\";\n\n private Pattern blacklistPattern;\n private Properties prop;\n private static Pattern pattern = Pattern.compile(\"\\\\$\\\\$([^$]+[^$])\\\\$\\\\$\");\n\n private static ConfigManager instance;\n\n private String hostName;\n\n private ConfigManager(Arguments args) {\n prop = new Properties();\n\n String filePath = System.getenv(\"MMS_CONFIG_FILE\");\n if (filePath == null) {\n filePath = args.getMmsConfigFile();\n if (filePath == null) {\n filePath = System.getProperty(\"mmsConfigFile\", \"config.properties\");\n }\n }\n\n File file = new File(filePath);\n if (file.exists()) {\n try (FileInputStream stream = new FileInputStream(file)) {\n prop.load(stream);\n prop.put(\"mmsConfigFile\", filePath);\n } catch (IOException e) {\n throw new IllegalStateException(\"Unable to read configuration file\", e);\n }\n }\n\n resolveEnvVarVals(prop);\n String modelStore = args.getModelStore();\n if (modelStore != null) {\n prop.setProperty(MMS_MODEL_STORE, modelStore);\n }\n\n String[] models = args.getModels();\n if (models != null) {\n prop.setProperty(MMS_LOAD_MODELS, String.join(\",\", models));\n }\n\n prop.setProperty(\n MMS_NUMBER_OF_GPU,\n String.valueOf(\n Integer.min(\n getAvailableGpu(),\n getIntProperty(MMS_NUMBER_OF_GPU, Integer.MAX_VALUE))));\n\n String pythonExecutable = args.getPythonExecutable();\n if (pythonExecutable != null) {\n prop.setProperty(\"PYTHON_EXECUTABLE\", pythonExecutable);\n }\n\n try {\n InetAddress ip = InetAddress.getLocalHost();\n hostName = ip.getHostName();\n } catch (UnknownHostException e) {\n hostName = \"Unknown\";\n }\n\n if (Boolean.parseBoolean(prop.getProperty(MMS_ASYNC_LOGGING))) {\n enableAsyncLogging();\n }\n\n if (Boolean.parseBoolean(getEnableEnvVarsConfig())) {\n // Environment variables have higher precedence over the config file variables\n setSystemVars();\n }\n }\n\n private void resolveEnvVarVals(Properties prop) {\n Set keys = prop.stringPropertyNames();\n for (String key : keys) {\n String val = prop.getProperty(key);\n Matcher matcher = pattern.matcher(val);\n if (matcher.find()) {\n StringBuffer sb = new StringBuffer();\n do {\n String envVar = matcher.group(1);\n if (System.getenv(envVar) == null) {\n throw new IllegalArgumentException(\n \"Invalid Environment Variable \" + envVar);\n }\n matcher.appendReplacement(sb, System.getenv(envVar));\n } while (matcher.find());\n matcher.appendTail(sb);\n prop.setProperty(key, sb.toString());\n }\n }\n }\n\n private void setSystemVars() {\n Class configClass = ConfigManager.class;\n Field[] fields = configClass.getDeclaredFields();\n for (Field f : fields) {\n if (f.getName().startsWith(\"MMS_\")) {\n String val = System.getenv(f.getName());\n if (val != null) {\n try {\n prop.setProperty((String) f.get(ConfigManager.class), val);\n } catch (IllegalAccessException e) {\n e.printStackTrace(); // NOPMD\n }\n }\n }\n }\n }\n\n String getEnableEnvVarsConfig() {\n return prop.getProperty(ENABLE_ENVVARS_CONFIG, \"false\");\n }\n\n public String getHostName() {\n return hostName;\n }\n\n public static void init(Arguments args) {\n instance = new ConfigManager(args);\n }\n\n public static ConfigManager getInstance() {\n return instance;\n }\n\n public boolean isDebug() {\n return Boolean.getBoolean(\"MMS_DEBUG\")\n || Boolean.parseBoolean(prop.getProperty(MMS_DEBUG, \"false\"));\n }\n\n public Connector getListener(boolean management) {\n String binding;\n if (management) {\n binding = prop.getProperty(MMS_MANAGEMENT_ADDRESS, \"http://127.0.0.1:8081\");\n } else {\n binding = prop.getProperty(MMS_INFERENCE_ADDRESS, \"http://127.0.0.1:8080\");\n }\n return Connector.parse(binding, management);\n }\n\n public String getPreloadModel() {\n return getProperty(MMS_PRELOAD_MODEL, \"false\");\n }\n\n public boolean getPreferDirectBuffer() {\n return Boolean.parseBoolean(getProperty(MMS_PREFER_DIRECT_BUFFER, \"false\"));\n }\n\n public int getNettyThreads() {\n return getIntProperty(MMS_NUMBER_OF_NETTY_THREADS, 0);\n }\n\n public int getNettyClientThreads() {\n return getIntProperty(MMS_NETTY_CLIENT_THREADS, 0);\n }\n\n public int getJobQueueSize() {\n return getIntProperty(MMS_JOB_QUEUE_SIZE, 100);\n }\n\n public int getNumberOfGpu() {\n return getIntProperty(MMS_NUMBER_OF_GPU, 0);\n }\n\n public String getMmsDefaultServiceHandler() {\n return getProperty(MMS_DEFAULT_SERVICE_HANDLER, null);\n }\n\n public Properties getConfiguration() {\n return new Properties(prop);\n }\n\n public int getConfiguredDefaultWorkersPerModel() {\n return getIntProperty(MMS_DEFAULT_WORKERS_PER_MODEL, 0);\n }\n\n public int getDefaultWorkers() {\n if (isDebug()) {\n return 1;\n }\n int workers = getConfiguredDefaultWorkersPerModel();\n\n if ((workers == 0) && (prop.getProperty(\"NUM_WORKERS\", null) != null)) {\n workers = getIntProperty(\"NUM_WORKERS\", 0);\n }\n\n if (workers == 0) {\n workers = getNumberOfGpu();\n }\n if (workers == 0) {\n workers = Runtime.getRuntime().availableProcessors();\n }\n setProperty(\"NUM_WORKERS\", Integer.toString(workers));\n return workers;\n }\n\n public int getMetricTimeInterval() {\n return getIntProperty(METRIC_TIME_INTERVAL, 60);\n }\n\n public String getModelServerHome() {\n String mmsHome = System.getenv(\"MODEL_SERVER_HOME\");\n if (mmsHome == null) {\n mmsHome = System.getProperty(MODEL_SERVER_HOME);\n if (mmsHome == null) {\n mmsHome = getProperty(MODEL_SERVER_HOME, null);\n if (mmsHome == null) {\n mmsHome = getCanonicalPath(findMmsHome());\n return mmsHome;\n }\n }\n }\n\n File dir = new File(mmsHome);\n if (!dir.exists()) {\n throw new IllegalArgumentException(\"Model server home not exist: \" + mmsHome);\n }\n mmsHome = getCanonicalPath(dir);\n return mmsHome;\n }\n\n public String getPythonExecutable() {\n return prop.getProperty(\"PYTHON_EXECUTABLE\", \"python\");\n }\n\n public String getModelStore() {\n return getCanonicalPath(prop.getProperty(MMS_MODEL_STORE));\n }\n\n public String getLoadModels() {\n return prop.getProperty(MMS_LOAD_MODELS);\n }\n\n public Pattern getBlacklistPattern() {\n return blacklistPattern;\n }\n\n public String getCorsAllowedOrigin() {\n return prop.getProperty(MMS_CORS_ALLOWED_ORIGIN);\n }\n\n public String getCorsAllowedMethods() {\n return prop.getProperty(MMS_CORS_ALLOWED_METHODS);\n }\n\n public String getCorsAllowedHeaders() {\n return prop.getProperty(MMS_CORS_ALLOWED_HEADERS);\n }\n\n public SslContext getSslContext() throws IOException, GeneralSecurityException {\n List supportedCiphers =\n Arrays.asList(\n \"TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA\",\n \"TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256\");\n\n PrivateKey privateKey;\n X509Certificate[] chain;\n String keyStoreFile = prop.getProperty(MMS_KEYSTORE);\n String privateKeyFile = prop.getProperty(MMS_PRIVATE_KEY_FILE);\n String certificateFile = prop.getProperty(MMS_CERTIFICATE_FILE);\n if (keyStoreFile != null) {\n char[] keystorePass = getProperty(MMS_KEYSTORE_PASS, \"changeit\").toCharArray();\n String keystoreType = getProperty(MMS_KEYSTORE_TYPE, \"PKCS12\");\n KeyStore keyStore = KeyStore.getInstance(keystoreType);\n try (InputStream is = new FileInputStream(keyStoreFile)) {\n keyStore.load(is, keystorePass);\n }\n\n Enumeration en = keyStore.aliases();\n String keyAlias = null;\n while (en.hasMoreElements()) {\n String alias = en.nextElement();\n if (keyStore.isKeyEntry(alias)) {\n keyAlias = alias;\n break;\n }\n }\n\n if (keyAlias == null) {\n throw new KeyException(\"No key entry found in keystore.\");\n }\n\n privateKey = (PrivateKey) keyStore.getKey(keyAlias, keystorePass);\n\n Certificate[] certs = keyStore.getCertificateChain(keyAlias);\n chain = new X509Certificate[certs.length];\n for (int i = 0; i < certs.length; ++i) {\n chain[i] = (X509Certificate) certs[i];\n }\n } else if (privateKeyFile != null && certificateFile != null) {\n privateKey = loadPrivateKey(privateKeyFile);\n chain = loadCertificateChain(certificateFile);\n } else {\n SelfSignedCertificate ssc = new SelfSignedCertificate();\n privateKey = ssc.key();\n chain = new X509Certificate[] {ssc.cert()};\n }\n\n return SslContextBuilder.forServer(privateKey, chain)\n .protocols(\"TLSv1.2\")\n .ciphers(supportedCiphers)\n .build();\n }\n\n private PrivateKey loadPrivateKey(String keyFile) throws IOException, GeneralSecurityException {\n KeyFactory keyFactory = KeyFactory.getInstance(\"RSA\");\n try (InputStream is = new FileInputStream(keyFile)) {\n String content = IOUtils.toString(is, StandardCharsets.UTF_8);\n content = content.replaceAll(\"-----(BEGIN|END)( RSA)? PRIVATE KEY-----\\\\s*\", \"\");\n byte[] buf = Base64.getMimeDecoder().decode(content);\n try {\n PKCS8EncodedKeySpec privKeySpec = new PKCS8EncodedKeySpec(buf);\n return keyFactory.generatePrivate(privKeySpec);\n } catch (InvalidKeySpecException e) {\n // old private key is OpenSSL format private key\n buf = OpenSslKey.convertPrivateKey(buf);\n PKCS8EncodedKeySpec privKeySpec = new PKCS8EncodedKeySpec(buf);\n return keyFactory.generatePrivate(privKeySpec);\n }\n }\n }\n\n private X509Certificate[] loadCertificateChain(String keyFile)\n throws IOException, GeneralSecurityException {\n CertificateFactory cf = CertificateFactory.getInstance(\"X.509\");\n try (InputStream is = new FileInputStream(keyFile)) {\n Collection certs = cf.generateCertificates(is);\n int i = 0;\n X509Certificate[] chain = new X509Certificate[certs.size()];\n for (Certificate cert : certs) {\n chain[i++] = (X509Certificate) cert;\n }\n return chain;\n }\n }\n\n public String getProperty(String key, String def) {\n return prop.getProperty(key, def);\n }\n\n public void validateConfigurations() throws InvalidPropertiesFormatException {\n String blacklistVars = prop.getProperty(MMS_BLACKLIST_ENV_VARS, \"\");\n try {\n blacklistPattern = Pattern.compile(blacklistVars);\n } catch (PatternSyntaxException e) {\n throw new InvalidPropertiesFormatException(e);\n }\n }\n\n public String dumpConfigurations() {\n Runtime runtime = Runtime.getRuntime();\n return \"\\nMMS Home: \"\n + getModelServerHome()\n + \"\\nCurrent directory: \"\n + getCanonicalPath(\".\")\n + \"\\nTemp directory: \"\n + System.getProperty(\"java.io.tmpdir\")\n + \"\\nNumber of GPUs: \"\n + getNumberOfGpu()\n + \"\\nNumber of CPUs: \"\n + runtime.availableProcessors()\n + \"\\nMax heap size: \"\n + (runtime.maxMemory() / 1024 / 1024)\n + \" M\\nPython executable: \"\n + (getPythonExecutable() == null ? \"N/A\" : getPythonExecutable())\n + \"\\nConfig file: \"\n + prop.getProperty(\"mmsConfigFile\", \"N/A\")\n + \"\\nInference address: \"\n + getListener(false)\n + \"\\nManagement address: \"\n + getListener(true)\n + \"\\nModel Store: \"\n + (getModelStore() == null ? \"N/A\" : getModelStore())\n + \"\\nInitial Models: \"\n + (getLoadModels() == null ? \"N/A\" : getLoadModels())\n + \"\\nLog dir: \"\n + getCanonicalPath(System.getProperty(\"LOG_LOCATION\"))\n + \"\\nMetrics dir: \"\n + getCanonicalPath(System.getProperty(\"METRICS_LOCATION\"))\n + \"\\nNetty threads: \"\n + getNettyThreads()\n + \"\\nNetty client threads: \"\n + getNettyClientThreads()\n + \"\\nDefault workers per model: \"\n + getDefaultWorkers()\n + \"\\nBlacklist Regex: \"\n + prop.getProperty(MMS_BLACKLIST_ENV_VARS, \"N/A\")\n + \"\\nMaximum Response Size: \"\n + prop.getProperty(MMS_MAX_RESPONSE_SIZE, \"6553500\")\n + \"\\nMaximum Request Size: \"\n + prop.getProperty(MMS_MAX_REQUEST_SIZE, \"6553500\")\n + \"\\nPreload model: \"\n + prop.getProperty(MMS_PRELOAD_MODEL, \"false\")\n + \"\\nPrefer direct buffer: \"\n + prop.getProperty(MMS_PREFER_DIRECT_BUFFER, \"false\");\n }\n\n public boolean useNativeIo() {\n return Boolean.parseBoolean(prop.getProperty(USE_NATIVE_IO, \"true\"));\n }\n\n public int getIoRatio() {\n return getIntProperty(IO_RATIO, 50);\n }\n\n public int getMaxResponseSize() {\n return getIntProperty(MMS_MAX_RESPONSE_SIZE, 6553500);\n }\n\n public int getMaxRequestSize() {\n return getIntProperty(MMS_MAX_REQUEST_SIZE, 6553500);\n }\n\n void setProperty(String key, String value) {\n prop.setProperty(key, value);\n }\n\n private int getIntProperty(String key, int def) {\n String value = prop.getProperty(key);\n if (value == null) {\n return def;\n }\n return Integer.parseInt(value);\n }\n\n public int getDefaultResponseTimeoutSeconds() {\n // TODO The MMS_DEFAULT_RESPONSE_TIMEOUT variable was never intended to represent minutes,\n // but due to a bug that's what it did. We'd like to remove this and match the documented\n // behavior, but for now we're being cautious about backward compatibility.\n\n // Check both properties, prefer seconds if provided, convert to seconds for return value\n int timeoutSeconds =\n Integer.parseInt(prop.getProperty(MMS_DEFAULT_RESPONSE_TIMEOUT_SECONDS, \"-1\"));\n if (timeoutSeconds < 0) {\n int timeoutMinutes =\n Integer.parseInt(prop.getProperty(MMS_DEFAULT_RESPONSE_TIMEOUT, \"120\"));\n timeoutSeconds = 60 * timeoutMinutes;\n }\n return timeoutSeconds;\n }\n\n public int getUnregisterModelTimeout() {\n return Integer.parseInt(prop.getProperty(MMS_UNREGISTER_MODEL_TIMEOUT, \"120\"));\n }\n\n private File findMmsHome() {\n File cwd = new File(getCanonicalPath(\".\"));\n File file = cwd;\n while (file != null) {\n File mms = new File(file, \"mms\");\n if (mms.exists()) {\n return file;\n }\n file = file.getParentFile();\n }\n return cwd;\n }\n\n private void enableAsyncLogging() {\n System.setProperty(\n \"log4j2.contextSelector\",\n \"org.apache.logging.log4j.core.async.AsyncLoggerContextSelector\");\n }\n\n public HashMap getBackendConfiguration() {\n HashMap config = new HashMap<>();\n // Append properties used by backend worker here\n config.put(\"MMS_DECODE_INPUT_REQUEST\", prop.getProperty(MMS_DECODE_INPUT_REQUEST, \"true\"));\n\n return config;\n }\n\n private static String getCanonicalPath(File file) {\n try {\n return file.getCanonicalPath();\n } catch (IOException e) {\n return file.getAbsolutePath();\n }\n }\n\n private static String getCanonicalPath(String path) {\n if (path == null) {\n return null;\n }\n return getCanonicalPath(new File(path));\n }\n\n private static int getAvailableGpu() {\n try {\n Process process =\n Runtime.getRuntime().exec(\"nvidia-smi --query-gpu=index --format=csv\");\n int ret = process.waitFor();\n if (ret != 0) {\n return 0;\n }\n List list = IOUtils.readLines(process.getInputStream(), StandardCharsets.UTF_8);\n if (list.isEmpty() || !\"index\".equals(list.get(0))) {\n throw new AssertionError(\"Unexpected nvidia-smi response.\");\n }\n return list.size() - 1;\n } catch (IOException | InterruptedException e) {\n return 0;\n }\n }\n\n public static final class Arguments {\n\n private String mmsConfigFile;\n private String pythonExecutable;\n private String modelStore;\n private String[] models;\n\n public Arguments() {}\n\n public Arguments(CommandLine cmd) {\n mmsConfigFile = cmd.getOptionValue(\"mms-config-file\");\n pythonExecutable = cmd.getOptionValue(\"python\");\n modelStore = cmd.getOptionValue(\"model-store\");\n models = cmd.getOptionValues(\"models\");\n }\n\n public static Options getOptions() {\n Options options = new Options();\n options.addOption(\n Option.builder(\"f\")\n .longOpt(\"mms-config-file\")\n .hasArg()\n .argName(\"MMS-CONFIG-FILE\")\n .desc(\"Path to the configuration properties file.\")\n .build());\n options.addOption(\n Option.builder(\"e\")\n .longOpt(\"python\")\n .hasArg()\n .argName(\"PYTHON\")\n .desc(\"Python runtime executable path.\")\n .build());\n options.addOption(\n Option.builder(\"m\")\n .longOpt(\"models\")\n .hasArgs()\n .argName(\"MODELS\")\n .desc(\"Models to be loaded at startup.\")\n .build());\n options.addOption(\n Option.builder(\"s\")\n .longOpt(\"model-store\")\n .hasArg()\n .argName(\"MODELS-STORE\")\n .desc(\"Model store location where models can be loaded.\")\n .build());\n return options;\n }\n\n public String getMmsConfigFile() {\n return mmsConfigFile;\n }\n\n public String getPythonExecutable() {\n return pythonExecutable;\n }\n\n public void setMmsConfigFile(String mmsConfigFile) {\n this.mmsConfigFile = mmsConfigFile;\n }\n\n public String getModelStore() {\n return modelStore;\n }\n\n public void setModelStore(String modelStore) {\n this.modelStore = modelStore;\n }\n\n public String[] getModels() {\n return models;\n }\n\n public void setModels(String[] models) {\n this.models = models;\n }\n }\n}\n" }, { "path": "frontend/server/src/main/java/com/amazonaws/ml/mms/util/Connector.java", "content": "/*\n * Copyright 2018 Amazon.com, Inc. or its affiliates. All Rights Reserved.\n *\n * Licensed under the Apache License, Version 2.0 (the \"License\"). You may not use this file except in compliance\n * with the License. A copy of the License is located at\n *\n * http://aws.amazon.com/apache2.0/\n *\n * or in the \"license\" file accompanying this file. This file is distributed on an \"AS IS\" BASIS, WITHOUT WARRANTIES\n * OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions\n * and limitations under the License.\n */\npackage com.amazonaws.ml.mms.util;\n\nimport io.netty.channel.Channel;\nimport io.netty.channel.EventLoopGroup;\nimport io.netty.channel.ServerChannel;\nimport io.netty.channel.epoll.Epoll;\nimport io.netty.channel.epoll.EpollDomainSocketChannel;\nimport io.netty.channel.epoll.EpollEventLoopGroup;\nimport io.netty.channel.epoll.EpollServerDomainSocketChannel;\nimport io.netty.channel.epoll.EpollServerSocketChannel;\nimport io.netty.channel.epoll.EpollSocketChannel;\nimport io.netty.channel.kqueue.KQueue;\nimport io.netty.channel.kqueue.KQueueDomainSocketChannel;\nimport io.netty.channel.kqueue.KQueueEventLoopGroup;\nimport io.netty.channel.kqueue.KQueueServerDomainSocketChannel;\nimport io.netty.channel.kqueue.KQueueServerSocketChannel;\nimport io.netty.channel.kqueue.KQueueSocketChannel;\nimport io.netty.channel.nio.NioEventLoopGroup;\nimport io.netty.channel.socket.nio.NioServerSocketChannel;\nimport io.netty.channel.socket.nio.NioSocketChannel;\nimport io.netty.channel.unix.DomainSocketAddress;\nimport java.io.File;\nimport java.net.InetSocketAddress;\nimport java.net.SocketAddress;\nimport java.util.Objects;\nimport java.util.regex.Matcher;\nimport java.util.regex.Pattern;\nimport org.apache.commons.io.FileUtils;\n\npublic class Connector {\n\n private static final Pattern ADDRESS_PATTERN =\n Pattern.compile(\n \"((https|http)://([^:^/]+)(:([0-9]+))?)|(unix:(/.*))\",\n Pattern.CASE_INSENSITIVE);\n\n private static boolean useNativeIo = ConfigManager.getInstance().useNativeIo();\n\n private boolean uds;\n private String socketPath;\n private String bindIp;\n private int port;\n private boolean ssl;\n private boolean management;\n\n public Connector(int port) {\n this(port, useNativeIo && (Epoll.isAvailable() || KQueue.isAvailable()));\n }\n\n private Connector(int port, boolean uds) {\n this.port = port;\n this.uds = uds;\n if (uds) {\n bindIp = \"\";\n socketPath = System.getProperty(\"java.io.tmpdir\") + \"/.mms.sock.\" + port;\n } else {\n bindIp = \"127.0.0.1\";\n socketPath = String.valueOf(port);\n }\n }\n\n private Connector(\n int port,\n boolean uds,\n String bindIp,\n String socketPath,\n boolean ssl,\n boolean management) {\n this.port = port;\n this.uds = uds;\n this.bindIp = bindIp;\n this.socketPath = socketPath;\n this.ssl = ssl;\n this.management = management;\n }\n\n public static Connector parse(String binding, boolean management) {\n Matcher matcher = ADDRESS_PATTERN.matcher(binding);\n if (!matcher.matches()) {\n throw new IllegalArgumentException(\"Invalid binding address: \" + binding);\n }\n\n boolean uds = matcher.group(7) != null;\n if (uds) {\n if (!useNativeIo) {\n throw new IllegalArgumentException(\n \"unix domain socket requires use_native_io set to true.\");\n }\n String path = matcher.group(7);\n return new Connector(-1, true, \"\", path, false, management);\n }\n\n String protocol = matcher.group(2);\n String host = matcher.group(3);\n String listeningPort = matcher.group(5);\n\n boolean ssl = \"https\".equalsIgnoreCase(protocol);\n int port;\n if (listeningPort == null) {\n if (management) {\n port = ssl ? 8444 : 8081;\n } else {\n port = ssl ? 443 : 80;\n }\n } else {\n port = Integer.parseInt(listeningPort);\n }\n if (port >= Short.MAX_VALUE) {\n throw new IllegalArgumentException(\"Invalid port number: \" + binding);\n }\n return new Connector(port, false, host, String.valueOf(port), ssl, management);\n }\n\n public String getSocketType() {\n return uds ? \"unix\" : \"tcp\";\n }\n\n public String getSocketPath() {\n return socketPath;\n }\n\n public boolean isUds() {\n return uds;\n }\n\n public boolean isSsl() {\n return ssl;\n }\n\n public boolean isManagement() {\n return management;\n }\n\n public SocketAddress getSocketAddress() {\n return uds ? new DomainSocketAddress(socketPath) : new InetSocketAddress(bindIp, port);\n }\n\n public String getPurpose() {\n return management ? \"Management\" : \"Inference\";\n }\n\n public static EventLoopGroup newEventLoopGroup(int threads) {\n if (useNativeIo && Epoll.isAvailable()) {\n return new EpollEventLoopGroup(threads);\n } else if (useNativeIo && KQueue.isAvailable()) {\n return new KQueueEventLoopGroup(threads);\n }\n\n NioEventLoopGroup eventLoopGroup = new NioEventLoopGroup(threads);\n eventLoopGroup.setIoRatio(ConfigManager.getInstance().getIoRatio());\n return eventLoopGroup;\n }\n\n public Class getServerChannel() {\n if (useNativeIo && Epoll.isAvailable()) {\n return uds ? EpollServerDomainSocketChannel.class : EpollServerSocketChannel.class;\n } else if (useNativeIo && KQueue.isAvailable()) {\n return uds ? KQueueServerDomainSocketChannel.class : KQueueServerSocketChannel.class;\n }\n\n return NioServerSocketChannel.class;\n }\n\n public Class getClientChannel() {\n if (useNativeIo && Epoll.isAvailable()) {\n return uds ? EpollDomainSocketChannel.class : EpollSocketChannel.class;\n } else if (useNativeIo && KQueue.isAvailable()) {\n return uds ? KQueueDomainSocketChannel.class : KQueueSocketChannel.class;\n }\n\n return NioSocketChannel.class;\n }\n\n public void clean() {\n if (uds) {\n FileUtils.deleteQuietly(new File(socketPath));\n }\n }\n\n @Override\n public boolean equals(Object o) {\n if (this == o) {\n return true;\n }\n if (o == null || getClass() != o.getClass()) {\n return false;\n }\n Connector connector = (Connector) o;\n return uds == connector.uds\n && port == connector.port\n && socketPath.equals(connector.socketPath)\n && bindIp.equals(connector.bindIp);\n }\n\n @Override\n public int hashCode() {\n return Objects.hash(uds, socketPath, bindIp, port);\n }\n\n @Override\n public String toString() {\n if (uds) {\n return \"unix:\" + socketPath;\n } else if (ssl) {\n return \"https://\" + bindIp + ':' + port;\n }\n return \"http://\" + bindIp + ':' + port;\n }\n}\n" }, { "path": "frontend/server/src/main/java/com/amazonaws/ml/mms/util/ConnectorType.java", "content": "package com.amazonaws.ml.mms.util;\n\npublic enum ConnectorType {\n INFERENCE_CONNECTOR,\n MANAGEMENT_CONNECTOR,\n BOTH\n}\n" }, { "path": "frontend/server/src/main/java/com/amazonaws/ml/mms/util/JsonUtils.java", "content": "/*\n * Copyright 2018 Amazon.com, Inc. or its affiliates. All Rights Reserved.\n *\n * Licensed under the Apache License, Version 2.0 (the \"License\"). You may not use this file except in compliance\n * with the License. A copy of the License is located at\n *\n * http://aws.amazon.com/apache2.0/\n *\n * or in the \"license\" file accompanying this file. This file is distributed on an \"AS IS\" BASIS, WITHOUT WARRANTIES\n * OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions\n * and limitations under the License.\n */\npackage com.amazonaws.ml.mms.util;\n\nimport com.google.gson.Gson;\nimport com.google.gson.GsonBuilder;\n\npublic final class JsonUtils {\n\n public static final Gson GSON_PRETTY =\n new GsonBuilder()\n .setDateFormat(\"yyyy-MM-dd'T'HH:mm:ss.SSS'Z'\")\n .setPrettyPrinting()\n .create();\n public static final Gson GSON = new GsonBuilder().create();\n\n private JsonUtils() {}\n}\n" }, { "path": "frontend/server/src/main/java/com/amazonaws/ml/mms/util/NettyUtils.java", "content": "/*\n * Copyright 2018 Amazon.com, Inc. or its affiliates. All Rights Reserved.\n *\n * Licensed under the Apache License, Version 2.0 (the \"License\"). You may not use this file except in compliance\n * with the License. A copy of the License is located at\n *\n * http://aws.amazon.com/apache2.0/\n *\n * or in the \"license\" file accompanying this file. This file is distributed on an \"AS IS\" BASIS, WITHOUT WARRANTIES\n * OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions\n * and limitations under the License.\n */\npackage com.amazonaws.ml.mms.util;\n\nimport com.amazonaws.ml.mms.http.ErrorResponse;\nimport com.amazonaws.ml.mms.http.Session;\nimport com.amazonaws.ml.mms.metrics.Dimension;\nimport com.amazonaws.ml.mms.metrics.Metric;\nimport com.amazonaws.ml.mms.util.messages.InputParameter;\nimport io.netty.buffer.ByteBuf;\nimport io.netty.buffer.Unpooled;\nimport io.netty.channel.Channel;\nimport io.netty.channel.ChannelFuture;\nimport io.netty.channel.ChannelFutureListener;\nimport io.netty.channel.ChannelHandlerContext;\nimport io.netty.handler.codec.http.DefaultFullHttpResponse;\nimport io.netty.handler.codec.http.DefaultHttpHeadersFactory;\nimport io.netty.handler.codec.http.FullHttpResponse;\nimport io.netty.handler.codec.http.HttpHeaderNames;\nimport io.netty.handler.codec.http.HttpHeaderValues;\nimport io.netty.handler.codec.http.HttpHeaders;\nimport io.netty.handler.codec.http.HttpRequest;\nimport io.netty.handler.codec.http.HttpResponseStatus;\nimport io.netty.handler.codec.http.HttpUtil;\nimport io.netty.handler.codec.http.HttpVersion;\nimport io.netty.handler.codec.http.QueryStringDecoder;\nimport io.netty.handler.codec.http.multipart.Attribute;\nimport io.netty.handler.codec.http.multipart.FileUpload;\nimport io.netty.handler.codec.http.multipart.InterfaceHttpData;\nimport io.netty.util.AttributeKey;\nimport io.netty.util.CharsetUtil;\nimport java.io.IOException;\nimport java.net.SocketAddress;\nimport java.util.List;\nimport org.slf4j.Logger;\nimport org.slf4j.LoggerFactory;\n\n/** A utility class that handling Netty request and response. */\npublic final class NettyUtils {\n\n private static final Logger logger = LoggerFactory.getLogger(\"ACCESS_LOG\");\n\n private static final String REQUEST_ID = \"x-request-id\";\n private static final AttributeKey SESSION_KEY = AttributeKey.valueOf(\"session\");\n private static final Dimension DIMENSION = new Dimension(\"Level\", \"Host\");\n private static final Metric REQUESTS_2_XX =\n new Metric(\n \"Requests2XX\",\n \"1\",\n \"Count\",\n ConfigManager.getInstance().getHostName(),\n DIMENSION);\n private static final Metric REQUESTS_4_XX =\n new Metric(\n \"Requests4XX\",\n \"1\",\n \"Count\",\n ConfigManager.getInstance().getHostName(),\n DIMENSION);\n private static final Metric REQUESTS_5_XX =\n new Metric(\n \"Requests5XX\",\n \"1\",\n \"Count\",\n ConfigManager.getInstance().getHostName(),\n DIMENSION);\n\n private static final Logger loggerMmsMetrics =\n LoggerFactory.getLogger(ConfigManager.MODEL_SERVER_METRICS_LOGGER);\n\n private NettyUtils() {}\n\n public static void requestReceived(Channel channel, HttpRequest request) {\n Session session = channel.attr(SESSION_KEY).get();\n assert session == null;\n\n SocketAddress address = channel.remoteAddress();\n String remoteIp;\n if (address == null) {\n // This is can be null on UDS, or on certain case in Windows\n remoteIp = \"0.0.0.0\";\n } else {\n remoteIp = address.toString();\n }\n channel.attr(SESSION_KEY).set(new Session(remoteIp, request));\n }\n\n public static String getRequestId(Channel channel) {\n Session accessLog = channel.attr(SESSION_KEY).get();\n if (accessLog != null) {\n return accessLog.getRequestId();\n }\n return null;\n }\n\n public static void sendJsonResponse(ChannelHandlerContext ctx, Object json) {\n sendJsonResponse(ctx, JsonUtils.GSON_PRETTY.toJson(json), HttpResponseStatus.OK);\n }\n\n public static void sendJsonResponse(\n ChannelHandlerContext ctx, Object json, HttpResponseStatus status) {\n sendJsonResponse(ctx, JsonUtils.GSON_PRETTY.toJson(json), status);\n }\n\n public static void sendJsonResponse(ChannelHandlerContext ctx, String json) {\n sendJsonResponse(ctx, json, HttpResponseStatus.OK);\n }\n\n public static void sendJsonResponse(\n ChannelHandlerContext ctx, String json, HttpResponseStatus status) {\n FullHttpResponse resp =\n new DefaultFullHttpResponse(\n HttpVersion.HTTP_1_1,\n status,\n Unpooled.directBuffer(),\n DefaultHttpHeadersFactory.headersFactory(),\n DefaultHttpHeadersFactory.trailersFactory());\n resp.headers().set(HttpHeaderNames.CONTENT_TYPE, HttpHeaderValues.APPLICATION_JSON);\n ByteBuf content = resp.content();\n content.writeCharSequence(json, CharsetUtil.UTF_8);\n content.writeByte('\\n');\n sendHttpResponse(ctx, resp, true);\n }\n\n public static void sendError(\n ChannelHandlerContext ctx, HttpResponseStatus status, Throwable t) {\n ErrorResponse error =\n new ErrorResponse(status.code(), t.getClass().getSimpleName(), t.getMessage());\n sendJsonResponse(ctx, error, status);\n }\n\n public static void sendError(\n ChannelHandlerContext ctx, HttpResponseStatus status, Throwable t, String msg) {\n ErrorResponse error = new ErrorResponse(status.code(), t.getClass().getSimpleName(), msg);\n sendJsonResponse(ctx, error, status);\n }\n\n /**\n * Send HTTP response to client.\n *\n * @param ctx ChannelHandlerContext\n * @param resp HttpResponse to send\n * @param keepAlive if keep the connection\n */\n public static void sendHttpResponse(\n ChannelHandlerContext ctx, FullHttpResponse resp, boolean keepAlive) {\n // Send the response and close the connection if necessary.\n Channel channel = ctx.channel();\n Session session = channel.attr(SESSION_KEY).getAndSet(null);\n HttpHeaders headers = resp.headers();\n\n ConfigManager configManager = ConfigManager.getInstance();\n if (session != null) {\n // session might be recycled if channel is closed already.\n session.setCode(resp.status().code());\n headers.set(REQUEST_ID, session.getRequestId());\n logger.info(session.toString());\n }\n int code = resp.status().code();\n if (code >= 200 && code < 300) {\n loggerMmsMetrics.info(\"{}\", REQUESTS_2_XX);\n } else if (code >= 400 && code < 500) {\n loggerMmsMetrics.info(\"{}\", REQUESTS_4_XX);\n } else {\n loggerMmsMetrics.info(\"{}\", REQUESTS_5_XX);\n }\n\n String allowedOrigin = configManager.getCorsAllowedOrigin();\n String allowedMethods = configManager.getCorsAllowedMethods();\n String allowedHeaders = configManager.getCorsAllowedHeaders();\n\n if (allowedOrigin != null\n && !allowedOrigin.isEmpty()\n && !headers.contains(HttpHeaderNames.ACCESS_CONTROL_ALLOW_ORIGIN)) {\n headers.set(HttpHeaderNames.ACCESS_CONTROL_ALLOW_ORIGIN, allowedOrigin);\n }\n if (allowedMethods != null\n && !allowedMethods.isEmpty()\n && !headers.contains(HttpHeaderNames.ACCESS_CONTROL_ALLOW_METHODS)) {\n headers.set(HttpHeaderNames.ACCESS_CONTROL_ALLOW_METHODS, allowedMethods);\n }\n if (allowedHeaders != null\n && !allowedHeaders.isEmpty()\n && !headers.contains(HttpHeaderNames.ACCESS_CONTROL_ALLOW_HEADERS)) {\n headers.set(HttpHeaderNames.ACCESS_CONTROL_ALLOW_HEADERS, allowedHeaders);\n }\n\n // Add cache-control headers to avoid browser cache response\n headers.set(\"Pragma\", \"no-cache\");\n headers.set(\"Cache-Control\", \"no-cache; no-store, must-revalidate, private\");\n headers.set(\"Expires\", \"Thu, 01 Jan 1970 00:00:00 UTC\");\n\n HttpUtil.setContentLength(resp, resp.content().readableBytes());\n if (!keepAlive || code >= 400) {\n headers.set(HttpHeaderNames.CONNECTION, HttpHeaderValues.CLOSE);\n ChannelFuture f = channel.writeAndFlush(resp);\n f.addListener(ChannelFutureListener.CLOSE);\n } else {\n headers.set(HttpHeaderNames.CONNECTION, HttpHeaderValues.KEEP_ALIVE);\n channel.writeAndFlush(resp);\n }\n }\n\n /** Closes the specified channel after all queued write requests are flushed. */\n public static void closeOnFlush(Channel ch) {\n if (ch.isActive()) {\n ch.writeAndFlush(Unpooled.EMPTY_BUFFER).addListener(ChannelFutureListener.CLOSE);\n }\n }\n\n public static byte[] getBytes(ByteBuf buf) {\n if (buf.hasArray()) {\n return buf.array();\n }\n\n byte[] ret = new byte[buf.readableBytes()];\n int readerIndex = buf.readerIndex();\n buf.getBytes(readerIndex, ret);\n return ret;\n }\n\n public static String getParameter(QueryStringDecoder decoder, String key, String def) {\n List param = decoder.parameters().get(key);\n if (param != null && !param.isEmpty()) {\n return param.get(0);\n }\n return def;\n }\n\n public static int getIntParameter(QueryStringDecoder decoder, String key, int def) {\n String value = getParameter(decoder, key, null);\n if (value == null) {\n return def;\n }\n try {\n return Integer.parseInt(value);\n } catch (NumberFormatException e) {\n return def;\n }\n }\n\n public static InputParameter getFormData(InterfaceHttpData data) {\n if (data == null) {\n return null;\n }\n\n String name = data.getName();\n switch (data.getHttpDataType()) {\n case Attribute:\n Attribute attribute = (Attribute) data;\n try {\n return new InputParameter(name, attribute.getValue());\n } catch (IOException e) {\n throw new AssertionError(e);\n }\n case FileUpload:\n FileUpload fileUpload = (FileUpload) data;\n String contentType = fileUpload.getContentType();\n try {\n return new InputParameter(name, getBytes(fileUpload.getByteBuf()), contentType);\n } catch (IOException e) {\n throw new AssertionError(e);\n }\n default:\n throw new IllegalArgumentException(\n \"Except form field, but got \" + data.getHttpDataType());\n }\n }\n}\n" }, { "path": "frontend/server/src/main/java/com/amazonaws/ml/mms/util/OpenSslKey.java", "content": "/*\n * Copyright 2018 Amazon.com, Inc. or its affiliates. All Rights Reserved.\n *\n * Licensed under the Apache License, Version 2.0 (the \"License\"). You may not use this file except in compliance\n * with the License. A copy of the License is located at\n *\n * http://aws.amazon.com/apache2.0/\n *\n * or in the \"license\" file accompanying this file. This file is distributed on an \"AS IS\" BASIS, WITHOUT WARRANTIES\n * OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions\n * and limitations under the License.\n */\npackage com.amazonaws.ml.mms.util;\n\n/** A utility class converting OpenSSL private key to PKCS8 private key. */\npublic final class OpenSslKey {\n\n private static final int[] RSA_ENCRYPTION = {1, 2, 840, 113549, 1, 1, 1};\n private static final byte[] NULL_BYTES = {0x05, 0x00};\n\n private OpenSslKey() {}\n\n /**\n * Convert OpenSSL private key to PKCS8 private key.\n *\n * @param keySpec OpenSSL key spec\n * @return PKCS8 encoded private key\n */\n public static byte[] convertPrivateKey(byte[] keySpec) {\n if (keySpec == null) {\n return null;\n }\n\n byte[] bytes = new byte[keySpec.length];\n System.arraycopy(keySpec, 0, bytes, 0, keySpec.length);\n byte[] octetBytes = encodeOctetString(bytes);\n\n byte[] oidBytes = encodeOID(RSA_ENCRYPTION);\n byte[] verBytes = {0x02, 0x01, 0x00};\n\n byte[][] seqBytes = new byte[4][];\n seqBytes[0] = oidBytes;\n seqBytes[1] = NULL_BYTES;\n seqBytes[2] = null;\n\n byte[] oidSeqBytes = encodeSequence(seqBytes);\n\n seqBytes[0] = verBytes;\n seqBytes[1] = oidSeqBytes;\n seqBytes[2] = octetBytes;\n seqBytes[3] = null;\n\n return encodeSequence(seqBytes);\n }\n\n private static byte[] encodeOID(int[] oid) {\n if (oid == null) {\n return null;\n }\n\n int oLen = 1;\n\n for (int i = 2; i < oid.length; i++) {\n oLen += getOIDCompLength(oid[i]);\n }\n\n int len = oLen + getLengthOfLengthField(oLen) + 1;\n\n byte[] bytes = new byte[len];\n\n bytes[0] = 0x06; // ASN Object ID\n int offset = writeLengthField(bytes, oLen);\n\n bytes[offset++] = (byte) (40 * oid[0] + oid[1]);\n\n for (int i = 2; i < oid.length; i++) {\n offset = writeOIDComp(oid[i], bytes, offset);\n }\n\n return bytes;\n }\n\n private static byte[] encodeOctetString(byte[] bytes) {\n if (bytes == null) {\n return null;\n }\n\n int oLen = bytes.length; // one byte for unused bits field\n int len = oLen + getLengthOfLengthField(oLen) + 1;\n\n byte[] newBytes = new byte[len];\n\n newBytes[0] = 0x04;\n int offset = writeLengthField(newBytes, oLen);\n\n if (len - oLen != offset) {\n return null;\n }\n\n System.arraycopy(bytes, 0, newBytes, offset, oLen);\n return newBytes;\n }\n\n private static byte[] encodeSequence(byte[][] byteArrays) {\n if (byteArrays == null) {\n return null;\n }\n\n int oLen = 0;\n for (byte[] b : byteArrays) {\n if (b == null) {\n break;\n }\n\n oLen += b.length;\n }\n\n int len = oLen + getLengthOfLengthField(oLen) + 1;\n\n byte[] bytes = new byte[len];\n bytes[0] = 0x10 | 0x20; // ASN sequence & constructed\n int offset = writeLengthField(bytes, oLen);\n\n if (len - oLen != offset) {\n return null;\n }\n\n for (byte[] b : byteArrays) {\n if (b == null) {\n break;\n }\n\n System.arraycopy(b, 0, bytes, offset, b.length);\n offset += b.length;\n }\n\n return bytes;\n }\n\n private static int writeLengthField(byte[] bytes, int len) {\n if (len < 127) {\n bytes[1] = (byte) len;\n return 2;\n }\n\n int lenOfLenField = getLengthOfLengthField(len);\n bytes[1] = (byte) ((lenOfLenField - 1) | 0x80); // record length of the length field\n\n for (int i = lenOfLenField; i >= 2; i--) { // write the length\n bytes[i] = (byte) (len >> ((lenOfLenField - i) * 8));\n }\n\n return lenOfLenField + 1;\n }\n\n private static int getLengthOfLengthField(int len) {\n if (len <= 127) { // highest bit is zero, one byte is enough\n return 1;\n } else if (len <= 0xFF) { // highest bit is 1, two bytes in the form {0x81, 0xab}\n return 2;\n } else if (len <= 0xFFFF) { // three bytes in the form {0x82, 0xab, 0xcd}\n return 3;\n } else if (len <= 0xFFFFFF) { // four bytes in the form {0x83, 0xab, 0xcd, 0xef}\n return 4;\n } else { // five bytes in the form {0x84, 0xab, 0xcd, 0xef, 0xgh}\n return 5;\n }\n }\n\n private static int getOIDCompLength(int comp) {\n if (comp <= 0x7F) {\n return 1;\n } else if (comp <= 0x3FFF) {\n return 2;\n } else if (comp <= 0x1FFFFF) {\n return 3;\n } else if (comp <= 0xFFFFFFF) {\n return 4;\n } else {\n return 5;\n }\n }\n\n private static int writeOIDComp(int comp, byte[] bytes, int offset) {\n int len = getOIDCompLength(comp);\n int off = offset;\n for (int i = len - 1; i > 0; i--) {\n bytes[off++] = (byte) ((comp >>> i * 7) | 0x80);\n }\n\n bytes[off++] = (byte) (comp & 0x7F);\n\n return off;\n }\n}\n" }, { "path": "frontend/server/src/main/java/com/amazonaws/ml/mms/util/ServerGroups.java", "content": "/*\n * Copyright 2018 Amazon.com, Inc. or its affiliates. All Rights Reserved.\n *\n * Licensed under the Apache License, Version 2.0 (the \"License\"). You may not use this file except in compliance\n * with the License. A copy of the License is located at\n *\n * http://aws.amazon.com/apache2.0/\n *\n * or in the \"license\" file accompanying this file. This file is distributed on an \"AS IS\" BASIS, WITHOUT WARRANTIES\n * OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions\n * and limitations under the License.\n */\npackage com.amazonaws.ml.mms.util;\n\nimport io.netty.channel.Channel;\nimport io.netty.channel.ChannelFuture;\nimport io.netty.channel.EventLoopGroup;\nimport io.netty.channel.group.ChannelGroup;\nimport io.netty.channel.group.ChannelGroupFuture;\nimport io.netty.channel.group.DefaultChannelGroup;\nimport io.netty.util.concurrent.GlobalEventExecutor;\nimport java.util.ArrayList;\nimport java.util.List;\nimport java.util.concurrent.TimeUnit;\nimport org.slf4j.Logger;\nimport org.slf4j.LoggerFactory;\n\npublic class ServerGroups {\n\n static final Logger logger = LoggerFactory.getLogger(ServerGroups.class);\n\n private ChannelGroup allChannels;\n\n private EventLoopGroup serverGroup;\n private EventLoopGroup childGroup;\n private EventLoopGroup backendGroup;\n\n private ConfigManager configManager;\n\n public ServerGroups(ConfigManager configManager) {\n this.configManager = configManager;\n init();\n }\n\n public final void init() {\n allChannels = new DefaultChannelGroup(GlobalEventExecutor.INSTANCE);\n\n serverGroup = Connector.newEventLoopGroup(2);\n childGroup = Connector.newEventLoopGroup(configManager.getNettyThreads());\n backendGroup = Connector.newEventLoopGroup(configManager.getNettyClientThreads());\n }\n\n public void shutdown(boolean graceful) {\n closeAllChannels(graceful);\n\n List allEventLoopGroups = new ArrayList<>();\n\n allEventLoopGroups.add(serverGroup);\n allEventLoopGroups.add(childGroup);\n\n for (EventLoopGroup group : allEventLoopGroups) {\n if (graceful) {\n group.shutdownGracefully();\n } else {\n group.shutdownGracefully(0, 0, TimeUnit.SECONDS);\n }\n }\n\n if (graceful) {\n for (EventLoopGroup group : allEventLoopGroups) {\n try {\n group.awaitTermination(60, TimeUnit.SECONDS);\n } catch (InterruptedException e) {\n Thread.currentThread().interrupt();\n }\n }\n }\n }\n\n public EventLoopGroup getServerGroup() {\n return serverGroup;\n }\n\n public EventLoopGroup getChildGroup() {\n return childGroup;\n }\n\n public EventLoopGroup getBackendGroup() {\n return backendGroup;\n }\n\n public void registerChannel(Channel channel) {\n allChannels.add(channel);\n }\n\n private void closeAllChannels(boolean graceful) {\n ChannelGroupFuture future = allChannels.close();\n\n // if this is a graceful shutdown, log any channel closing failures. if this isn't a\n // graceful shutdown, ignore them.\n if (graceful) {\n try {\n future.await(10, TimeUnit.SECONDS);\n } catch (InterruptedException e) {\n Thread.currentThread().interrupt();\n }\n\n if (!future.isSuccess()) {\n for (ChannelFuture cf : future) {\n if (!cf.isSuccess()) {\n logger.info(\"Unable to close channel: \" + cf.channel(), cf.cause());\n }\n }\n }\n }\n }\n}\n" }, { "path": "frontend/server/src/main/java/com/amazonaws/ml/mms/util/codec/CodecUtils.java", "content": "/*\n * Copyright 2018 Amazon.com, Inc. or its affiliates. All Rights Reserved.\n *\n * Licensed under the Apache License, Version 2.0 (the \"License\"). You may not use this file except in compliance\n * with the License. A copy of the License is located at\n *\n * http://aws.amazon.com/apache2.0/\n *\n * or in the \"license\" file accompanying this file. This file is distributed on an \"AS IS\" BASIS, WITHOUT WARRANTIES\n * OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions\n * and limitations under the License.\n */\npackage com.amazonaws.ml.mms.util.codec;\n\nimport io.netty.buffer.ByteBuf;\nimport io.netty.handler.codec.CorruptedFrameException;\nimport java.nio.charset.StandardCharsets;\nimport java.util.HashMap;\nimport java.util.Map;\n\npublic final class CodecUtils {\n\n public static final int END = -1;\n public static final int BUFFER_UNDER_RUN = -3;\n\n private CodecUtils() {}\n\n static int readLength(ByteBuf byteBuf, int maxLength) {\n int size = byteBuf.readableBytes();\n if (size < 4) {\n return BUFFER_UNDER_RUN;\n }\n\n int len = byteBuf.readInt();\n if (len > maxLength) {\n throw new CorruptedFrameException(\"Message size exceed limit: \" + len);\n }\n if (len > byteBuf.readableBytes()) {\n return BUFFER_UNDER_RUN;\n }\n return len;\n }\n\n static String readString(ByteBuf byteBuf, int len) {\n return new String(read(byteBuf, len), StandardCharsets.UTF_8);\n }\n\n static byte[] read(ByteBuf in, int len) {\n if (len < 0) {\n throw new CorruptedFrameException(\"Invalid message size: \" + len);\n }\n\n byte[] buf = new byte[len];\n in.readBytes(buf);\n return buf;\n }\n\n static Map readMap(ByteBuf in, int len) {\n HashMap ret = new HashMap<>();\n for (; len > 0; len--) {\n int l = readLength(in, in.readableBytes());\n String key = readString(in, l);\n l = readLength(in, in.readableBytes());\n String val = readString(in, l);\n ret.put(key, val);\n }\n return ret;\n }\n}\n" }, { "path": "frontend/server/src/main/java/com/amazonaws/ml/mms/util/codec/ModelRequestEncoder.java", "content": "/*\n * Copyright 2018 Amazon.com, Inc. or its affiliates. All Rights Reserved.\n *\n * Licensed under the Apache License, Version 2.0 (the \"License\"). You may not use this file except in compliance\n * with the License. A copy of the License is located at\n *\n * http://aws.amazon.com/apache2.0/\n *\n * or in the \"license\" file accompanying this file. This file is distributed on an \"AS IS\" BASIS, WITHOUT WARRANTIES\n * OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions\n * and limitations under the License.\n */\npackage com.amazonaws.ml.mms.util.codec;\n\nimport com.amazonaws.ml.mms.util.messages.BaseModelRequest;\nimport com.amazonaws.ml.mms.util.messages.InputParameter;\nimport com.amazonaws.ml.mms.util.messages.ModelInferenceRequest;\nimport com.amazonaws.ml.mms.util.messages.ModelLoadModelRequest;\nimport com.amazonaws.ml.mms.util.messages.RequestInput;\nimport io.netty.buffer.ByteBuf;\nimport io.netty.channel.ChannelHandler;\nimport io.netty.channel.ChannelHandlerContext;\nimport io.netty.handler.codec.MessageToByteEncoder;\nimport java.nio.charset.StandardCharsets;\nimport java.util.Map;\n\n@ChannelHandler.Sharable\npublic class ModelRequestEncoder extends MessageToByteEncoder {\n\n public ModelRequestEncoder(boolean preferDirect) {\n super(preferDirect);\n }\n\n @Override\n protected void encode(ChannelHandlerContext ctx, BaseModelRequest msg, ByteBuf out) {\n if (msg instanceof ModelLoadModelRequest) {\n out.writeByte('L');\n\n ModelLoadModelRequest request = (ModelLoadModelRequest) msg;\n byte[] buf = msg.getModelName().getBytes(StandardCharsets.UTF_8);\n out.writeInt(buf.length);\n out.writeBytes(buf);\n\n buf = request.getModelPath().getBytes(StandardCharsets.UTF_8);\n out.writeInt(buf.length);\n out.writeBytes(buf);\n\n int batchSize = request.getBatchSize();\n if (batchSize <= 0) {\n batchSize = 1;\n }\n out.writeInt(batchSize);\n\n buf = request.getHandler().getBytes(StandardCharsets.UTF_8);\n out.writeInt(buf.length);\n out.writeBytes(buf);\n\n out.writeInt(request.getGpuId());\n\n buf = request.getIoFileDescriptor().getBytes(StandardCharsets.UTF_8);\n out.writeInt(buf.length);\n out.writeBytes(buf);\n } else if (msg instanceof ModelInferenceRequest) {\n out.writeByte('I');\n ModelInferenceRequest request = (ModelInferenceRequest) msg;\n for (RequestInput input : request.getRequestBatch()) {\n encodeRequest(input, out);\n }\n out.writeInt(-1); // End of List\n }\n }\n\n private void encodeRequest(RequestInput req, ByteBuf out) {\n byte[] buf = req.getRequestId().getBytes(StandardCharsets.UTF_8);\n out.writeInt(buf.length);\n out.writeBytes(buf);\n\n for (Map.Entry entry : req.getHeaders().entrySet()) {\n encodeField(entry.getKey(), out);\n encodeField(entry.getValue(), out);\n }\n out.writeInt(-1); // End of List\n\n for (InputParameter input : req.getParameters()) {\n encodeParameter(input, out);\n }\n out.writeInt(-1); // End of List\n }\n\n private void encodeParameter(InputParameter parameter, ByteBuf out) {\n byte[] modelInputName = parameter.getName().getBytes(StandardCharsets.UTF_8);\n out.writeInt(modelInputName.length);\n out.writeBytes(modelInputName);\n\n encodeField(parameter.getContentType(), out);\n\n byte[] buf = parameter.getValue();\n out.writeInt(buf.length);\n out.writeBytes(buf);\n }\n\n private static void encodeField(CharSequence field, ByteBuf out) {\n if (field == null) {\n out.writeInt(0);\n return;\n }\n byte[] buf = field.toString().getBytes(StandardCharsets.UTF_8);\n out.writeInt(buf.length);\n out.writeBytes(buf);\n }\n}\n" }, { "path": "frontend/server/src/main/java/com/amazonaws/ml/mms/util/codec/ModelResponseDecoder.java", "content": "/*\n * Copyright 2018 Amazon.com, Inc. or its affiliates. All Rights Reserved.\n *\n * Licensed under the Apache License, Version 2.0 (the \"License\"). You may not use this file except in compliance\n * with the License. A copy of the License is located at\n *\n * http://aws.amazon.com/apache2.0/\n *\n * or in the \"license\" file accompanying this file. This file is distributed on an \"AS IS\" BASIS, WITHOUT WARRANTIES\n * OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions\n * and limitations under the License.\n */\npackage com.amazonaws.ml.mms.util.codec;\n\nimport com.amazonaws.ml.mms.util.messages.ModelWorkerResponse;\nimport com.amazonaws.ml.mms.util.messages.Predictions;\nimport io.netty.buffer.ByteBuf;\nimport io.netty.channel.ChannelHandlerContext;\nimport io.netty.handler.codec.ByteToMessageDecoder;\nimport java.util.ArrayList;\nimport java.util.List;\n\npublic class ModelResponseDecoder extends ByteToMessageDecoder {\n\n private final int maxBufferSize;\n\n public ModelResponseDecoder(int maxBufferSize) {\n this.maxBufferSize = maxBufferSize;\n }\n\n @Override\n protected void decode(ChannelHandlerContext ctx, ByteBuf in, List out) {\n int size = in.readableBytes();\n if (size < 9) {\n return;\n }\n\n in.markReaderIndex();\n boolean completed = false;\n try {\n ModelWorkerResponse resp = new ModelWorkerResponse();\n // Get Response overall Code\n resp.setCode(in.readInt());\n\n int len = CodecUtils.readLength(in, maxBufferSize);\n if (len == CodecUtils.BUFFER_UNDER_RUN) {\n return;\n }\n resp.setMessage(CodecUtils.readString(in, len));\n\n List predictions = new ArrayList<>();\n while ((len = CodecUtils.readLength(in, maxBufferSize)) != CodecUtils.END) {\n if (len == CodecUtils.BUFFER_UNDER_RUN) {\n return;\n }\n Predictions prediction = new Predictions();\n // Set response RequestId\n prediction.setRequestId(CodecUtils.readString(in, len));\n\n len = CodecUtils.readLength(in, maxBufferSize);\n if (len == CodecUtils.BUFFER_UNDER_RUN) {\n return;\n }\n // Set content type\n prediction.setContentType(CodecUtils.readString(in, len));\n\n // Set per request response code\n int httpStatusCode = in.readInt();\n prediction.setStatusCode(httpStatusCode);\n\n // Set the actual message\n len = CodecUtils.readLength(in, maxBufferSize);\n if (len == CodecUtils.BUFFER_UNDER_RUN) {\n return;\n }\n prediction.setReasonPhrase(CodecUtils.readString(in, len));\n\n len = CodecUtils.readLength(in, maxBufferSize);\n if (len == CodecUtils.BUFFER_UNDER_RUN) {\n return;\n }\n prediction.setHeaders(CodecUtils.readMap(in, len));\n\n len = CodecUtils.readLength(in, maxBufferSize);\n if (len == CodecUtils.BUFFER_UNDER_RUN) {\n return;\n }\n prediction.setResp(CodecUtils.read(in, len));\n predictions.add(prediction);\n }\n resp.setPredictions(predictions);\n out.add(resp);\n completed = true;\n } finally {\n if (!completed) {\n in.resetReaderIndex();\n }\n }\n }\n}\n" }, { "path": "frontend/server/src/main/java/com/amazonaws/ml/mms/util/logging/QLogLayout.java", "content": "/*\n * Copyright 2018 Amazon.com, Inc. or its affiliates. All Rights Reserved.\n *\n * Licensed under the Apache License, Version 2.0 (the \"License\"). You may not use this file except in compliance\n * with the License. A copy of the License is located at\n *\n * http://aws.amazon.com/apache2.0/\n *\n * or in the \"license\" file accompanying this file. This file is distributed on an \"AS IS\" BASIS, WITHOUT WARRANTIES\n * OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions\n * and limitations under the License.\n */\npackage com.amazonaws.ml.mms.util.logging;\n\nimport com.amazonaws.ml.mms.metrics.Dimension;\nimport com.amazonaws.ml.mms.metrics.Metric;\nimport org.apache.logging.log4j.core.Layout;\nimport org.apache.logging.log4j.core.LogEvent;\nimport org.apache.logging.log4j.core.config.Node;\nimport org.apache.logging.log4j.core.config.plugins.Plugin;\nimport org.apache.logging.log4j.core.config.plugins.PluginFactory;\nimport org.apache.logging.log4j.core.layout.AbstractStringLayout;\nimport org.apache.logging.log4j.message.Message;\n\n@Plugin(\n name = \"QLogLayout\",\n category = Node.CATEGORY,\n elementType = Layout.ELEMENT_TYPE,\n printObject = true)\npublic class QLogLayout extends AbstractStringLayout {\n\n public QLogLayout() {\n super(null, null, null);\n }\n\n /**\n * Model server also supports query log formatting.\n *\n *

To enable Query Log format, change the layout as follows\n *\n * 

\n     *     log4j.appender.model_metrics.layout = com.amazonaws.ml.mms.util.logging.QLogLayout\n     *
\n *\n * This enables logs which are shown as following\n *\n * 
\n     *     HostName=hostName\n     *     RequestId=004bd136-063c-4102-a070-d7aff5add939\n     *     Marketplace=US\n     *     StartTime=1542275707\n     *     Program=MXNetModelServer\n     *     Metrics=PredictionTime=45 Milliseconds ModelName|squeezenet  Level|Model\n     *     EOE\n     *
\n *\n * Note: The following entities in this metrics can be customized.\n *\n *
    \n *
  • Marketplace : This can be customized by setting the \"REALM\" system environment\n * variable.\n *
  • Program : This entity can be customized by setting \"MXNETMODELSERVER_PROGRAM\"\n * environment variable.\n *
\n *\n * Example: If the above environment variables are set to the following,\n *\n * 
\n     *     $ env\n     *     REALM=someRealm\n     *     MXNETMODELSERVER_PROGRAM=someProgram\n     *
\n *\n * This produces the metrics as follows\n *\n * 
\n     *    HostName=hostName\n     *    RequestId=004bd136-063c-4102-a070-d7aff5add939\n     *    Marketplace=someRealm\n     *    StartTime=1542275707\n     *    Program=someProgram\n     *    Metrics=PredictionTime=45 Milliseconds ModelName|squeezenet  Level|Model\n     *    EOE\n     *
\n *\n * @param event\n * @return\n */\n @Override\n public String toSerializable(LogEvent event) {\n Message eventMessage = event.getMessage();\n if (eventMessage == null || eventMessage.getParameters() == null) {\n return null;\n }\n String programName =\n getStringOrDefault(System.getenv(\"MXNETMODELSERVER_PROGRAM\"), \"MXNetModelServer\");\n String domain = getStringOrDefault(System.getenv(\"DOMAIN\"), \"Unknown\");\n long currentTimeInSec = System.currentTimeMillis() / 1000;\n Object[] parameters = eventMessage.getParameters();\n\n StringBuilder stringBuilder = new StringBuilder();\n for (Object obj : parameters) {\n if (obj instanceof Metric) {\n Metric metric = (Metric) obj;\n String marketPlace = System.getenv(\"REALM\");\n\n stringBuilder.append(\"HostName=\").append(metric.getHostName());\n\n if (metric.getRequestId() != null && !metric.getRequestId().isEmpty()) {\n stringBuilder.append(\"\\nRequestId=\").append(metric.getRequestId());\n }\n\n // Marketplace format should be : ::\n if (marketPlace != null && !marketPlace.isEmpty()) {\n stringBuilder\n .append(\"\\nMarketplace=\")\n .append(programName)\n .append(':')\n .append(domain)\n .append(':')\n .append(marketPlace);\n }\n\n stringBuilder\n .append(\"\\nStartTime=\")\n .append(\n getStringOrDefault(\n metric.getTimestamp(), Long.toString(currentTimeInSec)));\n\n stringBuilder\n .append(\"\\nProgram=\")\n .append(programName)\n .append(\"\\nMetrics=\")\n .append(metric.getMetricName())\n .append('=')\n .append(metric.getValue())\n .append(' ')\n .append(metric.getUnit());\n for (Dimension dimension : metric.getDimensions()) {\n stringBuilder\n .append(' ')\n .append(dimension.getName())\n .append('|')\n .append(dimension.getValue())\n .append(' ');\n }\n stringBuilder.append(\"\\nEOE\\n\");\n }\n }\n return stringBuilder.toString();\n }\n\n @PluginFactory\n public static QLogLayout createLayout() {\n return new QLogLayout();\n }\n\n private static String getStringOrDefault(String val, String defVal) {\n\n if (val == null || val.isEmpty()) {\n return defVal;\n }\n return val;\n }\n}\n" }, { "path": "frontend/server/src/main/java/com/amazonaws/ml/mms/util/messages/BaseModelRequest.java", "content": "/*\n * Copyright 2018 Amazon.com, Inc. or its affiliates. All Rights Reserved.\n *\n * Licensed under the Apache License, Version 2.0 (the \"License\"). You may not use this file except in compliance\n * with the License. A copy of the License is located at\n *\n * http://aws.amazon.com/apache2.0/\n *\n * or in the \"license\" file accompanying this file. This file is distributed on an \"AS IS\" BASIS, WITHOUT WARRANTIES\n * OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions\n * and limitations under the License.\n */\npackage com.amazonaws.ml.mms.util.messages;\n\npublic class BaseModelRequest {\n\n private WorkerCommands command;\n private String modelName;\n\n public BaseModelRequest() {}\n\n public BaseModelRequest(WorkerCommands command, String modelName) {\n this.command = command;\n this.modelName = modelName;\n }\n\n public WorkerCommands getCommand() {\n return command;\n }\n\n public String getModelName() {\n return modelName;\n }\n}\n" }, { "path": "frontend/server/src/main/java/com/amazonaws/ml/mms/util/messages/InputParameter.java", "content": "/*\n * Copyright 2018 Amazon.com, Inc. or its affiliates. All Rights Reserved.\n *\n * Licensed under the Apache License, Version 2.0 (the \"License\"). You may not use this file except in compliance\n * with the License. A copy of the License is located at\n *\n * http://aws.amazon.com/apache2.0/\n *\n * or in the \"license\" file accompanying this file. This file is distributed on an \"AS IS\" BASIS, WITHOUT WARRANTIES\n * OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions\n * and limitations under the License.\n */\npackage com.amazonaws.ml.mms.util.messages;\n\nimport java.nio.charset.StandardCharsets;\n\npublic class InputParameter {\n\n private String name;\n private byte[] value;\n private CharSequence contentType;\n\n public InputParameter() {}\n\n public InputParameter(String name, String value) {\n this.name = name;\n this.value = value.getBytes(StandardCharsets.UTF_8);\n }\n\n public InputParameter(String name, byte[] data) {\n this(name, data, null);\n }\n\n public InputParameter(String name, byte[] data, CharSequence contentType) {\n this.name = name;\n this.contentType = contentType;\n this.value = data;\n }\n\n public String getName() {\n return name;\n }\n\n public byte[] getValue() {\n return value;\n }\n\n public CharSequence getContentType() {\n return contentType;\n }\n}\n" }, { "path": "frontend/server/src/main/java/com/amazonaws/ml/mms/util/messages/ModelInferenceRequest.java", "content": "/*\n * Copyright 2018 Amazon.com, Inc. or its affiliates. All Rights Reserved.\n *\n * Licensed under the Apache License, Version 2.0 (the \"License\"). You may not use this file except in compliance\n * with the License. A copy of the License is located at\n *\n * http://aws.amazon.com/apache2.0/\n *\n * or in the \"license\" file accompanying this file. This file is distributed on an \"AS IS\" BASIS, WITHOUT WARRANTIES\n * OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions\n * and limitations under the License.\n */\npackage com.amazonaws.ml.mms.util.messages;\n\nimport java.util.ArrayList;\nimport java.util.List;\n\npublic class ModelInferenceRequest extends BaseModelRequest {\n\n private List batch;\n\n public ModelInferenceRequest(String modelName) {\n super(WorkerCommands.PREDICT, modelName);\n batch = new ArrayList<>();\n }\n\n public List getRequestBatch() {\n return batch;\n }\n\n public void setRequestBatch(List requestBatch) {\n this.batch = requestBatch;\n }\n\n public void addRequest(RequestInput req) {\n batch.add(req);\n }\n}\n" }, { "path": "frontend/server/src/main/java/com/amazonaws/ml/mms/util/messages/ModelLoadModelRequest.java", "content": "/*\n * Copyright 2018 Amazon.com, Inc. or its affiliates. All Rights Reserved.\n *\n * Licensed under the Apache License, Version 2.0 (the \"License\"). You may not use this file except in compliance\n * with the License. A copy of the License is located at\n *\n * http://aws.amazon.com/apache2.0/\n *\n * or in the \"license\" file accompanying this file. This file is distributed on an \"AS IS\" BASIS, WITHOUT WARRANTIES\n * OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions\n * and limitations under the License.\n */\npackage com.amazonaws.ml.mms.util.messages;\n\nimport com.amazonaws.ml.mms.wlm.Model;\n\npublic class ModelLoadModelRequest extends BaseModelRequest {\n\n /**\n * ModelLoadModelRequest is a interface between frontend and backend to notify the backend to\n * load a particular model.\n */\n private String modelPath;\n\n private String handler;\n private int batchSize;\n private int gpuId;\n private String ioFileDescriptor;\n\n public ModelLoadModelRequest(Model model, int gpuId, String fd) {\n super(WorkerCommands.LOAD, model.getModelName());\n this.gpuId = gpuId;\n modelPath = model.getModelDir().getAbsolutePath();\n handler = model.getModelArchive().getManifest().getModel().getHandler();\n batchSize = model.getBatchSize();\n ioFileDescriptor = fd;\n }\n\n public String getIoFileDescriptor() {\n return ioFileDescriptor;\n }\n\n public void setIoFileDescriptor(String ioFileDescriptor) {\n this.ioFileDescriptor = ioFileDescriptor;\n }\n\n public String getModelPath() {\n return modelPath;\n }\n\n public String getHandler() {\n return handler;\n }\n\n public int getBatchSize() {\n return batchSize;\n }\n\n public int getGpuId() {\n return gpuId;\n }\n}\n" }, { "path": "frontend/server/src/main/java/com/amazonaws/ml/mms/util/messages/ModelWorkerResponse.java", "content": "/*\n * Copyright 2018 Amazon.com, Inc. or its affiliates. All Rights Reserved.\n *\n * Licensed under the Apache License, Version 2.0 (the \"License\"). You may not use this file except in compliance\n * with the License. A copy of the License is located at\n *\n * http://aws.amazon.com/apache2.0/\n *\n * or in the \"license\" file accompanying this file. This file is distributed on an \"AS IS\" BASIS, WITHOUT WARRANTIES\n * OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions\n * and limitations under the License.\n */\npackage com.amazonaws.ml.mms.util.messages;\n\nimport java.util.List;\n\npublic class ModelWorkerResponse {\n\n private int code;\n private String message;\n private List predictions;\n\n public ModelWorkerResponse() {}\n\n public int getCode() {\n return code;\n }\n\n public void setCode(int code) {\n this.code = code;\n }\n\n public String getMessage() {\n return message;\n }\n\n public void setMessage(String message) {\n this.message = message;\n }\n\n public List getPredictions() {\n return predictions;\n }\n\n public void setPredictions(List predictions) {\n this.predictions = predictions;\n }\n\n public void appendPredictions(Predictions prediction) {\n this.predictions.add(prediction);\n }\n}\n" }, { "path": "frontend/server/src/main/java/com/amazonaws/ml/mms/util/messages/Predictions.java", "content": "/*\n * Copyright 2018 Amazon.com, Inc. or its affiliates. All Rights Reserved.\n *\n * Licensed under the Apache License, Version 2.0 (the \"License\"). You may not use this file except in compliance\n * with the License. A copy of the License is located at\n *\n * http://aws.amazon.com/apache2.0/\n *\n * or in the \"license\" file accompanying this file. This file is distributed on an \"AS IS\" BASIS, WITHOUT WARRANTIES\n * OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions\n * and limitations under the License.\n */\npackage com.amazonaws.ml.mms.util.messages;\n\nimport java.util.Map;\n\npublic class Predictions {\n\n private String requestId;\n private int statusCode;\n private String reasonPhrase;\n\n private String contentType;\n private Map headers;\n private byte[] resp;\n\n public Map getHeaders() {\n return headers;\n }\n\n public void setHeaders(Map headers) {\n this.headers = headers;\n }\n\n public Predictions() {}\n\n public String getRequestId() {\n return requestId;\n }\n\n public void setRequestId(String requestId) {\n this.requestId = requestId;\n }\n\n public byte[] getResp() {\n return resp;\n }\n\n public void setResp(byte[] resp) {\n this.resp = resp;\n }\n\n public String getContentType() {\n return contentType;\n }\n\n public void setStatusCode(int statusCode) {\n this.statusCode = statusCode;\n }\n\n public void setContentType(String contentType) {\n this.contentType = contentType;\n }\n\n public int getStatusCode() {\n return statusCode;\n }\n\n public String getReasonPhrase() {\n return reasonPhrase;\n }\n\n public void setReasonPhrase(String reasonPhrase) {\n this.reasonPhrase = reasonPhrase;\n }\n}\n" }, { "path": "frontend/server/src/main/java/com/amazonaws/ml/mms/util/messages/RequestInput.java", "content": "/*\n * Copyright 2018 Amazon.com, Inc. or its affiliates. All Rights Reserved.\n *\n * Licensed under the Apache License, Version 2.0 (the \"License\"). You may not use this file except in compliance\n * with the License. A copy of the License is located at\n *\n * http://aws.amazon.com/apache2.0/\n *\n * or in the \"license\" file accompanying this file. This file is distributed on an \"AS IS\" BASIS, WITHOUT WARRANTIES\n * OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions\n * and limitations under the License.\n */\npackage com.amazonaws.ml.mms.util.messages;\n\nimport java.nio.charset.StandardCharsets;\nimport java.util.ArrayList;\nimport java.util.HashMap;\nimport java.util.List;\nimport java.util.Map;\n\npublic class RequestInput {\n\n private String requestId;\n private Map headers;\n private List parameters;\n\n public RequestInput(String requestId) {\n this.requestId = requestId;\n headers = new HashMap<>();\n parameters = new ArrayList<>();\n }\n\n public String getRequestId() {\n return requestId;\n }\n\n public void setRequestId(String requestId) {\n this.requestId = requestId;\n }\n\n public Map getHeaders() {\n return headers;\n }\n\n public void setHeaders(Map headers) {\n this.headers = headers;\n }\n\n public void updateHeaders(String key, String val) {\n headers.put(key, val);\n }\n\n public List getParameters() {\n return parameters;\n }\n\n public void setParameters(List parameters) {\n this.parameters = parameters;\n }\n\n public void addParameter(InputParameter modelInput) {\n parameters.add(modelInput);\n }\n\n public String getStringParameter(String key) {\n for (InputParameter param : parameters) {\n if (key.equals(param.getName())) {\n return new String(param.getValue(), StandardCharsets.UTF_8);\n }\n }\n return null;\n }\n}\n" }, { "path": "frontend/server/src/main/java/com/amazonaws/ml/mms/util/messages/WorkerCommands.java", "content": "package com.amazonaws.ml.mms.util.messages;\n\nimport com.google.gson.annotations.SerializedName;\n\npublic enum WorkerCommands {\n @SerializedName(\"predict\")\n PREDICT(\"predict\"),\n @SerializedName(\"load\")\n LOAD(\"load\"),\n @SerializedName(\"unload\")\n UNLOAD(\"unload\"),\n @SerializedName(\"stats\")\n STATS(\"stats\");\n\n private String command;\n\n WorkerCommands(String command) {\n this.command = command;\n }\n\n public String getCommand() {\n return command;\n }\n}\n" }, { "path": "frontend/server/src/main/java/com/amazonaws/ml/mms/wlm/BatchAggregator.java", "content": "/*\n * Copyright 2018 Amazon.com, Inc. or its affiliates. All Rights Reserved.\n *\n * Licensed under the Apache License, Version 2.0 (the \"License\"). You may not use this file except in compliance\n * with the License. A copy of the License is located at\n *\n * http://aws.amazon.com/apache2.0/\n *\n * or in the \"license\" file accompanying this file. This file is distributed on an \"AS IS\" BASIS, WITHOUT WARRANTIES\n * OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions\n * and limitations under the License.\n */\npackage com.amazonaws.ml.mms.wlm;\n\nimport com.amazonaws.ml.mms.util.messages.BaseModelRequest;\nimport com.amazonaws.ml.mms.util.messages.ModelInferenceRequest;\nimport com.amazonaws.ml.mms.util.messages.ModelLoadModelRequest;\nimport com.amazonaws.ml.mms.util.messages.ModelWorkerResponse;\nimport com.amazonaws.ml.mms.util.messages.Predictions;\nimport com.amazonaws.ml.mms.util.messages.RequestInput;\nimport io.netty.handler.codec.http.HttpResponseStatus;\nimport java.util.LinkedHashMap;\nimport java.util.Map;\nimport org.slf4j.Logger;\nimport org.slf4j.LoggerFactory;\n\npublic class BatchAggregator {\n\n private static final Logger logger = LoggerFactory.getLogger(BatchAggregator.class);\n\n private Model model;\n private Map jobs;\n\n public BatchAggregator(Model model) {\n this.model = model;\n jobs = new LinkedHashMap<>();\n }\n\n public BaseModelRequest getRequest(String threadName, WorkerState state)\n throws InterruptedException {\n jobs.clear();\n\n ModelInferenceRequest req = new ModelInferenceRequest(model.getModelName());\n\n model.pollBatch(\n threadName, (state == WorkerState.WORKER_MODEL_LOADED) ? 0 : Long.MAX_VALUE, jobs);\n\n for (Job j : jobs.values()) {\n if (j.isControlCmd()) {\n if (jobs.size() > 1) {\n throw new IllegalStateException(\n \"Received more than 1 control command. \"\n + \"Control messages should be processed/retrieved one at a time.\");\n }\n RequestInput input = j.getPayload();\n int gpuId = -1;\n String gpu = input.getStringParameter(\"gpu\");\n if (gpu != null) {\n gpuId = Integer.parseInt(gpu);\n }\n return new ModelLoadModelRequest(model, gpuId, threadName);\n } else {\n j.setScheduled();\n req.addRequest(j.getPayload());\n }\n }\n return req;\n }\n\n public void sendResponse(ModelWorkerResponse message) {\n // TODO: Handle prediction level code\n\n if (message.getCode() == 200) {\n if (jobs.isEmpty()) {\n // this is from initial load.\n return;\n }\n\n for (Predictions prediction : message.getPredictions()) {\n String jobId = prediction.getRequestId();\n Job job = jobs.remove(jobId);\n if (job == null) {\n throw new IllegalStateException(\"Unexpected job: \" + jobId);\n }\n job.response(\n prediction.getResp(),\n prediction.getContentType(),\n prediction.getStatusCode(),\n prediction.getReasonPhrase(),\n prediction.getHeaders());\n }\n } else {\n for (String reqId : jobs.keySet()) {\n Job j = jobs.remove(reqId);\n if (j == null) {\n throw new IllegalStateException(\"Unexpected job: \" + reqId);\n }\n j.sendError(HttpResponseStatus.valueOf(message.getCode()), message.getMessage());\n }\n if (!jobs.isEmpty()) {\n throw new IllegalStateException(\"Not all jobs get response.\");\n }\n }\n }\n\n public void sendError(BaseModelRequest message, String error, HttpResponseStatus status) {\n if (message instanceof ModelLoadModelRequest) {\n logger.warn(\"Load model failed: {}, error: {}\", message.getModelName(), error);\n return;\n }\n\n if (message != null) {\n ModelInferenceRequest msg = (ModelInferenceRequest) message;\n for (RequestInput req : msg.getRequestBatch()) {\n String requestId = req.getRequestId();\n Job job = jobs.remove(requestId);\n if (job == null) {\n logger.error(\"Unexpected job: \" + requestId);\n } else {\n job.sendError(status, error);\n }\n }\n if (!jobs.isEmpty()) {\n jobs.clear();\n logger.error(\"Not all jobs get response.\");\n }\n } else {\n // Send the error message to all the jobs\n for (Map.Entry j : jobs.entrySet()) {\n String jobsId = j.getValue().getJobId();\n Job job = jobs.remove(jobsId);\n\n if (job.isControlCmd()) {\n job.sendError(status, error);\n } else {\n // Data message can be handled by other workers.\n // If batch has gone past its batch max delay timer?\n model.addFirst(job);\n }\n }\n }\n }\n}\n" }, { "path": "frontend/server/src/main/java/com/amazonaws/ml/mms/wlm/Job.java", "content": "/*\n * Copyright 2018 Amazon.com, Inc. or its affiliates. All Rights Reserved.\n *\n * Licensed under the Apache License, Version 2.0 (the \"License\"). You may not use this file except in compliance\n * with the License. A copy of the License is located at\n *\n * http://aws.amazon.com/apache2.0/\n *\n * or in the \"license\" file accompanying this file. This file is distributed on an \"AS IS\" BASIS, WITHOUT WARRANTIES\n * OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions\n * and limitations under the License.\n */\npackage com.amazonaws.ml.mms.wlm;\n\nimport com.amazonaws.ml.mms.http.InternalServerException;\nimport com.amazonaws.ml.mms.util.NettyUtils;\nimport com.amazonaws.ml.mms.util.messages.RequestInput;\nimport com.amazonaws.ml.mms.util.messages.WorkerCommands;\nimport io.netty.buffer.Unpooled;\nimport io.netty.channel.ChannelHandlerContext;\nimport io.netty.handler.codec.http.DefaultFullHttpResponse;\nimport io.netty.handler.codec.http.DefaultHttpHeadersFactory;\nimport io.netty.handler.codec.http.FullHttpResponse;\nimport io.netty.handler.codec.http.HttpHeaderNames;\nimport io.netty.handler.codec.http.HttpResponseStatus;\nimport io.netty.handler.codec.http.HttpVersion;\nimport java.util.Map;\nimport org.slf4j.Logger;\nimport org.slf4j.LoggerFactory;\n\npublic class Job {\n\n private static final Logger logger = LoggerFactory.getLogger(Job.class);\n\n private ChannelHandlerContext ctx;\n\n private String modelName;\n private WorkerCommands cmd; // Else its data msg or inf requests\n private RequestInput input;\n private long begin;\n private long scheduled;\n\n public Job(\n ChannelHandlerContext ctx, String modelName, WorkerCommands cmd, RequestInput input) {\n this.ctx = ctx;\n this.modelName = modelName;\n this.cmd = cmd;\n this.input = input;\n\n begin = System.currentTimeMillis();\n scheduled = begin;\n }\n\n public String getJobId() {\n return input.getRequestId();\n }\n\n public String getModelName() {\n return modelName;\n }\n\n public WorkerCommands getCmd() {\n return cmd;\n }\n\n public boolean isControlCmd() {\n return !WorkerCommands.PREDICT.equals(cmd);\n }\n\n public RequestInput getPayload() {\n return input;\n }\n\n public void setScheduled() {\n scheduled = System.currentTimeMillis();\n }\n\n public void response(\n byte[] body,\n CharSequence contentType,\n int statusCode,\n String statusPhrase,\n Map responseHeaders) {\n HttpResponseStatus status =\n (statusPhrase == null)\n ? HttpResponseStatus.valueOf(statusCode)\n : HttpResponseStatus.valueOf(statusCode, statusPhrase);\n FullHttpResponse resp =\n new DefaultFullHttpResponse(\n HttpVersion.HTTP_1_1,\n status,\n Unpooled.directBuffer(),\n DefaultHttpHeadersFactory.headersFactory(),\n DefaultHttpHeadersFactory.trailersFactory());\n\n if (contentType != null && contentType.length() > 0) {\n resp.headers().set(HttpHeaderNames.CONTENT_TYPE, contentType);\n }\n if (responseHeaders != null) {\n for (Map.Entry e : responseHeaders.entrySet()) {\n resp.headers().set(e.getKey(), e.getValue());\n }\n }\n resp.content().writeBytes(body);\n\n /*\n * We can load the models based on the configuration file.Since this Job is\n * not driven by the external connections, we could have a empty context for\n * this job. We shouldn't try to send a response to ctx if this is not triggered\n * by external clients.\n */\n if (ctx != null) {\n NettyUtils.sendHttpResponse(ctx, resp, true);\n }\n\n logger.debug(\n \"Waiting time: {}, Backend time: {}\",\n scheduled - begin,\n System.currentTimeMillis() - scheduled);\n }\n\n public void sendError(HttpResponseStatus status, String error) {\n /*\n * We can load the models based on the configuration file.Since this Job is\n * not driven by the external connections, we could have a empty context for\n * this job. We shouldn't try to send a response to ctx if this is not triggered\n * by external clients.\n */\n if (ctx != null) {\n NettyUtils.sendError(ctx, status, new InternalServerException(error));\n }\n\n logger.debug(\n \"Waiting time: {}, Inference time: {}\",\n scheduled - begin,\n System.currentTimeMillis() - begin);\n }\n}\n" }, { "path": "frontend/server/src/main/java/com/amazonaws/ml/mms/wlm/Model.java", "content": "/*\n * Copyright 2018 Amazon.com, Inc. or its affiliates. All Rights Reserved.\n *\n * Licensed under the Apache License, Version 2.0 (the \"License\"). You may not use this file except in compliance\n * with the License. A copy of the License is located at\n *\n * http://aws.amazon.com/apache2.0/\n *\n * or in the \"license\" file accompanying this file. This file is distributed on an \"AS IS\" BASIS, WITHOUT WARRANTIES\n * OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions\n * and limitations under the License.\n */\npackage com.amazonaws.ml.mms.wlm;\n\nimport com.amazonaws.ml.mms.archive.ModelArchive;\nimport com.amazonaws.ml.mms.util.ConfigManager;\nimport java.io.File;\nimport java.util.Map;\nimport java.util.Objects;\nimport java.util.concurrent.ConcurrentHashMap;\nimport java.util.concurrent.ConcurrentMap;\nimport java.util.concurrent.LinkedBlockingDeque;\nimport java.util.concurrent.TimeUnit;\nimport java.util.concurrent.atomic.AtomicInteger;\nimport java.util.concurrent.locks.ReentrantLock;\nimport org.slf4j.Logger;\nimport org.slf4j.LoggerFactory;\n\npublic class Model {\n\n public static final String DEFAULT_DATA_QUEUE = \"DATA_QUEUE\";\n private static final Logger logger = LoggerFactory.getLogger(Model.class);\n\n private ModelArchive modelArchive;\n private int minWorkers;\n private int maxWorkers;\n private int batchSize;\n private int maxBatchDelay;\n private String preloadModel;\n private AtomicInteger port; // Port on which the model server is running\n private ReentrantLock lock;\n private int responseTimeoutSeconds;\n private WorkerThread serverThread;\n // Total number of subsequent inference request failures\n private AtomicInteger failedInfReqs;\n\n // Per worker thread job queue. This separates out the control queue from data queue\n private ConcurrentMap> jobsDb;\n\n public Model(ModelArchive modelArchive, int queueSize, String preloadModel) {\n this.modelArchive = modelArchive;\n this.preloadModel = preloadModel;\n batchSize = 1;\n maxBatchDelay = 100;\n jobsDb = new ConcurrentHashMap<>();\n // Always have a queue for data\n jobsDb.putIfAbsent(DEFAULT_DATA_QUEUE, new LinkedBlockingDeque<>(queueSize));\n failedInfReqs = new AtomicInteger(0);\n port = new AtomicInteger(-1);\n lock = new ReentrantLock();\n }\n\n public String getModelName() {\n return modelArchive.getModelName();\n }\n\n public File getModelDir() {\n return modelArchive.getModelDir();\n }\n\n public String getModelUrl() {\n return modelArchive.getUrl();\n }\n\n public ModelArchive getModelArchive() {\n return modelArchive;\n }\n\n public int getMinWorkers() {\n return minWorkers;\n }\n\n public void setMinWorkers(int minWorkers) {\n this.minWorkers = minWorkers;\n }\n\n public int getMaxWorkers() {\n return maxWorkers;\n }\n\n public void setMaxWorkers(int maxWorkers) {\n this.maxWorkers = maxWorkers;\n }\n\n public int getBatchSize() {\n return batchSize;\n }\n\n public void setBatchSize(int batchSize) {\n this.batchSize = batchSize;\n }\n\n public int getMaxBatchDelay() {\n return maxBatchDelay;\n }\n\n public void setMaxBatchDelay(int maxBatchDelay) {\n this.maxBatchDelay = maxBatchDelay;\n }\n\n public void addJob(String threadId, Job job) {\n LinkedBlockingDeque blockingDeque = jobsDb.get(threadId);\n if (blockingDeque == null) {\n blockingDeque = new LinkedBlockingDeque<>();\n jobsDb.put(threadId, blockingDeque);\n }\n blockingDeque.offer(job);\n }\n\n public void removeJobQueue(String threadId) {\n if (!threadId.equals(DEFAULT_DATA_QUEUE)) {\n jobsDb.remove(threadId);\n }\n }\n\n public boolean addJob(Job job) {\n return jobsDb.get(DEFAULT_DATA_QUEUE).offer(job);\n }\n\n public void addFirst(Job job) {\n jobsDb.get(DEFAULT_DATA_QUEUE).addFirst(job);\n }\n\n public void pollBatch(String threadId, long waitTime, Map jobsRepo)\n throws InterruptedException {\n if (jobsRepo == null || threadId == null || threadId.isEmpty()) {\n throw new IllegalArgumentException(\"Invalid input given provided\");\n }\n\n if (!jobsRepo.isEmpty()) {\n throw new IllegalArgumentException(\n \"The jobs repo provided contains stale jobs. Clear them!!\");\n }\n\n LinkedBlockingDeque jobsQueue = jobsDb.get(threadId);\n if (jobsQueue != null && !jobsQueue.isEmpty()) {\n Job j = jobsQueue.poll(waitTime, TimeUnit.MILLISECONDS);\n if (j != null) {\n jobsRepo.put(j.getJobId(), j);\n return;\n }\n }\n\n try {\n lock.lockInterruptibly();\n long maxDelay = maxBatchDelay;\n jobsQueue = jobsDb.get(DEFAULT_DATA_QUEUE);\n\n Job j = jobsQueue.poll(Long.MAX_VALUE, TimeUnit.MILLISECONDS);\n logger.trace(\"get first job: {}\", Objects.requireNonNull(j).getJobId());\n\n jobsRepo.put(j.getJobId(), j);\n long begin = System.currentTimeMillis();\n for (int i = 0; i < batchSize - 1; ++i) {\n j = jobsQueue.poll(maxDelay, TimeUnit.MILLISECONDS);\n if (j == null) {\n break;\n }\n long end = System.currentTimeMillis();\n maxDelay -= end - begin;\n begin = end;\n jobsRepo.put(j.getJobId(), j);\n if (maxDelay <= 0) {\n break;\n }\n }\n logger.trace(\"sending jobs, size: {}\", jobsRepo.size());\n } finally {\n if (lock.isHeldByCurrentThread()) {\n lock.unlock();\n }\n }\n }\n\n public int getPort() {\n return port.get();\n }\n\n public void setPort(int port) {\n this.port.set(port);\n }\n\n public int incrFailedInfReqs() {\n return failedInfReqs.incrementAndGet();\n }\n\n public void resetFailedInfReqs() {\n failedInfReqs.set(0);\n }\n\n public int getResponseTimeoutSeconds() {\n return ConfigManager.getInstance().isDebug() ? Integer.MAX_VALUE : responseTimeoutSeconds;\n }\n\n public void setResponseTimeoutSeconds(int responseTimeoutSeconds) {\n this.responseTimeoutSeconds = responseTimeoutSeconds;\n }\n\n public WorkerThread getServerThread() {\n return serverThread;\n }\n\n public void setServerThread(WorkerThread serverThread) {\n this.serverThread = serverThread;\n }\n\n public String preloadModel() {\n return preloadModel;\n }\n}\n" }, { "path": "frontend/server/src/main/java/com/amazonaws/ml/mms/wlm/ModelManager.java", "content": "/*\n * Copyright 2018 Amazon.com, Inc. or its affiliates. All Rights Reserved.\n *\n * Licensed under the Apache License, Version 2.0 (the \"License\"). You may not use this file except in compliance\n * with the License. A copy of the License is located at\n *\n * http://aws.amazon.com/apache2.0/\n *\n * or in the \"license\" file accompanying this file. This file is distributed on an \"AS IS\" BASIS, WITHOUT WARRANTIES\n * OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions\n * and limitations under the License.\n */\npackage com.amazonaws.ml.mms.wlm;\n\nimport com.amazonaws.ml.mms.archive.Manifest;\nimport com.amazonaws.ml.mms.archive.ModelArchive;\nimport com.amazonaws.ml.mms.archive.ModelException;\nimport com.amazonaws.ml.mms.archive.ModelNotFoundException;\nimport com.amazonaws.ml.mms.http.ConflictStatusException;\nimport com.amazonaws.ml.mms.http.StatusResponse;\nimport com.amazonaws.ml.mms.util.ConfigManager;\nimport com.amazonaws.ml.mms.util.NettyUtils;\nimport io.netty.channel.ChannelHandlerContext;\nimport io.netty.handler.codec.http.HttpResponseStatus;\nimport java.io.IOException;\nimport java.util.HashSet;\nimport java.util.List;\nimport java.util.Map;\nimport java.util.Set;\nimport java.util.concurrent.CompletableFuture;\nimport java.util.concurrent.ConcurrentHashMap;\nimport java.util.concurrent.ExecutionException;\nimport java.util.concurrent.Executors;\nimport java.util.concurrent.ScheduledExecutorService;\nimport java.util.concurrent.TimeoutException;\nimport org.slf4j.Logger;\nimport org.slf4j.LoggerFactory;\n\npublic final class ModelManager {\n\n private static final Logger logger = LoggerFactory.getLogger(ModelManager.class);\n\n private static ModelManager modelManager;\n\n private ConfigManager configManager;\n private WorkLoadManager wlm;\n private ConcurrentHashMap models;\n private HashSet startupModels;\n private ScheduledExecutorService scheduler;\n\n private ModelManager(ConfigManager configManager, WorkLoadManager wlm) {\n this.configManager = configManager;\n this.wlm = wlm;\n models = new ConcurrentHashMap<>();\n scheduler = Executors.newScheduledThreadPool(2);\n this.startupModels = new HashSet<>();\n }\n\n public ScheduledExecutorService getScheduler() {\n return scheduler;\n }\n\n public static void init(ConfigManager configManager, WorkLoadManager wlm) {\n modelManager = new ModelManager(configManager, wlm);\n }\n\n public static ModelManager getInstance() {\n return modelManager;\n }\n\n public ModelArchive registerModel(String url, String defaultModelName, String preloadModel)\n throws ModelException, IOException, InterruptedException, ExecutionException,\n TimeoutException {\n return registerModel(\n url,\n null,\n null,\n null,\n 1,\n 100,\n configManager.getDefaultResponseTimeoutSeconds(),\n defaultModelName,\n preloadModel);\n }\n\n public ModelArchive registerModel(\n String url,\n String modelName,\n Manifest.RuntimeType runtime,\n String handler,\n int batchSize,\n int maxBatchDelay,\n int responseTimeoutSeconds,\n String defaultModelName,\n String preloadModel)\n throws ModelException, IOException, InterruptedException, ExecutionException,\n TimeoutException {\n\n ModelArchive archive = ModelArchive.downloadModel(configManager.getModelStore(), url);\n if (modelName == null || modelName.isEmpty()) {\n if (archive.getModelName() == null || archive.getModelName().isEmpty()) {\n archive.getManifest().getModel().setModelName(defaultModelName);\n }\n modelName = archive.getModelName();\n } else {\n archive.getManifest().getModel().setModelName(modelName);\n }\n if (runtime != null) {\n archive.getManifest().setRuntime(runtime);\n }\n if (handler != null) {\n archive.getManifest().getModel().setHandler(handler);\n } else if (archive.getHandler() == null || archive.getHandler().isEmpty()) {\n archive.getManifest()\n .getModel()\n .setHandler(configManager.getMmsDefaultServiceHandler());\n }\n\n archive.validate();\n\n Model model = new Model(archive, configManager.getJobQueueSize(), preloadModel);\n model.setBatchSize(batchSize);\n model.setMaxBatchDelay(maxBatchDelay);\n model.setResponseTimeoutSeconds(responseTimeoutSeconds);\n Model existingModel = models.putIfAbsent(modelName, model);\n if (existingModel != null) {\n // model already exists\n throw new ConflictStatusException(\"Model \" + modelName + \" is already registered.\");\n }\n\n if (configManager.isDebug()) {\n model.setPort(9000);\n } else {\n startBackendServer(model);\n }\n\n models.put(modelName, model);\n\n logger.info(\"Model {} loaded.\", model.getModelName());\n\n return archive;\n }\n\n public HttpResponseStatus unregisterModel(String modelName) {\n Model model = models.remove(modelName);\n if (model == null) {\n logger.warn(\"Model not found: \" + modelName);\n return HttpResponseStatus.NOT_FOUND;\n }\n model.setMinWorkers(0);\n model.setMaxWorkers(0);\n CompletableFuture futureStatus = wlm.modelChanged(model);\n HttpResponseStatus httpResponseStatus = HttpResponseStatus.OK;\n\n try {\n httpResponseStatus = futureStatus.get();\n } catch (InterruptedException | ExecutionException e) {\n logger.warn(\"Process was interrupted while cleaning resources.\");\n httpResponseStatus = HttpResponseStatus.INTERNAL_SERVER_ERROR;\n }\n\n // Only continue cleaning if resource cleaning succeeded\n if (httpResponseStatus == HttpResponseStatus.OK) {\n model.getModelArchive().clean();\n startupModels.remove(modelName);\n logger.info(\"Model {} unregistered.\", modelName);\n } else {\n models.put(modelName, model);\n }\n\n return httpResponseStatus;\n }\n\n public void startBackendServer(Model model)\n throws InterruptedException, ExecutionException, TimeoutException {\n CompletableFuture future = new CompletableFuture<>();\n if (model == null) {\n throw new AssertionError(\"Model not found\");\n }\n wlm.addServerThread(model, future);\n }\n\n public CompletableFuture updateModel(\n String modelName, int minWorkers, int maxWorkers) {\n Model model = models.get(modelName);\n if (model == null) {\n throw new AssertionError(\"Model not found: \" + modelName);\n }\n model.setMinWorkers(minWorkers);\n model.setMaxWorkers(maxWorkers);\n logger.debug(\"updateModel: {}, count: {}\", modelName, minWorkers);\n return wlm.modelChanged(model);\n }\n\n public Map getModels() {\n return models;\n }\n\n public List getWorkers(String modelName) {\n return wlm.getWorkers(modelName);\n }\n\n public Map getWorkers() {\n return wlm.getWorkers();\n }\n\n public boolean addJob(Job job) throws ModelNotFoundException {\n String modelName = job.getModelName();\n Model model = models.get(modelName);\n if (model == null) {\n throw new ModelNotFoundException(\"Model not found: \" + modelName);\n }\n\n if (wlm.hasNoWorker(modelName)) {\n return false;\n }\n\n return model.addJob(job);\n }\n\n public void workerStatus(final ChannelHandlerContext ctx) {\n Runnable r =\n () -> {\n String response = \"Healthy\";\n int numWorking = 0;\n int numScaled = 0;\n for (Map.Entry m : models.entrySet()) {\n numScaled += m.getValue().getMinWorkers();\n numWorking += wlm.getNumRunningWorkers(m.getValue().getModelName());\n }\n\n if ((numWorking > 0) && (numWorking < numScaled)) {\n response = \"Partial Healthy\";\n } else if ((numWorking == 0) && (numScaled > 0)) {\n response = \"Unhealthy\";\n }\n\n // TODO: Check if its OK to send other 2xx errors to ALB for \"Partial Healthy\"\n // and \"Unhealthy\"\n NettyUtils.sendJsonResponse(\n ctx, new StatusResponse(response), HttpResponseStatus.OK);\n };\n wlm.scheduleAsync(r);\n }\n\n public boolean scaleRequestStatus(String modelName) {\n Model model = ModelManager.getInstance().getModels().get(modelName);\n int numWorkers = wlm.getNumRunningWorkers(modelName);\n\n return model == null || model.getMinWorkers() <= numWorkers;\n }\n\n public void submitTask(Runnable runnable) {\n wlm.scheduleAsync(runnable);\n }\n\n public Set getStartupModels() {\n return startupModels;\n }\n}\n" }, { "path": "frontend/server/src/main/java/com/amazonaws/ml/mms/wlm/WorkLoadManager.java", "content": "/*\n * Copyright 2018 Amazon.com, Inc. or its affiliates. All Rights Reserved.\n *\n * Licensed under the Apache License, Version 2.0 (the \"License\"). You may not use this file except in compliance\n * with the License. A copy of the License is located at\n *\n * http://aws.amazon.com/apache2.0/\n *\n * or in the \"license\" file accompanying this file. This file is distributed on an \"AS IS\" BASIS, WITHOUT WARRANTIES\n * OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions\n * and limitations under the License.\n */\npackage com.amazonaws.ml.mms.wlm;\n\nimport com.amazonaws.ml.mms.util.ConfigManager;\nimport io.netty.channel.EventLoopGroup;\nimport io.netty.handler.codec.http.HttpResponseStatus;\nimport java.util.ArrayList;\nimport java.util.Collections;\nimport java.util.HashMap;\nimport java.util.List;\nimport java.util.Map;\nimport java.util.concurrent.CompletableFuture;\nimport java.util.concurrent.ConcurrentHashMap;\nimport java.util.concurrent.ExecutionException;\nimport java.util.concurrent.ExecutorService;\nimport java.util.concurrent.Executors;\nimport java.util.concurrent.TimeUnit;\nimport java.util.concurrent.TimeoutException;\nimport java.util.concurrent.atomic.AtomicInteger;\nimport org.slf4j.Logger;\nimport org.slf4j.LoggerFactory;\n\npublic class WorkLoadManager {\n\n private ExecutorService threadPool;\n\n private ConcurrentHashMap> workers;\n\n private ConfigManager configManager;\n private EventLoopGroup backendGroup;\n private AtomicInteger port;\n private AtomicInteger gpuCounter;\n private AtomicInteger threadNumber;\n\n private static final Logger logger = LoggerFactory.getLogger(WorkLoadManager.class);\n\n public WorkLoadManager(ConfigManager configManager, EventLoopGroup backendGroup) {\n this.configManager = configManager;\n this.backendGroup = backendGroup;\n this.port = new AtomicInteger(9000);\n this.gpuCounter = new AtomicInteger(0);\n threadPool = Executors.newCachedThreadPool();\n workers = new ConcurrentHashMap<>();\n threadNumber = new AtomicInteger(0);\n }\n\n public List getWorkers(String modelName) {\n List list = workers.get(modelName);\n if (list == null) {\n return Collections.emptyList();\n }\n return new ArrayList<>(list);\n }\n\n public Map getWorkers() {\n Map map = new HashMap<>();\n for (Map.Entry> entry : workers.entrySet()) {\n // Add server thread\n String modelName = entry.getKey();\n List workerThreads = entry.getValue();\n WorkerThread serverThread =\n ModelManager.getInstance().getModels().get(modelName).getServerThread();\n map.put(serverThread.getPid(), serverThread);\n for (WorkerThread worker : workerThreads) {\n map.put(worker.getPid(), worker);\n }\n }\n return map;\n }\n\n public boolean hasNoWorker(String modelName) {\n List worker = workers.get(modelName);\n if (worker == null) {\n return true;\n }\n return worker.isEmpty();\n }\n\n public int getNumRunningWorkers(String modelName) {\n int numWorking = 0;\n List threads = workers.getOrDefault(modelName, null);\n\n if (threads != null) {\n for (WorkerThread thread : threads) {\n if ((thread.getState() != WorkerState.WORKER_STOPPED)\n && (thread.getState() != WorkerState.WORKER_ERROR)\n && (thread.getState() != WorkerState.WORKER_SCALED_DOWN)) {\n numWorking += 1;\n }\n }\n }\n\n return numWorking;\n }\n\n public CompletableFuture modelChanged(Model model) {\n synchronized (model.getModelName()) {\n CompletableFuture future = new CompletableFuture<>();\n int minWorker = model.getMinWorkers();\n int maxWorker = model.getMaxWorkers();\n List threads;\n if (minWorker == 0) {\n threads = workers.remove(model.getModelName());\n if (threads == null) {\n if (maxWorker == 0) {\n return shutdownServerThread(model, future);\n }\n future.complete(HttpResponseStatus.OK);\n return future;\n }\n } else {\n threads = workers.computeIfAbsent(model.getModelName(), k -> new ArrayList<>());\n }\n\n int currentWorkers = threads.size();\n if (currentWorkers < minWorker) {\n addThreads(threads, model, minWorker - currentWorkers, future);\n } else {\n for (int i = currentWorkers - 1; i >= maxWorker; --i) {\n WorkerThread thread = threads.remove(i);\n thread.shutdown();\n }\n if (maxWorker == 0) {\n return shutdownServerThread(model, future);\n }\n future.complete(HttpResponseStatus.OK);\n }\n return future;\n }\n }\n\n private CompletableFuture shutdownServerThread(\n Model model, CompletableFuture future) {\n model.getServerThread().shutdown();\n WorkerLifeCycle lifecycle = model.getServerThread().getLifeCycle();\n Process workerProcess = lifecycle.getProcess();\n if (workerProcess.isAlive()) {\n boolean workerDestroyed = false;\n workerProcess.destroyForcibly();\n try {\n workerDestroyed =\n workerProcess.waitFor(\n configManager.getUnregisterModelTimeout(), TimeUnit.SECONDS);\n } catch (InterruptedException e) {\n logger.warn(\n \"WorkerThread interrupted during waitFor, possible asynch resource cleanup.\");\n future.complete(HttpResponseStatus.INTERNAL_SERVER_ERROR);\n return future;\n }\n if (!workerDestroyed) {\n logger.warn(\"WorkerThread timed out while cleaning, please resend request.\");\n future.complete(HttpResponseStatus.REQUEST_TIMEOUT);\n return future;\n }\n }\n future.complete(HttpResponseStatus.OK);\n return future;\n }\n\n public void addServerThread(Model model, CompletableFuture future)\n throws InterruptedException, ExecutionException, TimeoutException {\n WorkerStateListener listener = new WorkerStateListener(future, 1);\n BatchAggregator aggregator = new BatchAggregator(model);\n synchronized (model.getModelName()) {\n model.setPort(port.getAndIncrement());\n WorkerThread thread =\n new WorkerThread(\n configManager,\n backendGroup,\n model.getPort(),\n -1,\n model,\n aggregator,\n listener,\n threadNumber.getAndIncrement(),\n true);\n model.setServerThread(thread);\n threadPool.submit(thread);\n future.get(1, TimeUnit.MINUTES);\n }\n }\n\n private void addThreads(\n List threads,\n Model model,\n int count,\n CompletableFuture future) {\n WorkerStateListener listener = new WorkerStateListener(future, count);\n int maxGpu = configManager.getNumberOfGpu();\n for (int i = 0; i < count; ++i) {\n int gpuId = -1;\n\n if (maxGpu > 0) {\n gpuId = gpuCounter.accumulateAndGet(maxGpu, (prev, maxGpuId) -> ++prev % maxGpuId);\n }\n\n BatchAggregator aggregator = new BatchAggregator(model);\n WorkerThread thread =\n new WorkerThread(\n configManager,\n backendGroup,\n model.getPort(),\n gpuId,\n model,\n aggregator,\n listener,\n threadNumber.getAndIncrement(),\n false);\n\n threads.add(thread);\n threadPool.submit(thread);\n }\n }\n\n public void scheduleAsync(Runnable r) {\n threadPool.execute(r);\n }\n}\n" }, { "path": "frontend/server/src/main/java/com/amazonaws/ml/mms/wlm/WorkerInitializationException.java", "content": "/*\n * Copyright 2018 Amazon.com, Inc. or its affiliates. All Rights Reserved.\n *\n * Licensed under the Apache License, Version 2.0 (the \"License\"). You may not use this file except in compliance\n * with the License. A copy of the License is located at\n *\n * http://aws.amazon.com/apache2.0/\n *\n * or in the \"license\" file accompanying this file. This file is distributed on an \"AS IS\" BASIS, WITHOUT WARRANTIES\n * OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions\n * and limitations under the License.\n */\npackage com.amazonaws.ml.mms.wlm;\n\npublic class WorkerInitializationException extends Exception {\n\n static final long serialVersionUID = 1L;\n\n /** Creates a new {@code WorkerInitializationException} instance. */\n public WorkerInitializationException(String message) {\n super(message);\n }\n\n /**\n * Constructs a new {@code WorkerInitializationException} with the specified detail message and\n * cause.\n *\n * @param message the detail message (which is saved for later retrieval by the {@link\n * #getMessage()} method).\n * @param cause the cause (which is saved for later retrieval by the {@link #getCause()}\n * method). (A null value is permitted, and indicates that the cause is nonexistent\n * or unknown.)\n */\n public WorkerInitializationException(String message, Throwable cause) {\n super(message, cause);\n }\n}\n" }, { "path": "frontend/server/src/main/java/com/amazonaws/ml/mms/wlm/WorkerLifeCycle.java", "content": "/*\n * Copyright 2018 Amazon.com, Inc. or its affiliates. All Rights Reserved.\n *\n * Licensed under the Apache License, Version 2.0 (the \"License\"). You may not use this file except in compliance\n * with the License. A copy of the License is located at\n *\n * http://aws.amazon.com/apache2.0/\n *\n * or in the \"license\" file accompanying this file. This file is distributed on an \"AS IS\" BASIS, WITHOUT WARRANTIES\n * OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions\n * and limitations under the License.\n */\npackage com.amazonaws.ml.mms.wlm;\n\nimport com.amazonaws.ml.mms.archive.Manifest;\nimport com.amazonaws.ml.mms.metrics.Metric;\nimport com.amazonaws.ml.mms.util.ConfigManager;\nimport com.amazonaws.ml.mms.util.Connector;\nimport java.io.File;\nimport java.io.IOException;\nimport java.io.InputStream;\nimport java.nio.charset.StandardCharsets;\nimport java.util.ArrayList;\nimport java.util.HashMap;\nimport java.util.Map;\nimport java.util.Scanner;\nimport java.util.concurrent.CountDownLatch;\nimport java.util.concurrent.TimeUnit;\nimport java.util.concurrent.atomic.AtomicBoolean;\nimport java.util.regex.Pattern;\nimport org.slf4j.Logger;\nimport org.slf4j.LoggerFactory;\n\npublic class WorkerLifeCycle {\n\n static final Logger logger = LoggerFactory.getLogger(WorkerLifeCycle.class);\n\n private ConfigManager configManager;\n private Model model;\n private int pid = -1;\n private Process process;\n private CountDownLatch latch;\n private boolean success;\n private Connector connector;\n private ReaderThread errReader;\n private ReaderThread outReader;\n\n public WorkerLifeCycle(ConfigManager configManager, Model model) {\n this.configManager = configManager;\n this.model = model;\n this.latch = new CountDownLatch(1);\n }\n\n private String[] getEnvString(String cwd, String modelPath, String handler) {\n ArrayList envList = new ArrayList<>();\n Pattern blackList = configManager.getBlacklistPattern();\n String handlerFile = handler;\n if (handler.contains(\":\")) {\n handlerFile = handler.split(\":\")[0];\n if (handlerFile.contains(\"/\")) {\n handlerFile = handlerFile.substring(0, handlerFile.lastIndexOf('/'));\n }\n }\n\n StringBuilder pythonPath = new StringBuilder();\n HashMap environment = new HashMap<>(System.getenv());\n environment.putAll(configManager.getBackendConfiguration());\n pythonPath.append(handlerFile).append(File.pathSeparatorChar);\n if (System.getenv(\"PYTHONPATH\") != null) {\n pythonPath.append(System.getenv(\"PYTHONPATH\")).append(File.pathSeparatorChar);\n }\n\n pythonPath.append(modelPath);\n\n if (!cwd.contains(\"site-packages\") && !cwd.contains(\"dist-packages\")) {\n pythonPath.append(File.pathSeparatorChar).append(cwd);\n }\n\n environment.put(\"PYTHONPATH\", pythonPath.toString());\n\n for (Map.Entry entry : environment.entrySet()) {\n if (!blackList.matcher(entry.getKey()).matches()) {\n envList.add(entry.getKey() + '=' + entry.getValue());\n }\n }\n\n return envList.toArray(new String[0]); // NOPMD\n }\n\n public synchronized void attachIOStreams(\n String threadName, InputStream outStream, InputStream errStream) {\n logger.warn(\"attachIOStreams() threadName={}\", threadName);\n errReader = new ReaderThread(threadName, errStream, true, this);\n outReader = new ReaderThread(threadName, outStream, false, this);\n errReader.start();\n outReader.start();\n }\n\n public synchronized void terminateIOStreams() {\n if (errReader != null) {\n logger.warn(\"terminateIOStreams() threadName={}\", errReader.getName());\n errReader.terminate();\n }\n if (outReader != null) {\n logger.warn(\"terminateIOStreams() threadName={}\", outReader.getName());\n outReader.terminate();\n }\n }\n\n public void startBackendServer(int port)\n throws WorkerInitializationException, InterruptedException {\n\n File workingDir = new File(configManager.getModelServerHome());\n File modelPath;\n setPort(port);\n try {\n modelPath = model.getModelDir().getCanonicalFile();\n } catch (IOException e) {\n throw new WorkerInitializationException(\"Failed get MMS home directory\", e);\n }\n\n String[] args = new String[16];\n Manifest.RuntimeType runtime = model.getModelArchive().getManifest().getRuntime();\n if (runtime == Manifest.RuntimeType.PYTHON) {\n args[0] = configManager.getPythonExecutable();\n } else {\n args[0] = runtime.getValue();\n }\n args[1] = new File(workingDir, \"mms/model_service_worker.py\").getAbsolutePath();\n args[2] = \"--sock-type\";\n args[3] = connector.getSocketType();\n args[4] = connector.isUds() ? \"--sock-name\" : \"--port\";\n args[5] = connector.getSocketPath();\n args[6] = \"--handler\";\n args[7] = model.getModelArchive().getManifest().getModel().getHandler();\n args[8] = \"--model-path\";\n args[9] = model.getModelDir().getAbsolutePath();\n args[10] = \"--model-name\";\n args[11] = model.getModelName();\n args[12] = \"--preload-model\";\n args[13] = model.preloadModel();\n args[14] = \"--tmp-dir\";\n args[15] = System.getProperty(\"java.io.tmpdir\");\n\n String[] envp =\n getEnvString(\n workingDir.getAbsolutePath(),\n modelPath.getAbsolutePath(),\n model.getModelArchive().getManifest().getModel().getHandler());\n\n try {\n latch = new CountDownLatch(1);\n synchronized (this) {\n String threadName =\n \"W-\"\n + port\n + '-'\n + model.getModelName()\n .substring(0, Math.min(model.getModelName().length(), 25));\n process = Runtime.getRuntime().exec(args, envp, modelPath);\n attachIOStreams(threadName, process.getInputStream(), process.getErrorStream());\n }\n\n if (latch.await(2, TimeUnit.MINUTES)) {\n if (!success) {\n throw new WorkerInitializationException(\"Backend stream closed.\");\n }\n return;\n }\n throw new WorkerInitializationException(\"Backend worker startup time out.\");\n } catch (IOException e) {\n throw new WorkerInitializationException(\"Failed start worker process\", e);\n } finally {\n if (!success) {\n exit();\n }\n }\n }\n\n public synchronized void exit() {\n if (process != null) {\n process.destroyForcibly();\n connector.clean();\n terminateIOStreams();\n }\n }\n\n public synchronized Integer getExitValue() {\n if (process != null && !process.isAlive()) {\n return process.exitValue();\n }\n return null;\n }\n\n void setSuccess(boolean success) {\n this.success = success;\n latch.countDown();\n }\n\n public synchronized int getPid() {\n return pid;\n }\n\n public synchronized void setPid(int pid) {\n this.pid = pid;\n }\n\n private synchronized void setPort(int port) {\n connector = new Connector(port);\n }\n\n public Process getProcess() {\n return process;\n }\n\n private static final class ReaderThread extends Thread {\n\n private InputStream is;\n private boolean error;\n private WorkerLifeCycle lifeCycle;\n private AtomicBoolean isRunning = new AtomicBoolean(true);\n static final Logger loggerModelMetrics =\n LoggerFactory.getLogger(ConfigManager.MODEL_METRICS_LOGGER);\n\n public ReaderThread(String name, InputStream is, boolean error, WorkerLifeCycle lifeCycle) {\n super(name + (error ? \"-stderr\" : \"-stdout\"));\n this.is = is;\n this.error = error;\n this.lifeCycle = lifeCycle;\n }\n\n public void terminate() {\n isRunning.set(false);\n }\n\n @Override\n public void run() {\n try (Scanner scanner = new Scanner(is, StandardCharsets.UTF_8.name())) {\n while (isRunning.get() && scanner.hasNext()) {\n String result = scanner.nextLine();\n if (result == null) {\n break;\n }\n if (result.startsWith(\"[METRICS]\")) {\n loggerModelMetrics.info(\"{}\", Metric.parse(result.substring(9)));\n continue;\n }\n\n if (\"MMS worker started.\".equals(result)) {\n lifeCycle.setSuccess(true);\n } else if (result.startsWith(\"[PID]\")) {\n lifeCycle.setPid(Integer.parseInt(result.substring(\"[PID] \".length())));\n }\n if (error) {\n logger.warn(result);\n } else {\n logger.info(result);\n }\n }\n } catch (Exception e) {\n logger.error(\"Couldn't create scanner - {}\", getName(), e);\n } finally {\n logger.info(\"Stopped Scanner - {}\", getName());\n lifeCycle.setSuccess(false);\n try {\n is.close();\n } catch (IOException e) {\n logger.error(\"Failed to close stream for thread {}\", this.getName(), e);\n }\n }\n }\n }\n}\n" }, { "path": "frontend/server/src/main/java/com/amazonaws/ml/mms/wlm/WorkerState.java", "content": "/*\n * Copyright 2018 Amazon.com, Inc. or its affiliates. All Rights Reserved.\n *\n * Licensed under the Apache License, Version 2.0 (the \"License\"). You may not use this file except in compliance\n * with the License. A copy of the License is located at\n *\n * http://aws.amazon.com/apache2.0/\n *\n * or in the \"license\" file accompanying this file. This file is distributed on an \"AS IS\" BASIS, WITHOUT WARRANTIES\n * OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions\n * and limitations under the License.\n */\npackage com.amazonaws.ml.mms.wlm;\n\npublic enum WorkerState {\n WORKER_STARTED,\n WORKER_MODEL_LOADED,\n WORKER_STOPPED,\n WORKER_ERROR,\n WORKER_SCALED_DOWN\n}\n" }, { "path": "frontend/server/src/main/java/com/amazonaws/ml/mms/wlm/WorkerStateListener.java", "content": "/*\n * Copyright 2018 Amazon.com, Inc. or its affiliates. All Rights Reserved.\n *\n * Licensed under the Apache License, Version 2.0 (the \"License\"). You may not use this file except in compliance\n * with the License. A copy of the License is located at\n *\n * http://aws.amazon.com/apache2.0/\n *\n * or in the \"license\" file accompanying this file. This file is distributed on an \"AS IS\" BASIS, WITHOUT WARRANTIES\n * OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions\n * and limitations under the License.\n */\npackage com.amazonaws.ml.mms.wlm;\n\nimport io.netty.handler.codec.http.HttpResponseStatus;\nimport java.util.concurrent.CompletableFuture;\nimport java.util.concurrent.atomic.AtomicInteger;\n\npublic class WorkerStateListener {\n\n private CompletableFuture future;\n private AtomicInteger count;\n\n public WorkerStateListener(CompletableFuture future, int count) {\n this.future = future;\n this.count = new AtomicInteger(count);\n }\n\n public void notifyChangeState(String modelName, WorkerState state, HttpResponseStatus status) {\n // Update success and fail counts\n if (state == WorkerState.WORKER_MODEL_LOADED) {\n if (count.decrementAndGet() == 0) {\n future.complete(status);\n }\n }\n if (state == WorkerState.WORKER_ERROR || state == WorkerState.WORKER_STOPPED) {\n future.complete(status);\n }\n }\n}\n" }, { "path": "frontend/server/src/main/java/com/amazonaws/ml/mms/wlm/WorkerThread.java", "content": "/*\n * Copyright 2018 Amazon.com, Inc. or its affiliates. All Rights Reserved.\n *\n * Licensed under the Apache License, Version 2.0 (the \"License\"). You may not use this file except in compliance\n * with the License. A copy of the License is located at\n *\n * http://aws.amazon.com/apache2.0/\n *\n * or in the \"license\" file accompanying this file. This file is distributed on an \"AS IS\" BASIS, WITHOUT WARRANTIES\n * OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions\n * and limitations under the License.\n */\npackage com.amazonaws.ml.mms.wlm;\n\nimport com.amazonaws.ml.mms.metrics.Dimension;\nimport com.amazonaws.ml.mms.metrics.Metric;\nimport com.amazonaws.ml.mms.util.ConfigManager;\nimport com.amazonaws.ml.mms.util.Connector;\nimport com.amazonaws.ml.mms.util.NettyUtils;\nimport com.amazonaws.ml.mms.util.codec.ModelRequestEncoder;\nimport com.amazonaws.ml.mms.util.codec.ModelResponseDecoder;\nimport com.amazonaws.ml.mms.util.messages.BaseModelRequest;\nimport com.amazonaws.ml.mms.util.messages.InputParameter;\nimport com.amazonaws.ml.mms.util.messages.ModelWorkerResponse;\nimport com.amazonaws.ml.mms.util.messages.RequestInput;\nimport com.amazonaws.ml.mms.util.messages.WorkerCommands;\nimport io.netty.bootstrap.Bootstrap;\nimport io.netty.channel.Channel;\nimport io.netty.channel.ChannelFutureListener;\nimport io.netty.channel.ChannelHandler;\nimport io.netty.channel.ChannelHandlerContext;\nimport io.netty.channel.ChannelInitializer;\nimport io.netty.channel.ChannelPipeline;\nimport io.netty.channel.EventLoopGroup;\nimport io.netty.channel.SimpleChannelInboundHandler;\nimport io.netty.handler.codec.http.HttpResponseStatus;\nimport java.io.FileNotFoundException;\nimport java.io.IOException;\nimport java.io.RandomAccessFile;\nimport java.net.SocketAddress;\nimport java.nio.channels.Channels;\nimport java.util.UUID;\nimport java.util.concurrent.ArrayBlockingQueue;\nimport java.util.concurrent.CountDownLatch;\nimport java.util.concurrent.TimeUnit;\nimport java.util.concurrent.atomic.AtomicBoolean;\nimport java.util.concurrent.atomic.AtomicReference;\nimport org.slf4j.Logger;\nimport org.slf4j.LoggerFactory;\n\npublic class WorkerThread implements Runnable {\n\n static final Logger logger = LoggerFactory.getLogger(WorkerThread.class);\n private static final Logger loggerMmsMetrics =\n LoggerFactory.getLogger(ConfigManager.MODEL_SERVER_METRICS_LOGGER);\n\n private Metric workerLoadTime;\n\n private static final int[] BACK_OFF = {\n 0, 1, 1, 2, 3, 5, 8, 13, 21, 34, 55, 89, 144, 233, 377, 610, 987, 1597\n };\n\n static final long WORKER_TIMEOUT = ConfigManager.getInstance().isDebug() ? Long.MAX_VALUE : 2L;\n static final ModelRequestEncoder ENCODER =\n new ModelRequestEncoder(ConfigManager.getInstance().getPreferDirectBuffer());\n\n private EventLoopGroup backendEventGroup;\n private int port;\n private Model model;\n\n private Channel backendChannel;\n private AtomicBoolean running = new AtomicBoolean(true);\n\n private int backoffIdx;\n\n private BatchAggregator aggregator;\n private WorkerStateListener listener;\n ArrayBlockingQueue replies;\n private int gpuId;\n private long memory;\n private long startTime;\n private AtomicReference currentThread = new AtomicReference<>();\n private String workerId;\n private String threadName;\n private BaseModelRequest req;\n private WorkerState state;\n\n private WorkerLifeCycle lifeCycle;\n private boolean serverThread;\n private RandomAccessFile out;\n private RandomAccessFile err;\n private Connector connector;\n\n public WorkerState getState() {\n return state;\n }\n\n public WorkerLifeCycle getLifeCycle() {\n return lifeCycle;\n }\n\n public WorkerThread(\n ConfigManager configManager,\n EventLoopGroup backendEventGroup,\n int port,\n int gpuId,\n Model model,\n BatchAggregator aggregator,\n WorkerStateListener listener,\n int threadNumber,\n boolean serverThread) {\n this.workerId = String.valueOf(port); // Unique across all workers.\n this.backendEventGroup = backendEventGroup;\n this.port = port;\n this.model = model;\n this.aggregator = aggregator;\n this.gpuId = gpuId;\n this.listener = listener;\n startTime = System.currentTimeMillis();\n lifeCycle = new WorkerLifeCycle(configManager, model);\n replies = new ArrayBlockingQueue<>(1);\n this.serverThread = serverThread;\n this.threadName =\n !serverThread\n ? \"W-\"\n + model.getModelName()\n .substring(0, Math.min(model.getModelName().length(), 25))\n + '-'\n + threadNumber\n : \"BackendServer-\" + model.getModelName();\n workerLoadTime =\n new Metric(\n getWorkerName(),\n String.valueOf(System.currentTimeMillis()),\n \"ms\",\n ConfigManager.getInstance().getHostName(),\n new Dimension(\"Level\", \"Host\"));\n }\n\n private void runWorker()\n throws WorkerInitializationException, InterruptedException, FileNotFoundException {\n int responseTimeoutSeconds = model.getResponseTimeoutSeconds();\n while (isRunning()) {\n req = aggregator.getRequest(backendChannel.id().asLongText(), state);\n backendChannel.writeAndFlush(req).sync();\n long begin = System.currentTimeMillis();\n // TODO: Change this to configurable param\n ModelWorkerResponse reply = replies.poll(responseTimeoutSeconds, TimeUnit.SECONDS);\n long duration = System.currentTimeMillis() - begin;\n logger.info(\"Backend response time: {}\", duration);\n\n if (reply != null) {\n aggregator.sendResponse(reply);\n } else {\n int val = model.incrFailedInfReqs();\n logger.error(\"Number or consecutive unsuccessful inference {}\", val);\n throw new WorkerInitializationException(\n \"Backend worker did not respond in given time\");\n }\n switch (req.getCommand()) {\n case PREDICT:\n model.resetFailedInfReqs();\n break;\n case LOAD:\n String message = reply.getMessage();\n String tmpdir = System.getProperty(\"java.io.tmpdir\");\n out =\n new RandomAccessFile(\n tmpdir + '/' + backendChannel.id().asLongText() + \"-stdout\",\n \"rw\");\n err =\n new RandomAccessFile(\n tmpdir + '/' + backendChannel.id().asLongText() + \"-stderr\",\n \"rw\");\n if (reply.getCode() == 200) {\n setState(WorkerState.WORKER_MODEL_LOADED, HttpResponseStatus.OK);\n lifeCycle.setPid(\n Integer.parseInt(\n message.substring(\n message.indexOf(\"[PID]:\") + 6, message.length())));\n lifeCycle.attachIOStreams(\n threadName,\n Channels.newInputStream(out.getChannel()),\n Channels.newInputStream(err.getChannel()));\n backoffIdx = 0;\n } else {\n setState(\n WorkerState.WORKER_ERROR,\n HttpResponseStatus.valueOf(reply.getCode()));\n }\n break;\n case UNLOAD:\n case STATS:\n default:\n break;\n }\n req = null;\n }\n }\n\n @Override\n public void run() {\n Process process = null;\n Thread thread = Thread.currentThread();\n thread.setName(getWorkerName());\n currentThread.set(thread);\n HttpResponseStatus status = HttpResponseStatus.INTERNAL_SERVER_ERROR;\n try {\n if (!serverThread) {\n connect();\n runWorker();\n } else {\n // TODO: Move this logic to a seperate ServerThread class\n // This is server thread and shouldn't come out as long as process exists in CPU.\n model.setPort(port);\n lifeCycle.startBackendServer(port);\n setState(WorkerState.WORKER_MODEL_LOADED, HttpResponseStatus.OK);\n process = lifeCycle.getProcess();\n process.waitFor();\n }\n } catch (InterruptedException e) {\n if (state == WorkerState.WORKER_SCALED_DOWN) {\n logger.debug(\"Shutting down the thread .. Scaling down.\");\n } else {\n logger.debug(\n \"Backend worker monitoring thread interrupted or backend worker process died.\",\n e);\n }\n } catch (WorkerInitializationException e) {\n logger.error(\"Backend worker error\", e);\n } catch (OutOfMemoryError oom) {\n logger.error(\"Out of memory error when creating workers\", oom);\n status = HttpResponseStatus.INSUFFICIENT_STORAGE;\n } catch (Throwable t) {\n logger.warn(\"Backend worker thread exception.\", t);\n } finally {\n // WorkerThread is running in thread pool, the thread will be assigned to next\n // Runnable once this worker is finished. If currentThread keep holding the reference\n // of the thread, currentThread.interrupt() might kill next worker.\n backendChannel.disconnect();\n currentThread.set(null);\n Integer exitValue = lifeCycle.getExitValue();\n\n if (exitValue != null && exitValue == 137) {\n status = HttpResponseStatus.INSUFFICIENT_STORAGE;\n }\n\n if (!serverThread && req != null) {\n aggregator.sendError(req, \"Worker died.\", status);\n } else if (serverThread) {\n model.setPort(-1);\n if (process != null && process.isAlive()) {\n process.destroyForcibly();\n try {\n process.waitFor(1, TimeUnit.SECONDS);\n } catch (InterruptedException e) {\n logger.warn(\n \"WorkerThread interrupted during waitFor, possible asynch resource cleanup.\");\n }\n }\n }\n setState(WorkerState.WORKER_STOPPED, status);\n lifeCycle.exit();\n retry();\n }\n }\n\n public String getWorkerId() {\n return workerId;\n }\n\n public long getMemory() {\n return memory;\n }\n\n public void setMemory(long memory) {\n this.memory = memory;\n }\n\n private void connect()\n throws WorkerInitializationException, InterruptedException, FileNotFoundException {\n if (!this.serverThread && (model.getPort() == -1)) {\n throw new WorkerInitializationException(\"Backend server is not runniing\");\n }\n\n String modelName = model.getModelName();\n setState(WorkerState.WORKER_STARTED, HttpResponseStatus.OK);\n final CountDownLatch latch = new CountDownLatch(1);\n\n final int responseBufferSize = ConfigManager.getInstance().getMaxResponseSize();\n try {\n connector = new Connector(port);\n Bootstrap b = new Bootstrap();\n b.group(backendEventGroup)\n .channel(connector.getClientChannel())\n .handler(\n new ChannelInitializer() {\n @Override\n public void initChannel(Channel ch) {\n ChannelPipeline p = ch.pipeline();\n p.addLast(ENCODER);\n p.addLast(new ModelResponseDecoder(responseBufferSize));\n p.addLast(new WorkerHandler());\n }\n });\n\n SocketAddress address = connector.getSocketAddress();\n logger.info(\"Connecting to: {}\", address);\n backendChannel = b.connect(address).sync().channel();\n backendChannel\n .closeFuture()\n .addListener(\n (ChannelFutureListener)\n future -> {\n latch.countDown();\n logger.info(\n \"{} Worker disconnected. {}\", getWorkerId(), state);\n Thread thread = currentThread.getAndSet(null);\n if (thread != null) {\n thread.interrupt();\n }\n });\n\n backendChannel\n .newSucceededFuture()\n .addListener(\n (ChannelFutureListener)\n future -> {\n // TODO:\n // use gpu, batch size in load model command\n RequestInput input =\n new RequestInput(UUID.randomUUID().toString());\n if (gpuId >= 0) {\n input.addParameter(\n new InputParameter(\n \"gpu\", String.valueOf(gpuId)));\n }\n\n Job job =\n new Job(\n null,\n modelName,\n WorkerCommands.LOAD,\n input);\n model.addJob(backendChannel.id().asLongText(), job);\n latch.countDown();\n });\n\n if (!latch.await(WORKER_TIMEOUT, TimeUnit.MINUTES)) {\n throw new WorkerInitializationException(\n \"Worker failed to initialize within \" + WORKER_TIMEOUT + \" mins\");\n }\n workerId = workerId + \"-\" + backendChannel.id();\n running.set(true);\n } catch (Throwable t) {\n // https://github.com/netty/netty/issues/2597\n if (t instanceof IOException) {\n throw new WorkerInitializationException(\"Failed to connect to worker.\", t);\n }\n throw t;\n }\n }\n\n public boolean isRunning() {\n return running.get();\n }\n\n public int getGpuId() {\n return gpuId;\n }\n\n public long getStartTime() {\n return startTime;\n }\n\n public int getPid() {\n return lifeCycle.getPid();\n }\n\n public void shutdown() {\n running.set(false);\n setState(WorkerState.WORKER_SCALED_DOWN, HttpResponseStatus.OK);\n if (backendChannel != null) {\n model.removeJobQueue(backendChannel.id().asLongText());\n backendChannel.close();\n }\n if (this.serverThread && this.connector != null) {\n logger.debug(\"Cleaning connector socket\");\n this.connector.clean();\n }\n logger.debug(\"Terminating IOStreams for worker thread shutdown\");\n lifeCycle.terminateIOStreams();\n try {\n if (out != null) {\n out.close();\n }\n if (err != null) {\n err.close();\n }\n } catch (IOException e) {\n logger.error(\"Failed to close IO file handles\", e);\n }\n Thread thread = currentThread.getAndSet(null);\n if (thread != null) {\n thread.interrupt();\n aggregator.sendError(\n null, \"Worker scaled down.\", HttpResponseStatus.INTERNAL_SERVER_ERROR);\n }\n }\n\n public boolean isServerThread() {\n return serverThread;\n }\n\n private final String getWorkerName() {\n String modelName = model.getModelName();\n if (modelName.length() > 25) {\n modelName = modelName.substring(0, 25);\n }\n return \"W-\" + port + '-' + modelName;\n }\n\n void setState(WorkerState newState, HttpResponseStatus status) {\n listener.notifyChangeState(model.getModelName(), newState, status);\n logger.debug(\"{} State change {} -> {}\", getWorkerName(), state, newState);\n long timeTaken = System.currentTimeMillis() - startTime;\n if (state != WorkerState.WORKER_SCALED_DOWN) {\n // Don't update the state if it was terminated on purpose.. Scaling in..\n this.state = newState;\n }\n if (state == WorkerState.WORKER_MODEL_LOADED) {\n workerLoadTime.setValue(String.valueOf(timeTaken));\n workerLoadTime.setTimestamp(\n String.valueOf(TimeUnit.MILLISECONDS.toSeconds(System.currentTimeMillis())));\n loggerMmsMetrics.info(\"{}\", workerLoadTime);\n }\n }\n\n void retry() {\n if (state == WorkerState.WORKER_SCALED_DOWN) {\n logger.debug(\"Worker terminated due to scale-down call.\");\n return;\n }\n\n ModelManager manager = ModelManager.getInstance();\n\n if (backoffIdx < BACK_OFF.length - 1) {\n ++backoffIdx;\n }\n\n manager.getScheduler()\n .schedule(() -> manager.submitTask(this), BACK_OFF[backoffIdx], TimeUnit.SECONDS);\n logger.info(\"Retry worker: {} in {} seconds.\", workerId, BACK_OFF[backoffIdx]);\n }\n\n @ChannelHandler.Sharable\n private class WorkerHandler extends SimpleChannelInboundHandler {\n\n @Override\n public void channelRead0(ChannelHandlerContext ctx, ModelWorkerResponse msg) {\n if (!replies.offer(msg)) {\n throw new IllegalStateException(\"Reply queue is full.\");\n }\n }\n\n @Override\n public void exceptionCaught(ChannelHandlerContext ctx, Throwable cause) {\n logger.error(\"Unknown exception\", cause);\n if (cause instanceof OutOfMemoryError) {\n NettyUtils.sendError(ctx, HttpResponseStatus.INSUFFICIENT_STORAGE, cause);\n }\n ctx.close();\n }\n }\n}\n" }, { "path": "frontend/server/src/main/resources/log4j2.xml", "content": "\n\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n\n" }, { "path": "frontend/server/src/test/java/com/amazonaws/ml/mms/CoverageTest.java", "content": "/*\n * Copyright 2018 Amazon.com, Inc. or its affiliates. All Rights Reserved.\n *\n * Licensed under the Apache License, Version 2.0 (the \"License\"). You may not use this file except in compliance\n * with the License. A copy of the License is located at\n *\n * http://aws.amazon.com/apache2.0/\n *\n * or in the \"license\" file accompanying this file. This file is distributed on an \"AS IS\" BASIS, WITHOUT WARRANTIES\n * OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions\n * and limitations under the License.\n */\npackage com.amazonaws.ml.mms;\n\nimport com.amazonaws.ml.mms.test.TestHelper;\nimport java.io.IOException;\nimport org.testng.annotations.Test;\n\npublic class CoverageTest {\n\n @Test\n public void test() throws IOException, ClassNotFoundException {\n TestHelper.testGetterSetters(ModelServer.class);\n }\n}\n" }, { "path": "frontend/server/src/test/java/com/amazonaws/ml/mms/ModelServerTest.java", "content": "/*\n * Copyright 2018 Amazon.com, Inc. or its affiliates. All Rights Reserved.\n *\n * Licensed under the Apache License, Version 2.0 (the \"License\"). You may not use this file except in compliance\n * with the License. A copy of the License is located at\n *\n * http://aws.amazon.com/apache2.0/\n *\n * or in the \"license\" file accompanying this file. This file is distributed on an \"AS IS\" BASIS, WITHOUT WARRANTIES\n * OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions\n * and limitations under the License.\n */\npackage com.amazonaws.ml.mms;\n\nimport com.amazonaws.ml.mms.http.DescribeModelResponse;\nimport com.amazonaws.ml.mms.http.ErrorResponse;\nimport com.amazonaws.ml.mms.http.ListModelsResponse;\nimport com.amazonaws.ml.mms.http.StatusResponse;\nimport com.amazonaws.ml.mms.metrics.Dimension;\nimport com.amazonaws.ml.mms.metrics.Metric;\nimport com.amazonaws.ml.mms.metrics.MetricManager;\nimport com.amazonaws.ml.mms.servingsdk.impl.PluginsManager;\nimport com.amazonaws.ml.mms.util.ConfigManager;\nimport com.amazonaws.ml.mms.util.Connector;\nimport com.amazonaws.ml.mms.util.JsonUtils;\nimport com.google.gson.JsonParseException;\nimport io.netty.bootstrap.Bootstrap;\nimport io.netty.buffer.Unpooled;\nimport io.netty.channel.Channel;\nimport io.netty.channel.ChannelHandler;\nimport io.netty.channel.ChannelHandlerContext;\nimport io.netty.channel.ChannelInitializer;\nimport io.netty.channel.ChannelOption;\nimport io.netty.channel.ChannelPipeline;\nimport io.netty.channel.SimpleChannelInboundHandler;\nimport io.netty.handler.codec.http.DefaultFullHttpRequest;\nimport io.netty.handler.codec.http.FullHttpResponse;\nimport io.netty.handler.codec.http.HttpClientCodec;\nimport io.netty.handler.codec.http.HttpContentDecompressor;\nimport io.netty.handler.codec.http.HttpHeaderNames;\nimport io.netty.handler.codec.http.HttpHeaderValues;\nimport io.netty.handler.codec.http.HttpHeaders;\nimport io.netty.handler.codec.http.HttpMethod;\nimport io.netty.handler.codec.http.HttpObjectAggregator;\nimport io.netty.handler.codec.http.HttpRequest;\nimport io.netty.handler.codec.http.HttpResponseStatus;\nimport io.netty.handler.codec.http.HttpUtil;\nimport io.netty.handler.codec.http.HttpVersion;\nimport io.netty.handler.codec.http.multipart.HttpPostRequestEncoder;\nimport io.netty.handler.codec.http.multipart.MemoryFileUpload;\nimport io.netty.handler.ssl.SslContext;\nimport io.netty.handler.ssl.SslContextBuilder;\nimport io.netty.handler.ssl.util.InsecureTrustManagerFactory;\nimport io.netty.handler.stream.ChunkedWriteHandler;\nimport io.netty.handler.timeout.ReadTimeoutHandler;\nimport io.netty.util.CharsetUtil;\nimport io.netty.util.internal.logging.InternalLoggerFactory;\nimport io.netty.util.internal.logging.Slf4JLoggerFactory;\nimport java.io.File;\nimport java.io.FileInputStream;\nimport java.io.IOException;\nimport java.io.InputStream;\nimport java.lang.reflect.Field;\nimport java.nio.charset.StandardCharsets;\nimport java.security.GeneralSecurityException;\nimport java.util.List;\nimport java.util.Properties;\nimport java.util.Scanner;\nimport java.util.concurrent.CountDownLatch;\nimport org.apache.commons.io.IOUtils;\nimport org.slf4j.Logger;\nimport org.slf4j.LoggerFactory;\nimport org.testng.Assert;\nimport org.testng.annotations.AfterSuite;\nimport org.testng.annotations.BeforeSuite;\nimport org.testng.annotations.Test;\n\npublic class ModelServerTest {\n\n private static final String ERROR_NOT_FOUND =\n \"Requested resource is not found, please refer to API document.\";\n private static final String ERROR_METHOD_NOT_ALLOWED =\n \"Requested method is not allowed, please refer to API document.\";\n\n private ConfigManager configManager;\n private ModelServer server;\n CountDownLatch latch;\n HttpResponseStatus httpStatus;\n String result;\n HttpHeaders headers;\n private String listInferenceApisResult;\n private String listManagementApisResult;\n private String noopApiResult;\n\n static {\n TestUtils.init();\n }\n\n @BeforeSuite\n public void beforeSuite() throws InterruptedException, IOException, GeneralSecurityException {\n ConfigManager.init(new ConfigManager.Arguments());\n configManager = ConfigManager.getInstance();\n PluginsManager.getInstance().initialize();\n\n InternalLoggerFactory.setDefaultFactory(Slf4JLoggerFactory.INSTANCE);\n\n server = new ModelServer(configManager);\n server.start();\n\n try (InputStream is = new FileInputStream(\"src/test/resources/inference_open_api.json\")) {\n listInferenceApisResult = IOUtils.toString(is, StandardCharsets.UTF_8.name());\n }\n\n try (InputStream is = new FileInputStream(\"src/test/resources/management_open_api.json\")) {\n listManagementApisResult = IOUtils.toString(is, StandardCharsets.UTF_8.name());\n }\n\n try (InputStream is = new FileInputStream(\"src/test/resources/describe_api.json\")) {\n noopApiResult = IOUtils.toString(is, StandardCharsets.UTF_8.name());\n }\n }\n\n @AfterSuite\n public void afterSuite() {\n server.stop();\n }\n\n @Test\n public void test()\n throws InterruptedException, HttpPostRequestEncoder.ErrorDataEncoderException,\n IOException, NoSuchFieldException, IllegalAccessException {\n Channel channel = null;\n Channel managementChannel = null;\n for (int i = 0; i < 5; ++i) {\n channel = connect(false);\n if (channel != null) {\n break;\n }\n Thread.sleep(100);\n }\n\n for (int i = 0; i < 5; ++i) {\n managementChannel = connect(true);\n if (managementChannel != null) {\n break;\n }\n Thread.sleep(100);\n }\n\n Assert.assertNotNull(channel, \"Failed to connect to inference port.\");\n Assert.assertNotNull(managementChannel, \"Failed to connect to management port.\");\n testPing(channel);\n\n testRoot(channel, listInferenceApisResult);\n testRoot(managementChannel, listManagementApisResult);\n testApiDescription(channel, listInferenceApisResult);\n\n testDescribeApi(channel);\n testUnregisterModel(managementChannel);\n testLoadModel(managementChannel);\n testSyncScaleModel(managementChannel);\n testScaleModel(managementChannel);\n testListModels(managementChannel);\n testDescribeModel(managementChannel);\n testLoadModelWithInitialWorkers(managementChannel);\n testLoadModelWithInitialWorkersWithJSONReqBody(managementChannel);\n testPredictions(channel);\n testPredictionsBinary(channel);\n testPredictionsJson(channel);\n testInvocationsJson(channel);\n testInvocationsMultipart(channel);\n testModelsInvokeJson(channel);\n testModelsInvokeMultipart(channel);\n testLegacyPredict(channel);\n testPredictionsInvalidRequestSize(channel);\n testPredictionsValidRequestSize(channel);\n testPredictionsDecodeRequest(channel, managementChannel);\n testPredictionsDoNotDecodeRequest(channel, managementChannel);\n testPredictionsModifyResponseHeader(channel, managementChannel);\n testPredictionsNoManifest(channel, managementChannel);\n testModelRegisterWithDefaultWorkers(managementChannel);\n testLogging(channel, managementChannel);\n testLoggingUnload(channel, managementChannel);\n testLoadingMemoryError();\n testPredictionMemoryError();\n testPredictionCustomErrorCode();\n testMetricManager();\n testErrorBatch();\n\n channel.close().sync();\n managementChannel.close().sync();\n\n // negative test case, channel will be closed by server\n testInvalidRootRequest();\n testInvalidInferenceUri();\n testInvalidPredictionsUri();\n testInvalidDescribeModel();\n testPredictionsModelNotFound();\n\n testInvalidManagementUri();\n testInvalidModelsMethod();\n testInvalidModelMethod();\n testDescribeModelNotFound();\n testRegisterModelMissingUrl();\n testRegisterModelInvalidRuntime();\n testRegisterModelNotFound();\n testRegisterModelConflict();\n testRegisterModelMalformedUrl();\n testRegisterModelConnectionFailed();\n testRegisterModelHttpError();\n testRegisterModelInvalidPath();\n testScaleModelNotFound();\n testScaleModelFailure();\n testUnregisterModelNotFound();\n testUnregisterModelTimeout();\n testInvalidModel();\n }\n\n private void testRoot(Channel channel, String expected) throws InterruptedException {\n result = null;\n latch = new CountDownLatch(1);\n HttpRequest req = new DefaultFullHttpRequest(HttpVersion.HTTP_1_1, HttpMethod.OPTIONS, \"/\");\n channel.writeAndFlush(req).sync();\n latch.await();\n\n Assert.assertEquals(result, expected);\n }\n\n private void testPing(Channel channel) throws InterruptedException {\n result = null;\n latch = new CountDownLatch(1);\n HttpRequest req = new DefaultFullHttpRequest(HttpVersion.HTTP_1_1, HttpMethod.GET, \"/ping\");\n channel.writeAndFlush(req);\n latch.await();\n\n StatusResponse resp = JsonUtils.GSON.fromJson(result, StatusResponse.class);\n Assert.assertEquals(resp.getStatus(), \"Healthy\");\n Assert.assertTrue(headers.contains(\"x-request-id\"));\n }\n\n private void testApiDescription(Channel channel, String expected) throws InterruptedException {\n result = null;\n latch = new CountDownLatch(1);\n HttpRequest req =\n new DefaultFullHttpRequest(\n HttpVersion.HTTP_1_1, HttpMethod.GET, \"/api-description\");\n channel.writeAndFlush(req);\n latch.await();\n\n Assert.assertEquals(result, expected);\n }\n\n private void testDescribeApi(Channel channel) throws InterruptedException {\n result = null;\n latch = new CountDownLatch(1);\n HttpRequest req =\n new DefaultFullHttpRequest(\n HttpVersion.HTTP_1_1, HttpMethod.OPTIONS, \"/predictions/noop_v0.1\");\n channel.writeAndFlush(req);\n latch.await();\n\n Assert.assertEquals(result, noopApiResult);\n }\n\n private void testLoadModel(Channel channel) throws InterruptedException {\n result = null;\n latch = new CountDownLatch(1);\n HttpRequest req =\n new DefaultFullHttpRequest(\n HttpVersion.HTTP_1_1,\n HttpMethod.POST,\n \"/models?url=noop-v0.1&model_name=noop_v0.1&runtime=python&synchronous=false\");\n channel.writeAndFlush(req);\n latch.await();\n\n StatusResponse resp = JsonUtils.GSON.fromJson(result, StatusResponse.class);\n Assert.assertEquals(resp.getStatus(), \"Model \\\"noop_v0.1\\\" registered\");\n }\n\n private void testLoadModelWithInitialWorkers(Channel channel) throws InterruptedException {\n testUnregisterModel(channel);\n\n result = null;\n latch = new CountDownLatch(1);\n HttpRequest req =\n new DefaultFullHttpRequest(\n HttpVersion.HTTP_1_1,\n HttpMethod.POST,\n \"/models?url=noop-v0.1&model_name=noop_v0.1&initial_workers=1&synchronous=true\");\n channel.writeAndFlush(req);\n latch.await();\n\n StatusResponse resp = JsonUtils.GSON.fromJson(result, StatusResponse.class);\n Assert.assertEquals(resp.getStatus(), \"Workers scaled\");\n }\n\n private void testLoadModelWithInitialWorkersWithJSONReqBody(Channel channel)\n throws InterruptedException {\n testUnregisterModel(channel);\n\n result = null;\n latch = new CountDownLatch(1);\n DefaultFullHttpRequest req =\n new DefaultFullHttpRequest(HttpVersion.HTTP_1_1, HttpMethod.POST, \"/models\");\n req.headers().add(\"Content-Type\", \"application/json\");\n req.content()\n .writeCharSequence(\n \"{'url':'noop-v0.1', 'model_name':'noop_v0.1', 'initial_workers':'1', 'synchronous':'true'}\",\n CharsetUtil.UTF_8);\n HttpUtil.setContentLength(req, req.content().readableBytes());\n channel.writeAndFlush(req);\n latch.await();\n\n StatusResponse resp = JsonUtils.GSON.fromJson(result, StatusResponse.class);\n Assert.assertEquals(resp.getStatus(), \"Workers scaled\");\n }\n\n private void testScaleModel(Channel channel) throws InterruptedException {\n result = null;\n latch = new CountDownLatch(1);\n HttpRequest req =\n new DefaultFullHttpRequest(\n HttpVersion.HTTP_1_1, HttpMethod.PUT, \"/models/noop_v0.1?min_worker=2\");\n channel.writeAndFlush(req);\n latch.await();\n\n StatusResponse resp = JsonUtils.GSON.fromJson(result, StatusResponse.class);\n Assert.assertEquals(resp.getStatus(), \"Processing worker updates...\");\n }\n\n private void testSyncScaleModel(Channel channel) throws InterruptedException {\n result = null;\n latch = new CountDownLatch(1);\n HttpRequest req =\n new DefaultFullHttpRequest(\n HttpVersion.HTTP_1_1,\n HttpMethod.PUT,\n \"/models/noop_v0.1?synchronous=true&min_worker=1\");\n channel.writeAndFlush(req);\n latch.await();\n\n StatusResponse resp = JsonUtils.GSON.fromJson(result, StatusResponse.class);\n Assert.assertEquals(resp.getStatus(), \"Workers scaled\");\n }\n\n private void testUnregisterModel(Channel channel) throws InterruptedException {\n result = null;\n latch = new CountDownLatch(1);\n HttpRequest req =\n new DefaultFullHttpRequest(\n HttpVersion.HTTP_1_1, HttpMethod.DELETE, \"/models/noop_v0.1\");\n channel.writeAndFlush(req);\n latch.await();\n\n StatusResponse resp = JsonUtils.GSON.fromJson(result, StatusResponse.class);\n Assert.assertEquals(resp.getStatus(), \"Model \\\"noop_v0.1\\\" unregistered\");\n }\n\n private void testListModels(Channel channel) throws InterruptedException {\n result = null;\n latch = new CountDownLatch(1);\n HttpRequest req =\n new DefaultFullHttpRequest(\n HttpVersion.HTTP_1_1, HttpMethod.GET, \"/models?limit=200&nextPageToken=X\");\n channel.writeAndFlush(req);\n latch.await();\n\n ListModelsResponse resp = JsonUtils.GSON.fromJson(result, ListModelsResponse.class);\n Assert.assertEquals(resp.getModels().size(), 2);\n }\n\n private void testDescribeModel(Channel channel) throws InterruptedException {\n result = null;\n latch = new CountDownLatch(1);\n HttpRequest req =\n new DefaultFullHttpRequest(\n HttpVersion.HTTP_1_1, HttpMethod.GET, \"/models/noop_v0.1\");\n channel.writeAndFlush(req);\n latch.await();\n\n DescribeModelResponse resp = JsonUtils.GSON.fromJson(result, DescribeModelResponse.class);\n Assert.assertTrue(resp.getWorkers().size() > 1);\n }\n\n private void testPredictions(Channel channel) throws InterruptedException {\n result = null;\n latch = new CountDownLatch(1);\n DefaultFullHttpRequest req =\n new DefaultFullHttpRequest(\n HttpVersion.HTTP_1_1, HttpMethod.POST, \"/predictions/noop\");\n req.content().writeCharSequence(\"data=test\", CharsetUtil.UTF_8);\n HttpUtil.setContentLength(req, req.content().readableBytes());\n req.headers()\n .set(\n HttpHeaderNames.CONTENT_TYPE,\n HttpHeaderValues.APPLICATION_X_WWW_FORM_URLENCODED);\n channel.writeAndFlush(req);\n\n latch.await();\n Assert.assertEquals(result, \"OK\");\n }\n\n private void testPredictionsJson(Channel channel) throws InterruptedException {\n result = null;\n latch = new CountDownLatch(1);\n DefaultFullHttpRequest req =\n new DefaultFullHttpRequest(\n HttpVersion.HTTP_1_1, HttpMethod.POST, \"/predictions/noop\");\n req.content().writeCharSequence(\"{\\\"data\\\": \\\"test\\\"}\", CharsetUtil.UTF_8);\n HttpUtil.setContentLength(req, req.content().readableBytes());\n req.headers().set(HttpHeaderNames.CONTENT_TYPE, HttpHeaderValues.APPLICATION_JSON);\n channel.writeAndFlush(req);\n\n latch.await();\n Assert.assertEquals(result, \"OK\");\n }\n\n private void testPredictionsBinary(Channel channel) throws InterruptedException {\n result = null;\n latch = new CountDownLatch(1);\n DefaultFullHttpRequest req =\n new DefaultFullHttpRequest(\n HttpVersion.HTTP_1_1, HttpMethod.POST, \"/predictions/noop\");\n req.content().writeCharSequence(\"test\", CharsetUtil.UTF_8);\n HttpUtil.setContentLength(req, req.content().readableBytes());\n req.headers().set(HttpHeaderNames.CONTENT_TYPE, HttpHeaderValues.APPLICATION_OCTET_STREAM);\n channel.writeAndFlush(req);\n\n latch.await();\n\n Assert.assertEquals(result, \"OK\");\n }\n\n private void testInvocationsJson(Channel channel) throws InterruptedException {\n result = null;\n latch = new CountDownLatch(1);\n DefaultFullHttpRequest req =\n new DefaultFullHttpRequest(\n HttpVersion.HTTP_1_1, HttpMethod.POST, \"/invocations?model_name=noop\");\n req.content().writeCharSequence(\"{\\\"data\\\": \\\"test\\\"}\", CharsetUtil.UTF_8);\n HttpUtil.setContentLength(req, req.content().readableBytes());\n req.headers().set(HttpHeaderNames.CONTENT_TYPE, HttpHeaderValues.APPLICATION_JSON);\n channel.writeAndFlush(req);\n latch.await();\n\n Assert.assertEquals(result, \"OK\");\n }\n\n private void testInvocationsMultipart(Channel channel)\n throws InterruptedException, HttpPostRequestEncoder.ErrorDataEncoderException,\n IOException {\n result = null;\n latch = new CountDownLatch(1);\n DefaultFullHttpRequest req =\n new DefaultFullHttpRequest(HttpVersion.HTTP_1_1, HttpMethod.POST, \"/invocations\");\n\n HttpPostRequestEncoder encoder = new HttpPostRequestEncoder(req, true);\n encoder.addBodyAttribute(\"model_name\", \"noop_v0.1\");\n MemoryFileUpload body =\n new MemoryFileUpload(\"data\", \"test.txt\", \"text/plain\", null, null, 4);\n body.setContent(Unpooled.copiedBuffer(\"test\", StandardCharsets.UTF_8));\n encoder.addBodyHttpData(body);\n\n channel.writeAndFlush(encoder.finalizeRequest());\n if (encoder.isChunked()) {\n channel.writeAndFlush(encoder).sync();\n }\n\n latch.await();\n\n Assert.assertEquals(result, \"OK\");\n }\n\n private void testModelsInvokeJson(Channel channel) throws InterruptedException {\n result = null;\n latch = new CountDownLatch(1);\n DefaultFullHttpRequest req =\n new DefaultFullHttpRequest(\n HttpVersion.HTTP_1_1, HttpMethod.POST, \"/models/noop/invoke\");\n req.content().writeCharSequence(\"{\\\"data\\\": \\\"test\\\"}\", CharsetUtil.UTF_8);\n HttpUtil.setContentLength(req, req.content().readableBytes());\n req.headers().set(HttpHeaderNames.CONTENT_TYPE, HttpHeaderValues.APPLICATION_JSON);\n channel.writeAndFlush(req);\n latch.await();\n\n Assert.assertEquals(result, \"OK\");\n }\n\n private void testModelsInvokeMultipart(Channel channel)\n throws InterruptedException, HttpPostRequestEncoder.ErrorDataEncoderException,\n IOException {\n result = null;\n latch = new CountDownLatch(1);\n DefaultFullHttpRequest req =\n new DefaultFullHttpRequest(\n HttpVersion.HTTP_1_1, HttpMethod.POST, \"/models/noop/invoke\");\n\n HttpPostRequestEncoder encoder = new HttpPostRequestEncoder(req, true);\n MemoryFileUpload body =\n new MemoryFileUpload(\"data\", \"test.txt\", \"text/plain\", null, null, 4);\n body.setContent(Unpooled.copiedBuffer(\"test\", StandardCharsets.UTF_8));\n encoder.addBodyHttpData(body);\n\n channel.writeAndFlush(encoder.finalizeRequest());\n if (encoder.isChunked()) {\n channel.writeAndFlush(encoder).sync();\n }\n\n latch.await();\n\n Assert.assertEquals(result, \"OK\");\n }\n\n private void testPredictionsInvalidRequestSize(Channel channel) throws InterruptedException {\n result = null;\n latch = new CountDownLatch(1);\n DefaultFullHttpRequest req =\n new DefaultFullHttpRequest(\n HttpVersion.HTTP_1_1, HttpMethod.POST, \"/predictions/noop\");\n\n req.content().writeZero(11485760);\n HttpUtil.setContentLength(req, req.content().readableBytes());\n req.headers().set(HttpHeaderNames.CONTENT_TYPE, HttpHeaderValues.APPLICATION_OCTET_STREAM);\n channel.writeAndFlush(req);\n\n latch.await();\n\n Assert.assertEquals(httpStatus, HttpResponseStatus.REQUEST_ENTITY_TOO_LARGE);\n }\n\n private void testPredictionsValidRequestSize(Channel channel) throws InterruptedException {\n result = null;\n latch = new CountDownLatch(1);\n DefaultFullHttpRequest req =\n new DefaultFullHttpRequest(\n HttpVersion.HTTP_1_1, HttpMethod.POST, \"/predictions/noop\");\n\n req.content().writeZero(10385760);\n HttpUtil.setContentLength(req, req.content().readableBytes());\n req.headers().set(HttpHeaderNames.CONTENT_TYPE, HttpHeaderValues.APPLICATION_OCTET_STREAM);\n channel.writeAndFlush(req);\n\n latch.await();\n\n Assert.assertEquals(httpStatus, HttpResponseStatus.OK);\n }\n\n private void loadTests(Channel channel, String model, String modelName)\n throws InterruptedException {\n result = null;\n latch = new CountDownLatch(1);\n String url =\n \"/models?url=\"\n + model\n + \"&model_name=\"\n + modelName\n + \"&initial_workers=1&synchronous=true\";\n HttpRequest req = new DefaultFullHttpRequest(HttpVersion.HTTP_1_1, HttpMethod.POST, url);\n channel.writeAndFlush(req);\n latch.await();\n }\n\n private void unloadTests(Channel channel, String modelName) throws InterruptedException {\n result = null;\n latch = new CountDownLatch(1);\n String expected = \"Model \\\"\" + modelName + \"\\\" unregistered\";\n String url = \"/models/\" + modelName;\n HttpRequest req = new DefaultFullHttpRequest(HttpVersion.HTTP_1_1, HttpMethod.DELETE, url);\n channel.writeAndFlush(req);\n latch.await();\n StatusResponse resp = JsonUtils.GSON.fromJson(result, StatusResponse.class);\n Assert.assertEquals(resp.getStatus(), expected);\n }\n\n private void setConfiguration(String key, String val)\n throws NoSuchFieldException, IllegalAccessException {\n Field f = configManager.getClass().getDeclaredField(\"prop\");\n f.setAccessible(true);\n Properties p = (Properties) f.get(configManager);\n p.setProperty(key, val);\n }\n\n private void testModelRegisterWithDefaultWorkers(Channel mgmtChannel)\n throws NoSuchFieldException, IllegalAccessException, InterruptedException {\n setConfiguration(\"default_workers_per_model\", \"1\");\n loadTests(mgmtChannel, \"noop-v1.0\", \"noop_default_model_workers\");\n\n result = null;\n latch = new CountDownLatch(1);\n HttpRequest req =\n new DefaultFullHttpRequest(\n HttpVersion.HTTP_1_1, HttpMethod.GET, \"/models/noop_default_model_workers\");\n mgmtChannel.writeAndFlush(req);\n\n latch.await();\n DescribeModelResponse resp = JsonUtils.GSON.fromJson(result, DescribeModelResponse.class);\n Assert.assertEquals(httpStatus, HttpResponseStatus.OK);\n Assert.assertEquals(resp.getMinWorkers(), 1);\n unloadTests(mgmtChannel, \"noop_default_model_workers\");\n setConfiguration(\"default_workers_per_model\", \"0\");\n }\n\n private void testPredictionsDecodeRequest(Channel inferChannel, Channel mgmtChannel)\n throws InterruptedException, NoSuchFieldException, IllegalAccessException {\n setConfiguration(\"decode_input_request\", \"true\");\n loadTests(mgmtChannel, \"noop-v1.0-config-tests\", \"noop-config\");\n\n result = null;\n latch = new CountDownLatch(1);\n DefaultFullHttpRequest req =\n new DefaultFullHttpRequest(\n HttpVersion.HTTP_1_1, HttpMethod.POST, \"/predictions/noop-config\");\n req.content().writeCharSequence(\"{\\\"data\\\": \\\"test\\\"}\", CharsetUtil.UTF_8);\n HttpUtil.setContentLength(req, req.content().readableBytes());\n req.headers().set(HttpHeaderNames.CONTENT_TYPE, HttpHeaderValues.APPLICATION_JSON);\n inferChannel.writeAndFlush(req);\n\n latch.await();\n\n Assert.assertEquals(httpStatus, HttpResponseStatus.OK);\n Assert.assertFalse(result.contains(\"bytearray\"));\n unloadTests(mgmtChannel, \"noop-config\");\n }\n\n private void testPredictionsDoNotDecodeRequest(Channel inferChannel, Channel mgmtChannel)\n throws InterruptedException, NoSuchFieldException, IllegalAccessException {\n setConfiguration(\"decode_input_request\", \"false\");\n loadTests(mgmtChannel, \"noop-v1.0-config-tests\", \"noop-config\");\n\n result = null;\n latch = new CountDownLatch(1);\n DefaultFullHttpRequest req =\n new DefaultFullHttpRequest(\n HttpVersion.HTTP_1_1, HttpMethod.POST, \"/predictions/noop-config\");\n req.content().writeCharSequence(\"{\\\"data\\\": \\\"test\\\"}\", CharsetUtil.UTF_8);\n HttpUtil.setContentLength(req, req.content().readableBytes());\n req.headers().set(HttpHeaderNames.CONTENT_TYPE, HttpHeaderValues.APPLICATION_JSON);\n inferChannel.writeAndFlush(req);\n\n latch.await();\n\n Assert.assertEquals(httpStatus, HttpResponseStatus.OK);\n Assert.assertTrue(result.contains(\"bytearray\"));\n unloadTests(mgmtChannel, \"noop-config\");\n }\n\n private void testPredictionsModifyResponseHeader(\n Channel inferChannel, Channel managementChannel)\n throws NoSuchFieldException, IllegalAccessException, InterruptedException {\n setConfiguration(\"decode_input_request\", \"false\");\n loadTests(managementChannel, \"respheader-test\", \"respheader\");\n result = null;\n latch = new CountDownLatch(1);\n DefaultFullHttpRequest req =\n new DefaultFullHttpRequest(\n HttpVersion.HTTP_1_1, HttpMethod.POST, \"/predictions/respheader\");\n\n req.content().writeCharSequence(\"{\\\"data\\\": \\\"test\\\"}\", CharsetUtil.UTF_8);\n HttpUtil.setContentLength(req, req.content().readableBytes());\n req.headers().set(HttpHeaderNames.CONTENT_TYPE, HttpHeaderValues.APPLICATION_JSON);\n inferChannel.writeAndFlush(req);\n\n latch.await();\n\n Assert.assertEquals(httpStatus, HttpResponseStatus.OK);\n Assert.assertEquals(headers.get(\"dummy\"), \"1\");\n Assert.assertEquals(headers.get(\"content-type\"), \"text/plain\");\n Assert.assertTrue(result.contains(\"bytearray\"));\n unloadTests(managementChannel, \"respheader\");\n }\n\n private void testPredictionsNoManifest(Channel inferChannel, Channel mgmtChannel)\n throws InterruptedException, NoSuchFieldException, IllegalAccessException {\n setConfiguration(\"default_service_handler\", \"service:handle\");\n loadTests(mgmtChannel, \"noop-no-manifest\", \"nomanifest\");\n\n result = null;\n latch = new CountDownLatch(1);\n DefaultFullHttpRequest req =\n new DefaultFullHttpRequest(\n HttpVersion.HTTP_1_1, HttpMethod.POST, \"/predictions/nomanifest\");\n req.content().writeCharSequence(\"{\\\"data\\\": \\\"test\\\"}\", CharsetUtil.UTF_8);\n HttpUtil.setContentLength(req, req.content().readableBytes());\n req.headers().set(HttpHeaderNames.CONTENT_TYPE, HttpHeaderValues.APPLICATION_JSON);\n inferChannel.writeAndFlush(req);\n\n latch.await();\n\n Assert.assertEquals(httpStatus, HttpResponseStatus.OK);\n Assert.assertEquals(result, \"OK\");\n unloadTests(mgmtChannel, \"nomanifest\");\n }\n\n private void testLegacyPredict(Channel channel) throws InterruptedException {\n result = null;\n latch = new CountDownLatch(1);\n DefaultFullHttpRequest req =\n new DefaultFullHttpRequest(\n HttpVersion.HTTP_1_1, HttpMethod.GET, \"/noop/predict?data=test\");\n channel.writeAndFlush(req);\n\n latch.await();\n Assert.assertEquals(result, \"OK\");\n }\n\n private void testInvalidRootRequest() throws InterruptedException {\n Channel channel = connect(false);\n Assert.assertNotNull(channel);\n\n HttpRequest req = new DefaultFullHttpRequest(HttpVersion.HTTP_1_1, HttpMethod.GET, \"/\");\n channel.writeAndFlush(req).sync();\n channel.closeFuture().sync();\n\n ErrorResponse resp = JsonUtils.GSON.fromJson(result, ErrorResponse.class);\n\n Assert.assertEquals(resp.getCode(), HttpResponseStatus.METHOD_NOT_ALLOWED.code());\n Assert.assertEquals(resp.getMessage(), ERROR_METHOD_NOT_ALLOWED);\n }\n\n private void testInvalidInferenceUri() throws InterruptedException {\n Channel channel = connect(false);\n Assert.assertNotNull(channel);\n\n HttpRequest req =\n new DefaultFullHttpRequest(HttpVersion.HTTP_1_1, HttpMethod.GET, \"/InvalidUrl\");\n channel.writeAndFlush(req).sync();\n channel.closeFuture().sync();\n\n ErrorResponse resp = JsonUtils.GSON.fromJson(result, ErrorResponse.class);\n\n Assert.assertEquals(resp.getCode(), HttpResponseStatus.NOT_FOUND.code());\n Assert.assertEquals(resp.getMessage(), ERROR_NOT_FOUND);\n }\n\n private void testInvalidDescribeModel() throws InterruptedException {\n Channel channel = connect(false);\n Assert.assertNotNull(channel);\n\n HttpRequest req =\n new DefaultFullHttpRequest(\n HttpVersion.HTTP_1_1, HttpMethod.OPTIONS, \"/predictions/InvalidModel\");\n channel.writeAndFlush(req).sync();\n channel.closeFuture().sync();\n\n ErrorResponse resp = JsonUtils.GSON.fromJson(result, ErrorResponse.class);\n\n Assert.assertEquals(resp.getCode(), HttpResponseStatus.NOT_FOUND.code());\n Assert.assertEquals(resp.getMessage(), \"Model not found: InvalidModel\");\n }\n\n private void testInvalidPredictionsUri() throws InterruptedException {\n Channel channel = connect(false);\n Assert.assertNotNull(channel);\n\n HttpRequest req =\n new DefaultFullHttpRequest(HttpVersion.HTTP_1_1, HttpMethod.GET, \"/predictions\");\n channel.writeAndFlush(req).sync();\n channel.closeFuture().sync();\n\n ErrorResponse resp = JsonUtils.GSON.fromJson(result, ErrorResponse.class);\n\n Assert.assertEquals(resp.getCode(), HttpResponseStatus.NOT_FOUND.code());\n Assert.assertEquals(resp.getMessage(), ERROR_NOT_FOUND);\n }\n\n private void testPredictionsModelNotFound() throws InterruptedException {\n Channel channel = connect(false);\n Assert.assertNotNull(channel);\n\n HttpRequest req =\n new DefaultFullHttpRequest(\n HttpVersion.HTTP_1_1, HttpMethod.GET, \"/predictions/InvalidModel\");\n channel.writeAndFlush(req).sync();\n channel.closeFuture().sync();\n\n ErrorResponse resp = JsonUtils.GSON.fromJson(result, ErrorResponse.class);\n\n Assert.assertEquals(resp.getCode(), HttpResponseStatus.NOT_FOUND.code());\n Assert.assertEquals(resp.getMessage(), \"Model not found: InvalidModel\");\n }\n\n private void testInvalidManagementUri() throws InterruptedException {\n Channel channel = connect(true);\n Assert.assertNotNull(channel);\n\n HttpRequest req =\n new DefaultFullHttpRequest(HttpVersion.HTTP_1_1, HttpMethod.GET, \"/InvalidUrl\");\n channel.writeAndFlush(req).sync();\n channel.closeFuture().sync();\n\n ErrorResponse resp = JsonUtils.GSON.fromJson(result, ErrorResponse.class);\n\n Assert.assertEquals(resp.getCode(), HttpResponseStatus.NOT_FOUND.code());\n Assert.assertEquals(resp.getMessage(), ERROR_NOT_FOUND);\n }\n\n private void testInvalidModelsMethod() throws InterruptedException {\n Channel channel = connect(true);\n Assert.assertNotNull(channel);\n\n HttpRequest req =\n new DefaultFullHttpRequest(HttpVersion.HTTP_1_1, HttpMethod.PUT, \"/models\");\n channel.writeAndFlush(req).sync();\n channel.closeFuture().sync();\n\n ErrorResponse resp = JsonUtils.GSON.fromJson(result, ErrorResponse.class);\n\n Assert.assertEquals(resp.getCode(), HttpResponseStatus.METHOD_NOT_ALLOWED.code());\n Assert.assertEquals(resp.getMessage(), ERROR_METHOD_NOT_ALLOWED);\n }\n\n private void testInvalidModelMethod() throws InterruptedException {\n Channel channel = connect(true);\n Assert.assertNotNull(channel);\n\n HttpRequest req =\n new DefaultFullHttpRequest(HttpVersion.HTTP_1_1, HttpMethod.POST, \"/models/noop\");\n channel.writeAndFlush(req).sync();\n channel.closeFuture().sync();\n\n ErrorResponse resp = JsonUtils.GSON.fromJson(result, ErrorResponse.class);\n\n Assert.assertEquals(resp.getCode(), HttpResponseStatus.METHOD_NOT_ALLOWED.code());\n Assert.assertEquals(resp.getMessage(), ERROR_METHOD_NOT_ALLOWED);\n }\n\n private void testDescribeModelNotFound() throws InterruptedException {\n Channel channel = connect(true);\n Assert.assertNotNull(channel);\n\n HttpRequest req =\n new DefaultFullHttpRequest(\n HttpVersion.HTTP_1_1, HttpMethod.GET, \"/models/InvalidModel\");\n channel.writeAndFlush(req).sync();\n channel.closeFuture().sync();\n\n ErrorResponse resp = JsonUtils.GSON.fromJson(result, ErrorResponse.class);\n\n Assert.assertEquals(resp.getCode(), HttpResponseStatus.NOT_FOUND.code());\n Assert.assertEquals(resp.getMessage(), \"Model not found: InvalidModel\");\n }\n\n private void testRegisterModelMissingUrl() throws InterruptedException {\n Channel channel = connect(true);\n Assert.assertNotNull(channel);\n\n HttpRequest req =\n new DefaultFullHttpRequest(HttpVersion.HTTP_1_1, HttpMethod.POST, \"/models\");\n channel.writeAndFlush(req).sync();\n channel.closeFuture().sync();\n\n ErrorResponse resp = JsonUtils.GSON.fromJson(result, ErrorResponse.class);\n\n Assert.assertEquals(resp.getCode(), HttpResponseStatus.BAD_REQUEST.code());\n Assert.assertEquals(resp.getMessage(), \"Parameter url is required.\");\n }\n\n private void testRegisterModelInvalidRuntime() throws InterruptedException {\n Channel channel = connect(true);\n Assert.assertNotNull(channel);\n\n HttpRequest req =\n new DefaultFullHttpRequest(\n HttpVersion.HTTP_1_1,\n HttpMethod.POST,\n \"/models?url=InvalidUrl&runtime=InvalidRuntime\");\n channel.writeAndFlush(req).sync();\n channel.closeFuture().sync();\n\n ErrorResponse resp = JsonUtils.GSON.fromJson(result, ErrorResponse.class);\n\n Assert.assertEquals(resp.getCode(), HttpResponseStatus.BAD_REQUEST.code());\n Assert.assertEquals(resp.getMessage(), \"Invalid RuntimeType value: InvalidRuntime\");\n }\n\n private void testRegisterModelNotFound() throws InterruptedException {\n Channel channel = connect(true);\n Assert.assertNotNull(channel);\n\n HttpRequest req =\n new DefaultFullHttpRequest(\n HttpVersion.HTTP_1_1, HttpMethod.POST, \"/models?url=InvalidUrl\");\n channel.writeAndFlush(req).sync();\n channel.closeFuture().sync();\n\n ErrorResponse resp = JsonUtils.GSON.fromJson(result, ErrorResponse.class);\n\n Assert.assertEquals(resp.getCode(), HttpResponseStatus.NOT_FOUND.code());\n Assert.assertEquals(resp.getMessage(), \"Model not found in model store: InvalidUrl\");\n }\n\n private void testRegisterModelConflict() throws InterruptedException {\n Channel channel = connect(true);\n Assert.assertNotNull(channel);\n\n latch = new CountDownLatch(1);\n DefaultFullHttpRequest req =\n new DefaultFullHttpRequest(\n HttpVersion.HTTP_1_1,\n HttpMethod.POST,\n \"/models?url=noop-v0.1&model_name=noop_v0.1&runtime=python&synchronous=false\");\n channel.writeAndFlush(req);\n latch.await();\n\n req =\n new DefaultFullHttpRequest(\n HttpVersion.HTTP_1_1,\n HttpMethod.POST,\n \"/models?url=noop-v0.1&model_name=noop_v0.1&runtime=python&synchronous=false\");\n channel.writeAndFlush(req);\n channel.closeFuture().sync();\n\n ErrorResponse resp = JsonUtils.GSON.fromJson(result, ErrorResponse.class);\n Assert.assertEquals(resp.getCode(), HttpResponseStatus.CONFLICT.code());\n Assert.assertEquals(resp.getMessage(), \"Model noop_v0.1 is already registered.\");\n }\n\n private void testRegisterModelMalformedUrl() throws InterruptedException {\n Channel channel = connect(true);\n Assert.assertNotNull(channel);\n\n HttpRequest req =\n new DefaultFullHttpRequest(\n HttpVersion.HTTP_1_1,\n HttpMethod.POST,\n \"/models?url=http%3A%2F%2Flocalhost%3Aaaaa\");\n channel.writeAndFlush(req).sync();\n channel.closeFuture().sync();\n\n ErrorResponse resp = JsonUtils.GSON.fromJson(result, ErrorResponse.class);\n\n Assert.assertEquals(resp.getCode(), HttpResponseStatus.NOT_FOUND.code());\n Assert.assertEquals(resp.getMessage(), \"Invalid model url: http://localhost:aaaa\");\n }\n\n private void testRegisterModelConnectionFailed() throws InterruptedException {\n Channel channel = connect(true);\n Assert.assertNotNull(channel);\n\n HttpRequest req =\n new DefaultFullHttpRequest(\n HttpVersion.HTTP_1_1,\n HttpMethod.POST,\n \"/models?url=http%3A%2F%2Flocalhost%3A18888%2Ffake.mar&synchronous=false\");\n channel.writeAndFlush(req).sync();\n channel.closeFuture().sync();\n\n ErrorResponse resp = JsonUtils.GSON.fromJson(result, ErrorResponse.class);\n\n Assert.assertEquals(resp.getCode(), HttpResponseStatus.BAD_REQUEST.code());\n Assert.assertEquals(\n resp.getMessage(),\n \"Failed to download model from: http://localhost:18888/fake.mar\");\n }\n\n private void testRegisterModelHttpError() throws InterruptedException {\n Channel channel = connect(true);\n Assert.assertNotNull(channel);\n\n HttpRequest req =\n new DefaultFullHttpRequest(\n HttpVersion.HTTP_1_1,\n HttpMethod.POST,\n \"/models?url=https%3A%2F%2Flocalhost%3A8443%2Ffake.mar&synchronous=false\");\n channel.writeAndFlush(req).sync();\n channel.closeFuture().sync();\n\n ErrorResponse resp = JsonUtils.GSON.fromJson(result, ErrorResponse.class);\n\n Assert.assertEquals(resp.getCode(), HttpResponseStatus.BAD_REQUEST.code());\n Assert.assertEquals(\n resp.getMessage(),\n \"Failed to download model from: https://localhost:8443/fake.mar, code: 404\");\n }\n\n private void testRegisterModelInvalidPath() throws InterruptedException {\n Channel channel = connect(true);\n Assert.assertNotNull(channel);\n\n HttpRequest req =\n new DefaultFullHttpRequest(\n HttpVersion.HTTP_1_1,\n HttpMethod.POST,\n \"/models?url=..%2Ffake.mar&synchronous=false\");\n channel.writeAndFlush(req).sync();\n channel.closeFuture().sync();\n\n ErrorResponse resp = JsonUtils.GSON.fromJson(result, ErrorResponse.class);\n\n Assert.assertEquals(resp.getCode(), HttpResponseStatus.NOT_FOUND.code());\n Assert.assertEquals(resp.getMessage(), \"Relative path is not allowed in url: ../fake.mar\");\n }\n\n private void testScaleModelNotFound() throws InterruptedException {\n Channel channel = connect(true);\n Assert.assertNotNull(channel);\n\n HttpRequest req =\n new DefaultFullHttpRequest(HttpVersion.HTTP_1_1, HttpMethod.PUT, \"/models/fake\");\n channel.writeAndFlush(req).sync();\n channel.closeFuture().sync();\n\n ErrorResponse resp = JsonUtils.GSON.fromJson(result, ErrorResponse.class);\n\n Assert.assertEquals(resp.getCode(), HttpResponseStatus.NOT_FOUND.code());\n Assert.assertEquals(resp.getMessage(), \"Model not found: fake\");\n }\n\n private void testUnregisterModelNotFound() throws InterruptedException {\n Channel channel = connect(true);\n Assert.assertNotNull(channel);\n\n HttpRequest req =\n new DefaultFullHttpRequest(HttpVersion.HTTP_1_1, HttpMethod.DELETE, \"/models/fake\");\n channel.writeAndFlush(req).sync();\n channel.closeFuture().sync();\n\n ErrorResponse resp = JsonUtils.GSON.fromJson(result, ErrorResponse.class);\n\n Assert.assertEquals(resp.getCode(), HttpResponseStatus.NOT_FOUND.code());\n Assert.assertEquals(resp.getMessage(), \"Model not found: fake\");\n }\n\n private void testUnregisterModelTimeout()\n throws InterruptedException, NoSuchFieldException, IllegalAccessException {\n Channel channel = connect(true);\n setConfiguration(\"unregister_model_timeout\", \"0\");\n\n DefaultFullHttpRequest req =\n new DefaultFullHttpRequest(\n HttpVersion.HTTP_1_1, HttpMethod.DELETE, \"/models/noop_v0.1\");\n channel.writeAndFlush(req).sync();\n channel.closeFuture().sync();\n\n ErrorResponse resp = JsonUtils.GSON.fromJson(result, ErrorResponse.class);\n System.out.print(\"testUnregisterModelTimeout \" + resp.getCode());\n Assert.assertEquals(resp.getCode(), HttpResponseStatus.REQUEST_TIMEOUT.code());\n Assert.assertEquals(resp.getMessage(), \"Timed out while cleaning resources: noop_v0.1\");\n\n channel = connect(true);\n setConfiguration(\"unregister_model_timeout\", \"120\");\n\n req =\n new DefaultFullHttpRequest(\n HttpVersion.HTTP_1_1, HttpMethod.DELETE, \"/models/noop_v0.1\");\n channel.writeAndFlush(req).sync();\n channel.closeFuture().sync();\n\n Assert.assertEquals(httpStatus, HttpResponseStatus.OK);\n }\n\n private void testScaleModelFailure() throws InterruptedException {\n Channel channel = connect(true);\n Assert.assertNotNull(channel);\n\n httpStatus = null;\n result = null;\n latch = new CountDownLatch(1);\n DefaultFullHttpRequest req =\n new DefaultFullHttpRequest(\n HttpVersion.HTTP_1_1,\n HttpMethod.POST,\n \"/models?url=init-error&model_name=init-error&synchronous=false\");\n channel.writeAndFlush(req);\n latch.await();\n\n Assert.assertEquals(httpStatus, HttpResponseStatus.OK);\n\n httpStatus = null;\n result = null;\n latch = new CountDownLatch(1);\n req =\n new DefaultFullHttpRequest(\n HttpVersion.HTTP_1_1,\n HttpMethod.PUT,\n \"/models/init-error?synchronous=true&min_worker=1\");\n channel.writeAndFlush(req);\n latch.await();\n\n ErrorResponse resp = JsonUtils.GSON.fromJson(result, ErrorResponse.class);\n\n Assert.assertEquals(httpStatus, HttpResponseStatus.INTERNAL_SERVER_ERROR);\n Assert.assertEquals(resp.getCode(), HttpResponseStatus.INTERNAL_SERVER_ERROR.code());\n Assert.assertEquals(resp.getMessage(), \"Failed to start workers\");\n }\n\n private void testInvalidModel() throws InterruptedException {\n Channel channel = connect(true);\n Assert.assertNotNull(channel);\n\n httpStatus = null;\n result = null;\n latch = new CountDownLatch(1);\n DefaultFullHttpRequest req =\n new DefaultFullHttpRequest(\n HttpVersion.HTTP_1_1,\n HttpMethod.POST,\n \"/models?url=invalid&model_name=invalid&initial_workers=1&synchronous=true\");\n channel.writeAndFlush(req);\n latch.await();\n\n StatusResponse status = JsonUtils.GSON.fromJson(result, StatusResponse.class);\n Assert.assertEquals(status.getStatus(), \"Workers scaled\");\n\n channel.close().sync();\n\n channel = connect(false);\n Assert.assertNotNull(channel);\n\n result = null;\n latch = new CountDownLatch(1);\n req =\n new DefaultFullHttpRequest(\n HttpVersion.HTTP_1_1, HttpMethod.POST, \"/predictions/invalid\");\n req.content().writeCharSequence(\"data=invalid_output\", CharsetUtil.UTF_8);\n HttpUtil.setContentLength(req, req.content().readableBytes());\n req.headers()\n .set(\n HttpHeaderNames.CONTENT_TYPE,\n HttpHeaderValues.APPLICATION_X_WWW_FORM_URLENCODED);\n channel.writeAndFlush(req);\n\n latch.await();\n\n ErrorResponse resp = JsonUtils.GSON.fromJson(result, ErrorResponse.class);\n\n Assert.assertEquals(httpStatus, HttpResponseStatus.SERVICE_UNAVAILABLE);\n Assert.assertEquals(resp.getCode(), HttpResponseStatus.SERVICE_UNAVAILABLE.code());\n Assert.assertEquals(resp.getMessage(), \"Invalid model predict output\");\n }\n\n private void testLoadingMemoryError() throws InterruptedException {\n Channel channel = connect(true);\n Assert.assertNotNull(channel);\n result = null;\n latch = new CountDownLatch(1);\n HttpRequest req =\n new DefaultFullHttpRequest(\n HttpVersion.HTTP_1_1,\n HttpMethod.POST,\n \"/models?url=loading-memory-error&model_name=memory_error&runtime=python&initial_workers=1&synchronous=true\");\n channel.writeAndFlush(req);\n latch.await();\n\n Assert.assertEquals(httpStatus, HttpResponseStatus.INSUFFICIENT_STORAGE);\n channel.close().sync();\n }\n\n private void testPredictionMemoryError() throws InterruptedException {\n // Load the model\n Channel channel = connect(true);\n Assert.assertNotNull(channel);\n result = null;\n latch = new CountDownLatch(1);\n DefaultFullHttpRequest req =\n new DefaultFullHttpRequest(\n HttpVersion.HTTP_1_1,\n HttpMethod.POST,\n \"/models?url=prediction-memory-error&model_name=pred-err&runtime=python&initial_workers=1&synchronous=true\");\n channel.writeAndFlush(req);\n latch.await();\n Assert.assertEquals(httpStatus, HttpResponseStatus.OK);\n channel.close().sync();\n\n // Test for prediction\n channel = connect(false);\n Assert.assertNotNull(channel);\n result = null;\n latch = new CountDownLatch(1);\n req =\n new DefaultFullHttpRequest(\n HttpVersion.HTTP_1_1, HttpMethod.POST, \"/predictions/pred-err\");\n req.content().writeCharSequence(\"data=invalid_output\", CharsetUtil.UTF_8);\n\n channel.writeAndFlush(req);\n latch.await();\n\n Assert.assertEquals(httpStatus, HttpResponseStatus.INSUFFICIENT_STORAGE);\n channel.close().sync();\n\n // Unload the model\n channel = connect(true);\n httpStatus = null;\n latch = new CountDownLatch(1);\n Assert.assertNotNull(channel);\n req =\n new DefaultFullHttpRequest(\n HttpVersion.HTTP_1_1, HttpMethod.DELETE, \"/models/pred-err\");\n channel.writeAndFlush(req);\n latch.await();\n Assert.assertEquals(httpStatus, HttpResponseStatus.OK);\n }\n\n private void testPredictionCustomErrorCode() throws InterruptedException {\n // Load the model\n Channel channel = connect(true);\n Assert.assertNotNull(channel);\n result = null;\n latch = new CountDownLatch(1);\n DefaultFullHttpRequest req =\n new DefaultFullHttpRequest(\n HttpVersion.HTTP_1_1,\n HttpMethod.POST,\n \"/models?url=custom-return-code&model_name=custom-return-code&runtime=python&initial_workers=1&synchronous=true\");\n channel.writeAndFlush(req);\n latch.await();\n Assert.assertEquals(httpStatus, HttpResponseStatus.OK);\n channel.close().sync();\n\n // Test for prediction\n channel = connect(false);\n Assert.assertNotNull(channel);\n result = null;\n latch = new CountDownLatch(1);\n req =\n new DefaultFullHttpRequest(\n HttpVersion.HTTP_1_1, HttpMethod.POST, \"/predictions/custom-return-code\");\n req.content().writeCharSequence(\"data=invalid_output\", CharsetUtil.UTF_8);\n\n channel.writeAndFlush(req);\n latch.await();\n\n ErrorResponse resp = JsonUtils.GSON.fromJson(result, ErrorResponse.class);\n Assert.assertEquals(resp.getMessage(), \"Some Prediction Error\");\n Assert.assertEquals(resp.getCode(), 599);\n }\n\n private void testErrorBatch() throws InterruptedException {\n Channel channel = connect(true);\n Assert.assertNotNull(channel);\n\n httpStatus = null;\n result = null;\n latch = new CountDownLatch(1);\n DefaultFullHttpRequest req =\n new DefaultFullHttpRequest(\n HttpVersion.HTTP_1_1,\n HttpMethod.POST,\n \"/models?url=error_batch&model_name=err_batch&initial_workers=1&synchronous=true\");\n channel.writeAndFlush(req);\n latch.await();\n\n StatusResponse status = JsonUtils.GSON.fromJson(result, StatusResponse.class);\n Assert.assertEquals(status.getStatus(), \"Workers scaled\");\n\n channel.close().sync();\n\n channel = connect(false);\n Assert.assertNotNull(channel);\n\n result = null;\n latch = new CountDownLatch(1);\n httpStatus = null;\n req =\n new DefaultFullHttpRequest(\n HttpVersion.HTTP_1_1, HttpMethod.POST, \"/predictions/err_batch\");\n req.content().writeCharSequence(\"data=invalid_output\", CharsetUtil.UTF_8);\n HttpUtil.setContentLength(req, req.content().readableBytes());\n req.headers()\n .set(\n HttpHeaderNames.CONTENT_TYPE,\n HttpHeaderValues.APPLICATION_X_WWW_FORM_URLENCODED);\n channel.writeAndFlush(req);\n\n latch.await();\n\n Assert.assertEquals(httpStatus, HttpResponseStatus.INSUFFICIENT_STORAGE);\n Assert.assertEquals(result, \"Invalid response\");\n }\n\n private void testMetricManager() throws JsonParseException, InterruptedException {\n MetricManager.scheduleMetrics(configManager);\n MetricManager metricManager = MetricManager.getInstance();\n List metrics = metricManager.getMetrics();\n\n // Wait till first value is read in\n int count = 0;\n while (metrics.isEmpty()) {\n Thread.sleep(500);\n metrics = metricManager.getMetrics();\n Assert.assertTrue(++count < 5);\n }\n for (Metric metric : metrics) {\n if (metric.getMetricName().equals(\"CPUUtilization\")) {\n Assert.assertEquals(metric.getUnit(), \"Percent\");\n }\n if (metric.getMetricName().equals(\"MemoryUsed\")) {\n Assert.assertEquals(metric.getUnit(), \"Megabytes\");\n }\n if (metric.getMetricName().equals(\"DiskUsed\")) {\n List dimensions = metric.getDimensions();\n for (Dimension dimension : dimensions) {\n if (dimension.getName().equals(\"Level\")) {\n Assert.assertEquals(dimension.getValue(), \"Host\");\n }\n }\n }\n }\n }\n\n private void testLogging(Channel inferChannel, Channel mgmtChannel)\n throws NoSuchFieldException, IllegalAccessException, InterruptedException, IOException {\n setConfiguration(\"default_workers_per_model\", \"2\");\n loadTests(mgmtChannel, \"logging\", \"logging\");\n int niter = 5;\n int expected = 2;\n for (int i = 0; i < niter; i++) {\n latch = new CountDownLatch(1);\n DefaultFullHttpRequest req =\n new DefaultFullHttpRequest(\n HttpVersion.HTTP_1_1, HttpMethod.POST, \"/predictions/logging\");\n req.content().writeCharSequence(\"data=test\", CharsetUtil.UTF_8);\n HttpUtil.setContentLength(req, req.content().readableBytes());\n req.headers()\n .set(\n HttpHeaderNames.CONTENT_TYPE,\n HttpHeaderValues.APPLICATION_X_WWW_FORM_URLENCODED);\n inferChannel.writeAndFlush(req);\n latch.await();\n Assert.assertEquals(httpStatus, HttpResponseStatus.OK);\n }\n\n File logfile = new File(\"build/logs/mms_log.log\");\n Assert.assertTrue(logfile.exists());\n Scanner logscanner = new Scanner(logfile, \"UTF-8\");\n int count = 0;\n while (logscanner.hasNextLine()) {\n String line = logscanner.nextLine();\n if (line.contains(\"LoggingService inference [PID]:\")) {\n count = count + 1;\n }\n }\n Logger logger = LoggerFactory.getLogger(ModelServerTest.class);\n logger.info(\"testLogging, found {}, min expected {}.\", count, expected);\n Assert.assertTrue(count >= expected);\n unloadTests(mgmtChannel, \"logging\");\n }\n\n private void testLoggingUnload(Channel inferChannel, Channel mgmtChannel)\n throws NoSuchFieldException, IllegalAccessException, InterruptedException, IOException {\n setConfiguration(\"default_workers_per_model\", \"2\");\n loadTests(mgmtChannel, \"logging\", \"logging\");\n unloadTests(mgmtChannel, \"logging\");\n int expected = 1;\n int count = 0;\n File logfile = new File(\"build/logs/mms_log.log\");\n Assert.assertTrue(logfile.exists());\n Scanner logscanner = new Scanner(logfile, \"UTF-8\");\n while (logscanner.hasNextLine()) {\n String line = logscanner.nextLine();\n if (line.contains(\"Model logging unregistered\")) {\n count = count + 1;\n }\n }\n Assert.assertTrue(count >= expected);\n }\n\n private Channel connect(boolean management) {\n Logger logger = LoggerFactory.getLogger(ModelServerTest.class);\n\n final Connector connector = configManager.getListener(management);\n try {\n Bootstrap b = new Bootstrap();\n final SslContext sslCtx =\n SslContextBuilder.forClient()\n .trustManager(InsecureTrustManagerFactory.INSTANCE)\n .build();\n b.group(Connector.newEventLoopGroup(1))\n .channel(connector.getClientChannel())\n .option(ChannelOption.CONNECT_TIMEOUT_MILLIS, 10000)\n .handler(\n new ChannelInitializer() {\n @Override\n public void initChannel(Channel ch) {\n ChannelPipeline p = ch.pipeline();\n if (connector.isSsl()) {\n p.addLast(sslCtx.newHandler(ch.alloc()));\n }\n p.addLast(new ReadTimeoutHandler(30));\n p.addLast(new HttpClientCodec());\n p.addLast(new HttpContentDecompressor());\n p.addLast(new ChunkedWriteHandler());\n p.addLast(new HttpObjectAggregator(6553600));\n p.addLast(new TestHandler());\n }\n });\n\n return b.connect(connector.getSocketAddress()).sync().channel();\n } catch (Throwable t) {\n logger.warn(\"Connect error.\", t);\n }\n return null;\n }\n\n @ChannelHandler.Sharable\n private class TestHandler extends SimpleChannelInboundHandler {\n\n @Override\n public void channelRead0(ChannelHandlerContext ctx, FullHttpResponse msg) {\n httpStatus = msg.status();\n result = msg.content().toString(StandardCharsets.UTF_8);\n headers = msg.headers();\n latch.countDown();\n }\n\n @Override\n public void exceptionCaught(ChannelHandlerContext ctx, Throwable cause) {\n Logger logger = LoggerFactory.getLogger(TestHandler.class);\n logger.error(\"Unknown exception\", cause);\n ctx.close();\n latch.countDown();\n }\n }\n}\n" }, { "path": "frontend/server/src/test/java/com/amazonaws/ml/mms/TestUtils.java", "content": "/*\n * Copyright 2018 Amazon.com, Inc. or its affiliates. All Rights Reserved.\n *\n * Licensed under the Apache License, Version 2.0 (the \"License\"). You may not use this file except in compliance\n * with the License. A copy of the License is located at\n *\n * http://aws.amazon.com/apache2.0/\n *\n * or in the \"license\" file accompanying this file. This file is distributed on an \"AS IS\" BASIS, WITHOUT WARRANTIES\n * OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions\n * and limitations under the License.\n */\npackage com.amazonaws.ml.mms;\n\nimport io.netty.handler.ssl.util.InsecureTrustManagerFactory;\nimport java.security.GeneralSecurityException;\nimport javax.net.ssl.HttpsURLConnection;\nimport javax.net.ssl.SSLContext;\n\npublic final class TestUtils {\n\n private TestUtils() {}\n\n public static void init() {\n // set up system properties for local IDE debug\n if (System.getProperty(\"mmsConfigFile\") == null) {\n System.setProperty(\"mmsConfigFile\", \"src/test/resources/config.properties\");\n }\n if (System.getProperty(\"METRICS_LOCATION\") == null) {\n System.setProperty(\"METRICS_LOCATION\", \"build/logs\");\n }\n if (System.getProperty(\"LOG_LOCATION\") == null) {\n System.setProperty(\"LOG_LOCATION\", \"build/logs\");\n }\n\n try {\n SSLContext context = SSLContext.getInstance(\"TLS\");\n context.init(null, InsecureTrustManagerFactory.INSTANCE.getTrustManagers(), null);\n\n HttpsURLConnection.setDefaultSSLSocketFactory(context.getSocketFactory());\n\n HttpsURLConnection.setDefaultHostnameVerifier((s, sslSession) -> true);\n } catch (GeneralSecurityException e) {\n // ignore\n }\n }\n}\n" }, { "path": "frontend/server/src/test/java/com/amazonaws/ml/mms/test/TestHelper.java", "content": "/*\n * Copyright 2018 Amazon.com, Inc. or its affiliates. All Rights Reserved.\n *\n * Licensed under the Apache License, Version 2.0 (the \"License\"). You may not use this file except in compliance\n * with the License. A copy of the License is located at\n *\n * http://aws.amazon.com/apache2.0/\n *\n * or in the \"license\" file accompanying this file. This file is distributed on an \"AS IS\" BASIS, WITHOUT WARRANTIES\n * OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions\n * and limitations under the License.\n */\npackage com.amazonaws.ml.mms.test;\n\nimport java.io.File;\nimport java.io.IOException;\nimport java.lang.reflect.Constructor;\nimport java.lang.reflect.Method;\nimport java.net.URL;\nimport java.util.ArrayList;\nimport java.util.Collection;\nimport java.util.Collections;\nimport java.util.Enumeration;\nimport java.util.List;\nimport java.util.jar.JarEntry;\nimport java.util.jar.JarFile;\nimport org.apache.commons.io.FileUtils;\n\npublic final class TestHelper {\n\n private TestHelper() {}\n\n public static void testGetterSetters(Class baseClass)\n throws IOException, ClassNotFoundException {\n List> list = getClasses(baseClass);\n for (Class clazz : list) {\n Constructor[] constructors = clazz.getConstructors();\n Object obj = null;\n for (Constructor con : constructors) {\n try {\n Class[] types = con.getParameterTypes();\n Object[] args = new Object[types.length];\n for (int i = 0; i < args.length; ++i) {\n args[i] = getMockValue(types[i]);\n }\n\n obj = con.newInstance(args);\n } catch (ReflectiveOperationException ignore) {\n // ignore\n }\n }\n if (obj == null) {\n continue;\n }\n\n Method[] methods = clazz.getMethods();\n for (Method method : methods) {\n String methodName = method.getName();\n int parameterCount = method.getParameterCount();\n try {\n if (parameterCount == 0 && methodName.startsWith(\"get\")\n || methodName.startsWith(\"is\")) {\n method.invoke(obj);\n } else if (methodName.startsWith(\"set\") && parameterCount == 1) {\n Class type = method.getParameterTypes()[0];\n method.invoke(obj, getMockValue(type));\n }\n } catch (ReflectiveOperationException ignore) {\n // ignore\n }\n }\n }\n }\n\n private static List> getClasses(Class clazz)\n throws IOException, ClassNotFoundException {\n URL url = clazz.getProtectionDomain().getCodeSource().getLocation();\n String path = url.getPath();\n\n if (!\"file\".equalsIgnoreCase(url.getProtocol())) {\n return Collections.emptyList();\n }\n\n List> classList = new ArrayList<>();\n\n File classPath = new File(path);\n if (classPath.isDirectory()) {\n String rootPath = classPath.getCanonicalPath();\n String[] filters = new String[] {\"class\"};\n Collection files = FileUtils.listFiles(classPath, filters, true);\n for (File file : files) {\n String fileName = file.getCanonicalPath();\n fileName = fileName.substring(rootPath.length() + 1);\n fileName = fileName.substring(0, fileName.lastIndexOf(\".\"));\n fileName = fileName.replace(File.separatorChar, '.');\n\n classList.add(Class.forName(fileName));\n }\n } else if (path.toLowerCase().endsWith(\".jar\")) {\n try (JarFile jarFile = new JarFile(path)) {\n Enumeration en = jarFile.entries();\n while (en.hasMoreElements()) {\n JarEntry entry = en.nextElement();\n String fileName = entry.getName();\n if (fileName.endsWith(\".class\")) {\n fileName = fileName.substring(0, fileName.lastIndexOf(\".\"));\n fileName = fileName.replace('/', '.');\n classList.add(Class.forName(fileName));\n }\n }\n }\n }\n\n return classList;\n }\n\n private static Object getMockValue(Class type) {\n if (type.isPrimitive()) {\n if (type == Boolean.TYPE) {\n return Boolean.TRUE;\n }\n if (type == Character.TYPE) {\n return '0';\n }\n if (type == Byte.TYPE) {\n return (byte) 0;\n }\n if (type == Short.TYPE) {\n return (short) 0;\n }\n if (type == Integer.TYPE) {\n return 0;\n }\n if (type == Long.TYPE) {\n return 0L;\n }\n if (type == Float.TYPE) {\n return 0f;\n }\n if (type == Double.TYPE) {\n return 0d;\n }\n }\n return null;\n }\n}\n" }, { "path": "frontend/server/src/test/java/com/amazonaws/ml/mms/util/ConfigManagerTest.java", "content": "/*\n * Copyright 2018 Amazon.com, Inc. or its affiliates. All Rights Reserved.\n *\n * Licensed under the Apache License, Version 2.0 (the \"License\"). You may not use this file except in compliance\n * with the License. A copy of the License is located at\n *\n * http://aws.amazon.com/apache2.0/\n *\n * or in the \"license\" file accompanying this file. This file is distributed on an \"AS IS\" BASIS, WITHOUT WARRANTIES\n * OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions\n * and limitations under the License.\n */\npackage com.amazonaws.ml.mms.util;\n\nimport com.amazonaws.ml.mms.TestUtils;\nimport com.amazonaws.ml.mms.metrics.Dimension;\nimport com.amazonaws.ml.mms.metrics.Metric;\nimport io.netty.handler.ssl.SslContext;\nimport java.io.File;\nimport java.io.IOException;\nimport java.lang.reflect.Field;\nimport java.security.GeneralSecurityException;\nimport java.util.ArrayList;\nimport java.util.Collections;\nimport java.util.List;\nimport java.util.Map;\nimport org.junit.Assert;\nimport org.slf4j.Logger;\nimport org.slf4j.LoggerFactory;\nimport org.testng.annotations.Test;\n\npublic class ConfigManagerTest {\n static {\n TestUtils.init();\n }\n\n private Metric createMetric(String metricName, String requestId) {\n List dimensions = new ArrayList<>();\n Metric metric = new Metric();\n metric.setMetricName(metricName);\n metric.setRequestId(requestId);\n metric.setUnit(\"Milliseconds\");\n metric.setTimestamp(\"1542157988\");\n Dimension dimension = new Dimension();\n dimension.setName(\"Level\");\n dimension.setValue(\"Model\");\n dimensions.add(dimension);\n metric.setDimensions(dimensions);\n return metric;\n }\n\n @SuppressWarnings(\"unchecked\")\n private void modifyEnv(String key, String val)\n throws ClassNotFoundException, NoSuchFieldException, IllegalAccessException {\n try {\n Class processEnvironmentClass = Class.forName(\"java.lang.ProcessEnvironment\");\n Field theEnvironmentField = processEnvironmentClass.getDeclaredField(\"theEnvironment\");\n theEnvironmentField.setAccessible(true);\n Map env = (Map) theEnvironmentField.get(null);\n env.put(key, val);\n Field theCIEField =\n processEnvironmentClass.getDeclaredField(\"theCaseInsensitiveEnvironment\");\n theCIEField.setAccessible(true);\n Map cienv = (Map) theCIEField.get(null);\n cienv.put(key, val);\n } catch (NoSuchFieldException e) {\n Class[] classes = Collections.class.getDeclaredClasses();\n Map env = System.getenv();\n for (Class cl : classes) {\n if (\"java.util.Collections$UnmodifiableMap\".equals(cl.getName())) {\n Field field = cl.getDeclaredField(\"m\");\n field.setAccessible(true);\n Object obj = field.get(env);\n Map map = (Map) obj;\n map.clear();\n map.put(key, val);\n }\n }\n }\n }\n\n @Test\n public void test()\n throws IOException, GeneralSecurityException, IllegalAccessException,\n NoSuchFieldException, ClassNotFoundException {\n modifyEnv(\"MMS_DEFAULT_RESPONSE_TIMEOUT\", \"130\");\n ConfigManager.Arguments args = new ConfigManager.Arguments();\n args.setModels(new String[] {\"noop_v0.1\"});\n ConfigManager.init(args);\n ConfigManager configManager = ConfigManager.getInstance();\n configManager.setProperty(\"keystore\", \"src/test/resources/keystore.p12\");\n Assert.assertEquals(\"true\", configManager.getEnableEnvVarsConfig());\n Assert.assertEquals(60 * 130, configManager.getDefaultResponseTimeoutSeconds());\n\n Dimension dimension;\n List metrics = new ArrayList<>();\n // Create two metrics and add them to a list\n\n metrics.add(createMetric(\"TestMetric1\", \"12345\"));\n metrics.add(createMetric(\"TestMetric2\", \"23478\"));\n Logger logger = LoggerFactory.getLogger(ConfigManager.MODEL_SERVER_METRICS_LOGGER);\n logger.debug(\"{}\", metrics);\n Assert.assertTrue(new File(\"build/logs/mms_metrics.log\").exists());\n\n logger = LoggerFactory.getLogger(ConfigManager.MODEL_METRICS_LOGGER);\n logger.debug(\"{}\", metrics);\n Assert.assertTrue(new File(\"build/logs/model_metrics.log\").exists());\n\n Logger modelLogger = LoggerFactory.getLogger(ConfigManager.MODEL_LOGGER);\n modelLogger.debug(\"test model_log\");\n Assert.assertTrue(new File(\"build/logs/model_log.log\").exists());\n\n SslContext ctx = configManager.getSslContext();\n Assert.assertNotNull(ctx);\n }\n\n @Test\n public void testNoEnvVars()\n throws IllegalAccessException, NoSuchFieldException, ClassNotFoundException {\n System.setProperty(\"mmsConfigFile\", \"src/test/resources/config_test_env.properties\");\n modifyEnv(\"MMS_DEFAULT_RESPONSE_TIMEOUT\", \"130\");\n ConfigManager.Arguments args = new ConfigManager.Arguments();\n args.setModels(new String[] {\"noop_v0.1\"});\n ConfigManager.init(args);\n ConfigManager configManager = ConfigManager.getInstance();\n Assert.assertEquals(\"false\", configManager.getEnableEnvVarsConfig());\n Assert.assertEquals(60 * 120, configManager.getDefaultResponseTimeoutSeconds());\n }\n\n @Test\n public void testResponseTimeoutSeconds()\n throws IOException, GeneralSecurityException, IllegalAccessException,\n NoSuchFieldException, ClassNotFoundException {\n System.setProperty(\"mmsConfigFile\", \"src/test/resources/config.properties\");\n modifyEnv(\"MMS_DEFAULT_RESPONSE_TIMEOUT_SECONDS\", \"130\");\n ConfigManager.Arguments args = new ConfigManager.Arguments();\n args.setModels(new String[] {\"noop_v0.1\"});\n ConfigManager.init(args);\n ConfigManager configManager = ConfigManager.getInstance();\n Assert.assertEquals(\"true\", configManager.getEnableEnvVarsConfig());\n Assert.assertEquals(130, configManager.getDefaultResponseTimeoutSeconds());\n }\n}\n" }, { "path": "frontend/server/src/test/resources/certs.pem", "content": "-----BEGIN CERTIFICATE-----\nMIICiDCCAfGgAwIBAgIEeC8zQzANBgkqhkiG9w0BAQsFADB2MQswCQYDVQQGEwJV\nUzETMBEGA1UECBMKQ2FsaWZvcm5pYTESMBAGA1UEBxMJUGFsbyBBbHRvMRIwEAYD\nVQQKEwlBbWF6b24gQUkxDjAMBgNVBAsTBU14TmV0MRowGAYDVQQDExFtbXMuYW1h\nem9uYXdzLmNvbTAgFw0xODA2MjAwMjExMjhaGA8yMTE3MDExMjAyMTEyOFowdjEL\nMAkGA1UEBhMCVVMxEzARBgNVBAgTCkNhbGlmb3JuaWExEjAQBgNVBAcTCVBhbG8g\nQWx0bzESMBAGA1UEChMJQW1hem9uIEFJMQ4wDAYDVQQLEwVNeE5ldDEaMBgGA1UE\nAxMRbW1zLmFtYXpvbmF3cy5jb20wgZ8wDQYJKoZIhvcNAQEBBQADgY0AMIGJAoGB\nAMcbCEP6kn9pUcap5+kYO/5xEl7SL965gSQ2TbFrVv+sLVkLSK8yTtcILr7RUINz\nFsD151Q7VyQCvpVzkOFew2s2mAFWWxPJYmxo1j/R3IkJakrrTrMy1R3jsqOQMrxY\nTLGR5LIe2pjdAnb9xWe2NB125619WDG7RrdHWZDfvSPxAgMBAAGjITAfMB0GA1Ud\nDgQWBBRWjdEyNchYAkdPoyudKJY9YP3JPzANBgkqhkiG9w0BAQsFAAOBgQBMAvqG\ncqvD3ColO2Ihgb/LCfCdV14e1YhusVFeKyZkSKFYyQR+MoBOxqMQqJ24gVzgqTU/\nh+LkMqZcxxJAME08BzPgP5b06DBM4K0o0XUfYUViFpYXB0qCG5CA/0S7ONldBGaZ\nfv6JrnQ/a1NYBi92AaqXA4VmuaowWLVEFuPV1A==\n-----END CERTIFICATE-----\n" }, { "path": "frontend/server/src/test/resources/config.properties", "content": "# debug=true\n# vmargs=-Xmx128m -XX:-UseLargePages -XX:+UseG1GC -XX:MaxMetaspaceSize=32M -XX:MaxDirectMemorySize=10m -XX:+ExitOnOutOfMemoryError\ninference_address=https://127.0.0.1:8443\nmanagement_address=unix:/tmp/management.sock\n# model_server_home=../..\nmodel_store=../modelarchive/src/test/resources/models\nload_models=noop-v0.1,noop-v1.0\npreload_model=false\n# number_of_netty_threads=0\n# netty_client_threads=0\n# default_workers_per_model=0\n# job_queue_size=100\n# plugins_path=/tmp/plugins\nasync_logging=true\ndefault_response_timeout=120\nunregister_model_timeout=120\n# number_of_gpu=1\n# cors_allowed_origin\n# cors_allowed_methods\n# cors_allowed_headers\n# keystore=src/test/resources/keystore.p12\n# keystore_pass=changeit\n# keystore_type=PKCS12\nprivate_key_file=src/test/resources/key.pem\ncertificate_file=src/test/resources/certs.pem\n# max_response_size=6553500\nmax_request_size=10485760\n# blacklist_env_vars=.*USERNAME.*|.*PASSWORD.*\n# enable_env_vars_config=false\n# decode_input_request=true\nenable_envvars_config=true\n# default_service_handler=/path/to/service.py:handle\n" }, { "path": "frontend/server/src/test/resources/config_test_env.properties", "content": "# debug=true\n# vmargs=-Xmx128m -XX:-UseLargePages -XX:+UseG1GC -XX:MaxMetaspaceSize=32M -XX:MaxDirectMemorySize=10m -XX:+ExitOnOutOfMemoryError\ninference_address=https://127.0.0.1:8443\nmanagement_address=unix:/tmp/management.sock\n# model_server_home=../..\nmodel_store=../modelarchive/src/test/resources/models\nload_models=noop-v0.1,noop-v1.0\n# number_of_netty_threads=0\n# netty_client_threads=0\n# default_workers_per_model=0\n# job_queue_size=100\nasync_logging=true\ndefault_response_timeout=120\nunregister_model_timeout=120\n# number_of_gpu=1\n# cors_allowed_origin\n# cors_allowed_methods\n# cors_allowed_headers\n# keystore=src/test/resources/keystore.p12\n# keystore_pass=changeit\n# keystore_type=PKCS12\nprivate_key_file=src/test/resources/key.pem\ncertificate_file=src/test/resources/certs.pem\n# max_response_size=6553500\nmax_request_size=10485760\n# blacklist_env_vars=.*USERNAME.*|.*PASSWORD.*\n# enable_env_vars_config=false\n# decode_input_request=true\n# enable_envvars_config=false\n" }, { "path": "frontend/server/src/test/resources/describe_api.json", "content": "{\n \"openapi\": \"3.0.1\",\n \"info\": {\n \"title\": \"RESTful API for: noop_v0.1\",\n \"version\": \"1.0.0\"\n },\n \"paths\": {\n \"/prediction/noop_v0.1\": {\n \"post\": {\n \"description\": \"A predict entry point for model: noop_v0.1.\",\n \"operationId\": \"noop_v0.1\",\n \"parameters\": [],\n \"responses\": {\n \"200\": {\n \"description\": \"OK\"\n },\n \"500\": {\n \"description\": \"Internal Server Error\",\n \"content\": {\n \"application/json\": {\n \"schema\": {\n \"type\": \"object\",\n \"required\": [\n \"code\",\n \"type\",\n \"message\"\n ],\n \"properties\": {\n \"code\": {\n \"type\": \"integer\",\n \"description\": \"Error code.\"\n },\n \"type\": {\n \"type\": \"string\",\n \"description\": \"Error type.\"\n },\n \"message\": {\n \"type\": \"string\",\n \"description\": \"Error message.\"\n }\n }\n }\n }\n }\n }\n }\n }\n }\n }\n}\n" }, { "path": "frontend/server/src/test/resources/inference_open_api.json", "content": "{\n \"openapi\": \"3.0.1\",\n \"info\": {\n \"title\": \"Model Server APIs\",\n \"description\": \"Model Server is a flexible and easy to use tool for serving deep learning models\",\n \"version\": \"1.0.0\"\n },\n \"paths\": {\n \"/\": {\n \"options\": {\n \"operationId\": \"apiDescription\",\n \"parameters\": [],\n \"responses\": {\n \"200\": {\n \"description\": \"A openapi 3.0.1 descriptor\",\n \"content\": {\n \"application/json\": {\n \"schema\": {\n \"type\": \"object\",\n \"required\": [\n \"openapi\",\n \"info\",\n \"paths\"\n ],\n \"properties\": {\n \"openapi\": {\n \"type\": \"string\"\n },\n \"info\": {\n \"type\": \"object\"\n },\n \"paths\": {\n \"type\": \"object\"\n }\n }\n }\n }\n }\n },\n \"500\": {\n \"description\": \"Internal Server Error\",\n \"content\": {\n \"application/json\": {\n \"schema\": {\n \"type\": \"object\",\n \"required\": [\n \"code\",\n \"type\",\n \"message\"\n ],\n \"properties\": {\n \"code\": {\n \"type\": \"integer\",\n \"description\": \"Error code.\"\n },\n \"type\": {\n \"type\": \"string\",\n \"description\": \"Error type.\"\n },\n \"message\": {\n \"type\": \"string\",\n \"description\": \"Error message.\"\n }\n }\n }\n }\n }\n }\n }\n }\n },\n \"/{model_name}/predict\": {\n \"post\": {\n \"description\": \"A legacy predict entry point for each model.\",\n \"operationId\": \"predict\",\n \"parameters\": [\n {\n \"in\": \"path\",\n \"name\": \"model_name\",\n \"description\": \"Name of model to unregister.\",\n \"required\": true,\n \"schema\": {\n \"type\": \"string\"\n }\n }\n ],\n \"requestBody\": {\n \"description\": \"Input data format is defined by each model.\",\n \"content\": {\n \"*/*\": {\n \"schema\": {\n \"type\": \"string\",\n \"format\": \"binary\"\n }\n }\n },\n \"required\": true\n },\n \"responses\": {\n \"200\": {\n \"description\": \"Model specific output data format\",\n \"content\": {\n \"*/*\": {\n \"schema\": {\n \"type\": \"string\",\n \"format\": \"binary\"\n }\n }\n }\n },\n \"404\": {\n \"description\": \"Model not found\",\n \"content\": {\n \"application/json\": {\n \"schema\": {\n \"type\": \"object\",\n \"required\": [\n \"code\",\n \"type\",\n \"message\"\n ],\n \"properties\": {\n \"code\": {\n \"type\": \"integer\",\n \"description\": \"Error code.\"\n },\n \"type\": {\n \"type\": \"string\",\n \"description\": \"Error type.\"\n },\n \"message\": {\n \"type\": \"string\",\n \"description\": \"Error message.\"\n }\n }\n }\n }\n }\n },\n \"500\": {\n \"description\": \"Internal Server Error\",\n \"content\": {\n \"application/json\": {\n \"schema\": {\n \"type\": \"object\",\n \"required\": [\n \"code\",\n \"type\",\n \"message\"\n ],\n \"properties\": {\n \"code\": {\n \"type\": \"integer\",\n \"description\": \"Error code.\"\n },\n \"type\": {\n \"type\": \"string\",\n \"description\": \"Error type.\"\n },\n \"message\": {\n \"type\": \"string\",\n \"description\": \"Error message.\"\n }\n }\n }\n }\n }\n },\n \"503\": {\n \"description\": \"No worker is available to serve request\",\n \"content\": {\n \"application/json\": {\n \"schema\": {\n \"type\": \"object\",\n \"required\": [\n \"code\",\n \"type\",\n \"message\"\n ],\n \"properties\": {\n \"code\": {\n \"type\": \"integer\",\n \"description\": \"Error code.\"\n },\n \"type\": {\n \"type\": \"string\",\n \"description\": \"Error type.\"\n },\n \"message\": {\n \"type\": \"string\",\n \"description\": \"Error message.\"\n }\n }\n }\n }\n }\n }\n },\n \"deprecated\": true\n }\n },\n \"/ping\": {\n \"get\": {\n \"operationId\": \"ping\",\n \"parameters\": [],\n \"responses\": {\n \"200\": {\n \"description\": \"Model server status\",\n \"content\": {\n \"application/json\": {\n \"schema\": {\n \"type\": \"object\",\n \"required\": [\n \"status\"\n ],\n \"properties\": {\n \"status\": {\n \"type\": \"string\",\n \"description\": \"Overall status of the Model Server.\"\n }\n }\n }\n }\n }\n },\n \"500\": {\n \"description\": \"Internal Server Error\",\n \"content\": {\n \"application/json\": {\n \"schema\": {\n \"type\": \"object\",\n \"required\": [\n \"code\",\n \"type\",\n \"message\"\n ],\n \"properties\": {\n \"code\": {\n \"type\": \"integer\",\n \"description\": \"Error code.\"\n },\n \"type\": {\n \"type\": \"string\",\n \"description\": \"Error type.\"\n },\n \"message\": {\n \"type\": \"string\",\n \"description\": \"Error message.\"\n }\n }\n }\n }\n }\n }\n }\n }\n },\n \"/predictions/{model_name}\": {\n \"post\": {\n \"description\": \"Predictions entry point for each model. Use OPTIONS method to get detailed model API input and output description.\",\n \"operationId\": \"predictions\",\n \"parameters\": [\n {\n \"in\": \"path\",\n \"name\": \"model_name\",\n \"description\": \"Name of model.\",\n \"required\": true,\n \"schema\": {\n \"type\": \"string\"\n }\n }\n ],\n \"requestBody\": {\n \"description\": \"Input data format is defined by each model. Use OPTIONS method to get details for model input format.\",\n \"content\": {\n \"*/*\": {\n \"schema\": {\n \"type\": \"string\",\n \"format\": \"binary\"\n }\n }\n },\n \"required\": true\n },\n \"responses\": {\n \"200\": {\n \"description\": \"Output data format is defined by each model. Use OPTIONS method to get details for model output and output format.\",\n \"content\": {\n \"*/*\": {\n \"schema\": {\n \"type\": \"string\",\n \"format\": \"binary\"\n }\n }\n }\n },\n \"404\": {\n \"description\": \"Model not found\",\n \"content\": {\n \"application/json\": {\n \"schema\": {\n \"type\": \"object\",\n \"required\": [\n \"code\",\n \"type\",\n \"message\"\n ],\n \"properties\": {\n \"code\": {\n \"type\": \"integer\",\n \"description\": \"Error code.\"\n },\n \"type\": {\n \"type\": \"string\",\n \"description\": \"Error type.\"\n },\n \"message\": {\n \"type\": \"string\",\n \"description\": \"Error message.\"\n }\n }\n }\n }\n }\n },\n \"500\": {\n \"description\": \"Internal Server Error\",\n \"content\": {\n \"application/json\": {\n \"schema\": {\n \"type\": \"object\",\n \"required\": [\n \"code\",\n \"type\",\n \"message\"\n ],\n \"properties\": {\n \"code\": {\n \"type\": \"integer\",\n \"description\": \"Error code.\"\n },\n \"type\": {\n \"type\": \"string\",\n \"description\": \"Error type.\"\n },\n \"message\": {\n \"type\": \"string\",\n \"description\": \"Error message.\"\n }\n }\n }\n }\n }\n },\n \"503\": {\n \"description\": \"No worker is available to serve request\",\n \"content\": {\n \"application/json\": {\n \"schema\": {\n \"type\": \"object\",\n \"required\": [\n \"code\",\n \"type\",\n \"message\"\n ],\n \"properties\": {\n \"code\": {\n \"type\": \"integer\",\n \"description\": \"Error code.\"\n },\n \"type\": {\n \"type\": \"string\",\n \"description\": \"Error type.\"\n },\n \"message\": {\n \"type\": \"string\",\n \"description\": \"Error message.\"\n }\n }\n }\n }\n }\n }\n }\n },\n \"options\": {\n \"description\": \"Display details of per model input and output.\",\n \"operationId\": \"predictionsApi\",\n \"parameters\": [\n {\n \"in\": \"path\",\n \"name\": \"model_name\",\n \"description\": \"Name of model.\",\n \"required\": true,\n \"schema\": {\n \"type\": \"string\"\n }\n }\n ],\n \"responses\": {\n \"200\": {\n \"description\": \"OK\",\n \"content\": {\n \"application/json\": {\n \"schema\": {\n \"type\": \"object\"\n }\n }\n }\n }\n }\n }\n },\n \"/api-description\": {\n \"get\": {\n \"operationId\": \"apiDescription\",\n \"parameters\": [],\n \"responses\": {\n \"200\": {\n \"description\": \"A openapi 3.0.1 descriptor\",\n \"content\": {\n \"application/json\": {\n \"schema\": {\n \"type\": \"object\",\n \"required\": [\n \"openapi\",\n \"info\",\n \"paths\"\n ],\n \"properties\": {\n \"openapi\": {\n \"type\": \"string\"\n },\n \"info\": {\n \"type\": \"object\"\n },\n \"paths\": {\n \"type\": \"object\"\n }\n }\n }\n }\n }\n },\n \"500\": {\n \"description\": \"Internal Server Error\",\n \"content\": {\n \"application/json\": {\n \"schema\": {\n \"type\": \"object\",\n \"required\": [\n \"code\",\n \"type\",\n \"message\"\n ],\n \"properties\": {\n \"code\": {\n \"type\": \"integer\",\n \"description\": \"Error code.\"\n },\n \"type\": {\n \"type\": \"string\",\n \"description\": \"Error type.\"\n },\n \"message\": {\n \"type\": \"string\",\n \"description\": \"Error message.\"\n }\n }\n }\n }\n }\n }\n },\n \"deprecated\": true\n }\n },\n \"/invocations\": {\n \"post\": {\n \"description\": \"A generic invocation entry point for all models.\",\n \"operationId\": \"invocations\",\n \"parameters\": [\n {\n \"in\": \"query\",\n \"name\": \"model_name\",\n \"description\": \"Name of model\",\n \"required\": false,\n \"schema\": {\n \"type\": \"string\"\n }\n }\n ],\n \"requestBody\": {\n \"content\": {\n \"multipart/form-data\": {\n \"schema\": {\n \"required\": [\n \"data\"\n ],\n \"properties\": {\n \"model_name\": {\n \"type\": \"string\",\n \"description\": \"Name of model\"\n },\n \"data\": {\n \"type\": \"string\",\n \"format\": \"binary\",\n \"description\": \"Inference input data\"\n }\n }\n }\n }\n },\n \"required\": true\n },\n \"responses\": {\n \"200\": {\n \"description\": \"Model specific output data format\",\n \"content\": {\n \"*/*\": {\n \"schema\": {\n \"required\": [\n \"data\"\n ],\n \"properties\": {\n \"model_name\": {\n \"type\": \"string\",\n \"description\": \"Name of model\"\n },\n \"data\": {\n \"type\": \"string\",\n \"format\": \"binary\",\n \"description\": \"Inference input data\"\n }\n }\n }\n }\n }\n },\n \"400\": {\n \"description\": \"Missing model_name parameter\",\n \"content\": {\n \"application/json\": {\n \"schema\": {\n \"type\": \"object\",\n \"required\": [\n \"code\",\n \"type\",\n \"message\"\n ],\n \"properties\": {\n \"code\": {\n \"type\": \"integer\",\n \"description\": \"Error code.\"\n },\n \"type\": {\n \"type\": \"string\",\n \"description\": \"Error type.\"\n },\n \"message\": {\n \"type\": \"string\",\n \"description\": \"Error message.\"\n }\n }\n }\n }\n }\n },\n \"404\": {\n \"description\": \"Model not found\",\n \"content\": {\n \"application/json\": {\n \"schema\": {\n \"type\": \"object\",\n \"required\": [\n \"code\",\n \"type\",\n \"message\"\n ],\n \"properties\": {\n \"code\": {\n \"type\": \"integer\",\n \"description\": \"Error code.\"\n },\n \"type\": {\n \"type\": \"string\",\n \"description\": \"Error type.\"\n },\n \"message\": {\n \"type\": \"string\",\n \"description\": \"Error message.\"\n }\n }\n }\n }\n }\n },\n \"500\": {\n \"description\": \"Internal Server Error\",\n \"content\": {\n \"application/json\": {\n \"schema\": {\n \"type\": \"object\",\n \"required\": [\n \"code\",\n \"type\",\n \"message\"\n ],\n \"properties\": {\n \"code\": {\n \"type\": \"integer\",\n \"description\": \"Error code.\"\n },\n \"type\": {\n \"type\": \"string\",\n \"description\": \"Error type.\"\n },\n \"message\": {\n \"type\": \"string\",\n \"description\": \"Error message.\"\n }\n }\n }\n }\n }\n },\n \"503\": {\n \"description\": \"No worker is available to serve request\",\n \"content\": {\n \"application/json\": {\n \"schema\": {\n \"type\": \"object\",\n \"required\": [\n \"code\",\n \"type\",\n \"message\"\n ],\n \"properties\": {\n \"code\": {\n \"type\": \"integer\",\n \"description\": \"Error code.\"\n },\n \"type\": {\n \"type\": \"string\",\n \"description\": \"Error type.\"\n },\n \"message\": {\n \"type\": \"string\",\n \"description\": \"Error message.\"\n }\n }\n }\n }\n }\n }\n }\n }\n },\n \"/models/{model_name}/invoke\": {\n \"post\": {\n \"description\": \"A generic invocation entry point for all models.\",\n \"operationId\": \"invocations\",\n \"parameters\": [\n {\n \"in\": \"query\",\n \"name\": \"model_name\",\n \"description\": \"Name of model\",\n \"required\": false,\n \"schema\": {\n \"type\": \"string\"\n }\n }\n ],\n \"requestBody\": {\n \"content\": {\n \"multipart/form-data\": {\n \"schema\": {\n \"required\": [\n \"data\"\n ],\n \"properties\": {\n \"model_name\": {\n \"type\": \"string\",\n \"description\": \"Name of model\"\n },\n \"data\": {\n \"type\": \"string\",\n \"format\": \"binary\",\n \"description\": \"Inference input data\"\n }\n }\n }\n }\n },\n \"required\": true\n },\n \"responses\": {\n \"200\": {\n \"description\": \"Model specific output data format\",\n \"content\": {\n \"*/*\": {\n \"schema\": {\n \"required\": [\n \"data\"\n ],\n \"properties\": {\n \"model_name\": {\n \"type\": \"string\",\n \"description\": \"Name of model\"\n },\n \"data\": {\n \"type\": \"string\",\n \"format\": \"binary\",\n \"description\": \"Inference input data\"\n }\n }\n }\n }\n }\n },\n \"400\": {\n \"description\": \"Missing model_name parameter\",\n \"content\": {\n \"application/json\": {\n \"schema\": {\n \"type\": \"object\",\n \"required\": [\n \"code\",\n \"type\",\n \"message\"\n ],\n \"properties\": {\n \"code\": {\n \"type\": \"integer\",\n \"description\": \"Error code.\"\n },\n \"type\": {\n \"type\": \"string\",\n \"description\": \"Error type.\"\n },\n \"message\": {\n \"type\": \"string\",\n \"description\": \"Error message.\"\n }\n }\n }\n }\n }\n },\n \"404\": {\n \"description\": \"Model not found\",\n \"content\": {\n \"application/json\": {\n \"schema\": {\n \"type\": \"object\",\n \"required\": [\n \"code\",\n \"type\",\n \"message\"\n ],\n \"properties\": {\n \"code\": {\n \"type\": \"integer\",\n \"description\": \"Error code.\"\n },\n \"type\": {\n \"type\": \"string\",\n \"description\": \"Error type.\"\n },\n \"message\": {\n \"type\": \"string\",\n \"description\": \"Error message.\"\n }\n }\n }\n }\n }\n },\n \"500\": {\n \"description\": \"Internal Server Error\",\n \"content\": {\n \"application/json\": {\n \"schema\": {\n \"type\": \"object\",\n \"required\": [\n \"code\",\n \"type\",\n \"message\"\n ],\n \"properties\": {\n \"code\": {\n \"type\": \"integer\",\n \"description\": \"Error code.\"\n },\n \"type\": {\n \"type\": \"string\",\n \"description\": \"Error type.\"\n },\n \"message\": {\n \"type\": \"string\",\n \"description\": \"Error message.\"\n }\n }\n }\n }\n }\n },\n \"503\": {\n \"description\": \"No worker is available to serve request\",\n \"content\": {\n \"application/json\": {\n \"schema\": {\n \"type\": \"object\",\n \"required\": [\n \"code\",\n \"type\",\n \"message\"\n ],\n \"properties\": {\n \"code\": {\n \"type\": \"integer\",\n \"description\": \"Error code.\"\n },\n \"type\": {\n \"type\": \"string\",\n \"description\": \"Error type.\"\n },\n \"message\": {\n \"type\": \"string\",\n \"description\": \"Error message.\"\n }\n }\n }\n }\n }\n }\n }\n }\n }\n }\n}\n" }, { "path": "frontend/server/src/test/resources/key.pem", "content": "-----BEGIN RSA PRIVATE KEY-----\nMIICXAIBAAKBgQDHGwhD+pJ/aVHGqefpGDv+cRJe0i/euYEkNk2xa1b/rC1ZC0iv\nMk7XCC6+0VCDcxbA9edUO1ckAr6Vc5DhXsNrNpgBVlsTyWJsaNY/0dyJCWpK606z\nMtUd47KjkDK8WEyxkeSyHtqY3QJ2/cVntjQdduetfVgxu0a3R1mQ370j8QIDAQAB\nAoGANRoxlyfSQKcPR2PzVUjAX3k6xA1c9RMWrVjKWeJd/qymH5SR2yAYxOMKzJu4\n1IYycF5lRyLYd+M/f06mOmVyysH3D7hkrNz57Z07UrZ0dO/mmUKRL7zc44mo22ck\nJtQRwWJMplgew7N8OyqEZbcLOpahjlkL4+KZIWOuO7X5m30CQQDob/rzNY8gfhEm\noEHHQ4dCqa/b5as2OqpFoGBZ+iX3dumBf+UKuSHlvEozt4ZMm29DYSjhiGXgLUFw\n6NBhWxpXAkEA20oNdGiYAyyGJ6TKkD3FNZYoqB5+E/Cq6c0AACssB4OrJtiGiBFq\nR1h5HTEwYMe+ciZ4CI5MvBukjAdlfn7W9wJAXOIqyTe060oVdncB8ivlCFmgweHk\najZFRq+Q8UPKGjq1kx9VmtRiXFjC2inTjBds/eL8oCuOcmgDR6hxZQYv3wJAcMLv\nkECIinlGsvQGRY297wQ7+9dSNaa3/Gmx6mRIy8RlKiCFbUqnP/C6tswoeFu+DqzB\nZITn6IK+ZlMXWaiXmQJBAK7V4rR+4VdpYUu1OqPRxChkcM+Y4Wa985A46/8yoo3i\n0PEenvApVzhVzS9jF6WEqIKcffBAmOxaXOn3kmn8w2A=\n-----END RSA PRIVATE KEY-----\n" }, { "path": "frontend/server/src/test/resources/management_open_api.json", "content": "{\n \"openapi\": \"3.0.1\",\n \"info\": {\n \"title\": \"Model Server APIs\",\n \"description\": \"Model Server is a flexible and easy to use tool for serving deep learning models\",\n \"version\": \"1.0.0\"\n },\n \"paths\": {\n \"/\": {\n \"options\": {\n \"operationId\": \"apiDescription\",\n \"parameters\": [],\n \"responses\": {\n \"200\": {\n \"description\": \"A openapi 3.0.1 descriptor\",\n \"content\": {\n \"application/json\": {\n \"schema\": {\n \"type\": \"object\",\n \"required\": [\n \"openapi\",\n \"info\",\n \"paths\"\n ],\n \"properties\": {\n \"openapi\": {\n \"type\": \"string\"\n },\n \"info\": {\n \"type\": \"object\"\n },\n \"paths\": {\n \"type\": \"object\"\n }\n }\n }\n }\n }\n },\n \"500\": {\n \"description\": \"Internal Server Error\",\n \"content\": {\n \"application/json\": {\n \"schema\": {\n \"type\": \"object\",\n \"required\": [\n \"code\",\n \"type\",\n \"message\"\n ],\n \"properties\": {\n \"code\": {\n \"type\": \"integer\",\n \"description\": \"Error code.\"\n },\n \"type\": {\n \"type\": \"string\",\n \"description\": \"Error type.\"\n },\n \"message\": {\n \"type\": \"string\",\n \"description\": \"Error message.\"\n }\n }\n }\n }\n }\n }\n }\n }\n },\n \"/models\": {\n \"get\": {\n \"description\": \"List registered models in Model Server.\",\n \"operationId\": \"listModels\",\n \"parameters\": [\n {\n \"in\": \"query\",\n \"name\": \"limit\",\n \"description\": \"Use this parameter to specify the maximum number of items to return. When this value is present, Model Server does not return more than the specified number of items, but it might return fewer. This value is optional. If you include a value, it must be between 1 and 1000, inclusive. If you do not include a value, it defaults to 100.\",\n \"required\": false,\n \"schema\": {\n \"type\": \"integer\",\n \"default\": \"100\"\n }\n },\n {\n \"in\": \"query\",\n \"name\": \"next_page_token\",\n \"description\": \"The token to retrieve the next set of results. Model Server provides the token when the response from a previous call has more results than the maximum page size.\",\n \"required\": false,\n \"schema\": {\n \"type\": \"string\"\n }\n },\n {\n \"in\": \"query\",\n \"name\": \"model_name_pattern\",\n \"description\": \"A model name filter to list only matching models.\",\n \"required\": false,\n \"schema\": {\n \"type\": \"string\"\n }\n }\n ],\n \"responses\": {\n \"200\": {\n \"description\": \"OK\",\n \"content\": {\n \"application/json\": {\n \"schema\": {\n \"type\": \"object\",\n \"required\": [\n \"models\"\n ],\n \"properties\": {\n \"nextPageToken\": {\n \"type\": \"string\",\n \"description\": \"Use this parameter in a subsequent request after you receive a response with truncated results. Set it to the value of NextMarker from the truncated response you just received.\"\n },\n \"models\": {\n \"type\": \"array\",\n \"items\": {\n \"type\": \"object\",\n \"required\": [\n \"modelName\",\n \"modelUrl\"\n ],\n \"properties\": {\n \"modelName\": {\n \"type\": \"string\",\n \"description\": \"Name of the model.\"\n },\n \"modelUrl\": {\n \"type\": \"string\",\n \"description\": \"URL of the model.\"\n }\n }\n },\n \"description\": \"A list of registered models.\"\n }\n }\n }\n }\n }\n },\n \"500\": {\n \"description\": \"Internal Server Error\",\n \"content\": {\n \"application/json\": {\n \"schema\": {\n \"type\": \"object\",\n \"required\": [\n \"code\",\n \"type\",\n \"message\"\n ],\n \"properties\": {\n \"code\": {\n \"type\": \"integer\",\n \"description\": \"Error code.\"\n },\n \"type\": {\n \"type\": \"string\",\n \"description\": \"Error type.\"\n },\n \"message\": {\n \"type\": \"string\",\n \"description\": \"Error message.\"\n }\n }\n }\n }\n }\n }\n }\n },\n \"post\": {\n \"description\": \"Register a new model in Model Server.\",\n \"operationId\": \"registerModel\",\n \"parameters\": [\n {\n \"in\": \"query\",\n \"name\": \"model_url\",\n \"description\": \"Model archive download url, support local file or HTTP(s) protocol. For S3, consider use pre-signed url.\",\n \"required\": true,\n \"schema\": {\n \"type\": \"string\"\n }\n },\n {\n \"in\": \"query\",\n \"name\": \"model_name\",\n \"description\": \"Name of model. This value will override modelName in MANIFEST.json if present.\",\n \"required\": false,\n \"schema\": {\n \"type\": \"string\"\n }\n },\n {\n \"in\": \"query\",\n \"name\": \"handler\",\n \"description\": \"Inference handler entry-point. This value will override handler in MANIFEST.json if present.\",\n \"required\": false,\n \"schema\": {\n \"type\": \"string\"\n }\n },\n {\n \"in\": \"query\",\n \"name\": \"runtime\",\n \"description\": \"Runtime for the model custom service code. This value will override runtime in MANIFEST.json if present.\",\n \"required\": false,\n \"schema\": {\n \"type\": \"string\",\n \"enum\": [\n \"PYTHON\",\n \"PYTHON2\",\n \"PYTHON3\"\n ]\n }\n },\n {\n \"in\": \"query\",\n \"name\": \"batch_size\",\n \"description\": \"Inference batch size, default: 1.\",\n \"required\": false,\n \"schema\": {\n \"type\": \"integer\",\n \"default\": \"1\"\n }\n },\n {\n \"in\": \"query\",\n \"name\": \"max_batch_delay\",\n \"description\": \"Maximum delay for batch aggregation, default: 100.\",\n \"required\": false,\n \"schema\": {\n \"type\": \"integer\",\n \"default\": \"100\"\n }\n },\n {\n \"in\": \"query\",\n \"name\": \"response_timeout\",\n \"description\": \"Maximum time, in seconds, the Model Server waits for a response from the model inference code, default: 120.\",\n \"required\": false,\n \"schema\": {\n \"type\": \"integer\",\n \"default\": \"2\"\n }\n },\n {\n \"in\": \"query\",\n \"name\": \"initial_workers\",\n \"description\": \"Number of initial workers, default: 0.\",\n \"required\": false,\n \"schema\": {\n \"type\": \"integer\",\n \"default\": \"0\"\n }\n },\n {\n \"in\": \"query\",\n \"name\": \"synchronous\",\n \"description\": \"Decides whether creation of worker synchronous or not, default: false.\",\n \"required\": false,\n \"schema\": {\n \"type\": \"boolean\",\n \"default\": \"false\"\n }\n },\n {\n \"in\": \"query\",\n \"name\": \"preload_model\",\n \"description\": \"Decides if model should be preloaded, default: false.\",\n \"required\": false,\n \"schema\": {\n \"type\": \"boolean\",\n \"default\": \"false\"\n }\n }\n ],\n \"responses\": {\n \"200\": {\n \"description\": \"Model registered\",\n \"content\": {\n \"application/json\": {\n \"schema\": {\n \"type\": \"object\",\n \"required\": [\n \"status\"\n ],\n \"properties\": {\n \"status\": {\n \"type\": \"string\",\n \"description\": \"Error type.\"\n }\n }\n }\n }\n }\n },\n \"202\": {\n \"description\": \"Accepted\",\n \"content\": {\n \"application/json\": {\n \"schema\": {\n \"type\": \"object\",\n \"required\": [\n \"status\"\n ],\n \"properties\": {\n \"status\": {\n \"type\": \"string\",\n \"description\": \"Error type.\"\n }\n }\n }\n }\n }\n },\n \"210\": {\n \"description\": \"Partial Success\",\n \"content\": {\n \"application/json\": {\n \"schema\": {\n \"type\": \"object\",\n \"required\": [\n \"status\"\n ],\n \"properties\": {\n \"status\": {\n \"type\": \"string\",\n \"description\": \"Error type.\"\n }\n }\n }\n }\n }\n },\n \"400\": {\n \"description\": \"Bad request\",\n \"content\": {\n \"application/json\": {\n \"schema\": {\n \"type\": \"object\",\n \"required\": [\n \"code\",\n \"type\",\n \"message\"\n ],\n \"properties\": {\n \"code\": {\n \"type\": \"integer\",\n \"description\": \"Error code.\"\n },\n \"type\": {\n \"type\": \"string\",\n \"description\": \"Error type.\"\n },\n \"message\": {\n \"type\": \"string\",\n \"description\": \"Error message.\"\n }\n }\n }\n }\n }\n },\n \"404\": {\n \"description\": \"Model not found\",\n \"content\": {\n \"application/json\": {\n \"schema\": {\n \"type\": \"object\",\n \"required\": [\n \"code\",\n \"type\",\n \"message\"\n ],\n \"properties\": {\n \"code\": {\n \"type\": \"integer\",\n \"description\": \"Error code.\"\n },\n \"type\": {\n \"type\": \"string\",\n \"description\": \"Error type.\"\n },\n \"message\": {\n \"type\": \"string\",\n \"description\": \"Error message.\"\n }\n }\n }\n }\n }\n },\n \"409\": {\n \"description\": \"Model already registered\",\n \"content\": {\n \"application/json\": {\n \"schema\": {\n \"type\": \"object\",\n \"required\": [\n \"code\",\n \"type\",\n \"message\"\n ],\n \"properties\": {\n \"code\": {\n \"type\": \"integer\",\n \"description\": \"Error code.\"\n },\n \"type\": {\n \"type\": \"string\",\n \"description\": \"Error type.\"\n },\n \"message\": {\n \"type\": \"string\",\n \"description\": \"Error message.\"\n }\n }\n }\n }\n }\n },\n \"500\": {\n \"description\": \"Internal Server Error\",\n \"content\": {\n \"application/json\": {\n \"schema\": {\n \"type\": \"object\",\n \"required\": [\n \"code\",\n \"type\",\n \"message\"\n ],\n \"properties\": {\n \"code\": {\n \"type\": \"integer\",\n \"description\": \"Error code.\"\n },\n \"type\": {\n \"type\": \"string\",\n \"description\": \"Error type.\"\n },\n \"message\": {\n \"type\": \"string\",\n \"description\": \"Error message.\"\n }\n }\n }\n }\n }\n }\n }\n }\n },\n \"/models/{model_name}\": {\n \"get\": {\n \"description\": \"Provides detailed information about the specified model.\",\n \"operationId\": \"describeModel\",\n \"parameters\": [\n {\n \"in\": \"path\",\n \"name\": \"model_name\",\n \"description\": \"Name of model to describe.\",\n \"required\": true,\n \"schema\": {\n \"type\": \"string\"\n }\n }\n ],\n \"responses\": {\n \"200\": {\n \"description\": \"OK\",\n \"content\": {\n \"application/json\": {\n \"schema\": {\n \"type\": \"object\",\n \"required\": [\n \"modelName\",\n \"modelVersion\",\n \"modelUrl\",\n \"minWorkers\",\n \"maxWorkers\",\n \"status\",\n \"workers\",\n \"metrics\"\n ],\n \"properties\": {\n \"modelName\": {\n \"type\": \"string\",\n \"description\": \"Name of the model.\"\n },\n \"modelVersion\": {\n \"type\": \"string\",\n \"description\": \"Version of the model.\"\n },\n \"modelUrl\": {\n \"type\": \"string\",\n \"description\": \"URL of the model.\"\n },\n \"minWorkers\": {\n \"type\": \"integer\",\n \"description\": \"Configured minimum number of worker.\"\n },\n \"maxWorkers\": {\n \"type\": \"integer\",\n \"description\": \"Configured maximum number of worker.\"\n },\n \"batchSize\": {\n \"type\": \"integer\",\n \"description\": \"Configured batch size.\"\n },\n \"maxBatchDelay\": {\n \"type\": \"integer\",\n \"description\": \"Configured maximum batch delay in ms.\"\n },\n \"status\": {\n \"type\": \"string\",\n \"description\": \"Overall health status of the model\"\n },\n \"workers\": {\n \"type\": \"array\",\n \"items\": {\n \"type\": \"object\",\n \"required\": [\n \"id\",\n \"startTime\",\n \"status\"\n ],\n \"properties\": {\n \"id\": {\n \"type\": \"string\",\n \"description\": \"Worker id\"\n },\n \"startTime\": {\n \"type\": \"string\",\n \"description\": \"Worker start time\"\n },\n \"gpu\": {\n \"type\": \"boolean\",\n \"description\": \"If running on GPU\"\n },\n \"status\": {\n \"type\": \"string\",\n \"description\": \"Worker status\",\n \"enum\": [\n \"READY\",\n \"LOADING\",\n \"UNLOADING\"\n ]\n }\n }\n },\n \"description\": \"A list of active backend workers.\"\n },\n \"metrics\": {\n \"type\": \"object\",\n \"required\": [\n \"rejectedRequests\",\n \"waitingQueueSize\",\n \"requests\"\n ],\n \"properties\": {\n \"rejectedRequests\": {\n \"type\": \"integer\",\n \"description\": \"Number requests has been rejected in last 10 minutes.\"\n },\n \"waitingQueueSize\": {\n \"type\": \"integer\",\n \"description\": \"Number requests waiting in the queue.\"\n },\n \"requests\": {\n \"type\": \"integer\",\n \"description\": \"Number requests processed in last 10 minutes.\"\n }\n }\n }\n }\n }\n }\n }\n },\n \"500\": {\n \"description\": \"Internal Server Error\",\n \"content\": {\n \"application/json\": {\n \"schema\": {\n \"type\": \"object\",\n \"required\": [\n \"code\",\n \"type\",\n \"message\"\n ],\n \"properties\": {\n \"code\": {\n \"type\": \"integer\",\n \"description\": \"Error code.\"\n },\n \"type\": {\n \"type\": \"string\",\n \"description\": \"Error type.\"\n },\n \"message\": {\n \"type\": \"string\",\n \"description\": \"Error message.\"\n }\n }\n }\n }\n }\n }\n }\n },\n \"put\": {\n \"description\": \"Configure number of workers for a model, This is a asynchronous call by default. Caller need to call describeModel check if the model workers has been changed.\",\n \"operationId\": \"setAutoScale\",\n \"parameters\": [\n {\n \"in\": \"path\",\n \"name\": \"model_name\",\n \"description\": \"Name of model to describe.\",\n \"required\": true,\n \"schema\": {\n \"type\": \"string\"\n }\n },\n {\n \"in\": \"query\",\n \"name\": \"min_worker\",\n \"description\": \"Minimum number of worker processes.\",\n \"required\": false,\n \"schema\": {\n \"type\": \"integer\",\n \"default\": \"1\"\n }\n },\n {\n \"in\": \"query\",\n \"name\": \"max_worker\",\n \"description\": \"Maximum number of worker processes.\",\n \"required\": false,\n \"schema\": {\n \"type\": \"integer\",\n \"default\": \"1\"\n }\n },\n {\n \"in\": \"query\",\n \"name\": \"number_gpu\",\n \"description\": \"Number of GPU worker processes to create.\",\n \"required\": false,\n \"schema\": {\n \"type\": \"integer\",\n \"default\": \"0\"\n }\n },\n {\n \"in\": \"query\",\n \"name\": \"synchronous\",\n \"description\": \"Decides whether the call is synchronous or not, default: false.\",\n \"required\": false,\n \"schema\": {\n \"type\": \"boolean\",\n \"default\": \"false\"\n }\n },\n {\n \"in\": \"query\",\n \"name\": \"timeout\",\n \"description\": \"Waiting up to the specified wait time if necessary for a worker to complete all pending requests. Use 0 to terminate backend worker process immediately. Use -1 for wait infinitely.\",\n \"required\": false,\n \"schema\": {\n \"type\": \"integer\",\n \"default\": \"-1\"\n }\n }\n ],\n \"responses\": {\n \"200\": {\n \"description\": \"Model workers updated\",\n \"content\": {\n \"application/json\": {\n \"schema\": {\n \"type\": \"object\",\n \"required\": [\n \"status\"\n ],\n \"properties\": {\n \"status\": {\n \"type\": \"string\",\n \"description\": \"Error type.\"\n }\n }\n }\n }\n }\n },\n \"202\": {\n \"description\": \"Accepted\",\n \"content\": {\n \"application/json\": {\n \"schema\": {\n \"type\": \"object\",\n \"required\": [\n \"status\"\n ],\n \"properties\": {\n \"status\": {\n \"type\": \"string\",\n \"description\": \"Error type.\"\n }\n }\n }\n }\n }\n },\n \"210\": {\n \"description\": \"Partial Success\",\n \"content\": {\n \"application/json\": {\n \"schema\": {\n \"type\": \"object\",\n \"required\": [\n \"status\"\n ],\n \"properties\": {\n \"status\": {\n \"type\": \"string\",\n \"description\": \"Error type.\"\n }\n }\n }\n }\n }\n },\n \"400\": {\n \"description\": \"Bad request\",\n \"content\": {\n \"application/json\": {\n \"schema\": {\n \"type\": \"object\",\n \"required\": [\n \"code\",\n \"type\",\n \"message\"\n ],\n \"properties\": {\n \"code\": {\n \"type\": \"integer\",\n \"description\": \"Error code.\"\n },\n \"type\": {\n \"type\": \"string\",\n \"description\": \"Error type.\"\n },\n \"message\": {\n \"type\": \"string\",\n \"description\": \"Error message.\"\n }\n }\n }\n }\n }\n },\n \"404\": {\n \"description\": \"Model not found\",\n \"content\": {\n \"application/json\": {\n \"schema\": {\n \"type\": \"object\",\n \"required\": [\n \"code\",\n \"type\",\n \"message\"\n ],\n \"properties\": {\n \"code\": {\n \"type\": \"integer\",\n \"description\": \"Error code.\"\n },\n \"type\": {\n \"type\": \"string\",\n \"description\": \"Error type.\"\n },\n \"message\": {\n \"type\": \"string\",\n \"description\": \"Error message.\"\n }\n }\n }\n }\n }\n },\n \"500\": {\n \"description\": \"Internal Server Error\",\n \"content\": {\n \"application/json\": {\n \"schema\": {\n \"type\": \"object\",\n \"required\": [\n \"code\",\n \"type\",\n \"message\"\n ],\n \"properties\": {\n \"code\": {\n \"type\": \"integer\",\n \"description\": \"Error code.\"\n },\n \"type\": {\n \"type\": \"string\",\n \"description\": \"Error type.\"\n },\n \"message\": {\n \"type\": \"string\",\n \"description\": \"Error message.\"\n }\n }\n }\n }\n }\n }\n }\n },\n \"delete\": {\n \"description\": \"Unregister a model from Model Server. This is an asynchronous call by default. Caller can call listModels to confirm if all the works has be terminated.\",\n \"operationId\": \"unregisterModel\",\n \"parameters\": [\n {\n \"in\": \"path\",\n \"name\": \"model_name\",\n \"description\": \"Name of model to unregister.\",\n \"required\": true,\n \"schema\": {\n \"type\": \"string\"\n }\n },\n {\n \"in\": \"query\",\n \"name\": \"synchronous\",\n \"description\": \"Decides whether the call is synchronous or not, default: false.\",\n \"required\": false,\n \"schema\": {\n \"type\": \"boolean\",\n \"default\": \"false\"\n }\n },\n {\n \"in\": \"query\",\n \"name\": \"timeout\",\n \"description\": \"Waiting up to the specified wait time if necessary for a worker to complete all pending requests. Use 0 to terminate backend worker process immediately. Use -1 for wait infinitely.\",\n \"required\": false,\n \"schema\": {\n \"type\": \"integer\",\n \"default\": \"-1\"\n }\n }\n ],\n \"responses\": {\n \"200\": {\n \"description\": \"Model unregistered\",\n \"content\": {\n \"application/json\": {\n \"schema\": {\n \"type\": \"object\",\n \"required\": [\n \"status\"\n ],\n \"properties\": {\n \"status\": {\n \"type\": \"string\",\n \"description\": \"Error type.\"\n }\n }\n }\n }\n }\n },\n \"202\": {\n \"description\": \"Accepted\",\n \"content\": {\n \"application/json\": {\n \"schema\": {\n \"type\": \"object\",\n \"required\": [\n \"status\"\n ],\n \"properties\": {\n \"status\": {\n \"type\": \"string\",\n \"description\": \"Error type.\"\n }\n }\n }\n }\n }\n },\n \"404\": {\n \"description\": \"Model not found\",\n \"content\": {\n \"application/json\": {\n \"schema\": {\n \"type\": \"object\",\n \"required\": [\n \"code\",\n \"type\",\n \"message\"\n ],\n \"properties\": {\n \"code\": {\n \"type\": \"integer\",\n \"description\": \"Error code.\"\n },\n \"type\": {\n \"type\": \"string\",\n \"description\": \"Error type.\"\n },\n \"message\": {\n \"type\": \"string\",\n \"description\": \"Error message.\"\n }\n }\n }\n }\n }\n },\n \"408\": {\n \"description\": \"Request Timeout Error\",\n \"content\": {\n \"application/json\": {\n \"schema\": {\n \"type\": \"object\",\n \"required\": [\n \"code\",\n \"type\",\n \"message\"\n ],\n \"properties\": {\n \"code\": {\n \"type\": \"integer\",\n \"description\": \"Error code.\"\n },\n \"type\": {\n \"type\": \"string\",\n \"description\": \"Error type.\"\n },\n \"message\": {\n \"type\": \"string\",\n \"description\": \"Error message.\"\n }\n }\n }\n }\n }\n },\n \"500\": {\n \"description\": \"Internal Server Error\",\n \"content\": {\n \"application/json\": {\n \"schema\": {\n \"type\": \"object\",\n \"required\": [\n \"code\",\n \"type\",\n \"message\"\n ],\n \"properties\": {\n \"code\": {\n \"type\": \"integer\",\n \"description\": \"Error code.\"\n },\n \"type\": {\n \"type\": \"string\",\n \"description\": \"Error type.\"\n },\n \"message\": {\n \"type\": \"string\",\n \"description\": \"Error message.\"\n }\n }\n }\n }\n }\n }\n }\n }\n }\n }\n}\n" }, { "path": "frontend/settings.gradle", "content": "include 'server',\n\t'modelarchive',\n\t'cts'\n" }, { "path": "frontend/tools/conf/checkstyle.xml", "content": "\n\n\n \n \n\n \n \n\n \n \n \n \n \n \n\n \n \n\n \n \n\n \n \n \n \n \n \n \n \n\n \n \n\n \n \n \n \n \n\n \n \n\n \n \n\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n\n \n \n \n \n \n \n \n \n \n \n \n \n \n\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n\n \n \n\n \n \n\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n\n \n \n \n\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n\n \n \n\n \n \n \n \n\n \n \n\n \n\n\n" }, { "path": "frontend/tools/conf/findbugs-exclude.xml", "content": "\n\n \n \n \n \n \n \n \n \n \n \n\n\n" }, { "path": "frontend/tools/conf/pmd.xml", "content": "\n\n Java Rule in PMD\n\n \n\n \n \n \n \n \n \n\n \n\n \n \n \n\n \n\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n\n \n \n \n \n\n \n\n \n\n \n\n \n \n \n \n \n \n\n \n \n \n \n \n \n \n\n \n \n \n \n \n \n \n \n \n \n \n \n\n \n \n \n \n \n \n \n \n \n\n \n \n \n \n \n\n \n \n \n \n \n \n\n \n \n \n \n \n\n \n \n \n\n" }, { "path": "frontend/tools/conf/suppressions.xml", "content": "\n\n\n\n" }, { "path": "frontend/tools/gradle/check.gradle", "content": "apply plugin: 'findbugs'\nfindbugs {\n excludeFilter = file(\"${rootProject.projectDir}/tools/conf/findbugs-exclude.xml\")\n ignoreFailures = false\n findbugsTest.enabled = true\n}\ntasks.withType(FindBugs) {\n reports {\n xml.enabled false\n html.enabled true\n }\n}\n\napply plugin: 'pmd'\npmd {\n ignoreFailures = false\n pmdTest.enabled = false\n ruleSets = [] // workaround pmd gradle plugin bug\n ruleSetFiles = files(\"${rootProject.projectDir}/tools/conf/pmd.xml\")\n}\ntasks.withType(Pmd){\n reports{\n xml.enabled=true\n html.enabled=true\n }\n}\n\napply plugin: 'checkstyle'\ncheckstyle {\n toolVersion = '7.1.2'\n ignoreFailures = false\n checkstyleTest.enabled = true\n configProperties = [ \"checkstyle.suppressions.file\" : file(\"${rootProject.projectDir}/tools/conf/suppressions.xml\")]\n configFile = file(\"${rootProject.projectDir}/tools/conf/checkstyle.xml\")\n}\ncheckstyleMain {\n classpath += configurations.compile\n}\ntasks.withType(Checkstyle) {\n reports {\n xml.enabled false\n html.enabled true\n }\n}\n\napply plugin: \"jacoco\"\njacoco {\n toolVersion = \"0.8.1\"\n}\njacocoTestReport {\n group = \"Reporting\"\n reports {\n xml.enabled true\n csv.enabled false\n }\n}\n\ncheck.dependsOn jacocoTestReport\ncheck.dependsOn jacocoTestCoverageVerification\n" }, { "path": "frontend/tools/gradle/formatter.gradle", "content": "buildscript {\n repositories {\n maven {\n url \"https://plugins.gradle.org/m2/\"\n }\n }\n dependencies {\n classpath 'com.google.googlejavaformat:google-java-format:1.6'\n }\n}\n\napply plugin: FormatterPlugin\n\ncheck.dependsOn verifyJava\n\nimport com.google.googlejavaformat.java.Formatter\nimport com.google.googlejavaformat.java.ImportOrderer\nimport com.google.googlejavaformat.java.JavaFormatterOptions\nimport com.google.googlejavaformat.java.Main\nimport com.google.googlejavaformat.java.RemoveUnusedImports\n\nclass FormatterPlugin implements Plugin {\n void apply(Project project) {\n project.task('formatJava') {\n doLast {\n Main formatter = new Main(new PrintWriter(System.out, true), new PrintWriter(System.err, true), System.in)\n Project rootProject = project.getRootProject()\n for (item in project.sourceSets) {\n for (File file : item.getAllSource()) {\n if (!file.getName().endsWith(\".java\")) {\n continue\n }\n if (formatter.format(\"-a\", \"-i\", file.getAbsolutePath()) != 0) {\n throw new GradleException(\"Format java failed: \" + file.getAbsolutePath())\n }\n }\n }\n }\n }\n\n project.task('verifyJava') {\n doLast {\n def options = JavaFormatterOptions.builder().style(JavaFormatterOptions.Style.AOSP).build()\n Formatter formatter = new Formatter(options)\n Project rootProject = project.getRootProject()\n for (item in project.sourceSets) {\n for (File file : item.getAllSource()) {\n if (!file.getName().endsWith(\".java\")) {\n continue\n }\n\n String src = file.text\n String formatted = formatter.formatSource(src)\n formatted = RemoveUnusedImports.removeUnusedImports(formatted, RemoveUnusedImports.JavadocOnlyImports.KEEP)\n formatted = ImportOrderer.reorderImports(formatted);\n if (!src.equals(formatted)) {\n throw new GradleException(\"File not formatted: \" + file.getAbsolutePath())\n }\n }\n }\n }\n }\n }\n}\n" }, { "path": "frontend/tools/gradle/launcher.gradle", "content": "apply plugin: LauncherPlugin\n\nclean.dependsOn killServer\n\nimport org.gradle.internal.jvm.Jvm\n\nclass LauncherPlugin implements Plugin {\n void apply(Project project) {\n project.task('startServer') {\n dependsOn project.jar\n doLast {\n def pidFile = getPidFile()\n if (pidFile.exists()) {\n throw new GradleException(\"Server already running!\")\n }\n\n def list = []\n list.addAll(project.configurations.runtime.getFiles())\n list.add(project.jar.outputs.files.singleFile)\n String cp = CollectionUtils.join(File.pathSeparator, list)\n String jvmPath = Jvm.current().getJavaExecutable()\n\n def cmd = [jvmPath, \"-agentlib:jdwp=transport=dt_socket,address=0.0.0.0:4000,server=y,suspend=n\",\n \"-DmmsConfigFile=${project.projectDir}/src/test/resources/config.properties\",\n \"-DLOG_LOCATION=${project.buildDir}/logs\",\n \"-DMETRICS_LOCATION=${project.buildDir}/logs\",\n \"-cp\", cp, \"com.amazonaws.ml.mms.ModelServer\"] as String[]\n\n def builder = new ProcessBuilder(cmd)\n builder.redirectErrorStream(true)\n builder.directory(project.projectDir)\n Process process = builder.start()\n\n ReaderThread rt = new ReaderThread(process.getInputStream())\n rt.start()\n new ReaderThread(process.getErrorStream()).start()\n\n try {\n while (!rt.done) {\n try {\n process.exitValue();\n throw new GradleException(\"MMS stop unexpectedly.\")\n } catch(IllegalThreadStateException ex) {\n Thread.sleep(500);\n }\n }\n\n def pidField = process.class.getDeclaredField('pid')\n pidField.accessible = true\n\n pidFile << pidField.getInt(process)\n\n logger.quiet \"MMS service started.\"\n } catch (IllegalThreadStateException ignored) {\n }\n }\n }\n\n project.task('killServer') {\n doLast {\n def pidFile = getPidFile()\n if(!pidFile.exists()) {\n logger.quiet \"No server running!\"\n return\n }\n\n def pid = pidFile.text\n def process = \"kill $pid\".execute()\n\n try {\n process.waitFor()\n } finally {\n pidFile.delete()\n }\n }\n }\n\n project.task('restartServer') {\n dependsOn project.killServer\n dependsOn project.startServer\n }\n }\n\n private File getPidFile() {\n return new File(\"build/server.pid\")\n }\n}\n\nclass ReaderThread extends Thread {\n\n private InputStream is\n private boolean done;\n\n public ReaderThread(InputStream is) {\n this.is = is\n }\n\n public void run() {\n long begin = System.currentTimeMillis()\n def line\n def reader = new BufferedReader(new InputStreamReader(is))\n while ((line = reader.readLine()) != null) {\n if (!done) {\n done = line.matches(\"Model server started.*\")\n println line\n }\n }\n }\n}\n" }, { "path": "mms/.gitignore", "content": "frontend\n" }, { "path": "mms/__init__.py", "content": "# Copyright 2017 Amazon.com, Inc. or its affiliates. All Rights Reserved.\n# Licensed under the Apache License, Version 2.0 (the \"License\").\n# You may not use this file except in compliance with the License.\n# A copy of the License is located at\n# http://www.apache.org/licenses/LICENSE-2.0\n# or in the \"license\" file accompanying this file. This file is distributed\n# on an \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either\n# express or implied. See the License for the specific language governing\n# permissions and limitations under the License.\n\n\"\"\"\nThis module does the following:\na. Starts model-server.\nb. Creates end-points based on the configured models.\nc. Exposes standard \"ping\" and \"api-description\" endpoints.\nd. Waits for servicing inference requests.\n\"\"\"\nfrom . import version\n\n__version__ = version.__version__\n" }, { "path": "mms/arg_parser.py", "content": "# Copyright 2017 Amazon.com, Inc. or its affiliates. All Rights Reserved.\n# Licensed under the Apache License, Version 2.0 (the \"License\").\n# You may not use this file except in compliance with the License.\n# A copy of the License is located at\n# http://www.apache.org/licenses/LICENSE-2.0\n# or in the \"license\" file accompanying this file. This file is distributed\n# on an \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either\n# express or implied. See the License for the specific language governing\n# permissions and limitations under the License.\n\n\"\"\"\nThis module parses the arguments given through the multi-model-server command-line. This is used by model-server\nat runtime.\n\"\"\"\n\nimport argparse\n\n\n# noinspection PyTypeChecker\nclass ArgParser(object):\n \"\"\"\n Argument parser for multi-model-server and multi-model-export commands\n More detailed example is available at https://github.com/awslabs/multi-model-server/blob/master/README.md\n \"\"\"\n @staticmethod\n def mms_parser():\n \"\"\"\n Argument parser for multi-model-server start service\n \"\"\"\n parser = argparse.ArgumentParser(prog='multi-model-server', description='Multi Model Server')\n\n sub_parse = parser.add_mutually_exclusive_group(required=False)\n sub_parse.add_argument('--start', action='store_true', help='Start the model-server')\n sub_parse.add_argument('--stop', action='store_true', help='Stop the model-server')\n\n parser.add_argument('--mms-config',\n dest='mms_config',\n help='Configuration file for model server')\n parser.add_argument('--model-store',\n dest='model_store',\n help='Model store location where models can be loaded')\n parser.add_argument('--models',\n metavar='MODEL_PATH1 MODEL_NAME=MODEL_PATH2...',\n nargs='+',\n help='Models to be loaded using [model_name=]model_location format. '\n 'Location can be a HTTP URL, a model archive file or directory '\n 'contains model archive files in MODEL_STORE.')\n parser.add_argument('--log-config',\n dest='log_config',\n help='Log4j configuration file for model server')\n parser.add_argument('--foreground',\n help='Run the model server in foreground. If this option is disabled, the model server'\n ' will run in the background.',\n action='store_true')\n\n return parser\n\n @staticmethod\n def str2bool(v):\n if v.lower() in ('yes', 'true', 'y', '1'):\n return True\n if v.lower() in ('no', 'false', 'n', '0'):\n return False\n raise argparse.ArgumentTypeError('Boolean value expected {}'.format(v))\n\n @staticmethod\n def model_service_worker_args():\n \"\"\"\n ArgParser for backend worker. Takes the socket name and socket type.\n :return:\n \"\"\"\n parser = argparse.ArgumentParser(prog='model-server-worker', description='Model Server Worker')\n parser.add_argument('--sock-type',\n required=True,\n dest=\"sock_type\",\n type=str,\n choices=[\"unix\", \"tcp\"],\n help='Socket type the model service worker would use. The options are\\n'\n 'unix: The model worker expects to unix domain-socket\\n'\n 'tcp: The model worker expects a host-name and port-number')\n\n parser.add_argument('--sock-name',\n required=False,\n dest=\"sock_name\",\n type=str,\n help='If \\'sock-type\\' is \\'unix\\', sock-name is expected to be a string. '\n 'Eg: --sock-name \\\"test_sock\\\"')\n\n parser.add_argument('--host',\n type=str,\n help='If \\'sock-type\\' is \\'tcp\\' this is expected to have a host IP address')\n\n parser.add_argument('--port',\n type=str,\n help='If \\'sock-type\\' is \\'tcp\\' this is expected to have the host port to bind on')\n\n parser.add_argument('--handler',\n type=str,\n help='Entry point to the Model Server')\n\n parser.add_argument('--model-path',\n type=str,\n help='Path to the actual model location')\n\n parser.add_argument('--model-name',\n type=str,\n help='Name of the model')\n\n parser.add_argument('--preload-model',\n dest=\"preload_model\",\n required=True,\n type=ArgParser.str2bool,\n help='Determines if initialization should occur before spawning/forking child process')\n\n parser.add_argument('--tmp-dir',\n dest=\"tmp_dir\",\n required=True,\n type=str,\n help='Location of temporaty file descriptors')\n\n return parser\n\n @staticmethod\n def extract_args(args=None):\n parser = ArgParser.mms_parser()\n return parser.parse_args(args) if args else parser.parse_args()\n" }, { "path": "mms/configs/sagemaker_config.properties", "content": "vmargs=-XX:-UseContainerSupport\nmodel_store=$$SAGEMAKER_MODEL_STORE$$\ninference_address=http://0.0.0.0:$$SAGEMAKER_BIND_TO_PORT$$\nmanagement_address=http://0.0.0.0:$$SAGEMAKER_BIND_TO_PORT$$\ndefault_response_timeout=$$SAGEMAKER_RESPONSE_TIMEOUT$$\nunregister_model_timeout=$$SAGEMAKER_RESPONSE_TIMEOUT$$\ndefault_workers_per_model=$$SAGEMAKER_NUM_MODEL_WORKERS$$\ndefault_service_handler=$$SAGEMAKER_HANDLER$$\nasync_logging=true\nmax_response_size=$$SAGEMAKER_MAX_RESPONSE_SIZE$$\nmax_request_size=$$SAGEMAKER_MAX_REQUEST_SIZE$$\ndecode_input_request=false\n" }, { "path": "mms/context.py", "content": "# Copyright 2018 Amazon.com, Inc. or its affiliates. All Rights Reserved.\n# Licensed under the Apache License, Version 2.0 (the \"License\").\n# You may not use this file except in compliance with the License.\n# A copy of the License is located at\n# http://www.apache.org/licenses/LICENSE-2.0\n# or in the \"license\" file accompanying this file. This file is distributed\n# on an \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either\n# express or implied. See the License for the specific language governing\n# permissions and limitations under the License.\n\n\"\"\"\nContext object of incoming request\n\"\"\"\n\n\nclass Context(object):\n \"\"\"\n Context stores model relevant worker information\n Some fixed during load times and some\n \"\"\"\n\n def __init__(self, model_name, model_dir, manifest, batch_size, gpu, mms_version):\n self.model_name = model_name\n self.manifest = manifest\n self._system_properties = {\n \"model_dir\": model_dir,\n \"gpu_id\": gpu,\n \"batch_size\": batch_size,\n \"server_name\": \"MMS\",\n \"server_version\": mms_version\n }\n self.request_ids = None\n self.request_processor = None\n self._metrics = None\n\n @property\n def system_properties(self):\n return self._system_properties\n\n @property\n def request_processor(self):\n return self._request_processor\n\n @request_processor.setter\n def request_processor(self, request_processor):\n self._request_processor = request_processor\n\n @property\n def metrics(self):\n return self._metrics\n\n @metrics.setter\n def metrics(self, metrics):\n self._metrics = metrics\n\n def get_request_id(self, idx=0):\n return self.request_ids.get(idx)\n\n def get_request_header(self, idx, key):\n return self._request_processor[idx].get_request_property(key)\n\n def get_all_request_header(self, idx):\n return self._request_processor[idx].get_request_properties()\n\n def set_response_content_type(self, idx, value):\n self.set_response_header(idx, 'content-type', value)\n\n def get_response_content_type(self, idx):\n return self.get_response_headers(idx).get('content-type')\n\n def get_response_status(self, idx):\n return self._request_processor[idx].get_response_status_code(), \\\n self._request_processor[idx].get_response_status_phrase()\n\n def set_response_status(self, code=200, phrase=\"\", idx=0):\n \"\"\"\n Set the status code of individual requests\n :param phrase:\n :param idx: The index data in the list(data) that is sent to the handle() method\n :param code:\n :return:\n \"\"\"\n if self._request_processor is not None and self._request_processor[idx] is not None:\n self._request_processor[idx].report_status(code,\n reason_phrase=phrase)\n\n def set_all_response_status(self, code=200, phrase=\"\"):\n \"\"\"\n Set the status code of individual requests\n :param phrase:\n :param code:\n :return:\n \"\"\"\n for idx, _ in enumerate(self._request_processor):\n self._request_processor[idx].report_status(code, reason_phrase=phrase)\n\n def get_response_headers(self, idx):\n return self._request_processor[idx].get_response_headers()\n\n def set_response_header(self, idx, key, value):\n self._request_processor[idx].add_response_property(key, value)\n\n # TODO: Should we add \"add_header()\" interface, to have multiple values for a single header. EG: Accept headers.\n\n def __eq__(self, other):\n return isinstance(other, Context) and self.__dict__ == other.__dict__\n\n\nclass RequestProcessor(object):\n \"\"\"\n Request processor\n \"\"\"\n\n def __init__(self, request_header):\n self._status_code = 200\n self._reason_phrase = None\n self._response_header = {}\n self._request_header = request_header\n\n def get_request_property(self, key):\n return self._request_header.get(key)\n\n def report_status(self, code, reason_phrase=None):\n self._status_code = code\n self._reason_phrase = reason_phrase\n\n def get_response_status_code(self):\n return self._status_code\n\n def get_response_status_phrase(self):\n return self._reason_phrase\n\n def add_response_property(self, key, value):\n self._response_header[key] = value\n\n def get_response_headers(self):\n return self._response_header\n\n def get_response_header(self, key):\n return self._response_header.get(key)\n\n def get_request_properties(self):\n return self._request_header\n" }, { "path": "mms/export_model.py", "content": "# Copyright 2017 Amazon.com, Inc. or its affiliates. All Rights Reserved.\n# Licensed under the Apache License, Version 2.0 (the \"License\").\n# You may not use this file except in compliance with the License.\n# A copy of the License is located at\n# http://www.apache.org/licenses/LICENSE-2.0\n# or in the \"license\" file accompanying this file. This file is distributed\n# on an \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either\n# express or implied. See the License for the specific language governing\n# permissions and limitations under the License.\n\n\"\"\"\nThis command line interface is no longer used. Please refer to model-archiver tool for the new CLI for exporting models.\n\"\"\"\n\n\ndef main():\n print('\\033[93m' # Red Color start\n + \"multi-model-export is no longer supported.\\n\"\n \"Please use model-archiver to create 1.0 model archive.\\n\"\n \"For more detail, see: https://pypi.org/project/model-archiver\"\n + '\\033[0m') # Red Color end\n\n\nif __name__ == '__main__':\n main()\n" }, { "path": "mms/metrics/__init__.py", "content": "# Copyright 2018 Amazon.com, Inc. or its affiliates. All Rights Reserved.\n# Licensed under the Apache License, Version 2.0 (the \"License\").\n# You may not use this file except in compliance with the License.\n# A copy of the License is located at\n# http://www.apache.org/licenses/LICENSE-2.0\n# or in the \"license\" file accompanying this file. This file is distributed\n# on an \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either\n# express or implied. See the License for the specific language governing\n# permissions and limitations under the License.\n\n\"\"\"\nThis is a folder for all python worker metrics.\n\"\"\"\nfrom . import dimension\nfrom . import metric\nfrom . import metric_encoder\nfrom . import metrics_store\nfrom . import system_metrics\nfrom . import unit\n" }, { "path": "mms/metrics/dimension.py", "content": "# Copyright 2018 Amazon.com, Inc. or its affiliates. All Rights Reserved.\n# Licensed under the Apache License, Version 2.0 (the \"License\").\n# You may not use this file except in compliance with the License.\n# A copy of the License is located at\n# http://www.apache.org/licenses/LICENSE-2.0\n# or in the \"license\" file accompanying this file. This file is distributed\n# on an \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either\n# express or implied. See the License for the specific language governing\n# permissions and limitations under the License.\n\n\"\"\"\nDimension class for model server metrics\n\"\"\"\n\n\nclass Dimension(object):\n \"\"\"\n Dimension class defining key value pair\n \"\"\"\n def __init__(self, name, value):\n \"\"\"\n Constructor for Dimension class\n\n Parameters\n ----------\n name: str\n NAme of dimension\n value : str\n Unique Value of dimension\n \"\"\"\n self.name = name\n self.value = value\n\n def __str__(self):\n \"\"\"\n Return a string value\n :return:\n \"\"\"\n return \"{}:{}\".format(self.name, self.value)\n\n def to_dict(self):\n \"\"\"\n return an dictionary\n \"\"\"\n return {'Name': self.name, 'Value': self.value}\n" }, { "path": "mms/metrics/metric.py", "content": "# Copyright 2018 Amazon.com, Inc. or its affiliates. All Rights Reserved.\n# Licensed under the Apache License, Version 2.0 (the \"License\").\n# You may not use this file except in compliance with the License.\n# A copy of the License is located at\n# http://www.apache.org/licenses/LICENSE-2.0\n# or in the \"license\" file accompanying this file. This file is distributed\n# on an \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either\n# express or implied. See the License for the specific language governing\n# permissions and limitations under the License.\n\n\"\"\"\nMetric class for model server\n\"\"\"\nimport time\nimport socket\nfrom collections import OrderedDict\n\nfrom builtins import str\n\nfrom mms.metrics.unit import Units\n\nMetricUnit = Units()\n\n\nclass Metric(object):\n \"\"\"\n Class for generating metrics and printing it to stdout of the worker\n \"\"\"\n\n def __init__(self, name, value,\n unit, dimensions, request_id=None, metric_method=None):\n \"\"\"\n Constructor for Metric class\n\n Metric class will spawn a thread and report collected metrics to stdout of worker\n\n Parameters\n ----------\n name: str\n Name of metric\n value : int, float\n Can be integer or float\n unit: str\n unit can be one of ms, percent, count, MB, GB or a generic string\n dimensions: list\n list of dimension objects\n request_id: str\n req_id of metric\n metric_method: str\n useful for defining different operations, optional\n\n \"\"\"\n self.name = name\n self.unit = unit\n if unit in list(MetricUnit.units.keys()):\n self.unit = MetricUnit.units[unit]\n self.metric_method = metric_method\n self.value = value\n self.dimensions = dimensions\n self.request_id = request_id\n\n def update(self, value):\n \"\"\"\n Update function for Metric class\n\n Parameters\n ----------\n value : int, float\n metric to be updated\n \"\"\"\n\n if self.metric_method == 'counter':\n self.value += value\n else:\n self.value = value\n\n def __str__(self):\n dims = \",\".join([str(d) for d in self.dimensions])\n if self.request_id:\n return \"{}.{}:{}|#{}|#hostname:{},{},{}\".format(\n self.name, self.unit, self.value, dims, socket.gethostname(),\n int(time.time()), self.request_id)\n\n return \"{}.{}:{}|#{}|#hostname:{},{}\".format(\n self.name, self.unit, self.value, dims, socket.gethostname(), int(time.time()))\n\n def to_dict(self):\n \"\"\"\n return an Ordered Dictionary\n \"\"\"\n return OrderedDict({'MetricName': self.name, 'Value': self.value, 'Unit': self.unit,\n 'Dimensions': self.dimensions,\n 'Timestamp': int(time.time()),\n 'HostName': socket.gethostname(),\n 'RequestId': self.request_id})\n" }, { "path": "mms/metrics/metric_collector.py", "content": "# Copyright 2018 Amazon.com, Inc. or its affiliates. All Rights Reserved.\n# Licensed under the Apache License, Version 2.0 (the \"License\").\n# You may not use this file except in compliance with the License.\n# A copy of the License is located at\n# http://www.apache.org/licenses/LICENSE-2.0\n# or in the \"license\" file accompanying this file. This file is distributed\n# on an \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either\n# express or implied. See the License for the specific language governing\n# permissions and limitations under the License.\n\n\"\"\"\nSingle start point for system metrics and process metrics script\n\n\"\"\"\nimport logging\nimport sys\n\nfrom mms.metrics import system_metrics\nfrom mms.metrics.process_memory_metric import check_process_mem_usage\n\nif __name__ == '__main__':\n logging.basicConfig(stream=sys.stdout, format=\"%(message)s\", level=logging.INFO)\n\n system_metrics.collect_all(sys.modules['mms.metrics.system_metrics'])\n\n check_process_mem_usage(sys.stdin)\n" }, { "path": "mms/metrics/metric_encoder.py", "content": "# Copyright 2018 Amazon.com, Inc. or its affiliates. All Rights Reserved.\n# Licensed under the Apache License, Version 2.0 (the \"License\").\n# You may not use this file except in compliance with the License.\n# A copy of the License is located at\n# http://www.apache.org/licenses/LICENSE-2.0\n# or in the \"license\" file accompanying this file. This file is distributed\n# on an \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either\n# express or implied. See the License for the specific language governing\n# permissions and limitations under the License.\n\n\"\"\"\nMetric Encoder class for json dumps\n\"\"\"\n\nimport json\nfrom json import JSONEncoder\n\nfrom mms.metrics.dimension import Dimension\nfrom mms.metrics.metric import Metric\n\n\nclass MetricEncoder(JSONEncoder):\n \"\"\"\n Encoder class for json encoding Metric Object\n \"\"\"\n def default(self, obj): # pylint: disable=arguments-differ, method-hidden\n \"\"\"\n Override only when object is of type Metric\n \"\"\"\n if isinstance(obj, (Metric, Dimension)):\n return obj.to_dict()\n return json.JSONEncoder.default(self, obj)\n" }, { "path": "mms/metrics/metrics_store.py", "content": "# Copyright 2018 Amazon.com, Inc. or its affiliates. All Rights Reserved.\n# Licensed under the Apache License, Version 2.0 (the \"License\").\n# You may not use this file except in compliance with the License.\n# A copy of the License is located at\n# http://www.apache.org/licenses/LICENSE-2.0\n# or in the \"license\" file accompanying this file. This file is distributed\n# on an \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either\n# express or implied. See the License for the specific language governing\n# permissions and limitations under the License.\n\n\"\"\"\nMetrics collection module\n\"\"\"\nfrom builtins import str\n\nfrom mms.metrics.dimension import Dimension\nfrom mms.metrics.metric import Metric\n\n\nclass MetricsStore(object):\n \"\"\"\n Class for creating, modifying different metrics. And keep them in a dictionary\n \"\"\"\n\n def __init__(self, request_ids, model_name):\n \"\"\"\n Initialize metrics map,model name and request map\n \"\"\"\n self.store = list()\n self.request_ids = request_ids\n self.model_name = model_name\n self.cache = {}\n\n def _add_or_update(self, name, value, req_id, unit, metrics_method=None, dimensions=None):\n \"\"\"\n Add a metric key value pair\n\n Parameters\n ----------\n name : str\n metric name\n value: int, float\n value of metric\n req_id: str\n request id\n unit: str\n unit of metric\n value: int, float , str\n value of metric\n metrics_method: str, optional\n indicates type of metric operation if it is defined\n \"\"\"\n # IF req_id is none error Metric\n if dimensions is None:\n dimensions = list()\n elif not isinstance(dimensions, list):\n raise ValueError(\"Please provide a list of dimensions\")\n if req_id is None:\n dimensions.append(Dimension(\"Level\", \"Error\"))\n else:\n dimensions.append(Dimension(\"ModelName\", self.model_name))\n dimensions.append(Dimension(\"Level\", \"Model\"))\n\n # Cache the metric with an unique key for update\n dim_str = [name, unit, str(req_id)] + [str(d) for d in dimensions]\n dim_str = '-'.join(dim_str)\n if dim_str not in self.cache:\n metric = Metric(name, value, unit, dimensions, req_id, metrics_method)\n self.store.append(metric)\n self.cache[dim_str] = metric\n else:\n self.cache[dim_str].update(value)\n\n def _get_req(self, idx):\n \"\"\"\n Provide the request id dimension\n\n Parameters\n ----------\n\n idx : int\n request_id index in batch\n \"\"\"\n # check if request id for the metric is given, if so use it else have a list of all.\n req_id = self.request_ids\n if isinstance(req_id, dict):\n req_id = ','.join(self.request_ids.values())\n if idx is not None and self.request_ids is not None and idx in self.request_ids:\n req_id = self.request_ids[idx]\n return req_id\n\n def add_counter(self, name, value, idx=None, dimensions=None):\n \"\"\"\n Add a counter metric or increment an existing counter metric\n\n Parameters\n ----------\n name : str\n metric name\n value: int\n value of metric\n idx: int\n request_id index in batch\n dimensions: list\n list of dimensions for the metric\n \"\"\"\n unit = 'count'\n req_id = self._get_req(idx)\n self._add_or_update(name, value, req_id, unit, 'counter', dimensions)\n\n def add_time(self, name, value, idx=None, unit='ms', dimensions=None):\n \"\"\"\n Add a time based metric like latency, default unit is 'ms'\n\n Parameters\n ----------\n name : str\n metric name\n value: int\n value of metric\n idx: int\n request_id index in batch\n unit: str\n unit of metric, default here is ms, s is also accepted\n dimensions: list\n list of dimensions for the metric\n \"\"\"\n if unit not in ['ms', 's']:\n raise ValueError(\"the unit for a timed metric should be one of ['ms', 's']\")\n req_id = self._get_req(idx)\n self._add_or_update(name, value, req_id, unit, dimensions)\n\n def add_size(self, name, value, idx=None, unit='MB', dimensions=None):\n \"\"\"\n Add a size based metric\n\n Parameters\n ----------\n name : str\n metric name\n value: int, float\n value of metric\n idx: int\n request_id index in batch\n unit: str\n unit of metric, default here is 'MB', 'kB', 'GB' also supported\n dimensions: list\n list of dimensions for the metric\n \"\"\"\n if unit not in ['MB', 'kB', 'GB', 'B']:\n raise ValueError(\"The unit for size based metric is one of ['MB','kB', 'GB', 'B']\")\n req_id = self._get_req(idx)\n self._add_or_update(name, value, req_id, unit, dimensions)\n\n def add_percent(self, name, value, idx=None, dimensions=None):\n \"\"\"\n Add a percentage based metric\n\n Parameters\n ----------\n name : str\n metric name\n value: int, float\n value of metric\n idx: int\n request_id index in batch\n dimensions: list\n list of dimensions for the metric\n \"\"\"\n unit = 'percent'\n req_id = self._get_req(idx)\n self._add_or_update(name, value, req_id, unit, dimensions)\n\n def add_error(self, name, value, dimensions=None):\n \"\"\"\n Add a Error Metric\n Parameters\n ----------\n name : str\n metric name\n value: str\n value of metric, in this case a str\n dimensions: list\n list of dimensions for the metric\n \"\"\"\n unit = ''\n\n # noinspection PyTypeChecker\n self._add_or_update(name, value, None, unit, dimensions)\n\n def add_metric(self, name, value, idx=None, unit=None, dimensions=None):\n \"\"\"\n Add a metric which is generic with custom metrics\n\n Parameters\n ----------\n name : str\n metric name\n value: int, float\n value of metric\n idx: int\n request_id index in batch\n unit: str\n unit of metric\n dimensions: list\n list of dimensions for the metric\n \"\"\"\n req_id = self._get_req(idx)\n self._add_or_update(name, value, req_id, unit, dimensions)\n" }, { "path": "mms/metrics/process_memory_metric.py", "content": "# Copyright 2018 Amazon.com, Inc. or its affiliates. All Rights Reserved.\n# Licensed under the Apache License, Version 2.0 (the \"License\").\n# You may not use this file except in compliance with the License.\n# A copy of the License is located at\n# http://www.apache.org/licenses/LICENSE-2.0\n# or in the \"license\" file accompanying this file. This file is distributed\n# on an \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either\n# express or implied. See the License for the specific language governing\n# permissions and limitations under the License.\n\n\"\"\"\nCollect process memory usage metrics here\nPass a json, collection of pids and gpuID\n\"\"\"\n\nimport logging\n\nimport psutil\n\ndef get_cpu_usage(pid):\n \"\"\"\n use psutil for cpu memory\n :param pid: str\n :return: int\n \"\"\"\n try:\n process = psutil.Process(int(pid))\n except psutil.Error:\n logging.error(\"Failed to get process for pid: %s\", pid, exc_info=True)\n return 0\n\n mem_utilization = process.memory_info()[0]\n if mem_utilization == 0:\n logging.error(\"Failed to get memory utilization for pid: %s\", pid, exc_info=True)\n return 0\n return mem_utilization\n\n\ndef check_process_mem_usage(stdin):\n \"\"\"\n\n Return\n ------\n mem_utilization: float\n \"\"\"\n process_list = stdin.readline().strip().split(\",\")\n for process in process_list:\n if not process:\n continue\n mem_utilization = get_cpu_usage(process)\n if mem_utilization != 0:\n logging.info(\"%s:%d\", process, mem_utilization)\n" }, { "path": "mms/metrics/system_metrics.py", "content": "# Copyright 2018 Amazon.com, Inc. or its affiliates. All Rights Reserved.\n# Licensed under the Apache License, Version 2.0 (the \"License\").\n# You may not use this file except in compliance with the License.\n# A copy of the License is located at\n# http://www.apache.org/licenses/LICENSE-2.0\n# or in the \"license\" file accompanying this file. This file is distributed\n# on an \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either\n# express or implied. See the License for the specific language governing\n# permissions and limitations under the License.\n\"\"\"\nModule to collect system metrics for front-end\n\"\"\"\nimport logging\nimport types\nfrom builtins import str\n\nimport psutil\n\nfrom mms.metrics.dimension import Dimension\nfrom mms.metrics.metric import Metric\n\nsystem_metrics = []\ndimension = [Dimension('Level', 'Host')]\n\n\ndef cpu_utilization():\n data = psutil.cpu_percent()\n system_metrics.append(Metric('CPUUtilization', data, 'percent', dimension))\n\n\ndef memory_used():\n data = psutil.virtual_memory().used / (1024 * 1024) # in MB\n system_metrics.append(Metric('MemoryUsed', data, 'MB', dimension))\n\n\ndef memory_available():\n data = psutil.virtual_memory().available / (1024 * 1024) # in MB\n system_metrics.append(Metric('MemoryAvailable', data, 'MB', dimension))\n\n\ndef memory_utilization():\n data = psutil.virtual_memory().percent\n system_metrics.append(Metric('MemoryUtilization', data, 'percent', dimension))\n\n\ndef disk_used():\n data = psutil.disk_usage('/').used / (1024 * 1024 * 1024) # in GB\n system_metrics.append(Metric('DiskUsage', data, 'GB', dimension))\n\n\ndef disk_utilization():\n data = psutil.disk_usage('/').percent\n system_metrics.append(Metric('DiskUtilization', data, 'percent', dimension))\n\n\ndef disk_available():\n data = psutil.disk_usage('/').free / (1024 * 1024 * 1024) # in GB\n system_metrics.append(Metric('DiskAvailable', data, 'GB', dimension))\n\n\ndef collect_all(mod):\n \"\"\"\n Collect all system metrics.\n\n :param mod:\n :return:\n \"\"\"\n members = dir(mod)\n for i in members:\n value = getattr(mod, i)\n if isinstance(value, types.FunctionType) and value.__name__ not in ('collect_all', 'log_msg'):\n value()\n\n for met in system_metrics:\n logging.info(str(met))\n\n logging.info(\"\")\n" }, { "path": "mms/metrics/unit.py", "content": "# Copyright 2018 Amazon.com, Inc. or its affiliates. All Rights Reserved.\n# Licensed under the Apache License, Version 2.0 (the \"License\").\n# You may not use this file except in compliance with the License.\n# A copy of the License is located at\n# http://www.apache.org/licenses/LICENSE-2.0\n# or in the \"license\" file accompanying this file. This file is distributed\n# on an \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either\n# express or implied. See the License for the specific language governing\n# permissions and limitations under the License.\n\"\"\"\nModule to define Unit mappings\n\"\"\"\n\n\nclass Units(object):\n \"\"\"\n Define a unit of elements\n \"\"\"\n\n def __init__(self):\n self.units = {\n 'ms': \"Milliseconds\",\n 's': 'Seconds',\n 'percent': 'Percent',\n 'count': 'Count',\n 'MB': 'Megabytes',\n 'GB': 'Gigabytes',\n 'kB': 'Kilobytes',\n 'B': 'Bytes',\n None: 'unit',\n }\n" }, { "path": "mms/model_loader.py", "content": "# Copyright 2018 Amazon.com, Inc. or its affiliates. All Rights Reserved.\n# Licensed under the Apache License, Version 2.0 (the \"License\").\n# You may not use this file except in compliance with the License.\n# A copy of the License is located at\n# http://www.apache.org/licenses/LICENSE-2.0\n# or in the \"license\" file accompanying this file. This file is distributed\n# on an \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either\n# express or implied. See the License for the specific language governing\n# permissions and limitations under the License.\n\n\"\"\"\nModel loader.\n\"\"\"\nimport importlib\nimport inspect\nimport json\nimport logging\nimport os\nimport sys\nimport uuid\nfrom abc import ABCMeta, abstractmethod\n\nfrom builtins import str\n\nfrom mms.metrics.metrics_store import MetricsStore\nfrom mms.service import Service\n\n\nclass ModelLoaderFactory(object):\n \"\"\"\n ModelLoaderFactory\n \"\"\"\n\n @staticmethod\n def get_model_loader(model_dir):\n manifest_file = os.path.join(model_dir, \"MAR-INF/MANIFEST.json\")\n if os.path.exists(manifest_file):\n return MmsModelLoader()\n elif os.path.exists(os.path.join(model_dir, \"MANIFEST.json\")):\n return LegacyModelLoader()\n else:\n return MmsModelLoader()\n\n\nclass ModelLoader(object):\n \"\"\"\n Base Model Loader class.\n \"\"\"\n __metaclass__ = ABCMeta\n\n @abstractmethod\n def load(self, model_name, model_dir, handler, gpu_id, batch_size):\n \"\"\"\n Load model from file.\n\n :param model_name:\n :param model_dir:\n :param handler:\n :param gpu_id:\n :param batch_size:\n :return: Model\n \"\"\"\n pass # pylint: disable=unnecessary-pass\n\n @staticmethod\n def list_model_services(module, parent_class=None):\n \"\"\"\n Parse user defined module to get all model service classes in it.\n\n :param module:\n :param parent_class:\n :return: List of model service class definitions\n \"\"\"\n\n # Parsing the module to get all defined classes\n classes = [cls[1] for cls in inspect.getmembers(module, lambda member: inspect.isclass(member) and\n member.__module__ == module.__name__)]\n # filter classes that is subclass of parent_class\n if parent_class is not None:\n return [c for c in classes if issubclass(c, parent_class)]\n\n return classes\n\n\nclass MmsModelLoader(ModelLoader):\n \"\"\"\n MMS 1.0 Model Loader\n \"\"\"\n\n def load(self, model_name, model_dir, handler, gpu_id, batch_size):\n \"\"\"\n Load MMS 1.0 model from file.\n\n :param model_name:\n :param model_dir:\n :param handler:\n :param gpu_id:\n :param batch_size:\n :return:\n \"\"\"\n logging.debug(\"Loading model - working dir: %s\", os.getcwd())\n # TODO: Request ID is not given. UUID is a temp UUID.\n metrics = MetricsStore(uuid.uuid4(), model_name)\n manifest_file = os.path.join(model_dir, \"MAR-INF/MANIFEST.json\")\n manifest = None\n if os.path.exists(manifest_file):\n with open(manifest_file) as f:\n manifest = json.load(f)\n\n temp = handler.split(\":\", 1)\n module_name = temp[0]\n function_name = None if len(temp) == 1 else temp[1]\n if module_name.endswith(\".py\"):\n module_name = module_name[:-3]\n module_name = module_name.split(\"/\")[-1]\n self.module = importlib.import_module(module_name)\n if self.module is None:\n raise ValueError(\"Unable to load module {}, make sure it is added to python path\".format(module_name))\n if function_name is None:\n function_name = \"handle\"\n if hasattr(self.module, function_name):\n entry_point = getattr(self.module, function_name)\n service = Service(model_name, model_dir, manifest, entry_point, gpu_id, batch_size)\n\n service.context.metrics = metrics\n # initialize model at load time\n entry_point(None, service.context)\n else:\n model_class_definitions = ModelLoader.list_model_services(self.module)\n if len(model_class_definitions) != 1:\n raise ValueError(\"Expected only one class in custom service code or a function entry point {}\".format(\n model_class_definitions))\n\n model_class = model_class_definitions[0]\n model_service = model_class()\n handle = getattr(model_service, \"handle\")\n if handle is None:\n raise ValueError(\"Expect handle method in class {}\".format(str(model_class)))\n\n service = Service(model_name, model_dir, manifest, model_service.handle, gpu_id, batch_size)\n initialize = getattr(model_service, \"initialize\")\n if initialize is not None:\n # noinspection PyBroadException\n try:\n model_service.initialize(service.context)\n # pylint: disable=broad-except\n except Exception as e:\n # noinspection PyBroadException\n logging.exception(e)\n try:\n sys.exc_clear()\n # pylint: disable=broad-except\n except Exception:\n pass\n\n return service\n\n def unload(self):\n # to make sure logs emitted from model on exit get into mms logs,\n # do not delete logging python module\n module_vars = [var for var in vars(self.module) if not var.startswith('__') and not var == \"logging\"]\n for var in module_vars:\n delattr(self.module, var)\n del self.module\n\nclass LegacyModelLoader(ModelLoader):\n \"\"\"\n MMS 0.4 Model Loader\n \"\"\"\n\n def load(self, model_name, model_dir, handler, gpu_id, batch_size):\n \"\"\"\n Load MMS 0.3 model from file.\n\n :param model_name:\n :param model_dir:\n :param handler:\n :param gpu_id:\n :param batch_size:\n :return:\n \"\"\"\n manifest_file = os.path.join(model_dir, \"MANIFEST.json\")\n\n manifest = None\n if os.path.isfile(manifest_file):\n with open(manifest_file) as f:\n manifest = json.load(f)\n if not handler.endswith(\".py\"):\n handler = handler + \".py\"\n\n service_file = os.path.join(model_dir, handler)\n name = os.path.splitext(os.path.basename(service_file))[0]\n if sys.version_info[0] > 2:\n from importlib import util\n\n spec = util.spec_from_file_location(name, service_file)\n module = util.module_from_spec(spec)\n spec.loader.exec_module(module)\n else:\n import imp\n module = imp.load_source(name, service_file)\n\n if module is None:\n raise ValueError(\"Unable to load module {}\".format(service_file))\n\n from mms.model_service.mxnet_model_service import SingleNodeService\n\n model_class_definitions = ModelLoader.list_model_services(module, SingleNodeService)\n module_class = model_class_definitions[0]\n\n module = module_class(model_name, model_dir, manifest, gpu_id)\n service = Service(model_name, model_dir, manifest, module.handle, gpu_id, batch_size)\n\n module.initialize(service.context)\n\n return service\n" }, { "path": "mms/model_server.py", "content": "\"\"\"\nFile to define the entry point to Model Server\n\"\"\"\n\nimport os\nimport re\nimport subprocess\nimport sys\nimport tempfile\nfrom builtins import str\n\nimport psutil\n\nfrom mms.arg_parser import ArgParser\n\n\ndef old_start():\n \"\"\"\n This is the entry point for model server when using the old CLI name (mxnet-model-server).\n Please migrate to multi-model-server in the future\n :return:\n \"\"\"\n\n print(\"Warning: Calling MMS with mxnet-model-server. Please move to multi-model-server.\")\n start()\n\ndef start():\n \"\"\"\n This is the entry point for model server\n :return:\n \"\"\"\n args = ArgParser.mms_parser().parse_args()\n pid_file = os.path.join(tempfile.gettempdir(), \".model_server.pid\")\n pid = None\n if os.path.isfile(pid_file):\n with open(pid_file, \"r\") as f:\n pid = int(f.readline())\n\n # pylint: disable=too-many-nested-blocks\n if args.stop:\n if pid is None:\n print(\"Model server is not currently running.\")\n else:\n try:\n parent = psutil.Process(pid)\n for child in parent.children(recursive=True):\n child.terminate()\n for child in parent.children(recursive=True):\n if psutil.pid_exists(child.pid):\n child.kill()\n parent.terminate()\n if psutil.pid_exists(parent.pid):\n parent.kill()\n print(\"Model server stopped.\")\n except (OSError, psutil.Error):\n print(\"Model server already stopped.\")\n os.remove(pid_file)\n else:\n if pid is not None:\n try:\n psutil.Process(pid)\n print(\"Model server is already running, please use multi-model-server --stop to stop MMS.\")\n exit(1)\n except psutil.Error:\n print(\"Removing orphan pid file.\")\n os.remove(pid_file)\n\n java_home = os.environ.get(\"JAVA_HOME\")\n java = \"java\" if not java_home else \"{}/bin/java\".format(java_home)\n\n mms_home = os.path.dirname(os.path.dirname(os.path.realpath(__file__)))\n cmd = [java, \"-Dmodel_server_home={}\".format(mms_home)]\n if args.log_config:\n log_config = os.path.realpath(args.log_config)\n if not os.path.isfile(log_config):\n print(\"--log-config file not found: {}\".format(log_config))\n exit(1)\n\n cmd.append(\"-Dlog4j.configurationFile=file://{}\".format(log_config))\n\n tmp_dir = os.environ.get(\"TEMP\")\n if tmp_dir:\n if not os.path.isdir(tmp_dir):\n print(\"Invalid temp directory: {}, please check TEMP environment variable.\".format(tmp_dir))\n exit(1)\n\n cmd.append(\"-Djava.io.tmpdir={}\".format(tmp_dir))\n\n mms_config = args.mms_config\n mms_conf_file = None\n if mms_config:\n if mms_config == \"sagemaker\":\n mms_config = os.path.join(os.path.dirname(os.path.realpath(__file__)),\n \"configs\", \"sagemaker_config.properties\")\n if not os.path.isfile(mms_config):\n print(\"--mms-config file not found: {}\".format(mms_config))\n exit(1)\n mms_conf_file = mms_config\n else:\n mms_conf_file = \"config.properties\"\n\n class_path = \\\n \".:{}\".format(os.path.join(mms_home, \"mms/frontend/*\"))\n\n if os.path.isfile(mms_conf_file):\n props = load_properties(mms_conf_file)\n vm_args = props.get(\"vmargs\")\n if vm_args:\n print(\"Warning: MMS is using non-default JVM parameters: {}\".format(vm_args))\n arg_list = vm_args.split()\n if args.log_config:\n for word in arg_list[:]:\n if word.startswith(\"-Dlog4j.configurationFile=\"):\n arg_list.remove(word)\n cmd.extend(arg_list)\n plugins = props.get(\"plugins_path\", None)\n if plugins:\n class_path += \":\" + plugins + \"/*\" if \"*\" not in plugins else \":\" + plugins\n\n cmd.append(\"-cp\")\n cmd.append(class_path)\n\n cmd.append(\"com.amazonaws.ml.mms.ModelServer\")\n\n # model-server.jar command line parameters\n cmd.append(\"--python\")\n cmd.append(sys.executable)\n\n if mms_conf_file is not None:\n cmd.append(\"-f\")\n cmd.append(mms_conf_file)\n\n if args.model_store:\n if not os.path.isdir(args.model_store):\n print(\"--model-store directory not found: {}\".format(args.model_store))\n exit(1)\n\n cmd.append(\"-s\")\n cmd.append(args.model_store)\n\n if args.models:\n cmd.append(\"-m\")\n cmd.extend(args.models)\n if not args.model_store:\n pattern = re.compile(r\"(.+=)?http(s)?://.+\", re.IGNORECASE)\n for model_url in args.models:\n if not pattern.match(model_url) and model_url != \"ALL\":\n print(\"--model-store is required to load model locally.\")\n exit(1)\n\n try:\n process = subprocess.Popen(cmd)\n pid = process.pid\n with open(pid_file, \"w\") as pf:\n pf.write(str(pid))\n if args.foreground:\n process.wait()\n except OSError as e:\n if e.errno == 2:\n print(\"java not found, please make sure JAVA_HOME is set properly.\")\n else:\n print(\"start java frontend failed:\", sys.exc_info())\n\n\ndef load_properties(file_path):\n \"\"\"\n Read properties file into map.\n \"\"\"\n props = {}\n with open(file_path, \"rt\") as f:\n for line in f:\n line = line.strip()\n if not line.startswith(\"#\"):\n pair = line.split(\"=\", 1)\n if len(pair) > 1:\n key = pair[0].strip()\n props[key] = pair[1].strip()\n\n return props\n\n\nif __name__ == \"__main__\":\n start()\n" }, { "path": "mms/model_service/__init__.py", "content": "# Copyright 2017 Amazon.com, Inc. or its affiliates. All Rights Reserved.\n# Licensed under the Apache License, Version 2.0 (the \"License\").\n# You may not use this file except in compliance with the License.\n# A copy of the License is located at\n# http://www.apache.org/licenses/LICENSE-2.0\n# or in the \"license\" file accompanying this file. This file is distributed\n# on an \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either\n# express or implied. See the License for the specific language governing\n# permissions and limitations under the License.\n\"\"\"\nModel services code\n\"\"\"\nimport warnings\n\nfrom . import model_service\nfrom . import mxnet_model_service\nfrom . import mxnet_vision_service\n\nwarnings.warn(\"Module mms.model_service is deprecated, please migrate to model archive 1.0 format.\",\n DeprecationWarning, stacklevel=2)\n" }, { "path": "mms/model_service/gluon_vision_service.py", "content": "# Copyright 2018 Amazon.com, Inc. or its affiliates. All Rights Reserved.\n# Licensed under the Apache License, Version 2.0 (the \"License\").\n# You may not use this file except in compliance with the License.\n# A copy of the License is located at\n# http://www.apache.org/licenses/LICENSE-2.0\n# or in the \"license\" file accompanying this file. This file is distributed\n# on an \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either\n# express or implied. See the License for the specific language governing\n# permissions and limitations under the License.\n\n\"\"\"`Gluon vision service` defines a Gluon base vision service\n\"\"\"\nimport numpy as np\nimport mxnet\nfrom mms.model_service.mxnet_model_service import GluonImperativeBaseService\nfrom mms.utils.mxnet import ndarray\n\n\nclass GluonVisionService(GluonImperativeBaseService):\n \"\"\"MXNetVisionService defines a fundamental service for image classification task.\n In preprocess, input image buffer is read to NDArray and resized respect to input\n shape in signature.\n In post process, top-5 labels are returned.\n \"\"\"\n def _preprocess(self, data):\n img_list = []\n for idx, img in enumerate(data):\n input_shape = self.signature['inputs'][idx]['data_shape']\n # We are assuming input shape is NCHW\n [h, w] = input_shape[2:]\n img_arr = mxnet.img.imdecode(img)\n img_arr = mxnet.image.imresize(img_arr, w, h)\n img_arr = img_arr.astype(np.float32)\n img_arr /= 255\n img_arr = mxnet.image.color_normalize(img_arr,\n mean=mxnet.nd.array([0.485, 0.456, 0.406]),\n std=mxnet.nd.array([0.229, 0.224, 0.225]))\n img_arr = mxnet.nd.transpose(img_arr, (2, 0, 1))\n img_arr = img_arr.expand_dims(axis=0)\n img_list.append(img_arr)\n return img_list\n\n def _inference(self, data):\n \"\"\"\n Internal inference methods for MMS service. Run forward computation and\n return output.\n\n Parameters\n ----------\n data : list of NDArray\n Preprocessed inputs in NDArray format.\n\n Returns\n -------\n list of NDArray\n Inference output.\n \"\"\"\n # Check input shape\n super(GluonVisionService, self)._inference(data)\n output = self.net(data[0])\n return output.softmax()\n\n def _postprocess(self, data):\n assert hasattr(self, 'labels'), \\\n \"Can't find labels attribute. Did you put synset.txt file into \" \\\n \"model archive or manually load class label file in __init__?\"\n return [ndarray.top_probability(d, self.labels, top=5) for d in data]\n" }, { "path": "mms/model_service/model_service.py", "content": "# Copyright 2017 Amazon.com, Inc. or its affiliates. All Rights Reserved.\n# Licensed under the Apache License, Version 2.0 (the \"License\").\n# You may not use this file except in compliance with the License.\n# A copy of the License is located at\n# http://www.apache.org/licenses/LICENSE-2.0\n# or in the \"license\" file accompanying this file. This file is distributed\n# on an \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either\n# express or implied. See the License for the specific language governing\n# permissions and limitations under the License.\n\n\"\"\"`ModelService` defines an API for base model service.\n\"\"\"\n# pylint: disable=W0223\n\nimport ast\nimport json\nimport logging\nimport os\nimport time\nfrom abc import ABCMeta, abstractmethod\n\n\nclass ModelService(object):\n \"\"\"\n ModelService wraps up all preprocessing, inference and postprocessing\n functions used by model service. It is defined in a flexible manner to\n be easily extended to support different frameworks.\n \"\"\"\n __metaclass__ = ABCMeta\n\n # noinspection PyUnusedLocal\n def __init__(self, model_name, model_dir, manifest, gpu=None): # pylint: disable=unused-argument\n self.ctx = None\n self._context = None\n self._signature = None\n\n def initialize(self, context):\n \"\"\"\n Internal initialize ModelService.\n\n :param context: MMS context object\n :return:\n \"\"\"\n self._context = context\n properties = context.system_properties\n model_dir = properties.get(\"model_dir\")\n\n signature_file_path = os.path.join(model_dir, context.manifest['Model']['Signature'])\n if not os.path.isfile(signature_file_path):\n raise ValueError(\"Signature file is not found.\")\n\n with open(signature_file_path) as f:\n self._signature = json.load(f)\n\n @abstractmethod\n def inference(self, data):\n \"\"\"\n Wrapper function to run pre-process, inference and post-process functions.\n\n Parameters\n ----------\n data : list of object\n Raw input from request.\n\n Returns\n -------\n list of outputs to be sent back to client.\n data to be sent back\n \"\"\"\n # pylint: disable=unnecessary-pass\n pass\n\n @abstractmethod\n def ping(self):\n \"\"\"\n Ping to get system's health.\n\n Returns\n -------\n String\n A message, \"health\": \"healthy!\", to show system is healthy.\n \"\"\"\n pass # pylint: disable=unnecessary-pass\n\n def signature(self):\n \"\"\"\n Signature for model service.\n\n Returns\n -------\n Dict\n Model service signature.\n \"\"\"\n return self._signature\n\n # noinspection PyUnusedLocal\n def handle(self, data, context): # pylint: disable=unused-argument\n \"\"\"\n Backward compatible handle function.\n\n :param data:\n :param context:\n :return:\n\n \"\"\"\n input_type = self._signature['input_type']\n\n input_data = []\n data_name = self._signature[\"inputs\"][0][\"data_name\"]\n form_data = data[0].get(data_name)\n if form_data is None:\n form_data = data[0].get(\"body\")\n\n if form_data is None:\n form_data = data[0].get(\"data\")\n\n if input_type == \"application/json\":\n # user might not send content in HTTP request\n if isinstance(form_data, (bytes, bytearray)):\n form_data = ast.literal_eval(form_data.decode(\"utf-8\"))\n\n input_data.append(form_data)\n\n ret = self.inference(input_data)\n if isinstance(ret, list):\n return ret\n\n return [ret]\n\n\nclass SingleNodeService(ModelService):\n \"\"\"\n SingleNodeModel defines abstraction for model service which loads a\n single model.\n \"\"\"\n\n def inference(self, data):\n \"\"\"\n Wrapper function to run preprocess, inference and postprocess functions.\n\n Parameters\n ----------\n data : list of object\n Raw input from request.\n\n Returns\n -------\n list of outputs to be sent back to client.\n data to be sent back\n \"\"\"\n preprocess_start = time.time()\n data = self._preprocess(data)\n inference_start = time.time()\n data = self._inference(data)\n postprocess_start = time.time()\n data = self._postprocess(data)\n end_time = time.time()\n\n logging.info(\"preprocess time: %.2f\", (inference_start - preprocess_start) * 1000)\n logging.info(\"inference time: %.2f\", (postprocess_start - inference_start) * 1000)\n logging.info(\"postprocess time: %.2f\", (end_time - postprocess_start) * 1000)\n\n return data\n\n @abstractmethod\n def _inference(self, data):\n \"\"\"\n Internal inference methods. Run forward computation and\n return output.\n\n Parameters\n ----------\n data : list of NDArray\n Preprocessed inputs in NDArray format.\n\n Returns\n -------\n list of NDArray\n Inference output.\n \"\"\"\n return data\n\n def _preprocess(self, data):\n \"\"\"\n Internal preprocess methods. Do transformation on raw\n inputs and convert them to NDArray.\n\n Parameters\n ----------\n data : list of object\n Raw inputs from request.\n\n Returns\n -------\n list of NDArray\n Processed inputs in NDArray format.\n \"\"\"\n return data\n\n def _postprocess(self, data):\n \"\"\"\n Internal postprocess methods. Do transformation on inference output\n and convert them to MIME type objects.\n\n Parameters\n ----------\n data : list of NDArray\n Inference output.\n\n Returns\n -------\n list of object\n list of outputs to be sent back.\n \"\"\"\n return data\n" }, { "path": "mms/model_service/mxnet_model_service.py", "content": "# Copyright 2017 Amazon.com, Inc. or its affiliates. All Rights Reserved.\n# Licensed under the Apache License, Version 2.0 (the \"License\").\n# You may not use this file except in compliance with the License.\n# A copy of the License is located at\n# http://www.apache.org/licenses/LICENSE-2.0\n# or in the \"license\" file accompanying this file. This file is distributed\n# on an \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either\n# express or implied. See the License for the specific language governing\n# permissions and limitations under the License.\n\n\"\"\"\n`MXNetBaseService` defines an API for MXNet service.\n\"\"\"\nimport json\nimport os\nimport logging\n\nimport mxnet as mx\nfrom mxnet.io import DataBatch\n\nfrom .model_service import SingleNodeService\n\n\ndef check_input_shape(inputs, signature):\n \"\"\"\n Check input data shape consistency with signature.\n\n Parameters\n ----------\n inputs : List of NDArray\n Input data in NDArray format.\n signature : dict\n Dictionary containing model signature.\n \"\"\"\n assert isinstance(inputs, list), 'Input data must be a list.'\n assert len(inputs) == len(signature['inputs']), \\\n \"Input number mismatches with \" \\\n \"signature. %d expected but got %d.\" \\\n % (len(signature['inputs']), len(inputs))\n for input_data, sig_input in zip(inputs, signature[\"inputs\"]):\n assert isinstance(input_data, mx.nd.NDArray), 'Each input must be NDArray.'\n assert len(input_data.shape) == len(sig_input[\"data_shape\"]), \\\n 'Shape dimension of input %s mismatches with ' \\\n 'signature. %d expected but got %d.' \\\n % (sig_input['data_name'],\n len(sig_input['data_shape']),\n len(input_data.shape))\n for idx in range(len(input_data.shape)):\n if idx != 0 and sig_input['data_shape'][idx] != 0:\n assert sig_input['data_shape'][idx] == input_data.shape[idx], \\\n 'Input %s has different shape with ' \\\n 'signature. %s expected but got %s.' \\\n % (sig_input['data_name'], sig_input['data_shape'],\n input_data.shape)\n\n\nclass MXNetBaseService(SingleNodeService):\n \"\"\"\n MXNetBaseService defines the fundamental loading model and inference\n operations when serving MXNet model. This is a base class and needs to be\n inherited.\n \"\"\"\n\n def __init__(self, model_name, model_dir, manifest, gpu=None):\n super(MXNetBaseService, self).__init__(model_name, model_dir, manifest, gpu)\n self.param_filename = None\n self.model_name = model_name\n self.ctx = mx.gpu(int(gpu)) if gpu is not None else mx.cpu()\n signature_file_path = os.path.join(model_dir, manifest['Model']['Signature'])\n if not os.path.isfile(signature_file_path):\n raise RuntimeError('Signature file is not found. Please put signature.json '\n 'into the model file directory...' + signature_file_path)\n try:\n signature_file = open(signature_file_path)\n self._signature = json.load(signature_file)\n except Exception:\n raise Exception('Failed to open model signature file: %s' % signature_file_path)\n\n data_names = []\n data_shapes = []\n epoch = 0\n for input_data in self._signature['inputs']:\n data_names.append(input_data['data_name'])\n # Replace 0 entry in data shape with 1 for binding executor.\n # Set batch size as 1\n data_shape = input_data['data_shape']\n data_shape[0] = 1\n # pylint: disable=consider-using-enumerate\n for idx in range(len(data_shape)):\n if data_shape[idx] == 0:\n data_shape[idx] = 1\n data_shapes.append((input_data['data_name'], tuple(data_shape)))\n\n # Load MXNet module\n # noinspection PyBroadException\n try:\n self.param_filename = manifest['Model']['Parameters']\n epoch = int(self.param_filename[len(model_name) + 1: -len('.params')])\n except Exception: # pylint: disable=broad-except\n logging.info(\"Failed to parse epoch from param file, setting epoch to 0\")\n\n sym, arg_params, aux_params = mx.model.load_checkpoint('%s/%s' %\n (model_dir, manifest['Model']['Symbol'][:-12]), epoch)\n self.mx_model = mx.mod.Module(symbol=sym, context=self.ctx,\n data_names=data_names, label_names=None)\n self.mx_model.bind(for_training=False, data_shapes=data_shapes)\n self.mx_model.set_params(arg_params, aux_params, allow_missing=True, allow_extra=True)\n\n # Read synset file\n # If synset is not specified, check whether model archive contains synset file.\n archive_synset = os.path.join(model_dir, 'synset.txt')\n\n if os.path.isfile(archive_synset):\n synset = archive_synset\n self.labels = [line.strip() for line in open(synset).readlines()]\n\n def _preprocess(self, data):\n return map(mx.nd.array, data)\n\n def _postprocess(self, data):\n return [str(d.asnumpy().tolist()) for d in data]\n\n def _inference(self, data):\n \"\"\"Internal inference methods for MXNet. Run forward computation and\n return output.\n\n Parameters\n ----------\n data : list of NDArray\n Preprocessed inputs in NDArray format.\n\n Returns\n -------\n list of NDArray\n Inference output.\n \"\"\"\n # Check input shape\n check_input_shape(data, self.signature)\n data = [item.as_in_context(self.ctx) for item in data]\n self.mx_model.forward(DataBatch(data))\n data = self.mx_model.get_outputs()\n # by pass lazy evaluation get_outputs either returns a list of nd arrays\n # a list of list of NDArray\n for d in data:\n if isinstance(d, list):\n for n in data:\n if isinstance(n, mx.ndarray.ndarray.NDArray):\n n.wait_to_read()\n elif isinstance(d, mx.ndarray.ndarray.NDArray):\n d.wait_to_read()\n return data\n\n def ping(self):\n \"\"\"\n Ping to get system's health.\n\n Returns\n -------\n String\n MXNet version to show system is healthy.\n \"\"\"\n return mx.__version__\n\n @property\n def signature(self):\n \"\"\"\n Signature for model service.\n\n Returns\n -------\n Dict\n Model service signiture.\n \"\"\"\n return self._signature\n\n\nclass GluonImperativeBaseService(SingleNodeService):\n \"\"\"GluonImperativeBaseService defines the fundamental loading model and inference\n operations when serving Gluon model. This is a base class and needs to be\n inherited.\n \"\"\"\n\n def __init__(self, model_name, model_dir, manifest, net=None, gpu=None):\n super(GluonImperativeBaseService, self).__init__(model_name, model_dir, manifest, gpu)\n self.param_filename = None\n self.model_name = model_name\n self.ctx = mx.gpu(int(gpu)) if gpu is not None else mx.cpu()\n self.net = net\n signature_file_path = os.path.join(model_dir, manifest['Model']['Signature'])\n if not os.path.isfile(signature_file_path):\n raise RuntimeError('Signature file is not found. Please put signature.json '\n 'into the model file directory...' + signature_file_path)\n try:\n signature_file = open(signature_file_path)\n self._signature = json.load(signature_file)\n except Exception:\n raise Exception('Failed to open model signature file: %s' % signature_file_path)\n\n # Load MXNet module\n # noinspection PyBroadException\n try:\n self.param_filename = manifest['Model']['Parameters']\n if self.param_filename or self.net is not None:\n self.net.load_params(os.path.join(model_dir, self.param_filename), ctx=self.ctx)\n else:\n logging.info(\"No parameters file given for this imperative service\")\n except Exception: # pylint: disable=broad-except\n logging.info(\"Failed to parse epoch from param file, setting epoch to 0\")\n\n # Read synset file\n # If synset is not specified, check whether model archive contains synset file.\n archive_synset = os.path.join(model_dir, 'synset.txt')\n\n if os.path.isfile(archive_synset):\n synset = archive_synset\n self.labels = [line.strip() for line in open(synset).readlines()]\n\n def _preprocess(self, data):\n pass\n\n def _postprocess(self, data):\n pass\n\n def _inference(self, data):\n check_input_shape(data, self.signature)\n\n def ping(self):\n \"\"\"\n Ping to get system's health.\n\n Returns\n -------\n String\n MXNet version to show system is healthy.\n \"\"\"\n return mx.__version__\n\n @property\n def signature(self):\n \"\"\"\n Signature for model service.\n\n Returns\n -------\n Dict\n Model service signature.\n \"\"\"\n return self._signature\n" }, { "path": "mms/model_service/mxnet_vision_service.py", "content": "# Copyright 2017 Amazon.com, Inc. or its affiliates. All Rights Reserved.\n# Licensed under the Apache License, Version 2.0 (the \"License\").\n# You may not use this file except in compliance with the License.\n# A copy of the License is located at\n# http://www.apache.org/licenses/LICENSE-2.0\n# or in the \"license\" file accompanying this file. This file is distributed\n# on an \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either\n# express or implied. See the License for the specific language governing\n# permissions and limitations under the License.\n\n\"\"\"`MXNetVisionService` defines a MXNet base vision service\n\"\"\"\n\nfrom mms.model_service.mxnet_model_service import MXNetBaseService\nfrom mms.utils.mxnet import image, ndarray\n\n\nclass MXNetVisionService(MXNetBaseService):\n \"\"\"MXNetVisionService defines a fundamental service for image classification task.\n In preprocess, input image buffer is read to NDArray and resized respect to input\n shape in signature.\n In post process, top-5 labels are returned.\n \"\"\"\n def _preprocess(self, data):\n img_list = []\n for idx, img in enumerate(data):\n input_shape = self.signature['inputs'][idx]['data_shape']\n # We are assuming input shape is NCHW\n [h, w] = input_shape[2:]\n img_arr = image.read(img)\n img_arr = image.resize(img_arr, w, h)\n img_arr = image.transform_shape(img_arr)\n img_list.append(img_arr)\n return img_list\n\n def _postprocess(self, data):\n assert hasattr(self, 'labels'), \\\n \"Can't find labels attribute. Did you put synset.txt file into \" \\\n \"model archive or manually load class label file in __init__?\"\n return [ndarray.top_probability(d, self.labels, top=5) for d in data]\n" }, { "path": "mms/model_service_worker.py", "content": "# Copyright 2018 Amazon.com, Inc. or its affiliates. All Rights Reserved.\n# Licensed under the Apache License, Version 2.0 (the \"License\").\n# You may not use this file except in compliance with the License.\n# A copy of the License is located at\n# http://www.apache.org/licenses/LICENSE-2.0\n# or in the \"license\" file accompanying this file. This file is distributed\n# on an \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either\n# express or implied. See the License for the specific language governing\n# permissions and limitations under the License.\n\n\"\"\"\nModelServiceWorker is the worker that is started by the MMS front-end.\nCommunication message format: binary encoding\n\"\"\"\n\n# pylint: disable=redefined-builtin\n\nimport logging\nimport os\nimport multiprocessing\nimport platform\nimport socket\nimport sys\nimport signal\n\nfrom mms.arg_parser import ArgParser\nfrom mms.model_loader import ModelLoaderFactory\nfrom mms.protocol.otf_message_handler import retrieve_msg, create_load_model_response\nfrom mms.service import emit_metrics\n\nMAX_FAILURE_THRESHOLD = 5\nSOCKET_ACCEPT_TIMEOUT = 30.0\nDEBUG = False\n\n\nclass MXNetModelServiceWorker(object):\n \"\"\"\n Backend worker to handle Model Server's python service code\n \"\"\"\n def __init__(self, s_type=None, s_name=None, host_addr=None, port_num=None,\n model_request=None, preload_model=False, tmp_dir=\"/tmp\"):\n if os.environ.get(\"OMP_NUM_THREADS\") is None:\n os.environ[\"OMP_NUM_THREADS\"] = \"1\"\n if os.environ.get(\"MXNET_USE_OPERATOR_TUNING\") is None:\n # work around issue: https://github.com/apache/incubator-mxnet/issues/12255\n os.environ[\"MXNET_USE_OPERATOR_TUNING\"] = \"0\"\n\n self.sock_type = s_type\n if s_type == \"unix\":\n if s_name is None:\n raise ValueError(\"Wrong arguments passed. No socket name given.\")\n self.sock_name, self.port = s_name, -1\n try:\n os.remove(s_name)\n except OSError:\n if os.path.exists(s_name):\n raise RuntimeError(\"socket already in use: {}.\".format(s_name))\n\n elif s_type == \"tcp\":\n self.sock_name = host_addr if host_addr is not None else \"127.0.0.1\"\n if port_num is None:\n raise ValueError(\"Wrong arguments passed. No socket port given.\")\n self.port = port_num\n else:\n raise ValueError(\"Invalid socket type provided\")\n\n logging.info(\"Listening on port: %s\", s_name)\n socket_family = socket.AF_INET if s_type == \"tcp\" else socket.AF_UNIX\n self.sock = socket.socket(socket_family, socket.SOCK_STREAM)\n self.preload = preload_model\n self.service = None\n self.model_meta_data = model_request\n self.out = self.err = None\n self.tmp_dir = tmp_dir\n self.socket_name = s_name\n\n def load_model(self, load_model_request=None):\n \"\"\"\n Expected command\n {\n \"command\" : \"load\", string\n \"modelPath\" : \"/path/to/model/file\", string\n \"modelName\" : \"name\", string\n \"gpu\" : None if CPU else gpu_id, int\n \"handler\" : service handler entry point if provided, string\n \"batchSize\" : batch size, int\n }\n\n :param load_model_request:\n :return:\n \"\"\"\n try:\n model_dir = load_model_request[\"modelPath\"].decode(\"utf-8\")\n model_name = load_model_request[\"modelName\"].decode(\"utf-8\")\n handler = load_model_request[\"handler\"].decode(\"utf-8\")\n batch_size = 1\n if \"batchSize\" in load_model_request:\n batch_size = int(load_model_request[\"batchSize\"])\n\n gpu = None\n if \"gpu\" in load_model_request:\n gpu = int(load_model_request[\"gpu\"])\n io_fd = None\n if \"ioFileDescriptor\" in load_model_request:\n io_fd = load_model_request.get(\"ioFileDescriptor\").decode(\"utf-8\")\n self._create_io_files(self.tmp_dir, io_fd)\n if self.service is None or self.preload is False:\n self.model_loader = ModelLoaderFactory.get_model_loader(model_dir)\n self.service = self.model_loader.load(model_name, model_dir, handler, gpu, batch_size)\n logging.info(\"Model %s loaded io_fd=%s\", model_name, str(io_fd))\n return \"loaded model {}. [PID]:{}\".format(model_name, os.getpid()), 200\n\n except MemoryError:\n return \"System out of memory\", 507\n\n def _create_io_files(self, tmp_dir, io_fd):\n self.out = tmp_dir + '/' + io_fd + \"-stdout\"\n self.err = tmp_dir + '/' + io_fd + \"-stderr\"\n # TODO: Windows support\n os.mkfifo(self.out)\n os.mkfifo(self.err)\n\n def _remap_io(self):\n out_fd = open(self.out, \"w\")\n err_fd = open(self.err, \"w\")\n os.dup2(out_fd.fileno(), sys.stdout.fileno())\n os.dup2(err_fd.fileno(), sys.stderr.fileno())\n\n def handle_connection(self, cl_socket):\n \"\"\"\n Handle socket connection.\n\n :param cl_socket:\n :return:\n \"\"\"\n logging.basicConfig(stream=sys.stdout, format=\"%(message)s\", level=logging.INFO)\n cl_socket.setblocking(True)\n while True:\n cmd, msg = retrieve_msg(cl_socket)\n if cmd == b'I':\n resp = self.service.predict(msg)\n cl_socket.send(resp)\n elif cmd == b'L':\n result, code = self.load_model(msg)\n resp = bytearray()\n resp += create_load_model_response(code, result)\n cl_socket.send(resp)\n self._remap_io()\n if code != 200:\n raise RuntimeError(\"{} - {}\".format(code, result))\n else:\n raise ValueError(\"Received unknown command: {}\".format(cmd))\n\n if self.service is not None and self.service.context is not None \\\n and self.service.context.metrics is not None:\n emit_metrics(self.service.context.metrics.store)\n\n def sigterm_handler(self):\n for node in [self.socket_name, self.out, self.err]:\n try:\n os.remove(node)\n except OSError:\n pass\n\n def start_worker(self, cl_socket):\n \"\"\"\n Method to start the worker threads. These worker threads use multiprocessing to spawn a new worker.\n\n :param cl_socket:\n :return:\n \"\"\"\n self.sock.close() # close listening socket in the fork\n try:\n signal.signal(signal.SIGTERM, lambda signum, frame: self.sigterm_handler())\n self.handle_connection(cl_socket)\n except Exception: # pylint: disable=broad-except\n logging.error(\"Backend worker process died.\", exc_info=True)\n finally:\n try:\n self.model_loader.unload()\n sys.stdout.flush()\n os.remove(self.out)\n os.remove(self.err)\n finally:\n cl_socket.shutdown(socket.SHUT_RDWR)\n cl_socket.close()\n sys.exit(0)\n\n def run_server(self):\n \"\"\"\n Run the backend worker process and listen on a socket\n :return:\n \"\"\"\n if self.sock_type == \"unix\":\n self.sock.bind(self.sock_name)\n else:\n self.sock.bind((self.sock_name, int(self.port)))\n\n self.sock.listen(128)\n logging.info(\"[PID] %d\", os.getpid())\n logging.info(\"MMS worker started.\")\n logging.info(\"Python runtime: %s\", platform.python_version())\n while True:\n if self.service is None and self.preload is True:\n # Lazy loading the models\n self.load_model(self.model_meta_data)\n\n (cl_socket, _) = self.sock.accept()\n # workaround error(35, 'Resource temporarily unavailable') on OSX\n cl_socket.setblocking(True)\n\n logging.info(\"Connection accepted: %s.\", cl_socket.getsockname())\n p = multiprocessing.Process(target=self.start_worker, args=(cl_socket,))\n p.start()\n cl_socket.close() # close accepted socket in the parent\n\nif __name__ == \"__main__\":\n # Remove mms dir from python path to avoid module name conflict.\n mms_path = os.path.dirname(os.path.realpath(__file__))\n while mms_path in sys.path:\n sys.path.remove(mms_path)\n\n sock_type = None\n socket_name = None\n\n # noinspection PyBroadException\n try:\n logging.basicConfig(stream=sys.stdout, format=\"%(message)s\", level=logging.INFO)\n logging.info(\"model_service_worker started with args: %s\", \" \".join(sys.argv[1:]))\n model_req = dict()\n args = ArgParser.model_service_worker_args().parse_args()\n socket_name = args.sock_name\n sock_type = args.sock_type\n host = args.host\n port = args.port\n model_req[\"handler\"] = args.handler.encode('utf-8')\n model_req[\"modelPath\"] = args.model_path.encode('utf-8')\n model_req[\"modelName\"] = args.model_name.encode('utf-8')\n worker = MXNetModelServiceWorker(sock_type, socket_name, host, port, model_req,\n args.preload_model, args.tmp_dir)\n\n worker.run_server()\n except socket.timeout:\n logging.error(\"Backend worker did not receive connection in: %d\", SOCKET_ACCEPT_TIMEOUT)\n except Exception: # pylint: disable=broad-except\n logging.error(\"Backend worker process died\", exc_info=True)\n finally:\n if sock_type == 'unix' and os.path.exists(socket_name):\n os.remove(socket_name)\n\n exit(1)\n" }, { "path": "mms/protocol/__init__.py", "content": "# Copyright 2018 Amazon.com, Inc. or its affiliates. All Rights Reserved.\n# Licensed under the Apache License, Version 2.0 (the \"License\").\n# You may not use this file except in compliance with the License.\n# A copy of the License is located at\n# http://www.apache.org/licenses/LICENSE-2.0\n# or in the \"license\" file accompanying this file. This file is distributed\n# on an \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either\n# express or implied. See the License for the specific language governing\n# permissions and limitations under the License.\n" }, { "path": "mms/protocol/otf_message_handler.py", "content": "# Copyright 2018 Amazon.com, Inc. or its affiliates. All Rights Reserved.\n# Licensed under the Apache License, Version 2.0 (the \"License\").\n# You may not use this file except in compliance with the License.\n# A copy of the License is located at\n# http://www.apache.org/licenses/LICENSE-2.0\n# or in the \"license\" file accompanying this file. This file is distributed\n# on an \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either\n# express or implied. See the License for the specific language governing\n# permissions and limitations under the License.\n\n\"\"\"\nOTF Codec\n\"\"\"\nimport json\nimport logging\nimport struct\nimport os\n\nfrom builtins import bytearray\nfrom builtins import bytes\n\nint_size = 4\nEND_OF_LIST = -1\nLOAD_MSG = b'L'\nPREDICT_MSG = b'I'\nRESPONSE = 3\n\n\ndef retrieve_msg(conn):\n \"\"\"\n Retrieve a message from the socket channel.\n\n :param conn:\n :return:\n \"\"\"\n cmd = _retrieve_buffer(conn, 1)\n if cmd == LOAD_MSG:\n msg = _retrieve_load_msg(conn)\n elif cmd == PREDICT_MSG:\n msg = _retrieve_inference_msg(conn)\n else:\n raise ValueError(\"Invalid command: {}\".format(cmd))\n\n return cmd, msg\n\n\ndef encode_response_headers(resp_hdr_map):\n msg = bytearray()\n msg += struct.pack('!i', len(resp_hdr_map))\n for k, v in resp_hdr_map.items():\n msg += struct.pack('!i', len(k.encode('utf-8')))\n msg += k.encode('utf-8')\n msg += struct.pack('!i', len(v.encode('utf-8')))\n msg += v.encode('utf-8')\n return msg\n\n\ndef create_predict_response(ret, req_id_map, message, code, context=None):\n \"\"\"\n Create inference response.\n\n :param context:\n :param ret:\n :param req_id_map:\n :param message:\n :param code:\n :return:\n \"\"\"\n msg = bytearray()\n msg += struct.pack('!i', code)\n\n buf = message.encode(\"utf-8\")\n msg += struct.pack('!i', len(buf))\n msg += buf\n\n for idx in req_id_map:\n req_id = req_id_map.get(idx).encode('utf-8')\n msg += struct.pack(\"!i\", len(req_id))\n msg += req_id\n\n # Encoding Content-Type\n if context is None:\n msg += struct.pack('!i', 0) # content_type\n else:\n content_type = context.get_response_content_type(idx)\n if content_type is None or len(content_type) == 0:\n msg += struct.pack('!i', 0) # content_type\n else:\n msg += struct.pack('!i', len(content_type))\n msg += content_type.encode('utf-8')\n\n # Encoding the per prediction HTTP response code\n if context is None:\n # status code and reason phrase set to none\n msg += struct.pack('!i', code)\n msg += struct.pack('!i', 0) # No code phrase is returned\n # Response headers none\n msg += struct.pack('!i', 0)\n else:\n sc, phrase = context.get_response_status(idx)\n http_code = sc if sc is not None else 200\n http_phrase = phrase if phrase is not None else \"\"\n\n msg += struct.pack('!i', http_code)\n msg += struct.pack(\"!i\", len(http_phrase))\n msg += http_phrase.encode(\"utf-8\")\n # Response headers\n msg += encode_response_headers(context.get_response_headers(idx))\n\n if ret is None:\n buf = b\"error\"\n msg += struct.pack('!i', len(buf))\n msg += buf\n else:\n val = ret[idx]\n # NOTE: Process bytes/bytearray case before processing the string case.\n if isinstance(val, (bytes, bytearray)):\n msg += struct.pack('!i', len(val))\n msg += val\n elif isinstance(val, str):\n buf = val.encode(\"utf-8\")\n msg += struct.pack('!i', len(buf))\n msg += buf\n else:\n try:\n json_value = json.dumps(val, indent=2).encode(\"utf-8\")\n msg += struct.pack('!i', len(json_value))\n msg += json_value\n except TypeError:\n logging.warning(\"Unable to serialize model output.\", exc_info=True)\n return create_predict_response(None, req_id_map, \"Unsupported model output data type.\", 503)\n\n msg += struct.pack('!i', -1) # End of list\n return msg\n\n\ndef create_load_model_response(code, message):\n \"\"\"\n Create load model response.\n\n :param code:\n :param message:\n :return:\n \"\"\"\n msg = bytearray()\n msg += struct.pack('!i', code)\n\n buf = message.encode(\"utf-8\")\n msg += struct.pack('!i', len(buf))\n msg += buf\n msg += struct.pack('!i', -1) # no predictions\n\n return msg\n\n\ndef _retrieve_buffer(conn, length):\n data = bytearray()\n\n while length > 0:\n pkt = conn.recv(length)\n if len(pkt) == 0:\n logging.info(\"Frontend disconnected.\")\n raise ValueError(\"Frontend disconnected\")\n\n data += pkt\n length -= len(pkt)\n\n return data\n\n\ndef _retrieve_int(conn):\n data = _retrieve_buffer(conn, int_size)\n return struct.unpack(\"!i\", data)[0]\n\n\ndef _retrieve_load_msg(conn):\n \"\"\"\n MSG Frame Format:\n\n | cmd value |\n | int model-name length | model-name value |\n | int model-path length | model-path value |\n | int batch-size length |\n | int handler length | handler value |\n | int gpu id |\n\n :param conn:\n :return:\n \"\"\"\n msg = dict()\n length = _retrieve_int(conn)\n msg[\"modelName\"] = _retrieve_buffer(conn, length)\n length = _retrieve_int(conn)\n msg[\"modelPath\"] = _retrieve_buffer(conn, length)\n msg[\"batchSize\"] = _retrieve_int(conn)\n length = _retrieve_int(conn)\n msg[\"handler\"] = _retrieve_buffer(conn, length)\n gpu_id = _retrieve_int(conn)\n if gpu_id >= 0:\n msg[\"gpu\"] = gpu_id\n length = _retrieve_int(conn)\n msg[\"ioFileDescriptor\"] = _retrieve_buffer(conn, length)\n\n return msg\n\n\ndef _retrieve_inference_msg(conn):\n \"\"\"\n MSG Frame Format:\n\n | cmd value |\n | batch: list of requests |\n \"\"\"\n msg = []\n while True:\n request = _retrieve_request(conn)\n if request is None:\n break\n\n msg.append(request)\n\n return msg\n\n\ndef _retrieve_request(conn):\n \"\"\"\n MSG Frame Format:\n\n | request_id |\n | request_headers: list of request headers|\n | parameters: list of request parameters |\n \"\"\"\n length = _retrieve_int(conn)\n if length == -1:\n return None\n\n request = dict()\n request[\"requestId\"] = _retrieve_buffer(conn, length)\n\n headers = []\n while True:\n header = _retrieve_reqest_header(conn)\n if header is None:\n break\n headers.append(header)\n\n request[\"headers\"] = headers\n\n model_inputs = []\n while True:\n input_data = _retrieve_input_data(conn)\n if input_data is None:\n break\n model_inputs.append(input_data)\n\n request[\"parameters\"] = model_inputs\n return request\n\n\ndef _retrieve_reqest_header(conn):\n \"\"\"\n MSG Frame Format:\n\n | parameter_name |\n | content_type |\n | input data in bytes |\n \"\"\"\n length = _retrieve_int(conn)\n if length == -1:\n return None\n\n header = dict()\n header[\"name\"] = _retrieve_buffer(conn, length)\n\n length = _retrieve_int(conn)\n header[\"value\"] = _retrieve_buffer(conn, length)\n\n return header\n\n\ndef _retrieve_input_data(conn):\n \"\"\"\n MSG Frame Format:\n\n | parameter_name |\n | content_type |\n | input data in bytes |\n \"\"\"\n decode_req = os.environ.get(\"MMS_DECODE_INPUT_REQUEST\")\n length = _retrieve_int(conn)\n if length == -1:\n return None\n\n model_input = dict()\n model_input[\"name\"] = _retrieve_buffer(conn, length).decode(\"utf-8\")\n\n length = _retrieve_int(conn)\n content_type = _retrieve_buffer(conn, length).decode(\"utf-8\")\n model_input[\"contentType\"] = content_type\n\n length = _retrieve_int(conn)\n value = _retrieve_buffer(conn, length)\n if content_type == \"application/json\" and (decode_req is None or decode_req == \"true\"):\n model_input[\"value\"] = json.loads(value.decode(\"utf-8\"))\n elif content_type.startswith(\"text\") and (decode_req is None or decode_req == \"true\"):\n model_input[\"value\"] = value.decode(\"utf-8\")\n else:\n model_input[\"value\"] = value\n return model_input\n" }, { "path": "mms/service.py", "content": "# Copyright 2018 Amazon.com, Inc. or its affiliates. All Rights Reserved.\n# Licensed under the Apache License, Version 2.0 (the \"License\").\n# You may not use this file except in compliance with the License.\n# A copy of the License is located at\n# http://www.apache.org/licenses/LICENSE-2.0\n# or in the \"license\" file accompanying this file. This file is distributed\n# on an \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either\n# express or implied. See the License for the specific language governing\n# permissions and limitations under the License.\n\n\"\"\"\nCustomService class definitions\n\"\"\"\nimport logging\nimport time\n\nfrom builtins import str\n\nimport mms\nfrom mms.context import Context, RequestProcessor\nfrom mms.metrics.metrics_store import MetricsStore\nfrom mms.protocol.otf_message_handler import create_predict_response\n\nPREDICTION_METRIC = 'PredictionTime'\nlogger = logging.getLogger(__name__)\n\n\nclass Service(object):\n \"\"\"\n Wrapper for custom entry_point\n \"\"\"\n\n def __init__(self, model_name, model_dir, manifest, entry_point, gpu, batch_size):\n self._context = Context(model_name, model_dir, manifest, batch_size, gpu, mms.__version__)\n self._entry_point = entry_point\n\n @property\n def context(self):\n return self._context\n\n @staticmethod\n def retrieve_data_for_inference(batch):\n \"\"\"\n\n REQUEST_INPUT = {\n \"requestId\" : \"111-222-3333\",\n \"parameters\" : [ PARAMETER ]\n }\n\n PARAMETER = {\n \"name\" : parameter name\n \"contentType\": \"http-content-types\",\n \"value\": \"val1\"\n }\n\n :param batch:\n :return:\n \"\"\"\n if batch is None:\n raise ValueError(\"Received invalid inputs\")\n\n req_to_id_map = {}\n headers = []\n input_batch = []\n for batch_idx, request_batch in enumerate(batch):\n req_id = request_batch.get('requestId').decode(\"utf-8\")\n parameters = request_batch['parameters']\n model_in_headers = dict()\n\n model_in = dict()\n # Parameter level headers are updated here. multipart/form-data can have multiple headers.\n for parameter in parameters:\n model_in.update({parameter[\"name\"]: parameter[\"value\"]})\n model_in_headers.update({parameter[\"name\"]: {\"content-type\": parameter[\"contentType\"]}})\n\n # Request level headers are populated here\n if request_batch.get(\"headers\") is not None:\n for h in request_batch.get(\"headers\"):\n model_in_headers.update({h['name'].decode('utf-8'): h['value'].decode('utf-8')})\n\n headers.append(RequestProcessor(model_in_headers))\n input_batch.append(model_in)\n req_to_id_map[batch_idx] = req_id\n\n return headers, input_batch, req_to_id_map\n\n def predict(self, batch):\n \"\"\"\n PREDICT COMMAND = {\n \"command\": \"predict\",\n \"batch\": [ REQUEST_INPUT ]\n }\n :param batch: list of request\n :return:\n\n \"\"\"\n headers, input_batch, req_id_map = Service.retrieve_data_for_inference(batch)\n\n self.context.request_ids = req_id_map\n self.context.request_processor = headers\n metrics = MetricsStore(req_id_map, self.context.model_name)\n self.context.metrics = metrics\n\n start_time = time.time()\n\n # noinspection PyBroadException\n try:\n ret = self._entry_point(input_batch, self.context)\n except PredictionException as e:\n logger.error(\"Prediction error\", exc_info=True)\n return create_predict_response(None, req_id_map, e.message, e.error_code)\n except MemoryError:\n logger.error(\"System out of memory\", exc_info=True)\n return create_predict_response(None, req_id_map, \"Out of resources\", 507)\n except Exception: # pylint: disable=broad-except\n logger.warning(\"Invoking custom service failed.\", exc_info=True)\n return create_predict_response(None, req_id_map, \"Prediction failed\", 503)\n\n if not isinstance(ret, list):\n logger.warning(\"model: %s, Invalid return type: %s.\", self.context.model_name, type(ret))\n return create_predict_response(None, req_id_map, \"Invalid model predict output\", 503)\n\n if len(ret) != len(input_batch):\n logger.warning(\"model: %s, number of batch response mismatched, expect: %d, got: %d.\",\n self.context.model_name, len(input_batch), len(ret))\n return create_predict_response(None, req_id_map, \"number of batch response mismatched\", 503)\n\n duration = round((time.time() - start_time) * 1000, 2)\n metrics.add_time(PREDICTION_METRIC, duration)\n\n return create_predict_response(ret, req_id_map, \"Prediction success\", 200, context=self.context)\n\n\nclass PredictionException(Exception):\n def __init__(self, message, error_code=500):\n self.message = message\n self.error_code = error_code\n super(PredictionException, self).__init__(message)\n\n def __str__(self):\n return \"{message} : {error_code}\".format(message=self.message, error_code=self.error_code)\n\n\ndef emit_metrics(metrics):\n \"\"\"\n Emit the metrics in the provided Dictionary\n\n Parameters\n ----------\n metrics: Dictionary\n A dictionary of all metrics, when key is metric_name\n value is a metric object\n \"\"\"\n if metrics:\n for met in metrics:\n logger.info(\"[METRICS]%s\", str(met))\n" }, { "path": "mms/tests/README.md", "content": "# Testing MMS\n\n## Pre-requisites\n\nYou will need some additional Python modules to run the unit tests and linting.\n\n```bash\npip install mock pytest pylint\n```\n\nYou will also need the source for the project, so clone the project first.\n\n```bash\ngit clone https://github.com/awslabs/multi-model-server.git\ncd multi-model-server\n```\n\n## Unit Tests\n\nYou can run the unit tests with the following:\n\n```bash\npython -m pytest mms/tests/unit_tests/\n```\n\nTo get the coverage report of unit tests, you can run :\n\n```bash\npython -m pytest --cov-report term-missing --cov=mms/ mms/tests/unit_tests/\n```\n\nor:\n\n```bash\npython -m pytest --cov-report html:htmlcov --cov=mms/ mms/tests/unit_tests/\n```\n\n## Lint test\n\nYou can run the lint tests with the following:\n\n```bash\npylint -rn --rcfile=./mms/tests/pylintrc mms/.\n```\n" }, { "path": "mms/tests/pylintrc", "content": "[MASTER]\n\n# Licensed to the Apache Software Foundation (ASF) under one\n# or more contributor license agreements. See the NOTICE file\n# distributed with this work for additional information\n# regarding copyright ownership. The ASF licenses this file\n# to you under the Apache License, Version 2.0 (the\n# \"License\"); you may not use this file except in compliance\n# with the License. You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing,\n# software distributed under the License is distributed on an\n# \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n# KIND, either express or implied. See the License for the\n# specific language governing permissions and limitations\n# under the License.\n\n# Specify a configuration file.\n#rcfile=\n\n# Python code to execute, usually for sys.path manipulation such as\n# pygtk.require().\n#init-hook=\n\n# Add files or directories to the blacklist. They should be base names, not\n# paths.\nignore=CVS\n\n# Add files or directories matching the regex patterns to the blacklist. The\n# regex matches against base names, not paths.\nignore-patterns=\n\n# Pickle collected data for later comparisons.\npersistent=yes\n\n# List of plugins (as comma separated values of python modules names) to load,\n# usually to register additional checkers.\nload-plugins=\n\n# Use multiple processes to speed up Pylint.\njobs=8\n\n# Allow loading of arbitrary C extensions. Extensions are imported into the\n# active Python interpreter and may run arbitrary code.\nunsafe-load-any-extension=no\n\n# A comma-separated list of package or module names from where C extensions may\n# be loaded. Extensions are loading into the active Python interpreter and may\n# run arbitrary code\nextension-pkg-whitelist=numpy,opencv\n\n# Allow optimization of some AST trees. This will activate a peephole AST\n# optimizer, which will apply various small optimizations. For instance, it can\n# be used to obtain the result of joining multiple strings with the addition\n# operator. Joining a lot of strings can lead to a maximum recursion error in\n# Pylint and this flag can prevent that. It has one side effect, the resulting\n# AST will be different than the one from reality. This option is deprecated\n# and it will be removed in Pylint 2.0.\noptimize-ast=no\n\n\n[MESSAGES CONTROL]\n\n# Only show warnings with the listed confidence levels. Leave empty to show\n# all. Valid levels: HIGH, INFERENCE, INFERENCE_FAILURE, UNDEFINED\nconfidence=\n\n# Enable the message, report, category or checker with the given id(s). You can\n# either give multiple identifier separated by comma (,) or put this option\n# multiple time (only on the command line, not in the configuration file where\n# it should appear only once). See also the \"--disable\" option for examples.\nenable=indexing-exception,old-raise-syntax\n\n# Disable the message, report, category or checker with the given id(s). You\n# can either give multiple identifiers separated by comma (,) or put this\n# option multiple times (only on the command line, not in the configuration\n# file where it should appear only once).You can also use \"--disable=all\" to\n# disable everything first and then reenable specific checks. For example, if\n# you want to run only the similarities checker, you can use \"--disable=all\n# --enable=similarities\". If you want to run only the classes checker, but have\n# no Warning level messages displayed, use\"--disable=all --enable=classes\n# --disable=W\"\ndisable=design,similarities,no-self-use,attribute-defined-outside-init,locally-disabled,star-args,pointless-except,bad-option-value,global-statement,fixme,suppressed-message,useless-suppression,locally-enabled,no-member,no-name-in-module,import-error,unsubscriptable-object,unbalanced-tuple-unpacking,undefined-variable,protected-access,superfluous-parens,invalid-name,no-else-return,useless-super-delegation,len-as-condition,invalid-unary-operand-type,useless-object-inheritance\n# disable=unicode-builtin,delslice-method,using-cmp-argument,setslice-method,dict-view-method,parameter-unpacking,range-builtin-not-iterating,print-statement,file-builtin,old-raise-syntax,basestring-builtin,execfile-builtin,indexing-exception,import-star-module-level,coerce-method,long-builtin,old-ne-operator,old-division,no-absolute-import,raw_input-builtin,old-octal-literal,oct-method,xrange-builtin,hex-method,unpacking-in-except,nonzero-method,raising-string,intern-builtin,reload-builtin,metaclass-assignment,cmp-method,filter-builtin-not-iterating,apply-builtin,map-builtin-not-iterating,next-method-called,unichr-builtin,buffer-builtin,dict-iter-method,input-builtin,coerce-builtin,getslice-method,useless-suppression,standarderror-builtin,zip-builtin-not-iterating,suppressed-message,cmp-builtin,backtick,long-suffix,reduce-builtin,round-builtin\n\n\n[REPORTS]\n\n# Set the output format. Available formats are text, parseable, colorized, msvs\n# (visual studio) and html. You can also give a reporter class, eg\n# mypackage.mymodule.MyReporterClass.\noutput-format=text\n\n# Put messages in a separate file for each module / package specified on the\n# command line instead of printing them on stdout. Reports (if any) will be\n# written in a file name \"pylint_global.[txt|html]\". This option is deprecated\n# and it will be removed in Pylint 2.0.\nfiles-output=no\n\n# Tells whether to display a full report or only the messages\nreports=no\n\n# Python expression which should return a note less than 10 (10 is the highest\n# note). You have access to the variables errors warning, statement which\n# respectively contain the number of errors / warnings messages and the total\n# number of statements analyzed. This is used by the global evaluation report\n# (RP0004).\nevaluation=10.0 - ((float(5 * error + warning + refactor + convention) / statement) * 10)\n\n# Template used to display messages. This is a python new-style format string\n# used to format the message information. See doc for all details\n#msg-template=\n\n\n[FORMAT]\n\n# Maximum number of characters on a single line.\nmax-line-length=120\n\n# Regexp for a line that is allowed to be longer than the limit.\nignore-long-lines=^\\s*(# )??$\n\n# Allow the body of an if to be on the same line as the test if there is no\n# else.\nsingle-line-if-stmt=no\n\n# List of optional constructs for which whitespace checking is disabled. `dict-\n# separator` is used to allow tabulation in dicts, etc.: {1 : 1,\\n222: 2}.\n# `trailing-comma` allows a space between comma and closing bracket: (a, ).\n# `empty-line` allows space-only lines.\nno-space-check=trailing-comma,dict-separator\n\n# Maximum number of lines in a module\nmax-module-lines=1000\n\n# String used as indentation unit. This is usually \" \" (4 spaces) or \"\\t\" (1\n# tab).\nindent-string=' '\n\n# Number of spaces of indent required inside a hanging or continued line.\nindent-after-paren=4\n\n# Expected format of line ending, e.g. empty (any line ending), LF or CRLF.\nexpected-line-ending-format=\n\n\n[SPELLING]\n\n# Spelling dictionary name. Available dictionaries: none. To make it working\n# install python-enchant package.\nspelling-dict=\n\n# List of comma separated words that should not be checked.\nspelling-ignore-words=\n\n# A path to a file that contains private dictionary; one word per line.\nspelling-private-dict-file=\n\n# Tells whether to store unknown words to indicated private dictionary in\n# --spelling-private-dict-file option instead of raising a message.\nspelling-store-unknown-words=no\n\n\n[MISCELLANEOUS]\n\n# List of note tags to take in consideration, separated by a comma.\nnotes=FIXME,XXX,TODO\n\n\n[TYPECHECK]\n\n# Tells whether missing members accessed in mixin class should be ignored. A\n# mixin class is detected if its name ends with \"mixin\" (case insensitive).\nignore-mixin-members=yes\n\n# List of module names for which member attributes should not be checked\n# (useful for modules/projects where namespaces are manipulated during runtime\n# and thus existing member attributes cannot be deduced by static analysis. It\n# supports qualified module names, as well as Unix pattern matching.\nignored-modules=\n\n# List of class names for which member attributes should not be checked (useful\n# for classes with dynamically set attributes). This supports the use of\n# qualified names.\nignored-classes=optparse.Values,thread._local,_thread._local\n\n# List of members which are set dynamically and missed by pylint inference\n# system, and so shouldn't trigger E1101 when accessed. Python regular\n# expressions are accepted.\ngenerated-members=\n\n# List of decorators that produce context managers, such as\n# contextlib.contextmanager. Add to this list to register other decorators that\n# produce valid context managers.\ncontextmanager-decorators=contextlib.contextmanager\n\n\n[LOGGING]\n\n# Logging modules to check that the string format arguments are in logging\n# function parameter format\nlogging-modules=logging\n\n\n[SIMILARITIES]\n\n# Minimum lines number of a similarity.\nmin-similarity-lines=4\n\n# Ignore comments when computing similarities.\nignore-comments=yes\n\n# Ignore docstrings when computing similarities.\nignore-docstrings=yes\n\n# Ignore imports when computing similarities.\nignore-imports=no\n\n\n[VARIABLES]\n\n# Tells whether we should check for unused import in __init__ files.\ninit-import=no\n\n# A regular expression matching the name of dummy variables (i.e. expectedly\n# not used).\ndummy-variables-rgx=(_+[a-zA-Z0-9]*?$)|dummy\n\n# List of additional names supposed to be defined in builtins. Remember that\n# you should avoid to define new builtins when possible.\nadditional-builtins=\n\n# List of strings which can identify a callback function by name. A callback\n# name must start or end with one of those strings.\ncallbacks=cb_,_cb\n\n# List of qualified module names which can have objects that can redefine\n# builtins.\nredefining-builtins-modules=six.moves,future.builtins,builtins\n\n\n[BASIC]\n\n# Good variable names which should always be accepted, separated by a comma\ngood-names=i,j,_,a,b,op,x,y,wd,lr,kv,k,v,s,p,h,c,m,n,X,t,g,f\n\n# Bad variable names which should always be refused, separated by a comma\nbad-names=\n\n# Colon-delimited sets of names that determine each other's naming style when\n# the name regexes allow several styles.\nname-group=\n\n# Include a hint for the correct naming format with invalid-name\ninclude-naming-hint=no\n\n# List of decorators that produce properties, such as abc.abstractproperty. Add\n# to this list to register other decorators that produce valid properties.\nproperty-classes=abc.abstractproperty\n\n# Regular expression matching correct module names\nmodule-rgx=(([a-z_][a-z0-9_]*)|([A-Z][a-zA-Z0-9]+))$\n\n# Naming hint for module names\nmodule-name-hint=(([a-z_][a-z0-9_]*)|([A-Z][a-zA-Z0-9]+))$\n\n# Regular expression matching correct constant names\nconst-rgx=(([A-Z_][A-Z0-9_]*)|(__.*__))$\n\n# Naming hint for constant names\nconst-name-hint=(([A-Z_][A-Z0-9_]*)|(__.*__))$\n\n# Regular expression matching correct inline iteration names\ninlinevar-rgx=[A-Za-z_][A-Za-z0-9_]*$\n\n# Naming hint for inline iteration names\ninlinevar-name-hint=[A-Za-z_][A-Za-z0-9_]*$\n\n# Regular expression matching correct method names\nmethod-rgx=[a-z_][a-z0-9_]{2,30}$\n\n# Naming hint for method names\nmethod-name-hint=[a-z_][a-z0-9_]{2,30}$\n\n# Regular expression matching correct class attribute names\nclass-attribute-rgx=([A-Za-z_][A-Za-z0-9_]{2,30}|(__.*__))$\n\n# Naming hint for class attribute names\nclass-attribute-name-hint=([A-Za-z_][A-Za-z0-9_]{2,30}|(__.*__))$\n\n# Regular expression matching correct argument names\nargument-rgx=[a-z_][a-z0-9_]{2,30}$\n\n# Naming hint for argument names\nargument-name-hint=[a-z_][a-z0-9_]{2,30}$\n\n# Regular expression matching correct attribute names\nattr-rgx=[a-z_][a-z0-9_]{2,30}$\n\n# Naming hint for attribute names\nattr-name-hint=[a-z_][a-z0-9_]{2,30}$\n\n# Regular expression matching correct variable names\nvariable-rgx=[a-z_][a-z0-9_]{2,30}$\n\n# Naming hint for variable names\nvariable-name-hint=[a-z_][a-z0-9_]{2,30}$\n\n# Regular expression matching correct function names\nfunction-rgx=[a-z_][a-z0-9_]{2,30}$\n\n# Naming hint for function names\nfunction-name-hint=[a-z_][a-z0-9_]{2,30}$\n\n# Regular expression matching correct class names\nclass-rgx=[A-Z_][a-zA-Z0-9]+$\n\n# Naming hint for class names\nclass-name-hint=[A-Z_][a-zA-Z0-9]+$\n\n# Regular expression which should only match function or class names that do\n# not require a docstring.\nno-docstring-rgx=^_\n\n# Minimum line length for functions/classes that require docstrings, shorter\n# ones are exempt.\ndocstring-min-length=10\n\n\n[ELIF]\n\n# Maximum number of nested blocks for function / method body\nmax-nested-blocks=5\n\n\n[CLASSES]\n\n# List of method names used to declare (i.e. assign) instance attributes.\ndefining-attr-methods=__init__,__new__,setUp\n\n# List of valid names for the first argument in a class method.\nvalid-classmethod-first-arg=cls\n\n# List of valid names for the first argument in a metaclass class method.\nvalid-metaclass-classmethod-first-arg=mcs\n\n# List of member names, which should be excluded from the protected access\n# warning.\nexclude-protected=_asdict,_fields,_replace,_source,_make\n\n\n[IMPORTS]\n\n# Deprecated modules which should not be used, separated by a comma\ndeprecated-modules=optparse\n\n# Create a graph of every (i.e. internal and external) dependencies in the\n# given file (report RP0402 must not be disabled)\nimport-graph=\n\n# Create a graph of external dependencies in the given file (report RP0402 must\n# not be disabled)\next-import-graph=\n\n# Create a graph of internal dependencies in the given file (report RP0402 must\n# not be disabled)\nint-import-graph=\n\n# Force import order to recognize a module as part of the standard\n# compatibility libraries.\nknown-standard-library=\n\n# Force import order to recognize a module as part of a third party library.\nknown-third-party=enchant\n\n# Analyse import fallback blocks. This can be used to support both Python 2 and\n# 3 compatible code, which means that the block might have code that exists\n# only in one or another interpreter, leading to false positives when analysed.\nanalyse-fallback-blocks=no\n\n\n[DESIGN]\n\n# Maximum number of arguments for function / method\nmax-args=5\n\n# Argument names that match this expression will be ignored. Default to name\n# with leading underscore\nignored-argument-names=_.*\n\n# Maximum number of locals for function / method body\nmax-locals=15\n\n# Maximum number of return / yield for function / method body\nmax-returns=6\n\n# Maximum number of branch for function / method body\nmax-branches=12\n\n# Maximum number of statements in function / method body\nmax-statements=50\n\n# Maximum number of parents for a class (see R0901).\nmax-parents=7\n\n# Maximum number of attributes for a class (see R0902).\nmax-attributes=7\n\n# Minimum number of public methods for a class (see R0903).\nmin-public-methods=2\n\n# Maximum number of public methods for a class (see R0904).\nmax-public-methods=20\n\n# Maximum number of boolean expressions in a if statement\nmax-bool-expr=5\n\n\n[EXCEPTIONS]\n\n# Exceptions that will emit a warning when being caught. Defaults to\n# \"Exception\"\novergeneral-exceptions=Exception\n" }, { "path": "mms/tests/unit_tests/helper/__init__.py", "content": "# Copyright 2017 Amazon.com, Inc. or its affiliates. All Rights Reserved.\n# Licensed under the Apache License, Version 2.0 (the \"License\").\n# You may not use this file except in compliance with the License.\n# A copy of the License is located at\n# http://www.apache.org/licenses/LICENSE-2.0\n# or in the \"license\" file accompanying this file. This file is distributed\n# on an \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either\n# express or implied. See the License for the specific language governing\n# permissions and limitations under the License.\n\n" }, { "path": "mms/tests/unit_tests/helper/pixel2pixel_service.py", "content": "# Copyright 2017 Amazon.com, Inc. or its affiliates. All Rights Reserved.\n# Licensed under the Apache License, Version 2.0 (the \"License\").\n# You may not use this file except in compliance with the License.\n# A copy of the License is located at\n# http://www.apache.org/licenses/LICENSE-2.0\n# or in the \"license\" file accompanying this file. This file is distributed\n# on an \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either\n# express or implied. See the License for the specific language governing\n# permissions and limitations under the License.\n\nimport mxnet as mx\nimport numpy as np\nimport sys\nsys.path.append('../../..')\n\nfrom mms.model_service.mxnet_model_service import MXNetBaseService, check_input_shape\nfrom mms.utils.mxnet import image\nfrom mxnet import ndarray as nd\nfrom mxnet.gluon.nn import Dense, Activation, Conv2D, Conv2DTranspose, \\\n BatchNorm, LeakyReLU, Flatten, HybridSequential, HybridBlock, Dropout\n\n\n# Define Unet generator skip block\nclass UnetSkipUnit(HybridBlock):\n def __init__(self, inner_channels, outer_channels, inner_block=None, innermost=False, outermost=False,\n use_dropout=False, use_bias=False):\n super(UnetSkipUnit, self).__init__()\n\n with self.name_scope():\n self.outermost = outermost\n en_conv = Conv2D(channels=inner_channels, kernel_size=4, strides=2, padding=1,\n in_channels=outer_channels, use_bias=use_bias)\n en_relu = LeakyReLU(alpha=0.2)\n en_norm = BatchNorm(momentum=0.1, in_channels=inner_channels)\n de_relu = Activation(activation='relu')\n de_norm = BatchNorm(momentum=0.1, in_channels=outer_channels)\n\n if innermost:\n de_conv = Conv2DTranspose(channels=outer_channels, kernel_size=4, strides=2, padding=1,\n in_channels=inner_channels, use_bias=use_bias)\n encoder = [en_relu, en_conv]\n decoder = [de_relu, de_conv, de_norm]\n model = encoder + decoder\n elif outermost:\n de_conv = Conv2DTranspose(channels=outer_channels, kernel_size=4, strides=2, padding=1,\n in_channels=inner_channels * 2)\n encoder = [en_conv]\n decoder = [de_relu, de_conv, Activation(activation='tanh')]\n model = encoder + [inner_block] + decoder\n else:\n de_conv = Conv2DTranspose(channels=outer_channels, kernel_size=4, strides=2, padding=1,\n in_channels=inner_channels * 2, use_bias=use_bias)\n encoder = [en_relu, en_conv, en_norm]\n decoder = [de_relu, de_conv, de_norm]\n model = encoder + [inner_block] + decoder\n if use_dropout:\n model += [Dropout(rate=0.5)]\n\n self.model = HybridSequential()\n with self.model.name_scope():\n for block in model:\n self.model.add(block)\n\n def hybrid_forward(self, F, x):\n if self.outermost:\n return self.model(x)\n else:\n return F.concat(self.model(x), x, dim=1)\n\n\n# Define Unet generator\nclass UnetGenerator(HybridBlock):\n def __init__(self, in_channels, num_downs, ngf=64, use_dropout=True):\n super(UnetGenerator, self).__init__()\n\n # Build unet generator structure\n with self.name_scope():\n unet = UnetSkipUnit(ngf * 8, ngf * 8, innermost=True)\n for _ in range(num_downs - 5):\n unet = UnetSkipUnit(ngf * 8, ngf * 8, unet, use_dropout=use_dropout)\n unet = UnetSkipUnit(ngf * 8, ngf * 4, unet)\n unet = UnetSkipUnit(ngf * 4, ngf * 2, unet)\n unet = UnetSkipUnit(ngf * 2, ngf * 1, unet)\n unet = UnetSkipUnit(ngf, in_channels, unet, outermost=True)\n\n self.model = unet\n\n def hybrid_forward(self, F, x):\n return self.model(x)\n\nclass Pixel2pixelService(MXNetBaseService):\n\n def __init__(self, model_name, path):\n self.mx_model = UnetGenerator(in_channels=3, num_downs=8)\n self.mx_model.load_params('%s/%s.params' % (path, model_name), ctx=mx.cpu())\n\n def _preprocess(self, data):\n input_shape = self.signature['inputs'][0]['data_shape']\n height, width = input_shape[2:]\n img_arr = image.read(data[0])\n img_arr = image.resize(img_arr, width, height)\n img_arr = image.color_normalize(img_arr, nd.array([127.5]), nd.array([127.5]))\n img_arr = image.transform_shape(img_arr)\n return [img_arr]\n\n def _inference(self, data):\n check_input_shape(data, self.signature)\n return self.mx_model(*data)\n\n def _postprocess(self, data):\n img_arr = ((data[0] + 1.0) * 127.5).astype(np.uint8)\n return [image.write(img_arr)]\n\n\n\n\n" }, { "path": "mms/tests/unit_tests/model_service/dummy_model/MANIFEST.json", "content": "{\n \"Engine\": {\n \"MXNet\": 0.12\n },\n \"Model-Archive-Description\": \"dummy\",\n \"License\": \"Apache 2.0\",\n \"Model-Archive-Version\": 0.1,\n \"Model-Server\": 0.1,\n \"Model\": {\n \"Description\": \"dummy model\",\n \"Service\": \"dummy_model_service.py\",\n \"Model-Name\": \"dummy\",\n }\n}" }, { "path": "mms/tests/unit_tests/model_service/dummy_model/dummy_model_service.py", "content": "# Copyright 2018 Amazon.com, Inc. or its affiliates. All Rights Reserved.\n# Licensed under the Apache License, Version 2.0 (the \"License\").\n# You may not use this file except in compliance with the License.\n# A copy of the License is located at\n# http://www.apache.org/licenses/LICENSE-2.0\n# or in the \"license\" file accompanying this file. This file is distributed\n# on an \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either\n# express or implied. See the License for the specific language governing\n# permissions and limitations under the License.\n\nfrom mms.model_service.model_service import SingleNodeService\n\n\"\"\"\nThis file is a dummy file for the purpose of unit-testing test_service_manager.py\n\"\"\"\n\n\nclass DummyNodeService(SingleNodeService):\n def _inference(self, data):\n pass\n\n def signature(self):\n pass\n\n def ping(self):\n pass\n\n def inference(self):\n pass\n\n\nclass SomeOtherClass:\n def __init__(self):\n pass\n" }, { "path": "mms/tests/unit_tests/model_service/test_mxnet_image.py", "content": "# Copyright 2017 Amazon.com, Inc. or its affiliates. All Rights Reserved.\n# Licensed under the Apache License, Version 2.0 (the \"License\").\n # You may not use this file except in compliance with the License.\n# A copy of the License is located at\n# http://www.apache.org/licenses/LICENSE-2.0\n# or in the \"license\" file accompanying this file. This file is distributed\n# on an \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either\n# express or implied. See the License for the specific language governing\n# permissions and limitations under the License.\n\nimport os\nimport sys\ncurr_path = os.path.dirname(os.path.abspath(__file__))\nsys.path.append(curr_path + '/../../..')\n\nimport PIL\nimport unittest\nimport numpy as np\nimport mxnet as mx\nimport mms.utils.mxnet.image as image\nfrom io import BytesIO\n\nclass TestMXNetImageUtils(unittest.TestCase):\n def _write_image(self, img_arr, flag=1):\n img_arr = mx.nd.transpose(img_arr, (1, 2, 0))\n mode = 'RGB' if flag == 1 else 'L'\n if flag == 0:\n img_arr = mx.nd.reshape(img_arr, shape=(img_arr.shape[0], img_arr.shape[1]))\n img_arr = img_arr.astype(np.uint8).asnumpy()\n image = PIL.Image.fromarray(img_arr, mode)\n output = BytesIO()\n image.save(output, format='jpeg')\n return output.getvalue()\n\n def test_transform_shape(self):\n input1 = mx.nd.random.uniform(0, 255, shape=(32, 32, 3))\n output1 = image.transform_shape(input1)\n assert output1.shape == (1, 3, 32, 32), \"transform_shape method fail. Got %s shape.\" % (str(output1.shape))\n\n input2 = mx.nd.random.uniform(0, 255, shape=(28, 28, 3))\n output2 = image.transform_shape(input2, dim_order='NHWC')\n assert output2.shape == (1, 28, 28, 3), \"transform_shape method fail. Got %s shape.\" % (str(output2.shape))\n\n def test_read(self):\n input1 = mx.nd.random.uniform(0, 255, shape=(3, 256, 256))\n input_buf1 = self._write_image(input1)\n output1 = image.read(input_buf1)\n assert output1.shape == (256, 256, 3), \"Read method failed. Got %s shape.\" % (str(output1.shape))\n\n input2 = mx.nd.random.uniform(0, 255, shape=(1, 128, 128))\n input_buf2 = self._write_image(input2, flag=0)\n output2 = image.read(input_buf2, flag=0)\n assert output2.shape == (128, 128, 1), \"Read method failed. Got %s shape.\" % (str(output2.shape))\n\n def test_write(self):\n input1 = mx.nd.random.uniform(0, 255, shape=(3, 256, 256))\n output1 = image.write(input1)\n assert isinstance(output1, str), \"Write method failed. Output is not a string.\"\n\n input2 = mx.nd.random.uniform(0, 255, shape=(256, 256, 1))\n output2 = image.write(input2, flag=0, dim_order='HWC')\n assert isinstance(output2, str), \"Write method failed. Output is not a string.\"\n\n def test_resize(self):\n input1 = mx.nd.random.uniform(0, 255, shape=(245, 156, 3))\n output1 = image.resize(input1, 128, 256)\n assert output1.shape == (256, 128, 3), \"Resize method failed. Got %s shape.\" % (str(output1.shape))\n\n def test_fix_crop(self):\n input1 = mx.nd.random.uniform(0, 255, shape=(100, 100, 3))\n output1 = image.fixed_crop(input1, 10, 20, 50, 70)\n assert output1.shape == (70, 50, 3), \"Resize method failed. Got %s shape.\" % (str(output1.shape))\n\n def test_color_normalize(self):\n input1 = mx.nd.random.uniform(0, 255, shape=(1, 10, 10))\n output1 = image.color_normalize(input1, 127.5, 127.5).asnumpy()\n assert (output1 >= -1.0).all() and (output1 <= 1.0).all(), \"color_normalize method failed.\"\n\n def runTest(self):\n self.test_transform_shape()\n self.test_read()\n self.test_write()\n self.test_resize()\n self.test_fix_crop()\n self.test_color_normalize()" }, { "path": "mms/tests/unit_tests/model_service/test_mxnet_ndarray.py", "content": "# Copyright 2017 Amazon.com, Inc. or its affiliates. All Rights Reserved.\n# Licensed under the Apache License, Version 2.0 (the \"License\").\n# You may not use this file except in compliance with the License.\n# A copy of the License is located at\n# http://www.apache.org/licenses/LICENSE-2.0\n# or in the \"license\" file accompanying this file. This file is distributed\n# on an \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either\n# express or implied. See the License for the specific language governing\n# permissions and limitations under the License.\n\nimport os\nimport sys\ncurr_path = os.path.dirname(os.path.abspath(__file__))\nsys.path.append(curr_path + '/../../..')\n\nimport unittest\nimport mxnet as mx\nimport utils.mxnet.ndarray as ndarray\n\nclass TestMXNetNDArrayUtils(unittest.TestCase):\n def test_top_prob(self):\n labels = ['dummay' for _ in range(100)]\n data = mx.nd.random.uniform(0, 1, shape=(1, 100))\n top = 13\n output = ndarray.top_probability(data, labels, top=top)\n assert len(output) == top, \"top_probability method failed.\"\n\n def runTest(self):\n self.test_top_prob()\n" }, { "path": "mms/tests/unit_tests/model_service/test_mxnet_nlp.py", "content": "# Copyright 2017 Amazon.com, Inc. or its affiliates. All Rights Reserved.\n# Licensed under the Apache License, Version 2.0 (the \"License\").\n# You may not use this file except in compliance with the License.\n# A copy of the License is located at\n# http://www.apache.org/licenses/LICENSE-2.0\n# or in the \"license\" file accompanying this file. This file is distributed\n# on an \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either\n# express or implied. See the License for the specific language governing\n# permissions and limitations under the License.\n\nimport os\nimport sys\ncurr_path = os.path.dirname(os.path.abspath(__file__))\nsys.path.append(curr_path + '/../../..')\n\nimport unittest\nimport utils.mxnet.nlp as nlp\n\nfrom random import randint\n\nclass TestMXNetNLPUtils(unittest.TestCase):\n def test_encode_sentence(self):\n vocab = {}\n sentence = []\n for i in range(100):\n vocab['word%d' % (i)] = i\n sen_vec = [0, 56, 8, 10]\n for i in sen_vec:\n sentence.append('word%d' % (i))\n res1, out1 = nlp.encode_sentences([sentence], vocab)\n assert res1[0] == sen_vec, \"encode_sentence method failed. \" \\\n \"Result vector invalid.\"\n assert len(out1) == len(vocab), \"encode_sentence method failed. \" \\\n \"Generated vocab incorrect.\"\n\n res2, out2 = nlp.encode_sentences([sentence])\n assert res2[0] == [i for i in range(len(sentence))], \\\n \"encode_sentence method failed. Result vector invalid.\"\n assert len(out2) == len(sentence) + 1, \"encode_sentence method failed. \" \\\n \"Generated vocab incorrect.\"\n\n def test_pad_sentence(self):\n buckets = [10, 20, 30, 40, 50, 60]\n for _ in range(5):\n sent_length = randint(1, 60)\n sentence = [i for i in range(sent_length)]\n databatch = nlp.pad_sentence(sentence, buckets)\n assert databatch.data[0].shape[1] in buckets, \"pad_sentence failed. Padded sentence has length %d.\" \\\n % (databatch.data[0].shape[1])\n\n\n" }, { "path": "mms/tests/unit_tests/model_service/test_service.py", "content": "# Copyright 2017 Amazon.com, Inc. or its affiliates. All Rights Reserved.\n# Licensed under the Apache License, Version 2.0 (the \"License\").\n# You may not use this file except in compliance with the License.\n# A copy of the License is located at\n# http://www.apache.org/licenses/LICENSE-2.0\n# or in the \"license\" file accompanying this file. This file is distributed\n# on an \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either\n# express or implied. See the License for the specific language governing\n# permissions and limitations under the License.\n\nimport json\nimport os\nimport shutil\nimport sys\nimport tempfile\nimport unittest\nfrom io import BytesIO\n\nimport PIL\nimport mxnet as mx\nimport numpy as np\nimport pytest\nfrom helper.pixel2pixel_service import UnetGenerator\nfrom mms.model_service.mxnet_model_service import MXNetBaseService, GluonImperativeBaseService\n\ncurr_path = os.path.dirname(os.path.abspath(__file__))\nsys.path.append(curr_path + '/../..')\n\n\ndef empty_file(path):\n open(path, 'a').close()\n\n\ndef module_dir(tmpdir):\n path = '{}/test'.format(tmpdir)\n os.mkdir(path)\n empty_file('{}/test-symbol.json'.format(path))\n empty_file('{}/test-0000.params'.format(path))\n empty_file('{}/synset.txt'.format(path))\n\n with open('{}/signature.json'.format(path), 'w') as sig:\n signature = {\n \"input_type\": \"image/jpeg\",\n \"inputs\": [\n {\n 'data_name': 'data1',\n 'data_shape': [1, 3, 64, 64]\n },\n {\n 'data_name': 'data2',\n 'data_shape': [1, 3, 32, 32]\n }\n ],\n \"output_type\": \"application/json\",\n \"outputs\": [\n {\n 'data_name': 'softmax',\n 'data_shape': [1, 10]\n }\n ]\n }\n json.dump(signature, sig)\n\n return path\n\n\ndef create_symbolic_manifest(path):\n with open('{}/MANIFEST.json'.format(path), 'w') as man:\n manifest = {\n \"Engine\": {\n \"MXNet\": 0.12\n },\n \"Model-Archive-Description\": \"test\",\n \"License\": \"Apache 2.0\",\n \"Model-Archive-Version\": 0.1,\n \"Model-Server\": 0.1,\n \"Model\": {\n \"Description\": \"test\",\n \"Service\": \"test\",\n \"Symbol\": \"\",\n \"Parameters\": \"test-0000.params\",\n \"Signature\": \"signature.json\",\n \"Model-Name\": \"test\",\n \"Model-Format\": \"MXNet-Symbolic\"\n }\n }\n json.dump(manifest, man)\n\n\ndef create_imperative_manifest(path):\n with open('{}/MANIFEST.json'.format(path), 'w') as man:\n manifest = {\n \"Engine\": {\n \"MXNet\": 0.12\n },\n \"Model-Archive-Description\": \"test\",\n \"License\": \"Apache 2.0\",\n \"Model-Archive-Version\": 0.1,\n \"Model-Server\": 0.1,\n \"Model\": {\n \"Description\": \"test\",\n \"Service\": \"test\",\n \"Symbol\": \"\",\n \"Parameters\": \"\",\n \"Signature\": \"signature.json\",\n \"Model-Name\": \"test\",\n \"Model-Format\": \"Gluon-Imperative\"\n }\n }\n json.dump(manifest, man)\n\n\nclass TestService(unittest.TestCase):\n def setUp(self):\n self.test_dir = tempfile.mkdtemp()\n\n def tearDown(self):\n shutil.rmtree(self.test_dir)\n\n def _train_and_export(self, path):\n model_path = curr_path + '/' + path\n if not os.path.isdir(model_path):\n os.mkdir(model_path)\n num_class = 10\n data1 = mx.sym.Variable('data1')\n data2 = mx.sym.Variable('data2')\n conv1 = mx.sym.Convolution(data=data1, kernel=(2, 2), num_filter=2, stride=(2, 2))\n conv2 = mx.sym.Convolution(data=data2, kernel=(3, 3), num_filter=3, stride=(1, 1))\n pooling1 = mx.sym.Pooling(data=conv1, kernel=(2, 2), stride=(1, 1), pool_type=\"avg\")\n pooling2 = mx.sym.Pooling(data=conv2, kernel=(2, 2), stride=(1, 1), pool_type=\"max\")\n flatten1 = mx.sym.flatten(data=pooling1)\n flatten2 = mx.sym.flatten(data=pooling2)\n summary = mx.sym.sum(data=flatten1, axis=1) + mx.sym.sum(data=flatten2, axis=1)\n fc = mx.sym.FullyConnected(data=summary, num_hidden=num_class)\n sym = mx.sym.SoftmaxOutput(data=fc, name='softmax')\n\n dshape1 = (10, 3, 64, 64)\n dshape2 = (10, 3, 32, 32)\n lshape = (10,)\n\n mod = mx.mod.Module(symbol=sym, data_names=('data1', 'data2'),\n label_names=('softmax_label',))\n mod.bind(data_shapes=[('data1', dshape1), ('data2', dshape2)],\n label_shapes=[('softmax_label', lshape)])\n mod.init_params()\n mod.init_optimizer(optimizer_params={'learning_rate': 0.01})\n\n data_batch = mx.io.DataBatch(data=[mx.nd.random.uniform(0, 9, dshape1),\n mx.nd.random.uniform(5, 15, dshape2)],\n label=[mx.nd.ones(lshape)])\n mod.forward(data_batch)\n mod.backward()\n mod.update()\n with open('%s/synset.txt' % model_path, 'w') as synset:\n for i in range(10):\n synset.write('test label %d\\n' % i)\n\n def _write_image(self, img_arr):\n img_arr = mx.nd.transpose(img_arr, (1, 2, 0)).astype(np.uint8).asnumpy()\n mode = 'RGB'\n image = PIL.Image.fromarray(img_arr, mode)\n output = BytesIO()\n image.save(output, format='jpeg')\n return output.getvalue()\n\n def test_vision_init(self):\n path = 'test'\n self._train_and_export(path)\n model_path = curr_path + '/' + path\n os.system('rm -rf %s' % model_path)\n\n def test_vision_inference(self):\n path = 'test'\n self._train_and_export(path)\n\n os.system('rm -rf %s/test' % curr_path)\n\n def test_gluon_inference(self):\n path = 'gluon'\n model_name = 'gluon1'\n model_path = curr_path + '/' + path\n os.mkdir(model_path)\n ctx = mx.cpu()\n net_g = UnetGenerator(in_channels=3, num_downs=8)\n data = mx.nd.random_uniform(0, 255, shape=(1, 3, 256, 256))\n net_g.initialize(mx.init.Normal(0.02), ctx=ctx)\n net_g(data)\n net_g.save_params('%s/%s.params' % (model_path, model_name))\n with open('%s/signature.json' % model_path, 'w') as sig:\n signature = {\n \"input_type\": \"image/jpeg\",\n \"inputs\": [\n {\n 'data_name': 'data',\n 'data_shape': [1, 3, 256, 256]\n },\n ],\n \"output_type\": \"image/jpeg\",\n \"outputs\": [\n {\n 'data_name': 'output',\n 'data_shape': [1, 3, 256, 256]\n }\n ]\n }\n json.dump(signature, sig)\n\n cmd = 'python %s/../../export_model.py --model-name %s --model-path %s' \\\n % (curr_path, model_name, model_path)\n os.system(cmd)\n\n os.system('rm -rf %s %s/%s.model %s/%s' % (model_path, os.getcwd(),\n model_name, os.getcwd(), model_name))\n\n def test_mxnet_model_service(self):\n mod_dir = module_dir(self.test_dir)\n if mod_dir.startswith('~'):\n model_path = os.path.expanduser(mod_dir)\n else:\n model_path = mod_dir\n create_symbolic_manifest(model_path)\n manifest = json.load(open(os.path.join(model_path, 'MANIFEST.json')))\n with pytest.raises(Exception):\n MXNetBaseService('test', model_path, manifest)\n os.system('rm -rf %s' % model_path)\n\n def test_gluon_model_service(self):\n mod_dir = module_dir(self.test_dir)\n if mod_dir.startswith('~'):\n model_path = os.path.expanduser(mod_dir)\n else:\n model_path = mod_dir\n create_imperative_manifest(model_path)\n manifest = json.load(open(os.path.join(model_path, 'MANIFEST.json')))\n GluonImperativeBaseService('test', model_path, manifest,\n mx.gluon.model_zoo.vision.alexnet(pretrained=True))\n os.system('rm -rf %s' % model_path)\n\n def runTest(self):\n self.test_vision_init()\n self.test_vision_inference()\n self.test_gluon_inference()\n self.test_mxnet_model_service()\n self.test_gluon_model_service()\n self.test_incorrect_service()\n" }, { "path": "mms/tests/unit_tests/test_beckend_metric.py", "content": "import logging\nimport sys\n\nimport pytest\nfrom mms.metrics.dimension import Dimension\nfrom mms.metrics.metrics_store import MetricsStore\nfrom mms.service import emit_metrics\n\nlogging.basicConfig(stream=sys.stdout, format=\"%(message)s\", level=logging.INFO)\n\n\ndef get_model_key(name, unit, req_id, model_name):\n dimensions = list()\n dimensions.append(Dimension(\"ModelName\", model_name))\n dimensions.append(Dimension(\"Level\", \"Model\"))\n dim_str = [name, unit, str(req_id)] + [str(d) for d in dimensions]\n return '-'.join(dim_str)\n\n\ndef get_error_key(name, unit):\n dimensions = list()\n dimensions.append(Dimension(\"Level\", \"Error\"))\n dim_str = [name, unit, 'None'] + [str(d) for d in dimensions]\n return '-'.join(dim_str)\n\n\ndef test_metrics(caplog):\n \"\"\"\n Test if metric classes methods behave as expected\n Also checks global metric service methods\n \"\"\"\n caplog.set_level(logging.INFO)\n # Create a batch of request ids\n request_ids = {0: 'abcd', 1: \"xyz\", 2: \"qwerty\", 3: \"hjshfj\"}\n all_req_ids = ','.join(request_ids.values())\n model_name = \"dummy model\"\n\n # Create a metrics objects\n metrics = MetricsStore(request_ids, model_name)\n\n # Counter tests\n metrics.add_counter('CorrectCounter', 1, 1)\n test_metric = metrics.cache[get_model_key('CorrectCounter', 'count', 'xyz', model_name)]\n assert 'CorrectCounter' == test_metric.name\n metrics.add_counter('CorrectCounter', 1, 1)\n metrics.add_counter('CorrectCounter', 1, 3)\n metrics.add_counter('CorrectCounter', 1)\n test_metric = metrics.cache[get_model_key('CorrectCounter', 'count', all_req_ids, model_name)]\n assert 'CorrectCounter' == test_metric.name\n metrics.add_counter('CorrectCounter', 3)\n test_metric = metrics.cache[get_model_key('CorrectCounter', 'count', 'xyz', model_name)]\n assert test_metric.value == 2\n test_metric = metrics.cache[get_model_key('CorrectCounter', 'count', 'hjshfj', model_name)]\n assert test_metric.value == 1\n test_metric = metrics.cache[get_model_key('CorrectCounter', 'count', all_req_ids, model_name)]\n assert test_metric.value == 4\n # Check what is emitted is correct\n emit_metrics(metrics.store)\n\n assert \"hjshfj\" in caplog.text\n assert \"ModelName:dummy model\" in caplog.text\n\n # Adding other types of metrics\n # Check for time metric\n with pytest.raises(Exception) as e_info:\n metrics.add_time('WrongTime', 20, 1, 'ns')\n assert \"the unit for a timed metric should be one of ['ms', 's']\" == e_info.value.args[0]\n\n metrics.add_time('CorrectTime', 20, 2, 's')\n metrics.add_time('CorrectTime', 20, 0)\n test_metric = metrics.cache[get_model_key('CorrectTime', 'ms', 'abcd', model_name)]\n assert test_metric.value == 20\n assert test_metric.unit == 'Milliseconds'\n test_metric = metrics.cache[get_model_key('CorrectTime', 's', 'qwerty', model_name)]\n assert test_metric.value == 20\n assert test_metric.unit == 'Seconds'\n # Size based metrics\n with pytest.raises(Exception) as e_info:\n metrics.add_size('WrongSize', 20, 1, 'TB')\n assert \"The unit for size based metric is one of ['MB','kB', 'GB', 'B']\" == e_info.value.args[0]\n\n metrics.add_size('CorrectSize', 200, 0, 'GB')\n metrics.add_size('CorrectSize', 10, 2)\n test_metric = metrics.cache[get_model_key('CorrectSize', 'GB', 'abcd', model_name)]\n assert test_metric.value == 200\n assert test_metric.unit == 'Gigabytes'\n test_metric = metrics.cache[get_model_key('CorrectSize', 'MB', 'qwerty', model_name)]\n assert test_metric.value == 10\n assert test_metric.unit == 'Megabytes'\n\n # Check a percentage metric\n metrics.add_percent('CorrectPercent', 20.0, 3)\n test_metric = metrics.cache[get_model_key('CorrectPercent', 'percent', 'hjshfj', model_name)]\n assert test_metric.value == 20.0\n assert test_metric.unit == 'Percent'\n\n # Check a error metric\n metrics.add_error('CorrectError', 'Wrong values')\n test_metric = metrics.cache[get_error_key('CorrectError', '')]\n assert test_metric.value == 'Wrong values'\n" }, { "path": "mms/tests/unit_tests/test_model_loader.py", "content": "# Copyright 2018 Amazon.com, Inc. or its affiliates. All Rights Reserved.\n# Licensed under the Apache License, Version 2.0 (the \"License\").\n# You may not use this file except in compliance with the License.\n# A copy of the License is located at\n# http://www.apache.org/licenses/LICENSE-2.0\n# or in the \"license\" file accompanying this file. This file is distributed\n# on an \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either\n# express or implied. See the License for the specific language governing\n# permissions and limitations under the License.\n\nimport importlib\nimport inspect\nimport os\nimport sys\nimport types\nfrom collections import namedtuple\n\nimport mock\nimport pytest\n\nfrom mms.model_loader import LegacyModelLoader\nfrom mms.model_loader import MmsModelLoader\nfrom mms.model_loader import ModelLoaderFactory\nfrom mms.model_service.model_service import SingleNodeService\n\n\n# noinspection PyClassHasNoInit\n# @pytest.mark.skip(reason=\"Disabling it currently until the PR #467 gets merged\")\nclass TestModelFactory:\n\n def test_model_loader_factory_legacy(self):\n model_loader = ModelLoaderFactory.get_model_loader(\n os.path.abspath('mms/tests/unit_tests/model_service/dummy_model'))\n\n assert isinstance(model_loader, LegacyModelLoader)\n\n def test_model_loader_factory(self):\n model_loader = ModelLoaderFactory.get_model_loader(\n os.path.abspath('mms/tests/unit_tests/test_utils/'))\n\n assert isinstance(model_loader, MmsModelLoader)\n\n\n# noinspection PyClassHasNoInit\nclass TestListModels:\n\n def test_list_models_legacy(self):\n model_loader = ModelLoaderFactory.get_model_loader(\"legacy_mms\")\n sys.path.append(os.path.abspath('mms/tests/unit_tests/model_service/dummy_model'))\n module = importlib.import_module('dummy_model_service')\n classes = model_loader.list_model_services(module, SingleNodeService)\n assert len(classes) == 1\n assert issubclass(classes[0], SingleNodeService)\n\n def test_list_models(self):\n model_loader = ModelLoaderFactory.get_model_loader(\"mms\")\n sys.path.append(os.path.abspath('mms/tests/unit_tests/test_utils/'))\n module = importlib.import_module('dummy_class_model_service')\n classes = model_loader.list_model_services(module)\n assert len(classes) == 1\n assert classes[0].__name__ == 'CustomService'\n\n\n# noinspection PyProtectedMember\n# noinspection PyClassHasNoInit\nclass TestLoadModels:\n model_name = 'testmodel'\n model_dir = os.path.abspath('mms/tests/unit_tests/model_service/dummy_model')\n mock_manifest = '{\"Model\":{\"Service\":\"dummy_class_model_service.py\",' \\\n '\"Signature\":\"signature.json\",\"Model-Name\":\"testmodel\"}}'\n\n @pytest.fixture()\n def patches(self, mocker):\n Patches = namedtuple('Patches', ['mock_open', 'os_path', \"is_file\", \"open_signature\"])\n patches = Patches(\n mocker.patch('mms.model_loader.open'),\n mocker.patch('os.path.exists'),\n mocker.patch('os.path.isfile'),\n mocker.patch('mms.model_service.model_service.open')\n )\n return patches\n\n def test_load_model_legacy(self, patches):\n patches.mock_open.side_effect = [mock.mock_open(read_data=self.mock_manifest).return_value]\n patches.open_signature.side_effect = [mock.mock_open(read_data='{}').return_value]\n patches.is_file.return_value = True\n patches.os_path.side_effect = [False, True]\n sys.path.append(self.model_dir)\n handler = 'dummy_model_service'\n model_loader = ModelLoaderFactory.get_model_loader(self.model_dir)\n assert isinstance(model_loader, LegacyModelLoader)\n service = model_loader.load(self.model_name, self.model_dir, handler, 0, 1)\n\n assert inspect.ismethod(service._entry_point)\n\n def test_load_class_model(self, patches):\n patches.mock_open.side_effect = [mock.mock_open(read_data=self.mock_manifest).return_value]\n sys.path.append(os.path.abspath('mms/tests/unit_tests/test_utils/'))\n patches.os_path.return_value = True\n handler = 'dummy_class_model_service'\n model_loader = ModelLoaderFactory.get_model_loader(os.path.abspath('mms/unit_tests/test_utils/'))\n service = model_loader.load(self.model_name, self.model_dir, handler, 0, 1)\n\n assert inspect.ismethod(service._entry_point)\n\n def test_load_func_model(self, patches):\n patches.mock_open.side_effect = [mock.mock_open(read_data=self.mock_manifest).return_value]\n sys.path.append(os.path.abspath('mms/tests/unit_tests/test_utils/'))\n patches.os_path.return_value = True\n handler = 'dummy_func_model_service:infer'\n model_loader = ModelLoaderFactory.get_model_loader(os.path.abspath('mms/unit_tests/test_utils/'))\n service = model_loader.load(self.model_name, self.model_dir, handler, 0, 1)\n\n assert isinstance(service._entry_point, types.FunctionType)\n assert service._entry_point.__name__ == 'infer'\n\n def test_load_func_model_with_error(self, patches):\n patches.mock_open.side_effect = [mock.mock_open(read_data=self.mock_manifest).return_value]\n sys.path.append(os.path.abspath('mms/tests/unit_tests/test_utils/'))\n patches.os_path.return_value = True\n handler = 'dummy_func_model_service:wrong'\n model_loader = ModelLoaderFactory.get_model_loader(os.path.abspath('mms/unit_tests/test_utils/'))\n with pytest.raises(ValueError, match=r\"Expected only one class .*\"):\n model_loader.load(self.model_name, self.model_dir, handler, 0, 1)\n\n def test_load_model_with_error(self, patches):\n patches.mock_open.side_effect = [\n mock.mock_open(read_data='{\"test\" : \"h\"}').return_value]\n sys.path.append(os.path.abspath('mms/tests/unit_tests/test_utils/'))\n patches.os_path.return_value = True\n handler = 'dummy_func_model_service'\n model_loader = ModelLoaderFactory.get_model_loader(os.path.abspath('mms/unit_tests/test_utils/'))\n with pytest.raises(ValueError, match=r\"Expected only one class .*\"):\n model_loader.load(self.model_name, self.model_dir, handler, 0, 1)\n" }, { "path": "mms/tests/unit_tests/test_model_service_worker.py", "content": "# Copyright 2018 Amazon.com, Inc. or its affiliates. All Rights Reserved.\n# Licensed under the Apache License, Version 2.0 (the \"License\").\n# You may not use this file except in compliance with the License.\n# A copy of the License is located at\n# http://www.apache.org/licenses/LICENSE-2.0\n# or in the \"license\" file accompanying this file. This file is distributed\n# on an \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either\n# express or implied. See the License for the specific language governing\n# permissions and limitations under the License.\n\n\"\"\"\nModelServiceWorker is the worker that is started by the MMS front-end.\n\"\"\"\n\nimport socket\nfrom collections import namedtuple\n\nimport mock\nimport pytest\nfrom mock import Mock\n\nfrom mms.model_service_worker import MXNetModelServiceWorker\nfrom mms.service import Service\n\n\n@pytest.fixture()\ndef socket_patches(mocker):\n Patches = namedtuple('Patches', ['socket'])\n mock_patch = Patches(mocker.patch('socket.socket'))\n mock_patch.socket.recv.side_effect = [\n b\"L\",\n b\"\\x00\\x00\\x00\\x0a\", b\"model_name\",\n b\"\\x00\\x00\\x00\\x0a\", b\"model_path\",\n b\"\\x00\\x00\\x00\\x01\",\n b\"\\x00\\x00\\x00\\x07\", b\"handler\",\n b\"\\x00\\x00\\x00\\x01\"\n ]\n return mock_patch\n\n\n@pytest.fixture()\ndef model_service_worker(socket_patches):\n model_service_worker = MXNetModelServiceWorker('unix', 'my-socket', None, None)\n model_service_worker.sock = socket_patches.socket\n model_service_worker.service = Service('name', 'mpath', 'testmanifest', None, 0, 1)\n return model_service_worker\n\n\n# noinspection PyClassHasNoInit\nclass TestInit:\n socket_name = \"sampleSocketName\"\n\n def test_missing_socket_name(self):\n with pytest.raises(ValueError, match=\"Invalid socket type provided.*\"):\n MXNetModelServiceWorker()\n\n def test_socket_in_use(self, mocker):\n remove = mocker.patch('os.remove')\n path_exists = mocker.patch('os.path.exists')\n remove.side_effect = OSError()\n path_exists.return_value = True\n\n with pytest.raises(Exception, match=r\".*socket already in use: sampleSocketName.*\"):\n MXNetModelServiceWorker('unix', self.socket_name)\n\n @pytest.fixture()\n def patches(self, mocker):\n Patches = namedtuple('Patches', ['remove', 'socket'])\n patches = Patches(\n mocker.patch('os.remove'),\n mocker.patch('socket.socket')\n )\n return patches\n\n def test_success(self, patches):\n MXNetModelServiceWorker('unix', self.socket_name)\n patches.remove.assert_called_once_with(self.socket_name)\n patches.socket.assert_called_once_with(socket.AF_UNIX, socket.SOCK_STREAM)\n\n\n# noinspection PyClassHasNoInit\nclass TestRunServer:\n accept_result = (mock.MagicMock(), None)\n\n def test_with_socket_bind_error(self, socket_patches, model_service_worker):\n bind_exception = socket.error(\"binding error\")\n socket_patches.socket.bind.side_effect = bind_exception\n with pytest.raises(Exception):\n model_service_worker.run_server()\n\n socket_patches.socket.bind.assert_called()\n socket_patches.socket.listen.assert_not_called()\n\n def test_with_timeout(self, socket_patches, model_service_worker):\n exception = socket.timeout(\"Some Exception\")\n socket_patches.socket.accept.side_effect = exception\n\n with pytest.raises(socket.timeout):\n model_service_worker.run_server()\n socket_patches.socket.bind.assert_called()\n socket_patches.socket.listen.assert_called()\n socket_patches.socket.accept.assert_called()\n\n def test_with_run_server_debug(self, socket_patches, model_service_worker, mocker):\n exception = Exception(\"Some Exception\")\n socket_patches.socket.accept.side_effect = exception\n mocker.patch('mms.model_service_worker.DEBUG', True)\n model_service_worker.handle_connection = Mock()\n\n with pytest.raises(Exception):\n model_service_worker.run_server()\n\n socket_patches.socket.bind.assert_called()\n socket_patches.socket.listen.assert_called()\n socket_patches.socket.accept.assert_called()\n\n def test_success(self, model_service_worker):\n model_service_worker.sock.accept.return_value = self.accept_result\n model_service_worker.sock.recv.return_value = b\"\"\n exception = SystemExit\n model_service_worker.sock.accept.side_effect = exception\n with pytest.raises(SystemExit):\n model_service_worker.run_server()\n model_service_worker.sock.accept.assert_called_once()\n\n\n# noinspection PyClassHasNoInit\nclass TestLoadModel:\n data = {'modelPath': b'mpath', 'modelName': b'name', 'handler': b'handled'}\n\n @pytest.fixture()\n def patches(self, mocker):\n Patches = namedtuple('Patches', ['loader'])\n patches = Patches(mocker.patch('mms.model_service_worker.ModelLoaderFactory'))\n return patches\n\n def test_load_model(self, patches, model_service_worker):\n patches.loader.get_model_loader.return_value = Mock()\n model_service_worker.load_model(self.data)\n patches.loader.get_model_loader.assert_called()\n\n # noinspection PyUnusedLocal\n @pytest.mark.parametrize('batch_size', [(None, None), ('1', 1)])\n @pytest.mark.parametrize('gpu', [(None, None), ('2', 2)])\n def test_optional_args(self, patches, model_service_worker, batch_size, gpu):\n data = self.data.copy()\n if batch_size[0]:\n data['batchSize'] = batch_size[0]\n if gpu[0]:\n data['gpu'] = gpu[0]\n model_service_worker.load_model(data)\n\n\n# noinspection PyClassHasNoInit\nclass TestHandleConnection:\n data = {'modelPath': b'mpath', 'modelName': b'name', 'handler': b'handled'}\n\n @pytest.fixture()\n def patches(self, mocker):\n Patches = namedtuple(\"Patches\", [\"retrieve_msg\"])\n patches = Patches(\n mocker.patch(\"mms.model_service_worker.retrieve_msg\")\n )\n return patches\n\n def test_handle_connection(self, patches, model_service_worker):\n patches.retrieve_msg.side_effect = [(b\"L\", \"\"), (b\"I\", \"\"), (b\"U\", \"\")]\n model_service_worker.load_model = Mock()\n model_service_worker.service.predict = Mock()\n model_service_worker._remap_io = Mock()\n service = Mock()\n service.context = None\n model_service_worker.load_model.return_value = (\"\", 200)\n model_service_worker.service.predict.return_value = (\"OK\")\n model_service_worker._remap_io.return_value = (\"\")\n cl_socket = Mock()\n with pytest.raises(ValueError, match=r\"Received unknown command.*\"):\n model_service_worker.handle_connection(cl_socket)\n\n cl_socket.send.assert_called()\n" }, { "path": "mms/tests/unit_tests/test_otf_codec_protocol.py", "content": "# coding=utf-8\n\n# Copyright 2018 Amazon.com, Inc. or its affiliates. All Rights Reserved.\n# Licensed under the Apache License, Version 2.0 (the \"License\").\n# You may not use this file except in compliance with the License.\n# A copy of the License is located at\n# http://www.apache.org/licenses/LICENSE-2.0\n# or in the \"license\" file accompanying this file. This file is distributed\n# on an \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either\n# express or implied. See the License for the specific language governing\n# permissions and limitations under the License.\n\n\n\"\"\"\nOn The Fly Codec tester\n\"\"\"\n\nfrom collections import namedtuple\n\nimport pytest\n\nimport mms.protocol.otf_message_handler as codec\nfrom builtins import bytes\n\n\n@pytest.fixture()\ndef socket_patches(mocker):\n Patches = namedtuple('Patches', ['socket'])\n mock_patch = Patches(mocker.patch('socket.socket'))\n mock_patch.socket.recv.return_value = b'1'\n return mock_patch\n\n\n# noinspection PyClassHasNoInit\nclass TestOtfCodecHandler:\n\n def test_retrieve_msg_unknown(self, socket_patches):\n socket_patches.socket.recv.side_effect = [b\"U\", b\"\\x00\\x00\\x00\\x03\"]\n with pytest.raises(ValueError, match=r\"Invalid command: .*\"):\n codec.retrieve_msg(socket_patches.socket)\n\n def test_retrieve_msg_load_gpu(self, socket_patches):\n expected = {\"modelName\": b\"model_name\", \"modelPath\": b\"model_path\",\n \"batchSize\": 1, \"handler\": b\"handler\", \"gpu\": 1,\n \"ioFileDescriptor\": b\"0123456789\"}\n\n socket_patches.socket.recv.side_effect = [\n b\"L\",\n b\"\\x00\\x00\\x00\\x0a\", b\"model_name\",\n b\"\\x00\\x00\\x00\\x0a\", b\"model_path\",\n b\"\\x00\\x00\\x00\\x01\",\n b\"\\x00\\x00\\x00\\x07\", b\"handler\",\n b\"\\x00\\x00\\x00\\x01\",\n b\"\\x00\\x00\\x00\\x0a\", b\"0123456789\"\n ]\n cmd, ret = codec.retrieve_msg(socket_patches.socket)\n\n assert cmd == b\"L\"\n assert ret == expected\n\n def test_retrieve_msg_load_no_gpu(self, socket_patches):\n expected = {\"modelName\": b\"model_name\", \"modelPath\": b\"model_path\",\n \"batchSize\": 1, \"handler\": b\"handler\", \"ioFileDescriptor\": b\"0123456789\"}\n\n socket_patches.socket.recv.side_effect = [\n b\"L\",\n b\"\\x00\\x00\\x00\\x0a\", b\"model_name\",\n b\"\\x00\\x00\\x00\\x0a\", b\"model_path\",\n b\"\\x00\\x00\\x00\\x01\",\n b\"\\x00\\x00\\x00\\x07\", b\"handler\",\n b\"\\xFF\\xFF\\xFF\\xFF\",\n b\"\\x00\\x00\\x00\\x0a\", b\"0123456789\"\n ]\n cmd, ret = codec.retrieve_msg(socket_patches.socket)\n\n assert cmd == b\"L\"\n assert ret == expected\n\n def test_retrieve_msg_predict(self, socket_patches):\n expected = [{\n \"requestId\": b\"request_id\", \"headers\": [], \"parameters\": [\n {\"name\": \"input_name\",\n \"contentType\": \"application/json\",\n \"value\": {\"data\": \"value\"}\n }\n ]\n }]\n\n socket_patches.socket.recv.side_effect = [\n b\"I\",\n b\"\\x00\\x00\\x00\\x0a\", b\"request_id\",\n b\"\\xFF\\xFF\\xFF\\xFF\",\n b\"\\x00\\x00\\x00\\x0a\", b\"input_name\",\n b\"\\x00\\x00\\x00\\x0F\", b\"application/json\",\n b\"\\x00\\x00\\x00\\x0F\", b'{\"data\":\"value\"}',\n b\"\\xFF\\xFF\\xFF\\xFF\", # end of parameters\n b\"\\xFF\\xFF\\xFF\\xFF\" # end of batch\n ]\n cmd, ret = codec.retrieve_msg(socket_patches.socket)\n\n assert cmd == b'I'\n assert ret == expected\n\n def test_retrieve_msg_predict_text(self, socket_patches):\n expected = [{\n \"requestId\": b\"request_id\", \"headers\": [], \"parameters\": [\n {\"name\": \"input_name\",\n \"contentType\": \"text/plain\",\n \"value\": u\"text_value测试\"\n }\n ]\n }]\n\n socket_patches.socket.recv.side_effect = [\n b\"I\",\n b\"\\x00\\x00\\x00\\x0a\", b\"request_id\",\n b\"\\xFF\\xFF\\xFF\\xFF\",\n b\"\\x00\\x00\\x00\\x0a\", b\"input_name\",\n b\"\\x00\\x00\\x00\\x0a\", b\"text/plain\",\n b\"\\x00\\x00\\x00\\x0a\", bytes(u\"text_value测试\", \"utf-8\"),\n b\"\\xFF\\xFF\\xFF\\xFF\", # end of parameters\n b\"\\xFF\\xFF\\xFF\\xFF\" # end of batch\n ]\n cmd, ret = codec.retrieve_msg(socket_patches.socket)\n\n assert cmd == b'I'\n assert ret == expected\n\n def test_retrieve_msg_predict_binary(self, socket_patches):\n expected = [{\n \"requestId\": b\"request_id\", \"headers\": [], \"parameters\": [\n {\"name\": \"input_name\",\n \"contentType\": \"\",\n \"value\": b\"binary\"\n }\n ]\n }]\n\n socket_patches.socket.recv.side_effect = [\n b\"I\",\n b\"\\x00\\x00\\x00\\x0a\", b\"request_id\",\n b\"\\xFF\\xFF\\xFF\\xFF\",\n b\"\\x00\\x00\\x00\\x0a\", b\"input_name\",\n b\"\\x00\\x00\\x00\\x00\",\n b\"\\x00\\x00\\x00\\x06\", b\"binary\",\n b\"\\xFF\\xFF\\xFF\\xFF\", # end of parameters\n b\"\\xFF\\xFF\\xFF\\xFF\" # end of batch\n ]\n cmd, ret = codec.retrieve_msg(socket_patches.socket)\n\n assert cmd == b'I'\n assert ret == expected\n\n def test_create_load_model_response(self):\n msg = codec.create_load_model_response(200, \"model_loaded\")\n\n assert msg == b'\\x00\\x00\\x00\\xc8\\x00\\x00\\x00\\x0cmodel_loaded\\xff\\xff\\xff\\xff'\n\n def test_create_predict_response(self):\n msg = codec.create_predict_response([\"OK\"], {0: \"request_id\"}, \"success\", 200)\n assert msg == b'\\x00\\x00\\x00\\xc8\\x00\\x00\\x00\\x07success\\x00\\x00\\x00\\nrequest_id\\x00\\x00\\x00\\x00\\x00\\x00' \\\n b'\\x00\\xc8\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x02OK\\xff\\xff\\xff\\xff'\n\n def test_create_predict_response_with_error(self):\n msg = codec.create_predict_response(None, {0: \"request_id\"}, \"failed\", 200)\n\n assert msg == b'\\x00\\x00\\x00\\xc8\\x00\\x00\\x00\\x06failed\\x00\\x00\\x00\\nrequest_id\\x00\\x00\\x00\\x00\\x00\\x00\\x00' \\\n b'\\xc8\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x05error\\xff\\xff\\xff\\xff'\n" }, { "path": "mms/tests/unit_tests/test_utils/MAR-INF/MANIFEST.json", "content": "" }, { "path": "mms/tests/unit_tests/test_utils/dummy_class_model_service.py", "content": "# Copyright 2018 Amazon.com, Inc. or its affiliates. All Rights Reserved.\n# Licensed under the Apache License, Version 2.0 (the \"License\").\n# You may not use this file except in compliance with the License.\n# A copy of the License is located at\n# http://www.apache.org/licenses/LICENSE-2.0\n# or in the \"license\" file accompanying this file. This file is distributed\n# on an \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either\n# express or implied. See the License for the specific language governing\n# permissions and limitations under the License.\n\n\"\"\"\nDummy custom service which is class based\n\"\"\"\n\n\n# noinspection PyUnusedLocal\nclass CustomService(object):\n\n def initialize(self, context):\n pass\n\n # noinspection PyMethodMayBeStatic\n def handle(self, data, context):\n from mms.context import Context\n return [\"OK\"]\n" }, { "path": "mms/tests/unit_tests/test_utils/dummy_func_model_service.py", "content": "# Copyright 2018 Amazon.com, Inc. or its affiliates. All Rights Reserved.\n# Licensed under the Apache License, Version 2.0 (the \"License\").\n# You may not use this file except in compliance with the License.\n# A copy of the License is located at\n# http://www.apache.org/licenses/LICENSE-2.0\n# or in the \"license\" file accompanying this file. This file is distributed\n# on an \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either\n# express or implied. See the License for the specific language governing\n# permissions and limitations under the License.\n\n\"\"\"\nDummy custom service which is function based\n\"\"\"\n\nfrom mms.context import Context\n\n\n# noinspection PyUnusedLocal\ndef infer(data, context):\n return isinstance(context, Context)\n" }, { "path": "mms/tests/unit_tests/test_version.py", "content": "# Copyright 2017 Amazon.com, Inc. or its affiliates. All Rights Reserved.\n# Licensed under the Apache License, Version 2.0 (the \"License\").\n# You may not use this file except in compliance with the License.\n# A copy of the License is located at\n# http://www.apache.org/licenses/LICENSE-2.0\n# or in the \"license\" file accompanying this file. This file is distributed\n# on an \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either\n# express or implied. See the License for the specific language governing\n# permissions and limitations under the License.\n\nimport os\nimport re\n\nimport mms\n\n\ndef test_mms_version():\n with open(os.path.join(\"mms\", \"version.py\")) as f:\n exec(f.read(), globals())\n\n assert __version__ == str(mms.__version__), \"Versions don't match\"\n" }, { "path": "mms/tests/unit_tests/test_worker_service.py", "content": "import logging\nimport os\nimport sys\n\nimport pytest\n\nfrom mms.context import Context\nfrom mms.service import Service\nfrom mms.service import emit_metrics\n\nlogging.basicConfig(stream=sys.stdout, format=\"%(message)s\", level=logging.INFO)\n\n\n# noinspection PyClassHasNoInit\nclass TestService:\n\n model_name = 'testmodel'\n model_dir = os.path.abspath('mms/tests/unit_tests/test_utils/')\n manifest = \"testmanifest\"\n data = [\n {\"requestId\": b\"123\", \"parameters\": [\n {\"name\": \"xyz\", \"value\": \"abc\", \"contentType\": \"text/csv\"}\n ], \"data\": b\"\"}\n ]\n\n @pytest.fixture()\n def service(self, mocker):\n service = object.__new__(Service)\n service._entry_point = mocker.MagicMock(return_value=['prediction'])\n service._context = Context(self.model_name, self.model_dir, self.manifest, 1, 0, '1.0')\n return service\n\n def test_predict(self, service, mocker):\n create_predict_response = mocker.patch(\"mms.service.create_predict_response\")\n service.predict(self.data)\n create_predict_response.assert_called()\n\n def test_with_nil_request(self, service):\n with pytest.raises(ValueError, match=r\"Received invalid inputs\"):\n service.retrieve_data_for_inference(None)\n\n def test_valid_req(self, service):\n headers, input_batch, req_to_id_map = service.retrieve_data_for_inference(self.data)\n assert headers[0].get_request_property(\"xyz\").get(\"content-type\") == \"text/csv\"\n assert input_batch[0] == {\"xyz\": \"abc\"}\n assert req_to_id_map == {0: \"123\"}\n\n\n# noinspection PyClassHasNoInit\nclass TestEmitMetrics:\n\n def test_emit_metrics(self, caplog):\n caplog.set_level(logging.INFO)\n metrics = {'test_emit_metrics': True}\n emit_metrics(metrics)\n assert \"[METRICS]\" in caplog.text\n" }, { "path": "mms/utils/__init__.py", "content": "# Copyright 2017 Amazon.com, Inc. or its affiliates. All Rights Reserved.\n# Licensed under the Apache License, Version 2.0 (the \"License\").\n# You may not use this file except in compliance with the License.\n# A copy of the License is located at\n# http://www.apache.org/licenses/LICENSE-2.0\n# or in the \"license\" file accompanying this file. This file is distributed\n# on an \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either\n# express or implied. See the License for the specific language governing\n# permissions and limitations under the License.\n\n\"\"\"\nUtil files for MMS\n\"\"\"\nfrom . import timeit_decorator\n" }, { "path": "mms/utils/mxnet/__init__.py", "content": "# Copyright 2017 Amazon.com, Inc. or its affiliates. All Rights Reserved.\n# Licensed under the Apache License, Version 2.0 (the \"License\").\n# You may not use this file except in compliance with the License.\n# A copy of the License is located at\n# http://www.apache.org/licenses/LICENSE-2.0\n# or in the \"license\" file accompanying this file. This file is distributed\n# on an \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either\n# express or implied. See the License for the specific language governing\n# permissions and limitations under the License.\n\n\"\"\"\nMXNet Utils\n\"\"\"\nimport warnings\n\nwarnings.warn(\"Module mms.utils.mxnet is deprecated, please avoid using mms internal modules.\",\n DeprecationWarning, stacklevel=2)\n" }, { "path": "mms/utils/mxnet/image.py", "content": "# Copyright 2017 Amazon.com, Inc. or its affiliates. All Rights Reserved.\n# Licensed under the Apache License, Version 2.0 (the \"License\").\n# You may not use this file except in compliance with the License.\n# A copy of the License is located at\n# http://www.apache.org/licenses/LICENSE-2.0\n# or in the \"license\" file accompanying this file. This file is distributed\n# on an \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either\n# express or implied. See the License for the specific language governing\n# permissions and limitations under the License.\n\n\"\"\"\nImage utils\n\"\"\"\nimport sys\nimport base64\nfrom io import BytesIO\nimport numpy as np\nfrom PIL import Image\nimport mxnet as mx\nfrom mxnet import image as img\n\n\ndef transform_shape(img_arr, dim_order='NCHW'):\n \"\"\"Rearrange image NDArray shape to 'NCHW' or 'NHWC' which\n is valid for MXNet model input.\n Input image NDArray should has dim_order of 'HWC'.\n\n Parameters\n ----------\n img_arr : NDArray\n Image in NDArray format with shape (channel, width, height)\n dim_order : str\n Output image dimension order. Valid values are 'NCHW' and 'NHWC'\n\n Returns\n -------\n output : NDArray\n Image in NDArray format with dim_order shape\n \"\"\"\n assert dim_order in 'NCHW' or dim_order in 'NHWC', \"dim_order must be 'NCHW' or 'NHWC'.\"\n if dim_order == 'NCHW':\n img_arr = mx.nd.transpose(img_arr, (2, 0, 1))\n output = mx.nd.expand_dims(img_arr, axis=0)\n return output\n\n\ndef read(buf, flag=1, to_rgb=True, out=None):\n\n \"\"\"Read and decode an image to an NDArray.\n Input image NDArray should has dim_order of 'HWC'.\n\n Note: `imread` uses OpenCV (not the CV2 Python library).\n MXNet must have been built with USE_OPENCV=1 for `imdecode` to work.\n\n Parameters\n ----------\n buf : str/bytes or numpy.ndarray\n Binary image data as string or numpy ndarray.\n flag : {0, 1}, default 1\n 1 for three channel color output. 0 for grayscale output.\n to_rgb : bool, default True\n True for RGB formatted output (MXNet default).\n False for BGR formatted output (OpenCV default).\n out : NDArray, optional\n Output buffer. Use `None` for automatic allocation.\n\n Returns\n -------\n NDArray\n An `NDArray` containing the image.\n\n Example\n -------\n >>> buf = open(\"flower.jpg\", 'rb').read()\n >>> image.read(buf)\n \n \"\"\"\n return img.imdecode(buf, flag, to_rgb, out)\n\n\n# TODO: Check where this is used and rename format\ndef write(img_arr, flag=1, format='jpeg', dim_order='CHW'): # pylint: disable=redefined-builtin\n \"\"\"Write an NDArray to a base64 string\n\n Parameters\n ----------\n img_arr : NDArray\n Image in NDArray format with shape (channel, width, height).\n flag : {0, 1}, default 1\n 1 for three channel color output. 0 for grayscale output.\n format : str\n Output image format.\n dim_order : str\n Input image dimension order. Valid values are 'CHW' and 'HWC'\n\n Returns\n -------\n str\n Image in base64 string format\n \"\"\"\n assert dim_order in 'CHW' or dim_order in 'HWC', \"dim_order must be 'CHW' or 'HWC'.\"\n if dim_order == 'CHW':\n img_arr = mx.nd.transpose(img_arr, (1, 2, 0))\n if flag == 1:\n mode = 'RGB'\n else:\n mode = 'L'\n img_arr = mx.nd.reshape(img_arr, (img_arr.shape[0], img_arr.shape[1]))\n img_arr = img_arr.astype(np.uint8).asnumpy()\n image = Image.fromarray(img_arr, mode)\n output = BytesIO()\n image.save(output, format=format)\n output.seek(0)\n if sys.version_info[0] < 3:\n return base64.b64encode(output.getvalue())\n else:\n return base64.b64encode(output.getvalue()).decode(\"utf-8\")\n\n\ndef resize(src, new_width, new_height, interp=2):\n \"\"\"Resizes image to new_width and new_height.\n Input image NDArray should has dim_order of 'HWC'.\n\n Parameters\n ----------\n src : NDArray\n Source image in NDArray format\n new_width : int\n Width in pixel for resized image\n new_height : int\n Height in pixel for resized image\n interp : int\n interpolation method for all resizing operations\n\n Possible values:\n 0: Nearest Neighbors Interpolation.\n 1: Bilinear interpolation.\n 2: Area-based (resampling using pixel area relation). It may be a\n preferred method for image decimation, as it gives moire-free\n results. But when the image is zoomed, it is similar to the Nearest\n Neighbors method. (used by default).\n 3: Bicubic interpolation over 4x4 pixel neighborhood.\n 4: Lanczos interpolation over 8x8 pixel neighborhood.\n 9: Cubic for enlarge, area for shrink, bilinear for others\n 10: Random select from interpolation method metioned above.\n Note:\n When shrinking an image, it will generally look best with AREA-based\n interpolation, whereas, when enlarging an image, it will generally look best\n with Bicubic (slow) or Bilinear (faster but still looks OK).\n More details can be found in the documentation of OpenCV, please refer to\n http://docs.opencv.org/master/da/d54/group__imgproc__transform.html.\n\n Returns\n -------\n NDArray\n An `NDArray` containing the resized image.\n \"\"\"\n return img.imresize(src, new_width, new_height, interp)\n\n\ndef fixed_crop(src, x0, y0, w, h, size=None, interp=2):\n \"\"\"Crop src at fixed location, and (optionally) resize it to size.\n Input image NDArray should has dim_order of 'HWC'.\n\n Parameters\n ----------\n src : NDArray\n Input image\n x0 : int\n Left boundary of the cropping area\n y0 : int\n Top boundary of the cropping area\n w : int\n Width of the cropping area\n h : int\n Height of the cropping area\n size : tuple of (w, h)\n Optional, resize to new size after cropping\n interp : int, optional, default=2\n Interpolation method. See resize for details.\n\n Returns\n -------\n NDArray\n An `NDArray` containing the cropped image.\n \"\"\"\n return img.fixed_crop(src, x0, y0, w, h, size, interp)\n\n\ndef color_normalize(src, mean, std=None):\n \"\"\"Normalize src with mean and std.\n\n Parameters\n ----------\n src : NDArray\n Input image\n mean : NDArray\n RGB mean to be subtracted\n std : NDArray\n RGB standard deviation to be divided\n\n Returns\n -------\n NDArray\n An `NDArray` containing the normalized image.\n \"\"\"\n src = src.astype(np.float32)\n return img.color_normalize(src, mean, std)\n" }, { "path": "mms/utils/mxnet/ndarray.py", "content": "# Copyright 2017 Amazon.com, Inc. or its affiliates. All Rights Reserved.\n# Licensed under the Apache License, Version 2.0 (the \"License\").\n# You may not use this file except in compliance with the License.\n# A copy of the License is located at\n# http://www.apache.org/licenses/LICENSE-2.0\n# or in the \"license\" file accompanying this file. This file is distributed\n# on an \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either\n# express or implied. See the License for the specific language governing\n# permissions and limitations under the License.\n\n\"\"\"\nNDArray utils\n\"\"\"\nimport numpy as np\nimport mxnet as mx\n\n\ndef top_probability(data, labels, top=5):\n \"\"\"Get top probability prediction from NDArray.\n\n Parameters\n ----------\n data : NDArray\n Data to be predicted\n labels : List\n List of class labels\n\n Returns\n -------\n List\n List of probability: class pairs in sorted order\n \"\"\"\n dim = len(data.shape)\n if dim > 2:\n data = mx.nd.array(\n np.squeeze(data.asnumpy(), axis=tuple(range(dim)[2:])))\n sorted_prob = mx.nd.argsort(data[0], is_ascend=False)\n # pylint: disable=deprecated-lambda\n top_prob = map(lambda x: int(x.asscalar()), sorted_prob[0:top])\n return [{'probability': float(data[0, i].asscalar()), 'class': labels[i]}\n for i in top_prob]\n" }, { "path": "mms/utils/mxnet/nlp.py", "content": "# Copyright 2017 Amazon.com, Inc. or its affiliates. All Rights Reserved.\n# Licensed under the Apache License, Version 2.0 (the \"License\").\n# You may not use this file except in compliance with the License.\n# A copy of the License is located at\n# http://www.apache.org/licenses/LICENSE-2.0\n# or in the \"license\" file accompanying this file. This file is distributed\n# on an \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either\n# express or implied. See the License for the specific language governing\n# permissions and limitations under the License.\n\n\"\"\"\nNLP utils\n\"\"\"\nimport bisect\nimport numpy as np\nimport mxnet as mx\n\n\ndef encode_sentences(sentences, vocab=None, invalid_label=-1, invalid_key='\\n', start_label=0):\n \"\"\"Encode sentences and (optionally) build a mapping\n from string tokens to integer indices. Unknown keys\n will be added to vocabulary.\n\n Parameters\n ----------\n sentences : list of list of str\n A list of sentences to encode. Each sentence\n should be a list of string tokens.\n vocab : None or dict of str -> int\n Optional input Vocabulary\n invalid_label : int, default -1\n Index for invalid token, like \n invalid_key : str, default '\\\\n'\n Key for invalid token. Use '\\\\n' for end\n of sentence by default.\n start_label : int\n lowest index.\n\n Returns\n -------\n result : list of list of int\n encoded sentences\n vocab : dict of str -> int\n result vocabulary\n \"\"\"\n idx = start_label\n if vocab is None:\n vocab = {invalid_key: invalid_label}\n new_vocab = True\n else:\n new_vocab = False\n res = []\n for sent in sentences:\n coded = []\n for word in sent:\n if word not in vocab:\n if not new_vocab:\n coded.append(invalid_label)\n continue\n else:\n if idx == invalid_label:\n idx += 1\n vocab[word] = idx\n idx += 1\n coded.append(vocab[word])\n res.append(coded)\n\n return res, vocab\n\n\ndef pad_sentence(sentence, buckets, invalid_label=-1, data_name='data', layout='NT'):\n \"\"\"Pad a sentence to closest length in provided buckets.\n\n Parameters\n ----------\n sentence : list of int\n A list of integer representing an encoded sentence.\n buckets : list of int\n Size of the data buckets.\n invalid_label : int, optional\n Index for invalid token, like .\n data_name : str, optional\n Input data name.\n layoutlayout : str, optional\n Format of data and label. 'NT' means (batch_size, length)\n and 'TN' means (length, batch_size).\n\n Returns\n -------\n result : mx.io.DataBatch\n DataBatch contains sentence.\n \"\"\"\n buck = bisect.bisect_left(buckets, len(sentence))\n buff = np.full((buckets[buck],), invalid_label, dtype='float32')\n buff[:len(sentence)] = sentence\n sent_bucket = buckets[buck]\n pad_sent = mx.nd.array([buff], dtype='float32')\n shape = (1, sent_bucket) if layout == 'NT' else (sent_bucket, 1)\n return mx.io.DataBatch([pad_sent], pad=0, bucket_key=sent_bucket,\n provide_data=[mx.io.DataDesc(\n name=data_name,\n shape=shape,\n layout=layout)])\n" }, { "path": "mms/utils/timeit_decorator.py", "content": "# Copyright 2018 Amazon.com, Inc. or its affiliates. All Rights Reserved.\n# Licensed under the Apache License, Version 2.0 (the \"License\").\n# You may not use this file except in compliance with the License.\n# A copy of the License is located at\n# http://www.apache.org/licenses/LICENSE-2.0\n# or in the \"license\" file accompanying this file. This file is distributed\n# on an \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either\n# express or implied. See the License for the specific language governing\n# permissions and limitations under the License.\n\"\"\"\ntimeit decorator\n\"\"\"\n\nimport time\nfrom functools import wraps\n\n\ndef timeit(func):\n \"\"\"\n Use this decorator on a method to find it's execution time.\n :param func:\n :return:\n \"\"\"\n @wraps(func)\n def time_and_log(*args, **kwargs):\n start = time.time()\n start_cpu = time.clock()\n result = func(*args, **kwargs)\n end = time.time()\n end_cpu = time.clock()\n print(\"func: %r took a total of %2.4f sec to run and %2.4f sec of CPU time\\n\",\n (func.__name__, (end-start), (end_cpu - start_cpu)))\n return result\n return time_and_log\n" }, { "path": "mms/version.py", "content": "# Copyright 2017 Amazon.com, Inc. or its affiliates. All Rights Reserved.\n# Licensed under the Apache License, Version 2.0 (the \"License\").\n# You may not use this file except in compliance with the License.\n# A copy of the License is located at\n# http://www.apache.org/licenses/LICENSE-2.0\n# or in the \"license\" file accompanying this file. This file is distributed\n# on an \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either\n# express or implied. See the License for the specific language governing\n# permissions and limitations under the License.\n\"\"\"\nThis is the current version of MMS\n\"\"\"\n\n__version__ = '1.1.11'\n" }, { "path": "model-archiver/.coveragerc", "content": "# Copyright 2018 Amazon.com, Inc. or its affiliates. All Rights Reserved.\n# Licensed under the Apache License, Version 2.0 (the \"License\").\n# You may not use this file except in compliance with the License.\n# A copy of the License is located at\n# http://www.apache.org/licenses/LICENSE-2.0\n# or in the \"license\" file accompanying this file. This file is distributed\n# on an \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either\n# express or implied. See the License for the specific language governing\n# permissions and limitations under the License.\n\n[report]\nexclude_lines =\n pragma: no cover\n if __name__ == .__main__.:\n if __name__ == \"__main__\" :\n\n[run]\nbranch = True\nomit = */__init__.py\n model_archiver/tests/*\n model_archiver/manifest_components/*\n model_archiver/arg_parser.py\n model_archiver/setup.py\n\n" }, { "path": "model-archiver/LICENSE", "content": " Apache License\n Version 2.0, January 2004\n http://www.apache.org/licenses/\n\n TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION\n\n 1. Definitions.\n\n \"License\" shall mean the terms and conditions for use, reproduction,\n and distribution as defined by Sections 1 through 9 of this document.\n\n \"Licensor\" shall mean the copyright owner or entity authorized by\n the copyright owner that is granting the License.\n\n \"Legal Entity\" shall mean the union of the acting entity and all\n other entities that control, are controlled by, or are under common\n control with that entity. For the purposes of this definition,\n \"control\" means (i) the power, direct or indirect, to cause the\n direction or management of such entity, whether by contract or\n otherwise, or (ii) ownership of fifty percent (50%) or more of the\n outstanding shares, or (iii) beneficial ownership of such entity.\n\n \"You\" (or \"Your\") shall mean an individual or Legal Entity\n exercising permissions granted by this License.\n\n \"Source\" form shall mean the preferred form for making modifications,\n including but not limited to software source code, documentation\n source, and configuration files.\n\n \"Object\" form shall mean any form resulting from mechanical\n transformation or translation of a Source form, including but\n not limited to compiled object code, generated documentation,\n and conversions to other media types.\n\n \"Work\" shall mean the work of authorship, whether in Source or\n Object form, made available under the License, as indicated by a\n copyright notice that is included in or attached to the work\n (an example is provided in the Appendix below).\n\n \"Derivative Works\" shall mean any work, whether in Source or Object\n form, that is based on (or derived from) the Work and for which the\n editorial revisions, annotations, elaborations, or other modifications\n represent, as a whole, an original work of authorship. For the purposes\n of this License, Derivative Works shall not include works that remain\n separable from, or merely link (or bind by name) to the interfaces of,\n the Work and Derivative Works thereof.\n\n \"Contribution\" shall mean any work of authorship, including\n the original version of the Work and any modifications or additions\n to that Work or Derivative Works thereof, that is intentionally\n submitted to Licensor for inclusion in the Work by the copyright owner\n or by an individual or Legal Entity authorized to submit on behalf of\n the copyright owner. For the purposes of this definition, \"submitted\"\n means any form of electronic, verbal, or written communication sent\n to the Licensor or its representatives, including but not limited to\n communication on electronic mailing lists, source code control systems,\n and issue tracking systems that are managed by, or on behalf of, the\n Licensor for the purpose of discussing and improving the Work, but\n excluding communication that is conspicuously marked or otherwise\n designated in writing by the copyright owner as \"Not a Contribution.\"\n\n \"Contributor\" shall mean Licensor and any individual or Legal Entity\n on behalf of whom a Contribution has been received by Licensor and\n subsequently incorporated within the Work.\n\n 2. Grant of Copyright License. Subject to the terms and conditions of\n this License, each Contributor hereby grants to You a perpetual,\n worldwide, non-exclusive, no-charge, royalty-free, irrevocable\n copyright license to reproduce, prepare Derivative Works of,\n publicly display, publicly perform, sublicense, and distribute the\n Work and such Derivative Works in Source or Object form.\n\n 3. Grant of Patent License. Subject to the terms and conditions of\n this License, each Contributor hereby grants to You a perpetual,\n worldwide, non-exclusive, no-charge, royalty-free, irrevocable\n (except as stated in this section) patent license to make, have made,\n use, offer to sell, sell, import, and otherwise transfer the Work,\n where such license applies only to those patent claims licensable\n by such Contributor that are necessarily infringed by their\n Contribution(s) alone or by combination of their Contribution(s)\n with the Work to which such Contribution(s) was submitted. If You\n institute patent litigation against any entity (including a\n cross-claim or counterclaim in a lawsuit) alleging that the Work\n or a Contribution incorporated within the Work constitutes direct\n or contributory patent infringement, then any patent licenses\n granted to You under this License for that Work shall terminate\n as of the date such litigation is filed.\n\n 4. Redistribution. You may reproduce and distribute copies of the\n Work or Derivative Works thereof in any medium, with or without\n modifications, and in Source or Object form, provided that You\n meet the following conditions:\n\n (a) You must give any other recipients of the Work or\n Derivative Works a copy of this License; and\n\n (b) You must cause any modified files to carry prominent notices\n stating that You changed the files; and\n\n (c) You must retain, in the Source form of any Derivative Works\n that You distribute, all copyright, patent, trademark, and\n attribution notices from the Source form of the Work,\n excluding those notices that do not pertain to any part of\n the Derivative Works; and\n\n (d) If the Work includes a \"NOTICE\" text file as part of its\n distribution, then any Derivative Works that You distribute must\n include a readable copy of the attribution notices contained\n within such NOTICE file, excluding those notices that do not\n pertain to any part of the Derivative Works, in at least one\n of the following places: within a NOTICE text file distributed\n as part of the Derivative Works; within the Source form or\n documentation, if provided along with the Derivative Works; or,\n within a display generated by the Derivative Works, if and\n wherever such third-party notices normally appear. The contents\n of the NOTICE file are for informational purposes only and\n do not modify the License. You may add Your own attribution\n notices within Derivative Works that You distribute, alongside\n or as an addendum to the NOTICE text from the Work, provided\n that such additional attribution notices cannot be construed\n as modifying the License.\n\n You may add Your own copyright statement to Your modifications and\n may provide additional or different license terms and conditions\n for use, reproduction, or distribution of Your modifications, or\n for any such Derivative Works as a whole, provided Your use,\n reproduction, and distribution of the Work otherwise complies with\n the conditions stated in this License.\n\n 5. Submission of Contributions. Unless You explicitly state otherwise,\n any Contribution intentionally submitted for inclusion in the Work\n by You to the Licensor shall be under the terms and conditions of\n this License, without any additional terms or conditions.\n Notwithstanding the above, nothing herein shall supersede or modify\n the terms of any separate license agreement you may have executed\n with Licensor regarding such Contributions.\n\n 6. Trademarks. This License does not grant permission to use the trade\n names, trademarks, service marks, or product names of the Licensor,\n except as required for reasonable and customary use in describing the\n origin of the Work and reproducing the content of the NOTICE file.\n\n 7. Disclaimer of Warranty. Unless required by applicable law or\n agreed to in writing, Licensor provides the Work (and each\n Contributor provides its Contributions) on an \"AS IS\" BASIS,\n WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or\n implied, including, without limitation, any warranties or conditions\n of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A\n PARTICULAR PURPOSE. You are solely responsible for determining the\n appropriateness of using or redistributing the Work and assume any\n risks associated with Your exercise of permissions under this License.\n\n 8. Limitation of Liability. In no event and under no legal theory,\n whether in tort (including negligence), contract, or otherwise,\n unless required by applicable law (such as deliberate and grossly\n negligent acts) or agreed to in writing, shall any Contributor be\n liable to You for damages, including any direct, indirect, special,\n incidental, or consequential damages of any character arising as a\n result of this License or out of the use or inability to use the\n Work (including but not limited to damages for loss of goodwill,\n work stoppage, computer failure or malfunction, or any and all\n other commercial damages or losses), even if such Contributor\n has been advised of the possibility of such damages.\n\n 9. Accepting Warranty or Additional Liability. While redistributing\n the Work or Derivative Works thereof, You may choose to offer,\n and charge a fee for, acceptance of support, warranty, indemnity,\n or other liability obligations and/or rights consistent with this\n License. However, in accepting such obligations, You may act only\n on Your own behalf and on Your sole responsibility, not on behalf\n of any other Contributor, and only if You agree to indemnify,\n defend, and hold each Contributor harmless for any liability\n incurred by, or claims asserted against, such Contributor by reason\n of your accepting any such warranty or additional liability.\n\n END OF TERMS AND CONDITIONS\n\n APPENDIX: How to apply the Apache License to your work.\n\n To apply the Apache License to your work, attach the following\n boilerplate notice, with the fields enclosed by brackets \"{}\"\n replaced with your own identifying information. (Don't include\n the brackets!) The text should be enclosed in the appropriate\n comment syntax for the file format. We also recommend that a\n file or class name and description of purpose be included on the\n same \"printed page\" as the copyright notice for easier\n identification within third-party archives.\n\n Copyright {yyyy} {name of copyright owner}\n\n Licensed under the Apache License, Version 2.0 (the \"License\");\n you may not use this file except in compliance with the License.\n You may obtain a copy of the License at\n\n http://www.apache.org/licenses/LICENSE-2.0\n\n Unless required by applicable law or agreed to in writing, software\n distributed under the License is distributed on an \"AS IS\" BASIS,\n WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n See the License for the specific language governing permissions and\n limitations under the License.\n" }, { "path": "model-archiver/MANIFEST.in", "content": "include PyPiDescription.rst\n" }, { "path": "model-archiver/PyPiDescription.rst", "content": "Project Description\n===================\n\nModel Archiver is a tool used for creating archives of trained neural\nnet models that can be consumed by Multi-Model-Server inference.\n\nUse the Model Archiver CLI to start create a ``.mar`` file.\n\nModel Archiver is part of `MMS `__.\nHowever,you ca install Model Archiver stand alone.\n\nDetailed documentation and examples are provided in the `README\n`__.\n\nPrerequisites\n-------------\n\nONNX support is optional in `model-archiver` tool. It's not installed\nby default with `model-archiver`.\n\nIf you wish to package a ONNX model, you will need to first install a\n``protobuf`` compiler, ``onnx`` and ``mxnet`` manually.\n\n`Instructions for installing Model Archiver with\nONNX `__.\n\n\n\nInstallation\n------------\n\n::\n\n pip install model-archiver\n\nDevelopment\n-----------\n\nWe welcome new contributors of all experience levels. For information on\nhow to install MMS for development, refer to the `MMS\ndocs `__.\n\nImportant links\n---------------\n\n- `Official source code\n repo `__\n- `Download\n releases `__\n- `Issue\n tracker `__\n\nSource code\n-----------\n\nYou can check the latest source code as follows:\n\n::\n\n git clone https://github.com/awslabs/multi-model-server.git\n\nTesting\n-------\n\nAfter installation, try out the MMS Quickstart for `Create a\nmodel archive `__\nand `Serving a\nModel `__.\n\n\nHelp and Support\n----------------\n\n- `Documentation `__\n- `Forum `__\n\nCitation\n--------\n\nIf you use MMS in a publication or project, please cite MMS:\nhttps://github.com/awslabs/multi-model-server\n" }, { "path": "model-archiver/README.md", "content": "# Model archiver for MMS\n\n## Contents of this Document\n* [Overview](#overview)\n* [Model Archiver CLI](#model-archiver-command-line-interface)\n* [Artifact Details](#artifact-details)\n * [MAR-INFO](#mar-inf)\n * [Model name](#model-name)\n * [Runtime](#runtime)\n * [Handler](#handler)\n* [Quick Start: Creating a Model Archive](#creating-a-model-archive)\n\n## Other Relevant Documents\n* [Model Archive Examples](../examples/README.md)\n* [Packaging an ONNX Model](docs/convert_from_onnx.md)\n\n## Overview\n\nA key feature of MMS is the ability to package all model artifacts into a single model archive file. It is a separate command line interface (CLI), `model-archiver`, that can take model checkpoints and package them into a `.mar` file. This file can then be redistributed and served by anyone using MMS. It takes in the following model artifacts: a model composed of one or more files, the description of the model's inputs in the form of a signature file, a service file describing how to handle inputs and outputs, and other optional assets that may be required to serve the model. The CLI creates a `.mar` file that MMS's server CLI uses to serve the models.\n\n**Important**: Make sure you try the [Quick Start: Creating a Model Archive](#creating-a-model-archive) tutorial for a short example of using `model-archiver`.\n\nMMS can support any arbitrary model file. It is the custom service code's responsibility to locate and load the model files. The following information is required to create a standalone model archive:\n1. [Model name](#model-name)\n2. [Model path](#model-path)\n3. [Handler](#handler)\n\n## Model Archiver Command Line Interface\n\nNow let's cover the details on using the CLI tool: `model-archiver`.\n\nHere is an example usage with the squeezenet_v1.1 model archive which you can download or create by following the example in the [main README](../README.md):\n\n```bash\n\nmodel-archiver --model-name squeezenet_v1.1 --model-path squeezenet --handler mxnet_vision_service:handle\n\n```\n\n### Arguments\n\n```\n$ model-archiver -h\nusage: model-archiver [-h] --model-name MODEL_NAME --model-path MODEL_PATH\n --handler HANDLER [--runtime {python,python2,python3}]\n [--export-path EXPORT_PATH] [-f]\n\nModel Archiver Tool\n\noptional arguments:\n -h, --help show this help message and exit\n --model-name MODEL_NAME\n Exported model name. Exported file will be named as\n model-name.mar and saved in current working directory\n if no --export-path is specified, else it will be\n saved under the export path\n --model-path MODEL_PATH\n Path to the folder containing model related files.\n --handler HANDLER Handler path to handle custom MMS inference logic.\n --runtime {python,python2,python3}\n The runtime specifies which language to run your\n inference code on. The default runtime is\n RuntimeType.PYTHON. At the present moment we support\n the following runtimes python, python2, python3\n --export-path EXPORT_PATH\n Path where the exported .mar file will be saved. This\n is an optional parameter. If --export-path is not\n specified, the file will be saved in the current\n working directory.\n --archive-format {tgz,default}\n The format in which the model artifacts are archived.\n \"tgz\": This creates the model-archive in .tar.gz format.\n If platform hosting MMS requires model-artifacts to be in \".tar.gz\"\n use this option.\n \"no-archive\": This option creates an non-archived version of model artifacts\n at \"export-path/{model-name}\" location. As a result of this choice,\n MANIFEST file will be created at \"export-path/{model-name}\" location\n without archiving these model files\n \"default\": This creates the model-archive in .mar format.\n This is the default archiving format. Models archived in this format\n will be readily hostable on native MMS.\n -f, --force When the -f or --force flag is specified, an existing\n .mar file with same name as that provided in --model-\n name in the path specified by --export-path will\n overwritten\n```\n\n## Artifact Details\n\n### MAR-INF\n**MAR-INF** is a reserved folder name that will be used inside `.mar` file. This folder contains the model archive metadata files. Users should avoid using **MAR-INF** in their model path.\n\n### Runtime\n\n### Model name\n\nA valid model name must begin with a letter of the alphabet and can only contains letters, digits, underscores (_), dashes (-) and periods (.).\n\n**Note**: The model name can be overridden when you register the model with [Register Model API](../docs/management_api.md#register-a-model).\n\n### Model path\n\nA folder that contains all necessary files needed to run inference code for the model. All the files and sub-folders (except [excluded files](#excluded-files)) will be packaged into the `.mar` file.\n\n#### excluded files\nThe following types of file will be excluded during model archive packaging:\n1. hidden files\n2. Mac system files: __MACOSX and .DS_Store\n3. MANIFEST.json\n4. python compiled byte code (.pyc) files and cache folder __pycache__\n\n### handler\n\nA handler is a python entry point that MMS can invoke to execute inference code. The format of a Python handler is:\n* python_module_name[:function_name] (for example: lstm-service:handle).\n\nThe function name is optional if the provided python module follows one of predefined conventions:\n1. There is a `handle()` function available in the module\n2. The module contains only one Class and that class contains a `handle()` function.\n\nFurther details and specifications are found on the [custom service](../docs/custom_service.md) page.\n\n\n## Creating a Model Archive\n\n**1. Download these sample SqueezeNet model artifacts (if you don't have them handy)**\n\n```bash\nmkdir squeezenet\n\ncurl -o squeezenet/squeezenet_v1.1-symbol.json https://s3.amazonaws.com/model-server/model_archive_1.0/examples/squeezenet_v1.1/squeezenet_v1.1-symbol.json\ncurl -o squeezenet/squeezenet_v1.1-0000.params https://s3.amazonaws.com/model-server/model_archive_1.0/examples/squeezenet_v1.1/squeezenet_v1.1-0000.params\ncurl -o squeezenet/signature.json https://s3.amazonaws.com/model-server/model_archive_1.0/examples/squeezenet_v1.1/signature.json\ncurl -o squeezenet/synset.txt https://s3.amazonaws.com/model-server/model_archive_1.0/examples/squeezenet_v1.1/synset.txt\n```\n\nThe downloaded model artifact files are:\n\n* **Model Definition** (json file) - contains the layers and overall structure of the neural network.\n* **Model Params and Weights** (params file) - contains the parameters and the weights.\n* **Model Signature** (json file) - defines the inputs and outputs that MMS is expecting to hand-off to the API.\n* **assets** (text files) - auxiliary files that support model inference such as vocabularies, labels, etc. These vary depending on the model.\n\n\n**2. Download the model archiver source**\n```bash\ngit clone https://github.com/awslabs/multi-model-server.git\n```\n\n**3. Prepare your model custom service code**\n\nYou can implement your own model customer service code with a model archive entry point.\nHere we are going to use the MXNet vision service `model_service_template`.\nThis template is one of several provided with MMS.\nDownload the template and place it in your `squeezenet` folder.\n\n```bash\ncp -r multi-model-server/examples/model_service_template/* squeezenet/\n```\n\n**4. Package your model**\n\nWith the model artifacts available locally, you can use the `model-archiver` CLI to generate a `.mar` file that can be used to serve an inference API with MMS.\n\nIn this next step we'll run `model-archiver` and tell it our model's prefix is `squeezenet_v1.1` with the `model-name` argument. Then we're giving it the `model-path` to the model's assets.\n\n**Note**: For mxnet models, `model-name` must match prefix of the symbol and param file name.\n\n```bash\nmodel-archiver --model-name squeezenet_v1.1 --model-path squeezenet --handler mxnet_vision_service:handle\n```\n\nThis will package all the model artifacts files located in the `squeezenet` directory and output `squeezenet_v1.1.mar` in the current working directory. This `.mar` file is all you need to run MMS, serving inference requests for a simple image recognition API. Go back to the [Serve a Model tutorial](../README.md#serve-a-model) and try to run this model archive that you just created!\n" }, { "path": "model-archiver/docs/convert_from_onnx.md", "content": "# Converting an ONNX Model\n\n## Install model-archiver with ONNX support\nONNX support is optional in `model-archiver` tool. It's not installed by default with `model-archiver`.\n\nTo install MMS with ONNX support, you will need to have the [protobuf compiler](https://github.com/onnx/onnx#installation) installed:\n\nfor Ubuntu run:\n\n```bash\nsudo apt-get install protobuf-compiler libprotoc-dev\n\npip install model-archiver[onnx]\n```\n\nOr for Mac run:\n\n```bash\nconda install -c conda-forge protobuf numpy\n\npip install model-archiver[onnx]\n```\n\nMXNet is also required for conversion. You can choose different flavor is mxnet:\n\n```bash\npip install mxnet\n\nor\n\npip install mxnet-mkl\n\nor\n\npip install mxnet-cu90mkl\n```\n\n## ONNX model archive example\n\nYou can download a model from the [ONNX Model Zoo](https://github.com/onnx/models) then use `model-archiver` to covert it to a `.mar` file.\n\n**Note**: Some ONNX model authors upload their models to the zoo in the `.pb` or `.pb2` format. Just change the extension to `.onnx` before attempting to convert.\n\nLet's use the SqueezeNet ONNX model as an example. \n\n### Prepare ONNX model and labels\n\nTo create a model archive for MMS, you can get `.onnx` file and optionally a labels file (synset.txt) from our S3:\n\n* [SqueezeNet ONNX model](https://s3.amazonaws.com/model-server/model_archive_1.0/examples/onnx-squeezenet/squeezenet.onnx): a `.onnx` model file from the [ONNX Model Zoo](https://github.com/onnx/models)\n* [label file](https://s3.amazonaws.com/model-server/model_archive_1.0/examples/onnx-squeezenet/synset.txt): has the labels for 1,000 ImageNet classes\n\n```bash\ncd multi-model-server/examples\nmkdir onnx-squeezenet\ncd onnx-squeezenet\n\ncurl -O https://s3.amazonaws.com/model-server/model_archive_1.0/examples/onnx-squeezenet/squeezenet.onnx\ncurl -O https://s3.amazonaws.com/model-server/model_archive_1.0/examples/onnx-squeezenet/synset.txt\n```\n\n### Prepare your model custom service code\n\nYou can implement your own model customer service code as model archive entry point. In this example we just copy provided mxnet vision service template:\n\n```bash\ncd multi-model-server/examples\n\ncp -r model_service_template/* onnx-squeezenet/\n```\n\nThe mxnet_vision_service.py assume there is a signature.json file that describes input parameter name and shape. You can download example from: [signature file](https://s3.amazonaws.com/model-server/model_archive_1.0/examples/onnx-squeezenet/signature.json).\n\n\n```bash\ncd multi-model-server/examples/onnx-squeezenet\n\ncurl -o signature.json https://s3.amazonaws.com/model-server/model_archive_1.0/examples/onnx-squeezenet/signature.json\n```\n\n### Create a `.mar` file from onnx model\n\nThe model file in this example contains`.onnx` extension.\n\nIn order to convert the model with `.onnx` extension to an MXNet model, we would need to use the `-c` option of the model-archiver tool.\n\nNow you can use the `model-archiver` command to output `onnx-squeezenet.mar` file.\n\n```bash\ncd multi-model-server/examples\n\nmodel-archiver --model-name onnx-squeezenet --model-path onnx-squeezenet --handler mxnet_vision_service:handle -c -f\n```\n\nNow start the server:\n\n```bash\ncd multi-model-server\n\nmulti-model-server --start --model-store examples --models squeezenet=onnx-squeezenet.mar\n```\n\nAfter your server starts, you can use the following command to see the prediction results.\n\n```bash\ncurl -X POST http://127.0.0.1:8080/predictions/squeezenet -T docs/images/kitten_small.jpg\n```\n" }, { "path": "model-archiver/model_archiver/__init__.py", "content": "# Copyright 2018 Amazon.com, Inc. or its affiliates. All Rights Reserved.\n# Licensed under the Apache License, Version 2.0 (the \"License\").\n# You may not use this file except in compliance with the License.\n# A copy of the License is located at\n# http://www.apache.org/licenses/LICENSE-2.0\n# or in the \"license\" file accompanying this file. This file is distributed\n# on an \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either\n# express or implied. See the License for the specific language governing\n# permissions and limitations under the License.\n\n\"\"\"\nThis module does the following:\nExports the model folder to generate a Model Archive file out of it in .mar format\n\"\"\"\nfrom . import version\n\n__version__ = version.__version__\n" }, { "path": "model-archiver/model_archiver/arg_parser.py", "content": "# Copyright 2017 Amazon.com, Inc. or its affiliates. All Rights Reserved.\n# Licensed under the Apache License, Version 2.0 (the \"License\").\n# You may not use this file except in compliance with the License.\n# A copy of the License is located at\n# http://www.apache.org/licenses/LICENSE-2.0\n# or in the \"license\" file accompanying this file. This file is distributed\n# on an \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either\n# express or implied. See the License for the specific language governing\n# permissions and limitations under the License.\n\n\"\"\"\nThis module parses the arguments given through the multi-model-server command-line. This is used by model-server\nat runtime.\n\"\"\"\n\nimport argparse\nimport os\nfrom .manifest_components.manifest import RuntimeType\n\n\n# noinspection PyTypeChecker\nclass ArgParser(object):\n\n \"\"\"\n Argument parser for model-export-tool commands\n More detailed example is available at https://github.com/awslabs/multi-model-server/blob/master/README.md\n \"\"\"\n\n @staticmethod\n def export_model_args_parser():\n\n \"\"\" Argument parser for multi-model-export\n \"\"\"\n\n parser_export = argparse.ArgumentParser(prog='model-archiver', description='Model Archiver Tool',\n formatter_class=argparse.RawTextHelpFormatter)\n\n parser_export.add_argument('--model-name',\n required=True,\n type=str,\n default=None,\n help='Exported model name. Exported file will be named as\\n'\n 'model-name.mar and saved in current working directory if no --export-path is\\n'\n 'specified, else it will be saved under the export path')\n\n parser_export.add_argument('--model-path',\n required=True,\n type=str,\n default=None,\n help='Path to the folder containing model related files.')\n\n parser_export.add_argument('--handler',\n required=True,\n dest=\"handler\",\n type=str,\n default=None,\n help='Handler path to handle custom MMS inference logic.')\n\n parser_export.add_argument('--runtime',\n required=False,\n type=str,\n default=RuntimeType.PYTHON.value,\n choices=[s.value for s in RuntimeType],\n help='The runtime specifies which language to run your inference code on.\\n'\n 'The default runtime is \"python\".')\n\n parser_export.add_argument('--export-path',\n required=False,\n type=str,\n default=os.getcwd(),\n help='Path where the exported .mar file will be saved. This is an optional\\n'\n 'parameter. If --export-path is not specified, the file will be saved in the\\n'\n 'current working directory. ')\n\n parser_export.add_argument('--archive-format',\n required=False,\n type=str,\n default=\"default\",\n choices=[\"tgz\", \"no-archive\", \"default\"],\n help='The format in which the model artifacts are archived.\\n'\n '\"tgz\": This creates the model-archive in .tar.gz format.\\n'\n 'If platform hosting MMS requires model-artifacts to be in \".tar.gz\"\\n'\n 'use this option.\\n'\n '\"no-archive\": This option creates an non-archived version of model artifacts\\n'\n 'at \"export-path/{model-name}\" location. As a result of this choice, \\n'\n 'MANIFEST file will be created at \"export-path/{model-name}\" location\\n'\n 'without archiving these model files\\n'\n '\"default\": This creates the model-archive in .mar format.\\n'\n 'This is the default archiving format. Models archived in this format\\n'\n 'will be readily hostable on native MMS.\\n')\n\n parser_export.add_argument('-f', '--force',\n required=False,\n action='store_true',\n help='When the -f or --force flag is specified, an existing .mar file with same\\n'\n 'name as that provided in --model-name in the path specified by --export-path\\n'\n 'will overwritten')\n\n parser_export.add_argument('-c', '--convert',\n required=False,\n action='store_true',\n help='When this option is used, model-archiver looks for special files and tries\\n'\n 'preprocesses them. For example, if this option is chosen when running\\n'\n 'model-archiver tool on a model with \".onnx\" extension, the tool will try and\\n'\n 'convert \".onnx\" model into an Multi model.')\n\n\n return parser_export\n" }, { "path": "model-archiver/model_archiver/manifest_components/__init__.py", "content": "# Copyright 2018 Amazon.com, Inc. or its affiliates. All Rights Reserved.\n# Licensed under the Apache License, Version 2.0 (the \"License\").\n# You may not use this file except in compliance with the License.\n# A copy of the License is located at\n# http://www.apache.org/licenses/LICENSE-2.0\n# or in the \"license\" file accompanying this file. This file is distributed\n# on an \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either\n# express or implied. See the License for the specific language governing\n# permissions and limitations under the License.\n" }, { "path": "model-archiver/model_archiver/manifest_components/engine.py", "content": "# Copyright 2018 Amazon.com, Inc. or its affiliates. All Rights Reserved.\n# Licensed under the Apache License, Version 2.0 (the \"License\").\n# You may not use this file except in compliance with the License.\n# A copy of the License is located at\n# http://www.apache.org/licenses/LICENSE-2.0\n# or in the \"license\" file accompanying this file. This file is distributed\n# on an \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either\n# express or implied. See the License for the specific language governing\n# permissions and limitations under the License.\n\n# pylint: disable=missing-docstring\nimport json\nfrom enum import Enum\n\n\nclass EngineType(Enum):\n MXNET = \"MXNet\"\n\n # TODO Add more engines here as and when MMS supports more DL Frameworks\n\n\nclass Engine(object):\n \"\"\"\n Engine is a part of the final manifest.json. It defines which framework to run the inference on\n \"\"\"\n\n def __init__(self, engine_name, engine_version=None):\n self.engine_name = EngineType(engine_name)\n self.engine_version = engine_version\n\n self.engine_dict = self.__to_dict__()\n\n def __to_dict__(self):\n engine_dict = dict()\n engine_dict['engineName'] = self.engine_name.value\n\n if self.engine_version is not None:\n engine_dict['engineVersion'] = self.engine_version\n\n return engine_dict\n\n def __str__(self):\n return json.dumps(self.engine_dict)\n\n def __repr__(self):\n return json.dumps(self.engine_dict)\n" }, { "path": "model-archiver/model_archiver/manifest_components/manifest.py", "content": "# Copyright 2018 Amazon.com, Inc. or its affiliates. All Rights Reserved.\n# Licensed under the Apache License, Version 2.0 (the \"License\").\n# You may not use this file except in compliance with the License.\n# A copy of the License is located at\n# http://www.apache.org/licenses/LICENSE-2.0\n# or in the \"license\" file accompanying this file. This file is distributed\n# on an \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either\n# express or implied. See the License for the specific language governing\n# permissions and limitations under the License.\n\n# pylint: disable=redefined-builtin\n# pylint: disable=missing-docstring\nimport json\nfrom enum import Enum\n\n\nclass RuntimeType(Enum):\n\n PYTHON = \"python\"\n PYTHON2 = \"python2\"\n PYTHON3 = \"python3\"\n\n # TODO : Add more runtimes here when we support more runtimes such as Java/Go/Scala etc..\n\n\nclass Manifest(object):\n \"\"\"\n The main manifest object which gets written into the model archive as MANIFEST.json\n \"\"\"\n\n def __init__(self, runtime, model, engine=None, specification_version='1.0', implementation_version='1.0',\n description=None, publisher=None, model_server_version='1.0', license=None, user_data=None):\n\n self.runtime = RuntimeType(runtime)\n self.engine = engine\n self.model = model\n self.publisher = publisher\n self.specification_version = specification_version\n self.implementation_version = implementation_version\n self.model_server_version = model_server_version\n self.license = license\n self.description = description\n self.user_data = user_data\n self.manifest_dict = self.__to_dict__()\n\n def __to_dict__(self):\n manifest_dict = dict()\n\n manifest_dict['runtime'] = self.runtime.value\n\n manifest_dict['model'] = self.model.__to_dict__()\n\n if self.engine is not None:\n manifest_dict['engine'] = self.engine.__to_dict__()\n\n if self.license is not None:\n manifest_dict['license'] = self.license\n\n if self.model_server_version is not None:\n manifest_dict['modelServerVersion'] = self.model_server_version\n\n if self.description is not None:\n manifest_dict['description'] = self.description\n\n if self.implementation_version is not None:\n manifest_dict['implementationVersion'] = self.implementation_version\n\n if self.specification_version is not None:\n manifest_dict['specificationVersion'] = self.specification_version\n\n if self.user_data is not None:\n manifest_dict['userData'] = self.user_data\n\n if self.publisher is not None:\n manifest_dict['publisher'] = self.publisher.__to_dict__()\n\n return manifest_dict\n\n def __str__(self):\n return json.dumps(self.manifest_dict, indent=2)\n\n def __repr__(self):\n return json.dumps(self.manifest_dict, indent=2)\n" }, { "path": "model-archiver/model_archiver/manifest_components/model.py", "content": "# Copyright 2018 Amazon.com, Inc. or its affiliates. All Rights Reserved.\n# Licensed under the Apache License, Version 2.0 (the \"License\").\n# You may not use this file except in compliance with the License.\n# A copy of the License is located at\n# http://www.apache.org/licenses/LICENSE-2.0\n# or in the \"license\" file accompanying this file. This file is distributed\n# on an \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either\n# express or implied. See the License for the specific language governing\n# permissions and limitations under the License.\n\n# pylint: disable=missing-docstring\nimport json\n\n\nclass Model(object):\n \"\"\"\n Model is a part of the manifest.json. It defines the properties of the model such as name, version as weill\n as the entry point into the service code through the handler property\n \"\"\"\n\n def __init__(self, model_name, handler, description=None, model_version=None, extensions=None):\n self.model_name = model_name\n self.description = description\n self.model_version = model_version\n self.extensions = extensions\n self.handler = handler\n self.model_dict = self.__to_dict__()\n\n def __to_dict__(self):\n model_dict = dict()\n\n model_dict['modelName'] = self.model_name\n\n model_dict['handler'] = self.handler\n\n if self.description is not None:\n model_dict['description'] = self.description\n\n if self.model_version is not None:\n model_dict['modelVersion'] = self.model_version\n\n if self.extensions is not None:\n model_dict['extensions'] = self.extensions\n\n return model_dict\n\n def __str__(self):\n return json.dumps(self.model_dict)\n\n def __repr__(self):\n return json.dumps(self.model_dict)\n" }, { "path": "model-archiver/model_archiver/manifest_components/publisher.py", "content": "# Copyright 2018 Amazon.com, Inc. or its affiliates. All Rights Reserved.\n# Licensed under the Apache License, Version 2.0 (the \"License\").\n# You may not use this file except in compliance with the License.\n# A copy of the License is located at\n# http://www.apache.org/licenses/LICENSE-2.0\n# or in the \"license\" file accompanying this file. This file is distributed\n# on an \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either\n# express or implied. See the License for the specific language governing\n# permissions and limitations under the License.\n\n# pylint: disable=missing-docstring\nimport json\n\n\nclass Publisher(object):\n \"\"\"\n Publisher object is a part of Manifest.json\n \"\"\"\n\n def __init__(self, author, email):\n self.author = author\n self.email = email\n self.pub_dict = self.__to_dict__()\n\n def __to_dict__(self):\n pub_dict = dict()\n pub_dict['author'] = self.author\n pub_dict['email'] = self.email\n\n return pub_dict\n\n def __str__(self):\n return json.dumps(self.pub_dict)\n\n def __repr__(self):\n return json.dumps(self.pub_dict)\n" }, { "path": "model-archiver/model_archiver/model_archiver_error.py", "content": "# Copyright 2018 Amazon.com, Inc. or its affiliates. All Rights Reserved.\n# Licensed under the Apache License, Version 2.0 (the \"License\").\n# You may not use this file except in compliance with the License.\n# A copy of the License is located at\n# http://www.apache.org/licenses/LICENSE-2.0\n# or in the \"license\" file accompanying this file. This file is distributed\n# on an \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either\n# express or implied. See the License for the specific language governing\n# permissions and limitations under the License.\n\"\"\"\nModel Archiver Error\n\"\"\"\n\n\nclass ModelArchiverError(Exception):\n \"\"\"\n Error for Model Archiver module\n \"\"\"\n def __init__(self, message):\n super(ModelArchiverError, self).__init__(message)\n" }, { "path": "model-archiver/model_archiver/model_packaging.py", "content": "# Copyright 2018 Amazon.com, Inc. or its affiliates. All Rights Reserved.\n# Licensed under the Apache License, Version 2.0 (the \"License\").\n# You may not use this file except in compliance with the License.\n# A copy of the License is located at\n# http://www.apache.org/licenses/LICENSE-2.0\n# or in the \"license\" file accompanying this file. This file is distributed\n# on an \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either\n# express or implied. See the License for the specific language governing\n# permissions and limitations under the License.\n\n\"\"\"\nCommand line interface to export model files to be used for inference by Multi Model Server\n\"\"\"\n\nimport logging\nimport sys\nfrom .arg_parser import ArgParser\nfrom .model_packaging_utils import ModelExportUtils\nfrom .model_archiver_error import ModelArchiverError\n\n\ndef package_model(args, manifest):\n \"\"\"\n Internal helper for the exporting model command line interface.\n \"\"\"\n model_path = args.model_path\n model_name = args.model_name\n export_file_path = args.export_path\n temp_files = []\n\n try:\n ModelExportUtils.validate_inputs(model_path, model_name, export_file_path)\n # Step 1 : Check if .mar already exists with the given model name\n export_file_path = ModelExportUtils.check_mar_already_exists(model_name, export_file_path,\n args.force, args.archive_format)\n\n # Step 2 : Check if any special handling is required for custom models like onnx models\n files_to_exclude = []\n if args.convert:\n t, files_to_exclude = ModelExportUtils.check_custom_model_types(model_path, model_name)\n temp_files.extend(t)\n\n # Step 3 : Zip 'em all up\n ModelExportUtils.archive(export_file_path, model_name, model_path, files_to_exclude, manifest,\n args.archive_format)\n\n logging.info(\"Successfully exported model %s to file %s\", model_name, export_file_path)\n except ModelArchiverError as e:\n logging.error(e)\n sys.exit(1)\n finally:\n ModelExportUtils.clean_temp_files(temp_files)\n\n\ndef generate_model_archive():\n \"\"\"\n Generate a model archive file\n :return:\n \"\"\"\n logging.basicConfig(format='%(levelname)s - %(message)s')\n args = ArgParser.export_model_args_parser().parse_args()\n manifest = ModelExportUtils.generate_manifest_json(args)\n package_model(args, manifest=manifest)\n\n\nif __name__ == '__main__':\n generate_model_archive()\n" }, { "path": "model-archiver/model_archiver/model_packaging_utils.py", "content": "# Copyright 2018 Amazon.com, Inc. or its affiliates. All Rights Reserved.\n# Licensed under the Apache License, Version 2.0 (the \"License\").\n# You may not use this file except in compliance with the License.\n# A copy of the License is located at\n# http://www.apache.org/licenses/LICENSE-2.0\n# or in the \"license\" file accompanying this file. This file is distributed\n# on an \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either\n# express or implied. See the License for the specific language governing\n# permissions and limitations under the License.\n\n\"\"\"\nHelper utils for Model Export tool\n\"\"\"\n\nimport json\nimport logging\nimport os\nimport re\nimport zipfile\nimport shutil\nfrom .model_archiver_error import ModelArchiverError\n\nfrom .manifest_components.engine import Engine\nfrom .manifest_components.manifest import Manifest\nfrom .manifest_components.model import Model\nfrom .manifest_components.publisher import Publisher\n\narchiving_options = {\n \"tgz\": \".tar.gz\",\n \"no-archive\": \"\",\n \"default\": \".mar\"\n}\n\nMODEL_SERVER_VERSION = '1.0'\nMODEL_ARCHIVE_VERSION = '1.0'\nMANIFEST_FILE_NAME = 'MANIFEST.json'\nMAR_INF = 'MAR-INF'\nONNX_TYPE = '.onnx'\n\n\nclass ModelExportUtils(object):\n \"\"\"\n Helper utils for Model Archiver tool.\n This class lists out all the methods such as validations for model archiving, ONNX model checking etc.\n \"\"\"\n\n @staticmethod\n def get_archive_export_path(export_file_path, model_name, archive_format):\n return os.path.join(export_file_path, '{}{}'.format(model_name, archiving_options.get(archive_format)))\n\n @staticmethod\n def check_mar_already_exists(model_name, export_file_path, overwrite, archive_format=\"default\"):\n \"\"\"\n Function to check if .mar already exists\n :param archive_format:\n :param model_name:\n :param export_file_path:\n :param overwrite:\n :return:\n \"\"\"\n if export_file_path is None:\n export_file_path = os.getcwd()\n\n export_file = ModelExportUtils.get_archive_export_path(export_file_path, model_name, archive_format)\n\n if os.path.exists(export_file):\n if overwrite:\n logging.warning(\"Overwriting %s ...\", export_file)\n else:\n raise ModelArchiverError(\"%s already exists.\\n\"\n \"Please specify --force/-f option to overwrite the model archive \"\n \"output file.\\n\"\n \"See -h/--help for more details.\" + export_file)\n\n return export_file_path\n\n @staticmethod\n def check_custom_model_types(model_path, model_name=None):\n \"\"\"\n This functions checks whether any special handling is required for custom model extensions such as\n .onnx, or in the future, for Tensorflow and PyTorch extensions.\n :param model_path:\n :param model_name:\n :return:\n \"\"\"\n temp_files = [] # List of temp files added to handle custom models\n files_to_exclude = [] # List of files to be excluded from .mar packaging.\n\n files_set = set(os.listdir(model_path))\n onnx_file = ModelExportUtils.find_unique(files_set, ONNX_TYPE)\n if onnx_file is not None:\n logging.debug(\"Found ONNX files. Converting ONNX file to model archive...\")\n symbol_file, params_file = ModelExportUtils.convert_onnx_model(model_path, onnx_file, model_name)\n files_to_exclude.append(onnx_file)\n temp_files.append(os.path.join(model_path, symbol_file))\n temp_files.append(os.path.join(model_path, params_file))\n\n # More cases will go here as an if-else block\n\n return temp_files, files_to_exclude\n\n @staticmethod\n def find_unique(files, suffix):\n \"\"\"\n Function to find unique model params file\n :param files:\n :param suffix:\n :return:\n \"\"\"\n match = [f for f in files if f.endswith(suffix)]\n count = len(match)\n\n if count == 0:\n return None\n elif count == 1:\n return match[0]\n else:\n raise ModelArchiverError(\"model-archiver expects only one {} file in the folder.\"\n \" Found {} files {} in model-path.\".format(suffix, count, match))\n\n @staticmethod\n def convert_onnx_model(model_path, onnx_file, model_name):\n \"\"\"\n Util to convert onnx model to MXNet model\n :param model_name:\n :param model_path:\n :param onnx_file:\n :return:\n \"\"\"\n try:\n import mxnet as mx\n from mxnet.contrib import onnx as onnx_mxnet\n except ImportError:\n raise ModelArchiverError(\"MXNet package is not installed. Run command: pip install mxnet to install it.\")\n\n try:\n import onnx\n except ImportError:\n raise ModelArchiverError(\"Onnx package is not installed. Run command: pip install onnx to install it.\")\n\n symbol_file = '%s-symbol.json' % model_name\n params_file = '%s-0000.params' % model_name\n signature_file = 'signature.json'\n # Find input symbol name and shape\n try:\n model_proto = onnx.load(os.path.join(model_path, onnx_file))\n except:\n logging.error(\"Failed to load the %s model. Verify if the model file is valid\", onnx_file)\n raise\n\n graph = model_proto.graph\n _params = set()\n for tensor_vals in graph.initializer:\n _params.add(tensor_vals.name)\n\n input_data = []\n for graph_input in graph.input:\n shape = []\n if graph_input.name not in _params:\n for val in graph_input.type.tensor_type.shape.dim:\n shape.append(val.dim_value)\n input_data.append((graph_input.name, tuple(shape)))\n\n try:\n sym, arg_params, aux_params = onnx_mxnet.import_model(os.path.join(model_path, onnx_file))\n # UNION of argument and auxillary parameters\n params = dict(arg_params, **aux_params)\n except:\n logging.error(\"Failed to import %s file to onnx. Verify if the model file is valid\", onnx_file)\n raise\n\n try:\n # rewrite input data_name correctly\n with open(os.path.join(model_path, signature_file), 'r') as f:\n data = json.loads(f.read())\n data['inputs'][0]['data_name'] = input_data[0][0]\n data['inputs'][0]['data_shape'] = [int(i) for i in input_data[0][1]]\n with open(os.path.join(model_path, signature_file), 'w') as f:\n f.write(json.dumps(data, indent=2))\n\n with open(os.path.join(model_path, symbol_file), 'w') as f:\n f.write(sym.tojson())\n except:\n logging.error(\"Failed to write the signature or symbol files for %s model\", onnx_file)\n raise\n\n save_dict = {('arg:%s' % k): v.as_in_context(mx.cpu()) for k, v in params.items()}\n mx.nd.save(os.path.join(model_path, params_file), save_dict)\n return symbol_file, params_file\n\n @staticmethod\n def generate_publisher(publisherargs):\n publisher = Publisher(author=publisherargs.author, email=publisherargs.email)\n return publisher\n\n @staticmethod\n def generate_engine(engineargs):\n engine = Engine(engine_name=engineargs.engine)\n return engine\n\n @staticmethod\n def generate_model(modelargs):\n model = Model(model_name=modelargs.model_name, handler=modelargs.handler)\n return model\n\n @staticmethod\n def generate_manifest_json(args):\n \"\"\"\n Function to generate manifest as a json string from the inputs provided by the user in the command line\n :param args:\n :return:\n \"\"\"\n arg_dict = vars(args)\n\n publisher = ModelExportUtils.generate_publisher(args) if 'author' in arg_dict and 'email' in arg_dict else None\n\n engine = ModelExportUtils.generate_engine(args) if 'engine' in arg_dict else None\n\n model = ModelExportUtils.generate_model(args)\n\n manifest = Manifest(runtime=args.runtime, model=model, engine=engine, publisher=publisher)\n\n return str(manifest)\n\n @staticmethod\n def clean_temp_files(temp_files):\n for f in temp_files:\n os.remove(f)\n\n @staticmethod\n def make_dir(d):\n if not os.path.isdir(d):\n os.makedirs(d)\n\n @staticmethod\n def archive(export_file, model_name, model_path, files_to_exclude, manifest, archive_format=\"default\"):\n \"\"\"\n Create a model-archive\n :param archive_format:\n :param export_file:\n :param model_name:\n :param model_path:\n :param files_to_exclude:\n :param manifest:\n :return:\n \"\"\"\n mar_path = ModelExportUtils.get_archive_export_path(export_file, model_name, archive_format)\n try:\n if archive_format == \"tgz\":\n import tarfile\n from io import BytesIO\n with tarfile.open(mar_path, 'w:gz') as z:\n ModelExportUtils.archive_dir(model_path, z, set(files_to_exclude), archive_format, model_name)\n # Write the manifest here now as a json\n tar_manifest = tarfile.TarInfo(name=os.path.join(model_name, MAR_INF, MANIFEST_FILE_NAME))\n tar_manifest.size = len(manifest.encode('utf-8'))\n z.addfile(tarinfo=tar_manifest, fileobj=BytesIO(manifest.encode()))\n z.close()\n elif archive_format == \"no-archive\":\n if model_path != mar_path:\n # Copy files to export path if\n ModelExportUtils.archive_dir(model_path, mar_path,\n set(files_to_exclude), archive_format, model_name)\n # Write the MANIFEST in place\n manifest_path = os.path.join(mar_path, MAR_INF)\n ModelExportUtils.make_dir(manifest_path)\n with open(os.path.join(manifest_path, MANIFEST_FILE_NAME), \"w\") as f:\n f.write(manifest)\n else:\n with zipfile.ZipFile(mar_path, 'w', zipfile.ZIP_DEFLATED) as z:\n ModelExportUtils.archive_dir(model_path, z, set(files_to_exclude), archive_format, model_name)\n # Write the manifest here now as a json\n z.writestr(os.path.join(MAR_INF, MANIFEST_FILE_NAME), manifest)\n except IOError:\n logging.error(\"Failed to save the model-archive to model-path \\\"%s\\\". \"\n \"Check the file permissions and retry.\", export_file)\n raise\n except:\n logging.error(\"Failed to convert %s to the model-archive.\", model_name)\n raise\n\n @staticmethod\n def archive_dir(path, dst, files_to_exclude, archive_format, model_name):\n\n \"\"\"\n This method zips the dir and filters out some files based on a expression\n :param archive_format:\n :param path:\n :param dst:\n :param model_name:\n :param files_to_exclude:\n :return:\n \"\"\"\n unwanted_dirs = {'__MACOSX', '__pycache__'}\n\n for root, directories, files in os.walk(path):\n # Filter directories\n directories[:] = [d for d in directories if ModelExportUtils.directory_filter(d, unwanted_dirs)]\n # Filter files\n files[:] = [f for f in files if ModelExportUtils.file_filter(f, files_to_exclude)]\n for f in files:\n file_path = os.path.join(root, f)\n if archive_format == \"tgz\":\n dst.add(file_path, arcname=os.path.join(model_name, os.path.relpath(file_path, path)))\n elif archive_format == \"no-archive\":\n dst_dir = os.path.dirname(os.path.join(dst, os.path.relpath(file_path, path)))\n ModelExportUtils.make_dir(dst_dir)\n shutil.copy(file_path, dst_dir)\n else:\n dst.write(file_path, os.path.relpath(file_path, path))\n\n @staticmethod\n def directory_filter(directory, unwanted_dirs):\n \"\"\"\n This method weeds out unwanted hidden directories from the model archive .mar file\n :param directory:\n :param unwanted_dirs:\n :return:\n \"\"\"\n if directory in unwanted_dirs:\n return False\n if directory.startswith('.'):\n return False\n\n return True\n\n @staticmethod\n def file_filter(current_file, files_to_exclude):\n \"\"\"\n This method weeds out unwanted files\n :param current_file:\n :param files_to_exclude:\n :return:\n \"\"\"\n files_to_exclude.add('MANIFEST.json')\n if current_file in files_to_exclude:\n return False\n\n elif current_file.endswith(('.pyc', '.DS_Store', '.mar')):\n return False\n\n return True\n\n @staticmethod\n def check_model_name_regex_or_exit(model_name):\n \"\"\"\n Method checks whether model name passes regex filter.\n If the regex Filter fails, the method exits.\n :param model_name:\n :return:\n \"\"\"\n if not re.match(r'^[A-Za-z0-9][A-Za-z0-9_\\-.]*$', model_name):\n raise ModelArchiverError(\"Model name contains special characters.\\n\"\n \"The allowed regular expression filter for model \"\n \"name is: ^[A-Za-z0-9][A-Za-z0-9_\\\\-.]*$\")\n\n @staticmethod\n def validate_inputs(model_path, model_name, export_path):\n ModelExportUtils.check_model_name_regex_or_exit(model_name)\n if not os.path.isdir(os.path.abspath(export_path)):\n raise ModelArchiverError(\"Given export-path {} is not a directory. \"\n \"Point to a valid export-path directory.\".format(export_path))\n\n if not os.path.isdir(os.path.abspath(model_path)):\n raise ModelArchiverError(\"Given model-path {} is not a valid directory. \"\n \"Point to a valid model-path directory.\".format(model_path))\n" }, { "path": "model-archiver/model_archiver/tests/integ_tests/configuration.json", "content": " [{\n \"name\": \"packaging_mar\",\n \"modelName\": \"model\",\n \"modelPath\": \"model_archiver/tests/integ_tests/resources/regular_model\",\n \"handler\": \"service:handle\",\n \"exportPath\": \"/tmp/model\",\n \"archiveFormat\": \"default\",\n \"iterations\": 1\n },\n {\n \"name\": \"packaging_noarchive\",\n \"modelName\": \"model\",\n \"modelPath\": \"model_archiver/tests/integ_tests/resources/regular_model\",\n \"handler\": \"service:handle\",\n \"exportPath\": \"/tmp/model\",\n \"archiveFormat\": \"no-archive\",\n \"iterations\": 1\n },\n {\n \"name\": \"packaging_tgz\",\n \"modelName\": \"model\",\n \"modelPath\": \"model_archiver/tests/integ_tests/resources/regular_model\",\n \"handler\": \"service:handle\",\n \"exportPath\": \"/tmp/model\",\n \"archiveFormat\": \"tgz\",\n \"iterations\": 1\n },\n {\n \"name\": \"packaging_mar\",\n \"modelName\": \"model\",\n \"modelPath\": \"model_archiver/tests/integ_tests/resources/regular_model\",\n \"handler\": \"service:handle\",\n \"exportPath\": \"/tmp/model\",\n \"archiveFormat\": \"default\",\n \"iterations\": 2,\n \"expectError\": true\n },\n {\n \"name\": \"packaging_onnx_mar\",\n \"modelName\": \"model\",\n \"modelPath\": \"model_archiver/tests/integ_tests/resources/onnx_model\",\n \"handler\": \"service:handle\",\n \"exportPath\": \"/tmp/model\",\n \"archiveFormat\": \"default\",\n \"iterations\": 1\n },\n {\n \"name\": \"packaging_onnx_tgz\",\n \"modelName\": \"model\",\n \"modelPath\": \"model_archiver/tests/integ_tests/resources/onnx_model\",\n \"handler\": \"service:handle\",\n \"exportPath\": \"/tmp/model\",\n \"archiveFormat\": \"tgz\",\n \"iterations\": 1\n },\n {\n \"name\": \"packaging_onnx_no_archive\",\n \"modelName\": \"model\",\n \"modelPath\": \"model_archiver/tests/integ_tests/resources/onnx_model\",\n \"handler\": \"service:handle\",\n \"exportPath\": \"/tmp/model\",\n \"archiveFormat\": \"no-archive\",\n \"iterations\": 1\n },\n {\n \"name\": \"packaging_mar\",\n \"modelName\": \"model\",\n \"modelPath\": \"model_archiver/tests/integ_tests/resources/regular_model\",\n \"handler\": \"service:handle\",\n \"exportPath\": \"/tmp/model\",\n \"archiveFormat\": \"default\",\n \"iterations\": 2,\n \"force\": true\n } ]" }, { "path": "model-archiver/model_archiver/tests/integ_tests/resources/onnx_model/service.py", "content": "\"\"\"\nThis is a dummy source fiile\n\"\"\"\n\n\ndef handle():\n return \"Dummy model\"\n" }, { "path": "model-archiver/model_archiver/tests/integ_tests/resources/regular_model/dir/1.py", "content": "__version__ = 1" }, { "path": "model-archiver/model_archiver/tests/integ_tests/resources/regular_model/dummy-artifacts.txt", "content": "Some random text." }, { "path": "model-archiver/model_archiver/tests/integ_tests/resources/regular_model/service.py", "content": "\"\"\"\nThis is a dummy source fiile\n\"\"\"\n\n\ndef handle():\n return \"Dummy model\"\n" }, { "path": "model-archiver/model_archiver/tests/integ_tests/test_integration_model_archiver.py", "content": "import errno\nimport json\nimport os\nimport shutil\nimport subprocess\nimport requests\n\nDEFAULT_MODEL_PATH = \"model_archiver/tests/integ_tests/resources/regular_model\"\nDEFAULT_HANDLER = \"service:handle\"\nDEFAULT_RUNTIME = \"python\"\nDEFAULT_MODEL_NAME = \"model\"\nDEFAULT_EXPORT_PATH = \"/tmp/model\"\nMANIFEST_FILE = \"MAR-INF/MANIFEST.json\"\n\n\ndef update_tests(test):\n test[\"modelName\"] = test.get(\"modelName\", DEFAULT_MODEL_NAME)\n test[\"modelPath\"] = test.get(\"modelPath\", DEFAULT_MODEL_PATH)\n test[\"handler\"] = test.get(\"handler\", DEFAULT_HANDLER)\n test[\"runtime\"] = test.get(\"runtime\", DEFAULT_RUNTIME)\n test[\"exportPath\"] = test.get(\"exportPath\", DEFAULT_EXPORT_PATH)\n test[\"archiveFormat\"] = test.get(\"archiveFormat\", \"default\")\n return test\n\n\ndef create_file_path(path):\n try:\n os.makedirs(path)\n except OSError as exc:\n if exc.errno == errno.EEXIST and os.path.isdir(path):\n pass\n else:\n raise\n\n\ndef delete_file_path(path):\n try:\n if os.path.isfile(path):\n os.remove(path)\n if os.path.isdir(path):\n shutil.rmtree(path)\n except OSError:\n pass\n\n\ndef run_test(test, cmd):\n it = test.get(\"iterations\") if test.get(\"iterations\") is not None else 1\n for i in range(it):\n try:\n subprocess.check_call(cmd, shell=True)\n except subprocess.CalledProcessError as exc:\n if test.get(\"expectError\") is not True:\n assert 0, \"{}\".format(exc.output)\n else:\n return 0\n return 1\n\n\ndef validate_archive_exists(test):\n fmt = test.get(\"archiveFormat\")\n if fmt == \"tgz\":\n assert os.path.isfile(os.path.join(test.get(\"exportPath\"), test.get(\"modelName\")+\".tar.gz\"))\n elif fmt == \"no-archive\":\n assert os.path.isdir(os.path.join(test.get(\"exportPath\"), test.get(\"modelName\")))\n else:\n assert os.path.isfile(os.path.join(test.get(\"exportPath\"), test.get(\"modelName\")+\".mar\"))\n\n\ndef validate_manifest_file(manifest, test):\n \"\"\"\n Validate the MANIFEST file\n :param manifest:\n :param test:\n :return:\n \"\"\"\n assert manifest.get(\"runtime\") == test.get(\"runtime\")\n assert manifest.get(\"model\").get(\"modelName\") == test.get(\"modelName\")\n assert manifest.get(\"model\").get(\"handler\") == test.get(\"handler\")\n\n\ndef validate_files(file_list, prefix, regular):\n assert os.path.join(prefix, MANIFEST_FILE) in file_list\n assert os.path.join(prefix, \"service.py\") in file_list\n\n if regular:\n assert os.path.join(prefix, \"dummy-artifacts.txt\") in file_list\n assert os.path.join(prefix, \"dir/1.py\") in file_list\n else:\n assert os.path.join(prefix, \"model.onnx\") in file_list\n\n\ndef validate_tar_archive(test_cfg):\n import tarfile\n file_name = os.path.join(test_cfg.get(\"exportPath\"), test_cfg.get(\"modelName\") + \".tar.gz\")\n f = tarfile.open(file_name, \"r:gz\")\n manifest = json.loads(f.extractfile(os.path.join(test_cfg.get(\"modelName\"), MANIFEST_FILE)).read())\n validate_manifest_file(manifest, test_cfg)\n validate_files(f.getnames(), test_cfg.get(\"modelName\"), \"regular_model\" in test_cfg.get(\"modelPath\"))\n\n\ndef validate_noarchive_archive(test):\n file_name = os.path.join(test.get(\"exportPath\"), test.get(\"modelName\"), MANIFEST_FILE)\n manifest = json.loads(open(file_name).read())\n validate_manifest_file(manifest, test)\n\n\ndef validate_mar_archive(test):\n import zipfile\n file_name = os.path.join(test.get(\"exportPath\"), test.get(\"modelName\") + \".mar\")\n zf = zipfile.ZipFile(file_name, \"r\")\n manifest = json.loads(zf.open(MANIFEST_FILE).read())\n validate_manifest_file(manifest, test)\n\n\ndef validate_archive_content(test):\n fmt = test.get(\"archiveFormat\")\n if fmt == \"tgz\":\n validate_tar_archive(test)\n if fmt == \"no-archive\":\n validate_noarchive_archive(test)\n if fmt == \"default\":\n validate_mar_archive(test)\n\n\ndef validate(test):\n validate_archive_exists(test)\n validate_archive_content(test)\n\n\ndef test_model_archiver():\n \n f = open(\"model_archiver/tests/integ_tests/configuration.json\", \"r\")\n tests = json.loads(f.read())\n for t in tests:\n try:\n delete_file_path(t.get(\"exportPath\"))\n create_file_path(t.get(\"exportPath\"))\n t = update_tests(t)\n cmd = \"model-archiver \" \\\n \"--model-name {} \" \\\n \"--model-path {} \" \\\n \"--handler {} \" \\\n \"--runtime {} \" \\\n \"--export-path {} \" \\\n \"--archive-format {}\".format(t.get(\"modelName\"),\n t.get(\"modelPath\"),\n t.get(\"handler\"),\n t.get(\"runtime\"),\n t.get(\"exportPath\"),\n t.get(\"archiveFormat\"))\n if t.get(\"force\"):\n cmd += \" -f\"\n\n # TODO: Add tests to check for \"convert\" functionality\n if run_test(t, cmd):\n validate(t)\n finally:\n delete_file_path(t.get(\"exportPath\"))\n\n\nif __name__ == \"__main__\":\n test_model_archiver()\n" }, { "path": "model-archiver/model_archiver/tests/pylintrc", "content": "[MASTER]\n\n# Licensed to the Apache Software Foundation (ASF) under one\n# or more contributor license agreements. See the NOTICE file\n# distributed with this work for additional information\n# regarding copyright ownership. The ASF licenses this file\n# to you under the Apache License, Version 2.0 (the\n# \"License\"); you may not use this file except in compliance\n# with the License. You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing,\n# software distributed under the License is distributed on an\n# \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n# KIND, either express or implied. See the License for the\n# specific language governing permissions and limitations\n# under the License.\n\n# Specify a configuration file.\n#rcfile=\n\n# Python code to execute, usually for sys.path manipulation such as\n# pygtk.require().\n#init-hook=\n\n# Add files or directories to the blacklist. They should be base names, not\n# paths.\nignore=CVS\n\n# Add files or directories matching the regex patterns to the blacklist. The\n# regex matches against base names, not paths.\nignore-patterns=\n\n# Pickle collected data for later comparisons.\npersistent=yes\n\n# List of plugins (as comma separated values of python modules names) to load,\n# usually to register additional checkers.\nload-plugins=\n\n# Use multiple processes to speed up Pylint.\njobs=8\n\n# Allow loading of arbitrary C extensions. Extensions are imported into the\n# active Python interpreter and may run arbitrary code.\nunsafe-load-any-extension=no\n\n# A comma-separated list of package or module names from where C extensions may\n# be loaded. Extensions are loading into the active Python interpreter and may\n# run arbitrary code\nextension-pkg-whitelist=numpy,opencv\n\n# Allow optimization of some AST trees. This will activate a peephole AST\n# optimizer, which will apply various small optimizations. For instance, it can\n# be used to obtain the result of joining multiple strings with the addition\n# operator. Joining a lot of strings can lead to a maximum recursion error in\n# Pylint and this flag can prevent that. It has one side effect, the resulting\n# AST will be different than the one from reality. This option is deprecated\n# and it will be removed in Pylint 2.0.\noptimize-ast=no\n\n\n[MESSAGES CONTROL]\n\n# Only show warnings with the listed confidence levels. Leave empty to show\n# all. Valid levels: HIGH, INFERENCE, INFERENCE_FAILURE, UNDEFINED\nconfidence=\n\n# Enable the message, report, category or checker with the given id(s). You can\n# either give multiple identifier separated by comma (,) or put this option\n# multiple time (only on the command line, not in the configuration file where\n# it should appear only once). See also the \"--disable\" option for examples.\nenable=indexing-exception,old-raise-syntax\n\n# Disable the message, report, category or checker with the given id(s). You\n# can either give multiple identifiers separated by comma (,) or put this\n# option multiple times (only on the command line, not in the configuration\n# file where it should appear only once).You can also use \"--disable=all\" to\n# disable everything first and then reenable specific checks. For example, if\n# you want to run only the similarities checker, you can use \"--disable=all\n# --enable=similarities\". If you want to run only the classes checker, but have\n# no Warning level messages displayed, use\"--disable=all --enable=classes\n# --disable=W\"\ndisable=design,similarities,no-self-use,attribute-defined-outside-init,locally-disabled,star-args,pointless-except,bad-option-value,global-statement,fixme,suppressed-message,useless-suppression,locally-enabled,no-member,no-name-in-module,import-error,unsubscriptable-object,unbalanced-tuple-unpacking,undefined-variable,protected-access,superfluous-parens,invalid-name,no-else-return,useless-super-delegation,len-as-condition,invalid-unary-operand-type,useless-object-inheritance\n# disable=unicode-builtin,delslice-method,using-cmp-argument,setslice-method,dict-view-method,parameter-unpacking,range-builtin-not-iterating,print-statement,file-builtin,old-raise-syntax,basestring-builtin,execfile-builtin,indexing-exception,import-star-module-level,coerce-method,long-builtin,old-ne-operator,old-division,no-absolute-import,raw_input-builtin,old-octal-literal,oct-method,xrange-builtin,hex-method,unpacking-in-except,nonzero-method,raising-string,intern-builtin,reload-builtin,metaclass-assignment,cmp-method,filter-builtin-not-iterating,apply-builtin,map-builtin-not-iterating,next-method-called,unichr-builtin,buffer-builtin,dict-iter-method,input-builtin,coerce-builtin,getslice-method,useless-suppression,standarderror-builtin,zip-builtin-not-iterating,suppressed-message,cmp-builtin,backtick,long-suffix,reduce-builtin,round-builtin\n\n\n[REPORTS]\n\n# Set the output format. Available formats are text, parseable, colorized, msvs\n# (visual studio) and html. You can also give a reporter class, eg\n# mypackage.mymodule.MyReporterClass.\noutput-format=text\n\n# Put messages in a separate file for each module / package specified on the\n# command line instead of printing them on stdout. Reports (if any) will be\n# written in a file name \"pylint_global.[txt|html]\". This option is deprecated\n# and it will be removed in Pylint 2.0.\nfiles-output=no\n\n# Tells whether to display a full report or only the messages\nreports=no\n\n# Python expression which should return a note less than 10 (10 is the highest\n# note). You have access to the variables errors warning, statement which\n# respectively contain the number of errors / warnings messages and the total\n# number of statements analyzed. This is used by the global evaluation report\n# (RP0004).\nevaluation=10.0 - ((float(5 * error + warning + refactor + convention) / statement) * 10)\n\n# Template used to display messages. This is a python new-style format string\n# used to format the message information. See doc for all details\n#msg-template=\n\n\n[FORMAT]\n\n# Maximum number of characters on a single line.\nmax-line-length=120\n\n# Regexp for a line that is allowed to be longer than the limit.\nignore-long-lines=^\\s*(# )??$\n\n# Allow the body of an if to be on the same line as the test if there is no\n# else.\nsingle-line-if-stmt=no\n\n# List of optional constructs for which whitespace checking is disabled. `dict-\n# separator` is used to allow tabulation in dicts, etc.: {1 : 1,\\n222: 2}.\n# `trailing-comma` allows a space between comma and closing bracket: (a, ).\n# `empty-line` allows space-only lines.\nno-space-check=trailing-comma,dict-separator\n\n# Maximum number of lines in a module\nmax-module-lines=1000\n\n# String used as indentation unit. This is usually \" \" (4 spaces) or \"\\t\" (1\n# tab).\nindent-string=' '\n\n# Number of spaces of indent required inside a hanging or continued line.\nindent-after-paren=4\n\n# Expected format of line ending, e.g. empty (any line ending), LF or CRLF.\nexpected-line-ending-format=\n\n\n[SPELLING]\n\n# Spelling dictionary name. Available dictionaries: none. To make it working\n# install python-enchant package.\nspelling-dict=\n\n# List of comma separated words that should not be checked.\nspelling-ignore-words=\n\n# A path to a file that contains private dictionary; one word per line.\nspelling-private-dict-file=\n\n# Tells whether to store unknown words to indicated private dictionary in\n# --spelling-private-dict-file option instead of raising a message.\nspelling-store-unknown-words=no\n\n\n[MISCELLANEOUS]\n\n# List of note tags to take in consideration, separated by a comma.\nnotes=FIXME,XXX,TODO\n\n\n[TYPECHECK]\n\n# Tells whether missing members accessed in mixin class should be ignored. A\n# mixin class is detected if its name ends with \"mixin\" (case insensitive).\nignore-mixin-members=yes\n\n# List of module names for which member attributes should not be checked\n# (useful for modules/projects where namespaces are manipulated during runtime\n# and thus existing member attributes cannot be deduced by static analysis. It\n# supports qualified module names, as well as Unix pattern matching.\nignored-modules=\n\n# List of class names for which member attributes should not be checked (useful\n# for classes with dynamically set attributes). This supports the use of\n# qualified names.\nignored-classes=optparse.Values,thread._local,_thread._local\n\n# List of members which are set dynamically and missed by pylint inference\n# system, and so shouldn't trigger E1101 when accessed. Python regular\n# expressions are accepted.\ngenerated-members=\n\n# List of decorators that produce context managers, such as\n# contextlib.contextmanager. Add to this list to register other decorators that\n# produce valid context managers.\ncontextmanager-decorators=contextlib.contextmanager\n\n\n[LOGGING]\n\n# Logging modules to check that the string format arguments are in logging\n# function parameter format\nlogging-modules=logging\n\n\n[SIMILARITIES]\n\n# Minimum lines number of a similarity.\nmin-similarity-lines=4\n\n# Ignore comments when computing similarities.\nignore-comments=yes\n\n# Ignore docstrings when computing similarities.\nignore-docstrings=yes\n\n# Ignore imports when computing similarities.\nignore-imports=no\n\n\n[VARIABLES]\n\n# Tells whether we should check for unused import in __init__ files.\ninit-import=no\n\n# A regular expression matching the name of dummy variables (i.e. expectedly\n# not used).\ndummy-variables-rgx=(_+[a-zA-Z0-9]*?$)|dummy\n\n# List of additional names supposed to be defined in builtins. Remember that\n# you should avoid to define new builtins when possible.\nadditional-builtins=\n\n# List of strings which can identify a callback function by name. A callback\n# name must start or end with one of those strings.\ncallbacks=cb_,_cb\n\n# List of qualified module names which can have objects that can redefine\n# builtins.\nredefining-builtins-modules=six.moves,future.builtins\n\n\n[BASIC]\n\n# Good variable names which should always be accepted, separated by a comma\ngood-names=i,j,_,a,b,op,x,y,wd,lr,kv,k,v,s,p,h,c,m,n,X,t,g,f\n\n# Bad variable names which should always be refused, separated by a comma\nbad-names=\n\n# Colon-delimited sets of names that determine each other's naming style when\n# the name regexes allow several styles.\nname-group=\n\n# Include a hint for the correct naming format with invalid-name\ninclude-naming-hint=no\n\n# List of decorators that produce properties, such as abc.abstractproperty. Add\n# to this list to register other decorators that produce valid properties.\nproperty-classes=abc.abstractproperty\n\n# Regular expression matching correct module names\nmodule-rgx=(([a-z_][a-z0-9_]*)|([A-Z][a-zA-Z0-9]+))$\n\n# Naming hint for module names\nmodule-name-hint=(([a-z_][a-z0-9_]*)|([A-Z][a-zA-Z0-9]+))$\n\n# Regular expression matching correct constant names\nconst-rgx=(([A-Z_][A-Z0-9_]*)|(__.*__))$\n\n# Naming hint for constant names\nconst-name-hint=(([A-Z_][A-Z0-9_]*)|(__.*__))$\n\n# Regular expression matching correct inline iteration names\ninlinevar-rgx=[A-Za-z_][A-Za-z0-9_]*$\n\n# Naming hint for inline iteration names\ninlinevar-name-hint=[A-Za-z_][A-Za-z0-9_]*$\n\n# Regular expression matching correct method names\nmethod-rgx=[a-z_][a-z0-9_]{2,30}$\n\n# Naming hint for method names\nmethod-name-hint=[a-z_][a-z0-9_]{2,30}$\n\n# Regular expression matching correct class attribute names\nclass-attribute-rgx=([A-Za-z_][A-Za-z0-9_]{2,30}|(__.*__))$\n\n# Naming hint for class attribute names\nclass-attribute-name-hint=([A-Za-z_][A-Za-z0-9_]{2,30}|(__.*__))$\n\n# Regular expression matching correct argument names\nargument-rgx=[a-z_][a-z0-9_]{2,30}$\n\n# Naming hint for argument names\nargument-name-hint=[a-z_][a-z0-9_]{2,30}$\n\n# Regular expression matching correct attribute names\nattr-rgx=[a-z_][a-z0-9_]{2,30}$\n\n# Naming hint for attribute names\nattr-name-hint=[a-z_][a-z0-9_]{2,30}$\n\n# Regular expression matching correct variable names\nvariable-rgx=[a-z_][a-z0-9_]{2,30}$\n\n# Naming hint for variable names\nvariable-name-hint=[a-z_][a-z0-9_]{2,30}$\n\n# Regular expression matching correct function names\nfunction-rgx=[a-z_][a-z0-9_]{2,30}$\n\n# Naming hint for function names\nfunction-name-hint=[a-z_][a-z0-9_]{2,30}$\n\n# Regular expression matching correct class names\nclass-rgx=[A-Z_][a-zA-Z0-9]+$\n\n# Naming hint for class names\nclass-name-hint=[A-Z_][a-zA-Z0-9]+$\n\n# Regular expression which should only match function or class names that do\n# not require a docstring.\nno-docstring-rgx=^_\n\n# Minimum line length for functions/classes that require docstrings, shorter\n# ones are exempt.\ndocstring-min-length=10\n\n\n[ELIF]\n\n# Maximum number of nested blocks for function / method body\nmax-nested-blocks=5\n\n\n[CLASSES]\n\n# List of method names used to declare (i.e. assign) instance attributes.\ndefining-attr-methods=__init__,__new__,setUp\n\n# List of valid names for the first argument in a class method.\nvalid-classmethod-first-arg=cls\n\n# List of valid names for the first argument in a metaclass class method.\nvalid-metaclass-classmethod-first-arg=mcs\n\n# List of member names, which should be excluded from the protected access\n# warning.\nexclude-protected=_asdict,_fields,_replace,_source,_make\n\n\n[IMPORTS]\n\n# Deprecated modules which should not be used, separated by a comma\ndeprecated-modules=optparse\n\n# Create a graph of every (i.e. internal and external) dependencies in the\n# given file (report RP0402 must not be disabled)\nimport-graph=\n\n# Create a graph of external dependencies in the given file (report RP0402 must\n# not be disabled)\next-import-graph=\n\n# Create a graph of internal dependencies in the given file (report RP0402 must\n# not be disabled)\nint-import-graph=\n\n# Force import order to recognize a module as part of the standard\n# compatibility libraries.\nknown-standard-library=\n\n# Force import order to recognize a module as part of a third party library.\nknown-third-party=enchant\n\n# Analyse import fallback blocks. This can be used to support both Python 2 and\n# 3 compatible code, which means that the block might have code that exists\n# only in one or another interpreter, leading to false positives when analysed.\nanalyse-fallback-blocks=no\n\n\n[DESIGN]\n\n# Maximum number of arguments for function / method\nmax-args=5\n\n# Argument names that match this expression will be ignored. Default to name\n# with leading underscore\nignored-argument-names=_.*\n\n# Maximum number of locals for function / method body\nmax-locals=15\n\n# Maximum number of return / yield for function / method body\nmax-returns=6\n\n# Maximum number of branch for function / method body\nmax-branches=12\n\n# Maximum number of statements in function / method body\nmax-statements=50\n\n# Maximum number of parents for a class (see R0901).\nmax-parents=7\n\n# Maximum number of attributes for a class (see R0902).\nmax-attributes=7\n\n# Minimum number of public methods for a class (see R0903).\nmin-public-methods=2\n\n# Maximum number of public methods for a class (see R0904).\nmax-public-methods=20\n\n# Maximum number of boolean expressions in a if statement\nmax-bool-expr=5\n\n\n[EXCEPTIONS]\n\n# Exceptions that will emit a warning when being caught. Defaults to\n# \"Exception\"\novergeneral-exceptions=Exception\n" }, { "path": "model-archiver/model_archiver/tests/unit_tests/test_model_packaging.py", "content": "# Copyright 2018 Amazon.com, Inc. or its affiliates. All Rights Reserved.\n# Licensed under the Apache License, Version 2.0 (the \"License\").\n# You may not use this file except in compliance with the License.\n# A copy of the License is located at\n# http://www.apache.org/licenses/LICENSE-2.0\n# or in the \"license\" file accompanying this file. This file is distributed\n# on an \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either\n# express or implied. See the License for the specific language governing\n# permissions and limitations under the License.\n\nfrom collections import namedtuple\n\nimport pytest\n\nfrom model_archiver.manifest_components.engine import EngineType\nfrom model_archiver.manifest_components.manifest import RuntimeType\nfrom model_archiver.model_packaging import generate_model_archive, package_model\nfrom model_archiver.model_packaging_utils import ModelExportUtils\n\n\n# noinspection PyClassHasNoInit\nclass TestModelPackaging:\n\n class Namespace:\n def __init__(self, **kwargs):\n self.__dict__.update(kwargs)\n\n def update(self, **kwargs):\n self.__dict__.update(kwargs)\n\n author = 'ABC'\n email = 'ABC@XYZ.com'\n engine = EngineType.MXNET.value\n model_name = 'my-model'\n model_path = 'my-model/'\n handler = 'a.py::my-awesome-func'\n export_path = '/Users/dummyUser/'\n\n args = Namespace(author=author, email=email, engine=engine, model_name=model_name, handler=handler,\n runtime=RuntimeType.PYTHON.value, model_path=model_path, export_path=export_path, force=False,\n archive_format=\"default\", convert=False)\n\n @pytest.fixture()\n def patches(self, mocker):\n Patches = namedtuple('Patches', ['arg_parse', 'export_utils', 'export_method'])\n patches = Patches(mocker.patch('model_archiver.model_packaging.ArgParser'),\n mocker.patch('model_archiver.model_packaging.ModelExportUtils'),\n mocker.patch('model_archiver.model_packaging.package_model'))\n\n return patches\n\n def test_gen_model_archive(self, patches):\n patches.arg_parse.export_model_args_parser.parse_args.return_value = self.args\n generate_model_archive()\n patches.export_method.assert_called()\n\n def test_export_model_method(self, patches):\n patches.export_utils.check_mar_already_exists.return_value = '/Users/dummyUser/'\n patches.export_utils.check_custom_model_types.return_value = '/Users/dummyUser', ['a.txt', 'b.txt']\n patches.export_utils.zip.return_value = None\n\n package_model(self.args, ModelExportUtils.generate_manifest_json(self.args))\n patches.export_utils.validate_inputs.assert_called()\n patches.export_utils.archive.assert_called()\n patches.export_utils.clean_temp_files.assert_called()\n\n def test_export_model_method_tar(self, patches):\n self.args.update(archive_format=\"tar\")\n patches.export_utils.check_mar_already_exists.return_value = '/Users/dummyUser/'\n patches.export_utils.check_custom_model_types.return_value = '/Users/dummyUser', ['a.txt', 'b.txt']\n patches.export_utils.zip.return_value = None\n\n package_model(self.args, ModelExportUtils.generate_manifest_json(self.args))\n patches.export_utils.validate_inputs.assert_called()\n patches.export_utils.archive.assert_called()\n patches.export_utils.clean_temp_files.assert_called()\n\n def test_export_model_method_noarchive(self, patches):\n self.args.update(archive_format=\"no-archive\")\n patches.export_utils.check_mar_already_exists.return_value = '/Users/dummyUser/'\n patches.export_utils.check_custom_model_types.return_value = '/Users/dummyUser', ['a.txt', 'b.txt']\n patches.export_utils.zip.return_value = None\n\n package_model(self.args, ModelExportUtils.generate_manifest_json(self.args))\n patches.export_utils.validate_inputs.assert_called()\n patches.export_utils.archive.assert_called()\n patches.export_utils.clean_temp_files.assert_called()\n" }, { "path": "model-archiver/model_archiver/tests/unit_tests/test_model_packaging_utils.py", "content": "# Copyright 2018 Amazon.com, Inc. or its affiliates. All Rights Reserved.\n# Licensed under the Apache License, Version 2.0 (the \"License\").\n# You may not use this file except in compliance with the License.\n# A copy of the License is located at\n# http://www.apache.org/licenses/LICENSE-2.0\n# or in the \"license\" file accompanying this file. This file is distributed\n# on an \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either\n# express or implied. See the License for the specific language governing\n# permissions and limitations under the License.\n\nimport json\nimport os\nimport pytest\nfrom collections import namedtuple\nfrom model_archiver.model_packaging_utils import ModelExportUtils\nfrom model_archiver.manifest_components.engine import EngineType\nfrom model_archiver.manifest_components.manifest import RuntimeType\nfrom model_archiver.model_archiver_error import ModelArchiverError\n\n\n# noinspection PyClassHasNoInit\nclass TestExportModelUtils:\n\n # noinspection PyClassHasNoInit\n class TestMarExistence:\n\n @pytest.fixture()\n def patches(self, mocker):\n Patches = namedtuple('Patches', ['getcwd', 'path_exists'])\n patches = Patches(mocker.patch('os.getcwd'), mocker.patch('os.path.exists'))\n\n patches.getcwd.return_value = '/Users/dummyUser'\n\n return patches\n\n def test_export_file_is_none(self, patches):\n patches.path_exists.return_value = False\n ret_val = ModelExportUtils.check_mar_already_exists('some-model', None, False)\n\n patches.path_exists.assert_called_once_with(\"/Users/dummyUser/some-model.mar\")\n assert ret_val == \"/Users/dummyUser\"\n\n def test_export_file_is_not_none(self, patches):\n patches.path_exists.return_value = False\n ModelExportUtils.check_mar_already_exists('some-model', '/Users/dummyUser/', False)\n\n patches.path_exists.assert_called_once_with('/Users/dummyUser/some-model.mar')\n\n def test_export_file_already_exists_with_override(self, patches):\n patches.path_exists.return_value = True\n\n ModelExportUtils.check_mar_already_exists('some-model', None, True)\n\n patches.path_exists.assert_called_once_with('/Users/dummyUser/some-model.mar')\n\n def test_export_file_already_exists_with_override_false(self, patches):\n patches.path_exists.return_value = True\n\n with pytest.raises(ModelArchiverError):\n ModelExportUtils.check_mar_already_exists('some-model', None, False)\n\n patches.path_exists.assert_called_once_with('/Users/dummyUser/some-model.mar')\n\n def test_export_file_is_none_tar(self, patches):\n patches.path_exists.return_value = False\n ret_val = ModelExportUtils.check_mar_already_exists('some-model', None, False, archive_format='tgz')\n\n patches.path_exists.assert_called_once_with(\"/Users/dummyUser/some-model.tar.gz\")\n assert ret_val == \"/Users/dummyUser\"\n\n def test_export_file_is_none_tar(self, patches):\n patches.path_exists.return_value = False\n ret_val = ModelExportUtils.check_mar_already_exists('some-model', None, False, archive_format='no-archive')\n\n patches.path_exists.assert_called_once_with(\"/Users/dummyUser/some-model\")\n assert ret_val == \"/Users/dummyUser\"\n\n # noinspection PyClassHasNoInit\n class TestArchiveTypes:\n def test_archive_types(self):\n from model_archiver.model_packaging_utils import archiving_options as ar_opts\n assert ar_opts.get(\"tgz\") == \".tar.gz\"\n assert ar_opts.get(\"no-archive\") == \"\"\n assert ar_opts.get(\"default\") == \".mar\"\n assert len(ar_opts) == 3\n\n # noinspection PyClassHasNoInit\n class TestCustomModelTypes:\n\n model_path = '/Users/dummyUser'\n\n @pytest.fixture()\n def patches(self, mocker):\n Patches = namedtuple('Patches', ['utils', 'listdir'])\n patch = Patches(mocker.patch('model_archiver.model_packaging_utils.ModelExportUtils'),\n mocker.patch('os.listdir'))\n\n patch.listdir.return_value = {'a', 'b', 'c'}\n return patch\n\n def test_onnx_file_is_none(self, patches):\n patches.utils.find_unique.return_value = None\n ModelExportUtils.check_custom_model_types(model_path=self.model_path, model_name=None)\n\n patches.utils.find_unique.assert_called()\n patches.utils.convert_onnx_model.assert_not_called()\n\n def test_onnx_file_is_not_none(self, patches):\n onnx_file = 'some-file.onnx'\n patches.utils.find_unique.return_value = onnx_file\n patches.utils.convert_onnx_model.return_value = ('sym', 'param')\n\n temp, exclude = ModelExportUtils.check_custom_model_types(self.model_path)\n patches.utils.convert_onnx_model.assert_called_once_with(self.model_path, onnx_file, None)\n\n assert len(temp) == 2\n assert len(exclude) == 1\n assert temp[0] == os.path.join(self.model_path, 'sym')\n assert temp[1] == os.path.join(self.model_path, 'param')\n assert exclude[0] == onnx_file\n\n # noinspection PyClassHasNoInit\n class TestFindUnique:\n\n def test_with_count_zero(self):\n files = ['a.txt', 'b.txt', 'c.txt']\n suffix = '.mxnet'\n val = ModelExportUtils.find_unique(files, suffix)\n assert val is None\n\n def test_with_count_one(self):\n files = ['a.mxnet', 'b.txt', 'c.txt']\n suffix = '.mxnet'\n val = ModelExportUtils.find_unique(files, suffix)\n assert val == 'a.mxnet'\n\n def test_with_exit(self):\n files = ['a.onnx', 'b.onnx', 'c.txt']\n suffix = '.onnx'\n with pytest.raises(ModelArchiverError):\n ModelExportUtils.find_unique(files, suffix)\n\n # noinspection PyClassHasNoInit\n class TestCleanTempFiles:\n\n @pytest.fixture()\n def patches(self, mocker):\n Patches = namedtuple('Patches', ['remove'])\n patches = Patches(mocker.patch('os.remove'))\n\n patches.remove.return_value = True\n return patches\n\n def test_clean_call(self, patches):\n temp_files = ['a', 'b', 'c']\n ModelExportUtils.clean_temp_files(temp_files)\n\n patches.remove.assert_called()\n assert patches.remove.call_count == len(temp_files)\n\n # noinspection PyClassHasNoInit\n class TestGenerateManifestProps:\n\n class Namespace:\n def __init__(self, **kwargs):\n self.__dict__.update(kwargs)\n\n author = 'ABC'\n email = 'ABC@XYZ.com'\n engine = EngineType.MXNET.value\n model_name = 'my-model'\n handler = 'a.py::my-awesome-func'\n\n args = Namespace(author=author, email=email, engine=engine, model_name=model_name, handler=handler,\n runtime=RuntimeType.PYTHON.value)\n\n def test_publisher(self):\n pub = ModelExportUtils.generate_publisher(self.args)\n assert pub.email == self.email\n assert pub.author == self.author\n\n def test_engine(self):\n eng = ModelExportUtils.generate_engine(self.args)\n assert eng.engine_name == EngineType(self.engine)\n\n def test_model(self):\n mod = ModelExportUtils.generate_model(self.args)\n assert mod.model_name == self.model_name\n assert mod.handler == self.handler\n\n def test_manifest_json(self):\n manifest = ModelExportUtils.generate_manifest_json(self.args)\n manifest_json = json.loads(manifest)\n assert manifest_json['runtime'] == RuntimeType.PYTHON.value\n assert 'engine' in manifest_json\n assert 'model' in manifest_json\n assert 'publisher' in manifest_json\n assert 'license' not in manifest_json\n\n # noinspection PyClassHasNoInit\n class TestModelNameRegEx:\n\n def test_regex_pass(self):\n model_names = ['my-awesome-model', 'Aa.model', 'a', 'aA.model', 'a1234.model', 'a-A-A.model', '123-abc']\n for m in model_names:\n ModelExportUtils.check_model_name_regex_or_exit(m)\n\n def test_regex_fail(self):\n model_names = ['abc%', '123$abc', 'abc!123', '@123', '(model', 'mdoel)',\n '12*model-a.model', '##.model', '-.model']\n for m in model_names:\n with pytest.raises(ModelArchiverError):\n ModelExportUtils.check_model_name_regex_or_exit(m)\n\n # noinspection PyClassHasNoInit\n class TestFileFilter:\n\n files_to_exclude = {'abc.onnx'}\n\n def test_with_return_false(self):\n assert ModelExportUtils.file_filter('abc.onnx', self.files_to_exclude) is False\n\n def test_with_pyc(self):\n assert ModelExportUtils.file_filter('abc.pyc', self.files_to_exclude) is False\n\n def test_with_ds_store(self):\n assert ModelExportUtils.file_filter('.DS_Store', self.files_to_exclude) is False\n\n def test_with_return_true(self):\n assert ModelExportUtils.file_filter('abc.mxnet', self.files_to_exclude) is True\n\n # noinspection PyClassHasNoInit\n class TestDirectoryFilter:\n\n unwanted_dirs = {'__MACOSX', '__pycache__'}\n\n def test_with_unwanted_dirs(self):\n assert ModelExportUtils.directory_filter('__MACOSX', self.unwanted_dirs) is False\n\n def test_with_starts_with_dot(self):\n assert ModelExportUtils.directory_filter('.gitignore', self.unwanted_dirs) is False\n\n def test_with_return_true(self):\n assert ModelExportUtils.directory_filter('my-model', self.unwanted_dirs) is True\n" }, { "path": "model-archiver/model_archiver/tests/unit_tests/test_version.py", "content": "# Copyright 2018 Amazon.com, Inc. or its affiliates. All Rights Reserved.\n# Licensed under the Apache License, Version 2.0 (the \"License\").\n# You may not use this file except in compliance with the License.\n# A copy of the License is located at\n# http://www.apache.org/licenses/LICENSE-2.0\n# or in the \"license\" file accompanying this file. This file is distributed\n# on an \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either\n# express or implied. See the License for the specific language governing\n# permissions and limitations under the License.\n\nimport os\nimport model_archiver\n\n\ndef test_model_export_tool_version():\n \"\"\"\n Test the model archive version\n :return:\n \"\"\"\n with (open(os.path.join('model_archiver', 'version.py'))) as f:\n exec(f.read(), globals())\n\n assert __version__ == str(model_archiver.__version__), \"Versions do not match\"\n" }, { "path": "model-archiver/model_archiver/version.py", "content": "# Copyright 2018 Amazon.com, Inc. or its affiliates. All Rights Reserved.\n# Licensed under the Apache License, Version 2.0 (the \"License\").\n# You may not use this file except in compliance with the License.\n# A copy of the License is located at\n# http://www.apache.org/licenses/LICENSE-2.0\n# or in the \"license\" file accompanying this file. This file is distributed\n# on an \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either\n# express or implied. See the License for the specific language governing\n# permissions and limitations under the License.\n\"\"\"\nThis is the current version of Model Archiver Tool\n\"\"\"\n\n__version__ = '1.0.4'\n" }, { "path": "model-archiver/setup.py", "content": "# Copyright 2017 Amazon.com, Inc. or its affiliates. All Rights Reserved.\n# Licensed under the Apache License, Version 2.0 (the \"License\").\n# You may not use this file except in compliance with the License.\n# A copy of the License is located at\n# http://www.apache.org/licenses/LICENSE-2.0\n# or in the \"license\" file accompanying this file. This file is distributed\n# on an \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either\n# express or implied. See the License for the specific language governing\n# permissions and limitations under the License.\n\n# To build and upload a new version, follow the steps below.\n# Notes:\n# - this is a \"Universal Wheels\" package that is pure Python and supports both Python2 and Python3\n# - Twine is a secure PyPi upload package\n# - Make sure you have bumped the version! at mms/version.py\n# $ pip install twine\n# $ pip install wheel\n# $ python setup.py bdist_wheel --universal\n\n# *** TEST YOUR PACKAGE WITH TEST PI ******\n# twine upload --repository-url https://test.pypi.org/legacy/ dist/*\n\n# If this is successful then push it to actual pypi\n\n# $ twine upload dist/*\n\n\"\"\"\nSetup.py for the model-archiver tool\n\"\"\"\n\nfrom datetime import date\nimport sys\nfrom setuptools import setup, find_packages\n# pylint: disable = relative-import\nimport model_archiver\n\npkgs = find_packages()\n\n\ndef pypi_description():\n \"\"\"Imports the long description for the project page\"\"\"\n with open('PyPiDescription.rst') as df:\n return df.read()\n\n\ndef detect_model_archiver_version():\n if \"--release\" in sys.argv:\n sys.argv.remove(\"--release\")\n # pylint: disable = relative-import\n return model_archiver.__version__.strip()\n\n # pylint: disable = relative-import\n return model_archiver.__version__.strip() + 'b' + str(date.today()).replace('-', '')\n\n\nif __name__ == '__main__':\n version = detect_model_archiver_version()\n requirements = ['future', 'enum-compat']\n\n setup(\n name='model-archiver',\n version=version,\n description='Model Archiver is used for creating archives of trained neural net models that can be consumed '\n 'by Multi-Model-Server inference',\n long_description=pypi_description(),\n author='Trinity',\n author_email='noreply@amazon.com',\n url='https://github.com/awslabs/multi-model-server/model-archiver/',\n keywords='Multi Model Archive Archiver MMS Server Serving Deep Learning Inference AI',\n packages=pkgs,\n install_requires=requirements,\n extras_require={\n 'mxnet-mkl': ['mxnet-mkl==1.3.1'],\n 'mxnet-cu90mkl': ['mxnet-cu90mkl==1.3.1'],\n 'mxnet-cu92mkl': ['mxnet-cu92mkl==1.3.1'],\n 'mxnet': ['mxnet==1.3.1'],\n 'onnx': ['onnx==1.1.1']\n },\n entry_points={\n 'console_scripts': ['model-archiver=model_archiver.model_packaging:generate_model_archive']\n },\n include_package_data=True,\n license='Apache License Version 2.0'\n )\n" }, { "path": "performance_regression/imageInputModelPlan.jmx.yaml", "content": "---\nexecution:\n- concurrency: 1\n ramp-up: 5s\n hold-for: 1m\n scenario: Inference\nscenarios:\n Inference:\n default-address: ${__P(protocol,https)}://${__P(hostname,127.0.0.1)}:${__P(port,8443)}/\n requests:\n - follow-redirects: true\n label: Inference Request\n method: POST\n upload-files:\n - mime-type: image/jpeg\n param: data\n path: ${__P(input_filepath)}\n url: ${__P(protocol,http)}://${__P(hostname,127.0.0.1)}:${__P(port,8080)}/predictions/${model}\n store-cache: false\n store-cookie: false\n use-dns-cache-mgr: false\n variables:\n cnn_url: ${__P(url, https://s3.amazonaws.com/model-server/models/squeezenet_v1.1/squeezenet_v1.1.model)}\n model: ${__P(model_name,squeezenet_v1.1)}\n scale_down_workers: '0'\n scale_up_workers: ${__P(min_workers,1)}\nmodules:\n jmeter:\n properties:\n input_filepath : kitten.jpg\n model_name : squeezenet\nservices:\n - module: monitoring\n local:\n - interval: 2s\n logging: True\n metrics:\n - cpu\n - disk-space\n - mem\nreporting:\n- module: passfail\n criteria:\n - class: bzt.modules.monitoring.MonitoringCriteria\n subject: local/cpu\n condition: '>'\n threshold: 100\n timeframe: 6s\n- module: junit-xml\n filename: ${TAURUS_ARTIFACTS_DIR}/output/results.xml\n data-source: pass-fail\n" }, { "path": "plugins/build.gradle", "content": "/*\n * This file was generated by the Gradle 'init' task.\n *\n * This generated file contains a sample Java Library project to get you started.\n * For more details take a look at the Java Libraries chapter in the Gradle\n * User Manual available at https://docs.gradle.org/5.4.1/userguide/java_library_plugin.html\n */\n\nallprojects {\n apply plugin: 'idea'\n apply plugin: 'java'\n\n version = '1.0'\n\n repositories {\n jcenter()\n }\n\n idea {\n module {\n outputDir = file('build/classes/java/main')\n testOutputDir = file('build/classes/java/test')\n }\n }\n\n task buildSagemaker(\"type\": Jar) {\n\n doFirst{ task ->\n println \"building $task.project.name\"\n }\n\n with project.jar\n\n doLast {\n copy {\n def fromDir = project.jar\n def intoDir = \"${rootProject.projectDir}/build/plugins\"\n from fromDir\n into intoDir\n println \"Copying files from\" + fromDir + \" into \" + intoDir\n }\n }\n }\n\n buildSagemaker.onlyIf {project.hasProperty(\"sagemaker\")}\n}\n\ndef javaProjects() {\n return subprojects.findAll()\n}\n\nconfigure(javaProjects()) {\n sourceCompatibility = 1.8\n targetCompatibility = 1.8\n\n defaultTasks 'jar'\n\n apply from: file(\"${rootProject.projectDir}/tools/gradle/formatter.gradle\")\n apply from: file(\"${rootProject.projectDir}/tools/gradle/check.gradle\")\n\n test {\n useTestNG() {\n // suiteXmlFiles << new File(rootDir, \"testng.xml\") //This is how to add custom testng.xml\n }\n\n testLogging {\n showStandardStreams = true\n events \"passed\", \"skipped\", \"failed\", \"standardOut\", \"standardError\"\n }\n }\n\n test.finalizedBy(project.tasks.jacocoTestReport)\n\n compileJava {\n options.compilerArgs << \"-Xlint:all,-options,-static\" << \"-Werror\"\n }\n\n jacocoTestCoverageVerification {\n violationRules {\n rule {\n limit {\n minimum = 0.75\n }\n }\n }\n }\n}\n\n" }, { "path": "plugins/endpoints/build.gradle", "content": "dependencies {\n compile \"com.google.code.gson:gson:${gson_version}\"\n compile \"software.amazon.ai:mms-plugins-sdk:${mms_server_sdk_version}\"\n}\n\nproject.ext{\n sagemaker = true\n}\n\njar {\n includeEmptyDirs = false\n\n exclude \"META-INF/maven/**\"\n exclude \"META-INF/INDEX.LIST\"\n exclude \"META-INF/MANIFEST*\"\n exclude \"META-INF//LICENSE*\"\n exclude \"META-INF//NOTICE*\"\n}\n\n" }, { "path": "plugins/endpoints/src/main/java/software/amazon/ai/mms/plugins/endpoint/ExecutionParameters.java", "content": "package software.amazon.ai.mms.plugins.endpoint;\n\nimport com.google.gson.GsonBuilder;\nimport com.google.gson.annotations.SerializedName;\nimport java.io.IOException;\nimport java.nio.charset.StandardCharsets;\nimport java.util.Properties;\nimport software.amazon.ai.mms.servingsdk.Context;\nimport software.amazon.ai.mms.servingsdk.ModelServerEndpoint;\nimport software.amazon.ai.mms.servingsdk.annotations.Endpoint;\nimport software.amazon.ai.mms.servingsdk.annotations.helpers.EndpointTypes;\nimport software.amazon.ai.mms.servingsdk.http.Request;\nimport software.amazon.ai.mms.servingsdk.http.Response;\n\n@Endpoint(\n urlPattern = \"execution-parameters\",\n endpointType = EndpointTypes.INFERENCE,\n description = \"Execution parameters endpoint\")\npublic class ExecutionParameters extends ModelServerEndpoint {\n\n @Override\n public void doGet(Request req, Response rsp, Context ctx) throws IOException {\n Properties prop = ctx.getConfig();\n // 6 * 1024 * 1024\n int maxRequestSize = Integer.parseInt(prop.getProperty(\"max_request_size\", \"6291456\"));\n ExecutionParametersResponse r = new ExecutionParametersResponse();\n r.setMaxConcurrentTransforms(Integer.parseInt(prop.getProperty(\"NUM_WORKERS\", \"1\")));\n r.setBatchStrategy(\"MULTI_RECORD\");\n r.setMaxPayloadInMB(maxRequestSize / (1024 * 1024));\n rsp.getOutputStream()\n .write(\n new GsonBuilder()\n .setPrettyPrinting()\n .create()\n .toJson(r)\n .getBytes(StandardCharsets.UTF_8));\n }\n\n /** Response for Model server endpoint */\n public static class ExecutionParametersResponse {\n @SerializedName(\"MaxConcurrentTransforms\")\n private int maxConcurrentTransforms;\n\n @SerializedName(\"BatchStrategy\")\n private String batchStrategy;\n\n @SerializedName(\"MaxPayloadInMB\")\n private int maxPayloadInMB;\n\n public ExecutionParametersResponse() {\n maxConcurrentTransforms = 4;\n batchStrategy = \"MULTI_RECORD\";\n maxPayloadInMB = 6;\n }\n\n public int getMaxConcurrentTransforms() {\n return maxConcurrentTransforms;\n }\n\n public String getBatchStrategy() {\n return batchStrategy;\n }\n\n public int getMaxPayloadInMB() {\n return maxPayloadInMB;\n }\n\n public void setMaxConcurrentTransforms(int newMaxConcurrentTransforms) {\n maxConcurrentTransforms = newMaxConcurrentTransforms;\n }\n\n public void setBatchStrategy(String newBatchStrategy) {\n batchStrategy = newBatchStrategy;\n }\n\n public void setMaxPayloadInMB(int newMaxPayloadInMB) {\n maxPayloadInMB = newMaxPayloadInMB;\n }\n }\n}\n" }, { "path": "plugins/endpoints/src/main/java/software/amazon/ai/mms/plugins/endpoint/Ping.java", "content": "package software.amazon.ai.mms.plugins.endpoint;\n\nimport java.io.File;\nimport java.io.IOException;\nimport java.nio.charset.StandardCharsets;\nimport java.util.Map;\nimport software.amazon.ai.mms.servingsdk.Context;\nimport software.amazon.ai.mms.servingsdk.Model;\nimport software.amazon.ai.mms.servingsdk.ModelServerEndpoint;\nimport software.amazon.ai.mms.servingsdk.Worker;\nimport software.amazon.ai.mms.servingsdk.annotations.Endpoint;\nimport software.amazon.ai.mms.servingsdk.annotations.helpers.EndpointTypes;\nimport software.amazon.ai.mms.servingsdk.http.Request;\nimport software.amazon.ai.mms.servingsdk.http.Response;\n\n@Endpoint(\n urlPattern = \"ping\",\n endpointType = EndpointTypes.INFERENCE,\n description = \"Ping endpoint for sagemaker containers.\")\npublic class Ping extends ModelServerEndpoint {\n private boolean init;\n private byte[] success = \"{\\n\\t\\\"Status\\\": \\\"Healthy\\\"\\n}\\n\".getBytes(StandardCharsets.UTF_8);\n\n private boolean modelsLoaded(Context ctx) {\n Map modelMap = ctx.getModels();\n\n for (Map.Entry entry : modelMap.entrySet()) {\n for (Worker w : entry.getValue().getModelWorkers()) {\n if (w.isRunning()) {\n return true;\n }\n }\n }\n return false;\n }\n\n private boolean validConfig(String svc) {\n String fileName = svc;\n if (svc.contains(\":\")) {\n fileName = svc.substring(0, svc.lastIndexOf(':'));\n }\n if (!fileName.contains(\".py\")) {\n fileName = fileName.concat(\".py\");\n }\n return new File(fileName).exists();\n }\n\n @Override\n public void doGet(Request req, Response rsp, Context ctx) throws IOException {\n rsp.setStatus(200);\n String isMultiModelMode = System.getenv(\"SAGEMAKER_MULTI_MODE\");\n if (isMultiModelMode == null || \"false\".equalsIgnoreCase(isMultiModelMode)) {\n if (!init && !modelsLoaded(ctx)) {\n rsp.setStatus(503, \"Model loading...\");\n rsp.getOutputStream()\n .write(\"Models are not loaded\".getBytes(StandardCharsets.UTF_8));\n } else {\n init = true;\n rsp.getOutputStream().write(success);\n }\n } else {\n String svcFile = ctx.getConfig().getProperty(\"default_service_handler\");\n if ((svcFile == null) || !validConfig(svcFile)) {\n rsp.setStatus(503, \"Service file unavailable\");\n rsp.getOutputStream()\n .write(\"Service file unavailable\".getBytes(StandardCharsets.UTF_8));\n } else {\n rsp.getOutputStream().write(success);\n }\n }\n }\n}\n" }, { "path": "plugins/endpoints/src/main/resources/META-INF/services/software.amazon.ai.mms.servingsdk.ModelServerEndpoint", "content": "software.amazon.ai.mms.plugins.endpoint.Ping\n" }, { "path": "plugins/gradle/wrapper/gradle-wrapper.properties", "content": "#Tue Jun 04 12:29:27 PDT 2019\ndistributionBase=GRADLE_USER_HOME\ndistributionPath=wrapper/dists\nzipStoreBase=GRADLE_USER_HOME\nzipStorePath=wrapper/dists\ndistributionUrl=https\\://services.gradle.org/distributions/gradle-4.10-all.zip\n" }, { "path": "plugins/gradle.properties", "content": "gson_version=2.8.9\nmms_server_sdk_version=1.0.1\n" }, { "path": "plugins/gradlew", "content": "#!/usr/bin/env sh\n\n##############################################################################\n##\n## Gradle start up script for UN*X\n##\n##############################################################################\n\n# Attempt to set APP_HOME\n# Resolve links: $0 may be a link\nPRG=\"$0\"\n# Need this for relative symlinks.\nwhile [ -h \"$PRG\" ] ; do\n ls=`ls -ld \"$PRG\"`\n link=`expr \"$ls\" : '.*-> \\(.*\\)$'`\n if expr \"$link\" : '/.*' > /dev/null; then\n PRG=\"$link\"\n else\n PRG=`dirname \"$PRG\"`\"/$link\"\n fi\ndone\nSAVED=\"`pwd`\"\ncd \"`dirname \\\"$PRG\\\"`/\" >/dev/null\nAPP_HOME=\"`pwd -P`\"\ncd \"$SAVED\" >/dev/null\n\nAPP_NAME=\"Gradle\"\nAPP_BASE_NAME=`basename \"$0\"`\n\n# Add default JVM options here. You can also use JAVA_OPTS and GRADLE_OPTS to pass JVM options to this script.\nDEFAULT_JVM_OPTS=\"\"\n\n# Use the maximum available, or set MAX_FD != -1 to use that value.\nMAX_FD=\"maximum\"\n\nwarn () {\n echo \"$*\"\n}\n\ndie () {\n echo\n echo \"$*\"\n echo\n exit 1\n}\n\n# OS specific support (must be 'true' or 'false').\ncygwin=false\nmsys=false\ndarwin=false\nnonstop=false\ncase \"`uname`\" in\n CYGWIN* )\n cygwin=true\n ;;\n Darwin* )\n darwin=true\n ;;\n MINGW* )\n msys=true\n ;;\n NONSTOP* )\n nonstop=true\n ;;\nesac\n\nCLASSPATH=$APP_HOME/gradle/wrapper/gradle-wrapper.jar\n\n# Determine the Java command to use to start the JVM.\nif [ -n \"$JAVA_HOME\" ] ; then\n if [ -x \"$JAVA_HOME/jre/sh/java\" ] ; then\n # IBM's JDK on AIX uses strange locations for the executables\n JAVACMD=\"$JAVA_HOME/jre/sh/java\"\n else\n JAVACMD=\"$JAVA_HOME/bin/java\"\n fi\n if [ ! -x \"$JAVACMD\" ] ; then\n die \"ERROR: JAVA_HOME is set to an invalid directory: $JAVA_HOME\n\nPlease set the JAVA_HOME variable in your environment to match the\nlocation of your Java installation.\"\n fi\nelse\n JAVACMD=\"java\"\n which java >/dev/null 2>&1 || die \"ERROR: JAVA_HOME is not set and no 'java' command could be found in your PATH.\n\nPlease set the JAVA_HOME variable in your environment to match the\nlocation of your Java installation.\"\nfi\n\n# Increase the maximum file descriptors if we can.\nif [ \"$cygwin\" = \"false\" -a \"$darwin\" = \"false\" -a \"$nonstop\" = \"false\" ] ; then\n MAX_FD_LIMIT=`ulimit -H -n`\n if [ $? -eq 0 ] ; then\n if [ \"$MAX_FD\" = \"maximum\" -o \"$MAX_FD\" = \"max\" ] ; then\n MAX_FD=\"$MAX_FD_LIMIT\"\n fi\n ulimit -n $MAX_FD\n if [ $? -ne 0 ] ; then\n warn \"Could not set maximum file descriptor limit: $MAX_FD\"\n fi\n else\n warn \"Could not query maximum file descriptor limit: $MAX_FD_LIMIT\"\n fi\nfi\n\n# For Darwin, add options to specify how the application appears in the dock\nif $darwin; then\n GRADLE_OPTS=\"$GRADLE_OPTS \\\"-Xdock:name=$APP_NAME\\\" \\\"-Xdock:icon=$APP_HOME/media/gradle.icns\\\"\"\nfi\n\n# For Cygwin, switch paths to Windows format before running java\nif $cygwin ; then\n APP_HOME=`cygpath --path --mixed \"$APP_HOME\"`\n CLASSPATH=`cygpath --path --mixed \"$CLASSPATH\"`\n JAVACMD=`cygpath --unix \"$JAVACMD\"`\n\n # We build the pattern for arguments to be converted via cygpath\n ROOTDIRSRAW=`find -L / -maxdepth 1 -mindepth 1 -type d 2>/dev/null`\n SEP=\"\"\n for dir in $ROOTDIRSRAW ; do\n ROOTDIRS=\"$ROOTDIRS$SEP$dir\"\n SEP=\"|\"\n done\n OURCYGPATTERN=\"(^($ROOTDIRS))\"\n # Add a user-defined pattern to the cygpath arguments\n if [ \"$GRADLE_CYGPATTERN\" != \"\" ] ; then\n OURCYGPATTERN=\"$OURCYGPATTERN|($GRADLE_CYGPATTERN)\"\n fi\n # Now convert the arguments - kludge to limit ourselves to /bin/sh\n i=0\n for arg in \"$@\" ; do\n CHECK=`echo \"$arg\"|egrep -c \"$OURCYGPATTERN\" -`\n CHECK2=`echo \"$arg\"|egrep -c \"^-\"` ### Determine if an option\n\n if [ $CHECK -ne 0 ] && [ $CHECK2 -eq 0 ] ; then ### Added a condition\n eval `echo args$i`=`cygpath --path --ignore --mixed \"$arg\"`\n else\n eval `echo args$i`=\"\\\"$arg\\\"\"\n fi\n i=$((i+1))\n done\n case $i in\n (0) set -- ;;\n (1) set -- \"$args0\" ;;\n (2) set -- \"$args0\" \"$args1\" ;;\n (3) set -- \"$args0\" \"$args1\" \"$args2\" ;;\n (4) set -- \"$args0\" \"$args1\" \"$args2\" \"$args3\" ;;\n (5) set -- \"$args0\" \"$args1\" \"$args2\" \"$args3\" \"$args4\" ;;\n (6) set -- \"$args0\" \"$args1\" \"$args2\" \"$args3\" \"$args4\" \"$args5\" ;;\n (7) set -- \"$args0\" \"$args1\" \"$args2\" \"$args3\" \"$args4\" \"$args5\" \"$args6\" ;;\n (8) set -- \"$args0\" \"$args1\" \"$args2\" \"$args3\" \"$args4\" \"$args5\" \"$args6\" \"$args7\" ;;\n (9) set -- \"$args0\" \"$args1\" \"$args2\" \"$args3\" \"$args4\" \"$args5\" \"$args6\" \"$args7\" \"$args8\" ;;\n esac\nfi\n\n# Escape application args\nsave () {\n for i do printf %s\\\\n \"$i\" | sed \"s/'/'\\\\\\\\''/g;1s/^/'/;\\$s/\\$/' \\\\\\\\/\" ; done\n echo \" \"\n}\nAPP_ARGS=$(save \"$@\")\n\n# Collect all arguments for the java command, following the shell quoting and substitution rules\neval set -- $DEFAULT_JVM_OPTS $JAVA_OPTS $GRADLE_OPTS \"\\\"-Dorg.gradle.appname=$APP_BASE_NAME\\\"\" -classpath \"\\\"$CLASSPATH\\\"\" org.gradle.wrapper.GradleWrapperMain \"$APP_ARGS\"\n\n# by default we should be in the correct project dir, but when run from Finder on Mac, the cwd is wrong\nif [ \"$(uname)\" = \"Darwin\" ] && [ \"$HOME\" = \"$PWD\" ]; then\n cd \"$(dirname \"$0\")\"\nfi\n\nexec \"$JAVACMD\" \"$@\"\n" }, { "path": "plugins/gradlew.bat", "content": "@if \"%DEBUG%\" == \"\" @echo off\r\n@rem ##########################################################################\r\n@rem\r\n@rem Gradle startup script for Windows\r\n@rem\r\n@rem ##########################################################################\r\n\r\n@rem Set local scope for the variables with windows NT shell\r\nif \"%OS%\"==\"Windows_NT\" setlocal\r\n\r\nset DIRNAME=%~dp0\r\nif \"%DIRNAME%\" == \"\" set DIRNAME=.\r\nset APP_BASE_NAME=%~n0\r\nset APP_HOME=%DIRNAME%\r\n\r\n@rem Add default JVM options here. You can also use JAVA_OPTS and GRADLE_OPTS to pass JVM options to this script.\r\nset DEFAULT_JVM_OPTS=\r\n\r\n@rem Find java.exe\r\nif defined JAVA_HOME goto findJavaFromJavaHome\r\n\r\nset JAVA_EXE=java.exe\r\n%JAVA_EXE% -version >NUL 2>&1\r\nif \"%ERRORLEVEL%\" == \"0\" goto init\r\n\r\necho.\r\necho ERROR: JAVA_HOME is not set and no 'java' command could be found in your PATH.\r\necho.\r\necho Please set the JAVA_HOME variable in your environment to match the\r\necho location of your Java installation.\r\n\r\ngoto fail\r\n\r\n:findJavaFromJavaHome\r\nset JAVA_HOME=%JAVA_HOME:\"=%\r\nset JAVA_EXE=%JAVA_HOME%/bin/java.exe\r\n\r\nif exist \"%JAVA_EXE%\" goto init\r\n\r\necho.\r\necho ERROR: JAVA_HOME is set to an invalid directory: %JAVA_HOME%\r\necho.\r\necho Please set the JAVA_HOME variable in your environment to match the\r\necho location of your Java installation.\r\n\r\ngoto fail\r\n\r\n:init\r\n@rem Get command-line arguments, handling Windows variants\r\n\r\nif not \"%OS%\" == \"Windows_NT\" goto win9xME_args\r\n\r\n:win9xME_args\r\n@rem Slurp the command line arguments.\r\nset CMD_LINE_ARGS=\r\nset _SKIP=2\r\n\r\n:win9xME_args_slurp\r\nif \"x%~1\" == \"x\" goto execute\r\n\r\nset CMD_LINE_ARGS=%*\r\n\r\n:execute\r\n@rem Setup the command line\r\n\r\nset CLASSPATH=%APP_HOME%\\gradle\\wrapper\\gradle-wrapper.jar\r\n\r\n@rem Execute Gradle\r\n\"%JAVA_EXE%\" %DEFAULT_JVM_OPTS% %JAVA_OPTS% %GRADLE_OPTS% \"-Dorg.gradle.appname=%APP_BASE_NAME%\" -classpath \"%CLASSPATH%\" org.gradle.wrapper.GradleWrapperMain %CMD_LINE_ARGS%\r\n\r\n:end\r\n@rem End local scope for the variables with windows NT shell\r\nif \"%ERRORLEVEL%\"==\"0\" goto mainEnd\r\n\r\n:fail\r\nrem Set variable GRADLE_EXIT_CONSOLE if you need the _script_ return code instead of\r\nrem the _cmd.exe /c_ return code!\r\nif not \"\" == \"%GRADLE_EXIT_CONSOLE%\" exit 1\r\nexit /b 1\r\n\r\n:mainEnd\r\nif \"%OS%\"==\"Windows_NT\" endlocal\r\n\r\n:omega\r\n" }, { "path": "plugins/settings.gradle", "content": "/*\n * This file was generated by the Gradle 'init' task.\n *\n * The settings file is used to specify which projects to include in your build.\n *\n * Detailed information about configuring a multi-project build in Gradle can be found\n * in the user manual at https://docs.gradle.org/5.4.1/userguide/multi_project_builds.html\n */\n\nrootProject.name = 'plugins'\ninclude 'endpoints'\n" }, { "path": "plugins/tools/conf/checkstyle.xml", "content": "\n\n\n \n \n\n \n \n\n \n \n \n \n \n \n\n \n \n\n \n \n\n \n \n \n \n \n \n \n \n\n \n \n\n \n \n \n \n \n\n \n \n\n \n \n\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n\n \n \n \n \n \n \n \n \n \n \n \n \n \n\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n\n \n \n\n \n \n\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n\n \n \n \n\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n\n \n \n\n \n \n \n \n\n \n \n\n \n\n\n" }, { "path": "plugins/tools/conf/findbugs-exclude.xml", "content": "\n\n \n \n \n \n \n \n \n \n \n \n\n\n" }, { "path": "plugins/tools/conf/pmd.xml", "content": "\n\n Java Rule in PMD\n\n \n\n \n \n \n \n \n \n\n \n\n \n \n \n\n \n\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n\n \n \n \n \n\n \n\n \n\n \n\n \n \n \n \n \n \n\n \n \n \n \n \n \n \n\n \n \n \n \n \n \n \n \n \n \n \n \n\n \n \n \n \n \n \n \n \n \n\n \n \n \n \n \n\n \n \n \n \n \n \n\n \n \n \n \n \n\n \n \n \n\n" }, { "path": "plugins/tools/conf/suppressions.xml", "content": "\n\n\n\n" }, { "path": "plugins/tools/gradle/check.gradle", "content": "apply plugin: 'findbugs'\nfindbugs {\n excludeFilter = file(\"${rootProject.projectDir}/tools/conf/findbugs-exclude.xml\")\n ignoreFailures = false\n findbugsTest.enabled = true\n}\ntasks.withType(FindBugs) {\n reports {\n xml.enabled false\n html.enabled true\n }\n}\n\napply plugin: 'pmd'\npmd {\n ignoreFailures = false\n pmdTest.enabled = false\n ruleSets = [] // workaround pmd gradle plugin bug\n ruleSetFiles = files(\"${rootProject.projectDir}/tools/conf/pmd.xml\")\n}\ntasks.withType(Pmd){\n reports{\n xml.enabled=true\n html.enabled=true\n }\n}\n\napply plugin: 'checkstyle'\ncheckstyle {\n toolVersion = '7.1.2'\n ignoreFailures = false\n checkstyleTest.enabled = true\n configProperties = [ \"checkstyle.suppressions.file\" : file(\"${rootProject.projectDir}/tools/conf/suppressions.xml\")]\n configFile = file(\"${rootProject.projectDir}/tools/conf/checkstyle.xml\")\n}\ncheckstyleMain {\n classpath += configurations.compile\n}\ntasks.withType(Checkstyle) {\n reports {\n xml.enabled false\n html.enabled true\n }\n}\n\napply plugin: \"jacoco\"\njacoco {\n toolVersion = \"0.8.1\"\n}\njacocoTestReport {\n group = \"Reporting\"\n reports {\n xml.enabled true\n csv.enabled false\n }\n}\n\ncheck.dependsOn jacocoTestReport\ncheck.dependsOn jacocoTestCoverageVerification\n" }, { "path": "plugins/tools/gradle/formatter.gradle", "content": "buildscript {\n repositories {\n maven {\n url \"https://plugins.gradle.org/m2/\"\n }\n }\n dependencies {\n classpath 'com.google.googlejavaformat:google-java-format:1.6'\n }\n}\n\napply plugin: FormatterPlugin\n\ncheck.dependsOn verifyJava\n\nimport com.google.googlejavaformat.java.Formatter\nimport com.google.googlejavaformat.java.ImportOrderer\nimport com.google.googlejavaformat.java.JavaFormatterOptions\nimport com.google.googlejavaformat.java.Main\nimport com.google.googlejavaformat.java.RemoveUnusedImports\n\nclass FormatterPlugin implements Plugin {\n void apply(Project project) {\n project.task('formatJava') {\n doLast {\n Main formatter = new Main(new PrintWriter(System.out, true), new PrintWriter(System.err, true), System.in)\n Project rootProject = project.getRootProject()\n for (item in project.sourceSets) {\n for (File file : item.getAllSource()) {\n if (!file.getName().endsWith(\".java\")) {\n continue\n }\n if (formatter.format(\"-a\", \"-i\", file.getAbsolutePath()) != 0) {\n throw new GradleException(\"Format java failed: \" + file.getAbsolutePath())\n }\n }\n }\n }\n }\n\n project.task('verifyJava') {\n doLast {\n def options = JavaFormatterOptions.builder().style(JavaFormatterOptions.Style.AOSP).build()\n Formatter formatter = new Formatter(options)\n Project rootProject = project.getRootProject()\n for (item in project.sourceSets) {\n for (File file : item.getAllSource()) {\n if (!file.getName().endsWith(\".java\")) {\n continue\n }\n\n String src = file.text\n String formatted = formatter.formatSource(src)\n formatted = RemoveUnusedImports.removeUnusedImports(formatted, RemoveUnusedImports.JavadocOnlyImports.KEEP)\n formatted = ImportOrderer.reorderImports(formatted)\n if (!src.equals(formatted)) {\n throw new GradleException(\"File not formatted: \" + file.getAbsolutePath())\n }\n }\n }\n }\n }\n }\n}\n" }, { "path": "plugins/tools/gradle/launcher.gradle", "content": "apply plugin: LauncherPlugin\n\nclean.dependsOn killServer\n\nimport org.gradle.internal.jvm.Jvm\n\nclass LauncherPlugin implements Plugin {\n void apply(Project project) {\n project.task('startServer') {\n dependsOn project.jar\n doLast {\n def pidFile = getPidFile()\n if (pidFile.exists()) {\n throw new GradleException(\"Server already running!\")\n }\n\n def list = []\n list.addAll(project.configurations.runtime.getFiles())\n list.add(project.jar.outputs.files.singleFile)\n String cp = CollectionUtils.join(File.pathSeparator, list)\n String jvmPath = Jvm.current().getJavaExecutable()\n\n def cmd = [jvmPath, \"-agentlib:jdwp=transport=dt_socket,address=0.0.0.0:4000,server=y,suspend=n\",\n \"-DmmsConfigFile=${project.projectDir}/src/test/resources/config.properties\",\n \"-DLOG_LOCATION=${project.buildDir}/logs\",\n \"-DMETRICS_LOCATION=${project.buildDir}/logs\",\n \"-cp\", cp, \"com.amazonaws.ml.mms.ModelServer\"] as String[]\n\n def builder = new ProcessBuilder(cmd)\n builder.redirectErrorStream(true)\n builder.directory(project.projectDir)\n Process process = builder.start()\n\n ReaderThread rt = new ReaderThread(process.getInputStream())\n rt.start()\n new ReaderThread(process.getErrorStream()).start()\n\n try {\n while (!rt.done) {\n try {\n process.exitValue()\n throw new GradleException(\"MMS stop unexpectedly.\")\n } catch(IllegalThreadStateException ex) {\n Thread.sleep(500)\n }\n }\n\n def pidField = process.class.getDeclaredField('pid')\n pidField.accessible = true\n\n pidFile << pidField.getInt(process)\n\n logger.quiet \"MMS service started.\"\n } catch (IllegalThreadStateException ignored) {\n }\n }\n }\n\n project.task('killServer') {\n doLast {\n def pidFile = getPidFile()\n if(!pidFile.exists()) {\n logger.quiet \"No server running!\"\n return\n }\n\n def pid = pidFile.text\n def process = \"kill $pid\".execute()\n\n try {\n process.waitFor()\n } finally {\n pidFile.delete()\n }\n }\n }\n\n project.task('restartServer') {\n dependsOn project.killServer\n dependsOn project.startServer\n }\n }\n\n private File getPidFile() {\n return new File(\"build/server.pid\")\n }\n}\n\nclass ReaderThread extends Thread {\n\n private InputStream is\n private boolean done\n\n ReaderThread(InputStream is) {\n this.is = is\n }\n\n void run() {\n long begin = System.currentTimeMillis()\n def line\n def reader = new BufferedReader(new InputStreamReader(is))\n while ((line = reader.readLine()) != null) {\n if (!done) {\n done = line.matches(\"Model server started.*\")\n println line\n }\n }\n }\n}\n" }, { "path": "run_ci_tests.sh", "content": "#!/usr/bin/env bash\n#\n# A shell script to build MMS locally.\n# Developer should make sure local build passed before submit PR.\n\nset -e\n\nwhich docker\nif [ $? -ne 0 ]\nthen\n echo \"Please install docker.\"\n exit 1\nfi\n\nMMS_HOME=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd )\"\nBUILDSPEC=\"ci/buildspec.yml\"\n\ndocker pull amazon/aws-codebuild-local:latest\ndocker pull awsdeeplearningteam/mms-build:python2.7@sha256:2b743d6724dead806873cce1330f7b8a0197399a35af47dfd7035251fdade122\ndocker pull awsdeeplearningteam/mms-build:python3.6@sha256:2c1afa8834907ceec641d254dffbf4bcc659ca2d00fd6f2872d7521f32c9fa2e\n\nfind . -name __pycache__ | xargs rm -rf\n\ndocker run -it --rm -v /var/run/docker.sock:/var/run/docker.sock -e \"IMAGE_NAME=awsdeeplearningteam/mms-build:python2.7\" -e \"ARTIFACTS=${MMS_HOME}/build/artifacts2.7\" -e \"SOURCE=${MMS_HOME}\" -e \"BUILDSPEC=${BUILDSPEC}\" amazon/aws-codebuild-local\n\nfind . -name __pycache__ | xargs rm -rf\n\ndocker run -it --rm -v /var/run/docker.sock:/var/run/docker.sock -e \"IMAGE_NAME=awsdeeplearningteam/mms-build:python3.6\" -e \"ARTIFACTS=${MMS_HOME}/build/artifacts3.6\" -e \"SOURCE=${MMS_HOME}\" -e \"BUILDSPEC=${BUILDSPEC}\" amazon/aws-codebuild-local\n" }, { "path": "run_circleci_tests.py", "content": "#!/usr/bin/env python\n\"\"\"\n- This script helps to execute circleci jobs in a container on developer's local machine.\n- The script accepts workflow(mandatory), job(optional) and executor(optional) arguments.\n- The script used circleci cli's process command to generate a processed yaml.\n- The processed yaml, is parsed and twekaed to generate a new transformed yaml.\n- The transformed yaml contains a single job, which is merged and ordered list of job steps\nfrom the specfied and requird parent jobs.\n\"\"\"\n\n# Make sure you have following dependencies installed on your local machine\n# 1. PyYAML (pip install PyYaml)\n# 2. CircleCI cli from - https://circleci.com/docs/2.0/local-cli/#installation\n# 3. docker\n\nfrom collections import OrderedDict\nfrom functools import reduce\n\nimport subprocess\nimport sys\nimport copy\nimport argparse\nimport yaml\n\nparser = argparse.ArgumentParser(description='Execute circleci jobs in a container \\\n on your local machine')\nparser.add_argument('workflow', type=str, help='Workflow name from config.yml')\nparser.add_argument('-j', '--job', type=str, help='Job name from config.yml')\nparser.add_argument('-e', '--executor', type=str, help='Executor name from config.yml')\nargs = parser.parse_args()\n\nworkflow = args.workflow\njob = args.job\nexecutor = args.executor\n\nCCI_CONFIG_FILE = '.circleci/config.yml'\nPROCESSED_FILE = '.circleci/processed.yml'\nXFORMED_FILE = '.circleci/xformed.yml'\nCCI_CONFIG = {}\nPROCESSED_CONFIG = {}\nXFORMED_CONFIG = {}\nXFORMED_JOB_NAME = 'mms_xformed_job'\nBLACKLISTED_STEPS = ['persist_to_workspace', 'attach_workspace', 'store_artifacts']\n\n# Read CircleCI's config\nwith open(CCI_CONFIG_FILE) as fstream:\n try:\n CCI_CONFIG = yaml.safe_load(fstream)\n except yaml.YAMLError as err:\n print(err)\n\n# Create processed YAML using circleci cli's 'config process' commands\nPROCESS_CONFIG_CMD = 'circleci config process {} > {}'.format(CCI_CONFIG_FILE, PROCESSED_FILE)\nprint(\"Executing command : \", PROCESS_CONFIG_CMD)\nsubprocess.check_call(PROCESS_CONFIG_CMD, shell=True)\n\n# Read the processed config\nwith open(PROCESSED_FILE) as fstream:\n try:\n PROCESSED_CONFIG = yaml.safe_load(fstream)\n except yaml.YAMLError as err:\n print(err)\n\n# All executors available in the config file\navailable_executors = list(CCI_CONFIG['executors'])\n\n# All jobs available under the specified workflow\njobs_in_workflow = PROCESSED_CONFIG['workflows'][workflow]['jobs']\n\n\ndef get_processed_job_sequence(processed_job_name):\n \"\"\" Recursively iterate over jobs in the workflow to generate an ordered list of parent jobs \"\"\"\n jobs_in_sequence = []\n\n job_dict = next((jd for jd in jobs_in_workflow \\\n if isinstance(jd, dict) and processed_job_name == list(jd)[0]), None)\n if job_dict:\n # Find all parent jobs, recurse to find their respective ancestors\n parent_jobs = job_dict[processed_job_name].get('requires', [])\n for pjob in parent_jobs:\n jobs_in_sequence += get_processed_job_sequence(pjob)\n\n return jobs_in_sequence + [processed_job_name]\n\n\ndef get_jobs_to_exec(job_name):\n \"\"\" Returns a dictionary of executors and a list of jobs to be executed in them \"\"\"\n jobs_dict = {}\n executors = [executor] if executor else available_executors\n\n for exectr_name in executors:\n if job_name is None:\n # List of all job names(as string)\n jobs_dict[exectr_name] = map(lambda j: j if isinstance(j, str) \\\n else list(j)[0], jobs_in_workflow)\n # Filter processed job names as per the executor\n # \"job_name-executor_name\" is a convention set in config.yml\n # pylint: disable=cell-var-from-loop\n jobs_dict[exectr_name] = filter(lambda j: exectr_name in j, jobs_dict[exectr_name])\n else:\n # The list might contain duplicate parent jobs due to multiple fan-ins like config\n # - Remove the duplicates\n # \"job_name-executor_name\" is a convention set in config.yml\n jobs_dict[exectr_name] = \\\n OrderedDict.fromkeys(get_processed_job_sequence(job_name + '-' + exectr_name))\n jobs_dict[exectr_name] = list(jobs_dict[exectr_name])\n\n return jobs_dict\n\n\n# jobs_to_exec is a dict, with executor(s) as the key and list of jobs to be executed as its value\njobs_to_exec = get_jobs_to_exec(job)\n\n\ndef get_jobs_steps(steps, job_name):\n \"\"\" Merge all the steps from list of jobs to execute \"\"\"\n job_steps = PROCESSED_CONFIG['jobs'][job_name]['steps']\n filtered_job_steps = list(filter(lambda step: list(step)[0] not in BLACKLISTED_STEPS, \\\n job_steps))\n return steps + filtered_job_steps\n\n\nresult_dict = {}\n\nfor exectr, jobs in jobs_to_exec.items():\n merged_steps = reduce(get_jobs_steps, jobs, [])\n\n # Create a new job, using the first job as a reference\n # This ensures configs like executor, environment, etc are maintained from the first job\n first_job = jobs[0]\n xformed_job = copy.deepcopy(PROCESSED_CONFIG['jobs'][first_job])\n\n # Add the merged steps to this newly introduced job\n xformed_job['steps'] = merged_steps\n\n # Create a duplicate config(transformed) with the newly introduced job as the only job in config\n XFORMED_CONFIG = copy.deepcopy(PROCESSED_CONFIG)\n XFORMED_CONFIG['jobs'] = {}\n XFORMED_CONFIG['jobs'][XFORMED_JOB_NAME] = xformed_job\n\n # Create a transformed yaml\n with open(XFORMED_FILE, 'w+') as fstream:\n yaml.dump(XFORMED_CONFIG, fstream)\n\n try:\n # Locally execute the newly created job\n # This newly created job has all the steps (ordered and merged from steps in parent job(s))\n LOCAL_EXECUTE_CMD = 'circleci local execute -c {} --job {}'.format(XFORMED_FILE, \\\n XFORMED_JOB_NAME)\n print('Executing command : ', LOCAL_EXECUTE_CMD)\n result_dict[exectr] = subprocess.check_call(LOCAL_EXECUTE_CMD, shell=True)\n except subprocess.CalledProcessError as err:\n result_dict[exectr] = err.returncode\n\n# Clean up, remove the processed and transformed yml files\nCLEANUP_CMD = 'rm {} {}'.format(PROCESSED_FILE, XFORMED_FILE)\nprint('Executing command : ', CLEANUP_CMD)\nsubprocess.check_call(CLEANUP_CMD, shell=True)\n\n# Print job execution details\nfor exectr, retcode in result_dict.items():\n colorcode, status = ('\\033[0;37;42m', 'successful') if retcode == 0 \\\n else ('\\033[0;37;41m', 'failed')\n print(\"{} Job execution {} using {} executor \\x1b[0m\".format(colorcode, status, exectr))\n\n# Exit as per overall status\nSYS_EXIT_CODE = 0 if all(retcode == 0 for exectr, retcode in result_dict.items()) else 1\nsys.exit(SYS_EXIT_CODE)\n" }, { "path": "serving-sdk/checkstyle.xml", "content": "\n\n\n\n\n\n \n\n \n\n \n \n \n \n \n \n \n \n \n \n \n\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n\n" }, { "path": "serving-sdk/pom.xml", "content": "\n 4.0.0\n software.amazon.ai\n mms-plugins-sdk\n jar\n 1.0.1\n mms-plugins-sdk\n \n SDK for Model Server plugins\n \n \n \n The Apache Software License, Version 2.0\n http://www.apache.org/licenses/LICENSE-2.0.txt\n repo\n \n \n \n scm:git:git://github.com/awslabs/multi-model-server.git\n scm:git:ssh://git@github.com:deep-learning-mms-bot/multi-model-server.git\n https://github.com/awslabs/multi-model-server/tree/master/serving-sdk\n HEAD\n \n \n \n Multi Model Server\n https://github.com/awslabs/multi-model-server\n modelserver.io\n \n \n\n http://maven.apache.org\n \n file://${project.build.directory}/repo\n true\n UTF-8\n UTF-8\n \n\n \n \n junit\n junit\n 4.13.1\n test\n \n \n org.mockito\n mockito-all\n 1.10.19\n test\n \n \n\n \n \n staging\n \n https://oss.sonatype.org/service/local/staging/deploy/maven2/\n false\n \n \n \n \n \n \n org.apache.maven.plugins\n maven-checkstyle-plugin\n 3.1.0\n \n \n checkstyle\n validate\n \n check\n \n \n ${project.basedir}/checkstyle.xml\n true\n \n \n \n \n \n org.apache.maven.plugins\n maven-pmd-plugin\n 3.12.0\n \n \n compile\n \n check\n cpd-check\n \n \n \n \n \n org.apache.maven.plugins\n maven-source-plugin\n 3.1.0\n \n \n attach-sources\n \n jar\n \n \n \n \n \n org.apache.maven.plugins\n maven-javadoc-plugin\n 3.1.0\n \n \n attach-javadocs\n \n jar\n \n \n \n \n \n org.apache.maven.plugins\n maven-gpg-plugin\n 1.6\n \n ${skip.gpg}\n \n \n \n sign-artifacts\n verify\n \n sign\n \n \n \n \n \n org.apache.maven.plugins\n maven-deploy-plugin\n 2.8.2\n \n false\n deployrepo::default::${repo_url}\n \n \n \n org.apache.maven.plugins\n maven-compiler-plugin\n \n 8\n 8\n \n \n \n\n \n\n" }, { "path": "serving-sdk/src/main/java/software/amazon/ai/mms/servingsdk/Context.java", "content": "/*\n * Copyright 2019 Amazon.com, Inc. or its affiliates. All Rights Reserved.\n *\n * Licensed under the Apache License, Version 2.0 (the \"License\"). You may not use this file except in compliance\n * with the License. A copy of the License is located at\n *\n * http://aws.amazon.com/apache2.0/\n *\n * or in the \"license\" file accompanying this file. This file is distributed on an \"AS IS\" BASIS, WITHOUT WARRANTIES\n * OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions\n * and limitations under the License.\n */\npackage software.amazon.ai.mms.servingsdk;\n\nimport java.util.Map;\nimport java.util.Properties;\n\n/**\n * This interface provides access to the current running Model Server.\n */\npublic interface Context {\n /**\n * Get the configuration of the current running Model Server\n * @return Properties\n */\n Properties getConfig();\n\n /**\n * Get a list of Models registered with the Model Server\n * @return List of models\n */\n Map getModels();\n}\n" }, { "path": "serving-sdk/src/main/java/software/amazon/ai/mms/servingsdk/Model.java", "content": "/*\n * Copyright 2019 Amazon.com, Inc. or its affiliates. All Rights Reserved.\n *\n * Licensed under the Apache License, Version 2.0 (the \"License\"). You may not use this file except in compliance\n * with the License. A copy of the License is located at\n *\n * http://aws.amazon.com/apache2.0/\n *\n * or in the \"license\" file accompanying this file. This file is distributed on an \"AS IS\" BASIS, WITHOUT WARRANTIES\n * OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions\n * and limitations under the License.\n */\npackage software.amazon.ai.mms.servingsdk;\n\nimport java.util.List;\n\n/**\n * This provides information about the model which is currently registered with Model Server\n */\npublic interface Model {\n /**\n * Get the name of this model\n * @return The name of this model\n */\n String getModelName();\n\n /**\n * Get the URL of the Model location\n * @return models URL\n */\n String getModelUrl();\n\n /**\n * Get the model's entry-point\n * @return \"handler\" invoked to handle requests\n */\n String getModelHandler();\n\n /**\n * Returns the current list of workers for this model\n * @return list of Worker objects\n */\n List getModelWorkers();\n}\n" }, { "path": "serving-sdk/src/main/java/software/amazon/ai/mms/servingsdk/ModelServerEndpoint.java", "content": "/*\n * Copyright 2019 Amazon.com, Inc. or its affiliates. All Rights Reserved.\n *\n * Licensed under the Apache License, Version 2.0 (the \"License\"). You may not use this file except in compliance\n * with the License. A copy of the License is located at\n *\n * http://aws.amazon.com/apache2.0/\n *\n * or in the \"license\" file accompanying this file. This file is distributed on an \"AS IS\" BASIS, WITHOUT WARRANTIES\n * OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions\n * and limitations under the License.\n */\npackage software.amazon.ai.mms.servingsdk;\n\nimport software.amazon.ai.mms.servingsdk.http.Request;\nimport software.amazon.ai.mms.servingsdk.http.Response;\n\nimport java.io.IOException;\n\n/**\n * This class defines the abstract class for ModelServerEndpoint\n */\npublic abstract class ModelServerEndpoint {\n /**\n * This method is called when a HTTP GET method is invoked for the defined custom model server endpoint\n * @param req - Incoming request\n * @param res - Outgoing response\n * @param ctx - ModelServer's context which defines the current model-server system information\n * @throws IOException if I/O error occurs\n */\n public void doGet(Request req, Response res, Context ctx) throws ModelServerEndpointException, IOException {\n throw new ModelServerEndpointException(\"No implementation found .. Default implementation invoked\");\n }\n\n /**\n * This method is called when a HTTP PUT method is invoked for the defined custom model server endpoint\n * @param req - Incoming request\n * @param res - Outgoing response\n * @param ctx - ModelServer's context which defines the current model-server system information\n * @throws IOException if I/O error occurs\n */\n public void doPut(Request req, Response res, Context ctx) throws ModelServerEndpointException, IOException {\n throw new ModelServerEndpointException(\"No implementation found .. Default implementation invoked\");\n }\n\n /**\n * This method is called when a HTTP POST method is invoked for the defined custom model server endpoint\n * @param req - Incoming request\n * @param res - Outgoing response\n * @param ctx - ModelServer's context which defines the current model-server system information\n * @throws IOException if I/O error occurs\n */\n public void doPost(Request req, Response res, Context ctx) throws ModelServerEndpointException, IOException {\n throw new ModelServerEndpointException(\"No implementation found .. Default implementation invoked\");\n }\n\n /**\n * This method is called when a HTTP DELETE method is invoked for the defined custom model server endpoint\n * @param req - Incoming request\n * @param res - Outgoing response\n * @param ctx - ModelServer's context which defines the current model-server system information\n * @throws IOException if I/O error occurs\n */\n public void doDelete(Request req, Response res, Context ctx) throws ModelServerEndpointException, IOException {\n throw new ModelServerEndpointException(\"No implementation found .. Default implementation invoked\");\n }\n\n}\n" }, { "path": "serving-sdk/src/main/java/software/amazon/ai/mms/servingsdk/ModelServerEndpointException.java", "content": "/*\n * Copyright (c) 2019 Amazon.com, Inc. or its affiliates.\n * All Rights Reserved. Licensed under the Apache License, Version 2.0 (the \"License\").\n * You may not use this file except in compliance with the License. A copy of the License is located at\n * http://aws.amazon.com/apache2.0/ or in the \"license\" file accompanying this file. This file is distributed\n * on an \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for\n * the specific language governing permissions and limitations under the License.\n */\n\npackage software.amazon.ai.mms.servingsdk;\n\n/**\n * Runtime exception for custom model server endpoint plugins\n */\npublic class ModelServerEndpointException extends RuntimeException {\n public ModelServerEndpointException(String err) {super(err);}\n public ModelServerEndpointException(String err, Throwable t) {super(err, t);}\n}\n" }, { "path": "serving-sdk/src/main/java/software/amazon/ai/mms/servingsdk/Worker.java", "content": "/*\n * Copyright (c) 2019 Amazon.com, Inc. or its affiliates. All Rights Reserved. Licensed under the Apache\n * License, Version 2.0 (the \"License\"). You may not use this file except in compliance with the License. A copy of\n * the License is located at http://aws.amazon.com/apache2.0/ or in the \"license\" file accompanying this file. This file\n * is distributed on an \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or\n * implied. See the License for the specific language governing permissionsand limitations under the License.\n */\n\npackage software.amazon.ai.mms.servingsdk;\n\n/**\n * Describe the model worker\n */\npublic interface Worker {\n /**\n * Get the current running status of this model's worker\n * @return True - if the worker is currently running. False - the worker is currently not running.\n */\n boolean isRunning();\n\n /**\n * Get the current memory foot print of this worker\n * @return Current memory usage of this worker\n */\n long getWorkerMemory();\n}\n" }, { "path": "serving-sdk/src/main/java/software/amazon/ai/mms/servingsdk/annotations/Endpoint.java", "content": "/*\n * Copyright 2019 Amazon.com, Inc. or its affiliates. All Rights Reserved.\n *\n * Licensed under the Apache License, Version 2.0 (the \"License\"). You may not use this file except in compliance\n * with the License. A copy of the License is located at\n *\n * http://aws.amazon.com/apache2.0/\n *\n * or in the \"license\" file accompanying this file. This file is distributed on an \"AS IS\" BASIS, WITHOUT WARRANTIES\n * OR CONDITIONS OF ANY KIND, either express or implied.\n * See the License for the specific language governing permissions\n * and limitations under the License.\n */\npackage software.amazon.ai.mms.servingsdk.annotations;\n\nimport software.amazon.ai.mms.servingsdk.annotations.helpers.EndpointTypes;\n\nimport java.lang.annotation.Documented;\nimport java.lang.annotation.ElementType;\nimport java.lang.annotation.Retention;\nimport java.lang.annotation.RetentionPolicy;\nimport java.lang.annotation.Target;\n\n@Target(ElementType.TYPE)\n@Retention(RetentionPolicy.RUNTIME)\n@Documented\npublic @interface Endpoint {\n /**\n * @return URL pattern to which this class applies\n */\n String urlPattern() default \"\";\n\n /**\n * @return Type of this endpoint. Default NONE\n */\n EndpointTypes endpointType() default EndpointTypes.NONE;\n\n /**\n * @return Description of this endpoint. Default \"\"\n */\n String description() default \"\";\n}\n" }, { "path": "serving-sdk/src/main/java/software/amazon/ai/mms/servingsdk/annotations/helpers/EndpointTypes.java", "content": "/*\n * Copyright 2019 Amazon.com, Inc. or its affiliates. All Rights Reserved.\n *\n * Licensed under the Apache License, Version 2.0 (the \"License\"). You may not use this file except in compliance\n * with the License. A copy of the License is located at\n *\n * http://aws.amazon.com/apache2.0/\n *\n * or in the \"license\" file accompanying this file. This file is distributed on an \"AS IS\" BASIS, WITHOUT WARRANTIES\n * OR CONDITIONS OF ANY KIND, either express or implied.\n * See the License for the specific language governing permissions\n * and limitations under the License.\n */\npackage software.amazon.ai.mms.servingsdk.annotations.helpers;\n\n/**\n * Types of ModelServer endpoints\n */\npublic enum EndpointTypes {\n NONE, INFERENCE, MANAGEMENT;\n}\n" }, { "path": "serving-sdk/src/main/java/software/amazon/ai/mms/servingsdk/http/Request.java", "content": "/*\n * Copyright 2019 Amazon.com, Inc. or its affiliates. All Rights Reserved.\n *\n * Licensed under the Apache License, Version 2.0 (the \"License\"). You may not use this file except in compliance\n * with the License. A copy of the License is located at\n *\n * http://aws.amazon.com/apache2.0/\n *\n * or in the \"license\" file accompanying this file. This file is distributed on an \"AS IS\" BASIS, WITHOUT WARRANTIES\n * OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions\n * and limitations under the License.\n */\n\npackage software.amazon.ai.mms.servingsdk.http;\n\nimport java.io.IOException;\nimport java.io.InputStream;\nimport java.util.List;\nimport java.util.Map;\n\n/**\n * This defines the request object given to the custom endpoint\n */\npublic interface Request {\n /**\n * Get all header names in the request object\n * @return List of request header names\n */\n List getHeaderNames();\n\n /**\n * Get the URI of the request\n * @return URI of the endpoint\n */\n String getRequestURI();\n\n /**\n * Get all query parameters coming in for this endpoint\n * @return a dictionary of all the parameters in the query\n */\n Map> getParameterMap();\n\n /**\n * Get a query parameter\n * @param k - Parameter name\n * @return - value of the parameter\n */\n List getParameter(String k);\n\n /**\n * Get the content-type of the incoming request object\n * @return content-type string in the request\n */\n String getContentType();\n\n /**\n * Get the body content stream of the incoming request\n * @return the request content input stream\n * @throws IOException if there is an I/O error\n */\n InputStream getInputStream() throws IOException;\n}\n" }, { "path": "serving-sdk/src/main/java/software/amazon/ai/mms/servingsdk/http/Response.java", "content": "/*\n * Copyright 2019 Amazon.com, Inc. or its affiliates. All Rights Reserved.\n *\n * Licensed under the Apache License, Version 2.0 (the \"License\"). You may not use this file except in compliance\n * with the License. A copy of the License is located at\n *\n * http://aws.amazon.com/apache2.0/\n *\n * or in the \"license\" file accompanying this file. This file is distributed on an \"AS IS\" BASIS, WITHOUT WARRANTIES\n * OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions\n * and limitations under the License.\n */\n\npackage software.amazon.ai.mms.servingsdk.http;\n\nimport java.io.IOException;\nimport java.io.OutputStream;\n\n/**\n * Interface defining the response object sent to the custom defined endpoints\n */\npublic interface Response {\n /**\n * Set HTTP response status\n * @param sc - status code\n */\n void setStatus(int sc);\n\n /**\n * Set HTTP response status code and status phrase\n * @param sc - Integer value representing the status code of this response\n * @param phrase - String phrase representing the status phrase of this response\n */\n void setStatus(int sc, String phrase);\n\n /**\n * Set HTTP headers\n * @param k - Header name\n * @param v - Header value\n */\n void setHeader(String k, String v);\n\n /**\n * Add HTTP headers for an existing header name\n * @param k - Header name\n * @param v - Header value\n */\n void addHeader(String k, String v);\n\n /**\n * Set content type header in the response object\n * @param ct - Content-Type\n */\n void setContentType(String ct);\n\n /**\n * Get the output stream object for response\n * @return response body content as OutputStream\n * @throws IOException if I/O error occurs\n */\n OutputStream getOutputStream() throws IOException;\n}\n" }, { "path": "serving-sdk/src/test/java/software/amazon/ai/mms/servingsdk/ModelServerEndpointTest.java", "content": "/*\n * Copyright 2019 Amazon.com, Inc. or its affiliates. All Rights Reserved.\n *\n * Licensed under the Apache License, Version 2.0 (the \"License\"). You may not use this file except in compliance\n * with the License. A copy of the License is located at\n *\n * http://aws.amazon.com/apache2.0/\n *\n * or in the \"license\" file accompanying this file. This file is distributed on an \"AS IS\" BASIS, WITHOUT WARRANTIES\n * OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions\n * and limitations under the License.\n */\n\npackage software.amazon.ai.mms.servingsdk;\n\n\nimport org.junit.Assert;\nimport org.junit.Before;\nimport org.junit.Test;\nimport org.mockito.Mockito;\nimport org.mockito.stubbing.Answer;\nimport software.amazon.ai.mms.servingsdk.annotations.Endpoint;\nimport software.amazon.ai.mms.servingsdk.annotations.helpers.EndpointTypes;\nimport software.amazon.ai.mms.servingsdk.http.Request;\nimport software.amazon.ai.mms.servingsdk.http.Response;\n\nimport java.io.ByteArrayOutputStream;\nimport java.io.IOException;\nimport java.lang.reflect.Method;\nimport java.util.ArrayList;\nimport java.util.HashMap;\nimport java.util.List;\nimport java.util.Properties;\n\n/**\n * Unit test for simple App.\n */\npublic class ModelServerEndpointTest {\n Context c;\n Model m;\n Worker w;\n ModelServerEndpoint mse;\n Request req;\n Response rsp;\n ByteArrayOutputStream outputStream;\n Endpoint ea;\n\n @Before\n public void beforeSuite() throws IOException {\n c = Mockito.mock(Context.class);\n m = Mockito.mock(Model.class);\n w = Mockito.mock(Worker.class);\n ea = Mockito.mock(Endpoint.class);\n mse = Mockito.mock(ModelServerEndpoint.class);\n req = Mockito.mock(Request.class);\n rsp = Mockito.mock(Response.class);\n outputStream = new ByteArrayOutputStream();\n\n Properties p = new Properties();\n HashMap map = new HashMap<>();\n List l = new ArrayList<>();\n map.put(\"squeezenet\", m);\n p.setProperty(\"Hello\", \"World\");\n c.getConfig();\n l.add(w);\n\n Mockito.when(c.getConfig()).thenReturn(p);\n Mockito.when(c.getModels()).thenReturn(map);\n Mockito.when(m.getModelWorkers()).thenReturn(l);\n Mockito.when(m.getModelHandler()).thenReturn(\"mxnet_service:handle\");\n Mockito.when(m.getModelUrl()).thenReturn(\"/tmp/model/squeezenet.mar\");\n Mockito.when(m.getModelName()).thenReturn(\"squeezenet\");\n Mockito.when(w.getWorkerMemory()).thenReturn((long)100);\n Mockito.when(w.isRunning()).thenReturn(false);\n Mockito.when(ea.urlPattern()).thenReturn(\"myEndpoint\");\n Mockito.when(ea.description()).thenReturn(\"This is a test endpoint\");\n Mockito.when(ea.endpointType()).thenReturn(EndpointTypes.INFERENCE);\n Mockito.when(rsp.getOutputStream()).thenReturn(outputStream);\n }\n\n @Test\n public void test() throws IOException {\n testContextInterface();\n testEndpointAnnotation();\n testEndpointInterface();\n }\n\n private void testEndpointInterface() throws IOException {\n Class ep = ModelServerEndpoint.class;\n Assert.assertEquals(4, ep.getDeclaredMethods().length);\n for(Method m : ep.getDeclaredMethods()) {\n switch (m.getName()) {\n case \"doGet\":\n case \"doPost\":\n case \"doDelete\":\n case \"doPut\":\n break;\n default:\n Assert.fail(\"Invalid method found\");\n }\n }\n\n // Check signatures\n Mockito.doAnswer((Answer) i -> {\n Object rq = i.getArguments()[0];\n Object rs = i.getArguments()[1];\n Object ctx = i.getArguments()[2];\n\n ((Response)rs).getOutputStream().write(\"This is a test\".getBytes());\n return null;\n }).when(mse).doGet(req, rsp, c);\n\n mse.doGet(req, rsp, c);\n\n Assert.assertEquals(\"This is a test\", outputStream.toString());\n outputStream.reset();\n\n // Check signatures\n Mockito.doAnswer((Answer) i -> {\n Object rq = i.getArguments()[0];\n Object rs = i.getArguments()[1];\n Object ctx = i.getArguments()[2];\n ((Response)rs).getOutputStream().write(\"This is a test\".getBytes());\n return null;\n }).when(mse).doPost(req, rsp, c);\n\n mse.doPost(req, rsp, c);\n\n Assert.assertEquals(\"This is a test\", outputStream.toString());\n outputStream.reset();\n\n // Check signatures\n Mockito.doAnswer((Answer) i -> {\n Object rq = i.getArguments()[0];\n Object rs = i.getArguments()[1];\n Object ctx = i.getArguments()[2];\n\n ((Response)rs).getOutputStream().write(\"This is a test\".getBytes());\n return null;\n }).when(mse).doPut(req, rsp, c);\n\n mse.doPut(req, rsp, c);\n\n Assert.assertEquals(\"This is a test\", outputStream.toString());\n outputStream.reset();\n\n // Check signatures\n Mockito.doAnswer((Answer) i -> {\n Object rq = i.getArguments()[0];\n Object rs = i.getArguments()[1];\n Object ctx = i.getArguments()[2];\n\n ((Response)rs).getOutputStream().write(\"This is a test\".getBytes());\n return null;\n }).when(mse).doDelete(req, rsp, c);\n\n mse.doDelete(req, rsp, c);\n\n Assert.assertEquals(\"This is a test\", outputStream.toString());\n }\n\n private void testEndpointAnnotation() {\n Assert.assertEquals(3, Endpoint.class.getDeclaredMethods().length);\n Assert.assertEquals(\"myEndpoint\", ea.urlPattern());\n Assert.assertEquals(EndpointTypes.INFERENCE, ea.endpointType());\n Assert.assertEquals(\"This is a test endpoint\", ea.description());\n Assert.assertEquals(3, EndpointTypes.class.getFields().length);\n }\n\n private void testWorkerInterface(Worker w) {\n Assert.assertNotNull(w);\n Assert.assertFalse( w.isRunning());\n Assert.assertEquals(100, w.getWorkerMemory());\n Assert.assertEquals(2, Worker.class.getDeclaredMethods().length);\n }\n\n private void testModelInterface(Model m) {\n Assert.assertEquals(\"squeezenet\", m.getModelName());\n Assert.assertEquals(\"/tmp/model/squeezenet.mar\", m.getModelUrl());\n Assert.assertEquals(\"mxnet_service:handle\", m.getModelHandler());\n Assert.assertEquals(1, m.getModelWorkers().size());\n Assert.assertEquals(4, Model.class.getDeclaredMethods().length);\n testWorkerInterface(m.getModelWorkers().get(0));\n }\n\n private void testContextInterface() {\n Assert.assertNotNull(c.getModels());\n Assert.assertTrue(c.getModels().containsKey(\"squeezenet\"));\n Assert.assertTrue(c.getConfig().containsKey(\"Hello\"));\n Assert.assertEquals(\"World\", c.getConfig().getProperty(\"Hello\"));\n Assert.assertEquals(2, Context.class.getDeclaredMethods().length);\n testModelInterface(c.getModels().get(\"squeezenet\"));\n }\n}\n" }, { "path": "setup.py", "content": "# Copyright 2017 Amazon.com, Inc. or its affiliates. All Rights Reserved.\n# Licensed under the Apache License, Version 2.0 (the \"License\").\n# You may not use this file except in compliance with the License.\n# A copy of the License is located at\n# http://www.apache.org/licenses/LICENSE-2.0\n# or in the \"license\" file accompanying this file. This file is distributed\n# on an \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either\n# express or implied. See the License for the specific language governing\n# permissions and limitations under the License.\n\n# To build and upload a new version, follow the steps below.\n# Notes:\n# - this is a \"Universal Wheels\" package that is pure Python and supports both Python2 and Python3\n# - Twine is a secure PyPi upload package\n# - Make sure you have bumped the version! at mms/version.py\n# $ pip install twine\n# $ pip install wheel\n# $ python setup.py bdist_wheel --universal\n\n# *** TEST YOUR PACKAGE WITH TEST PI ******\n# twine upload --repository-url https://test.pypi.org/legacy/ dist/*\n\n# If this is successful then push it to actual pypi\n\n# $ twine upload dist/*\n\n\"\"\"\nSetup.py for the model server package\n\"\"\"\n\nimport errno\nimport os\nimport subprocess\nimport sys\nfrom datetime import date\nfrom shutil import copy2, rmtree\n\nimport setuptools.command.build_py\nfrom setuptools import setup, find_packages, Command\n\nimport mms\n\npkgs = find_packages()\n\ndef pypi_description():\n \"\"\"\n Imports the long description for the project page\n \"\"\"\n with open('PyPiDescription.rst') as df:\n return df.read()\n\n\ndef detect_model_server_version():\n sys.path.append(os.path.abspath(\"mms\"))\n if \"--release\" in sys.argv:\n sys.argv.remove(\"--release\")\n return mms.__version__.strip()\n\n return mms.__version__.strip() + 'b' + str(date.today()).replace('-', '')\n\n\nclass BuildFrontEnd(setuptools.command.build_py.build_py):\n \"\"\"\n Class defined to run custom commands.\n \"\"\"\n description = 'Build Model Server Frontend'\n source_server_file = os.path.abspath('frontend/server/build/libs/server-1.0.jar')\n dest_file_name = os.path.abspath('mms/frontend/model-server.jar')\n\n # noinspection PyMethodMayBeStatic\n def run(self):\n \"\"\"\n Actual method called to run the build command\n :return:\n \"\"\"\n front_end_bin_dir = os.path.abspath('.') + '/mms/frontend'\n try:\n os.mkdir(front_end_bin_dir)\n except OSError as exc:\n if exc.errno == errno.EEXIST and os.path.isdir(front_end_bin_dir):\n pass\n else:\n raise\n\n if os.path.exists(self.source_server_file):\n os.remove(self.source_server_file)\n\n # Remove build/lib directory.\n if os.path.exists('build/lib/'):\n rmtree('build/lib/')\n\n try:\n subprocess.check_call('frontend/gradlew -p frontend clean build', shell=True)\n except OSError:\n assert 0, \"build failed\"\n copy2(self.source_server_file, self.dest_file_name)\n\n\nclass BuildPy(setuptools.command.build_py.build_py):\n \"\"\"\n Class to invoke the custom command defined above.\n \"\"\"\n\n def run(self):\n sys.stderr.flush()\n self.run_command('build_frontend')\n setuptools.command.build_py.build_py.run(self)\n\n\nclass BuildPlugins(Command):\n description = 'Build Model Server Plugins'\n user_options = [('plugins=', 'p', 'Plugins installed')]\n source_plugin_dir = \\\n os.path.abspath('plugins/build/plugins')\n\n def initialize_options(self):\n self.plugins = None\n\n def finalize_options(self):\n if self.plugins is None:\n print(\"No plugin option provided. Defaulting to 'default'\")\n self.plugins = \"default\"\n\n # noinspection PyMethodMayBeStatic\n def run(self):\n if os.path.isdir(self.source_plugin_dir):\n rmtree(self.source_plugin_dir)\n\n try:\n if self.plugins == \"sagemaker\":\n subprocess.check_call('plugins/gradlew -p plugins clean bS', shell=True)\n else:\n raise OSError(\"No such rule exists\")\n except OSError:\n assert 0, \"build failed\"\n\n self.run_command('build_py')\n\n\nif __name__ == '__main__':\n version = detect_model_server_version()\n\n requirements = ['Pillow', 'psutil', 'future', 'model-archiver']\n\n setup(\n name='multi-model-server',\n version=version,\n description='Multi Model Server is a tool for serving neural net models for inference',\n author='Trinity team',\n author_email='noreply@amazon.com',\n long_description=pypi_description(),\n url='https://github.com/awslabs/multi-model-server',\n keywords='Multi Model Server Serving Deep Learning Inference AI',\n packages=pkgs,\n cmdclass={\n 'build_frontend': BuildFrontEnd,\n 'build_plugins': BuildPlugins,\n 'build_py': BuildPy,\n },\n install_requires=requirements,\n extras_require={\n 'mxnet-mkl': ['mxnet-mkl'],\n 'mxnet-cu90mkl': ['mxnet-cu90mkl'],\n 'mxnet': ['mxnet'],\n },\n entry_points={\n 'console_scripts': [\n 'multi-model-server=mms.model_server:start',\n 'mxnet-model-server=mms.model_server:old_start',\n 'multi-model-export=mms.export_model:main'\n ]\n },\n include_package_data=True,\n license='Apache License Version 2.0'\n )\n" }, { "path": "test/README.md", "content": "# MMS Regression Tests\n\nThis folder contains regression tests executed against MMS master.These tests use [POSTMAN](https://www.postman.com/downloads/) for exercising all the Management & Inference APIs.\n\n### Running the test manually.\n\nPull multi-model-server pre build docker image\n```\ndocker pull awsdeeplearningteam/multi-model-server\n```\n\nThis would build a docker Image with a awsdeeplearningteam/multi-model-server:latest in which we would run our Regression Tests.\n\n```\ndocker run -it --user root awsdeeplearningteam/multi-model-server:latest /bin/bash\n```\n\nIn the Docker CLI execute the following cmds.\n\n```\napt-get update \napt-get install -y git wget sudo \ngit clone https://github.com/awslabs/multi-model-server.git\ncd multi-model-server\n```\nTo execute tests on master run: \n\n`./test/regression_tests.sh `\n\nTo execute tests on different run: \n\n`./test/regression_tests.sh `\n\n\nYou can view the logs for Test execution & the Multi-model-server in the /tmp dir.\n\n```\ncat /tmp/test_exec.log\ncat /tmp/mms.log \n```\n\n### Adding tests\n\nTo add to the tests, import a collection (in /postman) to Postman and add new requests.\nSpecifically to test for inference against a new model\n* Open /postman/inference_data.json\n* Add new json object with the new model url and payload.\n\n![POSTMAN UI](screenshot/postman.png)\n\nAfterwards, export the collection as a v2.1 collection and replace the existing exported collection.\nTo add a new suite of tests, add a new collection to /postman and update regression_tests.sh to run the new collection and buldsepc.yml to keep track of the report.\n" }, { "path": "test/postman/environment.json", "content": "{\n\t\"id\": \"b9eb6b86-585b-4fca-abd0-972ec686e3e8\",\n\t\"name\": \"Multi-Model-server\",\n\t\"values\": [\n\t\t{\n\t\t\t\"key\": \"hostname\",\n\t\t\t\"value\": \"localhost\",\n\t\t\t\"enabled\": true\n\t\t},\n\t\t{\n\t\t\t\"key\": \"protocol\",\n\t\t\t\"value\": \"http\",\n\t\t\t\"enabled\": true\n\t\t},\n\t\t{\n\t\t\t\"key\": \"mgmt-port\",\n\t\t\t\"value\": \"8081\",\n\t\t\t\"enabled\": true\n\t\t},\n\t\t{\n\t\t\t\"key\": \"model-name\",\n\t\t\t\"value\": \"densenet161\",\n\t\t\t\"enabled\": true\n\t\t},\n\t\t{\n\t\t\t\"key\": \"pred-port\",\n\t\t\t\"value\": \"8080\",\n\t\t\t\"enabled\": true\n\t\t},\n\t\t{\n\t\t\t\"key\": \"sec-mgmt-port\",\n\t\t\t\"value\": \"8444\",\n\t\t\t\"enabled\": true\n\t\t},\n\t\t{\n\t\t\t\"key\": \"sec-pred-port\",\n\t\t\t\"value\": \"8443\",\n\t\t\t\"enabled\": true\n\t\t}\n\t],\n\t\"_postman_variable_scope\": \"environment\",\n\t\"_postman_exported_at\": \"2020-05-15T07:34:40.974Z\",\n\t\"_postman_exported_using\": \"Postman/7.24.0\"\n}" }, { "path": "test/postman/https_test_collection.json", "content": "{\n\t\"info\": {\n\t\t\"_postman_id\": \"cd002524-ce12-4ff7-ab75-932a05403f83\",\n\t\t\"name\": \"MMS - https_test_collection\",\n\t\t\"schema\": \"https://schema.getpostman.com/json/collection/v2.1.0/collection.json\"\n\t},\n\t\"item\": [\n\t\t{\n\t\t\t\"name\": \"HTTPS Inference API Description\",\n\t\t\t\"event\": [\n\t\t\t\t{\n\t\t\t\t\t\"listen\": \"test\",\n\t\t\t\t\t\"script\": {\n\t\t\t\t\t\t\"id\": \"638ec081-ebf4-4634-a9ea-f675613c2127\",\n\t\t\t\t\t\t\"exec\": [\n\t\t\t\t\t\t\t\"pm.test(\\\"Status code is 200\\\", function () {\",\n\t\t\t\t\t\t\t\" pm.response.to.have.status(200);\",\n\t\t\t\t\t\t\t\"});\"\n\t\t\t\t\t\t],\n\t\t\t\t\t\t\"type\": \"text/javascript\"\n\t\t\t\t\t}\n\t\t\t\t}\n\t\t\t],\n\t\t\t\"request\": {\n\t\t\t\t\"method\": \"OPTIONS\",\n\t\t\t\t\"header\": [],\n\t\t\t\t\"url\": {\n\t\t\t\t\t\"raw\": \"https://{{hostname}}:{{sec-pred-port}}\",\n\t\t\t\t\t\"protocol\": \"https\",\n\t\t\t\t\t\"host\": [\n\t\t\t\t\t\t\"{{hostname}}\"\n\t\t\t\t\t],\n\t\t\t\t\t\"port\": \"{{sec-pred-port}}\"\n\t\t\t\t}\n\t\t\t},\n\t\t\t\"response\": []\n\t\t},\n\t\t{\n\t\t\t\"name\": \"HTTPS Management API Description\",\n\t\t\t\"event\": [\n\t\t\t\t{\n\t\t\t\t\t\"listen\": \"test\",\n\t\t\t\t\t\"script\": {\n\t\t\t\t\t\t\"id\": \"81b8730a-0b89-4569-b042-1076266563ba\",\n\t\t\t\t\t\t\"exec\": [\n\t\t\t\t\t\t\t\"pm.test(\\\"Status code is 200\\\", function () {\",\n\t\t\t\t\t\t\t\" pm.response.to.have.status(200);\",\n\t\t\t\t\t\t\t\"});\"\n\t\t\t\t\t\t],\n\t\t\t\t\t\t\"type\": \"text/javascript\"\n\t\t\t\t\t}\n\t\t\t\t}\n\t\t\t],\n\t\t\t\"request\": {\n\t\t\t\t\"method\": \"OPTIONS\",\n\t\t\t\t\"header\": [],\n\t\t\t\t\"url\": {\n\t\t\t\t\t\"raw\": \"https://{{hostname}}:{{sec-pred-port}}\",\n\t\t\t\t\t\"protocol\": \"https\",\n\t\t\t\t\t\"host\": [\n\t\t\t\t\t\t\"{{hostname}}\"\n\t\t\t\t\t],\n\t\t\t\t\t\"port\": \"{{sec-pred-port}}\"\n\t\t\t\t}\n\t\t\t},\n\t\t\t\"response\": []\n\t\t},\n\t\t{\n\t\t\t\"name\": \"HTTPS Register Model - SqueezeNet\",\n\t\t\t\"event\": [\n\t\t\t\t{\n\t\t\t\t\t\"listen\": \"test\",\n\t\t\t\t\t\"script\": {\n\t\t\t\t\t\t\"id\": \"7e9a9528-e9f1-446c-b1ba-7dce112ffa30\",\n\t\t\t\t\t\t\"exec\": [\n\t\t\t\t\t\t\t\"pm.test(\\\"Successful POST request\\\", function () {\",\n\t\t\t\t\t\t\t\" pm.expect(pm.response.code).to.be.oneOf([200]);\",\n\t\t\t\t\t\t\t\"});\"\n\t\t\t\t\t\t],\n\t\t\t\t\t\t\"type\": \"text/javascript\"\n\t\t\t\t\t}\n\t\t\t\t}\n\t\t\t],\n\t\t\t\"request\": {\n\t\t\t\t\"method\": \"POST\",\n\t\t\t\t\"header\": [],\n\t\t\t\t\"url\": {\n\t\t\t\t\t\"raw\": \"https://{{hostname}}:{{sec-mgmt-port}}/models?url=https://s3.amazonaws.com/model-server/model_archive_1.0/squeezenet_v1.1.mar&model_name=squeezenetv1_1&initial_workers=1&synchronous=true\",\n\t\t\t\t\t\"protocol\": \"https\",\n\t\t\t\t\t\"host\": [\n\t\t\t\t\t\t\"{{hostname}}\"\n\t\t\t\t\t],\n\t\t\t\t\t\"port\": \"{{sec-mgmt-port}}\",\n\t\t\t\t\t\"path\": [\n\t\t\t\t\t\t\"models\"\n\t\t\t\t\t],\n\t\t\t\t\t\"query\": [\n\t\t\t\t\t\t{\n\t\t\t\t\t\t\t\"key\": \"url\",\n\t\t\t\t\t\t\t\"value\": \"https://s3.amazonaws.com/model-server/model_archive_1.0/squeezenet_v1.1.mar\"\n\t\t\t\t\t\t},\n\t\t\t\t\t\t{\n\t\t\t\t\t\t\t\"key\": \"model_name\",\n\t\t\t\t\t\t\t\"value\": \"squeezenetv1_1\"\n\t\t\t\t\t\t},\n\t\t\t\t\t\t{\n\t\t\t\t\t\t\t\"key\": \"initial_workers\",\n\t\t\t\t\t\t\t\"value\": \"1\"\n\t\t\t\t\t\t},\n\t\t\t\t\t\t{\n\t\t\t\t\t\t\t\"key\": \"synchronous\",\n\t\t\t\t\t\t\t\"value\": \"true\"\n\t\t\t\t\t\t}\n\t\t\t\t\t]\n\t\t\t\t}\n\t\t\t},\n\t\t\t\"response\": []\n\t\t},\n\t\t{\n\t\t\t\"name\": \"HTTPS Get SqueezeNet Model Description\",\n\t\t\t\"event\": [\n\t\t\t\t{\n\t\t\t\t\t\"listen\": \"test\",\n\t\t\t\t\t\"script\": {\n\t\t\t\t\t\t\"id\": \"cd11b9cc-335a-415a-8315-54ddda6f6d8a\",\n\t\t\t\t\t\t\"exec\": [\n\t\t\t\t\t\t\t\"pm.test(\\\"Successful GET request\\\", function () {\",\n\t\t\t\t\t\t\t\" pm.expect(pm.response.code).to.be.oneOf([200]);\",\n\t\t\t\t\t\t\t\"});\"\n\t\t\t\t\t\t],\n\t\t\t\t\t\t\"type\": \"text/javascript\"\n\t\t\t\t\t}\n\t\t\t\t}\n\t\t\t],\n\t\t\t\"request\": {\n\t\t\t\t\"method\": \"GET\",\n\t\t\t\t\"header\": [],\n\t\t\t\t\"url\": {\n\t\t\t\t\t\"raw\": \"https://{{hostname}}:{{sec-mgmt-port}}/models/squeezenetv1_1\",\n\t\t\t\t\t\"protocol\": \"https\",\n\t\t\t\t\t\"host\": [\n\t\t\t\t\t\t\"{{hostname}}\"\n\t\t\t\t\t],\n\t\t\t\t\t\"port\": \"{{sec-mgmt-port}}\",\n\t\t\t\t\t\"path\": [\n\t\t\t\t\t\t\"models\",\n\t\t\t\t\t\t\"squeezenetv1_1\"\n\t\t\t\t\t]\n\t\t\t\t}\n\t\t\t},\n\t\t\t\"response\": []\n\t\t},\n\t\t{\n\t\t\t\"name\": \"HTTPS Scale up Workers - Synchronous\",\n\t\t\t\"event\": [\n\t\t\t\t{\n\t\t\t\t\t\"listen\": \"test\",\n\t\t\t\t\t\"script\": {\n\t\t\t\t\t\t\"id\": \"ba06ca94-e630-4f72-adf4-49d1a80ded31\",\n\t\t\t\t\t\t\"exec\": [\n\t\t\t\t\t\t\t\"pm.test(\\\"Successful PUT request\\\", function () {\",\n\t\t\t\t\t\t\t\" pm.expect(pm.response.code).to.be.oneOf([200, 201, 202]);\",\n\t\t\t\t\t\t\t\"});\"\n\t\t\t\t\t\t],\n\t\t\t\t\t\t\"type\": \"text/javascript\"\n\t\t\t\t\t}\n\t\t\t\t}\n\t\t\t],\n\t\t\t\"request\": {\n\t\t\t\t\"method\": \"PUT\",\n\t\t\t\t\"header\": [],\n\t\t\t\t\"url\": {\n\t\t\t\t\t\"raw\": \"https://{{hostname}}:{{sec-mgmt-port}}/models/squeezenetv1_1?min_worker=5&max_worker=5&synchronous=true\",\n\t\t\t\t\t\"protocol\": \"https\",\n\t\t\t\t\t\"host\": [\n\t\t\t\t\t\t\"{{hostname}}\"\n\t\t\t\t\t],\n\t\t\t\t\t\"port\": \"{{sec-mgmt-port}}\",\n\t\t\t\t\t\"path\": [\n\t\t\t\t\t\t\"models\",\n\t\t\t\t\t\t\"squeezenetv1_1\"\n\t\t\t\t\t],\n\t\t\t\t\t\"query\": [\n\t\t\t\t\t\t{\n\t\t\t\t\t\t\t\"key\": \"min_worker\",\n\t\t\t\t\t\t\t\"value\": \"5\"\n\t\t\t\t\t\t},\n\t\t\t\t\t\t{\n\t\t\t\t\t\t\t\"key\": \"max_worker\",\n\t\t\t\t\t\t\t\"value\": \"5\"\n\t\t\t\t\t\t},\n\t\t\t\t\t\t{\n\t\t\t\t\t\t\t\"key\": \"synchronous\",\n\t\t\t\t\t\t\t\"value\": \"true\"\n\t\t\t\t\t\t}\n\t\t\t\t\t]\n\t\t\t\t}\n\t\t\t},\n\t\t\t\"response\": []\n\t\t},\n\t\t{\n\t\t\t\"name\": \"HTTPS Scale up Workers - Asynchronous\",\n\t\t\t\"event\": [\n\t\t\t\t{\n\t\t\t\t\t\"listen\": \"test\",\n\t\t\t\t\t\"script\": {\n\t\t\t\t\t\t\"id\": \"5a13e310-65a9-4982-84c7-d89f29d6c27a\",\n\t\t\t\t\t\t\"exec\": [\n\t\t\t\t\t\t\t\"pm.test(\\\"Successful PUT request\\\", function () {\",\n\t\t\t\t\t\t\t\" pm.expect(pm.response.code).to.be.oneOf([200,201,202]);\",\n\t\t\t\t\t\t\t\"});\"\n\t\t\t\t\t\t],\n\t\t\t\t\t\t\"type\": \"text/javascript\"\n\t\t\t\t\t}\n\t\t\t\t}\n\t\t\t],\n\t\t\t\"request\": {\n\t\t\t\t\"method\": \"PUT\",\n\t\t\t\t\"header\": [],\n\t\t\t\t\"url\": {\n\t\t\t\t\t\"raw\": \"https://{{hostname}}:{{sec-mgmt-port}}/models/squeezenetv1_1?min_worker=6&max_worker=6&synchronous=false\",\n\t\t\t\t\t\"protocol\": \"https\",\n\t\t\t\t\t\"host\": [\n\t\t\t\t\t\t\"{{hostname}}\"\n\t\t\t\t\t],\n\t\t\t\t\t\"port\": \"{{sec-mgmt-port}}\",\n\t\t\t\t\t\"path\": [\n\t\t\t\t\t\t\"models\",\n\t\t\t\t\t\t\"squeezenetv1_1\"\n\t\t\t\t\t],\n\t\t\t\t\t\"query\": [\n\t\t\t\t\t\t{\n\t\t\t\t\t\t\t\"key\": \"min_worker\",\n\t\t\t\t\t\t\t\"value\": \"6\"\n\t\t\t\t\t\t},\n\t\t\t\t\t\t{\n\t\t\t\t\t\t\t\"key\": \"max_worker\",\n\t\t\t\t\t\t\t\"value\": \"6\"\n\t\t\t\t\t\t},\n\t\t\t\t\t\t{\n\t\t\t\t\t\t\t\"key\": \"synchronous\",\n\t\t\t\t\t\t\t\"value\": \"false\"\n\t\t\t\t\t\t}\n\t\t\t\t\t]\n\t\t\t\t}\n\t\t\t},\n\t\t\t\"response\": []\n\t\t},\n\t\t{\n\t\t\t\"name\": \"HTTPS - Inference - SqueezeNet\",\n\t\t\t\"event\": [\n\t\t\t\t{\n\t\t\t\t\t\"listen\": \"test\",\n\t\t\t\t\t\"script\": {\n\t\t\t\t\t\t\"id\": \"7c1c4eaa-48f8-4734-8737-78b4b2766b29\",\n\t\t\t\t\t\t\"exec\": [\n\t\t\t\t\t\t\t\"pm.test(\\\"Status code is 200\\\", function () {\",\n\t\t\t\t\t\t\t\" pm.response.to.have.status(200);\",\n\t\t\t\t\t\t\t\"});\"\n\t\t\t\t\t\t],\n\t\t\t\t\t\t\"type\": \"text/javascript\"\n\t\t\t\t\t}\n\t\t\t\t}\n\t\t\t],\n\t\t\t\"protocolProfileBehavior\": {\n\t\t\t\t\"disabledSystemHeaders\": {\n\t\t\t\t\t\"content-type\": true\n\t\t\t\t}\n\t\t\t},\n\t\t\t\"request\": {\n\t\t\t\t\"method\": \"POST\",\n\t\t\t\t\"header\": [],\n\t\t\t\t\"body\": {\n\t\t\t\t\t\"mode\": \"file\",\n\t\t\t\t\t\"file\": {\n\t\t\t\t\t\t\"src\": \"../examples/image_classifier/kitten.jpg\"\n\t\t\t\t\t},\n\t\t\t\t\t\"options\": {\n\t\t\t\t\t\t\"raw\": {\n\t\t\t\t\t\t\t\"language\": \"text\"\n\t\t\t\t\t\t}\n\t\t\t\t\t}\n\t\t\t\t},\n\t\t\t\t\"url\": {\n\t\t\t\t\t\"raw\": \"https://{{hostname}}:{{sec-pred-port}}/predictions/squeezenetv1_1\",\n\t\t\t\t\t\"protocol\": \"https\",\n\t\t\t\t\t\"host\": [\n\t\t\t\t\t\t\"{{hostname}}\"\n\t\t\t\t\t],\n\t\t\t\t\t\"port\": \"{{sec-pred-port}}\",\n\t\t\t\t\t\"path\": [\n\t\t\t\t\t\t\"predictions\",\n\t\t\t\t\t\t\"squeezenetv1_1\"\n\t\t\t\t\t]\n\t\t\t\t}\n\t\t\t},\n\t\t\t\"response\": []\n\t\t},\n\t\t{\n\t\t\t\"name\": \"HTTPS UnRegister Model SqueezeNet\",\n\t\t\t\"event\": [\n\t\t\t\t{\n\t\t\t\t\t\"listen\": \"test\",\n\t\t\t\t\t\"script\": {\n\t\t\t\t\t\t\"id\": \"de94e7b6-d4fa-4e10-8b54-4052753de19e\",\n\t\t\t\t\t\t\"exec\": [\n\t\t\t\t\t\t\t\"pm.test(\\\"Successful DELETE request\\\", function () {\",\n\t\t\t\t\t\t\t\" pm.expect(pm.response.code).to.be.oneOf([200]);\",\n\t\t\t\t\t\t\t\"});\"\n\t\t\t\t\t\t],\n\t\t\t\t\t\t\"type\": \"text/javascript\"\n\t\t\t\t\t}\n\t\t\t\t}\n\t\t\t],\n\t\t\t\"request\": {\n\t\t\t\t\"method\": \"DELETE\",\n\t\t\t\t\"header\": [],\n\t\t\t\t\"url\": {\n\t\t\t\t\t\"raw\": \"https://{{hostname}}:{{sec-mgmt-port}}/models/squeezenetv1_1\",\n\t\t\t\t\t\"protocol\": \"https\",\n\t\t\t\t\t\"host\": [\n\t\t\t\t\t\t\"{{hostname}}\"\n\t\t\t\t\t],\n\t\t\t\t\t\"port\": \"{{sec-mgmt-port}}\",\n\t\t\t\t\t\"path\": [\n\t\t\t\t\t\t\"models\",\n\t\t\t\t\t\t\"squeezenetv1_1\"\n\t\t\t\t\t]\n\t\t\t\t}\n\t\t\t},\n\t\t\t\"response\": []\n\t\t}\n\t],\n\t\"protocolProfileBehavior\": {}\n}" }, { "path": "test/postman/inference_api_test_collection.json", "content": "{\n\t\"info\": {\n\t\t\"_postman_id\": \"e69000d9-d3c8-49bd-879a-ad42b95b042a\",\n\t\t\"name\": \"MMS - inference_api_collection\",\n\t\t\"schema\": \"https://schema.getpostman.com/json/collection/v2.1.0/collection.json\"\n\t},\n\t\"item\": [\n\t\t{\n\t\t\t\"name\": \"Model Zoo - Register Model\",\n\t\t\t\"event\": [\n\t\t\t\t{\n\t\t\t\t\t\"listen\": \"test\",\n\t\t\t\t\t\"script\": {\n\t\t\t\t\t\t\"id\": \"80fa33ea-ff6a-4535-9328-ddffb980062a\",\n\t\t\t\t\t\t\"exec\": [\n\t\t\t\t\t\t\t\"pm.test(\\\"Successful POST request\\\", function () {\",\n\t\t\t\t\t\t\t\" pm.expect(pm.response.code).to.be.oneOf([200]);\",\n\t\t\t\t\t\t\t\"});\"\n\t\t\t\t\t\t],\n\t\t\t\t\t\t\"type\": \"text/javascript\"\n\t\t\t\t\t}\n\t\t\t\t}\n\t\t\t],\n\t\t\t\"request\": {\n\t\t\t\t\"method\": \"POST\",\n\t\t\t\t\"header\": [],\n\t\t\t\t\"url\": {\n\t\t\t\t\t\"raw\": \"{{protocol}}://{{hostname}}:{{mgmt-port}}/models?url={{url}}&model_name={{model_name}}&initial_workers={{worker}}&synchronous={{synchronous}}\",\n\t\t\t\t\t\"protocol\": \"{{protocol}}\",\n\t\t\t\t\t\"host\": [\n\t\t\t\t\t\t\"{{hostname}}\"\n\t\t\t\t\t],\n\t\t\t\t\t\"port\": \"{{mgmt-port}}\",\n\t\t\t\t\t\"path\": [\n\t\t\t\t\t\t\"models\"\n\t\t\t\t\t],\n\t\t\t\t\t\"query\": [\n\t\t\t\t\t\t{\n\t\t\t\t\t\t\t\"key\": \"url\",\n\t\t\t\t\t\t\t\"value\": \"{{url}}\"\n\t\t\t\t\t\t},\n\t\t\t\t\t\t{\n\t\t\t\t\t\t\t\"key\": \"model_name\",\n\t\t\t\t\t\t\t\"value\": \"{{model_name}}\"\n\t\t\t\t\t\t},\n\t\t\t\t\t\t{\n\t\t\t\t\t\t\t\"key\": \"initial_workers\",\n\t\t\t\t\t\t\t\"value\": \"{{worker}}\"\n\t\t\t\t\t\t},\n\t\t\t\t\t\t{\n\t\t\t\t\t\t\t\"key\": \"synchronous\",\n\t\t\t\t\t\t\t\"value\": \"{{synchronous}}\"\n\t\t\t\t\t\t}\n\t\t\t\t\t]\n\t\t\t\t}\n\t\t\t},\n\t\t\t\"response\": []\n\t\t},\n\t\t{\n\t\t\t\"name\": \"Model Zoo - Inference Model\",\n\t\t\t\"event\": [\n\t\t\t\t{\n\t\t\t\t\t\"listen\": \"test\",\n\t\t\t\t\t\"script\": {\n\t\t\t\t\t\t\"id\": \"d6b1f2cf-6ffb-4850-b276-108f7f65fbd9\",\n\t\t\t\t\t\t\"exec\": [\n\t\t\t\t\t\t\t\"var type_response = pm.iterationData.get(\\\"content-type\\\");\",\n\t\t\t\t\t\t\t\"validators = {\",\n\t\t\t\t\t\t\t\" image_classification: validate_image_classification,\",\n\t\t\t\t\t\t\t\" default_json: validate_default\",\n\t\t\t\t\t\t\t\"};\",\n\t\t\t\t\t\t\t\"\",\n\t\t\t\t\t\t\t\"pm.test(\\\"Successful POST request\\\", function() {\",\n\t\t\t\t\t\t\t\" pm.expect(pm.response.code).to.be.oneOf([200]);\",\n\t\t\t\t\t\t\t\"});\",\n\t\t\t\t\t\t\t\"\",\n\t\t\t\t\t\t\t\"if (type_response === \\\"text/plain\\\") {\",\n\t\t\t\t\t\t\t\" pm.test(\\\"Test expected TEXT response\\\", function() {\",\n\t\t\t\t\t\t\t\" pm.response.to.have.body(pm.iterationData.get(\\\"expected\\\"));\",\n\t\t\t\t\t\t\t\" });\",\n\t\t\t\t\t\t\t\"\",\n\t\t\t\t\t\t\t\"} else if (type_response === \\\"application/json\\\") {\",\n\t\t\t\t\t\t\t\" if (pm.iterationData.has(\\\"validator\\\")) {\",\n\t\t\t\t\t\t\t\" var validator = pm.iterationData.get(\\\"validator\\\"); \",\n\t\t\t\t\t\t\t\" } else {\",\n\t\t\t\t\t\t\t\" var validator = \\\"default_json\\\";\",\n\t\t\t\t\t\t\t\" }\",\n\t\t\t\t\t\t\t\" pm.test(\\\"Test expected JSON response\\\", function() {\",\n\t\t\t\t\t\t\t\" var actual_obj = pm.response.json();\",\n\t\t\t\t\t\t\t\" var expected_obj = pm.iterationData.get(\\\"expected\\\");\",\n\t\t\t\t\t\t\t\" pm.expect(validators[validator](actual_obj, expected_obj)).to.be.true;\",\n\t\t\t\t\t\t\t\" });\",\n\t\t\t\t\t\t\t\"\",\n\t\t\t\t\t\t\t\"}\",\n\t\t\t\t\t\t\t\"\",\n\t\t\t\t\t\t\t\"function get_tolerance_value(expected_val) {\",\n\t\t\t\t\t\t\t\" var tolerance_percent = pm.iterationData.get(\\\"tolerance\\\")\",\n\t\t\t\t\t\t\t\" return (expected_val * tolerance_percent) / 100;\",\n\t\t\t\t\t\t\t\" \",\n\t\t\t\t\t\t\t\"}\",\n\t\t\t\t\t\t\t\"\",\n\t\t\t\t\t\t\t\"function validate_image_classification(actual_obj,expected_obj) {\",\n\t\t\t\t\t\t\t\" if (_.size(expected_obj) != _.size(actual_obj)) {\",\n\t\t\t\t\t\t\t\" return false;\",\n\t\t\t\t\t\t\t\" }\",\n\t\t\t\t\t\t\t\"\",\n\t\t\t\t\t\t\t\" for (i = 0; i < expected_obj.length; i += 1) {\",\n\t\t\t\t\t\t\t\" if(actual_obj[i][\\\"class\\\"] !== expected_obj[i][\\\"class\\\"]){\",\n\t\t\t\t\t\t\t\" return false\",\n\t\t\t\t\t\t\t\" }\",\n\t\t\t\t\t\t\t\"\",\n\t\t\t\t\t\t\t\" expected_val = expected_obj[i][\\\"probability\\\"]\",\n\t\t\t\t\t\t\t\" actual_val = actual_obj[i][\\\"probability\\\"]\",\n\t\t\t\t\t\t\t\"\",\n\t\t\t\t\t\t\t\" tolerance_value = get_tolerance_value(expected_val);\",\n\t\t\t\t\t\t\t\" if (!(Math.abs(expected_val - actual_val) < tolerance_value)) {\",\n\t\t\t\t\t\t\t\" return false;\",\n\t\t\t\t\t\t\t\" }\",\n\t\t\t\t\t\t\t\" }\",\n\t\t\t\t\t\t\t\" return true;\",\n\t\t\t\t\t\t\t\"}\",\n\t\t\t\t\t\t\t\"\",\n\t\t\t\t\t\t\t\"\",\n\t\t\t\t\t\t\t\"\",\n\t\t\t\t\t\t\t\"/* Simple and nested json object can be compared using validate_default when key and value are constant.\",\n\t\t\t\t\t\t\t\"-Notes-\",\n\t\t\t\t\t\t\t\"The order of keys within an object may change.\",\n\t\t\t\t\t\t\t\"If the output is array of objects then the objects compared are positional and cannot change order.\",\n\t\t\t\t\t\t\t\"*/\",\n\t\t\t\t\t\t\t\"function validate_default(actual_obj, expected_obj) {\",\n\t\t\t\t\t\t\t\" return _.isEqual(actual_obj, expected_obj);\",\n\t\t\t\t\t\t\t\"}\",\n\t\t\t\t\t\t\t\"\"\n\t\t\t\t\t\t],\n\t\t\t\t\t\t\"type\": \"text/javascript\"\n\t\t\t\t\t}\n\t\t\t\t}\n\t\t\t],\n\t\t\t\"protocolProfileBehavior\": {\n\t\t\t\t\"disabledSystemHeaders\": {\n\t\t\t\t\t\"content-type\": true\n\t\t\t\t}\n\t\t\t},\n\t\t\t\"request\": {\n\t\t\t\t\"method\": \"POST\",\n\t\t\t\t\"header\": [],\n\t\t\t\t\"body\": {\n\t\t\t\t\t\"mode\": \"file\",\n\t\t\t\t\t\"file\": {\n\t\t\t\t\t\t\"src\": \"{{file}}\"\n\t\t\t\t\t}\n\t\t\t\t},\n\t\t\t\t\"url\": {\n\t\t\t\t\t\"raw\": \"{{protocol}}://{{hostname}}:{{pred-port}}/predictions/{{model_name}}\",\n\t\t\t\t\t\"protocol\": \"{{protocol}}\",\n\t\t\t\t\t\"host\": [\n\t\t\t\t\t\t\"{{hostname}}\"\n\t\t\t\t\t],\n\t\t\t\t\t\"port\": \"{{pred-port}}\",\n\t\t\t\t\t\"path\": [\n\t\t\t\t\t\t\"predictions\",\n\t\t\t\t\t\t\"{{model_name}}\"\n\t\t\t\t\t]\n\t\t\t\t}\n\t\t\t},\n\t\t\t\"response\": []\n\t\t},\n\t\t{\n\t\t\t\"name\": \"Model Zoo - Unregister model\",\n\t\t\t\"event\": [\n\t\t\t\t{\n\t\t\t\t\t\"listen\": \"test\",\n\t\t\t\t\t\"script\": {\n\t\t\t\t\t\t\"id\": \"a14dd390-4176-45e7-af00-999676685f4a\",\n\t\t\t\t\t\t\"exec\": [\n\t\t\t\t\t\t\t\"pm.test(\\\"Successful DELETE request\\\", function () {\",\n\t\t\t\t\t\t\t\" pm.expect(pm.response.code).to.be.oneOf([200,201,202]);\",\n\t\t\t\t\t\t\t\"});\"\n\t\t\t\t\t\t],\n\t\t\t\t\t\t\"type\": \"text/javascript\"\n\t\t\t\t\t}\n\t\t\t\t}\n\t\t\t],\n\t\t\t\"request\": {\n\t\t\t\t\"method\": \"DELETE\",\n\t\t\t\t\"header\": [],\n\t\t\t\t\"url\": {\n\t\t\t\t\t\"raw\": \"{{protocol}}://{{hostname}}:{{mgmt-port}}/models/{{model_name}}\",\n\t\t\t\t\t\"protocol\": \"{{protocol}}\",\n\t\t\t\t\t\"host\": [\n\t\t\t\t\t\t\"{{hostname}}\"\n\t\t\t\t\t],\n\t\t\t\t\t\"port\": \"{{mgmt-port}}\",\n\t\t\t\t\t\"path\": [\n\t\t\t\t\t\t\"models\",\n\t\t\t\t\t\t\"{{model_name}}\"\n\t\t\t\t\t]\n\t\t\t\t}\n\t\t\t},\n\t\t\t\"response\": []\n\t\t}\n\t],\n\t\"protocolProfileBehavior\": {}\n}" }, { "path": "test/postman/inference_data.json", "content": "[\n {\n \"url\":\"https://s3.amazonaws.com/model-server/model_archive_1.0/alexnet.mar\",\n \"model_name\":\"alexnet\",\n \"worker\":1,\n \"synchronous\":\"true\",\n \"file\":\"test/resources/kitten.jpg\",\n \"content-type\":\"application/json\",\n \"validator\":\"image_classification\",\n \"expected\":[\n {\n \"probability\":0.4766186773777008,\n \"class\":\"n02127052 lynx, catamount\"\n },\n {\n \"probability\":0.20687074959278107,\n \"class\":\"n02128757 snow leopard, ounce, Panthera uncia\"\n },\n {\n \"probability\":0.135288268327713,\n \"class\":\"n02124075 Egyptian cat\"\n },\n {\n \"probability\":0.09019536525011063,\n \"class\":\"n02123045 tabby, tabby cat\"\n },\n {\n \"probability\":0.04814659804105759,\n \"class\":\"n02123159 tiger cat\"\n }\n ],\n \"tolerance\":1\n },\n {\n \"url\":\"https://s3.amazonaws.com/model-server/model_archive_1.0/caffenet.mar\",\n \"model_name\":\"caffenet\",\n \"worker\":1,\n \"synchronous\":\"true\",\n \"file\":\"test/resources/kitten.jpg\",\n \"content-type\":\"application/json\",\n \"validator\":\"image_classification\",\n \"expected\":[\n {\n \"probability\":0.7166884541511536,\n \"class\":\"n02127052 lynx, catamount\"\n },\n {\n \"probability\":0.09750154614448547,\n \"class\":\"n02123045 tabby, tabby cat\"\n },\n {\n \"probability\":0.0745730996131897,\n \"class\":\"n02123159 tiger cat\"\n },\n {\n \"probability\":0.06743090599775314,\n \"class\":\"n02124075 Egyptian cat\"\n },\n {\n \"probability\":0.03200334310531616,\n \"class\":\"n02128757 snow leopard, ounce, Panthera uncia\"\n }\n ],\n \"tolerance\":1\n },\n {\n \"url\":\"https://s3.amazonaws.com/model-server/model_archive_1.0/onnx-inception_v1.mar\",\n \"model_name\":\"inception_v1\",\n \"worker\":1,\n \"synchronous\":\"true\",\n \"file\":\"test/resources/kitten.jpg\",\n \"content-type\":\"application/json\",\n \"validator\":\"image_classification\",\n \"expected\":[\n {\n \"probability\":0.3833286464214325,\n \"class\":\"n02123159 tiger cat\"\n },\n {\n \"probability\":0.33825597167015076,\n \"class\":\"n02124075 Egyptian cat\"\n },\n {\n \"probability\":0.17002010345458984,\n \"class\":\"n02123045 tabby, tabby cat\"\n },\n {\n \"probability\":0.09881989657878876,\n \"class\":\"n02127052 lynx, catamount\"\n },\n {\n \"probability\":0.001223195344209671,\n \"class\":\"n02123394 Persian cat\"\n }\n ],\n \"tolerance\":1\n },\n {\n \"url\":\"https://s3.amazonaws.com/model-server/model_archive_1.0/inception-bn.mar\",\n \"model_name\":\"inception-bn\",\n \"worker\":1,\n \"synchronous\":\"true\",\n \"file\":\"test/resources/kitten.jpg\",\n \"content-type\":\"application/json\",\n \"validator\":\"image_classification\",\n \"expected\":[\n {\n \"probability\":0.5972931981086731,\n \"class\":\"n02123045 tabby, tabby cat\"\n },\n {\n \"probability\":0.30086880922317505,\n \"class\":\"n02123159 tiger cat\"\n },\n {\n \"probability\":0.08586657792329788,\n \"class\":\"n02124075 Egyptian cat\"\n },\n {\n \"probability\":0.009726772084832191,\n \"class\":\"n02127052 lynx, catamount\"\n },\n {\n \"probability\":0.0008877559448592365,\n \"class\":\"n03598930 jigsaw puzzle\"\n }\n ],\n \"tolerance\":1\n },\n {\n \"url\":\"https://s3.amazonaws.com/model-server/model_archive_1.0/onnx-mobilenet.mar\",\n \"model_name\":\"mobilenet\",\n \"worker\":1,\n \"synchronous\":\"true\",\n \"file\":\"test/resources/kitten.jpg\",\n \"content-type\":\"application/json\",\n \"validator\":\"image_classification\",\n \"expected\":[\n {\n \"probability\":216.0907745361328,\n \"class\":\"n02948072 candle, taper, wax light\"\n },\n {\n \"probability\":186.8531494140625,\n \"class\":\"n04456115 torch\"\n },\n {\n \"probability\":178.81483459472656,\n \"class\":\"n03347037 fire screen, fireguard\"\n },\n {\n \"probability\":169.5437469482422,\n \"class\":\"n03666591 lighter, light, igniter, ignitor\"\n },\n {\n \"probability\":145.08824157714844,\n \"class\":\"n09472597 volcano\"\n }\n ],\n \"tolerance\":1\n },\n {\n \"url\":\"https://s3.amazonaws.com/model-server/model_archive_1.0/nin.mar\",\n \"model_name\":\"nin\",\n \"worker\":1,\n \"synchronous\":\"true\",\n \"file\":\"test/resources/kitten.jpg\",\n \"content-type\":\"application/json\",\n \"validator\":\"image_classification\",\n \"expected\":[\n {\n \"probability\":0.978486180305481,\n \"class\":\"n02123045 tabby, tabby cat\"\n },\n {\n \"probability\":0.021282507106661797,\n \"class\":\"n02123159 tiger cat\"\n },\n {\n \"probability\":9.809057519305497e-05,\n \"class\":\"n02124075 Egyptian cat\"\n },\n {\n \"probability\":8.281067130155861e-05,\n \"class\":\"n02128385 leopard, Panthera pardus\"\n },\n {\n \"probability\":2.5670484319562092e-05,\n \"class\":\"n02128757 snow leopard, ounce, Panthera uncia\"\n }\n ],\n \"tolerance\":1\n },\n {\n \"url\":\"https://s3.amazonaws.com/model-server/model_archive_1.0/resnet-152.mar\",\n \"model_name\":\"resnet-152\",\n \"worker\":1,\n \"synchronous\":\"true\",\n \"file\":\"test/resources/kitten.jpg\",\n \"content-type\":\"application/json\",\n \"validator\":\"image_classification\",\n \"expected\":[\n {\n \"probability\":0.7149006128311157,\n \"class\":\"n02123045 tabby, tabby cat\"\n },\n {\n \"probability\":0.22877003252506256,\n \"class\":\"n02123159 tiger cat\"\n },\n {\n \"probability\":0.040323738008737564,\n \"class\":\"n02124075 Egyptian cat\"\n },\n {\n \"probability\":0.008370742201805115,\n \"class\":\"n02127052 lynx, catamount\"\n },\n {\n \"probability\":0.00067278987262398,\n \"class\":\"n02129604 tiger, Panthera tigris\"\n }\n ],\n \"tolerance\":1\n },\n {\n \"url\":\"https://s3.amazonaws.com/model-server/model_archive_1.0/resnet-18.mar\",\n \"model_name\":\"resnet-18\",\n \"worker\":1,\n \"synchronous\":\"true\",\n \"file\":\"test/resources/kitten.jpg\",\n \"content-type\":\"application/json\",\n \"validator\":\"image_classification\",\n \"expected\":[\n {\n \"probability\":0.3633013665676117,\n \"class\":\"n02123159 tiger cat\"\n },\n {\n \"probability\":0.2988913953304291,\n \"class\":\"n02124075 Egyptian cat\"\n },\n {\n \"probability\":0.18073132634162903,\n \"class\":\"n02123045 tabby, tabby cat\"\n },\n {\n \"probability\":0.07343611121177673,\n \"class\":\"n02127052 lynx, catamount\"\n },\n {\n \"probability\":0.02035967819392681,\n \"class\":\"n02128757 snow leopard, ounce, Panthera uncia\"\n }\n ],\n \"tolerance\":1\n },\n {\n \"url\":\"https://s3.amazonaws.com/model-server/model_archive_1.0/resnext-101-64x4d.mar\",\n \"model_name\":\"resnext101\",\n \"worker\":1,\n \"synchronous\":\"true\",\n \"file\":\"test/resources/kitten.jpg\",\n \"content-type\":\"application/json\",\n \"validator\":\"image_classification\",\n \"expected\":[\n {\n \"probability\":0.5961099863052368,\n \"class\":\"n02123159 tiger cat\"\n },\n {\n \"probability\":0.4005117118358612,\n \"class\":\"n02123045 tabby, tabby cat\"\n },\n {\n \"probability\":0.0012956340797245502,\n \"class\":\"n02124075 Egyptian cat\"\n },\n {\n \"probability\":0.0011538631515577435,\n \"class\":\"n04074963 remote control, remote\"\n },\n {\n \"probability\":0.000405249185860157,\n \"class\":\"n04286575 spotlight, spot\"\n }\n ],\n \"tolerance\":1\n },\n {\n \"url\":\"https://s3.amazonaws.com/model-server/model_archive_1.0/onnx-resnet18v1.mar\",\n \"model_name\":\"resnet18-v1\",\n \"worker\":1,\n \"synchronous\":\"true\",\n \"file\":\"test/resources/kitten.jpg\",\n \"content-type\":\"application/json\",\n \"validator\":\"image_classification\",\n \"expected\":[\n {\n \"probability\":16.35452651977539,\n \"class\":\"n02999410 chain\"\n },\n {\n \"probability\":16.15776824951172,\n \"class\":\"n10148035 groom, bridegroom\"\n },\n {\n \"probability\":15.857562065124512,\n \"class\":\"n04141076 sax, saxophone\"\n },\n {\n \"probability\":15.083999633789062,\n \"class\":\"n04507155 umbrella\"\n },\n {\n \"probability\":14.938633918762207,\n \"class\":\"n09229709 bubble\"\n }\n ],\n \"tolerance\":1\n },\n {\n \"url\":\"https://s3.amazonaws.com/model-server/model_archive_1.0/onnx-resnet34v1.mar\",\n \"model_name\":\"resnet34-v1\",\n \"worker\":1,\n \"synchronous\":\"true\",\n \"file\":\"test/resources/kitten.jpg\",\n \"content-type\":\"application/json\",\n \"validator\":\"image_classification\",\n \"expected\":[\n {\n \"probability\":12.617600440979004,\n \"class\":\"n04286575 spotlight, spot\"\n },\n {\n \"probability\":10.216148376464844,\n \"class\":\"n03637318 lampshade, lamp shade\"\n },\n {\n \"probability\":9.676478385925293,\n \"class\":\"n03942813 ping-pong ball\"\n },\n {\n \"probability\":9.529446601867676,\n \"class\":\"n02708093 analog clock\"\n },\n {\n \"probability\":9.494606018066406,\n \"class\":\"n03691459 loudspeaker, speaker, speaker unit, loudspeaker system, speaker system\"\n }\n ],\n \"tolerance\":1\n },\n {\n \"url\":\"https://s3.amazonaws.com/model-server/model_archive_1.0/onnx-resnet50v1.mar\",\n \"model_name\":\"resnet50-v1\",\n \"worker\":1,\n \"synchronous\":\"true\",\n \"file\":\"test/resources/kitten.jpg\",\n \"content-type\":\"application/json\",\n \"validator\":\"image_classification\",\n \"expected\":[\n {\n \"probability\":177.36087036132812,\n \"class\":\"n03982430 pool table, billiard table, snooker table\"\n },\n {\n \"probability\":174.36256408691406,\n \"class\":\"n03942813 ping-pong ball\"\n },\n {\n \"probability\":172.44488525390625,\n \"class\":\"n03661043 library\"\n },\n {\n \"probability\":163.6439971923828,\n \"class\":\"n02788148 bannister, banister, balustrade, balusters, handrail\"\n },\n {\n \"probability\":159.4976043701172,\n \"class\":\"n03065424 coil, spiral, volute, whorl, helix\"\n }\n ],\n \"tolerance\":1\n },\n {\n \"url\":\"https://s3.amazonaws.com/model-server/model_archive_1.0/onnx-resnet101v1.mar\",\n \"model_name\":\"resnet101-v1\",\n \"worker\":1,\n \"synchronous\":\"true\",\n \"file\":\"test/resources/kitten.jpg\",\n \"content-type\":\"application/json\",\n \"validator\":\"image_classification\",\n \"expected\":[\n {\n \"probability\":107.36915588378906,\n \"class\":\"n02823428 beer bottle\"\n },\n {\n \"probability\":99.51375579833984,\n \"class\":\"n04485082 tripod\"\n },\n {\n \"probability\":95.99050903320312,\n \"class\":\"n04069434 reflex camera\"\n },\n {\n \"probability\":84.3740463256836,\n \"class\":\"n04557648 water bottle\"\n },\n {\n \"probability\":83.7496566772461,\n \"class\":\"n02841315 binoculars, field glasses, opera glasses\"\n }\n ],\n \"tolerance\":1\n },\n {\n \"url\":\"https://s3.amazonaws.com/model-server/model_archive_1.0/onnx-resnet152v1.mar\",\n \"model_name\":\"resnet152-v1\",\n \"worker\":1,\n \"synchronous\":\"true\",\n \"file\":\"test/resources/kitten.jpg\",\n \"content-type\":\"application/json\",\n \"validator\":\"image_classification\",\n \"expected\":[\n {\n \"probability\":61.71363067626953,\n \"class\":\"n07930864 cup\"\n },\n {\n \"probability\":60.88921356201172,\n \"class\":\"n03832673 notebook, notebook computer\"\n },\n {\n \"probability\":60.11431121826172,\n \"class\":\"n03691459 loudspeaker, speaker, speaker unit, loudspeaker system, speaker system\"\n },\n {\n \"probability\":53.656005859375,\n \"class\":\"n07920052 espresso\"\n },\n {\n \"probability\":53.115779876708984,\n \"class\":\"n03492542 hard disc, hard disk, fixed disk\"\n }\n ],\n \"tolerance\":1\n },\n {\n \"url\":\"https://s3.amazonaws.com/model-server/model_archive_1.0/onnx-resnet18v2.mar\",\n \"model_name\":\"resnet18-v2\",\n \"worker\":1,\n \"synchronous\":\"true\",\n \"file\":\"test/resources/kitten.jpg\",\n \"content-type\":\"application/json\",\n \"validator\":\"image_classification\",\n \"expected\":[\n {\n \"probability\":24.102333068847656,\n \"class\":\"n04553703 washbasin, handbasin, washbowl, lavabo, wash-hand basin\"\n },\n {\n \"probability\":23.69866943359375,\n \"class\":\"n04254120 soap dispenser\"\n },\n {\n \"probability\":19.214643478393555,\n \"class\":\"n02747177 ashcan, trash can, garbage can, wastebin, ash bin, ash-bin, ashbin, dustbin, trash barrel, trash bin\"\n },\n {\n \"probability\":18.93875503540039,\n \"class\":\"n03691459 loudspeaker, speaker, speaker unit, loudspeaker system, speaker system\"\n },\n {\n \"probability\":18.90488052368164,\n \"class\":\"n04201297 shoji\"\n }\n ],\n \"tolerance\":1\n },\n {\n \"url\":\"https://s3.amazonaws.com/model-server/model_archive_1.0/onnx-resnet34v2.mar\",\n \"model_name\":\"resnet34-v2\",\n \"worker\":1,\n \"synchronous\":\"true\",\n \"file\":\"test/resources/kitten.jpg\",\n \"content-type\":\"application/json\",\n \"validator\":\"image_classification\",\n \"expected\":[\n {\n \"probability\":9.2193603515625,\n \"class\":\"n02708093 analog clock\"\n },\n {\n \"probability\":7.803028583526611,\n \"class\":\"n03782006 monitor\"\n },\n {\n \"probability\":7.681037425994873,\n \"class\":\"n04286575 spotlight, spot\"\n },\n {\n \"probability\":7.129834175109863,\n \"class\":\"n03028079 church, church building\"\n },\n {\n \"probability\":7.0597100257873535,\n \"class\":\"n04152593 screen, CRT screen\"\n }\n ],\n \"tolerance\":1\n },\n {\n \"url\":\"https://s3.amazonaws.com/model-server/model_archive_1.0/onnx-resnet50v2.mar\",\n \"model_name\":\"resnet50-v2\",\n \"worker\":1,\n \"synchronous\":\"true\",\n \"file\":\"test/resources/kitten.jpg\",\n \"content-type\":\"application/json\",\n \"validator\":\"image_classification\",\n \"expected\":[\n {\n \"probability\":15.76949691772461,\n \"class\":\"n06359193 web site, website, internet site, site\"\n },\n {\n \"probability\":14.25102710723877,\n \"class\":\"n07565083 menu\"\n },\n {\n \"probability\":13.321331024169922,\n \"class\":\"n03944341 pinwheel\"\n },\n {\n \"probability\":11.99173641204834,\n \"class\":\"n06596364 comic book\"\n },\n {\n \"probability\":11.768353462219238,\n \"class\":\"n03291819 envelope\"\n }\n ],\n \"tolerance\":1\n },\n {\n \"url\":\"https://s3.amazonaws.com/model-server/model_archive_1.0/onnx-resnet101v2.mar\",\n \"model_name\":\"resnet101-v2\",\n \"worker\":1,\n \"synchronous\":\"true\",\n \"file\":\"test/resources/kitten.jpg\",\n \"content-type\":\"application/json\",\n \"validator\":\"image_classification\",\n \"expected\":[\n {\n \"probability\":7.887596130371094,\n \"class\":\"n04380533 table lamp\"\n },\n {\n \"probability\":7.870771884918213,\n \"class\":\"n03627232 knot\"\n },\n {\n \"probability\":7.605418682098389,\n \"class\":\"n02093859 Kerry blue terrier\"\n },\n {\n \"probability\":7.55618143081665,\n \"class\":\"n04033995 quilt, comforter, comfort, puff\"\n },\n {\n \"probability\":7.256267547607422,\n \"class\":\"n03884397 panpipe, pandean pipe, syrinx\"\n }\n ],\n \"tolerance\":1\n },\n {\n \"url\":\"https://s3.amazonaws.com/model-server/model_archive_1.0/onnx-resnet152v2.mar\",\n \"model_name\":\"resnet152-v2\",\n \"worker\":1,\n \"synchronous\":\"true\",\n \"file\":\"test/resources/kitten.jpg\",\n \"content-type\":\"application/json\",\n \"validator\":\"image_classification\",\n \"expected\":[\n {\n \"probability\":36.13174819946289,\n \"class\":\"n03970156 plunger, plumber's helper\"\n },\n {\n \"probability\":29.110092163085938,\n \"class\":\"n02708093 analog clock\"\n },\n {\n \"probability\":28.711875915527344,\n \"class\":\"n04152593 screen, CRT screen\"\n },\n {\n \"probability\":28.073928833007812,\n \"class\":\"n01930112 nematode, nematode worm, roundworm\"\n },\n {\n \"probability\":28.058176040649414,\n \"class\":\"n04404412 television, television system\"\n }\n ],\n \"tolerance\":1\n },\n {\n \"url\":\"https://s3.amazonaws.com/model-server/model_archive_1.0/shufflenet.mar\",\n \"model_name\":\"shufflenet\",\n \"worker\":1,\n \"synchronous\":\"true\",\n \"file\":\"test/resources/kitten.jpg\",\n \"content-type\":\"application/json\",\n \"validator\":\"image_classification\",\n \"expected\":[\n {\n \"probability\":0.0010000000474974513,\n \"class\":\"n03792972 mountain tent\"\n },\n {\n \"probability\":0.0010000000474974513,\n \"class\":\"n03773504 missile\"\n },\n {\n \"probability\":0.0010000000474974513,\n \"class\":\"n03775071 mitten\"\n },\n {\n \"probability\":0.0010000000474974513,\n \"class\":\"n03775546 mixing bowl\"\n },\n {\n \"probability\":0.0010000000474974513,\n \"class\":\"n03776460 mobile home, manufactured home\"\n }\n ],\n \"tolerance\":1\n },\n {\n \"url\":\"https://s3.amazonaws.com/model-server/model_archive_1.0/onnx-squeezenet.mar\",\n \"model_name\":\"onnx-squeezenet\",\n \"worker\":1,\n \"synchronous\":\"true\",\n \"file\":\"test/resources/kitten.jpg\",\n \"content-type\":\"application/json\",\n \"validator\":\"image_classification\",\n \"expected\":[\n {\n \"probability\":0.6482629179954529,\n \"class\":\"n02124075 Egyptian cat\"\n },\n {\n \"probability\":0.2318260818719864,\n \"class\":\"n02123045 tabby, tabby cat\"\n },\n {\n \"probability\":0.10045160353183746,\n \"class\":\"n02123159 tiger cat\"\n },\n {\n \"probability\":0.013487360440194607,\n \"class\":\"n02127052 lynx, catamount\"\n },\n {\n \"probability\":0.003135664388537407,\n \"class\":\"n02128757 snow leopard, ounce, Panthera uncia\"\n }\n ],\n \"tolerance\":1\n },\n {\n \"url\":\"https://s3.amazonaws.com/model-server/model_archive_1.0/squeezenet_v1.1.mar\",\n \"model_name\":\"squeezenet_v1.1\",\n \"worker\":1,\n \"synchronous\":\"true\",\n \"file\":\"test/resources/kitten.jpg\",\n \"content-type\":\"application/json\",\n \"validator\":\"image_classification\",\n \"expected\":[\n {\n \"probability\":0.8582231402397156,\n \"class\":\"n02124075 Egyptian cat\"\n },\n {\n \"probability\":0.09159984439611435,\n \"class\":\"n02123045 tabby, tabby cat\"\n },\n {\n \"probability\":0.03748767822980881,\n \"class\":\"n02123159 tiger cat\"\n },\n {\n \"probability\":0.006165081635117531,\n \"class\":\"n02128385 leopard, Panthera pardus\"\n },\n {\n \"probability\":0.0031715999357402325,\n \"class\":\"n02127052 lynx, catamount\"\n }\n ],\n \"tolerance\":1\n },\n {\n \"url\":\"https://s3.amazonaws.com/model-server/model_archive_1.0/vgg16.mar\",\n \"model_name\":\"vgg16\",\n \"worker\":1,\n \"synchronous\":\"true\",\n \"file\":\"test/resources/kitten.jpg\",\n \"content-type\":\"application/json\",\n \"validator\":\"image_classification\",\n \"expected\":[\n {\n \"probability\":0.8178463578224182,\n \"class\":\"n02123159 tiger cat\"\n },\n {\n \"probability\":0.12500879168510437,\n \"class\":\"n02123045 tabby, tabby cat\"\n },\n {\n \"probability\":0.05412120372056961,\n \"class\":\"n02124075 Egyptian cat\"\n },\n {\n \"probability\":0.0020657714921981096,\n \"class\":\"n02127052 lynx, catamount\"\n },\n {\n \"probability\":0.0005614628316834569,\n \"class\":\"n02128757 snow leopard, ounce, Panthera uncia\"\n }\n ],\n \"tolerance\":1\n },\n {\n \"url\":\"https://s3.amazonaws.com/model-server/model_archive_1.0/onnx-vgg16.mar\",\n \"model_name\":\"onnx-vgg16\",\n \"worker\":1,\n \"synchronous\":\"true\",\n \"file\":\"test/resources/kitten.jpg\",\n \"content-type\":\"application/json\",\n \"validator\":\"image_classification\",\n \"expected\":[\n {\n \"probability\":101.13871765136719,\n \"class\":\"n02124075 Egyptian cat\"\n },\n {\n \"probability\":89.77296447753906,\n \"class\":\"n02123045 tabby, tabby cat\"\n },\n {\n \"probability\":88.40411376953125,\n \"class\":\"n02123159 tiger cat\"\n },\n {\n \"probability\":76.17413330078125,\n \"class\":\"n02125311 cougar, puma, catamount, mountain lion, painter, panther, Felis concolor\"\n },\n {\n \"probability\":74.09810638427734,\n \"class\":\"n07930864 cup\"\n }\n ],\n \"tolerance\":1\n },\n {\n \"url\":\"https://s3.amazonaws.com/model-server/model_archive_1.0/onnx-vgg16_bn.mar\",\n \"model_name\":\"onnx-vgg16_bn\",\n \"worker\":1,\n \"synchronous\":\"true\",\n \"file\":\"test/resources/kitten.jpg\",\n \"content-type\":\"application/json\",\n \"validator\":\"image_classification\",\n \"expected\":[\n {\n \"probability\":16.06477165222168,\n \"class\":\"n03196217 digital clock\"\n },\n {\n \"probability\":13.64653491973877,\n \"class\":\"n04286575 spotlight, spot\"\n },\n {\n \"probability\":13.565534591674805,\n \"class\":\"n03692522 loupe, jeweler's loupe\"\n },\n {\n \"probability\":13.479013442993164,\n \"class\":\"n03388043 fountain\"\n },\n {\n \"probability\":12.639715194702148,\n \"class\":\"n03666591 lighter, light, igniter, ignitor\"\n }\n ],\n \"tolerance\":1\n },\n {\n \"url\":\"https://s3.amazonaws.com/model-server/model_archive_1.0/vgg19.mar\",\n \"model_name\":\"vgg19\",\n \"worker\":1,\n \"synchronous\":\"true\",\n \"file\":\"test/resources/kitten.jpg\",\n \"content-type\":\"application/json\",\n \"validator\":\"image_classification\",\n \"expected\":[\n {\n \"probability\":0.5058671832084656,\n \"class\":\"n02123159 tiger cat\"\n },\n {\n \"probability\":0.28164851665496826,\n \"class\":\"n02124075 Egyptian cat\"\n },\n {\n \"probability\":0.20637290179729462,\n \"class\":\"n02123045 tabby, tabby cat\"\n },\n {\n \"probability\":0.0046674045734107494,\n \"class\":\"n02127052 lynx, catamount\"\n },\n {\n \"probability\":0.00011857607023557648,\n \"class\":\"n03598930 jigsaw puzzle\"\n }\n ],\n \"tolerance\":1\n },\n {\n \"url\":\"https://s3.amazonaws.com/model-server/model_archive_1.0/onnx-vgg19.mar\",\n \"model_name\":\"onnx-vgg19\",\n \"worker\":1,\n \"synchronous\":\"true\",\n \"file\":\"test/resources/kitten.jpg\",\n \"content-type\":\"application/json\",\n \"validator\":\"image_classification\",\n \"expected\":[\n {\n \"probability\":0.3949933350086212,\n \"class\":\"n02124075 Egyptian cat\"\n },\n {\n \"probability\":0.34417903423309326,\n \"class\":\"n02123159 tiger cat\"\n },\n {\n \"probability\":0.25566166639328003,\n \"class\":\"n02123045 tabby, tabby cat\"\n },\n {\n \"probability\":0.0027325130067765713,\n \"class\":\"n02127052 lynx, catamount\"\n },\n {\n \"probability\":0.0002417804644210264,\n \"class\":\"n07930864 cup\"\n }\n ],\n \"tolerance\":1\n },\n {\n \"url\":\"https://s3.amazonaws.com/model-server/model_archive_1.0/onnx-vgg19_bn.mar\",\n \"model_name\":\"onnx-vgg19_bn\",\n \"worker\":1,\n \"synchronous\":\"true\",\n \"file\":\"test/resources/kitten.jpg\",\n \"content-type\":\"application/json\",\n \"validator\":\"image_classification\",\n \"expected\":[\n {\n \"probability\":17.26873016357422,\n \"class\":\"n04589890 window screen\"\n },\n {\n \"probability\":16.25399398803711,\n \"class\":\"n03347037 fire screen, fireguard\"\n },\n {\n \"probability\":16.093460083007812,\n \"class\":\"n04286575 spotlight, spot\"\n },\n {\n \"probability\":16.02733039855957,\n \"class\":\"n04590129 window shade\"\n },\n {\n \"probability\":15.910074234008789,\n \"class\":\"n03637318 lampshade, lamp shade\"\n }\n ],\n \"tolerance\":1\n }\n]" }, { "path": "test/postman/management_api_test_collection.json", "content": "{\n\t\"info\": {\n\t\t\"_postman_id\": \"5c45fe29-4bb1-4df4-9e13-a4fd410e4775\",\n\t\t\"name\": \"MMS - management_api_collection\",\n\t\t\"schema\": \"https://schema.getpostman.com/json/collection/v2.1.0/collection.json\"\n\t},\n\t\"item\": [\n\t\t{\n\t\t\t\"name\": \"Register Model\",\n\t\t\t\"event\": [\n\t\t\t\t{\n\t\t\t\t\t\"listen\": \"test\",\n\t\t\t\t\t\"script\": {\n\t\t\t\t\t\t\"id\": \"4f706340-af84-431b-affd-6d6f53c89e80\",\n\t\t\t\t\t\t\"exec\": [\n\t\t\t\t\t\t\t\"pm.test(\\\"Successful POST request\\\", function () {\",\n\t\t\t\t\t\t\t\" pm.expect(pm.response.code).to.be.oneOf([200]);\",\n\t\t\t\t\t\t\t\"});\"\n\t\t\t\t\t\t],\n\t\t\t\t\t\t\"type\": \"text/javascript\"\n\t\t\t\t\t}\n\t\t\t\t}\n\t\t\t],\n\t\t\t\"request\": {\n\t\t\t\t\"method\": \"POST\",\n\t\t\t\t\"header\": [],\n\t\t\t\t\"url\": {\n\t\t\t\t\t\"raw\": \"{{protocol}}://{{hostname}}:{{mgmt-port}}/models?url=https://s3.amazonaws.com/model-server/model_archive_1.0/squeezenet_v1.1.mar&model_name=squeezenet1_1\",\n\t\t\t\t\t\"protocol\": \"{{protocol}}\",\n\t\t\t\t\t\"host\": [\n\t\t\t\t\t\t\"{{hostname}}\"\n\t\t\t\t\t],\n\t\t\t\t\t\"port\": \"{{mgmt-port}}\",\n\t\t\t\t\t\"path\": [\n\t\t\t\t\t\t\"models\"\n\t\t\t\t\t],\n\t\t\t\t\t\"query\": [\n\t\t\t\t\t\t{\n\t\t\t\t\t\t\t\"key\": \"url\",\n\t\t\t\t\t\t\t\"value\": \"https://s3.amazonaws.com/model-server/model_archive_1.0/squeezenet_v1.1.mar\"\n\t\t\t\t\t\t},\n\t\t\t\t\t\t{\n\t\t\t\t\t\t\t\"key\": \"model_name\",\n\t\t\t\t\t\t\t\"value\": \"squeezenet1_1\"\n\t\t\t\t\t\t}\n\t\t\t\t\t]\n\t\t\t\t}\n\t\t\t},\n\t\t\t\"response\": []\n\t\t},\n\t\t{\n\t\t\t\"name\": \"Get Valid Model\",\n\t\t\t\"event\": [\n\t\t\t\t{\n\t\t\t\t\t\"listen\": \"test\",\n\t\t\t\t\t\"script\": {\n\t\t\t\t\t\t\"id\": \"3d6c7158-f729-42cf-a2ec-4e62b7d764d0\",\n\t\t\t\t\t\t\"exec\": [\n\t\t\t\t\t\t\t\"pm.test(\\\"Successful POST request\\\", function () {\",\n\t\t\t\t\t\t\t\" pm.expect(pm.response.code).to.be.oneOf([200]);\",\n\t\t\t\t\t\t\t\"});\"\n\t\t\t\t\t\t],\n\t\t\t\t\t\t\"type\": \"text/javascript\"\n\t\t\t\t\t}\n\t\t\t\t}\n\t\t\t],\n\t\t\t\"request\": {\n\t\t\t\t\"method\": \"GET\",\n\t\t\t\t\"header\": [],\n\t\t\t\t\"url\": {\n\t\t\t\t\t\"raw\": \"{{protocol}}://{{hostname}}:{{mgmt-port}}/models/squeezenet1_1\",\n\t\t\t\t\t\"protocol\": \"{{protocol}}\",\n\t\t\t\t\t\"host\": [\n\t\t\t\t\t\t\"{{hostname}}\"\n\t\t\t\t\t],\n\t\t\t\t\t\"port\": \"{{mgmt-port}}\",\n\t\t\t\t\t\"path\": [\n\t\t\t\t\t\t\"models\",\n\t\t\t\t\t\t\"squeezenet1_1\"\n\t\t\t\t\t]\n\t\t\t\t}\n\t\t\t},\n\t\t\t\"response\": []\n\t\t},\n\t\t{\n\t\t\t\"name\": \"List Models\",\n\t\t\t\"event\": [\n\t\t\t\t{\n\t\t\t\t\t\"listen\": \"test\",\n\t\t\t\t\t\"script\": {\n\t\t\t\t\t\t\"id\": \"54f1209e-4819-4f06-9d26-f39593a889d9\",\n\t\t\t\t\t\t\"exec\": [\n\t\t\t\t\t\t\t\"pm.test(\\\"Successful POST request\\\", function () {\",\n\t\t\t\t\t\t\t\" pm.expect(pm.response.code).to.be.oneOf([200]);\",\n\t\t\t\t\t\t\t\"});\"\n\t\t\t\t\t\t],\n\t\t\t\t\t\t\"type\": \"text/javascript\"\n\t\t\t\t\t}\n\t\t\t\t}\n\t\t\t],\n\t\t\t\"request\": {\n\t\t\t\t\"method\": \"GET\",\n\t\t\t\t\"header\": [],\n\t\t\t\t\"url\": {\n\t\t\t\t\t\"raw\": \"{{protocol}}://{{hostname}}:{{mgmt-port}}/models\",\n\t\t\t\t\t\"protocol\": \"{{protocol}}\",\n\t\t\t\t\t\"host\": [\n\t\t\t\t\t\t\"{{hostname}}\"\n\t\t\t\t\t],\n\t\t\t\t\t\"port\": \"{{mgmt-port}}\",\n\t\t\t\t\t\"path\": [\n\t\t\t\t\t\t\"models\"\n\t\t\t\t\t]\n\t\t\t\t}\n\t\t\t},\n\t\t\t\"response\": []\n\t\t},\n\t\t{\n\t\t\t\"name\": \"Scale Min Workers - Asynchronous\",\n\t\t\t\"event\": [\n\t\t\t\t{\n\t\t\t\t\t\"listen\": \"test\",\n\t\t\t\t\t\"script\": {\n\t\t\t\t\t\t\"id\": \"47ac7b07-04c2-4128-9662-387e26446f7f\",\n\t\t\t\t\t\t\"exec\": [\n\t\t\t\t\t\t\t\"pm.test(\\\"Successful PUT request\\\", function () {\",\n\t\t\t\t\t\t\t\" pm.expect(pm.response.code).to.be.oneOf([200,201,202]);\",\n\t\t\t\t\t\t\t\"});\"\n\t\t\t\t\t\t],\n\t\t\t\t\t\t\"type\": \"text/javascript\"\n\t\t\t\t\t}\n\t\t\t\t}\n\t\t\t],\n\t\t\t\"request\": {\n\t\t\t\t\"method\": \"PUT\",\n\t\t\t\t\"header\": [],\n\t\t\t\t\"url\": {\n\t\t\t\t\t\"raw\": \"{{protocol}}://{{hostname}}:{{mgmt-port}}/models/squeezenet1_1?min_worker=3\",\n\t\t\t\t\t\"protocol\": \"{{protocol}}\",\n\t\t\t\t\t\"host\": [\n\t\t\t\t\t\t\"{{hostname}}\"\n\t\t\t\t\t],\n\t\t\t\t\t\"port\": \"{{mgmt-port}}\",\n\t\t\t\t\t\"path\": [\n\t\t\t\t\t\t\"models\",\n\t\t\t\t\t\t\"squeezenet1_1\"\n\t\t\t\t\t],\n\t\t\t\t\t\"query\": [\n\t\t\t\t\t\t{\n\t\t\t\t\t\t\t\"key\": \"min_worker\",\n\t\t\t\t\t\t\t\"value\": \"3\"\n\t\t\t\t\t\t}\n\t\t\t\t\t]\n\t\t\t\t}\n\t\t\t},\n\t\t\t\"response\": []\n\t\t},\n\t\t{\n\t\t\t\"name\": \"Scale Min Workers - Synchronous\",\n\t\t\t\"event\": [\n\t\t\t\t{\n\t\t\t\t\t\"listen\": \"test\",\n\t\t\t\t\t\"script\": {\n\t\t\t\t\t\t\"id\": \"3a8cf27e-b0d9-4df9-bdcf-9eadb239807d\",\n\t\t\t\t\t\t\"exec\": [\n\t\t\t\t\t\t\t\"pm.test(\\\"Successful PUT request\\\", function () {\",\n\t\t\t\t\t\t\t\" pm.expect(pm.response.code).to.be.oneOf([200,201,202]);\",\n\t\t\t\t\t\t\t\"});\"\n\t\t\t\t\t\t],\n\t\t\t\t\t\t\"type\": \"text/javascript\"\n\t\t\t\t\t}\n\t\t\t\t}\n\t\t\t],\n\t\t\t\"request\": {\n\t\t\t\t\"method\": \"PUT\",\n\t\t\t\t\"header\": [],\n\t\t\t\t\"url\": {\n\t\t\t\t\t\"raw\": \"{{protocol}}://{{hostname}}:{{mgmt-port}}/models/squeezenet1_1?min_worker=4&synchronous=true\",\n\t\t\t\t\t\"protocol\": \"{{protocol}}\",\n\t\t\t\t\t\"host\": [\n\t\t\t\t\t\t\"{{hostname}}\"\n\t\t\t\t\t],\n\t\t\t\t\t\"port\": \"{{mgmt-port}}\",\n\t\t\t\t\t\"path\": [\n\t\t\t\t\t\t\"models\",\n\t\t\t\t\t\t\"squeezenet1_1\"\n\t\t\t\t\t],\n\t\t\t\t\t\"query\": [\n\t\t\t\t\t\t{\n\t\t\t\t\t\t\t\"key\": \"min_worker\",\n\t\t\t\t\t\t\t\"value\": \"4\"\n\t\t\t\t\t\t},\n\t\t\t\t\t\t{\n\t\t\t\t\t\t\t\"key\": \"synchronous\",\n\t\t\t\t\t\t\t\"value\": \"true\"\n\t\t\t\t\t\t}\n\t\t\t\t\t]\n\t\t\t\t}\n\t\t\t},\n\t\t\t\"response\": []\n\t\t},\n\t\t{\n\t\t\t\"name\": \"Scale Min Workers with GPU\",\n\t\t\t\"event\": [\n\t\t\t\t{\n\t\t\t\t\t\"listen\": \"test\",\n\t\t\t\t\t\"script\": {\n\t\t\t\t\t\t\"id\": \"b73619d3-3abc-4628-8e52-ecbcda561191\",\n\t\t\t\t\t\t\"exec\": [\n\t\t\t\t\t\t\t\"pm.test(\\\"Successful PUT request\\\", function () {\",\n\t\t\t\t\t\t\t\" pm.expect(pm.response.code).to.be.oneOf([200,201,202]);\",\n\t\t\t\t\t\t\t\"});\"\n\t\t\t\t\t\t],\n\t\t\t\t\t\t\"type\": \"text/javascript\"\n\t\t\t\t\t}\n\t\t\t\t}\n\t\t\t],\n\t\t\t\"request\": {\n\t\t\t\t\"method\": \"PUT\",\n\t\t\t\t\"header\": [],\n\t\t\t\t\"url\": {\n\t\t\t\t\t\"raw\": \"{{protocol}}://{{hostname}}:{{mgmt-port}}/models/squeezenet1_1?min_worker=6&number_gpu=1\",\n\t\t\t\t\t\"protocol\": \"{{protocol}}\",\n\t\t\t\t\t\"host\": [\n\t\t\t\t\t\t\"{{hostname}}\"\n\t\t\t\t\t],\n\t\t\t\t\t\"port\": \"{{mgmt-port}}\",\n\t\t\t\t\t\"path\": [\n\t\t\t\t\t\t\"models\",\n\t\t\t\t\t\t\"squeezenet1_1\"\n\t\t\t\t\t],\n\t\t\t\t\t\"query\": [\n\t\t\t\t\t\t{\n\t\t\t\t\t\t\t\"key\": \"min_worker\",\n\t\t\t\t\t\t\t\"value\": \"6\"\n\t\t\t\t\t\t},\n\t\t\t\t\t\t{\n\t\t\t\t\t\t\t\"key\": \"number_gpu\",\n\t\t\t\t\t\t\t\"value\": \"1\"\n\t\t\t\t\t\t}\n\t\t\t\t\t]\n\t\t\t\t}\n\t\t\t},\n\t\t\t\"response\": []\n\t\t},\n\t\t{\n\t\t\t\"name\": \"UnRegister Model\",\n\t\t\t\"event\": [\n\t\t\t\t{\n\t\t\t\t\t\"listen\": \"test\",\n\t\t\t\t\t\"script\": {\n\t\t\t\t\t\t\"id\": \"0e10d089-7183-4adf-ba81-d14678f5ecc7\",\n\t\t\t\t\t\t\"exec\": [\n\t\t\t\t\t\t\t\"pm.test(\\\"Successful DELETE request\\\", function () {\",\n\t\t\t\t\t\t\t\" pm.expect(pm.response.code).to.be.oneOf([200,201,202]);\",\n\t\t\t\t\t\t\t\"});\"\n\t\t\t\t\t\t],\n\t\t\t\t\t\t\"type\": \"text/javascript\"\n\t\t\t\t\t}\n\t\t\t\t}\n\t\t\t],\n\t\t\t\"request\": {\n\t\t\t\t\"method\": \"DELETE\",\n\t\t\t\t\"header\": [],\n\t\t\t\t\"url\": {\n\t\t\t\t\t\"raw\": \"{{protocol}}://{{hostname}}:{{mgmt-port}}/models/squeezenet1_1\",\n\t\t\t\t\t\"protocol\": \"{{protocol}}\",\n\t\t\t\t\t\"host\": [\n\t\t\t\t\t\t\"{{hostname}}\"\n\t\t\t\t\t],\n\t\t\t\t\t\"port\": \"{{mgmt-port}}\",\n\t\t\t\t\t\"path\": [\n\t\t\t\t\t\t\"models\",\n\t\t\t\t\t\t\"squeezenet1_1\"\n\t\t\t\t\t]\n\t\t\t\t}\n\t\t\t},\n\t\t\t\"response\": []\n\t\t},\n\t\t{\n\t\t\t\"name\": \"Register Model with Additional Params\",\n\t\t\t\"event\": [\n\t\t\t\t{\n\t\t\t\t\t\"listen\": \"test\",\n\t\t\t\t\t\"script\": {\n\t\t\t\t\t\t\"id\": \"5f79bffa-f260-440d-9d49-a3553972b8eb\",\n\t\t\t\t\t\t\"exec\": [\n\t\t\t\t\t\t\t\"pm.test(\\\"Successful POST request\\\", function () {\",\n\t\t\t\t\t\t\t\" pm.expect(pm.response.code).to.be.oneOf([200]);\",\n\t\t\t\t\t\t\t\"});\"\n\t\t\t\t\t\t],\n\t\t\t\t\t\t\"type\": \"text/javascript\"\n\t\t\t\t\t}\n\t\t\t\t}\n\t\t\t],\n\t\t\t\"request\": {\n\t\t\t\t\"method\": \"POST\",\n\t\t\t\t\"header\": [],\n\t\t\t\t\"url\": {\n\t\t\t\t\t\"raw\": \"{{protocol}}://{{hostname}}:{{mgmt-port}}/models?url=https://s3.amazonaws.com/model-server/model_archive_1.0/squeezenet_v1.1.mar&model_name=squeezenet1_1&initial_workers=1&synchronous=true\",\n\t\t\t\t\t\"protocol\": \"{{protocol}}\",\n\t\t\t\t\t\"host\": [\n\t\t\t\t\t\t\"{{hostname}}\"\n\t\t\t\t\t],\n\t\t\t\t\t\"port\": \"{{mgmt-port}}\",\n\t\t\t\t\t\"path\": [\n\t\t\t\t\t\t\"models\"\n\t\t\t\t\t],\n\t\t\t\t\t\"query\": [\n\t\t\t\t\t\t{\n\t\t\t\t\t\t\t\"key\": \"url\",\n\t\t\t\t\t\t\t\"value\": \"https://s3.amazonaws.com/model-server/model_archive_1.0/squeezenet_v1.1.mar\"\n\t\t\t\t\t\t},\n\t\t\t\t\t\t{\n\t\t\t\t\t\t\t\"key\": \"model_name\",\n\t\t\t\t\t\t\t\"value\": \"squeezenet1_1\"\n\t\t\t\t\t\t},\n\t\t\t\t\t\t{\n\t\t\t\t\t\t\t\"key\": \"initial_workers\",\n\t\t\t\t\t\t\t\"value\": \"1\"\n\t\t\t\t\t\t},\n\t\t\t\t\t\t{\n\t\t\t\t\t\t\t\"key\": \"synchronous\",\n\t\t\t\t\t\t\t\"value\": \"true\"\n\t\t\t\t\t\t}\n\t\t\t\t\t]\n\t\t\t\t}\n\t\t\t},\n\t\t\t\"response\": []\n\t\t},\n\t\t{\n\t\t\t\"name\": \"Register Model Synchronous\",\n\t\t\t\"event\": [\n\t\t\t\t{\n\t\t\t\t\t\"listen\": \"test\",\n\t\t\t\t\t\"script\": {\n\t\t\t\t\t\t\"id\": \"74b42a61-dda4-4d70-9543-e61a5be2dd9b\",\n\t\t\t\t\t\t\"exec\": [\n\t\t\t\t\t\t\t\"pm.test(\\\"Successful POST request\\\", function () {\",\n\t\t\t\t\t\t\t\" pm.expect(pm.response.code).to.be.oneOf([200]);\",\n\t\t\t\t\t\t\t\"});\"\n\t\t\t\t\t\t],\n\t\t\t\t\t\t\"type\": \"text/javascript\"\n\t\t\t\t\t}\n\t\t\t\t}\n\t\t\t],\n\t\t\t\"request\": {\n\t\t\t\t\"method\": \"POST\",\n\t\t\t\t\"header\": [],\n\t\t\t\t\"url\": {\n\t\t\t\t\t\"raw\": \"{{protocol}}://{{hostname}}:{{mgmt-port}}/models?url=https://s3.amazonaws.com/model-server/model_archive_1.0/resnet-18.mar&model_name=resnet18&synchronous=true\",\n\t\t\t\t\t\"protocol\": \"{{protocol}}\",\n\t\t\t\t\t\"host\": [\n\t\t\t\t\t\t\"{{hostname}}\"\n\t\t\t\t\t],\n\t\t\t\t\t\"port\": \"{{mgmt-port}}\",\n\t\t\t\t\t\"path\": [\n\t\t\t\t\t\t\"models\"\n\t\t\t\t\t],\n\t\t\t\t\t\"query\": [\n\t\t\t\t\t\t{\n\t\t\t\t\t\t\t\"key\": \"url\",\n\t\t\t\t\t\t\t\"value\": \"https://s3.amazonaws.com/model-server/model_archive_1.0/resnet-18.mar\"\n\t\t\t\t\t\t},\n\t\t\t\t\t\t{\n\t\t\t\t\t\t\t\"key\": \"model_name\",\n\t\t\t\t\t\t\t\"value\": \"resnet18\"\n\t\t\t\t\t\t},\n\t\t\t\t\t\t{\n\t\t\t\t\t\t\t\"key\": \"synchronous\",\n\t\t\t\t\t\t\t\"value\": \"true\"\n\t\t\t\t\t\t}\n\t\t\t\t\t]\n\t\t\t\t}\n\t\t\t},\n\t\t\t\"response\": []\n\t\t},\n\t\t{\n\t\t\t\"name\": \"UnRegister Model\",\n\t\t\t\"event\": [\n\t\t\t\t{\n\t\t\t\t\t\"listen\": \"test\",\n\t\t\t\t\t\"script\": {\n\t\t\t\t\t\t\"id\": \"d74f53dc-5aac-4afc-8acf-849a309054e5\",\n\t\t\t\t\t\t\"exec\": [\n\t\t\t\t\t\t\t\"pm.test(\\\"Successful DELETE request\\\", function () {\",\n\t\t\t\t\t\t\t\" pm.expect(pm.response.code).to.be.oneOf([200]);\",\n\t\t\t\t\t\t\t\"});\"\n\t\t\t\t\t\t],\n\t\t\t\t\t\t\"type\": \"text/javascript\"\n\t\t\t\t\t}\n\t\t\t\t}\n\t\t\t],\n\t\t\t\"request\": {\n\t\t\t\t\"method\": \"DELETE\",\n\t\t\t\t\"header\": [],\n\t\t\t\t\"url\": {\n\t\t\t\t\t\"raw\": \"{{protocol}}://{{hostname}}:{{mgmt-port}}/models/resnet18\",\n\t\t\t\t\t\"protocol\": \"{{protocol}}\",\n\t\t\t\t\t\"host\": [\n\t\t\t\t\t\t\"{{hostname}}\"\n\t\t\t\t\t],\n\t\t\t\t\t\"port\": \"{{mgmt-port}}\",\n\t\t\t\t\t\"path\": [\n\t\t\t\t\t\t\"models\",\n\t\t\t\t\t\t\"resnet18\"\n\t\t\t\t\t]\n\t\t\t\t}\n\t\t\t},\n\t\t\t\"response\": []\n\t\t},\n\t\t{\n\t\t\t\"name\": \"Register Resnet Model\",\n\t\t\t\"event\": [\n\t\t\t\t{\n\t\t\t\t\t\"listen\": \"test\",\n\t\t\t\t\t\"script\": {\n\t\t\t\t\t\t\"id\": \"11206673-aec5-4d26-a06c-96fab1fc81f3\",\n\t\t\t\t\t\t\"exec\": [\n\t\t\t\t\t\t\t\"pm.test(\\\"Successful POST request\\\", function () {\",\n\t\t\t\t\t\t\t\" pm.expect(pm.response.code).to.be.oneOf([200]);\",\n\t\t\t\t\t\t\t\"});\"\n\t\t\t\t\t\t],\n\t\t\t\t\t\t\"type\": \"text/javascript\"\n\t\t\t\t\t}\n\t\t\t\t}\n\t\t\t],\n\t\t\t\"request\": {\n\t\t\t\t\"method\": \"POST\",\n\t\t\t\t\"header\": [],\n\t\t\t\t\"url\": {\n\t\t\t\t\t\"raw\": \"{{protocol}}://{{hostname}}:{{mgmt-port}}/models?url=https://s3.amazonaws.com/model-server/model_archive_1.0/resnet-18.mar&model_name=resnet18&synchronous=false\",\n\t\t\t\t\t\"protocol\": \"{{protocol}}\",\n\t\t\t\t\t\"host\": [\n\t\t\t\t\t\t\"{{hostname}}\"\n\t\t\t\t\t],\n\t\t\t\t\t\"port\": \"{{mgmt-port}}\",\n\t\t\t\t\t\"path\": [\n\t\t\t\t\t\t\"models\"\n\t\t\t\t\t],\n\t\t\t\t\t\"query\": [\n\t\t\t\t\t\t{\n\t\t\t\t\t\t\t\"key\": \"url\",\n\t\t\t\t\t\t\t\"value\": \"https://s3.amazonaws.com/model-server/model_archive_1.0/resnet-18.mar\"\n\t\t\t\t\t\t},\n\t\t\t\t\t\t{\n\t\t\t\t\t\t\t\"key\": \"model_name\",\n\t\t\t\t\t\t\t\"value\": \"resnet18\"\n\t\t\t\t\t\t},\n\t\t\t\t\t\t{\n\t\t\t\t\t\t\t\"key\": \"synchronous\",\n\t\t\t\t\t\t\t\"value\": \"false\"\n\t\t\t\t\t\t}\n\t\t\t\t\t]\n\t\t\t\t}\n\t\t\t},\n\t\t\t\"response\": []\n\t\t},\n\t\t{\n\t\t\t\"name\": \"List with Limit\",\n\t\t\t\"event\": [\n\t\t\t\t{\n\t\t\t\t\t\"listen\": \"test\",\n\t\t\t\t\t\"script\": {\n\t\t\t\t\t\t\"id\": \"55acdbb9-cae8-4ad1-b2a4-b50a3549e8be\",\n\t\t\t\t\t\t\"exec\": [\n\t\t\t\t\t\t\t\"pm.test(\\\"Successful POST request\\\", function () {\",\n\t\t\t\t\t\t\t\" pm.expect(pm.response.code).to.be.oneOf([200]);\",\n\t\t\t\t\t\t\t\"});\"\n\t\t\t\t\t\t],\n\t\t\t\t\t\t\"type\": \"text/javascript\"\n\t\t\t\t\t}\n\t\t\t\t}\n\t\t\t],\n\t\t\t\"request\": {\n\t\t\t\t\"method\": \"GET\",\n\t\t\t\t\"header\": [],\n\t\t\t\t\"url\": {\n\t\t\t\t\t\"raw\": \"{{protocol}}://{{hostname}}:{{mgmt-port}}/models?limit=1\",\n\t\t\t\t\t\"protocol\": \"{{protocol}}\",\n\t\t\t\t\t\"host\": [\n\t\t\t\t\t\t\"{{hostname}}\"\n\t\t\t\t\t],\n\t\t\t\t\t\"port\": \"{{mgmt-port}}\",\n\t\t\t\t\t\"path\": [\n\t\t\t\t\t\t\"models\"\n\t\t\t\t\t],\n\t\t\t\t\t\"query\": [\n\t\t\t\t\t\t{\n\t\t\t\t\t\t\t\"key\": \"limit\",\n\t\t\t\t\t\t\t\"value\": \"1\"\n\t\t\t\t\t\t}\n\t\t\t\t\t]\n\t\t\t\t}\n\t\t\t},\n\t\t\t\"response\": []\n\t\t},\n\t\t{\n\t\t\t\"name\": \"List with Pagination\",\n\t\t\t\"event\": [\n\t\t\t\t{\n\t\t\t\t\t\"listen\": \"test\",\n\t\t\t\t\t\"script\": {\n\t\t\t\t\t\t\"id\": \"d1b17cfc-90af-4b61-af07-081224ba7f81\",\n\t\t\t\t\t\t\"exec\": [\n\t\t\t\t\t\t\t\"pm.test(\\\"Successful POST request\\\", function () {\",\n\t\t\t\t\t\t\t\" pm.expect(pm.response.code).to.be.oneOf([200]);\",\n\t\t\t\t\t\t\t\"});\"\n\t\t\t\t\t\t],\n\t\t\t\t\t\t\"type\": \"text/javascript\"\n\t\t\t\t\t}\n\t\t\t\t}\n\t\t\t],\n\t\t\t\"request\": {\n\t\t\t\t\"method\": \"GET\",\n\t\t\t\t\"header\": [],\n\t\t\t\t\"url\": {\n\t\t\t\t\t\"raw\": \"{{protocol}}://{{hostname}}:{{mgmt-port}}/models?limit=1&next_page_token=1\",\n\t\t\t\t\t\"protocol\": \"{{protocol}}\",\n\t\t\t\t\t\"host\": [\n\t\t\t\t\t\t\"{{hostname}}\"\n\t\t\t\t\t],\n\t\t\t\t\t\"port\": \"{{mgmt-port}}\",\n\t\t\t\t\t\"path\": [\n\t\t\t\t\t\t\"models\"\n\t\t\t\t\t],\n\t\t\t\t\t\"query\": [\n\t\t\t\t\t\t{\n\t\t\t\t\t\t\t\"key\": \"limit\",\n\t\t\t\t\t\t\t\"value\": \"1\"\n\t\t\t\t\t\t},\n\t\t\t\t\t\t{\n\t\t\t\t\t\t\t\"key\": \"next_page_token\",\n\t\t\t\t\t\t\t\"value\": \"1\"\n\t\t\t\t\t\t}\n\t\t\t\t\t]\n\t\t\t\t}\n\t\t\t},\n\t\t\t\"response\": []\n\t\t},\n\t\t{\n\t\t\t\"name\": \"Update GPU Count\",\n\t\t\t\"event\": [\n\t\t\t\t{\n\t\t\t\t\t\"listen\": \"test\",\n\t\t\t\t\t\"script\": {\n\t\t\t\t\t\t\"id\": \"e00d8108-3ce9-4a02-9fe1-3e38907d7f7b\",\n\t\t\t\t\t\t\"exec\": [\n\t\t\t\t\t\t\t\"pm.test(\\\"Successful PUT request\\\", function () {\",\n\t\t\t\t\t\t\t\" pm.expect(pm.response.code).to.be.oneOf([200,201,202]);\",\n\t\t\t\t\t\t\t\"});\"\n\t\t\t\t\t\t],\n\t\t\t\t\t\t\"type\": \"text/javascript\"\n\t\t\t\t\t}\n\t\t\t\t}\n\t\t\t],\n\t\t\t\"request\": {\n\t\t\t\t\"method\": \"PUT\",\n\t\t\t\t\"header\": [],\n\t\t\t\t\"url\": {\n\t\t\t\t\t\"raw\": \"{{protocol}}://{{hostname}}:{{mgmt-port}}/models/resnet18?number_gpu=10\",\n\t\t\t\t\t\"protocol\": \"{{protocol}}\",\n\t\t\t\t\t\"host\": [\n\t\t\t\t\t\t\"{{hostname}}\"\n\t\t\t\t\t],\n\t\t\t\t\t\"port\": \"{{mgmt-port}}\",\n\t\t\t\t\t\"path\": [\n\t\t\t\t\t\t\"models\",\n\t\t\t\t\t\t\"resnet18\"\n\t\t\t\t\t],\n\t\t\t\t\t\"query\": [\n\t\t\t\t\t\t{\n\t\t\t\t\t\t\t\"key\": \"number_gpu\",\n\t\t\t\t\t\t\t\"value\": \"10\"\n\t\t\t\t\t\t}\n\t\t\t\t\t]\n\t\t\t\t}\n\t\t\t},\n\t\t\t\"response\": []\n\t\t},\n\t\t{\n\t\t\t\"name\": \"Scale up Workers - Synchronous\",\n\t\t\t\"event\": [\n\t\t\t\t{\n\t\t\t\t\t\"listen\": \"test\",\n\t\t\t\t\t\"script\": {\n\t\t\t\t\t\t\"id\": \"f575937f-1665-4cf4-91a7-31c351eb9424\",\n\t\t\t\t\t\t\"exec\": [\n\t\t\t\t\t\t\t\"pm.test(\\\"Successful PUT request\\\", function () {\",\n\t\t\t\t\t\t\t\" pm.expect(pm.response.code).to.be.oneOf([200,201,202]);\",\n\t\t\t\t\t\t\t\"});\"\n\t\t\t\t\t\t],\n\t\t\t\t\t\t\"type\": \"text/javascript\"\n\t\t\t\t\t}\n\t\t\t\t}\n\t\t\t],\n\t\t\t\"request\": {\n\t\t\t\t\"method\": \"PUT\",\n\t\t\t\t\"header\": [],\n\t\t\t\t\"url\": {\n\t\t\t\t\t\"raw\": \"{{protocol}}://{{hostname}}:{{mgmt-port}}/models/resnet18?min_worker=5&max_worker=5&synchronous=true\",\n\t\t\t\t\t\"protocol\": \"{{protocol}}\",\n\t\t\t\t\t\"host\": [\n\t\t\t\t\t\t\"{{hostname}}\"\n\t\t\t\t\t],\n\t\t\t\t\t\"port\": \"{{mgmt-port}}\",\n\t\t\t\t\t\"path\": [\n\t\t\t\t\t\t\"models\",\n\t\t\t\t\t\t\"resnet18\"\n\t\t\t\t\t],\n\t\t\t\t\t\"query\": [\n\t\t\t\t\t\t{\n\t\t\t\t\t\t\t\"key\": \"min_worker\",\n\t\t\t\t\t\t\t\"value\": \"5\"\n\t\t\t\t\t\t},\n\t\t\t\t\t\t{\n\t\t\t\t\t\t\t\"key\": \"max_worker\",\n\t\t\t\t\t\t\t\"value\": \"5\"\n\t\t\t\t\t\t},\n\t\t\t\t\t\t{\n\t\t\t\t\t\t\t\"key\": \"synchronous\",\n\t\t\t\t\t\t\t\"value\": \"true\"\n\t\t\t\t\t\t}\n\t\t\t\t\t]\n\t\t\t\t}\n\t\t\t},\n\t\t\t\"response\": []\n\t\t},\n\t\t{\n\t\t\t\"name\": \"Scale up Workers - Asynchronous\",\n\t\t\t\"event\": [\n\t\t\t\t{\n\t\t\t\t\t\"listen\": \"test\",\n\t\t\t\t\t\"script\": {\n\t\t\t\t\t\t\"id\": \"a65c2a28-de18-4ac5-a893-9aa2cf3cef77\",\n\t\t\t\t\t\t\"exec\": [\n\t\t\t\t\t\t\t\"pm.test(\\\"Successful PUT request\\\", function () {\",\n\t\t\t\t\t\t\t\" pm.expect(pm.response.code).to.be.oneOf([200,201,202]);\",\n\t\t\t\t\t\t\t\"});\"\n\t\t\t\t\t\t],\n\t\t\t\t\t\t\"type\": \"text/javascript\"\n\t\t\t\t\t}\n\t\t\t\t}\n\t\t\t],\n\t\t\t\"request\": {\n\t\t\t\t\"method\": \"PUT\",\n\t\t\t\t\"header\": [],\n\t\t\t\t\"url\": {\n\t\t\t\t\t\"raw\": \"{{protocol}}://{{hostname}}:{{mgmt-port}}/models/resnet18?min_worker=6&max_worker=6&synchronous=false\",\n\t\t\t\t\t\"protocol\": \"{{protocol}}\",\n\t\t\t\t\t\"host\": [\n\t\t\t\t\t\t\"{{hostname}}\"\n\t\t\t\t\t],\n\t\t\t\t\t\"port\": \"{{mgmt-port}}\",\n\t\t\t\t\t\"path\": [\n\t\t\t\t\t\t\"models\",\n\t\t\t\t\t\t\"resnet18\"\n\t\t\t\t\t],\n\t\t\t\t\t\"query\": [\n\t\t\t\t\t\t{\n\t\t\t\t\t\t\t\"key\": \"min_worker\",\n\t\t\t\t\t\t\t\"value\": \"6\"\n\t\t\t\t\t\t},\n\t\t\t\t\t\t{\n\t\t\t\t\t\t\t\"key\": \"max_worker\",\n\t\t\t\t\t\t\t\"value\": \"6\"\n\t\t\t\t\t\t},\n\t\t\t\t\t\t{\n\t\t\t\t\t\t\t\"key\": \"synchronous\",\n\t\t\t\t\t\t\t\"value\": \"false\"\n\t\t\t\t\t\t}\n\t\t\t\t\t]\n\t\t\t\t}\n\t\t\t},\n\t\t\t\"response\": []\n\t\t},\n\t\t{\n\t\t\t\"name\": \"Update Timeout to -1\",\n\t\t\t\"event\": [\n\t\t\t\t{\n\t\t\t\t\t\"listen\": \"test\",\n\t\t\t\t\t\"script\": {\n\t\t\t\t\t\t\"id\": \"96e4be74-20d4-4343-b950-2738d6034341\",\n\t\t\t\t\t\t\"exec\": [\n\t\t\t\t\t\t\t\"pm.test(\\\"Successful PUT request\\\", function () {\",\n\t\t\t\t\t\t\t\" pm.expect(pm.response.code).to.be.oneOf([200,201,202]);\",\n\t\t\t\t\t\t\t\"});\"\n\t\t\t\t\t\t],\n\t\t\t\t\t\t\"type\": \"text/javascript\"\n\t\t\t\t\t}\n\t\t\t\t}\n\t\t\t],\n\t\t\t\"request\": {\n\t\t\t\t\"method\": \"PUT\",\n\t\t\t\t\"header\": [],\n\t\t\t\t\"url\": {\n\t\t\t\t\t\"raw\": \"{{protocol}}://{{hostname}}:{{mgmt-port}}/models/resnet18?timeout=-1\",\n\t\t\t\t\t\"protocol\": \"{{protocol}}\",\n\t\t\t\t\t\"host\": [\n\t\t\t\t\t\t\"{{hostname}}\"\n\t\t\t\t\t],\n\t\t\t\t\t\"port\": \"{{mgmt-port}}\",\n\t\t\t\t\t\"path\": [\n\t\t\t\t\t\t\"models\",\n\t\t\t\t\t\t\"resnet18\"\n\t\t\t\t\t],\n\t\t\t\t\t\"query\": [\n\t\t\t\t\t\t{\n\t\t\t\t\t\t\t\"key\": \"timeout\",\n\t\t\t\t\t\t\t\"value\": \"-1\"\n\t\t\t\t\t\t}\n\t\t\t\t\t]\n\t\t\t\t}\n\t\t\t},\n\t\t\t\"response\": []\n\t\t},\n\t\t{\n\t\t\t\"name\": \"Update Timeout to 0\",\n\t\t\t\"event\": [\n\t\t\t\t{\n\t\t\t\t\t\"listen\": \"test\",\n\t\t\t\t\t\"script\": {\n\t\t\t\t\t\t\"id\": \"d2c89755-a457-48ef-8dea-fdd61e88d1e5\",\n\t\t\t\t\t\t\"exec\": [\n\t\t\t\t\t\t\t\"pm.test(\\\"Successful PUT request\\\", function () {\",\n\t\t\t\t\t\t\t\" pm.expect(pm.response.code).to.be.oneOf([200,201,202]);\",\n\t\t\t\t\t\t\t\"});\"\n\t\t\t\t\t\t],\n\t\t\t\t\t\t\"type\": \"text/javascript\"\n\t\t\t\t\t}\n\t\t\t\t}\n\t\t\t],\n\t\t\t\"request\": {\n\t\t\t\t\"method\": \"PUT\",\n\t\t\t\t\"header\": [],\n\t\t\t\t\"url\": {\n\t\t\t\t\t\"raw\": \"{{protocol}}://{{hostname}}:{{mgmt-port}}/models/resnet18?timeout=0\",\n\t\t\t\t\t\"protocol\": \"{{protocol}}\",\n\t\t\t\t\t\"host\": [\n\t\t\t\t\t\t\"{{hostname}}\"\n\t\t\t\t\t],\n\t\t\t\t\t\"port\": \"{{mgmt-port}}\",\n\t\t\t\t\t\"path\": [\n\t\t\t\t\t\t\"models\",\n\t\t\t\t\t\t\"resnet18\"\n\t\t\t\t\t],\n\t\t\t\t\t\"query\": [\n\t\t\t\t\t\t{\n\t\t\t\t\t\t\t\"key\": \"timeout\",\n\t\t\t\t\t\t\t\"value\": \"0\"\n\t\t\t\t\t\t}\n\t\t\t\t\t]\n\t\t\t\t}\n\t\t\t},\n\t\t\t\"response\": []\n\t\t},\n\t\t{\n\t\t\t\"name\": \"Register Model - Invalid URL\",\n\t\t\t\"event\": [\n\t\t\t\t{\n\t\t\t\t\t\"listen\": \"test\",\n\t\t\t\t\t\"script\": {\n\t\t\t\t\t\t\"id\": \"427b68f9-de96-49b8-9477-450fa580e257\",\n\t\t\t\t\t\t\"exec\": [\n\t\t\t\t\t\t\t\"pm.test(\\\"Invalid URL POST request\\\", function () {\",\n\t\t\t\t\t\t\t\" pm.expect(pm.response.code).to.be.oneOf([400]);\",\n\t\t\t\t\t\t\t\"});\"\n\t\t\t\t\t\t],\n\t\t\t\t\t\t\"type\": \"text/javascript\"\n\t\t\t\t\t}\n\t\t\t\t}\n\t\t\t],\n\t\t\t\"request\": {\n\t\t\t\t\"method\": \"POST\",\n\t\t\t\t\"header\": [],\n\t\t\t\t\"url\": {\n\t\t\t\t\t\"raw\": \"{{protocol}}://{{hostname}}:{{mgmt-port}}/models?url=https://s3.amazonaws.com/model-server/model_archive_1.0/invalid-resnet-18.mar&model_name=invalid-resnet18\",\n\t\t\t\t\t\"protocol\": \"{{protocol}}\",\n\t\t\t\t\t\"host\": [\n\t\t\t\t\t\t\"{{hostname}}\"\n\t\t\t\t\t],\n\t\t\t\t\t\"port\": \"{{mgmt-port}}\",\n\t\t\t\t\t\"path\": [\n\t\t\t\t\t\t\"models\"\n\t\t\t\t\t],\n\t\t\t\t\t\"query\": [\n\t\t\t\t\t\t{\n\t\t\t\t\t\t\t\"key\": \"url\",\n\t\t\t\t\t\t\t\"value\": \"https://s3.amazonaws.com/model-server/model_archive_1.0/invalid-resnet-18.mar\"\n\t\t\t\t\t\t},\n\t\t\t\t\t\t{\n\t\t\t\t\t\t\t\"key\": \"model_name\",\n\t\t\t\t\t\t\t\"value\": \"invalid-resnet18\"\n\t\t\t\t\t\t}\n\t\t\t\t\t]\n\t\t\t\t}\n\t\t\t},\n\t\t\t\"response\": []\n\t\t},\n\t\t{\n\t\t\t\"name\": \"Get Model - Invalid Model\",\n\t\t\t\"event\": [\n\t\t\t\t{\n\t\t\t\t\t\"listen\": \"test\",\n\t\t\t\t\t\"script\": {\n\t\t\t\t\t\t\"id\": \"36836026-4477-493b-898a-b642628a51d3\",\n\t\t\t\t\t\t\"exec\": [\n\t\t\t\t\t\t\t\"pm.test(\\\"Valid ERROR message\\\", function () {\",\n\t\t\t\t\t\t\t\" pm.expect(pm.response.code).to.be.oneOf([404]);\",\n\t\t\t\t\t\t\t\"});\"\n\t\t\t\t\t\t],\n\t\t\t\t\t\t\"type\": \"text/javascript\"\n\t\t\t\t\t}\n\t\t\t\t}\n\t\t\t],\n\t\t\t\"request\": {\n\t\t\t\t\"method\": \"GET\",\n\t\t\t\t\"header\": [],\n\t\t\t\t\"url\": {\n\t\t\t\t\t\"raw\": \"{{protocol}}://{{hostname}}:{{mgmt-port}}/models/invalid_squeezenet1_1\",\n\t\t\t\t\t\"protocol\": \"{{protocol}}\",\n\t\t\t\t\t\"host\": [\n\t\t\t\t\t\t\"{{hostname}}\"\n\t\t\t\t\t],\n\t\t\t\t\t\"port\": \"{{mgmt-port}}\",\n\t\t\t\t\t\"path\": [\n\t\t\t\t\t\t\"models\",\n\t\t\t\t\t\t\"invalid_squeezenet1_1\"\n\t\t\t\t\t]\n\t\t\t\t}\n\t\t\t},\n\t\t\t\"response\": []\n\t\t},\n\t\t{\n\t\t\t\"name\": \"List Models - Invalid Next Page Token\",\n\t\t\t\"event\": [\n\t\t\t\t{\n\t\t\t\t\t\"listen\": \"test\",\n\t\t\t\t\t\"script\": {\n\t\t\t\t\t\t\"id\": \"ccc07f12-6f45-4213-b4f5-e37f7d0ceadb\",\n\t\t\t\t\t\t\"exec\": [\n\t\t\t\t\t\t\t\"pm.test(\\\"Successful POST request\\\", function () {\",\n\t\t\t\t\t\t\t\" pm.expect(pm.response.code).to.be.oneOf([200]);\",\n\t\t\t\t\t\t\t\"});\"\n\t\t\t\t\t\t],\n\t\t\t\t\t\t\"type\": \"text/javascript\"\n\t\t\t\t\t}\n\t\t\t\t}\n\t\t\t],\n\t\t\t\"request\": {\n\t\t\t\t\"method\": \"GET\",\n\t\t\t\t\"header\": [],\n\t\t\t\t\"url\": {\n\t\t\t\t\t\"raw\": \"{{protocol}}://{{hostname}}:{{mgmt-port}}/models?next_page_token=12\",\n\t\t\t\t\t\"protocol\": \"{{protocol}}\",\n\t\t\t\t\t\"host\": [\n\t\t\t\t\t\t\"{{hostname}}\"\n\t\t\t\t\t],\n\t\t\t\t\t\"port\": \"{{mgmt-port}}\",\n\t\t\t\t\t\"path\": [\n\t\t\t\t\t\t\"models\"\n\t\t\t\t\t],\n\t\t\t\t\t\"query\": [\n\t\t\t\t\t\t{\n\t\t\t\t\t\t\t\"key\": \"next_page_token\",\n\t\t\t\t\t\t\t\"value\": \"12\"\n\t\t\t\t\t\t}\n\t\t\t\t\t]\n\t\t\t\t}\n\t\t\t},\n\t\t\t\"response\": []\n\t\t},\n\t\t{\n\t\t\t\"name\": \"Update Worker with Invalid Worker Count\",\n\t\t\t\"event\": [\n\t\t\t\t{\n\t\t\t\t\t\"listen\": \"test\",\n\t\t\t\t\t\"script\": {\n\t\t\t\t\t\t\"id\": \"ec7d233e-54bf-456a-a148-811385a34cc4\",\n\t\t\t\t\t\t\"exec\": [\n\t\t\t\t\t\t\t\"pm.test(\\\"Valid ERROR message\\\", function () {\",\n\t\t\t\t\t\t\t\" pm.expect(pm.response.code).to.be.oneOf([400]);\",\n\t\t\t\t\t\t\t\"});\"\n\t\t\t\t\t\t],\n\t\t\t\t\t\t\"type\": \"text/javascript\"\n\t\t\t\t\t}\n\t\t\t\t}\n\t\t\t],\n\t\t\t\"request\": {\n\t\t\t\t\"method\": \"PUT\",\n\t\t\t\t\"header\": [],\n\t\t\t\t\"url\": {\n\t\t\t\t\t\"raw\": \"{{protocol}}://{{hostname}}:{{mgmt-port}}/models/resnet18?min_worker=10&max_worker=9\",\n\t\t\t\t\t\"protocol\": \"{{protocol}}\",\n\t\t\t\t\t\"host\": [\n\t\t\t\t\t\t\"{{hostname}}\"\n\t\t\t\t\t],\n\t\t\t\t\t\"port\": \"{{mgmt-port}}\",\n\t\t\t\t\t\"path\": [\n\t\t\t\t\t\t\"models\",\n\t\t\t\t\t\t\"resnet18\"\n\t\t\t\t\t],\n\t\t\t\t\t\"query\": [\n\t\t\t\t\t\t{\n\t\t\t\t\t\t\t\"key\": \"min_worker\",\n\t\t\t\t\t\t\t\"value\": \"10\"\n\t\t\t\t\t\t},\n\t\t\t\t\t\t{\n\t\t\t\t\t\t\t\"key\": \"max_worker\",\n\t\t\t\t\t\t\t\"value\": \"9\"\n\t\t\t\t\t\t}\n\t\t\t\t\t]\n\t\t\t\t}\n\t\t\t},\n\t\t\t\"response\": []\n\t\t},\n\t\t{\n\t\t\t\"name\": \"UnRegister Invalid Model Name\",\n\t\t\t\"event\": [\n\t\t\t\t{\n\t\t\t\t\t\"listen\": \"test\",\n\t\t\t\t\t\"script\": {\n\t\t\t\t\t\t\"id\": \"08052d51-fd23-4249-a17f-933e557bdacc\",\n\t\t\t\t\t\t\"exec\": [\n\t\t\t\t\t\t\t\"pm.test(\\\"Valid ERROR message\\\", function () {\",\n\t\t\t\t\t\t\t\" pm.expect(pm.response.code).to.be.oneOf([404]);\",\n\t\t\t\t\t\t\t\"});\"\n\t\t\t\t\t\t],\n\t\t\t\t\t\t\"type\": \"text/javascript\"\n\t\t\t\t\t}\n\t\t\t\t}\n\t\t\t],\n\t\t\t\"request\": {\n\t\t\t\t\"method\": \"DELETE\",\n\t\t\t\t\"header\": [],\n\t\t\t\t\"url\": {\n\t\t\t\t\t\"raw\": \"{{protocol}}://{{hostname}}:{{mgmt-port}}/models/invalid_squeezenet1_1\",\n\t\t\t\t\t\"protocol\": \"{{protocol}}\",\n\t\t\t\t\t\"host\": [\n\t\t\t\t\t\t\"{{hostname}}\"\n\t\t\t\t\t],\n\t\t\t\t\t\"port\": \"{{mgmt-port}}\",\n\t\t\t\t\t\"path\": [\n\t\t\t\t\t\t\"models\",\n\t\t\t\t\t\t\"invalid_squeezenet1_1\"\n\t\t\t\t\t]\n\t\t\t\t}\n\t\t\t},\n\t\t\t\"response\": []\n\t\t}\n\t]\n}" }, { "path": "test/regression_tests.sh", "content": "#!/bin/bash\n\nset -x\nset -e\n\nMMS_REPO=\"https://github.com/awslabs/multi-model-server.git\"\nBRANCH=${1:-master}\nROOT_DIR=\"/workspace/\"\nCODEBUILD_WD=$(pwd)\nMODEL_STORE=$ROOT_DIR\"/model_store\"\nMMS_LOG_FILE=\"/tmp/mms.log\"\nTEST_EXECUTION_LOG_FILE=\"/tmp/test_exec.log\"\n\ninstall_mms_from_source() {\n echo \"Cloning & Building Multi Model Server Repo from \" $1\n\n sudo apt-get -y install nodejs-dev node-gyp libssl1.0-dev\n sudo apt-get -y install npm\n sudo npm install -g n\n sudo n latest\n export PATH=\"$PATH\"\n sudo npm install -g newman newman-reporter-html\n pip install mxnet-mkl\n # Clone & Build MMS\n echo \"Installing MMS from source\"\n git clone -b $2 $1\n cd multi-model-server\n pip install .\n cd -\n echo \"MMS Succesfully installed\"\n \n}\n\n\nstart_mms() {\n\n # Start MMS with Model Store\n multi-model-server --start --model-store $1 &>> $2\n sleep 10\n curl http://127.0.0.1:8081/models\n \n}\n\nstop_mms_serve() {\n multi-model-server --stop\n}\n\nstart_secure_mms() {\n\n # Start MMS with Model Store\n multi-model-server --start --mms-config test/resources/config.properties --model-store $1 &>> $2\n sleep 10\n curl --insecure -X GET https://127.0.0.1:8444/models\n}\n\n\nrun_postman_test() {\n # Run Postman Scripts\n mkdir $ROOT_DIR/report/\n cd $CODEBUILD_WD/\n set +e\n # Run Management API Tests\n stop_mms_serve\n start_mms $MODEL_STORE $MMS_LOG_FILE\n newman run -e test/postman/environment.json --bail --verbose test/postman/management_api_test_collection.json \\\n\t -r cli,html --reporter-html-export $ROOT_DIR/report/management_report.html >>$1 2>&1\n\n # Run Inference API Tests after Restart\n stop_mms_serve\n start_mms $MODEL_STORE $MMS_LOG_FILE\n newman run -e test/postman/environment.json --bail --verbose test/postman/inference_api_test_collection.json \\\n\t -d test/postman/inference_data.json -r cli,html --reporter-html-export $ROOT_DIR/report/inference_report.html >>$1 2>&1\n\n\n # Run Https test cases\n stop_mms_serve\n start_secure_mms $MODEL_STORE $MMS_LOG_FILE\n newman run --insecure -e test/postman/environment.json --bail --verbose test/postman/https_test_collection.json \\\n\t -r cli,html --reporter-html-export $ROOT_DIR/report/MMS_https_test_report.html >>$1 2>&1\n\n stop_mms_serve\n set -e\n cd -\n}\n\n\nsudo rm -rf $ROOT_DIR && sudo mkdir $ROOT_DIR\nsudo chown -R $USER:$USER $ROOT_DIR\ncd $ROOT_DIR\nmkdir $MODEL_STORE\n\nsudo rm -f $TEST_EXECUTION_LOG_FILE $MMS_LOG_FILE\n\necho \"** Execuing MMS Regression Test Suite executon for \" $MMS_REPO \" **\"\n\ninstall_mms_from_source $MMS_REPO $BRANCH\nrun_postman_test $TEST_EXECUTION_LOG_FILE\n\necho \"** Tests Complete ** \"\nexit 0\n" }, { "path": "test/resources/certs.pem", "content": "-----BEGIN CERTIFICATE-----\nMIICiDCCAfGgAwIBAgIEeC8zQzANBgkqhkiG9w0BAQsFADB2MQswCQYDVQQGEwJV\nUzETMBEGA1UECBMKQ2FsaWZvcm5pYTESMBAGA1UEBxMJUGFsbyBBbHRvMRIwEAYD\nVQQKEwlBbWF6b24gQUkxDjAMBgNVBAsTBU14TmV0MRowGAYDVQQDExFtbXMuYW1h\nem9uYXdzLmNvbTAgFw0xODA2MjAwMjExMjhaGA8yMTE3MDExMjAyMTEyOFowdjEL\nMAkGA1UEBhMCVVMxEzARBgNVBAgTCkNhbGlmb3JuaWExEjAQBgNVBAcTCVBhbG8g\nQWx0bzESMBAGA1UEChMJQW1hem9uIEFJMQ4wDAYDVQQLEwVNeE5ldDEaMBgGA1UE\nAxMRbW1zLmFtYXpvbmF3cy5jb20wgZ8wDQYJKoZIhvcNAQEBBQADgY0AMIGJAoGB\nAMcbCEP6kn9pUcap5+kYO/5xEl7SL965gSQ2TbFrVv+sLVkLSK8yTtcILr7RUINz\nFsD151Q7VyQCvpVzkOFew2s2mAFWWxPJYmxo1j/R3IkJakrrTrMy1R3jsqOQMrxY\nTLGR5LIe2pjdAnb9xWe2NB125619WDG7RrdHWZDfvSPxAgMBAAGjITAfMB0GA1Ud\nDgQWBBRWjdEyNchYAkdPoyudKJY9YP3JPzANBgkqhkiG9w0BAQsFAAOBgQBMAvqG\ncqvD3ColO2Ihgb/LCfCdV14e1YhusVFeKyZkSKFYyQR+MoBOxqMQqJ24gVzgqTU/\nh+LkMqZcxxJAME08BzPgP5b06DBM4K0o0XUfYUViFpYXB0qCG5CA/0S7ONldBGaZ\nfv6JrnQ/a1NYBi92AaqXA4VmuaowWLVEFuPV1A==\n-----END CERTIFICATE-----\n" }, { "path": "test/resources/config.properties", "content": "inference_address=https://127.0.0.1:8443\nmanagement_address=https://127.0.0.1:8444\nprivate_key_file=test/resources/key.pem\ncertificate_file=test/resources/certs.pem\n" }, { "path": "test/resources/key.pem", "content": "-----BEGIN RSA PRIVATE KEY-----\nMIICXAIBAAKBgQDHGwhD+pJ/aVHGqefpGDv+cRJe0i/euYEkNk2xa1b/rC1ZC0iv\nMk7XCC6+0VCDcxbA9edUO1ckAr6Vc5DhXsNrNpgBVlsTyWJsaNY/0dyJCWpK606z\nMtUd47KjkDK8WEyxkeSyHtqY3QJ2/cVntjQdduetfVgxu0a3R1mQ370j8QIDAQAB\nAoGANRoxlyfSQKcPR2PzVUjAX3k6xA1c9RMWrVjKWeJd/qymH5SR2yAYxOMKzJu4\n1IYycF5lRyLYd+M/f06mOmVyysH3D7hkrNz57Z07UrZ0dO/mmUKRL7zc44mo22ck\nJtQRwWJMplgew7N8OyqEZbcLOpahjlkL4+KZIWOuO7X5m30CQQDob/rzNY8gfhEm\noEHHQ4dCqa/b5as2OqpFoGBZ+iX3dumBf+UKuSHlvEozt4ZMm29DYSjhiGXgLUFw\n6NBhWxpXAkEA20oNdGiYAyyGJ6TKkD3FNZYoqB5+E/Cq6c0AACssB4OrJtiGiBFq\nR1h5HTEwYMe+ciZ4CI5MvBukjAdlfn7W9wJAXOIqyTe060oVdncB8ivlCFmgweHk\najZFRq+Q8UPKGjq1kx9VmtRiXFjC2inTjBds/eL8oCuOcmgDR6hxZQYv3wJAcMLv\nkECIinlGsvQGRY297wQ7+9dSNaa3/Gmx6mRIy8RlKiCFbUqnP/C6tswoeFu+DqzB\nZITn6IK+ZlMXWaiXmQJBAK7V4rR+4VdpYUu1OqPRxChkcM+Y4Wa985A46/8yoo3i\n0PEenvApVzhVzS9jF6WEqIKcffBAmOxaXOn3kmn8w2A=\n-----END RSA PRIVATE KEY-----\n" }, { "path": "tests/performance/README.md", "content": "# Performance Regression Suite\n\n## Motivation\nThe goal of this test suite is to ensure that performance regressions are detected early on. Ideally, with every commit \nmade into the source control system. \n\nThe salient features of the performance regression suite are\n\n* Non-intrusive - Does not need any code-changes or instrumentation on the server being monitored. \n* It can be used to monitor a wide variety of server metrics - memory, cpu, io - in addition to \ntraditional API level metrics such as latency, throughput etc. \n* It is easy to add custom metrics. For example, in MMS server, `the number of workers spawned` would be an interesting \nmetric to track. The platform allows for easy addition of these metrics.\n* Test cases are specified in human readable yaml files. Every test case has a pass or fail status. This is determined \nby evaluating expressions specified in the test case. Every expression checks metrics against threshold values. For \nexample, `memory consumed by all workers < 500M`, `number of worker processes < 3`.\n* Test cases execute against compute environments. The threshold values are specific to the compute environment. It is\npossible to specify multiple compute environments against which the test cases will run. It follows that each compute \nenvironment, will have its own threshold values.\n* This suite leverages the open source [Taurus framework](https://gettaurus.org/). \n* This suite extends the Taurus framework in the following ways\n * Adds resource monitoring service. This allows MMS specific metrics to be added. \n * Environments as described earlier.\n * Specification of pass/fail criterion between two commits. For example, memory consumed by workers should not \n increase by more than 10% between two commits for the given test case.\n * Custom reporting of results.\n \nThe building blocks of the performance regression suite and flow is captured in the following drawing\n\n![](assets/blocks.png) \n\n## Quickstart\n\n### A. Installation\n1. Install Taurus. Refer the [link](https://gettaurus.org/docs/Installation/) for more details on installation.\n ```bash \n pip install bzt # Needs python3.6+\n ``` \n2. Install performance regression suite dependencies.\n ```bash \n export MMS_HOME=\n pip install -r $MMS_HOME/tests/performance/requirements.txt\n ``` \n3. Make sure that `git` is installed and the test suites are run from the MMS working directory.\n\n### B. Running the test suite\n1. Make sure parameters set in [tests/common/global_config.yaml](tests/performance/tests/global_config.yaml) are correct.\n2. To run the test suite execute [run_performance_suite.py](run_performance_suite.py) with the following \nparameters\n\n * `--artifacts-dir` or `-a` is a directory where the test case results will be stored. The default value is \n`$MMS_HOME/tests/performance/run_artifacts`. \n\n * `--test-dir` or `-t` is a directory containing the test cases. The default value is \n`$MMS_HOME/tests/performance/tests`.\n \n * `--pattern` or `-p` glob pattern picks up certain test cases for execution within the `test-dir`. The default value picks up \nall test cases.\n \n * `--exclude-pattern` or `-x` glob pattern excludes certain test cases for execution within the `test-dir`. \nThe default value excludes nothing.\n \n * `--env-name` or `-e` specifies the environment name to use while running the test cases. The environment name is the name of \nthe file (minus the extension) found inside the environments folder in each test case. They encapsulate parameter \nvalues which are specific to the execution environment. This is a mandatory parameter. \n\n The script does the following: \n 1. Starts the metrics monitoring server.\n 2. Collects all the tests from test-dir satisfying the pattern\n 3. Executes the tests\n 4. Generates artifacts in the artifacts-dir against each test case. \n\n3. Check the console logs, $artifacts-dir$//performance_results.html report, comparison.csv, comparison.html \nand other artifacts.\n\n**Steps are provided below**\n\n```bash\nexport MMS_HOME=\ncd $MMS_HOME/tests/performance\n \n# Note that MMS server started and stopped by the individual test suite.\n# check variables such as MMS server PORT etc \n# vi tests/common/global_config.yaml \n\n#all tests\npython -m run_performance_suite -e xlarge\n\n#run a specific test \npython -m run_performance_suite -e xlarge -p inference_single_worker\n\n```\n\n### C. Understanding the test suite artifacts and reports\n1. The $artifacts-dir$//performance_results.html is a summary report of the test run. \n2. Each test yaml is treated as a test suite. Each criteria in the test suite is treated as a test case. \nIf the test suite does not specify any criteria, then the test suite is reported as skipped with 0 test cases.\n3. For each test suite, a sub-directory is created containing relevant run artifacts. Important files in this directory are\n * metrics.csv -- contains the values of the various system-monitored metrics over time\n * finals_stats.csv -- contains the values of the various api metrics over time \n4. The $artifacts-dir$//comparison_results.html is a summary report which shows performance difference between\nthe last two commits.\n5. The run completes with a console summary of the performance and comparision suites which have failed\n![](assets/console.png) \n\n## Add a new test\n\nFollow these three steps to add a new test case to the test suite.\n\n1. Add scenario (a.k.a test suite)\n2. Add metrics to monitor\n3. Add pass/fail criteria (a.k.a test case)\n\n\n#### 1. Add scenario (a.k.a test suite)\nCreate a folder for the test under `test_dir` location. A test generally comprises of a jmeter file - containing the \nload scenario and a yaml file which contains test scenarios specifying the conditions for failure or success. The\nfile-names should be identical to the folder name with their respective extensions. \n\nAn example [jmeter script](tests/examples_starter/examples_starter.jmx) \nand a [scenario](tests/examples_starter/examples_starter.yaml) is provided as a template to get started.\n \nPlease note that various global configuration settings used by examples_starter.jmx script are specified in \n[tests/global_config.yaml](tests/performance/tests/global_config.yaml) file.\n \n ```tests/examples_starter/examples_starter.yaml\n execution:\n - concurrency: 1\n ramp-up: 1s\n hold-for: 40s\n scenario: Inference\n\n scenarios:\n Inference:\n script: examples_starter.jmx\n\n ```\n \nTo execute this test suite, run the following command\n \n ```bash\n export MMS_HOME=\n cd $MMS_HOME/tests/performance\n python -m run_performance_suite -p examples_starter -e xlarge\n ```\n\n**Note**:\nTaurus provides support for different executors such as JMeter. Supported executor types can be found [here](https://gettaurus.org/docs/ExecutionSettings/).\nDetails about how to use an existing JMeter script are provided [here](https://gettaurus.org/docs/JMeter/). \n\n\n#### 2. Add metrics to monitor\nSpecify the metrics of interest in the services/monitoring section of the yaml.\n\n1. Standalone monitoring server\n\n Use this technique if MMS and the tests execute on different machines. Before running the test cases, \n please start the [metrics_monitoring_server.py](metrics_monitoring_server.py) script. It will communicate server \n metric data with the test client over sockets. The monitoring server runs on port 9009 by default.\n \n To start the monitoring server, run the following commands on the MMS host:\n ```bash \n export MMS_HOME=\n pip install -r $MMS_HOME/tests/performance/requirements.txt\n python $MMS_HOME/tests/performance/metrics_monitoring_server.py --start\n ``` \n \n The monitoring section configuration is shown below. \n \n ```yaml\n services:\n - module: monitoring\n server-agent:\n - address: :9009 # metric monitoring service address\n label: mms-inference-server # Specified label will be used in reports instead of ip:port\n interval: 1s # polling interval\n logging: True # those logs will be saved to \"SAlogs_192.168.0.1_9009.csv\" in the artifacts dir\n metrics: # metrics should be supported by monitoring service\n - sum_cpu_percent # cpu percent used by all the mms server processes and workers\n - sum_memory_percent\n - sum_num_handles\n - server_workers # no of mms workers\n ```\n The complete yaml can be found [here](tests/examples_remote_monitoring/examples_remote_monitoring.yaml)\n \n Use the command below to run the test suite.\n \n ```bash\n export MMS_HOME=\n cd $MMS_HOME/tests/performance\n python -m run_performance_suite -p examples_remote_monitoring -e xlarge\n ```\n\n2. Local monitoring plugin\n\n Use this technique if both MMS and the tests run on the same host. \n The monitoring section configuration is shown below.\n \n ```yaml\n modules:\n server_local_monitoring:\n # metrics_monitoring_taurus and dependencies should be in python path\n class : metrics_monitoring_taurus.Monitor # monitoring class.\n \n services:\n - module: server_local_monitoring # should be added in modules section\n ServerLocalClient: # keyword from metrics_monitoring_taurus.Monitor\n - interval: 1s\n metrics:\n - cpu\n - disk-space\n - mem\n - sum_memory_percent\n \n ```\n The complete yaml can be found [here](tests/examples_local_monitoring/examples_local_monitoring.yaml).\n \n Use the command below to run the test suite.\n \n ```bash\n export MMS_HOME=\n cd $MMS_HOME/tests/performance\n python -m run_performance_suite -p examples_local_monitoring -e xlarge\n ```\n\n#### 3. Add pass/fail criteria (a.k.a test case)\n\n1. **Specify the pass/fail criteria**. Each pass-fail criterion maps to a test case in the generated report. We leverage the\npass-fail module from Taurus to achieve this functionality. More details can be found [here](https://gettaurus.org/docs/PassFail/).\n\n A sample criterion is shown below\n\n ```yaml\n reporting:\n - module: passfail\n criteria:\n - class: bzt.modules.monitoring.MonitoringCriteria\n subject: mms-inference-server/sum_num_handles\n condition: '>'\n threshold: 180\n timeframe: 1s\n fail: true\n stop: true\n \n ```\n\n2. Specify the pass/fail criterion vis-a-vis a prior run. On completion, the test suite runner script compares the \nmonitoring metrics with values from a previous run which was executed on same environment. The run results are stored \nin either a local folder or a S3 bucket based on the `compare-local` option. Metrics which have 'diff_percent' value \nspecified in the pass/fail criterion are used for comparison with the previous run. \n\n A sample criterion is shown below\n ```yaml\n reporting:\n - module: passfail\n criteria:\n - class: bzt.modules.monitoring.MonitoringCriteria\n subject: mms-inference-server/sum_num_handles\n condition: '>'\n threshold: 180\n timeframe: 1s\n fail: true\n stop: true\n diff_percent : 30\n \n ```\n Note that \n 1. At least one test suite run on the same environment should have happened in order to do the comparison.\n 2. The $artifacts-dir$//comparison_results.html is a summary report which shows performance difference \n between the last two commits.\n 3. The test case fails if the diff_percent is greater than the specified value across runs.\n\n3. Metrics available for pass-fail criteria \n \n **System Metrics**\n > disk_used, memory_percent, read_count, write_count, read_bytes, write_byte\n\n | Syntax | Examples |\n | ------ | -------- |\n | system_{metricname} | system_disk_used, system_memory_percent, system_write_count |\n\n **Process Metrics**\n > cpu_percent, memory_percent, cpu_user_time, cpu_system_time, cpu_iowait_time, memory_rss, memory_vms, io_read_count, io_write_count, io_read_bytes, io_write_bytes, file_descriptors, threads\n\n - Frontend. Represents the Java process hosting the REST APIs \n\n | Syntax | Examples |\n | ------ | -------- |\n | frontend_{metricname} | frontend_cpu_percent, frontend_memory_percent, frontend_cpu_iowait_time, frontend_memory_rss, frontend_io_write_bytes, frontend_threads |\n\n - Workers. Represents the python worker processes. Metrics for worker(s) are always available with an aggregate \n > Aggregates \n > sum, avg, min, max\n\n | Syntax | Examples |\n | ------ | -------- |\n | {aggregate}\\_workers\\_{metricname} | total_workers, sum_workers_memory_percent, avg_workers_iowait_time, min_workers_io_write_bytes, max_workers_threads |\n\n - All (Frontend + Workers). Represents aggregate metrics for both frontend and worker processes.\n \n | Syntax | Examples |\n | ------ | -------- |\n | {aggregate}\\_all\\_{metricname} | sum_all_memory_percent, avg_all_iowait_time, min_all_io_write_bytes, max_all_threads |\n\n - Miscellaneous\n * total_processes - Total number of processes spawned for frontend & workers\n * total_workers - Total number of workers spawned\n * orphans - Total number of orphan processes\n\n## Test Strategy & Cases\nMore details about our testing strategy and test cases can be found [here](TESTS.md) \n\n## FAQ\n\nQ1. Is it possible to use the performance regression framework to test MMS on Python2.7?\n\nYes. Even though, the performance regression framework needs Python 3.7+ (as Taurus requires Python 3.7+), there are two\npossible ways to achieve this\n* Please create a Python 2.7 virtual env which runs MMS and a Python 3.7 virtual env which runs \n the test framework and test cases.\n* Alternatively, deploy the standalone monitoring agent on the MMS instance and run the test cases against the remote\nserver. Note that the standalone monitoring agent works on both Python 2/3. \n\n\n\n" }, { "path": "tests/performance/TESTS.md", "content": "|CODE|Test Types |Comments |\n|----|------------------------------------------------------------------------------------|-------------------------------------------------------------------------------------------------------------------------------------------|\n|A |Operation time is deterministic for given inputs and env configuration |Model Inference Time = (Wait time for worker to be free) + (Time taken by worker to infer) |\n|B |Minimum drift in user sla and system metrics over time |Current architecture doesnt support recycling of workers. Need to check whether params specified in sheet 2 remain within acceptable bounds|\n|C |Demonstrate performance isolation across registered models |User uploaded handlers cannot cause denial of service (dos) for other model type inference |\n|D |Operations should scale linearly on node |Time to scale N workers == ( Time to scale 1 worker ) * N |\n|E |Demonstrate expected service concurrency commensurate with environment configuration|Setup environment in a way to minimize false positives/negatives |\n|F |Demonstrate preservation of performance characteristics |Do not spawn additional workers, accept model registrations if they hamper system SLAs. Is this implemented currently? |\n|G |Demonstrate cleanup releases system resources |Unregistering model should free up commensurate resources held by workers |\n|H |Demonstrate that cleanup/termination of operations should be graceful |Scale down should wait for current inference operation to succeed. Is this current behavior? |\n|I |Demonstrate that operations rollback in case request cannot be satisifed |Ongoing inference should complete before scaledown operation is allowed to start |\n|J |Demonstrate that operations are idempotent |Multiple simultaneous scale operations with the same parameter value should result in the same system state |\n\n|API|CODE|YAML|\n|---|----|---|\n|Register Model|A,B,F,J|[register_unregister](tests/register_unregister)|\n|Inference|A,B,C|[inference_single_worker](tests/inference_single_worker), [inference_multiple_worker](tests/inference_multiple_worker)|\n|Batch Inference|A,B,C|[batch_inference](tests/batch_inference)|\n|Custom Model Handlers|C|[batch_and_single_inference](tests/batch_and_single_inference)|\n|Scale Workers - UP/DOWN|D,G,I,F,J|[scale_up_workers](tests/scale_up_workers), [scale_down_workers](tests/scale_down_workers)|\n|Unregister Models|D,G,I,J|[register_unregister](tests/register_unregister)|\n|Health Check|A,B,E|[health_check](tests/health_check)|\n|API Description|A,B,E|[api_description](tests/api_description)|\n|Model Describe|A,B,E|[model_description](tests/model_description)|\n|List Models|A,B,E|[list_models](tests/list_models)|\n" }, { "path": "tests/performance/agents/__init__.py", "content": "" }, { "path": "tests/performance/agents/config.ini", "content": "[server]\npid_file = model_server.pid\n\n[monitoring]\nHOST =\nPORT = 9009\n\n[suite]\ns3_bucket = mms-performance-regression-reports" }, { "path": "tests/performance/agents/configuration.py", "content": "#!/usr/bin/env python3\n\n# Copyright 2020 Amazon.com, Inc. or its affiliates. All Rights Reserved.\n# Licensed under the Apache License, Version 2.0 (the \"License\").\n# You may not use this file except in compliance with the License.\n# A copy of the License is located at\n# http://www.apache.org/licenses/LICENSE-2.0\n# or in the \"license\" file accompanying this file. This file is distributed\n# on an \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either\n# express or implied. See the License for the specific language governing\n# permissions and limitations under the License.\n\n\"\"\"\nRead configuration file\n\"\"\"\n# pylint: disable=redefined-builtin, bare-except\nimport os\nimport configparser\nimport pathlib\n\nconfig = configparser.ConfigParser()\npath = pathlib.Path(__file__).parent.absolute()\nconfig.read(os.path.join(path, 'config.ini'))\n\n\ndef get(section, key, default=''):\n try:\n return config[section][key]\n except:\n return default\n" }, { "path": "tests/performance/agents/metrics/__init__.py", "content": "#!/usr/bin/env python3\n\"\"\" Customised system and mms process metrics for monitoring and pass-fail criteria in taurus\"\"\"\n\n# Copyright 2018 Amazon.com, Inc. or its affiliates. All Rights Reserved.\n# Licensed under the Apache License, Version 2.0 (the \"License\").\n# You may not use this file except in compliance with the License.\n# A copy of the License is located at\n# http://www.apache.org/licenses/LICENSE-2.0\n# or in the \"license\" file accompanying this file. This file is distributed\n# on an \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either\n# express or implied. See the License for the specific language governing\n# permissions and limitations under the License.\n\nfrom enum import Enum\nfrom statistics import mean\n\nimport psutil\nfrom psutil import NoSuchProcess, ZombieProcess\n\n\nclass ProcessType(Enum):\n \"\"\" Type of MMS processes to compute metrics on \"\"\"\n FRONTEND = 1\n WORKER = 2\n ALL = 3\n\n\noperators = {\n 'sum': sum,\n 'avg': mean,\n 'min': min,\n 'max': max\n}\n\nprocess_metrics = {\n # cpu\n 'cpu_percent': lambda p: p.get('cpu_percent', 0),\n 'cpu_user_time': lambda p: getattr(p.get('cpu_times', {}), 'user', 0),\n 'cpu_system_time': lambda p: getattr(p.get('cpu_times', {}), 'system', 0),\n 'cpu_iowait_time': lambda p: getattr(p.get('cpu_times', {}), 'iowait', 0),\n # memory\n 'memory_percent': lambda p: p.get('memory_percent', 0),\n 'memory_rss': lambda p: getattr(p.get('memory_info', {}), 'rss', 0),\n 'memory_vms': lambda p: getattr(p.get('memory_info', {}), 'vms', 0),\n # io\n 'io_read_count': lambda p: getattr(p.get('io_counters', {}), 'read_count', 0),\n 'io_write_count': lambda p: getattr(p.get('io_counters', {}), 'write_count', 0),\n 'io_read_bytes': lambda p: getattr(p.get('io_counters', {}), 'read_bytes', 0),\n 'io_write_bytes': lambda p: getattr(p.get('io_counters', {}), 'write_bytes', 0),\n 'file_descriptors': lambda p: p.get('num_fds', 0),\n # processes\n 'threads': lambda p: p.get('num_threads', 0)\n}\n\nsystem_metrics = {\n 'system_disk_used': None,\n 'system_memory_percent': None,\n 'system_read_count': None,\n 'system_write_count': None,\n 'system_read_bytes': None,\n 'system_write_bytes': None,\n}\n\nmisc_metrics = {\n 'total_processes': None,\n 'total_workers': None,\n 'orphans': None\n}\n\nAVAILABLE_METRICS = list(system_metrics) + list(misc_metrics)\nWORKER_NAME = 'model_service_worker.py'\n\nfor metric in list(process_metrics):\n for ptype in list(ProcessType):\n if ptype == ProcessType.WORKER:\n PNAME = 'workers'\n for op in list(operators):\n AVAILABLE_METRICS.append('{}_{}_{}'.format(op, PNAME, metric))\n elif ptype == ProcessType.FRONTEND:\n PNAME = 'frontend'\n AVAILABLE_METRICS.append('{}_{}'.format(PNAME, metric))\n else:\n PNAME = 'all'\n for op in list(operators):\n AVAILABLE_METRICS.append('{}_{}_{}'.format(op, PNAME, metric))\n\nchildren = set()\n\n\ndef get_metrics(server_process, child_processes, logger):\n \"\"\" Get Server processes specific metrics\n \"\"\"\n result = {}\n children.update(child_processes)\n logger.debug(\"children : {0}\".format(\",\".join([str(c.pid) for c in children])))\n\n def update_metric(metric_name, proc_type, stats):\n stats = list(filter(lambda x: isinstance(x, (float, int)), stats))\n stats = stats if len(stats) else [0]\n\n if proc_type == ProcessType.WORKER:\n proc_name = 'workers'\n elif proc_type == ProcessType.FRONTEND:\n proc_name = 'frontend'\n result[proc_name + '_' + metric_name] = stats[0]\n return\n else:\n proc_name = 'all'\n\n for op_name in operators:\n result['{}_{}_{}'.format(op_name, proc_name, metric_name)] = operators[op_name](stats)\n\n processes_stats = []\n reclaimed_pids = []\n\n try:\n # as_dict() gets all stats in one shot\n processes_stats.append({'type': ProcessType.FRONTEND, 'stats': server_process.as_dict()})\n except:\n pass\n for child in children:\n try:\n child_cmdline = child.cmdline()\n if psutil.pid_exists(child.pid) and len(child_cmdline) >= 2 and WORKER_NAME in child_cmdline[1]:\n processes_stats.append({'type': ProcessType.WORKER, 'stats': child.as_dict()})\n else:\n reclaimed_pids.append(child)\n logger.debug('child {0} no longer available'.format(child.pid))\n except (NoSuchProcess, ZombieProcess):\n reclaimed_pids.append(child)\n logger.debug('child {0} no longer available'.format(child.pid))\n\n for p in reclaimed_pids:\n children.remove(p)\n\n ### PROCESS METRICS ###\n worker_stats = list(map(lambda x: x['stats'], \\\n filter(lambda x: x['type'] == ProcessType.WORKER, processes_stats)))\n server_stats = list(map(lambda x: x['stats'], \\\n filter(lambda x: x['type'] == ProcessType.FRONTEND, processes_stats)))\n all_stats = list(map(lambda x: x['stats'], processes_stats))\n\n for k in process_metrics:\n update_metric(k, ProcessType.WORKER, list(map(process_metrics[k], worker_stats)))\n update_metric(k, ProcessType.ALL, list(map(process_metrics[k], all_stats)))\n update_metric(k, ProcessType.FRONTEND, list(map(process_metrics[k], server_stats)))\n\n # Total processes\n result['total_processes'] = len(worker_stats) + 1\n result['total_workers'] = max(len(worker_stats) - 1, 0)\n result['orphans'] = len(list(filter(lambda p: p['ppid'] == 1, worker_stats)))\n\n ### SYSTEM METRICS ###\n result['system_disk_used'] = psutil.disk_usage('/').used\n result['system_memory_percent'] = psutil.virtual_memory().percent\n system_disk_io_counters = psutil.disk_io_counters()\n result['system_read_count'] = system_disk_io_counters.read_count\n result['system_write_count'] = system_disk_io_counters.write_count\n result['system_read_bytes'] = system_disk_io_counters.read_bytes\n result['system_write_bytes'] = system_disk_io_counters.write_bytes\n\n return result\n" }, { "path": "tests/performance/agents/metrics_collector.py", "content": "#!/usr/bin/env python3\n\n# Copyright 2020 Amazon.com, Inc. or its affiliates. All Rights Reserved.\n# Licensed under the Apache License, Version 2.0 (the \"License\").\n# You may not use this file except in compliance with the License.\n# A copy of the License is located at\n# http://www.apache.org/licenses/LICENSE-2.0\n# or in the \"license\" file accompanying this file. This file is distributed\n# on an \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either\n# express or implied. See the License for the specific language governing\n# permissions and limitations under the License.\n\n\"\"\"\nServer metrics collector\n\"\"\"\n# pylint: disable=redefined-builtin, broad-except, unused-variable\n\nimport argparse\nimport logging\nimport os\nimport sys\nimport tempfile\nimport time\nimport gevent\nimport psutil\n\nfrom utils.process import get_process_pid_from_file, get_child_processes, \\\n get_server_processes, get_server_pidfile\nfrom metrics import AVAILABLE_METRICS, get_metrics\nimport configuration\n\nlogger = logging.getLogger(__name__)\nlogging.basicConfig(stream=sys.stdout, format=\"%(message)s\", level=logging.INFO)\n\nTMP_DIR = tempfile.gettempdir()\nMETRICS_LOG_FILE = os.path.join(TMP_DIR, \"server_metrics_{}.log\".format(int(time.time())))\nMETRICS_COLLECTOR_PID_FILE = os.path.join(TMP_DIR, \"metrics_collector.pid\")\nPID_FILE = configuration.get('server', 'pid_file', 'model_server.pid')\n\nMONITOR_INTERVAL = 1\n\n\ndef store_pid(pid_file):\n \"\"\" Store the current process id to pid_file\"\"\"\n process = psutil.Process()\n pid_file = os.path.join(pid_file)\n with open(pid_file, \"w\") as pf:\n pf.write(str(process.pid))\n\n\ndef stop_process(pid_file):\n \"\"\"This will stop already running process .\n Note at a time only one pid file will be available.\n \"\"\"\n pid = get_process_pid_from_file(pid_file)\n if pid:\n try:\n process = psutil.Process(pid)\n if process.is_running():\n logger.info(\"Process with pid %s is running. Killing it.\", process.pid)\n process.kill()\n except Exception as e:\n pass\n else:\n logger.info(\"Dead process with pid %s found in '%s'.\", process.pid, pid_file)\n\n logger.info(\"Removing pid file '%s'.\", pid_file)\n os.remove(pid_file)\n\n\ndef check_is_running(pid_file):\n \"\"\"check if pid is running\"\"\"\n pid = get_process_pid_from_file(pid_file)\n if pid:\n try:\n perf_mon_process = psutil.Process(pid)\n except Exception as e:\n stop_process(pid_file)\n else:\n if perf_mon_process.is_running():\n logger.error(\"Performance monitoring script already running. \"\n \"Stop it using stop option.\")\n sys.exit()\n\n\ndef store_metrics_collector_pid():\n \"\"\" Store the process id of metrics collector process\"\"\"\n store_pid(METRICS_COLLECTOR_PID_FILE)\n\n\ndef stop_metrics_collector_process():\n \"\"\"This will stop already running metrics collector process.\n Note at a time only one pid file will be available.\n \"\"\"\n stop_process(METRICS_COLLECTOR_PID_FILE)\n\n\ndef monitor_processes(server_process, metrics, interval, socket):\n \"\"\" Monitor the metrics of server_process and its child processes\n \"\"\"\n while True:\n message = []\n collected_metrics = get_metrics(server_process, get_child_processes(server_process), logger)\n metrics_msg = []\n for metric in metrics:\n message.append(str(collected_metrics.get(metric, 0)))\n if collected_metrics.get(metric) is not None:\n metrics_msg.append(\"{0} : {1}\".format(metric, collected_metrics.get(metric, 0)))\n\n message = \"\\t\".join(message) + \"\\t\\n\"\n logger.info(\"%s\", \" -- \".join(metrics_msg))\n\n if socket:\n try:\n socket.send(message.encode(\"latin-1\"))\n except BrokenPipeError:\n logger.info(\"Stopping monitoring as socket connection is closed.\")\n break\n\n # TODO - log metrics to a file METRICS_LOG_FILE if METRICS_LOG_FILE is provided\n gevent.sleep(interval)\n\n\ndef start_metric_collection(server_process, metrics, interval, socket):\n bad_metrics = set(metrics) - set(AVAILABLE_METRICS)\n if bad_metrics:\n raise Exception(\"Metrics not available for monitoring {}.\".format(bad_metrics))\n\n logger.info(\"Started metric collection for target server processes.....\")\n thread = gevent.spawn(monitor_processes, server_process, metrics, interval, socket)\n gevent.joinall([thread])\n\n\ndef start_metric_collector_process():\n \"\"\"Spawn a metric collection process and keep on monitoring \"\"\"\n\n check_is_running(METRICS_COLLECTOR_PID_FILE)\n store_metrics_collector_pid()\n server_pid = get_process_pid_from_file(get_server_pidfile(PID_FILE))\n server_process = get_server_processes(server_pid)\n start_metric_collection(server_process, AVAILABLE_METRICS, MONITOR_INTERVAL, None)\n\n\nif __name__ == \"__main__\":\n logging.basicConfig(stream=sys.stdout, format=\"%(message)s\", level=logging.INFO)\n parser = argparse.ArgumentParser(prog='metric-collector', description='System Performance Metrics collector')\n sub_parse = parser.add_mutually_exclusive_group(required=True)\n sub_parse.add_argument('--start', action='store_true', help='Start the metric-collector')\n sub_parse.add_argument('--stop', action='store_true', help='Stop the metric-collector')\n\n args = parser.parse_args()\n\n if args.start:\n start_metric_collector_process()\n elif args.stop:\n stop_metrics_collector_process()\n" }, { "path": "tests/performance/agents/metrics_monitoring_inproc.py", "content": "#!/usr/bin/env python3\n\n# Copyright 2020 Amazon.com, Inc. or its affiliates. All Rights Reserved.\n# Licensed under the Apache License, Version 2.0 (the \"License\").\n# You may not use this file except in compliance with the License.\n# A copy of the License is located at\n# http://www.apache.org/licenses/LICENSE-2.0\n# or in the \"license\" file accompanying this file. This file is distributed\n# on an \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either\n# express or implied. See the License for the specific language governing\n# permissions and limitations under the License.\n\n\"\"\"\nTaurus Local plugin for server monitoring.\nShould be used when server and Taurus are running on same machine.\nThis file should be placed in Python Path along with monitoring package.\n\"\"\"\n# pylint: disable=redefined-builtin, unnecessary-comprehension\n\nimport csv\nimport sys\n\nfrom bzt import TaurusConfigError\nfrom bzt.modules import monitoring\nfrom bzt.utils import dehumanize_time\n\nimport configuration\nfrom metrics import get_metrics, AVAILABLE_METRICS as AVAILABLE_SERVER_METRICS\nfrom utils.process import get_process_pid_from_file, get_server_processes, \\\n get_child_processes, get_server_pidfile\n\n\nPY2 = sys.version_info[0] == 2\nPY3 = sys.version_info[0] == 3\nPID_FILE = configuration.get('server', 'pid_file', 'model_server.pid')\n\n\nclass Monitor(monitoring.Monitoring):\n \"\"\"Add ServerLocalClient to Monitoring by patching to monitoring.Monitoring\n \"\"\"\n\n def __init__(self):\n super(Monitor, self).__init__()\n self.client_classes.update({'ServerLocalClient': ServerLocalClient})\n\n\nclass ServerLocalClient(monitoring.LocalClient):\n \"\"\"Custom server local client \"\"\"\n\n AVAILABLE_METRICS = monitoring.LocalClient.AVAILABLE_METRICS + \\\n AVAILABLE_SERVER_METRICS\n\n def __init__(self, parent_log, label, config, engine=None):\n\n super(ServerLocalClient, self).__init__(parent_log, label, config, engine=engine)\n if label:\n self.label = label\n else:\n self.label = 'ServerLocalClient'\n\n def connect(self):\n exc = TaurusConfigError('Metric is required in Local monitoring client')\n metric_names = self.config.get('metrics', exc)\n\n bad_list = set(metric_names) - set(self.AVAILABLE_METRICS)\n if bad_list:\n self.log.warning('Wrong metrics found: %s', bad_list)\n\n good_list = set(metric_names) & set(self.AVAILABLE_METRICS)\n if not good_list:\n raise exc\n\n self.metrics = list(set(good_list))\n\n self.monitor = ServerLocalMonitor(self.log, self.metrics, self.engine)\n self.interval = dehumanize_time(self.config.get(\"interval\", self.engine.check_interval))\n\n if self.config.get(\"logging\", False):\n if not PY3:\n self.log.warning(\"Logging option doesn't work on python2.\")\n else:\n self.logs_file = self.engine.create_artifact(\"local_monitoring_logs\", \".csv\")\n with open(self.logs_file, \"a\", newline='') as mon_logs:\n logs_writer = csv.writer(mon_logs, delimiter=',')\n metrics = ['ts'] + sorted([metric for metric in good_list])\n logs_writer.writerow(metrics)\n\n\nclass ServerLocalMonitor(monitoring.LocalMonitor):\n \"\"\"Custom server local monitor\"\"\"\n\n def _calc_resource_stats(self, interval):\n result = super()._calc_resource_stats(interval)\n server_pid = get_process_pid_from_file(get_server_pidfile(PID_FILE))\n server_process = get_server_processes(server_pid)\n result.update(get_metrics(server_process, get_child_processes(server_process), self.log))\n\n metrics_msg = []\n\n updated_result = {}\n for key in self.metrics:\n if result.get(key) is not None:\n metrics_msg.append(\"{0} : {1}\".format(key, result[key]))\n updated_result[key] = result.get(key)\n self.log.info(\"{0}\".format(\" -- \".join(metrics_msg)))\n return updated_result\n" }, { "path": "tests/performance/agents/metrics_monitoring_server.py", "content": "#!/usr/bin/env python3\n\n# Copyright 2020 Amazon.com, Inc. or its affiliates. All Rights Reserved.\n# Licensed under the Apache License, Version 2.0 (the \"License\").\n# You may not use this file except in compliance with the License.\n# A copy of the License is located at\n# http://www.apache.org/licenses/LICENSE-2.0\n# or in the \"license\" file accompanying this file. This file is distributed\n# on an \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either\n# express or implied. See the License for the specific language governing\n# permissions and limitations under the License.\n\n\"\"\"\nRemote server monitoring script\n\"\"\"\n# pylint: disable=redefined-builtin, wrong-import-position, too-many-nested-blocks, broad-except\n\nimport argparse\nimport logging\nimport sys\nimport tempfile\nimport os\n\nfrom gevent import monkey\nfrom gevent import select\nfrom gevent import socket\n\nmonkey.patch_select()\nmonkey.patch_socket()\n\nfrom metrics_collector import start_metric_collection, stop_process, store_pid, check_is_running\nfrom utils.process import get_process_pid_from_file, \\\n get_server_processes, get_server_pidfile\nimport configuration\n\n\nlogger = logging.getLogger(__name__)\nlogging.basicConfig(stream=sys.stdout, format=\"%(message)s\", level=logging.INFO)\nTMP_DIR = tempfile.gettempdir()\nMETRICS_MON_SERVER_PID_FILE = os.path.join(TMP_DIR, \".metrics_monitoring_server.pid\")\nPID_FILE = configuration.get('server', 'pid_file', 'model_server.pid')\n\nHOST = str(configuration.get('monitoring', 'HOST'))\nPORT = int(configuration.get('monitoring', 'PORT', 9009))\n\nSOCKET_LIST = []\nRECV_BUFFER = 4096\ninterval = 1\n\n\ndef process_data(sock):\n \"\"\" process data recieved on socket\"\"\"\n # receiving data from the socket.\n data = sock.recv(RECV_BUFFER).decode()\n if data:\n if data == 'test\\n':\n send_message(sock, \"Yep\\n\")\n elif data == 'exit\\n':\n close_socket(sock)\n elif data.startswith('interval'):\n try:\n global interval\n interval = int(data.split(\":\")[1][:-1])\n except Exception:\n send_message(sock, \"In-correct interval data\")\n elif data.startswith('metrics'):\n metrics = data[:-1].split(\"metrics:\")[1].split(\"\\t\")\n server_pid = get_process_pid_from_file(get_server_pidfile(PID_FILE))\n server_process = get_server_processes(server_pid)\n start_metric_collection(server_process, metrics, interval, sock)\n else:\n # TODO - decide what to do here\n pass\n else:\n # remove the socket that's broken\n if sock in SOCKET_LIST:\n SOCKET_LIST.remove(sock)\n\ndef perf_server():\n \"\"\" start performance moniting server on a socket \"\"\"\n server_socket = socket.socket(socket.AF_INET, socket.SOCK_STREAM)\n server_socket.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)\n server_socket.bind((HOST, PORT))\n server_socket.listen(10)\n SOCKET_LIST.append(server_socket)\n logger.info(\"Started metrics monitoring server on port %s\", PORT)\n\n while True:\n ready_to_read, _, _ = select.select(SOCKET_LIST, [], [], 0)\n\n for sock in ready_to_read:\n # a new connection request recieved\n if sock == server_socket:\n sockfd, addr = server_socket.accept()\n SOCKET_LIST.append(sockfd)\n logger.info(\"client (%s, %s) connected\", addr[0], addr[1])\n\n # a message from a client, not a new connection\n else:\n try:\n process_data(sock)\n except Exception as e:\n logger.warning(\"Error %s\", str(e))\n continue\n\n server_socket.close()\n\n\ndef send_message(socket_, message):\n try:\n socket_.send(message.encode(\"latin-1\"))\n except Exception as e:\n logger.warning(\"Error while sending the message %s. Closing the socket.\", str(e))\n close_socket(socket_)\n\n\ndef close_socket(socket_):\n socket_.close()\n if socket_ in SOCKET_LIST:\n SOCKET_LIST.remove(socket_)\n\n\nif __name__ == \"__main__\":\n logging.basicConfig(stream=sys.stdout, format=\"%(message)s\", level=logging.INFO)\n parser = argparse.ArgumentParser(prog='perf-mon-script', description='System Performance Monitoring')\n sub_parse = parser.add_mutually_exclusive_group(required=True)\n sub_parse.add_argument('--start', action='store_true', help='Start the perf-mon-script')\n sub_parse.add_argument('--stop', action='store_true', help='Stop the perf-mon-script')\n\n args = parser.parse_args()\n\n if args.start:\n check_is_running(METRICS_MON_SERVER_PID_FILE)\n store_pid(METRICS_MON_SERVER_PID_FILE)\n perf_server()\n elif args.stop:\n stop_process(METRICS_MON_SERVER_PID_FILE)\n" }, { "path": "tests/performance/agents/utils/__init__.py", "content": "" }, { "path": "tests/performance/agents/utils/process.py", "content": "#!/usr/bin/env python3\n\n# Copyright 2020 Amazon.com, Inc. or its affiliates. All Rights Reserved.\n# Licensed under the Apache License, Version 2.0 (the \"License\").\n# You may not use this file except in compliance with the License.\n# A copy of the License is located at\n# http://www.apache.org/licenses/LICENSE-2.0\n# or in the \"license\" file accompanying this file. This file is distributed\n# on an \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either\n# express or implied. See the License for the specific language governing\n# permissions and limitations under the License.\n\n\"\"\"\nUtility methods for process related information\n\"\"\"\n# pylint: disable=redefined-builtin\n\nimport os\nimport tempfile\n\nimport psutil\n\n\ndef find_procs_by_name(name):\n \"\"\"Return a list of processes matching 'name'.\"\"\"\n ls = []\n for p in psutil.process_iter([\"name\", \"exe\", \"cmdline\"]):\n if name == p.info['name'] or \\\n p.info['exe'] and os.path.basename(p.info['exe']) == name or \\\n p.info['cmdline'] and p.info['cmdline'][0] == name:\n ls.append(p)\n\n if len(ls) > 1:\n raise Exception(\"Multiple processes found with name {}.\".format(name))\n\n return ls[0]\n\n\ndef get_process_pid_from_file(file_path):\n \"\"\"Get the process pid from pid file.\n \"\"\"\n pid = None\n if os.path.isfile(file_path):\n with open(file_path, \"r\") as f:\n pid = int(f.readline())\n\n return pid\n\n\ndef get_child_processes(process):\n \"\"\"Get all running child processes recursively\"\"\"\n child_processes = set()\n for p in process.children(recursive=True):\n child_processes.add(p)\n return child_processes\n\n\ndef get_server_processes(server_process_pid):\n \"\"\" It caches the main server and child processes at module level.\n Ensure that you call this process so that MMS process\n \"\"\"\n try:\n server_process = psutil.Process(server_process_pid)\n except Exception as e:\n print(\"Server process not found. Error: {}\".format(str(e)))\n raise\n return server_process\n\n\ndef get_server_pidfile(file):\n return os.path.join(tempfile.gettempdir(), \".{}\".format(file))\n" }, { "path": "tests/performance/pylintrc", "content": "[MASTER]\n\n# Licensed to the Apache Software Foundation (ASF) under one\n# or more contributor license agreements. See the NOTICE file\n# distributed with this work for additional information\n# regarding copyright ownership. The ASF licenses this file\n# to you under the Apache License, Version 2.0 (the\n# \"License\"); you may not use this file except in compliance\n# with the License. You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing,\n# software distributed under the License is distributed on an\n# \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n# KIND, either express or implied. See the License for the\n# specific language governing permissions and limitations\n# under the License.\n\n# Specify a configuration file.\n#rcfile=\n\n# Python code to execute, usually for sys.path manipulation such as\n# pygtk.require().\n#init-hook=\n\n# Add files or directories to the blacklist. They should be base names, not\n# paths.\nignore=CVS\n\n# Add files or directories matching the regex patterns to the blacklist. The\n# regex matches against base names, not paths.\nignore-patterns=\n\n# Pickle collected data for later comparisons.\npersistent=yes\n\n# List of plugins (as comma separated values of python modules names) to load,\n# usually to register additional checkers.\nload-plugins=\n\n# Use multiple processes to speed up Pylint.\njobs=8\n\n# Allow loading of arbitrary C extensions. Extensions are imported into the\n# active Python interpreter and may run arbitrary code.\nunsafe-load-any-extension=no\n\n# A comma-separated list of package or module names from where C extensions may\n# be loaded. Extensions are loading into the active Python interpreter and may\n# run arbitrary code\nextension-pkg-whitelist=numpy,opencv\n\n# Allow optimization of some AST trees. This will activate a peephole AST\n# optimizer, which will apply various small optimizations. For instance, it can\n# be used to obtain the result of joining multiple strings with the addition\n# operator. Joining a lot of strings can lead to a maximum recursion error in\n# Pylint and this flag can prevent that. It has one side effect, the resulting\n# AST will be different than the one from reality. This option is deprecated\n# and it will be removed in Pylint 2.0.\noptimize-ast=no\n\n\n[MESSAGES CONTROL]\n\n# Only show warnings with the listed confidence levels. Leave empty to show\n# all. Valid levels: HIGH, INFERENCE, INFERENCE_FAILURE, UNDEFINED\nconfidence=\n\n# Enable the message, report, category or checker with the given id(s). You can\n# either give multiple identifier separated by comma (,) or put this option\n# multiple time (only on the command line, not in the configuration file where\n# it should appear only once). See also the \"--disable\" option for examples.\nenable=indexing-exception,old-raise-syntax\n\n# Disable the message, report, category or checker with the given id(s). You\n# can either give multiple identifiers separated by comma (,) or put this\n# option multiple times (only on the command line, not in the configuration\n# file where it should appear only once).You can also use \"--disable=all\" to\n# disable everything first and then reenable specific checks. For example, if\n# you want to run only the similarities checker, you can use \"--disable=all\n# --enable=similarities\". If you want to run only the classes checker, but have\n# no Warning level messages displayed, use\"--disable=all --enable=classes\n# --disable=W\"\ndisable=design,similarities,no-self-use,attribute-defined-outside-init,locally-disabled,star-args,pointless-except,bad-option-value,global-statement,fixme,suppressed-message,useless-suppression,locally-enabled,no-member,no-name-in-module,import-error,unsubscriptable-object,unbalanced-tuple-unpacking,undefined-variable,protected-access,superfluous-parens,invalid-name,no-else-return,useless-super-delegation,len-as-condition,invalid-unary-operand-type,useless-object-inheritance\n# disable=unicode-builtin,delslice-method,using-cmp-argument,setslice-method,dict-view-method,parameter-unpacking,range-builtin-not-iterating,print-statement,file-builtin,old-raise-syntax,basestring-builtin,execfile-builtin,indexing-exception,import-star-module-level,coerce-method,long-builtin,old-ne-operator,old-division,no-absolute-import,raw_input-builtin,old-octal-literal,oct-method,xrange-builtin,hex-method,unpacking-in-except,nonzero-method,raising-string,intern-builtin,reload-builtin,metaclass-assignment,cmp-method,filter-builtin-not-iterating,apply-builtin,map-builtin-not-iterating,next-method-called,unichr-builtin,buffer-builtin,dict-iter-method,input-builtin,coerce-builtin,getslice-method,useless-suppression,standarderror-builtin,zip-builtin-not-iterating,suppressed-message,cmp-builtin,backtick,long-suffix,reduce-builtin,round-builtin\n\n\n[REPORTS]\n\n# Set the output format. Available formats are text, parseable, colorized, msvs\n# (visual studio) and html. You can also give a reporter class, eg\n# mypackage.mymodule.MyReporterClass.\noutput-format=text\n\n# Put messages in a separate file for each module / package specified on the\n# command line instead of printing them on stdout. Reports (if any) will be\n# written in a file name \"pylint_global.[txt|html]\". This option is deprecated\n# and it will be removed in Pylint 2.0.\nfiles-output=no\n\n# Tells whether to display a full report or only the messages\nreports=no\n\n# Python expression which should return a note less than 10 (10 is the highest\n# note). You have access to the variables errors warning, statement which\n# respectively contain the number of errors / warnings messages and the total\n# number of statements analyzed. This is used by the global evaluation report\n# (RP0004).\nevaluation=10.0 - ((float(5 * error + warning + refactor + convention) / statement) * 10)\n\n# Template used to display messages. This is a python new-style format string\n# used to format the message information. See doc for all details\n#msg-template=\n\n\n[FORMAT]\n\n# Maximum number of characters on a single line.\nmax-line-length=120\n\n# Regexp for a line that is allowed to be longer than the limit.\nignore-long-lines=^\\s*(# )??$\n\n# Allow the body of an if to be on the same line as the test if there is no\n# else.\nsingle-line-if-stmt=no\n\n# List of optional constructs for which whitespace checking is disabled. `dict-\n# separator` is used to allow tabulation in dicts, etc.: {1 : 1,\\n222: 2}.\n# `trailing-comma` allows a space between comma and closing bracket: (a, ).\n# `empty-line` allows space-only lines.\nno-space-check=trailing-comma,dict-separator\n\n# Maximum number of lines in a module\nmax-module-lines=1000\n\n# String used as indentation unit. This is usually \" \" (4 spaces) or \"\\t\" (1\n# tab).\nindent-string=' '\n\n# Number of spaces of indent required inside a hanging or continued line.\nindent-after-paren=4\n\n# Expected format of line ending, e.g. empty (any line ending), LF or CRLF.\nexpected-line-ending-format=\n\n\n[SPELLING]\n\n# Spelling dictionary name. Available dictionaries: none. To make it working\n# install python-enchant package.\nspelling-dict=\n\n# List of comma separated words that should not be checked.\nspelling-ignore-words=\n\n# A path to a file that contains private dictionary; one word per line.\nspelling-private-dict-file=\n\n# Tells whether to store unknown words to indicated private dictionary in\n# --spelling-private-dict-file option instead of raising a message.\nspelling-store-unknown-words=no\n\n\n[MISCELLANEOUS]\n\n# List of note tags to take in consideration, separated by a comma.\nnotes=FIXME,XXX,TODO\n\n\n[TYPECHECK]\n\n# Tells whether missing members accessed in mixin class should be ignored. A\n# mixin class is detected if its name ends with \"mixin\" (case insensitive).\nignore-mixin-members=yes\n\n# List of module names for which member attributes should not be checked\n# (useful for modules/projects where namespaces are manipulated during runtime\n# and thus existing member attributes cannot be deduced by static analysis. It\n# supports qualified module names, as well as Unix pattern matching.\nignored-modules=\n\n# List of class names for which member attributes should not be checked (useful\n# for classes with dynamically set attributes). This supports the use of\n# qualified names.\nignored-classes=optparse.Values,thread._local,_thread._local\n\n# List of members which are set dynamically and missed by pylint inference\n# system, and so shouldn't trigger E1101 when accessed. Python regular\n# expressions are accepted.\ngenerated-members=\n\n# List of decorators that produce context managers, such as\n# contextlib.contextmanager. Add to this list to register other decorators that\n# produce valid context managers.\ncontextmanager-decorators=contextlib.contextmanager\n\n\n[LOGGING]\n\n# Logging modules to check that the string format arguments are in logging\n# function parameter format\nlogging-modules=logging\n\n\n[SIMILARITIES]\n\n# Minimum lines number of a similarity.\nmin-similarity-lines=4\n\n# Ignore comments when computing similarities.\nignore-comments=yes\n\n# Ignore docstrings when computing similarities.\nignore-docstrings=yes\n\n# Ignore imports when computing similarities.\nignore-imports=no\n\n\n[VARIABLES]\n\n# Tells whether we should check for unused import in __init__ files.\ninit-import=no\n\n# A regular expression matching the name of dummy variables (i.e. expectedly\n# not used).\ndummy-variables-rgx=(_+[a-zA-Z0-9]*?$)|dummy\n\n# List of additional names supposed to be defined in builtins. Remember that\n# you should avoid to define new builtins when possible.\nadditional-builtins=\n\n# List of strings which can identify a callback function by name. A callback\n# name must start or end with one of those strings.\ncallbacks=cb_,_cb\n\n# List of qualified module names which can have objects that can redefine\n# builtins.\nredefining-builtins-modules=six.moves,future.builtins,builtins\n\n\n[BASIC]\n\n# Good variable names which should always be accepted, separated by a comma\ngood-names=i,j,_,a,b,op,x,y,wd,lr,kv,k,v,s,p,h,c,m,n,X,t,g,f\n\n# Bad variable names which should always be refused, separated by a comma\nbad-names=\n\n# Colon-delimited sets of names that determine each other's naming style when\n# the name regexes allow several styles.\nname-group=\n\n# Include a hint for the correct naming format with invalid-name\ninclude-naming-hint=no\n\n# List of decorators that produce properties, such as abc.abstractproperty. Add\n# to this list to register other decorators that produce valid properties.\nproperty-classes=abc.abstractproperty\n\n# Regular expression matching correct module names\nmodule-rgx=(([a-z_][a-z0-9_]*)|([A-Z][a-zA-Z0-9]+))$\n\n# Naming hint for module names\nmodule-name-hint=(([a-z_][a-z0-9_]*)|([A-Z][a-zA-Z0-9]+))$\n\n# Regular expression matching correct constant names\nconst-rgx=(([A-Z_][A-Z0-9_]*)|(__.*__))$\n\n# Naming hint for constant names\nconst-name-hint=(([A-Z_][A-Z0-9_]*)|(__.*__))$\n\n# Regular expression matching correct inline iteration names\ninlinevar-rgx=[A-Za-z_][A-Za-z0-9_]*$\n\n# Naming hint for inline iteration names\ninlinevar-name-hint=[A-Za-z_][A-Za-z0-9_]*$\n\n# Regular expression matching correct method names\nmethod-rgx=[a-z_][a-z0-9_]{2,30}$\n\n# Naming hint for method names\nmethod-name-hint=[a-z_][a-z0-9_]{2,30}$\n\n# Regular expression matching correct class attribute names\nclass-attribute-rgx=([A-Za-z_][A-Za-z0-9_]{2,30}|(__.*__))$\n\n# Naming hint for class attribute names\nclass-attribute-name-hint=([A-Za-z_][A-Za-z0-9_]{2,30}|(__.*__))$\n\n# Regular expression matching correct argument names\nargument-rgx=[a-z_][a-z0-9_]{2,30}$\n\n# Naming hint for argument names\nargument-name-hint=[a-z_][a-z0-9_]{2,30}$\n\n# Regular expression matching correct attribute names\nattr-rgx=[a-z_][a-z0-9_]{2,30}$\n\n# Naming hint for attribute names\nattr-name-hint=[a-z_][a-z0-9_]{2,30}$\n\n# Regular expression matching correct variable names\nvariable-rgx=[a-z_][a-z0-9_]{2,30}$\n\n# Naming hint for variable names\nvariable-name-hint=[a-z_][a-z0-9_]{2,30}$\n\n# Regular expression matching correct function names\nfunction-rgx=[a-z_][a-z0-9_]{2,30}$\n\n# Naming hint for function names\nfunction-name-hint=[a-z_][a-z0-9_]{2,30}$\n\n# Regular expression matching correct class names\nclass-rgx=[A-Z_][a-zA-Z0-9]+$\n\n# Naming hint for class names\nclass-name-hint=[A-Z_][a-zA-Z0-9]+$\n\n# Regular expression which should only match function or class names that do\n# not require a docstring.\nno-docstring-rgx=^_\n\n# Minimum line length for functions/classes that require docstrings, shorter\n# ones are exempt.\ndocstring-min-length=10\n\n\n[ELIF]\n\n# Maximum number of nested blocks for function / method body\nmax-nested-blocks=5\n\n\n[CLASSES]\n\n# List of method names used to declare (i.e. assign) instance attributes.\ndefining-attr-methods=__init__,__new__,setUp\n\n# List of valid names for the first argument in a class method.\nvalid-classmethod-first-arg=cls\n\n# List of valid names for the first argument in a metaclass class method.\nvalid-metaclass-classmethod-first-arg=mcs\n\n# List of member names, which should be excluded from the protected access\n# warning.\nexclude-protected=_asdict,_fields,_replace,_source,_make\n\n\n[IMPORTS]\n\n# Deprecated modules which should not be used, separated by a comma\ndeprecated-modules=optparse\n\n# Create a graph of every (i.e. internal and external) dependencies in the\n# given file (report RP0402 must not be disabled)\nimport-graph=\n\n# Create a graph of external dependencies in the given file (report RP0402 must\n# not be disabled)\next-import-graph=\n\n# Create a graph of internal dependencies in the given file (report RP0402 must\n# not be disabled)\nint-import-graph=\n\n# Force import order to recognize a module as part of the standard\n# compatibility libraries.\nknown-standard-library=\n\n# Force import order to recognize a module as part of a third party library.\nknown-third-party=enchant\n\n# Analyse import fallback blocks. This can be used to support both Python 2 and\n# 3 compatible code, which means that the block might have code that exists\n# only in one or another interpreter, leading to false positives when analysed.\nanalyse-fallback-blocks=no\n\n\n[DESIGN]\n\n# Maximum number of arguments for function / method\nmax-args=5\n\n# Argument names that match this expression will be ignored. Default to name\n# with leading underscore\nignored-argument-names=_.*\n\n# Maximum number of locals for function / method body\nmax-locals=15\n\n# Maximum number of return / yield for function / method body\nmax-returns=6\n\n# Maximum number of branch for function / method body\nmax-branches=12\n\n# Maximum number of statements in function / method body\nmax-statements=50\n\n# Maximum number of parents for a class (see R0901).\nmax-parents=7\n\n# Maximum number of attributes for a class (see R0902).\nmax-attributes=7\n\n# Minimum number of public methods for a class (see R0903).\nmin-public-methods=2\n\n# Maximum number of public methods for a class (see R0904).\nmax-public-methods=20\n\n# Maximum number of boolean expressions in a if statement\nmax-bool-expr=5\n\n\n[EXCEPTIONS]\n\n# Exceptions that will emit a warning when being caught. Defaults to\n# \"Exception\"\novergeneral-exceptions=Exception\n" }, { "path": "tests/performance/requirements.txt", "content": "gevent==20.5.2\njunitparser==1.4.1\ngit+https://github.com/maheshambule/vjunit.git#egg=vjunit\ntqdm==4.40.0\npathlib==1.0.1\nboto3==1.14.3\nawscli==1.18.80\nclick==7.1.2\ntabulate==0.8.7\npandas==1.0.3\ntermcolor==1.1.0" }, { "path": "tests/performance/run_performance_suite.py", "content": "\n\n# Copyright 2020 Amazon.com, Inc. or its affiliates. All Rights Reserved.\n# Licensed under the Apache License, Version 2.0 (the \"License\").\n# You may not use this file except in compliance with the License.\n# A copy of the License is located at\n# http://www.apache.org/licenses/LICENSE-2.0\n# or in the \"license\" file accompanying this file. This file is distributed\n# on an \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either\n# express or implied. See the License for the specific language governing\n# permissions and limitations under the License.\n\n\"\"\"\nRun Performance Regression Test Cases and Generate Reports\n\"\"\"\n# pylint: disable=redefined-builtin, no-value-for-parameter\n\nimport logging\nimport os\nimport subprocess\nimport sys\nimport time\n\nimport click\nimport pathlib\nfrom runs.context import ExecutionEnv\nfrom runs.taurus import get_taurus_options, x2junit, update_taurus_metric_files\nfrom tqdm import tqdm\n\nfrom utils import run_process, Timer, get_sub_dirs\n\nlogger = logging.getLogger(__name__)\nlogging.basicConfig(stream=sys.stdout, format=\"%(message)s\", level=logging.INFO)\n\nROOT_PATH = pathlib.Path(__file__).parent.absolute()\nRUN_ARTIFACTS_PATH = os.path.join(ROOT_PATH, \"run_artifacts\")\nGLOBAL_CONFIG_PATH = os.path.join(ROOT_PATH, \"tests\", \"global_config.yaml\")\nMONITORING_AGENT = os.path.join(ROOT_PATH, \"agents\", \"metrics_monitoring_server.py\")\n\n\ndef get_artifacts_dir(ctx, param, value):\n commit_id = subprocess.check_output('git rev-parse --short HEAD'.split()).decode(\"utf-8\")[:-1]\n run_name = \"{}__{}__{}\".format(ctx.params['env_name'], commit_id, int(time.time()))\n if value is None:\n artifacts_dir = os.path.join(RUN_ARTIFACTS_PATH, run_name)\n else:\n artifacts_dir = os.path.abspath(value)\n artifacts_dir = os.path.join(artifacts_dir, run_name)\n return artifacts_dir\n\n\ndef validate_env(ctx, param, value):\n try:\n if '__' in value:\n raise ValueError\n return value\n except ValueError:\n raise click.BadParameter('Environment name should not have double underscores in it.')\n\n\n@click.command()\n@click.option('-a', '--artifacts-dir', help='Directory to store artifacts.', type=click.Path(writable=True),\n callback=get_artifacts_dir)\n@click.option('-t', '--test-dir', help='Directory containing tests.', type=click.Path(exists=True),\n default=os.path.join(ROOT_PATH, \"tests\"))\n@click.option('-p', '--pattern', help='Test case folder name glob pattern', default=\"*\")\n@click.option('-x', '--exclude-pattern', help='Test case folder name glob pattern to exclude', default=None)\n@click.option('-j', '--jmeter-path', help='JMeter executable path.')\n@click.option('-e', '--env-name', help='Environment filename without the extension. Contains threshold values.',\n required=True, callback=validate_env)\n@click.option('--monit/--no-monit', help='Start Monitoring server', default=True)\n@click.option('--compare-local/--no-compare-local', help='Compare with previous run with files stored'\n ' in artifacts directory', default=True)\ndef run_test_suite(artifacts_dir, test_dir, pattern, exclude_pattern,\n jmeter_path, env_name, monit, compare_local):\n \"\"\"Collect test suites, run them and generate reports\"\"\"\n\n logger.info(\"Artifacts will be stored in directory %s\", artifacts_dir)\n test_dirs = get_sub_dirs(test_dir, exclude_list=[], include_pattern=pattern,\n exclude_pattern=exclude_pattern)\n if not test_dirs:\n logger.info(\"No test cases are collected...Exiting.\")\n sys.exit(3)\n else:\n logger.info(\"Collected tests %s\", test_dirs)\n\n with ExecutionEnv(MONITORING_AGENT, artifacts_dir, env_name, compare_local, monit) as prt:\n pre_command = 'export PYTHONPATH={}:$PYTHONPATH;'.format(os.path.join(str(ROOT_PATH), \"agents\"))\n for suite_name in tqdm(test_dirs, desc=\"Test Suites\"):\n with Timer(\"Test suite {} execution time\".format(suite_name)) as t:\n suite_artifacts_dir = os.path.join(artifacts_dir, suite_name)\n options_str = get_taurus_options(suite_artifacts_dir, jmeter_path)\n env_yaml_path = os.path.join(test_dir, suite_name, \"environments\", \"{}.yaml\".format(env_name))\n env_yaml_path = \"\" if not os.path.exists(env_yaml_path) else env_yaml_path\n test_file = os.path.join(test_dir, suite_name, \"{}.yaml\".format(suite_name))\n with x2junit.X2Junit(suite_name, suite_artifacts_dir, prt.reporter, t, env_name) as s:\n s.code, s.err = run_process(\"{} bzt {} {} {} {}\".format(pre_command, options_str,\n test_file, env_yaml_path,\n GLOBAL_CONFIG_PATH))\n\n update_taurus_metric_files(suite_artifacts_dir, test_file)\n\nif __name__ == \"__main__\":\n run_test_suite()\n" }, { "path": "tests/performance/runs/__init__.py", "content": "" }, { "path": "tests/performance/runs/compare.py", "content": "#!/usr/bin/env python\n\n# Copyright 2020 Amazon.com, Inc. or its affiliates. All Rights Reserved.\n# Licensed under the Apache License, Version 2.0 (the \"License\").\n# You may not use this file except in compliance with the License.\n# A copy of the License is located at\n# http://www.apache.org/licenses/LICENSE-2.0\n# or in the \"license\" file accompanying this file. This file is distributed\n# on an \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either\n# express or implied. See the License for the specific language governing\n# permissions and limitations under the License.\n\n\"\"\"\nCompare artifacts between runs\n\"\"\"\n# pylint: disable=redefined-builtin, self-assigning-variable, broad-except\n\n\nimport csv\nimport glob\nimport logging\nimport sys\nimport os\n\nimport pandas as pd\nfrom junitparser import TestCase, TestSuite, JUnitXml, Skipped, Error, Failure\nfrom runs.taurus import reader as taurus_reader\nfrom runs.storage import LocalStorage, S3Storage\n\nfrom utils import Timer, get_sub_dirs\n\nlogger = logging.getLogger(__name__)\nlogging.basicConfig(stream=sys.stdout, format=\"%(message)s\", level=logging.INFO)\n\n\nclass CompareReportGenerator():\n\n def __init__(self, path, env_name, local_run):\n self.artifacts_dir = path\n self.current_run_name = os.path.basename(path)\n self.env_name = env_name\n storage_class = LocalStorage if local_run else S3Storage\n self.storage = storage_class(self.artifacts_dir, self.env_name)\n self.junit_reporter = None\n self.pandas_result = None\n self.pass_fail = True\n\n def gen(self):\n \"\"\"Driver method to get comparison directory, do the comparison of it with current run directory\n and then store results\n \"\"\"\n compare_dir, compare_run_name = self.storage.get_dir_to_compare()\n if compare_run_name:\n self.junit_reporter, self.pandas_result = compare_artifacts(self.storage.artifacts_dir, compare_dir,\n self.storage.current_run_name, compare_run_name)\n self.pandas_result.to_csv(os.path.join(self.artifacts_dir, \"comparison_result.csv\"))\n else:\n logger.warning(\"The latest run not found for env.\")\n\n self.storage.store_results()\n return self.junit_reporter\n\n\nclass CompareTestSuite():\n \"\"\"\n Wrapper helper class over JUnit parser Test Suite\n \"\"\"\n\n result_types = {\"pass\": [lambda x: None, \"tests\"],\n \"fail\": [Failure, \"failures\"],\n \"error\": [Error, \"errors\"],\n \"skip\": [Skipped, \"skipped\"]}\n\n def __init__(self, name, hostname, t):\n self.ts = TestSuite(name)\n self.ts.errors, self.ts.failures, self.ts.skipped, self.ts.tests = 0, 0, 0, 0\n self.ts.hostname = hostname\n self.ts.timestamp = t.start\n\n def add_test_case(self, name, msg, type):\n tc = TestCase(name)\n result_type = CompareTestSuite.result_types[type]\n tc.result = result_type[0](msg)\n self.ts.add_testcase(tc)\n setattr(self.ts, result_type[1], getattr(self.ts, result_type[1]) + 1)\n\n\ndef get_log_file(dir, sub_dir):\n \"\"\"Get metric monitoring log files\"\"\"\n metrics_file = os.path.join(dir, sub_dir, \"metrics.csv\")\n return metrics_file if os.path.exists(metrics_file) else None\n\n\ndef get_aggregate_val(df, agg_func, col):\n \"\"\"Get aggregate values of a pandas datframe coulmn for given aggregate function\"\"\"\n val = None\n if str(col) in df:\n try:\n val = float(getattr(df[str(col)], agg_func)())\n except TypeError:\n val = None\n return val\n\n\ndef compare_values(val1, val2, diff_percent, run_name1, run_name2):\n \"\"\" Compare percentage diff values of val1 and val2 \"\"\"\n if pd.isna(val1) or pd.isna(val2):\n msg = \"Either of the value can not be determined. The run1 value is '{}' and \" \\\n \"run2 value is {}.\".format(val1, val2)\n pass_fail, diff, msg = \"error\", \"NA\", msg\n else:\n try:\n if val2 != val1:\n diff = (abs(val2 - val1) / ((val2 + val1) / 2)) * 100\n\n if diff < float(diff_percent):\n pass_fail, diff, msg = \"pass\", diff, \"passed\"\n else:\n msg = \"The diff_percent criteria has failed. The expected diff_percent is '{}' and actual \" \\\n \"diff percent is '{}' and the '{}' run value is '{}' and '{}' run value is '{}'. \". \\\n format(diff_percent, diff, run_name1, val1, run_name2, val2)\n\n pass_fail, diff, msg = \"fail\", diff, msg\n else: # special case of 0\n pass_fail, diff, msg = \"pass\", 0, \"\"\n except Exception as e:\n msg = \"error while calculating the diff for val1={} and val2={}.\" \\\n \"Error is: {}\".format(val1, val2, str(e))\n logger.info(msg)\n pass_fail, diff, msg = \"pass\", \"NA\", msg\n\n return diff, pass_fail, msg\n\n\ndef compare_artifacts(dir1, dir2, run_name1, run_name2):\n \"\"\"Compare artifacts from dir1 with di2 and store results in out_dir\"\"\"\n\n logger.info(\"Comparing artifacts from %s with %s\", dir1, dir2)\n sub_dirs_1 = get_sub_dirs(dir1)\n\n over_all_pass = True\n aggregates = [\"mean\", \"max\", \"min\"]\n header = [\"run_name1\", \"run_name2\", \"test_suite\", \"metric\", \"run1\", \"run2\",\n \"percentage_diff\", \"expected_diff\", \"result\", \"message\"]\n rows = [header]\n\n reporter = JUnitXml()\n for sub_dir1 in sub_dirs_1:\n with Timer(\"Comparison test suite {} execution time\".format(sub_dir1)) as t:\n comp_ts = CompareTestSuite(sub_dir1, run_name1 + \" and \" + run_name1, t)\n\n metrics_file1, metrics_file2 = get_log_file(dir1, sub_dir1), get_log_file(dir2, sub_dir1)\n if not (metrics_file1 and metrics_file2):\n msg = \"Metrics monitoring logs are not captured for {} in either \" \\\n \"of the runs.\".format(sub_dir1)\n logger.info(msg)\n rows.append([run_name1, run_name2, sub_dir1, \"metrics_log_file_availability\",\n \"NA\", \"NA\", \"NA\", \"NA\", \"pass\", msg])\n comp_ts.add_test_case(\"metrics_log_file_availability\", msg, \"skip\")\n continue\n\n metrics_from_file1 = pd.read_csv(metrics_file1)\n metrics_from_file2 = pd.read_csv(metrics_file2)\n metrics, diff_percents = taurus_reader.get_compare_metric_list(dir1, sub_dir1)\n\n for col, diff_percent in zip(metrics, diff_percents):\n for agg_func in aggregates:\n name = \"{}_{}\".format(agg_func, str(col))\n\n val1 = get_aggregate_val(metrics_from_file1, agg_func, col)\n val2 = get_aggregate_val(metrics_from_file2, agg_func, col)\n\n diff, pass_fail, msg = compare_values(val1, val2, diff_percent, run_name1, run_name2)\n\n if over_all_pass:\n over_all_pass = pass_fail == \"pass\"\n\n result_row = [run_name1, run_name2, sub_dir1, name, val1, val2,\n diff, diff_percent, pass_fail, msg]\n rows.append(result_row)\n test_name = \"{}: diff_percent < {}\".format(name, diff_percent)\n comp_ts.add_test_case(test_name, msg, pass_fail)\n\n comp_ts.ts.time = t.diff()\n comp_ts.ts.update_statistics()\n reporter.add_testsuite(comp_ts.ts)\n\n dataframe = pd.DataFrame(rows[1:], columns=rows[0])\n return reporter, dataframe\n\n" }, { "path": "tests/performance/runs/context.py", "content": "#!/usr/bin/env python\n\n# Copyright 2020 Amazon.com, Inc. or its affiliates. All Rights Reserved.\n# Licensed under the Apache License, Version 2.0 (the \"License\").\n# You may not use this file except in compliance with the License.\n# A copy of the License is located at\n# http://www.apache.org/licenses/LICENSE-2.0\n# or in the \"license\" file accompanying this file. This file is distributed\n# on an \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either\n# express or implied. See the License for the specific language governing\n# permissions and limitations under the License.\n\n\"\"\"\nStart and stop monitoring server\n\"\"\"\n# pylint: disable=redefined-builtin\n\nimport logging\nimport os\nimport sys\nimport time\nimport webbrowser\nfrom termcolor import colored\n\nfrom junitparser import JUnitXml\nfrom runs.compare import CompareReportGenerator\nfrom runs.junit import JunitConverter, junit2tabulate\n\nfrom utils import run_process\n\nlogger = logging.getLogger(__name__)\nlogging.basicConfig(stream=sys.stdout, format=\"%(message)s\", level=logging.INFO)\n\n\nclass ExecutionEnv(object):\n \"\"\"\n Context Manager class to run the performance regression suites\n \"\"\"\n\n def __init__(self, agent, artifacts_dir, env, local_run, use=True, check_mms_server_status=False):\n self.monitoring_agent = agent\n self.artifacts_dir = artifacts_dir\n self.use = use\n self.env = env\n self.local_run = local_run\n self.check_mms_server_status = check_mms_server_status\n self.reporter = JUnitXml()\n self.compare_reporter_generator = CompareReportGenerator(self.artifacts_dir, self.env, self.local_run)\n\n def __enter__(self):\n if self.use:\n start_monitoring_server = \"{} {} --start\".format(sys.executable, self.monitoring_agent)\n run_process(start_monitoring_server, wait=False)\n time.sleep(2)\n return self\n\n @staticmethod\n def open_report(file_path):\n if os.path.exists(file_path):\n return webbrowser.open_new_tab('file://' + os.path.realpath(file_path))\n return False\n\n @staticmethod\n def report_summary(reporter, suite_name):\n if reporter and os.path.exists(reporter.junit_html_path):\n status = reporter.junit_xml.errors or reporter.junit_xml.failures or reporter.junit_xml.skipped\n status, code, color = (\"failed\", 3, \"red\") if status else (\"passed\", 0, \"green\")\n\n msg = \"{} run has {}.\".format(suite_name, status)\n logger.info(colored(msg, color, attrs=['reverse', 'blink']))\n logger.info(\"%s report - %s\", suite_name, reporter.junit_html_path)\n logger.info(\"%s summary:\", suite_name)\n print(junit2tabulate(reporter.junit_xml))\n ExecutionEnv.open_report(reporter.junit_html_path)\n return code\n\n else:\n msg = \"{} run report is not generated.\".format(suite_name)\n logger.info(colored(msg, \"yellow\", attrs=['reverse', 'blink']))\n return 0\n\n def __exit__(self, type, value, traceback):\n if self.use:\n stop_monitoring_server = \"{} {} --stop\".format(sys.executable, self.monitoring_agent)\n run_process(stop_monitoring_server)\n\n junit_reporter = JunitConverter(self.reporter, self.artifacts_dir, 'performance_results')\n junit_reporter.generate_junit_report()\n junit_compare = self.compare_reporter_generator.gen()\n junit_compare_reporter = None\n if junit_compare:\n junit_compare_reporter = JunitConverter(junit_compare, self.artifacts_dir, 'comparison_results')\n junit_compare_reporter.generate_junit_report()\n\n compare_exit_code = ExecutionEnv.report_summary(junit_compare_reporter, \"Comparison Test suite\")\n exit_code = ExecutionEnv.report_summary(junit_reporter, \"Performance Regression Test suite\")\n\n sys.exit(0 if 0 == exit_code == compare_exit_code else 3)\n" }, { "path": "tests/performance/runs/junit.py", "content": "#!/usr/bin/env python\n\n# Copyright 2020 Amazon.com, Inc. or its affiliates. All Rights Reserved.\n# Licensed under the Apache License, Version 2.0 (the \"License\").\n# You may not use this file except in compliance with the License.\n# A copy of the License is located at\n# http://www.apache.org/licenses/LICENSE-2.0\n# or in the \"license\" file accompanying this file. This file is distributed\n# on an \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either\n# express or implied. See the License for the specific language governing\n# permissions and limitations under the License.\n\n\"\"\"\nStart and stop monitoring server\n\"\"\"\n# pylint: disable=redefined-builtin\n\nimport os\nimport html\nimport textwrap\nimport tabulate\nfrom utils import run_process\nfrom junitparser import JUnitXml\n\nheader = [\"suite_name\", \"test_case\", \"result\", \"message\"]\n\n\nclass JunitConverter():\n\n def __init__(self, junit_xml, out_dir, report_name):\n self.junit_xml = junit_xml\n self.junit_xml_path = os.path.join(out_dir, '{}.xml'.format(report_name))\n self.junit_html_path = os.path.join(out_dir, '{}.html'.format(report_name))\n\n def generate_junit_report(self):\n self.junit_xml.update_statistics()\n self.junit_xml.write(self.junit_xml_path)\n # vjunit pip package is used here\n run_process(\"vjunit -f {} -o {}\".format(self.junit_xml_path, self.junit_html_path))\n\n\ndef pretty_text(data):\n \"\"\"Unsescape the html characters from the data & wrap it\"\"\"\n if data is not None:\n return textwrap.fill(html.unescape(html.unescape(data)), width=60)\n else:\n return \"\"\n\n\ndef junit2array(junit_xml):\n \"\"\"convert junit xml junitparser.JUnitXml object to 2d array \"\"\"\n rows = [header]\n for i, suite in enumerate(junit_xml):\n if len(suite) == 0:\n rows.append([suite.name, \"\", \"skipped\",\n \"No criteria specified or there is an error.\"])\n else:\n for case in suite:\n result = case.result\n tag, msg = (result._tag, result.message) if result else (\"passed\", \"\")\n rows.append([suite.name, pretty_text(case.name), tag, pretty_text(msg)])\n\n return rows\n\n\ndef junit2tabulate(junit_xml):\n \"\"\"convert junit xml junitparser.JUnitXml object or a Junit xml to tabulate string \"\"\"\n if not isinstance(junit_xml, JUnitXml):\n if os.path.exists(junit_xml):\n junit_xml = JUnitXml.fromfile(junit_xml)\n else:\n return tabulate.tabulate([[header]], headers='firstrow')\n data = junit2array(junit_xml)\n return tabulate.tabulate(data, headers='firstrow', tablefmt=\"grid\")\n" }, { "path": "tests/performance/runs/storage.py", "content": "#!/usr/bin/env python\n\n# Copyright 2020 Amazon.com, Inc. or its affiliates. All Rights Reserved.\n# Licensed under the Apache License, Version 2.0 (the \"License\").\n# You may not use this file except in compliance with the License.\n# A copy of the License is located at\n# http://www.apache.org/licenses/LICENSE-2.0\n# or in the \"license\" file accompanying this file. This file is distributed\n# on an \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either\n# express or implied. See the License for the specific language governing\n# permissions and limitations under the License.\n\n\"\"\"\nResult store classes\n\"\"\"\n# pylint: disable=redefined-builtin\n\n\nimport logging\nimport os\nimport sys\nimport shutil\n\nimport boto3\nimport pathlib\nfrom agents import configuration\n\nfrom utils import run_process\n\nlogger = logging.getLogger(__name__)\nlogging.basicConfig(stream=sys.stdout, format=\"%(message)s\", level=logging.INFO)\nS3_BUCKET = configuration.get('suite', 's3_bucket')\n\n\nclass Storage():\n \"\"\"Class to store and retrieve artifacts\"\"\"\n\n def __init__(self, path, env_name):\n self.artifacts_dir = path\n self.current_run_name = os.path.basename(path)\n self.env_name = env_name\n\n def get_dir_to_compare(self):\n \"\"\"get the artifacts dir to compare to\"\"\"\n\n def store_results(self):\n \"\"\"Store the results\"\"\"\n\n @staticmethod\n def get_latest(names, env_name, exclude_name):\n \"\"\"\n Get latest directory for same env_name name given a list of them.\n :param names: list of folder names in the format env_name___commitid__timestamp\n :param env_name: filter for env_name\n :param exclude_name: any name to exclude\n :return: latest directory name\n \"\"\"\n max_ts = 0\n latest_run = ''\n for run_name in names:\n run_name_list = run_name.split('__')\n if env_name == run_name_list[0] and run_name != exclude_name:\n if int(run_name_list[2]) > max_ts:\n max_ts = int(run_name_list[2])\n latest_run = run_name\n\n return latest_run\n\n\nclass LocalStorage(Storage):\n \"\"\"\n Compare the monitoring metrics for current and previous run for the same env_name\n \"\"\"\n\n def get_dir_to_compare(self):\n \"\"\"Get latest run directory name to be compared with\"\"\"\n parent_dir = pathlib.Path(self.artifacts_dir).parent\n names = [di for di in os.listdir(parent_dir) if os.path.isdir(os.path.join(parent_dir, di))]\n latest_run = self.get_latest(names, self.env_name, self.current_run_name)\n return os.path.join(parent_dir, latest_run), latest_run\n\n\nclass S3Storage(Storage):\n \"\"\"Compare current run results with the results stored on S3\"\"\"\n\n def get_dir_to_compare(self):\n \"\"\"Get latest run result artifacts directory for same env_name from S3 bucket\n and store it locally for further comparison\n \"\"\"\n comp_data_path = os.path.join(self.artifacts_dir, \"comp_data\")\n s3 = boto3.resource('s3')\n bucket = s3.Bucket(S3_BUCKET)\n result = bucket.meta.client.list_objects(Bucket=bucket.name,\n Delimiter='/')\n run_names = []\n for o in result.get('CommonPrefixes'):\n run_names.append(o.get('Prefix')[:-1])\n\n latest_run = self.get_latest(run_names, self.env_name, self.current_run_name)\n if not latest_run:\n logger.info(\"No run found for env_id %s\", self.env_name)\n return '', ''\n\n if not os.path.exists(comp_data_path):\n os.makedirs(comp_data_path)\n\n tgt_path = os.path.join(comp_data_path, latest_run)\n run_process(\"aws s3 cp s3://{}/{} {} --recursive\".format(bucket.name, latest_run, tgt_path))\n\n return tgt_path, latest_run\n\n def store_results(self):\n \"\"\"Store the run results back to S3\"\"\"\n comp_data_path = os.path.join(self.artifacts_dir, \"comp_data\")\n if os.path.exists(comp_data_path):\n shutil.rmtree(comp_data_path)\n\n run_process(\"aws s3 cp {} s3://{}/{} --recursive\".format(self.artifacts_dir, S3_BUCKET,\n self.current_run_name))\n" }, { "path": "tests/performance/runs/taurus/__init__.py", "content": "#!/usr/bin/env python\n\n# Copyright 2020 Amazon.com, Inc. or its affiliates. All Rights Reserved.\n# Licensed under the Apache License, Version 2.0 (the \"License\").\n# You may not use this file except in compliance with the License.\n# A copy of the License is located at\n# http://www.apache.org/licenses/LICENSE-2.0\n# or in the \"license\" file accompanying this file. This file is distributed\n# on an \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either\n# express or implied. See the License for the specific language governing\n# permissions and limitations under the License.\n\n\"\"\"\nConvert the Taurus Test suite XML to Junit XML\n\"\"\"\n# pylint: disable=redefined-builtin\n\nimport glob\nimport shutil\nimport os\n\nfrom .reader import get_mon_metrics_list\n\n\ndef get_taurus_options(artifacts_dir, jmeter_path=None):\n \"\"\"The options for Taurus BZT command\"\"\"\n options = []\n if jmeter_path:\n options.append('-o modules.jmeter.path={}'.format(jmeter_path))\n options.append('-o settings.artifacts-dir={}'.format(artifacts_dir))\n options.append('-o modules.console.disable=true')\n options.append('-o settings.env.BASEDIR={}'.format(artifacts_dir))\n options_str = ' '.join(options)\n\n return options_str\n\n\ndef update_taurus_metric_files(suite_artifacts_dir, test_file):\n \"\"\"\n It renames the server and local metric monitoring log files to metrics.csv.\n The order of the columns in header of server metric monitoring SALogs file generated by taurus\n is not inline with data. So as a work around this function rewrites the header based on order\n defined in the test yaml.\n \"\"\"\n metrics_new_file = os.path.join(suite_artifacts_dir, \"metrics.csv\")\n\n server_metric_file_pattern = os.path.join(suite_artifacts_dir, \"SAlogs_*\")\n metrics_log_file = glob.glob(server_metric_file_pattern)\n if metrics_log_file:\n metrics = get_mon_metrics_list(test_file)\n if metrics:\n with open(metrics_log_file[0]) as from_file:\n line = from_file.readline()\n with open(metrics_log_file[0], mode=\"w\") as to_file:\n to_file.write(','.join(line.split(',')[0:1] + metrics) + \"\\n\")\n shutil.copyfileobj(from_file, to_file)\n\n os.rename(metrics_log_file[0], metrics_new_file)\n\n else:\n metrics_log_file = os.path.join(suite_artifacts_dir, \"local_monitoring_logs.csv\")\n if os.path.exists(metrics_log_file):\n os.rename(metrics_log_file, metrics_new_file)\n" }, { "path": "tests/performance/runs/taurus/reader.py", "content": "#!/usr/bin/env python\n\n# Copyright 2020 Amazon.com, Inc. or its affiliates. All Rights Reserved.\n# Licensed under the Apache License, Version 2.0 (the \"License\").\n# You may not use this file except in compliance with the License.\n# A copy of the License is located at\n# http://www.apache.org/licenses/LICENSE-2.0\n# or in the \"license\" file accompanying this file. This file is distributed\n# on an \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either\n# express or implied. See the License for the specific language governing\n# permissions and limitations under the License.\n\n\"\"\"\nRun shell command utilities\n\"\"\"\n# pylint: disable=redefined-builtin\n\nimport os\n\nimport yaml\n\n\ndef get_mon_metrics_list(test_yaml_path):\n \"\"\"Utility method to get list of server-agent metrics which are being monitored from a test yaml file\"\"\"\n metrics = []\n with open(test_yaml_path) as test_yaml:\n test_yaml = yaml.safe_load(test_yaml)\n for rep_section in test_yaml.get('services', []):\n if rep_section.get('module', None) == 'monitoring' and \"server-agent\" in rep_section:\n for mon_section in rep_section.get('server-agent', []):\n if isinstance(mon_section, dict):\n metrics.extend(mon_section.get('metrics', []))\n\n return metrics\n\n\ndef get_compare_metric_list(dir, sub_dir):\n \"\"\"Utility method to get list of compare monitoring metrics identified by diff_percent property\"\"\"\n diff_percents = []\n metrics = []\n test_yaml = os.path.join(dir, sub_dir, \"effective.yml\")\n with open(test_yaml) as test_yaml:\n test_yaml = yaml.safe_load(test_yaml)\n for rep_section in test_yaml.get('reporting', []):\n if rep_section.get('module', None) == 'passfail':\n for criterion in rep_section.get('criteria', []):\n if isinstance(criterion, dict) and 'monitoring' in criterion.get('class', ''):\n subject = criterion[\"subject\"]\n metric = subject.rsplit('/', 1)\n metric = metric[1] if len(metric) == 2 else metric[0]\n diff_percent = criterion.get(\"diff_percent\", None)\n\n if diff_percent:\n metrics.append(metric)\n diff_percents.append(diff_percent)\n\n return metrics, diff_percents\n" }, { "path": "tests/performance/runs/taurus/x2junit.py", "content": "#!/usr/bin/env python\n\n# Copyright 2020 Amazon.com, Inc. or its affiliates. All Rights Reserved.\n# Licensed under the Apache License, Version 2.0 (the \"License\").\n# You may not use this file except in compliance with the License.\n# A copy of the License is located at\n# http://www.apache.org/licenses/LICENSE-2.0\n# or in the \"license\" file accompanying this file. This file is distributed\n# on an \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either\n# express or implied. See the License for the specific language governing\n# permissions and limitations under the License.\n\n\"\"\"\nConvert the Taurus Test suite XML to Junit XML\n\"\"\"\n# pylint: disable=redefined-builtin\n\n\nimport os\n\nfrom junitparser import TestCase, TestSuite, JUnitXml, Skipped, Error, Failure\n\n\nclass X2Junit(object):\n \"\"\"\n Context Manager class to do convert Taurus Test suite XML report which is in Xunit specifications\n to JUnit XML report.\n \"\"\"\n def __init__(self, name, artifacts_dir, junit_xml, timer, env_name):\n self.ts = TestSuite(name)\n self.name = name\n self.junit_xml = junit_xml\n self.timer = timer\n self.artifacts_dir = artifacts_dir\n self.env_name = env_name\n\n def __enter__(self):\n return self\n\n def __exit__(self, type, value, traceback):\n xunit_file = os.path.join(self.artifacts_dir, \"xunit.xml\")\n tests, failures, skipped, errors = 0, 0, 0, 0\n if os.path.exists(xunit_file):\n xml = JUnitXml.fromfile(xunit_file)\n for i, suite in enumerate(xml):\n for case in suite:\n name = \"scenario_{}: {}\".format(i, case.name)\n result = case.result\n if isinstance(result, Error):\n failures += 1\n result = Failure(result.message, result.type)\n elif isinstance(result, Failure):\n errors += 1\n result = Error(result.message, result.type)\n elif isinstance(result, Skipped):\n skipped += 1\n else:\n tests += 1\n\n tc = TestCase(name)\n tc.result = result\n self.ts.add_testcase(tc)\n else:\n tc = TestCase(self.name)\n tc.result = Skipped()\n self.ts.add_testcase(tc)\n\n self.ts.hostname = self.env_name\n self.ts.timestamp = self.timer.start\n self.ts.time = self.timer.diff()\n self.ts.tests = tests\n self.ts.failures = failures\n self.ts.skipped = skipped\n self.ts.errors = errors\n self.ts.update_statistics()\n self.junit_xml.add_testsuite(self.ts)\n" }, { "path": "tests/performance/tests/api_description/api_description.jmx", "content": "\n\n \n \n \n false\n true\n false\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n ${__P(hostname,127.0.0.1)}\n ${__P(port,8443)}\n ${__P(protocol,https)}\n \n \n 6\n \n \n \n \n \n continue\n \n false\n ${__P(loops,4)}\n \n ${__P(threads,10)}\n ${__P(rampup,10)}\n false\n \n \n true\n \n \n \n \n \n \n \n ${__P(management_port,8444)}\n \n \n \n OPTIONS\n true\n false\n true\n false\n \n \n \n \n \n \n \n \n \n \n ${__P(port,8443)}\n \n \n \n OPTIONS\n true\n false\n true\n false\n \n \n \n \n \n \n \n \n\n" }, { "path": "tests/performance/tests/api_description/api_description.yaml", "content": "---\nexecution:\n- concurrency: 10\n ramp-up: 1s\n hold-for: 30s\n scenario: api_description\n\nscenarios:\n api_description:\n script: api_description.jmx\n\nmodules:\n server_local_monitoring:\n class : metrics_monitoring_inproc.Monitor\n\nservices:\n - module: shellexec\n prepare:\n - \"multi-model-server --start > /dev/null 2>&1\"\n - \"sleep 10s\"\n post-process:\n - \"multi-model-server --stop > /dev/null 2>&1\"\n - module: server_local_monitoring\n ServerLocalClient:\n - interval: 1s\n logging : True\n metrics:\n - total_processes\n - sum_all_file_descriptors\n - sum_all_memory_rss\n\nreporting:\n- module: passfail\n criteria:\n # Inbuilt Criteria\n - success of ManagementAPIDescription<${MGMT_DESC_SUCC}, stop as failed\n - success of InferenceAPIDescription<${INFR_DESC_SUCC}, stop as failed\n - avg-rt of ManagementAPIDescription>${MGMT_DESC_RT}, stop as failed\n - avg-rt of InferenceAPIDescription>${INFR_DESC_RT}, stop as failed\n # Custom Criteria\n - class: bzt.modules.monitoring.MonitoringCriteria\n subject: ServerLocalClient/total_processes\n condition: '>'\n threshold: ${TOTAL_PROCS}\n timeframe: 1s\n stop : true\n fail : true\n - class: bzt.modules.monitoring.MonitoringCriteria\n subject: ServerLocalClient/total_processes\n condition: '<'\n threshold: ${TOTAL_PROCS}\n timeframe: 1s\n stop : true\n fail : true\n - class: bzt.modules.monitoring.MonitoringCriteria\n subject: ServerLocalClient/sum_all_file_descriptors\n condition: '>'\n threshold: ${TOTAL_FDS}\n timeframe: 5s\n stop : true\n fail : true\n# - class: bzt.modules.monitoring.MonitoringCriteria\n# subject: ServerLocalClient/sum_all_memory_rss\n# condition: '>'\n# threshold: ${TOTAL_MEM}\n# timeframe: 5s\n# stop : true\n# fail : true" }, { "path": "tests/performance/tests/api_description/environments/xlarge.yaml", "content": "---\nsettings:\n env:\n MGMT_DESC_SUCC: 100%\n INFR_DESC_SUCC: 100%\n MGMT_DESC_RT : 10ms\n INFR_DESC_RT : 10ms\n TOTAL_PROCS : 1\n TOTAL_FDS : 73\n TOTAL_MEM: 100000000 #100MB\n" }, { "path": "tests/performance/tests/batch_and_single_inference/batch_and_single_inference.jmx", "content": "\n\n \n \n \n false\n true\n false\n \n \n \n \n \n \n \n \n \n model1\n ${__P(model_name1,resnet-152)}\n =\n \n \n model2\n ${__P(model_name2,squeezenet_v1.1)}\n =\n \n \n \n \n \n \n \n \n ${__P(hostname,127.0.0.1)}\n ${__P(port,8443)}\n ${__P(protocol,https)}\n \n \n 6\n \n \n \n \n \n continue\n \n false\n ${__P(loops,200)}\n \n ${__P(threads,20)}\n ${__P(rampup,5)}\n false\n \n \n true\n \n \n \n \n \n \n ${__P(input_filepath)}\n data\n image/jpeg\n \n \n \n \n \n \n \n \n \n \n /predictions/${model1}\n POST\n true\n false\n true\n true\n \n \n \n \n \n \n \n \n \n ${__P(input_filepath)}\n data\n image/jpeg\n \n \n \n \n \n \n \n \n \n \n /predictions/${model2}\n POST\n true\n false\n true\n true\n \n \n \n \n \n \n \n \n\n" }, { "path": "tests/performance/tests/batch_and_single_inference/batch_and_single_inference.yaml", "content": "---\nexecution:\n- concurrency: 10\n ramp-up: 5s\n hold-for: 20s\n scenario: Inference\n\nscenarios:\n Inference:\n script: batch_and_single_inference.jmx\n\nmodules:\n server_local_monitoring:\n # metrics_monitoring_inproc and dependencies should be in python path\n class : metrics_monitoring_inproc.Monitor # monitoring class.\n\nservices:\n - module: shellexec\n prepare:\n - \"curl -s -O https://s3.amazonaws.com/model-server/inputs/kitten.jpg\"\n - \"multi-model-server --start > /dev/null 2>&1\"\n - \"sleep 20s\"\n - \"curl -s -X POST http://localhost:8081/models?url=https://s3.amazonaws.com/model-server/model_archive_1.0/examples/resnet-152-batching/resnet-152.mar&batch_size=8&max_batch_delay=50\"\n # uncomment below and comment prev and use downloaded model with model-store\n #- curl -s -X POST \"http://localhost:8081/models?url=resnet-152.mar&batch_size=8&max_batch_delay=60&initial_workers=1\"\n - \"curl -s -X PUT http://localhost:8081/models/resnet-152?min_worker=2&synchronous=true\"\n - \"curl -s -X POST http://localhost:8081/models?url=https://s3.amazonaws.com/model-server/model_archive_1.0/squeezenet_v1.1.mar\"\n - \"curl -s -X PUT http://localhost:8081/models/squeezenet_v1.1?min_worker=2&synchronous=true\"\n post-process:\n - \"multi-model-server --stop > /dev/null 2>&1\"\n - \"rm kitten.jpg\"\n - module: server_local_monitoring # should be added in modules section\n ServerLocalClient: # keyword from metrics_monitoring_inproc.Monitor\n - interval: 1s\n logging : True\n metrics:\n - sum_workers_memory_rss\n - sum_workers_file_descriptors\n - total_workers\n - orphans\n\nreporting:\n- module: passfail\n criteria:\n - subject: avg-rt # required\n label: 'Inference1' # optional, default is ''\n condition: '>' # required\n threshold: ${INFR1_RT} # required\n logic: for # optional, logic to aggregate values within timeframe.\n # Default 'for' means take latest,\n # 'within' and 'over' means take sum/avg of all values within interval\n timeframe: 1s # optional, default is none\n stop: true # optional, default is true. false for nonstop testing until the end\n fail: true # optional, default is true\n - subject: avg-rt # required\n label: 'Inference2' # optional, default is ''\n condition: '>' # required\n threshold: ${INFR2_RT} # required\n logic: for # optional, logic to aggregate values within timeframe.\n # Default 'for' means take latest,\n # 'within' and 'over' means take sum/avg of all values within interval\n timeframe: 1s # optional, default is none\n stop: true # optional, default is true. false for nonstop testing until the end\n fail: true # optional, default is true\n - class: bzt.modules.monitoring.MonitoringCriteria\n subject: ServerLocalClient/sum_workers_memory_rss\n condition: '>'\n threshold: ${TOTAL_WORKERS_MEM}\n timeframe: 1s\n stop : true\n fail : true\n diff_percent : 30\n - class: bzt.modules.monitoring.MonitoringCriteria\n subject: ServerLocalClient/orphans\n condition: '>'\n threshold: ${TOTAL_ORPHANS}\n timeframe: 1s\n stop : true\n fail : true\n diff_percent : 0\n - class: bzt.modules.monitoring.MonitoringCriteria\n subject: ServerLocalClient/total_workers\n condition: '>'\n threshold: ${TOTAL_WORKERS}\n timeframe: 1s\n stop: true\n fail: true\n diff_percent: 0\n - class: bzt.modules.monitoring.MonitoringCriteria\n subject: ServerLocalClient/sum_workers_file_descriptors\n condition: '>'\n threshold: ${TOTAL_WORKERS_FDS}\n timeframe: 1s\n stop: true\n fail: true\n diff_percent: 30\n" }, { "path": "tests/performance/tests/batch_and_single_inference/environments/xlarge.yaml", "content": "---\nsettings:\n env:\n INFR1_RT : 6s\n INFR2_RT : 0.08s\n TOTAL_WORKERS_MEM : 4000000000 #4GB\n TOTAL_WORKERS : 9\n TOTAL_ORPHANS : 0\n TOTAL_WORKERS_FDS : 78\n" }, { "path": "tests/performance/tests/batch_inference/batch_inference.jmx", "content": "\n\n \n \n \n false\n true\n false\n \n \n \n \n \n \n \n \n \n model\n ${__P(model_name,resnet-152)}\n =\n \n \n \n \n \n \n \n \n ${__P(hostname,127.0.0.1)}\n ${__P(port,8443)}\n ${__P(protocol,https)}\n \n \n 6\n \n \n \n \n \n continue\n \n false\n ${__P(loops,200)}\n \n ${__P(threads,20)}\n ${__P(rampup,5)}\n false\n \n \n true\n \n \n \n \n \n \n ${__P(input_filepath)}\n data\n image/jpeg\n \n \n \n \n \n \n \n \n \n \n /predictions/${model}\n POST\n true\n false\n true\n true\n \n \n \n \n \n \n \n \n\n" }, { "path": "tests/performance/tests/batch_inference/batch_inference.yaml", "content": "---\nexecution:\n- concurrency: 10\n ramp-up: 5s\n hold-for: 20s\n scenario: Inference\n\nscenarios:\n Inference:\n script: batch_inference.jmx\n\nmodules:\n server_local_monitoring:\n # metrics_monitoring_inproc and dependencies should be in python path\n class : metrics_monitoring_inproc.Monitor # monitoring class.\n\nservices:\n - module: shellexec\n prepare:\n - \"curl -s -O https://s3.amazonaws.com/model-server/inputs/kitten.jpg\"\n - \"multi-model-server --start > /dev/null 2>&1\"\n - \"sleep 20s\"\n - \"curl -s -X POST http://localhost:8081/models?url=https://s3.amazonaws.com/model-server/model_archive_1.0/examples/resnet-152-batching/resnet-152.mar&batch_size=8&max_batch_delay=50\"\n # uncomment below and comment prev and use downloaded model with model-store\n #- \"curl -s -X POST http://localhost:8081/models?url=resnet-152.mar&batch_size=8&max_batch_delay=60&initial_workers=1\"\n - \"curl -s -X PUT http://localhost:8081/models/resnet-152?min_worker=2&synchronous=true\"\n post-process:\n - \"multi-model-server --stop > /dev/null 2>&1\"\n - \"rm kitten.jpg\"\n - module: server_local_monitoring # should be added in modules section\n ServerLocalClient: # keyword from metrics_monitoring_inproc.Monitor\n - interval: 1s\n logging : True\n metrics:\n - sum_workers_memory_rss\n - sum_workers_file_descriptors\n - total_workers\n - orphans\n\nreporting:\n- module: passfail\n criteria:\n - subject: avg-rt # required\n label: 'Inference' # optional, default is ''\n condition: '>' # required\n threshold: ${INFR_RT} # required\n logic: for # optional, logic to aggregate values within timeframe.\n # Default 'for' means take latest,\n # 'within' and 'over' means take sum/avg of all values within interval\n timeframe: 1s # optional, default is none\n stop: true # optional, default is true. false for nonstop testing until the end\n fail: true # optional, default is true\n - class: bzt.modules.monitoring.MonitoringCriteria\n subject: ServerLocalClient/sum_workers_memory_rss\n condition: '>'\n threshold: ${TOTAL_WORKERS_MEM}\n timeframe: 1s\n stop : true\n fail : true\n diff_percent : 30\n - class: bzt.modules.monitoring.MonitoringCriteria\n subject: ServerLocalClient/orphans\n condition: '>'\n threshold: ${TOTAL_ORPHANS}\n timeframe: 1s\n stop : true\n fail : true\n diff_percent : 0\n - class: bzt.modules.monitoring.MonitoringCriteria\n subject: ServerLocalClient/total_workers\n condition: '>'\n threshold: ${TOTAL_WORKERS}\n timeframe: 1s\n stop: true\n fail: true\n diff_percent: 0\n - class: bzt.modules.monitoring.MonitoringCriteria\n subject: ServerLocalClient/sum_workers_file_descriptors\n condition: '>'\n threshold: ${TOTAL_WORKERS_FDS}\n timeframe: 1s\n stop: true\n fail: true\n diff_percent: 30\n" }, { "path": "tests/performance/tests/batch_inference/environments/xlarge.yaml", "content": "---\nsettings:\n env:\n INFR_RT : 1.5s\n TOTAL_WORKERS_MEM : 3000000000 #3GB\n TOTAL_WORKERS : 4\n TOTAL_ORPHANS : 0\n TOTAL_WORKERS_FDS : 38" }, { "path": "tests/performance/tests/examples_local_criteria/environments/xlarge.yaml", "content": "---\nsettings:\n env:\n FAIL : 100%\n P90 : 290ms\n AVG_RT : 1s\n TOTAL_WORKERS_MEM : 132000000\n PERCENT_DIFF_TOTAL_WORKERS_MEM : 5\n" }, { "path": "tests/performance/tests/examples_local_criteria/examples_local_criteria.jmx", "content": "\n\n \n \n \n false\n true\n false\n \n \n \n \n \n \n \n \n \n cnn_url\n https://s3.amazonaws.com/model-server/model_archive_1.0/squeezenet_v1.1.mar\n =\n The url from where to fetch noop model from\n \n \n scale_up_workers\n ${__P(min_workers,1)}\n =\n The workers to scale No op model to\n \n \n scale_down_workers\n 0\n Offload the No-Op Model\n =\n \n \n model\n squeezenet_v1.1\n =\n \n \n \n \n \n \n \n \n ${__P(hostname,127.0.0.1)}\n ${__P(port,8443)}\n ${__P(protocol,https)}\n \n \n 6\n \n \n \n \n \n continue\n \n false\n 1\n \n 1\n 1\n false\n \n \n true\n \n \n \n \n \n \n \n ${__P(management_port,8444)}\n \n \n /models?url=${cnn_url}\n POST\n true\n false\n true\n false\n \n \n \n \n \n \n \n \n \n \n ${__P(management_port,8444)}\n \n \n /models/${model}?min_worker=${scale_up_workers}\n PUT\n true\n false\n true\n false\n \n \n \n \n \n \n \n continue\n \n false\n ${__P(loops,4)}\n \n ${__P(threads,10)}\n ${__P(rampup,10)}\n false\n \n \n true\n \n \n \n \n \n \n ${__P(input_filepath)}\n data\n image/jpeg\n \n \n \n \n \n \n \n \n \n \n /predictions/${model}\n POST\n true\n false\n true\n true\n \n \n \n \n \n \n \n continue\n \n false\n 1\n \n 1\n 1\n false\n \n \n true\n \n \n \n \n \n \n \n ${__P(management_port,8444)}\n \n \n /models/${model}?min_worker=${scale_down_workers}\n DELETE\n true\n false\n true\n false\n \n \n \n \n \n \n \n \n\n" }, { "path": "tests/performance/tests/examples_local_criteria/examples_local_criteria.yaml", "content": "---\nexecution:\n- concurrency: 1\n ramp-up: 5s\n hold-for: 20s\n scenario: Inference\n\nscenarios:\n Inference:\n script: examples_local_criteria.jmx\n\nmodules:\n server_local_monitoring:\n # metrics_monitoring_inproc and dependencies should be in python path\n class : metrics_monitoring_inproc.Monitor # monitoring class.\n\nservices:\n - module: shellexec\n prepare:\n - \"curl -s -O https://s3.amazonaws.com/model-server/inputs/kitten.jpg\"\n - \"multi-model-server --start > /dev/null 2>&1\"\n - \"sleep 10s\"\n post-process:\n - \"multi-model-server --stop > /dev/null 2>&1\"\n - \"rm kitten.jpg\"\n - module: server_local_monitoring # should be added in modules section\n ServerLocalClient: # keyword from metrics_monitoring_inproc.Monitor\n - interval: 1s\n logging : True\n metrics:\n - cpu\n - disk-space\n - mem\n - sum_workers_memory_rss\n\nreporting:\n- module: passfail\n criteria:\n - fail >${FAIL}, stop as failed\n - p90 >${P90} , stop as failed\n - avg-rt >${AVG_RT} , stop as failed\n - class: bzt.modules.monitoring.MonitoringCriteria\n subject: ServerLocalClient/sum_workers_memory_rss\n condition: '>'\n threshold: ${TOTAL_WORKERS_MEM}\n timeframe: 1s\n stop : true\n fail : true\n diff_percent : ${PERCENT_DIFF_TOTAL_WORKERS_MEM}\n" }, { "path": "tests/performance/tests/examples_local_monitoring/examples_local_monitoring.jmx", "content": "\n\n \n \n \n false\n true\n false\n \n \n \n \n \n \n \n \n \n cnn_url\n https://s3.amazonaws.com/model-server/model_archive_1.0/squeezenet_v1.1.mar\n =\n The url from where to fetch noop model from\n \n \n scale_up_workers\n ${__P(min_workers,1)}\n =\n The workers to scale No op model to\n \n \n scale_down_workers\n 0\n Offload the No-Op Model\n =\n \n \n model\n squeezenet_v1.1\n =\n \n \n \n \n \n \n \n \n ${__P(hostname,127.0.0.1)}\n ${__P(port,8443)}\n ${__P(protocol,https)}\n \n \n 6\n \n \n \n \n \n continue\n \n false\n 1\n \n 1\n 1\n false\n \n \n true\n \n \n \n \n \n \n \n ${__P(management_port,8444)}\n \n \n /models?url=${cnn_url}\n POST\n true\n false\n true\n false\n \n \n \n \n \n \n \n \n \n \n ${__P(management_port,8444)}\n \n \n /models/${model}?min_worker=${scale_up_workers}\n PUT\n true\n false\n true\n false\n \n \n \n \n \n \n \n continue\n \n false\n ${__P(loops,4)}\n \n ${__P(threads,10)}\n ${__P(rampup,10)}\n false\n \n \n true\n \n \n \n \n \n \n ${__P(input_filepath)}\n data\n image/jpeg\n \n \n \n \n \n \n \n \n \n \n /predictions/${model}\n POST\n true\n false\n true\n true\n \n \n \n \n \n \n \n continue\n \n false\n 1\n \n 1\n 1\n false\n \n \n true\n \n \n \n \n \n \n \n ${__P(management_port,8444)}\n \n \n /models/${model}?min_worker=${scale_down_workers}\n DELETE\n true\n false\n true\n false\n \n \n \n \n \n \n \n \n\n" }, { "path": "tests/performance/tests/examples_local_monitoring/examples_local_monitoring.yaml", "content": "---\nexecution:\n- concurrency: 1\n ramp-up: 5s\n hold-for: 20s\n scenario: Inference\nscenarios:\n Inference:\n script: examples_local_monitoring.jmx\n\nmodules:\n server_local_monitoring:\n # metrics_monitoring_inproc and dependencies should be in python path\n class : metrics_monitoring_inproc.Monitor # monitoring class.\n\nservices:\n - module: shellexec\n prepare:\n - \"curl -s -O https://s3.amazonaws.com/model-server/inputs/kitten.jpg\"\n - \"multi-model-server --start > /dev/null 2>&1\"\n - \"sleep 10s\"\n post-process:\n - \"multi-model-server --stop > /dev/null 2>&1\"\n - \"rm kitten.jpg\"\n - module: server_local_monitoring # should be added in modules section\n ServerLocalClient: # keyword from metrics_monitoring_inproc.Monitor\n - interval: 1s\n metrics:\n - cpu\n - disk-space\n - mem\n - sum_workers_memory_percent" }, { "path": "tests/performance/tests/examples_remote_criteria/environments/xlarge.yaml", "content": "---\nsettings:\n env:\n FAIL : 50%\n P90 : 250ms\n AVG_RT : 1s\n TOTAL_WORKERS_FDS : 80\n" }, { "path": "tests/performance/tests/examples_remote_criteria/examples_remote_criteria.jmx", "content": "\n\n \n \n \n false\n true\n false\n \n \n \n \n \n \n \n \n \n cnn_url\n https://s3.amazonaws.com/model-server/model_archive_1.0/squeezenet_v1.1.mar\n =\n The url from where to fetch noop model from\n \n \n scale_up_workers\n ${__P(min_workers,1)}\n =\n The workers to scale No op model to\n \n \n scale_down_workers\n 0\n Offload the No-Op Model\n =\n \n \n model\n squeezenet_v1.1\n =\n \n \n \n \n \n \n \n \n ${__P(hostname,127.0.0.1)}\n ${__P(port,8443)}\n ${__P(protocol,https)}\n \n \n 6\n \n \n \n \n \n continue\n \n false\n 1\n \n 1\n 1\n false\n \n \n true\n \n \n \n \n \n \n \n ${__P(management_port,8444)}\n \n \n /models?url=${cnn_url}\n POST\n true\n false\n true\n false\n \n \n \n \n \n \n \n \n \n \n ${__P(management_port,8444)}\n \n \n /models/${model}?min_worker=${scale_up_workers}\n PUT\n true\n false\n true\n false\n \n \n \n \n \n \n \n continue\n \n false\n ${__P(loops,4)}\n \n ${__P(threads,10)}\n ${__P(rampup,10)}\n false\n \n \n true\n \n \n \n \n \n \n ${__P(input_filepath)}\n data\n image/jpeg\n \n \n \n \n \n \n \n \n \n \n /predictions/${model}\n POST\n true\n false\n true\n true\n \n \n \n \n \n \n \n continue\n \n false\n 1\n \n 1\n 1\n false\n \n \n true\n \n \n \n \n \n \n \n ${__P(management_port,8444)}\n \n \n /models/${model}?min_worker=${scale_down_workers}\n DELETE\n true\n false\n true\n false\n \n \n \n \n \n \n \n \n\n" }, { "path": "tests/performance/tests/examples_remote_criteria/examples_remote_criteria.yaml", "content": "\nexecution:\n- concurrency: 4\n ramp-up: 1s\n hold-for: 20s\n scenario: Inference\n\nscenarios:\n Inference:\n script: examples_remote_criteria.jmx\n\nservices:\n - module: shellexec\n prepare:\n - \"curl -s -O https://s3.amazonaws.com/model-server/inputs/kitten.jpg\"\n - \"multi-model-server --start > /dev/null 2>&1\"\n - \"sleep 10s\"\n post-process:\n - \"multi-model-server --stop > /dev/null 2>&1\"\n - \"rm kitten.jpg\"\n - module: monitoring\n server-agent:\n - address: localhost:9009 # metric monitoring service address\n label: mms-inference-server # if you specify label, it will be used in reports instead of ip:port\n interval: 1s # polling interval\n logging: True # those logs will be saved to \"SAlogs_192.168.0.1_9009.csv\" in the artifacts dir\n metrics: # metrics should be supported by monitoring service\n - sum_workers_cpu_percent # cpu percent used by all the mms server processes and workers\n - sum_workers_memory_percent\n - sum_workers_file_descriptors\n - total_workers # no of mms workers\n\n\nreporting:\n- module: passfail\n criteria:\n - fail >${FAIL}, stop as failed\n - p90 >${P90} , stop as failed\n - avg-rt >${AVG_RT} , stop as failed\n - class: bzt.modules.monitoring.MonitoringCriteria\n subject: mms-inference-server/sum_workers_file_descriptors\n condition: '>'\n threshold: ${TOTAL_WORKERS_FDS}\n timeframe: 1s\n fail: true\n stop: true\n diff_percent : 35" }, { "path": "tests/performance/tests/examples_remote_monitoring/examples_remote_monitoring.jmx", "content": "\n\n \n \n \n false\n true\n false\n \n \n \n \n \n \n \n \n \n cnn_url\n https://s3.amazonaws.com/model-server/model_archive_1.0/squeezenet_v1.1.mar\n =\n The url from where to fetch noop model from\n \n \n scale_up_workers\n ${__P(min_workers,1)}\n =\n The workers to scale No op model to\n \n \n scale_down_workers\n 0\n Offload the No-Op Model\n =\n \n \n model\n squeezenet_v1.1\n =\n \n \n \n \n \n \n \n \n ${__P(hostname,127.0.0.1)}\n ${__P(port,8443)}\n ${__P(protocol,https)}\n \n \n 6\n \n \n \n \n \n continue\n \n false\n 1\n \n 1\n 1\n false\n \n \n true\n \n \n \n \n \n \n \n ${__P(management_port,8444)}\n \n \n /models?url=${cnn_url}\n POST\n true\n false\n true\n false\n \n \n \n \n \n \n \n \n \n \n ${__P(management_port,8444)}\n \n \n /models/${model}?min_worker=${scale_up_workers}\n PUT\n true\n false\n true\n false\n \n \n \n \n \n \n \n continue\n \n false\n ${__P(loops,4)}\n \n ${__P(threads,10)}\n ${__P(rampup,10)}\n false\n \n \n true\n \n \n \n \n \n \n ${__P(input_filepath)}\n data\n image/jpeg\n \n \n \n \n \n \n \n \n \n \n /predictions/${model}\n POST\n true\n false\n true\n true\n \n \n \n \n \n \n \n continue\n \n false\n 1\n \n 1\n 1\n false\n \n \n true\n \n \n \n \n \n \n \n ${__P(management_port,8444)}\n \n \n /models/${model}?min_worker=${scale_down_workers}\n DELETE\n true\n false\n true\n false\n \n \n \n \n \n \n \n \n\n" }, { "path": "tests/performance/tests/examples_remote_monitoring/examples_remote_monitoring.yaml", "content": "\nexecution:\n- concurrency: 4\n ramp-up: 1s\n hold-for: 20s\n scenario: Inference\n\nscenarios:\n Inference:\n script: examples_remote_monitoring.jmx\n\n\n\nservices:\n - module: shellexec\n prepare:\n - \"curl -s -O https://s3.amazonaws.com/model-server/inputs/kitten.jpg\"\n - \"multi-model-server --start > /dev/null 2>&1\"\n - \"sleep 10s\"\n post-process:\n - \"multi-model-server --stop > /dev/null 2>&1\"\n - \"rm kitten.jpg\"\n - module: monitoring\n server-agent:\n - address: localhost:9009 # metric monitoring service address\n label: mms-inference-server # if you specify label, it will be used in reports instead of ip:port\n interval: 1s # polling interval\n logging: True # those logs will be saved to \"SAlogs_192.168.0.1_9009.csv\" in the artifacts dir\n metrics: # metrics should be supported by monitoring service\n - sum_all_cpu_percent # cpu percent used by all the mms server processes and workers\n - sum_workers_memory_percent\n - frontend_file_descriptors\n - total_workers # no of mms workers\n\n" }, { "path": "tests/performance/tests/examples_starter/examples_starter.jmx", "content": "\n\n \n \n \n false\n true\n false\n \n \n \n \n \n \n \n \n \n cnn_url\n https://s3.amazonaws.com/model-server/model_archive_1.0/squeezenet_v1.1.mar\n =\n The url from where to fetch noop model from\n \n \n scale_up_workers\n ${__P(min_workers,1)}\n =\n The workers to scale No op model to\n \n \n scale_down_workers\n 0\n Offload the No-Op Model\n =\n \n \n model\n squeezenet_v1.1\n =\n \n \n \n \n \n \n \n \n ${__P(hostname,127.0.0.1)}\n ${__P(port,8443)}\n ${__P(protocol,https)}\n \n \n 6\n \n \n \n \n \n continue\n \n false\n 1\n \n 1\n 1\n false\n \n \n true\n \n \n \n \n \n \n \n ${__P(management_port,8444)}\n \n \n /models?url=${cnn_url}\n POST\n true\n false\n true\n false\n \n \n \n \n \n \n \n \n \n \n ${__P(management_port,8444)}\n \n \n /models/${model}?min_worker=${scale_up_workers}\n PUT\n true\n false\n true\n false\n \n \n \n \n \n \n \n continue\n \n false\n ${__P(loops,4)}\n \n ${__P(threads,10)}\n ${__P(rampup,10)}\n false\n \n \n true\n \n \n \n \n \n \n ${__P(input_filepath)}\n data\n image/jpeg\n \n \n \n \n \n \n \n \n \n \n /predictions/${model}\n POST\n true\n false\n true\n true\n \n \n \n \n \n \n \n continue\n \n false\n 1\n \n 1\n 1\n false\n \n \n true\n \n \n \n \n \n \n \n ${__P(management_port,8444)}\n \n \n /models/${model}?min_worker=${scale_down_workers}\n DELETE\n true\n false\n true\n false\n \n \n \n \n \n \n \n \n\n" }, { "path": "tests/performance/tests/examples_starter/examples_starter.yaml", "content": "---\nexecution:\n- concurrency: 1\n ramp-up: 1s\n hold-for: 40s\n scenario: Inference\nscenarios:\n Inference:\n script: examples_starter.jmx\n\nservices:\n - module: shellexec\n prepare:\n - \"curl -s -O https://s3.amazonaws.com/model-server/inputs/kitten.jpg\"\n - \"multi-model-server --start > /dev/null 2>&1\"\n - \"sleep 10s\"\n post-process:\n - \"multi-model-server --stop > /dev/null 2>&1\"\n - \"rm kitten.jpg\"\n\n" }, { "path": "tests/performance/tests/global_config.yaml", "content": "modules:\n jmeter:\n # These are JMeter test case properties. These variables are used in jmx files.\n # Change the vaues as per your setup\n properties:\n hostname : 127.0.0.1 # MMS properties\n port : 8080\n management_port : 8081\n protocol : http\n input_filepath : kitten.jpg # make sure jpg is available at this path\n # if relative path is provided this will be relative to current working directory\n\n# DO-NOT change properties below unless you know what you are doing.\n# They are needed for performance test suite runner script.\nreporting:\n- module: passfail # this is to enable passfail module\n- module: junit-xml\n data-source: pass-fail\n- module: junit-xml\n data-source: sample-labels\n- module: final-stats\n dump-csv : ${BASEDIR}/final_stats.csv\n\nsettings:\n env:\n BASEDIR : '.'\n" }, { "path": "tests/performance/tests/health_check/environments/xlarge.yaml", "content": "---\nsettings:\n env:\n HLTH_CHK_SUCC : 100%\n HLTH_CHK_RT : 14ms\n TOTAL_PROCS : 1\n TOTAL_FDS : 67\n TOTAL_MEM : 750000000 #750MB" }, { "path": "tests/performance/tests/health_check/health_check.jmx", "content": "\n\n \n \n \n false\n true\n false\n \n \n \n \n \n \n \n continue\n \n false\n ${__P(loops,4)}\n \n ${__P(threads,10)}\n ${__P(rampup,10)}\n false\n \n \n true\n \n \n \n \n \n \n ${__P(hostname,127.0.0.1)}\n ${__P(port,8443)}\n ${__P(protocol,https)}\n \n /ping\n GET\n true\n false\n true\n false\n \n \n \n \n \n \n \n \n\n" }, { "path": "tests/performance/tests/health_check/health_check.yaml", "content": "---\nexecution:\n- concurrency: 10\n ramp-up: 1s\n hold-for: 30s\n scenario: health_check\n\nscenarios:\n health_check:\n script: health_check.jmx\n\nmodules:\n server_local_monitoring:\n class : metrics_monitoring_inproc.Monitor\n\nservices:\n - module: shellexec\n prepare:\n - \"multi-model-server --start > /dev/null 2>&1\"\n - \"sleep 10s\"\n post-process:\n - \"multi-model-server --stop > /dev/null 2>&1\"\n - module: server_local_monitoring\n ServerLocalClient:\n - interval: 1s\n logging : True\n metrics:\n - total_processes\n - sum_all_file_descriptors\n - sum_all_memory_rss\n\nreporting:\n- module: passfail\n criteria:\n # Inbuilt Criteria\n - success of HealthCheck<${HLTH_CHK_SUCC}, stop as failed\n - avg-rt of HealthCheck>${HLTH_CHK_RT}, stop as failed\n # Custom Criteria\n - class: bzt.modules.monitoring.MonitoringCriteria\n subject: ServerLocalClient/total_processes\n condition: '>'\n threshold: ${TOTAL_PROCS}\n timeframe: 1s\n stop : true\n fail : true\n - class: bzt.modules.monitoring.MonitoringCriteria\n subject: ServerLocalClient/total_processes\n condition: '<'\n threshold: ${TOTAL_PROCS}\n timeframe: 1s\n stop : true\n fail : true\n - class: bzt.modules.monitoring.MonitoringCriteria\n subject: ServerLocalClient/sum_all_file_descriptors\n condition: '>'\n threshold: ${TOTAL_FDS}\n timeframe: 5s\n stop : true\n fail : true\n# - class: bzt.modules.monitoring.MonitoringCriteria\n# subject: ServerLocalClient/sum_all_memory_rss\n# condition: '>'\n# threshold: ${TOTAL_MEM}\n# timeframe: 5s\n# stop : true\n# fail : true" }, { "path": "tests/performance/tests/inference_multiple_models/environments/xlarge.yaml", "content": "---\nsettings:\n env:\n INFR1_SUCC : 100%\n INFR2_SUCC: 100%\n INFR1_RT : 290ms\n INFR2_RT: 450ms\n TOTAL_PROCS : 5\n TOTAL_FDS : 107\n TOTAL_MEM : 600000000 #600MB" }, { "path": "tests/performance/tests/inference_multiple_models/inference_multiple_models.jmx", "content": "\n\n \n \n \n false\n true\n false\n \n \n \n \n \n \n \n \n \n model1\n squeezenet_v1.1\n =\n Model1 Name\n \n \n model2\n resnet-18\n Model2 Name\n =\n \n \n \n \n \n \n \n \n ${__P(hostname,127.0.0.1)}\n ${__P(port,8443)}\n ${__P(protocol,https)}\n \n \n 6\n \n \n \n \n \n continue\n \n false\n ${__P(loops,4)}\n \n ${__P(threads,10)}\n ${__P(rampup,10)}\n false\n \n \n true\n \n \n \n \n \n \n ${__P(input_filepath)}\n data\n image/jpeg\n \n \n \n \n \n \n \n \n \n \n /predictions/${model1}\n POST\n true\n false\n true\n true\n \n \n \n \n \n \n \n continue\n \n false\n ${__P(loops,4)}\n \n ${__P(threads,10)}\n ${__P(rampup,10)}\n false\n \n \n true\n \n \n \n \n \n \n ${__P(input_filepath)}\n data\n image/jpeg\n \n \n \n \n \n \n \n \n \n \n /predictions/${model2}\n POST\n true\n false\n true\n true\n \n \n \n \n \n \n \n \n\n" }, { "path": "tests/performance/tests/inference_multiple_models/inference_multiple_models.yaml", "content": "---\nexecution:\n- concurrency: 10\n ramp-up: 1s\n hold-for: 30s\n scenario: inference_multiple_models\n\nscenarios:\n inference_multiple_models:\n script: inference_multiple_models.jmx\n\nmodules:\n server_local_monitoring:\n class : metrics_monitoring_inproc.Monitor\n\nservices:\n - module: shellexec\n prepare:\n - \"curl -s -O https://s3.amazonaws.com/model-server/inputs/kitten.jpg\"\n - \"multi-model-server --start > /dev/null 2>&1\"\n - \"sleep 20s\"\n - \"curl -s -X POST http://localhost:8081/models?url=https://s3.amazonaws.com/model-server/model_archive_1.0/squeezenet_v1.1.mar\"\n - \"curl -s -X POST http://localhost:8081/models?url=https://s3.amazonaws.com/model-server/model_archive_1.0/resnet-18.mar\"\n - \"curl -s -X PUT http://localhost:8081/models/squeezenet_v1.1?min_worker=1&synchronous=true\"\n - \"curl -s -X PUT http://localhost:8081/models/resnet-18?min_worker=1&synchronous=true\"\n post-process:\n - \"multi-model-server --stop > /dev/null 2>&1\"\n - \"rm kitten.jpg\"\n - module: server_local_monitoring\n ServerLocalClient:\n - interval: 1s\n logging : True\n metrics:\n - total_processes\n - sum_all_file_descriptors\n - sum_all_memory_rss\n\nreporting:\n- module: passfail\n criteria:\n # Inbuilt Criteria\n - success of Inference1<${INFR1_SUCC}, stop as failed\n - success of Inference2<${INFR2_SUCC}, stop as failed\n - avg-rt of Inference1>${INFR1_RT}, stop as failed\n - avg-rt of Inference2>${INFR2_RT}, stop as failed\n # Custom Criteria\n - class: bzt.modules.monitoring.MonitoringCriteria\n subject: ServerLocalClient/total_processes\n condition: '>'\n threshold: ${TOTAL_PROCS}\n timeframe: 1s\n stop : true\n fail : true\n - class: bzt.modules.monitoring.MonitoringCriteria\n subject: ServerLocalClient/total_processes\n condition: '<'\n threshold: ${TOTAL_PROCS}\n timeframe: 1s\n stop : true\n fail : true\n - class: bzt.modules.monitoring.MonitoringCriteria\n subject: ServerLocalClient/sum_all_file_descriptors\n condition: '>'\n threshold: ${TOTAL_FDS}\n timeframe: 5s\n stop : true\n fail : true\n# - class: bzt.modules.monitoring.MonitoringCriteria\n# subject: ServerLocalClient/sum_all_memory_rss\n# condition: '>'\n# threshold: ${TOTAL_MEM}\n# timeframe: 5s\n# stop : true\n# fail : true\n" }, { "path": "tests/performance/tests/inference_multiple_worker/environments/xlarge.yaml", "content": "---\nsettings:\n env:\n INFR_SUCC : 100%\n INFR_RT : 140ms\n TOTAL_PROCS : 6\n TOTAL_FDS : 126\n TOTAL_MEM : 750000000 #750MB" }, { "path": "tests/performance/tests/inference_multiple_worker/inference_multiple_worker.jmx", "content": "\n\n \n \n \n false\n true\n false\n \n \n \n \n \n \n \n \n \n model\n squeezenet_v1.1\n =\n Model Name\n \n \n \n \n \n \n \n \n ${__P(hostname,127.0.0.1)}\n ${__P(port,8443)}\n ${__P(protocol,https)}\n \n \n 6\n \n \n \n \n \n continue\n \n false\n ${__P(loops,4)}\n \n ${__P(threads,10)}\n ${__P(rampup,10)}\n false\n \n \n true\n \n \n \n \n \n \n ${__P(input_filepath)}\n data\n image/jpeg\n \n \n \n \n \n \n \n \n \n \n /predictions/${model}\n POST\n true\n false\n true\n true\n \n \n \n \n \n \n \n \n\n" }, { "path": "tests/performance/tests/inference_multiple_worker/inference_multiple_worker.yaml", "content": "---\nexecution:\n- concurrency: 10\n ramp-up: 1s\n hold-for: 1m\n iterations: 100\n scenario: inference_multiple_worker\n\nscenarios:\n inference_multiple_worker:\n script: inference_multiple_worker.jmx\n\nmodules:\n server_local_monitoring:\n class : metrics_monitoring_inproc.Monitor\n\nservices:\n - module: shellexec\n prepare:\n - \"curl -s -O https://s3.amazonaws.com/model-server/inputs/kitten.jpg\"\n - \"multi-model-server --start > /dev/null 2>&1\"\n - \"sleep 20s\"\n - \"curl -s -X POST http://localhost:8081/models?url=https://s3.amazonaws.com/model-server/model_archive_1.0/squeezenet_v1.1.mar\"\n - \"curl -s -X PUT http://localhost:8081/models/squeezenet_v1.1?min_worker=4&synchronous=true\"\n post-process:\n - \"multi-model-server --stop > /dev/null 2>&1\"\n - \"rm kitten.jpg\"\n - module: server_local_monitoring\n ServerLocalClient:\n - interval: 1s\n logging : True\n metrics:\n - total_processes\n - sum_all_file_descriptors\n - sum_all_memory_rss\n\nreporting:\n- module: passfail\n criteria:\n # Inbuilt Criteria\n - success of Inference<${INFR_SUCC}, stop as failed\n - avg-rt of Inference>${INFR_RT}, stop as failed\n # Custom Criteria\n - class: bzt.modules.monitoring.MonitoringCriteria\n subject: ServerLocalClient/total_processes\n condition: '>'\n threshold: ${TOTAL_PROCS}\n timeframe: 1s\n stop : true\n fail : true\n - class: bzt.modules.monitoring.MonitoringCriteria\n subject: ServerLocalClient/total_processes\n condition: '<'\n threshold: ${TOTAL_PROCS}\n timeframe: 1s\n stop : true\n fail : true\n - class: bzt.modules.monitoring.MonitoringCriteria\n subject: ServerLocalClient/sum_all_file_descriptors\n condition: '>'\n threshold: ${TOTAL_FDS}\n timeframe: 1s\n stop : true\n fail : true\n# - class: bzt.modules.monitoring.MonitoringCriteria\n# subject: ServerLocalClient/sum_all_memory_rss\n# condition: '>'\n# threshold: ${TOTAL_MEM}\n# timeframe: 5s\n# stop : true\n# fail : true" }, { "path": "tests/performance/tests/inference_single_worker/environments/xlarge.yaml", "content": "---\nsettings:\n env:\n INFR_SUCC : 100%\n INFR_RT : 290ms\n TOTAL_PROCS : 3\n TOTAL_FDS : 90\n TOTAL_MEM : 290000000 #290MB" }, { "path": "tests/performance/tests/inference_single_worker/inference_single_worker.jmx", "content": "\n\n \n \n \n false\n true\n false\n \n \n \n \n \n \n \n \n \n model\n squeezenet_v1.1\n =\n Model Name\n \n \n \n \n \n \n \n \n ${__P(hostname,127.0.0.1)}\n ${__P(port,8443)}\n ${__P(protocol,https)}\n \n \n 6\n \n \n \n \n \n continue\n \n false\n ${__P(loops,4)}\n \n ${__P(threads,10)}\n ${__P(rampup,10)}\n false\n \n \n true\n \n \n \n \n \n \n ${__P(input_filepath)}\n data\n image/jpeg\n \n \n \n \n \n \n \n \n \n \n /predictions/${model}\n POST\n true\n false\n true\n true\n \n \n \n \n \n \n \n \n\n" }, { "path": "tests/performance/tests/inference_single_worker/inference_single_worker.yaml", "content": "---\nexecution:\n- concurrency: 10\n ramp-up: 1s\n hold-for: 1m\n iterations: 100\n scenario: inference_single_worker\n\nscenarios:\n inference_single_worker:\n script: inference_single_worker.jmx\n\nmodules:\n server_local_monitoring:\n class : metrics_monitoring_inproc.Monitor\n\nservices:\n - module: shellexec\n prepare:\n - \"curl -s -O https://s3.amazonaws.com/model-server/inputs/kitten.jpg\"\n - \"multi-model-server --start > /dev/null 2>&1\"\n - \"sleep 20s\"\n - \"curl -s -X POST http://localhost:8081/models?url=https://s3.amazonaws.com/model-server/model_archive_1.0/squeezenet_v1.1.mar\"\n - \"curl -s -X PUT http://localhost:8081/models/squeezenet_v1.1?min_worker=1&synchronous=true\"\n post-process:\n - \"multi-model-server --stop > /dev/null 2>&1\"\n - \"rm kitten.jpg\"\n - module: server_local_monitoring\n ServerLocalClient:\n - interval: 1s\n logging : True\n metrics:\n - total_processes\n - sum_all_file_descriptors\n - sum_all_memory_rss\n\nreporting:\n- module: passfail\n criteria:\n # Inbuilt Criteria\n - success of Inference<${INFR_SUCC}, stop as failed\n - avg-rt of Inference>${INFR_RT}, stop as failed\n # Custom Criteria\n - class: bzt.modules.monitoring.MonitoringCriteria\n subject: ServerLocalClient/total_processes\n condition: '>'\n threshold: ${TOTAL_PROCS}\n timeframe: 1s\n stop : true\n fail : true\n - class: bzt.modules.monitoring.MonitoringCriteria\n subject: ServerLocalClient/total_processes\n condition: '<'\n threshold: ${TOTAL_PROCS}\n timeframe: 1s\n stop : true\n fail : true\n - class: bzt.modules.monitoring.MonitoringCriteria\n subject: ServerLocalClient/sum_all_file_descriptors\n condition: '>'\n threshold: ${TOTAL_FDS}\n timeframe: 1s\n stop : true\n fail : true\n# - class: bzt.modules.monitoring.MonitoringCriteria\n# subject: ServerLocalClient/sum_all_memory_rss\n# condition: '>'\n# threshold: ${TOTAL_MEM}\n# timeframe: 5s\n# stop : true\n# fail : true\n" }, { "path": "tests/performance/tests/list_models/environments/xlarge.yaml", "content": "---\nsettings:\n env:\n LST_MODLS_SUCC : 100%\n LST_MODLS_RT : 14ms\n TOTAL_PROCS : 3\n TOTAL_FDS : 86\n TOTAL_MEM : 185000000 #185MB" }, { "path": "tests/performance/tests/list_models/list_models.jmx", "content": "\n\n \n \n \n false\n true\n false\n \n \n \n \n \n \n \n \n \n model\n squeezenet_v1.1\n =\n Model name\n \n \n \n \n \n \n \n \n ${__P(hostname,127.0.0.1)}\n ${__P(port,8443)}\n ${__P(protocol,https)}\n \n \n 6\n \n \n \n \n \n continue\n \n false\n ${__P(loops,4)}\n \n ${__P(threads,10)}\n ${__P(rampup,10)}\n false\n \n \n true\n \n \n \n \n \n \n \n ${__P(management_port,8444)}\n \n \n /models\n GET\n true\n false\n true\n false\n \n \n \n \n \n \n \n \n\n" }, { "path": "tests/performance/tests/list_models/list_models.yaml", "content": "---\nexecution:\n- concurrency: 10\n ramp-up: 1s\n hold-for: 30s\n scenario: list_models\n\nscenarios:\n list_models:\n script: list_models.jmx\n\nmodules:\n server_local_monitoring:\n class : metrics_monitoring_inproc.Monitor\n\nservices:\n - module: shellexec\n prepare:\n - \"multi-model-server --start > /dev/null 2>&1\"\n - \"sleep 20s\"\n - \"curl -s -X POST http://localhost:8081/models?url=https://s3.amazonaws.com/model-server/model_archive_1.0/squeezenet_v1.1.mar\"\n - \"curl -s -X POST http://localhost:8081/models?url=https://s3.amazonaws.com/model-server/model_archive_1.0/shufflenet.mar\"\n post-process:\n - \"multi-model-server --stop > /dev/null 2>&1\"\n - module: server_local_monitoring\n ServerLocalClient:\n - interval: 1s\n logging : True\n metrics:\n - total_processes\n - sum_all_file_descriptors\n - sum_all_memory_rss\n\nreporting:\n- module: passfail\n criteria:\n # Inbuilt Criteria\n - success of ListModels<${LST_MODLS_SUCC}, stop as failed\n - avg-rt of ListModels>${LST_MODLS_RT}, stop as failed\n # Custom Criteria\n - class: bzt.modules.monitoring.MonitoringCriteria\n subject: ServerLocalClient/total_processes\n condition: '>'\n threshold: ${TOTAL_PROCS}\n timeframe: 1s\n stop : true\n fail : true\n - class: bzt.modules.monitoring.MonitoringCriteria\n subject: ServerLocalClient/total_processes\n condition: '<'\n threshold: ${TOTAL_PROCS}\n timeframe: 1s\n stop : true\n fail : true\n - class: bzt.modules.monitoring.MonitoringCriteria\n subject: ServerLocalClient/sum_all_file_descriptors\n condition: '>'\n threshold: ${TOTAL_FDS}\n timeframe: 1s\n stop : true\n fail : true\n# - class: bzt.modules.monitoring.MonitoringCriteria\n# subject: ServerLocalClient/sum_all_memory_rss\n# condition: '>'\n# threshold: ${TOTAL_MEM}\n# timeframe: 5s\n# stop : true\n# fail : true" }, { "path": "tests/performance/tests/model_description/environments/xlarge.yaml", "content": "---\nsettings:\n env:\n MODL_DESC_SUCC : 100%\n MODL_DESC_RT : 14ms\n TOTAL_PROCS : 3\n TOTAL_FDS : 90\n TOTAL_MEM : 300000000 #300MB" }, { "path": "tests/performance/tests/model_description/model_description.jmx", "content": "\n\n \n \n \n false\n true\n false\n \n \n \n \n \n \n \n \n \n model\n squeezenet_v1.1\n =\n Model name\n \n \n \n \n \n \n \n \n ${__P(hostname,127.0.0.1)}\n ${__P(port,8443)}\n ${__P(protocol,https)}\n \n \n 6\n \n \n \n \n \n continue\n \n false\n ${__P(loops,4)}\n \n ${__P(threads,10)}\n ${__P(rampup,10)}\n false\n \n \n true\n \n \n \n \n \n \n \n ${__P(management_port,8444)}\n \n \n /models/${model}\n GET\n true\n false\n true\n false\n \n \n \n \n \n \n \n \n\n" }, { "path": "tests/performance/tests/model_description/model_description.yaml", "content": "---\nexecution:\n- concurrency: 10\n ramp-up: 1s\n hold-for: 30s\n scenario: model_description\n\nscenarios:\n model_description:\n script: model_description.jmx\n\nmodules:\n server_local_monitoring:\n class : metrics_monitoring_inproc.Monitor\n\nservices:\n - module: shellexec\n prepare:\n - \"multi-model-server --start > /dev/null 2>&1\"\n - \"sleep 20s\"\n - \"curl -s -X POST http://localhost:8081/models?url=https://s3.amazonaws.com/model-server/model_archive_1.0/squeezenet_v1.1.mar\"\n - \"curl -s -X PUT http://localhost:8081/models/squeezenet_v1.1?min_worker=1&synchronous=true\"\n post-process:\n - \"multi-model-server --stop > /dev/null 2>&1\"\n - module: server_local_monitoring\n ServerLocalClient:\n - interval: 1s\n logging : True\n metrics:\n - total_processes\n - sum_all_file_descriptors\n - sum_all_memory_rss\n\nreporting:\n- module: passfail\n criteria:\n # Inbuilt Criteria\n - success of ModelDescription<${MODL_DESC_SUCC}, stop as failed\n - avg-rt of ModelDescription>${MODL_DESC_RT}, stop as failed\n # Custom Criteria\n - class: bzt.modules.monitoring.MonitoringCriteria\n subject: ServerLocalClient/total_processes\n condition: '>'\n threshold: ${TOTAL_PROCS}\n timeframe: 1s\n stop : true\n fail : true\n - class: bzt.modules.monitoring.MonitoringCriteria\n subject: ServerLocalClient/total_processes\n condition: '<'\n threshold: ${TOTAL_PROCS}\n timeframe: 1s\n stop : true\n fail : true\n - class: bzt.modules.monitoring.MonitoringCriteria\n subject: ServerLocalClient/sum_all_file_descriptors\n condition: '>'\n threshold: ${TOTAL_FDS}\n timeframe: 1s\n stop : true\n fail : true\n# - class: bzt.modules.monitoring.MonitoringCriteria\n# subject: ServerLocalClient/sum_all_memory_rss\n# condition: '>'\n# threshold: ${TOTAL_MEM}\n# timeframe: 5s\n# stop : true\n# fail : true" }, { "path": "tests/performance/tests/multiple_inference_and_scaling/environments/xlarge.yaml", "content": "---\nsettings:\n env:\n INFR1_SUCC : 100%\n INFR2_SUCC: 100%\n INFR1_RT : 290ms\n INFR2_RT: 450ms\n TOTAL_PROCS : 14\n TOTAL_FDS : 300\n TOTAL_MEM : 2000000000 #~2GB\n TOTAL_ORPHANS : 0\n FRNTEND_MEM : 1000000000 #~1GB" }, { "path": "tests/performance/tests/multiple_inference_and_scaling/multiple_inference_and_scaling.jmx", "content": "\n\n \n \n \n false\n true\n false\n \n \n \n \n \n \n \n \n \n model1\n squeezenet_v1.1\n =\n Model1 Name\n \n \n model2\n resnet-18\n Model2 Name\n =\n \n \n scale_up_workers1\n 4\n =\n \n \n scale_down_workers1\n 1\n =\n \n \n scale_up_workers2\n 4\n =\n \n \n scale_down_workers2\n 1\n =\n \n \n \n \n \n \n \n \n ${__P(hostname,127.0.0.1)}\n ${__P(port,8443)}\n ${__P(protocol,https)}\n \n \n 6\n \n \n \n \n \n continue\n \n false\n ${__P(loops,4)}\n \n ${__P(threads,10)}\n ${__P(rampup,10)}\n false\n \n \n true\n \n \n \n \n \n \n ${__P(input_filepath)}\n data\n image/jpeg\n \n \n \n \n \n \n \n \n \n \n /predictions/${model1}\n POST\n true\n false\n true\n true\n \n \n \n \n \n \n 1\n 0\n 20\n \n \n \n 1000\n 100\n \n \n \n \n \n \n \n \n ${__P(management_port,8444)}\n \n \n /models/${model1}?min_worker=${scale_down_workers1}&synchronous=true\n PUT\n true\n false\n true\n false\n \n \n \n \n \n \n 1\n 0\n 20\n \n \n \n 10000\n 100\n \n \n \n \n \n \n \n \n ${__P(management_port,8444)}\n \n \n /models/${model1}?min_worker=${scale_up_workers1}&synchronous=true\n PUT\n true\n false\n true\n false\n \n \n \n \n \n \n 1\n 0\n 20\n \n \n \n 10000\n 100\n \n \n \n \n \n continue\n \n false\n ${__P(loops,4)}\n \n ${__P(threads,10)}\n ${__P(rampup,10)}\n false\n \n \n true\n \n \n \n \n \n \n ${__P(input_filepath)}\n data\n image/jpeg\n \n \n \n \n \n \n \n \n \n \n /predictions/${model2}\n POST\n true\n false\n true\n true\n \n \n \n \n \n \n 1\n 0\n 20\n \n \n \n 1000\n 100\n \n \n \n \n \n \n \n \n ${__P(management_port,8444)}\n \n \n /models/${model2}?min_worker=${scale_down_workers2}&synchronous=true\n PUT\n true\n false\n true\n false\n \n \n \n \n \n \n 1\n 0\n 20\n \n \n \n 10000\n 100\n \n \n \n \n \n \n \n \n ${__P(management_port,8444)}\n \n \n /models/${model2}?min_worker=${scale_up_workers2}&synchronous=true\n PUT\n true\n false\n true\n false\n \n \n \n \n \n \n 1\n 0\n 20\n \n \n \n 10000\n 100\n \n \n \n \n \n \n\n" }, { "path": "tests/performance/tests/multiple_inference_and_scaling/multiple_inference_and_scaling.yaml", "content": "---\nexecution:\n- concurrency: 10\n ramp-up: 1s\n hold-for: 300s\n scenario: inference_multiple_models\n\nscenarios:\n inference_multiple_models:\n script: multiple_inference_and_scaling.jmx\n\nmodules:\n server_local_monitoring:\n class : metrics_monitoring_inproc.Monitor\n\nservices:\n - module: shellexec\n prepare:\n - \"curl -s -O https://s3.amazonaws.com/model-server/inputs/kitten.jpg\"\n - \"multi-model-server --start > /dev/null 2>&1\"\n - \"sleep 20s\"\n - \"curl -s -X POST http://localhost:8081/models?url=https://s3.amazonaws.com/model-server/model_archive_1.0/squeezenet_v1.1.mar\"\n - \"curl -s -X POST http://localhost:8081/models?url=https://s3.amazonaws.com/model-server/model_archive_1.0/resnet-18.mar\"\n - \"curl -s -X PUT http://localhost:8081/models/squeezenet_v1.1?min_worker=1&synchronous=true\"\n - \"curl -s -X PUT http://localhost:8081/models/resnet-18?min_worker=1&synchronous=true\"\n post-process:\n - \"multi-model-server --stop > /dev/null 2>&1\"\n - \"rm kitten.jpg\"\n - module: server_local_monitoring\n ServerLocalClient:\n - interval: 1s\n logging : True\n metrics:\n - total_processes\n - sum_all_file_descriptors\n - sum_all_memory_rss\n - frontend_memory_rss\n - orphans\n\nreporting:\n- module: passfail\n criteria:\n # Inbuilt Criteria\n - success of Inference1<${INFR1_SUCC}, stop as failed\n - success of Inference2<${INFR2_SUCC}, stop as failed\n - avg-rt of Inference1>${INFR1_RT}, stop as failed\n - avg-rt of Inference2>${INFR2_RT}, stop as failed\n # Custom Criteria\n - class: bzt.modules.monitoring.MonitoringCriteria\n subject: ServerLocalClient/total_processes\n condition: '>'\n threshold: ${TOTAL_PROCS}\n timeframe: 10s\n stop : true\n fail : true\n - class: bzt.modules.monitoring.MonitoringCriteria\n subject: ServerLocalClient/sum_all_file_descriptors\n condition: '>'\n threshold: ${TOTAL_FDS}\n timeframe: 5s\n stop : true\n fail : true\n - class: bzt.modules.monitoring.MonitoringCriteria\n subject: ServerLocalClient/sum_all_memory_rss\n condition: '>'\n threshold: ${TOTAL_MEM}\n timeframe: 5s\n stop : true\n fail : true\n - class: bzt.modules.monitoring.MonitoringCriteria\n subject: ServerLocalClient/orphans\n condition: '>'\n threshold: ${TOTAL_ORPHANS}\n timeframe: 1s\n stop : true\n fail : true\n - class: bzt.modules.monitoring.MonitoringCriteria\n subject: ServerLocalClient/frontend_memory_rss\n condition: '>'\n threshold: ${FRNTEND_MEM}\n timeframe: 5s\n stop : true\n fail : true\n" }, { "path": "tests/performance/tests/register_unregister/environments/xlarge.yaml", "content": "---\nsettings:\n env:\n REG_SUCC : 100%\n UNREG_SUCC: 100%\n REG_RT : 15s\n UNREG_RT: 10ms\n TOTAL_PROCS : 1\n TOTAL_FDS : 66\n TOTAL_ORPHANS : 0\n FRNTEND_MEM : 75000000 #75MB" }, { "path": "tests/performance/tests/register_unregister/register_unregister.jmx", "content": "\n\n \n \n \n false\n true\n false\n \n \n \n \n \n \n \n \n \n model\n squeezenet_v1.1\n =\n Model name\n \n \n model_url\n https://s3.amazonaws.com/model-server/model_archive_1.0/squeezenet_v1.1.mar\n URL to model store on s3\n =\n \n \n \n \n \n \n \n \n ${__P(hostname,127.0.0.1)}\n ${__P(port,8443)}\n ${__P(protocol,https)}\n \n \n 6\n \n \n \n \n \n continue\n \n false\n ${__P(loops,4)}\n \n ${__P(threads,10)}\n ${__P(rampup,10)}\n false\n \n \n true\n \n \n \n \n \n \n \n ${__P(management_port,8444)}\n \n \n /models?url=${model_url}\n POST\n true\n false\n true\n false\n \n \n \n \n \n \n \n \n \n \n ${__P(management_port,8444)}\n \n \n /models/${model}\n DELETE\n true\n false\n true\n false\n \n \n \n \n \n \n \n \n\n" }, { "path": "tests/performance/tests/register_unregister/register_unregister.yaml", "content": "---\nexecution:\n- concurrency: 1\n ramp-up: 0s\n# hold-for: 5h\n iterations: 5\n scenario: register_unregister\n\nscenarios:\n register_unregister:\n script: register_unregister.jmx\n\nmodules:\n server_local_monitoring:\n class : metrics_monitoring_inproc.Monitor\n\nservices:\n - module: shellexec\n prepare:\n - \"curl -s -O https://s3.amazonaws.com/model-server/inputs/kitten.jpg\"\n - \"multi-model-server --start > /dev/null 2>&1\"\n - \"sleep 10s\"\n post-process:\n - \"multi-model-server --stop > /dev/null 2>&1\"\n - \"rm kitten.jpg\"\n - module: server_local_monitoring\n ServerLocalClient:\n - interval: 1s\n logging : True\n metrics:\n - total_processes\n - sum_all_file_descriptors\n - frontend_memory_rss\n - orphans\n\nreporting:\n- module: passfail\n criteria:\n # Inbuilt Criteria\n - success of RegisterModel<${REG_SUCC}, stop as failed\n - success of UnregisterModel<${UNREG_SUCC}, stop as failed\n - avg-rt of RegisterModel>${REG_RT}, stop as failed\n - avg-rt of UnregisterModel>${UNREG_RT}, stop as failed\n # Custom Criteria\n - class: bzt.modules.monitoring.MonitoringCriteria\n subject: ServerLocalClient/total_processes\n condition: '>'\n threshold: ${TOTAL_PROCS}\n timeframe: 5s\n stop : true\n fail : true\n - class: bzt.modules.monitoring.MonitoringCriteria\n subject: ServerLocalClient/sum_all_file_descriptors\n condition: '>'\n threshold: ${TOTAL_FDS}\n timeframe: 5s\n stop : true\n fail : true\n - class: bzt.modules.monitoring.MonitoringCriteria\n subject: ServerLocalClient/orphans\n condition: '>'\n threshold: ${TOTAL_ORPHANS}\n timeframe: 1s\n stop : true\n fail : true\n# - class: bzt.modules.monitoring.MonitoringCriteria\n# subject: ServerLocalClient/frontend_memory_rss\n# condition: '>'\n# threshold: ${FRNTEND_MEM}\n# timeframe: 5s\n# stop : true\n# fail : true" }, { "path": "tests/performance/tests/register_unregister_multiple/environments/xlarge.yaml", "content": "---\nsettings:\n env:\n REG_SUCC : 100%\n SCL_UP_SUCC: 100%\n UNREG_SUCC: 100%\n REG_RT : 15s\n SCL_UP_RT: 1.5s\n UNREG_RT: 18ms\n TOTAL_PROCS : 2\n TOTAL_FDS : 73\n FRNTEND_MEM : 120000000 #120MB" }, { "path": "tests/performance/tests/register_unregister_multiple/register_unregister_multiple.jmx", "content": "\n\n \n \n \n false\n true\n false\n \n \n \n \n \n \n \n \n \n model\n squeezenet_v1.1\n =\n Model name\n \n \n model_url\n https://s3.amazonaws.com/model-server/model_archive_1.0/squeezenet_v1.1.mar\n URL to model store on s3\n =\n \n \n scale_up_workers\n 2\n Numer of workers to scale up to\n =\n \n \n \n \n \n \n \n \n ${__P(hostname,127.0.0.1)}\n ${__P(port,8443)}\n ${__P(protocol,https)}\n \n \n 6\n \n \n \n \n \n continue\n \n false\n ${__P(loops,4)}\n \n ${__P(threads,10)}\n ${__P(rampup,10)}\n false\n \n \n true\n \n \n \n \n \n \n \n ${__P(management_port,8444)}\n \n \n /models?url=${model_url}\n POST\n true\n false\n true\n false\n \n \n \n \n \n \n \n \n \n \n ${__P(management_port,8444)}\n \n \n /models/${model}?min_worker=${scale_up_workers}&synchronous=true\n PUT\n true\n false\n true\n false\n \n \n \n \n \n \n \n \n \n \n ${__P(management_port,8444)}\n \n \n /models/${model}\n DELETE\n true\n false\n true\n false\n \n \n \n \n \n \n \n \n\n" }, { "path": "tests/performance/tests/register_unregister_multiple/register_unregister_multiple.yaml", "content": "---\nexecution:\n- concurrency: 1\n ramp-up: 0s\n iterations: 5\n scenario: register_unregister_multiple\n\nscenarios:\n register_unregister_multiple:\n script: register_unregister_multiple.jmx\n\nmodules:\n server_local_monitoring:\n class : metrics_monitoring_inproc.Monitor\n\nservices:\n - module: shellexec\n prepare:\n - \"curl -s -O https://s3.amazonaws.com/model-server/inputs/kitten.jpg\"\n - \"multi-model-server --start > /dev/null 2>&1\"\n - \"sleep 20s\"\n - \"curl -s -X POST http://localhost:8081/models?url=https://s3.amazonaws.com/model-server/model_archive_1.0/resnet-18.mar\"\n post-process:\n - \"multi-model-server --stop > /dev/null 2>&1\"\n - \"rm kitten.jpg\"\n - module: server_local_monitoring\n ServerLocalClient:\n - interval: 1s\n logging : True\n metrics:\n - total_processes\n - sum_all_file_descriptors\n - frontend_memory_rss\n\nreporting:\n- module: passfail\n criteria:\n # Inbuilt Criteria\n - success of RegisterModel<${REG_SUCC}, stop as failed\n - success of ScaleUp<${SCL_UP_SUCC}, stop as failed\n - success of UnregisterModel<${UNREG_SUCC}, stop as failed\n - avg-rt of RegisterModel>${REG_RT}, stop as failed\n - avg-rt of ScaleUp>${SCL_UP_RT}, stop as failed\n - avg-rt of UnregisterModel>${UNREG_RT}, stop as failed\n # Custom Criteria\n - class: bzt.modules.monitoring.MonitoringCriteria\n subject: ServerLocalClient/total_processes\n condition: '>'\n threshold: ${TOTAL_PROCS}\n timeframe: 5s\n stop : true\n fail : true\n - class: bzt.modules.monitoring.MonitoringCriteria\n subject: ServerLocalClient/sum_all_file_descriptors\n condition: '>'\n threshold: ${TOTAL_FDS}\n timeframe: 5s\n stop : true\n fail : true\n# - class: bzt.modules.monitoring.MonitoringCriteria\n# subject: ServerLocalClient/frontend_memory_rss\n# condition: '>'\n# threshold: ${FRNTEND_MEM}\n# timeframe: 5s\n# stop : true\n# fail : true" }, { "path": "tests/performance/tests/scale_down_workers/environments/xlarge.yaml", "content": "---\nsettings:\n env:\n SCL_DWN_SUCC : 100%\n SCL_DWN_RT : 10ms\n TOTAL_PROCS_B4_SCL_DWN : 6\n TOTAL_PROCS_AFTR_SCL_DWN : 4\n TOTAL_WRKRS_B4_SCL_DWN : 4\n TOTAL_WRKRS_AFTR_SCL_DWN : 2\n FRNTEND_FDS : 78\n TOTAL_WRKRS_FDS_B4_SCL_DWN: 38\n TOTAL_WRKRS_FDS_AFTR_SCL_DWN: 23\n FRNTEND_MEM : 290000000 #290MB\n TOTAL_WRKRS_MEM_B4_SCL_DWN : 450000000 #450MB\n TOTAL_WRKRS_MEM_AFTR_SCL_DWN : 210000000 #210MB" }, { "path": "tests/performance/tests/scale_down_workers/scale_down_workers.jmx", "content": "\n\n \n \n \n false\n true\n false\n \n \n \n \n \n \n \n \n \n scale_down_workers\n 2\n Number of workers to scale down to\n =\n \n \n model\n squeezenet_v1.1\n =\n Model name\n \n \n \n \n \n \n \n \n ${__P(hostname,127.0.0.1)}\n ${__P(port,8443)}\n ${__P(protocol,https)}\n \n \n 6\n \n \n \n \n \n continue\n \n false\n ${__P(loops,4)}\n \n ${__P(threads,10)}\n ${__P(rampup,10)}\n false\n \n \n true\n \n \n \n \n \n \n \n ${__P(management_port,8444)}\n \n \n /models/${model}?min_worker=${scale_down_workers}&synchronous=true\n PUT\n true\n false\n true\n false\n \n \n \n \n \n \n \n \n\n" }, { "path": "tests/performance/tests/scale_down_workers/scale_down_workers.yaml", "content": "---\nexecution:\n- concurrency: 10\n ramp-up: 1s\n hold-for: 30s\n scenario: scaledown\n\nscenarios:\n scaledown:\n script: scale_down_workers.jmx\n\nmodules:\n server_local_monitoring:\n class : metrics_monitoring_inproc.Monitor\n\nservices:\n - module: shellexec\n prepare:\n - \"multi-model-server --start > /dev/null 2>&1\"\n - \"sleep 20s\"\n - \"curl -s -X POST http://localhost:8081/models?url=https://s3.amazonaws.com/model-server/model_archive_1.0/squeezenet_v1.1.mar\"\n - \"curl -s -X PUT http://localhost:8081/models/squeezenet_v1.1?min_worker=4&synchronous=true\"\n post-process:\n - \"multi-model-server --stop > /dev/null 2>&1\"\n - module: server_local_monitoring\n ServerLocalClient:\n - interval: 1s\n logging : True\n metrics:\n - total_processes\n - total_workers\n - frontend_file_descriptors\n - sum_workers_file_descriptors\n - frontend_memory_rss\n - sum_workers_memory_rss\n\nreporting:\n- module: passfail\n criteria:\n # Inbuilt Criteria\n - success of ScaleDown<${SCL_DWN_SUCC}, stop as failed\n - avg-rt of ScaleDown>${SCL_DWN_RT}, stop as failed\n # Custom Criteria\n - class: bzt.modules.monitoring.MonitoringCriteria\n subject: ServerLocalClient/total_processes\n condition: '>'\n threshold: ${TOTAL_PROCS_B4_SCL_DWN}\n timeframe: 1s\n stop : true\n fail : true\n - class: bzt.modules.monitoring.MonitoringCriteria\n subject: ServerLocalClient/total_processes\n condition: '<'\n threshold: ${TOTAL_PROCS_AFTR_SCL_DWN}\n timeframe: 1s\n stop : true\n fail : true\n - class: bzt.modules.monitoring.MonitoringCriteria\n subject: ServerLocalClient/total_workers\n condition: '>'\n threshold: ${TOTAL_WRKRS_B4_SCL_DWN}\n timeframe: 1s\n stop : true\n fail : true\n - class: bzt.modules.monitoring.MonitoringCriteria\n subject: ServerLocalClient/total_workers\n condition: '<'\n threshold: ${TOTAL_WRKRS_AFTR_SCL_DWN}\n timeframe: 1s\n stop : true\n fail : true\n - class: bzt.modules.monitoring.MonitoringCriteria\n subject: ServerLocalClient/frontend_file_descriptors\n condition: '>'\n threshold: ${FRNTEND_FDS}\n timeframe: 5s\n stop : true\n fail : true\n - class: bzt.modules.monitoring.MonitoringCriteria\n subject: ServerLocalClient/sum_workers_file_descriptors\n condition: '>'\n threshold: ${TOTAL_WRKRS_FDS_B4_SCL_DWN}\n timeframe: 5s\n stop : true\n fail : true\n - class: bzt.modules.monitoring.MonitoringCriteria\n subject: ServerLocalClient/sum_workers_file_descriptors\n condition: '<'\n threshold: ${TOTAL_WRKRS_FDS_AFTR_SCL_DWN}\n timeframe: 5s\n stop : true\n fail : true\n# - class: bzt.modules.monitoring.MonitoringCriteria\n# subject: ServerLocalClient/frontend_memory_rss\n# condition: '>'\n# threshold: ${FRNTEND_MEM}\n# timeframe: 5s\n# stop : true\n# fail : true\n - class: bzt.modules.monitoring.MonitoringCriteria\n subject: ServerLocalClient/sum_workers_memory_rss\n condition: '>'\n threshold: ${TOTAL_WRKRS_MEM_B4_SCL_DWN}\n timeframe: 5s\n stop : true\n fail : true\n - class: bzt.modules.monitoring.MonitoringCriteria\n subject: ServerLocalClient/sum_workers_memory_rss\n condition: '<'\n threshold: ${TOTAL_WRKRS_MEM_AFTR_SCL_DWN}\n timeframe: 5s\n stop : true\n fail : true" }, { "path": "tests/performance/tests/scale_up_workers/environments/xlarge.yaml", "content": "---\nsettings:\n env:\n SCL_UP_SUCC : 100%\n SCL_UP_RT : 10ms\n TOTAL_PROCS_AFTR_SCL_UP : 6\n TOTAL_PROCS_B4_SCL_UP : 3\n TOTAL_WRKRS_AFTR_SCL_UP : 4\n TOTAL_WRKRS_B4_SCL_UP : 1\n FRNTEND_FDS : 88\n TOTAL_WRKRS_FDS_AFTR_SCL_UP : 38\n TOTAL_WRKRS_FDS_B4_SCL_UP : 11\n FRNTEND_MEM : 290000000 #290MB\n TOTAL_WRKRS_MEM_AFTR_SCL_UP : 450000000 #450MB\n TOTAL_WRKRS_MEM_B4_SCL_UP : 115000000 #115MB" }, { "path": "tests/performance/tests/scale_up_workers/scale_up_workers.jmx", "content": "\n\n \n \n \n false\n true\n false\n \n \n \n \n \n \n \n \n \n scale_up_workers\n 4\n =\n Number of workers to scale up to\n \n \n model\n squeezenet_v1.1\n =\n Model name\n \n \n \n \n \n \n \n \n ${__P(hostname,127.0.0.1)}\n ${__P(port,8443)}\n ${__P(protocol,https)}\n \n \n 6\n \n \n \n \n \n continue\n \n false\n ${__P(loops,4)}\n \n ${__P(threads,10)}\n ${__P(rampup,10)}\n false\n \n \n true\n \n \n \n \n \n \n \n ${__P(management_port,8444)}\n \n \n /models/${model}?min_worker=${scale_up_workers}&synchronous=true\n PUT\n true\n false\n true\n false\n \n \n \n \n \n \n \n \n\n" }, { "path": "tests/performance/tests/scale_up_workers/scale_up_workers.yaml", "content": "---\nexecution:\n- concurrency: 10\n ramp-up: 1s\n hold-for: 30s\n scenario: scaleup\n\nscenarios:\n scaleup:\n script: scale_up_workers.jmx\n\nmodules:\n server_local_monitoring:\n class : metrics_monitoring_inproc.Monitor\n\nservices:\n - module: shellexec\n prepare:\n - \"multi-model-server --start > /dev/null 2>&1\"\n - \"sleep 20s\"\n - \"curl -s -X POST http://localhost:8081/models?url=https://s3.amazonaws.com/model-server/model_archive_1.0/squeezenet_v1.1.mar\"\n - \"curl -s -X PUT http://localhost:8081/models/squeezenet_v1.1?min_worker=1&synchronous=true\"\n post-process:\n - \"multi-model-server --stop > /dev/null 2>&1\"\n - module: server_local_monitoring\n ServerLocalClient:\n - interval: 1s\n logging : True\n metrics:\n - total_processes\n - total_workers\n - frontend_file_descriptors\n - sum_workers_file_descriptors\n - frontend_memory_rss\n - sum_workers_memory_rss\n\nreporting:\n- module: passfail\n criteria:\n # Inbuilt Criteria\n - success of ScaleUp<${SCL_UP_SUCC}, stop as failed\n - avg-rt of ScaleUp>${SCL_UP_RT}, stop as failed\n # Custom Criteria\n - class: bzt.modules.monitoring.MonitoringCriteria\n subject: ServerLocalClient/total_processes\n condition: '>'\n threshold: ${TOTAL_PROCS_AFTR_SCL_UP}\n timeframe: 1s\n stop : true\n fail : true\n - class: bzt.modules.monitoring.MonitoringCriteria\n subject: ServerLocalClient/total_processes\n condition: '<'\n threshold: ${TOTAL_PROCS_B4_SCL_UP}\n timeframe: 1s\n stop : true\n fail : true\n - class: bzt.modules.monitoring.MonitoringCriteria\n subject: ServerLocalClient/total_workers\n condition: '>'\n threshold: ${TOTAL_WRKRS_AFTR_SCL_UP}\n timeframe: 1s\n stop : true\n fail : true\n - class: bzt.modules.monitoring.MonitoringCriteria\n subject: ServerLocalClient/total_workers\n condition: '<'\n threshold: ${TOTAL_WRKRS_B4_SCL_UP}\n timeframe: 1s\n stop : true\n fail : true\n - class: bzt.modules.monitoring.MonitoringCriteria\n subject: ServerLocalClient/frontend_file_descriptors\n condition: '>'\n threshold: ${FRNTEND_FDS}\n timeframe: 5s\n stop : true\n fail : true\n - class: bzt.modules.monitoring.MonitoringCriteria\n subject: ServerLocalClient/sum_workers_file_descriptors\n condition: '>'\n threshold: ${TOTAL_WRKRS_FDS_AFTR_SCL_UP}\n timeframe: 5s\n stop : true\n fail : true\n - class: bzt.modules.monitoring.MonitoringCriteria\n subject: ServerLocalClient/sum_workers_file_descriptors\n condition: '<'\n threshold: ${TOTAL_WRKRS_FDS_B4_SCL_UP}\n timeframe: 5s\n stop : true\n fail : true\n# - class: bzt.modules.monitoring.MonitoringCriteria\n# subject: ServerLocalClient/frontend_memory_rss\n# condition: '>'\n# threshold: ${FRNTEND_MEM}\n# timeframe: 5s\n# stop : true\n# fail : true\n - class: bzt.modules.monitoring.MonitoringCriteria\n subject: ServerLocalClient/sum_workers_memory_rss\n condition: '>'\n threshold: ${TOTAL_WRKRS_MEM_AFTR_SCL_UP}\n timeframe: 5s\n stop : true\n fail : true\n - class: bzt.modules.monitoring.MonitoringCriteria\n subject: ServerLocalClient/sum_workers_memory_rss\n condition: '<'\n threshold: ${TOTAL_WRKRS_MEM_B4_SCL_UP}\n timeframe: 5s\n stop : true\n fail : true" }, { "path": "tests/performance/utils/__init__.py", "content": "#!/usr/bin/env python\n\n# Copyright 2020 Amazon.com, Inc. or its affiliates. All Rights Reserved.\n# Licensed under the Apache License, Version 2.0 (the \"License\").\n# You may not use this file except in compliance with the License.\n# A copy of the License is located at\n# http://www.apache.org/licenses/LICENSE-2.0\n# or in the \"license\" file accompanying this file. This file is distributed\n# on an \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either\n# express or implied. See the License for the specific language governing\n# permissions and limitations under the License.\n\n\"\"\"\nRun Tarus test cases and generate the Junit XML report\n\"\"\"\n# pylint: disable=redefined-builtin\n\nfrom .fs import get_sub_dirs\nfrom .timer import Timer\nfrom .pyshell import run_process\n" }, { "path": "tests/performance/utils/fs.py", "content": "#!/usr/bin/env python\n\n# Copyright 2020 Amazon.com, Inc. or its affiliates. All Rights Reserved.\n# Licensed under the Apache License, Version 2.0 (the \"License\").\n# You may not use this file except in compliance with the License.\n# A copy of the License is located at\n# http://www.apache.org/licenses/LICENSE-2.0\n# or in the \"license\" file accompanying this file. This file is distributed\n# on an \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either\n# express or implied. See the License for the specific language governing\n# permissions and limitations under the License.\n\n\"\"\"\nFile system utilities\n\"\"\"\n# pylint: disable=redefined-builtin, logging-format-interpolation, dangerous-default-value\nimport logging\nimport sys\nimport os\nimport glob\n\nlogger = logging.getLogger(__name__)\nlogging.basicConfig(stream=sys.stdout, format=\"%(message)s\", level=logging.INFO)\n\n\ndef get_sub_dirs(dir, exclude_list=['comp_data'], include_pattern='*', exclude_pattern=None):\n \"\"\"Utility method to get list of folders in a directory\"\"\"\n dir = dir.strip()\n if not os.path.exists(dir):\n msg = \"The path {} does not exit\".format(dir)\n logger.error(\"The path {} does not exit\".format(dir))\n raise Exception(msg)\n\n pattern_list = glob.glob(dir + \"/\" + include_pattern)\n exclude_pattern_list = glob.glob(dir + \"/\" + exclude_pattern) if exclude_pattern is not None else []\n return list([x for x in os.listdir(dir) if os.path.isdir(dir + \"/\" + x)\n and x not in exclude_list\n and dir + \"/\" + x in pattern_list\n and dir + \"/\" + x not in exclude_pattern_list])\n" }, { "path": "tests/performance/utils/pyshell.py", "content": "#!/usr/bin/env python\n\n# Copyright 2020 Amazon.com, Inc. or its affiliates. All Rights Reserved.\n# Licensed under the Apache License, Version 2.0 (the \"License\").\n# You may not use this file except in compliance with the License.\n# A copy of the License is located at\n# http://www.apache.org/licenses/LICENSE-2.0\n# or in the \"license\" file accompanying this file. This file is distributed\n# on an \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either\n# express or implied. See the License for the specific language governing\n# permissions and limitations under the License.\n\n\"\"\"\nRun shell command utilities\n\"\"\"\n# pylint: disable=redefined-builtin, logging-format-interpolation\n\nimport logging\nimport sys\nimport os\nimport subprocess\n\nlogger = logging.getLogger(__name__)\nlogging.basicConfig(stream=sys.stdout, format=\"%(message)s\", level=logging.INFO)\n\n\ndef run_process(cmd, wait=True):\n \"\"\"Utility method to run the shell commands\"\"\"\n logger.info(\"running command : %s\", cmd)\n\n if wait:\n os.environ[\"PYTHONUNBUFFERED\"] = \"1\"\n process = subprocess.Popen(cmd, stdout=subprocess.PIPE, stderr=subprocess.STDOUT,\n shell=True)\n lines = []\n while True:\n line = process.stdout.readline().decode('utf-8').rstrip()\n if not line:\n break\n lines.append(line)\n logger.info(line)\n\n return process.returncode, '\\n'.join(lines)\n else:\n process = subprocess.Popen(cmd, shell=True)\n return process.returncode, ''\n" }, { "path": "tests/performance/utils/timer.py", "content": "#!/usr/bin/env python\n\n# Copyright 2020 Amazon.com, Inc. or its affiliates. All Rights Reserved.\n# Licensed under the Apache License, Version 2.0 (the \"License\").\n# You may not use this file except in compliance with the License.\n# A copy of the License is located at\n# http://www.apache.org/licenses/LICENSE-2.0\n# or in the \"license\" file accompanying this file. This file is distributed\n# on an \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either\n# express or implied. See the License for the specific language governing\n# permissions and limitations under the License.\n\n\"\"\"\nTimer utilities\n\"\"\"\n# pylint: disable=redefined-builtin\n\nimport logging\nimport sys\nimport time\n\nlogger = logging.getLogger(__name__)\nlogging.basicConfig(stream=sys.stdout, format=\"%(message)s\", level=logging.INFO)\n\n\nclass Timer(object):\n \"\"\"\n Helper context manager class to capture time diff\n \"\"\"\n def __init__(self, description):\n self.description = description\n\n def __enter__(self):\n self.start = int(time.time())\n return self\n\n def __exit__(self, type, value, traceback):\n logger.info(\"%s: %ss\", self.description, self.diff())\n\n def diff(self):\n return int(time.time()) - self.start\n" } ]