[ { "path": ".github/workflows/onpush.yml", "content": "name: test\n\non:\n push:\n branches-ignore:\n - 'master' # the branch will be tested before push to master\n tags-ignore:\n - '*.*'\n paths-ignore:\n - 'README.md' # if we change documentation only, we don't need to run tests\n\njobs:\n test-e2e-azure-github-actions:\n\n runs-on: ubuntu-latest\n strategy:\n max-parallel: 4\n matrix:\n python-version: [3.7.5]\n\n steps:\n - uses: actions/checkout@v1\n\n - name: Set up Python ${{ matrix.python-version }}\n uses: actions/setup-python@v2\n with:\n python-version: ${{ matrix.python-version }}\n\n - name: Install pip\n run: |\n python -m pip install --upgrade pip\n\n - name: Install test dependencies\n run: |\n pip install -r requirements.txt" }, { "path": ".github/workflows/onrelease.yml", "content": "on:\n push:\n # Sequence of patterns matched against refs/tags\n tags:\n - 'v*' # Push events to matching v*, i.e. v1.0, v20.15.10\n\nname: release\n\njobs:\n build:\n name: Create Release\n runs-on: ubuntu-latest\n steps:\n - name: Checkout code\n uses: actions/checkout@v2\n\n - name: Create Release\n id: create_release\n uses: actions/create-release@v1\n env:\n GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }} # This token is provided by Actions, you do not need to create your own token\n with:\n tag_name: ${{ github.ref }}\n release_name: Release ${{ github.ref }}\n body: |\n Release for version ${{ github.ref }}. Please check CHANGELOG.md for more information.\n draft: false\n prerelease: false\n" }, { "path": ".gitignore", "content": "# Byte-compiled / optimized / DLL files\n__pycache__/\n*.py[cod]\n*$py.class\n\n# Distribution / packaging\n*.egg-info/\nbuild\ndist\n\n# Unit test / coverage reports\n.coverage\ncoverage.xml\njunit/*\nhtmlcov/*\n\n# Caches\n.pytest_cache/\n\n# VSCode\n.vscode/\n\n# Idea\n.idea/\n*.iml\n\n# MacOS\n.DS_Store\n\n# aux mlruns\nmlruns/\n\n# secrets\n.env" }, { "path": "CHANGELOG.md", "content": "# Changelog\n\n## Release 1.0.1\n\nThis release provides `init_adapter` method to make `dbx execute` method more user-friendly.\n\n## Release 1.0.2\n\nThis release provides hotfix for incorrect reference of `init_conf` and adds proper testing checks for github workflows.\n\n## Release 1.0.3\n\nThis release adds more compatibility with win-based development environments, as well as extensive tests for win-based launches.\n\n## Release 1.0.4\n\nAdded support for picking configuration properties from environment variable. \nFixed issue with non-existent lockfile. \n\n## Release 1.0.5\n\nMinor fixes in the dbx behaviour.\n\n## Release 1.0.6\n\nFixed multiple issues with run status checks:\n- status check code is now unified int one method\n- `--existing-runs=cancel` instabilities fixed\n- `--trace` stucks in case of skipped status fixed\n- proper exit code for failed integration tests\n\n## Release 1.0.7\n\n- Since dbx is moved to public, no more whl file is needed. whl file is deleted from the repository, as well as all references to it.\n\n## Release 1.0.8\n\n- Introduced support for Google Cloud\n- project template code is formatted with black for better readability\n\n## Release 1.0.9\n\n- Add installation of `dbx` to the `unit-requirements.txt` file\n- Switch generated top-level folder name to project_name variable, not project_slug.\n- Add `.coveragerc` to the generated project\n- Explicitly include support for `dbutils` in the `common.py` file\n\n## Release 1.0.10\n\n- Remove `init_adapter` logic, now all parameters are directly passed from the `deployment.json` file.\n\n## Release 1.0.11\n\n- Fixed issue with project name in Gitlab CI." }, { "path": "CONTRIBUTING.md", "content": "# Contributing\n\n## Legal\n\nThank you for your interest in contributing to the `cicd-templates` project (the “Project”). In order to clarify the intellectual property license granted with Contributions from any person or entity who contributes to the Project, Databricks, Inc. (\"Databricks\") must have a Contributor License Agreement (CLA) on file that has been signed by each such Contributor (or if an entity, an authorized representative of such entity). This license is for your protection as a Contributor as well as the protection of Databricks and its users; it does not change your rights to use your own Contributions for any other purpose.\nYou may sign this CLA either on your own behalf (with respect to any Contributions that are owned by you) and/or on behalf of an entity (the \"Corporation\") (with respect to any Contributions that are owned by such Corporation (e.g., those Contributions you make during the performance of your employment duties to the Corporation)). Please mark the corresponding box below.\nYou accept and agree to the following terms and conditions for Your present and future Contributions submitted to Databricks. Except for the licenses granted herein to Databricks, You reserve all right, title, and interest in and to Your Contributions.\n1. Definitions.\n\"You\" (or \"Your\") shall mean the copyright owner or legal entity authorized by the copyright owner that is making this Agreement with Databricks. For legal entities, the entity making a Contribution and all other entities that control, are controlled by, or are under common control with that entity are considered to be a single Contributor. For the purposes of this definition, \"control\" means (i) the power, direct or indirect, to cause the direction or management of such entity, whether by contract or otherwise, or (ii) ownership of fifty percent (50%) or more of the outstanding shares, or (iii) beneficial ownership of such entity.\n\"Contribution\" shall mean the code, documentation or any original work of authorship, including any modifications or additions to an existing work, that is submitted by You to Databricks for inclusion in, or documentation of, any of the products owned or managed by Databricks, including the Project, whether on, before or after the date You sign this CLA. For the purposes of this definition, \"submitted\" means any form of electronic, verbal, or written communication sent to Databricks or its representatives, including but not limited to communication on electronic mailing lists, source code control systems (e.g., Github), and issue tracking systems that are managed by, or on behalf of, Databricks for the purpose of discussing and improving the Project, but excluding communication that is conspicuously marked or otherwise designated in writing by You as \"Not a Contribution.\"\n2. Grant of Copyright License. Subject to the terms and conditions of this Agreement, You hereby grant to Databricks a perpetual, worldwide, non-exclusive, no-charge, royalty-free, irrevocable copyright license to reproduce, prepare derivative works of, publicly display, publicly perform, sublicense (through multiple tiers), and distribute Your Contributions and such derivative works. For the avoidance of doubt, and without limitation, this includes, at our option, the right to sublicense this license to recipients or users of any products or services (including software) distributed or otherwise made available (e.g., by SaaS offering) by Databricks (each, a “Downstream Recipient”).\n3. Grant of Patent License. Subject to the terms and conditions of this Agreement, You hereby grant to Databricks a perpetual, worldwide, non-exclusive, no-charge, royalty-free, irrevocable, sublicensable (through multiple tiers) (except as stated in this section) patent license to make, have made, use, offer to sell, sell, import, and otherwise transfer the Contribution in whole or in part, alone or in combination with any other products or services (including for the avoidance of doubt the Project), where such license applies only to those patent claims licensable by You that are necessarily infringed by Your Contribution(s) alone or by combination of Your Contribution(s) with the Project to which such Contribution(s) was submitted. For the avoidance of doubt, and without limitation, this includes, at our option, the right to sublicense this license to Downstream Recipients.\n4. Authorized Users. If you are signing this CLA on behalf of a Corporation, you may also add additional designated employees of the Corporation who will be covered by this CLA without the need to separately sign it (“Authorized Users”). Your Primary Point of Contact (you or the individual specified below) may add additional Authorized Users at any time by contacting Databricks at cla@databricks.com (or such other method as Databricks informs you).\n5. Representations. You represent that:\n 1. You are legally entitled to grant the above licenses, and, if You are signing on behalf of a Corporation and have added any Authorized Users, You represent further that each employee of the Corporation designated by You is authorized to submit Contributions on behalf of the Corporation;\n 2. each of Your Contributions is Your original creation;\n 3. to your knowledge, Your Contributions do not infringe or otherwise misappropriate the intellectual property rights of a third person; and\n 4. you will not assert any moral rights in your Contribution against us or any Downstream Recipients.\n6. Support. You are not expected to provide support for Your Contributions, except to the extent You desire to provide support. You may provide support for free, for a fee, or not at all. Unless required by applicable law or agreed to in writing, and except as specified above, You provide Your Contributions on an \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied, including, without limitation, any warranties or conditions of TITLE, MERCHANTABILITY, or FITNESS FOR A PARTICULAR PURPOSE.\n7. Notification. It is your responsibility to notify Databricks when any change is required to the list of Authorized Users, or to the Corporation's Primary Point of Contact with Databricks. You agree to notify Databricks of any facts or circumstances of which you become aware that would make the representations or warranties herein inaccurate in any respect.\n8. This CLA is governed by the laws of the State of California and applicable U.S. Federal law. Any choice of law rules will not apply.\n\nPlease check one of the applicable statement below. Please do NOT mark both statements:\n* ___ I am signing on behalf of myself as an individual and no other person or entity, including my employer, has or will have rights with respect my Contributions.\n* ___ I am signing on behalf of my employer or a legal entity and I have the actual authority to contractually bind such entity (the Corporation).\n\n\n| Name*: | |\n|-----------------------------------------------------------------------------------------------------------|---|\n| Corporation Entity Name (if applicable): | |\n| Title or Role (if applicable): | |\n| Mailing Address*: | |\n| Email*: | |\n| Signature*: | |\n| Date*: | |\n| Github Username (if applicable): | |\n| Primary Point of Contact (if not you) (please provide name and email and Github username, if applicable): | |\n| Authorized Users (please list Github usernames):** | |\n\n- \\* Required field\n- ** Please note that Authorized Users may not be immediately be granted authorization to submit Contributions; should more than one individual attempt to sign a CLA on behalf of a Corporation, the first such CLA will apply and later CLAs will be deemed void.\n" }, { "path": "LICENSE", "content": "Databricks Labs CI/CD Templates\n\nCopyright (2020) Databricks, Inc.\n\nThis library (the \"Software\") may not be used except in connection with the Licensee's use of the Databricks Platform Services pursuant \nto an Agreement (defined below) between Licensee (defined below) and Databricks, Inc. (\"Databricks\"). The Object Code version of the \nSoftware shall be deemed part of the Downloadable Services under the Agreement, or if the Agreement does not define Downloadable Services, \nSubscription Services, or if neither are defined then the term in such Agreement that refers to the applicable Databricks Platform \nServices (as defined below) shall be substituted herein for �Downloadable Services.� Licensee's use of the Software must comply at \nall times with any restrictions applicable to the Downlodable Services and Subscription Services, generally, and must be used in \naccordance with any applicable documentation. For the avoidance of doubt, the Software constitutes Databricks Confidential Information\nunder the Agreement.\nAdditionally, and notwithstanding anything in the Agreement to the contrary: \n* THE SOFTWARE IS PROVIDED \"AS IS\", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES \n OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE \n LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR \n IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.\n* you may view, make limited copies of, and may compile the Source Code version of the Software into an Object Code version of the\n Software. For the avoidance of doubt, you may not make derivative works of Software (or make any any changes to the Source Code \n version of the unless you have agreed to separate terms with Databricks permitting such modifications (e.g., a contribution license\n agreement)).\nIf you have not agreed to an Agreement or otherwise do not agree to these terms, you may not use the Software or view, copy or compile\nthe Source Code of the Software.\nThis license terminates automatically upon the termination of the Agreement or Licensee's breach of these terms. Additionally, \nDatabricks may terminate this license at any time on notice. Upon termination, you must permanently delete the Software and all\ncopies thereof (including the Source Code).\nAgreement: the agreement between Databricks and Licensee governing the use of the Databricks Platform Services, which shall be, with\nrespect to Databricks, the Databricks Terms of Service located at www.databricks.com/termsofservice, and with respect to Databricks\nCommunity Edition, the Community Edition Terms of Service located at www.databricks.com/ce-termsofuse, in each case unless Licensee \nhas entered into a separate written agreement with Databricks governing the use of the applicable Databricks Platform Services.\nDatabricks Platform Services: the Databricks services or the Databricks Community Edition services, according to where the Software is used.\nLicensee: the user of the Software, or, if the Software is being used on behalf of a company, the company.\nObject Code: is version of the Software produced when an interpreter or a compiler translates the Source Code into recognizable and \nexecutable machine code.\nSource Code: the human readable portion of the Software.\n" }, { "path": "Makefile", "content": "VERSION = 1.0.11\n\nrelease:\n\tgit add .\n\tgit tag -a v$(VERSION) -m \"Release tag for version $(VERSION)\"\n\tgit push origin --tags" }, { "path": "NOTICE", "content": "mlflow-deployments\n\nCopyright (2020) Databricks, Inc.\n\n\nThis Software includes software developed at Databricks (https://www.databricks.com/) and its use is subject to the included LICENSE file.\n\n\nAdditionally, this Software contains code from the following open source projects:\n\n1) MLflow - https://mlflow.org/\n2) Databricks CLI - https://github.com/databricks/databricks-cli\n3) Cookiecutter Data Sciense - https://github.com/drivendata/cookiecutter-data-science\n\n[mlflow-deployments - License]\n" }, { "path": "README.md", "content": "# [DEPRECATED] Databricks Labs CI/CD Templates\n\nThis repository provides a template for automated Databricks CI/CD pipeline creation and deployment.\n\n> **_NOTE:_** This repository is **deprecated** and provided for maintenance purposes only. Please use the [dbx init](https://dbx.readthedocs.io/en/latest/templates/python_basic.html) functionality instead.\n\n\n\nTable of Contents\n=================\n\n * [Databricks Labs CI/CD Templates](#databricks-labs-cicd-templates)\n * [Table of Contents](#table-of-contents)\n * [Sample project structure (with GitHub Actions)](#sample-project-structure-with-github-actions)\n * [Sample project structure (with Azure DevOps)](#sample-project-structure-with-azure-devops)\n * [Sample project structure (with GitLab)](#sample-project-structure-with-gitlab)\n * [Note on dbx](#note-on-dbx)\n * [Quickstart](#quickstart)\n * [Local steps](#local-steps)\n * [Setting up CI/CD pipeline on GitHub Actions](#setting-up-cicd-pipeline-on-github-actions)\n * [Setting up CI/CD pipeline on Azure DevOps](#setting-up-cicd-pipeline-on-azure-devops)\n * [Setting up CI/CD pipeline on Gitlab](#setting-up-cicd-pipeline-on-gitlab)\n * [Deployment file structure](#deployment-file-structure)\n * [Different deployment types](#different-deployment-types)\n * [Deployment for Run Submit API](#deployment-for-run-submit-api)\n * [Deployment for Run Now API](#deployment-for-run-now-api)\n * [Troubleshooting](#troubleshooting)\n * [FAQ](#faq)\n * [Legal Information](#legal-information)\n * [Feedback](#feedback)\n * [Contributing](#contributing)\n * [Kudos](#kudos)\n \n## Sample project structure (with GitHub Actions)\n```\n.\n├── .dbx\n│   └── project.json\n├── .github\n│   └── workflows\n│   ├── onpush.yml\n│   └── onrelease.yml\n├── .gitignore\n├── README.md\n├── conf\n│   ├── deployment.json\n│   └── test\n│   └── sample.json\n├── pytest.ini\n├── sample_project\n│   ├── __init__.py\n│   ├── common.py\n│   └── jobs\n│   ├── __init__.py\n│   └── sample\n│   ├── __init__.py\n│   └── entrypoint.py\n├── setup.py\n├── tests\n│   ├── integration\n│   │   └── sample_test.py\n│   └── unit\n│   └── sample_test.py\n└── unit-requirements.txt\n```\n\nSome explanations regarding structure:\n- `.dbx` folder is an auxiliary folder, where metadata about environments and execution context is located.\n- `sample_project` - Python package with your code (the directory name will follow your project name)\n- `tests` - directory with your package tests\n- `conf/deployment.json` - deployment configuration file. Please read the [following section](#deployment-file-structure) for a full reference.\n- `.github/workflows/` - workflow definitions for GitHub Actions\n\n## Sample project structure (with Azure DevOps)\n```\n.\n├── .dbx\n│   └── project.json\n├── .gitignore\n├── README.md\n├── azure-pipelines.yml\n├── conf\n│   ├── deployment.json\n│   └── test\n│   └── sample.json\n├── pytest.ini\n├── sample_project_azure_dev_ops\n│   ├── __init__.py\n│   ├── common.py\n│   └── jobs\n│   ├── __init__.py\n│   └── sample\n│   ├── __init__.py\n│   └── entrypoint.py\n├── setup.py\n├── tests\n│   ├── integration\n│   │   └── sample_test.py\n│   └── unit\n│   └── sample_test.py\n└── unit-requirements.txt\n```\n\nSome explanations regarding structure:\n- `.dbx` folder is an auxiliary folder, where metadata about environments and execution context is located.\n- `sample_project_azure_dev_ops` - Python package with your code (the directory name will follow your project name)\n- `tests` - directory with your package tests\n- `conf/deployment.json` - deployment configuration file. Please read the [following section](#deployment-file-structure) for a full reference.\n- `azure-pipelines.yml` - Azure DevOps Pipelines workflow definition\n\n## Sample project structure (with GitLab)\n```\n.\n├── .dbx\n│   └── project.json\n├── .gitignore\n├── README.md\n├── .gitlab-ci.yml\n├── conf\n│   ├── deployment.json\n│   └── test\n│   └── sample.json\n├── pytest.ini\n├── sample_project_gitlab\n│   ├── __init__.py\n│   ├── common.py\n│   └── jobs\n│   ├── __init__.py\n│   └── sample\n│   ├── __init__.py\n│   └── entrypoint.py\n├── setup.py\n├── tests\n│   ├── integration\n│   │   └── sample_test.py\n│   └── unit\n│   └── sample_test.py\n└── unit-requirements.txt\n```\n\nSome explanations regarding structure:\n- `.dbx` folder is an auxiliary folder, where metadata about environments and execution context is located.\n- `sample_project_gitlab` - Python package with your code (the directory name will follow your project name)\n- `tests` - directory with your package tests\n- `conf/deployment.json` - deployment configuration file. Please read the [following section](#deployment-file-structure) for a full reference.\n- `.gitlab-ci.yml` - GitLab CI/CD workflow definition\n\n## Note on dbx\n\n> **_NOTE:_** \n[dbx](https://github.com/databrickslabs/dbx) is a CLI tool for advanced Databricks jobs management. \nIt can be used separately from cicd-templates, and if you would like to preserve your project structure, please refer to dbx documentation on how to use it with customized project structure.\n\n\n## Quickstart\n\n> **_NOTE:_** \nAs a prerequisite, you need to install [databricks-cli](https://docs.databricks.com/dev-tools/cli/index.html) with a [configured profile](https://docs.databricks.com/dev-tools/cli/index.html#set-up-authentication).\nIn this instruction we're based on [Databricks Runtime 7.3 LTS ML](https://docs.databricks.com/release-notes/runtime/7.3ml.html). \nIf you don't need to use ML libraries, we still recommend to use ML-based version due to [`%pip` magic support](https://docs.databricks.com/libraries/notebooks-python-libraries.html).\n\n### Local steps\nPerform the following actions in your development environment:\n- Create new conda environment and activate it:\n```bash\nconda create -n python=3.7.5\nconda activate \n```\n- If you would like to be able to run local unit tests, you'll need JDK. If you don't have one, It can be installed via:\n```\nconda install -c anaconda \"openjdk=8.0.152\"\n```\n- Install cookiecutter and path:\n```bash\npip install cookiecutter path\n```\n- Create new project using cookiecutter template:\n```\ncookiecutter https://github.com/databrickslabs/cicd-templates\n```\n- Install development dependencies:\n```bash\npip install -r unit-requirements.txt\n```\n- Install generated package in development mode:\n```\npip install -e .\n```\n- In the generated directory you'll have a sample job with testing and launch configurations around it.\n- Launch and debug your code on an interactive cluster via the following command. Job name could be found in `conf/deployment.json`:\n```\ndbx execute --cluster-name= --job=\n```\n- Make your first deployment from the local machine:\n```\ndbx deploy\n```\n- Launch your first pipeline as a new separate job, and trace the job status. Job name could be found in `conf/deployment.json`:\n```\ndbx launch --job --trace\n```\n- For an in-depth local development and unit testing guidance, please refer to a generated `README.md` in the root of the project.\n\n### Setting up CI/CD pipeline on GitHub Actions\n\n- Create a new repository on GitHub\n- Configure `DATABRICKS_HOST` and `DATABRICKS_TOKEN` secrets for your project in [GitHub UI](https://docs.github.com/en/free-pro-team@latest/actions/reference/encrypted-secrets)\n- Add a remote origin to the local repo\n- Push the code \n- Open the GitHub Actions for your project to verify the state of the deployment pipeline\n\n### Setting up CI/CD pipeline on Azure DevOps\n\n- Create a new repository on GitHub\n- Connect the repository to Azure DevOps\n- Configure `DATABRICKS_HOST` and `DATABRICKS_TOKEN` secrets for your project in [Azure DevOps](https://docs.microsoft.com/en-us/azure/devops/pipelines/release/azure-key-vault?view=azure-devops). Note that secret variables must be mapped to env as mentioned [here](https://docs.microsoft.com/en-us/azure/devops/pipelines/process/variables?view=azure-devops&tabs=yaml%2Cbatch#secret-variables) using the syntax `env:` for example:\n```\nvariables:\n- group: Databricks-environment\nstages:\n...\n...\n - script: |\n dbx deploy\n env:\n DATABRICKS_TOKEN: $(DATABRICKS_TOKEN)\n```\n- Add a remote origin to the local repo\n- Push the code \n- Open the Azure DevOps UI to check the deployment status \n\n### Setting up CI/CD pipeline on Gitlab\n\n- Create a new repository on Gitlab\n- Configure `DATABRICKS_HOST` and `DATABRICKS_TOKEN` secrets for your project in [GitLab UI](https://docs.gitlab.com/ee/ci/variables/#create-a-custom-variable-in-the-ui)\n- Add a remote origin to the local repo\n- Push the code \n- Open the GitLab CI/CD UI to check the deployment status \n\n \n## Deployment file structure\nA sample deployment file could be found in a generated project.\n\nGeneral file structure could look like this:\n```json\n{\n \"\": {\n \"jobs\": [\n {\n \"name\": \"sample_project-sample\",\n \"existing_cluster_id\": \"some-cluster-id\", \n \"libraries\": [],\n \"max_retries\": 0,\n \"spark_python_task\": {\n \"python_file\": \"sample_project/jobs/sample/entrypoint.py\",\n \"parameters\": [\n \"--conf-file\",\n \"conf/test/sample.json\"\n ]\n }\n }\n ]\n }\n}\n```\nPer each environment you could describe any amount of jobs. Job description should follow the [Databricks Jobs API](https://docs.databricks.com/dev-tools/api/latest/jobs.html#create). \n\nHowever, there is some advanced behaviour for a `dbx deploy`.\n\nWhen you run `dbx deploy` with a given deployment file (by default it takes the deployment file from `conf/deployment.json`), the following actions will be performed:\n- Find the deployment configuration in `--deployment-file` (default: `conf/deployment.json`) \n- Build .whl package in a given project directory (could be disabled via `--no-rebuild` option)\n- Add this .whl package to a job definition\n- Add all requirements from `--requirements-file` (default: `requirements.txt`). Step will be skipped if requirements file is non-existent.\n- Create a new job or adjust existing job if the given job name exists. Job will be found by it's name.\n\nImportant thing about referencing is that you can also reference arbitrary local files. This is very handy for `python_file` section.\nIn the example above, the entrypoint file and the job configuration will be added to the job definition and uploaded to `dbfs` automatically. No explicit file upload is needed.\n\n## Different deployment types\n\nDatabricks Jobs API provides two methods for launching a particular workload:\n- [Run Submit API](https://docs.databricks.com/dev-tools/api/latest/jobs.html#runs-submit)\n- [Run Now API](https://docs.databricks.com/dev-tools/api/latest/jobs.html#run-now)\n\nMain logical difference between these methods is that Run Submit API allows to submit a workload directly without creating a job.\nTherefore, we have two deployment types - one for Run Submit API, and one for Run Now API. \n\n### Deployment for Run Submit API\n\nTo deploy only the files and not to override the job definitions, do the following:\n\n```bash\ndbx deploy --files-only\n```\n\nTo launch the file-based deployment:\n```\ndbx launch --as-run-submit --trace\n```\n\nThis type of deployment is handy for working in different branches, and it won't affect the job definition.\n\n### Deployment for Run Now API\n\nTo deploy files and update the job definitions:\n\n```bash\ndbx deploy\n```\n\nTo launch the file-based deployment:\n```\ndbx launch --job=\n```\n\nThis type of deployment shall be mainly used in automated way during new release. \n`dbx deploy` will change the job definition (unless `--files-only` option is provided).\n\n## Troubleshooting\n\n###\n*Q*: When running ```dbx deploy``` I'm getting the following exception ```json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)``` and stack trace:\n```\n...\n File \".../lib/python3.7/site-packages/dbx/utils/common.py\", line 215, in prepare_environment\n experiment = mlflow.get_experiment_by_name(environment_data[\"workspace_dir\"])\n...\n\njson.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)\n```\n\nWhat could be causing it and what is the potential fix?\n\n*A*: \nWe've seen this exception when in the profile the ```host=https://{domain}/?o={orgid}``` format is used for Azure. It is valid for the databricks cli, but not for the API. If that's the cause, once the \"?o={orgid}\" suffix is removed, the problem should be gone.\n\n## FAQ\n\n###\n*Q*: I'm using [poetry](https://python-poetry.org/) for package management. Is it possible to use poetry together with this template?\n\n*A*: \n Yes, it's also possible, but the library management during cluster execution should be performed via `libraries` section of job description. \n You also might need to disable the automatic rebuild for `dbx deploy` and `dbx execute` via `--no-rebuild` option. Finally, the built package should be in wheel format and located in `/dist/` directory.\n\n###\n*Q*: How can I add my Databricks Notebook to the `deployment.json`, so I can create a job out of it?\n \n*A*: \n Please follow [this](https://docs.databricks.com/dev-tools/api/latest/jobs.html#notebooktask) documentation section and add a notebook task definition into the deployment file.\n\n###\n*Q*: Is it possible to use `dbx` for non-Python based projects, for example Scala-based projects?\n\n*A*: \n Yes, it's possible, but the interactive mode `dbx execute` is not yet supported. However, you can just take the `dbx` wheel to your Scala-based project and reference your jar files in the deployment file, so the `dbx deploy` and `dbx launch` commands be available for you.\n\n###\n*Q*: I have a lot of interdependent jobs, and using solely JSON seems like a giant code duplication. What could solve this problem?\n\n*A*: \n You can implement any configuration logic and simply write the output into a custom `deployment.json` file and then pass it via `--deployment-file` option. \n As an example, you can generate your configuration using Python script, or [Jsonnet](https://jsonnet.org/).\n\n###\n*Q*: How can I secure the project environment?\n\n*A*: \nFrom the state serialization perspective, your code and deployments are stored in two separate storages:\n- workspace directory -this directory is stored in your workspace, described per-environment and defined in `.dbx/project.json`, in `workspace_dir` field.\n To control access to this directory, please use [Workspace ACLs](https://docs.databricks.com/security/access-control/workspace-acl.html). \n- artifact location - this location is stored in DBFS, described per-environment and defined in `.dbx/project.json`, in `artifact_location` field.\n To control access this location, please use credentials passthrough (docs for [ADLS](https://docs.microsoft.com/en-us/azure/databricks/security/credential-passthrough/adls-passthrough) and for [S3](https://docs.databricks.com/security/credential-passthrough/index.html)).\n\n###\n*Q*: I would like to use self-hosted (private) pypi repository. How can I configure my deployment and CI/CD pipeline?\n\n*A*: \nTo set up this scenario, there are some settings to be applied:\n- Databricks driver should have network access to your pypi repository\n- Additional step to deploy your package to pypi repo should be configured in CI/CD pipeline\n- Package re-build and generation should be disabled via `--no-rebuild --no-package` arguments for `dbx execute`\n- Package reference should be configured in job description\n\nHere is a sample for `dbx deploy` command:\n```\ndbx deploy --no-rebuild --no-package\n```\n\nSample section to `libraries` configuration:\n```json\n{\n \"pypi\": {\"package\": \"my-package-name==1.0.0\", \"repo\": \"my-repo.com\"}\n}\n```\n\n###\n*Q*: What is the purpose of `init_adapter` method in SampleJob?\n\n*A*: \nThis method should be primarily used for adapting configuration for `dbx execute` based run. \nBy using this method, you can provide an initial configuration in case if `--conf-file` option is not provided. \n\n###\n*Q*: I don't like the idea of hosting the host and token variables into `~/.databrickscfg` file inside the CI pipeline. How can I make this setup more secure?\n\n*A*: \n`dbx` now supports environment variables, provided via `DATABRICKS_HOST` and `DATABRICKS_TOKEN`. \nIf these variables are defined in env, no `~/.databrickscfg` file needed.\n\n## Legal Information\nThis software is provided as-is and is not officially supported by Databricks through customer technical support channels. \nSupport, questions, and feature requests can be communicated through the Issues page of this repo. \nPlease see the legal agreement and understand that issues with the use of this code will not be answered or investigated by Databricks Support.\n\n## Feedback\nIssues with template? Found a bug? Have a great idea for an addition? Feel free to file an issue.\n\n## Contributing\nHave a great idea that you want to add? Fork the repo and submit a PR!\n\n## Kudos\n- Project based on the [cookiecutter datascience project](https://drivendata.github.io/cookiecutter-data-science)\n- README.md ToC generated via [gh-md-toc](https://github.com/ekalinin/github-markdown-toc)\n" }, { "path": "cookiecutter.json", "content": "{\n \"project_name\": \"cicd-sample-project\",\n \"version\": \"0.0.1\",\n \"description\": \"Databricks Labs CICD Templates Sample Project\",\n \"author\": \"\",\n \"cloud\": [\n \"AWS\",\n \"Azure\",\n \"Google Cloud\"\n ],\n \"cicd_tool\": [\n \"GitHub Actions\",\n \"Azure DevOps\",\n \"GitLab\"\n ],\n \"project_slug\": \"{{ cookiecutter.project_name.lower().replace(' ', '_').replace('-', '_') }}\",\n \"workspace_dir\": \"/Shared/dbx/{{cookiecutter.project_slug}}\",\n \"artifact_location\": \"dbfs:/Shared/dbx/projects/{{cookiecutter.project_slug}}\",\n \"profile\": \"DEFAULT\",\n \"_copy_without_render\": [\n \"*.github\"\n ]\n}\n" }, { "path": "hooks/post_gen_project.py", "content": "import json\nimport os\nimport shutil\n\nfrom path import Path\n\ncicd_tool = \"{{cookiecutter.cicd_tool}}\"\ncloud = \"{{cookiecutter.cloud}}\"\nproject_slug = \"{{cookiecutter.project_slug}}\"\nproject_name = \"{{cookiecutter.project_name}}\"\nenvironment = \"default\"\nprofile = \"{{cookiecutter.profile}}\"\nworkspace_dir = \"{{cookiecutter.workspace_dir}}\"\nartifact_location = \"{{cookiecutter.artifact_location}}\"\n\nPROJECT_FILE_CONTENT = {\n \"environments\": {\n environment: {\n \"profile\": profile,\n \"workspace_dir\": workspace_dir,\n \"artifact_location\": artifact_location,\n }\n }\n}\n\nDEPLOYMENT = {\n \"AWS\": {\n environment: {\n \"jobs\": [\n {\n \"name\": \"%s-sample\" % project_name,\n \"new_cluster\": {\n \"spark_version\": \"7.3.x-cpu-ml-scala2.12\",\n \"node_type_id\": \"i3.xlarge\",\n \"aws_attributes\": {\n \"first_on_demand\": 0,\n \"availability\": \"SPOT\",\n },\n \"num_workers\": 2,\n },\n \"libraries\": [],\n \"email_notifications\": {\n \"on_start\": [],\n \"on_success\": [],\n \"on_failure\": [],\n },\n \"max_retries\": 0,\n \"spark_python_task\": {\n \"python_file\": \"%s/jobs/sample/entrypoint.py\" % project_slug,\n \"parameters\": [\"--conf-file\", \"conf/test/sample.json\"],\n },\n },\n {\n \"name\": \"%s-sample-integration-test\" % project_name,\n \"new_cluster\": {\n \"spark_version\": \"7.3.x-cpu-ml-scala2.12\",\n \"node_type_id\": \"i3.xlarge\",\n \"aws_attributes\": {\n \"first_on_demand\": 0,\n \"availability\": \"SPOT\",\n },\n \"num_workers\": 1,\n },\n \"libraries\": [],\n \"email_notifications\": {\n \"on_start\": [],\n \"on_success\": [],\n \"on_failure\": [],\n },\n \"max_retries\": 0,\n \"spark_python_task\": {\n \"python_file\": \"tests/integration/sample_test.py\"\n },\n },\n ]\n }\n },\n \"Azure\": {\n environment: {\n \"jobs\": [\n {\n \"name\": \"%s-sample\" % project_name,\n \"new_cluster\": {\n \"spark_version\": \"7.3.x-cpu-ml-scala2.12\",\n \"node_type_id\": \"Standard_F4s\",\n \"num_workers\": 2,\n },\n \"libraries\": [],\n \"email_notifications\": {\n \"on_start\": [],\n \"on_success\": [],\n \"on_failure\": [],\n },\n \"max_retries\": 0,\n \"spark_python_task\": {\n \"python_file\": \"%s/jobs/sample/entrypoint.py\" % project_slug,\n \"parameters\": [\"--conf-file\", \"conf/test/sample.json\"],\n },\n },\n {\n \"name\": \"%s-sample-integration-test\" % project_name,\n \"new_cluster\": {\n \"spark_version\": \"7.3.x-cpu-ml-scala2.12\",\n \"node_type_id\": \"Standard_F4s\",\n \"num_workers\": 1,\n },\n \"libraries\": [],\n \"email_notifications\": {\n \"on_start\": [],\n \"on_success\": [],\n \"on_failure\": [],\n },\n \"max_retries\": 0,\n \"spark_python_task\": {\n \"python_file\": \"tests/integration/sample_test.py\"\n },\n },\n ]\n }\n },\n \"Google Cloud\": {\n environment: {\n \"jobs\": [\n {\n \"name\": \"%s-sample\" % project_name,\n \"new_cluster\": {\n \"spark_version\": \"7.3.x-scala2.12\",\n \"node_type_id\": \"n1-standard-4\",\n \"num_workers\": 2,\n \"spark_env_vars\": {},\n \"cluster_source\": \"JOB\",\n \"gcp_attributes\": {\"use_preemptible_executors\": True},\n },\n \"libraries\": [],\n \"email_notifications\": {\n \"on_start\": [],\n \"on_success\": [],\n \"on_failure\": [],\n },\n \"max_retries\": 0,\n \"spark_python_task\": {\n \"python_file\": \"%s/jobs/sample/entrypoint.py\" % project_slug,\n \"parameters\": [\"--conf-file\", \"conf/test/sample.json\"],\n },\n },\n {\n \"name\": \"%s-sample-integration-test\" % project_name,\n \"new_cluster\": {\n \"spark_version\": \"7.3.x-scala2.12\",\n \"node_type_id\": \"n1-standard-4\",\n \"num_workers\": 2,\n \"spark_env_vars\": {},\n \"cluster_source\": \"JOB\",\n \"gcp_attributes\": {\"use_preemptible_executors\": True},\n },\n \"libraries\": [],\n \"email_notifications\": {\n \"on_start\": [],\n \"on_success\": [],\n \"on_failure\": [],\n },\n \"max_retries\": 0,\n \"spark_python_task\": {\n \"python_file\": \"tests/integration/sample_test.py\"\n },\n },\n ]\n }\n },\n}\n\n\ndef replace_vars(file_path: str):\n _path = Path(file_path)\n content = _path.read_text().format(\n project_name=project_name, environment=environment, profile=profile\n )\n _path.write_text(content)\n\n\nclass PostProcessor:\n @staticmethod\n def process():\n\n if cicd_tool == \"GitHub Actions\":\n os.remove(\"azure-pipelines.yml\")\n os.remove(\".gitlab-ci.yml\")\n\n replace_vars(\".github/workflows/onpush.yml\")\n replace_vars(\".github/workflows/onrelease.yml\")\n\n if cicd_tool == \"Azure DevOps\":\n shutil.rmtree(\".github\")\n os.remove(\".gitlab-ci.yml\")\n\n if cicd_tool == \"GitLab\":\n shutil.rmtree(\".github\")\n os.remove(\"azure-pipelines.yml\")\n\n deployment = json.dumps(DEPLOYMENT[cloud], indent=4)\n deployment_file = Path(\"conf/deployment.json\")\n if not deployment_file.parent.exists():\n deployment_file.parent.mkdir()\n deployment_file.write_text(deployment)\n project_file = Path(\".dbx/project.json\")\n if not project_file.parent.exists():\n project_file.parent.mkdir()\n deployment_file.write_text(deployment)\n project_file.write_text(json.dumps(PROJECT_FILE_CONTENT, indent=2))\n Path(\".dbx/lock.json\").write_text(\"{}\")\n os.system(\"git init\")\n\n\nif __name__ == \"__main__\":\n post_processor = PostProcessor()\n post_processor.process()\n" }, { "path": "pytest.ini", "content": "[pytest]\naddopts = -s -p no:warnings\nlog_cli = 1\nlog_cli_level = INFO\nlog_cli_format = [pytest][%(asctime)s][%(levelname)s][%(module)s][%(funcName)s] %(message)s\nlog_cli_date_format = %Y-%m-%d %H:%M:%S\nlog_level = INFO" }, { "path": "requirements.txt", "content": "setuptools\nwheel\ncookiecutter\npathlib\npytest\npath\npython-dotenv\npygithub\nrequests\npynacl\nclick\ndatabricks-cli" }, { "path": "tests/__init__.py", "content": "" }, { "path": "tests/test_e2e_local.py", "content": "import unittest\nfrom .utils import CicdTemplatesTest\nimport logging\n\n\nclass LocalExecuteTest(CicdTemplatesTest):\n def tearDown(self) -> None:\n logging.info(self.project_path)\n\n def test_local_execute_azure(self):\n with self.project_path:\n self.execute_command(\"pip install dbx\")\n self.execute_command(\"dbx deploy\")\n self.execute_command(\n f\"dbx execute --cluster-name=cicd-templates.testing --job={self.project_name}-sample\"\n )\n\n\nif __name__ == \"__main__\":\n unittest.main()\n" }, { "path": "tests/utils.py", "content": "import logging\nimport os\nimport pathlib\nimport shutil\nimport tempfile\nimport unittest\nfrom uuid import uuid4\n\nfrom cookiecutter.main import cookiecutter\nfrom path import Path\n\n\nclass CicdTemplatesTest(unittest.TestCase):\n TEMPLATE_PATH = str(pathlib.Path(\".\").absolute())\n\n def setUp(self) -> None:\n self.test_dir = tempfile.mkdtemp()\n self.project_name = \"cicd_templates_%s\" % str(uuid4()).split(\"-\")[0]\n logging.info(\n f\"Launching test with project name {self.project_name} and test dir {self.test_dir}\"\n )\n self.prepare_repository()\n self.project_path = Path(self.test_dir).joinpath(self.project_name)\n\n def tearDown(self) -> None:\n logging.info(f\"Deleting test directory: {self.test_dir}\")\n shutil.rmtree(self.test_dir)\n\n def prepare_repository(self):\n cookiecutter(\n template=self.TEMPLATE_PATH,\n no_input=True,\n output_dir=self.test_dir,\n extra_context={\n \"project_name\": self.project_name,\n \"cloud\": \"Azure\",\n \"cicd_tool\": \"GitHub Actions\",\n \"profile\": \"dbx-dev-azure\",\n },\n )\n\n def execute_command(self, cmd):\n exit_code = os.system(cmd)\n self.assertEqual(exit_code, 0)\n" }, { "path": "tox.ini", "content": "[flake8]\nmax-line-length = 79\nmax-complexity = 10\n" }, { "path": "utils/profile_creator.py", "content": "# platform independent profile configurator\nimport click\nfrom databricks_cli.configure.cli import DatabricksConfig, update_and_persist_config\n\n\n@click.command()\n@click.option(\"--profile\", required=True, type=str)\n@click.option(\"--host\", required=True, type=str)\n@click.option(\"--token\", required=True, type=str)\ndef configure(profile: str, host: str, token: str):\n new_config = DatabricksConfig.from_token(host, token, False)\n update_and_persist_config(profile, new_config)\n\n\nif __name__ == \"__main__\":\n configure()\n" }, { "path": "{{cookiecutter.project_name}}/.coveragerc", "content": "[run]\nbranch = True\nsource = {{cookiecutter.project_slug}}\n\n[report]\nexclude_lines =\n if self.debug:\n pragma: no cover\n raise NotImplementedError\n if __name__ == .__main__.:\n\nignore_errors = True\nomit =\n tests/*\n setup.py\n # this file is autogenerated by cicd-templates\n {{cookiecutter.project_slug}}/common.py" }, { "path": "{{cookiecutter.project_name}}/.github/workflows/onpush.yml", "content": "name: Test pipeline\n\non:\n push:\n branches:\n - '**'\n tags-ignore:\n - 'v*' # this tag type is used for release pipelines\n\njobs:\n test-pipeline:\n\n runs-on: ubuntu-latest\n strategy:\n max-parallel: 4\n\n env:\n DATABRICKS_HOST: ${{{{ secrets.DATABRICKS_HOST }}}}\n DATABRICKS_TOKEN: ${{{{ secrets.DATABRICKS_TOKEN }}}}\n\n steps:\n - uses: actions/checkout@v1\n\n - name: Set up Python\n uses: actions/setup-python@v2\n with:\n python-version: 3.7.5\n\n - name: Install pip\n run: |\n python -m pip install --upgrade pip\n\n - name: Install dependencies and project in dev mode\n run: |\n pip install -r unit-requirements.txt\n pip install -e .\n\n - name: Run unit tests\n run: |\n echo \"Launching unit tests\"\n pytest tests/unit\n\n - name: Deploy integration test\n run: |\n dbx deploy --jobs={project_name}-sample-integration-test --files-only\n\n - name: Run integration test\n run: |\n dbx launch --job={project_name}-sample-integration-test --as-run-submit --trace\n\n\n\n" }, { "path": "{{cookiecutter.project_name}}/.github/workflows/onrelease.yml", "content": "name: Release pipeline\n\non:\n push:\n tags:\n - 'v*' # Push events to matching v*, i.e. v1.0, v20.15.10\n\n\njobs:\n release-pipeline:\n\n runs-on: ubuntu-latest\n strategy:\n max-parallel: 4\n matrix:\n python-version: [ 3.7 ]\n\n env:\n DATABRICKS_HOST: ${{{{ secrets.DATABRICKS_HOST }}}}\n DATABRICKS_TOKEN: ${{{{ secrets.DATABRICKS_TOKEN }}}}\n\n steps:\n - uses: actions/checkout@v1\n\n - name: Set up Python\n uses: actions/setup-python@v1\n with:\n python-version: 3.7\n\n - name: Install pip\n run: |\n python -m pip install --upgrade pip\n\n - name: Install dependencies and project in dev mode\n run: |\n pip install -r unit-requirements.txt\n\n - name: Deploy the job\n run: |\n dbx deploy --jobs={project_name}-sample\n\n - name: Create Release\n id: create_release\n uses: actions/create-release@v1\n env:\n GITHUB_TOKEN: ${{{{ secrets.GITHUB_TOKEN }}}} # This token is provided by Actions\n with:\n tag_name: ${{{{ github.ref }}}}\n release_name: Release ${{{{ github.ref }}}}\n body: |\n Release for version ${{{{ github.ref }}}}.\n draft: false\n prerelease: false\n\n\n" }, { "path": "{{cookiecutter.project_name}}/.gitignore", "content": "# Byte-compiled / optimized / DLL files\n__pycache__/\n*.py[cod]\n*$py.class\n\n# Distribution / packaging\n*.egg-info/\nbuild\ndist\n\n# Unit test / coverage reports\n.coverage\ncoverage.xml\njunit/*\nhtmlcov/*\n\n# Caches\n.pytest_cache/\n\n# VSCode\n.vscode/\n\n# Idea\n.idea/\n*.iml\n\n# MacOS\n.DS_Store\n\n# Databricks eXtensions\n.dbx/lock.json\n\n# local mlflow files\nmlruns/" }, { "path": "{{cookiecutter.project_name}}/.gitlab-ci.yml", "content": "stages:\n - unit-testing\n - integration-testing\n - release\n\nunit-testing:\n stage: unit-testing\n image: python:3.7-stretch\n script:\n - echo \"Install dependencies\"\n - apt-get update\n - apt-get install default-jdk -y\n - pip install -r unit-requirements.txt\n - pip install -e .\n - echo \"Launching unit tests\"\n - pytest tests/unit\n\nintegration-testing:\n stage: integration-testing\n image: python:3.7-stretch\n script:\n - echo \"Install dependencies\"\n - pip install -r unit-requirements.txt\n - pip install -e .\n - echo \"Deploying tests\"\n - dbx deploy --jobs={{cookiecutter.project_name}}-integration-test --files-only\n - echo \"Running tests\"\n - dbx launch --job={{cookiecutter.project_name}}-integration-test --as-run-submit --trace\n\nrelease:\n stage: release\n image: python:3.7-stretch\n only:\n refs:\n - master\n script:\n - echo \"Install dependencies\"\n - pip install -r unit-requirements.txt\n - pip install -e .\n - echo \"Deploying Job\"\n - dbx deploy --jobs={{cookiecutter.project_name}}-sample\n" }, { "path": "{{cookiecutter.project_name}}/README.md", "content": "# {{cookiecutter.project_name}}\n\nThis is a sample project for Databricks, generated via cookiecutter.\n\nWhile using this project, you need Python 3.X and `pip` or `conda` for package management.\n\n## Installing project requirements\n\n```bash\npip install -r unit-requirements.txt\n```\n\n## Install project package in a developer mode\n\n```bash\npip install -e .\n```\n\n## Testing\n\nFor local unit testing, please use `pytest`:\n```\npytest tests/unit --cov\n```\n\nFor an integration test on interactive cluster, use the following command:\n```\ndbx execute --cluster-name= --job={{cookiecutter.project_name}}-sample-integration-test\n```\n\nFor a test on an automated job cluster, deploy the job files and then launch:\n```\ndbx deploy --jobs={{cookiecutter.project_name}}-sample-integration-test --files-only\ndbx launch --job={{cookiecutter.project_name}}-sample-integration-test --as-run-submit --trace\n```\n\n## Interactive execution and development\n\n1. `dbx` expects that cluster for interactive execution supports `%pip` and `%conda` magic [commands](https://docs.databricks.com/libraries/notebooks-python-libraries.html).\n2. Please configure your job in `conf/deployment.json` file. \n2. To execute the code interactively, provide either `--cluster-id` or `--cluster-name`.\n```bash\ndbx execute \\\n --cluster-name=\"\" \\\n --job=job-name\n```\n\nMultiple users also can use the same cluster for development. Libraries will be isolated per each execution context.\n\n## Preparing deployment file\n\nNext step would be to configure your deployment objects. To make this process easy and flexible, we're using JSON for configuration.\n\nBy default, deployment configuration is stored in `conf/deployment.json`.\n\n## Deployment for Run Submit API\n\nTo deploy only the files and not to override the job definitions, do the following:\n\n```bash\ndbx deploy --files-only\n```\n\nTo launch the file-based deployment:\n```\ndbx launch --as-run-submit --trace\n```\n\nThis type of deployment is handy for working in different branches, not to affect the main job definition.\n\n## Deployment for Run Now API\n\nTo deploy files and update the job definitions:\n\n```bash\ndbx deploy\n```\n\nTo launch the file-based deployment:\n```\ndbx launch --job=\n```\n\nThis type of deployment shall be mainly used from the CI pipeline in automated way during new release.\n\n\n## CICD pipeline settings\n\nPlease set the following secrets or environment variables for your CI provider:\n- `DATABRICKS_HOST`\n- `DATABRICKS_TOKEN`\n\n## Testing and releasing via CI pipeline\n\n- To trigger the CI pipeline, simply push your code to the repository. If CI provider is correctly set, it shall trigger the general testing pipeline\n- To trigger the release pipeline, get the current version from the `{{cookiecutter.project_slug}}/__init__.py` file and tag the current code version:\n```\ngit tag -a v -m \"Release tag for version \"\ngit push origin --tags\n```\n" }, { "path": "{{cookiecutter.project_name}}/azure-pipelines.yml", "content": "variables:\n- group: Databricks-environment\n\ntrigger:\n batch: true\n branches:\n include:\n - '*'\n\n tags:\n include:\n - v*.*\n - prod\n\nstages:\n- stage: onPush\n jobs:\n - job: onPushJob\n pool:\n vmImage: 'ubuntu-18.04'\n\n steps:\n - script: env | sort\n displayName: 'Environment / Context'\n\n - task: UsePythonVersion@0\n displayName: 'Use Python 3.7'\n inputs:\n versionSpec: 3.7\n\n - checkout: self\n persistCredentials: true\n clean: true\n displayName: 'Checkout & Build.Reason: $(Build.Reason) & Build.SourceBranchName: $(Build.SourceBranchName)'\n\n - script: |\n python -m pip install --upgrade pip\n pip install -r unit-requirements.txt\n pip install -e .\n displayName: 'Install dependencies'\n\n - script: |\n pytest tests/unit --junitxml=test-unit.xml\n displayName: 'Run Unit tests'\n\n - script: |\n dbx deploy --jobs={{cookiecutter.project_name}}-sample-integration-test --files-only\n displayName: 'Deploy integration test'\n\n - script: |\n dbx launch --job={{cookiecutter.project_name}}-sample-integration-test --as-run-submit --trace\n displayName: 'Launch integration on test'\n\n - task: PublishTestResults@2\n condition: succeededOrFailed()\n inputs:\n testResultsFormat: 'JUnit'\n testResultsFiles: '**/test-*.xml' \n failTaskOnFailedTests: true\n\n- stage: onRelease\n condition: |\n or(\n startsWith(variables['Build.SourceBranch'], 'refs/heads/releases'),\n startsWith(variables['Build.SourceBranch'], 'refs/tags/v')\n )\n jobs:\n - job: onReleaseJob\n pool:\n vmImage: 'ubuntu-18.04'\n\n steps:\n - script: env | sort\n displayName: 'Environment / Context'\n\n - task: UsePythonVersion@0\n displayName: 'Use Python 3.7'\n inputs:\n versionSpec: 3.7\n\n - checkout: self\n persistCredentials: true\n clean: true\n displayName: 'Checkout & Build.Reason: $(Build.Reason) & Build.SourceBranchName: $(Build.SourceBranchName)'\n\n - script: |\n python -m pip install --upgrade pip\n pip install -r unit-requirements.txt\n pip install -e .\n displayName: 'Install dependencies'\n\n - script: |\n pytest tests/unit --junitxml=test-unit.xml\n displayName: 'Run Unit tests'\n\n - script: |\n dbx deploy --jobs={{cookiecutter.project_name}}-sample\n displayName: 'Deploy the job'\n\n - task: PublishTestResults@2\n condition: succeededOrFailed()\n inputs:\n testResultsFormat: 'JUnit'\n testResultsFiles: '**/test-*.xml' \n failTaskOnFailedTests: true\n" }, { "path": "{{cookiecutter.project_name}}/conf/test/sample.json", "content": "{\n \"output_format\": \"delta\",\n \"output_path\": \"dbfs:/dbx/tmp/test/{{cookiecutter.project_slug}}\"\n}" }, { "path": "{{cookiecutter.project_name}}/pytest.ini", "content": "[pytest]\naddopts = -s -p no:warnings\nlog_cli = 1\nlog_cli_level = INFO\nlog_cli_format = [pytest][%(asctime)s][%(levelname)s][%(module)s][%(funcName)s] %(message)s\nlog_cli_date_format = %Y-%m-%d %H:%M:%S\nlog_level = INFO" }, { "path": "{{cookiecutter.project_name}}/setup.py", "content": "from setuptools import find_packages, setup\nfrom {{cookiecutter.project_slug}} import __version__\n\nsetup(\n name=\"{{cookiecutter.project_slug}}\",\n packages=find_packages(exclude=[\"tests\", \"tests.*\"]),\n setup_requires=[\"wheel\"],\n version=__version__,\n description=\"{{cookiecutter.description}}\",\n author=\"{{cookiecutter.author}}\",\n)\n" }, { "path": "{{cookiecutter.project_name}}/tests/integration/sample_test.py", "content": "import unittest\n\nfrom {{cookiecutter.project_slug}}.jobs.sample.entrypoint import SampleJob\nfrom uuid import uuid4\nfrom pyspark.dbutils import DBUtils # noqa\n\n\nclass SampleJobIntegrationTest(unittest.TestCase):\n def setUp(self):\n\n self.test_dir = \"dbfs:/tmp/tests/sample/%s\" % str(uuid4())\n self.test_config = {\"output_format\": \"delta\", \"output_path\": self.test_dir}\n\n self.job = SampleJob(init_conf=self.test_config)\n self.dbutils = DBUtils(self.job.spark)\n self.spark = self.job.spark\n\n def test_sample(self):\n\n self.job.launch()\n\n output_count = (\n self.spark.read.format(self.test_config[\"output_format\"])\n .load(self.test_config[\"output_path\"])\n .count()\n )\n\n self.assertGreater(output_count, 0)\n\n def tearDown(self):\n self.dbutils.fs.rm(self.test_dir, True)\n\n\nif __name__ == \"__main__\":\n # please don't change the logic of test result checks here\n # it's intentionally done in this way to comply with jobs run result checks\n # for other tests, please simply replace the SampleJobIntegrationTest with your custom class name\n loader = unittest.TestLoader()\n tests = loader.loadTestsFromTestCase(SampleJobIntegrationTest)\n runner = unittest.TextTestRunner(verbosity=2)\n result = runner.run(tests)\n if not result.wasSuccessful():\n raise RuntimeError(\n \"One or multiple tests failed. Please check job logs for additional information.\"\n )\n" }, { "path": "{{cookiecutter.project_name}}/tests/unit/sample_test.py", "content": "import unittest\nimport tempfile\nimport os\nimport shutil\n\nfrom {{cookiecutter.project_slug}}.jobs.sample.entrypoint import SampleJob\nfrom pyspark.sql import SparkSession\nfrom unittest.mock import MagicMock\n\nclass SampleJobUnitTest(unittest.TestCase):\n def setUp(self):\n self.test_dir = tempfile.TemporaryDirectory().name\n self.spark = SparkSession.builder.master(\"local[1]\").getOrCreate()\n self.test_config = {\n \"output_format\": \"parquet\",\n \"output_path\": os.path.join(self.test_dir, \"output\"),\n }\n self.job = SampleJob(spark=self.spark, init_conf=self.test_config)\n\n def test_sample(self):\n # feel free to add new methods to this magic mock to mock some particular functionality\n self.job.dbutils = MagicMock()\n\n self.job.launch()\n\n output_count = (\n self.spark.read.format(self.test_config[\"output_format\"])\n .load(self.test_config[\"output_path\"])\n .count()\n )\n\n self.assertGreater(output_count, 0)\n\n def tearDown(self):\n shutil.rmtree(self.test_dir)\n\n\nif __name__ == \"__main__\":\n unittest.main()\n" }, { "path": "{{cookiecutter.project_name}}/unit-requirements.txt", "content": "setuptools\nwheel\npyspark==3.0.0\npytest\npytest-cov\ndbx" }, { "path": "{{cookiecutter.project_name}}/{{cookiecutter.project_slug}}/__init__.py", "content": "__version__ = \"0.0.1\"\n" }, { "path": "{{cookiecutter.project_name}}/{{cookiecutter.project_slug}}/common.py", "content": "import json\nfrom abc import ABC, abstractmethod\nfrom argparse import ArgumentParser\nfrom logging import Logger\nfrom typing import Dict, Any\n\nfrom pyspark.sql import SparkSession\nimport sys\n\n\n# abstract class for jobs\nclass Job(ABC):\n\n def __init__(self, spark=None, init_conf=None):\n self.spark = self._prepare_spark(spark)\n self.logger = self._prepare_logger()\n self.dbutils = self.get_dbutils()\n if init_conf:\n self.conf = init_conf\n else:\n self.conf = self._provide_config()\n self._log_conf()\n\n @staticmethod\n def _prepare_spark(spark) -> SparkSession:\n if not spark:\n return SparkSession.builder.getOrCreate()\n else:\n return spark\n\n @staticmethod\n def _get_dbutils(spark: SparkSession):\n try:\n from pyspark.dbutils import DBUtils # noqa\n if \"dbutils\" not in locals():\n utils = DBUtils(spark)\n return utils\n else:\n return locals().get(\"dbutils\")\n except ImportError:\n return None\n\n def get_dbutils(self):\n utils = self._get_dbutils(self.spark)\n\n if not utils:\n self.logger.warn(\"No DBUtils defined in the runtime\")\n else:\n self.logger.info(\"DBUtils class initialized\")\n\n return utils\n\n def _provide_config(self):\n self.logger.info(\"Reading configuration from --conf-file job option\")\n conf_file = self._get_conf_file()\n if not conf_file:\n self.logger.info(\n \"No conf file was provided, setting configuration to empty dict.\"\n \"Please override configuration in subclass init method\"\n )\n return {}\n else:\n self.logger.info(\n f\"Conf file was provided, reading configuration from {conf_file}\"\n )\n return self._read_config(conf_file)\n\n @staticmethod\n def _get_conf_file():\n p = ArgumentParser()\n p.add_argument(\"--conf-file\", required=False, type=str)\n namespace = p.parse_known_args(sys.argv[1:])[0]\n return namespace.conf_file\n\n def _read_config(self, conf_file) -> Dict[str, Any]:\n raw_content = \"\".join(\n self.spark.read.format(\"text\").load(conf_file).toPandas()[\"value\"].tolist()\n )\n config = json.loads(raw_content)\n return config\n\n def _prepare_logger(self) -> Logger:\n log4j_logger = self.spark._jvm.org.apache.log4j # noqa\n return log4j_logger.LogManager.getLogger(self.__class__.__name__)\n\n def _log_conf(self):\n # log parameters\n self.logger.info(\"Launching job with configuration parameters:\")\n for key, item in self.conf.items():\n self.logger.info(\"\\t Parameter: %-30s with value => %-30s\" % (key, item))\n\n @abstractmethod\n def launch(self):\n \"\"\"\n Main method of the job.\n :return:\n \"\"\"\n pass\n" }, { "path": "{{cookiecutter.project_name}}/{{cookiecutter.project_slug}}/jobs/__init__.py", "content": "" }, { "path": "{{cookiecutter.project_name}}/{{cookiecutter.project_slug}}/jobs/sample/__init__.py", "content": "" }, { "path": "{{cookiecutter.project_name}}/{{cookiecutter.project_slug}}/jobs/sample/entrypoint.py", "content": "from {{cookiecutter.project_slug}}.common import Job\n\n\nclass SampleJob(Job):\n\n def launch(self):\n self.logger.info(\"Launching sample job\")\n\n listing = self.dbutils.fs.ls(\"dbfs:/\")\n\n for l in listing:\n self.logger.info(f\"DBFS directory: {l}\")\n\n df = self.spark.range(0, 1000)\n\n df.write.format(self.conf[\"output_format\"]).mode(\"overwrite\").save(\n self.conf[\"output_path\"]\n )\n\n self.logger.info(\"Sample job finished!\")\n\n\nif __name__ == \"__main__\":\n job = SampleJob()\n job.launch()\n" } ]