[ { "path": ".claude/commands/release.md", "content": "# DBDemos Release Workflow\n\nYou are helping with the dbdemos release process. This involves bundling demos from the `dbdemos-notebooks` repository, testing them, fixing any issues, and preparing for release.\n\n## ⛔ CRITICAL WARNINGS\n\n1. **NEVER run a release to PyPI by yourself** - Only the human can trigger `./build-and-distribute.sh`\n2. **NEVER commit secrets** - PAT tokens, GitHub tokens must never appear in commits or outputs\n3. **NEVER push directly to main** - Always use feature branches and PRs\n4. **NEVER cleanup workspace resources yourself** - Always ask the human to do cleanup\n\n## 📚 Notebook Code Quality Principles (CRITICAL)\n\n**The dbdemos-notebooks are state-of-the-art examples that customers will reuse.** Code must be:\n\n1. **Clean and minimal** - No unnecessary code, no hacks, no workarounds\n2. **Simple and readable** - Easy to understand for learning purposes\n3. **Safe to re-run** - Notebooks must work when run multiple times (idempotent)\n4. **No error handling hacks** - Don't add try/except blocks to work around specific errors\n5. **No comments explaining errors** - Don't add comments like \"handles BudgetPolicy error\"\n\n### What NOT to do:\n```python\n# BAD - Don't add error handling for specific workspace issues\ntry:\n agents.deploy(...)\nexcept NotFound as e:\n if \"BudgetPolicy\" in str(e):\n # cleanup and retry...\n```\n\n### What TO do instead:\n- If a job fails due to stale data/resources, **ASK THE HUMAN** to clean up the workspace\n- Never attempt cleanup yourself - the human must do it\n- Fix the root cause in the code, not the symptom\n\n### Handling Stale Resource Errors:\n**Note:** The bundler now automatically cleans up schemas before running (via `DROP SCHEMA CASCADE`).\nThis should prevent most stale resource errors. If you still encounter issues:\n\n- `BudgetPolicy not found` → Schema cleanup should fix this, or ask human to delete the serving endpoint\n- `Model version already exists` → Should be fixed by schema cleanup\n- `Endpoint already exists` → Should be fixed by schema cleanup\n- `Table already exists` → Should be fixed by schema cleanup\n\nIf automatic cleanup fails or you need manual intervention, **ask the human** to run:\n```sql\nDROP SCHEMA IF EXISTS main__build. CASCADE;\n```\nOr delete specific resources via the Databricks UI/API.\n\n## Overview\n\nThe dbdemos package bundles notebooks from the `dbdemos-notebooks` repository. The bundling process:\n1. Creates/updates jobs in a Databricks workspace that run the notebooks\n2. Waits for job completion\n3. Downloads executed notebooks with outputs\n4. Packages them into the `dbdemos/bundles/` directory\n\n## Environment Setup\n\nBefore starting, verify these are available:\n- `DATABRICKS_TOKEN` or token in `local_conf_E2TOOL.json`\n- `GITHUB_TOKEN` or token in `local_conf_E2TOOL.json`\n- Workspace: `https://e2-demo-tools.cloud.databricks.com/`\n- dbdemos-notebooks repo at: `../dbdemos-notebooks` (configurable)\n- Test cluster: Matches `cluster_name_pattern` in config (default: \"quentin\")\n\n## AI Release Tools Location\n\nAll AI-powered release tools are in `ai_release/`:\n- `ai_release/bundle.py` - Bundle and test demos\n- `ai_release/run_remote.py` - Execute code on Databricks clusters\n- `ai_release/compute.py` - Remote execution library\n- `ai_release/run_state.py` - Persistent state tracking for runs\n- `ai_release/jobs.py` - Job inspection library (uses Databricks SDK)\n- `ai_release/inspect_jobs.py` - CLI for job inspection\n\n## ⏱️ Important: Job Run Times\n\n**Bundle jobs typically take 15-30 minutes to complete.** Each job runs all notebooks in a demo on a Databricks cluster.\n\n- Do NOT wait synchronously for jobs to complete\n- Start the job, then work on other tasks or let the user know to check back later\n- Use `--status` to check job progress without blocking\n- The state tracking system persists progress across sessions\n\n---\n\n## Part 0: Run State Tracking\n\nThe AI release workflow tracks state persistently in `ai_release/runs/`:\n\n```\nai_release/runs/\n /\n state.json # Overall run state\n /\n status.json # Demo-specific status\n errors.json # Extracted errors from failed runs\n fix_attempts.json # History of fix attempts\n job_output.log # Raw job output\n notes.md # AI notes and observations\n```\n\n### Using Run State in Python\n```python\nfrom ai_release.run_state import get_run_state, get_latest_run\n\n# Get or create state for current commit\nstate = get_run_state()\n\n# Update demo status\nstate.update_demo_status(\"ai-agent\", \"running\", job_id=123, run_id=456)\n\n# Save errors\nstate.save_errors(\"ai-agent\", [{\"cell\": 5, \"error\": \"ImportError...\"}])\n\n# Record a fix attempt\nstate.add_fix_attempt(\"ai-agent\", \"Remove protobuf constraint\", \"ai-fix-ai-agent-pip\", [\"01_create_first_billing_agent.py\"])\n\n# Add notes\nstate.add_note(\"ai-agent\", \"The pip install fails due to protobuf<5 conflict with grpcio-status\")\n\n# Get summary\nprint(state.get_summary())\n\n# Resume from previous session\nstate = get_latest_run()\n```\n\n### When to Use State Tracking\n- Before starting a bundle job: `state.update_demo_status(demo, \"running\", ...)`\n- After job completes: `state.update_demo_status(demo, \"success\")` or `\"failed\"`\n- When extracting errors: `state.save_errors(demo, errors)`\n- When making a fix: `state.add_fix_attempt(demo, description, branch, files)`\n- To add context for future sessions: `state.add_note(demo, note)`\n\n---\n\n## Part 1: Remote Code Execution (Testing Fixes)\n\nBefore committing a fix to dbdemos-notebooks, test it interactively on a cluster.\n\n### List Available Clusters\n```bash\npython ai_release/run_remote.py --list-clusters\n```\n\n### Check/Start the Test Cluster\n```bash\n# Check status\npython ai_release/run_remote.py --cluster-status\n\n# Start if not running (will ask for confirmation)\npython ai_release/run_remote.py --start-cluster --wait-for-cluster\n```\n\n### Execute Code for Testing\n```bash\n# Execute Python code\npython ai_release/run_remote.py --code \"print(spark.version)\"\n\n# Execute SQL\npython ai_release/run_remote.py --code \"SELECT current_catalog()\" --language sql\n\n# Execute a file\npython ai_release/run_remote.py --file path/to/test_script.py\n\n# With longer timeout (default 300s)\npython ai_release/run_remote.py --code \"long_running_code()\" --timeout 600\n```\n\n### Context Reuse (Faster Follow-up Commands)\n```bash\n# First command - save context\npython ai_release/run_remote.py --code \"x = spark.range(100)\" --save-context\n\n# Follow-up commands reuse context (faster, keeps variables)\npython ai_release/run_remote.py --code \"x.count()\" --load-context\n\n# Clear context when done\npython ai_release/run_remote.py --clear-context\n```\n\n---\n\n## Part 2: Bundling Commands\n\n### Check Configuration\n```bash\npython ai_release/bundle.py --check-config\n```\n\n### Check Status of a Demo\n```bash\npython ai_release/bundle.py --demo --status\n```\nThis shows recent job runs, task status, and error details.\n\n### Bundle a Specific Demo (from main)\n```bash\npython ai_release/bundle.py --demo \n```\n\n### Bundle from a Feature Branch\n```bash\npython ai_release/bundle.py --demo --branch \n```\n\n### Force Re-run (ignore diff optimization)\n```bash\npython ai_release/bundle.py --demo --force\n```\n\n### Repair Failed Job (re-run only failed tasks)\n```bash\npython ai_release/bundle.py --demo --repair\n```\nUse this for quick iteration when debugging. After fixing, always do a full re-run.\n\nAdd `--wait` to wait for completion:\n```bash\npython ai_release/bundle.py --demo --repair --wait\n```\n\n### Schema Cleanup (Default: Enabled)\nBy default, the bundler automatically drops the demo schema (`main__build.`) before running.\nThis ensures a clean state and avoids stale resource errors.\n\n```bash\n# Cleanup is enabled by default - these are equivalent:\npython ai_release/bundle.py --demo \npython ai_release/bundle.py --demo --cleanup-schema\n\n# To skip cleanup (not recommended unless debugging):\npython ai_release/bundle.py --demo --no-cleanup-schema\n```\n\n### Bundle All Demos\n```bash\npython ai_release/bundle.py --all\n```\nThis uses GitHub diff API to only run demos with changed files.\n\n### List Available Demos\n```bash\npython ai_release/bundle.py --list-demos\n```\n\n---\n\n## Part 3: Fixing a Failed Demo - Complete Workflow\n\nWhen a demo fails, follow this workflow:\n\n### Step 1: Identify the Error\n```bash\n# Get job status with auto-extracted errors from notebook cells\npython ai_release/inspect_jobs.py --demo \n\n# For full error traces and failing code\npython ai_release/inspect_jobs.py --demo --errors\n\n# List all failed jobs\npython ai_release/inspect_jobs.py --list --failed-only\n```\nThe inspection tool automatically:\n- Fetches the job run details\n- Exports the notebook HTML\n- Extracts cell-level errors with traceback\n- Shows the exact code that failed\n- Suggests a fix workflow\n\nCommon issues:\n- Missing/incompatible dependencies (pip install failures)\n- API changes in Databricks\n- Data schema changes\n- Cluster configuration issues\n\n### Step 2: Test the Fix Interactively (Optional but Recommended)\n\nBefore touching the notebooks, test your fix on a cluster:\n```bash\n# Start cluster if needed\npython ai_release/run_remote.py --start-cluster --wait-for-cluster\n\n# Test your fix code\npython ai_release/run_remote.py --code \"\n# Your fix code here\ndf = spark.read.table('your_table')\n# ...\n\"\n```\n\n### Step 3: Create a Fix Branch in dbdemos-notebooks\n```bash\ncd ../dbdemos-notebooks\ngit checkout main\ngit pull origin main\ngit checkout -b ai-fix--\n```\n\n### Step 4: Make the Fix\nEdit the notebook files in `../dbdemos-notebooks`. The notebooks are `.py` files using Databricks notebook format.\n\n### Step 5: Commit and Push\n```bash\ncd ../dbdemos-notebooks\ngit add .\ngit commit -m \"fix: \"\ngit push origin ai-fix--\n```\n\n### Step 6: Test the Fix (Full Re-run)\n```bash\ncd ../dbdemos\npython ai_release/bundle.py --demo --branch ai-fix-- --force\n```\n\n### Step 7: If Still Failing - Iterate\n```bash\n# Make more fixes in dbdemos-notebooks\ncd ../dbdemos-notebooks\n# ... edit files ...\ngit add . && git commit -m \"fix: additional fixes\" && git push\n\n# Quick test with repair (faster, but use full re-run for final verification)\ncd ../dbdemos\npython ai_release/bundle.py --demo --repair --wait\n\n# Or full re-run if dependencies changed\npython ai_release/bundle.py --demo --branch ai-fix-- --force\n```\n\n### Step 8: Create PR (When Tests Pass)\n```bash\ncd ../dbdemos-notebooks\ngh pr create --title \"fix: \" --body \"## Summary\n- Fixed \n\n## Testing\n- Bundling job passed: \n\n🤖 Generated with Claude Code\"\n```\n\n### Step 9: After PR is Merged - Final Verification\nWait for the human to merge the PR, then:\n```bash\ncd ../dbdemos\npython ai_release/bundle.py --demo --force\n```\nReport the result to the human.\n\n---\n\n## Part 4: Full Release Workflow\n\nWhen all demos are working and you're asked to prepare a release:\n\n### Step 1: Bundle All Demos from Main\n```bash\npython ai_release/bundle.py --all --force\n```\n\n### Step 2: Verify All Passed\nCheck output for any failures. If any failed, fix them first.\n\n### Step 3: Report to Human\nTell the human:\n- All demos bundled successfully\n- Any changes made\n- Ready for PyPI release\n\n### Step 4: Human Runs Release\n**The human will run:** `./build-and-distribute.sh`\n**You must NEVER run this yourself.**\n\n---\n\n## Useful Information\n\n### Demo Path Structure\nDemos are located in paths like:\n- `product_demos/Delta-Lake/delta-lake`\n- `demo-retail/lakehouse-retail-c360`\n- `aibi/aibi-marketing-campaign`\n\n### Job Naming Convention\nJobs are named: `field-bundle_`\n\n### Bundle Config Location\nEach demo has a config at: `/_resources/bundle_config`\n\n### Workspace URLs\n- Jobs: `https://e2-demo-tools.cloud.databricks.com/#job/`\n- Runs: `https://e2-demo-tools.cloud.databricks.com/#job//run/`\n\n### Package Versioning Rules (IMPORTANT)\n\nWhen fixing `%pip install` lines in notebooks, follow these rules for Databricks packages:\n\n**Always use latest (no version pin):**\n- `databricks-langchain` - use latest\n- `databricks-agents` - use latest\n- `databricks-feature-engineering` - use latest (NOT pinned like `==0.12.1`)\n- `databricks-sdk` - use latest\n- `databricks-mcp` - use latest\n\n**Use minimum version (`>=`):**\n- `mlflow>=3.10.1` - minimum version constraint is OK\n\n**Never pin these constraints (they cause conflicts):**\n- `protobuf<5` - REMOVE, conflicts with grpcio-status\n- `cryptography<43` - REMOVE, unnecessary constraint\n\n**Example - BAD:**\n```\n%pip install mlflow>=3.10.1 databricks-feature-engineering==0.12.1 protobuf<5 cryptography<43\n```\n\n**Example - GOOD:**\n```\n%pip install mlflow>=3.10.1 databricks-langchain databricks-agents databricks-feature-engineering\n```\n\n### Common Errors and Fixes\n\n1. **\"couldn't get notebook for run... You probably did a run repair\"**\n - Solution: Do a full re-run with `--force`\n\n2. **\"last job failed for demo X. Can't package\"**\n - Solution: Fix the failing notebook, then re-run\n\n3. **API rate limits (429 errors)**\n - The script auto-retries. If persistent, wait a few minutes.\n\n4. **\"Couldn't pull the repo\"**\n - Git conflicts in workspace. May need manual resolution.\n\n5. **Cluster not running**\n - Use `python ai_release/run_remote.py --start-cluster --wait-for-cluster`\n\n6. **pip install CalledProcessError with protobuf/cryptography conflicts**\n - Remove `protobuf<5` and `cryptography<43` constraints\n - Remove pinned versions like `databricks-feature-engineering==0.12.1`\n - See \"Package Versioning Rules\" above\n\n---\n\n## Files Reference\n\n- `ai_release/inspect_jobs.py` - Job inspection CLI (auto-extracts errors from notebooks)\n- `ai_release/jobs.py` - Job inspection library (uses Databricks SDK)\n- `ai_release/bundle.py` - Main CLI for bundling\n- `ai_release/run_remote.py` - Remote code execution CLI\n- `ai_release/compute.py` - Remote execution library\n- `ai_release/run_state.py` - Persistent state tracking for runs\n- `ai_release/runs/` - Directory containing run state (gitignored)\n- `dbdemos/job_bundler.py` - Job creation and execution\n- `dbdemos/packager.py` - Packaging executed notebooks\n- `local_conf_E2TOOL.json` - Local configuration (gitignored)\n\n## SDK Documentation\n\nDatabricks SDK for Python: https://databricks-sdk-py.readthedocs.io/en/latest/\n- `../dbdemos-notebooks/` - Source notebooks repository\n" }, { "path": ".cursorignore", "content": "bdemos/exceptions/__pycache__\n__pycache__\nlocal_conf_awsevent.json\nlocal_conf_cse2.json\nlocal_conf_gcp.json\nlocal_conf_ioannis.json\nlocal_conf.json\nlocal_conf*\n.eggs\n.DS_Store\nbuild\ndbdemos/minisite/\ndbdemos/bundles\ndist\n*.egg-info/\ndbdemos/__pycache__\n.idea\nsend_to_e2.sh\ntest_package.py\ndbdemos/resources/local_conf.json\nfield-demo\nvenv\ndatabricks-demos.iml\ndist\nconf.json\n.DS_Store\n__pycache__\n.idea\nconfig.json\n\n" }, { "path": ".gitignore", "content": "dbdemos/exceptions/__pycache__\n__pycache__\nupdate-minisite-dbdemos-website.sh\nlocal_conf_awsevent.json\nlocal_conf_cse2.json\nlocal_conf_gcp.json\nlocal_conf_ioannis.json\nlocal_conf.json\nlocal_conf*\n.eggs\n.DS_Store\nbuild\ndbdemos/minisite/\ndbdemos/bundles\ndist\n*.egg-info/\ndbdemos/__pycache__\n.idea\nsend_to_e2.sh\ntest_package.py\ndbdemos/resources/local_conf.json\nfield-demo\nvenv\ndatabricks-demos.iml\nlocal_conf_azure.json\n" }, { "path": ".vscode/launch.json", "content": "{\n // Use IntelliSense to learn about possible attributes.\n // Hover to view descriptions of existing attributes.\n // For more information, visit: https://go.microsoft.com/fwlink/?linkid=830387\n \"version\": \"0.2.0\",\n \"configurations\": [\n {\n \"name\": \"Python Debugger: Launch main\",\n \"type\": \"debugpy\",\n \"request\": \"launch\",\n \"program\": \"${workspaceFolder}/main.py\",\n \"console\": \"integratedTerminal\"\n }\n ]\n}" }, { "path": ".vscode/settings.json", "content": "{\n \"python.testing.unittestArgs\": [\n \"-v\",\n \"-s\",\n \"./test\",\n \"-p\",\n \"test_*.py\"\n ],\n \"python.testing.pytestEnabled\": false,\n \"python.testing.unittestEnabled\": true,\n \"python-envs.defaultEnvManager\": \"ms-python.python:conda\",\n \"python-envs.defaultPackageManager\": \"ms-python.python:conda\",\n \"python-envs.pythonProjects\": []\n}" }, { "path": "CLAUDE.md", "content": "# CLAUDE.md\n\nThis file provides guidance to Claude Code (claude.ai/code) when working with code in this repository.\n\n## ⛔ CRITICAL: NEVER Release to PyPI\n\n**Claude Code must NEVER run `./build-and-distribute.sh` or release to PyPI.**\n\nOnly the human maintainer can trigger a PyPI release. When demos are ready:\n1. Report to the human that bundling is complete\n2. Wait for the human to run the release script manually\n\n## ⛔ CRITICAL: Never Commit Secrets\n\n**NEVER include PAT tokens, GitHub tokens, or any credentials in:**\n- Commits\n- PR descriptions\n- Tool outputs shown to user\n- Log messages\n\nTokens are stored in `local_conf.json` (gitignored) or environment variables.\n\n## CRITICAL: Do Not Modify Bundle and Minisite Directories\n\n**NEVER search, read, edit, or modify files under these directories unless explicitly asked:**\n- `dbdemos/bundles/` - Contains packaged demo bundles (generated artifacts)\n- `dbdemos/minisite/` - Contains generated minisite content\n\nThese directories contain packaged/generated demo content that should only be modified through the bundling workflow (`job_bundler.py` → `packager.py`). Direct edits to these files will be overwritten during the next bundling process and can break demo installations.\n\n**Work on the source code in the core modules instead** (`installer.py`, `packager.py`, `job_bundler.py`, etc.) or on the source repository (`dbdemos-notebooks`).\n\n## Project Overview\n\n`dbdemos` is a Python toolkit for installing and packaging Databricks demos. It automates deployment of complete demo environments including notebooks, Spark Declarative Pipeline (SDP) pipelines, DBSQL dashboards, workflows, ML models, and AI/BI Genie spaces. The project serves two main purposes:\n\n1. **End-user library**: Users install demos via `pip install dbdemos` and call `dbdemos.install('demo-name')`\n2. **Demo packaging system**: Maintainers package demos from source repositories (usually `dbdemos-notebooks`) into distributable bundles\n\n## Architecture\n\n### Core Components\n\n- **installer.py**: Main installation engine that deploys demos to Databricks workspaces\n - Creates clusters, SDP pipelines, workflows, dashboards, and ML models\n - Handles resource templating (replacing {{CURRENT_USER}}, {{DEMO_FOLDER}}, etc.)\n - Manages demo lifecycle from download to deployment\n\n- **job_bundler.py**: Manages the demo bundling workflow\n - Scans repositories for demos with `_resources/bundle_config` files\n - Executes pre-run jobs to generate notebook outputs\n - Tracks execution state and commit history to avoid redundant runs\n\n- **packager.py**: Packages demos into distributable bundles\n - Downloads notebooks (with or without pre-run results)\n - Extracts Lakeview dashboards from workspace\n - Processes notebook content (removes build tags, updates paths)\n - Generates minisite HTML for [dbdemos.ai](https://www.dbdemos.ai)\n\n- **dbdemos.py**: User-facing API layer providing `help()`, `list_demos()`, `install()` functions\n\n- **conf.py**: Configuration management including `DBClient` for Databricks REST API calls\n\n- **installer_*.py modules**: Specialized installers for different resource types:\n - `installer_workflows.py`: Job/workflow deployment\n - `installer_dashboard.py`: DBSQL dashboard installation\n - `installer_genie.py`: AI/BI Genie space setup\n - `installer_repos.py`: Repository management\n\n- **notebook_parser.py**: Parses and transforms notebook JSON/HTML content\n\n### Demo Bundle Structure\n\nEach demo lives in `dbdemos/bundles/{demo-name}/` with:\n- `_resources/bundle_config`: JSON configuration defining demo metadata, notebooks, pipelines, workflows, dashboards\n- Notebook files (`.html` format, pre-run with cell outputs)\n- `_resources/dashboards/*.lvdash.json`: Dashboard definitions\n\nBundle configs use template keys that get replaced during installation:\n- `{{CURRENT_USER}}`: Installing user's email\n- `{{CURRENT_USER_NAME}}`: Sanitized username\n- `{{DEMO_FOLDER}}`: Installation path\n- `{{DEMO_NAME}}`: Demo identifier\n- `{{TODAY}}`: Current date\n\nDemos are sourced from external repositories (typically `databricks-demos/dbdemos-notebooks`) and bundled into this package for distribution.\n\n## Common Development Commands\n\n### Building the Package\n\n```bash\n# Build wheel distribution\npython setup.py clean --all bdist_wheel\n\n# Build script (used locally)\n./build.sh\n```\n\n### Testing\n\n```bash\n# Run all tests\npytest\n\n# Run specific test file\npytest test/test_installer.py\n\n# Run specific test\npytest test/test_installer.py::TestInstaller::test_method_name\n```\n\n### Bundling Demos (Maintainer Workflow)\n\nCreate a `local_conf.json` file with workspace credentials (see `local_conf_example.json`):\n\n```json\n{\n \"username\": \"user@example.com\",\n \"url\": \"https://workspace.cloud.databricks.com\",\n \"org_id\": \"1234567890\",\n \"pat_token\": \"dapi...\",\n \"repo_staging_path\": \"/Repos/user@example.com\",\n \"repo_name\": \"dbdemos-notebooks\",\n \"repo_url\": \"https://github.com/databricks-demos/dbdemos-notebooks\",\n \"branch\": \"master\",\n \"github_token\": \"ghp_...\"\n}\n```\n\nThen use `main.py` to bundle demos:\n\n```python\nfrom dbdemos.job_bundler import JobBundler\nfrom dbdemos.packager import Packager\n\nbundler = JobBundler(conf)\nbundler.reset_staging_repo(skip_pull=False)\nbundler.add_bundle(\"product_demos/delta-lake\") # or use load_bundles_conf() to discover all\nbundler.start_and_wait_bundle_jobs(force_execution=False)\n\npackager = Packager(conf, bundler)\npackager.package_all()\n```\n\nSee `test_demo.py` for a complete bundling example.\n\n### Distribution and Release\n\n```bash\n# Full release process (bumps version, builds, uploads to PyPI, creates GitHub releases)\n./build-and-distribute.sh\n```\n\nThis script:\n1. Verifies GitHub CLI authentication and repository access\n2. Auto-increments version in `setup.py` and `dbdemos/__init__.py`\n3. Builds wheel package\n4. Uploads to PyPI via `twine`\n5. Creates release branch and pull request\n6. Creates GitHub releases on multiple repositories (`dbdemos`, `dbdemos-notebooks`, `dbdemos-dataset`, `dbdemos-resources`)\n\n## Key Implementation Details\n\n### Dynamic Link Replacement\n\nNotebooks contain special attributes in HTML links that get replaced during installation:\n- `dbdemos-pipeline-id=\"pipeline-id\"`: Links to SDP pipelines\n- `dbdemos-workflow-id=\"workflow-id\"`: Links to workflows\n- `dbdemos-dashboard-id=\"dashboard-id\"`: Links to dashboards\n\nThe installer updates these links with actual resource IDs/URLs after creation.\n\n### Resource Creation Flow\n\n1. Parse bundle configuration\n2. Create/update Git repo if specified\n3. Create demo cluster (with auto-termination)\n4. Install notebooks to workspace\n5. Create SDP pipelines\n6. Create workflows\n7. Create DBSQL dashboards\n8. Create Genie spaces (for AI/BI demos)\n9. Update notebook links to point to created resources\n10. Track installation metrics\n\n### Cluster Configuration\n\nDefault cluster configs are in `dbdemos/resources/`:\n- `default_cluster_config.json`: Standard demo cluster\n- `default_test_job_conf.json`: Job cluster configuration\n- Cloud-specific variants for AWS/Azure/GCP\n\nDemos can override cluster settings in their bundle config under the `cluster` key.\n\n### Multi-Cloud Support\n\nThe project supports AWS, Azure, and GCP. Cloud-specific configurations include:\n- Instance type selection\n- Storage paths (S3/ADLS/GCS)\n- Authentication mechanisms\n- DBR version selection\n\nCloud is detected automatically from workspace or specified via `cloud` parameter in `install()`.\n\n### Serverless Support\n\nSome demos support serverless compute. Set `serverless=True` when installing to use:\n- Serverless SDP pipelines\n- Serverless SQL warehouses\n- Serverless notebooks (where supported)\n\n## Testing Considerations\n\n- Tests use local configuration files (see `local_conf_*.json` examples)\n- Tests require a Databricks workspace with appropriate permissions\n- Most tests are in the `test/` directory\n- `test_demo.py` in root is for bundling workflow testing\n\n## Data Collection\n\nBy default, dbdemos collects usage metrics (views, installations) to improve demo quality. This can be disabled by setting `Tracker.enable_tracker = False` in `tracker.py`. No PII is collected; only aggregate usage data and org IDs.\n\n## Important Constraints\n\n- Users need cluster creation, SDP pipeline creation, and DBSQL dashboard permissions\n- Unity Catalog demos require a UC metastore\n- Some demos have resource quotas (compute, storage)\n- Pre-run notebooks require job execution in staging workspace\n- Dashboard API has rate limits (especially on GCP workspaces)\n\n## Claude Code Release Workflow\n\nFor the full release workflow, use the `/release` command which provides detailed instructions.\n\n### Quick Reference\n\n**Remote Execution** (`ai_release/run_remote.py`) - Test fixes before committing:\n```bash\n# List clusters\npython ai_release/run_remote.py --list-clusters\n\n# Start/check cluster\npython ai_release/run_remote.py --start-cluster --wait-for-cluster\n\n# Execute code\npython ai_release/run_remote.py --code \"print(spark.version)\"\n\n# Execute SQL\npython ai_release/run_remote.py --code \"SELECT 1\" --language sql\n```\n\n**Job Inspection** (`ai_release/inspect_jobs.py`) - Auto-extracts errors from notebook:\n```bash\n# List all jobs with status\npython ai_release/inspect_jobs.py --list\n\n# List only failed jobs\npython ai_release/inspect_jobs.py --list --failed-only\n\n# Get details for a demo (auto-fetches errors if failed)\npython ai_release/inspect_jobs.py --demo \n\n# Show full error traces and code\npython ai_release/inspect_jobs.py --demo --errors\n```\n\n**Bundle CLI** (`ai_release/bundle.py`):\n```bash\n# Check demo status\npython ai_release/bundle.py --demo --status\n\n# Bundle from main\npython ai_release/bundle.py --demo \n\n# Bundle from feature branch\npython ai_release/bundle.py --demo --branch \n\n# Repair failed job (quick iteration)\npython ai_release/bundle.py --demo --repair\n\n# Force full re-run\npython ai_release/bundle.py --demo --force\n\n# Bundle all demos\npython ai_release/bundle.py --all\n```\n\n**Fix Workflow Summary**:\n1. Inspect errors: `python ai_release/inspect_jobs.py --demo --errors`\n2. Test fix interactively: `python ai_release/run_remote.py --code \"...\"`\n3. Create fix branch in `../dbdemos-notebooks`: `ai-fix--`\n4. Make fix, commit, push\n5. Test: `--branch ai-fix-... --force`\n6. If fails, iterate with `--repair` or `--force`\n7. Create PR when green\n8. Human merges PR\n9. Final verification from main: `--force`\n10. Human runs `./build-and-distribute.sh`\n\n**GitHub CLI Account Switch** - Use the public account for PRs:\n```bash\n# List accounts\ngh auth status\n\n# Switch to public account (for creating PRs on public repos)\ngh auth switch --user QuentinAmbard\n\n# Switch to enterprise account (quentin-ambard_data is EMU, can't create PRs on public repos)\ngh auth switch --user quentin-ambard_data\n```\n\n**PR Status Verification** - Always check PR state before assuming:\n```bash\n# Check if PR is merged before running tests from main\ngh pr view --json state,mergedAt\n\n# Never assume a PR is not merged - always verify first\n```\n\n### Environment\n\n- **Workspace**: `https://e2-demo-tools.cloud.databricks.com/`\n- **Config**: `local_conf_E2TOOL.json` (primary) or environment variables\n- **Notebooks repo**: `../dbdemos-notebooks` (configurable)\n- **Test cluster**: Matches `cluster_name_pattern` in config (default: \"quentin\")\n\n### Key Files\n\n- `ai_release/inspect_jobs.py` - Job inspection CLI (auto-extracts errors)\n- `ai_release/jobs.py` - Job inspection library (uses Databricks SDK)\n- `ai_release/bundle.py` - Bundle CLI for demo packaging\n- `ai_release/run_remote.py` - Remote code execution on clusters\n- `ai_release/compute.py` - Remote execution library\n- `.claude/commands/release.md` - Full release workflow documentation\n- `dbdemos/job_bundler.py` - Job creation and execution logic\n- `dbdemos/packager.py` - Packaging logic\n- `local_conf.json` - Local configuration (gitignored, contains secrets)\n\n### Databricks SDK\n\nAll Databricks API operations use the Python SDK: https://databricks-sdk-py.readthedocs.io/en/latest/\n" }, { "path": "LICENSE", "content": "Copyright (2022) Databricks, Inc.\n\nThis library (the \"Software\") may not be used except in connection with the Licensee's use of the Databricks Platform Services pursuant to an Agreement (defined below) between Licensee (defined below) and Databricks, Inc. (\"Databricks\"). The Object Code version of the Software shall be deemed part of the Downloadable Services under the Agreement, or if the Agreement does not define Downloadable Services, Subscription Services, or if neither are defined then the term in such Agreement that refers to the applicable Databricks Platform Services (as defined below) shall be substituted herein for “Downloadable Services.” Licensee's use of the Software must comply at all times with any restrictions applicable to the Downlodable Services and Subscription Services, generally, and must be used in accordance with any applicable documentation. For the avoidance of doubt, the Software constitutes Databricks Confidential Information under the Agreement.\n\nAdditionally, and notwithstanding anything in the Agreement to the contrary:\n\nTHE SOFTWARE IS PROVIDED \"AS IS\", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.\nyou may view, make limited copies of, and may compile the Source Code version of the Software into an Object Code version of the Software. For the avoidance of doubt, you may not make derivative works of Software (or make any any changes to the Source Code version of the unless you have agreed to separate terms with Databricks permitting such modifications (e.g., a contribution license agreement)).\nIf you have not agreed to an Agreement or otherwise do not agree to these terms, you may not use the Software or view, copy or compile the Source Code of the Software.\n\nThis license terminates automatically upon the termination of the Agreement or Licensee's breach of these terms. Additionally, Databricks may terminate this license at any time on notice. Upon termination, you must permanently delete the Software and all copies thereof (including the Source Code).\n\nAgreement: the agreement between Databricks and Licensee governing the use of the Databricks Platform Services, which shall be, with respect to Databricks, the Databricks Terms of Service located at www.databricks.com/termsofservice, and with respect to Databricks Community Edition, the Community Edition Terms of Service located at www.databricks.com/ce-termsofuse, in each case unless Licensee has entered into a separate written agreement with Databricks governing the use of the applicable Databricks Platform Services.\n\nDatabricks Platform Services: the Databricks services or the Databricks Community Edition services, according to where the Software is used.\n\nLicensee: the user of the Software, or, if the Software is being used on behalf of a company, the company.\n\nObject Code: is version of the Software produced when an interpreter or a compiler translates the Source Code into recognizable and executable machine code.\n\nSource Code: the human readable portion of the Software." }, { "path": "MANIFEST.in", "content": "recursive-include dbdemos/bundles *\nrecursive-include dbdemos/template *\nrecursive-include dbdemos/resources *\n" }, { "path": "NOTICE", "content": "Copyright (2022) Databricks, Inc.\n## License\nThis Software includes software developed at Databricks (https://www.databricks.com/) and its use is subject to the included LICENSE file.\n\nThis Software contains code from the following open source projects, licensed under the Apache 2.0 license:\n\npsf/requests - https://github.com/psf/requests\nCopyright 2019 Kenneth Reitz\n\n## Data collection\nTo improve users experience and dbdemos asset quality, dbdemos sends report usage and capture views in the installed notebook (usually in the first cell) and other assets like dashboards. This information is captured for product improvement only and not for marketing purpose, and doesn't contain PII information. By using `dbdemos` and the assets it provides, you consent to this data collection. If you wish to disable it, you can set `Tracker.enable_tracker` to False in the `tracker.py` file.\n\n## Resource creation\nTo simplify your experience, `dbdemos` will create and start for you resources. As example, a demo could start (not exhaustive):\n- A cluster to run your demo\n- A Delta Live Table Pipeline to ingest data\n- A DBSQL endpoint to run DBSQL dashboard\n- An ML model\n\nWhile `dbdemos` does its best to limit the consumption and enforce resource auto-termination, you remain responsible for the resources created and the potential consumption associated.\n\n## Catalog/Database created\ndbdemos will try to create catalogs & databases (schemas). Demos are using the hive_metastore or UC catalogs. dbdemos will try to use the dbdemos catalog when possible.\n\nPermissions / ownership can be granted to all users (account users) in these datasets.\n\n## Support\nDatabricks does not offer official support for `dbdemos` and the associated assets.\n" }, { "path": "README.md", "content": "# dbdemos\n\nDBDemos is a toolkit to easily install Lakehouse demos for Databricks.\n\n**Looking for the dbdemos notebooks and content?** Access [https://github.com/databricks-demos/dbdemos](https://github.com/databricks-demos/dbdemos-notebooks)?\n\nSimply deploy & share demos on any workspace. dbdemos is packaged with a list of demos:\n\n- Lakehouse, end-to-end demos (ex: Lakehouse Retail Churn)\n- Product demos (ex: Delta Live Table, CDC, ML, DBSQL Dashboard, MLOps...)\n\n**Please visit [dbdemos.ai](https://www.dbdemos.ai) to explore all our demos.**\n\n## Installation \n**Do not clone the repo, just pip install dbdemos wheel:**\n\n```\n%pip install dbdemos\n```\n\n## Usage within Databricks\n\nSee [demo video](https://drive.google.com/file/d/12Iu50r7hlawVN01eE_GoUKBQ4kvUrR56/view?usp=sharing) \n```\nimport dbdemos\ndbdemos.help()\ndbdemos.list_demos()\n\ndbdemos.install('lakehouse-retail-c360', path='./', overwrite = True)\n```\n\n![Dbdemos install](https://github.com/databricks-demos/dbdemos/raw/main/resources/dbdemos-screenshot.png)\n\n## Requirements\n\n`dbdemos` requires the current user to have:\n* Cluster creation permission\n* SDP Pipeline creation permission\n* DBSQL dashboard & query creation permission\n* For UC demos: Unity Catalog metastore must be available (demo will be installed but won't work) \n\n\n## Features\n\n* Load demo notebooks (pre-run) to the given path\n* Start job to load dataset based on demo requirement\n* Start demo cluster customized for the demo & the current user\n* Setup SDP pipelines\n* Setup DBSQL dashboard\n* Create ML Model\n* Demo links are updated with resources created for an easy navigation\n\n## Feedback\n\nDemo not working? Can't use dbdemos? Please open a github issue.
\nMake sure you mention the name of the demo.\n\n# DBDemos Developer options\n\n## Adding an AI/BI demo to dbdemos\nopen [README_AIBI.md](README_AIBI.md) for more details on how to contribute & add an AI/BI demo.\n\nRead the following if you want to add a new demo bundle.\n\n## Packaging a demo with dbdemos\n\nYour demo must contain a `_resources` folder where you include all initialization scripts and your bundle configuration file.\n\n### Links & tags\nDBdemos will dynamically override the link to point to the resources created.\n\n**Always use links relative to the local path to support multi workspaces. Do not add the workspace id.**\n\n#### SDP pipelines:\nYour SDP pipeline must be added in the bundle file (see below).\nWithin your notebook, to identify your pipeline using the id in the bundle file, specify the id `dbdemos-pipeline-id=\"\"`as following:\n\n`Spark Declarative Pipeline`\n\n#### Workflows:\nYour workflows must be added in the bundle file (see below).\nWithin your notebook, to identify your workflow using the id in the bundle file, specify the id `dbdemos-workflow-id=\"\"`as following:\n\n`Access your workflow`\n\n\n#### DBSQL dashboards:\nSimilar to workflows, your dashboard id must match the one in the bundle file.\n\nDashboards definition should be added to the _dashboards folder (make sure the file name matches the dashboard id: `churn-prediction.lvdash.json`).\n\n` Churn Analysis Dashboard`\n\n\n\n### bundle_config\nThe demo must contain the a `./_resources/bundle_config` file containing your bundle definition.\nThis need to be a notebook & not a .json file (due to current api limitation).\n\n```json\n{\n \"name\": \"\",\n \"category\": \"\",\n \"title\": \".\",\n \"description\": \"<Description>\",\n \"bundle\": <Will bundle when True, skip when False>,\n \"tags\": [{\"sdp\": \"Spark Declarative Pipeline\"}],\n \"notebooks\": [\n {\n \"path\": \"<notebbok path from the demo folder (ex: resources/00-load-data)>\", \n \"pre_run\": <Will start a job to run it before packaging to get the cells results>, \n \"publish_on_website\": <Will add the notebook in the public website (with the results if it's pre_run=True)>, \n \"add_cluster_setup_cell\": <if True, add a cell with the name of the demo cluster>,\n \"title\": \"<Title>\", \n \"description\": \"<Description (will be in minisite also)>\",\n \"parameters\": {\"<key>\": \"<value. Will be sent to the pre_run job>\"}\n }\n ],\n \"init_job\": {\n \"settings\": {\n \"name\": \"demos_sdp_cdc_init_{{CURRENT_USER_NAME}}\",\n \"email_notifications\": {\n \"no_alert_for_skipped_runs\": False\n },\n \"timeout_seconds\": 0,\n \"max_concurrent_runs\": 1,\n \"tasks\": [\n {\n \"task_key\": \"init_data\",\n \"notebook_task\": {\n \"notebook_path\": \"{{DEMO_FOLDER}}/_resources/01-load-data-quality-dashboard\",\n \"source\": \"WORKSPACE\"\n },\n \"job_cluster_key\": \"Shared_job_cluster\",\n \"timeout_seconds\": 0,\n \"email_notifications\": {}\n }\n ]\n .... Full standard job definition\n }\n },\n \"pipelines\": <list of SDP pipelines if any>\n [\n {\n \"id\": \"sdp-cdc\", <id, used in the notebook links to go to the generated notebook: <a dbdemos-pipeline-id=\"sdp-cdc\" href=\"#joblist/pipelines/xxxx\">installed SDP pipeline</a> >\n \"run_after_creation\": True,\n \"definition\": {\n ... Any SDP pipeline configuration...\n \"libraries\": [\n {\n \"notebook\": {\n \"path\": \"{{DEMO_FOLDER}}/_resources/00-Data_CDC_Generator\"\n }\n }\n ],\n \"name\": \"demos_sdp_cdc_{{CURRENT_USER_NAME}}\",\n \"storage\": \"/demos/sdp/cdc/{{CURRENT_USER_NAME}}\",\n \"target\": \"demos_sdp_cdc_{{CURRENT_USER_NAME}}\"\n }\n }\n ],\n \"workflows\": [{\n \"start_on_install\": False,\n \"id\": \"credit-job\",\n \"definition\": {\n \"settings\": {\n ... full pipeline settings\n }\n }],\n \"dashboards\": [{\"name\": \"[dbdemos] Retail Churn Prediction Dashboard\", \"id\": \"churn-prediction\"}] \n}\n```\n\ndbdemos will replace the values defined as {{<KEY>}} based on who install the demo. Supported keys:\n* TODAY\n* CURRENT_USER (email)\n* CURRENT_USER_NAME (derivated from email)\n* DEMO_NAME\n* DEMO_FOLDER\n\n\n# DBDemo Installer configuration\n\nThe following describe how to package the demos created.\n\nThe installer needs to fetch data from a workspace & start jobs. To do so, it requires informations `local_conf.json`\n```json\n{\n \"pat_token\": \"xxx\",\n \"username\": \"xx.xx@databricks.com\",\n \"url\": \"https://xxx.databricks.com\",\n \"repo_staging_path\": \"/Repos/xx.xx@databricks.com\",\n \"repo_name\": \"dbdemos-notebooks\",\n \"repo_url\": \"https://github.com/databricks-demos/dbdemos-notebooks.git\", #put your clone here\n \"branch\": \"master\",\n \"current_folder\": \"<Used to mock the current folder outside of a notebook, ex: /Users/quentin.ambard@databricks.com/test_install_demo>\"\n}\n```\n\n### Creating the bundles:\n```python\nbundler = JobBundler(conf)\n# the bundler will use a stating repo dir in the workspace to analyze & run content.\nbundler.reset_staging_repo(skip_pull=False)\n# Discover bundles from repo:\nbundler.load_bundles_conf()\n# Or manually add bundle to run faster:\n#bundler.add_bundle(\"product_demos/Auto-Loader (cloudFiles)\")\n\n# Run the jobs (only if there is a new commit since the last time, or failure, or force execution)\nbundler.start_and_wait_bundle_jobs(force_execution = False)\n\npackager = Packager(conf, bundler)\npackager.package_all()\n```\n\n\n## Licence\nSee LICENSE file.\n\n## Data collection\nTo improve users experience and dbdemos asset quality, dbdemos sends report usage and capture views in the installed notebook (usually in the first cell) and dashboards. This information is captured for product improvement only and not for marketing purpose, and doesn't contain PII information. By using `dbdemos` and the assets it provides, you consent to this data collection. If you wish to disable it, you can set `Tracker.enable_tracker` to False in the `tracker.py` file.\n\n## Resource creation\nTo simplify your experience, `dbdemos` will create and start for you resources. As example, a demo could start (not exhaustive):\n- A cluster to run your demo\n- A Delta Live Table Pipeline to ingest data\n- A DBSQL endpoint to run DBSQL dashboard\n- An ML model\n\nWhile `dbdemos` does its best to limit the consumption and enforce resource auto-termination, you remain responsible for the resources created and the potential consumption associated.\n\n## Support\nDatabricks does not offer official support for `dbdemos` and the associated assets.\nFor any issue with `dbdemos` or the demos installed, please open an issue and the demo team will have a look on a best effort basis.\n\n" }, { "path": "README_AIBI.md", "content": "# Adding an AI-BI demo to dbdemos\n\n*Note: Adding new content from external contributors required special terms approval. Please open an issue if you'd like to contribute and are not part of the Databricks team.*\n\n*Note: if you're part of the Databricks team, please reach the demo team slack channel before starting the process for alignement and avoid duplicating work.*\n\n## Fork dbdemos-notebooks\nThe actual AI-BI demo content is in the [dbdemos-notebooks repository](https://github.com/databricks-demos/dbdemos-notebooks).\n\nStart by forking the repository and create a new branch there with your changes.\n\n## Create the demo\n\nStart by creating your dataset (must be all crafted/generated with DBRX to avoid any license issues), dashboard and genie space.\n\nOnce you're ready, add your dbdemos to the [aibi folder](https://github.com/databricks-demos/dbdemos-notebooks/tree/main/aibi).\nFor that, clone the the [aibi-marketing-campaign folder](https://github.com/databricks-demos/dbdemos-notebooks/tree/main/aibi/aibi-marketing-campaign) and replace the content with your own. \n\nMake sure the name of the folder has the similar pattern: `aibi-<use-case>`.\n\n## Data Transformation and Table Structure\n\n### Start with your story first\nThink about what would be a good Dashboard+Genie. Ideally you want to show some business outcome in the dashboard, and you see a spike somewhere. Then you open genie to ask a followup question.\n\n### Dataset\nOnce your story is ready, work backward to generate your dataset. Think about the gold table required, and then the raw dataset that you'll clean to create these tables.\n\n**Your dataset must be entirely crafted with tools like faker / DBRX. Double check any dataset license. Add a NOTICE file in your dataset folder explaining where the data is coming from / how it was created.**\n\nDatasets are stored in the [dbdemos-datasets repository](https://github.com/databricks-demos/dbdemos-datasets), and then mirrored in the dbdemos-dataset S3 bucket. Fork this repository and add your data in the `aibi` folder.\n\n### Defining the dbdemos genie room setup\n\nAll the configuration should go in the bundle file. See this example: [https://github.com/databricks-demos/dbdemos-notebooks/blob/main/aibi/aibi-marketing-campaign/_resources/bundle_config.py](https://github.com/databricks-demos/dbdemos-notebooks/blob/main/aibi/aibi-marketing-campaign/_resources/bundle_config.py)\n\nHere is what your bundle should look like:\n\n```json\n{\n \"name\": \"aibi-marketing-campaign\",\n \"category\": \"AI-BI\",\n \"title\": \"AI/BI: Marketing Campaign effectiveness\",\n \"custom_schema_supported\": True,\n \"default_catalog\": \"main\",\n \"default_schema\": \"dbdemos_aibi_cme_marketing_campaign\",\n \"description\": \"Analyze your Marketing Campaign effectiveness leveraging AI/BI Dashboard. Deep dive into your data and metrics, asking plain question through Genie Room.\",\n \"bundle\": True,\n \"notebooks\": [\n {\n \"path\": \"AI-BI-Marketing-campaign\", \n \"pre_run\": False, \n \"publish_on_website\": True, \n \"add_cluster_setup_cell\": False,\n \"title\": \"AI BI: Campaign effectiveness\", \n \"description\": \"Discover Databricks Intelligence Data Platform capabilities.\"\n }\n ],\n \"init_job\": {},\n \"cluster\": {}, \n \"pipelines\": [],\n \"dashboards\": [{\"name\": \"[dbdemos] AI/BI - Marketing Campaign\", \"id\": \"web-marketing\"}\n ],\n \"data_folders\":[\n {\"source_folder\":\"aibi/dbdemos_aibi_cme_marketing_campaign/raw_campaigns\", \"source_format\": \"parquet\", \"target_volume_folder\":\"raw_campaigns\", \"target_format\":\"parquet\"}],\n \"sql_queries\": [\n [\n \"CREATE OR REPLACE TABLE `{{CATALOG}}`.`{{SCHEMA}}`.raw_campaigns TBLPROPERTIES (delta.autooptimize.optimizewrite = TRUE, delta.autooptimize.autocompact = TRUE ) COMMENT 'This is the bronze table for campaigns created from parquet files' AS SELECT * FROM read_files('/Volumes/{{CATALOG}}/{{SCHEMA}}/dbdemos_raw_data/raw_campaigns', format => 'parquet', pathGlobFilter => '*.parquet')\"\n ],\n [\n \"... queries in here will be executed in parallel\", \" ... (don't forget to add comments) on the table, and PK/FK\"\n ],\n [\"CREATE OR REPLACE FUNCTION {{CATALOG}}.{{SCHEMA}}.my_ai_forecast(input_table STRING, target_column STRING, time_column STRING, periods INT) RETURN TABLE ...\"]\n ],\n \"genie_rooms\":[\n {\n \"id\": \"marketing-campaign\",\n \"display_name\": \"DBDemos - AI/BI - Marketing Campaign\", \n \"description\": \"Analyze your Marketing Campaign effectiveness leveraging AI/BI Dashboard. Deep dive into your data and metrics.\",\n \"table_identifiers\": [\"{{CATALOG}}.{{SCHEMA}}.campaigns\", \"...\"],\n \"sql_instructions\": [\n {\n \"title\": \"Compute rolling metrics\",\n \"content\": \"select date, unique_clicks, sum(unique_clicks) OVER (ORDER BY date RANGE BETWEEN 6 PRECEDING AND CURRENT ROW) AS clicks_t7d, sum(total_delivered) OVER (ORDER BY date RANGE BETWEEN 6 PRECEDING AND CURRENT ROW) AS delivered_t7d, sum(unique_clicks) OVER (ORDER BY date RANGE BETWEEN 27 PRECEDING AND CURRENT ROW) AS clicks_t28d, sum(total_delivered) OVER (ORDER BY date RANGE BETWEEN 27 PRECEDING AND CURRENT ROW) AS delivered_t28d, sum(unique_clicks) OVER (ORDER BY date RANGE BETWEEN 90 PRECEDING AND CURRENT ROW) AS clicks_t91d, sum(total_delivered) OVER (ORDER BY date RANGE BETWEEN 90 PRECEDING AND CURRENT ROW) AS delivered_t91d, unique_clicks / total_delivered as ctr, total_delivered / total_sent AS delivery_rate, total_optouts / total_delivered AS optout_rate, total_spam / total_delivered AS spam_rate, clicks_t7d / delivered_t7d as ctr_t7d, clicks_t28d / delivered_t28d as ctr_t28d, clicks_t91d / delivered_t91d as ctr_t91d from {{CATALOG}}.{{SCHEMA}}.metrics_daily_rolling\"\n }\n ],\n \"instructions\": \"If a customer ask a forecast, leverage the sql fonction ai_forecast\",\n \"function_names\": [\n \"{{CATALOG}}.{{SCHEMA}}.my_ai_forecast\"\n ], \n \"curated_questions\": [\n \"How has the total number of emails sent, delivered, and the unique clicks evolved over the last six months?\", \"...\"\n ]\n }\n ]\n}\n```\n\n### Data Loading and Transformation\nAIBI demos should start with raw data files in a volume and implement a few transformation steps to showcase data lineage. This helps demonstrate the end-to-end data workflow and provides a more comprehensive view of Databricks' capabilities.\n\n**Important:** Avoid using Materialized Views (MVs) for transformations as they can slow down the dbdemos installation process. Instead, use standard SQL transformations in your demo for now (we'll revisit soon).\n\nExample transformation flow:\n1. Start with raw data in volume, typically 3+ sources\n2. Create bronze table(s) directly from the volume files (~3+ tables)\n3. [optional] Create silver table(s) with basic transformations (cleaning, type conversion, etc.)\n4. Create gold table(s) with business-specific transformations and potentially a few joins (we want to keep at least 2 or 3 tables in the genie room)\n\n### Gold Table Requirements\n- Gold tables (used in the Genie room) should have PK and FK defined\n- Gold tables should include comprehensive comments on all fields. This improves the Genie experience by providing context for each column and helps users understand the data model.\n\nExample gold table creation with comments directly in the CREATE statement:\n```sql\nCREATE OR REPLACE TABLE {{CATALOG}}.{{SCHEMA}}.customer_gold (\n id STRING COMMENT 'Unique customer identifier' PRIMARY KEY,\n first_name STRING COMMENT 'Customer first name',\n last_name STRING COMMENT 'Customer last name',\n email STRING COMMENT 'Customer email address',\n signup_date DATE COMMENT 'Date when customer created their account',\n last_activity_date DATE COMMENT 'Most recent date of customer activity',\n customer_segment STRING COMMENT 'Customer segmentation category (New, Loyal, At-Risk, Churned)',\n lifetime_value DOUBLE COMMENT 'Calculated total customer spend in USD'\n)\nAS\nSELECT\n id, first_name, last_name, email, signup_date, last_activity_date, customer_segment, lifetime_value FROM {{CATALOG}}.{{SCHEMA}}.customer_silver;\n```\n\nThis approach is more concise and ensures all column comments are created in a single SQL statement.\n\n## SQL AI Functions\n\nWe need help implementing SQL AI Functions in the installer_genie.py file and the JSON configuration. These functions enhance the AI capabilities of the Genie room and enable more sophisticated queries.\n\nThe AI functions should be added as part of the SQL statement. Don't forget to add comments on them (at the function level and function param level).\n\nOnce created, you can add them to the genie room under `\"function_names\": [\"{{CATALOG}}.{{SCHEMA}}.ai_forecast\", \"...\"]`\n\n**Note: this isn't yet implemented. If you're interested in contributing to DBDemos, reach out to the demo team. The implementation should go in the `InstallerGenie` class to create these functions during demo installation and make them available in the Genie room.**\n\n## Update the Main notebook\nUpdate the notebook cloned from the folder above, with your use-case.\n\n### Present the use-case\nRename & Update the main notebook, detailing your use-case, what is the data and the insights you want to show.\n\n### Update tracking\nUpdate the demo name in the first cell in the tracker pixel, and the notebook name.\n\n### Update the dashboard link\n\nPut your dashboard in the dashboards folder. In the dashboard json, make sure you use the same catalog and schema as the one you have in the bundle configuration file, typically `main.dbdemos_aibi_xxxxxx`\n\nYou can then reference the dashboard like this:\n\n```html\n<a dbdemos-dashboard-id=\"web-marketing\" href='/sql/dashboardsv3/02ef00cc36721f9e1f2028ee75723cc1' target=\"_blank\">your dashboard</a>\n```\n\nthe ID here `web-marketing` must match the ID in the bundle configuration (and the dashboard file name):\n\n```\n\"dashboards\": [{\"name\": \"[dbdemos] AI/BI - Marketing Campaign\", \"id\": \"web-marketing\"}]\n```\n\n### Update the Genie Room link\n\n```html\n<a dbdemos-genie-id=\"marketing-campaign\" href='/genie/rooms/01ef775474091f7ba11a8a9d2075eb58' target=\"_blank\">your genie space</a>\n```\nthe ID here `marketing-campaign` must match the ID in the bundle configuration:\n\n```\n \"genie_rooms\":[\n {\n \"id\": \"marketing-campaign\",\n \"display_name\": \"DBDemos - AI/BI - Marketing Campaign\", \n ...\n }\n ]\n```\n\n## Update the bundle configuration\n\n- Make sure the dashboard ID and genie room ID match your links as above. The dashboard ID must match the dashboard file name in the dashboards folder (see below).\n- Keep the `default_catalog` to main, `default_schema` should follow the naming convention `dbdemos_aibi_<industry>_<use-case>`.\n- Make sure you add a sql instruction in the genie room, with curated questions, descriptions and an instruction.\n- Dataset folders must be the path in the databricks-datasets repository (see below).\n\n## Add your dashboard under the dashboards folder\n\n- create json file in the dashboards folder. Make sure you format it correctly as it's easy to read/diff\n- your catalog.schema in the queries must match the `default_catalog` and `default_schema` in the bundle configuration.\n- your dashboard name must match the id in the bundle configuration.\n- don't forget to update the dashboard tracking (add the tracker in the MD at the end of the dashboard, match the demo name)\n\n\n## Add your images\n\nImages are stored in the [dbdemos-resources repository](https://github.com/databricks-demos/dbdemos-resources). \n\nTo add an image, fork the repo and send a PR.\n\nYou need at least 2 images:\n- the miniature for the demo list: `https://github.com/databricks-demos/dbdemos-resources/raw/main/icon/<demo_name>.jpg`\n- the screenshot of the dashboard: `https://www.dbdemos.ai/assets/img/dbdemos/<demo_name>-dashboard-0.png` (1 per dashboard you add)\n\nReach out the demo team for a demo miniature\n\n\nhttps://www.dbdemos.ai/assets/img/dbdemos/aibi-marketing-campaign-dashboard-0.png\n\nhttps://github.com/databricks-demos/dbdemos-resources/raw/main/icon/aibi-marketing-campaign.jpg\n\n\n# Packaging & testing your demo\n\nOpen the `test_demo.py` file. Update the conf to match your databricks-notebooks repo fork/branch in the config json file.\n\ndbdemos needs a workspace and a repo to package the demo, make sure you configure it in the conf json file (use your fork).\n\nMake sure you update the bundle folder to match your demo:\n\n```\n bundle(conf, \"aibi/aibi-marketing-campaign\")\n```\n\n" }, { "path": "SECURITY.md", "content": "# Security Policy\n\n## Reporting a Vulnerability\n\nPlease email bugbounty@databricks.com to report any security vulnerabilities. We will acknowledge receipt of your vulnerability and strive to send you regular updates about our progress. If you're curious about the status of your disclosure please feel free to email us again. If you want to encrypt your disclosure email, you can use [this PGP key](https://keybase.io/arikfr/key.asc).\n\n" }, { "path": "ai_release/__init__.py", "content": "\"\"\"\nAI Release Tools for DBDemos\n\nThis module provides tools for Claude Code to:\n1. Execute code remotely on Databricks clusters for testing\n2. Bundle and test demos\n3. Fix issues in dbdemos-notebooks\n\nCRITICAL: Never release to PyPI - only the human can do that!\n\"\"\"\n\n__version__ = \"0.1.0\"\n" }, { "path": "ai_release/bundle.py", "content": "#!/usr/bin/env python3\n\"\"\"\nDBDemos Bundle CLI - For bundling and testing demos\n\nThis script is designed to be run by Claude Code for the release workflow.\nIt supports bundling specific demos, running from feature branches, and job repair.\n\nUsage:\n # Bundle a specific demo from main branch\n python ai_release/bundle.py --demo lakehouse-retail-c360\n\n # Bundle from a feature branch\n python ai_release/bundle.py --demo lakehouse-retail-c360 --branch fix/retail-bug\n\n # Bundle all demos (uses GitHub diff to only run changed ones)\n python ai_release/bundle.py --all\n\n # Repair a failed job (re-run only failed tasks)\n python ai_release/bundle.py --demo lakehouse-retail-c360 --repair\n\n # Force full re-run (ignore commit diff optimization)\n python ai_release/bundle.py --demo lakehouse-retail-c360 --force\n\n # Get job status and error details\n python ai_release/bundle.py --demo lakehouse-retail-c360 --status\n\n # List all available demos\n python ai_release/bundle.py --list-demos\n\nEnvironment Variables:\n DATABRICKS_HOST: Workspace URL (default: https://e2-demo-tools.cloud.databricks.com/)\n DATABRICKS_TOKEN: PAT token for Databricks\n GITHUB_TOKEN: GitHub token for API access\n DBDEMOS_NOTEBOOKS_PATH: Path to dbdemos-notebooks repo (default: ../dbdemos-notebooks)\n\nConfig File:\n Can also use local_conf.json in the repo root for configuration.\n\"\"\"\n\nimport argparse\nimport json\nimport os\nimport sys\nfrom pathlib import Path\n\n# Add parent directory to path for imports\nsys.path.insert(0, str(Path(__file__).parent.parent))\n\nfrom dbdemos.conf import Conf, DemoConf\nfrom dbdemos.job_bundler import JobBundler\nfrom dbdemos.packager import Packager\n\n\ndef load_config(args):\n \"\"\"Load configuration from environment variables and/or local_conf.json\"\"\"\n config = {}\n\n # Try to load from local_conf.json files (E2TOOL is primary for bundling)\n repo_root = Path(__file__).parent.parent\n conf_files = [\n repo_root / \"local_conf_E2TOOL.json\", # Primary for bundling/testing\n repo_root / \"local_conf.json\",\n ]\n\n for conf_file in conf_files:\n if conf_file.exists():\n with open(conf_file, \"r\") as f:\n config = json.load(f)\n print(f\"Loaded config from {conf_file}\")\n break\n\n # dbdemos-notebooks path\n default_notebooks_path = str(repo_root.parent / \"dbdemos-notebooks\")\n notebooks_path = os.environ.get(\"DBDEMOS_NOTEBOOKS_PATH\", config.get(\"dbdemos_notebooks_path\", default_notebooks_path))\n config[\"dbdemos_notebooks_path\"] = notebooks_path\n\n # Branch override from CLI\n if args.branch:\n config[\"branch\"] = args.branch\n elif \"branch\" not in config:\n config[\"branch\"] = \"main\"\n\n # Validate required fields\n required = [\"pat_token\", \"github_token\", \"url\"]\n missing = [f for f in required if not config.get(f)]\n if missing:\n print(f\"ERROR: Missing required config: {missing}\")\n print(\"Set via environment variables or local_conf.json\")\n sys.exit(1)\n\n return config\n\n\ndef load_cluster_templates():\n \"\"\"Load default cluster configuration templates\"\"\"\n repo_root = Path(__file__).parent.parent\n\n with open(repo_root / \"dbdemos/resources/default_cluster_config.json\", \"r\") as f:\n default_cluster_template = f.read()\n\n with open(repo_root / \"dbdemos/resources/default_test_job_conf.json\", \"r\") as f:\n default_cluster_job_template = f.read()\n\n return default_cluster_template, default_cluster_job_template\n\n\ndef create_conf(config):\n \"\"\"Create Conf object from config dict\"\"\"\n default_cluster_template, default_cluster_job_template = load_cluster_templates()\n\n # Strip .git from repo_url if present (Conf doesn't allow it)\n repo_url = config.get(\"repo_url\", \"https://github.com/databricks-demos/dbdemos-notebooks\")\n if repo_url.endswith(\".git\"):\n repo_url = repo_url[:-4]\n\n return Conf(\n username=config.get(\"username\", \"claude-code@databricks.com\"),\n workspace_url=config[\"url\"],\n org_id=config.get(\"org_id\", \"\"),\n pat_token=config[\"pat_token\"],\n default_cluster_template=default_cluster_template,\n default_cluster_job_template=default_cluster_job_template,\n repo_staging_path=config.get(\"repo_staging_path\", \"/Repos/quentin.ambard@databricks.com\"),\n repo_name=config.get(\"repo_name\", \"dbdemos-notebooks\"),\n repo_url=repo_url,\n branch=config[\"branch\"],\n github_token=config[\"github_token\"],\n run_test_as_username=config.get(\"run_test_as_username\", \"quentin.ambard@databricks.com\")\n )\n\n\ndef list_demos(bundler: JobBundler):\n \"\"\"List all available demos\"\"\"\n print(\"Scanning for available demos...\")\n bundler.reset_staging_repo(skip_pull=False)\n bundler.load_bundles_conf()\n\n print(f\"\\nFound {len(bundler.bundles)} demos:\\n\")\n for path, demo_conf in sorted(bundler.bundles.items()):\n print(f\" - {demo_conf.name:<40} ({path})\")\n\n return bundler.bundles\n\n\ndef get_job_status(bundler: JobBundler, demo_name: str):\n \"\"\"Get detailed job status for a demo\"\"\"\n # Find the demo\n bundler.reset_staging_repo(skip_pull=True)\n\n # Try to find the job\n job_name = f\"field-bundle_{demo_name}\"\n job = bundler.db.find_job(job_name)\n\n if not job:\n print(f\"No job found for demo: {demo_name}\")\n return None\n\n job_id = job[\"job_id\"]\n print(f\"\\n{'='*80}\")\n print(f\"Job: {job_name}\")\n print(f\"Job ID: {job_id}\")\n print(f\"URL: {bundler.conf.workspace_url}/#job/{job_id}\")\n print(f\"{'='*80}\\n\")\n\n # Get recent runs\n runs = bundler.db.get(\"2.1/jobs/runs/list\", {\"job_id\": job_id, \"limit\": 5, \"expand_tasks\": \"true\"})\n\n if \"runs\" not in runs or len(runs[\"runs\"]) == 0:\n print(\"No runs found for this job.\")\n return None\n\n for i, run in enumerate(runs[\"runs\"]):\n run_id = run[\"run_id\"]\n state = run[\"state\"]\n status = run.get(\"status\", {})\n\n print(f\"\\n--- Run {i+1}: {run_id} ---\")\n print(f\"State: {state.get('life_cycle_state', 'N/A')} / {state.get('result_state', 'N/A')}\")\n print(f\"URL: {bundler.conf.workspace_url}/#job/{job_id}/run/{run_id}\")\n\n if \"termination_details\" in status:\n print(f\"Termination: {status['termination_details']}\")\n\n # Show task details for the most recent run\n if i == 0 and \"tasks\" in run:\n print(f\"\\nTasks ({len(run['tasks'])} total):\")\n for task in run[\"tasks\"]:\n task_key = task[\"task_key\"]\n task_state = task.get(\"state\", {})\n task_result = task_state.get(\"result_state\", \"PENDING\")\n\n # Get error info if failed\n error_info = \"\"\n if task_result == \"FAILED\":\n # Try to get run output for error details\n task_run_id = task.get(\"run_id\")\n if task_run_id:\n task_output = bundler.db.get(\"2.1/jobs/runs/get-output\", {\"run_id\": task_run_id})\n if \"error\" in task_output:\n error_info = f\"\\n Error: {task_output['error'][:200]}...\"\n if \"error_trace\" in task_output:\n error_info += f\"\\n Trace: {task_output['error_trace'][:500]}...\"\n\n status_icon = \"✓\" if task_result == \"SUCCESS\" else \"✗\" if task_result == \"FAILED\" else \"○\"\n print(f\" {status_icon} {task_key}: {task_result}{error_info}\")\n\n return runs[\"runs\"][0] if runs[\"runs\"] else None\n\n\ndef wait_for_run(bundler: JobBundler, job_id: int, run_id: int):\n \"\"\"Wait for a job run to complete\"\"\"\n import time\n print(f\"Waiting for job completion...\")\n print(f\"URL: {bundler.conf.workspace_url}/#job/{job_id}/run/{run_id}\")\n\n i = 0\n while True:\n run = bundler.db.get(\"2.1/jobs/runs/get\", {\"run_id\": run_id})\n state = run.get(\"state\", {})\n life_cycle = state.get(\"life_cycle_state\", \"UNKNOWN\")\n\n if life_cycle not in [\"RUNNING\", \"PENDING\"]:\n result = state.get(\"result_state\", \"UNKNOWN\")\n print(f\"\\nJob finished: {life_cycle} / {result}\")\n return result == \"SUCCESS\"\n\n if i % 60 == 0: # Print every 5 minutes\n print(f\" Still running... ({i * 5}s elapsed)\")\n i += 1\n time.sleep(5)\n\n\ndef repair_job(bundler: JobBundler, demo_name: str, wait: bool = False):\n \"\"\"Repair a failed job (re-run only failed tasks)\"\"\"\n job_name = f\"field-bundle_{demo_name}\"\n job = bundler.db.find_job(job_name)\n\n if not job:\n print(f\"No job found for demo: {demo_name}\")\n return False\n\n job_id = job[\"job_id\"]\n\n # Get the most recent run\n runs = bundler.db.get(\"2.1/jobs/runs/list\", {\"job_id\": job_id, \"limit\": 1, \"expand_tasks\": \"true\"})\n\n if \"runs\" not in runs or len(runs[\"runs\"]) == 0:\n print(\"No runs found to repair.\")\n return False\n\n latest_run = runs[\"runs\"][0]\n run_id = latest_run[\"run_id\"]\n\n # Check if run is in a repairable state\n state = latest_run[\"state\"]\n if state.get(\"life_cycle_state\") != \"TERMINATED\":\n print(f\"Run is not terminated (state: {state.get('life_cycle_state')}). Cannot repair.\")\n return False\n\n if state.get(\"result_state\") == \"SUCCESS\":\n print(\"Run already succeeded. No repair needed.\")\n return True\n\n # Find failed tasks\n failed_tasks = []\n for task in latest_run.get(\"tasks\", []):\n task_state = task.get(\"state\", {})\n if task_state.get(\"result_state\") in [\"FAILED\", \"CANCELED\", \"TIMEDOUT\"]:\n failed_tasks.append(task[\"task_key\"])\n\n if not failed_tasks:\n print(\"No failed tasks found to repair.\")\n return True\n\n print(f\"Repairing run {run_id} - re-running tasks: {failed_tasks}\")\n\n # Call repair API\n repair_response = bundler.db.post(\"2.1/jobs/runs/repair\", {\n \"run_id\": run_id,\n \"rerun_tasks\": failed_tasks\n })\n\n if \"repair_id\" in repair_response:\n print(f\"Repair started. Repair ID: {repair_response['repair_id']}\")\n print(f\"URL: {bundler.conf.workspace_url}/#job/{job_id}/run/{run_id}\")\n\n if wait:\n return wait_for_run(bundler, job_id, run_id)\n return True\n else:\n print(f\"Failed to repair: {repair_response}\")\n return False\n\n\ndef cleanup_demo_schema(bundler: JobBundler, demo_conf):\n \"\"\"Drop the demo schema to ensure clean state before running.\n\n Uses main__build as the catalog (bundling catalog) and the demo's default_schema.\n Uses Databricks SDK: w.schemas.delete(full_name=schema_full_name, force=True)\n \"\"\"\n from databricks.sdk import WorkspaceClient\n from databricks.sdk.errors import NotFound\n\n # Bundling uses main__build catalog\n catalog = \"main__build\"\n schema = demo_conf.default_schema\n\n if not schema:\n print(f\" No default_schema defined for {demo_conf.name}, skipping cleanup\")\n return\n\n full_schema = f\"{catalog}.{schema}\"\n print(f\" Cleaning up schema: {full_schema}\")\n\n try:\n w = WorkspaceClient(\n host=bundler.conf.workspace_url,\n token=bundler.conf.pat_token\n )\n # force=True is equivalent to CASCADE\n w.schemas.delete(full_name=full_schema, force=True)\n print(f\" ✓ Schema {full_schema} dropped successfully\")\n except NotFound:\n print(f\" ✓ Schema {full_schema} does not exist (nothing to clean)\")\n except Exception as e:\n print(f\" WARNING: Error during schema cleanup: {e}\")\n\n\ndef bundle_demo(bundler: JobBundler, demo_path: str, force: bool = False, skip_packaging: bool = False, cleanup_schema: bool = True):\n \"\"\"Bundle a specific demo\"\"\"\n print(f\"\\nBundling demo: {demo_path}\")\n print(f\"Branch: {bundler.conf.branch}\")\n\n bundler.reset_staging_repo(skip_pull=False)\n bundler.add_bundle(demo_path)\n\n if len(bundler.bundles) == 0:\n print(f\"ERROR: Demo not found or not configured for bundling: {demo_path}\")\n return False\n\n # Clean up schema before running if requested\n if cleanup_schema:\n print(\"\\nCleaning up demo schemas...\")\n for path, demo_conf in bundler.bundles.items():\n cleanup_demo_schema(bundler, demo_conf)\n\n # Run the job\n bundler.start_and_wait_bundle_jobs(\n force_execution=force,\n skip_execution=False,\n recreate_jobs=False\n )\n\n # Check results\n for path, demo_conf in bundler.bundles.items():\n if demo_conf.run_id:\n run = bundler.db.get(\"2.1/jobs/runs/get\", {\"run_id\": demo_conf.run_id})\n result_state = run.get(\"state\", {}).get(\"result_state\", \"UNKNOWN\")\n\n if result_state == \"SUCCESS\":\n print(f\"\\n✓ Job succeeded for {demo_conf.name}\")\n\n if not skip_packaging:\n print(\"Packaging demo...\")\n packager = Packager(bundler.conf, bundler)\n packager.package_all()\n print(f\"✓ Demo packaged successfully\")\n\n return True\n else:\n print(f\"\\n✗ Job failed for {demo_conf.name}: {result_state}\")\n print(f\"Check: {bundler.conf.workspace_url}/#job/{demo_conf.job_id}/run/{demo_conf.run_id}\")\n return False\n\n return False\n\n\ndef bundle_all(bundler: JobBundler, force: bool = False, cleanup_schema: bool = True):\n \"\"\"Bundle all demos (uses diff optimization)\"\"\"\n print(\"\\nBundling all demos...\")\n print(f\"Branch: {bundler.conf.branch}\")\n\n bundler.reset_staging_repo(skip_pull=False)\n bundler.load_bundles_conf()\n\n print(f\"Found {len(bundler.bundles)} demos\")\n\n # Clean up schemas before running if requested\n if cleanup_schema:\n print(\"\\nCleaning up demo schemas...\")\n for path, demo_conf in bundler.bundles.items():\n cleanup_demo_schema(bundler, demo_conf)\n\n # Run jobs (will skip unchanged demos unless force=True)\n bundler.start_and_wait_bundle_jobs(\n force_execution=force,\n skip_execution=False,\n recreate_jobs=False\n )\n\n # Check results\n success_count = 0\n fail_count = 0\n skip_count = 0\n\n for path, demo_conf in bundler.bundles.items():\n if demo_conf.run_id:\n run = bundler.db.get(\"2.1/jobs/runs/get\", {\"run_id\": demo_conf.run_id})\n result_state = run.get(\"state\", {}).get(\"result_state\", \"UNKNOWN\")\n\n if result_state == \"SUCCESS\":\n success_count += 1\n else:\n fail_count += 1\n print(f\"✗ {demo_conf.name} failed: {result_state}\")\n else:\n skip_count += 1\n\n print(f\"\\nResults: {success_count} succeeded, {fail_count} failed, {skip_count} skipped\")\n\n if fail_count == 0:\n print(\"\\nPackaging all demos...\")\n packager = Packager(bundler.conf, bundler)\n packager.package_all()\n print(\"✓ All demos packaged successfully\")\n return True\n else:\n print(\"\\n✗ Some jobs failed. Fix errors before packaging.\")\n return False\n\n\ndef find_demo_path(bundler: JobBundler, demo_name: str) -> str:\n \"\"\"Find the full path for a demo by name\"\"\"\n bundler.reset_staging_repo(skip_pull=True)\n bundler.load_bundles_conf()\n\n # Check if it's already a path\n if demo_name in bundler.bundles:\n return demo_name\n\n # Search by demo name\n for path, demo_conf in bundler.bundles.items():\n if demo_conf.name == demo_name:\n return path\n\n # Partial match\n matches = []\n for path, demo_conf in bundler.bundles.items():\n if demo_name in demo_conf.name or demo_name in path:\n matches.append((path, demo_conf.name))\n\n if len(matches) == 1:\n return matches[0][0]\n elif len(matches) > 1:\n print(f\"Multiple matches for '{demo_name}':\")\n for path, name in matches:\n print(f\" - {name} ({path})\")\n print(\"\\nPlease be more specific.\")\n return None\n\n print(f\"Demo not found: {demo_name}\")\n return None\n\n\ndef main():\n parser = argparse.ArgumentParser(\n description=\"DBDemos Bundle CLI - Bundle and test demos\",\n formatter_class=argparse.RawDescriptionHelpFormatter,\n epilog=__doc__\n )\n\n # Actions\n parser.add_argument(\"--demo\", \"-d\", help=\"Demo name or path to bundle\")\n parser.add_argument(\"--all\", \"-a\", action=\"store_true\", help=\"Bundle all demos\")\n parser.add_argument(\"--list-demos\", \"-l\", action=\"store_true\", help=\"List all available demos\")\n parser.add_argument(\"--status\", \"-s\", action=\"store_true\", help=\"Get job status for a demo\")\n\n # Options\n parser.add_argument(\"--branch\", \"-b\", help=\"Git branch to use (overrides config)\")\n parser.add_argument(\"--force\", \"-f\", action=\"store_true\", help=\"Force re-run (ignore diff optimization)\")\n parser.add_argument(\"--repair\", \"-r\", action=\"store_true\", help=\"Repair failed job (re-run failed tasks only)\")\n parser.add_argument(\"--skip-packaging\", action=\"store_true\", help=\"Skip packaging step (useful for debugging)\")\n parser.add_argument(\"--check-config\", action=\"store_true\", help=\"Verify configuration without running anything\")\n parser.add_argument(\"--wait\", \"-w\", action=\"store_true\", help=\"Wait for job/repair completion\")\n parser.add_argument(\"--no-cleanup-schema\", action=\"store_true\", help=\"Skip schema cleanup (default: cleanup enabled)\")\n parser.add_argument(\"--cleanup-schema\", action=\"store_true\", default=True, help=\"Clean up demo schema before running (default: True)\")\n\n args = parser.parse_args()\n\n # Load config\n config = load_config(args)\n conf = create_conf(config)\n bundler = JobBundler(conf)\n\n print(f\"Workspace: {conf.workspace_url}\")\n print(f\"Branch: {conf.branch}\")\n\n # Check config only\n if args.check_config:\n print(f\"\\n✓ Configuration valid\")\n print(f\" - Username: {conf.username}\")\n print(f\" - Repo: {conf.repo_url}\")\n print(f\" - Repo path: {conf.get_repo_path()}\")\n print(f\" - Notebooks path: {config.get('dbdemos_notebooks_path', 'N/A')}\")\n return 0\n\n # Execute action\n if args.list_demos:\n list_demos(bundler)\n return 0\n\n if args.status:\n if not args.demo:\n print(\"ERROR: --status requires --demo\")\n return 1\n get_job_status(bundler, args.demo)\n return 0\n\n if args.repair:\n if not args.demo:\n print(\"ERROR: --repair requires --demo\")\n return 1\n success = repair_job(bundler, args.demo, wait=args.wait)\n return 0 if success else 1\n\n # Determine cleanup_schema setting (--no-cleanup-schema disables it)\n cleanup_schema = not args.no_cleanup_schema\n\n if args.demo:\n demo_path = find_demo_path(bundler, args.demo)\n if not demo_path:\n return 1\n\n # Reset bundler after find_demo_path used it\n bundler = JobBundler(conf)\n success = bundle_demo(bundler, demo_path, force=args.force, skip_packaging=args.skip_packaging, cleanup_schema=cleanup_schema)\n return 0 if success else 1\n\n if args.all:\n success = bundle_all(bundler, force=args.force, cleanup_schema=cleanup_schema)\n return 0 if success else 1\n\n parser.print_help()\n return 1\n\n\nif __name__ == \"__main__\":\n sys.exit(main())\n" }, { "path": "ai_release/compute.py", "content": "\"\"\"\nRemote Code Execution on Databricks Clusters\n\nThis module provides functions to execute code on Databricks clusters for testing\nnotebook fixes before committing them to dbdemos-notebooks.\n\nBased on databricks-tools-core from ai-dev-kit.\n\"\"\"\n\nimport datetime\nimport json\nimport time\nfrom pathlib import Path\nfrom typing import Optional, List, Dict, Any\n\nfrom databricks.sdk import WorkspaceClient\nfrom databricks.sdk.service.compute import (\n CommandStatus,\n ClusterSource,\n Language,\n ListClustersFilterBy,\n State,\n)\n\n\nclass ExecutionResult:\n \"\"\"Result from code execution on a Databricks cluster.\"\"\"\n\n def __init__(\n self,\n success: bool,\n output: Optional[str] = None,\n error: Optional[str] = None,\n cluster_id: Optional[str] = None,\n cluster_name: Optional[str] = None,\n context_id: Optional[str] = None,\n context_destroyed: bool = True,\n ):\n self.success = success\n self.output = output\n self.error = error\n self.cluster_id = cluster_id\n self.cluster_name = cluster_name\n self.context_id = context_id\n self.context_destroyed = context_destroyed\n\n if success and context_id and not context_destroyed:\n self.message = (\n f\"Execution successful. Reuse context_id='{context_id}' with \"\n f\"cluster_id='{cluster_id}' for follow-up commands.\"\n )\n elif success:\n self.message = \"Execution successful.\"\n else:\n self.message = f\"Execution failed: {error}\"\n\n def __repr__(self):\n if self.success:\n return f\"ExecutionResult(success=True, output={repr(self.output[:100] if self.output else None)}...)\"\n return f\"ExecutionResult(success=False, error={repr(self.error)})\"\n\n def to_dict(self) -> Dict[str, Any]:\n return {\n \"success\": self.success,\n \"output\": self.output,\n \"error\": self.error,\n \"cluster_id\": self.cluster_id,\n \"cluster_name\": self.cluster_name,\n \"context_id\": self.context_id,\n \"context_destroyed\": self.context_destroyed,\n \"message\": self.message,\n }\n\n\n_LANGUAGE_MAP = {\n \"python\": Language.PYTHON,\n \"scala\": Language.SCALA,\n \"sql\": Language.SQL,\n \"r\": Language.R,\n}\n\n\ndef get_workspace_client(host: str, token: str) -> WorkspaceClient:\n \"\"\"Create a WorkspaceClient with explicit credentials.\"\"\"\n return WorkspaceClient(\n host=host,\n token=token,\n auth_type=\"pat\",\n product=\"dbdemos-ai-release\",\n product_version=\"0.1.0\",\n )\n\n\ndef list_clusters(client: WorkspaceClient, include_terminated: bool = False) -> List[Dict[str, Any]]:\n \"\"\"List user-created clusters in the workspace.\"\"\"\n clusters = []\n\n # Only list user-created clusters\n user_sources = [ClusterSource.UI, ClusterSource.API]\n\n # Running clusters\n running_filter = ListClustersFilterBy(\n cluster_sources=user_sources,\n cluster_states=[State.RUNNING, State.PENDING, State.RESIZING, State.RESTARTING],\n )\n for cluster in client.clusters.list(filter_by=running_filter):\n clusters.append({\n \"cluster_id\": cluster.cluster_id,\n \"cluster_name\": cluster.cluster_name,\n \"state\": cluster.state.value if cluster.state else None,\n \"creator_user_name\": cluster.creator_user_name,\n })\n\n if include_terminated:\n terminated_filter = ListClustersFilterBy(\n cluster_sources=user_sources,\n cluster_states=[State.TERMINATED, State.TERMINATING, State.ERROR],\n )\n for cluster in client.clusters.list(filter_by=terminated_filter):\n clusters.append({\n \"cluster_id\": cluster.cluster_id,\n \"cluster_name\": cluster.cluster_name,\n \"state\": cluster.state.value if cluster.state else None,\n \"creator_user_name\": cluster.creator_user_name,\n })\n\n return clusters\n\n\ndef find_cluster_by_name(client: WorkspaceClient, name_pattern: str) -> Optional[Dict[str, Any]]:\n \"\"\"\n Find a cluster by name pattern (case-insensitive).\n\n Args:\n client: WorkspaceClient\n name_pattern: Pattern to match (e.g., \"quentin\")\n\n Returns:\n Cluster info dict or None\n \"\"\"\n clusters = list_clusters(client, include_terminated=True)\n pattern_lower = name_pattern.lower()\n\n # First try running clusters\n for cluster in clusters:\n if cluster[\"state\"] == \"RUNNING\" and pattern_lower in cluster[\"cluster_name\"].lower():\n return cluster\n\n # Then try any cluster\n for cluster in clusters:\n if pattern_lower in cluster[\"cluster_name\"].lower():\n return cluster\n\n return None\n\n\ndef start_cluster(client: WorkspaceClient, cluster_id: str) -> Dict[str, Any]:\n \"\"\"Start a terminated cluster.\"\"\"\n cluster = client.clusters.get(cluster_id)\n cluster_name = cluster.cluster_name or cluster_id\n current_state = cluster.state.value if cluster.state else \"UNKNOWN\"\n\n if current_state == \"RUNNING\":\n return {\n \"cluster_id\": cluster_id,\n \"cluster_name\": cluster_name,\n \"state\": \"RUNNING\",\n \"message\": f\"Cluster '{cluster_name}' is already running.\",\n }\n\n if current_state not in (\"TERMINATED\", \"ERROR\"):\n return {\n \"cluster_id\": cluster_id,\n \"cluster_name\": cluster_name,\n \"state\": current_state,\n \"message\": f\"Cluster '{cluster_name}' is in state {current_state}.\",\n }\n\n client.clusters.start(cluster_id)\n\n return {\n \"cluster_id\": cluster_id,\n \"cluster_name\": cluster_name,\n \"previous_state\": current_state,\n \"state\": \"PENDING\",\n \"message\": f\"Cluster '{cluster_name}' is starting (3-8 minutes).\",\n }\n\n\ndef get_cluster_status(client: WorkspaceClient, cluster_id: str) -> Dict[str, Any]:\n \"\"\"Get cluster status.\"\"\"\n cluster = client.clusters.get(cluster_id)\n return {\n \"cluster_id\": cluster_id,\n \"cluster_name\": cluster.cluster_name or cluster_id,\n \"state\": cluster.state.value if cluster.state else \"UNKNOWN\",\n }\n\n\ndef wait_for_cluster(client: WorkspaceClient, cluster_id: str, timeout: int = 600) -> bool:\n \"\"\"Wait for cluster to reach RUNNING state.\"\"\"\n start_time = time.time()\n while time.time() - start_time < timeout:\n status = get_cluster_status(client, cluster_id)\n state = status[\"state\"]\n\n if state == \"RUNNING\":\n print(f\"✓ Cluster '{status['cluster_name']}' is running\")\n return True\n elif state in (\"TERMINATED\", \"ERROR\"):\n print(f\"✗ Cluster '{status['cluster_name']}' is {state}\")\n return False\n\n print(f\" Cluster state: {state}... waiting\")\n time.sleep(30)\n\n print(f\"✗ Timeout waiting for cluster\")\n return False\n\n\ndef create_context(client: WorkspaceClient, cluster_id: str, language: str = \"python\") -> str:\n \"\"\"Create an execution context on a cluster.\"\"\"\n lang_enum = _LANGUAGE_MAP.get(language.lower(), Language.PYTHON)\n result = client.command_execution.create(\n cluster_id=cluster_id, language=lang_enum\n ).result()\n return result.id\n\n\ndef destroy_context(client: WorkspaceClient, cluster_id: str, context_id: str) -> None:\n \"\"\"Destroy an execution context.\"\"\"\n client.command_execution.destroy(cluster_id=cluster_id, context_id=context_id)\n\n\ndef execute_command(\n client: WorkspaceClient,\n code: str,\n cluster_id: str,\n context_id: Optional[str] = None,\n language: str = \"python\",\n timeout: int = 300,\n destroy_context_on_completion: bool = False,\n) -> ExecutionResult:\n \"\"\"\n Execute code on a Databricks cluster.\n\n Args:\n client: WorkspaceClient\n code: Code to execute\n cluster_id: Cluster ID\n context_id: Optional existing context ID (for state preservation)\n language: \"python\", \"scala\", \"sql\", or \"r\"\n timeout: Timeout in seconds\n destroy_context_on_completion: Whether to destroy context after execution\n\n Returns:\n ExecutionResult\n \"\"\"\n # Get cluster name for better output\n try:\n cluster_info = client.clusters.get(cluster_id)\n cluster_name = cluster_info.cluster_name\n except Exception:\n cluster_name = cluster_id\n\n # Create context if not provided\n context_created = False\n if context_id is None:\n context_id = create_context(client, cluster_id, language)\n context_created = True\n\n lang_enum = _LANGUAGE_MAP.get(language.lower(), Language.PYTHON)\n\n try:\n result = client.command_execution.execute(\n cluster_id=cluster_id,\n context_id=context_id,\n language=lang_enum,\n command=code,\n ).result(timeout=datetime.timedelta(seconds=timeout))\n\n if result.status == CommandStatus.FINISHED:\n # Check for error in results\n if result.results and result.results.result_type and result.results.result_type.value == \"error\":\n error_msg = result.results.cause if result.results.cause else \"Unknown error\"\n return ExecutionResult(\n success=False,\n error=error_msg,\n cluster_id=cluster_id,\n cluster_name=cluster_name,\n context_id=context_id,\n context_destroyed=False,\n )\n\n output = result.results.data if result.results and result.results.data else \"Success (no output)\"\n exec_result = ExecutionResult(\n success=True,\n output=str(output),\n cluster_id=cluster_id,\n cluster_name=cluster_name,\n context_id=context_id,\n context_destroyed=False,\n )\n elif result.status in [CommandStatus.ERROR, CommandStatus.CANCELLED]:\n error_msg = result.results.cause if result.results and result.results.cause else \"Unknown error\"\n exec_result = ExecutionResult(\n success=False,\n error=error_msg,\n cluster_id=cluster_id,\n cluster_name=cluster_name,\n context_id=context_id,\n context_destroyed=False,\n )\n else:\n exec_result = ExecutionResult(\n success=False,\n error=f\"Unexpected status: {result.status}\",\n cluster_id=cluster_id,\n cluster_name=cluster_name,\n context_id=context_id,\n context_destroyed=False,\n )\n\n # Destroy context if requested\n if destroy_context_on_completion:\n try:\n destroy_context(client, cluster_id, context_id)\n exec_result.context_destroyed = True\n except Exception:\n pass\n\n return exec_result\n\n except TimeoutError:\n return ExecutionResult(\n success=False,\n error=f\"Command timed out after {timeout}s\",\n cluster_id=cluster_id,\n cluster_name=cluster_name,\n context_id=context_id,\n context_destroyed=False,\n )\n except Exception as e:\n if context_created and destroy_context_on_completion:\n try:\n destroy_context(client, cluster_id, context_id)\n except Exception:\n pass\n return ExecutionResult(\n success=False,\n error=str(e),\n cluster_id=cluster_id,\n cluster_name=cluster_name,\n context_id=context_id if not destroy_context_on_completion else None,\n context_destroyed=destroy_context_on_completion,\n )\n\n\ndef execute_file(\n client: WorkspaceClient,\n file_path: str,\n cluster_id: str,\n context_id: Optional[str] = None,\n timeout: int = 600,\n destroy_context_on_completion: bool = False,\n) -> ExecutionResult:\n \"\"\"\n Execute a local Python file on a Databricks cluster.\n\n Args:\n client: WorkspaceClient\n file_path: Path to the Python file\n cluster_id: Cluster ID\n context_id: Optional existing context ID\n timeout: Timeout in seconds\n destroy_context_on_completion: Whether to destroy context after\n\n Returns:\n ExecutionResult\n \"\"\"\n try:\n with open(file_path, \"r\", encoding=\"utf-8\") as f:\n code = f.read()\n except FileNotFoundError:\n return ExecutionResult(success=False, error=f\"File not found: {file_path}\")\n except Exception as e:\n return ExecutionResult(success=False, error=f\"Failed to read file: {e}\")\n\n if not code.strip():\n return ExecutionResult(success=False, error=f\"File is empty: {file_path}\")\n\n return execute_command(\n client=client,\n code=code,\n cluster_id=cluster_id,\n context_id=context_id,\n language=\"python\",\n timeout=timeout,\n destroy_context_on_completion=destroy_context_on_completion,\n )\n" }, { "path": "ai_release/inspect_jobs.py", "content": "#!/usr/bin/env python3\n\"\"\"\nJob Inspection CLI for DBDemos\n\nInspect bundle jobs, check their status, and get detailed failure information.\nAutomatically extracts errors from notebook HTML when API doesn't provide them.\n\nUsage:\n # List all bundle jobs with their status\n python ai_release/inspect_jobs.py --list\n\n # List only failed jobs\n python ai_release/inspect_jobs.py --list --failed-only\n\n # Get detailed info for a specific demo (auto-fetches errors)\n python ai_release/inspect_jobs.py --demo ai-agent\n\n # Get detailed failure info with fix suggestions\n python ai_release/inspect_jobs.py --demo ai-agent --errors\n\n # Export notebook path for the failed task\n python ai_release/inspect_jobs.py --demo ai-agent --notebook-path\n\n # Check if job is up-to-date with HEAD commit\n python ai_release/inspect_jobs.py --demo ai-agent --check-commit\n\n # Get task output for debugging\n python ai_release/inspect_jobs.py --task-output <task_run_id>\n\n # Export failure summary to file\n python ai_release/inspect_jobs.py --demo ai-agent --errors --output errors.txt\n\"\"\"\n\nimport argparse\nimport json\nimport sys\nfrom datetime import datetime\nfrom pathlib import Path\n\n# Add parent directory to path\nsys.path.insert(0, str(Path(__file__).parent.parent))\n\nfrom ai_release.jobs import JobInspector, load_inspector_from_config, JobInfo\n\n\ndef format_timestamp(ts: int) -> str:\n \"\"\"Format a millisecond timestamp to human-readable string.\"\"\"\n if not ts:\n return \"N/A\"\n return datetime.fromtimestamp(ts / 1000).strftime(\"%Y-%m-%d %H:%M:%S\")\n\n\ndef format_duration(start: int, end: int) -> str:\n \"\"\"Format duration between two timestamps.\"\"\"\n if not start or not end:\n return \"N/A\"\n duration_sec = (end - start) / 1000\n if duration_sec < 60:\n return f\"{duration_sec:.0f}s\"\n elif duration_sec < 3600:\n return f\"{duration_sec / 60:.1f}m\"\n else:\n return f\"{duration_sec / 3600:.1f}h\"\n\n\ndef print_job_list(jobs: list, failed_only: bool = False):\n \"\"\"Print a formatted list of jobs.\"\"\"\n if failed_only:\n jobs = [j for j in jobs if j.latest_run and j.latest_run.failed]\n\n if not jobs:\n print(\"No jobs found.\")\n return\n\n print(f\"\\n{'Demo Name':<40} {'State':<12} {'Result':<10} {'Run Time':<12} {'Commit':<10}\")\n print(\"=\" * 90)\n\n for job in sorted(jobs, key=lambda j: j.demo_name):\n run = job.latest_run\n if run:\n state_icon = \"🟢\" if run.succeeded else \"🔴\" if run.failed else \"🟡\" if run.running else \"⚪\"\n result = run.result_state or run.state\n duration = format_duration(run.start_time, run.end_time)\n commit = (run.used_commit or \"\")[:8]\n else:\n state_icon = \"⚪\"\n result = \"NO RUNS\"\n duration = \"N/A\"\n commit = \"N/A\"\n\n print(f\"{state_icon} {job.demo_name:<38} {result:<12} {duration:<12} {commit:<10}\")\n\n # Summary\n total = len(jobs)\n succeeded = len([j for j in jobs if j.latest_run and j.latest_run.succeeded])\n failed = len([j for j in jobs if j.latest_run and j.latest_run.failed])\n running = len([j for j in jobs if j.latest_run and j.latest_run.running])\n\n print(f\"\\nTotal: {total} | ✓ Succeeded: {succeeded} | ✗ Failed: {failed} | ◐ Running: {running}\")\n\n\ndef print_fix_workflow(job: JobInfo, inspector: JobInspector):\n \"\"\"Print suggested fix workflow for a failed job.\"\"\"\n print(\"\\n\" + \"=\" * 80)\n print(\"SUGGESTED FIX WORKFLOW\")\n print(\"=\" * 80)\n\n demo_name = job.demo_name\n run = job.latest_run\n\n # Get the notebook path from the first failed task\n notebook_path = None\n if run and run.failed_tasks:\n notebook_path = run.failed_tasks[0].notebook_path\n\n print(f\"\"\"\n1. TEST FIX INTERACTIVELY (optional but recommended):\n python ai_release/run_remote.py --start-cluster --wait-for-cluster\n python ai_release/run_remote.py --code \"# test your fix code here\"\n\n2. CREATE FIX BRANCH in dbdemos-notebooks:\n cd ../dbdemos-notebooks\n git checkout main && git pull origin main\n git checkout -b ai-fix-{demo_name}-<issue>\n\n3. EDIT THE NOTEBOOK:\n {notebook_path or 'Check the failed task notebook path above'}\n\n4. COMMIT AND PUSH:\n git add . && git commit -m \"fix: <description>\" && git push origin ai-fix-{demo_name}-<issue>\n\n5. TEST THE FIX:\n cd ../dbdemos\n python ai_release/bundle.py --demo {demo_name} --branch ai-fix-{demo_name}-<issue> --force\n\n6. IF STILL FAILING - iterate with repair (faster):\n python ai_release/bundle.py --demo {demo_name} --repair --wait\n\n7. CREATE PR when tests pass:\n cd ../dbdemos-notebooks\n gh pr create --title \"fix: {demo_name} <issue>\" --body \"Fixed <issue>\"\n\n8. AFTER PR MERGED - final verification:\n cd ../dbdemos\n python ai_release/bundle.py --demo {demo_name} --force\n\"\"\")\n\n\ndef print_job_details(job: JobInfo, inspector: JobInspector, show_errors: bool = False,\n check_commit: bool = False, show_workflow: bool = True):\n \"\"\"Print detailed information about a job.\"\"\"\n print(f\"\\n{'=' * 80}\")\n print(f\"Demo: {job.demo_name}\")\n print(f\"Job ID: {job.job_id}\")\n print(f\"Job URL: {inspector.get_job_url(job.job_id)}\")\n print(f\"{'=' * 80}\")\n\n run = job.latest_run\n if not run:\n print(\"\\nNo runs found for this job.\")\n return\n\n print(f\"\\nLatest Run: {run.run_id}\")\n print(f\"Run URL: {inspector.get_job_url(job.job_id, run.run_id)}\")\n print(f\"State: {run.state}\")\n print(f\"Result: {run.result_state or 'N/A'}\")\n print(f\"Started: {format_timestamp(run.start_time)}\")\n print(f\"Ended: {format_timestamp(run.end_time)}\")\n print(f\"Duration: {format_duration(run.start_time, run.end_time)}\")\n print(f\"Git Commit: {run.used_commit or 'N/A'}\")\n\n if run.state_message:\n print(f\"Message: {run.state_message}\")\n\n # Check commit status\n if check_commit:\n print(f\"\\n--- Git Commit Check ---\")\n head = inspector.get_head_commit()\n if head:\n print(f\"HEAD Commit: {head}\")\n if run.used_commit:\n if run.used_commit == head:\n print(\"✓ Job is UP-TO-DATE with HEAD\")\n else:\n print(\"✗ Job is OUTDATED - HEAD has newer commits\")\n else:\n print(\"? Cannot determine - no commit info in job run\")\n else:\n print(\"Could not fetch HEAD commit from GitHub\")\n\n # Print tasks\n print(f\"\\n--- Tasks ({len(run.tasks)} total) ---\")\n for task in run.tasks:\n icon = \"✓\" if task.state == \"SUCCESS\" else \"✗\" if task.failed else \"○\"\n notebook = task.notebook_path.split(\"/\")[-1] if task.notebook_path else \"N/A\"\n print(f\" {icon} {task.task_key}: {task.state} ({notebook})\")\n\n # Print errors if job failed (always show for failed jobs, more detail with --errors)\n if run.failed_tasks:\n print(f\"\\n{'=' * 80}\")\n print(\"FAILURE DETAILS\")\n print(f\"{'=' * 80}\")\n\n for task in run.failed_tasks:\n print(f\"\\n--- Task: {task.task_key} ---\")\n if task.notebook_path:\n print(f\"Notebook: {task.notebook_path}\")\n\n # Show error summary\n if task.error_message:\n print(f\"\\nError: {task.error_message}\")\n\n # Show notebook errors if available\n if task.notebook_errors:\n print(f\"\\n--- Notebook Cell Errors ({len(task.notebook_errors)} found) ---\")\n for err in task.notebook_errors:\n print(f\"\\n[Cell {err.cell_index}] {err.error_name}: {err.error_message}\")\n if err.cell_source and show_errors:\n # Show the code that caused the error\n src = err.cell_source\n if len(src) > 500:\n src = src[:500] + \"\\n... (truncated)\"\n print(f\"\\nCode:\\n{src}\")\n if err.error_trace and show_errors:\n trace = err.error_trace\n if len(trace) > 2000:\n trace = trace[:2000] + \"\\n... (truncated)\"\n print(f\"\\nTraceback:\\n{trace}\")\n\n # Fallback to API trace if no notebook errors\n elif task.error_trace and show_errors:\n trace = task.error_trace\n if len(trace) > 3000:\n trace = trace[:3000] + \"\\n... (truncated, use --task-output for full trace)\"\n print(f\"\\nStack Trace:\\n{trace}\")\n\n # Show fix workflow for failed jobs\n if show_workflow:\n print_fix_workflow(job, inspector)\n\n\ndef print_task_output(inspector: JobInspector, task_run_id: int):\n \"\"\"Print the full output from a task run, including exported notebook errors.\"\"\"\n print(f\"\\n{'=' * 80}\")\n print(f\"Task Run ID: {task_run_id}\")\n print(f\"{'=' * 80}\")\n\n # First try standard API output\n output = inspector.get_task_output(task_run_id)\n if output:\n if output.get(\"error\"):\n print(f\"\\nAPI Error:\\n{output['error']}\")\n if output.get(\"error_trace\"):\n print(f\"\\nAPI Stack Trace:\\n{output['error_trace']}\")\n if output.get(\"notebook_output\"):\n print(f\"\\nNotebook Output:\\n{output['notebook_output']}\")\n\n # Also export and parse notebook HTML for cell-level errors\n print(\"\\n--- Extracting errors from notebook HTML ---\")\n html = inspector.export_notebook_html(task_run_id)\n if html:\n errors = inspector.extract_errors_from_html(html)\n if errors:\n print(f\"Found {len(errors)} error(s) in notebook cells:\")\n for err in errors:\n print(f\"\\n[Cell {err.cell_index}] {err.error_name}: {err.error_message}\")\n if err.cell_source:\n print(f\"\\nCode:\\n{err.cell_source}\")\n if err.error_trace:\n print(f\"\\nTraceback:\\n{err.error_trace}\")\n else:\n print(\"No cell errors found in notebook HTML\")\n else:\n print(\"Could not export notebook HTML\")\n\n\ndef main():\n parser = argparse.ArgumentParser(\n description=\"Inspect DBDemos bundle jobs\",\n formatter_class=argparse.RawDescriptionHelpFormatter,\n epilog=__doc__\n )\n\n # Main actions\n parser.add_argument(\"--list\", \"-l\", action=\"store_true\", help=\"List all bundle jobs\")\n parser.add_argument(\"--demo\", \"-d\", help=\"Get details for a specific demo\")\n parser.add_argument(\"--task-output\", type=int, help=\"Get output from a specific task run ID\")\n\n # Options\n parser.add_argument(\"--failed-only\", \"-f\", action=\"store_true\", help=\"Only show failed jobs\")\n parser.add_argument(\"--errors\", \"-e\", action=\"store_true\", help=\"Show detailed error traces and code\")\n parser.add_argument(\"--check-commit\", \"-c\", action=\"store_true\", help=\"Check if job is up-to-date with HEAD\")\n parser.add_argument(\"--no-workflow\", action=\"store_true\", help=\"Don't show fix workflow suggestions\")\n parser.add_argument(\"--notebook-path\", action=\"store_true\", help=\"Print only the notebook path for the first failed task\")\n parser.add_argument(\"--output\", \"-o\", help=\"Write output to file\")\n parser.add_argument(\"--json\", action=\"store_true\", help=\"Output as JSON\")\n\n args = parser.parse_args()\n\n # Load inspector\n try:\n inspector = load_inspector_from_config()\n print(f\"Workspace: {inspector.host}\")\n except Exception as e:\n print(f\"Error loading config: {e}\")\n return 1\n\n # Redirect output to file if requested\n output_file = None\n if args.output:\n output_file = open(args.output, \"w\")\n sys.stdout = output_file\n\n try:\n # List jobs\n if args.list:\n print(\"\\nFetching bundle jobs...\")\n jobs = inspector.list_bundle_jobs(include_run_details=True)\n print_job_list(jobs, failed_only=args.failed_only)\n return 0\n\n # Get demo details\n if args.demo:\n print(f\"\\nFetching job for demo: {args.demo}\")\n job = inspector.find_job(args.demo)\n if not job:\n print(f\"No job found for demo: {args.demo}\")\n return 1\n\n # Always get full details for failed jobs (to get errors)\n if job.latest_run and (job.latest_run.failed or args.errors or args.check_commit):\n print(\"Fetching error details...\")\n job.latest_run = inspector.get_job_run_details(job.job_id, job.latest_run.run_id)\n\n # Just print notebook path if requested\n if args.notebook_path:\n if job.latest_run and job.latest_run.failed_tasks:\n for task in job.latest_run.failed_tasks:\n if task.notebook_path:\n print(task.notebook_path)\n return 0\n\n if args.json:\n # Output as JSON for programmatic use\n data = {\n \"demo_name\": job.demo_name,\n \"job_id\": job.job_id,\n \"job_url\": inspector.get_job_url(job.job_id),\n }\n if job.latest_run:\n data[\"latest_run\"] = {\n \"run_id\": job.latest_run.run_id,\n \"state\": job.latest_run.state,\n \"result_state\": job.latest_run.result_state,\n \"used_commit\": job.latest_run.used_commit,\n \"failed_tasks\": [\n {\n \"task_key\": t.task_key,\n \"run_id\": t.run_id,\n \"notebook_path\": t.notebook_path,\n \"error_message\": t.error_message,\n \"error_trace\": t.error_trace,\n \"notebook_errors\": [\n {\n \"cell_index\": e.cell_index,\n \"error_name\": e.error_name,\n \"error_message\": e.error_message,\n \"cell_source\": e.cell_source,\n }\n for e in t.notebook_errors\n ] if t.notebook_errors else []\n }\n for t in job.latest_run.failed_tasks\n ]\n }\n print(json.dumps(data, indent=2))\n else:\n print_job_details(job, inspector,\n show_errors=args.errors,\n check_commit=args.check_commit,\n show_workflow=not args.no_workflow)\n return 0\n\n # Get task output\n if args.task_output:\n print_task_output(inspector, args.task_output)\n return 0\n\n # No action specified\n parser.print_help()\n return 1\n\n finally:\n if output_file:\n output_file.close()\n sys.stdout = sys.__stdout__\n print(f\"Output written to: {args.output}\")\n\n\nif __name__ == \"__main__\":\n sys.exit(main())\n" }, { "path": "ai_release/jobs.py", "content": "\"\"\"\nJob Inspection Module for DBDemos\n\nProvides functions to inspect bundle jobs, get failure details, and compare git commits.\nUses the Databricks SDK for all API operations.\n\nSDK Documentation: https://databricks-sdk-py.readthedocs.io/en/latest/\n\"\"\"\n\nimport json\nimport re\nimport requests\nimport urllib.parse\nfrom bs4 import BeautifulSoup\nfrom dataclasses import dataclass, field\nfrom typing import Optional, List, Dict, Any\nfrom pathlib import Path\nfrom html import unescape\n\nfrom databricks.sdk import WorkspaceClient\nfrom databricks.sdk.service.jobs import RunResultState, RunLifeCycleState, ViewsToExport\n\n\n@dataclass\nclass NotebookError:\n \"\"\"Error extracted from a notebook cell.\"\"\"\n cell_index: int\n cell_type: str # \"code\", \"markdown\"\n error_name: str # e.g., \"NameError\", \"ValueError\"\n error_message: str\n error_trace: Optional[str] = None\n cell_source: Optional[str] = None # The code that caused the error\n\n\n@dataclass\nclass TaskResult:\n \"\"\"Result from a single task in a job run.\"\"\"\n task_key: str\n run_id: int\n state: str # SUCCESS, FAILED, SKIPPED, etc.\n notebook_path: Optional[str] = None\n error_message: Optional[str] = None\n error_trace: Optional[str] = None\n used_commit: Optional[str] = None\n notebook_errors: List[NotebookError] = field(default_factory=list)\n\n @property\n def failed(self) -> bool:\n return self.state in (\"FAILED\", \"TIMEDOUT\", \"CANCELED\")\n\n def get_error_summary(self) -> str:\n \"\"\"Get a summary of all errors for this task.\"\"\"\n if not self.failed:\n return \"Task succeeded\"\n\n lines = []\n if self.error_message:\n lines.append(f\"Error: {self.error_message}\")\n if self.error_trace:\n lines.append(f\"Trace: {self.error_trace[:500]}...\")\n\n for err in self.notebook_errors:\n lines.append(f\"\\n[Cell {err.cell_index}] {err.error_name}: {err.error_message}\")\n if err.cell_source:\n # Show first 200 chars of source\n src = err.cell_source[:200]\n if len(err.cell_source) > 200:\n src += \"...\"\n lines.append(f\"Code: {src}\")\n if err.error_trace:\n lines.append(f\"Traceback:\\n{err.error_trace}\")\n\n return \"\\n\".join(lines) if lines else \"Unknown error\"\n\n\n@dataclass\nclass JobRunResult:\n \"\"\"Result from a job run with all task details.\"\"\"\n job_id: int\n job_name: str\n run_id: int\n state: str # RUNNING, TERMINATED, etc.\n result_state: Optional[str] = None # SUCCESS, FAILED, etc.\n state_message: Optional[str] = None\n start_time: Optional[int] = None\n end_time: Optional[int] = None\n tasks: List[TaskResult] = field(default_factory=list)\n used_commit: Optional[str] = None # Most recent commit from tasks\n\n @property\n def succeeded(self) -> bool:\n return self.result_state == \"SUCCESS\"\n\n @property\n def failed(self) -> bool:\n return self.result_state in (\"FAILED\", \"TIMEDOUT\", \"CANCELED\")\n\n @property\n def running(self) -> bool:\n return self.state == \"RUNNING\"\n\n @property\n def failed_tasks(self) -> List[TaskResult]:\n return [t for t in self.tasks if t.failed]\n\n def get_failure_summary(self) -> str:\n \"\"\"Get a human-readable summary of failures.\"\"\"\n if self.succeeded:\n return \"Job succeeded\"\n\n lines = [f\"Job {self.job_name} FAILED\"]\n if self.state_message:\n lines.append(f\"Message: {self.state_message}\")\n\n for task in self.failed_tasks:\n lines.append(f\"\\n--- Task: {task.task_key} ---\")\n if task.notebook_path:\n lines.append(f\"Notebook: {task.notebook_path}\")\n if task.error_message:\n lines.append(f\"Error: {task.error_message}\")\n if task.error_trace:\n # Truncate long traces\n trace = task.error_trace\n if len(trace) > 2000:\n trace = trace[:2000] + \"\\n... (truncated)\"\n lines.append(f\"Trace:\\n{trace}\")\n\n return \"\\n\".join(lines)\n\n\n@dataclass\nclass JobInfo:\n \"\"\"Information about a bundle job.\"\"\"\n job_id: int\n job_name: str\n demo_name: str\n latest_run: Optional[JobRunResult] = None\n head_commit: Optional[str] = None\n is_up_to_date: Optional[bool] = None # True if latest run used HEAD commit\n\n\nclass JobInspector:\n \"\"\"\n Inspects bundle jobs and retrieves detailed failure information.\n Uses the Databricks SDK for all API operations.\n\n Usage:\n inspector = JobInspector(host, token, github_token, repo_url)\n\n # List all bundle jobs\n jobs = inspector.list_bundle_jobs()\n\n # Get detailed failure info\n result = inspector.get_job_run_details(job_id, run_id)\n print(result.get_failure_summary())\n\n # Get task error output\n output = inspector.get_task_output(task_run_id)\n \"\"\"\n\n # Both prefixes are used - field-demos_ for demos, field-bundle_ for bundling\n JOB_PREFIXES = [\"field-demos_\", \"field-bundle_\"]\n\n def __init__(self, host: str, token: str, github_token: str = None, repo_url: str = None):\n self.host = host.rstrip(\"/\")\n self.token = token\n self.github_token = github_token\n self.repo_url = repo_url\n\n # Create Databricks SDK client\n self.ws = WorkspaceClient(\n host=host,\n token=token,\n auth_type=\"pat\",\n product=\"dbdemos-ai-release\",\n product_version=\"0.1.0\"\n )\n\n def _github_get(self, path: str) -> dict:\n \"\"\"Make a GET request to the GitHub API.\"\"\"\n if not self.github_token:\n raise ValueError(\"GitHub token required for this operation\")\n headers = {\n \"Accept\": \"application/vnd.github.v3+json\",\n \"Authorization\": f\"token {self.github_token}\"\n }\n url = f\"https://api.github.com/{path}\"\n resp = requests.get(url, headers=headers, timeout=60)\n return resp.json()\n\n def list_bundle_jobs(self, include_run_details: bool = True) -> List[JobInfo]:\n \"\"\"\n List all bundle jobs (jobs with 'field-bundle_' prefix).\n\n Args:\n include_run_details: If True, fetches latest run details for each job\n\n Returns:\n List of JobInfo objects\n \"\"\"\n jobs = []\n\n # List all jobs using SDK\n for job in self.ws.jobs.list():\n name = job.settings.name if job.settings else None\n if not name:\n continue\n\n # Check all known prefixes\n demo_name = None\n for prefix in self.JOB_PREFIXES:\n if name.startswith(prefix):\n demo_name = name[len(prefix):]\n break\n\n if demo_name:\n job_info = JobInfo(\n job_id=job.job_id,\n job_name=name,\n demo_name=demo_name\n )\n\n if include_run_details:\n # Get latest run using SDK\n runs = list(self.ws.jobs.list_runs(job_id=job.job_id, limit=1, expand_tasks=True))\n if runs:\n job_info.latest_run = self._parse_run(runs[0], name)\n\n jobs.append(job_info)\n\n return jobs\n\n def find_job(self, demo_name: str) -> Optional[JobInfo]:\n \"\"\"Find a bundle job by demo name. Tries all known prefixes.\"\"\"\n # Try each prefix\n for prefix in self.JOB_PREFIXES:\n job_name = f\"{prefix}{demo_name}\"\n\n # Search with name filter using SDK\n for job in self.ws.jobs.list(name=job_name):\n if job.settings and job.settings.name == job_name:\n job_info = JobInfo(\n job_id=job.job_id,\n job_name=job_name,\n demo_name=demo_name\n )\n\n # Get latest run\n runs = list(self.ws.jobs.list_runs(job_id=job.job_id, limit=1, expand_tasks=True))\n if runs:\n job_info.latest_run = self._parse_run(runs[0], job_name)\n\n return job_info\n\n return None\n\n def get_job_run_details(self, job_id: int, run_id: int = None) -> Optional[JobRunResult]:\n \"\"\"\n Get detailed information about a job run.\n\n Args:\n job_id: The job ID\n run_id: Specific run ID. If None, gets the latest run.\n\n Returns:\n JobRunResult with full task details and errors\n \"\"\"\n if run_id is None:\n # Get latest run\n runs = list(self.ws.jobs.list_runs(job_id=job_id, limit=1, expand_tasks=True))\n if not runs:\n return None\n run = runs[0]\n else:\n run = self.ws.jobs.get_run(run_id=run_id)\n\n # Get job name\n job = self.ws.jobs.get(job_id=job_id)\n job_name = job.settings.name if job.settings else f\"job_{job_id}\"\n\n result = self._parse_run(run, job_name)\n\n # For failed tasks, get detailed error output (API + notebook HTML)\n for task in result.failed_tasks:\n self.get_task_errors(task)\n\n return result\n\n def _parse_run(self, run, job_name: str) -> JobRunResult:\n \"\"\"Parse a run object from SDK into a JobRunResult.\"\"\"\n # Get state info\n state = run.state\n lifecycle_state = state.life_cycle_state.value if state and state.life_cycle_state else \"UNKNOWN\"\n result_state = state.result_state.value if state and state.result_state else None\n state_message = state.state_message if state else None\n\n # Parse tasks\n tasks = []\n most_recent_commit = None\n\n for task in (run.tasks or []):\n task_state = task.state\n task_result_state = task_state.result_state.value if task_state and task_state.result_state else \"UNKNOWN\"\n\n # Get commit from git_source\n used_commit = None\n if task.git_source and task.git_source.git_snapshot:\n used_commit = task.git_source.git_snapshot.used_commit\n if used_commit and (not most_recent_commit or used_commit > most_recent_commit):\n most_recent_commit = used_commit\n\n notebook_path = None\n if task.notebook_task:\n notebook_path = task.notebook_task.notebook_path\n\n task_result = TaskResult(\n task_key=task.task_key or \"unknown\",\n run_id=task.run_id or 0,\n state=task_result_state,\n notebook_path=notebook_path,\n used_commit=used_commit\n )\n tasks.append(task_result)\n\n return JobRunResult(\n job_id=run.job_id or 0,\n job_name=job_name,\n run_id=run.run_id or 0,\n state=lifecycle_state,\n result_state=result_state,\n state_message=state_message,\n start_time=run.start_time,\n end_time=run.end_time,\n tasks=tasks,\n used_commit=most_recent_commit\n )\n\n def get_task_output(self, task_run_id: int) -> Optional[Dict[str, Any]]:\n \"\"\"\n Get the output/error from a specific task run.\n\n Args:\n task_run_id: The task's run_id (not the job run_id)\n\n Returns:\n Dict with 'error' and 'error_trace' if available\n \"\"\"\n try:\n output = self.ws.jobs.get_run_output(run_id=task_run_id)\n return {\n \"error\": output.error,\n \"error_trace\": output.error_trace,\n \"metadata\": str(output.metadata) if output.metadata else None,\n \"notebook_output\": str(output.notebook_output) if output.notebook_output else None\n }\n except Exception as e:\n return {\"error\": str(e)}\n\n def export_notebook_html(self, task_run_id: int) -> Optional[str]:\n \"\"\"\n Export the notebook HTML from a task run.\n\n Args:\n task_run_id: The task's run_id\n\n Returns:\n HTML content of the notebook with outputs, or None if failed\n \"\"\"\n try:\n export = self.ws.jobs.export_run(run_id=task_run_id, views_to_export=ViewsToExport.ALL)\n if export.views and len(export.views) > 0:\n return export.views[0].content\n return None\n except Exception as e:\n print(f\"Failed to export notebook: {e}\")\n return None\n\n def extract_errors_from_html(self, html_content: str) -> List[NotebookError]:\n \"\"\"\n Parse notebook HTML and extract error information from failed cells.\n The HTML contains a base64+URL encoded JSON model with command details.\n\n Args:\n html_content: The HTML content from export_notebook_html\n\n Returns:\n List of NotebookError objects for each failed cell\n \"\"\"\n import base64\n\n errors = []\n\n # Find the notebook model in the HTML - it's base64 then URL encoded JSON\n match = re.search(r'__DATABRICKS_NOTEBOOK_MODEL = \\'([^\\']+)\\'', html_content)\n if not match:\n return errors\n\n try:\n encoded = match.group(1)\n # Decode: base64 -> URL encoding -> JSON\n decoded_bytes = base64.b64decode(encoded)\n url_encoded = decoded_bytes.decode('utf-8')\n json_str = urllib.parse.unquote(url_encoded)\n model = json.loads(json_str)\n except Exception as e:\n print(f\"Failed to parse notebook model: {e}\")\n return errors\n\n # Extract errors from commands\n for idx, cmd in enumerate(model.get('commands', [])):\n state = cmd.get('state')\n error_summary = cmd.get('errorSummary')\n error = cmd.get('error')\n\n # Skip non-error commands and \"Command skipped\" errors (not the root cause)\n if not (error or error_summary) or state != 'error':\n continue\n if error_summary == 'Command skipped':\n continue\n\n # Get command source\n cell_source = cmd.get('command', '')\n\n # Parse error name and message\n error_name = \"Error\"\n error_message = error_summary or \"Unknown error\"\n error_trace = None\n\n # Try to parse Python exception from error_summary\n exc_match = re.search(r'(\\w+Error|\\w+Exception):\\s*(.+)', error_summary or '')\n if exc_match:\n error_name = exc_match.group(1)\n error_message = exc_match.group(2).strip()\n\n # Clean ANSI codes from error trace\n if error:\n # Remove ANSI escape codes\n error_trace = re.sub(r'\\x1b\\[[0-9;]*m', '', str(error))\n\n errors.append(NotebookError(\n cell_index=idx,\n cell_type=\"code\",\n error_name=error_name,\n error_message=error_message,\n error_trace=error_trace,\n cell_source=cell_source[:500] if cell_source else None # Truncate source\n ))\n\n return errors\n\n def get_task_errors(self, task: TaskResult) -> TaskResult:\n \"\"\"\n Get comprehensive error information for a failed task.\n First tries API, then falls back to exporting and parsing notebook HTML.\n\n Args:\n task: TaskResult to enrich with error information\n\n Returns:\n The same TaskResult with error fields populated\n \"\"\"\n # First try the standard API\n output = self.get_task_output(task.run_id)\n if output:\n task.error_message = output.get(\"error\")\n task.error_trace = output.get(\"error_trace\")\n\n # If no error from API, export and parse the notebook\n if not task.error_message and not task.error_trace:\n html = self.export_notebook_html(task.run_id)\n if html:\n errors = self.extract_errors_from_html(html)\n task.notebook_errors = errors\n\n # Set primary error from first notebook error\n if errors:\n first_err = errors[0]\n task.error_message = f\"{first_err.error_name}: {first_err.error_message}\"\n task.error_trace = first_err.error_trace\n\n return task\n\n def get_head_commit(self) -> Optional[str]:\n \"\"\"Get the HEAD commit SHA from the GitHub repo.\"\"\"\n if not self.repo_url or not self.github_token:\n return None\n\n # Extract owner/repo from URL\n match = re.search(r'github\\.com[/:]([^/]+)/([^/\\.]+)', self.repo_url)\n if not match:\n return None\n\n owner, repo = match.groups()\n resp = self._github_get(f\"repos/{owner}/{repo}/commits/HEAD\")\n return resp.get(\"sha\")\n\n def check_job_up_to_date(self, job_info: JobInfo) -> bool:\n \"\"\"\n Check if a job's latest run used the HEAD commit.\n\n Args:\n job_info: JobInfo with latest_run populated\n\n Returns:\n True if the job was run with the latest commit\n \"\"\"\n if not job_info.latest_run or not job_info.latest_run.used_commit:\n return False\n\n head_commit = self.get_head_commit()\n if not head_commit:\n return False\n\n job_info.head_commit = head_commit\n job_info.is_up_to_date = job_info.latest_run.used_commit == head_commit\n return job_info.is_up_to_date\n\n def get_failed_jobs(self) -> List[JobInfo]:\n \"\"\"Get all bundle jobs that have a failed latest run.\"\"\"\n all_jobs = self.list_bundle_jobs(include_run_details=True)\n return [j for j in all_jobs if j.latest_run and j.latest_run.failed]\n\n def get_job_url(self, job_id: int, run_id: int = None) -> str:\n \"\"\"Get the workspace URL for a job or run.\"\"\"\n if run_id:\n return f\"{self.host}/#job/{job_id}/run/{run_id}\"\n return f\"{self.host}/#job/{job_id}\"\n\n\ndef load_inspector_from_config() -> JobInspector:\n \"\"\"Load a JobInspector using the local config file.\"\"\"\n repo_root = Path(__file__).parent.parent\n conf_files = [\n repo_root / \"local_conf_E2TOOL.json\",\n repo_root / \"local_conf.json\",\n ]\n\n config = None\n for conf_file in conf_files:\n if conf_file.exists():\n with open(conf_file, \"r\") as f:\n config = json.load(f)\n break\n\n if not config:\n raise FileNotFoundError(\"No config file found\")\n\n # Clean repo_url\n repo_url = config.get(\"repo_url\", \"\")\n if repo_url.endswith(\".git\"):\n repo_url = repo_url[:-4]\n\n return JobInspector(\n host=config[\"url\"],\n token=config[\"pat_token\"],\n github_token=config.get(\"github_token\"),\n repo_url=repo_url\n )\n" }, { "path": "ai_release/run_remote.py", "content": "#!/usr/bin/env python3\n\"\"\"\nRemote Code Execution CLI for DBDemos\n\nExecute Python code on a Databricks cluster for testing notebook fixes.\n\nUsage:\n # Execute code directly\n python ai_release/run_remote.py --code \"print('Hello from Databricks!')\"\n\n # Execute a file\n python ai_release/run_remote.py --file path/to/script.py\n\n # Execute SQL\n python ai_release/run_remote.py --code \"SELECT 1\" --language sql\n\n # List available clusters\n python ai_release/run_remote.py --list-clusters\n\n # Start a cluster\n python ai_release/run_remote.py --start-cluster\n\n # Check cluster status\n python ai_release/run_remote.py --cluster-status\n\n # Reuse context for faster follow-up commands\n python ai_release/run_remote.py --code \"x = 1\" --save-context\n python ai_release/run_remote.py --code \"print(x)\" --load-context\n\nEnvironment Variables / Config:\n Uses local_conf_E2TOOL.json for credentials.\n Cluster is auto-selected by matching \"cluster_name_pattern\" (default: \"quentin\")\n\"\"\"\n\nimport argparse\nimport json\nimport os\nimport sys\nfrom pathlib import Path\n\n# Add parent directory to path\nsys.path.insert(0, str(Path(__file__).parent.parent))\n\nfrom ai_release.compute import (\n get_workspace_client,\n list_clusters,\n find_cluster_by_name,\n start_cluster,\n get_cluster_status,\n wait_for_cluster,\n execute_command,\n execute_file,\n)\n\nCONTEXT_FILE = Path(__file__).parent / \".execution_context.json\"\n\n\ndef load_config():\n \"\"\"Load configuration from local_conf_E2TOOL.json\"\"\"\n repo_root = Path(__file__).parent.parent\n conf_files = [\n repo_root / \"local_conf_E2TOOL.json\",\n repo_root / \"local_conf.json\",\n ]\n\n for conf_file in conf_files:\n if conf_file.exists():\n with open(conf_file, \"r\") as f:\n config = json.load(f)\n print(f\"Loaded config from {conf_file.name}\")\n return config\n\n print(\"ERROR: No config file found (local_conf_E2TOOL.json or local_conf.json)\")\n sys.exit(1)\n\n\ndef save_context(cluster_id: str, context_id: str):\n \"\"\"Save execution context for reuse.\"\"\"\n with open(CONTEXT_FILE, \"w\") as f:\n json.dump({\"cluster_id\": cluster_id, \"context_id\": context_id}, f)\n print(f\"Context saved to {CONTEXT_FILE}\")\n\n\ndef load_context():\n \"\"\"Load saved execution context.\"\"\"\n if CONTEXT_FILE.exists():\n with open(CONTEXT_FILE, \"r\") as f:\n return json.load(f)\n return None\n\n\ndef clear_context():\n \"\"\"Clear saved context.\"\"\"\n if CONTEXT_FILE.exists():\n CONTEXT_FILE.unlink()\n print(\"Context cleared\")\n\n\ndef main():\n parser = argparse.ArgumentParser(\n description=\"Execute code on Databricks clusters\",\n formatter_class=argparse.RawDescriptionHelpFormatter,\n epilog=__doc__,\n )\n\n # Execution options\n parser.add_argument(\"--code\", \"-c\", help=\"Code to execute\")\n parser.add_argument(\"--file\", \"-f\", help=\"Python file to execute\")\n parser.add_argument(\"--language\", \"-l\", default=\"python\", choices=[\"python\", \"sql\", \"scala\", \"r\"])\n parser.add_argument(\"--timeout\", \"-t\", type=int, default=300, help=\"Timeout in seconds\")\n\n # Cluster management\n parser.add_argument(\"--list-clusters\", action=\"store_true\", help=\"List available clusters\")\n parser.add_argument(\"--start-cluster\", action=\"store_true\", help=\"Start the configured cluster\")\n parser.add_argument(\"--cluster-status\", action=\"store_true\", help=\"Check cluster status\")\n parser.add_argument(\"--wait-for-cluster\", action=\"store_true\", help=\"Wait for cluster to be running\")\n parser.add_argument(\"--cluster-name\", help=\"Cluster name pattern to match (default: from config or 'quentin')\")\n\n # Context management\n parser.add_argument(\"--save-context\", action=\"store_true\", help=\"Save context for reuse\")\n parser.add_argument(\"--load-context\", action=\"store_true\", help=\"Reuse saved context\")\n parser.add_argument(\"--clear-context\", action=\"store_true\", help=\"Clear saved context\")\n parser.add_argument(\"--destroy-context\", action=\"store_true\", help=\"Destroy context after execution\")\n\n args = parser.parse_args()\n\n # Load config\n config = load_config()\n host = config.get(\"url\", os.environ.get(\"DATABRICKS_HOST\"))\n token = config.get(\"pat_token\", os.environ.get(\"DATABRICKS_TOKEN\"))\n cluster_pattern = args.cluster_name or config.get(\"cluster_name_pattern\", \"quentin\")\n\n if not host or not token:\n print(\"ERROR: Missing workspace URL or token\")\n sys.exit(1)\n\n # Create client\n client = get_workspace_client(host, token)\n print(f\"Workspace: {host}\")\n\n # Clear context\n if args.clear_context:\n clear_context()\n return 0\n\n # List clusters\n if args.list_clusters:\n clusters = list_clusters(client, include_terminated=True)\n print(f\"\\nFound {len(clusters)} clusters:\\n\")\n for c in clusters:\n state_icon = \"🟢\" if c[\"state\"] == \"RUNNING\" else \"🔴\" if c[\"state\"] == \"TERMINATED\" else \"🟡\"\n print(f\" {state_icon} {c['cluster_name']:<40} {c['state']:<12} {c['cluster_id']}\")\n return 0\n\n # Find cluster\n cluster = find_cluster_by_name(client, cluster_pattern)\n if not cluster:\n print(f\"ERROR: No cluster found matching '{cluster_pattern}'\")\n print(\"Use --list-clusters to see available clusters\")\n return 1\n\n print(f\"Cluster: {cluster['cluster_name']} ({cluster['state']})\")\n cluster_id = cluster[\"cluster_id\"]\n\n # Cluster status\n if args.cluster_status:\n status = get_cluster_status(client, cluster_id)\n print(f\" State: {status['state']}\")\n return 0\n\n # Start cluster\n if args.start_cluster:\n result = start_cluster(client, cluster_id)\n print(f\" {result['message']}\")\n if args.wait_for_cluster and result.get(\"state\") != \"RUNNING\":\n wait_for_cluster(client, cluster_id)\n return 0\n\n # Wait for cluster\n if args.wait_for_cluster:\n success = wait_for_cluster(client, cluster_id)\n return 0 if success else 1\n\n # Execute code\n if args.code or args.file:\n # Check cluster is running\n if cluster[\"state\"] != \"RUNNING\":\n print(f\"ERROR: Cluster is {cluster['state']}, not RUNNING\")\n print(\"Use --start-cluster --wait-for-cluster to start it\")\n return 1\n\n # Load context if requested\n context_id = None\n if args.load_context:\n saved = load_context()\n if saved and saved.get(\"cluster_id\") == cluster_id:\n context_id = saved.get(\"context_id\")\n print(f\"Reusing context: {context_id}\")\n else:\n print(\"No saved context found or cluster changed, creating new context\")\n\n # Execute\n if args.file:\n print(f\"\\nExecuting file: {args.file}\")\n result = execute_file(\n client=client,\n file_path=args.file,\n cluster_id=cluster_id,\n context_id=context_id,\n timeout=args.timeout,\n destroy_context_on_completion=args.destroy_context,\n )\n else:\n print(f\"\\nExecuting {args.language} code...\")\n result = execute_command(\n client=client,\n code=args.code,\n cluster_id=cluster_id,\n context_id=context_id,\n language=args.language,\n timeout=args.timeout,\n destroy_context_on_completion=args.destroy_context,\n )\n\n # Print result\n print(\"\\n\" + \"=\" * 60)\n if result.success:\n print(\"✓ SUCCESS\")\n print(\"=\" * 60)\n print(result.output)\n else:\n print(\"✗ FAILED\")\n print(\"=\" * 60)\n print(result.error)\n\n # Save context if requested\n if args.save_context and result.context_id and not result.context_destroyed:\n save_context(cluster_id, result.context_id)\n print(f\"\\nContext saved. Use --load-context to reuse.\")\n\n return 0 if result.success else 1\n\n parser.print_help()\n return 1\n\n\nif __name__ == \"__main__\":\n sys.exit(main())\n" }, { "path": "ai_release/run_state.py", "content": "\"\"\"\nRun state management for AI release workflow.\n\nTracks job runs, errors, and fixes in a persistent folder structure:\n ai_release/runs/\n <commit_id>/\n state.json - Overall run state\n <demo_name>/\n status.json - Demo-specific status\n errors.json - Extracted errors from failed runs\n fix_attempts.json - History of fix attempts\n job_output.log - Raw job output\n notes.md - AI notes and observations\n\"\"\"\n\nimport json\nimport os\nfrom dataclasses import dataclass, field, asdict\nfrom datetime import datetime\nfrom pathlib import Path\nfrom typing import Optional, List, Dict, Any\n\n\nRUNS_DIR = Path(__file__).parent / \"runs\"\n\n\n@dataclass\nclass DemoRunState:\n \"\"\"State for a single demo run.\"\"\"\n demo_name: str\n status: str = \"pending\" # pending, running, success, failed, fixing\n job_id: Optional[int] = None\n run_id: Optional[int] = None\n branch: Optional[str] = None\n started_at: Optional[str] = None\n completed_at: Optional[str] = None\n error_summary: Optional[str] = None\n fix_attempts: List[Dict[str, Any]] = field(default_factory=list)\n\n def to_dict(self) -> dict:\n return asdict(self)\n\n @classmethod\n def from_dict(cls, data: dict) -> \"DemoRunState\":\n return cls(**data)\n\n\n@dataclass\nclass RunState:\n \"\"\"Overall state for a release run.\"\"\"\n commit_id: str\n branch: str = \"main\"\n started_at: str = field(default_factory=lambda: datetime.now().isoformat())\n demos: Dict[str, DemoRunState] = field(default_factory=dict)\n\n def to_dict(self) -> dict:\n return {\n \"commit_id\": self.commit_id,\n \"branch\": self.branch,\n \"started_at\": self.started_at,\n \"demos\": {k: v.to_dict() for k, v in self.demos.items()}\n }\n\n @classmethod\n def from_dict(cls, data: dict) -> \"RunState\":\n demos = {k: DemoRunState.from_dict(v) for k, v in data.get(\"demos\", {}).items()}\n return cls(\n commit_id=data[\"commit_id\"],\n branch=data.get(\"branch\", \"main\"),\n started_at=data.get(\"started_at\", \"\"),\n demos=demos\n )\n\n\nclass RunStateManager:\n \"\"\"Manages persistent run state for AI release workflow.\"\"\"\n\n def __init__(self, commit_id: Optional[str] = None):\n \"\"\"Initialize with a specific commit or auto-detect from git.\"\"\"\n if commit_id is None:\n commit_id = self._get_current_commit()\n self.commit_id = commit_id\n self.run_dir = RUNS_DIR / commit_id\n self.run_dir.mkdir(parents=True, exist_ok=True)\n self.state = self._load_or_create_state()\n\n def _get_current_commit(self) -> str:\n \"\"\"Get current git commit from dbdemos-notebooks.\"\"\"\n import subprocess\n try:\n result = subprocess.run(\n [\"git\", \"rev-parse\", \"--short\", \"HEAD\"],\n cwd=Path(__file__).parent.parent.parent / \"dbdemos-notebooks\",\n capture_output=True, text=True\n )\n if result.returncode == 0:\n return result.stdout.strip()\n except Exception:\n pass\n return datetime.now().strftime(\"%Y%m%d_%H%M%S\")\n\n def _load_or_create_state(self) -> RunState:\n \"\"\"Load existing state or create new one.\"\"\"\n state_file = self.run_dir / \"state.json\"\n if state_file.exists():\n with open(state_file) as f:\n return RunState.from_dict(json.load(f))\n return RunState(commit_id=self.commit_id)\n\n def save(self):\n \"\"\"Save current state to disk.\"\"\"\n state_file = self.run_dir / \"state.json\"\n with open(state_file, \"w\") as f:\n json.dump(self.state.to_dict(), f, indent=2)\n\n def get_demo_dir(self, demo_name: str) -> Path:\n \"\"\"Get or create directory for a demo.\"\"\"\n demo_dir = self.run_dir / demo_name\n demo_dir.mkdir(parents=True, exist_ok=True)\n return demo_dir\n\n def get_demo_state(self, demo_name: str) -> DemoRunState:\n \"\"\"Get state for a specific demo.\"\"\"\n if demo_name not in self.state.demos:\n self.state.demos[demo_name] = DemoRunState(demo_name=demo_name)\n return self.state.demos[demo_name]\n\n def update_demo_status(self, demo_name: str, status: str, **kwargs):\n \"\"\"Update demo status and save.\"\"\"\n demo_state = self.get_demo_state(demo_name)\n demo_state.status = status\n for key, value in kwargs.items():\n if hasattr(demo_state, key):\n setattr(demo_state, key, value)\n\n if status == \"running\" and not demo_state.started_at:\n demo_state.started_at = datetime.now().isoformat()\n elif status in (\"success\", \"failed\"):\n demo_state.completed_at = datetime.now().isoformat()\n\n self.save()\n self._save_demo_status(demo_name, demo_state)\n\n def _save_demo_status(self, demo_name: str, state: DemoRunState):\n \"\"\"Save demo-specific status file.\"\"\"\n demo_dir = self.get_demo_dir(demo_name)\n with open(demo_dir / \"status.json\", \"w\") as f:\n json.dump(state.to_dict(), f, indent=2)\n\n def save_errors(self, demo_name: str, errors: List[Dict[str, Any]]):\n \"\"\"Save extracted errors for a demo.\"\"\"\n demo_dir = self.get_demo_dir(demo_name)\n with open(demo_dir / \"errors.json\", \"w\") as f:\n json.dump({\n \"extracted_at\": datetime.now().isoformat(),\n \"errors\": errors\n }, f, indent=2)\n\n def save_job_output(self, demo_name: str, output: str):\n \"\"\"Save raw job output.\"\"\"\n demo_dir = self.get_demo_dir(demo_name)\n with open(demo_dir / \"job_output.log\", \"w\") as f:\n f.write(output)\n\n def add_fix_attempt(self, demo_name: str, description: str, branch: str, files_changed: List[str]):\n \"\"\"Record a fix attempt.\"\"\"\n demo_state = self.get_demo_state(demo_name)\n attempt = {\n \"timestamp\": datetime.now().isoformat(),\n \"description\": description,\n \"branch\": branch,\n \"files_changed\": files_changed,\n \"result\": \"pending\"\n }\n demo_state.fix_attempts.append(attempt)\n self.save()\n\n # Also save to fix_attempts.json\n demo_dir = self.get_demo_dir(demo_name)\n with open(demo_dir / \"fix_attempts.json\", \"w\") as f:\n json.dump(demo_state.fix_attempts, f, indent=2)\n\n def update_fix_result(self, demo_name: str, result: str):\n \"\"\"Update the result of the latest fix attempt.\"\"\"\n demo_state = self.get_demo_state(demo_name)\n if demo_state.fix_attempts:\n demo_state.fix_attempts[-1][\"result\"] = result\n self.save()\n\n def add_note(self, demo_name: str, note: str):\n \"\"\"Add a note to the demo's notes.md file.\"\"\"\n demo_dir = self.get_demo_dir(demo_name)\n notes_file = demo_dir / \"notes.md\"\n timestamp = datetime.now().strftime(\"%Y-%m-%d %H:%M:%S\")\n\n with open(notes_file, \"a\") as f:\n f.write(f\"\\n## {timestamp}\\n\\n{note}\\n\")\n\n def get_summary(self) -> str:\n \"\"\"Get a summary of all demo states.\"\"\"\n lines = [\n f\"# Release Run: {self.commit_id}\",\n f\"Branch: {self.state.branch}\",\n f\"Started: {self.state.started_at}\",\n \"\",\n \"## Demo Status\",\n \"\"\n ]\n\n for name, demo in sorted(self.state.demos.items()):\n status_emoji = {\n \"pending\": \"⏳\",\n \"running\": \"🔄\",\n \"success\": \"✅\",\n \"failed\": \"❌\",\n \"fixing\": \"🔧\"\n }.get(demo.status, \"❓\")\n\n line = f\"- {status_emoji} **{name}**: {demo.status}\"\n if demo.error_summary:\n line += f\" - {demo.error_summary[:50]}...\"\n if demo.fix_attempts:\n line += f\" ({len(demo.fix_attempts)} fix attempts)\"\n lines.append(line)\n\n return \"\\n\".join(lines)\n\n @classmethod\n def list_runs(cls) -> List[str]:\n \"\"\"List all existing run directories.\"\"\"\n if not RUNS_DIR.exists():\n return []\n return sorted([d.name for d in RUNS_DIR.iterdir() if d.is_dir()])\n\n @classmethod\n def get_latest_run(cls) -> Optional[\"RunStateManager\"]:\n \"\"\"Get the most recent run state manager.\"\"\"\n runs = cls.list_runs()\n if not runs:\n return None\n return cls(runs[-1])\n\n\n# Convenience functions\ndef get_run_state(commit_id: Optional[str] = None) -> RunStateManager:\n \"\"\"Get or create a run state manager.\"\"\"\n return RunStateManager(commit_id)\n\n\ndef get_latest_run() -> Optional[RunStateManager]:\n \"\"\"Get the latest run state.\"\"\"\n return RunStateManager.get_latest_run()\n" }, { "path": "ai_release/runs/.gitignore", "content": "*.log\n*/\n!.gitignore\n" }, { "path": "build-and-distribute.sh", "content": "#!/bin/bash\n\n# Check for pending changes before doing anything\nif ! git diff --quiet || ! git diff --cached --quiet; then\n echo \"Error: You have uncommitted changes.\"\n echo \"Please commit and push your changes before running a release.\"\n echo \"\"\n git status --short\n exit 1\nfi\n\nif [ -n \"$(git log origin/main..HEAD 2>/dev/null)\" ]; then\n echo \"Error: You have unpushed commits.\"\n echo \"Please push your changes before running a release.\"\n echo \"\"\n git log origin/main..HEAD --oneline\n exit 1\nfi\n\n# Check if gh CLI is installed\nif ! command -v gh &> /dev/null; then\n echo \"Error: GitHub CLI (gh) is not installed. Please install it first.\"\n echo \"Visit: https://cli.github.com/\"\n exit 1\nfi\n\n# Check if pip-compile is installed (from pip-tools)\nif ! command -v pip-compile &> /dev/null; then\n echo \"Error: pip-compile is not installed. Please install pip-tools first.\"\n echo \"Run: pip install pip-tools\"\n exit 1\nfi\n\n# Check authentication status\nif ! gh auth status &> /dev/null; then\n echo \"GitHub CLI not authenticated. Please login...\"\n gh auth login\nfi\n\n# Check if active account is Enterprise Managed User (ends with _data)\nACTIVE_ACCOUNT=$(gh auth status | grep \"Logged in to\" | head -1 | sed 's/.*Logged in to github.com account \\([^ ]*\\).*/\\1/')\nif [[ \"$ACTIVE_ACCOUNT\" == *\"_data\" ]]; then\n echo \"Warning: Active account '$ACTIVE_ACCOUNT' appears to be an Enterprise Managed User\"\n echo \"Switching to regular account...\"\n \n # Get list of available accounts by parsing auth status output\n AVAILABLE_ACCOUNTS=$(gh auth status | grep \"Logged in to\" | sed 's/.*Logged in to github.com account \\([^ ]*\\).*/\\1/')\n \n # Find first account that doesn't end with _data\n REGULAR_ACCOUNT=\"\"\n while IFS= read -r account; do\n if [[ \"$account\" != *\"_data\" ]]; then\n REGULAR_ACCOUNT=\"$account\"\n break\n fi\n done <<< \"$AVAILABLE_ACCOUNTS\"\n \n if [[ -n \"$REGULAR_ACCOUNT\" ]]; then\n echo \"Switching to regular account: $REGULAR_ACCOUNT\"\n gh auth switch --user \"$REGULAR_ACCOUNT\" || {\n echo \"Error: Failed to switch to regular account\"\n exit 1\n }\n else\n echo \"Error: No regular account found. Please add a regular GitHub account:\"\n echo \"gh auth login\"\n exit 1\n fi\nfi\n\n# Check access to required repositories\necho \"Checking access to required repositories...\"\nREPOS=(\"databricks-demos/dbdemos\" \"databricks-demos/dbdemos-notebooks\" \"databricks-demos/dbdemos-dataset\" \"databricks-demos/dbdemos-resources\")\n\nfor repo in \"${REPOS[@]}\"; do\n if ! gh api \"repos/$repo\" &> /dev/null; then\n echo \"Error: No access to repository $repo\"\n echo \"Please ensure you have the necessary permissions or try logging in again:\"\n echo \"gh auth login\"\n exit 1\n fi\n echo \"✓ Access confirmed for $repo\"\ndone\n\n# Switch to main and pull latest\ngit checkout main || exit 1\ngit pull || exit 1\n\n# Get current version from setup.py\nCURRENT_VERSION=$(grep \"version=\" setup.py | sed \"s/.*version='\\([^']*\\)'.*/\\1/\")\necho \"Current version: $CURRENT_VERSION\"\n\n# Bump version (patch increment)\nIFS='.' read -ra VERSION_PARTS <<< \"$CURRENT_VERSION\"\nNEW_PATCH=$((VERSION_PARTS[2] + 1))\nNEW_VERSION=\"${VERSION_PARTS[0]}.${VERSION_PARTS[1]}.$NEW_PATCH\"\necho \"New version: $NEW_VERSION\"\n\n# Update version in setup.py\nsed -i.bak \"s/version='[^']*'/version='$NEW_VERSION'/\" setup.py\nrm setup.py.bak\n\n# Update version in __init__.py\nsed -i.bak \"s/__version__ = \\\"[^\\\"]*\\\"/__version__ = \\\"$NEW_VERSION\\\"/\" dbdemos/__init__.py\nrm dbdemos/__init__.py.bak\n\n# Generate requirements.txt with hashes from trusted private index\necho \"Generating requirements.txt with hashes...\"\n\n# Extract dependencies from setup.py and write to requirements.in\npython3 -c \"\nimport ast\nimport sys\n\nwith open('setup.py', 'r') as f:\n content = f.read()\n\n# Parse the setup.py file\ntree = ast.parse(content)\n\n# Find the setup() call and extract install_requires\nfor node in ast.walk(tree):\n if isinstance(node, ast.Call) and getattr(node.func, 'id', None) == 'setup':\n for keyword in node.keywords:\n if keyword.arg == 'install_requires':\n # Extract the list of dependencies\n deps = ast.literal_eval(compile(ast.Expression(keyword.value), '<string>', 'eval'))\n for dep in deps:\n print(dep)\n sys.exit(0)\n\nprint('Error: Could not extract install_requires from setup.py', file=sys.stderr)\nsys.exit(1)\n\" > requirements.in\n\nif [ $? -ne 0 ]; then\n echo \"Error: Failed to extract dependencies from setup.py\"\n exit 1\nfi\n\necho \"Extracted dependencies:\"\ncat requirements.in\n\n# Run pip-compile with private index to get trusted hashes\nPRIVATE_INDEX=\"https://pypi-proxy.dev.databricks.com/simple/\"\npip-compile --generate-hashes --index-url=\"$PRIVATE_INDEX\" --output-file=requirements.txt requirements.in\n\nif [ $? -ne 0 ]; then\n echo \"Error: pip-compile failed\"\n exit 1\nfi\n\n# Remove the private index URL from requirements.txt (keep hashes, they're content-based)\nsed -i.bak '/^--index-url/d' requirements.txt\n# Also clean up the comment that references the private index\nsed -i.bak \"s|--index-url=$PRIVATE_INDEX ||g\" requirements.txt\nrm requirements.txt.bak\n\necho \"requirements.txt generated with hashes (private index removed)\"\n\n# Use the version we just bumped\nVERSION=$NEW_VERSION\necho \"Using bumped version: $VERSION\"\n\n#package\nrm -rf ./dist/*\nrm -rf ./dbdemos/bundles/.DS_Store\npython3 setup.py clean --all bdist_wheel\n\necho \"Package built under dist/ - updating pypi with new version...\"\nls -alh ./dist\nif ! twine upload dist/*; then\n echo \"Error: Failed to upload package to PyPI\"\n exit 1\nfi\necho \"Upload ok - available as pip install dbdemos\"\n\n# Create or switch to release branch and commit the bumped version\necho \"Creating/updating release branch with bumped version...\"\ngit checkout -b release/v$VERSION 2>/dev/null || git checkout release/v$VERSION\ngit add setup.py dbdemos/__init__.py requirements.in requirements.txt\ngit commit -m \"Bump version to $VERSION\"\ngit push origin release/v$VERSION\n\n# Create PR to main branch\necho \"Creating pull request to main branch...\"\nif gh pr create --title \"Release v$VERSION\" --body \"Automated release for version $VERSION\" --base main --head release/v$VERSION; then\n echo \"Pull request created successfully\"\nelse\n echo \"Warning: Failed to create pull request (may already exist)\"\nfi\n\n# Also update main with the version bump so it doesn't get lost\necho \"Syncing version bump to main...\"\ngit checkout main\ngit add setup.py dbdemos/__init__.py requirements.in requirements.txt\ngit commit -m \"Bump version to $VERSION\"\ngit push origin main\n\n# Find the wheel file\nWHL_FILE=$(find ./dist -name \"*.whl\" | head -n 1)\nif [ -z \"$WHL_FILE\" ]; then\n echo \"Error: No wheel file found in ./dist directory\"\n exit 1\nfi\n\necho \"Found wheel file: $WHL_FILE\"\n\n# Extract version from wheel filename (format: dbdemos-0.6.12-py3-none-any.whl)\nVERSION=$(basename \"$WHL_FILE\" | sed -E 's/dbdemos-([0-9]+\\.[0-9]+\\.[0-9]+).*/\\1/')\necho \"Extracted version from wheel file: $VERSION\"\n\n# Function to create a release and upload asset using gh CLI\ncreate_release_with_asset() {\n local repo=$1\n local tag_name=\"v$VERSION\"\n local release_name=\"v$VERSION\"\n \n echo \"Creating release $release_name for $repo...\"\n \n # Create the release using gh CLI\n if gh release create \"$tag_name\" \"$WHL_FILE\" --repo \"$repo\" --title \"$release_name\" --notes \"Release version $VERSION\"; then\n echo \"Release created and asset uploaded successfully for $repo\"\n return 0\n else\n echo \"Error creating release for $repo\"\n return 1\n fi\n}\n\n# Create releases with assets on all repositories\necho \"Creating releases for version v$VERSION...\"\ncreate_release_with_asset \"databricks-demos/dbdemos\"\ncreate_release_with_asset \"databricks-demos/dbdemos-notebooks\"\ncreate_release_with_asset \"databricks-demos/dbdemos-dataset\"\ncreate_release_with_asset \"databricks-demos/dbdemos-resources\"\n\necho \"Release process completed for v$VERSION!\"\n" }, { "path": "build.sh", "content": "python3 setup.py clean --all bdist_wheel\nconda activate test_dbdemos\n#pip3 install dist/dbdemos-0.3.0-py3-none-any.whl --force\n#python3 test_package.py\n\n#cp dist/dbdemos-* release/\n" }, { "path": "dbdemos/__init__.py", "content": "__version__ = \"0.6.34\"\n\nfrom .dbdemos import list_demos, install, create_cluster, help, install_all, check_status_all, check_status, get_html_list_demos\n\n" }, { "path": "dbdemos/conf.py", "content": "import json\nfrom pathlib import Path\nfrom typing import List\nimport requests\nimport urllib\nfrom datetime import date\nimport re\nimport threading\n\nfrom requests import Response\n\n\ndef merge_dict(a, b, path=None, override = True):\n \"\"\"merges dict b into a. Mutate a\"\"\"\n if path is None: path = []\n for key in b:\n if key in a:\n if isinstance(a[key], dict) and isinstance(b[key], dict):\n merge_dict(a[key], b[key], path + [str(key)])\n elif override:\n a[key] = b[key]\n else:\n a[key] = b[key]\n\nclass Conf():\n def __init__(self, username: str, workspace_url: str, org_id: str, pat_token: str, default_cluster_template: str = None, default_cluster_job_template = None,\n repo_staging_path: str = None, repo_name: str = None, repo_url: str = None, branch: str = \"master\", github_token = None, run_test_as_username=\"quentin.ambard@databricks.com\"):\n self.username = username\n name = self.username[:self.username.rfind('@')]\n self.name = re.sub(\"[^A-Za-z0-9]\", '_', name)\n self.workspace_url = workspace_url\n self.org_id = org_id\n self.pat_token = pat_token\n self.headers = {\"Authorization\": \"Bearer \" + pat_token, 'Content-type': 'application/json', 'User-Agent': 'dbdemos'}\n self.default_cluster_template = default_cluster_template\n self.default_cluster_job_template = default_cluster_job_template\n self.repo_staging_path = repo_staging_path\n self.repo_name = repo_name\n assert repo_url is None or \".git\" not in repo_url, \"repo_url should not contain .git\"\n self.repo_url = repo_url\n self.branch = branch\n self.github_token = github_token\n self.run_test_as_username = run_test_as_username\n\n def get_repo_path(self):\n return self.repo_staging_path+\"/\"+self.repo_name\n\n #Add internal pool id to accelerate our demos & unit tests\n def get_demo_pool(self):\n if self.org_id == \"1444828305810485\" or \"e2-demo-field-eng\" in self.workspace_url:\n return \"0727-104344-hauls13-pool-uftxk0r6\"\n if self.org_id == \"1660015457675682\" or self.is_dev_env():\n return \"1025-140806-yup112-pool-yz565bma\"\n if self.org_id == \"5206439413157315\":\n return \"1010-172835-slues66-pool-7dhzc23j\"\n if self.org_id == \"984752964297111\":\n return \"1010-173019-honor44-pool-ksw4stjz\"\n if self.org_id == \"2556758628403379\":\n return \"1010-173021-dance560-pool-hl7wefwy\"\n return None\n\n def is_dev_env(self):\n return \"e2-demo-tools\" in self.workspace_url or \"local\" in self.workspace_url\n\n def is_demo_env(self):\n return \"e2-demo-field-eng\" in self.workspace_url or \"eastus2\" or self.org_id in [\"1444828305810485\"]\n\n def is_fe_env(self):\n return \"e2-demo-field-eng\" in self.workspace_url or \"eastus2\" in self.workspace_url or \\\n self.org_id in [\"5206439413157315\", \"984752964297111\", \"local\", \"1444828305810485\", \"2556758628403379\"]\n\nclass DBClient():\n def __init__(self, conf: Conf):\n self.conf = conf\n\n def clean_path(self, path):\n if path.startswith(\"http\"):\n raise Exception(f\"Wrong path {path}, use with api path directly (no http://xxx..xxx).\")\n if path.startswith(\"/\"):\n path = path[1:]\n if path.startswith(\"api/\"):\n path = path[len(\"api/\"):]\n return path\n\n def post(self, path: str, json: dict = {}, retry = 0):\n url = self.conf.workspace_url+\"/api/\"+self.clean_path(path)\n with requests.post(url, headers = self.conf.headers, json=json, timeout=60) as r:\n if r.status_code == 429 and retry < 2:\n import time\n import random\n wait_time = 15 * (retry+1) + random.randint(2*retry, 10*retry)\n print(f'WARN: hitting api request limit 429 error: {path}. Sleeping {wait_time}sec and retrying...')\n time.sleep(wait_time)\n print('Retrying call.')\n return self.post(path, json, retry+1)\n else:\n return self.get_json_result(url, r)\n\n def put(self, path: str, json: dict = None, data: bytes = None):\n url = self.conf.workspace_url+\"/api/\"+self.clean_path(path)\n headers = self.conf.headers\n if data is not None:\n files = {'file': ('file', data, 'application/octet-stream')}\n with requests.put(url, headers=headers, files=files, timeout=60) as r:\n return self.get_json_result(url, r)\n else:\n with requests.put(url, headers=headers, json=json, timeout=60) as r:\n return self.get_json_result(url, r)\n\n def patch(self, path: str, json: dict = {}):\n url = self.conf.workspace_url+\"/api/\"+self.clean_path(path)\n with requests.patch(url, headers = self.conf.headers, json=json, timeout=60) as r:\n return self.get_json_result(url, r)\n\n def get(self, path: str, params: dict = {}, print_auth_error = True):\n url = self.conf.workspace_url+\"/api/\"+self.clean_path(path)\n with requests.get(url, headers = self.conf.headers, params=params, timeout=60) as r:\n return self.get_json_result(url, r, print_auth_error)\n\n def delete(self, path: str, params: dict = {}):\n url = self.conf.workspace_url+\"/api/\"+self.clean_path(path)\n with requests.delete(url, headers = self.conf.headers, params=params, timeout=60) as r:\n return self.get_json_result(url, r)\n\n def get_json_result(self, url: str, r: Response, print_auth_error = True):\n if r.status_code == 403:\n if print_auth_error:\n print(f\"Unauthorized call. Check your PAT token {r.text} - {r.url} - {url}\")\n try:\n return r.json()\n except Exception as e:\n print(f\"API CALL ERROR - can't read json. status: {r.status_code} {r.text} - URL: {url} - {e}\")\n raise e\n\n def search_cluster(self, cluster_name: str, tags: dict):\n clusters = self.db.get(\"2.1/clusters/list\")\n for c in clusters:\n if c['cluster_name'] == cluster_name:\n match = True\n #Check if all the tags are in the cluster conf\n for k, v in tags.items():\n if k not in c['custom_tags'] or c['custom_tags'][k] != v:\n match = False\n if match:\n return c\n return None\n\n def find_job(self, name, offset = 0, limit = 25):\n r = self.get(\"2.1/jobs/list\", {\"limit\": limit, \"offset\": offset, \"name\": urllib.parse.quote_plus(name)})\n if 'jobs' in r:\n for job in r['jobs']:\n if job[\"settings\"][\"name\"] == name:\n return job\n if r['has_more']:\n return self.find_job(name, offset+limit, limit)\n return None\n\nclass GenieRoom():\n def __init__(self, id: str, display_name: str, description: str, table_identifiers: List[str], curated_questions: List[str], instructions: str, sql_instructions: List[dict], function_names: List[str], benchmarks:List[dict]):\n self.display_name = display_name\n self.id = id\n self.description = description\n self.instructions = instructions\n self.table_identifiers = table_identifiers\n self.sql_instructions = sql_instructions\n self.curated_questions = curated_questions\n self.function_names = function_names\n self.benchmarks= benchmarks\n \nclass DataFolder():\n def __init__(self, source_folder: str, source_format: str, target_table_name: str = None, target_volume_folder_name: str = None, target_format: str = \"delta\"):\n assert target_volume_folder_name or target_table_name, \"Error, data folder should either has target_table_name or target_volume_folder_name set\"\n self.source_folder = source_folder\n self.source_format = source_format\n self.target_table_name = target_table_name\n self.target_format = target_format\n self.target_volume_folder_name = target_volume_folder_name\n\nclass DemoNotebook():\n def __init__(self, path: str, title: str, description: str, pre_run: bool = False, publish_on_website: bool = False,\n add_cluster_setup_cell: bool = False, parameters: dict = {}, depends_on_previous: bool = True, libraries: list = [], warehouse_id = None, object_type = None):\n self.path = path\n self.title = title\n self.description = description\n self.pre_run = pre_run\n self.publish_on_website = publish_on_website\n self.add_cluster_setup_cell = add_cluster_setup_cell\n self.parameters = parameters\n self.depends_on_previous = depends_on_previous\n self.libraries = libraries\n self.warehouse_id = warehouse_id\n self.object_type = object_type\n\n def __repr__(self):\n return self.path\n\n def get_folder(self):\n p = Path(self.get_clean_path())\n p.parts\n\n def get_clean_path(self):\n #Some notebook path are relatives, like ../../demo-retail/lakehouse-retail/_resources/xxx\n # DThis function removes it and returns _resources/xxx\n p = Path(self.path)\n parent_count = p.parts.count('..')\n if parent_count > 0:\n return str(p.relative_to(*p.parts[:parent_count*2-1]))\n return self.path\n\n\n def toJSON(self):\n return json.dumps(self, default=lambda o: o.__dict__)\n\nclass DemoConf():\n def __init__(self, path: str, json_conf: dict, catalog:str = None, schema: str = None):\n self.json_conf = json_conf\n self.notebooks = []\n self.cluster = json_conf.get('cluster', {})\n self.cluster_libraries = json_conf.get('cluster_libraries', [])\n self.workflows = json_conf.get('workflows', [])\n self.pipelines = json_conf.get('pipelines', [])\n self.repos = json_conf.get('repos', [])\n self.serverless_supported = json_conf.get('serverless_supported', False)\n self.init_job = json_conf.get('init_job', {})\n self.job_id = None\n self.run_id = None\n if path.startswith('/'):\n path = path[1:]\n self.path = path\n self.name = json_conf['name']\n self.category = json_conf['category']\n self.title = json_conf['title']\n self.description = json_conf['description']\n self.tags = json_conf.get('tags', [])\n self.custom_schema_supported = json_conf.get('custom_schema_supported', False)\n self.schema = schema\n self.catalog = catalog\n self.default_schema = json_conf.get('default_schema', \"\")\n self.default_catalog = json_conf.get('default_catalog', \"\")\n self.custom_message = json_conf.get('custom_message', \"\")\n self.create_cluster = json_conf.get('create_cluster', True)\n self.dashboards = json_conf.get('dashboards', [])\n self.sql_queries = json_conf.get('sql_queries', [])\n self.bundle = json_conf.get('bundle', False)\n self.env_version = json_conf.get('env_version', 2)\n \n self.data_folders: List[DataFolder] = []\n for data_folder in json_conf.get('data_folders', []):\n self.data_folders.append(DataFolder(data_folder['source_folder'], data_folder['source_format'], data_folder.get('target_table_name', None), \n data_folder.get('target_volume_folder', None), data_folder['target_format']))\n\n self.genie_rooms: List[GenieRoom] = []\n for genie_room in json_conf.get('genie_rooms', []):\n self.genie_rooms.append(GenieRoom(genie_room['id'], genie_room.get('display_name', None), genie_room.get('description', None),\n genie_room['table_identifiers'], genie_room.get('curated_questions', []),\n genie_room.get('instructions', None), genie_room.get('sql_instructions', []),\n genie_room.get('function_names', []),genie_room.get('benchmarks', [])))\n\n for n in json_conf.get('notebooks', []):\n add_cluster_setup_cell = n.get('add_cluster_setup_cell', False)\n params = n.get('parameters', {})\n depends_on_previous = n.get('depends_on_previous', True)\n libraries = n.get('libraries', [])\n warehouse_id = n.get('warehouse_id', None)\n self.notebooks.append(DemoNotebook(n['path'], n['title'], n['description'], n['pre_run'], n['publish_on_website'],\n add_cluster_setup_cell, params, depends_on_previous, libraries, warehouse_id, n.get('object_type', None)))\n\n self._notebook_lock = threading.Lock()\n\n def __repr__(self):\n return self.path + \"(\"+str(self.notebooks)+\")\"\n\n def update_notebook_object_type(self, notebook: DemoNotebook, object_type: str):\n with self._notebook_lock:\n for n in self.json_conf['notebooks']:\n if n['path'] == notebook.path:\n n['object_type'] = object_type\n break\n\n def add_notebook(self, notebook):\n self.notebooks.append(notebook)\n #TODO: this isn't clean, need a better solution\n self.json_conf[\"notebooks\"].append(notebook.__dict__)\n\n def set_pipeline_id(self, id, uid):\n j = json.dumps(self.init_job)\n j = j.replace(\"{{DYNAMIC_SDP_ID_\"+id+\"}}\", uid)\n self.init_job = json.loads(j)\n j = json.dumps(self.workflows)\n j = j.replace(\"{{DYNAMIC_SDP_ID_\"+id+\"}}\", uid)\n self.workflows = json.loads(j)\n\n def get_job_name(self):\n return \"field-bundle_\"+self.name\n\n def get_notebooks_to_run(self):\n return [n for n in self.notebooks if n.pre_run]\n\n def get_notebooks_to_publish(self) -> List[DemoNotebook]:\n return [n for n in self.notebooks if n.publish_on_website]\n\n def get_bundle_path(self):\n return self.get_bundle_root_path() + \"/install_package\"\n\n def get_bundle_dashboard_path(self):\n return self.get_bundle_root_path() + \"/dashboards\"\n\n def get_bundle_root_path(self):\n return \"dbdemos/bundles/\"+self.name\n\n def get_minisite_path(self):\n return \"dbdemos/minisite/\"+self.name\n\n\nclass ConfTemplate:\n def __init__(self, username, demo_name, catalog = None, schema = None, demo_folder = \"\"):\n self.catalog = catalog\n self.schema = schema\n self.username = username\n self.demo_name = demo_name\n self.demo_folder = demo_folder\n\n def template_TODAY(self):\n return date.today().strftime(\"%Y-%m-%d\")\n\n def template_CURRENT_USER(self):\n return self.username\n\n def template_CATALOG(self):\n return self.catalog\n\n def template_SCHEMA(self):\n return self.schema\n\n def template_CURRENT_USER_NAME(self):\n name = self.username[:self.username.rfind('@')]\n name = re.sub(\"[^A-Za-z0-9]\", '_', name)\n return name\n\n def template_DEMO_NAME(self):\n return self.demo_name\n\n def template_DEMO_FOLDER(self):\n return self.demo_folder\n\n def template_SHARED_WAREHOUSE_ID(self):\n return self.demo_folder\n\n def replace_template_key(self, text: str):\n for key in set(re.findall(r'\\{\\{(.*?)\\}\\}', text)):\n if \"Drift_detection\" not in key: #TODO need to improve that, mlops demo has {{}} in the product like tasks.Drift_detection.values.all_violations_count\n if not key.startswith(\"DYNAMIC\") and not key.startswith(\"SHARED_WAREHOUSE\"):\n func = getattr(self, f\"template_{key}\")\n replacement = func()\n text = text.replace(\"{{\"+key+\"}}\", replacement)\n return text" }, { "path": "dbdemos/dbdemos.py", "content": "from .exceptions.dbdemos_exception import TokenException\nfrom .installer import Installer\nfrom collections import defaultdict\n\nfrom .installer_report import InstallerReport\n\nCSS_LIST = \"\"\"\n<style>\n.dbdemo {\n font-family: -apple-system,BlinkMacSystemFont,Segoe UI,Roboto,Helvetica Neue,Arial,Noto Sans,sans-serif,Apple Color Emoji,Segoe UI Emoji,Segoe UI Symbol,Noto Color Emoji,FontAwesome;\n color: #3b3b3b;\n padding: 0px 0px 20px 0px;\n}\n.dbdemo_box {\n width: 400px;\n padding: 10px;\n box-shadow: 0 .15rem 1.15rem 0 rgba(58,59,69,.15)!important;\n float: left;\n min-height: 170px;\n margin: 0px 20px 20px 20px;\n}\n.dbdemo_category {\n clear: both;\n}\n.category {\n margin-left: 20px;\n margin-bottom: 5px;\n\n}\n.dbdemo_logo {\n width: 100%;\n height: 225px;\n}\n.code {\n padding: 5px;\n border: 1px solid #e4e4e4;\n font-family: monospace;\n background-color: #f5f5f5;\n margin: 5px 0px 0px 0px;\n}\n.dbdemo_description {\n height: 100px;\n}\n.menu_button {\n font-size: 15px;\n cursor: pointer;\n border: 0px;\n padding: 10px 20px 10px 20px;\n margin-right: 10px;\n background-color: rgb(238, 237, 233);\n border-radius: 20px;\n}\n.menu_button:hover {\n background-color: rgb(245, 244, 242)\n}\n.menu_button.selected {\n background-color: rgb(158, 214, 196)\n}\n.new_tag {\n background-color: red;\n color: white;\n font-size: 13px;\n padding: 2px 7px;\n border-radius: 3px;\n margin-right: 5px;\n}\n</style>\n\"\"\"\n\nJS_LIST = \"\"\"<script>\n const buttons = document.querySelectorAll('.menu_button');\n const sections = document.querySelectorAll('.dbdemo_category');\n\n buttons.forEach(button => {\n button.addEventListener('click', () => {\n const selectedCategory = button.getAttribute('category');\n\n sections.forEach(section => {\n if (section.id === `category-${selectedCategory}`) {\n section.style.display = 'block';\n } else {\n section.style.display = 'none';\n }\n });\n\n buttons.forEach(btn => {\n if (btn === button) {\n btn.classList.add('selected');\n } else {\n btn.classList.remove('selected');\n }\n });\n });\n });\n</script>\"\"\"\ndef help():\n installer = Installer()\n if installer.report.displayHTML_available():\n from dbruntime.display import displayHTML\n displayHTML(\"\"\"<style>\n .dbdemos_install{\n font-family: -apple-system,BlinkMacSystemFont,Segoe UI,Roboto,Helvetica Neue,Arial,Noto Sans,sans-serif,Apple Color Emoji,Segoe UI Emoji,Segoe UI Symbol,Noto Color Emoji,FontAwesome;\n color: #3b3b3b;\n box-shadow: 0 .15rem 1.15rem 0 rgba(58,59,69,.15)!important;\n padding: 10px;\n margin: 10px;\n }\n .code {\n padding: 0px 5px;\n border: 1px solid #e4e4e4;\n font-family: monospace;\n background-color: #f5f5f5;\n margin: 5px 0px 0px 0px;\n display: inline;\n }\n </style>\n <div class=\"dbdemos_install\">\n <h1>DBDemos</h1>\n <i>Install databricks demos: notebooks, Delta Live Table Pipeline, DBSQL Dashboards, ML Models etc.</i>\n <ul>\n <li>\n <div class=\"code\">dbdemos.help()</div>: display help.<br/><br/>\n </li>\n <li>\n <div class=\"code\">dbdemos.list_demos(category: str = None)</div>: list all demos available, can filter per category (ex: 'governance').<br/><br/>\n </li>\n <li>\n <div class=\"code\">dbdemos.install(demo_name: str, path: str = \"./\", overwrite: bool = False, use_current_cluster = False, username: str = None, pat_token: str = None, workspace_url: str = None, skip_dashboards: bool = False, cloud: str = \"AWS\", catalog: str = None, schema: str = None, serverless: bool = None, warehouse_name: str = None, skip_genie_rooms: bool = False, dlt_policy_id: str = None, dlt_compute_settings: dict = None)</div>: install the given demo to the given path.<br/><br/>\n <ul>\n <li>If overwrite is True, dbdemos will delete the given path folder and re-install the notebooks.</li>\n <li>use_current_cluster = True will not start a new cluster to init the demo but use the current cluster instead. <strong>Set it to True it if you don't have cluster creation permission</strong>.</li>\n <li>skip_dashboards = True will not load the DBSQL dashboard if any (faster, use it if the dashboard generation creates some issue).</li> \n <li>If no authentication are provided, dbdemos will use the current user credential & workspace + cloud to install the demo.</li>\n <li>catalog and schema options let you chose where to load the data and other assets.</li>\n <li>Dashboards require a warehouse, you can specify it with the warehouse_name='xx' option.</li>\n <li>Dbdemos will detect serverless compute and use the current cluster when you're running serverless. You can force it with the serverless=True option.</li>\n <li>Genie rooms are in beta. You can skip the genie room installation with skip_genie_rooms = True.</li>\n <li>dlt_policy_id will be used in the dlt (example: \"0003963E5B551CE4\"). Use it with dlt_compute_settings = {\"autoscale\": {\"min_workers\": 1, \"max_workers\": 5}} to respect the policy requirements.</li>\n </ul><br/>\n </li>\n <li>\n <div class=\"code\">dbdemos.create_cluster(demo_name: str)</div>: install update the interactive cluster for the demo (scoped to the user).<br/><br/>\n </li>\n <li>\n <div class=\"code\">dbdemos.install_all(path: str = \"./\", overwrite: bool = False, username: str = None, pat_token: str = None, workspace_url: str = None, skip_dashboards: bool = False, cloud: str = \"AWS\")</div>: install all the demos to the given path.<br/><br/>\n </li>\n </ul>\n </div>\"\"\")\n else:\n print(\"------------ DBDemos ------------------\")\n print(\"\"\"dbdemos.help(): display help.\"\"\")\n print(\"\"\"dbdemos.list_demos(category: str = None): list all demos available, can filter per category (ex: 'governance').\"\"\")\n print(\"\"\"dbdemos.install(demo_name: str, path: str = \"./\", overwrite: bool = False, username: str = None, pat_token: str = None, workspace_url: str = None, skip_dashboards: bool = False, cloud: str = \"AWS\"): install the given demo to the given path.\"\"\")\n print(\"\"\"dbdemos.create_cluster(demo_name: str): install update the interactive cluster for the demo (scoped to the user).\"\"\")\n print(\"\"\"dbdemos.install_all(path: str = \"./\", overwrite: bool = False, username: str = None, pat_token: str = None, workspace_url: str = None, skip_dashboards: bool = False, cloud: str = \"AWS\")</div>: install all the demos to the given path.\"\"\")\n\ndef list_demos(category = None, installer = None, pat_token = None):\n check_version()\n deprecated_demos = [\"uc-04-audit-log\", \"llm-dolly-chatbot\"]\n if installer == None:\n installer = Installer(pat_token=pat_token)\n installer.tracker.track_list()\n demos = defaultdict(lambda: [])\n #Define category order\n demos[\"lakehouse\"] = []\n demos[\"data-engineering\"] = []\n demos[\"governance\"] = []\n demos[\"DBSQL\"] = []\n demos[\"data-science\"] = []\n demos[\"AI-BI\"] = []\n for demo in installer.get_demos_available():\n conf = installer.get_demo_conf(demo)\n if (category is None or conf.category == category.lower()) and conf.name not in deprecated_demos:\n demos[conf.category].append(conf)\n if installer.report.displayHTML_available():\n content = get_html_list_demos(demos)\n from dbruntime.display import displayHTML\n displayHTML(content)\n else:\n list_console(demos)\n\ndef get_html_list_demos(demos):\n categories = list(demos.keys())\n content = f\"\"\"{CSS_LIST}<div class=\"dbdemo\">\n <div style=\"padding: 10px 0px 20px 20px\">\"\"\"\n for i, cat in enumerate(categories):\n content += f\"\"\"<button category=\"{cat}\" class=\"menu_button {\"selected\" if i == 0 else \"\"}\" type=\"button\">{f'<span class=\"new_tag\">NEW!</span>' if cat == 'AI-BI' else ''}<span>{cat.capitalize()}</span></button>\"\"\"\n content += \"\"\"</div>\"\"\"\n for i, cat in enumerate(categories):\n content += f\"\"\"<div class=\"dbdemo_category\" style=\"min-height: 200px; display: {\"block\" if i == 0 else \"none\"}\" id=\"category-{cat}\">\"\"\"\n ds = list(demos[cat])\n ds.sort(key=lambda d: d.name)\n for demo in ds:\n content += f\"\"\"\n <div class=\"dbdemo_box\">\n <img class=\"dbdemo_logo\" src=\"https://github.com/databricks-demos/dbdemos-resources/raw/main/icon/{demo.name}.jpg\" />\n <div class=\"dbdemo_description\">\n <h2>{demo.title}</h2>\n {demo.description}\n </div>\n <div class=\"code\"> \n dbdemos.install('{demo.name}')\n </div>\n </div>\"\"\"\n content += \"\"\"</div>\"\"\"\n content += f\"\"\"</div>{JS_LIST}\"\"\"\n return content\n\n\ndef list_console(demos):\n print(\"----------------------------------------------------\")\n print(\"----------------- Demos Available ------------------\")\n print(\"----------------------------------------------------\")\n categories = list(demos.keys())\n for cat in categories:\n print(f\"{cat.capitalize()}\")\n ds = list(demos[cat])\n ds.sort(key=lambda d: d.name)\n for demo in ds:\n print(f\" - {demo.name}: {demo.title} ({demo.description}) => dbdemos.install('{demo.name}')\")\n print(\"\")\n print(\"----------------------------------------------------\")\n\ndef list_delta_live_tables(category = None):\n pass\n\ndef list_dashboards(category = None):\n pass\n\ndef install(demo_name, path = None, overwrite = False, username = None, pat_token = None, workspace_url = None, skip_dashboards = False, cloud = \"AWS\", start_cluster: bool = None,\n use_current_cluster: bool = False, current_cluster_id = None, warehouse_name = None, debug = False, catalog = None, schema = None, serverless=None, skip_genie_rooms=False, \n create_schema=True, dlt_policy_id = None, dlt_compute_settings = None):\n check_version()\n \n if demo_name == \"llm-fine-tuning\" :\n print(\"ERROR: llm-fine-tuning is deprecated and has been removed. You can restore it from an older dbdemos version: %pip install dbdemos==0.6.28\")\n return\n elif demo_name == \"chatbot-rag-llm\" or demo_name == \"llm-tools-functions\" or demo_name == \"llm-rag-chatbot\":\n print(f\"ERROR: {demo_name} is deprecated and has been removed. You can restore it from an older dbdemos version: %pip install dbdemos==0.6.28\")\n print(\"We will instead install the new ai-agent demos\")\n demo_name = \"ai-agent\"\n elif demo_name == \"dlt-loans\" or demo_name == \"dlt-loan\" :\n print(\"ERROR: dlt-loans is deprecated and has been removed. You can restore it from an older dbdemos version: %pip install dbdemos==0.6.28\")\n print(\"We will instead install the new pipeline-bike demos\")\n demo_name = \"pipeline-bike\"\n elif demo_name == \"dlt-unit-test\":\n print(\"WARN: dlt-unit-test has been renamed to declarative-pipeline-unit-test\")\n demo_name = \"declarative-pipeline-unit-test\"\n elif demo_name == \"dlt-cdc\":\n print(\"WARN: dlt-cdc has been renamed to declarative-pipeline-cdc\")\n demo_name = \"declarative-pipeline-cdc\"\n elif demo_name == \"lakehouse-retail-churn\":\n print(\"WARN: lakehouse-retail-churn has been renamed to lakehouse-retail-c360\")\n demo_name = \"lakehouse-retail-c360\"\n elif demo_name == \"identity-pk-fk\":\n print(\"WARN: identity-pk-fk has been renamed to sql-warehouse\")\n demo_name = \"sql-warehouse\"\n elif demo_name == \"auto-loader\":\n print(\"WARN: auto-loader has been renamed to data-ingestion\")\n demo_name = \"data-ingestion\"\n\n \n \n try:\n installer = Installer(username, pat_token, workspace_url, cloud, current_cluster_id = current_cluster_id)\n except TokenException as e:\n report = InstallerReport(workspace_url)\n report.display_token_error(e, demo_name)\n\n if not installer.test_premium_pricing():\n #Force dashboard skip as dbsql isn't available to avoid any error.\n skip_dashboards = True\n installer.install_demo(demo_name, path, overwrite, skip_dashboards = skip_dashboards, start_cluster = start_cluster, use_current_cluster = use_current_cluster,\n debug = debug, catalog = catalog, schema = schema, serverless = serverless, warehouse_name=warehouse_name, skip_genie_rooms=skip_genie_rooms, create_schema=create_schema, dlt_policy_id = dlt_policy_id, dlt_compute_settings = dlt_compute_settings)\n\n\ndef install_all(path = None, overwrite = False, username = None, pat_token = None, workspace_url = None, skip_dashboards = False, cloud = \"AWS\", start_cluster = None, use_current_cluster = False, catalog = None, schema = None, dlt_policy_id = None, dlt_compute_settings = None):\n \"\"\"\n Install all the bundle demos.\n \"\"\"\n installer = Installer(username, pat_token, workspace_url, cloud)\n for demo_name in installer.get_demos_available():\n installer.install_demo(demo_name, path, overwrite, skip_dashboards = skip_dashboards, start_cluster = start_cluster, use_current_cluster = use_current_cluster, catalog = catalog, schema = schema, dlt_policy_id = dlt_policy_id, dlt_compute_settings = dlt_compute_settings)\n\ndef check_status_all(username = None, pat_token = None, workspace_url = None, cloud = \"AWS\"):\n \"\"\"\n Check all dbdemos bundle demos installation status (see #check_status)\n \"\"\"\n installer = Installer(username, pat_token, workspace_url, cloud)\n for demo_name in installer.get_demos_available():\n check_status(demo_name, username, pat_token, workspace_url, cloud)\n\ndef check_status(demo_name:str, username = None, pat_token = None, workspace_url = None, cloud = \"AWS\", catalog = None, schema = None):\n \"\"\"\n Check the status of the given demo installation. Will pool the installation job if any and wait for its completion.\n Throw an error if the job wasn't successful.\n \"\"\"\n installer = Installer(username, pat_token, workspace_url, cloud)\n demo_conf = installer.get_demo_conf(demo_name, catalog, schema)\n if schema is None:\n schema = demo_conf.default_schema\n if catalog is None:\n catalog = demo_conf.default_catalog\n if \"settings\" in demo_conf.init_job:\n job_name = demo_conf.init_job[\"settings\"][\"name\"]\n existing_job = installer.db.find_job(job_name)\n if existing_job == None:\n raise Exception(f\"Couldn't find job for demo {demo_name}. Did you install it first?\")\n installer.installer_workflow.wait_for_run_completion(existing_job['job_id'], debug=True)\n runs = installer.db.get(\"2.1/jobs/runs/list\", {\"job_id\": existing_job['job_id'], \"limit\": 1})\n if runs['runs'][0]['state']['result_state'] != \"SUCCESS\":\n raise Exception(f\"Job {existing_job['job_id']} for demo {demo_name} failed: {installer.db.conf.workspace_url}/#job/{existing_job['job_id']}/run/{runs['runs'][0]['run_id']} - {runs}\")\n\n\ndef create_cluster(demo_name, username = None, pat_token = None, workspace_url = None, cloud = \"AWS\"):\n installer = Installer(username, pat_token, workspace_url, cloud = cloud)\n installer.check_demo_name(demo_name)\n print(f\"Updating cluster for demo {demo_name}...\")\n demo_conf = installer.get_demo_conf(demo_name)\n installer.tracker.track_create_cluster(demo_conf.category, demo_name)\n cluster_id, cluster_name = installer.load_demo_cluster(demo_name, demo_conf, True)\n installer.report.display_install_result(demo_name, demo_conf.description, demo_conf.title, cluster_id = cluster_id, cluster_name = cluster_name)\n\n\ndef check_version():\n \"\"\"\n Check if a newer version of dbdemos is available on PyPI.\n Prints a warning if the installed version is outdated.\n \"\"\"\n try:\n import pkg_resources\n import requests\n import json\n \n # Get installed version\n installed_version = pkg_resources.get_distribution('dbdemos').version\n \n # Get latest version from PyPI\n pypi_response = requests.get(\"https://pypi.org/pypi/dbdemos/json\")\n latest_version = json.loads(pypi_response.text)['info']['version']\n \n # Compare versions\n if pkg_resources.parse_version(latest_version) > pkg_resources.parse_version(installed_version):\n print(f\"\\nWARNING: You are using dbdemos version {installed_version}, however version {latest_version} is available. You should consider upgrading:\")\n print(\"%pip install --upgrade dbdemos\")\n print(\"dbutils.library.restartPython()\")\n \n except Exception as e:\n # Silently handle any errors during version check\n pass\n" }, { "path": "dbdemos/exceptions/__init__.py", "content": "" }, { "path": "dbdemos/exceptions/dbdemos_exception.py", "content": "\nclass TokenException(Exception):\n def __init__(self, message):\n super().__init__(message)\n self.message = message\n\nclass ClusterException(Exception):\n def __init__(self, message, cluster_conf, response):\n super().__init__(message)\n self.response = response\n self.cluster_conf = cluster_conf\n\n\nclass ClusterPermissionException(ClusterException):\n def __init__(self, message, cluster_conf, response):\n super().__init__(message, cluster_conf, response)\n\n\nclass ClusterCreationException(ClusterException):\n def __init__(self, message, cluster_conf, response):\n super().__init__(message, cluster_conf, response)\n\n\nclass GenieCreationException(Exception):\n def __init__(self, message, genie_conf, response):\n super().__init__(message)\n self.response = response\n self.genie_conf = genie_conf\n\n\nclass ExistingResourceException(Exception):\n def __init__(self, install_path, response):\n super().__init__(f\"Folder {install_path} isn't empty.\")\n self.install_path = install_path\n self.response = response\n\nclass SQLQueryException(Exception):\n def __init__(self, message):\n super().__init__(message)\n\nclass DataLoaderException(Exception):\n def __init__(self, message):\n super().__init__(message)\n\nclass FolderDeletionException(Exception):\n def __init__(self, install_path, response):\n super().__init__(f\"Can't delete folder {install_path}.\")\n self.install_path = install_path\n self.response = response\n\nclass FolderCreationException(Exception):\n def __init__(self, install_path, response):\n super().__init__(f\"Can't load notebook {install_path}.\")\n self.install_path = install_path\n self.response = response\n\n\n\nclass SDPException(Exception):\n def __init__(self, message, description, pipeline_conf, response):\n super().__init__(message)\n self.description = description\n self.pipeline_conf = pipeline_conf\n self.response = response\n\nclass SDPNotAvailableException(SDPException):\n def __init__(self, message, pipeline_conf, response):\n super().__init__(\"SDP not available\", message, pipeline_conf, response)\n\nclass SDPCreationException(SDPException):\n def __init__(self, message, pipeline_conf, response):\n super().__init__(\"SDP creation failure\", message, pipeline_conf, response)\n\nclass WorkflowException(Exception):\n def __init__(self, message, details, job_config, response):\n super().__init__(message)\n self.details = details\n self.job_config = job_config\n self.response = response\n\n" }, { "path": "dbdemos/installer.py", "content": "import collections\n\nimport pkg_resources\n\n\nfrom .conf import DBClient, DemoConf, Conf, ConfTemplate, merge_dict, DemoNotebook\nfrom .exceptions.dbdemos_exception import ClusterPermissionException, ClusterCreationException, ClusterException, \\\n ExistingResourceException, FolderDeletionException, SDPNotAvailableException, SDPCreationException, SDPException, \\\n FolderCreationException, TokenException\nfrom .installer_report import InstallerReport\nfrom .installer_genie import InstallerGenie\nfrom .installer_dashboard import InstallerDashboard\nfrom .tracker import Tracker\nfrom .notebook_parser import NotebookParser\nfrom .installer_workflows import InstallerWorkflow\nfrom .installer_repos import InstallerRepo\nfrom pathlib import Path\nimport time\nimport json\nimport re\nimport base64\nfrom concurrent.futures import ThreadPoolExecutor\nfrom datetime import date\nimport urllib\nimport threading\nfrom dbdemos.sql_query import SQLQueryExecutor\nfrom databricks.sdk import WorkspaceClient\n\nclass Installer:\n def __init__(self, username = None, pat_token = None, workspace_url = None, cloud = \"AWS\", org_id: str = None, current_cluster_id: str = None):\n self.cloud = cloud\n self.dbutils = None\n if username is None:\n username = self.get_current_username()\n if workspace_url is None:\n workspace_url = self.get_current_url()\n if pat_token is None:\n pat_token = self.get_current_pat_token()\n if org_id is None:\n org_id = self.get_org_id()\n self.current_cluster_id = current_cluster_id\n if self.current_cluster_id is None:\n self.current_cluster_id = self.get_current_cluster_id()\n conf = Conf(username, workspace_url, org_id, pat_token)\n self.tracker = Tracker(org_id, self.get_uid(), username)\n self.db = DBClient(conf)\n self.report = InstallerReport(self.db.conf.workspace_url)\n self.installer_workflow = InstallerWorkflow(self)\n self.installer_repo = InstallerRepo(self)\n self.installer_dashboard = InstallerDashboard(self)\n self.installer_genie = InstallerGenie(self)\n self.sql_query_executor = SQLQueryExecutor()\n #Slows down on GCP as the dashboard API is very sensitive to back-pressure\n # 1 dashboard at a time to reduce import pression as it seems to be creating new errors.\n self.max_workers = 1 if self.get_current_cloud() == \"GCP\" else 1\n\n\n def get_dbutils(self):\n if self.dbutils is None:\n try:\n from pyspark.sql import SparkSession\n spark = SparkSession.getActiveSession()\n from pyspark.dbutils import DBUtils\n self.dbutils = DBUtils(spark)\n except:\n try:\n import IPython\n self.dbutils = IPython.get_ipython().user_ns[\"dbutils\"]\n except:\n #Can't get dbutils (local run)\n return None\n return self.dbutils\n\n\n def get_current_url(self):\n try:\n return \"https://\"+self.get_dbutils().notebook.entry_point.getDbutils().notebook().getContext().browserHostName().get()\n except:\n try:\n return \"https://\"+self.get_dbutils_tags_safe()['browserHostName']\n except:\n return \"local\"\n\n def get_dbutils_tags_safe(self):\n import json\n return json.loads(self.get_dbutils().notebook.entry_point.getDbutils().notebook().getContext().safeToJson())['attributes']\n\n def get_current_cluster_id(self):\n try:\n return self.get_dbutils().notebook.entry_point.getDbutils().notebook().getContext().tags().apply('clusterId')\n except:\n try:\n return self.get_dbutils().notebook.entry_point.getDbutils().notebook().getContext().clusterId().get()\n except:\n try:\n return self.get_dbutils_tags_safe()['clusterId']\n except:\n return \"local\"\n\n def get_org_id(self):\n try:\n return self.get_dbutils().notebook.entry_point.getDbutils().notebook().getContext().tags().apply('orgId')\n except:\n try:\n return self.get_dbutils_tags_safe()['orgId']\n except:\n return \"local\"\n\n def get_uid(self):\n try:\n return self.get_dbutils().notebook.entry_point.getDbutils().notebook().getContext().tags().apply('userId')\n except:\n return \"local\"\n\n def get_current_folder(self):\n try:\n current_notebook = self.get_dbutils().notebook.entry_point.getDbutils().notebook().getContext().notebookPath().get()\n return current_notebook[:current_notebook.rfind(\"/\")]\n except:\n try:\n current_notebook = self.get_dbutils_tags_safe()['notebook_path']\n return current_notebook[:current_notebook.rfind(\"/\")]\n except:\n return \"local\"\n def get_workspace_id(self):\n try:\n return self.get_dbutils().notebook.entry_point.getDbutils().notebook().getContext().workspaceId().get()\n except:\n try:\n return self.get_dbutils_tags_safe()['orgId']\n except:\n return \"local\"\n def get_current_pat_token(self):\n try:\n token = self.get_dbutils().notebook.entry_point.getDbutils().notebook().getContext().apiToken().get()\n except Exception as e:\n raise TokenException(\"Couldn't get a PAT Token: \"+str(e)+\". If you're installing it locally or from a batch, please use the pat_token='xxx' parameter instead using a secret.\")\n if len(token) == 0:\n raise TokenException(\"Empty PAT Token.\")\n return token\n\n def get_current_username(self):\n try:\n return self.get_dbutils().notebook.entry_point.getDbutils().notebook().getContext().tags().apply('user')\n except Exception as e2:\n try:\n return self.get_dbutils().notebook.entry_point.getDbutils().notebook().getContext().userName().get()\n except Exception as e:\n try:\n return self.get_dbutils_tags_safe()['user']\n except:\n print(f\"WARN: couldn't get current username. This shouldn't happen - unpredictable behavior - 2 errors: {e2} - {e} - will return 'unknown'\")\n return \"unknown\"\n\n def get_current_cloud(self):\n try:\n hostname = self.get_dbutils().notebook.entry_point.getDbutils().notebook().getContext().browserHostName().get()\n except:\n print(f\"WARNING: Can't get cloud from dbutils. Fallback to default local cloud {self.cloud}\")\n return self.cloud\n if \"gcp\" in hostname:\n return \"GCP\"\n elif \"azure\" in hostname:\n return \"AZURE\"\n else:\n return \"AWS\"\n \n def get_current_cluster_id(self):\n try:\n return self.get_dbutils().notebook.entry_point.getDbutils().notebook().getContext().tags().apply('clusterId')\n except:\n try:\n return self.get_dbutils().notebook.entry_point.getDbutils().notebook().getContext().clusterId().get()\n except:\n try:\n return self.get_dbutils_tags_safe()['clusterId']\n except:\n return \"local\"\n\n def get_workspace_url(self):\n try:\n workspace_url = \"https://\"+self.get_dbutils().notebook.entry_point.getDbutils().notebook().getContext().browserHostName().get()\n except Exception as e:\n raise Exception(\"Couldn't get workspace URL: \"+str(e))\n return workspace_url\n\n def check_demo_name(self, demo_name):\n demos = collections.defaultdict(lambda: [])\n #Define category order\n demos[\"lakehouse\"] = []\n demo_availables = self.get_demos_available()\n if demo_name not in demo_availables:\n for demo in demo_availables:\n conf = self.get_demo_conf(demo)\n demos[conf.category].append(conf)\n self.report.display_demo_name_error(demo_name, demos)\n\n def get_demos_available(self):\n return set(pkg_resources.resource_listdir(\"dbdemos\", \"bundles\"))\n\n def get_demo_conf(self, demo_name:str, catalog:str = None, schema:str = None, demo_folder: str = \"\"):\n demo = self.get_resource(f\"bundles/{demo_name}/conf.json\")\n raw_demo = json.loads(demo)\n catalog = catalog if catalog is not None else raw_demo.get('default_catalog', None)\n schema = schema if schema is not None else raw_demo.get('default_schema', None)\n conf_template = ConfTemplate(self.db.conf.username, demo_name, catalog, schema, demo_folder)\n return DemoConf(demo_name, json.loads(conf_template.replace_template_key(demo)), catalog, schema)\n\n def get_resource(self, path, decode=True):\n resource = pkg_resources.resource_string(\"dbdemos\", path)\n return resource.decode('UTF-8') if decode else resource\n \n def resource_isdir(self, path):\n return pkg_resources.resource_isdir(\"dbdemos\", path)\n\n def test_premium_pricing(self):\n try:\n w = self.db.get(\"2.0/sql/config/warehouses\", {\"limit\": 1}, print_auth_error = False)\n if \"error_code\" in w and (w[\"error_code\"] == \"FEATURE_DISABLED\" or w[\"error_code\"] == \"ENDPOINT_NOT_FOUND\"):\n self.report.display_non_premium_warn(Exception(f\"DBSQL not available, either at workspace level or user entitlement.\"), w)\n return False\n return True\n except Exception as e:\n print(e)\n self.report.display_non_premium_warn(Exception(f\"DBSQL not available\"), str(e))\n return False\n\n def cluster_is_serverless(self):\n try:\n cluster_details = self.db.get(\"2.0/clusters/get\", {\"cluster_id\": self.get_current_cluster_id()})\n return cluster_details.get(\"enable_serverless_compute\", False)\n except Exception as e:\n print(f\"Couldn't get cluster serverless status. Will consider it False. {e}\")\n return False\n\n def create_or_check_schema(self, demo_conf: DemoConf, create_schema: bool, debug=True):\n \"\"\"Create or verify schema exists based on create_schema parameter\"\"\"\n ws = WorkspaceClient(token=self.db.conf.pat_token, host=self.db.conf.workspace_url)\n try:\n catalog = ws.catalogs.get(demo_conf.catalog)\n except Exception as e:\n if create_schema:\n if debug:\n print(f\"Can't describe catalog {demo_conf.catalog}. Will now try to create it. Error: {e}\")\n try:\n print(f\"Catalog {demo_conf.catalog} doesn't exist. Creating it. You can set create_schema=False to avoid catalog and schema creation, or install in another catalog with catalog=<catalog_name>.\")\n self.sql_query_executor.execute_query(ws, f\"CREATE CATALOG IF NOT EXISTS {demo_conf.catalog}\")\n #note: ws.catalogs.create(demo_conf.catalog) this doesn't work properly in serverless workspaces with default storage for now (Metastore storage root URL does not exist error)\n except Exception as e:\n self.report.display_schema_creation_error(e, demo_conf)\n else:\n self.report.display_schema_not_found_error(e, demo_conf)\n\n schema_full_name = f\"{demo_conf.catalog}.{demo_conf.schema}\"\n try:\n schema = ws.schemas.get(schema_full_name)\n except Exception as e:\n if create_schema:\n if debug:\n print(f\"Can't describe schema {schema_full_name}. Will now try to create it. Error: {e}\")\n try:\n schema = ws.schemas.create(demo_conf.schema, catalog_name=demo_conf.catalog)\n except Exception as e:\n self.report.display_schema_creation_error(e, demo_conf)\n else:\n self.report.display_schema_not_found_error(e, demo_conf)\n\n def install_demo(self, demo_name, install_path, overwrite=False, update_cluster_if_exists = True, skip_dashboards = False, start_cluster = None,\n use_current_cluster = False, debug = False, catalog = None, schema = None, serverless=False, warehouse_name = None, skip_genie_rooms=False, \n create_schema=True, dlt_policy_id = None, dlt_compute_settings = None):\n # first get the demo conf.\n if install_path is None:\n install_path = self.get_current_folder()\n elif install_path.startswith(\"./\"):\n install_path = self.get_current_folder()+\"/\"+install_path[2:]\n elif not install_path.startswith(\"/\"):\n install_path = self.get_current_folder()+\"/\"+install_path\n if install_path.endswith(\"/\"):\n install_path = install_path[:-1]\n if serverless is None:\n serverless = self.cluster_is_serverless()\n self.check_demo_name(demo_name)\n demo_conf = self.get_demo_conf(demo_name, catalog, schema, install_path+\"/\"+demo_name)\n if (schema is not None or catalog is not None) and not demo_conf.custom_schema_supported:\n self.report.display_custom_schema_not_supported_error(Exception(f'Custom schema not supported for {demo_conf.name}'), demo_conf)\n if schema is None:\n schema = demo_conf.default_schema\n if catalog is None:\n catalog = demo_conf.default_catalog\n if \"-\" in schema or \"-\" in catalog:\n self.report.display_incorrect_schema_error(Exception('Please use a valid schema/catalog name.'), demo_conf)\n\n # Add schema validation/creation after demo_conf initialization\n if demo_conf.custom_schema_supported:\n self.create_or_check_schema(demo_conf, create_schema, debug)\n\n if demo_name.startswith(\"aibi\"):\n use_current_cluster = True\n if serverless:\n use_current_cluster = True\n if not demo_conf.serverless_supported:\n self.report.display_serverless_warn(Exception('This DBDemo content is not yet updated to Serverless/Express!'), demo_conf)\n\n self.report.display_install_info(demo_conf, install_path, catalog, schema)\n self.tracker.track_install(demo_conf.category, demo_name)\n use_cluster_id = self.current_cluster_id if use_current_cluster else None\n try:\n cluster_id, cluster_name = self.load_demo_cluster(demo_name, demo_conf, update_cluster_if_exists, start_cluster, use_cluster_id)\n except ClusterException as e:\n #Fallback to current cluster if we can't create a cluster.\n cluster_id = self.current_cluster_id\n self.report.display_cluster_creation_warn(e, demo_conf)\n cluster_name = \"Current Cluster\"\n self.check_if_install_folder_exists(demo_name, install_path, demo_conf, overwrite, debug)\n pipeline_ids = self.load_demo_pipelines(demo_name, demo_conf, debug, serverless, dlt_policy_id, dlt_compute_settings)\n # Create Genie rooms before dashboards so we can optionally inject their uid into dashboards\n genie_rooms = self.installer_genie.install_genies(demo_conf, install_path, warehouse_name, skip_genie_rooms, debug)\n dashboards = [] if skip_dashboards else self.installer_dashboard.install_dashboards(demo_conf, install_path, warehouse_name, debug, genie_rooms)\n repos = self.installer_repo.install_repos(demo_conf, debug)\n workflows = self.installer_workflow.install_workflows(demo_conf, use_cluster_id, warehouse_name, serverless, debug)\n init_job = self.installer_workflow.create_demo_init_job(demo_conf, use_cluster_id, warehouse_name, serverless, debug)\n all_workflows = workflows if init_job[\"id\"] is None else workflows + [init_job]\n notebooks = self.install_notebooks(demo_name, install_path, demo_conf, cluster_name, cluster_id, pipeline_ids, dashboards, all_workflows, repos, overwrite, use_current_cluster, genie_rooms, debug)\n self.installer_workflow.start_demo_init_job(demo_conf, init_job, debug)\n for pipeline in pipeline_ids:\n if \"run_after_creation\" in pipeline and pipeline[\"run_after_creation\"]:\n self.db.post(f\"2.0/pipelines/{pipeline['uid']}/updates\", { \"full_refresh\": True })\n\n self.report.display_install_result(demo_name, demo_conf.description, demo_conf.title, install_path, notebooks, init_job['uid'], init_job['run_id'], serverless, cluster_id, cluster_name, pipeline_ids, dashboards, workflows, genie_rooms)\n\n\n def get_demo_datasource(self, warehouse_name = None):\n data_sources = self.db.get(\"2.0/preview/sql/data_sources\")\n if warehouse_name is not None:\n for source in data_sources:\n if source['name'] == warehouse_name:\n return source\n raise Exception(f\"\"\"Error creating the dashboard: cannot find warehouse with warehouse_name='{warehouse_name}' to load your dashboards. Use a different name and make sure the endpoint exists.\"\"\")\n \n for source in data_sources:\n if source['name'] == \"dbdemos-shared-endpoint\":\n return source\n #Try to fallback to an existing shared endpoint.\n for source in data_sources:\n #Default serverless warehouse in express workspaces\n if \"serverless starter warehouse\" in source['name'].lower():\n return source\n for source in data_sources:\n if \"shared-sql-endpoint\" in source['name'].lower():\n return source\n for source in data_sources:\n if \"shared\" in source['name'].lower():\n return source\n return None\n\n def get_or_create_endpoint(self, username: str, demo_conf: DemoConf, default_endpoint_name: str =\"dbdemos-shared-endpoint\", warehouse_name: str = None, throw_error: bool = False):\n try:\n ds = self.get_demo_datasource(warehouse_name)\n except Exception as e:\n self.report.display_unknow_warehouse_error(e, demo_conf, warehouse_name)\n \n if ds is not None:\n return ds\n def get_definition(serverless, name):\n return {\n \"name\": name,\n \"cluster_size\": \"Small\",\n \"min_num_clusters\": 1,\n \"max_num_clusters\": 1,\n \"tags\": {\n \"project\": \"dbdemos\"\n },\n \"spot_instance_policy\": \"COST_OPTIMIZED\",\n \"warehouse_type\": \"PRO\",\n \"enable_photon\": \"true\",\n \"enable_serverless_compute\": serverless,\n \"channel\": { \"name\": \"CHANNEL_NAME_CURRENT\" }\n }\n def try_create_endpoint(serverless):\n w = self.db.post(\"2.0/sql/warehouses\", json=get_definition(serverless, default_endpoint_name))\n if \"message\" in w and \"already exists\" in w['message']:\n w = self.db.post(\"2.0/sql/warehouses\", json=get_definition(serverless, default_endpoint_name + \"-\" + username))\n if \"id\" in w:\n return w\n if serverless:\n print(f\"WARN: Couldn't create serverless warehouse ({default_endpoint_name}). Will fallback to standard SQL warehouse. Creation response: {w}\")\n else:\n print(f\"WARN: Couldn't create warehouse: {default_endpoint_name} and {default_endpoint_name}-{username}. Creation response: {w}. Use another warehouse to view your dashboard.\")\n return None\n\n if try_create_endpoint(True) is None:\n #Try to fallback with classic endpoint?\n try_create_endpoint(False)\n ds = self.get_demo_datasource()\n if ds is not None:\n return ds\n print(f\"ERROR: Couldn't create endpoint. Use the option warehouse_name={warehouse_name} to specify a different warehouse during the installation.\")\n if throw_error:\n self.report.display_warehouse_creation_error(Exception(\"Couldn't create endpoint - see WARNINGS for more details.\"), demo_conf)\n return None\n\n #Check if the folder already exists, and delete it if needed.\n def check_if_install_folder_exists(self, demo_name: str, install_path: str, demo_conf: DemoConf, overwrite=False, debug=False):\n install_path = install_path+\"/\"+demo_name\n s = self.db.get(\"2.0/workspace/get-status\", {\"path\": install_path})\n if 'object_type' in s:\n if not overwrite:\n self.report.display_folder_already_existing(ExistingResourceException(install_path, s), demo_conf)\n if debug:\n print(f\" Folder {install_path} already exists. Deleting the existing content...\")\n assert install_path.lower() not in ['/users', '/repos', '/shared', '/workspace', '/workspace/shared', '/workspace/users'],\\\n \"Demo name is missing, shouldn't happen. Fail to prevent main deletion.\"\n d = self.db.post(\"2.0/workspace/delete\", {\"path\": install_path, 'recursive': True})\n if 'error_code' in d:\n self.report.display_folder_permission(FolderDeletionException(install_path, d), demo_conf)\n\n def install_notebooks(self, demo_name: str, install_path: str, demo_conf: DemoConf, cluster_name: str, cluster_id: str,\n pipeline_ids, dashboards, workflows, repos, overwrite=False, use_current_cluster=False, genie_rooms = [], debug=False):\n assert len(demo_name) > 4, \"wrong demo name. Fail to prevent potential delete errors.\"\n if debug:\n print(f' Installing notebooks')\n install_path = install_path+\"/\"+demo_name\n folders_created = set()\n #Avoid multiple mkdirs in parallel as it's creating error.\n folders_created_lock = threading.Lock()\n def load_notebook(notebook):\n return load_notebook_path(notebook, \"bundles/\"+demo_name+\"/install_package/\"+notebook.get_clean_path())\n\n def load_notebook_path(notebook: DemoNotebook, template_path):\n parent = str(Path(install_path+\"/\"+notebook.get_clean_path()).parent)\n with folders_created_lock:\n if parent not in folders_created:\n r = self.db.post(\"2.0/workspace/mkdirs\", {\"path\": parent})\n folders_created.add(parent)\n if 'error_code' in r:\n if r['error_code'] == \"RESOURCE_ALREADY_EXISTS\":\n self.report.display_folder_creation_error(FolderCreationException(install_path, r), demo_conf)\n if notebook.object_type == \"FILE\":\n file = self.get_resource(template_path, decode=False)\n # Decode file content, replace schema, then re-encode\n file_content = file.decode('utf-8')\n file_content = NotebookParser.replace_schema_in_content(file_content, demo_conf)\n file_encoded = base64.b64encode(file_content.encode('utf-8')).decode(\"utf-8\")\n r = self.db.post(f\"2.0/workspace/import\", {\"path\": install_path+\"/\"+notebook.get_clean_path(), \"content\": file_encoded, \"format\": \"AUTO\", \"overwrite\": False})\n if 'error_code' in r:\n self.report.display_folder_creation_error(FolderCreationException(f\"{install_path}/{notebook.get_clean_path()}\", r), demo_conf)\n elif notebook.object_type == \"DIRECTORY\":\n zip_folder = self.get_resource(template_path+\".zip\", decode=False)\n zip_folder_encoded = base64.b64encode(zip_folder).decode(\"utf-8\")\n r = self.db.post(f\"2.0/workspace/import\", {\"path\": install_path+\"/\"+notebook.get_clean_path()+\".zip\", \"content\": zip_folder_encoded, \"format\": \"AUTO\", \"overwrite\": False})\n if 'error_code' in r:\n self.report.display_folder_creation_error(FolderCreationException(f\"{install_path}/{notebook.get_clean_path()}\", r), demo_conf)\n else:\n html = self.get_resource(template_path+\".html\")\n parser = NotebookParser(html)\n if notebook.add_cluster_setup_cell and not use_current_cluster:\n self.add_cluster_setup_cell(parser, demo_name, cluster_name, cluster_id, self.db.conf.workspace_url)\n parser.replace_dynamic_links_lakeview_dashboards(dashboards)\n parser.replace_dynamic_links_genie(genie_rooms)\n parser.remove_automl_result_links()\n parser.replace_schema(demo_conf)\n parser.replace_dynamic_links_pipeline(pipeline_ids)\n parser.replace_dynamic_links_repo(repos)\n parser.remove_delete_cell()\n parser.replace_dynamic_links_workflow(workflows)\n parser.set_tracker_tag(self.get_org_id(), self.get_uid(), demo_conf.category, demo_name, notebook.get_clean_path(), self.db.conf.username)\n content = parser.get_html()\n content = base64.b64encode(content.encode(\"utf-8\")).decode(\"utf-8\")\n r = self.db.post(\"2.0/workspace/import\", {\"path\": install_path+\"/\"+notebook.get_clean_path(), \"content\": content, \"format\": \"HTML\"})\n if 'error_code' in r:\n self.report.display_folder_creation_error(FolderCreationException(f\"{install_path}/{notebook.get_clean_path()}\", r), demo_conf)\n return notebook\n\n #Always adds the licence notebooks\n with ThreadPoolExecutor(max_workers=self.max_workers) as executor:\n notebooks = [\n DemoNotebook(\"_resources/LICENSE\", \"LICENSE\", \"Demo License\"),\n DemoNotebook(\"_resources/NOTICE\", \"NOTICE\", \"Demo Notice\"),\n DemoNotebook(\"_resources/README\", \"README\", \"Readme\")\n ]\n def load_notebook_template(notebook):\n load_notebook_path(notebook, f\"template/{notebook.title}\")\n collections.deque(executor.map(load_notebook_template, notebooks))\n with ThreadPoolExecutor(max_workers=self.max_workers) as executor:\n return [n for n in executor.map(load_notebook, demo_conf.notebooks)]\n\n def load_demo_pipelines(self, demo_name, demo_conf: DemoConf, debug=False, serverless=False, dlt_policy_id = None, dlt_compute_settings = None):\n #default cluster conf\n pipeline_ids = []\n for pipeline in demo_conf.pipelines:\n definition = pipeline[\"definition\"]\n if \"event_log\" not in definition:\n definition[\"event_log\"] = {\"catalog\": demo_conf.catalog, \"schema\": demo_conf.schema, \"name\": \"dlt_event_log_\"}\n if \"target\" in definition:\n definition[\"schema\"] = definition[\"target\"]\n del definition[\"target\"] #target is deprecated now (https://docs.databricks.com/api/workspace/pipelines/create#schema)\n #Force channel to current due to issue with PREVIEW on serverless with python verison\n definition[\"channel\"] = \"CURRENT\"\n today = date.today().strftime(\"%Y-%m-%d\")\n #modify cluster definitions if serverless\n if serverless:\n if \"clusters\" in definition:\n del definition['clusters']\n definition['photon'] = True\n definition['serverless'] = True\n if dlt_policy_id is not None:\n self.report.display_pipeline_error(SDPCreationException(f\"Policy ID is not supported for serverless pipelines, {dlt_policy_id}\", definition, None))\n else:\n #enforce demo tagging in the cluster\n for cluster in definition[\"clusters\"]:\n merge_dict(cluster, {\"custom_tags\": {\"project\": \"dbdemos\", \"demo\": demo_name, \"demo_install_date\": today}})\n if dlt_policy_id is not None:\n cluster[\"dlt_policy_id\"] = dlt_policy_id\n if self.db.conf.get_demo_pool() is not None:\n cluster[\"instance_pool_id\"] = self.db.conf.get_demo_pool()\n if \"node_type_id\" in cluster: del cluster[\"node_type_id\"]\n if \"enable_elastic_disk\" in cluster: del cluster[\"enable_elastic_disk\"]\n if \"aws_attributes\" in cluster: del cluster[\"aws_attributes\"]\n if dlt_compute_settings is not None:\n merge_dict(cluster, dlt_compute_settings)\n\n existing_pipeline = self.get_pipeline(definition[\"name\"])\n if debug:\n print(f' Installing pipeline {definition[\"name\"]}')\n if existing_pipeline == None:\n p = self.db.post(\"2.0/pipelines\", definition)\n if 'error_code' in p and p['error_code'] == 'FEATURE_DISABLED':\n message = f'SDP pipelines are not available in this workspace. Only Premium workspaces are supported on Azure.'\n pipeline_ids.append({\"name\": pipeline[\"definition\"][\"name\"], \"uid\": \"INSTALLATION_ERROR\", \"id\": pipeline[\"id\"], \"error\": True})\n self.report.display_pipeline_error(SDPNotAvailableException(message, definition, p))\n continue\n if 'error_code' in p:\n pipeline_ids.append({\"name\": pipeline[\"definition\"][\"name\"], \"uid\": \"INSTALLATION_ERROR\", \"id\": pipeline[\"id\"], \"error\": True})\n self.report.display_pipeline_error(SDPCreationException(f\"Error creating the SDP pipeline: {p['error_code']}\", definition, p))\n continue\n id = p['pipeline_id']\n else:\n if debug:\n print(\" Updating existing pipeline with last configuration\")\n id = existing_pipeline['pipeline_id']\n p = self.db.put(\"2.0/pipelines/\"+id, definition)\n if 'error_code' in p:\n pipeline_ids.append({\"name\": pipeline[\"definition\"][\"name\"], \"uid\": \"INSTALLATION_ERROR\", \"id\": pipeline[\"id\"], \"error\": True})\n if 'complete the migration' in str(p).lower() or 'CANNOT_SET_SCHEMA_FOR_EXISTING_PIPELINE' in str(p):\n self.report.display_pipeline_error_migration(SDPCreationException(f\"Please delete the existing SDP pipeline id {id} before re-installing this demo.\", definition, p))\n else:\n self.report.display_pipeline_error(SDPCreationException(f\"Error updating the SDP pipeline {id}: {p['error_code']}\", definition, p))\n continue\n permissions = self.db.patch(f\"2.0/preview/permissions/pipelines/{id}\", {\n \"access_control_list\": [{\"group_name\": \"users\", \"permission_level\": \"CAN_MANAGE\"}]\n })\n if 'error_code' in permissions:\n print(f\"WARN: Couldn't update the pipeline permission for all users to access: {permissions}. Try deleting the pipeline first?\")\n pipeline_ids.append({\"name\": definition['name'], \"uid\": id, \"id\": pipeline[\"id\"], \"run_after_creation\": pipeline[\"run_after_creation\"]})\n #Update the demo conf tags {{}} with the actual id (to be loaded as a job for example)\n demo_conf.set_pipeline_id(pipeline[\"id\"], id)\n return pipeline_ids\n\n def load_demo_cluster(self, demo_name, demo_conf: DemoConf, update_cluster_if_exists, start_cluster = None, use_cluster_id = None):\n if use_cluster_id is not None:\n return (use_cluster_id, \"Interactive cluster you used for installation - make sure the cluster configuration matches.\")\n \n if demo_conf.create_cluster == False:\n return (None, \"This demo doesn't require cluster\")\n \n #Do not start clusters by default in Databricks FE clusters to avoid costs as we have shared clusters for demos\n if start_cluster is None:\n start_cluster = not (self.db.conf.is_dev_env() or self.db.conf.is_fe_env())\n\n #default cluster conf\n conf_template = ConfTemplate(self.db.conf.username, demo_name)\n cluster_conf = self.get_resource(\"resources/default_cluster_config.json\")\n cluster_conf = json.loads(conf_template.replace_template_key(cluster_conf))\n #add cloud specific setup\n cloud = self.get_current_cloud()\n cluster_conf_cloud = self.get_resource(f\"resources/default_cluster_config-{cloud}.json\")\n cluster_conf_cloud = json.loads(conf_template.replace_template_key(cluster_conf_cloud))\n merge_dict(cluster_conf, cluster_conf_cloud)\n merge_dict(cluster_conf, demo_conf.cluster)\n\n if \"driver_node_type_id\" in cluster_conf:\n if cloud not in cluster_conf[\"driver_node_type_id\"] or cloud not in cluster_conf[\"node_type_id\"]:\n raise Exception(f\"\"\"ERROR CREATING CLUSTER FOR DEMO {demo_name}. You need to speficy the cloud type for all clouds: \"node_type_id\": {\"AWS\": \"g5.4xlarge\", \"AZURE\": \"Standard_NC8as_T4_v3\", \"GCP\": \"a2-highgpu-1g\"} and \"driver_node_type_id\" \"\"\")\n cluster_conf[\"node_type_id\"] = cluster_conf[\"node_type_id\"][cloud]\n cluster_conf[\"driver_node_type_id\"] = cluster_conf[\"driver_node_type_id\"][cloud]\n\n if \"spark.databricks.cluster.profile\" in cluster_conf[\"spark_conf\"] and cluster_conf[\"spark_conf\"][\"spark.databricks.cluster.profile\"] == \"singleNode\":\n del cluster_conf[\"autoscale\"]\n cluster_conf[\"num_workers\"] = 0\n\n existing_cluster = self.find_cluster(cluster_conf[\"cluster_name\"])\n if existing_cluster is None:\n cluster = self.db.post(\"2.0/clusters/create\", json = cluster_conf)\n if \"error_code\" in cluster and cluster[\"error_code\"] == \"PERMISSION_DENIED\":\n raise ClusterPermissionException(f\"Can't create cluster for demo {demo_name}\", cluster_conf, cluster)\n if \"cluster_id\" not in cluster or \"error_code\" in cluster:\n print(f\" WARN: couldn't create the cluster for the demo: {cluster}\")\n raise ClusterCreationException(f\"Can't create cluster for demo {demo_name}\", cluster_conf, cluster)\n else:\n cluster_conf[\"cluster_id\"] = cluster[\"cluster_id\"]\n else:\n cluster_conf[\"cluster_id\"] = existing_cluster[\"cluster_id\"]\n cluster = self.db.get(\"2.0/clusters/get\", params = {\"cluster_id\": cluster_conf[\"cluster_id\"]})\n self.wait_for_cluster_to_stop(cluster_conf, cluster)\n if update_cluster_if_exists:\n cluster = self.db.post(\"2.0/clusters/edit\", json = cluster_conf)\n if \"error_code\" in cluster and cluster[\"error_code\"] != \"INVALID_STATE\":\n raise ClusterCreationException(f\"couldn't edit the cluster conf for {demo_name}\", cluster_conf, cluster)\n self.wait_for_cluster_to_stop(cluster_conf, cluster)\n\n if len(demo_conf.cluster_libraries) > 0:\n install = self.db.post(\"2.0/libraries/install\", json = {\"cluster_id\": cluster_conf[\"cluster_id\"], \"libraries\": demo_conf.cluster_libraries})\n if \"error_code\" in install:\n print(f\"WARN: Couldn't install the libs: {cluster_conf}, libraries={demo_conf.cluster_libraries}\")\n\n # Only start if the cluster already exists (it's starting by default for new cluster)\n if existing_cluster is not None and start_cluster:\n start = self.db.post(\"2.0/clusters/start\", json = {\"cluster_id\": cluster_conf[\"cluster_id\"]})\n if \"error_code\" in start:\n if start[\"error_code\"] == \"INVALID_STATE\" and \\\n (\"unexpected state Pending\" in start[\"message\"] or \"unexpected state Restarting\" in start[\"message\"]):\n print(f\"INFO: looks like the cluster is already starting... full answer: {start}\")\n else:\n raise ClusterCreationException(f\"Couldn't start the cluster for {demo_name}: {start['error_code']} - {start['message']}\", cluster_conf, start)\n\n return cluster_conf['cluster_id'], cluster_conf['cluster_name']\n\n def wait_for_cluster_to_stop(self, cluster_conf, cluster):\n if \"error_code\" in cluster and cluster[\"error_code\"] == \"INVALID_STATE\":\n print(f\" Demo cluster {cluster_conf['cluster_name']} in invalid state. Stopping it...\")\n cluster = self.db.post(\"2.0/clusters/delete\", json = {\"cluster_id\": cluster_conf[\"cluster_id\"]})\n i = 0\n while i < 30:\n i += 1\n cluster = self.db.get(\"2.0/clusters/get\", params = {\"cluster_id\": cluster_conf[\"cluster_id\"]})\n if cluster[\"state\"] == \"TERMINATED\":\n print(\" Cluster properly stopped.\")\n break\n time.sleep(2)\n if cluster[\"state\"] != \"TERMINATED\":\n print(f\" WARNING: Couldn't stop the demo cluster properly. Unknown state. Please stop your cluster {cluster_conf['cluster_name']} before.\")\n\n #return the cluster with the given name or none\n def find_cluster(self, cluster_name):\n clusters = self.db.get(\"2.0/clusters/list\")\n if \"clusters\" in clusters:\n for c in clusters[\"clusters\"]:\n if c[\"cluster_name\"] == cluster_name:\n return c\n return None\n\n def get_pipeline(self, name):\n def get_pipelines(token = None):\n r = self.db.get(\"2.0/pipelines\", {\"max_results\": 100, \"page_token\": token})\n if \"statuses\" in r:\n for p in r[\"statuses\"]:\n if p[\"name\"] == name:\n return p\n if \"next_page_token\" in r:\n return get_pipelines(r[\"next_page_token\"])\n return None\n return get_pipelines()\n\n\n def add_cluster_setup_cell(self, parser: NotebookParser, demo_name, cluster_name, cluster_id, env_url):\n content = \"\"\"%md \\n### A cluster has been created for this demo\\nTo run this demo, just select the cluster `{{CLUSTER_NAME}}` from the dropdown menu ([open cluster configuration]({{ENV_URL}}/#setting/clusters/{{CLUSTER_ID}}/configuration)). <br />\\n*Note: If the cluster was deleted after 30 days, you can re-create it with `dbdemos.create_cluster('{{DEMO_NAME}}')` or re-install the demo: `dbdemos.install('{{DEMO_NAME}}')`*\"\"\"\n content = content.replace(\"{{DEMO_NAME}}\", demo_name) \\\n .replace(\"{{ENV_URL}}\", env_url) \\\n .replace(\"{{CLUSTER_NAME}}\", cluster_name) \\\n .replace(\"{{CLUSTER_ID}}\", cluster_id)\n parser.add_extra_cell(content)\n\n def add_extra_cell(self, html, cell_content, position = 0):\n command = {\n \"version\": \"CommandV1\",\n \"subtype\": \"command\",\n \"commandType\": \"auto\",\n \"position\": 1,\n \"command\": cell_content\n }\n raw_content, content = self.get_notebook_content(html)\n content = json.loads(urllib.parse.unquote(content))\n content[\"commands\"].insert(position, command)\n content = urllib.parse.quote(json.dumps(content), safe=\"()*''\")\n return html.replace(raw_content, base64.b64encode(content.encode('utf-8')).decode('utf-8'))\n\n def get_notebook_content(self, html):\n match = re.search(r'__DATABRICKS_NOTEBOOK_MODEL = \\'(.*?)\\'', html)\n raw_content = match.group(1)\n return raw_content, base64.b64decode(raw_content).decode('utf-8')" }, { "path": "dbdemos/installer_dashboard.py", "content": "from .conf import DemoConf\nimport pkg_resources\n\nfrom typing import TYPE_CHECKING\nif TYPE_CHECKING:\n from .installer import Installer\n\n\nclass InstallerDashboard:\n def __init__(self, installer: 'Installer'):\n self.installer = installer\n self.db = installer.db\n\n def install_dashboards(self, demo_conf: DemoConf, install_path, warehouse_name = None, debug = True, genie_rooms = None):\n if len(demo_conf.dashboards) > 0:\n try:\n if debug:\n print(f'installing {len(demo_conf.dashboards)} dashboards...')\n installed_dash = [self.load_lakeview_dashboard(demo_conf, install_path, d, warehouse_name, genie_rooms) for d in demo_conf.dashboards]\n if debug:\n print(f'dashboard installed: {installed_dash}')\n return installed_dash\n except Exception as e:\n self.installer.report.display_dashboard_error(e, demo_conf)\n elif \"dashboards\" in pkg_resources.resource_listdir(\"dbdemos\", \"bundles/\"+demo_conf.name):\n raise Exception(\"Old dashboard are not supported anymore. This shouldn't happen - please fill a bug\")\n return []\n\n def replace_dashboard_schema(self, demo_conf: DemoConf, definition: str):\n import re\n #main__build is used during the build process to avoid collision with default main. #main_build is used because agent don't support __ in their catalog name.\n definition = re.sub(r\"`?main[_]{1,2}build`\", \"main\", definition)\n definition = re.sub(r\"main[_]{1,2}build\\.\", \"main.\", definition)\n definition = re.sub(r\"`main[_]{1,2}build`\\.\", \"`main`.\", definition)\n if demo_conf.custom_schema_supported:\n return re.sub(r\"`?\" + re.escape(demo_conf.default_catalog) + r\"`?\\.`?\" + re.escape(demo_conf.default_schema) + r\"`?\", f\"`{demo_conf.catalog}`.`{demo_conf.schema}`\", definition)\n return definition\n\n def load_lakeview_dashboard(self, demo_conf: DemoConf, install_path, dashboard, warehouse_name = None, genie_rooms = None):\n endpoint = self.installer.get_or_create_endpoint(self.db.conf.name, demo_conf, warehouse_name = warehouse_name)\n try:\n definition = self.installer.get_resource(f\"bundles/{demo_conf.name}/install_package/_resources/dashboards/{dashboard['id']}.lvdash.json\")\n definition = self.replace_dashboard_schema(demo_conf, definition)\n except Exception as e:\n raise Exception(f\"Can't load dashboard {dashboard} in demo {demo_conf.name}. Check bundle configuration under dashboards: [..]. \"\n f\"The dashboard id should match the file name under the _resources/dashboard/<dashboard> folder.. {e}\")\n # If dashboard['genie_room_id'] matches a created Genie, set overrideId to that Genie's UID.\n try:\n target_room_uid = None\n mapped_room_id = dashboard.get(\"genie_room_id\")\n if mapped_room_id and genie_rooms:\n for room in genie_rooms:\n if room.get(\"id\") == mapped_room_id:\n target_room_uid = room.get(\"uid\")\n break\n if target_room_uid:\n import re as _re\n pattern = r'\"overrideId\"\\s*:\\s*\"\"'\n if _re.search(pattern, definition):\n definition = _re.sub(pattern, f'\"overrideId\": \"{target_room_uid}\"', definition, count=1)\n # If mapping missing or UID not found, skip injection silently.\n except Exception:\n pass\n dashboard_path = f\"{install_path}/{demo_conf.name}/_dashboards\"\n #Make sure the dashboard folder exists\n f = self.db.post(\"2.0/workspace/mkdirs\", {\"path\": dashboard_path})\n if \"error_code\" in f:\n raise Exception(f\"ERROR - wrong install path, can't save dashboard here: {f} - {dashboard_path}\")\n \n #Avoid issue with / in the dashboard name (such as AI/BI)\n dashboard['name'] = dashboard['name'].replace('/', '')\n dashboard_creation = self.db.post(f\"2.0/lakeview/dashboards\", {\n \"display_name\": dashboard['name'],\n \"warehouse_id\": endpoint['warehouse_id'],\n \"serialized_dashboard\": definition,\n \"parent_path\": dashboard_path\n })\n dashboard['uid'] = dashboard_creation['dashboard_id']\n dashboard['is_lakeview'] = True\n return dashboard" }, { "path": "dbdemos/installer_genie.py", "content": "import json\nfrom concurrent.futures import ThreadPoolExecutor, as_completed\nfrom databricks.sdk import WorkspaceClient\nfrom databricks.sdk.service.catalog import VolumeType\nfrom databricks.sdk.service.sql import StatementState\n\nfrom dbdemos.sql_query import SQLQueryExecutor\nfrom .conf import DataFolder, DemoConf, GenieRoom\nfrom .exceptions.dbdemos_exception import GenieCreationException, DataLoaderException, SQLQueryException\n\nfrom typing import TYPE_CHECKING\nif TYPE_CHECKING:\n from .installer import Installer\n\n\nclass InstallerGenie:\n VOLUME_NAME = \"dbdemos_raw_data\"\n\n def __init__(self, installer: 'Installer'):\n self.installer = installer\n self.db = installer.db\n self.sql_query_executor = SQLQueryExecutor()\n\n def install_genies(self, demo_conf: DemoConf, install_path: str, warehouse_name: str, skip_genie_rooms: bool, debug=True):\n rooms = []\n if len(demo_conf.genie_rooms) > 0 or len(demo_conf.data_folders) > 0:\n warehouse = self.installer.get_or_create_endpoint(self.db.conf.name, demo_conf, warehouse_name = warehouse_name, throw_error=True)\n try:\n warehouse_id = warehouse['endpoint_id']\n self.load_genie_data(demo_conf, warehouse_id, debug)\n if not skip_genie_rooms and len(demo_conf.genie_rooms) > 0:\n if debug:\n print(f\"Installing genie room {demo_conf.genie_rooms}\")\n genie_path = f\"{install_path}/{demo_conf.name}/_genie_spaces\"\n #Make sure the genie folder exists\n self.db.post(\"2.0/workspace/mkdirs\", {\"path\": genie_path})\n path = self.db.get(\"2.0/workspace/get-status\", {\"path\": genie_path})\n if \"error_code\" in path:\n raise Exception(f\"ERROR - wrong install path, can't save genie spaces here: {path}\")\n for room in demo_conf.genie_rooms:\n rooms.append(self.install_genie(room, path, warehouse_id, debug))\n except Exception as e:\n self.installer.report.display_genie_room_creation_error(e, demo_conf)\n return rooms\n\n def install_genie(self, room: GenieRoom, genie_path, warehouse_id, debug=True):\n #Genie rooms don't allow / anymore\n ws = WorkspaceClient(token=self.installer.db.conf.pat_token, host=self.installer.db.conf.workspace_url)\n self.create_temp_table_for_genie_creation(ws, room, warehouse_id, debug)\n room.display_name = room.display_name.replace(\"/\", \"-\")\n room_payload = {\n \"display_name\": room.display_name,\n \"description\": room.description,\n \"warehouse_id\": warehouse_id,\n \"table_identifiers\": room.table_identifiers,\n \"parent_folder\": f'folders/{genie_path[\"object_id\"]}',\n \"run_as_type\": \"VIEWER\"\n }\n created_room = self.db.post(\"2.0/data-rooms\", json=room_payload)\n if 'id' not in created_room:\n raise GenieCreationException(f\"Error creating room {room_payload} - {created_room}\", room, created_room)\n\n if debug:\n print(f\"Genie room created created_room: {created_room} - {room_payload}\")\n actions = [{\n \"action_type\": \"CREATE\",\n \"curated_question\": {\n \"data_room_id\": created_room['id'],\n \"question_text\": q,\n \"question_type\": \"SAMPLE_QUESTION\"\n }\n } for q in room.curated_questions]\n questions = self.db.post(f\"2.0/data-rooms/{created_room['id']}/curated-questions/batch-actions\", {\"actions\": actions})\n if debug:\n print(f\"Genie room question created:{questions}\")\n if room.instructions:\n instructions = self.db.post(f\"2.0/data-rooms/{created_room['id']}/instructions\", {\"title\": \"Notes\", \"content\": room.instructions, \"instruction_type\": \"TEXT_INSTRUCTION\"})\n if debug:\n print(f\"genie room instructions: {instructions}\")\n \n for function_name in room.function_names:\n instructions = self.db.post(f\"2.0/data-rooms/{created_room['id']}/instructions\", {\"title\": \"SQL Function\", \"content\": function_name, \"instruction_type\": \"CERTIFIED_ANSWER\"})\n if debug:\n print(f\"genie room function: {instructions}\")\n for sql in room.sql_instructions:\n instructions = self.db.post(f\"2.0/data-rooms/{created_room['id']}/instructions\", {\"title\": sql['title'], \"content\": sql['content'], \"instruction_type\": \"SQL_INSTRUCTION\"})\n if debug:\n print(f\"genie room SQL instructions: {instructions}\")\n for b in room.benchmarks:\n benchmark = {\n \"question_text\": b[\"question_text\"],\n \"question_type\":\"BENCHMARK\",\n \"answer_text\": b[\"answer_text\"],\n \"is_deprecated\": False,\n \"updatable_fields_mask\":[]\n }\n instructions = self.db.post(f\"2.0/data-rooms/{created_room['id']}/curated-questions\", {\"curated_question\": benchmark, \"data_room_id\":created_room['id']})\n if debug:\n print(f\"genie room benchmarks: {instructions}\")\n self.delete_temp_table_for_genie_creation(ws, room, debug)\n return {\"id\": room.id, \"uid\": created_room['id'], 'name': room.display_name}\n\n # we need to have the table existing before creating the genie room, however they're created in SDP which is in a job and not yet available.\n # This is a workaround to create a temp table with a property that will be used to delete it once the genie room is created so that the SDP table can run without issue.\n def create_temp_table_for_genie_creation(self, ws: WorkspaceClient, room: GenieRoom, warehouse_id, debug=False):\n for table in room.table_identifiers:\n if not ws.tables.exists(table).table_exists:\n sql_query = f\"CREATE TABLE IF NOT EXISTS {table} TBLPROPERTIES ('dbdemos.mock_table_for_genie' = 1);\"\n if debug:\n print(f\"Creating temp genie table {table}: {sql_query}\")\n self.sql_query_executor.execute_query(ws, sql_query, warehouse_id=warehouse_id, debug=debug)\n\n def delete_temp_table_for_genie_creation(self, ws, room: GenieRoom, debug=False):\n for table in room.table_identifiers:\n if ws.tables.exists(table).table_exists and 'dbdemos.mock_table_for_genie' in ws.tables.get(table).properties:\n if debug:\n print(f'Deleting temp genie table {table}')\n ws.tables.delete(table)\n\n def load_genie_data(self, demo_conf: DemoConf, warehouse_id, debug=True):\n if demo_conf.data_folders:\n print(f\"Loading data in your schema {demo_conf.catalog}.{demo_conf.schema} using warehouse {warehouse_id}, this might take a few seconds (you can use another warehouse with the option: warehouse_name='xxx')...\")\n ws = WorkspaceClient(token=self.installer.db.conf.pat_token, host=self.installer.db.conf.workspace_url)\n if any(d.target_volume_folder_name is not None for d in demo_conf.data_folders):\n self.create_raw_data_volume(ws, demo_conf, debug)\n\n with ThreadPoolExecutor(max_workers=3) as executor:\n futures = [executor.submit(self.load_data, ws, data_folder, warehouse_id, demo_conf, debug) \n for data_folder in demo_conf.data_folders]\n for future in futures:\n future.result()\n if demo_conf.sql_queries:\n self.run_sql_queries(ws, demo_conf, warehouse_id, debug)\n\n def run_sql_queries(self, ws: WorkspaceClient, demo_conf: DemoConf, warehouse_id, debug=True):\n for batch in demo_conf.sql_queries:\n with ThreadPoolExecutor(max_workers=5) as ex:\n futures = [ex.submit(self.sql_query_executor.execute_query, ws, q, warehouse_id=warehouse_id, debug=debug) for q in batch]\n for f in as_completed(futures):\n try:\n f.result()\n except SQLQueryException as e:\n if \"tag name\" in str(e).lower():\n print(f\"Warn - SQL error on tag ignored - probably free edition: {e}\")\n else: raise\n\n def get_current_cluster_id(self):\n return json.loads(self.installer.get_dbutils_tags_safe()['clusterId'])\n\n def load_data(self, ws: WorkspaceClient, data_folder: DataFolder, warehouse_id, conf: DemoConf, debug=True):\n # Load table to a table\n if data_folder.target_table_name:\n try:\n sql_query = f\"\"\"CREATE TABLE IF NOT EXISTS {conf.catalog}.{conf.schema}.{data_folder.target_table_name} as \n SELECT * FROM read_files('s3://dbdemos-dataset/{data_folder.source_folder}', \n format => '{data_folder.source_format}', \n pathGlobFilter => '*.{data_folder.source_format}')\"\"\"\n if debug:\n print(f\"Loading data {data_folder}: {sql_query}\")\n self.sql_query_executor.execute_query(ws, sql_query, warehouse_id=warehouse_id, debug=debug)\n except Exception as e:\n if \"com.amazonaws.auth.BasicSessionCredentials\" in str(e):\n print(\"INFO: Basic Credential error detected downloading the files from our demo bucket. Will try to load data to volume first, please wait as this is a slower workflow...\")\n self.create_raw_data_volume(ws, conf, debug)\n self.load_data_to_volume(ws, data_folder, conf, debug)\n self.create_table_from_volume(ws, data_folder, warehouse_id, conf, debug)\n else:\n raise DataLoaderException(f\"Error loading data from S3: {str(e)}\")\n else:\n self.load_data_to_volume(ws, data_folder, conf, debug)\n \n # Class-level lock for volume creation\n import threading\n _volume_creation_lock = threading.Lock()\n\n def create_raw_data_volume(self, ws: WorkspaceClient, demo_conf: DemoConf, debug=True):\n with InstallerGenie._volume_creation_lock:\n full_volume_name = f\"{demo_conf.catalog}/{demo_conf.schema}/{InstallerGenie.VOLUME_NAME}\"\n try:\n ws.volumes.read(f\"{demo_conf.catalog}.{demo_conf.schema}.{InstallerGenie.VOLUME_NAME}\")\n except Exception as e:\n if debug:\n print(f\"Volume {full_volume_name} doesn't seem to exist, creating it - {e}\")\n try:\n ws.volumes.create(\n catalog_name=demo_conf.catalog,\n schema_name=demo_conf.schema,\n name=InstallerGenie.VOLUME_NAME,\n volume_type=VolumeType.MANAGED\n )\n except Exception as e: \n raise DataLoaderException(f\"Can't create volume {full_volume_name} to load data demo, and it doesn't seem to be existing. <br/>\"\n f\"Please create the volume or grant you USAGE/READ permission, or install the demo in another catalog: dbdemos.install(xxx, catalog=xxx, schema=xxx, warehouse_id=xx).<br/>\"\n f\" {e}\")\n\n\n # -------------------------------------------------------------------------------------------------------------------------------------------- \n # Experimental, first upload data to the volume as some warehouse don't have access to the S3 bucket directly when instance profiles exist.\n # -------------------------------------------------------------------------------------------------------------------------------------------- \n def load_data_through_volume(self, ws: WorkspaceClient, data_folders: list[DataFolder], warehouse_id: str, demo_conf: DemoConf, debug=True):\n print('INFO: Basic Credential error detected downloading the files from our demo S3 bucket. Will try to load data to volume first, please wait as this might take a while...')\n self.create_raw_data_volume(ws, demo_conf, debug)\n\n def load_data_and_create_table(ws: WorkspaceClient, data_folder: DataFolder, warehouse_id: str, demo_conf: DemoConf, debug=True):\n self.load_data_to_volume(ws, demo_conf, data_folder, debug)\n self.create_table_from_volume(ws, data_folder, warehouse_id, demo_conf, debug)\n\n with ThreadPoolExecutor(max_workers=3) as executor:\n futures = [executor.submit(load_data_and_create_table, ws, data_folder, warehouse_id, demo_conf, debug)\n for data_folder in data_folders]\n for future in futures:\n future.result()\n\n\n def load_data_to_volume(self, ws: WorkspaceClient, data_folder: DataFolder, demo_conf: DemoConf, debug=True):\n assert data_folder.source_format in [\"csv\", \"json\", \"parquet\"], \"data loader through volume only support csv, json and parquet\"\n\n import requests\n import collections\n dbutils = self.installer.get_dbutils()\n try:\n folder = data_folder.target_volume_folder_name if data_folder.target_volume_folder_name else data_folder.source_folder\n #first try with a dbutils copy if available\n copied_successfully = False\n if debug:\n print(f\"Copying {data_folder.source_folder} to {f'/Volumes/{demo_conf.catalog}/{demo_conf.schema}/{InstallerGenie.VOLUME_NAME}/{folder}'} using dbutils fs.cp\")\n if dbutils is not None:\n try:\n dbutils.fs.cp(f\"s3://dbdemos-dataset/{data_folder.source_folder}\", f\"/Volumes/{demo_conf.catalog}/{demo_conf.schema}/{InstallerGenie.VOLUME_NAME}/{folder}\", recurse=True)\n copied_successfully = True\n except Exception as e:\n copied_successfully = False\n if debug:\n print(f\"Error copying {data_folder.source_folder} to {f'/Volumes/{demo_conf.catalog}/{demo_conf.schema}/{InstallerGenie.VOLUME_NAME}/{folder}'} using dbutils fs.cp: {e}\")\n if copied_successfully and debug:\n print(f\"Copied {data_folder.source_folder} to {f'/Volumes/{demo_conf.catalog}/{demo_conf.schema}/{InstallerGenie.VOLUME_NAME}/{folder}'} using dbutils fs.cp\")\n if not copied_successfully:\n # Get list of files from GitHub API, to avoid adding a S3 boto dependency just for this\n github_path = f\"https://api.github.com/repos/databricks-demos/dbdemos-dataset/contents/{data_folder.source_folder}\"\n if debug:\n print(f\"Getting files from {github_path}\")\n files = requests.get(github_path).json()\n if 'message' in files:\n print(f\"Error getting files from {github_path}: {files}\")\n files = [f['download_url'] for f in files]\n \n if debug:\n print(f\"Found {len(files)} files in GitHub repo for {data_folder.source_folder}\")\n \n def copy_file(file_url):\n if not file_url.endswith('/'):\n file_name = file_url.split('/')[-1]\n target_path = f\"/Volumes/{demo_conf.catalog}/{demo_conf.schema}/{InstallerGenie.VOLUME_NAME}/{folder}/{file_name}\"\n \n s3_url = file_url.replace(\"https://raw.githubusercontent.com/databricks-demos/dbdemos-dataset/main/\", \n \"https://dbdemos-dataset.s3.amazonaws.com/\")\n\n if debug:\n print(f\"Copying {s3_url} to {target_path}\")\n response = requests.get(s3_url)\n response.raise_for_status()\n if debug:\n print(f\"File {file_name} in memory. sending to volume...\")\n import io\n buffer = io.BytesIO(response.content)\n ws.files.upload(target_path, buffer, overwrite=True)\n if debug:\n print(f\"File {file_name} in volume!\")\n \n with ThreadPoolExecutor(max_workers=5) as executor:\n collections.deque(executor.map(copy_file, files))\n\n except Exception as e:\n raise DataLoaderException(f\"Error loading data from S3: {str(e)}\")\n\n def create_table_from_volume(self, ws: WorkspaceClient, data_folder: DataFolder, warehouse_id, conf: DemoConf, debug=True):\n self.sql_query_executor.execute_query(ws, f\"\"\"CREATE TABLE IF NOT EXISTS {conf.catalog}.{conf.schema}.{data_folder.target_table_name} as \n SELECT * FROM read_files('/Volumes/{conf.catalog}/{conf.schema}/{InstallerGenie.VOLUME_NAME}/{data_folder.source_folder}', \n format => '{data_folder.source_format}', \n pathGlobFilter => '*.{data_folder.source_format}')\"\"\", warehouse_id=warehouse_id, debug=debug)\n" }, { "path": "dbdemos/installer_report.py", "content": "from .conf import DemoConf\nfrom .exceptions.dbdemos_exception import ClusterCreationException, ExistingResourceException, FolderDeletionException, \\\n SDPException, WorkflowException, FolderCreationException, TokenException\nfrom pathlib import Path\nimport json\n\nclass InstallerReport:\n\n NOTEBOOK_SVG = \"\"\"<svg width=\"1em\" height=\"1em\" viewBox=\"0 0 16 16\" fill=\"none\" xmlns=\"http://www.w3.org/2000/svg\" aria-hidden=\"true\" focusable=\"false\" class=\"\"><path fill-rule=\"evenodd\" clip-rule=\"evenodd\" d=\"M3 1.75A.75.75 0 013.75 1h10.5a.75.75 0 01.75.75v12.5a.75.75 0 01-.75.75H3.75a.75.75 0 01-.75-.75V12.5H1V11h2V8.75H1v-1.5h2V5H1V3.5h2V1.75zm1.5.75v11H6v-11H4.5zm3 0v11h6v-11h-6z\" fill=\"currentColor\"></path></svg>\"\"\"\n FOLDER_SVG = \"\"\"<svg width=\"1em\" height=\"1em\" viewBox=\"0 0 16 16\" fill=\"none\" xmlns=\"http://www.w3.org/2000/svg\" aria-hidden=\"true\" focusable=\"false\" class=\"\"><path d=\"M.75 2a.75.75 0 00-.75.75v10.5c0 .414.336.75.75.75h14.5a.75.75 0 00.75-.75v-8.5a.75.75 0 00-.75-.75H7.81L6.617 2.805A2.75 2.75 0 004.672 2H.75z\" fill=\"currentColor\"></path></svg>\"\"\"\n DASHBOARD_SVG = \"\"\"<svg width=\"1em\" height=\"1em\" viewBox=\"0 0 16 16\" fill=\"none\" xmlns=\"http://www.w3.org/2000/svg\" aria-hidden=\"true\" focusable=\"false\" class=\"\"><path fill-rule=\"evenodd\" clip-rule=\"evenodd\" d=\"M1 1.75A.75.75 0 011.75 1h12.5a.75.75 0 01.75.75v12.5a.75.75 0 01-.75.75H1.75a.75.75 0 01-.75-.75V1.75zm1.5 8.75v3h4.75v-3H2.5zm0-1.5h4.75V2.5H2.5V9zm6.25-6.5v3h4.75v-3H8.75zm0 11V7h4.75v6.5H8.75z\" fill=\"currentColor\"></path></svg>\"\"\"\n GENIE_SVG = \"\"\"<svg xmlns=\"http://www.w3.org/2000/svg\" width=\"1em\" height=\"1em\" fill=\"none\" viewBox=\"0 0 16 16\" aria-hidden=\"true\" focusable=\"false\" class=\"\"><path fill=\"currentColor\" fill-rule=\"evenodd\" d=\"M0 2.75A.75.75 0 0 1 .75 2H8v1.5H1.5v9h13V10H16v3.25a.75.75 0 0 1-.75.75H.75a.75.75 0 0 1-.75-.75zm12.987-.14a.75.75 0 0 0-1.474 0l-.137.728a1.93 1.93 0 0 1-1.538 1.538l-.727.137a.75.75 0 0 0 0 1.474l.727.137c.78.147 1.39.758 1.538 1.538l.137.727a.75.75 0 0 0 1.474 0l.137-.727c.147-.78.758-1.39 1.538-1.538l.727-.137a.75.75 0 0 0 0-1.474l-.727-.137a1.93 1.93 0 0 1-1.538-1.538z\" clip-rule=\"evenodd\"></path></svg>\"\"\"\n\n CSS_REPORT = \"\"\"\n <style>\n .dbdemos_install{\n font-family: -apple-system,BlinkMacSystemFont,Segoe UI,Roboto,Helvetica Neue,Arial,Noto Sans,sans-serif,Apple Color Emoji,Segoe UI Emoji,Segoe UI Symbol,Noto Color Emoji,FontAwesome;\n color: #3b3b3b;\n box-shadow: 0 .15rem 1.15rem 0 rgba(58,59,69,.15)!important;\n padding: 10px 20px 20px 20px;\n margin: 10px;\n }\n .dbdemos_block{\n display: block !important;\n }\n .update_container {\n display: flex;\n gap: 20px;\n padding: 10px;\n margin: 10px 0;\n }\n .update_box {\n flex: 1;\n background-color: #f3fff9;\n padding: 15px;\n border-radius: 5px;\n box-shadow: 0 .15rem 1.15rem 0 rgba(58, 59, 69, .15);\n overflow: hidden;\n }\n .update_title {\n font-weight: bold;\n color: #34a853;\n margin-bottom: 10px;\n font-size: 1.4em;\n }\n .code {\n padding: 5px;\n border: 1px solid #e4e4e4;\n font-family: monospace;\n background-color: #f5f5f5;\n margin: 5px 0px 0px 0px;\n display: inline;\n }\n .update_image {\n float: right;\n width: 200px;\n margin: 0 0 10px 10px;\n border-radius: 5px;\n }\n .subfolder {\n padding-left: 30px;\n }\n .notebook {\n margin-bottom: 3px;\n }\n .dbdemos_install a {\n color: #3835a4;\n }\n .container_dbdemos {\n padding-left: 20px;\n }\n .path_desc {\n color: #928e9b;\n font-style: oblique;\n }\n </style>\"\"\"\n\n def __init__(self, workspace_url: str):\n self.workspace_url = workspace_url\n\n def displayHTML_available(self):\n try:\n from dbruntime.display import displayHTML\n return True\n except:\n return False\n\n def display_cluster_creation_warn(self, exception: ClusterCreationException, demo_conf: DemoConf):\n self.display_error(exception, f\"By default, dbdemos tries to create a new cluster for your demo with the proper settings. <br/>\"\n f\"dbdemos couldn't create a cluster for you (probably due to your permissions). Instead we will use the current cluster to run the setup job and load the data.<br/>\"\n f\"For the demo to run properly, <strong>make sure your cluster has UC enabled and using Databricks Runtime (DBR) version {exception.cluster_conf['spark_version']}</strong>.<br/>\"\n f\"<i>Note: you can avoid this message setting `use_current_cluster`:</i><br/>\"\n f\"\"\"<div class=\"code dbdemos_block\">dbdemos.install('{demo_conf.name}', use_current_cluster = True)</div><br/>\"\"\"\n f\"<strong>Cluster creation details</strong><br/>\"\n f\"\"\"Full cluster configuration: <div class=\"code dbdemos_block\">{json.dumps(exception.cluster_conf)}.</div><br/>\"\"\"\n f\"\"\"Full error: <div class=\"code dbdemos_block\">{json.dumps(exception.response)}</div>\"\"\", raise_error=False, warning=True)\n\n def display_serverless_warn(self, exception: Exception, demo_conf: DemoConf):\n self.display_error(exception, f\"This demo might not fully work on Serverless and Databricks Test Drive!<br/>\"\n f\"We're actively working to update this content to fully work on serverless.<br/>\"\n f\"Some of the notebooks might not work as expected as they are tested with DBRML, we'll be releasing an new version very shortly, stay tuned!<br/>\", raise_error=False, warning=True)\n \n def display_custom_schema_not_supported_error(self, exception: Exception, demo_conf: DemoConf):\n self.display_error(exception, f\"This demo doesn't support custom catalog/schema yet.<br/>\"\n f\"Please open a Github issue to accelerate the support for this demo.<br/>\"\n f\"Remove the 'catalog' and 'schema' option from your installation.<br/>\")\n\n def display_custom_schema_missing_error(self, exception: Exception, demo_conf: DemoConf):\n self.display_error(exception, f\"Both schema and catalog option must be defined.<br/>\"\n f\"\"\"<div class=\"code dbdemos_block\">dbdemos.install('{demo_conf.name}', catalog = 'xxx', schema = 'xxx')</div><br/>\"\"\")\n\n def display_incorrect_schema_error(self, exception: Exception, demo_conf: DemoConf):\n self.display_error(exception, f\"Incorrect schema/catalog name.<br/>\"\n f\"\"\"Please use a correct catalog/schema name. Use '_' instead of '-'.\"\"\")\n\n def display_warehouse_creation_error(self, exception: Exception, demo_conf: DemoConf):\n self.display_error(exception, f\"\"\"This demo requires a warehouse to work and couldn't find one or create one!<br/>\n You can specify the SQL warehouse you would like to use to load the dashboard with warehouse_name = 'xxx':\n <div class=\"code dbdemos_block\">dbdemos.install('{demo_conf.name}', warehouse_name = 'xxx')</div><br/>\"\"\")\n\n def display_unknow_warehouse_error(self, exception: Exception, demo_conf: DemoConf, warehouse_name: str):\n self.display_error(exception, f\"\"\"Can't find your warehouse!<br/>\n The warehouse you specified: {warehouse_name} can't be find. Make sure it exists and you have access to it.\n <div class=\"code dbdemos_block\">dbdemos.install('{demo_conf.name}', warehouse_name = 'xxx')</div><br/>\"\"\")\n\n def display_genie_room_creation_error(self, exception: Exception, demo_conf: DemoConf):\n self.display_error(exception, f\"\"\"This demo couldn't install the genie room properly.<br/>\n Genie room support for DBDemos is in beta. You can skip the genie room installation with skip_genie_rooms = True:\n <div class=\"code dbdemos_block\">dbdemos.install('{demo_conf.name}', skip_genie_rooms = True)</div><br/>\"\"\")\n\n def display_dashboard_error(self, exception: Exception, demo_conf: DemoConf):\n self.display_error(exception, f\"\"\"Couldn't create or update a dashboard. <br/>\n If this is a permission error, we recommend you to search the existing dashboard and delete it manually.<br/>\n You can skip the dashboard installation with skip_dashboards = True:\n <div class=\"code dbdemos_block\">dbdemos.install('{demo_conf.name}', skip_dashboards = True)</div><br/>\n You can also specify the SQL warehouse you'd like to use to load the dashboard with warehouse_name = 'xxx':\n <div class=\"code dbdemos_block\">dbdemos.install('{demo_conf.name}', warehouse_name = 'xxx')</div><br/>\"\"\")\n\n def display_folder_already_existing(self, exception: ExistingResourceException, demo_conf: DemoConf):\n self.display_error(exception, f\"\"\"Please install demo with overwrite=True to replace the existing folder content under {exception.install_path}:\n <div class=\"code dbdemos_block\">dbdemos.install('{demo_conf.name}', overwrite=True, path='{exception.install_path}')</div><br/>\n All content under {exception.install_path} will be deleted.<br/><br/>\n <strong>Details</strong><br/>\n Folder list response: <div class=\"code dbdemos_block\">{json.dumps(exception.response)}</div>\"\"\")\n\n def display_folder_permission(self, exception: FolderDeletionException, demo_conf: DemoConf):\n self.display_error(exception, f\"\"\"Can't delete the folder {exception.install_path}. <br/>\n Do you have read/write permission?<br/><br/>\n <strong>Details</strong><br/>\n Delete response: <div class=\"code dbdemos_block\">{json.dumps(exception.response)}</div>\"\"\")\n\n def display_folder_creation_error(self, exception: FolderCreationException, demo_conf: DemoConf):\n self.display_error(exception, f\"\"\"Couldn't load the model in the current folder. Do you have permissions to write in {exception.install_path}?\n Please install demo with overwrite=True to replace the existing folder:\n <div class=\"code dbdemos_block\">dbdemos.install('{demo_conf.name}', overwrite=True, path='{exception.install_path}')</div><br/>\n All content under {exception.install_path} will be deleted.<br/><br/>\n <strong>Details</strong><br/>\n Folder list response: <div class=\"code dbdemos_block\">{json.dumps(exception.response)}</div>\"\"\")\n\n def display_non_premium_warn(self, exception: Exception, response):\n self.display_error(exception, f\"\"\"DBSQL isn't available in this workspace. Only Premium/Enterprise workspaces are supported.<br/>\n dbdemos will try its best to install the demo and load the notebooks, but some component won't be available (SDP pipelines, Dashboards etc).<br/>\n Forcing skip_dashboards = True and continuing.<br/><br/>\n <strong>Details</strong><br/>\n API response: <div class=\"code dbdemos_block\">{json.dumps(response)}</div>\"\"\", raise_error=False, warning=True)\n\n def display_pipeline_error(self, exception: SDPException):\n self.display_error(exception, f\"\"\"{exception.description}. <br/>\n Skipping pipelines. Your demo will be installed without SDP pipelines.<br/><br/>\n <strong>Details</strong><br/>\n Pipeline configuration: <div class=\"code dbdemos_block\">{json.dumps(exception.pipeline_conf)}</div>\n API response: <div class=\"code dbdemos_block\">{json.dumps(exception.response)}</div>\"\"\", raise_error=False, warning=True)\n\n def display_pipeline_error_migration(self, exception: SDPException):\n self.display_error(exception, f\"\"\"{exception.description}. <br/>\n Skipping pipelines. Your demo will be installed without SDP pipelines.<br/><br/>\n <strong>Details</strong><br/>\n DBDemos updated its API to use the latest SDP features. You installed your pipeline on an older version which needs to be updated.<br/>\n The easiest fix is to delete the existing pipeline and re-install the demo to get the latest SDP features.<br/>\n Pipeline configuration: <div class=\"code dbdemos_block\">{json.dumps(exception.pipeline_conf)}</div>\n API response: <div class=\"code dbdemos_block\">{json.dumps(exception.response)}</div>\"\"\", raise_error=False, warning=True)\n\n def display_workflow_error(self, exception: WorkflowException, demo_name: str):\n self.display_error(exception, f\"\"\"{exception.details}. <br/>\n dbdemos creates jobs to load your demo data. If you don't have cluster creation permission, you can start the job using the current cluster.\n <div class=\"code dbdemos_block\">dbdemos.install('{demo_name}', use_current_cluster=True)</div><br/>\n <strong>Details</strong><br/>\n Pipeline configuration: <div class=\"code dbdemos_block\">{json.dumps(exception.job_config)}</div>\n API response: <div class=\"code dbdemos_block\">{json.dumps(exception.response)}</div>\"\"\")\n\n def display_token_error(self, exception: TokenException, demo_name: str):\n self.display_error(exception, f\"\"\"dbdemos couldn't not programmatically acquire a pat token to call the API to install the demo.<br/>\n This can be due to the following:\n <ul><li>Legacy cluster being used with admin protection for \"<a href=\"https://docs.databricks.com/administration-guide/account-settings/no-isolation-shared.html\">No isolation shared</a>\" (account level setting).\"</li>\n <li>Restriction on Shared cluster</li></ul>\n Please use a cluster with Access mode set to Isolation, Single User and re-run your dbdemos command.<br/>\n Alternatively, you can use a PAT token in the install:\n <div class=\"code dbdemos_block\">#Get pat token from the UI and save it as a token.<br/>\n pat_token = dbutils.secrets.get(scope=\"my_scope\", key=\"dbdemos_token\")<br/>\n dbdemos.install('{demo_name}', pat_token=pat_token)</div><br/>\n <strong>Details</strong><br/>\n Error: <div class=\"code dbdemos_block\">{exception.message}</div>\"\"\")\n\n def display_demo_name_error(self, name, demos):\n html = \"<h2>Demos available:</h2>\"\n for cat in demos:\n html += f\"<strong>{cat}</strong>\"\n for demo in demos[cat]:\n html += f\"\"\"<div style=\"padding-left: 40px\">{demo.name}: <span class=\"path_desc\">{demo.description}</span></div>\"\"\"\n self.display_error(Exception(f\"Demo '{name}' doesn't exist\"),\n f\"\"\"This demo doesn't exist, please check your demo name and re-run the installation.<br/>\n {html} \n <br/><br/>\n To get a full demo list, please run\n <div class=\"code dbdemos_block\">dbdemos.list_demos()</div>\"\"\")\n\n def display_error(self, exception, message, raise_error = True, warning = False):\n color = \"#d18b2a\" if warning else \"#eb0707\"\n level = \"warning\" if warning else \"error\"\n error = f\"\"\"{InstallerReport.CSS_REPORT}<div class=\"dbdemos_install\">\n <h1 style=\"color: {color}\">Installation {level}: {exception}</h1> \n {message}\n </div>\"\"\"\n if self.displayHTML_available():\n from dbruntime.display import displayHTML\n displayHTML(error)\n else:\n print(error)\n if raise_error:\n raise exception\n\n def display_install_info(self, demo_conf: DemoConf, install_path, catalog: str, schema: str):\n print(f\"Installing demo {demo_conf.name} under {install_path}, please wait...\")\n print(f\"\"\"Help us improving dbdemos, share your feedback or create an issue if something isn't working: https://github.com/databricks-demos/dbdemos\"\"\")\n # -----------------------------------------\n # Update the new demo here\n # -----------------------------------------\n info = \"\"\"\n <div class=\"update_container\">\n <div class=\"update_box\">\n <img src=\"https://github.com/databricks-demos/dbdemos-resources/raw/main/icon/declarative-pipelines.jpg\" class=\"update_image\">\n <div class=\"update_title\">Discover our Spark Declarative Pipelines demo!</div>\n <p>Discover how Lakeflow SDP simplifies batch and streaming ETL with automated reliability and built-in data quality:<br><br>\n <span class=\"code\">dbdemos.install('pipeline-bike')</span>\n </p>\n </div>\n <div class=\"update_box\">\n <img src=\"https://github.com/databricks-demos/dbdemos-resources/raw/main/icon/aibi-marketing-campaign.jpg\" class=\"update_image\">\n <div class=\"update_title\">New AI Agent demo!</div>\n <p>Discover how to build, package and evaluate a multi-agent system with Databricks and MLFlow 3.0!<br><br>\n <span class=\"code\">dbdemos.install('ai-agent')</span></p>\n </div>\n </div>\n \"\"\"\n if demo_conf.custom_schema_supported:\n if not catalog:\n info += f\"\"\"This demo supports utilizing custom Unity Catalog Schema's! The default schema is {demo_conf.default_catalog}.{demo_conf.default_schema}.\n To install it somewhere else, run <div class=\"code dbdemos_block\">dbdemos.install('{demo_conf.name}', catalog='xxx', schema='xxx')</div><br/>\"\"\"\n else:\n info += f\"\"\"This demo content will be installed in the schema `{catalog}`.`{schema}`<br/>\"\"\"\n if len(demo_conf.custom_message) > 0:\n info += \"<br/>\"+demo_conf.custom_message+\"<br/>\"\n self.display_info(info, \"Installation in progress...\")\n\n def display_info(self, info: str, title: str=\"\"):\n if len(info) > 0:\n if len(title) > 0:\n title = f\"\"\"<h2 style=\"color: #4875c2\">{title}</h2>\"\"\"\n html = f\"\"\"{InstallerReport.CSS_REPORT}\n <div class=\"dbdemos_install\">{title}\n {info}\n </div>\"\"\"\n if self.displayHTML_available():\n from dbruntime.display import displayHTML\n displayHTML(html)\n else:\n print(html)\n\n def display_install_result(self, demo_name, description, title, install_path = None, notebooks = [], job_id = None, run_id = None, serverless = False, cluster_id = None, cluster_name = None, \n pipelines_ids = [], dashboards = [], workflows = [], genie_rooms = []):\n if self.displayHTML_available():\n self.display_install_result_html(demo_name, description, title, install_path, notebooks, job_id, run_id, serverless, cluster_id, cluster_name, pipelines_ids, dashboards, workflows, genie_rooms)\n else:\n self.display_install_result_console(demo_name, description, title, install_path, notebooks, job_id, run_id, serverless, cluster_id, cluster_name, pipelines_ids, dashboards, workflows, genie_rooms)\n\n def get_install_result_html(self, demo_name, description, title, install_path = None, notebooks = [], job_id = None, run_id = None, serverless = False, cluster_id = None, cluster_name = None, \n pipelines_ids = [], dashboards = [], workflows = [], genie_rooms = []):\n html = f\"\"\"{InstallerReport.CSS_REPORT}\n <div class=\"dbdemos_install\">\n <img style=\"float:right; width: 180px; padding: 10px\" src=\"https://github.com/databricks-demos/dbdemos-resources/raw/main/icon/{demo_name}.jpg\" />\n <h1>Your demo: '{title}' is ready!</h1>\n <i>{description}</i><br/><br/>\n \"\"\"\n \n if not serverless and cluster_id is not None:\n cluster_section = f\"\"\"\n <h2>Interactive cluster for the demo:</h2>\n <a href=\"{self.workspace_url}/#setting/clusters/{cluster_id}/configuration\">{cluster_name}</a>. You can refresh your demo cluster with:\n <div class=\"code\">\n dbdemos.create_cluster('{demo_name}')\n </div>\"\"\"\n cluster_instruction = f' using the cluster <a href=\"{self.workspace_url}/#setting/clusters/{cluster_id}/configuration\">{cluster_name}</a>'\n else:\n cluster_section = \"\"\n cluster_instruction = \"\"\n if len(notebooks) > 0:\n first = list(filter(lambda n: \"/\" not in n.get_clean_path(), notebooks))\n if len(first) == 0:\n first = list(filter(lambda n: \"resources\" not in n.get_clean_path(), notebooks))\n first.sort(key=lambda n: n.get_clean_path())\n html += f\"\"\"Start with the first notebook {InstallerReport.NOTEBOOK_SVG} <a href=\"{self.workspace_url}/#workspace{install_path}/{demo_name}/{first[0].get_clean_path()}\">{demo_name}/{first[0].get_clean_path()}</a>{cluster_instruction}\\n\"\"\"\n html += \"\"\"<h2>Notebook installed:</h2><div class=\"container_dbdemos\">\\n \"\"\"\n if len(pipelines_ids)>0 or len(dashboards)>0:\n html += \"\"\"<div style=\"float: right; width: 300px\">\"\"\"\n if len(pipelines_ids)>0:\n html += f\"\"\"<img src=\"https://github.com/databricks-demos/dbdemos-resources/blob/main/icon/{demo_name}-dlt-0.png?raw=true\" style=\"width: 300px; margin-bottom: 10px\">\"\"\"\n if len(dashboards)>0:\n html += f\"\"\"<img src=\"https://github.com/databricks-demos/dbdemos-resources/blob/main/icon/{demo_name}-dashboard-0.png?raw=true\" style=\"width: 300px\">\"\"\"\n html += \"\"\"</div>\"\"\"\n previous_folder = \"\"\n for n in notebooks:\n if \"_resources\" not in n.get_clean_path():\n #from pathlib import Path\n parts = Path(n.get_clean_path()).parts\n path = n.get_clean_path()\n if len(parts) > 1 :\n path = str(Path(*parts[1:]))\n if previous_folder != parts[0]:\n div_class = \"subfolder\"\n html += f\"\"\"<div class=\"notebook\">{InstallerReport.FOLDER_SVG} {parts[0]}</div>\\n\"\"\"\n previous_folder = parts[0]\n elif len(parts) == 1:\n div_class = \"\"\n html += f\"\"\"<div class=\"notebook {div_class}\">{InstallerReport.NOTEBOOK_SVG} <a href=\"{self.workspace_url}/#workspace{install_path}/{demo_name}/{n.get_clean_path()}\">{path}</a>: <span class=\"path_desc\">{n.title}</span></div>\"\"\"\n html += \"\"\"</div>\"\"\"\n if len(pipelines_ids) > 0:\n html += f\"\"\"<h2>Spark Declarative Pipelines</h2><ul>\"\"\"\n for p in pipelines_ids:\n if 'error' in p:\n html += f\"\"\"<li>{p['name']}: Installation error</li>\"\"\"\n else:\n html += f\"\"\"<li><a href=\"{self.workspace_url}/#joblist/pipelines/{p['uid']}\">{p['name']}</a></li>\"\"\"\n html +=\"</ul>\"\n if len(dashboards) > 0:\n html += f\"\"\"<h2>Databricks AI/BI Dashboards</h2><div class=\"container_dbdemos\">\"\"\"\n for d in dashboards:\n if \"error\" in d:\n error_already_installed = \"\"\n html += f\"\"\"<div>ERROR INSTALLING DASHBOARD {d['name']}: {d['error']}. The Import/Export API must be enabled.{error_already_installed}</div>\"\"\"\n else:\n html += f\"\"\"<div>{InstallerReport.DASHBOARD_SVG} <a href=\"{self.workspace_url}/sql/dashboardsv3/{d['uid']}\">{d['name']}</a></div>\"\"\"\n html +=\"</div>\"\n if len(genie_rooms) > 0:\n html += f\"\"\"<h2>Databricks AI/BI Genie Spaces: Talk to your data</h2><div class=\"container_dbdemos\">\"\"\"\n for g in genie_rooms:\n html += f\"\"\"<div>{InstallerReport.GENIE_SVG} <a href=\"{self.workspace_url}/genie/rooms/{g['uid']}\">{g['name']}</a></div>\"\"\"\n html +=\"</div>\"\n if len(workflows) > 0:\n html += f\"\"\"<h2>Workflows</h2><ul>\"\"\"\n for w in workflows:\n if w['run_id'] is not None:\n html += f\"\"\"We created and started a <a href=\"{self.workspace_url}/#job/{w['uid']}/run/{w['run_id']}\">workflow</a> as part of your demo !\"\"\"\n else:\n html += f\"\"\"We created a <a href=\"{self.workspace_url}/#job/{w['uid']}\">workflow</a> as part of your demo !\"\"\"\n html +=\"</ul>\"\n if job_id is not None:\n html += f\"\"\"<h2>Initialization job started</h2>\n <div style=\"background-color: #e8f1ff; padding: 10px\">\n We started a <a href=\"{self.workspace_url}/#job/{job_id}/run/{run_id}\">job to initialize your demo data</a> (for DBSQL Dashboards & Delta Live Table). \n <strong>Please wait for the job completion to be able to access the dataset & dashboards...</strong>\n </div>\"\"\"\n html += cluster_section+\"</div>\"\n return html\n\n def display_install_result_html(self, demo_name, description, title, install_path = None, notebooks = [], job_id = None, run_id = None, serverless = False, cluster_id = None, cluster_name = None, \n pipelines_ids = [], dashboards = [], workflows = [], genie_rooms = []):\n from dbruntime.display import displayHTML\n html = self.get_install_result_html(demo_name, description, title, install_path, notebooks, job_id, run_id, serverless, cluster_id, cluster_name, pipelines_ids, dashboards, workflows, genie_rooms)\n displayHTML(html)\n\n def display_install_result_console(self, demo_name, description, title, install_path = None, notebooks = [], job_id = None, run_id = None, serverless = False, cluster_id = None, cluster_name = None, \n pipelines_ids = [], dashboards = [], workflows = [], genie_rooms = []):\n if len(notebooks) > 0:\n print(\"----------------------------------------------------\")\n print(\"-------------- Notebook installed: -----------------\")\n for n in notebooks:\n if \"_resources\" not in n.get_clean_path():\n print(f\" - {n.title}: {self.workspace_url}/#workspace{install_path}/{demo_name}/{n.get_clean_path()}\")\n if job_id is not None:\n print(\"----------------------------------------------------\")\n print(\"--- Job initialization started (load demo data): ---\")\n print(f\" - Job run available under: {self.workspace_url}/#job/{job_id}/run/{run_id}\")\n if not serverless and cluster_id is not None:\n print(\"----------------------------------------------------\")\n print(\"------------ Demo interactive cluster: -------------\")\n print(f\" - {cluster_name}: {self.workspace_url}/#setting/clusters/{cluster_id}/configuration\")\n cluster_instruction = f\" using the cluster {cluster_name}\"\n else:\n cluster_instruction = \"\"\n if len(pipelines_ids) > 0:\n print(\"----------------------------------------------------\")\n print(\"------------ Spark Declarative Pipelines available: -----------\")\n for p in pipelines_ids:\n if 'error' in p:\n print(f\" - {p['name']}: Installation error\")\n else:\n print(f\" - {p['name']}: {self.workspace_url}/#joblist/pipelines/{p['uid']}\")\n if len(dashboards) > 0:\n print(\"----------------------------------------------------\")\n print(\"------------- DBSQL Dashboard available: -----------\")\n for d in dashboards:\n error_already_installed = \"\"\n if \"error\" in d:\n print(f\" - ERROR INSTALLING DASHBOARD {d['name']}: {d['error']}. The Import/Export API must be enabled.{error_already_installed}\")\n else:\n print(f\" - {d['name']}: {self.workspace_url}/sql/dashboardsv3/{d['uid']}\")\n if len(genie_rooms) > 0:\n print(\"----------------------------------------------------\")\n print(\"------------- Genie Spaces available: -----------\")\n for g in genie_rooms:\n print(f\" - {g['name']}: {self.workspace_url}/genie/rooms/{g['uid']}\")\n if len(workflows) > 0:\n print(\"----------------------------------------------------\")\n print(\"-------------------- Workflows: --------------------\")\n for w in workflows:\n if w['run_id'] is not None:\n print(f\"\"\"We created and started a workflow as part of your demo: {self.workspace_url}/#job/{w['uid']}/run/{w['run_id']}\"\"\")\n else:\n print(f\"\"\"We created a workflow as part of your demo: {self.workspace_url}/#job/{w['uid']}/tasks\"\"\")\n print(\"----------------------------------------------------\")\n print(f\"Your demo {title} is ready! \")\n first = list(filter(lambda n: \"/\" not in n.get_clean_path(), notebooks))\n if len(first) > 0:\n first.sort(key=lambda n: n.get_clean_path())\n print(f\"Start with the first notebook {demo_name}/{first[0].get_clean_path()}{cluster_instruction}: {self.workspace_url}/#workspace{install_path}/{demo_name}/{first[0].get_clean_path()}.\")\n\n def display_schema_creation_error(self, exception: Exception, demo_conf: DemoConf):\n self.display_error(exception, f\"\"\"Can't create catalog/schema `{demo_conf.catalog}`.`{demo_conf.schema}`. <br/>\n Please verify you have the proper permissions to create catalogs and schemas, or install the demo in another location:<br/>\n <div class=\"code dbdemos_block\">dbdemos.install('{demo_conf.name}', catalog='{demo_conf.catalog}', schema='{demo_conf.schema}', create_schema=True)</div><br/>\n Error details: {str(exception)}\"\"\")\n\n def display_schema_not_found_error(self, exception: Exception, demo_conf: DemoConf):\n self.display_error(exception, f\"\"\"The catalog/schema `{demo_conf.catalog}`.`{demo_conf.schema}` doesn't exist. <br/>\n Either create it manually, or set create_schema=True to let dbdemos create it for you:<br/>\n <div class=\"code dbdemos_block\">dbdemos.install('{demo_conf.name}', catalog='{demo_conf.catalog}', schema='{demo_conf.schema}', create_schema=True)</div><br/>\n Error details: {str(exception)}\"\"\")\n" }, { "path": "dbdemos/installer_repos.py", "content": "from .conf import DemoConf\n\nfrom typing import TYPE_CHECKING\nif TYPE_CHECKING:\n from .installer import Installer\n\n\nclass InstallerRepo:\n def __init__(self, installer: 'Installer'):\n self.installer = installer\n self.db = installer.db\n\n #Start the init job if it exists\n def install_repos(self, demo_conf: DemoConf, debug = False):\n repos = []\n if len(demo_conf.repos) > 0:\n if debug:\n print(f\" Loading demo repos\")\n #We have an init jon\n for repo in demo_conf.repos:\n repo_id = self.update_or_create_repo(repo)\n #returns the path as id as that's what we'll change in the URL (#/workspace/<the/repo/path>/README.md)\n # see notebook_parser.replace_dynamic_links_repo for more details.\n repos.append({\"uid\": repo['path'], \"id\": repo['id'], \"repo_id\": repo_id})\n return repos\n\n def get_repos(self, path_prefix):\n assert len(path_prefix) > 2\n return self.db.get(\"/2.0/repos\", {\"path_prefix\": path_prefix})\n\n\n def update_or_create_repo(self, repo):\n repo_path = repo['path']\n folder = repo_path[:repo_path.rfind('/')]\n r = self.get_repos(repo_path)\n #No repo, clone it\n if 'repos' not in r:\n if repo_path.endswith('/'):\n repo_path = repo_path[:-1]\n f = self.installer.db.post(\"/2.0/workspace/mkdirs\", json = { \"path\": folder})\n data = {\n \"url\": repo['url'],\n \"branch\": repo['branch'],\n \"provider\": repo['provider'],\n \"path\": repo_path\n }\n r = self.db.post(\"/2.0/repos\", data)\n if 'error_code' in r:\n error = f\"ERROR - Could not clone the repo {repo['url']} under {repo_path}: {r}\"\n raise Exception(error)\n r = self.get_repos(repo_path)\n try:\n return r['repos'][0][\"id\"]\n except:\n raise Exception(f\"couldn't properly create the repository {data} - {r}\")\n\n\n repo_id = r['repos'][0][\"id\"]\n r = self.db.patch(f\"/2.0/repos/{repo_id}\", {\"branch\": repo[\"branch\"]})\n if 'error_code' in r and r['error_code'] == 'GIT_CONFLICT':\n print(f\"Error during repo pull {repo_path}: Git conflict. Please resolve manual conflict to get the last version.\")\n return repo_id\n" }, { "path": "dbdemos/installer_workflows.py", "content": "from .conf import DemoConf, merge_dict, ConfTemplate\nimport json\nimport time\n\nfrom .exceptions.dbdemos_exception import WorkflowException\nfrom typing import TYPE_CHECKING\nif TYPE_CHECKING:\n from .installer import Installer\n\n\nclass InstallerWorkflow:\n def __init__(self, installer: 'Installer'):\n self.installer = installer\n self.db = installer.db\n\n #Start the init job if it exists\n def install_workflows(self, demo_conf: DemoConf, use_cluster_id = None, warehouse_name: str = None, serverless = False, debug = False):\n workflows = []\n if len(demo_conf.workflows) > 0:\n if debug:\n print(f\" Loading demo workflows\")\n # We have an init jon\n for workflow in demo_conf.workflows:\n definition = workflow['definition']\n job_name = definition[\"settings\"][\"name\"]\n # add cloud specific setup\n job_id, run_id = self.create_or_replace_job(demo_conf, definition, job_name, workflow['start_on_install'], use_cluster_id, warehouse_name, serverless, debug)\n # print(f\" Demo workflow available: {self.installer.db.conf.workspace_url}/#job/{job_id}/tasks\")\n workflows.append({\"uid\": job_id, \"run_id\": run_id, \"id\": workflow['id']})\n return workflows\n\n #create or update the init job if it exists\n def create_demo_init_job(self, demo_conf: DemoConf, use_cluster_id = None, warehouse_name: str = None, serverless = False, debug = False):\n if \"settings\" in demo_conf.init_job:\n job_name = demo_conf.init_job[\"settings\"][\"name\"]\n if debug:\n print(f\" Searching for existing demo initialisation job {job_name}\")\n #We have an init json\n job_id, run_id = self.create_or_replace_job(demo_conf, demo_conf.init_job, job_name, False, use_cluster_id, warehouse_name, serverless, debug)\n return {\"uid\": job_id, \"run_id\": run_id, \"id\": \"init-job\"}\n return {\"uid\": None, \"run_id\": None, \"id\": None}\n\n #Start the init job if it exists.\n def start_demo_init_job(self, demo_conf: DemoConf, init_job, debug = False):\n if init_job['uid'] is not None:\n j = self.installer.db.post(\"2.1/jobs/run-now\", {\"job_id\": init_job['uid']})\n if debug:\n print(f'Starting init job {init_job}: {j}')\n if \"error_code\" in j:\n self.installer.report.display_workflow_error(WorkflowException(\"Can't start the workflow\", {\"job_id\": init_job['uid']}, init_job, j), demo_conf.name)\n init_job['run_id'] = j['run_id']\n return j['run_id']\n\n def create_or_replace_job(self, demo_conf: DemoConf, definition: dict, job_name: str, run_now: bool, use_cluster_id = None, warehouse_name: str = None, serverless = False, debug = False):\n cloud = self.installer.get_current_cloud()\n conf_template = ConfTemplate(self.db.conf.username, demo_conf.name)\n cluster_conf = self.installer.get_resource(\"resources/default_cluster_job_config.json\")\n cluster_conf = json.loads(conf_template.replace_template_key(cluster_conf))\n cluster_conf_cloud = json.loads(self.installer.get_resource(f\"resources/default_cluster_config-{cloud}.json\"))\n merge_dict(cluster_conf, cluster_conf_cloud)\n definition = self.replace_warehouse_id(demo_conf, definition, warehouse_name)\n #Use a given interactive cluster, change the job setting accordingly.\n if use_cluster_id is not None:\n del definition[\"settings\"][\"job_clusters\"]\n for task in definition[\"settings\"][\"tasks\"]:\n if \"job_cluster_key\" in task:\n del task[\"job_cluster_key\"]\n task[\"existing_cluster_id\"] = use_cluster_id\n #otherwise set the job properties based on the definition & add pool for our dev workspace.\n else:\n for cluster in definition[\"settings\"][\"job_clusters\"]:\n if \"new_cluster\" in cluster:\n merge_dict(cluster[\"new_cluster\"], cluster_conf, override=False)\n #Let's make sure we add our dev pool for faster startup\n if self.db.conf.get_demo_pool() is not None:\n cluster[\"new_cluster\"][\"instance_pool_id\"] = self.db.conf.get_demo_pool()\n cluster[\"new_cluster\"].pop(\"node_type_id\", None)\n cluster[\"new_cluster\"].pop(\"enable_elastic_disk\", None) \n cluster[\"new_cluster\"].pop(\"aws_attributes\", None)\n\n # Add support for clsuter specific task\n for task in definition[\"settings\"][\"tasks\"]:\n if \"new_cluster\" in task:\n merge_dict(task[\"new_cluster\"], cluster_conf, override=False)\n\n # if we're installing from a serverless cluster, update the job to be fully serverless\n if serverless:\n environments = []\n for task in definition[\"settings\"][\"tasks\"]:\n task.pop(\"new_cluster\", None)\n task.pop(\"job_cluster_key\", None)\n task.pop(\"existing_cluster_id\", None)\n \n # Serverless doesn't support libraries. Instead, they have environements and we can link these env to each task.\n # Extract libraries if they exist and convert to environment for serverless compute.\n if \"libraries\" in task:\n env_key = \"env_\" + task[\"task_key\"]\n dependencies = []\n for lib in task[\"libraries\"]:\n if \"pypi\" in lib and \"package\" in lib[\"pypi\"]:\n dependencies.append(lib[\"pypi\"][\"package\"])\n \n if dependencies:\n environments.append({\n \"environment_key\": env_key,\n \"spec\": {\n \"client\": \"2\",\n \"dependencies\": dependencies\n }\n })\n task[\"environment_key\"] = env_key\n task.pop(\"libraries\", None)\n \n definition[\"settings\"].pop(\"job_clusters\", None)\n if environments:\n definition[\"settings\"][\"environments\"] = environments\n \n existing_job = self.installer.db.find_job(job_name)\n if existing_job is not None:\n job_id = existing_job[\"job_id\"]\n self.installer.db.post(\"/2.1/jobs/runs/cancel-all\", {\"job_id\": job_id})\n self.wait_for_run_completion(job_id, debug=debug)\n if debug:\n print(\" Updating existing job\")\n job_config = {\"job_id\": job_id, \"new_settings\": definition[\"settings\"]}\n r = self.installer.db.post(\"2.1/jobs/reset\", job_config)\n if \"error_code\" in r:\n self.installer.report.display_workflow_error(WorkflowException(\"Can't update the workflow\",\n f\"error resetting the workflow, do you have permission?.\", job_config, r), demo_conf.name)\n else:\n if debug:\n print(\" Creating a new job for demo initialization (data & table setup).\")\n r_jobs = self.installer.db.post(\"2.1/jobs/create\", definition[\"settings\"])\n if \"error_code\" in r_jobs:\n self.installer.report.display_workflow_error(WorkflowException(\"Can't create the workflow\", {}, definition[\"settings\"], r_jobs), demo_conf.name)\n job_id = r_jobs[\"job_id\"]\n if run_now:\n j = self.installer.db.post(\"2.1/jobs/run-now\", {\"job_id\": job_id})\n if \"error_code\" in j:\n self.installer.report.display_workflow_error(WorkflowException(\"Can't start the workflow\", {\"job_id\": job_id}, j), demo_conf.name)\n return job_id, j['run_id']\n return job_id, None\n\n def replace_warehouse_id(self, demo_conf: DemoConf, definition, warehouse_name: str = None):\n # Jobs need a warehouse ID. Let's replace it with the one created. TODO: should be in the template?\n if \"{{SHARED_WAREHOUSE_ID}}\" in json.dumps(definition):\n endpoint = self.installer.get_or_create_endpoint(self.db.conf.name, demo_conf, warehouse_name = warehouse_name)\n if endpoint is None:\n print(\n \"ERROR: couldn't create or get a SQL endpoint for dbdemos. Do you have permission? Your workflow won't be able to execute the task.\")\n #TODO: quick & dirty, need to improve\n definition = json.loads(json.dumps(definition).replace(\"\"\", \"warehouse_id\": \"{{SHARED_WAREHOUSE_ID}}\"}\"\"\", \"\"))\n else:\n definition = json.loads(json.dumps(definition).replace(\"{{SHARED_WAREHOUSE_ID}}\", endpoint['warehouse_id']))\n return definition\n\n def wait_for_run_completion(self, job_id, max_retry=10, debug = False):\n def is_still_running(job_id):\n runs = self.installer.db.get(\"2.1/jobs/runs/list\", {\"job_id\": job_id, \"active_only\": \"true\"})\n return \"runs\" in runs and len(runs[\"runs\"]) > 0\n i = 0\n while i <= max_retry and is_still_running(job_id):\n if debug:\n print(f\" A run is still running for job {job_id}, waiting for termination...\")\n time.sleep(5)" }, { "path": "dbdemos/job_bundler.py", "content": "from .conf import DBClient, DemoConf, Conf, ConfTemplate, merge_dict\nimport time\nimport json\nimport re\nimport base64\nfrom concurrent.futures import ThreadPoolExecutor\nimport collections\nimport requests\n\nclass JobBundler:\n def __init__(self, conf: Conf):\n self.bundles = {}\n self.staging_reseted = False\n self.head_commit_id = None\n self.conf = conf\n self.db = DBClient(conf)\n\n def get_cluster_conf(self, demo_conf: DemoConf):\n conf_template = ConfTemplate(self.conf.username, demo_conf.name)\n #default conf\n cluster_conf = json.loads(conf_template.replace_template_key(self.conf.default_cluster_template))\n #demo specific\n demo_cluster_conf = json.loads(conf_template.replace_template_key(json.dumps(demo_conf.cluster)))\n merge_dict(cluster_conf, demo_cluster_conf)\n return cluster_conf\n\n def load_bundles_conf(self):\n #if not self.staging_reseted:\n # self.reset_staging_repo()\n print(\"scanning folder for bundles...\")\n from threading import Lock\n\n bundle_set = set()\n bundle_lock = Lock()\n\n def find_conf_files(path, depth = 0):\n objects = self.db.get(\"2.0/workspace/list\", {\"path\": path})\n if \"objects\" not in objects:\n return\n objects = objects[\"objects\"]\n with ThreadPoolExecutor(max_workers=3 if depth <= 2 else 1) as executor:\n params = [(o['path'], depth+1) for o in objects if o['object_type'] == 'DIRECTORY']\n for _ in executor.map(lambda args, f=find_conf_files: f(*args), params):\n pass\n for o in objects:\n if o['object_type'] == 'NOTEBOOK' and o['path'].endswith(\"/bundle_config\"):\n with bundle_lock:\n bundle_set.add(o['path'])\n\n find_conf_files(self.conf.get_repo_path())\n with ThreadPoolExecutor(max_workers=5) as executor:\n collections.deque(executor.map(self.add_bundle_from_config, bundle_set))\n\n def add_bundle_from_config(self, bundle_config_paths):\n #Remove the /Repos/xxx from the path (we need it from the repo root)\n path = bundle_config_paths[len(self.conf.get_repo_path()):]\n path = path[:-len(\"_resources/bundle_config\")-1]\n print(f\"add bundle under {path}\")\n self.add_bundle(path)\n\n def ignore_bundle(self, bundle_path):\n print(\"WARNING --------------------------------------------------------------------------------\")\n print(f\"TEMPORARY DISABLING DEMO - {bundle_path}\")\n print(\"WARNING --------------------------------------------------------------------------------\")\n print(self.bundles)\n del self.bundles[bundle_path]\n\n def add_bundle(self, bundle_path, config_path: str = \"_resources/bundle_config\"):\n if not self.staging_reseted:\n self.reset_staging_repo()\n #Let's get the demo conf from the demo folder.\n config_path = self.conf.get_repo_path()+\"/\"+bundle_path+\"/\"+config_path\n\n file = self.db.get(\"2.0/workspace/export\", {\"path\": config_path, \"format\": \"SOURCE\", \"direct_download\": False})\n if \"content\" not in file:\n raise Exception(f\"Couldn't download bundle file: {config_path}. Check your bundle path if you added it manualy.\")\n content = base64.b64decode(file['content']).decode('utf8')\n #TODO not great, we can't download a file so need to use a notebook. We could use eval() to eval the cell but it's not super safe.\n #Need to wait for file support via api (Q3)\n lines = [l for l in content.split('\\n') if not l.startswith(\"#\") and len(l) > 0]\n j = \"\\n\".join(lines)\n j = re.sub(r'[:\\s*]True', ' true', j)\n j = re.sub(r'[:\\s*]False', ' false', j)\n \n # Handle triple-quoted multi-line strings by converting them to valid JSON strings\n pattern = r'\"\"\"([\\s\\S]*?)\"\"\"'\n \n def replace_multiline(match):\n # Get the content of the multi-line string\n string_content = match.group(1)\n # Escape newlines and other special characters for JSON\n escaped = json.dumps(string_content)\n # Return the escaped string (without the outer quotes that dumps adds)\n return escaped\n \n j = re.sub(pattern, replace_multiline, j)\n \n try:\n json_conf = json.loads(j)\n except Exception as e:\n raise Exception(f\"incorrect json setting for {config_path}: {e}. The cell should contain a python object. Please use double quote.\\n {j}\")\n demo_conf = DemoConf(bundle_path, json_conf)\n if not demo_conf.bundle:\n print(f'SKIPPING DEMO {demo_conf.name} as it is not flagged for bundle.')\n else:\n self.bundles[bundle_path] = demo_conf\n\n def reset_staging_repo(self, skip_pull = False):\n repo_path = self.conf.get_repo_path()\n print(f\"Cloning repo { self.conf.repo_url} and pulling last content under {repo_path}...\")\n repos_response = self.db.get(\"2.0/repos\", {\"path_prefix\": repo_path})\n print(repos_response)\n if len(repos_response) == 0:\n print(f\"creating repo under {repo_path}\")\n self.db.post(\"2.0/repos\", {\"url\": self.conf.repo_url, \"provider\": \"gitHub\", \"path\": repo_path})\n repos = self.db.get(\"2.0/repos\", {\"path_prefix\": repo_path})['repos']\n else:\n repos = repos_response['repos']\n if skip_pull:\n self.head_commit_id = repos[0]['head_commit_id']\n else:\n print(f\"Pulling last content from branch {self.conf.branch}\")\n r = self.db.patch(f\"2.0/repos/{repos[0]['id']}\", {\"branch\": self.conf.branch})\n if 'error_code' in r :\n raise Exception(f\"Couldn't pull the repo: {r}. Please solve conflict or delete repo before.\")\n self.head_commit_id = r['head_commit_id']\n self.staging_reseted = True\n\n def start_and_wait_bundle_jobs(self, force_execution: bool = False, skip_execution: bool = False, recreate_jobs: bool = False):\n self.create_or_update_bundle_jobs(recreate_jobs)\n self.run_bundle_jobs(force_execution, skip_execution)\n self.wait_for_bundle_jobs_completion()\n\n def create_or_update_bundle_jobs(self, recreate_jobs: bool = False):\n with ThreadPoolExecutor(max_workers=10) as executor:\n confs = [c[1] for c in self.bundles.items()]\n def create_bundle_job(demo_conf):\n demo_conf.job_id = self.create_bundle_job(demo_conf, recreate_jobs)\n collections.deque(executor.map(create_bundle_job, confs))\n\n def get_head_commit(self):\n owner, repo = self.conf.repo_url.split('/')[-2:]\n headers = {\n \"Accept\": \"application/vnd.github.v3+json\",\n \"Authorization\": f\"token {self.conf.github_token}\"\n }\n # Get the latest commit (head) from the default branch\n response = requests.get(f\"https://api.github.com/repos/{owner}/{repo}/commits/HEAD\", headers=headers)\n if response.status_code != 200:\n raise Exception(f\"Error fetching head commit: {response.status_code}, {response.text}. Please check your github token in your conf file, or get a new in https://github.com/settings/tokens.\")\n return response.json()['sha']\n \n def run_bundle_jobs(self, force_execution: bool = False, skip_execution = False):\n if not force_execution:\n head_commit = self.get_head_commit()\n with ThreadPoolExecutor(max_workers=10) as executor:\n def run_job(demo_conf):\n if demo_conf.job_id is not None:\n execute = True\n runs = self.db.get(\"2.1/jobs/runs/list\", {\"job_id\": demo_conf.job_id, 'limit': 2, 'expand_tasks': \"true\"})\n #Last run was successful\n if 'runs' in runs and len(runs['runs']) > 0:\n run = runs['runs'][0]\n if run[\"status\"][\"state\"] != \"TERMINATED\":\n run = self.cancel_job_run(demo_conf, run)\n if not force_execution:\n if \"termination_details\" not in run[\"status\"]:\n raise Exception(f\"termination_details missing, should not happen. Job {demo_conf.name} status is {run['status']}\")\n elif run[\"status\"][\"termination_details\"][\"code\"] == \"SUCCESS\":\n print(f\"Job {demo_conf.name} status is {run['status']['termination_details']}...\")\n if skip_execution:\n execute = False\n demo_conf.run_id = run['run_id']\n print(f\"skipping job execution {demo_conf.name} as it was already run and skip_execution=True.\")\n else:\n #last run was using the same commit version.\n most_recent_commit = ''\n for task in run['tasks']:\n # Safely get the commit if git_source and git_snapshot exist\n task_commit = task.get('git_source', {}).get('git_snapshot', {}).get('used_commit', '')\n if task_commit > most_recent_commit:\n most_recent_commit = task_commit\n if not self.check_if_demo_file_changed_since_commit(demo_conf, most_recent_commit, head_commit) and most_recent_commit != '':\n execute = False\n demo_conf.run_id = run['run_id']\n print(f\"skipping job execution for {demo_conf.name} as no files changed since last run. run with force_execution=true to override this check.\")\n \n if execute:\n run = self.db.post(\"2.1/jobs/run-now\", {\"job_id\": demo_conf.job_id})\n demo_conf.run_id = run[\"run_id\"]\n\n collections.deque(executor.map(run_job, [c[1] for c in self.bundles.items()]))\n\n def wait_for_bundle_jobs_completion(self):\n for _, demo_conf in self.bundles.items():\n if demo_conf.run_id is not None:\n self.wait_for_bundle_job_completion(demo_conf)\n\n def wait_for_bundle_job_completion(self, demo_conf: DemoConf):\n if demo_conf.run_id is not None:\n i = 0\n while self.db.get(\"2.1/jobs/runs/get\", {\"run_id\": demo_conf.run_id})[\"state\"][\"life_cycle_state\"] == \"RUNNING\":\n if i % 200 == 0:\n print(f\"Waiting for {demo_conf.get_job_name()} completion... \"\n f\"{self.conf.workspace_url}/#job/{demo_conf.job_id}/run/{demo_conf.run_id}\")\n i += 1\n time.sleep(5)\n\n def create_bundle_job(self, demo_conf: DemoConf, recreate_jobs: bool = False):\n notebooks_to_run = demo_conf.get_notebooks_to_run()\n if len(notebooks_to_run) == 0:\n return None\n else:\n #default job conf. TODO: add specific job setup per demo if required?\n conf_template = ConfTemplate(self.conf.username, demo_conf.name)\n default_job_conf = json.loads(conf_template.replace_template_key(self.conf.default_cluster_job_template))\n default_job_conf[\"git_source\"][\"git_url\"] = self.conf.repo_url\n default_job_conf[\"git_source\"][\"git_branch\"] = self.conf.branch\n\n cluster_conf = self.get_cluster_conf(demo_conf)\n #Update the job cluster with the specific demo setup if any\n for job_cluster in default_job_conf[\"job_clusters\"]:\n merge_dict(job_cluster[\"new_cluster\"], cluster_conf)\n job_cluster[\"new_cluster\"][\"single_user_name\"] = self.conf.run_test_as_username\n # Custom instance (ex: gpu), not i3, remove the pool\n # expected format: {\"AWS\": \"g5.4xlarge\", \"AZURE\": \"Standard_NC8as_T4_v3\", \"GCP\": \"a2-highgpu-1g\"}\n if \"node_type_id\" in job_cluster[\"new_cluster\"] and \"AWS\" in job_cluster[\"new_cluster\"][\"node_type_id\"]:\n job_cluster[\"new_cluster\"].pop('instance_pool_id', None)\n job_cluster[\"new_cluster\"][\"node_type_id\"] = job_cluster[\"new_cluster\"][\"node_type_id\"][\"AWS\"]\n job_cluster[\"new_cluster\"][\"driver_node_type_id\"] = job_cluster[\"new_cluster\"][\"driver_node_type_id\"][\"AWS\"]\n elif self.db.conf.get_demo_pool() is not None:\n job_cluster[\"new_cluster\"][\"instance_pool_id\"] = self.db.conf.get_demo_pool()\n job_cluster[\"new_cluster\"].pop(\"node_type_id\", None)\n job_cluster[\"new_cluster\"].pop(\"enable_elastic_disk\", None)\n job_cluster[\"new_cluster\"].pop(\"aws_attributes\", None)\n elif 'instance_pool_id' in job_cluster[\"new_cluster\"]:\n job_cluster[\"new_cluster\"].pop('node_type_id', None)\n job_cluster[\"new_cluster\"].pop(\"enable_elastic_disk\", None)\n job_cluster[\"new_cluster\"].pop(\"aws_attributes\", None)\n\n job_cluster[\"new_cluster\"].pop('cluster_name', None)\n job_cluster[\"new_cluster\"].pop('autotermination_minutes', None)\n if job_cluster[\"new_cluster\"][\"spark_conf\"].get(\"spark.databricks.cluster.profile\", \"\") == \"singleNode\":\n del job_cluster[\"new_cluster\"][\"autoscale\"]\n job_cluster[\"new_cluster\"][\"num_workers\"] = 0\n default_job_conf['tasks'] = []\n\n # Added for unit testing 01/13/2025. Enforcing single user to reduce\n # complexity of testing.\n default_job_conf[\"run_as\"] = {\"user_name\": self.conf.run_test_as_username}\n\n for i, notebook in enumerate(notebooks_to_run):\n task = {\n \"task_key\": f\"bundle_{demo_conf.name}_{i}\",\n \"notebook_task\": {\n \"notebook_path\": demo_conf.path+\"/\"+notebook.path,\n \"base_parameters\": {\"reset_all_data\": \"false\"},\n \"source\": \"GIT\"\n },\n \"libraries\": notebook.libraries,\n \"job_cluster_key\": default_job_conf[\"job_clusters\"][0][\"job_cluster_key\"],\n \"timeout_seconds\": 0,\n \"email_notifications\": {}}\n merge_dict(task[\"notebook_task\"][\"base_parameters\"], notebook.parameters)\n if notebook.warehouse_id:\n del task[\"job_cluster_key\"]\n task[\"notebook_task\"][\"warehouse_id\"] = notebook.warehouse_id\n if notebook.depends_on_previous:\n task[\"depends_on\"] = [{\"task_key\": f\"bundle_{demo_conf.name}_{i-1}\"}]\n default_job_conf['tasks'].append(task)\n if \"depends_on\" in default_job_conf['tasks'][0]:\n del default_job_conf['tasks'][0][\"depends_on\"]\n\n return self.create_or_update_job(demo_conf, default_job_conf, recreate_jobs)\n\n def create_or_update_job(self, demo_conf: DemoConf, job_conf: dict, recreate_jobs: bool = False):\n print(f'searching for job {job_conf[\"name\"]}')\n existing_job = self.db.find_job(job_conf[\"name\"])\n if recreate_jobs:\n self.db.post(\"2.1/jobs/delete\", {'job_id': existing_job['job_id']})\n existing_job = None\n if existing_job is not None:\n # update the job\n print(f\"test job {existing_job['job_id']} already existing for {demo_conf.name}, updating it with last config\")\n self.db.post(\"2.1/jobs/reset\", {'job_id': existing_job['job_id'], 'new_settings': job_conf})\n return existing_job['job_id']\n else:\n # create the job from scratch\n print(f\"test job doesn't exist for {demo_conf.name}, creating a new one\")\n r = self.db.post(\"2.1/jobs/create\", job_conf)\n if 'job_id' not in r:\n raise Exception(f\"Error starting the job for demo {demo_conf.name}: {r}. Please check your cluster/job setup {job_conf}\")\n return r['job_id']\n\n def check_if_demo_file_changed_since_commit(self, demo_conf: DemoConf, base_commit, last_commit = None):\n if base_commit is None or base_commit == '':\n return True\n owner, repo = self.conf.repo_url.split('/')[-2:]\n files = self.get_changed_files_since_commit(owner, repo, base_commit, last_commit)\n return any(f.startswith(demo_conf.path) for f in files)\n\n def get_changed_files_since_commit(self, owner, repo, base_commit, last_commit = None):\n headers = {\n \"Accept\": \"application/vnd.github.v3+json\",\n \"Authorization\": f\"token {self.conf.github_token}\"\n }\n if last_commit is None:\n last_commit = self.get_head_commit()\n # Compare the base commit with the latest commit\n compare_url = f\"https://api.github.com/repos/{owner}/{repo}/compare/{base_commit}...{last_commit}\"\n compare_response = requests.get(compare_url, headers=headers)\n\n if compare_response.status_code == 200:\n data = compare_response.json()\n files = data['files']\n return [file['filename'] for file in files]\n else:\n raise Exception(f\"Error fetching latest commit: {compare_response.status_code}, {compare_response.text}\")\n\n def cancel_job_run(self, demo_conf: DemoConf, run):\n \"\"\"Cancel a running job and wait for termination\"\"\"\n print(f\"Job {demo_conf.name} status is {run['status']['state']}, cancelling it...\") \n self.db.post(\"2.1/jobs/runs/cancel-all\", {\"job_id\": demo_conf.job_id})\n time.sleep(5)\n while True:\n run = self.db.get(\"2.1/jobs/runs/get\", {\"run_id\": run['run_id']})\n if run[\"status\"][\"state\"] == \"TERMINATED\":\n break\n print(f\"Waiting for job {demo_conf.name} to be terminated after cancellation...\") \n time.sleep(10)\n return run\n\n \n" }, { "path": "dbdemos/notebook_parser.py", "content": "from dbdemos.conf import DemoConf\n\nfrom .tracker import Tracker\nimport urllib\nimport re\nimport base64\nimport json\n\nclass NotebookParser:\n\n def __init__(self, html):\n self.html = html\n self.raw_content, self.content = self.get_notebook_content(html)\n\n def get_notebook_content(self, html):\n match = re.search(r'__DATABRICKS_NOTEBOOK_MODEL = \\'(.*?)\\'', html)\n raw_content = match.group(1)\n content = base64.b64decode(raw_content).decode('utf-8')\n content = urllib.parse.unquote(content)\n return raw_content, content\n\n def get_html(self):\n content = json.loads(self.content)\n #force the position to avoid bug during import\n for i in range(len(content[\"commands\"])):\n content[\"commands\"][i]['position'] = i\n content = json.dumps(content)\n content = urllib.parse.quote(content, safe=\"()*''\")\n return self.html.replace(self.raw_content, base64.b64encode(content.encode('utf-8')).decode('utf-8'))\n\n def contains(self, str):\n return str in self.content\n\n def remove_static_settings(self):\n #Remove the static settings tags are it's too big & unecessary to repeat in each notebook.\n self.html = re.sub(\"\"\"<script>\\s?window\\.__STATIC_SETTINGS__.*</script>\"\"\", \"\", self.html)\n\n def set_tracker_tag(self, org_id, uid, category, demo_name, notebook, username):\n #Replace internal tags with dbdemos\n if Tracker.enable_tracker:\n tracker = Tracker(org_id, uid, username)\n #Our demos in the repo already have tags used when we clone the notebook directly.\n #We need to update the tracker with the demo configuration & dbdemos setup.\n tracker_url = tracker.get_track_url(category, demo_name, \"VIEW\", notebook)\n r = r\"\"\"(<img\\s*width=\\\\?\"1px\\\\?\"\\s*src=\\\\?\")(https:\\/\\/ppxrzfxige\\.execute-api\\.us-west-2\\.amazonaws\\.com\\/v1\\/analytics.*?)(\\\\?\"\\s?\\/?>)\"\"\"\n self.content = re.sub(r, rf'\\1{tracker_url}\\3', self.content)\n\n #old legacy tracker, to be migrted & emoved\n r = r\"\"\"(<img\\s*width=\\\\?\"1px\\\\?\"\\s*src=\\\\?\")(https:\\/\\/www\\.google-analytics\\.com\\/collect.*?)(\\\\?\"\\s?\\/?>)\"\"\"\n self.content = re.sub(r, rf'\\1{tracker_url}\\3', self.content)\n else:\n #Remove all the tracker from the notebook\n self.replace_in_notebook(r\"\"\"<img\\s*width=\\\\?\"1px\\\\?\"\\s*src=\\\\?\"https:\\/\\/www\\.google-analytics\\.com\\/collect.*?\\\\?\"\\s?\\/?>\"\"\", \"\", True)\n self.replace_in_notebook(r\"\"\"<img\\s*width=\\\\?\"1px\\\\?\"\\s*src=\\\\?\"https:\\/\\/ppxrzfxige\\.execute-api\\.us-west-2\\.amazonaws\\.com\\/v1\\/analytics.*?\\\\?\"\\s?\\/?>\"\"\", \"\", True)\n\n def remove_uncomment_tag(self):\n self.replace_in_notebook('[#-]{1,2}\\s*UNCOMMENT_FOR_DEMO ?', '', True)\n\n ##Remove the __build to avoid catalog conflict during build vs test\n # TODO: improve build and get a separate metastore for tests vs build.\n def remove_dbdemos_build(self):\n self.replace_in_notebook('dbdemos__build', 'dbdemos')\n\n def remove_robots_meta(self):\n #Drop the noindex tag\n self.html = self.html.replace('<meta name=\"robots\" content=\"nofollow, noindex\">', '')\n\n def add_cell_as_html_for_seo(self):\n #Add div as hidden HTML for SEO to capture the main information in the page.\n def md_to_html(text):\n if text.startswith('%md-sandbox'):\n text = text[len('%md-sandbox'):]\n if text.startswith('%md'):\n text = text[len('%md'):]\n #quick translation to html for seo\n for i in reversed(range(1,6)):\n tag = \"#\"*i\n text = re.sub(rf'\\s*{tag}\\s*(.*)', rf'<h{i}>\\1</h{i}>', text)\n text = text.replace('\\n', '<br/>')\n return text\n #Drop the noindex tag\n content = json.loads(self.content)\n html = \"\"\n for c in content[\"commands\"]:\n if c['command'].startswith('%md'):\n html += '<div>'+md_to_html(c['command'])+'</div>'\n if len(html) > 0:\n self.html = self.html.replace('<body>', f'''<body><div id='no_js_render' style='display: none'>{html}</div>''')\n self.html = self.html.replace('<script>', \"<script>window.addEventListener('load', function(event) { \"\n \"if (/bot|google|baidu|bing|msn|teoma|slurp|yandex/i.test(navigator.userAgent)) {\"\n \"document.getElementById('no_js_render').style.display = 'block';\"\n \"};\"\n \"});\", 1)\n\n @staticmethod\n def _replace_with_optional_escaped_quotes(content: str, old: str, new: str) -> str:\n \"\"\"\n Helper to replace text handling both escaped and unescaped quotes.\n In JSON content, quotes are escaped as \\\", but in parsed content they're not.\n We handle both by trying replacements with escaped quotes first, then unescaped.\n This is much faster than using regex.\n \"\"\"\n # Try with escaped quotes first (JSON format: \\\")\n old_escaped = old.replace('\"', '\\\\\"')\n new_escaped = new.replace('\"', '\\\\\"')\n content = content.replace(old_escaped, new_escaped)\n\n # Then try with unescaped quotes (plain text format: \")\n content = content.replace(old, new)\n\n return content\n\n @staticmethod\n def replace_schema_in_content(content: str, demo_conf: DemoConf) -> str:\n \"\"\"\n Static method to replace schema/catalog references in any content string.\n Used for both notebook content and FILE object types.\n \"\"\"\n #main__build is used during the build process to avoid collision with default main.\n # #main_build is used because agent don't support __ in their catalog name - TODO should improve this and move everything to main_build\n content = NotebookParser._replace_with_optional_escaped_quotes(content, 'catalog = \"main__build\"', 'catalog = \"main\"')\n content = NotebookParser._replace_with_optional_escaped_quotes(content, 'catalog = \"main_build\"', 'catalog = \"main\"')\n content = content.replace(f'main__build.{demo_conf.default_schema}', f'main.{demo_conf.default_schema}')\n content = content.replace(f'main_build.{demo_conf.default_schema}', f'main.{demo_conf.default_schema}')\n content = content.replace('Volumes/main__build', 'Volumes/main')\n content = content.replace('Volumes/main_build', 'Volumes/main')\n\n #TODO we need to unify this across all demos.\n if demo_conf.custom_schema_supported:\n content = re.sub(r\"\\$catalog=[0-9a-z_]*\\s{1,3}\\$schema=[0-9a-z_]*\", f\"$catalog={demo_conf.catalog} $schema={demo_conf.schema}\", content)\n content = re.sub(r\"\\$catalog=[0-9a-z_]*\\s{1,3}\\$db=[0-9a-z_]*\", f\"$catalog={demo_conf.catalog} $db={demo_conf.schema}\", content)\n content = content.replace(f\"{demo_conf.default_catalog}.{demo_conf.default_schema}\", f\"{demo_conf.catalog}.{demo_conf.schema}\")\n content = NotebookParser._replace_with_optional_escaped_quotes(content, f'dbutils.widgets.text(\"catalog\", \"{demo_conf.default_catalog}\"', f'dbutils.widgets.text(\"catalog\", \"{demo_conf.catalog}\"')\n content = NotebookParser._replace_with_optional_escaped_quotes(content, f'dbutils.widgets.text(\"schema\", \"{demo_conf.default_schema}\"', f'dbutils.widgets.text(\"schema\", \"{demo_conf.schema}\"')\n content = NotebookParser._replace_with_optional_escaped_quotes(content, f'dbutils.widgets.text(\"db\", \"{demo_conf.default_schema}\"', f'dbutils.widgets.text(\"db\", \"{demo_conf.schema}\"')\n content = content.replace(f'Volumes/{demo_conf.default_catalog}/{demo_conf.default_schema}', f'Volumes/{demo_conf.catalog}/{demo_conf.schema}')\n\n content = NotebookParser._replace_with_optional_escaped_quotes(content, f'catalog = \"{demo_conf.default_catalog}\"', f'catalog = \"{demo_conf.catalog}\"')\n content = NotebookParser._replace_with_optional_escaped_quotes(content, f'dbName = db = \"{demo_conf.default_schema}\"', f'dbName = db = \"{demo_conf.schema}\"')\n content = NotebookParser._replace_with_optional_escaped_quotes(content, f'schema = dbName = db = \"{demo_conf.default_schema}\"', f'schema = dbName = db = \"{demo_conf.schema}\"')\n content = NotebookParser._replace_with_optional_escaped_quotes(content, f'db = \"{demo_conf.default_schema}\"', f'db = \"{demo_conf.schema}\"')\n content = NotebookParser._replace_with_optional_escaped_quotes(content, f'schema = \"{demo_conf.default_schema}\"', f'schema = \"{demo_conf.schema}\"')\n content = content.replace(f'USE SCHEMA {demo_conf.default_schema}', f'USE SCHEMA {demo_conf.schema}')\n content = content.replace(f'USE CATALOG {demo_conf.default_catalog}', f'USE CATALOG {demo_conf.catalog}')\n content = content.replace(f'CREATE CATALOG IF NOT EXISTS {demo_conf.default_catalog}', f'CREATE CATALOG IF NOT EXISTS {demo_conf.catalog}')\n content = content.replace(f'CREATE SCHEMA IF NOT EXISTS {demo_conf.default_schema}', f'CREATE SCHEMA IF NOT EXISTS {demo_conf.schema}')\n\n return content\n\n def replace_schema(self, demo_conf: DemoConf):\n \"\"\"Replace schema/catalog in notebook content\"\"\"\n self.content = NotebookParser.replace_schema_in_content(self.content, demo_conf)\n\n def replace_in_notebook(self, old, new, regex = False):\n if regex:\n self.content = re.sub(old, new, self.content)\n else:\n self.content = self.content.replace(old, new)\n\n def add_extra_cell(self, cell_content, position = 1):\n command = {\n \"version\": \"CommandV1\",\n \"bindings\": {},\n \"subtype\": \"command\",\n \"commandType\": \"auto\",\n \"position\": position,\n \"command\": cell_content\n }\n content = json.loads(self.content)\n content[\"commands\"].insert(position, command)\n self.content = json.dumps(content)\n\n #as auto ml links are unique per workspace, we have to delete them\n def remove_automl_result_links(self):\n if \"display_automl_\" in self.content:\n content = json.loads(self.content)\n for c in content[\"commands\"]:\n if re.search('display_automl_[a-zA-Z]*_link', c[\"command\"]):\n if 'results' in c and c['results'] is not None and 'data' in c['results'] and c['results']['data'] is not None and len(c['results']['data']) > 0:\n contains_exp_link = len([d for d in c['results']['data'] if 'Data exploration notebook' in d['data']]) > 0\n if contains_exp_link:\n c['results']['data'] = [{'type': 'ansi', 'data': 'Please run the notebook cells to get your AutoML links (from the begining)', 'name': None, 'arguments': {}, 'addedWidgets': {}, 'removedWidgets': [], 'datasetInfos': [], 'metadata': {}}]\n self.content = json.dumps(content)\n\n\n #Will change the content to\n def change_relative_links_for_minisite(self):\n #self.replace_in_notebook(\"\"\"<a\\s*(?:target=\"_blank\")?\\s*(?:rel=\"noopener noreferrer\")?\\s*href=\"\\$\\.\\/(.*)\">\"\"\", \"\"\"<a href=\"./$1\">\"\"\", True)\n self.replace_in_notebook(\"\"\"\\]\\(\\$\\.\\/(.*?)\\)\"\"\", \"\"\"](./\\g<1>.html)\"\"\", True)\n\n\n def add_javascript_to_minisite_relative_links(self, notebook_path):\n # Add JavaScript to the HTML (not content) that intercepts link clicks\n # This is much more reliable than trying to modify the notebook content\n\n # Get the notebook's directory (remove filename)\n notebook_dir = '/'.join(notebook_path.split('/')[:-1])\n script = f\"\"\"\n <script type=\"text/javascript\">\n (function() {{\n const NOTEBOOK_PATH = '{notebook_path}';\n const NOTEBOOK_DIR = '{notebook_dir}';\n\n function resolvePath(relativePath) {{\n // If path starts with /, it's absolute from root\n if (relativePath.startsWith('/')) {{\n return relativePath.substring(1);\n }}\n\n // Otherwise, resolve relative to current notebook's directory\n if (!NOTEBOOK_DIR || NOTEBOOK_DIR === '') {{\n return relativePath;\n }}\n\n // Combine directory with relative path\n let parts = NOTEBOOK_DIR.split('/').filter(p => p !== '');\n let pathParts = relativePath.split('/').filter(p => p !== '');\n\n for (let part of pathParts) {{\n if (part === '..') {{\n parts.pop();\n }} else if (part !== '.') {{\n parts.push(part);\n }}\n }}\n\n return parts.join('/');\n }}\n\n function setupMinisiteLinks() {{\n // Find all links in the page\n const links = document.querySelectorAll('a[href]');\n\n links.forEach(function(link) {{\n const href = link.getAttribute('href');\n\n if (!href) return;\n\n // Check if link has $ (internal demo link)\n if (href.includes('$')) {{\n // All $ links are relative to the current notebook directory\n // Remove various prefixes: /$./ $./ /$. $.\n let relativePath = href;\n\n // Remove /$./\n if (relativePath.includes('/$' + './')) {{\n relativePath = relativePath.replace('/$' + './', '');\n }}\n // Remove $./\n else if (relativePath.includes('$' + './')) {{\n relativePath = relativePath.replace('$' + './', '');\n }}\n // Remove /$.\n else if (relativePath.includes('/$' + '.')) {{\n relativePath = relativePath.replace('/$' + '.', '');\n }}\n // Remove $.\n else if (relativePath.includes('$' + '.')) {{\n relativePath = relativePath.replace('$' + '.', '');\n }}\n // Just remove $\n else {{\n relativePath = relativePath.replace('$', '');\n }}\n\n // Remove only a single leading ./ if present (preserve ../ for navigation)\n if (relativePath.startsWith('./')) {{\n relativePath = relativePath.substring(2);\n }}\n\n // Always resolve against notebook directory\n let targetPath = resolvePath(relativePath) + '.html';\n\n // Remove target and rel attributes immediately\n link.removeAttribute('target');\n link.removeAttribute('rel');\n\n // Change href to # immediately to prevent navigation\n link.setAttribute('href', '#');\n\n // Store targetPath in data attribute for debugging\n link.setAttribute('data-target-path', targetPath);\n\n // Add click handler with capture phase\n link.addEventListener('click', function(e) {{\n e.preventDefault();\n e.stopPropagation();\n\n // Send message to parent window\n if (window.parent && window.parent !== window) {{\n window.parent.postMessage({{\n type: 'dbdemos-navigate',\n targetPath: targetPath\n }}, '*');\n }} else {{\n window.location.href = targetPath;\n }}\n\n return false;\n }}, true);\n }} else if (!href.startsWith('#') && !href.startsWith('http')) {{\n // Remove non-demo internal links (convert to plain text)\n const text = document.createTextNode(link.textContent);\n link.parentNode.replaceChild(text, link);\n }}\n }});\n }}\n\n // Run when DOM is ready\n if (document.readyState === 'loading') {{\n document.addEventListener('DOMContentLoaded', setupMinisiteLinks);\n }} else {{\n setupMinisiteLinks();\n }}\n\n // Also run after a short delay to catch dynamically loaded content\n setTimeout(setupMinisiteLinks, 500);\n }})();\n </script>\n \"\"\"\n\n # Insert the script before </body>\n self.html = self.html.replace('</body>', script + '</body>')\n\n #Set the environment metadata to the notebook.\n # TODO: might want to re-evaluate this once we move to ipynb format as it'll be set in the ipynb file, as metadata.\n def set_environement_metadata(self, client_version: str = \"3\"):\n content = json.loads(self.content)\n env_metadata = content.get(\"environmentMetadata\", {})\n if env_metadata is None:\n env_metadata = {}\n if (\"client\" not in env_metadata or \n env_metadata[\"client\"] is None or \n int(env_metadata[\"client\"]) < int(client_version)):\n env_metadata[\"client\"] = str(client_version)\n content[\"environmentMetadata\"] = env_metadata\n self.content = json.dumps(content)\n\n def hide_commands_and_results(self):\n #\n self.replace_in_notebook('e2-demo-tools', 'xxxx', True)\n content = json.loads(self.content)\n for c in content[\"commands\"]:\n if \"#hide_this_code\" in c[\"command\"].lower():\n c[\"hideCommandCode\"] = True\n if \"%run \" in c[\"command\"]:\n c[\"hideCommandResult\"] = True\n if \"results\" in c and c[\"results\"] is not None and \"data\" in c[\"results\"] and c[\"results\"][\"data\"] is not None and \\\n c[\"results\"][\"type\"] == \"table\" and len(c[\"results\"][\"data\"])>0 and str(c[\"results\"][\"data\"][0][0]).startswith(\"This Delta Live Tables query is syntactically valid\"):\n c[\"hideCommandResult\"] = True\n self.content = json.dumps(content)\n\n def remove_delete_cell(self):\n content = json.loads(self.content)\n content[\"commands\"] = [c for c in content[\"commands\"] if \"#dbdemos__delete_this_cell\" not in c[\"command\"].lower()]\n self.content = json.dumps(content)\n\n def replace_dynamic_links(self, items, name, link_path):\n if len(items) == 0:\n return\n matches = re.finditer(rf'<a\\s*dbdemos-{name}-id=\\\\?[\\'\"](?P<item_id>.*?)\\\\?[\\'\"]\\s*href=\\\\?[\\'\"].*?\\/?{link_path}\\/(?P<item_uid>[a-zA-Z0-9_-]*).*?>', self.content)\n for match in matches:\n item_id = match.groupdict()[\"item_id\"]\n installed = False\n for i in items:\n if i[\"id\"] == item_id:\n installed = True\n self.content = self.content.replace(match.groupdict()[\"item_uid\"], str(i['uid']))\n if not installed:\n print(f'''ERROR: couldn't find {name} with dbdemos-{name}-id={item_id}''')\n\n\n def replace_dynamic_links_workflow(self, workflows):\n \"\"\"\n Replace the links in the notebook with the workflow installed if any\n \"\"\"\n self.replace_dynamic_links(workflows, \"workflow\", \"#job\")\n\n def replace_dynamic_links_repo(self, repos):\n for r in repos:\n if r[\"uid\"].startswith(\"/\"):\n r[\"uid\"] = r[\"uid\"][1:]\n \"\"\"\n Replace the links in the notebook with the repos installed if any\n \"\"\"\n self.replace_dynamic_links(repos, \"repo\", \"#workspace\")\n\n def replace_dynamic_links_pipeline(self, pipelines_id):\n \"\"\"\n Replace the links in the notebook with the SDP pipeline installed if any\n \"\"\"\n self.replace_dynamic_links(pipelines_id, \"pipeline\", \"#joblist/pipelines\")\n\n\n def replace_dynamic_links_lakeview_dashboards(self, dashboards_id):\n \"\"\"\n Replace the links in the notebook with the Lakeview dashboard installed if any\n \"\"\"\n self.replace_dynamic_links(dashboards_id, \"dashboard\", \"/sql/dashboardsv3\")\n\n\n def replace_dynamic_links_genie(self, genie_rooms):\n \"\"\"\n Replace the links in the notebook with the Genie room installed if any\n \"\"\"\n self.replace_dynamic_links(genie_rooms, \"genie\", \"/genie/rooms\")" }, { "path": "dbdemos/packager.py", "content": "import pkg_resources\nfrom pathlib import Path\nfrom .conf import DBClient, DemoConf, Conf, DemoNotebook\nfrom .notebook_parser import NotebookParser\nimport json\nimport os\nimport re\nimport shutil\nimport base64\nfrom .job_bundler import JobBundler\nfrom concurrent.futures import ThreadPoolExecutor\nimport collections\nimport zipfile\nimport io\n\n\nclass Packager:\n DASHBOARD_IMPORT_API = \"_import_api\"\n def __init__(self, conf: Conf, jobBundler: JobBundler):\n self.db = DBClient(conf)\n self.jobBundler = jobBundler\n\n def package_all(self, iframe_root_src = \"./\"):\n def package_demo(demo_conf: DemoConf):\n self.clean_bundle(demo_conf)\n self.package_demo(demo_conf)\n if len(demo_conf.dashboards) > 0:\n self.extract_lakeview_dashboards(demo_conf)\n self.build_minisite(demo_conf, iframe_root_src)\n \n confs = [demo_conf for _, demo_conf in self.jobBundler.bundles.items()] \n with ThreadPoolExecutor(max_workers=3) as executor:\n collections.deque(executor.map(package_demo, confs))\n\n def clean_bundle(self, demo_conf: DemoConf):\n if Path(demo_conf.get_bundle_root_path()).exists():\n shutil.rmtree(demo_conf.get_bundle_root_path())\n\n\n def extract_lakeview_dashboards(self, demo_conf: DemoConf):\n for d in demo_conf.dashboards:\n repo_path = self.jobBundler.conf.get_repo_path()+\"/\"+demo_conf.path+\"/_resources/dashboards/\"+d['id']+\".lvdash.json\"\n repo_path = os.path.realpath(repo_path)\n dashboard_file = self.db.get(\"2.0/workspace/export\", {\"path\": repo_path, \"format\": \"SOURCE\", \"direct_download\": False})\n if 'error_code' in dashboard_file:\n raise Exception(f\"Couldn't find dashboard {repo_path} in repo. Check repo ID in bundle conf file and make sure the dashboard is here. \"\n f\"{dashboard_file['error_code']} - {dashboard_file['message']}\")\n dashboard_file = base64.b64decode(dashboard_file['content']).decode('utf-8')\n full_path = demo_conf.get_bundle_path()+\"/_resources/dashboards/\"+d['id']+\".lvdash.json\"\n Path(full_path[:full_path.rindex(\"/\")]).mkdir(parents=True, exist_ok=True)\n with open(full_path, \"w\") as f:\n f.write(dashboard_file)\n\n\n def process_file_content(self, file, destination_path, extension = \"\"):\n # Decode base64 content from the folder dict\n file_content = base64.b64decode(file['content'])\n with open(destination_path + extension, \"wb\") as f:\n f.write(file_content)\n\n def process_notebook_content(self, demo_conf: DemoConf, html, full_path):\n #Replace notebook content.\n parser = NotebookParser(html)\n parser.remove_uncomment_tag()\n parser.set_environement_metadata(demo_conf.env_version)\n parser.remove_dbdemos_build()\n #parser.remove_static_settings()\n parser.hide_commands_and_results()\n #Moving away from the initial 00-global-setup, remove it once migration is completed\n requires_global_setup_v2 = False\n if parser.contains(\"00-global-setup-v2\"):\n parser.replace_in_notebook('(?:\\.\\.\\/)*_resources\\/00-global-setup-v2', './00-global-setup-v2', True)\n requires_global_setup_v2 = True\n elif parser.contains(\"00-global-setup\"):\n raise Exception(\"00-global-setup is deprecated. Please use 00-global-setup-v2 instead.\")\n with open(full_path, \"w\") as f:\n f.write(parser.get_html())\n return requires_global_setup_v2\n\n def package_demo(self, demo_conf: DemoConf):\n print(f\"packaging demo {demo_conf.name} ({demo_conf.path})\")\n if len(demo_conf.get_notebooks_to_publish()) > 0 and not self.jobBundler.staging_reseted:\n self.jobBundler.reset_staging_repo()\n if len(demo_conf.get_notebooks_to_run()) > 0:\n run = self.db.get(\"2.1/jobs/runs/get\", {\"run_id\": demo_conf.run_id, \"include_history\": False})\n if 'state' not in run:\n raise Exception(f\"Can't get the last job {self.db.conf.workspace_url}/#job/{demo_conf.job_id}/run/{demo_conf.run_id} state for demo {demo_conf.name}: {run}\")\n if run['state']['result_state'] != 'SUCCESS':\n raise Exception(f\"last job {self.db.conf.workspace_url}/#job/{demo_conf.job_id}/run/{demo_conf.run_id} failed for demo {demo_conf.name}. Can't package the demo. {run['state']}\")\n\n def download_notebook_html(notebook: DemoNotebook):\n full_path = demo_conf.get_bundle_path()+\"/\"+notebook.get_clean_path()\n print(f\"downloading {notebook.path} to {full_path}\")\n Path(full_path[:full_path.rindex(\"/\")]).mkdir(parents=True, exist_ok=True)\n if not notebook.pre_run:\n repo_path = self.jobBundler.conf.get_repo_path()+\"/\"+demo_conf.path+\"/\"+notebook.path\n repo_path = os.path.realpath(repo_path)\n #print(f\"downloading from repo {repo_path}\")\n status = self.db.get(\"2.0/workspace/get-status\", {\"path\": repo_path})\n if 'error_code' in status:\n raise Exception(f\"Couldn't find file {repo_path} in workspace. Check notebook path in bundle conf file. {status['error_code']} - {status['message']}\")\n #We add the type of the object in the conf to know how to load it back.\n demo_conf.update_notebook_object_type(notebook, status['object_type'])\n if status['object_type'] == 'NOTEBOOK':\n file = self.db.get(\"2.0/workspace/export\", {\"path\": repo_path, \"format\": \"HTML\", \"direct_download\": False})\n if 'error_code' in file:\n raise Exception(f\"Couldn't find file {repo_path} in workspace. Check notebook path in bundle conf file. {file['error_code']} - {file['message']}\")\n html = base64.b64decode(file['content']).decode('utf-8')\n return self.process_notebook_content(demo_conf, html, full_path+\".html\")\n elif status['object_type'] == 'DIRECTORY':\n folder = self.db.get(\"2.0/workspace/export\", {\"path\": repo_path, \"format\": \"AUTO\", \"direct_download\": True})\n return self.process_file_content(folder, full_path, \".zip\")\n elif status['object_type'] == 'FILE':\n file = self.db.get(\"2.0/workspace/export\", {\"path\": repo_path, \"format\": \"AUTO\", \"direct_download\": True})\n return self.process_file_content(file, full_path)\n else:\n raise Exception(f\"Unsupported object type {status['object_type']} for {repo_path}\")\n else:\n tasks = [t for t in run['tasks'] if t['notebook_task']['notebook_path'].endswith(notebook.get_clean_path())]\n if len(tasks) == 0:\n raise Exception(f\"couldn't find task for notebook {notebook.path}. Please re-run the job & make sure the stating git repo is synch / reseted.\")\n #print(f\"Exporting notebook from job run {tasks[0]['run_id']}\")\n notebook_result = self.db.get(\"2.1/jobs/runs/export\", {'run_id': tasks[0]['run_id'], 'views_to_export': 'ALL'})\n if \"views\" not in notebook_result:\n raise Exception(f\"couldn't get notebook for run {tasks[0]['run_id']} - {notebook.path}. {demo_conf.name}. You probably did a run repair. Please re run the job. - {notebook_result}\")\n html = notebook_result[\"views\"][0][\"content\"]\n return self.process_notebook_content(demo_conf, html, full_path+\".html\")\n \n\n requires_global_setup_v2 = False\n \n # Process notebooks in parallel with max 5 workers\n with ThreadPoolExecutor(max_workers=10) as executor:\n # Submit all notebooks for processing and collect futures\n futures = [executor.submit(download_notebook_html, notebook) for notebook in demo_conf.notebooks]\n \n # Process results as they complete\n for future in futures:\n rv1 = future.result()\n if rv1:\n requires_global_setup_v2 = True\n\n #Add the global notebook if required\n if requires_global_setup_v2:\n init_notebook = DemoNotebook(\"_resources/00-global-setup-v2\", \"Global init\", \"Global init\")\n demo_conf.add_notebook(init_notebook)\n file = self.db.get(\"2.0/workspace/export\", {\"path\": self.jobBundler.conf.get_repo_path() +\"/\"+ init_notebook.path, \"format\": \"HTML\", \"direct_download\": False})\n if 'error_code' in file:\n raise Exception(f\"Couldn't find file '{self.jobBundler.conf.get_repo_path()}/{init_notebook.path}' in workspace. Check notebook path in bundle conf file. {file['error_code']} - {file['message']}\")\n html = base64.b64decode(file['content']).decode('utf-8')\n with open(demo_conf.get_bundle_path() + \"/\" + init_notebook.path+\".html\", \"w\") as f:\n f.write(html)\n\n def get_file_icon_svg(self, file_path: str) -> str:\n \"\"\"\n Get the appropriate SVG icon based on file type\n\n Args:\n file_path: Path to the file\n\n Returns:\n SVG icon as HTML string\n \"\"\"\n # Notebook icon (for .html notebook files)\n notebook_icon = '''<svg class=\"file-icon\" xmlns=\"http://www.w3.org/2000/svg\" width=\"1em\" height=\"1em\" fill=\"none\" viewBox=\"0 0 16 16\" aria-hidden=\"true\" focusable=\"false\"><path fill=\"currentColor\" fill-rule=\"evenodd\" d=\"M3 1.75A.75.75 0 0 1 3.75 1h10.5a.75.75 0 0 1 .75.75v12.5a.75.75 0 0 1-.75.75H3.75a.75.75 0 0 1-.75-.75V12.5H1V11h2V8.75H1v-1.5h2V5H1V3.5h2zm1.5.75v11H6v-11zm3 0v11h6v-11z\" clip-rule=\"evenodd\"></path></svg>'''\n\n # Generic file icon (for .py, .sql, and other code files)\n file_icon = '''<svg class=\"file-icon\" xmlns=\"http://www.w3.org/2000/svg\" width=\"1em\" height=\"1em\" fill=\"none\" viewBox=\"0 0 16 16\" aria-hidden=\"true\" focusable=\"false\"><path fill=\"currentColor\" fill-rule=\"evenodd\" d=\"M2 1.75A.75.75 0 0 1 2.75 1h6a.75.75 0 0 1 .53.22l4.5 4.5c.141.14.22.331.22.53v9a.75.75 0 0 1-.75.75H2.75a.75.75 0 0 1-.75-.75zm1.5.75v12h9V7H8.75A.75.75 0 0 1 8 6.25V2.5zm6 1.06 1.94 1.94H9.5z\" clip-rule=\"evenodd\"></path></svg>'''\n\n if file_path.endswith(('.py', '.sql')):\n return file_icon\n else:\n return notebook_icon\n\n def build_tree_structure(self, notebooks_to_publish):\n \"\"\"\n Build a hierarchical tree structure from notebook paths\n\n Returns a nested dict representing the folder/file hierarchy\n \"\"\"\n tree = {}\n\n for notebook in notebooks_to_publish:\n parts = notebook.get_clean_path().split('/')\n current = tree\n\n # Navigate/create the folder structure\n for i, part in enumerate(parts[:-1]):\n if part not in current:\n current[part] = {'__type__': 'folder', '__children__': {}}\n current = current[part]['__children__']\n\n # Add the file at the end\n filename = parts[-1]\n current[filename] = {\n '__type__': 'file',\n '__notebook__': notebook,\n '__path__': notebook.get_clean_path()\n }\n\n return tree\n\n def render_tree_html(self, tree, iframe_root_src=\"./\", level=0):\n \"\"\"\n Recursively render the tree structure as HTML with CSS-based tree lines\n\n Args:\n tree: The tree structure dict\n iframe_root_src: Root path for iframe sources\n level: Current depth level\n \"\"\"\n html = \"\"\n items = sorted(tree.items(), key=lambda x: (x[1].get('__type__') != 'folder', x[0]))\n\n for idx, (name, node) in enumerate(items):\n is_last = (idx == len(items) - 1)\n\n if node['__type__'] == 'folder':\n # Render folder\n folder_id = f\"folder_{level}_{idx}_{name.replace(' ', '_')}\"\n\n html += f'''\n <div class=\"tree-item tree-folder {'tree-last' if is_last else ''}\">\n <div class=\"tree-item-row folder-row expanded\" data-folder-id=\"{folder_id}\" onclick=\"toggleFolder('{folder_id}')\">\n <svg class=\"folder-icon\" xmlns=\"http://www.w3.org/2000/svg\" width=\"1em\" height=\"1em\" fill=\"none\" viewBox=\"0 0 16 16\">\n <path fill=\"currentColor\" d=\"M.75 2a.75.75 0 0 0-.75.75v10.5c0 .414.336.75.75.75h14.5a.75.75 0 0 0 .75-.75v-8.5a.75.75 0 0 0-.75-.75H7.81L6.617 2.805A2.75 2.75 0 0 0 4.672 2z\"></path>\n </svg>\n <span class=\"folder-name\">{name}</span>\n </div>\n <div class=\"tree-children\" id=\"{folder_id}\">\n '''\n\n # Recursively render children\n html += self.render_tree_html(node['__children__'], iframe_root_src, level + 1)\n\n html += '''\n </div>\n </div>\n '''\n else:\n # Render file\n notebook = node['__notebook__']\n path = node['__path__']\n file_icon = self.get_file_icon_svg(path)\n notebook_link = iframe_root_src + path + \".html\"\n # Use the filename (last part of the path)\n filename = path.split('/')[-1]\n\n html += f'''\n <div class=\"tree-item tree-file {'tree-last' if is_last else ''}\">\n <a href=\"#\" class=\"tree-item-row file-row _left_menu\" iframe-src=\"{notebook_link}\">\n {file_icon}\n <span class=\"file-name\">{filename}</span>\n </a>\n </div>\n '''\n\n return html\n\n def generate_html_from_code_file(self, code_file_path: str, output_html_path: str, demo_name: str):\n \"\"\"\n Generate HTML file from .py or .sql code file with syntax highlighting\n\n Args:\n code_file_path: Path to the source code file (.py or .sql)\n output_html_path: Path where the HTML file should be saved\n demo_name: Name of the demo (for metadata)\n \"\"\"\n import html\n\n # Determine file type and language\n file_extension = code_file_path.split('.')[-1]\n file_name = os.path.basename(code_file_path)\n file_path_display = code_file_path\n\n if file_extension == 'py':\n language = 'python'\n file_type = 'Python'\n elif file_extension == 'sql':\n language = 'sql'\n file_type = 'SQL'\n else:\n raise ValueError(f\"Unsupported file type: {file_extension}. Only .py and .sql are supported.\")\n\n # Read the code file\n with open(code_file_path, 'r', encoding='utf-8') as f:\n code_content = f.read()\n\n # HTML escape the code content to prevent XSS\n code_content_escaped = html.escape(code_content)\n\n # Load the code viewer template\n template = pkg_resources.resource_string(\"dbdemos\", \"template/code_viewer.html\").decode('UTF-8')\n\n # Replace placeholders\n template = template.replace(\"{{FILE_NAME}}\", file_name)\n template = template.replace(\"{{FILE_TYPE}}\", file_type)\n template = template.replace(\"{{FILE_PATH}}\", file_path_display)\n template = template.replace(\"{{LANGUAGE}}\", language)\n template = template.replace(\"{{CODE_CONTENT}}\", code_content_escaped)\n template = template.replace(\"{{DEMO_NAME}}\", demo_name)\n\n # Write the HTML file\n with open(output_html_path, 'w', encoding='utf-8') as f:\n f.write(template)\n\n #Build HTML pages with index.\n # - If the notebook is pre-run, load them from the install_package folder\n # - If the notebook isn't pre-run, download them from the pacakge workspace as HTML (ex: can't run SDP pipelines)\n def build_minisite(self, demo_conf: DemoConf, iframe_root_src = \"./\"):\n notebooks_to_publish = demo_conf.get_notebooks_to_publish()\n print(f\"Build minisite for demo {demo_conf.name} ({demo_conf.path}) - {notebooks_to_publish}\")\n minisite_path = demo_conf.get_minisite_path()\n\n for notebook in notebooks_to_publish:\n Path(minisite_path).mkdir(parents=True, exist_ok=True)\n full_path = minisite_path+\"/\"+notebook.get_clean_path()+\".html\"\n Path(full_path[:full_path.rindex(\"/\")]).mkdir(parents=True, exist_ok=True)\n\n # Check if we have a code file (.py or .sql) or notebook HTML file\n # Code files are stored with their full extension in the bundle\n clean_path = notebook.get_clean_path()\n\n if notebook.path.endswith(('.py', '.sql')):\n # Code file - path already includes extension (.py or .sql)\n source_file_path = demo_conf.get_bundle_path() + \"/\" + clean_path\n file_type = clean_path.split('.')[-1].upper()\n print(f\" Generating HTML from {file_type} file: {source_file_path}\")\n self.generate_html_from_code_file(source_file_path, full_path, demo_conf.name)\n else:\n # Standard notebook HTML file - append .html extension\n source_file_path = demo_conf.get_bundle_path() + \"/\" + clean_path + \".html\"\n if not os.path.exists(source_file_path):\n raise FileNotFoundError(f\"Could not find notebook file: {source_file_path}\")\n with open(source_file_path, \"r\") as f:\n parser = NotebookParser(f.read())\n with open(full_path, \"w\") as f:\n parser.remove_robots_meta()\n parser.add_cell_as_html_for_seo()\n parser.remove_delete_cell()\n parser.add_javascript_to_minisite_relative_links(notebook.get_clean_path())\n f.write(parser.get_html())\n\n # Build the tree structure from all notebooks\n tree = self.build_tree_structure(notebooks_to_publish)\n\n # Render the tree as HTML\n tree_html = self.render_tree_html(tree, iframe_root_src)\n\n # Create the index file\n template = pkg_resources.resource_string(\"dbdemos\", \"template/index.html\").decode('UTF-8')\n template = template.replace(\"{{LEFT_MENU}}\", tree_html)\n template = template.replace(\"{{TITLE}}\", demo_conf.title)\n template = template.replace(\"{{DESCRIPTION}}\", demo_conf.description)\n template = template.replace(\"{{DEMO_NAME}}\", demo_conf.name)\n with open(minisite_path+\"/index.html\", \"w\") as f:\n f.write(template)\n #dump the conf\n with open(demo_conf.get_bundle_root_path()+\"/conf.json\", \"w\") as f:\n f.write(json.dumps(demo_conf.json_conf))" }, { "path": "dbdemos/resources/default_cluster_config-AWS.json", "content": "{\n \"node_type_id\": \"i3.xlarge\",\n \"aws_attributes\": {\n \"first_on_demand\": 1,\n \"availability\": \"SPOT_WITH_FALLBACK\",\n \"instance_profile_arn\": null,\n \"spot_bid_price_percent\": 100,\n \"ebs_volume_count\": 0\n }\n}" }, { "path": "dbdemos/resources/default_cluster_config-AZURE.json", "content": "{\n \"node_type_id\": \"Standard_D8ds_v4\",\n \"azure_attributes\": {\n \"first_on_demand\": 1,\n \"availability\": \"ON_DEMAND_AZURE\",\n \"spot_bid_max_price\": -1\n }\n}" }, { "path": "dbdemos/resources/default_cluster_config-GCP.json", "content": "{\n \"node_type_id\": \"n1-standard-8\",\n \"gcp_attributes\": {\n \"use_preemptible_executors\": false,\n \"availability\": \"ON_DEMAND_GCP\",\n \"zone_id\": \"HA\"\n }\n}" }, { "path": "dbdemos/resources/default_cluster_config.json", "content": "{\n \"autoscale\": {\n \"min_workers\": 4,\n \"max_workers\": 4\n },\n \"cluster_name\": \"dbdemos-{{DEMO_NAME}}-{{CURRENT_USER_NAME}}\",\n \"spark_version\": \"16.4.x-cpu-ml-scala2.12\",\n \"spark_conf\": {\n \"spark.databricks.dataLineage.enabled\": \"true\"\n },\n \"ssh_public_keys\": [],\n \"custom_tags\": {\n \"project\": \"dbdemos\",\n \"demo\": \"{{DEMO_NAME}}\"\n },\n \"spark_env_vars\": {},\n \"autotermination_minutes\": 60,\n \"clus4ter_source\": \"UI\",\n \"init_scripts\": [],\n \"runtime_engine\": \"STANDARD\"\n}" }, { "path": "dbdemos/resources/default_cluster_job_config.json", "content": "{\n \"spark_version\": \"16.4.x-cpu-ml-scala2.12\",\n \"spark_conf\": {\n \"spark.databricks.dataLineage.enabled\": \"true\"\n },\n \"custom_tags\": {\n \"project\": \"dbdemos\",\n \"demo\": \"{{DEMO_NAME}}\"\n }\n}" }, { "path": "dbdemos/resources/default_test_job_conf.json", "content": "{\n \"name\": \"field-demos_{{DEMO_NAME}}\",\n \"email_notifications\": {\n \"no_alert_for_skipped_runs\": false\n },\n \"timeout_seconds\": 0,\n \"max_concurrent_runs\": 1,\n \"tasks\": [\n ],\n \"job_clusters\": [\n {\n \"job_cluster_key\": \"field_demo_test\",\n \"new_cluster\": {\n \"spark_version\": \"16.4.x-cpu-ml-scala2.12\",\n \"custom_tags\": {\n \"project\": \"dbdemos\",\n \"demo_bundle_job\": \"autoloader\",\n \"demo\": \"autoloader\"\n },\n \"spark_conf\": {\n \"spark.databricks.dataLineage.enabled\": \"true\"\n },\n \"instance_pool_id\": \"1025-140806-yup112-pool-yz565bma\",\n \"data_security_mode\": \"NONE\",\n \"runtime_engine\": \"STANDARD\",\n \"num_workers\": 3\n }\n }\n ],\n \"git_source\": {\n \"git_url\": \"WILL BE OVERRIDED BY THE LOCAL CONF SETUP\",\n \"git_provider\": \"gitHub\",\n \"git_branch\": \"WILL BE OVERRIDED BY THE LOCAL CONF SETUP\"\n },\n \"format\": \"MULTI_TASK\"\n}" }, { "path": "dbdemos/sql_query.py", "content": "import logging\nfrom databricks.sdk import WorkspaceClient\nfrom databricks.sdk.service.sql import StatementState, ExecuteStatementRequestOnWaitTimeout\nfrom databricks.sdk.service.sql import ResultData, ResultManifest\nfrom typing import List, Dict, Any\nimport time\n\nfrom dbdemos.exceptions.dbdemos_exception import SQLQueryException\n\nclass SQLQueryExecutor:\n def __init__(self):\n self.logger = logging.getLogger(__name__)\n\n def get_or_create_shared_warehouse(self, ws: WorkspaceClient) -> str:\n warehouses = ws.warehouses.list()\n \n # First, look for a shared warehouse\n for warehouse in warehouses:\n if 'shared' in warehouse.name.lower():\n return warehouse.id\n\n # If no shared warehouse, look for a running warehouse\n for warehouse in warehouses:\n if warehouse.state == 'RUNNING':\n return warehouse.id\n \n # If no running warehouse, return the first available warehouse\n if warehouses:\n return warehouses[0].id\n \n # If no warehouses at all, create a new one\n new_warehouse = ws.warehouses.create(\n name=\"shared-warehouse\",\n cluster_size=\"Small\",\n auto_stop_mins=10\n )\n return new_warehouse.id\n\n def execute_query_as_list(self, ws: WorkspaceClient, query: str, timeout: int = 50, warehouse_id: str = None, debug: bool = False) -> tuple[ResultData, ResultManifest]:\n data, manifest = self.execute_query(ws, query, timeout, warehouse_id, debug)\n return self.get_results_formatted_as_list(data, manifest)\n\n def execute_query(self, ws: WorkspaceClient, query: str, timeout: int = 50, warehouse_id: str = None, debug: bool = False) -> tuple[ResultData, ResultManifest]:\n if not warehouse_id:\n warehouse_id = self.get_or_create_shared_warehouse(ws)\n if debug:\n print(f\"Executing query: {query} with warehouse {warehouse_id}\")\n # Execute the query with a maximum wait timeout of 50 seconds\n statement = ws.statement_execution.execute_statement(\n warehouse_id=warehouse_id,\n statement=query,\n wait_timeout=f\"{timeout}s\",\n on_wait_timeout=ExecuteStatementRequestOnWaitTimeout.CONTINUE\n )\n \n # If the statement is not completed within the wait_timeout, poll for results\n while statement.status.state in [StatementState.PENDING, StatementState.RUNNING]:\n time.sleep(1)\n statement = ws.statement_execution.get_statement(statement.statement_id)\n if statement.status.state == StatementState.FAILED:\n raise SQLQueryException(f\"Query execution failed: {statement.status.error}\")\n \n # Fetch initial results\n results = ws.statement_execution.get_statement(statement.statement_id)\n \n # Initialize combined result data\n combined_data = ResultData(data_array=[])\n if results.result and results.result.data_array:\n combined_data.data_array.extend(results.result.data_array)\n \n # Fetch additional chunks if they exist\n if results.manifest and results.manifest.chunks:\n chunk_index = results.manifest.chunks[0].chunk_index + 1\n while chunk_index < results.manifest.total_chunk_count:\n chunk = ws.statement_execution.get_statement_result_chunk_n(\n statement_id=statement.statement_id,\n chunk_index=chunk_index\n )\n if chunk.data_array:\n combined_data.data_array.extend(chunk.data_array)\n chunk_index += 1\n \n return combined_data, results.manifest\n\n def get_results_formatted_as_list(self, result_data: ResultData, result_manifest: ResultManifest) -> List[Dict[str, Any]]:\n column_names = [col.name for col in result_manifest.schema.columns]\n \n result_list = []\n \n if result_data.data_array: \n for row in result_data.data_array:\n result_dict = {column_names[i]: value for i, value in enumerate(row)}\n result_list.append(result_dict)\n \n return result_list" }, { "path": "dbdemos/template/LICENSE.html", "content": "<!DOCTYPE html>\n<html>\n<head>\n <meta name=\"databricks-html-version\" content=\"1\">\n<title>LICENSE - Databricks\n\n\n\n\n\n\n\n\n\n\n\n\n\n \n\n\n\n\n" }, { "path": "dbdemos/template/NOTICE.html", "content": "\n\n\n \nNOTICE - Databricks\n\n\n\n\n\n\n\n\n\n\n\n\n\n \n\n\n\n\n" }, { "path": "dbdemos/template/README.html", "content": "\n\n\n \nREADME - Databricks\n\n\n\n\n\n\n\n\n\n\n\n\n\n \n\n\n\n\n" }, { "path": "dbdemos/template/code_viewer.html", "content": "\n\n\n \n \n \n {{FILE_NAME}}\n\n \n \n \n \n \n\n \n\n\n
\n

\n {{FILE_NAME}}\n {{FILE_TYPE}}\n

\n
\n\n
\n
\n {{FILE_PATH}}\n
\n 
{{CODE_CONTENT}}
\n
\n\n \n\n\n" }, { "path": "dbdemos/template/index.html", "content": "\n\n\n \n \n \n \n \n {{TITLE}}\n\n \n \n \n \n\n \n\n\n \n \n\n\n\n
\n
\n
\n 
%pip install dbdemos
\n 
import dbdemos\ndbdemos.install('{{DEMO_NAME}}')
\n
\n\n
Try this demo in your workspace!
Run the following in your notebook:
License | Notice\n
\n
\n\n \n \"databricks-logo\"\n \n
\n

{{TITLE}}

\n {{DESCRIPTION}}\n
\n
\n\n
\n
\n
\n {{LEFT_MENU}}\n
\n
\n
\n
\n \n
\n
\n\n\n\n\n\n\n\n\n\n" }, { "path": "dbdemos/tracker.py", "content": "import requests\nimport urllib.parse\nimport hashlib\n\nclass Tracker:\n #Set this value to false to disable dbdemo toolkit tracker.\n enable_tracker = True\n URL = \"https://ppxrzfxige.execute-api.us-west-2.amazonaws.com/v1/analytics\"\n\n def __init__(self, org_id, uid, email = None):\n self.org_id = org_id\n self.uid = uid\n # This is aggregating user behavior within Databricks at the org level to better understand dbdemos usage and improve the product.\n # We are not collecting any email/PII data. Please reach out to the demo team if you have any questions.\n if email is not None and email.endswith(\"@databricks.com\"):\n self.email = email\n else:\n self.email = None\n\n def track_install(self, category, demo_name):\n self.track(category, demo_name, \"INSTALL\")\n \n def track_create_cluster(self, category, demo_name):\n self.track(category, demo_name, \"CREATE_CLUSTER\")\n \n def track_list(self):\n self.track(\"list_demos\", \"list_demos\", \"LIST\")\n\n def get_user_hash(self):\n if self.email is None or not self.email.endswith(\"@databricks.com\"):\n return None\n return hashlib.sha256(self.email.encode()).hexdigest()\n\n def get_track_url(self, category, demo_name, event, notebook = \"\"):\n params = self.get_track_params(category, demo_name, event, notebook)\n return Tracker.URL+\"?\"+urllib.parse.urlencode(params)\n\n def get_track_params(self, category, demo_name, event, notebook =\"\"):\n if not Tracker.enable_tracker:\n return {}\n if len(notebook) > 0:\n notebook = '/'+notebook\n params = {\"category\": category,\n \"org_id\": self.org_id, #legacy \"cid\" -- ignore \"uid\": self.uid\n \"notebook\": notebook,\n \"demo_name\": demo_name,\n \"event\": event,\n \"path\": f\"/_dbdemos/{category}/{demo_name}{notebook}\", #legacy tracking \"dp\"\n \"version\": 1}\n user_hash = self.get_user_hash()\n if user_hash is not None:\n params[\"user_hash\"] = user_hash\n return params\n\n\n def track(self, category, demo_name, event):\n if self.org_id == \"1660015457675682\":\n print(\"skipping tracker for test / dev\")\n elif Tracker.enable_tracker:\n headers = {\"accept\": \"text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9\",\n \"accept-encoding\": \"gzip, deflate, br\",\n \"accept-language\": \"en-US,en;q=0.9\",\n \"cache-control\": \"max-age=0\",\n \"user-agent\": \"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/107.0.0.0 Safari/537.36\"}\n try:\n with requests.post(Tracker.URL, json = self.get_track_params(category, demo_name, event), headers=headers, timeout=5) as t:\n if t.status_code != 200:\n print(f\"Info - Usage report error (internet access not available?). See readme to disable it, you can ignore safely this. Details: {t.text}\")\n except Exception as e:\n print(\"Usage report error. See readme to disable it. \"+(str(e)))\n" }, { "path": "docs/CNAME", "content": "dbdemos.ai" }, { "path": "docs/index.html", "content": "\n\n\n\ndbdemos.ai - Demos for Databricks\ndbdemos.ai has a new home under databricks.com!
\nRedirecting to https://www.databricks.com/resources/demos/tutorials..\n\n\n\n\n\n\n\n\n\n" }, { "path": "main.py", "content": "import json\nfrom dbdemos.conf import Conf, DemoConf\nfrom dbdemos.installer import Installer\nfrom dbdemos.job_bundler import JobBundler\nfrom dbdemos.packager import Packager\nimport traceback\n\nwith open(\"./local_conf_E2TOOL.json\", \"r\") as r:\n c = json.loads(r.read())\nwith open(\"./dbdemos/resources/default_cluster_config.json\", \"r\") as cc:\n default_cluster_template = cc.read()\nwith open(\"./dbdemos/resources/default_test_job_conf.json\", \"r\") as cc:\n default_cluster_job_template = cc.read()\n\nconf = Conf(c['username'], c['url'], c['org_id'], c['pat_token'],\n default_cluster_template, default_cluster_job_template,\n c['repo_staging_path'], c['repo_name'], c['repo_url'], c['branch'], github_token=c['github_token'])\n\n\ndef bundle():\n bundler = JobBundler(conf)\n # the bundler will use a stating repo dir in the workspace to analyze & run content.\n bundler.reset_staging_repo(skip_pull=False, )\n \"\"\"\n bundler.load_bundles_conf()\n #bundler.add_bundle(\"product_demos/Unity-Catalog/uc-05-upgrade\")\n # Or manually add bundle to run faster:\n \n bundler.add_bundle(\"product_demos/Unity-Catalog/05-Upgrade-to-UC\")\n bundler.add_bundle(\"product_demos/Unity-Catalog/02-External-location\")\n #bundler.load_bundles_conf()\n \"\"\"\n bundler.load_bundles_conf()\n #bundler.ignore_bundle(\"/product_demos/delta-sharing-airlines\")\n #bundler.add_bundle(\"aibi/aibi-marketing-campaign\")\n \n #bundler.add_bundle(\"demo-retail/lakehouse-retail-c360\")\n # Run the jobs (only if there is a new commit since the last time, or failure, or force execution)\n bundler.start_and_wait_bundle_jobs(force_execution = False, skip_execution=False, recreate_jobs=False)\n\n packager = Packager(conf, bundler)\n packager.package_all()\n\nbundle()\n\n\n#Loads conf to install\nwith open(\"local_conf_E2FE.json\", \"r\") as r:\n c = json.loads(r.read())\n\nfrom dbdemos.installer import Installer\nimport dbdemos\n\n\n\n#dbdemos.list_demos(pat_token=c['pat_token'])\n#dbdemos.install_all(\"/Users/quentin.ambard@databricks.com/test_install_quentin\", True, c['username'], c['pat_token'], c['url'], cloud=\"AWS\", start_cluster = False, skip_dashboards=False, catalog='main_test_quentin')\n#dbdemos.check_status_all()\n#dbdemos.install(\"ai-agent\", \"/Users/quentin.ambard@databricks.com/test_install_quentin\", True, c['username'], c['pat_token'], c['url'], catalog='main', schema='test_quentin_dbdemos_ai_agent', cloud=\"AWS\", start_cluster = False, skip_dashboards=False, serverless=True)\n#dbdemos.install(\"computer-vision-pcb\", \"/Users/quentin.ambard@databricks.com/test_install_quentin\", True, c['username'], c['pat_token'], c['url'], catalog='main', schema='quentin_test2', cloud=\"AWS\", start_cluster = False, skip_dashboards=False)\n\n#dbdemos.install(\"uc-01-acl\", \"/Users/quentin.ambard@databricks.com/test_install_quentin\", True, c['username'], c['pat_token'], c['url'], start_cluster = False, schema='test_quentin_acl', catalog='dbdemos')\n#dbdemos.install(\"uc-02-external-location\", \"/Users/quentin.ambard@databricks.com/test_install_quentin\", True, c['username'], c['pat_token'], c['url'], cloud=\"AWS\")\n#dbdemos.install(\"uc-03-data-lineage\", \"/Users/quentin.ambard@databricks.com/test_install_quentin\", True, c['username'], c['pat_token'], c['url'], cloud=\"AWS\")\n#dbdemos.install(\"uc-05-upgrade\", \"/Users/quentin.ambard@databricks.com/test_install_quentin\", True, c['username'], c['pat_token'], c['url'])\n\n \n#dbdemos.install(\"dlt-cdc\", \"/Users/quentin.ambard@databricks.com/test_install_quentin\", True, c['username'], c['pat_token'], c['url'], catalog='main', schema='quentin_test2', cloud=\"AWS\", start_cluster = False, debug=True, dlt_policy_id = \"0003963E5B551CE4\", dlt_compute_settings = {\"autoscale\": {\"min_workers\": 1, \"max_workers\": 5}})\n#dbdemos.install(\"dlt-loans\", \"/Users/quentin.ambard@databricks.com/test_install_quentin\", True, c['username'], c['pat_token'], c['url'], catalog='main', schema='quentin_test2', cloud=\"AWS\", start_cluster = False, debug=True)\n#dbdemos.install(\"declarative-pipeline-unit-test\", \"/Users/quentin.ambard@databricks.com/test_install_quentin\", True, c['username'], c['pat_token'], c['url'], catalog='main', schema='test_quentin_pipeline_unit_test', cloud=\"AWS\", start_cluster = False, debug=True)\n\n#dbdemos.install(\"llm-tools-functions\", \"/Users/quentin.ambard@databricks.com/test_install_quentin\", True, c['username'], c['pat_token'], c['url'], cloud=\"AWS\", use_current_cluster=False, current_cluster_id=c[\"current_cluster_id\"], schema='test_quentin_rag', catalog='dbdemos', debug=True)\n\n#dbdemos.install(\"declarative-pipeline-unit-test\", \"/Users/quentin.ambard@databricks.com/test_install_quentin\", True, c['username'], c['pat_token'], c['url'], catalog='main', schema='quentin_test_unit_test', cloud=\"AWS\", start_cluster = False, debug=True)\n#dbdemos.install(\"auto-loader\", \"/Users/quentin.ambard@databricks.com/test_install_quentin\", True, c['username'], c['pat_token'], c['url'], catalog='main', schema='quentin_test2', cloud=\"AWS\", start_cluster = False, debug=True)\n\"\"\"\n\"\"\"\n\n\n\n#dbdemos.install_all(\"/Users/quentin.ambard@databricks.com/test_install_quentin\", True, c['username'], c['pat_token'], c['url'], cloud=\"AWS\", start_cluster = False, skip_dashboards = True)\n#dbdemos.check_status_all(c['username'], c['pat_token'], c['url'], cloud=\"AWS\")\n#dbdemos.install(\"delta-lake\", \"/Users/quentin.ambard@databricks.com/test_install_quentin\", True, c['username'], c['pat_token'], c['url'], cloud=\"AWS\", start_cluster = True, skip_dashboards=False)\n#dbdemos.install(\"delta-lake\", \"/Users/quentin.ambard@databricks.com/test_install_quentin\", True, c['username'], c['pat_token'], c['url'], cloud=\"AWS\", start_cluster = True, skip_dashboards=False)\n#installer = Installer()\n#for d in installer.get_demos_available():\n# dbdemos.install(d, \"/Users/quentin.ambard@databricks.com/test_dbdemos\", True, c['username'], c['pat_token'], c['url'], cloud=\"AWS\")\n\n\n#dbdemos.list_demos(None)\n\n#dbdemos.install(\"llm-rag-chatbot\", \"/Users/quentin.ambard@databricks.com/test_install_quentin\", True, c['username'], c['pat_token'], c['url'], catalog='main', schema='quentin_test_rag', cloud=\"AWS\", start_cluster = False, skip_dashboards=True, use_current_cluster=True, debug = True)\n#dbdemos.install(\"lakehouse-monitoring\", \"/Users/quentin.ambard@databricks.com/test_install_quentin\", True, c['username'], c['pat_token'], c['url'], catalog='main', schema='quentin_test_lhm', cloud=\"AWS\", start_cluster = False, skip_dashboards=False, use_current_cluster=True, debug = True)\n#dbdemos.install(\"uc-04-system-tables\", \"/Users/quentin.ambard@databricks.com/test_install_quentin\", True, c['username'], c['pat_token'], c['url'], catalog='main', schema='quentin_test_sys', cloud=\"AWS\", start_cluster = False, debug = True, serverless=True)\n\n#dbdemos.install(\"declarative-pipeline-cdc\", \"/Users/quentin.ambard@databricks.com/test_install_quentin\", True, c['username'], c['pat_token'], c['url'], catalog='main', schema='quentin_test2', cloud=\"AWS\", start_cluster = False, skip_dashboards=False)\n#dbdemos.install(\"lakehouse-retail-c360\", \"/Users/quentin.ambard@databricks.com/test_install_quentin\", True, c['username'], c['pat_token'], c['url'], catalog='main_test_quentin2', schema='quentin_test2', cloud=\"AWS\", start_cluster = False, skip_dashboards=False, create_schema=True, debug=True)\n#dbdemos.install(\"lakehouse-fsi-smart-claims\", \"/Users/quentin.ambard@databricks.com/test_install_quentin\", True, c['username'], c['pat_token'], c['url'], catalog='main', schema='quentin_test2', cloud=\"AWS\", start_cluster = False, skip_dashboards=False)\n#dbdemos.install(\"lakehouse-fsi-credit\", \"/Users/quentin.ambard@databricks.com/test_install_quentin\", True, c['username'], c['pat_token'], c['url'], catalog='main', schema='quentin_test2', cloud=\"AWS\", start_cluster = False, skip_dashboards=False)\n#dbdemos.install(\"lakehouse-fsi-fraud\", \"/Users/quentin.ambard@databricks.com/test_install_quentin\", True, c['username'], c['pat_token'], c['url'], catalog='main', schema='quentin_test2', cloud=\"AWS\", start_cluster = False, skip_dashboards=False)\n\n#dbdemos.install(\"lakehouse-iot-platform\", \"/Users/quentin.ambard@databricks.com/test_install_quentin\", True, c['username'], c['pat_token'], c['url'], catalog='main', schema='quentin_test3', cloud=\"AWS\", start_cluster = False)\n#dbdemos.install(\"pipeline-bike\", \"/Users/quentin.ambard@databricks.com/test_install_quentin\", True, c['username'], c['pat_token'], c['url'], catalog='main', schema='quentin_test3', cloud=\"AWS\", start_cluster = False)\n\n#dbdemos.install(\"feature-store\", \"/Users/quentin.ambard@databricks.com/test_install_quentin\", True, c['username'], c['pat_token'], c['url'], cloud=\"AWS\", use_current_cluster=False, current_cluster_id=c[\"current_cluster_id\"])\n#dbdemos.install(\"delta-lake\", \"/Users/quentin.ambard@databricks.com/test_install_quentin\", True, c['username'], c['pat_token'], c['url'], cloud=\"GCP\")\n#dbdemos.install(\"delta-lake\", \"/Users/quentin.ambard@databricks.com/test_install_quentin\", True, c['username'], c['pat_token'], c['url'], cloud=\"Azure\", use_current_cluster=True, current_cluster_id=c[\"current_cluster_id\"])\n#dbdemos.install(\"mlops-end2end\", \"/Users/quentin.ambard@databricks.com/test_install_quentin\", True, c['username'], c['pat_token'], c['url'], cloud=\"AWS\", skip_dashboards=True, schema='test_quentin_dbdemos', catalog='dbdemos')\n#dbdemos.install(\"pandas-on-spark\", \"/Users/quentin.ambard@databricks.com/test_install_quentin\", True, c['username'], c['pat_token'], c['url'], cloud=\"AWS\")\n#dbdemos.install(\"delta-sharing-airlines\", \"/Users/quentin.ambard@databricks.com/test_install_quentin\", True, c['username'], c['pat_token'], c['url'])\n#dbdemos.install(\"dlt-loans\", \"/Users/quentin.ambard@databricks.com/test_install_quentin\", True, c['username'], c['pat_token'], c['url'])\n#dbdemos.install(\"dlt-unit-test\", \"/Users/quentin.ambard@databricks.com/test_install\", True, c['username'], c['pat_token'], c['url'])\n#dbdemos.create_cluster(\"uc-05-upgrade\", c['username'], c['pat_token'], c['url'], \"GCP\")\n\n\ndbdemos.install(\"aibi-marketing-campaign\", \"/Users/quentin.ambard@databricks.com/test_install_quentin\", True, c['username'], c['pat_token'], c['url'], catalog='main', schema='quentin_test_sys', cloud=\"AWS\", start_cluster = False, debug = True)\n#dbdemos.install(\"aibi-portfolio-assistant\", \"/Users/quentin.ambard@databricks.com/test_install_quentin\", True, c['username'], c['pat_token'], c['url'], catalog='main', schema='quentin_test_sys', cloud=\"AWS\", start_cluster = False, debug = True)\n#dbdemos.install(\"aibi-supply-chain-forecasting\", \"/Users/quentin.ambard@databricks.com/test_install_quentin\", True, c['username'], c['pat_token'], c['url'], catalog='main', schema='quentin_test_sys', cloud=\"AWS\", start_cluster = False, debug = True)\n#dbdemos.install(\"aibi-sales-pipeline-review\", \"/Users/quentin.ambard@databricks.com/test_install_quentin\", True, c['username'], c['pat_token'], c['url'], catalog='main', schema='quentin_test_sys', cloud=\"AWS\", start_cluster = False, debug = True)\n#dbdemos.install(\"aibi-patient-genomics\", \"/Users/quentin.ambard@databricks.com/test_install_quentin\", True, c['username'], c['pat_token'], c['url'], catalog='main', schema='quentin_test_sys', cloud=\"AWS\", start_cluster = False, debug = True)\n#dbdemos.install(\"aibi-customer-support\", \"/Users/quentin.ambard@databricks.com/test_install_quentin\", True, c['username'], c['pat_token'], c['url'], catalog='main', schema='quentin_test_sys', cloud=\"AWS\", start_cluster = False, debug = True)\n\n\n\n#dbdemos.install(\"dbt-on-databricks\", \"/Users/quentin.ambard@databricks.com/test_install_quentin\", True, c['username'], c['pat_token'], c['url'], catalog='main', schema='quentin_test_sys', cloud=\"AWS\", start_cluster = False, debug = True, serverless=True) " }, { "path": "requirements.in", "content": "# Direct dependencies from setup.py\nrequests\npandas\ndatabricks-sdk>=0.38.0\n" }, { "path": "requirements.txt", "content": "#\n# This file is autogenerated by pip-compile with Python 3.11\n# by the following command:\n#\n# pip-compile --generate-hashes --output-file=requirements.txt requirements.in\n#\ncertifi==2026.2.25 \\\n --hash=sha256:027692e4402ad994f1c42e52a4997a9763c646b73e4096e4d5d6db8af1d6f0fa \\\n --hash=sha256:e887ab5cee78ea814d3472169153c2d12cd43b14bd03329a39a9c6e2e80bfba7\n # via requests\ncffi==2.0.0 \\\n --hash=sha256:00bdf7acc5f795150faa6957054fbbca2439db2f775ce831222b66f192f03beb \\\n --hash=sha256:07b271772c100085dd28b74fa0cd81c8fb1a3ba18b21e03d7c27f3436a10606b \\\n --hash=sha256:087067fa8953339c723661eda6b54bc98c5625757ea62e95eb4898ad5e776e9f \\\n --hash=sha256:0a1527a803f0a659de1af2e1fd700213caba79377e27e4693648c2923da066f9 \\\n --hash=sha256:0cf2d91ecc3fcc0625c2c530fe004f82c110405f101548512cce44322fa8ac44 \\\n --hash=sha256:0f6084a0ea23d05d20c3edcda20c3d006f9b6f3fefeac38f59262e10cef47ee2 \\\n --hash=sha256:12873ca6cb9b0f0d3a0da705d6086fe911591737a59f28b7936bdfed27c0d47c \\\n --hash=sha256:19f705ada2530c1167abacb171925dd886168931e0a7b78f5bffcae5c6b5be75 \\\n --hash=sha256:1cd13c99ce269b3ed80b417dcd591415d3372bcac067009b6e0f59c7d4015e65 \\\n --hash=sha256:1e3a615586f05fc4065a8b22b8152f0c1b00cdbc60596d187c2a74f9e3036e4e \\\n --hash=sha256:1f72fb8906754ac8a2cc3f9f5aaa298070652a0ffae577e0ea9bd480dc3c931a \\\n --hash=sha256:1fc9ea04857caf665289b7a75923f2c6ed559b8298a1b8c49e59f7dd95c8481e \\\n --hash=sha256:203a48d1fb583fc7d78a4c6655692963b860a417c0528492a6bc21f1aaefab25 \\\n --hash=sha256:2081580ebb843f759b9f617314a24ed5738c51d2aee65d31e02f6f7a2b97707a \\\n --hash=sha256:21d1152871b019407d8ac3985f6775c079416c282e431a4da6afe7aefd2bccbe \\\n --hash=sha256:24b6f81f1983e6df8db3adc38562c83f7d4a0c36162885ec7f7b77c7dcbec97b \\\n --hash=sha256:256f80b80ca3853f90c21b23ee78cd008713787b1b1e93eae9f3d6a7134abd91 \\\n --hash=sha256:28a3a209b96630bca57cce802da70c266eb08c6e97e5afd61a75611ee6c64592 \\\n --hash=sha256:2c8f814d84194c9ea681642fd164267891702542f028a15fc97d4674b6206187 \\\n --hash=sha256:2de9a304e27f7596cd03d16f1b7c72219bd944e99cc52b84d0145aefb07cbd3c \\\n --hash=sha256:38100abb9d1b1435bc4cc340bb4489635dc2f0da7456590877030c9b3d40b0c1 \\\n --hash=sha256:3925dd22fa2b7699ed2617149842d2e6adde22b262fcbfada50e3d195e4b3a94 \\\n --hash=sha256:3e17ed538242334bf70832644a32a7aae3d83b57567f9fd60a26257e992b79ba \\\n --hash=sha256:3e837e369566884707ddaf85fc1744b47575005c0a229de3327f8f9a20f4efeb \\\n --hash=sha256:3f4d46d8b35698056ec29bca21546e1551a205058ae1a181d871e278b0b28165 \\\n --hash=sha256:44d1b5909021139fe36001ae048dbdde8214afa20200eda0f64c068cac5d5529 \\\n --hash=sha256:45d5e886156860dc35862657e1494b9bae8dfa63bf56796f2fb56e1679fc0bca \\\n --hash=sha256:4647afc2f90d1ddd33441e5b0e85b16b12ddec4fca55f0d9671fef036ecca27c \\\n --hash=sha256:4671d9dd5ec934cb9a73e7ee9676f9362aba54f7f34910956b84d727b0d73fb6 \\\n --hash=sha256:53f77cbe57044e88bbd5ed26ac1d0514d2acf0591dd6bb02a3ae37f76811b80c \\\n --hash=sha256:5eda85d6d1879e692d546a078b44251cdd08dd1cfb98dfb77b670c97cee49ea0 \\\n --hash=sha256:5fed36fccc0612a53f1d4d9a816b50a36702c28a2aa880cb8a122b3466638743 \\\n --hash=sha256:61d028e90346df14fedc3d1e5441df818d095f3b87d286825dfcbd6459b7ef63 \\\n --hash=sha256:66f011380d0e49ed280c789fbd08ff0d40968ee7b665575489afa95c98196ab5 \\\n --hash=sha256:6824f87845e3396029f3820c206e459ccc91760e8fa24422f8b0c3d1731cbec5 \\\n --hash=sha256:6c6c373cfc5c83a975506110d17457138c8c63016b563cc9ed6e056a82f13ce4 \\\n --hash=sha256:6d02d6655b0e54f54c4ef0b94eb6be0607b70853c45ce98bd278dc7de718be5d \\\n --hash=sha256:6d50360be4546678fc1b79ffe7a66265e28667840010348dd69a314145807a1b \\\n --hash=sha256:730cacb21e1bdff3ce90babf007d0a0917cc3e6492f336c2f0134101e0944f93 \\\n --hash=sha256:737fe7d37e1a1bffe70bd5754ea763a62a066dc5913ca57e957824b72a85e205 \\\n --hash=sha256:74a03b9698e198d47562765773b4a8309919089150a0bb17d829ad7b44b60d27 \\\n --hash=sha256:7553fb2090d71822f02c629afe6042c299edf91ba1bf94951165613553984512 \\\n --hash=sha256:7a66c7204d8869299919db4d5069a82f1561581af12b11b3c9f48c584eb8743d \\\n --hash=sha256:7cc09976e8b56f8cebd752f7113ad07752461f48a58cbba644139015ac24954c \\\n --hash=sha256:81afed14892743bbe14dacb9e36d9e0e504cd204e0b165062c488942b9718037 \\\n --hash=sha256:8941aaadaf67246224cee8c3803777eed332a19d909b47e29c9842ef1e79ac26 \\\n --hash=sha256:89472c9762729b5ae1ad974b777416bfda4ac5642423fa93bd57a09204712322 \\\n --hash=sha256:8ea985900c5c95ce9db1745f7933eeef5d314f0565b27625d9a10ec9881e1bfb \\\n --hash=sha256:8eca2a813c1cb7ad4fb74d368c2ffbbb4789d377ee5bb8df98373c2cc0dee76c \\\n --hash=sha256:92b68146a71df78564e4ef48af17551a5ddd142e5190cdf2c5624d0c3ff5b2e8 \\\n --hash=sha256:9332088d75dc3241c702d852d4671613136d90fa6881da7d770a483fd05248b4 \\\n --hash=sha256:94698a9c5f91f9d138526b48fe26a199609544591f859c870d477351dc7b2414 \\\n --hash=sha256:9a67fc9e8eb39039280526379fb3a70023d77caec1852002b4da7e8b270c4dd9 \\\n --hash=sha256:9de40a7b0323d889cf8d23d1ef214f565ab154443c42737dfe52ff82cf857664 \\\n --hash=sha256:a05d0c237b3349096d3981b727493e22147f934b20f6f125a3eba8f994bec4a9 \\\n --hash=sha256:afb8db5439b81cf9c9d0c80404b60c3cc9c3add93e114dcae767f1477cb53775 \\\n --hash=sha256:b18a3ed7d5b3bd8d9ef7a8cb226502c6bf8308df1525e1cc676c3680e7176739 \\\n --hash=sha256:b1e74d11748e7e98e2f426ab176d4ed720a64412b6a15054378afdb71e0f37dc \\\n --hash=sha256:b21e08af67b8a103c71a250401c78d5e0893beff75e28c53c98f4de42f774062 \\\n --hash=sha256:b4c854ef3adc177950a8dfc81a86f5115d2abd545751a304c5bcf2c2c7283cfe \\\n --hash=sha256:b882b3df248017dba09d6b16defe9b5c407fe32fc7c65a9c69798e6175601be9 \\\n --hash=sha256:baf5215e0ab74c16e2dd324e8ec067ef59e41125d3eade2b863d294fd5035c92 \\\n --hash=sha256:c649e3a33450ec82378822b3dad03cc228b8f5963c0c12fc3b1e0ab940f768a5 \\\n --hash=sha256:c654de545946e0db659b3400168c9ad31b5d29593291482c43e3564effbcee13 \\\n --hash=sha256:c6638687455baf640e37344fe26d37c404db8b80d037c3d29f58fe8d1c3b194d \\\n --hash=sha256:c8d3b5532fc71b7a77c09192b4a5a200ea992702734a2e9279a37f2478236f26 \\\n --hash=sha256:cb527a79772e5ef98fb1d700678fe031e353e765d1ca2d409c92263c6d43e09f \\\n --hash=sha256:cf364028c016c03078a23b503f02058f1814320a56ad535686f90565636a9495 \\\n --hash=sha256:d48a880098c96020b02d5a1f7d9251308510ce8858940e6fa99ece33f610838b \\\n --hash=sha256:d68b6cef7827e8641e8ef16f4494edda8b36104d79773a334beaa1e3521430f6 \\\n --hash=sha256:d9b29c1f0ae438d5ee9acb31cadee00a58c46cc9c0b2f9038c6b0b3470877a8c \\\n --hash=sha256:d9b97165e8aed9272a6bb17c01e3cc5871a594a446ebedc996e2397a1c1ea8ef \\\n --hash=sha256:da68248800ad6320861f129cd9c1bf96ca849a2771a59e0344e88681905916f5 \\\n --hash=sha256:da902562c3e9c550df360bfa53c035b2f241fed6d9aef119048073680ace4a18 \\\n --hash=sha256:dbd5c7a25a7cb98f5ca55d258b103a2054f859a46ae11aaf23134f9cc0d356ad \\\n --hash=sha256:dd4f05f54a52fb558f1ba9f528228066954fee3ebe629fc1660d874d040ae5a3 \\\n --hash=sha256:de8dad4425a6ca6e4e5e297b27b5c824ecc7581910bf9aee86cb6835e6812aa7 \\\n --hash=sha256:e11e82b744887154b182fd3e7e8512418446501191994dbf9c9fc1f32cc8efd5 \\\n --hash=sha256:e6e73b9e02893c764e7e8d5bb5ce277f1a009cd5243f8228f75f842bf937c534 \\\n --hash=sha256:f73b96c41e3b2adedc34a7356e64c8eb96e03a3782b535e043a986276ce12a49 \\\n --hash=sha256:f93fd8e5c8c0a4aa1f424d6173f14a892044054871c771f8566e4008eaa359d2 \\\n --hash=sha256:fc33c5141b55ed366cfaad382df24fe7dcbc686de5be719b207bb248e3053dc5 \\\n --hash=sha256:fc7de24befaeae77ba923797c7c87834c73648a05a4bde34b3b7e5588973a453 \\\n --hash=sha256:fe562eb1a64e67dd297ccc4f5addea2501664954f2692b69a76449ec7913ecbf\n # via cryptography\ncharset-normalizer==3.4.6 \\\n --hash=sha256:06a7e86163334edfc5d20fe104db92fcd666e5a5df0977cb5680a506fe26cc8e \\\n --hash=sha256:0c173ce3a681f309f31b87125fecec7a5d1347261ea11ebbb856fa6006b23c8c \\\n --hash=sha256:0e28d62a8fc7a1fa411c43bd65e346f3bce9716dc51b897fbe930c5987b402d5 \\\n --hash=sha256:0e901eb1049fdb80f5bd11ed5ea1e498ec423102f7a9b9e4645d5b8204ff2815 \\\n --hash=sha256:11afb56037cbc4b1555a34dd69151e8e069bee82e613a73bef6e714ce733585f \\\n --hash=sha256:150b8ce8e830eb7ccb029ec9ca36022f756986aaaa7956aad6d9ec90089338c0 \\\n --hash=sha256:172985e4ff804a7ad08eebec0a1640ece87ba5041d565fff23c8f99c1f389484 \\\n --hash=sha256:197c1a244a274bb016dd8b79204850144ef77fe81c5b797dc389327adb552407 \\\n --hash=sha256:1ae6b62897110aa7c79ea2f5dd38d1abca6db663687c0b1ad9aed6f6bae3d9d6 \\\n --hash=sha256:1cf0a70018692f85172348fe06d3a4b63f94ecb055e13a00c644d368eb82e5b8 \\\n --hash=sha256:1ed80ff870ca6de33f4d953fda4d55654b9a2b340ff39ab32fa3adbcd718f264 \\\n --hash=sha256:22c6f0c2fbc31e76c3b8a86fba1a56eda6166e238c29cdd3d14befdb4a4e4815 \\\n --hash=sha256:231d4da14bcd9301310faf492051bee27df11f2bc7549bc0bb41fef11b82daa2 \\\n --hash=sha256:259695e2ccc253feb2a016303543d691825e920917e31f894ca1a687982b1de4 \\\n --hash=sha256:2a24157fa36980478dd1770b585c0f30d19e18f4fb0c47c13aa568f871718579 \\\n --hash=sha256:2b1a63e8224e401cafe7739f77efd3f9e7f5f2026bda4aead8e59afab537784f \\\n --hash=sha256:2bd9d128ef93637a5d7a6af25363cf5dec3fa21cf80e68055aad627f280e8afa \\\n --hash=sha256:2e1d8ca8611099001949d1cdfaefc510cf0f212484fe7c565f735b68c78c3c95 \\\n --hash=sha256:2ef7fedc7a6ecbe99969cd09632516738a97eeb8bd7258bf8a0f23114c057dab \\\n --hash=sha256:2f7fdd9b6e6c529d6a2501a2d36b240109e78a8ceaef5687cfcfa2bbe671d297 \\\n --hash=sha256:30f445ae60aad5e1f8bdbb3108e39f6fbc09f4ea16c815c66578878325f8f15a \\\n --hash=sha256:31215157227939b4fb3d740cd23fe27be0439afef67b785a1eb78a3ae69cba9e \\\n --hash=sha256:34315ff4fc374b285ad7f4a0bf7dcbfe769e1b104230d40f49f700d4ab6bbd84 \\\n --hash=sha256:3516bbb8d42169de9e61b8520cbeeeb716f12f4ecfe3fd30a9919aa16c806ca8 \\\n --hash=sha256:3778fd7d7cd04ae8f54651f4a7a0bd6e39a0cf20f801720a4c21d80e9b7ad6b0 \\\n --hash=sha256:39f5068d35621da2881271e5c3205125cc456f54e9030d3f723288c873a71bf9 \\\n --hash=sha256:404a1e552cf5b675a87f0651f8b79f5f1e6fd100ee88dc612f89aa16abd4486f \\\n --hash=sha256:419a9d91bd238052642a51938af8ac05da5b3343becde08d5cdeab9046df9ee1 \\\n --hash=sha256:423fb7e748a08f854a08a222b983f4df1912b1daedce51a72bd24fe8f26a1843 \\\n --hash=sha256:4482481cb0572180b6fd976a4d5c72a30263e98564da68b86ec91f0fe35e8565 \\\n --hash=sha256:461598cd852bfa5a61b09cae2b1c02e2efcd166ee5516e243d540ac24bfa68a7 \\\n --hash=sha256:47955475ac79cc504ef2704b192364e51d0d473ad452caedd0002605f780101c \\\n --hash=sha256:48696db7f18afb80a068821504296eb0787d9ce239b91ca15059d1d3eaacf13b \\\n --hash=sha256:4be9f4830ba8741527693848403e2c457c16e499100963ec711b1c6f2049b7c7 \\\n --hash=sha256:4d1d02209e06550bdaef34af58e041ad71b88e624f5d825519da3a3308e22687 \\\n --hash=sha256:4f41da960b196ea355357285ad1316a00099f22d0929fe168343b99b254729c9 \\\n --hash=sha256:517ad0e93394ac532745129ceabdf2696b609ec9f87863d337140317ebce1c14 \\\n --hash=sha256:51fb3c322c81d20567019778cb5a4a6f2dc1c200b886bc0d636238e364848c89 \\\n --hash=sha256:5273b9f0b5835ff0350c0828faea623c68bfa65b792720c453e22b25cc72930f \\\n --hash=sha256:530d548084c4a9f7a16ed4a294d459b4f229db50df689bfe92027452452943a0 \\\n --hash=sha256:530e8cebeea0d76bdcf93357aa5e41336f48c3dc709ac52da2bb167c5b8271d9 \\\n --hash=sha256:54fae94be3d75f3e573c9a1b5402dc593de19377013c9a0e4285e3d402dd3a2a \\\n --hash=sha256:572d7c822caf521f0525ba1bce1a622a0b85cf47ffbdae6c9c19e3b5ac3c4389 \\\n --hash=sha256:58c948d0d086229efc484fe2f30c2d382c86720f55cd9bc33591774348ad44e0 \\\n --hash=sha256:5d11595abf8dd942a77883a39d81433739b287b6aa71620f15164f8096221b30 \\\n --hash=sha256:5f8ddd609f9e1af8c7bd6e2aca279c931aefecd148a14402d4e368f3171769fd \\\n --hash=sha256:5feb91325bbceade6afab43eb3b508c63ee53579fe896c77137ded51c6b6958e \\\n --hash=sha256:60c74963d8350241a79cb8feea80e54d518f72c26db618862a8f53e5023deaf9 \\\n --hash=sha256:613f19aa6e082cf96e17e3ffd89383343d0d589abda756b7764cf78361fd41dc \\\n --hash=sha256:659a1e1b500fac8f2779dd9e1570464e012f43e580371470b45277a27baa7532 \\\n --hash=sha256:695f5c2823691a25f17bc5d5ffe79fa90972cc34b002ac6c843bb8a1720e950d \\\n --hash=sha256:69dd852c2f0ad631b8b60cfbe25a28c0058a894de5abb566619c205ce0550eae \\\n --hash=sha256:6cceb5473417d28edd20c6c984ab6fee6c6267d38d906823ebfe20b03d607dc2 \\\n --hash=sha256:71be7e0e01753a89cf024abf7ecb6bca2c81738ead80d43004d9b5e3f1244e64 \\\n --hash=sha256:74119174722c4349af9708993118581686f343adc1c8c9c007d59be90d077f3f \\\n --hash=sha256:74a2e659c7ecbc73562e2a15e05039f1e22c75b7c7618b4b574a3ea9118d1557 \\\n --hash=sha256:7504e9b7dc05f99a9bbb4525c67a2c155073b44d720470a148b34166a69c054e \\\n --hash=sha256:79090741d842f564b1b2827c0b82d846405b744d31e84f18d7a7b41c20e473ff \\\n --hash=sha256:7a6967aaf043bceabab5412ed6bd6bd26603dae84d5cb75bf8d9a74a4959d398 \\\n --hash=sha256:7bda6eebafd42133efdca535b04ccb338ab29467b3f7bf79569883676fc628db \\\n --hash=sha256:7edbed096e4a4798710ed6bc75dcaa2a21b68b6c356553ac4823c3658d53743a \\\n --hash=sha256:7f9019c9cb613f084481bd6a100b12e1547cf2efe362d873c2e31e4035a6fa43 \\\n --hash=sha256:802168e03fba8bbc5ce0d866d589e4b1ca751d06edee69f7f3a19c5a9fe6b597 \\\n --hash=sha256:80d0a5615143c0b3225e5e3ef22c8d5d51f3f72ce0ea6fb84c943546c7b25b6c \\\n --hash=sha256:82060f995ab5003a2d6e0f4ad29065b7672b6593c8c63559beefe5b443242c3e \\\n --hash=sha256:836ab36280f21fc1a03c99cd05c6b7af70d2697e374c7af0b61ed271401a72a2 \\\n --hash=sha256:8761ac29b6c81574724322a554605608a9960769ea83d2c73e396f3df896ad54 \\\n --hash=sha256:87725cfb1a4f1f8c2fc9890ae2f42094120f4b44db9360be5d99a4c6b0e03a9e \\\n --hash=sha256:899d28f422116b08be5118ef350c292b36fc15ec2daeb9ea987c89281c7bb5c4 \\\n --hash=sha256:8bc5f0687d796c05b1e28ab0d38a50e6309906ee09375dd3aff6a9c09dd6e8f4 \\\n --hash=sha256:8bea55c4eef25b0b19a0337dc4e3f9a15b00d569c77211fa8cde38684f234fb7 \\\n --hash=sha256:8e5a94886bedca0f9b78fecd6afb6629142fd2605aa70a125d49f4edc6037ee6 \\\n --hash=sha256:90ca27cd8da8118b18a52d5f547859cc1f8354a00cd1e8e5120df3e30d6279e5 \\\n --hash=sha256:92734d4d8d187a354a556626c221cd1a892a4e0802ccb2af432a1d85ec012194 \\\n --hash=sha256:947cf925bc916d90adba35a64c82aace04fa39b46b52d4630ece166655905a69 \\\n --hash=sha256:95b52c68d64c1878818687a473a10547b3292e82b6f6fe483808fb1468e2f52f \\\n --hash=sha256:97d0235baafca5f2b09cf332cc275f021e694e8362c6bb9c96fc9a0eb74fc316 \\\n --hash=sha256:9ca4c0b502ab399ef89248a2c84c54954f77a070f28e546a85e91da627d1301e \\\n --hash=sha256:9cc4fc6c196d6a8b76629a70ddfcd4635a6898756e2d9cac5565cf0654605d73 \\\n --hash=sha256:9cc6e6d9e571d2f863fa77700701dae73ed5f78881efc8b3f9a4398772ff53e8 \\\n --hash=sha256:a056d1ad2633548ca18ffa2f85c202cfb48b68615129143915b8dc72a806a923 \\\n --hash=sha256:a26611d9987b230566f24a0a125f17fe0de6a6aff9f25c9f564aaa2721a5fb88 \\\n --hash=sha256:a4474d924a47185a06411e0064b803c68be044be2d60e50e8bddcc2649957c1f \\\n --hash=sha256:a4ea868bc28109052790eb2b52a9ab33f3aa7adc02f96673526ff47419490e21 \\\n --hash=sha256:a9e68c9d88823b274cf1e72f28cb5dc89c990edf430b0bfd3e2fb0785bfeabf4 \\\n --hash=sha256:aa9cccf4a44b9b62d8ba8b4dd06c649ba683e4bf04eea606d2e94cfc2d6ff4d6 \\\n --hash=sha256:ab30e5e3e706e3063bc6de96b118688cb10396b70bb9864a430f67df98c61ecc \\\n --hash=sha256:ac2393c73378fea4e52aa56285a3d64be50f1a12395afef9cce47772f60334c2 \\\n --hash=sha256:ad8faf8df23f0378c6d527d8b0b15ea4a2e23c89376877c598c4870d1b2c7866 \\\n --hash=sha256:b35b200d6a71b9839a46b9b7fff66b6638bb52fc9658aa58796b0326595d3021 \\\n --hash=sha256:b3694e3f87f8ac7ce279d4355645b3c878d24d1424581b46282f24b92f5a4ae2 \\\n --hash=sha256:b4ff1d35e8c5bd078be89349b6f3a845128e685e751b6ea1169cf2160b344c4d \\\n --hash=sha256:bbc8c8650c6e51041ad1be191742b8b421d05bbd3410f43fa2a00c8db87678e8 \\\n --hash=sha256:bc72863f4d9aba2e8fd9085e63548a324ba706d2ea2c83b260da08a59b9482de \\\n --hash=sha256:bf625105bb9eef28a56a943fec8c8a98aeb80e7d7db99bd3c388137e6eb2d237 \\\n --hash=sha256:c2274ca724536f173122f36c98ce188fd24ce3dad886ec2b7af859518ce008a4 \\\n --hash=sha256:c45a03a4c69820a399f1dda9e1d8fbf3562eda46e7720458180302021b08f778 \\\n --hash=sha256:c8ae56368f8cc97c7e40a7ee18e1cedaf8e780cd8bc5ed5ac8b81f238614facb \\\n --hash=sha256:c907cdc8109f6c619e6254212e794d6548373cc40e1ec75e6e3823d9135d29cc \\\n --hash=sha256:ca0276464d148c72defa8bb4390cce01b4a0e425f3b50d1435aa6d7a18107602 \\\n --hash=sha256:cd5e2801c89992ed8c0a3f0293ae83c159a60d9a5d685005383ef4caca77f2c4 \\\n --hash=sha256:d08ec48f0a1c48d75d0356cea971921848fb620fdeba805b28f937e90691209f \\\n --hash=sha256:d1a2ee9c1499fc8f86f4521f27a973c914b211ffa87322f4ee33bb35392da2c5 \\\n --hash=sha256:d5f5d1e9def3405f60e3ca8232d56f35c98fb7bf581efcc60051ebf53cb8b611 \\\n --hash=sha256:d60377dce4511655582e300dc1e5a5f24ba0cb229005a1d5c8d0cb72bb758ab8 \\\n --hash=sha256:d73beaac5e90173ac3deb9928a74763a6d230f494e4bfb422c217a0ad8e629bf \\\n --hash=sha256:d7de2637729c67d67cf87614b566626057e95c303bc0a55ffe391f5205e7003d \\\n --hash=sha256:dad6e0f2e481fffdcf776d10ebee25e0ef89f16d691f1e5dee4b586375fdc64b \\\n --hash=sha256:dda86aba335c902b6149a02a55b38e96287157e609200811837678214ba2b1db \\\n --hash=sha256:df01808ee470038c3f8dc4f48620df7225c49c2d6639e38f96e6d6ac6e6f7b0e \\\n --hash=sha256:e1f6e2f00a6b8edb562826e4632e26d063ac10307e80f7461f7de3ad8ef3f077 \\\n --hash=sha256:e25369dc110d58ddf29b949377a93e0716d72a24f62bad72b2b39f155949c1fd \\\n --hash=sha256:e3c701e954abf6fc03a49f7c579cc80c2c6cc52525340ca3186c41d3f33482ef \\\n --hash=sha256:e5bcc1a1ae744e0bb59641171ae53743760130600da8db48cbb6e4918e186e4e \\\n --hash=sha256:e68c14b04827dd76dcbd1aeea9e604e3e4b78322d8faf2f8132c7138efa340a8 \\\n --hash=sha256:e8aeb10fcbe92767f0fa69ad5a72deca50d0dca07fbde97848997d778a50c9fe \\\n --hash=sha256:e985a16ff513596f217cee86c21371b8cd011c0f6f056d0920aa2d926c544058 \\\n --hash=sha256:ecbbd45615a6885fe3240eb9db73b9e62518b611850fdf8ab08bd56de7ad2b17 \\\n --hash=sha256:ee4ec14bc1680d6b0afab9aea2ef27e26d2024f18b24a2d7155a52b60da7e833 \\\n --hash=sha256:ef5960d965e67165d75b7c7ffc60a83ec5abfc5c11b764ec13ea54fbef8b4421 \\\n --hash=sha256:f0cdaecd4c953bfae0b6bb64910aaaca5a424ad9c72d85cb88417bb9814f7550 \\\n --hash=sha256:f1ce721c8a7dfec21fcbdfe04e8f68174183cf4e8188e0645e92aa23985c57ff \\\n --hash=sha256:f50498891691e0864dc3da965f340fada0771f6142a378083dc4608f4ea513e2 \\\n --hash=sha256:f5ea69428fa1b49573eef0cc44a1d43bebd45ad0c611eb7d7eac760c7ae771bc \\\n --hash=sha256:f61aa92e4aad0be58eb6eb4e0c21acf32cf8065f4b2cae5665da756c4ceef982 \\\n --hash=sha256:f6e4333fb15c83f7d1482a76d45a0818897b3d33f00efd215528ff7c51b8e35d \\\n --hash=sha256:f820f24b09e3e779fe84c3c456cb4108a7aa639b0d1f02c28046e11bfcd088ed \\\n --hash=sha256:f98059e4fcd3e3e4e2d632b7cf81c2faae96c43c60b569e9c621468082f1d104 \\\n --hash=sha256:fcce033e4021347d80ed9c66dcf1e7b1546319834b74445f561d2e2221de5659\n # via requests\ncryptography==46.0.6 \\\n --hash=sha256:02fad249cb0e090b574e30b276a3da6a149e04ee2f049725b1f69e7b8351ec70 \\\n --hash=sha256:063b67749f338ca9c5a0b7fe438a52c25f9526b851e24e6c9310e7195aad3b4d \\\n --hash=sha256:12cae594e9473bca1a7aceb90536060643128bb274fcea0fc459ab90f7d1ae7a \\\n --hash=sha256:12f0fa16cc247b13c43d56d7b35287ff1569b5b1f4c5e87e92cc4fcc00cd10c0 \\\n --hash=sha256:22259338084d6ae497a19bae5d4c66b7ca1387d3264d1c2c0e72d9e9b6a77b97 \\\n --hash=sha256:26031f1e5ca62fcb9d1fcb34b2b60b390d1aacaa15dc8b895a9ed00968b97b30 \\\n --hash=sha256:27550628a518c5c6c903d84f637fbecf287f6cb9ced3804838a1295dc1fd0759 \\\n --hash=sha256:2b417edbe8877cda9022dde3a008e2deb50be9c407eef034aeeb3a8b11d9db3c \\\n --hash=sha256:2ea0f37e9a9cf0df2952893ad145fd9627d326a59daec9b0802480fa3bcd2ead \\\n --hash=sha256:2ef9e69886cbb137c2aef9772c2e7138dc581fad4fcbcf13cc181eb5a3ab6275 \\\n --hash=sha256:341359d6c9e68834e204ceaf25936dffeafea3829ab80e9503860dcc4f4dac58 \\\n --hash=sha256:380343e0653b1c9d7e1f55b52aaa2dbb2fdf2730088d48c43ca1c7c0abb7cc2f \\\n --hash=sha256:3c21d92ed15e9cfc6eb64c1f5a0326db22ca9c2566ca46d845119b45b4400361 \\\n --hash=sha256:3dfa6567f2e9e4c5dceb8ccb5a708158a2a871052fa75c8b78cb0977063f1507 \\\n --hash=sha256:456b3215172aeefb9284550b162801d62f5f264a081049a3e94307fe20792cfa \\\n --hash=sha256:4668298aef7cddeaf5c6ecc244c2302a2b8e40f384255505c22875eebb47888b \\\n --hash=sha256:50575a76e2951fe7dbd1f56d181f8c5ceeeb075e9ff88e7ad997d2f42af06e7b \\\n --hash=sha256:639301950939d844a9e1c4464d7e07f902fe9a7f6b215bb0d4f28584729935d8 \\\n --hash=sha256:64235194bad039a10bb6d2d930ab3323baaec67e2ce36215fd0952fad0930ca8 \\\n --hash=sha256:6617f67b1606dfd9fe4dbfa354a9508d4a6d37afe30306fe6c101b7ce3274b72 \\\n --hash=sha256:67177e8a9f421aa2d3a170c3e56eca4e0128883cf52a071a7cbf53297f18b175 \\\n --hash=sha256:6728c49e3b2c180ef26f8e9f0a883a2c585638db64cf265b49c9ba10652d430e \\\n --hash=sha256:6739d56300662c468fddb0e5e291f9b4d084bead381667b9e654c7dd81705124 \\\n --hash=sha256:69cf0056d6947edc6e6760e5f17afe4bea06b56a9ac8a06de9d2bd6b532d4f3a \\\n --hash=sha256:760997a4b950ff00d418398ad73fbc91aa2894b5c1db7ccb45b4f68b42a63b3c \\\n --hash=sha256:79e865c642cfc5c0b3eb12af83c35c5aeff4fa5c672dc28c43721c2c9fdd2f0f \\\n --hash=sha256:7e6142674f2a9291463e5e150090b95a8519b2fb6e6aaec8917dd8d094ce750d \\\n --hash=sha256:7f417f034f91dcec1cb6c5c35b07cdbb2ef262557f701b4ecd803ee8cefed4f4 \\\n --hash=sha256:7f6690b6c55e9c5332c0b59b9c8a3fb232ebf059094c17f9019a51e9827df91c \\\n --hash=sha256:8927ccfbe967c7df312ade694f987e7e9e22b2425976ddbf28271d7e58845290 \\\n --hash=sha256:8ce35b77aaf02f3b59c90b2c8a05c73bac12cea5b4e8f3fbece1f5fddea5f0ca \\\n --hash=sha256:8e7304c4f4e9490e11efe56af6713983460ee0780f16c63f219984dab3af9d2d \\\n --hash=sha256:90e5f0a7b3be5f40c3a0a0eafb32c681d8d2c181fc2a1bdabe9b3f611d9f6b1a \\\n --hash=sha256:97c8115b27e19e592a05c45d0dd89c57f81f841cc9880e353e0d3bf25b2139ed \\\n --hash=sha256:9a693028b9cbe51b5a1136232ee8f2bc242e4e19d456ded3fa7c86e43c713b4a \\\n --hash=sha256:9a9c42a2723999a710445bc0d974e345c32adfd8d2fac6d8a251fa829ad31cfb \\\n --hash=sha256:a3e84d5ec9ba01f8fd03802b2147ba77f0c8f2617b2aff254cedd551844209c8 \\\n --hash=sha256:aad75154a7ac9039936d50cf431719a2f8d4ed3d3c277ac03f3339ded1a5e707 \\\n --hash=sha256:b12c6b1e1651e42ab5de8b1e00dc3b6354fdfd778e7fa60541ddacc27cd21410 \\\n --hash=sha256:b928a3ca837c77a10e81a814a693f2295200adb3352395fad024559b7be7a736 \\\n --hash=sha256:bcb87663e1f7b075e48c3be3ecb5f0b46c8fc50b50a97cf264e7f60242dca3f2 \\\n --hash=sha256:c797e2517cb7880f8297e2c0f43bb910e91381339336f75d2c1c2cbf811b70b4 \\\n --hash=sha256:c89eb37fae9216985d8734c1afd172ba4927f5a05cfd9bf0e4863c6d5465b013 \\\n --hash=sha256:cdcd3edcbc5d55757e5f5f3d330dd00007ae463a7e7aa5bf132d1f22a4b62b19 \\\n --hash=sha256:d24c13369e856b94892a89ddf70b332e0b70ad4a5c43cf3e9cb71d6d7ffa1f7b \\\n --hash=sha256:d4e4aadb7fc1f88687f47ca20bb7227981b03afaae69287029da08096853b738 \\\n --hash=sha256:d9528b535a6c4f8ff37847144b8986a9a143585f0540fbcb1a98115b543aa463 \\\n --hash=sha256:ed3775295fb91f70b4027aeba878d79b3e55c0b3e97eaa4de71f8f23a9f2eb77 \\\n --hash=sha256:ed418c37d095aeddf5336898a132fba01091f0ac5844e3e8018506f014b6d2c4\n # via google-auth\ndatabricks-sdk==0.102.0 \\\n --hash=sha256:75d1253276ee8f3dd5e7b00d62594b7051838435e618f74a8570a6dbd723ec12 \\\n --hash=sha256:8fa5f82317ee27cc46323c6e2543d2cfefb4468653f92ba558271043c6f72fb9\n # via -r requirements.in\ngoogle-auth==2.49.1 \\\n --hash=sha256:16d40da1c3c5a0533f57d268fe72e0ebb0ae1cc3b567024122651c045d879b64 \\\n --hash=sha256:195ebe3dca18eddd1b3db5edc5189b76c13e96f29e73043b923ebcf3f1a860f7\n # via databricks-sdk\nidna==3.11 \\\n --hash=sha256:771a87f49d9defaf64091e6e6fe9c18d4833f140bd19464795bc32d966ca37ea \\\n --hash=sha256:795dafcc9c04ed0c1fb032c2aa73654d8e8c5023a7df64a53f39190ada629902\n # via requests\nnumpy==2.4.3 \\\n --hash=sha256:0200b25c687033316fb39f0ff4e3e690e8957a2c3c8d22499891ec58c37a3eb5 \\\n --hash=sha256:0448e7f9caefb34b4b7dd2b77f21e8906e5d6f0365ad525f9f4f530b13df2afc \\\n --hash=sha256:0a195f4216be9305a73c0e91c9b026a35f2161237cf1c6de9b681637772ea657 \\\n --hash=sha256:0a60e17a14d640f49146cb38e3f105f571318db7826d9b6fef7e4dce758faecd \\\n --hash=sha256:120df8c0a81ebbf5b9020c91439fccd85f5e018a927a39f624845be194a2be02 \\\n --hash=sha256:148d59127ac95979d6f07e4d460f934ebdd6eed641db9c0db6c73026f2b2101a \\\n --hash=sha256:1ec84fd7c8e652b0f4aaaf2e6e9cc8eaa9b1b80a537e06b2e3a2fb176eedcb26 \\\n --hash=sha256:22654fe6be0e5206f553a9250762c653d3698e46686eee53b399ab90da59bd92 \\\n --hash=sha256:22c31dc07025123aedf7f2db9e91783df13f1776dc52c6b22c620870dc0fab22 \\\n --hash=sha256:23b46bb6d8ecb68b58c09944483c135ae5f0e9b8d8858ece5e4ead783771d2a9 \\\n --hash=sha256:2629289168f4897a3c4e23dc98d6f1731f0fc0fe52fb9db19f974041e4cc12b9 \\\n --hash=sha256:26952e18d82a1dbbc2f008d402021baa8d6fc8e84347a2072a25e08b46d698b9 \\\n --hash=sha256:29363fbfa6f8ee855d7569c96ce524845e3d726d6c19b29eceec7dd555dab152 \\\n --hash=sha256:297837823f5bc572c5f9379b0c9f3a3365f08492cbdc33bcc3af174372ebb168 \\\n --hash=sha256:2abad5c7fef172b3377502bde47892439bae394a71bc329f31df0fd829b41a9e \\\n --hash=sha256:2b3f8d2c4589b1a2028d2a770b0fc4d1f332fb5e01521f4de3199a896d158ddd \\\n --hash=sha256:2ddb7919366ee468342b91dea2352824c25b55814a987847b6c52003a7c97f15 \\\n --hash=sha256:2e03c05abaee1f672e9d67bc858f300b5ccba1c21397211e8d77d98350972093 \\\n --hash=sha256:32e3bef222ad6b052280311d1d60db8e259e4947052c3ae7dd6817451fc8a4c5 \\\n --hash=sha256:33b3bf58ee84b172c067f56aeadc7ee9ab6de69c5e800ab5b10295d54c581adb \\\n --hash=sha256:45f003dbdffb997a03da2d1d0cb41fbd24a87507fb41605c0420a3db5bd4667b \\\n --hash=sha256:483a201202b73495f00dbc83796c6ae63137a9bdade074f7648b3e32613412dd \\\n --hash=sha256:48da3a4ee1336454b07497ff7ec83903efa5505792c4e6d9bf83d99dc07a1e18 \\\n --hash=sha256:4b42639cdde6d24e732ff823a3fa5b701d8acad89c4142bc1d0bd6dc85200ba5 \\\n --hash=sha256:4bd4741a6a676770e0e97fe9ab2e51de01183df3dcbcec591d26d331a40de950 \\\n --hash=sha256:4d382735cecd7bcf090172489a525cd7d4087bc331f7df9f60ddc9a296cf208e \\\n --hash=sha256:52077feedeff7c76ed7c9f1a0428558e50825347b7545bbb8523da2cd55c547a \\\n --hash=sha256:54f29b877279d51e210e0c80709ee14ccbbad647810e8f3d375561c45ef613dd \\\n --hash=sha256:5884ce5c7acfae1e4e1b6fde43797d10aa506074d25b531b4f54bde33c0c31d4 \\\n --hash=sha256:5e10da9e93247e554bb1d22f8edc51847ddd7dde52d85ce31024c1b4312bfba0 \\\n --hash=sha256:61b0cbabbb6126c8df63b9a3a0c4b1f44ebca5e12ff6997b80fcf267fb3150ef \\\n --hash=sha256:65f3c2455188f09678355f5cae1f959a06b778bc66d535da07bf2ef20cd319d5 \\\n --hash=sha256:679f2a834bae9020f81534671c56fd0cc76dd7e5182f57131478e23d0dc59e24 \\\n --hash=sha256:6bd06731541f89cdc01b261ba2c9e037f1543df7472517836b78dfb15bd6e476 \\\n --hash=sha256:715de7f82e192e8cae5a507a347d97ad17598f8e026152ca97233e3666daaa71 \\\n --hash=sha256:737f630a337364665aba3b5a77e56a68cc42d350edd010c345d65a3efa3addcc \\\n --hash=sha256:7395e69ff32526710748f92cd8c9849b361830968ea3e24a676f272653e8983e \\\n --hash=sha256:76dbb9d4e43c16cf9aa711fcd8de1e2eeb27539dcefb60a1d5e9f12fae1d1ed8 \\\n --hash=sha256:76f0f283506c28b12bba319c0fab98217e9f9b54e6160e9c79e9f7348ba32e9c \\\n --hash=sha256:77e76d932c49a75617c6d13464e41203cd410956614d0a0e999b25e9e8d27eec \\\n --hash=sha256:7aa4e54f6469300ebca1d9eb80acd5253cdfa36f2c03d79a35883687da430875 \\\n --hash=sha256:7d1ce23cce91fcea443320a9d0ece9b9305d4368875bab09538f7a5b4131938a \\\n --hash=sha256:7e58765ad74dcebd3ef0208a5078fba32dc8ec3578fe84a604432950cd043d79 \\\n --hash=sha256:7f3408ff897f8ab07a07fbe2823d7aee6ff644c097cc1f90382511fe982f647f \\\n --hash=sha256:8ba7b51e71c05aa1f9bc3641463cd82308eab40ce0d5c7e1fd4038cbf9938147 \\\n --hash=sha256:8e236dbda4e1d319d681afcbb136c0c4a8e0f1a5c58ceec2adebb547357fe857 \\\n --hash=sha256:94f3c4a151a2e529adf49c1d54f0f57ff8f9b233ee4d44af623a81553ab86368 \\\n --hash=sha256:9684823a78a6cd6ad7511fc5e25b07947d1d5b5e2812c93fe99d7d4195130720 \\\n --hash=sha256:a016db5c5dba78fa8fe9f5d80d6708f9c42ab087a739803c0ac83a43d686a470 \\\n --hash=sha256:a111698b4a3f8dcbe54c64a7708f049355abd603e619013c346553c1fd4ca90b \\\n --hash=sha256:a1988292870c7cb9d0ebb4cc96b4d447513a9644801de54606dc7aabf2b7d920 \\\n --hash=sha256:a315e5234d88067f2d97e1f2ef670a7569df445d55400f1e33d117418d008d52 \\\n --hash=sha256:a749547700de0a20a6718293396ec237bb38218049cfce788e08fcb716e8cf73 \\\n --hash=sha256:a97cbf7e905c435865c2d939af3d93f99d18eaaa3cabe4256f4304fb51604349 \\\n --hash=sha256:abdce0f71dcb4a00e4e77f3faf05e4616ceccfe72ccaa07f47ee79cda3b7b0f4 \\\n --hash=sha256:b346845443716c8e542d54112966383b448f4a3ba5c66409771b8c0889485dd3 \\\n --hash=sha256:b44fd60341c4d9783039598efadd03617fa28d041fc37d22b62d08f2027fa0e7 \\\n --hash=sha256:bb2e3cf95854233799013779216c57e153c1ee67a0bf92138acca0e429aefaee \\\n --hash=sha256:bc71942c789ef415a37f0d4eab90341425a00d538cd0642445d30b41023d3395 \\\n --hash=sha256:be3b8487d725a77acccc9924f65fd8bce9af7fac8c9820df1049424a2115af6c \\\n --hash=sha256:c59020932feb24ed49ffd03704fbab89f22aa9c0d4b180ff45542fe8918f5611 \\\n --hash=sha256:c6b124bfcafb9e8d3ed09130dbee44848c20b3e758b6bbf006e641778927c028 \\\n --hash=sha256:c9619741e9da2059cd9c3f206110b97583c7152c1dc9f8aafd4beb450ac1c89d \\\n --hash=sha256:cd32fbacb9fd1bf041bf8e89e4576b6f00b895f06d00914820ae06a616bdfef7 \\\n --hash=sha256:d1b90d840b25874cf5cd20c219af10bac3667db3876d9a495609273ebe679070 \\\n --hash=sha256:d213c7e6e8d211888cc359bab7199670a00f5b82c0978b9d1c75baf1eddbeac0 \\\n --hash=sha256:d5f51900414fc9204a0e0da158ba2ac52b75656e7dce7e77fb9f84bfa343b4cc \\\n --hash=sha256:d71e379452a2f670ccb689ec801b1218cd3983e253105d6e83780967e899d687 \\\n --hash=sha256:d84f0f881cb2225c2dfd7f78a10a5645d487a496c6668d6cc39f0f114164f3d0 \\\n --hash=sha256:decb0eb8a53c3b009b0962378065589685d66b23467ef5dac16cbe818afde27f \\\n --hash=sha256:e7dd01a46700b1967487141a66ac1a3cf0dd8ebf1f08db37d46389401512ca97 \\\n --hash=sha256:eb610595dd91560905c132c709412b512135a60f1851ccbd2c959e136431ff67\n # via pandas\npandas==3.0.1 \\\n --hash=sha256:06aff2ad6f0b94a17822cf8b83bbb563b090ed82ff4fe7712db2ce57cd50d9b8 \\\n --hash=sha256:0ab749dfba921edf641d4036c4c21c0b3ea70fea478165cb98a998fb2a261955 \\\n --hash=sha256:0f463ebfd8de7f326d38037c7363c6dacb857c5881ab8961fb387804d6daf2f7 \\\n --hash=sha256:108dd1790337a494aa80e38def654ca3f0968cf4f362c85f44c15e471667102d \\\n --hash=sha256:15860b1fdb1973fffade772fdb931ccf9b2f400a3f5665aef94a00445d7d8dd5 \\\n --hash=sha256:1849f0bba9c8a2fb0f691d492b834cc8dadf617e29015c66e989448d58d011ee \\\n --hash=sha256:1ff8cf1d2896e34343197685f432450ec99a85ba8d90cce2030c5eee2ef98791 \\\n --hash=sha256:24ba315ba3d6e5806063ac6eb717504e499ce30bd8c236d8693a5fd3f084c796 \\\n --hash=sha256:331ca75a2f8672c365ae25c0b29e46f5ac0c6551fdace8eec4cd65e4fac271ff \\\n --hash=sha256:356e5c055ed9b0da1580d465657bc7d00635af4fd47f30afb23025352ba764d1 \\\n --hash=sha256:3b66857e983208654294bb6477b8a63dee26b37bdd0eb34d010556e91261784f \\\n --hash=sha256:406ce835c55bac912f2a0dcfaf27c06d73c6b04a5dde45f1fd3169ce31337389 \\\n --hash=sha256:4186a699674af418f655dbd420ed87f50d56b4cd6603784279d9eef6627823c8 \\\n --hash=sha256:44f1364411d5670efa692b146c748f4ed013df91ee91e9bec5677fb1fd58b937 \\\n --hash=sha256:476f84f8c20c9f5bc47252b66b4bb25e1a9fc2fa98cead96744d8116cb85771d \\\n --hash=sha256:4a68773d5a778afb31d12e34f7dd4612ab90de8c6fb1d8ffe5d4a03b955082a1 \\\n --hash=sha256:4e1b677accee34a09e0dc2ce5624e4a58a1870ffe56fc021e9caf7f23cd7668f \\\n --hash=sha256:5272627187b5d9c20e55d27caf5f2cd23e286aba25cadf73c8590e432e2b7262 \\\n --hash=sha256:532527a701281b9dd371e2f582ed9094f4c12dd9ffb82c0c54ee28d8ac9520c4 \\\n --hash=sha256:536232a5fe26dd989bd633e7a0c450705fdc86a207fec7254a55e9a22950fe43 \\\n --hash=sha256:56cf59638bf24dc9bdf2154c81e248b3289f9a09a6d04e63608c159022352749 \\\n --hash=sha256:58eeb1b2e0fb322befcf2bbc9ba0af41e616abadb3d3414a6bc7167f6cbfce32 \\\n --hash=sha256:5ae2ab1f166668b41e770650101e7090824fd34d17915dd9cd479f5c5e0065e9 \\\n --hash=sha256:661e0f665932af88c7877f31da0dc743fe9c8f2524bdffe23d24fdcb67ef9d56 \\\n --hash=sha256:6bf0603c2e30e2cafac32807b06435f28741135cb8697eae8b28c7d492fc7d76 \\\n --hash=sha256:6c426422973973cae1f4a23e51d4ae85974f44871b24844e4f7de752dd877098 \\\n --hash=sha256:75e6e292ff898679e47a2199172593d9f6107fd2dd3617c22c2946e97d5df46e \\\n --hash=sha256:830994d7e1f31dd7e790045235605ab61cff6c94defc774547e8b7fdfbff3dc7 \\\n --hash=sha256:84f0904a69e7365f79a0c77d3cdfccbfb05bf87847e3a51a41e1426b0edb9c79 \\\n --hash=sha256:85fe4c4df62e1e20f9db6ebfb88c844b092c22cd5324bdcf94bfa2fc1b391221 \\\n --hash=sha256:93325b0fe372d192965f4cca88d97667f49557398bbf94abdda3bf1b591dbe66 \\\n --hash=sha256:94f87a04984d6b63788327cd9f79dda62b7f9043909d2440ceccf709249ca988 \\\n --hash=sha256:97ca08674e3287c7148f4858b01136f8bdfe7202ad25ad04fec602dd1d29d132 \\\n --hash=sha256:9832c2c69da24b602c32e0c7b1b508a03949c18ba08d4d9f1c1033426685b447 \\\n --hash=sha256:99d0f92ed92d3083d140bf6b97774f9f13863924cf3f52a70711f4e7588f9d0a \\\n --hash=sha256:9d810036895f9ad6345b8f2a338dd6998a74e8483847403582cab67745bff821 \\\n --hash=sha256:9fea306c783e28884c29057a1d9baa11a349bbf99538ec1da44c8476563d1b25 \\\n --hash=sha256:a64ce8b0f2de1d2efd2ae40b0abe7f8ae6b29fbfb3812098ed5a6f8e235ad9bf \\\n --hash=sha256:a8d37a43c52917427e897cb2e429f67a449327394396a81034a4449b99afda59 \\\n --hash=sha256:a9cabbdcd03f1b6cd254d6dda8ae09b0252524be1592594c00b7895916cb1324 \\\n --hash=sha256:b03f91ae8c10a85c1613102c7bef5229b5379f343030a3ccefeca8a33414cf35 \\\n --hash=sha256:b8e36891080b87823aff3640c78649b91b8ff6eea3c0d70aeabd72ea43ab069b \\\n --hash=sha256:c1a9f55e0f46951874b863d1f3906dcb57df2d9be5c5847ba4dfb55b2c815249 \\\n --hash=sha256:c3d288439e11b5325b02ae6e9cc83e6805a62c40c5a6220bea9beb899c073b1c \\\n --hash=sha256:cd9af1276b5ca9e298bd79a26bda32fa9cc87ed095b2a9a60978d2ca058eaf87 \\\n --hash=sha256:d54855f04f8246ed7b6fc96b05d4871591143c46c0b6f4af874764ed0d2d6f06 \\\n --hash=sha256:de09668c1bf3b925c07e5762291602f0d789eca1b3a781f99c1c78f6cac0e7ea \\\n --hash=sha256:eca8b4510f6763f3d37359c2105df03a7a221a508f30e396a51d0713d462e68a\n # via -r requirements.in\nprotobuf==6.33.6 \\\n --hash=sha256:0cd27b587afca21b7cfa59a74dcbd48a50f0a6400cfb59391340ad729d91d326 \\\n --hash=sha256:77179e006c476e69bf8e8ce866640091ec42e1beb80b213c3900006ecfba6901 \\\n --hash=sha256:7d29d9b65f8afef196f8334e80d6bc1d5d4adedb449971fefd3723824e6e77d3 \\\n --hash=sha256:9720e6961b251bde64edfdab7d500725a2af5280f3f4c87e57c0208376aa8c3a \\\n --hash=sha256:a6768d25248312c297558af96a9f9c929e8c4cee0659cb07e780731095f38135 \\\n --hash=sha256:bd56799fb262994b2c2faa1799693c95cc2e22c62f56fb43af311cae45d26f0e \\\n --hash=sha256:c96c37eec15086b79762ed265d59ab204dabc53056e3443e702d2681f4b39ce3 \\\n --hash=sha256:e2afbae9b8e1825e3529f88d514754e094278bb95eadc0e199751cdd9a2e82a2 \\\n --hash=sha256:e9db7e292e0ab79dd108d7f1a94fe31601ce1ee3f7b79e0692043423020b0593 \\\n --hash=sha256:f443a394af5ed23672bc6c486be138628fbe5c651ccbc536873d7da23d1868cf\n # via databricks-sdk\npyasn1==0.6.3 \\\n --hash=sha256:697a8ecd6d98891189184ca1fa05d1bb00e2f84b5977c481452050549c8a72cf \\\n --hash=sha256:a80184d120f0864a52a073acc6fc642847d0be408e7c7252f31390c0f4eadcde\n # via pyasn1-modules\npyasn1-modules==0.4.2 \\\n --hash=sha256:29253a9207ce32b64c3ac6600edc75368f98473906e8fd1043bd6b5b1de2c14a \\\n --hash=sha256:677091de870a80aae844b1ca6134f54652fa2c8c5a52aa396440ac3106e941e6\n # via google-auth\npycparser==3.0 \\\n --hash=sha256:600f49d217304a5902ac3c37e1281c9fe94e4d0489de643a9504c5cdfdfc6b29 \\\n --hash=sha256:b727414169a36b7d524c1c3e31839a521725078d7b2ff038656844266160a992\n # via cffi\npython-dateutil==2.9.0.post0 \\\n --hash=sha256:37dd54208da7e1cd875388217d5e00ebd4179249f90fb72437e91a35459a0ad3 \\\n --hash=sha256:a8b2bc7bffae282281c8140a97d3aa9c14da0b136dfe83f850eea9a5f7470427\n # via pandas\nrequests==2.33.0 \\\n --hash=sha256:3324635456fa185245e24865e810cecec7b4caf933d7eb133dcde67d48cee69b \\\n --hash=sha256:c7ebc5e8b0f21837386ad0e1c8fe8b829fa5f544d8df3b2253bff14ef29d7652\n # via\n # -r requirements.in\n # databricks-sdk\nsix==1.17.0 \\\n --hash=sha256:4721f391ed90541fddacab5acf947aa0d3dc7d27b2e1e8eda2be8970586c3274 \\\n --hash=sha256:ff70335d468e7eb6ec65b95b99d3a2836546063f63acc5171de367e834932a81\n # via python-dateutil\nurllib3==2.6.3 \\\n --hash=sha256:1b62b6884944a57dbe321509ab94fd4d3b307075e0c2eae991ac71ee15ad38ed \\\n --hash=sha256:bf272323e553dfb2e87d9bfd225ca7b0f467b919d7bbd355436d3fd37cb0acd4\n # via requests\n" }, { "path": "setup.py", "content": "from setuptools import setup, find_packages\n\n#python setup.py clean --all bdist_wheel\nsetup(\n #this will be the package name you will see, e.g. the output of 'conda list' in anaconda prompt\n name = 'dbdemos',\n #some version number you may wish to add - increment this after every update\n version='0.6.34',\n author=\"Databricks\",\n author_email=[\"quentin.ambard@databricks.com\", \"cal.reynolds@databricks.com\"],\n description=\"Install databricks demos: notebooks, Delta Live Table Pipeline, DBSQL Dashboards, ML Models etc.\",\n long_description=open(\"README.md\").read(),\n long_description_content_type=\"text/markdown\",\n url=\"https://github.com/databricks-demos/dbdemos\",\n packages=find_packages(exclude=[\"tests\", \"tests.*\"]),\n setup_requires=[\"wheel\"],\n include_package_data=True,\n install_requires=[\"requests\", \"pandas\", \"databricks-sdk>=0.38.0\"],\n license=\"Databricks License\",\n license_files = ('LICENSE',),\n tests_require=[\n \"pytest\"\n ],\n python_requires=\">=3.7\"\n)\n" }, { "path": "test/__init__.py", "content": "# Empty file to mark the directory as a Python package " }, { "path": "test/test2.html", "content": "\n \n
\n \n

Your demo Lakehouse - Banking Fraud is ready!

\n Build your Banking platform and detect Fraud in real-time

\n Start with the first notebook lakehouse-fsi-fraud/00-FSI-fraud-detection-introduction-lakehouse using the cluster dbdemos-lakehouse-fsi-fraud-quentin_ambard\n

Notebook installed:

\n
00-FSI-fraud-detection-introduction-lakehouse: Lakehouse - Fraud introduction
01-Data-ingestion
\n
01.1-DLT-fraud-detection-SQL: Ingest data with Delta Live Table
02-Data-governance
\n
02-UC-data-governance-ACL-fsi-fraud: Governance with Unity Catalog
03-BI-data-warehousing
\n
03-BI-Datawarehousing-fraud: Datawarehousing & BI / Dashboarding
04-Data-Science-ML
\n
04.1-AutoML-FSI-fraud: Build Fraud prediction model (AutoML)
04.2-automl-generated-notebook-fraud: Explore Fraud Prediction generated model
04.3-Model-serving-realtime-inference-fraud: Infer Fraud in realtime - serverless API
04.4-Upgrade-to-imbalance-and-xgboost-model-fraud: Upgrade our model to XGboost
04.5-AB-testing-model-serving-fraud: Roll-out our new model with A/B testing.
05-Workflow-orchestration
\n
05-Workflow-orchestration-fsi-fraud: Orchestrate churn prevention with Workflow

Delta Live Table Pipelines

DBSQL Dashboards

Initialization job started

\n
\n We started a job to initialize your demo data (for DBSQL Dashboards & Delta Live Table). \n Please wait for the job completion to be able to access the dataset & dashboards...\n
\n

Interactive cluster for the demo:

\n dbdemos-lakehouse-fsi-fraud-quentin_ambard. You can refresh your demo cluster with:\n
\n dbdemos.create_cluster('lakehouse-fsi-fraud')\n
" }, { "path": "test/test_installer.py", "content": "import dbdemos\nfrom dbdemos.conf import DemoNotebook\n\nfrom dbdemos.installer import Installer\nfrom dbdemos.installer_report import InstallerReport\n\n\n\ndef test_html():\n installer = InstallerReport(\"http://localhost\")\n demo_name = \"lakehouse-fsi-fraud\"\n description = \"Build your Banking platform and detect Fraud in real-time\"\n title = \"Lakehouse - Banking Fraud\"\n install_path = \"/Users/quentin.ambard@databricks.com/test_install_quentin\"\n\n\n notebooks =[{\"path\": \"_resources/00-setup\", \"title\": \"Prep data\", \"description\": \"Helpers & setup.\", \"pre_run\": False, \"publish_on_website\": False, \"add_cluster_setup_cell\": False, \"parameters\": {}, \"depends_on_previous\": True},\n {\"path\": \"_resources/01-load-data\", \"title\": \"Prep data\", \"description\": \"Prep data for demo.\", \"pre_run\": False, \"publish_on_website\": False, \"add_cluster_setup_cell\": False, \"parameters\": {}, \"depends_on_previous\": True},\n {\"path\": \"00-FSI-fraud-detection-introduction-lakehouse\", \"title\": \"Lakehouse - Fraud introduction\", \"description\": \"Start here to explore the Lakehouse.\", \"pre_run\": False, \"publish_on_website\": True, \"add_cluster_setup_cell\": False, \"parameters\": {}, \"depends_on_previous\": True},\n {\"path\": \"01-Data-ingestion/01.1-DLT-fraud-detection-SQL\", \"title\": \"Ingest data with Delta Live Table\", \"description\": \"SQL DLT pipeline to ingest data & build clean tables.\", \"pre_run\": True, \"publish_on_website\": True, \"add_cluster_setup_cell\": False, \"parameters\": {}, \"depends_on_previous\": True},\n {\"path\": \"02-Data-governance/02-UC-data-governance-ACL-fsi-fraud\", \"title\": \"Governance with Unity Catalog\", \"description\": \"Secure your tables, lineage, auditlog...\", \"pre_run\": True, \"publish_on_website\": True, \"add_cluster_setup_cell\": True, \"parameters\": {}, \"depends_on_previous\": True},\n {\"path\": \"03-BI-data-warehousing/03-BI-Datawarehousing-fraud\", \"title\": \"Datawarehousing & BI / Dashboarding\", \"description\": \"Run interactive queries on top of your data\", \"pre_run\": False, \"publish_on_website\": True, \"add_cluster_setup_cell\": False, \"parameters\": {}, \"depends_on_previous\": True},\n {\"path\": \"04-Data-Science-ML/04.1-AutoML-FSI-fraud\", \"title\": \"Build Fraud prediction model (AutoML)\", \"description\": \"Leverage Databricks AutoML to create a Fraud model in a few clicks\", \"pre_run\": True, \"publish_on_website\": True, \"add_cluster_setup_cell\": True, \"parameters\": {}, \"depends_on_previous\": True},\n {\"path\": \"04-Data-Science-ML/04.2-automl-generated-notebook-fraud\", \"title\": \"Explore Fraud Prediction generated model\", \"description\": \"Explore the best Fraud model generated by AutoML and deploy it in production.\", \"pre_run\": True, \"publish_on_website\": True, \"add_cluster_setup_cell\": True, \"parameters\": {\"shap_enabled\": \"true\"}, \"depends_on_previous\": True},\n {\"path\": \"04-Data-Science-ML/04.3-Model-serving-realtime-inference-fraud\", \"title\": \"Infer Fraud in realtime - serverless API\", \"description\": \"Once your model is deployed, run low latency inferences.\", \"pre_run\": True, \"publish_on_website\": True, \"add_cluster_setup_cell\": True, \"parameters\": {}, \"depends_on_previous\": True},\n {\"path\": \"04-Data-Science-ML/04.4-Upgrade-to-imbalance-and-xgboost-model-fraud\", \"title\": \"Upgrade our model to XGboost\", \"description\": \"Improve AutoML model to handle imbalanced data.\", \"pre_run\": True, \"publish_on_website\": True, \"add_cluster_setup_cell\": True, \"parameters\": {}, \"depends_on_previous\": True},\n {\"path\": \"04-Data-Science-ML/04.5-AB-testing-model-serving-fraud\", \"title\": \"Roll-out our new model with A/B testing.\", \"description\": \"Deploy the new model comparing its performance with the previous one.\", \"pre_run\": True, \"publish_on_website\": True, \"add_cluster_setup_cell\": True, \"parameters\": {}, \"depends_on_previous\": True},\n {\"path\": \"05-Workflow-orchestration/05-Workflow-orchestration-fsi-fraud\", \"title\": \"Orchestrate churn prevention with Workflow\", \"description\": \"Orchestrate all tasks in a job and schedule your data/model refresh\", \"pre_run\": False, \"publish_on_website\": True, \"add_cluster_setup_cell\": False, \"parameters\": {}, \"depends_on_previous\": True},\n {\"path\": \"_resources/00-global-setup\", \"title\": \"Global init\", \"description\": \"Global init\", \"pre_run\": False, \"publish_on_website\": False, \"add_cluster_setup_cell\": False, \"parameters\": {}, \"depends_on_previous\": True}]\n\n notebooks = [DemoNotebook(n['path'], n['title'], n['description'], n['pre_run'], n['publish_on_website'], n['add_cluster_setup_cell'], n['parameters'], n['depends_on_previous']) for n in notebooks]\n\n job_id=720149424775058\n run_id=120544375\n cluster_id=\"0320-175126-exzit1ks\"\n cluster_name=\"dbdemos-lakehouse-fsi-fraud-quentin_ambard\"\n pipelines_ids= [{\"name\": \"dbdemos-fsi-fraud-detection\", \"uid\": \"30177e65-8729-4363-9e9c-7bff51caddc3\", \"id\": \"dlt-fsi-fraud\", \"run_after_creation\": True}]\n dashboards=[{\"id\": \"9fc6a3bb-ff36-4e06-b5f9-912d7e77dc05\", \"name\": \"FSI Fraud Detection - dbdemos\", \"uid\": \"e1fa43f0-865d-4f5b-b884-64806fe2526a\"}]\n workflows= []\n\n from pathlib import Path\n html = installer.get_install_result_html(demo_name, description, title, install_path, notebooks, job_id, run_id, cluster_id, cluster_name, pipelines_ids, dashboards, workflows)\n with open(\"./test2.html\", \"w\") as text_file:\n text_file.write(html)\n print(html)\n\ndef test_list():\n i = Installer(\"test\",\"test\",\"test\",\"test\",\"test\",\"test\")\n dbdemos.list_demos(None, i)\n\ndef test_list_html():\n deprecated_demos = [\"uc-04-audit-log\", \"llm-dolly-chatbot\"]\n installer = Installer(\"http://localhost\", pat_token=\"test\")\n from collections import defaultdict\n demos = defaultdict(lambda: [])\n \n #Define category order\n demos[\"lakehouse\"] = []\n for demo in installer.get_demos_available():\n conf = installer.get_demo_conf(demo)\n if len(demos[conf.category]) == 0:\n demos[conf.category].append(conf)\n content = dbdemos.get_html_list_demos(demos)\n with open(\"./test_list_html.html\", \"w\") as text_file:\n text_file.write(content)\n print(content)\n\n\n#test_list_html()\n#test_list()\n#test_html()" }, { "path": "test/test_installer_genie.py", "content": "import dbdemos\nfrom dbdemos.conf import DemoNotebook, DemoConf, DataFolder\n\nfrom dbdemos.installer import Installer\nfrom dbdemos.installer_genie import InstallerGenie\nfrom databricks.sdk import WorkspaceClient\n\n\n\ndef test_room_install():\n room = [{'display_name': 'test quentin API',\n 'description': 'test Desc',\n 'table_identifiers': ['main.quentin_test.*'],\n 'sql_instructions': [{\"title\": \"test\", \"content\": \"select * from test\"}],\n 'instructions': 'This is a description',\n 'curated_questions': [\"What's the Churn?\", \"Hoow many turbines do I have?\"]}]\n demo_conf = {\"genie_rooms\": room, \"name\": \"test\", \"category\": \"test\", \"title\": \"title\", \"description\": \"description\", \"bundle\": True}\n conf = DemoConf(path=\"/Users/quentin.ambard@databricks.com/test_install_quentin\", json_conf=demo_conf)\n with open(\"../local_conf_E2FE.json\", \"r\") as r:\n import json\n c = json.loads(r.read())\n installer = Installer(c['username'], c['pat_token'], c['url'], cloud = \"AWS\")\n genie_installer = InstallerGenie(installer)\n genie_installer.install_genies(conf, \"/Users/quentin.ambard@databricks.com/test_install_quentin\", warehouse_id=\"475b94ddc7cd5211\", debug=True)\n\n\ndef test_load_genie_data():\n data_folders = [\n {\"source_folder\":\"fsi/fraud-transaction/customers\", \"source_format\": \"parquet\", \"target_table_name\":\"iot_parts\", \"target_format\":\"delta\"},\n {\"source_folder\":\"fsi/fraud-transaction/fraud_report\", \"source_format\": \"parquet\", \"target_table_name\":\"iot_turbines\", \"target_format\":\"delta\"}\n ]\n demo_conf = {\"data_folders\": data_folders, \"name\": \"test\", \"category\": \"test\", \"title\": \"title\", \"description\": \"description\", \"bundle\": True}\n conf = DemoConf(path=\"/Users/quentin.ambard@databricks.com/test_install_quentin\", json_conf=demo_conf)\n with open(\"../local_conf_E2FE.json\", \"r\") as r:\n import json\n c = json.loads(r.read())\n installer = Installer(c['username'], c['pat_token'], c['url'], cloud = \"AWS\")\n genie_installer = InstallerGenie(installer)\n conf.catalog = 'dbdemos'\n conf.schema = 'test_quentin'\n genie_installer.load_genie_data(conf, warehouse_id=\"475b94ddc7cd5211\", debug=True)\n\ndef test_schema_creation():\n demo_conf = {\"name\": \"test\", \"category\": \"test\", \"title\": \"title\", \"description\": \"description\", \"bundle\": True}\n conf = DemoConf(path=\"/Users/quentin.ambard@databricks.com/test_install_quentin\", json_conf=demo_conf)\n with open(\"../local_conf_E2FE.json\", \"r\") as r:\n import json\n c = json.loads(r.read())\n installer = Installer(c['username'], c['pat_token'], c['url'], cloud = \"AWS\")\n genie_installer = InstallerGenie(installer)\n conf.catalog = 'dbdemos'\n conf.schema = 'test_quentin2'\n ws = WorkspaceClient(token=installer.db.conf.pat_token, host=installer.db.conf.workspace_url)\n genie_installer.create_schema(ws,conf, debug=True)\n\n\n\n\ndef load_data_to_volume():\n demo_conf = {\"name\": \"test\", \"category\": \"test\", \"title\": \"title\", \"description\": \"description\", \"bundle\": True}\n conf = DemoConf(path=\"/Users/quentin.ambard@databricks.com/test_install_quentin\", json_conf=demo_conf)\n with open(\"../local_conf_E2FE.json\", \"r\") as r:\n import json\n c = json.loads(r.read())\n installer = Installer(c['username'], c['pat_token'], c['url'], cloud = \"AWS\")\n genie_installer = InstallerGenie(installer)\n conf.catalog = 'dbdemos'\n conf.schema = 'test_quentin2'\n ws = WorkspaceClient(token=installer.db.conf.pat_token, host=installer.db.conf.workspace_url)\n data_folder = DataFolder(source_folder=\"fsi/fraud-transaction/customers\", source_format=\"parquet\", target_table_name=\"iot_parts\", target_format=\"delta\")\n genie_installer.load_data_to_volume(ws, data_folder, warehouse_id=\"475b94ddc7cd5211\", conf=conf, debug=True)\n\n\ndef test_load_data():\n demo_conf = {\"name\": \"test\", \"category\": \"test\", \"title\": \"title\", \"description\": \"description\", \"bundle\": True}\n conf = DemoConf(path=\"/Users/quentin.ambard@databricks.com/test_install_quentin\", json_conf=demo_conf)\n with open(\"../local_conf_E2FE.json\", \"r\") as r:\n import json\n c = json.loads(r.read())\n installer = Installer(c['username'], c['pat_token'], c['url'], cloud = \"AWS\")\n genie_installer = InstallerGenie(installer)\n conf.catalog = 'dbdemos'\n conf.schema = 'test_quentin2'\n ws = WorkspaceClient(token=installer.db.conf.pat_token, host=installer.db.conf.workspace_url)\n data_folder = DataFolder(source_folder=\"fsi/fraud-transaction/customers\", source_format=\"parquet\", target_table_name=\"dbdemos_aibi_customers\", target_format=\"delta\")\n genie_installer.load_data(ws, data_folder, warehouse_id=\"475b94ddc7cd5211\", conf=conf, debug=True)\n\n\n\n \n#test_load_data()\ntest_room_install()\n#test_schema_creation()\n#test_html()" }, { "path": "test/test_job_bundler.py", "content": "import unittest\nfrom dbdemos.job_bundler import JobBundler\nfrom dbdemos.conf import Conf\nimport json\n\nclass TestJobBundler(unittest.TestCase):\n def setUp(self):\n with open('local_conf.json', 'r') as f:\n local_conf = json.load(f)\n \n self.conf = Conf(username=\"test_user@test.com\", \n workspace_url=\"https://test.cloud.databricks.com\", \n org_id=\"1234567890\", \n pat_token=\"test_token\",\n repo_url=\"https://github.com/databricks-demos/dbdemos-notebooks\",\n github_token=local_conf.get('github_token'),\n branch=\"main\")\n self.job_bundler = JobBundler(self.conf)\n\n def test_get_changed_files_since_commit(self):\n # Test with a known repository and commit\n owner = \"databricks-demos\"\n repo = \"dbdemos-notebooks\"\n base_commit = \"0652f84e30f3ea2e6802dbe5f36538a30f0d8aa1\" # or use a specific commit SHA\n last_commit = \"b65796d63f628eacc32f7160033181c3477997dd\"\n files = self.job_bundler.get_changed_files_since_commit(owner, repo, base_commit, last_commit)\n\n # Check that we got a list of files back\n self.assertIsInstance(files, list, \"Should return a list of changed files\")\n self.assertEqual(files, ['demo-FSI/lakehouse-fsi-smart-claims/02-Data-Science-ML/02.1-Model-Training.py'])\n \n # Check that file paths are strings\n if files:\n self.assertIsInstance(files[0], str, \"File paths should be strings\")\n\n # Test with last_commit = None to get changes since base commit up to HEAD\n files_to_head = self.job_bundler.get_changed_files_since_commit(owner, repo, base_commit)\n \n # Check that we got a non-empty list back\n self.assertIsInstance(files_to_head, list, \"Should return a list of changed files\")\n self.assertGreater(len(files_to_head), 0, \"Should have at least one changed file\")\n \n # Check that file paths are strings\n self.assertIsInstance(files_to_head[0], str, \"File paths should be strings\")\n\n def test_check_if_demo_file_changed_since_commit(self):\n from dbdemos.conf import DemoConf\n \n # Create a demo config for testing with required name field\n demo_conf = DemoConf(\"demo-FSI/lakehouse-fsi-smart-claims\", {\n \"name\": \"lakehouse-fsi-smart-claims\", \n \"title\": \"FSI Smart Claims\",\n \"category\": \"test\",\n \"description\": \"description\",\n \"bundle\": True\n })\n \n # Test with known commits where we know a file changed in that demo\n base_commit = \"0652f84e30f3ea2e6802dbe5f36538a30f0d8aa1\"\n last_commit = \"b65796d63f628eacc32f7160033181c3477997dd\"\n \n # This should return True as we know a file changed in this demo between these commits\n has_changes = self.job_bundler.check_if_demo_file_changed_since_commit(demo_conf, base_commit, last_commit)\n self.assertTrue(has_changes, \"Should detect changes in demo files\")\n \n # Test with a demo path that we know didn't change\n demo_conf_no_changes = DemoConf(\"demo-retail\", {\n \"name\": \"demo-retail\",\n \"title\": \"Demo Retail\", \n \"category\": \"test\",\n \"description\": \"description\",\n \"bundle\": True\n })\n has_changes = self.job_bundler.check_if_demo_file_changed_since_commit(demo_conf_no_changes, base_commit, last_commit)\n self.assertFalse(has_changes, \"Should not detect changes in unmodified demo\")\n\nif __name__ == '__main__':\n unittest.main() " }, { "path": "test/test_list_demos.html", "content": "\n\n
\n\n
\n \n \n \n \n \n
\n\n\n\n
\n
\n \n
\n

Lakehouse - Credit Decisioning

\n Build your banking data platform and identify credit worthy customers\n
\n
\n dbdemos.install('lakehouse-fsi-credit')\n
\n
\n
\n \n
\n

Retail Banking - Fraud Detection

\n Build your Banking platform and detect Fraud in real-time. End 2 End demo, with Model Serving & realtime fraud inference A/B testing.\n
\n
\n dbdemos.install('lakehouse-fsi-fraud')\n
\n
\n
\n \n
\n

Lakehouse for HLS: Patient readmission

\n Build your data platform and personalized health care to reduce readmission risk\n
\n
\n dbdemos.install('lakehouse-hls-readmission')\n
\n
\n
\n \n
\n

Lakehouse for IoT & Predictive Maintenance

\n Detect faulty wind turbine: Ingestion (DLT), BI, Predictive Maintenance (ML), Governance (UC), Orchestration\n
\n
\n dbdemos.install('lakehouse-iot-platform')\n
\n
\n
\n \n
\n

Lakehouse for C360: Reducing Customer Churn

\n Centralize customer data and reduce churn: Ingestion (DLT), BI, Predictive Maintenance (ML), Governance (UC), Orchestration.\n
\n
\n dbdemos.install('lakehouse-retail-c360')\n
\n
\n
\n\n
\n
\n \n
\n

Data Warehousing with Identity, Primary Key & Foreign Key

\n Define your schema with auto incremental column and Primary + Foreign Key. Ideal for Data Warehouse & BI support!\n
\n
\n dbdemos.install('identity-pk-fk')\n
\n
\n
\n \n
\n

AI Functions: query LLM with DBSQL

\n Call Azure OpenAI's model from your Lakehouse data using AI_GENERATE_TEXT()\n
\n
\n dbdemos.install('sql-ai-functions')\n
\n
\n
\n\n
\n
\n \n
\n

Databricks Autoloader (cloudfile)

\n Incremental ingestion on your cloud storage folder.\n
\n
\n dbdemos.install('auto-loader')\n
\n
\n
Auto Loader
\n
\n
\n
\n \n
\n

CDC Pipeline with Delta

\n Process CDC data to build an entire pipeline and materialize your operational tables in your lakehouse.\n
\n
\n dbdemos.install('cdc-pipeline')\n
\n
\n
Delta Lake
\n
\n
\n
\n \n
\n

Orchestrate and run your dbt jobs

\n Launch your dbt pipelines in production using a SQL Warehouse. Leverage Databricks Workflow (orchestration) and add a dbt task in your transformation pipeline.\n
\n
\n dbdemos.install('dbt-on-databricks')\n
\n
\n
Delta Lake
\n
\n
\n
\n \n
\n

Delta Lake

\n Store your table with Delta Lake & discover how Delta Lake can simplify your Data Pipelines.\n
\n
\n dbdemos.install('delta-lake')\n
\n
\n
Delta Lake
\n
\n
\n
\n \n
\n

CDC pipeline with Delta Live Table.

\n Ingest Change Data Capture flow with APPLY INTO and simplify SCDT2 implementation.\n
\n
\n dbdemos.install('dlt-cdc')\n
\n
\n
Delta Live Table
\n
\n
\n
\n \n
\n

Full Delta Live Tables Pipeline - Loan

\n Ingest loan data and implement a DLT pipeline with quarantine.\n
\n
\n dbdemos.install('dlt-loans')\n
\n
\n
Delta Live Tables
\n
\n
\n
\n \n
\n

Unit Testing Delta Live Table (DLT) for production-grade pipelines

\n Deploy robust Delta Live Table pipelines with unit tests leveraging expectation.\n
\n
\n dbdemos.install('dlt-unit-test')\n
\n
\n
Delta Live Table
\n
\n
\n
\n \n
\n

Spark Streaming - Advanced

\n Deep dive on Spark Streaming with Delta to build webapp user sessions from clicks, with custom aggregation state management.\n
\n
\n dbdemos.install('streaming-sessionization')\n
\n
\n
Delta Lake
\n
\n
\n
\n
\n \n
\n

Delta Sharing - Airlines

\n Share your data to external organization using Delta Sharing.\n
\n
\n dbdemos.install('delta-sharing-airlines')\n
\n
\n
Delta Sharing
\n
\n
\n
\n \n
\n

Table ACL & Dynamic Views with UC

\n Discover to GRANT permission on your table with UC and implement more advanced control such as data masking at row-level, based on each user.\n
\n
\n dbdemos.install('uc-01-acl')\n
\n
\n
Unity Catalog
\n
\n
\n
\n \n
\n

Access data on External Location

\n Discover how you can secure files/table in external location (cloud storage like S3/ADLS/GCS) with simple GRANT command.\n
\n
\n dbdemos.install('uc-02-external-location')\n
\n
\n
Unity Catalog
\n
\n
\n
\n \n
\n

Data Lineage with Unity Catalog

\n Discover data lineage with Unity Catalog: table to table and column to column\n
\n
\n dbdemos.install('uc-03-data-lineage')\n
\n
\n
Unity Catalog
\n
\n
\n
\n \n
\n

Audit-log with Databricks

\n [DEPRECATED - prefer uc-04-system-tables] Track usage with Audit-log.\n
\n
\n dbdemos.install('uc-04-audit-log')\n
\n
\n
Unity Catalog
\n
\n
\n
\n \n
\n

System Tables: Billing Forecast, Usage and Audit

\n Track and analysis usage, billing & access with UC System tables.\n
\n
\n dbdemos.install('uc-04-system-tables')\n
\n
\n
Unity Catalog
\n
\n
\n
\n \n
\n

Upgrade table to Unity Catalog

\n Discover how to upgrade your hive_metastore tables to Unity Catalog to benefit from UC capabilities: Security/ACL/Row-level/Lineage/Audit...\n
\n
\n dbdemos.install('uc-05-upgrade')\n
\n
\n
Unity Catalog
\n
\n
\n\n
\n\n
\n \n
\n

Image classification - Default detection

\n Deep Learning using Databricks Lakehouse: detect defaults in PCBs with Hugging Face transformers and PyTorch Lightning.\n
\n
\n dbdemos.install('computer-vision-pcb')\n
\n
\n
Data Science
\n
\n
\n
\n \n
\n

Feature Store and Online Inference

\n Leverage Databricks Feature Store with streaming and online store.\n
\n
\n dbdemos.install('feature-store')\n
\n
\n
Data Science
\n
\n
\n
\n \n
\n

Build your Chat Bot with Dolly

\n Democratizing the magic of ChatGPT with open models and Databricks Lakehouse (starts GPU)\n
\n
\n dbdemos.install('llm-dolly-chatbot')\n
\n
\n
Data Science
\n
\n
\n
\n \n
\n

MLOps - End 2 end pipeline

\n Automate your model deployment with MLFlow webhook & repo, end 2 end!\n
\n
\n dbdemos.install('mlops-end2end')\n
\n
\n
Data Science
\n
\n
\n
\n \n
\n

Pandas API with spark backend (Koalas)

\n Let you Data Science team scale to TB of data while working with Pandas API, without having to learn & move to another framework.\n
\n
\n dbdemos.install('pandas-on-spark')\n
\n
\n
Data Science
\n
\n
\n\n\n" }, { "path": "test/test_list_demos2.html", "content": " \n
\n
\n
\n \n
\n

Lakehouse - Credit Decisioning

\n Build your banking data platform and identify credit worthy customers\n
\n
\n dbdemos.install('lakehouse-fsi-credit')\n
\n
\n
\n \n
\n

Retail Banking - Fraud Detection

\n Build your Banking platform and detect Fraud in real-time. End 2 End demo, with Model Serving & realtime fraud inference A/B testing.\n
\n
\n dbdemos.install('lakehouse-fsi-fraud')\n
\n
\n
\n \n
\n

Lakehouse for HLS: Patient readmission

\n Build your data platform and personalized health care to reduce readmission risk\n
\n
\n dbdemos.install('lakehouse-hls-readmission')\n
\n
\n
\n \n
\n

Lakehouse for IoT & Predictive Maintenance

\n Detect faulty wind turbine: Ingestion (DLT), BI, Predictive Maintenance (ML), Governance (UC), Orchestration\n
\n
\n dbdemos.install('lakehouse-iot-platform')\n
\n
\n
\n \n
\n

Lakehouse for C360: Reducing Customer Churn

\n Centralize customer data and reduce churn: Ingestion (DLT), BI, Predictive Maintenance (ML), Governance (UC), Orchestration.\n
\n
\n dbdemos.install('lakehouse-retail-c360')\n
\n
\n
\n \n
\n

Databricks Autoloader (cloudfile)

\n Incremental ingestion on your cloud storage folder.\n
\n
\n dbdemos.install('auto-loader')\n
\n
\n
\n \n
\n

CDC Pipeline with Delta

\n Process CDC data to build an entire pipeline and materialize your operational tables in your lakehouse.\n
\n
\n dbdemos.install('cdc-pipeline')\n
\n
\n
\n \n
\n

Orchestrate and run your dbt jobs

\n Launch your dbt pipelines in production using a SQL Warehouse. Leverage Databricks Workflow (orchestration) and add a dbt task in your transformation pipeline.\n
\n
\n dbdemos.install('dbt-on-databricks')\n
\n
\n
\n \n
\n

Delta Lake

\n Store your table with Delta Lake & discover how Delta Lake can simplify your Data Pipelines.\n
\n
\n dbdemos.install('delta-lake')\n
\n
\n
\n \n
\n

CDC pipeline with Delta Live Table.

\n Ingest Change Data Capture flow with APPLY INTO and simplify SCDT2 implementation.\n
\n
\n dbdemos.install('dlt-cdc')\n
\n
\n
\n \n
\n

Full Delta Live Tables Pipeline - Loan

\n Ingest loan data and implement a DLT pipeline with quarantine.\n
\n
\n dbdemos.install('dlt-loans')\n
\n
\n
\n \n
\n

Unit Testing Delta Live Table (DLT) for production-grade pipelines

\n Deploy robust Delta Live Table pipelines with unit tests leveraging expectation.\n
\n
\n dbdemos.install('dlt-unit-test')\n
\n
\n
\n \n
\n

Spark Streaming - Advanced

\n Deep dive on Spark Streaming with Delta to build webapp user sessions from clicks, with custom aggregation state management.\n
\n
\n dbdemos.install('streaming-sessionization')\n
\n
\n
\n \n
\n

Delta Sharing - Airlines

\n Share your data to external organization using Delta Sharing.\n
\n
\n dbdemos.install('delta-sharing-airlines')\n
\n
\n
\n \n
\n

Table ACL & Dynamic Views with UC

\n Discover to GRANT permission on your table with UC and implement more advanced control such as data masking at row-level, based on each user.\n
\n
\n dbdemos.install('uc-01-acl')\n
\n
\n
\n \n
\n

Access data on External Location

\n Discover how you can secure files/table in external location (cloud storage like S3/ADLS/GCS) with simple GRANT command.\n
\n
\n dbdemos.install('uc-02-external-location')\n
\n
\n
\n \n
\n

Data Lineage with Unity Catalog

\n Discover data lineage with Unity Catalog: table to table and column to column\n
\n
\n dbdemos.install('uc-03-data-lineage')\n
\n
\n
\n \n
\n

Audit-log with Databricks

\n [DEPRECATED - prefer uc-04-system-tables] Track usage with Audit-log.\n
\n
\n dbdemos.install('uc-04-audit-log')\n
\n
\n
\n \n
\n

System Tables: Billing Forecast, Usage and Audit

\n Track and analysis usage, billing & access with UC System tables.\n
\n
\n dbdemos.install('uc-04-system-tables')\n
\n
\n
\n \n
\n

Upgrade table to Unity Catalog

\n Discover how to upgrade your hive_metastore tables to Unity Catalog to benefit from UC capabilities: Security/ACL/Row-level/Lineage/Audit...\n
\n
\n dbdemos.install('uc-05-upgrade')\n
\n
\n
\n \n
\n

Image classification - Default detection

\n Deep Learning using Databricks Lakehouse: detect defaults in PCBs with Hugging Face transformers and PyTorch Lightning.\n
\n
\n dbdemos.install('computer-vision-pcb')\n
\n
\n
\n \n
\n

Feature Store and Online Inference

\n Leverage Databricks Feature Store with streaming and online store.\n
\n
\n dbdemos.install('feature-store')\n
\n
\n
\n \n
\n

Build your Chat Bot with Dolly

\n Democratizing the magic of ChatGPT with open models and Databricks Lakehouse (starts GPU)\n
\n
\n dbdemos.install('llm-dolly-chatbot')\n
\n
\n
\n \n
\n

MLOps - End 2 end pipeline

\n Automate your model deployment with MLFlow webhook & repo, end 2 end!\n
\n
\n dbdemos.install('mlops-end2end')\n
\n
\n
\n \n
\n

Pandas API with spark backend (Koalas)

\n Let you Data Science team scale to TB of data while working with Pandas API, without having to learn & move to another framework.\n
\n
\n dbdemos.install('pandas-on-spark')\n
\n
\n
\n \n
\n

Data Warehousing with Identity, Primary Key & Foreign Key

\n Define your schema with auto incremental column and Primary + Foreign Key. Ideal for Data Warehouse & BI support!\n
\n
\n dbdemos.install('identity-pk-fk')\n
\n
\n
\n \n
\n

AI Functions: query LLM with DBSQL

\n Call Azure OpenAI's model from your Lakehouse data using AI_GENERATE_TEXT()\n
\n
\n dbdemos.install('sql-ai-functions')\n
\n
\n" }, { "path": "test/test_notebook_parser.py", "content": "import re\nimport base64\nimport urllib.parse\nimport json\nfrom dbdemos.notebook_parser import NotebookParser\n\n\n\ndef test_close_cell():\n with open(\"../dbdemos/template/LICENSE.html\", \"r\") as f:\n p = NotebookParser(f.read())\n p.hide_commands_and_results()\n #print(p.get_html())\n #p.hide_command_result(0)\n\n\ndef test_automl():\n with open(\"../dbdemos/bundles/mlops-end2end/install_package/01_feature_engineering.html\", \"r\") as f:\n p = NotebookParser(f.read())\n assert \"Data exploration notebook\" in p.content\n assert \"Please run the notebook cells to get your AutoML links\" not in p.content\n p.remove_automl_result_links()\n assert \"Data exploration notebook\" not in p.content\n assert \"Please run the notebook cells to get your AutoML links\" in p.content\n #print(p.get_html())\n #p.hide_command_result(0)\n\ndef test_change_relative_links_for_minisite():\n with open(\"../dbdemos/bundles/llm-dolly-chatbot/install_package/01-Dolly-Introduction.html\", \"r\") as f:\n p = NotebookParser(f.read())\n assert p.contains(\"\"\"n the next [03-Q&A-prompt-engineering-for-dolly]($./03-Q&A-prompt-engineering-for-dolly) not\"\"\")\n p.change_relative_links_for_minisite()\n assert p.contains(\"\"\"n the next [03-Q&A-prompt-engineering-for-dolly](./03-Q&A-prompt-engineering-for-dolly.html) not\"\"\")\n\ndef test_parser_contains():\n with open(\"../dbdemos/bundles/mlops-end2end/install_package/_resources/00-setup.html\", \"r\") as f:\n p = NotebookParser(f.read())\n assert p.contains(\"00-global-setup\")\n p.replace_in_notebook('00-global-setup', './00-global-setup-test', True)\n assert p.contains(\"./00-global-setup-test\")\n #print(p.get_html())\n #p.hide_command_result(0)\n\ndef test_parser_notebook():\n\n with open(\"../dbdemos/bundles/lakehouse-retail-c360/install_package/01-Data-ingestion/01.1-DLT-churn-SQL.html\", \"r\") as f:\n p = NotebookParser(f.read())\n assert p.contains(\"\"\"\"\"\")\n p.replace_dynamic_links_pipeline([{\"id\": \"dlt-churn\", \"uid\": \"uuuiduuu\"}])\n assert p.contains(\"\"\"\"\"\")\n\n with open(\"../dbdemos/bundles/dlt-cdc/install_package/01-Retail_DLT_CDC_SQL.html\", \"r\") as f:\n p = NotebookParser(f.read())\n assert p.contains(\"\"\"\"\"\")\n p.replace_dynamic_links_pipeline([{\"id\": \"dlt-cdc\", \"uid\": \"uuuiduuu\"}])\n assert p.contains(\"\"\"\"\"\")\n\n with open(\"../dbdemos/bundles/dbt-on-databricks/install_package/00-DBT-on-databricks.html\", \"r\") as f:\n p = NotebookParser(f.read())\n p.replace_dynamic_links_pipeline([{\"id\": \"dlt-test\", \"uid\": \"uuuiduuu\"}])\n #assert \"\"\"Delta Live Table Pipeline for unit-test demo\"\"\")\n p.replace_dynamic_links_workflow([{'uid': 450396635732004, 'run_id': 3426479, 'id': 'dbt'}])\n assert p.contains(\"\"\"\"\"\")\n\n assert p.contains(\"\"\"\"\"\")\n p.replace_dynamic_links_repo([{'uid': '/Repos/quentin.ambard@databricks.com/dbdemos-dbt-databricks-c360', 'id': 'dbt-databricks-c360', 'repo_id': 3891038073826409}])\n assert p.contains(\"\"\"\"\"\")\n\n\n\ntest_automl()\ntest_close_cell()\ntest_automl()\ntest_parser_contains()\ntest_parser_notebook()\ntest_change_relative_links_for_minisite()\n" }, { "path": "test_demo.py", "content": "import json\nfrom dbdemos.conf import Conf\nfrom dbdemos.job_bundler import JobBundler\nfrom dbdemos.packager import Packager\n\n\ndef load_conf(conf_path):\n with open(conf_path, \"r\") as r:\n c = json.loads(r.read())\n\n with open(\"./dbdemos/resources/default_cluster_config.json\", \"r\") as cc:\n default_cluster_template = cc.read()\n with open(\"./dbdemos/resources/default_test_job_conf.json\", \"r\") as cc:\n default_cluster_job_template = cc.read()\n return Conf(c['username'], c['url'], c['org_id'], c['pat_token'],\n default_cluster_template, default_cluster_job_template,\n c['repo_staging_path'], c['repo_name'], c['repo_url'], c['branch'],github_token=c['github_token'])\n\ndef bundle(conf, demo_path_in_repo):\n bundler = JobBundler(conf)\n # the bundler will use a stating repo dir in the workspace to analyze & run content.\n bundler.reset_staging_repo(skip_pull=False)\n\n bundler.add_bundle(demo_path_in_repo)\n # Run the jobs (only if there is a new commit since the last time, or failure, or force execution)\n bundler.start_and_wait_bundle_jobs(force_execution = False, skip_execution=True)\n packager = Packager(conf, bundler)\n packager.package_all()\n\n#Loads conf (your workspace url & token) : local_conf.json.\n# Conf file example. This is what will be used as repo content to build the package. You can use the repo you're working on.\n\"\"\" {\n \"cloud\": \"AWS\",\n \"pat_token\": \"xxx\",\n \"username\": \"xx.xx@databricks.com\",\n \"url\": \"https://e2-demo-field-eng.cloud.databricks.com/\",\n \"org_id\": \"1444828305810485\",\n \"repo_staging_path\": \"/Repos/xx.xxx@databricks.com\",\n \"repo_name\": \"field-demos,\n \"repo_url\": \"\",\n \"branch\": \"master\",\n \"github_token\": \"ghp_xxx\"\n}\n\"\"\"\nconf = load_conf(\"local_conf_azure.json\")\n\n#This will create the bundle and save it in the local ./bundles and ./minisite folder.\n#change the path with your demo path in the https://github.com/databricks/field-demo repo (your fork)\ntry:\n bundle(conf, \"aibi/aibi-marketing-campaign\")\nexcept Exception as e:\n print(f\"Failure building the job: {e}\")\n raise e\n\n# Now that your demo is packaged, we can install it & test.\n# We recommend testing in a new workspace so that you have a fresh install\n# Load the conf for the workspace where you want to install the demo:\n\nimport dbdemos\ntry:\n #Install your demo in a given folder:\n dbdemos.install(\"aibi-marketing-campaign\", \"/Users/andrea.picasso@databricks.com/test_install_aibidemo\", True, conf.username,\n conf.pat_token, conf.workspace_url, cloud=\"AWS\", start_cluster = False)\n #Check if the init job is successful:\n #dbdemos.check_status(\"sql-ai-functions\", conf.username, conf.pat_token, conf.workspace_url, cloud=\"AWS\")\n print(\"looking good! Ready to send your PR with your new demo!\")\nexcept Exception as e:\n print(f\"Failure installing the demo: {e}\")\n raise e\n" }, { "path": "test_list_html.html", "content": "\n\n
\n
\n
\n \n
\n

Lakehouse for Retail Banking: Credit Decisioning

\n Build your banking data platform and identify credit worthy customers\n
\n
\n dbdemos.install('lakehouse-fsi-credit')\n
\n
\n
\n \n
\n

MLOps - End 2 end pipeline

\n Automate your model deployment with MLFlow and UC, end 2 end!\n
\n
\n dbdemos.install('mlops-end2end')\n
\n
\n
\n \n
\n

CDC Pipeline with Delta

\n Process CDC data to build an entire pipeline and materialize your operational tables in your lakehouse.\n
\n
\n dbdemos.install('cdc-pipeline')\n
\n
\n
\n \n
\n

AI/BI: Sales Pipeline Review

\n Optimize your sales pipeline visibility and insights with AI/BI Dashboards, and leverage Genie to ask questions about your data in natural language.\n
\n
\n dbdemos.install('aibi-sales-pipeline-review')\n
\n
\n
\n \n
\n

Table ACL & Row + Column Level security with UC

\n Discover how to GRANT permission on your table with Unity Catalog and implement fined grained control such as data masking at column level or filter rows based on each user.\n
\n
\n dbdemos.install('uc-01-acl')\n
\n
\n
\n \n
\n

Data Warehousing with Identity, Primary Key & Foreign Key

\n Define your schema with auto incremental column and Primary + Foreign Key. Ideal for Data Warehouse & BI support!\n
\n
\n dbdemos.install('identity-pk-fk')\n
\n
" } ]