[ { "path": ".changes/header.tpl.md", "content": "# Changelog\n" }, { "path": ".changes/unreleased/added-20260420-140247.yaml", "content": "kind: added\nbody: Add `$workspace.$name` and `$workspace.$name_encoded` dynamic replacement variables for target workspace display name\ntime: 2026-04-20T14:02:47.0254635+03:00\ncustom:\n Author: shirasassoon\n AuthorLink: https://github.com/shirasassoon\n Issue: \"895\"\n IssueLink: https://github.com/microsoft/fabric-cicd/issues/895\n" }, { "path": ".changes/unreleased/added-20260503-000000.yaml", "content": "kind: added\nbody: Add `configure_fabric_fqdn` to configure Fabric API URLs for private-link-enabled workspaces using per-workspace FQDN endpoints\ntime: 2026-05-03T00:00:00.0000000+00:00\ncustom:\n Author: SumeshKashyap\n AuthorLink: https://github.com/SumeshKashyap\n Issue: \"754\"\n IssueLink: https://github.com/microsoft/fabric-cicd/issues/754\n" }, { "path": ".changes/unreleased/fixed-20260424-103120.yaml", "content": "kind: fixed\nbody: Fix incomplete Sparkcompute settings deployment by using the item definition API instead of the staging/sparkcompute API when deploying Environments\ntime: 2026-04-24T10:31:20.3568472+02:00\ncustom:\n Author: lassevalentini\n AuthorLink: https://github.com/lassevalentini\n Issue: \"776\"\n IssueLink: https://github.com/microsoft/fabric-cicd/issues/776\n" }, { "path": ".changes/unreleased/fixed-20260428-121610.yaml", "content": "kind: fixed\nbody: Fix HTTP 400 errors when using ``items_to_include`` with post-publish operations (e.g. Lakehouse shortcut publishing)\ntime: 2026-04-28T12:16:10.6116726+03:00\ncustom:\n Author: shirasassoon\n AuthorLink: https://github.com/shirasassoon\n Issue: \"948\"\n IssueLink: https://github.com/microsoft/fabric-cicd/issues/948\n" }, { "path": ".changes/unreleased/new-items-20260505-123355.yaml", "content": "kind: new-items\nbody: Add support for DataBuildToolJob item\ntime: 2026-05-05T12:33:55.6581835-05:00\ncustom:\n Author: crazy-treyn\n AuthorLink: https://github.com/crazy-treyn\n Issue: \"864\"\n IssueLink: https://github.com/microsoft/fabric-cicd/issues/864\n" }, { "path": ".changes/unreleased/optimization-20260505-142743.yaml", "content": "kind: optimization\nbody: Remove version check on import to reduce noise and eliminate unnecessary PyPI network call at startup\ntime: 2026-05-05T14:27:43.6077342+03:00\ncustom:\n Author: shirasassoon\n AuthorLink: https://github.com/shirasassoon\n Issue: \"973\"\n IssueLink: https://github.com/microsoft/fabric-cicd/issues/973\n" }, { "path": ".changes/v0.1.0.md", "content": "## [v0.1.0](https://pypi.org/project/fabric-cicd/0.1.0) - January 23, 2025\n\n### ✨ New Functionality\n\n- Initial public preview release\n- Supports Notebook, Pipeline, Semantic Model, Report, and Environment deployments\n- Supports User and System Identity authentication\n- Released to PyPi\n- Onboarded to Github Pages\n" }, { "path": ".changes/v0.1.1.md", "content": "## [v0.1.1](https://pypi.org/project/fabric-cicd/0.1.1) - January 23, 2025\n\n### 🔧 Bug Fix\n\n- Fix Environment stuck in publish ([#51](https://github.com/microsoft/fabric-cicd/issues/51))\n" }, { "path": ".changes/v0.1.10.md", "content": "## [v0.1.10](https://pypi.org/project/fabric-cicd/0.1.10) - March 19, 2025\n\n### ✨ New Functionality\n\n- DataPipeline SPN Support ([#133](https://github.com/microsoft/fabric-cicd/issues/133))\n\n### 🔧 Bug Fix\n\n- Workspace ID replacement in data pipelines ([#164](https://github.com/microsoft/fabric-cicd/issues/164))\n\n### 📝 Documentation Update\n\n- Sample for passing in arguments from Azure DevOps Pipelines\n" }, { "path": ".changes/v0.1.11.md", "content": "## [v0.1.11](https://pypi.org/project/fabric-cicd/0.1.11) - March 25, 2025\n\n### ⚠️ Breaking Change\n\n- Parameterization refactor introducing a new parameter file structure and parameter file validation functionality ([#113](https://github.com/microsoft/fabric-cicd/issues/113))\n\n### ✨ New Functionality\n\n- Support regex for publish exclusion ([#121](https://github.com/microsoft/fabric-cicd/issues/121))\n- Override max retries via constants ([#146](https://github.com/microsoft/fabric-cicd/issues/146))\n\n### 📝 Documentation Update\n\n- Update to [parameterization](https://microsoft.github.io/fabric-cicd/latest/how_to/parameterization/) docs\n" }, { "path": ".changes/v0.1.12.md", "content": "## [v0.1.12](https://pypi.org/project/fabric-cicd/0.1.12) - March 27, 2025\n\n### 🔧 Bug Fix\n\n- Fix constant overwrite failures ([#190](https://github.com/microsoft/fabric-cicd/issues/190))\n- Fix bug where all workspace ids were not being replaced ([#186](https://github.com/microsoft/fabric-cicd/issues/186))\n- Fix type hints for older versions of Python ([#156](https://github.com/microsoft/fabric-cicd/issues/156))\n- Fix accepted item types constant in pre-build\n" }, { "path": ".changes/v0.1.13.md", "content": "## [v0.1.13](https://pypi.org/project/fabric-cicd/0.1.13) - April 07, 2025\n\n### ✨ New Functionality\n\n- Added support for Lakehouse Shortcuts\n- New `enable_environment_variable_replacement` feature flag ([#160](https://github.com/microsoft/fabric-cicd/issues/160))\n\n### 🆕 New Items Support\n\n- Onboard Workspace Folders ([#81](https://github.com/microsoft/fabric-cicd/issues/81))\n- Onboard Variable Library item type ([#206](https://github.com/microsoft/fabric-cicd/issues/206))\n\n### ⚡ Additional Optimizations\n\n- User-agent now available in API headers ([#207](https://github.com/microsoft/fabric-cicd/issues/207))\n- Fixed error log typo in fabric_endpoint\n\n### 🔧 Bug Fix\n\n- Fix break with invalid optional parameters ([#192](https://github.com/microsoft/fabric-cicd/issues/192))\n- Fix bug where all workspace ids were not being replaced by parameterization ([#186](https://github.com/microsoft/fabric-cicd/issues/186))\n" }, { "path": ".changes/v0.1.14.md", "content": "## [v0.1.14](https://pypi.org/project/fabric-cicd/0.1.14) - April 09, 2025\n\n### ✨ New Functionality\n\n- Optimized & beautified terminal output\n- Added changelog to output of old version check\n\n### 🔧 Bug Fix\n\n- Fix workspace folder deployments in root folder ([#221](https://github.com/microsoft/fabric-cicd/issues/221))\n- Fix unpublish of workspace folders without publish ([#222](https://github.com/microsoft/fabric-cicd/issues/222))\n\n### ⚡ Additional Optimizations\n\n- Removed Colorama and Colorlog Dependency\n" }, { "path": ".changes/v0.1.15.md", "content": "## [v0.1.15](https://pypi.org/project/fabric-cicd/0.1.15) - April 21, 2025\n\n### 🔧 Bug Fix\n\n- Fix folders moving with every publish ([#236](https://github.com/microsoft/fabric-cicd/issues/236))\n\n### ⚡ Additional Optimizations\n\n- Introduce parallel deployments to reduce publish times ([#237](https://github.com/microsoft/fabric-cicd/issues/237))\n- Improvements to check version logic\n\n### 📝 Documentation Update\n\n- Updated Examples section in docs\n" }, { "path": ".changes/v0.1.16.md", "content": "## [v0.1.16](https://pypi.org/project/fabric-cicd/0.1.16) - April 25, 2025\n\n### 🔧 Bug Fix\n\n- Fix bug with folder deployment to root ([#255](https://github.com/microsoft/fabric-cicd/issues/255))\n\n### ⚡ Additional Optimizations\n\n- Add Workspace Name in FabricWorkspaceObject ([#200](https://github.com/microsoft/fabric-cicd/issues/200))\n- New function to check SQL endpoint provision status ([#226](https://github.com/microsoft/fabric-cicd/issues/226))\n\n### 📝 Documentation Update\n\n- Updated Authentication docs + menu sort order\n" }, { "path": ".changes/v0.1.17.md", "content": "## [v0.1.17](https://pypi.org/project/fabric-cicd/0.1.17) - May 13, 2025\n\n### ⚠️ Breaking Change\n\n- Deprecate old parameter file structure ([#283](https://github.com/microsoft/fabric-cicd/issues/283))\n\n### 🆕 New Items Support\n\n- Onboard CopyJob item type ([#122](https://github.com/microsoft/fabric-cicd/issues/122))\n- Onboard Eventstream item type ([#170](https://github.com/microsoft/fabric-cicd/issues/170))\n- Onboard Eventhouse/KQL Database item type ([#169](https://github.com/microsoft/fabric-cicd/issues/169))\n- Onboard Data Activator item type ([#291](https://github.com/microsoft/fabric-cicd/issues/291))\n- Onboard KQL Queryset item type ([#292](https://github.com/microsoft/fabric-cicd/issues/292))\n\n### 🔧 Bug Fix\n\n- Fix post publish operations for skipped items ([#277](https://github.com/microsoft/fabric-cicd/issues/277))\n\n### ⚡ Additional Optimizations\n\n- New function `key_value_replace` for key-based replacement operations in JSON and YAML\n\n### 📝 Documentation Update\n\n- Add publish regex example to demonstrate how to use the `publish_all_items` with regex for excluding item names\n" }, { "path": ".changes/v0.1.18.md", "content": "## [v0.1.18](https://pypi.org/project/fabric-cicd/0.1.18) - May 14, 2025\n\n### 🔧 Bug Fix\n\n- Fix bug with check environment publish state ([#295](https://github.com/microsoft/fabric-cicd/issues/295))\n" }, { "path": ".changes/v0.1.19.md", "content": "## [v0.1.19](https://pypi.org/project/fabric-cicd/0.1.19) - May 21, 2025\n\n### 🆕 New Items Support\n\n- Onboard SQL Database item type (shell-only deployment) ([#301](https://github.com/microsoft/fabric-cicd/issues/301))\n- Onboard Warehouse item type (shell-only deployment) ([#204](https://github.com/microsoft/fabric-cicd/issues/204))\n\n### 🔧 Bug Fix\n\n- Fix bug with unpublish workspace folders ([#273](https://github.com/microsoft/fabric-cicd/issues/273))\n" }, { "path": ".changes/v0.1.2.md", "content": "## [v0.1.2](https://pypi.org/project/fabric-cicd/0.1.2) - January 27, 2025\n\n### ✨ New Functionality\n\n- Introduces max retry and backoff for long running / throttled calls ([#27](https://github.com/microsoft/fabric-cicd/issues/27))\n\n### 🔧 Bug Fix\n\n- Fix Environment publish uses arbitrary wait time ([#50](https://github.com/microsoft/fabric-cicd/issues/50))\n- Fix Environment publish doesn't wait for success ([#56](https://github.com/microsoft/fabric-cicd/issues/56))\n- Fix Long running operation steps out early for notebook publish ([#58](https://github.com/microsoft/fabric-cicd/issues/58))\n" }, { "path": ".changes/v0.1.20.md", "content": "## [v0.1.20](https://pypi.org/project/fabric-cicd/0.1.20) - June 12, 2025\n\n### ✨ New Functionality\n\n- Parameterization support for find_value regex and replace_value variables ([#326](https://github.com/microsoft/fabric-cicd/issues/326))\n\n### 🆕 New Items Support\n\n- Onboard KQL Dashboard item type ([#329](https://github.com/microsoft/fabric-cicd/issues/329))\n- Onboard Dataflow Gen2 item type ([#111](https://github.com/microsoft/fabric-cicd/issues/111))\n\n### 🔧 Bug Fix\n\n- Fix bug with deploying environment libraries with special chars ([#336](https://github.com/microsoft/fabric-cicd/issues/336))\n\n### ⚡ Additional Optimizations\n\n- Improved test coverage for subfolder creation/modification ([#211](https://github.com/microsoft/fabric-cicd/issues/211))\n" }, { "path": ".changes/v0.1.21.md", "content": "## [v0.1.21](https://pypi.org/project/fabric-cicd/0.1.21) - June 18, 2025\n\n### 🔧 Bug Fix\n\n- Fix bug with workspace ID replacement in JSON files for pipeline deployments ([#345](https://github.com/microsoft/fabric-cicd/issues/345))\n\n### ⚡ Additional Optimizations\n\n- Increased max retry for Warehouses and Dataflows\n" }, { "path": ".changes/v0.1.22.md", "content": "## [v0.1.22](https://pypi.org/project/fabric-cicd/0.1.22) - June 25, 2025\n\n### 🆕 New Items Support\n\n- Onboard API for GraphQL item type ([#287](https://github.com/microsoft/fabric-cicd/issues/287))\n\n### 🔧 Bug Fix\n\n- Fix Fabric API call error during dataflow publish ([#352](https://github.com/microsoft/fabric-cicd/issues/352))\n\n### ⚡ Additional Optimizations\n\n- Expanded test coverage to handle folder edge cases ([#358](https://github.com/microsoft/fabric-cicd/issues/358))\n" }, { "path": ".changes/v0.1.23.md", "content": "## [v0.1.23](https://pypi.org/project/fabric-cicd/0.1.23) - July 08, 2025\n\n### ✨ New Functionality\n\n- New functionalities for GitHub Copilot Agent and PR-to-Issue linking\n\n### 📝 Documentation Update\n\n- Fix formatting and examples in the How to and Examples pages\n\n### 🔧 Bug Fix\n\n- Fix issue with lakehouse shortcuts publishing ([#379](https://github.com/microsoft/fabric-cicd/issues/379))\n- Add validation for empty logical IDs to prevent deployment corruption ([#86](https://github.com/microsoft/fabric-cicd/issues/86))\n- Fix SQL provision print statement ([#329](https://github.com/microsoft/fabric-cicd/issues/329))\n- Rename the error code for reserved item name per updated Microsoft Fabric API ([#388](https://github.com/microsoft/fabric-cicd/issues/388))\n- Fix lakehouse exclude_regex to exclude shortcut publishing ([#385](https://github.com/microsoft/fabric-cicd/issues/385))\n- Remove max retry limit to handle large deployments ([#299](https://github.com/microsoft/fabric-cicd/issues/299))\n" }, { "path": ".changes/v0.1.24.md", "content": "## [v0.1.24](https://pypi.org/project/fabric-cicd/0.1.24) - August 04, 2025\n\n### ⚠️ Breaking Change\n\n- Require parameterization for Dataflow and Semantic Model references in Data Pipeline activities\n- Require specific parameterization for deploying a Dataflow that depends on another in the same workspace (see Parameterization docs)\n\n### 📝 Documentation Update\n\n- Improve Parameterization documentation ([#415](https://github.com/microsoft/fabric-cicd/issues/415))\n\n### ⚡ Additional Optimizations\n\n- Support for Eventhouse query URI parameterization ([#414](https://github.com/microsoft/fabric-cicd/issues/414))\n- Support for Warehouse SQL endpoint parameterization ([#392](https://github.com/microsoft/fabric-cicd/issues/392))\n\n### 🔧 Bug Fix\n\n- Fix Dataflow/Data Pipeline deployment failures caused by workspace permissions ([#419](https://github.com/microsoft/fabric-cicd/issues/419))\n- Prevent duplicate logical ID issue in Report and Semantic Model deployment ([#405](https://github.com/microsoft/fabric-cicd/issues/405))\n- Fix deployment of items without assigned capacity ([#402](https://github.com/microsoft/fabric-cicd/issues/402))\n" }, { "path": ".changes/v0.1.25.md", "content": "## [v0.1.25](https://pypi.org/project/fabric-cicd/0.1.25) - August 19, 2025\n\n### ⚠️ Breaking Change\n\n- Modify the default for item_types_in_scope and add thorough validation ([#464](https://github.com/microsoft/fabric-cicd/issues/464))\n\n### ✨ New Functionality\n\n- Add new experimental feature flag to enable selective deployment ([#384](https://github.com/microsoft/fabric-cicd/issues/384))\n- Support \"ALL\" environment concept in parameterization ([#320](https://github.com/microsoft/fabric-cicd/issues/320))\n\n### 📝 Documentation Update\n\n- Enhance Overview section in Parameterization docs ([#495](https://github.com/microsoft/fabric-cicd/issues/495))\n\n### ⚡ Additional Optimizations\n\n- Eliminate ACCEPTED_ITEM_TYPES_NON_UPN constant and unify with ACCEPTED_ITEM_TYPES ([#477](https://github.com/microsoft/fabric-cicd/issues/477))\n- Add comprehensive GitHub Copilot instructions for effective codebase development ([#468](https://github.com/microsoft/fabric-cicd/issues/468))\n\n### 🔧 Bug Fix\n\n- Add feature flags and warnings for Warehouse, SQL Database, and Eventhouse unpublish operations ([#483](https://github.com/microsoft/fabric-cicd/issues/483))\n- Fix code formatting inconsistencies in fabric_workspace unit test ([#474](https://github.com/microsoft/fabric-cicd/issues/474))\n- Fix KeyError when deploying Reports with Semantic Model dependencies in Report-only scope case ([#278](https://github.com/microsoft/fabric-cicd/issues/278))\n" }, { "path": ".changes/v0.1.26.md", "content": "## [v0.1.26](https://pypi.org/project/fabric-cicd/0.1.26) - September 05, 2025\n\n### ⚠️ Breaking Change\n\n- Deprecate Base API URL kwarg in Fabric Workspace ([#529](https://github.com/microsoft/fabric-cicd/issues/529))\n\n### ✨ New Functionality\n\n- Support Schedules parameterization ([#508](https://github.com/microsoft/fabric-cicd/issues/508))\n- Support YAML configuration file-based deployment ([#470](https://github.com/microsoft/fabric-cicd/issues/470))\n\n### 📝 Documentation Update\n\n- Add dynamically generated Python version requirements to documentation ([#520](https://github.com/microsoft/fabric-cicd/issues/520))\n\n### ⚡ Additional Optimizations\n\n- Enhance pytest output to limit console verbosity ([#514](https://github.com/microsoft/fabric-cicd/issues/514))\n\n### 🔧 Bug Fix\n\n- Fix Report item schema handling ([#518](https://github.com/microsoft/fabric-cicd/issues/518))\n- Fix deployment order to publish Mirrored Database before Lakehouse ([#482](https://github.com/microsoft/fabric-cicd/issues/482))\n" }, { "path": ".changes/v0.1.27.md", "content": "## [v0.1.27](https://pypi.org/project/fabric-cicd/0.1.27) - September 05, 2025\n\n### 🔧 Bug Fix\n\n- Fix trailing comma in report schema ([#534](https://github.com/microsoft/fabric-cicd/issues/534))\n" }, { "path": ".changes/v0.1.28.md", "content": "## [v0.1.28](https://pypi.org/project/fabric-cicd/0.1.28) - September 15, 2025\n\n### ✨ New Functionality\n\n- Add folder exclusion feature for publish operations ([#427](https://github.com/microsoft/fabric-cicd/issues/427))\n- Expand workspace ID dynamic replacement capabilities in parameterization ([#408](https://github.com/microsoft/fabric-cicd/issues/408))\n\n### 🔧 Bug Fix\n\n- Fix unexpected behavior with file_path parameter filter ([#545](https://github.com/microsoft/fabric-cicd/issues/545))\n- Fix unpublish exclude_regex bug in configuration file-based deployment ([#544](https://github.com/microsoft/fabric-cicd/issues/544))\n" }, { "path": ".changes/v0.1.29.md", "content": "## [v0.1.29](https://pypi.org/project/fabric-cicd/0.1.29) - October 01, 2025\n\n### ✨ New Functionality\n\n- Support dynamic replacement for cross-workspace item IDs ([#558](https://github.com/microsoft/fabric-cicd/issues/558))\n- Add option to return API response for publish operations in publish_all_items ([#497](https://github.com/microsoft/fabric-cicd/issues/497))\n\n### 🆕 New Items Support\n\n- Onboard Apache Airflow Job item type ([#565](https://github.com/microsoft/fabric-cicd/issues/565))\n- Onboard Mounted Data Factory item type ([#406](https://github.com/microsoft/fabric-cicd/issues/406))\n\n### 🔧 Bug Fix\n\n- Fix publish order of Eventhouses and Semantic Models ([#566](https://github.com/microsoft/fabric-cicd/issues/566))\n" }, { "path": ".changes/v0.1.3.md", "content": "## [v0.1.3](https://pypi.org/project/fabric-cicd/0.1.3) - January 29, 2025\n\n### ✨ New Functionality\n\n- Add PyPI check version to encourage version bumps ([#75](https://github.com/microsoft/fabric-cicd/issues/75))\n\n### 🔧 Bug Fix\n\n- Fix Semantic model initial publish results in None Url error ([#61](https://github.com/microsoft/fabric-cicd/issues/61))\n- Fix Integer parsed as float failing in handle_retry for <3.12 python ([#63](https://github.com/microsoft/fabric-cicd/issues/63))\n- Fix Default item types fail to unpublish ([#76](https://github.com/microsoft/fabric-cicd/issues/76))\n- Fix Items in subfolders are skipped ([#77](https://github.com/microsoft/fabric-cicd/issues/77))\n\n### 📝 Documentation Update\n\n- Update documentation & examples\n" }, { "path": ".changes/v0.1.30.md", "content": "## [v0.1.30](https://pypi.org/project/fabric-cicd/0.1.30) - October 20, 2025\n\n### ✨ New Functionality\n\n- Add support for binding semantic models to on-premise gateways in Fabric workspaces ([#569](https://github.com/microsoft/fabric-cicd/issues/569))\n\n### 🆕 New Items Support\n\n- Add support for publishing and managing Data Agent items ([#556](https://github.com/microsoft/fabric-cicd/issues/556))\n- Add OrgApp item type support ([#586](https://github.com/microsoft/fabric-cicd/issues/586))\n\n### ⚡ Additional Optimizations\n\n- Enhance cross-workspace variable support to allow referencing other attributes ([#583](https://github.com/microsoft/fabric-cicd/issues/583))\n\n### 🔧 Bug Fix\n\n- Fix workspace name extraction bug for non-ID attrs using ITEM_ATTR_LOOKUP ([#583](https://github.com/microsoft/fabric-cicd/issues/583))\n- Fix capacity requirement check ([#593](https://github.com/microsoft/fabric-cicd/issues/593))\n" }, { "path": ".changes/v0.1.31.md", "content": "## [v0.1.31](https://pypi.org/project/fabric-cicd/0.1.31) - December 01, 2025\n\n### ⚠️ Breaking Change\n\n- Migrate to the latest Fabric Environment item APIs to simplify deployment and improve compatibility ([#173](https://github.com/microsoft/fabric-cicd/issues/173))\n\n### ✨ New Functionality\n\n- Enable dynamic replacement of Lakehouse SQL Endpoint IDs ([#616](https://github.com/microsoft/fabric-cicd/issues/616))\n- Enable linking of Semantic Models to both cloud and gateway connections ([#602](https://github.com/microsoft/fabric-cicd/issues/602))\n- Allow use of the dynamic replacement variables within the key_value_replace parameter ([#567](https://github.com/microsoft/fabric-cicd/issues/567))\n- Add support for parameter file templates ([#499](https://github.com/microsoft/fabric-cicd/issues/499))\n\n### 🆕 New Items Support\n\n- Add support for the ML Experiment item type ([#600](https://github.com/microsoft/fabric-cicd/issues/600))\n- Add support for the User Data Function item type ([#588](https://github.com/microsoft/fabric-cicd/issues/588))\n\n### 📝 Documentation Update\n\n- Update the advanced Dataflow parameterization example with the correct file_path value ([#633](https://github.com/microsoft/fabric-cicd/issues/633))\n\n### 🔧 Bug Fix\n\n- Fix publishing issues for KQL Database items in folders ([#657](https://github.com/microsoft/fabric-cicd/issues/657))\n- Separate logic for 'items to include' feature between publish and unpublish operations ([#650](https://github.com/microsoft/fabric-cicd/issues/650))\n- Fix parameterization logic to properly handle find_value regex patterns and replacements ([#639](https://github.com/microsoft/fabric-cicd/issues/639))\n- Correct the publish order of Data Agent and Semantic Model items ([#628](https://github.com/microsoft/fabric-cicd/issues/628))\n- Fix Lakehouse item publishing errors when shortcuts refer to the default Lakehouse ID ([#610](https://github.com/microsoft/fabric-cicd/issues/610))\n" }, { "path": ".changes/v0.1.32.md", "content": "## [v0.1.32](https://pypi.org/project/fabric-cicd/0.1.32) - December 03, 2025\n\n### 🔧 Bug Fix\n\n- Fix publish bug for Environment items that contain only spark settings ([#664](https://github.com/microsoft/fabric-cicd/issues/664))\n" }, { "path": ".changes/v0.1.33.md", "content": "## [v0.1.33](https://pypi.org/project/fabric-cicd/0.1.33) - December 16, 2025\n\n### ✨ New Functionality\n\n- Add key_value_replace parameter support for YAML files ([#649](https://github.com/microsoft/fabric-cicd/issues/649))\n- Support selective shortcut publishing with regex exclusion ([#624](https://github.com/microsoft/fabric-cicd/issues/624))\n\n### ⚡ Additional Optimizations\n\n- Add Linux development environment bootstrapping script ([#680](https://github.com/microsoft/fabric-cicd/issues/680))\n- Update item types in scope to be an optional parameter in validate parameter file function ([#669](https://github.com/microsoft/fabric-cicd/issues/669))\n\n### 🔧 Bug Fix\n\n- Fix publish order for Notebook and Eventhouse dependent items ([#685](https://github.com/microsoft/fabric-cicd/issues/685))\n- Enable parameterizing multiple connections in the same Semantic Model item ([#674](https://github.com/microsoft/fabric-cicd/issues/674))\n- Fix missing description metadata in item payload for shell-only item deployments ([#672](https://github.com/microsoft/fabric-cicd/issues/672))\n- Resolve API long running operation handling when publishing Environment items ([#668](https://github.com/microsoft/fabric-cicd/issues/668))\n" }, { "path": ".changes/v0.1.34.md", "content": "## [v0.1.34](https://pypi.org/project/fabric-cicd/0.1.34) - January 20, 2026\n\n### ✨ New Functionality\n\n- Enable dynamic replacement of SQL endpoint values from SQL Database items ([#720](https://github.com/microsoft/fabric-cicd/issues/720))\n- Support Fabric Notebook Authentication ([#707](https://github.com/microsoft/fabric-cicd/issues/707))\n\n### 🆕 New Items Support\n\n- Onboard Spark Job Definition item type ([#115](https://github.com/microsoft/fabric-cicd/issues/115))\n\n### 📝 Documentation Update\n\n- Add `CONTRIBUTING.md` file to repository ([#723](https://github.com/microsoft/fabric-cicd/issues/723))\n- Add comprehensive troubleshooting guide to documentation ([#705](https://github.com/microsoft/fabric-cicd/issues/705))\n- Add parameterization documentation for Report items using ByConnection binding to Semantic Models ([#637](https://github.com/microsoft/fabric-cicd/issues/637))\n\n### ⚡ Additional Optimizations\n\n- Add debug file for local Fabric REST API testing ([#714](https://github.com/microsoft/fabric-cicd/issues/714))\n" }, { "path": ".changes/v0.1.4.md", "content": "## [v0.1.4](https://pypi.org/project/fabric-cicd/0.1.4) - February 12, 2025\n\n### ✨ New Functionality\n\n- Support Feature Flagging ([#96](https://github.com/microsoft/fabric-cicd/issues/96))\n\n### 🔧 Bug Fix\n\n- Fix Image support in report deployment ([#88](https://github.com/microsoft/fabric-cicd/issues/88))\n- Fix Broken README link ([#92](https://github.com/microsoft/fabric-cicd/issues/92))\n\n### ⚡ Additional Optimizations\n\n- Workspace ID replacement improved\n- Increased error handling in activate script\n- Onboard pytest and coverage\n- Improvements to nested dictionaries ([#37](https://github.com/microsoft/fabric-cicd/issues/37))\n- Support Python Installed From Windows Store ([#87](https://github.com/microsoft/fabric-cicd/issues/87))\n" }, { "path": ".changes/v0.1.5.md", "content": "## [v0.1.5](https://pypi.org/project/fabric-cicd/0.1.5) - February 18, 2025\n\n### 🔧 Bug Fix\n\n- Fix Environment Failure without Public Library ([#103](https://github.com/microsoft/fabric-cicd/issues/103))\n\n### ⚡ Additional Optimizations\n\n- Introduces pytest check for PRs ([#100](https://github.com/microsoft/fabric-cicd/issues/100))\n" }, { "path": ".changes/v0.1.6.md", "content": "## [v0.1.6](https://pypi.org/project/fabric-cicd/0.1.6) - February 24, 2025\n\n### 🆕 New Items Support\n\n- Onboard Lakehouse item type ([#116](https://github.com/microsoft/fabric-cicd/issues/116))\n\n### 📝 Documentation Update\n\n- Update example docs ([#25](https://github.com/microsoft/fabric-cicd/issues/25))\n- Update find_replace docs ([#110](https://github.com/microsoft/fabric-cicd/issues/110))\n\n### ⚡ Additional Optimizations\n\n- Standardized docstrings to Google format\n- Onboard file objects ([#46](https://github.com/microsoft/fabric-cicd/issues/46))\n- Leverage UpdateDefinition Flag ([#28](https://github.com/microsoft/fabric-cicd/issues/28))\n- Convert repo and workspace dictionaries ([#45](https://github.com/microsoft/fabric-cicd/issues/45))\n" }, { "path": ".changes/v0.1.7.md", "content": "## [v0.1.7](https://pypi.org/project/fabric-cicd/0.1.7) - February 26, 2025\n\n### 🔧 Bug Fix\n\n- Fix special character support in files ([#129](https://github.com/microsoft/fabric-cicd/issues/129))\n" }, { "path": ".changes/v0.1.8.md", "content": "## [v0.1.8](https://pypi.org/project/fabric-cicd/0.1.8) - March 04, 2025\n\n### 🔧 Bug Fix\n\n- Handle null byPath object in report definition file ([#143](https://github.com/microsoft/fabric-cicd/issues/143))\n- Support relative directories ([#136](https://github.com/microsoft/fabric-cicd/issues/136)) ([#132](https://github.com/microsoft/fabric-cicd/issues/132))\n- Increase special character support ([#134](https://github.com/microsoft/fabric-cicd/issues/134))\n\n### ⚡ Additional Optimizations\n\n- Changelog now available with version check ([#127](https://github.com/microsoft/fabric-cicd/issues/127))\n" }, { "path": ".changes/v0.1.9.md", "content": "## [v0.1.9](https://pypi.org/project/fabric-cicd/0.1.9) - March 11, 2025\n\n### 🆕 New Items Support\n\n- Support for Mirrored Database item type ([#145](https://github.com/microsoft/fabric-cicd/issues/145))\n\n### ⚡ Additional Optimizations\n\n- Increase reserved name wait time ([#135](https://github.com/microsoft/fabric-cicd/issues/135))\n" }, { "path": ".changes/v0.2.0.md", "content": "## [v0.2.0](https://pypi.org/project/fabric-cicd/0.2.0) - February 16, 2026\n\n### ✨ New Functionality\n\n- Support parallelize deployments within a given item type by [mdrakiburrahman](https://github.com/mdrakiburrahman) ([#719](https://github.com/microsoft/fabric-cicd/issues/719))\n- Add a black-box REST API testing harness by [mdrakiburrahman](https://github.com/mdrakiburrahman) ([#738](https://github.com/microsoft/fabric-cicd/issues/738))\n- Change header print messages to info log by [mwc360](https://github.com/mwc360) ([#771](https://github.com/microsoft/fabric-cicd/issues/771))\n- Add support for semantic model binding per environment by [shirasassoon](https://github.com/shirasassoon) ([#689](https://github.com/microsoft/fabric-cicd/issues/689))\n\n### 🔧 Bug Fix\n\n- Remove OrgApp item type support by [shirasassoon](https://github.com/shirasassoon) ([#758](https://github.com/microsoft/fabric-cicd/issues/758))\n- Improve environment-mapping behavior in optional config fields by [shirasassoon](https://github.com/shirasassoon) ([#716](https://github.com/microsoft/fabric-cicd/issues/716))\n- Fix duplicate YAML key detection in parameter validation by [shirasassoon](https://github.com/shirasassoon) ([#752](https://github.com/microsoft/fabric-cicd/issues/752))\n- Add caching for item attribute lookups by [MiSchroe](https://github.com/MiSchroe) ([#704](https://github.com/microsoft/fabric-cicd/issues/704))\n\n### ⚡ Additional Optimizations\n\n- Enable configuration-based deployment without feature flags by [shirasassoon](https://github.com/shirasassoon) ([#805](https://github.com/microsoft/fabric-cicd/issues/805))\n\n### 📝 Documentation Update\n\n- Fix troubleshooting docs by [shirasassoon](https://github.com/shirasassoon) ([#747](https://github.com/microsoft/fabric-cicd/issues/747))\n" }, { "path": ".changes/v0.3.0.md", "content": "## [v0.3.0](https://pypi.org/project/fabric-cicd/0.3.0) - March 09, 2026\n\n### ✨ New Functionality\n\n- Support selective folder deployment using inclusion list by [shirasassoon](https://github.com/shirasassoon) ([#757](https://github.com/microsoft/fabric-cicd/issues/757))\n- Add FABRIC_CICD_VERSION_CHECK_DISABLED environment variable to disable version check on startup by [Ricapar](https://github.com/Ricapar) ([#811](https://github.com/microsoft/fabric-cicd/issues/811))\n- Add enhanced logging configuration options via public functions by [shirasassoon](https://github.com/shirasassoon) ([#842](https://github.com/microsoft/fabric-cicd/issues/842))\n- Add deployment support for Notebook items with `.ipynb` file format by [shirasassoon](https://github.com/shirasassoon) ([#850](https://github.com/microsoft/fabric-cicd/issues/850))\n\n### 🔧 Bug Fix\n\n- Add max workers soft cap with override and handle garbage response from Fabric Data Plane by [mdrakiburrahman](https://github.com/mdrakiburrahman) ([#827](https://github.com/microsoft/fabric-cicd/issues/827))\n- Fix parameter validation failure for item names with accented characters by [shirasassoon](https://github.com/shirasassoon) ([#818](https://github.com/microsoft/fabric-cicd/issues/818))\n- Exclude items with placeholder logical ID from duplicate logical ID check by [shirasassoon](https://github.com/shirasassoon) ([#843](https://github.com/microsoft/fabric-cicd/issues/843))\n\n### ⚡ Additional Optimizations\n\n- Add return value to deploy_with_config by [ayeshurun](https://github.com/ayeshurun) ([#851](https://github.com/microsoft/fabric-cicd/issues/851))\n- Add support for Python 3.13 in the library by [shirasassoon](https://github.com/shirasassoon) ([#855](https://github.com/microsoft/fabric-cicd/issues/855))\n\n### 📝 Documentation Update\n\n- Remove incorrect statement on support for parameter replacements in Platform files by [shirasassoon](https://github.com/shirasassoon) ([#839](https://github.com/microsoft/fabric-cicd/issues/839))\n" }, { "path": ".changes/v0.3.1.md", "content": "## [v0.3.1](https://pypi.org/project/fabric-cicd/0.3.1) - March 12, 2026\n\n### 🔧 Bug Fix\n\n- Fix override behavior of feature flags and constants in config deployment by [shirasassoon](https://github.com/shirasassoon) ([#872](https://github.com/microsoft/fabric-cicd/issues/872))\n" }, { "path": ".changes/v1.0.0.md", "content": "## [v1.0.0](https://pypi.org/project/fabric-cicd/1.0.0) - April 20, 2026\n\n### ⚠️ Breaking Change\n\n- Remove default credential fallback; explicit token credential is now required by [shirasassoon](https://github.com/shirasassoon) ([#909](https://github.com/microsoft/fabric-cicd/issues/909))\n- Remove implicit authentication in Microsoft Fabric Notebook and identity logging; require explicit token credential and keyword-only arguments for `FabricWorkspace` and `deploy_with_config` by [shirasassoon](https://github.com/shirasassoon) ([#930](https://github.com/microsoft/fabric-cicd/issues/930))\n\n### 🆕 New Items Support\n\n- Add support for Ontology item type by [shirasassoon](https://github.com/shirasassoon) ([#796](https://github.com/microsoft/fabric-cicd/issues/796))\n\n### ✨ New Functionality\n\n- Improve transparency of `deploy_with_config` function in success and failure scenarios by [shirasassoon](https://github.com/shirasassoon) ([#695](https://github.com/microsoft/fabric-cicd/issues/695))\n- Extend API response collection to unpublish operations by [shirasassoon](https://github.com/shirasassoon) ([#877](https://github.com/microsoft/fabric-cicd/issues/877))\n- Add `get_changed_items()` utility function to detect Fabric items changed via git diff for use with selective deployment by [vipulb91](https://github.com/vipulb91) ([#865](https://github.com/microsoft/fabric-cicd/issues/865))\n\n### 🔧 Bug Fix\n\n- Prevent unintended GUID replacements in Variable Library item files during publish by [shirasassoon](https://github.com/shirasassoon) ([#884](https://github.com/microsoft/fabric-cicd/issues/884))\n- Fix YAML content check to reject notebook and other non-YAML files during `key_value_replace` parameterization, preventing file corruption by [shirasassoon](https://github.com/shirasassoon) ([#890](https://github.com/microsoft/fabric-cicd/issues/890))\n- Fix Notebook deployment failure caused by non-deterministic ordering of definition files in API payload by [shirasassoon](https://github.com/shirasassoon) ([#869](https://github.com/microsoft/fabric-cicd/issues/869))\n- Ignore parameter file when not explicitly defined in config file by [aviatco](https://github.com/aviatco) ([#866](https://github.com/microsoft/fabric-cicd/issues/866))\n- Add `enable_hard_delete` feature flag to bypass workspace recycle bin during unpublish by [shirasassoon](https://github.com/shirasassoon) ([#924](https://github.com/microsoft/fabric-cicd/issues/924))\n- Add timeout for long-running operation polling by [shirasassoon](https://github.com/shirasassoon) ([#919](https://github.com/microsoft/fabric-cicd/issues/919))\n" }, { "path": ".changie.yaml", "content": "# docs: https://changie.dev/config/\n---\nchangesDir: .changes\nunreleasedDir: unreleased\nheaderPath: header.tpl.md\nversionExt: md\nchangelogPath: docs/changelog.md\nversionFormat: '## [{{.Version}}](https://pypi.org/project/fabric-cicd/{{.Version}}) - {{.Time.Format \"January 02, 2006\"}}'\nkindFormat: \"### {{.Kind}}\"\nchangeFormat: \"* {{.Body}} by [{{.Custom.Author}}]({{.Custom.AuthorLink}}) ([#{{.Custom.Issue}}]({{.Custom.IssueLink}}))\"\nkinds:\n - label: \"⚠️ Breaking Change\"\n key: breaking\n auto: major\n - label: \"🆕 New Items Support\"\n key: new-items\n auto: minor\n - label: \"✨ New Functionality\"\n key: added\n auto: minor\n - label: \"🔧 Bug Fix\"\n key: fixed\n auto: patch\n - label: \"⚡ Additional Optimizations\"\n key: optimization\n auto: patch\n - label: \"📝 Documentation Update\"\n key: docs\n auto: patch\nnewlines:\n afterChangelogHeader: 0\n beforeChangelogVersion: 1\n endOfVersion: 1\n afterKind: 1\n beforeKind: 1\nenvPrefix: CHANGIE_\ncustom:\n - key: Issue\n label: Issue Number\n type: int\n minLength: 1\n - key: Author\n label: Author's GitHub Username\n type: string\n minLength: 3\npost:\n - key: AuthorLink\n value: \"https://github.com/{{.Custom.Author}}\"\n - key: IssueLink\n value: \"https://github.com/microsoft/fabric-cicd/issues/{{.Custom.Issue}}\"\nreplacements:\n - path: src/fabric_cicd/constants.py\n find: 'VERSION = \".*\"'\n replace: 'VERSION = \"{{.VersionNoPrefix}}\"'\n" }, { "path": ".github/CODEOWNERS", "content": "##########################################\n\n# CODE OWNERS\n\n##########################################\n\n# default ownership: default owners for everything in the repo (Unless a later match takes precedence)\n\n- @microsoft/fabric-cicd\n" }, { "path": ".github/ISSUE_TEMPLATE/1-bug.yml", "content": "name: \"🐛 Bug Report\"\ndescription: Report a bug\ntitle: \"[BUG] \"\nlabels: [\"bug\"]\nbody:\n - type: markdown\n attributes:\n value: Thanks for taking the time to report a bug! Please fill out the information below to help us diagnose and fix the issue.\n\n - type: input\n id: library-version\n attributes:\n label: Library Version\n description: What is the library version?\n validations:\n required: true\n\n - type: input\n id: python-version\n attributes:\n label: Python Version\n description: Run `python --version` to get your version\n validations:\n required: true\n\n - type: dropdown\n id: operating-system\n attributes:\n label: Operating System\n description: What operating system are you using?\n options:\n - Windows\n - macOS\n - Linux\n validations:\n required: true\n\n - type: dropdown\n id: auth-method\n attributes:\n label: Authentication Method\n description: How are you authenticating?\n options:\n - Azure CLI (az login)\n - Service principal (secret)\n - Service principal (certificate)\n - Service principal (federated credential)\n - Managed identity\n - Other\n validations:\n required: true\n\n - type: textarea\n id: problem-description\n attributes:\n label: What is the problem?\n description: Describe the bug you encountered and what the gap is.\n validations:\n required: true\n\n - type: textarea\n id: reproduction-steps\n attributes:\n label: Steps to reproduce\n description: Please provide step-by-step instructions to reproduce the bug\n validations:\n required: true\n\n - type: textarea\n id: expected-behavior\n attributes:\n label: Expected behavior\n description: What did you expect to happen?\n validations:\n required: true\n\n - type: textarea\n id: actual-behavior\n attributes:\n label: Actual behavior\n description: What actually happened? Include error messages if applicable.\n validations:\n required: true\n\n - type: textarea\n id: solution-description\n attributes:\n label: Is there an ideal solution?\n description: Describe the ideal solution if there is one.\n validations:\n required: false\n\n - type: textarea\n id: additional-context\n attributes:\n label: Additional context, screenshots, logs, error output, etc\n description: Add any other relevant details like specific workspace/item types involved, related issues or documentation, API request IDs. If it is long, please paste to https://gist.github.com/ and insert the link here. Ensure to strip any non public information.\n validations:\n required: false\n" }, { "path": ".github/ISSUE_TEMPLATE/2-feature.yml", "content": "name: \"🚀 Feature Request\"\ndescription: Suggest an idea or enhancement for fabric-cicd\ntitle: \"[FEATURE] \"\nlabels: [\"enhancement\"]\nbody:\n - type: markdown\n attributes:\n value: |\n Thanks for suggesting a new feature! Please provide as much detail as possible to help us understand and evaluate your request.\n\n - type: textarea\n id: use-case-problem\n attributes:\n label: Use Case / Problem\n description: |\n Describe the problem or use case that this feature would solve. Include:\n - What problem are you trying to solve?\n - Current limitations or pain points\n - How this fits into your workflow\n validations:\n required: true\n\n - type: textarea\n id: proposed-solution\n attributes:\n label: Proposed Solution\n description: |\n Describe the solution you'd like to see implemented:\n - Specific functionality\n - Expected behavior and output\n - Integration with existing fabric-cicd features\n validations:\n required: true\n\n - type: textarea\n id: alternatives-considered\n attributes:\n label: Alternatives Considered\n description: |\n Describe any alternative solutions or features you've considered:\n - Workarounds you're currently using\n - Other tools or approaches that could solve this\n - Why those alternatives are insufficient\n\n - type: checkboxes\n id: impact-assessment\n attributes:\n label: Impact Assessment\n description: Help us understand the impact of this feature\n options:\n - label: This would help me personally\n - label: This would help my team/organization\n - label: This would help the broader fabric-cicd community\n - label: This aligns with Microsoft Fabric roadmap items\n\n - type: checkboxes\n id: implementation-attestation\n attributes:\n label: Implementation Attestation\n description: Please confirm your understanding of implementation considerations (required)\n options:\n - label: I understand this feature should maintain backward compatibility (if applicable)\n required: true\n - label: I understand this feature should not introduce performance regressions for existing workflows\n required: true\n - label: I acknowledge that new features must follow fabric-cicd's established patterns and conventions\n required: true\n\n - type: textarea\n id: implementation-notes\n attributes:\n label: Implementation Notes\n description: |\n If you have technical suggestions for implementation, consider:\n - Which item types are affected? (Notebook, DataPipeline, Environment, etc.)\n - Does this require changes to core classes/functions? Which ones?\n - Does this require changes to the parameterization framework?\n - Which Fabric REST APIs would be involved (if any)?\n" }, { "path": ".github/ISSUE_TEMPLATE/3-documentation.yml", "content": "name: \"📝 Documentation Feedback\"\ndescription: Provide documentation feedback\ntitle: \"[DOCUMENTATION] \"\nlabels: [\"documentation\"]\nbody:\n - type: markdown\n attributes:\n value: Thanks for taking the time to provide documentation feedback! Please include as much detail as possible to help us improve our documentation.\n\n - type: input\n id: documentation-location\n attributes:\n label: URL to documentation\n description: Provide a URL to the documentation location.\n placeholder: \"https://microsoft.github.io/fabric-cicd/...\"\n validations:\n required: true\n\n - type: dropdown\n id: feedback-type\n attributes:\n label: Type of feedback\n description: What kind of documentation issue is this?\n options:\n - Unclear or confusing content\n - Missing information\n - Incorrect or outdated information\n - Typo or grammatical error\n - Code example issue\n - Broken link\n - Suggestion for improvement\n - Other\n validations:\n required: true\n\n - type: textarea\n id: current-content\n attributes:\n label: Current content\n description: Copy the relevant section of documentation that needs improvement (if applicable).\n placeholder: \"Paste the current text or describe what exists...\"\n validations:\n required: false\n\n - type: textarea\n id: feedback\n attributes:\n label: Feedback\n description: What is the issue or what improvement would you suggest?\n placeholder: \"Describe what's wrong or what could be better...\"\n validations:\n required: true\n\n - type: textarea\n id: suggested-change\n attributes:\n label: Suggested change\n description: If you have a specific suggestion, please provide it here.\n placeholder: \"Provide your suggested text, code, or improvement...\"\n validations:\n required: false\n\n - type: dropdown\n id: user-experience\n attributes:\n label: Experience with fabric-cicd\n description: This helps us understand our audience better.\n options:\n - New to fabric-cicd\n - Some experience with fabric-cicd\n - Experienced user\n validations:\n required: false\n\n - type: textarea\n id: additional-context\n attributes:\n label: Additional context\n description: Add any other context, screenshots, or related links here.\n validations:\n required: false\n" }, { "path": ".github/ISSUE_TEMPLATE/4-question.yml", "content": "name: \"❓ Question\"\ndescription: Ask a question about fabric-cicd\ntitle: \"[QUESTION] \"\nlabels: [\"question\"]\nbody:\n - type: markdown\n attributes:\n value: Thanks for your question! Please provide as much detail as possible to help us assist you effectively.\n\n - type: textarea\n id: question\n attributes:\n label: What is the question?\n description: A clear and concise description of what you'd like to know or need help with.\n validations:\n required: true\n\n - type: textarea\n id: context\n attributes:\n label: Context\n description: |\n Add context and include:\n - Your use case or scenario\n - What you've already tried\n - Specifics on what was involved (code snippets, configurations, etc.)\n validations:\n required: true\n\n - type: input\n id: library-version\n attributes:\n label: Library Version\n description: What is the library version? (if relevant)\n\n - type: dropdown\n id: operating-system\n attributes:\n label: Operating System\n description: What operating system are you using? (if relevant)\n options:\n - Not applicable\n - Windows\n - macOS\n - Linux\n default: 0\n\n - type: checkboxes\n id: documentation-check\n attributes:\n label: Have you checked the documentation?\n description: Please confirm you've reviewed the available resources\n options:\n - label: I have searched existing GitHub issues\n required: true\n - label: I have reviewed the fabric-cicd documentation\n required: true\n\n - type: textarea\n id: additional-information\n attributes:\n label: Additional Information\n description: |\n Any additional details that might help us answer your question:\n - Code snippets\n - Screenshots (if applicable)\n - Error messages (if any)\n\n - type: markdown\n attributes:\n value: |\n ## Alternative Support Channels\n\n For different types of questions, you might also consider:\n\n - **General Fabric questions**: [Developer Community Forum](https://community.fabric.microsoft.com/t5/Developer/bd-p/Developer)\n - **Feature suggestions**: [Fabric Ideas Portal](https://ideas.fabric.microsoft.com/)\n - **Enterprise support**: [Fabric Support Team](https://support.fabric.microsoft.com/)\n - **Community discussions**: [r/MicrosoftFabric on Reddit](https://www.reddit.com/r/MicrosoftFabric/)\n" }, { "path": ".github/ISSUE_TEMPLATE/config.yml", "content": "blank_issues_enabled: false\n" }, { "path": ".github/agents/new-item-type.agent.md", "content": "---\nname: New Item Type\ndescription: Guide and assist with onboarding a new Microsoft Fabric item type into fabric-cicd\nargument-hint: Tell me which Fabric item type you want to add (e.g., \"Add support for Ontology\")\ntools:\n - runInTerminal\n - terminalLastCommand\n - search\n - fetch\n - readFile\n - editFiles\n - createFile\n---\n\n# New Item Type Onboarding Agent\n\n> **Important:** If you are unsure about any detail — such as whether the item type supports definitions, has unique deployment requirements, depends on other item types, or requires parameterization — **always ask the requestor for clarification before proceeding**. List the specific unknowns and ask the requestor to confirm each one before continuing.\n\n## Prerequisites\n\nBefore starting, you need only the **display name** (PascalCase, as used by the Fabric API — e.g., `CopyJob`, `SnowflakeDatabase`). The eligibility gates below will determine the remaining core details.\n\n### Eligibility Gates\n\nBefore proceeding, confirm all of the following. If any gate fails, **stop** — the item type cannot be onboarded.\n\n1. The item type must be supported in source control / Git integration. Fetch the [item definition overview](https://learn.microsoft.com/en-us/rest/api/fabric/articles/item-management/definitions/item-definition-overview) page and check if the item type appears in the **\"Definition Details for Supported Item Types\"** list.\n2. The Fabric API must support deployment for the item type — either full definition deployment or shell-only creation (like Lakehouse/Warehouse). Verify using the steps in **\"How to verify gates 2 and 3\"** below.\n3. The Fabric API must support service principal (SPN) authentication for the item type's deployment operations. fabric-cicd is primarily used in CI/CD pipelines where SPN is the standard authentication method. Verify using the steps in **\"How to verify gates 2 and 3\"** below.\n\n#### How to verify gates 2 and 3\n\nThe Fabric REST API docs follow a predictable URL pattern. Use these steps to directly navigate to the right pages instead of searching the generic API root:\n\n1. **Construct the item type's API items page URL:**\n `https://learn.microsoft.com/en-us/rest/api/fabric/{itemtype-lowercase}/items`\n where `{itemtype-lowercase}` is the PascalCase item type name lowercased with no separators.\n - Examples: `SnowflakeDatabase` → `snowflakedatabase`, `DataPipeline` → `datapipeline`, `KQLDatabase` → `kqldatabase`\n\n2. **Fetch that page and confirm deployment operations exist (Gate 2):**\n - Full definition support: Look for **\"Get [Item] Definition\"** and **\"Update [Item] Definition\"** operations in the Operations table.\n - Shell-only support: If the documentation explicitly states that item definition is not supported (e.g., Warehouse's Create page states \"This API does not support item definition\") and the item has **\"Create [Item]\"** and **\"Update [Item]\"** operations, the item is shell-only. **Important:** There can be nuances with definition support — always present your findings to the requestor and ask them to confirm whether the item should be categorized as full definition or shell-only before proceeding.\n - If neither Create nor definition operations exist, Gate 2 fails.\n\n3. **Fetch the Create endpoint page to check SPN support (Gate 3):**\n `https://learn.microsoft.com/en-us/rest/api/fabric/{itemtype-lowercase}/items/create-{item-type-kebab-case}`\n where `{item-type-kebab-case}` inserts hyphens between PascalCase words and lowercases everything.\n - Examples: `SnowflakeDatabase` → `create-snowflake-database`, `DataPipeline` → `create-data-pipeline`, `KQLDatabase` → `create-kql-database`\n - Find the **\"Microsoft Entra supported identities\"** section and confirm the table shows **\"Service principal and Managed identities: Yes\"**. If it shows \"No\" or the section is missing, Gate 3 fails.\n\n**Important:** You MUST fetch the actual API pages listed above. Do not guess or assume gate results based on the item type name alone. If a URL returns a 404, try alternate kebab-case patterns (e.g., `graphqlapi` instead of `graph-q-l-api`), or fall back to fetching the [Fabric REST API root](https://learn.microsoft.com/en-us/rest/api/fabric/) and navigating to the item type's section. If the documentation is still inaccessible, ask the requestor to provide the relevant API page details.\n\n**Exceptions:** Gates 1 and 3 may be excepted on a case-by-case basis with fabric-cicd team approval — for example, Notebook `.ipynb` format is supported despite not being source-controlled (gate 1 exception). Gate 2 (API deployment support) has no exceptions. Any approved exception must be documented as a known limitation in `docs/how_to/item_types.md`.\n\n### Additional Details (Required)\n\nAfter the eligibility gates pass, you **must** ask the requestor about **every row** in this table and record the answer before starting implementation. Do not skip any row or assume the answer is \"no.\"\n\n| Question to ask the requestor | Example | Affects |\n| ----------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------ | ----------- |\n| Which existing item types must deploy **before** this one? Which existing types depend on this one deploying first? | Eventhouse → KQLDatabase, SemanticModel → Report | Step 1b |\n| Should certain file paths within the item folder be skipped during publish? | `.pbi/`, `.children/` | Step 1d |\n| Does the API use a non-standard definition format? If yes, provide the exact format string and instructions for how it should be handled. | `ipynb`, `SparkJobDefinitionV2` | Step 1e |\n| Does deleting this item destroy user data? | Lakehouse, Eventhouse | Step 1f |\n| Can items of this type reference each other? | Data Pipeline invokes another Data Pipeline | Steps 1g, 2 |\n| Does the item need custom deployment logic, dependency resolution, or special parameterization? If yes, describe what is needed. | Lakehouse creation payload, Environment async check, KQL items need query service URI (via dynamic replacement), SemanticModel binding parameter | Step 2 |\n\n---\n\n## Safety Rules\n\n- **Never hardcode secrets, tokens, or credentials** in publisher code or tests\n- **Use deterministic test data** — no real tenant IDs, workspace IDs, or user emails\n- **Follow existing patterns** — consistency is more important than cleverness\n- **Validate all assumptions** — if unsure about API behavior, ask the requestor\n\n---\n\n## Integration Checklist\n\nOnce eligibility gates pass and additional details are gathered, proceed through these steps in order. If you encounter something that doesn't fit the steps below, **stop and ask the requestor** rather than improvising.\n\n### Step 1 — Register the Item Type in Constants\n\n**File:** `src/fabric_cicd/constants.py`\n\nRead `constants.py` to see the existing patterns for each mapping below. Add entries following the same format.\n\n#### 1a. Add to the `ItemType` enum\n\nAdd a new member in alphabetical order within the enum:\n\n```python\nclass ItemType(str, Enum):\n # ... existing members ...\n NEW_TYPE = \"NewType\"\n```\n\n**Rules:**\n\n- Enum member name uses `UPPER_SNAKE_CASE`\n- Enum value uses `PascalCase` matching the Fabric API `type` field exactly\n\n#### 1b. Add to `SERIAL_ITEM_PUBLISH_ORDER`\n\nChoose the correct position based on the item's dependencies. Items that other items depend on must come **earlier** in the order. The unpublish order is automatically the reverse.\n\n##### How to determine the correct position\n\n> **How items reference each other:** In Fabric definition files, items reference other items via either **logical IDs** (workspace-agnostic GUIDs assigned by Git integration — automatically replaced by fabric-cicd) or **item IDs** (environment-specific GUIDs that differ per workspace — must be resolved via parameterization in `parameter.yml`). When reviewing definition files to identify dependencies, look for both types of references.\n\n1. **Read the current `SERIAL_ITEM_PUBLISH_ORDER`** in `constants.py` to see the latest order.\n2. **Identify upstream dependencies** — types the new item depends on. Check definition files for references (logical IDs, item IDs, paths, connection strings) and ask the requestor. The new item must go **after** the highest-numbered upstream dependency.\n3. **Identify downstream dependents** — existing types that will depend on the new item. The new item must go **before** the lowest-numbered downstream dependent.\n4. Place the new item anywhere in the valid range between those two bounds. If there is no gap, renumber existing entries to make room.\n5. If the new item has **no dependencies in either direction**, place it at the end of the order.\n6. **When in doubt, ask the requestor** — do not guess dependency relationships.\n\n#### 1c. Optionally add to `SHELL_ONLY_PUBLISH`\n\nIf the API does **not** support item definition and only supports metadata (shell) deployment — like Warehouse, ML Experiment, etc.\n\n#### 1d. Optionally add to `EXCLUDE_PATH_REGEX_MAPPING`\n\nIf certain file paths within the item should be excluded during publish (e.g., `.pbi/` folders for Report/SemanticModel, `.children/` for Eventhouse).\n\n#### 1e. Optionally add to `API_FORMAT_MAPPING`\n\nIf the Fabric API requires a specific format string for the item's definition (e.g., `\"ipynb\"` for Notebooks, `\"SparkJobDefinitionV2\"` for Spark Job Definitions).\n\nOnly add an API format if the format is supported in Fabric's Git integration (source control). If the format is not source-controlled, it generally should not be added unless approved by the fabric-cicd team.\n\n**Known exception:** Notebook `.ipynb` format is supported despite not being source-controlled. This is documented as a known limitation in `docs/how_to/item_types.md`.\n\n#### 1f. Optionally add to `UNPUBLISH_FLAG_MAPPING`\n\nIf unpublishing the item is destructive and should be gated behind a feature flag (like Lakehouse, Warehouse, Eventhouse). If so, also add a new `FeatureFlag` enum member.\n\n#### 1g. Optionally add to `ITEM_TYPE_TO_FILE`\n\nIf items of this type can reference other items of the **same** type (intra-type dependencies), register the content file that contains those references so the dependency module knows which file to parse. This is required when implementing intra-type dependency ordering (see Step 2 — Dependency Ordering below).\n\n---\n\n### Step 2 — Create a Publisher Class\n\n**File:** `src/fabric_cicd/_items/_newtype.py` (new file)\n\nCreate a publisher class that extends `ItemPublisher`. The simplest case:\n\n```python\n# src/fabric_cicd/_items/_newtype.py\n# Copyright (c) Microsoft Corporation.\n# Licensed under the MIT License.\n\n\"\"\"Functions to process and deploy NewType item.\"\"\"\n\nfrom fabric_cicd._items._base_publisher import ItemPublisher\nfrom fabric_cicd.constants import ItemType\n\n\nclass NewTypePublisher(ItemPublisher):\n \"\"\"Publisher for NewType items.\"\"\"\n\n item_type = ItemType.NEW_TYPE.value\n```\n\nFor more complex items, you can override these methods from `ItemPublisher`:\n\n| Method | Purpose | When to Override |\n| ------------------------------- | --------------------------------------- | -------------------------------------------------------- |\n| `publish_one(item_name, _item)` | Custom publish logic per item | Custom file processing, exclude paths, creation payloads |\n| `get_items_to_publish()` | Filter or order items before publishing | Custom item filtering |\n| `get_unpublish_order(items)` | Dependency-aware unpublish ordering | **Must also set `has_dependency_tracking = True`** |\n| `pre_publish_all()` | Pre-publish checks | e.g., Environment publish state check |\n| `post_publish_all()` | Post-publish actions | e.g., Semantic Model connection binding |\n| `post_publish_all_check()` | Async publish state verification | **Must also set `has_async_publish_check = True`** |\n\n#### Intra-Type Dependency Ordering (DAG)\n\nIf items of the same type can reference each other (e.g., a pipeline invoking another pipeline, a dataflow sourcing from another dataflow), publish and unpublish order must respect those internal dependencies. This requires:\n\n1. **A reference-finding function** that scans an item's content file and returns names of other items (of the same type) it depends on.\n2. **Sequential publish with dependency ordering** — set `has_dependency_tracking = True` and configure `parallel_config = ParallelConfig(enabled=False, ordered_items_func=...)` with a function that returns item names in topological order.\n3. **Dependency-aware unpublish** — override `get_unpublish_order()` to return items in reverse dependency order.\n4. **Choose or implement a sorting strategy:**\n - **Reuse `_manage_dependencies.py`** (preferred) — provides generic topological sort via `set_publish_order()` and `set_unpublish_order()`. You supply a `find_referenced_items_func(workspace, content, lookup_type) -> list[str]` callback. Used by `DataPipeline`. Requires `ITEM_TYPE_TO_FILE` registration (Step 1g).\n - **Custom DFS** — if the dependency resolution has unique requirements (e.g., Dataflow's parameterization-aware source detection), implement a custom ordering function as done in `_dataflowgen2.py`.\n\nSee `_datapipeline.py` (generic topological sort) and `_dataflowgen2.py` (custom DFS) for reference implementations.\n\n#### Custom File Processing Callback (`func_process_file`)\n\nThe standard publish pipeline automatically handles logical ID replacement, parameterization, and workspace ID replacement for all item types. If the item type requires **additional, item-type-specific content transformations** that the generic pipeline cannot handle, define a module-level `func_process_file(workspace_obj, item_obj, file_obj) -> str` callback and pass it to `_publish_item()` in `publish_one()`. This callback runs **first**, before the generic pipeline steps.\n\nSee `_report.py`, `_kqldashboard.py`, or `_kqlqueryset.py` for examples of this pattern.\n\n#### Parameterization\n\nGeneric parameterization (`find_replace`, `key_value_replace`) is applied automatically to all item types — no publisher code needed. If the item type requires a **specialized parameter key** in `parameter.yml` (e.g., Environment's `spark_pool`, SemanticModel's `semantic_model_binding`):\n\n- If the specialized parameterization is **not required for deployment** (e.g., connection binding can be done later), proceed with onboarding and coordinate with the fabric-cicd team to add it separately.\n- If the specialized parameterization **blocks deployment** (e.g., the item cannot be deployed without it), coordinate with the fabric-cicd team before completing the integration — the item type should not be onboarded until the parameterization is supported.\n\n---\n\n### Step 3 — Register the Publisher in the Factory Method\n\n**File:** `src/fabric_cicd/_items/_base_publisher.py`\n\nUpdate the `ItemPublisher.create()` factory method — add an import for the new publisher class and a mapping entry in `publisher_mapping`.\n\n**Rules:**\n\n- Follow the same ordering as `SERIAL_ITEM_PUBLISH_ORDER` for the mapping dictionary\n- Import must be inside the `create()` method (lazy imports to avoid circular dependencies)\n\n---\n\n### Step 4 — Add Tests\n\n**Directory:** `tests/`\n\nCreate or update test files for the new item type:\n\n- **Unit tests:** Add tests in `tests/test_publish.py` using the `create_test_item()` helper to create item data inline. For publisher-specific behavior, see `tests/test_environment_publish.py` as an example.\n- **Integration tests:** Add the item type to `sample/workspace/` and to the `item_types_to_deploy` list in `tests/test_integration_publish.py`. The HTTP trace (`tests/fixtures/http_trace.json.gz`) must be regenerated against a real Fabric workspace — see `tests/fixtures/README.md`. This step requires human intervention.\n\n**Rules:**\n\n- Use deterministic test data — no real tenant IDs, workspace IDs, or user emails\n- Never hardcode secrets, tokens, or credentials in tests\n- Mock all API interactions using the existing test patterns\n\n---\n\n### Step 5 — Documentation Updates\n\n#### 5a. Supported Item Types List (auto-generated)\n\nThe supported item types list auto-generates from the `ItemType` enum via `docs/config/pre-build/update_item_types.py` — **no manual update is needed for the list**.\n\n#### 5b. Item Types How-To Page\n\n**File:** `docs/how_to/item_types.md`\n\nAdd a new section for the item type following the existing pattern.\n\n---\n\n### Step 6 — Sample Files and Deployment Validation\n\n**Do not create sample files yourself.** Sample items must come from a real Fabric workspace exported via Git integration — the agent cannot generate valid definition files.\n\nEncourage the requestor to:\n\n1. **Add a sample item** to `sample/workspace/` by exporting an instance of the item type from a Fabric workspace connected to Git. The exported folder should follow the naming convention `{ItemName}.{ItemType}/` (e.g., `Hello Copy Job.CopyJob/`).\n2. **Add the item type** to `item_type_in_scope` in `devtools/debug_local.py` and run a sample deployment against a real workspace to validate end-to-end behavior.\n3. **Add the item type** to `item_types_to_deploy` in `tests/test_integration_publish.py`. The HTTP trace (`tests/fixtures/http_trace.json.gz`) must also be regenerated against a real workspace — this is a manual step.\n\n---\n\n## Patterns and Reference Examples\n\n**Default workflow**: All item types require steps 1a, 1b, 2, 3, 4, 5, 6. Use this table to determine which optional steps (1c, 1d, 1e, 1f, 1g) to skip based on your item type's characteristics. Read the example files for implementation details.\n\n| Pattern | Skip Steps | Example Files | Key Details |\n| -------------------------------- | ------------------ | --------------------------------------------------- | ---------------------------------------------------------------------------------------------- |\n| **Simple** (no special behavior) | 1c, 1d, 1e, 1f, 1g | `_graphqlapi.py`, `_copyjob.py` | Default publish behavior, no overrides needed |\n| **Exclude paths** | 1c, 1e, 1f, 1g | `_dataagent.py`, `_eventhouse.py` | Override `publish_one()` to pass `exclude_path` — do step 1d |\n| **API format** | 1c, 1d, 1f, 1g | `_notebook.py` | Override `publish_one()` to pass `api_format` — do step 1e |\n| **Custom file processing** | 1c, 1d, 1e, 1f, 1g | `_report.py`, `_kqldashboard.py`, `_kqlqueryset.py` | Define `func_process_file` callback, pass to `_publish_item()` |\n| **Shell-only** (metadata only) | 1d, 1e, 1f, 1g | `_warehouse.py`, `_lakehouse.py` | May need creation payload logic in `publish_one()` — do step 1c |\n| **Destructive unpublish** | 1c, 1d, 1e, 1g | `_lakehouse.py`, `_eventhouse.py` | Add new `FeatureFlag` enum member — do step 1f |\n| **Intra-type dependencies** | 1c, 1d, 1e, 1f | `_datapipeline.py`, `_dataflowgen2.py` | Set `has_dependency_tracking`, `ParallelConfig`, override `get_unpublish_order()` — do step 1g |\n| **Post-publish actions** | 1c, 1d, 1e, 1f, 1g | `_semanticmodel.py` | Override `post_publish_all()` |\n| **Async publish check** | 1c, 1d, 1e, 1f, 1g | `_environment.py` | Set `has_async_publish_check = True`, override `post_publish_all_check()` |\n\n**Combined patterns**: If your item type needs multiple patterns, skip only steps that appear in ALL applicable Skip Steps columns.\n\n---\n\n## Final Validation\n\nAfter completing all steps, run the mandatory validation suite:\n\n```\nuv run python -c \"from fabric_cicd import FabricWorkspace; print('Import successful')\"\nuv run pytest -v\nuv run ruff format\nuv run ruff check\n```\n\nAll four must pass before the task is complete.\n\n---\n\n## Key Files Quick Reference\n\n| File | Purpose |\n| ------------------------------------------------ | --------------------------------------------------------------- |\n| `src/fabric_cicd/constants.py` | Item type enum, publish order, feature flags, all type mappings |\n| `src/fabric_cicd/_items/_base_publisher.py` | Base publisher class and factory method |\n| `src/fabric_cicd/_items/` | All item publisher implementations |\n| `src/fabric_cicd/_items/_manage_dependencies.py` | Generic topological sort for intra-type dependencies |\n| `src/fabric_cicd/fabric_workspace.py` | Main workspace management class |\n| `src/fabric_cicd/publish.py` | Top-level publish/unpublish orchestration |\n| `tests/` | All test files |\n| `tests/fixtures/` | Test fixture data |\n| `docs/how_to/item_types.md` | Per-item-type documentation |\n| `docs/config/pre-build/update_item_types.py` | Auto-generates supported item types list from enum |\n| `sample/workspace/` | Example workspace item structures |\n" }, { "path": ".github/copilot-instructions.md", "content": "# Fabric CICD\n\nfabric-cicd is a Python library for Microsoft Fabric CI/CD automation. It supports code-first Continuous Integration/Continuous Deployment automations to integrate Source Controlled workspaces into a deployment framework.\n\nAlways reference these instructions first and fallback to search or bash commands only when you encounter unexpected information that does not match the info here.\n\n## Quick Command Reference\n\n**Prerequisites**: Requires Python 3.9+\n\n| Task | Command | Timeout |\n| ------------ | ---------------------------------------------------------------------------------------- | ------- |\n| Setup | `pip install uv && uv sync --dev` (NEVER CANCEL) | 120+s |\n| Test | `uv run pytest -v` (NEVER CANCEL) | 120+s |\n| Import check | `uv run python -c \"from fabric_cicd import FabricWorkspace; print('Import successful')\"` | 30s |\n| Format | `uv run ruff format` (Fix formatting issues) | 60s |\n| Lint check | `uv run ruff check` (Check for linting issues) | 60s |\n| Format check | `uv run ruff format --check` (Verify formatting is correct) | 60s |\n| Docs build | `uv run mkdocs build --clean` (Build documentation) | 60s |\n| Docs serve | `uv run mkdocs serve` (Start local documentation server) | 60s |\n\n**Mandatory Validation (ALWAYS):**\n\n1. Import check → 2. Run tests → 3. Format code → 4. Check linting → 5. Commit\n\n**Critical**: NEVER cancel build/test commands. CI (`.github/workflows/validate.yml`) will fail if validation workflow incomplete.\n\n## Authentication\n\nMust provide explicit `token_credential` parameter to `FabricWorkspace`.\n\n**Methods:**\n\n- **Local development**: `AzureCliCredential()` or `AzurePowerShellCredential()`\n- **CI/CD pipelines**: `ClientSecretCredential()` with service principal\n- **Testing/imports**: No authentication needed\n\n**Example:**\n\n```python\nfrom azure.identity import AzureCliCredential\nfrom fabric_cicd import FabricWorkspace\n\ntoken_credential = AzureCliCredential()\nworkspace = FabricWorkspace(\n workspace_id=\"your-id\",\n repository_directory=\"/path/to/workspace/items\",\n token_credential=token_credential\n)\n```\n\n## Basic Usage\n\n### Programmatic API\n\n```python\nfrom azure.identity import AzureCliCredential\nfrom fabric_cicd import FabricWorkspace, publish_all_items, unpublish_all_orphan_items\n\ntoken_credential = AzureCliCredential()\n# Initialize workspace (supports either workspace_id OR workspace_name)\nworkspace = FabricWorkspace(\n workspace_id=\"your-workspace-id\", # Alternative: workspace_name=\"your-workspace-name\"\n environment=\"DEV\",\n repository_directory=\"/path/to/workspace/items\",\n item_type_in_scope=[\"Notebook\", \"DataPipeline\", \"Environment\"],\n token_credential=token_credential\n)\n\n# Deploy items\npublish_all_items(workspace)\n\n# Clean up orphaned items\nunpublish_all_orphan_items(workspace)\n```\n\n### Config-Based Deployment\n\nAlternative: `deploy_with_config()` centralizes deployment settings in YAML.\n\n```python\nfrom azure.identity import AzureCliCredential\nfrom fabric_cicd import deploy_with_config\ntoken_credential = AzureCliCredential()\nresult = deploy_with_config(\n config_file_path=\"config.yml\",\n environment=\"dev\",\n token_credential=token_credential\n)\n```\n\n**Implementation files:**\n\n- Entry points: `deploy_with_config()`, `publish_all_items()`, `unpublish_all_orphan_items()` in `src/fabric_cicd/publish.py`\n- Config utilities: `src/fabric_cicd/_common/_config_utils.py` (loading, extraction)\n- Config validation: `src/fabric_cicd/_common/_config_validator.py`\n- Documentation: `docs/how_to/config_deployment.md`\n- Tests: `tests/test_deploy_with_config.py`, `tests/test_config_validator.py`\n\n### Public API Exports\n\nOnly import from the top-level package (`src/fabric_cicd/__init__.py`). Do not import internal modules directly.\n\n**Exported symbols:**\n\n- `FabricWorkspace` - Main workspace management class\n- `publish_all_items` - Deploy all items in scope\n- `unpublish_all_orphan_items` - Remove orphaned items\n- `deploy_with_config` - Config-based deployment\n- `DeploymentResult`, `DeploymentStatus` - Deployment result types\n- `ItemType` - Enum of supported Fabric item types\n- `FeatureFlag` - Enum of feature flags\n- `append_feature_flag` - Add feature flags programmatically\n- `change_log_level`, `configure_external_file_logging`, `disable_file_logging` - Logging utilities\n\n## Project Structure\n\n```\n/\n├── .github/workflows/ # CI/CD pipelines (test.yml, validate.yml, bump.yml)\n├── docs/ # Documentation source files\n├── docs/example/ # CI/CD scenario patterns (Azure DevOps, GitHub Actions, local development)\n├── sample/ # Example workspace structure and items\n├── src/fabric_cicd/ # Main library source code\n├── tests/ # Test files\n├── pyproject.toml # Project configuration and dependencies\n├── ruff.toml # Code formatting and linting configuration\n├── mkdocs.yml # Documentation configuration\n├── activate.ps1 # PowerShell setup script (Windows only)\n└── uv.lock # Dependency lock file\n```\n\n## Development Guidelines\n\n### Core Concepts\n\n- **Publisher Classes**: Handle deployment logic for each Fabric item type in `src/fabric_cicd/_items/`\n- **Serial Publishing**: Items deploy in dependency order via `SERIAL_ITEM_PUBLISH_ORDER`\n- **Parameterization**: YAML-based environment-specific value replacement\n\n### Supported Item Types\n\nValid values for `item_type_in_scope` are defined in the `ItemType` enum in `src/fabric_cicd/constants.py`. Always reference that file for the current list — do not hard-code item type strings without verifying them against the enum.\n\nThe publish/unpublish dependency order is defined in `SERIAL_ITEM_PUBLISH_ORDER` in the same file.\n\n### Common Development Patterns\n\n- **Adding constants**: Add to `ItemType` enum + `SERIAL_ITEM_PUBLISH_ORDER` in `src/fabric_cicd/constants.py`\n- **Adding publisher**: Extend `ItemPublisher` + register in `_base_publisher.py` factory\n- **Adding public exports**: Update `__all__` in `src/fabric_cicd/__init__.py`\n\n### Testing Guidelines\n\n**Always add/update tests for:**\n\n- New functionality or features\n- Bug fixes that change behavior\n- Core logic changes in any module\n- Publisher classes and deployment logic\n- Configuration and validation logic\n- API integrations and external calls\n\n**Testing approach**: Mock all external dependencies, use `requests_mock` for Azure APIs, `tmpdir` for file operations. Focus on testing business logic, error handling, and integration points.\n\n**Test file naming**: Follow the conventions in the `tests/` directory. Review existing test files to match the naming pattern before creating new ones.\n\n### Files to Avoid Modifying\n\n- `coverage_report/`, `site/`, `htmlcov/` - Auto-generated\n- `uv.lock` - Managed by uv\n- `.github/workflows/` - Affects CI validation\n\n### Dependencies & Testing\n\n**Runtime:** `azure-identity`, `dpath`, `pyyaml`, `requests` \n**Development:** `uv`, `ruff`, `pytest`, `mkdocs-material`\n\n**Test Types:** Unit (`tests/test_*.py`), Integration (mocked APIs), Parameter/File Handling, Workspace management\n\n**GitHub Actions:** `test.yml` (PR tests), `validate.yml` (formatting/linting), `bump.yml` (version bumps - vX.X.X format)\n\n**Microsoft Fabric APIs:** https://learn.microsoft.com/en-us/rest/api/fabric/\n\n## Pull Request Requirements\n\n**Base branch:** Always target `main` unless otherwise specified.\n\n**Title format:** \"Fixes #123 - Short Description\" where #123 is the issue number\n\n- Use \"Fixes\" for bug fixes, \"Closes\" for features, \"Resolves\" for other changes\n- Example: \"Fixes #520 - Add Python version requirements to documentation\"\n- Exception: Version bump PRs use \"vX.X.X\" format only\n\n**Requirements:**\n\n- PR description should be copilot generated summary\n- Pass ruff formatting and linting checks\n- Pass all tests\n- All PRs must be linked to valid GitHub issue\n\n## Do Not\n\n- Do not modify `uv.lock` manually — it is managed by `uv`\n- Do not import from internal modules (e.g., `fabric_cicd._items`) — only use the public API from `fabric_cicd`\n- Do not add `print()` statements — use the standard `logging` module\n- Do not create PRs without a linked GitHub issue\n- Do not modify `.github/workflows/` files unless explicitly required\n\n## Agent Troubleshooting\n\n**Common Failures:**\n\n- **Import errors**: Use `uv run python` prefix to ensure virtual environment\n- **Test pollution**: Azure credentials interfering - ensure proper mocking\n- **Setup failures**: Run `uv sync --dev` if modules missing\n- **Formatting issues**: Run `uv run ruff format` to auto-fix most issues\n- **CI failures**: Missing format/lint step in validation workflow\n\n**Authentication Strategy for Agents:**\n\n1. For testing/imports: No auth needed\n2. For publish operations: Use `AzureCliCredential()` (If `CredentialUnavailableError` occurs, user needs to run `az login` first)\n3. Context: Import check works without auth, but publish operations require credentials\n\n## Key Files to Monitor\n\n**Core System Files:**\n\n- `src/fabric_cicd/constants.py` - Version and configuration constants\n- `src/fabric_cicd/fabric_workspace.py` - Main workspace management class\n- `src/fabric_cicd/publish.py` - Main deployment entry points\n- `src/fabric_cicd/_items/` - Publisher classes for all item types\n- `src/fabric_cicd/_common/` - Config utilities, validation, and exceptions\n\n**Configuration Files:**\n\n- `pyproject.toml` - Project dependencies and configuration\n- `sample/workspace/parameter.yml` - Environment-specific parameter template\n\n**Project Structure:**\n\n- `sample/workspace/` - Example Microsoft Fabric item structures\n" }, { "path": ".github/policies/resourceManagement.yml", "content": "id: issue-triage\nname: GitOps.PullRequestIssueManagement\ndescription: Issue triage workflow\nowner:\nresource: repository\ndisabled: false\n\nwhere:\nconfiguration:\n resourceManagementConfiguration:\n\n eventResponderTasks:\n\n - description: Label new issues for triage\n if:\n - payloadType: Issues\n - isAction:\n action: Opened\n then:\n - addLabel:\n label: needs triage\n - addReply:\n reply: >\n Thank you for submitting this issue, ${issueAuthor}.\n \n A member of the Fabric CICD team will review your submission and provide feedback shortly.\n\n - description: Maintainer command to request more info\n if:\n - payloadType: Issue_Comment\n - commentContains:\n pattern: '^/needinfo'\n isRegex: true\n - or:\n - activitySenderHasPermission:\n permission: Admin\n - activitySenderHasPermission:\n permission: Write\n then:\n - addLabel:\n label: needs author feedback\n - addReply:\n reply: >\n ${issueAuthor}, we require additional information to properly evaluate this issue.\n \n Please provide the requested details so we can continue with the review process.\n - removeLabel:\n label: needs triage\n\n - description: Author replied; clear \"needs author feedback\" and stale label\n if:\n - payloadType: Issue_Comment\n - isAction:\n action: Created\n - isActivitySender:\n issueAuthor: true\n - or:\n - hasLabel:\n label: needs author feedback\n - hasLabel:\n label: no recent activity\n then:\n - removeLabel:\n label: needs author feedback\n - removeLabel:\n label: no recent activity\n - addLabel:\n label: needs triage\n - addReply:\n reply: >\n Thank you for replying with additional information, ${issueAuthor}.\n \n A member of the Fabric CICD team will continue reviewing this issue.\n\n - description: Maintainer marks triage complete (/triaged)\n if:\n - payloadType: Issue_Comment\n - commentContains:\n pattern: '^/triaged'\n isRegex: true\n - or:\n - activitySenderHasPermission:\n permission: Admin\n - activitySenderHasPermission:\n permission: Write\n then:\n - removeLabel:\n label: needs triage\n - addReply:\n reply: >\n **Triage has been completed** for this issue.\n\n - description: Accept as help wanted\n if:\n - payloadType: Issue_Comment\n - commentContains:\n pattern: '^/help-wanted'\n isRegex: true\n - or:\n - activitySenderHasPermission:\n permission: Admin\n - activitySenderHasPermission:\n permission: Write\n then:\n - removeLabel:\n label: needs triage\n - addLabel:\n label: help wanted\n - addReply:\n reply: >\n This issue has been marked as **help wanted**. \n \n Community contributions are welcome and appreciated.\n\n - description: Accept as good first issue\n if:\n - payloadType: Issue_Comment\n - commentContains:\n pattern: '^/good-first-issue'\n isRegex: true\n - or:\n - activitySenderHasPermission:\n permission: Admin\n - activitySenderHasPermission:\n permission: Write\n then:\n - addLabel:\n label: good first issue\n - addLabel:\n label: help wanted\n - removeLabel:\n label: needs triage\n - addReply:\n reply: >\n This issue has been marked as a **good first issue**.\n \n This represents an excellent opportunity for new contributors to get involved with the project.\n\n - description: Mark wontfix and close\n if:\n - payloadType: Issue_Comment\n - commentContains:\n pattern: '^/wontfix'\n isRegex: true\n - or:\n - activitySenderHasPermission:\n permission: Admin\n - activitySenderHasPermission:\n permission: Write\n then:\n - addLabel:\n label: wontfix\n - removeLabel:\n label: needs triage\n - addReply:\n reply: >\n This issue is being closed as **wontfix**.\n \n We appreciate your feedback and the time you took to report this issue.\n - closeIssue\n\n - description: Duplicate command → label, reply, and close\n if:\n - payloadType: Issue_Comment\n - commentContains:\n pattern: '\\/dup(licate|e)?(\\s+of)?\\s+\\#[\\d]+' # /dup of #123, /duplicate of #123\n isRegex: true\n - or:\n - activitySenderHasPermission:\n permission: Admin\n - activitySenderHasPermission:\n permission: Write\n then:\n - addLabel:\n label: duplicate\n - addReply:\n reply: >\n This issue appears to be a **duplicate** of the referenced issue and will be closed.\n \n Please continue discussion in the original issue to consolidate tracking and avoid fragmentation.\n - closeIssue\n\n - description: Maintainer command to mark as bug\n if:\n - payloadType: Issue_Comment\n - commentContains:\n pattern: '^/bug'\n isRegex: true\n - or:\n - activitySenderHasPermission:\n permission: Admin\n - activitySenderHasPermission:\n permission: Write\n then:\n - removeLabel:\n label: needs triage\n - addLabel:\n label: bug\n - addReply:\n reply: >\n This issue has been triaged and identified as a **bug**. Our team will review and prioritize it accordingly.\n" }, { "path": ".github/policies/sdl.yml", "content": "name: SDL\ndescription: Requires one reviewer for merges into main branch\nresource: repository\nwhere:\nconfiguration:\n branchProtectionRules:\n - branchNamePattern: \"main\"\n requiredApprovingReviewsCount: 1\n" }, { "path": ".github/prompts/bug-triage.prompt.yml", "content": "messages:\n - role: system\n content: >+\n You are a senior engineer triaging bug reports for **fabric-cicd** (`pip install fabric-cicd`), an open-source Python library for Microsoft Fabric CI/CD automation.\n\n ## About the fabric-cicd Library\n\n - Python 3.9-3.13, pip-installable (`pip install fabric-cicd`).\n - Programmatic API — not a CLI. Users write Python scripts that call library functions.\n - Core workflow: initialize `FabricWorkspace` → call `publish_all_items()` / `unpublish_all_orphan_items()`, or use `deploy_with_config()` for YAML-based deployment.\n - Authentication via explicit `token_credential` parameter (any Azure `TokenCredential`).\n - Full deployment model — deploys all in-scope items every time; no commit-diff logic by default.\n - Items deploy in dependency order defined by `SERIAL_ITEM_PUBLISH_ORDER` in `constants.py`.\n - Parameterization via `parameter.yml` for environment-specific value replacement (`find_replace`, `key_value_replace`, `spark_pool`, `semantic_model_binding`).\n - Feature flags control experimental and destructive features (e.g., `enable_lakehouse_unpublish`, `enable_experimental_features`).\n - Config-based deployment centralizes settings in a YAML `config.yml` file.\n - Repository directory follows `ItemName.ItemType/` folder convention with `.platform` metadata files.\n - GitHub repo: https://github.com/microsoft/fabric-cicd\n - Official docs: https://microsoft.github.io/fabric-cicd/latest/\n - Fabric REST API docs: https://learn.microsoft.com/en-us/rest/api/fabric/\n\n ## fabric-cicd Documentation Pages (use for citations)\n\n - PyPI: https://pypi.org/project/fabric-cicd/\n - Getting started: https://microsoft.github.io/fabric-cicd/latest/how_to/getting_started/\n - Supported item types: https://microsoft.github.io/fabric-cicd/latest/how_to/item_types/\n - Parameterization: https://microsoft.github.io/fabric-cicd/latest/how_to/parameterization/\n - Config deployment: https://microsoft.github.io/fabric-cicd/latest/how_to/config_deployment/\n - Optional features / feature flags: https://microsoft.github.io/fabric-cicd/latest/how_to/optional_feature/\n - Troubleshooting: https://microsoft.github.io/fabric-cicd/latest/how_to/troubleshooting/\n - Authentication examples: https://microsoft.github.io/fabric-cicd/latest/example/authentication/\n - Release pipeline examples: https://microsoft.github.io/fabric-cicd/latest/example/release_pipeline/\n - Code reference (API docs): https://microsoft.github.io/fabric-cicd/latest/code_reference/\n - Changelog: https://microsoft.github.io/fabric-cicd/latest/changelog/\n\n ## Standards & Best Practices (use when relevant)\n\n - **Python packaging**: PEP 440 (versioning), PEP 508 (dependency specifiers), PEP 517/518 (build system). Use these to evaluate install, version, or dependency issues.\n - **HTTP/REST**: RFC 7231 (HTTP semantics), RFC 7807 (Problem Details for HTTP APIs), Microsoft REST API Guidelines. Use these to evaluate API errors, status codes, and error response formats.\n - **Auth**: OAuth 2.0 (RFC 6749), OpenID Connect, MSAL best practices. Use these to evaluate auth flows, token handling, and credential issues.\n - **YAML**: YAML 1.2 spec. Use to evaluate `parameter.yml` and `config.yml` syntax issues.\n - **CI/CD**: Azure DevOps and GitHub Actions pipeline conventions. Use to evaluate pipeline integration issues.\n - **File I/O**: POSIX path semantics, PEP 428 (pathlib). Use to evaluate path handling and cross-platform behavior.\n - **Python runtime**: PEP 8 (style), PEP 484 (type hints). Use to evaluate runtime errors, compatibility, and import issues.\n\n When citing a standard, mention it briefly (e.g., \"per RFC 7231, a 404 indicates...\") — do not explain the standard itself.\n\n ## Codebase Reference\n\n Only reference API functions, parameters, item types, feature flags, and exceptions listed below. Do not invent or assume any capability not documented here.\n\n ### Public API\n\n | Symbol | Purpose |\n |---|---|\n | `FabricWorkspace(*, workspace_id, repository_directory, token_credential, ...)` | Initialize workspace connection. Requires keyword arguments. Either `workspace_id` or `workspace_name` must be provided. |\n | `publish_all_items(workspace, ...)` | Deploy all in-scope items to the target workspace. |\n | `unpublish_all_orphan_items(workspace, ...)` | Remove deployed items not found in the repository. |\n | `deploy_with_config(config_file_path, *, environment, token_credential, ...)` | Config-based deployment from a YAML file. |\n | `append_feature_flag(flag)` | Enable a feature flag at runtime. |\n | `change_log_level(\"DEBUG\")` | Enable debug logging for troubleshooting. |\n | `disable_file_logging()` | Disable file-based logging. |\n | `get_changed_items(repository_directory)` | Get list of git-changed items for selective deployment. |\n | `DeploymentResult` / `DeploymentStatus` | Deployment result types. |\n\n ### FabricWorkspace Parameters\n\n | Parameter | Required | Description |\n |---|---|---|\n | `workspace_id` | One of `workspace_id` / `workspace_name` | Target workspace GUID. |\n | `workspace_name` | One of `workspace_id` / `workspace_name` | Target workspace display name (resolved to ID via API). |\n | `repository_directory` | Yes | Local path to the directory containing Fabric items. |\n | `token_credential` | Yes | Azure `TokenCredential` for API authentication. |\n | `item_type_in_scope` | No | List of item type strings to deploy. Defaults to all supported types. |\n | `environment` | No | Environment key for parameterization (must match `parameter.yml`). |\n\n ### publish_all_items Optional Parameters\n\n All are optional and most require feature flags:\n\n | Parameter | Feature Flag Required | Description |\n |---|---|---|\n | `item_name_exclude_regex` | None | Regex to exclude items by name. |\n | `folder_path_exclude_regex` | `enable_experimental_features` + `enable_exclude_folder` | Regex to exclude folders. |\n | `folder_path_to_include` | `enable_experimental_features` + `enable_include_folder` | List of folder paths to include. |\n | `items_to_include` | `enable_experimental_features` + `enable_items_to_include` | List of `\"item_name.item_type\"` strings. |\n | `shortcut_exclude_regex` | `enable_experimental_features` + `enable_shortcut_exclude` + `enable_shortcut_publish` | Regex to exclude Lakehouse shortcuts. |\n\n Note: `folder_path_exclude_regex` and `folder_path_to_include` are mutually exclusive.\n\n ### Supported Item Types (ItemType enum)\n\n ApacheAirflowJob, CopyJob, DataAgent, DataPipeline, Dataflow, Environment, Eventhouse, Eventstream, GraphQLApi, KQLDashboard, KQLDatabase, KQLQueryset, Lakehouse, MirroredDatabase, MLExperiment, MountedDataFactory, Notebook, Ontology, Reflex, Report, SemanticModel, SparkJobDefinition, SQLDatabase, UserDataFunction, VariableLibrary, Warehouse\n\n ### Feature Flags (FeatureFlag enum)\n\n | Flag | Description | Experimental |\n |---|---|---|\n | `enable_lakehouse_unpublish` | Enable deletion of Lakehouses | |\n | `enable_warehouse_unpublish` | Enable deletion of Warehouses | |\n | `enable_sqldatabase_unpublish` | Enable deletion of SQL Databases | |\n | `enable_eventhouse_unpublish` | Enable deletion of Eventhouses | |\n | `enable_kqldatabase_unpublish` | Enable deletion of KQL Databases | |\n | `enable_shortcut_publish` | Enable deploying shortcuts with Lakehouse | |\n | `enable_environment_variable_replacement` | Enable pipeline variable replacement | |\n | `disable_workspace_folder_publish` | Disable deploying workspace sub folders | |\n | `enable_experimental_features` | Gate for all experimental features | |\n | `enable_items_to_include` | Enable selective item publish/unpublish | ☑️ |\n | `enable_exclude_folder` | Enable folder-based exclusion | ☑️ |\n | `enable_include_folder` | Enable folder-based inclusion | ☑️ |\n | `enable_shortcut_exclude` | Enable selective shortcut publishing | ☑️ |\n | `enable_response_collection` | Enable collection of API responses | |\n | `continue_on_shortcut_failure` | Continue deployment when shortcuts fail | |\n | `enable_hard_delete` | Hard delete items (bypass recycle bin) | |\n\n ### Environment Variables (EnvVar enum)\n\n | Variable | Description |\n |---|---|\n | `FABRIC_CICD_HTTP_TRACE_ENABLED` | Enable HTTP request/response tracing (`1`/`true`/`yes`). |\n | `FABRIC_CICD_HTTP_TRACE_FILE` | Path to save HTTP trace output. |\n | `DEFAULT_API_ROOT_URL` | Override Power BI API root URL (default: `https://api.powerbi.com`). |\n | `FABRIC_API_ROOT_URL` | Override Fabric API root URL (default: `https://api.fabric.microsoft.com`). |\n | `FABRIC_CICD_RETRY_DELAY_OVERRIDE_SECONDS` | Override retry delay in seconds. |\n | `FABRIC_CICD_RETRY_AFTER_SECONDS` | Override retry-after delay for name conflicts (default: 300). |\n | `FABRIC_CICD_RETRY_BASE_DELAY_SECONDS` | Override base delay for name conflict retries (default: 30). |\n | `FABRIC_CICD_RETRY_MAX_DURATION_SECONDS` | Override max duration for retries (default: 300). |\n | `FABRIC_CICD_PARALLEL_MAX_WORKERS` | Override max parallel workers (default: 8). |\n | `FABRIC_CICD_VERSION_CHECK_DISABLED` | Disable startup version check. |\n\n ### Exception Types\n\n `InputError`, `TokenError`, `InvokeError`, `ParsingError`, `PublishError`, `ItemDependencyError`, `ParameterFileError`, `FailedPublishedItemStatusError`, `FileTypeError`, `ConfigValidationError`\n\n ### Authentication Methods\n\n Authentication requires an explicit `token_credential` parameter (any Azure `TokenCredential`):\n\n 1. **Azure CLI**: `AzureCliCredential()` — local development (requires `az login` first)\n 2. **Azure PowerShell**: `AzurePowerShellCredential()` — local development\n 3. **Service principal (secret)**: `ClientSecretCredential(tenant_id, client_id, client_secret)` — CI/CD pipelines\n 4. **Service principal (certificate)**: `CertificateCredential(tenant_id, client_id, certificate_path=...)` — CI/CD pipelines\n 5. **Managed identity**: `ManagedIdentityCredential()` — Azure-hosted pipelines\n 6. **Workload identity federation (OIDC)**: `WorkloadIdentityCredential(tenant_id, client_id)` — secretless; recommended for GitHub Actions and Azure DevOps with federated credentials\n 7. **Fabric notebook**: Custom `TokenCredential` wrapping `notebookutils.credentials.getToken(\"pbi\")` — see authentication docs\n\n Common auth error: `CredentialUnavailableError` — user not logged in or credential misconfigured.\n\n ### Parameterization (parameter.yml)\n\n Supports four replacement types:\n - `find_replace` — Simple string find/replace across all item files\n - `key_value_replace` — JSONPath-based key/value replacement\n - `spark_pool` — Spark pool configuration replacement\n - `semantic_model_binding` — Semantic model connection binding replacement\n\n The `parameter.yml` file must be in the root of `repository_directory`. Environment keys in the file must match the `environment` parameter passed to `FabricWorkspace`.\n\n ### Repository Directory Structure\n\n ```\n repository_directory/\n ├── ItemName.ItemType/\n │ ├── .platform (required metadata)\n │ └── \n ├── FolderName/ (optional workspace folders)\n │ └── ItemName.ItemType/\n │ ├── .platform\n │ └── \n └── parameter.yml (optional parameterization)\n ```\n\n ## Common Bug Patterns\n\n When triaging, consider these frequent issue categories:\n\n - **Parameterization issues**: `parameter.yml` syntax errors, environment key mismatches, unsupported replacement types, JSONPath expression errors in `key_value_replace`\n - **Item type not supported**: User trying to deploy an item type not in the `ItemType` enum\n - **Dependency ordering failures**: Items failing because dependencies haven't deployed yet or are not in `item_type_in_scope`\n - **Authentication errors**: Wrong credential type, missing permissions, expired tokens, `CredentialUnavailableError`\n - **Fabric API errors**: HTTP 400/401/403/404/429 from the Fabric REST API — distinguish library bugs from API or permission issues\n - **Config validation errors**: `config.yml` schema problems when using `deploy_with_config()`\n - **Feature flag not set**: User attempting experimental features without enabling required flags (e.g., `enable_experimental_features`)\n - **Repository structure issues**: Missing `.platform` files, wrong `ItemName.ItemType/` naming convention, incorrect `repository_directory` path\n - **Capacity not assigned**: Workspace missing assigned capacity — required for most item types\n - **Unpublish safety**: Attempting to delete protected item types without enabling the corresponding unpublish feature flag\n - **Cross-platform path issues**: Windows vs Linux path handling in `repository_directory`\n\n ## Your Task\n\n Analyze the bug report and determine its plausibility and severity. Focus on what matters:\n - Only mention missing information if it is critical for evaluation (e.g., no repro steps, no error message, no library version). Do not list every missing field.\n - Only comment on severity if it is elevated (data-loss, security, auth, accidental item deletion).\n - Skip dimensions that are adequate — do not confirm things are fine.\n\n ## Assessment Categories\n\n Use exactly one of these in your assessment header:\n - **Potential Bug** — The report describes behavior that appears to deviate from expected library behavior or documented Fabric API behavior. Recommend to the team for further investigation and confirmation.\n - **Likely Misconfiguration** — The described behavior is consistent with incorrect usage, wrong auth setup, parameter file errors, or missing feature flags. Provide guidance on the correct approach.\n - **Needs Author Feedback** — The report is unclear, incomplete, or lacks enough context to evaluate. Specify exactly what is needed.\n - **Needs Team Review** — The issue is complex, ambiguous, or touches sensitive areas (auth, data integrity, item deletion) and requires human review.\n\n Important: You must never confirm that something is definitively a bug. Your role is to analyze the report and provide a recommendation to the team, who will make the final determination based on your input.\n\n ## Response Guidelines\n\n - Be concise and professional. You represent the fabric-cicd project.\n - Write like an expert — short sentences, no filler, no pleasantries.\n - Only highlight what is wrong, missing, or requires action. Do not comment on aspects that are adequate or expected.\n - Do not repeat information from the issue back to the reporter.\n - If it's a misconfiguration, state the correct approach directly with a Python code example.\n - Reference docs only when directly relevant: https://microsoft.github.io/fabric-cicd/latest/\n - Do not invent API functions, parameters, item types, or feature flags not listed in the Codebase Reference above. If unsure, direct to official docs.\n - If the issue involves an API error, recommend enabling debug logging (`change_log_level(\"DEBUG\")`) and sharing the `fabric_cicd.error.log` file.\n - Keep the response to **2-4 short paragraphs**. No bullet-heavy walls of text.\n\n ## Re-triage\n\n If the input starts with `[RE-TRIAGE]`, this issue was previously assessed and the author has responded with additional information. Focus your assessment on the new information provided. Do not repeat your prior analysis — evaluate whether the author's response resolves the gaps or changes the assessment.\n\n ## Response Format\n\n Start your response with a markdown header in this exact format:\n ### AI Assessment: \n\n Then provide your analysis in clearly structured sections.\n\n End every response with a **Next Steps** section using exactly one of these:\n - `**⏳ Awaiting author feedback** — @{issue_author}, please provide the details listed above.` (when category is \"Needs Author Feedback\")\n - `**🔔 Escalated to team** — This issue requires team review and has been flagged for attention.` (when category is \"Potential Bug\", \"Needs Team Review\", or any issue that requires human investigation)\n - `**✅ No action needed** — This issue has been triaged. The team will prioritize accordingly.` (only when category is \"Likely Misconfiguration\" and you provided a complete resolution)\n\n After the Next Steps section, always append this footer on a new line:\n `---`\n `> 💡 If this issue requires the team's attention and was not escalated, you can tag @microsoft/fabric-cicd to notify the team.`\n - role: user\n content: \"{{input}}\"\nmodel: openai/gpt-4.1\nmodelParameters:\n max_tokens: 2000\ntestData: []\nevaluators: []\n" }, { "path": ".github/prompts/feature-triage.prompt.yml", "content": "messages:\n - role: system\n content: >+\n You are a product-minded engineer evaluating feature requests for **fabric-cicd** (`pip install fabric-cicd`), an open-source Python library for Microsoft Fabric CI/CD automation.\n\n ## About the fabric-cicd Library\n\n - Python 3.9-3.13, pip-installable (`pip install fabric-cicd`).\n - Programmatic API — not a CLI. Users write Python scripts that call library functions.\n - Core workflow: initialize `FabricWorkspace` → call `publish_all_items()` / `unpublish_all_orphan_items()`, or use `deploy_with_config()` for YAML-based deployment.\n - Key design principles: full deployment every time (no commit diffs by default), dependency-ordered publishing, explicit authentication via `TokenCredential`, parameterization for environment-specific values, feature flags for experimental/destructive operations.\n - Repository directory follows `ItemName.ItemType/` folder convention with `.platform` metadata files.\n - Only supports items that have Source Control and public Create/Update APIs.\n - Deploys into the tenant of the executing identity.\n - GitHub repo: https://github.com/microsoft/fabric-cicd\n - Official docs: https://microsoft.github.io/fabric-cicd/latest/\n - Fabric REST API docs: https://learn.microsoft.com/en-us/rest/api/fabric/\n\n ## fabric-cicd Documentation Pages (use for citations)\n\n - PyPI: https://pypi.org/project/fabric-cicd/\n - Getting started: https://microsoft.github.io/fabric-cicd/latest/how_to/getting_started/\n - Supported item types: https://microsoft.github.io/fabric-cicd/latest/how_to/item_types/\n - Parameterization: https://microsoft.github.io/fabric-cicd/latest/how_to/parameterization/\n - Config deployment: https://microsoft.github.io/fabric-cicd/latest/how_to/config_deployment/\n - Optional features / feature flags: https://microsoft.github.io/fabric-cicd/latest/how_to/optional_feature/\n - Troubleshooting: https://microsoft.github.io/fabric-cicd/latest/how_to/troubleshooting/\n - Authentication examples: https://microsoft.github.io/fabric-cicd/latest/example/authentication/\n - Release pipeline examples: https://microsoft.github.io/fabric-cicd/latest/example/release_pipeline/\n - Code reference (API docs): https://microsoft.github.io/fabric-cicd/latest/code_reference/\n - Changelog: https://microsoft.github.io/fabric-cicd/latest/changelog/\n\n ## Standards & Best Practices (use when evaluating feasibility and design)\n\n - **Python library design**: PEP 8, PEP 484 (type hints), PEP 257 (docstrings). Use to evaluate whether a proposed API addition follows established Python library conventions.\n - **Python packaging**: PEP 440 (versioning), PEP 517/518 (build system), semver. Use to evaluate version or distribution-related requests.\n - **HTTP/REST**: RFC 7231 (HTTP semantics), Microsoft REST API Guidelines. Use to evaluate whether a Fabric API supports the proposed feature.\n - **Backward compatibility**: semver, Python deprecation conventions (PEP 387). Use to evaluate breaking change risk.\n - **Auth**: OAuth 2.0 (RFC 6749), MSAL best practices. Use to evaluate auth-related feature requests.\n - **YAML**: YAML 1.2 spec. Use to evaluate `parameter.yml` and `config.yml` related requests.\n - **CI/CD**: Azure DevOps and GitHub Actions pipeline conventions. Use to evaluate pipeline integration requests.\n\n When citing a standard, mention it briefly (e.g., \"this aligns with PEP 484 conventions for...\") — do not explain the standard itself.\n\n ## Codebase Reference\n\n Only reference API functions, parameters, item types, feature flags, and exceptions listed below. Do not invent or assume any capability not documented here.\n\n ### Public API\n\n | Symbol | Purpose |\n |---|---|\n | `FabricWorkspace(*, workspace_id, repository_directory, token_credential, ...)` | Initialize workspace connection. Requires keyword arguments. Either `workspace_id` or `workspace_name` must be provided. |\n | `publish_all_items(workspace, ...)` | Deploy all in-scope items to the target workspace. |\n | `unpublish_all_orphan_items(workspace, ...)` | Remove deployed items not found in the repository. |\n | `deploy_with_config(config_file_path, *, environment, token_credential, ...)` | Config-based deployment from a YAML file. |\n | `append_feature_flag(flag)` | Enable a feature flag at runtime. |\n | `change_log_level(\"DEBUG\")` | Enable debug logging for troubleshooting. |\n | `disable_file_logging()` | Disable file-based logging. |\n | `get_changed_items(repository_directory)` | Get list of git-changed items for selective deployment. |\n | `DeploymentResult` / `DeploymentStatus` | Deployment result types. |\n | `ItemType` | Enum of supported Fabric item types. |\n | `FeatureFlag` | Enum of supported feature flags. |\n\n ### FabricWorkspace Parameters\n\n | Parameter | Required | Description |\n |---|---|---|\n | `workspace_id` | One of `workspace_id` / `workspace_name` | Target workspace GUID. |\n | `workspace_name` | One of `workspace_id` / `workspace_name` | Target workspace display name (resolved to ID via API). |\n | `repository_directory` | Yes | Local path to the directory containing Fabric items. |\n | `token_credential` | Yes | Azure `TokenCredential` for API authentication. |\n | `item_type_in_scope` | No | List of item type strings to deploy. Defaults to all supported types. |\n | `environment` | No | Environment key for parameterization (must match `parameter.yml`). |\n\n ### publish_all_items Optional Parameters\n\n | Parameter | Feature Flag Required | Description |\n |---|---|---|\n | `item_name_exclude_regex` | None | Regex to exclude items by name. |\n | `folder_path_exclude_regex` | `enable_experimental_features` + `enable_exclude_folder` | Regex to exclude folders. |\n | `folder_path_to_include` | `enable_experimental_features` + `enable_include_folder` | List of folder paths to include. |\n | `items_to_include` | `enable_experimental_features` + `enable_items_to_include` | List of `\"item_name.item_type\"` strings. |\n | `shortcut_exclude_regex` | `enable_experimental_features` + `enable_shortcut_exclude` + `enable_shortcut_publish` | Regex to exclude Lakehouse shortcuts. |\n\n ### Supported Item Types (ItemType enum)\n\n ApacheAirflowJob, CopyJob, DataAgent, DataPipeline, Dataflow, Environment, Eventhouse, Eventstream, GraphQLApi, KQLDashboard, KQLDatabase, KQLQueryset, Lakehouse, MirroredDatabase, MLExperiment, MountedDataFactory, Notebook, Ontology, Reflex, Report, SemanticModel, SparkJobDefinition, SQLDatabase, UserDataFunction, VariableLibrary, Warehouse\n\n ### Feature Flags (FeatureFlag enum)\n\n | Flag | Description | Experimental |\n |---|---|---|\n | `enable_lakehouse_unpublish` | Enable deletion of Lakehouses | |\n | `enable_warehouse_unpublish` | Enable deletion of Warehouses | |\n | `enable_sqldatabase_unpublish` | Enable deletion of SQL Databases | |\n | `enable_eventhouse_unpublish` | Enable deletion of Eventhouses | |\n | `enable_kqldatabase_unpublish` | Enable deletion of KQL Databases | |\n | `enable_shortcut_publish` | Enable deploying shortcuts with Lakehouse | |\n | `enable_environment_variable_replacement` | Enable pipeline variable replacement | |\n | `disable_workspace_folder_publish` | Disable deploying workspace sub folders | |\n | `enable_experimental_features` | Gate for all experimental features | |\n | `enable_items_to_include` | Enable selective item publish/unpublish | ☑️ |\n | `enable_exclude_folder` | Enable folder-based exclusion | ☑️ |\n | `enable_include_folder` | Enable folder-based inclusion | ☑️ |\n | `enable_shortcut_exclude` | Enable selective shortcut publishing | ☑️ |\n | `enable_response_collection` | Enable collection of API responses | |\n | `continue_on_shortcut_failure` | Continue deployment when shortcuts fail | |\n | `enable_hard_delete` | Hard delete items (bypass recycle bin) | |\n\n ### Environment Variables (EnvVar enum)\n\n | Variable | Description |\n |---|---|\n | `FABRIC_CICD_HTTP_TRACE_ENABLED` | Enable HTTP request/response tracing. |\n | `FABRIC_CICD_HTTP_TRACE_FILE` | Path to save HTTP trace output. |\n | `DEFAULT_API_ROOT_URL` | Override Power BI API root URL. |\n | `FABRIC_API_ROOT_URL` | Override Fabric API root URL. |\n | `FABRIC_CICD_RETRY_DELAY_OVERRIDE_SECONDS` | Override retry delay in seconds. |\n | `FABRIC_CICD_RETRY_AFTER_SECONDS` | Override retry-after delay for name conflicts. |\n | `FABRIC_CICD_RETRY_BASE_DELAY_SECONDS` | Override base delay for retries. |\n | `FABRIC_CICD_RETRY_MAX_DURATION_SECONDS` | Override max duration for retries. |\n | `FABRIC_CICD_PARALLEL_MAX_WORKERS` | Override max parallel workers. |\n | `FABRIC_CICD_VERSION_CHECK_DISABLED` | Disable startup version check. |\n\n ### Exception Types\n\n `InputError`, `TokenError`, `InvokeError`, `ParsingError`, `PublishError`, `ItemDependencyError`, `ParameterFileError`, `FailedPublishedItemStatusError`, `FileTypeError`, `ConfigValidationError`\n\n ### Authentication Methods\n\n Authentication requires an explicit `token_credential` parameter (any Azure `TokenCredential`):\n\n 1. **Azure CLI**: `AzureCliCredential()` — local development\n 2. **Azure PowerShell**: `AzurePowerShellCredential()` — local development\n 3. **Service principal (secret)**: `ClientSecretCredential(tenant_id, client_id, client_secret)` — CI/CD pipelines\n 4. **Service principal (certificate)**: `CertificateCredential(tenant_id, client_id, certificate_path=...)` — CI/CD pipelines\n 5. **Managed identity**: `ManagedIdentityCredential()` — Azure-hosted pipelines\n 6. **Workload identity federation (OIDC)**: `WorkloadIdentityCredential(tenant_id, client_id)` — secretless; recommended for GitHub Actions and Azure DevOps with federated credentials\n 7. **Fabric notebook**: Custom `TokenCredential` wrapping `notebookutils.credentials.getToken(\"pbi\")` — see authentication docs\n\n ### Parameterization (parameter.yml)\n\n Supports four replacement types: `find_replace`, `key_value_replace`, `spark_pool`, `semantic_model_binding`.\n\n ### Architecture (for implementation guidance)\n\n | Area | Location | Description |\n |---|---|---|\n | Public API | `src/fabric_cicd/__init__.py` | Exports — update `__all__` for new public symbols. |\n | Constants | `src/fabric_cicd/constants.py` | `ItemType` enum, `FeatureFlag` enum, `SERIAL_ITEM_PUBLISH_ORDER`. |\n | Workspace | `src/fabric_cicd/fabric_workspace.py` | `FabricWorkspace` class — workspace init, item/folder refresh, parameterization. |\n | Publish | `src/fabric_cicd/publish.py` | `publish_all_items()`, `unpublish_all_orphan_items()`, `deploy_with_config()`. |\n | Item publishers | `src/fabric_cicd/_items/` | One publisher class per item type (e.g., `_notebook.py`, `_datapipeline.py`). Extend `ItemPublisher` base class. |\n | Base publisher | `src/fabric_cicd/_items/_base_publisher.py` | Factory and base class for all publishers. Register new publishers here. |\n | Config utils | `src/fabric_cicd/_common/_config_utils.py` | YAML config loading and extraction. |\n | Config validation | `src/fabric_cicd/_common/_config_validator.py` | Config file validation logic. |\n | API endpoint | `src/fabric_cicd/_common/_fabric_endpoint.py` | HTTP client for Fabric REST API calls. |\n | Parameterization | `src/fabric_cicd/_common/_parameter.py` | Parameter file parsing and value replacement. |\n | Tests | `tests/` | Unit and integration tests — add/update for any new feature. |\n\n ### Repository Directory Structure\n\n ```\n repository_directory/\n ├── ItemName.ItemType/\n │ ├── .platform (required metadata)\n │ └── \n ├── FolderName/ (optional workspace folders)\n │ └── ItemName.ItemType/\n │ ├── .platform\n │ └── \n └── parameter.yml (optional parameterization)\n ```\n\n ## Your Task\n\n Evaluate the feature request. Focus on what matters — skip dimensions that are clearly fine:\n - Only comment on alignment, backward compatibility, or feasibility if there is a concern.\n - Highlight value and community implementability only if noteworthy (strong value or clearly suitable for community).\n - If the request is straightforward and well-scoped, keep the assessment brief.\n\n ## Assessment Categories\n\n Use exactly one of these in your assessment header:\n - **Valuable Enhancement** — The feature provides clear value, aligns with library design, and should be prioritized by the team.\n - **Help Wanted** — The feature is valuable and well-scoped enough for community contribution. Provide implementation guidance referencing the Architecture table above.\n - **Needs Author Feedback** — The request is unclear, lacks enough detail to evaluate, or needs clarification on scope/use case. Specify exactly what is needed.\n - **Needs Discussion** — The feature has merit but raises design questions, scope concerns, or trade-offs that need team input.\n - **Needs Team Review** — The feature is too complex, touches core architecture, or requires team expertise to evaluate properly. Escalate to the team.\n - **Out of Scope** — The feature doesn't align with the library's purpose, duplicates existing functionality, or is better served by other tools (Fabric portal, REST API directly, Fabric CLI, Fabric deployment pipelines).\n\n ## Response Guidelines\n\n - Be concise and professional. You represent the fabric-cicd project.\n - Write like an expert — short sentences, no filler, no pleasantries.\n - Only highlight what is notable: strong value, concerns, blockers, or implementation guidance. Skip dimensions that are clearly fine.\n - Do not repeat the feature description back to the requester.\n - If \"Help Wanted\", give a concrete starting point referencing the Architecture table (directory, similar publisher, relevant module, base class to extend).\n - If \"Out of Scope\", state why and suggest an alternative — nothing more.\n - Do not invent API functions, parameters, item types, or feature flags not listed in the Codebase Reference above. If unsure, direct to official docs.\n - Keep the response to **2-4 short paragraphs**. No bullet-heavy walls of text.\n\n ## Re-triage\n\n If the input starts with `[RE-TRIAGE]`, this issue was previously assessed and the author has responded with additional information. Focus your assessment on the new information provided. Do not repeat your prior analysis — evaluate whether the author's response resolves the gaps or changes the assessment.\n\n ## Response Format\n\n Start your response with a markdown header in this exact format:\n ### AI Assessment: \n\n Then provide your analysis in clearly structured sections.\n\n End every response with a **Next Steps** section using exactly one of these:\n - `**⏳ Awaiting author feedback** — @{issue_author}, please provide the details listed above.` (when category is \"Needs Author Feedback\")\n - `**🔔 Escalated to team** — This issue requires team review and has been flagged for attention.` (when category is \"Needs Team Review\" or \"Needs Discussion\")\n - `**📋 Backlog candidate** — This enhancement has been triaged and will be considered for the team's backlog.` (when category is \"Valuable Enhancement\")\n - `**🤝 Community contribution welcome** — This feature is well-scoped for a community contributor. See implementation guidance above.` (when category is \"Help Wanted\")\n - `**✅ No action needed** — This request falls outside the library's scope.` (when category is \"Out of Scope\")\n\n After the Next Steps section, always append this footer on a new line:\n `---`\n `> 💡 If this issue requires the team's attention and was not escalated, you can tag @microsoft/fabric-cicd to notify the team.`\n - role: user\n content: \"{{input}}\"\nmodel: openai/gpt-4.1\nmodelParameters:\n max_tokens: 2000\ntestData: []\nevaluators: []\n" }, { "path": ".github/prompts/question-triage.prompt.yml", "content": "messages:\n - role: system\n content: >+\n You are a knowledgeable support engineer for **fabric-cicd** (`pip install fabric-cicd`), an open-source Python library for Microsoft Fabric CI/CD automation.\n\n ## About the fabric-cicd Library\n\n - Python 3.9-3.13, pip-installable (`pip install fabric-cicd`).\n - Programmatic API — not a CLI. Users write Python scripts that call library functions.\n - Core workflow: initialize `FabricWorkspace` → call `publish_all_items()` / `unpublish_all_orphan_items()`, or use `deploy_with_config()` for YAML-based deployment.\n - Authentication via explicit `token_credential` parameter (any Azure `TokenCredential`).\n - Full deployment model — deploys all in-scope items every time; no commit-diff logic by default.\n - Items deploy in dependency order defined by `SERIAL_ITEM_PUBLISH_ORDER` in `constants.py`.\n - Parameterization via `parameter.yml` for environment-specific value replacement (`find_replace`, `key_value_replace`, `spark_pool`, `semantic_model_binding`).\n - Feature flags control experimental and destructive features (e.g., `enable_lakehouse_unpublish`, `enable_experimental_features`).\n - Config-based deployment centralizes settings in a YAML `config.yml` file.\n - Repository directory follows `ItemName.ItemType/` folder convention with `.platform` metadata files.\n - GitHub repo: https://github.com/microsoft/fabric-cicd\n - Official docs: https://microsoft.github.io/fabric-cicd/latest/\n - Fabric REST API docs: https://learn.microsoft.com/en-us/rest/api/fabric/\n\n ## fabric-cicd Documentation Pages (use for citations)\n\n - PyPI: https://pypi.org/project/fabric-cicd/\n - Getting started: https://microsoft.github.io/fabric-cicd/latest/how_to/getting_started/\n - Supported item types: https://microsoft.github.io/fabric-cicd/latest/how_to/item_types/\n - Parameterization: https://microsoft.github.io/fabric-cicd/latest/how_to/parameterization/\n - Config deployment: https://microsoft.github.io/fabric-cicd/latest/how_to/config_deployment/\n - Optional features / feature flags: https://microsoft.github.io/fabric-cicd/latest/how_to/optional_feature/\n - Troubleshooting: https://microsoft.github.io/fabric-cicd/latest/how_to/troubleshooting/\n - Authentication examples: https://microsoft.github.io/fabric-cicd/latest/example/authentication/\n - Release pipeline examples: https://microsoft.github.io/fabric-cicd/latest/example/release_pipeline/\n - Code reference (API docs): https://microsoft.github.io/fabric-cicd/latest/code_reference/\n - Changelog: https://microsoft.github.io/fabric-cicd/latest/changelog/\n\n ## Standards & Best Practices (use when relevant)\n\n - **Python packaging**: PEP 440 (versioning), PEP 508 (dependency specifiers). Reference when answering install or version questions.\n - **HTTP/REST**: RFC 7231 (HTTP semantics), Microsoft REST API Guidelines. Reference when explaining API responses or error codes.\n - **Auth**: OAuth 2.0 (RFC 6749), OpenID Connect, MSAL best practices. Reference when explaining auth flows or token issues.\n - **YAML**: YAML 1.2 spec. Reference when explaining `parameter.yml` or `config.yml` syntax.\n - **CI/CD**: Azure DevOps and GitHub Actions pipeline conventions. Reference when explaining pipeline integration.\n - **File I/O**: POSIX path semantics, PEP 428 (pathlib). Reference when explaining path handling or cross-platform issues.\n\n When citing a standard, mention it briefly (e.g., \"per PEP 440, version specifiers...\") — do not explain the standard itself.\n\n ## Codebase Reference\n\n Only reference API functions, parameters, item types, feature flags, and exceptions listed below. Do not invent or assume any capability not documented here.\n\n ### Public API\n\n | Symbol | Purpose |\n |---|---|\n | `FabricWorkspace(*, workspace_id, repository_directory, token_credential, ...)` | Initialize workspace connection. Requires keyword arguments. Either `workspace_id` or `workspace_name` must be provided. |\n | `publish_all_items(workspace, ...)` | Deploy all in-scope items to the target workspace. |\n | `unpublish_all_orphan_items(workspace, ...)` | Remove deployed items not found in the repository. |\n | `deploy_with_config(config_file_path, *, environment, token_credential, ...)` | Config-based deployment from a YAML file. |\n | `append_feature_flag(flag)` | Enable a feature flag at runtime. |\n | `change_log_level(\"DEBUG\")` | Enable debug logging for troubleshooting. |\n | `disable_file_logging()` | Disable file-based logging. |\n | `get_changed_items(repository_directory)` | Get list of git-changed items for selective deployment. |\n | `DeploymentResult` / `DeploymentStatus` | Deployment result types. |\n | `ItemType` | Enum of supported Fabric item types. |\n | `FeatureFlag` | Enum of supported feature flags. |\n\n ### FabricWorkspace Parameters\n\n | Parameter | Required | Description |\n |---|---|---|\n | `workspace_id` | One of `workspace_id` / `workspace_name` | Target workspace GUID. |\n | `workspace_name` | One of `workspace_id` / `workspace_name` | Target workspace display name (resolved to ID via API). |\n | `repository_directory` | Yes | Local path to the directory containing Fabric items. |\n | `token_credential` | Yes | Azure `TokenCredential` for API authentication. |\n | `item_type_in_scope` | No | List of item type strings to deploy. Defaults to all supported types. |\n | `environment` | No | Environment key for parameterization (must match `parameter.yml`). |\n\n ### publish_all_items Optional Parameters\n\n | Parameter | Feature Flag Required | Description |\n |---|---|---|\n | `item_name_exclude_regex` | None | Regex to exclude items by name. |\n | `folder_path_exclude_regex` | `enable_experimental_features` + `enable_exclude_folder` | Regex to exclude folders. |\n | `folder_path_to_include` | `enable_experimental_features` + `enable_include_folder` | List of folder paths to include. |\n | `items_to_include` | `enable_experimental_features` + `enable_items_to_include` | List of `\"item_name.item_type\"` strings. |\n | `shortcut_exclude_regex` | `enable_experimental_features` + `enable_shortcut_exclude` + `enable_shortcut_publish` | Regex to exclude Lakehouse shortcuts. |\n\n Note: `folder_path_exclude_regex` and `folder_path_to_include` are mutually exclusive.\n\n ### Supported Item Types (ItemType enum)\n\n ApacheAirflowJob, CopyJob, DataAgent, DataPipeline, Dataflow, Environment, Eventhouse, Eventstream, GraphQLApi, KQLDashboard, KQLDatabase, KQLQueryset, Lakehouse, MirroredDatabase, MLExperiment, MountedDataFactory, Notebook, Ontology, Reflex, Report, SemanticModel, SparkJobDefinition, SQLDatabase, UserDataFunction, VariableLibrary, Warehouse\n\n ### Feature Flags (FeatureFlag enum)\n\n | Flag | Description | Experimental |\n |---|---|---|\n | `enable_lakehouse_unpublish` | Enable deletion of Lakehouses | |\n | `enable_warehouse_unpublish` | Enable deletion of Warehouses | |\n | `enable_sqldatabase_unpublish` | Enable deletion of SQL Databases | |\n | `enable_eventhouse_unpublish` | Enable deletion of Eventhouses | |\n | `enable_kqldatabase_unpublish` | Enable deletion of KQL Databases | |\n | `enable_shortcut_publish` | Enable deploying shortcuts with Lakehouse | |\n | `enable_environment_variable_replacement` | Enable pipeline variable replacement | |\n | `disable_workspace_folder_publish` | Disable deploying workspace sub folders | |\n | `enable_experimental_features` | Gate for all experimental features | |\n | `enable_items_to_include` | Enable selective item publish/unpublish | ☑️ |\n | `enable_exclude_folder` | Enable folder-based exclusion | ☑️ |\n | `enable_include_folder` | Enable folder-based inclusion | ☑️ |\n | `enable_shortcut_exclude` | Enable selective shortcut publishing | ☑️ |\n | `enable_response_collection` | Enable collection of API responses | |\n | `continue_on_shortcut_failure` | Continue deployment when shortcuts fail | |\n | `enable_hard_delete` | Hard delete items (bypass recycle bin) | |\n\n ### Environment Variables (EnvVar enum)\n\n | Variable | Description |\n |---|---|\n | `FABRIC_CICD_HTTP_TRACE_ENABLED` | Enable HTTP request/response tracing (`1`/`true`/`yes`). |\n | `FABRIC_CICD_HTTP_TRACE_FILE` | Path to save HTTP trace output. |\n | `DEFAULT_API_ROOT_URL` | Override Power BI API root URL (default: `https://api.powerbi.com`). |\n | `FABRIC_API_ROOT_URL` | Override Fabric API root URL (default: `https://api.fabric.microsoft.com`). |\n | `FABRIC_CICD_RETRY_DELAY_OVERRIDE_SECONDS` | Override retry delay in seconds. |\n | `FABRIC_CICD_RETRY_AFTER_SECONDS` | Override retry-after delay for name conflicts (default: 300). |\n | `FABRIC_CICD_RETRY_BASE_DELAY_SECONDS` | Override base delay for retries (default: 30). |\n | `FABRIC_CICD_RETRY_MAX_DURATION_SECONDS` | Override max duration for retries (default: 300). |\n | `FABRIC_CICD_PARALLEL_MAX_WORKERS` | Override max parallel workers (default: 8). |\n | `FABRIC_CICD_VERSION_CHECK_DISABLED` | Disable startup version check. |\n\n ### Exception Types\n\n `InputError`, `TokenError`, `InvokeError`, `ParsingError`, `PublishError`, `ItemDependencyError`, `ParameterFileError`, `FailedPublishedItemStatusError`, `FileTypeError`, `ConfigValidationError`\n\n ### Authentication Methods\n\n Authentication requires an explicit `token_credential` parameter (any Azure `TokenCredential`):\n\n 1. **Azure CLI**: `AzureCliCredential()` — local development (requires `az login` first)\n 2. **Azure PowerShell**: `AzurePowerShellCredential()` — local development\n 3. **Service principal (secret)**: `ClientSecretCredential(tenant_id, client_id, client_secret)` — CI/CD pipelines\n 4. **Service principal (certificate)**: `CertificateCredential(tenant_id, client_id, certificate_path=...)` — CI/CD pipelines\n 5. **Managed identity**: `ManagedIdentityCredential()` — Azure-hosted pipelines\n 6. **Workload identity federation (OIDC)**: `WorkloadIdentityCredential(tenant_id, client_id)` — secretless; recommended for GitHub Actions and Azure DevOps with federated credentials\n 7. **Fabric notebook**: Custom `TokenCredential` wrapping `notebookutils.credentials.getToken(\"pbi\")` — see authentication docs\n\n Common auth error: `CredentialUnavailableError` — user not logged in or credential misconfigured.\n\n ### Parameterization (parameter.yml)\n\n Supports four replacement types:\n - `find_replace` — Simple string find/replace across all item files\n - `key_value_replace` — JSONPath-based key/value replacement\n - `spark_pool` — Spark pool configuration replacement\n - `semantic_model_binding` — Semantic model connection binding replacement\n\n The `parameter.yml` file must be in the root of `repository_directory`. Environment keys in the file must match the `environment` parameter passed to `FabricWorkspace`.\n\n ### Repository Directory Structure\n\n ```\n repository_directory/\n ├── ItemName.ItemType/\n │ ├── .platform (required metadata)\n │ └── \n ├── FolderName/ (optional workspace folders)\n │ └── ItemName.ItemType/\n │ ├── .platform\n │ └── \n └── parameter.yml (optional parameterization)\n ```\n\n ## Common Question Topics\n\n When answering, consider these frequent areas users ask about:\n\n - **Getting started**: How to install, initialize `FabricWorkspace`, run a first deployment\n - **Authentication**: Which credential type to use, how to authenticate in CI/CD pipelines vs local dev vs Fabric notebooks\n - **Parameterization**: How to write `parameter.yml`, supported replacement types, why replacements aren't applying (environment key mismatch)\n - **Item types**: Which item types are supported, how to add a new one to `item_type_in_scope`\n - **Selective deployment**: How to use `items_to_include`, folder filtering, `item_name_exclude_regex`, and which feature flags are required\n - **Config-based deployment**: How to write `config.yml`, environment mappings, difference from programmatic API\n - **Feature flags**: Which flags exist, how to enable them, which are experimental\n - **Unpublish safety**: Which item types require explicit feature flags to delete, what `enable_hard_delete` does\n - **Debugging**: How to enable debug logging, where to find `fabric_cicd.error.log`, how to read HTTP traces\n - **Pipeline integration**: How to use fabric-cicd in Azure DevOps or GitHub Actions pipelines\n - **Errors**: What specific exceptions mean, common causes of `InputError`, `TokenError`, `PublishError`\n - **Repository structure**: How to organize items, `.platform` file requirements, `ItemName.ItemType/` naming convention\n\n ## Your Task\n\n Answer the user's question accurately and concisely. Focus on what matters:\n - Provide a direct answer. Do not pad with context the user already knows.\n - Only include Python code examples if they directly answer the question.\n - If redirecting, state where and why in one sentence — nothing more.\n\n ## Assessment Categories\n\n Use exactly one of these in your assessment header:\n - **Answered** — You were able to provide a complete, accurate answer to the question.\n - **Requires Additional Details** — The question is unclear, incomplete, or lacks enough context to provide a useful answer. Specify exactly what is needed.\n - **Needs Team Review** — The question requires internal knowledge, involves undocumented behavior, touches sensitive areas (auth, security, data integrity), or involves roadmap/design decisions that only the team can answer. Escalate to the team.\n - **Redirect to Docs** — The question is better answered by existing documentation or is about general Fabric (not fabric-cicd specific). Provide the relevant links.\n\n ## Response Guidelines\n\n - Be concise and professional. You represent the fabric-cicd project.\n - Write like an expert — short sentences, no filler, no pleasantries.\n - Only highlight what is relevant to the question. Do not add tangential context.\n - Do not repeat the question back to the user.\n - Link to docs only when directly relevant.\n - Do not invent API functions, parameters, item types, or feature flags not listed in the Codebase Reference above. If unsure, direct to official docs.\n - If the question involves troubleshooting, recommend enabling debug logging (`change_log_level(\"DEBUG\")`) and sharing the `fabric_cicd.error.log` file.\n - Keep the response to **2-4 short paragraphs**. No bullet-heavy walls of text.\n\n ## Re-triage\n\n If the input starts with `[RE-TRIAGE]`, this issue was previously assessed and the author has responded with additional information. Focus your assessment on the new information provided. Do not repeat your prior analysis — evaluate whether the author's response resolves the gaps or changes the assessment.\n\n ## Response Format\n\n Start your response with a markdown header in this exact format:\n ### AI Assessment: \n\n Then provide your answer in clearly structured sections.\n\n End every response with a **Next Steps** section using exactly one of these:\n - `**⏳ Awaiting author feedback** — @{issue_author}, please provide the details listed above.` (when category is \"Requires Additional Details\")\n - `**🔔 Escalated to team** — This issue requires team review and has been flagged for attention.` (when category is \"Needs Team Review\")\n - `**✅ No action needed** — This question has been answered. If you need further help, feel free to follow up.` (when category is \"Answered\" or \"Redirect to Docs\")\n\n After the Next Steps section, always append this footer on a new line:\n `---`\n `> 💡 If this issue requires the team's attention and was not escalated, you can tag @microsoft/fabric-cicd to notify the team.`\n - role: user\n content: \"{{input}}\"\nmodel: openai/gpt-4.1\nmodelParameters:\n max_tokens: 2000\ntestData: []\nevaluators: []\n" }, { "path": ".github/pull_request_template.md", "content": "## Description\n\nBriefly describe what this PR does and why.\n\n## Linked Issue (REQUIRED)\n\n\n" }, { "path": ".github/workflows/ai-issue-triage.yml", "content": "name: \"AI Issue Triage\"\n\non:\n issues:\n types: [labeled]\n workflow_dispatch:\n inputs:\n issue_number:\n description: \"Issue number to triage\"\n required: true\n type: number\n\njobs:\n ai-triage:\n if: github.event_name == 'workflow_dispatch' || github.event.label.name == 'needs triage'\n runs-on: ubuntu-latest\n permissions:\n issues: write\n models: read\n contents: read\n\n env:\n # -------------------------------------------------------\n # Phase control — toggle these two flags to switch phases:\n # Testing (fork): suppress both → true / true\n # Production: enable both → false / false\n # -------------------------------------------------------\n SUPPRESS_LABELS: \"false\"\n SUPPRESS_COMMENTS: \"false\"\n\n steps:\n - name: Checkout\n uses: actions/checkout@v4\n\n - name: Resolve issue details\n id: issue\n uses: actions/github-script@v7\n with:\n script: |\n const issueNumber = context.payload.issue?.number || ${{ inputs.issue_number || 0 }};\n const owner = context.repo.owner;\n const repo = context.repo.repo;\n\n const { data: issue } = await github.rest.issues.get({\n owner, repo, issue_number: issueNumber,\n });\n\n // Detect re-triage: check if any ai: labels exist from a prior assessment\n const hasAiLabels = issue.labels.some(l => l.name.startsWith('ai:'));\n\n // Check for prior AI assessment comments\n const { data: comments } = await github.rest.issues.listComments({\n owner, repo, issue_number: issueNumber, per_page: 100,\n });\n const hasAiComment = comments.some(c =>\n c.body && c.body.includes('### AI Assessment:')\n );\n\n const isRetriage = hasAiLabels || hasAiComment;\n let issueBody = issue.body || '';\n\n if (isRetriage) {\n // Find the last AI comment index\n const lastAiIdx = comments.reduce((acc, c, i) =>\n c.body && c.body.includes('### AI Assessment:') ? i : acc, -1);\n\n // Collect author replies after the last AI comment\n const authorReplies = comments\n .slice(lastAiIdx + 1)\n .filter(c => c.user.login === issue.user.login)\n .map(c => c.body)\n .join('\\n\\n---\\n\\n');\n\n if (authorReplies) {\n issueBody = `[RE-TRIAGE] The author has provided additional information in response to a prior AI assessment.\\n\\n`\n + `## Original Issue Summary\\n${issue.title}\\n\\n`\n + `## Author's Follow-up Response\\n${authorReplies}\\n\\n`\n + `Focus your assessment on the new information provided above. Reference the original issue only if needed for context.`;\n }\n }\n\n core.setOutput('number', issue.number);\n core.setOutput('body', issueBody);\n core.setOutput('title', issue.title);\n core.setOutput('html_url', issue.html_url);\n core.setOutput('labels', issue.labels.map(l => l.name).join(','));\n core.setOutput('is_retriage', isRetriage.toString());\n\n - name: Run AI assessment\n id: ai-assessment\n uses: github/ai-assessment-comment-labeler@v1.0.1\n with:\n token: ${{ secrets.GITHUB_TOKEN }}\n issue_number: ${{ steps.issue.outputs.number }}\n issue_body: ${{ steps.issue.outputs.body }}\n repo_name: ${{ github.event.repository.name || github.repository }}\n owner: ${{ github.repository_owner }}\n ai_review_label: \"needs triage\"\n prompts_directory: \".github/prompts\"\n labels_to_prompts_mapping: \"bug,bug-triage.prompt.yml|enhancement,feature-triage.prompt.yml|question,question-triage.prompt.yml\"\n model: \"openai/gpt-4.1\"\n max_tokens: 2000\n suppress_comments: ${{ env.SUPPRESS_COMMENTS }}\n suppress_labels: ${{ env.SUPPRESS_LABELS }}\n\n - name: Post-process triage results\n if: steps.ai-assessment.outputs.ai_assessments != ''\n uses: actions/github-script@v7\n env:\n ASSESSMENT_OUTPUT: ${{ steps.ai-assessment.outputs.ai_assessments }}\n SUPPRESS_LABELS: ${{ env.SUPPRESS_LABELS }}\n ISSUE_NUMBER: ${{ steps.issue.outputs.number }}\n with:\n github-token: ${{ secrets.GITHUB_TOKEN }}\n script: |\n const assessments = JSON.parse(process.env.ASSESSMENT_OUTPUT);\n const issueNumber = parseInt(process.env.ISSUE_NUMBER);\n const owner = context.repo.owner;\n const repo = context.repo.repo;\n const suppressLabels = process.env.SUPPRESS_LABELS === 'true';\n\n let needsHumanReview = false;\n let addHelpWanted = false;\n let needsAuthorFeedback = false;\n let canAutoClose = false;\n\n for (const assessment of assessments) {\n const label = (assessment.assessmentLabel || '').toLowerCase();\n\n // Check if the assessment requires human review\n if (label.includes('needs team review') || label.includes('needs maintainer input') || label.includes('potential bug') || label.includes('needs discussion')) {\n needsHumanReview = true;\n }\n\n // Check if feature should be tagged as help wanted\n if (label.includes('help wanted')) {\n addHelpWanted = true;\n }\n\n // Check if more information is needed from the issue author\n if (label.includes('needs author feedback') || label.includes('requires additional details')) {\n needsAuthorFeedback = true;\n }\n\n // Check if the AI fully resolved the issue (answered question, explained misconfiguration, redirected to docs)\n if (label.includes('answered') || label.includes('likely misconfiguration') || label.includes('redirect to docs')) {\n canAutoClose = true;\n }\n\n // Log for job summary\n core.info(`Prompt: ${assessment.prompt}, Label: ${assessment.assessmentLabel}`);\n }\n\n // Skip label changes in summary-only mode (Phase 3)\n if (suppressLabels) {\n core.info('Labels suppressed — logging decisions only.');\n core.exportVariable('LABEL_DECISIONS', JSON.stringify({\n needsHumanReview, addHelpWanted, needsAuthorFeedback, canAutoClose,\n assessmentLabels: assessments.map(a => a.assessmentLabel),\n }));\n return;\n }\n\n // Add 'help wanted' label if AI recommended community contribution\n if (addHelpWanted) {\n await github.rest.issues.addLabels({\n owner,\n repo,\n issue_number: issueNumber,\n labels: ['help wanted']\n });\n core.info('Added \"help wanted\" label based on AI assessment.');\n }\n\n // If AI fully handled the issue without needing team review\n if (!needsHumanReview) {\n // Auto-close if AI fully resolved (answered, misconfiguration, redirected)\n if (canAutoClose && !needsAuthorFeedback && !addHelpWanted) {\n await github.rest.issues.update({\n owner,\n repo,\n issue_number: issueNumber,\n state: 'closed',\n state_reason: 'completed'\n });\n core.info('Auto-closed issue — AI fully resolved it.');\n }\n } else {\n // Add consolidated label for easy filtering of all issues needing team attention\n await github.rest.issues.addLabels({\n owner,\n repo,\n issue_number: issueNumber,\n labels: ['ai:needs team attention']\n });\n\n // Notify team via comment on escalated issues\n await github.rest.issues.createComment({\n owner,\n repo,\n issue_number: issueNumber,\n body: '🔔 @microsoft/fabric-cicd — This issue has been flagged by AI triage as requiring team attention. Please review the assessment above.'\n });\n core.info('Escalated to team.');\n }\n\n // Always remove 'needs triage' — triage is complete regardless of outcome\n try {\n await github.rest.issues.removeLabel({\n owner,\n repo,\n issue_number: issueNumber,\n name: 'needs triage'\n });\n core.info('Removed \"needs triage\" — triage complete.');\n } catch (e) {\n core.info(`Could not remove \"needs triage\" label: ${e.message}`);\n }\n\n - name: Generate triage summary\n if: always() && steps.ai-assessment.outputs.ai_assessments != ''\n uses: actions/github-script@v7\n env:\n ASSESSMENT_OUTPUT: ${{ steps.ai-assessment.outputs.ai_assessments }}\n LABEL_DECISIONS: ${{ env.LABEL_DECISIONS }}\n ISSUE_NUMBER: ${{ steps.issue.outputs.number }}\n ISSUE_TITLE: ${{ steps.issue.outputs.title }}\n ISSUE_URL: ${{ steps.issue.outputs.html_url }}\n with:\n script: |\n const assessments = JSON.parse(process.env.ASSESSMENT_OUTPUT);\n const issueNumber = process.env.ISSUE_NUMBER;\n const issueTitle = process.env.ISSUE_TITLE;\n const issueUrl = process.env.ISSUE_URL;\n\n let summary = `## 🤖 AI Triage Report\\n\\n`;\n summary += `**Issue:** [#${issueNumber} — ${issueTitle}](${issueUrl})\\n\\n`;\n\n for (const assessment of assessments) {\n summary += `### Prompt: \\`${assessment.prompt}\\`\\n`;\n summary += `**Assessment:** \\`${assessment.assessmentLabel}\\`\\n\\n`;\n summary += `
Full AI Response\\n\\n`;\n summary += `${assessment.response}\\n\\n`;\n summary += `
\\n\\n`;\n }\n\n // Show label decisions\n const ld = process.env.LABEL_DECISIONS ? JSON.parse(process.env.LABEL_DECISIONS) : null;\n if (ld) {\n summary += `### 🏷️ Label Decisions (not applied — testing mode)\\n\\n`;\n summary += `| Decision | Value |\\n|----------|-------|\\n`;\n summary += `| AI assessment labels | ${(ld.assessmentLabels || []).map(l => '`' + l + '`').join(', ')} |\\n`;\n summary += `| Would add \\`help wanted\\` | ${ld.addHelpWanted ? '✅ Yes' : '❌ No'} |\\n`;\n summary += `| Would add \\`ai:needs team attention\\` | ${ld.needsHumanReview ? '✅ Yes' : '❌ No'} |\\n`;\n summary += `| Would request author feedback | ${ld.needsAuthorFeedback ? '✅ Yes' : '❌ No'} |\\n`;\n summary += `| Would remove \\`needs triage\\` | ✅ Yes (always) |\\n`;\n summary += `| Would auto-close | ${ld.canAutoClose && !ld.needsAuthorFeedback && !ld.addHelpWanted ? '✅ Yes' : '❌ No'} |\\n`;\n summary += `| Would notify team | ${ld.needsHumanReview ? '✅ Yes' : '❌ No'} |\\n\\n`;\n }\n\n summary += `---\\n\\n`;\n\n core.summary.addRaw(summary);\n await core.summary.write();\n\n const fs = require('fs');\n fs.mkdirSync('triage-reports', { recursive: true });\n fs.writeFileSync(\n `triage-reports/issue-${issueNumber}-triage.md`,\n summary\n );\n\n - name: Upload triage report\n if: always() && steps.ai-assessment.outputs.ai_assessments != ''\n uses: actions/upload-artifact@v4\n with:\n name: triage-report-issue-${{ steps.issue.outputs.number }}\n path: triage-reports/\n retention-days: 30\n" }, { "path": ".github/workflows/bump.yml", "content": "name: Bump Version\n\non:\n pull_request:\n types: [closed]\n branches:\n - main\n workflow_dispatch:\n inputs:\n pr_number:\n description: \"Pull Request number to create release from\"\n required: true\n type: string\n\npermissions:\n contents: write\n pull-requests: read\n\njobs:\n get-pr-details:\n name: Get PR Details\n runs-on: ubuntu-latest\n if: github.event_name == 'workflow_dispatch'\n outputs:\n pr_title: ${{ steps.pr-info.outputs.pr_title }}\n pr_head_sha: ${{ steps.pr-info.outputs.pr_head_sha }}\n steps:\n - name: Get PR Information\n id: pr-info\n uses: actions/github-script@v7\n with:\n script: |\n const prNumber = parseInt('${{ github.event.inputs.pr_number }}');\n\n const { data: pr } = await github.rest.pulls.get({\n owner: context.repo.owner,\n repo: context.repo.repo,\n pull_number: prNumber\n });\n\n console.log(`PR #${prNumber}: ${pr.title}`);\n console.log(`Merge commit SHA: ${pr.merge_commit_sha}`);\n\n core.setOutput('pr_title', pr.title);\n core.setOutput('pr_head_sha', pr.merge_commit_sha);\n\n validate-version-bump:\n name: Validate Version Bump\n runs-on: ubuntu-latest\n needs: [get-pr-details]\n if: always() && !cancelled()\n outputs:\n is_version_bump: ${{ steps.version_bump_check.outputs.is_version_bump }}\n steps:\n - name: Checkout code\n uses: actions/checkout@v4\n with:\n fetch-depth: 0\n\n - name: Validate Version Bump\n id: version_bump_check\n run: |\n set -e\n PR_TITLE=\"${{ github.event.pull_request.title || needs.get-pr-details.outputs.pr_title }}\"\n VERSION_REGEX='^v([0-9]+\\.[0-9]+\\.[0-9]+)$'\n\n if [[ \"$PR_TITLE\" =~ $VERSION_REGEX ]]; then\n echo \"✅ PR title matches version format: $PR_TITLE\"\n echo \"is_version_bump=true\" >> $GITHUB_OUTPUT\n else\n echo \"ℹ️ PR title does not match version format: $PR_TITLE\"\n echo \"is_version_bump=false\" >> $GITHUB_OUTPUT\n fi\n\n bump-version:\n name: Bump Version\n needs: [validate-version-bump]\n if: |\n (github.event_name == 'workflow_dispatch') ||\n (github.event_name == 'pull_request' && github.event.pull_request.merged == true && needs.validate-version-bump.outputs.is_version_bump == 'true')\n runs-on: ubuntu-latest\n steps:\n - name: Generate GitHub App Token\n id: app-token\n uses: actions/create-github-app-token@v1\n with:\n app-id: ${{ secrets.APP_ID }}\n private-key: ${{ secrets.APP_PRIVATE_KEY }}\n\n - name: Checkout repository\n uses: actions/checkout@v4\n with:\n fetch-depth: 0\n token: ${{ steps.app-token.outputs.token }}\n\n - name: Extract version\n id: version-check\n run: |\n # Determine which commit to get version from\n if [ \"${{ github.event_name }}\" = \"workflow_dispatch\" ]; then\n COMMIT_SHA=\"${{ needs.get-pr-details.outputs.pr_head_sha }}\"\n git fetch origin \"$COMMIT_SHA\"\n CONSTANTS_VERSION=$(git show ${COMMIT_SHA}:src/fabric_cicd/constants.py | grep -oP '(?<=^VERSION = \").*(?=\")')\n else\n # For pull_request events, use current working directory (HEAD)\n CONSTANTS_VERSION=$(grep -oP '(?<=^VERSION = \").*(?=\")' src/fabric_cicd/constants.py)\n fi\n\n echo \"Version: $CONSTANTS_VERSION\"\n\n if [ -z \"$CONSTANTS_VERSION\" ]; then\n echo \"ERROR: Could not extract version from constants.py\"\n exit 1\n fi\n\n echo \"version_number=$CONSTANTS_VERSION\" >> $GITHUB_OUTPUT\n echo \"version_tag=v$CONSTANTS_VERSION\" >> $GITHUB_OUTPUT\n\n # Log trigger type\n if [ \"${{ github.event_name }}\" = \"workflow_dispatch\" ]; then\n echo \"🚀 Manual trigger for PR #${{ github.event.inputs.pr_number }}\"\n elif [ \"${{ github.event_name }}\" = \"pull_request\" ]; then\n echo \"🚀 PR merge trigger - validated version bump\"\n fi\n\n - name: Create tag if needed\n if: needs.validate-version-bump.outputs.is_version_bump == 'true'\n id: tag-creation\n run: |\n TAG_NAME=\"${{ steps.version-check.outputs.version_tag }}\"\n echo \"Checking tag: $TAG_NAME\"\n\n # Check if tag exists locally or remotely\n if git tag -l | grep -q \"^$TAG_NAME$\" || git ls-remote --tags origin | grep -q \"refs/tags/$TAG_NAME$\"; then\n echo \"ℹ️ Tag $TAG_NAME already exists, will proceed with existing tag\"\n echo \"tag_created=false\" >> $GITHUB_OUTPUT\n else\n echo \"✅ Tag $TAG_NAME does not exist, creating new tag\"\n \n # Configure git\n git config user.name \"fabric-cicd-release[bot]\"\n git config user.email \"fabric-cicd-release[bot]@users.noreply.github.com\"\n\n # Create and push the tag\n git tag \"$TAG_NAME\"\n git push origin \"$TAG_NAME\"\n echo \"✅ Created and pushed tag: $TAG_NAME\"\n echo \"tag_created=true\" >> $GITHUB_OUTPUT\n fi\n\n - name: Create GitHub Release\n if: needs.validate-version-bump.outputs.is_version_bump == 'true'\n uses: actions/github-script@v7\n with:\n github-token: ${{ steps.app-token.outputs.token }}\n script: |\n const tagName = '${{ steps.version-check.outputs.version_tag }}';\n\n try {\n // Check if release already exists\n let existingRelease;\n try {\n existingRelease = await github.rest.repos.getReleaseByTag({\n owner: context.repo.owner,\n repo: context.repo.repo,\n tag: tagName\n });\n console.log(`ℹ️ Release ${tagName} already exists: ${existingRelease.data.html_url}`);\n console.log('Skipping release creation');\n return;\n } catch (error) {\n if (error.status !== 404) {\n throw error;\n }\n // Release doesn't exist, proceed to create it\n }\n\n // Extract changelog content for new release\n const versionNumber = '${{ steps.version-check.outputs.version_number }}';\n const fs = require('fs');\n const path = 'docs/changelog.md';\n \n let changelogContent = `Release ${tagName}`;\n \n if (fs.existsSync(path)) {\n try {\n const changelogText = fs.readFileSync(path, 'utf8');\n const lines = changelogText.split('\\n');\n let found = false;\n let content = [];\n \n for (const line of lines) {\n if (line.startsWith('## [v')) {\n if (found) break; // Hit next version, stop\n if (line.includes(`[v${versionNumber}]`)) {\n found = true;\n continue; // Skip the version header line\n }\n } else if (found) {\n // Skip span tags and empty lines\n if (line.trim() === '' || line.startsWith(' 0) {\n changelogContent = content.join('\\n').trim();\n } else {\n console.log(`Changelog content not found for version ${versionNumber}`);\n }\n } catch (error) {\n console.log('Error reading changelog:', error.message);\n }\n } else {\n console.log('Changelog file not found');\n }\n\n // Create new release\n const release = await github.rest.repos.createRelease({\n owner: context.repo.owner,\n repo: context.repo.repo,\n tag_name: tagName,\n name: tagName,\n body: changelogContent,\n draft: false,\n prerelease: false\n });\n \n console.log(`✅ Created GitHub release: ${tagName}`);\n console.log(`Release URL: ${release.data.html_url}`);\n } catch (error) {\n console.error('Failed to create release:', error);\n throw error;\n }\n" }, { "path": ".github/workflows/changelog.yml", "content": "# yaml-language-server: $schema=https://json.schemastore.org/github-workflow.json\n---\nname: 🔄 Changelog\n\non:\n pull_request:\n branches:\n - main\n types:\n - opened\n - reopened\n - labeled\n - unlabeled\n - synchronize\n workflow_dispatch:\n\nconcurrency:\n group: ${{ format('{0}-{1}-{2}-{3}-{4}', github.workflow, github.event_name, github.ref, github.base_ref || null, github.head_ref || null) }}\n cancel-in-progress: true\n\npermissions:\n contents: read\n\njobs:\n changelog-existence:\n name: 🔄 Check Changelog\n if: ${{ !contains(github.event.pull_request.labels.*.name, 'skip changelog') && github.actor != 'dependabot[bot]' }}\n runs-on: ubuntu-latest\n env:\n BASE_SHA: ${{ github.event.pull_request.base.sha }}\n HEAD_SHA: ${{ github.event.pull_request.head.sha }}\n steps:\n - name: ⤵️ Checkout\n uses: actions/checkout@v4\n with:\n fetch-depth: 0 # Required to access full commit history\n\n - name: ✔️ Check for changelog changes\n id: changelog_check\n uses: actions/github-script@v6\n with:\n script: |\n const { execSync } = require('child_process');\n const base = process.env.BASE_SHA;\n const head = process.env.HEAD_SHA;\n console.log(`Comparing changes from ${base} to ${head}`)\n const output = execSync(`git diff --name-only --no-renames --diff-filter=AM ${base} ${head}`).toString();\n const files = output.split('\\n').filter(Boolean);\n const changelogExists = files.some(file => file.startsWith('.changes/unreleased/') && file.endsWith('.yaml'));\n core.setOutput('exists', changelogExists);\n\n - name: 🚧 Setup Node\n if: steps.changelog_check.outputs.exists == 'true'\n uses: actions/setup-node@v6\n with:\n node-version: \"20\"\n\n - name: 🚧 Install Changie\n if: steps.changelog_check.outputs.exists == 'true'\n run: npm i -g changie\n\n - name: 🔄 Prepare comment (changelog)\n if: steps.changelog_check.outputs.exists == 'true'\n run: |\n echo -e \"# Changelog Preview\\n\" > changie.md\n changie batch patch --dry-run --prerelease 'dev' >> changie.md\n cat changie.md >> $GITHUB_STEP_SUMMARY\n\n - name: 🔄 Prepare comment (missing)\n if: steps.changelog_check.outputs.exists == 'false'\n run: |\n echo -e \"# 🛑 Changelog entry required to merge\\n\" > changie.md\n echo \"Run \\`changie new\\` to add a new changelog entry\" >> changie.md\n cat changie.md >> $GITHUB_STEP_SUMMARY\n\n - name: ✅ Pass if changelog entry exists\n if: steps.changelog_check.outputs.exists == 'true'\n run: |\n echo \"✅ Changelog entry exists.\"\n exit 0\n\n - name: 🛑 Fail if changelog entry is missing and required\n if: steps.changelog_check.outputs.exists == 'false'\n run: |\n echo \"🛑 Changelog entry required to merge.\"\n echo \"🛑 Please run 'changie new' to add a new changelog entry.\"\n exit 1\n\n changelog-skip:\n name: ⏭️ Skip Changelog\n if: ${{ contains(github.event.pull_request.labels.*.name, 'skip changelog') || github.actor == 'dependabot[bot]' }}\n runs-on: ubuntu-latest\n steps:\n - name: ✅ Pass (skip)\n run: |\n echo \"⏭️ Changelog check skipped.\" >> $GITHUB_STEP_SUMMARY\n exit 0\n" }, { "path": ".github/workflows/publish_docs.yml", "content": "name: Publish Docs\n\non:\n workflow_run:\n workflows: [\"Bump Version\"]\n types: [completed]\n workflow_dispatch:\n\npermissions:\n contents: write\n\njobs:\n publish-docs:\n name: Publish Docs\n runs-on: ubuntu-latest\n if: github.event_name == 'workflow_dispatch' || github.event.workflow_run.conclusion == 'success'\n steps:\n - name: Generate GitHub App Token\n id: app-token\n uses: actions/create-github-app-token@v1\n with:\n app-id: ${{ secrets.APP_ID }}\n private-key: ${{ secrets.APP_PRIVATE_KEY }}\n\n - name: Checkout repository\n uses: actions/checkout@v4\n with:\n fetch-depth: 0\n token: ${{ steps.app-token.outputs.token }}\n\n - name: Extract version\n id: version-check\n run: |\n CONSTANTS_VERSION=$(grep -oP '(?<=^VERSION = \").*(?=\")' src/fabric_cicd/constants.py)\n echo \"Version: $CONSTANTS_VERSION\"\n\n if [ -z \"$CONSTANTS_VERSION\" ]; then\n echo \"ERROR: Could not extract version from constants.py\"\n exit 1\n fi\n\n echo \"version_number=$CONSTANTS_VERSION\" >> $GITHUB_OUTPUT\n\n - name: Install Python\n uses: actions/setup-python@v5\n with:\n python-version: \"3.9\"\n\n - name: Install Requirements\n run: |\n python -m pip install --upgrade pip\n python -m pip install uv\n\n - name: Deploy GitHub Pages\n env:\n GITHUB_TOKEN: ${{ steps.app-token.outputs.token }}\n run: |\n git config user.name \"fabric-cicd-release[bot]\"\n git config user.email \"fabric-cicd-release[bot]@users.noreply.github.com\"\n git fetch --no-tags --prune --depth=1 origin +refs/heads/gh-pages:refs/remotes/origin/gh-pages\n uv sync\n\n VERSION=\"${{ steps.version-check.outputs.version_number }}\"\n echo \"Deploying docs for version: $VERSION\"\n\n uv run mike deploy \\\n --update-aliases \\\n --branch gh-pages \\\n --push \\\n $VERSION \\\n latest\n uv run mike set-default --push latest\n" }, { "path": ".github/workflows/test.yml", "content": "name: Test\ndescription: \"Run unit tests for the Fabric CICD project\"\n\non:\n pull_request:\n branches: [\"main\"]\n types: [opened, edited, synchronize, ready_for_review]\n push:\n branches: [\"main\"]\n\njobs:\n unit_test:\n name: Unit Test (Python ${{ matrix.python-version }})\n runs-on: ubuntu-latest\n strategy:\n fail-fast: false\n matrix:\n python-version: [\"3.9\", \"3.10\", \"3.11\", \"3.12\", \"3.13\"]\n steps:\n - name: Checkout code\n uses: actions/checkout@v4\n\n - name: Set up Python ${{ matrix.python-version }}\n uses: actions/setup-python@v5\n with:\n python-version: ${{ matrix.python-version }}\n\n - name: Run unit tests\n run: |\n python -m pip install --upgrade pip\n pip install uv\n uv sync --dev\n uv run pytest -v || exit 1 # Fail the job if any tests fail\n" }, { "path": ".github/workflows/validate.yml", "content": "name: Validate PR\ndescription: \"Validate pull requests for code conventions, naming conventions, linked issues, and version bumps\"\n\non:\n pull_request:\n branches: [\"main\"]\n types: [opened, edited, synchronize, ready_for_review, labeled, unlabeled]\n\npermissions:\n contents: read\n pull-requests: write\n issues: write\n statuses: write\n\njobs:\n format_ruff:\n if: ${{ github.event.action != 'labeled' && github.event.action != 'unlabeled' }}\n name: Code Formatted\n runs-on: ubuntu-latest\n steps:\n - name: Checkout code\n uses: actions/checkout@v2\n\n - name: Set up Python\n uses: actions/setup-python@v2\n with:\n python-version: \"3.x\"\n\n - name: Install dependencies\n run: |\n python -m pip install --upgrade pip\n pip install ruff\n\n - name: Run ruff format\n run: |\n if ruff format; then\n echo \"✅ ruff format passed.\"\n else\n echo \"❌ ruff format failed.\"\n exit 1\n fi\n lint_ruff:\n if: ${{ github.event.action != 'labeled' && github.event.action != 'unlabeled' }}\n name: Code Linted\n runs-on: ubuntu-latest\n steps:\n - name: Checkout code\n uses: actions/checkout@v2\n\n - name: Set up Python\n uses: actions/setup-python@v2\n with:\n python-version: \"3.x\"\n\n - name: Install dependencies\n run: |\n python -m pip install --upgrade pip\n pip install ruff\n\n - name: Run ruff lint\n run: |\n if ruff check; then\n echo \"✅ ruff lint passed.\"\n else\n echo \"❌ ruff lint failed.\"\n exit 1\n fi\n\n validate-version-bump:\n name: Proper Version Bump\n runs-on: ubuntu-latest\n outputs:\n is_version_bump: ${{ steps.version_bump_check.outputs.is_version_bump }}\n steps:\n - name: Checkout code\n uses: actions/checkout@v4\n with:\n fetch-depth: 0\n\n - name: Validate Version Bump\n id: version_bump_check\n env:\n PR_TITLE: ${{ github.event.pull_request.title }}\n PR_HEAD_SHA: ${{ github.event.pull_request.head.sha }}\n run: |\n set -e\n VERSION_REGEX='^v([0-9]+\\.[0-9]+\\.[0-9]+)$'\n BASE_REF=\"origin/main\"\n CHANGED_FILES=$(git diff --name-only \"$BASE_REF...$PR_HEAD_SHA\")\n\n VERSION_CHANGED=false\n TITLE_IS_VERSION=false\n OLD_VERSION=\"\"\n NEW_VERSION=\"\"\n # Get VERSION from main branch\n OLD_VERSION=$(git show origin/main:src/fabric_cicd/constants.py | grep '^VERSION = ' | sed -E 's/VERSION = \"([^\"]+)\"/\\1/')\n # Get VERSION from current branch\n NEW_VERSION=$(grep '^VERSION = ' src/fabric_cicd/constants.py | sed -E 's/VERSION = \"([^\"]+)\"/\\1/')\n\n if [ \"$OLD_VERSION\" != \"$NEW_VERSION\" ]; then\n VERSION_CHANGED=true\n fi\n\n if [[ \"$PR_TITLE\" =~ $VERSION_REGEX ]]; then\n TITLE_IS_VERSION=true\n fi\n\n if [ \"$TITLE_IS_VERSION\" = true ] || [ \"$VERSION_CHANGED\" = true ]; then\n echo \"is_version_bump=true\" >> $GITHUB_OUTPUT\n else\n echo \"is_version_bump=false\" >> $GITHUB_OUTPUT\n echo \"✅ Version bump validation passed.\"\n exit 0\n fi\n\n # 1. If version is updated, title must match\n if [ \"$VERSION_CHANGED\" = true ]; then\n EXPECTED_TITLE=\"v$NEW_VERSION\"\n if [ \"$PR_TITLE\" != \"$EXPECTED_TITLE\" ]; then\n echo \"❌ PR title must be vX.X.X when VERSION is changed. Expected title: $EXPECTED_TITLE\"\n exit 1\n fi\n fi\n\n # 2. If title is version format, constants.py, changelog.md, and .changes/v.md must be included in changed files\n # Note: Additional files (e.g., removed unreleased change files) are allowed alongside the required files\n if [[ \"$PR_TITLE\" =~ $VERSION_REGEX ]]; then\n VERSION_NUMBER=\"${BASH_REMATCH[1]}\"\n REQUIRED_FILES=(\"src/fabric_cicd/constants.py\" \"docs/changelog.md\" \".changes/v${VERSION_NUMBER}.md\")\n MISSING_FILES=()\n for file in \"${REQUIRED_FILES[@]}\"; do\n if ! echo \"$CHANGED_FILES\" | grep -Fqx \"$file\"; then\n MISSING_FILES+=(\"$file\")\n fi\n done\n if [ ${#MISSING_FILES[@]} -ne 0 ]; then\n echo \"❌ The following required files must be included in a PR titled vX.X.X: ${MISSING_FILES[*]}\"\n exit 1\n fi\n fi\n echo \"✅ Version bump validation passed.\"\n\n check-author-permissions:\n name: Check Author Permissions\n runs-on: ubuntu-latest\n outputs:\n skip_validation: ${{ steps.check_permissions.outputs.skip_validation }}\n permissions:\n pull-requests: read\n steps:\n - name: Check PR Author Collaborator Role\n id: check_permissions\n uses: actions/github-script@v7\n with:\n script: |\n const prAuthor = context.payload.pull_request.user.login;\n const repo = context.repo;\n\n // Skip validation for dependabot\n if (prAuthor === 'dependabot[bot]') {\n console.log('✅ Dependabot PR detected. Skipping linked issue validation.');\n core.setOutput('skip_validation', 'true');\n return;\n }\n\n console.log(`Checking collaborator role for PR author: ${prAuthor}`);\n\n try {\n const { data: collaborator } = await github.rest.repos.getCollaboratorPermissionLevel({\n owner: repo.owner,\n repo: repo.repo,\n username: prAuthor\n });\n \n const roleName = collaborator.role_name;\n console.log(`Collaborator role for ${prAuthor}: ${roleName}`);\n \n // Skip validation for users with admin, maintain, or write role\n const skipValidation = ['admin', 'maintain', 'write'].includes(roleName);\n \n core.setOutput('skip_validation', skipValidation.toString());\n \n if (skipValidation) {\n console.log(`✅ User ${prAuthor} has ${roleName} role. Validation job for linked issue will be skipped.`);\n } else {\n console.log(`ℹ️ User ${prAuthor} has ${roleName} role. Validation job for linked issue will run.`);\n }\n } catch (error) {\n console.log(`⚠️ Could not determine collaborator role for ${prAuthor}: ${error.message}`);\n console.log('Defaulting to running validation job for linked issue.');\n core.setOutput('skip_validation', 'false');\n }\n\n validate-linked-issue:\n name: Issue Linked\n runs-on: ubuntu-latest\n needs: [check-author-permissions, validate-version-bump]\n if: needs.check-author-permissions.outputs.skip_validation == 'false' && needs.validate-version-bump.outputs.is_version_bump == 'false' && !contains(github.event.pull_request.labels.*.name, 'skip changelog')\n permissions:\n pull-requests: read\n issues: read\n steps:\n - name: Validate Linked Issue\n uses: actions/github-script@v7\n with:\n script: |\n const prNumber = context.issue.number;\n const repo = context.repo;\n\n // Get PR details\n const pr = await github.rest.pulls.get({\n owner: repo.owner,\n repo: repo.repo,\n pull_number: prNumber\n });\n\n const prTitle = pr.data.title || '';\n\n // First, check for issues linked to the PR via GitHub's native linking\n let linkedIssues = [];\n try {\n // Use GraphQL to get linked issues\n const query = `\n query($owner: String!, $repo: String!, $number: Int!) {\n repository(owner: $owner, name: $repo) {\n pullRequest(number: $number) {\n closingIssuesReferences(first: 10) {\n nodes {\n number\n }\n }\n }\n }\n }\n `;\n \n const result = await github.graphql(query, {\n owner: repo.owner,\n repo: repo.repo,\n number: prNumber\n });\n \n linkedIssues = result.repository.pullRequest.closingIssuesReferences.nodes;\n \n if (linkedIssues.length > 0) {\n console.log(`✅ Found ${linkedIssues.length} linked issue(s): ${linkedIssues.map(issue => `#${issue.number}`).join(', ')}`);\n console.log('✅ Pull request is properly linked to an issue via GitHub linking.');\n return;\n }\n } catch (error) {\n console.log('⚠️ Could not check for linked issues via GraphQL, falling back to text analysis:', error.message);\n }\n\n // Issue reference patterns - more specific to avoid false positives\n const keywordPatterns = [\n /(?:close[sd]?|fix(?:e[sd])?|resolve[sd]?)\\s+#(\\d+)/gi,\n /(?:close[sd]?|fix(?:e[sd])?|resolve[sd]?)\\s+https:\\/\\/github\\.com\\/[^\\/]+\\/[^\\/]+\\/issues\\/(\\d+)/gi\n ];\n\n // Generic hash pattern (less specific, used as fallback)\n const hashPattern = /#(\\d+)(?!\\w)/g;\n\n let foundIssueNumbers = new Set();\n\n // Check PR title for issue references\n const textToCheck = prTitle;\n\n // First, look for keyword-based references (more reliable)\n for (const pattern of keywordPatterns) {\n const matches = textToCheck.matchAll(pattern);\n for (const match of matches) {\n const issueNumber = match[1];\n if (issueNumber && !isNaN(issueNumber)) {\n foundIssueNumbers.add(parseInt(issueNumber));\n }\n }\n }\n\n // If no keyword-based references found, look for simple hash references\n if (foundIssueNumbers.size === 0) {\n const matches = textToCheck.matchAll(hashPattern);\n for (const match of matches) {\n const issueNumber = match[1];\n if (issueNumber && !isNaN(issueNumber)) {\n foundIssueNumbers.add(parseInt(issueNumber));\n }\n }\n }\n\n if (foundIssueNumbers.size === 0) {\n core.setFailed(\n '❌ This pull request must be linked to an issue. Please:\\n' +\n '1. Reference an issue in the PR title using \"Fixes #123\", \"Closes #456\", or \"Resolves #789\"\\n' +\n '2. Make sure the referenced issue exists in this repository\\n\\n' +\n 'See our contribution guidelines for more details.'\n );\n return;\n }\n\n // Verify that the referenced issues actually exist\n let validIssueFound = false;\n const invalidIssues = [];\n\n for (const issueNumber of foundIssueNumbers) {\n try {\n await github.rest.issues.get({\n owner: repo.owner,\n repo: repo.repo,\n issue_number: issueNumber\n });\n validIssueFound = true;\n console.log(`✅ Found valid issue reference: #${issueNumber}`);\n } catch (error) {\n if (error.status === 404) {\n invalidIssues.push(issueNumber);\n console.log(`❌ Issue #${issueNumber} does not exist`);\n }\n }\n }\n\n if (!validIssueFound) {\n const invalidList = invalidIssues.length > 0 ?\n `\\n\\nInvalid issue references found: ${invalidIssues.map(n => `#${n}`).join(', ')}` : '';\n\n core.setFailed(\n '❌ This pull request must be linked to a valid issue in this repository.' +\n invalidList +\n '\\n\\nPlease:\\n' +\n '1. Create an issue first if one doesn\\'t exist\\n' +\n '2. Reference the issue in the PR title using \"Fixes #123\", \"Closes #456\", or \"Resolves #789\"\\n' +\n '3. Make sure the issue number is correct\\n\\n' +\n 'See our contribution guidelines for more details.'\n );\n return;\n }\n\n console.log('✅ Pull request is properly linked to an issue.');\n" }, { "path": ".gitignore", "content": "# Byte-compiled / optimized / DLL files\n__pycache__/\n*.py[cod]\n*$py.class\n\n# C extensions\n*.so\n\n# Distribution / packaging\n.Python\nbuild/\ndevelop-eggs/\ndist/\ndownloads/\neggs/\n.eggs/\nlib/\nlib64/\nparts/\nsdist/\nvar/\nwheels/\nshare/python-wheels/\n*.egg-info/\n.installed.cfg\n*.egg\nMANIFEST\n\n# PyInstaller\n# Usually these files are written by a python script from a template\n# before PyInstaller builds the exe, so as to inject date/other infos into it.\n*.manifest\n*.spec\n\n# Installer logs\npip-log.txt\npip-delete-this-directory.txt\n\n# Unit test / coverage reports\nhtmlcov/\n.tox/\n.nox/\n.coverage\n.coverage.*\n.cache\nnosetests.xml\ncoverage.xml\n*.cover\n*.py,cover\n.hypothesis/\n.pytest_cache/\ncover/\n\n# Translations\n*.mo\n*.pot\n\n# Django stuff:\n*.log\nlocal_settings.py\ndb.sqlite3\ndb.sqlite3-journal\n\n# Flask stuff:\ninstance/\n.webassets-cache\n\n# Scrapy stuff:\n.scrapy\n\n# Sphinx documentation\ndocs/_build/\n\n# PyBuilder\n.pybuilder/\ntarget/\n\n# Jupyter Notebook\n.ipynb_checkpoints\n\n# IPython\nprofile_default/\nipython_config.py\n\n# pyenv\n# For a library or package, you might want to ignore these files since the code is\n# intended to run in multiple environments; otherwise, check them in:\n# .python-version\n\n# pipenv\n# According to pypa/pipenv#598, it is recommended to include Pipfile.lock in version control.\n# However, in case of collaboration, if having platform-specific dependencies or dependencies\n# having no cross-platform support, pipenv may install dependencies that don't work, or not\n# install all needed dependencies.\n#Pipfile.lock\n\n# poetry\n# Similar to Pipfile.lock, it is generally recommended to include poetry.lock in version control.\n# This is especially recommended for binary packages to ensure reproducibility, and is more\n# commonly ignored for libraries.\n# https://python-poetry.org/docs/basic-usage/#commit-your-poetrylock-file-to-version-control\n#poetry.lock\n\n# pdm\n# Similar to Pipfile.lock, it is generally recommended to include pdm.lock in version control.\n#pdm.lock\n# pdm stores project-wide configurations in .pdm.toml, but it is recommended to not include it\n# in version control.\n# https://pdm.fming.dev/#use-with-ide\n.pdm.toml\n\n# PEP 582; used by e.g. github.com/David-OConnor/pyflow and github.com/pdm-project/pdm\n__pypackages__/\n\n# Celery stuff\ncelerybeat-schedule\ncelerybeat.pid\n\n# SageMath parsed files\n*.sage.py\n\n# Environments\n.env\n.venv\nenv/\nvenv/\nENV/\nenv.bak/\nvenv.bak/\n\n# Spyder project settings\n.spyderproject\n.spyproject\n\n# Rope project settings\n.ropeproject\n\n# mkdocs documentation\n/site\n\n# mypy\n.mypy_cache/\n.dmypy.json\ndmypy.json\n\n# ruff\n.ruff_cache/\n\n# Pyre type checker\n.pyre/\n\n# pytype static type analyzer\n.pytype/\n\n# Cython debug symbols\ncython_debug/\n\n# PyCharm\n# JetBrains specific template is maintained in a separate JetBrains.gitignore that can\n# be found at https://github.com/github/gitignore/blob/main/Global/JetBrains.gitignore\n# and can be added to the global gitignore or merged into this file. For a more nuclear\n# option (not recommended) you can uncomment the following to ignore the entire idea folder.\n#.idea/\n\n# http traces should only be committed at the fixture root\n/http_trace.json\n/http_trace.json.lock\n/http_trace.json.gz\n" }, { "path": ".prettierignore", "content": "sample/workspace/\ndocs/how_to/parameterization.md\n" }, { "path": ".prettierrc", "content": "{\n \"printWidth\": 100,\n \"tabWidth\": 4,\n \"useTabs\": false,\n \"semi\": true,\n \"trailingComma\": \"all\"\n}\n" }, { "path": ".python-version", "content": "3.11\n" }, { "path": ".vscode/extensions.json", "content": "{\n \"recommendations\": [\n \"ms-python.python\",\n \"esbenp.prettier-vscode\",\n \"ms-vscode.powershell\",\n \"charliermarsh.ruff\",\n \"tamasfe.even-better-toml\"\n ]\n}\n" }, { "path": ".vscode/launch.json", "content": "{\n \"version\": \"0.2.0\",\n \"configurations\": [\n {\n \"name\": \"Debug: Trace Publish All Items\",\n \"type\": \"debugpy\",\n \"request\": \"launch\",\n \"program\": \"${workspaceFolder}/devtools/debug_trace_deployment.py\",\n \"console\": \"integratedTerminal\",\n \"justMyCode\": false,\n \"env\": {\n \"PYTHONPATH\": \"${workspaceFolder}/src\",\n \"FABRIC_WORKSPACE_ID\": \"your-fabric-workspace-guid\"\n },\n \"cwd\": \"${workspaceFolder}\"\n }\n ]\n}\n" }, { "path": ".vscode/settings.json", "content": "{\n \"editor\": {\n \"trimAutoWhitespace\": false,\n \"defaultFormatter\": \"esbenp.prettier-vscode\",\n \"formatOnSave\": true\n },\n \"diffEditor\": {\n \"ignoreTrimWhitespace\": false\n },\n \"files\": {\n \"trimTrailingWhitespace\": false,\n \"trimTrailingWhitespaceInRegexAndStrings\": false,\n \"insertFinalNewline\": true,\n \"autoSave\": \"off\"\n },\n \"[python]\": {\n \"editor.defaultFormatter\": \"charliermarsh.ruff\",\n \"editor.codeActionsOnSave\": {\n \"source.fixAll\": \"explicit\"\n },\n \"editor.formatOnSave\": true\n },\n \"[powershell]\": {\n \"editor.defaultFormatter\": \"ms-vscode.powershell\",\n \"editor.formatOnSave\": true\n },\n \"[toml]\": {\n \"editor.defaultFormatter\": \"tamasfe.even-better-toml\",\n \"editor.formatOnSave\": true\n },\n \"git\": {\n \"branchPrefix\": \"users/\",\n \"enableSmartCommit\": true,\n \"confirmSync\": false,\n \"autofetch\": true\n },\n \"ruff\": {\n \"organizeImports\": true,\n \"fixAll\": true\n },\n \"python.terminal.activateEnvironment\": false,\n \"python.testing.pytestArgs\": [\"tests\"],\n \"python.testing.unittestEnabled\": false,\n \"python.testing.pytestEnabled\": true\n}\n" }, { "path": "CODE_OF_CONDUCT.md", "content": "# Microsoft Open Source Code of Conduct\n\nThis project has adopted the [Microsoft Open Source Code of Conduct](https://opensource.microsoft.com/codeofconduct/).\n\nResources:\n\n- [Microsoft Open Source Code of Conduct](https://opensource.microsoft.com/codeofconduct/)\n- [Microsoft Code of Conduct FAQ](https://opensource.microsoft.com/codeofconduct/faq/)\n- Contact [opencode@microsoft.com](mailto:opencode@microsoft.com) with questions or concerns\n" }, { "path": "CONTRIBUTING.md", "content": "# Contributing\n\nThis project welcomes contributions and suggestions. Most contributions require you to\nagree to a Contributor License Agreement (CLA) declaring that you have the right to,\nand actually do, grant us the rights to use your contribution. For details, visit\nhttps://cla.microsoft.com.\n\nWhen you submit a pull request, a CLA-bot will automatically determine whether you need\nto provide a CLA and decorate the PR appropriately (e.g., label, comment). Simply follow the\ninstructions provided by the bot. You will only need to do this once across all repositories using our CLA.\n\nThis project has adopted the [Microsoft Open Source Code of Conduct](https://opensource.microsoft.com/codeofconduct/).\nFor more information see the [Code of Conduct FAQ](https://opensource.microsoft.com/codeofconduct/faq/)\nor contact [opencode@microsoft.com](mailto:opencode@microsoft.com) with any additional questions or comments.\n\n## Ways to Contribute\n\nWe welcome several types of contributions:\n\n- 🔧 **Bug fixes** - Fix issues and improve reliability\n- ✨ **New features** - Add new commands or functionality\n- 🆕 **New Items Support** - Onboard new Fabric item types\n- 📝 **Documentation** - Improve guides, examples, and API docs\n- 🧪 **Tests** - Add or improve test coverage\n- 💬 **Help others** - Answer questions and provide support\n- 💡 **Feature suggestions** - Propose new capabilities\n\n## Prerequisites\n\nBefore you begin, ensure you have the following installed:\n\n- [Python](https://www.python.org/downloads/) (see [Installation](https://microsoft.github.io/fabric-cicd/#installation) for version requirements)\n- [Node.js and npm](https://nodejs.org/en/download/)\n- [PowerShell](https://docs.microsoft.com/en-us/powershell/scripting/install/installing-powershell)\n- [Azure CLI](https://learn.microsoft.com/en-us/cli/azure/install-azure-cli-windows) or [Az.Accounts PowerShell module](https://www.powershellgallery.com/packages/Az.Accounts/2.2.3)\n- [Visual Studio Code (VS Code)](https://code.visualstudio.com/)\n\n## Initial Configuration\n\n1. **Fork the Repository on GitHub**:\n - Go to the repository [fabric-cicd](https://github.com/microsoft/fabric-cicd) on GitHub\n - In the top right corner, click on the **Fork** button\n - This will create a copy of the repository in your own GitHub account\n\n1. **Clone Your Forked Repository**:\n - Once the fork is complete, go to your GitHub account and open the forked repository\n - Click on the **Code** button, and clone to VS Code\n\n1. **Run activate.ps1**:\n - Open the Project in VS Code\n - Open PowerShell terminal\n - Run `activate.ps1` which will install `uv`, and `ruff` if not already found. And set up the default environment leveraging `uv sync`\n ```powershell\n .\\activate.ps1\n ```\n _Note that this is technically optional and is designed to work with PowerShell. You can execute these steps manually as well, this is merely a helper_\n _For Linux, run `activate.sh` instead_\n\n1. **Select Python Interpreter**:\n - Open the Command Palette (`Ctrl+Shift+P`) and select `Python: Select Interpreter`\n - Choose the interpreter from the `.venv` directory\n\n1. **Ensure All VS Code Extensions Are Installed**:\n - Open the Command Palette (`Ctrl+Shift+P`) and select `Extensions: Show Recommended Extensions`\n - Install all extensions recommended for the workspace\n\n## Development\n\n### Managing Dependencies\n\n- All dependencies in this project are managed by `uv` which will resolve all dependencies and lock the versions to speed up virtual environment creation\n- For additions, run:\n ```sh\n uv add \n ```\n- For removals, run:\n ```sh\n uv remove \n ```\n\n### Code Formatting & Linting\n\n- The python code within this project is maintained by `ruff`\n- If you install the recommended extensions, `ruff` will auto format on save of any file\n- Before being able to merge a PR, `ruff` is ran in a GitHub Action to ensure the files are properly formatted and maintained\n- To force linting, run the following\n ```sh\n uv run ruff format\n uv run ruff check\n ```\n\n## Contribution process\n\nTo avoid cases where submitted PRs are rejected, please follow the following steps:\n\n- To report a new issue, follow [Create an issue](#creating-an-issue)\n- To work on existing issue, follow [Find an issue to work on](#finding-an-issue-to-work-on)\n- To contribute code, follow [Pull request process](#pull-request-process)\n\n### Creating an issue\n\nBefore reporting a new bug or suggesting a feature, please search the [GitHub Issues page](https://github.com/microsoft/fabric-cicd/issues) to check if one already exists.\n\nAll reported bugs or feature suggestions must start with creating an issue in the GitHub Issues pane. Please add as much information as possible to help us with triage and understanding. Once the issue is triaged, labels will be added to indicate its status (e.g., \"need more info\", \"help wanted\").\n\nWhen creating an issue please select the relevant template, e.g., bug, new feature, general question, etc. and provide all required input:\n\n- [Bug Report](https://github.com/microsoft/fabric-cicd/issues/new?template=1-bug.yml)\n- [Feature Request](https://github.com/microsoft/fabric-cicd/issues/new?template=2-feature.yml)\n- [Documentation](https://github.com/microsoft/fabric-cicd/issues/new?template=3-documentation.yml)\n- [Question](https://github.com/microsoft/fabric-cicd/issues/new?template=4-question.yml)\n\nWe aim to respond to new issues promptly, but response times may vary depending on workload and priority.\n\n### Finding an issue to work on\n\n#### For Beginners\n\nIf you're new to contributing, look for issues with these labels:\n\n- **`good-first-issue`** - Beginner-friendly tasks that are well-scoped and documented\n- **`help wanted`** - Issues where community contributions are especially welcome\n- **`documentation`** - Improve docs, examples, or help text (great for first contributions)\n\n#### Getting Started Tips\n\n1. **Start small** - Look for typo fixes, documentation improvements, or simple bug fixes\n2. **Read existing code** - Familiarize yourself with the codebase by exploring similar commands\n3. **Ask questions** - Comment on issues to clarify requirements or get guidance\n4. **Test locally** - Always test your changes thoroughly before submitting\n\n#### Before You Code\n\nAll PRs must be linked with a \"help wanted\" issue. To avoid rework after investing effort:\n\n1. **Comment on the issue** - Express interest and describe your planned approach\n2. **Wait for acknowledgment** - Get team confirmation before starting significant work\n3. **Ask for clarification** - Don't hesitate to ask questions about requirements\n\nPlease review [engineering guidelines](https://github.com/microsoft/fabric-cicd/wiki) for coding guidelines and common flows to help you with your task.\n\n### Pull request process\n\n**All pull requests must be linked to an approved issue,** see [PR Title Format](#pr-title-format). This ensures proper tracking and context for changes. Before creating a pull request:\n\n1. **Create or identify an existing issue** that describes the problem, feature request, or change you're addressing\n2. **Comment on the issue** to express interest and get team acknowledgment before starting work\n\n#### PR Title Format\n\nYour PR title MUST follow this exact format: `\"Fixes #123 - Short Description\"` where #123 is the issue number.\n\n- Use \"Fixes\" for bug fixes, \"Closes\" for features, \"Resolves\" for other changes\n- Example: \"Fixes #520 - Add Python version requirements to documentation\"\n- Version bump PRs are an exception: title must be \"vX.X.X\" format only\n- GitHub Actions will automatically check that your PR is linked to a valid issue and will fail if no valid reference is found\n\n#### Before Submitting PR\n\nVerify that:\n\n- The PR is focused on the related task\n- Tests coverage is kept and all tests pass\n- Your code is aligned with the code conventions of this project\n\n#### Review Process\n\n- Use a descriptive title and provide a clear summary of your changes\n- Address and resolve all review comments before merge\n- PRs will be labeled as \"need author feedback\" when there are comments to resolve\n- Approved PRs will be merged by the fabric-cicd team\n\n### Documenting Changes with Changie\n\nAll pull requests must include proper change documentation using [changie](https://changie.dev), which is pre-installed in the development container. This ensures that release notes are automatically generated and changes are properly tracked.\n\n#### Requirements\n\n**Every PR must include at least one change entry** created using `changie new`. You may add multiple entries if your PR introduces multiple distinct changes.\n\n#### How to Add Change Entries\n\n1. **From the Terminal, run `changie new` command**:\n\n ```bash\n changie new\n ```\n\n2. **Select the appropriate change type** from the available options:\n - **⚠️ Breaking Change** - For changes that break backward compatibility\n - **🆕 New Items Support** - For adding support for new Fabric item types\n - **✨ New Functionality** - For new features, commands, or capabilities\n - **🔧 Bug Fix** - For fixing existing issues or incorrect behavior\n - **⚡ Additional Optimizations** - For performance improvements or optimizations\n - **📝 Documentation Update** - For documentation improvements or updates\n\n3. **Provide a clear description** of your change:\n - Write in present tense (e.g., \"Add support for...\" not \"Added support for...\")\n - Be specific and user-focused\n - Include the affected command or feature if applicable\n - Keep it concise but informative\n\n#### Examples of Good Change Descriptions\n\n- `Fix timeout issue in LRO polling`\n- `Update workspace examples with new folder hierarchy patterns`\n- `Optimize API response caching to reduce network calls`\n\n#### Guidelines\n\n- **One logical change per entry**: If your PR fixes a bug and adds a feature, create two separate entries\n- **User-facing perspective**: Describe what users will experience, not internal implementation details\n- **Clear and actionable**: Users should understand what changed and how it affects them\n- **Consistent formatting**: Follow the examples and existing patterns in the changelog\n\nThe change entries will be automatically included in the release notes when a new version is published. This process ensures that all improvements, fixes, and new features are properly communicated to users.\n\n## Resources to help you get started\n\nHere are some resources to help you get started:\n\n- A good place to start learning about fabric-cicd is the [fabric-cicd documentation](https://microsoft.github.io/fabric-cicd/)\n- If you want to contribute code, please check more details about coding guidelines, major code flows and code building block in [Engineering guidelines](https://github.com/microsoft/fabric-cicd/wiki)\n\n## Engineering guidelines\n\nFor detailed engineering guidelines please refer to our [Wiki pages](https://github.com/microsoft/fabric-cicd/wiki).\n\nThe Wiki contains essential information and requirements for contributors, including: Code Style and Standards, Architecture Overview, Testing and more.\n\nBefore contributing code, please review these guidelines to ensure your contributions align with the project's standards and practices.\n\n## Areas with Restricted Contributions\n\nSome areas require special consideration:\n\n- **Core infrastructure** - Major architectural changes require team discussion, including within `FabricEndpoint` and `FabricWorkspace` classes\n- **Parameterization framework** - Changes require team discussion due to complex validation and parameter replacement logic\n\n## Need Help?\n\n### Getting Support\n\n- **[GitHub Issues](https://github.com/microsoft/fabric-cicd/issues)** - Report specific problems\n- **[Documentation](https://microsoft.github.io/fabric-cicd/)** - Check comprehensive guides\n\n### Communication Guidelines\n\n- **Be patient** - Maintainers balance multiple responsibilities\n- **Be respectful** - Follow the code of conduct\n- **Be specific** - Provide clear, detailed information\n- **Be collaborative** - Work together to improve the project\n\nThank you for contributing to Microsoft fabric-cicd! Your contributions help make this tool better for the entire Fabric community.\n" }, { "path": "CodeQL.yml", "content": "path_classifiers:\n tests:\n - \"devtools/*.py\"\n" }, { "path": "LICENSE", "content": " MIT License\n\n Copyright (c) Microsoft Corporation.\n\n Permission is hereby granted, free of charge, to any person obtaining a copy\n of this software and associated documentation files (the \"Software\"), to deal\n in the Software without restriction, including without limitation the rights\n to use, copy, modify, merge, publish, distribute, sublicense, and/or sell\n copies of the Software, and to permit persons to whom the Software is\n furnished to do so, subject to the following conditions:\n\n The above copyright notice and this permission notice shall be included in all\n copies or substantial portions of the Software.\n\n THE SOFTWARE IS PROVIDED \"AS IS\", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR\n IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,\n FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE\n AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER\n LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,\n OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE\n SOFTWARE\n" }, { "path": "README.md", "content": "# Fabric CICD\n\n[![Language](https://img.shields.io/badge/language-Python-blue.svg)](https://www.python.org/)\n[![PyPi version](https://badgen.net/pypi/v/fabric-cicd/)](https://pypi.org/project/fabric-cicd)\n[![Python version](https://img.shields.io/pypi/pyversions/fabric-cicd)](https://pypi.org/project/fabric-cicd)\n[![Code style: ruff](https://img.shields.io/badge/code%20style-ruff-000000.svg)](https://github.com/charliermarsh/ruff)\n[![Tests](https://img.shields.io/github/actions/workflow/status/microsoft/fabric-cicd/test.yml?logo=github&label=tests&branch=main)](https://github.com/microsoft/fabric-cicd/actions/workflows/test.yml)\n\n---\n\n## Project Overview\n\nfabric-cicd is a Python library designed for use with [Microsoft Fabric](https://learn.microsoft.com/en-us/fabric/) workspaces. This library supports code-first Continuous Integration / Continuous Deployment (CI/CD) automations to seamlessly integrate Source Controlled workspaces into a deployment framework. The goal is to assist CI/CD developers who prefer not to interact directly with the Microsoft Fabric APIs.\n\n## Documentation\n\nAll documentation is hosted on our [fabric-cicd](https://microsoft.github.io/fabric-cicd/) GitHub Pages\n\nSection Overview:\n\n- [Home](https://microsoft.github.io/fabric-cicd/latest/)\n- [How To](https://microsoft.github.io/fabric-cicd/latest/how_to/)\n- [Examples](https://microsoft.github.io/fabric-cicd/latest/example/)\n- [Contribution](https://microsoft.github.io/fabric-cicd/latest/contribution/)\n- [Changelog](https://microsoft.github.io/fabric-cicd/latest/changelog/)\n- [About](https://microsoft.github.io/fabric-cicd/latest/help/) - Inclusive of Support & Security Policies\n\n## Installation\n\nTo install fabric-cicd, run:\n\n```bash\npip install fabric-cicd\n```\n\n## Trademarks\n\nThis project may contain trademarks or logos for projects, products, or services. Authorized use of Microsoft trademarks or logos is subject to and must follow [Microsoft's Trademark & Brand Guidelines](https://www.microsoft.com/en-us/legal/intellectualproperty/trademarks/usage/general). Use of Microsoft trademarks or logos in modified versions of this project must not cause confusion or imply Microsoft sponsorship. Any use of third-party trademarks or logos are subject to those third-party's policies.\n" }, { "path": "SECURITY.md", "content": "\n\n## Security\n\nMicrosoft takes the security of our software products and services seriously, which includes all source code repositories managed through our GitHub organizations, which include [Microsoft](https://github.com/Microsoft), [Azure](https://github.com/Azure), [DotNet](https://github.com/dotnet), [AspNet](https://github.com/aspnet) and [Xamarin](https://github.com/xamarin).\n\nIf you believe you have found a security vulnerability in any Microsoft-owned repository that meets [Microsoft's definition of a security vulnerability](https://aka.ms/security.md/definition), please report it to us as described below.\n\n## Reporting Security Issues\n\n**Please do not report security vulnerabilities through public GitHub issues.**\n\nInstead, please report them to the Microsoft Security Response Center (MSRC) at [https://msrc.microsoft.com/create-report](https://aka.ms/security.md/msrc/create-report).\n\nIf you prefer to submit without logging in, send email to [secure@microsoft.com](mailto:secure@microsoft.com). If possible, encrypt your message with our PGP key; please download it from the [Microsoft Security Response Center PGP Key page](https://aka.ms/security.md/msrc/pgp).\n\nYou should receive a response within 24 hours. If for some reason you do not, please follow up via email to ensure we received your original message. Additional information can be found at [microsoft.com/msrc](https://www.microsoft.com/msrc).\n\nPlease include the requested information listed below (as much as you can provide) to help us better understand the nature and scope of the possible issue:\n\n- Type of issue (e.g. buffer overflow, SQL injection, cross-site scripting, etc.)\n- Full paths of source file(s) related to the manifestation of the issue\n- The location of the affected source code (tag/branch/commit or direct URL)\n- Any special configuration required to reproduce the issue\n- Step-by-step instructions to reproduce the issue\n- Proof-of-concept or exploit code (if possible)\n- Impact of the issue, including how an attacker might exploit the issue\n\nThis information will help us triage your report more quickly.\n\nIf you are reporting for a bug bounty, more complete reports can contribute to a higher bounty award. Please visit our [Microsoft Bug Bounty Program](https://aka.ms/security.md/msrc/bounty) page for more details about our active programs.\n\n## Preferred Languages\n\nWe prefer all communications to be in English.\n\n## Policy\n\nMicrosoft follows the principle of [Coordinated Vulnerability Disclosure](https://aka.ms/security.md/cvd).\n\n\n" }, { "path": "activate.ps1", "content": "# Copyright (c) Microsoft Corporation.\n# Licensed under the MIT License.\n\n<#\n.SYNOPSIS\nScript to check and install required Python packages, Node.js tools, add directories to PATH, and activate a virtual environment.\n\n.DESCRIPTION\nThis script performs the following tasks:\n1. Checks if Python is installed.\n2. Checks if pip is installed.\n3. Checks if Node.js and npm are installed.\n4. Checks and installs specified Python packages if they are not already installed.\n5. Installs changie globally via npm if not already installed.\n6. Adds a specified directory to the system PATH if it is not already included.\n7. Ensures the 'uv' command is available in the PATH.\n8. Activates a virtual environment using 'uv'.\n\n.NOTES\nMake sure that Python, pip, Node.js, and npm are installed and available in the system PATH.\n#>\n\n# Function to check if a dependency is available\nfunction Test-Dependancy {\n param (\n [string]$commandName\n )\n\n if (-not (Get-Command $commandName -ErrorAction SilentlyContinue)) {\n Write-Host \" $commandName is not installed or not in PATH. Please install $commandName and make sure it's available in the PATH.\"\n exit 1\n }\n else {\n $commandPath = (Get-Command $commandName).Path\n $commandDirectory = [System.IO.Path]::GetDirectoryName($commandPath)\n Write-Host \"$commandName is installed in $commandPath\" \n Add-DirectoryToPath -directory $commandDirectory\n }\n}\n\n# Function to install required packages if not already installed\nfunction Test-And-Install-Python-Package {\n param (\n [string]$packageName\n )\n\n pip show $packageName -q\n if ($LASTEXITCODE -ne 0) {\n Write-Host \"$packageName is not installed. Installing $packageName...\"\n try {\n pip install $packageName\n Write-Host \"$packageName installed successfully.\"\n }\n catch {\n Write-Host \"Failed to install $packageName. Please check your pip installation.\"\n exit 1\n }\n }\n else {\n Write-Host \"$packageName is already installed.\"\n }\n}\n\n# Function to install changie globally via npm if not already installed\nfunction Test-And-Install-Changie {\n if (-not (Get-Command changie -ErrorAction SilentlyContinue)) {\n Write-Host \"changie not found, installing globally via npm...\"\n try {\n npm install -g changie --registry https://registry.npmjs.org/\n \n # Add npm global bin to PATH if needed\n $npmGlobalPath = npm config get prefix\n if ($npmGlobalPath) {\n Add-DirectoryToPath -directory $npmGlobalPath\n }\n \n # Refresh PATH for the current session\n $env:Path = [System.Environment]::GetEnvironmentVariable(\"Path\",\"Machine\") + \";\" + [System.Environment]::GetEnvironmentVariable(\"Path\",\"User\")\n \n Test-Dependancy -commandName \"changie\"\n }\n catch {\n Write-Host \"Failed to install changie via npm. Please check your npm installation and connection.\"\n exit 1\n }\n }\n else {\n Write-Host \"changie is already installed.\"\n }\n}\n\n# Function to add a directory to PATH\nfunction Add-DirectoryToPath {\n param (\n [string]$directory\n )\n\n if (-not ($env:Path -split ';' | ForEach-Object { $_.Trim() } | Where-Object { $_ -eq $directory })) {\n $env:Path += \";$directory\"\n Write-Host \"Added $directory to PATH.\"\n }\n}\n\n# Check if dependencies are installed and add directory to PATH\nTest-Dependancy -commandName \"python\"\nTest-Dependancy -commandName \"pip\"\nTest-Dependancy -commandName \"node\"\nTest-Dependancy -commandName \"npm\"\n# Check and install required packages\nTest-And-Install-Python-Package -packageName \"uv\"\nTest-And-Install-Python-Package -packageName \"ruff\"\nTest-And-Install-Changie\n\n\n# uv fallback to default path if unavailable in python directory\nif (-not (Get-Command uv -ErrorAction SilentlyContinue)) {\n Write-Host \"uv is not recognized. Attempting to add uv to PATH...\"\n $localBinPath = [System.IO.Path]::Combine($env:USERPROFILE, '.local', 'bin')\n Add-DirectoryToPath -directory $localBinPath\n Test-Dependancy -commandName \"uv\"\n}\n\n# Activate the environment\nuv sync --python 3.11\n$venvPath = \".venv\\Scripts\\activate.ps1\"\n\nif (Test-Path $venvPath) {\n & $venvPath\n Write-Host \"venv activated\"\n}\nelse {\n Write-Host \"venv not found\"\n}\n\nWrite-Host \"\"\nWrite-Host \"To deactivate the environment, run \" -NoNewline\nWrite-Host \"deactivate\" -ForegroundColor Green\n" }, { "path": "activate.sh", "content": "#!/bin/bash\n#\n#\n# Script to check and install required Python packages, Node.js tools,\n# add directories to PATH, and activate a virtual environment.\n#\n# ---------------------------------------------------------------------------------------\n#\nset -e\n\nPACKAGES=\"\"\nif ! command -v python3.11 &> /dev/null; then PACKAGES=\"python3.11\"; fi\nif ! command -v pip &> /dev/null; then PACKAGES=\"${PACKAGES:+$PACKAGES }python3-pip\"; fi\nif ! command -v node &> /dev/null; then PACKAGES=\"${PACKAGES:+$PACKAGES }nodejs\"; fi\nif ! command -v npm &> /dev/null; then PACKAGES=\"${PACKAGES:+$PACKAGES }npm\"; fi\nif [[ \"$OSTYPE\" == \"linux-gnu\"* ]]; then\n if [ -n \"$PACKAGES\" ]; then\n echo \"Installing required packages for Linux: $PACKAGES\"\n sudo apt-get update > /dev/null 2>&1\n if sudo DEBIAN_FRONTEND=noninteractive apt-get install -y $PACKAGES > /dev/null 2>&1; then\n echo \"Packages installed successfully.\"\n else\n echo \"Failed to install packages.\"\n exit 1\n fi\n fi\nelif [[ \"$OSTYPE\" == \"darwin\"* ]]; then\n if ! command -v brew &> /dev/null; then\n echo \"Homebrew not found. Please install Homebrew first.\"\n exit 1\n fi\n \n BREW_PACKAGES=\"\"\n if ! command -v python3.11 &> /dev/null; then BREW_PACKAGES=\"python@3.11\"; fi\n if ! command -v node &> /dev/null; then BREW_PACKAGES=\"${BREW_PACKAGES:+$BREW_PACKAGES }node\"; fi\n\n if [ -n \"$BREW_PACKAGES\" ]; then\n echo \"Installing required packages for macOS: $BREW_PACKAGES\"\n if brew install $BREW_PACKAGES; then\n echo \"Packages installed successfully.\"\n else\n echo \"Failed to install packages.\"\n exit 1\n fi\n fi\nfi\n\n# Install uv if not present\nif ! command -v uv &> /dev/null; then\n echo \"Installing uv...\"\n if curl -LsSf https://astral.sh/uv/install.sh | sh; then\n echo \"uv installed successfully.\"\n else\n echo \"Failed to install uv.\"\n exit 1\n fi\nelse\n echo \"uv is already installed.\"\nfi\n\n# Install changie globally via npm if not present\nif ! command -v changie &> /dev/null; then\n echo \"Installing changie globally via npm...\"\n if npm install -g changie; then\n echo \"changie installed successfully.\"\n else\n echo \"Failed to install changie.\"\n exit 1\n fi\nelse\n echo \"changie is already installed.\"\nfi\n\n# Install VS Code Python extension if VS Code is available\nif command -v code &> /dev/null; then\n echo \"Installing VS Code Python extension...\"\n if code --install-extension ms-python.python --force > /dev/null 2>&1; then\n echo \"VS Code Python extension installed successfully.\"\n else\n echo \"Failed to install VS Code Python extension.\"\n fi\nelse\n echo \"VS Code not found, skipping extension installation.\"\nfi\n\n# Add required directories to PATH\nif [[ \":$PATH:\" != *\":$HOME/.local/bin:\"* ]]; then\n export PATH=\"$PATH:$HOME/.local/bin\"\n echo \"Added $HOME/.local/bin to PATH.\"\nfi\nif [[ \":$PATH:\" != *\":$HOME/.cargo/bin:\"* ]]; then\n export PATH=\"$PATH:$HOME/.cargo/bin\"\n echo \"Added $HOME/.cargo/bin to PATH.\"\nfi\n\n# Sync Python environment and activate\necho \"Syncing Python environment with uv...\"\nif uv sync --python 3.11; then\n echo \"Python environment synced successfully.\"\nelse\n echo \"Failed to sync Python environment.\"\n exit 1\nfi\n\nif [ -f .venv/bin/activate ]; then\n source .venv/bin/activate\n echo \"Virtual environment activated.\"\nelse\n echo \"Virtual environment not found.\"\nfi\n" }, { "path": "devtools/debug_api.py", "content": "# Copyright (c) Microsoft Corporation.\n# Licensed under the MIT License.\n\n# The following is intended for developers of fabric-cicd to debug and call Fabric REST APIs locally from the github repo\n\nfrom azure.identity import AzureCliCredential, AzurePowerShellCredential, ClientSecretCredential\n\nfrom fabric_cicd import change_log_level, constants\nfrom fabric_cicd._common._fabric_endpoint import FabricEndpoint\nfrom fabric_cicd._common._validate_input import validate_token_credential\n\n# Uncomment to enable debug\n# change_log_level()\n\nif __name__ == \"__main__\":\n # Azure CLI auth - comment out to use a different auth method\n token_credential = AzureCliCredential()\n\n # Uncomment to use PowerShell auth\n # token_credential = AzurePowerShellCredential()\n\n # Uncomment to use SPN auth\n # client_id = \"your-client-id\"\n # client_secret = \"your-client-secret\"\n # tenant_id = \"your-tenant-id\"\n # token_credential = ClientSecretCredential(client_id=client_id, client_secret=client_secret, tenant_id=tenant_id)\n\n # Create endpoint object\n fe = FabricEndpoint(token_credential=validate_token_credential(token_credential))\n\n # Set workspace id variable if needed in API url\n workspace_id = \"8f5c0cec-a8ea-48cd-9da4-871dc2642f4c\"\n\n # API endpoint url (placeholder)\n api_url = f\"{constants.DEFAULT_API_ROOT_URL}/v1/workspaces/{workspace_id}...\"\n\n print(\"Making API call...\")\n response = fe.invoke(\n method=\"POST\",\n url=api_url,\n body={},\n )\n print(\"Call completed.\")\n" }, { "path": "devtools/debug_local config.py", "content": "# Copyright (c) Microsoft Corporation.\n# Licensed under the MIT License.\n\n# The following is intended for developers of fabric-cicd to debug locally against the github repo\n\nimport sys\nfrom pathlib import Path\n\nfrom azure.identity import AzureCliCredential, AzurePowerShellCredential, ClientSecretCredential\n\nroot_directory = Path(__file__).resolve().parent.parent\nsys.path.insert(0, str(root_directory / \"src\"))\n\nfrom fabric_cicd import append_feature_flag, change_log_level, deploy_with_config\n\n# Uncomment to enable debug\n# change_log_level()\n\n# In this example, the config file sits within the root/sample/workspace directory\nconfig_file = str(root_directory / \"sample\" / \"workspace\" / \"config.yml\")\n\n# Azure CLI auth - comment out to use a different auth method\ntoken_credential = AzureCliCredential()\n\n# Uncomment to use PowerShell auth\n# token_credential = AzurePowerShellCredential()\n\n# Uncomment to use SPN auth\n# client_id = \"your-client-id\"\n# client_secret = \"your-client-secret\"\n# tenant_id = \"your-tenant-id\"\n# token_credential = ClientSecretCredential(client_id=client_id, client_secret=client_secret, tenant_id=tenant_id)\n\n# config_override_dict = {\"core\": {\"item_types_in_scope\": [\"Notebook\"]}, \"publish\": {\"skip\": {\"dev\": False}}}\n\ndeploy_with_config(\n config_file_path=config_file,\n # Comment out if environment is not needed\n environment=\"dev\",\n # Explicit token credential required for auth (choose one of the options above)\n token_credential=token_credential,\n # Uncomment to override specific config values (pass in a dictionary of override values)\n # config_override=config_override_dict\n)\n" }, { "path": "devtools/debug_local.py", "content": "# Copyright (c) Microsoft Corporation.\n# Licensed under the MIT License.\n\n# The following is intended for developers of fabric-cicd to debug locally against the github repo\n\nimport sys\nfrom pathlib import Path\n\nfrom azure.identity import AzureCliCredential, AzurePowerShellCredential, ClientSecretCredential\n\nroot_directory = Path(__file__).resolve().parent.parent\nsys.path.insert(0, str(root_directory / \"src\"))\n\nfrom fabric_cicd import (\n FabricWorkspace,\n append_feature_flag,\n change_log_level,\n constants,\n publish_all_items,\n unpublish_all_orphan_items,\n)\n\n# Uncomment to enable debug\n# change_log_level()\n\n# Uncomment to add feature flag\nappend_feature_flag(\"enable_shortcut_publish\")\n\n# The defined environment values should match the names found in the parameter.yml file\nworkspace_id = \"8f5c0cec-a8ea-48cd-9da4-871dc2642f4c\"\nenvironment = \"PPE\"\n\n# In this example, our workspace content sits within the root/sample/workspace directory\nrepository_directory = str(root_directory / \"sample\" / \"workspace\")\n\n# Explicitly define which of the item types we want to deploy\nitem_type_in_scope = [\n \"Lakehouse\",\n \"VariableLibrary\",\n \"DataBuildToolJob\",\n \"Dataflow\",\n \"DataPipeline\",\n \"Notebook\",\n \"Environment\",\n \"SemanticModel\",\n \"Report\",\n \"Eventhouse\",\n \"KQLDatabase\",\n \"KQLQueryset\",\n \"Reflex\",\n \"Eventstream\",\n \"SparkJobDefinition\",\n \"Ontology\",\n]\n\n# Azure CLI auth - comment out to use a different auth method\ntoken_credential = AzureCliCredential()\n\n# Uncomment to use PowerShell auth\n# token_credential = AzurePowerShellCredential()\n\n# Uncomment to use SPN auth\n# client_id = \"your-client-id\"\n# client_secret = \"your-client-secret\"\n# tenant_id = \"your-tenant-id\"\n# token_credential = ClientSecretCredential(client_id=client_id, client_secret=client_secret, tenant_id=tenant_id)\n\nconstants.DEFAULT_API_ROOT_URL = \"https://msitapi.fabric.microsoft.com\"\n\n# Initialize the FabricWorkspace object with the required parameters\ntarget_workspace = FabricWorkspace(\n workspace_id=workspace_id,\n environment=environment,\n repository_directory=repository_directory,\n item_type_in_scope=item_type_in_scope,\n # Explicit token credential required for auth (choose one of the options above)\n token_credential=token_credential,\n)\n\n# Uncomment to publish\n# Publish all items defined in item_type_in_scope\n# publish_all_items(target_workspace)\n\n# Uncomment to unpublish\n# Unpublish all items defined in scope not found in repository\n# unpublish_all_orphan_items(target_workspace, item_name_exclude_regex=r\"^DEBUG.*\")\n" }, { "path": "devtools/debug_parameterization.py", "content": "# Copyright (c) Microsoft Corporation.\n# Licensed under the MIT License.\n\n# The following is intended for developers of fabric-cicd to debug parameter.yml file locally against the github repo\n\nimport sys\nfrom pathlib import Path\n\nimport fabric_cicd.constants as constants\nfrom fabric_cicd import change_log_level\nfrom fabric_cicd._parameter._utils import validate_parameter_file\n\nroot_directory = Path(__file__).resolve().parent.parent\nsys.path.insert(0, str(root_directory / \"src\"))\n\n# Uncomment to enable debug\n# change_log_level()\n\n# In this example, the parameter.yml file sits within the root/sample/workspace directory\nrepository_directory = str(root_directory / \"sample\" / \"workspace\")\n\n# Explicitly define valid item types\nitem_type_in_scope = [\"DataPipeline\", \"Notebook\", \"Environment\", \"SemanticModel\", \"Report\"]\n\n# Set target environment\nenvironment = \"PPE\"\n\n# Uncomment to use a parameter file in a different location (default location is within repository directory)\n# Use absolute path\n# parameter_file_path = str(root_directory / \"sample\" / \"config\" / \"parameter.yml\")\n# or use relative path\n# parameter_file_path = \"../config/parameter.yml\"\n\nvalidate_parameter_file(\n repository_directory=repository_directory,\n item_type_in_scope=item_type_in_scope,\n # Comment to exclude target environment in validation\n environment=environment,\n # Uncomment to use a different parameter file name within the repository directory (default name: parameter.yml)\n # Assign to the constant in constants.py or pass in a string directly\n # parameter_file_name=constants.PARAMETER_FILE_NAME,\n # Uncomment to use a parameter file from outside the repository (takes precedence over parameter_file_name)\n # parameter_file_path=parameter_file_path\n)\n" }, { "path": "devtools/debug_trace_deployment.py", "content": "# Copyright (c) Microsoft Corporation.\n# Licensed under the MIT License.\n\n# Captures route traces from a live Fabric workspace deployment into a JSON GZIP trace file.\n\nimport gzip\nimport os\nimport shutil\nimport sys\nfrom pathlib import Path\n\nfrom azure.identity import AzureCliCredential\n\nroot_directory = Path(__file__).resolve().parent.parent\nsys.path.insert(0, str(root_directory / \"src\"))\n\nimport fabric_cicd\n\n\ndef main():\n \"\"\"Capture HTTP trace while publishing all items to Fabric workspace.\"\"\"\n\n os.environ[\"FABRIC_CICD_HTTP_TRACE_ENABLED\"] = \"1\"\n os.environ[\"FABRIC_CICD_HTTP_TRACE_FILE\"] = str(root_directory / \"http_trace.json\")\n\n workspace_id = os.environ.get(\"FABRIC_WORKSPACE_ID\")\n if not workspace_id:\n msg = \"FABRIC_WORKSPACE_ID environment variable must be set\"\n raise ValueError(msg)\n\n environment = \"PPE\"\n repository_directory = str(root_directory / \"sample\" / \"workspace\")\n item_type_in_scope = [\n \"DataBuildToolJob\",\n \"Dataflow\",\n \"DataPipeline\",\n \"Environment\",\n \"Eventhouse\",\n \"Eventstream\",\n \"KQLDatabase\",\n \"KQLQueryset\",\n \"Lakehouse\",\n \"MirroredDatabase\",\n \"MLExperiment\",\n \"Notebook\",\n \"Ontology\",\n \"Reflex\",\n \"Report\",\n \"SemanticModel\",\n \"SparkJobDefinition\",\n \"SQLDatabase\",\n \"VariableLibrary\",\n \"Warehouse\",\n ]\n token_credential = AzureCliCredential()\n for flag in [\"enable_shortcut_publish\", \"continue_on_shortcut_failure\"]:\n fabric_cicd.append_feature_flag(flag)\n target_workspace = fabric_cicd.FabricWorkspace(\n workspace_id=workspace_id,\n environment=environment,\n repository_directory=str(repository_directory),\n item_type_in_scope=item_type_in_scope,\n token_credential=token_credential,\n )\n fabric_cicd.publish_all_items(target_workspace)\n\n print(\"Publish completed successfully\")\n\n # The raw JSON trace file is very large; GZIP compress it generate a compact version\n # that can be used by tests. The raw trace file is still left in place for\n # debugging purposes.\n #\n trace_file = root_directory / \"http_trace.json\"\n compressed_file = root_directory / \"http_trace.json.gz\"\n with trace_file.open(\"rb\") as f_in, gzip.open(compressed_file, \"wb\") as f_out:\n shutil.copyfileobj(f_in, f_out)\n print(f\"Compressed trace file to {compressed_file}\")\n\n\nif __name__ == \"__main__\":\n main()\n" }, { "path": "devtools/pypi_build_release_dev.ps1", "content": "# Copyright (c) Microsoft Corporation.\n# Licensed under the MIT License.\n\nRemove-Item -Recurse -Force dist/*\n\npython -m build\n\npython -m twine upload --repository testpypi dist/*\n\npip install --upgrade --index-url https://test.pypi.org/simple/ fabric-cicd[dev]\n" }, { "path": "docs/about.md", "content": "# About\n\n## Security\n\n{%\n include-markdown \"../SECURITY.md\"\n start=\"## Security\"\n heading-offset=1\n%}\n\n## License\n\n{% include \"../LICENSE\" %}\n\n## Get help\n\nThis project uses [GitHub Issues](https://github.com/microsoft/fabric-cicd/issues) to track bugs, feature requests, and questions. Please search the existing issues before filing new issues to avoid duplicates. For new issues, file your [bug](https://github.com/microsoft/fabric-cicd/issues/new?template=1-bug.yml), [feature request](https://github.com/microsoft/fabric-cicd/issues/new?template=2-feature.yml), or [question](https://github.com/microsoft/fabric-cicd/issues/new?template=4-question.yml) as a new issue.\n\n**Found a bug or have a suggestion?**\n\n- [Raise a GitHub issue](https://github.com/microsoft/fabric-cicd/issues)\n- [Submit ideas to Fabric Ideas Portal](https://ideas.fabric.microsoft.com/)\n\n**Need help using the fabric-cicd library?**\n\n- For debugging information, including how to use the error log file and debug scripts, see the [Troubleshooting Guide](how_to/troubleshooting.md).\n- Connect with the community on [r/MicrosoftFabric](https://www.reddit.com/r/MicrosoftFabric/)\n- Join the [Microsoft Developer Community](https://community.fabric.microsoft.com/t5/Developer/bd-p/Developer)\n\n**Need enterprise assistance?**\n\n- Contact your Microsoft account manager or:\n- Open a ticket with the [Fabric Support Team](https://support.fabric.microsoft.com/)\n" }, { "path": "docs/changelog.md", "content": "# Changelog\n\n## [v1.0.0](https://pypi.org/project/fabric-cicd/1.0.0) - April 20, 2026\n\n### ⚠️ Breaking Change\n\n- Remove default credential fallback; explicit token credential is now required by [shirasassoon](https://github.com/shirasassoon) ([#909](https://github.com/microsoft/fabric-cicd/issues/909))\n- Remove implicit authentication in Microsoft Fabric Notebook and identity logging; require explicit token credential and keyword-only arguments for `FabricWorkspace` and `deploy_with_config` by [shirasassoon](https://github.com/shirasassoon) ([#930](https://github.com/microsoft/fabric-cicd/issues/930))\n\n### 🆕 New Items Support\n\n- Add support for Ontology item type by [shirasassoon](https://github.com/shirasassoon) ([#796](https://github.com/microsoft/fabric-cicd/issues/796))\n\n### ✨ New Functionality\n\n- Improve transparency of `deploy_with_config` function in success and failure scenarios by [shirasassoon](https://github.com/shirasassoon) ([#695](https://github.com/microsoft/fabric-cicd/issues/695))\n- Extend API response collection to unpublish operations by [shirasassoon](https://github.com/shirasassoon) ([#877](https://github.com/microsoft/fabric-cicd/issues/877))\n- Add `get_changed_items()` utility function to detect Fabric items changed via git diff for use with selective deployment by [vipulb91](https://github.com/vipulb91) ([#865](https://github.com/microsoft/fabric-cicd/issues/865))\n\n### 🔧 Bug Fix\n\n- Prevent unintended GUID replacements in Variable Library item files during publish by [shirasassoon](https://github.com/shirasassoon) ([#884](https://github.com/microsoft/fabric-cicd/issues/884))\n- Fix YAML content check to reject notebook and other non-YAML files during `key_value_replace` parameterization, preventing file corruption by [shirasassoon](https://github.com/shirasassoon) ([#890](https://github.com/microsoft/fabric-cicd/issues/890))\n- Fix Notebook deployment failure caused by non-deterministic ordering of definition files in API payload by [shirasassoon](https://github.com/shirasassoon) ([#869](https://github.com/microsoft/fabric-cicd/issues/869))\n- Ignore parameter file when not explicitly defined in config file by [aviatco](https://github.com/aviatco) ([#866](https://github.com/microsoft/fabric-cicd/issues/866))\n- Add `enable_hard_delete` feature flag to bypass workspace recycle bin during unpublish by [shirasassoon](https://github.com/shirasassoon) ([#924](https://github.com/microsoft/fabric-cicd/issues/924))\n- Add timeout for long-running operation polling by [shirasassoon](https://github.com/shirasassoon) ([#919](https://github.com/microsoft/fabric-cicd/issues/919))\n\n## [v0.3.1](https://pypi.org/project/fabric-cicd/0.3.1) - March 12, 2026\n\n### 🔧 Bug Fix\n\n- Fix override behavior of feature flags and constants in config deployment by [shirasassoon](https://github.com/shirasassoon) ([#872](https://github.com/microsoft/fabric-cicd/issues/872))\n\n## [v0.3.0](https://pypi.org/project/fabric-cicd/0.3.0) - March 09, 2026\n\n### ✨ New Functionality\n\n- Support selective folder deployment using inclusion list by [shirasassoon](https://github.com/shirasassoon) ([#757](https://github.com/microsoft/fabric-cicd/issues/757))\n- Add FABRIC_CICD_VERSION_CHECK_DISABLED environment variable to disable version check on startup by [Ricapar](https://github.com/Ricapar) ([#811](https://github.com/microsoft/fabric-cicd/issues/811))\n- Add enhanced logging configuration options via public functions by [shirasassoon](https://github.com/shirasassoon) ([#842](https://github.com/microsoft/fabric-cicd/issues/842))\n- Add deployment support for Notebook items with `.ipynb` file format by [shirasassoon](https://github.com/shirasassoon) ([#850](https://github.com/microsoft/fabric-cicd/issues/850))\n\n### 🔧 Bug Fix\n\n- Add max workers soft cap with override and handle garbage response from Fabric Data Plane by [mdrakiburrahman](https://github.com/mdrakiburrahman) ([#827](https://github.com/microsoft/fabric-cicd/issues/827))\n- Fix parameter validation failure for item names with accented characters by [shirasassoon](https://github.com/shirasassoon) ([#818](https://github.com/microsoft/fabric-cicd/issues/818))\n- Exclude items with placeholder logical ID from duplicate logical ID check by [shirasassoon](https://github.com/shirasassoon) ([#843](https://github.com/microsoft/fabric-cicd/issues/843))\n\n### ⚡ Additional Optimizations\n\n- Add return value to deploy_with_config by [ayeshurun](https://github.com/ayeshurun) ([#851](https://github.com/microsoft/fabric-cicd/issues/851))\n- Add support for Python 3.13 in the library by [shirasassoon](https://github.com/shirasassoon) ([#855](https://github.com/microsoft/fabric-cicd/issues/855))\n\n### 📝 Documentation Update\n\n- Remove incorrect statement on support for parameter replacements in Platform files by [shirasassoon](https://github.com/shirasassoon) ([#839](https://github.com/microsoft/fabric-cicd/issues/839))\n\n## [v0.2.0](https://pypi.org/project/fabric-cicd/0.2.0) - February 16, 2026\n\n### ✨ New Functionality\n\n- Support parallelize deployments within a given item type by [mdrakiburrahman](https://github.com/mdrakiburrahman) ([#719](https://github.com/microsoft/fabric-cicd/issues/719))\n- Add a black-box REST API testing harness by [mdrakiburrahman](https://github.com/mdrakiburrahman) ([#738](https://github.com/microsoft/fabric-cicd/issues/738))\n- Change header print messages to info log by [mwc360](https://github.com/mwc360) ([#771](https://github.com/microsoft/fabric-cicd/issues/771))\n- Add support for semantic model binding per environment by [shirasassoon](https://github.com/shirasassoon) ([#689](https://github.com/microsoft/fabric-cicd/issues/689))\n\n### 🔧 Bug Fix\n\n- Remove OrgApp item type support by [shirasassoon](https://github.com/shirasassoon) ([#758](https://github.com/microsoft/fabric-cicd/issues/758))\n- Improve environment-mapping behavior in optional config fields by [shirasassoon](https://github.com/shirasassoon) ([#716](https://github.com/microsoft/fabric-cicd/issues/716))\n- Fix duplicate YAML key detection in parameter validation by [shirasassoon](https://github.com/shirasassoon) ([#752](https://github.com/microsoft/fabric-cicd/issues/752))\n- Add caching for item attribute lookups by [MiSchroe](https://github.com/MiSchroe) ([#704](https://github.com/microsoft/fabric-cicd/issues/704))\n\n### ⚡ Additional Optimizations\n\n- Enable configuration-based deployment without feature flags by [shirasassoon](https://github.com/shirasassoon) ([#805](https://github.com/microsoft/fabric-cicd/issues/805))\n\n### 📝 Documentation Update\n\n- Fix troubleshooting docs by [shirasassoon](https://github.com/shirasassoon) ([#747](https://github.com/microsoft/fabric-cicd/issues/747))\n\n## [v0.1.34](https://pypi.org/project/fabric-cicd/0.1.34) - January 20, 2026\n\n### ✨ New Functionality\n\n- Enable dynamic replacement of SQL endpoint values from SQL Database items ([#720](https://github.com/microsoft/fabric-cicd/issues/720))\n- Support Fabric Notebook Authentication ([#707](https://github.com/microsoft/fabric-cicd/issues/707))\n\n### 🆕 New Items Support\n\n- Onboard Spark Job Definition item type ([#115](https://github.com/microsoft/fabric-cicd/issues/115))\n\n### 📝 Documentation Update\n\n- Add `CONTRIBUTING.md` file to repository ([#723](https://github.com/microsoft/fabric-cicd/issues/723))\n- Add comprehensive troubleshooting guide to documentation ([#705](https://github.com/microsoft/fabric-cicd/issues/705))\n- Add parameterization documentation for Report items using ByConnection binding to Semantic Models ([#637](https://github.com/microsoft/fabric-cicd/issues/637))\n\n### ⚡ Additional Optimizations\n\n- Add debug file for local Fabric REST API testing ([#714](https://github.com/microsoft/fabric-cicd/issues/714))\n\n## [v0.1.33](https://pypi.org/project/fabric-cicd/0.1.33) - December 16, 2025\n\n### ✨ New Functionality\n\n- Add key_value_replace parameter support for YAML files ([#649](https://github.com/microsoft/fabric-cicd/issues/649))\n- Support selective shortcut publishing with regex exclusion ([#624](https://github.com/microsoft/fabric-cicd/issues/624))\n\n### ⚡ Additional Optimizations\n\n- Add Linux development environment bootstrapping script ([#680](https://github.com/microsoft/fabric-cicd/issues/680))\n- Update item types in scope to be an optional parameter in validate parameter file function ([#669](https://github.com/microsoft/fabric-cicd/issues/669))\n\n### 🔧 Bug Fix\n\n- Fix publish order for Notebook and Eventhouse dependent items ([#685](https://github.com/microsoft/fabric-cicd/issues/685))\n- Enable parameterizing multiple connections in the same Semantic Model item ([#674](https://github.com/microsoft/fabric-cicd/issues/674))\n- Fix missing description metadata in item payload for shell-only item deployments ([#672](https://github.com/microsoft/fabric-cicd/issues/672))\n- Resolve API long running operation handling when publishing Environment items ([#668](https://github.com/microsoft/fabric-cicd/issues/668))\n\n## [v0.1.32](https://pypi.org/project/fabric-cicd/0.1.32) - December 03, 2025\n\n### 🔧 Bug Fix\n\n- Fix publish bug for Environment items that contain only spark settings ([#664](https://github.com/microsoft/fabric-cicd/issues/664))\n\n## [v0.1.31](https://pypi.org/project/fabric-cicd/0.1.31) - December 01, 2025\n\n### ⚠️ Breaking Change\n\n- Migrate to the latest Fabric Environment item APIs to simplify deployment and improve compatibility ([#173](https://github.com/microsoft/fabric-cicd/issues/173))\n\n### ✨ New Functionality\n\n- Enable dynamic replacement of Lakehouse SQL Endpoint IDs ([#616](https://github.com/microsoft/fabric-cicd/issues/616))\n- Enable linking of Semantic Models to both cloud and gateway connections ([#602](https://github.com/microsoft/fabric-cicd/issues/602))\n- Allow use of the dynamic replacement variables within the key_value_replace parameter ([#567](https://github.com/microsoft/fabric-cicd/issues/567))\n- Add support for parameter file templates ([#499](https://github.com/microsoft/fabric-cicd/issues/499))\n\n### 🆕 New Items Support\n\n- Add support for the ML Experiment item type ([#600](https://github.com/microsoft/fabric-cicd/issues/600))\n- Add support for the User Data Function item type ([#588](https://github.com/microsoft/fabric-cicd/issues/588))\n\n### 📝 Documentation Update\n\n- Update the advanced Dataflow parameterization example with the correct file_path value ([#633](https://github.com/microsoft/fabric-cicd/issues/633))\n\n### 🔧 Bug Fix\n\n- Fix publishing issues for KQL Database items in folders ([#657](https://github.com/microsoft/fabric-cicd/issues/657))\n- Separate logic for 'items to include' feature between publish and unpublish operations ([#650](https://github.com/microsoft/fabric-cicd/issues/650))\n- Fix parameterization logic to properly handle find_value regex patterns and replacements ([#639](https://github.com/microsoft/fabric-cicd/issues/639))\n- Correct the publish order of Data Agent and Semantic Model items ([#628](https://github.com/microsoft/fabric-cicd/issues/628))\n- Fix Lakehouse item publishing errors when shortcuts refer to the default Lakehouse ID ([#610](https://github.com/microsoft/fabric-cicd/issues/610))\n\n## [v0.1.30](https://pypi.org/project/fabric-cicd/0.1.30) - October 20, 2025\n\n### ✨ New Functionality\n\n- Add support for binding semantic models to on-premise gateways in Fabric workspaces ([#569](https://github.com/microsoft/fabric-cicd/issues/569))\n\n### 🆕 New Items Support\n\n- Add support for publishing and managing Data Agent items ([#556](https://github.com/microsoft/fabric-cicd/issues/556))\n- Add OrgApp item type support ([#586](https://github.com/microsoft/fabric-cicd/issues/586))\n\n### ⚡ Additional Optimizations\n\n- Enhance cross-workspace variable support to allow referencing other attributes ([#583](https://github.com/microsoft/fabric-cicd/issues/583))\n\n### 🔧 Bug Fix\n\n- Fix workspace name extraction bug for non-ID attrs using ITEM_ATTR_LOOKUP ([#583](https://github.com/microsoft/fabric-cicd/issues/583))\n- Fix capacity requirement check ([#593](https://github.com/microsoft/fabric-cicd/issues/593))\n\n## [v0.1.29](https://pypi.org/project/fabric-cicd/0.1.29) - October 01, 2025\n\n### ✨ New Functionality\n\n- Support dynamic replacement for cross-workspace item IDs ([#558](https://github.com/microsoft/fabric-cicd/issues/558))\n- Add option to return API response for publish operations in publish_all_items ([#497](https://github.com/microsoft/fabric-cicd/issues/497))\n\n### 🆕 New Items Support\n\n- Onboard Apache Airflow Job item type ([#565](https://github.com/microsoft/fabric-cicd/issues/565))\n- Onboard Mounted Data Factory item type ([#406](https://github.com/microsoft/fabric-cicd/issues/406))\n\n### 🔧 Bug Fix\n\n- Fix publish order of Eventhouses and Semantic Models ([#566](https://github.com/microsoft/fabric-cicd/issues/566))\n\n## [v0.1.28](https://pypi.org/project/fabric-cicd/0.1.28) - September 15, 2025\n\n### ✨ New Functionality\n\n- Add folder exclusion feature for publish operations ([#427](https://github.com/microsoft/fabric-cicd/issues/427))\n- Expand workspace ID dynamic replacement capabilities in parameterization ([#408](https://github.com/microsoft/fabric-cicd/issues/408))\n\n### 🔧 Bug Fix\n\n- Fix unexpected behavior with file_path parameter filter ([#545](https://github.com/microsoft/fabric-cicd/issues/545))\n- Fix unpublish exclude_regex bug in configuration file-based deployment ([#544](https://github.com/microsoft/fabric-cicd/issues/544))\n\n## [v0.1.27](https://pypi.org/project/fabric-cicd/0.1.27) - September 05, 2025\n\n### 🔧 Bug Fix\n\n- Fix trailing comma in report schema ([#534](https://github.com/microsoft/fabric-cicd/issues/534))\n\n## [v0.1.26](https://pypi.org/project/fabric-cicd/0.1.26) - September 05, 2025\n\n### ⚠️ Breaking Change\n\n- Deprecate Base API URL kwarg in Fabric Workspace ([#529](https://github.com/microsoft/fabric-cicd/issues/529))\n\n### ✨ New Functionality\n\n- Support Schedules parameterization ([#508](https://github.com/microsoft/fabric-cicd/issues/508))\n- Support YAML configuration file-based deployment ([#470](https://github.com/microsoft/fabric-cicd/issues/470))\n\n### 📝 Documentation Update\n\n- Add dynamically generated Python version requirements to documentation ([#520](https://github.com/microsoft/fabric-cicd/issues/520))\n\n### ⚡ Additional Optimizations\n\n- Enhance pytest output to limit console verbosity ([#514](https://github.com/microsoft/fabric-cicd/issues/514))\n\n### 🔧 Bug Fix\n\n- Fix Report item schema handling ([#518](https://github.com/microsoft/fabric-cicd/issues/518))\n- Fix deployment order to publish Mirrored Database before Lakehouse ([#482](https://github.com/microsoft/fabric-cicd/issues/482))\n\n## [v0.1.25](https://pypi.org/project/fabric-cicd/0.1.25) - August 19, 2025\n\n### ⚠️ Breaking Change\n\n- Modify the default for item_types_in_scope and add thorough validation ([#464](https://github.com/microsoft/fabric-cicd/issues/464))\n\n### ✨ New Functionality\n\n- Add new experimental feature flag to enable selective deployment ([#384](https://github.com/microsoft/fabric-cicd/issues/384))\n- Support \"ALL\" environment concept in parameterization ([#320](https://github.com/microsoft/fabric-cicd/issues/320))\n\n### 📝 Documentation Update\n\n- Enhance Overview section in Parameterization docs ([#495](https://github.com/microsoft/fabric-cicd/issues/495))\n\n### ⚡ Additional Optimizations\n\n- Eliminate ACCEPTED_ITEM_TYPES_NON_UPN constant and unify with ACCEPTED_ITEM_TYPES ([#477](https://github.com/microsoft/fabric-cicd/issues/477))\n- Add comprehensive GitHub Copilot instructions for effective codebase development ([#468](https://github.com/microsoft/fabric-cicd/issues/468))\n\n### 🔧 Bug Fix\n\n- Add feature flags and warnings for Warehouse, SQL Database, and Eventhouse unpublish operations ([#483](https://github.com/microsoft/fabric-cicd/issues/483))\n- Fix code formatting inconsistencies in fabric_workspace unit test ([#474](https://github.com/microsoft/fabric-cicd/issues/474))\n- Fix KeyError when deploying Reports with Semantic Model dependencies in Report-only scope case ([#278](https://github.com/microsoft/fabric-cicd/issues/278))\n\n## [v0.1.24](https://pypi.org/project/fabric-cicd/0.1.24) - August 04, 2025\n\n### ⚠️ Breaking Change\n\n- Require parameterization for Dataflow and Semantic Model references in Data Pipeline activities\n- Require specific parameterization for deploying a Dataflow that depends on another in the same workspace (see Parameterization docs)\n\n### 📝 Documentation Update\n\n- Improve Parameterization documentation ([#415](https://github.com/microsoft/fabric-cicd/issues/415))\n\n### ⚡ Additional Optimizations\n\n- Support for Eventhouse query URI parameterization ([#414](https://github.com/microsoft/fabric-cicd/issues/414))\n- Support for Warehouse SQL endpoint parameterization ([#392](https://github.com/microsoft/fabric-cicd/issues/392))\n\n### 🔧 Bug Fix\n\n- Fix Dataflow/Data Pipeline deployment failures caused by workspace permissions ([#419](https://github.com/microsoft/fabric-cicd/issues/419))\n- Prevent duplicate logical ID issue in Report and Semantic Model deployment ([#405](https://github.com/microsoft/fabric-cicd/issues/405))\n- Fix deployment of items without assigned capacity ([#402](https://github.com/microsoft/fabric-cicd/issues/402))\n\n## [v0.1.23](https://pypi.org/project/fabric-cicd/0.1.23) - July 08, 2025\n\n### ✨ New Functionality\n\n- New functionalities for GitHub Copilot Agent and PR-to-Issue linking\n\n### 📝 Documentation Update\n\n- Fix formatting and examples in the How to and Examples pages\n\n### 🔧 Bug Fix\n\n- Fix issue with lakehouse shortcuts publishing ([#379](https://github.com/microsoft/fabric-cicd/issues/379))\n- Add validation for empty logical IDs to prevent deployment corruption ([#86](https://github.com/microsoft/fabric-cicd/issues/86))\n- Fix SQL provision print statement ([#329](https://github.com/microsoft/fabric-cicd/issues/329))\n- Rename the error code for reserved item name per updated Microsoft Fabric API ([#388](https://github.com/microsoft/fabric-cicd/issues/388))\n- Fix lakehouse exclude_regex to exclude shortcut publishing ([#385](https://github.com/microsoft/fabric-cicd/issues/385))\n- Remove max retry limit to handle large deployments ([#299](https://github.com/microsoft/fabric-cicd/issues/299))\n\n## [v0.1.22](https://pypi.org/project/fabric-cicd/0.1.22) - June 25, 2025\n\n### 🆕 New Items Support\n\n- Onboard API for GraphQL item type ([#287](https://github.com/microsoft/fabric-cicd/issues/287))\n\n### 🔧 Bug Fix\n\n- Fix Fabric API call error during dataflow publish ([#352](https://github.com/microsoft/fabric-cicd/issues/352))\n\n### ⚡ Additional Optimizations\n\n- Expanded test coverage to handle folder edge cases ([#358](https://github.com/microsoft/fabric-cicd/issues/358))\n\n## [v0.1.21](https://pypi.org/project/fabric-cicd/0.1.21) - June 18, 2025\n\n### 🔧 Bug Fix\n\n- Fix bug with workspace ID replacement in JSON files for pipeline deployments ([#345](https://github.com/microsoft/fabric-cicd/issues/345))\n\n### ⚡ Additional Optimizations\n\n- Increased max retry for Warehouses and Dataflows\n\n## [v0.1.20](https://pypi.org/project/fabric-cicd/0.1.20) - June 12, 2025\n\n### ✨ New Functionality\n\n- Parameterization support for find_value regex and replace_value variables ([#326](https://github.com/microsoft/fabric-cicd/issues/326))\n\n### 🆕 New Items Support\n\n- Onboard KQL Dashboard item type ([#329](https://github.com/microsoft/fabric-cicd/issues/329))\n- Onboard Dataflow Gen2 item type ([#111](https://github.com/microsoft/fabric-cicd/issues/111))\n\n### 🔧 Bug Fix\n\n- Fix bug with deploying environment libraries with special chars ([#336](https://github.com/microsoft/fabric-cicd/issues/336))\n\n### ⚡ Additional Optimizations\n\n- Improved test coverage for subfolder creation/modification ([#211](https://github.com/microsoft/fabric-cicd/issues/211))\n\n## [v0.1.19](https://pypi.org/project/fabric-cicd/0.1.19) - May 21, 2025\n\n### 🆕 New Items Support\n\n- Onboard SQL Database item type (shell-only deployment) ([#301](https://github.com/microsoft/fabric-cicd/issues/301))\n- Onboard Warehouse item type (shell-only deployment) ([#204](https://github.com/microsoft/fabric-cicd/issues/204))\n\n### 🔧 Bug Fix\n\n- Fix bug with unpublish workspace folders ([#273](https://github.com/microsoft/fabric-cicd/issues/273))\n\n## [v0.1.18](https://pypi.org/project/fabric-cicd/0.1.18) - May 14, 2025\n\n### 🔧 Bug Fix\n\n- Fix bug with check environment publish state ([#295](https://github.com/microsoft/fabric-cicd/issues/295))\n\n## [v0.1.17](https://pypi.org/project/fabric-cicd/0.1.17) - May 13, 2025\n\n### ⚠️ Breaking Change\n\n- Deprecate old parameter file structure ([#283](https://github.com/microsoft/fabric-cicd/issues/283))\n\n### 🆕 New Items Support\n\n- Onboard CopyJob item type ([#122](https://github.com/microsoft/fabric-cicd/issues/122))\n- Onboard Eventstream item type ([#170](https://github.com/microsoft/fabric-cicd/issues/170))\n- Onboard Eventhouse/KQL Database item type ([#169](https://github.com/microsoft/fabric-cicd/issues/169))\n- Onboard Data Activator item type ([#291](https://github.com/microsoft/fabric-cicd/issues/291))\n- Onboard KQL Queryset item type ([#292](https://github.com/microsoft/fabric-cicd/issues/292))\n\n### 🔧 Bug Fix\n\n- Fix post publish operations for skipped items ([#277](https://github.com/microsoft/fabric-cicd/issues/277))\n\n### ⚡ Additional Optimizations\n\n- New function `key_value_replace` for key-based replacement operations in JSON and YAML\n\n### 📝 Documentation Update\n\n- Add publish regex example to demonstrate how to use the `publish_all_items` with regex for excluding item names\n\n## [v0.1.16](https://pypi.org/project/fabric-cicd/0.1.16) - April 25, 2025\n\n### 🔧 Bug Fix\n\n- Fix bug with folder deployment to root ([#255](https://github.com/microsoft/fabric-cicd/issues/255))\n\n### ⚡ Additional Optimizations\n\n- Add Workspace Name in FabricWorkspaceObject ([#200](https://github.com/microsoft/fabric-cicd/issues/200))\n- New function to check SQL endpoint provision status ([#226](https://github.com/microsoft/fabric-cicd/issues/226))\n\n### 📝 Documentation Update\n\n- Updated Authentication docs + menu sort order\n\n## [v0.1.15](https://pypi.org/project/fabric-cicd/0.1.15) - April 21, 2025\n\n### 🔧 Bug Fix\n\n- Fix folders moving with every publish ([#236](https://github.com/microsoft/fabric-cicd/issues/236))\n\n### ⚡ Additional Optimizations\n\n- Introduce parallel deployments to reduce publish times ([#237](https://github.com/microsoft/fabric-cicd/issues/237))\n- Improvements to check version logic\n\n### 📝 Documentation Update\n\n- Updated Examples section in docs\n\n## [v0.1.14](https://pypi.org/project/fabric-cicd/0.1.14) - April 09, 2025\n\n### ✨ New Functionality\n\n- Optimized & beautified terminal output\n- Added changelog to output of old version check\n\n### 🔧 Bug Fix\n\n- Fix workspace folder deployments in root folder ([#221](https://github.com/microsoft/fabric-cicd/issues/221))\n- Fix unpublish of workspace folders without publish ([#222](https://github.com/microsoft/fabric-cicd/issues/222))\n\n### ⚡ Additional Optimizations\n\n- Removed Colorama and Colorlog Dependency\n\n## [v0.1.13](https://pypi.org/project/fabric-cicd/0.1.13) - April 07, 2025\n\n### ✨ New Functionality\n\n- Added support for Lakehouse Shortcuts\n- New `enable_environment_variable_replacement` feature flag ([#160](https://github.com/microsoft/fabric-cicd/issues/160))\n\n### 🆕 New Items Support\n\n- Onboard Workspace Folders ([#81](https://github.com/microsoft/fabric-cicd/issues/81))\n- Onboard Variable Library item type ([#206](https://github.com/microsoft/fabric-cicd/issues/206))\n\n### ⚡ Additional Optimizations\n\n- User-agent now available in API headers ([#207](https://github.com/microsoft/fabric-cicd/issues/207))\n- Fixed error log typo in fabric_endpoint\n\n### 🔧 Bug Fix\n\n- Fix break with invalid optional parameters ([#192](https://github.com/microsoft/fabric-cicd/issues/192))\n- Fix bug where all workspace ids were not being replaced by parameterization ([#186](https://github.com/microsoft/fabric-cicd/issues/186))\n\n## [v0.1.12](https://pypi.org/project/fabric-cicd/0.1.12) - March 27, 2025\n\n### 🔧 Bug Fix\n\n- Fix constant overwrite failures ([#190](https://github.com/microsoft/fabric-cicd/issues/190))\n- Fix bug where all workspace ids were not being replaced ([#186](https://github.com/microsoft/fabric-cicd/issues/186))\n- Fix type hints for older versions of Python ([#156](https://github.com/microsoft/fabric-cicd/issues/156))\n- Fix accepted item types constant in pre-build\n\n## [v0.1.11](https://pypi.org/project/fabric-cicd/0.1.11) - March 25, 2025\n\n### ⚠️ Breaking Change\n\n- Parameterization refactor introducing a new parameter file structure and parameter file validation functionality ([#113](https://github.com/microsoft/fabric-cicd/issues/113))\n\n### ✨ New Functionality\n\n- Support regex for publish exclusion ([#121](https://github.com/microsoft/fabric-cicd/issues/121))\n- Override max retries via constants ([#146](https://github.com/microsoft/fabric-cicd/issues/146))\n\n### 📝 Documentation Update\n\n- Update to [parameterization](https://microsoft.github.io/fabric-cicd/latest/how_to/parameterization/) docs\n\n## [v0.1.10](https://pypi.org/project/fabric-cicd/0.1.10) - March 19, 2025\n\n### ✨ New Functionality\n\n- DataPipeline SPN Support ([#133](https://github.com/microsoft/fabric-cicd/issues/133))\n\n### 🔧 Bug Fix\n\n- Workspace ID replacement in data pipelines ([#164](https://github.com/microsoft/fabric-cicd/issues/164))\n\n### 📝 Documentation Update\n\n- Sample for passing in arguments from Azure DevOps Pipelines\n\n## [v0.1.9](https://pypi.org/project/fabric-cicd/0.1.9) - March 11, 2025\n\n### 🆕 New Items Support\n\n- Support for Mirrored Database item type ([#145](https://github.com/microsoft/fabric-cicd/issues/145))\n\n### ⚡ Additional Optimizations\n\n- Increase reserved name wait time ([#135](https://github.com/microsoft/fabric-cicd/issues/135))\n\n## [v0.1.8](https://pypi.org/project/fabric-cicd/0.1.8) - March 04, 2025\n\n### 🔧 Bug Fix\n\n- Handle null byPath object in report definition file ([#143](https://github.com/microsoft/fabric-cicd/issues/143))\n- Support relative directories ([#136](https://github.com/microsoft/fabric-cicd/issues/136)) ([#132](https://github.com/microsoft/fabric-cicd/issues/132))\n- Increase special character support ([#134](https://github.com/microsoft/fabric-cicd/issues/134))\n\n### ⚡ Additional Optimizations\n\n- Changelog now available with version check ([#127](https://github.com/microsoft/fabric-cicd/issues/127))\n\n## [v0.1.7](https://pypi.org/project/fabric-cicd/0.1.7) - February 26, 2025\n\n### 🔧 Bug Fix\n\n- Fix special character support in files ([#129](https://github.com/microsoft/fabric-cicd/issues/129))\n\n## [v0.1.6](https://pypi.org/project/fabric-cicd/0.1.6) - February 24, 2025\n\n### 🆕 New Items Support\n\n- Onboard Lakehouse item type ([#116](https://github.com/microsoft/fabric-cicd/issues/116))\n\n### 📝 Documentation Update\n\n- Update example docs ([#25](https://github.com/microsoft/fabric-cicd/issues/25))\n- Update find_replace docs ([#110](https://github.com/microsoft/fabric-cicd/issues/110))\n\n### ⚡ Additional Optimizations\n\n- Standardized docstrings to Google format\n- Onboard file objects ([#46](https://github.com/microsoft/fabric-cicd/issues/46))\n- Leverage UpdateDefinition Flag ([#28](https://github.com/microsoft/fabric-cicd/issues/28))\n- Convert repo and workspace dictionaries ([#45](https://github.com/microsoft/fabric-cicd/issues/45))\n\n## [v0.1.5](https://pypi.org/project/fabric-cicd/0.1.5) - February 18, 2025\n\n### 🔧 Bug Fix\n\n- Fix Environment Failure without Public Library ([#103](https://github.com/microsoft/fabric-cicd/issues/103))\n\n### ⚡ Additional Optimizations\n\n- Introduces pytest check for PRs ([#100](https://github.com/microsoft/fabric-cicd/issues/100))\n\n## [v0.1.4](https://pypi.org/project/fabric-cicd/0.1.4) - February 12, 2025\n\n### ✨ New Functionality\n\n- Support Feature Flagging ([#96](https://github.com/microsoft/fabric-cicd/issues/96))\n\n### 🔧 Bug Fix\n\n- Fix Image support in report deployment ([#88](https://github.com/microsoft/fabric-cicd/issues/88))\n- Fix Broken README link ([#92](https://github.com/microsoft/fabric-cicd/issues/92))\n\n### ⚡ Additional Optimizations\n\n- Workspace ID replacement improved\n- Increased error handling in activate script\n- Onboard pytest and coverage\n- Improvements to nested dictionaries ([#37](https://github.com/microsoft/fabric-cicd/issues/37))\n- Support Python Installed From Windows Store ([#87](https://github.com/microsoft/fabric-cicd/issues/87))\n\n## [v0.1.3](https://pypi.org/project/fabric-cicd/0.1.3) - January 29, 2025\n\n### ✨ New Functionality\n\n- Add PyPI check version to encourage version bumps ([#75](https://github.com/microsoft/fabric-cicd/issues/75))\n\n### 🔧 Bug Fix\n\n- Fix Semantic model initial publish results in None Url error ([#61](https://github.com/microsoft/fabric-cicd/issues/61))\n- Fix Integer parsed as float failing in handle_retry for <3.12 python ([#63](https://github.com/microsoft/fabric-cicd/issues/63))\n- Fix Default item types fail to unpublish ([#76](https://github.com/microsoft/fabric-cicd/issues/76))\n- Fix Items in subfolders are skipped ([#77](https://github.com/microsoft/fabric-cicd/issues/77))\n\n### 📝 Documentation Update\n\n- Update documentation & examples\n\n## [v0.1.2](https://pypi.org/project/fabric-cicd/0.1.2) - January 27, 2025\n\n### ✨ New Functionality\n\n- Introduces max retry and backoff for long running / throttled calls ([#27](https://github.com/microsoft/fabric-cicd/issues/27))\n\n### 🔧 Bug Fix\n\n- Fix Environment publish uses arbitrary wait time ([#50](https://github.com/microsoft/fabric-cicd/issues/50))\n- Fix Environment publish doesn't wait for success ([#56](https://github.com/microsoft/fabric-cicd/issues/56))\n- Fix Long running operation steps out early for notebook publish ([#58](https://github.com/microsoft/fabric-cicd/issues/58))\n\n## [v0.1.1](https://pypi.org/project/fabric-cicd/0.1.1) - January 23, 2025\n\n### 🔧 Bug Fix\n\n- Fix Environment stuck in publish ([#51](https://github.com/microsoft/fabric-cicd/issues/51))\n\n## [v0.1.0](https://pypi.org/project/fabric-cicd/0.1.0) - January 23, 2025\n\n### ✨ New Functionality\n\n- Initial public preview release\n- Supports Notebook, Pipeline, Semantic Model, Report, and Environment deployments\n- Supports User and System Identity authentication\n- Released to PyPi\n- Onboarded to Github Pages\n" }, { "path": "docs/code_reference.md", "content": "# Code Reference\n::: fabric_cicd\n" }, { "path": "docs/config/overrides/main.html", "content": "{% extends \"base.html\" %}\n\n{% block announce %}\n

ℹ️ This library is open source. Please raise issues & feature requests as they arise.

\n{% endblock %}\n" }, { "path": "docs/config/pre-build/section_toc.py", "content": "from pathlib import Path\nimport re\nimport unicodedata\n\nimport yaml\n\n\ndef slugify(title):\n \"\"\"\n Generate an anchor slug from a heading title, matching MkDocs/Python-Markdown toc behavior.\n Strips markdown formatting (backticks, escape chars), lowercases, replaces spaces with hyphens,\n and removes characters that aren't alphanumeric, hyphens, or underscores.\n \"\"\"\n # Remove markdown escape backslashes (e.g. \\_ALL\\_ -> _ALL_)\n slug = title.replace(\"\\\\\", \"\")\n # Remove backticks (inline code markers)\n slug = slug.replace(\"`\", \"\")\n # Normalize unicode\n slug = unicodedata.normalize(\"NFKD\", slug)\n # Lowercase\n slug = slug.lower()\n # Replace spaces with hyphens\n slug = slug.replace(\" \", \"-\")\n # Remove characters that aren't alphanumeric, hyphens, or underscores\n slug = re.sub(r\"[^\\w\\-]\", \"\", slug)\n # Strip leading/trailing hyphens\n slug = slug.strip(\"-\")\n return slug\n\n\ndef get_section_order(nav, current_dir_str):\n \"\"\"\n Recursively find the order of markdown files in the current directory as defined in nav.\n \"\"\"\n order = []\n for item in nav:\n if isinstance(item, dict):\n for key, value in item.items():\n if isinstance(value, list):\n # Recurse into subsections\n order += get_section_order(value, current_dir_str)\n elif isinstance(value, str):\n path = Path(value)\n if str(path.parent).replace(\"\\\\\", \"/\") == current_dir_str:\n order.append(path.name)\n elif isinstance(item, str):\n path = Path(item)\n if str(path.parent).replace(\"\\\\\", \"/\") == current_dir_str:\n order.append(path.name)\n return order\n\ndef on_page_markdown(markdown, page, config, files):\n if \"\\n\" in markdown:\n start_index = markdown.index(\"\\n\") + len(\"\\n\")\n end_index = markdown.index(\"\\n\")\n\n current_page_path = Path(page.file.abs_src_path)\n current_page_dir = current_page_path.parent\n\n # Compute docs root (where mkdocs.yml lives)\n mkdocs_yml = Path(config['config_file_path'])\n docs_root = mkdocs_yml.parent\n\n # Get current dir relative to docs root, as string\n current_dir_rel = str(current_page_dir.name)\n\n # Load mkdocs.yml and extract nav order\n with mkdocs_yml.open(\"r\", encoding=\"utf-8\") as f:\n mkdocs_config = yaml.safe_load(f)\n nav = mkdocs_config.get('nav', [])\n section_order = get_section_order(nav, current_dir_rel)\n \n\n toc = []\n for md_name in section_order:\n md_file = current_page_dir / md_name\n if not md_file.exists() or md_file == current_page_path:\n continue\n with md_file.open(\"r\", encoding=\"utf-8\") as f:\n content = f.read()\n content_no_code = re.sub(r\"```.*?```\", \"\", content, flags=re.DOTALL)\n headers = re.findall(r\"^(#{1,6})\\s+(.*)\", content_no_code, re.MULTILINE)\n seen_slugs = {}\n for header in headers:\n level = len(header[0])\n title = header[1]\n base_anchor = slugify(title)\n # MkDocs appends _1, _2, etc. for duplicate heading IDs\n count = seen_slugs.get(base_anchor, 0)\n seen_slugs[base_anchor] = count + 1\n anchor = base_anchor if count == 0 else f\"{base_anchor}_{count}\"\n toc.append(f\"{' ' * level}- [{title}]({md_file.name}#{anchor})\")\n\n toc_content = \"\\n\".join(toc)\n new_markdown = markdown[:start_index] + toc_content + markdown[end_index:]\n return new_markdown\n return markdown\n" }, { "path": "docs/config/pre-build/update_item_types.py", "content": "import sys\nfrom pathlib import Path\n\nroot_directory = Path(__file__).resolve().parent.parent.parent.parent\nsys.path.insert(0, str(root_directory / \"src\"))\n\nimport fabric_cicd.constants as constants\n\n\ndef on_page_markdown(markdown, **kwargs):\n if \"\\n\" in markdown:\n start_index = markdown.index(\"\\n\") + len(\"\\n\")\n end_index = markdown.index(\"\\n\")\n\n supported_item_types = constants.ACCEPTED_ITEM_TYPES\n markdown_content = \"\\n\".join([f\"- {item}\" for item in supported_item_types])\n\n new_markdown = markdown[:start_index] + markdown_content + markdown[end_index:]\n return new_markdown\n return markdown\n" }, { "path": "docs/config/pre-build/update_python_version.py", "content": "import re\ntry:\n import tomllib # Python 3.11+\nexcept ImportError:\n import toml as tomllib # Fallback for older Python versions\nfrom pathlib import Path\n\n\ndef on_page_markdown(markdown, **kwargs):\n \"\"\"\n Replace Python version placeholders with versions from pyproject.toml\n \"\"\"\n if \"\" in markdown or \"\" in markdown:\n # Get pyproject.toml path (4 levels up from this file)\n root_directory = Path(__file__).resolve().parent.parent.parent.parent\n pyproject_path = root_directory / \"pyproject.toml\"\n \n try:\n # Load pyproject.toml\n with open(pyproject_path, 'rb') as f:\n try:\n pyproject_data = tomllib.load(f)\n except AttributeError:\n # Fallback for older toml library\n f.seek(0)\n content = f.read().decode('utf-8')\n pyproject_data = tomllib.loads(content)\n \n # Extract requires-python\n requires_python = pyproject_data.get('project', {}).get('requires-python', '')\n \n # Extract min and max versions\n min_version = \"3.9\" # fallback\n max_version = \"3.12\" # fallback\n \n if requires_python:\n min_match = re.search(r'>=(\\d+\\.\\d+)', requires_python)\n max_match = re.search(r'<(\\d+\\.\\d+)', requires_python)\n \n if min_match:\n min_version = min_match.group(1)\n if max_match:\n # Convert exclusive max to inclusive (e.g., <3.13 means up to 3.12)\n max_parts = max_match.group(1).split('.')\n max_major = int(max_parts[0])\n max_minor = int(max_parts[1])\n if max_minor > 0:\n max_version = f\"{max_major}.{max_minor - 1}\"\n \n except Exception:\n # Use fallback values\n min_version = \"3.9\"\n max_version = \"3.13\"\n \n # Replace placeholders\n markdown = markdown.replace(\"\", min_version)\n markdown = markdown.replace(\"\", max_version)\n \n return markdown" }, { "path": "docs/config/stylesheets/extra.css", "content": "[data-md-color-scheme=\"fabric\"] {\n --md-primary-fg-color: #117865;\n --md-primary-fg-color--light: #e3f7ef;\n --md-primary-fg-color--dark: #012826;\n --md-typeset-a-color: #012826;\n --md-accent-fg-color: #012826;\n --md-default-fg-color--light: #333333;\n --md-typeset-a-color: #117865;\n --md-primary-bg-color: #ffffff;\n}\n\n/* Remove opaqueness to top nav */\n.md-tabs__link,\n.md-tabs__link:hover,\n.md-tabs__item--active {\n opacity: 1;\n}\n\n/* Add line under active top nav item */\n.md-tabs__item--active {\n opacity: 1;\n box-shadow: inset 0 -6px 0 0 var(--md-primary-fg-color),\n inset 0 -8px 0 0 var(--md-primary-bg-color);\n}\n\n/* Add underline to links */\n.md-content a {\n text-decoration: underline;\n color: var(--md-primary-fg-color);\n}\n\n/* Keep underline on hover */\n.md-content a:hover {\n text-decoration: underline;\n color: var(--md-primary-fg-color--dark);\n}\n\n.md-typeset .tabbed-labels > label > [href]:first-child {\n color: var(--md-typeset-a-color);\n text-decoration: inherit;\n}\n\n.js .md-typeset .tabbed-labels:before {\n background: var(--md-typeset-a-color);\n}\n\n/*Code inside of tables are of white background*/\n.md-typeset table code {\n background-color: var(--md-primary-bg-color);\n}\n\n/*Code inside of tables are colored when hovered*/\n.md-typeset table tr:hover code {\n background-color: inherit;\n}\n\n/*Default header color*/\n.md-content h1,\n.md-content h2,\n.md-content h3 {\n color: var(--md-accent-fg-color);\n font-weight: 500; /* Slight bold */\n margin: 0 0 1em;\n}\n\n/*Add a border line to all H2*/\n.md-content h2 {\n padding-top: 1em;\n border-top: 1px solid #ddd;\n}\n\n/*Custom class for subheaders under h2*/\n.md-h2-subheader {\n font-size: smaller;\n color: inherit;\n}\n\n/*Custom class for non header h3*/\n.md-h3-nonanchor {\n color: var(--md-accent-fg-color);\n font-weight: 500; /* Slight bold */\n margin: 0 0 1em;\n font-size: 1.25em;\n}\n\n.md-h4-nonanchor {\n color: var(--md-accent-fg-color);\n font-weight: 700;\n letter-spacing: -0.01em;\n margin: 1em 0;\n}\n\n/*Remove all margins from the

element containing .md-h2-subheader*/\np:has(.md-h2-subheader) {\n margin: 0;\n}\n\n/*Remove bottom margin from h2 so it's closer to subheader*/\n.md-content h2:has(+ p .md-h2-subheader) {\n margin-bottom: 0;\n}\n\n/* Override default font size for typeset */\n.md-typeset {\n font-size: 0.64rem;\n}\n\n.md-typeset h1 {\n font-size: 1.25rem;\n}\n\n.md-typeset h2 {\n font-size: 1rem;\n}\n\n.md-nav--lifted .md-nav__item {\n font-size: 0.7rem;\n}\n\n/* Markdown image resizing */\n.md-content .md-typeset img {\n max-width: 40rem;\n width: 100%;\n height: auto; /* Maintain aspect ratio */\n}\n" }, { "path": "docs/example/authentication.md", "content": "# Authentication Examples\n\nThe following are the most common authentication flows for fabric-cicd. However, because fabric-cicd supports any [TokenCredential](https://learn.microsoft.com/en-us/dotnet/api/azure.core.tokencredential), there are multiple authentication methods available beyond the ones described here. These examples provide starting points that should be adapted for your specific environment and security requirements.\n\n> **⚠️ NOTICE:** Due to security best practices, the **Default Credential** (`DefaultAzureCredential` fallback) and **implicit Fabric Notebook authentication** (without a `token_credential` parameter) methods are no longer supported. `token_credential` is now a required parameter.\n\n**Notes:**\n\n- Fabric Notebook users must provide an explicit `token_credential`. See [Fabric Notebook Authentication](#fabric-notebook-authentication) for options.\n- Avoid hardcoding credentials. Use environment variables or secret management services. SPN + Secret auth can also be achieved via `az login --service-principal` or `Connect-AzAccount -ServicePrincipal` in the CLI/PowerShell flows below.\n\n## CLI Credential\n\nThis approach utilizes the CLI credential flow, meaning it only refers to the authentication established with `az login`. This is agnostic of the executing user; it can be UPN, SPN, Managed Identity, etc. Whatever is used to log in will be used.\n\n=== \"Local\"\n\n ```python\n '''Log in with Azure CLI (az login) prior to execution'''\n\n from pathlib import Path\n\n from azure.identity import AzureCliCredential\n from fabric_cicd import FabricWorkspace, publish_all_items, unpublish_all_orphan_items\n\n # Assumes your script is one level down from root\n root_directory = Path(__file__).resolve().parent\n\n # Sample values for FabricWorkspace parameters\n workspace_id = \"your-workspace-id\"\n environment = \"your-environment\"\n repository_directory = str(root_directory / \"your-workspace-directory\")\n item_type_in_scope = [\"Notebook\", \"DataPipeline\", \"Environment\"]\n\n # Use Azure CLI credential to authenticate\n token_credential = AzureCliCredential()\n\n # Initialize the FabricWorkspace object with the required parameters\n target_workspace = FabricWorkspace(\n workspace_id=workspace_id,\n environment=environment,\n repository_directory=repository_directory,\n item_type_in_scope=item_type_in_scope,\n token_credential=token_credential,\n )\n\n # Publish all items defined in item_type_in_scope\n publish_all_items(target_workspace)\n\n # Unpublish all items defined in item_type_in_scope not found in repository\n unpublish_all_orphan_items(target_workspace)\n ```\n\n=== \"Azure DevOps\"\n\n ```python\n '''\n Log in with Azure CLI (az login) prior to execution\n OR (Preferred) Use Az CLI ADO Tasks with a Service Connection\n '''\n\n import sys\n import os\n from pathlib import Path\n\n from azure.identity import AzureCliCredential\n from fabric_cicd import FabricWorkspace, publish_all_items, unpublish_all_orphan_items, change_log_level\n\n # Force unbuffered output like `python -u`\n sys.stdout.reconfigure(line_buffering=True, write_through=True)\n sys.stderr.reconfigure(line_buffering=True, write_through=True)\n\n # Enable debugging if defined in Azure DevOps pipeline\n if os.getenv(\"SYSTEM_DEBUG\", \"false\").lower() == \"true\":\n change_log_level(\"DEBUG\")\n\n # Assumes your script is one level down from root\n root_directory = Path(__file__).resolve().parent\n\n # Sample values for FabricWorkspace parameters\n workspace_id = \"your-workspace-id\"\n environment = \"your-environment\"\n repository_directory = str(root_directory / \"your-workspace-directory\")\n item_type_in_scope = [\"Notebook\", \"DataPipeline\", \"Environment\"]\n\n # Use Azure CLI credential to authenticate\n token_credential = AzureCliCredential()\n\n # Initialize the FabricWorkspace object with the required parameters\n target_workspace = FabricWorkspace(\n workspace_id=workspace_id,\n environment=environment,\n repository_directory=repository_directory,\n item_type_in_scope=item_type_in_scope,\n token_credential=token_credential,\n )\n\n # Publish all items defined in item_type_in_scope\n publish_all_items(target_workspace)\n\n # Unpublish all items defined in item_type_in_scope not found in repository\n unpublish_all_orphan_items(target_workspace)\n ```\n\n=== \"GitHub\"\n\n ```python\n '''\n Log in with Azure CLI (az login) prior to execution\n Requires: azure/login workflow step in GitHub Actions\n '''\n\n import os\n from pathlib import Path\n from azure.identity import AzureCliCredential\n from fabric_cicd import FabricWorkspace, publish_all_items, unpublish_all_orphan_items\n\n # GitHub Actions sets GITHUB_WORKSPACE automatically\n root_directory = Path(os.getenv(\"GITHUB_WORKSPACE\", \".\")).resolve()\n\n # Sample values for FabricWorkspace parameters\n workspace_id = os.getenv(\"WORKSPACE_ID\")\n environment = os.getenv(\"ENVIRONMENT\", \"PROD\")\n repository_directory = str(root_directory / \"your-workspace-directory\")\n item_type_in_scope = [\"Notebook\", \"DataPipeline\", \"Environment\"]\n\n # Use Azure CLI credential (assumes 'az login' in workflow step)\n token_credential = AzureCliCredential()\n\n # Initialize the FabricWorkspace object with the required parameters\n target_workspace = FabricWorkspace(\n workspace_id=workspace_id,\n environment=environment,\n repository_directory=repository_directory,\n item_type_in_scope=item_type_in_scope,\n token_credential=token_credential,\n )\n\n # Publish all items defined in item_type_in_scope\n publish_all_items(target_workspace)\n\n # Unpublish all items defined in item_type_in_scope not found in repository\n unpublish_all_orphan_items(target_workspace)\n ```\n\n## AZ PowerShell Credential\n\nThis approach utilizes the AZ PowerShell credential flow, meaning it only refers to the authentication established with `Connect-AzAccount`. This is agnostic of the executing user; it can be UPN, SPN, Managed Identity, etc. Whatever is used to log in will be used.\n\n=== \"Local\"\n\n ```python\n '''Log in with Azure PowerShell (Connect-AzAccount) prior to execution'''\n\n from pathlib import Path\n\n from azure.identity import AzurePowerShellCredential\n from fabric_cicd import FabricWorkspace, publish_all_items, unpublish_all_orphan_items\n\n # Assumes your script is one level down from root\n root_directory = Path(__file__).resolve().parent\n\n # Sample values for FabricWorkspace parameters\n workspace_id = \"your-workspace-id\"\n environment = \"your-environment\"\n repository_directory = str(root_directory / \"your-workspace-directory\")\n item_type_in_scope = [\"Notebook\", \"DataPipeline\", \"Environment\"]\n\n # Use Azure PowerShell credential to authenticate\n token_credential = AzurePowerShellCredential()\n\n # Initialize the FabricWorkspace object with the required parameters\n target_workspace = FabricWorkspace(\n workspace_id=workspace_id,\n environment=environment,\n repository_directory=repository_directory,\n item_type_in_scope=item_type_in_scope,\n token_credential=token_credential,\n )\n\n # Publish all items defined in item_type_in_scope\n publish_all_items(target_workspace)\n\n # Unpublish all items defined in item_type_in_scope not found in repository\n unpublish_all_orphan_items(target_workspace)\n ```\n\n=== \"Azure DevOps\"\n\n ```python\n '''\n Log in with Azure PowerShell (Connect-AzAccount) prior to execution\n OR (Preferred) Use AzPowerShell ADO Tasks with a Service Connection\n '''\n\n import sys\n import os\n from pathlib import Path\n\n from azure.identity import AzurePowerShellCredential\n from fabric_cicd import FabricWorkspace, publish_all_items, unpublish_all_orphan_items, change_log_level\n\n # Force unbuffered output like `python -u`\n sys.stdout.reconfigure(line_buffering=True, write_through=True)\n sys.stderr.reconfigure(line_buffering=True, write_through=True)\n\n # Enable debugging if defined in Azure DevOps pipeline\n if os.getenv(\"SYSTEM_DEBUG\", \"false\").lower() == \"true\":\n change_log_level(\"DEBUG\")\n\n # Assumes your script is one level down from root\n root_directory = Path(__file__).resolve().parent\n\n # Sample values for FabricWorkspace parameters\n workspace_id = \"your-workspace-id\"\n environment = \"your-environment\"\n repository_directory = str(root_directory / \"your-workspace-directory\")\n item_type_in_scope = [\"Notebook\", \"DataPipeline\", \"Environment\"]\n\n # Use Azure PowerShell credential to authenticate\n token_credential = AzurePowerShellCredential()\n\n # Initialize the FabricWorkspace object with the required parameters\n target_workspace = FabricWorkspace(\n workspace_id=workspace_id,\n environment=environment,\n repository_directory=repository_directory,\n item_type_in_scope=item_type_in_scope,\n token_credential=token_credential,\n )\n\n # Publish all items defined in item_type_in_scope\n publish_all_items(target_workspace)\n\n # Unpublish all items defined in item_type_in_scope not found in repository\n unpublish_all_orphan_items(target_workspace)\n ```\n\n=== \"GitHub\"\n\n ```python\n '''\n Log in with Azure PowerShell (Connect-AzAccount) prior to execution\n Requires: azure/powershell workflow step in GitHub Actions\n '''\n\n import os\n from pathlib import Path\n from azure.identity import AzurePowerShellCredential\n from fabric_cicd import FabricWorkspace, publish_all_items, unpublish_all_orphan_items\n\n # GitHub Actions sets GITHUB_WORKSPACE automatically\n root_directory = Path(os.getenv(\"GITHUB_WORKSPACE\", \".\")).resolve()\n\n # Sample values for FabricWorkspace parameters\n workspace_id = os.getenv(\"WORKSPACE_ID\")\n environment = os.getenv(\"ENVIRONMENT\", \"PROD\")\n repository_directory = str(root_directory / \"your-workspace-directory\")\n item_type_in_scope = [\"Notebook\", \"DataPipeline\", \"Environment\"]\n\n # Use Azure PowerShell credential (assumes 'Connect-AzAccount' in workflow step)\n token_credential = AzurePowerShellCredential()\n\n # Initialize the FabricWorkspace object with the required parameters\n target_workspace = FabricWorkspace(\n workspace_id=workspace_id,\n environment=environment,\n repository_directory=repository_directory,\n item_type_in_scope=item_type_in_scope,\n token_credential=token_credential,\n )\n\n # Publish all items defined in item_type_in_scope\n publish_all_items(target_workspace)\n\n # Unpublish all items defined in item_type_in_scope not found in repository\n unpublish_all_orphan_items(target_workspace)\n ```\n\n## Managed Identity Credential\n\nThis approach uses Azure Managed Identity, eliminating the need to manage secrets. Managed identities provide an automatically managed identity in Azure AD for applications to use when connecting to resources.\n\n=== \"Azure DevOps\"\n\n ```python\n '''\n Running on Azure DevOps self-hosted agents with system-assigned managed identity\n OR Azure DevOps agents hosted on Azure VMs with managed identity\n '''\n\n import sys\n import os\n from pathlib import Path\n\n from azure.identity import ManagedIdentityCredential\n from fabric_cicd import FabricWorkspace, publish_all_items, unpublish_all_orphan_items, change_log_level\n\n # Force unbuffered output like `python -u`\n sys.stdout.reconfigure(line_buffering=True, write_through=True)\n sys.stderr.reconfigure(line_buffering=True, write_through=True)\n\n # Enable debugging if defined in Azure DevOps pipeline\n if os.getenv(\"SYSTEM_DEBUG\", \"false\").lower() == \"true\":\n change_log_level(\"DEBUG\")\n\n # Assumes your script is one level down from root\n root_directory = Path(__file__).resolve().parent\n\n # Sample values for FabricWorkspace parameters\n workspace_id = \"your-workspace-id\"\n environment = \"your-environment\"\n repository_directory = str(root_directory / \"your-workspace-directory\")\n item_type_in_scope = [\"Notebook\", \"DataPipeline\", \"Environment\"]\n\n # Use system-assigned managed identity\n token_credential = ManagedIdentityCredential()\n\n # Initialize the FabricWorkspace object with the required parameters\n target_workspace = FabricWorkspace(\n workspace_id=workspace_id,\n environment=environment,\n repository_directory=repository_directory,\n item_type_in_scope=item_type_in_scope,\n token_credential=token_credential,\n )\n\n # Publish all items defined in item_type_in_scope\n publish_all_items(target_workspace)\n\n # Unpublish all items defined in item_type_in_scope not found in repository\n unpublish_all_orphan_items(target_workspace)\n ```\n\n=== \"GitHub\"\n\n ```python\n '''\n Running on GitHub self-hosted runners with system-assigned managed identity\n OR GitHub Actions hosted on Azure VMs with managed identity\n '''\n\n import os\n from pathlib import Path\n from azure.identity import ManagedIdentityCredential\n from fabric_cicd import FabricWorkspace, publish_all_items, unpublish_all_orphan_items\n\n # GitHub Actions sets GITHUB_WORKSPACE automatically\n root_directory = Path(os.getenv(\"GITHUB_WORKSPACE\", \".\")).resolve()\n\n # Sample values for FabricWorkspace parameters\n workspace_id = os.getenv(\"WORKSPACE_ID\")\n environment = os.getenv(\"ENVIRONMENT\", \"PROD\")\n repository_directory = str(root_directory / \"your-workspace-directory\")\n item_type_in_scope = [\"Notebook\", \"DataPipeline\", \"Environment\"]\n\n # Use system-assigned managed identity\n token_credential = ManagedIdentityCredential()\n\n # Initialize the FabricWorkspace object with the required parameters\n target_workspace = FabricWorkspace(\n workspace_id=workspace_id,\n environment=environment,\n repository_directory=repository_directory,\n item_type_in_scope=item_type_in_scope,\n token_credential=token_credential,\n )\n\n # Publish all items defined in item_type_in_scope\n publish_all_items(target_workspace)\n\n # Unpublish all items defined in item_type_in_scope not found in repository\n unpublish_all_orphan_items(target_workspace)\n ```\n\n## Fabric Notebook Authentication\n\nWhen running fabric-cicd within Microsoft Fabric Notebooks, an explicit `token_credential` parameter is required, consistent with all other environments. The simplest approach uses `notebookutils.credentials.getToken()` to create a credential from the current user session. Alternatively, you can use a service principal for specific identity requirements.\n\n=== \"Session Credential\"\n\n ```python\n '''\n Use the Fabric Notebook session identity via notebookutils\n Most common pattern: clone repository and deploy from within notebook\n '''\n\n import time\n import tempfile\n import subprocess\n import os\n from azure.core.credentials import AccessToken, TokenCredential\n from fabric_cicd import FabricWorkspace, publish_all_items, unpublish_all_orphan_items\n\n # Define a credential that wraps the Fabric session token\n class FabricNotebookCredential(TokenCredential):\n def get_token(self, *scopes, **kwargs):\n token = notebookutils.credentials.getToken(\"pbi\")\n return AccessToken(token, int(time.time()) + 3600) # 1 hour\n\n # Sample configuration values\n workspace_id = \"your-workspace-id\"\n environment = \"your-environment\"\n repo_url = \"https://github.com/your-org/your-repo.git\"\n repo_ref = \"main\"\n workspace_directory = \"your-workspace-directory\"\n\n # Use context manager for automatic cleanup (even on exceptions)\n with tempfile.TemporaryDirectory(prefix=\"cloned_repo_\") as temp_dir:\n print(f\"Created temporary directory: {temp_dir}\")\n\n # Clone the repository\n print(f\"Cloning {repo_url} (ref: {repo_ref})...\")\n result = subprocess.run(\n [\"git\", \"clone\", \"--branch\", repo_ref, \"--single-branch\", repo_url, temp_dir],\n capture_output=True,\n text=True\n )\n\n if result.returncode != 0:\n raise Exception(f\"Git clone failed: {result.stderr}\")\n\n workspace_root = os.path.join(temp_dir, workspace_directory)\n\n # Deploy workspace items from cloned repository\n item_type_in_scope = [\"Notebook\", \"DataPipeline\", \"Environment\"]\n\n # Initialize FabricWorkspace with explicit session credential\n target_workspace = FabricWorkspace(\n workspace_id=workspace_id,\n environment=environment,\n repository_directory=workspace_root,\n item_type_in_scope=item_type_in_scope,\n token_credential=FabricNotebookCredential(),\n )\n\n # Publish all items defined in item_type_in_scope\n publish_all_items(target_workspace)\n\n # Unpublish all items defined in item_type_in_scope not found in repository\n unpublish_all_orphan_items(target_workspace)\n\n # Directory automatically cleaned up here\n print(\"Cleaned up temporary directory\")\n ```\n\n=== \"SPN Credential\"\n\n ```python\n '''\n Use a service principal for specific identity requirements\n Retrieve secrets from Azure Key Vault using notebookutils\n '''\n\n import tempfile\n import subprocess\n import os\n from azure.identity import ClientSecretCredential\n from fabric_cicd import FabricWorkspace, publish_all_items, unpublish_all_orphan_items\n\n # Sample configuration values\n workspace_id = \"your-workspace-id\"\n environment = \"your-environment\"\n repo_url = \"https://github.com/your-org/your-repo.git\"\n repo_ref = \"main\"\n workspace_directory = \"your-workspace-directory\"\n\n # Retrieve secrets from Azure Key Vault using notebookutils\n key_vault_url = \"https://your-keyvault.vault.azure.net/\"\n client_id = notebookutils.credentials.getSecret(key_vault_url, \"client-id\")\n client_secret = notebookutils.credentials.getSecret(key_vault_url, \"client-secret\")\n tenant_id = notebookutils.credentials.getSecret(key_vault_url, \"tenant-id\")\n\n token_credential = ClientSecretCredential(\n client_id=client_id,\n client_secret=client_secret,\n tenant_id=tenant_id\n )\n\n # Use context manager for automatic cleanup (even on exceptions)\n with tempfile.TemporaryDirectory(prefix=\"cloned_repo_\") as temp_dir:\n print(f\"Created temporary directory: {temp_dir}\")\n\n # Clone the repository\n print(f\"Cloning {repo_url} (ref: {repo_ref})...\")\n result = subprocess.run(\n [\"git\", \"clone\", \"--branch\", repo_ref, \"--single-branch\", repo_url, temp_dir],\n capture_output=True,\n text=True\n )\n\n if result.returncode != 0:\n raise Exception(f\"Git clone failed: {result.stderr}\")\n\n workspace_root = os.path.join(temp_dir, workspace_directory)\n\n # Deploy workspace items from cloned repository\n item_type_in_scope = [\"Notebook\", \"DataPipeline\", \"Environment\"]\n\n # Initialize with explicit SPN credential\n target_workspace = FabricWorkspace(\n workspace_id=workspace_id,\n environment=environment,\n repository_directory=workspace_root,\n item_type_in_scope=item_type_in_scope,\n token_credential=token_credential,\n )\n\n # Publish all items defined in item_type_in_scope\n publish_all_items(target_workspace)\n\n # Unpublish all items defined in item_type_in_scope not found in repository\n unpublish_all_orphan_items(target_workspace)\n\n # Directory automatically cleaned up here\n print(\"Cleaned up temporary directory\")\n ```\n" }, { "path": "docs/example/deployment_variable.md", "content": "# Deployment Variable Examples\n\nA key concept in CI/CD is defining environment-specific deployment variables. The following are examples of how to inject variables from outside of the python script to handle values that are environment-specific, or common across other tooling. These examples provide starting points that should be adapted for your specific environment and security requirements.\n\n> **⚠️ NOTICE**: Due to security best practices, the **Default Credential** (`DefaultAzureCredential` fallback) and **implicit Fabric Notebook authentication** (without a `token_credential` parameter) methods are no longer supported. `token_credential` is now a required parameter.\n\n**Note:** All examples below use the `AzureCliCredential` token for demonstration purposes. You can substitute this with other explicit credential methods based on your environment.\n\n## Branch Based\n\nLeverage the following when you have specific values that you need to define per branch you are deploying from.\n\n=== \"Local\"\n\n ```python\n '''Determines variables based on locally checked out branch'''\n\n from pathlib import Path\n import git # Depends on pip install gitpython\n from azure.identity import AzureCliCredential\n from fabric_cicd import FabricWorkspace, publish_all_items, unpublish_all_orphan_items\n\n # Use Azure CLI credential to authenticate\n token_credential = AzureCliCredential()\n\n # Assumes your script is one level down from root\n root_directory = Path(__file__).resolve().parent\n\n repo = git.Repo(root_directory)\n repo.remotes.origin.pull()\n branch = repo.active_branch.name\n\n # The defined environment values should match the names found in the parameter.yml file\n if branch == \"dev\":\n workspace_id = \"dev-workspace-id\"\n environment = \"DEV\"\n elif branch == \"main\":\n workspace_id = \"prod-workspace-id\"\n environment = \"PROD\"\n else:\n raise ValueError(\"Invalid branch to deploy from\")\n\n # Sample values for FabricWorkspace parameters\n repository_directory = str(root_directory / \"your-workspace-directory\")\n item_type_in_scope = [\"Notebook\", \"DataPipeline\", \"Environment\"]\n\n # Initialize the FabricWorkspace object with the required parameters\n target_workspace = FabricWorkspace(\n workspace_id=workspace_id,\n environment=environment,\n repository_directory=repository_directory,\n item_type_in_scope=item_type_in_scope,\n token_credential=token_credential, # or any other TokenCredential\n )\n\n # Publish all items defined in item_type_in_scope\n publish_all_items(target_workspace)\n\n # Unpublish all items defined in item_type_in_scope not found in repository\n unpublish_all_orphan_items(target_workspace)\n ```\n\n=== \"Azure DevOps\"\n\n ```python\n '''\n Determines variables based on the branch that originated the build\n Uses Azure CLI credential with service connection\n '''\n\n import sys\n import os\n from pathlib import Path\n from azure.identity import AzureCliCredential\n from fabric_cicd import FabricWorkspace, publish_all_items, unpublish_all_orphan_items, change_log_level\n\n # Force unbuffered output like `python -u`\n sys.stdout.reconfigure(line_buffering=True, write_through=True)\n sys.stderr.reconfigure(line_buffering=True, write_through=True)\n\n # Enable debugging if defined in Azure DevOps pipeline\n if os.getenv(\"SYSTEM_DEBUG\", \"false\").lower() == \"true\":\n change_log_level(\"DEBUG\")\n\n # Use Azure CLI credential to authenticate\n token_credential = AzureCliCredential()\n\n # Assumes your script is one level down from root\n root_directory = Path(__file__).resolve().parent\n\n branch = os.getenv(\"BUILD_SOURCEBRANCHNAME\")\n\n # The defined environment values should match the names found in the parameter.yml file\n if branch == \"dev\":\n workspace_id = \"dev-workspace-id\"\n environment = \"DEV\"\n elif branch == \"main\":\n workspace_id = \"prod-workspace-id\"\n environment = \"PROD\"\n else:\n raise ValueError(\"Invalid branch to deploy from\")\n\n # Sample values for FabricWorkspace parameters\n repository_directory = str(root_directory / \"your-workspace-directory\")\n item_type_in_scope = [\"Notebook\", \"DataPipeline\", \"Environment\"]\n\n # Initialize the FabricWorkspace object with the required parameters\n target_workspace = FabricWorkspace(\n workspace_id=workspace_id,\n environment=environment,\n repository_directory=repository_directory,\n item_type_in_scope=item_type_in_scope,\n token_credential=token_credential, # or any other TokenCredential\n )\n\n # Publish all items defined in item_type_in_scope\n publish_all_items(target_workspace)\n\n # Unpublish all items defined in item_type_in_scope not found in repository\n unpublish_all_orphan_items(target_workspace)\n ```\n\n=== \"GitHub\"\n\n ```python\n '''\n Determines variables based on the branch that originated the build\n Uses Azure CLI credential (requires az login in GitHub Actions workflow)\n '''\n\n import os\n from pathlib import Path\n from azure.identity import AzureCliCredential\n from fabric_cicd import FabricWorkspace, publish_all_items, unpublish_all_orphan_items\n\n # Use Azure CLI credential to authenticate\n token_credential = AzureCliCredential()\n\n # GitHub Actions sets GITHUB_WORKSPACE automatically\n root_directory = Path(os.getenv(\"GITHUB_WORKSPACE\", \".\")).resolve()\n\n # Get branch from GitHub environment variable\n branch = os.getenv(\"GITHUB_REF_NAME\")\n\n # The defined environment values should match the names found in the parameter.yml file\n if branch == \"dev\":\n workspace_id = \"dev-workspace-id\"\n environment = \"DEV\"\n elif branch == \"main\":\n workspace_id = \"prod-workspace-id\"\n environment = \"PROD\"\n else:\n raise ValueError(\"Invalid branch to deploy from\")\n\n # Sample values for FabricWorkspace parameters\n repository_directory = str(root_directory / \"your-workspace-directory\")\n item_type_in_scope = [\"Notebook\", \"DataPipeline\", \"Environment\"]\n\n # Initialize the FabricWorkspace object with the required parameters\n target_workspace = FabricWorkspace(\n workspace_id=workspace_id,\n environment=environment,\n repository_directory=repository_directory,\n item_type_in_scope=item_type_in_scope,\n token_credential=token_credential, # or any other TokenCredential\n )\n\n # Publish all items defined in item_type_in_scope\n publish_all_items(target_workspace)\n\n # Unpublish all items defined in item_type_in_scope not found in repository\n unpublish_all_orphan_items(target_workspace)\n ```\n\n## Passed Arguments\n\nLeverage the following when you want to pass in variables outside of the python script. This is most common for scenarios where you want to use one py script, but have multiple deployments.\n\n=== \"Local\"\n\n ```python\n '''Accepts parameters passed into Python during execution'''\n\n import argparse\n from azure.identity import AzureCliCredential\n from fabric_cicd import FabricWorkspace, publish_all_items, unpublish_all_orphan_items\n\n # Use Azure CLI credential to authenticate\n token_credential = AzureCliCredential()\n\n # Accept parsed arguments\n parser = argparse.ArgumentParser(description='Process deployment arguments.')\n parser.add_argument('--workspace_id', type=str, required=True)\n parser.add_argument('--environment', type=str)\n parser.add_argument('--repository_directory', type=str, required=True)\n parser.add_argument('--items_in_scope', type=str)\n args = parser.parse_args()\n\n # Sample values for FabricWorkspace parameters\n workspace_id = args.workspace_id\n environment = args.environment\n repository_directory = args.repository_directory\n item_type_in_scope = args.items_in_scope.split(\",\") if args.items_in_scope else [\"Notebook\", \"DataPipeline\", \"Environment\"]\n\n # Initialize the FabricWorkspace object with the required parameters\n target_workspace = FabricWorkspace(\n workspace_id=workspace_id,\n environment=environment,\n repository_directory=repository_directory,\n item_type_in_scope=item_type_in_scope,\n token_credential=token_credential, # or any other TokenCredential\n )\n\n # Publish all items defined in item_type_in_scope\n publish_all_items(target_workspace)\n\n # Unpublish all items defined in item_type_in_scope not found in repository\n unpublish_all_orphan_items(target_workspace)\n ```\n\n=== \"Azure DevOps\"\n\n ```python\n '''\n Accepts parameters passed into Python during execution\n Uses Azure CLI credential with service connection\n '''\n\n import sys\n import os\n import argparse\n from pathlib import Path\n from azure.identity import AzureCliCredential\n from fabric_cicd import FabricWorkspace, publish_all_items, unpublish_all_orphan_items, change_log_level\n\n # Force unbuffered output like `python -u`\n sys.stdout.reconfigure(line_buffering=True, write_through=True)\n sys.stderr.reconfigure(line_buffering=True, write_through=True)\n\n # Enable debugging if defined in Azure DevOps pipeline\n if os.getenv(\"SYSTEM_DEBUG\", \"false\").lower() == \"true\":\n change_log_level(\"DEBUG\")\n\n # Use Azure CLI credential to authenticate\n token_credential = AzureCliCredential()\n\n # Accept parsed arguments\n parser = argparse.ArgumentParser(description='Process deployment arguments.')\n parser.add_argument('--workspace_id', type=str, required=True)\n parser.add_argument('--environment', type=str)\n parser.add_argument('--repository_directory', type=str, required=True)\n parser.add_argument('--items_in_scope', type=str)\n args = parser.parse_args()\n\n # Sample values for FabricWorkspace parameters\n workspace_id = args.workspace_id\n environment = args.environment\n repository_directory = args.repository_directory\n item_type_in_scope = args.items_in_scope.split(\",\") if args.items_in_scope else [\"Notebook\", \"DataPipeline\", \"Environment\"]\n\n # Initialize the FabricWorkspace object with the required parameters\n target_workspace = FabricWorkspace(\n workspace_id=workspace_id,\n environment=environment,\n repository_directory=repository_directory,\n item_type_in_scope=item_type_in_scope,\n token_credential=token_credential, # or any other TokenCredential\n )\n\n # Publish all items defined in item_type_in_scope\n publish_all_items(target_workspace)\n\n # Unpublish all items defined in item_type_in_scope not found in repository\n unpublish_all_orphan_items(target_workspace)\n ```\n\n=== \"GitHub\"\n\n ```python\n '''Unconfirmed example at this time, however, the Azure DevOps example is a good starting point'''\n ```\n" }, { "path": "docs/example/index.md", "content": "# How To\n\nWelcome to the Examples section! Here you will find all necessary code samples to leverage fabric-cicd. If there is any missing information, raise a [documentation issue](https://github.com/microsoft/fabric-cicd/issues/new?template=3-documentation.yml) on GitHub.\n\n## Contents\n\n\n\n" }, { "path": "docs/example/release_pipeline.md", "content": "# Release Pipeline Examples\n\nThe following are some common examples of how to deploy from tooling like Azure DevOps and GitHub. Note that this is not an exhaustive list, nor is it a recommendation to not use a proper Build/Release stage. These are simplified to show the potential.\n\n## Azure CLI\n\nThis approach uses the Azure CLI Credential Flow. An explicit credential method is required. This avoids ambiguity when multiple identities are present in the build VM.\n\n=== \"Azure DevOps\"\n\n ```yml\n trigger:\n branches:\n include:\n - dev\n - main\n stages:\n - stage: Build_Release\n jobs:\n - job: Build\n pool:\n vmImage: windows-latest\n steps:\n - checkout: self\n - task: UsePythonVersion@0\n inputs:\n versionSpec: '3.12'\n addToPath: true\n - script: |\n pip install fabric-cicd\n displayName: 'Install fabric-cicd'\n - task: AzureCLI@2\n displayName: \"Deploy Fabric Workspace\"\n inputs:\n azureSubscription: \"your-service-connection\"\n scriptType: \"ps\"\n scriptLocation: \"inlineScript\"\n inlineScript: |\n python -u $(System.DefaultWorkingDirectory)/.deploy/fabric_workspace.py\n ```\n\n=== \"GitHub\"\n\n ```yaml\n # Unconfirmed example at this time. The Azure DevOps example is a good starting point.\n ```\n\n## Azure PowerShell\n\nThis approach uses the Azure PowerShell Credential Flow. An explicit credential method is required. This avoids ambiguity when multiple identities are present in the build VM.\n\n=== \"Azure DevOps\"\n\n ```yml\n trigger:\n branches:\n include:\n - dev\n - main\n stages:\n - stage: Build_Release\n jobs:\n - job: Build\n pool:\n vmImage: windows-latest\n steps:\n - checkout: self\n - task: UsePythonVersion@0\n inputs:\n versionSpec: '3.12'\n addToPath: true\n - script: |\n pip install fabric-cicd\n displayName: 'Install fabric-cicd'\n - task: AzurePowerShell@5\n displayName: \"Deploy Fabric Workspace\"\n inputs:\n azureSubscription: \"your-service-connection\"\n scriptType: \"InlineScript\"\n scriptLocation: \"inlineScript\"\n pwsh: true\n Inline: |\n python -u $(System.DefaultWorkingDirectory)/.deploy/fabric_workspace.py\n ```\n\n=== \"GitHub\"\n\n ```yaml\n # Unconfirmed example at this time. The Azure DevOps example is a good starting point.\n ```\n\n## Variable Groups\n\nThis approach is best suited for the Passed Arguments example found in the Deployment Variable Examples, in combination with a `ClientSecretCredential` as shown in the [Authentication Examples](authentication.md). The goal is to define values within the pipeline (or outside the pipeline in Azure DevOps variable groups) and inject them into the python script. Note this also doesn't take a dependency on PowerShell for those organizations or scenarios where PowerShell is not allowed.\n=== \"Azure DevOps\"\n\n ```yml\n trigger:\n branches:\n include:\n - dev\n - main\n\n parameters:\n - name: items_in_scope\n displayName: Enter Fabric items to be deployed\n type: string\n default: '[\"Notebook\",\"DataPipeline\",\"Environment\"]'\n\n variables:\n - group: Fabric_Deployment_Group_KeyVault # Linked to Azure Key Vault and contains tenant id, SPN client id, and SPN secret\n - group: Fabric_Deployment_Group # Contains workspace_name and repository directory name\n\n stages:\n - stage: Build_Release\n jobs:\n - job: Build\n pool:\n vmImage: windows-latest\n steps:\n - checkout: self\n - task: UsePythonVersion@0\n inputs:\n versionSpec: '3.12'\n addToPath: true\n - script: |\n pip install fabric-cicd\n displayName: 'Install fabric-cicd'\n - task: PythonScript@0\n inputs:\n scriptSource: 'filePath'\n scriptPath: '.deploy/fabric_workspace.py'\n arguments: >-\n --spn_client_id $(client_id) # from Fabric_Deployment_Group_KeyVault\n --spn_client_secret $(client_secret) # from Fabric_Deployment_Group_KeyVault\n --tenant_id $(tenant_id) # from Fabric_Deployment_Group_KeyVault\n --workspace_id $(workspace_id) # from Fabric_Deployment_Group\n --environment $(environment_name) # from Fabric_Deployment_Group\n --repository_directory $(repository_directory) # from Fabric_Deployment_Group\n --item_types_in_scope ${{ parameters.items_in_scope }}\n ```\n\n=== \"GitHub\"\n\n ```yaml\n # Unconfirmed example at this time. The Azure DevOps example is a good starting point.\n ```\n" }, { "path": "docs/how_to/config_deployment.md", "content": "# Configuration Deployment\n\n## Overview\n\nConfiguration-based deployment provides an alternative way to manage the deployment of Fabric items across multiple environments. Instead of using the traditional approach of defining a workspace object with various parameters and then running the publish/unpublish functions, this approach centralizes all deployment settings in a single YAML configuration file and simplifies the deployment into one function call.\n\nConfiguration file location (supports any location in the Git repository):\n\n```\nC:/dev/workspace\n /HelloWorld.Notebook\n ...\n /GoodbyeWorld.Notebook\n ...\n /config.yml\n```\n\nBasic example of configuration-based deployment:\n\n> **Note:** All parameters except `config_file_path` must be passed as keyword arguments to `deploy_with_config()`.\n\n```python\nfrom fabric_cicd import deploy_with_config\nfrom azure.identity import AzureCliCredential\n\n# Deploy using a config file\ndeploy_with_config(\n config_file_path=\"C:/dev/workspace/config.yml\", # required\n token_credential=AzureCliCredential(), # required\n environment=\"dev\"\n)\n```\n\nRaise a [feature request](https://github.com/microsoft/fabric-cicd/issues/new?template=2-feature.yml) for additional capabilities or a [bug report](https://github.com/microsoft/fabric-cicd/issues/new?template=1-bug.yml) for issues.\n\n## Configuration File Setup\n\nThe configuration file includes several sections with configurable settings for different aspects of the deployment process.\n\n> **Note:** Configuration values can be specified in two ways: as a single value (applied to any target environment provided) or as an environment mapping. Both approaches can be used within the same configuration file — for example, using environment mappings for workspace IDs while keeping a single value for repository directory.\n\n### Core Settings\n\nThe `core` section is **required** as it defines the fundamental settings for the deployment, most importantly the **target workspace** and **repository directory**. Other optional settings can be configured within the `core` section, including **item types in scope** and **parameter**.\n\n```yaml\ncore:\n # Only one workspace identifier field is required\n workspace: \n\n workspace_id: \n\n # Required - path to the directory containing Fabric items\n repository_directory: \n\n # Optional - specific item types to include in deployment\n item_types_in_scope:\n - \n - \n - \n\n # Optional - path to parameter file\n parameter: \n```\n\nWith environment mapping:\n\n```yaml\ncore:\n # Only one workspace identifier field is required\n workspace:\n : \n : \n\n workspace_id:\n : \n : \n\n # Required - path to the directory containing Fabric items\n repository_directory:\n : \n : \n\n # Optional - specific item types to include in deployment\n item_types_in_scope:\n :\n - \n - \n :\n - \n - \n\n # Optional - path to parameter file\n parameter:\n : \n : \n```\n\nRequired Fields:\n\n- **Workspace Identifier:**\n - Workspace ID takes precedence over workspace name when both are provided.\n - `workspace_id` must be a valid string GUID.\n- **Repository Directory Path:**\n - Supports relative or absolute path.\n - Relative path must be relative to the `config.yml` file location.\n\nOptional Fields:\n\n- **Item Types in Scope:**\n - If `item_types_in_scope` is not specified, all item types will be included by default.\n - Item types must be provided as a list; use `-` or `[]` notation.\n - Only accepts supported item types.\n- **Parameter Path:**\n - Supports relative or absolute path.\n - Relative path must be relative to the `config.yml` file location.\n\n### Publish Settings\n\nThe `publish` section is optional and controls item publishing behavior. If this section is omitted entirely, publishing will run with **default behavior — all items published, no exclusions.** It includes various optional settings to enable/disable publishing operations or selectively publish items.\n\n> **Note:** `folder_exclude_regex` and `folder_path_to_include` are mutually exclusive — providing both for the same environment will result in a validation error. For detailed information about folder and item filtering behavior, see [Selective Deployment Features](optional_feature.md#selective-deployment-features).\n\n```yaml\npublish:\n # Optional - pattern to exclude items from publishing (no feature flag required)\n exclude_regex: \n\n # Optional - pattern to exclude specific folder paths with items from publishing (requires feature flags)\n folder_exclude_regex: \n\n # Optional - specific folder paths with items to publish (requires feature flags)\n folder_path_to_include:\n - \n - \n - # publish items found in nested folder - subfolder_3\n\n # Optional - specific items to publish (requires feature flags)\n items_to_include:\n - \n - \n\n # Optional - pattern to exclude Lakehouse shortcuts from publishing (requires feature flags)\n shortcut_exclude_regex: \n\n # Optional - control publishing by environment\n skip: \n```\n\nWith environment mapping:\n\n```yaml\npublish:\n # Optional - pattern to exclude items from publishing\n exclude_regex:\n : \n : \n\n # Optional - pattern to exclude specific folder paths with items from publishing (requires feature flags)\n folder_exclude_regex:\n : \n : \n\n # Optional - specific folder paths with items to publish (requires feature flags)\n folder_path_to_include:\n :\n - \n - \n :\n - \n\n # Optional - specific items to publish (requires feature flags)\n items_to_include:\n :\n - \n - \n :\n - \n - \n\n # Optional - pattern to exclude Lakehouse shortcuts from publishing (requires feature flags)\n shortcut_exclude_regex:\n : \n : \n\n # Optional - control publishing by environment\n skip:\n : \n : \n```\n\n### Unpublish Settings\n\nThe `unpublish` section is optional and controls item unpublishing behavior. If this section is omitted entirely, unpublishing will run with **default behavior — all orphan items unpublished, no exclusions.** It includes various optional settings to enable/disable unpublishing or selectively unpublish items.\n\n```yaml\nunpublish:\n # Optional - pattern to exclude items from unpublishing (no feature flag required)\n exclude_regex: \n\n # Optional - specific items to unpublish (requires feature flags)\n items_to_include:\n - \n - \n\n # Optional - control unpublishing by environment\n skip: \n```\n\nWith environment mapping:\n\n```yaml\nunpublish:\n # Optional - pattern to exclude items from unpublishing\n exclude_regex:\n : \n : \n\n # Optional - specific items to unpublish (requires feature flags)\n items_to_include:\n :\n - \n - \n :\n - \n - \n\n # Optional - control unpublishing by environment\n skip:\n : \n : \n```\n\n> **Warning:** While selective deployment is supported in fabric-cicd, it is not recommended due to potential issues with dependency management.\n\n### Features Setting\n\nThe `features` section is optional and allows you to set a list of specific feature flags.\n\n```yaml\nfeatures:\n - \n - \n```\n\nWith environment mapping:\n\n```yaml\nfeatures:\n :\n - \n - \n :\n - \n - \n```\n\n### Constants Setting\n\nThe `constants` section is optional and allows you to override supported library constants.\n\n```yaml\nconstants:\n CONSTANT_NAME: \n```\n\nWith environment mapping:\n\n```yaml\nconstants:\n CONSTANT_NAME:\n : \n : \n```\n\n## Environment-Specific Values\n\nAll configuration fields support environment-specific values using a mapping format:\n\n```yaml\ncore:\n workspace_id:\n dev: \"dev-workspace-id\"\n test: \"test-workspace-id\"\n prod: \"prod-workspace-id\"\n```\n\n### Required vs Optional Fields\n\nFields are categorized as **required** or **optional**, which affects how missing environment values are handled when environment is passed into `deploy_with_config()`:\n\n> **Note:** When the `publish` or `unpublish` sections are omitted entirely, both operations run by default. To skip either operation, explicitly set `skip: true` for that section.\n\n| Field | Required | Environment Missing Behavior |\n| --------------------------------------- | -------- | ------------------------------- |\n| `core.workspace_id` or `core.workspace` | ✅ | Validation error |\n| `core.repository_directory` | ✅ | Validation error |\n| `core.item_types_in_scope` | ❌ | Warning logged, setting skipped |\n| `core.parameter` | ❌ | Warning logged, setting skipped |\n| `publish.exclude_regex` | ❌ | Debug logged, setting skipped |\n| `publish.folder_exclude_regex` | ❌ | Debug logged, setting skipped |\n| `publish.shortcut_exclude_regex` | ❌ | Debug logged, setting skipped |\n| `publish.folder_path_to_include` | ❌ | Debug logged, setting skipped |\n| `publish.items_to_include` | ❌ | Debug logged, setting skipped |\n| `publish.skip` | ❌ | Defaults to `False` |\n| `unpublish.exclude_regex` | ❌ | Debug logged, setting skipped |\n| `unpublish.items_to_include` | ❌ | Debug logged, setting skipped |\n| `unpublish.skip` | ❌ | Defaults to `False` |\n| `features` | ❌ | Warning logged, setting skipped |\n| `constants` | ❌ | Warning logged, setting skipped |\n\n### Selective Environment Configuration\n\nOptional fields allow you to apply settings to specific environments without affecting others. This is useful when you want different behavior per environment:\n\n```yaml\ncore:\n workspace_id:\n dev: \"dev-workspace-id\"\n test: \"test-workspace-id\"\n prod: \"prod-workspace-id\"\n repository_directory: \"./workspace\" # Same for all environments\n\npublish:\n # Only exclude legacy folders in prod environment\n folder_exclude_regex:\n prod: \"^/legacy_.*\"\n # dev and test not specified - no folder exclusion applied\n\n # Skip publish in dev, run in test and prod\n skip:\n dev: true\n # test and prod default to false\n```\n\nIn this example:\n\n- Deploying to `dev`: No folder exclusion applied, `skip` = `true`\n- Deploying to `test`: No folder exclusion applied, `skip` = `false`\n- Deploying to `prod`: `folder_exclude_regex` = `\"^/legacy_.*\"`, `skip` = `false`\n\n### Logging Behavior\n\nWhen an optional field uses environment mapping and does not include the target environment:\n\n- **Important optional fields** (`item_types_in_scope`, `parameter`): A **warning** is logged to alert users that the setting is being skipped.\n- **Other optional fields**: A **debug** message is logged, visible only when debug logging is enabled.\n\nExample log output when deploying to `prod` with the configuration above:\n\n```\n[Debug] - No value for 'folder_exclude_regex' in environment 'prod'. Available environments: ['dev']. This setting will be skipped.\n```\n\nTo enable debug logging:\n\n```python\nfrom fabric_cicd import change_log_level\nchange_log_level()\n```\n\n## Sample `config.yml` File\n\n```yaml\ncore:\n workspace:\n dev: \"Fabric-Dev-Engineering\"\n test: \"Fabric-Test-Engineering\"\n prod: \"Fabric-Prod-Engineering\"\n\n workspace_id:\n dev: \"8b6e2c7a-4c1f-4e3a-9b2e-7d8f2e1a6c3b\"\n test: \"2f4b9e8d-1a7c-4d3e-b8e2-5c9f7a2d4e1b\"\n prod: \"7c3e1f8b-2d4a-4b9e-8f2c-1a6c3b7d8e2f\"\n\n repository_directory: \".\" # relative path\n\n item_types_in_scope:\n - Notebook\n - DataPipeline\n - Environment\n - Lakehouse\n\n parameter: \"parameter.yml\" # relative path\n\npublish:\n # Don't publish items matching this pattern (no feature flag required)\n exclude_regex: \"^DONT_DEPLOY.*\"\n\n # Use folder_exclude_regex OR folder_path_to_include, not both for the same environment\n folder_exclude_regex:\n dev: \"^/DONT_DEPLOY_FOLDER\"\n\n folder_path_to_include:\n prod:\n - \"/DEPLOY_FOLDER\"\n - \"/DEPLOY_FOLDER/DEPLOY_NESTED_FOLDER\"\n\n items_to_include:\n - \"Hello World.Notebook\"\n - \"Run Hello World.DataPipeline\"\n\n shortcut_exclude_regex:\n test: \"^temp_.*\"\n\n skip:\n dev: true\n test: false\n prod: false\n\nunpublish:\n # Don't unpublish items matching this pattern (no feature flag required)\n exclude_regex: \"^DEBUG.*\"\n\n skip:\n dev: false\n test: false\n prod: true\n\nfeatures:\n - enable_shortcut_publish\n - enable_experimental_features\n - enable_items_to_include\n - enable_exclude_folder\n - enable_include_folder\n - enable_shortcut_exclude\n\nconstants:\n DEFAULT_API_ROOT_URL: \"https://api.fabric.microsoft.com\"\n```\n\n## Configuration File Deployment\n\n### Basic Usage\n\n```python\nfrom fabric_cicd import deploy_with_config\nfrom azure.identity import AzureCliCredential\n\n# Deploy using a config file\ndeploy_with_config(\n config_file_path=\"path/to/config.yml\", # required\n token_credential=AzureCliCredential(), # required\n environment=\"dev\" # optional (recommended)\n)\n```\n\n### Custom Authentication\n\n```python\nfrom fabric_cicd import deploy_with_config\nfrom azure.identity import ClientSecretCredential\n\n# Create a credential\ncredential = ClientSecretCredential(\n tenant_id=\"your-tenant-id\",\n client_id=\"your-client-id\",\n client_secret=\"your-client-secret\"\n)\n\n# Deploy with custom credential\ndeploy_with_config(\n config_file_path=\"path/to/config.yml\",\n token_credential=credential,\n environment=\"prod\"\n)\n```\n\n### Configuration Override\n\nThe `config_override` parameter in `deploy_with_config()` allows you to dynamically modify configuration values at runtime without changing the base configuration file. This is particularly useful for debugging or making temporary deployment adjustments.\n\n```python\nfrom fabric_cicd import deploy_with_config\nfrom azure.identity import AzureCliCredential\n\nconfig_override_dict = {\n \"core\": {\n \"item_types_in_scope\": [\"Notebook\", \"DataPipeline\"]\n },\n \"publish\": {\n \"skip\": {\n \"dev\": False\n }\n }\n}\n\n# Deploy with configuration override\ndeploy_with_config(\n config_file_path=\"path/to/config.yml\",\n token_credential=AzureCliCredential(),\n environment=\"dev\",\n config_override=config_override_dict\n)\n```\n\n**Important Considerations:**\n\n- **Caution:** Exercise caution when overriding configuration values for _production_ environments.\n- **Support:** Configuration overrides are supported for all sections and settings in the configuration file.\n- **Rules:**\n - Existing values can be overridden for any field in the configuration.\n - New values can only be added for optional fields that aren't present in the original configuration.\n - Required fields must exist in the original configuration in order to override.\n\n## Troubleshooting Guide\n\nThe configuration file undergoes validation prior to reaching the deployment phase. Here are some common issues that may occur:\n\n1. **File Not Found:** Ensure the configuration file path is correct and accessible (must be an absolute path).\n\n2. **Invalid YAML:** Check YAML syntax for errors (indentation, missing quotes, etc.).\n\n3. **Missing Required Fields:** Ensure the `core` section is present and contains the required fields (workspace identifier, repository directory path).\n\n4. **Path Resolution Errors:** Relative paths are resolved relative to the `config.yml` file location. Check that path inputs are valid and accessible.\n\n5. **Environment Not Found:** The `environment` parameter must match one of the environment keys (e.g., \"dev\", \"test\", \"prod\") used in the configuration mappings for required fields (`workspace`/`workspace_id`, `repository_directory`).\n" }, { "path": "docs/how_to/getting_started.md", "content": "# Getting Started\n\n## Installation\n\nTo install fabric-cicd, run:\n\n```bash\npip install fabric-cicd\n```\n\n## Authentication\n\n> **⚠️ NOTICE**: Due to security best practices, the **Default Credential** (`DefaultAzureCredential` fallback) and **implicit Fabric Notebook authentication** (without a `token_credential` parameter) methods are no longer supported. `token_credential` is now a required parameter.\n\n- You must provide your own credential object that aligns with the `TokenCredential` class (from [azure.identity](https://learn.microsoft.com/en-us/python/api/azure-identity/azure.identity?view=azure-python)). For more details, see the [TokenCredential](https://learn.microsoft.com/en-us/python/api/azure-core/azure.core.credentials.tokencredential?view=azure-python) documentation.\n- When running in Fabric Notebook runtime, provide an explicit credential. See Authentication examples for details.\n\n**Recommended Authentication Methods:**\n\n- For local development: `AzureCliCredential` or `AzurePowerShellCredential` (user authentication)\n- For CI/CD pipelines: `AzureCliCredential`/`AzurePowerShellCredential` (platform authentication), `ClientSecretCredential` (service principal), or `ManagedIdentityCredential` (self-hosted agents)\n\n**Basic Example:**\n\n```python\nfrom azure.identity import AzureCliCredential\nfrom fabric_cicd import FabricWorkspace\n\ntoken_credential = AzureCliCredential()\n\nworkspace = FabricWorkspace(\n workspace_id=\"your-workspace-id\",\n environment=\"your-target-environment\",\n repository_directory=\"your-repository-directory\",\n item_type_in_scope=[\"Notebook\", \"DataPipeline\", \"Environment\"],\n token_credential=token_credential, # or any other TokenCredential\n)\n```\n\nSee the [Authentication Examples](../example/authentication.md) for specific implementation patterns.\n\n## Directory Structure\n\nThis library deploys from a directory containing files and directories committed via the Fabric Source Control UI. Ensure the `repository_directory` includes only these committed items, with the exception of the `parameter.yml` file.\n\n```\n/\n /.\n ...\n /.\n ...\n /\n /.\n ...\n /.\n ...\n /parameter.yml\n```\n\n## GIT Flow\n\nThe flow pictured below is the hero scenario for this library and is the recommendation if you're just starting out.\n\n- `Deployed` branches are not connected to workspaces via [GIT Sync](https://learn.microsoft.com/en-us/fabric/cicd/git-integration/git-get-started?tabs=azure-devops%2CAzure%2Ccommit-to-git#connect-a-workspace-to-a-git-repo)\n- `Feature` branches are connected to workspaces via [GIT Sync](https://learn.microsoft.com/en-us/fabric/cicd/git-integration/git-get-started?tabs=azure-devops%2CAzure%2Ccommit-to-git#connect-a-workspace-to-a-git-repo)\n- `Deployed` workspaces are only updated through script-based deployments, such as through the fabric-cicd library\n- `Feature` branches are created from the default branch, merged back into the default `Deployed` branch, and cherry picked into the upper `Deployed` branches\n- Each deployment is a full deployment and does not consider commit diffs\n\n![GIT Flow](../config/assets/git_flow.png)\n" }, { "path": "docs/how_to/index.md", "content": "# How To\n\nWelcome to the How To section! Here you will find all necessary information to leverage fabric-cicd. If there is any missing information, raise a [documentation issue](https://github.com/microsoft/fabric-cicd/issues/new?template=3-documentation.yml) on GitHub.\n\n## Contents\n\n\n\n" }, { "path": "docs/how_to/item_types.md", "content": "# Item Types\n\n## Activator\n\n- **Parameterization:**\n - The `find_replace` section in the `parameter.yml` file is not applied.\n- **Initial deployment** may not reflect streaming data immediately.\n- **Reflex** is the item name in source control. Source control may not support all activators/reflexes, as not all sources are compatible.\n\n## Apache Airflow Job\n\n- **Parameterization:**\n - The referenced items in DAG files will always point to the original item unless parameterized in the `find_replace` section of the `parameter.yml` file.\n- **Connections** are not source controlled and must be created manually.\n- See known CI/CD limitations [here](https://learn.microsoft.com/en-us/fabric/data-factory/cicd-apache-airflow-jobs#known-limitations).\n\n## API for GraphQL\n\n- **Parameterization:**\n - The source will always point to the source in the original workspace unless parameterized in the `find_replace` section of the `parameter.yml` file.\n - It is recommended to use the supported variables in `find_replace` for dynamic replacement of the source workspace and item IDs.\n - If you are using a connection and expect it to change between different environments, then it needs to be parameterized in the `parameter.yml` file.\n- When using the **Saved Credential** method to connect to data sources, developers must have access to the Saved Credential information in order to successfully deploy GraphQL item.\n- Changes made to the original API query are not source controlled. You will need to manually update the query in the GraphQL item's query editor within the target workspace.\n- Only user authentication is currently supported for GraphQL items that source data from the SQL Analytics Endpoint.\n\n## Copy Job\n\n- **Parameterization:**\n - Connections will always point to the original data source unless parameterized in the `find_replace` section of the `parameter.yml` file.\n- **Initial deployment** requires manual configuration of the connection after deployment.\n\n## Dataflow\n\n- **Parameterization:**\n - Source/destination items (e.g., Dataflow, Lakehouse, Warehouse) will always reference the original item unless parameterized in the `find_replace` section of the `parameter.yml` file.\n - The recommended approach for re-pointing source/destination items that exist in the _same_ workspace as the Dataflow is to use **`replace_value` variables** in the `find_replace` parameter along with a `find_value` regex (literal string works too). For more guidance, see [Parameterization -> Dataflows](parameterization.md#dataflows).\n - **Important** for Dataflows that reference another Dataflow in the _same_ workspace, ONLY parameterize the referenced **dataflowId** in the `mashup.pq` file (workspaceId re-pointing is handled automatically in this approach) using the following `replace_value`: `$items.Dataflow..id`. This ensures proper dependency resolution and ordered publishing.\n- **Initial deployment** will require a manual publish and refresh of the dataflow. After re-pointing dataflows during deployment, temporary errors may appear but should resolve after refreshing and allowing time for processing (especially when a dataflow sources from another dataflow in the target workspace).\n- `fabric ci-cd` automatically manages ordered deployment of dataflows that source from other dataflows in the same workspace.\n- **Connections** are not source controlled and require manual creation.\n- If you use connections that differ between environments, parameterize them in the `parameter.yml` file.\n\n## Data Agent\n\n- **Parameterization:**\n - Data source items (e.g., Lakehouse, Warehouse, Semantic Model) will always reference the original item unless parameterized in the `find_replace` section of the `parameter.yml` file.\n\n## Data Build Tool Job\n\n- **Parameterization:**\n - Connection and profile references will always point to the original values unless parameterized in the `find_replace` section of the `parameter.yml` file.\n- **Initial deployment:**\n - Validate connection/profile settings in the target workspace, especially when promoting between test and prod workspaces.\n\n## Data Pipeline\n\n- **Parameterization:**\n - Activities connected to items that exist in a different workspace will always point to the original item unless parameterized in the `find_replace` section of the `parameter.yml` file.\n - Activities connected to items within the same workspace are automatically re-pointed to the new item in the target workspace. **Note: Activities referencing items that don't use _logical ID_ and _default workspace ID_ (such as Refresh Dataflow and Refresh Semantic Model) require parameterization.**\n- **Connections** are not source controlled and must be created manually.\n- If you are using connections and expect them to change between different environments, then those need to be parameterized in the `parameter.yml` file.\n- The **executing identity** of the deployment must have access to the connections, or the deployment will fail.\n\n## Environment\n\n- **Parameterization:**\n - Environments attached to custom spark pools attach to the default starter pool unless parameterized in the `spark_pools` section of the `parameter.yml` file.\n - The `find_replace` section in the `parameter.yml` file is not applied to Environments.\n- **Resources** are not source controlled and will not be deployed.\n- Environments with libraries will have **high initial publish times** (sometimes 20+ minutes).\n\n## Eventhouse\n\n- **Parameterization:**\n - The `find_replace` section in the `parameter.yml` file is not applied.\n- The `exclude_path` variable is required when deploying an **Eventhouse** that is attached to a **KQL Database** (common scenario).\n- There may be significant _differences_ in the streaming data between the source eventhouse and the deployed eventhouse.\n- **Unpublish** is disabled by default, enable with feature flag `enable_eventhouse_unpublish`.\n\n## Eventstream\n\n- **Parameterization:**\n - Destinations connected to items that exist in a different workspace will always point to the original item unless parameterized in the `find_replace` section of the `parameter.yml` file.\n - Destinations connected to items within the same workspace are re-pointed to the new item in the target workspace.\n- **Initial deployment** requires waiting for the table to populate in the destination lakehouse if a lakehouse destination is present in the eventstream.\n\n## KQL Database\n\n- **Parameterization:**\n - The `find_replace` section in the `parameter.yml` file is not applied.\n- In Fabric, a KQL database is not a standalone item. However, during deployment, it is treated as such. Its source control files are located within a `.children` folder under the directory of the attached eventhouse.\n- Data in KQL database tables is not source controlled and may not consistently appear in the database UI after deployment. Some tables may be empty post-deployment.\n\n## KQL Queryset\n\n- **Parameterization:**\n - KQL querysets attached to KQL databases always point to the original KQL database unless parameterized in the `find_replace` section of the `parameter.yml` file.\n- The **cluster/query URI** of the KQL database must be present in the KQL queryset JSON for rebinding. If the KQL queryset is attached to a KQL database within the same workspace, the cluster URI value is empty and needs to be re-added. `fabric ci-cd` handles this automatically.\n- KQL querysets can still exist after the KQL database source has been deleted. However, errors will reflect in the KQL queryset.\n\n## Lakehouse\n\n- **Parameterization:**\n - The `find_replace` section in the `parameter.yml` file is not applied.\n- **Shortcut** publish is disabled by default (for now), enable with feature flag `enable_shortcut_publish`.\n- **Schemas are not deployed** unless the schema has a shortcut present.\n- **Unpublish** is disabled by default, enable with feature flag `enable_lakehouse_unpublish`.\n\n## Mirrored Database\n\n- **Parameterization:**\n - Connections will always point to the original source database unless parameterized in the `find_replace` section of the `parameter.yml` file.\n- **Initial deployment** for Azure SQL Database or Azure SQL Managed Instance requires manual granting of System Assigned Managed Identity (SAMI) Read and Write permission to the mirrored database for replication to be successful after deployment. ref -> ([Prerequisites](https://learn.microsoft.com/en-us/fabric/database/mirrored-database/mirrored-database-rest-api#create-mirrored-database))\n- **Unpublish** - a warning is shown for any default Semantic Models created by the Mirror Database. This is a current limitation of the Fabric API and can be ignored.\n\n## ML Experiment\n\n- **Parameterization:**\n - The `find_replace` section in the `parameter.yml` file is not applied.\n- **Only the ML Shell is created.** The create API does not support the creation an machine learning experiment with definition.\n\n## Mounted Data Factory\n\n- **Parameterization:**\n - The `find_replace` section in the `parameter.yml` file is not applied.\n- Deploys a **mounted** Azure Data Factory to a Fabric Workspace.\n- Before deployment, ensure the Azure Data Factory resource exists with proper permissions.\n\n## Notebook\n\n- **Parameterization:**\n - Notebooks attached to lakehouses always point to the original lakehouse unless parameterized in the `find_replace` section of the `parameter.yml` file.\n- **Resources** are not source controlled and will not be deployed.\n- **Note:** Both `.py` and `.ipynb` formats are supported. Git-integrated Notebooks only support `.py`, while Notebooks exported via the Get Item Definition API support both formats.\n\n## Ontology\n\n- **Parameterization:**\n - Referenced items that exist in a different workspace will always point to the original item unless parameterized in the `find_replace` section of the `parameter.yml` file.\n - Referenced items within the same workspace are automatically re-pointed to the new item in the target workspace.\n\n## Real-Time Dashboard\n\n- **Parameterization:**\n - Real-Time Dashboard attached to KQL databases always point to the original KQL database unless parameterized in the `find_replace` section of the `parameter.yml` file.\n- The **cluster/query URI** of the KQL database(s) used in the dashboard must be present in the Real-Time Dashboard JSON for rebinding. If the Real-Time Dashboard is attached to a KQL database within the same workspace, the cluster URI value is empty and needs to be re-added. `fabric ci-cd` handles this automatically.\n\n## Report\n\n- **Parameterization:**\n - Reports connected to Semantic Models outside of the same workspace always point to the original Semantic Model unless parameterized in the `parameter.yml` file.\n - Reports with `byPath` references to Semantic Models within the same workspace are automatically re-pointed to the new item in the target workspace.\n - Reports with `byConnection` references to Semantic Models within or outside of the same workspace can be parameterized using `find_replace` or `key_value_replace` parameters to update workspace IDs, semantic model names, and semantic model IDs in the connection string. See [Reports Parameterization Example](parameterization.md#reports).\n\n## Semantic Model\n\n- **Parameterization:**\n - Semantic Models connected to sources outside of the same workspace always point to the original item unless parameterized in the `find_replace` section of the `parameter.yml` file.\n - Semantic Models connected to sources within the same workspace may or may not be re-pointed; it is best to test this before taking a dependency. Use the `find_replace` section of the `parameter.yml` file as needed.\n- **Automatic Connection Binding:**\n - Use the `semantic_model_binding` parameter in `parameter.yml` to automatically bind Semantic Model connections after deployment, removing the need for manual connection configuration on initial deployment.\n - **Note:** Only a single connection binding per Semantic Model is currently supported. If your Semantic Model uses multiple connections, additional connections must be configured manually after deployment.\n - See [Parameterization -> semantic_model_binding](parameterization.md#semantic_model_binding) for configuration details.\n- **Initial deployment** requires manual configuration of the connection after deployment **unless** `semantic_model_binding` is configured in the `parameter.yml` file.\n\n## Spark Job Definition\n\n- **Parameterization:**\n - Spark Job Definitions attached to lakehouses always point to the original lakehouse unless parameterized in the `find_replace` section of the `parameter.yml` file.\n - When connected to an Environment within the same workspace, the deployed Spark Job Definition will be re-pointed to the new Environment in the target workspace.\n- **File Types Supported:** The v2 API which is used only supports Spark Job Definitions with file formats of `.py` or `.scala`. The `.jar` file format isn't supported.\n\n## SQL Database\n\n- **Parameterization:**\n - The `find_replace` section in the `parameter.yml` file is not applied.\n- **SQL Database content is not deployed.** Only the item shell is deployed. The SQL database schema (DDL) must be deployed separately using a DACPAC or other tools such as dbt.\n- **Unpublish** is disabled by default, enable with feature flag `enable_sqldatabase_unpublish`.\n\n## User Data Function\n\n- **Parameterization:**\n - Connection references and libraries in the `definitions.json` file will always point to the original items unless parameterized in the `find_replace` section of the `parameter.yml` file.\n - **Important:** When parameterizing connections or libraries that reference items in different workspaces, use the appropriate `replace_value` variables with workspace and item IDs.\n- **Deployment**:\n - Includes all connections and libraries specified in the source workspace.\n - **Important:** Due to the nature of UserDataFunctions they could take up to a few minutes to publish.\n- **Metadata and configuration** are automatically managed through the `functions.json` file in the resources folder.\n\n## Variable Library\n\n- **Parameterization:**\n - The active value set of the variable library is defined by the `environment` field passed into the `FabricWorkspace` object. If no `environment` is specified, the active Value Set will not be changed.\n- **Changing Value Sets:**\n - Variable Libraries do not support programmatically changing the name of value set which is active\n - After the initial deployment, if an active set is renamed, or removed, the deployment will fail\n - Manual intervention will be required to make the necessary changes in the Fabric UI and then restart the deployment\n\n## Warehouse\n\n- **Parameterization:**\n - The `find_replace` section in the `parameter.yml` file is not applied.\n- **Warehouse content is not deployed.** Only the item shell is deployed. Warehouse DDL must be deployed separately using a DACPAC or other tools such as dbt.\n- **Case insensitive collation is supported** custom collation must be manually edited in the `.platform` file creation payload. See [How to: Create a warehouse with case-insensitive (CI) collation](https://learn.microsoft.com/en-us/fabric/data-warehouse/collation) for more details.\n- **Unpublish** is disabled by default, enable with feature flag `enable_warehouse_unpublish`.\n" }, { "path": "docs/how_to/optional_feature.md", "content": "# Optional Features\n\nfabric-cicd has an expected default flow; however, there will always be cases where overriding default behavior is required.\n\n## Feature Flags\n\nFor scenarios that aren't supported by default, fabric-cicd offers feature flags. Below is an exhaustive list of currently supported features.\n\n| Flag Name | Description | Experimental |\n| ----------------------------------------- | --------------------------------------------------------------------------------------------------------- | ------------ |\n| `enable_lakehouse_unpublish` | Set to enable the deletion of Lakehouses | |\n| `enable_warehouse_unpublish` | Set to enable the deletion of Warehouses | |\n| `enable_sqldatabase_unpublish` | Set to enable the deletion of SQL Databases | |\n| `enable_eventhouse_unpublish` | Set to enable the deletion of Eventhouses | |\n| `enable_kqldatabase_unpublish` | Set to enable the deletion of KQL Databases (attached to Eventhouses) | |\n| `enable_shortcut_publish` | Set to enable deploying shortcuts with the Lakehouse | |\n| `enable_environment_variable_replacement` | Set to enable the use of pipeline variables | |\n| `disable_workspace_folder_publish` | Set to disable deploying workspace sub folders | |\n| `enable_experimental_features` | Set to enable experimental features, such as selective deployments | |\n| `enable_items_to_include` | Set to enable selective publishing/unpublishing of items | ☑️ |\n| `enable_exclude_folder` | Set to enable folder-based exclusion during publish operations | ☑️ |\n| `enable_include_folder` | Set to enable folder-based inclusion during publish operations | ☑️ |\n| `enable_shortcut_exclude` | Set to enable selective publishing of shortcuts in a Lakehouse | ☑️ |\n| `enable_response_collection` | Set to enable collection of API responses during publish and unpublish operations | |\n| `continue_on_shortcut_failure` | Set to allow deployment to continue even when shortcuts fail to publish | |\n| `enable_hard_delete` | Set to enable hard deletion of items, bypassing the workspace recycle bin. Requires workspace Admin role. | |\n\nExample\n\n```python\nfrom fabric_cicd import append_feature_flag\nappend_feature_flag(\"enable_lakehouse_unpublish\")\nappend_feature_flag(\"enable_warehouse_unpublish\")\nappend_feature_flag(\"enable_environment_variable_replacement\")\nappend_feature_flag(\"enable_response_collection\")\n```\n\n## Selective Deployment Features\n\nBy default, fabric-cicd performs a full deployment of all repository items. Selective deployment is an experimental feature due to the risk of deploying Fabric items that have dependencies on other items, which can result in broken deployments. These features support a range of filtering options, from broader folder-based selection to more granular item-level and shortcut-level filtering. To use these features, you must enable both the `enable_experimental_features` flag and the specific feature flag (if applicable).\n\n**Warning:** Selective deployment is not recommended due to potential issues with dependency management.\n\n### Folder-Level Filtering\n\nA subset of items in the repository that exist within a Fabric workspace folder can be published using one of the following experimental features. Only one of these features can be applied during a deployment. Use case: selectively deploy a **group** of Fabric items (must be contained within folders). Folder-based item exclusion/inclusion is not supported in the unpublish scenario.\n\n1. **`folder_path_exclude_regex`**\n - Optional parameter in `publish_all_items()`, set to a regex pattern that matches Fabric folder path(s) containing items in the repository.\n - Requires the `enable_exclude_folder` feature flag.\n - The folder path(s) and items contained within that match the regex will be excluded from the publish operation.\n - When using `folder_path_exclude_regex`, the pattern is matched using `search()` (substring match), so a pattern like `subfolder1` will match any folder path containing \"subfolder1\" (e.g., `/subfolder1`, `/subfolder1/subfolder2`, `/other/subfolder1`).\n - To target a specific folder, use an anchored pattern (e.g., `^/subfolder1$`) — this ensures only the exact folder path matches.\n - Child folders like `/subfolder1/subfolder2` will also be excluded automatically since their parent folder was excluded, preserving a consistent folder hierarchy.\n\n2. **`folder_path_to_include`**\n - Optional parameter in `publish_all_items()`, set to a list of strings that exactly match the folder path(s) containing items in the repository.\n - Requires the `enable_include_folder` feature flag.\n - Folder paths must start with `/` (e.g., `/folder_name` or `/folder_name/nested_folder`). The matching folder path(s) and their contained items will be included in the publish operation; any other items contained within Fabric folders will be excluded.\n - When using `folder_path_to_include` with nested paths (e.g., `/subfolder1/subfolder2`), ancestor folders (e.g., `/subfolder1`) are automatically created to preserve the correct folder hierarchy, but items directly under the ancestor folder are **not** published unless the ancestor folder is also explicitly included in the list.\n\n**Note:** `folder_path_exclude_regex` and `folder_path_to_include` are mutually exclusive and cannot be used together for the same deployment. These filters are ignored when the `disable_workspace_folder_publish` feature flag is set. Folder-based filtering does not impact standalone items.\n\n### Item-Level Filtering\n\nA subset of items in the repository can be published/unpublished using one of the following features. Both features are technically supported, but **it is recommended to use one feature per deployment to avoid unexpected results**.\n\n1. **`item_name_exclude_regex`**\n - Optional parameter in `publish_all_items()` and `unpublish_all_orphan_items()`, set to a regex pattern that matches item name(s) found in the repository.\n - **This feature does not require any feature flags.**\n - Fabric items that match the regex will be excluded from the publish/unpublish operation.\n\n2. **`items_to_include`**\n - Optional parameter in `publish_all_items()` and `unpublish_all_orphan_items()`, set to a list of strings that exactly match items in the repository.\n - Requires the `enable_items_to_include` feature flag.\n - Must be in the format: `\"item_name.item_type\"`. The matching item(s) will be included in the publish/unpublish operation.\n\n**Note:** `item_name_exclude_regex` and `items_to_include` can be applied to items within Fabric folders or standalone items. Item-level filtering can be combined with folder-level filtering, but be cautious when using both to avoid unexpected results.\n\n### Filter Precedence\n\nFilters are evaluated in the following order:\n\n1. **`items_to_include`** — Scope is narrowed upfront; only explicitly listed items proceed to further checks\n2. **`item_name_exclude_regex`** — Items matching the regex are excluded\n3. **`folder_path_exclude_regex`** — Items in matching folders are excluded\n4. **`folder_path_to_include`** — Only items in specified folders are published\n\n**Note:** `folder_path_exclude_regex` and `folder_path_to_include` are mutually exclusive — only one can be used per deployment. Standalone items (items not in any folder) are not impacted by folder-level filters. When `items_to_include` is combined with exclusion filters, an item must first be in the include list before exclusion filters are evaluated against it.\n\n### Lakehouse Shortcut Filtering\n\nShortcuts are items associated with Lakehouse items and can be selectively published using the following experimental feature:\n\n1. **`shortcut_exclude_regex`**\n - Optional parameter in `publish_all_items()`, set to a regex pattern that matches the shortcut name(s) found within Lakehouse item(s) in the repository.\n - Requires the `enable_shortcut_exclude` feature flag.\n - The matching shortcut(s) will be excluded from publishing.\n\n**Note:** This feature can be applied along with the other selective deployment features — please be cautious when using to avoid unexpected results.\n\n## Git-Based Change Detection\n\n`get_changed_items()` is a public utility function that uses `git diff` to detect which Fabric items have been added, modified, or renamed relative to a given git reference. It returns a list of strings in `\"item_name.item_type\"` format that can be passed directly to `items_to_include` in `publish_all_items()`.\n\nWhile `get_changed_items()` itself requires no feature flags, passing its output to `items_to_include` requires the experimental feature flags.\n\n**Important:** If `get_changed_items()` returns an empty list (no changes detected), do not call `publish_all_items()` without an explicit `items_to_include` list, as this would default to a full deployment. Always guard against the empty-list case:\n\n```python\nfrom fabric_cicd import FabricWorkspace, publish_all_items, get_changed_items, append_feature_flag\n\nappend_feature_flag(\"enable_experimental_features\")\nappend_feature_flag(\"enable_items_to_include\")\n\nworkspace = FabricWorkspace(\n workspace_id=\"your-workspace-id\",\n repository_directory=\"/path/to/repo\",\n item_type_in_scope=[\"Notebook\", \"DataPipeline\"],\n token_credential=token_credential, # or any other TokenCredential\n)\n\nchanged = get_changed_items(workspace.repository_directory)\n\nif changed:\n publish_all_items(workspace, items_to_include=changed)\nelse:\n print(\"No changed items detected — skipping deployment.\")\n```\n\nTo compare against a branch or a specific commit instead of the previous commit, pass a custom `git_compare_ref`:\n\n```python\nchanged = get_changed_items(workspace.repository_directory, git_compare_ref=\"main\")\n```\n\n**Note:** `get_changed_items()` returns only items that were **modified or added** (i.e., candidates for publishing). It does not return deleted items. Passing `items_to_include` to `publish_all_items()` requires enabling the `enable_experimental_features` and `enable_items_to_include` feature flags.\n\n## Debugging\n\nIf an error arises, or you want full transparency to all calls being made outside the library, enable debugging. Enabling debugging will write all API calls to the terminal. The logs can also be found in the `fabric_cicd.error.log` file.\n\n```python\nfrom fabric_cicd import change_log_level\nchange_log_level(\"DEBUG\")\n```\n\n**Note:** The `\"DEBUG\"` parameter can be omitted as it is the default value.\n\nFor comprehensive debugging information, including how to use the error log file and debug scripts, see the [Troubleshooting Guide](troubleshooting.md).\n" }, { "path": "docs/how_to/parameterization.md", "content": "# Parameterization\n\n## Overview\n\nTo handle environment-specific values committed to git, use a `parameter.yml` file. This file supports programmatically changing values based on the `environment` field passed into the `FabricWorkspace` object. If the environment value is not found in the `parameter.yml` file, any dependent replacements will be skipped. This file should sit in the root of the `repository_directory` folder specified in the FabricWorkspace object.\n\nExample of parameter.yml location based on provided repository directory:\n\n```python\nfrom azure.identity import AzureCliCredential\nfrom fabric_cicd import FabricWorkspace\n\ntoken_credential = AzureCliCredential()\nworkspace = FabricWorkspace(\n workspace_id=\"your-workspace-id\",\n repository_directory=\"C:/dev/workspace\",\n environment=\"PROD\",\n item_type_in_scope=[\"Notebook\"],\n token_credential=token_credential, # or any other TokenCredential\n)\n```\n\n```\nC:/dev/workspace\n /HelloWorld.Notebook\n ...\n /GoodbyeWorld.Notebook\n ...\n /parameter.yml\n```\n\nExample of parameter.yml file content:\n\n```yaml\nfind_replace:\n - find_value: \"your-dev-lakehouse-id\"\n replace_value:\n PPE: \"ppe-lakehouse-id\"\n PROD: \"prod-lakehouse-id\"\n\nkey_value_replace:\n - find_key: $.variables[?(@.name==\"Environment\")].value\n replace_value:\n PPE: \"PPE\"\n PROD: \"PROD\"\n\nspark_pool:\n - instance_pool_id: \"your-dev-pool-instance-id\"\n replace_value:\n PPE:\n type: \"Capacity\"\n name: \"PPE-Pool-name\"\n PROD:\n type: \"Capacity\"\n name: \"PROD-Pool-name\"\n\nsemantic_model_binding:\n default:\n connection_id:\n PPE: \"PPE-connection_id\"\n PROD: \"PROD-connection_id\"\n models:\n - semantic_model_name: \"semantic_model_name\"\n connection_id:\n PPE: \"PPE-connection_id\"\n PROD: \"PROD-connection_id\"\n```\n\nRaise a [feature request](https://github.com/microsoft/fabric-cicd/issues/new?template=2-feature.yml) for additional parameterization capabilities.\n\n## Parameter Inputs\n\n### `find_replace`\n\nFor generic find-and-replace operations. This will replace every instance of a specified string in every file. Specify the `find_value` and the `replace_value` for each environment (e.g., PPE, PROD). Optional fields, including `item_type`, `item_name`, and `file_path`, can be used as file filters for more fine-grained control over where the replacement occurs. The `is_regex` field can be added and set to `\"true\"` to enable regex pattern matching for the `find_value`.\n\nNote: A common use case for this function is to replace values in text based file types like notebooks.\n\n```yaml\nfind_replace:\n # Required fields: value must be a string\n - find_value: \n replace_value:\n : \n : \n # Optional fields\n # Set to \"true\" to treat find_value as a regex pattern\n is_regex: \"\"\n # Filter values must be a string or array of strings\n item_type: \n item_name: \n file_path: \n```\n\n### `key_value_replace`\n\nProvides the ability to perform key based replacement operations in JSON and YAML files. This will look for a specific key using a valid JSONPath expression and replace every found instance in every file. Specify the `find_key` and the `replace_value` for each environment (e.g., PPE, PROD). Optional fields, including `item_type`, `item_name`, and `file_path`, can be used as file filters for more fine-grained control over where the replacement occurs. Refer to https://jsonpath.com/ for a simple to use JSONPath evaluator.\n\nNote: A common use case for this parameter is to replace values in key/value file types like Data Pipeline, Schedule files, etc. The `key_value_replace` parameter automatically detects and processes any file containing valid JSON or YAML content, regardless of file extension (e.g., `.schedules` files). This does not apply to `.platform` files, which are intentionally excluded from parameterization to maintain item integrity.\n\nThe `replace_value` field supports the same dynamic replacement variables as `find_replace`, including `$items` and `$workspace` notation. See the **Dynamic Replacement** section under `find_replace` for details on supported variables and attributes.\n\n```yaml\nkey_value_replace:\n # Required fields: key must be JSONPath\n - find_key: \n replace_value:\n : \n : \n # Optional fields: value must be a string or array of strings\n item_type: \n item_name: \n file_path: \n```\n\nExample with `$items` notation:\n\n```yaml\nkey_value_replace:\n - find_key: $.properties.activities[?(@.name==\"Run Notebook\")].typeProperties.notebookId\n replace_value:\n PPE: \"$items.Notebook.Hello World.$id\" # PPE Hello World Notebook GUID\n PROD: \"$items.Notebook.Hello World.$id\" # PROD Hello World Notebook GUID\n item_type: \"DataPipeline\"\n - find_key: $.properties.activities[?(@.name==\"Run Notebook\")].typeProperties.workspaceId\n replace_value:\n PPE: \"$workspace.$id\" # PPE workspace ID\n PROD: \"$workspace.$id\" # PROD workspace ID\n item_type: \"DataPipeline\"\n```\n\n### `spark_pool`\n\nEnvironments attached to custom spark pools need to be parameterized because the `instance_pool_id` in the `Sparkcompute.yml` file is workspace-specific and must be mapped to the target workspace's pool. Provide the `instance_pool_id` value, and the pool `type` and `name` values as the `replace_value` for each environment (e.g., PPE, PROD). An optional field, `item_name`, can be used to filter the specific environment item where the replacement will occur.\n\n```yaml\nspark_pool:\n # Required fields: value must be a string\n - instance_pool_id: \n replace_value:\n :\n type: \n name: \n :\n type: \n name: \n # Optional field: value must be a string or array of strings\n item_name: \n```\n\n### `semantic_model_binding`\n\nSemantic model binding automatically connects semantic models to the appropriate data source connection (e.g., cloud or gateway/on-premises) after deployment, ensuring your models can refresh data in the target environment.\n\n```yaml\nsemantic_model_binding:\n # Default connection for all models not explicitly listed\n default:\n connection_id:\n PPE: \n PROD: \n # OR use _ALL_ for same connection across environments\n # _ALL_: \n\n # Explicit bindings override default\n models:\n - semantic_model_name: \"\"\n connection_id:\n PPE: \n PROD: \n\n - semantic_model_name: [\"\", \"\", ...]\n connection_id:\n _ALL_: \n```\n\n**Notes:**\n\n- The `_ALL_` environment key (case-insensitive) can be used in the `connection_id` dictionary to apply the same connection to any target environment.\n- Connection ID values must be valid GUIDs.\n- **Only a single connection binding per Semantic Model is currently supported.** If your Semantic Model uses multiple connections (e.g., connecting to both a SQL database and a Lakehouse), only one can be configured through `semantic_model_binding`. Additional connections must be configured manually after deployment.\n\n## Advanced Find and Replace\n\n### `find_value` Regex\n\nIn the `find_replace` parameter, the `find_value` can be set to a regex pattern instead of a literal string to find a value in the files to replace. When a match is found, the `find_value` is assigned to the matched string and can be used to replace all occurrences of that value in the file.\n\n- **How to** use this feature:\n - Set the `find_value` to a **valid regex pattern** wrapped in quotes.\n - Include the optional field `is_regex` and set it to the value `\"true\"`, see [more details](#regex-pattern-match).\n- **Important:**\n - The user is solely **responsible for providing a valid and correctly matching regex pattern**. If the pattern is invalid (i.e., it cannot be compiled) or fails to match any content in the target files, deployment will fail.\n - A valid regex pattern requires the following:\n - Ensure that all special characters in the regex pattern are properly **escaped**.\n - The exact value intended to be replaced must be enclosed in parentheses `( )`.\n - The parentheses creates a **capture group 1**, which must always be used as the replacement target. Capture group 1 should isolate values like a GUID, SQL connection string, etc.\n - Include the **surrounding context** in the pattern, such as property/field names, quotes, etc. to ensure it matches the correct value and not a value with a similar format elsewhere in the file.\n- **Example:**\n - Use a regex `find_value` to match a lakehouse ID inside a Notebook file. **Note:** avoid using a pattern that ONLY matches the GUID format as doing so would risk replacing any matching GUID in the file, not just the intended one. Include the surrounding context in your pattern—such as `# META \"default_lakehouse\": \"123456\"`—and capture only the `123456` GUID in group 1. This ensures that only the correct, context-specific GUID is replaced.\n\n```yaml\nfind_replace:\n # A valid regex pattern to match the default_lakehouse ID\n - find_value: \\#\\s*META\\s+\"default_lakehouse\":\\s*\"([0-9a-fA-F]{8}-[0-9a-fA-F]{4}-[0-9a-fA-F]{4}-[0-9a-fA-F]{4}-[0-9a-fA-F]{12})\"\n replace_value:\n PPE: \"e8a7f3c6-9b2d-4f5e-a1b0-7c98d4e6a5f3\" # PPE Lakehouse GUID\n PROD: \"12c45d67-89ab-4cde-f012-3456789abcde\" # PROD Lakehouse GUID\n # Optional field: Set to \"true\" to treat find_value as a regex pattern\n is_regex: \"true\" # \"\"\n item_type: \"Notebook\" # filter on notebook files\n item_name: [\"Hello World\", \"Goodbye World\"] # filter on specific notebook files\n```\n\n### Dynamic Replacement\n\nThe `replace_value` field in the `find_replace` and `key_value_replace` parameters supports fabric-cicd defined _variables_ that reference workspace or deployed item metadata:\n\n- **Dynamic workspace/item metadata replacement ONLY works for referenced items that exist in the `repository_directory`.**\n- Dynamic replacement works in tandem with `find_value` (for `find_replace`) as either a regex or a literal string, or with `find_key` (for `key_value_replace`) as a JSONPath expression.\n- The `replace_value` can contain a mix of input values within the _same_ parameter input, e.g. `PPE` is set to a static string and `PROD` is set to a variable.\n- **Supported variables:**\n - **Workspace variable:**\n\n | Workspace Variable | Description | Example |\n | --------------------------------------------------------------- | ----------------------------------------------------------------------------------------- | ---------------------------------------------------------- |\n | `$workspace.$id` or `$workspace.id` | Workspace ID of the target environment | `$workspace.$id` or `$workspace.id` |\n | `$workspace.$name` | Display name of the target workspace | `$workspace.$name` |\n | `$workspace.$name_encoded` | URL-encoded display name of the target workspace (spaces become `%20`, etc.) | `$workspace.$name_encoded` |\n | `$workspace..$id` or `$workspace.` | Workspace ID of the specified workspace name | `$workspace.TestWorkspace.$id` or `$workspace.TestWorkspace` |\n | `$workspace..$items...$` | Attribute value of the specified item in a specified workspace (see supported attributes) | `$workspace.TestWorkspace.$items.Lakehouse.Example_LH.$id` |\n\n > **Notes:**\n >\n > - When using `$workspace.$name`, `$workspace.$name_encoded`, `$workspace..$id` / `$workspace.`, or `$workspace..$items...$`, ensure the executing identity has proper permissions to access the relevant workspace. \n > - For `$workspace..$id` / `$workspace.` and `$workspace..$items...$`, the provided workspace display name must be an exact, case-sensitive match.\n > - `$workspace.$name` returns the raw workspace display name. If the target value must remain URL-encoded, use `$workspace.$name_encoded` instead, which automatically percent-encodes the name (e.g., `My Workspace` becomes `My%20Workspace`).\n\n - **Item attribute variable:** replaces the item's attribute value with the corresponding attribute value of the item in the deployed/target workspace.\n - `$items...` (legacy format)\n - **`$items...$`** (new format)\n - **Supported attributes:**\n\n | Attribute Variable | Supported Items | Example | Sample Replace Value |\n | ------------------------------------------------- | --------------------------------- | ------------------------------------------------- | -------------------------------------------------------------- |\n | `$items...$id` | All | `$items.Notebook.MyNotebook.$id` | `123e4567-e89b-12d3-a456-426614174000` |\n | `$items...$sqlendpoint` | Lakehouse, SQLDatabase, Warehouse | `$items.Lakehouse.MyLakehouse.$sqlendpoint` | `abc123def456.datawarehouse.fabric.microsoft.com` |\n | `$items...$sqlendpointid` | Lakehouse | `$items.Lakehouse.MyLakehouse.$sqlendpointid` | `37dc8a41-dea9-465d-b528-3e95043b2356` |\n | `$items...$queryserviceuri` | Eventhouse | `$items.Eventhouse.MyEventhouse.$queryserviceuri` | `https://trd-a1b2c3d4e5f6g7h8i9.z4.kusto.fabric.microsoft.com` |\n \n - Attributes should be **lowercase**.\n - Item type and name are **case-sensitive**.\n - Item type must be valid and in scope.\n - Item name must be an **exact match** (include spaces, if present).\n - **Example:** set `$items.Notebook.Hello World.$id` to get the item ID of the `\"Hello World\"` Notebook in the target workspace.\n - **Important**: Deployment will fail in the following cases:\n - Incorrect variable syntax used, e.g., `$item.Notebook.Hello World.$id` instead of `$items.Notebook.Hello World.$id`.\n - The specified **item type** or **name** is invalid or does NOT exist in the deployed workspace, e.g., `$items.Notebook.HelloWorld.$id` or `$items.Environment.Hello World.$id`.\n - An invalid attribute name is provided, e.g., `$items.Notebook.Hello World.$guid` instead of `$items.Notebook.Hello World.$id`.\n - The attribute value does NOT exist, e.g., `$items.Notebook.Hello World.$sqlendpoint` (Notebook items don't have a SQL Endpoint).\n\n - For example use-cases, see the **Notebook/Dataflow Advanced `find_replace` Parameterization Case.**\n\n### Environment Variable Replacement\n\nIn the `find_replace` parameter, if the `enable_environment_variable_replacement` feature flag is set, pipeline/environment variables will be used to replace the values in the `parameter.yml` file with the corresponding values from the variables dictionary. **Only Environment Variable beginning with '$ENV:' will be used as replacement values.** See example below:\n\n```yaml\nfind_replace:\n # Lakehouse GUID\n - find_value: \"db52be81-c2b2-4261-84fa-840c67f4bbd0\"\n replace_value:\n PPE: \"$ENV:ppe_lakehouse\"\n PROD: \"$ENV:prod_lakehouse\"\n```\n\n### File Filters\n\nFile filtering is supported in all parameters. This feature is optional and can be used to specify the files where replacement is intended to occur.\n\n- **Supported filters:** `item_type`, `item_name`, and `file_path`, see [more details](#supported-file-filters).\n - **Note:** only `item_name` filter is supported in `spark_pool` parameter.\n- **Expected behavior:**\n - If at least one filter value does not match, the replacement will be skipped for that file.\n - If none of the optional filter fields or values are provided, the value found in _any_ repository file is subject to replacement.\n- **Filter input:**\n - Input values are **case sensitive**.\n - Input values must be **string** or **array** (enables one or many values to filter on).\n - YAML supports array inputs using bracket ( **[ ]** ) or dash ( **-** ) notation.\n\nfind_replace/key_value_replace\n\n```yaml\n:\n # Required fields: value must be a string\n - : \n replace_value:\n : \n : \n # Optional fields\n # Filter values must be a string or array of strings\n item_type: \n item_name: \n file_path: \n```\n\nspark_pool\n\n```yaml\nspark_pool:\n # Required fields: value must be a string\n - instance_pool_id: \n replace_value:\n :\n type: \n name: \n :\n type: \n name: \n # Optional field: value must be a string or array of strings\n item_name: \n```\n\n### \\_ALL\\_ Environment Key in `replace_value`\n\nThe `_ALL_` environment key (case-insensitive) in `replace_value` is supported for all parameter types (`find_replace`, `key_value_replace`, `spark_pool`) and applies the replacement to any target environment. When `_ALL_` is used, it must be the only environment key in the `replace_value` dictionary. Using `ALL` without underscores will be treated as a regular environment key.\n\nUse case: when the same replacement value applies to all target environments (particularly valuable in dynamic replacement scenarios).\n\n```yaml\nfind_replace:\n # Lakehouse GUID\n - find_value: \"db52be81-c2b2-4261-84fa-840c67f4bbd0\"\n replace_value:\n # use _ALL_ or _all_ or _All_\n _ALL_: \"$items.Lakehouse.Example_LH.$id\"\n```\n\n## Optional Fields\n\nWhen optional fields are omitted or left empty, only basic parameterization functionality will be available. To enable advanced features, you must add the specific optional field(s) (if applicable) and set appropriately.\n\n**Important:**\n\n- String input values should be wrapped in quotes. Remember to escape special characters, such as **\\\\** in `file_path` inputs.\n- `is_regex` and filter fields can be used in the same parameter configuration.\n\n### Regex Pattern Match\n\n#### `is_regex`\n\n- Only applicable to the `find_replace` parameter.\n- Include `is_regex` field when setting the `find_value` to a **valid regex pattern.**\n- When the `is_regex` field is set to the **string** value `\"true\"` or `\"True\"` (case-insensitive), regex pattern matching is enabled.\n- When regex pattern matching is enabled, the `find_value` is interpreted as a regex pattern rather than a literal string.\n\n### Supported File Filters\n\n#### `item_type`\n\n- Item types must be valid and within scope of deployment.\n- See valid [types](https://learn.microsoft.com/en-us/rest/api/fabric/core/items/create-item?tabs=HTTP#itemtype).\n\n#### `item_name`\n\n- Item names must match the exact names of items in the `repository_directory`.\n\n#### `file_path`\n\n- `file_path` accepts three types of paths within the _repository directory_ boundary:\n - **Absolute paths:** Full path starting from the drive root.\n - **Relative paths:** Paths relative to the _repository directory_.\n - **Wildcard paths:** Paths containing glob patterns.\n- When using _wildcard paths_:\n - Common patterns include `*` (matches any characters in a filename), `**` (matches any directory depth).\n - All matched files must exist within the _repository directory_.\n - When using wildcard patterns, verify your syntax carefully to avoid unexpected matching behavior.\n - **Examples:** `**/notebook-content.py` matches all notebook files in the repository directory, `Sample Pipelines/*.json` matches json files in the Sample Pipelines folder in the repository directory.\n\n## Parameter File Validation\n\nValidation of the `parameter.yml` file is a built-in feature of fabric-cicd, managed by the `Parameter` class. Validation is utilized in the following scenarios:\n\n**Debuggability:** Users can debug and validate their parameter file to ensure it meets the acceptable structure and input value criteria before running a deployment. Simply run the `debug_parameterization.py` script located in the `devtools` directory.\n\n**Deployment:** At the start of a deployment, an automated validation checks the validity of the `parameter.yml` file, if it is present. This step ensures that valid parameters are loaded, allowing deployment to run smoothly with correctly applied parameterized configurations. If the parameter file is invalid, the deployment will NOT proceed.\n\n## Parameter File Templates\n\nThis option supports splitting a large parameter file into smaller parameter file \"templates\". Create template YAML files in any location (in the following example, the files are located in a `templates` directory within the repository directory). In the main `parameter.yml` file, add the `extend` key with a list of template parameter file paths **relative to the main parameter file location.**\n\nRepository directory\n\n```\nC:/dev/workspace\n /HelloWorld.Notebook\n ...\n /GoodbyeWorld.Notebook\n ...\n /parameter.yml\n ...\n /templates\n /nb_parameters.yml\n /pl_parameters.yml\n /df_parameters.yml\n```\n\nMain `parameter.yml` file\n\n```yaml\nextend:\n - \"./templates/nb_parameters.yml\"\n # - \"./templates/pl_parameters.yml\"\n # - \"./templates/df_parameters.yml\"\n\nfind_replace:\n # Lakehouse Connection Guid\n - find_value: \"db52be81-c2b2-4261-84fa-840c67f4bbd0\"\n replace_value:\n PPE: \"81bbb339-8d0b-46e8-bfa6-289a159c0733\"\n PROD: \"5d6a1b16-447f-464a-b959-45d0fed35ca0\"\n # Optional fields:\n item_type: \"Notebook\"\n item_name: [\"Hello World\", \"Hello World Subfolder\"]\n file_path:\n - \"/Hello World.Notebook/notebook-content.py\"\n - \"/subfolder/Hello World Subfolder.Notebook/notebook-content.py\"\n\nspark_pool:\n # CapacityPool_Large\n - instance_pool_id: \"72c68dbc-0775-4d59-909d-a47896f4573b\"\n replace_value:\n PPE:\n type: \"Capacity\"\n name: \"CapacityPool_Large_PPE\"\n PROD:\n type: \"Capacity\"\n name: \"CapacityPool_Large_PROD\"\n # Optional field:\n item_name: \"World\"\n```\n\n`nb_parameters.yml` file\n\n```yaml\nfind_replace:\n # Lakehouse Connection Guid regex\n - find_value: \\#\\s*META\\s+\"default_lakehouse\":\\s*\"([0-9a-fA-F]{8}-[0-9a-fA-F]{4}-[0-9a-fA-F]{4}-[0-9a-fA-F]{4}-[0-9a-fA-F]{12})\"\n replace_value:\n # Variable: $items.type.name.attribute (Note: item type and name values are CASE SENSITIVE; id attribute returns the deployed item's id/guid)\n PPE: \"$items.Lakehouse.WithoutSchema.id\"\n PROD: \"$items.Lakehouse.WithoutSchema.id\"\n # Optional fields:\n is_regex: \"true\"\n file_path: \"/Example Notebook.Notebook/notebook-content.py\"\n # Lakehouse workspace id regex\n - find_value: \\#\\s*META\\s+\"default_lakehouse_workspace_id\":\\s*\"([0-9a-fA-F]{8}-[0-9a-fA-F]{4}-[0-9a-fA-F]{4}-[0-9a-fA-F]{4}-[0-9a-fA-F]{12})\"\n replace_value:\n # Variable: $workspace.id -> target workspace id\n PPE: \"$workspace.id\"\n PROD: \"$workspace.id\"\n # Optional fields:\n is_regex: \"true\"\n file_path: \"/Example Notebook.Notebook/notebook-content.py\"\n```\n\nParameter dictionary\n\n```json\n{\n \"find_replace\": [\n {\n \"find_value\": \"db52be81-c2b2-4261-84fa-840c67f4bbd0\",\n \"replace_value\": {\n \"PPE\": \"81bbb339-8d0b-46e8-bfa6-289a159c0733\",\n \"PROD\": \"5d6a1b16-447f-464a-b959-45d0fed35ca0\"\n },\n \"item_type\": \"Notebook\",\n \"item_name\": [\"Hello World\", \"Hello World Subfolder\"],\n \"file_path\": [\n \"/Hello World.Notebook/notebook-content.py\",\n \"/subfolder/Hello World Subfolder.Notebook/notebook-content.py\"\n ]\n },\n {\n \"find_value\": \"\\\\#\\\\s*META\\\\s+\\\"default_lakehouse\\\":\\\\s*\\\"([0-9a-fA-F]{8}-[0-9a-fA-F]{4}-[0-9a-fA-F]{4}-[0-9a-fA-F]{4}-[0-9a-fA-F]{12})\\\"\",\n \"replace_value\": {\n \"PPE\": \"$items.Lakehouse.WithoutSchema.id\",\n \"PROD\": \"$items.Lakehouse.WithoutSchema.id\"\n },\n \"is_regex\": \"true\",\n \"file_path\": \"/Example Notebook.Notebook/notebook-content.py\"\n },\n {\n \"find_value\": \"\\\\#\\\\s*META\\\\s+\\\"default_lakehouse_workspace_id\\\":\\\\s*\\\"([0-9a-fA-F]{8}-[0-9a-fA-F]{4}-[0-9a-fA-F]{4}-[0-9a-fA-F]{4}-[0-9a-fA-F]{12})\\\"\",\n \"replace_value\": { \"PPE\": \"$workspace.id\", \"PROD\": \"$workspace.id\" },\n \"is_regex\": \"true\",\n \"file_path\": \"/Example Notebook.Notebook/notebook-content.py\"\n }\n ],\n \"spark_pool\": [\n {\n \"instance_pool_id\": \"72c68dbc-0775-4d59-909d-a47896f4573b\",\n \"replace_value\": {\n \"PPE\": { \"type\": \"Capacity\", \"name\": \"CapacityPool_Large_PPE\" },\n \"PROD\": { \"type\": \"Capacity\", \"name\": \"CapacityPool_Large_PROD\" }\n },\n \"item_name\": \"World\"\n }\n ]\n}\n```\n\n## Sample Parameter File\n\nAn exhaustive example of all capabilities currently supported in the `parameter.yml` file.\n\n```yaml\nfind_replace:\n - find_value: \"123e4567-e89b-12d3-a456-426614174000\" # lakehouse GUID to be replaced\n replace_value:\n PPE: \"f47ac10b-58cc-4372-a567-0e02b2c3d479\" # PPE lakehouse GUID\n PROD: \"9b2e5f4c-8d3a-4f1b-9c3e-2d5b6e4a7f8c\" # PROD lakehouse GUID\n item_type: \"Notebook\" # filter on notebook files\n item_name: [\"Hello World\", \"Goodbye World\"] # filter on specific notebook files\n\n # enable_environment_variable_replacement feature flag to replace workspace ID\n - find_value: \"8f5c0cec-a8ea-48cd-9da4-871dc2642f4c\" # workspace ID to be replaced\n replace_value:\n PPE: \"$ENV:ppe_workspace_id\" # PPE workspace ID (ENV variable)\n PROD: \"$ENV:prod_workspace_id\" # PROD workspace ID (ENV variable)\n file_path: # filter on notebook files with these paths\n - \"/Hello World.Notebook/notebook-content.py\"\n - \"\\\\Goodbye World.Notebook\\\\notebook-content.py\"\n\n # lakehouse GUID to be replaced (using regex pattern)\n - find_value: \\#\\s*META\\s+\"default_lakehouse\":\\s*\"([0-9a-fA-F]{8}-[0-9a-fA-F]{4}-[0-9a-fA-F]{4}-[0-9a-fA-F]{4}-[0-9a-fA-F]{12})\"\n replace_value:\n PPE: \"$items.Lakehouse.Example_LH.$id\" # PPE lakehouse GUID (dynamic)\n PROD: \"$items.Lakehouse.Example_LH.$id\" # PROD lakehouse GUID (dynamic)\n is_regex: \"true\" # enable regex pattern matching\n item_type: \"Notebook\" # filter on notebook files\n item_name: [\"Hello World\", \"Goodbye World\"] # filter on specific notebook files\n\n # lakehouse workspace ID to be replaced (using regex pattern)\n - find_value: \\#\\s*META\\s+\"default_lakehouse_workspace_id\":\\s*\"([0-9a-fA-F]{8}-[0-9a-fA-F]{4}-[0-9a-fA-F]{4}-[0-9a-fA-F]{4}-[0-9a-fA-F]{12})\"\n replace_value:\n _ALL_: \"$workspace.$id\" # workspace ID of the target environment (dynamic)\n is_regex: \"true\" # enable regex pattern matching\n item_name: # filter on specific notebook files\n - \"Hello World\"\n - \"Goodbye World\"\n file_path: \"**/notebook-content.py\" # filter on notebook files using wildcard paths\n\nkey_value_replace:\n # SQL Server Connection to be replaced\n - find_key: $.properties.activities[?(@.name==\"Load_Intake\")].typeProperties.source.datasetSettings.externalReferences.connection\n replace_value:\n PPE: \"6c517159-d27a-41d5-b71e-ca1ecff6542b\" # PPE SQL Server Connection\n PROD: \"6c517159-d27a-41d5-b71e-ca1ecff6542b\" # PROD SQL Server Connection\n item_type: \"DataPipeline\" # filter on data pipeline files\n\n # Schedule enabled state to be replaced\n - find_key: $.schedules[?(@.jobType==\"Execute\")].enabled\n replace_value:\n PPE: false # disable execution in PPE environment\n PROD: true # enable execution in PROD environment\n file_path: \"**/.schedules\" # filter on all .schedules files\n\nspark_pool:\n - instance_pool_id: \"72c68dbc-0775-4d59-909d-a47896f4573b\" # spark_pool_instance_id to be replaced\n replace_value:\n PPE:\n type: \"Capacity\" # target spark pool type, only supports Capacity or Workspace\n name: \"CapacityPool_Medium\" # target spark pool name\n PROD:\n type: \"Capacity\" # target spark pool type, only supports Capacity or Workspace\n name: \"CapacityPool_Large\" # target spark pool name\n item_name: \"World\" # filter on environment file for environment named \"World\"\n\n - instance_pool_id: \"e7b8f1c4-4a6e-4b8b-9b2e-8f1e5d6a9c3d\" # spark_pool_instance_id to be replaced\n replace_value:\n PPE:\n type: \"Workspace\" # target spark pool type, only supports Capacity or Workspace\n name: \"WorkspacePool_Medium\" # target spark pool name\n item_name: [\"World_1\", \"World_2\", \"World_3\"] # filter on environment files for environments with these names\n```\n\n## Examples by Item Type\n\n### Notebooks\n\n#### `find_replace` Parameterization Case\n\n**Case:** A Notebook is attached to a Lakehouse which resides in different workspaces. The Workspace and Lakehouse GUIDs in the Notebook need to be updated to ensure the Notebook points to the correct Lakehouse once deployed.\n\n**Solution:** In the `notebook-content.py` file, the default_lakehouse `47592d55-9a83-41a8-9b21-e1ef44264161`, and default_lakehouse_workspace_id `2190baad-a374-4114-addd-0dcf0533e69d` must be replaced with the corresponding GUIDs of the Lakehouse in the target environment (PPE/PROD/etc). This replacement is managed by the `find_replace` input in the `parameter.yml` file where fabric-cicd finds every instance of the string within the specified repository files and replaces it with the string for the deployed environment.\n\nparameter.yml file\n\n```yaml\nfind_replace:\n - find_value: \"47592d55-9a83-41a8-9b21-e1ef44264161\" # lakehouse GUID to be replaced\n replace_value:\n PPE: \"a21e502a-51a5-4455-bb3d-6faf1e3e21fb\" # PPE lakehouse GUID\n PROD: \"1069f2ff-bb30-42a0-97b3-1f4655705b8a\" # PROD lakehouse GUID\n item_type: \"Notebook\" # filter on notebook files\n item_name: [\"Hello World\", \"Goodbye World\"] # filter on specific notebook files\n - find_value: \"2190baad-a374-4114-addd-0dcf0533e69d\" # workspace ID to be replaced\n replace_value:\n PPE: \"5a6ebbe6-9289-4105-b47c-cf158247b911\" # PPE workspace ID\n PROD: \"f9e8cbe0-2669-4e06-a026-7c75e5af8107\" # PROD workspace ID\n file_path: # filter on notebook files with these paths\n - \"/Hello World.Notebook/notebook-content.py\"\n - \"\\\\Goodbye World.Notebook\\\\notebook-content.py\"\n```\n\nnotebook-content.py file\n\n```python\n# Fabric notebook source\n\n# METADATA ********************\n\n# META {\n# META \"kernel_info\": {\n# META \"name\": \"synapse_pyspark\"\n# META },\n# META \"dependencies\": {\n# META \"lakehouse\": {\n# META \"default_lakehouse\": \"47592d55-9a83-41a8-9b21-e1ef44264161\",\n# META \"default_lakehouse_name\": \"Example_LH\",\n# META \"default_lakehouse_workspace_id\": \"2190baad-a374-4114-addd-0dcf0533e69d\"\n# META },\n# META \"environment\": {\n# META \"environmentId\": \"a277ea4a-e87f-8537-4ce0-39db11d4aade\",\n# META \"workspaceId\": \"00000000-0000-0000-0000-000000000000\"\n# META }\n# META }\n# META }\n\n# CELL ********************\n\ndf = spark.sql(\"SELECT * FROM Example_LH.Table1 LIMIT 1000\")\ndisplay(df)\n\n# METADATA ********************\n\n# META {\n# META \"language\": \"python\",\n# META \"language_group\": \"synapse_pyspark\"\n# META }\n```\n\n#### Advanced `find_replace` Parameterization Case\n\n**Case:** A Notebook is attached to a Lakehouse which resides in the same workspace. When deploying both the Lakehouse and the Notebook to a target environment (PPE/PROD/etc), the Workspace and Lakehouse GUIDs referenced in the Notebook must be updated to ensure it correctly points to the corresponding Lakehouse in the new environment.\n\n**Solution:** This approach uses `find_value` [**regex**](#find_value-regex)\\*\\* and [**dynamic variables**](#dynamic-replacement) to manage replacement. In the `find_replace` input in the `parameter.yml` file, the `is_regex` field is set to `\"true\"`, enabling fabric-cicd to find a string value within the _specified_ repository files that matches the provided regex pattern.\n\nThis approach is particularly useful for replacing values that are not known until deployment time, such as item IDs.\n\n\\*\\*The regex pattern must include a capture group, defined using `()`, and the `find_value` must always match **group 1**. The value captured in this group will be dynamically replaced with the appropriate value for the deployed environment.\n\nparameter.yml file\n\n```yaml\nfind_replace:\n # lakehouse GUID matching group 1 of regex pattern to be replaced\n - find_value: \\#\\s*META\\s+\"default_lakehouse\":\\s*\"([0-9a-fA-F]{8}-[0-9a-fA-F]{4}-[0-9a-fA-F]{4}-[0-9a-fA-F]{4}-[0-9a-fA-F]{12})\"\n replace_value:\n PPE: \"$items.Lakehouse.Example_LH.$id\" # PPE lakehouse GUID (dynamic)\n PROD: \"$items.Lakehouse.Example_LH.$id\" # PROD lakehouse GUID (dynamic)\n is_regex: \"true\"\n item_type: \"Notebook\" # filter on notebook files\n item_name: [\"Hello World\", \"Goodbye World\"] # filter on specific notebook files\n # workspace ID matching group 1 of regex pattern to be replaced\n - find_value: \\#\\s*META\\s+\"default_lakehouse_workspace_id\":\\s*\"([0-9a-fA-F]{8}-[0-9a-fA-F]{4}-[0-9a-fA-F]{4}-[0-9a-fA-F]{4}-[0-9a-fA-F]{12})\"\n replace_value:\n PPE: \"$workspace.$id\" # PPE workspace ID (dynamic)\n PROD: \"$workspace.$id\" # PROD workspace ID (dynamic)\n is_regex: \"true\"\n file_path: # filter on notebook files with these paths\n - \"/Hello World.Notebook/notebook-content.py\"\n - \"\\\\Goodbye World.Notebook\\\\notebook-content.py\"\n```\n\nnotebook-content.py file\n\n```python\n# Fabric notebook source\n\n# METADATA ********************\n\n# META {\n# META \"kernel_info\": {\n# META \"name\": \"synapse_pyspark\"\n# META },\n# META \"dependencies\": {\n# META \"lakehouse\": {\n# META \"default_lakehouse\": \"123e4567-e89b-12d3-a456-426614174000\",\n# META \"default_lakehouse_name\": \"Example_LH\",\n# META \"default_lakehouse_workspace_id\": \"8f5c0cec-a8ea-48cd-9da4-871dc2642f4c\"\n# META },\n# META \"environment\": {\n# META \"environmentId\": \"a277ea4a-e87f-8537-4ce0-39db11d4aade\",\n# META \"workspaceId\": \"00000000-0000-0000-0000-000000000000\"\n# META }\n# META }\n# META }\n\n# CELL ********************\n\ndf = spark.sql(\"SELECT * FROM Example_LH.Table1 LIMIT 1000\")\ndisplay(df)\n\n# METADATA ********************\n\n# META {\n# META \"language\": \"python\",\n# META \"language_group\": \"synapse_pyspark\"\n# META }\n```\n\n#### TSQL Notebook with SQL Endpoint Parameterization Case\n\n**Case:** A TSQL Notebook is attached to a Lakehouse SQL Endpoint. The SQL Endpoint ID in the Notebook needs to be updated to point to the corresponding SQL Endpoint in the target environment.\n\n**Solution:** Use `$items.Lakehouse..$sqlendpointid` to dynamically retrieve the SQL Endpoint ID.\n\nparameter.yml file\n\n```yaml\nfind_replace:\n - find_value: \"37dc8a41-dea9-465d-b528-3e95043b2356\"\n replace_value:\n PPE: \"$items.Lakehouse.Example_LH.$sqlendpointid\"\n PROD: \"$items.Lakehouse.Example_LH.$sqlendpointid\"\n item_type: \"Notebook\"\n```\n\nnotebook-content.py file\n\n```python\n# META {\n# META \"dependencies\": {\n# META \"lakehouse\": {\n# META \"default_lakehouse\": \"123e4567-e89b-12d3-a456-426614174000\",\n# META \"default_lakehouse_name\": \"Example_LH\",\n# META \"default_lakehouse_sql_endpoint\": \"37dc8a41-dea9-465d-b528-3e95043b2356\"\n# META }\n# META }\n# META }\n```\n\n### Data Pipelines\n\n#### `key_value_replace` Parameterization Case\n\n**Case:** A Data Pipeline is attached to data sources via the Connection Id. Connections are not deployed with fabric-cicd and therefore need to be parameterized. In the `pipeline-content.json` file, the SQL Server Connection Id `c517e095-ed87-4665-95fa-8cdb1e751fba`, must be replaced with the corresponding GUIDs of the SQL Server in the target environment (PPE/PROD/etc).\n\n**Solution:** This replacement is managed by the `find_key` input in the `parameter.yml` file where fabric-cicd finds every instance of the key within the _specified_ repository files and replaces it with the string for the deployed environment.\n\nparameter.yml file\n\n```yaml\nkey_value_replace:\n - find_key: $.properties.activities[?(@.name==\"Copy Data\")].typeProperties.source.datasetSettings.externalReferences.connection\n replace_value:\n PPE: \"f47ac10b-58cc-4372-a567-0e02b2c3d479\" # PPE SQL Connection GUID\n PROD: \"9b2e5f4c-8d3a-4f1b-9c3e-2d5b6e4a7f8c\" # PROD SQL Connection GUID\n item_type: \"DataPipeline\" # filter on Data Pipelines files\n item_name: \"Example Pipeline\" # filter on specific Data Pipelines files\n```\n\npipeline-content.json file\n\n```json\n{\n \"properties\": {\n \"activities\": [\n {\n \"name\": \"Copy Data\",\n \"type\": \"Copy\",\n \"dependsOn\": [],\n \"policy\": {\n \"timeout\": \"0.12:00:00\",\n \"retry\": 0,\n \"retryIntervalInSeconds\": 30,\n \"secureOutput\": false,\n \"secureInput\": false\n },\n \"typeProperties\": {\n \"source\": {\n \"type\": \"AzureSqlSource\",\n \"queryTimeout\": \"02:00:00\",\n \"partitionOption\": \"None\",\n \"datasetSettings\": {\n \"annotations\": [],\n \"type\": \"AzureSqlTable\",\n \"schema\": [],\n \"typeProperties\": {\n \"schema\": \"Dataprod\",\n \"table\": \"DIM_Calendar\",\n \"database\": \"unified\"\n },\n \"externalReferences\": {\n \"connection\": \"c517e095-ed87-4665-95fa-8cdb1e751fba\"\n }\n }\n },\n \"sink\": {\n \"type\": \"LakehouseTableSink\",\n \"tableActionOption\": \"Append\",\n \"datasetSettings\": {\n \"annotations\": [],\n \"linkedService\": {\n \"name\": \"Unified\",\n \"properties\": {\n \"annotations\": [],\n \"type\": \"Lakehouse\",\n \"typeProperties\": {\n \"workspaceId\": \"2d2e0ae2-9505-4f0c-ab42-e76cc11fb07d\",\n \"artifactId\": \"31dd665e-95f3-4575-9f46-70ea5903d89b\",\n \"rootFolder\": \"Tables\"\n }\n }\n },\n \"type\": \"LakehouseTable\",\n \"schema\": [],\n \"typeProperties\": {\n \"schema\": \"Dataprod\",\n \"table\": \"DIM_Calendar\"\n }\n }\n },\n \"enableStaging\": false,\n \"translator\": {\n \"type\": \"TabularTranslator\",\n \"typeConversion\": true,\n \"typeConversionSettings\": {\n \"allowDataTruncation\": true,\n \"treatBooleanAsNumber\": false\n }\n }\n }\n }\n ]\n }\n}\n```\n\n### Schedules\n\n#### `key_value_replace` Parameterization Case\n\n**Case:** Items with schedules need to have their execution settings parameterized across environments. The `enabled` field in `.schedules` files typically needs different values for different environments (e.g., disabled in test environments, enabled in production).\n\n**Solution:** In the `.schedules` file, the `enabled` field for Execute jobs must be replaced with environment-specific boolean values. This replacement is managed by the `key_value_replace` input in the `parameter.yml` file where fabric-cicd finds the JSONPath expression within the specified repository files and replaces it with the appropriate value for the deployed environment.\n\nparameter.yml file\n\n```yaml\nkey_value_replace:\n - find_key: $.schedules[?(@.jobType==\"Execute\")].enabled\n replace_value:\n PPE: false\n PROD: true\n file_path: \"**/.schedules\"\n```\n\n.schedules file\n\n```json\n{\n \"$schema\": \"https://developer.microsoft.com/json-schemas/fabric/gitIntegration/schedules/1.0.0/schema.json\",\n \"schedules\": [\n {\n \"enabled\": true,\n \"jobType\": \"Execute\",\n \"configuration\": {\n \"type\": \"Cron\",\n \"startDateTime\": \"2025-07-01T12:00:00\",\n \"endDateTime\": \"2029-07-01T12:00:00\",\n \"localTimeZoneId\": \"Pacific Standard Time\",\n \"interval\": 15\n }\n }\n ]\n}\n```\n\n**Note:** The `.schedules` file contains JSON content but does not have a `.json` file extension. The fabric-cicd library automatically detects and processes files with valid JSON content regardless of their file extension, making this parameterization work seamlessly.\n\n### Environments\n\n#### `spark_pool` Parameterization Case\n\n**Case:** An Environment is attached to a Capacity level Custom Pool. The `instance_pool_id` in `Sparkcompute.yml` is workspace-specific and will differ between environments. The Spark Pool needs to be parameterized so fabric-cicd can resolve the correct pool in the target workspace. **Note:** Defining different names per environment is supported in the `parameter.yml` file. In the `Sparkcompute.yml` file, the referenced instance_pool_id `72c68dbc-0775-4d59-909d-a47896f4573b` points to a capacity custom pool named `CapacityPool_Large` of pool type `Capacity` for the `PROD` environment.\n\n**Solution:** This replacement is managed by the `spark_pool` input in the `parameter.yml` file where fabric-cicd finds every instance of the `instance_pool_id` and replaces it with the pool type and pool name for the _specified_ environment file.\n\nparameter.yml file\n\n```yaml\nspark_pool:\n - instance_pool_id: \"72c68dbc-0775-4d59-909d-a47896f4573b\" # spark_pool_instance_id to be replaced\n replace_value:\n PPE:\n type: \"Capacity\" # target spark pool type, only supports Capacity or Workspace\n name: \"CapacityPool_Medium\" # target spark pool name\n PROD:\n type: \"Capacity\" # target spark pool type, only supports Capacity or Workspace\n name: \"CapacityPool_Large\" # target spark pool name\n item_name: \"World\" # filter on environment file for environment named \"World\"\n```\n\nSparkcompute.yml\n\n```yaml\nenable_native_execution_engine: false\ninstance_pool_id: 72c68dbc-0775-4d59-909d-a47896f4573b\ndriver_cores: 16\ndriver_memory: 112g\nexecutor_cores: 16\nexecutor_memory: 112g\ndynamic_executor_allocation:\n enabled: false\n min_executors: 31\n max_executors: 31\nruntime_version: 1.3\n```\n\n### Dataflows\n\nDataflows can have different kinds of Fabric sources and destinations that need to be parameterized, depending on the scenario.\n\n#### Parameterization Overview\n\nTake a Lakehouse source/destination as an example, the Lakehouse is connected to a Dataflow in the following ways:\n\n1. Connection Id in the `queryMetadata.json` file:\n - Connections are not deployed with fabric-cicd and therefore need to be parameterized.\n2. Workspace and item IDs in the `mashup.pq` file:\n - Source and/or destination item references, such as a Dataflow (source only\\*\\*), Lakehouse, Warehouse, etc. appear in the `mashup.pq` file and need to be parameterized to ensure proper deployment across environments.\n\n**\\*\\*Note:** A Dataflow that sources from another Dataflow introduces a dependency that may require a specific order of deploying (source first then dependent). A Dataflow is referenced by the item ID in the workspace and the actual workspace ID, this makes re-pointing more complex (see parameterization guidance below).\n\n#### Parameterization Guidance\n\nConnections must be parameterized in addition to item references.\n\nScenarios When Deploying a Dataflow that contains a source Dataflow reference:\n\n1. Source Dataflow exists in the **same workspace** as the dependent Dataflow:\n - The source Dataflow must be deployed BEFORE the dependent Dataflow (especially during first time deployment).\n - To handle this dependency correctly and prevent deployment errors, set up the `find_replace` parameter with the following requirements (incorrect setup may introduce failure during Dataflow deployment):\n - Set `find_value` to match the `dataflowId` GUID referenced in the `mashup.pq` file (literal string or [regex](#find_value-regex)).\n - Set `replace_value` to the variable `$items.Dataflow..$id`. **Important:** Make sure the **item type** is `\"Dataflow\"` and the **item name** matches the source Dataflow name in the repository directory exactly (case sensitive, include any spaces).\n - File filters are optional but recommended when using a regex pattern for `find_value`.\n - **You don't need to parameterize the source Dataflow workspace ID here** as the library automatically handles this replacement when you use the items variable in _this_ Dataflow scenario.\n - **How this works:** This parameterization approach ensures correct deployment of interdependent Dataflows while automatically updating references to point to the newly deployed Dataflow in the target workspace.\n - Example parameter input:\n\n ```yaml\n find_replace:\n # The ID of the source Dataflow referenced in mashup.pq\n - find_value: \"0187104d-7a35-4abe-a2ca-a241ec81c8f1\"\n # Type = Dataflow and Name = , Attribute = id\n replace_value:\n PPE: \"$items.Dataflow.Source Dataflow.$id\"\n PROD: \"$items.Dataflow.Source Dataflow.$id\"\n # Optional fields:\n file_path:\n - \"\\\\Referencing Dataflow.Dataflow\\\\mashup.pq\"\n ```\n\n2. Source Dataflow exists in a **different workspace** from the dependent Dataflow:\n - When the source Dataflow exists in a different workspace, deployment order doesn't matter.\n - To re-point the source Dataflow from one workspace to another workspace, you can parameterize using the `find_replace` parameter. The Dataflow ID AND Workspace ID of the source Dataflow both need to be parameterized.\n - **Note:** dynamic replacement for item ID and workspace ID will NOT work here since the source Dataflow does not exist in the _repository directory_.\n\nScenarios When Deploying a Dataflow that contains other Fabric items (e.g., Lakehouse, Warehouse, etc.) references:\n\n1. Source and/or destination item exists in the **same workspace** as the dependent Dataflow:\n - Use the `find_replace` parameter to update references so they point to the corresponding items in the target workspace.\n - You need to parameterize both the item ID and workspace ID found in the `mashup.pq` file.\n - Best practices for Dataflow parameterization:\n - Use a [regex](#find_value-regex) for the `find_value` to avoid hardcoding GUIDs and simplify maintenance\n - Use [dynamic replacement](#dynamic-replacement) to eliminate multi-phase deployments\n - Adding file filters to target specific Dataflow files provides more precise control.\n\n2. Source/destination item exists in a **different workspace** from the dependent Dataflow:\n - Use the `find_replace` parameter to update references so they point to items in the different workspace.\n - Parameterize both the item ID and workspace ID found in the `mashup.pq` file.\n - Use a regex pattern for the `find_value` to avoid hardcoding GUIDs and simplify maintenance.\n - **Note:** dynamic replacement won't work in this scenario - it only works for items in the same workspace as the Dataflow.\n - Adding file filters helps target specific Dataflow files for more precise control.\n\n#### Advanced `find_replace` Parameterization Case\n\n**Case:** A Dataflow points to a destination Lakehouse. The Lakehouse exists in the same workspace as the Dataflow. In the `mashup.pq` file, the following GUIDs need to be replaced:\n\n- The workspaceId `e6a8c59f-4b27-48d1-ae03-7f92b1c6458d` with the target workspace Id.\n- The lakehouseId `3d72f90e-61b5-42a8-9c7e-b085d4e31fa2` with the corresponding Id of the Lakehouse in the target environment (PPE/PROD/etc).\n\n**Solution:** These replacements are managed using a regex pattern as input for the `find_value` in the `parameter.yml` file, which finds the matching value in the _specified_ repository files and replaces it with the dynamically retrieved workspace or item Id of the target environment.\n\n**Note:** While Connection IDs are shown in this example, they are not the main focus. Connection parameterization may vary depending on your specific scenario.\n\nparameter.yml file\n\n```yaml\nfind_replace:\n # Lakehouse workspace ID regex - matches the workspaceId GUID\n - find_value: Navigation_1\\s*=\\s*Pattern\\{\\[workspaceId\\s*=\\s*\"([0-9a-fA-F]{8}-[0-9a-fA-F]{4}-[0-9a-fA-F]{4}-[0-9a-fA-F]{4}-[0-9a-fA-F]{12})\"\\]\\}\n replace_value:\n PPE: \"$workspace.$id\" # PPE workspace ID (dynamic)\n PROD: \"$workspace.$id\"\n is_regex: \"true\" # Activate find_value regex matching\n file_path: \"/Sample Dataflow.Dataflow/mashup.pq\"\n\n # Lakehouse ID regex - matches the lakehouseId GUID\n - find_value: Navigation_2\\s*=\\s*Navigation_1\\{\\[lakehouseId\\s*=\\s*\"([0-9a-fA-F]{8}-[0-9a-fA-F]{4}-[0-9a-fA-F]{4}-[0-9a-fA-F]{4}-[0-9a-fA-F]{12})\"\\]\\}\n replace_value:\n PPE: \"$items.Lakehouse.Sample_LH.$id\" # Sample_LH Lakehouse ID in PPE (dynamic)\n PROD: \"$items.Lakehouse.Sample_LH.$id\"\n is_regex: \"true\" # Activate find_value regex matching\n file_path: \"/Sample Dataflow.Dataflow/mashup.pq\"\n\n # Connection ID - Cluster ID\n - find_value: \"8e4f92a7-3c18-49d5-b6d0-7f2e591ca4e8\"\n replace_value:\n PPE: \"76a8f5c3-e4b2-48d1-9c7f-382d69a5e7b0\" # PPE Cluster ID\n PROD: \"f297e14d-6c83-42a5-b718-59d40e3f8c2d\" # PROD Cluster ID\n file_path: \"/Sample Dataflow.Dataflow/queryMetadata.json\"\n\n # Connection ID - Datasource ID\n - find_value: \"d12c5f7b-90a3-47e6-8d2c-3fb59e01d47a\"\n replace_value:\n PPE: \"25b9a417-3d8e-4f62-901c-75de6ba84f35\" # PPE Datasource ID\n PROD: \"cb718d96-5ae2-47fc-8b93-1d24c0f5e8a7\" # PROD Datasource ID\n file_path: \"/Sample Dataflow.Dataflow/queryMetadata.json\"\n```\n\nqueryMetadata.json file\n\n```json\n{\n \"formatVersion\": \"202502\",\n \"computeEngineSettings\": {},\n \"name\": \"Sample Dataflow\",\n \"queryGroups\": [],\n \"documentLocale\": \"en-US\",\n \"queriesMetadata\": {\n \"Table\": {\n \"queryId\": \"ba67667b-14c0-4536-a92d-feafc73baa4b\",\n \"queryName\": \"Table\",\n \"loadEnabled\": false\n },\n \"Table_DataDestination\": {\n \"queryId\": \"a157a378-b510-4d95-bb82-5a7c80df8b4c\",\n \"queryName\": \"Table_DataDestination\",\n \"isHidden\": true,\n \"loadEnabled\": false\n }\n },\n \"connections\": [\n {\n \"path\": \"Lakehouse\",\n \"kind\": \"Lakehouse\",\n \"connectionId\": \"{\\\"ClusterId\\\":\\\"8e4f92a7-3c18-49d5-b6d0-7f2e591ca4e8\\\",\\\"DatasourceId\\\":\\\"d12c5f7b-90a3-47e6-8d2c-3fb59e01d47a\\\"}\"\n }\n ]\n}\n```\n\nmashup.pq file\n\n```pq\n[StagingDefinition = [Kind = \"FastCopy\"]]\nsection Section1;\n[DataDestinations = {[Definition = [Kind = \"Reference\", QueryName = \"Table_DataDestination\", IsNewTarget = true], Settings = [Kind = \"Automatic\", TypeSettings = [Kind = \"Table\"]]]}]\nshared Table = let\n Source = Table.FromRows(Json.Document(Binary.Decompress(Binary.FromText(\"i45WckksSUzLyS9X0lEyBGKP1JycfKVYHYhEQGZBak5mXipQwgiIw/OLclLAkn75JalJ+fnZQEFjmC4FhHRwam5iXklmsm9+SmoOUN4EiMFsBVTzoRabArGLG0x/LAA=\", BinaryEncoding.Base64), Compression.Deflate)), let _t = ((type nullable text) meta [Serialized.Text = true]) in type table [Item = _t, Id = _t, Name = _t]),\n #\"Changed column type\" = Table.TransformColumnTypes(Source, {{\"Item\", type text}, {\"Id\", Int64.Type}, {\"Name\", type text}}),\n #\"Added custom\" = Table.TransformColumnTypes(Table.AddColumn(#\"Changed column type\", \"IsDataflow\", each if [Item] = \"Dataflow\" then true else false), {{\"IsDataflow\", type logical}}),\n #\"Added custom 1\" = Table.TransformColumnTypes(Table.AddColumn(#\"Added custom\", \"ContainsHello\", each if Text.Contains([Name], \"Hello\") then 1 else 0), {{\"ContainsHello\", Int64.Type}})\nin\n #\"Added custom 1\";\nshared Table_DataDestination = let\n Pattern = Lakehouse.Contents([CreateNavigationProperties = false, EnableFolding = false]),\n Navigation_1 = Pattern{[workspaceId = \"e6a8c59f-4b27-48d1-ae03-7f92b1c6458d\"]}[Data],\n Navigation_2 = Navigation_1{[lakehouseId = \"3d72f90e-61b5-42a8-9c7e-b085d4e31fa2\"]}[Data],\n TableNavigation = Navigation_2{[Id = \"Items\", ItemKind = \"Table\"]}?[Data]?\nin\n TableNavigation;\n```\n\n### Reports\n\nReports can reference Semantic Models in two ways: `byPath` (relative path to a model in the same repository) or `byConnection` (connection string to a model in Power BI service).\n\n#### Parameterization Overview\n\n**`byPath` - No Parameterization Needed**\n\nWhen a Report uses `byPath` to reference a Semantic Model in the same repository, the library automatically converts it to `byConnection` format during deployment. The Semantic Model must exist in the repository for this to work.\n\n**`byConnection` - Requires Parameterization**\n\nWhen Reports and Semantic Models are deployed separately (e.g., models first, then reports), or when Reports need to connect to models in Power BI Online service, use `byConnection` with parameterization to rebind reports to different semantic models across environments. This approach supports both same-workspace and cross-workspace binding scenarios.\n\n**Case:** A Report uses `byConnection` to reference a Semantic Model deployed to Power BI Online. The connection string contains environment-specific values (workspace ID, semantic model name, and semantic model ID) that must be updated for each target environment (PPE, PROD, etc.). The semantic model can be in the same workspace as the report or in a different workspace.\n\n**Note:** This case does not apply to the `semantic_model_binding` parameter.\n\n**Solution:** Use `find_replace` or `key_value_replace` in the `parameter.yml` file to parameterize the connection string components.\n\n#### `find_replace` Parameterization Case\n\nThis approach replaces individual parts of the connection string (workspace ID, model name, model ID) with environment-specific values. This enables granular control over each component in the connection string and allows the option to apply dynamic variables where needed.\n\n**Note:** The examples below use placeholder values (e.g., `MyReport`, `YourSemanticModelName`). Replace these with your actual report and semantic model names. For a working example, see `sample/workspace/parameter.yml` which references the `ByConnection.Report` and `ABC.SemanticModel` items.\n\nparameter.yml file\n\n```yaml\nfind_replace:\n # Replace workspace ID in connection string\n - find_value: \"dev-workspace-id\"\n replace_value:\n PPE: \"$workspace.$id\" # PPE workspace ID\n PROD: \"$workspace.$id\" # PROD workspace ID\n item_type: \"Report\"\n file_path: \"/MyReport.Report/definition.pbir\"\n\n # Replace semantic model name in connection string\n - find_value: \"dev-semantic-model\"\n replace_value:\n PPE: \"ppe-semantic-model-name\"\n PROD: \"prod-semantic-model-name\"\n item_type: \"Report\"\n file_path: \"/MyReport.Report/definition.pbir\"\n\n # Replace semantic model ID in connection string with dynamic replacement\n - find_value: \"00000000-0000-0000-0000-000000000000\"\n replace_value:\n PPE: \"$items.SemanticModel.YourSemanticModelName.$id\"\n PROD: \"$items.SemanticModel.YourSemanticModelName.$id\"\n item_type: \"Report\"\n file_path: \"/MyReport.Report/definition.pbir\"\n```\n\ndefinition.pbir\n\n```json\n{\n \"$schema\": \"https://developer.microsoft.com/json-schemas/fabric/item/report/definitionProperties/2.0.0/schema.json\",\n \"version\": \"4.0\",\n \"datasetReference\": {\n \"byConnection\": {\n \"connectionString\": \"Data Source=powerbi://api.powerbi.com/v1.0/myorg/dev-workspace-id;initial catalog=dev-semantic-model;access mode=readonly;integrated security=ClaimsToken;semanticmodelid=00000000-0000-0000-0000-000000000000\"\n }\n }\n}\n```\n\n#### `key_value_replace` Parameterization Case\n\nThis approach replaces the entire connection string with environment-specific values. This simplifies the parameter configuration, however, dynamic variables are not supported in this example as they cannot be embedded within a larger string value.\n\nparameter.yml file\n\n```yaml\nkey_value_replace:\n - find_key: $.datasetReference.byConnection.connectionString\n replace_value:\n PPE: \"Data Source=powerbi://api.powerbi.com/v1.0/myorg/ppe-workspace-guid;initial catalog=ppe-semantic-model;access mode=readonly;integrated security=ClaimsToken;semanticmodelid=ppe-model-guid\"\n PROD: \"Data Source=powerbi://api.powerbi.com/v1.0/myorg/prod-workspace-guid;initial catalog=prod-semantic-model;access mode=readonly;integrated security=ClaimsToken;semanticmodelid=prod-model-guid\"\n item_type: \"Report\"\n item_name: \"MyReport\"\n```\n\ndefinition.pbir\n\n```json\n{\n \"$schema\": \"https://developer.microsoft.com/json-schemas/fabric/item/report/definitionProperties/2.0.0/schema.json\",\n \"version\": \"4.0\",\n \"datasetReference\": {\n \"byConnection\": {\n \"connectionString\": \"Data Source=powerbi://api.powerbi.com/v1.0/myorg/dev-workspace-id;initial catalog=dev-semantic-model;access mode=readonly;integrated security=ClaimsToken;semanticmodelid=00000000-0000-0000-0000-000000000000\"\n }\n }\n}\n```\n" }, { "path": "docs/how_to/troubleshooting.md", "content": "# Troubleshooting\n\nThis guide provides comprehensive debugging and troubleshooting resources for both users deploying with fabric-cicd and contributors developing within the repository.\n\n## Debugging Deployments\n\n### Enable Debug Logging\n\nfabric-cicd includes a debug logging feature that provides detailed visibility into all operations, including API calls made during deployment.\n\n**Default Behavior:**\n\n- Without debug logging enabled, fabric-cicd displays only high-level progress messages, warnings, and errors\n- The `fabric_cicd.error.log` file will contain the same lines printed to the console along with stack traces for any errors\n\n**Enabling Debug Logging:**\n\nTo enable debug logging, add the following to your deployment script:\n\n```python\nfrom fabric_cicd import change_log_level\n\n# Enable debug logging (call before other fabric-cicd operations)\nchange_log_level()\n```\n\nWhen debug logging is enabled, all API calls are logged with detailed request/response information, and additional context about internal operations is displayed. Both the console and the `fabric_cicd.error.log` file will contain the detailed information.\n\n**Important:** Always enable debug logging when troubleshooting deployment issues. The additional output helps identify whether problems originate from API calls, authentication, or configuration. See [Understanding Error Logs](#understanding-error-logs) for details on interpreting log output.\n\n### Testing Deployments Locally\n\nBefore running deployments via CI/CD pipelines, users can test the deployment workflow locally by running the provided debug scripts. This helps with:\n\n- Validating configuration changes without affecting production\n- Testing parameter file configurations\n- Debugging deployment issues\n- Verifying authentication and permissions\n\nfabric-cicd includes several debug scripts in the `devtools/` directory that allow users to run deployments against real workspaces in a controlled environment. See [Debug Scripts](#debug-scripts) for detailed information on:\n\n- `debug_local.py` or `debug_local config.py` - Test full deployment workflows\n- `debug_parameterization.py` - Validate parameter files without deploying\n- `debug_api.py` - Test Fabric REST API calls directly\n- `debug_trace_deployment.py` - Perform and end-to-end deployment against a Fabric Workspace and capture HTTP Traces to be used for Integration Tests\n\n**Tip:** Using these scripts locally can catch configuration errors early, saving time in your CI/CD pipeline.\n\n### Sample Workspace Directory\n\nfabric-cicd includes the `sample/workspace/` directory that demonstrates the recommended repository structure for Fabric item source control files. It contains sample items of various supported item types (e.g., Environment, Notebook, Data Pipeline, etc.).\n\n**Repository Directory Structure:**\n\n```\nsample/workspace/\n├── Sample Pipeline.DataPipeline/\n│ ├── .platform\n│ └── pipeline-content.json\n├── Sample_Notebook.Notebook/\n│ ├── .platform\n│ └── notebook-content.py\n...\n└── parameter.yml\n```\n\nEach item folder follows the naming convention `ItemName.ItemType/` and contains:\n\n- `.platform` file which contains the item metadata\n- Item definition files (e.g., `pipeline-content.json`, `notebook-content.py`)\n\n**Using the Sample:**\n\nUse this sample structure as a template for organizing your Fabric items. To test deployments with the items found in the sample workspace, set `repository_directory = \"sample/workspace\"` in `debug_local.py` or in `config.yml` when running `debug_local config.py`.\n\n### Understanding Error Logs\n\nWhen running a deployment, fabric-cicd automatically creates a `fabric_cicd.error.log` file in your working directory. The level of detail captured depends on whether [debug logging is enabled](#enable-debug-logging).\n\n**Tip:** Always enable debug logging when troubleshooting deployment issues to capture full API traces in the log file.\n\n#### Accessing API Traces\n\nWhen an error occurs during deployment, the console will display:\n\n```\nError: [Brief error message]\n\nSee /path/to/fabric_cicd.error.log for full details.\n```\n\nOpen the `fabric_cicd.error.log` file to view:\n\n1. **Request Details**: The exact API endpoint called, HTTP method, and request body\n2. **Response Details**: Status code, response headers, and complete response body\n3. **Timing Information**: When the call was made\n4. **Stack Trace**: The complete call stack leading to the error\n5. **Additional Logs**: Information on internal operations that occurred during deployment\n\nThis information is critical for determining if issues are caused by:\n\n- API failures or service issues\n- Authentication/authorization problems\n- Invalid request payloads\n- Network connectivity issues\n\n#### Example Error Log Entry\n\n```\n2024-01-06 10:30:45 - ERROR - fabric_cicd.api - API call failed\nRequest: POST https://api.fabric.microsoft.com/v1/workspaces/{workspace_id}/items\nHeaders: {'Authorization': '***', 'Content-Type': 'application/json'}\nBody: {\"displayName\": \"MyNotebook\", \"type\": \"Notebook\", ...}\n\nResponse: 400 Bad Request\nBody: {\"error\": {\"code\": \"InvalidRequest\", \"message\": \"Item name contains invalid characters\"}}\n\nTraceback (most recent call last):\n File \"fabric_cicd/publish.py\", line 123, in publish_item\n response = api.create_item(...)\n ...\n```\n\n### Common Issues and Solutions\n\n#### Authentication Failures\n\n**Symptom**: Errors mentioning \"authentication failed\" or \"401 Unauthorized\"\n\n**Solution**:\n\n1. Explicit authentication is **required** as the `DefaultAzureCredential` fallback has been removed. `token_credential` is now a required parameter. fabric-cicd accepts any [`TokenCredential`](https://learn.microsoft.com/en-us/python/api/azure-core/azure.core.credentials.tokencredential) — choose the appropriate one for your scenario:\n - Local development: `AzureCliCredential` (requires `az login`) or `AzurePowerShellCredential` (requires `Connect-AzAccount`)\n - CI/CD pipelines with platform auth: `AzureCliCredential` or `AzurePowerShellCredential` (requires a prior login step in the workflow, e.g., `azure/login` or AzCLI task)\n - CI/CD pipelines with OIDC / workload identity federation: `WorkloadIdentityCredential` (secretless; recommended for GitHub Actions and Azure DevOps with federated credentials)\n - CI/CD pipelines with service principals: `ClientSecretCredential` (requires client ID, secret, and tenant ID)\n - CI/CD pipelines with managed identity: `ManagedIdentityCredential` (requires Azure-hosted self-hosted runners)\n - Fabric Notebooks: Provide an explicit credential. See Authentication Examples for details.\n\n2. Verify authentication setup:\n ```bash\n az login\n ```\n or\n ```powershell\n Connect-AzAccount\n ```\n3. Check permissions: ensure your account has appropriate permissions on the target workspace\n\n4. For Service Principal authentication: verify client ID, secret, and tenant ID are correct\n\n5. See detailed examples: refer to [authentication examples](../example/authentication.md) for platform-specific implementation guidance\n\n#### Item Deployment Failures\n\n**Symptom**: Specific items fail to deploy while others succeed\n\n**Solution**:\n\n1. Enable debug logging to see the exact API error\n2. Check `fabric_cicd.error.log` for detailed API response\n3. Verify the item definition files exist and are properly formatted\n4. Check if the item type is included in your `item_type_in_scope` list\n5. Ensure item dependencies exist (e.g., a Data Pipeline referencing a Notebook must be deployed along with the Notebook)\n6. If deleting and recreating an item with the same name, wait 5 minutes between operations due to Fabric API item name reservation\n\n#### Parameter Substitution Issues\n\n**Symptom**: Deployed items contain literal find value instead of the proper replace value\n\n**Solution**:\n\n1. Verify your `parameter.yml` file is in the correct location (repository directory by default)\n2. Check that find values in your files exactly match those in `parameter.yml`\n3. Ensure the environment name matches between your script and `parameter.yml`\n4. Validate the find value regex and/or dynamic replacement variables in `parameter.yml`\n5. Use the [debug_parameterization.py](#debug_parameterizationpy) script to validate parameter files\n\n#### Private Link Connection Failures\n\n**Symptom**: API calls fail with connection errors when deploying to a workspace with \"Allow connections only from workspace level private links\" enabled.\n\n**Solution**: Call `configure_fabric_fqdn` before initializing `FabricWorkspace`:\n\n```python\nfrom fabric_cicd import configure_fabric_fqdn, FabricWorkspace\n\nconfigure_fabric_fqdn(workspace_id)\nworkspace = FabricWorkspace(workspace_id=workspace_id, ...)\n```\n\n#### API Rate Limiting\n\n**Symptom**: Deployments fail with \"429 Too Many Requests\" errors\n\n**Solution**:\n\n1. Consider deploying in smaller batches\n2. Check `fabric_cicd.error.log` for retry-after headers in API responses\n\n### Debug Scripts\n\nThe `devtools/` directory contains pre-built scripts to help test and validate deployments, parameter files, and Fabric REST APIs locally. These scripts already exist in the repository - you just need to configure them for your scenario.\n\n#### debug_local.py\n\n**Purpose**: Test full deployment workflows locally against a Microsoft Fabric workspace.\n\n**Key Configuration Options**:\n\n| Configuration | Description | Required |\n| ---------------------- | ---------------------------------------------------------------- | -------- |\n| `workspace_id` | Target Fabric workspace ID | Yes |\n| `environment` | Target environment (used for parameterization) | No |\n| `repository_directory` | Path to Fabric workspace items files (absolute or relative path) | Yes |\n| `item_type_in_scope` | Specific item types to deploy (defaults to all supported types) | No |\n| `token_credential` | Explicit credential method (`AzureCliCredential`, etc.) | Yes |\n\n**Quick Start**:\n\n1. Open `devtools/debug_local.py`\n2. Set `workspace_id`, `environment`, and `repository_directory` at the top\n3. Uncomment `change_log_level()` to enable debug logging\n4. Add necessary [feature flags](optional_feature.md#feature-flags) required for deployment\n5. Uncomment `publish_all_items(target_workspace)` and/or `unpublish_all_orphan_items(target_workspace)` to test deployment\n6. Run: `uv run python devtools/debug_local.py`\n\n**Common Configurations**:\n\n```python\n# Enable debug logging\nchange_log_level()\n\n# Add feature flag(s)\nappend_feature_flag(\"enable_shortcut_publish\")\n\n# Use sample workspace for testing\nrepository_directory = \"sample/workspace\"\n\n# Deploy only specific item types\nitem_type_in_scope = [\"Environment\", \"Notebook\", \"DataPipeline\"]\n\n# Authentication examples - choose one:\n\n# For local development with Azure CLI\nfrom azure.identity import AzureCliCredential\ntoken_credential = AzureCliCredential()\n\n# For local development with Azure PowerShell\nfrom azure.identity import AzurePowerShellCredential\ntoken_credential = AzurePowerShellCredential()\n\n# For CI/CD with Service Principal\nfrom azure.identity import ClientSecretCredential\ntoken_credential = ClientSecretCredential(\n client_id=\"your-client-id\",\n client_secret=\"your-client-secret\",\n tenant_id=\"your-tenant-id\"\n)\n\n# For Azure-hosted runners with Managed Identity\nfrom azure.identity import ManagedIdentityCredential\ntoken_credential = ManagedIdentityCredential()\n\n# Override constant value\nconstants.DEFAULT_API_ROOT_URL = \"https://api.fabric.microsoft.com\"\n```\n\n#### debug_local config.py\n\n**Purpose**: Test configuration-based deployment workflows using a `config.yml` file.\n\n**Key Configuration Options**:\n\n| Configuration | Description | Required |\n| ------------------ | ----------------------------------------------------------------------------------- | -------- |\n| `config_file` | Path to your `config.yml` file | Yes |\n| `token_credential` | Explicit credential method (`AzureCliCredential`, etc.) | Yes |\n| `environment` | Target environment (used for parameterization and environment-based configurations) | No |\n| `config_override` | Dictionary to override configuration values within `config.yml` | No |\n\n**Quick Start**:\n\n1. Open `devtools/debug_local config.py`\n2. Set `config_file` path and `environment` (can use the sample `config.yml` file found in `sample/workspace`)\n3. Uncomment `change_log_level()` to enable debug logging\n4. Ensure required feature flags are enabled (already set in script)\n5. Run: `uv run python \"devtools/debug_local config.py\"`\n\nSee [configuration deployment](config_deployment.md) for details on creating `config.yml`.\n\n#### debug_parameterization.py\n\n**Purpose**: Validate parameter file without deploying items - useful for catching parameterization errors early.\n\n**Key Configuration Options**:\n\n| Configuration | Description | Required |\n| ---------------------- | ------------------------------------------------------------------------------------ | -------- |\n| `repository_directory` | Path to Fabric workspace items files and `parameter.yml` file (default location) | Yes |\n| `environment` | Target environment (used for parameterization) | No |\n| `item_type_in_scope` | Item types to validate (defaults to all) | No |\n| `parameter_file_name` | Alternate parameter file name within repository directory (default: `parameter.yml`) | No |\n| `parameter_file_path` | Alternate location of parameter file | No |\n\n**Quick Start**:\n\n1. Open `devtools/debug_parameterization.py`\n2. Set `repository_directory` and `environment` (the `parameter.yml` file is located in the repository directory in this case)\n3. Uncomment `change_log_level()` to view all the validation steps\n4. Run: `uv run python devtools/debug_parameterization.py`\n\nSee [parameterization](parameterization.md#parameter-file-validation) for more information.\n\n#### debug_api.py\n\n**Purpose**: Test Fabric REST API calls directly without going through full deployment workflows.\n\n**Key Configuration Options**:\n\n| Configuration | Description | Required |\n| ------------------ | ------------------------------------------------------------------- | -------- |\n| `token_credential` | Explicit credential method (`AzureCliCredential`, etc.) | Yes |\n| `api_url` | Full API endpoint URL | Yes |\n| `method` | HTTP method (GET, POST, DELETE, PATCH) | Yes |\n| `body` | Request payload (for POST/PATCH) | Varies |\n| other | View `invoke()` in `FabricEndpoint` class for additional parameters | No |\n\n**Quick Start**:\n\n1. Open `devtools/debug_api.py`\n2. Configure the API endpoint, method, body (if any)\n3. Uncomment `change_log_level()` to view API request/response details\n4. Run: `uv run python devtools/debug_api.py`\n\n#### debug_trace_deployment.py\n\n**Purpose**: Debug the public APIs called in `publish_all_items()` workflow with breakpoints using VS Code's debugger.\n\n**Quick Start**:\n\n1. Open `devtools/debug_trace_deployment.py`\n2. Set breakpoint(s) in the code - e.g. prior to `publish_all_items`\n3. Update `.vscode/launch.json` with your workspace ID in `FABRIC_WORKSPACE_ID`\n4. Press **F5** → Select \"Debug: Publish All Items\"\n\n## Getting Help\n\nIf you're still experiencing issues after following this guide:\n\n1. **Enable debug logging** and capture the complete error log\n2. **Check existing issues** on [GitHub](https://github.com/microsoft/fabric-cicd/issues)\n3. **Create a new issue** using the appropriate template:\n - [Bug Report](https://github.com/microsoft/fabric-cicd/issues/new?template=1-bug.yml)\n - [Question](https://github.com/microsoft/fabric-cicd/issues/new?template=4-question.yml)\n\n## Additional Resources\n\n- [Authentication Examples](../example/authentication.md) - Comprehensive authentication implementation examples\n- [Contribution Guide](https://github.com/microsoft/fabric-cicd/blob/main/CONTRIBUTING.md) - Setup instructions and PR requirements\n- [Feature Flags](optional_feature.md#feature-flags) - Available feature flags for advanced scenarios\n- [Getting Started](getting_started.md) - Basic installation and authentication\n- [Microsoft Fabric API Documentation](https://learn.microsoft.com/en-us/rest/api/fabric/) - Official API reference\n" }, { "path": "docs/index.md", "content": "fabric-cicd is a Python library designed for use with [Microsoft Fabric](https://learn.microsoft.com/en-us/fabric/) workspaces. This library supports code-first Continuous Integration / Continuous Deployment (CI/CD) automations to seamlessly integrate Source Controlled workspaces into a deployment framework. The goal is to assist CI/CD developers who prefer not to interact directly with the Microsoft Fabric APIs.\n\n## Base Expectations\n\n- Full deployment every time, without considering commit diffs\n- Deploys into the tenant of the executing identity\n- Only supports items that have Source Control, and Public Create/Update APIs\n\n## Supported Item Types\n\n\n\n\n## Installation\n\n**Requirements**: Python to \n\nTo install fabric-cicd, run:\n\n```bash\npip install fabric-cicd\n```\n\n## Basic Example\n\n```python\nfrom azure.identity import AzureCliCredential\nfrom fabric_cicd import FabricWorkspace, publish_all_items, unpublish_all_orphan_items\n\ntoken_credential = AzureCliCredential()\n\n# Initialize the FabricWorkspace object with the required parameters\ntarget_workspace = FabricWorkspace(\n workspace_id=\"your-workspace-id\",\n environment=\"your-target-environment\",\n repository_directory=\"your-repository-directory\",\n item_type_in_scope=[\"Notebook\", \"DataPipeline\", \"Environment\"],\n token_credential=token_credential, # or any other TokenCredential\n)\n\n# Publish all items defined in item_type_in_scope\npublish_all_items(target_workspace)\n\n# Unpublish all items defined in item_type_in_scope not found in repository\nunpublish_all_orphan_items(target_workspace)\n```\n\n> **Notes:**\n>\n> - All parameters for `FabricWorkspace` must be passed as keyword arguments.\n> - The `environment` parameter is required for parameter replacement to work properly. It must match one of the environment keys defined in your `parameter.yml` file (e.g., \"PPE\", \"PROD\", \"DEV\"). If you don't need parameter replacement, you can omit this parameter.\n" }, { "path": "mkdocs.yml", "content": "site_name: fabric-cicd\n\nrepo_name: microsoft/fabric-cicd\nrepo_url: https://github.com/microsoft/fabric-cicd\nsite_url: https://microsoft.github.io/fabric-cicd/\nremote_branch: gh-pages\nremote_name: origin\n\nnav:\n - Home: index.md\n - How To:\n - How To: how_to/index.md\n - Getting Started: how_to/getting_started.md\n - Item Types: how_to/item_types.md\n - Configuration Deployment: how_to/config_deployment.md\n - Parameterization: how_to/parameterization.md\n - Optional Features: how_to/optional_feature.md\n - Troubleshooting: how_to/troubleshooting.md\n - Examples:\n - Examples: example/index.md\n - Authentication: example/authentication.md\n - Deployment Variables: example/deployment_variable.md\n - Release Pipelines: example/release_pipeline.md\n - Code Reference: code_reference.md\n - Changelog: changelog.md\n - About: about.md\n\ntheme:\n name: material\n custom_dir: docs/config/overrides\n icon:\n logo: fontawesome/solid/code\n favicon: config/assets/favicon.ico\n font:\n text: Roboto\n code: Roboto Mono\n\n features:\n - content.code.copy\n - content.tooltips\n - navigation.expand\n - navigation.indexes\n - navigation.sections\n - navigation.tabs\n - navigation.tabs.sticky\n - navigation.top\n - search.highlight\n - search.suggest\n - toc.follow\n - toc.integrate\n - announce.dismiss\n language: en\n palette:\n scheme: fabric\n\nextra_css:\n - config/stylesheets/extra.css\nhooks:\n - docs/config/pre-build/update_item_types.py\n - docs/config/pre-build/section_toc.py\n - docs/config/pre-build/update_python_version.py\n\nplugins:\n - search:\n separator: '[\\s\\u200b\\-_,:!=\\[\\]()\"`/]+|\\.(?!\\d)|&[lg]t;|(?!\\b)(?=[A-Z][a-z])'\n - minify:\n minify_html: true\n - include-markdown\n - mkdocstrings:\n handlers:\n python:\n paths: [\"src\"]\n options:\n docstring_style: google\n docstring_options:\n ignore_init_summary: true\n summary:\n modules: false\n functions: true\n classes: true\n show_source: false\n show_root_full_path: false\n separate_signature: true\n show_signature_annotations: true\n signature_crossrefs: true\n filters: [\"!^_\", \"^__init__$\"]\n merge_init_into_class: true\n show_symbol_type_heading: true\n show_symbol_type_toc: true\n line_length: 80\n\nextra:\n version:\n provider: mike\n default: latest\n alias: true\n social:\n - icon: fontawesome/brands/github\n link: https://github.com/microsoft/fabric-cicd\n generator: false\n analytics:\n feedback:\n title: Was this page helpful?\n ratings:\n - icon: material/thumb-up-outline\n name: This page was helpful\n data: 1\n note: Thanks for your feedback!\n - icon: material/thumb-down-outline\n name: This page could be improved\n data: 0\n note: >-\n Thanks for your feedback! Help us improve this page by using our \n feedback form.\n\nmarkdown_extensions:\n - abbr\n - md_in_html\n - toc:\n permalink: true\n toc_depth: 2\n - admonition\n - pymdownx.highlight:\n anchor_linenums: true\n line_spans: __span\n pygments_lang_class: true\n - pymdownx.inlinehilite\n - pymdownx.snippets\n - pymdownx.superfences\n - pymdownx.tabbed:\n alternate_style: true\n - pymdownx.superfences\n\ncopyright: Copyright © Microsoft Corporation\n" }, { "path": "pyproject.toml", "content": "[project]\nname = \"fabric-cicd\"\nauthors = [{ name = \"Microsoft Corporation\" }]\ndescription = \"Microsoft Fabric CI/CD\"\nreadme = \"README.md\"\nrequires-python = \">=3.9,<3.14\"\nlicense = \"MIT\"\nlicense-files = [\"LICENSE\"]\ndynamic = [\"version\"]\nclassifiers = [\n \"Programming Language :: Python\",\n \"Programming Language :: Python :: 3.9\",\n \"Programming Language :: Python :: 3.10\",\n \"Programming Language :: Python :: 3.11\",\n \"Programming Language :: Python :: 3.12\",\n \"Programming Language :: Python :: 3.13\",\n]\ndependencies = [\n \"azure-identity>=1.25.0\",\n \"dpath>=2.2.0\",\n \"filetype>=1.2.0\",\n \"jsonpath-ng>=1.8.0\",\n \"pyyaml>=6.0.2\",\n \"requests>=2.32.3\",\n]\n\n[project.urls]\nRepository = \"https://github.com/microsoft/fabric-cicd.git\"\nChangelog = \"https://github.com/microsoft/fabric-cicd/blob/main/docs/changelog.md\"\n\n[dependency-groups]\ndev = [\n \"coverage>=7.6.10\",\n \"gitpython>=3.1.44\",\n \"mike>=2.1.3\",\n \"mkdocs-include-markdown-plugin>=7.1.2\",\n \"mkdocs-material>=9.6.5\",\n \"mkdocs-minify-plugin>=0.8.0\",\n \"mkdocstrings-python>=1.13.0\",\n \"pygments>=2.18.0,<2.20.0\",\n \"pytest>=8.3.4\",\n \"pytest-mock>=3.14.0\",\n \"ruff>=0.9.5\",\n]\n\n[build-system]\nrequires = [\"setuptools>=61.0\"]\nbuild-backend = \"setuptools.build_meta\"\n\n[tool.setuptools]\ndynamic.version = { attr = \"fabric_cicd.constants.VERSION\" }\npackages.find.where = [\"src\"]\n\n[tool.uv]\npackage = true\npython-preference = \"only-managed\"\n\n[tool.pytest.ini_options]\naddopts = \"-v --tb=short\"\ntestpaths = [\"tests\"]\npythonpath = [\"src\"]\n\n[tool.coverage.run]\nsource = [\"src\"]\nomit = [\"tests/*\"]\n" }, { "path": "ruff.toml", "content": "line-length = 120\nexclude = [\"sample/*\", \"docs/*\"]\n\n[lint]\n# https://docs.astral.sh/ruff/rules/\nselect = [\n \"A\", # flake8-builtins\n \"ANN\", # flake8-annotations-complexity\n \"ARG\", # flake8-unused-arguments\n \"B\", # flake8-bugbear\n \"D\", # pydocstyle\n \"EM\", # flake8-errmsg\n \"F\", # Pyflakes\n \"I\", # isort\n \"INP\", # flake8-no-pep420\n \"N\", # pep8-naming\n \"PT\", # flake8-pytest-style\n \"PTH\", # flake8-use-pathlib\n \"RET\", # flake8-return\n \"RUF\", # Ruff-specific rules\n \"SIM\", # flake8-simplify\n \"UP\", # pyupgrade\n]\nignore = [\n \"D203\", # No blank lines between a section header and its content\n \"D205\", # 1 blank line required between summary line and description\n \"D212\", # Multi-line docstring summary should start at the first line\n \"D400\", # First line should end with a period\n \"D401\", # First line should be in imperative mood\n \"D415\", # First line should end with a period, question mark, or exclamation point\n \"ANN003\", # Missing type annotation for kwargs\n \"RUF100\", # Unused noqa directive (version-dependent on ruff and not deterministic)\n]\n\npydocstyle.convention = \"google\"\n\n[lint.per-file-ignores]\n\"{tests}/*\" = [\n \"D\", # pydocstyle\n \"INP\", # flake8-no-pep420\n \"ANN\", # flake8-annotations-complexity\n]\n\n\"{devtools}/*\" = [\n \"D\", # pydocstyle\n \"INP\", # flake8-no-pep420\n \"F401\", # unused import\n \"ANN\", # flake8-annotations-complexity\n]\n\n[format]\npreview = true\ndocstring-code-format = false\ndocstring-code-line-length = \"dynamic\"\n" }, { "path": "sample/workspace/ABC.Report/.platform", "content": "{\n \"$schema\": \"https://developer.microsoft.com/json-schemas/fabric/gitIntegration/platformProperties/2.0.0/schema.json\",\n \"metadata\": {\n \"type\": \"Report\",\n \"displayName\": \"ABC\"\n },\n \"config\": {\n \"version\": \"2.0\",\n \"logicalId\": \"40db346f-1c24-8aa6-48c7-a3aec4199351\"\n }\n}\n" }, { "path": "sample/workspace/ABC.Report/StaticResources/SharedResources/BaseThemes/CY24SU10.json", "content": "{\n \"name\": \"CY24SU10\",\n \"dataColors\": [\n \"#118DFF\",\n \"#12239E\",\n \"#E66C37\",\n \"#6B007B\",\n \"#E044A7\",\n \"#744EC2\",\n \"#D9B300\",\n \"#D64550\",\n \"#197278\",\n \"#1AAB40\",\n \"#15C6F4\",\n \"#4092FF\",\n \"#FFA058\",\n \"#BE5DC9\",\n \"#F472D0\",\n \"#B5A1FF\",\n \"#C4A200\",\n \"#FF8080\",\n \"#00DBBC\",\n \"#5BD667\",\n \"#0091D5\",\n \"#4668C5\",\n \"#FF6300\",\n \"#99008A\",\n \"#EC008C\",\n \"#533285\",\n \"#99700A\",\n \"#FF4141\",\n \"#1F9A85\",\n \"#25891C\",\n \"#0057A2\",\n \"#002050\",\n \"#C94F0F\",\n \"#450F54\",\n \"#B60064\",\n \"#34124F\",\n \"#6A5A29\",\n \"#1AAB40\",\n \"#BA141A\",\n \"#0C3D37\",\n \"#0B511F\"\n ],\n \"foreground\": \"#252423\",\n \"foregroundNeutralSecondary\": \"#605E5C\",\n \"foregroundNeutralTertiary\": \"#B3B0AD\",\n \"background\": \"#FFFFFF\",\n \"backgroundLight\": \"#F3F2F1\",\n \"backgroundNeutral\": \"#C8C6C4\",\n \"tableAccent\": \"#118DFF\",\n \"good\": \"#1AAB40\",\n \"neutral\": \"#D9B300\",\n \"bad\": \"#D64554\",\n \"maximum\": \"#118DFF\",\n \"center\": \"#D9B300\",\n \"minimum\": \"#DEEFFF\",\n \"null\": \"#FF7F48\",\n \"hyperlink\": \"#0078d4\",\n \"visitedHyperlink\": \"#0078d4\",\n \"textClasses\": {\n \"callout\": {\n \"fontSize\": 45,\n \"fontFace\": \"DIN\",\n \"color\": \"#252423\"\n },\n \"title\": {\n \"fontSize\": 12,\n \"fontFace\": \"DIN\",\n \"color\": \"#252423\"\n },\n \"header\": {\n \"fontSize\": 12,\n \"fontFace\": \"Segoe UI Semibold\",\n \"color\": \"#252423\"\n },\n \"label\": {\n \"fontSize\": 10,\n \"fontFace\": \"Segoe UI\",\n \"color\": \"#252423\"\n }\n },\n \"visualStyles\": {\n \"*\": {\n \"*\": {\n \"*\": [\n {\n \"wordWrap\": true\n }\n ],\n \"line\": [\n {\n \"transparency\": 0\n }\n ],\n \"outline\": [\n {\n \"transparency\": 0\n }\n ],\n \"plotArea\": [\n {\n \"transparency\": 0\n }\n ],\n \"categoryAxis\": [\n {\n \"showAxisTitle\": true,\n \"gridlineStyle\": \"dotted\",\n \"concatenateLabels\": false\n }\n ],\n \"valueAxis\": [\n {\n \"showAxisTitle\": true,\n \"gridlineStyle\": \"dotted\"\n }\n ],\n \"y2Axis\": [\n {\n \"show\": true\n }\n ],\n \"title\": [\n {\n \"titleWrap\": true\n }\n ],\n \"lineStyles\": [\n {\n \"strokeWidth\": 3\n }\n ],\n \"wordWrap\": [\n {\n \"show\": true\n }\n ],\n \"background\": [\n {\n \"show\": true,\n \"transparency\": 0\n }\n ],\n \"border\": [\n {\n \"width\": 1\n }\n ],\n \"outspacePane\": [\n {\n \"backgroundColor\": {\n \"solid\": {\n \"color\": \"#ffffff\"\n }\n },\n \"transparency\": 0,\n \"border\": true,\n \"borderColor\": {\n \"solid\": {\n \"color\": \"#B3B0AD\"\n }\n }\n }\n ],\n \"filterCard\": [\n {\n \"$id\": \"Applied\",\n \"transparency\": 0,\n \"foregroundColor\": {\n \"solid\": {\n \"color\": \"#252423\"\n }\n },\n \"border\": true\n },\n {\n \"$id\": \"Available\",\n \"transparency\": 0,\n \"foregroundColor\": {\n \"solid\": {\n \"color\": \"#252423\"\n }\n },\n \"border\": true\n }\n ]\n }\n },\n \"scatterChart\": {\n \"*\": {\n \"bubbles\": [\n {\n \"bubbleSize\": -10,\n \"markerRangeType\": \"auto\"\n }\n ],\n \"general\": [\n {\n \"responsive\": true\n }\n ],\n \"fillPoint\": [\n {\n \"show\": true\n }\n ],\n \"legend\": [\n {\n \"showGradientLegend\": true\n }\n ]\n }\n },\n \"lineChart\": {\n \"*\": {\n \"general\": [\n {\n \"responsive\": true\n }\n ],\n \"smallMultiplesLayout\": [\n {\n \"backgroundTransparency\": 0,\n \"gridLineType\": \"inner\"\n }\n ],\n \"forecast\": [\n {\n \"matchSeriesInterpolation\": true\n }\n ]\n }\n },\n \"map\": {\n \"*\": {\n \"bubbles\": [\n {\n \"bubbleSize\": -10,\n \"markerRangeType\": \"auto\"\n }\n ]\n }\n },\n \"azureMap\": {\n \"*\": {\n \"bubbleLayer\": [\n {\n \"bubbleRadius\": 8,\n \"minBubbleRadius\": 8,\n \"maxRadius\": 40\n }\n ],\n \"barChart\": [\n {\n \"barHeight\": 3,\n \"thickness\": 3\n }\n ]\n }\n },\n \"pieChart\": {\n \"*\": {\n \"legend\": [\n {\n \"show\": true,\n \"position\": \"RightCenter\"\n }\n ],\n \"labels\": [\n {\n \"labelStyle\": \"Data value, percent of total\"\n }\n ]\n }\n },\n \"donutChart\": {\n \"*\": {\n \"legend\": [\n {\n \"show\": true,\n \"position\": \"RightCenter\"\n }\n ],\n \"labels\": [\n {\n \"labelStyle\": \"Data value, percent of total\"\n }\n ]\n }\n },\n \"pivotTable\": {\n \"*\": {\n \"rowHeaders\": [\n {\n \"showExpandCollapseButtons\": true,\n \"legacyStyleDisabled\": true\n }\n ]\n }\n },\n \"multiRowCard\": {\n \"*\": {\n \"card\": [\n {\n \"outlineWeight\": 2,\n \"barShow\": true,\n \"barWeight\": 2\n }\n ]\n }\n },\n \"kpi\": {\n \"*\": {\n \"trendline\": [\n {\n \"transparency\": 20\n }\n ]\n }\n },\n \"cardVisual\": {\n \"*\": {\n \"layout\": [\n {\n \"maxTiles\": 3\n }\n ],\n \"overflow\": [\n {\n \"type\": 0\n }\n ],\n \"image\": [\n {\n \"fixedSize\": false\n },\n {\n \"imageAreaSize\": 50\n }\n ]\n }\n },\n \"advancedSlicerVisual\": {\n \"*\": {\n \"layout\": [\n {\n \"maxTiles\": 3\n }\n ]\n }\n },\n \"slicer\": {\n \"*\": {\n \"general\": [\n {\n \"responsive\": true\n }\n ],\n \"date\": [\n {\n \"hideDatePickerButton\": false\n }\n ],\n \"items\": [\n {\n \"padding\": 4,\n \"accessibilityContrastProperties\": true\n }\n ]\n }\n },\n \"waterfallChart\": {\n \"*\": {\n \"general\": [\n {\n \"responsive\": true\n }\n ]\n }\n },\n \"columnChart\": {\n \"*\": {\n \"general\": [\n {\n \"responsive\": true\n }\n ],\n \"legend\": [\n {\n \"showGradientLegend\": true\n }\n ],\n \"smallMultiplesLayout\": [\n {\n \"backgroundTransparency\": 0,\n \"gridLineType\": \"inner\"\n }\n ]\n }\n },\n \"clusteredColumnChart\": {\n \"*\": {\n \"general\": [\n {\n \"responsive\": true\n }\n ],\n \"legend\": [\n {\n \"showGradientLegend\": true\n }\n ],\n \"smallMultiplesLayout\": [\n {\n \"backgroundTransparency\": 0,\n \"gridLineType\": \"inner\"\n }\n ]\n }\n },\n \"hundredPercentStackedColumnChart\": {\n \"*\": {\n \"general\": [\n {\n \"responsive\": true\n }\n ],\n \"legend\": [\n {\n \"showGradientLegend\": true\n }\n ],\n \"smallMultiplesLayout\": [\n {\n \"backgroundTransparency\": 0,\n \"gridLineType\": \"inner\"\n }\n ]\n }\n },\n \"barChart\": {\n \"*\": {\n \"general\": [\n {\n \"responsive\": true\n }\n ],\n \"legend\": [\n {\n \"showGradientLegend\": true\n }\n ],\n \"smallMultiplesLayout\": [\n {\n \"backgroundTransparency\": 0,\n \"gridLineType\": \"inner\"\n }\n ]\n }\n },\n \"clusteredBarChart\": {\n \"*\": {\n \"general\": [\n {\n \"responsive\": true\n }\n ],\n \"legend\": [\n {\n \"showGradientLegend\": true\n }\n ],\n \"smallMultiplesLayout\": [\n {\n \"backgroundTransparency\": 0,\n \"gridLineType\": \"inner\"\n }\n ]\n }\n },\n \"hundredPercentStackedBarChart\": {\n \"*\": {\n \"general\": [\n {\n \"responsive\": true\n }\n ],\n \"legend\": [\n {\n \"showGradientLegend\": true\n }\n ],\n \"smallMultiplesLayout\": [\n {\n \"backgroundTransparency\": 0,\n \"gridLineType\": \"inner\"\n }\n ]\n }\n },\n \"areaChart\": {\n \"*\": {\n \"general\": [\n {\n \"responsive\": true\n }\n ],\n \"smallMultiplesLayout\": [\n {\n \"backgroundTransparency\": 0,\n \"gridLineType\": \"inner\"\n }\n ]\n }\n },\n \"stackedAreaChart\": {\n \"*\": {\n \"general\": [\n {\n \"responsive\": true\n }\n ],\n \"smallMultiplesLayout\": [\n {\n \"backgroundTransparency\": 0,\n \"gridLineType\": \"inner\"\n }\n ]\n }\n },\n \"lineClusteredColumnComboChart\": {\n \"*\": {\n \"general\": [\n {\n \"responsive\": true\n }\n ],\n \"smallMultiplesLayout\": [\n {\n \"backgroundTransparency\": 0,\n \"gridLineType\": \"inner\"\n }\n ]\n }\n },\n \"lineStackedColumnComboChart\": {\n \"*\": {\n \"general\": [\n {\n \"responsive\": true\n }\n ],\n \"smallMultiplesLayout\": [\n {\n \"backgroundTransparency\": 0,\n \"gridLineType\": \"inner\"\n }\n ]\n }\n },\n \"ribbonChart\": {\n \"*\": {\n \"general\": [\n {\n \"responsive\": true\n }\n ],\n \"smallMultiplesLayout\": [\n {\n \"backgroundTransparency\": 0,\n \"gridLineType\": \"inner\"\n }\n ],\n \"valueAxis\": [\n {\n \"show\": true\n }\n ]\n }\n },\n \"hundredPercentStackedAreaChart\": {\n \"*\": {\n \"general\": [\n {\n \"responsive\": true\n }\n ],\n \"smallMultiplesLayout\": [\n {\n \"backgroundTransparency\": 0,\n \"gridLineType\": \"inner\"\n }\n ]\n }\n },\n \"group\": {\n \"*\": {\n \"background\": [\n {\n \"show\": false\n }\n ]\n }\n },\n \"basicShape\": {\n \"*\": {\n \"background\": [\n {\n \"show\": false\n }\n ],\n \"general\": [\n {\n \"keepLayerOrder\": true\n }\n ],\n \"visualHeader\": [\n {\n \"show\": false\n }\n ]\n }\n },\n \"shape\": {\n \"*\": {\n \"background\": [\n {\n \"show\": false\n }\n ],\n \"general\": [\n {\n \"keepLayerOrder\": true\n }\n ],\n \"visualHeader\": [\n {\n \"show\": false\n }\n ]\n }\n },\n \"image\": {\n \"*\": {\n \"background\": [\n {\n \"show\": false\n }\n ],\n \"general\": [\n {\n \"keepLayerOrder\": true\n }\n ],\n \"visualHeader\": [\n {\n \"show\": false\n }\n ],\n \"lockAspect\": [\n {\n \"show\": true\n }\n ]\n }\n },\n \"actionButton\": {\n \"*\": {\n \"background\": [\n {\n \"show\": false\n }\n ],\n \"visualHeader\": [\n {\n \"show\": false\n }\n ]\n }\n },\n \"pageNavigator\": {\n \"*\": {\n \"background\": [\n {\n \"show\": false\n }\n ],\n \"visualHeader\": [\n {\n \"show\": false\n }\n ]\n }\n },\n \"bookmarkNavigator\": {\n \"*\": {\n \"background\": [\n {\n \"show\": false\n }\n ],\n \"visualHeader\": [\n {\n \"show\": false\n }\n ]\n }\n },\n \"textbox\": {\n \"*\": {\n \"general\": [\n {\n \"keepLayerOrder\": true\n }\n ],\n \"visualHeader\": [\n {\n \"show\": false\n }\n ]\n }\n },\n \"page\": {\n \"*\": {\n \"outspace\": [\n {\n \"color\": {\n \"solid\": {\n \"color\": \"#FFFFFF\"\n }\n }\n }\n ],\n \"background\": [\n {\n \"transparency\": 100\n }\n ]\n }\n }\n }\n}" }, { "path": "sample/workspace/ABC.Report/definition.pbir", "content": "{\n \"version\": \"4.0\",\n \"datasetReference\": {\n \"byPath\": {\n \"path\": \"../ABC.SemanticModel\"\n }\n }\n}" }, { "path": "sample/workspace/ABC.Report/report.json", "content": "{\n \"config\": \"{\\\"version\\\":\\\"5.59\\\",\\\"themeCollection\\\":{\\\"baseTheme\\\":{\\\"name\\\":\\\"CY24SU10\\\",\\\"version\\\":\\\"5.61\\\",\\\"type\\\":2}},\\\"activeSectionIndex\\\":3,\\\"defaultDrillFilterOtherVisuals\\\":true,\\\"linguisticSchemaSyncVersion\\\":2,\\\"settings\\\":{\\\"useNewFilterPaneExperience\\\":true,\\\"allowChangeFilterTypes\\\":true,\\\"useStylableVisualContainerHeader\\\":true,\\\"queryLimitOption\\\":6,\\\"useEnhancedTooltips\\\":true,\\\"exportDataMode\\\":1,\\\"useDefaultAggregateDisplayName\\\":true},\\\"objects\\\":{\\\"section\\\":[{\\\"properties\\\":{\\\"verticalAlignment\\\":{\\\"expr\\\":{\\\"Literal\\\":{\\\"Value\\\":\\\"'Top'\\\"}}}}}]}}\",\n \"layoutOptimization\": 0,\n \"resourcePackages\": [\n {\n \"resourcePackage\": {\n \"disabled\": false,\n \"items\": [\n {\n \"name\": \"CY24SU10\",\n \"path\": \"BaseThemes/CY24SU10.json\",\n \"type\": 202\n }\n ],\n \"name\": \"SharedResources\",\n \"type\": 2\n }\n },\n {\n \"resourcePackage\": {\n \"disabled\": false,\n \"items\": [\n {\n \"name\": \"test_image13726994662591707.bmp\",\n \"path\": \"test_image13726994662591707.bmp\",\n \"type\": 100\n },\n {\n \"name\": \"test_image15532910574958114.gif\",\n \"path\": \"test_image15532910574958114.gif\",\n \"type\": 100\n },\n {\n \"name\": \"test_image25861853181026917.tif\",\n \"path\": \"test_image25861853181026917.tif\",\n \"type\": 100\n },\n {\n \"name\": \"test_image9544675947263683.jpg\",\n \"path\": \"test_image9544675947263683.jpg\",\n \"type\": 100\n },\n {\n \"name\": \"test_image9896482226816141.png\",\n \"path\": \"test_image9896482226816141.png\",\n \"type\": 100\n }\n ],\n \"name\": \"RegisteredResources\",\n \"type\": 1\n }\n }\n ],\n \"sections\": [\n {\n \"config\": \"{}\",\n \"displayName\": \"Page 3\",\n \"displayOption\": 1,\n \"filters\": \"[]\",\n \"height\": 720.00,\n \"name\": \"63e290aa19e7bea3d03d\",\n \"ordinal\": 2,\n \"visualContainers\": [\n {\n \"config\": \"{\\\"name\\\":\\\"39eb354ddb07467c1948\\\",\\\"layouts\\\":[{\\\"id\\\":0,\\\"position\\\":{\\\"x\\\":525.312,\\\"y\\\":169.984,\\\"z\\\":0,\\\"width\\\":279.552,\\\"height\\\":279.552,\\\"tabOrder\\\":0}}],\\\"singleVisual\\\":{\\\"visualType\\\":\\\"hundredPercentStackedBarChart\\\",\\\"projections\\\":{\\\"Category\\\":[{\\\"queryRef\\\":\\\"Table.Column1\\\",\\\"active\\\":true}]},\\\"prototypeQuery\\\":{\\\"Version\\\":2,\\\"From\\\":[{\\\"Name\\\":\\\"t\\\",\\\"Entity\\\":\\\"Table\\\",\\\"Type\\\":0}],\\\"Select\\\":[{\\\"Column\\\":{\\\"Expression\\\":{\\\"SourceRef\\\":{\\\"Source\\\":\\\"t\\\"}},\\\"Property\\\":\\\"Column1\\\"},\\\"Name\\\":\\\"Table.Column1\\\",\\\"NativeReferenceName\\\":\\\"Column1\\\"}]},\\\"drillFilterOtherVisuals\\\":true}}\",\n \"filters\": \"[]\",\n \"height\": 279.55,\n \"width\": 279.55,\n \"x\": 525.31,\n \"y\": 169.98,\n \"z\": 0.00\n }\n ],\n \"width\": 1280.00\n },\n {\n \"config\": \"{}\",\n \"displayName\": \"Page 1\",\n \"displayOption\": 1,\n \"filters\": \"[]\",\n \"height\": 720.00,\n \"name\": \"90cb47b35aee21c5a77c\",\n \"visualContainers\": [\n {\n \"config\": \"{\\\"name\\\":\\\"d15aa7ce4fc6c2acb4f2\\\",\\\"layouts\\\":[{\\\"id\\\":0,\\\"position\\\":{\\\"x\\\":342,\\\"y\\\":104,\\\"z\\\":0,\\\"width\\\":280,\\\"height\\\":280,\\\"tabOrder\\\":0}}],\\\"singleVisual\\\":{\\\"visualType\\\":\\\"tableEx\\\",\\\"projections\\\":{\\\"Values\\\":[{\\\"queryRef\\\":\\\"Table.Column1\\\"},{\\\"queryRef\\\":\\\"Table.Column2\\\"}]},\\\"prototypeQuery\\\":{\\\"Version\\\":2,\\\"From\\\":[{\\\"Name\\\":\\\"t\\\",\\\"Entity\\\":\\\"Table\\\",\\\"Type\\\":0}],\\\"Select\\\":[{\\\"Column\\\":{\\\"Expression\\\":{\\\"SourceRef\\\":{\\\"Source\\\":\\\"t\\\"}},\\\"Property\\\":\\\"Column1\\\"},\\\"Name\\\":\\\"Table.Column1\\\",\\\"NativeReferenceName\\\":\\\"Column1\\\"},{\\\"Column\\\":{\\\"Expression\\\":{\\\"SourceRef\\\":{\\\"Source\\\":\\\"t\\\"}},\\\"Property\\\":\\\"Column2\\\"},\\\"Name\\\":\\\"Table.Column2\\\",\\\"NativeReferenceName\\\":\\\"Column2\\\"}]},\\\"drillFilterOtherVisuals\\\":true}}\",\n \"filters\": \"[]\",\n \"height\": 280.00,\n \"width\": 280.00,\n \"x\": 342.00,\n \"y\": 104.00,\n \"z\": 0.00\n }\n ],\n \"width\": 1280.00\n },\n {\n \"config\": \"{}\",\n \"displayName\": \"Page 4\",\n \"displayOption\": 1,\n \"filters\": \"[]\",\n \"height\": 720.00,\n \"name\": \"c744048b570d475206d0\",\n \"ordinal\": 3,\n \"visualContainers\": [\n {\n \"config\": \"{\\\"name\\\":\\\"63c15486d69a1d77ad82\\\",\\\"layouts\\\":[{\\\"id\\\":0,\\\"position\\\":{\\\"x\\\":937.5,\\\"y\\\":27,\\\"width\\\":161,\\\"height\\\":161,\\\"z\\\":4,\\\"tabOrder\\\":4}}],\\\"singleVisual\\\":{\\\"visualType\\\":\\\"image\\\",\\\"drillFilterOtherVisuals\\\":true,\\\"objects\\\":{\\\"general\\\":[{\\\"properties\\\":{\\\"imageUrl\\\":{\\\"expr\\\":{\\\"ResourcePackageItem\\\":{\\\"PackageName\\\":\\\"RegisteredResources\\\",\\\"PackageType\\\":1,\\\"ItemName\\\":\\\"test_image25861853181026917.tif\\\"}}}}}]}},\\\"howCreated\\\":\\\"InsertVisualButton\\\"}\",\n \"filters\": \"[]\",\n \"height\": 161.00,\n \"width\": 161.00,\n \"x\": 937.50,\n \"y\": 27.00,\n \"z\": 4.00\n },\n {\n \"config\": \"{\\\"name\\\":\\\"6ac48140b12eb0ab5bac\\\",\\\"layouts\\\":[{\\\"id\\\":0,\\\"position\\\":{\\\"x\\\":22.426995457495135,\\\"y\\\":49.00713822193381,\\\"z\\\":0,\\\"width\\\":161.14211550940948,\\\"height\\\":161.14211550940948,\\\"tabOrder\\\":0}}],\\\"singleVisual\\\":{\\\"visualType\\\":\\\"image\\\",\\\"drillFilterOtherVisuals\\\":true,\\\"objects\\\":{\\\"general\\\":[{\\\"properties\\\":{\\\"imageUrl\\\":{\\\"expr\\\":{\\\"ResourcePackageItem\\\":{\\\"PackageName\\\":\\\"RegisteredResources\\\",\\\"PackageType\\\":1,\\\"ItemName\\\":\\\"test_image13726994662591707.bmp\\\"}}}}}]}},\\\"howCreated\\\":\\\"InsertVisualButton\\\"}\",\n \"filters\": \"[]\",\n \"height\": 161.14,\n \"width\": 161.14,\n \"x\": 22.43,\n \"y\": 49.01,\n \"z\": 0.00\n },\n {\n \"config\": \"{\\\"name\\\":\\\"8bce33fe60a0c78d1459\\\",\\\"layouts\\\":[{\\\"id\\\":0,\\\"position\\\":{\\\"x\\\":224.26995457495133,\\\"y\\\":49.00713822193381,\\\"z\\\":1,\\\"width\\\":161.14211550940948,\\\"height\\\":161.14211550940948,\\\"tabOrder\\\":1}}],\\\"singleVisual\\\":{\\\"visualType\\\":\\\"image\\\",\\\"drillFilterOtherVisuals\\\":true,\\\"objects\\\":{\\\"general\\\":[{\\\"properties\\\":{\\\"imageUrl\\\":{\\\"expr\\\":{\\\"ResourcePackageItem\\\":{\\\"PackageName\\\":\\\"RegisteredResources\\\",\\\"PackageType\\\":1,\\\"ItemName\\\":\\\"test_image15532910574958114.gif\\\"}}}}}]}},\\\"howCreated\\\":\\\"InsertVisualButton\\\"}\",\n \"filters\": \"[]\",\n \"height\": 161.14,\n \"width\": 161.14,\n \"x\": 224.27,\n \"y\": 49.01,\n \"z\": 1.00\n },\n {\n \"config\": \"{\\\"name\\\":\\\"ac2f524856312c1c0882\\\",\\\"layouts\\\":[{\\\"id\\\":0,\\\"position\\\":{\\\"x\\\":706.8656716417911,\\\"y\\\":27.410772225827387,\\\"z\\\":3,\\\"width\\\":161.14211550940948,\\\"height\\\":161.14211550940948,\\\"tabOrder\\\":3}}],\\\"singleVisual\\\":{\\\"visualType\\\":\\\"image\\\",\\\"drillFilterOtherVisuals\\\":true,\\\"objects\\\":{\\\"general\\\":[{\\\"properties\\\":{\\\"imageUrl\\\":{\\\"expr\\\":{\\\"ResourcePackageItem\\\":{\\\"PackageName\\\":\\\"RegisteredResources\\\",\\\"PackageType\\\":1,\\\"ItemName\\\":\\\"test_image9896482226816141.png\\\"}}}}}]}},\\\"howCreated\\\":\\\"InsertVisualButton\\\"}\",\n \"filters\": \"[]\",\n \"height\": 161.14,\n \"width\": 161.14,\n \"x\": 706.87,\n \"y\": 27.41,\n \"z\": 3.00\n },\n {\n \"config\": \"{\\\"name\\\":\\\"bb2e1aaeee837cbc07c6\\\",\\\"layouts\\\":[{\\\"id\\\":0,\\\"position\\\":{\\\"x\\\":443.5561323815704,\\\"y\\\":49.00713822193381,\\\"z\\\":2,\\\"width\\\":161.14211550940948,\\\"height\\\":161.14211550940948,\\\"tabOrder\\\":2}}],\\\"singleVisual\\\":{\\\"visualType\\\":\\\"image\\\",\\\"drillFilterOtherVisuals\\\":true,\\\"objects\\\":{\\\"general\\\":[{\\\"properties\\\":{\\\"imageUrl\\\":{\\\"expr\\\":{\\\"ResourcePackageItem\\\":{\\\"PackageName\\\":\\\"RegisteredResources\\\",\\\"PackageType\\\":1,\\\"ItemName\\\":\\\"test_image9544675947263683.jpg\\\"}}}}}]}},\\\"howCreated\\\":\\\"InsertVisualButton\\\"}\",\n \"filters\": \"[]\",\n \"height\": 161.14,\n \"width\": 161.14,\n \"x\": 443.56,\n \"y\": 49.01,\n \"z\": 2.00\n }\n ],\n \"width\": 1280.00\n },\n {\n \"config\": \"{}\",\n \"displayName\": \"Page 2\",\n \"displayOption\": 1,\n \"filters\": \"[]\",\n \"height\": 720.00,\n \"name\": \"f072cad6a715c38915c7\",\n \"ordinal\": 1,\n \"visualContainers\": [\n {\n \"config\": \"{\\\"name\\\":\\\"1e78447fe04d60e1956f\\\",\\\"layouts\\\":[{\\\"id\\\":0,\\\"position\\\":{\\\"x\\\":317.44,\\\"y\\\":69.632,\\\"z\\\":0,\\\"width\\\":279.552,\\\"height\\\":279.552,\\\"tabOrder\\\":0}}],\\\"singleVisual\\\":{\\\"visualType\\\":\\\"areaChart\\\",\\\"projections\\\":{\\\"Category\\\":[{\\\"queryRef\\\":\\\"Table.Column1\\\",\\\"active\\\":true}],\\\"Y\\\":[{\\\"queryRef\\\":\\\"CountNonNull(Table.Column2)\\\"}],\\\"Series\\\":[{\\\"queryRef\\\":\\\"Table_2.Column1\\\"}]},\\\"prototypeQuery\\\":{\\\"Version\\\":2,\\\"From\\\":[{\\\"Name\\\":\\\"t\\\",\\\"Entity\\\":\\\"Table\\\",\\\"Type\\\":0},{\\\"Name\\\":\\\"t1\\\",\\\"Entity\\\":\\\"Table_2\\\",\\\"Type\\\":0}],\\\"Select\\\":[{\\\"Column\\\":{\\\"Expression\\\":{\\\"SourceRef\\\":{\\\"Source\\\":\\\"t\\\"}},\\\"Property\\\":\\\"Column1\\\"},\\\"Name\\\":\\\"Table.Column1\\\",\\\"NativeReferenceName\\\":\\\"Column1\\\"},{\\\"Aggregation\\\":{\\\"Expression\\\":{\\\"Column\\\":{\\\"Expression\\\":{\\\"SourceRef\\\":{\\\"Source\\\":\\\"t\\\"}},\\\"Property\\\":\\\"Column2\\\"}},\\\"Function\\\":5},\\\"Name\\\":\\\"CountNonNull(Table.Column2)\\\",\\\"NativeReferenceName\\\":\\\"Column2\\\"},{\\\"Column\\\":{\\\"Expression\\\":{\\\"SourceRef\\\":{\\\"Source\\\":\\\"t1\\\"}},\\\"Property\\\":\\\"Column1\\\"},\\\"Name\\\":\\\"Table_2.Column1\\\",\\\"NativeReferenceName\\\":\\\"Column11\\\"}],\\\"OrderBy\\\":[{\\\"Direction\\\":2,\\\"Expression\\\":{\\\"Aggregation\\\":{\\\"Expression\\\":{\\\"Column\\\":{\\\"Expression\\\":{\\\"SourceRef\\\":{\\\"Source\\\":\\\"t\\\"}},\\\"Property\\\":\\\"Column2\\\"}},\\\"Function\\\":5}}}]},\\\"drillFilterOtherVisuals\\\":true,\\\"hasDefaultSort\\\":true}}\",\n \"filters\": \"[]\",\n \"height\": 279.55,\n \"width\": 279.55,\n \"x\": 317.44,\n \"y\": 69.63,\n \"z\": 0.00\n }\n ],\n \"width\": 1280.00\n }\n ]\n}" }, { "path": "sample/workspace/ABC.SemanticModel/.platform", "content": "{\n \"$schema\": \"https://developer.microsoft.com/json-schemas/fabric/gitIntegration/platformProperties/2.0.0/schema.json\",\n \"metadata\": {\n \"type\": \"SemanticModel\",\n \"displayName\": \"ABC\"\n },\n \"config\": {\n \"version\": \"2.0\",\n \"logicalId\": \"5ead2a08-6831-80df-4110-6265e29e3ee7\"\n }\n}" }, { "path": "sample/workspace/ABC.SemanticModel/definition/cultures/en-US.tmdl", "content": "cultureInfo en-US\n\n\tlinguisticMetadata =\n\t\t\t{\n\t\t\t \"Version\": \"1.0.0\",\n\t\t\t \"Language\": \"en-US\"\n\t\t\t}\n\t\tcontentType: json\n\n" }, { "path": "sample/workspace/ABC.SemanticModel/definition/database.tmdl", "content": "database\n\tcompatibilityLevel: 1550\n\n" }, { "path": "sample/workspace/ABC.SemanticModel/definition/model.tmdl", "content": "model Model\n\tculture: en-US\n\tdefaultPowerBIDataSourceVersion: powerBI_V3\n\tsourceQueryCulture: en-US\n\tdataAccessOptions\n\t\tlegacyRedirects\n\t\treturnErrorValuesAsNull\n\nannotation PBI_QueryOrder = [\"Table\",\"Table_2\"]\n\nannotation __PBI_TimeIntelligenceEnabled = 1\n\nannotation PBIDesktopVersion = 2.140.679.0 (25.02)+e2f7796f7ddf473b3f87e4d9e2bee0b29f9956bf\n\nannotation PBI_ProTooling = [\"DevMode\"]\n\nref table Table\nref table Table_2\n\nref cultureInfo en-US\n\n" }, { "path": "sample/workspace/ABC.SemanticModel/definition/relationships.tmdl", "content": "relationship AutoDetected_981379f6-d1c0-48e8-a4d4-cf3718d1fc57\n\tcrossFilteringBehavior: bothDirections\n\tfromCardinality: one\n\tfromColumn: Table.Column2\n\ttoColumn: Table_2.Column1\n\n" }, { "path": "sample/workspace/ABC.SemanticModel/definition/tables/Table.tmdl", "content": "table Table\n\tlineageTag: e9a135a4-d745-4125-9c08-01f5a8ee959e\n\n\tcolumn Column1\n\t\tdataType: string\n\t\tlineageTag: 2b0b4597-fb3d-432e-9047-671b8e2a3970\n\t\tsummarizeBy: none\n\t\tsourceColumn: Column1\n\n\t\tannotation SummarizationSetBy = Automatic\n\n\tcolumn Column2\n\t\tdataType: int64\n\t\tformatString: 0\n\t\tlineageTag: def47816-d442-4e60-aed3-6b05c2fb0265\n\t\tsummarizeBy: none\n\t\tsourceColumn: Column2\n\n\t\tannotation SummarizationSetBy = Automatic\n\n\tpartition Table = m\n\t\tmode: import\n\t\tsource =\n\t\t\t\tlet\n\t\t\t\t Source = Table.FromRows(Json.Document(Binary.Decompress(Binary.FromText(\"i45WclTSUTJUitWJVnICsozALGcgy1gpNhYA\", BinaryEncoding.Base64), Compression.Deflate)), let _t = ((type nullable text) meta [Serialized.Text = true]) in type table [Column1 = _t, Column2 = _t]),\n\t\t\t\t #\"Changed Type\" = Table.TransformColumnTypes(Source,{{\"Column1\", type text}, {\"Column2\", Int64.Type}})\n\t\t\t\tin\n\t\t\t\t #\"Changed Type\"\n\n\tannotation PBI_NavigationStepName = Navigation\n\n\tannotation PBI_ResultType = Table\n\n" }, { "path": "sample/workspace/ABC.SemanticModel/definition/tables/Table_2.tmdl", "content": "table Table_2\n\tlineageTag: 2f801d69-7502-4625-aba2-356b8f4378b2\n\n\tcolumn Column1\n\t\tdataType: int64\n\t\tformatString: 0\n\t\tlineageTag: adbc29fb-d481-4ebf-96b8-ed12b1d1d7e1\n\t\tsummarizeBy: none\n\t\tsourceColumn: Column1\n\n\t\tannotation SummarizationSetBy = Automatic\n\n\tpartition Table_2 = m\n\t\tmode: import\n\t\tsource =\n\t\t\t\tlet\n\t\t\t\t Source = Table.FromRows(Json.Document(Binary.Decompress(Binary.FromText(\"i45WMlSK1YlWMgKTxmDSBEyaKsXGAgA=\", BinaryEncoding.Base64), Compression.Deflate)), let _t = ((type nullable text) meta [Serialized.Text = true]) in type table [Column1 = _t]),\n\t\t\t\t #\"Changed Type\" = Table.TransformColumnTypes(Source,{{\"Column1\", Int64.Type}})\n\t\t\t\tin\n\t\t\t\t #\"Changed Type\"\n\n\tannotation PBI_NavigationStepName = Navigation\n\n\tannotation PBI_ResultType = Table\n\n" }, { "path": "sample/workspace/ABC.SemanticModel/definition.pbism", "content": "{\n \"version\": \"4.0\",\n \"settings\": {}\n}" }, { "path": "sample/workspace/ABC.SemanticModel/diagramLayout.json", "content": "{\n \"version\": \"1.1.0\",\n \"diagrams\": [\n {\n \"ordinal\": 0,\n \"scrollPosition\": {\n \"x\": 0,\n \"y\": 0\n },\n \"nodes\": [\n {\n \"location\": {\n \"x\": 294,\n \"y\": -10\n },\n \"nodeIndex\": \"Table\",\n \"nodeLineageTag\": \"e9a135a4-d745-4125-9c08-01f5a8ee959e\",\n \"size\": {\n \"height\": 128,\n \"width\": 234\n },\n \"zIndex\": 0\n },\n {\n \"location\": {\n \"x\": 10,\n \"y\": 2\n },\n \"nodeIndex\": \"Table_2\",\n \"nodeLineageTag\": \"2f801d69-7502-4625-aba2-356b8f4378b2\",\n \"size\": {\n \"height\": 104,\n \"width\": 234\n },\n \"zIndex\": 0\n }\n ],\n \"name\": \"All tables\",\n \"zoomValue\": 100,\n \"pinKeyFieldsToTop\": false,\n \"showExtraHeaderInfo\": false,\n \"hideKeyFieldsWhenCollapsed\": false,\n \"tablesLocked\": false\n }\n ],\n \"selectedDiagram\": \"All tables\",\n \"defaultDiagram\": \"All tables\"\n}" }, { "path": "sample/workspace/ABCD.Report/.platform", "content": "{\n \"$schema\": \"https://developer.microsoft.com/json-schemas/fabric/gitIntegration/platformProperties/2.0.0/schema.json\",\n \"metadata\": {\n \"type\": \"Report\",\n \"displayName\": \"ABCD\"\n },\n \"config\": {\n \"version\": \"2.0\",\n \"logicalId\": \"40db346f-1c24-8aa6-48c7-a3aec4199350\"\n }\n}\n" }, { "path": "sample/workspace/ABCD.Report/StaticResources/SharedResources/BaseThemes/CY24SU10.json", "content": "{\n \"name\": \"CY24SU10\",\n \"dataColors\": [\n \"#118DFF\",\n \"#12239E\",\n \"#E66C37\",\n \"#6B007B\",\n \"#E044A7\",\n \"#744EC2\",\n \"#D9B300\",\n \"#D64550\",\n \"#197278\",\n \"#1AAB40\",\n \"#15C6F4\",\n \"#4092FF\",\n \"#FFA058\",\n \"#BE5DC9\",\n \"#F472D0\",\n \"#B5A1FF\",\n \"#C4A200\",\n \"#FF8080\",\n \"#00DBBC\",\n \"#5BD667\",\n \"#0091D5\",\n \"#4668C5\",\n \"#FF6300\",\n \"#99008A\",\n \"#EC008C\",\n \"#533285\",\n \"#99700A\",\n \"#FF4141\",\n \"#1F9A85\",\n \"#25891C\",\n \"#0057A2\",\n \"#002050\",\n \"#C94F0F\",\n \"#450F54\",\n \"#B60064\",\n \"#34124F\",\n \"#6A5A29\",\n \"#1AAB40\",\n \"#BA141A\",\n \"#0C3D37\",\n \"#0B511F\"\n ],\n \"foreground\": \"#252423\",\n \"foregroundNeutralSecondary\": \"#605E5C\",\n \"foregroundNeutralTertiary\": \"#B3B0AD\",\n \"background\": \"#FFFFFF\",\n \"backgroundLight\": \"#F3F2F1\",\n \"backgroundNeutral\": \"#C8C6C4\",\n \"tableAccent\": \"#118DFF\",\n \"good\": \"#1AAB40\",\n \"neutral\": \"#D9B300\",\n \"bad\": \"#D64554\",\n \"maximum\": \"#118DFF\",\n \"center\": \"#D9B300\",\n \"minimum\": \"#DEEFFF\",\n \"null\": \"#FF7F48\",\n \"hyperlink\": \"#0078d4\",\n \"visitedHyperlink\": \"#0078d4\",\n \"textClasses\": {\n \"callout\": {\n \"fontSize\": 45,\n \"fontFace\": \"DIN\",\n \"color\": \"#252423\"\n },\n \"title\": {\n \"fontSize\": 12,\n \"fontFace\": \"DIN\",\n \"color\": \"#252423\"\n },\n \"header\": {\n \"fontSize\": 12,\n \"fontFace\": \"Segoe UI Semibold\",\n \"color\": \"#252423\"\n },\n \"label\": {\n \"fontSize\": 10,\n \"fontFace\": \"Segoe UI\",\n \"color\": \"#252423\"\n }\n },\n \"visualStyles\": {\n \"*\": {\n \"*\": {\n \"*\": [\n {\n \"wordWrap\": true\n }\n ],\n \"line\": [\n {\n \"transparency\": 0\n }\n ],\n \"outline\": [\n {\n \"transparency\": 0\n }\n ],\n \"plotArea\": [\n {\n \"transparency\": 0\n }\n ],\n \"categoryAxis\": [\n {\n \"showAxisTitle\": true,\n \"gridlineStyle\": \"dotted\",\n \"concatenateLabels\": false\n }\n ],\n \"valueAxis\": [\n {\n \"showAxisTitle\": true,\n \"gridlineStyle\": \"dotted\"\n }\n ],\n \"y2Axis\": [\n {\n \"show\": true\n }\n ],\n \"title\": [\n {\n \"titleWrap\": true\n }\n ],\n \"lineStyles\": [\n {\n \"strokeWidth\": 3\n }\n ],\n \"wordWrap\": [\n {\n \"show\": true\n }\n ],\n \"background\": [\n {\n \"show\": true,\n \"transparency\": 0\n }\n ],\n \"border\": [\n {\n \"width\": 1\n }\n ],\n \"outspacePane\": [\n {\n \"backgroundColor\": {\n \"solid\": {\n \"color\": \"#ffffff\"\n }\n },\n \"transparency\": 0,\n \"border\": true,\n \"borderColor\": {\n \"solid\": {\n \"color\": \"#B3B0AD\"\n }\n }\n }\n ],\n \"filterCard\": [\n {\n \"$id\": \"Applied\",\n \"transparency\": 0,\n \"foregroundColor\": {\n \"solid\": {\n \"color\": \"#252423\"\n }\n },\n \"border\": true\n },\n {\n \"$id\": \"Available\",\n \"transparency\": 0,\n \"foregroundColor\": {\n \"solid\": {\n \"color\": \"#252423\"\n }\n },\n \"border\": true\n }\n ]\n }\n },\n \"scatterChart\": {\n \"*\": {\n \"bubbles\": [\n {\n \"bubbleSize\": -10,\n \"markerRangeType\": \"auto\"\n }\n ],\n \"general\": [\n {\n \"responsive\": true\n }\n ],\n \"fillPoint\": [\n {\n \"show\": true\n }\n ],\n \"legend\": [\n {\n \"showGradientLegend\": true\n }\n ]\n }\n },\n \"lineChart\": {\n \"*\": {\n \"general\": [\n {\n \"responsive\": true\n }\n ],\n \"smallMultiplesLayout\": [\n {\n \"backgroundTransparency\": 0,\n \"gridLineType\": \"inner\"\n }\n ],\n \"forecast\": [\n {\n \"matchSeriesInterpolation\": true\n }\n ]\n }\n },\n \"map\": {\n \"*\": {\n \"bubbles\": [\n {\n \"bubbleSize\": -10,\n \"markerRangeType\": \"auto\"\n }\n ]\n }\n },\n \"azureMap\": {\n \"*\": {\n \"bubbleLayer\": [\n {\n \"bubbleRadius\": 8,\n \"minBubbleRadius\": 8,\n \"maxRadius\": 40\n }\n ],\n \"barChart\": [\n {\n \"barHeight\": 3,\n \"thickness\": 3\n }\n ]\n }\n },\n \"pieChart\": {\n \"*\": {\n \"legend\": [\n {\n \"show\": true,\n \"position\": \"RightCenter\"\n }\n ],\n \"labels\": [\n {\n \"labelStyle\": \"Data value, percent of total\"\n }\n ]\n }\n },\n \"donutChart\": {\n \"*\": {\n \"legend\": [\n {\n \"show\": true,\n \"position\": \"RightCenter\"\n }\n ],\n \"labels\": [\n {\n \"labelStyle\": \"Data value, percent of total\"\n }\n ]\n }\n },\n \"pivotTable\": {\n \"*\": {\n \"rowHeaders\": [\n {\n \"showExpandCollapseButtons\": true,\n \"legacyStyleDisabled\": true\n }\n ]\n }\n },\n \"multiRowCard\": {\n \"*\": {\n \"card\": [\n {\n \"outlineWeight\": 2,\n \"barShow\": true,\n \"barWeight\": 2\n }\n ]\n }\n },\n \"kpi\": {\n \"*\": {\n \"trendline\": [\n {\n \"transparency\": 20\n }\n ]\n }\n },\n \"cardVisual\": {\n \"*\": {\n \"layout\": [\n {\n \"maxTiles\": 3\n }\n ],\n \"overflow\": [\n {\n \"type\": 0\n }\n ],\n \"image\": [\n {\n \"fixedSize\": false\n },\n {\n \"imageAreaSize\": 50\n }\n ]\n }\n },\n \"advancedSlicerVisual\": {\n \"*\": {\n \"layout\": [\n {\n \"maxTiles\": 3\n }\n ]\n }\n },\n \"slicer\": {\n \"*\": {\n \"general\": [\n {\n \"responsive\": true\n }\n ],\n \"date\": [\n {\n \"hideDatePickerButton\": false\n }\n ],\n \"items\": [\n {\n \"padding\": 4,\n \"accessibilityContrastProperties\": true\n }\n ]\n }\n },\n \"waterfallChart\": {\n \"*\": {\n \"general\": [\n {\n \"responsive\": true\n }\n ]\n }\n },\n \"columnChart\": {\n \"*\": {\n \"general\": [\n {\n \"responsive\": true\n }\n ],\n \"legend\": [\n {\n \"showGradientLegend\": true\n }\n ],\n \"smallMultiplesLayout\": [\n {\n \"backgroundTransparency\": 0,\n \"gridLineType\": \"inner\"\n }\n ]\n }\n },\n \"clusteredColumnChart\": {\n \"*\": {\n \"general\": [\n {\n \"responsive\": true\n }\n ],\n \"legend\": [\n {\n \"showGradientLegend\": true\n }\n ],\n \"smallMultiplesLayout\": [\n {\n \"backgroundTransparency\": 0,\n \"gridLineType\": \"inner\"\n }\n ]\n }\n },\n \"hundredPercentStackedColumnChart\": {\n \"*\": {\n \"general\": [\n {\n \"responsive\": true\n }\n ],\n \"legend\": [\n {\n \"showGradientLegend\": true\n }\n ],\n \"smallMultiplesLayout\": [\n {\n \"backgroundTransparency\": 0,\n \"gridLineType\": \"inner\"\n }\n ]\n }\n },\n \"barChart\": {\n \"*\": {\n \"general\": [\n {\n \"responsive\": true\n }\n ],\n \"legend\": [\n {\n \"showGradientLegend\": true\n }\n ],\n \"smallMultiplesLayout\": [\n {\n \"backgroundTransparency\": 0,\n \"gridLineType\": \"inner\"\n }\n ]\n }\n },\n \"clusteredBarChart\": {\n \"*\": {\n \"general\": [\n {\n \"responsive\": true\n }\n ],\n \"legend\": [\n {\n \"showGradientLegend\": true\n }\n ],\n \"smallMultiplesLayout\": [\n {\n \"backgroundTransparency\": 0,\n \"gridLineType\": \"inner\"\n }\n ]\n }\n },\n \"hundredPercentStackedBarChart\": {\n \"*\": {\n \"general\": [\n {\n \"responsive\": true\n }\n ],\n \"legend\": [\n {\n \"showGradientLegend\": true\n }\n ],\n \"smallMultiplesLayout\": [\n {\n \"backgroundTransparency\": 0,\n \"gridLineType\": \"inner\"\n }\n ]\n }\n },\n \"areaChart\": {\n \"*\": {\n \"general\": [\n {\n \"responsive\": true\n }\n ],\n \"smallMultiplesLayout\": [\n {\n \"backgroundTransparency\": 0,\n \"gridLineType\": \"inner\"\n }\n ]\n }\n },\n \"stackedAreaChart\": {\n \"*\": {\n \"general\": [\n {\n \"responsive\": true\n }\n ],\n \"smallMultiplesLayout\": [\n {\n \"backgroundTransparency\": 0,\n \"gridLineType\": \"inner\"\n }\n ]\n }\n },\n \"lineClusteredColumnComboChart\": {\n \"*\": {\n \"general\": [\n {\n \"responsive\": true\n }\n ],\n \"smallMultiplesLayout\": [\n {\n \"backgroundTransparency\": 0,\n \"gridLineType\": \"inner\"\n }\n ]\n }\n },\n \"lineStackedColumnComboChart\": {\n \"*\": {\n \"general\": [\n {\n \"responsive\": true\n }\n ],\n \"smallMultiplesLayout\": [\n {\n \"backgroundTransparency\": 0,\n \"gridLineType\": \"inner\"\n }\n ]\n }\n },\n \"ribbonChart\": {\n \"*\": {\n \"general\": [\n {\n \"responsive\": true\n }\n ],\n \"smallMultiplesLayout\": [\n {\n \"backgroundTransparency\": 0,\n \"gridLineType\": \"inner\"\n }\n ],\n \"valueAxis\": [\n {\n \"show\": true\n }\n ]\n }\n },\n \"hundredPercentStackedAreaChart\": {\n \"*\": {\n \"general\": [\n {\n \"responsive\": true\n }\n ],\n \"smallMultiplesLayout\": [\n {\n \"backgroundTransparency\": 0,\n \"gridLineType\": \"inner\"\n }\n ]\n }\n },\n \"group\": {\n \"*\": {\n \"background\": [\n {\n \"show\": false\n }\n ]\n }\n },\n \"basicShape\": {\n \"*\": {\n \"background\": [\n {\n \"show\": false\n }\n ],\n \"general\": [\n {\n \"keepLayerOrder\": true\n }\n ],\n \"visualHeader\": [\n {\n \"show\": false\n }\n ]\n }\n },\n \"shape\": {\n \"*\": {\n \"background\": [\n {\n \"show\": false\n }\n ],\n \"general\": [\n {\n \"keepLayerOrder\": true\n }\n ],\n \"visualHeader\": [\n {\n \"show\": false\n }\n ]\n }\n },\n \"image\": {\n \"*\": {\n \"background\": [\n {\n \"show\": false\n }\n ],\n \"general\": [\n {\n \"keepLayerOrder\": true\n }\n ],\n \"visualHeader\": [\n {\n \"show\": false\n }\n ],\n \"lockAspect\": [\n {\n \"show\": true\n }\n ]\n }\n },\n \"actionButton\": {\n \"*\": {\n \"background\": [\n {\n \"show\": false\n }\n ],\n \"visualHeader\": [\n {\n \"show\": false\n }\n ]\n }\n },\n \"pageNavigator\": {\n \"*\": {\n \"background\": [\n {\n \"show\": false\n }\n ],\n \"visualHeader\": [\n {\n \"show\": false\n }\n ]\n }\n },\n \"bookmarkNavigator\": {\n \"*\": {\n \"background\": [\n {\n \"show\": false\n }\n ],\n \"visualHeader\": [\n {\n \"show\": false\n }\n ]\n }\n },\n \"textbox\": {\n \"*\": {\n \"general\": [\n {\n \"keepLayerOrder\": true\n }\n ],\n \"visualHeader\": [\n {\n \"show\": false\n }\n ]\n }\n },\n \"page\": {\n \"*\": {\n \"outspace\": [\n {\n \"color\": {\n \"solid\": {\n \"color\": \"#FFFFFF\"\n }\n }\n }\n ],\n \"background\": [\n {\n \"transparency\": 100\n }\n ]\n }\n }\n }\n}" }, { "path": "sample/workspace/ABCD.Report/definition.pbir", "content": "{\n \"version\": \"4.0\",\n \"datasetReference\": {\n \"byPath\": {\n \"path\": \"../ABC.SemanticModel\"\n }\n }\n}" }, { "path": "sample/workspace/ABCD.Report/report.json", "content": "{\n \"config\": \"{\\\"version\\\":\\\"5.59\\\",\\\"themeCollection\\\":{\\\"baseTheme\\\":{\\\"name\\\":\\\"CY24SU10\\\",\\\"version\\\":\\\"5.61\\\",\\\"type\\\":2}},\\\"activeSectionIndex\\\":0,\\\"defaultDrillFilterOtherVisuals\\\":true,\\\"linguisticSchemaSyncVersion\\\":2,\\\"settings\\\":{\\\"useNewFilterPaneExperience\\\":true,\\\"allowChangeFilterTypes\\\":true,\\\"useStylableVisualContainerHeader\\\":true,\\\"queryLimitOption\\\":6,\\\"useEnhancedTooltips\\\":true,\\\"exportDataMode\\\":1,\\\"useDefaultAggregateDisplayName\\\":true},\\\"objects\\\":{\\\"section\\\":[{\\\"properties\\\":{\\\"verticalAlignment\\\":{\\\"expr\\\":{\\\"Literal\\\":{\\\"Value\\\":\\\"'Top'\\\"}}}}}]}}\",\n \"layoutOptimization\": 0,\n \"resourcePackages\": [\n {\n \"resourcePackage\": {\n \"disabled\": false,\n \"items\": [\n {\n \"name\": \"CY24SU10\",\n \"path\": \"BaseThemes/CY24SU10.json\",\n \"type\": 202\n }\n ],\n \"name\": \"SharedResources\",\n \"type\": 2\n }\n }\n ],\n \"sections\": [\n {\n \"config\": \"{}\",\n \"displayName\": \"Page 3\",\n \"displayOption\": 1,\n \"filters\": \"[]\",\n \"height\": 720.00,\n \"name\": \"63e290aa19e7bea3d03d\",\n \"ordinal\": 2,\n \"visualContainers\": [\n {\n \"config\": \"{\\\"name\\\":\\\"39eb354ddb07467c1948\\\",\\\"layouts\\\":[{\\\"id\\\":0,\\\"position\\\":{\\\"x\\\":525.312,\\\"y\\\":169.984,\\\"z\\\":0,\\\"width\\\":279.552,\\\"height\\\":279.552,\\\"tabOrder\\\":0}}],\\\"singleVisual\\\":{\\\"visualType\\\":\\\"hundredPercentStackedBarChart\\\",\\\"projections\\\":{\\\"Category\\\":[{\\\"queryRef\\\":\\\"Table.Column1\\\",\\\"active\\\":true}]},\\\"prototypeQuery\\\":{\\\"Version\\\":2,\\\"From\\\":[{\\\"Name\\\":\\\"t\\\",\\\"Entity\\\":\\\"Table\\\",\\\"Type\\\":0}],\\\"Select\\\":[{\\\"Column\\\":{\\\"Expression\\\":{\\\"SourceRef\\\":{\\\"Source\\\":\\\"t\\\"}},\\\"Property\\\":\\\"Column1\\\"},\\\"Name\\\":\\\"Table.Column1\\\",\\\"NativeReferenceName\\\":\\\"Column1\\\"}]},\\\"drillFilterOtherVisuals\\\":true}}\",\n \"filters\": \"[]\",\n \"height\": 279.55,\n \"width\": 279.55,\n \"x\": 525.31,\n \"y\": 169.98,\n \"z\": 0.00\n }\n ],\n \"width\": 1280.00\n },\n {\n \"config\": \"{}\",\n \"displayName\": \"Page 1\",\n \"displayOption\": 1,\n \"filters\": \"[]\",\n \"height\": 720.00,\n \"name\": \"90cb47b35aee21c5a77c\",\n \"visualContainers\": [\n {\n \"config\": \"{\\\"name\\\":\\\"d15aa7ce4fc6c2acb4f2\\\",\\\"layouts\\\":[{\\\"id\\\":0,\\\"position\\\":{\\\"x\\\":342,\\\"y\\\":104,\\\"z\\\":0,\\\"width\\\":280,\\\"height\\\":280,\\\"tabOrder\\\":0}}],\\\"singleVisual\\\":{\\\"visualType\\\":\\\"tableEx\\\",\\\"projections\\\":{\\\"Values\\\":[{\\\"queryRef\\\":\\\"Table.Column1\\\"},{\\\"queryRef\\\":\\\"Table.Column2\\\"}]},\\\"prototypeQuery\\\":{\\\"Version\\\":2,\\\"From\\\":[{\\\"Name\\\":\\\"t\\\",\\\"Entity\\\":\\\"Table\\\",\\\"Type\\\":0}],\\\"Select\\\":[{\\\"Column\\\":{\\\"Expression\\\":{\\\"SourceRef\\\":{\\\"Source\\\":\\\"t\\\"}},\\\"Property\\\":\\\"Column1\\\"},\\\"Name\\\":\\\"Table.Column1\\\",\\\"NativeReferenceName\\\":\\\"Column1\\\"},{\\\"Column\\\":{\\\"Expression\\\":{\\\"SourceRef\\\":{\\\"Source\\\":\\\"t\\\"}},\\\"Property\\\":\\\"Column2\\\"},\\\"Name\\\":\\\"Table.Column2\\\",\\\"NativeReferenceName\\\":\\\"Column2\\\"}]},\\\"drillFilterOtherVisuals\\\":true}}\",\n \"filters\": \"[]\",\n \"height\": 280.00,\n \"width\": 280.00,\n \"x\": 342.00,\n \"y\": 104.00,\n \"z\": 0.00\n }\n ],\n \"width\": 1280.00\n },\n {\n \"config\": \"{}\",\n \"displayName\": \"Page 2\",\n \"displayOption\": 1,\n \"filters\": \"[]\",\n \"height\": 720.00,\n \"name\": \"f072cad6a715c38915c7\",\n \"ordinal\": 1,\n \"visualContainers\": [\n {\n \"config\": \"{\\\"name\\\":\\\"1e78447fe04d60e1956f\\\",\\\"layouts\\\":[{\\\"id\\\":0,\\\"position\\\":{\\\"x\\\":317.44,\\\"y\\\":69.632,\\\"z\\\":0,\\\"width\\\":279.552,\\\"height\\\":279.552,\\\"tabOrder\\\":0}}],\\\"singleVisual\\\":{\\\"visualType\\\":\\\"areaChart\\\",\\\"projections\\\":{\\\"Category\\\":[{\\\"queryRef\\\":\\\"Table.Column1\\\",\\\"active\\\":true}],\\\"Y\\\":[{\\\"queryRef\\\":\\\"CountNonNull(Table.Column2)\\\"}],\\\"Series\\\":[{\\\"queryRef\\\":\\\"Table_2.Column1\\\"}]},\\\"prototypeQuery\\\":{\\\"Version\\\":2,\\\"From\\\":[{\\\"Name\\\":\\\"t\\\",\\\"Entity\\\":\\\"Table\\\",\\\"Type\\\":0},{\\\"Name\\\":\\\"t1\\\",\\\"Entity\\\":\\\"Table_2\\\",\\\"Type\\\":0}],\\\"Select\\\":[{\\\"Column\\\":{\\\"Expression\\\":{\\\"SourceRef\\\":{\\\"Source\\\":\\\"t\\\"}},\\\"Property\\\":\\\"Column1\\\"},\\\"Name\\\":\\\"Table.Column1\\\",\\\"NativeReferenceName\\\":\\\"Column1\\\"},{\\\"Aggregation\\\":{\\\"Expression\\\":{\\\"Column\\\":{\\\"Expression\\\":{\\\"SourceRef\\\":{\\\"Source\\\":\\\"t\\\"}},\\\"Property\\\":\\\"Column2\\\"}},\\\"Function\\\":5},\\\"Name\\\":\\\"CountNonNull(Table.Column2)\\\",\\\"NativeReferenceName\\\":\\\"Column2\\\"},{\\\"Column\\\":{\\\"Expression\\\":{\\\"SourceRef\\\":{\\\"Source\\\":\\\"t1\\\"}},\\\"Property\\\":\\\"Column1\\\"},\\\"Name\\\":\\\"Table_2.Column1\\\",\\\"NativeReferenceName\\\":\\\"Column11\\\"}],\\\"OrderBy\\\":[{\\\"Direction\\\":2,\\\"Expression\\\":{\\\"Aggregation\\\":{\\\"Expression\\\":{\\\"Column\\\":{\\\"Expression\\\":{\\\"SourceRef\\\":{\\\"Source\\\":\\\"t\\\"}},\\\"Property\\\":\\\"Column2\\\"}},\\\"Function\\\":5}}}]},\\\"drillFilterOtherVisuals\\\":true,\\\"hasDefaultSort\\\":true}}\",\n \"filters\": \"[]\",\n \"height\": 279.55,\n \"width\": 279.55,\n \"x\": 317.44,\n \"y\": 69.63,\n \"z\": 0.00\n }\n ],\n \"width\": 1280.00\n }\n ]\n}" }, { "path": "sample/workspace/ByConnection.Report/.platform", "content": "{\n \"$schema\": \"https://developer.microsoft.com/json-schemas/fabric/gitIntegration/platformProperties/2.0.0/schema.json\",\n \"metadata\": {\n \"type\": \"Report\",\n \"displayName\": \"ByConnection\"\n },\n \"config\": {\n \"version\": \"2.0\",\n \"logicalId\": \"12345678-1234-1234-1234-123456789abc\"\n }\n}\n" }, { "path": "sample/workspace/ByConnection.Report/definition.pbir", "content": "{\n \"$schema\": \"https://developer.microsoft.com/json-schemas/fabric/item/report/definitionProperties/2.0.0/schema.json\",\n \"version\": \"4.0\",\n \"datasetReference\": {\n \"byConnection\": {\n \"connectionString\": \"Data Source=powerbi://api.powerbi.com/v1.0/myorg/dev-workspace-id;initial catalog=dev-semantic-model;access mode=readonly;integrated security=ClaimsToken;semanticmodelid=00000000-0000-0000-0000-000000000000\"\n }\n }\n}\n" }, { "path": "sample/workspace/ByConnection.Report/report.json", "content": "{\n \"config\": \"{\\\"version\\\":\\\"5.59\\\",\\\"themeCollection\\\":{\\\"baseTheme\\\":{\\\"name\\\":\\\"CY24SU10\\\",\\\"version\\\":\\\"5.61\\\",\\\"type\\\":2}},\\\"activeSectionIndex\\\":0,\\\"defaultDrillFilterOtherVisuals\\\":true,\\\"linguisticSchemaSyncVersion\\\":2,\\\"settings\\\":{\\\"useNewFilterPaneExperience\\\":true,\\\"allowChangeFilterTypes\\\":true,\\\"useStylableVisualContainerHeader\\\":true,\\\"queryLimitOption\\\":6,\\\"useEnhancedTooltips\\\":true,\\\"exportDataMode\\\":1,\\\"useDefaultAggregateDisplayName\\\":true}}\",\n \"layoutOptimization\": 0,\n \"resourcePackages\": [\n {\n \"resourcePackage\": {\n \"disabled\": false,\n \"items\": [\n {\n \"name\": \"CY24SU10\",\n \"path\": \"BaseThemes/CY24SU10.json\",\n \"type\": 202\n }\n ],\n \"name\": \"SharedResources\",\n \"type\": 2\n }\n }\n ],\n \"sections\": [\n {\n \"config\": \"{}\",\n \"displayName\": \"Page 1\",\n \"displayOption\": 1,\n \"filters\": \"[]\",\n \"height\": 720.00,\n \"name\": \"ReportSection\",\n \"visualContainers\": [],\n \"width\": 1280.00\n }\n ]\n}\n" }, { "path": "sample/workspace/Default.Warehouse/.platform", "content": "{\n \"$schema\": \"https://developer.microsoft.com/json-schemas/fabric/gitIntegration/platformProperties/2.0.0/schema.json\",\n \"metadata\": {\n \"type\": \"Warehouse\",\n \"displayName\": \"Default\"\n },\n \"config\": {\n \"version\": \"2.0\",\n \"logicalId\": \"4f344f2d-b51a-a14e-4706-c94957a91175\"\n }\n}\n" }, { "path": "sample/workspace/DefaultCaseInsensitive.Warehouse/.platform", "content": "{\n \"$schema\": \"https://developer.microsoft.com/json-schemas/fabric/gitIntegration/platformProperties/2.0.0/schema.json\",\n \"metadata\": {\n \"type\": \"Warehouse\",\n \"displayName\": \"DefaultCaseInsensitive\",\n \"creationPayload\": {\n \"defaultCollation\": \"Latin1_General_100_CI_AS_KS_WS_SC_UTF8\"\n }\n },\n \"config\": {\n \"version\": \"2.0\",\n \"logicalId\": \"0e63f8ad-3b03-4f91-8d5e-b33750b3beec\"\n }\n}\n" }, { "path": "sample/workspace/Example Notebook.Notebook/.platform", "content": "{\n \"$schema\": \"https://developer.microsoft.com/json-schemas/fabric/gitIntegration/platformProperties/2.0.0/schema.json\",\n \"metadata\": {\n \"type\": \"Notebook\",\n \"displayName\": \"Example Notebook\",\n \"description\": \"New notebook\"\n },\n \"config\": {\n \"version\": \"2.0\",\n \"logicalId\": \"21d371bb-4ca3-bf25-46cf-c634f603cbf1\"\n }\n}\n" }, { "path": "sample/workspace/Example Notebook.Notebook/notebook-content.py", "content": "# Fabric notebook source\n\n# METADATA ********************\n\n# META {\n# META \"kernel_info\": {\n# META \"name\": \"synapse_pyspark\"\n# META },\n# META \"dependencies\": {\n# META \"lakehouse\": {\n# META \"default_lakehouse\": \"f3b9c1e2-7d4a-4b3e-9f2a-1c6e8d9a7b3c\",\n# META \"default_lakehouse_name\": \"WithoutSchema\",\n# META \"default_lakehouse_workspace_id\": \"c7e4b9f1-2a3d-4e6f-9c8b-7d2f1a4e5b6d\",\n# META \"known_lakehouses\": [\n# META {\n# META \"id\": \"f3b9c1e2-7d4a-4b3e-9f2a-1c6e8d9a7b3c\"\n# META }\n# META ]\n# META }\n# META }\n# META }\n\n# CELL ********************\n\n# Welcome to your new notebook\n# Type here in the cell editor to add code!\nprint(\"This notebook is attached to the 'WithoutSchema' lakehouse.\")\nprint(\"This its SQL analytics endpoint: sqlserverconnectionstringinoriginlakehouse.com\")\nprint(\"This workspace includes 'SampleEventhouse' eventhouse.\")\nprint(\"This its eventhouse query URI: https://trd-origineventhouse.z4.kusto.fabric.microsoft.com\")\n\n# METADATA ********************\n\n# META {\n# META \"language\": \"python\",\n# META \"language_group\": \"synapse_pyspark\"\n# META }\n" }, { "path": "sample/workspace/Hello Copy Job.CopyJob/.platform", "content": "{\n \"$schema\": \"https://developer.microsoft.com/json-schemas/fabric/gitIntegration/platformProperties/2.0.0/schema.json\",\n \"metadata\": {\n \"type\": \"CopyJob\",\n \"displayName\": \"Hello Copy Job\"\n },\n \"config\": {\n \"version\": \"2.0\",\n \"logicalId\": \"b0eb37b3-73d6-bae9-40a6-d47ae162a23b\"\n }\n}" }, { "path": "sample/workspace/Hello Copy Job.CopyJob/copyjob-content.json", "content": "{\n \"properties\": {\n \"jobMode\": \"Batch\",\n \"source\": {\n \"type\": \"LakehouseTable\",\n \"connectionSettings\": {\n \"type\": \"Lakehouse\",\n \"typeProperties\": {\n \"workspaceId\": \"e96609ad-cc50-4c63-8829-c8499910e044\",\n \"artifactId\": \"0d88b8d7-e73a-418c-8b6c-2a4016602f45\",\n \"rootFolder\": \"Tables\"\n }\n }\n },\n \"destination\": {\n \"type\": \"LakehouseTable\",\n \"connectionSettings\": {\n \"type\": \"Lakehouse\",\n \"typeProperties\": {\n \"workspaceId\": \"e96609ad-cc50-4c63-8829-c8499910e044\",\n \"artifactId\": \"d0e31750-29de-4992-b01d-ed022494141f\",\n \"rootFolder\": \"Tables\"\n }\n }\n },\n \"policy\": {\n \"timeout\": \"0.12:00:00\"\n }\n },\n \"activities\": [\n {\n \"id\": \"2905df0f-1421-42e5-b769-0b0a32cb5321\",\n \"properties\": {\n \"source\": {\n \"datasetSettings\": {\n \"schema\": \"dbo\",\n \"table\": \"dimension_city\"\n }\n },\n \"destination\": {\n \"partitionOption\": \"None\",\n \"writeBehavior\": \"Overwrite\",\n \"datasetSettings\": {\n \"schema\": \"dbo\",\n \"table\": \"dimension_city\"\n }\n },\n \"enableStaging\": false,\n \"translator\": {\n \"type\": \"TabularTranslator\"\n },\n \"typeConversionSettings\": {\n \"typeConversion\": {\n \"allowDataTruncation\": true,\n \"treatBooleanAsNumber\": false\n }\n }\n }\n },\n {\n \"id\": \"468840f4-e88a-42a4-b96c-4b95d0c3cd3a\",\n \"properties\": {\n \"source\": {\n \"datasetSettings\": {\n \"schema\": \"dbo\",\n \"table\": \"dimension_customer\"\n }\n },\n \"destination\": {\n \"partitionOption\": \"None\",\n \"writeBehavior\": \"Overwrite\",\n \"datasetSettings\": {\n \"schema\": \"dbo\",\n \"table\": \"dimension_customer\"\n }\n },\n \"enableStaging\": false,\n \"translator\": {\n \"type\": \"TabularTranslator\"\n },\n \"typeConversionSettings\": {\n \"typeConversion\": {\n \"allowDataTruncation\": true,\n \"treatBooleanAsNumber\": false\n }\n }\n }\n },\n {\n \"id\": \"27c592c0-21d1-4d19-b193-0eac8a2b864d\",\n \"properties\": {\n \"source\": {\n \"datasetSettings\": {\n \"schema\": \"dbo\",\n \"table\": \"dimension_date\"\n }\n },\n \"destination\": {\n \"partitionOption\": \"None\",\n \"writeBehavior\": \"Overwrite\",\n \"datasetSettings\": {\n \"schema\": \"dbo\",\n \"table\": \"dimension_date\"\n }\n },\n \"enableStaging\": false,\n \"translator\": {\n \"type\": \"TabularTranslator\"\n },\n \"typeConversionSettings\": {\n \"typeConversion\": {\n \"allowDataTruncation\": true,\n \"treatBooleanAsNumber\": false\n }\n }\n }\n },\n {\n \"id\": \"a9b14f12-b243-45f9-ab21-7642ae6c3afa\",\n \"properties\": {\n \"source\": {\n \"datasetSettings\": {\n \"schema\": \"dbo\",\n \"table\": \"dimension_employee\"\n }\n },\n \"destination\": {\n \"partitionOption\": \"None\",\n \"writeBehavior\": \"Overwrite\",\n \"datasetSettings\": {\n \"schema\": \"dbo\",\n \"table\": \"dimension_employee\"\n }\n },\n \"enableStaging\": false,\n \"translator\": {\n \"type\": \"TabularTranslator\"\n },\n \"typeConversionSettings\": {\n \"typeConversion\": {\n \"allowDataTruncation\": true,\n \"treatBooleanAsNumber\": false\n }\n }\n }\n },\n {\n \"id\": \"f76f4474-489b-4ceb-b885-a2a33df301c9\",\n \"properties\": {\n \"source\": {\n \"datasetSettings\": {\n \"schema\": \"dbo\",\n \"table\": \"dimension_stock_item\"\n }\n },\n \"destination\": {\n \"partitionOption\": \"None\",\n \"writeBehavior\": \"Overwrite\",\n \"datasetSettings\": {\n \"schema\": \"dbo\",\n \"table\": \"dimension_stock_item\"\n }\n },\n \"enableStaging\": false,\n \"translator\": {\n \"type\": \"TabularTranslator\"\n },\n \"typeConversionSettings\": {\n \"typeConversion\": {\n \"allowDataTruncation\": true,\n \"treatBooleanAsNumber\": false\n }\n }\n }\n },\n {\n \"id\": \"10390776-5fb0-4454-bcbe-b6d1cffc2da8\",\n \"properties\": {\n \"source\": {\n \"datasetSettings\": {\n \"schema\": \"dbo\",\n \"table\": \"fact_sale\"\n }\n },\n \"destination\": {\n \"partitionOption\": \"None\",\n \"writeBehavior\": \"Overwrite\",\n \"datasetSettings\": {\n \"schema\": \"dbo\",\n \"table\": \"fact_sale\"\n }\n },\n \"enableStaging\": false,\n \"translator\": {\n \"type\": \"TabularTranslator\"\n },\n \"typeConversionSettings\": {\n \"typeConversion\": {\n \"allowDataTruncation\": true,\n \"treatBooleanAsNumber\": false\n }\n }\n }\n }\n ]\n}\n" }, { "path": "sample/workspace/Hello Dataflow.Dataflow/.platform", "content": "{\n \"$schema\": \"https://developer.microsoft.com/json-schemas/fabric/gitIntegration/platformProperties/2.0.0/schema.json\",\n \"metadata\": {\n \"type\": \"Dataflow\",\n \"displayName\": \"Hello Dataflow\"\n },\n \"config\": {\n \"version\": \"2.0\",\n \"logicalId\": \"35869aa1-a7be-8319-4fa2-a84dac178bf1\"\n }\n}" }, { "path": "sample/workspace/Hello Dataflow.Dataflow/mashup.pq", "content": "[StagingDefinition = [Kind = \"FastCopy\"]]\nsection Section1;\nshared Table = let\n Source = Table.FromRows(Json.Document(Binary.Decompress(Binary.FromText(\"i45W8ssvSU3Kz89W0lEyBGKP1JycfIXw/KKcFKVYnWgll8SSxLSc/HKglBFMGi6hEJBZkJqTmZcKlDEGYoS24NTcxLySzGQF3/yU1ByglAkQg9kKqEZAzTYFYhc3mL2xAA==\", BinaryEncoding.Base64), Compression.Deflate)), let _t = ((type nullable text) meta [Serialized.Text = true]) in type table [Item = _t, Id = _t, Name = _t]),\n #\"Changed column type\" = Table.TransformColumnTypes(Source, {{\"Item\", type text}, {\"Id\", Int64.Type}, {\"Name\", type text}}),\n #\"Added custom\" = Table.TransformColumnTypes(Table.AddColumn(#\"Changed column type\", \"IsDataflow\", each if [Item] = \"Dataflow\" then true else false), {{\"IsDataflow\", type logical}}),\n #\"Filtered rows\" = Table.SelectRows(#\"Added custom\", each ([IsDataflow] = true))\nin\n #\"Filtered rows\";\n" }, { "path": "sample/workspace/Hello Dataflow.Dataflow/queryMetadata.json", "content": "{\n \"formatVersion\": \"202502\",\n \"computeEngineSettings\": {},\n \"name\": \"Hello Dataflow\",\n \"queryGroups\": [],\n \"documentLocale\": \"en-US\",\n \"queriesMetadata\": {\n \"Table\": {\n \"queryId\": \"4f5f3e2c-e1c3-4abe-825e-c98ea699ac8f\",\n \"queryName\": \"Table\"\n }\n },\n \"connections\": []\n}" }, { "path": "sample/workspace/Hello World.Notebook/.platform", "content": "{\n \"$schema\": \"https://developer.microsoft.com/json-schemas/fabric/gitIntegration/platformProperties/2.0.0/schema.json\",\n \"metadata\": {\n \"type\": \"Notebook\",\n \"displayName\": \"Hello World\",\n \"description\": \"Sample notebook\"\n },\n \"config\": {\n \"version\": \"2.0\",\n \"logicalId\": \"99b570c5-0c79-9dc4-4c9b-fa16c621384c\"\n }\n}" }, { "path": "sample/workspace/Hello World.Notebook/notebook-content.py", "content": "# Fabric notebook source\n\n# METADATA ********************\n\n# META {\n# META \"kernel_info\": {\n# META \"name\": \"synapse_pyspark\"\n# META },\n# META \"dependencies\": {\n# META \"environment\": {\n# META \"environmentId\": \"a277ea4a-e87f-8537-4ce0-39db11d4aade\",\n# META \"workspaceId\": \"00000000-0000-0000-0000-000000000000\"\n# META }\n# META }\n# META }\n\n# CELL ********************\n\nprint(\"Hello World\")\n\n# METADATA ********************\n\n# META {\n# META \"language\": \"python\",\n# META \"language_group\": \"synapse_pyspark\"\n# META }\n" }, { "path": "sample/workspace/Hello db.SQLDatabase/.gitignore", "content": "## Ignore Visual Studio temporary files, build results, and\n## files generated by popular Visual Studio add-ons.\n##\n## Get latest from `dotnet new gitignore`\n\n# dotenv files\n.env\n\n# User-specific files\n*.rsuser\n*.suo\n*.user\n*.userosscache\n*.sln.docstates\n\n# User-specific files (MonoDevelop/Xamarin Studio)\n*.userprefs\n\n# Mono auto generated files\nmono_crash.*\n\n# Build results\n[Dd]ebug/\n[Dd]ebugPublic/\n[Rr]elease/\n[Rr]eleases/\nx64/\nx86/\n[Ww][Ii][Nn]32/\n[Aa][Rr][Mm]/\n[Aa][Rr][Mm]64/\nbld/\n[Bb]in/\n[Oo]bj/\n[Ll]og/\n[Ll]ogs/\n\n# Visual Studio 2015/2017 cache/options directory\n.vs/\n# Uncomment if you have tasks that create the project's static files in wwwroot\n#wwwroot/\n\n# Visual Studio 2017 auto generated files\nGenerated\\ Files/\n\n# MSTest test Results\n[Tt]est[Rr]esult*/\n[Bb]uild[Ll]og.*\n\n# NUnit\n*.VisualState.xml\nTestResult.xml\nnunit-*.xml\n\n# Build Results of an ATL Project\n[Dd]ebugPS/\n[Rr]eleasePS/\ndlldata.c\n\n# Benchmark Results\nBenchmarkDotNet.Artifacts/\n\n# .NET\nproject.lock.json\nproject.fragment.lock.json\nartifacts/\n\n# Tye\n.tye/\n\n# ASP.NET Scaffolding\nScaffoldingReadMe.txt\n\n# StyleCop\nStyleCopReport.xml\n\n# Files built by Visual Studio\n*_i.c\n*_p.c\n*_h.h\n*.ilk\n*.meta\n*.obj\n*.iobj\n*.pch\n*.pdb\n*.ipdb\n*.pgc\n*.pgd\n*.rsp\n*.sbr\n*.tlb\n*.tli\n*.tlh\n*.tmp\n*.tmp_proj\n*_wpftmp.csproj\n*.log\n*.tlog\n*.vspscc\n*.vssscc\n.builds\n*.pidb\n*.svclog\n*.scc\n\n# Chutzpah Test files\n_Chutzpah*\n\n# Visual C++ cache files\nipch/\n*.aps\n*.ncb\n*.opendb\n*.opensdf\n*.sdf\n*.cachefile\n*.VC.db\n*.VC.VC.opendb\n\n# Visual Studio profiler\n*.psess\n*.vsp\n*.vspx\n*.sap\n\n# Visual Studio Trace Files\n*.e2e\n\n# TFS 2012 Local Workspace\n$tf/\n\n# Guidance Automation Toolkit\n*.gpState\n\n# ReSharper is a .NET coding add-in\n_ReSharper*/\n*.[Rr]e[Ss]harper\n*.DotSettings.user\n\n# TeamCity is a build add-in\n_TeamCity*\n\n# DotCover is a Code Coverage Tool\n*.dotCover\n\n# AxoCover is a Code Coverage Tool\n.axoCover/*\n!.axoCover/settings.json\n\n# Coverlet is a free, cross platform Code Coverage Tool\ncoverage*.json\ncoverage*.xml\ncoverage*.info\n\n# Visual Studio code coverage results\n*.coverage\n*.coveragexml\n\n# NCrunch\n_NCrunch_*\n.*crunch*.local.xml\nnCrunchTemp_*\n\n# MightyMoose\n*.mm.*\nAutoTest.Net/\n\n# Web workbench (sass)\n.sass-cache/\n\n# Installshield output folder\n[Ee]xpress/\n\n# DocProject is a documentation generator add-in\nDocProject/buildhelp/\nDocProject/Help/*.HxT\nDocProject/Help/*.HxC\nDocProject/Help/*.hhc\nDocProject/Help/*.hhk\nDocProject/Help/*.hhp\nDocProject/Help/Html2\nDocProject/Help/html\n\n# Click-Once directory\npublish/\n\n# Publish Web Output\n*.[Pp]ublish.xml\n*.azurePubxml\n# Note: Comment the next line if you want to checkin your web deploy settings,\n# but database connection strings (with potential passwords) will be unencrypted\n*.pubxml\n*.publishproj\n\n# Microsoft Azure Web App publish settings. Comment the next line if you want to\n# checkin your Azure Web App publish settings, but sensitive information contained\n# in these scripts will be unencrypted\nPublishScripts/\n\n# NuGet Packages\n*.nupkg\n# NuGet Symbol Packages\n*.snupkg\n# The packages folder can be ignored because of Package Restore\n**/[Pp]ackages/*\n# except build/, which is used as an MSBuild target.\n!**/[Pp]ackages/build/\n# Uncomment if necessary however generally it will be regenerated when needed\n#!**/[Pp]ackages/repositories.config\n# NuGet v3's project.json files produces more ignorable files\n*.nuget.props\n*.nuget.targets\n\n# Microsoft Azure Build Output\ncsx/\n*.build.csdef\n\n# Microsoft Azure Emulator\necf/\nrcf/\n\n# Windows Store app package directories and files\nAppPackages/\nBundleArtifacts/\nPackage.StoreAssociation.xml\n_pkginfo.txt\n*.appx\n*.appxbundle\n*.appxupload\n\n# Visual Studio cache files\n# files ending in .cache can be ignored\n*.[Cc]ache\n# but keep track of directories ending in .cache\n!?*.[Cc]ache/\n\n# Others\nClientBin/\n~$*\n*~\n*.dbmdl\n*.dbproj.schemaview\n*.jfm\n*.pfx\n*.publishsettings\norleans.codegen.cs\n\n# Including strong name files can present a security risk\n# (https://github.com/github/gitignore/pull/2483#issue-259490424)\n#*.snk\n\n# Since there are multiple workflows, uncomment next line to ignore bower_components\n# (https://github.com/github/gitignore/pull/1529#issuecomment-104372622)\n#bower_components/\n\n# RIA/Silverlight projects\nGenerated_Code/\n\n# Backup & report files from converting an old project file\n# to a newer Visual Studio version. Backup files are not needed,\n# because we have git ;-)\n_UpgradeReport_Files/\nBackup*/\nUpgradeLog*.XML\nUpgradeLog*.htm\nServiceFabricBackup/\n*.rptproj.bak\n\n# SQL Server files\n*.mdf\n*.ldf\n*.ndf\n\n# Business Intelligence projects\n*.rdl.data\n*.bim.layout\n*.bim_*.settings\n*.rptproj.rsuser\n*- [Bb]ackup.rdl\n*- [Bb]ackup ([0-9]).rdl\n*- [Bb]ackup ([0-9][0-9]).rdl\n\n# Microsoft Fakes\nFakesAssemblies/\n\n# GhostDoc plugin setting file\n*.GhostDoc.xml\n\n# Node.js Tools for Visual Studio\n.ntvs_analysis.dat\nnode_modules/\n\n# Visual Studio 6 build log\n*.plg\n\n# Visual Studio 6 workspace options file\n*.opt\n\n# Visual Studio 6 auto-generated workspace file (contains which files were open etc.)\n*.vbw\n\n# Visual Studio 6 auto-generated project file (contains which files were open etc.)\n*.vbp\n\n# Visual Studio 6 workspace and project file (working project files containing files to include in project)\n*.dsw\n*.dsp\n\n# Visual Studio 6 technical files\n*.ncb\n*.aps\n\n# Visual Studio LightSwitch build output\n**/*.HTMLClient/GeneratedArtifacts\n**/*.DesktopClient/GeneratedArtifacts\n**/*.DesktopClient/ModelManifest.xml\n**/*.Server/GeneratedArtifacts\n**/*.Server/ModelManifest.xml\n_Pvt_Extensions\n\n# Paket dependency manager\n.paket/paket.exe\npaket-files/\n\n# FAKE - F# Make\n.fake/\n\n# CodeRush personal settings\n.cr/personal\n\n# Python Tools for Visual Studio (PTVS)\n__pycache__/\n*.pyc\n\n# Cake - Uncomment if you are using it\n# tools/**\n# !tools/packages.config\n\n# Tabs Studio\n*.tss\n\n# Telerik's JustMock configuration file\n*.jmconfig\n\n# BizTalk build output\n*.btp.cs\n*.btm.cs\n*.odx.cs\n*.xsd.cs\n\n# OpenCover UI analysis results\nOpenCover/\n\n# Azure Stream Analytics local run output\nASALocalRun/\n\n# MSBuild Binary and Structured Log\n*.binlog\n\n# NVidia Nsight GPU debugger configuration file\n*.nvuser\n\n# MFractors (Xamarin productivity tool) working folder\n.mfractor/\n\n# Local History for Visual Studio\n.localhistory/\n\n# Visual Studio History (VSHistory) files\n.vshistory/\n\n# BeatPulse healthcheck temp database\nhealthchecksdb\n\n# Backup folder for Package Reference Convert tool in Visual Studio 2017\nMigrationBackup/\n\n# Ionide (cross platform F# VS Code tools) working folder\n.ionide/\n\n# Fody - auto-generated XML schema\nFodyWeavers.xsd\n\n# VS Code files for those working on multiple tools\n.vscode/*\n!.vscode/settings.json\n!.vscode/tasks.json\n!.vscode/launch.json\n!.vscode/extensions.json\n*.code-workspace\n\n# Local History for Visual Studio Code\n.history/\n\n# Windows Installer files from build outputs\n*.cab\n*.msi\n*.msix\n*.msm\n*.msp\n\n# JetBrains Rider\n*.sln.iml\n.idea\n\n##\n## Visual studio for Mac\n##\n\n\n# globs\nMakefile.in\n*.userprefs\n*.usertasks\nconfig.make\nconfig.status\naclocal.m4\ninstall-sh\nautom4te.cache/\n*.tar.gz\ntarballs/\ntest-results/\n\n# Mac bundle stuff\n*.dmg\n*.app\n\n# content below from: https://github.com/github/gitignore/blob/master/Global/macOS.gitignore\n# General\n.DS_Store\n.AppleDouble\n.LSOverride\n\n# Icon must end with two \\r\nIcon\n\n\n# Thumbnails\n._*\n\n# Files that might appear in the root of a volume\n.DocumentRevisions-V100\n.fseventsd\n.Spotlight-V100\n.TemporaryItems\n.Trashes\n.VolumeIcon.icns\n.com.apple.timemachine.donotpresent\n\n# Directories potentially created on remote AFP share\n.AppleDB\n.AppleDesktop\nNetwork Trash Folder\nTemporary Items\n.apdisk\n\n# content below from: https://github.com/github/gitignore/blob/master/Global/Windows.gitignore\n# Windows thumbnail cache files\nThumbs.db\nehthumbs.db\nehthumbs_vista.db\n\n# Dump file\n*.stackdump\n\n# Folder config file\n[Dd]esktop.ini\n\n# Recycle Bin used on file shares\n$RECYCLE.BIN/\n\n# Windows Installer files\n*.cab\n*.msi\n*.msix\n*.msm\n*.msp\n\n# Windows shortcuts\n*.lnk\n\n# Vim temporary swap files\n*.swp\n" }, { "path": "sample/workspace/Hello db.SQLDatabase/.platform", "content": "{\n \"$schema\": \"https://developer.microsoft.com/json-schemas/fabric/gitIntegration/platformProperties/2.0.0/schema.json\",\n \"metadata\": {\n \"type\": \"SQLDatabase\",\n \"displayName\": \"Hello db\"\n },\n \"config\": {\n \"version\": \"2.0\",\n \"logicalId\": \"fab68254-2c13-9624-451f-7ef2912347a6\"\n }\n}\n" }, { "path": "sample/workspace/Hello db.SQLDatabase/Hello db.sqlproj", "content": "\n\n \n \n Hello db\n {00000000-0000-0000-0000-000000000000}\n Microsoft.Data.Tools.Schema.Sql.SqlDbFabricDatabaseSchemaProvider\n 1033, CI\n \n \n \n \n\n" }, { "path": "sample/workspace/HelloEventhouse.Eventhouse/.children/HelloEventhouse.KQLDatabase/.platform", "content": "{\n \"$schema\": \"https://developer.microsoft.com/json-schemas/fabric/gitIntegration/platformProperties/2.0.0/schema.json\",\n \"metadata\": {\n \"type\": \"KQLDatabase\",\n \"displayName\": \"HelloEventhouse\",\n \"description\": \"HelloEventhouse\"\n },\n \"config\": {\n \"version\": \"2.0\",\n \"logicalId\": \"9ec49357-fad2-8ac5-4295-0dcf7d164fc7\"\n }\n}" }, { "path": "sample/workspace/HelloEventhouse.Eventhouse/.children/HelloEventhouse.KQLDatabase/DatabaseProperties.json", "content": "{\n \"databaseType\": \"ReadWrite\",\n \"parentEventhouseItemId\": \"50adae54-3e43-9fda-464a-757b7f9fb86b\",\n \"oneLakeCachingPeriod\": \"P36500D\",\n \"oneLakeStandardStoragePeriod\": \"P36500D\"\n}" }, { "path": "sample/workspace/HelloEventhouse.Eventhouse/.children/HelloEventhouse.KQLDatabase/DatabaseSchema.kql", "content": "// KQL script\n// Use management commands in this script to configure your database items, such as tables, functions, materialized views, and more.\n\n\n.create-merge table YellowTaxi (vendorID:string, tpepPickupDateTime:datetime, tpepDropoffDateTime:datetime, passengerCount:int, tripDistance:real, puLocationId:string, doLocationId:string, startLon:string, startLat:string, endLon:string, endLat:string, rateCodeId:string, storeAndFwdFlag:string, paymentType:string, fareAmount:real, extra:int, mtaTax:real, improvementSurcharge:real, tipAmount:real, tollsAmount:int, totalAmount:real, EventProcessedUtcTime:datetime, PartitionId:string, EventEnqueuedUtcTime:datetime) \n.create-or-alter table YellowTaxi ingestion json mapping 'YellowTaxi_mapping'\n```\n[{\"Properties\":{\"Path\":\"$['vendorID']\"},\"column\":\"vendorID\",\"datatype\":\"\"},{\"Properties\":{\"Path\":\"$['tpepPickupDateTime']\",\"Transform\":\"DateTimeFromUnixMilliseconds\"},\"column\":\"tpepPickupDateTime\",\"datatype\":\"\"},{\"Properties\":{\"Path\":\"$['tpepDropoffDateTime']\",\"Transform\":\"DateTimeFromUnixMilliseconds\"},\"column\":\"tpepDropoffDateTime\",\"datatype\":\"\"},{\"Properties\":{\"Path\":\"$['passengerCount']\"},\"column\":\"passengerCount\",\"datatype\":\"\"},{\"Properties\":{\"Path\":\"$['tripDistance']\"},\"column\":\"tripDistance\",\"datatype\":\"\"},{\"Properties\":{\"Path\":\"$['puLocationId']\"},\"column\":\"puLocationId\",\"datatype\":\"\"},{\"Properties\":{\"Path\":\"$['doLocationId']\"},\"column\":\"doLocationId\",\"datatype\":\"\"},{\"Properties\":{\"Path\":\"$['startLon']\"},\"column\":\"startLon\",\"datatype\":\"\"},{\"Properties\":{\"Path\":\"$['startLat']\"},\"column\":\"startLat\",\"datatype\":\"\"},{\"Properties\":{\"Path\":\"$['endLon']\"},\"column\":\"endLon\",\"datatype\":\"\"},{\"Properties\":{\"Path\":\"$['endLat']\"},\"column\":\"endLat\",\"datatype\":\"\"},{\"Properties\":{\"Path\":\"$['rateCodeId']\"},\"column\":\"rateCodeId\",\"datatype\":\"\"},{\"Properties\":{\"Path\":\"$['storeAndFwdFlag']\"},\"column\":\"storeAndFwdFlag\",\"datatype\":\"\"},{\"Properties\":{\"Path\":\"$['paymentType']\"},\"column\":\"paymentType\",\"datatype\":\"\"},{\"Properties\":{\"Path\":\"$['fareAmount']\"},\"column\":\"fareAmount\",\"datatype\":\"\"},{\"Properties\":{\"Path\":\"$['extra']\"},\"column\":\"extra\",\"datatype\":\"\"},{\"Properties\":{\"Path\":\"$['mtaTax']\"},\"column\":\"mtaTax\",\"datatype\":\"\"},{\"Properties\":{\"Path\":\"$['improvementSurcharge']\"},\"column\":\"improvementSurcharge\",\"datatype\":\"\"},{\"Properties\":{\"Path\":\"$['tipAmount']\"},\"column\":\"tipAmount\",\"datatype\":\"\"},{\"Properties\":{\"Path\":\"$['tollsAmount']\"},\"column\":\"tollsAmount\",\"datatype\":\"\"},{\"Properties\":{\"Path\":\"$['totalAmount']\"},\"column\":\"totalAmount\",\"datatype\":\"\"},{\"Properties\":{\"Path\":\"$['EventProcessedUtcTime']\"},\"column\":\"EventProcessedUtcTime\",\"datatype\":\"\"},{\"Properties\":{\"Path\":\"$['PartitionId']\"},\"column\":\"PartitionId\",\"datatype\":\"\"},{\"Properties\":{\"Path\":\"$['EventEnqueuedUtcTime']\"},\"column\":\"EventEnqueuedUtcTime\",\"datatype\":\"\"}]\n```\n" }, { "path": "sample/workspace/HelloEventhouse.Eventhouse/.platform", "content": "{\n \"$schema\": \"https://developer.microsoft.com/json-schemas/fabric/gitIntegration/platformProperties/2.0.0/schema.json\",\n \"metadata\": {\n \"type\": \"Eventhouse\",\n \"displayName\": \"HelloEventhouse\",\n \"description\": \"HelloEventhouse\"\n },\n \"config\": {\n \"version\": \"2.0\",\n \"logicalId\": \"50adae54-3e43-9fda-464a-757b7f9fb86b\"\n }\n}" }, { "path": "sample/workspace/HelloEventhouse.Eventhouse/EventhouseProperties.json", "content": "{}" }, { "path": "sample/workspace/HelloRealTimeDashboard.KQLDashboard/.platform", "content": "{\n \"$schema\": \"https://developer.microsoft.com/json-schemas/fabric/gitIntegration/platformProperties/2.0.0/schema.json\",\n \"metadata\": {\n \"type\": \"KQLDashboard\",\n \"displayName\": \"HelloRealTimeDashboard\",\n \"description\": \"HelloRealTimeDashboard\"\n },\n \"config\": {\n \"version\": \"2.0\",\n \"logicalId\": \"d105911e-7acf-b25d-45a2-0cf90211a6d0\"\n }\n}" }, { "path": "sample/workspace/HelloRealTimeDashboard.KQLDashboard/RealTimeDashboard.json", "content": "{\n \"schema_version\": \"60\",\n \"tiles\": [\n {\n \"id\": \"bad34d7e-6462-4811-b17a-5ed946fa0bf1\",\n \"title\": \"New tile\",\n \"visualType\": \"table\",\n \"pageId\": \"48f62f8e-cb94-443b-8a98-9dae1edb6a35\",\n \"layout\": {\n \"x\": 0,\n \"y\": 0,\n \"width\": 15,\n \"height\": 8\n },\n \"queryRef\": {\n \"kind\": \"query\",\n \"queryId\": \"8e1eced0-4d8a-43a8-8932-ec36f799b06f\"\n },\n \"visualOptions\": {\n \"table__enableRenderLinks\": true,\n \"colorRulesDisabled\": true,\n \"colorStyle\": \"light\",\n \"crossFilterDisabled\": false,\n \"drillthroughDisabled\": false,\n \"crossFilter\": [],\n \"drillthrough\": [],\n \"table__renderLinks\": [],\n \"colorRules\": []\n }\n },\n {\n \"id\": \"197cd6d1-d4fb-48c2-8abd-cb293552caa5\",\n \"title\": \"New tile\",\n \"visualType\": \"table\",\n \"pageId\": \"48f62f8e-cb94-443b-8a98-9dae1edb6a35\",\n \"layout\": {\n \"x\": 15,\n \"y\": 0,\n \"width\": 9,\n \"height\": 7\n },\n \"queryRef\": {\n \"kind\": \"query\",\n \"queryId\": \"80edd254-bbbe-4d44-af41-a1bf292ee045\"\n },\n \"visualOptions\": {\n \"table__enableRenderLinks\": true,\n \"colorRulesDisabled\": true,\n \"colorStyle\": \"light\",\n \"crossFilter\": [],\n \"crossFilterDisabled\": false,\n \"drillthroughDisabled\": false,\n \"drillthrough\": [],\n \"table__renderLinks\": [],\n \"colorRules\": []\n }\n }\n ],\n \"baseQueries\": [\n {\n \"id\": \"c6be13e1-fc14-44ef-a4bf-5670ebb67943\",\n \"queryId\": \"6632d0f7-5d1e-47de-9aca-10c04ecd230b\",\n \"variableName\": \"PassengerCount\"\n }\n ],\n \"parameters\": [\n {\n \"kind\": \"duration\",\n \"id\": \"b9410c12-493a-455c-bac9-7d2a7eecd110\",\n \"displayName\": \"Time range\",\n \"description\": \"\",\n \"beginVariableName\": \"_startTime\",\n \"endVariableName\": \"_endTime\",\n \"defaultValue\": {\n \"kind\": \"dynamic\",\n \"count\": 1,\n \"unit\": \"hours\"\n },\n \"showOnPages\": {\n \"kind\": \"all\"\n }\n },\n {\n \"kind\": \"int\",\n \"id\": \"e8acee0e-b0ce-42a9-b340-bd934d434fd3\",\n \"displayName\": \"PassengerCount\",\n \"description\": \"\",\n \"variableName\": \"PassengerCounts\",\n \"selectionType\": \"scalar\",\n \"includeAllOption\": false,\n \"defaultValue\": {\n \"kind\": \"value\",\n \"value\": 1\n },\n \"dataSource\": {\n \"kind\": \"static\",\n \"values\": [\n {\n \"value\": 1\n }\n ]\n },\n \"showOnPages\": {\n \"kind\": \"all\"\n }\n }\n ],\n \"dataSources\": [\n {\n \"kind\": \"kusto-trident\",\n \"scopeId\": \"kusto-trident\",\n \"clusterUri\": \"\",\n \"database\": \"9ec49357-fad2-8ac5-4295-0dcf7d164fc7\",\n \"name\": \"HelloEventhouse\",\n \"id\": \"13efefa9-c29f-420a-9f53-834aece3eacf\",\n \"workspace\": \"00000000-0000-0000-0000-000000000000\"\n },\n {\n \"kind\": \"kusto-trident\",\n \"scopeId\": \"kusto-trident\",\n \"clusterUri\": \"https://trd-srpupn7qadyes7d72g.z2.kusto.fabric.microsoft.com\",\n \"database\": \"a65d86d8-ba4d-4fbf-b264-b3a504ced0ab\",\n \"name\": \"Test1\",\n \"id\": \"c33c6d70-d6fa-4d50-a8b4-3c44b969c033\",\n \"workspace\": \"90c87160-6b48-4215-994f-710bb0755246\"\n }\n ],\n \"pages\": [\n {\n \"name\": \"Passengers\",\n \"id\": \"48f62f8e-cb94-443b-8a98-9dae1edb6a35\"\n }\n ],\n \"queries\": [\n {\n \"dataSource\": {\n \"kind\": \"inline\",\n \"dataSourceId\": \"13efefa9-c29f-420a-9f53-834aece3eacf\"\n },\n \"text\": \"// Please enter your KQL query (Example):\\n// \\n// | where between (['_startTime'] .. ['_endTime']) // Time range filtering\\n// | take 100\\nYellowTaxi\\n| where passengerCount > 1\\n\",\n \"id\": \"8e1eced0-4d8a-43a8-8932-ec36f799b06f\",\n \"usedVariables\": []\n },\n {\n \"dataSource\": {\n \"kind\": \"inline\",\n \"dataSourceId\": \"c33c6d70-d6fa-4d50-a8b4-3c44b969c033\"\n },\n \"text\": \"// Please enter your KQL query (Example):\\n//
\\n// | where between (['_startTime'] .. ['_endTime']) // Time range filtering\\n// | where isempty(['PassengerCounts']) or == ['PassengerCounts'] // Single selection filtering\\n// | take 100\\nWeather\\n| take 100\",\n \"id\": \"80edd254-bbbe-4d44-af41-a1bf292ee045\",\n \"usedVariables\": []\n },\n {\n \"id\": \"6632d0f7-5d1e-47de-9aca-10c04ecd230b\",\n \"dataSource\": {\n \"kind\": \"inline\",\n \"dataSourceId\": \"13efefa9-c29f-420a-9f53-834aece3eacf\"\n },\n \"text\": \"// Please enter your KQL query (Example):\\n//
\\n// | where between (['_startTime'] .. ['_endTime']) // Time range filtering\\n// | take 100\\nYellowTaxi\\n| where passengerCount > 1\",\n \"usedVariables\": []\n }\n ]\n}\n" }, { "path": "sample/workspace/MirroredDatabase_1.MirroredDatabase/.platform", "content": "{\n \"$schema\": \"https://developer.microsoft.com/json-schemas/fabric/gitIntegration/platformProperties/2.0.0/schema.json\",\n \"metadata\": {\n \"type\": \"MirroredDatabase\",\n \"displayName\": \"MirroredDatabase_1\"\n },\n \"config\": {\n \"version\": \"2.0\",\n \"logicalId\": \"7b1dcb5e-9744-86a5-447a-67e0a4bef6f8\"\n }\n}" }, { "path": "sample/workspace/MirroredDatabase_1.MirroredDatabase/mirroring.json", "content": "{\n \"properties\": {\n \"source\": {\n \"type\": \"GenericMirror\",\n \"typeProperties\": null\n },\n \"target\": {\n \"type\": \"MountedRelationalDatabase\",\n \"typeProperties\": {\n \"format\": \"Delta\",\n \"defaultSchema\": \"dbo\"\n }\n }\n }\n}" }, { "path": "sample/workspace/OntologyDataLH.Lakehouse/.platform", "content": "{\n \"$schema\": \"https://developer.microsoft.com/json-schemas/fabric/gitIntegration/platformProperties/2.0.0/schema.json\",\n \"metadata\": {\n \"type\": \"Lakehouse\",\n \"displayName\": \"OntologyDataLH\"\n },\n \"config\": {\n \"version\": \"2.0\",\n \"logicalId\": \"19b772b2-c916-8ae7-41fa-1fa21fbf1fb0\"\n }\n}" }, { "path": "sample/workspace/OntologyDataLH.Lakehouse/alm.settings.json", "content": "{\n \"version\": \"1.0.1\",\n \"objectTypes\": [\n {\n \"name\": \"Shortcuts\",\n \"state\": \"Enabled\",\n \"subObjectTypes\": [\n {\n \"name\": \"Shortcuts.OneLake\",\n \"state\": \"Enabled\"\n },\n {\n \"name\": \"Shortcuts.AdlsGen2\",\n \"state\": \"Enabled\"\n },\n {\n \"name\": \"Shortcuts.Dataverse\",\n \"state\": \"Enabled\"\n },\n {\n \"name\": \"Shortcuts.AmazonS3\",\n \"state\": \"Enabled\"\n },\n {\n \"name\": \"Shortcuts.S3Compatible\",\n \"state\": \"Enabled\"\n },\n {\n \"name\": \"Shortcuts.GoogleCloudStorage\",\n \"state\": \"Enabled\"\n },\n {\n \"name\": \"Shortcuts.AzureBlobStorage\",\n \"state\": \"Enabled\"\n },\n {\n \"name\": \"Shortcuts.OneDriveSharePoint\",\n \"state\": \"Enabled\"\n }\n ]\n },\n {\n \"name\": \"DataAccessRoles\",\n \"state\": \"Disabled\"\n }\n ]\n}" }, { "path": "sample/workspace/OntologyDataLH.Lakehouse/lakehouse.metadata.json", "content": "{}" }, { "path": "sample/workspace/OntologyDataLH.Lakehouse/shortcuts.metadata.json", "content": "[]" }, { "path": "sample/workspace/RetailSalesOntology.Ontology/.platform", "content": "{\n \"$schema\": \"https://developer.microsoft.com/json-schemas/fabric/gitIntegration/platformProperties/2.0.0/schema.json\",\n \"metadata\": {\n \"type\": \"Ontology\",\n \"displayName\": \"RetailSalesOntology\"\n },\n \"config\": {\n \"version\": \"2.0\",\n \"logicalId\": \"707f3e76-8ff7-867d-4c53-982a5010d492\"\n }\n}" }, { "path": "sample/workspace/RetailSalesOntology.Ontology/EntityTypes/205398164146535/DataBindings/a790fdb3-e356-4f42-acf4-4420557c0fd7.json", "content": "{\n \"$schema\": \"https://developer.microsoft.com/json-schemas/fabric/item/ontology/dataBinding/1.0.0/schema.json\",\n \"id\": \"a790fdb3-e356-4f42-acf4-4420557c0fd7\",\n \"dataBindingConfiguration\": {\n \"dataBindingType\": \"NonTimeSeries\",\n \"propertyBindings\": [\n {\n \"sourceColumnName\": \"StoreId\",\n \"targetPropertyId\": \"4251100807708466193\"\n },\n {\n \"sourceColumnName\": \"StoreName\",\n \"targetPropertyId\": \"4251100806981390461\"\n },\n {\n \"sourceColumnName\": \"City\",\n \"targetPropertyId\": \"4251100807999019669\"\n },\n {\n \"sourceColumnName\": \"Region\",\n \"targetPropertyId\": \"4251100809740297011\"\n },\n {\n \"sourceColumnName\": \"Latitude\",\n \"targetPropertyId\": \"4251100809564483949\"\n },\n {\n \"sourceColumnName\": \"Longitude\",\n \"targetPropertyId\": \"4251100807720423576\"\n }\n ],\n \"sourceTableProperties\": {\n \"sourceType\": \"LakehouseTable\",\n \"workspaceId\": \"00000000-0000-0000-0000-000000000000\",\n \"itemId\": \"19b772b2-c916-8ae7-41fa-1fa21fbf1fb0\",\n \"sourceTableName\": \"dimstore\",\n \"sourceSchema\": null\n }\n }\n}" }, { "path": "sample/workspace/RetailSalesOntology.Ontology/EntityTypes/205398164146535/definition.json", "content": "{\n \"$schema\": \"https://developer.microsoft.com/json-schemas/fabric/item/ontology/entityType/1.0.0/schema.json\",\n \"id\": \"205398164146535\",\n \"namespace\": \"usertypes\",\n \"baseEntityTypeId\": null,\n \"name\": \"Store\",\n \"entityIdParts\": [\n \"4251100807708466193\"\n ],\n \"displayNamePropertyId\": null,\n \"namespaceType\": \"Custom\",\n \"visibility\": \"Visible\",\n \"properties\": [\n {\n \"id\": \"4251100807708466193\",\n \"name\": \"StoreId\",\n \"redefines\": null,\n \"baseTypeNamespaceType\": null,\n \"valueType\": \"String\"\n },\n {\n \"id\": \"4251100806981390461\",\n \"name\": \"StoreName\",\n \"redefines\": null,\n \"baseTypeNamespaceType\": null,\n \"valueType\": \"String\"\n },\n {\n \"id\": \"4251100807999019669\",\n \"name\": \"City\",\n \"redefines\": null,\n \"baseTypeNamespaceType\": null,\n \"valueType\": \"String\"\n },\n {\n \"id\": \"4251100809740297011\",\n \"name\": \"Region\",\n \"redefines\": null,\n \"baseTypeNamespaceType\": null,\n \"valueType\": \"String\"\n },\n {\n \"id\": \"4251100809564483949\",\n \"name\": \"Latitude\",\n \"redefines\": null,\n \"baseTypeNamespaceType\": null,\n \"valueType\": \"Double\"\n },\n {\n \"id\": \"4251100807720423576\",\n \"name\": \"Longitude\",\n \"redefines\": null,\n \"baseTypeNamespaceType\": null,\n \"valueType\": \"Double\"\n }\n ],\n \"timeseriesProperties\": []\n}" }, { "path": "sample/workspace/RetailSalesOntology.Ontology/EntityTypes/267812974919544/DataBindings/275d574a-4d0d-4935-9cfd-54e59ee36d7f.json", "content": "{\n \"$schema\": \"https://developer.microsoft.com/json-schemas/fabric/item/ontology/dataBinding/1.0.0/schema.json\",\n \"id\": \"275d574a-4d0d-4935-9cfd-54e59ee36d7f\",\n \"dataBindingConfiguration\": {\n \"dataBindingType\": \"NonTimeSeries\",\n \"propertyBindings\": [\n {\n \"sourceColumnName\": \"ProductId\",\n \"targetPropertyId\": \"4247990175846782469\"\n },\n {\n \"sourceColumnName\": \"ProductName\",\n \"targetPropertyId\": \"4247990172388291209\"\n },\n {\n \"sourceColumnName\": \"Category\",\n \"targetPropertyId\": \"4247990173156094732\"\n },\n {\n \"sourceColumnName\": \"Subcategory\",\n \"targetPropertyId\": \"4247990174411431168\"\n },\n {\n \"sourceColumnName\": \"Brand\",\n \"targetPropertyId\": \"4247990176233268513\"\n }\n ],\n \"sourceTableProperties\": {\n \"sourceType\": \"LakehouseTable\",\n \"workspaceId\": \"00000000-0000-0000-0000-000000000000\",\n \"itemId\": \"19b772b2-c916-8ae7-41fa-1fa21fbf1fb0\",\n \"sourceTableName\": \"dimproducts\",\n \"sourceSchema\": null\n }\n }\n}" }, { "path": "sample/workspace/RetailSalesOntology.Ontology/EntityTypes/267812974919544/definition.json", "content": "{\n \"$schema\": \"https://developer.microsoft.com/json-schemas/fabric/item/ontology/entityType/1.0.0/schema.json\",\n \"id\": \"267812974919544\",\n \"namespace\": \"usertypes\",\n \"baseEntityTypeId\": null,\n \"name\": \"Products\",\n \"entityIdParts\": [\n \"4247990175846782469\"\n ],\n \"displayNamePropertyId\": null,\n \"namespaceType\": \"Custom\",\n \"visibility\": \"Visible\",\n \"properties\": [\n {\n \"id\": \"4247990175846782469\",\n \"name\": \"ProductId\",\n \"redefines\": null,\n \"baseTypeNamespaceType\": null,\n \"valueType\": \"String\"\n },\n {\n \"id\": \"4247990172388291209\",\n \"name\": \"ProductName\",\n \"redefines\": null,\n \"baseTypeNamespaceType\": null,\n \"valueType\": \"String\"\n },\n {\n \"id\": \"4247990173156094732\",\n \"name\": \"Category\",\n \"redefines\": null,\n \"baseTypeNamespaceType\": null,\n \"valueType\": \"String\"\n },\n {\n \"id\": \"4247990174411431168\",\n \"name\": \"Subcategory\",\n \"redefines\": null,\n \"baseTypeNamespaceType\": null,\n \"valueType\": \"String\"\n },\n {\n \"id\": \"4247990176233268513\",\n \"name\": \"Brand\",\n \"redefines\": null,\n \"baseTypeNamespaceType\": null,\n \"valueType\": \"String\"\n }\n ],\n \"timeseriesProperties\": []\n}" }, { "path": "sample/workspace/RetailSalesOntology.Ontology/EntityTypes/28747097105824/DataBindings/5f66bbb3-9bb6-415a-9cd9-9d7421d0e553.json", "content": "{\n \"$schema\": \"https://developer.microsoft.com/json-schemas/fabric/item/ontology/dataBinding/1.0.0/schema.json\",\n \"id\": \"5f66bbb3-9bb6-415a-9cd9-9d7421d0e553\",\n \"dataBindingConfiguration\": {\n \"dataBindingType\": \"NonTimeSeries\",\n \"propertyBindings\": [\n {\n \"sourceColumnName\": \"FreezerId\",\n \"targetPropertyId\": \"4184311779038199817\"\n },\n {\n \"sourceColumnName\": \"Model\",\n \"targetPropertyId\": \"4184311778721622097\"\n },\n {\n \"sourceColumnName\": \"minSafeTempC\",\n \"targetPropertyId\": \"4184311777711646739\"\n },\n {\n \"sourceColumnName\": \"StoreId\",\n \"targetPropertyId\": \"4184311777870346068\"\n }\n ],\n \"sourceTableProperties\": {\n \"sourceType\": \"LakehouseTable\",\n \"workspaceId\": \"00000000-0000-0000-0000-000000000000\",\n \"itemId\": \"19b772b2-c916-8ae7-41fa-1fa21fbf1fb0\",\n \"sourceTableName\": \"freezer\",\n \"sourceSchema\": null\n }\n }\n}" }, { "path": "sample/workspace/RetailSalesOntology.Ontology/EntityTypes/28747097105824/DataBindings/f0767964-7f82-40a1-9ca2-f7e7fe931fcd.json", "content": "{\n \"$schema\": \"https://developer.microsoft.com/json-schemas/fabric/item/ontology/dataBinding/1.0.0/schema.json\",\n \"id\": \"f0767964-7f82-40a1-9ca2-f7e7fe931fcd\",\n \"dataBindingConfiguration\": {\n \"dataBindingType\": \"TimeSeries\",\n \"timestampColumnName\": \"timestamp\",\n \"propertyBindings\": [\n {\n \"sourceColumnName\": \"timestamp\",\n \"targetPropertyId\": \"4162420792245339175\"\n },\n {\n \"sourceColumnName\": \"storeId\",\n \"targetPropertyId\": \"4184311777870346068\"\n },\n {\n \"sourceColumnName\": \"freezerId\",\n \"targetPropertyId\": \"4184311779038199817\"\n },\n {\n \"sourceColumnName\": \"temperatureC\",\n \"targetPropertyId\": \"4162420789917117594\"\n },\n {\n \"sourceColumnName\": \"humidityPct\",\n \"targetPropertyId\": \"4162420789093079066\"\n },\n {\n \"sourceColumnName\": \"doorOpen\",\n \"targetPropertyId\": \"4162420791058306832\"\n }\n ],\n \"sourceTableProperties\": {\n \"sourceType\": \"KustoTable\",\n \"workspaceId\": \"00000000-0000-0000-0000-000000000000\",\n \"itemId\": \"971dc008-7a5d-a198-4021-03a663f7d3b7\",\n \"clusterUri\": \"https://trd-frevj0bfsve4ey3cpu.z1.kusto.fabric.microsoft.com\",\n \"databaseName\": \"TelemetryDataEH\",\n \"sourceTableName\": \"FreezerTelemetry\"\n }\n }\n}" }, { "path": "sample/workspace/RetailSalesOntology.Ontology/EntityTypes/28747097105824/definition.json", "content": "{\n \"$schema\": \"https://developer.microsoft.com/json-schemas/fabric/item/ontology/entityType/1.0.0/schema.json\",\n \"id\": \"28747097105824\",\n \"namespace\": \"usertypes\",\n \"baseEntityTypeId\": null,\n \"name\": \"Freezer\",\n \"entityIdParts\": [\n \"4184311779038199817\"\n ],\n \"displayNamePropertyId\": null,\n \"namespaceType\": \"Custom\",\n \"visibility\": \"Visible\",\n \"properties\": [\n {\n \"id\": \"4184311779038199817\",\n \"name\": \"FreezerId\",\n \"redefines\": null,\n \"baseTypeNamespaceType\": null,\n \"valueType\": \"String\"\n },\n {\n \"id\": \"4184311778721622097\",\n \"name\": \"Model\",\n \"redefines\": null,\n \"baseTypeNamespaceType\": null,\n \"valueType\": \"String\"\n },\n {\n \"id\": \"4184311777711646739\",\n \"name\": \"minSafeTempC\",\n \"redefines\": null,\n \"baseTypeNamespaceType\": null,\n \"valueType\": \"String\"\n },\n {\n \"id\": \"4184311777870346068\",\n \"name\": \"StoreId\",\n \"redefines\": null,\n \"baseTypeNamespaceType\": null,\n \"valueType\": \"String\"\n }\n ],\n \"timeseriesProperties\": [\n {\n \"id\": \"4162420792245339175\",\n \"name\": \"timestamp\",\n \"redefines\": null,\n \"baseTypeNamespaceType\": null,\n \"valueType\": \"DateTime\"\n },\n {\n \"id\": \"4162420789917117594\",\n \"name\": \"temperatureC\",\n \"redefines\": null,\n \"baseTypeNamespaceType\": null,\n \"valueType\": \"Double\"\n },\n {\n \"id\": \"4162420789093079066\",\n \"name\": \"humidityPct\",\n \"redefines\": null,\n \"baseTypeNamespaceType\": null,\n \"valueType\": \"Double\"\n },\n {\n \"id\": \"4162420791058306832\",\n \"name\": \"doorOpen\",\n \"redefines\": null,\n \"baseTypeNamespaceType\": null,\n \"valueType\": \"BigInt\"\n }\n ]\n}" }, { "path": "sample/workspace/RetailSalesOntology.Ontology/EntityTypes/52068896499199/DataBindings/3d6fc8a5-b442-46b2-9bee-c22d08038f2e.json", "content": "{\n \"$schema\": \"https://developer.microsoft.com/json-schemas/fabric/item/ontology/dataBinding/1.0.0/schema.json\",\n \"id\": \"3d6fc8a5-b442-46b2-9bee-c22d08038f2e\",\n \"dataBindingConfiguration\": {\n \"dataBindingType\": \"NonTimeSeries\",\n \"propertyBindings\": [\n {\n \"sourceColumnName\": \"SaleId\",\n \"targetPropertyId\": \"4246041465402381596\"\n },\n {\n \"sourceColumnName\": \"SaleDate\",\n \"targetPropertyId\": \"4246041463854715983\"\n },\n {\n \"sourceColumnName\": \"StoreId\",\n \"targetPropertyId\": \"4246041466467721141\"\n },\n {\n \"sourceColumnName\": \"ProductId\",\n \"targetPropertyId\": \"4246041463975607651\"\n },\n {\n \"sourceColumnName\": \"Units\",\n \"targetPropertyId\": \"4246041465946373253\"\n },\n {\n \"sourceColumnName\": \"RevenueUSD\",\n \"targetPropertyId\": \"4246041466087597094\"\n }\n ],\n \"sourceTableProperties\": {\n \"sourceType\": \"LakehouseTable\",\n \"workspaceId\": \"00000000-0000-0000-0000-000000000000\",\n \"itemId\": \"19b772b2-c916-8ae7-41fa-1fa21fbf1fb0\",\n \"sourceTableName\": \"factsales\",\n \"sourceSchema\": null\n }\n }\n}" }, { "path": "sample/workspace/RetailSalesOntology.Ontology/EntityTypes/52068896499199/definition.json", "content": "{\n \"$schema\": \"https://developer.microsoft.com/json-schemas/fabric/item/ontology/entityType/1.0.0/schema.json\",\n \"id\": \"52068896499199\",\n \"namespace\": \"usertypes\",\n \"baseEntityTypeId\": null,\n \"name\": \"SaleEvent\",\n \"entityIdParts\": [\n \"4246041465402381596\"\n ],\n \"displayNamePropertyId\": null,\n \"namespaceType\": \"Custom\",\n \"visibility\": \"Visible\",\n \"properties\": [\n {\n \"id\": \"4246041465402381596\",\n \"name\": \"SaleId\",\n \"redefines\": null,\n \"baseTypeNamespaceType\": null,\n \"valueType\": \"BigInt\"\n },\n {\n \"id\": \"4246041463854715983\",\n \"name\": \"SaleDate\",\n \"redefines\": null,\n \"baseTypeNamespaceType\": null,\n \"valueType\": \"DateTime\"\n },\n {\n \"id\": \"4246041466467721141\",\n \"name\": \"StoreId\",\n \"redefines\": null,\n \"baseTypeNamespaceType\": null,\n \"valueType\": \"String\"\n },\n {\n \"id\": \"4246041463975607651\",\n \"name\": \"ProductId\",\n \"redefines\": null,\n \"baseTypeNamespaceType\": null,\n \"valueType\": \"String\"\n },\n {\n \"id\": \"4246041465946373253\",\n \"name\": \"Units\",\n \"redefines\": null,\n \"baseTypeNamespaceType\": null,\n \"valueType\": \"BigInt\"\n },\n {\n \"id\": \"4246041466087597094\",\n \"name\": \"RevenueUSD\",\n \"redefines\": null,\n \"baseTypeNamespaceType\": null,\n \"valueType\": \"Double\"\n }\n ],\n \"timeseriesProperties\": []\n}" }, { "path": "sample/workspace/RetailSalesOntology.Ontology/RelationshipTypes/4160405290834524422/Contextualizations/088a81ce-2a4c-4dbc-8887-ab6b831fa047.json", "content": "{\n \"$schema\": \"https://developer.microsoft.com/json-schemas/fabric/item/ontology/contextualization/1.0.0/schema.json\",\n \"id\": \"088a81ce-2a4c-4dbc-8887-ab6b831fa047\",\n \"dataBindingTable\": {\n \"workspaceId\": \"00000000-0000-0000-0000-000000000000\",\n \"itemId\": \"19b772b2-c916-8ae7-41fa-1fa21fbf1fb0\",\n \"sourceTableName\": \"freezer\",\n \"sourceSchema\": null,\n \"sourceType\": \"LakehouseTable\"\n },\n \"sourceKeyRefBindings\": [\n {\n \"sourceColumnName\": \"StoreId\",\n \"targetPropertyId\": \"4251100807708466193\"\n }\n ],\n \"targetKeyRefBindings\": [\n {\n \"sourceColumnName\": \"FreezerId\",\n \"targetPropertyId\": \"4184311779038199817\"\n }\n ]\n}" }, { "path": "sample/workspace/RetailSalesOntology.Ontology/RelationshipTypes/4160405290834524422/definition.json", "content": "{\n \"$schema\": \"https://developer.microsoft.com/json-schemas/fabric/item/ontology/relationshipType/1.0.0/schema.json\",\n \"namespace\": \"usertypes\",\n \"id\": \"4160405290834524422\",\n \"name\": \"operates\",\n \"namespaceType\": \"Custom\",\n \"source\": {\n \"entityTypeId\": \"205398164146535\"\n },\n \"target\": {\n \"entityTypeId\": \"28747097105824\"\n }\n}" }, { "path": "sample/workspace/RetailSalesOntology.Ontology/RelationshipTypes/4194354367812289411/Contextualizations/7be8afdf-2557-4950-930e-26c762fcb5a5.json", "content": "{\n \"$schema\": \"https://developer.microsoft.com/json-schemas/fabric/item/ontology/contextualization/1.0.0/schema.json\",\n \"id\": \"7be8afdf-2557-4950-930e-26c762fcb5a5\",\n \"dataBindingTable\": {\n \"workspaceId\": \"00000000-0000-0000-0000-000000000000\",\n \"itemId\": \"19b772b2-c916-8ae7-41fa-1fa21fbf1fb0\",\n \"sourceTableName\": \"dimproducts\",\n \"sourceSchema\": null,\n \"sourceType\": \"LakehouseTable\"\n },\n \"sourceKeyRefBindings\": [\n {\n \"sourceColumnName\": \"ProductId\",\n \"targetPropertyId\": \"4247990175846782469\"\n }\n ],\n \"targetKeyRefBindings\": [\n {\n \"sourceColumnName\": \"ProductId\",\n \"targetPropertyId\": \"4246041465402381596\"\n }\n ]\n}" }, { "path": "sample/workspace/RetailSalesOntology.Ontology/RelationshipTypes/4194354367812289411/definition.json", "content": "{\n \"$schema\": \"https://developer.microsoft.com/json-schemas/fabric/item/ontology/relationshipType/1.0.0/schema.json\",\n \"namespace\": \"usertypes\",\n \"id\": \"4194354367812289411\",\n \"name\": \"soldIn\",\n \"namespaceType\": \"Custom\",\n \"source\": {\n \"entityTypeId\": \"267812974919544\"\n },\n \"target\": {\n \"entityTypeId\": \"52068896499199\"\n }\n}" }, { "path": "sample/workspace/RetailSalesOntology.Ontology/RelationshipTypes/4244194862547506054/Contextualizations/7f23e2a5-25f6-4c87-8b9a-43b3b9a142ec.json", "content": "{\n \"$schema\": \"https://developer.microsoft.com/json-schemas/fabric/item/ontology/contextualization/1.0.0/schema.json\",\n \"id\": \"7f23e2a5-25f6-4c87-8b9a-43b3b9a142ec\",\n \"dataBindingTable\": {\n \"workspaceId\": \"00000000-0000-0000-0000-000000000000\",\n \"itemId\": \"19b772b2-c916-8ae7-41fa-1fa21fbf1fb0\",\n \"sourceTableName\": \"dimstore\",\n \"sourceSchema\": null,\n \"sourceType\": \"LakehouseTable\"\n },\n \"sourceKeyRefBindings\": [\n {\n \"sourceColumnName\": \"StoreId\",\n \"targetPropertyId\": \"4251100807708466193\"\n }\n ],\n \"targetKeyRefBindings\": [\n {\n \"sourceColumnName\": \"StoreId\",\n \"targetPropertyId\": \"4246041465402381596\"\n }\n ]\n}" }, { "path": "sample/workspace/RetailSalesOntology.Ontology/RelationshipTypes/4244194862547506054/definition.json", "content": "{\n \"$schema\": \"https://developer.microsoft.com/json-schemas/fabric/item/ontology/relationshipType/1.0.0/schema.json\",\n \"namespace\": \"usertypes\",\n \"id\": \"4244194862547506054\",\n \"name\": \"has\",\n \"namespaceType\": \"Custom\",\n \"source\": {\n \"entityTypeId\": \"205398164146535\"\n },\n \"target\": {\n \"entityTypeId\": \"52068896499199\"\n }\n}" }, { "path": "sample/workspace/RetailSalesOntology.Ontology/definition.json", "content": "{}" }, { "path": "sample/workspace/Run Hello World.DataPipeline/.platform", "content": "{\n \"$schema\": \"https://developer.microsoft.com/json-schemas/fabric/gitIntegration/platformProperties/2.0.0/schema.json\",\n \"metadata\": {\n \"type\": \"DataPipeline\",\n \"displayName\": \"Run Hello World\"\n },\n \"config\": {\n \"version\": \"2.0\",\n \"logicalId\": \"70a8992d-af56-801f-4046-ad0813bac453\"\n }\n}" }, { "path": "sample/workspace/Run Hello World.DataPipeline/.schedules", "content": "{\n \"$schema\": \"https://developer.microsoft.com/json-schemas/fabric/gitIntegration/schedules/1.0.0/schema.json\",\n \"schedules\": [\n {\n \"enabled\": true,\n \"jobType\": \"Execute\",\n \"configuration\": {\n \"type\": \"Cron\",\n \"startDateTime\": \"2025-07-01T12:00:00\",\n \"endDateTime\": \"2029-07-01T12:00:00\",\n \"localTimeZoneId\": \"Pacific Standard Time\",\n \"interval\": 15\n }\n }\n ]\n}\n" }, { "path": "sample/workspace/Run Hello World.DataPipeline/pipeline-content.json", "content": "{\n \"properties\": {\n \"activities\": [\n {\n \"type\": \"TridentNotebook\",\n \"typeProperties\": {\n \"notebookId\": \"99b570c5-0c79-9dc4-4c9b-fa16c621384c\",\n \"workspaceId\": \"00000000-0000-0000-0000-000000000000\"\n },\n \"policy\": {\n \"timeout\": \"0.12:00:00\",\n \"retry\": 0,\n \"retryIntervalInSeconds\": 30,\n \"secureInput\": false,\n \"secureOutput\": false\n },\n \"name\": \"Run Hello World\",\n \"dependsOn\": []\n }\n ]\n }\n}" }, { "path": "sample/workspace/Sample.GraphQLApi/.platform", "content": "{\n \"$schema\": \"https://developer.microsoft.com/json-schemas/fabric/gitIntegration/platformProperties/2.0.0/schema.json\",\n \"metadata\": {\n \"type\": \"GraphQLApi\",\n \"displayName\": \"Sample\"\n },\n \"config\": {\n \"version\": \"2.0\",\n \"logicalId\": \"c1e7dc70-264e-928d-43fa-a371558f47e4\"\n }\n}" }, { "path": "sample/workspace/Sample.GraphQLApi/graphql-definition.json", "content": "{\n \"$schema\": \"https://developer.microsoft.com/json-schemas/fabric/item/graphqlApi/definition/1.0.0/schema.json\",\n \"datasources\": []\n}" }, { "path": "sample/workspace/SampleDataActivator.Reflex/.platform", "content": "{\n \"$schema\": \"https://developer.microsoft.com/json-schemas/fabric/gitIntegration/platformProperties/2.0.0/schema.json\",\n \"metadata\": {\n \"type\": \"Reflex\",\n \"displayName\": \"SampleDataActivator\"\n },\n \"config\": {\n \"version\": \"2.0\",\n \"logicalId\": \"c3bf82de-14b6-af39-4852-dda67eccd7c0\"\n }\n}\n" }, { "path": "sample/workspace/SampleDataActivator.Reflex/ReflexEntities.json", "content": "[]" }, { "path": "sample/workspace/SampleDataBuildToolJob.DataBuildToolJob/.platform", "content": "{\n \"$schema\": \"https://developer.microsoft.com/json-schemas/fabric/gitIntegration/platformProperties/2.0.0/schema.json\",\n \"metadata\": {\n \"type\": \"DataBuildToolJob\",\n \"displayName\": \"SampleDataBuildToolJob\",\n \"description\": \"Sample DBT job\"\n },\n \"config\": {\n \"version\": \"2.0\",\n \"logicalId\": \"6f8a77bf-5ff2-4b65-9223-8c10d4fc6a4b\"\n }\n}\n" }, { "path": "sample/workspace/SampleDataBuildToolJob.DataBuildToolJob/dbt-content.json", "content": "{\n \"project\": {\n \"projectType\": \"OneLake\",\n \"folderPath\": \"dbt\"\n },\n \"profile\": {\n \"profileType\": \"DataWarehouse\",\n \"schema\": \"analytics_schema\",\n \"connectionSettings\": {\n \"name\": \"sample_warehouse\",\n \"properties\": {\n \"type\": \"DataWarehouse\",\n \"typeProperties\": {\n \"workspaceId\": \"00000000-0000-0000-0000-000000000000\",\n \"artifactId\": \"cccccccc-3333-4444-5555-dddddddddddd\",\n \"endPoint\": \"sampleworkspace-samplewarehouse.datawarehouse.fabric.microsoft.com\"\n }\n }\n }\n },\n \"command\": {\n \"operation\": \"build\",\n \"arguments\": {\n \"exclude\": \"\",\n \"threads\": 4\n }\n }\n}\n" }, { "path": "sample/workspace/SampleEventhouse.Eventhouse/.children/TaxiDB.KQLDatabase/.platform", "content": "{\n \"$schema\": \"https://developer.microsoft.com/json-schemas/fabric/gitIntegration/platformProperties/2.0.0/schema.json\",\n \"metadata\": {\n \"type\": \"KQLDatabase\",\n \"displayName\": \"TaxiDB\"\n },\n \"config\": {\n \"version\": \"2.0\",\n \"logicalId\": \"a51e98dd-5993-8e1c-443f-02aa53d4db74\"\n }\n}\n" }, { "path": "sample/workspace/SampleEventhouse.Eventhouse/.children/TaxiDB.KQLDatabase/DatabaseProperties.json", "content": "{\n \"databaseType\": \"ReadWrite\",\n \"parentEventhouseItemId\": \"959f24d2-d283-ad08-4897-52f5ff52f4d3\",\n \"oneLakeCachingPeriod\": \"P30D\",\n \"oneLakeStandardStoragePeriod\": \"P365D\"\n}" }, { "path": "sample/workspace/SampleEventhouse.Eventhouse/.children/TaxiDB.KQLDatabase/DatabaseSchema.kql", "content": "// KQL script\n// Use management commands in this script to configure your database items, such as tables, functions, materialized views, and more.\n\n\n.create-merge table TaxiRaw (VendorID:string, tpep_pickup_datetime:datetime, tpep_dropoff_datetime:datetime, passenger_count:real, trip_distance:real, RatecodeID:real, store_and_fwd_flag:string, PULocationID:string, DOLocationID:string, payment_type:long, fare_amount:real, extra:real, mta_tax:real, tip_amount:real, tolls_amount:real, improvement_surcharge:real, total_amount:real, congestion_surcharge:real, airport_fee:real) with (folder = \"\", docstring = \"\") \n.create-or-alter table TaxiRaw ingestion json mapping 'TaxiRaw_mapping'\n```\n[{\"Properties\":{\"Path\":\"$['VendorID']\"},\"column\":\"VendorID\",\"datatype\":\"\"},{\"Properties\":{\"Path\":\"$['tpep_pickup_datetime']\"},\"column\":\"tpep_pickup_datetime\",\"datatype\":\"\"},{\"Properties\":{\"Path\":\"$['tpep_dropoff_datetime']\"},\"column\":\"tpep_dropoff_datetime\",\"datatype\":\"\"},{\"Properties\":{\"Path\":\"$['passenger_count']\"},\"column\":\"passenger_count\",\"datatype\":\"\"},{\"Properties\":{\"Path\":\"$['trip_distance']\"},\"column\":\"trip_distance\",\"datatype\":\"\"},{\"Properties\":{\"Path\":\"$['RatecodeID']\"},\"column\":\"RatecodeID\",\"datatype\":\"\"},{\"Properties\":{\"Path\":\"$['store_and_fwd_flag']\"},\"column\":\"store_and_fwd_flag\",\"datatype\":\"\"},{\"Properties\":{\"Path\":\"$['PULocationID']\"},\"column\":\"PULocationID\",\"datatype\":\"\"},{\"Properties\":{\"Path\":\"$['DOLocationID']\"},\"column\":\"DOLocationID\",\"datatype\":\"\"},{\"Properties\":{\"Path\":\"$['payment_type']\"},\"column\":\"payment_type\",\"datatype\":\"\"},{\"Properties\":{\"Path\":\"$['fare_amount']\"},\"column\":\"fare_amount\",\"datatype\":\"\"},{\"Properties\":{\"Path\":\"$['extra']\"},\"column\":\"extra\",\"datatype\":\"\"},{\"Properties\":{\"Path\":\"$['mta_tax']\"},\"column\":\"mta_tax\",\"datatype\":\"\"},{\"Properties\":{\"Path\":\"$['tip_amount']\"},\"column\":\"tip_amount\",\"datatype\":\"\"},{\"Properties\":{\"Path\":\"$['tolls_amount']\"},\"column\":\"tolls_amount\",\"datatype\":\"\"},{\"Properties\":{\"Path\":\"$['improvement_surcharge']\"},\"column\":\"improvement_surcharge\",\"datatype\":\"\"},{\"Properties\":{\"Path\":\"$['total_amount']\"},\"column\":\"total_amount\",\"datatype\":\"\"},{\"Properties\":{\"Path\":\"$['congestion_surcharge']\"},\"column\":\"congestion_surcharge\",\"datatype\":\"\"},{\"Properties\":{\"Path\":\"$['airport_fee']\"},\"column\":\"airport_fee\",\"datatype\":\"\"}]\n```\n.create-merge table ZoneLookup (LocationID:string, Borough:string, Zone:string, service_zone:string) \n.create-merge table TaxiRecords (VendorID:string, tpep_pickup_datetime:datetime, tpep_dropoff_datetime:datetime, passenger_count:real, trip_distance:real, RatecodeID:real, store_and_fwd_flag:string, PULocationID:string, DOLocationID:string, payment_type:long, fare_amount:real, extra:real, mta_tax:real, tip_amount:real, tolls_amount:real, improvement_surcharge:real, total_amount:real, congestion_surcharge:real, airport_fee:real, DOBourough:string, DOZone:string, DOService_Zone:string, PUBourough:string, PUZone:string, PUService_Zone:string) \n.create-or-alter function with (skipvalidation = \"true\") TaxiUpdate() {TaxiRaw | lookup (ZoneLookup | project DOLocationID=LocationID, DOBourough=Borough, DOZone=Zone, DOService_Zone=service_zone) on DOLocationID | lookup (ZoneLookup | project PULocationID=LocationID, PUBourough=Borough, PUZone=Zone, PUService_Zone=service_zone) on PULocationID }\n.create-or-alter materialized-view TaxiRecordsDedup on table TaxiRecords { TaxiRecords | summarize take_any(*) by VendorID, tpep_pickup_datetime, tpep_dropoff_datetime, PULocationID, DOLocationID, trip_distance }\n.alter materialized-view TaxiRecordsDedup policy caching hotdata=time(14.00:00:00) hotindex=time(14.00:00:00)\n.create-or-alter materialized-view TaxiRecordsHourly on materialized-view TaxiRecordsDedup { TaxiRecordsDedup | summarize Avg_Passenger_Count=avg(passenger_count), Avg_Trip_Distance=avg(trip_distance), Avg_Fare_Amount=avg(fare_amount), Avg_Extra=avg(extra), Avg_MTA_Tax=avg(mta_tax), Avg_Tip_Amount=avg(tip_amount), Avg_Tolls_Amount=avg(tolls_amount), Avg_Improvement_Surcharge=avg(improvement_surcharge), Avg_Total_Amount=avg(total_amount), Avg_Congestion_Surcharge=avg(congestion_surcharge), Avg_Airport_Fee=avg(airport_fee) by bin(tpep_pickup_datetime,1h), bin(tpep_dropoff_datetime,1h), store_and_fwd_flag, PULocationID, DOLocationID, payment_type, DOBourough, DOZone, DOService_Zone, PUBourough, PUZone, PUService_Zone }\n.alter materialized-view TaxiRecordsHourly policy caching hotdata=time(60.00:00:00) hotindex=time(60.00:00:00)\n.alter table TaxiRaw policy retention @'{\"SoftDeletePeriod\":\"1.00:00:00\",\"Recoverability\":\"Enabled\"}'\n.alter table TaxiRaw policy caching hotdata = time(1.00:00:00) hotindex = time(1.00:00:00)\n.alter table TaxiRaw policy streamingingestion \"{\\\"IsEnabled\\\":false,\\\"HintAllocatedRate\\\":null,\\\"NumberOfRowStores\\\":null,\\\"SealIntervalLimit\\\":null,\\\"SealThresholdBytes\\\":null,\\\"UsageTags\\\":[],\\\"IsMaintenanceActive\\\":false}\"\n.alter table TaxiRecords policy retention @'{\"SoftDeletePeriod\":\"10.00:00:00\",\"Recoverability\":\"Enabled\"}'\n.alter table TaxiRecords policy caching hotdata = time(3.00:00:00) hotindex = time(3.00:00:00)\n.alter table TaxiRecords policy update \"[{\\\"IsEnabled\\\":true,\\\"Source\\\":\\\"TaxiRaw\\\",\\\"Query\\\":\\\"TaxiUpdate()\\\",\\\"IsTransactional\\\":true,\\\"PropagateIngestionProperties\\\":false,\\\"ManagedIdentity\\\":null}]\"\n" }, { "path": "sample/workspace/SampleEventhouse.Eventhouse/.platform", "content": "{\n \"$schema\": \"https://developer.microsoft.com/json-schemas/fabric/gitIntegration/platformProperties/2.0.0/schema.json\",\n \"metadata\": {\n \"type\": \"Eventhouse\",\n \"displayName\": \"SampleEventhouse\"\n },\n \"config\": {\n \"version\": \"2.0\",\n \"logicalId\": \"959f24d2-d283-ad08-4897-52f5ff52f4d3\"\n }\n}\n" }, { "path": "sample/workspace/SampleEventhouse.Eventhouse/EventhouseProperties.json", "content": "{}" }, { "path": "sample/workspace/SampleEventstream.Eventstream/.platform", "content": "{\n \"$schema\": \"https://developer.microsoft.com/json-schemas/fabric/gitIntegration/platformProperties/2.0.0/schema.json\",\n \"metadata\": {\n \"type\": \"Eventstream\",\n \"displayName\": \"SampleEventstream\"\n },\n \"config\": {\n \"version\": \"2.0\",\n \"logicalId\": \"5d252c47-5ea6-b389-447d-d5f9aec1c6e7\"\n }\n}\n" }, { "path": "sample/workspace/SampleEventstream.Eventstream/eventstream.json", "content": "{\n \"sources\": [\n {\n \"id\": \"d4686b8c-e228-4328-871d-1f2d2bcd6248\",\n \"name\": \"Taxi\",\n \"type\": \"SampleData\",\n \"properties\": {\n \"type\": \"YellowTaxi\"\n }\n }\n ],\n \"destinations\": [\n {\n \"id\": \"07890dca-ce9e-4cb0-ad46-c4a6ee442119\",\n \"name\": \"DataActivator\",\n \"type\": \"Activator\",\n \"properties\": {\n \"workspaceId\": \"00000000-0000-0000-0000-000000000000\",\n \"itemId\": \"c3bf82de-14b6-af39-4852-dda67eccd7c0\",\n \"inputSerialization\": {\n \"type\": \"Json\",\n \"properties\": {\n \"encoding\": \"UTF8\"\n }\n }\n },\n \"inputNodes\": [\n {\n \"name\": \"ManageFields\"\n }\n ],\n \"inputSchemas\": [\n {\n \"name\": \"ManageFields\",\n \"schema\": {\n \"columns\": [\n {\n \"name\": \"VendorID\",\n \"type\": \"Nvarchar(max)\",\n \"fields\": null,\n \"items\": null\n },\n {\n \"name\": \"tpep_pickup_datetime\",\n \"type\": \"DateTime\",\n \"fields\": null,\n \"items\": null\n },\n {\n \"name\": \"tpep_dropoff_datetime\",\n \"type\": \"DateTime\",\n \"fields\": null,\n \"items\": null\n },\n {\n \"name\": \"passenger_count\",\n \"type\": \"Float\",\n \"fields\": null,\n \"items\": null\n },\n {\n \"name\": \"trip_distance\",\n \"type\": \"Float\",\n \"fields\": null,\n \"items\": null\n },\n {\n \"name\": \"RatecodeID\",\n \"type\": \"Float\",\n \"fields\": null,\n \"items\": null\n },\n {\n \"name\": \"store_and_fwd_flag\",\n \"type\": \"Nvarchar(max)\",\n \"fields\": null,\n \"items\": null\n },\n {\n \"name\": \"PULocationID\",\n \"type\": \"Nvarchar(max)\",\n \"fields\": null,\n \"items\": null\n },\n {\n \"name\": \"DOLocationID\",\n \"type\": \"Nvarchar(max)\",\n \"fields\": null,\n \"items\": null\n },\n {\n \"name\": \"payment_type\",\n \"type\": \"BigInt\",\n \"fields\": null,\n \"items\": null\n },\n {\n \"name\": \"fare_amount\",\n \"type\": \"Float\",\n \"fields\": null,\n \"items\": null\n },\n {\n \"name\": \"extra\",\n \"type\": \"Float\",\n \"fields\": null,\n \"items\": null\n },\n {\n \"name\": \"mta_tax\",\n \"type\": \"Float\",\n \"fields\": null,\n \"items\": null\n },\n {\n \"name\": \"tip_amount\",\n \"type\": \"Float\",\n \"fields\": null,\n \"items\": null\n },\n {\n \"name\": \"tolls_amount\",\n \"type\": \"Float\",\n \"fields\": null,\n \"items\": null\n },\n {\n \"name\": \"improvement_surcharge\",\n \"type\": \"Float\",\n \"fields\": null,\n \"items\": null\n },\n {\n \"name\": \"total_amount\",\n \"type\": \"Float\",\n \"fields\": null,\n \"items\": null\n },\n {\n \"name\": \"congestion_surcharge\",\n \"type\": \"Float\",\n \"fields\": null,\n \"items\": null\n },\n {\n \"name\": \"airport_fee\",\n \"type\": \"Float\",\n \"fields\": null,\n \"items\": null\n }\n ]\n }\n }\n ]\n },\n {\n \"id\": \"124fb4cb-6c67-4c0a-ab0a-8a99bfefcdca\",\n \"name\": \"Lakehouse\",\n \"type\": \"Lakehouse\",\n \"properties\": {\n \"workspaceId\": \"00000000-0000-0000-0000-000000000000\",\n \"itemId\": \"c916eeb0-dd6a-ae32-4f4f-966d2414b239\",\n \"schema\": \"\",\n \"deltaTable\": \"Taxi\",\n \"inputSerialization\": {\n \"type\": \"Json\",\n \"properties\": {\n \"encoding\": \"UTF8\"\n }\n }\n },\n \"inputNodes\": [\n {\n \"name\": \"ManageFields\"\n }\n ],\n \"inputSchemas\": [\n {\n \"name\": \"ManageFields\",\n \"schema\": {\n \"columns\": [\n {\n \"name\": \"VendorID\",\n \"type\": \"Nvarchar(max)\",\n \"fields\": null,\n \"items\": null\n },\n {\n \"name\": \"tpep_pickup_datetime\",\n \"type\": \"DateTime\",\n \"fields\": null,\n \"items\": null\n },\n {\n \"name\": \"tpep_dropoff_datetime\",\n \"type\": \"DateTime\",\n \"fields\": null,\n \"items\": null\n },\n {\n \"name\": \"passenger_count\",\n \"type\": \"Float\",\n \"fields\": null,\n \"items\": null\n },\n {\n \"name\": \"trip_distance\",\n \"type\": \"Float\",\n \"fields\": null,\n \"items\": null\n },\n {\n \"name\": \"RatecodeID\",\n \"type\": \"Float\",\n \"fields\": null,\n \"items\": null\n },\n {\n \"name\": \"store_and_fwd_flag\",\n \"type\": \"Nvarchar(max)\",\n \"fields\": null,\n \"items\": null\n },\n {\n \"name\": \"PULocationID\",\n \"type\": \"Nvarchar(max)\",\n \"fields\": null,\n \"items\": null\n },\n {\n \"name\": \"DOLocationID\",\n \"type\": \"Nvarchar(max)\",\n \"fields\": null,\n \"items\": null\n },\n {\n \"name\": \"payment_type\",\n \"type\": \"BigInt\",\n \"fields\": null,\n \"items\": null\n },\n {\n \"name\": \"fare_amount\",\n \"type\": \"Float\",\n \"fields\": null,\n \"items\": null\n },\n {\n \"name\": \"extra\",\n \"type\": \"Float\",\n \"fields\": null,\n \"items\": null\n },\n {\n \"name\": \"mta_tax\",\n \"type\": \"Float\",\n \"fields\": null,\n \"items\": null\n },\n {\n \"name\": \"tip_amount\",\n \"type\": \"Float\",\n \"fields\": null,\n \"items\": null\n },\n {\n \"name\": \"tolls_amount\",\n \"type\": \"Float\",\n \"fields\": null,\n \"items\": null\n },\n {\n \"name\": \"improvement_surcharge\",\n \"type\": \"Float\",\n \"fields\": null,\n \"items\": null\n },\n {\n \"name\": \"total_amount\",\n \"type\": \"Float\",\n \"fields\": null,\n \"items\": null\n },\n {\n \"name\": \"congestion_surcharge\",\n \"type\": \"Float\",\n \"fields\": null,\n \"items\": null\n },\n {\n \"name\": \"airport_fee\",\n \"type\": \"Float\",\n \"fields\": null,\n \"items\": null\n }\n ]\n }\n }\n ]\n },\n {\n \"id\": \"cecf7a31-bfba-437a-b878-ac6ddb3d50a2\",\n \"name\": \"Eventhouse\",\n \"type\": \"Eventhouse\",\n \"properties\": {\n \"dataIngestionMode\": \"ProcessedIngestion\",\n \"workspaceId\": \"00000000-0000-0000-0000-000000000000\",\n \"itemId\": \"a51e98dd-5993-8e1c-443f-02aa53d4db74\",\n \"databaseName\": \"TaxiDB\",\n \"tableName\": \"TaxiRaw\",\n \"inputSerialization\": {\n \"type\": \"Json\",\n \"properties\": {\n \"encoding\": \"UTF8\"\n }\n }\n },\n \"inputNodes\": [\n {\n \"name\": \"ManageFields\"\n }\n ],\n \"inputSchemas\": [\n {\n \"name\": \"ManageFields\",\n \"schema\": {\n \"columns\": [\n {\n \"name\": \"VendorID\",\n \"type\": \"Nvarchar(max)\",\n \"fields\": null,\n \"items\": null\n },\n {\n \"name\": \"tpep_pickup_datetime\",\n \"type\": \"DateTime\",\n \"fields\": null,\n \"items\": null\n },\n {\n \"name\": \"tpep_dropoff_datetime\",\n \"type\": \"DateTime\",\n \"fields\": null,\n \"items\": null\n },\n {\n \"name\": \"passenger_count\",\n \"type\": \"Float\",\n \"fields\": null,\n \"items\": null\n },\n {\n \"name\": \"trip_distance\",\n \"type\": \"Float\",\n \"fields\": null,\n \"items\": null\n },\n {\n \"name\": \"RatecodeID\",\n \"type\": \"Float\",\n \"fields\": null,\n \"items\": null\n },\n {\n \"name\": \"store_and_fwd_flag\",\n \"type\": \"Nvarchar(max)\",\n \"fields\": null,\n \"items\": null\n },\n {\n \"name\": \"PULocationID\",\n \"type\": \"Nvarchar(max)\",\n \"fields\": null,\n \"items\": null\n },\n {\n \"name\": \"DOLocationID\",\n \"type\": \"Nvarchar(max)\",\n \"fields\": null,\n \"items\": null\n },\n {\n \"name\": \"payment_type\",\n \"type\": \"BigInt\",\n \"fields\": null,\n \"items\": null\n },\n {\n \"name\": \"fare_amount\",\n \"type\": \"Float\",\n \"fields\": null,\n \"items\": null\n },\n {\n \"name\": \"extra\",\n \"type\": \"Float\",\n \"fields\": null,\n \"items\": null\n },\n {\n \"name\": \"mta_tax\",\n \"type\": \"Float\",\n \"fields\": null,\n \"items\": null\n },\n {\n \"name\": \"tip_amount\",\n \"type\": \"Float\",\n \"fields\": null,\n \"items\": null\n },\n {\n \"name\": \"tolls_amount\",\n \"type\": \"Float\",\n \"fields\": null,\n \"items\": null\n },\n {\n \"name\": \"improvement_surcharge\",\n \"type\": \"Float\",\n \"fields\": null,\n \"items\": null\n },\n {\n \"name\": \"total_amount\",\n \"type\": \"Float\",\n \"fields\": null,\n \"items\": null\n },\n {\n \"name\": \"congestion_surcharge\",\n \"type\": \"Float\",\n \"fields\": null,\n \"items\": null\n },\n {\n \"name\": \"airport_fee\",\n \"type\": \"Float\",\n \"fields\": null,\n \"items\": null\n }\n ]\n }\n }\n ]\n }\n ],\n \"streams\": [\n {\n \"id\": \"bc291021-eb52-49b7-8c15-01bb51d8a940\",\n \"name\": \"Taxi-Stream\",\n \"type\": \"DefaultStream\",\n \"properties\": {},\n \"inputNodes\": [\n {\n \"name\": \"Taxi\"\n }\n ]\n }\n ],\n \"operators\": [\n {\n \"name\": \"ManageFields\",\n \"type\": \"ManageFields\",\n \"inputNodes\": [\n {\n \"name\": \"Taxi-Stream\"\n }\n ],\n \"properties\": {\n \"columns\": [\n {\n \"type\": \"Rename\",\n \"properties\": {\n \"column\": {\n \"expressionType\": \"ColumnReference\",\n \"node\": null,\n \"columnName\": \"VendorID\",\n \"columnPathSegments\": []\n }\n },\n \"alias\": \"VendorID\"\n },\n {\n \"type\": \"Cast\",\n \"properties\": {\n \"targetDataType\": \"DateTime\",\n \"column\": {\n \"expressionType\": \"ColumnReference\",\n \"node\": null,\n \"columnName\": \"tpep_pickup_datetime\",\n \"columnPathSegments\": []\n }\n },\n \"alias\": \"tpep_pickup_datetime\"\n },\n {\n \"type\": \"Cast\",\n \"properties\": {\n \"targetDataType\": \"DateTime\",\n \"column\": {\n \"expressionType\": \"ColumnReference\",\n \"node\": null,\n \"columnName\": \"tpep_dropoff_datetime\",\n \"columnPathSegments\": []\n }\n },\n \"alias\": \"tpep_dropoff_datetime\"\n },\n {\n \"type\": \"Cast\",\n \"properties\": {\n \"targetDataType\": \"Float\",\n \"column\": {\n \"expressionType\": \"ColumnReference\",\n \"node\": null,\n \"columnName\": \"passenger_count\",\n \"columnPathSegments\": []\n }\n },\n \"alias\": \"passenger_count\"\n },\n {\n \"type\": \"Cast\",\n \"properties\": {\n \"targetDataType\": \"Float\",\n \"column\": {\n \"expressionType\": \"ColumnReference\",\n \"node\": null,\n \"columnName\": \"trip_distance\",\n \"columnPathSegments\": []\n }\n },\n \"alias\": \"trip_distance\"\n },\n {\n \"type\": \"Cast\",\n \"properties\": {\n \"targetDataType\": \"Float\",\n \"column\": {\n \"expressionType\": \"ColumnReference\",\n \"node\": null,\n \"columnName\": \"RatecodeID\",\n \"columnPathSegments\": []\n }\n },\n \"alias\": \"RatecodeID\"\n },\n {\n \"type\": \"Rename\",\n \"properties\": {\n \"column\": {\n \"expressionType\": \"ColumnReference\",\n \"node\": null,\n \"columnName\": \"store_and_fwd_flag\",\n \"columnPathSegments\": []\n }\n },\n \"alias\": \"store_and_fwd_flag\"\n },\n {\n \"type\": \"Rename\",\n \"properties\": {\n \"column\": {\n \"expressionType\": \"ColumnReference\",\n \"node\": null,\n \"columnName\": \"PULocationID\",\n \"columnPathSegments\": []\n }\n },\n \"alias\": \"PULocationID\"\n },\n {\n \"type\": \"Rename\",\n \"properties\": {\n \"column\": {\n \"expressionType\": \"ColumnReference\",\n \"node\": null,\n \"columnName\": \"DOLocationID\",\n \"columnPathSegments\": []\n }\n },\n \"alias\": \"DOLocationID\"\n },\n {\n \"type\": \"Cast\",\n \"properties\": {\n \"targetDataType\": \"BigInt\",\n \"column\": {\n \"expressionType\": \"ColumnReference\",\n \"node\": null,\n \"columnName\": \"payment_type\",\n \"columnPathSegments\": []\n }\n },\n \"alias\": \"payment_type\"\n },\n {\n \"type\": \"Cast\",\n \"properties\": {\n \"targetDataType\": \"Float\",\n \"column\": {\n \"expressionType\": \"ColumnReference\",\n \"node\": null,\n \"columnName\": \"fare_amount\",\n \"columnPathSegments\": []\n }\n },\n \"alias\": \"fare_amount\"\n },\n {\n \"type\": \"Cast\",\n \"properties\": {\n \"targetDataType\": \"Float\",\n \"column\": {\n \"expressionType\": \"ColumnReference\",\n \"node\": null,\n \"columnName\": \"extra\",\n \"columnPathSegments\": []\n }\n },\n \"alias\": \"extra\"\n },\n {\n \"type\": \"Cast\",\n \"properties\": {\n \"targetDataType\": \"Float\",\n \"column\": {\n \"expressionType\": \"ColumnReference\",\n \"node\": null,\n \"columnName\": \"mta_tax\",\n \"columnPathSegments\": []\n }\n },\n \"alias\": \"mta_tax\"\n },\n {\n \"type\": \"Cast\",\n \"properties\": {\n \"targetDataType\": \"Float\",\n \"column\": {\n \"expressionType\": \"ColumnReference\",\n \"node\": null,\n \"columnName\": \"tip_amount\",\n \"columnPathSegments\": []\n }\n },\n \"alias\": \"tip_amount\"\n },\n {\n \"type\": \"Cast\",\n \"properties\": {\n \"targetDataType\": \"Float\",\n \"column\": {\n \"expressionType\": \"ColumnReference\",\n \"node\": null,\n \"columnName\": \"tolls_amount\",\n \"columnPathSegments\": []\n }\n },\n \"alias\": \"tolls_amount\"\n },\n {\n \"type\": \"Cast\",\n \"properties\": {\n \"targetDataType\": \"Float\",\n \"column\": {\n \"expressionType\": \"ColumnReference\",\n \"node\": null,\n \"columnName\": \"improvement_surcharge\",\n \"columnPathSegments\": []\n }\n },\n \"alias\": \"improvement_surcharge\"\n },\n {\n \"type\": \"Cast\",\n \"properties\": {\n \"targetDataType\": \"Float\",\n \"column\": {\n \"expressionType\": \"ColumnReference\",\n \"node\": null,\n \"columnName\": \"total_amount\",\n \"columnPathSegments\": []\n }\n },\n \"alias\": \"total_amount\"\n },\n {\n \"type\": \"Cast\",\n \"properties\": {\n \"targetDataType\": \"Float\",\n \"column\": {\n \"expressionType\": \"ColumnReference\",\n \"node\": null,\n \"columnName\": \"congestion_surcharge\",\n \"columnPathSegments\": []\n }\n },\n \"alias\": \"congestion_surcharge\"\n },\n {\n \"type\": \"Cast\",\n \"properties\": {\n \"targetDataType\": \"Float\",\n \"column\": {\n \"expressionType\": \"ColumnReference\",\n \"node\": null,\n \"columnName\": \"airport_fee\",\n \"columnPathSegments\": []\n }\n },\n \"alias\": \"airport_fee\"\n }\n ]\n },\n \"inputSchemas\": [\n {\n \"name\": \"Taxi-Stream\",\n \"schema\": {\n \"columns\": [\n {\n \"name\": \"VendorID\",\n \"type\": \"Nvarchar(max)\",\n \"fields\": null,\n \"items\": null\n },\n {\n \"name\": \"tpep_pickup_datetime\",\n \"type\": \"Nvarchar(max)\",\n \"fields\": null,\n \"items\": null\n },\n {\n \"name\": \"tpep_dropoff_datetime\",\n \"type\": \"Nvarchar(max)\",\n \"fields\": null,\n \"items\": null\n },\n {\n \"name\": \"passenger_count\",\n \"type\": \"Nvarchar(max)\",\n \"fields\": null,\n \"items\": null\n },\n {\n \"name\": \"trip_distance\",\n \"type\": \"Nvarchar(max)\",\n \"fields\": null,\n \"items\": null\n },\n {\n \"name\": \"RatecodeID\",\n \"type\": \"Nvarchar(max)\",\n \"fields\": null,\n \"items\": null\n },\n {\n \"name\": \"store_and_fwd_flag\",\n \"type\": \"Nvarchar(max)\",\n \"fields\": null,\n \"items\": null\n },\n {\n \"name\": \"PULocationID\",\n \"type\": \"Nvarchar(max)\",\n \"fields\": null,\n \"items\": null\n },\n {\n \"name\": \"DOLocationID\",\n \"type\": \"Nvarchar(max)\",\n \"fields\": null,\n \"items\": null\n },\n {\n \"name\": \"payment_type\",\n \"type\": \"Nvarchar(max)\",\n \"fields\": null,\n \"items\": null\n },\n {\n \"name\": \"fare_amount\",\n \"type\": \"Nvarchar(max)\",\n \"fields\": null,\n \"items\": null\n },\n {\n \"name\": \"extra\",\n \"type\": \"Nvarchar(max)\",\n \"fields\": null,\n \"items\": null\n },\n {\n \"name\": \"mta_tax\",\n \"type\": \"Nvarchar(max)\",\n \"fields\": null,\n \"items\": null\n },\n {\n \"name\": \"tip_amount\",\n \"type\": \"Nvarchar(max)\",\n \"fields\": null,\n \"items\": null\n },\n {\n \"name\": \"tolls_amount\",\n \"type\": \"Nvarchar(max)\",\n \"fields\": null,\n \"items\": null\n },\n {\n \"name\": \"improvement_surcharge\",\n \"type\": \"Nvarchar(max)\",\n \"fields\": null,\n \"items\": null\n },\n {\n \"name\": \"total_amount\",\n \"type\": \"Nvarchar(max)\",\n \"fields\": null,\n \"items\": null\n },\n {\n \"name\": \"congestion_surcharge\",\n \"type\": \"Nvarchar(max)\",\n \"fields\": null,\n \"items\": null\n },\n {\n \"name\": \"airport_fee\",\n \"type\": \"Nvarchar(max)\",\n \"fields\": null,\n \"items\": null\n }\n ]\n }\n }\n ]\n }\n ],\n \"compatibilityLevel\": \"1.0\"\n}\n" }, { "path": "sample/workspace/SampleEventstream.Eventstream/eventstreamProperties.json", "content": "{\n \"retentionTimeInDays\": 1,\n \"eventThroughputLevel\": \"Low\"\n}" }, { "path": "sample/workspace/SampleKQLQueryset.KQLQueryset/.platform", "content": "{\n \"$schema\": \"https://developer.microsoft.com/json-schemas/fabric/gitIntegration/platformProperties/2.0.0/schema.json\",\n \"metadata\": {\n \"type\": \"KQLQueryset\",\n \"displayName\": \"SampleKQLQueryset\",\n \"description\": \"Sample\"\n },\n \"config\": {\n \"version\": \"2.0\",\n \"logicalId\": \"6f343e19-fcaa-a492-4d58-6bd62a31fb94\"\n }\n}\n" }, { "path": "sample/workspace/SampleKQLQueryset.KQLQueryset/RealTimeQueryset.json", "content": "{\n \"queryset\": {\n \"version\": \"1.0.0\",\n \"dataSources\": [\n {\n \"id\": \"50c1256c-4b67-4f03-a048-9aeadd277887\",\n \"clusterUri\": \"\",\n \"type\": \"Fabric\",\n \"databaseItemId\": \"a51e98dd-5993-8e1c-443f-02aa53d4db74\",\n \"databaseItemName\": \"TaxiDB\"\n }\n ],\n \"tabs\": [\n {\n \"id\": \"472f60f7-4cdb-4cb2-a2fb-a7fce8377aa0\",\n \"content\": \"//***********************************************************************************************************\\n// Here are two articles to help you get started with KQL:\\n// KQL reference guide - https://aka.ms/KQLguide\\n// SQL - KQL conversions - https://aka.ms/sqlcheatsheet\\n//***********************************************************************************************************\\n\\n// Use \\\"take\\\" to view a sample number of records in the table and check the data.\\nTaxiRaw\\n| take 100\\n\\n// See how many records are in the table.\\nTaxiRaw\\n| count\\n\\n// This query returns the number of ingestions per hour in the given table.\\nTaxiRecords\\n| summarize IngestionCount = count() by bin(ingestion_time(), 1h)\\n\\n\",\n \"title\": \"\",\n \"dataSourceId\": \"50c1256c-4b67-4f03-a048-9aeadd277887\"\n }\n ]\n }\n}\n" }, { "path": "sample/workspace/SampleSparkJobDefinition.SparkJobDefinition/.platform", "content": "{\n \"$schema\": \"https://developer.microsoft.com/json-schemas/fabric/gitIntegration/platformProperties/2.0.0/schema.json\",\n \"metadata\": {\n \"type\": \"SparkJobDefinition\",\n \"displayName\": \"SampleSparkJobDefinition\",\n \"description\": \"Spark job definition\"\n },\n \"config\": {\n \"version\": \"2.0\",\n \"logicalId\": \"674fe1e5-9c35-8b9d-48cd-98a3fd65b0b2\"\n }\n}\n" }, { "path": "sample/workspace/SampleSparkJobDefinition.SparkJobDefinition/Libs/pipeline_config.py", "content": "config = {\n 'range_limit': 1000\n}\n" }, { "path": "sample/workspace/SampleSparkJobDefinition.SparkJobDefinition/Main/main.py", "content": "from pyspark.sql import SparkSession\nimport logging\nimport sys\n\nfrom pipeline_config import config\n\n# Configure logging\nlogging.basicConfig(\n level=logging.INFO,\n format='%(asctime)s - %(name)s - %(levelname)s - %(message)s',\n handlers=[logging.StreamHandler(sys.stdout)]\n)\n\nlogger = logging.getLogger(__name__)\n\nif __name__ == \"__main__\":\n\n # Step 1: Create Spark Session\n spark = (SparkSession\n .builder\n .appName(\"sjdsampleapp\")\n .getOrCreate())\n\n spark_context = spark.sparkContext\n spark_context.setLogLevel(\"ERROR\")\n\n logger.info(\"=\" * 80)\n logger.info(\"Starting Job\")\n logger.info(\"=\" * 80)\n\n # Step 2: Sample Data Processing\n df = spark.range(config['range_limit']).collect()\n\n logger.info(\"=\" * 80)\n logger.info(\"Completing Job\")\n logger.info(\"=\" * 80)\n" }, { "path": "sample/workspace/SampleSparkJobDefinition.SparkJobDefinition/SparkJobDefinitionV1.json", "content": "{\n \"executableFile\": \"main.py\",\n \"defaultLakehouseArtifactId\": \"eb8e6aef-81f9-894c-4eda-7be021fdfc5d\",\n \"mainClass\": \"\",\n \"additionalLakehouseIds\": [],\n \"retryPolicy\": null,\n \"commandLineArguments\": \"arg1 true\",\n \"additionalLibraryUris\": [\"pipeline_config.py\"],\n \"language\": \"Python\",\n \"environmentArtifactId\": \"a277ea4a-e87f-8537-4ce0-39db11d4aade\"\n}\n" }, { "path": "sample/workspace/SampleUserDataFunction.UserDataFunction/.platform", "content": "{\n \"$schema\": \"https://developer.microsoft.com/json-schemas/fabric/gitIntegration/platformProperties/2.0.0/schema.json\",\n \"metadata\": {\n \"type\": \"UserDataFunction\",\n \"displayName\": \"SampleUserDataFunction\"\n },\n \"config\": {\n \"version\": \"2.0\",\n \"logicalId\": \"ee3a4088-9b35-8f20-4977-ffbae687acc4\"\n }\n}\n" }, { "path": "sample/workspace/SampleUserDataFunction.UserDataFunction/.resources/functions.json", "content": "{\n \"runtime\": \"PYTHON\",\n \"functionsMetadata\": [\n {\n \"name\": \"hello_fabric\",\n \"scriptFile\": \"function_app.py\",\n \"bindings\": [\n {\n \"name\": \"req\",\n \"type\": \"HttpTrigger\",\n \"direction\": \"In\",\n \"authLevel\": \"Anonymous\",\n \"methods\": [\n \"post\"\n ],\n \"route\": \"hello_fabric\"\n }\n ],\n \"fabricProperties\": {\n \"fabricMetadataSchemaVersion\": null,\n \"fabricFunctionReturnType\": \"str\",\n \"fabricFunctionParameters\": [\n {\n \"name\": \"name\",\n \"dataType\": \"str\"\n }\n ]\n }\n }\n ]\n}" }, { "path": "sample/workspace/SampleUserDataFunction.UserDataFunction/definition.json", "content": "{\n \"$schema\": \"https://developer.microsoft.com/json-schemas/fabric/item/userDataFunction/definition/1.1.0/schema.json\",\n \"runtime\": \"PYTHON\",\n \"connectedDataSources\": [],\n \"functions\": [\n {\n \"name\": \"hello_fabric\",\n \"description\": \"\",\n \"isPublicEndpointEnabled\": true\n }\n ],\n \"libraries\": {\n \"public\": [\n {\n \"name\": \"fabric-user-data-functions\",\n \"type\": \"PYPI\",\n \"version\": \"1.0\"\n }\n ],\n \"private\": []\n }\n}" }, { "path": "sample/workspace/SampleUserDataFunction.UserDataFunction/function_app.py", "content": "import datetime\nimport fabric.functions as fn\nimport logging\n\nudf = fn.UserDataFunctions()\n\n@udf.function()\ndef hello_fabric(name: str) -> str:\n logging.info('Python UDF trigger function processed a request.')\n\n return f\"Welcome to Fabric Functions, {name}, at {datetime.datetime.now()}!\"\n" }, { "path": "sample/workspace/SourceForShortcutLH.Lakehouse/.platform", "content": "{\n \"$schema\": \"https://developer.microsoft.com/json-schemas/fabric/gitIntegration/platformProperties/2.0.0/schema.json\",\n \"metadata\": {\n \"type\": \"Lakehouse\",\n \"displayName\": \"SourceForShortcutLH\"\n },\n \"config\": {\n \"version\": \"2.0\",\n \"logicalId\": \"c0cec3c6-bff6-8a77-47b1-ab19d15a52cc\"\n }\n}" }, { "path": "sample/workspace/SourceForShortcutLH.Lakehouse/lakehouse.metadata.json", "content": "{\"defaultSchema\":\"dbo\"}" }, { "path": "sample/workspace/SourceForShortcutLH.Lakehouse/shortcuts.metadata.json", "content": "[]" }, { "path": "sample/workspace/TargetForShortcutLH.Lakehouse/.platform", "content": "{\n \"$schema\": \"https://developer.microsoft.com/json-schemas/fabric/gitIntegration/platformProperties/2.0.0/schema.json\",\n \"metadata\": {\n \"type\": \"Lakehouse\",\n \"displayName\": \"TargetForShortcutLH\"\n },\n \"config\": {\n \"version\": \"2.0\",\n \"logicalId\": \"30cba7ea-14f6-810a-4c89-6ab6f4632da6\"\n }\n}" }, { "path": "sample/workspace/TargetForShortcutLH.Lakehouse/lakehouse.metadata.json", "content": "{\"defaultSchema\":\"dbo\"}" }, { "path": "sample/workspace/TargetForShortcutLH.Lakehouse/shortcuts.metadata.json", "content": "[\n {\n \"name\": \"publicholidays\",\n \"path\": \"/Tables/dbo\",\n \"target\": {\n \"type\": \"OneLake\",\n \"oneLake\": {\n \"path\": \"Tables/dbo/publicholidays\",\n \"itemId\": \"c0cec3c6-bff6-8a77-47b1-ab19d15a52cc\",\n \"workspaceId\": \"00000000-0000-0000-0000-000000000000\",\n \"artifactType\": \"Lakehouse\"\n }\n }\n },\n {\n \"name\": \"sample_datasets\",\n \"path\": \"/Files\",\n \"target\": {\n \"type\": \"OneLake\",\n \"oneLake\": {\n \"path\": \"Files/sample_datasets\",\n \"itemId\": \"c0cec3c6-bff6-8a77-47b1-ab19d15a52cc\",\n \"workspaceId\": \"00000000-0000-0000-0000-000000000000\",\n \"artifactType\": \"Lakehouse\"\n }\n }\n },\n {\n \"name\": \"images\",\n \"path\": \"/Files\",\n \"target\": {\n \"type\": \"OneLake\",\n \"oneLake\": {\n \"path\": \"Files/images\",\n \"itemId\": \"c0cec3c6-bff6-8a77-47b1-ab19d15a52cc\",\n \"workspaceId\": \"00000000-0000-0000-0000-000000000000\",\n \"artifactType\": \"Lakehouse\"\n }\n }\n }\n]" }, { "path": "sample/workspace/TelemetryDataEH.Eventhouse/.children/TelemetryDataEH.KQLDatabase/.platform", "content": "{\n \"$schema\": \"https://developer.microsoft.com/json-schemas/fabric/gitIntegration/platformProperties/2.0.0/schema.json\",\n \"metadata\": {\n \"type\": \"KQLDatabase\",\n \"displayName\": \"TelemetryDataEH\",\n \"description\": \"TelemetryDataEH\"\n },\n \"config\": {\n \"version\": \"2.0\",\n \"logicalId\": \"65341627-8b62-85ef-49a4-65cc214b4411\"\n }\n}" }, { "path": "sample/workspace/TelemetryDataEH.Eventhouse/.children/TelemetryDataEH.KQLDatabase/DatabaseProperties.json", "content": "{\n \"databaseType\": \"ReadWrite\",\n \"parentEventhouseItemId\": \"971dc008-7a5d-a198-4021-03a663f7d3b7\",\n \"oneLakeCachingPeriod\": \"P36500D\",\n \"oneLakeStandardStoragePeriod\": \"P36500D\"\n}" }, { "path": "sample/workspace/TelemetryDataEH.Eventhouse/.children/TelemetryDataEH.KQLDatabase/DatabaseSchema.kql", "content": "// KQL script\n// Use management commands in this script to configure your database items, such as tables, functions, materialized views, and more.\n\n\n.create-merge table FreezerTelemetry (timestamp:datetime, storeId:string, freezerId:string, temperatureC:real, humidityPct:real, doorOpen:long) \n.create-or-alter table FreezerTelemetry ingestion csv mapping 'FreezerTelemetry_mapping'\n```\n[{\"Properties\":{\"Ordinal\":\"0\"},\"column\":\"timestamp\",\"datatype\":\"\"},{\"Properties\":{\"Ordinal\":\"1\"},\"column\":\"storeId\",\"datatype\":\"\"},{\"Properties\":{\"Ordinal\":\"2\"},\"column\":\"freezerId\",\"datatype\":\"\"},{\"Properties\":{\"Ordinal\":\"3\"},\"column\":\"temperatureC\",\"datatype\":\"\"},{\"Properties\":{\"Ordinal\":\"4\"},\"column\":\"humidityPct\",\"datatype\":\"\"},{\"Properties\":{\"Ordinal\":\"5\"},\"column\":\"doorOpen\",\"datatype\":\"\"}]\n```\n" }, { "path": "sample/workspace/TelemetryDataEH.Eventhouse/.platform", "content": "{\n \"$schema\": \"https://developer.microsoft.com/json-schemas/fabric/gitIntegration/platformProperties/2.0.0/schema.json\",\n \"metadata\": {\n \"type\": \"Eventhouse\",\n \"displayName\": \"TelemetryDataEH\",\n \"description\": \"TelemetryDataEH\"\n },\n \"config\": {\n \"version\": \"2.0\",\n \"logicalId\": \"971dc008-7a5d-a198-4021-03a663f7d3b7\"\n }\n}" }, { "path": "sample/workspace/TelemetryDataEH.Eventhouse/EventhouseProperties.json", "content": "{}" }, { "path": "sample/workspace/Vars.VariableLibrary/.platform", "content": "{\n \"$schema\": \"https://developer.microsoft.com/json-schemas/fabric/gitIntegration/platformProperties/2.0.0/schema.json\",\n \"metadata\": {\n \"type\": \"VariableLibrary\",\n \"displayName\": \"Vars\",\n \"description\": \"\"\n },\n \"config\": {\n \"version\": \"2.0\",\n \"logicalId\": \"b2380160-e03e-a112-414f-13442f7f96af\"\n }\n}" }, { "path": "sample/workspace/Vars.VariableLibrary/settings.json", "content": "{\n \"$schema\": \"https://developer.microsoft.com/json-schemas/fabric/item/variableLibrary/definition/settings/1.0.0/schema.json\",\n \"valueSetsOrder\": [\n \"PPE\",\n \"PROD\"\n ]\n}\n" }, { "path": "sample/workspace/Vars.VariableLibrary/valueSets/PPE.json", "content": "{\n \"$schema\": \"https://developer.microsoft.com/json-schemas/fabric/item/variableLibrary/definition/valueSet/1.0.0/schema.json\",\n \"name\": \"PPE\",\n \"variableOverrides\": []\n}\n" }, { "path": "sample/workspace/Vars.VariableLibrary/valueSets/PROD.json", "content": "{\n \"$schema\": \"https://developer.microsoft.com/json-schemas/fabric/item/variableLibrary/definition/valueSet/1.0.0/schema.json\",\n \"name\": \"PROD\",\n \"variableOverrides\": [\n {\n \"name\": \"Environment\",\n \"value\": \"Prod\"\n },\n {\n \"name\": \"SQL_Server\",\n \"value\": \"contoso-prod.database.windows.net\"\n }\n ]\n}\n" }, { "path": "sample/workspace/Vars.VariableLibrary/variables.json", "content": "{\n \"$schema\": \"https://developer.microsoft.com/json-schemas/fabric/item/variableLibrary/definition/variables/1.0.0/schema.json\",\n \"variables\": [\n {\n \"name\": \"Environment\",\n \"note\": \"\",\n \"type\": \"String\",\n \"value\": \"PPE\"\n },\n {\n \"name\": \"SQL_Server\",\n \"note\": \"\",\n \"type\": \"String\",\n \"value\": \"contoso-ppe.database.windows.net\"\n }\n ]\n}\n" }, { "path": "sample/workspace/WithSchema.Lakehouse/.platform", "content": "{\n \"$schema\": \"https://developer.microsoft.com/json-schemas/fabric/gitIntegration/platformProperties/2.0.0/schema.json\",\n \"metadata\": {\n \"type\": \"Lakehouse\",\n \"displayName\": \"WithSchema\"\n },\n \"config\": {\n \"version\": \"2.0\",\n \"logicalId\": \"eb8e6aef-81f9-894c-4eda-7be021fdfc5d\"\n }\n}\n" }, { "path": "sample/workspace/WithSchema.Lakehouse/lakehouse.metadata.json", "content": "{\"defaultSchema\":\"dbo\"}" }, { "path": "sample/workspace/WithSchema.Lakehouse/shortcuts.metadata.json", "content": "[\n {\n \"name\": \"Testing\",\n \"path\": \"/Files\",\n \"target\": {\n \"type\": \"OneLake\",\n \"oneLake\": {\n \"path\": \"Files/Testing\",\n \"itemId\": \"c916eeb0-dd6a-ae32-4f4f-966d2414b239\",\n \"workspaceId\": \"00000000-0000-0000-0000-000000000000\",\n \"artifactType\": \"Lakehouse\"\n }\n }\n }\n]\n" }, { "path": "sample/workspace/WithoutSchema.Lakehouse/.platform", "content": "{\n \"$schema\": \"https://developer.microsoft.com/json-schemas/fabric/gitIntegration/platformProperties/2.0.0/schema.json\",\n \"metadata\": {\n \"type\": \"Lakehouse\",\n \"displayName\": \"WithoutSchema\"\n },\n \"config\": {\n \"version\": \"2.0\",\n \"logicalId\": \"c916eeb0-dd6a-ae32-4f4f-966d2414b239\"\n }\n}\n" }, { "path": "sample/workspace/WithoutSchema.Lakehouse/lakehouse.metadata.json", "content": "{}" }, { "path": "sample/workspace/WithoutSchema.Lakehouse/shortcuts.metadata.json", "content": "[]" }, { "path": "sample/workspace/World.Environment/.platform", "content": "{\n \"$schema\": \"https://developer.microsoft.com/json-schemas/fabric/gitIntegration/platformProperties/2.0.0/schema.json\",\n \"metadata\": {\n \"type\": \"Environment\",\n \"displayName\": \"World\",\n \"description\": \"Environment\"\n },\n \"config\": {\n \"version\": \"2.0\",\n \"logicalId\": \"a277ea4a-e87f-8537-4ce0-39db11d4aade\"\n }\n}" }, { "path": "sample/workspace/World.Environment/Libraries/PublicLibraries/environment.yml", "content": "dependencies:\n - pip:\n - fuzzywuzzy==0.18.0\n - python-levenshtein==0.26.0\n" }, { "path": "sample/workspace/World.Environment/Setting/Sparkcompute.yml", "content": "enable_native_execution_engine: false\ndriver_cores: 8\ndriver_memory: 56g\nexecutor_cores: 8\nexecutor_memory: 56g\ndynamic_executor_allocation:\n enabled: true\n min_executors: 1\n max_executors: 9\nruntime_version: 1.2\n" }, { "path": "sample/workspace/cicd_experiment.MLExperiment/.platform", "content": "{\n \"$schema\": \"https://developer.microsoft.com/json-schemas/fabric/gitIntegration/platformProperties/2.0.0/schema.json\",\n \"metadata\": {\n \"type\": \"MLExperiment\",\n \"displayName\": \"cicd_experiment\"\n },\n \"config\": {\n \"version\": \"2.0\",\n \"logicalId\": \"ab6b3e5c-9425-b94c-40be-0ed409f844eb\"\n }\n}\n" }, { "path": "sample/workspace/cicd_experiment.MLExperiment/mlexperiment.metadata.json", "content": "{\"dependencies\":[]}" }, { "path": "sample/workspace/config.yml", "content": "# Sample configuration file for fabric-cicd deployment\n# This file demonstrates the YAML configuration structure for simplified deployment workflow\n\ncore: # Core configurations\n # Either workspace or workspace_id must be provided\n workspace: # Workspace names by environment\n dev: Fabric-Dev-Engineering\n test: Fabric-Test-Engineering\n prod: Fabric-Prod-Engineering\n\n workspace_id: # Workspace IDs by environment (takes precedence over workspace if both are provided)\n dev: 8b6e2c7a-4c1f-4e3a-9b2e-7d8f2e1a6c3b\n test: 2f4b9e8d-1a7c-4d3e-b8e2-5c9f7a2d4e1b\n prod: 7c3e1f8b-2d4a-4b9e-8f2c-1a6c3b7d8e2f\n\n repository_directory: \".\" # Path to workspace items directory (relative to config.yml location) (required)\n\n item_types_in_scope: # Item types to include in deployment (optional)\n - VariableLibrary\n - Dataflow\n - DataPipeline\n - Notebook\n - Environment\n\n parameter: \"parameter.yml\" # Path to parameter file (relative to config.yml location) (optional)\n\npublish: # Publish configuration (optional)\n exclude_regex: \"^DONT_DEPLOY.*\" # Regex pattern to exclude items from publishing\n\n # folder_exclude_regex: \"^/DONT_DEPLOY_FOLDER\" # Regex pattern to exclude folder paths with items from publishing (requires feature flags)\n \n # folder_path_to_include: # Optional list of specific folder paths with items to publish (requires feature flags)\n # - \"/subfolderA\"\n # - \"/subfolderA/subfolderB\"\n\n # items_to_include: # Optional list of specific items to publish (requires feature flags)\n # - \"Hello World.Notebook\"\n # - \"Run Hello World.DataPipeline\"\n \n # shortcut_exclude_regex: \"^DONT_DEPLOY_SHORTCUT.*\" # Regex pattern to exclude Lakehouse shortcuts from publishing (requires feature flags)\n\n skip: # Skip publishing for specific environments\n dev: true # Skip publishing in dev environment\n test: false # Enable publishing in test environment\n prod: false # Enable publishing in prod environment\n\nunpublish: # Unpublish configuration (optional)\n exclude_regex: \"^DEBUG.*\" # Regex pattern to exclude items from unpublishing\n # items_to_include: # Optional list of specific items to unpublish (requires feature flags)\n\n skip: # Skip unpublishing for specific environments\n dev: true # Skip unpublishing in dev environment\n test: false # Enable unpublishing in test environment\n prod: false # Enable unpublishing in prod environment\n\nfeatures: # Feature flags to enable (optional)\n - enable_shortcut_publish\n\nconstants: # Global constants to override (optional)\n DEFAULT_API_ROOT_URL: \"https://msitapi.fabric.microsoft.com\"\n" }, { "path": "sample/workspace/parameter template.yml", "content": "# Add template parameter files\nextend:\n - \"./templates/nb parameter template 1.yml\"\n - \"./templates/nb parameter template 2.yml\"\n\nfind_replace:\n # Lakehouse Connection Guid\n - find_value: \"db52be81-c2b2-4261-84fa-840c67f4bbd0\"\n replace_value:\n PPE: \"81bbb339-8d0b-46e8-bfa6-289a159c0733\"\n PROD: \"5d6a1b16-447f-464a-b959-45d0fed35ca0\"\n # Optional fields:\n item_type: \"Notebook\"\n item_name: [\"Hello World\", \"Hello World Subfolder\"]\n file_path:\n - \"/Hello World.Notebook/notebook-content.py\"\n - \"/subfolder/Hello World Subfolder.Notebook/notebook-content.py\"\n" }, { "path": "sample/workspace/parameter.yml", "content": "find_replace:\n # Lakehouse Connection Guid\n - find_value: \"db52be81-c2b2-4261-84fa-840c67f4bbd0\"\n replace_value:\n PPE: \"81bbb339-8d0b-46e8-bfa6-289a159c0733\"\n PROD: \"5d6a1b16-447f-464a-b959-45d0fed35ca0\"\n # Optional fields:\n item_type: \"Notebook\"\n item_name: [\"Hello World\", \"Hello World Subfolder\"]\n file_path:\n - \"/Hello World.Notebook/notebook-content.py\"\n - \"/subfolder/Hello World Subfolder.Notebook/notebook-content.py\"\n # Lakehouse Connection Guid regex\n - find_value: \\#\\s*META\\s+\"default_lakehouse\":\\s*\"([0-9a-fA-F]{8}-[0-9a-fA-F]{4}-[0-9a-fA-F]{4}-[0-9a-fA-F]{4}-[0-9a-fA-F]{12})\"\n replace_value:\n # Variable: $items.type.name.attribute (Note: item type and name values are CASE SENSITIVE; id attribute returns the deployed item's id/guid)\n PPE: \"$items.Lakehouse.WithoutSchema.id\" \n PROD: \"$items.Lakehouse.WithoutSchema.id\" \n # Optional fields:\n is_regex: \"true\"\n file_path: \"/Example Notebook.Notebook/notebook-content.py\"\n # Lakehouse workspace id regex\n - find_value: \\#\\s*META\\s+\"default_lakehouse_workspace_id\":\\s*\"([0-9a-fA-F]{8}-[0-9a-fA-F]{4}-[0-9a-fA-F]{4}-[0-9a-fA-F]{4}-[0-9a-fA-F]{12})\"\n replace_value:\n # Variable: $workspace.id -> target workspace id\n PPE: \"$workspace.id\"\n PROD: \"$workspace.id\"\n # Optional fields:\n is_regex: \"true\"\n file_path: \"/Example Notebook.Notebook/notebook-content.py\"\n # SQL connection string\n - find_value: \"sqlserverconnectionstringinoriginlakehouse.com\"\n replace_value:\n # Variable: $items.type.name.attribute (Note: item type and name values are CASE SENSITIVE; sqlendpoint attribute returns the deployed item's sql endpoint)\n PPE: \"$items.Lakehouse.WithoutSchema.sqlendpoint\" \n PROD: \"$items.Lakehouse.WithoutSchema.sqlendpoint\" \n # Optional fields:\n file_path: \"/Example Notebook.Notebook/notebook-content.py\"\n # Eventhouse query URI\n - find_value: \"https://trd-origineventhouse.z4.kusto.fabric.microsoft.com\"\n replace_value:\n # Variable: $items.type.name.attribute (Note: item type and name values are CASE SENSITIVE; queryserviceuri attribute returns the deployed item's query service URI)\n PPE: \"$items.Eventhouse.SampleEventhouse.queryserviceuri\" \n PROD: \"$items.Eventhouse.SampleEventhouse.queryserviceuri\" \n # Optional fields:\n file_path: \"/Example Notebook.Notebook/notebook-content.py\"\n # Report byConnection - workspace id in connection string\n - find_value: \"dev-workspace-id\"\n replace_value:\n PPE: \"$workspace.$id\"\n PROD: \"$workspace.$id\"\n # Optional fields:\n file_path: \"/ByConnection.Report/definition.pbir\"\n # Report byConnection - semantic model name in connection string\n - find_value: \"dev-semantic-model\"\n replace_value:\n PPE: \"ABC\"\n PROD: \"ABC\"\n # Optional fields:\n file_path: \"/ByConnection.Report/definition.pbir\"\n # Report byConnection - semantic model id in connection string using dynamic replacement\n - find_value: \"00000000-0000-0000-0000-000000000000\"\n replace_value:\n # Variable: $items.type.name.attribute (Note: item type and name values are CASE SENSITIVE; id attribute returns the deployed item's id/guid)\n PPE: \"$items.SemanticModel.ABC.$id\"\n PROD: \"$items.SemanticModel.ABC.$id\"\n # Optional fields:\n file_path: \"/ByConnection.Report/definition.pbir\"\n\nkey_value_replace:\n - find_key: $.variables[?(@.name==\"SQL_Server\")].value\n replace_value:\n PPE: \"contoso-ppe.database.windows.net\"\n PROD: \"contoso-prod.database.windows.net\"\n UAT: \"contoso-uat.database.windows.net\"\n # Optional fields:\n item_type: \"VariableLibrary\"\n item_name: \"Vars\"\n - find_key: $.variables[?(@.name==\"Environment\")].value\n replace_value:\n PPE: \"PPE\"\n PROD: \"PROD\"\n UAT: \"UAT\"\n # Optional fields:\n item_type: \"VariableLibrary\"\n item_name: \"Vars\"\n - find_key: $.variableOverrides[?(@.name==\"SQL_Server\")].value\n replace_value:\n PROD: \"contoso-production-override.database.windows.net\"\n file_path: Vars.VariableLibrary/valueSets/PROD.json\n item_type: \"VariableLibrary\"\n item_name: \"Vars\"\n - find_key: $.variableOverrides[?(@.name==\"Environment\")].value\n replace_value:\n PROD: \"PROD_ENV\"\n file_path: Vars.VariableLibrary/valueSets/PROD.json\n item_type: \"VariableLibrary\"\n item_name: \"Vars\"\n - find_key: $.schedules[?(@.jobType==\"Execute\")].enabled\n replace_value:\n PPE: false\n PROD: true\n file_path: \"**/.schedules\"\n\nspark_pool:\n # CapacityPool_Large\n - instance_pool_id: \"72c68dbc-0775-4d59-909d-a47896f4573b\"\n replace_value:\n PPE: \n type: \"Capacity\"\n name: \"CapacityPool_Large_PPE\"\n PROD: \n type: \"Capacity\"\n name: \"CapacityPool_Large_PROD\"\n # Optional field:\n item_name: \"World\"\n # CapacityPool_Medium\n - instance_pool_id: \"e7b8f1c4-4a6e-4b8b-9b2e-8f1e5d6a9c3d\"\n replace_value:\n PPE:\n type: \"Workspace\"\n name: \"WorkspacePool_Medium\"\n PROD:\n type: \"Workspace\"\n name: \"WorkspacePool_Medium\"\n # Optional field:\n item_name:\n \n# Supports default connections and per-item overrides (single connection only per model)\nsemantic_model_binding:\n default:\n connection_id:\n PPE: \"76e05dfe-9855-4e3d-a410-1dda048dbe99\"\n PROD: \"c4f8e2b1-3d2a-4f5b-9c6e-7a8b9c0d1e2f\"\n models:\n - semantic_model_name: [\"cloudconnections\", \"MySemanticModel_ADLS_Gen2\"]\n connection_id:\n PPE: \"f96870d5-5f86-49ad-bf41-5967fd7c1c6d\"\n PROD: \"a1b2c3d4-5678-90ab-cdef-1234567890ab\"\n" }, { "path": "sample/workspace/sample apache airflow job.ApacheAirflowJob/.platform", "content": "{\n \"$schema\": \"https://developer.microsoft.com/json-schemas/fabric/gitIntegration/platformProperties/2.0.0/schema.json\",\n \"metadata\": {\n \"type\": \"ApacheAirflowJob\",\n \"displayName\": \"sample apache airflow job\"\n },\n \"config\": {\n \"version\": \"2.0\",\n \"logicalId\": \"f31cd938-c3ab-b977-4dd4-3e4a56d7c124\"\n }\n}\n" }, { "path": "sample/workspace/sample apache airflow job.ApacheAirflowJob/apacheairflowjob-content.json", "content": "{\n \"properties\": {\n \"type\": \"Airflow\",\n \"typeProperties\": {\n \"airflowProperties\": {\n \"airflowConfigurationOverrides\": {},\n \"airflowEnvironment\": \"FabricAirflowJob-1.0.0\",\n \"airflowRequirements\": [],\n \"airflowVersion\": \"2.10.5\",\n \"enableAADIntegration\": true,\n \"enableTriggerers\": false,\n \"environmentVariables\": {},\n \"packageProviderPath\": \"plugins\",\n \"pythonVersion\": \"3.12\"\n },\n \"computeProperties\": {\n \"computePool\": \"StarterPool\",\n \"computeSize\": \"Small\",\n \"enableAutoscale\": false,\n \"enableAvailabilityZones\": false,\n \"extraNodes\": 0\n }\n }\n }\n}" }, { "path": "sample/workspace/sample apache airflow job.ApacheAirflowJob/dags/dag1.py", "content": "from datetime import datetime\nfrom airflow import DAG\nfrom airflow.operators.bash import BashOperator\n\n# Define the default arguments for the DAG\ndefault_args = {\n 'owner': 'airflow',\n 'depends_on_past': False,\n 'start_date': datetime(2023, 5, 1),\n 'email_on_failure': False,\n 'email_on_retry': False,\n 'retries': 1\n}\n\n# Instantiate the DAG object\nwith DAG(\n 'dags-dag1.py',\n default_args=default_args,\n description='A simple Hello World DAG',\n schedule_interval=None,\n catchup=False\n) as dag:\n\n # Define the tasks\n hello_task = BashOperator(\n task_id='hello_world_task',\n bash_command='echo \"Hello, World!\"'\n )\n\n # Set the task dependencies\n hello_task" }, { "path": "sample/workspace/subfolder/Hello World Subfolder.Notebook/.platform", "content": "{\n \"$schema\": \"https://developer.microsoft.com/json-schemas/fabric/gitIntegration/platformProperties/2.0.0/schema.json\",\n \"metadata\": {\n \"type\": \"Notebook\",\n \"displayName\": \"Hello World Subfolder\",\n \"description\": \"New notebook\"\n },\n \"config\": {\n \"version\": \"2.0\",\n \"logicalId\": \"2d147568-6f37-9881-461d-c07a95bcb35d\"\n }\n}" }, { "path": "sample/workspace/subfolder/Hello World Subfolder.Notebook/notebook-content.py", "content": "# Fabric notebook source\n\n# METADATA ********************\n\n# META {\n# META \"kernel_info\": {\n# META \"name\": \"synapse_pyspark\"\n# META },\n# META \"dependencies\": {\n# META \"environment\": {\n# META \"environmentId\": \"a277ea4a-e87f-8537-4ce0-39db11d4aade\",\n# META \"workspaceId\": \"00000000-0000-0000-0000-000000000000\"\n# META }\n# META }\n# META }\n\n# CELL ********************\n\nprint(\"Hello World\")\n\n# METADATA ********************\n\n# META {\n# META \"language\": \"python\",\n# META \"language_group\": \"synapse_pyspark\"\n# META }\n" }, { "path": "sample/workspace/subfolder/subfolder/Hello World SubfolderSubfolder.Notebook/.platform", "content": "{\n \"$schema\": \"https://developer.microsoft.com/json-schemas/fabric/gitIntegration/platformProperties/2.0.0/schema.json\",\n \"metadata\": {\n \"type\": \"Notebook\",\n \"displayName\": \"Hello World SubfolderSubfolder\",\n \"description\": \"New notebook\"\n },\n \"config\": {\n \"version\": \"2.0\",\n \"logicalId\": \"66dd4a2e-45b9-aff1-418c-695af0f1885a\"\n }\n}" }, { "path": "sample/workspace/subfolder/subfolder/Hello World SubfolderSubfolder.Notebook/notebook-content.py", "content": "# Fabric notebook source\n\n# METADATA ********************\n\n# META {\n# META \"kernel_info\": {\n# META \"name\": \"synapse_pyspark\"\n# META },\n# META \"dependencies\": {\n# META \"environment\": {\n# META \"environmentId\": \"a277ea4a-e87f-8537-4ce0-39db11d4aade\",\n# META \"workspaceId\": \"00000000-0000-0000-0000-000000000000\"\n# META }\n# META }\n# META }\n\n# CELL ********************\n\nprint(\"Hello World\")\n\n# METADATA ********************\n\n# META {\n# META \"language\": \"python\",\n# META \"language_group\": \"synapse_pyspark\"\n# META }\n" }, { "path": "sample/workspace/templates/nb parameter template 1.yml", "content": "find_replace:\n # SQL connection string\n - find_value: \"sqlserverconnectionstringinoriginlakehouse.com\"\n replace_value:\n # Variable: $items.type.name.attribute (Note: item type and name values are CASE SENSITIVE; sqlendpoint attribute returns the deployed item's sql endpoint)\n PPE: \"$items.Lakehouse.WithoutSchema.sqlendpoint\" \n PROD: \"$items.Lakehouse.WithoutSchema.sqlendpoint\" \n # Optional fields:\n file_path: \"/Example Notebook.Notebook/notebook-content.py\"\n # Eventhouse query URI\n - find_value: \"https://trd-origineventhouse.z4.kusto.fabric.microsoft.com\"\n replace_value:\n # Variable: $items.type.name.attribute (Note: item type and name values are CASE SENSITIVE; queryserviceuri attribute returns the deployed item's query service URI)\n PPE: \"$items.Eventhouse.SampleEventhouse.queryserviceuri\" \n PROD: \"$items.Eventhouse.SampleEventhouse.queryserviceuri\" \n # Optional fields:\n file_path: \"/Example Notebook.Notebook/notebook-content.py\"\n" }, { "path": "sample/workspace/templates/nb parameter template 2.yml", "content": "find_replace:\n # Lakehouse Connection Guid regex\n - find_value: \\#\\s*META\\s+\"default_lakehouse\":\\s*\"([0-9a-fA-F]{8}-[0-9a-fA-F]{4}-[0-9a-fA-F]{4}-[0-9a-fA-F]{4}-[0-9a-fA-F]{12})\"\n replace_value:\n # Variable: $items.type.name.attribute (Note: item type and name values are CASE SENSITIVE; id attribute returns the deployed item's id/guid)\n PPE: \"$items.Lakehouse.WithoutSchema.id\" \n PROD: \"$items.Lakehouse.WithoutSchema.id\" \n # Optional fields:\n is_regex: \"true\"\n file_path: \"/Example Notebook.Notebook/notebook-content.py\"\n # Lakehouse workspace id regex\n - find_value: \\#\\s*META\\s+\"default_lakehouse_workspace_id\":\\s*\"([0-9a-fA-F]{8}-[0-9a-fA-F]{4}-[0-9a-fA-F]{4}-[0-9a-fA-F]{4}-[0-9a-fA-F]{12})\"\n replace_value:\n # Variable: $workspace.id -> target workspace id\n PPE: \"$workspace.id\"\n PROD: \"$workspace.id\"\n # Optional fields:\n is_regex: \"true\"\n file_path: \"/Example Notebook.Notebook/notebook-content.py\"\n" }, { "path": "src/fabric_cicd/__init__.py", "content": "# Copyright (c) Microsoft Corporation.\n# Licensed under the MIT License.\n\n\"\"\"Provides tools for managing and publishing items in a Fabric workspace.\"\"\"\n\nimport logging\nimport sys\n\nimport fabric_cicd.constants as constants\nfrom fabric_cicd._common._deployment_result import DeploymentResult, DeploymentStatus\nfrom fabric_cicd._common._git_diff_utils import get_changed_items\nfrom fabric_cicd._common._logging import configure_logger, exception_handler, get_file_handler\nfrom fabric_cicd._common._validate_env_vars import _get_fabric_fqdn_url, validate_api_url\nfrom fabric_cicd._common._validate_input import validate_workspace_id\nfrom fabric_cicd.constants import FeatureFlag, ItemType\nfrom fabric_cicd.fabric_workspace import FabricWorkspace\nfrom fabric_cicd.publish import deploy_with_config, publish_all_items, unpublish_all_orphan_items\n\nlogger = logging.getLogger(__name__)\n\n\ndef append_feature_flag(feature: str) -> None:\n \"\"\"\n Append a feature flag to the global feature_flag set.\n\n Args:\n feature: The feature flag to be included.\n\n Examples:\n Basic usage\n >>> from fabric_cicd import append_feature_flag\n >>> append_feature_flag(\"enable_lakehouse_unpublish\")\n \"\"\"\n constants.FEATURE_FLAG.add(feature)\n\n\ndef change_log_level(level: str = \"DEBUG\") -> None:\n \"\"\"\n Sets the log level for all loggers within the fabric_cicd package. Currently only supports DEBUG.\n\n Args:\n level: The logging level to set (e.g., DEBUG).\n\n Examples:\n Basic usage\n >>> from fabric_cicd import change_log_level\n >>> change_log_level(\"DEBUG\")\n \"\"\"\n if level.upper() == \"DEBUG\":\n configure_logger(logging.DEBUG)\n logger.info(\"Changed log level to DEBUG\")\n else:\n logger.warning(f\"Log level '{level}' not supported. Only DEBUG is supported at this time. No changes made.\")\n\n\ndef configure_external_file_logging(external_logger: logging.Logger) -> None:\n \"\"\"\n Configure fabric_cicd package logging to integrate with an external logger's\n file handler. This is an advanced alternative to the default file logging\n configuration when level is set to DEBUG via `change_log_level()`.\n\n Extracts the file handler from the provided logger and configures fabric_cicd\n to append only DEBUG logs (e.g., API request/response details) to the same file.\n The external logger retains full ownership of the handler, including file\n rotation (if applicable) and lifecycle management.\n\n Note:\n - This function resets logging configuration. Use as an alternative to\n ``change_log_level()`` or ``disable_file_logging()``, not in combination\n\n - Only DEBUG logs from the fabric_cicd package are written to the log file.\n Exception messages are displayed on the console, but full stack traces\n are not written to the external log file\n\n - Console output remains at INFO level (default fabric_cicd console behavior)\n\n Args:\n external_logger: The external logger instance that has\n a `FileHandler` or `RotatingFileHandler` attached.\n\n Raises:\n ValueError: If no file handler is found on the provided logger.\n\n Examples:\n General usage:\n >>> import logging\n >>> from logging.handlers import RotatingFileHandler\n >>> from fabric_cicd import configure_external_file_logging\n ...\n >>> # Set up your own logger with a file handler\n >>> my_logger = logging.getLogger(\"MyApp\")\n >>> handler = RotatingFileHandler(\"app.log\", maxBytes=5*1024*1024, backupCount=7)\n >>> handler.setFormatter(logging.Formatter(\"%(asctime)s - %(levelname)s - %(message)s\"))\n >>> my_logger.addHandler(handler)\n ...\n >>> # Configure fabric_cicd to use the same file\n >>> configure_external_file_logging(my_logger)\n \"\"\"\n # Extract file handler from external logger\n file_handler = get_file_handler(external_logger)\n if file_handler is None:\n msg = \"No FileHandler or RotatingFileHandler found on the provided logger.\"\n raise ValueError(msg)\n\n configure_logger(\n level=logging.DEBUG,\n suppress_debug_console=True,\n debug_only_file=True,\n external_file_handler=file_handler,\n )\n\n\ndef disable_file_logging() -> None:\n \"\"\"\n Disable file logging for the fabric_cicd package.\n\n When called, no log file will be created and only console logging will occur\n at the default INFO level.\n\n Note:\n - This function is intended to be used as an alternative to\n `change_log_level()` or `configure_external_file_logging()`, not in\n combination with them as this will reset logging configurations\n to INFO-level console output only.\n - Exception messages will still be displayed on the console, but full\n stack traces will not be written to any log file or console.\n\n Examples:\n Basic usage\n >>> from fabric_cicd import disable_file_logging\n >>> disable_file_logging()\n \"\"\"\n configure_logger(disable_log_file=True)\n\n\ndef configure_fabric_fqdn(workspace_id: str) -> None:\n \"\"\"\n Configure Fabric API URLs for private-link-enabled workspaces.\n\n Updates the global Fabric API URL constants to use the FQDN format required\n for private-link-enabled workspaces. Call this function before initializing\n a FabricWorkspace if you are using a private-link-enabled workspace.\n\n Args:\n workspace_id: The workspace ID string in standard GUID format with dashes\n (e.g., \"f953f3da-c5f0-4e36-a644-c85933e35e2f\").\n\n Side Effects:\n Updates the module-level constants in fabric_cicd.constants:\n - FABRIC_API_ROOT_URL: Set to the FQDN URL derived from workspace_id\n - DEFAULT_API_ROOT_URL: Set to the same FQDN URL\n\n Examples:\n Basic usage with FabricWorkspace:\n >>> from fabric_cicd import configure_fabric_fqdn, FabricWorkspace\n >>> from azure.identity import AzureCliCredential\n >>>\n >>> workspace_id = \"f953f3da-c5f0-4e36-a644-c85933e35e2f\"\n >>> configure_fabric_fqdn(workspace_id)\n >>>\n >>> token_credential = AzureCliCredential()\n >>> workspace = FabricWorkspace(\n ... workspace_id=workspace_id,\n ... repository_directory=\"/path/to/workspace\",\n ... token_credential=token_credential\n ... )\n \"\"\"\n workspace_id = validate_workspace_id(workspace_id)\n fqdn_url = _get_fabric_fqdn_url(workspace_id)\n fqdn_url = validate_api_url(fqdn_url, \"configure_fabric_fqdn\")\n\n if constants.FABRIC_API_ROOT_URL != \"https://api.fabric.microsoft.com\":\n logger.warning(\n f\"configure_fabric_fqdn: overwriting previously set FABRIC_API_ROOT_URL '{constants.FABRIC_API_ROOT_URL}'\"\n )\n\n constants.FABRIC_API_ROOT_URL = fqdn_url\n constants.DEFAULT_API_ROOT_URL = fqdn_url\n\n\nconfigure_logger()\nsys.excepthook = exception_handler\n\n__all__ = [\n \"DeploymentResult\",\n \"DeploymentStatus\",\n \"FabricWorkspace\",\n \"FeatureFlag\",\n \"ItemType\",\n \"append_feature_flag\",\n \"change_log_level\",\n \"configure_external_file_logging\",\n \"configure_fabric_fqdn\",\n \"deploy_with_config\",\n \"disable_file_logging\",\n \"get_changed_items\",\n \"publish_all_items\",\n \"unpublish_all_orphan_items\",\n]\n" }, { "path": "src/fabric_cicd/_common/__init__.py", "content": "# Copyright (c) Microsoft Corporation.\n# Licensed under the MIT License.\n\n" }, { "path": "src/fabric_cicd/_common/_check_utils.py", "content": "# Copyright (c) Microsoft Corporation.\n# Licensed under the MIT License.\n\n\"\"\"Utility functions for checking file types and versions.\"\"\"\n\nimport json\nimport logging\nimport re\nfrom pathlib import Path\n\nimport filetype\nimport yaml\n\nfrom fabric_cicd._common._exceptions import FileTypeError\n\nlogger = logging.getLogger(__name__)\n\n\ndef check_file_type(file_path: Path) -> str:\n \"\"\"\n Check the type of the provided file.\n\n Args:\n file_path: The path to the file.\n \"\"\"\n try:\n kind = filetype.guess(file_path)\n except Exception as e:\n msg = f\"Error determining file type of {file_path}: {e}\"\n FileTypeError(msg, logger)\n\n if kind is not None:\n if kind.mime.startswith(\"application/\"):\n return \"binary\"\n if kind.mime.startswith(\"image/\"):\n return \"image\"\n return \"text\"\n\n\ndef check_regex(regex: str) -> re.Pattern:\n \"\"\"\n Check if a regex pattern is valid and returns the pattern.\n\n Args:\n regex: The regex pattern to match.\n \"\"\"\n try:\n regex_pattern = re.compile(regex)\n except Exception as e:\n msg = f\"An error occurred with the regex provided: {e}\"\n raise ValueError(msg) from e\n return regex_pattern\n\n\ndef check_valid_json_content(content: str) -> bool:\n \"\"\"\n Check if the given string content is valid JSON.\n\n Args:\n content: The string content to validate as JSON.\n\n Returns:\n bool: True if the content is valid JSON, False otherwise.\n \"\"\"\n try:\n json.loads(content)\n return True\n except json.JSONDecodeError:\n return False\n\n\ndef check_valid_yaml_content(content: str) -> bool:\n \"\"\"\n Check if the given string content is valid structured YAML (mapping or sequence).\n\n Args:\n content: The string content to validate as YAML.\n\n Returns:\n bool: True if the content parses as a YAML mapping or sequence, False otherwise.\n \"\"\"\n try:\n result = yaml.safe_load(content)\n return isinstance(result, (dict, list))\n except yaml.YAMLError:\n return False\n" }, { "path": "src/fabric_cicd/_common/_color.py", "content": "class Fore:\n BLACK = \"\\033[30m\"\n RED = \"\\033[31m\"\n GREEN = \"\\033[32m\"\n YELLOW = \"\\033[33m\"\n BLUE = \"\\033[34m\"\n MAGENTA = \"\\033[35m\"\n CYAN = \"\\033[36m\"\n WHITE = \"\\033[37m\"\n RESET = \"\\033[39m\"\n\n\nclass Back:\n BLACK = \"\\033[40m\"\n RED = \"\\033[41m\"\n GREEN = \"\\033[42m\"\n YELLOW = \"\\033[43m\"\n BLUE = \"\\033[44m\"\n MAGENTA = \"\\033[45m\"\n CYAN = \"\\033[46m\"\n WHITE = \"\\033[47m\"\n RESET = \"\\033[49m\"\n\n\nclass Style:\n BRIGHT = \"\\033[1m\"\n DIM = \"\\033[2m\"\n NORMAL = \"\\033[22m\"\n RESET_ALL = \"\\033[0m\"\n" }, { "path": "src/fabric_cicd/_common/_config_utils.py", "content": "# Copyright (c) Microsoft Corporation.\n# Licensed under the MIT License.\n\n\"\"\"Utilities for YAML-based deployment configuration.\"\"\"\n\nimport contextlib\nimport logging\nfrom collections.abc import Generator\nfrom typing import Optional, Union\n\nfrom fabric_cicd import constants\nfrom fabric_cicd._common._config_validator import ConfigValidator\n\nlogger = logging.getLogger(__name__)\n\n\ndef load_config_file(config_file_path: str, environment: str, config_override: Optional[dict] = None) -> dict:\n \"\"\"Load and validate YAML configuration file.\n\n Args:\n config_file_path: Path to the YAML config file\n environment: Target environment for deployment\n config_override: Optional dictionary to override specific configuration values\n\n Returns:\n Parsed and validated configuration dictionary\n \"\"\"\n validator = ConfigValidator()\n return validator.validate_config_file(config_file_path, environment, config_override)\n\n\ndef get_config_value(config_section: dict, key: str, environment: str) -> Optional[Union[str, list, bool]]:\n \"\"\"Extract a value from config, handling both single and environment-specific formats.\n\n Args:\n config_section: The config section to extract from\n key: The key to extract\n environment: Target environment\n\n Returns:\n The extracted value, or None if key doesn't exist or environment not found in dict\n \"\"\"\n if key not in config_section:\n return None\n\n value = config_section[key]\n\n if isinstance(value, dict):\n return value.get(environment)\n\n return value\n\n\ndef update_setting(\n settings: dict,\n config: dict,\n key: str,\n environment: str,\n default_value: Optional[str] = None,\n output_key: Optional[str] = None,\n) -> None:\n \"\"\"\n Gets a config value using get_config_value and updates the settings dictionary\n if the value is not None.\n\n Args:\n settings: The settings dictionary to update\n config: The configuration dictionary\n key: The key to extract from the config\n environment: Target environment\n default_value: The default value to set if the config value is None\n output_key: The key to use in the settings dictionary (defaults to `key` if None)\n \"\"\"\n value = get_config_value(config, key, environment)\n target_key = output_key or key\n if value is not None:\n settings[target_key] = value\n elif default_value is not None:\n settings[target_key] = default_value\n\n\ndef extract_workspace_settings(config: dict, environment: str) -> dict:\n \"\"\"Extract workspace-specific settings from config for the given environment.\"\"\"\n environment = environment.strip()\n core = config[\"core\"]\n settings = {}\n\n # Workspace ID or name - required, validation ensures value exists for target environment\n if \"workspace_id\" in core:\n settings[\"workspace_id\"] = get_config_value(core, \"workspace_id\", environment)\n logger.info(f\"Using workspace ID '{settings['workspace_id']}'\")\n elif \"workspace\" in core:\n settings[\"workspace_name\"] = get_config_value(core, \"workspace\", environment)\n logger.info(f\"Using workspace '{settings['workspace_name']}'\")\n\n # Repository directory - required, validation ensures value exists for target environment\n if \"repository_directory\" in core:\n settings[\"repository_directory\"] = get_config_value(core, \"repository_directory\", environment)\n\n # Optional settings - validation logs warning if value not found for target environment\n update_setting(settings, core, \"item_types_in_scope\", environment)\n update_setting(settings, core, \"parameter\", environment, output_key=\"parameter_file_path\")\n\n return settings\n\n\ndef extract_publish_settings(config: dict, environment: str) -> dict:\n \"\"\"Extract publish-specific settings from config for the given environment.\"\"\"\n settings = {}\n\n if \"publish\" in config:\n publish_config = config[\"publish\"]\n\n # Optional settings - validation logs debug if value not found for target environment\n settings_to_update = [\n \"exclude_regex\",\n \"folder_exclude_regex\",\n \"folder_path_to_include\",\n \"items_to_include\",\n \"shortcut_exclude_regex\",\n ]\n for key in settings_to_update:\n update_setting(settings, publish_config, key, environment)\n\n # Skip defaults to False if setting not found\n update_setting(settings, publish_config, \"skip\", environment, default_value=False)\n\n return settings\n\n\ndef extract_unpublish_settings(config: dict, environment: str) -> dict:\n \"\"\"Extract unpublish-specific settings from config for the given environment.\"\"\"\n settings = {}\n\n if \"unpublish\" in config:\n unpublish_config = config[\"unpublish\"]\n\n # Optional settings - validation logs debug if value not found for target environment\n settings_to_update = [\n \"exclude_regex\",\n \"items_to_include\",\n ]\n for key in settings_to_update:\n update_setting(settings, unpublish_config, key, environment)\n\n # Skip defaults to False if setting not found\n update_setting(settings, unpublish_config, \"skip\", environment, default_value=False)\n\n return settings\n\n\n@contextlib.contextmanager\ndef config_overrides_scope(config: dict, environment: str) -> Generator[None, None, None]:\n \"\"\"\n Context manager that applies feature flags and constants\n overrides from config and guarantees cleanup.\n\n Feature flags and constants are restored to their pre-call values on exit,\n ensuring no state leaks between deployments.\n\n Args:\n config: Configuration dictionary\n environment: Target environment for deployment\n \"\"\"\n # Snapshot current state before any changes\n original_feature_flags = constants.FEATURE_FLAG.copy()\n overridden_keys = {}\n\n try:\n # Set feature flags\n if \"features\" in config:\n features = config[\"features\"]\n features_list = features.get(environment, []) if isinstance(features, dict) else features\n for feature in features_list:\n constants.FEATURE_FLAG.add(feature)\n logger.info(f\"Enabled feature flag: {feature}\")\n\n # Apply constants overrides\n if \"constants\" in config:\n constants_section = config[\"constants\"]\n for key in list(constants_section.keys()):\n value = get_config_value(constants_section, key, environment)\n if value is not None and hasattr(constants, key):\n overridden_keys[key] = getattr(constants, key)\n setattr(constants, key, value)\n logger.warning(f\"Override constant {key} = {value}\")\n yield\n\n finally:\n # Restore original state — guaranteed even if deployment raises\n constants.FEATURE_FLAG.clear()\n constants.FEATURE_FLAG.update(original_feature_flags)\n\n for key, original_value in overridden_keys.items():\n setattr(constants, key, original_value)\n" }, { "path": "src/fabric_cicd/_common/_config_validator.py", "content": "# Copyright (c) Microsoft Corporation.\n# Licensed under the MIT License.\n\n\"\"\"Configuration validation for YAML-based deployment configuration.\"\"\"\n\nimport logging\nimport re\nfrom pathlib import Path\nfrom typing import Any, Optional, Union\n\nimport yaml\n\nfrom fabric_cicd import constants\nfrom fabric_cicd._common._exceptions import InputError\nfrom fabric_cicd._common._validate_env_vars import _URL_CONSTANTS, validate_api_url\n\nlogger = logging.getLogger(__name__)\n\n\nclass ConfigValidationError(InputError):\n \"\"\"Specific exception for configuration validation errors.\"\"\"\n\n def __init__(self, errors: list[str], logger_instance: logging.Logger) -> None:\n \"\"\"Initialize with list of validation errors.\"\"\"\n self.validation_errors = errors\n error_msg = f\"Configuration validation failed with {len(errors)} error(s):\\n\" + \"\\n\".join(\n f\" - {error}\" for error in errors\n )\n super().__init__(error_msg, logger_instance)\n\n\nclass ConfigValidator:\n \"\"\"Validates YAML configuration files for fabric-cicd deployment.\"\"\"\n\n def __init__(self) -> None:\n \"\"\"Initialize the validator.\"\"\"\n self.errors: list = []\n self.config: dict = None\n self.config_path: Path = None\n self.environment: str = None\n self.config_override: Optional[dict] = None\n\n def validate_config_file(\n self, config_file_path: str, environment: str, config_override: Optional[dict] = None\n ) -> dict[str, Any]:\n \"\"\"\n Validate configuration file and return parsed config if valid.\n\n Args:\n config_file_path: String path to the configuration file\n environment: The target environment for the deployment\n config_override: Optional dictionary to override specific configuration values\n\n Returns:\n Parsed configuration dictionary (includes overrides, if any)\n\n Raises:\n ConfigValidationError: If validation fails\n \"\"\"\n self.errors = []\n self.environment = environment\n self.config_override = config_override\n\n # Step 1: Validate file existence and accessibility\n config_path = self._validate_file_existence(config_file_path)\n\n # Step 2: Validate file content and YAML syntax\n self.config = self._validate_yaml_content(config_path)\n\n # Step 3: Apply and validate config overrides\n if self.config is not None and self.config_override is not None:\n self._apply_and_validate_overrides()\n\n # Step 4: Validate configuration structure and required fields\n if self.config is not None:\n self._validate_config_structure()\n self._validate_config_sections()\n\n # Step 5: Validate environment-specific mapping\n self._validate_environment_exists()\n\n # Step 6: Resolve paths after environment validation passes\n if not self.errors:\n self._resolve_repository_path()\n self._resolve_parameter_path()\n\n # If there are validation errors, raise them all at once\n if self.errors:\n raise ConfigValidationError(self.errors, logger)\n\n return self.config\n\n def _validate_file_existence(self, config_file_path: str) -> Path:\n \"\"\"Validate file path and existence.\"\"\"\n if not config_file_path or not isinstance(config_file_path, str):\n self.errors.append(constants.CONFIG_VALIDATION_MSGS[\"file\"][\"path_empty\"])\n return None\n\n try:\n config_path = Path(config_file_path).resolve()\n except (OSError, RuntimeError) as e:\n self.errors.append(constants.CONFIG_VALIDATION_MSGS[\"file\"][\"invalid_path\"].format(config_file_path, e))\n return None\n\n if not config_path.exists():\n self.errors.append(constants.CONFIG_VALIDATION_MSGS[\"file\"][\"not_found\"].format(config_file_path))\n return None\n\n if not config_path.is_file():\n self.errors.append(constants.CONFIG_VALIDATION_MSGS[\"file\"][\"not_file\"].format(config_file_path))\n return None\n\n self.config_path = config_path\n return config_path\n\n def _validate_yaml_content(self, config_path: Optional[Path]) -> Optional[dict]:\n \"\"\"Validate YAML syntax and basic structure.\"\"\"\n if config_path is None:\n return None\n\n try:\n with config_path.open(encoding=\"utf-8\") as f:\n config = yaml.safe_load(f)\n except yaml.YAMLError as e:\n self.errors.append(constants.CONFIG_VALIDATION_MSGS[\"file\"][\"yaml_syntax\"].format(e))\n return None\n except UnicodeDecodeError as e:\n self.errors.append(constants.CONFIG_VALIDATION_MSGS[\"file\"][\"encoding_error\"].format(e))\n return None\n except PermissionError as e:\n self.errors.append(constants.CONFIG_VALIDATION_MSGS[\"file\"][\"permission_denied\"].format(e))\n return None\n except Exception as e:\n self.errors.append(constants.CONFIG_VALIDATION_MSGS[\"file\"][\"unexpected_error\"].format(e))\n return None\n\n # Handle empty file case\n if config is None:\n self.errors.append(constants.CONFIG_VALIDATION_MSGS[\"file\"][\"empty_file\"])\n return None\n\n if not isinstance(config, dict):\n self.errors.append(constants.CONFIG_VALIDATION_MSGS[\"file\"][\"not_dict\"].format(type(config).__name__))\n return None\n\n return config\n\n def _apply_and_validate_overrides(self) -> None:\n \"\"\"Apply and validate config overrides.\"\"\"\n if not self.config_override:\n return\n\n for section, value in self.config_override.items():\n try:\n # Validate the section\n if not self._valid_override_section(section, value):\n continue\n\n # Merge overrides into config\n self._merge_overrides(section, value)\n\n except Exception as e:\n self.errors.append(constants.CONFIG_VALIDATION_MSGS[\"override\"][\"apply_failed\"].format(section, e))\n\n def _valid_override_section(self, section: str, value: any) -> bool:\n \"\"\"Validates the override section and structure are correct.\"\"\"\n # Check section is supported\n if section not in constants.CONFIG_SECTIONS:\n self.errors.append(\n constants.CONFIG_VALIDATION_MSGS[\"override\"][\"unsupported_section\"].format(\n section, list(constants.CONFIG_SECTIONS.keys())\n )\n )\n return False\n\n # Check type is valid\n expected_types = constants.CONFIG_SECTIONS[section][\"type\"]\n if not isinstance(value, expected_types):\n type_names = (\n \" or \".join(t.__name__ for t in expected_types)\n if isinstance(expected_types, tuple)\n else expected_types.__name__\n )\n self.errors.append(\n constants.CONFIG_VALIDATION_MSGS[\"override\"][\"wrong_type\"].format(\n section, type_names, type(value).__name__\n )\n )\n return False\n\n # Check setting is supported for applicable sections\n if isinstance(value, dict) and section in [\"core\", \"publish\", \"unpublish\"]:\n supported = constants.CONFIG_SECTIONS[section][\"settings\"]\n for setting in value:\n if setting not in supported:\n self.errors.append(\n constants.CONFIG_VALIDATION_MSGS[\"override\"][\"unsupported_setting\"].format(\n section, setting, supported\n )\n )\n return False\n\n return True\n\n def _merge_overrides(self, section: str, value: Union[dict, list]) -> None:\n \"\"\"Merge section and setting overrides into config file.\"\"\"\n # Special handling for features and constants sections\n if section == \"features\":\n action = \"added\" if not self.config.get(\"features\") else \"updated\"\n self.config[\"features\"] = value\n logger.warning(constants.CONFIG_VALIDATION_MSGS[\"log\"][\"override_section\"].format(action, section, value))\n return\n\n if section == \"constants\":\n action = \"added\" if \"constants\" not in self.config else \"updated\"\n self.config[\"constants\"] = value\n logger.warning(constants.CONFIG_VALIDATION_MSGS[\"log\"][\"override_section\"].format(action, section, value))\n return\n\n # Add section if it doesn't already exist (publish, unpublish only)\n if section not in self.config:\n if section == \"core\":\n self.errors.append(constants.CONFIG_VALIDATION_MSGS[\"override\"][\"cannot_create_core\"])\n return\n\n self.config[section] = {}\n logger.warning(constants.CONFIG_VALIDATION_MSGS[\"log\"][\"override_added_section\"].format(section))\n\n # Process field by field for other sections (core, publish, unpublish)\n for setting, setting_value in value.items():\n exists = setting in self.config[section]\n\n # Validate required fields can only be overridden, not added\n if not exists and section == \"core\":\n if setting == \"repository_directory\":\n self.errors.append(\n constants.CONFIG_VALIDATION_MSGS[\"override\"][\"cannot_create_required\"].format(setting)\n )\n continue\n if setting in [\"workspace_id\", \"workspace\"]:\n # Check if the other workspace identifier exists\n other_workspace_field = \"workspace\" if setting == \"workspace_id\" else \"workspace_id\"\n if other_workspace_field not in self.config[section]:\n self.errors.append(\n constants.CONFIG_VALIDATION_MSGS[\"override\"][\"cannot_create_workspace_id\"].format(setting)\n )\n continue\n\n # Handle environment specific override\n if isinstance(setting_value, dict) and self.environment in setting_value:\n env_value = setting_value[self.environment]\n\n # Replace existing environment value with override value\n if exists and isinstance(self.config[section][setting], dict):\n self.config[section][setting][self.environment] = env_value\n logger.warning(\n constants.CONFIG_VALIDATION_MSGS[\"log\"][\"override_env_specific\"].format(\n section, setting, self.environment, env_value\n )\n )\n\n # Otherwise, add new environment value\n else:\n self.config[section][setting] = {self.environment: env_value}\n logger.warning(\n constants.CONFIG_VALIDATION_MSGS[\"log\"][\"override_env_mapping\"].format(\n section, setting, self.environment, env_value\n )\n )\n\n # Otherwise, handle direct value override\n else:\n self.config[section][setting] = setting_value\n action = \"updated\" if exists else \"added\"\n logger.warning(\n constants.CONFIG_VALIDATION_MSGS[\"log\"][\"override_setting\"].format(\n action, section, setting, setting_value\n )\n )\n\n def _validate_config_structure(self) -> None:\n \"\"\"Validate top-level configuration structure.\"\"\"\n if not isinstance(self.config, dict):\n return\n\n # Check for required top-level sections\n if \"core\" not in self.config:\n self.errors.append(constants.CONFIG_VALIDATION_MSGS[\"structure\"][\"missing_core\"])\n return\n\n if not isinstance(self.config[\"core\"], dict):\n self.errors.append(\n constants.CONFIG_VALIDATION_MSGS[\"structure\"][\"core_not_dict\"].format(\n type(self.config[\"core\"]).__name__\n )\n )\n\n def _validate_config_sections(self) -> None:\n \"\"\"Validate the configuration sections\"\"\"\n # Validate core section (required)\n if \"core\" not in self.config or not isinstance(self.config[\"core\"], dict):\n self.errors.append(constants.CONFIG_VALIDATION_MSGS[\"structure\"][\"missing_core\"])\n return\n\n core = self.config[\"core\"]\n\n # Validate workspace identification (must have either workspace_id or workspace)\n has_workspace_id = self._validate_workspace_field(core, \"workspace_id\")\n has_workspace_name = self._validate_workspace_field(core, \"workspace\")\n\n if not has_workspace_id and not has_workspace_name:\n self.errors.append(constants.CONFIG_VALIDATION_MSGS[\"structure\"][\"missing_workspace_id\"])\n\n # Validate repository_directory\n self._validate_repository_directory(core)\n\n # Validate optional item_types_in_scope\n self._validate_item_types_in_scope(core)\n\n # Validate optional parameter field\n self._validate_parameter_field(core)\n\n # Validate optional sections\n # publish section\n if \"publish\" in self.config:\n self._validate_operation_section(self.config[\"publish\"], \"publish\")\n\n # unpublish section\n if \"unpublish\" in self.config:\n self._validate_operation_section(self.config[\"unpublish\"], \"unpublish\")\n\n # features section\n if \"features\" in self.config:\n self._validate_features_section(self.config[\"features\"])\n\n # constants section\n if \"constants\" in self.config:\n self._validate_constants_section(self.config[\"constants\"])\n\n def _validate_environment_exists(self) -> None:\n \"\"\"Validate that target environment exists in all environment mappings.\"\"\"\n if self.environment == \"N/A\":\n # Handle no target environment case\n if any(\n field_name in section and isinstance(section[field_name], dict)\n for section, field_name, _, _, _ in _get_config_fields(self.config)\n if field_name != \"constants\"\n ):\n self.errors.append(constants.CONFIG_VALIDATION_MSGS[\"environment\"][\"no_env_with_mappings\"])\n return\n\n # Check each field for target environment presence\n for section, field_name, display_name, is_required, log_warning in _get_config_fields(self.config):\n if field_name in section:\n field_value = section[field_name]\n # Handle constants special case — check each constant's value individually\n if field_name == \"constants\":\n if isinstance(field_value, dict):\n for const_key, const_val in field_value.items():\n if isinstance(const_val, dict) and self.environment not in const_val:\n available_envs = list(const_val.keys())\n logger.warning(\n f\"Environment '{self.environment}' not found in 'constants.{const_key}'. \"\n f\"Available environments: {available_envs}. This constant will be skipped.\"\n )\n continue\n\n # If it's a dict (environment mapping), check if target environment exists\n if isinstance(field_value, dict) and self.environment not in field_value:\n available_envs = list(field_value.keys())\n msg = (\n f\"Environment '{self.environment}' not found in '{display_name}'. \"\n f\"Available environments: {available_envs}. This setting will be skipped.\"\n )\n\n if is_required:\n self.errors.append(\n constants.CONFIG_VALIDATION_MSGS[\"environment\"][\"env_not_found\"].format(\n self.environment, display_name, available_envs\n )\n )\n elif log_warning:\n logger.warning(msg)\n else:\n logger.debug(msg)\n\n def _validate_environment_mapping(self, field_value: dict, field_name: str, accepted_type: type) -> bool:\n \"\"\"Validate field with environment mapping.\"\"\"\n if not field_value:\n self.errors.append(constants.CONFIG_VALIDATION_MSGS[\"environment\"][\"empty_mapping\"].format(field_name))\n return False\n\n valid = True\n for env, value in field_value.items():\n # Validate environment key\n if not isinstance(env, str) or not env.strip():\n self.errors.append(\n constants.CONFIG_VALIDATION_MSGS[\"environment\"][\"invalid_env_key\"].format(\n field_name, type(env).__name__\n )\n )\n valid = False\n continue\n\n # Validate environment value type\n if not isinstance(value, accepted_type):\n self.errors.append(\n f\"'{field_name}' value for environment '{env}' must be a {accepted_type.__name__}, got {type(value).__name__}\"\n )\n valid = False\n continue\n\n # Validate environment value content (type-specific)\n if accepted_type == str:\n if not value.strip():\n self.errors.append(\n constants.CONFIG_VALIDATION_MSGS[\"environment\"][\"empty_env_value\"].format(field_name, env)\n )\n valid = False\n elif accepted_type == list and not value:\n self.errors.append(\n constants.CONFIG_VALIDATION_MSGS[\"environment\"][\"empty_env_value\"].format(field_name, env)\n )\n valid = False\n\n return valid\n\n def _validate_workspace_field(self, core: dict, field_name: str) -> bool:\n \"\"\"Validate workspace_id or workspace field.\"\"\"\n if field_name not in core:\n return False\n\n field_value = core[field_name]\n\n # Support both string values and environment mappings\n if isinstance(field_value, str):\n if not field_value.strip():\n self.errors.append(constants.CONFIG_VALIDATION_MSGS[\"field\"][\"empty_value\"].format(field_name))\n return False\n\n return self._validate_workspace_value(field_value, field_name, field_name)\n\n if isinstance(field_value, dict):\n valid = self._validate_environment_mapping(field_value, field_name, str)\n\n # Apply field-specific validation to each environment value\n if valid:\n for env, value in field_value.items():\n if isinstance(value, str) and not self._validate_workspace_value(\n value, field_name, f\"{field_name}.{env}\"\n ):\n valid = False\n\n return valid\n\n self.errors.append(\n constants.CONFIG_VALIDATION_MSGS[\"field\"][\"string_or_dict\"].format(field_name, type(field_value).__name__)\n )\n return False\n\n def _validate_workspace_value(self, value: str, field_name: str, context: str) -> bool:\n \"\"\"Validate a workspace value (applies GUID validation for workspace_id).\"\"\"\n if field_name == \"workspace_id\" and not _validate_guid_format(value):\n self.errors.append(constants.CONFIG_VALIDATION_MSGS[\"field\"][\"invalid_guid\"].format(context, value))\n return False\n return True\n\n def _validate_repository_directory(self, core: dict) -> None:\n \"\"\"Validate repository_directory field.\"\"\"\n if \"repository_directory\" not in core:\n self.errors.append(constants.CONFIG_VALIDATION_MSGS[\"structure\"][\"missing_repository_dir\"])\n return\n\n repository_directory = core[\"repository_directory\"]\n\n # Support both string values and environment mappings\n if isinstance(repository_directory, str):\n if not repository_directory.strip():\n self.errors.append(\n constants.CONFIG_VALIDATION_MSGS[\"field\"][\"empty_value\"].format(\"repository_directory\")\n )\n return\n\n elif isinstance(repository_directory, dict):\n if not self._validate_environment_mapping(repository_directory, \"repository_directory\", str):\n return\n\n else:\n self.errors.append(\n constants.CONFIG_VALIDATION_MSGS[\"field\"][\"string_or_dict\"].format(\n \"repository_directory\", type(repository_directory).__name__\n )\n )\n return\n\n def _validate_item_types_in_scope(self, core: dict[str, Any]) -> None:\n \"\"\"Validate item_types_in_scope field if present.\"\"\"\n if \"item_types_in_scope\" not in core:\n return # Optional field\n\n item_types = core[\"item_types_in_scope\"]\n\n if isinstance(item_types, list):\n if not item_types:\n self.errors.append(\n constants.CONFIG_VALIDATION_MSGS[\"field\"][\"empty_list\"].format(\"item_types_in_scope\")\n )\n return\n\n self._validate_item_types(item_types)\n return\n\n if isinstance(item_types, dict):\n # Validate environment mapping\n if not self._validate_environment_mapping(item_types, \"item_types_in_scope\", list):\n return\n\n # Validate each environment's item types\n for env, item_type_list in item_types.items():\n self._validate_item_types(item_type_list, env_context=env)\n return\n\n self.errors.append(\n constants.CONFIG_VALIDATION_MSGS[\"field\"][\"item_types_list_or_dict\"].format(type(item_types).__name__)\n )\n\n def _validate_item_types(self, item_types: list, env_context: Optional[str] = None) -> None:\n \"\"\"Validate a list of item types.\"\"\"\n if not item_types:\n self.errors.append(constants.CONFIG_VALIDATION_MSGS[\"field\"][\"empty_list\"].format(\"item_types_in_scope\"))\n return\n\n # Validate each item type\n for item_type in item_types:\n if not isinstance(item_type, str):\n self.errors.append(\n constants.CONFIG_VALIDATION_MSGS[\"field\"][\"invalid_item_type\"].format(\n type(item_type).__name__, item_type\n )\n )\n continue\n\n if item_type not in constants.ACCEPTED_ITEM_TYPES:\n available_types = \", \".join(sorted(constants.ACCEPTED_ITEM_TYPES))\n if env_context:\n self.errors.append(\n constants.CONFIG_VALIDATION_MSGS[\"field\"][\"unsupported_item_type_env\"].format(\n item_type, env_context, available_types\n )\n )\n else:\n self.errors.append(\n constants.CONFIG_VALIDATION_MSGS[\"field\"][\"unsupported_item_type\"].format(\n item_type, available_types\n )\n )\n\n def _validate_parameter_field(self, core: dict) -> None:\n \"\"\"Validate parameter field if present.\"\"\"\n if \"parameter\" not in core:\n return # Optional field\n\n parameter_value = core[\"parameter\"]\n\n # Support both string values and environment mappings\n if isinstance(parameter_value, str):\n if not parameter_value.strip():\n self.errors.append(constants.CONFIG_VALIDATION_MSGS[\"field\"][\"empty_value\"].format(\"parameter\"))\n return\n elif isinstance(parameter_value, dict):\n if not self._validate_environment_mapping(parameter_value, \"parameter\", str):\n return\n else:\n self.errors.append(\n constants.CONFIG_VALIDATION_MSGS[\"field\"][\"string_or_dict\"].format(\n \"parameter\", type(parameter_value).__name__\n )\n )\n\n def _resolve_path_field(\n self, field_value: Union[str, dict], field_name: str, section_name: str, path_type: str = \"directory\"\n ) -> None:\n \"\"\"Path resolution for configuration \"path\" fields (e.g, repository_directory, parameter).\"\"\"\n # Prepare paths for resolution\n paths_to_resolve = {\"_default\": field_value} if isinstance(field_value, str) else field_value\n\n # Skip resolution if config validation failed\n if not self.config_path:\n logger.debug(constants.CONFIG_VALIDATION_MSGS[\"path\"][\"skip\"].format(field_name))\n return\n\n # If environment mapping is used and target environment is provided, only process that environment path\n if self.environment and self.environment != \"N/A\" and isinstance(field_value, dict):\n if self.environment not in paths_to_resolve:\n # Skip if environment not in mapping (for parameter field, which is optional)\n logger.debug(\n f\"Skipping path resolution for '{field_name}' - environment '{self.environment}' not in mapping\"\n )\n return\n paths_to_resolve = {self.environment: paths_to_resolve[self.environment]}\n\n for env_key, path_str in paths_to_resolve.items():\n try:\n env_desc = f\" for environment '{env_key}'\" if env_key != \"_default\" else \"\"\n path = Path(path_str)\n\n if path.is_absolute():\n resolved_path = path\n logger.info(\n constants.CONFIG_VALIDATION_MSGS[\"path\"][\"absolute\"].format(field_name, env_desc, resolved_path)\n )\n\n # Validate absolute paths are in the same git repository as config file\n config_repo_root = _find_git_root(self.config_path.parent)\n path_repo_root = _find_git_root(resolved_path.parent if path_type == \"file\" else resolved_path)\n\n if config_repo_root and path_repo_root and config_repo_root != path_repo_root:\n self.errors.append(\n constants.CONFIG_VALIDATION_MSGS[\"path\"][\"git_repo\"].format(\n field_name, env_desc, config_repo_root, field_name, path_repo_root\n )\n )\n continue\n else:\n # Resolve relative to config path location\n config_dir = self.config_path.parent\n resolved_path = (config_dir / path_str).resolve()\n logger.info(\n constants.CONFIG_VALIDATION_MSGS[\"path\"][\"resolved\"].format(\n field_name, path_str, env_desc, resolved_path\n )\n )\n\n # Validate the resolved path exists\n if not resolved_path.exists():\n self.errors.append(\n constants.CONFIG_VALIDATION_MSGS[\"path\"][\"not_found\"].format(\n field_name, env_desc, resolved_path\n )\n )\n continue\n\n # Path type-specific validation\n if path_type == \"directory\":\n if not resolved_path.is_dir():\n self.errors.append(\n constants.CONFIG_VALIDATION_MSGS[\"path\"][\"not_directory\"].format(\n field_name, env_desc, resolved_path\n )\n )\n continue\n\n elif path_type == \"file\" and not resolved_path.is_file():\n self.errors.append(\n constants.CONFIG_VALIDATION_MSGS[\"path\"][\"not_file\"].format(field_name, env_desc, resolved_path)\n )\n continue\n\n # Store the resolved path back in config\n if isinstance(field_value, str):\n if section_name:\n self.config[section_name][field_name] = str(resolved_path)\n else:\n if section_name:\n self.config[section_name][field_name][env_key] = str(resolved_path)\n\n except (OSError, ValueError) as e:\n self.errors.append(\n constants.CONFIG_VALIDATION_MSGS[\"path\"][\"invalid\"].format(field_name, path_str, env_desc, e)\n )\n\n def _resolve_repository_path(self) -> None:\n \"\"\"Resolve repository directory paths after environment validation.\"\"\"\n core = self.config[\"core\"]\n repository_directory = core[\"repository_directory\"]\n self._resolve_path_field(repository_directory, \"repository_directory\", \"core\", \"directory\")\n\n def _resolve_parameter_path(self) -> None:\n \"\"\"Resolve parameter file paths after environment validation.\"\"\"\n core = self.config[\"core\"]\n if \"parameter\" not in core:\n return # Optional field\n\n parameter_value = core[\"parameter\"]\n self._resolve_path_field(parameter_value, \"parameter\", \"core\", \"file\")\n\n def _validate_operation_section(self, section: dict[str, Any], section_name: str) -> None:\n \"\"\"Validate publish/unpublish section structure.\"\"\"\n if not isinstance(section, dict):\n self.errors.append(\n constants.CONFIG_VALIDATION_MSGS[\"operation\"][\"not_dict\"].format(section_name, type(section).__name__)\n )\n return\n\n # Validate exclude_regex if present\n if \"exclude_regex\" in section:\n exclude_regex = section[\"exclude_regex\"]\n\n if isinstance(exclude_regex, str):\n if not exclude_regex.strip():\n self.errors.append(\n constants.CONFIG_VALIDATION_MSGS[\"operation\"][\"empty_string\"].format(\n f\"{section_name}.exclude_regex\"\n )\n )\n else:\n self._validate_regex(exclude_regex, f\"{section_name}.exclude_regex\")\n\n elif isinstance(exclude_regex, dict):\n # Validate environment mapping\n if not self._validate_environment_mapping(exclude_regex, f\"{section_name}.exclude_regex\", str):\n return\n\n # Validate each environment's regex pattern\n for env, regex_pattern in exclude_regex.items():\n self._validate_regex(regex_pattern, f\"{section_name}.exclude_regex.{env}\")\n\n else:\n self.errors.append(\n constants.CONFIG_VALIDATION_MSGS[\"field\"][\"string_or_dict\"].format(\n f\"{section_name}.exclude_regex\", type(exclude_regex).__name__\n )\n )\n\n # Validate items_to_include if present\n if \"items_to_include\" in section:\n items = section[\"items_to_include\"]\n\n if isinstance(items, list):\n if not items:\n self.errors.append(\n constants.CONFIG_VALIDATION_MSGS[\"operation\"][\"empty_list\"].format(\n f\"{section_name}.items_to_include\"\n )\n )\n else:\n self._validate_items_list(items, f\"{section_name}.items_to_include\")\n\n elif isinstance(items, dict):\n # Validate environment mapping\n if not self._validate_environment_mapping(items, f\"{section_name}.items_to_include\", list):\n return\n\n # Validate each environment's items list\n for env, items_list in items.items():\n self._validate_items_list(items_list, f\"{section_name}.items_to_include.{env}\")\n\n else:\n self.errors.append(\n constants.CONFIG_VALIDATION_MSGS[\"field\"][\"list_or_dict\"].format(\n f\"{section_name}.items_to_include\", type(items).__name__\n )\n )\n\n # Validate folder_exclude_regex if present (publish only)\n if \"folder_exclude_regex\" in section:\n if section_name != \"publish\":\n self.errors.append(\n constants.CONFIG_VALIDATION_MSGS[\"operation\"][\"unsupported_field\"].format(\n \"folder_exclude_regex\", section_name\n )\n )\n\n folder_exclude_regex = section[\"folder_exclude_regex\"]\n if isinstance(folder_exclude_regex, str):\n if not folder_exclude_regex.strip():\n self.errors.append(\n constants.CONFIG_VALIDATION_MSGS[\"operation\"][\"empty_string\"].format(\n f\"{section_name}.folder_exclude_regex\"\n )\n )\n else:\n self._validate_regex(folder_exclude_regex, f\"{section_name}.folder_exclude_regex\")\n\n elif isinstance(folder_exclude_regex, dict):\n # Validate environment mapping\n if not self._validate_environment_mapping(\n folder_exclude_regex, f\"{section_name}.folder_exclude_regex\", str\n ):\n return\n\n # Validate each environment's regex pattern\n for env, regex_pattern in folder_exclude_regex.items():\n self._validate_regex(regex_pattern, f\"{section_name}.folder_exclude_regex.{env}\")\n\n else:\n self.errors.append(\n constants.CONFIG_VALIDATION_MSGS[\"field\"][\"string_or_dict\"].format(\n f\"{section_name}.folder_exclude_regex\", type(folder_exclude_regex).__name__\n )\n )\n\n # Validate folder_path_to_include if present (publish only)\n if \"folder_path_to_include\" in section:\n if section_name != \"publish\":\n self.errors.append(\n constants.CONFIG_VALIDATION_MSGS[\"operation\"][\"unsupported_field\"].format(\n \"folder_path_to_include\", section_name\n )\n )\n\n folders = section[\"folder_path_to_include\"]\n if isinstance(folders, list):\n if not folders:\n self.errors.append(\n constants.CONFIG_VALIDATION_MSGS[\"operation\"][\"empty_list\"].format(\n f\"{section_name}.folder_path_to_include\"\n )\n )\n else:\n self._validate_folders_list(folders, f\"{section_name}.folder_path_to_include\")\n\n elif isinstance(folders, dict):\n # Validate environment mapping\n if not self._validate_environment_mapping(folders, f\"{section_name}.folder_path_to_include\", list):\n return\n\n # Validate each environment's folders list\n for env, folders_list in folders.items():\n self._validate_folders_list(folders_list, f\"{section_name}.folder_path_to_include.{env}\")\n\n else:\n self.errors.append(\n constants.CONFIG_VALIDATION_MSGS[\"field\"][\"list_or_dict\"].format(\n f\"{section_name}.folder_path_to_include\", type(folders).__name__\n )\n )\n\n # Validate shortcut_exclude_regex if present (publish only)\n if \"shortcut_exclude_regex\" in section:\n if section_name != \"publish\":\n self.errors.append(\n f\"'{section_name}.shortcut_exclude_regex' is not supported - shortcut exclusion is only available in the 'publish' section\"\n )\n\n shortcut_exclude_regex = section[\"shortcut_exclude_regex\"]\n if isinstance(shortcut_exclude_regex, str):\n if not shortcut_exclude_regex.strip():\n self.errors.append(\n constants.CONFIG_VALIDATION_MSGS[\"operation\"][\"empty_string\"].format(\n f\"{section_name}.shortcut_exclude_regex\"\n )\n )\n else:\n self._validate_regex(shortcut_exclude_regex, f\"{section_name}.shortcut_exclude_regex\")\n\n elif isinstance(shortcut_exclude_regex, dict):\n # Validate environment mapping\n if not self._validate_environment_mapping(\n shortcut_exclude_regex, f\"{section_name}.shortcut_exclude_regex\", str\n ):\n return\n\n # Validate each environment's regex pattern\n for env, regex_pattern in shortcut_exclude_regex.items():\n self._validate_regex(regex_pattern, f\"{section_name}.shortcut_exclude_regex.{env}\")\n\n else:\n self.errors.append(\n constants.CONFIG_VALIDATION_MSGS[\"field\"][\"string_or_dict\"].format(\n f\"{section_name}.shortcut_exclude_regex\", type(shortcut_exclude_regex).__name__\n )\n )\n\n # Validate skip if present\n if \"skip\" in section:\n skip_value = section[\"skip\"]\n\n if isinstance(skip_value, bool):\n # Single boolean value\n return\n\n if isinstance(skip_value, dict):\n # Use the reusable environment mapping validation\n if not self._validate_environment_mapping(skip_value, f\"{section_name}.skip\", bool):\n return\n\n else:\n self.errors.append(\n constants.CONFIG_VALIDATION_MSGS[\"field\"][\"string_or_dict\"]\n .format(f\"{section_name}.skip\", type(skip_value).__name__)\n .replace(\"a string\", \"a boolean\")\n )\n\n # Validate mutual exclusivity of folder filtering options\n self._validate_mutually_exclusive_fields(\n section, \"folder_exclude_regex\", \"folder_path_to_include\", section_name\n )\n\n def _validate_regex(self, regex: str, section_name: str) -> None:\n \"\"\"Validate regex value.\"\"\"\n try:\n re.compile(regex)\n except re.error as e:\n self.errors.append(\n constants.CONFIG_VALIDATION_MSGS[\"operation\"][\"invalid_regex\"].format(regex, section_name, e)\n )\n\n def _validate_items_list(self, items_list: list, context: str) -> None:\n \"\"\"Validate a list of items with proper context for error messages.\"\"\"\n for i, item in enumerate(items_list):\n if not isinstance(item, str):\n self.errors.append(\n constants.CONFIG_VALIDATION_MSGS[\"operation\"][\"list_entry_type\"].format(\n context, i, type(item).__name__\n )\n )\n elif not item.strip():\n self.errors.append(constants.CONFIG_VALIDATION_MSGS[\"operation\"][\"list_entry_empty\"].format(context, i))\n\n def _validate_folders_list(self, folders_list: list, context: str) -> None:\n \"\"\"Validate a list of folder paths with proper context for error messages.\"\"\"\n for i, folder in enumerate(folders_list):\n if not isinstance(folder, str):\n self.errors.append(\n constants.CONFIG_VALIDATION_MSGS[\"operation\"][\"list_entry_type\"].format(\n context, i, type(folder).__name__\n )\n )\n elif not folder.strip():\n self.errors.append(constants.CONFIG_VALIDATION_MSGS[\"operation\"][\"list_entry_empty\"].format(context, i))\n elif not folder.startswith(\"/\"):\n self.errors.append(\n constants.CONFIG_VALIDATION_MSGS[\"operation\"][\"folders_list_prefix\"].format(context, i, folder)\n )\n\n def _validate_mutually_exclusive_fields(self, section: dict, field1: str, field2: str, section_name: str) -> None:\n \"\"\"Validate that two fields are not both specified for the same environment.\"\"\"\n if field1 not in section or field2 not in section:\n return\n\n value1 = section[field1]\n value2 = section[field2]\n\n # Both are direct values (not environment-specific), throw error\n if not isinstance(value1, dict) and not isinstance(value2, dict):\n self.errors.append(\n constants.CONFIG_VALIDATION_MSGS[\"operation\"][\"mutually_exclusive\"].format(\n f\"{section_name}.{field1}\", f\"{section_name}.{field2}\"\n )\n )\n return\n\n # Determine which environments each field contains (if they are environment mappings)\n value1_envs = set(value1.keys()) if isinstance(value1, dict) else set()\n value2_envs = set(value2.keys()) if isinstance(value2, dict) else set()\n\n # Determine if it is a direct value\n value1_is_direct = not isinstance(value1, dict)\n value2_is_direct = not isinstance(value2, dict)\n\n # Check if both fields would resolve for the target environment\n value1_applies = value1_is_direct or self.environment in value1_envs\n value2_applies = value2_is_direct or self.environment in value2_envs\n\n if value1_applies and value2_applies:\n self.errors.append(\n constants.CONFIG_VALIDATION_MSGS[\"operation\"][\"mutually_exclusive_env\"].format(\n f\"{section_name}.{field1}\",\n f\"{section_name}.{field2}\",\n [self.environment],\n )\n )\n\n def _validate_features_section(self, features: any) -> None:\n \"\"\"Validate features section.\"\"\"\n if isinstance(features, list):\n if not features:\n self.errors.append(constants.CONFIG_VALIDATION_MSGS[\"operation\"][\"empty_section\"].format(\"features\"))\n return\n\n self._validate_features_list(features, \"features\")\n return\n\n if isinstance(features, dict):\n # Validate environment mapping\n if not self._validate_environment_mapping(features, \"features\", list):\n return\n\n # Validate each environment's features list\n for env, features_list in features.items():\n self._validate_features_list(features_list, f\"features.{env}\")\n return\n\n self.errors.append(\n constants.CONFIG_VALIDATION_MSGS[\"operation\"][\"features_type\"].format(type(features).__name__)\n )\n\n def _validate_features_list(self, features_list: list, context: str) -> None:\n \"\"\"Validate a list of features with proper context for error messages.\"\"\"\n for i, feature in enumerate(features_list):\n if not isinstance(feature, str):\n self.errors.append(\n constants.CONFIG_VALIDATION_MSGS[\"operation\"][\"list_entry_type\"].format(\n context, i, type(feature).__name__\n )\n )\n elif not feature.strip():\n self.errors.append(constants.CONFIG_VALIDATION_MSGS[\"operation\"][\"list_entry_empty\"].format(context, i))\n\n def _validate_constants_section(self, constants_section: any) -> None:\n \"\"\"Validate constants section.\"\"\"\n if not isinstance(constants_section, dict):\n self.errors.append(\n constants.CONFIG_VALIDATION_MSGS[\"operation\"][\"not_dict\"].format(\n \"constants\", type(constants_section).__name__\n )\n )\n return\n\n # Validate each constant key individually — values can be flat or per-key env mappings\n for key, value in constants_section.items():\n if not isinstance(key, str) or not key.strip():\n self.errors.append(\n constants.CONFIG_VALIDATION_MSGS[\"operation\"][\"invalid_constant_key\"].format(\"constants\", key)\n )\n continue\n\n # Validate that the constant exists in the constants module\n if not hasattr(constants, key):\n self.errors.append(\n constants.CONFIG_VALIDATION_MSGS[\"operation\"][\"unknown_constant\"].format(key, \"constants\")\n )\n\n if isinstance(value, dict):\n # Per-key environment mapping: { KEY: { dev: val, prod: val } }\n for env, env_value in value.items():\n if not isinstance(env, str) or not env.strip():\n self.errors.append(\n constants.CONFIG_VALIDATION_MSGS[\"environment\"][\"invalid_env_key\"].format(\n f\"constants.{key}\", type(env).__name__\n )\n )\n continue\n self._validate_single_constant(key, env_value, f\"constants.{key}.{env}\")\n else:\n # Flat value: { KEY: val }\n self._validate_single_constant(key, value, f\"constants.{key}\")\n\n def _validate_single_constant(self, key: str, value: any, context: str) -> None:\n \"\"\"Validate a single constant value.\"\"\"\n if key in _URL_CONSTANTS:\n if not isinstance(value, str):\n self.errors.append(f\"'{context}' must be a string URL, got {type(value).__name__}\")\n return\n try:\n validate_api_url(value, context)\n except InputError as e:\n self.errors.append(str(e))\n\n\ndef _get_config_fields(config: dict) -> list[tuple[dict, str, str, bool, bool]]:\n \"\"\"Get list of all fields that support environment mappings.\n\n Returns:\n List of tuples: (section_dict, field_name, display_name, is_required, log_warning)\n - is_required: If True, missing environment causes error.\n - log_warning: logging type (e.g., warning (True), debug (False)).\n \"\"\"\n return [\n # Core section fields - required\n (config.get(\"core\", {}), \"workspace_id\", \"core.workspace_id\", True, False),\n (config.get(\"core\", {}), \"workspace\", \"core.workspace\", True, False),\n (config.get(\"core\", {}), \"repository_directory\", \"core.repository_directory\", True, False),\n # Core section fields - optional but important (warn if missing)\n (config.get(\"core\", {}), \"item_types_in_scope\", \"core.item_types_in_scope\", False, True),\n (config.get(\"core\", {}), \"parameter\", \"core.parameter\", False, True),\n # Publish section fields - optional (debug if missing)\n (config.get(\"publish\", {}), \"exclude_regex\", \"publish.exclude_regex\", False, False),\n (config.get(\"publish\", {}), \"folder_exclude_regex\", \"publish.folder_exclude_regex\", False, False),\n (config.get(\"publish\", {}), \"folder_path_to_include\", \"publish.folder_path_to_include\", False, False),\n (config.get(\"publish\", {}), \"shortcut_exclude_regex\", \"publish.shortcut_exclude_regex\", False, False),\n (config.get(\"publish\", {}), \"items_to_include\", \"publish.items_to_include\", False, False),\n (config.get(\"publish\", {}), \"skip\", \"publish.skip\", False, False),\n # Unpublish section fields - optional (debug if missing)\n (config.get(\"unpublish\", {}), \"exclude_regex\", \"unpublish.exclude_regex\", False, False),\n (config.get(\"unpublish\", {}), \"items_to_include\", \"unpublish.items_to_include\", False, False),\n (config.get(\"unpublish\", {}), \"skip\", \"unpublish.skip\", False, False),\n # Top-level sections - optional (warn if missing)\n (config, \"features\", \"features\", False, True),\n (config, \"constants\", \"constants\", False, True),\n ]\n\n\ndef _find_git_root(path: Path) -> Optional[Path]:\n \"\"\"Find the git repository root for a given path.\"\"\"\n current = path.resolve()\n while current != current.parent:\n if (current / \".git\").exists():\n return current\n current = current.parent\n return None\n\n\ndef _validate_guid_format(guid: str) -> bool:\n \"\"\"Validate GUID format using the pattern from constants.\"\"\"\n return bool(re.match(constants.VALID_GUID_REGEX, guid))\n" }, { "path": "src/fabric_cicd/_common/_deployment_result.py", "content": "# Copyright (c) Microsoft Corporation.\n# Licensed under the MIT License.\n\n\"\"\"Deployment result types for config-based deployment operations.\"\"\"\n\nfrom dataclasses import dataclass, field\nfrom enum import Enum\nfrom typing import Optional\n\n\nclass DeploymentStatus(str, Enum):\n \"\"\"Enumeration of deployment status values for deploy_with_config results.\"\"\"\n\n COMPLETED = \"completed\"\n \"\"\"Deployment completed successfully without any errors.\"\"\"\n FAILED = \"failed\"\n \"\"\"Deployment failed due to one or more errors.\"\"\"\n\n\n@dataclass\nclass DeploymentResult:\n \"\"\"Structured result of a config-based deployment operation.\n\n Returned by ``deploy_with_config`` on success. On failure, an instance is\n attached to the raised exception as ``e.deployment_result`` with ``status``\n set to ``DeploymentStatus.FAILED``.\n\n Attributes:\n status: The deployment status.\n message: A human-readable message describing the result.\n responses: Optional dictionary of API response data from the deployment.\n \"\"\"\n\n status: DeploymentStatus\n message: str\n responses: Optional[dict] = field(default=None)\n" }, { "path": "src/fabric_cicd/_common/_exceptions.py", "content": "# Copyright (c) Microsoft Corporation.\n# Licensed under the MIT License.\n\n\"\"\"Custom exceptions for the fabric-cicd package.\"\"\"\n\nfrom logging import Logger\nfrom typing import Optional\n\n\nclass BaseCustomError(Exception):\n def __init__(self, message: str, logger: Logger, additional_info: Optional[str] = None) -> None:\n \"\"\"\n Initialize the BaseCustomError.\n\n Args:\n message: The error message.\n logger: The logger instance.\n additional_info: Additional information about the error.\n \"\"\"\n super().__init__(message)\n self.logger = logger\n self.additional_info = additional_info\n\n\nclass ParsingError(BaseCustomError):\n pass\n\n\nclass InputError(BaseCustomError):\n pass\n\n\nclass TokenError(BaseCustomError):\n pass\n\n\nclass InvokeError(BaseCustomError):\n pass\n\n\nclass ItemDependencyError(BaseCustomError):\n pass\n\n\nclass FileTypeError(BaseCustomError):\n pass\n\n\nclass ParameterFileError(BaseCustomError):\n pass\n\n\nclass FailedPublishedItemStatusError(BaseCustomError):\n pass\n\n\nclass PublishError(BaseCustomError):\n \"\"\"Exception raised when one or more publish operations fail.\n\n Attributes:\n errors: List of (item_name, exception) tuples for all failed items.\n \"\"\"\n\n def __init__(self, errors: list[tuple[str, Exception]], logger: Logger) -> None:\n \"\"\"Initialize with a list of (item_name, exception) tuples.\"\"\"\n self.errors = errors\n failed_names = [name for name, _ in errors]\n message = f\"Failed to publish {len(errors)} item(s): {failed_names}\"\n\n additional_info_parts = []\n for item_name, exc in errors:\n additional_info_parts.append(f\"\\n--- {item_name} ---\\n{exc!s}\")\n additional_info = \"\\n\".join(additional_info_parts) if additional_info_parts else None\n\n super().__init__(message, logger, additional_info)\n" }, { "path": "src/fabric_cicd/_common/_fabric_endpoint.py", "content": "# Copyright (c) Microsoft Corporation.\n# Licensed under the MIT License.\n\n\"\"\"Handles interactions with the Fabric API, including authentication and request management.\"\"\"\n\nimport datetime\nimport json\nimport logging\nimport os\nimport time\nfrom typing import Optional\n\nimport requests\nfrom azure.core.credentials import TokenCredential\nfrom azure.core.exceptions import (\n ClientAuthenticationError,\n)\n\nimport fabric_cicd.constants as constants\nfrom fabric_cicd._common._exceptions import InvokeError, TokenError\nfrom fabric_cicd._common._http_tracer import HTTPTracer, HTTPTracerFactory\n\nlogger = logging.getLogger(__name__)\n\n\nclass FabricEndpoint:\n \"\"\"Handles interactions with the Fabric API, including authentication and request management.\"\"\"\n\n def __init__(\n self,\n token_credential: TokenCredential,\n requests_module: requests = requests,\n http_tracer: Optional[HTTPTracer] = None,\n ) -> None:\n \"\"\"\n Initializes the FabricEndpoint instance, sets up the authentication token.\n\n Args:\n token_credential: The token credential.\n requests_module: The requests module.\n http_tracer: Optional HTTP tracer for debugging. If None, create using factory.\n \"\"\"\n self.aad_token = None\n self.aad_token_expiration = None\n self.token_credential = token_credential\n self.requests = requests_module\n self.http_tracer = http_tracer if http_tracer is not None else HTTPTracerFactory.create()\n\n self._refresh_token()\n\n def invoke(\n self,\n method: str,\n url: str,\n body: str = \"{}\",\n files: Optional[dict] = None,\n poll_long_running: bool = True,\n max_duration: int = 300,\n **kwargs,\n ) -> dict:\n \"\"\"\n Sends an HTTP request to the specified URL with the given method and body.\n\n Args:\n method: HTTP method to use for the request (e.g., 'GET', 'POST', 'PATCH', 'DELETE').\n url: URL to send the request to.\n body: The JSON body to include in the request. Defaults to an empty JSON object.\n files: The file path to be included in the request. Defaults to None.\n poll_long_running: A flag to poll for long-running operations. Defaults to True.\n max_duration: Maximum execution duration in seconds. Defaults to 300 (5 minutes).\n **kwargs: Additional keyword arguments to pass to the method.\n \"\"\"\n exit_loop = False\n iteration_count = 0\n long_running = False\n start_time = time.time()\n invoke_log_message = \"\"\n\n while not exit_loop:\n try:\n headers = {\n \"Authorization\": f\"Bearer {self.aad_token}\",\n \"User-Agent\": f\"{constants.USER_AGENT}\",\n }\n if files is None:\n headers[\"Content-Type\"] = \"application/json; charset=utf-8\"\n\n self.http_tracer.capture_request(method, url, headers, body, files)\n response = self.requests.request(method=method, url=url, headers=headers, json=body, files=files)\n self.http_tracer.capture_response(response)\n\n iteration_count += 1\n\n invoke_log_message = _format_invoke_log(response, method, url, body)\n\n # Handle expired authentication token\n if response.status_code == 401 and response.headers.get(\"x-ms-public-api-error-code\") == \"TokenExpired\":\n logger.info(f\"{constants.INDENT}AAD token expired. Refreshing token.\")\n self._refresh_token()\n # Handle long-running operations without polling (e.g., for environment item publish)\n elif response.status_code == 202 and not poll_long_running:\n # Accept 202, do not poll\n exit_loop = True\n else:\n exit_loop, method, url, body, long_running = _handle_response(\n response,\n method,\n url,\n body,\n long_running,\n iteration_count,\n max_duration,\n start_time,\n **kwargs,\n )\n\n # Log if reached to end of loop iteration\n if logger.isEnabledFor(logging.DEBUG):\n logger.debug(invoke_log_message)\n\n except (requests.exceptions.ConnectionError, requests.exceptions.Timeout) as e:\n iteration_count += 1\n if max_duration is not None and time.time() - start_time >= max_duration:\n logger.debug(invoke_log_message)\n raise InvokeError(e, logger, invoke_log_message) from e\n handle_retry(\n attempt=iteration_count,\n base_delay=10,\n max_duration=max_duration,\n start_time=start_time,\n prepend_message=\"Connection error encountered.\",\n )\n\n except Exception as e:\n logger.debug(invoke_log_message)\n raise InvokeError(e, logger, invoke_log_message) from e\n\n end_time = time.time()\n logger.debug(f\"Request completed in {end_time - start_time} seconds\")\n\n if exit_loop:\n self.http_tracer.save()\n\n return {\n \"header\": dict(response.headers),\n \"body\": (response.json() if \"application/json\" in response.headers.get(\"Content-Type\") else {}),\n \"status_code\": response.status_code,\n }\n\n def _refresh_token(self) -> None:\n \"\"\"Refreshes the AAD token if empty or expiration has passed.\"\"\"\n if (\n self.aad_token is None\n or self.aad_token_expiration is None\n or self.aad_token_expiration < datetime.datetime.now(datetime.timezone.utc)\n ):\n resource_url = \"https://api.fabric.microsoft.com/.default\"\n\n try:\n access_token = self.token_credential.get_token(resource_url)\n self.aad_token = access_token.token\n self.aad_token_expiration = datetime.datetime.fromtimestamp(\n access_token.expires_on, tz=datetime.timezone.utc\n )\n except ClientAuthenticationError as e:\n msg = f\"Failed to acquire AAD token. {e}\"\n raise TokenError(msg, logger) from e\n except Exception as e:\n msg = f\"An unexpected error occurred when generating the AAD token. {e}\"\n raise TokenError(msg, logger) from e\n\n\ndef _handle_response(\n response: requests.Response,\n method: str,\n url: str,\n body: str,\n long_running: bool,\n iteration_count: int,\n max_duration: Optional[int] = None,\n start_time: Optional[float] = None,\n) -> tuple:\n \"\"\"\n Handles the response from an HTTP request, including retries, throttling, and token expiration.\n Technical debt: this method needs to be refactored to be more testable and requires less parameters.\n Initial approach is only temporary to support testing, but only temporary.\n\n Args:\n response: The response object from the HTTP request.\n method: The HTTP method used in the request.\n url: The URL used in the request.\n body: The JSON body used in the request.\n long_running: A boolean indicating if the operation is long-running.\n iteration_count: The current iteration count of the loop.\n max_duration: Maximum execution duration in seconds. Defaults to None.\n start_time: The start time of the request in seconds since epoch. Defaults to None.\n \"\"\"\n exit_loop = False\n retry_after = response.headers.get(\"Retry-After\", 60)\n\n # Handle long-running operations\n # https://learn.microsoft.com/en-us/rest/api/fabric/core/long-running-operations/get-operation-result\n if (response.status_code == 200 and long_running) or response.status_code == 202:\n url = response.headers.get(\"Location\")\n method = \"GET\"\n body = \"{}\"\n response_json = response.json() if response.text else {}\n\n if long_running:\n status = response_json.get(\"status\")\n if status == \"Succeeded\":\n long_running = False\n # If location not included in operation success call, no body is expected to be returned\n exit_loop = url is None\n\n elif status == \"Failed\":\n response_error = response_json[\"error\"]\n msg = (\n f\"Operation failed. Error Code: {response_error['errorCode']}. \"\n f\"Error Message: {response_error['message']}\"\n )\n raise Exception(msg)\n elif status == \"Undefined\":\n msg = f\"Operation is in an undefined state. Full Body: {response_json}\"\n raise Exception(msg)\n else:\n handle_retry(\n attempt=iteration_count - 1,\n base_delay=0.5,\n response_retry_after=retry_after,\n max_duration=max_duration,\n start_time=start_time,\n prepend_message=f\"{constants.INDENT}Operation in progress.\",\n )\n else:\n if url is None:\n # No Location header means operation completed immediately\n exit_loop = True\n else:\n # We check for system level config for retry delay override, e.g. in unit\n # tests where we want to rip through thousands of API calls quickly. If not\n # set, e.g. at runtime, we use normal polling delay of default 1 second.\n time.sleep(float(os.environ.get(constants.EnvVar.RETRY_DELAY_OVERRIDE_SECONDS.value, 1)))\n long_running = True\n\n # Handle successful responses\n elif response.status_code in {200, 201} or (\n # Valid response for environmentlibrariesnotfound\n response.status_code == 404\n and response.headers.get(\"x-ms-public-api-error-code\") == \"EnvironmentLibrariesNotFound\"\n ):\n exit_loop = True\n\n # Handle API throttling\n elif response.status_code == 429:\n handle_retry(\n attempt=iteration_count,\n base_delay=10,\n max_duration=max_duration,\n start_time=start_time,\n response_retry_after=retry_after,\n prepend_message=\"API is throttled.\",\n )\n\n # Handle internal server errors via retry,\n # rather than failing the deployment run\n elif response.status_code == 500:\n handle_retry(\n attempt=iteration_count,\n base_delay=10,\n max_duration=max_duration,\n start_time=start_time,\n response_retry_after=retry_after,\n prepend_message=\"Server error encountered.\",\n )\n\n # Handle unauthorized access\n elif response.status_code == 401 and response.headers.get(\"x-ms-public-api-error-code\") == \"Unauthorized\":\n msg = f\"The executing identity is not authorized to call {method} on '{url}'.\"\n raise Exception(msg)\n\n # Handle item name conflicts\n elif (\n response.status_code == 400\n and response.headers.get(\"x-ms-public-api-error-code\") == \"ItemDisplayNameNotAvailableYet\"\n ):\n handle_retry(\n attempt=iteration_count,\n base_delay=constants.RETRY_BASE_DELAY_SECONDS,\n max_duration=constants.RETRY_MAX_DURATION_SECONDS,\n start_time=start_time,\n response_retry_after=constants.RETRY_AFTER_SECONDS,\n prepend_message=\"Item name is reserved.\",\n )\n\n # Handle scenario where library removed from environment before being removed from repo\n elif response.status_code == 400 and \"is not present in the environment.\" in response.json().get(\n \"message\", \"No message provided\"\n ):\n msg = (\n f\"Deployment attempted to remove a library that is not present in the environment. \"\n f\"Description: {response.json().get('message')}\"\n )\n raise Exception(msg)\n\n # Handle unsupported principal type\n elif (\n response.status_code == 400\n and response.headers.get(\"x-ms-public-api-error-code\") == \"PrincipalTypeNotSupported\"\n ):\n msg = f\"The executing principal type is not supported to call {method} on '{url}'.\"\n raise Exception(msg)\n\n # Handle unsupported item types\n elif response.status_code == 403 and response.reason == \"FeatureNotAvailable\":\n msg = f\"Item type not supported. Description: {response.reason}\"\n raise Exception(msg)\n\n # Handle unexpected errors\n else:\n err_msg = (\n f\" Message: {response.json()['message']}. {response.json().get('moreDetails', '')}\"\n if \"application/json\" in (response.headers.get(\"Content-Type\") or \"\")\n else \"\"\n )\n msg = f\"Unhandled error occurred calling {method} on '{url}'.{err_msg}\"\n raise Exception(msg)\n\n return exit_loop, method, url, body, long_running\n\n\ndef handle_retry(\n attempt: int,\n base_delay: float,\n response_retry_after: float = 60,\n prepend_message: str = \"\",\n max_duration: Optional[int] = None,\n start_time: Optional[float] = None,\n) -> None:\n \"\"\"\n Handles retry logic with exponential backoff based on the response, retrying\n until the maximum duration is reached.\n\n Args:\n attempt: The current attempt number.\n base_delay: Base delay in seconds for backoff.\n response_retry_after: The value of the Retry-After header from the response.\n prepend_message: Message to prepend to the retry log.\n max_duration: Maximum execution duration in seconds. If None, retries indefinitely.\n start_time: The start time of the request in seconds since epoch. Required if max_duration is set.\n \"\"\"\n if max_duration is None or (start_time is not None and time.time() - start_time < max_duration):\n retry_delay_override = os.environ.get(constants.EnvVar.RETRY_DELAY_OVERRIDE_SECONDS.value)\n if retry_delay_override is not None:\n delay = float(retry_delay_override)\n else:\n retry_after = float(response_retry_after)\n base_delay = float(base_delay)\n delay = min(retry_after, base_delay * (2**attempt))\n\n # modify output for proper plurality and formatting\n delay_str = f\"{delay:.0f}\" if delay.is_integer() else f\"{delay:.2f}\"\n second_str = \"second\" if delay == 1 else \"seconds\"\n prepend_message += \" \" if prepend_message else \"\"\n\n logger.info(\n f\"{constants.INDENT}{prepend_message}Checking again in {delay_str} {second_str} (Attempt {attempt})...\"\n )\n time.sleep(delay)\n else:\n elapsed = time.time() - start_time if start_time is not None else 0\n msg = f\"Maximum execution duration ({max_duration} seconds) exceeded after {elapsed:.1f} seconds.\"\n raise Exception(msg)\n\n\ndef _format_invoke_log(response: requests.Response, method: str, url: str, body: str) -> str:\n \"\"\"\n Format the log message for the invoke method.\n\n Args:\n response: The response object from the HTTP request.\n method: The HTTP method used in the request.\n url: The URL used in the request.\n body: The JSON body used in the request.\n \"\"\"\n message = [\n f\"\\nURL: {url}\",\n f\"Method: {method}\",\n (f\"Request Body:\\n{json.dumps(body, indent=4)}\" if body else \"Request Body: None\"),\n ]\n if response is not None:\n message.extend([\n f\"Response Status: {response.status_code}\",\n \"Response Headers:\",\n json.dumps(dict(response.headers), indent=4),\n \"Response Body:\",\n (\n json.dumps(response.json(), indent=4)\n if response.headers.get(\"Content-Type\") == \"application/json\"\n else response.text\n ),\n \"\",\n ])\n\n return \"\\n\".join(message)\n" }, { "path": "src/fabric_cicd/_common/_file.py", "content": "# Copyright (c) Microsoft Corporation.\n# Licensed under the MIT License.\n\n\"\"\"Functions and classes to manage file operations.\"\"\"\n\nimport base64\nimport logging\nfrom dataclasses import dataclass, field\nfrom pathlib import Path\nfrom typing import ClassVar\n\nfrom fabric_cicd._common._check_utils import check_file_type\nfrom fabric_cicd._common._exceptions import FileTypeError\n\nlogger = logging.getLogger(__name__)\n\n\n@dataclass()\nclass File:\n \"\"\"A class to represent a single file in an item object.\"\"\"\n\n item_path: Path\n file_path: Path\n type: str = field(default=\"text\", init=False)\n contents: str = field(default=\"\", init=False)\n IMMUTABLE_FIELDS: ClassVar[set] = {\"item_path\", \"file_path\"}\n\n def __setattr__(self, key: str, value: any) -> None:\n \"\"\"\n Override setattr for 'immutable' fields.\n\n Args:\n key: The attribute name.\n value: The attribute value.\n \"\"\"\n if key in self.IMMUTABLE_FIELDS and hasattr(self, key):\n msg = f\"item {key} is immutable\"\n raise AttributeError(msg)\n\n # Image file contents cannot be set\n if key == \"contents\" and self.type != \"text\":\n msg = f\"item {key} is immutable for non text files\"\n raise AttributeError(msg)\n super().__setattr__(key, value)\n\n def __post_init__(self) -> None:\n \"\"\"After initializing the object, read the file contents and set the type.\"\"\"\n file_type = check_file_type(self.file_path)\n\n if file_type != \"text\":\n try:\n self.contents = self.file_path.read_bytes()\n except Exception as e:\n msg = (\n f\"Error reading file {self.file_path} as binary. \"\n f\"Please submit this as a bug https://github.com/microsoft/fabric-cicd/issues/new?template=1-bug.yml.md. Exception: {e}\"\n )\n FileTypeError(msg, logger)\n else:\n try:\n self.contents = self.file_path.read_text(encoding=\"utf-8\")\n except Exception as e:\n msg = (\n f\"Error reading file {self.file_path} as text. \"\n f\"Please submit this as a bug https://github.com/microsoft/fabric-cicd/issues/new?template=1-bug.yml.md. Exception: {e}\"\n )\n FileTypeError(msg, logger)\n\n # set after as image contents are now immutable\n self.type = file_type\n\n @property\n def name(self) -> str:\n \"\"\"Return the file name.\"\"\"\n return self.file_path.name\n\n @property\n def relative_path(self) -> str:\n \"\"\"Return the relative path of the file.\"\"\"\n return str(self.file_path.relative_to(self.item_path).as_posix())\n\n @property\n def base64_payload(self) -> dict:\n \"\"\"Return the file contents as a base64 encoded payload.\"\"\"\n byte_file = self.contents.encode(\"utf-8\") if self.type == \"text\" else self.contents\n\n return {\n \"path\": self.relative_path,\n \"payload\": base64.b64encode(byte_file).decode(\"utf-8\"),\n \"payloadType\": \"InlineBase64\",\n }\n" }, { "path": "src/fabric_cicd/_common/_file_lock.py", "content": "# Copyright (c) Microsoft Corporation.\n# Licensed under the MIT License.\n\n\"\"\"Cross-platform file locking.\"\"\"\n\nimport sys\nfrom pathlib import Path\nfrom types import TracebackType\nfrom typing import Callable, Optional, TypeVar\n\nT = TypeVar(\"T\")\n\n\nclass FileLock:\n \"\"\"File lock context manager.\"\"\"\n\n def __init__(self, lock_file: str) -> None:\n self.lock_path = Path(f\"{lock_file}.lock\")\n self._lock_file: Optional[object] = None\n\n def __enter__(self) -> \"FileLock\":\n self._lock_file = self.lock_path.open(\"w\")\n if sys.platform == \"win32\":\n import msvcrt\n\n msvcrt.locking(self._lock_file.fileno(), msvcrt.LK_LOCK, 1)\n else:\n import fcntl\n\n fcntl.flock(self._lock_file.fileno(), fcntl.LOCK_EX)\n return self\n\n def __exit__(\n self,\n exc_type: Optional[type[BaseException]],\n exc_val: Optional[BaseException],\n exc_tb: Optional[TracebackType],\n ) -> bool:\n if self._lock_file:\n if sys.platform == \"win32\":\n import msvcrt\n\n msvcrt.locking(self._lock_file.fileno(), msvcrt.LK_UNLCK, 1)\n else:\n import fcntl\n\n fcntl.flock(self._lock_file.fileno(), fcntl.LOCK_UN)\n self._lock_file.close()\n return False\n\n @staticmethod\n def run_with_lock(lock_file: str, func: Callable[[], T]) -> T:\n \"\"\"\n Execute a function while holding an exclusive file lock.\n\n Args:\n lock_file: Path to the file to lock (a .lock suffix will be added)\n func: The function to execute while holding the lock\n\n Returns:\n The return value of the function\n \"\"\"\n with FileLock(lock_file):\n return func()\n" }, { "path": "src/fabric_cicd/_common/_git_diff_utils.py", "content": "# Copyright (c) Microsoft Corporation.\n# Licensed under the MIT License.\n\n\"\"\"Utility functions for detecting Fabric items changed via git diff.\"\"\"\n\nimport json\nimport logging\nimport subprocess\nfrom pathlib import Path\nfrom typing import Optional\n\nlogger = logging.getLogger(__name__)\n\n\ndef _find_platform_item(file_path: Path, repo_root: Path) -> Optional[tuple[str, str]]:\n \"\"\"\n Walk up from file_path towards repo_root looking for a .platform file.\n\n The .platform file marks the boundary of a Fabric item directory.\n Its JSON content contains ``metadata.type`` (item type) and\n ``metadata.displayName`` (item name).\n\n Returns:\n A ``(item_name, item_type)`` tuple, or ``None`` if not found.\n \"\"\"\n current = file_path.parent\n while True:\n platform_file = current / \".platform\"\n if platform_file.exists():\n try:\n data = json.loads(platform_file.read_text(encoding=\"utf-8\"))\n metadata = data.get(\"metadata\", {})\n item_type = metadata.get(\"type\")\n item_name = metadata.get(\"displayName\") or current.name\n if item_type:\n return item_name, item_type\n except Exception as exc:\n logger.debug(f\"Could not parse .platform file at '{platform_file}': {exc}\")\n # Stop if we have reached the repository root or the filesystem root\n if current == repo_root or current == current.parent:\n break\n current = current.parent\n return None\n\n\ndef _resolve_git_diff_path(\n file_path_str: str,\n git_root: Path,\n repository_directory: Path,\n) -> Optional[Path]:\n \"\"\"\n Resolve and validate a file path from git diff output.\n\n Follows the same resolve → boundary-check → reject contract as\n ``_resolve_file_path`` in ``_parameter/_utils.py``, adapted for\n paths that are relative to a git root with containment checked\n against a (potentially different) repository subdirectory.\n\n Args:\n file_path_str: Relative path string from git diff output.\n git_root: Resolved absolute path of the git repository root.\n repository_directory: Resolved absolute path of the configured\n repository directory (may be a subdirectory of git_root).\n\n Returns:\n Resolved absolute Path if valid and within boundary, None otherwise.\n \"\"\"\n raw_path = Path(file_path_str)\n\n # Reject absolute paths — git diff should only produce relative paths\n if raw_path.is_absolute():\n logger.debug(f\"get_changed_items: skipping absolute path '{file_path_str}'\")\n return None\n\n # Reject traversal sequences before resolution (mirrors _validate_wildcard_syntax)\n if \"..\" in raw_path.parts:\n logger.debug(f\"get_changed_items: skipping path with traversal '{file_path_str}'\")\n return None\n\n # Reject null bytes\n if \"\\x00\" in file_path_str:\n logger.debug(\"get_changed_items: skipping path with null bytes\")\n return None\n\n # Step 1: Resolve relative to git root (analogous to _resolve_file_path Step 1)\n resolved_path = (git_root / file_path_str).resolve()\n\n # Step 2: Boundary check against repository_directory (analogous to _resolve_file_path Step 2)\n try:\n resolved_path.relative_to(repository_directory)\n except ValueError:\n return None\n\n # Note: No Step 3 (existence check) — deleted files won't exist on disk\n return resolved_path\n\n\ndef get_changed_items(\n repository_directory: Path,\n git_compare_ref: str = \"HEAD~1\",\n) -> list[str]:\n \"\"\"\n Return the list of Fabric items that were added, modified, or renamed relative to ``git_compare_ref``.\n\n The returned list is in ``\"item_name.item_type\"`` format and can be passed directly\n to the ``items_to_include`` parameter of :func:`publish_all_items` to deploy only\n what has changed since the last commit.\n\n Args:\n repository_directory: Path to the local git repository directory\n (e.g. ``FabricWorkspace.repository_directory``).\n git_compare_ref: Git ref to compare against. Defaults to ``\"HEAD~1\"``.\n\n Returns:\n List of strings in ``\"item_name.item_type\"`` format. Returns an empty list when\n no changes are detected, the git root cannot be found, or git is unavailable.\n\n Examples:\n Deploy only changed items\n >>> from azure.identity import AzureCliCredential\n >>> from fabric_cicd import FabricWorkspace, publish_all_items, get_changed_items\n >>> workspace = FabricWorkspace(\n ... workspace_id=\"your-workspace-id\",\n ... repository_directory=\"/path/to/repo\",\n ... item_type_in_scope=[\"Notebook\", \"DataPipeline\"],\n ... token_credential=AzureCliCredential()\n ... )\n >>> changed = get_changed_items(workspace.repository_directory)\n >>> if changed:\n ... publish_all_items(workspace, items_to_include=changed)\n\n With a custom git ref\n >>> changed = get_changed_items(workspace.repository_directory, git_compare_ref=\"main\")\n >>> if changed:\n ... publish_all_items(workspace, items_to_include=changed)\n \"\"\"\n changed, _ = _resolve_changed_items(Path(repository_directory), git_compare_ref)\n return changed\n\n\ndef _resolve_changed_items(\n repository_directory: Path,\n git_compare_ref: str,\n) -> tuple[list[str], list[str]]:\n \"\"\"\n Use ``git diff --name-status`` to detect Fabric items that changed or were\n deleted relative to *git_compare_ref*.\n\n Args:\n repository_directory: Absolute path to the local repository directory\n (as stored on ``FabricWorkspace.repository_directory``).\n git_compare_ref: Git ref to diff against (e.g. ``\"HEAD~1\"``).\n\n Returns:\n A two-element tuple ``(changed_items, deleted_items)`` where each\n element is a list of strings in ``\"item_name.item_type\"`` format.\n Both lists are empty when the git root cannot be found or git fails.\n \"\"\"\n from fabric_cicd._common._config_validator import _find_git_root\n from fabric_cicd._common._validate_input import validate_git_compare_ref\n\n validate_git_compare_ref(git_compare_ref)\n\n git_root = _find_git_root(repository_directory)\n if git_root is None:\n logger.warning(\"get_changed_items: could not locate a git repository root — returning empty list.\")\n return [], []\n\n try:\n result = subprocess.run(\n [\"git\", \"diff\", \"--name-status\", git_compare_ref],\n cwd=str(git_root),\n capture_output=True,\n text=True,\n check=True,\n timeout=30,\n )\n except subprocess.CalledProcessError as exc:\n logger.warning(f\"get_changed_items: 'git diff' failed ({exc.stderr.strip()}) — returning empty list.\")\n return [], []\n except subprocess.TimeoutExpired:\n logger.warning(\"get_changed_items: 'git diff' timed out — returning empty list.\")\n return [], []\n\n changed_items: set[str] = set()\n deleted_items: set[str] = set()\n\n git_root_resolved = git_root.resolve()\n repo_dir_resolved = repository_directory.resolve()\n\n for line in result.stdout.splitlines():\n line = line.strip()\n if not line:\n continue\n\n parts = line.split(\"\\t\")\n status = parts[0].strip()\n\n # Renames produce three tab-separated fields: R\\told\\tnew\n if status.startswith(\"R\") and len(parts) >= 3:\n file_path_str = parts[2]\n elif len(parts) >= 2:\n file_path_str = parts[1]\n else:\n continue\n\n abs_path = _resolve_git_diff_path(file_path_str, git_root_resolved, repo_dir_resolved)\n if abs_path is None:\n continue\n\n if status == \"D\":\n if abs_path.name == \".platform\":\n try:\n show_result = subprocess.run(\n [\"git\", \"show\", f\"{git_compare_ref}:{file_path_str}\"],\n cwd=str(git_root_resolved),\n capture_output=True,\n text=True,\n check=True,\n timeout=30,\n )\n data = json.loads(show_result.stdout)\n metadata = data.get(\"metadata\", {})\n item_type = metadata.get(\"type\")\n item_name = metadata.get(\"displayName\") or abs_path.parent.name\n if item_type and item_name:\n deleted_items.add(f\"{item_name}.{item_type}\")\n except Exception as exc:\n logger.debug(f\"get_changed_items: could not read deleted .platform '{file_path_str}': {exc}\")\n else:\n item_info = _find_platform_item(abs_path, repo_dir_resolved)\n if item_info:\n changed_items.add(f\"{item_info[0]}.{item_info[1]}\")\n\n return list(changed_items), list(deleted_items)\n" }, { "path": "src/fabric_cicd/_common/_http_tracer.py", "content": "# Copyright (c) Microsoft Corporation.\n# Licensed under the MIT License.\n\n\"\"\"HTTP request/response tracer for debugging and mock server generation.\"\"\"\n\nimport base64\nimport hashlib\nimport json\nimport logging\nimport os\nimport uuid\nfrom dataclasses import asdict, dataclass\nfrom datetime import date, datetime, timezone\nfrom pathlib import Path\nfrom typing import Any, Optional, Protocol\nfrom urllib.parse import urlparse\n\nimport requests\n\nfrom fabric_cicd._common._file_lock import FileLock\nfrom fabric_cicd.constants import AUTHORIZATION_HEADER, EnvVar\n\nlogger = logging.getLogger(__name__)\n\n\ndef _trace_default(obj: object) -> str:\n \"\"\"Deterministic JSON fallback for non-native objects in HTTP traces.\"\"\"\n if isinstance(obj, (datetime, date)):\n return obj.isoformat()\n if isinstance(obj, uuid.UUID):\n return str(obj)\n if isinstance(obj, (bytes, bytearray)):\n try:\n return bytes(obj).decode(\"utf-8\")\n except UnicodeDecodeError:\n return base64.b64encode(bytes(obj)).decode(\"ascii\")\n return f\"\"\n\n\n@dataclass\nclass HTTPRequest:\n \"\"\"Represents an HTTP request with metadata.\"\"\"\n\n method: str\n url: str\n headers: dict[str, str]\n body: Any\n timestamp: str\n\n def to_b64(self) -> str:\n \"\"\"Serialize to base64-encoded JSON.\"\"\"\n request_json = json.dumps(asdict(self), separators=(\",\", \":\"), default=_trace_default)\n return base64.b64encode(request_json.encode()).decode()\n\n @classmethod\n def from_b64(cls, b64_str: str) -> \"HTTPRequest\":\n \"\"\"Deserialize from base64-encoded JSON.\"\"\"\n json_str = base64.b64decode(b64_str).decode()\n data = json.loads(json_str)\n return cls(**data)\n\n def get_unique_signature(self) -> str:\n \"\"\"Generate unique signature from URL, method, and body using SHA256.\"\"\"\n body_str = json.dumps(self.body, sort_keys=True) if isinstance(self.body, dict) else str(self.body or \"\")\n return hashlib.sha256(f\"{self.url}{self.method}{body_str}\".encode()).hexdigest()\n\n def get_route_key(self) -> str:\n \"\"\"Extract route key (method + path + query) from the request.\"\"\"\n try:\n parsed_url = urlparse(self.url)\n route = parsed_url.path\n if parsed_url.query:\n route += f\"?{parsed_url.query}\"\n return f\"{self.method} {route}\"\n except Exception:\n return \"\"\n\n\n@dataclass\nclass HTTPResponse:\n \"\"\"Represents an HTTP response with metadata.\"\"\"\n\n status_code: int\n headers: dict[str, str]\n body: Any\n timestamp: str\n\n def to_b64(self) -> str:\n \"\"\"Serialize to base64-encoded JSON.\"\"\"\n response_json = json.dumps(asdict(self), separators=(\",\", \":\"), default=_trace_default)\n return base64.b64encode(response_json.encode()).decode()\n\n @classmethod\n def from_b64(cls, b64_str: str) -> \"HTTPResponse\":\n \"\"\"Deserialize from base64-encoded JSON.\"\"\"\n json_str = base64.b64decode(b64_str).decode()\n data = json.loads(json_str)\n return cls(**data)\n\n def get_unique_signature(self) -> str:\n \"\"\"Generate unique signature from status code and body using SHA256.\"\"\"\n body_str = json.dumps(self.body, sort_keys=True) if isinstance(self.body, dict) else str(self.body or \"\")\n return hashlib.sha256(f\"{self.status_code}{body_str}\".encode()).hexdigest()\n\n\nclass HTTPTracer(Protocol):\n \"\"\"Protocol for HTTP request/response tracers.\"\"\"\n\n def capture_request(self, method: str, url: str, headers: dict, body: str, files: Optional[dict]) -> None:\n \"\"\"Capture HTTP request details.\"\"\"\n ...\n\n def capture_response(self, response: requests.Response) -> None:\n \"\"\"Capture HTTP response details.\"\"\"\n ...\n\n def save(self) -> None:\n \"\"\"Save captured data.\"\"\"\n ...\n\n\nclass NoOpTracer:\n \"\"\"No-op tracer that does nothing.\"\"\"\n\n def capture_request(self, method: str, url: str, headers: dict, body: str, files: Optional[dict]) -> None:\n \"\"\"No-op capture request.\"\"\"\n pass\n\n def capture_response(self, response: requests.Response) -> None:\n \"\"\"No-op capture response.\"\"\"\n pass\n\n def save(self) -> None:\n \"\"\"No-op save.\"\"\"\n pass\n\n\nclass FileTracer:\n \"\"\"Captures HTTP requests and responses to a JSON file.\"\"\"\n\n def __init__(self, output_file: Optional[str] = None) -> None:\n \"\"\"\n Initialize the file tracer.\n\n Args:\n output_file: Path to save the trace file. If None, checks EnvVar.HTTP_TRACE_FILE.\n \"\"\"\n trace_file_from_env = os.environ.get(EnvVar.HTTP_TRACE_FILE.value)\n\n if output_file is None:\n self.output_file = trace_file_from_env if trace_file_from_env else \"http_trace.json\"\n else:\n self.output_file = output_file\n\n self.captures: list[dict] = []\n\n def capture_request(self, method: str, url: str, headers: dict, body: str, files: Optional[dict]) -> None: # noqa: ARG002\n \"\"\"\n Capture HTTP request details with base64 encoding, if enabled.\n\n Args:\n method: HTTP method (GET, POST, etc.)\n url: Request URL\n headers: Request headers. Note: Authorization headers are excluded.\n body: Request body\n files: Files being uploaded\n \"\"\"\n request = HTTPRequest(\n method=method,\n url=url,\n headers={k: v for k, v in headers.items() if k.lower() not in [AUTHORIZATION_HEADER]},\n body=body,\n timestamp=datetime.now(timezone.utc).isoformat(),\n )\n\n self.captures.append({\"request_b64\": request.to_b64(), \"response_b64\": None})\n\n def capture_response(self, response: requests.Response) -> None:\n \"\"\"\n Add response data to the most recent capture entry.\n\n Args:\n response: The HTTP response object\n \"\"\"\n if not self.captures:\n return\n\n try:\n if hasattr(response, \"json\") and \"application/json\" in response.headers.get(\"Content-Type\", \"\"):\n response_body = response.json()\n else:\n response_body = response.text if hasattr(response, \"text\") else \"\"\n except Exception:\n response_body = \"\"\n\n http_response = HTTPResponse(\n status_code=response.status_code,\n headers=dict(response.headers),\n body=response_body,\n timestamp=datetime.now(timezone.utc).isoformat(),\n )\n\n self.captures[-1][\"response_b64\"] = http_response.to_b64()\n\n def save(self) -> None:\n \"\"\"Save all HTTP captures to a JSON file.\"\"\"\n if not self.captures:\n return\n\n try:\n FileLock.run_with_lock(self.output_file, self._flush_traces_to_file)\n except Exception as e:\n logger.warning(f\"Failed to save HTTP trace: {e}\")\n\n def _flush_traces_to_file(self) -> None:\n \"\"\"Flush captured traces to the output file (called within lock).\"\"\"\n output_path = Path(self.output_file)\n existing_traces: list[dict] = []\n if output_path.exists() and output_path.stat().st_size > 0:\n with output_path.open(\"r\") as f:\n existing_data = json.load(f)\n existing_traces = existing_data.get(\"traces\", [])\n\n for capture in self.captures:\n request_b64 = capture.get(\"request_b64\", \"\")\n response_b64 = capture.get(\"response_b64\", \"\")\n\n request_data = None\n response_data = None\n\n if request_b64:\n request_data = json.loads(base64.b64decode(request_b64).decode())\n if response_b64:\n response_data = json.loads(base64.b64decode(response_b64).decode())\n\n existing_traces.append({\"request\": request_data, \"response\": response_data})\n\n existing_traces.sort(key=lambda x: x[\"request\"].get(\"timestamp\", \"\") if x.get(\"request\") else \"\")\n output_data = {\n \"description\": \"HTTP trace data from Fabric API interactions\",\n \"total_traces\": len(existing_traces),\n \"traces\": existing_traces,\n }\n\n with output_path.open(\"w\") as f:\n json.dump(output_data, f, indent=2)\n\n\nclass HTTPTracerFactory:\n \"\"\"Factory class for creating HTTP tracer instances.\"\"\"\n\n @staticmethod\n def create() -> HTTPTracer:\n \"\"\"\n Create an HTTP tracer based on environment configuration.\n\n Returns:\n FileTracer if tracing is enabled via environment variable, NoOpTracer otherwise.\n \"\"\"\n from fabric_cicd.constants import VALID_ENABLE_FLAGS\n\n trace_enabled = os.environ.get(EnvVar.HTTP_TRACE_ENABLED.value, \"\").lower() in VALID_ENABLE_FLAGS\n return FileTracer() if trace_enabled else NoOpTracer()\n" }, { "path": "src/fabric_cicd/_common/_item.py", "content": "# Copyright (c) Microsoft Corporation.\n# Licensed under the MIT License.\n\n\"\"\"Functions and classes to manage Item operations.\"\"\"\n\nimport os\nfrom dataclasses import dataclass, field\nfrom pathlib import Path\nfrom typing import ClassVar\n\nfrom fabric_cicd._common._file import File\n\n\n@dataclass\nclass Item:\n \"\"\"A class to represent a single item.\"\"\"\n\n type: str\n name: str\n description: str\n guid: str\n logical_id: str = field(default=\"\")\n path: Path = field(default_factory=Path)\n item_files: list = field(default_factory=list)\n folder_id: str = field(default=\"\")\n folder_path: str = field(default=\"\")\n IMMUTABLE_FIELDS: ClassVar[set] = {\"type\", \"name\", \"description\"}\n skip_publish: bool = field(default=False)\n\n def __setattr__(self, key: str, value: any) -> None:\n \"\"\"\n Override setattr for 'immutable' fields.\n\n Args:\n key: The attribute name.\n value: The attribute value.\n \"\"\"\n if key in self.IMMUTABLE_FIELDS and hasattr(self, key):\n msg = f\"item {key} is immutable\"\n raise AttributeError(msg)\n super().__setattr__(key, value)\n\n @property\n def relative_path(self) -> str:\n \"\"\"Return the relative path of the file.\"\"\"\n return str(self.file_path.relative_to(self.item_path).as_posix())\n\n def collect_item_files(self) -> None:\n \"\"\"Collect all files in the item path.\"\"\"\n self.item_files = []\n for root, _dirs, files in os.walk(self.path):\n for file in files:\n full_path = Path(root, file)\n self.item_files.append(File(self.path, full_path))\n" }, { "path": "src/fabric_cicd/_common/_logging.py", "content": "# Copyright (c) Microsoft Corporation.\n# Licensed under the MIT License.\n\n\"\"\"Logging utilities for the fabric_cicd package.\"\"\"\n\nimport inspect\nimport logging\nimport re\nimport sys\nimport traceback\nfrom logging import LogRecord\nfrom logging.handlers import RotatingFileHandler\nfrom pathlib import Path\nfrom typing import ClassVar, Optional, Union\n\nfrom fabric_cicd import constants\nfrom fabric_cicd._common import _exceptions\nfrom fabric_cicd._common._color import Fore, Style\n\n\nclass CustomFormatter(logging.Formatter):\n LEVEL_COLORS: ClassVar[dict[str, str]] = {\n \"DEBUG\": Fore.BLACK,\n \"INFO\": Fore.WHITE + Style.BRIGHT,\n \"WARNING\": Fore.YELLOW,\n \"ERROR\": Fore.RED,\n \"CRITICAL\": Style.BRIGHT + Fore.RED,\n }\n\n def format(self, record: LogRecord) -> str:\n level_color = self.LEVEL_COLORS.get(record.levelname, \"\")\n level_name = {\n \"WARNING\": \"warn\",\n \"DEBUG\": \"debug\",\n \"INFO\": \"info\",\n \"ERROR\": \"error\",\n \"CRITICAL\": \"crit\",\n }.get(record.levelname, \"unknown\")\n\n level_name = f\"{level_color}[{level_name}]\"\n timestamp = f\"{self.formatTime(record, self.datefmt)}\"\n message = f\"{record.getMessage()}{Style.RESET_ALL}\"\n\n # indent if the message contains \"->\"\n if constants.INDENT in message:\n message = message.replace(constants.INDENT, \"\")\n full_message = f\"{' ' * 8} {timestamp} - {message}\"\n else:\n # Calculate visual length by removing ANSI escape codes\n\n ansi_escape = re.compile(r\"\\x1B(?:[@-Z\\\\-_]|\\[[0-?]*[ -/]*[@-~])\")\n\n # Get visual length of level_name without ANSI codes\n visual_level_length = len(ansi_escape.sub(\"\", level_name))\n # Pad to 16 visual characters\n padding = \" \" * max(0, 8 - visual_level_length)\n\n full_message = f\"{level_name}{padding} {timestamp} - {message}\"\n return full_message\n\n\nclass PackageFilter(logging.Filter):\n \"\"\"\n Filter that only allows records from the fabric_cicd package logs.\n\n Args:\n debug_only: If True, only allows DEBUG level records. If False, allows all levels.\n \"\"\"\n\n def __init__(self, debug_only: bool = False) -> None:\n super().__init__()\n self.debug_only = debug_only\n\n def filter(self, record: LogRecord) -> bool:\n is_fabric_cicd = record.name.startswith(\"fabric_cicd\")\n if self.debug_only:\n return is_fabric_cicd and record.levelno == logging.DEBUG\n return is_fabric_cicd\n\n\n\"\"\"Helper functions to configure logging and handle exceptions across the fabric_cicd package.\"\"\"\n\n_DEFAULT_LOG_FILENAME = \"fabric_cicd.error.log\"\n_DEFAULT_LOG_FILE_FORMATTER = \"%(asctime)s - %(levelname)s - %(name)s - %(message)s\"\n_FABRIC_CICD_HANDLER_ATTR = \"_fabric_cicd_managed\"\n_FABRIC_CICD_EXTERNAL_HANDLER_ATTR = \"_fabric_cicd_external\"\n\n\ndef _cleanup_external_handler_filters(root_logger: logging.Logger) -> None:\n \"\"\"Remove PackageFilter from any external handler previously configured by fabric_cicd.\"\"\"\n for handler in list(root_logger.handlers):\n if getattr(handler, _FABRIC_CICD_EXTERNAL_HANDLER_ATTR, False):\n # Remove all PackageFilters that were added\n for f in list(handler.filters):\n if isinstance(f, PackageFilter):\n handler.removeFilter(f)\n # Remove the marker attribute\n delattr(handler, _FABRIC_CICD_EXTERNAL_HANDLER_ATTR)\n # Remove handler from root logger (don't close it - caller owns it)\n root_logger.removeHandler(handler)\n\n\ndef _cleanup_managed_handlers(*loggers: logging.Logger) -> None:\n \"\"\"Close and remove only handlers previously added by fabric_cicd.\"\"\"\n for logger_instance in loggers:\n # First, clean up any external handlers configured (filters only, don't close)\n _cleanup_external_handler_filters(logger_instance)\n\n # Then clean up fabric_cicd-managed handlers (close and remove)\n for handler in list(logger_instance.handlers):\n if getattr(handler, _FABRIC_CICD_HANDLER_ATTR, False):\n handler.close()\n logger_instance.removeHandler(handler)\n\n\ndef _mark_handler(handler: logging.Handler) -> logging.Handler:\n \"\"\"Mark a handler as managed by fabric_cicd.\"\"\"\n setattr(handler, _FABRIC_CICD_HANDLER_ATTR, True)\n return handler\n\n\ndef _mark_external_handler(handler: logging.Handler) -> logging.Handler:\n \"\"\"Mark an external handler as configured by fabric_cicd (for filter cleanup only).\"\"\"\n setattr(handler, _FABRIC_CICD_EXTERNAL_HANDLER_ATTR, True)\n return handler\n\n\ndef _configure_default_file_handler() -> logging.Handler:\n \"\"\"Configure the default file handler for standalone fabric_cicd usage.\"\"\"\n handler = logging.FileHandler(\n filename=_DEFAULT_LOG_FILENAME,\n mode=\"w\",\n delay=True,\n )\n handler.setFormatter(logging.Formatter(_DEFAULT_LOG_FILE_FORMATTER))\n handler.addFilter(PackageFilter()) # All levels from fabric_cicd package logs\n\n return _mark_handler(handler)\n\n\ndef _configure_external_file_handler(\n external_handler: Union[logging.FileHandler, RotatingFileHandler],\n level: int,\n debug_only_file: bool,\n) -> logging.Handler:\n \"\"\"\n Configure an external file handler for fabric_cicd package logs.\n\n Reuses the external handler directly to preserve rotation behavior (if any).\n The external handler is not marked as fabric_cicd-managed (won't be closed),\n but is marked as external so filters can be cleaned up on reconfiguration.\n\n Note: Any existing PackageFilter is already removed by _cleanup_managed_handlers()\n before this function is called.\n \"\"\"\n # Add the appropriate filter\n if level == logging.DEBUG and debug_only_file:\n external_handler.addFilter(PackageFilter(debug_only=True))\n else:\n external_handler.addFilter(PackageFilter())\n\n # Mark as external in order to clean up filters later (but don't close it)\n return _mark_external_handler(external_handler)\n\n\ndef _configure_console_handler(level: int) -> logging.StreamHandler:\n \"\"\"Configure a console handler with the standard fabric_cicd formatter.\"\"\"\n handler = logging.StreamHandler()\n handler.setLevel(level)\n handler.setFormatter(\n CustomFormatter(\n \"[%(levelname)s] %(asctime)s - %(message)s\",\n datefmt=\"%H:%M:%S\",\n )\n )\n return _mark_handler(handler)\n\n\ndef _build_console_message(exception: BaseException, file_handler: Optional[logging.FileHandler] = None) -> str:\n \"\"\"Build the user-facing console error message, optionally referencing the log file.\"\"\"\n # Write exception to console when file logging is disabled or when using an external file handler\n if file_handler is None or Path(file_handler.baseFilename).name != _DEFAULT_LOG_FILENAME:\n return f\"{exception!s}\"\n\n # Only reference the default log file which contains full error details\n log_file_path = Path(file_handler.baseFilename).resolve()\n return f\"{exception!s}\\n\\nSee {log_file_path} for full details.\"\n\n\ndef _build_file_message(exception: BaseException) -> str:\n \"\"\"Build the log file message, including additional info if available.\"\"\"\n additional_info = getattr(exception, \"additional_info\", None)\n if additional_info is not None:\n return f\"%s\\n\\nAdditional Info: \\n{additional_info}\"\n return \"%s\"\n\n\n\"\"\"Main logging configuration and exception handling functions for fabric_cicd.\"\"\"\n\n\ndef get_file_handler(\n logger: Optional[logging.Logger] = None,\n) -> Optional[Union[logging.FileHandler, RotatingFileHandler]]:\n \"\"\"\n Get a file handler from a logger.\n\n Args:\n logger: The logger to search for a file handler. If None, searches the root logger\n for fabric_cicd-managed handlers only.\n\n Returns:\n The first FileHandler or RotatingFileHandler found, or None if not found.\n \"\"\"\n target_logger = logger if logger is not None else logging.getLogger()\n\n # When searching root logger, only return fabric_cicd-managed handlers\n check_managed = logger is None\n\n return next(\n (\n h\n for h in target_logger.handlers\n if isinstance(h, (logging.FileHandler, RotatingFileHandler))\n and (getattr(h, _FABRIC_CICD_HANDLER_ATTR, False) if check_managed else True)\n ),\n None,\n )\n\n\ndef configure_logger(\n level: int = logging.INFO,\n suppress_debug_console: bool = False,\n debug_only_file: bool = False,\n disable_log_file: bool = False,\n external_file_handler: Optional[Union[logging.FileHandler, RotatingFileHandler]] = None,\n) -> None:\n \"\"\"\n Configure the logger.\n\n Args:\n level: The log level to set. Must be one of the standard logging levels.\n suppress_debug_console: Suppress DEBUG output to console (only applies when level is DEBUG).\n debug_only_file: Only write DEBUG messages to file (only applies when level is DEBUG).\n disable_log_file: Disable file logging entirely.\n external_file_handler: External file handler to append logs to instead of creating the default one.\n \"\"\"\n # Determine console level - suppress DEBUG to console if specified, otherwise same as level\n console_level = logging.INFO if suppress_debug_console and level == logging.DEBUG else level\n\n # Get all loggers\n root_logger = logging.getLogger()\n package_logger = logging.getLogger(\"fabric_cicd\")\n console_only_logger = logging.getLogger(\"console_only\")\n\n # Close and remove old handlers before adding new ones\n # This also cleans up any PackageFilter added to external handlers\n _cleanup_managed_handlers(root_logger, package_logger, console_only_logger)\n\n # Root logger - receives propagated records from fabric_cicd loggers\n # Holds the file handler so all fabric_cicd.* child loggers write to file via propagation\n # Set root logger level - for non-fabric_cicd packages: INFO if DEBUG, else ERROR\n root_logger.setLevel(level=logging.INFO if level == logging.DEBUG else logging.ERROR)\n\n # Configure file handler based on parameters\n if external_file_handler is not None:\n # Use provided external file handler for fabric_cicd logs\n root_logger.addHandler(_configure_external_file_handler(external_file_handler, level, debug_only_file))\n\n elif not disable_log_file:\n # Use the default file handler for fabric_cicd logs\n root_logger.addHandler(_configure_default_file_handler())\n\n # Package logger - primary logger for all fabric_cicd library logging\n # Writes to console via its own handler and to file via propagation to root\n package_logger.setLevel(level)\n package_logger.addHandler(_configure_console_handler(console_level))\n\n # Console-only logger - used exclusively by exception_handler() to display\n # user-facing error messages on the terminal without writing them to the log file\n console_only_logger.setLevel(console_level)\n console_only_logger.addHandler(_configure_console_handler(console_level))\n console_only_logger.propagate = False\n\n\ndef exception_handler(exception_type: type[BaseException], exception: BaseException, traceback: traceback) -> None:\n \"\"\"\n Handle exceptions that are instances of any class from the _common._exceptions module.\n\n Args:\n exception_type: The type of the exception.\n exception: The exception instance.\n traceback: The traceback object.\n \"\"\"\n # Get all exception classes from the _common._exceptions module\n exception_classes = [cls for _, cls in inspect.getmembers(_exceptions, inspect.isclass)]\n\n # If the exception is not from _common._exceptions, use the default exception handler\n if not any(isinstance(exception, cls) for cls in exception_classes):\n sys.__excepthook__(exception_type, exception, traceback)\n return\n\n # Step 1: Write user-facing error message to console only (no file)\n file_handler = get_file_handler() # searches root logger for managed handlers\n console_message = _build_console_message(exception, file_handler)\n logging.getLogger(\"console_only\").error(console_message)\n\n # Step 2: Write full stack trace to file only (not terminal)\n # Only write to file if using fabric_cicd default file handler (includes ERROR level logs)\n is_default_file_handler = file_handler is not None and Path(file_handler.baseFilename).name == _DEFAULT_LOG_FILENAME\n if is_default_file_handler:\n # Remove console handler from package logger so stack trace doesn't print to terminal\n package_logger = logging.getLogger(\"fabric_cicd\")\n _cleanup_managed_handlers(package_logger)\n file_message = _build_file_message(exception)\n exception.logger.exception(file_message, exception, exc_info=(exception_type, exception, traceback))\n\n\ndef log_header(logger: logging.Logger, message: str) -> None:\n \"\"\"\n Logs a header message with a decorative line above and below it.\n\n Args:\n logger: The logger to use for logging the header message.\n message: The header message to log.\n \"\"\"\n line_separator = \"#\" * 100\n formatted_message = f\"########## {message}\"\n formatted_message = f\"{formatted_message} {line_separator[len(formatted_message) + 1 :]}\"\n\n logger.info(\"\") # Log a blank line before the header\n logger.info(f\"{Fore.GREEN}{Style.BRIGHT}{line_separator}{Style.RESET_ALL}\")\n logger.info(f\"{Fore.GREEN}{Style.BRIGHT}{formatted_message}{Style.RESET_ALL}\")\n logger.info(f\"{Fore.GREEN}{Style.BRIGHT}{line_separator}{Style.RESET_ALL}\")\n logger.info(\"\")\n" }, { "path": "src/fabric_cicd/_common/_validate_env_vars.py", "content": "# Copyright (c) Microsoft Corporation.\n# Licensed under the MIT License.\n\n\"\"\"Functions for validating environment variables and constants used by fabric-cicd.\"\"\"\n\nimport logging\nimport os\nimport re\nfrom urllib.parse import urlsplit\n\nfrom fabric_cicd._common._exceptions import InputError\n\nlogger = logging.getLogger(__name__)\n\n# Define a regular expression for valid hostnames\n# Matches: any subdomain of []api.fabric.microsoft.com or []api.powerbi.com\n_VALID_HOSTNAME_REGEX = re.compile(r\"^([\\w-]+\\.)*[\\w-]*api\\.(fabric\\.microsoft\\.com|powerbi\\.com)\\Z\", re.IGNORECASE)\n\n\n# Regular expression for valid GUIDs with dashes\nVALID_GUID_REGEX = r\"^[a-fA-F0-9]{8}-[a-fA-F0-9]{4}-[a-fA-F0-9]{4}-[a-fA-F0-9]{4}-[a-fA-F0-9]{12}$\"\n\n# Constants that hold API URLs and require URL validation\n_URL_CONSTANTS = {\"DEFAULT_API_ROOT_URL\", \"FABRIC_API_ROOT_URL\"}\n\n\ndef validate_api_url(url: str, label: str) -> str:\n \"\"\"\n Validates an API URL string.\n Validates the value is non-empty, the scheme is https, the hostname matches\n allowed patterns, and no path components are present.\n\n Args:\n url: The URL string to validate.\n label: A human-readable label for error messages (e.g., env var name or config key).\n\n Returns:\n str: The validated URL with trailing slashes removed.\n \"\"\"\n if not url.strip():\n msg = f\"'{label}' must resolve to a non-empty string.\"\n raise InputError(msg, logger)\n\n # Parse the URL using urlsplit\n parsed = urlsplit(url)\n\n if parsed.scheme != \"https\":\n msg = f\"Invalid or missing scheme in '{label}': '{url}'. URL must use HTTPS scheme.\"\n raise InputError(msg, logger)\n\n hostname = parsed.hostname or \"\"\n\n if not _VALID_HOSTNAME_REGEX.match(hostname):\n msg = f\"'{label}' has invalid hostname: {hostname}\"\n raise InputError(msg, logger)\n\n if parsed.path and parsed.path not in (\"\", \"/\"):\n msg = f\"'{label}' should be a root URL without path components. Got path: '{parsed.path}'\"\n raise InputError(msg, logger)\n\n return url.rstrip(\"/\")\n\n\ndef validate_env_var_api_url(env_var_name: str, default_value: str) -> str:\n \"\"\"\n Validates and returns the API URL from an environment variable.\n Validates the scheme is https and the hostname matches allowed patterns.\n\n Args:\n env_var_name: Name of the environment variable\n default_value: Default value if environment variable is not set (full URL with https://)\n\n Returns:\n str: The original validated API URL value, or the default if env var is not set.\n \"\"\"\n value = os.environ.get(env_var_name, default_value)\n return validate_api_url(value, f\"environment variable '{env_var_name}'\")\n\n\ndef _get_fabric_fqdn_url(workspace_id: str) -> str:\n \"\"\"\n Transform workspace ID to FQDN format for private-link-enabled Fabric workspaces.\n\n Args:\n workspace_id: The workspace ID string in standard GUID format with dashes\n (e.g., \"f953f3da-c5f0-4e36-a644-c85933e35e2f\").\n\n Returns:\n The FQDN URL string in the format:\n https://.z.w.api.fabric.microsoft.com\n\n Examples:\n >>> url = _get_fabric_fqdn_url(\"f953f3da-c5f0-4e36-a644-c85933e35e2f\")\n >>> url\n 'https://f953f3dac5f04e36a644c85933e35e2f.zf9.w.api.fabric.microsoft.com'\n \"\"\"\n if not re.match(VALID_GUID_REGEX, workspace_id):\n msg = f\"workspace_id must be a valid GUID with dashes, got: '{workspace_id}'\"\n raise ValueError(msg)\n no_dashes = workspace_id.replace(\"-\", \"\")\n first_two = no_dashes[:2]\n return f\"https://{no_dashes}.z{first_two}.w.api.fabric.microsoft.com\"\n" }, { "path": "src/fabric_cicd/_common/_validate_input.py", "content": "# Copyright (c) Microsoft Corporation.\n# Licensed under the MIT License.\n\n\"\"\"\nFollowing functions are leveraged to validate user input for the fabric-cicd package\nPrimarily used for the FabricWorkspace class, but also intended to be leveraged for\nany user input throughout the package\n\"\"\"\n\nimport logging\nimport re\nfrom pathlib import Path\nfrom typing import Optional\n\nfrom azure.core.credentials import TokenCredential\n\nimport fabric_cicd.constants as constants\nfrom fabric_cicd._common._exceptions import InputError\nfrom fabric_cicd.constants import FeatureFlag, OperationType\nfrom fabric_cicd.fabric_workspace import FabricWorkspace\n\nlogger = logging.getLogger(__name__)\n\n\ndef validate_data_type(expected_type: str, variable_name: str, input_value: any) -> any:\n \"\"\"\n Validate the data type of the input value.\n\n Args:\n expected_type: The expected data type.\n variable_name: The name of the variable.\n input_value: The input value to validate.\n \"\"\"\n # Mapping of expected types to their validation functions\n type_validators = {\n \"string\": lambda x: isinstance(x, str),\n \"bool\": lambda x: isinstance(x, bool),\n \"list\": lambda x: isinstance(x, list),\n \"list[string]\": lambda x: isinstance(x, list) and all(isinstance(item, str) for item in x),\n \"FabricWorkspace\": lambda x: isinstance(x, FabricWorkspace),\n \"TokenCredential\": lambda x: isinstance(x, TokenCredential),\n }\n\n # Check if the expected type is valid and if the input matches the expected type\n if expected_type not in type_validators or not type_validators[expected_type](input_value):\n msg = f\"The provided {variable_name} is not of type {expected_type}.\"\n raise InputError(msg, logger)\n\n return input_value\n\n\ndef validate_item_type_in_scope(input_value: Optional[list]) -> list:\n \"\"\"\n Validate the item type in scope.\n\n Args:\n input_value: The input value to validate. If None, defaults to all supported item types.\n \"\"\"\n accepted_item_types = constants.ACCEPTED_ITEM_TYPES\n\n # If None, return all accepted item types\n if input_value is None:\n return list(accepted_item_types)\n\n validate_data_type(\"list[string]\", \"item_type_in_scope\", input_value)\n\n for item_type in input_value:\n if item_type not in accepted_item_types:\n msg = f\"Invalid or unsupported item type: '{item_type}'. Must be one of {', '.join(accepted_item_types)}.\"\n raise InputError(msg, logger)\n\n return input_value\n\n\ndef validate_repository_directory(input_value: str) -> Path:\n \"\"\"\n Validate the repository directory and convert string to Path object\n\n Args:\n input_value: The input value to validate.\n \"\"\"\n validate_data_type(\"string\", \"repository_directory\", input_value)\n\n directory = Path(input_value)\n\n if not directory.is_dir():\n msg = f\"The provided repository_directory '{input_value}' does not exist.\"\n raise InputError(msg, logger)\n\n if not directory.is_absolute():\n absolute_directory = directory.resolve()\n logger.info(f\"Relative directory path '{directory}' resolved as '{absolute_directory}'\")\n directory = absolute_directory\n\n return directory\n\n\ndef validate_workspace_id(input_value: str) -> str:\n \"\"\"\n Validate the workspace ID.\n\n Args:\n input_value: The input value to validate.\n \"\"\"\n validate_data_type(\"string\", \"workspace_id\", input_value)\n\n if not re.match(constants.VALID_GUID_REGEX, input_value):\n msg = \"The provided workspace_id is not a valid guid.\"\n raise InputError(msg, logger)\n\n return input_value\n\n\ndef validate_workspace_name(input_value: str) -> str:\n \"\"\"\n Validate the workspace name.\n\n Args:\n input_value: The input value to validate.\n \"\"\"\n validate_data_type(\"string\", \"workspace_name\", input_value)\n\n return input_value\n\n\ndef validate_environment(input_value: str) -> str:\n \"\"\"\n Validate the environment.\n\n Args:\n input_value: The input value to validate.\n \"\"\"\n validate_data_type(\"string\", \"environment\", input_value)\n\n return input_value\n\n\ndef validate_fabric_workspace_obj(input_value: FabricWorkspace) -> FabricWorkspace:\n \"\"\"\n Validate the FabricWorkspace object.\n\n Args:\n input_value: The input value to validate.\n \"\"\"\n validate_data_type(\"FabricWorkspace\", \"fabric_workspace_obj\", input_value)\n\n return input_value\n\n\ndef validate_token_credential(input_value: TokenCredential) -> TokenCredential:\n \"\"\"\n Validate the token credential.\n\n Args:\n input_value: The input value to validate.\n \"\"\"\n validate_data_type(\"TokenCredential\", \"credential\", input_value)\n\n return input_value\n\n\ndef validate_experimental_param(\n param_value: Optional[str],\n required_flag: \"FeatureFlag\",\n warning_message: str,\n risk_warning: str,\n) -> None:\n \"\"\"\n Generic validation for optional parameters requiring experimental feature flags.\n\n Args:\n param_value: The parameter value (None means skip validation).\n required_flag: The specific feature flag required (in addition to experimental).\n warning_message: Primary warning message when feature is enabled.\n risk_warning: Risk/caution warning message.\n\n Raises:\n InputError: If required feature flags are not enabled.\n \"\"\"\n if param_value is None:\n return\n\n if (\n FeatureFlag.ENABLE_EXPERIMENTAL_FEATURES.value not in constants.FEATURE_FLAG\n or required_flag.value not in constants.FEATURE_FLAG\n ):\n msg = f\"Feature flags 'enable_experimental_features' and '{required_flag.value}' must be set.\"\n raise InputError(msg, logger)\n\n logger.warning(warning_message)\n logger.warning(risk_warning)\n\n\ndef validate_items_to_include(items_to_include: Optional[list[str]], operation: \"OperationType\") -> None:\n \"\"\"\n Validate items_to_include parameter and check required feature flags.\n\n Args:\n items_to_include: List of items in \"item_name.item_type\" format, or None.\n operation: The type of operation being performed (publish or unpublish).\n\n Raises:\n InputError: If required feature flags are not enabled.\n \"\"\"\n validate_experimental_param(\n param_value=items_to_include,\n required_flag=FeatureFlag.ENABLE_ITEMS_TO_INCLUDE,\n warning_message=f\"Selective {operation.value} is enabled.\",\n risk_warning=f\"Using items_to_include is risky as it can prevent needed dependencies from being {operation.value}. Use at your own risk.\",\n )\n\n\ndef validate_folder_path_exclude_regex(folder_path_exclude_regex: Optional[str]) -> None:\n \"\"\"\n Validate folder_path_exclude_regex parameter and check required feature flags.\n\n Args:\n folder_path_exclude_regex: Regex pattern to exclude items based on their folder path, or None.\n\n Raises:\n InputError: If required feature flags are not enabled.\n \"\"\"\n validate_experimental_param(\n param_value=folder_path_exclude_regex,\n required_flag=FeatureFlag.ENABLE_EXCLUDE_FOLDER,\n warning_message=\"Folder path exclusion is enabled.\",\n risk_warning=\"Using folder_path_exclude_regex is risky as it can prevent needed dependencies from being deployed. Use at your own risk.\",\n )\n\n if not isinstance(folder_path_exclude_regex, str):\n msg = \"folder_path_exclude_regex must be a string.\"\n raise InputError(msg, logger)\n\n if folder_path_exclude_regex == \"\":\n msg = \"folder_path_exclude_regex must not be an empty string. Provide a valid regex pattern or omit the parameter.\"\n raise InputError(msg, logger)\n\n\ndef validate_folder_path_to_include(folder_path_to_include: Optional[list[str]]) -> None:\n \"\"\"\n Validate folder_path_to_include parameter and check required feature flags.\n\n Args:\n folder_path_to_include: List of folder paths with format [\"/folder1\", \"/folder2\", ...], or None.\n\n Raises:\n InputError: If required feature flags are not enabled.\n \"\"\"\n validate_experimental_param(\n param_value=folder_path_to_include,\n required_flag=FeatureFlag.ENABLE_INCLUDE_FOLDER,\n warning_message=\"Folder path inclusion is enabled.\",\n risk_warning=\"Using folder_path_to_include is risky as it can prevent needed dependencies from being deployed. Use at your own risk.\",\n )\n\n if not isinstance(folder_path_to_include, list):\n msg = \"folder_path_to_include must be a list of folder paths.\"\n raise InputError(msg, logger)\n\n if not folder_path_to_include:\n msg = \"folder_path_to_include must not be an empty list. Provide folder paths or omit the parameter.\"\n raise InputError(msg, logger)\n\n\ndef validate_shortcut_exclude_regex(shortcut_exclude_regex: Optional[str]) -> None:\n \"\"\"\n Validate shortcut_exclude_regex parameter and check required feature flags.\n\n Args:\n shortcut_exclude_regex: Regex pattern to exclude specific shortcuts from being published, or None.\n\n Raises:\n InputError: If required feature flags are not enabled.\n \"\"\"\n validate_experimental_param(\n param_value=shortcut_exclude_regex,\n required_flag=FeatureFlag.ENABLE_SHORTCUT_EXCLUDE,\n warning_message=\"Shortcut exclusion is enabled.\",\n risk_warning=\"Using shortcut_exclude_regex will selectively exclude shortcuts from being deployed to lakehouses. Use with caution.\",\n )\n\n\ndef validate_git_compare_ref(git_compare_ref: str) -> str:\n \"\"\"\n Validate the git_compare_ref parameter to prevent git flag injection.\n\n Args:\n git_compare_ref: The git ref to compare against.\n\n Raises:\n InputError: If the ref is empty, starts with '-', or contains invalid characters.\n \"\"\"\n validate_data_type(\"string\", \"git_compare_ref\", git_compare_ref)\n\n if not git_compare_ref.strip():\n msg = \"git_compare_ref must not be an empty string.\"\n raise InputError(msg, logger)\n\n if git_compare_ref.startswith(\"-\"):\n msg = \"git_compare_ref must not start with '-' to prevent git flag injection.\"\n raise InputError(msg, logger)\n\n # Allow only characters valid in git refs: alphanumeric, /, ., ~, ^, -, _\n if not re.match(r\"^[a-zA-Z0-9/_.\\-~^@{}]+$\", git_compare_ref):\n msg = f\"git_compare_ref '{git_compare_ref}' contains invalid characters.\"\n raise InputError(msg, logger)\n\n return git_compare_ref\n" }, { "path": "src/fabric_cicd/_items/__init__.py", "content": "# Copyright (c) Microsoft Corporation.\n# Licensed under the MIT License.\n\nfrom fabric_cicd._common._exceptions import PublishError\nfrom fabric_cicd._items._base_publisher import ItemPublisher, ParallelConfig\n\n__all__ = [\n \"ItemPublisher\",\n \"ParallelConfig\",\n \"PublishError\",\n]\n" }, { "path": "src/fabric_cicd/_items/_activator.py", "content": "# Copyright (c) Microsoft Corporation.\n# Licensed under the MIT License.\n\n\"\"\"Functions to process and deploy Reflex item.\"\"\"\n\nfrom fabric_cicd._items._base_publisher import ItemPublisher\nfrom fabric_cicd.constants import ItemType\n\n\nclass ActivatorPublisher(ItemPublisher):\n \"\"\"Publisher for Reflex AKA Activator items.\"\"\"\n\n item_type = ItemType.REFLEX.value\n" }, { "path": "src/fabric_cicd/_items/_apacheairflowjob.py", "content": "# Copyright (c) Microsoft Corporation.\n# Licensed under the MIT License.\n\n\"\"\"Functions to process and deploy Apache Airflow Job item.\"\"\"\n\nfrom fabric_cicd._items._base_publisher import ItemPublisher\nfrom fabric_cicd.constants import ItemType\n\n\nclass ApacheAirflowJobPublisher(ItemPublisher):\n \"\"\"Publisher for Apache Airflow Job items.\"\"\"\n\n item_type = ItemType.APACHE_AIRFLOW_JOB.value\n" }, { "path": "src/fabric_cicd/_items/_base_publisher.py", "content": "# Copyright (c) Microsoft Corporation.\n# Licensed under the MIT License.\n\n\"\"\"Base interface for all item publishers.\"\"\"\n\nimport logging\nfrom abc import ABC, abstractmethod\nfrom concurrent.futures import ThreadPoolExecutor, as_completed\nfrom dataclasses import dataclass\nfrom typing import Callable, Optional\n\nfrom fabric_cicd._common._exceptions import PublishError\nfrom fabric_cicd._common._item import Item\nfrom fabric_cicd.constants import PARALLEL_MAX_WORKERS, ItemType\nfrom fabric_cicd.fabric_workspace import FabricWorkspace\n\nlogger = logging.getLogger(__name__)\n\n\n@dataclass\nclass ParallelConfig:\n \"\"\"Configuration for parallel execution behavior of a publisher.\n\n This dataclass controls how the base ItemPublisher.publish_all() method\n executes publish_one() calls - either in parallel or sequentially.\n\n Attributes:\n enabled: If True, publish_one calls run in parallel using ThreadPoolExecutor.\n If False, items are published sequentially. Default is True.\n max_workers: Maximum number of concurrent threads. None means use ThreadPoolExecutor default.\n ordered_items_func: Optional callable that returns an ordered list of item names.\n When provided, items are published sequentially in this order.\n This takes precedence over `enabled=True`.\n \"\"\"\n\n enabled: bool = True\n max_workers: Optional[int] = PARALLEL_MAX_WORKERS\n ordered_items_func: Optional[Callable[[\"ItemPublisher\"], list[str]]] = None\n\n\nclass Publisher(ABC):\n \"\"\"Base interface for all publishers.\"\"\"\n\n def __init__(self, fabric_workspace_obj: \"FabricWorkspace\") -> None:\n \"\"\"\n Initialize the publisher with a FabricWorkspace object.\n\n Args:\n fabric_workspace_obj: The FabricWorkspace object containing items to be published.\n \"\"\"\n self.fabric_workspace_obj = fabric_workspace_obj\n\n @abstractmethod\n def publish_one(self, name: str, obj: object) -> None:\n \"\"\"\n Publish a single object.\n\n Args:\n name: The name of the object to publish.\n obj: The object to publish.\n \"\"\"\n raise NotImplementedError\n\n @abstractmethod\n def publish_all(self) -> None:\n \"\"\"Publish all objects.\"\"\"\n raise NotImplementedError\n\n\nclass ItemPublisher(Publisher):\n \"\"\"\n Base interface for all item type publishers.\n\n Provides a default parallel publish_all() implementation that:\n - Executes publish_one() calls in parallel using ThreadPoolExecutor\n - Aggregates errors from all failed items into a single PublishError\n - Supports pre/post hooks via pre_publish_all() and post_publish_all()\n - Can be configured via the parallel_config class attribute\n\n Subclasses can customize behavior by:\n - Setting parallel_config to control parallelization\n - Overriding pre_publish_all() for setup before publishing\n - Overriding post_publish_all() for cleanup after publishing\n - Overriding get_items_to_publish() to filter or order items\n - Overriding get_unpublish_order() for dependency-aware unpublishing\n - Overriding post_publish_all_check() for async publish state verification\n\n Publish Lifecycle:\n 1. pre_publish_all()\n 2. get_items_to_publish()\n 3. publish_one() - called for each item\n 4. post_publish_all()\n 5. post_publish_all_check() - if has_async_publish_check\n\n Unpublish Hook:\n - get_unpublish_order() - if has_dependency_tracking\n \"\"\"\n\n # region Class Attributes\n\n item_type: str\n \"\"\"Mandatory property to be set by each publisher subclass\"\"\"\n\n parallel_config: ParallelConfig = ParallelConfig()\n \"\"\"Configuration for parallel execution - subclasses can override with their own ParallelConfig\"\"\"\n\n has_async_publish_check: bool = False\n \"\"\"Set to True if this publisher implements post_publish_all_check() for async state verification\"\"\"\n\n has_dependency_tracking: bool = False\n \"\"\"Set to True if this publisher implements get_unpublish_order() for dependency ordering\"\"\"\n\n # endregion\n\n # region Initialization & Factory\n\n def __init__(self, fabric_workspace_obj: \"FabricWorkspace\") -> None:\n \"\"\"\n Initialize the publisher with a FabricWorkspace object.\n\n Args:\n fabric_workspace_obj: The FabricWorkspace object containing items to be published.\n \"\"\"\n super().__init__(fabric_workspace_obj)\n\n @staticmethod\n def create(item_type: ItemType, fabric_workspace_obj: \"FabricWorkspace\") -> \"ItemPublisher\":\n \"\"\"\n Factory method to create the appropriate publisher for a given item type.\n\n Args:\n item_type: The ItemType enum value for which to create a publisher.\n fabric_workspace_obj: The FabricWorkspace object containing items to be published.\n\n Returns:\n An instance of the appropriate ItemPublisher subclass.\n\n Raises:\n ValueError: If the item type is not supported.\n \"\"\"\n from fabric_cicd._items._activator import ActivatorPublisher\n from fabric_cicd._items._apacheairflowjob import ApacheAirflowJobPublisher\n from fabric_cicd._items._copyjob import CopyJobPublisher\n from fabric_cicd._items._dataagent import DataAgentPublisher\n from fabric_cicd._items._databuildtooljob import DataBuildToolJobPublisher\n from fabric_cicd._items._dataflowgen2 import DataflowPublisher\n from fabric_cicd._items._datapipeline import DataPipelinePublisher\n from fabric_cicd._items._environment import EnvironmentPublisher\n from fabric_cicd._items._eventhouse import EventhousePublisher\n from fabric_cicd._items._eventstream import EventstreamPublisher\n from fabric_cicd._items._graphqlapi import GraphQLApiPublisher\n from fabric_cicd._items._kqldashboard import KQLDashboardPublisher\n from fabric_cicd._items._kqldatabase import KQLDatabasePublisher\n from fabric_cicd._items._kqlqueryset import KQLQuerysetPublisher\n from fabric_cicd._items._lakehouse import LakehousePublisher\n from fabric_cicd._items._mirroreddatabase import MirroredDatabasePublisher\n from fabric_cicd._items._mlexperiment import MLExperimentPublisher\n from fabric_cicd._items._mounteddatafactory import MountedDataFactoryPublisher\n from fabric_cicd._items._notebook import NotebookPublisher\n from fabric_cicd._items._ontology import OntologyPublisher\n from fabric_cicd._items._report import ReportPublisher\n from fabric_cicd._items._semanticmodel import SemanticModelPublisher\n from fabric_cicd._items._sparkjobdefinition import SparkJobDefinitionPublisher\n from fabric_cicd._items._sqldatabase import SQLDatabasePublisher\n from fabric_cicd._items._userdatafunction import UserDataFunctionPublisher\n from fabric_cicd._items._variablelibrary import VariableLibraryPublisher\n from fabric_cicd._items._warehouse import WarehousePublisher\n\n publisher_mapping = {\n ItemType.VARIABLE_LIBRARY: VariableLibraryPublisher,\n ItemType.WAREHOUSE: WarehousePublisher,\n ItemType.MIRRORED_DATABASE: MirroredDatabasePublisher,\n ItemType.LAKEHOUSE: LakehousePublisher,\n ItemType.SQL_DATABASE: SQLDatabasePublisher,\n ItemType.ENVIRONMENT: EnvironmentPublisher,\n ItemType.USER_DATA_FUNCTION: UserDataFunctionPublisher,\n ItemType.EVENTHOUSE: EventhousePublisher,\n ItemType.SPARK_JOB_DEFINITION: SparkJobDefinitionPublisher,\n ItemType.NOTEBOOK: NotebookPublisher,\n ItemType.SEMANTIC_MODEL: SemanticModelPublisher,\n ItemType.REPORT: ReportPublisher,\n ItemType.COPY_JOB: CopyJobPublisher,\n ItemType.DATA_BUILD_TOOL_JOB: DataBuildToolJobPublisher,\n ItemType.KQL_DATABASE: KQLDatabasePublisher,\n ItemType.KQL_QUERYSET: KQLQuerysetPublisher,\n ItemType.REFLEX: ActivatorPublisher,\n ItemType.EVENTSTREAM: EventstreamPublisher,\n ItemType.KQL_DASHBOARD: KQLDashboardPublisher,\n ItemType.DATAFLOW: DataflowPublisher,\n ItemType.DATA_PIPELINE: DataPipelinePublisher,\n ItemType.GRAPHQL_API: GraphQLApiPublisher,\n ItemType.APACHE_AIRFLOW_JOB: ApacheAirflowJobPublisher,\n ItemType.MOUNTED_DATA_FACTORY: MountedDataFactoryPublisher,\n ItemType.DATA_AGENT: DataAgentPublisher,\n ItemType.ML_EXPERIMENT: MLExperimentPublisher,\n ItemType.ONTOLOGY: OntologyPublisher,\n }\n\n publisher_class = publisher_mapping.get(item_type)\n if publisher_class is None:\n msg = f\"No publisher found for item type: {item_type}\"\n raise ValueError(msg)\n\n return publisher_class(fabric_workspace_obj)\n\n @staticmethod\n def get_item_types_to_publish(fabric_workspace_obj: \"FabricWorkspace\") -> list[tuple[int, ItemType]]:\n \"\"\"\n Get the ordered list of item types that should be published.\n\n Returns item types that are both in scope and have items in the repository,\n ordered according to SERIAL_ITEM_PUBLISH_ORDER.\n\n Args:\n fabric_workspace_obj: The FabricWorkspace object containing scope and repository info.\n\n Returns:\n List of (order_num, ItemType) tuples for item types that should be published.\n \"\"\"\n from fabric_cicd import constants\n\n result = []\n for order_num, item_type in constants.SERIAL_ITEM_PUBLISH_ORDER.items():\n if (\n item_type.value in fabric_workspace_obj.item_type_in_scope\n and item_type.value in fabric_workspace_obj.repository_items\n ):\n result.append((order_num, item_type))\n return result\n\n @staticmethod\n def get_item_types_to_unpublish(fabric_workspace_obj: \"FabricWorkspace\") -> list[str]:\n \"\"\"\n Get the ordered list of item types that should be unpublished.\n\n Returns item types in reverse publish order that are in scope, have deployed items,\n and meet feature flag requirements. Logs warnings for skipped item types.\n\n Args:\n fabric_workspace_obj: The FabricWorkspace object containing scope and deployed items info.\n\n Returns:\n List of item type strings in the order they should be unpublished.\n \"\"\"\n from fabric_cicd import constants\n\n unpublish_order = []\n for item_type in reversed(list(constants.SERIAL_ITEM_PUBLISH_ORDER.values())):\n if (\n item_type.value in fabric_workspace_obj.item_type_in_scope\n and item_type.value in fabric_workspace_obj.deployed_items\n ):\n unpublish_flag = constants.UNPUBLISH_FLAG_MAPPING.get(item_type.value)\n # Append item_type if no feature flag is required or the corresponding flag is enabled\n if not unpublish_flag or unpublish_flag in constants.FEATURE_FLAG:\n unpublish_order.append(item_type.value)\n elif unpublish_flag and unpublish_flag not in constants.FEATURE_FLAG:\n # Log warning when unpublish is skipped due to missing feature flag\n logger.warning(\n f\"Skipping unpublish for {item_type.value} items because the '{unpublish_flag}' feature flag is not enabled.\"\n )\n return unpublish_order\n\n @staticmethod\n def get_orphaned_items(\n fabric_workspace_obj: \"FabricWorkspace\",\n item_type: str,\n item_name_exclude_regex: Optional[str] = None,\n items_to_include: Optional[list[str]] = None,\n ) -> list[str]:\n \"\"\"\n Get the list of orphaned items that should be unpublished for a given item type.\n\n Orphaned items are those deployed but not present in the repository,\n filtered by exclusion regex or items_to_include list.\n\n Args:\n fabric_workspace_obj: The FabricWorkspace object containing deployed and repository items.\n item_type: The item type string to check for orphans.\n item_name_exclude_regex: Optional regex pattern to exclude items from unpublishing.\n items_to_include: Optional list of items in \"name.type\" format to include for unpublishing.\n\n Returns:\n List of item names that should be unpublished.\n \"\"\"\n import re\n\n deployed_names = set(fabric_workspace_obj.deployed_items.get(item_type, {}).keys())\n repository_names = set(fabric_workspace_obj.repository_items.get(item_type, {}).keys())\n to_delete_set = deployed_names - repository_names\n\n if items_to_include is not None:\n # Filter to only items in the include list\n return [name for name in to_delete_set if f\"{name}.{item_type}\" in items_to_include]\n if item_name_exclude_regex:\n # Filter out items matching the exclude regex\n regex_pattern = re.compile(item_name_exclude_regex)\n return [name for name in to_delete_set if not regex_pattern.match(name)]\n return list(to_delete_set)\n\n # endregion\n\n # region Public Methods\n\n def publish_all(self) -> None:\n \"\"\"\n Execute the publish operation for this item type.\n\n 1. Calls pre_publish_all() for any setup operations\n 2. Gets items via get_items_to_publish()\n 3. Publishes items (parallel or sequential based on parallel_config)\n 4. Calls post_publish_all() for any finalization\n 5. Raises PublishError if any items failed\n\n The parallel_config class attribute controls execution:\n - If ordered_items_func is set: publishes in that order sequentially\n - If enabled=True: publishes in parallel\n - If enabled=False: publishes sequentially\n\n Raises:\n PublishError: If one or more items failed to publish.\n \"\"\"\n self.pre_publish_all()\n all_items = self.fabric_workspace_obj.repository_items.get(self.item_type, {})\n items = self.get_items_to_publish()\n\n # Mark excluded items as skip_publish=True so post_publish_all() hooks\n # can reliably skip them. Included items are left unchanged (False).\n skipped = [name for name in all_items if name not in items]\n if skipped:\n logger.info(f\"Skipping {self.item_type} item(s) due to items_to_include filter: {skipped}\")\n for name in skipped:\n all_items[name].skip_publish = True\n\n if not items:\n self.post_publish_all()\n return\n\n config = getattr(self.__class__, \"parallel_config\", ParallelConfig())\n\n if config.ordered_items_func is not None:\n order = config.ordered_items_func(self)\n errors = self._publish_items_ordered(items, order)\n elif config.enabled:\n errors = self._publish_items_parallel(items)\n else:\n errors = self._publish_items_sequential(items)\n\n self.post_publish_all()\n\n if errors:\n raise PublishError(errors, logger)\n\n def publish_one(self, item_name: str, _item: \"Item\") -> None:\n \"\"\"\n Publish a single item.\n\n Args:\n item_name: The name of the item to publish.\n _item: The Item object to publish.\n\n Default implementation publishes the item using _publish_item.\n Subclasses can override this method for custom publishing logic.\n \"\"\"\n self.fabric_workspace_obj._publish_item(item_name=item_name, item_type=self.item_type)\n\n def get_items_to_publish(self) -> dict[str, \"Item\"]:\n \"\"\"\n Get the items to publish for this item type.\n\n Returns:\n Dictionary mapping item names to Item objects, pre-filtered by\n items_to_include when set so that only relevant items are iterated.\n\n Subclasses can override to filter or transform the items.\n\n Note:\n The base implementation applies ``FabricWorkspace.items_to_include`` filtering.\n To override this method and preserve this behavior, call ``super().get_items_to_publish()``\n to keep ``items_to_include`` support, then apply any additional selection logic.\n\n Items NOT returned by this method will have ``skip_publish=True`` set on them\n by ``publish_all()`` before ``post_publish_all()`` is called. This ensures that\n ``post_publish_all()`` hooks (e.g. shortcut publishing, connection binding) can\n reliably use ``item.skip_publish`` to determine whether an item was published.\n \"\"\"\n all_items = self.fabric_workspace_obj.repository_items.get(self.item_type, {})\n items_to_include = self.fabric_workspace_obj.items_to_include\n if not items_to_include:\n return all_items\n normalized_include_set = {i.lower() for i in items_to_include}\n return {\n name: item\n for name, item in all_items.items()\n if f\"{name}.{self.item_type}\".lower() in normalized_include_set\n }\n\n def get_unpublish_order(self, items_to_unpublish: list[str]) -> list[str]:\n \"\"\"\n Get the ordered list of item names based on dependencies for unpublishing.\n\n Args:\n items_to_unpublish: List of item names to be unpublished.\n\n Returns:\n List of item names in the order they should be unpublished (reverse dependency order).\n\n Default implementation returns items in their original order.\n Subclasses with dependency tracking should override for proper ordering.\n \"\"\"\n return items_to_unpublish\n\n def pre_publish_all(self) -> None:\n \"\"\"\n Hook called before publishing any items.\n\n Subclasses can override to perform setup, validation, or refresh operations.\n Default implementation does nothing.\n \"\"\"\n pass\n\n def post_publish_all(self) -> None:\n \"\"\"\n Hook called after all items have been published successfully.\n\n Subclasses can override to perform cleanup, binding, or finalization.\n Default implementation does nothing.\n \"\"\"\n pass\n\n def post_publish_all_check(self) -> None:\n \"\"\"\n Hook called after publish_all completes to verify async publish state.\n\n Subclasses can override this to check the state of asynchronous publish\n operations (e.g., Environment items that have async publish workflows).\n Default implementation does nothing.\n\n This method is called separately from publish_all() and should be invoked\n by the orchestration layer after all items of this type have been published.\n \"\"\"\n pass\n\n # endregion\n\n # region Publishing\n\n def _publish_items_parallel(self, items: dict[str, \"Item\"]) -> list[tuple[str, Exception]]:\n \"\"\"\n Publish items in parallel using ThreadPoolExecutor.\n\n Args:\n items: Dictionary mapping item names to Item objects.\n\n Returns:\n List of (item_name, exception) tuples for failed items.\n \"\"\"\n errors: list[tuple[str, Exception]] = []\n config = getattr(self.__class__, \"parallel_config\", ParallelConfig())\n\n with ThreadPoolExecutor(max_workers=config.max_workers) as executor:\n futures = {\n executor.submit(self.publish_one, item_name, item): (item_name, item)\n for item_name, item in items.items()\n }\n\n for future in as_completed(futures):\n item_name, _ = futures[future]\n try:\n future.result()\n except Exception as e:\n logger.error(f\"Failed to publish {self.item_type} '{item_name}': {e}\")\n errors.append((item_name, e))\n\n return errors\n\n def _publish_items_sequential(self, items: dict[str, \"Item\"]) -> list[tuple[str, Exception]]:\n \"\"\"\n Publish items sequentially.\n\n Args:\n items: Dictionary mapping item names to Item objects.\n\n Returns:\n List of (item_name, exception) tuples for failed items.\n \"\"\"\n errors: list[tuple[str, Exception]] = []\n\n for item_name, item in items.items():\n try:\n self.publish_one(item_name, item)\n except Exception as e:\n logger.error(f\"Failed to publish {self.item_type} '{item_name}': {e}\")\n errors.append((item_name, e))\n\n return errors\n\n def _publish_items_ordered(self, items: dict[str, \"Item\"], order: list[str]) -> list[tuple[str, Exception]]:\n \"\"\"\n Publish items in a specific order sequentially.\n\n Args:\n items: Dictionary mapping item names to Item objects.\n order: List of item names in the order they should be published.\n\n Returns:\n List of (item_name, exception) tuples for failed items.\n \"\"\"\n errors: list[tuple[str, Exception]] = []\n\n for item_name in order:\n if item_name in items:\n item = items[item_name]\n try:\n self.publish_one(item_name, item)\n except Exception as e:\n logger.error(f\"Failed to publish {self.item_type} '{item_name}': {e}\")\n errors.append((item_name, e))\n\n return errors\n\n # endregion\n" }, { "path": "src/fabric_cicd/_items/_copyjob.py", "content": "# Copyright (c) Microsoft Corporation.\n# Licensed under the MIT License.\n\n\"\"\"Functions to process and deploy Copy Job item.\"\"\"\n\nfrom fabric_cicd._items._base_publisher import ItemPublisher\nfrom fabric_cicd.constants import ItemType\n\n\nclass CopyJobPublisher(ItemPublisher):\n \"\"\"Publisher for Copy Job items.\"\"\"\n\n item_type = ItemType.COPY_JOB.value\n" }, { "path": "src/fabric_cicd/_items/_dataagent.py", "content": "# Copyright (c) Microsoft Corporation.\n# Licensed under the MIT License.\n\n\"\"\"Functions to process and deploy Data Agent item.\"\"\"\n\nimport logging\n\nfrom fabric_cicd._common._item import Item\nfrom fabric_cicd._items._base_publisher import ItemPublisher\nfrom fabric_cicd.constants import EXCLUDE_PATH_REGEX_MAPPING, ItemType\n\nlogger = logging.getLogger(__name__)\n\n\nclass DataAgentPublisher(ItemPublisher):\n \"\"\"Publisher for Data Agent items.\"\"\"\n\n item_type = ItemType.DATA_AGENT.value\n\n def publish_one(self, item_name: str, _item: Item) -> None:\n \"\"\"Publish a single Data Agent item.\"\"\"\n self.fabric_workspace_obj._publish_item(\n item_name=item_name, item_type=self.item_type, exclude_path=EXCLUDE_PATH_REGEX_MAPPING.get(self.item_type)\n )\n" }, { "path": "src/fabric_cicd/_items/_databuildtooljob.py", "content": "# Copyright (c) Microsoft Corporation.\n# Licensed under the MIT License.\n\n\"\"\"Functions to process and deploy Data Build Tool Job item.\"\"\"\n\nfrom fabric_cicd._items._base_publisher import ItemPublisher\nfrom fabric_cicd.constants import ItemType\n\n\nclass DataBuildToolJobPublisher(ItemPublisher):\n \"\"\"Publisher for Data Build Tool Job items.\"\"\"\n\n item_type = ItemType.DATA_BUILD_TOOL_JOB.value\n" }, { "path": "src/fabric_cicd/_items/_dataflowgen2.py", "content": "# Copyright (c) Microsoft Corporation.\n# Licensed under the MIT License.\n\n\"\"\"Functions to process and deploy Dataflow Gen2 item.\"\"\"\n\nimport logging\nimport re\n\nfrom fabric_cicd import FabricWorkspace, constants\nfrom fabric_cicd._common._exceptions import ParsingError\nfrom fabric_cicd._common._file import File\nfrom fabric_cicd._common._item import Item\nfrom fabric_cicd._items._base_publisher import ItemPublisher, ParallelConfig\nfrom fabric_cicd._parameter._utils import (\n check_replacement,\n extract_find_value,\n extract_parameter_filters,\n extract_replace_value,\n)\nfrom fabric_cicd.constants import ItemType\n\nlogger = logging.getLogger(__name__)\n\n\ndef set_dataflow_publish_order(workspace_obj: FabricWorkspace, item_type: str) -> list[str]:\n \"\"\"\n Sets the publish order where the source dataflow, if present always proceeds the referencing dataflow.\n This only applies to dataflows that reference another dataflow in the repository and the referenced\n dataflowId is parameterized correctly (using the items attribute variable).\n\n Algorithm for determining dataflow publish order:\n 1. Find all dataflows that reference other dataflows in the repository\n 2. Build a dependency graph where each dataflow depends on its source dataflow\n 3. Use a modified depth-first search with cycle detection to create a topological sort\n ensuring that source dataflows are published before the dataflows that reference them\n 4. Add any remaining standalone dataflows (without dependencies) to the end of the publish order\n\n Args:\n workspace_obj: The FabricWorkspace object.\n item_type: Type of item (e.g., 'Dataflow').\n \"\"\"\n publish_order = []\n visited = set()\n temp_visited = set()\n\n # If the find_replace parameter doesn't exist, skip sorting dataflows by dependencies\n param_dict = workspace_obj.environment_parameter.get(\"find_replace\", [])\n if not param_dict:\n logger.warning(\n \"find_replace parameter not found - dataflows will not be checked for dependencies. \"\n \"Dataflows will be published in arbitrary order, which may cause errors in the deployed items\"\n )\n return list(workspace_obj.repository_items.get(item_type, {}).keys())\n\n # Otherwise, collect dataflow items with a source dataflow\n for item in workspace_obj.repository_items.get(item_type, {}).values():\n for file in item.item_files:\n # Check if a dataflow is referenced in the file\n if file.type == \"text\" and str(file.file_path).endswith(\".pq\") and contains_source_dataflow(file.contents):\n # Try to get info associated with the dataflow reference\n dataflow_name, dataflow_workspace_id, dataflow_id = get_source_dataflow_name(\n workspace_obj, file.contents, item.name, file.file_path\n )\n # If the dataflow is found in the repository, add it to the dependencies dictionary\n if dataflow_name:\n workspace_obj.dataflow_dependencies[item.name] = {\n \"source_name\": dataflow_name,\n \"source_workspace_id\": dataflow_workspace_id,\n \"source_id\": dataflow_id,\n }\n else:\n logger.warning(f\"The '{item.name}' dataflow will be published without considering its dependency\")\n\n def add_dataflow_with_dependency(item: str) -> bool:\n \"\"\"\n Recursively adds an item and its dependency to the publish order.\n Returns True if successful, False if a cycle is detected.\n \"\"\"\n # Dataflow was already processed, no need to process again\n if item in visited:\n return True\n\n # If the item is already in the temporary visited set, it indicates a cycle\n if item in temp_visited:\n msg = f\"Circular dependency found for item {item}. Cannot determine a valid publish order\"\n raise ParsingError(msg, logger)\n\n # Add the item to the temporary visited set\n temp_visited.add(item)\n\n # First add the dependency if it exists\n if workspace_obj.dataflow_dependencies.get(item):\n dependency = workspace_obj.dataflow_dependencies[item][\"source_name\"]\n # Propagate cycle detection\n if not add_dataflow_with_dependency(dependency):\n return False\n\n # Then add the current item\n publish_order.append(item)\n visited.add(item)\n # Remove from temporary set\n temp_visited.remove(item)\n\n return True\n\n # Process each item in the dataflow dependencies\n for item in list(workspace_obj.dataflow_dependencies.keys()):\n add_dataflow_with_dependency(item)\n\n # Add any remaining dataflows from the repository that aren't in the publish order (standalone dataflows)\n for item_name in workspace_obj.repository_items.get(item_type, {}):\n if item_name not in visited:\n publish_order.append(item_name)\n\n return publish_order\n\n\ndef contains_source_dataflow(file_content: str) -> bool:\n \"\"\"A helper function to check if the file content contains a source dataflow reference.\"\"\"\n try:\n # Check if file contains the PowerPlatform.Dataflows pattern (group 1 of the regex)\n match = re.search(constants.DATAFLOW_SOURCE_REGEX, file_content, re.DOTALL)\n return match is not None and bool(match.group(1))\n except (re.error, TypeError, IndexError) as e:\n logger.debug(f\"Error checking for source dataflow: {e}\")\n return False\n\n\ndef get_source_dataflow_ids(file_content: str, item_name: str) -> tuple[str, str]:\n \"\"\"A helper function to get the dataflow ID and workspace ID of a referenced dataflow.\"\"\"\n try:\n match = re.search(constants.DATAFLOW_SOURCE_REGEX, file_content, re.DOTALL)\n if not match:\n msg = f\"No dataflow source pattern found in the '{item_name}' file content\"\n raise ParsingError(msg, logger)\n\n # Extract the source dataflow IDs from the regex match\n dataflow_workspace_id = match.group(2)\n dataflow_id = match.group(3)\n\n except Exception as e:\n msg = f\"Error extracting dataflow information from file content: {e}\"\n raise ParsingError(msg, logger) from e\n\n # Validate the extracted IDs are valid GUIDs\n if not dataflow_workspace_id or not re.match(constants.VALID_GUID_REGEX, dataflow_workspace_id):\n msg = f\"Invalid workspace ID: {dataflow_workspace_id} in '{item_name}' file content\"\n raise ParsingError(msg, logger)\n if not dataflow_id or not re.match(constants.VALID_GUID_REGEX, dataflow_id):\n msg = f\"Invalid dataflow ID: {dataflow_id} in '{item_name}' file content\"\n raise ParsingError(msg, logger)\n\n return dataflow_workspace_id, dataflow_id\n\n\ndef get_source_dataflow_name(\n workspace_obj: FabricWorkspace, file_content: str, item_name: str, file_path: str\n) -> tuple[str, str, str]:\n \"\"\"\n A helper function to extract the dataflow name, dataflow workspaceId and dataflowId\n associated with the source dataflow referenced in the file content. The source dataflow\n name is obtained by using the matching parameter dictionary input, if present.\n \"\"\"\n # Get the IDs of the source dataflow\n dataflow_workspace_id, dataflow_id = get_source_dataflow_ids(file_content, item_name)\n\n # Look for a parameter that contains the dataflow ID\n for param in workspace_obj.environment_parameter.get(\"find_replace\", []):\n # Extract values from the parameter\n input_type, input_name, input_path = extract_parameter_filters(workspace_obj, param)\n filter_match = check_replacement(\n input_type, input_name, input_path, ItemType.DATAFLOW.value, item_name, file_path\n )\n find_info = extract_find_value(param, file_content, filter_match)\n\n # Skip if this parameter doesn't match the dataflow ID\n if find_info[\"pattern\"] != dataflow_id:\n logger.debug(\n f\"Find value: {find_info['pattern']} does not match the dataflow ID: {dataflow_id}, skipping this parameter\"\n )\n continue\n\n # Extract the replace value for the current environment\n replace_value = param.get(\"replace_value\", {}).get(workspace_obj.environment, \"\")\n\n # Pass in get_dataflow_name=True to get the source dataflow name, if it exists\n source_dataflow_name = extract_replace_value(workspace_obj, replace_value, get_dataflow_name=True)\n\n if source_dataflow_name:\n # Return the source dataflow name along with the IDs\n return source_dataflow_name, dataflow_workspace_id, dataflow_id\n\n return \"\", \"\", \"\"\n\n\ndef func_process_file(workspace_obj: FabricWorkspace, item_obj: Item, file_obj: File) -> str:\n \"\"\"\n Custom file processing for dataflow items.\n\n Args:\n workspace_obj: The FabricWorkspace object.\n item_obj: The item object.\n file_obj: The file object.\n \"\"\"\n # Replace the dataflow ID with the logical ID of the source dataflow in the file content\n return replace_source_dataflow_ids(workspace_obj, item_obj, file_obj)\n\n\ndef replace_source_dataflow_ids(workspace_obj: FabricWorkspace, item_obj: Item, file_obj: File) -> str:\n \"\"\"\n Replaces both the dataflow ID and workspace ID of the source dataflow\n with logical values for cross-environment compatibility.\n\n Args:\n workspace_obj: The FabricWorkspace object.\n item_obj: The item object.\n file_obj: The file object.\n \"\"\"\n if str(file_obj.file_path).endswith(\".pq\"):\n # Get source dataflow info from the dependency dictionary\n source_dataflow_info = workspace_obj.dataflow_dependencies.get(item_obj.name, {})\n\n # If the info was tracked, proceed to replace the IDs\n if source_dataflow_info:\n source_dataflow_name = source_dataflow_info[\"source_name\"]\n source_dataflow_workspace_id = source_dataflow_info[\"source_workspace_id\"]\n source_dataflow_id = source_dataflow_info[\"source_id\"]\n\n # Get the logical ID of the source dataflow from repository items\n logical_id = (\n workspace_obj.repository_items.get(ItemType.DATAFLOW.value, {}).get(source_dataflow_name, {}).logical_id\n )\n\n # Replace the dataflow ID with its logical ID and the workspace ID with the default workspace ID\n if logical_id:\n file_obj.contents = file_obj.contents.replace(source_dataflow_id, logical_id)\n file_obj.contents = file_obj.contents.replace(source_dataflow_workspace_id, constants.DEFAULT_GUID)\n logger.debug(\n f\"Replaced dataflow ID '{source_dataflow_id}' with logical ID '{logical_id}' and workspace ID \"\n f\"'{source_dataflow_workspace_id}' with default workspace ID '{constants.DEFAULT_GUID}' \"\n f\"in '{item_obj.name}' file\"\n )\n\n return file_obj.contents\n\n\ndef _get_dataflow_publish_order(publisher: \"DataflowPublisher\") -> list[str]:\n \"\"\"Get the ordered list of dataflow names based on dependencies.\"\"\"\n return set_dataflow_publish_order(publisher.fabric_workspace_obj, publisher.item_type)\n\n\nclass DataflowPublisher(ItemPublisher):\n \"\"\"Publisher for Dataflow items.\"\"\"\n\n item_type = ItemType.DATAFLOW.value\n\n parallel_config = ParallelConfig(enabled=False, ordered_items_func=_get_dataflow_publish_order)\n \"\"\"Dataflows must be published in dependency order (sequential)\"\"\"\n\n def publish_one(self, item_name: str, _item: Item) -> None:\n \"\"\"Publish a single Dataflow item.\"\"\"\n self.fabric_workspace_obj._publish_item(\n item_name=item_name, item_type=self.item_type, func_process_file=func_process_file\n )\n" }, { "path": "src/fabric_cicd/_items/_datapipeline.py", "content": "# Copyright (c) Microsoft Corporation.\n# Licensed under the MIT License.\n\n\"\"\"Functions to process and deploy DataPipeline item.\"\"\"\n\nimport logging\nimport re\n\nimport dpath\n\nfrom fabric_cicd import FabricWorkspace, constants\nfrom fabric_cicd._common._item import Item\nfrom fabric_cicd._items._base_publisher import ItemPublisher, ParallelConfig\nfrom fabric_cicd._items._manage_dependencies import set_publish_order, set_unpublish_order\nfrom fabric_cicd.constants import ItemType\n\nlogger = logging.getLogger(__name__)\n\n\ndef find_referenced_datapipelines(fabric_workspace_obj: FabricWorkspace, file_content: dict, lookup_type: str) -> list:\n \"\"\"\n Scan through pipeline file json dictionary and find pipeline references (including nested pipelines).\n\n Args:\n fabric_workspace_obj: The FabricWorkspace object.\n file_content: Dict representation of the pipeline-content file.\n lookup_type: Finding references in deployed file or repo file (Deployed or Repository).\n \"\"\"\n item_type = ItemType.DATA_PIPELINE.value\n reference_list = []\n guid_pattern = re.compile(constants.VALID_GUID_REGEX)\n\n # Use the dpath library to search through the dictionary for all values that match the GUID pattern\n for _, value in dpath.search(file_content, \"**\", yielded=True):\n if isinstance(value, str):\n match = guid_pattern.search(value)\n if match:\n # If a valid GUID is found, convert it to name. If name is not None, it's a pipeline and will be added to the reference list\n referenced_id = match.group(0)\n referenced_name = fabric_workspace_obj._convert_id_to_name(\n item_type=item_type, generic_id=referenced_id, lookup_type=lookup_type\n )\n # Add pipeline to the reference list if it's not already present\n if referenced_name and referenced_name not in reference_list:\n reference_list.append(referenced_name)\n\n return reference_list\n\n\ndef _get_datapipeline_publish_order(publisher: \"DataPipelinePublisher\") -> list[str]:\n \"\"\"Get the ordered list of data pipeline names based on dependencies.\"\"\"\n return set_publish_order(publisher.fabric_workspace_obj, publisher.item_type, find_referenced_datapipelines)\n\n\nclass DataPipelinePublisher(ItemPublisher):\n \"\"\"Publisher for Data Pipeline items.\"\"\"\n\n item_type = ItemType.DATA_PIPELINE.value\n has_dependency_tracking = True\n\n parallel_config = ParallelConfig(enabled=False, ordered_items_func=_get_datapipeline_publish_order)\n \"\"\"Pipelines must be published in dependency order (sequential)\"\"\"\n\n def get_unpublish_order(self, items_to_unpublish: list[str]) -> list[str]:\n \"\"\"\n Get the ordered list of item names based on dependencies for unpublishing.\n\n Args:\n items_to_unpublish: List of item names to be unpublished.\n\n Returns:\n List of item names in the order they should be unpublished (reverse dependency order).\n \"\"\"\n return set_unpublish_order(\n self.fabric_workspace_obj, self.item_type, items_to_unpublish, find_referenced_datapipelines\n )\n\n def publish_one(self, item_name: str, _item: Item) -> None:\n \"\"\"Publish a single Data Pipeline item.\"\"\"\n self.fabric_workspace_obj._publish_item(item_name=item_name, item_type=self.item_type)\n\n def pre_publish_all(self) -> None:\n \"\"\"Refresh deployed items before publishing to resolve references.\"\"\"\n self.fabric_workspace_obj._refresh_deployed_items()\n" }, { "path": "src/fabric_cicd/_items/_environment.py", "content": "# Copyright (c) Microsoft Corporation.\n# Licensed under the MIT License.\n\n\"\"\"Functions to process and deploy Environment item.\"\"\"\n\nimport logging\nimport re\n\nimport dpath\nimport yaml\n\nfrom fabric_cicd import FabricWorkspace, constants\nfrom fabric_cicd._common._exceptions import InputError\nfrom fabric_cicd._common._fabric_endpoint import handle_retry\nfrom fabric_cicd._common._file import File\nfrom fabric_cicd._common._item import Item\nfrom fabric_cicd._items._base_publisher import ItemPublisher\nfrom fabric_cicd.constants import ItemType\n\nlogger = logging.getLogger(__name__)\n\n\ndef _process_environment_file(\n fabric_workspace_obj: FabricWorkspace,\n item: Item,\n file_obj: File,\n) -> str:\n \"\"\"\n Process an Environment item file before it is included in the item definition payload.\n\n For ``Setting/Sparkcompute.yml`` this performs ``instance_pool_id`` replacement\n using the ``spark_pool`` parameter configuration so that the correct pool\n reference is embedded directly in the YAML sent to the Fabric Items API.\n\n All other files are returned unchanged.\n\n Args:\n fabric_workspace_obj: The FabricWorkspace object.\n item: The Item object representing the Environment.\n file_obj: The File object being processed.\n\n Returns:\n The (possibly modified) file contents as a string.\n \"\"\"\n if not (file_obj.file_path.name == \"Sparkcompute.yml\" and file_obj.file_path.parent.name == \"Setting\"):\n return file_obj.contents\n\n contents = file_obj.contents\n\n if \"instance_pool_id\" not in contents:\n return contents\n\n yaml_body = yaml.safe_load(contents)\n if not isinstance(yaml_body, dict):\n return contents\n\n if \"instance_pool_id\" in yaml_body:\n yaml_body = _replace_instance_pool_id(fabric_workspace_obj, yaml_body, item.name)\n\n return yaml.dump(yaml_body, default_flow_style=False, sort_keys=False)\n\n\ndef _replace_instance_pool_id(fabric_workspace_obj: FabricWorkspace, yaml_body: dict, item_name: str) -> dict:\n \"\"\"\n Replace ``instance_pool_id`` in parsed Sparkcompute YAML with a resolved pool GUID.\n\n This function reads ``spark_pool`` parameter mappings from\n ``fabric_workspace_obj.environment_parameter`` and finds the entry whose\n ``instance_pool_id`` matches the current YAML value. If an ``item_name`` is\n provided in the mapping, it must match the current Environment item name;\n otherwise, the mapping applies globally.\n\n The mapped target pool ``name`` and ``type`` are then resolved against the\n workspace custom pool list returned by the Fabric API, and the resolved pool\n ``id`` is written back to ``yaml_body[\"instance_pool_id\"]``.\n\n Args:\n fabric_workspace_obj: Workspace context containing environment, parameters,\n and endpoint configuration.\n yaml_body: Parsed contents of ``Setting/Sparkcompute.yml``.\n item_name: Environment item name used for optional per-item mapping filters.\n\n Returns:\n The YAML dictionary, updated if a matching mapping is found; otherwise unchanged.\n \"\"\"\n from fabric_cicd._parameter._utils import process_environment_key\n\n pool_id = yaml_body[\"instance_pool_id\"]\n if \"spark_pool\" in fabric_workspace_obj.environment_parameter:\n pools = fabric_workspace_obj._get_workspace_pools()\n parameter_dict = fabric_workspace_obj.environment_parameter[\"spark_pool\"]\n for key in parameter_dict:\n instance_pool_id = key[\"instance_pool_id\"]\n replace_value = process_environment_key(fabric_workspace_obj.environment, key[\"replace_value\"])\n input_name = key.get(\"item_name\")\n if instance_pool_id == pool_id and (input_name == item_name or not input_name):\n pool_config = replace_value[fabric_workspace_obj.environment]\n resolved_id = _resolve_pool_id(\n pools,\n pool_name=pool_config[\"name\"],\n pool_type=pool_config[\"type\"],\n )\n yaml_body[\"instance_pool_id\"] = resolved_id\n break\n\n return yaml_body\n\n\ndef _resolve_pool_id(pools: list[dict], pool_name: str, pool_type: str) -> str:\n \"\"\"\n Resolve a workspace custom Spark pool ID by pool ``name`` and ``type``.\n\n Args:\n pools: Pool objects from ``GET /spark/pools`` (expected to include\n ``name``, ``type``, and ``id`` fields).\n pool_name: Target pool display name.\n pool_type: Target pool type (for example, ``\"Capacity\"`` or ``\"Workspace\"``).\n\n Returns:\n The matching pool GUID.\n\n Raises:\n InputError: If no pool exists with the specified ``name`` and ``type``.\n \"\"\"\n for pool in pools:\n if pool[\"name\"] == pool_name and pool[\"type\"] == pool_type:\n return pool[\"id\"]\n\n msg = (\n f\"Could not resolve custom Spark pool: name='{pool_name}', type='{pool_type}'. \"\n f\"No matching pool found in the target workspace.\"\n )\n raise InputError(msg, logger)\n\n\ndef _check_environment_publish_state(fabric_workspace_obj: FabricWorkspace, initial_check: bool = False) -> None:\n \"\"\"\n Checks the publish state of environments after deployment.\n\n Args:\n fabric_workspace_obj: The FabricWorkspace object.\n initial_check: Flag to ignore publish failures on initial check.\n \"\"\"\n ongoing_publish = True\n iteration = 1\n\n environments = fabric_workspace_obj.repository_items.get(ItemType.ENVIRONMENT.value, {})\n filtered_environments = [\n k\n for k in environments\n if (\n # Check exclude regex\n (\n not fabric_workspace_obj.publish_item_name_exclude_regex\n or not re.search(fabric_workspace_obj.publish_item_name_exclude_regex, k)\n )\n # Check items_to_include list\n and (\n not fabric_workspace_obj.items_to_include or k + \".Environment\" in fabric_workspace_obj.items_to_include\n )\n )\n ]\n\n logger.info(f\"Checking Environment Publish State for {filtered_environments}\")\n\n while ongoing_publish:\n ongoing_publish = False\n completed = []\n running = []\n failed = []\n\n response_state = fabric_workspace_obj.endpoint.invoke(\n method=\"GET\", url=f\"{fabric_workspace_obj.base_api_url}/environments/\"\n )\n\n for item in response_state[\"body\"][\"value\"]:\n item_name = item[\"displayName\"]\n item_state = dpath.get(item, \"properties/publishDetails/state\", default=\"\").lower()\n if item_name in filtered_environments:\n if item_state == \"running\":\n running.append(item_name)\n ongoing_publish = True\n elif item_state == \"success\":\n completed.append(item_name)\n elif item_state in [\"failed\", \"cancelled\"]:\n failed.append(item_name)\n if not initial_check:\n msg = f\"Publish {item_state} for Environment '{item_name}'\"\n raise Exception(msg)\n logger.debug(\n f\"Environment publish states - Running: {running}, Succeeded: {completed}, Failed/Cancelled: {failed}\"\n )\n if ongoing_publish:\n handle_retry(\n attempt=iteration,\n base_delay=5,\n response_retry_after=120,\n prepend_message=f\"{constants.INDENT}Operation in progress.\",\n )\n iteration += 1\n\n if not initial_check:\n logger.info(f\"{constants.INDENT}Published: {completed}\")\n\n\ndef _submit_environment_publish(fabric_workspace_obj: FabricWorkspace, item_name: str) -> None:\n \"\"\"\n Submit a publish request for an Environment item.\n\n Triggers the asynchronous publish of the environment's staged settings and\n libraries. The publish state is monitored separately by the async publish\n check hooks.\n\n Args:\n fabric_workspace_obj: The FabricWorkspace object.\n item_name: Name of the environment item to publish.\n \"\"\"\n item_type = ItemType.ENVIRONMENT.value\n item_guid = fabric_workspace_obj.repository_items[item_type][item_name].guid\n\n # Publish updated settings - compute settings and libraries (long-running operation)\n # https://learn.microsoft.com/en-us/rest/api/fabric/environment/items/publish-environment\n fabric_workspace_obj.endpoint.invoke(\n method=\"POST\",\n url=f\"{fabric_workspace_obj.base_api_url}/environments/{item_guid}/staging/publish?beta=False\",\n poll_long_running=False,\n )\n logger.info(f\"{constants.INDENT}Publish Submitted for Environment '{item_name}'\")\n\n\nclass EnvironmentPublisher(ItemPublisher):\n \"\"\"Publisher for Environment items.\"\"\"\n\n item_type = ItemType.ENVIRONMENT.value\n has_async_publish_check = True\n\n def publish_one(self, item_name: str, item: Item) -> None:\n \"\"\"Publish a single Environment item.\"\"\"\n self.fabric_workspace_obj._publish_item(\n item_name=item_name,\n item_type=self.item_type,\n func_process_file=_process_environment_file,\n skip_publish_logging=True,\n )\n if item.skip_publish:\n return\n _submit_environment_publish(self.fabric_workspace_obj, item_name)\n\n def pre_publish_all(self) -> None:\n \"\"\"Check environment publish state before publishing.\"\"\"\n _check_environment_publish_state(self.fabric_workspace_obj, True)\n\n def post_publish_all_check(self) -> None:\n \"\"\"Check environment publish state after all environments have been published.\"\"\"\n _check_environment_publish_state(self.fabric_workspace_obj, False)\n" }, { "path": "src/fabric_cicd/_items/_eventhouse.py", "content": "# Copyright (c) Microsoft Corporation.\n# Licensed under the MIT License.\n\n\"\"\"Functions to process and deploy Eventhouse item.\"\"\"\n\nimport logging\n\nfrom fabric_cicd._common._item import Item\nfrom fabric_cicd._items._base_publisher import ItemPublisher\nfrom fabric_cicd.constants import EXCLUDE_PATH_REGEX_MAPPING, ItemType\n\nlogger = logging.getLogger(__name__)\n\n\nclass EventhousePublisher(ItemPublisher):\n \"\"\"Publisher for Eventhouse items.\"\"\"\n\n item_type = ItemType.EVENTHOUSE.value\n\n def publish_one(self, item_name: str, _item: Item) -> None:\n \"\"\"Publish a single Eventhouse item.\"\"\"\n self.fabric_workspace_obj._publish_item(\n item_name=item_name, item_type=self.item_type, exclude_path=EXCLUDE_PATH_REGEX_MAPPING.get(self.item_type)\n )\n" }, { "path": "src/fabric_cicd/_items/_eventstream.py", "content": "# Copyright (c) Microsoft Corporation.\n# Licensed under the MIT License.\n\n\"\"\"Functions to process and deploy Eventstream item.\"\"\"\n\nfrom fabric_cicd._items._base_publisher import ItemPublisher\nfrom fabric_cicd.constants import ItemType\n\n\nclass EventstreamPublisher(ItemPublisher):\n \"\"\"Publisher for Eventstream items.\"\"\"\n\n item_type = ItemType.EVENTSTREAM.value\n" }, { "path": "src/fabric_cicd/_items/_graphqlapi.py", "content": "# Copyright (c) Microsoft Corporation.\n# Licensed under the MIT License.\n\n\"\"\"Functions to process and deploy API for GraphQL item.\"\"\"\n\nfrom fabric_cicd._items._base_publisher import ItemPublisher\nfrom fabric_cicd.constants import ItemType\n\n\nclass GraphQLApiPublisher(ItemPublisher):\n \"\"\"Publisher for GraphQL API items.\"\"\"\n\n item_type = ItemType.GRAPHQL_API.value\n" }, { "path": "src/fabric_cicd/_items/_kqldashboard.py", "content": "# Copyright (c) Microsoft Corporation.\n# Licensed under the MIT License.\n\n\"\"\"Functions to process and deploy Real-Time Dashboard item.\"\"\"\n\nimport json\nimport logging\n\nfrom fabric_cicd import FabricWorkspace\nfrom fabric_cicd._common._exceptions import ParsingError\nfrom fabric_cicd._common._file import File\nfrom fabric_cicd._common._item import Item\nfrom fabric_cicd._items._base_publisher import ItemPublisher\nfrom fabric_cicd.constants import ItemType\n\nlogger = logging.getLogger(__name__)\n\n\ndef func_process_file(workspace_obj: FabricWorkspace, item_obj: Item, file_obj: File) -> str:\n \"\"\"\n Custom file processing for KQL Dashboard items.\n\n Args:\n workspace_obj: The FabricWorkspace object.\n item_obj: The item object.\n file_obj: The file object.\n \"\"\"\n # For KQL Dashboard, we do not need to process the file content\n return (\n replace_cluster_uri(workspace_obj, file_obj)\n if item_obj.type == ItemType.KQL_DASHBOARD.value\n else file_obj.contents\n )\n\n\ndef replace_cluster_uri(fabric_workspace_obj: FabricWorkspace, file_obj: File) -> str:\n \"\"\"\n Replaces an empty cluster URI value in a Real-Time Dashboard item with the cluster URI associated\n with its KQL Database source in the raw file content.\n\n Args:\n fabric_workspace_obj: The FabricWorkspace object.\n file_obj: The file object.\n \"\"\"\n # Create a dictionary from the raw file\n json_content_dict = json.loads(file_obj.contents)\n\n data_sources = json_content_dict.get(\"dataSources\")\n\n # Get the KQL Database items from the deployed items\n database_items = fabric_workspace_obj.deployed_items.get(ItemType.KQL_DATABASE.value, {})\n\n for data_source in data_sources:\n if not data_source:\n msg = \"No data sources found in the KQL Dashboard item.\"\n raise ParsingError(msg, logger)\n if data_source.get(\"clusterUri\") == \"\":\n database_item_name = data_source.get(\"name\")\n database_item = database_items.get(database_item_name)\n\n if not database_item:\n msg = f\"Cannot find the KQL Database source with name '{database_item_name}' as it is not yet deployed.\"\n raise ParsingError(msg, logger)\n\n database_item_guid = database_item.guid\n # Get the cluster URI of the KQL database\n kqldatabase_data = fabric_workspace_obj.endpoint.invoke(\n method=\"GET\",\n url=f\"{fabric_workspace_obj.base_api_url}/kqlDatabases/{database_item_guid}\",\n )\n kqldatabase_cluster_uri = kqldatabase_data.get(\"body\", {}).get(\"properties\", {}).get(\"queryServiceUri\")\n # Replace the cluster URI value\n if not kqldatabase_cluster_uri:\n msg = f\"Cluster URI for KQL Database '{database_item_name}' is not found.\"\n raise ParsingError(msg, logger)\n\n data_source[\"clusterUri\"] = kqldatabase_cluster_uri\n\n return json.dumps(json_content_dict, indent=2)\n\n\nclass KQLDashboardPublisher(ItemPublisher):\n \"\"\"Publisher for KQL Dashboard items.\"\"\"\n\n item_type = ItemType.KQL_DASHBOARD.value\n\n def publish_one(self, item_name: str, _item: Item) -> None:\n \"\"\"Publish a single KQL Dashboard item.\"\"\"\n self.fabric_workspace_obj._publish_item(\n item_name=item_name, item_type=self.item_type, func_process_file=func_process_file\n )\n\n def pre_publish_all(self) -> None:\n \"\"\"Refresh deployed items to get KQL Database cluster URIs.\"\"\"\n self.fabric_workspace_obj._refresh_deployed_items()\n" }, { "path": "src/fabric_cicd/_items/_kqldatabase.py", "content": "# Copyright (c) Microsoft Corporation.\n# Licensed under the MIT License.\n\n\"\"\"Functions to process and deploy KQL Database item.\"\"\"\n\nfrom fabric_cicd._items._base_publisher import ItemPublisher\nfrom fabric_cicd.constants import ItemType\n\n\nclass KQLDatabasePublisher(ItemPublisher):\n \"\"\"Publisher for KQL Database items.\"\"\"\n\n item_type = ItemType.KQL_DATABASE.value\n" }, { "path": "src/fabric_cicd/_items/_kqlqueryset.py", "content": "# Copyright (c) Microsoft Corporation.\n# Licensed under the MIT License.\n\n\"\"\"Functions to process and deploy KQL Queryset item.\"\"\"\n\nimport json\nimport logging\n\nfrom fabric_cicd import FabricWorkspace\nfrom fabric_cicd._common._exceptions import ParsingError\nfrom fabric_cicd._common._file import File\nfrom fabric_cicd._common._item import Item\nfrom fabric_cicd._items._base_publisher import ItemPublisher\nfrom fabric_cicd.constants import ItemType\n\nlogger = logging.getLogger(__name__)\n\n\ndef func_process_file(workspace_obj: FabricWorkspace, item_obj: Item, file_obj: File) -> str:\n \"\"\"\n Custom file processing for kql queryset items.\n\n Args:\n workspace_obj: The FabricWorkspace object.\n item_obj: The item object.\n file_obj: The file object.\n \"\"\"\n return (\n replace_cluster_uri(workspace_obj, file_obj)\n if item_obj.type == ItemType.KQL_QUERYSET.value\n else file_obj.contents\n )\n\n\ndef replace_cluster_uri(fabric_workspace_obj: FabricWorkspace, file_obj: File) -> str:\n \"\"\"\n Replaces an empty cluster URI value in a KQL Queryset item with the cluster URI associated\n with its KQL Database source in the raw file content.\n\n Args:\n fabric_workspace_obj: The FabricWorkspace object.\n file_obj: The file object.\n \"\"\"\n # Create a dictionary from the raw file\n json_content_dict = json.loads(file_obj.contents)\n\n queryset = json_content_dict.get(\"queryset\")\n data_sources = queryset.get(\"dataSources\") if queryset else None\n if not data_sources:\n logger.debug(\"No data sources found in KQL Queryset.\")\n return file_obj.contents\n\n # Get the KQL Database items from the deployed items\n database_items = fabric_workspace_obj.deployed_items.get(ItemType.KQL_DATABASE.value, {})\n\n # If the cluster URI is empty, replace it with the cluster URI of the KQL database\n for data_source in data_sources:\n if data_source.get(\"clusterUri\") == \"\":\n database_item_name = data_source.get(\"databaseItemName\")\n logger.debug(f\"Found empty cluster URI for database '{database_item_name}'\")\n\n database_item = database_items.get(database_item_name)\n if not database_item:\n msg = f\"Cannot find the KQL Database source with name '{database_item_name}' as it is not yet deployed.\"\n raise ParsingError(msg, logger)\n\n database_item_guid = database_item.guid\n # Get the cluster URI of the KQL database\n kqldatabase_data = fabric_workspace_obj.endpoint.invoke(\n method=\"GET\",\n url=f\"{fabric_workspace_obj.base_api_url}/kqlDatabases/{database_item_guid}\",\n )\n try:\n kqldatabase_cluster_uri = kqldatabase_data[\"body\"][\"properties\"][\"queryServiceUri\"]\n except (KeyError, TypeError):\n kqldatabase_cluster_uri = None\n\n if not kqldatabase_cluster_uri:\n msg = f\"Cannot find the cluster URI for KQL Database '{database_item_name}'.\"\n raise ParsingError(msg, logger)\n # Replace the cluster URI value\n data_source[\"clusterUri\"] = kqldatabase_cluster_uri\n logger.debug(\n f\"Updated the cluster URI for data source '{database_item_name}' with '{kqldatabase_cluster_uri}'\"\n )\n\n logger.debug(\"Successfully updated all empty cluster URIs.\")\n return json.dumps(json_content_dict, indent=2)\n\n\nclass KQLQuerysetPublisher(ItemPublisher):\n \"\"\"Publisher for KQL Queryset items.\"\"\"\n\n item_type = ItemType.KQL_QUERYSET.value\n\n def publish_one(self, item_name: str, _item: Item) -> None:\n \"\"\"Publish a single KQL Queryset item.\"\"\"\n self.fabric_workspace_obj._publish_item(\n item_name=item_name, item_type=self.item_type, func_process_file=func_process_file\n )\n\n def pre_publish_all(self) -> None:\n \"\"\"Refresh deployed items to get KQL Database cluster URIs.\"\"\"\n self.fabric_workspace_obj._refresh_deployed_items()\n" }, { "path": "src/fabric_cicd/_items/_lakehouse.py", "content": "# Copyright (c) Microsoft Corporation.\n# Licensed under the MIT License.\n\n\"\"\"Functions to process and deploy Lakehouse item.\"\"\"\n\nimport json\nimport logging\n\nimport dpath\n\nfrom fabric_cicd import FabricWorkspace, constants\nfrom fabric_cicd._common._exceptions import FailedPublishedItemStatusError\nfrom fabric_cicd._common._fabric_endpoint import handle_retry\nfrom fabric_cicd._common._item import Item\nfrom fabric_cicd._items._base_publisher import ItemPublisher, Publisher\nfrom fabric_cicd.constants import FeatureFlag, ItemType\n\nlogger = logging.getLogger(__name__)\n\n\ndef check_sqlendpoint_provision_status(fabric_workspace_obj: FabricWorkspace, item_obj: Item) -> None:\n \"\"\"\n Check the SQL endpoint status of the published lakehouses\n\n Args:\n fabric_workspace_obj: The FabricWorkspace object containing the items to be published\n item_obj: The item object to check the SQL endpoint status for\n\n \"\"\"\n iteration = 1\n\n while True:\n sql_endpoint_status = None\n\n response_state = fabric_workspace_obj.endpoint.invoke(\n method=\"GET\", url=f\"{fabric_workspace_obj.base_api_url}/lakehouses/{item_obj.guid}\"\n )\n\n sql_endpoint_status = dpath.get(\n response_state, \"body/properties/sqlEndpointProperties/provisioningStatus\", default=None\n )\n\n if sql_endpoint_status == \"Success\":\n logger.info(f\"{constants.INDENT}SQL Endpoint provisioned successfully\")\n break\n\n if sql_endpoint_status == \"Failed\":\n msg = f\"Cannot resolve SQL endpoint for lakehouse {item_obj.name}\"\n raise FailedPublishedItemStatusError(msg, logger)\n\n handle_retry(\n attempt=iteration,\n base_delay=5,\n response_retry_after=30,\n prepend_message=f\"{constants.INDENT}SQL Endpoint provisioning in progress\",\n )\n iteration += 1\n\n\ndef list_deployed_shortcuts(fabric_workspace_obj: FabricWorkspace, item_obj: Item) -> list:\n \"\"\"\n Lists all deployed shortcut paths\n\n Args:\n fabric_workspace_obj: The FabricWorkspace object containing the items to be published\n item_obj: The item object to list the shortcuts for\n \"\"\"\n request_url = f\"{fabric_workspace_obj.base_api_url}/items/{item_obj.guid}/shortcuts\"\n deployed_shortcut_paths = []\n\n while request_url:\n # https://learn.microsoft.com/en-us/rest/api/fabric/core/onelake-shortcuts/list-shortcuts\n response = fabric_workspace_obj.endpoint.invoke(method=\"GET\", url=request_url)\n\n # Handle cases where the response body is empty\n shortcuts = response[\"body\"].get(\"value\", [])\n deployed_shortcut_paths.extend(f\"{shortcut['path']}/{shortcut['name']}\" for shortcut in shortcuts)\n\n request_url = response[\"header\"].get(\"continuationUri\", None)\n\n return deployed_shortcut_paths\n\n\ndef replace_default_lakehouse_id(shortcut: dict, item_obj: Item) -> dict:\n \"\"\"\n Replaces the default lakehouse ID (all zeros) with the actual lakehouse ID\n in the shortcut definition when present.\n\n Args:\n shortcut: The shortcut definition dictionary\n item_obj: The item object used to get the default lakehouse ID\n \"\"\"\n if dpath.get(shortcut, \"target/oneLake/itemId\", default=None) == constants.DEFAULT_GUID:\n shortcut[\"target\"][\"oneLake\"][\"itemId\"] = item_obj.guid\n\n return shortcut\n\n\nclass LakehousePublisher(ItemPublisher):\n \"\"\"Publisher for Lakehouse items.\"\"\"\n\n item_type = ItemType.LAKEHOUSE.value\n\n def publish_one(self, item_name: str, item: Item) -> None:\n \"\"\"Publish a single Lakehouse item.\"\"\"\n creation_payload = next(\n (\n {\"enableSchemas\": True}\n for file in item.item_files\n if file.name == \"lakehouse.metadata.json\" and \"defaultSchema\" in file.contents\n ),\n None,\n )\n\n self.fabric_workspace_obj._publish_item(\n item_name=item_name,\n item_type=self.item_type,\n creation_payload=creation_payload,\n skip_publish_logging=True,\n )\n\n # Check if the item is published to avoid any post publish actions\n if item.skip_publish:\n return\n\n check_sqlendpoint_provision_status(self.fabric_workspace_obj, item)\n\n logger.info(f\"{constants.INDENT}Published Lakehouse '{item_name}'\")\n\n def post_publish_all(self) -> None:\n \"\"\"Publish shortcuts after all lakehouses are published to protect interrelationships.\"\"\"\n if FeatureFlag.ENABLE_SHORTCUT_PUBLISH.value in constants.FEATURE_FLAG:\n for item_obj in self.fabric_workspace_obj.repository_items.get(self.item_type, {}).values():\n # Check if the item is published to avoid any post publish actions\n if not item_obj.skip_publish and item_obj.guid:\n shortcut_publisher = ShortcutPublisher(self.fabric_workspace_obj, item_obj)\n shortcut_publisher.publish_all()\n\n\nclass ShortcutPublisher(Publisher):\n \"\"\"Publisher for Lakehouse shortcuts.\"\"\"\n\n def __init__(self, fabric_workspace_obj: FabricWorkspace, item_obj: Item) -> None:\n \"\"\"\n Initialize the shortcut publisher.\n\n Args:\n fabric_workspace_obj: The FabricWorkspace object containing the items to be published.\n item_obj: The lakehouse item object to publish shortcuts for.\n \"\"\"\n super().__init__(fabric_workspace_obj)\n self.item_obj = item_obj\n\n def _unpublish_shortcuts(self, shortcut_paths: list) -> None:\n \"\"\"\n Unpublish shortcuts from the lakehouse.\n\n Args:\n shortcut_paths: The list of shortcut paths to unpublish.\n \"\"\"\n for deployed_shortcut_path in shortcut_paths:\n # https://learn.microsoft.com/en-us/rest/api/fabric/core/onelake-shortcuts/delete-shortcut\n self.fabric_workspace_obj.endpoint.invoke(\n method=\"DELETE\",\n url=f\"{self.fabric_workspace_obj.base_api_url}/items/{self.item_obj.guid}/shortcuts/{deployed_shortcut_path}\",\n )\n\n def publish_one(self, _shortcut_name: str, shortcut: dict) -> None:\n \"\"\"\n Publish a single shortcut.\n\n Args:\n _shortcut_name: The name/path of the shortcut to publish.\n shortcut: The shortcut definition to publish.\n \"\"\"\n shortcut = replace_default_lakehouse_id(shortcut, self.item_obj)\n # https://learn.microsoft.com/en-us/rest/api/fabric/core/onelake-shortcuts/create-shortcut\n try:\n self.fabric_workspace_obj.endpoint.invoke(\n method=\"POST\",\n url=f\"{self.fabric_workspace_obj.base_api_url}/items/{self.item_obj.guid}/shortcuts?shortcutConflictPolicy=CreateOrOverwrite\",\n body=shortcut,\n )\n logger.info(f\"{constants.INDENT}Published Shortcut '{shortcut['name']}'\")\n except Exception as e:\n if FeatureFlag.CONTINUE_ON_SHORTCUT_FAILURE.value in constants.FEATURE_FLAG:\n logger.warning(\n f\"Failed to publish Shortcut '{shortcut['name']}'. This usually happens when the lakehouse containing the source for this shortcut is published as a shell and has no data yet.\"\n )\n logger.info(\"The publish process will continue with the other items.\")\n return\n msg = f\"Failed to publish '{shortcut['name']}' for lakehouse {self.item_obj.name}\"\n raise FailedPublishedItemStatusError(msg, logger) from e\n\n def publish_all(self) -> None:\n \"\"\"\n Publish all shortcuts for the lakehouse item.\n\n Loads shortcuts from metadata, filters based on exclude regex,\n unpublishes orphaned shortcuts, and publishes all remaining shortcuts.\n \"\"\"\n from fabric_cicd._common._check_utils import check_regex\n\n deployed_shortcuts = list_deployed_shortcuts(self.fabric_workspace_obj, self.item_obj)\n\n shortcut_file_obj = next(\n (file for file in self.item_obj.item_files if file.name == \"shortcuts.metadata.json\"), None\n )\n\n if shortcut_file_obj:\n shortcut_file_obj.contents = self.fabric_workspace_obj._replace_parameters(shortcut_file_obj, self.item_obj)\n shortcut_file_obj.contents = self.fabric_workspace_obj._replace_logical_ids(shortcut_file_obj.contents)\n shortcut_file_obj.contents = self.fabric_workspace_obj._replace_workspace_ids(shortcut_file_obj.contents)\n\n shortcuts = json.loads(shortcut_file_obj.contents) or []\n else:\n logger.debug(\"No shortcuts.metadata.json found\")\n shortcuts = []\n\n # Filter shortcuts based on exclude regex if provided\n if self.fabric_workspace_obj.shortcut_exclude_regex:\n regex_pattern = check_regex(self.fabric_workspace_obj.shortcut_exclude_regex)\n original_count = len(shortcuts)\n excluded_shortcuts = [s[\"name\"] for s in shortcuts if \"name\" in s and regex_pattern.match(s[\"name\"])]\n shortcuts = [s for s in shortcuts if \"name\" in s and not regex_pattern.match(s[\"name\"])]\n excluded_count = original_count - len(shortcuts)\n if excluded_count > 0:\n logger.info(\n f\"{constants.INDENT}Excluded {excluded_count} shortcut(s) from {self.item_obj.name} deployment based on regex pattern\"\n )\n logger.info(f\"{constants.INDENT}Excluded shortcuts: {excluded_shortcuts}\")\n\n shortcuts_to_publish = {f\"{shortcut['path']}/{shortcut['name']}\": shortcut for shortcut in shortcuts}\n\n if shortcuts_to_publish:\n logger.info(f\"Publishing Lakehouse '{self.item_obj.name}' Shortcuts\")\n shortcut_paths_to_unpublish = [path for path in deployed_shortcuts if path not in shortcuts_to_publish]\n self._unpublish_shortcuts(shortcut_paths_to_unpublish)\n # Deploy and overwrite shortcuts\n for shortcut_path, shortcut in shortcuts_to_publish.items():\n self.publish_one(shortcut_path, shortcut)\n" }, { "path": "src/fabric_cicd/_items/_manage_dependencies.py", "content": "# Copyright (c) Microsoft Corporation.\n# Licensed under the MIT License.\n\n\"\"\"Functions to process items with dependencies.\"\"\"\n\nimport base64\nimport json\nimport logging\nfrom collections import defaultdict, deque\nfrom pathlib import Path\nfrom typing import Callable\n\nfrom fabric_cicd import FabricWorkspace, constants\nfrom fabric_cicd._common._exceptions import ParsingError\n\nlogger = logging.getLogger(__name__)\n\n\ndef set_publish_order(\n fabric_workspace_obj: FabricWorkspace, item_type: str, find_referenced_items_func: Callable\n) -> list:\n \"\"\"\n Creates a publish order list for items of the same type, considering their dependencies.\n\n Args:\n fabric_workspace_obj: The FabricWorkspace object.\n item_type: Type of item to order (e.g., 'DataPipeline').\n find_referenced_items_func: Function to find referenced items in content.\n \"\"\"\n # Get all items of the given type from the repository\n items = fabric_workspace_obj.repository_items.get(item_type, {})\n\n # Construct the unsorted_dict with an item and its associated file content\n unsorted_dict = {}\n # Set the file name based on the item type (e.g., 'pipeline-content.json' for DataPipeline)\n file_name = constants.ITEM_TYPE_TO_FILE[item_type]\n\n for item_name, item_details in items.items():\n with Path(item_details.path, file_name).open(encoding=\"utf-8\") as f:\n raw_file = f.read()\n\n # If the file is a JSON, load as dict; otherwise, keep as the raw file\n item_content = json.loads(raw_file) if file_name.endswith(\".json\") else raw_file\n unsorted_dict[item_name] = item_content\n\n # Return a list of items sorted by their dependencies\n return sort_items(fabric_workspace_obj, unsorted_dict, \"Repository\", find_referenced_items_func)\n\n\ndef set_unpublish_order(\n fabric_workspace_obj: FabricWorkspace,\n item_type: str,\n unpublish_list: list,\n find_referenced_items_func: Callable,\n) -> list:\n \"\"\"\n Creates an unpublish order list for items of the same type, considering their dependencies.\n\n Args:\n fabric_workspace_obj: The FabricWorkspace object.\n item_type: Type of item to order (e.g., 'DataPipeline').\n unpublish_list: List of items to unpublish.\n find_referenced_items_func: Function to find referenced items in content.\n \"\"\"\n unsorted_item_dict = {}\n file_name = constants.ITEM_TYPE_TO_FILE[item_type]\n\n for item_name in unpublish_list:\n # Get deployed item definition\n # https://learn.microsoft.com/en-us/rest/api/fabric/core/items/get-item-definition\n item_guid = fabric_workspace_obj.deployed_items[item_type][item_name].guid\n response = fabric_workspace_obj.endpoint.invoke(\n method=\"POST\", url=f\"{fabric_workspace_obj.base_api_url}/items/{item_guid}/getDefinition\"\n )\n for part in response[\"body\"][\"definition\"][\"parts\"]:\n if part[\"path\"] == file_name:\n # Decode Base64 string to dictionary\n decoded_bytes = base64.b64decode(part[\"payload\"])\n decoded_string = decoded_bytes.decode(\"utf-8\")\n unsorted_item_dict[item_name] = (\n json.loads(decoded_string) if file_name.endswith(\".json\") else decoded_string\n )\n break\n\n # Determine order to delete w/o dependencies\n return sort_items(fabric_workspace_obj, unsorted_item_dict, \"Deployed\", find_referenced_items_func)\n\n\ndef sort_items(\n fabric_workspace_obj: FabricWorkspace, unsorted_dict: dict, lookup_type: str, find_referenced_items_func: Callable\n) -> list:\n \"\"\"\n Performs topological sort on items of a given item type based on their dependencies.\n\n Args:\n fabric_workspace_obj: The FabricWorkspace object.\n unsorted_dict: Dictionary mapping items to their file content.\n lookup_type: Finding references in deployed file or repo file (Deployed or Repository).\n find_referenced_items_func: Function to find referenced items in content.\n \"\"\"\n # Step 1: Create a graph to manage dependencies\n graph = defaultdict(list)\n in_degree = defaultdict(int)\n unpublish_items = []\n\n # Step 2: Build the graph and count the in-degrees\n for item_name, item_content in unsorted_dict.items():\n logger.debug(f\"Processing item: '{item_name}'\")\n # In an unpublish case, keep track of items to get unpublished\n if lookup_type == \"Deployed\":\n unpublish_items.append(item_name)\n\n referenced_items = find_referenced_items_func(fabric_workspace_obj, item_content, lookup_type)\n\n for referenced_name in referenced_items:\n graph[referenced_name].append(item_name)\n in_degree[item_name] += 1\n # Ensure every item has an entry in the in-degree map\n if item_name not in in_degree:\n in_degree[item_name] = 0\n\n logger.debug(f\"Graph: {graph}\")\n logger.debug(f\"In-degree map: {in_degree}\")\n\n # In an unpublish case, adjust in_degree to include entire dependency chain\n if lookup_type == \"Deployed\":\n for item_name in graph:\n if item_name not in in_degree:\n in_degree[item_name] = 0\n for neighbor in graph[item_name]:\n if neighbor not in in_degree:\n in_degree[neighbor] += 1\n\n # Step 3: Perform a topological sort to determine the correct publish order\n zero_in_degree_queue = deque([item_name for item_name in in_degree if in_degree[item_name] == 0])\n sorted_items = []\n logger.debug(f\"Zero_in_degree_queue: {zero_in_degree_queue}\")\n\n while zero_in_degree_queue:\n item_name = zero_in_degree_queue.popleft()\n sorted_items.append(item_name)\n\n for neighbor in graph[item_name]:\n in_degree[neighbor] -= 1\n if in_degree[neighbor] == 0:\n zero_in_degree_queue.append(neighbor)\n\n if len(sorted_items) != len(in_degree):\n msg = \"There is a cycle in the graph. Cannot determine a valid publish order.\"\n raise ParsingError(msg, logger)\n\n # Remove items not present in unpublish list and invert order for deployed sort\n if lookup_type == \"Deployed\":\n sorted_items = [item_name for item_name in sorted_items if item_name in unpublish_items]\n sorted_items = sorted_items[::-1]\n\n logger.debug(f\"Sorted items in {lookup_type}: {sorted_items}\")\n return sorted_items\n" }, { "path": "src/fabric_cicd/_items/_mirroreddatabase.py", "content": "# Copyright (c) Microsoft Corporation.\n# Licensed under the MIT License.\n\n\"\"\"Functions to process and deploy Mirrored Database item.\"\"\"\n\nfrom fabric_cicd._items._base_publisher import ItemPublisher\nfrom fabric_cicd.constants import ItemType\n\n\nclass MirroredDatabasePublisher(ItemPublisher):\n \"\"\"Publisher for Mirrored Database items.\"\"\"\n\n item_type = ItemType.MIRRORED_DATABASE.value\n" }, { "path": "src/fabric_cicd/_items/_mlexperiment.py", "content": "# Copyright (c) Microsoft Corporation.\n# Licensed under the MIT License.\n\n\"\"\"Functions to process and deploy ML Experiment item.\"\"\"\n\nfrom fabric_cicd._items._base_publisher import ItemPublisher\nfrom fabric_cicd.constants import ItemType\n\n\nclass MLExperimentPublisher(ItemPublisher):\n \"\"\"Publisher for ML Experiment items.\"\"\"\n\n item_type = ItemType.ML_EXPERIMENT.value\n" }, { "path": "src/fabric_cicd/_items/_mounteddatafactory.py", "content": "# Copyright (c) Microsoft Corporation.\n# Licensed under the MIT License.\n\n\"\"\"Functions to process and deploy Mounted Data Factory item.\"\"\"\n\nfrom fabric_cicd._items._base_publisher import ItemPublisher\nfrom fabric_cicd.constants import ItemType\n\n\nclass MountedDataFactoryPublisher(ItemPublisher):\n \"\"\"Publisher for Mounted Data Factory items.\"\"\"\n\n item_type = ItemType.MOUNTED_DATA_FACTORY.value\n" }, { "path": "src/fabric_cicd/_items/_notebook.py", "content": "# Copyright (c) Microsoft Corporation.\n# Licensed under the MIT License.\n\n\"\"\"Functions to process and deploy Notebook item.\"\"\"\n\nfrom fabric_cicd._common._item import Item\nfrom fabric_cicd._items._base_publisher import ItemPublisher\nfrom fabric_cicd.constants import API_FORMAT_MAPPING, ItemType\n\n\nclass NotebookPublisher(ItemPublisher):\n \"\"\"Publisher for Notebook items.\"\"\"\n\n item_type = ItemType.NOTEBOOK.value\n\n def publish_one(self, item_name: str, item: Item) -> None:\n \"\"\"Publish a Notebook item.\"\"\"\n is_ipynb = any(file.file_path.suffix == \".ipynb\" for file in item.item_files)\n\n # Sort files to ensure consistent payload order for Fabric API notebook processing\n # Fabric API expects content file (.py) to be processed before settings file (.json) when both are present\n def _sort_key(f: Item) -> tuple[int, str]:\n # .ipynb included for completeness; in practice, .json settings only exist with .py notebooks (git integrated format)\n priority = {\".platform\": 0, \".py\": 1, \".ipynb\": 1, \".json\": 3}\n # Account for other file types that may be added later to the notebook item and assign to priority 2\n return (priority.get(f.file_path.name, priority.get(f.file_path.suffix, 2)), f.file_path.name)\n\n item.item_files.sort(key=_sort_key)\n\n kwargs = {}\n if is_ipynb:\n api_format = API_FORMAT_MAPPING.get(self.item_type)\n if api_format:\n kwargs[\"api_format\"] = api_format\n\n self.fabric_workspace_obj._publish_item(\n item_name=item_name,\n item_type=self.item_type,\n **kwargs,\n )\n" }, { "path": "src/fabric_cicd/_items/_ontology.py", "content": "# Copyright (c) Microsoft Corporation.\n# Licensed under the MIT License.\n\n\"\"\"Functions to process and deploy Ontology item.\"\"\"\n\nfrom fabric_cicd._items._base_publisher import ItemPublisher\nfrom fabric_cicd.constants import ItemType\n\n\nclass OntologyPublisher(ItemPublisher):\n \"\"\"Publisher for Ontology items.\"\"\"\n\n item_type = ItemType.ONTOLOGY.value\n" }, { "path": "src/fabric_cicd/_items/_report.py", "content": "# Copyright (c) Microsoft Corporation.\n# Licensed under the MIT License.\n\n\"\"\"Functions to process and deploy Report item.\"\"\"\n\nimport json\nimport logging\n\nfrom fabric_cicd import FabricWorkspace\nfrom fabric_cicd._common._exceptions import ItemDependencyError\nfrom fabric_cicd._common._file import File\nfrom fabric_cicd._common._item import Item\nfrom fabric_cicd._items._base_publisher import ItemPublisher\nfrom fabric_cicd.constants import EXCLUDE_PATH_REGEX_MAPPING, ItemType\n\nlogger = logging.getLogger(__name__)\n\n\ndef func_process_file(workspace_obj: FabricWorkspace, item_obj: Item, file_obj: File) -> str:\n \"\"\"\n Custom file processing for report items.\n\n Args:\n workspace_obj: The FabricWorkspace object.\n item_obj: The item object.\n file_obj: The file object.\n \"\"\"\n if file_obj.name == \"definition.pbir\":\n definition_body = json.loads(file_obj.contents)\n if (\n \"datasetReference\" in definition_body\n and \"byPath\" in definition_body[\"datasetReference\"]\n and definition_body[\"datasetReference\"][\"byPath\"] is not None\n ):\n model_rel_path = definition_body[\"datasetReference\"][\"byPath\"][\"path\"]\n model_path = str((item_obj.path / model_rel_path).resolve())\n model_id = workspace_obj._convert_path_to_id(ItemType.SEMANTIC_MODEL.value, model_path)\n\n if not model_id:\n msg = \"Semantic model not found in the repository. Cannot deploy a report with a relative path without deploying the model.\"\n raise ItemDependencyError(msg, logger)\n\n definition_body[\"$schema\"] = (\n \"https://developer.microsoft.com/json-schemas/fabric/item/report/definitionProperties/1.0.0/schema.json\"\n )\n\n definition_body[\"datasetReference\"] = {\n \"byConnection\": {\n \"connectionString\": None,\n \"pbiServiceModelId\": None,\n \"pbiModelVirtualServerName\": \"sobe_wowvirtualserver\",\n \"pbiModelDatabaseName\": f\"{model_id}\",\n \"name\": \"EntityDataSource\",\n \"connectionType\": \"pbiServiceXmlaStyleLive\",\n }\n }\n\n return json.dumps(definition_body, indent=4)\n return file_obj.contents\n\n\nclass ReportPublisher(ItemPublisher):\n \"\"\"Publisher for Report items.\"\"\"\n\n item_type = ItemType.REPORT.value\n\n def publish_one(self, item_name: str, _item: Item) -> None:\n \"\"\"Publish a single Report item.\"\"\"\n self.fabric_workspace_obj._publish_item(\n item_name=item_name,\n item_type=self.item_type,\n exclude_path=EXCLUDE_PATH_REGEX_MAPPING.get(self.item_type),\n func_process_file=func_process_file,\n )\n" }, { "path": "src/fabric_cicd/_items/_semanticmodel.py", "content": "# Copyright (c) Microsoft Corporation.\n# Licensed under the MIT License.\n\n\"\"\"Functions to process and deploy Semantic Model item.\"\"\"\n\nimport logging\n\nfrom fabric_cicd import FabricWorkspace, constants\nfrom fabric_cicd._common._item import Item\nfrom fabric_cicd._items._base_publisher import ItemPublisher\nfrom fabric_cicd._parameter._utils import process_environment_key\nfrom fabric_cicd.constants import EXCLUDE_PATH_REGEX_MAPPING, ItemType\n\nlogger = logging.getLogger(__name__)\n\n\ndef build_binding_mapping_legacy(fabric_workspace_obj: FabricWorkspace, semantic_model_binding: list) -> dict:\n \"\"\"\n Build the connection mapping from legacy list-based semantic_model_binding parameter.\n\n Args:\n fabric_workspace_obj: The FabricWorkspace object\n semantic_model_binding: The semantic_model_binding parameter as a list\n\n Returns:\n Dictionary mapping semantic model names to connection IDs\n \"\"\"\n logger.warning(\n \"The legacy 'semantic_model_binding' list format is deprecated and will be removed in a future release. \"\n \"Please migrate to the new dictionary format with 'default' and 'models' keys. \"\n \"See: https://microsoft.github.io/fabric-cicd/how_to/parameterization/\"\n )\n item_type = \"SemanticModel\"\n binding_mapping = {}\n repository_models = set(fabric_workspace_obj.repository_items.get(item_type, {}).keys())\n\n for entry in semantic_model_binding:\n connection_id = entry.get(\"connection_id\")\n model_names = entry.get(\"semantic_model_name\", [])\n\n if not connection_id:\n logger.debug(\"No connection_id found in semantic_model_binding entry, skipping\")\n continue\n\n # Legacy format only supports string connection_id\n if isinstance(connection_id, dict):\n logger.warning(\n \"Environment-specific connection_id dictionaries are not supported in the legacy format. \"\n \"Please migrate to the new dictionary format to use environment-specific values.\"\n )\n continue\n\n if isinstance(model_names, str):\n model_names = [model_names]\n\n for name in model_names:\n if name not in repository_models:\n logger.warning(f\"Semantic model '{name}' specified in parameter.yml not found in repository\")\n continue\n binding_mapping[name] = connection_id\n\n return binding_mapping\n\n\ndef build_binding_mapping(\n fabric_workspace_obj: FabricWorkspace, semantic_model_binding: dict, environment: str\n) -> dict:\n \"\"\"\n Build the connection mapping from semantic_model_binding parameter. The new format requires\n environment-specific connection_id values (use '_ALL_' for all environments).\n\n Supports:\n - default.connection_id: Applied to all models in the repository that are not explicitly listed\n - models: List of explicit model-to-connection mappings\n\n Args:\n fabric_workspace_obj: The FabricWorkspace object\n semantic_model_binding: The semantic_model_binding parameter dictionary\n environment: The target environment name (_ALL_ key can be used)\n\n Returns:\n Dictionary mapping semantic model names to connection IDs\n \"\"\"\n item_type = \"SemanticModel\"\n binding_mapping = {}\n repository_models = set(fabric_workspace_obj.repository_items.get(item_type, {}).keys())\n\n # Get default connection_id for this environment\n default_connection_id = None\n default_config = semantic_model_binding.get(\"default\", {})\n if default_config:\n connection_id_config = default_config.get(\"connection_id\", {})\n connection_id_config = process_environment_key(environment, connection_id_config)\n default_connection_id = connection_id_config.get(environment)\n if not default_connection_id:\n logger.debug(f\"Environment '{environment}' not found in default.connection_id\")\n\n # Process explicit model bindings\n explicit_models = set()\n models_config = semantic_model_binding.get(\"models\", [])\n\n for model in models_config:\n model_names = model.get(\"semantic_model_name\", [])\n connection_id_config = model.get(\"connection_id\", {})\n\n if isinstance(model_names, str):\n model_names = [model_names]\n\n connection_id_config = process_environment_key(environment, connection_id_config)\n connection_id = connection_id_config.get(environment)\n if not connection_id:\n logger.debug(f\"Environment '{environment}' not found in connection_id for semantic model(s): {model_names}\")\n continue\n\n # Track models with explicit bindings to exclude from default connection assignment\n explicit_models.update(model_names)\n\n for name in model_names:\n if name not in repository_models:\n logger.warning(f\"Semantic model '{name}' specified in parameter.yml not found in repository\")\n continue\n binding_mapping[name] = connection_id\n\n # Apply default connection to non-explicit models\n if default_connection_id:\n default_models = repository_models - explicit_models\n for model_name in default_models:\n binding_mapping[model_name] = default_connection_id\n logger.debug(f\"Applying default connection to semantic model '{model_name}'\")\n\n return binding_mapping\n\n\ndef get_connections(fabric_workspace_obj: FabricWorkspace) -> dict:\n \"\"\"\n Get all connections from the workspace.\n\n Args:\n fabric_workspace_obj: The FabricWorkspace object\n\n Returns:\n Dictionary with connection ID as key and connection details as value\n \"\"\"\n # https://learn.microsoft.com/en-us/rest/api/fabric/core/connections/list-connections\n connections_url = f\"{constants.FABRIC_API_ROOT_URL}/v1/connections\"\n\n try:\n response = fabric_workspace_obj.endpoint.invoke(method=\"GET\", url=connections_url)\n connections_list = response.get(\"body\", {}).get(\"value\", [])\n\n connections_dict = {}\n for connection in connections_list:\n connection_id = connection.get(\"id\")\n if connection_id:\n connections_dict[connection_id] = {\n \"id\": connection_id,\n \"connectivityType\": connection.get(\"connectivityType\"),\n \"connectionDetails\": connection.get(\"connectionDetails\", {}),\n }\n\n return connections_dict\n except Exception as e:\n logger.error(f\"Failed to retrieve connections: {e}\")\n return {}\n\n\ndef bind_semanticmodel_to_connection(\n fabric_workspace_obj: FabricWorkspace, connections: dict, connection_details: dict\n) -> None:\n \"\"\"\n Binds semantic models to their specified connections.\n\n Args:\n fabric_workspace_obj: The FabricWorkspace object containing the items to be published.\n connections: Dictionary of connection objects with connection ID as key.\n connection_details: Dictionary mapping semantic model names to connection IDs from parameter.yml.\n \"\"\"\n item_type = ItemType.SEMANTIC_MODEL.value\n\n for model_name, connection_id in connection_details.items():\n # Check if the connection ID exists in the connections dict\n if connection_id not in connections:\n logger.warning(f\"Connection ID '{connection_id}' not found for semantic model '{model_name}'\")\n continue\n\n # Get the semantic model object (validated during binding mapping creation)\n item_obj = fabric_workspace_obj.repository_items[item_type][model_name]\n\n # Skip models excluded by items_to_include (skip_publish=True) or with no deployed\n # GUID — binding would produce an empty-ID URL (HTTP 400) and fail with a server error.\n if item_obj.skip_publish or not item_obj.guid:\n logger.debug(\n f\"Skipping connection binding for semantic model '{model_name}' \"\n f\"(skip_publish={item_obj.skip_publish}, guid='{item_obj.guid}')\"\n )\n continue\n\n model_id = item_obj.guid\n\n logger.info(f\"Binding semantic model '{model_name}' (ID: {model_id}) to connection '{connection_id}'\")\n\n try:\n # Get the connection details for this semantic model from Fabric API\n # https://learn.microsoft.com/en-us/rest/api/fabric/core/items/list-item-connections\n item_connections_url = f\"{constants.FABRIC_API_ROOT_URL}/v1/workspaces/{fabric_workspace_obj.workspace_id}/items/{model_id}/connections\"\n connections_response = fabric_workspace_obj.endpoint.invoke(method=\"GET\", url=item_connections_url)\n connections_data = connections_response.get(\"body\", {}).get(\"value\", [])\n\n if not connections_data:\n logger.debug(f\"No existing connections found for semantic model '{model_name}', skipping binding\")\n continue\n\n # Use the first connection as the template\n connection_binding = connections_data[0]\n\n # Update the connection binding with the target connection ID from parameter.yml\n connection_binding[\"id\"] = connection_id\n connection_binding[\"connectivityType\"] = connections[connection_id][\"connectivityType\"]\n connection_binding[\"connectionDetails\"] = connections[connection_id][\"connectionDetails\"]\n\n # Build the request body\n request_body = build_request_body({\"connectionBinding\": connection_binding})\n\n # Make the bind connection API call\n # https://learn.microsoft.com/en-us/rest/api/fabric/semanticmodel/items/bind-semantic-model-connection\n binding_url = f\"{constants.FABRIC_API_ROOT_URL}/v1/workspaces/{fabric_workspace_obj.workspace_id}/semanticModels/{model_id}/bindConnection\"\n bind_response = fabric_workspace_obj.endpoint.invoke(\n method=\"POST\",\n url=binding_url,\n body=request_body,\n )\n\n status_code = bind_response.get(\"status_code\")\n\n if status_code == 200:\n logger.info(f\"Successfully bound semantic model '{model_name}' to connection '{connection_id}'\")\n else:\n logger.warning(f\"Failed to bind semantic model '{model_name}'. Status code: {status_code}\")\n\n except Exception as e:\n logger.error(f\"Failed to bind semantic model '{model_name}' to connection: {e!s}\")\n continue\n\n\ndef build_request_body(body: dict) -> dict:\n \"\"\"\n Build request body with specific order of fields for connection binding.\n\n Args:\n body: Dictionary containing connectionBinding data\n\n Returns:\n Ordered dictionary with id, connectivityType, and connectionDetails\n \"\"\"\n connection_binding = body.get(\"connectionBinding\", {})\n connection_details = connection_binding.get(\"connectionDetails\", {})\n\n return {\n \"connectionBinding\": {\n \"id\": connection_binding.get(\"id\"),\n \"connectivityType\": connection_binding.get(\"connectivityType\"),\n \"connectionDetails\": {\n \"type\": connection_details.get(\"type\") if \"type\" in connection_details else None,\n \"path\": connection_details.get(\"path\") if \"path\" in connection_details else None,\n },\n }\n }\n\n\nclass SemanticModelPublisher(ItemPublisher):\n \"\"\"Publisher for Semantic Model items.\"\"\"\n\n item_type = ItemType.SEMANTIC_MODEL.value\n\n def publish_one(self, item_name: str, _item: Item) -> None:\n \"\"\"Publish a single Semantic Model item.\"\"\"\n self.fabric_workspace_obj._publish_item(\n item_name=item_name, item_type=self.item_type, exclude_path=EXCLUDE_PATH_REGEX_MAPPING.get(self.item_type)\n )\n\n def post_publish_all(self) -> None:\n \"\"\"Bind semantic models to connections after all models are published.\"\"\"\n semantic_model_binding = self.fabric_workspace_obj.environment_parameter.get(\"semantic_model_binding\", {})\n if not semantic_model_binding:\n return\n\n # Build connection mapping from semantic_model_binding parameter (support legacy or new formats)\n environment = self.fabric_workspace_obj.environment\n\n if isinstance(semantic_model_binding, list):\n binding_mapping = build_binding_mapping_legacy(self.fabric_workspace_obj, semantic_model_binding)\n elif isinstance(semantic_model_binding, dict):\n binding_mapping = build_binding_mapping(self.fabric_workspace_obj, semantic_model_binding, environment)\n else:\n logger.warning(\n f\"Invalid 'semantic_model_binding' type: {type(semantic_model_binding).__name__}. \"\n \"Expected list or dict. Skipping semantic model binding.\"\n )\n return\n\n if binding_mapping:\n connections = get_connections(self.fabric_workspace_obj)\n bind_semanticmodel_to_connection(\n fabric_workspace_obj=self.fabric_workspace_obj,\n connections=connections,\n connection_details=binding_mapping,\n )\n" }, { "path": "src/fabric_cicd/_items/_sparkjobdefinition.py", "content": "# Copyright (c) Microsoft Corporation.\n# Licensed under the MIT License.\n\n\"\"\"Functions to process and deploy Spark Job Definition item.\"\"\"\n\nimport logging\n\nfrom fabric_cicd._common._item import Item\nfrom fabric_cicd._items._base_publisher import ItemPublisher\nfrom fabric_cicd.constants import API_FORMAT_MAPPING, ItemType\n\nlogger = logging.getLogger(__name__)\n\n\nclass SparkJobDefinitionPublisher(ItemPublisher):\n \"\"\"Publisher for Spark Job Definition items.\"\"\"\n\n item_type = ItemType.SPARK_JOB_DEFINITION.value\n\n def publish_one(self, item_name: str, _item: Item) -> None:\n \"\"\"Publish a single Spark Job Definition item.\"\"\"\n self.fabric_workspace_obj._publish_item(\n item_name=item_name, item_type=self.item_type, api_format=API_FORMAT_MAPPING.get(self.item_type)\n )\n" }, { "path": "src/fabric_cicd/_items/_sqldatabase.py", "content": "# Copyright (c) Microsoft Corporation.\n# Licensed under the MIT License.\n\n\"\"\"Functions to process and deploy SQL Database item.\"\"\"\n\nimport logging\n\nfrom fabric_cicd import constants\nfrom fabric_cicd._common._item import Item\nfrom fabric_cicd._items._base_publisher import ItemPublisher\nfrom fabric_cicd.constants import ItemType\n\nlogger = logging.getLogger(__name__)\n\n\nclass SQLDatabasePublisher(ItemPublisher):\n \"\"\"Publisher for SQL Database items.\"\"\"\n\n item_type = ItemType.SQL_DATABASE.value\n\n def publish_one(self, item_name: str, item: Item) -> None:\n \"\"\"Publish a single SQL Database item.\"\"\"\n self.fabric_workspace_obj._publish_item(\n item_name=item_name,\n item_type=self.item_type,\n skip_publish_logging=True,\n )\n\n # Check if the item is published to avoid any post publish actions\n if item.skip_publish:\n return\n\n logger.info(f\"{constants.INDENT}Published SQLDatabase '{item_name}'\")\n" }, { "path": "src/fabric_cicd/_items/_userdatafunction.py", "content": "# Copyright (c) Microsoft Corporation.\n# Licensed under the MIT License.\n\n\"\"\"Functions to process and deploy User Data Function item.\"\"\"\n\nfrom fabric_cicd._items._base_publisher import ItemPublisher\nfrom fabric_cicd.constants import ItemType\n\n\nclass UserDataFunctionPublisher(ItemPublisher):\n \"\"\"Publisher for User Data Function items.\"\"\"\n\n item_type = ItemType.USER_DATA_FUNCTION.value\n" }, { "path": "src/fabric_cicd/_items/_variablelibrary.py", "content": "# Copyright (c) Microsoft Corporation.\n# Licensed under the MIT License.\n\n\"\"\"Functions to process and deploy Variable Library item.\"\"\"\n\nimport json\nimport logging\n\nfrom fabric_cicd import FabricWorkspace, constants\nfrom fabric_cicd._common._item import Item\nfrom fabric_cicd._items._base_publisher import ItemPublisher\nfrom fabric_cicd.constants import ItemType\n\nlogger = logging.getLogger(__name__)\n\n\ndef activate_value_set(fabric_workspace_obj: FabricWorkspace, item_obj: Item) -> None:\n \"\"\"\n Activates the value set for the given Variable Library item.\n\n Args:\n fabric_workspace_obj: The FabricWorkspace object.\n item_obj: The item object.\n \"\"\"\n settings_file_obj = next((file for file in item_obj.item_files if file.name == \"settings.json\"), None)\n\n if settings_file_obj:\n settings_dict = json.loads(settings_file_obj.contents)\n if fabric_workspace_obj.environment in settings_dict[\"valueSetsOrder\"]:\n active_value_set = fabric_workspace_obj.environment\n else:\n active_value_set = \"Default value set\"\n logger.warning(\n f\"Provided target environment '{fabric_workspace_obj.environment}' does not match any value sets. Using '{active_value_set}'\"\n )\n\n body = {\"properties\": {\"activeValueSetName\": active_value_set}}\n\n fabric_workspace_obj.endpoint.invoke(\n method=\"PATCH\", url=f\"{fabric_workspace_obj.base_api_url}/VariableLibraries/{item_obj.guid}\", body=body\n )\n\n logger.info(f\"{constants.INDENT}Active value set changed to '{active_value_set}'\")\n\n else:\n logger.warning(f\"settings.json file not found for item {item_obj.name}. Active value set not changed.\")\n\n\nclass VariableLibraryPublisher(ItemPublisher):\n \"\"\"Publisher for Variable Library items.\"\"\"\n\n item_type = ItemType.VARIABLE_LIBRARY.value\n\n def publish_one(self, item_name: str, item: Item) -> None:\n \"\"\"Publish a single Variable Library item.\"\"\"\n self.fabric_workspace_obj._publish_item(item_name=item_name, item_type=self.item_type)\n if not item.skip_publish:\n activate_value_set(self.fabric_workspace_obj, item)\n" }, { "path": "src/fabric_cicd/_items/_warehouse.py", "content": "# Copyright (c) Microsoft Corporation.\n# Licensed under the MIT License.\n\n\"\"\"Functions to process and deploy Warehouse item.\"\"\"\n\nimport json\nimport logging\n\nfrom fabric_cicd import constants\nfrom fabric_cicd._common._item import Item\nfrom fabric_cicd._items._base_publisher import ItemPublisher\nfrom fabric_cicd.constants import ItemType\n\nlogger = logging.getLogger(__name__)\n\n\nclass WarehousePublisher(ItemPublisher):\n \"\"\"Publisher for Warehouse items.\"\"\"\n\n item_type = ItemType.WAREHOUSE.value\n\n def publish_one(self, item_name: str, item: Item) -> None:\n \"\"\"Publish a single Warehouse item.\"\"\"\n creation_payload = next(\n (\n json.loads(file.contents)[\"metadata\"][\"creationPayload\"]\n for file in item.item_files\n if file.name == \".platform\" and \"creationPayload\" in file.contents\n ),\n None,\n )\n\n self.fabric_workspace_obj._publish_item(\n item_name=item_name,\n item_type=self.item_type,\n creation_payload=creation_payload,\n skip_publish_logging=True,\n )\n\n # Check if the item is published to avoid any post publish actions\n if item.skip_publish:\n return\n\n logger.info(f\"{constants.INDENT}Published Warehouse '{item_name}'\")\n" }, { "path": "src/fabric_cicd/_parameter/__init__.py", "content": "# Copyright (c) Microsoft Corporation.\n# Licensed under the MIT License.\n\n" }, { "path": "src/fabric_cicd/_parameter/_parameter.py", "content": "# Copyright (c) Microsoft Corporation.\n# Licensed under the MIT License.\n\n\"\"\"Module provides the Parameter class to load and validate the parameter file used for deployment configurations.\"\"\"\n\nimport json\nimport logging\nimport os\nimport re\nfrom pathlib import Path\nfrom typing import ClassVar, Optional\n\nimport yaml\n\nimport fabric_cicd.constants as constants\nfrom fabric_cicd._parameter._utils import (\n is_valid_structure,\n process_input_path,\n replace_variables_in_parameter_file,\n)\n\n# Configure logging to output to the console\nlogger = logging.getLogger(__name__)\n\n\nclass Parameter:\n \"\"\"A class to validate the parameter file.\"\"\"\n\n PARAMETER_KEYS: ClassVar[dict] = {\n \"find_replace\": {\n \"minimum\": {\"find_value\", \"replace_value\"},\n \"maximum\": {\"find_value\", \"replace_value\", \"is_regex\", \"item_type\", \"item_name\", \"file_path\"},\n },\n \"spark_pool\": {\n \"minimum\": {\"instance_pool_id\", \"replace_value\"},\n \"maximum\": {\"instance_pool_id\", \"replace_value\", \"item_name\"},\n },\n \"spark_pool_replace_value\": {\"type\", \"name\"},\n \"key_value_replace\": {\n \"minimum\": {\"find_key\", \"replace_value\"},\n \"maximum\": {\"find_key\", \"replace_value\", \"item_type\", \"item_name\", \"file_path\"},\n },\n \"gateway_binding\": {\n \"minimum\": {\"gateway_id\", \"dataset_name\"},\n \"maximum\": {\"gateway_id\", \"dataset_name\"},\n },\n \"semantic_model_binding\": {\n \"minimum\": set(),\n \"maximum\": {\"connection_id\", \"semantic_model_name\", \"default\", \"models\"},\n },\n \"extend\": {\"minimum\": set(), \"maximum\": set()},\n }\n\n LOAD_ERROR_MSG = \"\"\n\n def __init__(\n self,\n repository_directory: Path,\n item_type_in_scope: list[str],\n environment: str,\n parameter_file_name: str = \"parameter.yml\",\n parameter_file_path: Optional[str] = None,\n ) -> None:\n \"\"\"\n Initializes the Parameter instance.\n\n Args:\n repository_directory: Local directory path of the repository where items are to be deployed from and parameter file lives.\n item_type_in_scope: Item types that should be deployed for a given workspace.\n environment: The environment to be used for parameterization.\n parameter_file_name: The name of the parameter file, default is \"parameter.yml\".\n parameter_file_path: The path to the parameter file, if not using the default.\n \"\"\"\n # Set class variables\n self.repository_directory = repository_directory\n self.item_type_in_scope = item_type_in_scope\n self.environment = environment\n self.parameter_file_name = parameter_file_name\n self.parameter_file_path = parameter_file_path\n\n self._set_parameter_file_path()\n self._refresh_parameter_file()\n\n def _set_parameter_file_path(self) -> None:\n \"\"\"Set the parameter file path based on the provided path or default name.\"\"\"\n is_param_path = False\n original_param_path = None\n\n # Determine which input to use for parameter file path\n if self.parameter_file_path and isinstance(self.parameter_file_path, str):\n original_param_path = self.parameter_file_path\n if self.parameter_file_name != \"parameter.yml\":\n is_param_path = True\n logger.warning(\n constants.PARAMETER_MSGS[\"both_param_path_and_name\"].format(\n self.parameter_file_name, original_param_path\n )\n )\n else:\n is_param_path = True\n\n try:\n # Resolve parameter file path, if provided\n if is_param_path and original_param_path:\n try:\n param_path = Path(original_param_path)\n # Handle relative path (must be relative to repository_directory)\n if not param_path.is_absolute():\n logger.debug(constants.PARAMETER_MSGS[\"resolving_relative_path\"].format(original_param_path))\n param_path = Path(self.repository_directory, original_param_path)\n\n self.parameter_file_path = param_path.resolve()\n logger.debug(constants.PARAMETER_MSGS[\"using_param_file_path\"].format(self.parameter_file_path))\n\n except (TypeError, ValueError) as e:\n logger.error(f\"Error setting parameter file path: {e}\")\n is_param_path = False\n\n # Otherwise, resolve with default path\n if not is_param_path:\n self.parameter_file_path = Path(self.repository_directory, self.parameter_file_name).resolve()\n logger.debug(constants.PARAMETER_MSGS[\"using_default_param_file_path\"].format(self.parameter_file_path))\n\n except Exception as e:\n logger.error(f\"Unexpected error setting parameter file path: {e}\")\n self.parameter_file_path = None\n\n def _refresh_parameter_file(self) -> None:\n \"\"\"Load parameters if file is present.\"\"\"\n self.environment_parameter = {}\n\n # Only proceed if the parameter file exists\n if self._validate_parameter_file_exists():\n is_valid, environment_parameter = self._validate_load_parameters_to_dict()\n if is_valid:\n self.environment_parameter = environment_parameter\n\n def _validate_parameter_file_exists(self) -> bool:\n \"\"\"Validate the parameter file exists.\"\"\"\n if self.parameter_file_path is None:\n return False\n\n return self.parameter_file_path.is_file()\n\n def _validate_load_parameters_to_dict(self) -> tuple[bool, dict]:\n \"\"\"Validate loading the parameter file to a dictionary, including any templates.\"\"\"\n parameter_dict = {}\n try:\n # Load the base parameter file\n with Path.open(self.parameter_file_path, encoding=\"utf-8\") as yaml_file:\n yaml_content = yaml_file.read()\n yaml_content = replace_variables_in_parameter_file(yaml_content)\n\n # Check for empty YAML content\n if not yaml_content.strip():\n self.LOAD_ERROR_MSG = constants.PARAMETER_MSGS[\"invalid load\"].format(\n constants.PARAMETER_MSGS[\"empty yaml\"]\n )\n return False, parameter_dict\n\n # Use custom loader that detects duplicate keys\n parameter_dict = yaml.load(yaml_content, Loader=_DuplicateKeyLoader) or {}\n logger.debug(constants.PARAMETER_MSGS[\"passed\"].format(\"YAML content is valid\"))\n\n if parameter_dict.get(\"extend\"):\n parameter_dict = self._process_template_parameters(parameter_dict)\n\n return True, parameter_dict\n\n except (UnicodeDecodeError, yaml.YAMLError) as e:\n self.LOAD_ERROR_MSG = constants.PARAMETER_MSGS[\"invalid load\"].format(e)\n return False, parameter_dict\n\n def _process_template_parameters(self, base_parameter_dict: dict) -> dict:\n \"\"\"\n Process template parameter files and merge them with the base parameter dictionary.\n Template files are resolved relative to the main parameter file's location.\n \"\"\"\n # Step 1: Check extend contains files\n if not isinstance(base_parameter_dict.get(\"extend\"), list):\n logger.warning(\"No template parameter files specified under 'extend'\")\n del base_parameter_dict[\"extend\"]\n return base_parameter_dict\n\n template_files = base_parameter_dict[\"extend\"]\n successful_templates = 0\n failed_templates = []\n processed_templates = set()\n\n # Step 2: Get the directory containing the main parameter file\n param_file_dir = self.parameter_file_path.parent\n\n # Step 3: Process each template file\n for param_file in template_files:\n try:\n # Check if this template file has been already processed to prevent duplication\n if param_file in processed_templates:\n logger.warning(f\"Skipping duplicate template parameter file reference: {param_file}\")\n continue\n\n # Step a: Resolve the path relative to the main parameter file's directory\n template_path = (param_file_dir / str(param_file)).resolve()\n\n # Check if the template file exists\n if not template_path.is_file():\n error_msg = f\"Template file not found: {param_file}\"\n failed_templates.append((param_file, error_msg))\n continue\n\n # Step b: Load and validate the parameter file\n template_dict = self._load_template_parameter_file(template_path)\n if not template_dict:\n continue\n\n # Step c: Check for nested templates\n if \"extend\" in template_dict:\n error_msg = f\"Nested templates are not supported in {param_file}\"\n failed_templates.append((param_file, error_msg))\n continue\n\n # Step d: Merge the template dict with the base parameter dict\n base_parameter_dict = self._merge_template_dict(base_parameter_dict, template_dict)\n successful_templates += 1\n # Mark template parameter file as processed\n processed_templates.add(param_file)\n logger.debug(constants.PARAMETER_MSGS[\"template_file_loaded\"].format(template_path))\n\n except Exception as e:\n error_msg = f\"Error processing template file: {e!s}\"\n failed_templates.append((param_file, error_msg))\n continue\n\n # Step 4: Log results\n if successful_templates > 0:\n logger.debug(constants.PARAMETER_MSGS[\"template_files_processed\"].format(successful_templates))\n\n if failed_templates:\n for failed_file, reason in failed_templates:\n logger.error(f\"Validation failed for template file: {failed_file}\")\n logger.error(f\"{reason}\")\n logger.warning(\n f\"Template parameter '{failed_file}' content will not be included in the parameter dictionary\"\n )\n elif successful_templates == 0:\n logger.warning(constants.PARAMETER_MSGS[\"template_files_none_valid\"])\n\n # Step 5: Remove the extend key after processing\n del base_parameter_dict[\"extend\"]\n return base_parameter_dict\n\n def _load_template_parameter_file(self, file_path: Path) -> dict:\n \"\"\"Load and validate a template parameter file.\"\"\"\n try:\n with Path.open(file_path, encoding=\"utf-8\") as param_file:\n param_content = param_file.read()\n param_content = replace_variables_in_parameter_file(param_content)\n\n # Check for empty YAML content\n if not param_content.strip():\n logger.error(\n constants.PARAMETER_MSGS[\"template_file_invalid\"].format(\n file_path, constants.PARAMETER_MSGS[\"empty yaml\"]\n )\n )\n return {}\n\n # Use custom loader that detects duplicate keys\n return yaml.load(param_content, Loader=_DuplicateKeyLoader) or {}\n\n except (UnicodeDecodeError, yaml.YAMLError) as e:\n logger.error(constants.PARAMETER_MSGS[\"template_file_error\"].format(file_path, e))\n return {}\n\n def _merge_template_dict(self, base_dict: dict, template_dict: dict) -> dict:\n \"\"\"\n Merge the template dictionary with the base dictionary, properly handling lists and nested structures.\n Preserves all entries, letting validation handle any issues later.\n \"\"\"\n result = base_dict.copy()\n\n for key, template_value in template_dict.items():\n # Skip the 'extend' key as it's processed separately\n if key == \"extend\":\n continue\n\n # If the key doesn't exist in the base dict, just add it\n if key not in result:\n result[key] = template_value\n continue\n\n base_value = result[key]\n\n # Handle merging based on value types\n if isinstance(base_value, list) and isinstance(template_value, list):\n # For parameter lists like find_replace, append items from template\n result[key] = base_value + template_value\n\n elif isinstance(base_value, dict) and isinstance(template_value, dict):\n # For nested dictionaries, recursively merge them\n result[key] = self._merge_template_dict(base_value, template_value)\n\n else:\n # Add both values into a list for later validation\n result[key] = [base_value, template_value]\n logger.debug(f\"Type mismatch for key '{key}': creating list of values for validation\")\n\n return result\n\n def _validate_parameter_load(self) -> tuple[bool, str]:\n \"\"\"Validate the parameter file load.\"\"\"\n if self.parameter_file_path is None:\n return False, \"not set\"\n\n if not self.environment_parameter:\n # Check if the file exists\n if not self._validate_parameter_file_exists():\n logger.warning(constants.PARAMETER_MSGS[\"not found\"].format(self.parameter_file_path))\n return False, \"not found\"\n logger.debug(constants.PARAMETER_MSGS[\"found\"])\n return False, self.LOAD_ERROR_MSG\n\n return True, constants.PARAMETER_MSGS[\"valid load\"]\n\n def _validate_parameter_file(self) -> bool:\n \"\"\"Validate the parameter file.\"\"\"\n # Handle gateway_binding deprecation\n if \"gateway_binding\" in self.environment_parameter or \"semantic_model_binding\" in self.environment_parameter:\n self._handle_gateway_binding_parameter()\n\n validation_steps = [\n (\"parameter file load\", self._validate_parameter_load),\n (\"parameter names\", self._validate_parameter_names),\n (\"parameter file structure\", self._validate_parameter_structure),\n (\"find_replace parameter\", lambda: self._validate_parameter(\"find_replace\")),\n (\"spark_pool parameter\", lambda: self._validate_parameter(\"spark_pool\")),\n (\"key_value_replace parameter\", lambda: self._validate_parameter(\"key_value_replace\")),\n (\"semantic_model_binding parameter\", lambda: self._validate_parameter(\"semantic_model_binding\")),\n ]\n for step, validation_func in validation_steps:\n logger.debug(constants.PARAMETER_MSGS[\"validating\"].format(step))\n is_valid, msg = validation_func()\n if not is_valid:\n # Return True for specific not is_valid case\n if step == \"parameter file load\" and msg == \"not found\":\n logger.warning(constants.PARAMETER_MSGS[\"terminate\"].format(msg))\n return True\n # Discontinue validation check for absent parameter\n if (\n step\n in (\n \"find_replace parameter\",\n \"key_value_replace parameter\",\n \"spark_pool parameter\",\n \"semantic_model_binding parameter\",\n )\n and msg == \"parameter not found\"\n ):\n continue\n # Otherwise, return False with error message\n logger.error(constants.PARAMETER_MSGS[\"failed\"].format(msg))\n return False\n logger.debug(constants.PARAMETER_MSGS[\"passed\"].format(msg))\n\n # Return True if all validation steps pass\n logger.info(constants.PARAMETER_MSGS[\"validation_complete\"])\n return True\n\n def _handle_gateway_binding_parameter(self) -> None:\n \"\"\"This code will be removed in future releases, after deprecating 'gateway_binding' support.\"\"\"\n # Extend semantic_model_binding by adding gateway_binding into one dict\n sm_param = self.environment_parameter.get(\"semantic_model_binding\", [])\n gw_list = self.environment_parameter.get(\"gateway_binding\", [])\n\n if gw_list:\n # Only merge gateway_binding if semantic_model_binding is legacy format (list) or doesn't exist\n if isinstance(sm_param, list):\n # Transform gateway entries to semantic_model_binding shape\n sm_param.extend(\n {\n \"connection_id\": entry.get(\"gateway_id\"),\n \"semantic_model_name\": entry.get(\"dataset_name\", []),\n }\n for entry in gw_list\n )\n self.environment_parameter[\"semantic_model_binding\"] = sm_param\n else:\n # New format (dict) - cannot merge gateway_binding, log warning\n logger.warning(\n \"Cannot merge 'gateway_binding' with new format 'semantic_model_binding'. \"\n \"Please migrate gateway_binding entries to the new semantic_model_binding format.\"\n )\n\n # Remove original gateway_binding after processing\n del self.environment_parameter[\"gateway_binding\"]\n logger.warning(constants.PARAMETER_MSGS[\"gateway_deprecated\"])\n\n def _validate_parameter_structure(self) -> tuple[bool, str]:\n \"\"\"Validate the parameter file structure.\"\"\"\n if not is_valid_structure(self.environment_parameter):\n return False, constants.PARAMETER_MSGS[\"invalid structure\"]\n\n return True, constants.PARAMETER_MSGS[\"valid structure\"]\n\n def _validate_parameter_names(self) -> tuple[bool, str]:\n \"\"\"Validate the parameter names in the parameter dictionary.\"\"\"\n params = list(self.PARAMETER_KEYS.keys())[:6]\n for param in self.environment_parameter:\n if param not in params:\n return False, constants.PARAMETER_MSGS[\"invalid name\"].format(param)\n\n return True, constants.PARAMETER_MSGS[\"valid name\"]\n\n def _validate_parameter(self, param_name: str) -> tuple[bool, str]:\n \"\"\"Validate the specific parameter and its contents.\"\"\"\n if param_name not in self.environment_parameter:\n logger.debug(constants.PARAMETER_MSGS[\"param_not_found\"].format(param_name))\n return False, \"parameter not found\"\n\n logger.debug(constants.PARAMETER_MSGS[\"param_found\"].format(param_name))\n\n param_count = len(self.environment_parameter[param_name])\n multiple_param = param_count > 1\n if multiple_param:\n logger.debug(constants.PARAMETER_MSGS[\"param_count\"].format(param_count, param_name))\n\n # Run a separate validation for semantic_model_binding parameter\n if param_name == \"semantic_model_binding\":\n return self._validate_semantic_model_binding_parameter(param_name, multiple_param)\n\n # Validation for other kinds of parameters\n validation_steps = [\n (\"keys\", lambda param_dict: self._validate_parameter_keys(param_name, list(param_dict.keys()))),\n (\"required values\", lambda param_dict: self._validate_required_values(param_name, param_dict)),\n (\"replace_value\", lambda param_dict: self._validate_replace_value(param_name, param_dict[\"replace_value\"])),\n (\"optional values\", lambda param_dict: self._validate_optional_values(param_name, param_dict)),\n ]\n # Set the proper find_value key name based on the parameter\n if param_name == \"key_value_replace\":\n key_name = \"find_key\"\n elif param_name == \"spark_pool\":\n key_name = \"instance_pool_id\"\n else:\n key_name = \"find_value\"\n\n for param_num, parameter_dict in enumerate(self.environment_parameter[param_name], start=1):\n param_num_str = str(param_num) if multiple_param else \"\"\n find_value = parameter_dict.get(key_name)\n for step, validation_func in validation_steps:\n logger.debug(constants.PARAMETER_MSGS[\"validating\"].format(f\"{param_name} {param_num_str} {step}\"))\n is_valid, msg = validation_func(parameter_dict)\n if not is_valid:\n return False, msg\n logger.debug(constants.PARAMETER_MSGS[\"passed\"].format(msg))\n\n # Check if replacement will be skipped for a given find value\n is_valid_env, env_type = self._validate_environment(parameter_dict[\"replace_value\"])\n is_valid_optional_val, msg = self._validate_optional_values(param_name, parameter_dict, check_match=True)\n log_func = logger.debug if param_name == \"key_value_replace\" else logger.warning\n\n # Set value_type based on regex flag once\n value_type = (\n \"find value regex\"\n if (parameter_dict.get(\"is_regex\") and parameter_dict[\"is_regex\"].lower() == \"true\")\n else \"find value\"\n )\n\n if self.environment != \"N/A\" and not is_valid_env:\n if env_type.lower() == \"_all_\":\n return False, constants.PARAMETER_MSGS[\"other target env\"].format(\n env_type, parameter_dict[\"replace_value\"]\n )\n\n skip_msg = constants.PARAMETER_MSGS[\"no target env\"].format(self.environment, param_name)\n log_func(\n constants.PARAMETER_MSGS[\"skip\"].format(\n value_type, find_value, skip_msg, param_name + \" \" + param_num_str\n )\n )\n continue\n\n if env_type.lower() == \"_all_\":\n logger.warning(\n constants.PARAMETER_MSGS[\"all target env\"].format(parameter_dict[\"replace_value\"][env_type])\n )\n\n if msg == \"no match\" and not is_valid_optional_val:\n skip_msg = constants.PARAMETER_MSGS[\"no filter match\"]\n log_func(\n constants.PARAMETER_MSGS[\"skip\"].format(\n value_type, find_value, skip_msg, param_name + \" \" + param_num_str\n )\n )\n\n return True, constants.PARAMETER_MSGS[\"valid parameter\"].format(param_name)\n\n def _validate_semantic_model_binding_parameter(\n self, param_name: str, multiple_param: bool = False\n ) -> tuple[bool, str]:\n \"\"\"Validate semantic_model_binding parameter, supporting both legacy and new formats.\n\n Legacy format (list): Only string connection_id allowed (no environment mapping)\n New format (dict): Only dict connection_id allowed (environment mapping required, use _ALL_ for all)\n \"\"\"\n param_value = self.environment_parameter.get(param_name)\n is_new_format = isinstance(param_value, dict)\n legacy_keys = {\"connection_id\", \"semantic_model_name\"}\n new_keys = {\"default\", \"models\"}\n\n if is_new_format:\n # New format: dict with 'default' and/or 'models'\n param_keys_set = set(param_value.keys())\n\n # Validate top-level keys\n if param_keys_set & legacy_keys:\n return False, constants.PARAMETER_MSGS[\"mixed format\"].format(param_name)\n if not param_keys_set <= new_keys:\n return False, constants.PARAMETER_MSGS[\"invalid key\"].format(param_name)\n\n # Require at least one of 'default' or 'models'\n if \"default\" not in param_value and \"models\" not in param_value:\n return False, constants.PARAMETER_MSGS[\"missing key\"].format(\n f\"{param_name} (requires 'default' or 'models')\"\n )\n\n # Validate 'default' section\n if \"default\" in param_value:\n default_value = param_value[\"default\"]\n is_valid, msg = self._validate_data_type(default_value, \"dictionary\", \"default\", param_name)\n if not is_valid:\n return False, msg\n if \"connection_id\" not in default_value:\n return False, constants.PARAMETER_MSGS[\"missing key\"].format(\n f\"{param_name}.default (requires 'connection_id')\"\n )\n # New format requires dict connection_id\n is_valid, msg = self._validate_connection_id(\n default_value[\"connection_id\"], f\"{param_name}.default.connection_id\", require_dict=True\n )\n if not is_valid:\n return False, msg\n\n # Validate 'models' section\n if \"models\" in param_value:\n models_value = param_value[\"models\"]\n section_name = f\"{param_name}.models\"\n\n if not isinstance(models_value, list) or not models_value:\n return False, f\"{section_name} must be a non-empty list\"\n\n for idx, entry in enumerate(models_value):\n entry_name = f\"{section_name}[{idx}]\"\n\n is_valid, msg = self._validate_data_type(entry, \"dictionary\", entry_name, section_name)\n if not is_valid:\n return False, msg\n if \"semantic_model_name\" not in entry:\n return False, constants.PARAMETER_MSGS[\"missing key\"].format(\n f\"{entry_name} (requires 'semantic_model_name')\"\n )\n if \"connection_id\" not in entry:\n return False, constants.PARAMETER_MSGS[\"missing key\"].format(\n f\"{entry_name} (requires 'connection_id')\"\n )\n\n is_valid, msg = self._validate_data_type(\n entry[\"semantic_model_name\"], \"string or list[string]\", \"semantic_model_name\", entry_name\n )\n if not is_valid:\n return False, msg\n\n # New format requires dict connection_id\n is_valid, msg = self._validate_connection_id(\n entry[\"connection_id\"], f\"{entry_name}.connection_id\", require_dict=True\n )\n if not is_valid:\n return False, msg\n\n else:\n # Legacy format: list of dicts with 'connection_id' and 'semantic_model_name'\n for param_num, entry in enumerate(param_value, start=1):\n context_name = f\"{param_name} {param_num}\" if multiple_param else param_name\n param_keys_set = set(entry.keys())\n\n # Validate keys\n if param_keys_set & new_keys:\n return False, constants.PARAMETER_MSGS[\"mixed format\"].format(param_name)\n if not legacy_keys <= param_keys_set:\n return False, constants.PARAMETER_MSGS[\"missing key\"].format(param_name)\n if not param_keys_set <= legacy_keys:\n return False, constants.PARAMETER_MSGS[\"invalid key\"].format(param_name)\n\n connection_id = entry.get(\"connection_id\")\n semantic_model_name = entry.get(\"semantic_model_name\")\n\n if not connection_id:\n return False, constants.PARAMETER_MSGS[\"missing required value\"].format(\"connection_id\", param_name)\n if not semantic_model_name:\n return False, constants.PARAMETER_MSGS[\"missing required value\"].format(\n \"semantic_model_name\", param_name\n )\n\n is_valid, msg = self._validate_data_type(\n semantic_model_name, \"string or list[string]\", \"semantic_model_name\", param_name\n )\n if not is_valid:\n return False, msg\n\n # Legacy format requires string connection_id (no environment mapping)\n is_valid, msg = self._validate_connection_id(connection_id, context_name, require_string=True)\n if not is_valid:\n return False, msg\n\n # Check for duplicate semantic model names\n self._check_duplicate_semantic_model_names(param_value, is_new_format)\n\n return True, constants.PARAMETER_MSGS[\"valid parameter\"].format(param_name)\n\n def _validate_connection_id(\n self, connection_id: any, context_name: str, require_string: bool = False, require_dict: bool = False\n ) -> tuple[bool, str]:\n \"\"\"Validate a connection_id value.\"\"\"\n # Legacy format: require string GUID only\n if require_string:\n if not isinstance(connection_id, str):\n return False, (\n f\"connection_id in {context_name} must be a string GUID. \"\n \"Environment-specific dictionaries are not supported in the legacy format. \"\n \"Please migrate to the new dictionary format with 'default' and 'models' keys.\"\n )\n if not re.match(constants.VALID_GUID_REGEX, connection_id):\n return False, f\"connection_id '{connection_id}' is not a valid GUID in {context_name}\"\n return True, f\"Valid {context_name}\"\n\n # New format: require dict with environment keys\n if require_dict:\n if not isinstance(connection_id, dict):\n return False, (\n f\"{context_name} must be a dictionary with environment keys (e.g., DEV, PPE, PROD) or '_ALL_'. \"\n \"Use '_ALL_' to apply the same connection to all environments.\"\n )\n if not connection_id:\n return False, f\"{context_name} must be a non-empty dictionary\"\n\n for env_key, guid_value in connection_id.items():\n if not isinstance(guid_value, str):\n return False, f\"connection_id value for environment '{env_key}' must be a string (GUID)\"\n if not re.match(constants.VALID_GUID_REGEX, guid_value):\n return False, f\"connection_id for environment '{env_key}' is not a valid GUID: '{guid_value}'\"\n\n # Validate environment exists\n is_valid_env, env_type = self._validate_environment(connection_id)\n if self.environment != \"N/A\" and not is_valid_env:\n if env_type.lower() == \"_all_\":\n return False, constants.PARAMETER_MSGS[\"other target env\"].format(env_type, connection_id)\n logger.warning(constants.PARAMETER_MSGS[\"no target env\"].format(self.environment, context_name))\n\n return True, f\"Valid {context_name}\"\n\n # Should not reach here - caller must specify require_string or require_dict\n return False, f\"Invalid validation call for {context_name}: must specify require_string or require_dict\"\n\n def _check_duplicate_semantic_model_names(self, param_value: any, is_new_format: bool) -> None:\n \"\"\"Check for duplicate semantic model names and log a warning if found.\"\"\"\n names = []\n\n if is_new_format:\n for item in param_value.get(\"models\", []):\n raw = item.get(\"semantic_model_name\", [])\n if isinstance(raw, str):\n names.append(raw)\n elif isinstance(raw, list):\n names.extend(n for n in raw if isinstance(n, str))\n else:\n for entry in param_value:\n raw = entry.get(\"semantic_model_name\", [])\n if isinstance(raw, str):\n names.append(raw)\n elif isinstance(raw, list):\n names.extend(n for n in raw if isinstance(n, str))\n\n duplicates = {n for n in names if names.count(n) > 1}\n if duplicates:\n logger.warning(constants.PARAMETER_MSGS[\"duplicate_semantic_model\"].format(\", \".join(sorted(duplicates))))\n\n def _validate_parameter_keys(self, param_name: str, param_keys: list) -> tuple[bool, str]:\n \"\"\"Validate the keys in the parameter.\"\"\"\n param_keys_set = set(param_keys)\n\n # Validate minimum set\n if not self.PARAMETER_KEYS[param_name][\"minimum\"] <= param_keys_set:\n return False, constants.PARAMETER_MSGS[\"missing key\"].format(param_name)\n\n # Validate maximum set\n if not param_keys_set <= self.PARAMETER_KEYS[param_name][\"maximum\"]:\n return False, constants.PARAMETER_MSGS[\"invalid key\"].format(param_name)\n\n return True, constants.PARAMETER_MSGS[\"valid keys\"].format(param_name)\n\n def _validate_required_values(self, param_name: str, param_dict: dict) -> tuple[bool, str]:\n \"\"\"Validate required values in the parameter.\"\"\"\n for key in self.PARAMETER_KEYS[param_name][\"minimum\"]:\n if not param_dict.get(key):\n return False, constants.PARAMETER_MSGS[\"missing required value\"].format(key, param_name)\n\n expected_type = \"dictionary\" if key == \"replace_value\" else \"string\"\n is_valid, msg = self._validate_data_type(param_dict[key], expected_type, key, param_name)\n if not is_valid:\n return False, msg\n\n # Validate find_value is a valid regex if is_regex is set to true\n if param_name == \"find_replace\":\n is_valid, msg = self._validate_find_regex(param_name, param_dict)\n if not is_valid:\n return False, msg\n\n if param_name == \"key_value_replace\":\n is_valid, msg = self._validate_key_value_find_key(param_dict)\n if not is_valid:\n return False, msg\n\n return True, constants.PARAMETER_MSGS[\"valid required values\"].format(param_name)\n\n def _validate_key_value_find_key(self, param_dict: dict) -> tuple[bool, str]:\n \"\"\"Validate the `find_key` JSONPath for key_value_replace (compile-only).\"\"\"\n find_key = param_dict.get(\"find_key\")\n if not find_key or not isinstance(find_key, str) or not find_key.strip():\n return False, \"Missing or empty 'find_key' for key_value_replace\"\n\n # Require absolute JSONPath root to avoid ambiguous relative paths\n if not find_key.strip().startswith(\"$\"):\n return False, \"find_key must be an absolute JSONPath starting with '$'\"\n\n try:\n # jsonpath_ng.ext.parse supports extended JSONPath (dot/bracket)\n from jsonpath_ng.ext import parse\n\n parse(find_key)\n except Exception as e:\n return False, f\"Invalid JSONPath expression '{find_key}': {e}\"\n\n return True, \"Valid JSONPath\"\n\n def _validate_find_regex(self, param_name: str, param_dict: dict) -> tuple[bool, str]:\n \"\"\"Validate the find_value is a valid regex if is_regex is set to true.\"\"\"\n # Return True if is_regex is not present or set\n if not param_dict.get(\"is_regex\"):\n return True, \"No regex present\"\n\n # First validate is_regex value\n is_valid, msg = self._validate_data_type(param_dict.get(\"is_regex\"), \"string\", \"is_regex\", param_name)\n if not is_valid:\n return False, msg\n\n # Skip regex validation if is_regex is not set to true\n if param_dict[\"is_regex\"].lower() != \"true\":\n logger.warning(constants.PARAMETER_MSGS[\"regex_ignored\"])\n return True, \"Skip regex validation\"\n\n # Validate the find_value is a valid regex\n pattern = param_dict[\"find_value\"]\n try:\n re.compile(pattern)\n return True, \"Valid regex\"\n except re.error as e:\n return False, f\"Invalid regex {pattern}: {e}\"\n\n def _validate_replace_value(self, param_name: str, replace_value: dict) -> tuple[bool, str]:\n \"\"\"Validate the replace_value dictionary.\"\"\"\n # Validate replace_value dictionary values\n if param_name == \"find_replace\":\n is_valid, msg = self._validate_find_replace_replace_value(replace_value)\n\n if param_name == \"key_value_replace\":\n is_valid, msg = self._validate_key_value_replace_replace_value(replace_value)\n\n if param_name == \"spark_pool\":\n is_valid, msg = self._validate_spark_pool_replace_value(replace_value)\n\n if not is_valid:\n return False, msg\n\n return True, msg\n\n def _validate_find_replace_replace_value(self, replace_value: dict) -> tuple[bool, str]:\n \"\"\"Validate the replace_value dictionary values in find_replace parameters.\"\"\"\n for environment in replace_value:\n if not replace_value[environment]:\n return False, constants.PARAMETER_MSGS[\"missing replace value\"].format(\"find_replace\", environment)\n is_valid, msg = self._validate_data_type(\n replace_value[environment], \"string\", environment + \" replace_value\", param_name=\"find_replace\"\n )\n if not is_valid:\n return False, msg\n\n return True, constants.PARAMETER_MSGS[\"valid replace value\"].format(\"find_replace\")\n\n def _validate_key_value_replace_replace_value(self, replace_value: dict) -> tuple[bool, str]:\n \"\"\"Validate the replace_value dictionary values in key_value_replace parameters.\n\n For key_value_replace, we allow any data type but all values should be of the same type\n to ensure consistency when replacing values in JSON/YAML files.\n \"\"\"\n if not replace_value:\n return False, constants.PARAMETER_MSGS[\"missing replace value\"].format(\"key_value_replace\", \"any\")\n\n # Get the first value to determine the expected type\n first_env = next(iter(replace_value))\n first_value = replace_value[first_env]\n expected_type = type(first_value)\n\n for environment in replace_value:\n value = replace_value[environment]\n\n # Check if value is None/empty (not allowed)\n if value is None:\n return False, constants.PARAMETER_MSGS[\"missing replace value\"].format(\"key_value_replace\", environment)\n\n # Check type consistency across all environments\n if type(value) != expected_type:\n return (\n False,\n f\"Inconsistent data types in key_value_replace replace_value: \"\n f\"'{first_env}' has type {expected_type.__name__} but \"\n f\"'{environment}' has type {type(value).__name__}. \"\n f\"All values must be of the same type.\",\n )\n\n return True, constants.PARAMETER_MSGS[\"valid replace value\"].format(\"key_value_replace\")\n\n def _validate_spark_pool_replace_value(self, replace_value: dict) -> tuple[bool, str]:\n \"\"\"Validate the replace_value dictionary values in spark_pool parameter.\"\"\"\n for environment, environment_dict in replace_value.items():\n # Check if environment_dict is empty\n if not environment_dict:\n return False, constants.PARAMETER_MSGS[\"missing replace value\"].format(\"spark_pool\", environment)\n\n is_valid, msg = self._validate_data_type(\n environment_dict, \"dictionary\", environment + \" key\", param_name=\"spark_pool\"\n )\n if not is_valid:\n return False, msg\n\n msgs = constants.PARAMETER_MSGS[\"invalid replace value\"]\n # Validate keys for the environment\n config_keys = list(environment_dict.keys())\n required_keys = self.PARAMETER_KEYS[\"spark_pool_replace_value\"]\n if not required_keys.issubset(config_keys) or len(config_keys) != len(required_keys):\n return False, msgs[\"missing key\"].format(environment)\n\n # Validate values for the environment dict\n for key in config_keys:\n if not environment_dict[key]:\n return False, msgs[\"missing value\"].format(environment, key)\n\n is_valid, msg = self._validate_data_type(environment_dict[key], \"string\", key, param_name=\"spark_pool\")\n if not is_valid:\n return False, msg\n\n if environment_dict[\"type\"] not in [\"Capacity\", \"Workspace\"]:\n return False, msgs[\"invalid value\"].format(environment)\n\n return True, constants.PARAMETER_MSGS[\"valid replace value\"].format(\"spark_pool\")\n\n def _validate_optional_values(\n self, param_name: str, param_dict: dict, check_match: bool = False\n ) -> tuple[bool, str]:\n \"\"\"Validate the optional filter values in the parameter.\"\"\"\n optional_values = {\n \"item_type\": param_dict.get(\"item_type\"),\n \"item_name\": param_dict.get(\"item_name\"),\n \"file_path\": param_dict.get(\"file_path\"),\n }\n if (param_name == \"find_replace\" and not any(optional_values.values())) or (\n param_name == \"spark_pool\" and not optional_values[\"item_name\"]\n ):\n return True, constants.PARAMETER_MSGS[\"no optional\"].format(param_name)\n\n validation_methods = {\n \"item_type\": self._validate_item_type,\n \"item_name\": self._validate_item_name,\n \"file_path\": self._validate_file_path,\n }\n\n for param, value in optional_values.items():\n if value:\n # Check value data type\n is_valid, msg = self._validate_data_type(value, \"string or list[string]\", param, param_name)\n if not is_valid:\n return False, msg\n\n # Validate specific optional values and check for matches\n if check_match and param in validation_methods:\n values = value if isinstance(value, list) else [value]\n if param == \"file_path\":\n is_valid, msg = validation_methods[param](values)\n else:\n for item in values:\n is_valid, msg = validation_methods[param](item)\n\n if not is_valid:\n logger.error(msg)\n return False, \"no match\"\n\n return True, constants.PARAMETER_MSGS[\"valid optional\"].format(param_name)\n\n def _validate_data_type(\n self, input_value: any, expected_type: str, input_name: str, param_name: str\n ) -> tuple[bool, str]:\n \"\"\"Validate the data type of the input value.\"\"\"\n type_validators = {\n \"string\": lambda x: isinstance(x, str),\n \"string or list[string]\": lambda x: (isinstance(x, str))\n or (isinstance(x, list) and all(isinstance(item, str) for item in x)),\n \"dictionary\": lambda x: isinstance(x, dict),\n }\n # Check if the expected type is valid and if the input matches the expected type\n if expected_type not in type_validators or not type_validators[expected_type](input_value):\n return False, constants.PARAMETER_MSGS[\"invalid data type\"].format(input_name, expected_type, param_name)\n\n return True, \"Data type is valid\"\n\n def _validate_environment(self, replace_value: dict) -> tuple[bool, str]:\n \"\"\"\n Check the target environment exists as a key in the replace_value dictionary.\n If \"_ALL_\" (case insensitive) is present, it must be the only key.\n \"\"\"\n # Check for _ALL_ in any case variation\n all_key = None\n for key in replace_value:\n if key.lower() == \"_all_\":\n logger.warning(f\"Found the reserved environment key '{key}'\")\n all_key = key\n break\n if all_key:\n # If _ALL_ is present, it must be the only key\n return len(replace_value) == 1, all_key\n\n # If _ALL_ is not present, check if target environment is present\n return self.environment in replace_value, \"env\"\n\n def _validate_item_type(self, input_type: str) -> tuple[bool, str]:\n \"\"\"Validate the item type is in scope.\"\"\"\n if input_type not in self.item_type_in_scope:\n return False, constants.PARAMETER_MSGS[\"invalid item type\"].format(input_type)\n\n return True, \"Valid item type\"\n\n def _validate_item_name(self, input_name: str) -> tuple[bool, str]:\n \"\"\"Validate the item name is found in the repository directory.\"\"\"\n item_name_list = []\n for root, _dirs, files in os.walk(self.repository_directory):\n directory = Path(root)\n # valid item directory with .platform file within\n if \".platform\" in files:\n item_metadata_path = Path(directory, \".platform\")\n with Path.open(item_metadata_path, encoding=\"utf-8\") as file:\n item_metadata = json.load(file)\n # Ensure required metadata fields are present\n if item_metadata and \"type\" in item_metadata[\"metadata\"] and \"displayName\" in item_metadata[\"metadata\"]:\n item_name = item_metadata[\"metadata\"][\"displayName\"]\n item_name_list.append(item_name)\n\n # Check if item name is valid\n if input_name not in item_name_list:\n return False, constants.PARAMETER_MSGS[\"invalid item name\"].format(input_name)\n\n return True, \"Valid item name\"\n\n def _validate_file_path(self, input_path: list[str]) -> tuple[bool, str]:\n \"\"\"Validate that the file paths exist within the repository directory.\"\"\"\n # Convert input path to Path objects, returned as a list of valid paths\n valid_paths = process_input_path(self.repository_directory, input_path, validation_flag=True)\n\n # If list of paths is empty, all paths were invalid\n if not valid_paths:\n return False, constants.PARAMETER_MSGS[\"no valid file path\"].format(input_path)\n\n # Check for some invalid paths\n path_diff = len(input_path) - len(valid_paths)\n if path_diff > 0:\n return False, constants.PARAMETER_MSGS[\"invalid file path\"].format(input_path, path_diff)\n\n return True, \"Valid file path\"\n\n\nclass _DuplicateKeyLoader(yaml.SafeLoader):\n \"\"\"Custom YAML loader that raises an error on duplicate keys.\"\"\"\n\n pass\n\n\ndef _collect_duplicate_key_errors(root_node: yaml.MappingNode, loader: _DuplicateKeyLoader) -> list[str]:\n \"\"\"Collect duplicate key errors from all mapping nodes using iterative traversal.\"\"\"\n errors: list[str] = []\n stack = [root_node]\n\n while stack:\n node = stack.pop()\n # Group original key names by their lowercase form\n seen_keys: dict[str, list[str]] = {}\n\n for key_node, value_node in node.value:\n key = loader.construct_object(key_node)\n # Case-insensitive comparison (e.g., \"PROD\" and \"prod\" are duplicates)\n normalized_key = key.lower() if isinstance(key, str) else key\n if normalized_key not in seen_keys:\n seen_keys[normalized_key] = []\n seen_keys[normalized_key].append(str(key))\n\n # Queue child mappings for checking at their own level\n if isinstance(value_node, yaml.MappingNode):\n stack.append(value_node)\n elif isinstance(value_node, yaml.SequenceNode):\n for item_node in value_node.value:\n if isinstance(item_node, yaml.MappingNode):\n stack.append(item_node)\n\n for _, variants in seen_keys.items():\n if len(variants) > 1:\n unique_variants = set(variants)\n count = len(variants)\n # Show key once if all cases match, otherwise show all variants\n if len(unique_variants) == 1:\n errors.append(f\"'{next(iter(unique_variants))}' ({count})\")\n else:\n errors.append(f\"{sorted(unique_variants)} ({count})\")\n\n return errors\n\n\ndef _check_duplicate_keys_constructor(loader: _DuplicateKeyLoader, node: yaml.MappingNode) -> dict:\n \"\"\"Constructor that checks for duplicate keys in YAML mappings.\"\"\"\n # Run full traversal only from the root node; inner nodes skip to avoid redundant checks\n if not getattr(loader, \"_root_checked\", False):\n loader._root_checked = True\n errors = _collect_duplicate_key_errors(node, loader)\n\n if errors:\n dupe_details = \", \".join(errors)\n raise yaml.YAMLError(constants.PARAMETER_MSGS[\"duplicate key\"].format(dupe_details))\n\n return loader.construct_mapping(node)\n\n\n_DuplicateKeyLoader.add_constructor(yaml.resolver.BaseResolver.DEFAULT_MAPPING_TAG, _check_duplicate_keys_constructor)\n" }, { "path": "src/fabric_cicd/_parameter/_utils.py", "content": "# Copyright (c) Microsoft Corporation.\n# Licensed under the MIT License.\n\n\"\"\"\nFollowing functions are parameter utilities used by the FabricWorkspace and Parameter classes,\nand for debugging the parameter file. The utilities include validating the parameter.yml file, determining\nparameter dictionary structure, processing parameter values, and handling parameter value replacements.\n\"\"\"\n\nimport glob\nimport json\nimport logging\nimport os\nimport re\nimport urllib.parse\nfrom pathlib import Path\nfrom typing import Optional, Union\n\nimport yaml\nfrom jsonpath_ng.ext import parse\n\nimport fabric_cicd.constants as constants\nfrom fabric_cicd import FabricWorkspace\nfrom fabric_cicd._common._exceptions import InputError, ParsingError\nfrom fabric_cicd.constants import ItemType\n\nlogger = logging.getLogger(__name__)\n\n\"\"\"Functions to extract parameter values\"\"\"\n\n\ndef _validate_regex_structure(pattern: re.Pattern, find_value: str) -> None:\n \"\"\"\n Validates regex pattern structure to ensure it has exactly one capturing group.\n This validation is performed independently of whether the pattern matches any content.\n\n Args:\n pattern: Compiled regex pattern\n find_value: The regex pattern string for error messages\n\n Raises:\n InputError: If the regex doesn't have exactly one capturing group\n \"\"\"\n # Check the number of capturing groups in the pattern\n if pattern.groups != 1:\n msg = f\"Regex pattern '{find_value}' must contain exactly one capturing group.\"\n raise InputError(msg, logger)\n\n\ndef _validate_regex_pattern(matches: list, find_value: str) -> None:\n \"\"\"\n Validates regex pattern matches to ensure the captured value is not empty.\n\n Args:\n matches: List of regex match objects\n find_value: The regex pattern string for error messages\n\n Raises:\n InputError: If validation fails\n \"\"\"\n if matches:\n # Check if the captured group is empty (which would be invalid)\n captured_value = matches[0].group(1)\n if not captured_value:\n msg = f\"Regex pattern '{find_value}' captured an empty value.\"\n raise InputError(msg, logger)\n\n\ndef extract_find_value(param_dict: dict, file_content: str, filter_match: bool) -> dict:\n \"\"\"\n Extracts the find_value and sets the value. Processes the find_value if a valid regex is provided.\n Returns replacement information for use with re.sub() or string replace().\n\n Args:\n param_dict: The parameter dictionary containing the find_value and is_regex keys.\n file_content: The content of the file where the find_value will be searched.\n filter_match: A boolean to check for a regex match in filtered files only.\n\n Returns:\n Dictionary with keys:\n - 'pattern': The find pattern (original string or regex pattern)\n - 'is_regex': Whether this is a regex pattern\n - 'has_matches': Whether any matches were found\n \"\"\"\n find_value = param_dict.get(\"find_value\")\n is_regex = param_dict.get(\"is_regex\", \"\").lower() == \"true\"\n\n # No find value -> nothing to do\n if not find_value:\n return {\"pattern\": \"\", \"is_regex\": False, \"has_matches\": False}\n\n # Regex find_value\n if is_regex:\n try:\n compiled = re.compile(find_value)\n except re.error as re_err:\n msg = f\"Invalid regex '{find_value}': {re_err}\"\n raise InputError(msg, logger) from re_err\n\n # Validate structure (to catch bad patterns early)\n _validate_regex_structure(compiled, find_value)\n\n # If file excluded by filters, do not search — return no-match but keep validation\n if not filter_match:\n return {\"pattern\": find_value, \"is_regex\": True, \"has_matches\": False}\n\n matches = list(re.finditer(compiled, file_content))\n _validate_regex_pattern(matches, find_value)\n\n return {\"pattern\": find_value, \"is_regex\": True, \"has_matches\": bool(matches)}\n\n # Non-regex find_value\n if not filter_match:\n return {\"pattern\": find_value, \"is_regex\": False, \"has_matches\": False}\n\n return {\"pattern\": find_value, \"is_regex\": False, \"has_matches\": find_value in file_content}\n\n\ndef extract_replace_value(workspace_obj: FabricWorkspace, replace_value: str, get_dataflow_name: bool = False) -> str:\n \"\"\"Extracts the replace_value and sets the value. Processes the replace_value if a valid variable is provided.\"\"\"\n if not replace_value.startswith(\"$\"):\n if get_dataflow_name:\n logger.debug(\n \"Can't get dataflow name as the replace_value was set to a regular string, not the items variable\"\n )\n return None\n return replace_value\n\n # If $workspace variable, return the workspace ID value\n if replace_value.startswith(\"$workspace.\"):\n if get_dataflow_name:\n msg = \"Invalid replace_value variable: '$workspace'. Expected format to get dataflow name: $items.type.name.$attribute\"\n raise InputError(msg, logger)\n\n return _extract_workspace_id(workspace_obj, replace_value)\n\n # If $items variable, return the item attribute value if found\n if replace_value.startswith(\"$items.\"):\n return _extract_item_attribute(workspace_obj, replace_value, get_dataflow_name)\n\n # Otherwise, raise an error for invalid variable syntax\n msg = f\"Invalid replace_value variable format: '{replace_value}'. Expected format: $items.type.name.$attribute or $workspace.$id or $workspace.$name or $workspace.$name_encoded or $workspace..$id or $workspace..$items...$attribute\"\n raise InputError(msg, logger)\n\n\ndef _extract_workspace_id(workspace_obj: FabricWorkspace, replace_value: str) -> str:\n \"\"\"\n Extracts workspace ID or display name from the $workspace variable to set as the replace_value.\n\n Supports the following formats:\n - $workspace.id or $workspace.$id - Returns the target workspace ID\n - $workspace.$name - Returns the target workspace display name\n - $workspace.$name_encoded - Returns the target workspace display name, URL-encoded\n - $workspace..$items...$ - Resolves an item attribute from the specified workspace,\n where $attribute is any supported attribute in constants.ITEM_ATTR_LOOKUP\n - $workspace. or $workspace..$id - Resolves the workspace ID from the name\n \"\"\"\n # Case 1: $workspace.$id ($workspace.id supported for backward compatibility)\n if replace_value == \"$workspace.id\" or replace_value == \"$workspace.$id\":\n return workspace_obj.workspace_id\n\n try:\n # Case 2: $workspace.$name\n if replace_value == \"$workspace.$name\":\n return workspace_obj._resolve_workspace_name()\n\n # Case 3: $workspace.$name_encoded - URL-encoded display name\n if replace_value == \"$workspace.$name_encoded\":\n name = workspace_obj._resolve_workspace_name()\n return urllib.parse.quote(name, safe=\"\")\n\n # Extract the variable string without the prefix\n var_string = replace_value.removeprefix(\"$workspace.\")\n\n # Case 4: Check if this is a cross-workspace item reference\n if \"$items.\" in var_string:\n # Check if the variable ends with a valid attribute\n valid_attribute = False\n attribute = None\n\n for attr in constants.ITEM_ATTR_LOOKUP:\n if var_string.endswith(f\".${attr}\"):\n valid_attribute = True\n attribute = attr\n break\n\n if not valid_attribute:\n msg = f\"Invalid syntax or missing attribute in cross-workspace variable '{replace_value}'. Expected format: $workspace.name.$items.type.name.$attribute where attribute is one of: {', '.join(constants.ITEM_ATTR_LOOKUP)}\"\n raise ParsingError(msg, logger)\n\n # Split on the $items prefix to get workspace name\n workspace_part, items_part = var_string.split(\".$items.\", 1)\n workspace_name = workspace_part.strip()\n logger.debug(f\"Extracted workspace name: {workspace_name}\")\n\n # Get workspace ID\n workspace_id = workspace_obj._resolve_workspace_id(workspace_name)\n\n # Remove the trailing .$attribute to get the item info\n items_info = items_part.removesuffix(f\".${attribute}\")\n\n # Find the last period to separate item type from item name\n last_period_pos = items_info.rfind(\".\")\n if last_period_pos == -1:\n msg = f\"Invalid $workspace variable syntax: {replace_value}. Expected format: $workspace.name.$items.type.name.$attribute\"\n raise ParsingError(msg, logger)\n\n # Extract item_type and item_name\n item_type = items_info[:last_period_pos].strip()\n item_name = items_info[last_period_pos + 1 :].strip()\n\n logger.debug(f\"Extracted item type: {item_type}, item name: {item_name}, attribute: {attribute}\")\n\n if item_type not in constants.ACCEPTED_ITEM_TYPES:\n msg = f\"Item type '{item_type}' is invalid or not supported\"\n raise ParsingError(msg, logger)\n\n # Look up the attribute value of the item in the specified workspace\n attribute_value = workspace_obj._lookup_item_attribute(workspace_id, item_type, item_name, attribute)\n logger.debug(f\"Found item {attribute}: {attribute_value}\")\n return attribute_value\n\n # Case 5: Pattern: $workspace..$id (explicit) or $workspace. (backward-compatible)\n workspace_name = var_string.removesuffix(\".$id\").strip()\n logger.debug(f\"Extracted workspace name: {workspace_name}\")\n\n # Resolve workspace ID\n return workspace_obj._resolve_workspace_id(workspace_name)\n\n except Exception as e:\n # Re-raise exceptions\n if isinstance(e, (ParsingError, InputError)):\n raise e\n\n # Otherwise, wrap it in a ParsingError\n msg = f\"Error parsing $workspace variable: {e}\"\n raise ParsingError(msg, logger) from e\n\n\ndef _extract_item_attribute(workspace_obj: FabricWorkspace, variable: str, get_dataflow_name: bool) -> str:\n \"\"\"\n Extracts the item attribute value from the $items variable to set as the replace_value.\n\n Args:\n workspace_obj: The FabricWorkspace object containing the workspace items dictionary used to access item metadata.\n variable: The $items variable string to be parsed and processed.\n Supports the following formats:\n - $items.type.name.attribute (legacy format)\n - $items.type.name.$attribute (new format)\n Supported attributes: id, sqlendpoint, queryserviceuri\n get_dataflow_name: A boolean flag to indicate if the dataflow item name should be returned instead of the attribute value.\n \"\"\"\n error = None\n try:\n var_string = variable.removeprefix(\"$items.\")\n\n # Check for new pattern with $attribute\n if \".$\" in var_string:\n # Split on the $attribute marker\n name_part, attr_part = var_string.rsplit(\".$\", 1)\n\n # Extract attribute name\n attribute = attr_part.strip()\n\n # Find the last period to separate item_type from item_name\n last_period_pos = name_part.rfind(\".\")\n if last_period_pos == -1:\n msg = f\"Invalid $items variable syntax: {variable}. Expected format: $items.type.name.$attribute\"\n error = ParsingError(msg, logger)\n return None\n\n # Extract item_type and item_name\n item_type = name_part[:last_period_pos].strip()\n item_name = name_part[last_period_pos + 1 :].strip()\n\n # Backward compatibility for legacy pattern\n else:\n msg = f\"Invalid $items variable syntax: {variable}. Expected format: $items.type.name.attribute\"\n # Split the string to get item_type (first part)\n parts = var_string.split(\".\", 1)\n if len(parts) < 2:\n error = ParsingError(msg, logger)\n return None\n\n item_type = parts[0].strip()\n\n # Get the attribute (last part)\n if \".\" not in parts[1]:\n error = ParsingError(msg, logger)\n return None\n\n # Find the position of the last period which separates item_name from attribute\n last_period_pos = parts[1].rfind(\".\")\n if last_period_pos == -1:\n error = ParsingError(msg, logger)\n return None\n\n # Extract item_name and attribute\n item_name = parts[1][:last_period_pos].strip()\n attribute = parts[1][last_period_pos + 1 :].strip()\n\n # Validate attribute before further processing\n attr_name = attribute.lower()\n if attr_name not in constants.ITEM_ATTR_LOOKUP:\n msg = f\"Attribute '{attribute}' is invalid. Supported attributes: {list(constants.ITEM_ATTR_LOOKUP)}\"\n error = ParsingError(msg, logger)\n return None\n\n logger.debug(\n f\"Processing $items variable with item_type={item_type}, item_name={item_name}, attribute={attribute}\"\n )\n\n # Refresh the workspace items to get the latest deployed items\n workspace_obj._refresh_deployed_items()\n\n # Validate item type exists in the deployed workspace\n if item_type not in workspace_obj.workspace_items and not get_dataflow_name:\n msg = f\"Item type '{item_type}' is invalid or not found in deployed items\"\n error = ParsingError(msg, logger)\n return None\n\n # Check if the specific item is deployed\n if item_name not in workspace_obj.workspace_items.get(item_type, {}) and not get_dataflow_name:\n msg = f\"Item '{item_name}' not found as a deployed {item_type}\"\n error = ParsingError(msg, logger)\n return None\n\n # Special case: set to True in the context of a Dataflow that references another Dataflow\n if get_dataflow_name:\n if (\n item_type in workspace_obj.repository_items\n and item_type == ItemType.DATAFLOW.value\n and item_name in workspace_obj.repository_items[item_type]\n and attribute == \"id\"\n ):\n logger.debug(\"Source Dataflow reference will be replaced separately\")\n return item_name\n # Return None for non-existent item\n return None\n\n # Get the item's attributes from workspace items\n item_attr_values = workspace_obj.workspace_items[item_type][item_name]\n\n # Get the attribute value and check if it exists\n attr_value = item_attr_values.get(attr_name)\n if not attr_value:\n msg = f\"Value does not exist for attribute '{attribute}' in the {item_type} item '{item_name}'\"\n error = ParsingError(msg, logger)\n return None\n\n logger.debug(f\"Found attribute '{attr_name}' with value '{attr_value}'\")\n return attr_value\n\n except Exception as e:\n # If it's not a ParsingError, create a new one\n if not isinstance(e, ParsingError):\n error = ParsingError(f\"Error parsing $items variable: {e!s}\", logger)\n error = e\n return None\n\n finally:\n # Raise error at the very end (only once)\n if error is not None:\n raise error\n\n\ndef extract_parameter_filters(workspace_obj: FabricWorkspace, param_dict: dict) -> tuple[str, str, Path]:\n \"\"\"Extracts the item type, name, and path filters from the parameter dictionary, if present.\"\"\"\n item_type = param_dict.get(\"item_type\")\n item_name = param_dict.get(\"item_name\")\n file_path = process_input_path(workspace_obj.repository_directory, param_dict.get(\"file_path\"))\n\n return item_type, item_name, file_path\n\n\ndef process_environment_key(environment: str, replace_value_dict: dict) -> dict:\n \"\"\"Processes the replace_value dictionary to replace the '_ALL_' environment key with the target environment when present.\"\"\"\n # If there's only one key, check if it's \"_ALL_\" (case insensitive) and replace it\n # Note: When other env keys are present with _ALL_, upstream parameter validation fails\n if len(replace_value_dict) == 1:\n key = next(iter(replace_value_dict))\n if key.lower() == \"_all_\":\n replace_value_dict[environment] = replace_value_dict.pop(key)\n\n return replace_value_dict\n\n\n\"\"\"Functions to replace key values in JSON or YAML\"\"\"\n\n\ndef replace_key_value(\n workspace_obj: FabricWorkspace, param_dict: dict, content: str, env: str, is_yaml: bool = False\n) -> str:\n \"\"\"A function to replace key values in a JSON or YAML using parameterization. It uses jsonpath_ng to find and replace values in the JSON.\n\n Args:\n workspace_obj: The FabricWorkspace object.\n param_dict: The parameter dictionary.\n content: the JSON/YAML content to be modified.\n env: The environment variable to be used for replacement.\n is_yaml: A boolean indicating if the content is YAML (default is False for JSON).\n \"\"\"\n # Parse content to a dictionary based on format (YAML or JSON)\n if is_yaml:\n try:\n data = yaml.safe_load(content)\n except yaml.YAMLError as ye:\n raise ValueError(ye) from ye\n\n # Handle empty YAML files\n if data is None:\n return content\n else:\n try:\n data = json.loads(content)\n except json.JSONDecodeError as jde:\n raise ValueError(jde) from jde\n\n # Extract the jsonpath expression from the find_key attribute of the param_dict\n jsonpath_expr = parse(param_dict[\"find_key\"])\n replace_value_dict = process_environment_key(workspace_obj.environment, param_dict[\"replace_value\"])\n for match in jsonpath_expr.find(data):\n # If the env is present in the replace_value array perform the replacement\n if env in replace_value_dict:\n try:\n # Process the replace value to handle $items notation\n processed_value = replace_value_dict[env]\n if isinstance(processed_value, str):\n processed_value = extract_replace_value(workspace_obj, processed_value)\n match.full_path.update(data, processed_value)\n logger.debug(\n f\"Value: {match.value} found at path: {match.full_path} to be replaced with value: {processed_value}\"\n )\n except Exception as match_e:\n raise ValueError(match_e) from match_e\n\n return yaml.dump(data, default_flow_style=False, allow_unicode=True) if is_yaml else json.dumps(data)\n\n\ndef replace_variables_in_parameter_file(raw_file: str) -> str:\n \"\"\"\n A function to replace tokens in the parameter.yml file with environment variables.\n\n Args:\n raw_file: The parameter.yml file content as a string.\n \"\"\"\n if \"enable_environment_variable_replacement\" in constants.FEATURE_FLAG:\n # filter os.environ dict to only allow variables that begin with $ENV:\n env_vars = {k[len(\"$ENV:\") :]: v for k, v in os.environ.items() if k.startswith(\"$ENV:\")}\n # block of code to support both variants of the parameters.yml file\n\n # Perform replacements\n for var_name, var_value in env_vars.items():\n placeholder = f\"$ENV:{var_name}\"\n if placeholder in raw_file:\n raw_file = raw_file.replace(placeholder, var_value)\n logger.debug(f\"Replaced {placeholder} with {var_value}\")\n\n return raw_file\n return raw_file\n\n\n\"\"\"Functions to validate the parameter file\"\"\"\n\n\ndef validate_parameter_file(\n repository_directory: str,\n item_type_in_scope: Optional[list] = None,\n environment: str = \"N/A\",\n parameter_file_name: str = \"parameter.yml\",\n parameter_file_path: Optional[str] = None,\n) -> bool:\n \"\"\"\n A wrapper function that validates a parameter.yml file, using\n the Parameter class.\n\n Args:\n repository_directory: The directory containing the items and parameter.yml file.\n item_type_in_scope: A list of item types to validate. If omitted, defaults to all supported item types.\n environment: The target environment.\n parameter_file_name: The name of the parameter file, default is \"parameter.yml\".\n parameter_file_path: The path to the parameter file, if different from the default.\n \"\"\"\n from fabric_cicd._common._validate_input import (\n validate_environment,\n validate_item_type_in_scope,\n validate_repository_directory,\n )\n\n # Import the Parameter class here to avoid circular imports\n from fabric_cicd._parameter._parameter import Parameter\n\n # Initialize the Parameter object with the validated inputs\n parameter_obj = Parameter(\n repository_directory=validate_repository_directory(repository_directory),\n item_type_in_scope=validate_item_type_in_scope(item_type_in_scope),\n environment=validate_environment(environment),\n parameter_file_name=parameter_file_name,\n parameter_file_path=parameter_file_path,\n )\n # Validate with _validate_parameter_file() method\n return parameter_obj._validate_parameter_file()\n\n\ndef is_valid_structure(param_dict: dict, param_name: Optional[str] = None) -> bool:\n \"\"\"\n Checks the parameter dictionary structure and determines if it\n contains the valid structure (i.e. a list of values when indexed by the key,\n or the new dict format for semantic_model_binding).\n\n Args:\n param_dict: The parameter dictionary to check.\n param_name: The name of the parameter to check, if specified.\n \"\"\"\n # Check the structure of the specified parameter\n if param_name:\n # Special case for semantic_model_binding - can be list (legacy) or dict (new)\n if param_name == \"semantic_model_binding\":\n is_valid, _ = _check_semantic_model_binding_structure(param_dict.get(param_name))\n return is_valid\n return _check_parameter_structure(param_dict.get(param_name))\n\n # Get only parameters that exist in param_dict\n existing_params = [name for name in constants.PARAM_NAMES if name in param_dict]\n\n # If no parameters found, return False\n if not existing_params:\n return False\n\n # Check all existing parameters have valid structure\n for name in existing_params:\n # Special case for semantic_model_binding\n if name == \"semantic_model_binding\":\n is_valid, _ = _check_semantic_model_binding_structure(param_dict.get(name))\n if not is_valid:\n return False\n elif not _check_parameter_structure(param_dict.get(name)):\n return False\n\n return True\n\n\ndef _check_parameter_structure(param_value: any) -> bool:\n \"\"\"Checks the structure of a parameter value\"\"\"\n return isinstance(param_value, list)\n\n\ndef _check_semantic_model_binding_structure(param_value: any) -> tuple[bool, bool]:\n \"\"\"\n Checks the structure of semantic_model_binding parameter value.\n Supports both legacy (list) and new (dict with 'default' or 'models') formats.\n \"\"\"\n # Legacy format: list\n if isinstance(param_value, list):\n return (True, False)\n\n # New format: dict with at least 'default' or 'models'\n if isinstance(param_value, dict):\n has_default = \"default\" in param_value\n has_models = \"models\" in param_value\n is_valid = has_default or has_models\n return (is_valid, True)\n\n return (False, False)\n\n\n\"\"\"Functions to process and validate file paths from the optional filter\"\"\"\n\n\ndef process_input_path(\n repository_directory: Path, input_path: Union[str, list[str], None], validation_flag: bool = False\n) -> Union[list[Path], None]:\n \"\"\"\n Processes the input_path value according to its type. Supports both\n regular paths and wildcard paths, including mixed lists.\n\n Args:\n repository_directory: The directory of the repository.\n input_path: The input path value to process (None, a string, or list of strings).\n validation_flag: Flag to indicate the context of the function call to set the logging type.\n \"\"\"\n # Set the logging function based on validation_flag\n log_func = logger.error if validation_flag else logger.debug\n\n # Return None for None or empty input\n if not input_path:\n return None\n\n # Use a set to avoid duplicate paths\n valid_paths = set()\n\n # Standardize to list for consistent processing\n paths_to_process = [input_path] if isinstance(input_path, str) else input_path\n\n for path in paths_to_process:\n # Process path based on whether it contains wildcard characters\n has_wildcard = False\n try:\n has_wildcard = glob.has_magic(path)\n except Exception as e:\n log_func(f\"Error checking for wildcard in path '{path}': {e}\")\n continue\n\n if has_wildcard:\n _process_wildcard_path(path, repository_directory, valid_paths, log_func)\n else:\n _process_regular_path(path, repository_directory, valid_paths, log_func)\n\n return list(valid_paths)\n\n\ndef _process_regular_path(\n path: str, repository_directory: Path, valid_paths: set[Path], log_func: logging.Logger\n) -> None:\n \"\"\"Process a regular (non-wildcard) path and add to valid_paths if valid.\"\"\"\n # Normalize path for consistent handling\n normalized_path = Path(path.lstrip(\"/\\\\\"))\n\n # Set the path type based on whether it is absolute or relative\n path_type = \"Relative\" if not normalized_path.is_absolute() else \"Absolute\"\n\n # Validate the path and add to set if valid\n valid_path = _resolve_file_path(normalized_path, repository_directory, path_type, log_func)\n if valid_path:\n valid_paths.add(valid_path)\n\n\ndef _process_wildcard_path(\n path: str, repository_directory: Path, valid_paths: set[Path], log_func: logging.Logger\n) -> None:\n \"\"\"Process a wildcard path and add matching files to valid_paths if valid.\"\"\"\n search_pattern = _set_wildcard_path_pattern(path, repository_directory, log_func)\n\n if not search_pattern:\n return\n\n # Track if matches are found\n initial_paths_count = len(valid_paths)\n\n # Get all matching files that exist\n try:\n for matched_path in [p for p in repository_directory.glob(search_pattern) if p.is_file()]:\n # Validate path and add to set if valid\n valid_path = _resolve_file_path(matched_path, repository_directory, \"Wildcard\", log_func)\n if valid_path:\n valid_paths.add(valid_path)\n\n # Only log if matches were not found\n if len(valid_paths) == initial_paths_count:\n log_func(f\"Wildcard path '{path}' did not match any files\")\n\n except Exception as e:\n log_func(f\"Error processing wildcard pattern '{search_pattern}': {e}\")\n\n\ndef _set_wildcard_path_pattern(wildcard_path: str, repository_directory: Path, log_func: logging.Logger) -> str:\n \"\"\"Determine the glob search pattern for a wildcard path.\"\"\"\n normalized_wildcard_path = wildcard_path.replace(\"\\\\\", \"/\")\n\n # Step 1: Validate wildcard pattern syntax\n if not _validate_wildcard_syntax(normalized_wildcard_path, log_func):\n return \"\"\n\n # Step 2: Determine search pattern based on path type\n if normalized_wildcard_path.startswith(\"**/\"):\n logger.debug(\"Recursive wildcard path detected\")\n return f\"**/{normalized_wildcard_path[3:]}\"\n\n if Path(normalized_wildcard_path).is_absolute():\n logger.debug(\"Absolute wildcard path detected\")\n try:\n # Check if the path is within the repository\n rel_path = Path(normalized_wildcard_path).relative_to(repository_directory)\n return str(rel_path)\n except ValueError:\n log_func(f\"Invalid absolute wildcard path. '{wildcard_path}' is outside the repository directory\")\n return \"\"\n else:\n logger.debug(\"Non-recursive and non-absolute wildcard path detected\")\n return normalized_wildcard_path\n\n\ndef _resolve_file_path(\n input_path: Path, repository_directory: Path, path_type: str, log_func: logging.Logger\n) -> Optional[Path]:\n \"\"\"\n Validates that a path exists, is a file, and is within the repository directory.\n Returns the resolved absolute path if valid, None otherwise.\n \"\"\"\n try:\n # Step 1: Resolve the input path based on its type\n if path_type == \"Relative\":\n resolved_path = (repository_directory / input_path).resolve()\n logger.debug(f\"{path_type} path '{input_path}' resolved as '{resolved_path}'\")\n elif path_type == \"Absolute\":\n resolved_path = input_path.resolve()\n else:\n resolved_path = input_path\n\n # Step 2: Check if the path is within the repository directory\n try:\n _ = resolved_path.relative_to(repository_directory)\n except ValueError:\n log_func(f\"{path_type} path '{input_path}' is outside the repository directory\")\n return None\n\n # Step 3: For non-wildcard paths, check existence and file type\n if path_type != \"Wildcard\":\n # Check path existence\n if not resolved_path.exists():\n log_func(f\"{path_type} path '{input_path}' does not exist\")\n return None\n\n # Check file validation\n if not resolved_path.is_file():\n log_func(f\"{path_type} path '{input_path}' is not a file\")\n return None\n\n logger.debug(f\"Path '{resolved_path}' is valid and within the repository directory\")\n return resolved_path\n\n except Exception as e:\n log_func(f\"Error validating {path_type.lower()} path '{input_path}': {e}\")\n return None\n\n\ndef _validate_wildcard_syntax(pattern: str, log_func: logging.Logger) -> bool:\n \"\"\"Validates wildcard pattern syntax before using glob.\"\"\"\n # Check for empty or whitespace-only patterns\n if not pattern or pattern.isspace():\n log_func(\"Wildcard pattern is empty\")\n return False\n\n # Check for problematic absolute paths with recursive patterns\n if pattern.startswith(\"/\") and pattern[1:].startswith(\"**/\"):\n log_func(f\"Absolute path with recursive pattern is not allowed: '{pattern}'\")\n return False\n\n # Handle Windows-style absolute paths with recursive patterns\n if re.match(r\"^[a-zA-Z]:\\\\\", pattern) and \"**\\\\\" in pattern:\n log_func(f\"Absolute path with recursive pattern is not allowed: '{pattern}'\")\n return False\n\n # Apply standard validations from constants\n for validation in constants.WILDCARD_PATH_VALIDATIONS:\n if validation[\"check\"](pattern):\n log_func(validation[\"message\"](pattern))\n return False\n\n # Validate proper nesting of brackets and braces\n if not _validate_nested_brackets_braces(pattern, log_func):\n return False\n\n # Validate character classes (bracket expressions)\n for section in re.findall(r\"\\[(.*?)\\]\", pattern):\n if not section or section.startswith(\"]\") or section.startswith(\"-\") or \"--\" in section:\n log_func(f\"Invalid character class in pattern: '{pattern}'\")\n return False\n\n # Validate brace expansions\n try:\n for section in re.findall(r\"\\{(.*?)\\}\", pattern):\n if (\n not section # Empty braces\n or \",\" not in section # No comma separator\n or section.startswith(\",\") # Starts with comma\n or section.endswith(\",\") # Ends with comma\n or \",,\" in section\n ): # Adjacent commas\n log_func(f\"Invalid brace expansion in pattern: '{pattern}'\")\n return False\n\n except Exception as e:\n log_func(f\"Error validating brace content in pattern '{pattern}': {e}\")\n return False\n\n # Check for path traversal sequences\n pattern_lower = pattern.lower()\n traversal_patterns = [\n \"../\",\n \"..\" + os.sep,\n \"..\" + os.altsep if os.altsep else \"\",\n \"..%2F\",\n \"..%5C\",\n \"..%2f\",\n \"..%5c\",\n ]\n\n for traversal in traversal_patterns:\n if traversal and traversal in pattern_lower:\n log_func(f\"Path traversal sequences not allowed: '{pattern}'\")\n return False\n\n return True\n\n\ndef _validate_nested_brackets_braces(pattern: str, log_func: logging.Logger) -> bool:\n \"\"\"Validates proper nesting of brackets and braces in a wildcard pattern.\"\"\"\n stack = []\n\n for pos, char in enumerate(pattern):\n if char in \"[{\":\n stack.append(char)\n elif char in \"]}\":\n # Check if stack is empty (closing without opening)\n if not stack:\n log_func(f\"Unmatched closing '{char}' at position {pos} in pattern: '{pattern}'\")\n return False\n\n # Check for proper matching\n last_open = stack.pop()\n if (char == \"]\" and last_open != \"[\") or (char == \"}\" and last_open != \"{\"):\n log_func(f\"Mismatched bracket/brace pair '{last_open}{char}' at position {pos} in pattern: '{pattern}'\")\n return False\n\n # Check if all brackets and braces were closed\n if stack:\n log_func(f\"Unclosed bracket(s) or brace(s) in pattern: '{pattern}'\")\n return False\n\n return True\n\n\n\"\"\"Functions to determine replacement based on optional filters\"\"\"\n\n\ndef check_replacement(\n input_type: Union[str, list[str], None],\n input_name: Union[str, list[str], None],\n input_path: Union[list[Path], None],\n item_type: str,\n item_name: str,\n file_path: Path,\n) -> bool:\n \"\"\"\n Determines whether a find and replace is applied or not based on the provided optional filters.\n\n Args:\n input_type: The input item_type value to check.\n input_name: The input item_name value to check.\n input_path: The input file_path value to check.\n item_type: The item_type value to compare with.\n item_name: The item_name value to compare with.\n file_path: The file_path value to compare with.\n \"\"\"\n # No optional parameters found\n if input_type is None and input_name is None and input_path is None:\n logger.debug(\"No optional filters found. Find and replace applied in this repository file\")\n return True\n\n # Otherwise, find matches for the optional parameters\n item_type_match = _find_match(input_type, item_type)\n item_name_match = _find_match(input_name, item_name)\n file_path_match = _find_match(input_path, file_path)\n\n if item_type_match and item_name_match and file_path_match:\n if input_type:\n logger.debug(f\"Item type match found: {item_type_match}\")\n if input_name:\n logger.debug(f\"Item name match found: {item_name_match}\")\n if input_path:\n logger.debug(f\"File path match found: {file_path_match}\")\n\n # Optional filters match found. Find and replace applied in this repository file\n return True\n\n # Optional filters match not found. Find and replace skipped for this repository file\n return False\n\n\ndef _find_match(\n param_value: Union[str, list, None],\n compare_value: Union[str, Path],\n) -> bool:\n \"\"\"\n Checks for a match between the parameter value and\n the compare value based on parameter value type.\n\n Args:\n param_value: The parameter value to compare (can be a string, list, Path, or None type).\n compare_value: The value to compare with.\n \"\"\"\n # If no parameter value, checking for matches is not required\n if param_value is None:\n return True\n\n # Otherwise, check for matches based on the parameter value type\n if isinstance(param_value, list):\n match_condition = any(compare_value == value for value in param_value)\n elif isinstance(param_value, (str, Path)):\n match_condition = compare_value == param_value\n else:\n match_condition = False\n\n return match_condition\n" }, { "path": "src/fabric_cicd/constants.py", "content": "# Copyright (c) Microsoft Corporation.\n# Licensed under the MIT License.\n\n\"\"\"Constants for the fabric-cicd package.\"\"\"\n\nimport os\nfrom enum import Enum\n\nfrom fabric_cicd._common._validate_env_vars import VALID_GUID_REGEX as VALID_GUID_REGEX\nfrom fabric_cicd._common._validate_env_vars import validate_env_var_api_url\n\n# General\nVERSION = \"1.0.0\"\nDEFAULT_GUID = \"00000000-0000-0000-0000-000000000000\"\nFEATURE_FLAG = set()\nUSER_AGENT = f\"ms-fabric-cicd/{VERSION}\"\nVALID_ENABLE_FLAGS = [\"1\", \"true\", \"yes\"]\n\n\nclass EnvVar(str, Enum):\n \"\"\"Enumeration of environment variables used by fabric-cicd.\"\"\"\n\n HTTP_TRACE_ENABLED = \"FABRIC_CICD_HTTP_TRACE_ENABLED\"\n \"\"\"Set to '1', 'true', or 'yes' to enable HTTP request/response tracing.\"\"\"\n HTTP_TRACE_FILE = \"FABRIC_CICD_HTTP_TRACE_FILE\"\n \"\"\"Path to save HTTP trace output. Only used if HTTP tracing is enabled.\"\"\"\n DEFAULT_API_ROOT_URL = \"DEFAULT_API_ROOT_URL\"\n \"\"\"Override the default Power BI API root URL. Defaults to 'https://api.powerbi.com'.\"\"\"\n FABRIC_API_ROOT_URL = \"FABRIC_API_ROOT_URL\"\n \"\"\"Override the Fabric API root URL. Defaults to 'https://api.fabric.microsoft.com'.\"\"\"\n RETRY_DELAY_OVERRIDE_SECONDS = \"FABRIC_CICD_RETRY_DELAY_OVERRIDE_SECONDS\"\n \"\"\"Override retry delay in seconds (e.g., '0' for instant retries - useful in tests).\"\"\"\n RETRY_AFTER_SECONDS = \"FABRIC_CICD_RETRY_AFTER_SECONDS\"\n \"\"\"Override retry-after delay for item name conflicts (HTTP 400). Defaults to 300 seconds.\"\"\"\n RETRY_BASE_DELAY_SECONDS = \"FABRIC_CICD_RETRY_BASE_DELAY_SECONDS\"\n \"\"\"Override base delay for item name conflict retries. Defaults to 30 seconds.\"\"\"\n RETRY_MAX_DURATION_SECONDS = \"FABRIC_CICD_RETRY_MAX_DURATION_SECONDS\"\n \"\"\"Override max duration for item name conflict retries. Defaults to 300 seconds.\"\"\"\n PARALLEL_MAX_WORKERS = \"FABRIC_CICD_PARALLEL_MAX_WORKERS\"\n \"\"\"Override max parallel workers for concurrent item publishing. Defaults to 8.\"\"\"\n\n\nclass ItemType(str, Enum):\n \"\"\"Enumeration of supported Microsoft Fabric item types.\"\"\"\n\n APACHE_AIRFLOW_JOB = \"ApacheAirflowJob\"\n COPY_JOB = \"CopyJob\"\n DATA_AGENT = \"DataAgent\"\n DATA_BUILD_TOOL_JOB = \"DataBuildToolJob\"\n DATA_PIPELINE = \"DataPipeline\"\n DATAFLOW = \"Dataflow\"\n ENVIRONMENT = \"Environment\"\n EVENTHOUSE = \"Eventhouse\"\n EVENTSTREAM = \"Eventstream\"\n GRAPHQL_API = \"GraphQLApi\"\n KQL_DASHBOARD = \"KQLDashboard\"\n KQL_DATABASE = \"KQLDatabase\"\n KQL_QUERYSET = \"KQLQueryset\"\n LAKEHOUSE = \"Lakehouse\"\n MIRRORED_DATABASE = \"MirroredDatabase\"\n ML_EXPERIMENT = \"MLExperiment\"\n MOUNTED_DATA_FACTORY = \"MountedDataFactory\"\n NOTEBOOK = \"Notebook\"\n ONTOLOGY = \"Ontology\"\n REFLEX = \"Reflex\"\n REPORT = \"Report\"\n SEMANTIC_MODEL = \"SemanticModel\"\n SPARK_JOB_DEFINITION = \"SparkJobDefinition\"\n SQL_DATABASE = \"SQLDatabase\"\n USER_DATA_FUNCTION = \"UserDataFunction\"\n VARIABLE_LIBRARY = \"VariableLibrary\"\n WAREHOUSE = \"Warehouse\"\n\n\n# Serial execution order for publishing items determines dependency order.\n# Unpublish order is the reverse of this.\nSERIAL_ITEM_PUBLISH_ORDER: dict[int, ItemType] = {\n 1: ItemType.VARIABLE_LIBRARY,\n 2: ItemType.WAREHOUSE,\n 3: ItemType.MIRRORED_DATABASE,\n 4: ItemType.LAKEHOUSE,\n 5: ItemType.SQL_DATABASE,\n 6: ItemType.ENVIRONMENT,\n 7: ItemType.USER_DATA_FUNCTION,\n 8: ItemType.EVENTHOUSE,\n 9: ItemType.SPARK_JOB_DEFINITION,\n 10: ItemType.NOTEBOOK,\n 11: ItemType.SEMANTIC_MODEL,\n 12: ItemType.REPORT,\n 13: ItemType.COPY_JOB,\n 14: ItemType.DATA_BUILD_TOOL_JOB,\n 15: ItemType.KQL_DATABASE,\n 16: ItemType.KQL_QUERYSET,\n 17: ItemType.REFLEX,\n 18: ItemType.EVENTSTREAM,\n 19: ItemType.KQL_DASHBOARD,\n 20: ItemType.DATAFLOW,\n 21: ItemType.DATA_PIPELINE,\n 22: ItemType.GRAPHQL_API,\n 23: ItemType.APACHE_AIRFLOW_JOB,\n 24: ItemType.MOUNTED_DATA_FACTORY,\n 25: ItemType.DATA_AGENT,\n 26: ItemType.ML_EXPERIMENT,\n 27: ItemType.ONTOLOGY,\n}\n\n\nclass FeatureFlag(str, Enum):\n \"\"\"Enumeration of supported feature flags for fabric-cicd.\"\"\"\n\n ENABLE_LAKEHOUSE_UNPUBLISH = \"enable_lakehouse_unpublish\"\n \"\"\"Set to enable the deletion of Lakehouses.\"\"\"\n ENABLE_WAREHOUSE_UNPUBLISH = \"enable_warehouse_unpublish\"\n \"\"\"Set to enable the deletion of Warehouses.\"\"\"\n ENABLE_SQLDATABASE_UNPUBLISH = \"enable_sqldatabase_unpublish\"\n \"\"\"Set to enable the deletion of SQL Databases.\"\"\"\n ENABLE_EVENTHOUSE_UNPUBLISH = \"enable_eventhouse_unpublish\"\n \"\"\"Set to enable the deletion of Eventhouses.\"\"\"\n ENABLE_KQLDATABASE_UNPUBLISH = \"enable_kqldatabase_unpublish\"\n \"\"\"Set to enable the deletion of KQL Databases (attached to Eventhouses).\"\"\"\n ENABLE_SHORTCUT_PUBLISH = \"enable_shortcut_publish\"\n \"\"\"Set to enable deploying shortcuts with the lakehouse.\"\"\"\n DISABLE_WORKSPACE_FOLDER_PUBLISH = \"disable_workspace_folder_publish\"\n \"\"\"Set to disable deploying workspace sub folders.\"\"\"\n CONTINUE_ON_SHORTCUT_FAILURE = \"continue_on_shortcut_failure\"\n \"\"\"Set to allow deployment to continue even when shortcuts fail to publish.\"\"\"\n ENABLE_ENVIRONMENT_VARIABLE_REPLACEMENT = \"enable_environment_variable_replacement\"\n \"\"\"Set to enable the use of pipeline variables.\"\"\"\n ENABLE_EXPERIMENTAL_FEATURES = \"enable_experimental_features\"\n \"\"\"Set to enable experimental features, such as selective deployments.\"\"\"\n ENABLE_ITEMS_TO_INCLUDE = \"enable_items_to_include\"\n \"\"\"Set to enable selective publishing/unpublishing of items.\"\"\"\n ENABLE_EXCLUDE_FOLDER = \"enable_exclude_folder\"\n \"\"\"Set to enable folder-based exclusion during publish operations.\"\"\"\n ENABLE_INCLUDE_FOLDER = \"enable_include_folder\"\n \"\"\"Set to enable folder-based inclusion during publish operations.\"\"\"\n ENABLE_SHORTCUT_EXCLUDE = \"enable_shortcut_exclude\"\n \"\"\"Set to enable selective publishing of shortcuts in a Lakehouse.\"\"\"\n ENABLE_RESPONSE_COLLECTION = \"enable_response_collection\"\n \"\"\"Set to enable collection of API responses during publish operations.\"\"\"\n ENABLE_HARD_DELETE = \"enable_hard_delete\"\n \"\"\"Set to enable hard deletion of items, bypassing the workspace recycle bin.\"\"\"\n\n\nclass OperationType(str, Enum):\n \"\"\"Enumeration of operation types for publish/unpublish workflows.\"\"\"\n\n PUBLISH = \"deployment\"\n \"\"\"Publishing items to the workspace.\"\"\"\n UNPUBLISH = \"unpublish\"\n \"\"\"Unpublishing/removing items from the workspace.\"\"\"\n\n\n# The following resources can be unpublished only if their feature flags are set\nUNPUBLISH_FLAG_MAPPING = {\n ItemType.LAKEHOUSE.value: FeatureFlag.ENABLE_LAKEHOUSE_UNPUBLISH.value,\n ItemType.SQL_DATABASE.value: FeatureFlag.ENABLE_SQLDATABASE_UNPUBLISH.value,\n ItemType.WAREHOUSE.value: FeatureFlag.ENABLE_WAREHOUSE_UNPUBLISH.value,\n ItemType.EVENTHOUSE.value: FeatureFlag.ENABLE_EVENTHOUSE_UNPUBLISH.value,\n ItemType.KQL_DATABASE.value: FeatureFlag.ENABLE_KQLDATABASE_UNPUBLISH.value,\n}\n\n# Item Type\nACCEPTED_ITEM_TYPES = tuple(item_type.value for item_type in ItemType)\n\n# API URLs\nDEFAULT_API_ROOT_URL = validate_env_var_api_url(EnvVar.DEFAULT_API_ROOT_URL.value, \"https://api.powerbi.com\")\nFABRIC_API_ROOT_URL = validate_env_var_api_url(EnvVar.FABRIC_API_ROOT_URL.value, \"https://api.fabric.microsoft.com\")\n\n# Retry Settings\nRETRY_AFTER_SECONDS = float(os.environ.get(EnvVar.RETRY_AFTER_SECONDS.value, 300))\nRETRY_BASE_DELAY_SECONDS = float(os.environ.get(EnvVar.RETRY_BASE_DELAY_SECONDS.value, 30))\nRETRY_MAX_DURATION_SECONDS = int(os.environ.get(EnvVar.RETRY_MAX_DURATION_SECONDS.value, 300))\n\n# Parallel Settings\n_parallel_max_workers_raw = os.environ.get(EnvVar.PARALLEL_MAX_WORKERS.value)\nPARALLEL_MAX_WORKERS: int = (\n int(_parallel_max_workers_raw) if _parallel_max_workers_raw and _parallel_max_workers_raw.isdigit() else 8\n)\n\n# HTTP Headers\nAUTHORIZATION_HEADER = \"authorization\"\n\n# Publish\nSHELL_ONLY_PUBLISH = [\n ItemType.LAKEHOUSE.value,\n ItemType.WAREHOUSE.value,\n ItemType.SQL_DATABASE.value,\n ItemType.ML_EXPERIMENT.value,\n]\n\n# Items that do not require assigned capacity\nNO_ASSIGNED_CAPACITY_REQUIRED = [ItemType.SEMANTIC_MODEL.value, ItemType.REPORT.value]\n\n# Exclude Path Regex Patterns for filtering files during publish\nEXCLUDE_PATH_REGEX_MAPPING = {\n ItemType.DATA_AGENT.value: r\".*\\.pbi[/\\\\].*\",\n ItemType.REPORT.value: r\".*\\.pbi[/\\\\].*\",\n ItemType.SEMANTIC_MODEL.value: r\".*\\.pbi[/\\\\].*\",\n ItemType.EVENTHOUSE.value: r\".*\\.children[/\\\\].*\",\n}\n\n# API Format Mapping for item types that require specific API formats\nAPI_FORMAT_MAPPING = {\n ItemType.SPARK_JOB_DEFINITION.value: \"SparkJobDefinitionV2\",\n ItemType.NOTEBOOK.value: \"ipynb\",\n}\n\n# REGEX Constants\nWORKSPACE_ID_REFERENCE_REGEX = r\"\\\"?(default_lakehouse_workspace_id|workspaceId|workspace)\\\"?\\s*[:=]\\s*\\\"(.*?)\\\"\"\nDATAFLOW_SOURCE_REGEX = (\n r'(PowerPlatform\\.Dataflows)(?:\\(\\[\\]\\))?[\\s\\S]*?workspaceId\\s*=\\s*\"(.*?)\"[\\s\\S]*?dataflowId\\s*=\\s*\"(.*?)\"'\n)\nINVALID_FOLDER_CHAR_REGEX = r'[~\"#.%&*:<>?/\\\\{|}]'\nKQL_DATABASE_FOLDER_PATH_REGEX = r\"(?i)^(.*)/[^/]+\\.Eventhouse/\\.children(?:/.*)?$\"\n\n# Well known file names\nDATA_PIPELINE_CONTENT_FILE_JSON = \"pipeline-content.json\"\n\n# Item Type to File Mapping (to check for item dependencies)\nITEM_TYPE_TO_FILE = {ItemType.DATA_PIPELINE.value: DATA_PIPELINE_CONTENT_FILE_JSON}\n\n# Property path to get SQL Endpoint or Eventhouse URI\nPROPERTY_PATH_ATTR_MAPPING = {\n ItemType.LAKEHOUSE.value: {\n \"sqlendpoint\": \"body/properties/sqlEndpointProperties/connectionString\",\n \"sqlendpointid\": \"body/properties/sqlEndpointProperties/id\",\n },\n ItemType.WAREHOUSE.value: {\n \"sqlendpoint\": \"body/properties/connectionString\",\n },\n ItemType.SQL_DATABASE.value: {\n \"sqlendpoint\": \"body/properties/serverFqdn\",\n },\n ItemType.EVENTHOUSE.value: {\n \"queryserviceuri\": \"body/properties/queryServiceUri\",\n },\n}\n\n# Parameter file configs\nPARAMETER_FILE_NAME = \"parameter.yml\"\n# Parameters to validate\nPARAM_NAMES = [\"find_replace\", \"key_value_replace\", \"spark_pool\", \"semantic_model_binding\"]\n\nITEM_ATTR_LOOKUP = [\"id\", \"sqlendpoint\", \"sqlendpointid\", \"queryserviceuri\"]\n\n# Parameter file validation messages\nINVALID_REPLACE_VALUE_SPARK_POOL = {\n \"missing key\": \"The '{}' environment dict in spark_pool must contain a 'type' and a 'name' key\",\n \"missing value\": \"The '{}' environment in spark_pool is missing a value for '{}' key\",\n \"invalid value\": \"The '{}' environment in spark_pool must contain 'Capacity' or 'Workspace' as a value for 'type'\",\n}\nPARAMETER_MSGS = {\n \"validating\": \"Validating {}\",\n \"passed\": \"Validation passed: {}\",\n \"failed\": \"Validation failed with error: {}\",\n \"terminate\": \"Validation terminated: {}\",\n \"found\": f\"Found {PARAMETER_FILE_NAME} file\",\n \"not found\": \"Parameter file not found with path: {}\",\n \"not set\": \"Parameter file path is not set\",\n \"empty yaml\": \"YAML content is empty\",\n \"duplicate key\": \"duplicate key(s) found: {}\",\n \"valid load\": f\"Successfully loaded {PARAMETER_FILE_NAME}\",\n \"invalid load\": f\"Error loading {PARAMETER_FILE_NAME} \" + \"'{}'\",\n \"invalid structure\": \"Invalid parameter file structure\",\n \"valid structure\": \"Parameter file structure is valid\",\n \"invalid name\": \"Invalid parameter name '{}' found in the parameter file\",\n \"valid name\": \"Parameter names are valid\",\n \"invalid data type\": \"The provided '{}' is not of type {} in {}\",\n \"missing key\": \"{} is missing keys\",\n \"invalid key\": \"{} contains invalid keys\",\n \"valid keys\": \"{} contains valid keys\",\n \"mixed format\": \"Parameter '{}' contains mixed format keys (legacy and new format cannot be combined)\",\n \"missing required value\": \"Missing value for '{}' key in {}\",\n \"valid required values\": \"Required values in {} are valid\",\n \"missing replace value\": \"{} is missing a replace value for '{}' environment'\",\n \"valid replace value\": \"Values in 'replace_value' dict in {} are valid\",\n \"invalid replace value\": INVALID_REPLACE_VALUE_SPARK_POOL,\n \"no optional\": \"No optional values provided in {}\",\n \"invalid item type\": \"Item type '{}' not in scope\",\n \"invalid item name\": \"Item name '{}' not found in the repository directory\",\n \"invalid file path\": \"Number of paths in list '{}' that are invalid or not found in the repository directory: {}\",\n \"no valid file path\": \"No valid file path found in the repository directory for {}\",\n \"valid optional\": \"Optional values in {} are valid. Checking for file matches in the repository directory\",\n \"valid parameter\": \"{} parameter is valid\",\n \"skip\": \"The {} '{}' replacement will be skipped due to {} in parameter {}\",\n \"no target env\": \"target environment '{}' not found\",\n \"all target env\": \"The replace value: '{}' will be applied for any target environment\",\n \"other target env\": \"The '{}' environment key can only be used alone. Other environment keys found in replace_value: '{}'\",\n \"no filter match\": \"unmatched optional filters\",\n # Path resolution messages\n \"resolving_relative_path\": \"Resolving path '{}' to be relative to repository directory\",\n \"using_param_file_path\": \"Using parameter file path: '{}'\",\n \"using_default_param_file_path\": \"Using default parameter file path: '{}'\",\n \"param_file_not_found\": \"Parameter file path not found at: '{}'. The path was resolved from: '{}' relative to repository directory: '{}'\",\n \"param_path_not_file\": \"The specified parameter path '{}' exists but is not a file.\",\n \"both_param_path_and_name\": \"Both parameter_file_name: '{}' and parameter_file_path: '{}' were provided. Using parameter_file_path\",\n # Parameter validation messages\n \"param_not_found\": \"The {} parameter was not found\",\n \"param_found\": \"Found the {} parameter\",\n \"param_count\": \"{} {} parameters found\",\n \"regex_ignored\": \"The provided is_regex value is not set to 'true', regex matching will be ignored.\",\n \"validation_complete\": \"Parameter file validation passed\",\n \"gateway_deprecated\": \"The 'gateway_binding' parameter is deprecated and will be removed in future releases. Please use 'semantic_model_binding' instead.\",\n \"duplicate_semantic_model\": \"Duplicate semantic model names found: {}. Each semantic model should only appear once in the configuration as only one connection can be bound per semantic model. Please remove duplicate entries to avoid unpredictable binding behavior.\",\n # Template parameter file messages\n \"template_file_not_found\": \"Template parameter file not found: {}\",\n \"template_file_invalid\": \"Invalid template parameter file {}: {}\",\n \"template_file_error\": \"Error loading template parameter file {}: {}\",\n \"template_file_loaded\": \"Successfully loaded template parameter file: {}\",\n \"template_files_processed\": \"Successfully processed {} template parameter file(s)\",\n \"template_files_none_valid\": \"None of the template parameter files were valid or found, content will not be added\",\n}\n\n# Wildcard path support validations\nWILDCARD_PATH_VALIDATIONS = [\n # Invalid combinations\n {\n \"check\": lambda p: any(bad in p for bad in [\"/**/*/\", \"**/**\", \"//\", \"\\\\\\\\\", \"**/**/\"]),\n \"message\": lambda p: f\"Invalid wildcard combination in pattern: '{p}'\",\n },\n # Incorrect recursive wildcard format\n {\n \"check\": lambda p: \"**\" in p and not (\"**/\" in p or \"/**\" in p),\n \"message\": lambda p: f\"Invalid recursive wildcard format (use **/ or /**): '{p}'\",\n },\n]\n\n\nINDENT = \"->\"\n\n\n# Define supported sections and settings for config file\nCONFIG_SECTIONS = {\n \"core\": {\n \"type\": dict,\n \"settings\": [\"workspace_id\", \"workspace\", \"repository_directory\", \"item_types_in_scope\", \"parameter\"],\n },\n \"publish\": {\n \"type\": dict,\n \"settings\": [\n \"exclude_regex\",\n \"folder_exclude_regex\",\n \"folder_path_to_include\",\n \"items_to_include\",\n \"shortcut_exclude_regex\",\n \"skip\",\n ],\n },\n \"unpublish\": {\"type\": dict, \"settings\": [\"exclude_regex\", \"items_to_include\", \"skip\"]},\n \"features\": {\"type\": (list, dict), \"settings\": []},\n \"constants\": {\"type\": dict, \"settings\": []},\n}\n\n# Config deployment validation messages\nCONFIG_VALIDATION_MSGS = {\n # File validation\n \"file\": {\n \"path_empty\": \"Configuration file path must be a non-empty string\",\n \"invalid_path\": \"Invalid file path '{}': {}\",\n \"not_found\": \"Configuration file not found: {}\",\n \"not_file\": \"Path is not a file: {}\",\n \"yaml_syntax\": \"Invalid YAML syntax: {}\",\n \"encoding_error\": \"File encoding error (expected UTF-8): {}\",\n \"permission_denied\": \"Permission denied reading file: {}\",\n \"unexpected_error\": \"Unexpected error reading file: {}\",\n \"empty_file\": \"Configuration file is empty or contains only comments\",\n \"not_dict\": \"Configuration must be a dictionary, got {}\",\n },\n # Override validation\n \"override\": {\n \"apply_failed\": \"Failed to apply config override for section '{}': {}\",\n \"unsupported_section\": \"Cannot override unsupported config section: '{}'. Supported: {}\",\n \"wrong_type\": \"Override section '{}' must be a {}, got {}\",\n \"unsupported_setting\": \"Cannot override unsupported setting '{}.{}'. Supported: {}\",\n \"cannot_create_core\": \"Cannot create 'core' section - required section must exist in the config file to override\",\n \"cannot_create_required\": \"Cannot create required field 'core.{}'\",\n \"cannot_create_workspace_id\": \"Cannot create workspace identifier 'core.{}'\",\n },\n # Structure validation\n \"structure\": {\n \"missing_core\": \"Configuration must contain a 'core' section\",\n \"core_not_dict\": \"'core' section must be a dictionary, got {}\",\n \"missing_workspace_id\": \"Configuration must specify either 'workspace_id' or 'workspace' in core section\",\n \"missing_repository_dir\": \"Configuration must specify 'repository_directory' in core section\",\n },\n # Environment validation\n \"environment\": {\n \"no_env_with_mappings\": \"Configuration contains environment mappings but no environment was provided. Please specify an environment or remove environment mappings.\",\n \"env_not_found\": \"Environment '{}' not found in '{}' mappings. Available: {}\",\n \"empty_mapping\": \"'{}' environment mapping cannot be empty\",\n \"invalid_env_key\": \"Environment key in '{}' must be a non-empty string, got: {}\",\n \"empty_env_value\": \"'{}' value for environment '{}' cannot be empty\",\n },\n # Field validation\n \"field\": {\n \"string_or_dict\": \"'{}' must be either a string or environment mapping dictionary (e.g., {{dev: 'dev_value', prod: 'prod_value'}}), got type {}\",\n \"list_or_dict\": \"'{}' must be either a list or environment mapping dictionary (e.g., {{dev: ['dev_value1', 'dev_value2'], prod: ['prod_value']}}), got type {}\",\n \"empty_value\": \"'{}' cannot be empty\",\n \"empty_list\": \"'{}' cannot be empty if specified\",\n \"invalid_guid\": \"'{}' must be a valid GUID format: {}\",\n \"item_types_list_or_dict\": \"'item_types_in_scope' must be either a list or environment mapping dictionary (e.g., {{dev: ['Notebook'], prod: ['DataPipeline']}}), got type {}\",\n \"invalid_item_type\": \"Item type must be a string, got {}: {}\",\n \"unsupported_item_type_env\": \"Invalid item type '{}' in environment '{}'. Available types: {}\",\n \"unsupported_item_type\": \"Invalid item type '{}'. Available types: {}\",\n },\n # Path validation\n \"path\": {\n \"skip\": \"Skipping {} path resolution due to config file validation failure\",\n \"absolute\": \"Using absolute {} path{}: '{}'\",\n \"git_repo\": \"{}{} must be in the same git repository as the configuration file. Config repository: {}, {} repository: {}\",\n \"resolved\": \"{} '{}' resolved relative to config path{}: '{}'\",\n \"not_found\": \"{} not found at resolved path{}: '{}'\",\n \"not_directory\": \"{} path exists but is not a directory{}: '{}'\",\n \"not_file\": \"{} path exists but is not a file{}: '{}'\",\n \"invalid\": \"Invalid {} path '{}'{}: {}\",\n },\n # Operation section validation\n \"operation\": {\n \"unsupported_field\": \"'{}' field is not supported in '{}' section\",\n \"not_dict\": \"'{}' section must be a dictionary, got {}\",\n \"invalid_regex\": \"'{}' in {} is not a valid regex pattern: {}\",\n \"empty_string\": \"'{}' cannot be an empty string\",\n \"empty_list\": \"'{}' cannot be an empty list\",\n \"list_entry_type\": \"'{}[{}]' must be a string, got {}\",\n \"list_entry_empty\": \"'{}[{}]' cannot be an empty string\",\n \"features_type\": \"'features' section must be either a list or environment mapping dictionary, got {}\",\n \"empty_section\": \"'{}' section cannot be empty if specified\",\n \"empty_section_env\": \"'{}.{}' cannot be empty if specified\",\n \"invalid_constant_key\": \"Constant key in '{}' must be a non-empty string, got: {}\",\n \"unknown_constant\": \"Unknown constant '{}' in '{}' - this constant does not exist in fabric_cicd.constants\",\n \"folders_list_prefix\": \"'{}[{}]' entry must start with '/' (got '{}')\",\n \"mutually_exclusive\": \"Cannot specify both '{}' and '{}'. Choose one filtering strategy.\",\n \"mutually_exclusive_env\": \"Cannot specify both '{}' and '{}' for the same environment(s): {}. Choose one filtering strategy per environment.\",\n },\n # Log messages\n \"log\": {\n \"override_section\": \"Override: {} '{}' section with value: '{}'\",\n \"override_setting\": \"Override: {} {}.{} with value: '{}'\",\n \"override_env_specific\": \"Override: updated {}.{}.{} with value: '{}'\",\n \"override_env_mapping\": \"Override: {}.{} added with environment mapping, with {} value: '{}'\",\n \"override_added_section\": \"Override: added '{}' section\",\n },\n}\n" }, { "path": "src/fabric_cicd/fabric_workspace.py", "content": "# Copyright (c) Microsoft Corporation.\n# Licensed under the MIT License.\n\n\"\"\"Module provides the FabricWorkspace class to manage and publish workspace items to the Fabric API.\"\"\"\n\nimport json\nimport logging\nimport os\nimport re\nimport threading\nfrom pathlib import Path\nfrom typing import Optional\n\nimport dpath\nfrom azure.core.credentials import TokenCredential\n\nfrom fabric_cicd import constants\nfrom fabric_cicd._common._check_utils import check_regex, check_valid_json_content, check_valid_yaml_content\nfrom fabric_cicd._common._exceptions import FailedPublishedItemStatusError, InputError, ParameterFileError, ParsingError\nfrom fabric_cicd._common._fabric_endpoint import FabricEndpoint\nfrom fabric_cicd._common._item import Item\nfrom fabric_cicd._common._logging import log_header\nfrom fabric_cicd.constants import FeatureFlag, ItemType\n\nlogger = logging.getLogger(__name__)\n\n\nclass FabricWorkspace:\n \"\"\"A class to manage and publish workspace items to the Fabric API.\"\"\"\n\n def __init__(\n self,\n *,\n repository_directory: str,\n token_credential: TokenCredential,\n item_type_in_scope: Optional[list[str]] = None,\n environment: str = \"N/A\",\n workspace_id: Optional[str] = None,\n workspace_name: Optional[str] = None,\n **kwargs: object,\n ) -> None:\n \"\"\"\n Initializes the FabricWorkspace instance.\n\n Args:\n repository_directory: Local directory path of the repository where items are to be deployed from.\n token_credential: The token credential to use for API requests (e.g., AzureCliCredential, ClientSecretCredential) - required.\n item_type_in_scope: Item types that should be deployed for a given workspace. If omitted, defaults to all available item types.\n environment: The environment to be used for parameterization.\n workspace_id: The ID of the workspace to interact with. Either `workspace_id` or `workspace_name` must be provided. Considers only `workspace_id` if both are specified.\n workspace_name: The name of the workspace to interact with. Either `workspace_id` or `workspace_name` must be provided. Considers only `workspace_id` if both are specified.\n kwargs: Additional keyword arguments.\n\n Examples:\n Basic usage\n >>> from fabric_cicd import FabricWorkspace\n >>> from azure.identity import AzureCliCredential\n >>> workspace = FabricWorkspace(\n ... workspace_id=\"your-workspace-id\",\n ... repository_directory=\"/path/to/repo\",\n ... item_type_in_scope=[\"Environment\", \"Notebook\", \"DataPipeline\"],\n ... token_credential=AzureCliCredential() # or any other TokenCredential\n ... )\n\n Basic usage with workspace_name\n >>> from fabric_cicd import FabricWorkspace\n >>> from azure.identity import AzureCliCredential\n >>> workspace = FabricWorkspace(\n ... workspace_name=\"your-workspace-name\",\n ... repository_directory=\"/path/to/repo\",\n ... token_credential=AzureCliCredential() # or any other TokenCredential\n ... )\n\n With optional parameters\n >>> from fabric_cicd import FabricWorkspace\n >>> from azure.identity import AzureCliCredential\n >>> workspace = FabricWorkspace(\n ... workspace_id=\"your-workspace-id\",\n ... repository_directory=\"/your/path/to/repo\",\n ... item_type_in_scope=[\"Environment\", \"Notebook\", \"DataPipeline\"],\n ... environment=\"your-target-environment\",\n ... token_credential=AzureCliCredential() # or any other TokenCredential\n ... )\n\n With service principal credentials\n >>> from fabric_cicd import FabricWorkspace\n >>> from azure.identity import ClientSecretCredential\n >>> client_id = \"your-client-id\"\n >>> client_secret = \"your-client-secret\"\n >>> tenant_id = \"your-tenant-id\"\n >>> token_credential = ClientSecretCredential(\n ... client_id=client_id, client_secret=client_secret, tenant_id=tenant_id\n ... )\n >>> workspace = FabricWorkspace(\n ... workspace_id=\"your-workspace-id\",\n ... repository_directory=\"/your/path/to/repo\",\n ... item_type_in_scope=[\"Environment\", \"Notebook\", \"DataPipeline\"],\n ... token_credential=token_credential\n ... )\n \"\"\"\n from fabric_cicd._common._validate_input import (\n validate_environment,\n validate_item_type_in_scope,\n validate_repository_directory,\n validate_token_credential,\n validate_workspace_id,\n validate_workspace_name,\n )\n\n # Validate token_credential. A TokenCredential is required to authenticate API requests\n token_credential = validate_token_credential(token_credential)\n\n # Initialize endpoint\n self.endpoint = FabricEndpoint(token_credential=token_credential)\n\n # Snapshot at construction so subsequent configure_fabric_fqdn calls for a\n # different workspace don't retarget this instance.\n self._api_root_url = constants.DEFAULT_API_ROOT_URL\n\n # Set workspace_id class variable\n if workspace_id:\n self.workspace_id = validate_workspace_id(workspace_id)\n elif workspace_name:\n self.workspace_id = self._resolve_workspace_id(validate_workspace_name(workspace_name))\n else:\n msg = \"Either workspace_name or workspace_id must be specified.\"\n raise InputError(msg, logger)\n\n # Validate and set class variables\n self.repository_directory: Path = validate_repository_directory(repository_directory)\n self.item_type_in_scope = validate_item_type_in_scope(item_type_in_scope)\n self.environment = validate_environment(environment)\n self.publish_item_name_exclude_regex = None\n self.publish_folder_path_exclude_regex = None\n self.publish_folder_path_to_include = None\n self.shortcut_exclude_regex = None\n self.items_to_include = None\n self.responses = None\n self.unpublish_responses = None\n self.repository_folders = {}\n self.repository_items = {}\n self.deployed_folders = {}\n self.deployed_items = {}\n\n # Initialize dataflow dependencies dictionary (used in dataflow item processing)\n self.dataflow_dependencies = {}\n\n # Initialize workspace pools cache (used in Environment item processing)\n self._workspace_pools_cache: Optional[list[dict]] = None\n self._workspace_pools_cache_lock = threading.Lock()\n\n # Initialize cache for _get_item_attribute method\n self._item_attribute_cache = {}\n self._item_attribute_cache_lock = threading.Lock()\n\n # Get parameter_file_path from kwargs\n self.parameter_file_path = kwargs.get(\"parameter_file_path\")\n\n # base_api_url is no longer supported - raise error if provided\n if \"base_api_url\" in kwargs:\n msg = (\n \"Setting base_api_url is no longer supported. Please use the following instead:\\n\"\n \">>> import fabric_cicd.constants\\n\"\n \">>> constants.DEFAULT_API_ROOT_URL = ''\"\n )\n raise InputError(msg, logger)\n\n # Initialize parameter file — skipped when config-based deployment omits the\n # 'parameter' field, ensuring repository parameter.yml is not auto-discovered.\n skip_parameterization = kwargs.get(\"skip_parameterization\", False)\n if not skip_parameterization:\n self._refresh_parameter_file()\n else:\n self.environment_parameter = {}\n logger.info(\n \"Parameterization skipped: no parameter file configured/provided (environment=%s).\",\n self.environment,\n )\n\n @property\n def base_api_url(self) -> str:\n \"\"\"Construct the base API URL using constants.\"\"\"\n return f\"{self._api_root_url}/v1/workspaces/{self.workspace_id}\"\n\n def _resolve_workspace_id(self, workspace_name: str) -> str:\n \"\"\"Resolve workspace ID based on the workspace name given.\"\"\"\n response = self.endpoint.invoke(method=\"GET\", url=f\"{self._api_root_url}/v1/workspaces\")\n for workspace in response[\"body\"][\"value\"]:\n if workspace[\"displayName\"] == workspace_name:\n return workspace[\"id\"]\n msg = f\"Workspace ID could not be resolved from workspace name: {workspace_name}.\"\n raise InputError(msg, logger)\n\n def _resolve_workspace_name(self) -> str:\n \"\"\"Resolve workspace display name of the target workspace ID.\"\"\"\n response = self.endpoint.invoke(method=\"GET\", url=f\"{self._api_root_url}/v1/workspaces/{self.workspace_id}\")\n if \"displayName\" in response.get(\"body\", {}):\n return response[\"body\"][\"displayName\"]\n msg = f\"Workspace name could not be resolved from workspace ID: {self.workspace_id}.\"\n raise InputError(msg, logger)\n\n def _lookup_item_attribute(self, workspace_id: str, item_type: str, item_name: str, attribute_name: str) -> str:\n \"\"\"Lookup item attribute in the specified workspace based on item type and name.\"\"\"\n response = self.endpoint.invoke(\n method=\"GET\", url=f\"{self._api_root_url}/v1/workspaces/{workspace_id}/items\"\n )\n for item in response[\"body\"][\"value\"]:\n if item[\"type\"] == item_type and item[\"displayName\"] == item_name:\n item_guid = item[\"id\"]\n if attribute_name == \"id\":\n return item_guid\n # For other attribute, use the item guid to get the attribute value\n return self._get_item_attribute(workspace_id, item_type, item_guid, item_name, attribute_name)\n\n msg = f\"Failed to look up item in workspace: {workspace_id}, item_type: {item_type}, item_name: {item_name}\"\n raise InputError(msg, logger)\n\n def _get_item_attribute(\n self, workspace_id: str, item_type: str, item_guid: str, item_name: str, attribute_name: str\n ) -> str:\n \"\"\"Returns the attribute value of an item in the specified workspace based on item type and id\"\"\"\n # No need to make API calls if we don't have an item guid\n if not item_guid:\n return \"\"\n\n # Create a cache key for this request\n cache_key = (workspace_id, item_type, item_guid, item_name, attribute_name)\n\n # Check if result is already cached\n with self._item_attribute_cache_lock:\n if cache_key in self._item_attribute_cache:\n return self._item_attribute_cache[cache_key]\n\n # Check if this item type has property mappings\n if item_type not in constants.PROPERTY_PATH_ATTR_MAPPING:\n logger.debug(f\"No property path mappings defined for {item_type}\")\n return \"\"\n\n # Get the attribute mappings for this item type\n attribute_mappings = constants.PROPERTY_PATH_ATTR_MAPPING.get(item_type)\n\n # Check if the requested attribute is supported\n if attribute_name not in attribute_mappings:\n logger.debug(\n f\"Attribute '{attribute_name}' not supported for {item_type} '{item_name}'. Supported: {list(attribute_mappings.keys())}\"\n )\n return \"\"\n\n # Get the property path for this attribute\n property_path = attribute_mappings[attribute_name]\n\n response = self.endpoint.invoke(\n method=\"GET\",\n url=f\"{self._api_root_url}/v1/workspaces/{workspace_id}/{item_type.lower()}s/{item_guid}\",\n )\n # Extract the attribute value using the path\n attribute_value = dpath.get(response, property_path, default=\"\")\n if not attribute_value:\n msg = f\"Attribute value not found for {item_type} '{item_name}'\"\n raise InputError(msg, logger)\n\n # Cache the result before returning\n with self._item_attribute_cache_lock:\n self._item_attribute_cache[cache_key] = attribute_value\n return attribute_value\n\n def _get_workspace_pools(self) -> list[dict]:\n \"\"\"Return the list of workspace custom Spark pools, fetching from the API on first call.\n\n The result is cached so that subsequent calls during the same deployment\n do not make additional API requests. Thread-safe via a lock.\n\n Returns:\n A list of pool dictionaries from the Fabric Spark custom-pools API.\n \"\"\"\n with self._workspace_pools_cache_lock:\n if self._workspace_pools_cache is None:\n # https://learn.microsoft.com/en-us/rest/api/fabric/spark/custom-pools/list-workspace-custom-pools\n response = self.endpoint.invoke(\n method=\"GET\",\n url=f\"{self.base_api_url}/spark/pools\",\n )\n\n pools = response.get(\"body\", {}).get(\"value\") if isinstance(response, dict) else None\n if not isinstance(pools, list):\n msg = f\"Unexpected response from Spark pools API: expected 'body.value' to be a list. Response: {response}\"\n raise InputError(msg, logger)\n self._workspace_pools_cache = pools\n\n return self._workspace_pools_cache\n\n def _refresh_parameter_file(self) -> None:\n \"\"\"Load parameters if file is present.\"\"\"\n from fabric_cicd._parameter._parameter import Parameter\n\n log_header(logger, \"Validating Parameter File\")\n\n # Initialize the parameter dict and Parameter object\n self.environment_parameter = {}\n parameter_obj = Parameter(\n repository_directory=self.repository_directory,\n item_type_in_scope=self.item_type_in_scope,\n environment=self.environment,\n parameter_file_name=constants.PARAMETER_FILE_NAME,\n parameter_file_path=self.parameter_file_path,\n )\n is_valid = parameter_obj._validate_parameter_file()\n if is_valid:\n self.environment_parameter = parameter_obj.environment_parameter\n else:\n msg = \"Deployment terminated due to an invalid parameter file\"\n raise ParameterFileError(msg, logger)\n\n def _refresh_repository_items(self) -> None:\n \"\"\"Refreshes the repository_items dictionary by scanning the repository directory.\"\"\"\n self.repository_items = {}\n empty_logical_id_paths = [] # Collect all paths with empty logical IDs\n visited_logical_ids = set() # Track visited logical IDs to avoid duplicates\n\n for root, _dirs, files in os.walk(self.repository_directory):\n directory = Path(root)\n # valid item directory with .platform file within\n if \".platform\" in files:\n item_metadata_path = directory / \".platform\"\n\n # Print a warning and skip directory if empty\n if not any(directory.iterdir()):\n logger.warning(f\"Directory {directory.name} is empty.\")\n continue\n\n # Attempt to read metadata file\n try:\n with Path.open(item_metadata_path, encoding=\"utf-8\") as file:\n item_metadata = json.load(file)\n except FileNotFoundError as e:\n msg = f\"{item_metadata_path} path does not exist in the specified repository. {e}\"\n ParsingError(msg, logger)\n except json.JSONDecodeError as e:\n msg = f\"Error decoding JSON in {item_metadata_path}. {e}\"\n ParsingError(msg, logger)\n\n # Ensure required metadata fields are present\n if \"type\" not in item_metadata[\"metadata\"] or \"displayName\" not in item_metadata[\"metadata\"]:\n msg = f\"displayName & type are required in {item_metadata_path}\"\n raise ParsingError(msg, logger)\n\n item_type = item_metadata[\"metadata\"][\"type\"]\n item_description = item_metadata[\"metadata\"].get(\"description\", \"\")\n item_name = item_metadata[\"metadata\"][\"displayName\"]\n item_logical_id = item_metadata[\"config\"][\"logicalId\"]\n\n # Check for empty logical ID and collect the path\n if not item_logical_id or item_logical_id.strip() == \"\":\n empty_logical_id_paths.append(str(item_metadata_path))\n continue # Skip processing this item further\n\n # Validate duplicate logical IDs (skip default GUID as export API uses it as a placeholder)\n if item_logical_id != constants.DEFAULT_GUID:\n if item_logical_id in visited_logical_ids:\n msg = f\"Duplicate logicalId '{item_logical_id}' found in {item_metadata_path}\"\n raise FailedPublishedItemStatusError(msg, logger)\n visited_logical_ids.add(item_logical_id)\n\n item_path = directory\n relative_path = f\"/{directory.relative_to(self.repository_directory).as_posix()}\"\n # Special handling for KQLDatabase items:\n # .Eventhouse/.children/ directory structure, requires extracting the\n # parent folder path before the Eventhouse container, not just\n # the immediate parent directory\n if item_type == ItemType.KQL_DATABASE.value:\n pattern = re.compile(constants.KQL_DATABASE_FOLDER_PATH_REGEX)\n match = pattern.match(relative_path)\n relative_parent_path = match.group(1) if match else None\n else:\n relative_parent_path = \"/\".join(relative_path.split(\"/\")[:-1])\n\n if FeatureFlag.DISABLE_WORKSPACE_FOLDER_PUBLISH.value not in constants.FEATURE_FLAG:\n item_folder_id = self.repository_folders.get(relative_parent_path, \"\")\n else:\n item_folder_id = \"\"\n\n # Get the GUID if the item is already deployed\n item_guid = self.deployed_items.get(item_type, {}).get(item_name, Item(\"\", \"\", \"\", \"\")).guid\n\n if item_type not in self.repository_items:\n self.repository_items[item_type] = {}\n\n # Add the item to the repository_items dictionary\n self.repository_items[item_type][item_name] = Item(\n type=item_type,\n name=item_name,\n description=item_description,\n guid=item_guid,\n logical_id=item_logical_id,\n path=item_path,\n folder_id=item_folder_id,\n folder_path=relative_parent_path,\n )\n\n self.repository_items[item_type][item_name].collect_item_files()\n\n # If we found any empty logical IDs, raise an error with all paths\n if empty_logical_id_paths:\n if len(empty_logical_id_paths) == 1:\n msg = f\"logicalId cannot be empty in {empty_logical_id_paths[0]}\"\n else:\n paths_list = \"\\n - \".join(empty_logical_id_paths)\n msg = f\"logicalId cannot be empty in the following files:\\n - {paths_list}\"\n raise ParsingError(msg, logger)\n\n def _refresh_deployed_items(self) -> None:\n \"\"\"Refreshes the deployed_items dictionary by querying the Fabric workspace items API.\"\"\"\n # Get all items in workspace\n # https://learn.microsoft.com/en-us/rest/api/fabric/core/items/get-item\n response = self.endpoint.invoke(method=\"GET\", url=f\"{self.base_api_url}/items\")\n\n self.deployed_items = {}\n self.workspace_items = {}\n\n for item in response[\"body\"][\"value\"]:\n item_type = item[\"type\"]\n item_description = item[\"description\"]\n item_name = item[\"displayName\"]\n item_guid = item[\"id\"]\n item_folder_id = item.get(\"folderId\", \"\")\n sql_endpoint = \"\"\n sql_endpoint_id = \"\"\n query_service_uri = \"\"\n\n # Add an empty dictionary if the item type hasn't been added yet\n if item_type not in self.deployed_items:\n self.deployed_items[item_type] = {}\n\n if item_type not in self.workspace_items:\n self.workspace_items[item_type] = {}\n\n # Get additional properties\n if item_type in [ItemType.LAKEHOUSE.value, ItemType.WAREHOUSE.value, ItemType.SQL_DATABASE.value]:\n sql_endpoint = self._get_item_attribute(\n self.workspace_id, item_type, item_guid, item_name, \"sqlendpoint\"\n )\n sql_endpoint_id = self._get_item_attribute(\n self.workspace_id, item_type, item_guid, item_name, \"sqlendpointid\"\n )\n if item_type in [ItemType.EVENTHOUSE.value]:\n query_service_uri = self._get_item_attribute(\n self.workspace_id, item_type, item_guid, item_name, \"queryserviceuri\"\n )\n\n # Add item details to the deployed_items dictionary\n self.deployed_items[item_type][item_name] = Item(\n type=item_type,\n name=item_name,\n description=item_description,\n guid=item_guid,\n folder_id=item_folder_id,\n )\n\n # Add item details to the workspace_items dictionary required for parameterization (public-facing attributes)\n self.workspace_items[item_type][item_name] = {\n \"id\": item_guid,\n \"sqlendpoint\": sql_endpoint,\n \"sqlendpointid\": sql_endpoint_id,\n \"queryserviceuri\": query_service_uri,\n }\n\n def _replace_logical_ids(self, raw_file: str) -> str:\n \"\"\"\n Replaces logical IDs with deployed GUIDs in the raw file content.\n\n Args:\n raw_file: The raw file content where logical IDs need to be replaced.\n \"\"\"\n for item_name in self.repository_items.values():\n for item_details in item_name.values():\n logical_id = item_details.logical_id\n item_guid = item_details.guid\n\n # Skip placeholder logical IDs (default GUID) used by items via export API\n if logical_id == constants.DEFAULT_GUID:\n continue\n\n if logical_id in raw_file:\n if item_guid == \"\":\n msg = f\"Cannot replace logical ID '{logical_id}' as referenced item is not yet deployed.\"\n raise ParsingError(msg, logger)\n raw_file = raw_file.replace(logical_id, item_guid)\n\n return raw_file\n\n def _replace_parameters(self, file_obj: object, item_obj: object) -> str:\n \"\"\"\n Replaces values found in parameter file with the chosen environment value. Handles two parameter dictionary structures.\n\n Args:\n file_obj: The File object instance that provides the file content and file path.\n item_obj: The Item object instance that provides the item type and item name.\n \"\"\"\n from fabric_cicd._parameter._utils import (\n check_replacement,\n extract_find_value,\n extract_parameter_filters,\n extract_replace_value,\n process_environment_key,\n replace_key_value,\n )\n\n # Parse the file_obj and item_obj\n raw_file = file_obj.contents\n item_type = item_obj.type\n item_name = item_obj.name\n file_path = file_obj.file_path\n\n if \"key_value_replace\" in self.environment_parameter:\n for parameter_dict in self.environment_parameter.get(\"key_value_replace\"):\n # Extract the file filter values and set the match condition\n input_type, input_name, input_path = extract_parameter_filters(self, parameter_dict)\n filter_match = check_replacement(input_type, input_name, input_path, item_type, item_name, file_path)\n\n # Perform replacement if condition is met and file contains valid JSON or YAML\n if filter_match:\n if check_valid_json_content(raw_file):\n raw_file = replace_key_value(self, parameter_dict, raw_file, self.environment)\n elif check_valid_yaml_content(raw_file):\n raw_file = replace_key_value(self, parameter_dict, raw_file, self.environment, is_yaml=True)\n\n if \"find_replace\" in self.environment_parameter:\n for parameter_dict in self.environment_parameter.get(\"find_replace\"):\n # Extract the file filter values and set the match condition\n input_type, input_name, input_path = extract_parameter_filters(self, parameter_dict)\n filter_match = check_replacement(input_type, input_name, input_path, item_type, item_name, file_path)\n\n # Extract the find_pattern and replace_value_dict\n find_info = extract_find_value(parameter_dict, raw_file, filter_match)\n replace_value_dict = process_environment_key(self.environment, parameter_dict.get(\"replace_value\", {}))\n\n # Replace any found references with specified environment value if conditions are met\n if filter_match and self.environment in replace_value_dict and find_info[\"has_matches\"]:\n replace_value = extract_replace_value(self, replace_value_dict[self.environment])\n if replace_value:\n pattern = find_info[\"pattern\"]\n is_regex = find_info[\"is_regex\"]\n\n if is_regex:\n # For regex patterns, use re.sub with lambda to replace only the captured group\n # Use string slicing to precisely replace only the captured group (group 1)\n # The slicing calculates relative positions: match.start(1) - match.start(0) gives\n # the start position of group 1 within the full match, and similarly for end position\n raw_file = re.sub(\n pattern,\n lambda match, repl=replace_value: (\n match.group(0)[: match.start(1) - match.start(0)]\n + repl\n + match.group(0)[match.end(1) - match.start(0) :]\n ),\n raw_file,\n )\n logger.debug(\n f\"Replacing regex pattern '{pattern}' captured group with '{replace_value}' in {item_name}.{item_type}\"\n )\n else:\n # For non-regex matches, replace as before\n raw_file = raw_file.replace(pattern, replace_value)\n logger.debug(f\"Replacing '{pattern}' with '{replace_value}' in {item_name}.{item_type}\")\n\n return raw_file\n\n def _replace_workspace_ids(self, raw_file: str) -> str:\n \"\"\"\n Replaces feature branch workspace ID, default (i.e. 00000000-0000-0000-0000-000000000000) and non-default\n (actual workspace ID guid) values, with target workspace ID in the raw file content.\n\n Args:\n raw_file: The raw file content where workspace IDs need to be replaced.\n \"\"\"\n # Use re.sub to replace all matches\n return re.sub(\n constants.WORKSPACE_ID_REFERENCE_REGEX,\n lambda match: (\n match.group(0).replace(constants.DEFAULT_GUID, self.workspace_id)\n if match.group(2) == constants.DEFAULT_GUID\n else match.group(0)\n ),\n raw_file,\n )\n\n def _convert_id_to_name(self, item_type: str, generic_id: str, lookup_type: str) -> str:\n \"\"\"\n For a given item_type and id, returns the item name. Special handling for both deployed and repository items.\n\n Args:\n item_type: Type of the item (e.g., Notebook, Environment).\n generic_id: Logical id or item guid of the item based on lookup_type.\n lookup_type: Finding references in deployed file or repo file (Deployed or Repository).\n \"\"\"\n lookup_dict = self.repository_items if lookup_type == \"Repository\" else self.deployed_items\n\n for item_details in lookup_dict[item_type].values():\n lookup_id = item_details.logical_id if lookup_type == \"Repository\" else item_details.guid\n if lookup_id == generic_id:\n return item_details.name\n # if not found\n return None\n\n def _convert_path_to_id(self, item_type: str, path: str) -> str:\n \"\"\"\n For a given path and item type, returns the logical id.\n\n Args:\n item_type: Type of the item (e.g., Notebook, Environment).\n path: Full path of the desired item.\n \"\"\"\n if item_type in self.repository_items:\n for item_details in self.repository_items[item_type].values():\n if item_details.path == Path(path):\n return item_details.logical_id\n # if not found\n return None\n\n def _publish_item(\n self,\n item_name: str,\n item_type: str,\n exclude_path: str = r\"^(?!.*)\",\n func_process_file: Optional[callable] = None,\n **kwargs,\n ) -> None:\n \"\"\"\n Publishes or updates an item in the Fabric Workspace.\n\n Args:\n item_name: Name of the item to publish.\n item_type: Type of the item (e.g., Notebook, Environment).\n exclude_path: Regex string of paths to exclude. Defaults to r\"^(?!.*)\".\n func_process_file: Custom function to process file contents. Defaults to None.\n **kwargs: Additional keyword arguments.\n \"\"\"\n item = self.repository_items[item_type][item_name]\n folder_path = item.folder_path or \"\"\n\n # Initialize response collection for this item if responses are being tracked\n api_response = None\n\n # ===== FILTER ORDER (applied in _publish_item): Item Exclusion → Folder Exclusion → Folder Inclusion =====\n # Note: items_to_include filtering is applied upstream in publish_all() via get_items_to_publish().\n\n # 1. Skip publishing if the item is excluded by the regex\n if self.publish_item_name_exclude_regex:\n regex_pattern = check_regex(self.publish_item_name_exclude_regex)\n if regex_pattern.match(item_name):\n item.skip_publish = True\n logger.info(f\"Skipping publishing of {item_type} '{item_name}' due to exclusion regex.\")\n return\n\n # 2. Skip publishing if the item's folder path is excluded by the regex\n if self.publish_folder_path_exclude_regex and folder_path:\n regex_pattern = check_regex(self.publish_folder_path_exclude_regex)\n # Walk up the folder hierarchy checking each level against the exclusion regex.\n # Cases handled:\n # 1. Direct match — item's folder matches the regex (e.g., item in /A/B, regex matches /A/B)\n # 2. Ancestor match — item's ancestor folder matches (e.g., item in /A/B/C, regex matches /A)\n # 3. No match at any level — no exclusion applied, continue to next checks\n # Note: Root-level items (empty folder_path) are not impacted by folder path exclusion.\n # This ensures excluding a parent folder cascades to all descendants.\n path_to_check = folder_path\n while path_to_check:\n # If the current path (or ancestor) matches the exclusion pattern, skip this item\n if regex_pattern.search(path_to_check):\n item.skip_publish = True\n logger.info(f\"Skipping publishing of {item_type} '{item_name}' due to folder path exclusion regex.\")\n return\n # Move one level up by stripping the last path segment (e.g., \"/a/b/c\" -> \"/a/b\")\n if \"/\" in path_to_check and path_to_check != \"\":\n path_to_check = path_to_check.rsplit(\"/\", 1)[0]\n else:\n # Reached the root level with no match; stop checking\n break\n\n # 3. Skip publishing if the item's folder path is not in the include list\n # If the item's folder is not in the explicit include list, skip item publish (even though folder has been created).\n # Note: unlike exclusion, this does NOT walk ancestors — only exact folder match is checked.\n # (e.g., including /A does NOT include items in /A/B, or including /A/B does NOT include items in /A, but the folder /A will still exist).\n if (\n self.publish_folder_path_to_include\n and folder_path\n and folder_path not in self.publish_folder_path_to_include\n ):\n item.skip_publish = True\n logger.info(\n f\"Skipping publishing of {item_type} '{item_name}' under {folder_path} as it is not in the include list.\"\n )\n return\n\n item_guid = item.guid\n item_description = item.description\n item_files = item.item_files\n\n metadata_body = {\"displayName\": item_name, \"type\": item_type, \"description\": item_description}\n\n # Only shell deployment, no definition support (item_type can be overridden via kwargs)\n shell_only_publish = kwargs.get(\"shell_only_publish\", item_type in constants.SHELL_ONLY_PUBLISH)\n\n if kwargs.get(\"creation_payload\"):\n creation_payload = {\"creationPayload\": kwargs[\"creation_payload\"]}\n combined_body = {**metadata_body, **creation_payload}\n elif shell_only_publish:\n combined_body = metadata_body\n else:\n item_payload = []\n for file in item_files:\n if not re.match(exclude_path, file.relative_path):\n if file.type == \"text\" and not str(file.file_path).endswith(\".platform\"):\n # Only enable parameter replacement in Variable Library item definition files\n if item_type == ItemType.VARIABLE_LIBRARY.value:\n file.contents = self._replace_parameters(file, item)\n # Apply default processing for all other item definition files\n else:\n file.contents = func_process_file(self, item, file) if func_process_file else file.contents\n file.contents = self._replace_logical_ids(file.contents)\n file.contents = self._replace_parameters(file, item)\n file.contents = self._replace_workspace_ids(file.contents)\n\n item_payload.append(file.base64_payload)\n # Some item definitions require specifying the format as multiple API versions exist (i.e. Spark Job Definitions)\n if kwargs.get(\"api_format\"):\n definition_body = {\"definition\": {\"format\": kwargs[\"api_format\"], \"parts\": item_payload}}\n else:\n definition_body = {\"definition\": {\"parts\": item_payload}}\n combined_body = {**metadata_body, **definition_body}\n\n logger.info(f\"Publishing {item_type} '{item_name}'\")\n\n is_deployed = bool(item_guid)\n\n if not is_deployed:\n combined_body = {**combined_body, **{\"folderId\": item.folder_id}}\n\n # Create a new item if it does not exist\n # https://learn.microsoft.com/en-us/rest/api/fabric/core/items/create-item\n item_create_response = self.endpoint.invoke(\n method=\"POST\", url=f\"{self.base_api_url}/items\", body=combined_body\n )\n api_response = item_create_response\n item_guid = item_create_response[\"body\"][\"id\"]\n self.repository_items[item_type][item_name].guid = item_guid\n\n elif is_deployed and not shell_only_publish:\n # Update the item's definition if full publish is required\n # https://learn.microsoft.com/en-us/rest/api/fabric/core/items/update-item-definition\n update_response = self.endpoint.invoke(\n method=\"POST\",\n url=f\"{self.base_api_url}/items/{item_guid}/updateDefinition?updateMetadata=True\",\n body=definition_body,\n )\n api_response = update_response\n elif is_deployed and shell_only_publish:\n # Remove the 'type' key as it's not supported in the update-item API\n metadata_body.pop(\"type\", None)\n\n # Update the item's metadata\n # https://learn.microsoft.com/en-us/rest/api/fabric/core/items/update-item\n metadata_update_response = self.endpoint.invoke(\n method=\"PATCH\",\n url=f\"{self.base_api_url}/items/{item_guid}\",\n body=metadata_body,\n )\n api_response = metadata_update_response\n\n if FeatureFlag.DISABLE_WORKSPACE_FOLDER_PUBLISH.value not in constants.FEATURE_FLAG:\n deployed_item = self.deployed_items.get(item_type, {}).get(item_name) if is_deployed else None\n # Check if the folder has changed\n if deployed_item is not None and deployed_item.folder_id != item.folder_id:\n # Move the item to the correct folder if it has been moved\n # https://learn.microsoft.com/en-us/rest/api/fabric/core/items/move-item\n move_response = self.endpoint.invoke(\n method=\"POST\",\n url=f\"{self.base_api_url}/items/{item_guid}/move\",\n body={\"targetFolderId\": f\"{item.folder_id}\"},\n )\n # For move operations, combine responses if we're tracking them\n if self.responses is not None:\n if api_response:\n # If we already have a response, combine them\n api_response = {\"publish_response\": api_response, \"move_response\": move_response}\n else:\n # If move is the only operation, use the move response\n api_response = move_response\n logger.debug(\n f\"Moved {item_guid} from folder_id {self.deployed_items[item_type][item_name].folder_id} to folder_id {item.folder_id}\"\n )\n\n # Store response if responses are being tracked\n if self.responses is not None and api_response:\n # Initialize item_type dictionary if it doesn't exist\n if item_type not in self.responses:\n self.responses[item_type] = {}\n self.responses[item_type][item_name] = api_response\n\n # skip_publish_logging provided in kwargs to suppress logging if further processing is to be done\n if not kwargs.get(\"skip_publish_logging\", False):\n logger.info(f\"{constants.INDENT}Published {item_type} '{item_name}'\")\n return\n\n def _unpublish_item(self, item_name: str, item_type: str) -> None:\n \"\"\"\n Unpublishes an item from the Fabric workspace.\n\n Args:\n item_name: Name of the item to unpublish.\n item_type: Type of the item (e.g., Notebook, Environment).\n \"\"\"\n item_guid = self.deployed_items[item_type][item_name].guid\n\n logger.info(f\"Unpublishing {item_type} '{item_name}'\")\n\n # Delete the item from the workspace\n # https://learn.microsoft.com/en-us/rest/api/fabric/core/items/delete-item\n try:\n # Apply hard delete if the feature flag is enabled, otherwise defaults to soft deleting (moves the item to the recycle bin)\n hard_delete = FeatureFlag.ENABLE_HARD_DELETE.value in constants.FEATURE_FLAG\n delete_url = f\"{self.base_api_url}/items/{item_guid}\" + (\"?hardDelete=true\" if hard_delete else \"\")\n api_response = self.endpoint.invoke(method=\"DELETE\", url=delete_url)\n logger.info(f\"{constants.INDENT}Unpublished {item_type} '{item_name}'\")\n\n # Store response if responses are being tracked\n if self.unpublish_responses is not None and api_response:\n self.unpublish_responses.setdefault(item_type, {})[item_name] = api_response\n\n except Exception as e:\n msg = f\"Failed to unpublish {item_type} '{item_name}'. Raw exception: {e}\"\n if not hard_delete:\n msg += (\n f\" Consider enabling the '{FeatureFlag.ENABLE_HARD_DELETE.value}' feature flag\"\n \" to perform a permanent deletion, which bypasses the recycle bin\"\n \" and may resolve this issue (requires workspace Admin role).\"\n )\n logger.warning(msg)\n\n def _refresh_deployed_folders(self) -> None:\n \"\"\"\n Converts the folder list payload into a structure of folder name and their ids\n\n output should be like this:\n {\n \"/Pipeline\": \"323eaa75-d70b-498c-8544-6c4219bf336e\",\n \"/Notebook\": \"f802fd90-c70e-4d77-b079-538f617646d3\",\n \"/Notebook/Processing\": \"36ed1a63-be82-4a7a-9364-2e4ff3a66b31\"\n }\n\n \"\"\"\n self.deployed_folders = {}\n request_url = f\"{self.base_api_url}/folders\"\n folders = []\n\n while request_url:\n # https://learn.microsoft.com/en-us/rest/api/fabric/core/folders/list-folders\n response = self.endpoint.invoke(method=\"GET\", url=request_url)\n\n # Handle cases where the response body is empty\n folder_response = response[\"body\"].get(\"value\", [])\n folders.extend(folder for folder in folder_response)\n\n request_url = response[\"header\"].get(\"continuationUri\", None)\n\n # Create a lookup table for folders by their ID\n folder_lookup = {folder[\"id\"]: folder for folder in folders}\n\n # Build the folder hierarchy\n folder_hierarchy = {}\n\n def get_full_path(folder: dict) -> str:\n \"\"\"Recursively build the full path for a folder\"\"\"\n parent_id = folder.get(\"parentFolderId\")\n if parent_id:\n parent_folder = folder_lookup.get(parent_id)\n if parent_folder:\n return f\"{get_full_path(parent_folder)}/{folder['displayName']}\"\n return f\"/{folder['displayName']}\"\n\n for folder in folders:\n full_path = get_full_path(folder)\n folder_hierarchy[full_path] = folder[\"id\"]\n\n self.deployed_folders = folder_hierarchy\n\n def _refresh_repository_folders(self) -> None:\n \"\"\"\n Converts the folder list payload into a structure of folder name and their ids,\n skipping empty folders or folders that only contain other empty folders.\n\n output should be like this:\n {\n \"/Pipeline\": \"\",\n \"/Notebook\": \"\",\n \"/Notebook/Processing\": \"\"\n }\n \"\"\"\n self.repository_folders = {}\n\n root_path = Path(self.repository_directory)\n folder_hierarchy = {}\n\n # Collect all folders that directly contain a .platform file\n platform_folders = set(p.parent for p in root_path.rglob(\".platform\"))\n\n # Now, for every folder, check if any of its subfolders is in platform_folders\n for folder in root_path.rglob(\"*\"):\n if not folder.is_dir() or folder == root_path or folder.name == \".children\":\n continue\n\n # Skip folders that directly contain a .platform file\n if folder in platform_folders:\n continue\n\n # Check if any subfolder (at any depth) is in platform_folders\n if any(sub in platform_folders for sub in folder.rglob(\"*\") if sub.is_dir()):\n relative_path = f\"/{folder.relative_to(root_path).as_posix()}\"\n folder_hierarchy[relative_path] = \"\"\n\n self.repository_folders = folder_hierarchy\n\n def _publish_folders(self) -> None:\n \"\"\"Publishes all folders from the repository.\"\"\"\n # Sort folders by the number of '/' in their paths (ascending order)\n sorted_folders = sorted(self.repository_folders.keys(), key=lambda path: path.count(\"/\"))\n log_header(logger, \"Publishing Workspace Folders\")\n logger.info(\"Publishing Workspace Folders\")\n for folder_path in sorted_folders:\n # Skip folders matching the exclusion regex\n if self.publish_folder_path_exclude_regex:\n regex_pattern = check_regex(self.publish_folder_path_exclude_regex)\n if regex_pattern.search(folder_path):\n logger.info(f\"Skipping publishing of folder '{folder_path}' due to folder path exclusion regex.\")\n continue\n # If any ancestor folder was excluded by the regex, skip this\n # descendant folder too to preserve a consistent hierarchy\n ancestor_path = folder_path\n ancestor_excluded = False\n while \"/\" in ancestor_path and ancestor_path != \"\":\n ancestor_path = ancestor_path.rsplit(\"/\", 1)[0]\n if ancestor_path and regex_pattern.search(ancestor_path):\n ancestor_excluded = True\n break\n if ancestor_excluded:\n logger.info(\n f\"Skipping publishing of folder '{folder_path}' as its ancestor folder was excluded by regex.\"\n )\n continue\n # Skip folders not in the include list\n # Ancestor folders must be published to preserve the correct hierarchy.\n # Even though they may not be explicitly included, (e.g., if /A/B is included, /A must also be published).\n if self.publish_folder_path_to_include:\n is_included = folder_path in self.publish_folder_path_to_include\n is_ancestor_of_included = any(\n included.startswith(folder_path + \"/\") for included in self.publish_folder_path_to_include\n )\n if not is_included and not is_ancestor_of_included:\n logger.info(f\"Skipping publishing of folder '{folder_path}' as it is not in the include list.\")\n continue\n if folder_path in self.deployed_folders:\n # Folder already deployed, update local hierarchy\n self.repository_folders[folder_path] = self.deployed_folders[folder_path]\n logger.debug(f\"Folder exists: {folder_path}\")\n continue\n\n # Publish the folder\n folder_name = folder_path.split(\"/\")[-1]\n folder_parent_path = \"/\".join(folder_path.split(\"/\")[:-1])\n folder_parent_id = self.repository_folders.get(folder_parent_path, None)\n\n if re.search(constants.INVALID_FOLDER_CHAR_REGEX, folder_name):\n msg = f\"Folder name '{folder_name}' contains invalid characters.\"\n raise InputError(msg, logger)\n\n request_body = {\"displayName\": folder_name}\n if folder_parent_id:\n request_body[\"parentFolderId\"] = folder_parent_id\n\n request_url = f\"{self.base_api_url}/folders\"\n response = self.endpoint.invoke(method=\"POST\", url=request_url, body=request_body)\n\n # Update local hierarchy with the new folder ID\n self.repository_folders[folder_path] = response[\"body\"][\"id\"]\n logger.debug(f\"Published folder: {folder_path}\")\n\n logger.info(f\"{constants.INDENT}Published\")\n\n def _unpublish_folders(self) -> None:\n \"\"\"Unpublishes all empty folders in workspace.\"\"\"\n # Sort folders by the number of '/' in their paths (descending order)\n sorted_folder_ids = [\n self.deployed_folders[key]\n for key in sorted(self.deployed_folders.keys(), key=lambda path: path.count(\"/\"), reverse=True)\n ]\n\n ## Any folder that neither contains items nor is an ancestor of a folder\n ## containing items is considered orphaned\n\n # Create a set of folders that contain items\n unorphaned_folders = {\n item.folder_id for items in self.deployed_items.values() for item in items.values() if item.folder_id\n }\n # Skip deletion if all deployed folders are unorphaned\n if unorphaned_folders == set(sorted_folder_ids):\n return\n\n # Create a reversed mapping for folder_id to folder_path lookups\n folder_id_to_path_mapping = {folder_id: folder_path for folder_path, folder_id in self.deployed_folders.items()}\n\n # Create a copy of the unorphaned_folders set to safely iterate while modifying the original set\n folder_lookup = unorphaned_folders.copy()\n\n # For each folder containing items, identify and protect all its ancestor folders from deletion\n for folder_id in folder_lookup:\n if folder_id in folder_id_to_path_mapping:\n # Get the folder path\n folder_path = folder_id_to_path_mapping[folder_id]\n\n # Move up the folder hierarchy and add all ancestor folders\n current_folder_path = folder_path\n while current_folder_path != \"/\":\n # Get the parent folder path\n current_folder_path = current_folder_path.rsplit(\"/\", 1)[0] or \"/\"\n\n # Get the folder_id for this path and add to the unorphaned_folder set\n parent_folder_id = self.deployed_folders.get(current_folder_path)\n if parent_folder_id:\n unorphaned_folders.add(parent_folder_id)\n\n # Check if deletion can be skipped after update to unorphaned_folder set\n if unorphaned_folders == set(sorted_folder_ids):\n return\n\n logger.info(\"Unpublishing Workspace Folders\")\n\n # Pop all folders\n\n for folder_id in sorted_folder_ids:\n if folder_id not in unorphaned_folders:\n # Folder deployed, but not in repository\n\n # Delete the folder from the workspace\n # https://learn.microsoft.com/en-us/rest/api/fabric/core/folders/delete-folder\n try:\n self.endpoint.invoke(method=\"DELETE\", url=f\"{self.base_api_url}/folders/{folder_id}\")\n logger.debug(f\"Unpublished folder: {folder_id}\")\n except Exception as e:\n logger.warning(f\"Failed to unpublish folder {folder_id}. Raw exception: {e}\")\n\n logger.info(f\"{constants.INDENT}Unpublished\")\n" }, { "path": "src/fabric_cicd/publish.py", "content": "# Copyright (c) Microsoft Corporation.\n# Licensed under the MIT License.\n\n\"\"\"Module for publishing and unpublishing Fabric workspace items.\"\"\"\n\nimport logging\nfrom typing import Optional\n\nimport dpath\nfrom azure.core.credentials import TokenCredential\n\nimport fabric_cicd._items as items\nfrom fabric_cicd import constants\nfrom fabric_cicd._common._config_utils import (\n config_overrides_scope,\n extract_publish_settings,\n extract_unpublish_settings,\n extract_workspace_settings,\n load_config_file,\n)\nfrom fabric_cicd._common._deployment_result import DeploymentResult, DeploymentStatus\nfrom fabric_cicd._common._exceptions import FailedPublishedItemStatusError, InputError\nfrom fabric_cicd._common._logging import log_header\nfrom fabric_cicd._common._validate_input import (\n validate_environment,\n validate_fabric_workspace_obj,\n validate_folder_path_exclude_regex,\n validate_folder_path_to_include,\n validate_items_to_include,\n validate_shortcut_exclude_regex,\n)\nfrom fabric_cicd.constants import FeatureFlag, ItemType\nfrom fabric_cicd.fabric_workspace import FabricWorkspace\n\nlogger = logging.getLogger(__name__)\n\n\ndef publish_all_items(\n fabric_workspace_obj: FabricWorkspace,\n item_name_exclude_regex: Optional[str] = None,\n folder_path_exclude_regex: Optional[str] = None,\n folder_path_to_include: Optional[list[str]] = None,\n items_to_include: Optional[list[str]] = None,\n shortcut_exclude_regex: Optional[str] = None,\n) -> Optional[dict]:\n \"\"\"\n Publishes all items defined in the `item_type_in_scope` list of the given FabricWorkspace object.\n\n Args:\n fabric_workspace_obj: The FabricWorkspace object containing the items to be published.\n item_name_exclude_regex: Regex pattern to exclude specific items from being published.\n folder_path_exclude_regex: Regex pattern matched against folder paths (e.g., \"/folder_name\") to exclude folders and their items from being published.\n folder_path_to_include: List of folder paths in the format \"/folder_name\"; only the specified folders and their items will be published.\n items_to_include: List of items in the format \"item_name.item_type\" that should be published.\n shortcut_exclude_regex: Regex pattern to exclude specific shortcuts from being published in lakehouses.\n\n Returns:\n Dict containing all API responses if the ``enable_response_collection`` feature flag is enabled\n and at least one response was collected; otherwise, None.\n\n folder_path_exclude_regex:\n This is an experimental feature in fabric-cicd. Use at your own risk as selective deployments are\n not recommended due to item dependencies. Cannot be used together with ``folder_path_to_include``\n for the same environment. To enable this feature, see How To -> Optional Features for information\n on which flags to enable.\n\n folder_path_to_include:\n This is an experimental feature in fabric-cicd. Use at your own risk as selective deployments are\n not recommended due to item dependencies. Cannot be used together with ``folder_path_exclude_regex``\n for the same environment. To enable this feature, see How To -> Optional Features for information\n on which flags to enable.\n\n items_to_include:\n This is an experimental feature in fabric-cicd. Use at your own risk as selective deployments are\n not recommended due to item dependencies. To enable this feature, see How To -> Optional Features\n for information on which flags to enable.\n\n shortcut_exclude_regex:\n This is an experimental feature in fabric-cicd. Use at your own risk as selective shortcut deployments\n may result in missing data dependencies. To enable this feature, see How To -> Optional Features\n for information on which flags to enable.\n\n Examples:\n Basic usage\n >>> from fabric_cicd import FabricWorkspace, publish_all_items\n >>> from azure.identity import AzureCliCredential\n >>> workspace = FabricWorkspace(\n ... workspace_id=\"your-workspace-id\",\n ... repository_directory=\"/path/to/repo\",\n ... item_type_in_scope=[\"Environment\", \"Notebook\", \"DataPipeline\"],\n ... token_credential=AzureCliCredential() # or any other TokenCredential\n ... )\n >>> publish_all_items(workspace)\n\n With regex name exclusion\n >>> from fabric_cicd import FabricWorkspace, publish_all_items\n >>> from azure.identity import AzureCliCredential\n >>> workspace = FabricWorkspace(\n ... workspace_id=\"your-workspace-id\",\n ... repository_directory=\"/path/to/repo\",\n ... item_type_in_scope=[\"Environment\", \"Notebook\", \"DataPipeline\"],\n ... token_credential=AzureCliCredential() # or any other TokenCredential\n ... )\n >>> exclude_regex = \".*_do_not_publish\"\n >>> publish_all_items(workspace, item_name_exclude_regex=exclude_regex)\n\n With folder exclusion\n >>> from fabric_cicd import FabricWorkspace, publish_all_items, append_feature_flag\n >>> from azure.identity import AzureCliCredential\n >>> append_feature_flag(\"enable_experimental_features\")\n >>> append_feature_flag(\"enable_exclude_folder\")\n >>> workspace = FabricWorkspace(\n ... workspace_id=\"your-workspace-id\",\n ... repository_directory=\"/path/to/repo\",\n ... item_type_in_scope=[\"Environment\", \"Notebook\", \"DataPipeline\"],\n ... token_credential=AzureCliCredential() # or any other TokenCredential\n ... )\n >>> folder_exclude_regex = \"^/legacy\"\n >>> publish_all_items(workspace, folder_path_exclude_regex=folder_exclude_regex)\n\n With folder inclusion\n >>> from fabric_cicd import FabricWorkspace, publish_all_items, append_feature_flag\n >>> from azure.identity import AzureCliCredential\n >>> append_feature_flag(\"enable_experimental_features\")\n >>> append_feature_flag(\"enable_include_folder\")\n >>> workspace = FabricWorkspace(\n ... workspace_id=\"your-workspace-id\",\n ... repository_directory=\"/path/to/repo\",\n ... item_type_in_scope=[\"Environment\", \"Notebook\", \"DataPipeline\"],\n ... token_credential=AzureCliCredential() # or any other TokenCredential\n ... )\n >>> folder_path_to_include = [\"/subfolder\"]\n >>> publish_all_items(workspace, folder_path_to_include=folder_path_to_include)\n\n With items to include\n >>> from fabric_cicd import FabricWorkspace, publish_all_items, append_feature_flag\n >>> from azure.identity import AzureCliCredential\n >>> append_feature_flag(\"enable_experimental_features\")\n >>> append_feature_flag(\"enable_items_to_include\")\n >>> workspace = FabricWorkspace(\n ... workspace_id=\"your-workspace-id\",\n ... repository_directory=\"/path/to/repo\",\n ... item_type_in_scope=[\"Environment\", \"Notebook\", \"DataPipeline\"],\n ... token_credential=AzureCliCredential() # or any other TokenCredential\n ... )\n >>> items_to_include = [\"Hello World.Notebook\", \"Hello.Environment\"]\n >>> publish_all_items(workspace, items_to_include=items_to_include)\n\n With shortcut exclusion\n >>> from fabric_cicd import FabricWorkspace, publish_all_items, append_feature_flag\n >>> from azure.identity import AzureCliCredential\n >>> append_feature_flag(\"enable_experimental_features\")\n >>> append_feature_flag(\"enable_shortcut_exclude\")\n >>> append_feature_flag(\"enable_shortcut_publish\")\n >>> workspace = FabricWorkspace(\n ... workspace_id=\"your-workspace-id\",\n ... repository_directory=\"/path/to/repo\",\n ... item_type_in_scope=[\"Lakehouse\"],\n ... token_credential=AzureCliCredential() # or any other TokenCredential\n ... )\n >>> shortcut_exclude_regex = \"^temp_.*\" # Exclude shortcuts starting with \"temp_\"\n >>> publish_all_items(workspace, shortcut_exclude_regex=shortcut_exclude_regex)\n\n With response collection\n >>> from fabric_cicd import FabricWorkspace, publish_all_items, append_feature_flag\n >>> from azure.identity import AzureCliCredential\n >>> append_feature_flag(\"enable_response_collection\")\n >>> workspace = FabricWorkspace(\n ... workspace_id=\"your-workspace-id\",\n ... repository_directory=\"/path/to/repo\",\n ... item_type_in_scope=[\"Environment\", \"Notebook\", \"DataPipeline\"],\n ... token_credential=AzureCliCredential() # or any other TokenCredential\n ... )\n >>> responses = publish_all_items(workspace)\n >>> # Access all responses\n >>> print(responses)\n >>> # Access individual item response (dict with \"header\", \"body\", \"status_code\" keys)\n >>> notebook_response = workspace.responses[\"Notebook\"][\"Hello World\"]\n >>> print(notebook_response[\"status_code\"]) # e.g., 200\n\n With get_changed_items (deploy only git-changed items)\n >>> from fabric_cicd import FabricWorkspace, publish_all_items, get_changed_items\n >>> from azure.identity import AzureCliCredential\n >>> workspace = FabricWorkspace(\n ... workspace_id=\"your-workspace-id\",\n ... repository_directory=\"/path/to/repo\",\n ... item_type_in_scope=[\"Notebook\", \"DataPipeline\"],\n ... token_credential=AzureCliCredential() # or any other TokenCredential\n ... )\n >>> changed = get_changed_items(workspace.repository_directory)\n >>> if changed:\n ... publish_all_items(workspace, items_to_include=changed)\n \"\"\"\n fabric_workspace_obj = validate_fabric_workspace_obj(fabric_workspace_obj)\n responses_enabled = FeatureFlag.ENABLE_RESPONSE_COLLECTION.value in constants.FEATURE_FLAG\n\n # Initialize response collection if feature flag is enabled\n if responses_enabled:\n fabric_workspace_obj.responses = {}\n\n # Check if workspace has assigned capacity, if not, exit\n has_assigned_capacity = None\n\n response_state = fabric_workspace_obj.endpoint.invoke(\n method=\"GET\", url=f\"{constants.DEFAULT_API_ROOT_URL}/v1/workspaces/{fabric_workspace_obj.workspace_id}\"\n )\n\n has_assigned_capacity = dpath.get(response_state, \"body/capacityId\", default=None)\n\n if not has_assigned_capacity and not set(fabric_workspace_obj.item_type_in_scope).issubset(\n set(constants.NO_ASSIGNED_CAPACITY_REQUIRED)\n ):\n msg = f\"Workspace {fabric_workspace_obj.workspace_id} does not have an assigned capacity. Please assign a capacity before publishing items.\"\n raise FailedPublishedItemStatusError(msg, logger)\n\n if FeatureFlag.DISABLE_WORKSPACE_FOLDER_PUBLISH.value not in constants.FEATURE_FLAG:\n if folder_path_exclude_regex is not None and folder_path_to_include is not None:\n msg = \"Cannot use both 'folder_path_exclude_regex' and 'folder_path_to_include' simultaneously. Choose one filtering strategy.\"\n raise InputError(msg, logger)\n\n if folder_path_exclude_regex is not None:\n validate_folder_path_exclude_regex(folder_path_exclude_regex)\n fabric_workspace_obj.publish_folder_path_exclude_regex = folder_path_exclude_regex\n\n if folder_path_to_include is not None:\n validate_folder_path_to_include(folder_path_to_include)\n fabric_workspace_obj.publish_folder_path_to_include = folder_path_to_include\n\n fabric_workspace_obj._refresh_deployed_folders()\n fabric_workspace_obj._refresh_repository_folders()\n fabric_workspace_obj._publish_folders()\n\n fabric_workspace_obj._refresh_deployed_items()\n fabric_workspace_obj._refresh_repository_items()\n\n if item_name_exclude_regex:\n logger.warning(\n \"Using item_name_exclude_regex is risky as it can prevent needed dependencies from being deployed. Use at your own risk.\"\n )\n fabric_workspace_obj.publish_item_name_exclude_regex = item_name_exclude_regex\n\n if items_to_include:\n validate_items_to_include(items_to_include, operation=constants.OperationType.PUBLISH)\n fabric_workspace_obj.items_to_include = items_to_include\n\n if shortcut_exclude_regex:\n validate_shortcut_exclude_regex(shortcut_exclude_regex)\n fabric_workspace_obj.shortcut_exclude_regex = shortcut_exclude_regex\n\n # Publish items in the defined order synchronously\n total_item_types = len(constants.SERIAL_ITEM_PUBLISH_ORDER)\n publishers_with_async_check: list[items.ItemPublisher] = []\n for order_num, item_type in items.ItemPublisher.get_item_types_to_publish(fabric_workspace_obj):\n log_header(logger, f\"Publishing Item {order_num}/{total_item_types}: {item_type.value}\")\n publisher = items.ItemPublisher.create(item_type, fabric_workspace_obj)\n publisher.publish_all()\n if publisher.has_async_publish_check:\n publishers_with_async_check.append(publisher)\n\n # Check asynchronous publish status for relevant item types\n for publisher in publishers_with_async_check:\n log_header(logger, f\"Checking {publisher.item_type} Publish State\")\n publisher.post_publish_all_check()\n\n # Return response data if feature flag is enabled and responses were collected\n return fabric_workspace_obj.responses if responses_enabled and fabric_workspace_obj.responses else None\n\n\ndef unpublish_all_orphan_items(\n fabric_workspace_obj: FabricWorkspace,\n item_name_exclude_regex: str = \"^$\",\n items_to_include: Optional[list[str]] = None,\n) -> Optional[dict]:\n \"\"\"\n Unpublishes all orphaned items not present in the repository except for those matching the exclude regex.\n\n Args:\n fabric_workspace_obj: The FabricWorkspace object containing the items to be unpublished.\n item_name_exclude_regex: Regex pattern to exclude specific items from being unpublished. Default is '^$' which will exclude nothing.\n items_to_include: List of items in the format \"item_name.item_type\" that should be unpublished.\n\n Returns:\n Dict containing all collected API responses if the ``enable_response_collection`` feature flag is enabled\n and at least one response was collected; otherwise, None.\n\n Note:\n By default, the Fabric Delete Item API moves deleted items to the workspace recycle bin.\n However, not all item types support soft delete; for those types, deletion requires the\n ``enable_hard_delete`` feature flag. Enabling this flag bypasses the recycle bin and\n permanently deletes items. Hard delete requires the workspace **Admin** role.\n\n items_to_include:\n This is an experimental feature in fabric-cicd. Use at your own risk as selective unpublishing is not recommended due to item dependencies.\n To enable this feature, see How To -> Optional Features for information on which flags to enable.\n\n Examples:\n Basic usage\n >>> from fabric_cicd import FabricWorkspace, publish_all_items, unpublish_all_orphan_items\n >>> from azure.identity import AzureCliCredential\n >>> workspace = FabricWorkspace(\n ... workspace_id=\"your-workspace-id\",\n ... repository_directory=\"/path/to/repo\",\n ... item_type_in_scope=[\"Environment\", \"Notebook\", \"DataPipeline\"],\n ... token_credential=AzureCliCredential() # or any other TokenCredential\n ... )\n >>> publish_all_items(workspace)\n >>> unpublish_all_orphan_items(workspace)\n\n With regex name exclusion\n >>> from fabric_cicd import FabricWorkspace, publish_all_items, unpublish_all_orphan_items\n >>> from azure.identity import AzureCliCredential\n >>> workspace = FabricWorkspace(\n ... workspace_id=\"your-workspace-id\",\n ... repository_directory=\"/path/to/repo\",\n ... item_type_in_scope=[\"Environment\", \"Notebook\", \"DataPipeline\"],\n ... token_credential=AzureCliCredential()\n ... )\n >>> publish_all_items(workspace)\n >>> exclude_regex = \".*_do_not_delete\"\n >>> unpublish_all_orphan_items(workspace, item_name_exclude_regex=exclude_regex)\n\n With items to include\n >>> from fabric_cicd import FabricWorkspace, publish_all_items, unpublish_all_orphan_items, append_feature_flag\n >>> from azure.identity import AzureCliCredential\n >>> append_feature_flag(\"enable_experimental_features\")\n >>> append_feature_flag(\"enable_items_to_include\")\n >>> workspace = FabricWorkspace(\n ... workspace_id=\"your-workspace-id\",\n ... repository_directory=\"/path/to/repo\",\n ... item_type_in_scope=[\"Environment\", \"Notebook\", \"DataPipeline\"],\n ... token_credential=AzureCliCredential() # or any other TokenCredential\n ... )\n >>> publish_all_items(workspace)\n >>> items_to_include = [\"Hello World.Notebook\", \"Run Hello World.DataPipeline\"]\n >>> unpublish_all_orphan_items(workspace, items_to_include=items_to_include)\n\n With response collection\n >>> from fabric_cicd import FabricWorkspace, publish_all_items, unpublish_all_orphan_items, append_feature_flag\n >>> from azure.identity import AzureCliCredential\n >>> append_feature_flag(\"enable_response_collection\")\n >>> workspace = FabricWorkspace(\n ... workspace_id=\"your-workspace-id\",\n ... repository_directory=\"/path/to/repo\",\n ... item_type_in_scope=[\"Environment\", \"Notebook\", \"DataPipeline\"],\n ... token_credential=AzureCliCredential() # or any other TokenCredential\n ... )\n >>> publish_all_items(workspace)\n >>> responses = unpublish_all_orphan_items(workspace)\n >>> # Access all unpublish responses\n >>> print(responses)\n >>> # Access individual item response (dict with \"header\", \"body\", \"status_code\" keys)\n >>> notebook_response = workspace.unpublish_responses[\"Notebook\"][\"Hello World\"]\n >>> print(notebook_response[\"status_code\"]) # e.g., 200\n \"\"\"\n fabric_workspace_obj = validate_fabric_workspace_obj(fabric_workspace_obj)\n\n validate_items_to_include(items_to_include, operation=constants.OperationType.UNPUBLISH)\n\n responses_enabled = FeatureFlag.ENABLE_RESPONSE_COLLECTION.value in constants.FEATURE_FLAG\n\n # Initialize response collection if feature flag is enabled\n if responses_enabled:\n fabric_workspace_obj.unpublish_responses = {}\n\n fabric_workspace_obj._refresh_deployed_items()\n fabric_workspace_obj._refresh_repository_items()\n log_header(logger, \"Unpublishing Orphaned Items\")\n\n # Build unpublish order based on reversed publish order, scope, and feature flags\n for item_type in items.ItemPublisher.get_item_types_to_unpublish(fabric_workspace_obj):\n to_delete_list = items.ItemPublisher.get_orphaned_items(\n fabric_workspace_obj,\n item_type,\n item_name_exclude_regex=item_name_exclude_regex if not items_to_include else None,\n items_to_include=items_to_include,\n )\n\n if items_to_include and to_delete_list:\n logger.debug(f\"Items to include for unpublishing ({item_type}): {to_delete_list}\")\n\n publisher = items.ItemPublisher.create(ItemType(item_type), fabric_workspace_obj)\n if to_delete_list and publisher.has_dependency_tracking:\n to_delete_list = publisher.get_unpublish_order(to_delete_list)\n\n for item_name in to_delete_list:\n fabric_workspace_obj._unpublish_item(item_name=item_name, item_type=item_type)\n\n fabric_workspace_obj._refresh_deployed_items()\n fabric_workspace_obj._refresh_deployed_folders()\n if FeatureFlag.DISABLE_WORKSPACE_FOLDER_PUBLISH.value not in constants.FEATURE_FLAG:\n fabric_workspace_obj._unpublish_folders()\n\n # Return response data if feature flag is enabled and responses were collected\n return (\n fabric_workspace_obj.unpublish_responses\n if responses_enabled and fabric_workspace_obj.unpublish_responses\n else None\n )\n\n\ndef deploy_with_config(\n config_file_path: str,\n *,\n token_credential: TokenCredential,\n environment: str = \"N/A\",\n config_override: Optional[dict] = None,\n) -> DeploymentResult:\n \"\"\"\n Deploy items using YAML configuration file with environment-specific settings.\n This function provides a simplified deployment interface that loads configuration\n from a YAML file and executes deployment operations based on environment-specific\n settings. It constructs the necessary FabricWorkspace object internally\n and handles publish/unpublish operations according to the configuration.\n\n Args:\n config_file_path: Path to the YAML configuration file as a string.\n token_credential: Azure token credential for authentication (e.g., AzureCliCredential, ClientSecretCredential) - required.\n environment: Environment name to use for deployment (e.g., 'dev', 'test', 'prod'), if missing defaults to 'N/A'.\n config_override: Optional dictionary to override specific configuration values.\n\n Returns:\n DeploymentResult: A result object containing the deployment status, message, and\n responses (opt-in). The status will be DeploymentStatus.COMPLETED on success.\n The responses field contains a dictionary with ``\"publish\"`` and/or ``\"unpublish\"``\n keys mapping to their respective API response data when the\n ``enable_response_collection`` feature flag is enabled and responses were collected,\n otherwise None.\n\n Raises:\n InputError: If configuration is invalid, environment not found, or input validation fails.\n ConfigValidationError: If configuration file is missing or fails structural validation.\n\n Note:\n On failure, the raised exception will have a ``deployment_result`` attribute\n containing a ``DeploymentResult`` with ``status`` set to\n ``DeploymentStatus.FAILED``, ``message`` set to the error description, and\n ``responses`` containing any partial API responses collected before the failure\n (requires the ``enable_response_collection`` feature flag, otherwise None).\n\n Examples:\n Basic usage\n >>> from fabric_cicd import deploy_with_config\n >>> from azure.identity import AzureCliCredential\n >>> credential = AzureCliCredential()\n >>> result = deploy_with_config(\n ... config_file_path=\"workspace/config.yml\",\n ... token_credential=credential,\n ... environment=\"prod\"\n ... )\n >>> print(result.status) # DeploymentStatus.COMPLETED\n >>> print(result.message) # \"Deployment completed successfully\"\n >>> print(result.responses) # {\"publish\": {...}, \"unpublish\": {...}} or None\n\n With custom authentication\n >>> from fabric_cicd import deploy_with_config\n >>> from azure.identity import ClientSecretCredential\n >>> credential = ClientSecretCredential(tenant_id, client_id, client_secret)\n >>> result = deploy_with_config(\n ... config_file_path=\"workspace/config.yml\",\n ... token_credential=credential,\n ... environment=\"prod\"\n ... )\n\n With override configuration\n >>> from fabric_cicd import deploy_with_config\n >>> from azure.identity import ClientSecretCredential\n >>> credential = ClientSecretCredential(tenant_id, client_id, client_secret)\n >>> result = deploy_with_config(\n ... config_file_path=\"workspace/config.yml\",\n ... token_credential=credential,\n ... environment=\"prod\",\n ... config_override={\n ... \"core\": {\n ... \"item_types_in_scope\": [\"Notebook\"]\n ... },\n ... \"publish\": {\n ... \"skip\": {\n ... \"prod\": False\n ... }\n ... }\n ... }\n ... )\n\n Handling deployment failures\n >>> from fabric_cicd import deploy_with_config\n >>> from azure.identity import AzureCliCredential\n >>> credential = AzureCliCredential()\n >>> try:\n ... result = deploy_with_config(\n ... config_file_path=\"workspace/config.yml\",\n ... token_credential=credential,\n ... environment=\"prod\"\n ... )\n ... print(result.status) # DeploymentStatus.COMPLETED\n ... print(result.message) # \"Deployment completed successfully\"\n ... print(result.responses) # {\"publish\": {...}, \"unpublish\": {...}} or None\n ... except Exception as e:\n ... print(e.deployment_result.status) # DeploymentStatus.FAILED\n ... print(e.deployment_result.message) # Original error message\n ... print(e.deployment_result.responses) # Partial API responses or None\n \"\"\"\n log_header(logger, \"Config-Based Deployment\")\n logger.info(f\"Loading configuration from {config_file_path} for environment '{environment}'\")\n\n # Initialize workspace as None so it exists in except block scope\n workspace = None\n responses_enabled = False\n\n try:\n # Validate environment\n environment = validate_environment(environment)\n\n # Load and validate configuration file\n config = load_config_file(config_file_path, environment, config_override)\n\n # Extract environment-specific settings\n workspace_settings = extract_workspace_settings(config, environment)\n publish_settings = extract_publish_settings(config, environment)\n unpublish_settings = extract_unpublish_settings(config, environment)\n\n # Apply feature flags and constants if specified\n with config_overrides_scope(config, environment):\n # Determine if response collection flag has been enabled in the config file\n responses_enabled = FeatureFlag.ENABLE_RESPONSE_COLLECTION.value in constants.FEATURE_FLAG\n\n # When no parameter file is configured or resolved for this environment,\n # parameter_file_path is None and parameterization must be skipped entirely —\n # any parameter.yml that happens to exist in the repository must NOT be auto-discovered.\n skip_parameterization = workspace_settings.get(\"parameter_file_path\") is None\n\n # Create FabricWorkspace object with extracted settings\n workspace = FabricWorkspace(\n repository_directory=workspace_settings[\"repository_directory\"],\n item_type_in_scope=workspace_settings.get(\"item_types_in_scope\"),\n environment=environment,\n workspace_id=workspace_settings.get(\"workspace_id\"),\n workspace_name=workspace_settings.get(\"workspace_name\"),\n token_credential=token_credential,\n parameter_file_path=workspace_settings.get(\"parameter_file_path\"),\n skip_parameterization=skip_parameterization,\n )\n # Execute deployment operations based on skip settings\n if not publish_settings.get(\"skip\", False):\n publish_all_items(\n workspace,\n item_name_exclude_regex=publish_settings.get(\"exclude_regex\"),\n folder_path_exclude_regex=publish_settings.get(\"folder_exclude_regex\"),\n folder_path_to_include=publish_settings.get(\"folder_path_to_include\"),\n items_to_include=publish_settings.get(\"items_to_include\"),\n shortcut_exclude_regex=publish_settings.get(\"shortcut_exclude_regex\"),\n )\n else:\n logger.info(f\"Skipping publish operation for environment '{environment}'\")\n\n if not unpublish_settings.get(\"skip\", False):\n unpublish_all_orphan_items(\n workspace,\n item_name_exclude_regex=unpublish_settings.get(\"exclude_regex\", \"^$\"),\n items_to_include=unpublish_settings.get(\"items_to_include\"),\n )\n else:\n logger.info(f\"Skipping unpublish operation for environment '{environment}'\")\n\n except Exception as e:\n e.deployment_result = DeploymentResult(\n status=DeploymentStatus.FAILED,\n message=str(e),\n responses=_collect_responses(workspace, responses_enabled),\n )\n raise\n\n logger.info(\"Config-based deployment completed successfully\")\n return DeploymentResult(\n status=DeploymentStatus.COMPLETED,\n message=\"Deployment completed successfully\",\n responses=_collect_responses(workspace, responses_enabled),\n )\n\n\ndef _collect_responses(workspace: Optional[FabricWorkspace], responses_enabled: bool) -> Optional[dict]:\n \"\"\"Return collected API responses if available, otherwise None.\"\"\"\n if not responses_enabled or workspace is None:\n return None\n result = {}\n if workspace.responses:\n result[\"publish\"] = workspace.responses\n if workspace.unpublish_responses:\n result[\"unpublish\"] = workspace.unpublish_responses\n return result or None\n" }, { "path": "tests/fixtures/.gitignore", "content": "# Only the gzipped http trace should be committed to git\n# Snapshots are not human reviewable in a PR, it's too large.\nhttp_trace.json\n" }, { "path": "tests/fixtures/README.md", "content": "# Test Fixtures\n\n## `mock_fabric_server`: A mock Fabric REST API\n\n### Use Cases\n\nBasically, this fixture allows Integration Testing without the costs associated with E2E tests.\n\nIf you peek inside `http_trace.json`, each full deployment of `fabric-cicd` project makes thousands of API calls.\nAs the project grows in scope, so does the amount of tests that need to be written to exercise coverage. This becomes\nunruly via PyTest mocks or [monkey patching](https://docs.pytest.org/en/stable/how-to/monkeypatch.html).\n\nIn an ideal scenario, whenver a PR is opened, we'd test it against a real Fabric Workspace, but - stateful tests are \ndifficult, any outages in Fabric API can cause PRs to fail - etc; this means testing against a real workspace is not\nrealistic.\n\nBut what if we could _mimic_ the Fabric API locally?\n\nThat's what this fixture provides.\n\nThis is a mock REST API Server that can mimic any REST API, including `https://api.powerbi.com` and `https://api.fabric.microsoft.com`. \nWe can add any number of further mocks in the future as well - as long as the API calls are traced and snapshotted.\n\nThe idea is, to exercise the public facing `fabric_cicd` API E2E rapidly.\nThe mock server loads an `http_trace.json` file to dictate the behavior.\n\n### Why not VCR\n\nThis is very similar to [VCR Cassettes](https://github.com/vcr/vcr). The downside of VCR is, the actual HTTP interactions are\nabstracted away via the SDK, and as a result, the test code does not truly make calls to a real REST API that we have full control over.\n\nFor example, in our mock REST API, we can add logic to override exceptions at a route level, which is not possible in VCR (without opening a PR).\n\n> In both VCR and this fixture, the one thing is common, generating snapshots takes time, since you have to interact\n> with the real control plane (in this case Fabric), which is costly.\n>\n> The way to think about the situation is, we're able to enjoy near 100% test coverage for _all_ of our customer facing calls\n> without using a real control plane, this allows us to _significantly_ increase code velocity as all code paths are tested by\n> nature of a full deployment.\n\n### What is this?\n\nThe 4 steps outlined in the image below are as follows:\n\n1. Add new workloads into the codebase\n2. Capture REST calls from Fabric using `debug_trace_deployment.py`\n3. Move `http_trace.json.gz` into fixture\n4. Enjoy rapid test coverage!\n\n![Test Harness](.imgs/test-harness.png)\n\n### Capturing HTTP Trace for new Fabric API calls\n\nSuppose you need to add payloads for a new fabric item type or an API call.\n\nThe following script creates an HTTP snapshot that is stored in `http_trace.json.gz`, which is moved into `fabric-cicd/tests/fixtures`\n\nUpdate `item_type_in_scope` in the script with the item type you want to capture HTTP traffic for, then run:\n\n```bash\nexport FABRIC_WORKSPACE_ID=\"your-fabric-workspace-guid\"\nuv run python devtools/debug_trace_deployment.py\ncp -f http_trace.json.gz tests/fixtures/http_trace.json.gz\n```\n\nYou can validate the integration test works with the mock server with:\n\n```bash\nuv run pytest -v -s --log-cli-level=INFO tests/test_integration_publish.py::test_publish_all_items_integration\n```\n\n### Important Notes\n\n* The `http_trace.json` must be generated in one shot, i.e. the Mock Server is not guaranteed to incrementally process new lines added to `http_trace.json`.\n What that means is - you should capture as many items as possible in `debug_trace_deployment.py`, and use that payload in the tests.\n\n### Troubleshooting\n\n* If a Unit Test fails on your branch but it wasn't failing on a previous branch, that means your branch contains new changes that have not\n been snapshotted. If you rever your branch to `main`, the tests should run green.\n \n Therefore, due to the logic/API call additions in your PR, you must regenerate the snapshot using the steps outlined above.\n\n* Copilot generated PRs that change API signatures will probably need human intervention to generate snapshots, as Copilot cannot make\n REST API calls to a real Fabric instance to regenerate the snapshot. In this situation, pull the Copilot generated PR locally and rerun\n the snapshot generation.\n" }, { "path": "tests/fixtures/__init__.py", "content": "# Copyright (c) Microsoft Corporation.\n# Licensed under the MIT License.\n\n\"\"\"Test fixtures and utilities.\"\"\"\n\nfrom fixtures.credentials import DummyTokenCredential, create_dummy_jwt\nfrom fixtures.mock_fabric_server import MOCK_SERVER_PORT, MockFabricServer\n\n__all__ = [\n \"MOCK_SERVER_PORT\",\n \"DummyTokenCredential\",\n \"MockFabricServer\",\n \"create_dummy_jwt\",\n]\n" }, { "path": "tests/fixtures/credentials.py", "content": "# Copyright (c) Microsoft Corporation.\n# Licensed under the MIT License.\n\n\"\"\"Test credentials and authentication utilities.\"\"\"\n\nimport base64\nimport json\nimport logging\nfrom datetime import datetime, timedelta, timezone\nfrom typing import Any\n\nfrom azure.core.credentials import AccessToken, TokenCredential\n\n\ndef create_dummy_jwt(expiry_timestamp: int) -> str:\n \"\"\"\n Create a dummy JWT token for testing.\n\n Args:\n expiry_timestamp: Unix timestamp for token expiry\n\n Returns:\n A properly formatted JWT token string\n \"\"\"\n header = {\"alg\": \"HS256\", \"typ\": \"JWT\"}\n payload = {\n \"exp\": expiry_timestamp,\n \"upn\": \"test@example.com\",\n \"aud\": \"https://api.fabric.microsoft.com\",\n }\n\n header_b64 = base64.urlsafe_b64encode(json.dumps(header).encode()).decode().rstrip(\"=\")\n payload_b64 = base64.urlsafe_b64encode(json.dumps(payload).encode()).decode().rstrip(\"=\")\n signature = \"dummy_signature\"\n\n return f\"{header_b64}.{payload_b64}.{signature}\"\n\n\nclass DummyTokenCredential(TokenCredential):\n \"\"\"A static token credential for testing.\"\"\"\n\n def __init__(self, expiry_days: int = 365):\n \"\"\"\n Initialize static token credential.\n\n Args:\n expiry_days: Number of days until expiry. Defaults to 365.\n \"\"\"\n self.expiry = int((datetime.now(timezone.utc) + timedelta(days=expiry_days)).timestamp())\n self._token = create_dummy_jwt(self.expiry)\n self.logger = logging.getLogger(__name__)\n\n def get_token(self, *scopes: str, **kwargs: Any) -> AccessToken: # noqa: ARG002\n \"\"\"Get the static access token.\"\"\"\n self.logger.debug(f\"Static token credential - getting token for scopes: {scopes}\")\n return AccessToken(self._token, self.expiry)\n\n def get_expire(self) -> int:\n \"\"\"Get the token expiry timestamp.\"\"\"\n return self.expiry\n" }, { "path": "tests/fixtures/mock_fabric_server.py", "content": "# Copyright (c) Microsoft Corporation.\n# Licensed under the MIT License.\n\n\"\"\"\nStateful Mock Fabric REST API server for integration testing.\n\nProvides transactionally correct responses via:\n- Content-based matching for POST/PATCH (by displayName + type)\n- Operation ID correlation for async 202 responses\n- State machine for long-running operations (Running -> Succeeded)\n- Generic fallback responses for unknown mutation routes\n\"\"\"\n\nimport json\nimport logging\nimport re\nimport threading\nfrom collections import defaultdict\nfrom http.server import BaseHTTPRequestHandler, HTTPServer\nfrom pathlib import Path\nfrom typing import Any, ClassVar, Optional\nfrom urllib.parse import urlparse\n\nfrom fabric_cicd._common._http_tracer import HTTPRequest, HTTPResponse\n\nlogger = logging.getLogger(__name__)\n\nMOCK_SERVER_PORT = 8765\nGUID_PATTERN = re.compile(r\"[a-fA-F0-9]{8}-[a-fA-F0-9]{4}-[a-fA-F0-9]{4}-[a-fA-F0-9]{4}-[a-fA-F0-9]{12}\")\nOPERATION_PATTERN = re.compile(r\"/v1/operations/([a-fA-F0-9-]+)(/result)?$\")\nSKIP_HEADERS = {\n \"content-length\",\n \"content-encoding\",\n \"transfer-encoding\",\n \"server\",\n \"date\",\n \"home-cluster-uri\",\n \"request-redirected\",\n}\n\n\nclass TraceIndex:\n \"\"\"\n Multi-index for trace lookup: by normalized route, by content (displayName, type),\n and by operation ID for async operation correlation.\n \"\"\"\n\n def __init__(self):\n self.by_route: dict[str, list[tuple[HTTPRequest, HTTPResponse]]] = defaultdict(list)\n self.by_content: dict[tuple[str, str, str], list[tuple[HTTPRequest, HTTPResponse]]] = defaultdict(list)\n self.by_operation: dict[str, dict[str, Any]] = defaultdict(lambda: {\"post\": None, \"poll\": [], \"result\": None})\n\n @staticmethod\n def normalize_route(route: str) -> str:\n \"\"\"Replace all GUIDs with {GUID} placeholder.\"\"\"\n return GUID_PATTERN.sub(\"{GUID}\", route)\n\n @staticmethod\n def extract_content_key(body: Any, method: str) -> Optional[tuple[str, str, str]]:\n \"\"\"Extract (displayName, type, method) from request body if present.\"\"\"\n if isinstance(body, dict) and body.get(\"displayName\") and body.get(\"type\"):\n return (body[\"displayName\"], body[\"type\"], method)\n return None\n\n def add_trace(self, request: HTTPRequest, response: HTTPResponse):\n \"\"\"Index a trace by route, content, and operation ID.\"\"\"\n parsed = urlparse(request.url)\n route = parsed.path + (f\"?{parsed.query}\" if parsed.query else \"\")\n normalized_key = f\"{request.method} {self.normalize_route(route)}\"\n\n self.by_route[normalized_key].append((request, response))\n\n if (\n request.method in (\"POST\", \"PATCH\")\n and request.body\n and (content_key := self.extract_content_key(request.body, request.method))\n ):\n self.by_content[content_key].append((request, response))\n\n if (op_id := response.headers.get(\"x-ms-operation-id\")) and response.status_code == 202:\n self.by_operation[op_id][\"post\"] = (request, response)\n\n if op_match := OPERATION_PATTERN.search(route):\n op_id, is_result = op_match.group(1), op_match.group(2) == \"/result\"\n if is_result:\n self.by_operation[op_id][\"result\"] = (request, response)\n else:\n self.by_operation[op_id][\"poll\"].append((request, response))\n\n\nclass MockFabricAPIHandler(BaseHTTPRequestHandler):\n \"\"\"HTTP handler with stateful matching: content-based for items, state machine for operations.\"\"\"\n\n trace_index: ClassVar[TraceIndex] = TraceIndex()\n route_lock: ClassVar[threading.Lock] = threading.Lock()\n operation_poll_counts: ClassVar[dict[str, int]] = {}\n content_to_operation: ClassVar[dict[tuple[str, str], str]] = {}\n\n def log_message(self, format, *args): # noqa: A002\n pass\n\n def do_GET(self): # noqa: N802\n self._handle_request(\"GET\")\n\n def do_POST(self): # noqa: N802\n self._handle_request(\"POST\")\n\n def do_PATCH(self): # noqa: N802\n self._handle_request(\"PATCH\")\n\n def do_DELETE(self): # noqa: N802\n self._handle_request(\"DELETE\")\n\n def _read_request_body(self) -> Optional[dict]:\n \"\"\"Read and parse JSON request body.\"\"\"\n if content_length := self.headers.get(\"Content-Length\"):\n try:\n return json.loads(self.rfile.read(int(content_length)).decode(\"utf-8\"))\n except (json.JSONDecodeError, UnicodeDecodeError):\n pass\n return None\n\n def _handle_request(self, method: str):\n \"\"\"Route request to appropriate handler, with fallback for unknown routes.\"\"\"\n route_key = f\"{method} {self.path}\"\n logger.info(f\"Mock server received: {route_key}\")\n\n request_body = self._read_request_body() if method in (\"POST\", \"PATCH\") else None\n response = self._find_matching_response(method, self.path, request_body)\n\n if response is None:\n if method in (\"PATCH\", \"DELETE\"):\n logger.info(f\"No trace for {route_key}, returning 200 OK\")\n response = HTTPResponse(\n status_code=200, headers={\"Content-Type\": \"application/json\"}, body={}, timestamp=None\n )\n elif method == \"POST\":\n logger.info(f\"No trace for {route_key}, returning 202 Accepted\")\n response = HTTPResponse(\n status_code=202, headers={\"Content-Type\": \"application/json\"}, body={}, timestamp=None\n )\n else:\n logger.warning(f\"No trace data for {route_key}\")\n self.send_error(404, f\"No trace data found for {route_key}\")\n return\n\n self._send_response(response, route_key)\n\n def _find_matching_response(self, method: str, route: str, request_body: Optional[dict]) -> Optional[HTTPResponse]:\n \"\"\"Find response by: 1) operation polling, 2) content match, 3) route match.\"\"\"\n normalized_key = f\"{method} {self.trace_index.normalize_route(route)}\"\n\n if op_match := OPERATION_PATTERN.search(route):\n return self._handle_operation_request(op_match.group(1), op_match.group(2) == \"/result\")\n\n if (\n method == \"POST\"\n and route.endswith(\"/items\")\n and request_body\n and (content_key := self.trace_index.extract_content_key(request_body, method))\n ):\n return self._handle_item_creation(content_key)\n\n if (\n method == \"POST\"\n and \"updateDefinition\" in route\n and (traces := self.trace_index.by_route.get(normalized_key))\n ):\n return traces[-1][1]\n\n if traces := self.trace_index.by_route.get(normalized_key):\n return traces[-1][1]\n\n return None\n\n def _handle_item_creation(self, content_key: tuple[str, str, str]) -> Optional[HTTPResponse]:\n \"\"\"Match POST /items by (displayName, type), track operation for async responses.\"\"\"\n display_name, item_type, _ = content_key\n traces = self.trace_index.by_content.get(content_key, [])\n\n if not traces:\n logger.warning(f\"No trace for item creation: {display_name} ({item_type})\")\n return None\n\n for _, response in traces:\n if response.status_code in (200, 201, 202):\n if response.status_code == 202 and (op_id := response.headers.get(\"x-ms-operation-id\")):\n with self.route_lock:\n self.content_to_operation[(display_name, item_type)] = op_id\n self.operation_poll_counts[op_id] = 0\n logger.info(f\"Tracking operation {op_id} for {display_name} ({item_type})\")\n logger.info(f\"Matched item creation: {display_name} ({item_type}) -> {response.status_code}\")\n return response\n\n return traces[-1][1]\n\n def _handle_operation_request(self, operation_id: str, is_result: bool) -> Optional[HTTPResponse]:\n \"\"\"\n State machine for operation polling: first poll returns Running, subsequent return Succeeded.\n Result endpoint returns the created item details.\n \"\"\"\n op_data = self.trace_index.by_operation.get(operation_id)\n if not op_data:\n for _, known_data in self.trace_index.by_operation.items():\n if known_data[\"post\"] or known_data[\"result\"]:\n op_data = known_data\n break\n\n if not op_data:\n logger.warning(f\"No operation data for {operation_id}\")\n return None\n\n if is_result:\n if op_data[\"result\"]:\n logger.info(f\"Returning operation result for {operation_id}\")\n return op_data[\"result\"][1]\n return None\n\n with self.route_lock:\n poll_count = self.operation_poll_counts.get(operation_id, 0)\n self.operation_poll_counts[operation_id] = poll_count + 1\n\n poll_traces = op_data.get(\"poll\", [])\n if not poll_traces:\n logger.warning(f\"No poll traces for operation {operation_id}\")\n return None\n\n target_status = \"Running\" if poll_count == 0 else \"Succeeded\"\n for _, response in poll_traces:\n if isinstance(response.body, dict) and response.body.get(\"status\") == target_status:\n logger.info(f\"Operation {operation_id} poll #{poll_count}: {target_status}\")\n return response\n\n return poll_traces[-1][1]\n\n def _send_response(self, response: HTTPResponse, route_key: str):\n \"\"\"Send HTTP response, rewriting Location headers for operations.\"\"\"\n self.send_response(response.status_code)\n\n body_bytes = json.dumps(response.body if isinstance(response.body, dict) else {}).encode()\n if isinstance(response.body, str) and response.body:\n body_bytes = response.body.encode()\n\n for name, value in response.headers.items():\n lower = name.lower()\n if lower in (\"x-ms-operation-id\", \"retry-after\"):\n self.send_header(name, value)\n elif lower == \"location\" and \"operations\" in value and (op_match := OPERATION_PATTERN.search(value)):\n suffix = \"/result\" if op_match.group(2) == \"/result\" else \"\"\n self.send_header(\n \"Location\", f\"http://127.0.0.1:{MOCK_SERVER_PORT}/v1/operations/{op_match.group(1)}{suffix}\"\n )\n elif lower not in SKIP_HEADERS:\n self.send_header(name, value)\n\n self.send_header(\"Content-Length\", len(body_bytes))\n self.end_headers()\n self.wfile.write(body_bytes)\n logger.debug(f\"Responded to {route_key}: {response.status_code}\")\n\n @classmethod\n def load_trace_data(cls, trace_file: Path):\n \"\"\"Load trace data from JSON file and build indices.\"\"\"\n cls.trace_index = TraceIndex()\n cls.operation_poll_counts.clear()\n cls.content_to_operation.clear()\n\n with trace_file.open(\"r\") as f:\n data = json.load(f)\n\n loaded = 0\n for trace in data.get(\"traces\", []):\n try:\n req, resp = trace.get(\"request\"), trace.get(\"response\")\n if not req or not resp:\n continue\n cls.trace_index.add_trace(\n HTTPRequest(\n req.get(\"method\", \"\"),\n req.get(\"url\", \"\"),\n req.get(\"headers\", {}),\n req.get(\"body\"),\n req.get(\"timestamp\"),\n ),\n HTTPResponse(\n resp.get(\"status_code\", 200), resp.get(\"headers\", {}), resp.get(\"body\"), resp.get(\"timestamp\")\n ),\n )\n loaded += 1\n except Exception as e:\n logger.warning(f\"Failed to parse trace: {e}\")\n\n logger.info(\n f\"Loaded {loaded} traces (routes={len(cls.trace_index.by_route)}, content={len(cls.trace_index.by_content)}, ops={len(cls.trace_index.by_operation)})\"\n )\n\n\nclass MockFabricServer:\n \"\"\"Mock Fabric API server for testing.\"\"\"\n\n HTTP_TRACE_FILE = \"http_trace.json.gz\"\n\n def __init__(self, trace_file: Path, port: int = MOCK_SERVER_PORT):\n self.port = port\n self.trace_file = trace_file\n self.server: Optional[HTTPServer] = None\n self.server_thread: Optional[threading.Thread] = None\n\n def start(self):\n \"\"\"Start the mock server in a background thread.\"\"\"\n MockFabricAPIHandler.load_trace_data(self.trace_file)\n self.server = HTTPServer((\"127.0.0.1\", self.port), MockFabricAPIHandler)\n self.server_thread = threading.Thread(target=self.server.serve_forever, daemon=True)\n self.server_thread.start()\n logger.info(f\"Mock Fabric API server started on http://127.0.0.1:{self.port}\")\n\n def stop(self):\n \"\"\"Stop the mock server.\"\"\"\n if self.server:\n self.server.shutdown()\n self.server.server_close()\n if self.server_thread:\n self.server_thread.join(timeout=5)\n logger.info(\"Mock Fabric API server stopped\")\n" }, { "path": "tests/test__check_utils.py", "content": "# Copyright (c) Microsoft Corporation.\n# Licensed under the MIT License.\n\nimport json\n\nimport pytest\n\nfrom fabric_cicd._common._check_utils import check_file_type, check_valid_json_content, check_valid_yaml_content\n\n\n@pytest.fixture\ndef text_file(tmp_path):\n file_path = tmp_path / \"test.txt\"\n file_path.write_text(\"sample text\")\n return file_path\n\n\n@pytest.fixture\ndef binary_file(tmp_path):\n file_path = tmp_path / \"test.bin\"\n file_path.write_bytes(\n b\"PK\\x03\\x04\\x14\\x00\\x00\\x00\\x08\\x00\\x00\\x00!\\x00\\xb7\\xac\\xce\\x02\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\"\n )\n return file_path\n\n\n@pytest.fixture\ndef image_file(tmp_path):\n file_path = tmp_path / \"test.png\"\n file_path.write_bytes(b\"\\x89PNG\\r\\n\\x1a\\n\\x00\\x00\\x00\\rIHDR\\x00\\x00\\x00\\x01\")\n return file_path\n\n\ndef test_check_file_type_text(text_file):\n assert check_file_type(text_file) == \"text\"\n\n\ndef test_check_file_type_binary(binary_file):\n assert check_file_type(binary_file) == \"binary\"\n\n\ndef test_check_file_type_image(image_file):\n assert check_file_type(image_file) == \"image\"\n\n\n@pytest.fixture\ndef real_schedules_file(tmp_path):\n \"\"\"Create a realistic .schedules file with exact structure like fabric-cicd uses.\"\"\"\n # Create a DataPipeline directory structure\n pipeline_dir = tmp_path / \"Test Pipeline.DataPipeline\"\n pipeline_dir.mkdir()\n\n schedules_file = pipeline_dir / \".schedules\"\n schedules_content = {\n \"schedules\": [\n {\n \"jobType\": \"Execute\",\n \"enabled\": True,\n \"cronExpression\": \"0 0 12 * * ?\",\n \"timeZone\": \"UTC\",\n \"description\": \"Daily execution at noon\",\n },\n {\n \"jobType\": \"Refresh\",\n \"enabled\": False,\n \"cronExpression\": \"0 0 6 * * ?\",\n \"timeZone\": \"UTC\",\n \"description\": \"Morning refresh\",\n },\n ]\n }\n schedules_file.write_text(json.dumps(schedules_content, indent=2))\n return schedules_file\n\n\ndef test_schedules_file_json_validation_and_structure(real_schedules_file):\n \"\"\"Test that .schedules files are properly validated and contain expected structure.\"\"\"\n # Test that check_valid_json_content correctly identifies .schedules content as valid JSON\n content = real_schedules_file.read_text(encoding=\"utf-8\")\n assert check_valid_json_content(content) is True\n\n # Verify the file has the expected structure for key_value_replace\n data = json.loads(content)\n\n # Verify the structure matches what the JSONPath expression expects\n assert \"schedules\" in data\n assert isinstance(data[\"schedules\"], list)\n assert len(data[\"schedules\"]) >= 1\n\n # Find Execute job and verify it has enabled field\n execute_jobs = [schedule for schedule in data[\"schedules\"] if schedule.get(\"jobType\") == \"Execute\"]\n assert len(execute_jobs) >= 1\n\n execute_job = execute_jobs[0]\n assert \"enabled\" in execute_job\n assert isinstance(execute_job[\"enabled\"], bool)\n\n # Verify file path ends with .schedules\n assert real_schedules_file.name == \".schedules\"\n assert str(real_schedules_file).endswith(\".schedules\")\n\n\ndef test_schedules_file_jsonpath_compatibility(real_schedules_file):\n \"\"\"Test that .schedules files work with the specific JSONPath expression used in parameter.yml.\"\"\"\n try:\n from jsonpath_ng.ext import parse\n except ImportError:\n pytest.skip(\"jsonpath_ng not available for testing\")\n\n # Read and parse the .schedules file\n content = real_schedules_file.read_text(encoding=\"utf-8\")\n data = json.loads(content)\n\n # Test the exact JSONPath expression from the parameter.yml\n jsonpath_expr = parse('$.schedules[?(@.jobType==\"Execute\")].enabled')\n matches = [match.value for match in jsonpath_expr.find(data)]\n\n # Should find at least one enabled field from Execute jobs\n assert len(matches) >= 1\n assert all(isinstance(match, bool) for match in matches)\n\n # Verify we can access the specific value that would be replaced\n first_match = matches[0]\n assert first_match is True # Our test data has enabled=True\n\n\ndef test_real_sample_schedules_file():\n \"\"\"Test that the actual sample .schedules file works with our function.\"\"\"\n from pathlib import Path\n\n schedules_file = Path(\"sample/workspace/Run Hello World.DataPipeline/.schedules\")\n\n # Skip if sample file doesn't exist (optional test)\n if not schedules_file.exists():\n pytest.skip(\"Sample .schedules file not found\")\n\n # Test that our function works with the real file content\n content = schedules_file.read_text(encoding=\"utf-8\")\n assert check_valid_json_content(content) is True\n\n # Verify the structure contains what we expect\n data = json.loads(content)\n\n assert \"schedules\" in data\n assert isinstance(data[\"schedules\"], list)\n\n\ndef test_check_valid_json_content_with_valid_json():\n \"\"\"Test check_valid_json_content with valid JSON string.\"\"\"\n valid_json = '{\"key\": \"value\", \"number\": 123, \"boolean\": true}'\n assert check_valid_json_content(valid_json) is True\n\n\ndef test_check_valid_json_content_with_invalid_json():\n \"\"\"Test check_valid_json_content with invalid JSON string.\"\"\"\n invalid_json = '{\"key\": \"value\" invalid json}'\n assert check_valid_json_content(invalid_json) is False\n\n\ndef test_check_valid_json_content_with_empty_string():\n \"\"\"Test check_valid_json_content with empty string.\"\"\"\n assert check_valid_json_content(\"\") is False\n\n\ndef test_check_valid_json_content_with_schedules_structure():\n \"\"\"Test check_valid_json_content with realistic schedules JSON structure.\"\"\"\n schedules_json = json.dumps({\n \"schedules\": [{\"jobType\": \"Execute\", \"enabled\": True, \"cronExpression\": \"0 0 12 * * ?\"}]\n })\n assert check_valid_json_content(schedules_json) is True\n\n\ndef test_check_valid_yaml_content_with_valid_yaml():\n \"\"\"Test check_valid_yaml_content with valid YAML string.\"\"\"\n valid_yaml = \"\"\"\nserver:\n host: localhost\n port: 8080\n\"\"\"\n assert check_valid_yaml_content(valid_yaml) is True\n\n\ndef test_check_valid_yaml_content_with_invalid_yaml():\n \"\"\"Test check_valid_yaml_content with invalid YAML string.\"\"\"\n invalid_yaml = \"invalid: yaml: [unclosed\"\n assert check_valid_yaml_content(invalid_yaml) is False\n\n\ndef test_check_valid_yaml_content_with_empty_string():\n \"\"\"Test check_valid_yaml_content with empty string.\"\"\"\n # Empty string parses as None, not a structured YAML mapping/sequence\n assert check_valid_yaml_content(\"\") is False\n\n\ndef test_check_valid_yaml_content_with_spark_compute_structure():\n \"\"\"Test check_valid_yaml_content with realistic SparkCompute.yml structure.\"\"\"\n spark_compute_yaml = \"\"\"\nenable_native_execution_engine: false\ndriver_cores: 8\ndriver_memory: 56g\nexecutor_cores: 8\nexecutor_memory: 56g\ndynamic_executor_allocation:\n enabled: true\n min_executors: 1\n max_executors: 9\nruntime_version: \"1.2\"\n\"\"\"\n assert check_valid_yaml_content(spark_compute_yaml) is True\n\n\ndef test_check_valid_yaml_content_with_complex_structure():\n \"\"\"Test check_valid_yaml_content with complex nested YAML structure.\"\"\"\n complex_yaml = \"\"\"\nconfig:\n database:\n host: localhost\n port: 5432\n credentials:\n username: admin\n password: secret\n features:\n - name: feature1\n enabled: true\n - name: feature2\n enabled: false\n limits:\n max_connections: 100\n timeout: 30.5\n\"\"\"\n assert check_valid_yaml_content(complex_yaml) is True\n\n\ndef test_check_valid_yaml_content_vs_json_content():\n \"\"\"Test that JSON is also valid YAML (JSON is a subset of YAML).\"\"\"\n json_content = '{\"key\": \"value\", \"number\": 123}'\n # JSON is valid YAML\n assert check_valid_yaml_content(json_content) is True\n assert check_valid_json_content(json_content) is True\n\n # YAML-only content is not valid JSON\n yaml_only_content = \"\"\"\nkey: value\nnumber: 123\n\"\"\"\n assert check_valid_yaml_content(yaml_only_content) is True\n assert check_valid_json_content(yaml_only_content) is False\n\n\ndef test_check_valid_yaml_content_with_notebook_py_content():\n \"\"\"Test that notebook .py files with # comments are not treated as valid YAML.\"\"\"\n notebook_content = \"\"\"# Fabric notebook source\n\n# METADATA ********************\n\n# META {\n# META \"kernel_info\": {\n# META \"name\": \"synapse_pyspark\"\n# META }\n# META }\n\n# CELL ********************\n\nprint(\"hello world\")\n\"\"\"\n assert check_valid_yaml_content(notebook_content) is False\n\n\ndef test_check_valid_yaml_content_with_kql_content():\n \"\"\"Test that KQL script files are not treated as valid YAML.\"\"\"\n kql_content = \"\"\"// KQL script\n// Use management commands to configure your database items.\n\n.create-merge table YellowTaxi (vendorID:string, tpepPickupDateTime:datetime)\n.create-or-alter table YellowTaxi ingestion json mapping 'YellowTaxi_mapping'\n\"\"\"\n assert check_valid_yaml_content(kql_content) is False\n\n\ndef test_check_valid_yaml_content_with_plain_python_content():\n \"\"\"Test that plain Python files (e.g., SparkJobDefinition) are not treated as valid YAML.\"\"\"\n python_content = \"\"\"from pyspark.sql import SparkSession\nimport logging\n\nlogger = logging.getLogger(__name__)\n\nif __name__ == \"__main__\":\n spark = SparkSession.builder.appName(\"test\").getOrCreate()\n df = spark.read.format(\"delta\").load(\"Tables/my_table\")\n df.show()\n\"\"\"\n assert check_valid_yaml_content(python_content) is False\n" }, { "path": "tests/test__fabric_endpoint.py", "content": "# Copyright (c) Microsoft Corporation.\n# Licensed under the MIT License.\n\nimport datetime\nimport time\nfrom unittest.mock import Mock\n\nimport pytest\nfrom azure.core.exceptions import ClientAuthenticationError\n\nfrom fabric_cicd import constants\nfrom fabric_cicd._common._exceptions import InvokeError, TokenError\nfrom fabric_cicd._common._fabric_endpoint import FabricEndpoint, _format_invoke_log, _handle_response\n\n\nclass DummyLogger:\n def __init__(self):\n self.messages = []\n\n def info(self, message):\n self.messages.append(message)\n\n def debug(self, message):\n self.messages.append(message)\n\n\nclass DummyCredential:\n def __init__(self, token, expires_on=9999999999):\n self.token = token\n self.expires_on = expires_on\n self.raise_exception = None\n\n def get_token(self, *_, **__):\n if self.raise_exception:\n raise self.raise_exception\n return Mock(token=self.token, expires_on=self.expires_on)\n\n\n@pytest.fixture\ndef setup_mocks(monkeypatch, mocker):\n dl = DummyLogger()\n mock_logger = mocker.Mock()\n mock_logger.isEnabledFor.return_value = True\n mock_logger.info.side_effect = dl.info\n mock_logger.debug.side_effect = dl.debug\n monkeypatch.setattr(\"fabric_cicd._common._fabric_endpoint.logger\", mock_logger)\n mock_requests = mocker.patch(\"requests.request\")\n return dl, mock_requests\n\n\ndef generate_mock_token():\n return \"mock_token_value\"\n\n\ndef test_integration(setup_mocks):\n \"\"\"Test integration of FabricEndpoint for GET request.\"\"\"\n _, mock_requests = setup_mocks\n mock_requests.return_value = Mock(\n status_code=200, headers={\"Content-Type\": \"application/json\"}, json=Mock(return_value={})\n )\n mock_token_credential = Mock()\n mock_token_credential.get_token.return_value = Mock(token=generate_mock_token(), expires_on=9999999999)\n endpoint = FabricEndpoint(token_credential=mock_token_credential)\n response = endpoint.invoke(\"GET\", \"http://example.com\")\n assert response[\"status_code\"] == 200\n\n\ndef test_performance(setup_mocks):\n \"\"\"Test that _handle_response completes quickly under long-running simulation.\"\"\"\n _, _mock_requests = setup_mocks\n response = Mock(status_code=200, headers={}, json=Mock(return_value={\"status\": \"Succeeded\"}))\n start_time = time.time()\n _handle_response(\n response=response,\n method=\"GET\",\n url=\"old\",\n body=\"{}\",\n long_running=True,\n iteration_count=2,\n )\n end_time = time.time()\n assert (end_time - start_time) < 1 # Ensure the function completes within 1 second\n\n\n@pytest.mark.parametrize(\n (\"method\", \"url\", \"body\", \"files\"),\n [\n (\"GET\", \"http://example.com\", \"{}\", None),\n (\"POST\", \"http://example.com\", \"{}\", {\"file\": \"test.txt\"}),\n ],\n ids=[\"invoke\", \"invoke_with_files\"],\n)\ndef test_invoke(setup_mocks, method, url, body, files):\n \"\"\"Test FabricEndpoint invoke method success + with optional files.\"\"\"\n _, mock_requests = setup_mocks\n mock_requests.return_value = Mock(\n status_code=200, headers={\"Content-Type\": \"application/json\"}, json=Mock(return_value={})\n )\n mock_token_credential = Mock()\n mock_token_credential.get_token.return_value = Mock(token=generate_mock_token(), expires_on=9999999999)\n endpoint = FabricEndpoint(token_credential=mock_token_credential)\n response = endpoint.invoke(method, url, body, files)\n assert response[\"status_code\"] == 200\n\n\ndef test_invoke_token_expired(setup_mocks, monkeypatch):\n \"\"\"Test invoking endpoint when the AAD token is expired and refreshed.\"\"\"\n dl, mock_requests = setup_mocks\n mock_requests.side_effect = [\n Mock(status_code=401, headers={\"x-ms-public-api-error-code\": \"TokenExpired\"}),\n Mock(status_code=200, headers={\"Content-Type\": \"application/json\"}, json=Mock(return_value={})),\n ]\n mock_token_credential = Mock()\n mock_token_credential.get_token.return_value = Mock(token=generate_mock_token(), expires_on=9999999999)\n endpoint = FabricEndpoint(token_credential=mock_token_credential)\n\n endpoint.aad_token_expiration = datetime.datetime.now(datetime.timezone.utc) - datetime.timedelta(seconds=1)\n endpoint._refresh_token = Mock()\n monkeypatch.setattr(\"fabric_cicd._common._fabric_endpoint._format_invoke_log\", lambda *_, **__: \"\")\n\n response = endpoint.invoke(\"GET\", \"http://example.com\")\n\n assert f\"{constants.INDENT}AAD token expired. Refreshing token.\" in dl.messages\n assert response[\"status_code\"] == 200\n\n\ndef test_invoke_exception(setup_mocks):\n \"\"\"Test invoking endpoint when the AAD token is expired and refreshed.\"\"\"\n _, mock_requests = setup_mocks\n mock_requests.side_effect = Exception(\"Test exception\")\n mock_token_credential = Mock()\n mock_token_credential.get_token.return_value = Mock(token=generate_mock_token(), expires_on=9999999999)\n endpoint = FabricEndpoint(token_credential=mock_token_credential)\n with pytest.raises(InvokeError):\n endpoint.invoke(\"GET\", \"http://example.com\")\n\n\ndef test_invoke_poll_long_running_false_with_202(setup_mocks):\n \"\"\"Test invoke method with poll_long_running=False exits early on 202 response.\"\"\"\n _, mock_requests = setup_mocks\n mock_requests.return_value = Mock(\n status_code=202,\n headers={\"Content-Type\": \"application/json\", \"Location\": \"http://example.com/status\"},\n json=Mock(return_value={}),\n )\n mock_token_credential = Mock()\n mock_token_credential.get_token.return_value = Mock(token=generate_mock_token(), expires_on=9999999999)\n endpoint = FabricEndpoint(token_credential=mock_token_credential)\n\n response = endpoint.invoke(\"POST\", \"http://example.com\", poll_long_running=False)\n\n # Should exit immediately without polling\n assert response[\"status_code\"] == 202\n assert mock_requests.call_count == 1 # Only one request, no polling\n\n\ndef test_invoke_poll_long_running_true_with_202(setup_mocks, monkeypatch):\n \"\"\"Test invoke method with poll_long_running=True polls on 202 response.\"\"\"\n _, mock_requests = setup_mocks\n\n # First call returns 202 with Location header, second call returns 200 with Succeeded status\n mock_requests.side_effect = [\n Mock(\n status_code=202,\n headers={\"Content-Type\": \"application/json\", \"Location\": \"http://example.com/status\"},\n json=Mock(return_value={}),\n text=\"{}\",\n ),\n Mock(\n status_code=200,\n headers={\"Content-Type\": \"application/json\"},\n json=Mock(return_value={\"status\": \"Succeeded\"}),\n text='{\"status\": \"Succeeded\"}',\n ),\n ]\n\n mock_token_credential = Mock()\n mock_token_credential.get_token.return_value = Mock(token=generate_mock_token(), expires_on=9999999999)\n endpoint = FabricEndpoint(token_credential=mock_token_credential)\n\n # Mock time.sleep to avoid delays in tests\n monkeypatch.setattr(\"time.sleep\", lambda _: None)\n\n response = endpoint.invoke(\"POST\", \"http://example.com\", poll_long_running=True)\n\n # Should poll and return final status\n assert response[\"status_code\"] == 200\n assert mock_requests.call_count == 2 # Initial request + polling request\n\n\ndef test_invoke_poll_long_running_default_with_202(setup_mocks, monkeypatch):\n \"\"\"Test invoke method with default poll_long_running (True) polls on 202 response.\"\"\"\n _, mock_requests = setup_mocks\n\n # First call returns 202 with Location header, second call returns 200 with Succeeded status\n mock_requests.side_effect = [\n Mock(\n status_code=202,\n headers={\"Content-Type\": \"application/json\", \"Location\": \"http://example.com/status\"},\n json=Mock(return_value={}),\n text=\"{}\",\n ),\n Mock(\n status_code=200,\n headers={\"Content-Type\": \"application/json\"},\n json=Mock(return_value={\"status\": \"Succeeded\"}),\n text='{\"status\": \"Succeeded\"}',\n ),\n ]\n\n mock_token_credential = Mock()\n mock_token_credential.get_token.return_value = Mock(token=generate_mock_token(), expires_on=9999999999)\n endpoint = FabricEndpoint(token_credential=mock_token_credential)\n\n # Mock time.sleep to avoid delays in tests\n monkeypatch.setattr(\"time.sleep\", lambda _: None)\n\n # Don't pass poll_long_running, should default to True\n response = endpoint.invoke(\"POST\", \"http://example.com\")\n\n # Should poll and return final status\n assert response[\"status_code\"] == 200\n assert mock_requests.call_count == 2 # Initial request + polling request\n\n\ndef test_refresh_token(setup_mocks):\n \"\"\"Test refreshing token sets token and expiration from AccessToken.\"\"\"\n _dl, _mock_requests = setup_mocks\n mock_token_credential = Mock()\n mock_token_credential.get_token.return_value = Mock(token=\"test_token\", expires_on=9999999999)\n endpoint = FabricEndpoint(token_credential=mock_token_credential)\n assert endpoint.aad_token == \"test_token\"\n assert endpoint.aad_token_expiration == datetime.datetime.fromtimestamp(9999999999, tz=datetime.timezone.utc)\n\n\n@pytest.mark.parametrize(\n (\"raise_exception\", \"expected_msg\"),\n [\n (ClientAuthenticationError(\"Auth failed\"), \"Failed to acquire AAD token. Auth failed\"),\n (Exception(\"Unexpected error\"), \"An unexpected error occurred when generating the AAD token. Unexpected error\"),\n ],\n ids=[\"auth_error\", \"unexpected_exception\"],\n)\ndef test_refresh_token_exceptions(raise_exception, expected_msg):\n \"\"\"Test token refresh exception handling for authentication failures.\"\"\"\n credential = DummyCredential(\"irrelevant\")\n credential.raise_exception = raise_exception\n with pytest.raises(TokenError, match=expected_msg):\n FabricEndpoint(token_credential=credential)\n\n\n@pytest.mark.parametrize(\n (\n \"status_code\",\n \"request_method\",\n \"expected_long_running\",\n \"expected_exit_loop\",\n \"input_long_running\",\n \"input_iteration_count\",\n \"response_header\",\n \"response_json\",\n ),\n [\n (200, \"POST\", False, True, False, 1, {}, {}),\n (202, \"POST\", True, False, False, 1, {\"Retry-After\": 20, \"Location\": \"new\"}, {}),\n (200, \"GET\", True, False, True, 2, {\"Retry-After\": 20, \"Location\": \"old\"}, {\"status\": \"Running\"}),\n (200, \"GET\", False, True, True, 2, {}, {\"status\": \"Succeeded\"}),\n (200, \"GET\", False, False, True, 2, {\"Retry-After\": 20, \"Location\": \"old\"}, {\"status\": \"Succeeded\"}),\n ],\n ids=[\n \"success\",\n \"long_running_redirect\",\n \"long_running_running\",\n \"long_running_success\",\n \"long_running_success_with_result\",\n ],\n)\ndef test_handle_response(\n status_code,\n request_method,\n expected_long_running,\n expected_exit_loop,\n input_long_running,\n input_iteration_count,\n response_header,\n response_json,\n):\n \"\"\"Test _handle_response behavior for various HTTP responses and long-running operations.\"\"\"\n response = Mock(status_code=status_code, headers=response_header, json=Mock(return_value=response_json))\n\n exit_loop, _method, _url, _body, long_running = _handle_response(\n response=response,\n method=request_method,\n url=\"old\",\n body=\"{}\",\n long_running=input_long_running,\n iteration_count=input_iteration_count,\n )\n assert exit_loop == expected_exit_loop\n assert long_running == expected_long_running\n\n\n@pytest.mark.parametrize(\n (\"exception_match\", \"response_json\"),\n [\n (\n \"[Operation failed].*\",\n {\"status\": \"Failed\", \"error\": {\"errorCode\": \"SampleErrorCode\", \"message\": \"Sample failure message\"}},\n ),\n (\"[Operation is in an undefined state].*\", {\"status\": \"Undefined\"}),\n ],\n ids=[\"failed\", \"undefined\"],\n)\ndef test_handle_response_longrunning_exception(exception_match, response_json):\n \"\"\"Test _handle_response raises exception for longrunning failure conditions.\"\"\"\n response = Mock(status_code=200, headers={}, json=Mock(return_value=response_json))\n\n with pytest.raises(Exception, match=exception_match):\n _handle_response(\n response=response,\n method=\"GET\",\n url=\"old\",\n body=\"{}\",\n long_running=True,\n iteration_count=2,\n )\n\n\n@pytest.mark.parametrize(\n (\n \"status_code\",\n \"input_iteration_count\",\n \"input_long_running\",\n \"response_header\",\n \"return_value\",\n \"exception_match\",\n \"max_duration\",\n \"start_time\",\n ),\n [\n (\n 401,\n 1,\n False,\n {\"x-ms-public-api-error-code\": \"Unauthorized\"},\n {},\n \"The executing identity is not authorized to call GET on 'http://example.com'.\",\n None,\n None,\n ),\n (\n 400,\n 1,\n False,\n {\"x-ms-public-api-error-code\": \"PrincipalTypeNotSupported\"},\n {},\n \"The executing principal type is not supported to call GET on 'http://example.com'.\",\n None,\n None,\n ),\n (\n 400,\n 1,\n False,\n {\"x-ms-public-api-error-code\": \"PrincipalTypeNotSupported\"},\n {\"message\": \"Test Libabry is not present in the environment.\"},\n \"Deployment attempted to remove a library that is not present in the environment. \",\n None,\n None,\n ),\n (\n 500,\n 5,\n False,\n {\"Content-Type\": \"application/json\"},\n {\"message\": \"Internal Server Error\"},\n r\"Maximum execution duration \\(0 seconds\\) exceeded\",\n 0,\n 0.0,\n ),\n (429, 5, True, {\"Retry-After\": \"10\"}, {}, r\"Maximum execution duration \\(0 seconds\\) exceeded\", 0, 0.0),\n (\n 200,\n 5,\n True,\n {},\n {\"status\": \"Running\"},\n r\"Maximum execution duration \\(0 seconds\\) exceeded\",\n 0,\n 0.0,\n ),\n ],\n ids=[\n \"unauthorized\",\n \"principal_type_not_supported\",\n \"failed_library_removal\",\n \"retry_500\",\n \"retry_429\",\n \"long_running_timeout\",\n ],\n)\ndef test_handle_response_exceptions(\n status_code,\n input_iteration_count,\n input_long_running,\n response_header,\n return_value,\n exception_match,\n max_duration,\n start_time,\n):\n \"\"\"Test _handle_response raises appropriate exceptions based on response error codes.\"\"\"\n response = Mock(status_code=status_code, headers=response_header, json=Mock(return_value=return_value))\n with pytest.raises(Exception, match=exception_match):\n _handle_response(\n response=response,\n method=\"GET\",\n url=\"http://example.com\",\n body=\"{}\",\n long_running=input_long_running,\n iteration_count=input_iteration_count,\n max_duration=max_duration,\n start_time=start_time,\n )\n\n\ndef test_handle_response_feature_not_available():\n \"\"\"Test _handle_response for feature not available\"\"\"\n response = Mock(status_code=403, reason=\"FeatureNotAvailable\")\n with pytest.raises(Exception, match=r\"Item type not supported. Description: FeatureNotAvailable\"):\n _handle_response(\n response=response,\n method=\"GET\",\n url=\"http://example.com\",\n body=\"{}\",\n long_running=False,\n iteration_count=1,\n )\n\n\ndef test_handle_response_item_display_name_already_in_use(setup_mocks, monkeypatch):\n \"\"\"\n Test _handle_response logs a retry message when item display name is already in use.\n\n Mocks time.sleep to avoid actual test execution delays.\n \"\"\"\n import time\n\n dl, _mock_requests = setup_mocks\n monkeypatch.setattr(\"time.sleep\", lambda _: None)\n response = Mock(status_code=400, headers={\"x-ms-public-api-error-code\": \"ItemDisplayNameNotAvailableYet\"})\n _handle_response(response, \"GET\", \"http://example.com\", \"{}\", False, 1, max_duration=300, start_time=time.time())\n expected = f\"{constants.INDENT}Item name is reserved. Checking again in 60 seconds (Attempt 1)...\"\n assert dl.messages == [expected]\n\n\ndef test_handle_response_environment_libraries_not_found(setup_mocks):\n \"\"\"Test _handle_response exits loop when environment libraries are not found (404).\"\"\"\n _, _mock_requests = setup_mocks\n response = Mock(status_code=404, headers={\"x-ms-public-api-error-code\": \"EnvironmentLibrariesNotFound\"})\n exit_loop, _method, _url, _body, long_running = _handle_response(\n response=response,\n method=\"GET\",\n url=\"http://example.com\",\n body=\"{}\",\n long_running=False,\n iteration_count=1,\n )\n assert exit_loop is True\n assert long_running is False\n\n\ndef test_format_invoke_log():\n \"\"\"Test formatting of the invoke log message.\"\"\"\n response = Mock(status_code=200, headers={\"Content-Type\": \"application/json\"}, json=Mock(return_value={}))\n log_message = _format_invoke_log(response, \"GET\", \"http://example.com\", \"{}\")\n assert \"Method: GET\" in log_message\n assert \"URL: http://example.com\" in log_message\n" }, { "path": "tests/test__file.py", "content": "# Copyright (c) Microsoft Corporation.\n# Licensed under the MIT License.\n\nimport base64\nfrom pathlib import Path\n\nimport pytest\n\nfrom fabric_cicd._common._file import File\n\nSAMPLE_IMAGE_DATA = b\"\\x89PNG\\r\\n\\x1a\\n\\x00\\x00\\x00\\rIHDR\\x00\\x00\\x00\\x01\"\nSAMPLE_TEXT_DATA = \"sample text\"\n\n\n@pytest.fixture\ndef text_file(tmp_path):\n item_path = tmp_path / \"workspace/ABC.SemanticModel\"\n file_path = item_path / \"definition/tables/Table.tmdl\"\n file_path.parent.mkdir(parents=True, exist_ok=True) # Ensure the parent directories are created\n file_path.write_text(SAMPLE_TEXT_DATA)\n return File(item_path=item_path, file_path=file_path)\n\n\n@pytest.fixture\ndef image_file(tmp_path):\n item_path = tmp_path / \"workspace/ABC.Report\"\n file_path = item_path / \"StaticResources/RegisteredResources/image.png\"\n file_path.parent.mkdir(parents=True, exist_ok=True) # Ensure the parent directories are created\n file_path.write_bytes(SAMPLE_IMAGE_DATA)\n return File(item_path=item_path, file_path=file_path)\n\n\ndef test_file_text_initialization(text_file):\n assert text_file.name == \"Table.tmdl\"\n assert text_file.contents == SAMPLE_TEXT_DATA\n assert text_file.relative_path == \"definition/tables/Table.tmdl\"\n\n\ndef test_file_text_payload(text_file):\n expected_payload = base64.b64encode(SAMPLE_TEXT_DATA.encode(\"utf-8\")).decode(\"utf-8\")\n assert text_file.base64_payload == {\n \"path\": \"definition/tables/Table.tmdl\",\n \"payload\": expected_payload,\n \"payloadType\": \"InlineBase64\",\n }\n\n\ndef test_file_text_set_contents(text_file):\n text_file.contents = \"New contents\"\n assert text_file.contents == \"New contents\"\n\n\ndef test_file_image_immutable_fields(image_file):\n with pytest.raises(AttributeError):\n image_file.item_path = Path(\"/new/path\")\n with pytest.raises(AttributeError):\n image_file.file_path = Path(\"/new/path\")\n with pytest.raises(AttributeError):\n image_file.contents = \"new contents\"\n\n\ndef test_file_image_payload(image_file):\n expected_payload = base64.b64encode(SAMPLE_IMAGE_DATA).decode(\"utf-8\")\n assert image_file.base64_payload == {\n \"path\": \"StaticResources/RegisteredResources/image.png\",\n \"payload\": expected_payload,\n \"payloadType\": \"InlineBase64\",\n }\n\n\ndef test_file_text_special_characters(tmp_path):\n special_text = (\n \"Greek: a, β, y, δ, ε, ζ\\n\"\n \"Latin: æ, ø, å, ß, é, ñ, ü\\n\"\n \"Cyrillic: Ж, Д, и, ю\\n\"\n \"Arabic: مرحبا, سلام\\n\"\n \"Chinese: 你好, 世界\\n\"\n \"Emoji: 😊, 🚀, 🌟\\n\"\n \"Symbols: ©, ®, ™, €, £, ¥, ∞, ≠, ≤, ≥\"\n )\n item_path = tmp_path / \"workspace/SpecialTextModel\"\n file_path = item_path / \"definition/special.txt\"\n file_path.parent.mkdir(parents=True, exist_ok=True)\n file_path.write_text(special_text, encoding=\"utf-8\")\n file_obj = File(item_path=item_path, file_path=file_path)\n expected_payload = base64.b64encode(special_text.encode(\"utf-8\")).decode(\"utf-8\")\n assert file_obj.contents == special_text\n assert file_obj.base64_payload == {\n \"path\": \"definition/special.txt\",\n \"payload\": expected_payload,\n \"payloadType\": \"InlineBase64\",\n }\n" }, { "path": "tests/test_config_validator.py", "content": "# Copyright (c) Microsoft Corporation.\n# Licensed under the MIT License.\n\n\"\"\"Unit tests for ConfigValidator class.\"\"\"\n\nfrom pathlib import Path\nfrom unittest.mock import patch\n\nimport pytest\nimport yaml\n\nfrom fabric_cicd import constants\nfrom fabric_cicd._common._config_validator import ConfigValidationError, ConfigValidator\n\n\nclass TestConfigValidator:\n \"\"\"Unit tests for ConfigValidator class.\"\"\"\n\n def setup_method(self):\n \"\"\"Set up for each test method.\"\"\"\n self.validator = ConfigValidator()\n\n def test_init(self):\n \"\"\"Test ConfigValidator initialization.\"\"\"\n assert self.validator.errors == []\n assert self.validator.config is None\n assert self.validator.config_path is None\n assert self.validator.environment is None\n\n def test_validate_file_existence_valid_file(self, tmp_path):\n \"\"\"Test _validate_file_existence with valid file.\"\"\"\n config_file = tmp_path / \"config.yaml\"\n config_file.write_text(\"test: value\")\n\n result = self.validator._validate_file_existence(str(config_file))\n\n assert result == config_file.resolve()\n assert self.validator.errors == []\n\n def test_validate_file_existence_missing_file(self):\n \"\"\"Test _validate_file_existence with missing file.\"\"\"\n result = self.validator._validate_file_existence(\"nonexistent.yaml\")\n\n assert result is None\n assert len(self.validator.errors) == 1\n assert (\n constants.CONFIG_VALIDATION_MSGS[\"file\"][\"not_found\"].format(\"nonexistent.yaml\") in self.validator.errors[0]\n )\n\n @pytest.mark.parametrize(\"path_value\", [\"\", None])\n def test_validate_file_existence_empty_or_none_path(self, path_value):\n \"\"\"Test _validate_file_existence with empty or None path.\"\"\"\n result = self.validator._validate_file_existence(path_value)\n\n assert result is None\n assert len(self.validator.errors) == 1\n assert constants.CONFIG_VALIDATION_MSGS[\"file\"][\"path_empty\"] in self.validator.errors[0]\n\n def test_validate_file_existence_directory_instead_of_file(self, tmp_path):\n \"\"\"Test _validate_file_existence with directory instead of file.\"\"\"\n result = self.validator._validate_file_existence(str(tmp_path))\n\n assert result is None\n assert len(self.validator.errors) == 1\n assert constants.CONFIG_VALIDATION_MSGS[\"file\"][\"not_file\"].format(str(tmp_path)) in self.validator.errors[0]\n\n def test_validate_file_existence_invalid_path_os_error(self):\n \"\"\"Test _validate_file_existence with a path that triggers OSError.\"\"\"\n with patch(\"fabric_cicd._common._config_validator.Path.resolve\", side_effect=OSError(\"bad path\")):\n result = self.validator._validate_file_existence(\"some_path\")\n\n assert result is None\n assert len(self.validator.errors) == 1\n assert (\n constants.CONFIG_VALIDATION_MSGS[\"file\"][\"invalid_path\"].format(\"some_path\", \"bad path\")\n in self.validator.errors[0]\n )\n\n def test_validate_yaml_content_valid_yaml(self, tmp_path):\n \"\"\"Test _validate_yaml_content with valid YAML.\"\"\"\n config_file = tmp_path / \"config.yaml\"\n config_data = {\"core\": {\"workspace_id\": \"test-id\"}}\n config_file.write_text(yaml.dump(config_data))\n\n self.validator.config_path = config_file\n result = self.validator._validate_yaml_content(config_file)\n\n assert result == config_data\n assert self.validator.errors == []\n\n def test_validate_yaml_content_invalid_yaml(self, tmp_path):\n \"\"\"Test _validate_yaml_content with invalid YAML syntax.\"\"\"\n config_file = tmp_path / \"config.yaml\"\n config_file.write_text(\"invalid: yaml: content: [\")\n\n self.validator.config_path = config_file\n result = self.validator._validate_yaml_content(config_file)\n\n assert result is None\n assert len(self.validator.errors) == 1\n # We can't test the exact error message as it includes the specific parse error\n assert constants.CONFIG_VALIDATION_MSGS[\"file\"][\"yaml_syntax\"].split(\":\")[0] in self.validator.errors[0]\n\n def test_validate_yaml_content_unicode_decode_error(self, tmp_path):\n \"\"\"Test _validate_yaml_content with file that triggers UnicodeDecodeError.\"\"\"\n config_file = tmp_path / \"config.yaml\"\n config_file.write_bytes(b\"\\x80\\x81\\x82\") # Invalid UTF-8\n\n self.validator.config_path = config_file\n result = self.validator._validate_yaml_content(config_file)\n\n assert result is None\n assert len(self.validator.errors) == 1\n assert constants.CONFIG_VALIDATION_MSGS[\"file\"][\"encoding_error\"].split(\":\")[0] in self.validator.errors[0]\n\n def test_validate_yaml_content_permission_error(self, tmp_path):\n \"\"\"Test _validate_yaml_content with file that triggers PermissionError.\"\"\"\n config_file = tmp_path / \"config.yaml\"\n config_file.write_text(\"valid: yaml\")\n\n self.validator.config_path = config_file\n with patch(\"pathlib.Path.open\", side_effect=PermissionError(\"access denied\")):\n result = self.validator._validate_yaml_content(config_file)\n\n assert result is None\n assert len(self.validator.errors) == 1\n assert constants.CONFIG_VALIDATION_MSGS[\"file\"][\"permission_denied\"].split(\":\")[0] in self.validator.errors[0]\n\n def test_validate_yaml_content_unexpected_error(self, tmp_path):\n \"\"\"Test _validate_yaml_content with file that triggers unexpected error.\"\"\"\n config_file = tmp_path / \"config.yaml\"\n config_file.write_text(\"valid: yaml\")\n\n self.validator.config_path = config_file\n with patch(\"pathlib.Path.open\", side_effect=RuntimeError(\"unexpected\")):\n result = self.validator._validate_yaml_content(config_file)\n\n assert result is None\n assert len(self.validator.errors) == 1\n assert constants.CONFIG_VALIDATION_MSGS[\"file\"][\"unexpected_error\"].split(\":\")[0] in self.validator.errors[0]\n\n def test_validate_yaml_content_non_dict_yaml(self, tmp_path):\n \"\"\"Test _validate_yaml_content with non-dictionary YAML.\"\"\"\n config_file = tmp_path / \"config.yaml\"\n config_file.write_text(\"- item1\\n- item2\")\n\n self.validator.config_path = config_file\n result = self.validator._validate_yaml_content(config_file)\n\n assert result is None\n assert len(self.validator.errors) == 1\n assert constants.CONFIG_VALIDATION_MSGS[\"file\"][\"not_dict\"].format(\"list\") in self.validator.errors[0]\n\n def test_validate_yaml_content_none_path(self):\n \"\"\"Test _validate_yaml_content with None path.\"\"\"\n result = self.validator._validate_yaml_content(None)\n\n assert result is None\n assert self.validator.errors == [] # Error should already be added by file existence check\n\n def test_validate_yaml_content_empty_file(self, tmp_path):\n \"\"\"Test _validate_yaml_content with empty file.\"\"\"\n config_file = tmp_path / \"config.yaml\"\n config_file.write_text(\"\")\n\n self.validator.config_path = config_file\n result = self.validator._validate_yaml_content(config_file)\n\n assert result is None\n assert len(self.validator.errors) == 1\n assert constants.CONFIG_VALIDATION_MSGS[\"file\"][\"empty_file\"] in self.validator.errors[0]\n\n def test_validate_config_structure_valid(self):\n \"\"\"Test _validate_config_structure with valid config.\"\"\"\n self.validator.config = {\"core\": {\"workspace_id\": \"test-id\"}}\n\n self.validator._validate_config_structure()\n\n assert self.validator.errors == []\n\n @pytest.mark.parametrize(\"config_value\", [[\"not\", \"a\", \"dict\"], None])\n def test_validate_config_structure_non_dict_or_none(self, config_value):\n \"\"\"Test _validate_config_structure with non-dictionary or None config.\"\"\"\n self.validator.config = config_value\n\n self.validator._validate_config_structure()\n\n # The structure validation doesn't add errors for non-dict/None configs\n # as this is handled by YAML content validation\n assert self.validator.errors == []\n\n def test_validate_config_structure_missing_core(self):\n \"\"\"Test _validate_config_structure with config missing 'core' section.\"\"\"\n self.validator.config = {\"features\": [\"f1\"]}\n\n self.validator._validate_config_structure()\n\n assert len(self.validator.errors) == 1\n assert constants.CONFIG_VALIDATION_MSGS[\"structure\"][\"missing_core\"] in self.validator.errors[0]\n\n def test_validate_config_structure_core_not_dict(self):\n \"\"\"Test _validate_config_structure with 'core' as non-dict.\"\"\"\n self.validator.config = {\"core\": \"not a dict\"}\n\n self.validator._validate_config_structure()\n\n assert len(self.validator.errors) == 1\n assert constants.CONFIG_VALIDATION_MSGS[\"structure\"][\"core_not_dict\"].format(\"str\") in self.validator.errors[0]\n\n def test_validate_workspace_field_valid_string(self):\n \"\"\"Test _validate_workspace_field with valid string.\"\"\"\n core = {\"workspace\": \"test-workspace\"}\n\n result = self.validator._validate_workspace_field(core, \"workspace\")\n\n assert result is True\n assert self.validator.errors == []\n\n def test_validate_workspace_field_valid_workspace_id_guid(self):\n \"\"\"Test _validate_workspace_field with valid workspace_id GUID.\"\"\"\n core = {\"workspace_id\": \"8b6e2c7a-4c1f-4e3a-9b2e-7d8f2e1a6c3b\"}\n\n result = self.validator._validate_workspace_field(core, \"workspace_id\")\n\n assert result is True\n assert self.validator.errors == []\n\n def test_validate_workspace_field_invalid_workspace_id_guid(self):\n \"\"\"Test _validate_workspace_field with invalid workspace_id GUID format.\"\"\"\n core = {\"workspace_id\": \"invalid-guid-format\"}\n\n result = self.validator._validate_workspace_field(core, \"workspace_id\")\n\n assert result is False\n assert len(self.validator.errors) == 1\n assert \"must be a valid GUID format\" in self.validator.errors[0]\n\n def test_validate_workspace_field_valid_dict(self):\n \"\"\"Test _validate_workspace_field with valid environment mapping.\"\"\"\n core = {\"workspace\": {\"dev\": \"dev-workspace\", \"prod\": \"prod-workspace\"}}\n\n result = self.validator._validate_workspace_field(core, \"workspace\")\n\n assert result is True\n assert self.validator.errors == []\n\n def test_validate_workspace_field_valid_workspace_id_dict(self):\n \"\"\"Test _validate_workspace_field with valid workspace_id environment mapping.\"\"\"\n core = {\n \"workspace_id\": {\n \"dev\": \"8b6e2c7a-4c1f-4e3a-9b2e-7d8f2e1a6c3b\",\n \"prod\": \"2f4b9e8d-1a7c-4d3e-b8e2-5c9f7a2d4e1b\",\n }\n }\n\n result = self.validator._validate_workspace_field(core, \"workspace_id\")\n\n assert result is True\n assert self.validator.errors == []\n\n def test_validate_workspace_field_invalid_workspace_id_dict(self):\n \"\"\"Test _validate_workspace_field with invalid workspace_id GUID in environment mapping.\"\"\"\n core = {\"workspace_id\": {\"dev\": \"valid-8b6e2c7a-4c1f-4e3a-9b2e-7d8f2e1a6c3b\", \"prod\": \"invalid-guid\"}}\n\n result = self.validator._validate_workspace_field(core, \"workspace_id\")\n\n assert result is False\n assert len(self.validator.errors) == 2 # One for each invalid GUID\n assert \"must be a valid GUID format\" in self.validator.errors[0]\n assert \"must be a valid GUID format\" in self.validator.errors[1]\n\n def test_validate_workspace_field_missing(self):\n \"\"\"Test _validate_workspace_field with missing field.\"\"\"\n core = {}\n\n result = self.validator._validate_workspace_field(core, \"workspace_id\")\n\n assert result is False\n assert self.validator.errors == []\n\n def test_validate_workspace_field_empty_string(self):\n \"\"\"Test _validate_workspace_field with empty/whitespace string.\"\"\"\n core = {\"workspace_id\": \" \"}\n\n result = self.validator._validate_workspace_field(core, \"workspace_id\")\n\n assert result is False\n assert len(self.validator.errors) == 1\n assert (\n constants.CONFIG_VALIDATION_MSGS[\"field\"][\"empty_value\"].format(\"workspace_id\") in self.validator.errors[0]\n )\n\n def test_validate_workspace_field_invalid_type(self):\n \"\"\"Test _validate_workspace_field with invalid type.\"\"\"\n core = {\"workspace_id\": 123}\n\n result = self.validator._validate_workspace_field(core, \"workspace_id\")\n\n assert result is False\n assert len(self.validator.errors) == 1\n assert \"must be either a string or environment mapping\" in self.validator.errors[0]\n\n def test_validate_environment_mapping_valid(self):\n \"\"\"Test _validate_environment_mapping with valid mapping.\"\"\"\n field_value = {\"dev\": \"dev-value\", \"prod\": \"prod-value\"}\n\n result = self.validator._validate_environment_mapping(field_value, \"test_field\", str)\n\n assert result is True\n assert self.validator.errors == []\n\n def test_validate_environment_mapping_empty(self):\n \"\"\"Test _validate_environment_mapping with empty mapping.\"\"\"\n field_value = {}\n\n result = self.validator._validate_environment_mapping(field_value, \"test_field\", str)\n\n assert result is False\n assert len(self.validator.errors) == 1\n assert \"environment mapping cannot be empty\" in self.validator.errors[0]\n\n def test_validate_environment_mapping_invalid_env_key(self):\n \"\"\"Test _validate_environment_mapping with invalid environment key.\"\"\"\n field_value = {\"\": \"value\", \"dev\": \"dev-value\"}\n\n result = self.validator._validate_environment_mapping(field_value, \"test_field\", str)\n\n assert result is False\n assert len(self.validator.errors) == 1\n assert \"Environment key in 'test_field' must be a non-empty string\" in self.validator.errors[0]\n\n def test_validate_environment_mapping_wrong_value_type(self):\n \"\"\"Test _validate_environment_mapping with wrong value type.\"\"\"\n field_value = {\"dev\": 123, \"prod\": \"prod-value\"}\n\n result = self.validator._validate_environment_mapping(field_value, \"test_field\", str)\n\n assert result is False\n assert len(self.validator.errors) == 1\n assert \"must be a str, got int\" in self.validator.errors[0]\n\n def test_validate_environment_mapping_empty_string_value(self):\n \"\"\"Test _validate_environment_mapping with empty string value.\"\"\"\n field_value = {\"dev\": \"\", \"prod\": \"prod-value\"}\n\n result = self.validator._validate_environment_mapping(field_value, \"test_field\", str)\n\n assert result is False\n assert len(self.validator.errors) == 1\n assert \"value for environment 'dev' cannot be empty\" in self.validator.errors[0]\n\n def test_validate_environment_mapping_empty_list_value(self):\n \"\"\"Test _validate_environment_mapping with empty list value.\"\"\"\n field_value = {\"dev\": [], \"prod\": [\"item1\"]}\n\n result = self.validator._validate_environment_mapping(field_value, \"test_field\", list)\n\n assert result is False\n assert len(self.validator.errors) == 1\n assert \"value for environment 'dev' cannot be empty\" in self.validator.errors[0]\n\n def test_validate_repository_directory_empty_string(self):\n \"\"\"Test _validate_repository_directory with empty string value.\"\"\"\n core = {\"repository_directory\": \" \"}\n\n self.validator._validate_repository_directory(core)\n\n assert len(self.validator.errors) == 1\n assert (\n constants.CONFIG_VALIDATION_MSGS[\"field\"][\"empty_value\"].format(\"repository_directory\")\n in self.validator.errors[0]\n )\n\n def test_validate_repository_directory_valid_string(self):\n \"\"\"Test _validate_repository_directory with valid string.\"\"\"\n core = {\"repository_directory\": \"/path/to/repo\"}\n\n self.validator._validate_repository_directory(core)\n\n assert self.validator.errors == []\n\n def test_validate_repository_directory_missing(self):\n \"\"\"Test _validate_repository_directory with missing field.\"\"\"\n core = {}\n\n self.validator._validate_repository_directory(core)\n\n assert len(self.validator.errors) == 1\n assert \"must specify 'repository_directory'\" in self.validator.errors[0]\n\n def test_validate_repository_directory_invalid_type(self):\n \"\"\"Test _validate_repository_directory with invalid type.\"\"\"\n core = {\"repository_directory\": 123}\n\n self.validator._validate_repository_directory(core)\n\n assert len(self.validator.errors) == 1\n assert \"must be either a string or environment mapping\" in self.validator.errors[0]\n\n def test_validate_repository_directory_valid_env_mapping(self):\n \"\"\"Test _validate_repository_directory with valid environment mapping.\"\"\"\n core = {\"repository_directory\": {\"dev\": \"/dev/path\", \"prod\": \"/prod/path\"}}\n\n self.validator._validate_repository_directory(core)\n\n assert self.validator.errors == []\n\n def test_validate_repository_directory_invalid_env_mapping(self):\n \"\"\"Test _validate_repository_directory with invalid environment mapping.\"\"\"\n core = {\"repository_directory\": {\"\": \"/dev/path\"}}\n\n self.validator._validate_repository_directory(core)\n\n assert len(self.validator.errors) == 1\n assert (\n constants.CONFIG_VALIDATION_MSGS[\"environment\"][\"invalid_env_key\"].format(\"repository_directory\", \"str\")\n in self.validator.errors[0]\n )\n\n def test_validate_item_types_valid_list(self):\n \"\"\"Test _validate_item_types with valid item types.\"\"\"\n item_types = [\"Notebook\", \"DataPipeline\"]\n\n self.validator._validate_item_types(item_types)\n\n assert self.validator.errors == []\n\n def test_validate_item_types_empty_list(self):\n \"\"\"Test _validate_item_types with empty list.\"\"\"\n item_types = []\n\n self.validator._validate_item_types(item_types)\n\n assert len(self.validator.errors) == 1\n assert \"'item_types_in_scope' cannot be empty\" in self.validator.errors[0]\n\n def test_validate_item_types_invalid_type(self):\n \"\"\"Test _validate_item_types with invalid item type.\"\"\"\n item_types = [\"Notebook\", 123, \"DataPipeline\"]\n\n self.validator._validate_item_types(item_types)\n\n assert len(self.validator.errors) == 1\n assert \"Item type must be a string, got int\" in self.validator.errors[0]\n\n def test_validate_item_types_unknown_item_type(self):\n \"\"\"Test _validate_item_types with unknown item type.\"\"\"\n item_types = [\"Notebook\", \"UnknownType\"]\n\n self.validator._validate_item_types(item_types)\n\n assert len(self.validator.errors) == 1\n assert \"Invalid item type 'UnknownType'\" in self.validator.errors[0]\n assert \"Available types:\" in self.validator.errors[0]\n\n def test_validate_item_types_with_env_context(self):\n \"\"\"Test _validate_item_types with environment context.\"\"\"\n item_types = [\"UnknownType\"]\n\n self.validator._validate_item_types(item_types, env_context=\"dev\")\n\n assert len(self.validator.errors) == 1\n assert \"Invalid item type 'UnknownType' in environment 'dev'\" in self.validator.errors[0]\n\n def test_validate_item_types_in_scope_invalid_env_mapping(self):\n \"\"\"Test _validate_item_types_in_scope with invalid environment mapping.\"\"\"\n core = {\"item_types_in_scope\": {\"\": [\"Notebook\"]}}\n\n self.validator._validate_item_types_in_scope(core)\n\n assert len(self.validator.errors) == 1\n assert (\n constants.CONFIG_VALIDATION_MSGS[\"environment\"][\"invalid_env_key\"].format(\"item_types_in_scope\", \"str\")\n in self.validator.errors[0]\n )\n\n def test_validate_regex_valid(self):\n \"\"\"Test _validate_regex with valid regex.\"\"\"\n self.validator._validate_regex(\"^test.*\", \"test_section\")\n\n assert self.validator.errors == []\n\n def test_validate_regex_invalid(self):\n \"\"\"Test _validate_regex with invalid regex.\"\"\"\n self.validator._validate_regex(\"[invalid\", \"test_section\")\n\n assert len(self.validator.errors) == 1\n assert \"is not a valid regex pattern\" in self.validator.errors[0]\n\n def test_validate_items_list_valid(self):\n \"\"\"Test _validate_items_list with valid items.\"\"\"\n items_list = [\"item1.Notebook\", \"item2.DataPipeline\"]\n\n self.validator._validate_items_list(items_list, \"test_context\")\n\n assert self.validator.errors == []\n\n def test_validate_items_list_invalid_type(self):\n \"\"\"Test _validate_items_list with invalid item type.\"\"\"\n items_list = [\"item1.Notebook\", 123]\n\n self.validator._validate_items_list(items_list, \"test_context\")\n\n assert len(self.validator.errors) == 1\n assert \"'test_context[1]' must be a string\" in self.validator.errors[0]\n\n def test_validate_items_list_empty_item(self):\n \"\"\"Test _validate_items_list with empty item.\"\"\"\n items_list = [\"item1.Notebook\", \"\"]\n\n self.validator._validate_items_list(items_list, \"test_context\")\n\n assert len(self.validator.errors) == 1\n assert (\n constants.CONFIG_VALIDATION_MSGS[\"operation\"][\"list_entry_empty\"].format(\"test_context\", 1)\n in self.validator.errors[0]\n )\n\n def test_validate_features_list_valid(self):\n \"\"\"Test _validate_features_list with valid features.\"\"\"\n features_list = [\"enable_shortcut_publish\"]\n\n self.validator._validate_features_list(features_list, \"test_context\")\n\n assert self.validator.errors == []\n\n def test_validate_features_list_invalid_type(self):\n \"\"\"Test _validate_features_list with invalid feature type.\"\"\"\n features_list = [\"enable_shortcut_publish\", 123]\n\n self.validator._validate_features_list(features_list, \"test_context\")\n\n assert len(self.validator.errors) == 1\n assert \"'test_context[1]' must be a string\" in self.validator.errors[0]\n\n def test_validate_features_list_empty_feature(self):\n \"\"\"Test _validate_features_list with empty feature.\"\"\"\n features_list = [\"enable_shortcut_publish\", \"\"]\n\n self.validator._validate_features_list(features_list, \"test_context\")\n\n assert len(self.validator.errors) == 1\n assert (\n constants.CONFIG_VALIDATION_MSGS[\"operation\"][\"list_entry_empty\"].format(\"test_context\", 1)\n in self.validator.errors[0]\n )\n\n @pytest.mark.parametrize(\"key_value\", [123, \"\"])\n def test_validate_constants_dict_invalid_or_empty_key(self, key_value):\n \"\"\"Test _validate_constants_section with invalid or empty key.\"\"\"\n constants_dict = {key_value: \"value\"}\n\n self.validator._validate_constants_section(constants_dict)\n\n assert len(self.validator.errors) == 1\n assert \"Constant key in 'constants' must be a non-empty string\" in self.validator.errors[0]\n\n def test_validate_constants_dict_unknown_constant(self):\n \"\"\"Test _validate_constants_section with unknown constant.\"\"\"\n constants_dict = {\"UNKNOWN_CONSTANT\": \"value\"}\n\n self.validator._validate_constants_section(constants_dict)\n\n assert len(self.validator.errors) == 1\n assert \"Unknown constant 'UNKNOWN_CONSTANT'\" in self.validator.errors[0]\n\n def test_validate_constants_dict_valid_various_types(self):\n \"\"\"Test _validate_constants_section with valid constants of various types.\"\"\"\n constants_dict = {\n \"DEFAULT_API_ROOT_URL\": \"https://api.fabric.microsoft.com\",\n \"ACCEPTED_ITEM_TYPES\": [\"Notebook\", \"DataPipeline\"],\n }\n\n self.validator._validate_constants_section(constants_dict)\n\n assert self.validator.errors == []\n\n def test_validate_constants_dict_url_constant_non_string_type(self):\n \"\"\"Test _validate_constants_section rejects non-string URL constant.\"\"\"\n constants_dict = {\"DEFAULT_API_ROOT_URL\": 12345}\n\n self.validator._validate_constants_section(constants_dict)\n\n assert len(self.validator.errors) == 1\n assert \"'constants.DEFAULT_API_ROOT_URL' must be a string URL, got int\" in self.validator.errors[0]\n\n def test_validate_constants_dict_url_constant_invalid_hostname(self):\n \"\"\"Test _validate_constants_section rejects URL with invalid hostname.\"\"\"\n constants_dict = {\"DEFAULT_API_ROOT_URL\": \"https://evil.example.com\"}\n\n self.validator._validate_constants_section(constants_dict)\n\n assert len(self.validator.errors) == 1\n assert \"invalid hostname\" in self.validator.errors[0].lower()\n\n def test_validate_constants_dict_url_constant_http_scheme(self):\n \"\"\"Test _validate_constants_section rejects URL with HTTP scheme.\"\"\"\n constants_dict = {\"DEFAULT_API_ROOT_URL\": \"http://api.fabric.microsoft.com\"}\n\n self.validator._validate_constants_section(constants_dict)\n\n assert len(self.validator.errors) == 1\n assert \"must use HTTPS scheme\" in self.validator.errors[0]\n\n def test_validate_constants_dict_url_constant_with_path(self):\n \"\"\"Test _validate_constants_section rejects URL with path components.\"\"\"\n constants_dict = {\"DEFAULT_API_ROOT_URL\": \"https://api.fabric.microsoft.com/v1\"}\n\n self.validator._validate_constants_section(constants_dict)\n\n assert len(self.validator.errors) == 1\n assert \"without path components\" in self.validator.errors[0]\n\n def test_validate_constants_dict_url_constant_valid_powerbi(self):\n \"\"\"Test _validate_constants_section accepts valid PowerBI URL constant.\"\"\"\n constants_dict = {\"FABRIC_API_ROOT_URL\": \"https://api.powerbi.com\"}\n\n self.validator._validate_constants_section(constants_dict)\n\n assert self.validator.errors == []\n\n def test_validate_constants_dict_non_url_constant_skips_url_validation(self):\n \"\"\"Test _validate_constants_section skips URL validation for non-URL constants.\"\"\"\n constants_dict = {\"ACCEPTED_ITEM_TYPES\": [\"Notebook\", \"DataPipeline\"]}\n\n self.validator._validate_constants_section(constants_dict)\n\n assert self.validator.errors == []\n\n def test_validate_item_types_in_scope_valid_list(self):\n \"\"\"Test _validate_item_types_in_scope with valid list.\"\"\"\n core = {\"item_types_in_scope\": [\"Notebook\", \"DataPipeline\"]}\n\n self.validator._validate_item_types_in_scope(core)\n\n assert self.validator.errors == []\n\n def test_validate_item_types_in_scope_empty_list(self):\n \"\"\"Test _validate_item_types_in_scope with empty list.\"\"\"\n core = {\"item_types_in_scope\": []}\n\n self.validator._validate_item_types_in_scope(core)\n\n assert len(self.validator.errors) == 1\n assert \"'item_types_in_scope' cannot be empty if specified\" in self.validator.errors[0]\n\n def test_validate_item_types_in_scope_environment_mapping(self):\n \"\"\"Test _validate_item_types_in_scope with environment mapping.\"\"\"\n core = {\"item_types_in_scope\": {\"dev\": [\"Notebook\"], \"prod\": [\"DataPipeline\", \"Notebook\"]}}\n\n self.validator._validate_item_types_in_scope(core)\n\n assert self.validator.errors == []\n\n def test_validate_item_types_in_scope_invalid_type(self):\n \"\"\"Test _validate_item_types_in_scope with invalid type.\"\"\"\n core = {\"item_types_in_scope\": \"invalid\"}\n\n self.validator._validate_item_types_in_scope(core)\n\n assert len(self.validator.errors) == 1\n assert \"must be either a list or environment mapping dictionary\" in self.validator.errors[0]\n\n def test_validate_item_types_in_scope_missing_field(self):\n \"\"\"Test _validate_item_types_in_scope with missing field (should be okay).\"\"\"\n core = {\"workspace_id\": \"12345678-1234-1234-1234-123456789abc\"}\n\n self.validator._validate_item_types_in_scope(core)\n\n assert self.validator.errors == []\n\n def test_resolve_repository_path_absolute_path(self, tmp_path):\n \"\"\"Test _resolve_repository_path with absolute path.\"\"\"\n # Create actual directory\n repo_dir = tmp_path / \"workspace\"\n repo_dir.mkdir()\n\n self.validator.config = {\"core\": {\"repository_directory\": str(repo_dir)}}\n self.validator.config_path = tmp_path / \"config.yaml\"\n\n self.validator._resolve_repository_path()\n\n assert self.validator.errors == []\n assert Path(self.validator.config[\"core\"][\"repository_directory\"]) == repo_dir\n\n def test_resolve_repository_path_relative_path(self, tmp_path):\n \"\"\"Test _resolve_repository_path with relative path.\"\"\"\n # Create actual directory structure\n config_dir = tmp_path / \"configs\"\n config_dir.mkdir()\n repo_dir = tmp_path / \"workspace\"\n repo_dir.mkdir()\n\n self.validator.config = {\"core\": {\"repository_directory\": \"../workspace\"}}\n self.validator.config_path = config_dir / \"config.yaml\"\n\n self.validator._resolve_repository_path()\n\n assert self.validator.errors == []\n resolved_path = Path(self.validator.config[\"core\"][\"repository_directory\"])\n assert resolved_path.is_absolute()\n assert resolved_path.exists()\n\n def test_resolve_repository_path_nonexistent_directory(self, tmp_path):\n \"\"\"Test _resolve_repository_path with nonexistent directory.\"\"\"\n self.validator.config = {\"core\": {\"repository_directory\": \"nonexistent\"}}\n self.validator.config_path = tmp_path / \"config.yaml\"\n\n self.validator._resolve_repository_path()\n\n assert len(self.validator.errors) == 1\n assert \"repository_directory not found at resolved path\" in self.validator.errors[0]\n\n def test_resolve_repository_path_file_instead_of_directory(self, tmp_path):\n \"\"\"Test _resolve_repository_path with file instead of directory.\"\"\"\n # Create a file instead of directory\n not_a_dir = tmp_path / \"not_a_dir.txt\"\n not_a_dir.write_text(\"content\")\n\n self.validator.config = {\"core\": {\"repository_directory\": str(not_a_dir)}}\n self.validator.config_path = tmp_path / \"config.yaml\"\n\n self.validator._resolve_repository_path()\n\n assert len(self.validator.errors) == 1\n assert \"repository_directory path exists but is not a directory\" in self.validator.errors[0]\n\n def test_resolve_repository_path_environment_mapping(self, tmp_path):\n \"\"\"Test _resolve_repository_path with environment mapping.\"\"\"\n # Create actual directories\n dev_repo = tmp_path / \"dev_workspace\"\n dev_repo.mkdir()\n prod_repo = tmp_path / \"prod_workspace\"\n prod_repo.mkdir()\n\n self.validator.config = {\"core\": {\"repository_directory\": {\"dev\": str(dev_repo), \"prod\": str(prod_repo)}}}\n self.validator.config_path = tmp_path / \"config.yaml\"\n\n self.validator._resolve_repository_path()\n\n assert self.validator.errors == []\n repo_dirs = self.validator.config[\"core\"][\"repository_directory\"]\n assert Path(repo_dirs[\"dev\"]).is_absolute()\n assert Path(repo_dirs[\"prod\"]).is_absolute()\n\n def test_validate_parameter_field_valid_configurations(self):\n \"\"\"Test parameter field validation with valid string and environment mapping.\"\"\"\n # Test valid string\n core_string = {\"parameter\": \"parameter.yml\"}\n self.validator._validate_parameter_field(core_string)\n assert self.validator.errors == []\n\n # Reset for next test\n self.validator.errors = []\n\n # Test valid environment mapping\n core_mapping = {\"parameter\": {\"dev\": \"dev-parameter.yml\", \"prod\": \"prod-parameter.yml\"}}\n self.validator._validate_parameter_field(core_mapping)\n assert self.validator.errors == []\n\n def test_validate_parameter_field_invalid_configurations(self):\n \"\"\"Test parameter field validation with invalid configurations.\"\"\"\n # Test empty string\n core_empty = {\"parameter\": \"\"}\n self.validator._validate_parameter_field(core_empty)\n assert len(self.validator.errors) == 1\n assert constants.CONFIG_VALIDATION_MSGS[\"field\"][\"empty_value\"].format(\"parameter\") in self.validator.errors[0]\n\n # Reset for next test\n self.validator.errors = []\n\n # Test invalid type\n core_invalid_type = {\"parameter\": 123}\n self.validator._validate_parameter_field(core_invalid_type)\n assert len(self.validator.errors) == 1\n assert (\n constants.CONFIG_VALIDATION_MSGS[\"field\"][\"string_or_dict\"].format(\"parameter\", \"int\")\n in self.validator.errors[0]\n )\n\n def test_validate_parameter_field_invalid_env_mapping(self):\n \"\"\"Test _validate_parameter_field with invalid environment mapping.\"\"\"\n core = {\"parameter\": {\"\": \"param.yml\"}}\n\n self.validator._validate_parameter_field(core)\n\n assert len(self.validator.errors) == 1\n assert (\n constants.CONFIG_VALIDATION_MSGS[\"environment\"][\"invalid_env_key\"].format(\"parameter\", \"str\")\n in self.validator.errors[0]\n )\n\n def test_resolve_parameter_path_basic_functionality(self, tmp_path):\n \"\"\"Test basic parameter path resolution functionality.\"\"\"\n # Create parameter file\n param_file = tmp_path / \"parameter.yml\"\n param_file.write_text(\"find_replace: []\")\n\n self.validator.config = {\n \"core\": {\n \"workspace_id\": \"12345678-1234-1234-1234-123456789abc\",\n \"repository_directory\": \"workspace\",\n \"parameter\": \"parameter.yml\",\n }\n }\n self.validator.config_path = tmp_path / \"config.yml\"\n\n self.validator._resolve_parameter_path()\n\n assert self.validator.errors == []\n resolved_path = Path(self.validator.config[\"core\"][\"parameter\"])\n assert resolved_path.is_absolute()\n assert resolved_path.exists()\n assert resolved_path.name == \"parameter.yml\"\n\n def test_resolve_path_field_directory_relative_path(self, tmp_path):\n \"\"\"Test _resolve_path_field with relative directory path.\"\"\"\n # Create directory\n test_dir = tmp_path / \"test_dir\"\n test_dir.mkdir()\n\n self.validator.config = {\"test_section\": {\"test_field\": \"test_dir\"}}\n self.validator.config_path = tmp_path / \"config.yml\"\n\n self.validator._resolve_path_field(\"test_dir\", \"test_field\", \"test_section\", \"directory\")\n\n assert self.validator.errors == []\n resolved_path = Path(self.validator.config[\"test_section\"][\"test_field\"])\n assert resolved_path.is_absolute()\n assert resolved_path.exists()\n assert resolved_path.is_dir()\n\n def test_resolve_path_field_file_absolute_path(self, tmp_path):\n \"\"\"Test _resolve_path_field with absolute file path.\"\"\"\n # Create file\n test_file = tmp_path / \"test_file.txt\"\n test_file.write_text(\"test content\")\n\n self.validator.config = {\"test_section\": {\"test_field\": str(test_file)}}\n self.validator.config_path = tmp_path / \"config.yml\"\n\n self.validator._resolve_path_field(str(test_file), \"test_field\", \"test_section\", \"file\")\n\n assert self.validator.errors == []\n resolved_path = Path(self.validator.config[\"test_section\"][\"test_field\"])\n assert resolved_path.is_absolute()\n assert resolved_path.exists()\n assert resolved_path.is_file()\n\n def test_resolve_path_field_git_repo_mismatch(self, tmp_path):\n \"\"\"Test _resolve_path_field detects different git repositories.\"\"\"\n test_dir = tmp_path / \"test_dir\"\n test_dir.mkdir()\n\n self.validator.config = {\"test_section\": {\"test_field\": str(test_dir)}}\n self.validator.config_path = tmp_path / \"config.yml\"\n\n with patch(\n \"fabric_cicd._common._config_validator._find_git_root\",\n side_effect=[Path(\"/repo1\"), Path(\"/repo2\")],\n ):\n self.validator._resolve_path_field(str(test_dir), \"test_field\", \"test_section\", \"directory\")\n\n assert len(self.validator.errors) == 1\n assert \"same git repository\" in self.validator.errors[0]\n\n def test_resolve_path_field_file_type_but_is_directory(self, tmp_path):\n \"\"\"Test _resolve_path_field when expecting file but path is a directory.\"\"\"\n test_dir = tmp_path / \"a_dir\"\n test_dir.mkdir()\n\n self.validator.config = {\"test_section\": {\"test_field\": \"a_dir\"}}\n self.validator.config_path = tmp_path / \"config.yml\"\n\n self.validator._resolve_path_field(\"a_dir\", \"test_field\", \"test_section\", \"file\")\n\n assert len(self.validator.errors) == 1\n assert \"test_field path exists but is not a file\" in self.validator.errors[0]\n\n def test_resolve_path_field_os_error(self, tmp_path):\n \"\"\"Test _resolve_path_field with OSError during path resolution.\"\"\"\n self.validator.config = {\"test_section\": {\"test_field\": \"some_path\"}}\n self.validator.config_path = tmp_path / \"config.yml\"\n\n with patch.object(Path, \"resolve\", side_effect=OSError(\"bad path\")):\n self.validator._resolve_path_field(\"some_path\", \"test_field\", \"test_section\", \"directory\")\n\n assert len(self.validator.errors) == 1\n assert \"Invalid test_field path\" in self.validator.errors[0]\n assert \"some_path\" in self.validator.errors[0]\n assert \"bad path\" in self.validator.errors[0]\n\n def test_resolve_path_field_environment_mapping(self, tmp_path):\n \"\"\"Test _resolve_path_field with environment mapping.\"\"\"\n self.validator.environment = \"DEV\"\n\n # Create directories for different environments\n dev_dir = tmp_path / \"dev_dir\"\n prod_dir = tmp_path / \"prod_dir\"\n dev_dir.mkdir()\n prod_dir.mkdir()\n\n field_value = {\"DEV\": \"dev_dir\", \"PROD\": \"prod_dir\"}\n\n self.validator.config = {\"test_section\": {\"test_field\": field_value}}\n self.validator.config_path = tmp_path / \"config.yml\"\n\n self.validator._resolve_path_field(field_value, \"test_field\", \"test_section\", \"directory\")\n\n assert self.validator.errors == []\n # Only DEV environment should be resolved since that's the target environment\n resolved_path = Path(self.validator.config[\"test_section\"][\"test_field\"][\"DEV\"])\n assert resolved_path.is_absolute()\n assert resolved_path.exists()\n assert resolved_path.is_dir()\n # PROD should remain unchanged since it wasn't the target environment\n assert self.validator.config[\"test_section\"][\"test_field\"][\"PROD\"] == \"prod_dir\"\n\n def test_resolve_path_field_environment_not_in_mapping(self, tmp_path):\n \"\"\"Test _resolve_path_field skips gracefully when environment is not in mapping.\"\"\"\n self.validator.environment = \"prod\" # Target environment\n\n # Create directory only for 'dev' environment\n dev_dir = tmp_path / \"dev_dir\"\n dev_dir.mkdir()\n\n # Parameter mapping only has 'dev', not 'prod'\n field_value = {\"dev\": \"dev_dir\"}\n\n self.validator.config = {\"test_section\": {\"test_field\": field_value}}\n self.validator.config_path = tmp_path / \"config.yml\"\n\n # Should NOT raise KeyError - should skip gracefully\n self.validator._resolve_path_field(field_value, \"test_field\", \"test_section\", \"directory\")\n\n # No errors should be added (optional field behavior)\n assert self.validator.errors == []\n # Config should remain unchanged since resolution was skipped\n assert self.validator.config[\"test_section\"][\"test_field\"] == {\"dev\": \"dev_dir\"}\n\n def test_resolve_path_field_nonexistent_path(self, tmp_path):\n \"\"\"Test _resolve_path_field with nonexistent path.\"\"\"\n self.validator.config = {\"test_section\": {\"test_field\": \"nonexistent_dir\"}}\n self.validator.config_path = tmp_path / \"config.yml\"\n\n self.validator._resolve_path_field(\"nonexistent_dir\", \"test_field\", \"test_section\", \"directory\")\n\n assert len(self.validator.errors) == 1\n assert \"test_field not found at resolved path\" in self.validator.errors[0]\n\n def test_resolve_path_field_wrong_type_file_vs_directory(self, tmp_path):\n \"\"\"Test _resolve_path_field when path exists but is wrong type.\"\"\"\n # Create a file but try to resolve it as a directory\n test_file = tmp_path / \"test_file.txt\"\n test_file.write_text(\"test content\")\n\n self.validator.config = {\"test_section\": {\"test_field\": \"test_file.txt\"}}\n self.validator.config_path = tmp_path / \"config.yml\"\n\n self.validator._resolve_path_field(\"test_file.txt\", \"test_field\", \"test_section\", \"directory\")\n\n assert len(self.validator.errors) == 1\n assert \"test_field path exists but is not a directory\" in self.validator.errors[0]\n\n def test_resolve_path_field_no_config_path(self):\n \"\"\"Test _resolve_path_field when config_path is None (validation failed).\"\"\"\n self.validator.config_path = None # Simulate config validation failure\n\n self.validator.config = {\"test_section\": {\"test_field\": \"test_dir\"}}\n\n self.validator._resolve_path_field(\"test_dir\", \"test_field\", \"test_section\", \"directory\")\n\n # Should skip resolution and not add any errors\n assert self.validator.errors == []\n assert self.validator.config[\"test_section\"][\"test_field\"] == \"test_dir\" # Unchanged\n\n def test_environment_exists_valid(self):\n \"\"\"Test _validate_environment_exists with valid environment.\"\"\"\n self.validator.config = {\"core\": {\"workspace_id\": {\"dev\": \"dev-id\", \"prod\": \"prod-id\"}}}\n self.validator.environment = \"dev\"\n\n self.validator._validate_environment_exists()\n\n assert self.validator.errors == []\n\n def test_environment_exists_missing_environment(self):\n \"\"\"Test _validate_environment_exists with missing environment.\"\"\"\n self.validator.config = {\"core\": {\"workspace_id\": {\"dev\": \"dev-id\", \"prod\": \"prod-id\"}}}\n self.validator.environment = \"test\"\n\n self.validator._validate_environment_exists()\n\n assert len(self.validator.errors) == 1\n assert \"Environment 'test' not found in 'core.workspace_id' mappings\" in self.validator.errors[0]\n\n def test_environment_exists_no_environment_with_mapping(self):\n \"\"\"Test _validate_environment_exists with N/A environment but config has mappings.\"\"\"\n self.validator.config = {\"core\": {\"workspace_id\": {\"dev\": \"dev-id\", \"prod\": \"prod-id\"}}}\n self.validator.environment = \"N/A\"\n\n self.validator._validate_environment_exists()\n\n assert len(self.validator.errors) == 1\n assert \"Configuration contains environment mappings but no environment was provided\" in self.validator.errors[0]\n\n def test_environment_exists_no_environment_no_mapping(self):\n \"\"\"Test _validate_environment_exists with N/A environment and no mappings.\"\"\"\n self.validator.config = {\"core\": {\"workspace_id\": \"single-id\", \"repository_directory\": \"/path/to/repo\"}}\n self.validator.environment = \"N/A\"\n\n self.validator._validate_environment_exists()\n\n assert self.validator.errors == []\n\n # Config Override Tests\n @pytest.mark.parametrize(\n (\"section\", \"value\", \"expected_result\", \"expected_error_msg\"),\n [\n # Valid cases - expect True with no errors\n (\"core\", {\"workspace_id\": \"new-id\"}, True, None),\n (\"features\", [\"feature1\", \"feature2\"], True, None),\n (\"features\", {\"dev\": [\"feature1\"], \"prod\": [\"feature2\"]}, True, None),\n # Publish/Unpublish section cases\n (\"publish\", {\"skip\": False}, True, None),\n (\"publish\", {\"exclude_regex\": \"^TEST.*\"}, True, None),\n (\"publish\", {\"items_to_include\": [\"item1.Notebook\"]}, True, None),\n (\"unpublish\", {\"skip\": True}, True, None),\n (\"unpublish\", {\"exclude_regex\": \"^OLD.*\", \"skip\": False}, True, None),\n # Basic validation only - these pass but would be validated later\n (\"features\", [\"feature1\", 123], True, None), # Contains non-string\n (\"constants\", {123: \"value\"}, True, None), # Invalid key\n (\"constants\", {\"UNKNOWN_CONSTANT\": \"value\"}, True, None), # Unknown constant\n # Invalid cases - expect False with errors\n (\n \"invalid_section\",\n {\"field\": \"value\"},\n False,\n \"Cannot override unsupported config section: 'invalid_section'\",\n ),\n (\"core\", \"invalid_type\", False, \"Override section 'core' must be a dict, got str\"),\n (\"core\", {\"invalid_setting\": \"value\"}, False, \"Cannot override unsupported setting 'core.invalid_setting'\"),\n (\"publish\", \"invalid_type\", False, \"Override section 'publish' must be a dict, got str\"),\n (\"unpublish\", \"invalid_type\", False, \"Override section 'unpublish' must be a dict, got str\"),\n (\n \"publish\",\n {\"invalid_setting\": \"value\"},\n False,\n \"Cannot override unsupported setting 'publish.invalid_setting'\",\n ),\n (\n \"unpublish\",\n {\"invalid_setting\": \"value\"},\n False,\n \"Cannot override unsupported setting 'unpublish.invalid_setting'\",\n ),\n ],\n )\n def test_valid_override_section(self, section, value, expected_result, expected_error_msg):\n \"\"\"Test _valid_override_section with various inputs.\"\"\"\n # Reset errors before each test case\n self.validator.errors = []\n\n result = self.validator._valid_override_section(section, value)\n\n assert result is expected_result\n\n if expected_error_msg:\n assert len(self.validator.errors) == 1\n assert expected_error_msg in self.validator.errors[0]\n else:\n assert self.validator.errors == []\n\n @pytest.mark.parametrize(\n (\"section\", \"initial_config\", \"override_value\", \"expected_result\"),\n [\n # Features replacement\n (\n \"features\",\n {\"features\": [\"existing_feature\"]},\n [\"new_feature1\", \"new_feature2\"],\n {\"features\": [\"new_feature1\", \"new_feature2\"]},\n ),\n # Constants merge\n (\n \"constants\",\n {\"constants\": {\"EXISTING_CONST\": \"existing_value\"}},\n {\"NEW_CONST\": \"new_value\"},\n {\"constants\": {\"NEW_CONST\": \"new_value\"}},\n ),\n # Constants create section\n (\n \"constants\",\n {\"core\": {\"workspace_id\": \"test\"}},\n {\"NEW_CONST\": \"new_value\"},\n {\"core\": {\"workspace_id\": \"test\"}, \"constants\": {\"NEW_CONST\": \"new_value\"}},\n ),\n # Publish section overrides\n (\n \"publish\",\n {\"publish\": {\"skip\": True, \"exclude_regex\": \"^OLD.*\"}},\n {\"skip\": False},\n {\"publish\": {\"skip\": False, \"exclude_regex\": \"^OLD.*\"}},\n ),\n # Unpublish section overrides\n (\n \"unpublish\",\n {\"unpublish\": {\"skip\": False}},\n {\"skip\": True, \"exclude_regex\": \"^TEST.*\"},\n {\"unpublish\": {\"skip\": True, \"exclude_regex\": \"^TEST.*\"}},\n ),\n # Create publish section with multiple settings\n (\n \"publish\",\n {\"core\": {\"workspace_id\": \"test-id\"}},\n {\"skip\": False, \"exclude_regex\": \"^TEST.*\", \"items_to_include\": [\"item1.Notebook\"]},\n {\n \"core\": {\"workspace_id\": \"test-id\"},\n \"publish\": {\"skip\": False, \"exclude_regex\": \"^TEST.*\", \"items_to_include\": [\"item1.Notebook\"]},\n },\n ),\n # Create unpublish section\n (\n \"unpublish\",\n {\"core\": {\"workspace_id\": \"test-id\"}},\n {\"skip\": True},\n {\"core\": {\"workspace_id\": \"test-id\"}, \"unpublish\": {\"skip\": True}},\n ),\n ],\n )\n def test_merge_overrides_basic_sections(self, section, initial_config, override_value, expected_result):\n \"\"\"Test _merge_overrides with various basic section operations.\"\"\"\n self.validator.config = initial_config.copy()\n self.validator.errors = []\n\n self.validator._merge_overrides(section, override_value)\n\n assert self.validator.errors == []\n assert self.validator.config == expected_result\n\n @pytest.mark.parametrize(\n (\"initial_config\", \"override_value\", \"expected_config\", \"expected_error\", \"test_description\"),\n [\n # Direct value override\n (\n {\"core\": {\"workspace_id\": \"original-id\", \"repository_directory\": \"/original/path\"}},\n {\"workspace_id\": \"new-id\"},\n {\"core\": {\"workspace_id\": \"new-id\", \"repository_directory\": \"/original/path\"}},\n None,\n \"Direct value override\",\n ),\n # Environment-specific override\n (\n {\"core\": {\"workspace_id\": {\"dev\": \"original-dev-id\", \"prod\": \"prod-id\"}}},\n {\"workspace_id\": {\"dev\": \"new-dev-id\"}},\n {\"core\": {\"workspace_id\": {\"dev\": \"new-dev-id\", \"prod\": \"prod-id\"}}},\n None,\n \"Environment-specific override\",\n ),\n # Create environment mapping from simple value\n (\n {\"core\": {\"workspace_id\": \"simple-id\"}},\n {\"workspace_id\": {\"dev\": \"new-dev-id\"}},\n {\"core\": {\"workspace_id\": {\"dev\": \"new-dev-id\"}}},\n None,\n \"Create environment mapping\",\n ),\n # Add new optional field\n (\n {\"core\": {\"workspace_id\": \"test-id\"}},\n {\"item_types_in_scope\": [\"Notebook\"]},\n {\"core\": {\"workspace_id\": \"test-id\", \"item_types_in_scope\": [\"Notebook\"]}},\n None,\n \"Add new optional field\",\n ),\n # Create new publish section\n (\n {\"core\": {\"workspace_id\": \"test-id\"}},\n {\"skip\": False},\n {\"core\": {\"workspace_id\": \"test-id\"}, \"publish\": {\"skip\": False}},\n None,\n \"Create new section\",\n ),\n # Environment-specific publish section\n (\n {\"core\": {\"workspace_id\": \"test-id\"}},\n {\"skip\": {\"dev\": False}},\n {\"core\": {\"workspace_id\": \"test-id\"}, \"publish\": {\"skip\": {\"dev\": False}}},\n None,\n \"Environment-specific publish setting\",\n ),\n # Environment-specific unpublish section\n (\n {\"core\": {\"workspace_id\": \"test-id\"}},\n {\"exclude_regex\": {\"dev\": \"^TEST_DEV.*\"}},\n {\"core\": {\"workspace_id\": \"test-id\"}, \"unpublish\": {\"exclude_regex\": {\"dev\": \"^TEST_DEV.*\"}}},\n None,\n \"Environment-specific unpublish setting\",\n ),\n # Cannot create core section\n (\n {\"features\": [\"test\"]},\n {\"workspace_id\": \"test-id\"},\n {\"features\": [\"test\"]}, # Unchanged\n \"Cannot create 'core' section\",\n \"Prevent creating core section\",\n ),\n # Cannot create required repository_directory\n (\n {\"core\": {\"workspace_id\": \"test-id\"}},\n {\"repository_directory\": \"/new/path\"},\n {\"core\": {\"workspace_id\": \"test-id\"}}, # Unchanged\n \"Cannot create required field 'core.repository_directory'\",\n \"Prevent creating required field\",\n ),\n # Can override existing repository_directory\n (\n {\"core\": {\"workspace_id\": \"test-id\", \"repository_directory\": \"/original/path\"}},\n {\"repository_directory\": \"/new/path\"},\n {\"core\": {\"workspace_id\": \"test-id\", \"repository_directory\": \"/new/path\"}},\n None,\n \"Allow overriding existing required field\",\n ),\n ],\n )\n def test_merge_overrides_core_section(\n self, initial_config, override_value, expected_config, expected_error, test_description\n ):\n \"\"\"Test _merge_overrides with core section operations.\"\"\"\n # Set environment for environment mapping tests\n if \"environment\" in test_description.lower():\n self.validator.environment = \"dev\"\n else:\n self.validator.environment = \"N/A\"\n\n self.validator.config = initial_config.copy()\n self.validator.errors = []\n\n section = \"publish\" if \"publish\" in str(expected_config) else \"core\"\n section = \"unpublish\" if \"unpublish\" in str(expected_config) else section\n self.validator._merge_overrides(section, override_value)\n\n if expected_error:\n assert len(self.validator.errors) == 1\n assert expected_error in self.validator.errors[0]\n else:\n assert self.validator.errors == []\n\n assert self.validator.config == expected_config\n\n @pytest.mark.parametrize(\n (\"initial_config\", \"override_value\", \"expected_result\", \"expected_error\"),\n [\n # Cannot create workspace_id without existing workspace\n (\n {\"core\": {\"repository_directory\": \"/path\"}},\n {\"workspace_id\": \"new-id\"},\n {\"core\": {\"repository_directory\": \"/path\"}},\n \"Cannot create workspace identifier 'core.workspace_id'\",\n ),\n # Cannot create workspace without existing workspace_id\n (\n {\"core\": {\"repository_directory\": \"/path\"}},\n {\"workspace\": \"new-workspace\"},\n {\"core\": {\"repository_directory\": \"/path\"}},\n \"Cannot create workspace identifier 'core.workspace'\",\n ),\n # Can create workspace_id when workspace already exists\n (\n {\"core\": {\"workspace\": \"existing-workspace\", \"repository_directory\": \"/path\"}},\n {\"workspace_id\": \"new-id\"},\n {\n \"core\": {\n \"workspace\": \"existing-workspace\",\n \"repository_directory\": \"/path\",\n \"workspace_id\": \"new-id\",\n }\n },\n None,\n ),\n # Can create workspace when workspace_id already exists\n (\n {\"core\": {\"workspace_id\": \"existing-id\", \"repository_directory\": \"/path\"}},\n {\"workspace\": \"new-workspace\"},\n {\n \"core\": {\n \"workspace_id\": \"existing-id\",\n \"repository_directory\": \"/path\",\n \"workspace\": \"new-workspace\",\n }\n },\n None,\n ),\n # Can override existing workspace identifiers\n (\n {\n \"core\": {\n \"workspace_id\": \"original-id\",\n \"workspace\": \"original-workspace\",\n \"repository_directory\": \"/path\",\n }\n },\n {\"workspace_id\": \"new-id\", \"workspace\": \"new-workspace\"},\n {\"core\": {\"workspace_id\": \"new-id\", \"workspace\": \"new-workspace\", \"repository_directory\": \"/path\"}},\n None,\n ),\n ],\n )\n def test_merge_overrides_workspace_identifiers(\n self, initial_config, override_value, expected_result, expected_error\n ):\n \"\"\"Test _merge_overrides with workspace identifier operations.\"\"\"\n self.validator.config = initial_config.copy()\n self.validator.errors = []\n\n self.validator._merge_overrides(\"core\", override_value)\n\n if expected_error:\n assert len(self.validator.errors) == 1\n assert expected_error in self.validator.errors[0]\n else:\n assert self.validator.errors == []\n\n assert self.validator.config == expected_result\n\n @pytest.mark.parametrize(\n (\"initial_config\", \"config_override\", \"expected_config\", \"expected_error\", \"mock_side_effect\"),\n [\n # Successful override\n (\n {\"core\": {\"workspace_id\": \"original-id\"}},\n {\"core\": {\"workspace_id\": \"new-id\"}, \"constants\": {\"DEFAULT_API_ROOT_URL\": \"https://api.test.com\"}},\n {\"core\": {\"workspace_id\": \"new-id\"}, \"constants\": {\"DEFAULT_API_ROOT_URL\": \"https://api.test.com\"}},\n None,\n None,\n ),\n # Publish and unpublish sections\n (\n {\"core\": {\"workspace_id\": \"original-id\"}},\n {\n \"publish\": {\"skip\": False, \"exclude_regex\": \"^TEST.*\"},\n \"unpublish\": {\"skip\": True, \"items_to_include\": [\"item1.Notebook\"]},\n },\n {\n \"core\": {\"workspace_id\": \"original-id\"},\n \"publish\": {\"skip\": False, \"exclude_regex\": \"^TEST.*\"},\n \"unpublish\": {\"skip\": True, \"items_to_include\": [\"item1.Notebook\"]},\n },\n None,\n None,\n ),\n # Empty publish section (should be rejected in real validation)\n (\n {\"core\": {\"workspace_id\": \"original-id\"}},\n {\"publish\": {}},\n {\"core\": {\"workspace_id\": \"original-id\"}, \"publish\": {}},\n None, # No error at override level, but would fail in section validation\n None,\n ),\n # Validation failure\n (\n {\"core\": {\"workspace_id\": \"original-id\"}},\n {\"core\": {\"invalid_setting\": \"value\"}},\n {\"core\": {\"workspace_id\": \"original-id\"}}, # Config remains unchanged\n \"Cannot override unsupported setting 'core.invalid_setting'\",\n None,\n ),\n # Exception handling\n (\n {\"core\": {\"workspace_id\": \"original-id\"}},\n {\"core\": {\"workspace_id\": \"new-id\"}},\n {\"core\": {\"workspace_id\": \"original-id\"}}, # Config remains unchanged\n \"Failed to apply config override for section 'core': Test exception\",\n Exception(\"Test exception\"),\n ),\n # No overrides\n (\n {\"core\": {\"workspace_id\": \"original-id\"}},\n None,\n {\"core\": {\"workspace_id\": \"original-id\"}}, # Config remains unchanged\n None,\n None,\n ),\n ],\n )\n def test_apply_and_validate_overrides(\n self, initial_config, config_override, expected_config, expected_error, mock_side_effect\n ):\n \"\"\"Test _apply_and_validate_overrides with various scenarios.\"\"\"\n self.validator.config = initial_config.copy()\n self.validator.config_override = config_override\n self.validator.errors = []\n\n if mock_side_effect:\n with patch.object(self.validator, \"_merge_overrides\", side_effect=mock_side_effect):\n self.validator._apply_and_validate_overrides()\n else:\n self.validator._apply_and_validate_overrides()\n\n if expected_error:\n assert len(self.validator.errors) == 1\n assert expected_error in self.validator.errors[0]\n else:\n assert self.validator.errors == []\n\n assert self.validator.config == expected_config\n\n def test_validate_config_file_with_overrides_integration(self, tmp_path):\n \"\"\"Integration test for validate_config_file with config overrides.\"\"\"\n # Create a valid config file\n config_content = \"\"\"\ncore:\n workspace_id: \"12345678-1234-1234-1234-123456789abc\"\n repository_directory: \"workspace\"\nconstants:\n DEFAULT_API_ROOT_URL: \"https://api.fabric.microsoft.com\"\n\"\"\"\n config_file = tmp_path / \"config.yaml\"\n config_file.write_text(config_content)\n\n # Create workspace directory\n workspace_dir = tmp_path / \"workspace\"\n workspace_dir.mkdir()\n\n config_override = {\n \"core\": {\"workspace_id\": \"87654321-4321-4321-4321-123456789abc\"},\n \"constants\": {\"DEFAULT_API_ROOT_URL\": \"https://api.powerbi.com\"},\n }\n\n result = self.validator.validate_config_file(str(config_file), \"DEV\", config_override)\n\n assert result[\"core\"][\"workspace_id\"] == \"87654321-4321-4321-4321-123456789abc\"\n assert result[\"constants\"][\"DEFAULT_API_ROOT_URL\"] == \"https://api.powerbi.com\"\n\n def test_validate_config_file_with_invalid_overrides_integration(self, tmp_path):\n \"\"\"Integration test for validate_config_file with invalid overrides.\"\"\"\n # Create a valid config file\n config_content = \"\"\"\ncore:\n workspace_id: \"12345678-1234-1234-1234-123456789abc\"\n repository_directory: \"workspace\"\n\"\"\"\n config_file = tmp_path / \"config.yaml\"\n config_file.write_text(config_content)\n\n # Create workspace directory\n workspace_dir = tmp_path / \"workspace\"\n workspace_dir.mkdir()\n\n config_override = {\"core\": {\"invalid_field\": \"value\"}}\n\n with pytest.raises(ConfigValidationError) as exc_info:\n self.validator.validate_config_file(str(config_file), \"DEV\", config_override)\n\n assert \"Cannot override unsupported setting 'core.invalid_field'\" in str(exc_info.value)\n\n\n# Tests for utility functions\nclass TestConfigValidatorUtilityFunctions:\n \"\"\"Tests for standalone utility functions in the config validator module.\"\"\"\n\n def test_find_git_root_with_git_repo(self, tmp_path):\n \"\"\"Test _find_git_root when path is in a git repository.\"\"\"\n from fabric_cicd._common._config_validator import _find_git_root\n\n # Create a fake git repo structure\n git_dir = tmp_path / \".git\"\n git_dir.mkdir()\n\n # Test from root\n result = _find_git_root(tmp_path)\n assert result == tmp_path\n\n # Test from subdirectory\n sub_dir = tmp_path / \"subdir\" / \"deep\"\n sub_dir.mkdir(parents=True)\n result = _find_git_root(sub_dir)\n assert result == tmp_path\n\n def test_find_git_root_no_git_repo(self, tmp_path):\n \"\"\"Test _find_git_root when path is not in a git repository.\"\"\"\n from fabric_cicd._common._config_validator import _find_git_root\n\n result = _find_git_root(tmp_path)\n assert result is None\n\n def test_validate_guid_format_valid(self):\n \"\"\"Test _validate_guid_format with valid GUIDs.\"\"\"\n from fabric_cicd._common._config_validator import _validate_guid_format\n\n valid_guids = [\n \"12345678-1234-1234-1234-123456789abc\",\n \"ABCDEF12-3456-7890-ABCD-EF1234567890\",\n \"00000000-0000-0000-0000-000000000000\",\n ]\n\n for guid in valid_guids:\n assert _validate_guid_format(guid) is True\n\n def test_validate_guid_format_invalid(self):\n \"\"\"Test _validate_guid_format with invalid GUIDs.\"\"\"\n from fabric_cicd._common._config_validator import _validate_guid_format\n\n invalid_guids = [\n \"invalid-guid\",\n \"12345678-1234-1234-1234\", # too short\n \"12345678-1234-1234-1234-123456789abcd\", # too long\n \"12345678_1234_1234_1234_123456789abc\", # wrong separators\n \"\",\n \"not-a-guid-at-all\",\n ]\n\n for guid in invalid_guids:\n assert _validate_guid_format(guid) is False\n\n def test_get_config_fields_complete_config(self):\n \"\"\"Test _get_config_fields with complete configuration.\"\"\"\n from fabric_cicd._common._config_validator import _get_config_fields\n\n config = {\n \"core\": {\n \"workspace_id\": \"test-id\",\n \"workspace\": \"test-workspace\",\n \"repository_directory\": \"/path\",\n \"item_types_in_scope\": [\"Notebook\"],\n \"parameter\": \"param.yml\",\n },\n \"publish\": {\n \"exclude_regex\": \".*_test\",\n \"folder_exclude_regex\": \"^/temp\",\n \"folder_path_to_include\": [\"/subfolder\"],\n \"shortcut_exclude_regex\": \"^shortcut_temp/\",\n \"items_to_include\": [\"item1\"],\n \"skip\": False,\n },\n \"unpublish\": {\"exclude_regex\": \".*_old\", \"items_to_include\": [\"item2\"], \"skip\": True},\n \"features\": [\"feature1\"],\n \"constants\": {\"KEY\": \"value\"},\n }\n\n fields = _get_config_fields(config)\n\n # Should return all fields from all sections\n assert len(fields) == 16 # Updated count with folder_exclude_regex and shortcut_exclude_regex\n\n # Check some specific fields\n field_names = [field[1] for field in fields]\n assert \"workspace_id\" in field_names\n assert \"repository_directory\" in field_names\n assert \"parameter\" in field_names\n assert \"folder_exclude_regex\" in field_names\n assert \"shortcut_exclude_regex\" in field_names\n assert \"features\" in field_names\n assert \"constants\" in field_names\n\n # Check required vs optional flags\n for _section, field_name, _display_name, is_required, warn_if_missing in fields:\n if field_name in [\"workspace_id\", \"workspace\", \"repository_directory\"]:\n assert is_required is True, f\"{field_name} should be required\"\n else:\n assert is_required is False, f\"{field_name} should be optional\"\n\n if field_name in [\"item_types_in_scope\", \"parameter\"]:\n assert warn_if_missing is True, f\"{field_name} should warn if missing\"\n\n\nclass TestConfigValidatorIntegration:\n \"\"\"Integration tests for ConfigValidator.validate_config_file method.\"\"\"\n\n def test_validate_config_file_complete_success(self, tmp_path):\n \"\"\"Test validate_config_file with complete valid configuration.\"\"\"\n # Create actual directory structure\n repo_dir = tmp_path / \"workspace\"\n repo_dir.mkdir()\n\n config_data = {\n \"core\": {\n \"workspace_id\": {\"dev\": \"8b6e2c7a-4c1f-4e3a-9b2e-7d8f2e1a6c3b\"},\n \"repository_directory\": \"workspace\",\n \"item_types_in_scope\": [\"Notebook\", \"DataPipeline\"],\n },\n \"publish\": {\"exclude_regex\": \"^DONT_DEPLOY.*\", \"skip\": {\"dev\": False}},\n }\n\n config_file = tmp_path / \"config.yaml\"\n with Path.open(config_file, \"w\") as f:\n yaml.dump(config_data, f)\n\n validator = ConfigValidator()\n result = validator.validate_config_file(str(config_file), \"dev\")\n\n assert result is not None\n assert \"core\" in result\n assert \"publish\" in result\n # Path should be resolved to absolute\n assert Path(result[\"core\"][\"repository_directory\"]).is_absolute()\n\n def test_validate_config_file_accumulates_errors(self, tmp_path):\n \"\"\"Test validate_config_file accumulates multiple errors.\"\"\"\n config_data = {\n \"core\": {\n \"workspace_id\": 123, # Invalid type\n \"item_types_in_scope\": [\"InvalidType\"], # Invalid item type\n }\n # Missing repository_directory\n }\n\n config_file = tmp_path / \"config.yaml\"\n with Path.open(config_file, \"w\") as f:\n yaml.dump(config_data, f)\n\n validator = ConfigValidator()\n\n with pytest.raises(ConfigValidationError) as exc_info:\n validator.validate_config_file(str(config_file), \"dev\")\n\n # Should have multiple errors\n assert len(exc_info.value.validation_errors) >= 3\n error_messages = \" \".join(exc_info.value.validation_errors)\n assert \"must be either a string or environment mapping\" in error_messages\n assert \"must specify 'repository_directory'\" in error_messages\n assert \"Invalid item type 'InvalidType'\" in error_messages\n\n def test_validate_config_file_stops_at_yaml_parse_error(self, tmp_path):\n \"\"\"Test validate_config_file stops at YAML parse error.\"\"\"\n config_file = tmp_path / \"config.yaml\"\n config_file.write_text(\"invalid: yaml: [\")\n\n validator = ConfigValidator()\n\n with pytest.raises(ConfigValidationError) as exc_info:\n validator.validate_config_file(str(config_file), \"dev\")\n\n assert len(exc_info.value.validation_errors) == 1\n assert \"Invalid YAML syntax:\" in exc_info.value.validation_errors[0]\n\n def test_validate_config_file_catches_guid_and_constants_errors(self, tmp_path):\n \"\"\"Test validate_config_file catches GUID format and unknown constants errors.\"\"\"\n config_data = {\n \"core\": {\n \"workspace_id\": {\n \"dev\": \"invalid-guid-format\", # Invalid GUID\n \"prod\": \"8b6e2c7a-4c1f-4e3a-9b2e-7d8f2e1a6c3b\", # Valid GUID\n },\n \"repository_directory\": \"workspace\",\n \"item_types_in_scope\": [\"Notebook\"],\n },\n \"constants\": {\n \"UNKNOWN_CONSTANT\": \"some_value\", # Unknown constant\n \"DEFAULT_API_ROOT_URL\": \"https://api.fabric.microsoft.com\", # Valid URL constant\n },\n }\n\n config_file = tmp_path / \"config.yaml\"\n with Path.open(config_file, \"w\") as f:\n yaml.dump(config_data, f)\n\n validator = ConfigValidator()\n\n with pytest.raises(ConfigValidationError) as exc_info:\n validator.validate_config_file(str(config_file), \"dev\")\n\n # Should catch both GUID format error and unknown constant error\n assert len(exc_info.value.validation_errors) >= 2\n error_messages = \" \".join(exc_info.value.validation_errors)\n assert \"must be a valid GUID format\" in error_messages\n assert \"Unknown constant 'UNKNOWN_CONSTANT'\" in error_messages\n\n\nclass TestConfigSectionValidation:\n \"\"\"Tests for section validation - required vs optional sections.\"\"\"\n\n def setup_method(self):\n \"\"\"Set up for each test method.\"\"\"\n self.validator = ConfigValidator()\n\n def test_validate_config_sections_missing_core(self):\n \"\"\"Test _validate_config_sections with missing core section.\"\"\"\n self.validator.config = {\"publish\": {\"skip\": False}, \"unpublish\": {\"skip\": True}}\n\n self.validator._validate_config_sections()\n\n assert len(self.validator.errors) == 1\n assert \"Configuration must contain a 'core' section\" in self.validator.errors[0]\n\n def test_validate_config_sections_core_not_dict(self):\n \"\"\"Test _validate_config_sections with core section not being a dictionary.\"\"\"\n self.validator.config = {\"core\": \"not a dict\"}\n\n self.validator._validate_config_sections()\n\n assert len(self.validator.errors) == 1\n assert \"Configuration must contain a 'core' section\" in self.validator.errors[0]\n\n def test_validate_config_sections_core_only(self):\n \"\"\"Test _validate_config_sections with only required core section.\"\"\"\n self.validator.config = {\n \"core\": {\"workspace_id\": \"8b6e2c7a-4c1f-4e3a-9b2e-7d8f2e1a6c3b\", \"repository_directory\": \"/path/to/repo\"}\n }\n\n self.validator._validate_config_sections()\n\n assert self.validator.errors == []\n\n def test_validate_config_sections_with_optional_sections(self):\n \"\"\"Test _validate_config_sections with optional sections present.\"\"\"\n self.validator.config = {\n \"core\": {\"workspace_id\": \"8b6e2c7a-4c1f-4e3a-9b2e-7d8f2e1a6c3b\", \"repository_directory\": \"/path/to/repo\"},\n \"publish\": {\"skip\": False},\n \"unpublish\": {\"skip\": True},\n \"features\": [\"enable_shortcut_publish\"],\n \"constants\": {\"DEFAULT_API_ROOT_URL\": \"https://api.example.com\"},\n }\n\n with patch.object(constants, \"DEFAULT_API_ROOT_URL\", \"original_value\"):\n self.validator._validate_config_sections()\n\n assert self.validator.errors == [\"'constants.DEFAULT_API_ROOT_URL' has invalid hostname: api.example.com\"]\n\n def test_validate_config_sections_missing_workspace_identifier(self):\n \"\"\"Test _validate_config_sections with missing workspace identifier.\"\"\"\n self.validator.config = {\"core\": {\"repository_directory\": \"/path/to/repo\"}}\n\n self.validator._validate_config_sections()\n\n assert len(self.validator.errors) == 1\n assert \"Configuration must specify either 'workspace_id' or 'workspace'\" in self.validator.errors[0]\n\n\nclass TestOperationSectionValidation:\n \"\"\"Tests for publish/unpublish operation section validation.\"\"\"\n\n def setup_method(self):\n \"\"\"Set up for each test method.\"\"\"\n self.validator = ConfigValidator()\n\n def test_validate_operation_section_valid_basic(self):\n \"\"\"Test _validate_operation_section with valid basic configuration.\"\"\"\n section = {\n \"exclude_regex\": \"^TEST.*\",\n \"items_to_include\": [\"item1.Notebook\", \"item2.DataPipeline\"],\n \"skip\": False,\n }\n\n self.validator._validate_operation_section(section, \"publish\")\n\n assert self.validator.errors == []\n\n def test_validate_operation_section_not_dict(self):\n \"\"\"Test _validate_operation_section with non-dictionary section.\"\"\"\n section = \"not a dict\"\n\n self.validator._validate_operation_section(section, \"publish\")\n\n assert len(self.validator.errors) == 1\n assert \"'publish' section must be a dictionary\" in self.validator.errors[0]\n\n # --- Parametrized regex field tests ---\n\n @pytest.mark.parametrize(\"regex_field\", [\"exclude_regex\", \"folder_exclude_regex\", \"shortcut_exclude_regex\"])\n def test_validate_operation_section_valid_regex_field(self, regex_field):\n \"\"\"Test _validate_operation_section with valid regex field.\"\"\"\n section = {regex_field: \"^DONT_DEPLOY.*\"}\n\n self.validator._validate_operation_section(section, \"publish\")\n\n assert self.validator.errors == []\n\n @pytest.mark.parametrize(\"regex_field\", [\"exclude_regex\", \"folder_exclude_regex\", \"shortcut_exclude_regex\"])\n def test_validate_operation_section_invalid_regex_field(self, regex_field):\n \"\"\"Test _validate_operation_section with invalid regex field.\"\"\"\n section = {regex_field: \"[invalid\"}\n\n self.validator._validate_operation_section(section, \"publish\")\n\n assert len(self.validator.errors) == 1\n assert \"is not a valid regex pattern\" in self.validator.errors[0]\n\n @pytest.mark.parametrize(\n (\"regex_field\", \"empty_value\"),\n [\n (\"exclude_regex\", \"\"),\n (\"folder_exclude_regex\", \" \"),\n (\"shortcut_exclude_regex\", \" \"),\n ],\n )\n def test_validate_operation_section_empty_regex_field(self, regex_field, empty_value):\n \"\"\"Test _validate_operation_section with empty regex field.\"\"\"\n section = {regex_field: empty_value}\n\n self.validator._validate_operation_section(section, \"publish\")\n\n assert len(self.validator.errors) == 1\n assert \"empty\" in self.validator.errors[0].lower()\n\n @pytest.mark.parametrize(\"regex_field\", [\"exclude_regex\", \"folder_exclude_regex\", \"shortcut_exclude_regex\"])\n def test_validate_operation_section_regex_field_invalid_type(self, regex_field):\n \"\"\"Test _validate_operation_section with regex field invalid type.\"\"\"\n section = {regex_field: 123}\n\n self.validator._validate_operation_section(section, \"publish\")\n\n assert len(self.validator.errors) == 1\n assert \"must be either a string or environment mapping\" in self.validator.errors[0]\n\n @pytest.mark.parametrize(\"regex_field\", [\"exclude_regex\", \"folder_exclude_regex\", \"shortcut_exclude_regex\"])\n def test_validate_operation_section_regex_field_environment_mapping(self, regex_field):\n \"\"\"Test _validate_operation_section with regex field environment mapping.\"\"\"\n section = {regex_field: {\"dev\": \"^DEV_.*\", \"prod\": \"^PROD_.*\"}}\n\n self.validator._validate_operation_section(section, \"publish\")\n\n assert self.validator.errors == []\n\n @pytest.mark.parametrize(\"regex_field\", [\"exclude_regex\", \"folder_exclude_regex\", \"shortcut_exclude_regex\"])\n def test_validate_operation_section_regex_field_invalid_env_mapping(self, regex_field):\n \"\"\"Test regex field dict with invalid environment key causes early return.\"\"\"\n section = {regex_field: {123: \"^pattern\"}}\n self.validator._validate_operation_section(section, \"publish\")\n assert len(self.validator.errors) == 1\n assert (\n constants.CONFIG_VALIDATION_MSGS[\"environment\"][\"invalid_env_key\"].format(f\"publish.{regex_field}\", \"int\")\n in self.validator.errors[0]\n )\n\n @pytest.mark.parametrize(\"regex_field\", [\"folder_exclude_regex\", \"shortcut_exclude_regex\"])\n def test_validate_operation_section_regex_field_rejected_in_unpublish(self, regex_field):\n \"\"\"Test publish-only regex fields are rejected in unpublish section.\"\"\"\n section = {regex_field: \"^temp\"}\n self.validator._validate_operation_section(section, \"unpublish\")\n assert len(self.validator.errors) == 1\n assert regex_field in self.validator.errors[0]\n assert \"unpublish\" in self.validator.errors[0]\n\n # --- items_to_include tests ---\n\n def test_validate_operation_section_empty_items_to_include(self):\n \"\"\"Test _validate_operation_section with empty items_to_include list.\"\"\"\n section = {\"items_to_include\": []}\n\n self.validator._validate_operation_section(section, \"publish\")\n\n assert len(self.validator.errors) == 1\n assert (\n constants.CONFIG_VALIDATION_MSGS[\"operation\"][\"empty_list\"].format(\"publish.items_to_include\")\n in self.validator.errors[0]\n )\n\n def test_validate_operation_section_items_to_include_valid_env_mapping(self):\n \"\"\"Test _validate_operation_section with valid items_to_include environment mapping.\"\"\"\n section = {\"items_to_include\": {\"dev\": [\"item1.Notebook\"], \"prod\": [\"item2.DataPipeline\"]}}\n\n self.validator._validate_operation_section(section, \"publish\")\n\n assert self.validator.errors == []\n\n def test_validate_operation_section_items_to_include_invalid_env_mapping(self):\n \"\"\"Test items_to_include dict with invalid environment key causes early return.\"\"\"\n section = {\"items_to_include\": {123: [\"item1\"]}}\n self.validator._validate_operation_section(section, \"publish\")\n assert len(self.validator.errors) == 1\n assert (\n constants.CONFIG_VALIDATION_MSGS[\"environment\"][\"invalid_env_key\"].format(\"publish.items_to_include\", \"int\")\n in self.validator.errors[0]\n )\n\n def test_validate_operation_section_items_to_include_invalid_type(self):\n \"\"\"Test _validate_operation_section with items_to_include invalid type.\"\"\"\n section = {\"items_to_include\": \"not a list or dict\"}\n\n self.validator._validate_operation_section(section, \"publish\")\n\n assert len(self.validator.errors) == 1\n assert (\n constants.CONFIG_VALIDATION_MSGS[\"field\"][\"list_or_dict\"].format(\"publish.items_to_include\", \"str\")\n in self.validator.errors[0]\n )\n\n # --- skip tests ---\n\n def test_validate_operation_section_skip_boolean(self):\n \"\"\"Test _validate_operation_section with skip as boolean.\"\"\"\n section = {\"skip\": True}\n\n self.validator._validate_operation_section(section, \"unpublish\")\n\n assert self.validator.errors == []\n\n def test_validate_operation_section_skip_environment_mapping(self):\n \"\"\"Test _validate_operation_section with skip environment mapping.\"\"\"\n section = {\"skip\": {\"dev\": True, \"test\": False, \"prod\": False}}\n\n self.validator._validate_operation_section(section, \"unpublish\")\n\n assert self.validator.errors == []\n\n def test_validate_operation_section_skip_invalid_type(self):\n \"\"\"Test _validate_operation_section with skip invalid type.\"\"\"\n section = {\"skip\": \"not a boolean\"}\n\n self.validator._validate_operation_section(section, \"unpublish\")\n\n assert len(self.validator.errors) == 1\n modified_msg = (\n constants.CONFIG_VALIDATION_MSGS[\"field\"][\"string_or_dict\"]\n .format(\"unpublish.skip\", \"str\")\n .replace(\"a string\", \"a boolean\")\n )\n assert modified_msg in self.validator.errors[0]\n\n def test_validate_operation_section_skip_invalid_env_mapping(self):\n \"\"\"Test skip dict with invalid environment key causes early return.\"\"\"\n section = {\"skip\": {123: True}}\n self.validator._validate_operation_section(section, \"publish\")\n assert len(self.validator.errors) == 1\n assert (\n constants.CONFIG_VALIDATION_MSGS[\"environment\"][\"invalid_env_key\"].format(\"publish.skip\", \"int\")\n in self.validator.errors[0]\n )\n\n # --- folder_path_to_include tests ---\n\n def test_validate_operation_section_folder_path_to_include_valid_list(self):\n \"\"\"Test _validate_operation_section with valid folder_path_to_include list.\"\"\"\n section = {\"folder_path_to_include\": [\"/FolderA\", \"/FolderB\"]}\n\n self.validator._validate_operation_section(section, \"publish\")\n\n assert len(self.validator.errors) == 0\n\n def test_validate_operation_section_folder_path_to_include_valid_env_mapping(self):\n \"\"\"Test _validate_operation_section with valid folder_path_to_include environment mapping.\"\"\"\n section = {\"folder_path_to_include\": {\"dev\": [\"/FolderA\"], \"prod\": [\"/FolderA\", \"/FolderB\"]}}\n\n self.validator._validate_operation_section(section, \"publish\")\n\n assert len(self.validator.errors) == 0\n\n def test_validate_operation_section_folder_path_to_include_empty_list(self):\n \"\"\"Test _validate_operation_section with empty folder_path_to_include list.\"\"\"\n section = {\"folder_path_to_include\": []}\n\n self.validator._validate_operation_section(section, \"publish\")\n\n assert len(self.validator.errors) == 1\n assert (\n constants.CONFIG_VALIDATION_MSGS[\"operation\"][\"empty_list\"].format(\"publish.folder_path_to_include\")\n in self.validator.errors[0]\n )\n\n def test_validate_operation_section_folder_path_to_include_invalid_type(self):\n \"\"\"Test _validate_operation_section with folder_path_to_include invalid type.\"\"\"\n section = {\"folder_path_to_include\": \"not a list or dict\"}\n\n self.validator._validate_operation_section(section, \"publish\")\n\n assert len(self.validator.errors) == 1\n assert (\n constants.CONFIG_VALIDATION_MSGS[\"field\"][\"list_or_dict\"].format(\"publish.folder_path_to_include\", \"str\")\n in self.validator.errors[0]\n )\n\n def test_validate_operation_section_folder_path_to_include_unsupported_in_unpublish(self):\n \"\"\"Test _validate_operation_section with folder_path_to_include in unpublish section.\"\"\"\n section = {\"folder_path_to_include\": [\"/FolderA\"]}\n\n self.validator._validate_operation_section(section, \"unpublish\")\n\n assert len(self.validator.errors) == 1\n assert (\n constants.CONFIG_VALIDATION_MSGS[\"operation\"][\"unsupported_field\"].format(\n \"folder_path_to_include\", \"unpublish\"\n )\n in self.validator.errors[0]\n )\n\n def test_validate_operation_section_folder_path_to_include_entry_not_string(self):\n \"\"\"Test _validate_operation_section with non-string entry in folder_path_to_include.\"\"\"\n section = {\"folder_path_to_include\": [\"/FolderA\", 123, \"/FolderB\"]}\n\n self.validator._validate_operation_section(section, \"publish\")\n\n assert len(self.validator.errors) == 1\n assert (\n constants.CONFIG_VALIDATION_MSGS[\"operation\"][\"list_entry_type\"].format(\n \"publish.folder_path_to_include\", 1, \"int\"\n )\n in self.validator.errors[0]\n )\n\n def test_validate_operation_section_folder_path_to_include_empty_string_entry(self):\n \"\"\"Test _validate_operation_section with empty string entry in folder_path_to_include.\"\"\"\n section = {\"folder_path_to_include\": [\"/FolderA\", \"\", \"/FolderB\"]}\n\n self.validator._validate_operation_section(section, \"publish\")\n\n assert len(self.validator.errors) == 1\n assert (\n constants.CONFIG_VALIDATION_MSGS[\"operation\"][\"list_entry_empty\"].format(\n \"publish.folder_path_to_include\", 1\n )\n in self.validator.errors[0]\n )\n\n def test_validate_operation_section_folder_path_to_include_missing_prefix(self):\n \"\"\"Test _validate_operation_section with folder entry missing leading slash.\"\"\"\n section = {\"folder_path_to_include\": [\"/FolderA\", \"FolderB\"]}\n\n self.validator._validate_operation_section(section, \"publish\")\n\n assert len(self.validator.errors) == 1\n assert (\n constants.CONFIG_VALIDATION_MSGS[\"operation\"][\"folders_list_prefix\"].format(\n \"publish.folder_path_to_include\", 1, \"FolderB\"\n )\n in self.validator.errors[0]\n )\n\n def test_validate_operation_section_folder_path_to_include_env_mapping_empty_list(self):\n \"\"\"Test _validate_operation_section with empty list in environment mapping for folder_path_to_include.\"\"\"\n section = {\"folder_path_to_include\": {\"dev\": [\"/FolderA\"], \"prod\": []}}\n\n self.validator._validate_operation_section(section, \"publish\")\n\n assert len(self.validator.errors) == 1\n assert (\n constants.CONFIG_VALIDATION_MSGS[\"environment\"][\"empty_env_value\"].format(\n \"publish.folder_path_to_include\", \"prod\"\n )\n in self.validator.errors[0]\n )\n\n def test_validate_operation_section_folder_path_to_include_env_mapping_invalid_entry(self):\n \"\"\"Test _validate_operation_section with invalid entry in environment mapping for folder_path_to_include.\"\"\"\n section = {\"folder_path_to_include\": {\"dev\": [\"/FolderA\", \"NoSlash\"]}}\n\n self.validator._validate_operation_section(section, \"publish\")\n\n assert len(self.validator.errors) == 1\n assert (\n constants.CONFIG_VALIDATION_MSGS[\"operation\"][\"folders_list_prefix\"].format(\n \"publish.folder_path_to_include.dev\", 1, \"NoSlash\"\n )\n in self.validator.errors[0]\n )\n\n def test_validate_operation_section_folder_path_to_include_nested_path(self):\n \"\"\"Test _validate_operation_section with nested folder paths in folder_path_to_include.\"\"\"\n section = {\"folder_path_to_include\": [\"/FolderA/SubFolder\", \"/FolderB/Sub1/Sub2\"]}\n\n self.validator._validate_operation_section(section, \"publish\")\n\n assert len(self.validator.errors) == 0\n\n def test_validate_operation_section_folder_path_to_include_whitespace_entry(self):\n \"\"\"Test _validate_operation_section with whitespace-only entry in folder_path_to_include.\"\"\"\n section = {\"folder_path_to_include\": [\"/FolderA\", \" \"]}\n\n self.validator._validate_operation_section(section, \"publish\")\n\n assert len(self.validator.errors) == 1\n assert (\n constants.CONFIG_VALIDATION_MSGS[\"operation\"][\"list_entry_empty\"].format(\n \"publish.folder_path_to_include\", 1\n )\n in self.validator.errors[0]\n )\n\n # --- Mutual exclusivity tests ---\n\n def test_validate_operation_section_mutually_exclusive_both_direct_values(self):\n \"\"\"Test that both folder_exclude_regex and folder_path_to_include as direct values raises error.\"\"\"\n section = {\"folder_exclude_regex\": \"^/legacy\", \"folder_path_to_include\": [\"/subfolder\"]}\n\n self.validator._validate_operation_section(section, \"publish\")\n\n error_messages = \" \".join(self.validator.errors)\n assert (\n constants.CONFIG_VALIDATION_MSGS[\"operation\"][\"mutually_exclusive\"].format(\n \"publish.folder_exclude_regex\", \"publish.folder_path_to_include\"\n )\n in error_messages\n )\n\n def test_validate_operation_section_mutually_exclusive_both_env_mapped_overlapping(self):\n \"\"\"Test that both fields with overlapping environment mappings raises error.\"\"\"\n self.validator.environment = \"dev\"\n section = {\n \"folder_exclude_regex\": {\"dev\": \"^/legacy\"},\n \"folder_path_to_include\": {\"dev\": [\"/subfolder\"]},\n }\n\n self.validator._validate_operation_section(section, \"publish\")\n\n error_messages = \" \".join(self.validator.errors)\n assert (\n constants.CONFIG_VALIDATION_MSGS[\"operation\"][\"mutually_exclusive_env\"].format(\n \"publish.folder_exclude_regex\", \"publish.folder_path_to_include\", [\"dev\"]\n )\n in error_messages\n )\n\n def test_validate_operation_section_mutually_exclusive_both_env_mapped_no_overlap(self):\n \"\"\"Test that both fields with non-overlapping environment mappings is valid.\"\"\"\n self.validator.environment = \"dev\"\n section = {\n \"folder_exclude_regex\": {\"dev\": \"^/legacy\"},\n \"folder_path_to_include\": {\"prod\": [\"/subfolder\"]},\n }\n\n self.validator._validate_operation_section(section, \"publish\")\n\n # Filter errors to only mutual exclusivity errors\n mutual_errors = [\n e for e in self.validator.errors if \"mutually exclusive\" in e.lower() or \"Cannot specify both\" in e\n ]\n assert len(mutual_errors) == 0\n\n def test_validate_operation_section_mutually_exclusive_direct_and_env_mapped_conflict(self):\n \"\"\"Test that direct value + env-mapped value conflicts when target env matches.\"\"\"\n self.validator.environment = \"dev\"\n section = {\n \"folder_exclude_regex\": \"^/legacy\", # direct, applies to all\n \"folder_path_to_include\": {\"dev\": [\"/subfolder\"]}, # env-mapped, applies to dev\n }\n\n self.validator._validate_operation_section(section, \"publish\")\n\n error_messages = \" \".join(self.validator.errors)\n assert (\n constants.CONFIG_VALIDATION_MSGS[\"operation\"][\"mutually_exclusive_env\"].format(\n \"publish.folder_exclude_regex\", \"publish.folder_path_to_include\", [\"dev\"]\n )\n in error_messages\n )\n\n def test_validate_operation_section_mutually_exclusive_direct_and_env_mapped_no_conflict(self):\n \"\"\"Test that direct value + env-mapped value is valid when target env doesn't match.\"\"\"\n self.validator.environment = \"prod\"\n section = {\n \"folder_exclude_regex\": \"^/legacy\", # direct, applies to all\n \"folder_path_to_include\": {\"dev\": [\"/subfolder\"]}, # env-mapped, only dev\n }\n\n self.validator._validate_operation_section(section, \"publish\")\n\n # The direct value applies to prod, but folder_path_to_include doesn't apply to prod\n mutual_errors = [\n e for e in self.validator.errors if \"mutually exclusive\" in e.lower() or \"Cannot specify both\" in e\n ]\n assert len(mutual_errors) == 0\n\n def test_validate_operation_section_mutually_exclusive_env_mapped_and_direct_conflict(self):\n \"\"\"Test that env-mapped value + direct value conflicts when target env matches.\"\"\"\n self.validator.environment = \"dev\"\n section = {\n \"folder_exclude_regex\": {\"dev\": \"^/legacy\"}, # env-mapped, applies to dev\n \"folder_path_to_include\": [\"/subfolder\"], # direct, applies to all\n }\n\n self.validator._validate_operation_section(section, \"publish\")\n\n error_messages = \" \".join(self.validator.errors)\n assert (\n constants.CONFIG_VALIDATION_MSGS[\"operation\"][\"mutually_exclusive_env\"].format(\n \"publish.folder_exclude_regex\", \"publish.folder_path_to_include\", [\"dev\"]\n )\n in error_messages\n )\n\n def test_validate_operation_section_mutually_exclusive_only_one_field_present(self):\n \"\"\"Test that having only one of the mutually exclusive fields is valid.\"\"\"\n section = {\"folder_exclude_regex\": \"^/legacy\"}\n\n self.validator._validate_operation_section(section, \"publish\")\n\n mutual_errors = [\n e for e in self.validator.errors if \"mutually exclusive\" in e.lower() or \"Cannot specify both\" in e\n ]\n assert len(mutual_errors) == 0\n\n\nclass TestFeaturesSectionValidation:\n \"\"\"Tests for features section validation.\"\"\"\n\n def setup_method(self):\n \"\"\"Set up for each test method.\"\"\"\n self.validator = ConfigValidator()\n\n def test_validate_features_section_list(self):\n \"\"\"Test _validate_features_section with list of features.\"\"\"\n features = [\"enable_shortcut_publish\", \"feature2\"]\n\n self.validator._validate_features_section(features)\n\n assert self.validator.errors == []\n\n def test_validate_features_section_empty_list(self):\n \"\"\"Test _validate_features_section with empty list.\"\"\"\n features = []\n\n self.validator._validate_features_section(features)\n\n assert len(self.validator.errors) == 1\n assert \"'features' section cannot be empty if specified\" in self.validator.errors[0]\n\n def test_validate_features_section_environment_mapping(self):\n \"\"\"Test _validate_features_section with environment mapping.\"\"\"\n features = {\"dev\": [\"enable_shortcut_publish\"], \"prod\": [\"feature2\", \"feature3\"]}\n\n self.validator._validate_features_section(features)\n\n assert self.validator.errors == []\n\n def test_validate_features_section_invalid_type(self):\n \"\"\"Test _validate_features_section with invalid type.\"\"\"\n features = \"not a list or dict\"\n\n self.validator._validate_features_section(features)\n\n assert len(self.validator.errors) == 1\n assert constants.CONFIG_VALIDATION_MSGS[\"operation\"][\"features_type\"].format(\"str\") in self.validator.errors[0]\n\n def test_validate_features_section_env_mapping_empty_list_for_env(self):\n \"\"\"Test _validate_features_section with empty feature list for one environment.\"\"\"\n features = {\"dev\": [\"feature1\"], \"prod\": []}\n\n self.validator._validate_features_section(features)\n\n assert len(self.validator.errors) == 1\n assert (\n constants.CONFIG_VALIDATION_MSGS[\"environment\"][\"empty_env_value\"].format(\"features\", \"prod\")\n in self.validator.errors[0]\n )\n\n\nclass TestConstantsSectionValidation:\n \"\"\"Tests for constants section validation.\"\"\"\n\n def setup_method(self):\n \"\"\"Set up for each test method.\"\"\"\n self.validator = ConfigValidator()\n\n def test_validate_constants_section_dict(self):\n \"\"\"Test _validate_constants_section with valid constants dictionary.\"\"\"\n constants_section = {\"DEFAULT_API_ROOT_URL\": \"https://api.fabric.microsoft.com\"}\n\n with patch.object(constants, \"DEFAULT_API_ROOT_URL\", \"original_value\"):\n self.validator._validate_constants_section(constants_section)\n\n assert self.validator.errors == []\n\n def test_validate_constants_section_not_dict(self):\n \"\"\"Test _validate_constants_section with non-dictionary.\"\"\"\n constants_section = \"not a dict\"\n\n self.validator._validate_constants_section(constants_section)\n\n assert len(self.validator.errors) == 1\n assert \"'constants' section must be a dictionary\" in self.validator.errors[0]\n\n def test_validate_constants_section_environment_mapping(self):\n \"\"\"Test _validate_constants_section with per-key environment mapping.\"\"\"\n constants_section = {\n \"DEFAULT_API_ROOT_URL\": {\n \"dev\": \"https://api.fabric.microsoft.com\",\n \"prod\": \"https://api.powerbi.com\",\n },\n }\n\n self.validator._validate_constants_section(constants_section)\n\n assert self.validator.errors == []\n\n def test_validate_constants_section_rejects_env_at_top_format(self):\n \"\"\"Test _validate_constants_section rejects invalid env-at-top format.\"\"\"\n constants_section = {\n \"dev\": {\"DEFAULT_API_ROOT_URL\": \"https://api.fabric.microsoft.com\"},\n \"prod\": {\"DEFAULT_API_ROOT_URL\": \"https://api.powerbi.com\"},\n }\n\n self.validator._validate_constants_section(constants_section)\n\n assert len(self.validator.errors) == 2\n assert \"Unknown constant 'dev'\" in self.validator.errors[0]\n assert \"Unknown constant 'prod'\" in self.validator.errors[1]\n\n def test_validate_constants_section_env_mapping_invalid_env_key(self):\n \"\"\"Test _validate_constants_section with invalid env key in per-key mapping.\"\"\"\n constants_section = {\n \"DEFAULT_API_ROOT_URL\": {123: \"https://api.fabric.microsoft.com\"},\n }\n\n self.validator._validate_constants_section(constants_section)\n\n assert len(self.validator.errors) == 1\n assert (\n constants.CONFIG_VALIDATION_MSGS[\"environment\"][\"invalid_env_key\"].format(\n \"constants.DEFAULT_API_ROOT_URL\", \"int\"\n )\n in self.validator.errors[0]\n )\n assert \"DEFAULT_API_ROOT_URL\" in self.validator.errors[0]\n\n def test_validate_constants_section_env_mapping_url_validation(self):\n \"\"\"Test _validate_constants_section validates URL in per-key env mapping.\"\"\"\n constants_section = {\n \"DEFAULT_API_ROOT_URL\": {\n \"dev\": \"http://api.fabric.microsoft.com\", # HTTP, not HTTPS\n },\n }\n\n self.validator._validate_constants_section(constants_section)\n\n assert len(self.validator.errors) == 1\n assert \"must use HTTPS scheme\" in self.validator.errors[0]\n assert \"DEFAULT_API_ROOT_URL\" in self.validator.errors[0]\n\n def test_validate_constants_section_env_mapping_non_string_url(self):\n \"\"\"Test _validate_constants_section rejects non-string URL in per-key env mapping.\"\"\"\n constants_section = {\n \"DEFAULT_API_ROOT_URL\": {\"dev\": 12345},\n }\n\n self.validator._validate_constants_section(constants_section)\n\n assert len(self.validator.errors) == 1\n assert \"must be a string URL\" in self.validator.errors[0]\n assert \"DEFAULT_API_ROOT_URL\" in self.validator.errors[0]\n assert \"int\" in self.validator.errors[0]\n\n\nclass TestEnvironmentMismatchValidation:\n \"\"\"Tests for environment mismatch scenarios.\"\"\"\n\n def setup_method(self):\n \"\"\"Set up for each test method.\"\"\"\n self.validator = ConfigValidator()\n\n def test_environment_mismatch_in_workspace_id(self):\n \"\"\"Test environment exists validation with mismatch in workspace_id.\"\"\"\n self.validator.config = {\n \"core\": {\"workspace_id\": {\"dev\": \"dev-id\", \"prod\": \"prod-id\"}, \"repository_directory\": \"/path/to/repo\"}\n }\n self.validator.environment = \"staging\" # Not in the mapping\n\n self.validator._validate_environment_exists()\n\n assert len(self.validator.errors) == 1\n assert \"Environment 'staging' not found in 'core.workspace_id' mappings\" in self.validator.errors[0]\n assert \"Available: ['dev', 'prod']\" in self.validator.errors[0]\n\n def test_environment_mismatch_in_multiple_fields(self):\n \"\"\"Test environment exists validation with mismatches in multiple fields.\"\"\"\n self.validator.config = {\n \"core\": {\n \"workspace_id\": {\"dev\": \"dev-id\", \"prod\": \"prod-id\"},\n \"repository_directory\": {\"dev\": \"/dev/path\", \"prod\": \"/prod/path\"},\n \"item_types_in_scope\": {\"dev\": [\"Notebook\"], \"prod\": [\"DataPipeline\"]},\n },\n \"publish\": {\"skip\": {\"dev\": True, \"prod\": False}},\n }\n self.validator.environment = \"test\" # Not in any mapping\n\n self.validator._validate_environment_exists()\n\n # Only required fields (workspace_id, repository_directory) cause errors\n # Optional fields (item_types_in_scope, skip) only log warnings/debug\n assert len(self.validator.errors) == 2\n error_text = \" \".join(self.validator.errors)\n assert \"workspace_id\" in error_text\n assert \"repository_directory\" in error_text\n\n def test_environment_mapping_vs_basic_values_mixed(self):\n \"\"\"Test configuration with both environment mappings and basic values.\"\"\"\n self.validator.config = {\n \"core\": {\n \"workspace_id\": {\"dev\": \"dev-id\", \"prod\": \"prod-id\"}, # Environment mapping\n \"repository_directory\": \"/single/path\", # Basic value\n \"item_types_in_scope\": [\"Notebook\", \"DataPipeline\"], # Basic value\n },\n \"publish\": {\n \"skip\": True # Basic boolean\n },\n \"unpublish\": {\n \"skip\": {\"dev\": False, \"prod\": True} # Environment mapping\n },\n }\n self.validator.environment = \"dev\"\n\n self.validator._validate_environment_exists()\n\n # Should only validate the environment mappings, not the basic values\n assert self.validator.errors == []\n\n def test_environment_mapping_vs_basic_values_mismatch(self):\n \"\"\"Test environment mismatch only in fields with environment mappings.\"\"\"\n self.validator.config = {\n \"core\": {\n \"workspace_id\": {\"dev\": \"dev-id\"}, # Environment mapping - missing 'prod'\n \"repository_directory\": \"/single/path\", # Basic value - should be ignored\n \"item_types_in_scope\": [\"Notebook\"], # Basic value - should be ignored\n },\n \"publish\": {\n \"exclude_regex\": \"^TEST.*\", # Basic value - should be ignored\n \"skip\": {\"dev\": True}, # Environment mapping - missing 'prod' (optional, no error)\n },\n }\n self.validator.environment = \"prod\"\n\n self.validator._validate_environment_exists()\n\n # Should get error only for required field with environment mapping\n # Optional fields (skip) only log debug, not errors\n assert len(self.validator.errors) == 1\n error_text = \" \".join(self.validator.errors)\n assert \"workspace_id\" in error_text\n assert \"skip\" not in error_text # Optional field should not cause error\n\n def test_environment_mismatch_optional_fields_log_only(self):\n \"\"\"Test that optional fields only log warnings/debug when environment is missing.\"\"\"\n import logging\n\n self.validator.config = {\n \"core\": {\n \"workspace_id\": \"simple-id\", # Not a mapping, won't trigger error\n \"repository_directory\": \"/path\",\n \"item_types_in_scope\": {\"dev\": [\"Notebook\"]}, # Optional, warn_if_missing=True\n \"parameter\": {\"dev\": \"param.yml\"}, # Optional, warn_if_missing=True\n },\n \"publish\": {\n \"exclude_regex\": {\"dev\": \"^TEST.*\"}, # Optional, warn_if_missing=False\n \"skip\": {\"dev\": True}, # Optional, warn_if_missing=False\n },\n }\n self.validator.environment = \"prod\"\n\n with (\n patch.object(logging.getLogger(\"fabric_cicd._common._config_validator\"), \"warning\") as mock_warning,\n patch.object(logging.getLogger(\"fabric_cicd._common._config_validator\"), \"debug\") as mock_debug,\n ):\n self.validator._validate_environment_exists()\n\n # No errors for optional fields\n assert self.validator.errors == []\n\n # Warnings should be logged for item_types_in_scope and parameter\n assert mock_warning.call_count == 2\n\n # Debug should be logged for exclude_regex and skip\n assert mock_debug.call_count == 2\n\n def test_environment_mismatch_required_fields_error(self):\n \"\"\"Test that required fields cause errors when environment is missing.\"\"\"\n self.validator.config = {\n \"core\": {\n \"workspace_id\": {\"dev\": \"dev-id\"}, # Required\n \"repository_directory\": {\"dev\": \"/dev/path\"}, # Required\n },\n }\n self.validator.environment = \"prod\"\n\n self.validator._validate_environment_exists()\n\n # Should get errors for both required fields\n assert len(self.validator.errors) == 2\n error_text = \" \".join(self.validator.errors)\n assert \"workspace_id\" in error_text\n assert \"repository_directory\" in error_text\n\n def test_environment_exists_constants_per_key_env_missing(self):\n \"\"\"Test _validate_environment_exists logs debug when env missing from per-key constants.\"\"\"\n import logging\n\n self.validator.config = {\n \"core\": {\n \"workspace_id\": \"simple-id\",\n \"repository_directory\": \"/path\",\n },\n \"constants\": {\n \"DEFAULT_API_ROOT_URL\": {\n \"dev\": \"https://api.fabric.microsoft.com\",\n },\n },\n }\n self.validator.environment = \"prod\"\n\n with patch.object(logging.getLogger(\"fabric_cicd._common._config_validator\"), \"warning\") as mock_warning:\n self.validator._validate_environment_exists()\n\n assert self.validator.errors == []\n assert mock_warning.call_count == 1\n assert \"constants.DEFAULT_API_ROOT_URL\" in mock_warning.call_args[0][0]\n\n def test_environment_exists_constants_per_key_env_present(self):\n \"\"\"Test _validate_environment_exists passes when env exists in per-key constants.\"\"\"\n self.validator.config = {\n \"core\": {\n \"workspace_id\": {\"dev\": \"dev-id\"},\n \"repository_directory\": \"/path\",\n },\n \"constants\": {\n \"DEFAULT_API_ROOT_URL\": {\n \"dev\": \"https://api.fabric.microsoft.com\",\n },\n },\n }\n self.validator.environment = \"dev\"\n\n self.validator._validate_environment_exists()\n\n assert self.validator.errors == []\n\n def test_environment_exists_constants_flat_value_no_env_check(self):\n \"\"\"Test _validate_environment_exists skips flat constants (no env mapping to check).\"\"\"\n self.validator.config = {\n \"core\": {\n \"workspace_id\": \"simple-id\",\n \"repository_directory\": \"/path\",\n },\n \"constants\": {\n \"DEFAULT_API_ROOT_URL\": \"https://api.fabric.microsoft.com\",\n },\n }\n self.validator.environment = \"prod\"\n\n self.validator._validate_environment_exists()\n\n assert self.validator.errors == []\n\n def test_environment_na_with_constants_env_mapping_no_error(self):\n \"\"\"Test N/A environment does not flag constants per-key env mappings.\"\"\"\n self.validator.config = {\n \"core\": {\n \"workspace_id\": \"simple-id\",\n \"repository_directory\": \"/path\",\n },\n \"constants\": {\n \"DEFAULT_API_ROOT_URL\": {\n \"dev\": \"https://api.fabric.microsoft.com\",\n },\n },\n }\n self.validator.environment = \"N/A\"\n\n self.validator._validate_environment_exists()\n\n # Constants per-key dicts are excluded from the N/A check\n assert self.validator.errors == []\n" }, { "path": "tests/test_deploy_with_config.py", "content": "# Copyright (c) Microsoft Corporation.\n# Licensed under the MIT License.\n\n\"\"\"Tests for the config-based deployment functionality.\"\"\"\n\nfrom pathlib import Path\nfrom unittest.mock import MagicMock, patch\n\nimport pytest\nimport yaml\n\nfrom fabric_cicd import DeploymentResult, DeploymentStatus, constants, deploy_with_config\nfrom fabric_cicd._common._config_utils import (\n config_overrides_scope,\n extract_publish_settings,\n extract_unpublish_settings,\n extract_workspace_settings,\n load_config_file,\n)\nfrom fabric_cicd._common._config_validator import ConfigValidationError\nfrom fabric_cicd._common._exceptions import InputError, PublishError\n\n\nclass TestConfigFileLoading:\n \"\"\"Test config file loading and validation.\"\"\"\n\n def test_load_valid_config_file(self, tmp_path):\n \"\"\"Test loading a valid YAML config file.\"\"\"\n # Create the actual directory structure that the config references\n test_repo_dir = tmp_path / \"test\" / \"path\"\n test_repo_dir.mkdir(parents=True)\n\n config_data = {\n \"core\": {\n \"workspace_id\": {\"dev\": \"12345678-1234-1234-1234-123456789abc\"},\n \"repository_directory\": \"test/path\",\n }\n }\n config_file = tmp_path / \"config.yml\"\n with Path.open(config_file, \"w\") as f:\n yaml.dump(config_data, f)\n\n result = load_config_file(str(config_file), \"dev\")\n # Verify the structure is correct\n assert result[\"core\"][\"workspace_id\"] == config_data[\"core\"][\"workspace_id\"]\n # Verify path was resolved to absolute path and exists\n resolved_path = Path(result[\"core\"][\"repository_directory\"])\n assert resolved_path.is_absolute()\n assert resolved_path.exists()\n assert resolved_path.is_dir()\n\n def test_load_config_file_with_override(self, tmp_path):\n \"\"\"Test loading a YAML config file with overrides.\"\"\"\n # Create the actual directory structure that the config references\n test_repo_dir = tmp_path / \"test\" / \"path\"\n test_repo_dir.mkdir(parents=True)\n\n config_data = {\n \"core\": {\n \"workspace_id\": {\"dev\": \"12345678-1234-1234-1234-123456789abc\"},\n \"repository_directory\": \"test/path\",\n }\n }\n config_file = tmp_path / \"config.yml\"\n with Path.open(config_file, \"w\") as f:\n yaml.dump(config_data, f)\n\n # Define override values\n config_override = {\n \"core\": {\"workspace_id\": {\"dev\": \"87654321-4321-4321-4321-123456789abc\"}},\n \"publish\": {\"skip\": False, \"exclude_regex\": \"^TEST.*\"},\n }\n\n result = load_config_file(str(config_file), \"dev\", config_override)\n\n # Verify the overridden values\n assert result[\"core\"][\"workspace_id\"][\"dev\"] == \"87654321-4321-4321-4321-123456789abc\"\n assert result[\"publish\"][\"skip\"] == False\n assert result[\"publish\"][\"exclude_regex\"] == \"^TEST.*\"\n\n # Verify path was still resolved to absolute path and exists\n resolved_path = Path(result[\"core\"][\"repository_directory\"])\n assert resolved_path.is_absolute()\n assert resolved_path.exists()\n assert resolved_path.is_dir()\n\n def test_load_nonexistent_config_file(self):\n \"\"\"Test loading a non-existent config file raises ConfigValidationError.\"\"\"\n with pytest.raises(ConfigValidationError, match=\"Configuration file not found\"):\n load_config_file(\"nonexistent.yml\", \"N/A\")\n\n def test_load_invalid_yaml_syntax(self, tmp_path):\n \"\"\"Test loading a file with invalid YAML syntax raises InputError.\"\"\"\n config_file = tmp_path / \"invalid.yml\"\n config_file.write_text(\"invalid: yaml: content: [\")\n\n with pytest.raises(InputError, match=\"Invalid YAML syntax\"):\n load_config_file(str(config_file), \"N/A\")\n\n def test_load_non_dict_yaml(self, tmp_path):\n \"\"\"Test loading a YAML file that doesn't contain a dictionary.\"\"\"\n config_file = tmp_path / \"list.yml\"\n config_file.write_text(\"- item1\\n- item2\")\n\n with pytest.raises(ConfigValidationError, match=\"Configuration must be a dictionary\"):\n load_config_file(str(config_file), \"N/A\")\n\n def test_load_config_missing_core_section(self, tmp_path):\n \"\"\"Test loading a config file without required 'core' section.\"\"\"\n config_data = {\"publish\": {\"skip\": {\"dev\": True}}}\n config_file = tmp_path / \"no_core.yml\"\n with Path.open(config_file, \"w\") as f:\n yaml.dump(config_data, f)\n\n with pytest.raises(ConfigValidationError, match=\"must contain a 'core' section\"):\n load_config_file(str(config_file), \"N/A\")\n\n\nclass TestWorkspaceSettingsExtraction:\n \"\"\"Test workspace settings extraction from config.\"\"\"\n\n def test_extract_workspace_id_by_environment(self):\n \"\"\"Test extracting workspace ID based on environment.\"\"\"\n config = {\n \"core\": {\n \"workspace_id\": {\n \"dev\": \"11111111-1111-1111-1111-111111111111\",\n \"prod\": \"22222222-2222-2222-2222-222222222222\",\n },\n \"repository_directory\": \"test/path\",\n }\n }\n\n settings = extract_workspace_settings(config, \"dev\")\n assert settings[\"workspace_id\"] == \"11111111-1111-1111-1111-111111111111\"\n assert settings[\"repository_directory\"] == \"test/path\"\n\n def test_extract_workspace_name_by_environment(self):\n \"\"\"Test extracting workspace name based on environment.\"\"\"\n config = {\n \"core\": {\n \"workspace\": {\"dev\": \"dev-workspace\", \"prod\": \"prod-workspace\"},\n \"repository_directory\": \"test/path\",\n }\n }\n\n settings = extract_workspace_settings(config, \"dev\")\n assert settings[\"workspace_name\"] == \"dev-workspace\"\n assert settings[\"repository_directory\"] == \"test/path\"\n\n def test_extract_single_workspace_id(self, tmp_path):\n \"\"\"Test config with single workspace ID (non-environment-specific).\"\"\"\n # Create the actual directory structure that the config references\n test_repo_dir = tmp_path / \"test\" / \"path\"\n test_repo_dir.mkdir(parents=True)\n\n config_data = {\n \"core\": {\n \"workspace_id\": \"33333333-3333-3333-3333-333333333333\",\n \"repository_directory\": \"test/path\",\n }\n }\n config_file = tmp_path / \"config.yml\"\n config_file.write_text(yaml.dump(config_data))\n\n # Single workspace IDs are supported\n config = load_config_file(str(config_file), \"N/A\")\n assert config[\"core\"][\"workspace_id\"] == \"33333333-3333-3333-3333-333333333333\"\n\n def test_extract_missing_environment(self, tmp_path):\n \"\"\"Test error when environment not found in workspace mappings during config loading.\"\"\"\n config_data = {\n \"core\": {\n \"workspace_id\": {\"dev\": \"44444444-4444-4444-4444-444444444444\"},\n \"repository_directory\": \"test/path\",\n }\n }\n config_file = tmp_path / \"config.yml\"\n config_file.write_text(yaml.dump(config_data))\n\n # Environment validation should happen during config loading, not extraction\n with pytest.raises(\n ConfigValidationError, match=r\"Environment 'prod' not found in 'core.workspace_id' mappings\"\n ):\n load_config_file(str(config_file), \"prod\")\n\n def test_extract_missing_workspace_config(self, tmp_path):\n \"\"\"Test error when neither workspace_id nor workspace is provided.\"\"\"\n config_data = {\n \"core\": {\n \"repository_directory\": \"test/path\",\n }\n }\n config_file = tmp_path / \"config.yml\"\n config_file.write_text(yaml.dump(config_data))\n\n with pytest.raises(ConfigValidationError, match=\"must specify either 'workspace_id' or 'workspace'\"):\n load_config_file(str(config_file), \"N/A\")\n\n def test_extract_missing_repository_directory(self, tmp_path):\n \"\"\"Test error when repository_directory is missing.\"\"\"\n config_data = {\n \"core\": {\n \"workspace_id\": {\"dev\": \"55555555-5555-5555-5555-555555555555\"},\n }\n }\n config_file = tmp_path / \"config.yml\"\n config_file.write_text(yaml.dump(config_data))\n\n with pytest.raises(ConfigValidationError, match=\"must specify 'repository_directory'\"):\n load_config_file(str(config_file), \"N/A\")\n\n def test_extract_optional_item_types(self):\n \"\"\"Test extracting optional item_types_in_scope.\"\"\"\n config = {\n \"core\": {\n \"workspace_id\": \"66666666-6666-6666-6666-666666666666\",\n \"repository_directory\": \"test/path\",\n \"item_types_in_scope\": [\"Notebook\", \"DataPipeline\"],\n }\n }\n\n settings = extract_workspace_settings(config, \"dev\")\n assert settings[\"item_types_in_scope\"] == [\"Notebook\", \"DataPipeline\"]\n\n def test_extract_parameter_file_path_string(self):\n \"\"\"Test extracting parameter file path as string.\"\"\"\n config = {\n \"core\": {\n \"workspace_id\": \"12345678-1234-1234-1234-123456789abc\",\n \"repository_directory\": \"test/path\",\n \"parameter\": \"parameter.yml\",\n }\n }\n\n settings = extract_workspace_settings(config, \"dev\")\n assert settings[\"parameter_file_path\"] == \"parameter.yml\"\n\n def test_extract_parameter_file_path_environment_mapping(self):\n \"\"\"Test extracting parameter file path from environment mapping.\"\"\"\n config = {\n \"core\": {\n \"workspace_id\": {\n \"dev\": \"11111111-1111-1111-1111-111111111111\",\n \"prod\": \"22222222-2222-2222-2222-222222222222\",\n },\n \"repository_directory\": \"test/path\",\n \"parameter\": {\"dev\": \"dev-parameter.yml\", \"prod\": \"prod-parameter.yml\"},\n }\n }\n\n settings = extract_workspace_settings(config, \"dev\")\n assert settings[\"parameter_file_path\"] == \"dev-parameter.yml\"\n\n settings_prod = extract_workspace_settings(config, \"prod\")\n assert settings_prod[\"parameter_file_path\"] == \"prod-parameter.yml\"\n\n def test_extract_parameter_file_path_missing(self):\n \"\"\"Test extracting workspace settings when parameter field is missing.\"\"\"\n config = {\n \"core\": {\n \"workspace_id\": \"33333333-3333-3333-3333-333333333333\",\n \"repository_directory\": \"test/path\",\n }\n }\n\n settings = extract_workspace_settings(config, \"dev\")\n assert \"parameter_file_path\" not in settings\n\n\nclass TestPublishSettingsExtraction:\n \"\"\"Test publish settings extraction from config.\"\"\"\n\n def testextract_publish_settings_with_skip(self):\n \"\"\"Test extracting publish settings with environment-specific skip.\"\"\"\n config = {\n \"publish\": {\n \"exclude_regex\": \"^DONT_DEPLOY.*\",\n \"skip\": {\"dev\": True, \"prod\": False},\n }\n }\n\n settings = extract_publish_settings(config, \"dev\")\n assert settings[\"exclude_regex\"] == \"^DONT_DEPLOY.*\"\n assert settings[\"skip\"] is True\n\n settings = extract_publish_settings(config, \"prod\")\n assert settings[\"skip\"] is False\n\n def testextract_publish_settings_with_items_to_include(self):\n \"\"\"Test extracting publish settings with items_to_include.\"\"\"\n config = {\n \"publish\": {\n \"items_to_include\": [\"item1.Notebook\", \"item2.DataPipeline\"],\n }\n }\n\n settings = extract_publish_settings(config, \"dev\")\n assert settings[\"items_to_include\"] == [\"item1.Notebook\", \"item2.DataPipeline\"]\n\n def testextract_publish_settings_no_config(self):\n \"\"\"Test extracting publish settings when no publish config exists.\"\"\"\n config = {}\n\n settings = extract_publish_settings(config, \"dev\")\n assert settings == {}\n\n def testextract_publish_settings_single_skip_value(self):\n \"\"\"Test extracting publish settings with single skip value (not environment-specific).\"\"\"\n config = {\n \"publish\": {\n \"skip\": True,\n }\n }\n\n settings = extract_publish_settings(config, \"dev\")\n assert settings[\"skip\"] is True\n\n def test_extract_publish_settings_with_shortcut_exclude_regex(self):\n \"\"\"Test extracting publish settings with shortcut_exclude_regex.\"\"\"\n config = {\n \"publish\": {\n \"shortcut_exclude_regex\": \"^temp_.*\",\n }\n }\n\n settings = extract_publish_settings(config, \"dev\")\n assert settings[\"shortcut_exclude_regex\"] == \"^temp_.*\"\n\n def test_extract_publish_settings_with_environment_specific_shortcut_exclude_regex(self):\n \"\"\"Test extracting publish settings with environment-specific shortcut_exclude_regex.\"\"\"\n config = {\n \"publish\": {\n \"shortcut_exclude_regex\": {\"dev\": \"^dev_temp_.*\", \"prod\": \"^staging_.*\"},\n }\n }\n\n settings = extract_publish_settings(config, \"dev\")\n assert settings[\"shortcut_exclude_regex\"] == \"^dev_temp_.*\"\n\n settings = extract_publish_settings(config, \"prod\")\n assert settings[\"shortcut_exclude_regex\"] == \"^staging_.*\"\n\n\nclass TestUnpublishSettingsExtraction:\n \"\"\"Test unpublish settings extraction from config.\"\"\"\n\n def testextract_unpublish_settings_with_skip(self):\n \"\"\"Test extracting unpublish settings with environment-specific skip.\"\"\"\n config = {\n \"unpublish\": {\n \"exclude_regex\": \"^DEBUG.*\",\n \"skip\": {\"dev\": True, \"prod\": False},\n }\n }\n\n settings = extract_unpublish_settings(config, \"dev\")\n assert settings[\"exclude_regex\"] == \"^DEBUG.*\"\n assert settings[\"skip\"] is True\n\n settings = extract_unpublish_settings(config, \"prod\")\n assert settings[\"skip\"] is False\n\n def testextract_unpublish_settings_no_config(self):\n \"\"\"Test extracting unpublish settings when no unpublish config exists.\"\"\"\n config = {}\n\n settings = extract_unpublish_settings(config, \"dev\")\n assert settings == {}\n\n\nclass TestConfigOverrides:\n \"\"\"Test feature flags and constants overrides.\"\"\"\n\n def test_feature_flags_applied_within_scope(self):\n \"\"\"Test feature flags are active inside config_overrides_scope.\"\"\"\n foo = \"enable_foo_feature\"\n bar = \"enable_bar_feature\"\n config = {\"features\": [foo, bar]}\n\n original_flags = constants.FEATURE_FLAG.copy()\n\n with config_overrides_scope(config, \"N/A\"):\n assert foo in constants.FEATURE_FLAG\n assert bar in constants.FEATURE_FLAG\n\n # Verify flags are restored after exiting scope\n assert original_flags == constants.FEATURE_FLAG\n\n def test_feature_flags_restored_after_scope(self):\n \"\"\"Test feature flags set by config do not persist after scope exits.\"\"\"\n config = {\"features\": [\"enable_test_feature\"]}\n\n original_flags = constants.FEATURE_FLAG.copy()\n\n with config_overrides_scope(config, \"N/A\"):\n assert \"enable_test_feature\" in constants.FEATURE_FLAG\n\n assert \"enable_test_feature\" not in constants.FEATURE_FLAG\n assert original_flags == constants.FEATURE_FLAG\n\n def test_constants_overrides_applied_within_scope(self):\n \"\"\"Test constants overrides are active inside config_overrides_scope.\"\"\"\n original_url = constants.DEFAULT_API_ROOT_URL\n config = {\"constants\": {\"DEFAULT_API_ROOT_URL\": \"https://custom.api.com\"}}\n\n with config_overrides_scope(config, \"N/A\"):\n assert constants.DEFAULT_API_ROOT_URL == \"https://custom.api.com\"\n\n # Verify constant is restored after exiting scope\n assert original_url == constants.DEFAULT_API_ROOT_URL\n\n def test_constants_overrides_restored_after_scope(self):\n \"\"\"Test constants set by config do not persist after scope exits.\"\"\"\n original_url = constants.DEFAULT_API_ROOT_URL\n config = {\"constants\": {\"DEFAULT_API_ROOT_URL\": \"https://temporary.api.com\"}}\n\n with config_overrides_scope(config, \"N/A\"):\n pass\n\n assert original_url == constants.DEFAULT_API_ROOT_URL\n\n def test_user_constants_preserved_across_scope(self):\n \"\"\"Test that constants set by user before scope are preserved after scope exits.\"\"\"\n original_url = constants.DEFAULT_API_ROOT_URL\n constants.DEFAULT_API_ROOT_URL = \"https://user-set.api.com\"\n\n config = {\"constants\": {\"FABRIC_API_ROOT_URL\": \"https://config-set.fabric.com\"}}\n original_fabric_url = constants.FABRIC_API_ROOT_URL\n\n with config_overrides_scope(config, \"N/A\"):\n assert constants.DEFAULT_API_ROOT_URL == \"https://user-set.api.com\"\n assert constants.FABRIC_API_ROOT_URL == \"https://config-set.fabric.com\"\n\n assert constants.DEFAULT_API_ROOT_URL == \"https://user-set.api.com\"\n assert original_fabric_url == constants.FABRIC_API_ROOT_URL\n\n # Clean up\n constants.DEFAULT_API_ROOT_URL = original_url\n\n def test_user_constant_same_key_restored_after_scope(self):\n \"\"\"Test that when user sets a constant and config overrides the same key, user value is restored.\"\"\"\n original_url = constants.DEFAULT_API_ROOT_URL\n constants.DEFAULT_API_ROOT_URL = \"https://user-set.api.com\"\n\n config = {\"constants\": {\"DEFAULT_API_ROOT_URL\": \"https://config-override.api.com\"}}\n\n with config_overrides_scope(config, \"N/A\"):\n assert constants.DEFAULT_API_ROOT_URL == \"https://config-override.api.com\"\n\n assert constants.DEFAULT_API_ROOT_URL == \"https://user-set.api.com\"\n\n # Clean up\n constants.DEFAULT_API_ROOT_URL = original_url\n\n def test_no_overrides_does_not_error(self):\n \"\"\"Test config_overrides_scope works with empty config.\"\"\"\n original_flags = constants.FEATURE_FLAG.copy()\n config = {}\n\n with config_overrides_scope(config, \"N/A\"):\n pass\n\n assert original_flags == constants.FEATURE_FLAG\n\n def test_overrides_restored_on_exception(self):\n \"\"\"Test that overrides are restored even when an exception occurs inside the scope.\"\"\"\n original_url = constants.DEFAULT_API_ROOT_URL\n original_flags = constants.FEATURE_FLAG.copy()\n config = {\n \"features\": [\"enable_test_feature\"],\n \"constants\": {\"DEFAULT_API_ROOT_URL\": \"https://will-fail.api.com\"},\n }\n\n msg = \"deployment failed\"\n with pytest.raises(RuntimeError, match=msg), config_overrides_scope(config, \"N/A\"):\n raise RuntimeError(msg)\n\n assert original_url == constants.DEFAULT_API_ROOT_URL\n assert original_flags == constants.FEATURE_FLAG\n\n def test_user_flags_preserved_across_scope(self):\n \"\"\"Test that flags set by user before scope are preserved after scope exits.\"\"\"\n original_flags = constants.FEATURE_FLAG.copy()\n constants.FEATURE_FLAG.add(\"user_set_flag\")\n\n config = {\"features\": [\"config_set_flag\"]}\n\n with config_overrides_scope(config, \"N/A\"):\n assert \"user_set_flag\" in constants.FEATURE_FLAG\n assert \"config_set_flag\" in constants.FEATURE_FLAG\n\n assert \"user_set_flag\" in constants.FEATURE_FLAG\n assert \"config_set_flag\" not in constants.FEATURE_FLAG\n\n # Clean up\n constants.FEATURE_FLAG.clear()\n constants.FEATURE_FLAG.update(original_flags)\n\n def test_environment_specific_feature_flags(self):\n \"\"\"Test environment-specific feature flags are resolved correctly.\"\"\"\n original_flags = constants.FEATURE_FLAG.copy()\n config = {\n \"features\": {\n \"dev\": [\"enable_dev_feature\"],\n \"prod\": [\"enable_prod_feature\"],\n }\n }\n\n with config_overrides_scope(config, \"dev\"):\n assert \"enable_dev_feature\" in constants.FEATURE_FLAG\n assert \"enable_prod_feature\" not in constants.FEATURE_FLAG\n\n assert original_flags == constants.FEATURE_FLAG\n\n def test_environment_specific_constants(self):\n \"\"\"Test environment-specific constants overrides are resolved correctly.\"\"\"\n original_url = constants.DEFAULT_API_ROOT_URL\n config = {\n \"constants\": {\n \"DEFAULT_API_ROOT_URL\": {\n \"dev\": \"https://dev.api.com\",\n \"prod\": \"https://prod.api.com\",\n }\n }\n }\n\n with config_overrides_scope(config, \"dev\"):\n assert constants.DEFAULT_API_ROOT_URL == \"https://dev.api.com\"\n\n assert original_url == constants.DEFAULT_API_ROOT_URL\n\n\nclass TestConfigOverridesIntegration:\n \"\"\"Integration tests for config_overrides_scope with deploy_with_config.\"\"\"\n\n @patch(\"fabric_cicd.publish.FabricWorkspace\")\n @patch(\"fabric_cicd.publish.publish_all_items\")\n @patch(\"fabric_cicd.publish.unpublish_all_orphan_items\")\n def test_response_collection_via_config_features(self, mock_unpublish, mock_publish, mock_workspace, tmp_path):\n \"\"\"Test that enable_response_collection set in config features enables response collection.\"\"\"\n _ = mock_unpublish\n _ = mock_publish\n\n test_repo_dir = tmp_path / \"test\" / \"path\"\n test_repo_dir.mkdir(parents=True)\n\n config_data = {\n \"core\": {\n \"workspace_id\": {\"dev\": \"77777777-7777-7777-7777-777777777777\"},\n \"repository_directory\": \"test/path\",\n },\n \"features\": [\"enable_response_collection\"],\n }\n config_file = tmp_path / \"config.yml\"\n with Path.open(config_file, \"w\") as f:\n yaml.dump(config_data, f)\n\n mock_workspace_instance = MagicMock()\n mock_workspace_instance.responses = {\"Notebook\": {\"MyNotebook\": {\"body\": {\"id\": \"123\"}}}}\n mock_workspace_instance.unpublish_responses = None\n mock_workspace.return_value = mock_workspace_instance\n\n result = deploy_with_config(config_file_path=str(config_file), token_credential=MagicMock(), environment=\"dev\")\n\n assert isinstance(result, DeploymentResult)\n assert result.responses == {\"publish\": {\"Notebook\": {\"MyNotebook\": {\"body\": {\"id\": \"123\"}}}}}\n\n # Verify feature flag was restored after scope exit\n assert \"enable_response_collection\" not in constants.FEATURE_FLAG\n\n @patch(\"fabric_cicd.publish.FabricWorkspace\")\n @patch(\"fabric_cicd.publish.publish_all_items\")\n @patch(\"fabric_cicd.publish.unpublish_all_orphan_items\")\n def test_failure_with_partial_responses_via_config_features(\n self, mock_unpublish, mock_publish, mock_workspace, tmp_path\n ):\n \"\"\"Test that partial responses are attached to exceptions when enable_response_collection is set via config.\"\"\"\n _ = mock_publish\n\n test_repo_dir = tmp_path / \"test\" / \"path\"\n test_repo_dir.mkdir(parents=True)\n\n config_data = {\n \"core\": {\n \"workspace_id\": {\"dev\": \"77777777-7777-7777-7777-777777777777\"},\n \"repository_directory\": \"test/path\",\n },\n \"features\": [\"enable_response_collection\"],\n }\n config_file = tmp_path / \"config.yml\"\n with Path.open(config_file, \"w\") as f:\n yaml.dump(config_data, f)\n\n mock_workspace_instance = MagicMock()\n mock_workspace_instance.responses = {\"Notebook\": {\"MyNotebook\": {\"body\": {\"id\": \"123\"}}}}\n mock_workspace_instance.unpublish_responses = None\n mock_workspace.return_value = mock_workspace_instance\n\n mock_unpublish.side_effect = RuntimeError(\"Unpublish failed\")\n\n with pytest.raises(RuntimeError) as exc_info:\n deploy_with_config(config_file_path=str(config_file), token_credential=MagicMock(), environment=\"dev\")\n\n e = exc_info.value\n assert hasattr(e, \"deployment_result\")\n assert e.deployment_result.responses == {\"publish\": {\"Notebook\": {\"MyNotebook\": {\"body\": {\"id\": \"123\"}}}}}\n\n # Verify feature flag was restored after scope exit\n assert \"enable_response_collection\" not in constants.FEATURE_FLAG\n\n\nclass TestDeployWithConfig:\n \"\"\"Test the main deploy_with_config function.\"\"\"\n\n @patch(\"fabric_cicd.publish.FabricWorkspace\")\n @patch(\"fabric_cicd.publish.publish_all_items\")\n @patch(\"fabric_cicd.publish.unpublish_all_orphan_items\")\n def test_deploy_with_config_full_deployment(self, mock_unpublish, mock_publish, mock_workspace, tmp_path):\n \"\"\"Test full deployment with config file.\"\"\"\n # Create the actual directory structure that the config references\n test_repo_dir = tmp_path / \"test\" / \"path\"\n test_repo_dir.mkdir(parents=True)\n\n # Create test config file\n config_data = {\n \"core\": {\n \"workspace_id\": {\"dev\": \"77777777-7777-7777-7777-777777777777\"},\n \"repository_directory\": \"test/path\",\n \"item_types_in_scope\": [\"Notebook\", \"DataPipeline\"],\n },\n \"publish\": {\n \"exclude_regex\": \"^DONT_DEPLOY.*\",\n \"skip\": {\"dev\": False},\n },\n \"unpublish\": {\n \"exclude_regex\": \"^DEBUG.*\",\n \"skip\": {\"dev\": False},\n },\n }\n config_file = tmp_path / \"config.yml\"\n with Path.open(config_file, \"w\") as f:\n yaml.dump(config_data, f)\n\n # Mock workspace instance\n mock_workspace_instance = MagicMock()\n mock_workspace.return_value = mock_workspace_instance\n mock_credential = MagicMock()\n\n # Execute deployment\n deploy_with_config(config_file_path=str(config_file), token_credential=mock_credential, environment=\"dev\")\n\n # Verify workspace creation\n # Note: repository_directory will be resolved to absolute path during validation\n call_args = mock_workspace.call_args[1]\n assert call_args[\"workspace_id\"] == \"77777777-7777-7777-7777-777777777777\"\n assert call_args[\"workspace_name\"] is None\n assert \"test\" in call_args[\"repository_directory\"] # Path will be resolved to absolute\n assert \"path\" in call_args[\"repository_directory\"]\n assert call_args[\"item_type_in_scope\"] == [\"Notebook\", \"DataPipeline\"]\n assert call_args[\"environment\"] == \"dev\"\n assert call_args[\"token_credential\"] == mock_credential\n\n # Verify publish and unpublish calls\n mock_publish.assert_called_once_with(\n mock_workspace_instance,\n item_name_exclude_regex=\"^DONT_DEPLOY.*\",\n folder_path_exclude_regex=None,\n folder_path_to_include=None,\n items_to_include=None,\n shortcut_exclude_regex=None,\n )\n mock_unpublish.assert_called_once_with(\n mock_workspace_instance,\n item_name_exclude_regex=\"^DEBUG.*\",\n items_to_include=None,\n )\n\n @patch(\"fabric_cicd.publish.FabricWorkspace\")\n @patch(\"fabric_cicd.publish.publish_all_items\")\n @patch(\"fabric_cicd.publish.unpublish_all_orphan_items\")\n def test_deploy_with_config_skip_operations(self, mock_unpublish, mock_publish, mock_workspace, tmp_path):\n \"\"\"Test deployment with skip flags enabled.\"\"\"\n # Create the actual directory structure that the config references\n test_repo_dir = tmp_path / \"test\" / \"path\"\n test_repo_dir.mkdir(parents=True)\n\n # Create test config file with skip flags\n config_data = {\n \"core\": {\n \"workspace_id\": {\"dev\": \"88888888-8888-8888-8888-888888888888\"},\n \"repository_directory\": \"test/path\",\n },\n \"publish\": {\n \"skip\": {\"dev\": True},\n },\n \"unpublish\": {\n \"skip\": {\"dev\": True},\n },\n }\n config_file = tmp_path / \"config.yml\"\n with Path.open(config_file, \"w\") as f:\n yaml.dump(config_data, f)\n\n # Mock workspace instance\n mock_workspace_instance = MagicMock()\n mock_workspace.return_value = mock_workspace_instance\n\n # Execute deployment\n deploy_with_config(config_file_path=str(config_file), token_credential=MagicMock(), environment=\"dev\")\n\n # Verify workspace creation\n mock_workspace.assert_called_once()\n\n # Verify that publish and unpublish are NOT called due to skip flags\n mock_publish.assert_not_called()\n mock_unpublish.assert_not_called()\n\n def test_deploy_with_config_missing_file(self):\n \"\"\"Test deployment with missing config file.\"\"\"\n with pytest.raises(ConfigValidationError, match=\"Configuration file not found\"):\n deploy_with_config(config_file_path=\"nonexistent.yml\", token_credential=MagicMock(), environment=\"dev\")\n\n @patch(\"fabric_cicd.publish.FabricWorkspace\")\n @patch(\"fabric_cicd.publish.publish_all_items\")\n @patch(\"fabric_cicd.publish.unpublish_all_orphan_items\")\n def test_deploy_with_config_with_token_credential(self, mock_unpublish, mock_publish, mock_workspace, tmp_path):\n \"\"\"Test deployment with custom token credential.\"\"\"\n # Mark unused mocks to avoid linting warnings\n _ = mock_unpublish\n _ = mock_publish\n\n # Create the actual directory structure that the config references\n test_repo_dir = tmp_path / \"test\" / \"path\"\n test_repo_dir.mkdir(parents=True)\n\n # Create test config file\n config_data = {\n \"core\": {\n \"workspace_id\": {\"dev\": \"99999999-9999-9999-9999-999999999999\"},\n \"repository_directory\": \"test/path\",\n },\n }\n config_file = tmp_path / \"config.yml\"\n with Path.open(config_file, \"w\") as f:\n yaml.dump(config_data, f)\n\n # Mock workspace instance and token credential\n mock_workspace_instance = MagicMock()\n mock_workspace.return_value = mock_workspace_instance\n mock_credential = MagicMock()\n\n # Execute deployment\n deploy_with_config(config_file_path=str(config_file), token_credential=mock_credential, environment=\"dev\")\n\n # Verify workspace creation with token credential\n # Note: repository_directory will be resolved to absolute path during validation\n call_args = mock_workspace.call_args[1]\n assert call_args[\"workspace_id\"] == \"99999999-9999-9999-9999-999999999999\"\n assert call_args[\"workspace_name\"] is None\n assert \"test\" in call_args[\"repository_directory\"] # Path will be resolved to absolute\n assert \"path\" in call_args[\"repository_directory\"]\n assert call_args[\"item_type_in_scope\"] is None\n assert call_args[\"environment\"] == \"dev\"\n assert call_args[\"token_credential\"] == mock_credential\n\n @patch(\"fabric_cicd.publish.FabricWorkspace\")\n @patch(\"fabric_cicd.publish.publish_all_items\")\n @patch(\"fabric_cicd.publish.unpublish_all_orphan_items\")\n def test_deploy_with_config_with_config_override(self, mock_unpublish, mock_publish, mock_workspace, tmp_path):\n \"\"\"Test deployment with config override.\"\"\"\n # Create the actual directory structure that the config references\n test_repo_dir = tmp_path / \"test\" / \"path\"\n test_repo_dir.mkdir(parents=True)\n\n # Create test config file with default publish.skip = True to skip publishing\n config_data = {\n \"core\": {\n \"workspace_id\": {\"dev\": \"12345678-1234-1234-1234-123456789abc\"},\n \"repository_directory\": \"test/path\",\n },\n \"publish\": {\n \"skip\": {\"dev\": True},\n },\n \"unpublish\": {\n \"skip\": {\"dev\": True},\n },\n }\n config_file = tmp_path / \"config.yml\"\n with Path.open(config_file, \"w\") as f:\n yaml.dump(config_data, f)\n\n # Define config override to override the skip flags\n config_override = {\n \"publish\": {\"skip\": {\"dev\": False}}, # Override to NOT skip publish\n \"unpublish\": {\"skip\": {\"dev\": False}}, # Override to NOT skip unpublish\n }\n\n # Mock workspace instance\n mock_workspace_instance = MagicMock()\n mock_workspace.return_value = mock_workspace_instance\n\n # Execute deployment with config override\n deploy_with_config(\n config_file_path=str(config_file),\n token_credential=MagicMock(),\n environment=\"dev\",\n config_override=config_override,\n )\n\n # Verify workspace creation\n mock_workspace.assert_called_once()\n\n # Verify that publish and unpublish ARE called because the override turns off the skip flags\n mock_publish.assert_called_once()\n mock_unpublish.assert_called_once()\n\n @patch(\"fabric_cicd.publish.FabricWorkspace\")\n @patch(\"fabric_cicd.publish.publish_all_items\")\n @patch(\"fabric_cicd.publish.unpublish_all_orphan_items\")\n def test_deploy_with_config_shortcut_exclude_regex(self, mock_unpublish, mock_publish, mock_workspace, tmp_path):\n \"\"\"Test deployment with shortcut_exclude_regex in config.\"\"\"\n # Create the actual directory structure that the config references\n test_repo_dir = tmp_path / \"test\" / \"path\"\n test_repo_dir.mkdir(parents=True)\n\n # Create test config file with shortcut_exclude_regex\n config_data = {\n \"core\": {\n \"workspace_id\": \"12345678-1234-1234-1234-123456789abc\",\n \"repository_directory\": \"test/path\",\n \"item_types_in_scope\": [\"Lakehouse\"],\n },\n \"publish\": {\n \"shortcut_exclude_regex\": \"^temp_.*\",\n },\n }\n config_file = tmp_path / \"config.yml\"\n with Path.open(config_file, \"w\") as f:\n yaml.dump(config_data, f)\n\n # Mock workspace instance\n mock_workspace_instance = MagicMock()\n mock_workspace.return_value = mock_workspace_instance\n\n # Execute deployment\n deploy_with_config(config_file_path=str(config_file), token_credential=MagicMock(), environment=\"dev\")\n\n # Verify publish was called with shortcut_exclude_regex parameter\n mock_publish.assert_called_once_with(\n mock_workspace_instance,\n item_name_exclude_regex=None,\n folder_path_exclude_regex=None,\n folder_path_to_include=None,\n items_to_include=None,\n shortcut_exclude_regex=\"^temp_.*\",\n )\n # Verify unpublish was also called (but without shortcut_exclude_regex since it's publish-only)\n mock_unpublish.assert_called_once()\n\n @patch(\"fabric_cicd.publish.FabricWorkspace\")\n @patch(\"fabric_cicd.publish.publish_all_items\")\n @patch(\"fabric_cicd.publish.unpublish_all_orphan_items\")\n def test_folder_path_to_include_passed_to_publish(self, _mock_unpublish, mock_publish, mock_workspace, tmp_path): # noqa: PT019\n \"\"\"Test that folder_path_to_include from config is passed to publish_all_items.\"\"\"\n test_repo_dir = tmp_path / \"repo\"\n test_repo_dir.mkdir(parents=True)\n\n config_data = {\n \"core\": {\n \"workspace_id\": \"11111111-1111-1111-1111-111111111111\",\n \"repository_directory\": str(test_repo_dir),\n },\n \"publish\": {\n \"folder_path_to_include\": [\"/my/folder/path\"],\n },\n }\n config_file = tmp_path / \"config.yml\"\n with Path.open(config_file, \"w\") as f:\n yaml.dump(config_data, f)\n\n mock_workspace.return_value = MagicMock()\n deploy_with_config(config_file_path=str(config_file), token_credential=MagicMock(), environment=\"dev\")\n\n call_args = mock_publish.call_args[1]\n assert call_args[\"folder_path_to_include\"] == [\"/my/folder/path\"]\n\n @patch(\"fabric_cicd.publish.FabricWorkspace\")\n @patch(\"fabric_cicd.publish.publish_all_items\")\n @patch(\"fabric_cicd.publish.unpublish_all_orphan_items\")\n def test_folder_path_to_include_defaults_to_none(self, _mock_unpublish, mock_publish, mock_workspace, tmp_path): # noqa: PT019\n \"\"\"Test that folder_path_to_include defaults to None when not specified.\"\"\"\n test_repo_dir = tmp_path / \"repo\"\n test_repo_dir.mkdir(parents=True)\n\n config_data = {\n \"core\": {\n \"workspace_id\": \"11111111-1111-1111-1111-111111111111\",\n \"repository_directory\": str(test_repo_dir),\n },\n }\n config_file = tmp_path / \"config.yml\"\n with Path.open(config_file, \"w\") as f:\n yaml.dump(config_data, f)\n\n mock_workspace.return_value = MagicMock()\n deploy_with_config(config_file_path=str(config_file), token_credential=MagicMock(), environment=\"dev\")\n\n call_args = mock_publish.call_args[1]\n assert call_args[\"folder_path_to_include\"] is None\n\n @patch(\"fabric_cicd.publish.FabricWorkspace\")\n @patch(\"fabric_cicd.publish.publish_all_items\")\n @patch(\"fabric_cicd.publish.unpublish_all_orphan_items\")\n def test_folder_path_to_include_environment_specific(self, _mock_unpublish, mock_publish, mock_workspace, tmp_path): # noqa: PT019\n \"\"\"Test that folder_path_to_include resolves environment-specific values.\"\"\"\n test_repo_dir = tmp_path / \"repo\"\n test_repo_dir.mkdir(parents=True)\n\n config_data = {\n \"core\": {\n \"workspace_id\": {\n \"dev\": \"11111111-1111-1111-1111-111111111111\",\n \"prod\": \"22222222-2222-2222-2222-222222222222\",\n },\n \"repository_directory\": str(test_repo_dir),\n },\n \"publish\": {\n \"folder_path_to_include\": {\"dev\": [\"/dev/folder\"], \"prod\": [\"/prod/folder\"]},\n },\n }\n config_file = tmp_path / \"config.yml\"\n with Path.open(config_file, \"w\") as f:\n yaml.dump(config_data, f)\n\n mock_workspace.return_value = MagicMock()\n deploy_with_config(config_file_path=str(config_file), token_credential=MagicMock(), environment=\"dev\")\n\n call_args = mock_publish.call_args[1]\n assert call_args[\"folder_path_to_include\"] == [\"/dev/folder\"]\n\n def test_deploy_with_config_skips_parameterization_when_parameter_absent(self, tmp_path):\n \"\"\"Integration: deploy_with_config must not auto-discover parameter.yml\n when the 'parameter' field is absent from the config file.\"\"\"\n # Set up repo directory WITH a parameter.yml that would be auto-discovered\n repo_dir = tmp_path / \"workspace\"\n repo_dir.mkdir()\n (repo_dir / \"parameter.yml\").write_text(\n \"find_replace:\\n - find_value: 'old'\\n replace_value:\\n dev: 'new'\\n\"\n )\n\n config_data = {\n \"core\": {\n \"workspace_id\": {\"dev\": \"11111111-1111-1111-1111-111111111111\"},\n \"repository_directory\": str(repo_dir),\n # 'parameter' intentionally omitted\n }\n }\n config_file = tmp_path / \"config.yml\"\n config_file.write_text(yaml.dump(config_data))\n\n with patch(\"fabric_cicd.publish.FabricWorkspace\") as mock_fabric_ws:\n mock_ws = MagicMock()\n mock_ws.environment_parameter = {}\n mock_fabric_ws.return_value = mock_ws\n\n # Call deploy_with_config and verify skip_parameterization=True was passed\n # (mock publish/unpublish to avoid real API calls)\n with (\n patch(\"fabric_cicd.publish.publish_all_items\"),\n patch(\"fabric_cicd.publish.unpublish_all_orphan_items\"),\n ):\n deploy_with_config(config_file_path=str(config_file), token_credential=MagicMock(), environment=\"dev\")\n\n # Assert FabricWorkspace was constructed with skip_parameterization=True\n call_kwargs = mock_fabric_ws.call_args[1]\n assert call_kwargs.get(\"skip_parameterization\") is True\n\n def test_deploy_with_config_loads_parameter_when_field_present(self, tmp_path):\n \"\"\"Integration: deploy_with_config must pass skip_parameterization=False\n when the 'parameter' field IS present in config.\"\"\"\n repo_dir = tmp_path / \"workspace\"\n repo_dir.mkdir()\n\n config_data = {\n \"core\": {\n \"workspace_id\": {\"dev\": \"11111111-1111-1111-1111-111111111111\"},\n \"repository_directory\": str(repo_dir),\n \"parameter\": \"my-params.yml\",\n }\n }\n config_file = tmp_path / \"config.yml\"\n config_file.write_text(yaml.dump(config_data))\n\n parameter_file = tmp_path / \"my-params.yml\"\n parameter_file.write_text(\"find_replace:\\n - find_value: 'old'\\n replace_value:\\n dev: 'new'\\n\")\n\n with patch(\"fabric_cicd.publish.FabricWorkspace\") as mock_fabric_ws:\n mock_ws = MagicMock()\n mock_fabric_ws.return_value = mock_ws\n\n with (\n patch(\"fabric_cicd.publish.publish_all_items\"),\n patch(\"fabric_cicd.publish.unpublish_all_orphan_items\"),\n ):\n deploy_with_config(config_file_path=str(config_file), token_credential=MagicMock(), environment=\"dev\")\n\n call_kwargs = mock_fabric_ws.call_args[1]\n assert call_kwargs.get(\"skip_parameterization\") is False\n\n\nclass TestConfigIntegration:\n \"\"\"Integration tests for config functionality.\"\"\"\n\n def test_sample_config_file_structure(self):\n \"\"\"Test that the sample config file can be loaded and parsed correctly.\"\"\"\n # Test with the actual sample config file\n sample_config_path = Path(__file__).parent.parent / \"sample\" / \"workspace\" / \"config.yml\"\n\n if sample_config_path.exists():\n # The sample config file might have directory references that don't exist in the test environment\n # So we just verify it can be parsed as valid YAML\n import yaml\n\n with sample_config_path.open(encoding=\"utf-8\") as f:\n config = yaml.safe_load(f)\n\n # Verify basic structure\n assert \"core\" in config\n\n # If we have a valid environment, test basic functionality\n if (\n \"core\" in config\n and \"workspace_id\" in config[\"core\"]\n and isinstance(config[\"core\"][\"workspace_id\"], dict)\n ):\n test_env = next(iter(config[\"core\"][\"workspace_id\"].keys()))\n\n # Only test the config extraction without path validation\n workspace_settings = extract_workspace_settings(config, test_env)\n assert \"repository_directory\" in workspace_settings\n\n extract_publish_settings(config, test_env)\n extract_unpublish_settings(config, test_env)\n with config_overrides_scope(config, test_env):\n pass\n\n def test_config_validation_comprehensive(self, tmp_path):\n \"\"\"Test comprehensive config validation with all sections.\"\"\"\n # Create the actual directory structure that the config references\n sample_workspace_dir = tmp_path / \"sample\" / \"workspace\"\n sample_workspace_dir.mkdir(parents=True)\n\n config_data = {\n \"core\": {\n \"workspace_id\": {\n \"dev\": \"aaaaaaaa-aaaa-aaaa-aaaa-aaaaaaaaaaaa\",\n \"test\": \"bbbbbbbb-bbbb-bbbb-bbbb-bbbbbbbbbbbb\",\n \"prod\": \"cccccccc-cccc-cccc-cccc-cccccccccccc\",\n },\n \"repository_directory\": \"sample/workspace\",\n \"item_types_in_scope\": [\"Environment\", \"Notebook\", \"DataPipeline\"],\n },\n \"publish\": {\n \"exclude_regex\": \"^DONT_DEPLOY.*\",\n \"items_to_include\": [\"item1.Notebook\"],\n \"skip\": {\"dev\": True, \"test\": False, \"prod\": False},\n },\n \"unpublish\": {\"exclude_regex\": \"^DEBUG.*\", \"skip\": {\"dev\": True, \"test\": False, \"prod\": False}},\n \"features\": [\"enable_shortcut_publish\"],\n \"constants\": {\"DEFAULT_API_ROOT_URL\": \"https://msitapi.fabric.microsoft.com\"},\n }\n\n config_file = tmp_path / \"comprehensive_config.yml\"\n with Path.open(config_file, \"w\") as f:\n yaml.dump(config_data, f)\n\n # Test loading and parsing\n config = load_config_file(str(config_file), \"dev\")\n\n # Config validation may modify the config (e.g., resolve paths)\n # So we test the important parts separately\n assert \"core\" in config\n assert config[\"core\"][\"workspace_id\"] == config_data[\"core\"][\"workspace_id\"]\n assert \"Notebook\" in config[\"core\"][\"item_types_in_scope\"]\n assert \"publish\" in config\n assert config[\"publish\"][\"exclude_regex\"] == config_data[\"publish\"][\"exclude_regex\"]\n\n # Test all environment extractions\n for env in [\"dev\", \"test\", \"prod\"]:\n workspace_settings = extract_workspace_settings(config, env)\n assert workspace_settings[\"workspace_id\"] == config_data[\"core\"][\"workspace_id\"][env]\n\n publish_settings = extract_publish_settings(config, env)\n assert publish_settings[\"skip\"] == config_data[\"publish\"][\"skip\"][env]\n\n unpublish_settings = extract_unpublish_settings(config, env)\n assert unpublish_settings[\"skip\"] == config_data[\"unpublish\"][\"skip\"][env]\n\n\nclass TestConfigUtilsExtractSettings:\n \"\"\"Test config utility functions for extracting settings.\"\"\"\n\n def test_extract_publish_settings_with_folder_exclude_regex(self):\n \"\"\"Test extracting publish settings with folder_exclude_regex.\"\"\"\n config = {\n \"publish\": {\n \"folder_exclude_regex\": \"^/DONT_DEPLOY_FOLDER\",\n }\n }\n\n settings = extract_publish_settings(config, \"dev\")\n assert settings[\"folder_exclude_regex\"] == \"^/DONT_DEPLOY_FOLDER\"\n\n def test_extract_publish_settings_with_environment_specific_folder_exclude_regex(self):\n \"\"\"Test extracting publish settings with environment-specific folder_exclude_regex.\"\"\"\n config = {\n \"publish\": {\n \"folder_exclude_regex\": {\"dev\": \"^/DEV_FOLDER\", \"prod\": \"^/PROD_FOLDER\"},\n }\n }\n\n settings = extract_publish_settings(config, \"dev\")\n assert settings[\"folder_exclude_regex\"] == \"^/DEV_FOLDER\"\n\n settings = extract_publish_settings(config, \"prod\")\n assert settings[\"folder_exclude_regex\"] == \"^/PROD_FOLDER\"\n\n def test_extract_publish_settings_missing_environment_skips_setting(self):\n \"\"\"Test that missing environment in optional publish settings skips the setting.\"\"\"\n config = {\n \"publish\": {\n \"exclude_regex\": {\"dev\": \"^DEV.*\"}, # Only dev defined\n \"folder_exclude_regex\": {\"dev\": \"^/DEV_FOLDER\"}, # Only dev defined\n }\n }\n\n # prod environment not defined - settings should be skipped\n settings = extract_publish_settings(config, \"prod\")\n assert \"exclude_regex\" not in settings\n assert \"folder_exclude_regex\" not in settings\n\n def test_extract_unpublish_settings_missing_environment_skips_setting(self):\n \"\"\"Test that missing environment in optional unpublish settings skips the setting.\"\"\"\n config = {\n \"unpublish\": {\n \"exclude_regex\": {\"dev\": \"^DEV.*\"}, # Only dev defined\n \"items_to_include\": {\"dev\": [\"item1\"]}, # Only dev defined\n }\n }\n\n # prod environment not defined - settings should be skipped\n settings = extract_unpublish_settings(config, \"prod\")\n assert \"exclude_regex\" not in settings\n assert \"items_to_include\" not in settings\n\n def test_extract_publish_settings_skip_defaults_false_when_env_missing(self):\n \"\"\"Test that skip defaults to False when environment is not in skip mapping.\"\"\"\n config = {\n \"publish\": {\n \"skip\": {\"dev\": True}, # Only dev defined\n }\n }\n\n # prod environment not defined - skip should default to False\n settings = extract_publish_settings(config, \"prod\")\n assert settings[\"skip\"] is False\n\n def test_extract_unpublish_settings_skip_defaults_false_when_env_missing(self):\n \"\"\"Test that skip defaults to False when environment is not in skip mapping.\"\"\"\n config = {\n \"unpublish\": {\n \"skip\": {\"dev\": True}, # Only dev defined\n }\n }\n\n # prod environment not defined - skip should default to False\n settings = extract_unpublish_settings(config, \"prod\")\n assert settings[\"skip\"] is False\n\n def test_extract_workspace_settings_optional_fields_missing_environment(self):\n \"\"\"Test that optional workspace fields are skipped when environment is missing.\"\"\"\n config = {\n \"core\": {\n \"workspace_id\": \"12345678-1234-1234-1234-123456789abc\", # Simple value\n \"repository_directory\": \"/path/to/repo\",\n \"item_types_in_scope\": {\"dev\": [\"Notebook\"]}, # Only dev defined\n \"parameter\": {\"dev\": \"dev-param.yml\"}, # Only dev defined\n }\n }\n\n # prod environment not defined for optional fields - they should be skipped\n settings = extract_workspace_settings(config, \"prod\")\n assert \"item_types_in_scope\" not in settings\n assert \"parameter_file_path\" not in settings\n # Required fields should still be present\n assert settings[\"workspace_id\"] == \"12345678-1234-1234-1234-123456789abc\"\n assert settings[\"repository_directory\"] == \"/path/to/repo\"\n\n def test_extract_publish_settings_shortcut_exclude_regex_missing_environment(self):\n \"\"\"Test that shortcut_exclude_regex is skipped when environment is missing.\"\"\"\n config = {\n \"publish\": {\n \"shortcut_exclude_regex\": {\"dev\": \"^dev_temp_.*\"}, # Only dev defined\n }\n }\n\n # prod environment not defined - setting should be skipped\n settings = extract_publish_settings(config, \"prod\")\n assert \"shortcut_exclude_regex\" not in settings\n\n def test_extract_publish_settings_items_to_include_missing_environment(self):\n \"\"\"Test that items_to_include is skipped when environment is missing.\"\"\"\n config = {\n \"publish\": {\n \"items_to_include\": {\"dev\": [\"item1.Notebook\", \"item2.DataPipeline\"]}, # Only dev defined\n }\n }\n\n # prod environment not defined - setting should be skipped\n settings = extract_publish_settings(config, \"prod\")\n assert \"items_to_include\" not in settings\n\n def test_extract_publish_settings_folder_path_to_include_list(self):\n \"\"\"Test extract_publish_settings returns folder_path_to_include as a list.\"\"\"\n config = {\n \"publish\": {\n \"folder_path_to_include\": [\"/my/folder/path\"],\n },\n }\n result = extract_publish_settings(config, \"dev\")\n assert result[\"folder_path_to_include\"] == [\"/my/folder/path\"]\n\n def test_extract_publish_settings_folder_path_to_include_env_specific(self):\n \"\"\"Test extract_publish_settings resolves folder_path_to_include per environment.\"\"\"\n config = {\n \"publish\": {\n \"folder_path_to_include\": {\"dev\": [\"/dev/folder\"], \"prod\": [\"/prod/folder\"]},\n },\n }\n result = extract_publish_settings(config, \"dev\")\n assert result[\"folder_path_to_include\"] == [\"/dev/folder\"]\n\n def test_extract_publish_settings_folder_path_to_include_missing(self):\n \"\"\"Test extract_publish_settings defaults folder_path_to_include to None.\"\"\"\n config = {\n \"publish\": {\n \"exclude_regex\": \"^SKIP.*\",\n },\n }\n settings = extract_publish_settings(config, \"dev\")\n assert \"folder_path_to_include\" not in settings\n\n def test_extract_publish_settings_no_publish_section_folder_path_to_include(self):\n \"\"\"Test extract_publish_settings defaults folder_path_to_include to None when no publish section.\"\"\"\n config = {}\n settings = extract_publish_settings(config, \"dev\")\n assert \"folder_path_to_include\" not in settings\n\n\nclass TestGetConfigValue:\n \"\"\"Test the get_config_value utility function.\"\"\"\n\n def test_get_config_value_key_not_present(self):\n \"\"\"Test get_config_value when key doesn't exist.\"\"\"\n from fabric_cicd._common._config_utils import get_config_value\n\n config = {\"other_key\": \"value\"}\n result = get_config_value(config, \"missing_key\", \"dev\")\n assert result is None\n\n def test_get_config_value_simple_value(self):\n \"\"\"Test get_config_value with simple (non-dict) value.\"\"\"\n from fabric_cicd._common._config_utils import get_config_value\n\n config = {\"key\": \"simple_value\"}\n result = get_config_value(config, \"key\", \"dev\")\n assert result == \"simple_value\"\n\n def test_get_config_value_dict_with_environment(self):\n \"\"\"Test get_config_value with dict containing target environment.\"\"\"\n from fabric_cicd._common._config_utils import get_config_value\n\n config = {\"key\": {\"dev\": \"dev_value\", \"prod\": \"prod_value\"}}\n result = get_config_value(config, \"key\", \"dev\")\n assert result == \"dev_value\"\n\n def test_get_config_value_dict_missing_environment(self):\n \"\"\"Test get_config_value with dict missing target environment.\"\"\"\n from fabric_cicd._common._config_utils import get_config_value\n\n config = {\"key\": {\"dev\": \"dev_value\"}}\n result = get_config_value(config, \"key\", \"prod\")\n assert result is None\n\n def test_get_config_value_list_value(self):\n \"\"\"Test get_config_value with list value.\"\"\"\n from fabric_cicd._common._config_utils import get_config_value\n\n config = {\"key\": [\"item1\", \"item2\"]}\n result = get_config_value(config, \"key\", \"dev\")\n assert result == [\"item1\", \"item2\"]\n\n def test_get_config_value_bool_value(self):\n \"\"\"Test get_config_value with boolean value.\"\"\"\n from fabric_cicd._common._config_utils import get_config_value\n\n config = {\"key\": True}\n result = get_config_value(config, \"key\", \"dev\")\n assert result is True\n\n\nclass TestDeploymentResult:\n \"\"\"Test DeploymentResult and DeploymentStatus types.\"\"\"\n\n def test_deployment_status_completed_value(self):\n \"\"\"Test DeploymentStatus.COMPLETED has expected value.\"\"\"\n assert DeploymentStatus.COMPLETED.value == \"completed\"\n\n def test_deployment_status_failed_value(self):\n \"\"\"Test DeploymentStatus.FAILED has expected value.\"\"\"\n assert DeploymentStatus.FAILED.value == \"failed\"\n\n def test_deployment_status_string_comparison(self):\n \"\"\"Test DeploymentStatus supports string comparison via str inheritance.\"\"\"\n assert DeploymentStatus.COMPLETED == \"completed\"\n assert DeploymentStatus.FAILED == \"failed\"\n\n def test_deployment_result_structure(self):\n \"\"\"Test DeploymentResult fields are set correctly.\"\"\"\n result = DeploymentResult(\n status=DeploymentStatus.COMPLETED,\n message=\"Test message\",\n )\n assert result.status == DeploymentStatus.COMPLETED\n assert result.message == \"Test message\"\n\n def test_deployment_result_responses_defaults_to_none(self):\n \"\"\"Test DeploymentResult.responses defaults to None when not provided.\"\"\"\n result = DeploymentResult(\n status=DeploymentStatus.COMPLETED,\n message=\"Test message\",\n )\n assert result.responses is None\n\n def test_deployment_result_with_responses(self):\n \"\"\"Test DeploymentResult stores responses when provided.\"\"\"\n responses = {\"Notebook\": {\"MyNotebook\": {\"body\": {\"id\": \"123\"}}}}\n result = DeploymentResult(\n status=DeploymentStatus.FAILED,\n message=\"Deployment failed\",\n responses=responses,\n )\n assert result.status == DeploymentStatus.FAILED\n assert result.message == \"Deployment failed\"\n assert result.responses == responses\n\n\nclass TestDeployWithConfigReturnValue:\n \"\"\"Test deploy_with_config returns DeploymentResult.\"\"\"\n\n @patch(\"fabric_cicd.publish.FabricWorkspace\")\n @patch(\"fabric_cicd.publish.publish_all_items\")\n @patch(\"fabric_cicd.publish.unpublish_all_orphan_items\")\n @patch(\"fabric_cicd.constants.FEATURE_FLAG\", set([\"enable_experimental_features\", \"enable_config_deploy\"]))\n def test_deploy_with_config_returns_deployment_result(self, mock_unpublish, mock_publish, mock_workspace, tmp_path):\n \"\"\"Test that deploy_with_config returns a DeploymentResult on success.\"\"\"\n # Mark unused mocks to avoid linting warnings\n _ = mock_unpublish\n _ = mock_publish\n\n # Create the actual directory structure that the config references\n test_repo_dir = tmp_path / \"test\" / \"path\"\n test_repo_dir.mkdir(parents=True)\n\n # Create test config file\n config_data = {\n \"core\": {\n \"workspace_id\": {\"dev\": \"77777777-7777-7777-7777-777777777777\"},\n \"repository_directory\": \"test/path\",\n },\n }\n config_file = tmp_path / \"config.yml\"\n with Path.open(config_file, \"w\") as f:\n yaml.dump(config_data, f)\n\n # Mock workspace instance\n mock_workspace_instance = MagicMock()\n mock_workspace.return_value = mock_workspace_instance\n\n # Execute deployment\n result = deploy_with_config(config_file_path=str(config_file), token_credential=MagicMock(), environment=\"dev\")\n\n # Verify result is a DeploymentResult\n assert isinstance(result, DeploymentResult)\n assert result.status == DeploymentStatus.COMPLETED\n assert \"completed successfully\" in result.message\n\n @patch(\"fabric_cicd.publish.FabricWorkspace\")\n @patch(\"fabric_cicd.publish.publish_all_items\")\n @patch(\"fabric_cicd.publish.unpublish_all_orphan_items\")\n @patch(\"fabric_cicd.constants.FEATURE_FLAG\", set([\"enable_experimental_features\", \"enable_config_deploy\"]))\n def test_deploy_with_config_returns_completed_when_skipping_operations(\n self, mock_unpublish, mock_publish, mock_workspace, tmp_path\n ):\n \"\"\"Test that deploy_with_config returns COMPLETED status even when skipping operations.\"\"\"\n # Create the actual directory structure that the config references\n test_repo_dir = tmp_path / \"test\" / \"path\"\n test_repo_dir.mkdir(parents=True)\n\n # Create test config file with skip flags\n config_data = {\n \"core\": {\n \"workspace_id\": {\"dev\": \"88888888-8888-8888-8888-888888888888\"},\n \"repository_directory\": \"test/path\",\n },\n \"publish\": {\n \"skip\": {\"dev\": True},\n },\n \"unpublish\": {\n \"skip\": {\"dev\": True},\n },\n }\n config_file = tmp_path / \"config.yml\"\n with Path.open(config_file, \"w\") as f:\n yaml.dump(config_data, f)\n\n # Mock workspace instance\n mock_workspace_instance = MagicMock()\n mock_workspace.return_value = mock_workspace_instance\n\n # Execute deployment\n result = deploy_with_config(config_file_path=str(config_file), token_credential=MagicMock(), environment=\"dev\")\n\n # Verify result is a DeploymentResult with COMPLETED status\n assert isinstance(result, DeploymentResult)\n assert result.status == DeploymentStatus.COMPLETED\n\n # Verify that publish and unpublish are NOT called due to skip flags\n mock_publish.assert_not_called()\n mock_unpublish.assert_not_called()\n\n\nclass TestDeployWithConfigFailures:\n \"\"\"Test deploy_with_config raises exceptions properly on failure.\"\"\"\n\n def test_deploy_with_config_invalid_yaml_raises_input_error(self, tmp_path):\n \"\"\"Test that deploy_with_config raises InputError for invalid YAML syntax.\"\"\"\n config_file = tmp_path / \"invalid.yml\"\n config_file.write_text(\"invalid: yaml: content: [\")\n\n with pytest.raises(InputError, match=\"Invalid YAML syntax\"):\n deploy_with_config(config_file_path=str(config_file), token_credential=MagicMock(), environment=\"dev\")\n\n def test_deploy_with_config_missing_core_raises_config_validation_error(self, tmp_path):\n \"\"\"Test that deploy_with_config raises ConfigValidationError when core section is missing.\"\"\"\n config_data = {\"publish\": {\"skip\": True}}\n config_file = tmp_path / \"no_core.yml\"\n with Path.open(config_file, \"w\") as f:\n yaml.dump(config_data, f)\n\n with pytest.raises(ConfigValidationError, match=\"must contain a 'core' section\"):\n deploy_with_config(config_file_path=str(config_file), token_credential=MagicMock(), environment=\"dev\")\n\n def test_deploy_with_config_missing_environment_raises_config_validation_error(self, tmp_path):\n \"\"\"Test that deploy_with_config raises ConfigValidationError when environment is not in workspace mappings.\"\"\"\n test_repo_dir = tmp_path / \"test\" / \"path\"\n test_repo_dir.mkdir(parents=True)\n\n config_data = {\n \"core\": {\n \"workspace_id\": {\"dev\": \"12345678-1234-1234-1234-123456789abc\"},\n \"repository_directory\": \"test/path\",\n }\n }\n config_file = tmp_path / \"config.yml\"\n with Path.open(config_file, \"w\") as f:\n yaml.dump(config_data, f)\n\n with pytest.raises(ConfigValidationError, match=\"Environment 'prod' not found\"):\n deploy_with_config(config_file_path=str(config_file), token_credential=MagicMock(), environment=\"prod\")\n\n def test_deploy_with_config_missing_workspace_id_raises_config_validation_error(self, tmp_path):\n \"\"\"Test that deploy_with_config raises ConfigValidationError when workspace_id is missing.\"\"\"\n config_data = {\n \"core\": {\n \"repository_directory\": \"test/path\",\n }\n }\n config_file = tmp_path / \"config.yml\"\n with Path.open(config_file, \"w\") as f:\n yaml.dump(config_data, f)\n\n with pytest.raises(ConfigValidationError, match=\"must specify either 'workspace_id' or 'workspace'\"):\n deploy_with_config(config_file_path=str(config_file), token_credential=MagicMock(), environment=\"dev\")\n\n @patch(\"fabric_cicd.publish.FabricWorkspace\")\n @patch(\"fabric_cicd.publish.publish_all_items\")\n @patch(\"fabric_cicd.publish.unpublish_all_orphan_items\")\n def test_deploy_with_config_publish_error_propagates(self, mock_unpublish, mock_publish, mock_workspace, tmp_path):\n \"\"\"Test that PublishError from publish_all_items propagates through deploy_with_config.\"\"\"\n _ = mock_unpublish\n\n test_repo_dir = tmp_path / \"test\" / \"path\"\n test_repo_dir.mkdir(parents=True)\n\n config_data = {\n \"core\": {\n \"workspace_id\": {\"dev\": \"77777777-7777-7777-7777-777777777777\"},\n \"repository_directory\": \"test/path\",\n },\n }\n config_file = tmp_path / \"config.yml\"\n with Path.open(config_file, \"w\") as f:\n yaml.dump(config_data, f)\n\n mock_workspace.return_value = MagicMock()\n mock_publish.side_effect = PublishError(\n errors=[(\"FailedNotebook\", RuntimeError(\"API call failed\"))],\n logger=MagicMock(),\n )\n\n with pytest.raises(PublishError, match=\"Failed to publish 1 item\"):\n deploy_with_config(config_file_path=str(config_file), token_credential=MagicMock(), environment=\"dev\")\n\n @patch(\"fabric_cicd.publish.FabricWorkspace\")\n @patch(\"fabric_cicd.publish.publish_all_items\")\n @patch(\"fabric_cicd.publish.unpublish_all_orphan_items\")\n def test_deploy_with_config_workspace_creation_error_propagates(\n self, mock_unpublish, mock_publish, mock_workspace, tmp_path\n ):\n \"\"\"Test that exceptions from FabricWorkspace constructor propagate through deploy_with_config.\"\"\"\n _ = mock_unpublish\n _ = mock_publish\n\n test_repo_dir = tmp_path / \"test\" / \"path\"\n test_repo_dir.mkdir(parents=True)\n\n config_data = {\n \"core\": {\n \"workspace_id\": {\"dev\": \"77777777-7777-7777-7777-777777777777\"},\n \"repository_directory\": \"test/path\",\n },\n }\n config_file = tmp_path / \"config.yml\"\n with Path.open(config_file, \"w\") as f:\n yaml.dump(config_data, f)\n\n mock_workspace.side_effect = Exception(\"Workspace initialization failed\")\n\n with pytest.raises(Exception, match=\"Workspace initialization failed\"):\n deploy_with_config(config_file_path=str(config_file), token_credential=MagicMock(), environment=\"dev\")\n\n mock_publish.assert_not_called()\n mock_unpublish.assert_not_called()\n\n @patch(\"fabric_cicd.publish.FabricWorkspace\")\n @patch(\"fabric_cicd.publish.publish_all_items\")\n @patch(\"fabric_cicd.publish.unpublish_all_orphan_items\")\n def test_deploy_with_config_unpublish_error_propagates(\n self, mock_unpublish, mock_publish, mock_workspace, tmp_path\n ):\n \"\"\"Test that exceptions from unpublish_all_orphan_items propagate and publish was already called.\"\"\"\n test_repo_dir = tmp_path / \"test\" / \"path\"\n test_repo_dir.mkdir(parents=True)\n\n config_data = {\n \"core\": {\n \"workspace_id\": {\"dev\": \"77777777-7777-7777-7777-777777777777\"},\n \"repository_directory\": \"test/path\",\n },\n }\n config_file = tmp_path / \"config.yml\"\n with Path.open(config_file, \"w\") as f:\n yaml.dump(config_data, f)\n\n mock_workspace.return_value = MagicMock()\n mock_unpublish.side_effect = RuntimeError(\"Unpublish operation failed\")\n\n with pytest.raises(RuntimeError, match=\"Unpublish operation failed\"):\n deploy_with_config(config_file_path=str(config_file), token_credential=MagicMock(), environment=\"dev\")\n\n # Verify publish was called successfully before unpublish failed\n mock_publish.assert_called_once()\n\n\nclass TestDeployWithConfigExceptionAttributes:\n \"\"\"Test that deploy_with_config attaches deployment attributes to exceptions.\"\"\"\n\n @patch(\"fabric_cicd.publish.FabricWorkspace\")\n @patch(\"fabric_cicd.publish.publish_all_items\")\n @patch(\"fabric_cicd.publish.unpublish_all_orphan_items\")\n def test_exception_has_deployment_status_and_message(self, mock_unpublish, mock_publish, mock_workspace, tmp_path):\n \"\"\"Test that raised exceptions have deployment_status and deployment_message attributes.\"\"\"\n _ = mock_unpublish\n\n test_repo_dir = tmp_path / \"test\" / \"path\"\n test_repo_dir.mkdir(parents=True)\n\n config_data = {\n \"core\": {\n \"workspace_id\": {\"dev\": \"77777777-7777-7777-7777-777777777777\"},\n \"repository_directory\": \"test/path\",\n },\n }\n config_file = tmp_path / \"config.yml\"\n with Path.open(config_file, \"w\") as f:\n yaml.dump(config_data, f)\n\n mock_workspace.return_value = MagicMock()\n mock_publish.side_effect = RuntimeError(\"Something broke\")\n\n with pytest.raises(RuntimeError) as exc_info:\n deploy_with_config(config_file_path=str(config_file), token_credential=MagicMock(), environment=\"dev\")\n\n e = exc_info.value\n assert hasattr(e, \"deployment_result\")\n assert e.deployment_result.status == DeploymentStatus.FAILED\n assert \"Something broke\" in e.deployment_result.message\n\n @patch(\"fabric_cicd.publish.FabricWorkspace\")\n @patch(\"fabric_cicd.publish.publish_all_items\")\n @patch(\"fabric_cicd.publish.unpublish_all_orphan_items\")\n @patch(\n \"fabric_cicd.constants.FEATURE_FLAG\",\n set([\"enable_experimental_features\", \"enable_config_deploy\", \"enable_response_collection\"]),\n )\n def test_exception_has_partial_responses_when_enabled(self, mock_unpublish, mock_publish, mock_workspace, tmp_path):\n \"\"\"Test that partial responses are attached to exceptions when response collection is enabled.\"\"\"\n _ = mock_publish\n\n test_repo_dir = tmp_path / \"test\" / \"path\"\n test_repo_dir.mkdir(parents=True)\n\n config_data = {\n \"core\": {\n \"workspace_id\": {\"dev\": \"77777777-7777-7777-7777-777777777777\"},\n \"repository_directory\": \"test/path\",\n },\n }\n config_file = tmp_path / \"config.yml\"\n with Path.open(config_file, \"w\") as f:\n yaml.dump(config_data, f)\n\n mock_workspace_instance = MagicMock()\n mock_workspace_instance.responses = {\"Notebook\": {\"MyNotebook\": {\"body\": {\"id\": \"123\"}}}}\n mock_workspace_instance.unpublish_responses = None\n mock_workspace.return_value = mock_workspace_instance\n\n mock_unpublish.side_effect = RuntimeError(\"Unpublish failed\")\n\n with pytest.raises(RuntimeError) as exc_info:\n deploy_with_config(config_file_path=str(config_file), token_credential=MagicMock(), environment=\"dev\")\n\n e = exc_info.value\n assert hasattr(e, \"deployment_result\")\n assert e.deployment_result.responses == {\"publish\": {\"Notebook\": {\"MyNotebook\": {\"body\": {\"id\": \"123\"}}}}}\n\n @patch(\"fabric_cicd.publish.FabricWorkspace\")\n @patch(\"fabric_cicd.publish.publish_all_items\")\n @patch(\"fabric_cicd.publish.unpublish_all_orphan_items\")\n def test_exception_no_responses_when_flag_disabled(self, mock_unpublish, mock_publish, mock_workspace, tmp_path):\n \"\"\"Test that responses are not attached to exceptions when response collection is disabled.\"\"\"\n _ = mock_unpublish\n\n test_repo_dir = tmp_path / \"test\" / \"path\"\n test_repo_dir.mkdir(parents=True)\n\n config_data = {\n \"core\": {\n \"workspace_id\": {\"dev\": \"77777777-7777-7777-7777-777777777777\"},\n \"repository_directory\": \"test/path\",\n },\n }\n config_file = tmp_path / \"config.yml\"\n with Path.open(config_file, \"w\") as f:\n yaml.dump(config_data, f)\n\n mock_workspace.return_value = MagicMock()\n mock_publish.side_effect = RuntimeError(\"Publish failed\")\n\n with pytest.raises(RuntimeError) as exc_info:\n deploy_with_config(config_file_path=str(config_file), token_credential=MagicMock(), environment=\"dev\")\n\n e = exc_info.value\n assert hasattr(e, \"deployment_result\")\n assert e.deployment_result.responses is None\n\n def test_pre_workspace_failure_has_deployment_attributes(self, tmp_path):\n \"\"\"Test that failures before workspace creation still have deployment attributes.\"\"\"\n config_file = tmp_path / \"invalid.yml\"\n config_file.write_text(\"invalid: yaml: content: [\")\n\n with pytest.raises(InputError) as exc_info:\n deploy_with_config(config_file_path=str(config_file), token_credential=MagicMock(), environment=\"dev\")\n\n e = exc_info.value\n assert hasattr(e, \"deployment_result\")\n assert e.deployment_result.status == DeploymentStatus.FAILED\n assert e.deployment_result.message is not None\n assert e.deployment_result.responses is None\n\n\nclass TestDeployWithConfigResponseCollection:\n \"\"\"Test deploy_with_config response collection integration.\"\"\"\n\n @patch(\"fabric_cicd.publish.FabricWorkspace\")\n @patch(\"fabric_cicd.publish.publish_all_items\")\n @patch(\"fabric_cicd.publish.unpublish_all_orphan_items\")\n @patch(\n \"fabric_cicd.constants.FEATURE_FLAG\",\n set([\"enable_experimental_features\", \"enable_config_deploy\", \"enable_response_collection\"]),\n )\n def test_result_responses_is_dict_when_enabled(self, mock_unpublish, mock_publish, mock_workspace, tmp_path):\n \"\"\"Test that result.responses is a dict (not string) when response collection is enabled.\"\"\"\n _ = mock_unpublish\n _ = mock_publish\n\n test_repo_dir = tmp_path / \"test\" / \"path\"\n test_repo_dir.mkdir(parents=True)\n\n config_data = {\n \"core\": {\n \"workspace_id\": {\"dev\": \"77777777-7777-7777-7777-777777777777\"},\n \"repository_directory\": \"test/path\",\n },\n }\n config_file = tmp_path / \"config.yml\"\n with Path.open(config_file, \"w\") as f:\n yaml.dump(config_data, f)\n\n mock_workspace_instance = MagicMock()\n mock_workspace_instance.responses = {\"Notebook\": {\"MyNotebook\": {\"body\": {\"id\": \"123\"}}}}\n mock_workspace_instance.unpublish_responses = None\n mock_workspace.return_value = mock_workspace_instance\n\n result = deploy_with_config(config_file_path=str(config_file), token_credential=MagicMock(), environment=\"dev\")\n\n assert isinstance(result, DeploymentResult)\n assert isinstance(result.responses, dict)\n assert result.responses == {\"publish\": {\"Notebook\": {\"MyNotebook\": {\"body\": {\"id\": \"123\"}}}}}\n\n @patch(\"fabric_cicd.publish.FabricWorkspace\")\n @patch(\"fabric_cicd.publish.publish_all_items\")\n @patch(\"fabric_cicd.publish.unpublish_all_orphan_items\")\n @patch(\n \"fabric_cicd.constants.FEATURE_FLAG\",\n set([\"enable_experimental_features\", \"enable_config_deploy\", \"enable_response_collection\"]),\n )\n def test_result_responses_contains_both_publish_and_unpublish(\n self, mock_unpublish, mock_publish, mock_workspace, tmp_path\n ):\n \"\"\"Test that result.responses contains both publish and unpublish keys when both are collected.\"\"\"\n _ = mock_unpublish\n _ = mock_publish\n\n test_repo_dir = tmp_path / \"test\" / \"path\"\n test_repo_dir.mkdir(parents=True)\n\n config_data = {\n \"core\": {\n \"workspace_id\": {\"dev\": \"77777777-7777-7777-7777-777777777777\"},\n \"repository_directory\": \"test/path\",\n },\n }\n config_file = tmp_path / \"config.yml\"\n with Path.open(config_file, \"w\") as f:\n yaml.dump(config_data, f)\n\n mock_workspace_instance = MagicMock()\n mock_workspace_instance.responses = {\"Notebook\": {\"nb1\": {\"body\": {\"id\": \"123\"}}}}\n mock_workspace_instance.unpublish_responses = {\"Notebook\": {\"nb2\": {\"body\": {\"id\": \"456\"}}}}\n mock_workspace.return_value = mock_workspace_instance\n\n result = deploy_with_config(config_file_path=str(config_file), token_credential=MagicMock(), environment=\"dev\")\n\n assert isinstance(result, DeploymentResult)\n assert result.responses == {\n \"publish\": {\"Notebook\": {\"nb1\": {\"body\": {\"id\": \"123\"}}}},\n \"unpublish\": {\"Notebook\": {\"nb2\": {\"body\": {\"id\": \"456\"}}}},\n }\n\n @patch(\"fabric_cicd.publish.FabricWorkspace\")\n @patch(\"fabric_cicd.publish.publish_all_items\")\n @patch(\"fabric_cicd.publish.unpublish_all_orphan_items\")\n @patch(\n \"fabric_cicd.constants.FEATURE_FLAG\",\n set([\"enable_experimental_features\", \"enable_config_deploy\"]),\n )\n def test_result_responses_is_none_when_disabled(self, mock_unpublish, mock_publish, mock_workspace, tmp_path):\n \"\"\"Test that result.responses is None when response collection is not enabled.\"\"\"\n _ = mock_unpublish\n _ = mock_publish\n\n test_repo_dir = tmp_path / \"test\" / \"path\"\n test_repo_dir.mkdir(parents=True)\n\n config_data = {\n \"core\": {\n \"workspace_id\": {\"dev\": \"77777777-7777-7777-7777-777777777777\"},\n \"repository_directory\": \"test/path\",\n },\n }\n config_file = tmp_path / \"config.yml\"\n with Path.open(config_file, \"w\") as f:\n yaml.dump(config_data, f)\n\n mock_workspace_instance = MagicMock()\n mock_workspace.return_value = mock_workspace_instance\n\n result = deploy_with_config(config_file_path=str(config_file), token_credential=MagicMock(), environment=\"dev\")\n\n assert isinstance(result, DeploymentResult)\n assert result.responses is None\n\n\nclass TestCollectResponses:\n \"\"\"Test the _collect_responses helper function.\"\"\"\n\n def test_returns_none_when_responses_disabled(self):\n from fabric_cicd.publish import _collect_responses\n\n workspace = MagicMock()\n workspace.responses = {\"Notebook\": {\"nb1\": {\"body\": {}}}}\n assert _collect_responses(workspace, responses_enabled=False) is None\n\n def test_returns_none_when_workspace_is_none(self):\n from fabric_cicd.publish import _collect_responses\n\n assert _collect_responses(None, responses_enabled=True) is None\n\n def test_returns_none_when_both_responses_empty_dict(self):\n from fabric_cicd.publish import _collect_responses\n\n workspace = MagicMock()\n workspace.responses = {}\n workspace.unpublish_responses = {}\n assert _collect_responses(workspace, responses_enabled=True) is None\n\n def test_returns_none_when_both_responses_none(self):\n from fabric_cicd.publish import _collect_responses\n\n workspace = MagicMock()\n workspace.responses = None\n workspace.unpublish_responses = None\n assert _collect_responses(workspace, responses_enabled=True) is None\n\n def test_returns_none_when_responses_empty_and_unpublish_none(self):\n from fabric_cicd.publish import _collect_responses\n\n workspace = MagicMock()\n workspace.responses = {}\n workspace.unpublish_responses = None\n assert _collect_responses(workspace, responses_enabled=True) is None\n\n def test_returns_none_when_responses_none_and_unpublish_empty(self):\n from fabric_cicd.publish import _collect_responses\n\n workspace = MagicMock()\n workspace.responses = None\n workspace.unpublish_responses = {}\n assert _collect_responses(workspace, responses_enabled=True) is None\n\n def test_returns_publish_only_when_publish_available(self):\n from fabric_cicd.publish import _collect_responses\n\n workspace = MagicMock()\n workspace.responses = {\"Notebook\": {\"nb1\": {\"body\": {\"id\": \"123\"}}}}\n workspace.unpublish_responses = None\n result = _collect_responses(workspace, responses_enabled=True)\n assert result == {\"publish\": {\"Notebook\": {\"nb1\": {\"body\": {\"id\": \"123\"}}}}}\n\n def test_returns_unpublish_only_when_unpublish_available(self):\n from fabric_cicd.publish import _collect_responses\n\n workspace = MagicMock()\n workspace.responses = None\n workspace.unpublish_responses = {\"Notebook\": {\"nb1\": {\"body\": {\"id\": \"456\"}}}}\n result = _collect_responses(workspace, responses_enabled=True)\n assert result == {\"unpublish\": {\"Notebook\": {\"nb1\": {\"body\": {\"id\": \"456\"}}}}}\n\n def test_returns_both_when_both_available(self):\n from fabric_cicd.publish import _collect_responses\n\n workspace = MagicMock()\n workspace.responses = {\"Notebook\": {\"nb1\": {\"body\": {\"id\": \"123\"}}}}\n workspace.unpublish_responses = {\"Notebook\": {\"nb2\": {\"body\": {\"id\": \"456\"}}}}\n result = _collect_responses(workspace, responses_enabled=True)\n assert result == {\n \"publish\": {\"Notebook\": {\"nb1\": {\"body\": {\"id\": \"123\"}}}},\n \"unpublish\": {\"Notebook\": {\"nb2\": {\"body\": {\"id\": \"456\"}}}},\n }\n" }, { "path": "tests/test_environment_publish.py", "content": "from pathlib import Path\nfrom typing import ClassVar\n\nimport yaml\n\nfrom fabric_cicd._items import _environment as env_module\n\n\nclass DummyFile:\n def __init__(self, file_path):\n # accept Path or string\n self.file_path = Path(file_path)\n # keep attributes other code may inspect\n self.relative_path = str(self.file_path).replace(\"\\\\\", \"/\")\n self.type = \"text\"\n self.base64_payload = \"payload\"\n # Read contents from file if it exists, otherwise empty string\n if self.file_path.exists():\n self.contents = self.file_path.read_text(encoding=\"utf-8\")\n else:\n self.contents = \"\"\n\n\nclass DummyItem:\n def __init__(self, name, file_paths):\n self.name = name\n self.item_files = [DummyFile(p) for p in file_paths]\n # Set path to the environment item directory (parent of the folder that\n # contains the file), e.g. if file is /tmp/Env/Setting/Sparkcompute.yml ->\n # path should be /tmp/Env\n if file_paths:\n p = Path(file_paths[0])\n # if parent has at least one parent, choose grandparent; otherwise use parent\n self.path = p.parent.parent if p.parent.parent != Path() else p.parent\n else:\n self.path = Path()\n self.guid = None\n self.skip_publish = False\n\n\n# ---------- func_process_file tests ----------\n\n\ndef test_process_environment_file_non_sparkcompute(tmp_path):\n \"\"\"Non-Sparkcompute files are returned unchanged.\"\"\"\n f = tmp_path / \"EnvA\" / \"Libraries\" / \"lib.txt\"\n f.parent.mkdir(parents=True, exist_ok=True)\n f.write_text(\"original content\", encoding=\"utf-8\")\n\n dummy = DummyFile(f)\n result = env_module._process_environment_file(None, DummyItem(\"EnvA\", [f]), dummy)\n assert result == \"original content\"\n\n\ndef test_process_environment_file_no_instance_pool(tmp_path):\n \"\"\"Sparkcompute.yml without instance_pool_id is returned as re-serialized YAML.\"\"\"\n env_dir = tmp_path / \"EnvB\"\n setting_dir = env_dir / \"Setting\"\n setting_dir.mkdir(parents=True, exist_ok=True)\n sc = setting_dir / \"Sparkcompute.yml\"\n sc.write_text(\"driver_cores: 8\\ndriver_memory: 56g\\n\", encoding=\"utf-8\")\n\n dummy = DummyFile(sc)\n result = env_module._process_environment_file(None, DummyItem(\"EnvB\", [sc]), dummy)\n parsed = yaml.safe_load(result)\n assert parsed[\"driver_cores\"] == 8\n assert parsed[\"driver_memory\"] == \"56g\"\n assert \"instance_pool_id\" not in parsed\n\n\ndef test_process_environment_file_replaces_instance_pool(tmp_path):\n \"\"\"Sparkcompute.yml with instance_pool_id is resolved to the target pool GUID via API.\"\"\"\n env_dir = tmp_path / \"EnvC\"\n setting_dir = env_dir / \"Setting\"\n setting_dir.mkdir(parents=True, exist_ok=True)\n sc = setting_dir / \"Sparkcompute.yml\"\n sc.write_text(\"instance_pool_id: pool-123\\ndriver_cores: 4\\n\", encoding=\"utf-8\")\n\n class FakeWS:\n environment = \"DEV\"\n environment_parameter: ClassVar[dict] = {\n \"spark_pool\": [\n {\n \"instance_pool_id\": \"pool-123\",\n \"replace_value\": {\"DEV\": {\"type\": \"Capacity\", \"name\": \"MyPool\"}},\n }\n ]\n }\n base_api_url = \"https://api.example/v1/workspaces/ws-id\"\n\n def _get_workspace_pools(self):\n return [\n {\"id\": \"resolved-guid-abc\", \"name\": \"MyPool\", \"type\": \"Capacity\"},\n {\"id\": \"other-guid\", \"name\": \"OtherPool\", \"type\": \"Workspace\"},\n ]\n\n dummy = DummyFile(sc)\n result = env_module._process_environment_file(FakeWS(), DummyItem(\"EnvC\", [sc]), dummy)\n parsed = yaml.safe_load(result)\n assert parsed[\"instance_pool_id\"] == \"resolved-guid-abc\"\n assert \"instance_pool\" not in parsed\n assert parsed[\"driver_cores\"] == 4\n\n\ndef test_process_environment_file_pool_with_item_name_filter(tmp_path):\n \"\"\"instance_pool_id replacement respects the optional item_name filter.\"\"\"\n env_dir = tmp_path / \"EnvD\"\n setting_dir = env_dir / \"Setting\"\n setting_dir.mkdir(parents=True, exist_ok=True)\n sc = setting_dir / \"Sparkcompute.yml\"\n sc.write_text(\"instance_pool_id: pool-456\\n\", encoding=\"utf-8\")\n\n class FakeWS:\n environment = \"PROD\"\n environment_parameter: ClassVar[dict] = {\n \"spark_pool\": [\n {\n \"instance_pool_id\": \"pool-456\",\n \"replace_value\": {\"PROD\": {\"type\": \"Workspace\", \"name\": \"WsPool\"}},\n \"item_name\": \"EnvD\",\n }\n ]\n }\n base_api_url = \"https://api.example/v1/workspaces/ws-id\"\n\n def _get_workspace_pools(self):\n return [{\"id\": \"ws-pool-guid\", \"name\": \"WsPool\", \"type\": \"Workspace\"}]\n\n dummy = DummyFile(sc)\n result = env_module._process_environment_file(FakeWS(), DummyItem(\"EnvD\", [sc]), dummy)\n parsed = yaml.safe_load(result)\n assert parsed[\"instance_pool_id\"] == \"ws-pool-guid\"\n\n\ndef test_process_environment_file_pool_no_match(tmp_path):\n \"\"\"When no spark_pool entry matches, instance_pool_id is left as-is.\"\"\"\n env_dir = tmp_path / \"EnvE\"\n setting_dir = env_dir / \"Setting\"\n setting_dir.mkdir(parents=True, exist_ok=True)\n sc = setting_dir / \"Sparkcompute.yml\"\n sc.write_text(\"instance_pool_id: unmatched-pool\\n\", encoding=\"utf-8\")\n\n class FakeWS:\n environment = \"DEV\"\n environment_parameter: ClassVar[dict] = {\n \"spark_pool\": [{\"instance_pool_id\": \"different-pool\", \"replace_value\": {\"DEV\": \"something\"}}]\n }\n base_api_url = \"https://api.example/v1/workspaces/ws-id\"\n\n def _get_workspace_pools(self):\n return [{\"id\": \"guid-other\", \"name\": \"OtherPool\", \"type\": \"Capacity\"}]\n\n dummy = DummyFile(sc)\n result = env_module._process_environment_file(FakeWS(), DummyItem(\"EnvE\", [sc]), dummy)\n parsed = yaml.safe_load(result)\n assert \"instance_pool_id\" in parsed\n assert parsed[\"instance_pool_id\"] == \"unmatched-pool\"\n\n\ndef test_process_environment_file_no_spark_pool_param(tmp_path):\n \"\"\"When environment_parameter has no spark_pool, instance_pool_id is left as-is.\"\"\"\n env_dir = tmp_path / \"EnvF\"\n setting_dir = env_dir / \"Setting\"\n setting_dir.mkdir(parents=True, exist_ok=True)\n sc = setting_dir / \"Sparkcompute.yml\"\n sc.write_text(\"instance_pool_id: some-pool\\n\", encoding=\"utf-8\")\n\n class FakeWS:\n environment = \"DEV\"\n environment_parameter: ClassVar[dict] = {}\n base_api_url = \"https://api.example/v1/workspaces/ws-id\"\n\n def _get_workspace_pools(self):\n return []\n\n dummy = DummyFile(sc)\n result = env_module._process_environment_file(FakeWS(), DummyItem(\"EnvF\", [sc]), dummy)\n parsed = yaml.safe_load(result)\n assert parsed[\"instance_pool_id\"] == \"some-pool\"\n\n\ndef test_resolve_pool_id_success():\n \"\"\"_resolve_pool_id returns the matching pool GUID from the API response.\"\"\"\n pools = [\n {\"id\": \"guid-cap\", \"name\": \"CapPool\", \"type\": \"Capacity\"},\n {\"id\": \"guid-ws\", \"name\": \"WsPool\", \"type\": \"Workspace\"},\n ]\n assert env_module._resolve_pool_id(pools, pool_name=\"CapPool\", pool_type=\"Capacity\") == \"guid-cap\"\n assert env_module._resolve_pool_id(pools, pool_name=\"WsPool\", pool_type=\"Workspace\") == \"guid-ws\"\n\n\ndef test_resolve_pool_id_not_found():\n \"\"\"_resolve_pool_id raises when no matching pool is found.\"\"\"\n import pytest\n\n pools = [{\"id\": \"guid-other\", \"name\": \"OtherPool\", \"type\": \"Capacity\"}]\n\n with pytest.raises(Exception, match=\"Could not resolve custom Spark pool\"):\n env_module._resolve_pool_id(pools, pool_name=\"MissingPool\", pool_type=\"Workspace\")\n\n\n# ---------- Publisher integration tests ----------\n\n\ndef test_publish_environments_passes_func_process_file(tmp_path):\n \"\"\"\n Ensure EnvironmentPublisher passes func_process_file to _publish_item\n and no longer passes shell_only_publish or exclude_path.\n \"\"\"\n captured = {}\n\n class FakeEndpoint:\n def invoke(self, *_args, **_kwargs):\n return {\"body\": {\"value\": []}}\n\n class FakeWorkspace:\n def __init__(self):\n p = tmp_path / \"EnvX\" / \"Setting\" / \"Sparkcompute.yml\"\n self.repository_items = {\"Environment\": {\"EnvX\": DummyItem(\"EnvX\", [p])}}\n self.publish_item_name_exclude_regex = None\n self.publish_folder_path_exclude_regex = None\n self.items_to_include = None\n self.base_api_url = \"https://example\"\n self.endpoint = FakeEndpoint()\n self.repository_directory = tmp_path\n self.responses = None\n\n def _get_workspace_pools(self):\n return []\n\n def _publish_item(self, item_name, item_type, **kwargs):\n captured[\"called_with\"] = kwargs\n self.repository_items[item_type][item_name].skip_publish = True\n\n ws = FakeWorkspace()\n env_module.EnvironmentPublisher(ws).publish_all()\n assert \"func_process_file\" in captured[\"called_with\"]\n assert captured[\"called_with\"][\"func_process_file\"] is env_module._process_environment_file\n assert \"shell_only_publish\" not in captured[\"called_with\"]\n assert \"exclude_path\" not in captured[\"called_with\"]\n\n\n# ---------- End-to-end style tests ----------\n\n\ndef test_end_to_end_environment_setting_only(tmp_path):\n \"\"\"\n End-to-end style test for an Environment item that contains only Setting.\n Verifies: create item (POST /items) and submit publish (POST /staging/publish).\n Sparkcompute.yml is included in the regular item definition—no separate\n PATCH sparkcompute call is made.\n \"\"\"\n env_dir = tmp_path / \"EnvEnd1\"\n setting_dir = env_dir / \"Setting\"\n setting_dir.mkdir(parents=True, exist_ok=True)\n spark_yaml = setting_dir / \"Sparkcompute.yml\"\n spark_yaml.write_text(\"driver_cores: 4\\n\", encoding=\"utf-8\")\n\n calls = []\n\n class FakeEndpoint:\n def invoke(self, method=None, url=None, body=None, **_kwargs):\n calls.append((method, url, body))\n if method == \"GET\" and url.endswith(\"/environments/\"):\n return {\"body\": {\"value\": []}}\n if method == \"POST\" and url.endswith(\"/items\"):\n return {\"body\": {\"id\": \"guid-123\"}}\n if method == \"POST\" and url.endswith(\"/staging/publish?beta=False\"):\n return {\"status\": 202}\n return {}\n\n class FakeWorkspace:\n def __init__(self):\n self.repository_items = {\"Environment\": {\"EnvEnd1\": DummyItem(\"EnvEnd1\", [spark_yaml])}}\n self.publish_item_name_exclude_regex = None\n self.publish_folder_path_exclude_regex = None\n self.items_to_include = None\n self.base_api_url = \"https://api.example\"\n self.endpoint = FakeEndpoint()\n self.repository_directory = tmp_path\n self.responses = None\n self.environment_parameter = {}\n\n def _get_workspace_pools(self):\n return []\n\n def _publish_item(self, item_name, item_type, **_kwargs):\n item = self.repository_items[item_type][item_name]\n if not item.guid:\n resp = self.endpoint.invoke(method=\"POST\", url=f\"{self.base_api_url}/items\", body={})\n item.guid = resp[\"body\"][\"id\"]\n\n ws = FakeWorkspace()\n env_module.EnvironmentPublisher(ws).publish_all()\n\n urls = [c[1] for c in calls]\n assert any(\"/items\" in u and u.endswith(\"/items\") for u in urls), \"Create item call missing\"\n assert any(u.endswith(\"/staging/publish?beta=False\") for u in urls), \"Publish submit missing\"\n assert not any(\"sparkcompute\" in u for u in urls), \"Unexpected sparkcompute PATCH call\"\n\n\ndef test_end_to_end_environment_with_libraries(tmp_path):\n \"\"\"\n End-to-end style test for an Environment item that contains both Setting and\n Libraries. Verifies create/update flow and staging publish—no separate\n PATCH sparkcompute call.\n \"\"\"\n env_dir = tmp_path / \"EnvEnd2\"\n setting_dir = env_dir / \"Setting\"\n libs_dir = env_dir / \"Libraries\"\n setting_dir.mkdir(parents=True, exist_ok=True)\n libs_dir.mkdir(parents=True, exist_ok=True)\n spark_yaml = setting_dir / \"Sparkcompute.yml\"\n spark_yaml.write_text(\"driver_cores: 8\\n\", encoding=\"utf-8\")\n (libs_dir / \"lib.zip\").write_text(\"dummy\", encoding=\"utf-8\")\n\n calls = []\n\n class FakeEndpoint:\n def invoke(self, method=None, url=None, body=None, **_kwargs):\n calls.append((method, url, body))\n if method == \"GET\" and url.endswith(\"/environments/\"):\n return {\"body\": {\"value\": []}}\n if method == \"POST\" and url.endswith(\"/items\"):\n return {\"body\": {\"id\": \"guid-456\"}}\n if method == \"POST\" and \"updateDefinition\" in url:\n return {\"status\": 200}\n if method == \"POST\" and url.endswith(\"/staging/publish?beta=False\"):\n return {\"status\": 202}\n return {}\n\n class FakeWorkspace:\n def __init__(self):\n p_set = spark_yaml\n p_lib = libs_dir / \"lib.zip\"\n self.repository_items = {\"Environment\": {\"EnvEnd2\": DummyItem(\"EnvEnd2\", [p_set, p_lib])}}\n self.publish_item_name_exclude_regex = None\n self.publish_folder_path_exclude_regex = None\n self.items_to_include = None\n self.base_api_url = \"https://api.example\"\n self.endpoint = FakeEndpoint()\n self.repository_directory = tmp_path\n self.responses = None\n self.environment_parameter = {}\n\n def _get_workspace_pools(self):\n return []\n\n def _publish_item(self, item_name, item_type, **_kwargs):\n item = self.repository_items[item_type][item_name]\n if not item.guid:\n resp = self.endpoint.invoke(method=\"POST\", url=f\"{self.base_api_url}/items\", body={})\n item.guid = resp[\"body\"][\"id\"]\n self.endpoint.invoke(\n method=\"POST\",\n url=f\"{self.base_api_url}/items/{item.guid}/updateDefinition?updateMetadata=True\",\n body={},\n )\n\n ws = FakeWorkspace()\n env_module.EnvironmentPublisher(ws).publish_all()\n\n urls = [c[1] for c in calls]\n assert any(\"/items\" in u and u.endswith(\"/items\") for u in urls), \"Create item call missing\"\n assert any(\"updateDefinition\" in u for u in urls), \"updateDefinition call missing\"\n assert any(u.endswith(\"/staging/publish?beta=False\") for u in urls), \"Publish submit missing\"\n assert not any(\"sparkcompute\" in u for u in urls), \"Unexpected sparkcompute PATCH call\"\n" }, { "path": "tests/test_fabric_workspace.py", "content": "# Copyright (c) Microsoft Corporation.\n# Licensed under the MIT License.\n\nimport json\nimport re\nimport tempfile\nfrom pathlib import Path\nfrom unittest.mock import MagicMock, patch\n\nimport pytest\nimport yaml\nfrom fixtures.credentials import DummyTokenCredential\n\nfrom fabric_cicd import configure_fabric_fqdn\nfrom fabric_cicd.fabric_workspace import FabricWorkspace, constants\n\n\n@pytest.fixture\ndef mock_endpoint():\n \"\"\"Mock FabricEndpoint to avoid real API calls.\"\"\"\n mock = MagicMock()\n\n def mock_invoke(method, url, body=None, **_kwargs):\n if method == \"POST\" and url.endswith(\"/items\"):\n return {\n \"body\": {\n \"id\": \"mock-item-id-12345\",\n \"workspaceId\": \"mock-workspace-id\",\n \"displayName\": body.get(\"displayName\", \"Test Item\") if body else \"Test Item\",\n \"type\": body.get(\"type\", \"Unknown\") if body else \"Unknown\",\n }\n }\n if method == \"POST\" and \"updateDefinition\" in url:\n return {\"body\": {\"message\": \"Definition updated successfully\"}}\n if method == \"PATCH\" and \"items/\" in url:\n return {\"body\": {\"message\": \"Item metadata updated successfully\"}}\n return {\"body\": {\"value\": []}}\n\n mock.invoke.side_effect = mock_invoke\n return mock\n\n\n@pytest.fixture\ndef temp_workspace_dir():\n \"\"\"Create a temporary directory structure for testing.\"\"\"\n with tempfile.TemporaryDirectory() as temp_dir:\n yield Path(temp_dir)\n\n\n@pytest.fixture\ndef valid_workspace_id():\n \"\"\"Return a valid workspace ID in GUID format.\"\"\"\n return \"12345678-1234-5678-abcd-1234567890ab\"\n\n\n@pytest.fixture\ndef utf8_test_chars():\n \"\"\"Provide sample UTF-8 characters for testing.\"\"\"\n return {\"nordic\": \"Ö æ ø\", \"european\": \"ñ é ü ß ç\", \"asian\": \"你好\", \"mixed\": \"ñ é ü ß ç 你好\"}\n\n\ndef create_parameter_file(dir_path, utf8_chars):\n \"\"\"Create a parameter file with UTF-8 characters.\"\"\"\n parameter_file_path = dir_path / \"parameter.yml\"\n parameter_content = {\n \"find_replace\": [\n {\n \"find_value\": f\"Production {utf8_chars['mixed']}\",\n \"replace_value\": {\n utf8_chars[\"nordic\"]: \"12345678-1234-5678-abcd-1234567890ab\",\n utf8_chars[\"asian\"]: \"21345678-1234-5678-abcd-1234567890ab\",\n },\n }\n ]\n }\n\n with parameter_file_path.open(\"w\", encoding=\"utf-8\") as f:\n yaml.dump(parameter_content, f, allow_unicode=True)\n\n return parameter_content\n\n\ndef create_platform_metadata(dir_path, utf8_chars):\n \"\"\"Create a .platform metadata file with UTF-8 characters.\"\"\"\n item_dir = dir_path / \"test_item\"\n item_dir.mkdir(parents=True, exist_ok=True)\n platform_file_path = item_dir / \".platform\"\n metadata_content = {\n \"metadata\": {\n \"type\": \"Notebook\",\n \"displayName\": f\"Test Notebook with {utf8_chars['nordic']}\",\n \"description\": f\"Description with {utf8_chars['mixed']}\",\n },\n \"config\": {\"logicalId\": \"test-logical-id\"},\n }\n\n with platform_file_path.open(\"w\", encoding=\"utf-8\") as f:\n json.dump(metadata_content, f, ensure_ascii=False)\n\n with (item_dir / \"dummy.txt\").open(\"w\", encoding=\"utf-8\") as f:\n f.write(\"Dummy file\")\n\n return metadata_content\n\n\n@pytest.fixture\ndef patched_fabric_workspace(mock_endpoint):\n \"\"\"Return a factory function to create a patched FabricWorkspace.\"\"\"\n\n def _create_workspace(workspace_id, repository_directory, item_type_in_scope=None, **kwargs):\n fabric_endpoint_patch = patch(\"fabric_cicd.fabric_workspace.FabricEndpoint\", return_value=mock_endpoint)\n refresh_items_patch = patch.object(\n FabricWorkspace, \"_refresh_deployed_items\", new=lambda self: setattr(self, \"deployed_items\", {})\n )\n refresh_folders_patch = patch.object(\n FabricWorkspace, \"_refresh_deployed_folders\", new=lambda self: setattr(self, \"deployed_folders\", {})\n )\n\n with fabric_endpoint_patch, refresh_items_patch, refresh_folders_patch:\n workspace = FabricWorkspace(\n workspace_id=workspace_id,\n repository_directory=repository_directory,\n item_type_in_scope=item_type_in_scope,\n token_credential=DummyTokenCredential(),\n **kwargs,\n )\n # Call refresh methods to populate workspace data\n workspace._refresh_deployed_folders()\n workspace._refresh_repository_folders()\n workspace._refresh_deployed_items()\n workspace._refresh_repository_items()\n\n return workspace\n\n return _create_workspace\n\n\ndef test_parameter_file_with_utf8_chars(\n temp_workspace_dir, patched_fabric_workspace, valid_workspace_id, utf8_test_chars\n):\n \"\"\"Test that parameter file with UTF-8 characters is read correctly.\"\"\"\n create_parameter_file(temp_workspace_dir, utf8_test_chars)\n with patch.object(FabricWorkspace, \"_refresh_repository_items\"):\n workspace = patched_fabric_workspace(\n workspace_id=valid_workspace_id,\n repository_directory=str(temp_workspace_dir),\n item_type_in_scope=[\"Environment\"],\n )\n\n key1 = f\"Production {utf8_test_chars['mixed']}\"\n key2 = utf8_test_chars[\"nordic\"]\n key3 = utf8_test_chars[\"asian\"]\n\n for param_dict in workspace.environment_parameter.get(\"find_replace\"):\n assert key1 == param_dict[\"find_value\"]\n assert key2 in param_dict[\"replace_value\"]\n assert key3 in param_dict[\"replace_value\"]\n\n\ndef test_platform_metadata_with_utf8_chars(\n temp_workspace_dir, patched_fabric_workspace, valid_workspace_id, utf8_test_chars\n):\n \"\"\"Test that .platform metadata file with UTF-8 characters is read correctly.\"\"\"\n create_platform_metadata(temp_workspace_dir, utf8_test_chars)\n with patch.object(FabricWorkspace, \"_refresh_parameter_file\"):\n workspace = patched_fabric_workspace(\n workspace_id=valid_workspace_id,\n repository_directory=str(temp_workspace_dir),\n item_type_in_scope=[\"Notebook\"],\n )\n\n item_name = f\"Test Notebook with {utf8_test_chars['nordic']}\"\n item = workspace.repository_items[\"Notebook\"][item_name]\n\n assert \"Notebook\" in workspace.repository_items\n assert item_name in workspace.repository_items[\"Notebook\"]\n assert item.name == item_name\n assert item.description == f\"Description with {utf8_test_chars['mixed']}\"\n\n\ndef test_environment_param_with_utf8_chars(\n temp_workspace_dir, patched_fabric_workspace, valid_workspace_id, utf8_test_chars\n):\n \"\"\"Test that environment parameter with UTF-8 characters is preserved.\"\"\"\n with patch.object(FabricWorkspace, \"_refresh_repository_items\"):\n workspace = patched_fabric_workspace(\n workspace_id=valid_workspace_id,\n repository_directory=str(temp_workspace_dir),\n item_type_in_scope=[\"Environment\"],\n environment=utf8_test_chars[\"nordic\"],\n )\n\n assert workspace.environment == utf8_test_chars[\"nordic\"]\n\n\ndef test_workspace_id_replacement_in_json(patched_fabric_workspace, valid_workspace_id, temp_workspace_dir):\n \"\"\"Test that workspace IDs are properly replaced in JSON files (like pipeline-content.json).\"\"\"\n # JSON content with workspace ID that should be replaced\n json_content = \"\"\"{\n \"properties\": {\n \"activities\": [\n {\n \"type\": \"TridentNotebook\",\n \"typeProperties\": {\n \"notebookId\": \"99b570c5-0c79-9dc4-4c9b-fa16c621384c\",\n \"workspaceId\": \"00000000-0000-0000-0000-000000000000\"\n }\n }\n ]\n }\n}\"\"\"\n\n with patch.object(FabricWorkspace, \"_refresh_repository_items\"):\n workspace = patched_fabric_workspace(\n workspace_id=valid_workspace_id,\n repository_directory=str(temp_workspace_dir),\n item_type_in_scope=[\"DataPipeline\"],\n )\n\n # Test the workspace ID replacement function\n result = workspace._replace_workspace_ids(json_content)\n\n # Verify that the default workspace ID was replaced with the target workspace ID\n assert \"00000000-0000-0000-0000-000000000000\" not in result\n assert valid_workspace_id in result\n assert '\"workspaceId\": \"' + valid_workspace_id + '\"' in result\n\n\ndef test_workspace_id_replacement_in_python(patched_fabric_workspace, valid_workspace_id, temp_workspace_dir):\n \"\"\"Test that workspace IDs are properly replaced in Python files (like notebook-content.py).\"\"\"\n # Python content with workspace ID that should be replaced (as in notebook metadata)\n python_content = \"\"\"# META {\n# META \"dependencies\": {\n# META \"environment\": {\n# META \"environmentId\": \"a277ea4a-e87f-8537-4ce0-39db11d4aade\",\n# META \"workspaceId\": \"00000000-0000-0000-0000-000000000000\"\n# META }\n# META }\n# META }\"\"\"\n\n with patch.object(FabricWorkspace, \"_refresh_repository_items\"):\n workspace = patched_fabric_workspace(\n workspace_id=valid_workspace_id,\n repository_directory=str(temp_workspace_dir),\n item_type_in_scope=[\"Notebook\"],\n )\n\n # Test the workspace ID replacement function\n result = workspace._replace_workspace_ids(python_content)\n\n # Verify that the default workspace ID was replaced with the target workspace ID\n assert \"00000000-0000-0000-0000-000000000000\" not in result\n assert valid_workspace_id in result\n assert 'workspaceId\": \"' + valid_workspace_id + '\"' in result\n\n\ndef test_workspace_id_replacement_eventstream_json(patched_fabric_workspace, valid_workspace_id, temp_workspace_dir):\n \"\"\"Test workspace ID replacement in Eventstream JSON files with multiple occurrences.\"\"\"\n eventstream_content = \"\"\"{\n \"destinations\": [\n {\n \"name\": \"DataActivator\",\n \"type\": \"Activator\",\n \"properties\": {\n \"workspaceId\": \"00000000-0000-0000-0000-000000000000\",\n \"itemId\": \"c3bf82de-14b6-af39-4852-dda67eccd7c0\"\n }\n },\n {\n \"name\": \"Lakehouse\",\n \"type\": \"Lakehouse\",\n \"properties\": {\n \"workspaceId\": \"00000000-0000-0000-0000-000000000000\",\n \"itemId\": \"c916eeb0-dd6a-ae32-4f4f-966d2414b239\"\n }\n },\n {\n \"name\": \"Eventhouse\",\n \"type\": \"Eventhouse\",\n \"properties\": {\n \"workspaceId\": \"00000000-0000-0000-0000-000000000000\",\n \"itemId\": \"a51e98dd-5993-8e1c-443f-02aa53d4db74\"\n }\n }\n ]\n}\"\"\"\n\n with patch.object(FabricWorkspace, \"_refresh_repository_items\"):\n workspace = patched_fabric_workspace(\n workspace_id=valid_workspace_id,\n repository_directory=str(temp_workspace_dir),\n item_type_in_scope=[\"Eventstream\"],\n )\n\n result = workspace._replace_workspace_ids(eventstream_content)\n\n # Verify all three workspace IDs were replaced\n assert \"00000000-0000-0000-0000-000000000000\" not in result\n assert result.count(f'\"workspaceId\": \"{valid_workspace_id}\"') == 3\n\n\ndef test_workspace_id_replacement_yaml_format(patched_fabric_workspace, valid_workspace_id, temp_workspace_dir):\n \"\"\"Test workspace ID replacement in YAML-style formats.\"\"\"\n yaml_content = \"\"\"\nconfiguration:\n lakehouse:\n default_lakehouse_workspace_id: \"00000000-0000-0000-0000-000000000000\"\n environment:\n workspaceId = \"00000000-0000-0000-0000-000000000000\"\n other:\n workspace: \"00000000-0000-0000-0000-000000000000\"\n\"\"\"\n\n with patch.object(FabricWorkspace, \"_refresh_repository_items\"):\n workspace = patched_fabric_workspace(\n workspace_id=valid_workspace_id,\n repository_directory=str(temp_workspace_dir),\n item_type_in_scope=[\"Environment\"],\n )\n\n result = workspace._replace_workspace_ids(yaml_content)\n\n # Verify all different property name formats are replaced\n assert \"00000000-0000-0000-0000-000000000000\" not in result\n assert f'default_lakehouse_workspace_id: \"{valid_workspace_id}\"' in result\n assert f'workspaceId = \"{valid_workspace_id}\"' in result\n assert f'workspace: \"{valid_workspace_id}\"' in result\n\n\ndef test_workspace_id_replacement_mixed_formats(patched_fabric_workspace, valid_workspace_id, temp_workspace_dir):\n \"\"\"Test workspace ID replacement with mixed JSON and YAML formats in same content.\"\"\"\n mixed_content = \"\"\"{\n \"pipeline\": {\n \"properties\": {\n \"workspaceId\": \"00000000-0000-0000-0000-000000000000\"\n }\n },\n \"configuration\": {\n \"default_lakehouse_workspace_id\": \"00000000-0000-0000-0000-000000000000\",\n \"workspace\" = \"00000000-0000-0000-0000-000000000000\"\n }\n}\"\"\"\n\n with patch.object(FabricWorkspace, \"_refresh_repository_items\"):\n workspace = patched_fabric_workspace(\n workspace_id=valid_workspace_id,\n repository_directory=str(temp_workspace_dir),\n item_type_in_scope=[\"DataPipeline\"],\n )\n\n result = workspace._replace_workspace_ids(mixed_content)\n\n # Verify all formats are replaced correctly\n assert \"00000000-0000-0000-0000-000000000000\" not in result\n assert f'\"workspaceId\": \"{valid_workspace_id}\"' in result\n assert f'\"default_lakehouse_workspace_id\": \"{valid_workspace_id}\"' in result\n assert f'\"workspace\" = \"{valid_workspace_id}\"' in result\n\n\ndef test_workspace_id_replacement_whitespace_variations(\n patched_fabric_workspace, valid_workspace_id, temp_workspace_dir\n):\n \"\"\"Test workspace ID replacement with various whitespace patterns.\"\"\"\n whitespace_content = \"\"\"\n{\n \"test1\": {\n \"workspaceId\":\"00000000-0000-0000-0000-000000000000\"\n },\n \"test2\": {\n \"workspaceId\" : \"00000000-0000-0000-0000-000000000000\"\n },\n \"test3\": {\n workspaceId = \"00000000-0000-0000-0000-000000000000\"\n },\n \"test4\": {\n \"workspace\" : \"00000000-0000-0000-0000-000000000000\"\n }\n}\n\"\"\"\n\n with patch.object(FabricWorkspace, \"_refresh_repository_items\"):\n workspace = patched_fabric_workspace(\n workspace_id=valid_workspace_id,\n repository_directory=str(temp_workspace_dir),\n item_type_in_scope=[\"DataPipeline\"],\n )\n\n result = workspace._replace_workspace_ids(whitespace_content)\n\n # Verify all whitespace variations are handled\n assert \"00000000-0000-0000-0000-000000000000\" not in result\n assert result.count(valid_workspace_id) == 4\n\n\ndef test_workspace_id_replacement_non_default_values_preserved(\n patched_fabric_workspace, valid_workspace_id, temp_workspace_dir\n):\n \"\"\"Test that non-default workspace IDs are NOT replaced (regression test).\"\"\"\n # Use a different workspace ID that should not be replaced\n other_workspace_id = \"12345678-1234-1234-1234-123456789012\"\n content_with_other_id = f'''{{\n \"properties\": {{\n \"activities\": [\n {{\n \"type\": \"TridentNotebook\",\n \"typeProperties\": {{\n \"workspaceId\": \"{other_workspace_id}\",\n \"notebookId\": \"99b570c5-0c79-9dc4-4c9b-fa16c621384c\"\n }}\n }},\n {{\n \"type\": \"TridentNotebook\",\n \"typeProperties\": {{\n \"workspaceId\": \"00000000-0000-0000-0000-000000000000\",\n \"notebookId\": \"88a570c5-0c79-9dc4-4c9b-fa16c621384c\"\n }}\n }}\n ]\n }}\n}}'''\n\n with patch.object(FabricWorkspace, \"_refresh_repository_items\"):\n workspace = patched_fabric_workspace(\n workspace_id=valid_workspace_id,\n repository_directory=str(temp_workspace_dir),\n item_type_in_scope=[\"DataPipeline\"],\n )\n\n result = workspace._replace_workspace_ids(content_with_other_id)\n\n # Verify only default workspace ID was replaced, other ID preserved\n assert \"00000000-0000-0000-0000-000000000000\" not in result\n assert other_workspace_id in result # This should be preserved\n assert result.count(valid_workspace_id) == 1 # Only one replacement\n assert result.count(other_workspace_id) == 1 # Original preserved\n\n\ndef test_workspace_id_replacement_edge_cases(patched_fabric_workspace, valid_workspace_id, temp_workspace_dir):\n \"\"\"Test workspace ID replacement edge cases and current regex behavior.\"\"\"\n edge_cases_content = \"\"\"\n// Comment with workspaceId: \"00000000-0000-0000-0000-000000000000\" - this gets replaced due to current regex\n{\n \"validCase1\": {\n \"workspaceId\": \"00000000-0000-0000-0000-000000000000\"\n },\n \"validCase2\": {\n \"default_lakehouse_workspace_id\": \"00000000-0000-0000-0000-000000000000\"\n },\n \"invalidCase1\": {\n \"workspaceIdNot\": \"00000000-0000-0000-0000-000000000000\"\n },\n \"invalidCase2\": {\n \"notworkspaceId\": \"00000000-0000-0000-0000-000000000000\"\n },\n \"validCase3\": {\n workspace: \"00000000-0000-0000-0000-000000000000\"\n }\n}\n\"\"\"\n\n with patch.object(FabricWorkspace, \"_refresh_repository_items\"):\n workspace = patched_fabric_workspace(\n workspace_id=valid_workspace_id,\n repository_directory=str(temp_workspace_dir),\n item_type_in_scope=[\"DataPipeline\"],\n )\n\n result = workspace._replace_workspace_ids(edge_cases_content)\n\n # Current regex behavior: matches comments and partial matches like \"notworkspaceId\"\n # This documents the current behavior for regression testing\n assert result.count(valid_workspace_id) == 5 # comment, validCase1, validCase2, invalidCase2, validCase3\n assert '\"workspaceIdNot\": \"00000000-0000-0000-0000-000000000000\"' in result # Should not be replaced (prefix case)\n assert f'\"notworkspaceId\": \"{valid_workspace_id}\"' in result # Gets replaced (suffix matches workspaceId)\n assert f'// Comment with workspaceId: \"{valid_workspace_id}\"' in result # Comment gets replaced\n\n\ndef test_workspace_id_replacement_comprehensive_item_types(\n patched_fabric_workspace, valid_workspace_id, temp_workspace_dir\n):\n \"\"\"Test workspace ID replacement across different item type contexts.\"\"\"\n # Test content that might appear in different item types\n comprehensive_content = \"\"\"\n{\n \"notebook\": {\n \"metadata\": {\n \"environment\": {\n \"workspaceId\": \"00000000-0000-0000-0000-000000000000\"\n }\n }\n },\n \"pipeline\": {\n \"activities\": [{\n \"typeProperties\": {\n \"workspaceId\": \"00000000-0000-0000-0000-000000000000\"\n }\n }]\n },\n \"eventstream\": {\n \"destinations\": [{\n \"properties\": {\n \"workspaceId\": \"00000000-0000-0000-0000-000000000000\"\n }\n }]\n },\n \"lakehouse\": {\n \"default_lakehouse_workspace_id\": \"00000000-0000-0000-0000-000000000000\"\n },\n \"environment\": {\n \"workspace\": \"00000000-0000-0000-0000-000000000000\"\n }\n}\n\"\"\"\n\n # Test with different item types to ensure the replacement works regardless of item type context\n item_types_to_test = [\"Notebook\", \"DataPipeline\", \"Eventstream\", \"Lakehouse\", \"Environment\"]\n\n for item_type in item_types_to_test:\n with patch.object(FabricWorkspace, \"_refresh_repository_items\"):\n workspace = patched_fabric_workspace(\n workspace_id=valid_workspace_id,\n repository_directory=str(temp_workspace_dir),\n item_type_in_scope=[item_type],\n )\n\n result = workspace._replace_workspace_ids(comprehensive_content)\n\n # Verify all workspace IDs are replaced regardless of item type context\n assert \"00000000-0000-0000-0000-000000000000\" not in result, f\"Failed for item type: {item_type}\"\n assert result.count(valid_workspace_id) == 5, f\"Incorrect replacement count for item type: {item_type}\"\n\n\ndef test_environment_parameter_replacement_issue(patched_fabric_workspace, temp_workspace_dir, valid_workspace_id):\n \"\"\"Test that parameter replacement works correctly with different environment values.\n\n This test ensures that the issue where parameter replacement doesn't work when\n environment defaults to 'N/A' is properly handled.\n \"\"\"\n # Create parameter.yml file with environment-specific replacements\n parameter_content = \"\"\"\nfind_replace:\n - find_value: \"test-guid-to-replace\"\n replace_value:\n PPE: \"ppe-replacement-value\"\n PROD: \"prod-replacement-value\"\n item_type: \"Notebook\"\n item_name: [\"Test Notebook\"]\n\"\"\"\n\n # Create notebook structure\n notebook_dir = temp_workspace_dir / \"Test Notebook.Notebook\"\n notebook_dir.mkdir(parents=True)\n\n notebook_content = 'test_value = \"test-guid-to-replace\"'\n\n # Write files\n (temp_workspace_dir / \"parameter.yml\").write_text(parameter_content)\n (notebook_dir / \"notebook-content.py\").write_text(notebook_content)\n\n from fabric_cicd._common._file import File\n from fabric_cicd._common._item import Item\n\n # Test 1: Without environment parameter (defaults to 'N/A')\n with patch.object(FabricWorkspace, \"_refresh_repository_items\"):\n workspace_no_env = patched_fabric_workspace(\n workspace_id=valid_workspace_id,\n repository_directory=str(temp_workspace_dir),\n item_type_in_scope=[\"Notebook\"],\n )\n\n # Test 2: With environment parameter (PPE)\n with patch.object(FabricWorkspace, \"_refresh_repository_items\"):\n workspace_with_env = patched_fabric_workspace(\n workspace_id=valid_workspace_id,\n repository_directory=str(temp_workspace_dir),\n item_type_in_scope=[\"Notebook\"],\n environment=\"PPE\",\n )\n\n # Create test objects for parameter replacement\n test_item = Item(type=\"Notebook\", name=\"Test Notebook\", description=\"\", guid=\"test-guid\", path=notebook_dir)\n test_file = File(item_path=notebook_dir, file_path=notebook_dir / \"notebook-content.py\")\n\n # Test parameter replacement with default environment\n replaced_content_no_env = workspace_no_env._replace_parameters(test_file, test_item)\n\n # Test parameter replacement with specific environment\n replaced_content_with_env = workspace_with_env._replace_parameters(test_file, test_item)\n\n # Assertions\n # With default environment ('N/A'), replacement should NOT occur\n assert \"test-guid-to-replace\" in replaced_content_no_env, \"Original value should remain when environment is N/A\"\n assert \"ppe-replacement-value\" not in replaced_content_no_env, (\n \"Replacement should not occur with default environment\"\n )\n\n # With specific environment (PPE), replacement SHOULD occur\n assert \"test-guid-to-replace\" not in replaced_content_with_env, (\n \"Original value should be replaced when environment matches\"\n )\n assert \"ppe-replacement-value\" in replaced_content_with_env, \"Replacement should occur with matching environment\"\n\n\ndef test_empty_logical_id_validation(temp_workspace_dir, patched_fabric_workspace, valid_workspace_id):\n \"\"\"Test that empty logical IDs raise a ParsingError during repository refresh.\"\"\"\n from fabric_cicd._common._exceptions import ParsingError\n\n # Create a .platform file with empty logical ID\n item_dir = temp_workspace_dir / \"TestItem.Notebook\"\n item_dir.mkdir(parents=True, exist_ok=True)\n platform_file_path = item_dir / \".platform\"\n\n metadata_content = {\n \"metadata\": {\n \"type\": \"Notebook\",\n \"displayName\": \"Test Item with Empty Logical ID\",\n \"description\": \"Test item for empty logical ID validation\",\n },\n \"config\": {\"logicalId\": \"\"}, # Empty logical ID\n }\n\n with platform_file_path.open(\"w\", encoding=\"utf-8\") as f:\n json.dump(metadata_content, f, ensure_ascii=False)\n\n # Create a dummy content file\n with (item_dir / \"dummy.txt\").open(\"w\", encoding=\"utf-8\") as f:\n f.write(\"Dummy file content\")\n\n # Test that ParsingError is raised when trying to refresh repository items\n with pytest.raises(ParsingError) as exc_info:\n patched_fabric_workspace(\n workspace_id=valid_workspace_id,\n repository_directory=str(temp_workspace_dir),\n item_type_in_scope=[\"Notebook\"],\n )\n\n # Verify the error message contains the expected information\n assert \"logicalId cannot be empty\" in str(exc_info.value)\n assert str(platform_file_path) in str(exc_info.value)\n\n\ndef test_whitespace_only_logical_id_validation(temp_workspace_dir, patched_fabric_workspace, valid_workspace_id):\n \"\"\"Test that logical IDs with only whitespace raise a ParsingError.\"\"\"\n from fabric_cicd._common._exceptions import ParsingError\n\n # Create a .platform file with whitespace-only logical ID\n item_dir = temp_workspace_dir / \"TestItem.Notebook\"\n item_dir.mkdir(parents=True, exist_ok=True)\n platform_file_path = item_dir / \".platform\"\n\n metadata_content = {\n \"metadata\": {\n \"type\": \"Notebook\",\n \"displayName\": \"Test Item with Whitespace Logical ID\",\n \"description\": \"Test item for whitespace logical ID validation\",\n },\n \"config\": {\"logicalId\": \" \"}, # Whitespace-only logical ID\n }\n\n with platform_file_path.open(\"w\", encoding=\"utf-8\") as f:\n json.dump(metadata_content, f, ensure_ascii=False)\n\n # Create a dummy content file\n with (item_dir / \"dummy.txt\").open(\"w\", encoding=\"utf-8\") as f:\n f.write(\"Dummy file content\")\n\n # Test that ParsingError is raised when trying to refresh repository items\n with pytest.raises(ParsingError) as exc_info:\n patched_fabric_workspace(\n workspace_id=valid_workspace_id,\n repository_directory=str(temp_workspace_dir),\n item_type_in_scope=[\"Notebook\"],\n )\n\n # Verify the error message\n assert \"logicalId cannot be empty\" in str(exc_info.value)\n\n\ndef test_valid_logical_id_works_correctly(temp_workspace_dir, patched_fabric_workspace, valid_workspace_id):\n \"\"\"Test that valid logical IDs continue to work correctly after adding validation.\"\"\"\n # Create a .platform file with valid logical ID\n item_dir = temp_workspace_dir / \"TestItem.Notebook\"\n item_dir.mkdir(parents=True, exist_ok=True)\n platform_file_path = item_dir / \".platform\"\n\n metadata_content = {\n \"metadata\": {\n \"type\": \"Notebook\",\n \"displayName\": \"Test Item with Valid Logical ID\",\n \"description\": \"Test item for valid logical ID verification\",\n },\n \"config\": {\"logicalId\": \"valid-logical-id-123\"}, # Valid logical ID\n }\n\n with platform_file_path.open(\"w\", encoding=\"utf-8\") as f:\n json.dump(metadata_content, f, ensure_ascii=False)\n\n # Create a dummy content file\n with (item_dir / \"dummy.txt\").open(\"w\", encoding=\"utf-8\") as f:\n f.write(\"Dummy file content\")\n\n # This should work without raising any exception\n workspace = patched_fabric_workspace(\n workspace_id=valid_workspace_id, repository_directory=str(temp_workspace_dir), item_type_in_scope=[\"Notebook\"]\n )\n\n # Verify the item was loaded correctly (validation happens automatically during refresh)\n assert \"Notebook\" in workspace.repository_items\n assert \"Test Item with Valid Logical ID\" in workspace.repository_items[\"Notebook\"]\n assert (\n workspace.repository_items[\"Notebook\"][\"Test Item with Valid Logical ID\"].logical_id == \"valid-logical-id-123\"\n )\n\n\ndef test_empty_logical_id_validation_during_publish(temp_workspace_dir, patched_fabric_workspace, valid_workspace_id):\n \"\"\"Test that empty logical IDs are caught during workspace initialization.\"\"\"\n from fabric_cicd._common._exceptions import ParsingError\n\n # Create a .platform file with empty logical ID\n item_dir = temp_workspace_dir / \"TestItem.Notebook\"\n item_dir.mkdir(parents=True, exist_ok=True)\n platform_file_path = item_dir / \".platform\"\n\n metadata_content = {\n \"metadata\": {\n \"type\": \"Notebook\",\n \"displayName\": \"Test Item with Empty Logical ID\",\n \"description\": \"Test item for empty logical ID validation during publish\",\n },\n \"config\": {\"logicalId\": \"\"}, # Empty logical ID\n }\n\n with platform_file_path.open(\"w\", encoding=\"utf-8\") as f:\n json.dump(metadata_content, f, ensure_ascii=False)\n\n # Create a dummy content file\n with (item_dir / \"dummy.txt\").open(\"w\", encoding=\"utf-8\") as f:\n f.write(\"Dummy file content\")\n\n # Test that ParsingError is raised during workspace initialization\n with pytest.raises(ParsingError) as exc_info:\n patched_fabric_workspace(\n workspace_id=valid_workspace_id,\n repository_directory=str(temp_workspace_dir),\n item_type_in_scope=[\"Notebook\"],\n )\n\n # Verify the error message contains the expected information\n assert \"logicalId cannot be empty\" in str(exc_info.value)\n assert str(platform_file_path) in str(exc_info.value)\n\n\ndef test_multiple_empty_logical_ids_validation(temp_workspace_dir, patched_fabric_workspace, valid_workspace_id):\n \"\"\"Test that multiple empty logical IDs are all reported at once.\"\"\"\n from fabric_cicd._common._exceptions import ParsingError\n\n # Create multiple .platform files with empty logical IDs\n item_dirs = [\"TestItem1.Notebook\", \"TestItem2.Notebook\", \"TestItem3.Environment\"]\n platform_file_paths = []\n\n for item_dir_name in item_dirs:\n item_dir = temp_workspace_dir / item_dir_name\n item_dir.mkdir(parents=True, exist_ok=True)\n platform_file_path = item_dir / \".platform\"\n platform_file_paths.append(platform_file_path)\n\n item_type = \"Notebook\" if \"Notebook\" in item_dir_name else \"Environment\"\n metadata_content = {\n \"metadata\": {\n \"type\": item_type,\n \"displayName\": f\"Test Item {item_dir_name}\",\n \"description\": \"Test item for multiple empty logical ID validation\",\n },\n \"config\": {\"logicalId\": \"\"}, # Empty logical ID\n }\n\n with platform_file_path.open(\"w\", encoding=\"utf-8\") as f:\n json.dump(metadata_content, f, ensure_ascii=False)\n\n # Create a dummy content file\n with (item_dir / \"dummy.txt\").open(\"w\", encoding=\"utf-8\") as f:\n f.write(\"Dummy file content\")\n\n # Test that ParsingError is raised when trying to refresh repository items\n with pytest.raises(ParsingError) as exc_info:\n patched_fabric_workspace(\n workspace_id=valid_workspace_id,\n repository_directory=str(temp_workspace_dir),\n item_type_in_scope=[\"Notebook\", \"Environment\"],\n )\n\n # Verify the error message contains information about all empty logical IDs\n error_message = str(exc_info.value)\n assert \"logicalId cannot be empty in the following files:\" in error_message\n for platform_file_path in platform_file_paths:\n assert str(platform_file_path) in error_message\n\n\ndef test_single_empty_logical_id_validation_message(temp_workspace_dir, patched_fabric_workspace, valid_workspace_id):\n \"\"\"Test that a single empty logical ID shows the original error format.\"\"\"\n from fabric_cicd._common._exceptions import ParsingError\n\n # Create a .platform file with empty logical ID\n item_dir = temp_workspace_dir / \"TestItem.Notebook\"\n item_dir.mkdir(parents=True, exist_ok=True)\n platform_file_path = item_dir / \".platform\"\n\n metadata_content = {\n \"metadata\": {\n \"type\": \"Notebook\",\n \"displayName\": \"Test Item with Empty Logical ID\",\n \"description\": \"Test item for single empty logical ID validation\",\n },\n \"config\": {\"logicalId\": \"\"}, # Empty logical ID\n }\n\n with platform_file_path.open(\"w\", encoding=\"utf-8\") as f:\n json.dump(metadata_content, f, ensure_ascii=False)\n\n # Create a dummy content file\n with (item_dir / \"dummy.txt\").open(\"w\", encoding=\"utf-8\") as f:\n f.write(\"Dummy file content\")\n\n # Test that ParsingError is raised when trying to refresh repository items\n with pytest.raises(ParsingError) as exc_info:\n patched_fabric_workspace(\n workspace_id=valid_workspace_id,\n repository_directory=str(temp_workspace_dir),\n item_type_in_scope=[\"Notebook\"],\n )\n\n # Verify the error message uses single file format (not \"following files:\")\n error_message = str(exc_info.value)\n assert \"logicalId cannot be empty in \" in error_message\n assert \"following files:\" not in error_message\n assert str(platform_file_path) in error_message\n\n\ndef test_fabric_workspace_with_none_item_types_defaults_to_all(\n temp_workspace_dir, patched_fabric_workspace, valid_workspace_id\n):\n \"\"\"Test that FabricWorkspace works correctly when initialized with None item_type_in_scope (defaults to all available types).\"\"\"\n # Create a sample item to test with\n item_dir = temp_workspace_dir / \"TestNotebook.Notebook\"\n item_dir.mkdir(parents=True, exist_ok=True)\n platform_file_path = item_dir / \".platform\"\n\n metadata_content = {\n \"metadata\": {\n \"type\": \"Notebook\",\n \"displayName\": \"Test Notebook\",\n \"description\": \"Test notebook for None item types test\",\n },\n \"config\": {\"logicalId\": \"test-logical-id-none\"},\n }\n\n with platform_file_path.open(\"w\", encoding=\"utf-8\") as f:\n json.dump(metadata_content, f, ensure_ascii=False)\n\n # Create a dummy content file\n with (item_dir / \"dummy.txt\").open(\"w\", encoding=\"utf-8\") as f:\n f.write(\"Dummy file content\")\n\n # Test that workspace initializes correctly with None (default behavior)\n workspace = patched_fabric_workspace(\n workspace_id=valid_workspace_id,\n repository_directory=str(temp_workspace_dir), # item_type_in_scope=None (default)\n )\n\n # Verify that item_type_in_scope was expanded to all available types\n import fabric_cicd.constants as constants\n\n expected_types = list(constants.ACCEPTED_ITEM_TYPES)\n assert set(workspace.item_type_in_scope) == set(expected_types), (\n f\"Expected all item types, got {workspace.item_type_in_scope}\"\n )\n\n # Verify that the notebook item was loaded correctly\n assert \"Notebook\" in workspace.repository_items\n assert \"Test Notebook\" in workspace.repository_items[\"Notebook\"]\n\n\ndef test_parameter_file_path_types(temp_workspace_dir, patched_fabric_workspace, valid_workspace_id):\n \"\"\"Test different path types for parameter_file_path in FabricWorkspace.\"\"\"\n\n # Absolute path - accepted\n param_file = temp_workspace_dir / \"parameters.yml\"\n param_file.write_text(\"\"\"\nfind_replace:\n - find_value: \"test-value\"\n replace_value:\n DEV: \"dev-replacement\"\n\"\"\")\n\n workspace = patched_fabric_workspace(\n workspace_id=valid_workspace_id,\n repository_directory=str(temp_workspace_dir),\n parameter_file_path=str(param_file),\n )\n\n assert workspace.parameter_file_path == str(param_file)\n\n # Relative path - now resolved against repository directory but file doesn't exist\n # This should not raise an exception now, it's handled gracefully\n workspace = patched_fabric_workspace(\n workspace_id=valid_workspace_id,\n repository_directory=str(temp_workspace_dir),\n parameter_file_path=\"relative/path/parameters.yml\",\n )\n\n # The workspace should be created successfully but with empty parameters\n assert workspace is not None\n assert hasattr(workspace, \"environment_parameter\")\n assert not workspace.environment_parameter\n\n\ndef test_parameter_file_path_none(temp_workspace_dir, patched_fabric_workspace, valid_workspace_id):\n \"\"\"Test None cases for parameter_file_path in FabricWorkspace.\"\"\"\n # Create a workspace with parameter_file_path set to None\n workspace = patched_fabric_workspace(\n workspace_id=valid_workspace_id, repository_directory=str(temp_workspace_dir), parameter_file_path=None\n )\n assert workspace.parameter_file_path is None\n\n # Create a workspace without parameter_file_path provided\n workspace = patched_fabric_workspace(workspace_id=valid_workspace_id, repository_directory=str(temp_workspace_dir))\n assert workspace.parameter_file_path is None\n\n\ndef test_skip_parameterization_prevents_parameter_yml_auto_discovery(\n temp_workspace_dir, patched_fabric_workspace, valid_workspace_id\n):\n \"\"\"When skip_parameterization=True (config 'parameter' field absent), a parameter.yml\n present in the repository directory must NOT be loaded — _refresh_parameter_file must\n not be called at all, and environment_parameter must remain empty.\"\"\"\n # Create a parameter.yml in the repo — it must NOT be picked up\n param_file = temp_workspace_dir / \"parameter.yml\"\n param_file.write_text(\"find_replace:\\n - find_value: 'secret'\\n replace_value:\\n DEV: 'replaced'\\n\")\n assert param_file.exists()\n\n workspace = patched_fabric_workspace(\n workspace_id=valid_workspace_id,\n repository_directory=str(temp_workspace_dir),\n skip_parameterization=True,\n )\n\n # environment_parameter must be empty — parameter.yml was not auto-discovered\n assert workspace.environment_parameter == {}\n\n\ndef test_skip_parameterization_false_loads_explicit_parameter_file(\n temp_workspace_dir, patched_fabric_workspace, valid_workspace_id\n):\n \"\"\"When skip_parameterization=False (default) and an explicit parameter_file_path is\n given, the parameter file IS loaded and parameterization IS applied.\"\"\"\n param_file = temp_workspace_dir / \"my_params.yml\"\n param_file.write_text(\"find_replace:\\n - find_value: 'secret'\\n replace_value:\\n DEV: 'replaced'\\n\")\n\n workspace = patched_fabric_workspace(\n workspace_id=valid_workspace_id,\n repository_directory=str(temp_workspace_dir),\n parameter_file_path=str(param_file),\n environment=\"DEV\",\n skip_parameterization=False,\n )\n\n # Parameterization must be populated — the explicit file was loaded\n assert \"find_replace\" in workspace.environment_parameter\n\n\ndef test_parameter_file_path_with_environment(temp_workspace_dir, patched_fabric_workspace, valid_workspace_id):\n \"\"\"Test that FabricWorkspace works with both parameter_file_path and environment.\"\"\"\n param_file = temp_workspace_dir / \"dev_parameters.yml\"\n param_file.write_text(\"\"\"\nfind_replace:\n - find_value: \"test-value\"\n replace_value:\n DEV: \"dev-replacement\"\n\"\"\")\n\n workspace = patched_fabric_workspace(\n workspace_id=valid_workspace_id,\n repository_directory=str(temp_workspace_dir),\n parameter_file_path=str(param_file),\n environment=\"DEV\",\n )\n\n assert workspace.parameter_file_path == str(param_file)\n assert workspace.environment == \"DEV\"\n\n\ndef test_parameter_file_path_backward_compatibility(temp_workspace_dir, patched_fabric_workspace, valid_workspace_id):\n \"\"\"Test that existing code without parameter_file_path continues to work.\"\"\"\n workspace = patched_fabric_workspace(\n workspace_id=valid_workspace_id,\n repository_directory=str(temp_workspace_dir),\n item_type_in_scope=[\"Notebook\"],\n environment=\"PROD\",\n )\n\n # Should work as before without parameter_file_path\n assert workspace.parameter_file_path is None\n assert workspace.environment == \"PROD\"\n assert workspace.item_type_in_scope == [\"Notebook\"]\n\n\ndef test_parameter_file_path_integration_with_parameter_class(\n temp_workspace_dir, patched_fabric_workspace, valid_workspace_id\n):\n \"\"\"Test that parameter_file_path integrates correctly with Parameter class.\"\"\"\n param_file = temp_workspace_dir / \"test_parameters.yml\"\n param_content = \"\"\"\nfind_replace:\n - find_value: \"test-value\"\n replace_value:\n DEV: \"dev-replacement\"\n PROD: \"prod-replacement\"\n\"\"\"\n param_file.write_text(param_content)\n\n workspace = patched_fabric_workspace(\n workspace_id=valid_workspace_id,\n repository_directory=str(temp_workspace_dir),\n parameter_file_path=str(param_file),\n environment=\"DEV\",\n )\n\n # The workspace should use the parameter file path\n assert workspace.parameter_file_path == str(param_file)\n\n # The parameter data should be loaded correctly\n # (Note: This is testing the integration, actual Parameter behavior tested separately)\n assert hasattr(workspace, \"environment_parameter\")\n assert \"find_replace\" in workspace.environment_parameter\n\n\ndef test_parameter_file_path_invalid_type_rejected(temp_workspace_dir, patched_fabric_workspace, valid_workspace_id):\n \"\"\"Test that FabricWorkspace handles invalid types for parameter_file_path.\"\"\"\n # This should not raise an exception now since Parameter handles the error internally\n workspace = patched_fabric_workspace(\n workspace_id=valid_workspace_id,\n repository_directory=str(temp_workspace_dir),\n parameter_file_path=123, # Invalid type\n )\n\n # The workspace should be created, but parameter loading should fail silently\n assert workspace is not None\n assert hasattr(workspace, \"environment_parameter\")\n # Environment parameter should be empty since the parameter file path was invalid\n assert not workspace.environment_parameter\n\n\ndef test_no_token_credential_raises_error(temp_workspace_dir, valid_workspace_id):\n \"\"\"Test that constructing FabricWorkspace without token_credential raises TypeError.\"\"\"\n\n # Create a simple platform file so directory validation passes\n notebook_dir = temp_workspace_dir / \"Test Notebook\"\n notebook_dir.mkdir()\n platform_file = notebook_dir / \".platform\"\n platform_content = {\n \"metadata\": {\"type\": \"Notebook\", \"displayName\": \"Test Notebook\"},\n \"config\": {\"logicalId\": \"12345678-1234-5678-abcd-1234567890ab\"},\n }\n with platform_file.open(\"w\", encoding=\"utf-8\") as f:\n json.dump(platform_content, f)\n\n with pytest.raises(TypeError) as exc_info:\n FabricWorkspace(\n workspace_id=valid_workspace_id,\n repository_directory=str(temp_workspace_dir),\n )\n\n assert \"token_credential\" in str(exc_info.value)\n\n\ndef test_base_api_url_kwarg_raises_error(temp_workspace_dir, valid_workspace_id):\n \"\"\"Test that passing base_api_url as kwarg raises an error.\"\"\"\n from fabric_cicd._common._exceptions import InputError\n\n # Create a simple platform file\n notebook_dir = temp_workspace_dir / \"Test Notebook\"\n notebook_dir.mkdir()\n platform_file = notebook_dir / \".platform\"\n platform_content = {\n \"$schema\": \"https://developer.microsoft.com/json-schemas/fabric/gitIntegration/platformProperties/2.0.0/schema.json\",\n \"metadata\": {\"type\": \"Notebook\", \"displayName\": \"Test Notebook\"},\n \"config\": {\"version\": \"2.0\", \"logicalId\": \"12345678-1234-5678-abcd-1234567890ab\"},\n }\n\n with platform_file.open(\"w\", encoding=\"utf-8\") as f:\n json.dump(platform_content, f)\n\n # Test that base_api_url kwarg raises InputError\n with patch(\"fabric_cicd.fabric_workspace.FabricEndpoint\"):\n with pytest.raises(InputError) as exc_info:\n FabricWorkspace(\n workspace_id=valid_workspace_id,\n repository_directory=str(temp_workspace_dir),\n base_api_url=\"https://custom.api.url\",\n token_credential=DummyTokenCredential(),\n )\n\n # Verify the error message contains the expected text\n assert \"base_api_url is no longer supported\" in str(exc_info.value)\n assert \"constants.DEFAULT_API_ROOT_URL\" in str(exc_info.value)\n\n\ndef test_resolve_workspace_name(patched_fabric_workspace, valid_workspace_id, temp_workspace_dir):\n \"\"\"Tests _resolve_workspace_name resolves display name from workspace ID.\"\"\"\n mock_endpoint = MagicMock()\n mock_endpoint.invoke.return_value = {\n \"body\": {\n \"id\": \"mock-workspace-id\",\n \"displayName\": \"My Workspace [DEV]\",\n }\n }\n\n with patch(\"fabric_cicd.fabric_workspace.FabricEndpoint\", return_value=mock_endpoint):\n workspace = patched_fabric_workspace(\n workspace_id=valid_workspace_id,\n repository_directory=str(temp_workspace_dir),\n )\n\n workspace.endpoint = mock_endpoint\n result = workspace._resolve_workspace_name()\n assert result == \"My Workspace [DEV]\"\n\n\ndef test_resolve_workspace_name_not_found(patched_fabric_workspace, valid_workspace_id, temp_workspace_dir):\n \"\"\"Tests _resolve_workspace_name raises InputError when displayName not in response.\"\"\"\n from fabric_cicd._common._exceptions import InputError\n\n mock_endpoint = MagicMock()\n mock_endpoint.invoke.return_value = {\"body\": {}}\n\n with patch(\"fabric_cicd.fabric_workspace.FabricEndpoint\", return_value=mock_endpoint):\n workspace = patched_fabric_workspace(\n workspace_id=valid_workspace_id,\n repository_directory=str(temp_workspace_dir),\n )\n\n workspace.endpoint = mock_endpoint\n with pytest.raises(InputError, match=\"Workspace name could not be resolved from workspace ID\"):\n workspace._resolve_workspace_name()\n\n\ndef test_lookup_item_attribute(patched_fabric_workspace, valid_workspace_id, temp_workspace_dir):\n \"\"\"Test that _lookup_item_attribute correctly finds items in another workspace.\"\"\"\n # Mock endpoint response for workspace items\n mock_endpoint = MagicMock()\n\n # Ensure the mock response exactly matches what's expected\n mock_response = {\n \"body\": {\n \"value\": [\n {\"id\": \"item-id-1234\", \"type\": \"Notebook\", \"displayName\": \"Test Notebook\"},\n {\"id\": \"item-id-5678\", \"type\": \"DataPipeline\", \"displayName\": \"Test Pipeline\"},\n ]\n }\n }\n mock_endpoint.invoke.return_value = mock_response\n\n # Create a workspace with our mocked endpoint\n with patch(\"fabric_cicd.fabric_workspace.FabricEndpoint\", return_value=mock_endpoint):\n workspace = patched_fabric_workspace(\n workspace_id=valid_workspace_id,\n repository_directory=str(temp_workspace_dir),\n item_type_in_scope=[\"Notebook\", \"DataPipeline\"],\n )\n\n # Replace the endpoint attribute to ensure our mock is being used\n workspace.endpoint = mock_endpoint\n\n # Test finding an existing item\n item_id = workspace._lookup_item_attribute(\"target-workspace-id\", \"Notebook\", \"Test Notebook\", \"id\")\n assert item_id == \"item-id-1234\"\n\n # Test API was called with correct parameters\n mock_endpoint.invoke.assert_called_with(\n method=\"GET\", url=f\"{constants.DEFAULT_API_ROOT_URL}/v1/workspaces/target-workspace-id/items\"\n )\n\n # Test finding a different item type\n item_id = workspace._lookup_item_attribute(\"target-workspace-id\", \"DataPipeline\", \"Test Pipeline\", \"id\")\n assert item_id == \"item-id-5678\"\n\n # Test item not found - should raise InputError\n from fabric_cicd._common._exceptions import InputError\n\n with pytest.raises(InputError) as exc_info:\n workspace._lookup_item_attribute(\"target-workspace-id\", \"Notebook\", \"Non-Existent Notebook\", \"id\")\n\n assert \"Failed to look up item in workspace\" in str(exc_info.value)\n assert \"target-workspace-id\" in str(exc_info.value)\n assert \"Notebook\" in str(exc_info.value)\n assert \"Non-Existent Notebook\" in str(exc_info.value)\n\n # Test item type not found - should raise InputError\n with pytest.raises(InputError) as exc_info:\n workspace._lookup_item_attribute(\"target-workspace-id\", \"NonExistentType\", \"Test Item\", \"id\")\n\n assert \"Failed to look up item in workspace\" in str(exc_info.value)\n assert \"target-workspace-id\" in str(exc_info.value)\n assert \"NonExistentType\" in str(exc_info.value)\n assert \"Test Item\" in str(exc_info.value)\n\n\ndef test_kqldatabase_folder_regex_root_eventhouse():\n \"\"\"KQLDatabase under top-level Eventhouse .children: group(1) is empty string.\"\"\"\n pattern = re.compile(constants.KQL_DATABASE_FOLDER_PATH_REGEX)\n relative_path = \"/SampleEventhouse.Eventhouse/.children/TaxiDB.KQLDatabase\"\n match = pattern.match(relative_path)\n assert match is not None, \"Regex should match a top-level Eventhouse .children path\"\n assert match.group(1) == \"\", \"Expected empty string for group(1) when Eventhouse is at repository root\"\n\n\ndef test_kqldatabase_folder_regex_nested_subfolder():\n \"\"\"KQLDatabase nested under a subfolder before Eventhouse: group(1) captures the subfolder path.\"\"\"\n pattern = re.compile(constants.KQL_DATABASE_FOLDER_PATH_REGEX)\n relative_path = \"/subfolder/EventhouseName.Eventhouse/.children/DB.KQLDatabase\"\n match = pattern.match(relative_path)\n assert match is not None, \"Regex should match nested Eventhouse .children path\"\n assert match.group(1) == \"/subfolder\", \"Expected '/subfolder' captured as the parent path\"\n\n\ndef test_kqldatabase_folder_regex_no_match_edge_case():\n \"\"\"Edge case: paths that do not follow the Eventhouse/.children pattern should not match.\"\"\"\n pattern = re.compile(constants.KQL_DATABASE_FOLDER_PATH_REGEX)\n # Missing '.Eventhouse/.children' sequence\n bad_paths = [\n \"/SomeFolder/TaxiDB.KQLDatabase\", # no Eventhouse container\n \"/Another.Eventhouse/TaxiDB.KQLDatabase\", # missing '.children'\n \"/prefix/.children/TaxiDB.KQLDatabase\", # missing Eventhouse segment\n ]\n for p in bad_paths:\n assert pattern.match(p) is None, f\"Regex should not match path: {p}\"\n\n\ndef test_get_item_attribute_caching_basic(patched_fabric_workspace, valid_workspace_id, temp_workspace_dir):\n \"\"\"Test that _get_item_attribute caches results and returns expected values.\"\"\"\n mock_endpoint = MagicMock()\n\n # Mock response for Lakehouse sqlendpoint attribute\n mock_response = {\"body\": {\"properties\": {\"sqlEndpointProperties\": {\"connectionString\": \"test-connection-string\"}}}}\n mock_endpoint.invoke.return_value = mock_response\n\n # Create workspace with mocked endpoint\n with patch(\"fabric_cicd.fabric_workspace.FabricEndpoint\", return_value=mock_endpoint):\n workspace = patched_fabric_workspace(\n workspace_id=valid_workspace_id,\n repository_directory=str(temp_workspace_dir),\n )\n workspace.endpoint = mock_endpoint\n\n # Test fetching an attribute\n result = workspace._get_item_attribute(\n workspace_id=\"test-workspace-id\",\n item_type=\"Lakehouse\",\n item_guid=\"test-item-guid\",\n item_name=\"Test Lakehouse\",\n attribute_name=\"sqlendpoint\",\n )\n\n # Verify the result is as expected\n assert result == \"test-connection-string\"\n\n # Verify API was called once\n assert mock_endpoint.invoke.call_count == 1\n mock_endpoint.invoke.assert_called_with(\n method=\"GET\",\n url=f\"{constants.DEFAULT_API_ROOT_URL}/v1/workspaces/test-workspace-id/lakehouses/test-item-guid\",\n )\n\n\ndef test_get_item_attribute_caching_prevents_api_call(patched_fabric_workspace, valid_workspace_id, temp_workspace_dir):\n \"\"\"Test that fetching the same attribute again uses cache and doesn't make API call.\"\"\"\n mock_endpoint = MagicMock()\n\n # Mock response for Lakehouse sqlendpoint attribute\n mock_response = {\"body\": {\"properties\": {\"sqlEndpointProperties\": {\"connectionString\": \"test-connection-string\"}}}}\n mock_endpoint.invoke.return_value = mock_response\n\n # Create workspace with mocked endpoint\n with patch(\"fabric_cicd.fabric_workspace.FabricEndpoint\", return_value=mock_endpoint):\n workspace = patched_fabric_workspace(\n workspace_id=valid_workspace_id,\n repository_directory=str(temp_workspace_dir),\n )\n workspace.endpoint = mock_endpoint\n\n # First call - should make API call\n result1 = workspace._get_item_attribute(\n workspace_id=\"test-workspace-id\",\n item_type=\"Lakehouse\",\n item_guid=\"test-item-guid\",\n item_name=\"Test Lakehouse\",\n attribute_name=\"sqlendpoint\",\n )\n\n # Second call with same parameters - should use cache\n result2 = workspace._get_item_attribute(\n workspace_id=\"test-workspace-id\",\n item_type=\"Lakehouse\",\n item_guid=\"test-item-guid\",\n item_name=\"Test Lakehouse\",\n attribute_name=\"sqlendpoint\",\n )\n\n # Verify results are the same\n assert result1 == result2 == \"test-connection-string\"\n\n # Verify API was called only once (cached on second call)\n assert mock_endpoint.invoke.call_count == 1\n\n\ndef test_get_item_attribute_different_cache_keys(patched_fabric_workspace, valid_workspace_id, temp_workspace_dir):\n \"\"\"Test that different cache keys don't collide and each makes separate API calls.\"\"\"\n mock_endpoint = MagicMock()\n\n # Mock response for different item types\n def mock_invoke_side_effect(*args, **kwargs):\n url = kwargs.get(\"url\", args[1] if len(args) > 1 else \"\")\n if \"lakehouses\" in url:\n return {\n \"body\": {\n \"properties\": {\n \"sqlEndpointProperties\": {\n \"id\": \"endpoint-id-123\",\n \"connectionString\": \"lakehouse-connection-string\",\n }\n }\n }\n }\n if \"warehouses\" in url:\n return {\"body\": {\"properties\": {\"connectionString\": \"warehouse-connection-string\"}}}\n if \"eventhouses\" in url:\n return {\"body\": {\"properties\": {\"queryServiceUri\": \"eventhouse-query-uri\"}}}\n return {\"body\": {}}\n\n mock_endpoint.invoke.side_effect = mock_invoke_side_effect\n\n # Create workspace with mocked endpoint\n with patch(\"fabric_cicd.fabric_workspace.FabricEndpoint\", return_value=mock_endpoint):\n workspace = patched_fabric_workspace(\n workspace_id=valid_workspace_id,\n repository_directory=str(temp_workspace_dir),\n )\n workspace.endpoint = mock_endpoint\n\n # Test different combinations to ensure no cache collisions\n\n # Different item types\n lakehouse_result = workspace._get_item_attribute(\"ws1\", \"Lakehouse\", \"guid1\", \"name1\", \"sqlendpoint\")\n warehouse_result = workspace._get_item_attribute(\"ws1\", \"Warehouse\", \"guid1\", \"name1\", \"sqlendpoint\")\n eventhouse_result = workspace._get_item_attribute(\"ws1\", \"Eventhouse\", \"guid1\", \"name1\", \"queryserviceuri\")\n\n # Different workspace IDs\n lakehouse_ws2_result = workspace._get_item_attribute(\"ws2\", \"Lakehouse\", \"guid1\", \"name1\", \"sqlendpoint\")\n\n # Different item GUIDs\n lakehouse_guid2_result = workspace._get_item_attribute(\"ws1\", \"Lakehouse\", \"guid2\", \"name1\", \"sqlendpoint\")\n\n # Different item names\n lakehouse_name2_result = workspace._get_item_attribute(\"ws1\", \"Lakehouse\", \"guid1\", \"name2\", \"sqlendpoint\")\n\n # Different attributes\n lakehouse_sqlendpointid_result = workspace._get_item_attribute(\n \"ws1\", \"Lakehouse\", \"guid1\", \"name1\", \"sqlendpointid\"\n )\n\n # Verify all results are different and correct\n assert lakehouse_result == \"lakehouse-connection-string\"\n assert warehouse_result == \"warehouse-connection-string\"\n assert eventhouse_result == \"eventhouse-query-uri\"\n assert lakehouse_ws2_result == \"lakehouse-connection-string\" # Same API response\n assert lakehouse_guid2_result == \"lakehouse-connection-string\" # Same API response\n assert lakehouse_name2_result == \"lakehouse-connection-string\" # Same API response\n\n # Mock the API to return different values for sqlendpointid\n def mock_invoke_side_effect_extended(*args, **kwargs):\n url = kwargs.get(\"url\", args[1] if len(args) > 1 else \"\")\n if \"lakehouses\" in url:\n if \"guid1\" in url:\n return {\n \"body\": {\n \"properties\": {\n \"sqlEndpointProperties\": {\n \"id\": \"endpoint-id-123\",\n \"connectionString\": \"lakehouse-connection-string\",\n }\n }\n }\n }\n # guid2\n return {\n \"body\": {\n \"properties\": {\n \"sqlEndpointProperties\": {\n \"id\": \"endpoint-id-456\",\n \"connectionString\": \"lakehouse-connection-string-2\",\n }\n }\n }\n }\n return {\"body\": {}}\n\n mock_endpoint.invoke.side_effect = mock_invoke_side_effect_extended\n\n # Fetch sqlendpointid for guid1\n lakehouse_sqlendpointid_result = workspace._get_item_attribute(\n \"ws1\", \"Lakehouse\", \"guid1\", \"name1\", \"sqlendpointid\"\n )\n assert lakehouse_sqlendpointid_result == \"endpoint-id-123\"\n\n # Verify API was called for each unique cache key\n # We expect 7 calls: 3 initial + 1 for ws2 + 1 for guid2 + 1 for name2 + 1 for sqlendpointid\n assert mock_endpoint.invoke.call_count == 7\n\n\ndef test_get_item_attribute_edge_cases(patched_fabric_workspace, valid_workspace_id, temp_workspace_dir):\n \"\"\"Test edge cases for _get_item_attribute to ensure cache doesn't introduce regressions.\"\"\"\n mock_endpoint = MagicMock()\n mock_endpoint.invoke.return_value = {\"body\": {}}\n\n # Create workspace with mocked endpoint\n with patch(\"fabric_cicd.fabric_workspace.FabricEndpoint\", return_value=mock_endpoint):\n workspace = patched_fabric_workspace(\n workspace_id=valid_workspace_id,\n repository_directory=str(temp_workspace_dir),\n )\n workspace.endpoint = mock_endpoint\n\n # Test empty item_guid - should return empty string without API call\n result = workspace._get_item_attribute(\"ws1\", \"Lakehouse\", \"\", \"name1\", \"sqlendpoint\")\n assert result == \"\"\n assert mock_endpoint.invoke.call_count == 0 # No API call made\n\n # Test None item_guid - should return empty string without API call\n result = workspace._get_item_attribute(\"ws1\", \"Lakehouse\", None, \"name1\", \"sqlendpoint\")\n assert result == \"\"\n assert mock_endpoint.invoke.call_count == 0 # No API call made\n\n # Test unsupported item type - should return empty string without API call\n result = workspace._get_item_attribute(\"ws1\", \"UnsupportedType\", \"guid1\", \"name1\", \"someattr\")\n assert result == \"\"\n assert mock_endpoint.invoke.call_count == 0 # No API call made\n\n # Test unsupported attribute for supported item type - should return empty string without API call\n result = workspace._get_item_attribute(\"ws1\", \"Lakehouse\", \"guid1\", \"name1\", \"unsupportedattr\")\n assert result == \"\"\n assert mock_endpoint.invoke.call_count == 0 # No API call made\n\n # Test valid call that results in empty attribute value - should raise InputError\n mock_endpoint.invoke.return_value = {\n \"body\": {\n \"properties\": {\n \"sqlEndpointProperties\": {\n \"connectionString\": \"\" # Empty value\n }\n }\n }\n }\n\n from fabric_cicd._common._exceptions import InputError\n\n with pytest.raises(InputError) as exc_info:\n workspace._get_item_attribute(\"ws1\", \"Lakehouse\", \"guid1\", \"name1\", \"sqlendpoint\")\n\n assert \"Attribute value not found\" in str(exc_info.value)\n assert \"Lakehouse\" in str(exc_info.value)\n assert \"name1\" in str(exc_info.value)\n\n # Verify the error case was not cached\n with pytest.raises(InputError):\n workspace._get_item_attribute(\"ws1\", \"Lakehouse\", \"guid1\", \"name1\", \"sqlendpoint\")\n # Should still be only 1 API call (cached error)\n assert mock_endpoint.invoke.call_count == 2\n\n\ndef test_multiple_items_with_default_guid_logical_id(temp_workspace_dir, patched_fabric_workspace, valid_workspace_id):\n \"\"\"Test that multiple items with DEFAULT_GUID as logical ID don't raise a duplicate error.\"\"\"\n # Create multiple items all using the default GUID (export API scenario)\n default_guid = constants.DEFAULT_GUID\n\n for i, item_dir_name in enumerate([\"Notebook1.Notebook\", \"Notebook2.Notebook\", \"Pipeline1.DataPipeline\"]):\n item_dir = temp_workspace_dir / item_dir_name\n item_dir.mkdir(parents=True, exist_ok=True)\n\n item_type = \"Notebook\" if \"Notebook\" in item_dir_name else \"DataPipeline\"\n metadata_content = {\n \"metadata\": {\n \"type\": item_type,\n \"displayName\": f\"Item {i}\",\n \"description\": \"\",\n },\n \"config\": {\"logicalId\": default_guid},\n }\n\n with (item_dir / \".platform\").open(\"w\", encoding=\"utf-8\") as f:\n json.dump(metadata_content, f, ensure_ascii=False)\n\n with (item_dir / \"dummy.txt\").open(\"w\", encoding=\"utf-8\") as f:\n f.write(\"Dummy file content\")\n\n # Should NOT raise any error\n workspace = patched_fabric_workspace(\n workspace_id=valid_workspace_id,\n repository_directory=str(temp_workspace_dir),\n item_type_in_scope=[\"Notebook\", \"DataPipeline\"],\n )\n\n # Verify all items were loaded\n assert \"Notebook\" in workspace.repository_items\n assert len(workspace.repository_items[\"Notebook\"]) == 2\n assert \"DataPipeline\" in workspace.repository_items\n assert len(workspace.repository_items[\"DataPipeline\"]) == 1\n\n\ndef test_duplicate_non_default_logical_id_raises_error(\n temp_workspace_dir, patched_fabric_workspace, valid_workspace_id\n):\n \"\"\"Test that duplicate non-default logical IDs still raise an error.\"\"\"\n from fabric_cicd._common._exceptions import FailedPublishedItemStatusError\n\n duplicate_logical_id = \"aaaaaaaa-bbbb-cccc-dddd-eeeeeeeeeeee\"\n\n for i, item_dir_name in enumerate([\"Notebook1.Notebook\", \"Notebook2.Notebook\"]):\n item_dir = temp_workspace_dir / item_dir_name\n item_dir.mkdir(parents=True, exist_ok=True)\n\n metadata_content = {\n \"metadata\": {\n \"type\": \"Notebook\",\n \"displayName\": f\"Duplicate Item {i}\",\n \"description\": \"\",\n },\n \"config\": {\"logicalId\": duplicate_logical_id},\n }\n\n with (item_dir / \".platform\").open(\"w\", encoding=\"utf-8\") as f:\n json.dump(metadata_content, f, ensure_ascii=False)\n\n with (item_dir / \"dummy.txt\").open(\"w\", encoding=\"utf-8\") as f:\n f.write(\"Dummy file content\")\n\n with pytest.raises(FailedPublishedItemStatusError) as exc_info:\n patched_fabric_workspace(\n workspace_id=valid_workspace_id,\n repository_directory=str(temp_workspace_dir),\n item_type_in_scope=[\"Notebook\"],\n )\n\n assert \"Duplicate logicalId\" in str(exc_info.value)\n assert duplicate_logical_id in str(exc_info.value)\n\n\ndef test_replace_logical_ids_skips_default_guid(temp_workspace_dir, patched_fabric_workspace, valid_workspace_id):\n \"\"\"Test that _replace_logical_ids skips items with DEFAULT_GUID as their logical ID.\"\"\"\n from fabric_cicd._common._item import Item\n\n with patch.object(FabricWorkspace, \"_refresh_repository_items\"):\n workspace = patched_fabric_workspace(\n workspace_id=valid_workspace_id,\n repository_directory=str(temp_workspace_dir),\n item_type_in_scope=[\"Notebook\"],\n )\n\n # Set up repository items with DEFAULT_GUID logical IDs\n workspace.repository_items = {\n \"Notebook\": {\n \"Notebook1\": Item(\n type=\"Notebook\",\n name=\"Notebook1\",\n description=\"\",\n guid=\"actual-guid-1111\",\n logical_id=constants.DEFAULT_GUID,\n ),\n \"Notebook2\": Item(\n type=\"Notebook\",\n name=\"Notebook2\",\n description=\"\",\n guid=\"actual-guid-2222\",\n logical_id=constants.DEFAULT_GUID,\n ),\n }\n }\n\n # File content containing the default GUID (e.g., as a workspace ID placeholder)\n raw_file = f'{{\"workspaceId\": \"{constants.DEFAULT_GUID}\"}}'\n\n result = workspace._replace_logical_ids(raw_file)\n\n # DEFAULT_GUID should NOT have been replaced by any item GUID\n assert constants.DEFAULT_GUID in result\n assert \"actual-guid-1111\" not in result\n assert \"actual-guid-2222\" not in result\n\n\ndef test_replace_logical_ids_replaces_non_default_guid(\n temp_workspace_dir, patched_fabric_workspace, valid_workspace_id\n):\n \"\"\"Test that _replace_logical_ids still replaces non-default logical IDs correctly.\"\"\"\n from fabric_cicd._common._item import Item\n\n with patch.object(FabricWorkspace, \"_refresh_repository_items\"):\n workspace = patched_fabric_workspace(\n workspace_id=valid_workspace_id,\n repository_directory=str(temp_workspace_dir),\n item_type_in_scope=[\"Notebook\"],\n )\n\n logical_id = \"aaaaaaaa-bbbb-cccc-dddd-eeeeeeeeeeee\"\n item_guid = \"11111111-2222-3333-4444-555555555555\"\n\n workspace.repository_items = {\n \"Notebook\": {\n \"MyNotebook\": Item(\n type=\"Notebook\",\n name=\"MyNotebook\",\n description=\"\",\n guid=item_guid,\n logical_id=logical_id,\n ),\n }\n }\n\n raw_file = f'{{\"notebookId\": \"{logical_id}\"}}'\n result = workspace._replace_logical_ids(raw_file)\n\n assert logical_id not in result\n assert item_guid in result\n\n\ndef test_mix_of_default_and_non_default_logical_ids(temp_workspace_dir, patched_fabric_workspace, valid_workspace_id):\n \"\"\"Test repository with a mix of DEFAULT_GUID and unique logical IDs.\"\"\"\n unique_logical_id = \"aaaaaaaa-bbbb-cccc-dddd-eeeeeeeeeeee\"\n\n # Item 1: export API item with default GUID\n item_dir_1 = temp_workspace_dir / \"ExportedNotebook.Notebook\"\n item_dir_1.mkdir(parents=True, exist_ok=True)\n metadata_1 = {\n \"metadata\": {\"type\": \"Notebook\", \"displayName\": \"Exported Notebook\", \"description\": \"\"},\n \"config\": {\"logicalId\": constants.DEFAULT_GUID},\n }\n with (item_dir_1 / \".platform\").open(\"w\", encoding=\"utf-8\") as f:\n json.dump(metadata_1, f)\n with (item_dir_1 / \"dummy.txt\").open(\"w\", encoding=\"utf-8\") as f:\n f.write(\"content\")\n\n # Item 2: git integration item with unique logical ID\n item_dir_2 = temp_workspace_dir / \"GitNotebook.Notebook\"\n item_dir_2.mkdir(parents=True, exist_ok=True)\n metadata_2 = {\n \"metadata\": {\"type\": \"Notebook\", \"displayName\": \"Git Notebook\", \"description\": \"\"},\n \"config\": {\"logicalId\": unique_logical_id},\n }\n with (item_dir_2 / \".platform\").open(\"w\", encoding=\"utf-8\") as f:\n json.dump(metadata_2, f)\n with (item_dir_2 / \"dummy.txt\").open(\"w\", encoding=\"utf-8\") as f:\n f.write(\"content\")\n\n # Item 3: another export API item with default GUID\n item_dir_3 = temp_workspace_dir / \"ExportedPipeline.DataPipeline\"\n item_dir_3.mkdir(parents=True, exist_ok=True)\n metadata_3 = {\n \"metadata\": {\"type\": \"DataPipeline\", \"displayName\": \"Exported Pipeline\", \"description\": \"\"},\n \"config\": {\"logicalId\": constants.DEFAULT_GUID},\n }\n with (item_dir_3 / \".platform\").open(\"w\", encoding=\"utf-8\") as f:\n json.dump(metadata_3, f)\n with (item_dir_3 / \"dummy.txt\").open(\"w\", encoding=\"utf-8\") as f:\n f.write(\"content\")\n\n # Should NOT raise any error\n workspace = patched_fabric_workspace(\n workspace_id=valid_workspace_id,\n repository_directory=str(temp_workspace_dir),\n item_type_in_scope=[\"Notebook\", \"DataPipeline\"],\n )\n\n assert len(workspace.repository_items[\"Notebook\"]) == 2\n assert len(workspace.repository_items[\"DataPipeline\"]) == 1\n assert workspace.repository_items[\"Notebook\"][\"Exported Notebook\"].logical_id == constants.DEFAULT_GUID\n assert workspace.repository_items[\"Notebook\"][\"Git Notebook\"].logical_id == unique_logical_id\n assert workspace.repository_items[\"DataPipeline\"][\"Exported Pipeline\"].logical_id == constants.DEFAULT_GUID\n\n\ndef test_publish_variable_library_only_calls_replace_parameters(\n temp_workspace_dir, patched_fabric_workspace, valid_workspace_id\n):\n \"\"\"Test that Variable Library items only have _replace_parameters called, not logical ID or workspace ID replacement.\"\"\"\n workspace = patched_fabric_workspace(valid_workspace_id, str(temp_workspace_dir))\n\n mock_file = MagicMock()\n mock_file.relative_path = \"valueSets/Default.json\"\n mock_file.type = \"text\"\n mock_file.file_path = Path(\"valueSets/Default.json\")\n mock_file.contents = '{\"key\": \"value\", \"workspace\": \"00000000-0000-0000-0000-000000000000\"}'\n mock_file.base64_payload = {\"path\": \"valueSets/Default.json\", \"payloadType\": \"InlineBase64\"}\n\n mock_item = MagicMock()\n mock_item.guid = None\n mock_item.folder_id = \"\"\n mock_item.folder_path = \"\"\n mock_item.description = \"\"\n mock_item.logical_id = \"test-logical-id\"\n mock_item.item_files = [mock_file]\n mock_item.skip_publish = False\n mock_item.type = \"VariableLibrary\"\n mock_item.name = \"TestVars\"\n\n workspace.repository_items = {\"VariableLibrary\": {\"TestVars\": mock_item}}\n workspace.deployed_items = {}\n\n with (\n patch.object(workspace, \"_replace_logical_ids\", wraps=workspace._replace_logical_ids) as mock_logical,\n patch.object(workspace, \"_replace_parameters\", side_effect=lambda file, _: file.contents) as mock_params,\n patch.object(workspace, \"_replace_workspace_ids\", wraps=workspace._replace_workspace_ids) as mock_ws,\n ):\n workspace._publish_item(item_name=\"TestVars\", item_type=\"VariableLibrary\")\n\n # _replace_parameters should be called\n mock_params.assert_called_once()\n # _replace_logical_ids and _replace_workspace_ids should NOT be called\n mock_logical.assert_not_called()\n mock_ws.assert_not_called()\n\n\ndef test_publish_non_variable_library_calls_all_replacements(\n temp_workspace_dir, patched_fabric_workspace, valid_workspace_id\n):\n \"\"\"Test that non-Variable Library items still go through the full replacement pipeline.\"\"\"\n workspace = patched_fabric_workspace(valid_workspace_id, str(temp_workspace_dir))\n\n mock_file = MagicMock()\n mock_file.relative_path = \"notebook-content.py\"\n mock_file.type = \"text\"\n mock_file.file_path = Path(\"notebook-content.py\")\n mock_file.contents = \"print('hello')\"\n mock_file.base64_payload = {\"path\": \"notebook-content.py\", \"payloadType\": \"InlineBase64\"}\n\n mock_item = MagicMock()\n mock_item.guid = None\n mock_item.folder_id = \"\"\n mock_item.folder_path = \"\"\n mock_item.description = \"\"\n mock_item.logical_id = \"test-logical-id\"\n mock_item.item_files = [mock_file]\n mock_item.skip_publish = False\n mock_item.type = \"Notebook\"\n mock_item.name = \"TestNotebook\"\n\n workspace.repository_items = {\"Notebook\": {\"TestNotebook\": mock_item}}\n workspace.deployed_items = {}\n\n with (\n patch.object(workspace, \"_replace_logical_ids\", side_effect=lambda x: x) as mock_logical,\n patch.object(workspace, \"_replace_parameters\", side_effect=lambda file, _: file.contents) as mock_params,\n patch.object(workspace, \"_replace_workspace_ids\", side_effect=lambda x: x) as mock_ws,\n ):\n workspace._publish_item(item_name=\"TestNotebook\", item_type=\"Notebook\")\n\n # All three replacement methods should be called\n mock_logical.assert_called_once()\n mock_params.assert_called_once()\n mock_ws.assert_called_once()\n\n\ndef test_api_root_url_snapshot_is_not_retargeted_by_second_configure_call(\n temp_workspace_dir, patched_fabric_workspace, monkeypatch\n):\n \"\"\"Test that a constructed FabricWorkspace retains its snapshotted URL\n even if configure_fabric_fqdn is called again for a different workspace.\"\"\"\n workspace_id_a = \"f953f3da-c5f0-4e36-a644-c85933e35e2f\"\n workspace_id_b = \"a1b2c3d4-e5f6-7890-abcd-ef1234567890\"\n\n # Reset globals to defaults before test\n monkeypatch.setattr(constants, \"DEFAULT_API_ROOT_URL\", \"https://api.powerbi.com\")\n monkeypatch.setattr(constants, \"FABRIC_API_ROOT_URL\", \"https://api.fabric.microsoft.com\")\n\n # Configure for workspace a and construct it\n configure_fabric_fqdn(workspace_id_a)\n expected_fqdn_a = constants.DEFAULT_API_ROOT_URL # snapshot what was set\n\n with patch.object(FabricWorkspace, \"_refresh_repository_items\"):\n workspace_a = patched_fabric_workspace(\n workspace_id=workspace_id_a,\n repository_directory=str(temp_workspace_dir),\n )\n\n # Now configure for workspace b\n configure_fabric_fqdn(workspace_id_b)\n\n # workspace_a should still use fqdn_a, not fqdn_b\n assert workspace_a._api_root_url == expected_fqdn_a\n assert workspace_a.base_api_url.startswith(expected_fqdn_a)\n" }, { "path": "tests/test_fqdn_workspace_id.py", "content": "# Copyright (c) Microsoft Corporation.\n# Licensed under the MIT License.\n\nimport pytest\n\nimport fabric_cicd\nimport fabric_cicd.constants as constants\nfrom fabric_cicd import configure_fabric_fqdn\nfrom fabric_cicd._common._validate_env_vars import _get_fabric_fqdn_url\n\n\nclass TestGetFabricFqdnUrl:\n \"\"\"Tests for the _get_fabric_fqdn_url helper function.\"\"\"\n\n def test_produces_correct_fqdn_url(self):\n url = _get_fabric_fqdn_url(\"f953f3da-c5f0-4e36-a644-c85933e35e2f\")\n assert url == \"https://f953f3dac5f04e36a644c85933e35e2f.zf9.w.api.fabric.microsoft.com\"\n\n def test_rejects_workspace_id_without_dashes(self):\n with pytest.raises(ValueError, match=\"valid GUID with dashes\"):\n _get_fabric_fqdn_url(\"f953f3dac5f04e36a644c85933e35e2f\")\n\n\nclass TestConfigureFabricFqdn:\n \"\"\"Tests for configure_fabric_fqdn.\"\"\"\n\n def test_globals_updated(self, monkeypatch):\n monkeypatch.setattr(constants, \"FABRIC_API_ROOT_URL\", \"https://api.fabric.microsoft.com\")\n monkeypatch.setattr(constants, \"DEFAULT_API_ROOT_URL\", \"https://api.powerbi.com\")\n\n configure_fabric_fqdn(\"f953f3da-c5f0-4e36-a644-c85933e35e2f\")\n\n expected = \"https://f953f3dac5f04e36a644c85933e35e2f.zf9.w.api.fabric.microsoft.com\"\n assert expected == constants.FABRIC_API_ROOT_URL\n assert expected == constants.DEFAULT_API_ROOT_URL\n\n def test_overwrite_warning_on_second_call(self, monkeypatch, mocker):\n monkeypatch.setattr(constants, \"FABRIC_API_ROOT_URL\", \"https://api.fabric.microsoft.com\")\n monkeypatch.setattr(constants, \"DEFAULT_API_ROOT_URL\", \"https://api.powerbi.com\")\n\n mock_logger = mocker.Mock()\n monkeypatch.setattr(fabric_cicd, \"logger\", mock_logger)\n\n configure_fabric_fqdn(\"f953f3da-c5f0-4e36-a644-c85933e35e2f\")\n mock_logger.warning.assert_not_called()\n\n configure_fabric_fqdn(\"f953f3da-c5f0-4e36-a644-c85933e35e2f\")\n mock_logger.warning.assert_called_once()\n" }, { "path": "tests/test_git_diff_utils.py", "content": "# Copyright (c) Microsoft Corporation.\n# Licensed under the MIT License.\n\n\"\"\"Tests for git diff utilities: get_changed_items() and validate_git_compare_ref().\"\"\"\n\nfrom unittest.mock import patch\n\nimport pytest\n\nimport fabric_cicd._common._git_diff_utils as git_utils\nfrom fabric_cicd._common._exceptions import InputError\nfrom fabric_cicd._common._validate_input import validate_git_compare_ref\n\n# =============================================================================\n# Tests for validate_git_compare_ref()\n# =============================================================================\n\n\nclass TestValidateGitCompareRef:\n def test_accepts_common_valid_refs(self):\n assert validate_git_compare_ref(\"HEAD~1\") == \"HEAD~1\"\n assert validate_git_compare_ref(\"main\") == \"main\"\n assert validate_git_compare_ref(\"feature/my_branch\") == \"feature/my_branch\"\n assert validate_git_compare_ref(\"release/v1.2.3\") == \"release/v1.2.3\"\n\n def test_rejects_empty_string(self):\n with pytest.raises(InputError):\n validate_git_compare_ref(\"\")\n\n def test_rejects_whitespace_only(self):\n with pytest.raises(InputError):\n validate_git_compare_ref(\" \")\n\n def test_rejects_dash_prefixed(self):\n with pytest.raises(InputError):\n validate_git_compare_ref(\"-n\")\n with pytest.raises(InputError):\n validate_git_compare_ref(\"--help\")\n\n def test_rejects_invalid_characters(self):\n with pytest.raises(InputError):\n validate_git_compare_ref(\"ref;rm -rf /\")\n\n def test_rejects_shell_metacharacters(self):\n \"\"\"Prevent shell injection via backticks, pipes, dollar signs, etc.\"\"\"\n for ref in [\"$(whoami)\", \"`id`\", \"ref|cat /etc/passwd\", \"ref&echo bad\", \"ref\\nnewline\"]:\n with pytest.raises(InputError):\n validate_git_compare_ref(ref)\n\n def test_rejects_non_string_input(self):\n \"\"\"Non-string types must be rejected.\"\"\"\n with pytest.raises(InputError):\n validate_git_compare_ref(123)\n with pytest.raises(InputError):\n validate_git_compare_ref(None)\n\n def test_accepts_advanced_git_ref_syntax(self):\n \"\"\"Valid git ref syntax including caret, tilde, and reflog notation.\"\"\"\n assert validate_git_compare_ref(\"HEAD^\") == \"HEAD^\"\n assert validate_git_compare_ref(\"HEAD~3\") == \"HEAD~3\"\n assert validate_git_compare_ref(\"main@{1}\") == \"main@{1}\"\n assert validate_git_compare_ref(\"origin/main\") == \"origin/main\"\n assert validate_git_compare_ref(\"v2.0.0\") == \"v2.0.0\"\n assert validate_git_compare_ref(\"abc123def\") == \"abc123def\"\n\n\n# =============================================================================\n# Tests for _resolve_git_diff_path()\n# =============================================================================\n\n\nclass TestResolveGitDiffPath:\n \"\"\"Tests for path validation/resolution from git diff output.\"\"\"\n\n def test_rejects_absolute_paths(self, tmp_path):\n result = git_utils._resolve_git_diff_path(\"/etc/passwd\", tmp_path, tmp_path)\n assert result is None\n\n def test_rejects_path_traversal(self, tmp_path):\n result = git_utils._resolve_git_diff_path(\"../../../etc/passwd\", tmp_path, tmp_path)\n assert result is None\n\n def test_rejects_null_bytes(self, tmp_path):\n result = git_utils._resolve_git_diff_path(\"file\\x00.txt\", tmp_path, tmp_path)\n assert result is None\n\n def test_accepts_valid_relative_path(self, tmp_path):\n sub = tmp_path / \"MyItem\"\n sub.mkdir()\n result = git_utils._resolve_git_diff_path(\"MyItem/file.py\", tmp_path, tmp_path)\n assert result is not None\n assert result.name == \"file.py\"\n\n def test_rejects_path_outside_repo_directory(self, tmp_path):\n \"\"\"A file under git root but outside the repo subdirectory is rejected.\"\"\"\n repo_sub = tmp_path / \"workspace\"\n repo_sub.mkdir()\n result = git_utils._resolve_git_diff_path(\"other/file.py\", tmp_path, repo_sub)\n assert result is None\n\n\n# =============================================================================\n# Tests for _find_platform_item()\n# =============================================================================\n\n\nclass TestFindPlatformItem:\n \"\"\"Tests for .platform file discovery and parsing.\"\"\"\n\n def test_finds_platform_in_same_directory(self, tmp_path):\n item_dir = tmp_path / \"MyItem.Notebook\"\n item_dir.mkdir()\n (item_dir / \".platform\").write_text(\n '{\"metadata\": {\"type\": \"Notebook\", \"displayName\": \"MyItem\"}}', encoding=\"utf-8\"\n )\n file_path = item_dir / \"notebook.py\"\n file_path.touch()\n result = git_utils._find_platform_item(file_path, tmp_path)\n assert result == (\"MyItem\", \"Notebook\")\n\n def test_returns_none_when_no_platform_file(self, tmp_path):\n item_dir = tmp_path / \"NoItem\"\n item_dir.mkdir()\n file_path = item_dir / \"file.py\"\n file_path.touch()\n result = git_utils._find_platform_item(file_path, tmp_path)\n assert result is None\n\n def test_returns_none_for_malformed_platform_json(self, tmp_path):\n item_dir = tmp_path / \"BadItem\"\n item_dir.mkdir()\n (item_dir / \".platform\").write_text(\"not valid json\", encoding=\"utf-8\")\n file_path = item_dir / \"file.py\"\n file_path.touch()\n result = git_utils._find_platform_item(file_path, tmp_path)\n assert result is None\n\n def test_returns_none_when_metadata_missing_type(self, tmp_path):\n item_dir = tmp_path / \"NoType\"\n item_dir.mkdir()\n (item_dir / \".platform\").write_text(\n '{\"metadata\": {\"displayName\": \"NoType\"}}',\n encoding=\"utf-8\",\n )\n file_path = item_dir / \"file.py\"\n file_path.touch()\n result = git_utils._find_platform_item(file_path, tmp_path)\n assert result is None\n\n\n# =============================================================================\n# Tests for get_changed_items()\n# =============================================================================\n\n\nclass TestGetChangedItems:\n \"\"\"Tests for the public get_changed_items() utility function.\"\"\"\n\n def _make_git_diff_output(self, lines: list[str]) -> str:\n return \"\\n\".join(lines)\n\n def test_returns_changed_items_from_git_diff(self, tmp_path):\n \"\"\"Returns items detected as modified/added by git diff.\"\"\"\n # Set up a fake item directory with a .platform file\n item_dir = tmp_path / \"MyNotebook.Notebook\"\n item_dir.mkdir()\n platform = item_dir / \".platform\"\n platform.write_text(\n '{\"metadata\": {\"type\": \"Notebook\", \"displayName\": \"MyNotebook\"}}',\n encoding=\"utf-8\",\n )\n changed_file = item_dir / \"notebook.py\"\n changed_file.write_text(\"print('hello')\", encoding=\"utf-8\")\n\n diff_output = self._make_git_diff_output([\"M\\tMyNotebook.Notebook/notebook.py\"])\n\n git_root_patch = \"fabric_cicd._common._config_validator._find_git_root\"\n\n with (\n patch(git_root_patch, return_value=tmp_path),\n patch(\"subprocess.run\") as mock_run,\n ):\n mock_run.return_value.stdout = diff_output\n mock_run.return_value.returncode = 0\n\n result = git_utils.get_changed_items(tmp_path)\n\n assert result == [\"MyNotebook.Notebook\"]\n\n def test_returns_empty_list_when_no_changes(self, tmp_path):\n \"\"\"Returns an empty list when git diff reports no changed files.\"\"\"\n git_root_patch = \"fabric_cicd._common._config_validator._find_git_root\"\n\n with (\n patch(git_root_patch, return_value=tmp_path),\n patch(\"subprocess.run\") as mock_run,\n ):\n mock_run.return_value.stdout = \"\"\n mock_run.return_value.returncode = 0\n\n result = git_utils.get_changed_items(tmp_path)\n\n assert result == []\n\n def test_returns_empty_list_when_git_root_not_found(self, tmp_path):\n \"\"\"Returns an empty list and logs a warning when no git root is found.\"\"\"\n git_root_patch = \"fabric_cicd._common._config_validator._find_git_root\"\n\n with patch(git_root_patch, return_value=None):\n result = git_utils.get_changed_items(tmp_path)\n\n assert result == []\n\n def test_returns_empty_list_when_git_diff_fails(self, tmp_path):\n \"\"\"Returns an empty list and logs a warning when git diff fails.\"\"\"\n import subprocess\n\n git_root_patch = \"fabric_cicd._common._config_validator._find_git_root\"\n\n with (\n patch(git_root_patch, return_value=tmp_path),\n patch(\"subprocess.run\", side_effect=subprocess.CalledProcessError(1, \"git\", stderr=\"bad ref\")),\n ):\n result = git_utils.get_changed_items(tmp_path)\n\n assert result == []\n\n def test_uses_custom_git_compare_ref(self, tmp_path):\n \"\"\"Passes the custom git_compare_ref to the underlying git command.\"\"\"\n git_root_patch = \"fabric_cicd._common._config_validator._find_git_root\"\n\n with (\n patch(git_root_patch, return_value=tmp_path),\n patch(\"subprocess.run\") as mock_run,\n ):\n mock_run.return_value.stdout = \"\"\n mock_run.return_value.returncode = 0\n\n git_utils.get_changed_items(tmp_path, git_compare_ref=\"main\")\n\n call_args = mock_run.call_args[0][0]\n assert call_args == [\"git\", \"diff\", \"--name-status\", \"main\"]\n\n def test_excludes_files_outside_repository_directory(self, tmp_path):\n \"\"\"Files changed outside the configured repository_directory are ignored.\"\"\"\n outside_dir = tmp_path / \"other_repo\" / \"SomeItem.Notebook\"\n outside_dir.mkdir(parents=True)\n\n diff_output = self._make_git_diff_output([\"M\\tother_repo/SomeItem.Notebook/item.py\"])\n\n git_root_patch = \"fabric_cicd._common._config_validator._find_git_root\"\n\n with (\n patch(git_root_patch, return_value=tmp_path),\n patch(\"subprocess.run\") as mock_run,\n ):\n mock_run.return_value.stdout = diff_output\n mock_run.return_value.returncode = 0\n\n # Use a subdirectory as the repository_directory so \"other_repo\" is out of scope\n repo_subdir = tmp_path / \"my_workspace\"\n repo_subdir.mkdir()\n result = git_utils.get_changed_items(repo_subdir)\n\n assert result == []\n\n def test_deduplicates_multiple_files_in_same_item(self, tmp_path):\n \"\"\"Multiple changed files in the same item should produce a single entry.\"\"\"\n item_dir = tmp_path / \"MyNotebook.Notebook\"\n item_dir.mkdir()\n (item_dir / \".platform\").write_text(\n '{\"metadata\": {\"type\": \"Notebook\", \"displayName\": \"MyNotebook\"}}',\n encoding=\"utf-8\",\n )\n (item_dir / \"file1.py\").write_text(\"a\", encoding=\"utf-8\")\n (item_dir / \"file2.py\").write_text(\"b\", encoding=\"utf-8\")\n\n diff_output = self._make_git_diff_output([\n \"M\\tMyNotebook.Notebook/file1.py\",\n \"M\\tMyNotebook.Notebook/file2.py\",\n ])\n\n git_root_patch = \"fabric_cicd._common._config_validator._find_git_root\"\n\n with (\n patch(git_root_patch, return_value=tmp_path),\n patch(\"subprocess.run\") as mock_run,\n ):\n mock_run.return_value.stdout = diff_output\n mock_run.return_value.returncode = 0\n result = git_utils.get_changed_items(tmp_path)\n\n assert result == [\"MyNotebook.Notebook\"]\n\n def test_handles_renamed_files(self, tmp_path):\n \"\"\"Renamed files (R status) should be detected via the new path.\"\"\"\n item_dir = tmp_path / \"Renamed.Notebook\"\n item_dir.mkdir()\n (item_dir / \".platform\").write_text(\n '{\"metadata\": {\"type\": \"Notebook\", \"displayName\": \"Renamed\"}}',\n encoding=\"utf-8\",\n )\n (item_dir / \"new_name.py\").write_text(\"x\", encoding=\"utf-8\")\n\n diff_output = self._make_git_diff_output([\"R100\\tOld.Notebook/old.py\\tRenamed.Notebook/new_name.py\"])\n\n git_root_patch = \"fabric_cicd._common._config_validator._find_git_root\"\n\n with (\n patch(git_root_patch, return_value=tmp_path),\n patch(\"subprocess.run\") as mock_run,\n ):\n mock_run.return_value.stdout = diff_output\n mock_run.return_value.returncode = 0\n result = git_utils.get_changed_items(tmp_path)\n\n assert result == [\"Renamed.Notebook\"]\n\n def test_returns_empty_list_on_timeout(self, tmp_path):\n \"\"\"A git diff timeout should return an empty list gracefully.\"\"\"\n import subprocess\n\n git_root_patch = \"fabric_cicd._common._config_validator._find_git_root\"\n\n with (\n patch(git_root_patch, return_value=tmp_path),\n patch(\"subprocess.run\", side_effect=subprocess.TimeoutExpired(\"git\", 30)),\n ):\n result = git_utils.get_changed_items(tmp_path)\n\n assert result == []\n\n def test_multiple_distinct_items(self, tmp_path):\n \"\"\"Changes across multiple items should all be returned.\"\"\"\n for name, item_type in [(\"NB1\", \"Notebook\"), (\"Pipeline1\", \"DataPipeline\")]:\n item_dir = tmp_path / f\"{name}.{item_type}\"\n item_dir.mkdir()\n (item_dir / \".platform\").write_text(\n f'{{\"metadata\": {{\"type\": \"{item_type}\", \"displayName\": \"{name}\"}}}}',\n encoding=\"utf-8\",\n )\n (item_dir / \"file.py\").write_text(\"content\", encoding=\"utf-8\")\n\n diff_output = self._make_git_diff_output([\n \"M\\tNB1.Notebook/file.py\",\n \"A\\tPipeline1.DataPipeline/file.py\",\n ])\n\n git_root_patch = \"fabric_cicd._common._config_validator._find_git_root\"\n\n with (\n patch(git_root_patch, return_value=tmp_path),\n patch(\"subprocess.run\") as mock_run,\n ):\n mock_run.return_value.stdout = diff_output\n mock_run.return_value.returncode = 0\n result = git_utils.get_changed_items(tmp_path)\n\n assert sorted(result) == [\"NB1.Notebook\", \"Pipeline1.DataPipeline\"]\n\n def test_rejects_dangerous_git_compare_ref(self, tmp_path):\n \"\"\"Passing an invalid git_compare_ref should raise InputError before running git.\"\"\"\n with pytest.raises(InputError):\n git_utils.get_changed_items(tmp_path, git_compare_ref=\"--exec=whoami\")\n" }, { "path": "tests/test_hard_delete.py", "content": "# Copyright (c) Microsoft Corporation.\n# Licensed under the MIT License.\n\n\"\"\"Test enable_hard_delete feature flag functionality.\"\"\"\n\nimport json\nimport tempfile\nfrom pathlib import Path\nfrom unittest.mock import MagicMock, patch\n\nimport pytest\nfrom fixtures.credentials import DummyTokenCredential\n\nimport fabric_cicd.constants as constants\nfrom fabric_cicd.constants import FeatureFlag\nfrom fabric_cicd.fabric_workspace import FabricWorkspace\n\n\n@pytest.fixture\ndef mock_endpoint():\n \"\"\"Mock FabricEndpoint to capture DELETE requests.\"\"\"\n mock = MagicMock()\n mock.delete_urls = []\n\n def mock_invoke(method, url, **_kwargs):\n if method == \"DELETE\":\n mock.delete_urls.append(url)\n return {\"body\": {}, \"header\": {}, \"status_code\": 200}\n if method == \"GET\" and \"workspaces\" in url and not url.endswith(\"/items\"):\n return {\"body\": {\"value\": [], \"capacityId\": \"test-capacity\"}}\n if method == \"GET\" and url.endswith(\"/items\"):\n return {\"body\": {\"value\": []}}\n return {\"body\": {\"value\": []}}\n\n mock.invoke.side_effect = mock_invoke\n return mock\n\n\n@pytest.fixture\ndef test_workspace(mock_endpoint):\n \"\"\"Create a test workspace with a notebook item.\"\"\"\n with tempfile.TemporaryDirectory() as temp_dir:\n temp_path = Path(temp_dir)\n\n notebook_dir = temp_path / \"TestNotebook.Notebook\"\n notebook_dir.mkdir(parents=True, exist_ok=True)\n\n platform_file = notebook_dir / \".platform\"\n platform_file.write_text(\n json.dumps({\n \"metadata\": {\n \"kernel_info\": {\"name\": \"synapse_pyspark\"},\n \"language_info\": {\"name\": \"python\"},\n }\n })\n )\n\n notebook_file = notebook_dir / \"notebook-content.py\"\n notebook_file.write_text(\"# Test notebook content\\nprint('Hello World')\")\n\n with (\n patch(\"fabric_cicd.fabric_workspace.FabricEndpoint\", return_value=mock_endpoint),\n patch.object(\n FabricWorkspace, \"_refresh_deployed_items\", new=lambda self: setattr(self, \"deployed_items\", {})\n ),\n patch.object(\n FabricWorkspace, \"_refresh_deployed_folders\", new=lambda self: setattr(self, \"deployed_folders\", {})\n ),\n patch.object(FabricWorkspace, \"_refresh_repository_items\", new=lambda _: None),\n patch.object(FabricWorkspace, \"_refresh_repository_folders\", new=lambda _: None),\n ):\n workspace = FabricWorkspace(\n workspace_id=\"12345678-1234-5678-abcd-1234567890ab\",\n repository_directory=str(temp_path),\n item_type_in_scope=[\"Notebook\"],\n token_credential=DummyTokenCredential(),\n )\n yield workspace\n\n\n@pytest.fixture(autouse=True)\ndef _clear_feature_flags():\n \"\"\"Clear feature flags before and after each test to avoid state leakage.\"\"\"\n constants.FEATURE_FLAG.discard(FeatureFlag.ENABLE_HARD_DELETE.value)\n yield\n constants.FEATURE_FLAG.discard(FeatureFlag.ENABLE_HARD_DELETE.value)\n\n\ndef test_unpublish_item_without_hard_delete_flag(test_workspace, mock_endpoint):\n \"\"\"Test that _unpublish_item uses a plain DELETE URL when flag is not set.\"\"\"\n item_guid = \"mock-guid-123\"\n test_workspace.deployed_items = {\"Notebook\": {\"TestNotebook\": MagicMock(guid=item_guid)}}\n\n mock_endpoint.delete_urls.clear()\n\n test_workspace._unpublish_item(item_name=\"TestNotebook\", item_type=\"Notebook\")\n\n assert len(mock_endpoint.delete_urls) == 1\n delete_url = mock_endpoint.delete_urls[0]\n assert delete_url == f\"{test_workspace.base_api_url}/items/{item_guid}\"\n assert \"hardDelete=true\" not in delete_url\n\n\ndef test_unpublish_item_with_hard_delete_flag(test_workspace, mock_endpoint):\n \"\"\"Test that _unpublish_item appends ?hardDelete=True when flag is set.\"\"\"\n item_guid = \"mock-guid-456\"\n test_workspace.deployed_items = {\"Notebook\": {\"TestNotebook\": MagicMock(guid=item_guid)}}\n\n constants.FEATURE_FLAG.add(FeatureFlag.ENABLE_HARD_DELETE.value)\n mock_endpoint.delete_urls.clear()\n\n test_workspace._unpublish_item(item_name=\"TestNotebook\", item_type=\"Notebook\")\n\n assert len(mock_endpoint.delete_urls) == 1\n delete_url = mock_endpoint.delete_urls[0]\n assert delete_url == f\"{test_workspace.base_api_url}/items/{item_guid}?hardDelete=true\"\n\n\ndef test_hard_delete_flag_via_append_feature_flag(test_workspace, mock_endpoint):\n \"\"\"Test that enable_hard_delete works when set via append_feature_flag.\"\"\"\n from fabric_cicd import append_feature_flag\n\n item_guid = \"mock-guid-789\"\n test_workspace.deployed_items = {\"Notebook\": {\"TestNotebook\": MagicMock(guid=item_guid)}}\n\n append_feature_flag(FeatureFlag.ENABLE_HARD_DELETE.value)\n mock_endpoint.delete_urls.clear()\n\n test_workspace._unpublish_item(item_name=\"TestNotebook\", item_type=\"Notebook\")\n\n assert len(mock_endpoint.delete_urls) == 1\n assert \"hardDelete=true\" in mock_endpoint.delete_urls[0]\n" }, { "path": "tests/test_integration_publish.py", "content": "# Copyright (c) Microsoft Corporation.\n# Licensed under the MIT License.\n\n\"\"\"Integration test for publish operations using mock Fabric API server.\"\"\"\n\nimport gzip\nimport importlib\nimport os\nimport shutil\nfrom pathlib import Path\nfrom urllib.parse import urlparse\n\nimport pytest\nfrom fixtures.credentials import DummyTokenCredential\nfrom fixtures.mock_fabric_server import MOCK_SERVER_PORT, MockFabricServer\n\nimport fabric_cicd\nimport fabric_cicd._common._validate_env_vars as validate_env_vars\nimport fabric_cicd.constants\n\n\n@pytest.fixture\ndef allow_localhost_http_for_integration(monkeypatch: pytest.MonkeyPatch):\n \"\"\"\n Test-only override: allow http://localhost for mocked integration servers.\n \"\"\"\n real_validate = validate_env_vars.validate_env_var_api_url\n\n def _validate_api_url_test(env_var_name: str, default_value: str) -> str:\n value = os.environ.get(env_var_name, default_value)\n parsed = urlparse(value)\n host = (parsed.hostname or \"\").lower()\n\n if parsed.scheme == \"http\" and host in {\"localhost\", \"127.0.0.1\", \"::1\"}:\n return value.rstrip(\"/\")\n\n return real_validate(env_var_name, default_value)\n\n monkeypatch.setattr(validate_env_vars, \"validate_env_var_api_url\", _validate_api_url_test)\n return\n\n\n@pytest.fixture\ndef mock_fabric_api_server(allow_localhost_http_for_integration): # noqa: ARG001\n \"\"\"\n Start mock Fabric API server for the test.\n\n Yields the server and sets environment variables for API URLs.\n \"\"\"\n tests_dir = Path(__file__).parent\n trace_file_gz = tests_dir / \"fixtures\" / MockFabricServer.HTTP_TRACE_FILE\n trace_file = trace_file_gz.with_suffix(\"\")\n\n if not trace_file_gz.exists():\n pytest.skip(\n \"http_trace.json.gz not found - run devtools/debug_trace_deployment.py first to generate trace data\"\n )\n\n if trace_file.exists():\n trace_file.unlink()\n with gzip.open(trace_file_gz, \"rb\") as f_in, trace_file.open(\"wb\") as f_out:\n shutil.copyfileobj(f_in, f_out)\n\n server = MockFabricServer(trace_file, port=MOCK_SERVER_PORT)\n\n original_default_api = os.environ.get(\"DEFAULT_API_ROOT_URL\")\n original_fabric_api = os.environ.get(\"FABRIC_API_ROOT_URL\")\n original_retry_delay = os.environ.get(\"FABRIC_CICD_RETRY_DELAY_OVERRIDE_SECONDS\")\n\n os.environ[\"DEFAULT_API_ROOT_URL\"] = f\"http://127.0.0.1:{MOCK_SERVER_PORT}\"\n os.environ[\"FABRIC_API_ROOT_URL\"] = f\"http://127.0.0.1:{MOCK_SERVER_PORT}\"\n os.environ[\"FABRIC_CICD_RETRY_DELAY_OVERRIDE_SECONDS\"] = \"0\"\n\n # reload only after env is set and override fixture is active\n importlib.reload(fabric_cicd.constants)\n\n server.start()\n\n yield server\n\n server.stop()\n\n if original_default_api is not None:\n os.environ[\"DEFAULT_API_ROOT_URL\"] = original_default_api\n else:\n os.environ.pop(\"DEFAULT_API_ROOT_URL\", None)\n\n if original_fabric_api is not None:\n os.environ[\"FABRIC_API_ROOT_URL\"] = original_fabric_api\n else:\n os.environ.pop(\"FABRIC_API_ROOT_URL\", None)\n\n if original_retry_delay is not None:\n os.environ[\"FABRIC_CICD_RETRY_DELAY_OVERRIDE_SECONDS\"] = original_retry_delay\n else:\n os.environ.pop(\"FABRIC_CICD_RETRY_DELAY_OVERRIDE_SECONDS\", None)\n\n importlib.reload(fabric_cicd.constants)\n\n\ndef test_publish_all_items_integration(mock_fabric_api_server): # noqa: ARG001\n \"\"\"Test full publish_all_items workflow using mocked API responses.\"\"\"\n workspace_id = \"00000000-0000-0000-0000-000000000000\"\n environment_key = \"PPE\"\n\n root_directory = Path(__file__).resolve().parent.parent\n artifacts_folder = root_directory / \"sample\" / \"workspace\"\n\n item_types_to_deploy = [\n \"Dataflow\",\n \"DataPipeline\",\n \"Environment\",\n \"Eventhouse\",\n \"Eventstream\",\n \"KQLDatabase\",\n \"KQLQueryset\",\n \"Lakehouse\",\n \"MirroredDatabase\",\n \"MLExperiment\",\n \"Notebook\",\n \"Ontology\",\n \"Reflex\",\n \"Report\",\n \"SemanticModel\",\n \"SparkJobDefinition\",\n \"SQLDatabase\",\n \"VariableLibrary\",\n \"Warehouse\",\n ]\n\n token_credential = DummyTokenCredential()\n\n for flag in [\"enable_shortcut_publish\", \"continue_on_shortcut_failure\"]:\n fabric_cicd.append_feature_flag(flag)\n\n target_workspace = fabric_cicd.FabricWorkspace(\n workspace_id=workspace_id,\n environment=environment_key,\n repository_directory=str(artifacts_folder),\n item_type_in_scope=item_types_to_deploy,\n token_credential=token_credential,\n )\n\n fabric_cicd.publish_all_items(target_workspace)\n\n assert True, \"Publish completed successfully\"\n" }, { "path": "tests/test_logging.py", "content": "# Copyright (c) Microsoft Corporation.\n# Licensed under the MIT License.\n\n\"\"\"Tests for the logging module and wrapper functions.\"\"\"\n\nimport logging\nimport shutil\nimport tempfile\nfrom logging.handlers import RotatingFileHandler\nfrom pathlib import Path\nfrom unittest.mock import patch\n\nimport pytest\n\nfrom fabric_cicd import (\n append_feature_flag,\n change_log_level,\n configure_external_file_logging,\n constants,\n disable_file_logging,\n)\nfrom fabric_cicd._common._logging import (\n CustomFormatter,\n PackageFilter,\n _build_console_message,\n _build_file_message,\n _cleanup_managed_handlers,\n _configure_console_handler,\n _configure_default_file_handler,\n _configure_external_file_handler,\n _mark_external_handler,\n _mark_handler,\n configure_logger,\n exception_handler,\n get_file_handler,\n log_header,\n)\n\n\ndef _close_all_file_handlers():\n \"\"\"Close all file handlers to release file locks on Windows.\"\"\"\n for logger_name in (\"\", \"fabric_cicd\"):\n logger = logging.getLogger(logger_name)\n for handler in logger.handlers[:]:\n if isinstance(handler, (logging.FileHandler, RotatingFileHandler)):\n handler.close()\n logger.removeHandler(handler)\n\n\ndef _reset_logger(logger_name: str) -> None:\n \"\"\"Reset a logger to clean state.\"\"\"\n logger = logging.getLogger(logger_name)\n logger.handlers = []\n\n\n@pytest.fixture(autouse=True)\ndef _clean_logging_state():\n \"\"\"Reset logging state before and after each test to release file locks on Windows.\"\"\"\n _close_all_file_handlers()\n for logger_name in (\"\", \"fabric_cicd\", \"console_only\"):\n _reset_logger(logger_name)\n\n yield\n\n _close_all_file_handlers()\n\n\n@pytest.fixture\ndef temp_log_dir():\n \"\"\"Create a temporary directory for log files.\"\"\"\n tmpdir = Path(tempfile.mkdtemp())\n yield tmpdir\n _close_all_file_handlers()\n shutil.rmtree(tmpdir, ignore_errors=True)\n\n\n@pytest.fixture\ndef external_rotating_handler(temp_log_dir):\n \"\"\"Create an external RotatingFileHandler for testing.\"\"\"\n log_file = temp_log_dir / \"external.log\"\n handler = RotatingFileHandler(str(log_file), maxBytes=1024, backupCount=1)\n handler.setFormatter(logging.Formatter(\"%(asctime)s - %(levelname)s - %(message)s\"))\n yield handler\n handler.close()\n\n\n@pytest.fixture\ndef external_logger_with_handler(temp_log_dir):\n \"\"\"Create an external logger with a RotatingFileHandler attached.\"\"\"\n log_file = temp_log_dir / \"external.log\"\n handler = RotatingFileHandler(str(log_file), maxBytes=1024, backupCount=1)\n handler.setFormatter(logging.Formatter(\"%(message)s\"))\n\n logger = logging.getLogger(f\"ExternalLogger_{id(handler)}\")\n logger.addHandler(handler)\n logger.setLevel(logging.DEBUG)\n\n yield logger, handler, log_file\n\n handler.close()\n logger.removeHandler(handler)\n\n\nclass TestCustomFormatter:\n \"\"\"Tests for the CustomFormatter class.\"\"\"\n\n @pytest.mark.parametrize(\n (\"level\", \"level_name\", \"message\"),\n [\n (logging.DEBUG, \"debug\", \"Debug message\"),\n (logging.INFO, \"info\", \"Info message\"),\n (logging.WARNING, \"warn\", \"Warning message\"),\n (logging.ERROR, \"error\", \"Error message\"),\n (logging.CRITICAL, \"crit\", \"Critical message\"),\n ],\n )\n def test_format_levels(self, level, level_name, message):\n \"\"\"Test formatting of various log levels.\"\"\"\n formatter = CustomFormatter(\"[%(levelname)s] %(asctime)s - %(message)s\", datefmt=\"%H:%M:%S\")\n record = logging.LogRecord(\n name=\"fabric_cicd\",\n level=level,\n pathname=\"\",\n lineno=0,\n msg=message,\n args=(),\n exc_info=None,\n )\n formatted = formatter.format(record)\n assert level_name in formatted.lower()\n assert message in formatted\n\n def test_format_with_indent(self):\n \"\"\"Test formatting of messages with indent marker.\"\"\"\n\n formatter = CustomFormatter(\"[%(levelname)s] %(asctime)s - %(message)s\", datefmt=\"%H:%M:%S\")\n record = logging.LogRecord(\n name=\"fabric_cicd\",\n level=logging.INFO,\n pathname=\"\",\n lineno=0,\n msg=f\"{constants.INDENT}Indented message\",\n args=(),\n exc_info=None,\n )\n formatted = formatter.format(record)\n assert \"Indented message\" in formatted\n assert formatted.startswith(\" \" * 8)\n\n\nclass TestPackageFilter:\n \"\"\"Tests for the PackageFilter class.\"\"\"\n\n @pytest.mark.parametrize(\n (\"logger_name\", \"expected\"),\n [\n (\"fabric_cicd\", True),\n (\"fabric_cicd.publish\", True),\n (\"fabric_cicd._common._logging\", True),\n (\"azure.identity\", False),\n (\"urllib3.connectionpool\", False),\n (\"other_package\", False),\n ],\n )\n def test_namespace_filtering(self, logger_name, expected):\n \"\"\"Test filter correctly handles fabric_cicd and third-party namespaces.\"\"\"\n filter_instance = PackageFilter()\n record = logging.LogRecord(\n name=logger_name, level=logging.INFO, pathname=\"\", lineno=0, msg=\"test\", args=(), exc_info=None\n )\n assert filter_instance.filter(record) is expected\n\n @pytest.mark.parametrize(\n (\"level\", \"expected\"),\n [\n (logging.DEBUG, True),\n (logging.INFO, False),\n (logging.WARNING, False),\n (logging.ERROR, False),\n (logging.CRITICAL, False),\n ],\n )\n def test_debug_only_mode(self, level, expected):\n \"\"\"Test debug_only=True only allows DEBUG level from fabric_cicd.\"\"\"\n filter_instance = PackageFilter(debug_only=True)\n record = logging.LogRecord(\n name=\"fabric_cicd\", level=level, pathname=\"\", lineno=0, msg=\"test\", args=(), exc_info=None\n )\n assert filter_instance.filter(record) is expected\n\n def test_debug_only_still_checks_namespace(self):\n \"\"\"Test debug_only=True still blocks non-fabric_cicd DEBUG logs.\"\"\"\n filter_instance = PackageFilter(debug_only=True)\n record = logging.LogRecord(\n name=\"azure.identity\", level=logging.DEBUG, pathname=\"\", lineno=0, msg=\"debug\", args=(), exc_info=None\n )\n assert filter_instance.filter(record) is False\n\n def test_default_allows_all_levels_from_package(self):\n \"\"\"Test default filter (debug_only=False) allows all levels from fabric_cicd.\"\"\"\n filter_instance = PackageFilter(debug_only=False)\n levels = [logging.DEBUG, logging.INFO, logging.WARNING, logging.ERROR, logging.CRITICAL]\n for level in levels:\n record = logging.LogRecord(\n name=\"fabric_cicd\", level=level, pathname=\"\", lineno=0, msg=\"test\", args=(), exc_info=None\n )\n assert filter_instance.filter(record) is True\n\n\nclass TestMarkHandler:\n \"\"\"Tests for the _mark_handler and _mark_external_handler functions.\"\"\"\n\n def test_mark_handler(self):\n \"\"\"Test that _mark_handler sets attribute and returns same handler.\"\"\"\n handler = logging.StreamHandler()\n marked = _mark_handler(handler)\n assert getattr(marked, \"_fabric_cicd_managed\", False) is True\n assert marked is handler\n\n def test_mark_external_handler(self):\n \"\"\"Test that _mark_external_handler sets attribute and returns same handler.\"\"\"\n handler = logging.StreamHandler()\n marked = _mark_external_handler(handler)\n assert getattr(marked, \"_fabric_cicd_external\", False) is True\n assert getattr(marked, \"_fabric_cicd_managed\", False) is False\n assert marked is handler\n\n\nclass TestCleanupManagedHandlers:\n \"\"\"Tests for the _cleanup_managed_handlers function.\"\"\"\n\n def test_removes_managed_preserves_external(self):\n \"\"\"Test that managed handlers are removed while external are preserved.\"\"\"\n logger = logging.getLogger(\"test_cleanup\")\n external_handler = logging.StreamHandler()\n managed_handler = _mark_handler(logging.StreamHandler())\n logger.addHandler(external_handler)\n logger.addHandler(managed_handler)\n\n _cleanup_managed_handlers(logger)\n\n assert external_handler in logger.handlers\n assert managed_handler not in logger.handlers\n\n logger.removeHandler(external_handler)\n\n def test_cleanup_multiple_loggers(self):\n \"\"\"Test cleanup across multiple loggers.\"\"\"\n logger_a = logging.getLogger(\"test_cleanup_a\")\n logger_b = logging.getLogger(\"test_cleanup_b\")\n handler_a = _mark_handler(logging.StreamHandler())\n handler_b = _mark_handler(logging.StreamHandler())\n logger_a.addHandler(handler_a)\n logger_b.addHandler(handler_b)\n\n _cleanup_managed_handlers(logger_a, logger_b)\n\n assert handler_a not in logger_a.handlers\n assert handler_b not in logger_b.handlers\n\n logger_a.handlers = []\n logger_b.handlers = []\n\n def test_cleanup_external_handler_removes_filters(self, temp_log_dir):\n \"\"\"Test cleanup removes PackageFilter from external handlers.\"\"\"\n log_file = temp_log_dir / \"external.log\"\n external_handler = RotatingFileHandler(str(log_file), maxBytes=1024, backupCount=1)\n\n try:\n _mark_external_handler(external_handler)\n external_handler.addFilter(PackageFilter(debug_only=True))\n\n root_logger = logging.getLogger()\n root_logger.addHandler(external_handler)\n\n assert len(external_handler.filters) == 1\n assert isinstance(external_handler.filters[0], PackageFilter)\n\n _cleanup_managed_handlers(root_logger)\n\n assert external_handler not in root_logger.handlers\n assert len(external_handler.filters) == 0\n assert getattr(external_handler, \"_fabric_cicd_external\", False) is False\n\n finally:\n external_handler.close()\n\n\nclass TestConfigureDefaultFileHandler:\n \"\"\"Tests for the _configure_default_file_handler function.\"\"\"\n\n def test_default_file_handler_configuration(self):\n \"\"\"Test default file handler has correct configuration.\"\"\"\n handler = _configure_default_file_handler()\n try:\n assert isinstance(handler, logging.FileHandler)\n assert not isinstance(handler, RotatingFileHandler)\n assert getattr(handler, \"_fabric_cicd_managed\", False) is True\n assert handler.baseFilename.endswith(\"fabric_cicd.error.log\")\n assert handler.mode == \"w\"\n assert handler.stream is None # delay=True\n assert len(handler.filters) == 1\n assert isinstance(handler.filters[0], PackageFilter)\n assert handler.filters[0].debug_only is False\n assert handler.formatter is not None\n finally:\n handler.close()\n\n\nclass TestConfigureExternalFileHandler:\n \"\"\"Tests for the _configure_external_file_handler function.\"\"\"\n\n def test_reuses_handler_directly(self, temp_log_dir):\n \"\"\"Test external file handler is reused directly (preserving rotation).\"\"\"\n log_file = temp_log_dir / \"external.log\"\n external_handler = RotatingFileHandler(str(log_file), maxBytes=1024, backupCount=1)\n custom_formatter = logging.Formatter(\"CUSTOM - %(message)s\")\n external_handler.setFormatter(custom_formatter)\n\n try:\n handler = _configure_external_file_handler(external_handler, logging.DEBUG, debug_only_file=True)\n\n assert handler is external_handler\n assert isinstance(handler, RotatingFileHandler)\n assert handler.baseFilename == str(log_file)\n assert getattr(handler, \"_fabric_cicd_managed\", False) is False\n assert getattr(handler, \"_fabric_cicd_external\", False) is True\n assert handler.formatter is custom_formatter\n assert len(handler.filters) == 1\n assert isinstance(handler.filters[0], PackageFilter)\n assert handler.filters[0].debug_only is True\n finally:\n external_handler.close()\n\n def test_preserves_caller_formatter(self, temp_log_dir):\n \"\"\"Test external file handler preserves caller's formatter.\"\"\"\n log_file = temp_log_dir / \"external.log\"\n custom_formatter = logging.Formatter(\"CUSTOM - %(levelname)s - %(message)s\")\n external_handler = RotatingFileHandler(str(log_file), maxBytes=1024, backupCount=1)\n external_handler.setFormatter(custom_formatter)\n\n try:\n handler = _configure_external_file_handler(external_handler, logging.DEBUG, debug_only_file=True)\n\n # Verify formatter is preserved by checking it formats correctly\n record = logging.LogRecord(\n name=\"fabric_cicd\", level=logging.DEBUG, pathname=\"\", lineno=0, msg=\"test\", args=(), exc_info=None\n )\n formatted = handler.formatter.format(record)\n assert formatted.startswith(\"CUSTOM - DEBUG - test\")\n finally:\n external_handler.close()\n\n def test_info_level_ignores_debug_only(self, temp_log_dir):\n \"\"\"Test external file handler at INFO level ignores debug_only_file flag.\"\"\"\n log_file = temp_log_dir / \"external.log\"\n external_handler = RotatingFileHandler(str(log_file), maxBytes=1024, backupCount=1)\n\n try:\n handler = _configure_external_file_handler(external_handler, logging.INFO, debug_only_file=True)\n assert handler.filters[0].debug_only is False\n finally:\n external_handler.close()\n\n def test_works_with_regular_file_handler(self, temp_log_dir):\n \"\"\"Test external file handler works with regular FileHandler.\"\"\"\n log_file = temp_log_dir / \"external.log\"\n external_handler = logging.FileHandler(str(log_file))\n\n try:\n handler = _configure_external_file_handler(external_handler, logging.DEBUG, debug_only_file=False)\n\n assert handler is external_handler\n assert isinstance(handler, logging.FileHandler)\n assert not isinstance(handler, RotatingFileHandler)\n assert len(handler.filters) == 1\n assert handler.filters[0].debug_only is False\n finally:\n external_handler.close()\n\n\nclass TestConfigureConsoleHandler:\n \"\"\"Tests for the _configure_console_handler function.\"\"\"\n\n def test_console_handler_configuration(self):\n \"\"\"Test console handler has correct configuration.\"\"\"\n handler = _configure_console_handler(logging.WARNING)\n assert isinstance(handler, logging.StreamHandler)\n assert handler.level == logging.WARNING\n assert getattr(handler, \"_fabric_cicd_managed\", False) is True\n assert isinstance(handler.formatter, CustomFormatter)\n\n\nclass TestGetFileHandler:\n \"\"\"Tests for the get_file_handler function.\"\"\"\n\n def test_returns_none_when_no_file_handler(self):\n \"\"\"Test returns None when no file handler exists.\"\"\"\n assert get_file_handler() is None\n\n def test_returns_managed_file_handler_from_root(self):\n \"\"\"Test returns the managed file handler from root logger.\"\"\"\n root_logger = logging.getLogger()\n handler = _mark_handler(logging.FileHandler(\"test_get.log\", delay=True))\n root_logger.addHandler(handler)\n\n try:\n result = get_file_handler()\n assert result is handler\n finally:\n handler.close()\n root_logger.removeHandler(handler)\n\n def test_ignores_unmanaged_file_handler_on_root(self):\n \"\"\"Test ignores file handlers not marked as managed on root logger.\"\"\"\n root_logger = logging.getLogger()\n handler = logging.FileHandler(\"test_unmanaged.log\", delay=True)\n root_logger.addHandler(handler)\n\n try:\n assert get_file_handler() is None\n finally:\n handler.close()\n root_logger.removeHandler(handler)\n\n def test_ignores_external_file_handler_on_root(self):\n \"\"\"Test ignores file handlers marked as external on root logger.\"\"\"\n root_logger = logging.getLogger()\n handler = _mark_external_handler(logging.FileHandler(\"test_external.log\", delay=True))\n root_logger.addHandler(handler)\n\n try:\n assert get_file_handler() is None\n finally:\n handler.close()\n root_logger.removeHandler(handler)\n\n def test_returns_any_file_handler_from_provided_logger(self):\n \"\"\"Test returns any file handler from provided logger.\"\"\"\n external_logger = logging.getLogger(\"external_test\")\n handler = logging.FileHandler(\"test_external.log\", delay=True)\n external_logger.addHandler(handler)\n\n try:\n result = get_file_handler(external_logger)\n assert result is handler\n finally:\n handler.close()\n external_logger.removeHandler(handler)\n\n\nclass TestBuildConsoleMessage:\n \"\"\"Tests for the _build_console_message function.\"\"\"\n\n def test_no_file_handler(self):\n \"\"\"Test message without file handler reference.\"\"\"\n exception = Exception(\"Something failed\")\n result = _build_console_message(exception, None)\n assert result == \"Something failed\"\n\n def test_with_default_file_handler(self):\n \"\"\"Test message includes file path for default FileHandler.\"\"\"\n handler = logging.FileHandler(\"fabric_cicd.error.log\", delay=True)\n try:\n exception = Exception(\"Something failed\")\n result = _build_console_message(exception, handler)\n assert \"Something failed\" in result\n assert \"See\" in result\n assert \"fabric_cicd.error.log\" in result\n finally:\n handler.close()\n\n def test_with_non_default_file_handler(self, temp_log_dir):\n \"\"\"Test message excludes file path for non-default file handlers.\"\"\"\n log_file = temp_log_dir / \"program.log\"\n handler = logging.FileHandler(str(log_file), delay=True)\n\n try:\n exception = Exception(\"Something failed\")\n result = _build_console_message(exception, handler)\n assert result == \"Something failed\"\n assert \"See\" not in result\n finally:\n handler.close()\n\n\nclass TestBuildFileMessage:\n \"\"\"Tests for the _build_file_message function.\"\"\"\n\n @pytest.mark.parametrize(\n (\"additional_info\", \"expected_in_result\"),\n [\n (None, False),\n (\"status: 403\", True),\n ],\n )\n def test_file_message(self, additional_info, expected_in_result):\n \"\"\"Test file message with and without additional info.\"\"\"\n exception = Exception(\"Something failed\")\n if additional_info is not None:\n exception.additional_info = additional_info\n\n result = _build_file_message(exception)\n assert \"%s\" in result\n if expected_in_result:\n assert \"Additional Info\" in result\n assert additional_info in result\n else:\n assert result == \"%s\"\n\n\nclass TestConfigureLogger:\n \"\"\"Tests for the configure_logger function.\"\"\"\n\n @pytest.mark.parametrize(\n (\"level\", \"expected_package_level\", \"expected_root_level\"),\n [\n (logging.INFO, logging.INFO, logging.ERROR),\n (logging.DEBUG, logging.DEBUG, logging.INFO),\n ],\n )\n def test_logger_levels(self, level, expected_package_level, expected_root_level):\n \"\"\"Test logger level configuration.\"\"\"\n configure_logger(level=level, disable_log_file=True)\n\n assert logging.getLogger(\"fabric_cicd\").level == expected_package_level\n assert logging.getLogger().level == expected_root_level\n\n def test_default_includes_file_handler(self):\n \"\"\"Test default configuration includes file handler.\"\"\"\n configure_logger()\n root_logger = logging.getLogger()\n file_handlers = [h for h in root_logger.handlers if isinstance(h, logging.FileHandler)]\n assert len(file_handlers) == 1\n\n def test_disable_file_logging(self):\n \"\"\"Test file logging can be disabled.\"\"\"\n configure_logger(disable_log_file=True)\n root_logger = logging.getLogger()\n file_handlers = [h for h in root_logger.handlers if isinstance(h, logging.FileHandler)]\n assert len(file_handlers) == 0\n\n def test_with_external_file_handler(self, external_rotating_handler, temp_log_dir):\n \"\"\"Test configuration with external file handler.\"\"\"\n log_file = temp_log_dir / \"external.log\"\n\n configure_logger(\n level=logging.DEBUG,\n external_file_handler=external_rotating_handler,\n suppress_debug_console=True,\n debug_only_file=True,\n )\n\n root_logger = logging.getLogger()\n external_handlers = [\n h\n for h in root_logger.handlers\n if isinstance(h, logging.FileHandler) and getattr(h, \"_fabric_cicd_external\", False)\n ]\n assert len(external_handlers) == 1\n assert external_handlers[0] is external_rotating_handler\n assert external_handlers[0].baseFilename == str(log_file)\n assert isinstance(external_handlers[0], RotatingFileHandler)\n\n def test_suppress_debug_console(self):\n \"\"\"Test suppressing DEBUG output to console.\"\"\"\n configure_logger(level=logging.DEBUG, suppress_debug_console=True, disable_log_file=True)\n\n package_logger = logging.getLogger(\"fabric_cicd\")\n console_handlers = [h for h in package_logger.handlers if isinstance(h, logging.StreamHandler)]\n assert len(console_handlers) == 1\n assert console_handlers[0].level == logging.INFO\n\n def test_console_only_logger_configured(self):\n \"\"\"Test console_only logger is properly configured.\"\"\"\n configure_logger(disable_log_file=True)\n\n console_only_logger = logging.getLogger(\"console_only\")\n package_logger = logging.getLogger(\"fabric_cicd\")\n\n assert console_only_logger.propagate is False\n assert len(console_only_logger.handlers) == 1\n assert package_logger.handlers[0] is not console_only_logger.handlers[0]\n\n def test_preserves_unmanaged_handlers(self):\n \"\"\"Test that unmanaged handlers survive reconfiguration.\"\"\"\n root_logger = logging.getLogger()\n external_handler = logging.StreamHandler()\n root_logger.addHandler(external_handler)\n\n configure_logger(disable_log_file=True)\n configure_logger(disable_log_file=True)\n\n assert external_handler in root_logger.handlers\n root_logger.removeHandler(external_handler)\n\n def test_package_logger_propagates(self):\n \"\"\"Test that package logger propagates to root.\"\"\"\n configure_logger(disable_log_file=True)\n assert logging.getLogger(\"fabric_cicd\").propagate is True\n\n\nclass TestLogHeader:\n \"\"\"Tests for the log_header function.\"\"\"\n\n def test_logs_expected_messages(self, caplog):\n \"\"\"Test log_header logs the expected messages.\"\"\"\n logger = logging.getLogger(\"fabric_cicd.test\")\n logger.setLevel(logging.INFO)\n\n with caplog.at_level(logging.INFO, logger=\"fabric_cicd.test\"):\n log_header(logger, \"Test Header\")\n\n assert len(caplog.records) >= 3\n assert any(\"Test Header\" in record.message for record in caplog.records)\n\n\nclass TestWrapperFunctions:\n \"\"\"Tests for the wrapper functions in __init__.py.\"\"\"\n\n @pytest.fixture(autouse=True)\n def _clear_feature_flags(self):\n \"\"\"Clear feature flags before each wrapper test.\"\"\"\n\n constants.FEATURE_FLAG.clear()\n\n def test_append_feature_flag(self):\n \"\"\"Test append_feature_flag adds flags correctly.\"\"\"\n append_feature_flag(\"feature_1\")\n append_feature_flag(\"feature_2\")\n append_feature_flag(\"feature_1\") # Duplicate\n\n assert \"feature_1\" in constants.FEATURE_FLAG\n assert \"feature_2\" in constants.FEATURE_FLAG\n assert len([f for f in constants.FEATURE_FLAG if f == \"feature_1\"]) == 1\n\n @pytest.mark.parametrize(\"level_input\", [\"DEBUG\", \"debug\"])\n def test_change_log_level(self, level_input):\n \"\"\"Test change_log_level sets level correctly.\"\"\"\n change_log_level(level_input)\n assert logging.getLogger(\"fabric_cicd\").level == logging.DEBUG\n\n def test_change_log_level_unsupported(self, capsys):\n \"\"\"Test change_log_level warns on unsupported level.\"\"\"\n configure_logger(disable_log_file=True)\n change_log_level(\"TRACE\")\n\n captured = capsys.readouterr()\n assert \"not supported\" in captured.err\n\n def test_disable_file_logging(self):\n \"\"\"Test disable_file_logging removes file handlers.\"\"\"\n\n configure_logger()\n disable_file_logging()\n\n root_logger = logging.getLogger()\n file_handlers = [h for h in root_logger.handlers if isinstance(h, logging.FileHandler)]\n assert len(file_handlers) == 0\n\n\nclass TestConfigureExternalFileLogging:\n \"\"\"Tests for the configure_external_file_logging wrapper function.\"\"\"\n\n def test_configures_correctly(self, external_logger_with_handler):\n \"\"\"Test that configure_external_file_logging configures correctly.\"\"\"\n external_logger, external_handler, log_file = external_logger_with_handler\n\n configure_external_file_logging(external_logger)\n\n package_logger = logging.getLogger(\"fabric_cicd\")\n assert package_logger.level == logging.DEBUG\n\n console_handlers = [\n h\n for h in package_logger.handlers\n if isinstance(h, logging.StreamHandler) and not isinstance(h, logging.FileHandler)\n ]\n assert len(console_handlers) == 1\n assert console_handlers[0].level == logging.INFO\n\n root_logger = logging.getLogger()\n external_handlers = [\n h\n for h in root_logger.handlers\n if isinstance(h, logging.FileHandler) and getattr(h, \"_fabric_cicd_external\", False)\n ]\n assert len(external_handlers) == 1\n assert external_handlers[0] is external_handler\n assert external_handlers[0].baseFilename == str(log_file)\n\n def test_raises_without_handler(self):\n \"\"\"Test that configure_external_file_logging raises ValueError if no file handler.\"\"\"\n external_logger = logging.getLogger(\"NoFileHandler\")\n external_logger.handlers = []\n\n with pytest.raises(ValueError, match=\"No FileHandler or RotatingFileHandler found\"):\n configure_external_file_logging(external_logger)\n\n def test_writes_only_debug_logs(self, external_logger_with_handler):\n \"\"\"Test that only DEBUG logs from fabric_cicd are written to external file.\"\"\"\n external_logger, _external_handler, log_file = external_logger_with_handler\n\n configure_external_file_logging(external_logger)\n\n fabric_logger = logging.getLogger(\"fabric_cicd\")\n fabric_logger.debug(\"Debug message\")\n fabric_logger.info(\"Info message\")\n\n azure_logger = logging.getLogger(\"azure.identity\")\n azure_logger.setLevel(logging.DEBUG)\n azure_logger.debug(\"Azure debug\")\n\n for handler in logging.getLogger().handlers:\n if hasattr(handler, \"flush\"):\n handler.flush()\n\n content = log_file.read_text(encoding=\"utf-8\")\n assert \"Debug message\" in content\n assert \"Info message\" not in content\n assert \"Azure debug\" not in content\n\n\nclass TestExceptionHandler:\n \"\"\"Tests for the exception_handler function.\"\"\"\n\n def test_handles_custom_exception(self):\n \"\"\"Test exception handler handles custom exceptions.\"\"\"\n from fabric_cicd._common._exceptions import InputError\n\n test_logger = logging.getLogger(\"fabric_cicd.test\")\n exception = InputError(\"Test error message\", logger=test_logger)\n configure_logger(disable_log_file=True)\n\n try:\n exception_handler(InputError, exception, None)\n except Exception:\n pytest.fail(\"exception_handler raised an unexpected exception\")\n\n def test_falls_back_for_standard_exception(self):\n \"\"\"Test exception handler falls back to default for standard exceptions.\"\"\"\n with patch(\"sys.__excepthook__\") as mock_excepthook:\n exception = ValueError(\"Standard error\")\n exception_handler(ValueError, exception, None)\n mock_excepthook.assert_called_once()\n\n def test_writes_to_console_only_logger(self):\n \"\"\"Test that exception handler writes to console_only logger.\"\"\"\n from fabric_cicd._common._exceptions import InputError\n\n configure_logger(disable_log_file=True)\n test_logger = logging.getLogger(\"fabric_cicd.test\")\n exception = InputError(\"User-facing error\", logger=test_logger)\n\n with patch.object(logging.getLogger(\"console_only\"), \"error\") as mock_error:\n exception_handler(InputError, exception, None)\n mock_error.assert_called_once()\n message = mock_error.call_args[0][0]\n assert \"User-facing error\" in message\n assert \"See\" not in message\n\n def test_removes_console_handler_when_using_default_file(self):\n \"\"\"Test that exception handler removes console handler when using default file handler.\"\"\"\n from fabric_cicd._common._exceptions import InputError\n\n configure_logger()\n test_logger = logging.getLogger(\"fabric_cicd.test\")\n exception = InputError(\"Test error\", logger=test_logger)\n\n package_logger = logging.getLogger(\"fabric_cicd\")\n assert len(package_logger.handlers) >= 1\n\n exception_handler(InputError, exception, None)\n\n managed_handlers = [h for h in package_logger.handlers if getattr(h, \"_fabric_cicd_managed\", False)]\n assert len(managed_handlers) == 0\n\n\nclass TestFileLoggingIntegration:\n \"\"\"Integration tests for file logging functionality.\"\"\"\n\n def test_default_file_handler_writes_logs(self, temp_log_dir):\n \"\"\"Test that default file handler actually writes logs.\"\"\"\n import os\n\n original_cwd = Path.cwd()\n\n try:\n os.chdir(temp_log_dir)\n configure_logger()\n\n logger = logging.getLogger(\"fabric_cicd\")\n logger.error(\"Error message for test\")\n\n for handler in logging.getLogger().handlers:\n if hasattr(handler, \"flush\"):\n handler.flush()\n\n log_file = temp_log_dir / \"fabric_cicd.error.log\"\n assert log_file.exists()\n content = log_file.read_text(encoding=\"utf-8\")\n assert \"Error message for test\" in content\n\n finally:\n os.chdir(original_cwd)\n\n def test_file_not_created_until_log_written(self, temp_log_dir):\n \"\"\"Test that log file is not created until first log is written (delay=True).\"\"\"\n import os\n\n original_cwd = Path.cwd()\n\n try:\n os.chdir(temp_log_dir)\n configure_logger()\n\n log_file = temp_log_dir / \"fabric_cicd.error.log\"\n assert not log_file.exists()\n\n finally:\n os.chdir(original_cwd)\n\n def test_external_handler_writes_fabric_cicd_logs(self, external_logger_with_handler):\n \"\"\"Test that external handler writes fabric_cicd logs to the shared file.\"\"\"\n external_logger, external_handler, log_file = external_logger_with_handler\n\n external_logger.debug(\"Program message 1\")\n\n configure_external_file_logging(external_logger)\n\n fabric_logger = logging.getLogger(\"fabric_cicd\")\n fabric_logger.debug(\"Fabric CICD message\")\n\n for handler in logging.getLogger().handlers:\n if hasattr(handler, \"flush\"):\n handler.flush()\n external_handler.flush()\n\n content = log_file.read_text(encoding=\"utf-8\")\n assert \"Program message 1\" in content\n assert \"Fabric CICD message\" in content\n\n def test_console_only_logger_does_not_propagate_to_file(self, external_logger_with_handler):\n \"\"\"Test that console_only logger does not write to file.\"\"\"\n external_logger, _external_handler, log_file = external_logger_with_handler\n\n configure_external_file_logging(external_logger)\n\n console_only_logger = logging.getLogger(\"console_only\")\n console_only_logger.error(\"Console only error\")\n\n for handler in logging.getLogger().handlers:\n if hasattr(handler, \"flush\"):\n handler.flush()\n\n if log_file.exists():\n content = log_file.read_text(encoding=\"utf-8\")\n assert \"Console only error\" not in content\n\n\nclass TestExternalHandlerReconfiguration:\n \"\"\"Tests for external handler reconfiguration scenarios.\"\"\"\n\n def test_debug_to_non_debug_cleans_up_filter(self, external_logger_with_handler):\n \"\"\"Test switching from debug mode to non-debug mode cleans up filter.\"\"\"\n external_logger, external_handler, _log_file = external_logger_with_handler\n\n configure_external_file_logging(external_logger)\n assert len(external_handler.filters) == 1\n assert isinstance(external_handler.filters[0], PackageFilter)\n\n disable_file_logging()\n\n assert len(external_handler.filters) == 0\n assert getattr(external_handler, \"_fabric_cicd_external\", False) is False\n\n def test_non_debug_to_debug_adds_filter(self, external_logger_with_handler):\n \"\"\"Test switching from non-debug mode to debug mode adds filter correctly.\"\"\"\n external_logger, external_handler, _log_file = external_logger_with_handler\n\n disable_file_logging()\n assert len(external_handler.filters) == 0\n\n configure_external_file_logging(external_logger)\n\n assert len(external_handler.filters) == 1\n assert isinstance(external_handler.filters[0], PackageFilter)\n assert external_handler.filters[0].debug_only is True\n\n def test_multiple_debug_runs_no_filter_accumulation(self, external_logger_with_handler):\n \"\"\"Test multiple debug runs don't accumulate filters on the same handler.\"\"\"\n external_logger, external_handler, _log_file = external_logger_with_handler\n\n configure_external_file_logging(external_logger)\n configure_external_file_logging(external_logger)\n configure_external_file_logging(external_logger)\n\n assert len(external_handler.filters) == 1\n assert isinstance(external_handler.filters[0], PackageFilter)\n\n def test_handler_not_closed_on_disable(self, external_logger_with_handler):\n \"\"\"Test external handler is not closed when file logging is disabled.\"\"\"\n external_logger, external_handler, log_file = external_logger_with_handler\n\n configure_external_file_logging(external_logger)\n disable_file_logging()\n\n external_logger.debug(\"Message after disable\")\n external_handler.flush()\n\n content = log_file.read_text(encoding=\"utf-8\")\n assert \"Message after disable\" in content\n\n def test_rotating_handler_preserves_rotation_settings(self, external_logger_with_handler):\n \"\"\"Test that RotatingFileHandler rotation settings are preserved.\"\"\"\n external_logger, external_handler, _log_file = external_logger_with_handler\n\n original_max_bytes = external_handler.maxBytes\n original_backup_count = external_handler.backupCount\n\n configure_external_file_logging(external_logger)\n\n root_logger = logging.getLogger()\n external_handlers = [\n h\n for h in root_logger.handlers\n if isinstance(h, RotatingFileHandler) and getattr(h, \"_fabric_cicd_external\", False)\n ]\n\n assert len(external_handlers) == 1\n handler = external_handlers[0]\n assert handler.maxBytes == original_max_bytes\n assert handler.backupCount == original_backup_count\n" }, { "path": "tests/test_parameter.py", "content": "# Copyright (c) Microsoft Corporation.\n# Licensed under the MIT License.\n\nimport json\nimport re\nfrom pathlib import Path\nfrom unittest import mock\n\nimport pytest\nimport yaml\n\nimport fabric_cicd.constants as constants\nfrom fabric_cicd._parameter._parameter import Parameter\n\nSAMPLE_PARAMETER_FILE = \"\"\" \nfind_replace:\n # Required Fields \n - find_value: \"db52be81-c2b2-4261-84fa-840c67f4bbd0\"\n replace_value:\n PPE: \"81bbb339-8d0b-46e8-bfa6-289a159c0733\"\n PROD: \"5d6a1b16-447f-464a-b959-45d0fed35ca0\"\n # Optional Fields\n item_type: \"Notebook\"\n item_name: [\"Hello World\"] \n file_path: \"/Hello World.Notebook/notebook-content.py\"\nspark_pool:\n # Required Fields\n - instance_pool_id: \"72c68dbc-0775-4d59-909d-a47896f4573b\"\n replace_value:\n PPE:\n type: \"Capacity\"\n name: \"CapacityPool_Large_PPE\"\n PROD:\n type: \"Capacity\"\n name: \"CapacityPool_Large_PROD\"\n # Optional Fields\n item_name: \n\"\"\"\n\nSAMPLE_PARAMETER_FILE_MULTIPLE = \"\"\" \nfind_replace:\n # Required Fields \n - find_value: \"db52be81-c2b2-4261-84fa-840c67f4bbd0\"\n replace_value:\n PPE: \"81bbb339-8d0b-46e8-bfa6-289a159c0733\"\n PROD: \"5d6a1b16-447f-464a-b959-45d0fed35ca0\"\n # Optional Fields\n item_type: \"Notebook\"\n item_name: [\"Hello World\"] \n file_path: \"/Hello World.Notebook/notebook-content.py\"\n # Required Fields \n - find_value: \"db52be81-c2b2-4261-84fa-840c67f4bbd0\"\n replace_value:\n PPE: \"81bbb339-8d0b-46e8-bfa6-289a159c0733\"\n PROD: \"5d6a1b16-447f-464a-b959-45d0fed35ca0\"\n # Optional Fields\n item_type: \"Notebook\"\n item_name: [\"Hello World\"] \n file_path: \"/Hello World.Notebook/notebook-content.py\"\nkey_value_replace:\n - find_key: $.variables[?(@.name==\"SQL_Server\")].value\n replace_value:\n PPE: \"contoso-ppe.database.windows.net\"\n PROD: \"contoso-prod.database.windows.net\"\n UAT: \"contoso-uat.database.windows.net\"\n # Optional fields:\n item_type: \"VariableLibrary\"\n item_name: \"Vars\"\n - find_key: $.variables[?(@.name==\"Environment\")].value\n replace_value:\n PPE: \"PPE\"\n PROD: \"PROD\"\n UAT: \"UAT\"\n # Optional fields:\n item_type: \"VariableLibrary\"\n item_name: \"Vars\"\n - find_key: $.variableOverrides[?(@.name==\"SQL_Server\")].value\n replace_value:\n PROD: \"contoso-production-override.database.windows.net\"\n file_path: Vars.VariableLibrary/valueSets/PROD.json\n item_type: \"VariableLibrary\"\n item_name: \"Vars\"\n - find_key: $.variableOverrides[?(@.name==\"Environment\")].value\n replace_value:\n PROD: \"PROD_ENV\"\n file_path: Vars.VariableLibrary/valueSets/PROD.json\n item_type: \"VariableLibrary\"\n item_name: \"Vars\"\n\"\"\"\n\nSAMPLE_INVALID_PARAMETER_FILE = \"\"\"\nfind_replace:\n # Required Fields \n - find_value: \"db52be81-c2b2-4261-84fa-840c67f4bbd0\"\n replace_value:\n PPE: \"81bbb339-8d0b-46e8-bfa6-289a159c0733\"\n PROD: \"5d6a1b16-447f-464a-b959-45d0fed35ca0\"\n # Optional Fields\n item_type: \"Notebook\"\n item_name: [\"Hello World\"] \n file_path: \"/Hello World.Notebook/notebook-content.py\"\nspark_pool:\n # CapacityPool_Large\n \"72c68dbc-0775-4d59-909d-a47896f4573b\":\n type: \"Capacity\"\n name: \"CapacityPool_Large\"\n # CapacityPool_Medium\n \"e7b8f1c4-4a6e-4b8b-9b2e-8f1e5d6a9c3d\":\n type: \"Workspace\"\n name: \"WorkspacePool_Medium\"\n\"\"\"\n\nSAMPLE_PARAMETER_NO_TARGET_ENV = \"\"\" \nfind_replace:\n # Required Fields \n - find_value: \"db52be81-c2b2-4261-84fa-840c67f4bbd0\"\n replace_value:\n DEV: \"81bbb339-8d0b-46e8-bfa6-289a159c0733\"\n PROD: \"5d6a1b16-447f-464a-b959-45d0fed35ca0\"\n # Optional Fields\n item_type: \"Notebook\"\n item_name: [\"Hello World\"] \n file_path: \"/Hello World.Notebook/notebook-content.py\"\n\"\"\"\n\nSAMPLE_PARAMETER_MISSING_FIND_VAL = \"\"\" \nfind_replace:\n # Required Fields \n - find_value: \n replace_value:\n PPE: \"81bbb339-8d0b-46e8-bfa6-289a159c0733\"\n PROD: \"5d6a1b16-447f-464a-b959-45d0fed35ca0\"\n # Optional Fields\n item_type: \"Notebook\"\n item_name: [\"Hello World\"] \n file_path: \"/Hello World.Notebook/notebook-content.py\"\n\"\"\"\n\nSAMPLE_PARAMETER_MISMATCH_FILTER = \"\"\" \nfind_replace:\n # Required Fields \n - find_value: \"db52be81-c2b2-4261-84fa-840c67f4bbd0\"\n replace_value:\n PPE: \"81bbb339-8d0b-46e8-bfa6-289a159c0733\"\n PROD: \"5d6a1b16-447f-464a-b959-45d0fed35ca0\"\n # Optional Fields\n item_type: \"Notebook\"\n item_name: [\"Hello World\", 'Hello World Subfolder'] \n file_path: \"/Hello World.Notebook/notebook-content.py\"\n\"\"\"\n\nSAMPLE_PARAMETER_MISSING_REPLACE_VAL = \"\"\"\nspark_pool:\n # Required Fields\n - instance_pool_id: \"72c68dbc-0775-4d59-909d-a47896f4573b\"\n replace_value:\n # Optional Fields\n item_name:\n\"\"\"\n\nSAMPLE_PARAMETER_INVALID_NAME = \"\"\"\nspark_pool_param:\n # Required Fields\n - instance_pool_id: \"72c68dbc-0775-4d59-909d-a47896f4573b\"\n replace_value:\n PPE:\n type: \"Capacity\"\n name: \"CapacityPool_Large_PPE\"\n PROD:\n type: \"Capacity\"\n name: \"CapacityPool_Large_PROD\"\n # Optional Fields\n item_name: \n\"\"\"\n\nSAMPLE_PARAMETER_INVALID_YAML_STRUC = \"\"\"\nspark_pool:\n # Required Fields\n instance_pool_id: \"72c68dbc-0775-4d59-909d-a47896f4573b\"\n replace_value:\n PPE:\n type: \"Capacity\"\n name: \"CapacityPool_Large_PPE\"\n PROD:\n type: \"Capacity\"\n name: \"CapacityPool_Large_PROD\"\n # Optional Fields\n item_name: \n\"\"\"\n\nSAMPLE_PARAMETER_INVALID_YAML_CHAR = \"\"\"\nfind_replace:\n # Required Fields \n - find_value: '\"db52be81-c2b2-4261-84fa-840c67f4bbd0\"\n replace_value:\n PPE: \"81bbb339-8d0b-46e8-bfa6-289a159c0733\"\n PROD: \"5d6a1b16-447f-464a-b959-45d0fed35ca0\"\n # Optional Fields\n item_type: \"Notebook\"\n item_name: [\"Hello World\"] \n file_path: \"/Hello World.Notebook/notebook-content.py\"\n\"\"\"\n\nSAMPLE_PARAMETER_INVALID_IS_REGEX = \"\"\"\nfind_replace:\n # Required Fields\n - find_value: \"\\\\#\\\\s*META\\\\s+\\\"default_lakehouse\\\":\\\\s*\\\"([0-9a-fA-F]{8}-[0-9a-fA-F]{4}-[0-9a-fA-F]{4}-[0-9a-fA-F]{4}-[0-9a-fA-F]{12})\\\"\"\n replace_value:\n PPE: \"81bbb339-8d0b-46e8-bfa6-289a159c0733\"\n PROD: \"5d6a1b16-447f-464a-b959-45d0fed35ca0\"\n # Optional Fields\n is_regex: True\n item_type: \"Notebook\"\n\"\"\"\n\nSAMPLE_PARAMETER_ALL_ENV = \"\"\"\nfind_replace:\n # Required Fields \n - find_value: \"db52be81-c2b2-4261-84fa-840c67f4bbd0\"\n replace_value:\n ALL: \"universal-workspace-id-12345\"\n # Optional Fields\n item_type: \"Notebook\"\n item_name: [\"Hello World\"] \n file_path: \"/Hello World.Notebook/notebook-content.py\"\nkey_value_replace:\n - find_key: $.variables[?(@.name==\"Environment\")].value\n replace_value:\n ALL: \"ANY_ENV\"\n # Optional fields:\n item_type: \"VariableLibrary\"\n item_name: \"Vars\"\n\"\"\"\n\nSAMPLE_PLATFORM_FILE = \"\"\"\n{\n \"$schema\": \"https://developer.microsoft.com/json-schemas/fabric/gitIntegration/platformProperties/2.0.0/schema.json\",\n \"metadata\": {\n \"type\": \"Notebook\",\n \"displayName\": \"Hello World\",\n \"description\": \"Sample notebook\"\n },\n \"config\": {\n \"version\": \"2.0\",\n \"logicalId\": \"99b570c5-0c79-9dc4-4c9b-fa16c621384c\"\n }\n}\n\"\"\"\n\nSAMPLE_NOTEBOOK_FILE = \"print('Hello World and replace connection string: db52be81-c2b2-4261-84fa-840c67f4bbd0')\"\n\nSAMPLE_PARAMETER_FILE_DUPLICATE_KEYS = \"\"\"\nfind_replace:\n - find_value: \"first-value\"\n replace_value:\n PPE: \"first-ppe\"\n PROD: \"first-prod\"\n\nfind_replace:\n - find_value: \"second-value\"\n replace_value:\n PPE: \"second-ppe\"\n PROD: \"second-prod\"\n\"\"\"\n\nSAMPLE_PARAMETER_FILE_MULTIPLE_DUPLICATE_KEYS = \"\"\"\nfind_replace:\n - find_value: \"first-value\"\n replace_value:\n PPE: \"first-ppe\"\n\nspark_pool:\n - instance_pool_id: \"pool-1\"\n replace_value:\n PPE:\n type: \"Capacity\"\n name: \"Pool1\"\n\nfind_replace:\n - find_value: \"second-value\"\n replace_value:\n PPE: \"second-ppe\"\n\nspark_pool:\n - instance_pool_id: \"pool-2\"\n replace_value:\n PPE:\n type: \"Capacity\"\n name: \"Pool2\"\n\"\"\"\n\nSAMPLE_PARAMETER_FILE_TRIPLE_DUPLICATE_KEY = \"\"\"\nfind_replace:\n - find_value: \"first-value\"\n replace_value:\n PPE: \"first-ppe\"\n\nfind_replace:\n - find_value: \"second-value\"\n replace_value:\n PPE: \"second-ppe\"\n\nfind_replace:\n - find_value: \"third-value\"\n replace_value:\n PPE: \"third-ppe\"\n\"\"\"\n\n\n@pytest.fixture\ndef item_type_in_scope():\n return [\"Notebook\", \"DataPipeline\", \"Environment\"]\n\n\n@pytest.fixture\ndef target_environment():\n return \"PPE\"\n\n\n@pytest.fixture\ndef repository_directory(tmp_path):\n # Create the sample workspace structure\n workspace_dir = tmp_path / \"sample\" / \"workspace\"\n workspace_dir.mkdir(parents=True, exist_ok=True)\n\n # Create the sample parameter file\n parameter_file_path = workspace_dir / constants.PARAMETER_FILE_NAME\n parameter_file_path.write_text(SAMPLE_PARAMETER_FILE)\n\n # Create sample invalid parameter files\n invalid_parameter_file_path = workspace_dir / \"invalid_parameter.yml\"\n invalid_parameter_file_path.write_text(SAMPLE_INVALID_PARAMETER_FILE)\n\n invalid_parameter_file_path1 = workspace_dir / \"no_target_env_parameter.yml\"\n invalid_parameter_file_path1.write_text(SAMPLE_PARAMETER_NO_TARGET_ENV)\n\n invalid_parameter_file_path2 = workspace_dir / \"missing_find_val_parameter.yml\"\n invalid_parameter_file_path2.write_text(SAMPLE_PARAMETER_MISSING_FIND_VAL)\n\n invalid_parameter_file_path3 = workspace_dir / \"mismatch_filter_parameter.yml\"\n invalid_parameter_file_path3.write_text(SAMPLE_PARAMETER_MISMATCH_FILTER)\n\n invalid_parameter_file_path4 = workspace_dir / \"missing_replace_val_parameter.yml\"\n invalid_parameter_file_path4.write_text(SAMPLE_PARAMETER_MISSING_REPLACE_VAL)\n\n invalid_parameter_file_path5 = workspace_dir / \"invalid_name_parameter.yml\"\n invalid_parameter_file_path5.write_text(SAMPLE_PARAMETER_INVALID_NAME)\n\n invalid_parameter_file_path6 = workspace_dir / \"invalid_yaml_struc_parameter.yml\"\n invalid_parameter_file_path6.write_text(SAMPLE_PARAMETER_INVALID_YAML_STRUC)\n\n invalid_parameter_file_path7 = workspace_dir / \"invalid_yaml_char_parameter.yml\"\n invalid_parameter_file_path7.write_text(SAMPLE_PARAMETER_INVALID_YAML_CHAR)\n\n invalid_parameter_file_path8 = workspace_dir / \"invalid_is_regex_parameter.yml\"\n invalid_parameter_file_path8.write_text(SAMPLE_PARAMETER_INVALID_IS_REGEX)\n\n # Create duplicate keys parameter files\n duplicate_keys_file = workspace_dir / \"duplicate_keys_parameter.yml\"\n duplicate_keys_file.write_text(SAMPLE_PARAMETER_FILE_DUPLICATE_KEYS)\n\n multiple_duplicate_keys_file = workspace_dir / \"multiple_duplicate_keys_parameter.yml\"\n multiple_duplicate_keys_file.write_text(SAMPLE_PARAMETER_FILE_MULTIPLE_DUPLICATE_KEYS)\n\n triple_duplicate_key_file = workspace_dir / \"triple_duplicate_key_parameter.yml\"\n triple_duplicate_key_file.write_text(SAMPLE_PARAMETER_FILE_TRIPLE_DUPLICATE_KEY)\n\n # Create the sample parameter file with ALL environment key\n all_env_parameter_file_path = workspace_dir / \"all_env_parameter.yml\"\n all_env_parameter_file_path.write_text(SAMPLE_PARAMETER_ALL_ENV)\n\n # Create the sample parameter file with multiple of a parameter\n multiple_parameter_file_path = workspace_dir / \"multiple_parameter.yml\"\n multiple_parameter_file_path.write_text(SAMPLE_PARAMETER_FILE_MULTIPLE)\n\n # Create the sample notebook files\n notebook_dir = workspace_dir / \"Hello World.Notebook\"\n notebook_dir.mkdir(parents=True, exist_ok=True)\n\n notebook_platform_file_path = notebook_dir / \".platform\"\n notebook_platform_file_path.write_text(SAMPLE_PLATFORM_FILE)\n\n notebook_file_path = notebook_dir / \"notebook-content.py\"\n notebook_file_path.write_text(SAMPLE_NOTEBOOK_FILE)\n\n return workspace_dir\n\n\n@pytest.fixture\ndef parameter_object(repository_directory, item_type_in_scope, target_environment):\n \"\"\"Fixture to create a Parameter object.\"\"\"\n return Parameter(\n repository_directory=repository_directory,\n item_type_in_scope=item_type_in_scope,\n environment=target_environment,\n parameter_file_name=constants.PARAMETER_FILE_NAME,\n )\n\n\ndef test_parameter_class_initialization(parameter_object, repository_directory, item_type_in_scope, target_environment):\n \"\"\"Test the Parameter class initialization.\"\"\"\n parameter_file_name = constants.PARAMETER_FILE_NAME\n\n # Check if the object is initialized correctly\n assert parameter_object.repository_directory == repository_directory\n assert parameter_object.item_type_in_scope == item_type_in_scope\n assert parameter_object.environment == target_environment\n assert parameter_object.parameter_file_name == parameter_file_name\n assert parameter_object.parameter_file_path == repository_directory / parameter_file_name\n\n\ndef test_parameter_file_validation(parameter_object):\n \"\"\"Test the validation methods for the parameter file\"\"\"\n assert parameter_object._validate_parameter_file_exists() == True\n assert parameter_object._validate_load_parameters_to_dict() == (True, parameter_object.environment_parameter)\n assert parameter_object._validate_parameter_load() == (True, constants.PARAMETER_MSGS[\"valid load\"])\n assert parameter_object._validate_parameter_names() == (True, constants.PARAMETER_MSGS[\"valid name\"])\n assert parameter_object._validate_parameter_structure() == (True, constants.PARAMETER_MSGS[\"valid structure\"])\n assert parameter_object._validate_parameter(\"find_replace\") == (\n True,\n constants.PARAMETER_MSGS[\"valid parameter\"].format(\"find_replace\"),\n )\n assert parameter_object._validate_parameter(\"spark_pool\") == (\n True,\n constants.PARAMETER_MSGS[\"valid parameter\"].format(\"spark_pool\"),\n )\n assert parameter_object._validate_parameter_file() == True\n\n\ndef test_multiple_parameter_validation(repository_directory, item_type_in_scope, target_environment):\n \"\"\"Test the validation methods for multiple parameters case\"\"\"\n multi_param_obj = Parameter(\n repository_directory=repository_directory,\n item_type_in_scope=item_type_in_scope,\n environment=target_environment,\n parameter_file_name=\"multiple_parameter.yml\",\n )\n assert multi_param_obj._validate_parameter(\"find_replace\") == (\n True,\n constants.PARAMETER_MSGS[\"valid parameter\"].format(\"find_replace\"),\n )\n assert multi_param_obj._validate_parameter(\"key_value_replace\") == (\n True,\n constants.PARAMETER_MSGS[\"valid parameter\"].format(\"key_value_replace\"),\n )\n assert multi_param_obj._validate_parameter_file() == True\n\n\n@pytest.mark.parametrize(\n (\"param_name\", \"param_value\", \"result\", \"msg\"),\n [\n (\"find_replace\", [\"find_value\", \"replace_value\"], True, \"valid keys\"),\n (\"find_replace\", [\"find_value\", \"item_type\", \"item_name\", \"file_path\"], False, \"missing key\"),\n (\"find_replace\", [\"find_value\", \"replace_value\", \"is_regex\", \"item_type\"], True, \"valid keys\"),\n (\"spark_pool\", [\"instance_pool_id\", \"replace_value\", \"item_name\"], True, \"valid keys\"),\n (\"spark_pool\", [\"instance_pool_id\", \"replace_value\", \"item_name\", \"file_path\"], False, \"invalid key\"),\n ],\n)\ndef test_validate_parameter_keys(parameter_object, param_name, param_value, result, msg):\n \"\"\"Test the validation methods for the find_replace parameter\"\"\"\n\n assert parameter_object._validate_parameter_keys(param_name, param_value) == (\n result,\n constants.PARAMETER_MSGS[msg].format(param_name),\n )\n\n\n@pytest.mark.parametrize((\"param_name\"), [(\"find_replace\"), (\"spark_pool\")])\ndef test_validate_parameter(parameter_object, param_name):\n \"\"\"Test the validation methods for a specific parameter\"\"\"\n param_dict = parameter_object.environment_parameter.get(param_name)\n for param in param_dict:\n assert parameter_object._validate_required_values(param_name, param) == (\n True,\n constants.PARAMETER_MSGS[\"valid required values\"].format(param_name),\n )\n assert parameter_object._validate_replace_value(param_name, param[\"replace_value\"]) == (\n True,\n constants.PARAMETER_MSGS[\"valid replace value\"].format(param_name),\n )\n\n\n@pytest.mark.parametrize(\n (\"replace_value\", \"result\", \"msg\"),\n [\n (\n {\"PPE\": \"81bbb339-8d0b-46e8-bfa6-289a159c0733\", \"PROD\": \"5d6a1b16-447f-464a-b959-45d0fed35ca0\"},\n True,\n \"valid replace value\",\n ),\n (\n {\"PPE\": \"81bbb339-8d0b-46e8-bfa6-289a159c0733\", \"PROD\": None},\n False,\n \"missing replace value\",\n ),\n ],\n)\ndef test_validate_find_replace_replace_value(parameter_object, replace_value, result, msg):\n \"\"\"Test the _validate_find_replace_replace_value method.\"\"\"\n assert parameter_object._validate_find_replace_replace_value(replace_value) == (\n result,\n constants.PARAMETER_MSGS[msg].format(\"find_replace\", \"PROD\")\n if msg == \"missing replace value\"\n else constants.PARAMETER_MSGS[msg].format(\"find_replace\"),\n )\n\n\n@pytest.mark.parametrize(\n (\"replace_value\", \"result\", \"msg\"),\n [\n # Valid cases - all values are same type\n (\n {\"PPE\": \"string_value\", \"PROD\": \"another_string\"},\n True,\n \"valid replace value\",\n ),\n (\n {\"PPE\": True, \"PROD\": False},\n True,\n \"valid replace value\",\n ),\n (\n {\"PPE\": 123, \"PROD\": 456},\n True,\n \"valid replace value\",\n ),\n (\n {\"PPE\": 1.5, \"PROD\": 2.7},\n True,\n \"valid replace value\",\n ),\n (\n {\"PPE\": [\"item1\", \"item2\"], \"PROD\": [\"item3\", \"item4\"]},\n True,\n \"valid replace value\",\n ),\n (\n {\"PPE\": {\"key\": \"value1\"}, \"PROD\": {\"key\": \"value2\"}},\n True,\n \"valid replace value\",\n ),\n # Invalid cases - missing values\n (\n {\"PPE\": \"value\", \"PROD\": None},\n False,\n \"missing replace value\",\n ),\n # Invalid cases - mixed types\n (\n {\"PPE\": \"string_value\", \"PROD\": 123},\n False,\n \"mixed types\",\n ),\n (\n {\"PPE\": True, \"PROD\": \"false\"},\n False,\n \"mixed types\",\n ),\n (\n {\"PPE\": 123, \"PROD\": 45.6},\n False,\n \"mixed types\",\n ),\n ],\n)\ndef test_validate_key_value_replace_replace_value(parameter_object, replace_value, result, msg):\n \"\"\"Test the _validate_key_value_replace_replace_value method.\"\"\"\n is_valid, actual_msg = parameter_object._validate_key_value_replace_replace_value(replace_value)\n\n if msg == \"valid replace value\":\n expected_msg = constants.PARAMETER_MSGS[msg].format(\"key_value_replace\")\n assert (is_valid, actual_msg) == (result, expected_msg)\n elif msg == \"missing replace value\":\n # For missing replace value, check that the message contains the expected format\n assert is_valid == result\n assert \"key_value_replace is missing a replace value for\" in actual_msg\n elif msg == \"mixed types\":\n # For mixed types, check that the message contains the expected content\n assert is_valid == result\n assert \"Inconsistent data types in key_value_replace replace_value\" in actual_msg\n\n\n@pytest.mark.parametrize(\n (\"replace_value\", \"result\", \"msg\", \"desc\"),\n [\n (\n {\n \"PPE\": {\"type\": \"Capacity\", \"name\": \"CapacityPool_Large_PPE\"},\n \"PROD\": {\"type\": \"Capacity\", \"name\": \"CapacityPool_Large_PROD\"},\n },\n True,\n \"valid replace value\",\n None,\n ),\n (\n {\n \"PPE\": {},\n \"PROD\": {\"type\": \"Capacity\", \"name\": \"CapacityPool_Large_PROD\"},\n },\n False,\n \"missing replace value\",\n None,\n ),\n (\n {\n \"PPE\": {\"name\": \"CapacityPool_Large_PPE\"},\n \"PROD\": {\"type\": \"Capacity\", \"name\": \"CapacityPool_Large_PROD\"},\n },\n False,\n \"invalid replace value\",\n \"missing key\",\n ),\n (\n {\n \"PPE\": {\"type\": \"Capacity\", \"name\": \"CapacityPool_Large_PPE\"},\n \"PROD\": {\"type\": \"Capacity\", \"name\": None},\n },\n False,\n \"invalid replace value\",\n \"missing value\",\n ),\n (\n {\n \"PPE\": {\"type\": \"Capacity\", \"name\": \"CapacityPool_Large_PPE\"},\n \"PROD\": {\"type\": \"Test\", \"name\": \"CapacityPool_Large_PROD\"},\n },\n False,\n \"invalid replace value\",\n \"invalid value\",\n ),\n ],\n)\ndef test_validate_spark_pool_replace_value(parameter_object, replace_value, result, msg, desc):\n \"\"\"Test the _validate_spark_pool_replace_value method.\"\"\"\n if msg == \"valid replace value\":\n msg = constants.PARAMETER_MSGS[msg].format(\"spark_pool\")\n if msg == \"missing replace value\":\n msg = constants.PARAMETER_MSGS[msg].format(\"spark_pool\", \"PPE\")\n if msg == \"invalid replace value\" and desc == \"missing key\":\n msg = constants.PARAMETER_MSGS[msg][desc].format(\"PPE\")\n if msg == \"invalid replace value\" and desc == \"missing value\":\n msg = constants.PARAMETER_MSGS[msg][desc].format(\"PROD\", \"name\")\n if msg == \"invalid replace value\" and desc == \"invalid value\":\n msg = constants.PARAMETER_MSGS[msg][desc].format(\"PROD\")\n\n assert parameter_object._validate_spark_pool_replace_value(replace_value) == (result, msg)\n assert parameter_object._validate_replace_value(\n \"spark_pool\",\n {\n \"PPE\": {},\n \"PROD\": {\"type\": \"Capacity\", \"name\": \"CapacityPool_Large_PROD\"},\n },\n ) == (False, constants.PARAMETER_MSGS[\"missing replace value\"].format(\"spark_pool\", \"PPE\"))\n\n\ndef test_validate_data_type(parameter_object):\n \"\"\"Test data type validation\"\"\"\n # General data type validation\n assert parameter_object._validate_data_type([1, 2, 3], \"string or list[string]\", \"key\", \"param_name\") == (\n False,\n constants.PARAMETER_MSGS[\"invalid data type\"].format(\"key\", \"string or list[string]\", \"param_name\"),\n )\n\n required_values = {\n \"find_value\": [\"db52be81-c2b2-4261-84fa-840c67f4bbd0\"],\n \"replace_value\": {\n \"PPE\": \"81bbb339-8d0b-46e8-bfa6-289a159c0733\",\n \"PROD\": \"5d6a1b16-447f-464a-b959-45d0fed35ca0\",\n },\n }\n # Data type error in required values\n assert parameter_object._validate_required_values(\"find_replace\", required_values) == (\n False,\n constants.PARAMETER_MSGS[\"invalid data type\"].format(\"find_value\", \"string\", \"find_replace\"),\n )\n\n find_replace_value = {\n \"PPE\": \"81bbb339-8d0b-46e8-bfa6-289a159c0733\",\n \"PROD\": 123,\n }\n # Data type error in find_replace replace value dict\n assert parameter_object._validate_find_replace_replace_value(find_replace_value) == (\n False,\n constants.PARAMETER_MSGS[\"invalid data type\"].format(\"PROD replace_value\", \"string\", \"find_replace\"),\n )\n\n spark_pool_replace_value_1 = {\n \"PPE\": \"string\",\n \"PROD\": {\"type\": \"Capacity\", \"name\": \"CapacityPool_Large_PROD\"},\n }\n # Data type error in spark_pool replace value dict\n assert parameter_object._validate_spark_pool_replace_value(spark_pool_replace_value_1) == (\n False,\n constants.PARAMETER_MSGS[\"invalid data type\"].format(\"PPE key\", \"dictionary\", \"spark_pool\"),\n )\n\n spark_pool_replace_value_2 = {\n \"PPE\": {\"type\": \"Capacity\", \"name\": \"CapacityPool_Large_PPE\"},\n \"PROD\": {\"type\": [\"Capacity\"], \"name\": \"CapacityPool_Large_PROD\"},\n }\n # Data type error in spark_pool replace value environment dict\n assert parameter_object._validate_spark_pool_replace_value(spark_pool_replace_value_2) == (\n False,\n constants.PARAMETER_MSGS[\"invalid data type\"].format(\"type\", \"string\", \"spark_pool\"),\n )\n\n param_dict = {\n \"item_type\": \"Notebook\",\n \"item_name\": {\"Hello World\"},\n \"file_path\": \"/Hello World.Notebook/notebook-content.py\",\n }\n # Data type error in optional values\n assert parameter_object._validate_optional_values(\"find_replace\", param_dict) == (\n False,\n constants.PARAMETER_MSGS[\"invalid data type\"].format(\"item_name\", \"string or list[string]\", \"find_replace\"),\n )\n\n\ndef test_validate_yaml_content_empty():\n \"\"\"Test that empty YAML content is handled correctly.\"\"\"\n import tempfile\n\n # Test empty content - create a temp file with empty content\n with tempfile.NamedTemporaryFile(mode=\"w\", suffix=\".yml\", delete=False) as temp_file:\n temp_file.write(\"\\n\\n\\n\\t\")\n temp_file_path = temp_file.name\n\n try:\n param = Parameter(\n repository_directory=Path(temp_file_path).parent,\n item_type_in_scope=[\"Notebook\"],\n environment=\"TEST\",\n parameter_file_name=Path(temp_file_path).name,\n )\n # Empty content should fail to load\n assert not param.environment_parameter, \"Empty YAML should not load successfully\"\n is_valid, msg = param._validate_parameter_load()\n assert is_valid is False\n assert constants.PARAMETER_MSGS[\"empty yaml\"] in msg\n finally:\n Path(temp_file_path).unlink()\n\n\ndef test_utf8_validation_at_file_read():\n \"\"\"Test that Python validates UTF-8 encoding when reading parameter files.\"\"\"\n import tempfile\n\n # Create a file with invalid UTF-8 bytes\n with tempfile.NamedTemporaryFile(mode=\"wb\", suffix=\".yml\", delete=False) as temp_file:\n # Write valid YAML structure with invalid UTF-8 byte sequence\n temp_file.write(b\"find_replace:\\n - find_value: 'invalid \\x80\\x81\\x82 bytes'\\n\")\n temp_file_path = temp_file.name\n\n try:\n # Parameter creation should succeed but loading should fail gracefully\n param = Parameter(\n repository_directory=Path(temp_file_path).parent,\n item_type_in_scope=[\"Notebook\"],\n environment=\"TEST\",\n parameter_file_name=Path(temp_file_path).name,\n )\n\n # Parameter should fail to load due to invalid UTF-8\n assert not param.environment_parameter, \"Invalid UTF-8 file should not load successfully\"\n\n # Verify that validation reports the file as invalid\n is_valid, msg = param._validate_parameter_load()\n assert is_valid is False, \"Parameter load should fail for invalid UTF-8\"\n\n # Verify the error message uses the constant format with UnicodeDecodeError details\n assert constants.PARAMETER_MSGS[\"invalid load\"].split(\"{}\")[0] in msg, (\n f\"Error message should use 'invalid load' constant format: {msg}\"\n )\n # UnicodeDecodeError message contains 'utf-8' and 'decode'\n assert \"utf-8\" in msg.lower(), f\"Error message should contain UTF-8 details: {msg}\"\n\n finally:\n # Clean up temporary file\n Path(temp_file_path).unlink()\n\n\ndef test_validate_yaml_content_duplicate_keys():\n \"\"\"Test that duplicate keys are detected via _DuplicateKeyLoader.\"\"\"\n import tempfile\n\n # Test duplicate key detection - create a temp file with duplicate keys\n with tempfile.NamedTemporaryFile(mode=\"w\", suffix=\".yml\", delete=False) as temp_file:\n temp_file.write(\"\"\"find_replace:\n - find_value: \"first\"\n replace_value:\n TEST: \"first-value\"\nfind_replace:\n - find_value: \"second\"\n replace_value:\n TEST: \"second-value\"\n\"\"\")\n temp_file_path = temp_file.name\n\n try:\n param = Parameter(\n repository_directory=Path(temp_file_path).parent,\n item_type_in_scope=[\"Notebook\"],\n environment=\"TEST\",\n parameter_file_name=Path(temp_file_path).name,\n )\n # Duplicate keys should fail to load\n assert not param.environment_parameter, \"YAML with duplicate keys should not load successfully\"\n is_valid, msg = param._validate_parameter_load()\n assert is_valid is False\n assert constants.PARAMETER_MSGS[\"duplicate key\"].split(\"{}\")[0].lower() in msg.lower()\n finally:\n Path(temp_file_path).unlink()\n\n\ndef test_duplicate_keys_single_duplicate(repository_directory, item_type_in_scope, target_environment):\n \"\"\"Test detection of a single duplicate root-level key via _DuplicateKeyLoader.\"\"\"\n param_obj = Parameter(\n repository_directory=repository_directory,\n item_type_in_scope=item_type_in_scope,\n environment=target_environment,\n parameter_file_name=\"duplicate_keys_parameter.yml\",\n )\n\n # Duplicate keys should fail to load - _DuplicateKeyLoader catches them during yaml.load()\n assert not param_obj.environment_parameter, \"YAML with duplicate keys should not load successfully\"\n\n is_valid, _msg = param_obj._validate_parameter_load()\n assert is_valid is False\n assert constants.PARAMETER_MSGS[\"duplicate key\"].split(\"{}\")[0].lower() in _msg.lower()\n\n\ndef test_duplicate_keys_multiple_duplicates(repository_directory, item_type_in_scope, target_environment):\n \"\"\"Test detection of multiple duplicate root-level keys via _DuplicateKeyLoader.\"\"\"\n param_obj = Parameter(\n repository_directory=repository_directory,\n item_type_in_scope=item_type_in_scope,\n environment=target_environment,\n parameter_file_name=\"multiple_duplicate_keys_parameter.yml\",\n )\n\n # Duplicate keys should fail to load - _DuplicateKeyLoader catches them during yaml.load()\n assert not param_obj.environment_parameter, \"YAML with duplicate keys should not load successfully\"\n\n is_valid, _msg = param_obj._validate_parameter_load()\n assert is_valid is False\n assert constants.PARAMETER_MSGS[\"duplicate key\"].split(\"{}\")[0].lower() in _msg.lower()\n\n\ndef test_duplicate_keys_no_duplicates(parameter_object):\n \"\"\"Test that valid YAML with no duplicate keys passes.\"\"\"\n # parameter_object fixture uses a valid parameter file without duplicates\n assert parameter_object.environment_parameter, \"Valid YAML should load successfully\"\n\n is_valid, _msg = parameter_object._validate_parameter_load()\n assert is_valid is True\n\n\ndef test_duplicate_keys_ignores_comments():\n \"\"\"Test that comment lines starting with # are ignored.\"\"\"\n import tempfile\n\n content = \"\"\"\n# This is a comment\nfind_replace:\n - find_value: \"value1\"\n replace_value:\n PPE: \"ppe-value\"\n# Another comment\n# find_replace: this should be ignored\nspark_pool:\n - instance_pool_id: \"pool-id\"\n replace_value:\n PPE:\n type: \"Capacity\"\n name: \"Pool\"\n\"\"\"\n with tempfile.NamedTemporaryFile(mode=\"w\", suffix=\".yml\", delete=False) as temp_file:\n temp_file.write(content)\n temp_file_path = temp_file.name\n\n try:\n param = Parameter(\n repository_directory=Path(temp_file_path).parent,\n item_type_in_scope=[\"Notebook\"],\n environment=\"PPE\",\n parameter_file_name=Path(temp_file_path).name,\n )\n # Valid YAML with comments should load successfully\n assert param.environment_parameter, \"Valid YAML with comments should load successfully\"\n is_valid, _msg = param._validate_parameter_load()\n assert is_valid is True\n finally:\n Path(temp_file_path).unlink()\n\n\ndef test_duplicate_keys_nested_keys_not_flagged():\n \"\"\"Test that nested keys (indented) are not flagged as root-level duplicates.\"\"\"\n import tempfile\n\n content = \"\"\"\nfind_replace:\n - find_value: \"value1\"\n replace_value:\n PPE: \"ppe-value\"\n - find_value: \"value2\"\n replace_value:\n PPE: \"ppe-value2\"\n\"\"\"\n with tempfile.NamedTemporaryFile(mode=\"w\", suffix=\".yml\", delete=False) as temp_file:\n temp_file.write(content)\n temp_file_path = temp_file.name\n\n try:\n param = Parameter(\n repository_directory=Path(temp_file_path).parent,\n item_type_in_scope=[\"Notebook\"],\n environment=\"PPE\",\n parameter_file_name=Path(temp_file_path).name,\n )\n # Valid YAML with nested repeated keys should load successfully\n assert param.environment_parameter, \"Valid YAML with nested keys should load successfully\"\n is_valid, _msg = param._validate_parameter_load()\n assert is_valid is True\n finally:\n Path(temp_file_path).unlink()\n\n\n@pytest.mark.parametrize(\n (\"content\", \"duplicate_keys\"),\n [\n # Duplicate environment key within replace_value\n (\n \"\"\"\nfind_replace:\n - find_value: \"value1\"\n replace_value:\n PPE: \"ppe-value1\"\n PPE: \"ppe-value2\"\n\"\"\",\n [\"PPE\"],\n ),\n # Duplicate find_value within a single list entry\n (\n \"\"\"\nfind_replace:\n - find_value: \"first\"\n find_value: \"second\"\n replace_value:\n PPE: \"ppe-value\"\n\"\"\",\n [\"find_value\"],\n ),\n # Duplicate replace_value within a single list entry\n (\n \"\"\"\nfind_replace:\n - find_value: \"test-id\"\n replace_value:\n PPE: \"first-ppe\"\n replace_value:\n PPE: \"second-ppe\"\n\"\"\",\n [\"replace_value\"],\n ),\n # Duplicate optional field (item_type)\n (\n \"\"\"\nfind_replace:\n - find_value: \"test-id\"\n replace_value:\n PPE: \"ppe-value\"\n item_type: \"Notebook\"\n item_type: \"DataPipeline\"\n\"\"\",\n [\"item_type\"],\n ),\n # Case-insensitive duplicate environment key\n (\n \"\"\"\nfind_replace:\n - find_value: \"abc\"\n replace_value:\n PROD: \"value1\"\n prod: \"value2\"\n\"\"\",\n [\"prod\"],\n ),\n # Duplicate item_name within a single list entry\n (\n \"\"\"\nfind_replace:\n - find_value: \"test-id\"\n replace_value:\n PPE: \"ppe-value\"\n item_name: \"Notebook1\"\n item_name: \"Notebook2\"\n\"\"\",\n [\"item_name\"],\n ),\n # Duplicate file_path within a single list entry\n (\n \"\"\"\nfind_replace:\n - find_value: \"test-id\"\n replace_value:\n PPE: \"ppe-value\"\n file_path: \"/path/one.py\"\n file_path: \"/path/two.py\"\n\"\"\",\n [\"file_path\"],\n ),\n # Multiple different duplicate keys in the same entry\n (\n \"\"\"\nfind_replace:\n - find_value: \"test-id\"\n find_value: \"test-id-2\"\n replace_value:\n PPE: \"first-ppe\"\n replace_value:\n PPE: \"second-ppe\"\n item_type: \"Notebook\"\n item_type: \"DataPipeline\"\n\"\"\",\n [\"find_value\", \"replace_value\", \"item_type\"],\n ),\n ],\n ids=[\n \"duplicate_environment_key\",\n \"duplicate_find_value\",\n \"duplicate_replace_value\",\n \"duplicate_item_type\",\n \"case_insensitive_duplicate\",\n \"duplicate_item_name\",\n \"duplicate_file_path\",\n \"multiple_different_duplicate_keys\",\n ],\n)\ndef test_duplicate_keys_nested_duplicate_detected(content, duplicate_keys):\n \"\"\"Test that duplicate keys within nested mappings are detected by _DuplicateKeyLoader.\"\"\"\n import tempfile\n\n with tempfile.NamedTemporaryFile(mode=\"w\", suffix=\".yml\", delete=False) as temp_file:\n temp_file.write(content)\n temp_file_path = temp_file.name\n\n try:\n param = Parameter(\n repository_directory=Path(temp_file_path).parent,\n item_type_in_scope=[\"Notebook\"],\n environment=\"PPE\",\n parameter_file_name=Path(temp_file_path).name,\n )\n # Duplicate keys should fail to load\n assert not param.environment_parameter, (\n f\"YAML with duplicate '{duplicate_keys}' keys should not load successfully\"\n )\n is_valid, msg = param._validate_parameter_load()\n assert is_valid is False\n for key in duplicate_keys:\n assert key.lower() in msg.lower(), f\"Expected '{key}' in error message: {msg}\"\n finally:\n Path(temp_file_path).unlink()\n\n\ndef test_duplicate_keys_triple_occurrence(repository_directory, item_type_in_scope, target_environment):\n \"\"\"Test detection of a key appearing more than twice via _DuplicateKeyLoader.\"\"\"\n param_obj = Parameter(\n repository_directory=repository_directory,\n item_type_in_scope=item_type_in_scope,\n environment=target_environment,\n parameter_file_name=\"triple_duplicate_key_parameter.yml\",\n )\n\n # Duplicate keys should fail to load - _DuplicateKeyLoader catches them during yaml.load()\n assert not param_obj.environment_parameter, \"YAML with duplicate keys should not load successfully\"\n\n is_valid, msg = param_obj._validate_parameter_load()\n assert is_valid is False\n assert constants.PARAMETER_MSGS[\"duplicate key\"].split(\"{}\")[0].lower() in msg.lower()\n\n\ndef test_validate_parameter_file_structure(repository_directory, item_type_in_scope, target_environment):\n \"\"\"Test the validation of the parameter file structure\"\"\"\n param_obj = Parameter(\n repository_directory=repository_directory,\n item_type_in_scope=item_type_in_scope,\n environment=target_environment,\n parameter_file_name=\"invalid_parameter.yml\",\n )\n assert param_obj._validate_parameter_structure() == (False, constants.PARAMETER_MSGS[\"invalid structure\"])\n\n\ndef test_validate_optional_values(parameter_object):\n \"\"\"Test the _validate_optional_values method.\"\"\"\n param_dict_1 = {\n \"item_type\": \"Notebook\",\n \"item_name\": [\"Hello World\"],\n \"file_path\": \"/Hello World.Notebook/notebook-content.py\",\n }\n assert parameter_object._validate_optional_values(\"find_replace\", param_dict_1) == (\n True,\n constants.PARAMETER_MSGS[\"valid optional\"].format(\"find_replace\"),\n )\n\n param_dict_2 = {\n \"item_type\": \"SparkNotebook\",\n \"item_name\": [\"Hello World\"],\n \"file_path\": \"/Hello World.Notebook/notebook-content.py\",\n }\n assert parameter_object._validate_optional_values(\"find_replace\", param_dict_2, check_match=True) == (\n False,\n \"no match\",\n )\n\n param_dict_3 = {\"item_name\": \"Hello World\"}\n assert parameter_object._validate_optional_values(\"spark_pool\", param_dict_3, check_match=True) == (\n True,\n constants.PARAMETER_MSGS[\"valid optional\"].format(\"spark_pool\"),\n )\n\n\n@pytest.mark.parametrize(\n (\"param_name\"),\n [\"find_replace\", \"spark_pool\"],\n)\ndef test_validate_parameter_environment_and_filters(parameter_object, param_name):\n \"\"\"Test the validation methods for environment and filters\"\"\"\n for param_dict in parameter_object.environment_parameter.get(param_name):\n # Environment validation\n assert parameter_object._validate_environment(param_dict[\"replace_value\"]) == (True, \"env\")\n\n # Optional filters validation\n assert parameter_object._validate_item_type(\"Pipeline\") == (\n False,\n constants.PARAMETER_MSGS[\"invalid item type\"].format(\"Pipeline\"),\n )\n assert parameter_object._validate_item_name(\"Hello World 2\") == (\n False,\n constants.PARAMETER_MSGS[\"invalid item name\"].format(\"Hello World 2\"),\n )\n assert parameter_object._validate_file_path([\"Hello World 2.Notebook/notebook-content.py\"]) == (\n False,\n constants.PARAMETER_MSGS[\"no valid file path\"].format([\"Hello World 2.Notebook/notebook-content.py\"]),\n )\n\n\ndef test_validate_item_name_with_accented_characters(repository_directory, item_type_in_scope, target_environment):\n \"\"\"Test that _validate_item_name correctly handles item names with accented characters.\"\"\"\n # Create a notebook directory with accented characters in the displayName\n accented_name = \"Rôles et parcours de formation\"\n accented_dir = repository_directory / f\"{accented_name}.Report\"\n accented_dir.mkdir(parents=True, exist_ok=True)\n\n platform_content = f\"\"\"{{\n \"$schema\": \"https://developer.microsoft.com/json-schemas/fabric/gitIntegration/platformProperties/2.0.0/schema.json\",\n \"metadata\": {{\n \"type\": \"Report\",\n \"displayName\": \"{accented_name}\",\n \"description\": \"Report with accented characters\"\n }},\n \"config\": {{\n \"version\": \"2.0\",\n \"logicalId\": \"99b570c5-0c79-9dc4-4c9b-fa16c621384c\"\n }}\n}}\n\"\"\"\n platform_file = accented_dir / \".platform\"\n platform_file.write_text(platform_content, encoding=\"utf-8\")\n\n param_obj = Parameter(\n repository_directory=repository_directory,\n item_type_in_scope=item_type_in_scope,\n environment=target_environment,\n parameter_file_name=constants.PARAMETER_FILE_NAME,\n )\n\n # Should find the item with accented characters\n assert param_obj._validate_item_name(accented_name) == (True, \"Valid item name\")\n # Should not find a non-existent item\n assert param_obj._validate_item_name(\"Roles et parcours de formation\") == (\n False,\n constants.PARAMETER_MSGS[\"invalid item name\"].format(\"Roles et parcours de formation\"),\n )\n\n\n@pytest.mark.parametrize(\n (\"param_file_name\", \"result\", \"msg\"),\n [\n (\"no_target_env_parameter.yml\", True, \"valid parameter\"),\n (\"missing_find_val_parameter.yml\", False, \"missing required value\"),\n (\"mismatch_filter_parameter.yml\", True, \"valid parameter\"),\n (\"missing_replace_val_parameter.yml\", False, \"missing required value\"),\n (\"invalid_name_parameter.yml\", False, \"invalid name\"),\n (\"invalid_yaml_struc_parameter.yml\", False, \"invalid load\"),\n (\"invalid_yaml_char_parameter.yml\", False, \"invalid load\"),\n (\"invalid_is_regex_parameter.yml\", False, \"invalid data type\"),\n ],\n)\ndef test_validate_invalid_parameters(\n repository_directory, item_type_in_scope, target_environment, param_file_name, result, msg\n):\n \"\"\"Test the validation of invalid or error-prone parameter files\"\"\"\n param_obj = Parameter(\n repository_directory=repository_directory,\n item_type_in_scope=item_type_in_scope,\n environment=target_environment,\n parameter_file_name=param_file_name,\n )\n\n # Target environment not present in find_replace parameter (error-prone case)\n if param_file_name == \"no_target_env_parameter.yml\":\n assert param_obj._validate_parameter(\"find_replace\") == (\n result,\n constants.PARAMETER_MSGS[msg].format(\"find_replace\"),\n )\n\n # Missing required value in find_replace parameter\n if param_file_name == \"missing_find_val_parameter.yml\":\n for param_dict in param_obj.environment_parameter.get(\"find_replace\"):\n assert param_obj._validate_required_values(\"find_replace\", param_dict) == (\n result,\n constants.PARAMETER_MSGS[msg].format(\"find_value\", \"find_replace\"),\n )\n assert param_obj._validate_parameter_file() == result\n\n # Mismatched optional filters in find_replace parameter (error-prone case)\n if param_file_name == \"mismatch_filter_parameter.yml\":\n assert param_obj._validate_parameter(\"find_replace\") == (\n result,\n constants.PARAMETER_MSGS[msg].format(\"find_replace\"),\n )\n\n # Missing required value in spark_pool parameter\n if param_file_name == \"missing_replace_val_parameter.yml\":\n assert param_obj._validate_parameter(\"spark_pool\") == (\n result,\n constants.PARAMETER_MSGS[msg].format(\"replace_value\", \"spark_pool\"),\n )\n\n # Invalid parameter name\n if param_file_name == \"invalid_name_parameter.yml\":\n assert param_obj._validate_parameter_names() == (\n result,\n constants.PARAMETER_MSGS[msg].format(\"spark_pool_param\"),\n )\n\n # Errors in YAML content structure\n if param_file_name == \"invalid_yaml_struc_parameter.yml\":\n is_valid, msg = param_obj._validate_parameter_load()\n try:\n with Path.open(repository_directory / param_file_name, encoding=\"utf-8\") as yaml_file:\n yaml_content = yaml_file.read()\n yaml.full_load(yaml_content)\n except yaml.YAMLError as e:\n error_message = str(e)\n\n assert is_valid == result\n assert msg == constants.PARAMETER_MSGS[\"invalid load\"].format(error_message)\n\n # Mismatched quotes in YAML content\n if param_file_name == \"invalid_yaml_char_parameter.yml\":\n is_valid, msg = param_obj._validate_parameter_load()\n try:\n with Path.open(repository_directory / param_file_name, encoding=\"utf-8\") as yaml_file:\n yaml_content = yaml_file.read()\n yaml.full_load(yaml_content)\n except yaml.YAMLError as e:\n error_message = str(e)\n\n assert is_valid == result\n assert msg == constants.PARAMETER_MSGS[\"invalid load\"].format(error_message)\n\n # Invalid is_regex value in find_replace parameter\n if param_file_name == \"invalid_is_regex_parameter.yml\":\n # Mock the environment_parameter to have the invalid is_regex (boolean instead of string)\n param_obj.environment_parameter = {\n \"find_replace\": [\n {\n \"find_value\": '\\\\#\\\\s*META\\\\s+\"default_lakehouse\":\\\\s*\"([0-9a-fA-F]{8}-[0-9a-fA-F]{4}-[0-9a-fA-F]{4}-[0-9a-fA-F]{4}-[0-9a-fA-F]{12})\"',\n \"replace_value\": {\n \"PPE\": \"81bbb339-8d0b-46e8-bfa6-289a159c0733\",\n \"PROD\": \"5d6a1b16-447f-464a-b959-45d0fed35ca0\",\n },\n \"is_regex\": True, # This is a boolean, not a string\n }\n ]\n }\n assert param_obj._validate_parameter(\"find_replace\") == (\n result,\n constants.PARAMETER_MSGS[msg].format(\"is_regex\", \"string\", \"find_replace\"),\n )\n\n\ndef test_validate_file_path_scenarios(parameter_object):\n \"\"\"Test _validate_file_path with different scenarios.\"\"\"\n # Test 1: Single invalid path - expects \"no valid file path\" error\n single_invalid_path = [\"nonexistent_file.py\"]\n with mock.patch(\"fabric_cicd._parameter._utils.process_input_path\", return_value=[]):\n result, msg = parameter_object._validate_file_path(single_invalid_path)\n assert result is False\n assert msg == constants.PARAMETER_MSGS[\"no valid file path\"].format(single_invalid_path)\n\n # Test 2: Multiple invalid paths - expects \"no valid file path\" error\n multiple_invalid_paths = [\"nonexistent_file1.py\", \"nonexistent_file2.py\"]\n with mock.patch(\"fabric_cicd._parameter._utils.process_input_path\", return_value=[]):\n result, msg = parameter_object._validate_file_path(multiple_invalid_paths)\n assert result is False\n assert msg == constants.PARAMETER_MSGS[\"no valid file path\"].format(multiple_invalid_paths)\n\n # Test 3: Mixed valid/invalid paths - expects \"invalid file path\" error for missing paths\n mixed_paths = [\"valid_path.py\", \"invalid_path.py\"]\n\n # Create a custom test implementation that simulates the behavior we want to test\n def mock_validate_file_path(input_path):\n \"\"\"Custom implementation to test the mixed valid/invalid paths case\"\"\"\n # For test purposes, we'll simulate that process_input_path returned a valid path\n valid_paths = [Path(\"valid_path.py\")]\n\n # If there are no valid paths, return the \"no valid file path\" error\n if not valid_paths:\n return False, constants.PARAMETER_MSGS[\"no valid file path\"].format(input_path)\n\n # Normalize paths for comparison\n processed_paths = {str(p).replace(\"\\\\\", \"/\") for p in valid_paths}\n original_paths = {str(p).replace(\"\\\\\", \"/\") for p in input_path}\n\n # Find invalid paths\n missing_paths = original_paths - processed_paths\n\n if missing_paths:\n path_diff = len(original_paths) - len(processed_paths)\n return False, constants.PARAMETER_MSGS[\"invalid file path\"].format(input_path, path_diff)\n\n return True, \"Valid file path\"\n\n # Save the original method\n original_method = parameter_object._validate_file_path\n\n # Replace with our mock implementation\n parameter_object._validate_file_path = mock_validate_file_path\n\n try:\n # Call the method with our test data\n result, msg = parameter_object._validate_file_path(mixed_paths)\n\n # Should return False and the invalid file path message\n assert result is False\n\n # The message should contain the invalid path\n assert \"invalid_path.py\" in msg\n\n # Make sure we're getting the specific \"invalid file path\" message\n # We need to match the new format which takes input_path and path_diff\n mixed_paths = [\"valid_path.py\", \"invalid_path.py\"]\n path_diff = 1 # One invalid path\n expected_msg = constants.PARAMETER_MSGS[\"invalid file path\"].format(mixed_paths, path_diff)\n assert msg == expected_msg\n finally:\n # Restore the original method\n parameter_object._validate_file_path = original_method\n\n # Test 4: All valid paths - should return True\n valid_paths = [\"valid_path1.py\", \"valid_path2.py\"]\n\n # Create a mock function that always returns success\n def mock_validate_all_valid(_):\n \"\"\"Mock function that simulates all paths being valid\"\"\"\n return True, \"Valid file path\"\n\n # Save the original method and replace with our mock\n original_method = parameter_object._validate_file_path\n parameter_object._validate_file_path = mock_validate_all_valid\n\n try:\n # Call the method with our test data\n result, msg = parameter_object._validate_file_path(valid_paths)\n assert result is True\n assert msg == \"Valid file path\"\n finally:\n # Restore the original method\n parameter_object._validate_file_path = original_method\n\n\ndef test_validate_all_environment_key_valid():\n \"\"\"Test the validation of _ALL_ environment key in valid scenarios\"\"\"\n # Test that _ALL_ key is accepted as a valid environment\n import tempfile\n\n from fabric_cicd._parameter._parameter import Parameter\n\n # Create a temporary parameter file with _ALL_ environment key\n with tempfile.NamedTemporaryFile(mode=\"w\", suffix=\".yml\", delete=False) as temp_file:\n temp_file.write(\"\"\"\nfind_replace:\n - find_value: \"test-value\"\n replace_value:\n _ALL_: \"universal-value\"\nkey_value_replace:\n - find_key: $.test\n replace_value:\n _All_: \"universal-key-value\"\nspark_pool:\n - instance_pool_id: \"test-pool-id\"\n replace_value:\n _all_:\n type: \"Capacity\"\n name: \"UniversalPool\"\n\"\"\")\n temp_file_path = temp_file.name\n\n try:\n param_obj = Parameter(\n repository_directory=Path(temp_file_path).parent,\n item_type_in_scope=[\"Notebook\"],\n environment=\"TEST\",\n parameter_file_name=Path(temp_file_path).name,\n )\n\n # Should pass validation since _ALL_ is a valid environment key\n assert param_obj._validate_parameter(\"find_replace\") == (\n True,\n constants.PARAMETER_MSGS[\"valid parameter\"].format(\"find_replace\"),\n )\n\n assert param_obj._validate_parameter(\"key_value_replace\") == (\n True,\n constants.PARAMETER_MSGS[\"valid parameter\"].format(\"key_value_replace\"),\n )\n\n assert param_obj._validate_parameter(\"spark_pool\") == (\n True,\n constants.PARAMETER_MSGS[\"valid parameter\"].format(\"spark_pool\"),\n )\n\n # Overall parameter file should be valid\n assert param_obj._validate_parameter_file() == True\n\n # Test that the _ALL_ environment key is properly recognized (case-insensitive)\n for param_dict in param_obj.environment_parameter.get(\"find_replace\"):\n assert param_obj._validate_environment(param_dict[\"replace_value\"]) == (True, \"_ALL_\")\n\n for param_dict in param_obj.environment_parameter.get(\"key_value_replace\"):\n assert param_obj._validate_environment(param_dict[\"replace_value\"]) == (True, \"_All_\")\n\n for param_dict in param_obj.environment_parameter.get(\"spark_pool\"):\n assert param_obj._validate_environment(param_dict[\"replace_value\"]) == (True, \"_all_\")\n\n finally:\n # Clean up temporary file\n Path(temp_file_path).unlink()\n\n\ndef test_validate_all_environment_key_invalid():\n \"\"\"Test validation of _ALL_ environment key in invalid scenarios\"\"\"\n import tempfile\n\n from fabric_cicd._parameter._parameter import Parameter\n\n # Create a parameter file with multiple environment keys including ALL\n with tempfile.NamedTemporaryFile(mode=\"w\", suffix=\".yml\", delete=False) as temp_file:\n temp_file.write(\"\"\"\nfind_replace:\n - find_value: \"test-connection-string\"\n replace_value:\n DEV: \"dev-connection-string\"\n TEST: \"test-connection-string\" \n PROD: \"prod-connection-string\"\n _ALL_: \"universal-connection-string\"\nspark_pool:\n - instance_pool_id: \"multi-env-pool-id\"\n replace_value:\n DEV:\n type: \"Workspace\"\n name: \"DevPool\"\n TEST:\n type: \"Workspace\" \n name: \"TestPool\"\n PROD:\n type: \"Capacity\"\n name: \"ProdPool\"\n _all_:\n type: \"Capacity\"\n name: \"UniversalPool\"\n\"\"\")\n temp_file_path = temp_file.name\n\n try:\n param_obj = Parameter(\n repository_directory=Path(temp_file_path).parent,\n item_type_in_scope=[\"Notebook\"],\n environment=\"TEST\",\n parameter_file_name=Path(temp_file_path).name,\n )\n\n # Should fail validation since ALL cannot coexist with other environment keys\n assert param_obj._validate_parameter(\"find_replace\") == (\n False,\n constants.PARAMETER_MSGS[\"other target env\"].format(\n \"_ALL_\", param_obj.environment_parameter[\"find_replace\"][0][\"replace_value\"]\n ),\n )\n\n assert param_obj._validate_parameter(\"spark_pool\") == (\n False,\n constants.PARAMETER_MSGS[\"other target env\"].format(\n \"_all_\", param_obj.environment_parameter[\"spark_pool\"][0][\"replace_value\"]\n ),\n )\n\n # Overall parameter file should be invalid\n assert param_obj._validate_parameter_file() == False\n\n # Test that mixed environment combinations are invalid\n for param_dict in param_obj.environment_parameter.get(\"find_replace\"):\n assert param_obj._validate_environment(param_dict[\"replace_value\"]) == (False, \"_ALL_\")\n\n for param_dict in param_obj.environment_parameter.get(\"spark_pool\"):\n assert param_obj._validate_environment(param_dict[\"replace_value\"]) == (False, \"_all_\")\n\n finally:\n # Clean up temporary file\n Path(temp_file_path).unlink()\n\n\ndef test_validate_all_environment_key_with_logging():\n \"\"\"Test that _ALL_ environment key triggers appropriate logging\"\"\"\n import tempfile\n\n # Create a temporary parameter file with _ALL_ environment key\n with tempfile.NamedTemporaryFile(mode=\"w\", suffix=\".yml\", delete=False) as temp_file:\n temp_file.write(\"\"\"\nfind_replace:\n - find_value: \"test-value\"\n replace_value:\n _ALL_: \"universal-value\"\nspark_pool:\n - instance_pool_id: \"test-pool-id\"\n replace_value:\n _all_:\n type: \"Capacity\"\n name: \"UniversalPool\"\n\"\"\")\n temp_file_path = temp_file.name\n\n try:\n param_obj = Parameter(\n repository_directory=Path(temp_file_path).parent,\n item_type_in_scope=[\"Notebook\"],\n environment=\"TEST\",\n parameter_file_name=Path(temp_file_path).name,\n )\n\n # Test environment validation specifically for _ALL_ key\n replace_value_with_all = {\"_all_\": \"universal-value\"}\n assert param_obj._validate_environment(replace_value_with_all) == (True, \"_all_\")\n\n # Test spark_pool with _ALL_ environment key\n spark_pool_replace_value_with_all = {\"_ALL_\": {\"type\": \"Capacity\", \"name\": \"UniversalPool\"}}\n assert param_obj._validate_environment(spark_pool_replace_value_with_all) == (True, \"_ALL_\")\n\n # Test environment validation specifically for all key (not reserved)\n replace_value_with_all = {\"all\": \"universal-value\"}\n assert param_obj._validate_environment(replace_value_with_all) == (False, \"env\")\n\n # Test environment validation with both target env and _ALL_ key (should fail)\n replace_value_mixed = {\"TEST\": \"test-value\", \"_ALL_\": \"universal-value\"}\n assert param_obj._validate_environment(replace_value_mixed) == (False, \"_ALL_\")\n\n # Test spark_pool with mixed environment keys (should fail)\n spark_pool_mixed = {\n \"TEST\": {\"type\": \"Workspace\", \"name\": \"TestPool\"},\n \"_ALL_\": {\"type\": \"Capacity\", \"name\": \"UniversalPool\"},\n }\n assert param_obj._validate_environment(spark_pool_mixed) == (False, \"_ALL_\")\n\n # Test environment validation with multiple environment keys including all (not reserved)\n replace_value_multiple_envs = {\"TEST\": \"test-value\", \"PROD\": \"prod-value\", \"all\": \"universal-value\"}\n assert param_obj._validate_environment(replace_value_multiple_envs) == (True, \"env\")\n\n # Test spark_pool with multiple environment keys including ALL (not reserved)\n spark_pool_multiple_envs = {\n \"PROD\": {\"type\": \"Workspace\", \"name\": \"ProdPool\"},\n \"All\": {\"type\": \"Capacity\", \"name\": \"UniversalPool\"},\n }\n assert param_obj._validate_environment(spark_pool_multiple_envs) == (False, \"env\")\n\n # Test environment validation with only target env (no _ALL_ key)\n replace_value_target_only = {\"TEST\": \"test-value\"}\n assert param_obj._validate_environment(replace_value_target_only) == (True, \"env\")\n\n # Test environment validation with neither target env nor _ALL_ key\n replace_value_other = {\"PROD\": \"prod-value\"}\n assert param_obj._validate_environment(replace_value_other) == (False, \"env\")\n\n finally:\n # Clean up temporary file\n Path(temp_file_path).unlink()\n\n\ndef test_parameter_file_path_absolute():\n \"\"\"Test that Parameter class accepts absolute parameter_file_path.\"\"\"\n import tempfile\n\n with tempfile.NamedTemporaryFile(mode=\"w\", suffix=\".yml\", delete=False) as temp_file:\n temp_file.write(\"\"\"\nfind_replace:\n - find_value: \"test-value\"\n replace_value:\n TEST: \"test-replacement\"\n\"\"\")\n temp_file_path = temp_file.name\n\n try:\n param_obj = Parameter(\n repository_directory=Path(temp_file_path).parent,\n item_type_in_scope=[\"Notebook\"],\n environment=\"TEST\",\n parameter_file_path=temp_file_path,\n )\n\n # Should work without errors\n assert param_obj.environment == \"TEST\"\n assert param_obj.item_type_in_scope == [\"Notebook\"]\n\n finally:\n Path(temp_file_path).unlink()\n\n\ndef test_parameter_file_path_relative():\n \"\"\"Test that Parameter class handles relative parameter_file_path by resolving it against repository_directory.\"\"\"\n import tempfile\n from pathlib import Path\n\n # Create a temporary directory to act as the repository\n with tempfile.TemporaryDirectory() as temp_dir:\n repo_dir = Path(temp_dir)\n\n # Create a nested directory and parameter file\n relative_dir = \"relative/path\"\n (repo_dir / relative_dir).mkdir(parents=True)\n\n param_file = \"parameters.yml\"\n param_file_path = Path(repo_dir, relative_dir, param_file)\n param_file_path.write_text(\"key: value\") # Simple valid YAML\n\n # Test with relative path that exists\n relative_path = f\"{relative_dir}/{param_file}\"\n\n # Create a Parameter instance with a relative path\n param = Parameter(\n repository_directory=repo_dir,\n item_type_in_scope=[\"Notebook\"],\n environment=\"TEST\",\n parameter_file_path=relative_path,\n )\n\n # Verify the path was resolved relative to repository_directory\n expected_path = param_file_path.resolve()\n assert param.parameter_file_path == expected_path\n\n # Test with relative path that doesn't exist\n non_existent_path = \"relative/path/non_existent.yml\"\n\n # This should not raise an error but should log an error message\n param2 = Parameter(\n repository_directory=repo_dir,\n item_type_in_scope=[\"Notebook\"],\n environment=\"TEST\",\n parameter_file_path=non_existent_path,\n )\n\n # Verify the path was resolved but parameter loading failed\n assert param2.parameter_file_path is not None\n assert not param2.environment_parameter\n\n # Test with path that exists but is a directory, not a file\n param3 = Parameter(\n repository_directory=repo_dir,\n item_type_in_scope=[\"Notebook\"],\n environment=\"TEST\",\n parameter_file_path=relative_dir,\n )\n\n # Verify the path was resolved but parameter loading failed\n assert param3.parameter_file_path is not None\n assert not param3.environment_parameter\n\n\ndef test_parameter_file_path_none():\n \"\"\"Test that Parameter class accepts None for parameter_file_path (uses parameter_file_name).\"\"\"\n import tempfile\n\n with tempfile.TemporaryDirectory() as temp_dir:\n temp_dir_path = Path(temp_dir)\n param_file = temp_dir_path / \"parameters.yml\"\n param_file.write_text(\"\"\"\nfind_replace:\n - find_value: \"test-value\"\n replace_value:\n TEST: \"test-replacement\"\n\"\"\")\n\n param_obj = Parameter(\n repository_directory=temp_dir_path,\n item_type_in_scope=[\"Notebook\"],\n environment=\"TEST\",\n parameter_file_name=\"parameters.yml\",\n parameter_file_path=None,\n )\n\n # Should work with parameter_file_name fallback\n assert param_obj.environment == \"TEST\"\n assert param_obj.item_type_in_scope == [\"Notebook\"]\n # Verify the parameter_file_path was set correctly from parameter_file_name\n assert param_obj.parameter_file_path == (temp_dir_path / \"parameters.yml\").resolve()\n\n\ndef test_parameter_file_path_and_name_inputs():\n \"\"\"Test that parameter_file_path takes precedence over parameter_file_name.\"\"\"\n import tempfile\n\n with tempfile.TemporaryDirectory() as temp_dir:\n temp_dir_path = Path(temp_dir)\n\n # Create file referenced by parameter_file_name\n fallback_file = temp_dir_path / \"parameters.yml\"\n fallback_file.write_text(\"\"\"\nfind_replace:\n - find_value: \"fallback-value\"\n replace_value:\n TEST: \"fallback-replacement\"\n\"\"\")\n\n # Create file referenced by parameter_file_path\n primary_file = temp_dir_path / \"primary_parameters.yml\"\n primary_file.write_text(\"\"\"\nfind_replace:\n - find_value: \"primary-value\"\n replace_value:\n TEST: \"primary-replacement\"\n\"\"\")\n\n param_obj = Parameter(\n repository_directory=temp_dir_path,\n item_type_in_scope=[\"Notebook\"],\n environment=\"TEST\",\n parameter_file_name=\"parameters.yml\", # This should be ignored\n parameter_file_path=str(primary_file), # This should be used\n )\n\n # Should use primary_file content\n assert param_obj.environment == \"TEST\"\n # Check that the primary file was used by examining parameter content\n assert \"find_replace\" in param_obj.environment_parameter\n assert len(param_obj.environment_parameter[\"find_replace\"]) == 1\n assert param_obj.environment_parameter[\"find_replace\"][0][\"find_value\"] == \"primary-value\"\n\n\ndef test_no_provided_parameter_file_path():\n \"\"\"Test that default behavior without parameter_file_path remains unchanged.\"\"\"\n import tempfile\n\n with tempfile.TemporaryDirectory() as temp_dir:\n temp_dir_path = Path(temp_dir)\n param_file = temp_dir_path / \"parameters.yml\"\n param_file.write_text(\"\"\"\nfind_replace:\n - find_value: \"test-value\"\n replace_value:\n TEST: \"test-replacement\"\n\"\"\")\n\n # Original behavior - no parameter_file_path specified\n param_obj = Parameter(\n repository_directory=temp_dir_path,\n item_type_in_scope=[\"Notebook\"],\n environment=\"TEST\",\n parameter_file_name=\"parameters.yml\",\n )\n\n # Should work exactly as before\n assert param_obj.environment == \"TEST\"\n assert param_obj.item_type_in_scope == [\"Notebook\"]\n assert \"find_replace\" in param_obj.environment_parameter\n assert len(param_obj.environment_parameter[\"find_replace\"]) == 1\n\n\ndef test_parameter_file_path_nonexistent():\n \"\"\"Test behavior when parameter_file_path points to nonexistent file.\"\"\"\n import tempfile\n\n with tempfile.TemporaryDirectory() as temp_dir:\n nonexistent_path = str(Path(temp_dir) / \"nonexistent\" / \"parameters.yml\")\n\n # Should log an error but not raise an exception\n param = Parameter(\n repository_directory=Path.cwd(),\n item_type_in_scope=[\"Notebook\"],\n environment=\"TEST\",\n parameter_file_path=nonexistent_path,\n )\n\n # Parameter file path should be set but the environment_parameter should be empty\n assert param.parameter_file_path is not None\n assert not param.environment_parameter\n\n\ndef test_validate_parameter_file_exists_none():\n \"\"\"Test that _validate_parameter_file_exists returns False when parameter_file_path is None.\"\"\"\n import tempfile\n\n with tempfile.TemporaryDirectory() as temp_dir:\n # Create a Parameter instance with parameter_file_path set to None in _set_parameter_file_path\n param = Parameter(\n repository_directory=Path(temp_dir),\n item_type_in_scope=[\"Notebook\"],\n environment=\"TEST\",\n parameter_file_name=\"does_not_exist.yml\", # This file doesn't exist\n )\n\n # Force parameter_file_path to None\n param.parameter_file_path = None\n\n # Method should return False without raising errors\n assert param._validate_parameter_file_exists() is False\n\n\ndef test_parameter_file_path_invalid_type():\n \"\"\"Test that Parameter class handles invalid types for parameter_file_path.\"\"\"\n import tempfile\n\n with tempfile.TemporaryDirectory() as temp_dir:\n # Parameter class should handle the invalid type internally without raising an exception\n param = Parameter(\n repository_directory=Path(temp_dir),\n item_type_in_scope=[\"Notebook\"],\n environment=\"TEST\",\n parameter_file_path=123, # Invalid type\n )\n\n # The error handling in _set_parameter_file_path sets is_param_path to False\n # and falls back to the default parameter file path\n assert param.parameter_file_path is not None\n assert param.parameter_file_path == (Path(temp_dir) / \"parameter.yml\").resolve()\n\n\ndef test_set_parameter_file_path_error_handling():\n \"\"\"Test error handling in _set_parameter_file_path method.\"\"\"\n import tempfile\n\n # Create a mock that raises an exception when called with any arguments\n path_mock = mock.Mock(side_effect=Exception(\"Simulated error\"))\n\n with tempfile.TemporaryDirectory() as temp_dir, mock.patch(\"fabric_cicd._parameter._parameter.Path\", path_mock):\n # Create parameter with both parameters to test the error handling\n param = Parameter(\n repository_directory=temp_dir, # Using string path to avoid early Path conversion\n item_type_in_scope=[\"Notebook\"],\n environment=\"TEST\",\n parameter_file_name=\"parameters.yml\",\n parameter_file_path=\"custom_path.yml\",\n )\n\n # The method should have caught the exception and set parameter_file_path to None\n assert param.parameter_file_path is None\n\n\ndef test_basic_template_processing(tmp_path):\n \"\"\"Test basic template parameter file processing with valid files.\"\"\"\n # Setup repository structure\n repo_dir = tmp_path / \"repo\"\n repo_dir.mkdir()\n templates_dir = repo_dir / \"templates\"\n templates_dir.mkdir()\n\n # Create base parameter file\n base_file = repo_dir / \"parameter.yml\"\n base_content = \"\"\"\n extend:\n - ./templates/template1.yml\n find_replace:\n - find_value: \"base-id\"\n replace_value:\n DEV: \"dev-base\"\n PROD: \"prod-base\"\n \"\"\"\n base_file.write_text(base_content)\n\n # Create template file\n template_file = templates_dir / \"template1.yml\"\n template_content = \"\"\"\n find_replace:\n - find_value: \"template-id\"\n replace_value:\n DEV: \"dev-template\"\n PROD: \"prod-template\"\n spark_pool:\n - instance_pool_id: \"pool-id\"\n replace_value:\n DEV:\n type: \"Workspace\"\n name: \"dev-pool\"\n \"\"\"\n template_file.write_text(template_content)\n\n # Initialize parameter object\n param = Parameter(repository_directory=repo_dir, item_type_in_scope=[\"Notebook\"], environment=\"DEV\")\n\n # Verify template processing results\n assert \"extend\" not in param.environment_parameter\n assert len(param.environment_parameter[\"find_replace\"]) == 2\n assert \"spark_pool\" in param.environment_parameter\n\n # Verify specific values were merged correctly\n find_values = [item[\"find_value\"] for item in param.environment_parameter[\"find_replace\"]]\n assert \"base-id\" in find_values\n assert \"template-id\" in find_values\n\n\ndef test_missing_templates_directory(tmp_path):\n \"\"\"Test handling of missing templates directory.\"\"\"\n # Setup repository without templates directory\n repo_dir = tmp_path / \"repo\"\n repo_dir.mkdir()\n\n # Create base parameter file\n base_file = repo_dir / \"parameter.yml\"\n base_content = \"\"\"\n extend:\n - template1.yml\n find_replace:\n - find_value: \"base-id\"\n replace_value:\n DEV: \"dev-base\"\n \"\"\"\n base_file.write_text(base_content)\n\n # Initialize parameter object\n param = Parameter(repository_directory=repo_dir, item_type_in_scope=[\"Notebook\"], environment=\"DEV\")\n\n # Verify base parameters remain but extend key is removed\n assert \"extend\" not in param.environment_parameter\n assert len(param.environment_parameter[\"find_replace\"]) == 1\n assert param.environment_parameter[\"find_replace\"][0][\"find_value\"] == \"base-id\"\n\n\ndef test_nested_template_prevention(tmp_path):\n \"\"\"Test prevention of nested template extensions.\"\"\"\n # Setup repository structure\n repo_dir = tmp_path / \"repo\"\n repo_dir.mkdir()\n templates_dir = repo_dir / \"templates\"\n templates_dir.mkdir()\n\n # Create base parameter file\n base_file = repo_dir / \"parameter.yml\"\n base_content = \"\"\"\n extend:\n - parent.yml\n find_replace:\n - find_value: \"base-id\"\n replace_value:\n DEV: \"dev-base\"\n \"\"\"\n base_file.write_text(base_content)\n\n # Create parent template with nested extend\n parent_file = templates_dir / \"parent.yml\"\n parent_content = \"\"\"\n extend:\n - child.yml\n find_replace:\n - find_value: \"parent-id\"\n replace_value:\n DEV: \"dev-parent\"\n \"\"\"\n parent_file.write_text(parent_content)\n\n # Create child template\n child_file = templates_dir / \"child.yml\"\n child_content = \"\"\"\n find_replace:\n - find_value: \"child-id\"\n replace_value:\n DEV: \"dev-child\"\n \"\"\"\n child_file.write_text(child_content)\n\n # Initialize parameter object\n param = Parameter(repository_directory=repo_dir, item_type_in_scope=[\"Notebook\"], environment=\"DEV\")\n\n # Verify nested template was prevented\n assert \"extend\" not in param.environment_parameter # extend key should be removed regardless\n assert len(param.environment_parameter[\"find_replace\"]) == 1 # only base parameters should be processed\n find_values = [item[\"find_value\"] for item in param.environment_parameter[\"find_replace\"]]\n assert \"base-id\" in find_values\n assert \"parent-id\" not in find_values # parent template should be skipped due to nested extend\n assert \"child-id\" not in find_values # child template should not be processed\n\n\ndef test_template_path_resolution(tmp_path):\n \"\"\"Test that template files are resolved relative to the parameter file location.\"\"\"\n # Setup repository structure\n repo_dir = tmp_path / \"repo\"\n repo_dir.mkdir()\n\n # Create a directory at the same level as repo for \"outside\" templates\n shared_dir = tmp_path / \"shared\"\n shared_dir.mkdir()\n\n # Create base parameter file\n base_file = repo_dir / \"parameter.yml\"\n base_content = \"\"\"\n extend:\n - normal.yml # Same directory\n - ../shared/shared.yml # Outside repo (should work now)\n - /absolute/path.yml # Absolute path that doesn't exist (should fail)\n - nonexistent.yml # File doesn't exist (should fail)\n find_replace:\n - find_value: \"base-id\"\n replace_value:\n DEV: \"dev-base\"\n \"\"\"\n base_file.write_text(base_content)\n\n # Create template in same directory\n normal_file = repo_dir / \"normal.yml\"\n normal_content = \"\"\"\n find_replace:\n - find_value: \"normal-id\"\n replace_value:\n DEV: \"dev-normal\"\n \"\"\"\n normal_file.write_text(normal_content)\n\n # Create shared template outside repo\n shared_file = shared_dir / \"shared.yml\"\n shared_content = \"\"\"\n find_replace:\n - find_value: \"shared-id\"\n replace_value:\n DEV: \"dev-shared\"\n \"\"\"\n shared_file.write_text(shared_content)\n\n # Initialize parameter object\n param = Parameter(repository_directory=repo_dir, item_type_in_scope=[\"Notebook\"], environment=\"DEV\")\n\n # Verify template processing results\n assert \"extend\" not in param.environment_parameter\n # Should have: base, normal, and shared (3 total)\n assert len(param.environment_parameter[\"find_replace\"]) == 3\n find_values = {item[\"find_value\"] for item in param.environment_parameter[\"find_replace\"]}\n assert find_values == {\"base-id\", \"normal-id\", \"shared-id\"}\n\n\ndef test_missing_template_files(tmp_path):\n \"\"\"Test that missing template files are handled gracefully.\"\"\"\n repo_dir = tmp_path / \"repo\"\n repo_dir.mkdir()\n\n base_file = repo_dir / \"parameter.yml\"\n base_content = \"\"\"\n extend:\n - existing.yml\n - missing.yml\n - /absolute/missing.yml\n find_replace:\n - find_value: \"base-id\"\n replace_value:\n DEV: \"dev-base\"\n \"\"\"\n base_file.write_text(base_content)\n\n existing_file = repo_dir / \"existing.yml\"\n existing_file.write_text(\"\"\"\n find_replace:\n - find_value: \"existing-id\"\n replace_value:\n DEV: \"dev-existing\"\n \"\"\")\n\n param = Parameter(repository_directory=repo_dir, item_type_in_scope=[\"Notebook\"], environment=\"DEV\")\n\n # Only base and existing should be loaded\n assert len(param.environment_parameter[\"find_replace\"]) == 2\n find_values = {item[\"find_value\"] for item in param.environment_parameter[\"find_replace\"]}\n assert find_values == {\"base-id\", \"existing-id\"}\n\n\ndef test_template_merge_validation(tmp_path):\n \"\"\"Test validation of merged template content.\"\"\"\n # Setup repository structure\n repo_dir = tmp_path / \"repo\"\n repo_dir.mkdir()\n templates_dir = repo_dir / \"templates\"\n templates_dir.mkdir()\n\n # Create base parameter file\n base_file = repo_dir / \"parameter.yml\"\n base_content = \"\"\"\n extend:\n - ./templates/template1.yml\n - ./templates/invalid.yml\n find_replace:\n - find_value: \"base-id\"\n replace_value:\n DEV: \"dev-base\"\n \"\"\"\n base_file.write_text(base_content)\n\n # Create valid template\n template1_file = templates_dir / \"template1.yml\"\n template1_content = \"\"\"\n find_replace:\n - find_value: \"template-id\"\n replace_value:\n DEV: \"dev-template\"\n \"\"\"\n template1_file.write_text(template1_content)\n\n # Create invalid template\n invalid_file = templates_dir / \"invalid.yml\"\n invalid_content = \"\"\"\n find_replace:\n - replace_value:\n DEV: \"dev-invalid\"\n optional_field: \"value\"\n \"\"\"\n invalid_file.write_text(invalid_content)\n\n # Initialize parameter object\n param = Parameter(repository_directory=repo_dir, item_type_in_scope=[\"Notebook\"], environment=\"DEV\")\n\n # Verify all content was initially merged\n assert \"extend\" not in param.environment_parameter\n assert len(param.environment_parameter[\"find_replace\"]) == 3 # All entries are merged\n\n # Verify the merged content includes both valid and invalid entries (by design)\n entries = param.environment_parameter[\"find_replace\"]\n assert any(e.get(\"find_value\") == \"base-id\" for e in entries) # Base entry\n assert any(e.get(\"find_value\") == \"template-id\" for e in entries) # Valid template\n assert any(e.get(\"optional_field\") == \"value\" for e in entries) # Invalid template entry\n\n # Verify that validation fails due to invalid content\n is_valid, message = param._validate_parameter(\"find_replace\")\n assert is_valid == False\n assert message == constants.PARAMETER_MSGS[\"missing key\"].format(\"find_replace\")\n assert param._validate_parameter_file() == False\n\n\ndef test_template_duplicate_keys_detected(tmp_path):\n \"\"\"Test that duplicate keys in template files are detected by _DuplicateKeyLoader.\"\"\"\n repo_dir = tmp_path / \"repo\"\n repo_dir.mkdir()\n templates_dir = repo_dir / \"templates\"\n templates_dir.mkdir()\n\n # Create base parameter file\n base_file = repo_dir / \"parameter.yml\"\n base_content = \"\"\"\nextend:\n - ./templates/duplicate_template.yml\nfind_replace:\n - find_value: \"base-id\"\n replace_value:\n DEV: \"dev-base\"\n\"\"\"\n base_file.write_text(base_content)\n\n # Create template file with duplicate keys\n template_file = templates_dir / \"duplicate_template.yml\"\n template_content = \"\"\"\nfind_replace:\n - find_value: \"template-id\"\n replace_value:\n DEV: \"dev-template\"\nfind_replace:\n - find_value: \"duplicate-template-id\"\n replace_value:\n DEV: \"dev-duplicate\"\n\"\"\"\n template_file.write_text(template_content)\n\n # Initialize parameter object\n param = Parameter(repository_directory=repo_dir, item_type_in_scope=[\"Notebook\"], environment=\"DEV\")\n\n # Template with duplicate keys should be skipped, only base content should remain\n assert \"extend\" not in param.environment_parameter\n assert len(param.environment_parameter[\"find_replace\"]) == 1\n assert param.environment_parameter[\"find_replace\"][0][\"find_value\"] == \"base-id\"\n\n\ndef test_circular_template_reference(tmp_path):\n \"\"\"Test handling of circular template references.\"\"\"\n repo_dir = tmp_path / \"repo\"\n repo_dir.mkdir()\n templates_dir = repo_dir / \"templates\"\n templates_dir.mkdir()\n\n # Create base parameter file that references template1\n base_file = repo_dir / \"parameter.yml\"\n base_content = \"\"\"\n extend:\n - template1.yml\n find_replace:\n - find_value: \"base-id\"\n replace_value:\n DEV: \"dev-base\"\n \"\"\"\n base_file.write_text(base_content)\n\n # Create template1 that references template2\n template1_file = templates_dir / \"template1.yml\"\n template1_content = \"\"\"\n extend:\n - template2.yml\n find_replace:\n - find_value: \"template1-id\"\n replace_value:\n DEV: \"dev-template1\"\n \"\"\"\n template1_file.write_text(template1_content)\n\n # Create template2 that references template1 (circular)\n template2_file = templates_dir / \"template2.yml\"\n template2_content = \"\"\"\n extend:\n - template1.yml\n find_replace:\n - find_value: \"template2-id\"\n replace_value:\n DEV: \"dev-template2\"\n \"\"\"\n template2_file.write_text(template2_content)\n\n # Initialize parameter object\n param = Parameter(repository_directory=repo_dir, item_type_in_scope=[\"Notebook\"], environment=\"DEV\")\n\n # Verify only base content remains due to circular reference detection\n assert \"extend\" not in param.environment_parameter\n assert len(param.environment_parameter[\"find_replace\"]) == 1\n assert param.environment_parameter[\"find_replace\"][0][\"find_value\"] == \"base-id\"\n\n\ndef test_multiple_template_references(tmp_path):\n \"\"\"Test handling of multiple template references with various scenarios.\"\"\"\n repo_dir = tmp_path / \"repo\"\n repo_dir.mkdir()\n templates_dir = repo_dir / \"templates\"\n templates_dir.mkdir()\n\n # Create different types of template files\n template_configs = [\n # Basic template with single find_replace\n (\n \"template1.yml\",\n \"\"\"\nfind_replace:\n - find_value: \"template1-id\"\n replace_value:\n DEV: \"dev-template1\"\n PROD: \"prod-template1\"\n\"\"\",\n ),\n # Template with multiple find_replace entries\n (\n \"template2.yml\",\n \"\"\"\nfind_replace:\n - find_value: \"template2-id1\"\n replace_value:\n DEV: \"dev-template2-1\"\n - find_value: \"template2-id2\"\n replace_value:\n DEV: \"dev-template2-2\"\n\"\"\",\n ),\n # Template with regex and item filters\n (\n \"template3.yml\",\n \"\"\"\nfind_replace:\n - find_value: \"template3-.*\"\n is_regex: \"true\"\n item_type: \"Notebook\"\n item_name: \"Test Notebook\"\n replace_value:\n DEV: \"dev-template3\"\n\"\"\",\n ),\n # Template with _ALL_ environment\n (\n \"template4.yml\",\n \"\"\"\nfind_replace:\n - find_value: \"template4-id\"\n replace_value:\n _ALL_: \"all-template4\"\n\"\"\",\n ),\n # Template with key_value_replace\n (\n \"template5.yml\",\n \"\"\"\nkey_value_replace:\n - find_key: \"connectionString\"\n replace_value:\n DEV: \"dev-connection\"\n PROD: \"prod-connection\"\n\"\"\",\n ),\n ]\n\n # Create template files\n template_refs = []\n for template_name, content in template_configs:\n template_refs.append(template_name)\n template_file = templates_dir / template_name\n template_file.write_text(content.strip(), encoding=\"utf-8\")\n\n # Create base parameter file\n base_file = repo_dir / \"parameter.yml\"\n template_refs_with_path = [f\"./templates/{ref}\" for ref in template_refs]\n base_content = \"\"\"\nfind_replace:\n - find_value: \"base-id\"\n replace_value:\n DEV: \"dev-base\"\n PROD: \"prod-base\"\n - find_value: \"base-regex-.*\"\n is_regex: \"true\"\n replace_value:\n DEV: \"dev-base-regex\"\nkey_value_replace:\n - find_key: \"baseKey\"\n replace_value:\n DEV: \"dev-base-value\"\nextend:\n\"\"\" + yaml.safe_dump(template_refs_with_path, allow_unicode=True, indent=2)\n\n base_file.write_text(base_content.strip(), encoding=\"utf-8\")\n\n # Create a test notebook item for validation\n notebook_dir = repo_dir / \"TestNotebook\"\n notebook_dir.mkdir()\n platform_file = notebook_dir / \".platform\"\n platform_content = {\"metadata\": {\"type\": \"Notebook\", \"displayName\": \"Test Notebook\"}}\n platform_file.write_text(json.dumps(platform_content), encoding=\"utf-8\")\n\n # Test with DEV environment\n param_dev = Parameter(repository_directory=repo_dir, item_type_in_scope=[\"Notebook\"], environment=\"DEV\")\n\n # Validate basic file operations\n assert param_dev._validate_parameter_file_exists(), \"Parameter file does not exist\"\n is_valid, message = param_dev._validate_load_parameters_to_dict()\n assert is_valid, f\"Failed to load parameters: {message}\"\n\n # Validate merged parameters\n find_replace_params = param_dev.environment_parameter[\"find_replace\"]\n key_value_params = param_dev.environment_parameter.get(\"key_value_replace\", [])\n\n # Test base parameter presence\n assert any(p[\"find_value\"] == \"base-id\" for p in find_replace_params), \"Base find_replace missing\"\n assert any(p[\"find_value\"] == \"base-regex-.*\" for p in find_replace_params), \"Base regex find_replace missing\"\n assert any(p[\"find_key\"] == \"baseKey\" for p in key_value_params), \"Base key_value_replace missing\"\n\n # Test template merging\n assert any(p[\"find_value\"] == \"template1-id\" for p in find_replace_params), \"Template1 not merged\"\n assert any(p[\"find_value\"] == \"template2-id1\" for p in find_replace_params), \"Template2 first entry not merged\"\n assert any(p[\"find_value\"] == \"template2-id2\" for p in find_replace_params), \"Template2 second entry not merged\"\n assert any(p[\"find_value\"] == \"template3-.*\" for p in find_replace_params), \"Template3 regex not merged\"\n assert any(p[\"find_value\"] == \"template4-id\" for p in find_replace_params), \"Template4 _ALL_ not merged\"\n assert any(p[\"find_key\"] == \"connectionString\" for p in key_value_params), \"Template5 key_value_replace not merged\"\n\n # Test regex validation\n regex_entries = [p for p in find_replace_params if p.get(\"is_regex\") == \"true\"]\n assert len(regex_entries) == 2, \"Expected exactly 2 regex entries\"\n for entry in regex_entries:\n assert re.compile(entry[\"find_value\"]), f\"Invalid regex pattern: {entry['find_value']}\"\n\n # Test item filtering\n filtered_entries = [p for p in find_replace_params if p.get(\"item_name\") or p.get(\"item_type\")]\n assert len(filtered_entries) == 1, \"Expected exactly 1 filtered entry\"\n filtered_entry = filtered_entries[0]\n assert filtered_entry[\"item_type\"] == \"Notebook\", \"Incorrect item type filter\"\n assert filtered_entry[\"item_name\"] == \"Test Notebook\", \"Incorrect item name filter\"\n\n # Test _ALL_ environment handling\n all_env_entries = [p for p in find_replace_params if \"_ALL_\" in p[\"replace_value\"]]\n assert len(all_env_entries) == 1, \"Expected exactly 1 _ALL_ environment entry\"\n assert all_env_entries[0][\"replace_value\"][\"_ALL_\"] == \"all-template4\", \"Incorrect _ALL_ value\"\n\n # Test with PROD environment\n param_prod = Parameter(repository_directory=repo_dir, item_type_in_scope=[\"Notebook\"], environment=\"PROD\")\n assert param_prod._validate_parameter_file_exists(), \"Parameter file does not exist for PROD\"\n is_valid, message = param_prod._validate_parameter(\"find_replace\")\n assert is_valid, f\"Parameter validation failed for PROD: {message}\"\n\n # Verify environment-specific values\n prod_params = param_prod.environment_parameter[\"find_replace\"]\n assert any(\n p[\"find_value\"] == \"template1-id\" and p[\"replace_value\"][\"PROD\"] == \"prod-template1\" for p in prod_params\n ), \"PROD environment value not correctly loaded\"\n\n\ndef test_template_merge_behavior(tmp_path):\n \"\"\"Test template merging behavior including order, duplicates, and identical entries.\"\"\"\n repo_dir = tmp_path / \"repo\"\n repo_dir.mkdir()\n templates_dir = repo_dir / \"templates\"\n templates_dir.mkdir()\n\n # Create base parameter file\n base_file = repo_dir / \"parameter.yml\"\n base_content = \"\"\"\n extend:\n - ./templates/template1.yml\n - ./templates/template1.yml # Duplicate reference\n - ./templates/template2.yml\n find_replace:\n - find_value: \"id-1\"\n replace_value:\n DEV: \"base-1\"\n PROD: \"base-2\"\n \"\"\"\n base_file.write_text(base_content)\n\n # Create template1 with identical and different entries\n template1_file = templates_dir / \"template1.yml\"\n template1_content = \"\"\"\n find_replace:\n - find_value: \"id-1\" # Identical to base\n replace_value:\n DEV: \"base-1\"\n PROD: \"base-2\"\n - find_value: \"id-2\" # Unique entry\n replace_value:\n DEV: \"template1-1\"\n PROD: \"template1-2\"\n \"\"\"\n template1_file.write_text(template1_content)\n\n # Create template2 with different values\n template2_file = templates_dir / \"template2.yml\"\n template2_content = \"\"\"\n find_replace:\n - find_value: \"id-1\" # Different values\n replace_value:\n DEV: \"template2-1\"\n PROD: \"template2-2\"\n \"\"\"\n template2_file.write_text(template2_content)\n\n # Initialize parameter object\n param = Parameter(repository_directory=repo_dir, item_type_in_scope=[\"Notebook\"], environment=\"DEV\")\n\n # Test 1: Template deduplication\n find_values = {item[\"find_value\"] for item in param.environment_parameter[\"find_replace\"]}\n assert len(find_values) == 2, \"Duplicate template references should be processed only once\"\n\n # Test 2: Merge order preservation\n find_replace_entries = param.environment_parameter[\"find_replace\"]\n assert find_replace_entries[0][\"find_value\"] == \"id-1\" # Base entry first\n assert find_replace_entries[-1][\"find_value\"] == \"id-1\" # Template2 entry last\n\n # Test 3: Value preservation\n dev_values = {item[\"replace_value\"][\"DEV\"] for item in find_replace_entries if item[\"find_value\"] == \"id-1\"}\n assert \"base-1\" in dev_values, \"Base values should be preserved\"\n assert \"template2-1\" in dev_values, \"Template values should be preserved\"\n\n\ndef test_template_reference_handling(tmp_path):\n \"\"\"Test template reference handling including circular references and deep nesting.\"\"\"\n repo_dir = tmp_path / \"repo\"\n repo_dir.mkdir()\n templates_dir = repo_dir / \"templates\"\n templates_dir.mkdir()\n\n # Create base file\n base_file = repo_dir / \"parameter.yml\"\n base_content = \"\"\"\n extend:\n - circular1.yml\n - deep1.yml\n find_replace:\n - find_value: \"base-id\"\n replace_value:\n DEV: \"dev-base\"\n \"\"\"\n base_file.write_text(base_content)\n\n # Create circular reference templates\n circular1_content = \"\"\"\n extend:\n - circular2.yml\n find_replace:\n - find_value: \"circular1-id\"\n replace_value:\n DEV: \"dev-circular1\"\n \"\"\"\n (templates_dir / \"circular1.yml\").write_text(circular1_content)\n\n circular2_content = \"\"\"\n extend:\n - circular1.yml\n find_replace:\n - find_value: \"circular2-id\"\n replace_value:\n DEV: \"dev-circular2\"\n \"\"\"\n (templates_dir / \"circular2.yml\").write_text(circular2_content)\n\n # Create deep nesting templates\n for i in range(1, 6):\n template_content = f\"\"\"\n extend:\n - deep{i + 1}.yml\n find_replace:\n - find_value: \"deep{i}-id\"\n replace_value:\n DEV: \"dev-deep{i}\"\n \"\"\"\n (templates_dir / f\"deep{i}.yml\").write_text(template_content)\n\n # Create final template in deep chain\n final_content = \"\"\"\n find_replace:\n - find_value: \"deep6-id\"\n replace_value:\n DEV: \"dev-deep6\"\n \"\"\"\n (templates_dir / \"deep6.yml\").write_text(final_content)\n\n # Initialize parameter object\n param = Parameter(repository_directory=repo_dir, item_type_in_scope=[\"Notebook\"], environment=\"DEV\")\n\n # Test 1: Circular reference handling\n circular_values = {\n item[\"find_value\"] for item in param.environment_parameter[\"find_replace\"] if \"circular\" in item[\"find_value\"]\n }\n assert not circular_values, \"Circular references should be prevented\"\n\n # Test 2: Deep nesting limit\n deep_entries = [item for item in param.environment_parameter[\"find_replace\"] if \"deep\" in item[\"find_value\"]]\n assert len(deep_entries) <= 5, \"Deep nesting should be limited\"\n\n # Test 3: Base content preservation\n assert any(item[\"find_value\"] == \"base-id\" for item in param.environment_parameter[\"find_replace\"]), (\n \"Base content should be preserved\"\n )\n\n\n@pytest.fixture\ndef empty_parameter(tmp_path):\n # Parameter expects a repository directory; use an empty temporary path.\n return Parameter(repository_directory=tmp_path, item_type_in_scope=[\"Notebook\"], environment=\"DEV\")\n\n\ndef test_validate_key_value_find_key_valid_dot_notation(empty_parameter):\n param = {\"find_key\": \"$.server.host\"}\n ok, msg = empty_parameter._validate_key_value_find_key(param)\n assert ok is True\n assert msg == \"Valid JSONPath\"\n\n\ndef test_validate_key_value_find_key_valid_filter_syntax(empty_parameter):\n param = {\"find_key\": '$.variables[?(@.name==\"SQL_Server\")].value'}\n ok, msg = empty_parameter._validate_key_value_find_key(param)\n assert ok is True\n assert msg == \"Valid JSONPath\"\n\n\ndef test_validate_key_value_find_key_missing_key(empty_parameter):\n param = {}\n ok, msg = empty_parameter._validate_key_value_find_key(param)\n assert ok is False\n assert \"Missing or empty 'find_key'\" in msg\n\n\ndef test_validate_key_value_find_key_non_string(empty_parameter):\n param = {\"find_key\": 123}\n ok, msg = empty_parameter._validate_key_value_find_key(param)\n assert ok is False\n assert \"Missing or empty 'find_key'\" in msg\n\n\ndef test_validate_key_value_find_key_empty_string(empty_parameter):\n param = {\"find_key\": \"\"}\n ok, msg = empty_parameter._validate_key_value_find_key(param)\n assert ok is False\n assert \"Missing or empty 'find_key'\" in msg\n\n\ndef test_validate_key_value_find_key_requires_root(empty_parameter):\n param = {\"find_key\": 'variables[?(@.name==\"SQL_Server\")].value'}\n ok, msg = empty_parameter._validate_key_value_find_key(param)\n assert ok is False\n assert \"must be an absolute JSONPath\" in msg\n\n\ndef test_validate_key_value_find_key_unbalanced_filter(empty_parameter):\n param = {\"find_key\": '$.variables[?(@.name==\"SQL_Server\"].value'}\n ok, msg = empty_parameter._validate_key_value_find_key(param)\n assert ok is False\n assert \"Invalid JSONPath expression\" in msg\n\n\ndef test_validate_key_value_find_key_unsupported_regex_operator(empty_parameter):\n # expressions using =~ are commonly unsupported by jsonpath_ng; ensure validator rejects them\n param = {\"find_key\": \"$.variables[?(@.name =~ /SQL_.*/)].value\"}\n ok, msg = empty_parameter._validate_key_value_find_key(param)\n assert ok is False\n assert \"Invalid JSONPath expression\" in msg\n\n\ndef test_validate_required_values_integration_calls_find_key_validator(empty_parameter):\n # Integration: ensure _validate_required_values uses the find_key validator for key_value_replace\n param_dict = {\"find_key\": \"no-root\", \"replace_value\": {\"DEV\": \"x\"}}\n ok, msg = empty_parameter._validate_required_values(\"key_value_replace\", param_dict)\n assert ok is False\n assert \"must be an absolute JSONPath\" in msg\n\n\ndef test_validate_and_evaluate_bracket_key_with_yaml(empty_parameter):\n \"\"\"JSONPath with bracket notation should parse and match YAML keys with spaces.\"\"\"\n import yaml\n from jsonpath_ng.ext import parse\n\n yaml_str = \"\"\"\n\"my key\":\n value: 42\n\"\"\"\n param = {\"find_key\": '$[\"my key\"].value'}\n\n ok, msg = empty_parameter._validate_key_value_find_key(param)\n assert ok is True\n assert msg == \"Valid JSONPath\"\n\n data = yaml.safe_load(yaml_str)\n matches = parse(param[\"find_key\"]).find(data)\n assert len(matches) == 1\n assert matches[0].value == 42\n\n\ndef test_yaml_boolean_filter_evaluation(empty_parameter):\n \"\"\"JSONPath filter on boolean YAML scalars should evaluate correctly.\"\"\"\n import yaml\n from jsonpath_ng.ext import parse\n\n yaml_str = \"\"\"\nservers:\n - name: \"a\"\n enabled: true\n - name: \"b\"\n enabled: false\n\"\"\"\n param = {\"find_key\": \"$.servers[?(@.enabled==true)].name\"}\n\n ok, msg = empty_parameter._validate_key_value_find_key(param)\n assert ok is True\n assert msg == \"Valid JSONPath\"\n\n data = yaml.safe_load(yaml_str)\n matches = parse(param[\"find_key\"]).find(data)\n # Expect exactly one matching server name (\"a\")\n assert len(matches) == 1\n assert matches[0].value == \"a\"\n\n\ndef test_yaml_no_match_is_no_op(empty_parameter):\n \"\"\"A JSONPath that matches nothing in YAML should be a no-op (no exception).\"\"\"\n import yaml\n from jsonpath_ng.ext import parse\n\n yaml_str = \"\"\"\nconfig:\n flag: false\n\"\"\"\n param = {\"find_key\": \"$.config.nonexistent\"}\n\n ok, msg = empty_parameter._validate_key_value_find_key(param)\n assert ok is True\n assert msg == \"Valid JSONPath\"\n\n data = yaml.safe_load(yaml_str)\n matches = parse(param[\"find_key\"]).find(data)\n assert len(matches) == 0\n\n\n@pytest.mark.parametrize(\n (\"param_value\", \"expected_ok\", \"expected_msg_contains\"),\n [\n # ===== Legacy format tests =====\n pytest.param(\n [{\"connection_id\": \"76e05dfe-9855-4e3d-a410-1dda048dbe99\", \"semantic_model_name\": [\"model1\", \"model2\"]}],\n True,\n \"parameter is valid\",\n id=\"legacy_string_connection_id\",\n ),\n pytest.param(\n [\n {\n \"connection_id\": {\n \"PPE\": \"76e05dfe-9855-4e3d-a410-1dda048dbe99\",\n \"PROD\": \"a1b2c3d4-5678-90ab-cdef-1234567890ab\",\n },\n \"semantic_model_name\": [\"model1\", \"model2\"],\n }\n ],\n False,\n \"must be a string guid\",\n id=\"legacy_dict_connection_id_not_supported\",\n ),\n pytest.param(\n [\n {\n \"connection_id\": \"invalid-guid-format\",\n \"semantic_model_name\": [\"model1\"],\n }\n ],\n False,\n \"not a valid guid\",\n id=\"legacy_invalid_guid\",\n ),\n pytest.param(\n [\n {\n \"connection_id\": 12345,\n \"semantic_model_name\": [\"model1\"],\n }\n ],\n False,\n \"must be a string guid\",\n id=\"legacy_connection_id_not_string\",\n ),\n pytest.param(\n [{\"connection_id\": \"\", \"semantic_model_name\": [\"model1\"]}],\n False,\n \"missing value\",\n id=\"legacy_empty_connection_id\",\n ),\n pytest.param(\n [\n {\n \"connection_id\": \"76e05dfe-9855-4e3d-a410-1dda048dbe99\",\n \"semantic_model_name\": \"Model1\",\n \"default\": \"x\",\n }\n ],\n False,\n \"mixed format\",\n id=\"legacy_mixed_with_new_keys\",\n ),\n # ===== New format tests =====\n pytest.param(\n {\"default\": {\"connection_id\": {\"DEV\": \"76e05dfe-9855-4e3d-a410-1dda048dbe99\"}}},\n True,\n \"parameter is valid\",\n id=\"new_default_only\",\n ),\n pytest.param(\n {\n \"models\": [\n {\"semantic_model_name\": \"MyModel\", \"connection_id\": {\"DEV\": \"76e05dfe-9855-4e3d-a410-1dda048dbe99\"}}\n ]\n },\n True,\n \"parameter is valid\",\n id=\"new_models_only\",\n ),\n pytest.param(\n {\n \"default\": {\"connection_id\": {\"DEV\": \"76e05dfe-9855-4e3d-a410-1dda048dbe99\"}},\n \"models\": [\n {\n \"semantic_model_name\": [\"Model1\", \"Model2\"],\n \"connection_id\": {\n \"DEV\": \"a1b2c3d4-5678-90ab-cdef-1234567890ab\",\n \"PROD\": \"b2c3d4e5-6789-0abc-def1-234567890abc\",\n },\n }\n ],\n },\n True,\n \"parameter is valid\",\n id=\"new_default_and_models\",\n ),\n pytest.param(\n {},\n False,\n \"requires 'default' or 'models'\",\n id=\"new_empty_dict\",\n ),\n pytest.param(\n {\"default\": {\"connection_id\": {\"DEV\": \"76e05dfe-9855-4e3d-a410-1dda048dbe99\"}}, \"invalid_key\": \"x\"},\n False,\n \"invalid key\",\n id=\"new_invalid_key\",\n ),\n pytest.param(\n {\"default\": \"not-a-dict\"},\n False,\n \"dictionary\",\n id=\"new_default_not_dict\",\n ),\n pytest.param(\n {\"default\": {\"some_key\": \"value\"}},\n False,\n \"connection_id\",\n id=\"new_default_missing_connection_id\",\n ),\n pytest.param(\n {\"models\": []},\n False,\n \"non-empty list\",\n id=\"new_models_empty_list\",\n ),\n pytest.param(\n {\"models\": \"not-a-list\"},\n False,\n \"non-empty list\",\n id=\"new_models_not_list\",\n ),\n pytest.param(\n {\"models\": [\"not-a-dict\"]},\n False,\n \"dictionary\",\n id=\"new_model_entry_not_dict\",\n ),\n pytest.param(\n {\"models\": [{\"connection_id\": {\"DEV\": \"76e05dfe-9855-4e3d-a410-1dda048dbe99\"}}]},\n False,\n \"semantic_model_name\",\n id=\"new_missing_semantic_model_name\",\n ),\n pytest.param(\n {\"models\": [{\"semantic_model_name\": \"MyModel\"}]},\n False,\n \"connection_id\",\n id=\"new_missing_connection_id\",\n ),\n pytest.param(\n {\n \"models\": [\n {\"semantic_model_name\": 12345, \"connection_id\": {\"DEV\": \"76e05dfe-9855-4e3d-a410-1dda048dbe99\"}}\n ]\n },\n False,\n \"string or list[string]\",\n id=\"new_invalid_semantic_model_name_type\",\n ),\n pytest.param(\n {\"models\": [{\"semantic_model_name\": \"MyModel\", \"connection_id\": \"76e05dfe-9855-4e3d-a410-1dda048dbe99\"}]},\n False,\n \"must be a dictionary\",\n id=\"new_string_connection_id_not_supported\",\n ),\n pytest.param(\n {\"models\": [{\"semantic_model_name\": \"MyModel\", \"connection_id\": {\"DEV\": \"invalid-guid\"}}]},\n False,\n \"not a valid guid\",\n id=\"new_invalid_connection_guid\",\n ),\n pytest.param(\n {\"models\": [{\"semantic_model_name\": \"MyModel\", \"connection_id\": {}}]},\n False,\n \"non-empty dictionary\",\n id=\"new_empty_connection_id_dict\",\n ),\n ],\n)\ndef test_semantic_model_binding_validation(empty_parameter, param_value, expected_ok, expected_msg_contains):\n \"\"\"Parametrized test for semantic_model_binding validation covering legacy and new formats.\"\"\"\n empty_parameter.environment_parameter = {\"semantic_model_binding\": param_value}\n ok, msg = empty_parameter._validate_semantic_model_binding_parameter(\"semantic_model_binding\")\n assert ok is expected_ok\n assert expected_msg_contains.lower() in msg.lower()\n\n\n@pytest.mark.parametrize(\n (\"connection_id\", \"require_string\", \"require_dict\", \"expected_ok\", \"expected_msg_contains\"),\n [\n pytest.param(\n \"76e05dfe-9855-4e3d-a410-1dda048dbe99\",\n True,\n False,\n True,\n \"Valid\",\n id=\"legacy_valid_string_guid\",\n ),\n pytest.param(\n \"invalid-guid\",\n True,\n False,\n False,\n \"not a valid GUID\",\n id=\"legacy_invalid_guid\",\n ),\n pytest.param(\n {\"DEV\": \"76e05dfe-9855-4e3d-a410-1dda048dbe99\"},\n True,\n False,\n False,\n \"must be a string GUID\",\n id=\"legacy_dict_not_supported\",\n ),\n pytest.param(\n {\n \"PPE\": \"76e05dfe-9855-4e3d-a410-1dda048dbe99\",\n \"PROD\": \"a1b2c3d4-5678-90ab-cdef-1234567890ab\",\n },\n False,\n True,\n True,\n \"Valid\",\n id=\"new_valid_multi_env\",\n ),\n pytest.param(\n {\"PPE\": \"not-a-guid\", \"PROD\": \"a1b2c3d4-5678-90ab-cdef-1234567890ab\"},\n False,\n True,\n False,\n \"not a valid GUID\",\n id=\"new_invalid_guid_format\",\n ),\n pytest.param(\n \"76e05dfe-9855-4e3d-a410-1dda048dbe99\",\n False,\n True,\n False,\n \"must be a dictionary\",\n id=\"new_string_not_supported\",\n ),\n ],\n)\ndef test_validate_connection_id(\n empty_parameter, connection_id, require_string, require_dict, expected_ok, expected_msg_contains\n):\n \"\"\"Test _validate_connection_id with various inputs.\"\"\"\n ok, msg = empty_parameter._validate_connection_id(\n connection_id, \"semantic_model_binding\", require_string=require_string, require_dict=require_dict\n )\n assert ok is expected_ok\n assert expected_msg_contains in msg\n\n\ndef test_semantic_model_binding_new_format_models_invalid_connection_guid(empty_parameter):\n \"\"\"Test semantic_model_binding new format with invalid GUID in models connections.\"\"\"\n empty_parameter.environment_parameter = {\n \"semantic_model_binding\": {\n \"models\": [{\"semantic_model_name\": \"MyModel\", \"connection_id\": {\"DEV\": \"not-a-valid-guid\"}}]\n }\n }\n ok, msg = empty_parameter._validate_semantic_model_binding_parameter(\"semantic_model_binding\")\n assert ok is False\n assert \"not a valid guid\" in msg.lower()\n\n\ndef test_semantic_model_binding_legacy_format_mixed_with_new_keys(empty_parameter):\n \"\"\"Test semantic_model_binding legacy format with new format keys mixed in (should fail).\"\"\"\n empty_parameter.environment_parameter = {\n \"semantic_model_binding\": [\n {\n \"connection_id\": \"76e05dfe-9855-4e3d-a410-1dda048dbe99\",\n \"semantic_model_name\": \"Model1\",\n \"default\": \"should-not-be-here\", # New format key in legacy entry\n }\n ]\n }\n ok, msg = empty_parameter._validate_semantic_model_binding_parameter(\"semantic_model_binding\")\n assert ok is False\n assert \"mixed format\" in msg.lower()\n\n\n@pytest.mark.parametrize(\n (\"param_value\", \"is_new_format\", \"expected_duplicates\"),\n [\n # New format: duplicate name triggers warning\n (\n {\n \"default\": {\"connection_id\": {\"PPE\": \"00000000-0000-0000-0000-000000000001\"}},\n \"models\": [\n {\"semantic_model_name\": \"ModelA\", \"connection_id\": {\"PPE\": \"00000000-0000-0000-0000-000000000002\"}},\n {\"semantic_model_name\": \"ModelA\", \"connection_id\": {\"PPE\": \"00000000-0000-0000-0000-000000000003\"}},\n ],\n },\n True,\n {\"ModelA\"},\n ),\n # New format: no duplicates, no warning\n (\n {\n \"default\": {\"connection_id\": {\"PPE\": \"00000000-0000-0000-0000-000000000001\"}},\n \"models\": [\n {\"semantic_model_name\": \"ModelA\", \"connection_id\": {\"PPE\": \"00000000-0000-0000-0000-000000000002\"}},\n {\"semantic_model_name\": \"ModelB\", \"connection_id\": {\"PPE\": \"00000000-0000-0000-0000-000000000003\"}},\n ],\n },\n True,\n set(),\n ),\n # Legacy format: duplicate name triggers warning\n (\n [\n {\"semantic_model_name\": \"ModelA\", \"connection_id\": \"00000000-0000-0000-0000-000000000001\"},\n {\"semantic_model_name\": \"ModelA\", \"connection_id\": \"00000000-0000-0000-0000-000000000002\"},\n ],\n False,\n {\"ModelA\"},\n ),\n # Legacy format: no duplicates, no warning\n (\n [\n {\"semantic_model_name\": \"ModelA\", \"connection_id\": \"00000000-0000-0000-0000-000000000001\"},\n {\"semantic_model_name\": \"ModelB\", \"connection_id\": \"00000000-0000-0000-0000-000000000002\"},\n ],\n False,\n set(),\n ),\n # New format: duplicate within a list value\n (\n {\n \"default\": {\"connection_id\": {\"PPE\": \"00000000-0000-0000-0000-000000000001\"}},\n \"models\": [\n {\n \"semantic_model_name\": [\"ModelA\", \"ModelA\"],\n \"connection_id\": {\"PPE\": \"00000000-0000-0000-0000-000000000002\"},\n },\n ],\n },\n True,\n {\"ModelA\"},\n ),\n # Legacy format: duplicate across list values in different entries\n (\n [\n {\"semantic_model_name\": [\"ModelA\", \"ModelB\"], \"connection_id\": \"00000000-0000-0000-0000-000000000001\"},\n {\"semantic_model_name\": [\"ModelB\", \"ModelC\"], \"connection_id\": \"00000000-0000-0000-0000-000000000002\"},\n ],\n False,\n {\"ModelB\"},\n ),\n ],\n ids=[\n \"new_format_duplicate\",\n \"new_format_no_duplicate\",\n \"legacy_duplicate\",\n \"legacy_no_duplicate\",\n \"new_format_duplicate_within_list\",\n \"legacy_duplicate_across_lists\",\n ],\n)\ndef test_check_duplicate_semantic_model_names(empty_parameter, param_value, is_new_format, expected_duplicates, caplog):\n \"\"\"Test that _check_duplicate_semantic_model_names warns on duplicate names.\"\"\"\n import logging\n\n with caplog.at_level(logging.WARNING):\n empty_parameter._check_duplicate_semantic_model_names(param_value, is_new_format)\n\n if expected_duplicates:\n expected_msg = constants.PARAMETER_MSGS[\"duplicate_semantic_model\"].format(\n \", \".join(sorted(expected_duplicates))\n )\n assert expected_msg in caplog.messages, (\n f\"Expected warning message not found.\\nExpected: {expected_msg}\\nActual messages: {caplog.messages}\"\n )\n # Verify exactly one warning was logged\n duplicate_warnings = [m for m in caplog.messages if \"Duplicate semantic model names found\" in m]\n assert len(duplicate_warnings) == 1, (\n f\"Expected exactly 1 duplicate warning, found {len(duplicate_warnings)}: {duplicate_warnings}\"\n )\n # Verify duplicates produce a warning but do not cause validation failure\n empty_parameter.environment_parameter = {\"semantic_model_binding\": param_value}\n ok, _ = empty_parameter._validate_semantic_model_binding_parameter(\"semantic_model_binding\")\n assert ok is True, \"Duplicate semantic model names should warn but not fail validation\"\n else:\n assert not any(\"Duplicate semantic model names found\" in m for m in caplog.messages), (\n f\"Unexpected duplicate warning found in messages: {caplog.messages}\"\n )\n" }, { "path": "tests/test_parameter_utils.py", "content": "# Copyright (c) Microsoft Corporation.\n# Licensed under the MIT License.\n\n\"\"\"\nTests for the parameter utility functions in _utils.py.\nThe tests focused on path handling functions should be compatible with both Windows and Linux.\n\"\"\"\n\nimport json\nimport logging\nimport re\nimport shutil\nimport tempfile\nfrom pathlib import Path\nfrom unittest import mock\nfrom unittest.mock import MagicMock\n\nimport pytest\nimport yaml\n\nimport fabric_cicd.constants as constants\n\n# Logger for testing\nlogger = logging.getLogger(__name__)\n\n\n@pytest.fixture\ndef temp_repository():\n \"\"\"Creates a temporary directory structure mocking a repository for testing.\"\"\"\n temp_dir = Path(tempfile.mkdtemp())\n try:\n # Create test directory structure\n (temp_dir / \"folder1\").mkdir()\n (temp_dir / \"folder1\" / \"subfolder\").mkdir()\n (temp_dir / \"folder2\").mkdir()\n\n # Create test files\n (temp_dir / \"file1.txt\").write_text(\"content1\")\n (temp_dir / \"file2.json\").write_text(\"content2\")\n (temp_dir / \"folder1\" / \"file3.py\").write_text(\"content3\")\n (temp_dir / \"folder1\" / \"subfolder\" / \"file4.md\").write_text(\"content4\")\n (temp_dir / \"folder2\" / \"file5.txt\").write_text(\"content5\")\n\n # Return the temporary directory path\n yield temp_dir\n finally:\n # Clean up temporary directory after tests\n shutil.rmtree(temp_dir)\n\n\nfrom fabric_cicd._common._exceptions import InputError, ParsingError\nfrom fabric_cicd._parameter._utils import (\n _check_parameter_structure,\n _extract_item_attribute,\n _find_match,\n _process_regular_path,\n _process_wildcard_path,\n _resolve_file_path,\n _validate_wildcard_syntax,\n check_replacement,\n extract_find_value,\n extract_parameter_filters,\n extract_replace_value,\n is_valid_structure,\n process_environment_key,\n process_input_path,\n replace_key_value,\n replace_variables_in_parameter_file,\n)\n\n\nclass TestParameterUtilities:\n \"\"\"Tests for parameter utilities in _utils.py.\"\"\"\n\n @pytest.fixture\n def mock_workspace(self):\n \"\"\"Creates a mock FabricWorkspace for testing.\"\"\"\n mock_ws = mock.MagicMock()\n mock_ws.repository_directory = Path(\"/mock/repository\")\n mock_ws.workspace_id = \"mock-workspace-id\"\n mock_ws.workspace_items = {\n \"Notebook\": {\n \"Test Notebook\": {\"id\": \"notebook-id\", \"sqlendpoint\": \"\", \"sqlendpointid\": \"\", \"queryserviceuri\": \"\"},\n },\n \"Warehouse\": {\n \"TestWarehouse\": {\n \"id\": \"warehouse-id\",\n \"sqlendpoint\": \"warehouse-endpoint\",\n \"sqlendpointid\": \"\",\n \"queryserviceuri\": \"\",\n },\n },\n \"Lakehouse\": {\n \"Test_Lakehouse\": {\n \"id\": \"lakehouse-id\",\n \"sqlendpoint\": \"lakehouse-endpoint\",\n \"sqlendpointid\": \"lakehouse-sql-endpoint-id\",\n \"queryserviceuri\": \"\",\n },\n },\n \"Eventhouse\": {\n \"Test Eventhouse\": {\n \"id\": \"eventhouse-id\",\n \"sqlendpoint\": \"\",\n \"sqlendpointid\": \"\",\n \"queryserviceuri\": \"eventhouse-query-uri\",\n },\n },\n \"SQLDatabase\": {\n \"TestSQLDatabase\": {\n \"id\": \"sqldatabase-id\",\n \"sqlendpoint\": \"test-sql-server.database.fabric.microsoft.com,1433\",\n \"sqlendpointid\": \"\",\n \"queryserviceuri\": \"\",\n },\n },\n }\n mock_ws.repository_items = {\n \"Dataflow\": {\n \"Source Dataflow\": {\"id\": \"source-dataflow-id\"},\n }\n }\n # Mock _refresh_deployed_items to avoid API calls in all tests using this fixture\n mock_ws._refresh_deployed_items = MagicMock()\n return mock_ws\n\n def test_extract_find_value(self):\n \"\"\"Tests extract_find_value with string.\"\"\"\n # Test with plain text\n param_dict = {\"find_value\": \"test-value\"}\n expected = {\"pattern\": \"test-value\", \"is_regex\": False, \"has_matches\": True}\n assert extract_find_value(param_dict, \"content with test-value\", True) == expected\n expected_no_match = {\"pattern\": \"test-value\", \"is_regex\": False, \"has_matches\": False}\n assert extract_find_value(param_dict, \"unrelated content\", True) == expected_no_match\n\n def test_extract_find_value_valid_regex(self):\n \"\"\"Tests extract_find_value with regex pattern.\"\"\"\n param_dict = {\"find_value\": \"id=([\\\\w-]+)\", \"is_regex\": \"true\"}\n\n # Test with regex\n expected = {\"pattern\": \"id=([\\\\w-]+)\", \"is_regex\": True, \"has_matches\": True}\n assert extract_find_value(param_dict, \"content with id=abc-123\", True) == expected\n # Test with non-matching regex\n expected_no_match = {\"pattern\": \"id=([\\\\w-]+)\", \"is_regex\": True, \"has_matches\": False}\n assert extract_find_value(param_dict, \"unrelated content\", True) == expected_no_match\n # Test with regex but filter_match=False - should still return is_regex=True\n expected = {\"pattern\": \"id=([\\\\w-]+)\", \"is_regex\": True, \"has_matches\": False}\n assert extract_find_value(param_dict, \"content with id=abc-123\", False) == expected\n\n def test_extract_find_value_invalid_regex(self):\n \"\"\"Tests extract_find_value with invalid regex capturing groups.\"\"\"\n # Test with regex that has no capturing groups\n param_dict = {\"find_value\": \"id=\\\\w+\", \"is_regex\": \"true\"}\n with pytest.raises(InputError):\n extract_find_value(param_dict, \"content with id=abc123\", True)\n\n # Test with regex that has multiple capturing groups\n param_dict = {\"find_value\": \"(id)=([\\\\w-]+)\", \"is_regex\": \"true\"}\n with pytest.raises(InputError):\n extract_find_value(param_dict, \"content with id=abc-123\", True)\n\n # Test with regex that captures empty value\n param_dict = {\"find_value\": \"id=()\", \"is_regex\": \"true\"}\n with pytest.raises(InputError):\n extract_find_value(param_dict, \"content with id=\", True)\n\n # Test that structure validation happens even when there are no matches\n # (regex with no capturing groups should still fail even if no matches)\n param_dict = {\"find_value\": \"id=\\\\w+\", \"is_regex\": \"true\"}\n with pytest.raises(InputError):\n extract_find_value(param_dict, \"unrelated content without matches\", True)\n\n # Test that structure validation happens with filter_match=False too\n param_dict = {\"find_value\": \"(id)=([\\\\w-]+)\", \"is_regex\": \"true\"}\n with pytest.raises(InputError):\n extract_find_value(param_dict, \"unrelated content without matches\", False)\n\n def test_extract_find_value_multiple_matches(self):\n \"\"\"Tests extract_find_value with regex pattern that has multiple matches.\"\"\"\n param_dict = {\"find_value\": \"id=([\\\\w-]+)\", \"is_regex\": \"true\"}\n\n # Test with multiple matches in content - now returns simple dict format\n content_with_multiple = \"content with id=abc-123 and id=def-456 and id=ghi-789\"\n result = extract_find_value(param_dict, content_with_multiple, True)\n expected = {\"pattern\": \"id=([\\\\w-]+)\", \"is_regex\": True, \"has_matches\": True}\n assert result == expected\n\n # Test that content is detected as having matches\n content_with_duplicates = \"content with id=abc-123 and id=def-456 and id=abc-123\"\n result = extract_find_value(param_dict, content_with_duplicates, True)\n expected = {\"pattern\": \"id=([\\\\w-]+)\", \"is_regex\": True, \"has_matches\": True}\n assert result == expected\n\n def test_extract_replace_value_default(self, mock_workspace):\n \"\"\"Tests extract_replace_value with different inputs, get_dataflow_name=False.\"\"\"\n # Regular string should be returned as is\n assert extract_replace_value(mock_workspace, \"literal string\") == \"literal string\"\n\n # Workspace ID variable should return the workspace ID\n assert extract_replace_value(mock_workspace, \"$workspace.id\", False) == \"mock-workspace-id\"\n\n # Workspace ID variable should return the workspace ID\n assert extract_replace_value(mock_workspace, \"$workspace.$id\", False) == \"mock-workspace-id\"\n\n # Workspace name variable should resolve to workspace ID\n with mock.patch(\"fabric_cicd._parameter._utils._extract_workspace_id\") as mock_extract_ws:\n mock_extract_ws.return_value = \"resolved-workspace-id\"\n result = extract_replace_value(mock_workspace, \"$workspace.dev\")\n assert result == \"resolved-workspace-id\"\n mock_extract_ws.assert_called_once_with(mock_workspace, \"$workspace.dev\")\n\n # Item attribute variables should extract values from workspace items\n with mock.patch(\"fabric_cicd._parameter._utils._extract_item_attribute\") as mock_extract:\n mock_extract.return_value = \"notebook-id\"\n result = extract_replace_value(mock_workspace, \"$items.Notebook.Test Notebook.id\")\n assert result == \"notebook-id\"\n mock_extract.assert_called_once_with(mock_workspace, \"$items.Notebook.Test Notebook.id\", False)\n\n def test_extract_replace_value_get_dataflow_name(self, mock_workspace):\n \"\"\"Tests extract_replace_value with different inputs, get_dataflow_name=True.\"\"\"\n # With get_dataflow_name=True for regular string, should return None\n assert extract_replace_value(mock_workspace, \"literal string\", True) is None\n\n # With get_dataflow_name=True for workspace ID, should return an error\n with pytest.raises(\n InputError,\n match=re.escape(\n \"Invalid replace_value variable: '$workspace'. Expected format to get dataflow name: $items.type.name.$attribute\"\n ),\n ):\n result = extract_replace_value(mock_workspace, \"$workspace.id\", True)\n\n # With get_dataflow_name=True for non-Dataflow item, should return None\n with mock.patch(\"fabric_cicd._parameter._utils._extract_item_attribute\") as mock_extract:\n mock_extract.return_value = None\n result = extract_replace_value(mock_workspace, \"$items.Notebook.Test Notebook.id\", True)\n assert result is None\n mock_extract.assert_called_once_with(mock_workspace, \"$items.Notebook.Test Notebook.id\", True)\n\n # With get_dataflow_name=True for a Dataflow item, should return the Dataflow name\n with mock.patch(\"fabric_cicd._parameter._utils._extract_item_attribute\") as mock_extract:\n mock_extract.return_value = \"Source Dataflow\"\n result = extract_replace_value(mock_workspace, \"$items.Dataflow.Source Dataflow.id\", True)\n assert result == \"Source Dataflow\"\n mock_extract.assert_called_once_with(mock_workspace, \"$items.Dataflow.Source Dataflow.id\", True)\n\n def test_extract_item_attribute_valid(self, mock_workspace):\n \"\"\"Tests _extract_item_attribute with valid variables.\"\"\"\n # Test with valid notebook item\n result = _extract_item_attribute(mock_workspace, \"$items.Notebook.Test Notebook.id\", False)\n assert result == \"notebook-id\"\n result = _extract_item_attribute(mock_workspace, \"$items.Notebook.Test Notebook.$id\", False)\n assert result == \"notebook-id\"\n\n # Test with valid lakehouse item\n result = _extract_item_attribute(mock_workspace, \"$items.Lakehouse.Test_Lakehouse.sqlendpoint\", False)\n assert result == \"lakehouse-endpoint\"\n result = _extract_item_attribute(mock_workspace, \"$items.Lakehouse.Test_Lakehouse.$sqlendpoint\", False)\n assert result == \"lakehouse-endpoint\"\n\n # Test with valid lakehouse sqlendpointid attribute\n result = _extract_item_attribute(mock_workspace, \"$items.Lakehouse.Test_Lakehouse.sqlendpointid\", False)\n assert result == \"lakehouse-sql-endpoint-id\"\n result = _extract_item_attribute(mock_workspace, \"$items.Lakehouse.Test_Lakehouse.$sqlendpointid\", False)\n assert result == \"lakehouse-sql-endpoint-id\"\n\n # Test with valid warehouse item\n result = _extract_item_attribute(mock_workspace, \"$items.Warehouse.TestWarehouse.id\", False)\n assert result == \"warehouse-id\"\n result = _extract_item_attribute(mock_workspace, \"$items.Warehouse.TestWarehouse.$id\", False)\n assert result == \"warehouse-id\"\n\n # Test with valid SQLDatabase item\n result = _extract_item_attribute(mock_workspace, \"$items.SQLDatabase.TestSQLDatabase.sqlendpoint\", False)\n assert result == \"test-sql-server.database.fabric.microsoft.com,1433\"\n result = _extract_item_attribute(mock_workspace, \"$items.SQLDatabase.TestSQLDatabase.$sqlendpoint\", False)\n assert result == \"test-sql-server.database.fabric.microsoft.com,1433\"\n\n # Test with valid eventhouse item\n result = _extract_item_attribute(mock_workspace, \"$items.Eventhouse.Test Eventhouse.queryserviceuri\", False)\n assert result == \"eventhouse-query-uri\"\n result = _extract_item_attribute(mock_workspace, \"$items.Eventhouse.Test Eventhouse.$queryserviceuri\", False)\n assert result == \"eventhouse-query-uri\"\n\n def test_extract_item_attribute_invalid(self, mock_workspace):\n \"\"\"Tests _extract_item_attribute with invalid variable cases.\"\"\"\n # Test with invalid syntax\n with pytest.raises(ParsingError, match=\"Invalid \\\\$items variable syntax\"):\n _extract_item_attribute(mock_workspace, \"$items.Notebook\", False)\n with pytest.raises(ParsingError, match=\"Invalid \\\\$items variable syntax\"):\n _extract_item_attribute(mock_workspace, \"$items.Notebook.Test Notebook\", False)\n\n # Test with too many segments - now check for invalid attribute instead of invalid syntax\n mock_items_attr_lookup = list(constants.ITEM_ATTR_LOOKUP)\n with pytest.raises(\n ParsingError,\n match=re.escape(f\"Attribute 'extra' is invalid. Supported attributes: {mock_items_attr_lookup}\"),\n ):\n _extract_item_attribute(mock_workspace, \"$items.Notebook.Test Notebook.id.extra\", False)\n\n def test_extract_item_attribute_get_dataflow_name(self, mock_workspace):\n \"\"\"Test _extract_item_attribute with special handling for Dataflow references.\"\"\"\n # Test when Dataflow references another Dataflow in the repository\n result = _extract_item_attribute(mock_workspace, \"$items.Dataflow.Source Dataflow.id\", True)\n assert result == \"Source Dataflow\"\n\n # Test when source Dataflow doesn't exist in repository - should return None\n result = _extract_item_attribute(mock_workspace, \"$items.Dataflow.NonExistentDataflow.id\", True)\n assert result is None\n\n # Test when source Dataflow type doesn't match (case sensitive) - should return None\n result = _extract_item_attribute(mock_workspace, \"$items.dataflow.Source Dataflow.id\", get_dataflow_name=True)\n assert result is None\n\n # Test when source Dataflow name doesn't match (case sensitive) - should return None\n result = _extract_item_attribute(mock_workspace, \"$items.Dataflow.source dataflow.id\", get_dataflow_name=True)\n assert result is None\n\n def test_extract_workspace_id_direct(self, mock_workspace):\n \"\"\"Tests _extract_workspace_id with direct workspace ID variable.\"\"\"\n from fabric_cicd._parameter._utils import _extract_workspace_id\n\n # Test with $workspace.id - should return workspace_id directly\n result = _extract_workspace_id(mock_workspace, \"$workspace.id\")\n assert result == \"mock-workspace-id\"\n\n result = _extract_workspace_id(mock_workspace, \"$workspace.$id\")\n assert result == \"mock-workspace-id\"\n\n def test_extract_workspace_id_resolve(self, mock_workspace):\n \"\"\"Tests _extract_workspace_id with workspace name resolution.\"\"\"\n from fabric_cicd._parameter._utils import _extract_workspace_id\n\n # Mock the _resolve_workspace_id method\n mock_workspace._resolve_workspace_id.return_value = \"resolved-workspace-id\"\n\n # Test with both backward-compatible and explicit ID syntaxes\n result = _extract_workspace_id(mock_workspace, \"$workspace.test_workspace\")\n assert result == \"resolved-workspace-id\"\n mock_workspace._resolve_workspace_id.assert_called_once_with(\"test_workspace\")\n\n mock_workspace._resolve_workspace_id.reset_mock()\n result = _extract_workspace_id(mock_workspace, \"$workspace.test_workspace.$id\")\n assert result == \"resolved-workspace-id\"\n mock_workspace._resolve_workspace_id.assert_called_once_with(\"test_workspace\")\n\n def test_extract_workspace_id_with_workspace_name_variable(self, mock_workspace):\n \"\"\"Tests _extract_workspace_id with workspace name variable.\"\"\"\n from fabric_cicd._parameter._utils import _extract_workspace_id\n\n mock_workspace._resolve_workspace_name = mock.MagicMock(return_value=\"My Target Workspace [PPE]\")\n\n result = _extract_workspace_id(mock_workspace, \"$workspace.$name\")\n assert result == \"My Target Workspace [PPE]\"\n mock_workspace._resolve_workspace_name.assert_called_once_with()\n\n def test_extract_workspace_id_name_encoded(self, mock_workspace):\n \"\"\"Tests _extract_workspace_id with $workspace.$name_encoded returns URL-encoded name.\"\"\"\n from fabric_cicd._parameter._utils import _extract_workspace_id\n\n mock_workspace._resolve_workspace_name = mock.MagicMock(return_value=\"My Target Workspace [PPE]\")\n\n result = _extract_workspace_id(mock_workspace, \"$workspace.$name_encoded\")\n assert result == \"My%20Target%20Workspace%20%5BPPE%5D\"\n mock_workspace._resolve_workspace_name.assert_called_once_with()\n\n def test_extract_workspace_id_resolve_error(self, mock_workspace):\n \"\"\"Tests _extract_workspace_id when workspace name resolution fails.\"\"\"\n from fabric_cicd._parameter._utils import _extract_workspace_id\n\n # Mock the _resolve_workspace_id method to raise InputError\n mock_workspace._resolve_workspace_id.side_effect = InputError(\"Workspace name not found\", logger)\n\n # Should re-raise the same InputError\n with pytest.raises(InputError, match=r\"Workspace name not found\"):\n _extract_workspace_id(mock_workspace, \"$workspace.unknown_workspace\")\n\n def test_extract_workspace_id_general_error(self, mock_workspace):\n \"\"\"Tests _extract_workspace_id with unexpected errors.\"\"\"\n from fabric_cicd._parameter._utils import _extract_workspace_id\n\n # Mock the _resolve_workspace_id method to raise a general exception\n mock_workspace._resolve_workspace_id.side_effect = Exception(\"Unexpected error\")\n\n # Should wrap general exceptions in ParsingError\n with pytest.raises(ParsingError, match=r\"Error parsing \\$workspace variable\"):\n _extract_workspace_id(mock_workspace, \"$workspace.test_workspace\")\n\n def test_extract_item_attribute_null_return(self, mock_workspace):\n \"\"\"Tests _extract_item_attribute cases that return None.\"\"\"\n # Test with non-Dataflow item in get_dataflow_name mode should return None\n result = _extract_item_attribute(mock_workspace, \"$items.Lakehouse.Test Lakehouse.id\", True)\n assert result is None\n\n # Test with Dataflow item, but incorrect attribute should return None\n result = _extract_item_attribute(mock_workspace, \"$items.Dataflow.Source Dataflow.sqlendpoint\", True)\n assert result is None\n\n def test_extract_item_attribute_invalid_attribute(self, mock_workspace):\n \"\"\"Tests _extract_item_attribute with invalid attribute.\"\"\"\n # Test with invalid attribute\n mock_items_attr_lookup = list(constants.ITEM_ATTR_LOOKUP)\n with pytest.raises(\n ParsingError,\n match=re.escape(f\"Attribute 'guid' is invalid. Supported attributes: {mock_items_attr_lookup}\"),\n ):\n _extract_item_attribute(mock_workspace, \"$items.Dataflow.Source Dataflow.guid\", True)\n\n def test_extract_workspace_id_with_item_lookup(self, mock_workspace):\n \"\"\"Tests _extract_workspace_id with item lookup in another workspace.\"\"\"\n from fabric_cicd._parameter._utils import _extract_workspace_id\n\n # Mock the _resolve_workspace_id method\n mock_workspace._resolve_workspace_id.return_value = \"resolved-workspace-id\"\n\n # Mock the _lookup_item_attribute method\n mock_workspace._lookup_item_attribute = mock.MagicMock(return_value=\"item-123-id\")\n\n # Test with $workspace..$items...$id format\n result = _extract_workspace_id(mock_workspace, \"$workspace.test_workspace.$items.Notebook.Test Notebook.$id\")\n\n assert result == \"item-123-id\"\n mock_workspace._resolve_workspace_id.assert_called_once_with(\"test_workspace\")\n # attribute should be passed without leading '$'\n mock_workspace._lookup_item_attribute.assert_called_once_with(\n \"resolved-workspace-id\", \"Notebook\", \"Test Notebook\", \"id\"\n )\n\n def test_extract_workspace_id_with_item_lookup_not_found(self, mock_workspace):\n \"\"\"Tests _extract_workspace_id when item lookup fails.\"\"\"\n from fabric_cicd._parameter._utils import _extract_workspace_id\n\n # Mock the _resolve_workspace_id method\n mock_workspace._resolve_workspace_id.return_value = \"resolved-workspace-id\"\n\n # Mock the _lookup_item_attribute method to raise InputError (item not found)\n error_msg = (\n \"Failed to look up item in workspace: resolved-workspace-id, item_type: Notebook, item_name: Test Notebook\"\n )\n mock_workspace._lookup_item_attribute = mock.MagicMock(side_effect=InputError(error_msg, logger))\n\n # Should re-raise the InputError\n with pytest.raises(InputError, match=re.escape(error_msg)):\n _extract_workspace_id(mock_workspace, \"$workspace.test_workspace.$items.Notebook.Test Notebook.$id\")\n\n mock_workspace._resolve_workspace_id.assert_called_once_with(\"test_workspace\")\n mock_workspace._lookup_item_attribute.assert_called_once_with(\n \"resolved-workspace-id\", \"Notebook\", \"Test Notebook\", \"id\"\n )\n\n @pytest.mark.parametrize(\n \"invalid_var\",\n [\n \"$workspace.$items.Notebook.Test Notebook.$id\", # Missing workspace name\n \"$workspace.test_workspace.$items.InvalidType.Test Notebook.$id\", # Invalid item type\n \"$workspace.test_workspace.$items.Notebook.$id\", # Missing item name\n ],\n )\n def test_extract_workspace_id_with_item_lookup_invalid_format(self, mock_workspace, invalid_var):\n \"\"\"Tests _extract_workspace_id with invalid item lookup format.\"\"\"\n from fabric_cicd._parameter._utils import _extract_workspace_id\n\n # Test with invalid formats\n with pytest.raises(ParsingError):\n _extract_workspace_id(mock_workspace, invalid_var)\n\n def test_extract_workspace_id_with_item_lookup_sqlendpoint(self, mock_workspace):\n \"\"\"Tests _extract_workspace_id resolves sqlendpoint from another workspace via $items reference.\"\"\"\n from fabric_cicd._parameter._utils import _extract_workspace_id\n\n mock_workspace._resolve_workspace_id.return_value = \"resolved-workspace-id\"\n mock_workspace._lookup_item_attribute = mock.MagicMock(return_value=\"lakehouse-endpoint-value\")\n\n result = _extract_workspace_id(\n mock_workspace, \"$workspace.test_workspace.$items.Lakehouse.Test_Lakehouse.$sqlendpoint\"\n )\n\n assert result == \"lakehouse-endpoint-value\"\n mock_workspace._resolve_workspace_id.assert_called_once_with(\"test_workspace\")\n mock_workspace._lookup_item_attribute.assert_called_once_with(\n \"resolved-workspace-id\", \"Lakehouse\", \"Test_Lakehouse\", \"sqlendpoint\"\n )\n\n def test_extract_workspace_id_with_item_lookup_queryserviceuri(self, mock_workspace):\n \"\"\"Tests _extract_workspace_id resolves queryserviceuri from another workspace via $items reference.\"\"\"\n from fabric_cicd._parameter._utils import _extract_workspace_id\n\n mock_workspace._resolve_workspace_id.return_value = \"resolved-workspace-id\"\n mock_workspace._lookup_item_attribute = mock.MagicMock(return_value=\"eventhouse-query-uri-value\")\n\n result = _extract_workspace_id(\n mock_workspace, \"$workspace.test_workspace.$items.Eventhouse.Test Eventhouse.$queryserviceuri\"\n )\n\n assert result == \"eventhouse-query-uri-value\"\n mock_workspace._resolve_workspace_id.assert_called_once_with(\"test_workspace\")\n mock_workspace._lookup_item_attribute.assert_called_once_with(\n \"resolved-workspace-id\", \"Eventhouse\", \"Test Eventhouse\", \"queryserviceuri\"\n )\n\n def test_extract_workspace_id_with_item_lookup_sqlendpointid(self, mock_workspace):\n \"\"\"Tests _extract_workspace_id resolves sqlendpointid from another workspace via $items reference.\"\"\"\n from fabric_cicd._parameter._utils import _extract_workspace_id\n\n mock_workspace._resolve_workspace_id.return_value = \"resolved-workspace-id\"\n mock_workspace._lookup_item_attribute = mock.MagicMock(return_value=\"lakehouse-sql-endpoint-id-value\")\n\n result = _extract_workspace_id(\n mock_workspace, \"$workspace.test_workspace.$items.Lakehouse.Test_Lakehouse.$sqlendpointid\"\n )\n\n assert result == \"lakehouse-sql-endpoint-id-value\"\n mock_workspace._resolve_workspace_id.assert_called_once_with(\"test_workspace\")\n mock_workspace._lookup_item_attribute.assert_called_once_with(\n \"resolved-workspace-id\", \"Lakehouse\", \"Test_Lakehouse\", \"sqlendpointid\"\n )\n\n def test_extract_replace_value_workspace_name(self, mock_workspace):\n \"\"\"Tests extract_replace_value returns workspace name for $workspace.$name.\"\"\"\n mock_workspace._resolve_workspace_name = mock.MagicMock(return_value=\"My Target Workspace [PPE]\")\n\n result = extract_replace_value(mock_workspace, \"$workspace.$name\")\n assert result == \"My Target Workspace [PPE]\"\n\n # $workspace.name (without $) should resolve \"name\" as a workspace name, not return display name\n mock_workspace._resolve_workspace_id.return_value = \"resolved-id-for-name\"\n result = extract_replace_value(mock_workspace, \"$workspace.name\")\n assert result == \"resolved-id-for-name\"\n mock_workspace._resolve_workspace_id.assert_called_once_with(\"name\")\n\n mock_workspace._resolve_workspace_id.reset_mock()\n result = extract_replace_value(mock_workspace, \"$workspace.TestWorkspace.$id\")\n assert result == \"resolved-id-for-name\"\n mock_workspace._resolve_workspace_id.assert_called_once_with(\"TestWorkspace\")\n\n def test_extract_parameter_filters(self, mock_workspace):\n \"\"\"Tests extract_parameter_filters function.\"\"\"\n # Test with all filters\n param_dict = {\"item_type\": \"Notebook\", \"item_name\": \"TestNotebook\", \"file_path\": \"path/to/file.txt\"}\n\n with mock.patch(\"fabric_cicd._parameter._utils.process_input_path\") as mock_process:\n # Return a list of Path objects as expected\n processed_path = Path(\"processed/path\")\n mock_process.return_value = [processed_path]\n item_type, item_name, file_path = extract_parameter_filters(mock_workspace, param_dict)\n\n assert item_type == \"Notebook\"\n assert item_name == \"TestNotebook\"\n # Assert that file_path is a list containing the processed path\n assert file_path == [processed_path]\n mock_process.assert_called_once_with(mock_workspace.repository_directory, \"path/to/file.txt\")\n\n # Test with missing filters\n param_dict = {}\n with mock.patch(\"fabric_cicd._parameter._utils.process_input_path\") as mock_process:\n # When no file_path in param_dict, process_input_path should return an empty list\n mock_process.return_value = []\n item_type, item_name, file_path = extract_parameter_filters(mock_workspace, param_dict)\n\n assert item_type is None\n assert item_name is None\n assert file_path == []\n\n def test_check_parameter_structure(self):\n \"\"\"Tests _check_parameter_structure function.\"\"\"\n # Test with valid list\n assert _check_parameter_structure([1, 2, 3]) is True\n assert _check_parameter_structure([]) is True\n\n # Test with invalid types\n assert _check_parameter_structure(\"string\") is False\n assert _check_parameter_structure(123) is False\n assert _check_parameter_structure({\"key\": \"value\"}) is False\n assert _check_parameter_structure(None) is False\n\n def test_is_valid_structure(self):\n \"\"\"Tests is_valid_structure function.\"\"\"\n # Test with valid structures\n valid_dict = {\n \"find_replace\": [{\"find_value\": \"test\"}],\n \"key_value_replace\": [{\"find_key\": \"$.test\"}],\n \"spark_pool\": [{\"instance_pool_id\": \"test\"}],\n \"semantic_model_binding\": [{\"connection_id\": \"test\"}],\n }\n assert is_valid_structure(valid_dict) is True\n assert is_valid_structure(valid_dict, \"find_replace\") is True\n\n # Test with invalid structures\n invalid_dict = {\n \"find_replace\": \"not a list\",\n \"key_value_replace\": [{\"find_key\": \"$.test\"}],\n \"semantic_model_binding\": \"not a list\",\n }\n assert is_valid_structure(invalid_dict) is False\n assert is_valid_structure(invalid_dict, \"find_replace\") is False\n\n # Test with missing parameters\n missing_dict = {\n \"unknown_param\": [{\"test\": \"value\"}],\n }\n assert is_valid_structure(missing_dict) is False\n\n # Test with empty dict\n assert is_valid_structure({}) is False\n\n def test_is_valid_structure_semantic_model_binding_new_format(self):\n \"\"\"Tests is_valid_structure with new dictionary format for semantic_model_binding.\"\"\"\n # New format with default and models\n valid_new_format = {\n \"semantic_model_binding\": {\n \"default\": {\n \"connection_id\": {\n \"PPE\": \"76e05dfe-9855-4e3d-a410-1dda048dbe99\",\n }\n },\n \"models\": [\n {\n \"semantic_model_name\": [\"Model1\", \"Model2\"],\n \"connection_id\": {\n \"PPE\": \"f96870d5-5f86-49ad-bf41-5967fd7c1c6d\",\n },\n }\n ],\n }\n }\n assert is_valid_structure(valid_new_format) is True\n assert is_valid_structure(valid_new_format, \"semantic_model_binding\") is True\n\n # New format with only default\n valid_default_only = {\n \"semantic_model_binding\": {\"default\": {\"connection_id\": {\"_ALL_\": \"76e05dfe-9855-4e3d-a410-1dda048dbe99\"}}}\n }\n assert is_valid_structure(valid_default_only) is True\n\n # New format with only models\n valid_models_only = {\n \"semantic_model_binding\": {\n \"models\": [\n {\n \"semantic_model_name\": \"SingleModel\",\n \"connection_id\": {\"DEV\": \"76e05dfe-9855-4e3d-a410-1dda048dbe99\"},\n }\n ]\n }\n }\n assert is_valid_structure(valid_models_only) is True\n\n # String is invalid\n invalid_string = {\"semantic_model_binding\": \"not valid\"}\n assert is_valid_structure(invalid_string) is False\n\n # Empty dict is invalid (no bindings configured)\n invalid_empty = {\"semantic_model_binding\": {}}\n assert is_valid_structure(invalid_empty) is False\n\n @mock.patch(\"fabric_cicd._parameter._parameter.Parameter\")\n @mock.patch(\"fabric_cicd._common._validate_input.validate_repository_directory\")\n @mock.patch(\"fabric_cicd._common._validate_input.validate_item_type_in_scope\")\n @mock.patch(\"fabric_cicd._common._validate_input.validate_environment\")\n def test_validate_parameter_file(self, mock_validate_env, mock_validate_item_type, mock_validate_repo, mock_param):\n \"\"\"Tests validate_parameter_file function with default parameters.\"\"\"\n # Setup mocks\n mock_validate_repo.return_value = Path(\"/mock/repo\")\n mock_validate_item_type.return_value = [\"Notebook\", \"Lakehouse\"]\n mock_validate_env.return_value = \"Test\"\n mock_param_instance = mock.MagicMock()\n mock_param.return_value = mock_param_instance\n mock_param_instance._validate_parameter_file.return_value = True\n\n # Call the function\n from fabric_cicd._parameter._utils import validate_parameter_file\n\n # Patch the FabricEndpoint inside the test since we need it to run successfully\n with mock.patch(\"fabric_cicd._common._fabric_endpoint.FabricEndpoint\", return_value=mock.MagicMock()):\n result = validate_parameter_file(\n repository_directory=Path(\"/mock/repo\"),\n item_type_in_scope=[\"Notebook\", \"Lakehouse\"],\n environment=\"Test\",\n )\n\n # Verify the result\n assert result is True\n mock_param.assert_called_once_with(\n repository_directory=Path(\"/mock/repo\"),\n item_type_in_scope=[\"Notebook\", \"Lakehouse\"],\n environment=\"Test\",\n parameter_file_name=\"parameter.yml\",\n parameter_file_path=None,\n )\n mock_param_instance._validate_parameter_file.assert_called_once()\n\n @mock.patch(\"fabric_cicd._parameter._parameter.Parameter\")\n @mock.patch(\"fabric_cicd._common._validate_input.validate_repository_directory\")\n @mock.patch(\"fabric_cicd._common._validate_input.validate_item_type_in_scope\")\n @mock.patch(\"fabric_cicd._common._validate_input.validate_environment\")\n def test_validate_parameter_file_with_custom_file_name(\n self, mock_validate_env, mock_validate_item_type, mock_validate_repo, mock_param\n ):\n \"\"\"Tests validate_parameter_file function with custom parameter file name.\"\"\"\n # Setup mocks\n mock_validate_repo.return_value = Path(\"/mock/repo\")\n mock_validate_item_type.return_value = [\"Notebook\", \"Lakehouse\"]\n mock_validate_env.return_value = \"Test\"\n mock_param_instance = mock.MagicMock()\n mock_param.return_value = mock_param_instance\n mock_param_instance._validate_parameter_file.return_value = True\n\n # Call the function\n from fabric_cicd._parameter._utils import validate_parameter_file\n\n # Patch the FabricEndpoint inside the test since we need it to run successfully\n with mock.patch(\"fabric_cicd._common._fabric_endpoint.FabricEndpoint\", return_value=mock.MagicMock()):\n result = validate_parameter_file(\n repository_directory=Path(\"/mock/repo\"),\n item_type_in_scope=[\"Notebook\", \"Lakehouse\"],\n environment=\"Test\",\n parameter_file_name=\"custom_params.yml\",\n )\n\n # Verify the result\n assert result is True\n mock_param.assert_called_once_with(\n repository_directory=Path(\"/mock/repo\"),\n item_type_in_scope=[\"Notebook\", \"Lakehouse\"],\n environment=\"Test\",\n parameter_file_name=\"custom_params.yml\",\n parameter_file_path=None,\n )\n mock_param_instance._validate_parameter_file.assert_called_once()\n\n @mock.patch(\"fabric_cicd._parameter._parameter.Parameter\")\n @mock.patch(\"fabric_cicd._common._validate_input.validate_repository_directory\")\n @mock.patch(\"fabric_cicd._common._validate_input.validate_item_type_in_scope\")\n @mock.patch(\"fabric_cicd._common._validate_input.validate_environment\")\n def test_validate_parameter_file_with_custom_file_path(\n self, mock_validate_env, mock_validate_item_type, mock_validate_repo, mock_param\n ):\n \"\"\"Tests validate_parameter_file function with custom parameter file path.\"\"\"\n # Setup mocks\n mock_validate_repo.return_value = Path(\"/mock/repo\")\n mock_validate_item_type.return_value = [\"Notebook\", \"Lakehouse\"]\n mock_validate_env.return_value = \"Test\"\n mock_param_instance = mock.MagicMock()\n mock_param.return_value = mock_param_instance\n mock_param_instance._validate_parameter_file.return_value = True\n\n # Call the function\n from fabric_cicd._parameter._utils import validate_parameter_file\n\n # Patch the FabricEndpoint inside the test since we need it to run successfully\n with mock.patch(\"fabric_cicd._common._fabric_endpoint.FabricEndpoint\", return_value=mock.MagicMock()):\n result = validate_parameter_file(\n repository_directory=Path(\"/mock/repo\"),\n item_type_in_scope=[\"Notebook\", \"Lakehouse\"],\n environment=\"Test\",\n parameter_file_path=\"/custom/path/to/parameters.yml\",\n )\n\n # Verify the result\n assert result is True\n mock_param.assert_called_once_with(\n repository_directory=Path(\"/mock/repo\"),\n item_type_in_scope=[\"Notebook\", \"Lakehouse\"],\n environment=\"Test\",\n parameter_file_name=\"parameter.yml\",\n parameter_file_path=\"/custom/path/to/parameters.yml\",\n )\n mock_param_instance._validate_parameter_file.assert_called_once()\n\n @mock.patch(\"fabric_cicd._parameter._parameter.Parameter\")\n @mock.patch(\"fabric_cicd._common._validate_input.validate_repository_directory\")\n @mock.patch(\"fabric_cicd._common._validate_input.validate_item_type_in_scope\")\n @mock.patch(\"fabric_cicd._common._validate_input.validate_environment\")\n def test_validate_parameter_file_with_both_custom_name_and_path(\n self, mock_validate_env, mock_validate_item_type, mock_validate_repo, mock_param\n ):\n \"\"\"Tests validate_parameter_file function with both custom file name and path.\"\"\"\n # Setup mocks\n mock_validate_repo.return_value = Path(\"/mock/repo\")\n mock_validate_item_type.return_value = [\"Notebook\", \"Lakehouse\"]\n mock_validate_env.return_value = \"Test\"\n mock_param_instance = mock.MagicMock()\n mock_param.return_value = mock_param_instance\n mock_param_instance._validate_parameter_file.return_value = True\n\n # Call the function\n from fabric_cicd._parameter._utils import validate_parameter_file\n\n # Patch the FabricEndpoint inside the test since we need it to run successfully\n with mock.patch(\"fabric_cicd._common._fabric_endpoint.FabricEndpoint\", return_value=mock.MagicMock()):\n result = validate_parameter_file(\n repository_directory=Path(\"/mock/repo\"),\n item_type_in_scope=[\"Notebook\", \"Lakehouse\"],\n environment=\"Test\",\n parameter_file_name=\"custom_params.yml\",\n parameter_file_path=\"/custom/path/to/parameters.yml\",\n )\n\n # Verify the result\n assert result is True\n mock_param.assert_called_once_with(\n repository_directory=Path(\"/mock/repo\"),\n item_type_in_scope=[\"Notebook\", \"Lakehouse\"],\n environment=\"Test\",\n parameter_file_name=\"custom_params.yml\",\n parameter_file_path=\"/custom/path/to/parameters.yml\",\n )\n mock_param_instance._validate_parameter_file.assert_called_once()\n\n @mock.patch(\"fabric_cicd._parameter._parameter.Parameter\")\n @mock.patch(\"fabric_cicd._common._validate_input.validate_repository_directory\")\n @mock.patch(\"fabric_cicd._common._validate_input.validate_item_type_in_scope\")\n @mock.patch(\"fabric_cicd._common._validate_input.validate_environment\")\n def test_validate_parameter_file_with_none_item_type_in_scope(\n self, mock_validate_env, mock_validate_item_type, mock_validate_repo, mock_param\n ):\n \"\"\"Tests validate_parameter_file function when item_type_in_scope is omitted (None).\"\"\"\n # Setup mocks\n mock_validate_repo.return_value = Path(\"/mock/repo\")\n # Mock validate_item_type_in_scope to return all supported types when None is passed\n mock_validate_item_type.return_value = list(constants.ACCEPTED_ITEM_TYPES)\n mock_validate_env.return_value = \"Test\"\n mock_param_instance = mock.MagicMock()\n mock_param.return_value = mock_param_instance\n mock_param_instance._validate_parameter_file.return_value = True\n\n # Call the function\n from fabric_cicd._parameter._utils import validate_parameter_file\n\n # Patch the FabricEndpoint inside the test since we need it to run successfully\n with mock.patch(\"fabric_cicd._common._fabric_endpoint.FabricEndpoint\", return_value=mock.MagicMock()):\n result = validate_parameter_file(\n repository_directory=Path(\"/mock/repo\"),\n environment=\"Test\",\n )\n\n # Verify the result\n assert result is True\n # Verify that validate_item_type_in_scope was called with None\n mock_validate_item_type.assert_called_once_with(None)\n # Verify Parameter was called with all supported item types\n mock_param.assert_called_once_with(\n repository_directory=Path(\"/mock/repo\"),\n item_type_in_scope=list(constants.ACCEPTED_ITEM_TYPES),\n environment=\"Test\",\n parameter_file_name=\"parameter.yml\",\n parameter_file_path=None,\n )\n mock_param_instance._validate_parameter_file.assert_called_once()\n\n def test_find_match(self):\n \"\"\"Tests _find_match function with various inputs.\"\"\"\n # Test with None param_value\n assert _find_match(None, \"value\") is True\n\n # Test with string param_value\n assert _find_match(\"value\", \"value\") is True\n assert _find_match(\"value\", \"other\") is False\n\n # Test with list param_value\n assert _find_match([\"value1\", \"value2\"], \"value1\") is True\n assert _find_match([\"value1\", \"value2\"], \"value3\") is False\n\n # Test with list of Paths\n path_list = [Path(\"test1.txt\"), Path(\"test2.txt\")]\n assert _find_match(path_list, Path(\"test1.txt\")) is True\n assert _find_match(path_list, Path(\"test3.txt\")) is False\n\n # Test with invalid type\n assert _find_match(123, \"value\") is False\n\n def test_check_replacement(self, temp_repository):\n \"\"\"Tests check_replacement function with various combinations of inputs.\"\"\"\n file_path = temp_repository / \"file1.txt\"\n\n # Test with no filters\n assert check_replacement(None, None, None, \"type1\", \"name1\", file_path) is True\n\n # Test with matching filters\n assert check_replacement(\"type1\", \"name1\", [file_path], \"type1\", \"name1\", file_path) is True\n\n # Test with non-matching filters\n assert check_replacement(\"type2\", \"name1\", [file_path], \"type1\", \"name1\", file_path) is False\n assert check_replacement(\"type1\", \"name2\", [file_path], \"type1\", \"name1\", file_path) is False\n assert check_replacement(\"type1\", \"name1\", [Path(\"other.txt\")], \"type1\", \"name1\", file_path) is False\n\n # Test with combination of matching/non-matching filters\n assert check_replacement(\"type1\", \"name2\", [file_path], \"type1\", \"name1\", file_path) is False\n assert check_replacement(\"type1\", \"name1\", [Path(\"other.txt\")], \"type1\", \"name1\", file_path) is False\n\n def test_replace_key_value_valid_json(self, mock_workspace):\n \"\"\"Tests replace_key_value with valid JSON content and environment.\"\"\"\n # Test JSON with server host configuration\n test_json = '{\"server\": {\"host\": \"localhost\", \"port\": 8080}}'\n param_dict = {\n \"find_key\": \"$.server.host\",\n \"replace_value\": {\"dev\": \"dev-server.example.com\", \"prod\": \"prod-server.example.com\"},\n }\n\n # Test successful replacement for dev environment\n result = replace_key_value(mock_workspace, param_dict, test_json, \"dev\")\n result_data = json.loads(result)\n assert result_data[\"server\"][\"host\"] == \"dev-server.example.com\"\n assert result_data[\"server\"][\"port\"] == 8080 # Verify other values unchanged\n\n # Test successful replacement for prod environment\n result = replace_key_value(mock_workspace, param_dict, test_json, \"prod\")\n result_data = json.loads(result)\n assert result_data[\"server\"][\"host\"] == \"prod-server.example.com\"\n\n def test_replace_key_value_environment_not_found(self, mock_workspace):\n \"\"\"Tests replace_key_value when environment is not in the replace_value dictionary.\"\"\"\n test_json = '{\"server\": {\"host\": \"localhost\", \"port\": 8080}}'\n param_dict = {\n \"find_key\": \"$.server.host\",\n \"replace_value\": {\"dev\": \"dev-server.example.com\", \"prod\": \"prod-server.example.com\"},\n }\n\n # Test when environment not in replace_value\n result = replace_key_value(mock_workspace, param_dict, test_json, \"test\")\n result_data = json.loads(result)\n assert result_data[\"server\"][\"host\"] == \"localhost\" # Original value unchanged\n\n def test_replace_key_value_invalid_json(self, mock_workspace):\n \"\"\"Tests replace_key_value with invalid JSON content.\"\"\"\n invalid_json = \"{invalid json content}\"\n param_dict = {\"find_key\": \"$.server.host\", \"replace_value\": {\"dev\": \"test-server\"}}\n\n # JSONDecodeError will be raised for invalid JSON and wrapped in ValueError\n with pytest.raises(ValueError, match=\"Expecting property name\"):\n replace_key_value(mock_workspace, param_dict, invalid_json, \"dev\")\n\n def test_replace_key_value(self, mock_workspace):\n \"\"\"Test replace_key_value function with JSON content.\"\"\"\n # Create test parameter dictionary and JSON content\n param_dict = {\n \"find_key\": \"$.server.host\",\n \"replace_value\": {\"dev\": \"dev-server.example.com\", \"prod\": \"prod-server.example.com\"},\n }\n json_content = '{\"server\": {\"host\": \"localhost\", \"port\": 8080}}'\n\n # Test successful replacement\n result = replace_key_value(mock_workspace, param_dict, json_content, \"dev\")\n\n # Parse the JSON result and check the exact value (avoid substring sanitization issues)\n result_json = json.loads(result)\n assert result_json[\"server\"][\"host\"] == \"dev-server.example.com\"\n\n # Test with environment not in replace_value\n result = replace_key_value(mock_workspace, param_dict, json_content, \"test\")\n result_json = json.loads(result)\n assert result_json[\"server\"][\"host\"] == \"localhost\"\n\n # Test with invalid JSON content\n with pytest.raises(ValueError, match=\"Expecting property name\"):\n replace_key_value(mock_workspace, param_dict, \"{invalid json}\", \"dev\")\n\n def test_replace_key_value_with_items_notation(self, mock_workspace):\n \"\"\"Test replace_key_value function with $items notation.\"\"\"\n # Mock the workspace to return item attributes\n mock_workspace.workspace_items = {\n \"Lakehouse\": {\n \"TestLakehouse\": {\n \"id\": \"test-lakehouse-id-12345\",\n \"sqlendpoint\": \"test-lakehouse.database.windows.net\",\n }\n },\n \"Warehouse\": {\n \"TestWarehouse\": {\n \"id\": \"test-warehouse-id-67890\",\n \"sqlendpoint\": \"test-warehouse.database.windows.net\",\n }\n },\n }\n mock_workspace._refresh_deployed_items = MagicMock()\n\n # Test JSON with item references\n test_json = '{\"lakehouse\": {\"id\": \"placeholder-id\", \"endpoint\": \"placeholder-endpoint\"}}'\n\n # Test with $items notation for lakehouse id\n param_dict = {\n \"find_key\": \"$.lakehouse.id\",\n \"replace_value\": {\n \"dev\": \"$items.Lakehouse.TestLakehouse.$id\",\n \"prod\": \"$items.Lakehouse.TestLakehouse.$id\",\n },\n }\n\n # Test successful replacement with $items notation\n result = replace_key_value(mock_workspace, param_dict, test_json, \"dev\")\n result_data = json.loads(result)\n assert result_data[\"lakehouse\"][\"id\"] == \"test-lakehouse-id-12345\"\n assert result_data[\"lakehouse\"][\"endpoint\"] == \"placeholder-endpoint\" # Unchanged\n\n # Test with $items notation for sqlendpoint\n param_dict = {\n \"find_key\": \"$.lakehouse.endpoint\",\n \"replace_value\": {\n \"dev\": \"$items.Lakehouse.TestLakehouse.$sqlendpoint\",\n \"prod\": \"$items.Warehouse.TestWarehouse.$sqlendpoint\",\n },\n }\n\n result = replace_key_value(mock_workspace, param_dict, test_json, \"dev\")\n result_data = json.loads(result)\n assert result_data[\"lakehouse\"][\"endpoint\"] == \"test-lakehouse.database.windows.net\"\n\n result = replace_key_value(mock_workspace, param_dict, test_json, \"prod\")\n result_data = json.loads(result)\n assert result_data[\"lakehouse\"][\"endpoint\"] == \"test-warehouse.database.windows.net\"\n\n def test_replace_key_value_with_items_notation_and_non_string_values(self, mock_workspace):\n \"\"\"Test replace_key_value function with $items notation mixed with other value types.\"\"\"\n # Mock the workspace to return item attributes\n mock_workspace.workspace_items = {\n \"Lakehouse\": {\n \"TestLakehouse\": {\n \"id\": \"test-lakehouse-id-12345\",\n }\n },\n }\n mock_workspace._refresh_deployed_items = MagicMock()\n\n # Test JSON with mixed value types\n test_json = '{\"config\": {\"enabled\": false, \"count\": 100, \"lakehouse_id\": \"placeholder\"}}'\n\n # Test with boolean value (should not process as $items)\n param_dict = {\n \"find_key\": \"$.config.enabled\",\n \"replace_value\": {\"dev\": True, \"prod\": False},\n }\n\n result = replace_key_value(mock_workspace, param_dict, test_json, \"dev\")\n result_data = json.loads(result)\n assert result_data[\"config\"][\"enabled\"] is True\n\n # Test with integer value (should not process as $items)\n param_dict = {\n \"find_key\": \"$.config.count\",\n \"replace_value\": {\"dev\": 200, \"prod\": 300},\n }\n\n result = replace_key_value(mock_workspace, param_dict, test_json, \"dev\")\n result_data = json.loads(result)\n assert result_data[\"config\"][\"count\"] == 200\n\n # Test with string $items notation\n param_dict = {\n \"find_key\": \"$.config.lakehouse_id\",\n \"replace_value\": {\n \"dev\": \"$items.Lakehouse.TestLakehouse.$id\",\n \"prod\": \"$items.Lakehouse.TestLakehouse.$id\",\n },\n }\n\n result = replace_key_value(mock_workspace, param_dict, test_json, \"dev\")\n result_data = json.loads(result)\n assert result_data[\"config\"][\"lakehouse_id\"] == \"test-lakehouse-id-12345\"\n\n def test_replace_key_value_yaml_valid(self, mock_workspace):\n \"\"\"Tests replace_key_value_yaml with valid YAML content and environment.\"\"\"\n # Test YAML with server host configuration\n test_yaml = \"\"\"server:\n host: localhost\n port: 8080\n\"\"\"\n param_dict = {\n \"find_key\": \"$.server.host\",\n \"replace_value\": {\"dev\": \"dev-server.example.com\", \"prod\": \"prod-server.example.com\"},\n }\n\n # Test successful replacement for dev environment\n result = replace_key_value(mock_workspace, param_dict, test_yaml, \"dev\", is_yaml=True)\n result_data = yaml.safe_load(result)\n assert result_data[\"server\"][\"host\"] == \"dev-server.example.com\"\n assert result_data[\"server\"][\"port\"] == 8080 # Verify other values unchanged\n\n # Test successful replacement for prod environment\n result = replace_key_value(mock_workspace, param_dict, test_yaml, \"prod\", is_yaml=True)\n result_data = yaml.safe_load(result)\n assert result_data[\"server\"][\"host\"] == \"prod-server.example.com\"\n\n def test_replace_key_value_yaml_environment_not_found(self, mock_workspace):\n \"\"\"Tests replace_key_value_yaml when environment is not in the replace_value dictionary.\"\"\"\n test_yaml = \"\"\"server:\n host: localhost\n port: 8080\n\"\"\"\n param_dict = {\n \"find_key\": \"$.server.host\",\n \"replace_value\": {\"dev\": \"dev-server.example.com\", \"prod\": \"prod-server.example.com\"},\n }\n\n # Test when environment not in replace_value\n result = replace_key_value(mock_workspace, param_dict, test_yaml, \"test\", is_yaml=True)\n result_data = yaml.safe_load(result)\n assert result_data[\"server\"][\"host\"] == \"localhost\" # Original value unchanged\n\n def test_replace_key_value_yaml_invalid(self, mock_workspace):\n \"\"\"Tests replace_key_value_yaml with invalid YAML content.\"\"\"\n invalid_yaml = \"invalid: yaml: content: [unclosed\"\n param_dict = {\"find_key\": \"$.server.host\", \"replace_value\": {\"dev\": \"test-server\"}}\n\n # YAMLError will be raised for invalid YAML and wrapped in ValueError\n with pytest.raises(ValueError, match=\"mapping values are not allowed\"):\n replace_key_value(mock_workspace, param_dict, invalid_yaml, \"dev\", is_yaml=True)\n\n def test_replace_key_value_yaml_empty_content(self, mock_workspace):\n \"\"\"Tests replace_key_value_yaml with empty YAML content.\"\"\"\n empty_yaml = \"\"\n param_dict = {\"find_key\": \"$.server.host\", \"replace_value\": {\"dev\": \"test-server\"}}\n\n # Empty YAML should return as-is\n result = replace_key_value(mock_workspace, param_dict, empty_yaml, \"dev\", is_yaml=True)\n assert result == empty_yaml\n\n def test_replace_key_value_yaml_nested_structure(self, mock_workspace):\n \"\"\"Tests replace_key_value_yaml with nested YAML structure like SparkCompute.yml.\"\"\"\n # Test YAML similar to SparkCompute.yml\n test_yaml = \"\"\"enable_native_execution_engine: false\ndriver_cores: 8\ndriver_memory: 56g\nexecutor_cores: 8\nexecutor_memory: 56g\ndynamic_executor_allocation:\n enabled: true\n min_executors: 1\n max_executors: 9\nruntime_version: \"1.2\"\n\"\"\"\n # Test replacing a nested value\n param_dict = {\n \"find_key\": \"$.dynamic_executor_allocation.max_executors\",\n \"replace_value\": {\"dev\": 5, \"prod\": 20},\n }\n\n result = replace_key_value(mock_workspace, param_dict, test_yaml, \"dev\", is_yaml=True)\n result_data = yaml.safe_load(result)\n assert result_data[\"dynamic_executor_allocation\"][\"max_executors\"] == 5\n assert result_data[\"dynamic_executor_allocation\"][\"min_executors\"] == 1 # Unchanged\n\n # Test replacing a top-level value\n param_dict = {\n \"find_key\": \"$.driver_cores\",\n \"replace_value\": {\"dev\": 4, \"prod\": 16},\n }\n\n result = replace_key_value(mock_workspace, param_dict, test_yaml, \"prod\", is_yaml=True)\n result_data = yaml.safe_load(result)\n assert result_data[\"driver_cores\"] == 16\n\n def test_replace_key_value_yaml_with_items_notation(self, mock_workspace):\n \"\"\"Test replace_key_value_yaml function with $items notation.\"\"\"\n # Mock the workspace to return item attributes\n mock_workspace.workspace_items = {\n \"Lakehouse\": {\n \"TestLakehouse\": {\n \"id\": \"test-lakehouse-id-12345\",\n \"sqlendpoint\": \"test-lakehouse.database.windows.net\",\n }\n },\n }\n mock_workspace._refresh_deployed_items = MagicMock()\n\n # Test YAML with item references\n test_yaml = \"\"\"lakehouse:\n id: placeholder-id\n endpoint: placeholder-endpoint\n\"\"\"\n\n # Test with $items notation for lakehouse id\n param_dict = {\n \"find_key\": \"$.lakehouse.id\",\n \"replace_value\": {\n \"dev\": \"$items.Lakehouse.TestLakehouse.$id\",\n \"prod\": \"$items.Lakehouse.TestLakehouse.$id\",\n },\n }\n\n # Test successful replacement with $items notation\n result = replace_key_value(mock_workspace, param_dict, test_yaml, \"dev\", is_yaml=True)\n result_data = yaml.safe_load(result)\n assert result_data[\"lakehouse\"][\"id\"] == \"test-lakehouse-id-12345\"\n assert result_data[\"lakehouse\"][\"endpoint\"] == \"placeholder-endpoint\" # Unchanged\n\n def test_replace_key_value_yaml_with_non_string_values(self, mock_workspace):\n \"\"\"Test replace_key_value_yaml function with non-string value types.\"\"\"\n # Test YAML with mixed value types\n test_yaml = \"\"\"config:\n enabled: false\n count: 100\n threshold: 0.5\n items:\n - item1\n - item2\n\"\"\"\n\n # Test with boolean value\n param_dict = {\n \"find_key\": \"$.config.enabled\",\n \"replace_value\": {\"dev\": True, \"prod\": False},\n }\n\n result = replace_key_value(mock_workspace, param_dict, test_yaml, \"dev\", is_yaml=True)\n result_data = yaml.safe_load(result)\n assert result_data[\"config\"][\"enabled\"] is True\n\n # Test with integer value\n param_dict = {\n \"find_key\": \"$.config.count\",\n \"replace_value\": {\"dev\": 200, \"prod\": 300},\n }\n\n result = replace_key_value(mock_workspace, param_dict, test_yaml, \"dev\", is_yaml=True)\n result_data = yaml.safe_load(result)\n assert result_data[\"config\"][\"count\"] == 200\n\n # Test with float value\n param_dict = {\n \"find_key\": \"$.config.threshold\",\n \"replace_value\": {\"dev\": 0.8, \"prod\": 0.95},\n }\n\n result = replace_key_value(mock_workspace, param_dict, test_yaml, \"dev\", is_yaml=True)\n result_data = yaml.safe_load(result)\n assert result_data[\"config\"][\"threshold\"] == 0.8\n\n def test_replace_variables_in_parameter_file(self, monkeypatch):\n \"\"\"Test replace_variables_in_parameter_file with feature flag enabled.\"\"\"\n # Set up test environment variables\n test_env_vars = {\n \"$ENV:TEST_VAR\": \"test_value\",\n \"$ENV:ANOTHER_VAR\": \"another_value\",\n \"NORMAL_VAR\": \"normal_value\", # Should be ignored\n }\n # Mock os.environ\n monkeypatch.setattr(\"os.environ\", test_env_vars)\n\n # Mock feature flag to be enabled\n monkeypatch.setattr(constants, \"FEATURE_FLAG\", [\"enable_environment_variable_replacement\"])\n\n # Test parameter file content with environment variables\n test_content = \"\"\"\n parameter:\n value: $ENV:TEST_VAR\n other: $ENV:ANOTHER_VAR\n normal: NORMAL_VAR\n \"\"\"\n result = replace_variables_in_parameter_file(test_content)\n # Verify replacements\n assert \"value: test_value\" in result\n assert \"other: another_value\" in result\n assert \"normal: NORMAL_VAR\" in result # Normal var unchanged\n\n def test_replace_variables_in_parameter_file_feature_disabled(self, monkeypatch):\n \"\"\"Test replace_variables_in_parameter_file with feature flag disabled.\"\"\"\n # Set up test environment variables with $ENV: prefix\n test_env_vars = {\n \"$ENV:TEST_VAR\": \"test_value\",\n \"$ENV:ANOTHER_VAR\": \"another_value\",\n }\n # Mock os.environ\n monkeypatch.setattr(\"os.environ\", test_env_vars)\n\n # Mock feature flag to be disabled (empty list)\n monkeypatch.setattr(constants, \"FEATURE_FLAG\", [])\n\n # Test parameter file content with environment variables\n test_content = \"\"\"\n parameter:\n value: $ENV:TEST_VAR\n other: $ENV:ANOTHER_VAR\n normal: NORMAL_VAR\n \"\"\"\n result = replace_variables_in_parameter_file(test_content)\n\n # Verify NO replacements occurred since feature is disabled\n # Environment variables should remain as-is in the output\n assert \"$ENV:TEST_VAR\" in result\n assert \"$ENV:ANOTHER_VAR\" in result\n assert \"NORMAL_VAR\" in result # Normal var unchanged\n\n # Make sure no replacements happened\n assert \"test_value\" not in result\n assert \"another_value\" not in result\n\n def test_replace_env_variables_in_content(self, monkeypatch):\n \"\"\"Test replace_variables_in_parameter_file with feature flag enabled.\"\"\"\n # Set up test environment variables with $ENV: prefix\n # This is required because the function filters os.environ for keys starting with $ENV:\n test_env_vars = {\n \"$ENV:TEST_VAR\": \"test_value\",\n \"$ENV:ANOTHER_VAR\": \"another_value\",\n \"NORMAL_VAR\": \"normal_value\", # Should be ignored (no $ENV: prefix)\n }\n # Mock os.environ\n monkeypatch.setattr(\"os.environ\", test_env_vars)\n\n # Mock feature flag to be enabled\n monkeypatch.setattr(constants, \"FEATURE_FLAG\", [\"enable_environment_variable_replacement\"])\n\n # Test parameter file content with environment variables\n test_content = \"\"\"\n parameter:\n value: $ENV:TEST_VAR\n other: $ENV:ANOTHER_VAR\n normal: NORMAL_VAR\n \"\"\"\n result = replace_variables_in_parameter_file(test_content)\n # Verify replacements\n assert \"value: test_value\" in result\n assert \"other: another_value\" in result\n assert \"normal: NORMAL_VAR\" in result # Normal var unchanged\n\n def test_process_environment_key(self, mock_workspace):\n \"\"\"Test process_environment_key function with ALL environment key replacement.\"\"\"\n # Test with ALL key only - should replace with target environment\n replace_value_dict_1 = {\"_ALL_\": \"universal-value\"}\n replace_value_dict_2 = {\"_all_\": \"universal-value\"}\n replace_value_dict_3 = {\"_All_\": \"universal-value\"}\n replace_value_dict_4 = {\"ALL\": \"universal-value\"}\n\n # Mock the workspace environment\n mock_workspace.environment = \"TEST\"\n\n # Call the function\n result_1 = process_environment_key(mock_workspace.environment, replace_value_dict_1)\n result_2 = process_environment_key(mock_workspace.environment, replace_value_dict_2)\n result_3 = process_environment_key(mock_workspace.environment, replace_value_dict_3)\n result_4 = process_environment_key(mock_workspace.environment, replace_value_dict_4)\n\n # Verify _ALL_ key is replaced with the target environment\n assert \"_ALL_\" not in result_1\n assert \"TEST\" in result_1\n assert result_1[\"TEST\"] == \"universal-value\"\n\n # Verify _all_ key is replaced with the target environment\n assert \"_all_\" not in result_2\n assert \"TEST\" in result_2\n assert result_2[\"TEST\"] == \"universal-value\"\n\n # Verify _All_ key is replaced with the target environment\n assert \"_All_\" not in result_3\n assert \"TEST\" in result_3\n assert result_3[\"TEST\"] == \"universal-value\"\n\n # Verify ALL key is replaced with the target environment\n assert \"ALL\" in result_4\n assert \"TEST\" not in result_4\n assert result_4[\"ALL\"] == \"universal-value\"\n\n assert result_1 == {\"TEST\": \"universal-value\"}\n assert result_1 == result_2 == result_3 != result_4\n\n # Test without ALL key - should return unchanged dictionary\n replace_value_dict_5 = {\n \"DEV\": \"dev-value\",\n \"PROD\": \"prod-value\",\n }\n\n # Mock the workspace environment\n mock_workspace.environment = \"TEST\"\n\n # Call the function\n result = process_environment_key(mock_workspace.environment, replace_value_dict_5)\n\n # Dictionary should remain unchanged\n assert result == replace_value_dict_5\n assert \"TEST\" not in result\n\n def test_validate_item_type_in_scope_with_none(self):\n \"\"\"Tests validate_item_type_in_scope function when None is passed.\"\"\"\n from fabric_cicd._common._validate_input import validate_item_type_in_scope\n\n # Test with None - should return all accepted item types\n result = validate_item_type_in_scope(None)\n assert result == list(constants.ACCEPTED_ITEM_TYPES)\n assert len(result) > 0 # Ensure we got some types back\n # Verify a few expected types are in the result\n assert \"Notebook\" in result\n assert \"DataPipeline\" in result\n assert \"Environment\" in result\n\n def test_validate_item_type_in_scope_with_valid_list(self):\n \"\"\"Tests validate_item_type_in_scope function with a valid list.\"\"\"\n from fabric_cicd._common._validate_input import validate_item_type_in_scope\n\n # Test with valid list\n valid_types = [\"Notebook\", \"Lakehouse\", \"Environment\"]\n result = validate_item_type_in_scope(valid_types)\n assert result == valid_types\n\n def test_validate_item_type_in_scope_with_invalid_type(self):\n \"\"\"Tests validate_item_type_in_scope function with invalid item type.\"\"\"\n from fabric_cicd._common._validate_input import validate_item_type_in_scope\n\n # Test with invalid item type\n invalid_types = [\"Notebook\", \"InvalidType\", \"Environment\"]\n with pytest.raises(InputError, match=\"Invalid or unsupported item type: 'InvalidType'\"):\n validate_item_type_in_scope(invalid_types)\n\n\nclass TestPathUtilities:\n \"\"\"Tests for path utility functions in _utils.py.\"\"\"\n\n def test_process_input_path_none(self, temp_repository):\n \"\"\"Tests process_input_path with none input.\"\"\"\n result = process_input_path(temp_repository, None)\n assert result is None\n\n def test_process_input_path_string(self, temp_repository, monkeypatch):\n \"\"\"Tests process_input_path with string input.\"\"\"\n\n # Mock the helper functions and glob.has_magic\n def mock_process_regular_path(path, repo, valid_paths, _):\n if path == \"file1.txt\":\n valid_paths.add(repo / \"file1.txt\")\n\n def mock_process_wildcard_path(path, repo, valid_paths, _):\n if path == \"*.txt\":\n valid_paths.add(repo / \"file1.txt\")\n valid_paths.add(repo / \"file2.txt\")\n\n def mock_has_magic(path):\n return \"*\" in path\n\n # Apply the mocks\n monkeypatch.setattr(\"fabric_cicd._parameter._utils._process_regular_path\", mock_process_regular_path)\n monkeypatch.setattr(\"fabric_cicd._parameter._utils._process_wildcard_path\", mock_process_wildcard_path)\n monkeypatch.setattr(\"glob.has_magic\", mock_has_magic)\n\n # Test with string path\n result = process_input_path(temp_repository, \"file1.txt\")\n assert isinstance(result, list)\n assert len(result) == 1\n assert result[0].name == \"file1.txt\"\n\n # Test with wildcard string\n result = process_input_path(temp_repository, \"*.txt\")\n assert isinstance(result, list)\n assert len(result) == 2 # Should find the 2 .txt files in root\n\n def test_process_input_path_list(self, temp_repository, monkeypatch):\n \"\"\"Tests process_input_path with list input.\"\"\"\n # Create a mapping of paths to the files they should find\n path_results = {\n \"file1.txt\": [temp_repository / \"file1.txt\"],\n \"*.json\": [temp_repository / \"file2.json\"],\n \"folder1/*.py\": [temp_repository / \"folder1\" / \"file3.py\"],\n }\n\n # Mock the helper functions and glob.has_magic\n def mock_process_regular_path(path, _, valid_paths, __):\n if path in path_results and \"*\" not in path:\n valid_paths.update(path_results[path])\n\n def mock_process_wildcard_path(path, _, valid_paths, __):\n if path in path_results and \"*\" in path:\n valid_paths.update(path_results[path])\n\n def mock_has_magic(path):\n return \"*\" in path\n\n # Apply the mocks\n monkeypatch.setattr(\"fabric_cicd._parameter._utils._process_regular_path\", mock_process_regular_path)\n monkeypatch.setattr(\"fabric_cicd._parameter._utils._process_wildcard_path\", mock_process_wildcard_path)\n monkeypatch.setattr(\"glob.has_magic\", mock_has_magic)\n\n # Test with list of paths including both regular and wildcard patterns\n paths = [\"file1.txt\", \"*.json\", \"folder1/*.py\"]\n result = process_input_path(temp_repository, paths)\n assert isinstance(result, list)\n assert len(result) == 3 # Should find file1.txt, file2.json, and folder1/file3.py\n assert any(p.name == \"file1.txt\" for p in result)\n assert any(p.name == \"file2.json\" for p in result)\n assert any(p.name == \"file3.py\" for p in result)\n\n def test_process_input_path_has_magic_exception(self, temp_repository, monkeypatch):\n \"\"\"Tests process_input_path when glob.has_magic raises an exception.\"\"\"\n # Create a mock logger to verify logging\n mock_logger = mock.MagicMock()\n monkeypatch.setattr(\"fabric_cicd._parameter._utils.logger\", mock_logger)\n\n # Mock glob.has_magic to raise an exception\n def mock_has_magic(_):\n msg = \"Mock exception in has_magic\"\n raise ValueError(msg)\n\n monkeypatch.setattr(\"glob.has_magic\", mock_has_magic)\n\n # Test with a single path - should handle the exception gracefully\n result = process_input_path(temp_repository, \"file1.txt\", False)\n\n # Verify the result is a list and it's empty (since we couldn't process the path)\n assert isinstance(result, list)\n assert len(result) == 0\n\n # Verify that the error was logged\n assert mock_logger.debug.called\n assert \"Error checking for wildcard\" in mock_logger.debug.call_args_list[0][0][0]\n mock_logger.reset_mock()\n\n # Test with a list of paths - should attempt to process each path but return empty list\n # since all paths will fail the glob.has_magic check with an exception\n result = process_input_path(temp_repository, [\"file1.txt\", \"file2.txt\"], False)\n assert isinstance(result, list)\n assert len(result) == 0\n\n # Verify that errors were logged for both paths\n assert mock_logger.debug.call_count == 2\n assert \"Error checking for wildcard\" in mock_logger.debug.call_args_list[0][0][0]\n assert \"Error checking for wildcard\" in mock_logger.debug.call_args_list[1][0][0]\n\n def test_resolve_input_path_with_invalid_wildcard_syntax(self, temp_repository, monkeypatch):\n \"\"\"Tests _resolve_input_path when _validate_wildcard_syntax returns False.\"\"\"\n # Create a valid path in the temp repository\n valid_path = temp_repository / \"test.txt\"\n valid_path.write_text(\"test content\")\n\n # Mock _validate_wildcard_syntax to return False for our test pattern\n def mock_validate_wildcard_syntax(pattern, _):\n return pattern != \"invalid*.txt\" # Return False only for our test pattern\n\n monkeypatch.setattr(\"fabric_cicd._parameter._utils._validate_wildcard_syntax\", mock_validate_wildcard_syntax)\n\n # Use a public function that calls _resolve_input_path with wildcard=True\n result = process_input_path(temp_repository, \"invalid*.txt\")\n\n # Should be empty because the wildcard validation failed\n assert len(result) == 0\n\n def test_process_input_path_some_invalid(self, temp_repository, monkeypatch):\n \"\"\"Tests process_input_path with some invalid paths.\"\"\"\n # Create a mock logger\n mock_logger = mock.MagicMock()\n monkeypatch.setattr(\"fabric_cicd._parameter._utils.logger\", mock_logger)\n\n # Create test files we need for this test\n (temp_repository / \"valid_file.txt\").write_text(\"valid content\")\n\n # Mock glob.has_magic to succeed for specific paths and fail for others\n import glob as glob_module\n\n original_has_magic = glob_module.has_magic\n\n def mock_has_magic(path):\n if path == \"error_path.txt\":\n msg = \"Mock error for specific path\"\n raise ValueError(msg)\n return original_has_magic(path)\n\n monkeypatch.setattr(\"glob.has_magic\", mock_has_magic)\n\n # Mock _resolve_file_path to return a valid path for specific files\n def mock_resolve_file_path(path, *_):\n if \"valid_file.txt\" in str(path):\n return path\n return None\n\n monkeypatch.setattr(\"fabric_cicd._parameter._utils._resolve_file_path\", mock_resolve_file_path)\n\n # Test with a mix of valid and problematic paths\n result = process_input_path(temp_repository, [\"valid_file.txt\", \"error_path.txt\", \"nonexistent_file.txt\"])\n\n # Should return only valid paths\n assert isinstance(result, list)\n assert len(result) == 1\n assert \"valid_file.txt\" in str(result[0])\n\n # Verify errors were logged for problematic paths\n assert mock_logger.debug.called\n assert any(\"Error checking for wildcard\" in call[0][0] for call in mock_logger.debug.call_args_list)\n\n def test_process_wildcard_path(self, temp_repository, monkeypatch):\n \"\"\"Tests _process_wildcard_path function.\"\"\"\n # Create the test files we need for this test\n (temp_repository / \"file1.txt\").write_text(\"content1\")\n (temp_repository / \"file2.txt\").write_text(\"content2\")\n (temp_repository / \"folder2\" / \"file5.txt\").write_text(\"content5\")\n\n # We need to patch the actual Path.glob method with our own implementation\n original_glob = Path.glob\n\n def patched_glob(self, pattern):\n # Special case for our test - return predefined results\n if str(self) == str(temp_repository):\n if pattern == \"*.txt\":\n return [temp_repository / \"file1.txt\", temp_repository / \"file2.txt\"]\n if pattern == \"**/*.txt\":\n return [\n temp_repository / \"file1.txt\",\n temp_repository / \"file2.txt\",\n temp_repository / \"folder2\" / \"file5.txt\",\n ]\n # Fall back to original method for other cases\n return original_glob(self, pattern)\n\n # Apply the patch\n monkeypatch.setattr(Path, \"glob\", patched_glob)\n\n # Set up a valid paths set\n valid_paths = set()\n mock_log = mock.MagicMock()\n\n # Mock _set_wildcard_path_pattern to return our test pattern\n def mock_set_pattern(pattern, _repo, _log):\n return \"*.txt\" if pattern == \"*.txt\" else \"**/*.txt\"\n\n monkeypatch.setattr(\"fabric_cicd._parameter._utils._set_wildcard_path_pattern\", mock_set_pattern)\n\n # Mock _resolve_file_path to return valid paths\n def mock_resolve_path(path, _repo, _path_type, _log):\n return path\n\n monkeypatch.setattr(\"fabric_cicd._parameter._utils._resolve_file_path\", mock_resolve_path)\n\n # Test with wildcard pattern for txt files\n _process_wildcard_path(\"*.txt\", temp_repository, valid_paths, mock_log)\n assert len(valid_paths) == 2 # Should find file1.txt and file2.txt in root\n assert all(path.suffix == \".txt\" for path in valid_paths)\n\n # Reset paths and test with recursive pattern\n valid_paths.clear()\n _process_wildcard_path(\"**/*.txt\", temp_repository, valid_paths, mock_log)\n assert len(valid_paths) == 3 # Should find all .txt files (including in subdirectories)\n\n def test_process_regular_path(self, temp_repository, monkeypatch):\n \"\"\"Tests _process_regular_path with regular file paths.\"\"\"\n\n # Set up a valid paths set\n valid_paths = set()\n mock_log = mock.MagicMock()\n\n # Mock _resolve_file_path to return valid paths for specific files\n def mock_resolve_file_path(path, _repo, _path_type, _log):\n if path.name == \"file1.txt\" or path.name == \"file2.json\":\n return path.resolve()\n return None\n\n monkeypatch.setattr(\"fabric_cicd._parameter._utils._resolve_file_path\", mock_resolve_file_path)\n\n # Test with specific file path\n _process_regular_path(\"file1.txt\", temp_repository, valid_paths, mock_log)\n assert len(valid_paths) == 1\n assert next(iter(valid_paths)).name == \"file1.txt\"\n\n # Reset and test with absolute path\n valid_paths.clear()\n abs_path = str(temp_repository / \"file2.json\")\n _process_regular_path(abs_path, temp_repository, valid_paths, mock_log)\n assert len(valid_paths) == 1\n assert next(iter(valid_paths)).name == \"file2.json\"\n\n # Test with nonexistent file\n valid_paths.clear()\n _process_regular_path(\"nonexistent.txt\", temp_repository, valid_paths, mock_log)\n assert len(valid_paths) == 0 # Should not add nonexistent files\n\n def test_resolve_nonexistent_file_path(self, temp_repository):\n \"\"\"Tests _resolve_file_path with nonexistent files.\"\"\"\n # Test nonexistent file\n file_path = temp_repository / \"nonexistent.txt\"\n result = _resolve_file_path(file_path, temp_repository, \"Relative\", logger.debug)\n assert result is None\n\n def test_resolve_directory_file_path(self, temp_repository):\n \"\"\"Tests _resolve_file_path with directories.\"\"\"\n # Test with directory instead of file\n dir_path = temp_repository / \"folder1\"\n result = _resolve_file_path(dir_path, temp_repository, \"Relative\", logger.debug)\n assert result is None\n\n def test_resolve_input_path_absolute_path(self):\n \"\"\"Test _resolve_input_path with absolute path.\"\"\"\n # Using a standard logger function format that takes a string message\n mock_logger = MagicMock()\n repo_dir = Path(\"c:/test_repo\").resolve() # Make sure it's resolved\n\n # Test with absolute path outside repository\n outside_path = Path(\"c:/outside/file.txt\").resolve() # Make sure it's resolved\n\n # Simulate a path outside the repo by mocking the relative_to method\n with mock.patch.object(Path, \"relative_to\", side_effect=ValueError(\"Path outside repo\")):\n result = _resolve_file_path(outside_path, repo_dir, \"Absolute\", mock_logger)\n # Check that the function returns None (path rejected)\n assert result is None\n # Check that the logger was called with an error about the path being outside\n mock_logger.assert_called_once_with(f\"Absolute path '{outside_path}' is outside the repository directory\")\n\n def test_resolve_outside_repo_file_path(self, temp_repository):\n \"\"\"Tests _resolve_file_path with paths outside the repository.\"\"\"\n # Create a file outside the repository\n outside_dir = Path(tempfile.mkdtemp())\n try:\n outside_file = outside_dir / \"outside.txt\"\n outside_file.write_text(\"outside content\")\n\n # Test with file outside repository\n result = _resolve_file_path(outside_file, temp_repository, \"Absolute\", logger.debug)\n assert result is None\n finally:\n shutil.rmtree(outside_dir)\n\n def test_resolve_invalid_file_path(self, temp_repository, monkeypatch):\n \"\"\"Tests _resolve_file_path with a path that causes exception.\"\"\"\n\n # Set up a mock that raises an exception when checking if file exists\n def mock_path_exists(_):\n msg = \"Permission denied\"\n raise PermissionError(msg)\n\n # Apply the mock\n monkeypatch.setattr(Path, \"exists\", mock_path_exists)\n\n # Test the exception handling\n file_path = temp_repository / \"file1.txt\"\n result = _resolve_file_path(file_path, temp_repository, \"Test\", logger.debug)\n assert result is None\n\n def test_validate_wildcard_syntax_invalid(self):\n \"\"\"Test _validate_wildcard_syntax with invalid wildcard syntax.\"\"\"\n # Create a mock function to pass as log_func\n mock_log_func = MagicMock()\n\n # Test with invalid recursive wildcard format - double asterisk without proper format\n # This will trigger the check: \"**\" in p and not (\"**/\" in p or \"/**\" in p)\n invalid_path = \"src**invalid.py\" # Missing slash between src and **\n\n # Call the function being tested\n result = _validate_wildcard_syntax(invalid_path, mock_log_func)\n\n # Verify validation fails\n assert result is False\n\n # Check that log_func was called exactly once with the expected message\n mock_log_func.assert_called_once_with(f\"Invalid recursive wildcard format (use **/ or /**): '{invalid_path}'\")\n\n def test_valid_wildcard_syntax(self):\n \"\"\"Tests that valid wildcard patterns pass validation.\"\"\"\n # Create a mock logger\n mock_log_func = mock.MagicMock()\n\n valid_patterns = [\n \"*.txt\",\n \"**/*.py\",\n \"folder1/*.json\",\n \"folder?/*.txt\",\n \"folder[1-3]/*.txt\",\n \"file[!1-3].txt\",\n \"file{1,2,3}.txt\",\n \"**/subfolder/*.md\",\n ]\n\n for pattern in valid_patterns:\n assert _validate_wildcard_syntax(pattern, mock_log_func) is True, f\"Pattern should be valid: {pattern}\"\n mock_log_func.assert_not_called() # No errors should be logged\n\n def test_invalid_wildcard_syntax(self):\n \"\"\"Tests that invalid wildcard patterns fail validation, including complex bracket/brace nesting issues.\"\"\"\n # Create a mock logger\n mock_log_func = mock.MagicMock()\n\n # Group 1: Basic validation errors\n basic_invalid_patterns = [\n \"\", # Empty string\n \" \", # Whitespace only\n \"../file.txt\", # Path traversal\n \"folder/../file.txt\", # Path traversal\n \"..%2Ffile.txt\", # Encoded path traversal\n ]\n\n # Group 2: Wildcard pattern errors\n wildcard_invalid_patterns = [\n \"/**/*/\", # Invalid combination\n \"**/**\", # Invalid combination\n \"folder//file.txt\", # Double slashes\n \"folder\\\\\\\\file.txt\", # Double backslashes\n \"**file.txt\", # Incorrect recursive format\n \"//**/test.txt\", # Absolute path with recursive pattern\n ]\n\n # Group 3: Bracket/brace validation errors\n bracket_brace_invalid_patterns = [\n \"folder[].txt\", # Empty brackets\n \"folder[abc.txt\", # Unclosed bracket\n \"folder{}.txt\", # Empty braces\n \"folder{abc.txt\", # Unclosed brace\n \"folder{,}.txt\", # Invalid comma in braces\n \"folder{a,,b}.txt\", # Empty option in braces\n \"folder{abc}.txt\", # Brace without comma\n \"folder[a-\", # Unclosed bracket with range\n ]\n\n # Test all invalid patterns\n all_invalid_patterns = basic_invalid_patterns + wildcard_invalid_patterns + bracket_brace_invalid_patterns\n for pattern in all_invalid_patterns:\n assert _validate_wildcard_syntax(pattern, mock_log_func) is False, f\"Pattern should be invalid: {pattern}\"\n mock_log_func.assert_called() # Error should be logged\n mock_log_func.reset_mock()\n\n complex_invalid_nested_patterns = [\n \"folder[[a[b]c].txt\", # Unbalanced nested brackets\n \"folder{a{b,c}.txt\", # Unbalanced nested braces\n ]\n\n # Test more complex bracket/brace nesting scenarios\n for pattern in complex_invalid_nested_patterns:\n assert _validate_wildcard_syntax(pattern, mock_log_func) is False, f\"Pattern should be invalid: {pattern}\"\n mock_log_func.assert_called() # Error should be logged\n mock_log_func.reset_mock()\n\n def test_validate_nested_brackets_braces(self):\n \"\"\"Tests the _validate_nested_brackets_braces function to ensure proper validation of bracket/brace nesting.\"\"\"\n from fabric_cicd._parameter._utils import _validate_nested_brackets_braces as validate_func\n\n mock_log_func = mock.MagicMock()\n\n valid_nested_patterns = [\n \"file[abc].txt\", # Simple bracket\n \"file{a,b,c}.txt\", # Simple brace\n \"file[abc]{1,2,3}.txt\", # Both brackets and braces\n \"file[a[b]c].txt\", # Nested brackets (valid in some glob implementations)\n \"file{a{b,c},d}.txt\", # Nested braces\n \"file[[]].txt\", # Escaped bracket in character class\n \"file[a-z].{txt,md}\", # Multiple bracket/brace pairs\n ]\n\n # Test valid patterns\n for pattern in valid_nested_patterns:\n assert validate_func(pattern, mock_log_func) is True, f\"Pattern should be valid: {pattern}\"\n mock_log_func.assert_not_called()\n mock_log_func.reset_mock()\n\n invalid_nested_patterns = [\n \"file[abc.txt\", # Unclosed bracket\n \"file{a,b.txt\", # Unclosed brace\n \"file]abc[.txt\", # Closing before opening\n \"file}abc{.txt\", # Closing before opening\n \"file[abc}.txt\", # Mismatched pairs\n \"file{abc].txt\", # Mismatched pairs\n \"file[a{b]c}.txt\", # Interleaved mismatched pairs\n \"file{a[b}c].txt\", # Interleaved mismatched pairs\n ]\n\n # Test invalid patterns\n for pattern in invalid_nested_patterns:\n assert validate_func(pattern, mock_log_func) is False, f\"Pattern should be invalid: {pattern}\"\n mock_log_func.assert_called_once()\n mock_log_func.reset_mock()\n" }, { "path": "tests/test_publish.py", "content": "# Copyright (c) Microsoft Corporation.\n# Licensed under the MIT License.\n\n\"\"\"Test publishing functionality including selective publishing based on repository content.\"\"\"\n\nimport json\nimport logging\nimport tempfile\nfrom pathlib import Path\nfrom typing import Optional\nfrom unittest.mock import MagicMock, patch\n\nimport pytest\nfrom fixtures.credentials import DummyTokenCredential\n\nimport fabric_cicd.publish as publish\nfrom fabric_cicd import constants\nfrom fabric_cicd._common._exceptions import InputError\nfrom fabric_cicd._items._notebook import NotebookPublisher\nfrom fabric_cicd.constants import API_FORMAT_MAPPING, ItemType\nfrom fabric_cicd.fabric_workspace import FabricWorkspace\n\n# =============================================================================\n# Shared Fixtures and Helpers\n# =============================================================================\n\n\n@pytest.fixture\ndef mock_endpoint():\n \"\"\"Mock FabricEndpoint to avoid real API calls.\"\"\"\n mock = MagicMock()\n\n def mock_invoke(method, url, **_kwargs):\n if method == \"GET\" and \"workspaces\" in url and not url.endswith(\"/items\"):\n return {\"body\": {\"value\": [], \"capacityId\": \"test-capacity\"}}\n if method == \"GET\" and url.endswith(\"/items\"):\n return {\"body\": {\"value\": []}}\n if method == \"POST\" and url.endswith(\"/folders\"):\n return {\"body\": {\"id\": \"mock-folder-id\"}}\n if method == \"POST\" and url.endswith(\"/items\"):\n return {\"body\": {\"id\": \"mock-item-id\", \"workspaceId\": \"mock-workspace-id\"}}\n return {\"body\": {\"value\": [], \"capacityId\": \"test-capacity\"}}\n\n mock.invoke.side_effect = mock_invoke\n return mock\n\n\n@pytest.fixture\ndef temp_workspace_dir():\n \"\"\"Create a temporary directory for test workspaces.\"\"\"\n with tempfile.TemporaryDirectory() as temp_dir:\n yield Path(temp_dir)\n\n\n@pytest.fixture\ndef experimental_feature_flags():\n \"\"\"Enable experimental feature flags for tests.\"\"\"\n original_flags = constants.FEATURE_FLAG.copy()\n constants.FEATURE_FLAG.add(\"enable_experimental_features\")\n constants.FEATURE_FLAG.add(\"enable_exclude_folder\")\n constants.FEATURE_FLAG.add(\"enable_include_folder\")\n constants.FEATURE_FLAG.add(\"enable_items_to_include\")\n yield\n constants.FEATURE_FLAG.clear()\n constants.FEATURE_FLAG.update(original_flags)\n\n\ndef create_test_item(base_path: Path, folder: Optional[str], name: str, item_type: str, logical_id: str) -> Path:\n \"\"\"Helper to create a test item with .platform file.\n\n Args:\n base_path: Root directory for the workspace.\n folder: Subfolder path (e.g., \"legacy\" or \"projects/team1\") or None for root-level.\n name: Display name of the item.\n item_type: Type of the item (e.g., \"Notebook\", \"SemanticModel\").\n logical_id: Logical ID for the item.\n\n Returns:\n Path to the created item directory.\n \"\"\"\n item_dir = base_path / folder / f\"{name}.{item_type}\" if folder else base_path / f\"{name}.{item_type}\"\n\n item_dir.mkdir(parents=True, exist_ok=True)\n\n platform_file = item_dir / \".platform\"\n metadata = {\n \"metadata\": {\n \"type\": item_type,\n \"displayName\": name,\n \"description\": f\"Test {item_type}\",\n },\n \"config\": {\"logicalId\": logical_id},\n }\n\n with platform_file.open(\"w\", encoding=\"utf-8\") as f:\n json.dump(metadata, f)\n\n with (item_dir / \"dummy.txt\").open(\"w\", encoding=\"utf-8\") as f:\n f.write(\"Dummy file content\")\n\n return item_dir\n\n\n# =============================================================================\n# Basic Publishing Tests\n# =============================================================================\n\n\ndef test_publish_only_existing_item_types(mock_endpoint, temp_workspace_dir):\n \"\"\"Test that publish_all_items only attempts to publish item types that exist in repository.\"\"\"\n create_test_item(temp_workspace_dir, None, \"TestNotebook\", \"Notebook\", \"test-notebook-id\")\n\n with (\n patch(\"fabric_cicd.fabric_workspace.FabricEndpoint\", return_value=mock_endpoint),\n patch.object(FabricWorkspace, \"_refresh_deployed_items\", new=lambda self: setattr(self, \"deployed_items\", {})),\n patch.object(\n FabricWorkspace, \"_refresh_deployed_folders\", new=lambda self: setattr(self, \"deployed_folders\", {})\n ),\n patch(\"fabric_cicd._items._notebook.NotebookPublisher\") as mock_notebook_cls,\n patch(\"fabric_cicd._items._environment.EnvironmentPublisher\") as mock_env_cls,\n ):\n mock_notebook_instance = mock_notebook_cls.return_value\n\n workspace = FabricWorkspace(\n workspace_id=\"12345678-1234-5678-abcd-1234567890ab\",\n repository_directory=str(temp_workspace_dir),\n token_credential=DummyTokenCredential(),\n )\n\n publish.publish_all_items(workspace)\n\n assert \"Notebook\" in workspace.repository_items\n assert \"Environment\" not in workspace.repository_items\n\n mock_notebook_cls.assert_called_once_with(workspace)\n mock_notebook_instance.publish_all.assert_called_once()\n mock_env_cls.assert_not_called()\n\n\ndef test_publish_ontology_item(mock_endpoint, temp_workspace_dir):\n \"\"\"Test that publish_all_items publishes Ontology items when present in repository.\"\"\"\n create_test_item(temp_workspace_dir, None, \"TestOntology\", \"Ontology\", \"test-ontology-id\")\n\n with (\n patch(\"fabric_cicd.fabric_workspace.FabricEndpoint\", return_value=mock_endpoint),\n patch.object(FabricWorkspace, \"_refresh_deployed_items\", new=lambda self: setattr(self, \"deployed_items\", {})),\n patch.object(\n FabricWorkspace, \"_refresh_deployed_folders\", new=lambda self: setattr(self, \"deployed_folders\", {})\n ),\n patch(\"fabric_cicd._items._ontology.OntologyPublisher\") as mock_ontology_cls,\n ):\n mock_ontology_instance = mock_ontology_cls.return_value\n\n workspace = FabricWorkspace(\n workspace_id=\"12345678-1234-5678-abcd-1234567890ab\",\n repository_directory=str(temp_workspace_dir),\n token_credential=DummyTokenCredential(),\n )\n\n publish.publish_all_items(workspace)\n\n assert \"Ontology\" in workspace.repository_items\n mock_ontology_cls.assert_called_once_with(workspace)\n mock_ontology_instance.publish_all.assert_called_once()\n\n\ndef test_publish_data_build_tool_job_item(mock_endpoint, temp_workspace_dir):\n \"\"\"Test that publish_all_items publishes DataBuildToolJob items when present in repository.\"\"\"\n create_test_item(temp_workspace_dir, None, \"TestDbtJob\", \"DataBuildToolJob\", \"test-dbt-job-id\")\n\n with (\n patch(\"fabric_cicd.fabric_workspace.FabricEndpoint\", return_value=mock_endpoint),\n patch.object(FabricWorkspace, \"_refresh_deployed_items\", new=lambda self: setattr(self, \"deployed_items\", {})),\n patch.object(\n FabricWorkspace, \"_refresh_deployed_folders\", new=lambda self: setattr(self, \"deployed_folders\", {})\n ),\n patch(\"fabric_cicd._items._databuildtooljob.DataBuildToolJobPublisher\") as mock_dbt_job_cls,\n ):\n mock_dbt_job_instance = mock_dbt_job_cls.return_value\n\n workspace = FabricWorkspace(\n workspace_id=\"12345678-1234-5678-abcd-1234567890ab\",\n repository_directory=str(temp_workspace_dir),\n token_credential=DummyTokenCredential(),\n )\n\n publish.publish_all_items(workspace)\n\n assert \"DataBuildToolJob\" in workspace.repository_items\n mock_dbt_job_cls.assert_called_once_with(workspace)\n mock_dbt_job_instance.publish_all.assert_called_once()\n\n\ndef test_default_none_item_type_in_scope_includes_all_types(mock_endpoint, temp_workspace_dir):\n \"\"\"Test that when item_type_in_scope is None (default), all available item types are included.\"\"\"\n with (\n patch(\"fabric_cicd.fabric_workspace.FabricEndpoint\", return_value=mock_endpoint),\n patch.object(FabricWorkspace, \"_refresh_deployed_items\", new=lambda self: setattr(self, \"deployed_items\", {})),\n patch.object(\n FabricWorkspace, \"_refresh_deployed_folders\", new=lambda self: setattr(self, \"deployed_folders\", {})\n ),\n ):\n workspace = FabricWorkspace(\n workspace_id=\"12345678-1234-5678-abcd-1234567890ab\",\n repository_directory=str(temp_workspace_dir),\n token_credential=DummyTokenCredential(),\n )\n\n expected_types = list(constants.ACCEPTED_ITEM_TYPES)\n assert set(workspace.item_type_in_scope) == set(expected_types)\n\n\ndef test_empty_item_type_in_scope_list(mock_endpoint, temp_workspace_dir):\n \"\"\"Test that passing an empty item_type_in_scope list works (no items to process).\"\"\"\n with patch(\"fabric_cicd.fabric_workspace.FabricEndpoint\", return_value=mock_endpoint):\n workspace = FabricWorkspace(\n workspace_id=\"12345678-1234-5678-abcd-1234567890ab\",\n repository_directory=str(temp_workspace_dir),\n item_type_in_scope=[],\n token_credential=DummyTokenCredential(),\n )\n assert workspace.item_type_in_scope == []\n\n\n# =============================================================================\n# Invalid Item Type Tests\n# =============================================================================\n\n\ndef test_invalid_item_types_in_scope(mock_endpoint, temp_workspace_dir):\n \"\"\"Test that passing invalid item types raises appropriate errors.\"\"\"\n with (\n patch(\"fabric_cicd.fabric_workspace.FabricEndpoint\", return_value=mock_endpoint),\n pytest.raises(InputError, match=\"Invalid or unsupported item type: 'InvalidItemType'\"),\n ):\n FabricWorkspace(\n workspace_id=\"12345678-1234-5678-abcd-1234567890ab\",\n repository_directory=str(temp_workspace_dir),\n item_type_in_scope=[\"InvalidItemType\"],\n token_credential=DummyTokenCredential(),\n )\n\n\ndef test_multiple_invalid_item_types_in_scope(mock_endpoint, temp_workspace_dir):\n \"\"\"Test that passing multiple invalid item types raises error for the first invalid one.\"\"\"\n with (\n patch(\"fabric_cicd.fabric_workspace.FabricEndpoint\", return_value=mock_endpoint),\n pytest.raises(InputError, match=\"Invalid or unsupported item type: 'FakeType'\"),\n ):\n FabricWorkspace(\n workspace_id=\"12345678-1234-5678-abcd-1234567890ab\",\n repository_directory=str(temp_workspace_dir),\n item_type_in_scope=[\"FakeType\", \"AnotherInvalidType\"],\n token_credential=DummyTokenCredential(),\n )\n\n\ndef test_mixed_valid_and_invalid_item_types_in_scope(mock_endpoint, temp_workspace_dir):\n \"\"\"Test that passing a mix of valid and invalid item types raises error for the invalid one.\"\"\"\n with (\n patch(\"fabric_cicd.fabric_workspace.FabricEndpoint\", return_value=mock_endpoint),\n pytest.raises(InputError, match=\"Invalid or unsupported item type: 'BadType'\"),\n ):\n FabricWorkspace(\n workspace_id=\"12345678-1234-5678-abcd-1234567890ab\",\n repository_directory=str(temp_workspace_dir),\n item_type_in_scope=[\"Notebook\", \"BadType\", \"Environment\"],\n token_credential=DummyTokenCredential(),\n )\n\n\n# =============================================================================\n# Unpublish Feature Flag Tests\n# =============================================================================\n\n\ndef test_unpublish_feature_flag_warnings(mock_endpoint, temp_workspace_dir, caplog):\n \"\"\"Test that warnings are logged when unpublish feature flags are missing.\"\"\"\n test_items = [\n (\"legacy\", \"TestLakehouse\", \"Lakehouse\", \"test-lakehouse-id\"),\n (\"legacy\", \"TestWarehouse\", \"Warehouse\", \"test-warehouse-id\"),\n (\"legacy\", \"TestSQLDB\", \"SQLDatabase\", \"test-sqldb-id\"),\n (\"legacy\", \"TestEventhouse\", \"Eventhouse\", \"test-eventhouse-id\"),\n ]\n\n for folder, name, item_type, logical_id in test_items:\n create_test_item(temp_workspace_dir, folder, name, item_type, logical_id)\n\n deployed_items = {item_type: {name: MagicMock()} for _, name, item_type, _ in test_items}\n\n with (\n patch(\"fabric_cicd.fabric_workspace.FabricEndpoint\", return_value=mock_endpoint),\n patch.object(\n FabricWorkspace,\n \"_refresh_deployed_items\",\n new=lambda self: setattr(self, \"deployed_items\", deployed_items),\n ),\n patch.object(\n FabricWorkspace, \"_refresh_deployed_folders\", new=lambda self: setattr(self, \"deployed_folders\", {})\n ),\n patch.object(FabricWorkspace, \"_unpublish_folders\", new=lambda _: None),\n caplog.at_level(logging.WARNING),\n ):\n workspace = FabricWorkspace(\n workspace_id=\"12345678-1234-5678-abcd-1234567890ab\",\n repository_directory=str(temp_workspace_dir),\n item_type_in_scope=[\"Lakehouse\", \"Warehouse\", \"SQLDatabase\", \"Eventhouse\"],\n token_credential=DummyTokenCredential(),\n )\n\n publish.unpublish_all_orphan_items(workspace)\n\n expected_warnings = [\n \"Skipping unpublish for Lakehouse items because the 'enable_lakehouse_unpublish' feature flag is not enabled.\",\n \"Skipping unpublish for Warehouse items because the 'enable_warehouse_unpublish' feature flag is not enabled.\",\n \"Skipping unpublish for SQLDatabase items because the 'enable_sqldatabase_unpublish' feature flag is not enabled.\",\n \"Skipping unpublish for Eventhouse items because the 'enable_eventhouse_unpublish' feature flag is not enabled.\",\n ]\n\n for expected_warning in expected_warnings:\n assert expected_warning in caplog.text\n\n\ndef test_unpublish_with_feature_flags_enabled(mock_endpoint, temp_workspace_dir, caplog):\n \"\"\"Test that no warnings are logged when unpublish feature flags are enabled.\"\"\"\n create_test_item(temp_workspace_dir, None, \"TestLakehouse\", \"Lakehouse\", \"test-lakehouse-id\")\n\n deployed_items = {\"Lakehouse\": {\"TestLakehouse\": MagicMock()}}\n\n original_flags = constants.FEATURE_FLAG.copy()\n constants.FEATURE_FLAG.add(\"enable_lakehouse_unpublish\")\n\n try:\n with (\n patch(\"fabric_cicd.fabric_workspace.FabricEndpoint\", return_value=mock_endpoint),\n patch.object(\n FabricWorkspace,\n \"_refresh_deployed_items\",\n new=lambda self: setattr(self, \"deployed_items\", deployed_items),\n ),\n patch.object(\n FabricWorkspace, \"_refresh_deployed_folders\", new=lambda self: setattr(self, \"deployed_folders\", {})\n ),\n patch.object(FabricWorkspace, \"_unpublish_folders\", new=lambda _: None),\n patch.object(FabricWorkspace, \"_unpublish_item\", new=lambda _, __, ___: None),\n caplog.at_level(logging.WARNING),\n ):\n workspace = FabricWorkspace(\n workspace_id=\"12345678-1234-5678-abcd-1234567890ab\",\n repository_directory=str(temp_workspace_dir),\n item_type_in_scope=[\"Lakehouse\"],\n token_credential=DummyTokenCredential(),\n )\n\n publish.unpublish_all_orphan_items(workspace)\n\n assert \"enable_lakehouse_unpublish\" not in caplog.text\n assert \"Skipping unpublish for Lakehouse\" not in caplog.text\n\n finally:\n constants.FEATURE_FLAG.clear()\n constants.FEATURE_FLAG.update(original_flags)\n\n\ndef test_unpublish_orphan_item_is_deleted(mock_endpoint, temp_workspace_dir):\n \"\"\"Test that unpublish_all_orphan_items deletes an orphaned item not in the repository.\"\"\"\n create_test_item(temp_workspace_dir, None, \"KeepMe\", \"Notebook\", \"keep-me-id\")\n\n orphan_deployed = {\n \"Notebook\": {\n \"KeepMe\": MagicMock(guid=\"keep-guid\"),\n \"OrphanNotebook\": MagicMock(guid=\"orphan-guid-123\"),\n }\n }\n orphan_repo = {\"Notebook\": {\"KeepMe\": MagicMock()}}\n\n unpublish_calls = []\n\n def track_unpublish(_self, item_name, item_type):\n unpublish_calls.append((item_name, item_type))\n\n with (\n patch(\"fabric_cicd.fabric_workspace.FabricEndpoint\", return_value=mock_endpoint),\n patch.object(\n FabricWorkspace,\n \"_refresh_deployed_items\",\n new=lambda self: setattr(self, \"deployed_items\", orphan_deployed),\n ),\n patch.object(\n FabricWorkspace,\n \"_refresh_repository_items\",\n new=lambda self: setattr(self, \"repository_items\", orphan_repo),\n ),\n patch.object(\n FabricWorkspace, \"_refresh_deployed_folders\", new=lambda self: setattr(self, \"deployed_folders\", {})\n ),\n patch.object(FabricWorkspace, \"_unpublish_folders\", new=lambda _: None),\n patch.object(FabricWorkspace, \"_unpublish_item\", new=track_unpublish),\n ):\n workspace = FabricWorkspace(\n workspace_id=\"12345678-1234-5678-abcd-1234567890ab\",\n repository_directory=str(temp_workspace_dir),\n item_type_in_scope=[\"Notebook\"],\n token_credential=DummyTokenCredential(),\n )\n\n publish.unpublish_all_orphan_items(workspace)\n\n assert len(unpublish_calls) == 1\n assert unpublish_calls[0] == (\"OrphanNotebook\", \"Notebook\")\n\n\ndef test_unpublish_orphan_excluded_by_regex(mock_endpoint, temp_workspace_dir):\n \"\"\"Test that orphaned items matching the exclude regex are NOT unpublished.\"\"\"\n create_test_item(temp_workspace_dir, None, \"KeepMe\", \"Notebook\", \"keep-me-id\")\n\n orphan_deployed = {\n \"Notebook\": {\n \"KeepMe\": MagicMock(guid=\"keep-guid\"),\n \"ProtectedOrphan\": MagicMock(guid=\"protected-guid\"),\n \"DeleteMe\": MagicMock(guid=\"delete-guid\"),\n }\n }\n orphan_repo = {\"Notebook\": {\"KeepMe\": MagicMock()}}\n\n unpublish_calls = []\n\n def track_unpublish(_self, item_name, item_type):\n unpublish_calls.append((item_name, item_type))\n\n with (\n patch(\"fabric_cicd.fabric_workspace.FabricEndpoint\", return_value=mock_endpoint),\n patch.object(\n FabricWorkspace,\n \"_refresh_deployed_items\",\n new=lambda self: setattr(self, \"deployed_items\", orphan_deployed),\n ),\n patch.object(\n FabricWorkspace,\n \"_refresh_repository_items\",\n new=lambda self: setattr(self, \"repository_items\", orphan_repo),\n ),\n patch.object(\n FabricWorkspace, \"_refresh_deployed_folders\", new=lambda self: setattr(self, \"deployed_folders\", {})\n ),\n patch.object(FabricWorkspace, \"_unpublish_folders\", new=lambda _: None),\n patch.object(FabricWorkspace, \"_unpublish_item\", new=track_unpublish),\n ):\n workspace = FabricWorkspace(\n workspace_id=\"12345678-1234-5678-abcd-1234567890ab\",\n repository_directory=str(temp_workspace_dir),\n item_type_in_scope=[\"Notebook\"],\n token_credential=DummyTokenCredential(),\n )\n\n publish.unpublish_all_orphan_items(workspace, item_name_exclude_regex=r\"^Protected.*\")\n\n assert (\"DeleteMe\", \"Notebook\") in unpublish_calls\n assert (\"ProtectedOrphan\", \"Notebook\") not in unpublish_calls\n\n\n@pytest.mark.usefixtures(\"experimental_feature_flags\")\ndef test_unpublish_orphan_filtered_by_items_to_include(mock_endpoint, temp_workspace_dir):\n \"\"\"Test that items_to_include limits which orphaned items are unpublished.\"\"\"\n create_test_item(temp_workspace_dir, None, \"KeepMe\", \"Notebook\", \"keep-me-id\")\n\n orphan_deployed = {\n \"Notebook\": {\n \"KeepMe\": MagicMock(guid=\"keep-guid\"),\n \"TargetOrphan\": MagicMock(guid=\"target-guid\"),\n \"OtherOrphan\": MagicMock(guid=\"other-guid\"),\n }\n }\n orphan_repo = {\"Notebook\": {\"KeepMe\": MagicMock()}}\n\n unpublish_calls = []\n\n def track_unpublish(_self, item_name, item_type):\n unpublish_calls.append((item_name, item_type))\n\n with (\n patch(\"fabric_cicd.fabric_workspace.FabricEndpoint\", return_value=mock_endpoint),\n patch.object(\n FabricWorkspace,\n \"_refresh_deployed_items\",\n new=lambda self: setattr(self, \"deployed_items\", orphan_deployed),\n ),\n patch.object(\n FabricWorkspace,\n \"_refresh_repository_items\",\n new=lambda self: setattr(self, \"repository_items\", orphan_repo),\n ),\n patch.object(\n FabricWorkspace, \"_refresh_deployed_folders\", new=lambda self: setattr(self, \"deployed_folders\", {})\n ),\n patch.object(FabricWorkspace, \"_unpublish_folders\", new=lambda _: None),\n patch.object(FabricWorkspace, \"_unpublish_item\", new=track_unpublish),\n ):\n workspace = FabricWorkspace(\n workspace_id=\"12345678-1234-5678-abcd-1234567890ab\",\n repository_directory=str(temp_workspace_dir),\n item_type_in_scope=[\"Notebook\"],\n token_credential=DummyTokenCredential(),\n )\n\n publish.unpublish_all_orphan_items(workspace, items_to_include=[\"TargetOrphan.Notebook\"])\n\n assert (\"TargetOrphan\", \"Notebook\") in unpublish_calls\n assert (\"OtherOrphan\", \"Notebook\") not in unpublish_calls\n\n\ndef test_unpublish_no_orphans_no_deletion(mock_endpoint, temp_workspace_dir):\n \"\"\"Test that unpublish_all_orphan_items does not call _unpublish_item when there are no orphans.\"\"\"\n create_test_item(temp_workspace_dir, None, \"MyNotebook\", \"Notebook\", \"my-notebook-id\")\n\n matching_items = {\"Notebook\": {\"MyNotebook\": MagicMock(guid=\"my-guid\")}}\n\n unpublish_calls = []\n\n def track_unpublish(_self, item_name, item_type):\n unpublish_calls.append((item_name, item_type))\n\n with (\n patch(\"fabric_cicd.fabric_workspace.FabricEndpoint\", return_value=mock_endpoint),\n patch.object(\n FabricWorkspace,\n \"_refresh_deployed_items\",\n new=lambda self: setattr(self, \"deployed_items\", matching_items),\n ),\n patch.object(\n FabricWorkspace,\n \"_refresh_repository_items\",\n new=lambda self: setattr(self, \"repository_items\", matching_items),\n ),\n patch.object(\n FabricWorkspace, \"_refresh_deployed_folders\", new=lambda self: setattr(self, \"deployed_folders\", {})\n ),\n patch.object(FabricWorkspace, \"_unpublish_folders\", new=lambda _: None),\n patch.object(FabricWorkspace, \"_unpublish_item\", new=track_unpublish),\n ):\n workspace = FabricWorkspace(\n workspace_id=\"12345678-1234-5678-abcd-1234567890ab\",\n repository_directory=str(temp_workspace_dir),\n item_type_in_scope=[\"Notebook\"],\n token_credential=DummyTokenCredential(),\n )\n\n publish.unpublish_all_orphan_items(workspace)\n\n assert len(unpublish_calls) == 0\n\n\n# =============================================================================\n# Publishing Order Tests\n# =============================================================================\n\n\ndef test_mirrored_database_published_before_lakehouse(mock_endpoint, temp_workspace_dir):\n \"\"\"Test that MirroredDatabase items are published before Lakehouse items to enable shortcuts.\"\"\"\n call_order = []\n\n def mock_publish_lakehouses():\n call_order.append(\"Lakehouse\")\n\n def mock_publish_mirroreddatabase():\n call_order.append(\"MirroredDatabase\")\n\n create_test_item(temp_workspace_dir, None, \"TestLakehouse\", \"Lakehouse\", \"test-lakehouse-id\")\n create_test_item(temp_workspace_dir, None, \"TestMirroredDB\", \"MirroredDatabase\", \"test-mirrored-db-id\")\n\n with (\n patch(\"fabric_cicd.fabric_workspace.FabricEndpoint\", return_value=mock_endpoint),\n patch.object(FabricWorkspace, \"_refresh_deployed_items\", new=lambda self: setattr(self, \"deployed_items\", {})),\n patch.object(\n FabricWorkspace, \"_refresh_deployed_folders\", new=lambda self: setattr(self, \"deployed_folders\", {})\n ),\n patch(\"fabric_cicd._items._lakehouse.LakehousePublisher\") as mock_lakehouse_cls,\n patch(\"fabric_cicd._items._mirroreddatabase.MirroredDatabasePublisher\") as mock_mirrored_cls,\n ):\n mock_lakehouse_instance = mock_lakehouse_cls.return_value\n mock_lakehouse_instance.publish_all.side_effect = mock_publish_lakehouses\n mock_mirrored_instance = mock_mirrored_cls.return_value\n mock_mirrored_instance.publish_all.side_effect = mock_publish_mirroreddatabase\n\n workspace = FabricWorkspace(\n workspace_id=\"12345678-1234-5678-abcd-1234567890ab\",\n repository_directory=str(temp_workspace_dir),\n item_type_in_scope=[\"Lakehouse\", \"MirroredDatabase\"],\n token_credential=DummyTokenCredential(),\n )\n\n publish.publish_all_items(workspace)\n\n assert len(call_order) == 2\n assert \"MirroredDatabase\" in call_order\n assert \"Lakehouse\" in call_order\n\n mirrored_db_index = call_order.index(\"MirroredDatabase\")\n lakehouse_index = call_order.index(\"Lakehouse\")\n assert mirrored_db_index < lakehouse_index, (\n f\"MirroredDatabase should be published before Lakehouse, but got order: {call_order}\"\n )\n\n\n# =============================================================================\n# Folder Exclusion Tests\n# =============================================================================\n\n\n@pytest.mark.usefixtures(\"experimental_feature_flags\")\ndef test_folder_exclusion_with_regex(mock_endpoint, temp_workspace_dir):\n \"\"\"Test that folder_path_exclude_regex can exclude entire folders of items.\"\"\"\n create_test_item(temp_workspace_dir, \"legacy\", \"LegacyNotebook\", \"Notebook\", \"legacy-notebook-id\")\n create_test_item(temp_workspace_dir, \"legacy\", \"LegacyModel\", \"SemanticModel\", \"legacy-model-id\")\n create_test_item(temp_workspace_dir, \"current\", \"CurrentNotebook\", \"Notebook\", \"current-notebook-id\")\n create_test_item(temp_workspace_dir, None, \"RootNotebook\", \"Notebook\", \"root-notebook-id\")\n\n with (\n patch(\"fabric_cicd.fabric_workspace.FabricEndpoint\", return_value=mock_endpoint),\n patch.object(FabricWorkspace, \"_refresh_deployed_items\", new=lambda self: setattr(self, \"deployed_items\", {})),\n patch.object(\n FabricWorkspace, \"_refresh_deployed_folders\", new=lambda self: setattr(self, \"deployed_folders\", {})\n ),\n ):\n workspace = FabricWorkspace(\n workspace_id=\"12345678-1234-5678-abcd-1234567890ab\",\n repository_directory=str(temp_workspace_dir),\n item_type_in_scope=[\"Notebook\", \"SemanticModel\"],\n token_credential=DummyTokenCredential(),\n )\n\n exclude_regex = r\".*legacy.*\"\n publish.publish_all_items(workspace, folder_path_exclude_regex=exclude_regex)\n\n assert \"Notebook\" in workspace.repository_items\n assert \"SemanticModel\" in workspace.repository_items\n\n assert workspace.repository_items[\"Notebook\"][\"LegacyNotebook\"].skip_publish is True\n assert workspace.repository_items[\"SemanticModel\"][\"LegacyModel\"].skip_publish is True\n\n assert workspace.repository_items[\"Notebook\"][\"CurrentNotebook\"].skip_publish is False\n assert workspace.repository_items[\"Notebook\"][\"RootNotebook\"].skip_publish is False\n\n\n@pytest.mark.usefixtures(\"experimental_feature_flags\")\ndef test_folder_exclusion_with_anchored_regex(mock_endpoint, temp_workspace_dir):\n \"\"\"Test that excluding a parent folder with an anchored regex also excludes\n items in child folders, preserving consistent hierarchy behavior.\"\"\"\n create_test_item(temp_workspace_dir, \"legacy\", \"LegacyNotebook\", \"Notebook\", \"legacy-notebook-id\")\n create_test_item(temp_workspace_dir, \"legacy/archived\", \"ArchivedNotebook\", \"Notebook\", \"archived-notebook-id\")\n create_test_item(temp_workspace_dir, \"current\", \"CurrentNotebook\", \"Notebook\", \"current-notebook-id\")\n\n with (\n patch(\"fabric_cicd.fabric_workspace.FabricEndpoint\", return_value=mock_endpoint),\n patch.object(FabricWorkspace, \"_refresh_deployed_items\", new=lambda self: setattr(self, \"deployed_items\", {})),\n patch.object(\n FabricWorkspace, \"_refresh_deployed_folders\", new=lambda self: setattr(self, \"deployed_folders\", {})\n ),\n ):\n workspace = FabricWorkspace(\n workspace_id=\"12345678-1234-5678-abcd-1234567890ab\",\n repository_directory=str(temp_workspace_dir),\n item_type_in_scope=[\"Notebook\"],\n token_credential=DummyTokenCredential(),\n )\n\n exclude_regex = r\"^/legacy$\"\n publish.publish_all_items(workspace, folder_path_exclude_regex=exclude_regex)\n\n assert workspace.repository_items[\"Notebook\"][\"LegacyNotebook\"].skip_publish is True\n assert workspace.repository_items[\"Notebook\"][\"ArchivedNotebook\"].skip_publish is True\n assert workspace.repository_items[\"Notebook\"][\"CurrentNotebook\"].skip_publish is False\n\n\ndef test_item_name_exclusion_still_works(mock_endpoint, temp_workspace_dir):\n \"\"\"Test that existing item name exclusion still works with the new folder exclusion feature.\"\"\"\n create_test_item(temp_workspace_dir, None, \"TestNotebook\", \"Notebook\", \"test-notebook-id\")\n create_test_item(temp_workspace_dir, None, \"DoNotPublish\", \"Notebook\", \"excluded-notebook-id\")\n\n with (\n patch(\"fabric_cicd.fabric_workspace.FabricEndpoint\", return_value=mock_endpoint),\n patch.object(FabricWorkspace, \"_refresh_deployed_items\", new=lambda self: setattr(self, \"deployed_items\", {})),\n patch.object(\n FabricWorkspace, \"_refresh_deployed_folders\", new=lambda self: setattr(self, \"deployed_folders\", {})\n ),\n ):\n workspace = FabricWorkspace(\n workspace_id=\"12345678-1234-5678-abcd-1234567890ab\",\n repository_directory=str(temp_workspace_dir),\n item_type_in_scope=[\"Notebook\"],\n token_credential=DummyTokenCredential(),\n )\n\n exclude_regex = r\".*DoNotPublish.*\"\n publish.publish_all_items(workspace, item_name_exclude_regex=exclude_regex)\n\n assert workspace.repository_items[\"Notebook\"][\"DoNotPublish\"].skip_publish is True\n assert workspace.repository_items[\"Notebook\"][\"TestNotebook\"].skip_publish is False\n\n\n# =============================================================================\n# Folder Inclusion Tests\n# =============================================================================\n\n\n@pytest.mark.usefixtures(\"experimental_feature_flags\")\ndef test_folder_inclusion_with_folder_path_to_include(mock_endpoint, temp_workspace_dir):\n \"\"\"Test that folder_path_to_include only filters items found within a Fabric folder.\"\"\"\n create_test_item(temp_workspace_dir, \"active\", \"ActiveNotebook\", \"Notebook\", \"active-notebook-id\")\n create_test_item(temp_workspace_dir, \"active\", \"ActiveModel\", \"SemanticModel\", \"active-model-id\")\n create_test_item(temp_workspace_dir, \"archive\", \"ArchivedNotebook\", \"Notebook\", \"archived-notebook-id\")\n create_test_item(temp_workspace_dir, None, \"RootNotebook\", \"Notebook\", \"root-notebook-id\")\n create_test_item(temp_workspace_dir, \"projects\", \"ProjectNotebook\", \"Notebook\", \"projects-notebook-id\")\n create_test_item(temp_workspace_dir, \"projects/team1\", \"NestedNotebook\", \"Notebook\", \"nested-notebook-id\")\n create_test_item(temp_workspace_dir, \"dept\", \"DeptNotebook\", \"Notebook\", \"dept-notebook-id\")\n create_test_item(temp_workspace_dir, \"dept/eng\", \"EngNotebook\", \"Notebook\", \"eng-notebook-id\")\n\n with (\n patch(\"fabric_cicd.fabric_workspace.FabricEndpoint\", return_value=mock_endpoint),\n patch.object(FabricWorkspace, \"_refresh_deployed_items\", new=lambda self: setattr(self, \"deployed_items\", {})),\n patch.object(\n FabricWorkspace, \"_refresh_deployed_folders\", new=lambda self: setattr(self, \"deployed_folders\", {})\n ),\n ):\n workspace = FabricWorkspace(\n workspace_id=\"12345678-1234-5678-abcd-1234567890ab\",\n repository_directory=str(temp_workspace_dir),\n item_type_in_scope=[\"Notebook\", \"SemanticModel\"],\n token_credential=DummyTokenCredential(),\n )\n\n publish.publish_all_items(\n workspace,\n folder_path_to_include=[\"/active\", \"/projects/team1\", \"/dept\", \"/dept/eng\"],\n )\n\n assert \"Notebook\" in workspace.repository_items\n assert \"SemanticModel\" in workspace.repository_items\n\n assert workspace.repository_items[\"Notebook\"][\"ActiveNotebook\"].skip_publish is False\n assert workspace.repository_items[\"SemanticModel\"][\"ActiveModel\"].skip_publish is False\n assert workspace.repository_items[\"Notebook\"][\"ArchivedNotebook\"].skip_publish is True\n assert workspace.repository_items[\"Notebook\"][\"RootNotebook\"].skip_publish is False\n assert workspace.repository_items[\"Notebook\"][\"NestedNotebook\"].skip_publish is False\n assert workspace.repository_items[\"Notebook\"][\"ProjectNotebook\"].skip_publish is True\n assert workspace.repository_items[\"Notebook\"][\"DeptNotebook\"].skip_publish is False\n assert workspace.repository_items[\"Notebook\"][\"EngNotebook\"].skip_publish is False\n\n\n@pytest.mark.usefixtures(\"experimental_feature_flags\")\ndef test_folder_inclusion_and_exclusion_together(mock_endpoint, temp_workspace_dir):\n \"\"\"Test that using both folder_path_to_include and folder_path_exclude_regex raises InputError.\"\"\"\n create_test_item(temp_workspace_dir, \"deploy\", \"DeployNotebook\", \"Notebook\", \"deploy-notebook-id\")\n\n with (\n patch(\"fabric_cicd.fabric_workspace.FabricEndpoint\", return_value=mock_endpoint),\n patch.object(FabricWorkspace, \"_refresh_deployed_items\", new=lambda self: setattr(self, \"deployed_items\", {})),\n patch.object(\n FabricWorkspace, \"_refresh_deployed_folders\", new=lambda self: setattr(self, \"deployed_folders\", {})\n ),\n ):\n workspace = FabricWorkspace(\n workspace_id=\"12345678-1234-5678-abcd-1234567890ab\",\n repository_directory=str(temp_workspace_dir),\n item_type_in_scope=[\"Notebook\"],\n token_credential=DummyTokenCredential(),\n )\n\n with pytest.raises(\n InputError,\n match=\"Cannot use both 'folder_path_exclude_regex' and 'folder_path_to_include'\",\n ):\n publish.publish_all_items(\n workspace,\n folder_path_to_include=[\"/deploy\"],\n folder_path_exclude_regex=r\"^/deploy/legacy\",\n )\n\n\n@pytest.mark.usefixtures(\"experimental_feature_flags\")\ndef test_empty_folder_path_to_include_raises_error(mock_endpoint, temp_workspace_dir):\n \"\"\"Test that passing an empty list for folder_path_to_include raises an InputError.\"\"\"\n with (\n patch(\"fabric_cicd.fabric_workspace.FabricEndpoint\", return_value=mock_endpoint),\n patch.object(FabricWorkspace, \"_refresh_deployed_items\", new=lambda self: setattr(self, \"deployed_items\", {})),\n patch.object(\n FabricWorkspace, \"_refresh_deployed_folders\", new=lambda self: setattr(self, \"deployed_folders\", {})\n ),\n ):\n workspace = FabricWorkspace(\n workspace_id=\"12345678-1234-5678-abcd-1234567890ab\",\n repository_directory=str(temp_workspace_dir),\n item_type_in_scope=[\"Notebook\"],\n token_credential=DummyTokenCredential(),\n )\n\n with pytest.raises(InputError, match=\"folder_path_to_include must not be an empty list\"):\n publish.publish_all_items(workspace, folder_path_to_include=[])\n\n\n# =============================================================================\n# Combined Filter Tests\n# =============================================================================\n\n\n@pytest.mark.usefixtures(\"experimental_feature_flags\")\ndef test_folder_exclusion_with_items_to_include(mock_endpoint, temp_workspace_dir):\n \"\"\"Test that folder exclusion takes precedence over items_to_include.\"\"\"\n create_test_item(temp_workspace_dir, \"legacy\", \"ImportantNotebook\", \"Notebook\", \"important-notebook-id\")\n create_test_item(temp_workspace_dir, None, \"StandaloneNotebook\", \"Notebook\", \"standalone-notebook-id\")\n create_test_item(temp_workspace_dir, None, \"OtherNotebook\", \"Notebook\", \"other-notebook-id\")\n\n with (\n patch(\"fabric_cicd.fabric_workspace.FabricEndpoint\", return_value=mock_endpoint),\n patch.object(FabricWorkspace, \"_refresh_deployed_items\", new=lambda self: setattr(self, \"deployed_items\", {})),\n patch.object(\n FabricWorkspace, \"_refresh_deployed_folders\", new=lambda self: setattr(self, \"deployed_folders\", {})\n ),\n ):\n workspace = FabricWorkspace(\n workspace_id=\"12345678-1234-5678-abcd-1234567890ab\",\n repository_directory=str(temp_workspace_dir),\n item_type_in_scope=[\"Notebook\"],\n token_credential=DummyTokenCredential(),\n )\n\n publish.publish_all_items(\n workspace,\n folder_path_exclude_regex=r\"^/legacy\",\n items_to_include=[\"ImportantNotebook.Notebook\", \"StandaloneNotebook.Notebook\"],\n )\n\n assert workspace.repository_items[\"Notebook\"][\"ImportantNotebook\"].skip_publish is True\n assert workspace.repository_items[\"Notebook\"][\"StandaloneNotebook\"].skip_publish is False\n # OtherNotebook is excluded by get_items_to_publish() because it is not in\n # items_to_include, so publish_all() marks it skip_publish=True.\n assert workspace.repository_items[\"Notebook\"][\"OtherNotebook\"].skip_publish is True\n\n\n@pytest.mark.usefixtures(\"experimental_feature_flags\")\ndef test_folder_inclusion_with_item_exclusion(mock_endpoint, temp_workspace_dir):\n \"\"\"Test that item_name_exclude_regex can exclude specific items within an included folder.\"\"\"\n create_test_item(temp_workspace_dir, \"active\", \"ActiveNotebook\", \"Notebook\", \"active-notebook-id\")\n create_test_item(temp_workspace_dir, \"active\", \"DebugNotebook\", \"Notebook\", \"debug-notebook-id\")\n\n with (\n patch(\"fabric_cicd.fabric_workspace.FabricEndpoint\", return_value=mock_endpoint),\n patch.object(FabricWorkspace, \"_refresh_deployed_items\", new=lambda self: setattr(self, \"deployed_items\", {})),\n patch.object(\n FabricWorkspace, \"_refresh_deployed_folders\", new=lambda self: setattr(self, \"deployed_folders\", {})\n ),\n ):\n workspace = FabricWorkspace(\n workspace_id=\"12345678-1234-5678-abcd-1234567890ab\",\n repository_directory=str(temp_workspace_dir),\n item_type_in_scope=[\"Notebook\"],\n token_credential=DummyTokenCredential(),\n )\n\n publish.publish_all_items(\n workspace,\n folder_path_to_include=[\"/active\"],\n item_name_exclude_regex=r\"^Debug.*\",\n )\n\n assert workspace.repository_items[\"Notebook\"][\"DebugNotebook\"].skip_publish is True\n assert workspace.repository_items[\"Notebook\"][\"ActiveNotebook\"].skip_publish is False\n\n\n@pytest.mark.usefixtures(\"experimental_feature_flags\")\ndef test_folder_inclusion_with_items_to_include(mock_endpoint, temp_workspace_dir):\n \"\"\"Test that folder_path_to_include and items_to_include work together to narrow the scope.\"\"\"\n create_test_item(temp_workspace_dir, \"active\", \"Notebook1\", \"Notebook\", \"notebook1-id\")\n create_test_item(temp_workspace_dir, \"active\", \"Notebook2\", \"Notebook\", \"notebook2-id\")\n create_test_item(temp_workspace_dir, \"archive\", \"ArchivedNotebook\", \"Notebook\", \"archived-notebook-id\")\n\n with (\n patch(\"fabric_cicd.fabric_workspace.FabricEndpoint\", return_value=mock_endpoint),\n patch.object(FabricWorkspace, \"_refresh_deployed_items\", new=lambda self: setattr(self, \"deployed_items\", {})),\n patch.object(\n FabricWorkspace, \"_refresh_deployed_folders\", new=lambda self: setattr(self, \"deployed_folders\", {})\n ),\n ):\n workspace = FabricWorkspace(\n workspace_id=\"12345678-1234-5678-abcd-1234567890ab\",\n repository_directory=str(temp_workspace_dir),\n item_type_in_scope=[\"Notebook\"],\n token_credential=DummyTokenCredential(),\n )\n\n publish.publish_all_items(\n workspace,\n folder_path_to_include=[\"/active\"],\n items_to_include=[\"Notebook1.Notebook\"],\n )\n\n assert workspace.repository_items[\"Notebook\"][\"Notebook1\"].skip_publish is False\n # Notebook2 and ArchivedNotebook are excluded by get_items_to_publish()\n # because they are not in items_to_include, so publish_all() marks them skip_publish=True.\n assert workspace.repository_items[\"Notebook\"][\"Notebook2\"].skip_publish is True\n assert workspace.repository_items[\"Notebook\"][\"ArchivedNotebook\"].skip_publish is True\n\n\n@pytest.mark.usefixtures(\"experimental_feature_flags\")\ndef test_all_filters_combined(mock_endpoint, temp_workspace_dir):\n \"\"\"Test the complete filter evaluation order with all filters applied.\"\"\"\n create_test_item(temp_workspace_dir, \"active\", \"DebugNotebook\", \"Notebook\", \"debug-id\")\n create_test_item(temp_workspace_dir, \"active\", \"TargetNotebook\", \"Notebook\", \"target-id\")\n create_test_item(temp_workspace_dir, \"active\", \"OtherNotebook\", \"Notebook\", \"other-id\")\n create_test_item(temp_workspace_dir, \"archive\", \"ArchivedNotebook\", \"Notebook\", \"archive-id\")\n\n with (\n patch(\"fabric_cicd.fabric_workspace.FabricEndpoint\", return_value=mock_endpoint),\n patch.object(FabricWorkspace, \"_refresh_deployed_items\", new=lambda self: setattr(self, \"deployed_items\", {})),\n patch.object(\n FabricWorkspace, \"_refresh_deployed_folders\", new=lambda self: setattr(self, \"deployed_folders\", {})\n ),\n ):\n workspace = FabricWorkspace(\n workspace_id=\"12345678-1234-5678-abcd-1234567890ab\",\n repository_directory=str(temp_workspace_dir),\n item_type_in_scope=[\"Notebook\"],\n token_credential=DummyTokenCredential(),\n )\n\n publish.publish_all_items(\n workspace,\n item_name_exclude_regex=r\"^Debug.*\",\n folder_path_to_include=[\"/active\"],\n items_to_include=[\"TargetNotebook.Notebook\"],\n )\n\n # DebugNotebook, OtherNotebook, and ArchivedNotebook are excluded by\n # get_items_to_publish() because they are not in items_to_include, so\n # publish_all() marks them skip_publish=True.\n assert workspace.repository_items[\"Notebook\"][\"DebugNotebook\"].skip_publish is True\n assert workspace.repository_items[\"Notebook\"][\"TargetNotebook\"].skip_publish is False\n assert workspace.repository_items[\"Notebook\"][\"OtherNotebook\"].skip_publish is True\n assert workspace.repository_items[\"Notebook\"][\"ArchivedNotebook\"].skip_publish is True\n\n\n# =============================================================================\n# NotebookPublisher Tests\n# =============================================================================\n\n\nclass TestNotebookPublisher:\n \"\"\"Tests for NotebookPublisher.publish_one method.\"\"\"\n\n @pytest.fixture\n def mock_workspace(self):\n \"\"\"Create a mock FabricWorkspace object.\"\"\"\n workspace = MagicMock()\n workspace._publish_item = MagicMock()\n return workspace\n\n @pytest.fixture\n def publisher(self, mock_workspace):\n \"\"\"Create a NotebookPublisher instance.\"\"\"\n publisher = NotebookPublisher.__new__(NotebookPublisher)\n publisher.fabric_workspace_obj = mock_workspace\n return publisher\n\n def _create_mock_item(self, file_suffix: str) -> MagicMock:\n \"\"\"Create a mock Item with a file of the given suffix.\"\"\"\n mock_file = MagicMock()\n mock_file.file_path = Path(f\"notebook{file_suffix}\")\n mock_item = MagicMock()\n mock_item.item_files = [mock_file]\n return mock_item\n\n def test_publish_ipynb_includes_api_format(self, publisher, mock_workspace):\n \"\"\"Test that .ipynb files include api_format in kwargs.\"\"\"\n item = self._create_mock_item(\".ipynb\")\n\n publisher.publish_one(\"test_notebook\", item)\n\n expected_api_format = API_FORMAT_MAPPING.get(ItemType.NOTEBOOK.value)\n mock_workspace._publish_item.assert_called_once_with(\n item_name=\"test_notebook\",\n item_type=ItemType.NOTEBOOK.value,\n api_format=expected_api_format,\n )\n\n def test_publish_non_ipynb_excludes_api_format(self, publisher, mock_workspace):\n \"\"\"Test that non-.ipynb files do not include api_format.\"\"\"\n item = self._create_mock_item(\".py\")\n\n publisher.publish_one(\"test_notebook\", item)\n\n mock_workspace._publish_item.assert_called_once_with(\n item_name=\"test_notebook\",\n item_type=ItemType.NOTEBOOK.value,\n )\n\n def test_publish_mixed_files_with_ipynb(self, publisher, mock_workspace):\n \"\"\"Test that if any file is .ipynb, api_format is included.\"\"\"\n mock_file_py = MagicMock()\n mock_file_py.file_path = Path(\"script.py\")\n mock_file_ipynb = MagicMock()\n mock_file_ipynb.file_path = Path(\"notebook.ipynb\")\n\n mock_item = MagicMock()\n mock_item.item_files = [mock_file_py, mock_file_ipynb]\n\n publisher.publish_one(\"test_notebook\", mock_item)\n\n expected_api_format = API_FORMAT_MAPPING.get(ItemType.NOTEBOOK.value)\n mock_workspace._publish_item.assert_called_once_with(\n item_name=\"test_notebook\",\n item_type=ItemType.NOTEBOOK.value,\n api_format=expected_api_format,\n )\n\n def test_item_type_is_notebook(self, publisher):\n \"\"\"Test that item_type is correctly set to Notebook.\"\"\"\n assert publisher.item_type == ItemType.NOTEBOOK.value\n\n def test_files_sorted_same_stem_content_before_settings(self, publisher):\n \"\"\"Test content file precedes settings even when filenames share the same stem.\"\"\"\n mock_platform = MagicMock()\n mock_platform.file_path = Path(\".platform\")\n mock_settings = MagicMock()\n mock_settings.file_path = Path(\"notebook-settings.json\")\n mock_content = MagicMock()\n mock_content.file_path = Path(\"notebook-content.py\")\n\n mock_item = MagicMock()\n mock_item.item_files = [mock_settings, mock_content, mock_platform]\n\n publisher.publish_one(\"test_notebook\", mock_item)\n\n assert mock_item.item_files[0].file_path.name == \".platform\"\n assert mock_item.item_files[1].file_path.name == \"notebook-content.py\"\n assert mock_item.item_files[2].file_path.name == \"notebook-settings.json\"\n\n def test_files_sorted_with_unknown_extension(self, publisher):\n \"\"\"Test that unknown file extensions get default priority (2) between content and settings.\"\"\"\n mock_platform = MagicMock()\n mock_platform.file_path = Path(\".platform\")\n mock_settings = MagicMock()\n mock_settings.file_path = Path(\"notebook.json\")\n mock_content = MagicMock()\n mock_content.file_path = Path(\"notebook.py\")\n mock_other = MagicMock()\n mock_other.file_path = Path(\"readme.md\") # Unknown extension\n\n mock_item = MagicMock()\n mock_item.item_files = [mock_settings, mock_other, mock_content, mock_platform]\n\n publisher.publish_one(\"test_notebook\", mock_item)\n\n # Expected order: .platform (0), notebook.py (1), readme.md (2), notebook.json (3)\n assert mock_item.item_files[0].file_path.name == \".platform\"\n assert mock_item.item_files[1].file_path.name == \"notebook.py\"\n assert mock_item.item_files[2].file_path.name == \"readme.md\"\n assert mock_item.item_files[3].file_path.name == \"notebook.json\"\n" }, { "path": "tests/test_response_collection.py", "content": "# Copyright (c) Microsoft Corporation.\n# Licensed under the MIT License.\n\n\"\"\"Test response collection functionality.\"\"\"\n\nimport json\nimport tempfile\nfrom pathlib import Path\nfrom unittest.mock import MagicMock, patch\n\nimport pytest\nfrom fixtures.credentials import DummyTokenCredential\n\nimport fabric_cicd.constants as constants\nimport fabric_cicd.publish as publish\nfrom fabric_cicd import append_feature_flag\nfrom fabric_cicd.fabric_workspace import FabricWorkspace\n\n\n@pytest.fixture\ndef mock_endpoint():\n \"\"\"Mock FabricEndpoint to return realistic responses.\"\"\"\n mock = MagicMock()\n\n def mock_invoke(method, url, body=None, **_kwargs):\n if method == \"GET\" and \"workspaces\" in url and not url.endswith(\"/items\"):\n return {\"body\": {\"value\": [], \"capacityId\": \"test-capacity\"}}\n if method == \"GET\" and url.endswith(\"/items\"):\n return {\"body\": {\"value\": []}}\n if method == \"POST\" and url.endswith(\"/folders\"):\n return {\"body\": {\"id\": \"mock-folder-id\"}}\n if method == \"POST\" and url.endswith(\"/items\"):\n return {\n \"body\": {\n \"id\": \"mock-item-id-12345\",\n \"workspaceId\": \"mock-workspace-id\",\n \"displayName\": body.get(\"displayName\", \"Test Item\"),\n \"type\": body.get(\"type\", \"Notebook\"),\n }\n }\n if method == \"POST\" and \"updateDefinition\" in url:\n return {\"body\": {\"message\": \"Definition updated successfully\"}}\n if method == \"PATCH\" and \"items/\" in url:\n return {\"body\": {\"message\": \"Item metadata updated successfully\"}}\n if method == \"POST\" and url.endswith(\"/move\"):\n return {\"body\": {\"message\": \"Item moved successfully\"}}\n if method == \"DELETE\" and \"items/\" in url:\n return {\"body\": {}, \"header\": {}, \"status_code\": 200}\n return {\"body\": {\"value\": [], \"capacityId\": \"test-capacity\"}}\n\n mock.invoke.side_effect = mock_invoke\n return mock\n\n\n@pytest.fixture\ndef test_workspace_with_notebook(mock_endpoint):\n \"\"\"Create a test workspace with a notebook item.\"\"\"\n with tempfile.TemporaryDirectory() as temp_dir:\n temp_path = Path(temp_dir)\n\n # Create a notebook item\n notebook_dir = temp_path / \"TestNotebook.Notebook\"\n notebook_dir.mkdir(parents=True, exist_ok=True)\n\n platform_file = notebook_dir / \".platform\"\n platform_file.write_text(\n json.dumps({\n \"metadata\": {\n \"kernel_info\": {\"name\": \"synapse_pyspark\"},\n \"language_info\": {\"name\": \"python\"},\n }\n })\n )\n\n notebook_file = notebook_dir / \"notebook-content.py\"\n notebook_file.write_text(\"# Test notebook content\\nprint('Hello World')\")\n\n # Patch FabricEndpoint before creating workspace\n with (\n patch(\"fabric_cicd.fabric_workspace.FabricEndpoint\", return_value=mock_endpoint),\n patch.object(\n FabricWorkspace, \"_refresh_deployed_items\", new=lambda self: setattr(self, \"deployed_items\", {})\n ),\n patch.object(\n FabricWorkspace, \"_refresh_deployed_folders\", new=lambda self: setattr(self, \"deployed_folders\", {})\n ),\n patch.object(FabricWorkspace, \"_refresh_repository_items\", new=lambda _: None),\n patch.object(FabricWorkspace, \"_refresh_repository_folders\", new=lambda _: None),\n ):\n workspace = FabricWorkspace(\n workspace_id=\"12345678-1234-5678-abcd-1234567890ab\",\n repository_directory=str(temp_path),\n item_type_in_scope=[\"Notebook\"],\n token_credential=DummyTokenCredential(),\n )\n # Manually set up repository items since we're patching the refresh methods\n workspace.repository_items = {\n \"Notebook\": {\n \"TestNotebook\": MagicMock(\n guid=None,\n folder_id=\"mock-folder-id\",\n logical_id=\"test-notebook-logical-id\",\n item_files=[\n MagicMock(\n relative_path=\"notebook-content.py\",\n type=\"text\",\n file_path=notebook_file,\n contents=\"# Test notebook content\\nprint('Hello World')\",\n base64_payload={\"path\": \"notebook-content.py\", \"payloadType\": \"InlineBase64\"},\n )\n ],\n skip_publish=False,\n path=notebook_dir,\n )\n }\n }\n workspace.deployed_items = {}\n # Set up parameter data to avoid parameter file warnings\n workspace.parameter_data = {}\n workspace.parameter_file_path = None\n yield workspace\n\n\n# =============================================================================\n# Initialization Tests\n# =============================================================================\n\n\ndef test_responses_initialized_as_none(test_workspace_with_notebook):\n \"\"\"Test that responses and unpublish_responses attributes are initialized as None by default.\"\"\"\n workspace = test_workspace_with_notebook\n assert workspace.responses is None\n assert workspace.unpublish_responses is None\n\n\n# =============================================================================\n# Publish Response Collection Tests\n# =============================================================================\n\n\ndef test_publish_item_without_response_collection(test_workspace_with_notebook):\n \"\"\"Test that _publish_item works normally when responses is None.\"\"\"\n workspace = test_workspace_with_notebook\n\n with (\n patch.object(workspace, \"_replace_logical_ids\", side_effect=lambda x: x),\n patch.object(workspace, \"_replace_parameters\", side_effect=lambda file, _: file.contents),\n patch.object(workspace, \"_replace_workspace_ids\", side_effect=lambda x: x),\n ):\n workspace._publish_item(item_name=\"TestNotebook\", item_type=\"Notebook\")\n assert workspace.responses is None\n assert workspace.unpublish_responses is None\n\n\ndef test_publish_item_with_response_collection(test_workspace_with_notebook):\n \"\"\"Test that _publish_item stores responses when feature flag is enabled.\"\"\"\n workspace = test_workspace_with_notebook\n\n constants.FEATURE_FLAG.add(\"enable_response_collection\")\n\n try:\n workspace.responses = {}\n\n with (\n patch.object(workspace, \"_replace_logical_ids\", side_effect=lambda x: x),\n patch.object(workspace, \"_replace_parameters\", side_effect=lambda file, _: file.contents),\n patch.object(workspace, \"_replace_workspace_ids\", side_effect=lambda x: x),\n ):\n workspace._publish_item(item_name=\"TestNotebook\", item_type=\"Notebook\")\n\n assert workspace.responses is not None\n assert \"Notebook\" in workspace.responses\n assert \"TestNotebook\" in workspace.responses[\"Notebook\"]\n response = workspace.responses[\"Notebook\"][\"TestNotebook\"]\n assert response[\"body\"][\"id\"] == \"mock-item-id-12345\"\n finally:\n constants.FEATURE_FLAG.discard(\"enable_response_collection\")\n\n\ndef test_publish_all_items_no_feature_flag(test_workspace_with_notebook):\n \"\"\"Test that publish_all_items doesn't enable responses by default.\"\"\"\n workspace = test_workspace_with_notebook\n\n result = publish.publish_all_items(workspace)\n\n assert result is None\n assert workspace.responses is None\n\n\ndef test_publish_all_items_with_feature_flag(test_workspace_with_notebook):\n \"\"\"Test that publish_all_items enables response collection when feature flag is set.\"\"\"\n workspace = test_workspace_with_notebook\n\n constants.FEATURE_FLAG.add(\"enable_response_collection\")\n\n try:\n result = publish.publish_all_items(workspace)\n\n assert workspace.responses is not None\n assert isinstance(workspace.responses, dict)\n assert result is workspace.responses\n finally:\n constants.FEATURE_FLAG.discard(\"enable_response_collection\")\n\n\ndef test_workspace_responses_access_pattern(test_workspace_with_notebook):\n \"\"\"Test the recommended access pattern for responses.\"\"\"\n workspace = test_workspace_with_notebook\n constants.FEATURE_FLAG.add(\"enable_response_collection\")\n\n try:\n publish.publish_all_items(workspace)\n\n assert hasattr(workspace, \"responses\")\n assert workspace.responses is not None\n\n if workspace.responses:\n for item_type, items in workspace.responses.items():\n assert isinstance(item_type, str)\n assert isinstance(items, dict)\n for item_name, response in items.items():\n assert isinstance(item_name, str)\n assert isinstance(response, dict)\n finally:\n constants.FEATURE_FLAG.discard(\"enable_response_collection\")\n\n\ndef test_publish_item_skipped_no_response_stored(test_workspace_with_notebook):\n \"\"\"Test that skipped items don't store responses even when collection is enabled.\"\"\"\n workspace = test_workspace_with_notebook\n\n constants.FEATURE_FLAG.add(\"enable_response_collection\")\n\n try:\n workspace.responses = {}\n workspace.publish_item_name_exclude_regex = \"TestNotebook\"\n\n workspace._publish_item(item_name=\"TestNotebook\", item_type=\"Notebook\")\n\n assert \"Notebook\" not in workspace.responses\n finally:\n constants.FEATURE_FLAG.discard(\"enable_response_collection\")\n\n\ndef test_append_feature_flag_enables_response_collection(test_workspace_with_notebook):\n \"\"\"Test that using append_feature_flag enables response collection.\"\"\"\n workspace = test_workspace_with_notebook\n\n append_feature_flag(\"enable_response_collection\")\n\n try:\n result = publish.publish_all_items(workspace)\n\n assert workspace.responses is not None\n assert isinstance(workspace.responses, dict)\n assert result is workspace.responses\n finally:\n constants.FEATURE_FLAG.discard(\"enable_response_collection\")\n\n\n# =============================================================================\n# Unpublish Response Collection Tests\n# =============================================================================\n\n\ndef test_unpublish_item_without_response_collection(test_workspace_with_notebook):\n \"\"\"Test that _unpublish_item does not store responses when collection is disabled.\"\"\"\n workspace = test_workspace_with_notebook\n workspace.deployed_items = {\"Notebook\": {\"TestNotebook\": MagicMock(guid=\"mock-guid-123\")}}\n\n workspace._unpublish_item(item_name=\"TestNotebook\", item_type=\"Notebook\")\n\n assert workspace.unpublish_responses is None\n\n\ndef test_unpublish_item_with_response_collection(test_workspace_with_notebook):\n \"\"\"Test that _unpublish_item stores responses in unpublish_responses.\"\"\"\n workspace = test_workspace_with_notebook\n workspace.deployed_items = {\"Notebook\": {\"TestNotebook\": MagicMock(guid=\"mock-guid-123\")}}\n\n constants.FEATURE_FLAG.add(\"enable_response_collection\")\n\n try:\n workspace.unpublish_responses = {}\n\n workspace._unpublish_item(item_name=\"TestNotebook\", item_type=\"Notebook\")\n\n assert \"Notebook\" in workspace.unpublish_responses\n assert \"TestNotebook\" in workspace.unpublish_responses[\"Notebook\"]\n finally:\n constants.FEATURE_FLAG.discard(\"enable_response_collection\")\n\n\ndef test_unpublish_item_does_not_write_to_publish_responses(test_workspace_with_notebook):\n \"\"\"Test that _unpublish_item does not write to self.responses.\"\"\"\n workspace = test_workspace_with_notebook\n workspace.deployed_items = {\"Notebook\": {\"TestNotebook\": MagicMock(guid=\"mock-guid-123\")}}\n\n constants.FEATURE_FLAG.add(\"enable_response_collection\")\n\n try:\n workspace.responses = {\"Notebook\": {\"ExistingItem\": {\"body\": {\"id\": \"existing\"}}}}\n workspace.unpublish_responses = {}\n\n workspace._unpublish_item(item_name=\"TestNotebook\", item_type=\"Notebook\")\n\n # publish responses unchanged — no TestNotebook added\n assert \"TestNotebook\" not in workspace.responses.get(\"Notebook\", {})\n assert workspace.responses[\"Notebook\"][\"ExistingItem\"][\"body\"][\"id\"] == \"existing\"\n finally:\n constants.FEATURE_FLAG.discard(\"enable_response_collection\")\n\n\ndef test_unpublish_item_failure_does_not_store_response(test_workspace_with_notebook, mock_endpoint):\n \"\"\"Test that _unpublish_item does not store responses when the DELETE call fails.\"\"\"\n workspace = test_workspace_with_notebook\n workspace.deployed_items = {\"Notebook\": {\"TestNotebook\": MagicMock(guid=\"mock-guid-123\")}}\n\n constants.FEATURE_FLAG.add(\"enable_response_collection\")\n\n # Make DELETE raise an exception\n original_side_effect = mock_endpoint.invoke.side_effect\n\n def failing_invoke(method, url, **kwargs):\n if method == \"DELETE\":\n msg = \"API error\"\n raise Exception(msg)\n return original_side_effect(method, url, **kwargs)\n\n mock_endpoint.invoke.side_effect = failing_invoke\n\n try:\n workspace.unpublish_responses = {}\n\n workspace._unpublish_item(item_name=\"TestNotebook\", item_type=\"Notebook\")\n\n # No response stored due to failure\n assert \"Notebook\" not in workspace.unpublish_responses\n finally:\n mock_endpoint.invoke.side_effect = original_side_effect\n constants.FEATURE_FLAG.discard(\"enable_response_collection\")\n\n\ndef test_unpublish_all_orphan_items_no_feature_flag(test_workspace_with_notebook):\n \"\"\"Test that unpublish_all_orphan_items returns None without the feature flag.\"\"\"\n workspace = test_workspace_with_notebook\n workspace.deployed_items = {}\n\n result = publish.unpublish_all_orphan_items(workspace)\n\n assert result is None\n assert workspace.unpublish_responses is None\n\n\ndef test_unpublish_all_orphan_items_with_feature_flag(test_workspace_with_notebook):\n \"\"\"Test that unpublish_all_orphan_items initializes unpublish_responses and returns populated dict.\"\"\"\n workspace = test_workspace_with_notebook\n\n # Set up an orphaned item: deployed but not in repository\n orphan_deployed = {\"Notebook\": {\"OrphanNotebook\": MagicMock(guid=\"orphan-guid-456\")}}\n orphan_repo = {}\n\n constants.FEATURE_FLAG.add(\"enable_response_collection\")\n\n try:\n assert workspace.unpublish_responses is None\n\n with (\n patch.object(\n FabricWorkspace,\n \"_refresh_deployed_items\",\n new=lambda self: setattr(self, \"deployed_items\", orphan_deployed),\n ),\n patch.object(\n FabricWorkspace,\n \"_refresh_repository_items\",\n new=lambda self: setattr(self, \"repository_items\", orphan_repo),\n ),\n ):\n result = publish.unpublish_all_orphan_items(workspace)\n\n assert workspace.unpublish_responses is not None\n assert isinstance(workspace.unpublish_responses, dict)\n assert \"Notebook\" in workspace.unpublish_responses\n assert \"OrphanNotebook\" in workspace.unpublish_responses[\"Notebook\"]\n assert result is workspace.unpublish_responses\n finally:\n constants.FEATURE_FLAG.discard(\"enable_response_collection\")\n\n\ndef test_unpublish_all_orphan_items_empty_returns_none(test_workspace_with_notebook):\n \"\"\"Test that unpublish_all_orphan_items returns None when no items are orphaned.\"\"\"\n workspace = test_workspace_with_notebook\n workspace.deployed_items = {}\n\n constants.FEATURE_FLAG.add(\"enable_response_collection\")\n\n try:\n result = publish.unpublish_all_orphan_items(workspace)\n\n # Empty responses dict is falsy, so return value is None\n assert result is None\n assert workspace.unpublish_responses is not None\n assert isinstance(workspace.unpublish_responses, dict)\n finally:\n constants.FEATURE_FLAG.discard(\"enable_response_collection\")\n\n\n# =============================================================================\n# Publish and Unpublish Response Separation Tests\n# =============================================================================\n\n\ndef test_unpublish_does_not_modify_publish_responses(test_workspace_with_notebook):\n \"\"\"Test that unpublish_all_orphan_items does not modify publish responses.\"\"\"\n workspace = test_workspace_with_notebook\n workspace.deployed_items = {}\n\n constants.FEATURE_FLAG.add(\"enable_response_collection\")\n\n try:\n workspace.responses = {\"Notebook\": {\"TestNotebook\": {\"body\": {\"id\": \"publish-response\"}}}}\n\n publish.unpublish_all_orphan_items(workspace)\n\n # Publish responses untouched\n assert workspace.responses[\"Notebook\"][\"TestNotebook\"][\"body\"][\"id\"] == \"publish-response\"\n # Unpublish responses initialized separately\n assert workspace.unpublish_responses is not None\n assert isinstance(workspace.unpublish_responses, dict)\n finally:\n constants.FEATURE_FLAG.discard(\"enable_response_collection\")\n\n\ndef test_publish_does_not_modify_unpublish_responses(test_workspace_with_notebook):\n \"\"\"Test that publish_all_items does not modify unpublish responses.\"\"\"\n workspace = test_workspace_with_notebook\n\n constants.FEATURE_FLAG.add(\"enable_response_collection\")\n\n try:\n workspace.unpublish_responses = {\"Notebook\": {\"OldNotebook\": {\"body\": {\"id\": \"unpublish-response\"}}}}\n\n publish.publish_all_items(workspace)\n\n # Unpublish responses untouched\n assert workspace.unpublish_responses[\"Notebook\"][\"OldNotebook\"][\"body\"][\"id\"] == \"unpublish-response\"\n # Publish responses initialized separately\n assert workspace.responses is not None\n assert isinstance(workspace.responses, dict)\n finally:\n constants.FEATURE_FLAG.discard(\"enable_response_collection\")\n\n\ndef test_publish_and_unpublish_responses_are_separate_dicts(test_workspace_with_notebook):\n \"\"\"Test that publish and unpublish use separate response dictionaries.\"\"\"\n workspace = test_workspace_with_notebook\n workspace.deployed_items = {}\n\n constants.FEATURE_FLAG.add(\"enable_response_collection\")\n\n try:\n publish.publish_all_items(workspace)\n publish.unpublish_all_orphan_items(workspace)\n\n assert workspace.responses is not workspace.unpublish_responses\n finally:\n constants.FEATURE_FLAG.discard(\"enable_response_collection\")\n" }, { "path": "tests/test_semantic_model_exclude.py", "content": "# Copyright (c) Microsoft Corporation.\n# Licensed under the MIT License.\n\n\"\"\"\nRegression tests: items_to_include + semantic model connection binding.\n\nWhen items_to_include scopes to a subset of semantic models, excluded models\nhave skip_publish=True and guid=\"\" after publish_all(). bind_semanticmodel_to_connection()\nmust not attempt to bind them — doing so would produce URLs like\n GET items//connections\n POST semanticModels//bindConnection\nwhich return HTTP 400.\n\"\"\"\n\nfrom unittest.mock import MagicMock\n\nfrom fabric_cicd._common._item import Item\nfrom fabric_cicd._items._semanticmodel import SemanticModelPublisher, bind_semanticmodel_to_connection\nfrom fabric_cicd.fabric_workspace import FabricWorkspace\n\n# ---------------------------------------------------------------------------\n# Helpers\n# ---------------------------------------------------------------------------\n\n\ndef _make_sm(name: str, guid: str = \"\", skip_publish: bool = False) -> Item:\n \"\"\"Create a real SemanticModel Item for testing.\"\"\"\n item = Item(type=\"SemanticModel\", name=name, description=\"\", guid=guid)\n item.skip_publish = skip_publish\n return item\n\n\ndef _make_connections(*conn_ids: str) -> dict:\n \"\"\"Build a minimal connections dict keyed by connection ID.\"\"\"\n return {\n cid: {\"id\": cid, \"connectivityType\": \"ShareableCloud\", \"connectionDetails\": {\"type\": \"SQL\", \"path\": \"srv\"}}\n for cid in conn_ids\n }\n\n\n# ---------------------------------------------------------------------------\n# Unit tests for bind_semanticmodel_to_connection()\n# ---------------------------------------------------------------------------\n\n\ndef test_bind_skips_model_with_skip_publish_true():\n \"\"\"\n bind_semanticmodel_to_connection() must skip models whose skip_publish=True.\n No HTTP call should be made for the excluded model.\n \"\"\"\n included = _make_sm(\"SalesModel\", guid=\"sales-guid\", skip_publish=False)\n excluded = _make_sm(\"DevModel\", guid=\"\", skip_publish=True) # excluded via items_to_include\n\n workspace = MagicMock(spec=FabricWorkspace)\n workspace.workspace_id = \"ws-123\"\n workspace.endpoint = MagicMock()\n workspace.repository_items = {\"SemanticModel\": {\"SalesModel\": included, \"DevModel\": excluded}}\n workspace.endpoint.invoke.return_value = {\n \"body\": {\"value\": [{\"id\": \"old-conn\", \"connectivityType\": \"ShareableCloud\", \"connectionDetails\": {}}]},\n \"status_code\": 200,\n }\n\n connections = _make_connections(\"conn-001\")\n connection_details = {\"SalesModel\": \"conn-001\", \"DevModel\": \"conn-001\"}\n\n bind_semanticmodel_to_connection(workspace, connections, connection_details)\n\n called_urls = [c[1][\"url\"] for c in workspace.endpoint.invoke.call_args_list]\n\n # Positive assertion: SalesModel MUST have triggered API calls (guards against vacuous all([]))\n assert any(\"sales-guid\" in url for url in called_urls), (\n f\"SalesModel should have triggered API calls, got: {called_urls}\"\n )\n # Negative assertion: DevModel must be entirely skipped\n assert not any(\"DevModel\" in url for url in called_urls), (\n f\"Empty-GUID or DevModel URL must not be called, got: {called_urls}\"\n )\n # Extra guard: no empty-segment URLs at all\n assert not any(\"//\" in url.split(\"://\", 1)[-1] for url in called_urls), (\n f\"Empty-GUID URL must not appear, got: {called_urls}\"\n )\n\n\ndef test_bind_skips_model_without_guid():\n \"\"\"\n bind_semanticmodel_to_connection() must skip models whose guid is empty,\n even when skip_publish is False. This is a defensive safety net against\n any future code path that leaves skip_publish unset.\n \"\"\"\n deployed = _make_sm(\"SalesModel\", guid=\"sales-guid\", skip_publish=False)\n not_deployed = _make_sm(\"DevModel\", guid=\"\", skip_publish=False) # guid guard must catch this\n\n workspace = MagicMock(spec=FabricWorkspace)\n workspace.workspace_id = \"ws-123\"\n workspace.endpoint = MagicMock()\n workspace.repository_items = {\"SemanticModel\": {\"SalesModel\": deployed, \"DevModel\": not_deployed}}\n workspace.endpoint.invoke.return_value = {\n \"body\": {\"value\": [{\"id\": \"old-conn\", \"connectivityType\": \"ShareableCloud\", \"connectionDetails\": {}}]},\n \"status_code\": 200,\n }\n\n connections = _make_connections(\"conn-001\")\n connection_details = {\"SalesModel\": \"conn-001\", \"DevModel\": \"conn-001\"}\n\n bind_semanticmodel_to_connection(workspace, connections, connection_details)\n\n called_urls = [c[1][\"url\"] for c in workspace.endpoint.invoke.call_args_list]\n # An empty GUID produces a path like \"items//connections\"; strip the scheme to detect it.\n assert not any(\"//\" in url.split(\"://\", 1)[-1] for url in called_urls), (\n f\"Empty-GUID URL must not appear, got: {called_urls}\"\n )\n assert not any(\"DevModel\" in url for url in called_urls), f\"DevModel must be skipped, got: {called_urls}\"\n\n\ndef test_bind_processes_included_models_normally():\n \"\"\"\n Models with skip_publish=False and a valid guid must still be bound.\n \"\"\"\n model_a = _make_sm(\"ModelA\", guid=\"guid-a\", skip_publish=False)\n model_b = _make_sm(\"ModelB\", guid=\"guid-b\", skip_publish=False)\n\n workspace = MagicMock(spec=FabricWorkspace)\n workspace.workspace_id = \"ws-123\"\n workspace.endpoint = MagicMock()\n workspace.repository_items = {\"SemanticModel\": {\"ModelA\": model_a, \"ModelB\": model_b}}\n workspace.endpoint.invoke.return_value = {\n \"body\": {\"value\": [{\"id\": \"old-conn\", \"connectivityType\": \"ShareableCloud\", \"connectionDetails\": {}}]},\n \"status_code\": 200,\n }\n\n connections = _make_connections(\"conn-001\")\n connection_details = {\"ModelA\": \"conn-001\", \"ModelB\": \"conn-001\"}\n\n bind_semanticmodel_to_connection(workspace, connections, connection_details)\n\n called_urls = [c[1][\"url\"] for c in workspace.endpoint.invoke.call_args_list]\n assert any(\"guid-a\" in url for url in called_urls), \"ModelA should have been processed\"\n assert any(\"guid-b\" in url for url in called_urls), \"ModelB should have been processed\"\n\n\n# ---------------------------------------------------------------------------\n# Integration test through SemanticModelPublisher.post_publish_all()\n# ---------------------------------------------------------------------------\n\n\ndef test_post_publish_all_skips_excluded_semantic_models():\n \"\"\"\n End-to-end regression: SemanticModelPublisher.post_publish_all() must not\n attempt connection binding for models excluded by items_to_include.\n\n publish_all() marks excluded models with skip_publish=True; the guard in\n bind_semanticmodel_to_connection() must then prevent any API call for them.\n \"\"\"\n included_model = _make_sm(\"SalesModel\", guid=\"sales-guid\", skip_publish=False)\n excluded_model = _make_sm(\"DevModel\", guid=\"\", skip_publish=True) # set by publish_all()\n\n workspace = MagicMock(spec=FabricWorkspace)\n workspace.workspace_id = \"ws-123\"\n workspace.environment = \"UAT\"\n workspace.endpoint = MagicMock()\n workspace.repository_items = {\"SemanticModel\": {\"SalesModel\": included_model, \"DevModel\": excluded_model}}\n workspace.environment_parameter = {\n \"semantic_model_binding\": {\n \"default\": {\n \"connection_id\": {\"_ALL_\": \"conn-001\"} # applies to ALL models by default\n }\n }\n }\n workspace.items_to_include = [\"SalesModel.SemanticModel\"]\n\n # Simulate the connections API and bind call responses\n def fake_invoke(method, url, **_kwargs):\n if method == \"GET\" and url.split(\"?\")[0].split(\"api.fabric.microsoft.com\")[-1] == \"/v1/connections\":\n return {\n \"body\": {\n \"value\": [\n {\n \"id\": \"conn-001\",\n \"connectivityType\": \"ShareableCloud\",\n \"connectionDetails\": {\"type\": \"SQL\", \"path\": \"srv\"},\n }\n ]\n }\n }\n if method == \"GET\" and \"/connections\" in url:\n return {\n \"body\": {\n \"value\": [\n {\n \"id\": \"old-conn\",\n \"connectivityType\": \"ShareableCloud\",\n \"connectionDetails\": {\"type\": \"SQL\", \"path\": \"srv\"},\n }\n ]\n }\n }\n if method == \"POST\" and \"bindConnection\" in url:\n return {\"status_code\": 200}\n return {\"body\": {}}\n\n workspace.endpoint.invoke.side_effect = fake_invoke\n\n publisher = SemanticModelPublisher.__new__(SemanticModelPublisher)\n publisher.fabric_workspace_obj = workspace\n publisher.item_type = \"SemanticModel\"\n\n publisher.post_publish_all()\n\n called_urls = [c[1][\"url\"] for c in workspace.endpoint.invoke.call_args_list]\n\n # An empty GUID produces a path like \"semanticModels//bindConnection\"; strip the scheme to detect it.\n assert not any(\"//\" in url.split(\"://\", 1)[-1] for url in called_urls), f\"Empty-GUID URL produced: {called_urls}\"\n # DevModel (excluded, guid='') must never appear in any URL\n assert not any(\"DevModel\" in url for url in called_urls), f\"DevModel must be skipped entirely: {called_urls}\"\n # SalesModel should be bound (bindConnection called with sales-guid)\n assert any(\"sales-guid\" in url and \"bindConnection\" in url for url in called_urls), (\n f\"SalesModel bindConnection not called: {called_urls}\"\n )\n" }, { "path": "tests/test_shortcut_exclude.py", "content": "# Copyright (c) Microsoft Corporation.\n# Licensed under the MIT License.\n\n\"\"\"Test shortcut exclusion functionality.\"\"\"\n\nimport json\nfrom unittest.mock import MagicMock, patch\n\nimport pytest\n\nfrom fabric_cicd import constants\nfrom fabric_cicd._common._item import Item\nfrom fabric_cicd._items._lakehouse import LakehousePublisher, ShortcutPublisher\nfrom fabric_cicd.constants import FeatureFlag\nfrom fabric_cicd.fabric_workspace import FabricWorkspace\n\n\n@pytest.fixture\ndef mock_fabric_workspace():\n \"\"\"Create a mock FabricWorkspace object.\"\"\"\n workspace = MagicMock(spec=FabricWorkspace)\n workspace.base_api_url = \"https://api.fabric.microsoft.com/v1\"\n workspace.shortcut_exclude_regex = None\n workspace.endpoint = MagicMock()\n\n # Mock the endpoint invoke method to return empty shortcuts list\n def mock_invoke(method, url, **_kwargs):\n if method == \"GET\" and \"shortcuts\" in url:\n return {\"body\": {\"value\": []}, \"header\": {}}\n if method == \"POST\" and \"shortcuts\" in url:\n return {\"body\": {\"id\": \"mock-shortcut-id\"}}\n return {\"body\": {}}\n\n workspace.endpoint.invoke.side_effect = mock_invoke\n\n # Mock parameter replacement methods to return content as-is\n workspace._replace_parameters = lambda file_obj, _item_obj: file_obj.contents\n workspace._replace_logical_ids = lambda contents: contents\n workspace._replace_workspace_ids = lambda contents: contents\n\n return workspace\n\n\n@pytest.fixture\ndef mock_item():\n \"\"\"Create a mock Item object.\"\"\"\n item = MagicMock(spec=Item)\n item.name = \"TestLakehouse\"\n item.guid = \"test-lakehouse-guid\"\n return item\n\n\ndef create_shortcut_file(shortcuts_data):\n \"\"\"Helper to create a mock file object with shortcut data.\"\"\"\n file_obj = MagicMock()\n file_obj.name = \"shortcuts.metadata.json\"\n file_obj.contents = json.dumps(shortcuts_data)\n return file_obj\n\n\ndef test_process_shortcuts_with_exclude_regex_filters_shortcuts(mock_fabric_workspace, mock_item):\n \"\"\"Test that shortcut_exclude_regex correctly filters shortcuts from deployment.\"\"\"\n\n # Create shortcuts data\n shortcuts_data = [\n {\n \"name\": \"temp_shortcut1\",\n \"path\": \"/Tables\",\n \"target\": {\n \"type\": \"OneLake\",\n \"oneLake\": {\n \"path\": \"Tables/temp1\",\n \"itemId\": \"test-item-id\",\n \"workspaceId\": \"test-workspace-id\",\n \"artifactType\": \"Lakehouse\",\n },\n },\n },\n {\n \"name\": \"production_shortcut\",\n \"path\": \"/Tables\",\n \"target\": {\n \"type\": \"OneLake\",\n \"oneLake\": {\n \"path\": \"Tables/prod\",\n \"itemId\": \"test-item-id\",\n \"workspaceId\": \"test-workspace-id\",\n \"artifactType\": \"Lakehouse\",\n },\n },\n },\n {\n \"name\": \"temp_shortcut2\",\n \"path\": \"/Files\",\n \"target\": {\n \"type\": \"OneLake\",\n \"oneLake\": {\n \"path\": \"Files/temp2\",\n \"itemId\": \"test-item-id\",\n \"workspaceId\": \"test-workspace-id\",\n \"artifactType\": \"Lakehouse\",\n },\n },\n },\n ]\n\n # Create mock file with shortcuts\n shortcut_file = create_shortcut_file(shortcuts_data)\n mock_item.item_files = [shortcut_file]\n\n # Set exclude regex to filter out shortcuts starting with \"temp_\"\n mock_fabric_workspace.shortcut_exclude_regex = \"^temp_.*\"\n\n # Call process_shortcuts\n ShortcutPublisher(mock_fabric_workspace, mock_item).publish_all()\n\n # Verify that only the production_shortcut was published\n post_calls = [\n call\n for call in mock_fabric_workspace.endpoint.invoke.call_args_list\n if call[1].get(\"method\") == \"POST\" and \"shortcuts\" in call[1].get(\"url\", \"\")\n ]\n\n # Should have only 1 shortcut published (production_shortcut)\n assert len(post_calls) == 1\n\n # Verify the published shortcut is the production one\n published_shortcut = post_calls[0][1][\"body\"]\n assert published_shortcut[\"name\"] == \"production_shortcut\"\n\n\ndef test_process_shortcuts_without_exclude_regex_publishes_all(mock_fabric_workspace, mock_item):\n \"\"\"Test that when shortcut_exclude_regex is None, all shortcuts are published.\"\"\"\n\n # Create shortcuts data\n shortcuts_data = [\n {\n \"name\": \"shortcut1\",\n \"path\": \"/Tables\",\n \"target\": {\n \"type\": \"OneLake\",\n \"oneLake\": {\n \"path\": \"Tables/s1\",\n \"itemId\": \"test-item-id\",\n \"workspaceId\": \"test-workspace-id\",\n \"artifactType\": \"Lakehouse\",\n },\n },\n },\n {\n \"name\": \"shortcut2\",\n \"path\": \"/Files\",\n \"target\": {\n \"type\": \"OneLake\",\n \"oneLake\": {\n \"path\": \"Files/s2\",\n \"itemId\": \"test-item-id\",\n \"workspaceId\": \"test-workspace-id\",\n \"artifactType\": \"Lakehouse\",\n },\n },\n },\n ]\n\n # Create mock file with shortcuts\n shortcut_file = create_shortcut_file(shortcuts_data)\n mock_item.item_files = [shortcut_file]\n\n # No exclude regex set (None)\n mock_fabric_workspace.shortcut_exclude_regex = None\n\n # Call process_shortcuts\n ShortcutPublisher(mock_fabric_workspace, mock_item).publish_all()\n\n # Verify that both shortcuts were published\n post_calls = [\n call\n for call in mock_fabric_workspace.endpoint.invoke.call_args_list\n if call[1].get(\"method\") == \"POST\" and \"shortcuts\" in call[1].get(\"url\", \"\")\n ]\n\n # Should have 2 shortcuts published\n assert len(post_calls) == 2\n\n\ndef test_process_shortcuts_exclude_regex_excludes_all_matching(mock_fabric_workspace, mock_item):\n \"\"\"Test that shortcut_exclude_regex excludes all matching shortcuts.\"\"\"\n\n # Create shortcuts data with all matching the pattern\n shortcuts_data = [\n {\n \"name\": \"temp_shortcut1\",\n \"path\": \"/Tables\",\n \"target\": {\n \"type\": \"OneLake\",\n \"oneLake\": {\n \"path\": \"Tables/temp1\",\n \"itemId\": \"test-item-id\",\n \"workspaceId\": \"test-workspace-id\",\n \"artifactType\": \"Lakehouse\",\n },\n },\n },\n {\n \"name\": \"temp_shortcut2\",\n \"path\": \"/Files\",\n \"target\": {\n \"type\": \"OneLake\",\n \"oneLake\": {\n \"path\": \"Files/temp2\",\n \"itemId\": \"test-item-id\",\n \"workspaceId\": \"test-workspace-id\",\n \"artifactType\": \"Lakehouse\",\n },\n },\n },\n ]\n\n # Create mock file with shortcuts\n shortcut_file = create_shortcut_file(shortcuts_data)\n mock_item.item_files = [shortcut_file]\n\n # Set exclude regex that matches all shortcuts\n mock_fabric_workspace.shortcut_exclude_regex = \"^temp_.*\"\n\n # Call process_shortcuts\n ShortcutPublisher(mock_fabric_workspace, mock_item).publish_all()\n\n # Verify that no shortcuts were published\n post_calls = [\n call\n for call in mock_fabric_workspace.endpoint.invoke.call_args_list\n if call[1].get(\"method\") == \"POST\" and \"shortcuts\" in call[1].get(\"url\", \"\")\n ]\n\n # Should have 0 shortcuts published\n assert len(post_calls) == 0\n\n\ndef test_process_shortcuts_with_complex_regex_pattern(mock_fabric_workspace, mock_item):\n \"\"\"Test shortcut exclusion with a more complex regex pattern.\"\"\"\n\n # Create shortcuts data\n shortcuts_data = [\n {\n \"name\": \"dev_temp_shortcut\",\n \"path\": \"/Tables\",\n \"target\": {\n \"type\": \"OneLake\",\n \"oneLake\": {\n \"path\": \"Tables/dev_temp\",\n \"itemId\": \"test-item-id\",\n \"workspaceId\": \"test-workspace-id\",\n \"artifactType\": \"Lakehouse\",\n },\n },\n },\n {\n \"name\": \"prod_shortcut\",\n \"path\": \"/Tables\",\n \"target\": {\n \"type\": \"OneLake\",\n \"oneLake\": {\n \"path\": \"Tables/prod\",\n \"itemId\": \"test-item-id\",\n \"workspaceId\": \"test-workspace-id\",\n \"artifactType\": \"Lakehouse\",\n },\n },\n },\n {\n \"name\": \"staging_temp_data\",\n \"path\": \"/Files\",\n \"target\": {\n \"type\": \"OneLake\",\n \"oneLake\": {\n \"path\": \"Files/staging_temp\",\n \"itemId\": \"test-item-id\",\n \"workspaceId\": \"test-workspace-id\",\n \"artifactType\": \"Lakehouse\",\n },\n },\n },\n ]\n\n # Create mock file with shortcuts\n shortcut_file = create_shortcut_file(shortcuts_data)\n mock_item.item_files = [shortcut_file]\n\n # Set exclude regex to filter shortcuts containing \"_temp\"\n mock_fabric_workspace.shortcut_exclude_regex = \".*_temp.*\"\n\n # Call process_shortcuts\n ShortcutPublisher(mock_fabric_workspace, mock_item).publish_all()\n\n # Verify that only prod_shortcut was published\n post_calls = [\n call\n for call in mock_fabric_workspace.endpoint.invoke.call_args_list\n if call[1].get(\"method\") == \"POST\" and \"shortcuts\" in call[1].get(\"url\", \"\")\n ]\n\n # Should have only 1 shortcut published (prod_shortcut)\n assert len(post_calls) == 1\n\n # Verify the published shortcut is the prod one\n published_shortcut = post_calls[0][1][\"body\"]\n assert published_shortcut[\"name\"] == \"prod_shortcut\"\n\n\n# =============================================================================\n# Regression tests: items_to_include + shortcut publishing\n# =============================================================================\n\n\n@pytest.fixture\ndef shortcut_publish_enabled():\n \"\"\"Enable the ENABLE_SHORTCUT_PUBLISH feature flag for the duration of a test.\"\"\"\n original_flags = constants.FEATURE_FLAG.copy()\n constants.FEATURE_FLAG.add(FeatureFlag.ENABLE_SHORTCUT_PUBLISH.value)\n yield\n constants.FEATURE_FLAG.clear()\n constants.FEATURE_FLAG.update(original_flags)\n\n\ndef _make_item(name: str, guid: str = \"\") -> Item:\n \"\"\"Create a real Item for testing skip_publish and guid behaviour.\"\"\"\n return Item(type=\"Lakehouse\", name=name, description=\"\", guid=guid)\n\n\ndef test_excluded_lakehouses_marked_skip_publish_with_items_to_include():\n \"\"\"\n When items_to_include is set and the workspace contains lakehouses that are\n NOT in the include list, publish_all() must mark those lakehouses\n skip_publish=True so that post_publish_all() does not attempt to publish\n their shortcuts (which would fail with a 400 error because guid is \"\").\n \"\"\"\n lh_bronze = _make_item(\"lh_bronze\", guid=\"bronze-guid\")\n lh_silver = _make_item(\"lh_silver\", guid=\"\") # not deployed to this environment\n\n workspace = MagicMock(spec=FabricWorkspace)\n workspace.repository_items = {\"Lakehouse\": {\"lh_bronze\": lh_bronze, \"lh_silver\": lh_silver}}\n workspace.items_to_include = [\"lh_bronze.Lakehouse\"]\n\n publisher = LakehousePublisher.__new__(LakehousePublisher)\n publisher.fabric_workspace_obj = workspace\n\n # items_to_include filtering inside get_items_to_publish()\n items = publisher.get_items_to_publish()\n assert \"lh_bronze\" in items\n assert \"lh_silver\" not in items\n\n # Simulate what publish_all() now does: mark excluded items skip_publish=True\n all_items = workspace.repository_items.get(\"Lakehouse\", {})\n for item_name, item_obj in all_items.items():\n if item_name not in items:\n item_obj.skip_publish = True\n\n # The excluded lakehouse must be marked as skip_publish\n assert lh_silver.skip_publish is True\n # The included lakehouse must NOT be marked as skip_publish\n assert lh_bronze.skip_publish is False\n\n\n@pytest.mark.usefixtures(\"shortcut_publish_enabled\")\ndef test_lakehouses_without_guid_are_not_shortcut_published():\n \"\"\"\n Regression test for the guid guard in post_publish_all().\n\n Even if skip_publish is somehow False for a lakehouse with no guid, the\n guid guard in post_publish_all() must prevent shortcut publishing for it,\n avoiding the 'items//shortcuts' URL that returns a 400 error.\n \"\"\"\n lh_bronze = _make_item(\"lh_bronze\", guid=\"bronze-guid\")\n lh_silver = _make_item(\"lh_silver\", guid=\"\") # empty guid, skip_publish stays False\n\n workspace = MagicMock(spec=FabricWorkspace)\n workspace.repository_items = {\"Lakehouse\": {\"lh_bronze\": lh_bronze, \"lh_silver\": lh_silver}}\n workspace.items_to_include = [\"lh_bronze.Lakehouse\"]\n workspace.shortcut_exclude_regex = None\n\n publisher = LakehousePublisher.__new__(LakehousePublisher)\n publisher.fabric_workspace_obj = workspace\n\n with patch(\"fabric_cicd._items._lakehouse.ShortcutPublisher\") as mock_shortcut_cls:\n publisher.post_publish_all()\n\n # ShortcutPublisher should only be instantiated for lh_bronze (has guid)\n calls = mock_shortcut_cls.call_args_list\n assert len(calls) == 1\n assert calls[0][0][1] is lh_bronze\n\n\n@pytest.mark.usefixtures(\"shortcut_publish_enabled\")\ndef test_publish_all_marks_excluded_items_skip_publish():\n \"\"\"\n End-to-end regression: publish_all() must mark items excluded by items_to_include\n as skip_publish=True before post_publish_all() runs, so that shortcut publishing\n is never attempted for lakehouses with empty guids.\n \"\"\"\n lh_bronze = _make_item(\"lh_bronze\", guid=\"bronze-guid\")\n lh_silver = _make_item(\"lh_silver\", guid=\"\") # not deployed in this environment\n\n workspace = MagicMock(spec=FabricWorkspace)\n workspace.repository_items = {\"Lakehouse\": {\"lh_bronze\": lh_bronze, \"lh_silver\": lh_silver}}\n workspace.items_to_include = [\"lh_bronze.Lakehouse\"]\n workspace.shortcut_exclude_regex = None\n\n publisher = LakehousePublisher.__new__(LakehousePublisher)\n publisher.fabric_workspace_obj = workspace\n publisher.item_type = \"Lakehouse\"\n\n # Track which items get shortcut-published\n shortcut_published_guids = []\n\n def fake_shortcut_publish_all(self_inner):\n shortcut_published_guids.append(self_inner.item_obj.guid)\n\n with (\n patch.object(LakehousePublisher, \"pre_publish_all\", return_value=None),\n patch.object(LakehousePublisher, \"publish_one\", return_value=None),\n patch(\"fabric_cicd._items._lakehouse.ShortcutPublisher.publish_all\", fake_shortcut_publish_all),\n ):\n publisher.publish_all()\n\n # lh_silver must be marked as skip_publish after publish_all()\n assert lh_silver.skip_publish is True\n # lh_bronze was in items_to_include so must NOT be skip_publish\n assert lh_bronze.skip_publish is False\n\n # Shortcuts must only be published for lh_bronze (has guid); never for lh_silver\n assert shortcut_published_guids == [\"bronze-guid\"]\n" }, { "path": "tests/test_subfolders.py", "content": "# Copyright (c) Microsoft Corporation.\n# Licensed under the MIT License.\n\n\"\"\"Test subfolder creation and modification in the fabric workspace.\"\"\"\n\nimport json\nimport os\nimport re\nfrom unittest.mock import MagicMock, patch\n\nimport pytest\nfrom fixtures.credentials import DummyTokenCredential\n\nfrom fabric_cicd.fabric_workspace import FabricWorkspace\n\n\n@pytest.fixture\ndef mock_endpoint():\n \"\"\"Mock FabricEndpoint to avoid real API calls.\"\"\"\n mock = MagicMock()\n mock.invoke.return_value = {\"body\": {\"value\": []}, \"header\": {}}\n return mock\n\n\n@pytest.fixture\ndef temp_workspace_dir(tmp_path):\n \"\"\"Create a temporary directory structure for testing.\"\"\"\n # Use pytest's tmp_path for better isolation\n return tmp_path\n\n\n@pytest.fixture\ndef valid_workspace_id():\n \"\"\"Return a valid workspace ID in GUID format.\"\"\"\n return \"12345678-1234-5678-abcd-1234567890ab\"\n\n\ndef create_platform_file(item_path, item_type=\"Notebook\", item_name=\"Test Item\"):\n \"\"\"Create a .platform file for an item.\"\"\"\n platform_file_path = item_path / \".platform\"\n item_path.mkdir(parents=True, exist_ok=True)\n\n metadata_content = {\n \"metadata\": {\n \"type\": item_type,\n \"displayName\": item_name,\n \"description\": f\"Test {item_type}\",\n },\n \"config\": {\"logicalId\": f\"test-logical-id-{item_name}\"},\n }\n\n with platform_file_path.open(\"w\", encoding=\"utf-8\") as f:\n json.dump(metadata_content, f, ensure_ascii=False)\n\n # Create a dummy content file\n with (item_path / \"dummy.txt\").open(\"w\", encoding=\"utf-8\") as f:\n f.write(\"Dummy file\")\n\n return metadata_content\n\n\n@pytest.fixture\ndef repository_with_subfolders(tmp_path):\n \"\"\"Create a repository with subfolders for testing - isolated per test.\"\"\"\n # Create root level items\n create_platform_file(tmp_path / \"RootNotebook.Notebook\", item_type=\"Notebook\", item_name=\"Root Notebook\")\n create_platform_file(tmp_path / \"RootPipeline.DataPipeline\", item_type=\"DataPipeline\", item_name=\"Root Pipeline\")\n\n # Create first level subfolders with items\n create_platform_file(\n tmp_path / \"Folder1\" / \"Folder1Notebook.Notebook\", item_type=\"Notebook\", item_name=\"Folder1 Notebook\"\n )\n create_platform_file(\n tmp_path / \"Folder2\" / \"Folder2Pipeline.DataPipeline\", item_type=\"DataPipeline\", item_name=\"Folder2 Pipeline\"\n )\n\n # Create second level subfolders with items\n create_platform_file(\n tmp_path / \"Folder1\" / \"Subfolder1\" / \"Subfolder1Notebook.Notebook\",\n item_type=\"Notebook\",\n item_name=\"Subfolder1 Notebook\",\n )\n create_platform_file(\n tmp_path / \"Folder2\" / \"Subfolder2\" / \"Subfolder2Pipeline.DataPipeline\",\n item_type=\"DataPipeline\",\n item_name=\"Subfolder2 Pipeline\",\n )\n\n # Create empty folder (should not be included in repository_folders)\n (tmp_path / \"EmptyFolder\").mkdir(parents=True, exist_ok=True)\n\n # Create a folder with only empty subfolders (should not be included)\n (tmp_path / \"FolderWithEmptySubfolders\" / \"EmptySubfolder\").mkdir(parents=True, exist_ok=True)\n\n return tmp_path\n\n\n@pytest.fixture\ndef patched_fabric_workspace(mock_endpoint):\n \"\"\"Return a factory function to create a patched FabricWorkspace.\"\"\"\n\n def _create_workspace(workspace_id, repository_directory, item_type_in_scope, **kwargs):\n fabric_endpoint_patch = patch(\"fabric_cicd.fabric_workspace.FabricEndpoint\", return_value=mock_endpoint)\n parameter_patch = patch.object(\n FabricWorkspace, \"_refresh_parameter_file\", new=lambda self: setattr(self, \"environment_parameter\", {})\n )\n\n with fabric_endpoint_patch, parameter_patch:\n return FabricWorkspace(\n workspace_id=workspace_id,\n repository_directory=repository_directory,\n item_type_in_scope=item_type_in_scope,\n token_credential=DummyTokenCredential(),\n **kwargs,\n )\n\n return _create_workspace\n\n\ndef test_refresh_repository_folders(repository_with_subfolders, patched_fabric_workspace, valid_workspace_id):\n \"\"\"Test the _refresh_repository_folders method.\"\"\"\n workspace = patched_fabric_workspace(\n workspace_id=valid_workspace_id,\n repository_directory=str(repository_with_subfolders),\n item_type_in_scope=[\"Notebook\", \"DataPipeline\"],\n )\n\n # Call the method under test\n workspace._refresh_repository_folders()\n\n # Verify folders are correctly identified\n assert \"/Folder1\" in workspace.repository_folders\n assert \"/Folder2\" in workspace.repository_folders\n assert \"/Folder1/Subfolder1\" in workspace.repository_folders\n assert \"/Folder2/Subfolder2\" in workspace.repository_folders\n\n # Verify empty folders are not included\n assert \"/EmptyFolder\" not in workspace.repository_folders\n assert \"/FolderWithEmptySubfolders\" not in workspace.repository_folders\n assert \"/FolderWithEmptySubfolders/EmptySubfolder\" not in workspace.repository_folders\n\n # Verify all folder IDs are initially empty strings\n for folder_id in workspace.repository_folders.values():\n assert folder_id == \"\"\n\n\ndef test_publish_folders_hierarchy(repository_with_subfolders, patched_fabric_workspace, valid_workspace_id):\n \"\"\"Test that the folder hierarchy is correctly established.\"\"\"\n workspace = patched_fabric_workspace(\n workspace_id=valid_workspace_id,\n repository_directory=str(repository_with_subfolders),\n item_type_in_scope=[\"Notebook\", \"DataPipeline\"],\n )\n\n # Call the method under test\n workspace._refresh_repository_folders()\n\n # Verify folders are correctly identified\n assert \"/Folder1\" in workspace.repository_folders\n assert \"/Folder2\" in workspace.repository_folders\n assert \"/Folder1/Subfolder1\" in workspace.repository_folders\n assert \"/Folder2/Subfolder2\" in workspace.repository_folders\n\n # Sort folders by path depth\n sorted_folders = sorted(workspace.repository_folders.keys(), key=lambda path: path.count(\"/\"))\n\n # Check parent-child relationships in the sorted folder list\n # Parents should always come before their children\n assert sorted_folders.index(\"/Folder1\") < sorted_folders.index(\"/Folder1/Subfolder1\")\n assert sorted_folders.index(\"/Folder2\") < sorted_folders.index(\"/Folder2/Subfolder2\")\n\n # Verify direct parent-child relationships by checking path structure\n for folder_path in workspace.repository_folders:\n if folder_path.count(\"/\") > 1: # It's a subfolder\n parent_path = \"/\".join(folder_path.split(\"/\")[:-1])\n assert parent_path in workspace.repository_folders, (\n f\"Parent folder {parent_path} not found for {folder_path}\"\n )\n\n\ndef test_folder_hierarchy_preservation(repository_with_subfolders, patched_fabric_workspace, valid_workspace_id):\n \"\"\"Test that the folder hierarchy is preserved when reusing existing folders.\"\"\"\n workspace = patched_fabric_workspace(\n workspace_id=valid_workspace_id,\n repository_directory=str(repository_with_subfolders),\n item_type_in_scope=[\"Notebook\", \"DataPipeline\"],\n )\n\n # Mock the deployed folders API to return existing folders\n folder1_id = \"folder1-id-12345\"\n folder2_id = \"folder2-id-67890\"\n\n def mock_invoke_side_effect(method, url, **_kwargs):\n if method == \"GET\" and url.endswith(\"/folders\"):\n # Mock API response for existing folders\n return {\n \"body\": {\n \"value\": [\n {\"id\": folder1_id, \"displayName\": \"Folder1\", \"parentFolderId\": None},\n {\"id\": folder2_id, \"displayName\": \"Folder2\", \"parentFolderId\": None},\n ]\n },\n \"header\": {},\n }\n\n return {\"body\": {\"value\": []}, \"header\": {}}\n\n workspace.endpoint.invoke.side_effect = mock_invoke_side_effect\n\n # Call methods in the intended order\n workspace._refresh_repository_folders()\n workspace._refresh_deployed_folders()\n\n # Capture initial repository folders\n initial_folders = set(workspace.repository_folders.keys())\n\n # Verify the folder hierarchy remains intact\n assert set(workspace.repository_folders.keys()) == initial_folders\n\n # Verify deployed folder IDs were detected correctly\n assert workspace.deployed_folders[\"/Folder1\"] == folder1_id\n assert workspace.deployed_folders[\"/Folder2\"] == folder2_id\n\n # Verify subfolder paths still exist in repository (even if not deployed)\n assert \"/Folder1/Subfolder1\" in workspace.repository_folders\n assert \"/Folder2/Subfolder2\" in workspace.repository_folders\n\n\ndef test_item_folder_association(repository_with_subfolders, valid_workspace_id):\n \"\"\"Test that items are correctly associated with their parent folders.\"\"\"\n # Set up mock folder IDs\n folder1_id = \"folder1-id-12345\"\n folder2_id = \"folder2-id-67890\"\n subfolder1_id = \"subfolder1-id-12345\"\n subfolder2_id = \"subfolder2-id-67890\"\n\n # Mock responses for API calls\n def mock_invoke_side_effect(method, url, **_kwargs):\n if method == \"GET\" and url.endswith(\"/items\"):\n return {\"body\": {\"value\": []}}\n\n if method == \"GET\" and url.endswith(\"/folders\"):\n # Mock API response for deployed folders\n return {\n \"body\": {\n \"value\": [\n {\"id\": folder1_id, \"displayName\": \"Folder1\", \"parentFolderId\": None},\n {\"id\": folder2_id, \"displayName\": \"Folder2\", \"parentFolderId\": None},\n {\"id\": subfolder1_id, \"displayName\": \"Subfolder1\", \"parentFolderId\": folder1_id},\n {\"id\": subfolder2_id, \"displayName\": \"Subfolder2\", \"parentFolderId\": folder2_id},\n ]\n },\n \"header\": {},\n }\n\n return {\"body\": {\"value\": []}}\n\n mock_endpoint = MagicMock()\n mock_endpoint.invoke.side_effect = mock_invoke_side_effect\n\n fabric_endpoint_patch = patch(\"fabric_cicd.fabric_workspace.FabricEndpoint\", return_value=mock_endpoint)\n parameter_patch = patch.object(\n FabricWorkspace, \"_refresh_parameter_file\", new=lambda self: setattr(self, \"environment_parameter\", {})\n )\n\n with fabric_endpoint_patch, parameter_patch:\n workspace = FabricWorkspace(\n workspace_id=valid_workspace_id,\n repository_directory=str(repository_with_subfolders),\n item_type_in_scope=[\"Notebook\", \"DataPipeline\"],\n token_credential=DummyTokenCredential(),\n )\n\n # Call methods in the intended order to populate folder structures\n workspace._refresh_repository_folders()\n workspace._refresh_deployed_folders()\n\n # Simulate the effect of _publish_folders by updating repository_folders\n # with deployed folder IDs (this normally happens in _publish_folders)\n for folder_path, folder_id in workspace.deployed_folders.items():\n if folder_path in workspace.repository_folders:\n workspace.repository_folders[folder_path] = folder_id\n\n workspace._refresh_repository_items()\n\n # Verify folder IDs are correctly assigned to items\n assert workspace.repository_items[\"Notebook\"][\"Root Notebook\"].folder_id == \"\"\n assert workspace.repository_items[\"Notebook\"][\"Folder1 Notebook\"].folder_id == folder1_id\n assert workspace.repository_items[\"Notebook\"][\"Subfolder1 Notebook\"].folder_id == subfolder1_id\n\n assert workspace.repository_items[\"DataPipeline\"][\"Root Pipeline\"].folder_id == \"\"\n assert workspace.repository_items[\"DataPipeline\"][\"Folder2 Pipeline\"].folder_id == folder2_id\n assert workspace.repository_items[\"DataPipeline\"][\"Subfolder2 Pipeline\"].folder_id == subfolder2_id\n\n\ndef test_deeply_nested_subfolders(tmp_path, patched_fabric_workspace, valid_workspace_id):\n \"\"\"Test handling of deeply nested folder structures (15+ levels deep).\"\"\"\n # Create a deeply nested folder structure\n current_path = tmp_path\n folder_names = []\n\n # Create 15 levels of nested folders (adjust depth for Windows if needed)\n depth = 15\n prefix = \"Level\"\n if os.name == \"nt\":\n depth = 8\n prefix = \"L\"\n\n for i in range(1, depth + 1):\n folder_name = f\"{prefix}{i:02d}\"\n folder_names.append(folder_name)\n current_path = current_path / folder_name\n\n # Create an item in the deepest folder\n create_platform_file(\n current_path / \"DeepNotebook.Notebook\",\n item_type=\"Notebook\",\n item_name=\"Deep Notebook\",\n )\n\n mid_level_path = tmp_path\n for i in range(min(7, depth)): # Create item at level 7\n mid_level_path = mid_level_path / folder_names[i]\n\n create_platform_file(\n mid_level_path / \"MidLevelNotebook.Notebook\",\n item_type=\"Notebook\",\n item_name=\"Mid Level Notebook\",\n )\n\n workspace = patched_fabric_workspace(\n workspace_id=valid_workspace_id,\n repository_directory=str(tmp_path),\n item_type_in_scope=[\"Notebook\"],\n )\n\n # Test that _refresh_repository_folders can handle deep nesting\n workspace._refresh_repository_folders()\n\n # Verify all folder levels were detected\n expected_deep_path = \"/\" + \"/\".join(folder_names)\n expected_mid_path = \"/\" + \"/\".join(folder_names[: min(7, depth)])\n\n assert expected_deep_path in workspace.repository_folders\n assert expected_mid_path in workspace.repository_folders\n\n # Verify folder hierarchy ordering (parents before children)\n sorted_folders = sorted(workspace.repository_folders.keys(), key=lambda path: path.count(\"/\"))\n\n for i in range(1, depth):\n current_level_path = \"/\" + \"/\".join(folder_names[:i])\n next_level_path = \"/\" + \"/\".join(folder_names[: i + 1])\n if current_level_path in workspace.repository_folders and next_level_path in workspace.repository_folders:\n assert sorted_folders.index(current_level_path) < sorted_folders.index(next_level_path)\n\n # Verify no stack overflow or performance issues by checking reasonable execution time\n import time\n\n start_time = time.time()\n workspace._refresh_repository_folders()\n execution_time = time.time() - start_time\n\n # Should complete in reasonable time (< 1 second for 15 levels)\n assert execution_time < 1.0, f\"Deep folder processing took too long: {execution_time:.2f}s\"\n\n\ndef test_folder_rename_operations(tmp_path, patched_fabric_workspace, valid_workspace_id):\n \"\"\"Test folder rename operations and verify child items and subfolders are updated correctly.\"\"\"\n # Create initial folder structure in isolated tmp_path\n original_folder = tmp_path / \"OriginalFolder\"\n original_subfolder = original_folder / \"OriginalSubfolder\"\n\n # Create items in original folders\n create_platform_file(original_folder / \"ParentNotebook.Notebook\", item_type=\"Notebook\", item_name=\"Parent Notebook\")\n\n create_platform_file(\n original_subfolder / \"ChildNotebook.Notebook\", item_type=\"Notebook\", item_name=\"Child Notebook\"\n )\n\n workspace = patched_fabric_workspace(\n workspace_id=valid_workspace_id,\n repository_directory=str(tmp_path),\n item_type_in_scope=[\"Notebook\"],\n )\n\n # Initial state\n workspace._refresh_repository_folders()\n workspace._refresh_repository_items()\n\n assert \"/OriginalFolder\" in workspace.repository_folders\n assert \"/OriginalFolder/OriginalSubfolder\" in workspace.repository_folders\n\n # Create a separate workspace instance for testing renamed structure\n # to avoid contaminating the original workspace state\n renamed_tmp_path = tmp_path.parent / \"renamed_workspace\"\n renamed_tmp_path.mkdir()\n\n # Create the renamed folder structure in the new location\n renamed_folder = renamed_tmp_path / \"RenamedFolder\"\n renamed_subfolder = renamed_folder / \"RenamedSubfolder\"\n\n # Create items in renamed folders\n create_platform_file(renamed_folder / \"ParentNotebook.Notebook\", item_type=\"Notebook\", item_name=\"Parent Notebook\")\n\n create_platform_file(renamed_subfolder / \"ChildNotebook.Notebook\", item_type=\"Notebook\", item_name=\"Child Notebook\")\n\n # Create new workspace instance for renamed structure\n renamed_workspace = patched_fabric_workspace(\n workspace_id=valid_workspace_id,\n repository_directory=str(renamed_tmp_path),\n item_type_in_scope=[\"Notebook\"],\n )\n\n # Refresh after \"rename\" (using the new workspace)\n renamed_workspace._refresh_repository_folders()\n renamed_workspace._refresh_repository_items()\n\n # Verify old folder paths are no longer present in renamed workspace\n assert \"/OriginalFolder\" not in renamed_workspace.repository_folders\n assert \"/OriginalFolder/OriginalSubfolder\" not in renamed_workspace.repository_folders\n\n # Verify new folder paths are detected in renamed workspace\n assert \"/RenamedFolder\" in renamed_workspace.repository_folders\n assert \"/RenamedFolder/RenamedSubfolder\" in renamed_workspace.repository_folders\n\n # Verify items are detected in new locations\n assert \"Parent Notebook\" in renamed_workspace.repository_items[\"Notebook\"]\n assert \"Child Notebook\" in renamed_workspace.repository_items[\"Notebook\"]\n\n # Verify folder hierarchy is maintained\n sorted_folders = sorted(renamed_workspace.repository_folders.keys(), key=lambda path: path.count(\"/\"))\n assert sorted_folders.index(\"/RenamedFolder\") < sorted_folders.index(\"/RenamedFolder/RenamedSubfolder\")\n\n\ndef test_special_character_handling(tmp_path, patched_fabric_workspace, valid_workspace_id):\n \"\"\"Test handling of special characters in folder names.\"\"\"\n test_cases = [\n # Valid cases - should be accepted\n (\"ValidFolder\", True, \"Basic valid folder name\"),\n (\"Folder_With_Underscores\", True, \"Underscores should be valid\"),\n (\"Folder-With-Hyphens\", True, \"Hyphens should be valid\"),\n (\"Folder With Spaces\", True, \"Spaces should be valid\"),\n (\"FolderWithUnicode_测试\", True, \"Unicode characters should be valid\"),\n (\"FolderWith123Numbers\", True, \"Numbers should be valid\"),\n (\" SpacesAroundName \", True, \"Leading/trailing spaces should be handled\"),\n # Invalid cases - should be rejected by regex\n (\"Folder*WithAsterisk\", False, \"Asterisk should be invalid\"),\n (\"Folder#WithHash\", False, \"Hash should be invalid\"),\n (\"FolderWithBracket\", False, \"Angle bracket should be invalid\"),\n (\"Folder:WithColon\", False, \"Colon should be invalid\"),\n ('Folder\"WithQuote', False, \"Quote should be invalid\"),\n (\"Folder|WithPipe\", False, \"Pipe should be invalid\"),\n (\"Folder?WithQuestion\", False, \"Question mark should be invalid\"),\n (\"Folder\\\\WithBackslash\", False, \"Backslash should be invalid\"),\n (\"Folder/WithSlash\", False, \"Forward slash should be invalid\"),\n (\"Folder{WithBrace\", False, \"Curly brace should be invalid\"),\n (\"Folder}WithBrace\", False, \"Curly brace should be invalid\"),\n (\"Folder~WithTilde\", False, \"Tilde should be invalid\"),\n (\"Folder.WithDot\", False, \"Dot should be invalid\"),\n (\"Folder%WithPercent\", False, \"Percent should be invalid\"),\n (\"Folder&WithAmpersand\", False, \"Ampersand should be invalid\"),\n ]\n\n from fabric_cicd import constants\n\n # Test regex validation for each case\n for folder_name, should_be_valid, description in test_cases:\n has_invalid_chars = bool(re.search(constants.INVALID_FOLDER_CHAR_REGEX, folder_name))\n\n if should_be_valid:\n assert not has_invalid_chars, f\"{description}: '{folder_name}' should be valid but was rejected\"\n else:\n assert has_invalid_chars, f\"{description}: '{folder_name}' should be invalid but was accepted\"\n\n # Test actual folder creation with some valid cases\n valid_folders = [\"ValidFolder\", \"Folder_With_Underscores\", \"Folder-With-Hyphens\", \"FolderWithUnicode_测试\"]\n\n for folder_name in valid_folders:\n folder_path = tmp_path / folder_name\n create_platform_file(\n folder_path / \"TestNotebook.Notebook\", item_type=\"Notebook\", item_name=f\"Test {folder_name}\"\n )\n\n workspace = patched_fabric_workspace(\n workspace_id=valid_workspace_id,\n repository_directory=str(tmp_path),\n item_type_in_scope=[\"Notebook\"],\n )\n\n workspace._refresh_repository_folders()\n\n # Verify valid folders were detected\n for folder_name in valid_folders:\n expected_path = f\"/{folder_name}\"\n assert expected_path in workspace.repository_folders, f\"Valid folder '{folder_name}' was not detected\"\n\n\ndef test_parent_folder_with_only_subfolder_containing_items(tmp_path, patched_fabric_workspace, valid_workspace_id):\n \"\"\"Test scenario where parent folder contains only a subfolder with items (no direct items in parent).\"\"\"\n # Create parent folder with NO direct items\n parent_folder = tmp_path / \"ParentWithNoDirectItems\"\n parent_folder.mkdir(parents=True, exist_ok=True)\n\n # Create subfolder with items (parent has no direct items)\n create_platform_file(\n parent_folder / \"SubfolderWithItems\" / \"SubfolderNotebook.Notebook\",\n item_type=\"Notebook\",\n item_name=\"Subfolder Notebook\",\n )\n\n # Also create a more complex scenario with nested empty parents\n deeply_nested_parent = tmp_path / \"Level1EmptyParent\" / \"Level2EmptyParent\"\n deeply_nested_parent.mkdir(parents=True, exist_ok=True)\n\n create_platform_file(\n deeply_nested_parent / \"FinalSubfolder\" / \"DeepNestedNotebook.Notebook\",\n item_type=\"Notebook\",\n item_name=\"Deep Nested Notebook\",\n )\n\n workspace = patched_fabric_workspace(\n workspace_id=valid_workspace_id,\n repository_directory=str(tmp_path),\n item_type_in_scope=[\"Notebook\"],\n )\n\n workspace._refresh_repository_folders()\n workspace._refresh_repository_items()\n\n # Verify parent folders are detected even though they have no direct items\n assert \"/ParentWithNoDirectItems\" in workspace.repository_folders\n assert \"/ParentWithNoDirectItems/SubfolderWithItems\" in workspace.repository_folders\n\n # Verify deeply nested scenario\n assert \"/Level1EmptyParent\" in workspace.repository_folders\n assert \"/Level1EmptyParent/Level2EmptyParent\" in workspace.repository_folders\n assert \"/Level1EmptyParent/Level2EmptyParent/FinalSubfolder\" in workspace.repository_folders\n\n # Verify folder hierarchy is maintained (parents before children)\n sorted_folders = sorted(workspace.repository_folders.keys(), key=lambda path: path.count(\"/\"))\n\n parent_idx = sorted_folders.index(\"/ParentWithNoDirectItems\")\n subfolder_idx = sorted_folders.index(\"/ParentWithNoDirectItems/SubfolderWithItems\")\n assert parent_idx < subfolder_idx, \"Parent folder should come before subfolder\"\n\n # Verify items are correctly associated with their folders\n assert \"Subfolder Notebook\" in workspace.repository_items[\"Notebook\"]\n assert \"Deep Nested Notebook\" in workspace.repository_items[\"Notebook\"]\n\n # Verify that parent folders have empty folder IDs (since they have no direct deployment)\n # but their subfolders would get proper folder IDs when deployed\n assert workspace.repository_folders[\"/ParentWithNoDirectItems\"] == \"\"\n assert workspace.repository_folders[\"/Level1EmptyParent\"] == \"\"\n assert workspace.repository_folders[\"/Level1EmptyParent/Level2EmptyParent\"] == \"\"\n\n\ndef test_large_number_of_folders_and_items(tmp_path, patched_fabric_workspace, valid_workspace_id):\n \"\"\"Test performance and scalability with a large number of folders and items.\"\"\"\n import time\n\n # Create a large number of folders and items (100 folders with multiple items each)\n num_folders = 100\n items_per_folder = 3\n\n # Create folders at multiple levels\n for i in range(num_folders):\n if i < 50:\n # First 50 folders at root level\n folder_path = tmp_path / f\"Folder{i:03d}\"\n else:\n # Next 50 folders nested under first 25 folders\n parent_idx = (i - 50) % 25\n folder_path = tmp_path / f\"Folder{parent_idx:03d}\" / f\"Subfolder{i:03d}\"\n\n # Create multiple items in each folder\n for j in range(items_per_folder):\n create_platform_file(\n folder_path / f\"Item{j:02d}.Notebook\", item_type=\"Notebook\", item_name=f\"Item {j:02d} in Folder {i:03d}\"\n )\n\n workspace = patched_fabric_workspace(\n workspace_id=valid_workspace_id,\n repository_directory=str(tmp_path),\n item_type_in_scope=[\"Notebook\"],\n )\n\n # Test _refresh_repository_folders performance\n start_time = time.time()\n workspace._refresh_repository_folders()\n folders_time = time.time() - start_time\n\n # Verify we detected a reasonable number of folders\n assert len(workspace.repository_folders) >= 50, \"Should detect at least 50 folders\"\n assert len(workspace.repository_folders) <= 125, \"Should not detect more than 125 folders\"\n\n # Test _refresh_repository_items performance\n start_time = time.time()\n workspace._refresh_repository_items()\n items_time = time.time() - start_time\n\n # Verify we detected the expected number of items\n expected_items = num_folders * items_per_folder\n assert len(workspace.repository_items[\"Notebook\"]) == expected_items\n\n # Performance assertions - should complete in reasonable time\n assert folders_time < 15.0, f\"Folder detection took too long: {folders_time:.2f}s\"\n assert items_time < 30.0, f\"Item detection took too long: {items_time:.2f}s\"\n\n # Memory usage test - verify folder hierarchy is correct\n # Check that parent-child relationships are maintained even with large numbers\n nested_folders = [path for path in workspace.repository_folders if path.count(\"/\") > 1]\n\n for folder_path in nested_folders:\n parent_path = \"/\".join(folder_path.split(\"/\")[:-1])\n assert parent_path in workspace.repository_folders, f\"Parent {parent_path} not found for {folder_path}\"\n\n # Test that folder sorting still works correctly with large numbers\n sorted_folders = sorted(workspace.repository_folders.keys(), key=lambda path: path.count(\"/\"))\n\n # Verify sorting is correct - all level 1 folders should come before level 2 folders\n level_1_folders = [f for f in sorted_folders if f.count(\"/\") == 1]\n level_2_folders = [f for f in sorted_folders if f.count(\"/\") == 2]\n\n if level_1_folders and level_2_folders:\n last_level_1_index = max(sorted_folders.index(f) for f in level_1_folders)\n first_level_2_index = min(sorted_folders.index(f) for f in level_2_folders)\n assert last_level_1_index < first_level_2_index, \"Folder sorting is incorrect with large numbers\"\n" }, { "path": "tests/test_validate_env_vars.py", "content": "# Copyright (c) Microsoft Corporation.\n# Licensed under the MIT License.\n\nimport os\n\nimport pytest\n\nfrom fabric_cicd._common._exceptions import InputError\nfrom fabric_cicd._common._validate_env_vars import (\n _VALID_HOSTNAME_REGEX,\n validate_api_url,\n validate_env_var_api_url,\n)\n\n\nclass TestValidHostnameRegex:\n \"\"\"Tests for the VALID_HOSTNAME_REGEX pattern.\"\"\"\n\n @pytest.mark.parametrize(\n \"hostname\",\n [\n \"api.fabric.microsoft.com\",\n \"api.powerbi.com\",\n \"myapi.fabric.microsoft.com\",\n \"myapi.powerbi.com\",\n \"someapi.fabric.microsoft.com\",\n \"someapi.powerbi.com\",\n \"some.api.fabric.microsoft.com\",\n \"some.api.powerbi.com\",\n \"my-org.api.fabric.microsoft.com\",\n \"a.b.api.fabric.microsoft.com\",\n \"abcdef01234567890abcdef012345678.z01.w.api.fabric.microsoft.com\",\n \"abcdef01234567890abcdef012345678.z42.w.api.powerbi.com\",\n # Hyphen and underscore in labels\n \"staging-api.fabric.microsoft.com\",\n \"staging-api.powerbi.com\",\n \"my_org.api.fabric.microsoft.com\",\n # Deeply nested subdomains\n \"a.b.c.d.api.fabric.microsoft.com\",\n # Numeric subdomain label\n \"123.api.powerbi.com\",\n # Case insensitive\n \"API.fabric.microsoft.com\",\n \"api.FABRIC.microsoft.com\",\n \"api.fabric.MICROSOFT.com\",\n \"Api.PowerBI.Com\",\n ],\n )\n def test_valid_hostnames(self, hostname):\n assert _VALID_HOSTNAME_REGEX.match(hostname), f\"Expected '{hostname}' to be valid\"\n\n @pytest.mark.parametrize(\n \"hostname\",\n [\n \"fabric.microsoft.com\",\n \"powerbi.com\",\n \"contoso.com\",\n \"api.fabric.microsoft.com.contoso.com\",\n \"api.powerbi.com.contoso.com\",\n \"\",\n \"https://api.fabric.microsoft.com\",\n \"api.fabric.microsoft.com/path\",\n \"random.hostname.com\",\n # Label doesn't end with 'api'\n \"dfs.fabric.microsoft.com\",\n \"management.fabric.microsoft.com\",\n \"apix.fabric.microsoft.com\",\n \"my-apix.fabric.microsoft.com\",\n # Trailing dot\n \"api.fabric.microsoft.com.\",\n # Leading dot\n \".api.fabric.microsoft.com\",\n # Domain suffix spoofing\n \"api.fabric.microsoft.com.br\",\n \"api.fabric.microsoft.community\",\n \"api.notfabric.microsoft.com\",\n \"api.fakepowerbi.com\",\n ],\n )\n def test_invalid_hostnames(self, hostname):\n assert not _VALID_HOSTNAME_REGEX.match(hostname), f\"Expected '{hostname}' to be invalid\"\n\n\nclass TestValidateApiUrl:\n \"\"\"Tests for the standalone validate_api_url function.\"\"\"\n\n def test_accepts_valid_fabric_url(self):\n result = validate_api_url(\"https://api.fabric.microsoft.com\", \"test_label\")\n assert result == \"https://api.fabric.microsoft.com\"\n\n def test_accepts_valid_powerbi_url(self):\n result = validate_api_url(\"https://api.powerbi.com\", \"test_label\")\n assert result == \"https://api.powerbi.com\"\n\n def test_strips_trailing_slash(self):\n result = validate_api_url(\"https://api.fabric.microsoft.com/\", \"test_label\")\n assert result == \"https://api.fabric.microsoft.com\"\n\n def test_label_appears_in_error_message(self):\n with pytest.raises(InputError, match=\"my_config_key\"):\n validate_api_url(\"http://api.fabric.microsoft.com\", \"my_config_key\")\n\n def test_rejects_empty_string(self):\n with pytest.raises(InputError, match=\"must resolve to a non-empty string\"):\n validate_api_url(\"\", \"test_label\")\n\n def test_rejects_whitespace_only(self):\n with pytest.raises(InputError, match=\"must resolve to a non-empty string\"):\n validate_api_url(\" \", \"test_label\")\n\n def test_rejects_http_scheme(self):\n with pytest.raises(InputError, match=\"HTTPS scheme\"):\n validate_api_url(\"http://api.fabric.microsoft.com\", \"test_label\")\n\n def test_rejects_invalid_hostname(self):\n with pytest.raises(InputError, match=\"invalid hostname\"):\n validate_api_url(\"https://evil.com\", \"test_label\")\n\n def test_rejects_path_components(self):\n with pytest.raises(InputError, match=\"root URL without path components\"):\n validate_api_url(\"https://api.fabric.microsoft.com/v1/workspaces\", \"test_label\")\n\n def test_accepts_private_link_url(self):\n url = \"https://abcdef01234567890abcdef012345678.z01.w.api.fabric.microsoft.com\"\n result = validate_api_url(url, \"test_label\")\n assert result == url\n\n\nclass TestValidateApiUrlHostname:\n \"\"\"Tests for the validate_env_var_api_url function (env var wrapper).\"\"\"\n\n def test_returns_default_when_env_not_set(self):\n env_var = \"TEST_HOSTNAME_NOT_SET_12345\"\n assert env_var not in os.environ\n result = validate_env_var_api_url(env_var, \"https://api.fabric.microsoft.com\")\n assert result == \"https://api.fabric.microsoft.com\"\n\n def test_returns_default_powerbi_url_when_env_not_set(self):\n env_var = \"TEST_HOSTNAME_NOT_SET_12345\"\n assert env_var not in os.environ\n result = validate_env_var_api_url(env_var, \"https://api.powerbi.com\")\n assert result == \"https://api.powerbi.com\"\n\n def test_returns_env_value_when_set(self, monkeypatch):\n monkeypatch.setenv(\"TEST_HOSTNAME_VAR\", \"https://api.powerbi.com\")\n result = validate_env_var_api_url(\"TEST_HOSTNAME_VAR\", \"https://api.fabric.microsoft.com\")\n assert result == \"https://api.powerbi.com\"\n\n def test_rejects_path_components(self, monkeypatch):\n monkeypatch.setenv(\"TEST_HOSTNAME_VAR\", \"https://api.fabric.microsoft.com/v1/workspaces\")\n with pytest.raises(InputError, match=\"root URL without path components\"):\n validate_env_var_api_url(\"TEST_HOSTNAME_VAR\", \"https://api.fabric.microsoft.com\")\n\n def test_raises_on_invalid_hostname(self, monkeypatch):\n monkeypatch.setenv(\"TEST_HOSTNAME_VAR\", \"https://evil.com\")\n with pytest.raises(InputError, match=\"invalid hostname\"):\n validate_env_var_api_url(\"TEST_HOSTNAME_VAR\", \"https://api.fabric.microsoft.com\")\n\n def test_raises_on_empty_hostname(self, monkeypatch):\n monkeypatch.setenv(\"TEST_HOSTNAME_VAR\", \"\")\n with pytest.raises(InputError, match=\"must resolve to a non-empty string\"):\n validate_env_var_api_url(\"TEST_HOSTNAME_VAR\", \"https://api.fabric.microsoft.com\")\n\n def test_prefixed_hostname_accepted(self, monkeypatch):\n monkeypatch.setenv(\"TEST_HOSTNAME_VAR\", \"https://myapi.fabric.microsoft.com\")\n result = validate_env_var_api_url(\"TEST_HOSTNAME_VAR\", \"https://api.fabric.microsoft.com\")\n assert result == \"https://myapi.fabric.microsoft.com\"\n\n def test_workspace_id_pattern_accepted(self, monkeypatch):\n url = \"https://abcdef01234567890abcdef012345678.z01.w.api.fabric.microsoft.com\"\n monkeypatch.setenv(\"TEST_HOSTNAME_VAR\", url)\n result = validate_env_var_api_url(\"TEST_HOSTNAME_VAR\", \"https://api.fabric.microsoft.com\")\n assert result == url\n\n def test_dotted_prefix_hostname_accepted(self, monkeypatch):\n monkeypatch.setenv(\"TEST_HOSTNAME_VAR\", \"https://some.api.fabric.microsoft.com\")\n result = validate_env_var_api_url(\"TEST_HOSTNAME_VAR\", \"https://api.fabric.microsoft.com\")\n assert result == \"https://some.api.fabric.microsoft.com\"\n\n def test_hostname_without_scheme_rejected(self, monkeypatch):\n monkeypatch.setenv(\"TEST_HOSTNAME_VAR\", \"api.fabric.microsoft.com\")\n with pytest.raises(InputError, match=\"Invalid or missing scheme\"):\n validate_env_var_api_url(\"TEST_HOSTNAME_VAR\", \"https://api.fabric.microsoft.com\")\n\n def test_rejects_http_scheme(self, monkeypatch):\n monkeypatch.setenv(\"TEST_HOSTNAME_VAR\", \"http://api.fabric.microsoft.com\")\n with pytest.raises(InputError, match=\"Invalid or missing scheme\"):\n validate_env_var_api_url(\"TEST_HOSTNAME_VAR\", \"https://api.fabric.microsoft.com\")\n\n def test_rejects_ftp_scheme(self, monkeypatch):\n monkeypatch.setenv(\"TEST_HOSTNAME_VAR\", \"ftp://api.fabric.microsoft.com\")\n with pytest.raises(InputError, match=\"Invalid or missing scheme\"):\n validate_env_var_api_url(\"TEST_HOSTNAME_VAR\", \"https://api.fabric.microsoft.com\")\n\n def test_raises_on_whitespace_only(self, monkeypatch):\n monkeypatch.setenv(\"TEST_HOSTNAME_VAR\", \" \")\n with pytest.raises(InputError, match=\"must resolve to a non-empty string\"):\n validate_env_var_api_url(\"TEST_HOSTNAME_VAR\", \"https://api.fabric.microsoft.com\")\n\n def test_strips_trailing_slash(self, monkeypatch):\n monkeypatch.setenv(\"TEST_HOSTNAME_VAR\", \"https://api.fabric.microsoft.com/\")\n result = validate_env_var_api_url(\"TEST_HOSTNAME_VAR\", \"https://api.fabric.microsoft.com\")\n assert result == \"https://api.fabric.microsoft.com\"\n\n def test_rejects_url_with_no_authority(self, monkeypatch):\n monkeypatch.setenv(\"TEST_HOSTNAME_VAR\", \"https:///\")\n with pytest.raises(InputError, match=\"invalid hostname\"):\n validate_env_var_api_url(\"TEST_HOSTNAME_VAR\", \"https://api.fabric.microsoft.com\")\n" } ]