[
{
"path": ".gitattributes",
"content": "# Auto detect text files and perform LF normalization\n* text=auto\n"
},
{
"path": ".github/FUNDING.yml",
"content": "# These are supported funding model platforms\n\ngithub: [mrpaulandrew]\n"
},
{
"path": ".github/ISSUE_TEMPLATE/bug-found.md",
"content": "---\nname: Bug Found\nabout: Tell me about the bug you've found in the procfwk please\ntitle: ''\nlabels: bug\nassignees: mrpaulandrew\n\n---\n\n**Describe the bug**\nA clear and concise description of what the bug is.\n\n**Affected services**\nWhich resource within the processing framework does this affect?\n* Data Factory/Synapse\n* SQL Database\n* Functions\n* All of them\n* Other\n\n**To Reproduce**\nSteps to reproduce the behavior:\n1. Go to '...'\n2. Click on '....'\n3. Scroll down to '....'\n4. See error\n\n**Expected behaviour**\nA clear and concise description of what you expected to happen.\n\n**Screenshots**\nIf applicable, add screenshots to help explain your problem.\n\n**Additional context**\nAdd any other context about the problem here.\n"
},
{
"path": ".github/ISSUE_TEMPLATE/feature-request.md",
"content": "---\nname: Feature Request\nabout: Tell me about your idea to enhance the procfwk please\ntitle: ''\nlabels: enhancement\nassignees: mrpaulandrew\n\n---\n\n**Share Your Idea**\nAll features considered.\n"
},
{
"path": ".github/ISSUE_TEMPLATE/help---support-request.md",
"content": "---\nname: Help & Support Request\nabout: Tell me about the problem or error you are facing when using the procfwk\ntitle: ''\nlabels: help wanted\nassignees: mrpaulandrew\n\n---\n\n**Describe the error**\nA clear and concise description of what the bug is.\n\n**Error message**\nAn output of the error message presented.\n\n**Affected services**\nWhich resource within the processing framework does this affect?\n* Data Factory/Synapse\n* SQL Database\n* Functions\n* All of them\n* Other\n\n**To Reproduce**\nSteps to reproduce the behavior:\n1. Go to '...'\n2. Click on '....'\n3. Scroll down to '....'\n4. See error\n\n**Screenshots**\nIf applicable, add screenshots to help explain your problem.\n"
},
{
"path": ".gitignore",
"content": "################################################################################\n# This .gitignore file was automatically created by Microsoft(R) Visual Studio.\n################################################################################\n\n## Ignore Visual Studio temporary files, build results, and\n## files generated by popular Visual Studio add-ons.\n\n## https://stackoverflow.com/questions/30868544/gitignore-wont-ignore-vs-folder-for-visual-studio-2015-rc-on-windows7-8/39520183\n\n# User-specific files\n*.suo\n*.user\n*.userosscache\n*.sln.docstates\n\n# User-specific files (MonoDevelop/Xamarin Studio)\n*.userprefs\n\n# Build results\n[Dd]ebug/\n[Dd]ebugPublic/\n[Rr]elease/\n[Rr]eleases/\nx64/\nx86/\nbld/\n[Bb]in/\n[Oo]bj/\n[Ll]og/\n\n# Visual Studio 2015 cache/options directory\n.vs/\n# Uncomment if you have tasks that create the project's static files in wwwroot\n#wwwroot/\n\n# MSTest test Results\n[Tt]est[Rr]esult*/\n[Bb]uild[Ll]og.*\n\n# NUNIT\n*.VisualState.xml\nTestResult.xml\n\n# Build Results of an ATL Project\n[Dd]ebugPS/\n[Rr]eleasePS/\ndlldata.c\n\n# DNX\nproject.lock.json\nartifacts/\n\n*_i.c\n*_p.c\n*_i.h\n*.ilk\n*.meta\n*.obj\n*.pch\n*.pdb\n*.pgc\n*.pgd\n*.rsp\n*.sbr\n*.tlb\n*.tli\n*.tlh\n*.tmp\n*.tmp_proj\n*.log\n*.vspscc\n*.vssscc\n.builds\n*.pidb\n*.svclog\n*.scc\n\n# Chutzpah Test files\n_Chutzpah*\n\n# Visual C++ cache files\nipch/\n*.aps\n*.ncb\n*.opendb\n*.opensdf\n*.sdf\n*.cachefile\n\n# Visual Studio profiler\n*.psess\n*.vsp\n*.vspx\n*.sap\n\n# TFS 2012 Local Workspace\n$tf/\n\n# Guidance Automation Toolkit\n*.gpState\n\n# ReSharper is a .NET coding add-in\n_ReSharper*/\n*.[Rr]e[Ss]harper\n*.DotSettings.user\n\n# JustCode is a .NET coding add-in\n.JustCode\n\n# TeamCity is a build add-in\n_TeamCity*\n\n# DotCover is a Code Coverage Tool\n*.dotCover\n\n# NCrunch\n_NCrunch_*\n.*crunch*.local.xml\nnCrunchTemp_*\n\n# MightyMoose\n*.mm.*\nAutoTest.Net/\n\n# Web workbench (sass)\n.sass-cache/\n\n# Installshield output folder\n[Ee]xpress/\n\n# DocProject is a documentation generator add-in\nDocProject/buildhelp/\nDocProject/Help/*.HxT\nDocProject/Help/*.HxC\nDocProject/Help/*.hhc\nDocProject/Help/*.hhk\nDocProject/Help/*.hhp\nDocProject/Help/Html2\nDocProject/Help/html\n\n# Click-Once directory\npublish/\n\n# Publish Web Output\n*.[Pp]ublish.xml\n*.azurePubxml\n# TODO: Comment the next line if you want to checkin your web deploy settings \n# but database connection strings (with potential passwords) will be unencrypted\n*.pubxml\n*.publishproj\n\n# NuGet Packages\n*.nupkg\n# The packages folder can be ignored because of Package Restore\n**/packages/*\n# except build/, which is used as an MSBuild target.\n!**/packages/build/\n# Uncomment if necessary however generally it will be regenerated when needed\n#!**/packages/repositories.config\n# NuGet v3's project.json files produces more ignoreable files\n*.nuget.props\n*.nuget.targets\n\n# Microsoft Azure Build Output\ncsx/\n*.build.csdef\n\n# Microsoft Azure Emulator\necf/\nrcf/\n\n# Windows Store app package directories and files\nAppPackages/\nBundleArtifacts/\nPackage.StoreAssociation.xml\n_pkginfo.txt\n\n# Visual Studio cache files\n# files ending in .cache can be ignored\n*.[Cc]ache\n# but keep track of directories ending in .cache\n!*.[Cc]ache/\n\n# Others\nClientBin/\n~$*\n*~\n*.dbmdl\n*.dbproj.schemaview\n*.pfx\n*.publishsettings\nnode_modules/\norleans.codegen.cs\n\n# Since there are multiple workflows, uncomment next line to ignore bower_components \n# (https://github.com/github/gitignore/pull/1529#issuecomment-104372622)\n#bower_components/\n\n# RIA/Silverlight projects\nGenerated_Code/\n\n# Backup & report files from converting an old project file\n# to a newer Visual Studio version. Backup files are not needed,\n# because we have git ;-)\n_UpgradeReport_Files/\nBackup*/\nUpgradeLog*.XML\nUpgradeLog*.htm\n\n# SQL Server files\n*.mdf\n*.ldf\n*.jfm\n\n# Business Intelligence projects\n*.rdl.data\n*.bim.layout\n*.bim_*.settings\n\n# Microsoft Fakes\nFakesAssemblies/\n\n# GhostDoc plugin setting file\n*.GhostDoc.xml\n\n# Node.js Tools for Visual Studio\n.ntvs_analysis.dat\n\n# Visual Studio 6 build log\n*.plg\n\n# Visual Studio 6 workspace options file\n*.opt\n\n# Visual Studio LightSwitch build output\n**/*.HTMLClient/GeneratedArtifacts\n**/*.DesktopClient/GeneratedArtifacts\n**/*.DesktopClient/ModelManifest.xml\n**/*.Server/GeneratedArtifacts\n**/*.Server/ModelManifest.xml\n_Pvt_Extensions\n\n# Paket dependency manager\n.paket/paket.exe\npaket-files/\n\n# FAKE - F# Make\n.fake/\n\n# JetBrains Rider\n.idea/\n*.sln.iml\n/PipelineExecutor/local.settings.json\n/Functions/local.settings.json\n/docs\n"
},
{
"path": ".vscode/extensions.json",
"content": "{\n \"recommendations\": [\n \"ms-azuretools.vscode-azurefunctions\",\n \"ms-vscode.csharp\"\n ]\n}\n"
},
{
"path": ".vscode/launch.json",
"content": "{\n \"version\": \"0.2.0\",\n \"configurations\": [\n {\n \"name\": \"Attach to .NET Functions\",\n \"type\": \"coreclr\",\n \"request\": \"attach\",\n \"processId\": \"${command:azureFunctions.pickProcess}\"\n }\n ]\n}"
},
{
"path": ".vscode/settings.json",
"content": "{\n \"azureFunctions.deploySubpath\": \"PipelineExecutor/bin/Release/netcoreapp3.1/publish\",\n \"azureFunctions.projectLanguage\": \"C#\",\n \"azureFunctions.projectRuntime\": \"~3\",\n \"debug.internalConsoleOptions\": \"neverOpen\",\n \"azureFunctions.preDeployTask\": \"publish\"\n}"
},
{
"path": ".vscode/tasks.json",
"content": "{\n \"version\": \"2.0.0\",\n \"tasks\": [\n {\n \"label\": \"clean\",\n \"command\": \"dotnet\",\n \"args\": [\n \"clean\",\n \"/property:GenerateFullPaths=true\",\n \"/consoleloggerparameters:NoSummary\"\n ],\n \"type\": \"process\",\n \"problemMatcher\": \"$msCompile\",\n \"options\": {\n \"cwd\": \"${workspaceFolder}/PipelineExecutor\"\n }\n },\n {\n \"label\": \"build\",\n \"command\": \"dotnet\",\n \"args\": [\n \"build\",\n \"/property:GenerateFullPaths=true\",\n \"/consoleloggerparameters:NoSummary\"\n ],\n \"type\": \"process\",\n \"dependsOn\": \"clean\",\n \"group\": {\n \"kind\": \"build\",\n \"isDefault\": true\n },\n \"problemMatcher\": \"$msCompile\",\n \"options\": {\n \"cwd\": \"${workspaceFolder}/PipelineExecutor\"\n }\n },\n {\n \"label\": \"clean release\",\n \"command\": \"dotnet\",\n \"args\": [\n \"clean\",\n \"--configuration\",\n \"Release\",\n \"/property:GenerateFullPaths=true\",\n \"/consoleloggerparameters:NoSummary\"\n ],\n \"type\": \"process\",\n \"problemMatcher\": \"$msCompile\",\n \"options\": {\n \"cwd\": \"${workspaceFolder}/PipelineExecutor\"\n }\n },\n {\n \"label\": \"publish\",\n \"command\": \"dotnet\",\n \"args\": [\n \"publish\",\n \"--configuration\",\n \"Release\",\n \"/property:GenerateFullPaths=true\",\n \"/consoleloggerparameters:NoSummary\"\n ],\n \"type\": \"process\",\n \"dependsOn\": \"clean release\",\n \"problemMatcher\": \"$msCompile\",\n \"options\": {\n \"cwd\": \"${workspaceFolder}/PipelineExecutor\"\n }\n },\n {\n \"type\": \"func\",\n \"dependsOn\": \"build\",\n \"options\": {\n \"cwd\": \"${workspaceFolder}/PipelineExecutor/bin/Debug/netcoreapp3.1\"\n },\n \"command\": \"host start\",\n \"isBackground\": true,\n \"problemMatcher\": \"$func-watch\"\n }\n ]\n}"
},
{
"path": "ARM Templates/Data Factory/v1.0 Export.json",
"content": "{\n \"$schema\": \"http://schema.management.azure.com/schemas/2015-01-01/deploymentTemplate.json#\",\n \"contentVersion\": \"1.0.0.0\",\n \"parameters\": {\n \"factoryName\": {\n \"type\": \"string\",\n \"metadata\": \"Data Factory name\"\n },\n \"Keys_properties_typeProperties_baseUrl\": {\n \"type\": \"string\"\n },\n \"SupportDatabase_properties_typeProperties_connectionString_secretName\": {\n \"type\": \"string\"\n }\n },\n \"variables\": {\n \"factoryId\": \"[concat('Microsoft.DataFactory/factories/', parameters('factoryName'))]\"\n },\n \"resources\": [\n {\n \"name\": \"[concat(parameters('factoryName'), '/BootStrap')]\",\n \"type\": \"Microsoft.DataFactory/factories/pipelines\",\n \"apiVersion\": \"2018-06-01\",\n \"properties\": {\n \"activities\": [\n {\n \"name\": \"Get Stages\",\n \"type\": \"Lookup\",\n \"dependsOn\": [\n {\n \"activity\": \"Create New Execution\",\n \"dependencyConditions\": [\n \"Succeeded\"\n ]\n }\n ],\n \"policy\": {\n \"timeout\": \"7.00:00:00\",\n \"retry\": 0,\n \"retryIntervalInSeconds\": 30,\n \"secureOutput\": false,\n \"secureInput\": false\n },\n \"userProperties\": [],\n \"typeProperties\": {\n \"source\": {\n \"type\": \"AzureSqlSource\",\n \"sqlReaderStoredProcedureName\": \"[[procfwk].[GetProcessStages]\",\n \"storedProcedureParameters\": {\n \"ExecutionId\": {\n \"type\": \"Guid\",\n \"value\": {\n \"value\": \"@activity('Create New Execution').output.firstRow.ExecutionId\",\n \"type\": \"Expression\"\n }\n }\n },\n \"queryTimeout\": \"02:00:00\"\n },\n \"dataset\": {\n \"referenceName\": \"GetSetMetadata\",\n \"type\": \"DatasetReference\",\n \"parameters\": {}\n },\n \"firstRowOnly\": false\n }\n },\n {\n \"name\": \"Execute Stages\",\n \"type\": \"ForEach\",\n \"dependsOn\": [\n {\n \"activity\": \"Get Stages\",\n \"dependencyConditions\": [\n \"Succeeded\"\n ]\n }\n ],\n \"userProperties\": [],\n \"typeProperties\": {\n \"items\": {\n \"value\": \"@activity('Get Stages').output.value\",\n \"type\": \"Expression\"\n },\n \"isSequential\": true,\n \"activities\": [\n {\n \"name\": \"Stage Executor\",\n \"type\": \"ExecutePipeline\",\n \"dependsOn\": [\n {\n \"activity\": \"Log Stage Start\",\n \"dependencyConditions\": [\n \"Succeeded\"\n ]\n }\n ],\n \"userProperties\": [],\n \"typeProperties\": {\n \"pipeline\": {\n \"referenceName\": \"Executor\",\n \"type\": \"PipelineReference\"\n },\n \"waitOnCompletion\": true,\n \"parameters\": {\n \"StageId\": {\n \"value\": \"@item().StageId\",\n \"type\": \"Expression\"\n },\n \"ExecutionId\": {\n \"value\": \"@activity('Create New Execution').output.firstRow.ExecutionId\",\n \"type\": \"Expression\"\n }\n }\n }\n },\n {\n \"name\": \"Log Stage Start\",\n \"type\": \"SqlServerStoredProcedure\",\n \"dependsOn\": [],\n \"policy\": {\n \"timeout\": \"7.00:00:00\",\n \"retry\": 0,\n \"retryIntervalInSeconds\": 30,\n \"secureOutput\": false,\n \"secureInput\": false\n },\n \"userProperties\": [],\n \"typeProperties\": {\n \"storedProcedureName\": \"[[procfwk].[SetLogStageStart]\",\n \"storedProcedureParameters\": {\n \"ExecutionId\": {\n \"value\": {\n \"value\": \"@activity('Create New Execution').output.firstRow.ExecutionId\",\n \"type\": \"Expression\"\n },\n \"type\": \"Guid\"\n },\n \"StageId\": {\n \"value\": {\n \"value\": \"@item().StageId\",\n \"type\": \"Expression\"\n },\n \"type\": \"Int32\"\n }\n }\n },\n \"linkedServiceName\": {\n \"referenceName\": \"SupportDatabase\",\n \"type\": \"LinkedServiceReference\"\n }\n }\n ]\n }\n },\n {\n \"name\": \"Create New Execution\",\n \"type\": \"Lookup\",\n \"dependsOn\": [],\n \"policy\": {\n \"timeout\": \"7.00:00:00\",\n \"retry\": 0,\n \"retryIntervalInSeconds\": 30,\n \"secureOutput\": false,\n \"secureInput\": false\n },\n \"userProperties\": [],\n \"typeProperties\": {\n \"source\": {\n \"type\": \"AzureSqlSource\",\n \"sqlReaderStoredProcedureName\": \"[[procfwk].[CreateNewExecution]\",\n \"queryTimeout\": \"02:00:00\"\n },\n \"dataset\": {\n \"referenceName\": \"GetSetMetadata\",\n \"type\": \"DatasetReference\",\n \"parameters\": {}\n }\n }\n },\n {\n \"name\": \"Update Execution Log\",\n \"type\": \"SqlServerStoredProcedure\",\n \"dependsOn\": [\n {\n \"activity\": \"Execute Stages\",\n \"dependencyConditions\": [\n \"Succeeded\"\n ]\n }\n ],\n \"policy\": {\n \"timeout\": \"7.00:00:00\",\n \"retry\": 0,\n \"retryIntervalInSeconds\": 30,\n \"secureOutput\": false,\n \"secureInput\": false\n },\n \"userProperties\": [],\n \"typeProperties\": {\n \"storedProcedureName\": \"[[procfwk].[UpdateExecutionLog]\"\n },\n \"linkedServiceName\": {\n \"referenceName\": \"SupportDatabase\",\n \"type\": \"LinkedServiceReference\"\n }\n }\n ],\n \"folder\": {\n \"name\": \"Framework\"\n },\n \"annotations\": []\n },\n \"dependsOn\": [\n \"[concat(variables('factoryId'), '/datasets/GetSetMetadata')]\",\n \"[concat(variables('factoryId'), '/linkedServices/SupportDatabase')]\",\n \"[concat(variables('factoryId'), '/pipelines/Executor')]\"\n ]\n },\n {\n \"name\": \"[concat(parameters('factoryName'), '/Child 1')]\",\n \"type\": \"Microsoft.DataFactory/factories/pipelines\",\n \"apiVersion\": \"2018-06-01\",\n \"properties\": {\n \"activities\": [\n {\n \"name\": \"Wait1\",\n \"type\": \"Wait\",\n \"dependsOn\": [],\n \"userProperties\": [],\n \"typeProperties\": {\n \"waitTimeInSeconds\": 1\n }\n }\n ],\n \"folder\": {\n \"name\": \"Child Pipelines\"\n },\n \"annotations\": []\n },\n \"dependsOn\": []\n },\n {\n \"name\": \"[concat(parameters('factoryName'), '/Child 2')]\",\n \"type\": \"Microsoft.DataFactory/factories/pipelines\",\n \"apiVersion\": \"2018-06-01\",\n \"properties\": {\n \"activities\": [\n {\n \"name\": \"Wait1\",\n \"type\": \"Wait\",\n \"dependsOn\": [],\n \"userProperties\": [],\n \"typeProperties\": {\n \"waitTimeInSeconds\": 1\n }\n }\n ],\n \"folder\": {\n \"name\": \"Child Pipelines\"\n },\n \"annotations\": []\n },\n \"dependsOn\": []\n },\n {\n \"name\": \"[concat(parameters('factoryName'), '/Child 3')]\",\n \"type\": \"Microsoft.DataFactory/factories/pipelines\",\n \"apiVersion\": \"2018-06-01\",\n \"properties\": {\n \"activities\": [\n {\n \"name\": \"Wait1\",\n \"type\": \"Wait\",\n \"dependsOn\": [],\n \"userProperties\": [],\n \"typeProperties\": {\n \"waitTimeInSeconds\": 1\n }\n }\n ],\n \"folder\": {\n \"name\": \"Child Pipelines\"\n },\n \"annotations\": []\n },\n \"dependsOn\": []\n },\n {\n \"name\": \"[concat(parameters('factoryName'), '/Child 4')]\",\n \"type\": \"Microsoft.DataFactory/factories/pipelines\",\n \"apiVersion\": \"2018-06-01\",\n \"properties\": {\n \"activities\": [\n {\n \"name\": \"Wait1\",\n \"type\": \"Wait\",\n \"dependsOn\": [],\n \"userProperties\": [],\n \"typeProperties\": {\n \"waitTimeInSeconds\": 1\n }\n }\n ],\n \"folder\": {\n \"name\": \"Child Pipelines\"\n },\n \"annotations\": []\n },\n \"dependsOn\": []\n },\n {\n \"name\": \"[concat(parameters('factoryName'), '/Child 5')]\",\n \"type\": \"Microsoft.DataFactory/factories/pipelines\",\n \"apiVersion\": \"2018-06-01\",\n \"properties\": {\n \"activities\": [\n {\n \"name\": \"Wait1\",\n \"type\": \"Wait\",\n \"dependsOn\": [],\n \"userProperties\": [],\n \"typeProperties\": {\n \"waitTimeInSeconds\": 1\n }\n }\n ],\n \"folder\": {\n \"name\": \"Child Pipelines\"\n },\n \"annotations\": []\n },\n \"dependsOn\": []\n },\n {\n \"name\": \"[concat(parameters('factoryName'), '/Child 6')]\",\n \"type\": \"Microsoft.DataFactory/factories/pipelines\",\n \"apiVersion\": \"2018-06-01\",\n \"properties\": {\n \"activities\": [\n {\n \"name\": \"Wait1\",\n \"type\": \"Wait\",\n \"dependsOn\": [],\n \"userProperties\": [],\n \"typeProperties\": {\n \"waitTimeInSeconds\": 1\n }\n }\n ],\n \"folder\": {\n \"name\": \"Child Pipelines\"\n },\n \"annotations\": []\n },\n \"dependsOn\": []\n },\n {\n \"name\": \"[concat(parameters('factoryName'), '/Executor')]\",\n \"type\": \"Microsoft.DataFactory/factories/pipelines\",\n \"apiVersion\": \"2018-06-01\",\n \"properties\": {\n \"activities\": [\n {\n \"name\": \"Get Pipelines\",\n \"type\": \"Lookup\",\n \"dependsOn\": [],\n \"policy\": {\n \"timeout\": \"7.00:00:00\",\n \"retry\": 0,\n \"retryIntervalInSeconds\": 30,\n \"secureOutput\": false,\n \"secureInput\": false\n },\n \"userProperties\": [],\n \"typeProperties\": {\n \"source\": {\n \"type\": \"AzureSqlSource\",\n \"sqlReaderStoredProcedureName\": \"[[procfwk].[GetPipelinesInStage]\",\n \"storedProcedureParameters\": {\n \"StageId\": {\n \"type\": \"Int32\",\n \"value\": {\n \"value\": \"@pipeline().parameters.StageId\",\n \"type\": \"Expression\"\n }\n }\n },\n \"queryTimeout\": \"02:00:00\"\n },\n \"dataset\": {\n \"referenceName\": \"GetSetMetadata\",\n \"type\": \"DatasetReference\",\n \"parameters\": {}\n },\n \"firstRowOnly\": false\n }\n },\n {\n \"name\": \"Execute Pipelines\",\n \"type\": \"ForEach\",\n \"dependsOn\": [\n {\n \"activity\": \"Get Pipelines\",\n \"dependencyConditions\": [\n \"Succeeded\"\n ]\n }\n ],\n \"userProperties\": [],\n \"typeProperties\": {\n \"items\": {\n \"value\": \"@activity('Get Pipelines').output.value\",\n \"type\": \"Expression\"\n },\n \"activities\": [\n {\n \"name\": \"Execute Pipeline\",\n \"type\": \"AzureFunctionActivity\",\n \"dependsOn\": [\n {\n \"activity\": \"Set Body\",\n \"dependencyConditions\": [\n \"Succeeded\"\n ]\n },\n {\n \"activity\": \"Log Pipeline Running\",\n \"dependencyConditions\": [\n \"Succeeded\"\n ]\n }\n ],\n \"policy\": {\n \"timeout\": \"7.00:00:00\",\n \"retry\": 0,\n \"retryIntervalInSeconds\": 30,\n \"secureOutput\": false,\n \"secureInput\": false\n },\n \"userProperties\": [],\n \"typeProperties\": {\n \"functionName\": \"ExecutePipeline\",\n \"method\": \"POST\",\n \"headers\": {},\n \"body\": {\n \"value\": \"@variables('FunctionBody')\",\n \"type\": \"Expression\"\n }\n },\n \"linkedServiceName\": {\n \"referenceName\": \"PipelineExecutor\",\n \"type\": \"LinkedServiceReference\"\n }\n },\n {\n \"name\": \"Set Body\",\n \"type\": \"SetVariable\",\n \"dependsOn\": [\n {\n \"activity\": \"Get Pipeline Params\",\n \"dependencyConditions\": [\n \"Succeeded\"\n ]\n }\n ],\n \"userProperties\": [],\n \"typeProperties\": {\n \"variableName\": \"FunctionBody\",\n \"value\": {\n \"value\": \"@concat('\\n{\\n\\t\\\"tenantId\\\": \\\"1234-1234-1234-1234-1234\\\",\\n\\t\\\"applicationId\\\": \\\"1234-1234-1234-1234-1234\\\",\\n\\t\\\"authenticationKey\\\": \\\"Passw0rd123!\\\",\\n\\t\\\"subscriptionId\\\": \\\"1234-1234-1234-1234-1234\\\",\\n\\t\\\"resourceGroup\\\": \\\"Demos\\\",\\n\\t\\\"factoryName\\\": \\\"FunFactory\\\",\\n\\t\\\"pipelineName\\\": \\\"',item().PipelineName,'\\\"',activity('Get Pipeline Params').output.firstRow.Params,'\\n}')\",\n \"type\": \"Expression\"\n }\n }\n },\n {\n \"name\": \"Get Pipeline Params\",\n \"type\": \"Lookup\",\n \"dependsOn\": [],\n \"policy\": {\n \"timeout\": \"7.00:00:00\",\n \"retry\": 0,\n \"retryIntervalInSeconds\": 30,\n \"secureOutput\": false,\n \"secureInput\": false\n },\n \"userProperties\": [],\n \"typeProperties\": {\n \"source\": {\n \"type\": \"AzureSqlSource\",\n \"sqlReaderStoredProcedureName\": \"[[procfwk].[GetPipelineParameters]\",\n \"storedProcedureParameters\": {\n \"PipelineId\": {\n \"type\": \"Int32\",\n \"value\": {\n \"value\": \"@item().PipelineId\",\n \"type\": \"Expression\"\n }\n }\n },\n \"queryTimeout\": \"02:00:00\"\n },\n \"dataset\": {\n \"referenceName\": \"GetSetMetadata\",\n \"type\": \"DatasetReference\",\n \"parameters\": {}\n }\n }\n },\n {\n \"name\": \"Log Pipeline Running\",\n \"type\": \"SqlServerStoredProcedure\",\n \"dependsOn\": [],\n \"policy\": {\n \"timeout\": \"7.00:00:00\",\n \"retry\": 0,\n \"retryIntervalInSeconds\": 30,\n \"secureOutput\": false,\n \"secureInput\": false\n },\n \"userProperties\": [],\n \"typeProperties\": {\n \"storedProcedureName\": \"[[procfwk].[SetLogPipelineRunning]\",\n \"storedProcedureParameters\": {\n \"ExecutionId\": {\n \"value\": {\n \"value\": \"@pipeline().parameters.ExecutionId\",\n \"type\": \"Expression\"\n },\n \"type\": \"Guid\"\n },\n \"PipelineId\": {\n \"value\": {\n \"value\": \"@item().PipelineId\",\n \"type\": \"Expression\"\n },\n \"type\": \"Int32\"\n },\n \"StageId\": {\n \"value\": {\n \"value\": \"@pipeline().parameters.StageId\",\n \"type\": \"Expression\"\n },\n \"type\": \"Int32\"\n }\n }\n },\n \"linkedServiceName\": {\n \"referenceName\": \"SupportDatabase\",\n \"type\": \"LinkedServiceReference\"\n }\n },\n {\n \"name\": \"Log Pipeline Success\",\n \"type\": \"SqlServerStoredProcedure\",\n \"dependsOn\": [\n {\n \"activity\": \"Execute Pipeline\",\n \"dependencyConditions\": [\n \"Succeeded\"\n ]\n }\n ],\n \"policy\": {\n \"timeout\": \"7.00:00:00\",\n \"retry\": 0,\n \"retryIntervalInSeconds\": 30,\n \"secureOutput\": false,\n \"secureInput\": false\n },\n \"userProperties\": [],\n \"typeProperties\": {\n \"storedProcedureName\": \"[[procfwk].[SetLogPipelineSuccess]\",\n \"storedProcedureParameters\": {\n \"ExecutionId\": {\n \"value\": {\n \"value\": \"@pipeline().parameters.ExecutionId\",\n \"type\": \"Expression\"\n },\n \"type\": \"Guid\"\n },\n \"PipelineId\": {\n \"value\": {\n \"value\": \"@item().PipelineId\",\n \"type\": \"Expression\"\n },\n \"type\": \"Int32\"\n },\n \"StageId\": {\n \"value\": {\n \"value\": \"@pipeline().parameters.StageId\",\n \"type\": \"Expression\"\n },\n \"type\": \"Int32\"\n }\n }\n },\n \"linkedServiceName\": {\n \"referenceName\": \"SupportDatabase\",\n \"type\": \"LinkedServiceReference\"\n }\n },\n {\n \"name\": \"Log Pipeline Failure\",\n \"type\": \"SqlServerStoredProcedure\",\n \"dependsOn\": [\n {\n \"activity\": \"Execute Pipeline\",\n \"dependencyConditions\": [\n \"Failed\"\n ]\n }\n ],\n \"policy\": {\n \"timeout\": \"7.00:00:00\",\n \"retry\": 0,\n \"retryIntervalInSeconds\": 30,\n \"secureOutput\": false,\n \"secureInput\": false\n },\n \"userProperties\": [],\n \"typeProperties\": {\n \"storedProcedureName\": \"[[procfwk].[SetLogPipelineFailed]\",\n \"storedProcedureParameters\": {\n \"ExecutionId\": {\n \"value\": {\n \"value\": \"@pipeline().parameters.ExecutionId\",\n \"type\": \"Expression\"\n },\n \"type\": \"Guid\"\n },\n \"PipelineId\": {\n \"value\": {\n \"value\": \"@item().PipelineId\",\n \"type\": \"Expression\"\n },\n \"type\": \"Int32\"\n },\n \"StageId\": {\n \"value\": {\n \"value\": \"@pipeline().parameters.StageId\",\n \"type\": \"Expression\"\n },\n \"type\": \"Int32\"\n }\n }\n },\n \"linkedServiceName\": {\n \"referenceName\": \"SupportDatabase\",\n \"type\": \"LinkedServiceReference\"\n }\n }\n ]\n }\n }\n ],\n \"parameters\": {\n \"StageId\": {\n \"type\": \"int\"\n },\n \"ExecutionId\": {\n \"type\": \"string\"\n }\n },\n \"variables\": {\n \"FunctionBody\": {\n \"type\": \"String\"\n }\n },\n \"folder\": {\n \"name\": \"Framework\"\n },\n \"annotations\": []\n },\n \"dependsOn\": [\n \"[concat(variables('factoryId'), '/datasets/GetSetMetadata')]\",\n \"[concat(variables('factoryId'), '/linkedServices/PipelineExecutor')]\",\n \"[concat(variables('factoryId'), '/linkedServices/SupportDatabase')]\"\n ]\n },\n {\n \"name\": \"[concat(parameters('factoryName'), '/Grandparent')]\",\n \"type\": \"Microsoft.DataFactory/factories/pipelines\",\n \"apiVersion\": \"2018-06-01\",\n \"properties\": {\n \"activities\": [\n {\n \"name\": \"Framework Processing\",\n \"type\": \"ExecutePipeline\",\n \"dependsOn\": [],\n \"userProperties\": [],\n \"typeProperties\": {\n \"pipeline\": {\n \"referenceName\": \"BootStrap\",\n \"type\": \"PipelineReference\"\n },\n \"waitOnCompletion\": true,\n \"parameters\": {}\n }\n }\n ],\n \"annotations\": []\n },\n \"dependsOn\": [\n \"[concat(variables('factoryId'), '/pipelines/BootStrap')]\"\n ]\n },\n {\n \"name\": \"[concat(parameters('factoryName'), '/GetSetMetadata')]\",\n \"type\": \"Microsoft.DataFactory/factories/datasets\",\n \"apiVersion\": \"2018-06-01\",\n \"properties\": {\n \"linkedServiceName\": {\n \"referenceName\": \"SupportDatabase\",\n \"type\": \"LinkedServiceReference\"\n },\n \"annotations\": [],\n \"type\": \"AzureSqlTable\",\n \"schema\": [],\n \"typeProperties\": {}\n },\n \"dependsOn\": [\n \"[concat(variables('factoryId'), '/linkedServices/SupportDatabase')]\"\n ]\n },\n {\n \"name\": \"[concat(parameters('factoryName'), '/Keys')]\",\n \"type\": \"Microsoft.DataFactory/factories/linkedServices\",\n \"apiVersion\": \"2018-06-01\",\n \"properties\": {\n \"annotations\": [],\n \"type\": \"AzureKeyVault\",\n \"typeProperties\": {\n \"baseUrl\": \"[parameters('Keys_properties_typeProperties_baseUrl')]\"\n }\n },\n \"dependsOn\": []\n },\n {\n \"name\": \"[concat(parameters('factoryName'), '/PipelineExecutor')]\",\n \"type\": \"Microsoft.DataFactory/factories/linkedServices\",\n \"apiVersion\": \"2018-06-01\",\n \"properties\": {\n \"annotations\": [],\n \"type\": \"AzureFunction\",\n \"typeProperties\": {\n \"functionAppUrl\": \"https://pipelineexecutor.azurewebsites.net\",\n \"functionKey\": {\n \"type\": \"AzureKeyVaultSecret\",\n \"store\": {\n \"referenceName\": \"Keys\",\n \"type\": \"LinkedServiceReference\"\n },\n \"secretName\": \"ExecutorFunctionAppKey\"\n }\n }\n },\n \"dependsOn\": [\n \"[concat(variables('factoryId'), '/linkedServices/Keys')]\"\n ]\n },\n {\n \"name\": \"[concat(parameters('factoryName'), '/SupportDatabase')]\",\n \"type\": \"Microsoft.DataFactory/factories/linkedServices\",\n \"apiVersion\": \"2018-06-01\",\n \"properties\": {\n \"annotations\": [],\n \"type\": \"AzureSqlDatabase\",\n \"typeProperties\": {\n \"connectionString\": {\n \"type\": \"AzureKeyVaultSecret\",\n \"store\": {\n \"referenceName\": \"Keys\",\n \"type\": \"LinkedServiceReference\"\n },\n \"secretName\": \"[parameters('SupportDatabase_properties_typeProperties_connectionString_secretName')]\"\n }\n }\n },\n \"dependsOn\": [\n \"[concat(variables('factoryId'), '/linkedServices/Keys')]\"\n ]\n }\n ]\n}"
},
{
"path": "ARM Templates/Data Factory/v1.1 Export.json",
"content": "{\n \"$schema\": \"http://schema.management.azure.com/schemas/2015-01-01/deploymentTemplate.json#\",\n \"contentVersion\": \"1.0.0.0\",\n \"parameters\": {\n \"factoryName\": {\n \"type\": \"string\",\n \"metadata\": \"Data Factory name\",\n \"defaultValue\": \"\"\n },\n \"Keys_properties_typeProperties_baseUrl\": {\n \"type\": \"string\",\n \"defaultValue\": \"\"\n },\n \"SupportDatabase_properties_typeProperties_connectionString_secretName\": {\n \"type\": \"string\",\n \"defaultValue\": \"\"\n }\n },\n \"variables\": {\n \"factoryId\": \"[concat('Microsoft.DataFactory/factories/', parameters('factoryName'))]\"\n },\n \"resources\": [\n {\n \"name\": \"[concat(parameters('factoryName'), '/BootStrap')]\",\n \"type\": \"Microsoft.DataFactory/factories/pipelines\",\n \"apiVersion\": \"2018-06-01\",\n \"properties\": {\n \"activities\": [\n {\n \"name\": \"Get Stages\",\n \"type\": \"Lookup\",\n \"dependsOn\": [\n {\n \"activity\": \"Create New Execution\",\n \"dependencyConditions\": [\n \"Succeeded\"\n ]\n }\n ],\n \"policy\": {\n \"timeout\": \"7.00:00:00\",\n \"retry\": 0,\n \"retryIntervalInSeconds\": 30,\n \"secureOutput\": false,\n \"secureInput\": false\n },\n \"userProperties\": [],\n \"typeProperties\": {\n \"source\": {\n \"type\": \"AzureSqlSource\",\n \"sqlReaderStoredProcedureName\": \"[[procfwk].[GetProcessStages]\",\n \"storedProcedureParameters\": {\n \"ExecutionId\": {\n \"type\": \"Guid\",\n \"value\": {\n \"value\": \"@activity('Create New Execution').output.firstRow.ExecutionId\",\n \"type\": \"Expression\"\n }\n }\n },\n \"queryTimeout\": \"02:00:00\"\n },\n \"dataset\": {\n \"referenceName\": \"GetSetMetadata\",\n \"type\": \"DatasetReference\",\n \"parameters\": {}\n },\n \"firstRowOnly\": false\n }\n },\n {\n \"name\": \"Execute Stages\",\n \"type\": \"ForEach\",\n \"dependsOn\": [\n {\n \"activity\": \"Get Stages\",\n \"dependencyConditions\": [\n \"Succeeded\"\n ]\n },\n {\n \"activity\": \"Get Tenant Id\",\n \"dependencyConditions\": [\n \"Succeeded\"\n ]\n },\n {\n \"activity\": \"Get Subscription Id\",\n \"dependencyConditions\": [\n \"Succeeded\"\n ]\n }\n ],\n \"userProperties\": [],\n \"typeProperties\": {\n \"items\": {\n \"value\": \"@activity('Get Stages').output.value\",\n \"type\": \"Expression\"\n },\n \"isSequential\": true,\n \"activities\": [\n {\n \"name\": \"Stage Executor\",\n \"type\": \"ExecutePipeline\",\n \"dependsOn\": [\n {\n \"activity\": \"Log Stage Start\",\n \"dependencyConditions\": [\n \"Succeeded\"\n ]\n }\n ],\n \"userProperties\": [],\n \"typeProperties\": {\n \"pipeline\": {\n \"referenceName\": \"Executor\",\n \"type\": \"PipelineReference\"\n },\n \"waitOnCompletion\": true,\n \"parameters\": {\n \"StageId\": {\n \"value\": \"@item().StageId\",\n \"type\": \"Expression\"\n },\n \"ExecutionId\": {\n \"value\": \"@activity('Create New Execution').output.firstRow.ExecutionId\",\n \"type\": \"Expression\"\n },\n \"TenantId\": {\n \"value\": \"@activity('Get Tenant Id').output.firstRow.PropertyValue\",\n \"type\": \"Expression\"\n },\n \"SubscriptionId\": {\n \"value\": \"@activity('Get Subscription Id').output.firstRow.PropertyValue\",\n \"type\": \"Expression\"\n }\n }\n }\n },\n {\n \"name\": \"Log Stage Start\",\n \"type\": \"SqlServerStoredProcedure\",\n \"dependsOn\": [],\n \"policy\": {\n \"timeout\": \"7.00:00:00\",\n \"retry\": 0,\n \"retryIntervalInSeconds\": 30,\n \"secureOutput\": false,\n \"secureInput\": false\n },\n \"userProperties\": [],\n \"typeProperties\": {\n \"storedProcedureName\": \"[[procfwk].[SetLogStageStart]\",\n \"storedProcedureParameters\": {\n \"ExecutionId\": {\n \"value\": {\n \"value\": \"@activity('Create New Execution').output.firstRow.ExecutionId\",\n \"type\": \"Expression\"\n },\n \"type\": \"Guid\"\n },\n \"StageId\": {\n \"value\": {\n \"value\": \"@item().StageId\",\n \"type\": \"Expression\"\n },\n \"type\": \"Int32\"\n }\n }\n },\n \"linkedServiceName\": {\n \"referenceName\": \"SupportDatabase\",\n \"type\": \"LinkedServiceReference\"\n }\n }\n ]\n }\n },\n {\n \"name\": \"Create New Execution\",\n \"type\": \"Lookup\",\n \"dependsOn\": [],\n \"policy\": {\n \"timeout\": \"7.00:00:00\",\n \"retry\": 0,\n \"retryIntervalInSeconds\": 30,\n \"secureOutput\": false,\n \"secureInput\": false\n },\n \"userProperties\": [],\n \"typeProperties\": {\n \"source\": {\n \"type\": \"AzureSqlSource\",\n \"sqlReaderStoredProcedureName\": \"[[procfwk].[CreateNewExecution]\",\n \"queryTimeout\": \"02:00:00\"\n },\n \"dataset\": {\n \"referenceName\": \"GetSetMetadata\",\n \"type\": \"DatasetReference\",\n \"parameters\": {}\n }\n }\n },\n {\n \"name\": \"Update Execution Log\",\n \"type\": \"SqlServerStoredProcedure\",\n \"dependsOn\": [\n {\n \"activity\": \"Execute Stages\",\n \"dependencyConditions\": [\n \"Succeeded\"\n ]\n }\n ],\n \"policy\": {\n \"timeout\": \"7.00:00:00\",\n \"retry\": 0,\n \"retryIntervalInSeconds\": 30,\n \"secureOutput\": false,\n \"secureInput\": false\n },\n \"userProperties\": [],\n \"typeProperties\": {\n \"storedProcedureName\": \"[[procfwk].[UpdateExecutionLog]\"\n },\n \"linkedServiceName\": {\n \"referenceName\": \"SupportDatabase\",\n \"type\": \"LinkedServiceReference\"\n }\n },\n {\n \"name\": \"Get Tenant Id\",\n \"type\": \"Lookup\",\n \"dependsOn\": [],\n \"policy\": {\n \"timeout\": \"7.00:00:00\",\n \"retry\": 0,\n \"retryIntervalInSeconds\": 30,\n \"secureOutput\": false,\n \"secureInput\": false\n },\n \"userProperties\": [],\n \"typeProperties\": {\n \"source\": {\n \"type\": \"AzureSqlSource\",\n \"sqlReaderStoredProcedureName\": \"[[procfwk].[GetPropertyValue]\",\n \"storedProcedureParameters\": {\n \"PropertyName\": {\n \"type\": \"String\",\n \"value\": \"TenantId\"\n }\n },\n \"queryTimeout\": \"02:00:00\"\n },\n \"dataset\": {\n \"referenceName\": \"GetSetMetadata\",\n \"type\": \"DatasetReference\",\n \"parameters\": {}\n },\n \"firstRowOnly\": true\n }\n },\n {\n \"name\": \"Get Subscription Id\",\n \"type\": \"Lookup\",\n \"dependsOn\": [],\n \"policy\": {\n \"timeout\": \"7.00:00:00\",\n \"retry\": 0,\n \"retryIntervalInSeconds\": 30,\n \"secureOutput\": false,\n \"secureInput\": true\n },\n \"userProperties\": [],\n \"typeProperties\": {\n \"source\": {\n \"type\": \"AzureSqlSource\",\n \"sqlReaderStoredProcedureName\": \"[[procfwk].[GetPropertyValue]\",\n \"storedProcedureParameters\": {\n \"PropertyName\": {\n \"type\": \"String\",\n \"value\": \"SubscriptionId\"\n }\n },\n \"queryTimeout\": \"02:00:00\"\n },\n \"dataset\": {\n \"referenceName\": \"GetSetMetadata\",\n \"type\": \"DatasetReference\",\n \"parameters\": {}\n }\n }\n }\n ],\n \"folder\": {\n \"name\": \"Framework\"\n },\n \"annotations\": []\n },\n \"dependsOn\": [\n \"[concat(variables('factoryId'), '/datasets/GetSetMetadata')]\",\n \"[concat(variables('factoryId'), '/linkedServices/SupportDatabase')]\",\n \"[concat(variables('factoryId'), '/pipelines/Executor')]\"\n ]\n },\n {\n \"name\": \"[concat(parameters('factoryName'), '/Child 1')]\",\n \"type\": \"Microsoft.DataFactory/factories/pipelines\",\n \"apiVersion\": \"2018-06-01\",\n \"properties\": {\n \"activities\": [\n {\n \"name\": \"Wait1\",\n \"type\": \"Wait\",\n \"dependsOn\": [],\n \"userProperties\": [],\n \"typeProperties\": {\n \"waitTimeInSeconds\": 1\n }\n }\n ],\n \"folder\": {\n \"name\": \"Child Pipelines\"\n },\n \"annotations\": []\n },\n \"dependsOn\": []\n },\n {\n \"name\": \"[concat(parameters('factoryName'), '/Child 2')]\",\n \"type\": \"Microsoft.DataFactory/factories/pipelines\",\n \"apiVersion\": \"2018-06-01\",\n \"properties\": {\n \"activities\": [\n {\n \"name\": \"Wait1\",\n \"type\": \"Wait\",\n \"dependsOn\": [],\n \"userProperties\": [],\n \"typeProperties\": {\n \"waitTimeInSeconds\": 1\n }\n }\n ],\n \"folder\": {\n \"name\": \"Child Pipelines\"\n },\n \"annotations\": []\n },\n \"dependsOn\": []\n },\n {\n \"name\": \"[concat(parameters('factoryName'), '/Child 3')]\",\n \"type\": \"Microsoft.DataFactory/factories/pipelines\",\n \"apiVersion\": \"2018-06-01\",\n \"properties\": {\n \"activities\": [\n {\n \"name\": \"Wait1\",\n \"type\": \"Wait\",\n \"dependsOn\": [],\n \"userProperties\": [],\n \"typeProperties\": {\n \"waitTimeInSeconds\": 1\n }\n }\n ],\n \"folder\": {\n \"name\": \"Child Pipelines\"\n },\n \"annotations\": []\n },\n \"dependsOn\": []\n },\n {\n \"name\": \"[concat(parameters('factoryName'), '/Child 4')]\",\n \"type\": \"Microsoft.DataFactory/factories/pipelines\",\n \"apiVersion\": \"2018-06-01\",\n \"properties\": {\n \"activities\": [\n {\n \"name\": \"Wait1\",\n \"type\": \"Wait\",\n \"dependsOn\": [],\n \"userProperties\": [],\n \"typeProperties\": {\n \"waitTimeInSeconds\": 1\n }\n }\n ],\n \"folder\": {\n \"name\": \"Child Pipelines\"\n },\n \"annotations\": []\n },\n \"dependsOn\": []\n },\n {\n \"name\": \"[concat(parameters('factoryName'), '/Child 5')]\",\n \"type\": \"Microsoft.DataFactory/factories/pipelines\",\n \"apiVersion\": \"2018-06-01\",\n \"properties\": {\n \"activities\": [\n {\n \"name\": \"Wait1\",\n \"type\": \"Wait\",\n \"dependsOn\": [],\n \"userProperties\": [],\n \"typeProperties\": {\n \"waitTimeInSeconds\": 1\n }\n }\n ],\n \"folder\": {\n \"name\": \"Child Pipelines\"\n },\n \"annotations\": []\n },\n \"dependsOn\": []\n },\n {\n \"name\": \"[concat(parameters('factoryName'), '/Child 6')]\",\n \"type\": \"Microsoft.DataFactory/factories/pipelines\",\n \"apiVersion\": \"2018-06-01\",\n \"properties\": {\n \"activities\": [\n {\n \"name\": \"Wait1\",\n \"type\": \"Wait\",\n \"dependsOn\": [],\n \"userProperties\": [],\n \"typeProperties\": {\n \"waitTimeInSeconds\": 1\n }\n }\n ],\n \"folder\": {\n \"name\": \"Child Pipelines\"\n },\n \"annotations\": []\n },\n \"dependsOn\": []\n },\n {\n \"name\": \"[concat(parameters('factoryName'), '/Executor')]\",\n \"type\": \"Microsoft.DataFactory/factories/pipelines\",\n \"apiVersion\": \"2018-06-01\",\n \"properties\": {\n \"activities\": [\n {\n \"name\": \"Get Pipelines\",\n \"type\": \"Lookup\",\n \"dependsOn\": [],\n \"policy\": {\n \"timeout\": \"7.00:00:00\",\n \"retry\": 0,\n \"retryIntervalInSeconds\": 30,\n \"secureOutput\": false,\n \"secureInput\": false\n },\n \"userProperties\": [],\n \"typeProperties\": {\n \"source\": {\n \"type\": \"AzureSqlSource\",\n \"sqlReaderStoredProcedureName\": \"[[procfwk].[GetPipelinesInStage]\",\n \"storedProcedureParameters\": {\n \"StageId\": {\n \"type\": \"Int32\",\n \"value\": {\n \"value\": \"@pipeline().parameters.StageId\",\n \"type\": \"Expression\"\n }\n }\n },\n \"queryTimeout\": \"02:00:00\"\n },\n \"dataset\": {\n \"referenceName\": \"GetSetMetadata\",\n \"type\": \"DatasetReference\",\n \"parameters\": {}\n },\n \"firstRowOnly\": false\n }\n },\n {\n \"name\": \"Execute Pipelines\",\n \"type\": \"ForEach\",\n \"dependsOn\": [\n {\n \"activity\": \"Get Pipelines\",\n \"dependencyConditions\": [\n \"Succeeded\"\n ]\n }\n ],\n \"userProperties\": [],\n \"typeProperties\": {\n \"items\": {\n \"value\": \"@activity('Get Pipelines').output.value\",\n \"type\": \"Expression\"\n },\n \"activities\": [\n {\n \"name\": \"Execute Pipeline\",\n \"type\": \"AzureFunctionActivity\",\n \"dependsOn\": [\n {\n \"activity\": \"Log Pipeline Running\",\n \"dependencyConditions\": [\n \"Succeeded\"\n ]\n },\n {\n \"activity\": \"Get Pipeline Params\",\n \"dependencyConditions\": [\n \"Succeeded\"\n ]\n },\n {\n \"activity\": \"Get SPN Details\",\n \"dependencyConditions\": [\n \"Succeeded\"\n ]\n }\n ],\n \"policy\": {\n \"timeout\": \"7.00:00:00\",\n \"retry\": 0,\n \"retryIntervalInSeconds\": 30,\n \"secureOutput\": false,\n \"secureInput\": false\n },\n \"userProperties\": [],\n \"typeProperties\": {\n \"functionName\": \"ExecutePipeline\",\n \"method\": \"POST\",\n \"headers\": {},\n \"body\": {\n \"value\": \"@concat('\\n{\\n\\t\\\"tenantId\\\": \\\"',pipeline().parameters.TenantId,'\\\",\\n\\t\\\"applicationId\\\": \\\"',activity('Get SPN Details').output.firstRow.Id,'\\\",\\n\\t\\\"authenticationKey\\\": \\\"',activity('Get SPN Details').output.firstRow.Secret,'\\\",\\n\\t\\\"subscriptionId\\\": \\\"',pipeline().parameters.SubscriptionId,'\\\",\\n\\t\\\"resourceGroup\\\": \\\"',item().ResourceGroupName,'\\\",\\n\\t\\\"factoryName\\\": \\\"',item().DataFactoryName,'\\\",\\n\\t\\\"pipelineName\\\": \\\"',item().PipelineName,'\\\"',activity('Get Pipeline Params').output.firstRow.Params,'\\n}')\",\n \"type\": \"Expression\"\n }\n },\n \"linkedServiceName\": {\n \"referenceName\": \"PipelineExecutor\",\n \"type\": \"LinkedServiceReference\"\n }\n },\n {\n \"name\": \"Get Pipeline Params\",\n \"type\": \"Lookup\",\n \"dependsOn\": [],\n \"policy\": {\n \"timeout\": \"7.00:00:00\",\n \"retry\": 0,\n \"retryIntervalInSeconds\": 30,\n \"secureOutput\": false,\n \"secureInput\": false\n },\n \"userProperties\": [],\n \"typeProperties\": {\n \"source\": {\n \"type\": \"AzureSqlSource\",\n \"sqlReaderStoredProcedureName\": \"[[procfwk].[GetPipelineParameters]\",\n \"storedProcedureParameters\": {\n \"PipelineId\": {\n \"type\": \"Int32\",\n \"value\": {\n \"value\": \"@item().PipelineId\",\n \"type\": \"Expression\"\n }\n }\n },\n \"queryTimeout\": \"02:00:00\"\n },\n \"dataset\": {\n \"referenceName\": \"GetSetMetadata\",\n \"type\": \"DatasetReference\",\n \"parameters\": {}\n }\n }\n },\n {\n \"name\": \"Log Pipeline Running\",\n \"type\": \"SqlServerStoredProcedure\",\n \"dependsOn\": [],\n \"policy\": {\n \"timeout\": \"7.00:00:00\",\n \"retry\": 0,\n \"retryIntervalInSeconds\": 30,\n \"secureOutput\": false,\n \"secureInput\": false\n },\n \"userProperties\": [],\n \"typeProperties\": {\n \"storedProcedureName\": \"[[procfwk].[SetLogPipelineRunning]\",\n \"storedProcedureParameters\": {\n \"ExecutionId\": {\n \"value\": {\n \"value\": \"@pipeline().parameters.ExecutionId\",\n \"type\": \"Expression\"\n },\n \"type\": \"Guid\"\n },\n \"PipelineId\": {\n \"value\": {\n \"value\": \"@item().PipelineId\",\n \"type\": \"Expression\"\n },\n \"type\": \"Int32\"\n },\n \"StageId\": {\n \"value\": {\n \"value\": \"@pipeline().parameters.StageId\",\n \"type\": \"Expression\"\n },\n \"type\": \"Int32\"\n }\n }\n },\n \"linkedServiceName\": {\n \"referenceName\": \"SupportDatabase\",\n \"type\": \"LinkedServiceReference\"\n }\n },\n {\n \"name\": \"Log Pipeline Success\",\n \"type\": \"SqlServerStoredProcedure\",\n \"dependsOn\": [\n {\n \"activity\": \"Execute Pipeline\",\n \"dependencyConditions\": [\n \"Succeeded\"\n ]\n }\n ],\n \"policy\": {\n \"timeout\": \"7.00:00:00\",\n \"retry\": 0,\n \"retryIntervalInSeconds\": 30,\n \"secureOutput\": false,\n \"secureInput\": false\n },\n \"userProperties\": [],\n \"typeProperties\": {\n \"storedProcedureName\": \"[[procfwk].[SetLogPipelineSuccess]\",\n \"storedProcedureParameters\": {\n \"ExecutionId\": {\n \"value\": {\n \"value\": \"@pipeline().parameters.ExecutionId\",\n \"type\": \"Expression\"\n },\n \"type\": \"Guid\"\n },\n \"PipelineId\": {\n \"value\": {\n \"value\": \"@item().PipelineId\",\n \"type\": \"Expression\"\n },\n \"type\": \"Int32\"\n },\n \"StageId\": {\n \"value\": {\n \"value\": \"@pipeline().parameters.StageId\",\n \"type\": \"Expression\"\n },\n \"type\": \"Int32\"\n }\n }\n },\n \"linkedServiceName\": {\n \"referenceName\": \"SupportDatabase\",\n \"type\": \"LinkedServiceReference\"\n }\n },\n {\n \"name\": \"Log Pipeline Failure\",\n \"type\": \"SqlServerStoredProcedure\",\n \"dependsOn\": [\n {\n \"activity\": \"Execute Pipeline\",\n \"dependencyConditions\": [\n \"Failed\"\n ]\n }\n ],\n \"policy\": {\n \"timeout\": \"7.00:00:00\",\n \"retry\": 0,\n \"retryIntervalInSeconds\": 30,\n \"secureOutput\": false,\n \"secureInput\": false\n },\n \"userProperties\": [],\n \"typeProperties\": {\n \"storedProcedureName\": \"[[procfwk].[SetLogPipelineFailed]\",\n \"storedProcedureParameters\": {\n \"ExecutionId\": {\n \"value\": {\n \"value\": \"@pipeline().parameters.ExecutionId\",\n \"type\": \"Expression\"\n },\n \"type\": \"Guid\"\n },\n \"PipelineId\": {\n \"value\": {\n \"value\": \"@item().PipelineId\",\n \"type\": \"Expression\"\n },\n \"type\": \"Int32\"\n },\n \"StageId\": {\n \"value\": {\n \"value\": \"@pipeline().parameters.StageId\",\n \"type\": \"Expression\"\n },\n \"type\": \"Int32\"\n }\n }\n },\n \"linkedServiceName\": {\n \"referenceName\": \"SupportDatabase\",\n \"type\": \"LinkedServiceReference\"\n }\n },\n {\n \"name\": \"Get SPN Details\",\n \"type\": \"Lookup\",\n \"dependsOn\": [],\n \"policy\": {\n \"timeout\": \"7.00:00:00\",\n \"retry\": 0,\n \"retryIntervalInSeconds\": 30,\n \"secureOutput\": false,\n \"secureInput\": false\n },\n \"userProperties\": [],\n \"typeProperties\": {\n \"source\": {\n \"type\": \"AzureSqlSource\",\n \"sqlReaderStoredProcedureName\": \"[[procfwk].[GetServicePrincipal]\",\n \"storedProcedureParameters\": {\n \"DataFactory\": {\n \"type\": \"String\",\n \"value\": {\n \"value\": \"@item().DataFactoryName\",\n \"type\": \"Expression\"\n }\n },\n \"PipelineName\": {\n \"type\": \"String\",\n \"value\": {\n \"value\": \"@item().PipelineName\",\n \"type\": \"Expression\"\n }\n }\n },\n \"queryTimeout\": \"02:00:00\"\n },\n \"dataset\": {\n \"referenceName\": \"GetSetMetadata\",\n \"type\": \"DatasetReference\",\n \"parameters\": {}\n }\n }\n }\n ]\n }\n }\n ],\n \"parameters\": {\n \"StageId\": {\n \"type\": \"int\"\n },\n \"ExecutionId\": {\n \"type\": \"string\"\n },\n \"TenantId\": {\n \"type\": \"string\"\n },\n \"SubscriptionId\": {\n \"type\": \"string\"\n }\n },\n \"variables\": {\n \"FunctionBody\": {\n \"type\": \"String\"\n }\n },\n \"folder\": {\n \"name\": \"Framework\"\n },\n \"annotations\": []\n },\n \"dependsOn\": [\n \"[concat(variables('factoryId'), '/datasets/GetSetMetadata')]\",\n \"[concat(variables('factoryId'), '/linkedServices/PipelineExecutor')]\",\n \"[concat(variables('factoryId'), '/linkedServices/SupportDatabase')]\"\n ]\n },\n {\n \"name\": \"[concat(parameters('factoryName'), '/Grandparent')]\",\n \"type\": \"Microsoft.DataFactory/factories/pipelines\",\n \"apiVersion\": \"2018-06-01\",\n \"properties\": {\n \"activities\": [\n {\n \"name\": \"Framework Processing\",\n \"type\": \"ExecutePipeline\",\n \"dependsOn\": [],\n \"userProperties\": [],\n \"typeProperties\": {\n \"pipeline\": {\n \"referenceName\": \"BootStrap\",\n \"type\": \"PipelineReference\"\n },\n \"waitOnCompletion\": true,\n \"parameters\": {}\n }\n }\n ],\n \"annotations\": []\n },\n \"dependsOn\": [\n \"[concat(variables('factoryId'), '/pipelines/BootStrap')]\"\n ]\n },\n {\n \"name\": \"[concat(parameters('factoryName'), '/GetSetMetadata')]\",\n \"type\": \"Microsoft.DataFactory/factories/datasets\",\n \"apiVersion\": \"2018-06-01\",\n \"properties\": {\n \"linkedServiceName\": {\n \"referenceName\": \"SupportDatabase\",\n \"type\": \"LinkedServiceReference\"\n },\n \"annotations\": [],\n \"type\": \"AzureSqlTable\",\n \"schema\": [],\n \"typeProperties\": {}\n },\n \"dependsOn\": [\n \"[concat(variables('factoryId'), '/linkedServices/SupportDatabase')]\"\n ]\n },\n {\n \"name\": \"[concat(parameters('factoryName'), '/Keys')]\",\n \"type\": \"Microsoft.DataFactory/factories/linkedServices\",\n \"apiVersion\": \"2018-06-01\",\n \"properties\": {\n \"annotations\": [],\n \"type\": \"AzureKeyVault\",\n \"typeProperties\": {\n \"baseUrl\": \"[parameters('Keys_properties_typeProperties_baseUrl')]\"\n }\n },\n \"dependsOn\": []\n },\n {\n \"name\": \"[concat(parameters('factoryName'), '/PipelineExecutor')]\",\n \"type\": \"Microsoft.DataFactory/factories/linkedServices\",\n \"apiVersion\": \"2018-06-01\",\n \"properties\": {\n \"annotations\": [],\n \"type\": \"AzureFunction\",\n \"typeProperties\": {\n \"functionAppUrl\": \"https://pipelineexecutor.azurewebsites.net\",\n \"functionKey\": {\n \"type\": \"AzureKeyVaultSecret\",\n \"store\": {\n \"referenceName\": \"Keys\",\n \"type\": \"LinkedServiceReference\"\n },\n \"secretName\": \"ExecutorFunctionAppKey\"\n }\n }\n },\n \"dependsOn\": [\n \"[concat(variables('factoryId'), '/linkedServices/Keys')]\"\n ]\n },\n {\n \"name\": \"[concat(parameters('factoryName'), '/SupportDatabase')]\",\n \"type\": \"Microsoft.DataFactory/factories/linkedServices\",\n \"apiVersion\": \"2018-06-01\",\n \"properties\": {\n \"annotations\": [],\n \"type\": \"AzureSqlDatabase\",\n \"typeProperties\": {\n \"connectionString\": {\n \"type\": \"AzureKeyVaultSecret\",\n \"store\": {\n \"referenceName\": \"Keys\",\n \"type\": \"LinkedServiceReference\"\n },\n \"secretName\": \"[parameters('SupportDatabase_properties_typeProperties_connectionString_secretName')]\"\n }\n }\n },\n \"dependsOn\": [\n \"[concat(variables('factoryId'), '/linkedServices/Keys')]\"\n ]\n }\n ]\n}"
},
{
"path": "ARM Templates/Data Factory/v1.2 Export.json",
"content": "{\n \"$schema\": \"http://schema.management.azure.com/schemas/2015-01-01/deploymentTemplate.json#\",\n \"contentVersion\": \"1.0.0.0\",\n \"parameters\": {\n \"factoryName\": {\n \"type\": \"string\",\n \"metadata\": \"Data Factory name\",\n \"defaultValue\": \"\"\n },\n \"Keys_properties_typeProperties_baseUrl\": {\n \"type\": \"string\",\n \"defaultValue\": \"\"\n },\n \"SupportDatabase_properties_typeProperties_connectionString_secretName\": {\n \"type\": \"string\",\n \"defaultValue\": \"\"\n }\n },\n \"variables\": {\n \"factoryId\": \"[concat('Microsoft.DataFactory/factories/', parameters('factoryName'))]\"\n },\n \"resources\": [\n {\n \"name\": \"[concat(parameters('factoryName'), '/01 Grandparent')]\",\n \"type\": \"Microsoft.DataFactory/factories/pipelines\",\n \"apiVersion\": \"2018-06-01\",\n \"properties\": {\n \"activities\": [\n {\n \"name\": \"Framework Processing\",\n \"type\": \"ExecutePipeline\",\n \"dependsOn\": [],\n \"userProperties\": [],\n \"typeProperties\": {\n \"pipeline\": {\n \"referenceName\": \"02 Parent\",\n \"type\": \"PipelineReference\"\n },\n \"waitOnCompletion\": true,\n \"parameters\": {}\n }\n }\n ],\n \"folder\": {\n \"name\": \"_ProcFwk\"\n },\n \"annotations\": []\n },\n \"dependsOn\": [\n \"[concat(variables('factoryId'), '/pipelines/02 Parent')]\"\n ]\n },\n {\n \"name\": \"[concat(parameters('factoryName'), '/02 Parent')]\",\n \"type\": \"Microsoft.DataFactory/factories/pipelines\",\n \"apiVersion\": \"2018-06-01\",\n \"properties\": {\n \"description\": \"ADF.procfwk parent pipeline used to bootstrap the orchestration framework in perform the first level ForEach calls in sequence for the metadata stages.\",\n \"activities\": [\n {\n \"name\": \"Get Stages\",\n \"description\": \"Returns a distinct list of execution stages within the framework metadata.\",\n \"type\": \"Lookup\",\n \"dependsOn\": [\n {\n \"activity\": \"Execution Wrapper\",\n \"dependencyConditions\": [\n \"Succeeded\"\n ]\n }\n ],\n \"policy\": {\n \"timeout\": \"7.00:00:00\",\n \"retry\": 0,\n \"retryIntervalInSeconds\": 30,\n \"secureOutput\": false,\n \"secureInput\": false\n },\n \"userProperties\": [],\n \"typeProperties\": {\n \"source\": {\n \"type\": \"AzureSqlSource\",\n \"sqlReaderStoredProcedureName\": \"[[procfwk].[GetStages]\",\n \"storedProcedureParameters\": {\n \"ExecutionId\": {\n \"type\": \"Guid\",\n \"value\": {\n \"value\": \"@activity('Execution Wrapper').output.firstRow.ExecutionId\",\n \"type\": \"Expression\"\n }\n }\n },\n \"queryTimeout\": \"02:00:00\"\n },\n \"dataset\": {\n \"referenceName\": \"GetSetMetadata\",\n \"type\": \"DatasetReference\",\n \"parameters\": {}\n },\n \"firstRowOnly\": false\n }\n },\n {\n \"name\": \"Execute Stages\",\n \"description\": \"Top level ForEach to sequentially call all processing stages within the framework metadata. Items for iteration passed from the Get Stages lookup activity.\",\n \"type\": \"ForEach\",\n \"dependsOn\": [\n {\n \"activity\": \"Get Stages\",\n \"dependencyConditions\": [\n \"Succeeded\"\n ]\n },\n {\n \"activity\": \"Get Tenant Id\",\n \"dependencyConditions\": [\n \"Succeeded\"\n ]\n },\n {\n \"activity\": \"Get Subscription Id\",\n \"dependencyConditions\": [\n \"Succeeded\"\n ]\n }\n ],\n \"userProperties\": [],\n \"typeProperties\": {\n \"items\": {\n \"value\": \"@activity('Get Stages').output.value\",\n \"type\": \"Expression\"\n },\n \"isSequential\": true,\n \"activities\": [\n {\n \"name\": \"Stage Executor\",\n \"description\": \"Call to the framework generic child pipeline for a given execution stage.\",\n \"type\": \"ExecutePipeline\",\n \"dependsOn\": [\n {\n \"activity\": \"Log Stage Preparing\",\n \"dependencyConditions\": [\n \"Succeeded\"\n ]\n }\n ],\n \"userProperties\": [],\n \"typeProperties\": {\n \"pipeline\": {\n \"referenceName\": \"03 Child\",\n \"type\": \"PipelineReference\"\n },\n \"waitOnCompletion\": true,\n \"parameters\": {\n \"StageId\": {\n \"value\": \"@item().StageId\",\n \"type\": \"Expression\"\n },\n \"ExecutionId\": {\n \"value\": \"@activity('Execution Wrapper').output.firstRow.ExecutionId\",\n \"type\": \"Expression\"\n },\n \"TenantId\": {\n \"value\": \"@activity('Get Tenant Id').output.firstRow.PropertyValue\",\n \"type\": \"Expression\"\n },\n \"SubscriptionId\": {\n \"value\": \"@activity('Get Subscription Id').output.firstRow.PropertyValue\",\n \"type\": \"Expression\"\n }\n }\n }\n },\n {\n \"name\": \"Log Stage Preparing\",\n \"description\": \"Update the current execution table flagging all pipelines within the stage as preparing.\",\n \"type\": \"SqlServerStoredProcedure\",\n \"dependsOn\": [\n {\n \"activity\": \"Check for Blockers\",\n \"dependencyConditions\": [\n \"Succeeded\"\n ]\n }\n ],\n \"policy\": {\n \"timeout\": \"7.00:00:00\",\n \"retry\": 0,\n \"retryIntervalInSeconds\": 30,\n \"secureOutput\": false,\n \"secureInput\": false\n },\n \"userProperties\": [],\n \"typeProperties\": {\n \"storedProcedureName\": \"[[procfwk].[SetLogStagePreparing]\",\n \"storedProcedureParameters\": {\n \"ExecutionId\": {\n \"value\": {\n \"value\": \"@activity('Execution Wrapper').output.firstRow.ExecutionId\",\n \"type\": \"Expression\"\n },\n \"type\": \"Guid\"\n },\n \"StageId\": {\n \"value\": {\n \"value\": \"@item().StageId\",\n \"type\": \"Expression\"\n },\n \"type\": \"Int32\"\n }\n }\n },\n \"linkedServiceName\": {\n \"referenceName\": \"SupportDatabase\",\n \"type\": \"LinkedServiceReference\"\n }\n },\n {\n \"name\": \"Check for Blockers\",\n \"description\": \"Used to double check and stop the next execution stage if failures and blockers have be incurred. Without this step processing would continue regardless of upstream failures.\",\n \"type\": \"SqlServerStoredProcedure\",\n \"dependsOn\": [],\n \"policy\": {\n \"timeout\": \"7.00:00:00\",\n \"retry\": 0,\n \"retryIntervalInSeconds\": 30,\n \"secureOutput\": false,\n \"secureInput\": false\n },\n \"userProperties\": [],\n \"typeProperties\": {\n \"storedProcedureName\": \"[[procfwk].[CheckForBlockedPipelines]\",\n \"storedProcedureParameters\": {\n \"StageId\": {\n \"value\": {\n \"value\": \"@item().StageId\",\n \"type\": \"Expression\"\n },\n \"type\": \"Int32\"\n }\n }\n },\n \"linkedServiceName\": {\n \"referenceName\": \"SupportDatabase\",\n \"type\": \"LinkedServiceReference\"\n }\n }\n ]\n }\n },\n {\n \"name\": \"Execution Wrapper\",\n \"description\": \"Wrapper to reset and restart processing or create a completely new execution instance of the framework metadata.\",\n \"type\": \"Lookup\",\n \"dependsOn\": [],\n \"policy\": {\n \"timeout\": \"7.00:00:00\",\n \"retry\": 0,\n \"retryIntervalInSeconds\": 30,\n \"secureOutput\": false,\n \"secureInput\": false\n },\n \"userProperties\": [],\n \"typeProperties\": {\n \"source\": {\n \"type\": \"AzureSqlSource\",\n \"sqlReaderStoredProcedureName\": \"[[procfwk].[ExecutionWrapper]\",\n \"queryTimeout\": \"02:00:00\"\n },\n \"dataset\": {\n \"referenceName\": \"GetSetMetadata\",\n \"type\": \"DatasetReference\",\n \"parameters\": {}\n }\n }\n },\n {\n \"name\": \"Archive Execution Log\",\n \"description\": \"After a successful execution run the current execution metadata is moved to the long term logging table by this stored procedure call.\",\n \"type\": \"SqlServerStoredProcedure\",\n \"dependsOn\": [\n {\n \"activity\": \"Execute Stages\",\n \"dependencyConditions\": [\n \"Succeeded\"\n ]\n }\n ],\n \"policy\": {\n \"timeout\": \"7.00:00:00\",\n \"retry\": 0,\n \"retryIntervalInSeconds\": 30,\n \"secureOutput\": false,\n \"secureInput\": false\n },\n \"userProperties\": [],\n \"typeProperties\": {\n \"storedProcedureName\": \"[[procfwk].[UpdateExecutionLog]\"\n },\n \"linkedServiceName\": {\n \"referenceName\": \"SupportDatabase\",\n \"type\": \"LinkedServiceReference\"\n }\n },\n {\n \"name\": \"Get Tenant Id\",\n \"description\": \"Returning the Azure Tenant Id from the metadata properties table.\",\n \"type\": \"Lookup\",\n \"dependsOn\": [],\n \"policy\": {\n \"timeout\": \"7.00:00:00\",\n \"retry\": 0,\n \"retryIntervalInSeconds\": 30,\n \"secureOutput\": false,\n \"secureInput\": false\n },\n \"userProperties\": [],\n \"typeProperties\": {\n \"source\": {\n \"type\": \"AzureSqlSource\",\n \"sqlReaderStoredProcedureName\": \"[[procfwk].[GetPropertyValue]\",\n \"storedProcedureParameters\": {\n \"PropertyName\": {\n \"type\": \"String\",\n \"value\": \"TenantId\"\n }\n },\n \"queryTimeout\": \"02:00:00\"\n },\n \"dataset\": {\n \"referenceName\": \"GetSetMetadata\",\n \"type\": \"DatasetReference\",\n \"parameters\": {}\n },\n \"firstRowOnly\": true\n }\n },\n {\n \"name\": \"Get Subscription Id\",\n \"description\": \"Returning the Azure Subscription Id from the metadata properties table.\",\n \"type\": \"Lookup\",\n \"dependsOn\": [],\n \"policy\": {\n \"timeout\": \"7.00:00:00\",\n \"retry\": 0,\n \"retryIntervalInSeconds\": 30,\n \"secureOutput\": false,\n \"secureInput\": true\n },\n \"userProperties\": [],\n \"typeProperties\": {\n \"source\": {\n \"type\": \"AzureSqlSource\",\n \"sqlReaderStoredProcedureName\": \"[[procfwk].[GetPropertyValue]\",\n \"storedProcedureParameters\": {\n \"PropertyName\": {\n \"type\": \"String\",\n \"value\": \"SubscriptionId\"\n }\n },\n \"queryTimeout\": \"02:00:00\"\n },\n \"dataset\": {\n \"referenceName\": \"GetSetMetadata\",\n \"type\": \"DatasetReference\",\n \"parameters\": {}\n }\n }\n }\n ],\n \"folder\": {\n \"name\": \"_ProcFwk\"\n },\n \"annotations\": [\n \"ADF.procfwk\"\n ]\n },\n \"dependsOn\": [\n \"[concat(variables('factoryId'), '/datasets/GetSetMetadata')]\",\n \"[concat(variables('factoryId'), '/linkedServices/SupportDatabase')]\",\n \"[concat(variables('factoryId'), '/pipelines/03 Child')]\"\n ]\n },\n {\n \"name\": \"[concat(parameters('factoryName'), '/03 Child')]\",\n \"type\": \"Microsoft.DataFactory/factories/pipelines\",\n \"apiVersion\": \"2018-06-01\",\n \"properties\": {\n \"description\": \"ADF.procfwk child pipeline used to execute processing pipelines within a given execution stage. This pipeline will be called multi times in parallel.\",\n \"activities\": [\n {\n \"name\": \"Get Pipelines\",\n \"description\": \"Returns all pipelines from the metadata to be executed within a given processing stage.\",\n \"type\": \"Lookup\",\n \"dependsOn\": [],\n \"policy\": {\n \"timeout\": \"7.00:00:00\",\n \"retry\": 0,\n \"retryIntervalInSeconds\": 30,\n \"secureOutput\": false,\n \"secureInput\": false\n },\n \"userProperties\": [],\n \"typeProperties\": {\n \"source\": {\n \"type\": \"AzureSqlSource\",\n \"sqlReaderStoredProcedureName\": \"[[procfwk].[GetPipelinesInStage]\",\n \"storedProcedureParameters\": {\n \"StageId\": {\n \"type\": \"Int32\",\n \"value\": {\n \"value\": \"@pipeline().parameters.StageId\",\n \"type\": \"Expression\"\n }\n }\n },\n \"queryTimeout\": \"02:00:00\"\n },\n \"dataset\": {\n \"referenceName\": \"GetSetMetadata\",\n \"type\": \"DatasetReference\",\n \"parameters\": {}\n },\n \"firstRowOnly\": false\n }\n },\n {\n \"name\": \"Execute Pipelines\",\n \"description\": \"Second level ForEach to run in parallel all pipelines within the stage. Items for iteration passed from the Get Pipelines lookup activity.\",\n \"type\": \"ForEach\",\n \"dependsOn\": [\n {\n \"activity\": \"Get Pipelines\",\n \"dependencyConditions\": [\n \"Succeeded\"\n ]\n }\n ],\n \"userProperties\": [],\n \"typeProperties\": {\n \"items\": {\n \"value\": \"@activity('Get Pipelines').output.value\",\n \"type\": \"Expression\"\n },\n \"activities\": [\n {\n \"name\": \"Execute Pipeline\",\n \"description\": \"The lowest level executor with the metadata framework to call existing processing pipelines within Data Factory. The function called will block processing and wait for an outcome.\",\n \"type\": \"AzureFunctionActivity\",\n \"dependsOn\": [\n {\n \"activity\": \"Log Pipeline Running\",\n \"dependencyConditions\": [\n \"Succeeded\"\n ]\n },\n {\n \"activity\": \"Get Pipeline Params\",\n \"dependencyConditions\": [\n \"Succeeded\"\n ]\n },\n {\n \"activity\": \"Get SPN Details\",\n \"dependencyConditions\": [\n \"Succeeded\"\n ]\n }\n ],\n \"policy\": {\n \"timeout\": \"7.00:00:00\",\n \"retry\": 0,\n \"retryIntervalInSeconds\": 30,\n \"secureOutput\": false,\n \"secureInput\": false\n },\n \"userProperties\": [],\n \"typeProperties\": {\n \"functionName\": \"ExecutePipeline\",\n \"method\": \"POST\",\n \"headers\": {},\n \"body\": {\n \"value\": \"@concat('\\n{\\n\\t\\\"tenantId\\\": \\\"',pipeline().parameters.TenantId,'\\\",\\n\\t\\\"applicationId\\\": \\\"',activity('Get SPN Details').output.firstRow.Id,'\\\",\\n\\t\\\"authenticationKey\\\": \\\"',activity('Get SPN Details').output.firstRow.Secret,'\\\",\\n\\t\\\"subscriptionId\\\": \\\"',pipeline().parameters.SubscriptionId,'\\\",\\n\\t\\\"resourceGroup\\\": \\\"',item().ResourceGroupName,'\\\",\\n\\t\\\"factoryName\\\": \\\"',item().DataFactoryName,'\\\",\\n\\t\\\"pipelineName\\\": \\\"',item().PipelineName,'\\\"',activity('Get Pipeline Params').output.firstRow.Params,'\\n}')\",\n \"type\": \"Expression\"\n }\n },\n \"linkedServiceName\": {\n \"referenceName\": \"PipelineExecutor\",\n \"type\": \"LinkedServiceReference\"\n }\n },\n {\n \"name\": \"Get Pipeline Params\",\n \"description\": \"Returns any parameters from metadata required for the processing pipeline being called. The output can be an empty string if no parameters are required.\",\n \"type\": \"Lookup\",\n \"dependsOn\": [],\n \"policy\": {\n \"timeout\": \"7.00:00:00\",\n \"retry\": 0,\n \"retryIntervalInSeconds\": 30,\n \"secureOutput\": false,\n \"secureInput\": false\n },\n \"userProperties\": [],\n \"typeProperties\": {\n \"source\": {\n \"type\": \"AzureSqlSource\",\n \"sqlReaderStoredProcedureName\": \"[[procfwk].[GetPipelineParameters]\",\n \"storedProcedureParameters\": {\n \"PipelineId\": {\n \"type\": \"Int32\",\n \"value\": {\n \"value\": \"@item().PipelineId\",\n \"type\": \"Expression\"\n }\n }\n },\n \"queryTimeout\": \"02:00:00\"\n },\n \"dataset\": {\n \"referenceName\": \"GetSetMetadata\",\n \"type\": \"DatasetReference\",\n \"parameters\": {}\n }\n }\n },\n {\n \"name\": \"Log Pipeline Running\",\n \"description\": \"Sets the current pipeline with a status of running within the current execution database table.\",\n \"type\": \"SqlServerStoredProcedure\",\n \"dependsOn\": [],\n \"policy\": {\n \"timeout\": \"7.00:00:00\",\n \"retry\": 0,\n \"retryIntervalInSeconds\": 30,\n \"secureOutput\": false,\n \"secureInput\": false\n },\n \"userProperties\": [],\n \"typeProperties\": {\n \"storedProcedureName\": \"[[procfwk].[SetLogPipelineRunning]\",\n \"storedProcedureParameters\": {\n \"ExecutionId\": {\n \"value\": {\n \"value\": \"@pipeline().parameters.ExecutionId\",\n \"type\": \"Expression\"\n },\n \"type\": \"Guid\"\n },\n \"PipelineId\": {\n \"value\": {\n \"value\": \"@item().PipelineId\",\n \"type\": \"Expression\"\n },\n \"type\": \"Int32\"\n },\n \"StageId\": {\n \"value\": {\n \"value\": \"@pipeline().parameters.StageId\",\n \"type\": \"Expression\"\n },\n \"type\": \"Int32\"\n }\n }\n },\n \"linkedServiceName\": {\n \"referenceName\": \"SupportDatabase\",\n \"type\": \"LinkedServiceReference\"\n }\n },\n {\n \"name\": \"Get SPN Details\",\n \"description\": \"Return the SPN ID and Secret for the processing pipeline being executed. Called at this level as each pipeline can have a different SPN.\",\n \"type\": \"Lookup\",\n \"dependsOn\": [],\n \"policy\": {\n \"timeout\": \"7.00:00:00\",\n \"retry\": 0,\n \"retryIntervalInSeconds\": 30,\n \"secureOutput\": false,\n \"secureInput\": false\n },\n \"userProperties\": [],\n \"typeProperties\": {\n \"source\": {\n \"type\": \"AzureSqlSource\",\n \"sqlReaderStoredProcedureName\": \"[[procfwk].[GetServicePrincipal]\",\n \"storedProcedureParameters\": {\n \"DataFactory\": {\n \"type\": \"String\",\n \"value\": {\n \"value\": \"@item().DataFactoryName\",\n \"type\": \"Expression\"\n }\n },\n \"PipelineName\": {\n \"type\": \"String\",\n \"value\": {\n \"value\": \"@item().PipelineName\",\n \"type\": \"Expression\"\n }\n }\n },\n \"queryTimeout\": \"02:00:00\"\n },\n \"dataset\": {\n \"referenceName\": \"GetSetMetadata\",\n \"type\": \"DatasetReference\",\n \"parameters\": {}\n }\n }\n },\n {\n \"name\": \"Handle Function Output\",\n \"description\": \"Receives the outcome from the function execution for a given processing pipeline and updates the current execution table with different pipelines status values depending on the result (case).\",\n \"type\": \"Switch\",\n \"dependsOn\": [\n {\n \"activity\": \"Execute Pipeline\",\n \"dependencyConditions\": [\n \"Succeeded\"\n ]\n }\n ],\n \"userProperties\": [],\n \"typeProperties\": {\n \"on\": {\n \"value\": \"@activity('Execute Pipeline').output.Status\",\n \"type\": \"Expression\"\n },\n \"cases\": [\n {\n \"value\": \"Failed\",\n \"activities\": [\n {\n \"name\": \"Log Pipeline Failure\",\n \"description\": \"Updates the current execution table with a pipeline status of failed if the function outcome is failed. Also blocks pipelines in the downstream execution stage.\",\n \"type\": \"SqlServerStoredProcedure\",\n \"dependsOn\": [],\n \"policy\": {\n \"timeout\": \"7.00:00:00\",\n \"retry\": 0,\n \"retryIntervalInSeconds\": 30,\n \"secureOutput\": false,\n \"secureInput\": false\n },\n \"userProperties\": [],\n \"typeProperties\": {\n \"storedProcedureName\": \"[[procfwk].[SetLogPipelineFailed]\",\n \"storedProcedureParameters\": {\n \"ExecutionId\": {\n \"value\": {\n \"value\": \"@pipeline().parameters.ExecutionId\",\n \"type\": \"Expression\"\n },\n \"type\": \"Guid\"\n },\n \"PipelineId\": {\n \"value\": {\n \"value\": \"@item().PipelineId\",\n \"type\": \"Expression\"\n },\n \"type\": \"Int32\"\n },\n \"RunId\": {\n \"value\": {\n \"value\": \"@activity('Execute Pipeline').output.RunIdUsed\",\n \"type\": \"Expression\"\n },\n \"type\": \"Guid\"\n },\n \"StageId\": {\n \"value\": {\n \"value\": \"@pipeline().parameters.StageId\",\n \"type\": \"Expression\"\n },\n \"type\": \"Int32\"\n }\n }\n },\n \"linkedServiceName\": {\n \"referenceName\": \"SupportDatabase\",\n \"type\": \"LinkedServiceReference\"\n }\n }\n ]\n },\n {\n \"value\": \"Succeeded\",\n \"activities\": [\n {\n \"name\": \"Log Pipeline Success\",\n \"description\": \"Updates the current execution table with a pipeline status of success if the function outcome is succeeded.\",\n \"type\": \"SqlServerStoredProcedure\",\n \"dependsOn\": [],\n \"policy\": {\n \"timeout\": \"7.00:00:00\",\n \"retry\": 0,\n \"retryIntervalInSeconds\": 30,\n \"secureOutput\": false,\n \"secureInput\": false\n },\n \"userProperties\": [],\n \"typeProperties\": {\n \"storedProcedureName\": \"[[procfwk].[SetLogPipelineSuccess]\",\n \"storedProcedureParameters\": {\n \"ExecutionId\": {\n \"value\": {\n \"value\": \"@pipeline().parameters.ExecutionId\",\n \"type\": \"Expression\"\n },\n \"type\": \"Guid\"\n },\n \"PipelineId\": {\n \"value\": {\n \"value\": \"@item().PipelineId\\n\",\n \"type\": \"Expression\"\n },\n \"type\": \"Int32\"\n },\n \"StageId\": {\n \"value\": {\n \"value\": \"@pipeline().parameters.StageId\",\n \"type\": \"Expression\"\n },\n \"type\": \"Int32\"\n }\n }\n },\n \"linkedServiceName\": {\n \"referenceName\": \"SupportDatabase\",\n \"type\": \"LinkedServiceReference\"\n }\n }\n ]\n }\n ],\n \"defaultActivities\": [\n {\n \"name\": \"Log Pipeline Unknown\",\n \"description\": \"Updates the current execution table with a pipeline status of unknown if the function returns an unexpected outcome.\",\n \"type\": \"SqlServerStoredProcedure\",\n \"dependsOn\": [],\n \"policy\": {\n \"timeout\": \"7.00:00:00\",\n \"retry\": 0,\n \"retryIntervalInSeconds\": 30,\n \"secureOutput\": false,\n \"secureInput\": false\n },\n \"userProperties\": [],\n \"typeProperties\": {\n \"storedProcedureName\": \"[[procfwk].[SetLogPipelineUnknown]\",\n \"storedProcedureParameters\": {\n \"ExecutionId\": {\n \"value\": {\n \"value\": \"@pipeline().parameters.ExecutionId\",\n \"type\": \"Expression\"\n },\n \"type\": \"Guid\"\n },\n \"PipelineId\": {\n \"value\": {\n \"value\": \"@item().PipelineId\",\n \"type\": \"Expression\"\n },\n \"type\": \"Int32\"\n },\n \"StageId\": {\n \"value\": {\n \"value\": \"@pipeline().parameters.StageId\",\n \"type\": \"Expression\"\n },\n \"type\": \"Int32\"\n }\n }\n },\n \"linkedServiceName\": {\n \"referenceName\": \"SupportDatabase\",\n \"type\": \"LinkedServiceReference\"\n }\n }\n ]\n }\n },\n {\n \"name\": \"Log Activity Failure\",\n \"description\": \"Handle true failures from calling out to the Azure Function and update the current execution table accordingly so a restart can occur.\",\n \"type\": \"SqlServerStoredProcedure\",\n \"dependsOn\": [\n {\n \"activity\": \"Execute Pipeline\",\n \"dependencyConditions\": [\n \"Failed\"\n ]\n }\n ],\n \"policy\": {\n \"timeout\": \"7.00:00:00\",\n \"retry\": 0,\n \"retryIntervalInSeconds\": 30,\n \"secureOutput\": false,\n \"secureInput\": false\n },\n \"userProperties\": [],\n \"typeProperties\": {\n \"storedProcedureName\": \"[[procfwk].[SetLogActivityFailed]\",\n \"storedProcedureParameters\": {\n \"ExecutionId\": {\n \"value\": {\n \"value\": \"@pipeline().parameters.ExecutionId\",\n \"type\": \"Expression\"\n },\n \"type\": \"Guid\"\n },\n \"PipelineId\": {\n \"value\": {\n \"value\": \"@item().PipelineId\",\n \"type\": \"Expression\"\n },\n \"type\": \"Int32\"\n },\n \"RunId\": {\n \"value\": {\n \"value\": \"@pipeline().RunId\",\n \"type\": \"Expression\"\n },\n \"type\": \"Guid\"\n },\n \"StageId\": {\n \"value\": {\n \"value\": \"@pipeline().parameters.StageId\",\n \"type\": \"Expression\"\n },\n \"type\": \"Int32\"\n }\n }\n },\n \"linkedServiceName\": {\n \"referenceName\": \"SupportDatabase\",\n \"type\": \"LinkedServiceReference\"\n }\n }\n ]\n }\n }\n ],\n \"parameters\": {\n \"StageId\": {\n \"type\": \"int\"\n },\n \"ExecutionId\": {\n \"type\": \"string\"\n },\n \"TenantId\": {\n \"type\": \"string\"\n },\n \"SubscriptionId\": {\n \"type\": \"string\"\n }\n },\n \"variables\": {\n \"FunctionBody\": {\n \"type\": \"String\"\n }\n },\n \"folder\": {\n \"name\": \"_ProcFwk\"\n },\n \"annotations\": [\n \"ADF.procfwk\"\n ]\n },\n \"dependsOn\": [\n \"[concat(variables('factoryId'), '/datasets/GetSetMetadata')]\",\n \"[concat(variables('factoryId'), '/linkedServices/PipelineExecutor')]\",\n \"[concat(variables('factoryId'), '/linkedServices/SupportDatabase')]\"\n ]\n },\n {\n \"name\": \"[concat(parameters('factoryName'), '/Intention Error')]\",\n \"type\": \"Microsoft.DataFactory/factories/pipelines\",\n \"apiVersion\": \"2018-06-01\",\n \"properties\": {\n \"description\": \"Used just so the ADF.procfwk has something to call during development.\",\n \"activities\": [\n {\n \"name\": \"Call Fail Procedure\",\n \"type\": \"SqlServerStoredProcedure\",\n \"dependsOn\": [],\n \"policy\": {\n \"timeout\": \"7.00:00:00\",\n \"retry\": 0,\n \"retryIntervalInSeconds\": 30,\n \"secureOutput\": false,\n \"secureInput\": false\n },\n \"userProperties\": [],\n \"typeProperties\": {\n \"storedProcedureName\": \"[[dbo].[FailProcedure]\"\n },\n \"linkedServiceName\": {\n \"referenceName\": \"SupportDatabase\",\n \"type\": \"LinkedServiceReference\"\n }\n }\n ],\n \"folder\": {\n \"name\": \"Process Pipelines\"\n },\n \"annotations\": []\n },\n \"dependsOn\": [\n \"[concat(variables('factoryId'), '/linkedServices/SupportDatabase')]\"\n ]\n },\n {\n \"name\": \"[concat(parameters('factoryName'), '/Wait 1')]\",\n \"type\": \"Microsoft.DataFactory/factories/pipelines\",\n \"apiVersion\": \"2018-06-01\",\n \"properties\": {\n \"description\": \"Used just so the ADF.procfwk has something to call during development.\",\n \"activities\": [\n {\n \"name\": \"Wait1\",\n \"type\": \"Wait\",\n \"dependsOn\": [],\n \"userProperties\": [],\n \"typeProperties\": {\n \"waitTimeInSeconds\": {\n \"value\": \"@pipeline().parameters.WaitTime\",\n \"type\": \"Expression\"\n }\n }\n }\n ],\n \"parameters\": {\n \"WaitTime\": {\n \"type\": \"int\",\n \"defaultValue\": 5\n }\n },\n \"folder\": {\n \"name\": \"Process Pipelines\"\n },\n \"annotations\": []\n },\n \"dependsOn\": []\n },\n {\n \"name\": \"[concat(parameters('factoryName'), '/Wait 10')]\",\n \"type\": \"Microsoft.DataFactory/factories/pipelines\",\n \"apiVersion\": \"2018-06-01\",\n \"properties\": {\n \"description\": \"Used just so the ADF.procfwk has something to call during development.\",\n \"activities\": [\n {\n \"name\": \"Wait10\",\n \"type\": \"Wait\",\n \"dependsOn\": [],\n \"userProperties\": [],\n \"typeProperties\": {\n \"waitTimeInSeconds\": {\n \"value\": \"@pipeline().parameters.WaitTime\",\n \"type\": \"Expression\"\n }\n }\n }\n ],\n \"parameters\": {\n \"WaitTime\": {\n \"type\": \"int\",\n \"defaultValue\": 5\n }\n },\n \"folder\": {\n \"name\": \"Process Pipelines\"\n },\n \"annotations\": []\n },\n \"dependsOn\": []\n },\n {\n \"name\": \"[concat(parameters('factoryName'), '/Wait 2')]\",\n \"type\": \"Microsoft.DataFactory/factories/pipelines\",\n \"apiVersion\": \"2018-06-01\",\n \"properties\": {\n \"description\": \"Used just so the ADF.procfwk has something to call during development.\",\n \"activities\": [\n {\n \"name\": \"Wait2\",\n \"type\": \"Wait\",\n \"dependsOn\": [],\n \"userProperties\": [],\n \"typeProperties\": {\n \"waitTimeInSeconds\": {\n \"value\": \"@pipeline().parameters.WaitTime\",\n \"type\": \"Expression\"\n }\n }\n }\n ],\n \"parameters\": {\n \"WaitTime\": {\n \"type\": \"int\",\n \"defaultValue\": 5\n }\n },\n \"folder\": {\n \"name\": \"Process Pipelines\"\n },\n \"annotations\": []\n },\n \"dependsOn\": []\n },\n {\n \"name\": \"[concat(parameters('factoryName'), '/Wait 3')]\",\n \"type\": \"Microsoft.DataFactory/factories/pipelines\",\n \"apiVersion\": \"2018-06-01\",\n \"properties\": {\n \"description\": \"Used just so the ADF.procfwk has something to call during development.\",\n \"activities\": [\n {\n \"name\": \"Wait3\",\n \"type\": \"Wait\",\n \"dependsOn\": [],\n \"userProperties\": [],\n \"typeProperties\": {\n \"waitTimeInSeconds\": {\n \"value\": \"@pipeline().parameters.WaitTime\",\n \"type\": \"Expression\"\n }\n }\n }\n ],\n \"parameters\": {\n \"WaitTime\": {\n \"type\": \"int\",\n \"defaultValue\": 5\n }\n },\n \"folder\": {\n \"name\": \"Process Pipelines\"\n },\n \"annotations\": []\n },\n \"dependsOn\": []\n },\n {\n \"name\": \"[concat(parameters('factoryName'), '/Wait 4')]\",\n \"type\": \"Microsoft.DataFactory/factories/pipelines\",\n \"apiVersion\": \"2018-06-01\",\n \"properties\": {\n \"description\": \"Used just so the ADF.procfwk has something to call during development.\",\n \"activities\": [\n {\n \"name\": \"Wait4\",\n \"type\": \"Wait\",\n \"dependsOn\": [],\n \"userProperties\": [],\n \"typeProperties\": {\n \"waitTimeInSeconds\": {\n \"value\": \"@pipeline().parameters.WaitTime\",\n \"type\": \"Expression\"\n }\n }\n }\n ],\n \"parameters\": {\n \"WaitTime\": {\n \"type\": \"int\",\n \"defaultValue\": 5\n }\n },\n \"folder\": {\n \"name\": \"Process Pipelines\"\n },\n \"annotations\": []\n },\n \"dependsOn\": []\n },\n {\n \"name\": \"[concat(parameters('factoryName'), '/Wait 5')]\",\n \"type\": \"Microsoft.DataFactory/factories/pipelines\",\n \"apiVersion\": \"2018-06-01\",\n \"properties\": {\n \"description\": \"Used just so the ADF.procfwk has something to call during development.\",\n \"activities\": [\n {\n \"name\": \"Wait5\",\n \"type\": \"Wait\",\n \"dependsOn\": [],\n \"userProperties\": [],\n \"typeProperties\": {\n \"waitTimeInSeconds\": {\n \"value\": \"@pipeline().parameters.WaitTime\",\n \"type\": \"Expression\"\n }\n }\n }\n ],\n \"parameters\": {\n \"WaitTime\": {\n \"type\": \"int\",\n \"defaultValue\": 5\n }\n },\n \"folder\": {\n \"name\": \"Process Pipelines\"\n },\n \"annotations\": []\n },\n \"dependsOn\": []\n },\n {\n \"name\": \"[concat(parameters('factoryName'), '/Wait 6')]\",\n \"type\": \"Microsoft.DataFactory/factories/pipelines\",\n \"apiVersion\": \"2018-06-01\",\n \"properties\": {\n \"description\": \"Used just so the ADF.procfwk has something to call during development.\",\n \"activities\": [\n {\n \"name\": \"Wait6\",\n \"type\": \"Wait\",\n \"dependsOn\": [],\n \"userProperties\": [],\n \"typeProperties\": {\n \"waitTimeInSeconds\": {\n \"value\": \"@pipeline().parameters.WaitTime\",\n \"type\": \"Expression\"\n }\n }\n }\n ],\n \"parameters\": {\n \"WaitTime\": {\n \"type\": \"int\",\n \"defaultValue\": 5\n }\n },\n \"folder\": {\n \"name\": \"Process Pipelines\"\n },\n \"annotations\": []\n },\n \"dependsOn\": []\n },\n {\n \"name\": \"[concat(parameters('factoryName'), '/Wait 7')]\",\n \"type\": \"Microsoft.DataFactory/factories/pipelines\",\n \"apiVersion\": \"2018-06-01\",\n \"properties\": {\n \"description\": \"Used just so the ADF.procfwk has something to call during development.\",\n \"activities\": [\n {\n \"name\": \"Wait7\",\n \"type\": \"Wait\",\n \"dependsOn\": [],\n \"userProperties\": [],\n \"typeProperties\": {\n \"waitTimeInSeconds\": {\n \"value\": \"@pipeline().parameters.WaitTime\",\n \"type\": \"Expression\"\n }\n }\n }\n ],\n \"parameters\": {\n \"WaitTime\": {\n \"type\": \"int\",\n \"defaultValue\": 5\n }\n },\n \"folder\": {\n \"name\": \"Process Pipelines\"\n },\n \"annotations\": []\n },\n \"dependsOn\": []\n },\n {\n \"name\": \"[concat(parameters('factoryName'), '/Wait 8')]\",\n \"type\": \"Microsoft.DataFactory/factories/pipelines\",\n \"apiVersion\": \"2018-06-01\",\n \"properties\": {\n \"description\": \"Used just so the ADF.procfwk has something to call during development.\",\n \"activities\": [\n {\n \"name\": \"Wait8\",\n \"type\": \"Wait\",\n \"dependsOn\": [],\n \"userProperties\": [],\n \"typeProperties\": {\n \"waitTimeInSeconds\": {\n \"value\": \"@pipeline().parameters.WaitTime\",\n \"type\": \"Expression\"\n }\n }\n }\n ],\n \"parameters\": {\n \"WaitTime\": {\n \"type\": \"int\",\n \"defaultValue\": 5\n }\n },\n \"folder\": {\n \"name\": \"Process Pipelines\"\n },\n \"annotations\": []\n },\n \"dependsOn\": []\n },\n {\n \"name\": \"[concat(parameters('factoryName'), '/Wait 9')]\",\n \"type\": \"Microsoft.DataFactory/factories/pipelines\",\n \"apiVersion\": \"2018-06-01\",\n \"properties\": {\n \"description\": \"Used just so the ADF.procfwk has something to call during development.\",\n \"activities\": [\n {\n \"name\": \"Wait9\",\n \"type\": \"Wait\",\n \"dependsOn\": [],\n \"userProperties\": [],\n \"typeProperties\": {\n \"waitTimeInSeconds\": {\n \"value\": \"@pipeline().parameters.WaitTime\",\n \"type\": \"Expression\"\n }\n }\n }\n ],\n \"parameters\": {\n \"WaitTime\": {\n \"type\": \"int\",\n \"defaultValue\": 5\n }\n },\n \"folder\": {\n \"name\": \"Process Pipelines\"\n },\n \"annotations\": []\n },\n \"dependsOn\": []\n },\n {\n \"name\": \"[concat(parameters('factoryName'), '/GetSetMetadata')]\",\n \"type\": \"Microsoft.DataFactory/factories/datasets\",\n \"apiVersion\": \"2018-06-01\",\n \"properties\": {\n \"linkedServiceName\": {\n \"referenceName\": \"SupportDatabase\",\n \"type\": \"LinkedServiceReference\"\n },\n \"annotations\": [\n \"ADF.procfwk\"\n ],\n \"type\": \"AzureSqlTable\",\n \"schema\": [],\n \"typeProperties\": {}\n },\n \"dependsOn\": [\n \"[concat(variables('factoryId'), '/linkedServices/SupportDatabase')]\"\n ]\n },\n {\n \"name\": \"[concat(parameters('factoryName'), '/Keys')]\",\n \"type\": \"Microsoft.DataFactory/factories/linkedServices\",\n \"apiVersion\": \"2018-06-01\",\n \"properties\": {\n \"description\": \"Connection to Key Vault for all other ADF linked service credentials required to run the processing framework.\",\n \"annotations\": [\n \"ADF.procfwk\"\n ],\n \"type\": \"AzureKeyVault\",\n \"typeProperties\": {\n \"baseUrl\": \"[parameters('Keys_properties_typeProperties_baseUrl')]\"\n }\n },\n \"dependsOn\": []\n },\n {\n \"name\": \"[concat(parameters('factoryName'), '/PipelineExecutor')]\",\n \"type\": \"Microsoft.DataFactory/factories/linkedServices\",\n \"apiVersion\": \"2018-06-01\",\n \"properties\": {\n \"description\": \"Connection to the Function App from ADF for calling the processing pipeline function within the orchestration framework.\",\n \"annotations\": [\n \"ADF.procfwk\"\n ],\n \"type\": \"AzureFunction\",\n \"typeProperties\": {\n \"functionAppUrl\": \"https://pipelineexecutor.azurewebsites.net\",\n \"functionKey\": {\n \"type\": \"AzureKeyVaultSecret\",\n \"store\": {\n \"referenceName\": \"Keys\",\n \"type\": \"LinkedServiceReference\"\n },\n \"secretName\": \"ExecutorFunctionAppKey\"\n }\n }\n },\n \"dependsOn\": [\n \"[concat(variables('factoryId'), '/linkedServices/Keys')]\"\n ]\n },\n {\n \"name\": \"[concat(parameters('factoryName'), '/SupportDatabase')]\",\n \"type\": \"Microsoft.DataFactory/factories/linkedServices\",\n \"apiVersion\": \"2018-06-01\",\n \"properties\": {\n \"description\": \"Connection between ADF and processing framework metadata SQLDB.\",\n \"annotations\": [\n \"ADF.procfwk\"\n ],\n \"type\": \"AzureSqlDatabase\",\n \"typeProperties\": {\n \"connectionString\": {\n \"type\": \"AzureKeyVaultSecret\",\n \"store\": {\n \"referenceName\": \"Keys\",\n \"type\": \"LinkedServiceReference\"\n },\n \"secretName\": \"[parameters('SupportDatabase_properties_typeProperties_connectionString_secretName')]\"\n }\n }\n },\n \"dependsOn\": [\n \"[concat(variables('factoryId'), '/linkedServices/Keys')]\"\n ]\n }\n ]\n}"
},
{
"path": "ARM Templates/Data Factory/v1.3 Export.json",
"content": "{\n \"$schema\": \"http://schema.management.azure.com/schemas/2015-01-01/deploymentTemplate.json#\",\n \"contentVersion\": \"1.0.0.0\",\n \"parameters\": {\n \"factoryName\": {\n \"type\": \"string\",\n \"metadata\": \"Data Factory name\",\n \"defaultValue\": \"\"\n },\n \"Keys_properties_typeProperties_baseUrl\": {\n \"type\": \"string\",\n \"defaultValue\": \"\"\n },\n \"SupportDatabase_properties_typeProperties_connectionString_secretName\": {\n \"type\": \"string\",\n \"defaultValue\": \"\"\n }\n },\n \"variables\": {\n \"factoryId\": \"[concat('Microsoft.DataFactory/factories/', parameters('factoryName'))]\"\n },\n \"resources\": [\n {\n \"name\": \"[concat(parameters('factoryName'), '/01-Grandparent')]\",\n \"type\": \"Microsoft.DataFactory/factories/pipelines\",\n \"apiVersion\": \"2018-06-01\",\n \"properties\": {\n \"description\": \"ADF.procfwk grandparent pipeline used optionally to bootstrap any wider processes in your Data Factory that then calls the processing framework.\",\n \"activities\": [\n {\n \"name\": \"Framework Processing\",\n \"type\": \"ExecutePipeline\",\n \"dependsOn\": [\n {\n \"activity\": \"Set Random Waits\",\n \"dependencyConditions\": [\n \"Succeeded\"\n ]\n }\n ],\n \"userProperties\": [],\n \"typeProperties\": {\n \"pipeline\": {\n \"referenceName\": \"02-Parent\",\n \"type\": \"PipelineReference\"\n },\n \"waitOnCompletion\": true,\n \"parameters\": {}\n }\n },\n {\n \"name\": \"Set Random Waits\",\n \"description\": \"For functional testing only.\",\n \"type\": \"SqlServerStoredProcedure\",\n \"dependsOn\": [],\n \"policy\": {\n \"timeout\": \"7.00:00:00\",\n \"retry\": 0,\n \"retryIntervalInSeconds\": 30,\n \"secureOutput\": false,\n \"secureInput\": false\n },\n \"userProperties\": [],\n \"typeProperties\": {\n \"storedProcedureName\": \"[[dbo].[SetRandomWaitValues]\"\n },\n \"linkedServiceName\": {\n \"referenceName\": \"SupportDatabase\",\n \"type\": \"LinkedServiceReference\"\n }\n }\n ],\n \"folder\": {\n \"name\": \"_ProcFwk\"\n },\n \"annotations\": [\n \"ADF.procfwk\"\n ]\n },\n \"dependsOn\": [\n \"[concat(variables('factoryId'), '/pipelines/02-Parent')]\",\n \"[concat(variables('factoryId'), '/linkedServices/SupportDatabase')]\"\n ]\n },\n {\n \"name\": \"[concat(parameters('factoryName'), '/02-Parent')]\",\n \"type\": \"Microsoft.DataFactory/factories/pipelines\",\n \"apiVersion\": \"2018-06-01\",\n \"properties\": {\n \"description\": \"ADF.procfwk parent pipeline used to bootstrap the orchestration framework in perform the first level ForEach calls in sequence for the metadata stages.\",\n \"activities\": [\n {\n \"name\": \"Get Stages\",\n \"description\": \"Returns a distinct list of execution stages within the framework metadata.\",\n \"type\": \"Lookup\",\n \"dependsOn\": [\n {\n \"activity\": \"Execution Wrapper\",\n \"dependencyConditions\": [\n \"Succeeded\"\n ]\n }\n ],\n \"policy\": {\n \"timeout\": \"7.00:00:00\",\n \"retry\": 0,\n \"retryIntervalInSeconds\": 30,\n \"secureOutput\": false,\n \"secureInput\": false\n },\n \"userProperties\": [],\n \"typeProperties\": {\n \"source\": {\n \"type\": \"AzureSqlSource\",\n \"sqlReaderStoredProcedureName\": \"[[procfwk].[GetStages]\",\n \"storedProcedureParameters\": {\n \"ExecutionId\": {\n \"type\": \"Guid\",\n \"value\": {\n \"value\": \"@activity('Execution Wrapper').output.firstRow.ExecutionId\",\n \"type\": \"Expression\"\n }\n }\n },\n \"queryTimeout\": \"02:00:00\"\n },\n \"dataset\": {\n \"referenceName\": \"GetSetMetadata\",\n \"type\": \"DatasetReference\",\n \"parameters\": {}\n },\n \"firstRowOnly\": false\n }\n },\n {\n \"name\": \"Execute Stages\",\n \"description\": \"Top level ForEach to sequentially call all processing stages within the framework metadata. Items for iteration passed from the Get Stages lookup activity.\",\n \"type\": \"ForEach\",\n \"dependsOn\": [\n {\n \"activity\": \"Get Stages\",\n \"dependencyConditions\": [\n \"Succeeded\"\n ]\n },\n {\n \"activity\": \"Get Tenant Id\",\n \"dependencyConditions\": [\n \"Succeeded\"\n ]\n },\n {\n \"activity\": \"Get Subscription Id\",\n \"dependencyConditions\": [\n \"Succeeded\"\n ]\n }\n ],\n \"userProperties\": [],\n \"typeProperties\": {\n \"items\": {\n \"value\": \"@activity('Get Stages').output.value\",\n \"type\": \"Expression\"\n },\n \"isSequential\": true,\n \"activities\": [\n {\n \"name\": \"Stage Executor\",\n \"description\": \"Call to the framework generic child pipeline for a given execution stage.\",\n \"type\": \"ExecutePipeline\",\n \"dependsOn\": [\n {\n \"activity\": \"Log Stage Preparing\",\n \"dependencyConditions\": [\n \"Succeeded\"\n ]\n }\n ],\n \"userProperties\": [],\n \"typeProperties\": {\n \"pipeline\": {\n \"referenceName\": \"03-Child\",\n \"type\": \"PipelineReference\"\n },\n \"waitOnCompletion\": true,\n \"parameters\": {\n \"StageId\": {\n \"value\": \"@item().StageId\",\n \"type\": \"Expression\"\n },\n \"ExecutionId\": {\n \"value\": \"@activity('Execution Wrapper').output.firstRow.ExecutionId\",\n \"type\": \"Expression\"\n },\n \"TenantId\": {\n \"value\": \"@activity('Get Tenant Id').output.firstRow.PropertyValue\",\n \"type\": \"Expression\"\n },\n \"SubscriptionId\": {\n \"value\": \"@activity('Get Subscription Id').output.firstRow.PropertyValue\",\n \"type\": \"Expression\"\n }\n }\n }\n },\n {\n \"name\": \"Log Stage Preparing\",\n \"description\": \"Update the current execution table flagging all pipelines within the stage as preparing.\",\n \"type\": \"SqlServerStoredProcedure\",\n \"dependsOn\": [\n {\n \"activity\": \"Check for Blockers\",\n \"dependencyConditions\": [\n \"Succeeded\"\n ]\n }\n ],\n \"policy\": {\n \"timeout\": \"7.00:00:00\",\n \"retry\": 0,\n \"retryIntervalInSeconds\": 30,\n \"secureOutput\": false,\n \"secureInput\": false\n },\n \"userProperties\": [],\n \"typeProperties\": {\n \"storedProcedureName\": \"[[procfwk].[SetLogStagePreparing]\",\n \"storedProcedureParameters\": {\n \"ExecutionId\": {\n \"value\": {\n \"value\": \"@activity('Execution Wrapper').output.firstRow.ExecutionId\",\n \"type\": \"Expression\"\n },\n \"type\": \"Guid\"\n },\n \"StageId\": {\n \"value\": {\n \"value\": \"@item().StageId\",\n \"type\": \"Expression\"\n },\n \"type\": \"Int32\"\n }\n }\n },\n \"linkedServiceName\": {\n \"referenceName\": \"SupportDatabase\",\n \"type\": \"LinkedServiceReference\"\n }\n },\n {\n \"name\": \"Check for Blockers\",\n \"description\": \"Used to double check and stop the next execution stage if failures and blockers have be incurred. Without this step processing would continue regardless of upstream failures.\",\n \"type\": \"SqlServerStoredProcedure\",\n \"dependsOn\": [],\n \"policy\": {\n \"timeout\": \"7.00:00:00\",\n \"retry\": 0,\n \"retryIntervalInSeconds\": 30,\n \"secureOutput\": false,\n \"secureInput\": false\n },\n \"userProperties\": [],\n \"typeProperties\": {\n \"storedProcedureName\": \"[[procfwk].[CheckForBlockedPipelines]\",\n \"storedProcedureParameters\": {\n \"StageId\": {\n \"value\": {\n \"value\": \"@item().StageId\",\n \"type\": \"Expression\"\n },\n \"type\": \"Int32\"\n }\n }\n },\n \"linkedServiceName\": {\n \"referenceName\": \"SupportDatabase\",\n \"type\": \"LinkedServiceReference\"\n }\n }\n ]\n }\n },\n {\n \"name\": \"Execution Wrapper\",\n \"description\": \"Wrapper to reset and restart processing or create a completely new execution instance of the framework metadata.\",\n \"type\": \"Lookup\",\n \"dependsOn\": [\n {\n \"activity\": \"Metadata Integrity Checks\",\n \"dependencyConditions\": [\n \"Succeeded\"\n ]\n }\n ],\n \"policy\": {\n \"timeout\": \"7.00:00:00\",\n \"retry\": 0,\n \"retryIntervalInSeconds\": 30,\n \"secureOutput\": false,\n \"secureInput\": false\n },\n \"userProperties\": [],\n \"typeProperties\": {\n \"source\": {\n \"type\": \"AzureSqlSource\",\n \"sqlReaderStoredProcedureName\": \"[[procfwk].[ExecutionWrapper]\",\n \"storedProcedureParameters\": {\n \"CallingDataFactory\": {\n \"type\": \"String\",\n \"value\": {\n \"value\": \"@pipeline().DataFactory\",\n \"type\": \"Expression\"\n }\n }\n },\n \"queryTimeout\": \"02:00:00\"\n },\n \"dataset\": {\n \"referenceName\": \"GetSetMetadata\",\n \"type\": \"DatasetReference\",\n \"parameters\": {}\n }\n }\n },\n {\n \"name\": \"Archive Execution Log\",\n \"description\": \"After a successful execution run the current execution metadata is moved to the long term logging table by this stored procedure call.\",\n \"type\": \"SqlServerStoredProcedure\",\n \"dependsOn\": [\n {\n \"activity\": \"Execute Stages\",\n \"dependencyConditions\": [\n \"Succeeded\"\n ]\n }\n ],\n \"policy\": {\n \"timeout\": \"7.00:00:00\",\n \"retry\": 0,\n \"retryIntervalInSeconds\": 30,\n \"secureOutput\": false,\n \"secureInput\": false\n },\n \"userProperties\": [],\n \"typeProperties\": {\n \"storedProcedureName\": \"[[procfwk].[UpdateExecutionLog]\"\n },\n \"linkedServiceName\": {\n \"referenceName\": \"SupportDatabase\",\n \"type\": \"LinkedServiceReference\"\n }\n },\n {\n \"name\": \"Get Tenant Id\",\n \"description\": \"Returning the Azure Tenant Id from the metadata properties table.\",\n \"type\": \"Lookup\",\n \"dependsOn\": [\n {\n \"activity\": \"Metadata Integrity Checks\",\n \"dependencyConditions\": [\n \"Succeeded\"\n ]\n }\n ],\n \"policy\": {\n \"timeout\": \"7.00:00:00\",\n \"retry\": 0,\n \"retryIntervalInSeconds\": 30,\n \"secureOutput\": false,\n \"secureInput\": false\n },\n \"userProperties\": [],\n \"typeProperties\": {\n \"source\": {\n \"type\": \"AzureSqlSource\",\n \"sqlReaderStoredProcedureName\": \"[[procfwk].[GetPropertyValue]\",\n \"storedProcedureParameters\": {\n \"PropertyName\": {\n \"type\": \"String\",\n \"value\": \"TenantId\"\n }\n },\n \"queryTimeout\": \"02:00:00\"\n },\n \"dataset\": {\n \"referenceName\": \"GetSetMetadata\",\n \"type\": \"DatasetReference\",\n \"parameters\": {}\n },\n \"firstRowOnly\": true\n }\n },\n {\n \"name\": \"Get Subscription Id\",\n \"description\": \"Returning the Azure Subscription Id from the metadata properties table.\",\n \"type\": \"Lookup\",\n \"dependsOn\": [\n {\n \"activity\": \"Metadata Integrity Checks\",\n \"dependencyConditions\": [\n \"Succeeded\"\n ]\n }\n ],\n \"policy\": {\n \"timeout\": \"7.00:00:00\",\n \"retry\": 0,\n \"retryIntervalInSeconds\": 30,\n \"secureOutput\": false,\n \"secureInput\": true\n },\n \"userProperties\": [],\n \"typeProperties\": {\n \"source\": {\n \"type\": \"AzureSqlSource\",\n \"sqlReaderStoredProcedureName\": \"[[procfwk].[GetPropertyValue]\",\n \"storedProcedureParameters\": {\n \"PropertyName\": {\n \"type\": \"String\",\n \"value\": \"SubscriptionId\"\n }\n },\n \"queryTimeout\": \"02:00:00\"\n },\n \"dataset\": {\n \"referenceName\": \"GetSetMetadata\",\n \"type\": \"DatasetReference\",\n \"parameters\": {}\n }\n }\n },\n {\n \"name\": \"Metadata Integrity Checks\",\n \"description\": \"Performs a series of checks on all metadata held in the framework SQLDB. This is intended to raise errors before an execution run even starts.\",\n \"type\": \"SqlServerStoredProcedure\",\n \"dependsOn\": [],\n \"policy\": {\n \"timeout\": \"7.00:00:00\",\n \"retry\": 0,\n \"retryIntervalInSeconds\": 30,\n \"secureOutput\": false,\n \"secureInput\": false\n },\n \"userProperties\": [],\n \"typeProperties\": {\n \"storedProcedureName\": \"[[procfwk].[CheckMetadataIntegrity]\",\n \"storedProcedureParameters\": {\n \"DebugMode\": {\n \"value\": \"false\",\n \"type\": \"Boolean\"\n }\n }\n },\n \"linkedServiceName\": {\n \"referenceName\": \"SupportDatabase\",\n \"type\": \"LinkedServiceReference\"\n }\n }\n ],\n \"folder\": {\n \"name\": \"_ProcFwk\"\n },\n \"annotations\": [\n \"ADF.procfwk\"\n ]\n },\n \"dependsOn\": [\n \"[concat(variables('factoryId'), '/datasets/GetSetMetadata')]\",\n \"[concat(variables('factoryId'), '/linkedServices/SupportDatabase')]\",\n \"[concat(variables('factoryId'), '/pipelines/03-Child')]\"\n ]\n },\n {\n \"name\": \"[concat(parameters('factoryName'), '/03-Child')]\",\n \"type\": \"Microsoft.DataFactory/factories/pipelines\",\n \"apiVersion\": \"2018-06-01\",\n \"properties\": {\n \"description\": \"ADF.procfwk child pipeline used to execute processing pipelines within a given execution stage. This pipeline will be called multi times in parallel.\",\n \"activities\": [\n {\n \"name\": \"Get Pipelines\",\n \"description\": \"Returns all pipelines from the metadata to be executed within a given processing stage.\",\n \"type\": \"Lookup\",\n \"dependsOn\": [],\n \"policy\": {\n \"timeout\": \"7.00:00:00\",\n \"retry\": 0,\n \"retryIntervalInSeconds\": 30,\n \"secureOutput\": false,\n \"secureInput\": false\n },\n \"userProperties\": [],\n \"typeProperties\": {\n \"source\": {\n \"type\": \"AzureSqlSource\",\n \"sqlReaderStoredProcedureName\": \"[[procfwk].[GetPipelinesInStage]\",\n \"storedProcedureParameters\": {\n \"StageId\": {\n \"type\": \"Int32\",\n \"value\": {\n \"value\": \"@pipeline().parameters.StageId\",\n \"type\": \"Expression\"\n }\n }\n },\n \"queryTimeout\": \"02:00:00\"\n },\n \"dataset\": {\n \"referenceName\": \"GetSetMetadata\",\n \"type\": \"DatasetReference\",\n \"parameters\": {}\n },\n \"firstRowOnly\": false\n }\n },\n {\n \"name\": \"Execute Pipelines\",\n \"description\": \"Second level ForEach to run in parallel all pipelines within the stage. Items for iteration passed from the Get Pipelines lookup activity.\",\n \"type\": \"ForEach\",\n \"dependsOn\": [\n {\n \"activity\": \"Get Pipelines\",\n \"dependencyConditions\": [\n \"Succeeded\"\n ]\n }\n ],\n \"userProperties\": [],\n \"typeProperties\": {\n \"items\": {\n \"value\": \"@activity('Get Pipelines').output.value\",\n \"type\": \"Expression\"\n },\n \"activities\": [\n {\n \"name\": \"Execute Pipeline\",\n \"description\": \"The lowest level executor with the metadata framework to call existing processing pipelines within Data Factory. The function called will block processing and wait for an outcome.\",\n \"type\": \"AzureFunctionActivity\",\n \"dependsOn\": [\n {\n \"activity\": \"Log Pipeline Running\",\n \"dependencyConditions\": [\n \"Succeeded\"\n ]\n },\n {\n \"activity\": \"Get Pipeline Params\",\n \"dependencyConditions\": [\n \"Succeeded\"\n ]\n },\n {\n \"activity\": \"Get SPN Details\",\n \"dependencyConditions\": [\n \"Succeeded\"\n ]\n }\n ],\n \"policy\": {\n \"timeout\": \"7.00:00:00\",\n \"retry\": 0,\n \"retryIntervalInSeconds\": 30,\n \"secureOutput\": false,\n \"secureInput\": false\n },\n \"userProperties\": [],\n \"typeProperties\": {\n \"functionName\": \"ExecutePipeline\",\n \"method\": \"POST\",\n \"headers\": {},\n \"body\": {\n \"value\": \"@concat('\\n{\\n\\t\\\"tenantId\\\": \\\"',pipeline().parameters.TenantId,'\\\",\\n\\t\\\"applicationId\\\": \\\"',activity('Get SPN Details').output.firstRow.Id,'\\\",\\n\\t\\\"authenticationKey\\\": \\\"',activity('Get SPN Details').output.firstRow.Secret,'\\\",\\n\\t\\\"subscriptionId\\\": \\\"',pipeline().parameters.SubscriptionId,'\\\",\\n\\t\\\"resourceGroup\\\": \\\"',item().ResourceGroupName,'\\\",\\n\\t\\\"factoryName\\\": \\\"',item().DataFactoryName,'\\\",\\n\\t\\\"pipelineName\\\": \\\"',item().PipelineName,'\\\"',activity('Get Pipeline Params').output.firstRow.Params,'\\n}')\",\n \"type\": \"Expression\"\n }\n },\n \"linkedServiceName\": {\n \"referenceName\": \"PipelineExecutor\",\n \"type\": \"LinkedServiceReference\"\n }\n },\n {\n \"name\": \"Get Pipeline Params\",\n \"description\": \"Returns any parameters from metadata required for the processing pipeline being called. The output can be an empty string if no parameters are required.\",\n \"type\": \"Lookup\",\n \"dependsOn\": [],\n \"policy\": {\n \"timeout\": \"7.00:00:00\",\n \"retry\": 0,\n \"retryIntervalInSeconds\": 30,\n \"secureOutput\": false,\n \"secureInput\": false\n },\n \"userProperties\": [],\n \"typeProperties\": {\n \"source\": {\n \"type\": \"AzureSqlSource\",\n \"sqlReaderStoredProcedureName\": \"[[procfwk].[GetPipelineParameters]\",\n \"storedProcedureParameters\": {\n \"PipelineId\": {\n \"type\": \"Int32\",\n \"value\": {\n \"value\": \"@item().PipelineId\",\n \"type\": \"Expression\"\n }\n }\n },\n \"queryTimeout\": \"02:00:00\"\n },\n \"dataset\": {\n \"referenceName\": \"GetSetMetadata\",\n \"type\": \"DatasetReference\",\n \"parameters\": {}\n }\n }\n },\n {\n \"name\": \"Log Pipeline Running\",\n \"description\": \"Sets the current pipeline with a status of running within the current execution database table.\",\n \"type\": \"SqlServerStoredProcedure\",\n \"dependsOn\": [],\n \"policy\": {\n \"timeout\": \"7.00:00:00\",\n \"retry\": 0,\n \"retryIntervalInSeconds\": 30,\n \"secureOutput\": false,\n \"secureInput\": false\n },\n \"userProperties\": [],\n \"typeProperties\": {\n \"storedProcedureName\": \"[[procfwk].[SetLogPipelineRunning]\",\n \"storedProcedureParameters\": {\n \"ExecutionId\": {\n \"value\": {\n \"value\": \"@pipeline().parameters.ExecutionId\",\n \"type\": \"Expression\"\n },\n \"type\": \"Guid\"\n },\n \"PipelineId\": {\n \"value\": {\n \"value\": \"@item().PipelineId\",\n \"type\": \"Expression\"\n },\n \"type\": \"Int32\"\n },\n \"StageId\": {\n \"value\": {\n \"value\": \"@pipeline().parameters.StageId\",\n \"type\": \"Expression\"\n },\n \"type\": \"Int32\"\n }\n }\n },\n \"linkedServiceName\": {\n \"referenceName\": \"SupportDatabase\",\n \"type\": \"LinkedServiceReference\"\n }\n },\n {\n \"name\": \"Get SPN Details\",\n \"description\": \"Return the SPN ID and Secret for the processing pipeline being executed. Called at this level as each pipeline can have a different SPN.\",\n \"type\": \"Lookup\",\n \"dependsOn\": [],\n \"policy\": {\n \"timeout\": \"7.00:00:00\",\n \"retry\": 0,\n \"retryIntervalInSeconds\": 30,\n \"secureOutput\": false,\n \"secureInput\": false\n },\n \"userProperties\": [],\n \"typeProperties\": {\n \"source\": {\n \"type\": \"AzureSqlSource\",\n \"sqlReaderStoredProcedureName\": \"[[procfwk].[GetServicePrincipal]\",\n \"storedProcedureParameters\": {\n \"DataFactory\": {\n \"type\": \"String\",\n \"value\": {\n \"value\": \"@item().DataFactoryName\",\n \"type\": \"Expression\"\n }\n },\n \"PipelineName\": {\n \"type\": \"String\",\n \"value\": {\n \"value\": \"@item().PipelineName\",\n \"type\": \"Expression\"\n }\n }\n },\n \"queryTimeout\": \"02:00:00\"\n },\n \"dataset\": {\n \"referenceName\": \"GetSetMetadata\",\n \"type\": \"DatasetReference\",\n \"parameters\": {}\n }\n }\n },\n {\n \"name\": \"Handle Function Output\",\n \"description\": \"Receives the outcome from the function execution for a given processing pipeline and updates the current execution table with different pipelines status values depending on the result (case).\",\n \"type\": \"Switch\",\n \"dependsOn\": [\n {\n \"activity\": \"Execute Pipeline\",\n \"dependencyConditions\": [\n \"Succeeded\"\n ]\n }\n ],\n \"userProperties\": [],\n \"typeProperties\": {\n \"on\": {\n \"value\": \"@activity('Execute Pipeline').output.Status\",\n \"type\": \"Expression\"\n },\n \"cases\": [\n {\n \"value\": \"Failed\",\n \"activities\": [\n {\n \"name\": \"Log Pipeline Failure\",\n \"description\": \"Updates the current execution table with a pipeline status of failed if the function outcome is failed. Also blocks pipelines in the downstream execution stage.\",\n \"type\": \"SqlServerStoredProcedure\",\n \"dependsOn\": [],\n \"policy\": {\n \"timeout\": \"7.00:00:00\",\n \"retry\": 0,\n \"retryIntervalInSeconds\": 30,\n \"secureOutput\": false,\n \"secureInput\": false\n },\n \"userProperties\": [],\n \"typeProperties\": {\n \"storedProcedureName\": \"[[procfwk].[SetLogPipelineFailed]\",\n \"storedProcedureParameters\": {\n \"ExecutionId\": {\n \"value\": {\n \"value\": \"@pipeline().parameters.ExecutionId\",\n \"type\": \"Expression\"\n },\n \"type\": \"Guid\"\n },\n \"PipelineId\": {\n \"value\": {\n \"value\": \"@item().PipelineId\",\n \"type\": \"Expression\"\n },\n \"type\": \"Int32\"\n },\n \"RunId\": {\n \"value\": {\n \"value\": \"@activity('Execute Pipeline').output.RunIdUsed\",\n \"type\": \"Expression\"\n },\n \"type\": \"Guid\"\n },\n \"StageId\": {\n \"value\": {\n \"value\": \"@pipeline().parameters.StageId\",\n \"type\": \"Expression\"\n },\n \"type\": \"Int32\"\n }\n }\n },\n \"linkedServiceName\": {\n \"referenceName\": \"SupportDatabase\",\n \"type\": \"LinkedServiceReference\"\n }\n }\n ]\n },\n {\n \"value\": \"Succeeded\",\n \"activities\": [\n {\n \"name\": \"Log Pipeline Success\",\n \"description\": \"Updates the current execution table with a pipeline status of success if the function outcome is succeeded.\",\n \"type\": \"SqlServerStoredProcedure\",\n \"dependsOn\": [],\n \"policy\": {\n \"timeout\": \"7.00:00:00\",\n \"retry\": 0,\n \"retryIntervalInSeconds\": 30,\n \"secureOutput\": false,\n \"secureInput\": false\n },\n \"userProperties\": [],\n \"typeProperties\": {\n \"storedProcedureName\": \"[[procfwk].[SetLogPipelineSuccess]\",\n \"storedProcedureParameters\": {\n \"ExecutionId\": {\n \"value\": {\n \"value\": \"@pipeline().parameters.ExecutionId\",\n \"type\": \"Expression\"\n },\n \"type\": \"Guid\"\n },\n \"PipelineId\": {\n \"value\": {\n \"value\": \"@item().PipelineId\\n\",\n \"type\": \"Expression\"\n },\n \"type\": \"Int32\"\n },\n \"StageId\": {\n \"value\": {\n \"value\": \"@pipeline().parameters.StageId\",\n \"type\": \"Expression\"\n },\n \"type\": \"Int32\"\n }\n }\n },\n \"linkedServiceName\": {\n \"referenceName\": \"SupportDatabase\",\n \"type\": \"LinkedServiceReference\"\n }\n }\n ]\n }\n ],\n \"defaultActivities\": [\n {\n \"name\": \"Log Pipeline Unknown\",\n \"description\": \"Updates the current execution table with a pipeline status of unknown if the function returns an unexpected outcome.\",\n \"type\": \"SqlServerStoredProcedure\",\n \"dependsOn\": [],\n \"policy\": {\n \"timeout\": \"7.00:00:00\",\n \"retry\": 0,\n \"retryIntervalInSeconds\": 30,\n \"secureOutput\": false,\n \"secureInput\": false\n },\n \"userProperties\": [],\n \"typeProperties\": {\n \"storedProcedureName\": \"[[procfwk].[SetLogPipelineUnknown]\",\n \"storedProcedureParameters\": {\n \"ExecutionId\": {\n \"value\": {\n \"value\": \"@pipeline().parameters.ExecutionId\",\n \"type\": \"Expression\"\n },\n \"type\": \"Guid\"\n },\n \"PipelineId\": {\n \"value\": {\n \"value\": \"@item().PipelineId\",\n \"type\": \"Expression\"\n },\n \"type\": \"Int32\"\n },\n \"StageId\": {\n \"value\": {\n \"value\": \"@pipeline().parameters.StageId\",\n \"type\": \"Expression\"\n },\n \"type\": \"Int32\"\n }\n }\n },\n \"linkedServiceName\": {\n \"referenceName\": \"SupportDatabase\",\n \"type\": \"LinkedServiceReference\"\n }\n }\n ]\n }\n },\n {\n \"name\": \"Log Activity Failure\",\n \"description\": \"Handle true failures from calling out to the Azure Function and update the current execution table accordingly so a restart can occur.\",\n \"type\": \"SqlServerStoredProcedure\",\n \"dependsOn\": [\n {\n \"activity\": \"Execute Pipeline\",\n \"dependencyConditions\": [\n \"Failed\"\n ]\n }\n ],\n \"policy\": {\n \"timeout\": \"7.00:00:00\",\n \"retry\": 0,\n \"retryIntervalInSeconds\": 30,\n \"secureOutput\": false,\n \"secureInput\": false\n },\n \"userProperties\": [],\n \"typeProperties\": {\n \"storedProcedureName\": \"[[procfwk].[SetLogActivityFailed]\",\n \"storedProcedureParameters\": {\n \"ExecutionId\": {\n \"value\": {\n \"value\": \"@pipeline().parameters.ExecutionId\",\n \"type\": \"Expression\"\n },\n \"type\": \"Guid\"\n },\n \"PipelineId\": {\n \"value\": {\n \"value\": \"@item().PipelineId\",\n \"type\": \"Expression\"\n },\n \"type\": \"Int32\"\n },\n \"RunId\": {\n \"value\": {\n \"value\": \"@pipeline().RunId\",\n \"type\": \"Expression\"\n },\n \"type\": \"Guid\"\n },\n \"StageId\": {\n \"value\": {\n \"value\": \"@pipeline().parameters.StageId\",\n \"type\": \"Expression\"\n },\n \"type\": \"Int32\"\n }\n }\n },\n \"linkedServiceName\": {\n \"referenceName\": \"SupportDatabase\",\n \"type\": \"LinkedServiceReference\"\n }\n }\n ]\n }\n }\n ],\n \"parameters\": {\n \"StageId\": {\n \"type\": \"int\"\n },\n \"ExecutionId\": {\n \"type\": \"string\"\n },\n \"TenantId\": {\n \"type\": \"string\"\n },\n \"SubscriptionId\": {\n \"type\": \"string\"\n }\n },\n \"variables\": {\n \"FunctionBody\": {\n \"type\": \"String\"\n }\n },\n \"folder\": {\n \"name\": \"_ProcFwk\"\n },\n \"annotations\": [\n \"ADF.procfwk\"\n ]\n },\n \"dependsOn\": [\n \"[concat(variables('factoryId'), '/datasets/GetSetMetadata')]\",\n \"[concat(variables('factoryId'), '/linkedServices/PipelineExecutor')]\",\n \"[concat(variables('factoryId'), '/linkedServices/SupportDatabase')]\"\n ]\n },\n {\n \"name\": \"[concat(parameters('factoryName'), '/Intention Error')]\",\n \"type\": \"Microsoft.DataFactory/factories/pipelines\",\n \"apiVersion\": \"2018-06-01\",\n \"properties\": {\n \"description\": \"Used just so the ADF.procfwk has something to call during development.\",\n \"activities\": [\n {\n \"name\": \"Call Fail Procedure\",\n \"type\": \"SqlServerStoredProcedure\",\n \"dependsOn\": [],\n \"policy\": {\n \"timeout\": \"7.00:00:00\",\n \"retry\": 0,\n \"retryIntervalInSeconds\": 30,\n \"secureOutput\": false,\n \"secureInput\": false\n },\n \"userProperties\": [],\n \"typeProperties\": {\n \"storedProcedureName\": \"[[dbo].[FailProcedure]\"\n },\n \"linkedServiceName\": {\n \"referenceName\": \"SupportDatabase\",\n \"type\": \"LinkedServiceReference\"\n }\n }\n ],\n \"folder\": {\n \"name\": \"Processes\"\n },\n \"annotations\": []\n },\n \"dependsOn\": [\n \"[concat(variables('factoryId'), '/linkedServices/SupportDatabase')]\"\n ]\n },\n {\n \"name\": \"[concat(parameters('factoryName'), '/Wait 1')]\",\n \"type\": \"Microsoft.DataFactory/factories/pipelines\",\n \"apiVersion\": \"2018-06-01\",\n \"properties\": {\n \"description\": \"Used just so the ADF.procfwk has something to call during development.\",\n \"activities\": [\n {\n \"name\": \"Wait1\",\n \"type\": \"Wait\",\n \"dependsOn\": [],\n \"userProperties\": [],\n \"typeProperties\": {\n \"waitTimeInSeconds\": {\n \"value\": \"@pipeline().parameters.WaitTime\",\n \"type\": \"Expression\"\n }\n }\n }\n ],\n \"parameters\": {\n \"WaitTime\": {\n \"type\": \"int\",\n \"defaultValue\": 5\n }\n },\n \"folder\": {\n \"name\": \"Processes\"\n },\n \"annotations\": []\n },\n \"dependsOn\": []\n },\n {\n \"name\": \"[concat(parameters('factoryName'), '/Wait 10')]\",\n \"type\": \"Microsoft.DataFactory/factories/pipelines\",\n \"apiVersion\": \"2018-06-01\",\n \"properties\": {\n \"description\": \"Used just so the ADF.procfwk has something to call during development.\",\n \"activities\": [\n {\n \"name\": \"Wait10\",\n \"type\": \"Wait\",\n \"dependsOn\": [],\n \"userProperties\": [],\n \"typeProperties\": {\n \"waitTimeInSeconds\": {\n \"value\": \"@pipeline().parameters.WaitTime\",\n \"type\": \"Expression\"\n }\n }\n }\n ],\n \"parameters\": {\n \"WaitTime\": {\n \"type\": \"int\",\n \"defaultValue\": 5\n }\n },\n \"folder\": {\n \"name\": \"Processes\"\n },\n \"annotations\": []\n },\n \"dependsOn\": []\n },\n {\n \"name\": \"[concat(parameters('factoryName'), '/Wait 2')]\",\n \"type\": \"Microsoft.DataFactory/factories/pipelines\",\n \"apiVersion\": \"2018-06-01\",\n \"properties\": {\n \"description\": \"Used just so the ADF.procfwk has something to call during development.\",\n \"activities\": [\n {\n \"name\": \"Wait2\",\n \"type\": \"Wait\",\n \"dependsOn\": [],\n \"userProperties\": [],\n \"typeProperties\": {\n \"waitTimeInSeconds\": {\n \"value\": \"@pipeline().parameters.WaitTime\",\n \"type\": \"Expression\"\n }\n }\n }\n ],\n \"parameters\": {\n \"WaitTime\": {\n \"type\": \"int\",\n \"defaultValue\": 5\n }\n },\n \"folder\": {\n \"name\": \"Processes\"\n },\n \"annotations\": []\n },\n \"dependsOn\": []\n },\n {\n \"name\": \"[concat(parameters('factoryName'), '/Wait 3')]\",\n \"type\": \"Microsoft.DataFactory/factories/pipelines\",\n \"apiVersion\": \"2018-06-01\",\n \"properties\": {\n \"description\": \"Used just so the ADF.procfwk has something to call during development.\",\n \"activities\": [\n {\n \"name\": \"Wait3\",\n \"type\": \"Wait\",\n \"dependsOn\": [],\n \"userProperties\": [],\n \"typeProperties\": {\n \"waitTimeInSeconds\": {\n \"value\": \"@pipeline().parameters.WaitTime\",\n \"type\": \"Expression\"\n }\n }\n }\n ],\n \"parameters\": {\n \"WaitTime\": {\n \"type\": \"int\",\n \"defaultValue\": 5\n }\n },\n \"folder\": {\n \"name\": \"Processes\"\n },\n \"annotations\": []\n },\n \"dependsOn\": []\n },\n {\n \"name\": \"[concat(parameters('factoryName'), '/Wait 4')]\",\n \"type\": \"Microsoft.DataFactory/factories/pipelines\",\n \"apiVersion\": \"2018-06-01\",\n \"properties\": {\n \"description\": \"Used just so the ADF.procfwk has something to call during development.\",\n \"activities\": [\n {\n \"name\": \"Wait4\",\n \"type\": \"Wait\",\n \"dependsOn\": [],\n \"userProperties\": [],\n \"typeProperties\": {\n \"waitTimeInSeconds\": {\n \"value\": \"@pipeline().parameters.WaitTime\",\n \"type\": \"Expression\"\n }\n }\n }\n ],\n \"parameters\": {\n \"WaitTime\": {\n \"type\": \"int\",\n \"defaultValue\": 5\n }\n },\n \"folder\": {\n \"name\": \"Processes\"\n },\n \"annotations\": []\n },\n \"dependsOn\": []\n },\n {\n \"name\": \"[concat(parameters('factoryName'), '/Wait 5')]\",\n \"type\": \"Microsoft.DataFactory/factories/pipelines\",\n \"apiVersion\": \"2018-06-01\",\n \"properties\": {\n \"description\": \"Used just so the ADF.procfwk has something to call during development.\",\n \"activities\": [\n {\n \"name\": \"Wait5\",\n \"type\": \"Wait\",\n \"dependsOn\": [],\n \"userProperties\": [],\n \"typeProperties\": {\n \"waitTimeInSeconds\": {\n \"value\": \"@pipeline().parameters.WaitTime\",\n \"type\": \"Expression\"\n }\n }\n }\n ],\n \"parameters\": {\n \"WaitTime\": {\n \"type\": \"int\",\n \"defaultValue\": 5\n }\n },\n \"folder\": {\n \"name\": \"Processes\"\n },\n \"annotations\": []\n },\n \"dependsOn\": []\n },\n {\n \"name\": \"[concat(parameters('factoryName'), '/Wait 6')]\",\n \"type\": \"Microsoft.DataFactory/factories/pipelines\",\n \"apiVersion\": \"2018-06-01\",\n \"properties\": {\n \"description\": \"Used just so the ADF.procfwk has something to call during development.\",\n \"activities\": [\n {\n \"name\": \"Wait6\",\n \"type\": \"Wait\",\n \"dependsOn\": [],\n \"userProperties\": [],\n \"typeProperties\": {\n \"waitTimeInSeconds\": {\n \"value\": \"@pipeline().parameters.WaitTime\",\n \"type\": \"Expression\"\n }\n }\n }\n ],\n \"parameters\": {\n \"WaitTime\": {\n \"type\": \"int\",\n \"defaultValue\": 5\n }\n },\n \"folder\": {\n \"name\": \"Processes\"\n },\n \"annotations\": []\n },\n \"dependsOn\": []\n },\n {\n \"name\": \"[concat(parameters('factoryName'), '/Wait 7')]\",\n \"type\": \"Microsoft.DataFactory/factories/pipelines\",\n \"apiVersion\": \"2018-06-01\",\n \"properties\": {\n \"description\": \"Used just so the ADF.procfwk has something to call during development.\",\n \"activities\": [\n {\n \"name\": \"Wait7\",\n \"type\": \"Wait\",\n \"dependsOn\": [],\n \"userProperties\": [],\n \"typeProperties\": {\n \"waitTimeInSeconds\": {\n \"value\": \"@pipeline().parameters.WaitTime\",\n \"type\": \"Expression\"\n }\n }\n }\n ],\n \"parameters\": {\n \"WaitTime\": {\n \"type\": \"int\",\n \"defaultValue\": 5\n }\n },\n \"folder\": {\n \"name\": \"Processes\"\n },\n \"annotations\": []\n },\n \"dependsOn\": []\n },\n {\n \"name\": \"[concat(parameters('factoryName'), '/Wait 8')]\",\n \"type\": \"Microsoft.DataFactory/factories/pipelines\",\n \"apiVersion\": \"2018-06-01\",\n \"properties\": {\n \"description\": \"Used just so the ADF.procfwk has something to call during development.\",\n \"activities\": [\n {\n \"name\": \"Wait8\",\n \"type\": \"Wait\",\n \"dependsOn\": [],\n \"userProperties\": [],\n \"typeProperties\": {\n \"waitTimeInSeconds\": {\n \"value\": \"@pipeline().parameters.WaitTime\",\n \"type\": \"Expression\"\n }\n }\n }\n ],\n \"parameters\": {\n \"WaitTime\": {\n \"type\": \"int\",\n \"defaultValue\": 5\n }\n },\n \"folder\": {\n \"name\": \"Processes\"\n },\n \"annotations\": []\n },\n \"dependsOn\": []\n },\n {\n \"name\": \"[concat(parameters('factoryName'), '/Wait 9')]\",\n \"type\": \"Microsoft.DataFactory/factories/pipelines\",\n \"apiVersion\": \"2018-06-01\",\n \"properties\": {\n \"description\": \"Used just so the ADF.procfwk has something to call during development.\",\n \"activities\": [\n {\n \"name\": \"Wait9\",\n \"type\": \"Wait\",\n \"dependsOn\": [],\n \"userProperties\": [],\n \"typeProperties\": {\n \"waitTimeInSeconds\": {\n \"value\": \"@pipeline().parameters.WaitTime\",\n \"type\": \"Expression\"\n }\n }\n }\n ],\n \"parameters\": {\n \"WaitTime\": {\n \"type\": \"int\",\n \"defaultValue\": 5\n }\n },\n \"folder\": {\n \"name\": \"Processes\"\n },\n \"annotations\": []\n },\n \"dependsOn\": []\n },\n {\n \"name\": \"[concat(parameters('factoryName'), '/GetSetMetadata')]\",\n \"type\": \"Microsoft.DataFactory/factories/datasets\",\n \"apiVersion\": \"2018-06-01\",\n \"properties\": {\n \"linkedServiceName\": {\n \"referenceName\": \"SupportDatabase\",\n \"type\": \"LinkedServiceReference\"\n },\n \"annotations\": [\n \"ADF.procfwk\"\n ],\n \"type\": \"AzureSqlTable\",\n \"schema\": [],\n \"typeProperties\": {}\n },\n \"dependsOn\": [\n \"[concat(variables('factoryId'), '/linkedServices/SupportDatabase')]\"\n ]\n },\n {\n \"name\": \"[concat(parameters('factoryName'), '/Keys')]\",\n \"type\": \"Microsoft.DataFactory/factories/linkedServices\",\n \"apiVersion\": \"2018-06-01\",\n \"properties\": {\n \"description\": \"Connection to Key Vault for all other ADF linked service credentials required to run the processing framework.\",\n \"annotations\": [\n \"ADF.procfwk\"\n ],\n \"type\": \"AzureKeyVault\",\n \"typeProperties\": {\n \"baseUrl\": \"[parameters('Keys_properties_typeProperties_baseUrl')]\"\n }\n },\n \"dependsOn\": []\n },\n {\n \"name\": \"[concat(parameters('factoryName'), '/PipelineExecutor')]\",\n \"type\": \"Microsoft.DataFactory/factories/linkedServices\",\n \"apiVersion\": \"2018-06-01\",\n \"properties\": {\n \"description\": \"Connection to the Function App from ADF for calling the processing pipeline function within the orchestration framework.\",\n \"annotations\": [\n \"ADF.procfwk\"\n ],\n \"type\": \"AzureFunction\",\n \"typeProperties\": {\n \"functionAppUrl\": \"https://pipelineexecutor.azurewebsites.net\",\n \"functionKey\": {\n \"type\": \"AzureKeyVaultSecret\",\n \"store\": {\n \"referenceName\": \"Keys\",\n \"type\": \"LinkedServiceReference\"\n },\n \"secretName\": \"ExecutorFunctionAppKey\"\n }\n }\n },\n \"dependsOn\": [\n \"[concat(variables('factoryId'), '/linkedServices/Keys')]\"\n ]\n },\n {\n \"name\": \"[concat(parameters('factoryName'), '/SupportDatabase')]\",\n \"type\": \"Microsoft.DataFactory/factories/linkedServices\",\n \"apiVersion\": \"2018-06-01\",\n \"properties\": {\n \"description\": \"Connection between ADF and processing framework metadata SQLDB.\",\n \"annotations\": [\n \"ADF.procfwk\"\n ],\n \"type\": \"AzureSqlDatabase\",\n \"typeProperties\": {\n \"connectionString\": {\n \"type\": \"AzureKeyVaultSecret\",\n \"store\": {\n \"referenceName\": \"Keys\",\n \"type\": \"LinkedServiceReference\"\n },\n \"secretName\": \"[parameters('SupportDatabase_properties_typeProperties_connectionString_secretName')]\"\n }\n }\n },\n \"dependsOn\": [\n \"[concat(variables('factoryId'), '/linkedServices/Keys')]\"\n ]\n },\n {\n \"name\": \"[concat(parameters('factoryName'), '/FunctionalTestingTrigger')]\",\n \"type\": \"Microsoft.DataFactory/factories/triggers\",\n \"apiVersion\": \"2018-06-01\",\n \"properties\": {\n \"description\": \"Used internally to run the processing framework hourly.\",\n \"annotations\": [\n \"ADF.procfwk\"\n ],\n \"runtimeState\": \"Stopped\",\n \"pipelines\": [\n {\n \"pipelineReference\": {\n \"referenceName\": \"01-Grandparent\",\n \"type\": \"PipelineReference\"\n },\n \"parameters\": {}\n }\n ],\n \"type\": \"ScheduleTrigger\",\n \"typeProperties\": {\n \"recurrence\": {\n \"frequency\": \"Hour\",\n \"interval\": 1,\n \"startTime\": \"2020-04-06T15:00:00.000Z\",\n \"timeZone\": \"UTC\"\n }\n }\n },\n \"dependsOn\": [\n \"[concat(variables('factoryId'), '/pipelines/01-Grandparent')]\"\n ]\n }\n ]\n}"
},
{
"path": "ARM Templates/Data Factory/v1.4 Export.json",
"content": "{\n \"$schema\": \"http://schema.management.azure.com/schemas/2015-01-01/deploymentTemplate.json#\",\n \"contentVersion\": \"1.0.0.0\",\n \"parameters\": {\n \"factoryName\": {\n \"type\": \"string\",\n \"metadata\": \"Data Factory name\",\n \"defaultValue\": \"\"\n },\n \"Keys_properties_typeProperties_baseUrl\": {\n \"type\": \"string\",\n \"defaultValue\": \"\"\n },\n \"SupportDatabase_properties_typeProperties_connectionString_secretName\": {\n \"type\": \"string\",\n \"defaultValue\": \"\"\n }\n },\n \"variables\": {\n \"factoryId\": \"[concat('Microsoft.DataFactory/factories/', parameters('factoryName'))]\"\n },\n \"resources\": [\n {\n \"name\": \"[concat(parameters('factoryName'), '/01-Grandparent')]\",\n \"type\": \"Microsoft.DataFactory/factories/pipelines\",\n \"apiVersion\": \"2018-06-01\",\n \"properties\": {\n \"description\": \"ADF.procfwk grandparent pipeline used optionally to bootstrap any wider processes in your Data Factory that then calls the processing framework.\",\n \"activities\": [\n {\n \"name\": \"Framework Processing\",\n \"type\": \"ExecutePipeline\",\n \"dependsOn\": [\n {\n \"activity\": \"Set Random Waits\",\n \"dependencyConditions\": [\n \"Succeeded\"\n ]\n }\n ],\n \"userProperties\": [],\n \"typeProperties\": {\n \"pipeline\": {\n \"referenceName\": \"02-Parent\",\n \"type\": \"PipelineReference\"\n },\n \"waitOnCompletion\": true,\n \"parameters\": {}\n }\n },\n {\n \"name\": \"Set Random Waits\",\n \"description\": \"For functional testing only.\",\n \"type\": \"SqlServerStoredProcedure\",\n \"dependsOn\": [],\n \"policy\": {\n \"timeout\": \"7.00:00:00\",\n \"retry\": 0,\n \"retryIntervalInSeconds\": 30,\n \"secureOutput\": false,\n \"secureInput\": false\n },\n \"userProperties\": [],\n \"typeProperties\": {\n \"storedProcedureName\": \"[[dbo].[SetRandomWaitValues]\"\n },\n \"linkedServiceName\": {\n \"referenceName\": \"SupportDatabase\",\n \"type\": \"LinkedServiceReference\"\n }\n }\n ],\n \"folder\": {\n \"name\": \"_ProcFwk\"\n },\n \"annotations\": [\n \"ADF.procfwk\"\n ]\n },\n \"dependsOn\": [\n \"[concat(variables('factoryId'), '/pipelines/02-Parent')]\",\n \"[concat(variables('factoryId'), '/linkedServices/SupportDatabase')]\"\n ]\n },\n {\n \"name\": \"[concat(parameters('factoryName'), '/02-Parent')]\",\n \"type\": \"Microsoft.DataFactory/factories/pipelines\",\n \"apiVersion\": \"2018-06-01\",\n \"properties\": {\n \"description\": \"ADF.procfwk parent pipeline used to bootstrap the orchestration framework in perform the first level ForEach calls in sequence for the metadata stages.\",\n \"activities\": [\n {\n \"name\": \"Get Stages\",\n \"description\": \"Returns a distinct list of execution stages within the framework metadata.\",\n \"type\": \"Lookup\",\n \"dependsOn\": [\n {\n \"activity\": \"Execution Wrapper\",\n \"dependencyConditions\": [\n \"Succeeded\"\n ]\n }\n ],\n \"policy\": {\n \"timeout\": \"7.00:00:00\",\n \"retry\": 0,\n \"retryIntervalInSeconds\": 30,\n \"secureOutput\": false,\n \"secureInput\": false\n },\n \"userProperties\": [],\n \"typeProperties\": {\n \"source\": {\n \"type\": \"AzureSqlSource\",\n \"sqlReaderStoredProcedureName\": \"[[procfwk].[GetStages]\",\n \"storedProcedureParameters\": {\n \"ExecutionId\": {\n \"type\": \"Guid\",\n \"value\": {\n \"value\": \"@activity('Execution Wrapper').output.firstRow.ExecutionId\",\n \"type\": \"Expression\"\n }\n }\n },\n \"queryTimeout\": \"02:00:00\"\n },\n \"dataset\": {\n \"referenceName\": \"GetSetMetadata\",\n \"type\": \"DatasetReference\",\n \"parameters\": {}\n },\n \"firstRowOnly\": false\n }\n },\n {\n \"name\": \"Execute Stages\",\n \"description\": \"Top level ForEach to sequentially call all processing stages within the framework metadata. Items for iteration passed from the Get Stages lookup activity.\",\n \"type\": \"ForEach\",\n \"dependsOn\": [\n {\n \"activity\": \"Get Stages\",\n \"dependencyConditions\": [\n \"Succeeded\"\n ]\n },\n {\n \"activity\": \"Get Tenant Id\",\n \"dependencyConditions\": [\n \"Succeeded\"\n ]\n },\n {\n \"activity\": \"Get Subscription Id\",\n \"dependencyConditions\": [\n \"Succeeded\"\n ]\n }\n ],\n \"userProperties\": [],\n \"typeProperties\": {\n \"items\": {\n \"value\": \"@activity('Get Stages').output.value\",\n \"type\": \"Expression\"\n },\n \"isSequential\": true,\n \"activities\": [\n {\n \"name\": \"Stage Executor\",\n \"description\": \"Call to the framework generic child pipeline for a given execution stage.\",\n \"type\": \"ExecutePipeline\",\n \"dependsOn\": [\n {\n \"activity\": \"Log Stage Preparing\",\n \"dependencyConditions\": [\n \"Succeeded\"\n ]\n }\n ],\n \"userProperties\": [],\n \"typeProperties\": {\n \"pipeline\": {\n \"referenceName\": \"03-Child\",\n \"type\": \"PipelineReference\"\n },\n \"waitOnCompletion\": true,\n \"parameters\": {\n \"StageId\": {\n \"value\": \"@item().StageId\",\n \"type\": \"Expression\"\n },\n \"ExecutionId\": {\n \"value\": \"@activity('Execution Wrapper').output.firstRow.ExecutionId\",\n \"type\": \"Expression\"\n },\n \"TenantId\": {\n \"value\": \"@activity('Get Tenant Id').output.firstRow.PropertyValue\",\n \"type\": \"Expression\"\n },\n \"SubscriptionId\": {\n \"value\": \"@activity('Get Subscription Id').output.firstRow.PropertyValue\",\n \"type\": \"Expression\"\n }\n }\n }\n },\n {\n \"name\": \"Log Stage Preparing\",\n \"description\": \"Update the current execution table flagging all pipelines within the stage as preparing.\",\n \"type\": \"SqlServerStoredProcedure\",\n \"dependsOn\": [\n {\n \"activity\": \"Check for Blockers\",\n \"dependencyConditions\": [\n \"Succeeded\"\n ]\n }\n ],\n \"policy\": {\n \"timeout\": \"7.00:00:00\",\n \"retry\": 0,\n \"retryIntervalInSeconds\": 30,\n \"secureOutput\": false,\n \"secureInput\": false\n },\n \"userProperties\": [],\n \"typeProperties\": {\n \"storedProcedureName\": \"[[procfwk].[SetLogStagePreparing]\",\n \"storedProcedureParameters\": {\n \"ExecutionId\": {\n \"value\": {\n \"value\": \"@activity('Execution Wrapper').output.firstRow.ExecutionId\",\n \"type\": \"Expression\"\n },\n \"type\": \"Guid\"\n },\n \"StageId\": {\n \"value\": {\n \"value\": \"@item().StageId\",\n \"type\": \"Expression\"\n },\n \"type\": \"Int32\"\n }\n }\n },\n \"linkedServiceName\": {\n \"referenceName\": \"SupportDatabase\",\n \"type\": \"LinkedServiceReference\"\n }\n },\n {\n \"name\": \"Check for Blockers\",\n \"description\": \"Used to double check and stop the next execution stage if failures and blockers have be incurred. Without this step processing would continue regardless of upstream failures.\",\n \"type\": \"SqlServerStoredProcedure\",\n \"dependsOn\": [],\n \"policy\": {\n \"timeout\": \"7.00:00:00\",\n \"retry\": 0,\n \"retryIntervalInSeconds\": 30,\n \"secureOutput\": false,\n \"secureInput\": false\n },\n \"userProperties\": [],\n \"typeProperties\": {\n \"storedProcedureName\": \"[[procfwk].[CheckForBlockedPipelines]\",\n \"storedProcedureParameters\": {\n \"StageId\": {\n \"value\": {\n \"value\": \"@item().StageId\",\n \"type\": \"Expression\"\n },\n \"type\": \"Int32\"\n }\n }\n },\n \"linkedServiceName\": {\n \"referenceName\": \"SupportDatabase\",\n \"type\": \"LinkedServiceReference\"\n }\n }\n ]\n }\n },\n {\n \"name\": \"Execution Wrapper\",\n \"description\": \"Wrapper to reset and restart processing or create a completely new execution instance of the framework metadata.\",\n \"type\": \"Lookup\",\n \"dependsOn\": [\n {\n \"activity\": \"Metadata Integrity Checks\",\n \"dependencyConditions\": [\n \"Succeeded\"\n ]\n }\n ],\n \"policy\": {\n \"timeout\": \"7.00:00:00\",\n \"retry\": 0,\n \"retryIntervalInSeconds\": 30,\n \"secureOutput\": false,\n \"secureInput\": false\n },\n \"userProperties\": [],\n \"typeProperties\": {\n \"source\": {\n \"type\": \"AzureSqlSource\",\n \"sqlReaderStoredProcedureName\": \"[[procfwk].[ExecutionWrapper]\",\n \"storedProcedureParameters\": {\n \"CallingDataFactory\": {\n \"type\": \"String\",\n \"value\": {\n \"value\": \"@pipeline().DataFactory\",\n \"type\": \"Expression\"\n }\n }\n },\n \"queryTimeout\": \"02:00:00\"\n },\n \"dataset\": {\n \"referenceName\": \"GetSetMetadata\",\n \"type\": \"DatasetReference\",\n \"parameters\": {}\n }\n }\n },\n {\n \"name\": \"Archive Execution Log\",\n \"description\": \"After a successful execution run the current execution metadata is moved to the long term logging table by this stored procedure call.\",\n \"type\": \"SqlServerStoredProcedure\",\n \"dependsOn\": [\n {\n \"activity\": \"Execute Stages\",\n \"dependencyConditions\": [\n \"Succeeded\"\n ]\n }\n ],\n \"policy\": {\n \"timeout\": \"7.00:00:00\",\n \"retry\": 0,\n \"retryIntervalInSeconds\": 30,\n \"secureOutput\": false,\n \"secureInput\": false\n },\n \"userProperties\": [],\n \"typeProperties\": {\n \"storedProcedureName\": \"[[procfwk].[UpdateExecutionLog]\"\n },\n \"linkedServiceName\": {\n \"referenceName\": \"SupportDatabase\",\n \"type\": \"LinkedServiceReference\"\n }\n },\n {\n \"name\": \"Get Tenant Id\",\n \"description\": \"Returning the Azure Tenant Id from the metadata properties table.\",\n \"type\": \"Lookup\",\n \"dependsOn\": [\n {\n \"activity\": \"Metadata Integrity Checks\",\n \"dependencyConditions\": [\n \"Succeeded\"\n ]\n }\n ],\n \"policy\": {\n \"timeout\": \"7.00:00:00\",\n \"retry\": 0,\n \"retryIntervalInSeconds\": 30,\n \"secureOutput\": false,\n \"secureInput\": false\n },\n \"userProperties\": [],\n \"typeProperties\": {\n \"source\": {\n \"type\": \"AzureSqlSource\",\n \"sqlReaderStoredProcedureName\": \"[[procfwk].[GetPropertyValue]\",\n \"storedProcedureParameters\": {\n \"PropertyName\": {\n \"type\": \"String\",\n \"value\": \"TenantId\"\n }\n },\n \"queryTimeout\": \"02:00:00\"\n },\n \"dataset\": {\n \"referenceName\": \"GetSetMetadata\",\n \"type\": \"DatasetReference\",\n \"parameters\": {}\n },\n \"firstRowOnly\": true\n }\n },\n {\n \"name\": \"Get Subscription Id\",\n \"description\": \"Returning the Azure Subscription Id from the metadata properties table.\",\n \"type\": \"Lookup\",\n \"dependsOn\": [\n {\n \"activity\": \"Metadata Integrity Checks\",\n \"dependencyConditions\": [\n \"Succeeded\"\n ]\n }\n ],\n \"policy\": {\n \"timeout\": \"7.00:00:00\",\n \"retry\": 0,\n \"retryIntervalInSeconds\": 30,\n \"secureOutput\": false,\n \"secureInput\": true\n },\n \"userProperties\": [],\n \"typeProperties\": {\n \"source\": {\n \"type\": \"AzureSqlSource\",\n \"sqlReaderStoredProcedureName\": \"[[procfwk].[GetPropertyValue]\",\n \"storedProcedureParameters\": {\n \"PropertyName\": {\n \"type\": \"String\",\n \"value\": \"SubscriptionId\"\n }\n },\n \"queryTimeout\": \"02:00:00\"\n },\n \"dataset\": {\n \"referenceName\": \"GetSetMetadata\",\n \"type\": \"DatasetReference\",\n \"parameters\": {}\n }\n }\n },\n {\n \"name\": \"Metadata Integrity Checks\",\n \"description\": \"Performs a series of checks on all metadata held in the framework SQLDB. This is intended to raise errors before an execution run even starts.\",\n \"type\": \"SqlServerStoredProcedure\",\n \"dependsOn\": [],\n \"policy\": {\n \"timeout\": \"7.00:00:00\",\n \"retry\": 0,\n \"retryIntervalInSeconds\": 30,\n \"secureOutput\": false,\n \"secureInput\": false\n },\n \"userProperties\": [],\n \"typeProperties\": {\n \"storedProcedureName\": \"[[procfwk].[CheckMetadataIntegrity]\",\n \"storedProcedureParameters\": {\n \"DebugMode\": {\n \"value\": \"false\",\n \"type\": \"Boolean\"\n }\n }\n },\n \"linkedServiceName\": {\n \"referenceName\": \"SupportDatabase\",\n \"type\": \"LinkedServiceReference\"\n }\n }\n ],\n \"folder\": {\n \"name\": \"_ProcFwk\"\n },\n \"annotations\": [\n \"ADF.procfwk\"\n ]\n },\n \"dependsOn\": [\n \"[concat(variables('factoryId'), '/datasets/GetSetMetadata')]\",\n \"[concat(variables('factoryId'), '/linkedServices/SupportDatabase')]\",\n \"[concat(variables('factoryId'), '/pipelines/03-Child')]\"\n ]\n },\n {\n \"name\": \"[concat(parameters('factoryName'), '/03-Child')]\",\n \"type\": \"Microsoft.DataFactory/factories/pipelines\",\n \"apiVersion\": \"2018-06-01\",\n \"properties\": {\n \"description\": \"ADF.procfwk child pipeline used to execute Worker pipelines within a given execution stage. This pipeline will be called once for each stage, then execute all Workers in parallel.\",\n \"activities\": [\n {\n \"name\": \"Get Pipelines\",\n \"description\": \"Returns all pipelines from the metadata to be executed within a given processing stage.\",\n \"type\": \"Lookup\",\n \"dependsOn\": [],\n \"policy\": {\n \"timeout\": \"7.00:00:00\",\n \"retry\": 0,\n \"retryIntervalInSeconds\": 30,\n \"secureOutput\": false,\n \"secureInput\": false\n },\n \"userProperties\": [],\n \"typeProperties\": {\n \"source\": {\n \"type\": \"AzureSqlSource\",\n \"sqlReaderStoredProcedureName\": \"[[procfwk].[GetPipelinesInStage]\",\n \"storedProcedureParameters\": {\n \"StageId\": {\n \"type\": \"Int32\",\n \"value\": {\n \"value\": \"@pipeline().parameters.StageId\",\n \"type\": \"Expression\"\n }\n }\n },\n \"queryTimeout\": \"02:00:00\"\n },\n \"dataset\": {\n \"referenceName\": \"GetSetMetadata\",\n \"type\": \"DatasetReference\",\n \"parameters\": {}\n },\n \"firstRowOnly\": false\n }\n },\n {\n \"name\": \"Execute Pipelines\",\n \"description\": \"Second level ForEach to run in parallel all pipelines within the stage. Items for iteration passed from the Get Pipelines lookup activity.\",\n \"type\": \"ForEach\",\n \"dependsOn\": [\n {\n \"activity\": \"Get Pipelines\",\n \"dependencyConditions\": [\n \"Succeeded\"\n ]\n }\n ],\n \"userProperties\": [],\n \"typeProperties\": {\n \"items\": {\n \"value\": \"@activity('Get Pipelines').output.value\",\n \"type\": \"Expression\"\n },\n \"isSequential\": false,\n \"activities\": [\n {\n \"name\": \"Execute Pipeline\",\n \"description\": \"The lowest level executor with the metadata framework to call existing processing pipelines within Data Factory. The function called will block processing and wait for an outcome.\",\n \"type\": \"AzureFunctionActivity\",\n \"dependsOn\": [\n {\n \"activity\": \"Log Pipeline Running\",\n \"dependencyConditions\": [\n \"Succeeded\"\n ]\n },\n {\n \"activity\": \"Get Pipeline Params\",\n \"dependencyConditions\": [\n \"Succeeded\"\n ]\n },\n {\n \"activity\": \"Get SPN Details\",\n \"dependencyConditions\": [\n \"Succeeded\"\n ]\n }\n ],\n \"policy\": {\n \"timeout\": \"7.00:00:00\",\n \"retry\": 0,\n \"retryIntervalInSeconds\": 30,\n \"secureOutput\": false,\n \"secureInput\": false\n },\n \"userProperties\": [],\n \"typeProperties\": {\n \"functionName\": \"ExecutePipelineV2\",\n \"method\": \"POST\",\n \"headers\": {},\n \"body\": {\n \"value\": \"@concat('\\n{\\n\\t\\\"tenantId\\\": \\\"',pipeline().parameters.TenantId,'\\\",\\n\\t\\\"applicationId\\\": \\\"',activity('Get SPN Details').output.firstRow.Id,'\\\",\\n\\t\\\"authenticationKey\\\": \\\"',activity('Get SPN Details').output.firstRow.Secret,'\\\",\\n\\t\\\"subscriptionId\\\": \\\"',pipeline().parameters.SubscriptionId,'\\\",\\n\\t\\\"resourceGroup\\\": \\\"',item().ResourceGroupName,'\\\",\\n\\t\\\"factoryName\\\": \\\"',item().DataFactoryName,'\\\",\\n\\t\\\"pipelineName\\\": \\\"',item().PipelineName,'\\\"',activity('Get Pipeline Params').output.firstRow.Params,'\\n}')\",\n \"type\": \"Expression\"\n }\n },\n \"linkedServiceName\": {\n \"referenceName\": \"PipelineExecutor\",\n \"type\": \"LinkedServiceReference\"\n }\n },\n {\n \"name\": \"Get Pipeline Params\",\n \"description\": \"Returns any parameters from metadata required for the processing pipeline being called. The output can be an empty string if no parameters are required.\",\n \"type\": \"Lookup\",\n \"dependsOn\": [],\n \"policy\": {\n \"timeout\": \"7.00:00:00\",\n \"retry\": 0,\n \"retryIntervalInSeconds\": 30,\n \"secureOutput\": false,\n \"secureInput\": false\n },\n \"userProperties\": [],\n \"typeProperties\": {\n \"source\": {\n \"type\": \"AzureSqlSource\",\n \"sqlReaderStoredProcedureName\": \"[[procfwk].[GetPipelineParameters]\",\n \"storedProcedureParameters\": {\n \"PipelineId\": {\n \"type\": \"Int32\",\n \"value\": {\n \"value\": \"@item().PipelineId\",\n \"type\": \"Expression\"\n }\n }\n },\n \"queryTimeout\": \"02:00:00\"\n },\n \"dataset\": {\n \"referenceName\": \"GetSetMetadata\",\n \"type\": \"DatasetReference\",\n \"parameters\": {}\n }\n }\n },\n {\n \"name\": \"Log Pipeline Running\",\n \"description\": \"Sets the current pipeline with a status of running within the current execution database table.\",\n \"type\": \"SqlServerStoredProcedure\",\n \"dependsOn\": [],\n \"policy\": {\n \"timeout\": \"7.00:00:00\",\n \"retry\": 0,\n \"retryIntervalInSeconds\": 30,\n \"secureOutput\": false,\n \"secureInput\": false\n },\n \"userProperties\": [],\n \"typeProperties\": {\n \"storedProcedureName\": \"[[procfwk].[SetLogPipelineRunning]\",\n \"storedProcedureParameters\": {\n \"ExecutionId\": {\n \"value\": {\n \"value\": \"@pipeline().parameters.ExecutionId\",\n \"type\": \"Expression\"\n },\n \"type\": \"Guid\"\n },\n \"PipelineId\": {\n \"value\": {\n \"value\": \"@item().PipelineId\",\n \"type\": \"Expression\"\n },\n \"type\": \"Int32\"\n },\n \"StageId\": {\n \"value\": {\n \"value\": \"@pipeline().parameters.StageId\",\n \"type\": \"Expression\"\n },\n \"type\": \"Int32\"\n }\n }\n },\n \"linkedServiceName\": {\n \"referenceName\": \"SupportDatabase\",\n \"type\": \"LinkedServiceReference\"\n }\n },\n {\n \"name\": \"Get SPN Details\",\n \"description\": \"Return the SPN ID and Secret for the processing pipeline being executed. Called at this level as each pipeline can have a different SPN.\",\n \"type\": \"Lookup\",\n \"dependsOn\": [],\n \"policy\": {\n \"timeout\": \"7.00:00:00\",\n \"retry\": 0,\n \"retryIntervalInSeconds\": 30,\n \"secureOutput\": false,\n \"secureInput\": false\n },\n \"userProperties\": [],\n \"typeProperties\": {\n \"source\": {\n \"type\": \"AzureSqlSource\",\n \"sqlReaderStoredProcedureName\": \"[[procfwk].[GetServicePrincipal]\",\n \"storedProcedureParameters\": {\n \"DataFactory\": {\n \"type\": \"String\",\n \"value\": {\n \"value\": \"@item().DataFactoryName\",\n \"type\": \"Expression\"\n }\n },\n \"PipelineName\": {\n \"type\": \"String\",\n \"value\": {\n \"value\": \"@item().PipelineName\",\n \"type\": \"Expression\"\n }\n }\n },\n \"queryTimeout\": \"02:00:00\"\n },\n \"dataset\": {\n \"referenceName\": \"GetSetMetadata\",\n \"type\": \"DatasetReference\",\n \"parameters\": {}\n }\n }\n },\n {\n \"name\": \"Log Activity Failure\",\n \"description\": \"Handle true failures from calling out to the Azure Function and update the current execution table accordingly so a restart can occur.\",\n \"type\": \"SqlServerStoredProcedure\",\n \"dependsOn\": [\n {\n \"activity\": \"Execute Pipeline\",\n \"dependencyConditions\": [\n \"Failed\"\n ]\n }\n ],\n \"policy\": {\n \"timeout\": \"7.00:00:00\",\n \"retry\": 0,\n \"retryIntervalInSeconds\": 30,\n \"secureOutput\": false,\n \"secureInput\": false\n },\n \"userProperties\": [],\n \"typeProperties\": {\n \"storedProcedureName\": \"[[procfwk].[SetLogActivityFailed]\",\n \"storedProcedureParameters\": {\n \"ExecutionId\": {\n \"value\": {\n \"value\": \"@pipeline().parameters.ExecutionId\",\n \"type\": \"Expression\"\n },\n \"type\": \"Guid\"\n },\n \"PipelineId\": {\n \"value\": {\n \"value\": \"@item().PipelineId\",\n \"type\": \"Expression\"\n },\n \"type\": \"Int32\"\n },\n \"RunId\": {\n \"value\": {\n \"value\": \"@pipeline().RunId\",\n \"type\": \"Expression\"\n },\n \"type\": \"Guid\"\n },\n \"StageId\": {\n \"value\": {\n \"value\": \"@pipeline().parameters.StageId\",\n \"type\": \"Expression\"\n },\n \"type\": \"Int32\"\n }\n }\n },\n \"linkedServiceName\": {\n \"referenceName\": \"SupportDatabase\",\n \"type\": \"LinkedServiceReference\"\n }\n },\n {\n \"name\": \"Running Pipeline Handler\",\n \"type\": \"ExecutePipeline\",\n \"dependsOn\": [\n {\n \"activity\": \"Execute Pipeline\",\n \"dependencyConditions\": [\n \"Succeeded\"\n ]\n }\n ],\n \"userProperties\": [],\n \"typeProperties\": {\n \"pipeline\": {\n \"referenceName\": \"04-Infant\",\n \"type\": \"PipelineReference\"\n },\n \"waitOnCompletion\": true,\n \"parameters\": {\n \"tenantId\": {\n \"value\": \"@pipeline().parameters.TenantId\",\n \"type\": \"Expression\"\n },\n \"applicationId\": {\n \"value\": \"@activity('Get SPN Details').output.firstRow.Id\",\n \"type\": \"Expression\"\n },\n \"authenticationKey\": {\n \"value\": \"@activity('Get SPN Details').output.firstRow.Secret\",\n \"type\": \"Expression\"\n },\n \"subscriptionId\": {\n \"value\": \"@pipeline().parameters.SubscriptionId\",\n \"type\": \"Expression\"\n },\n \"resourceGroup\": {\n \"value\": \"@item().ResourceGroupName\",\n \"type\": \"Expression\"\n },\n \"factoryName\": {\n \"value\": \"@item().DataFactoryName\",\n \"type\": \"Expression\"\n },\n \"pipelineName\": {\n \"value\": \"@item().PipelineName\",\n \"type\": \"Expression\"\n },\n \"runId\": {\n \"value\": \"@activity('Execute Pipeline').output.RunId\",\n \"type\": \"Expression\"\n },\n \"executionId\": {\n \"value\": \"@pipeline().parameters.ExecutionId\",\n \"type\": \"Expression\"\n },\n \"stageId\": {\n \"value\": \"@pipeline().parameters.StageId\",\n \"type\": \"Expression\"\n },\n \"pipelineId\": {\n \"value\": \"@item().PipelineId\",\n \"type\": \"Expression\"\n }\n }\n }\n }\n ]\n }\n }\n ],\n \"parameters\": {\n \"StageId\": {\n \"type\": \"int\"\n },\n \"ExecutionId\": {\n \"type\": \"string\"\n },\n \"TenantId\": {\n \"type\": \"string\"\n },\n \"SubscriptionId\": {\n \"type\": \"string\"\n }\n },\n \"variables\": {\n \"FunctionBody\": {\n \"type\": \"String\"\n }\n },\n \"folder\": {\n \"name\": \"_ProcFwk\"\n },\n \"annotations\": [\n \"ADF.procfwk\"\n ]\n },\n \"dependsOn\": [\n \"[concat(variables('factoryId'), '/datasets/GetSetMetadata')]\",\n \"[concat(variables('factoryId'), '/linkedServices/PipelineExecutor')]\",\n \"[concat(variables('factoryId'), '/linkedServices/SupportDatabase')]\",\n \"[concat(variables('factoryId'), '/pipelines/04-Infant')]\"\n ]\n },\n {\n \"name\": \"[concat(parameters('factoryName'), '/04-Infant')]\",\n \"type\": \"Microsoft.DataFactory/factories/pipelines\",\n \"apiVersion\": \"2018-06-01\",\n \"properties\": {\n \"description\": \"ADF.procfwk infant pipeline used to check when the processing pipeline called by the Child completes and passes the resulting status back to the metadata database.\",\n \"activities\": [\n {\n \"name\": \"Wait Until Pipeline Completes\",\n \"description\": \"Loops until the pipeline called completes.\\n\\nSimple status:\\n- Running = new iteration.\\n- Done = break.\",\n \"type\": \"Until\",\n \"dependsOn\": [\n {\n \"activity\": \"Get Wait Duration\",\n \"dependencyConditions\": [\n \"Succeeded\"\n ]\n }\n ],\n \"userProperties\": [],\n \"typeProperties\": {\n \"expression\": {\n \"value\": \"@equals('Done',activity('Get Pipeline Status').output.SimpleStatus)\",\n \"type\": \"Expression\"\n },\n \"activities\": [\n {\n \"name\": \"Get Pipeline Status\",\n \"description\": \"Checks the status of a given processing pipeline and provides the value for the downstream framework activities to act upon.\",\n \"type\": \"AzureFunctionActivity\",\n \"dependsOn\": [],\n \"policy\": {\n \"timeout\": \"7.00:00:00\",\n \"retry\": 0,\n \"retryIntervalInSeconds\": 30,\n \"secureOutput\": false,\n \"secureInput\": false\n },\n \"userProperties\": [],\n \"typeProperties\": {\n \"functionName\": \"CheckPipelineStatus\",\n \"method\": \"POST\",\n \"headers\": {},\n \"body\": {\n \"value\": \"@concat('\\n{\\n \\\"tenantId\\\": \\\"',pipeline().parameters.TenantId,'\\\",\\n \\\"applicationId\\\": \\\"',pipeline().parameters.applicationId,'\\\",\\n \\\"authenticationKey\\\": \\\"',pipeline().parameters.authenticationKey,'\\\",\\n \\\"subscriptionId\\\": \\\"',pipeline().parameters.subscriptionId,'\\\",\\n \\\"resourceGroup\\\": \\\"',pipeline().parameters.resourceGroup,'\\\",\\n \\\"factoryName\\\": \\\"',pipeline().parameters.factoryName,'\\\",\\n \\\"pipelineName\\\": \\\"',pipeline().parameters.pipelineName,'\\\",\\n \\\"runId\\\": \\\"',pipeline().parameters.runId,'\\\"\\n}')\",\n \"type\": \"Expression\"\n }\n },\n \"linkedServiceName\": {\n \"referenceName\": \"PipelineExecutor\",\n \"type\": \"LinkedServiceReference\"\n }\n },\n {\n \"name\": \"Wait If Running\",\n \"description\": \"True = Do nothing.\\nFalse = Wait.\",\n \"type\": \"IfCondition\",\n \"dependsOn\": [\n {\n \"activity\": \"Get Pipeline Status\",\n \"dependencyConditions\": [\n \"Succeeded\"\n ]\n }\n ],\n \"userProperties\": [],\n \"typeProperties\": {\n \"expression\": {\n \"value\": \"@equals('Done',activity('Get Pipeline Status').output.SimpleStatus)\",\n \"type\": \"Expression\"\n },\n \"ifFalseActivities\": [\n {\n \"name\": \"Wait for Pipeline\",\n \"description\": \"The processing pipeline is still running so Wait before checking its status again.\",\n \"type\": \"Wait\",\n \"dependsOn\": [],\n \"userProperties\": [],\n \"typeProperties\": {\n \"waitTimeInSeconds\": {\n \"value\": \"@activity('Get Wait Duration').output.firstRow.PropertyValue\",\n \"type\": \"Expression\"\n }\n }\n }\n ]\n }\n },\n {\n \"name\": \"Wait to Retry Function\",\n \"type\": \"Wait\",\n \"dependsOn\": [\n {\n \"activity\": \"Get Pipeline Status\",\n \"dependencyConditions\": [\n \"Failed\"\n ]\n }\n ],\n \"userProperties\": [],\n \"typeProperties\": {\n \"waitTimeInSeconds\": {\n \"value\": \"@activity('Get Wait Duration').output.firstRow.PropertyValue\",\n \"type\": \"Expression\"\n }\n }\n },\n {\n \"name\": \"Set Last Update\",\n \"type\": \"SqlServerStoredProcedure\",\n \"dependsOn\": [\n {\n \"activity\": \"Get Pipeline Status\",\n \"dependencyConditions\": [\n \"Succeeded\"\n ]\n }\n ],\n \"policy\": {\n \"timeout\": \"7.00:00:00\",\n \"retry\": 0,\n \"retryIntervalInSeconds\": 30,\n \"secureOutput\": false,\n \"secureInput\": false\n },\n \"userProperties\": [],\n \"typeProperties\": {\n \"storedProcedureName\": \"[[procfwk].[SetLogPipelineLastStatusCheck]\",\n \"storedProcedureParameters\": {\n \"ExecutionId\": {\n \"value\": {\n \"value\": \"@pipeline().parameters.executionId\",\n \"type\": \"Expression\"\n },\n \"type\": \"Guid\"\n },\n \"PipelineId\": {\n \"value\": {\n \"value\": \"@pipeline().parameters.pipelineId\",\n \"type\": \"Expression\"\n },\n \"type\": \"Int32\"\n },\n \"StageId\": {\n \"value\": {\n \"value\": \"@pipeline().parameters.stageId\",\n \"type\": \"Expression\"\n },\n \"type\": \"Int32\"\n }\n }\n },\n \"linkedServiceName\": {\n \"referenceName\": \"SupportDatabase\",\n \"type\": \"LinkedServiceReference\"\n }\n }\n ],\n \"timeout\": \"7.00:00:00\"\n }\n },\n {\n \"name\": \"Pipeline Result\",\n \"description\": \"Receives the outcome from the function execution for a given processing pipeline and updates the current execution table with different pipelines status values depending on the result (case).\",\n \"type\": \"Switch\",\n \"dependsOn\": [\n {\n \"activity\": \"Wait Until Pipeline Completes\",\n \"dependencyConditions\": [\n \"Completed\"\n ]\n }\n ],\n \"userProperties\": [],\n \"typeProperties\": {\n \"on\": {\n \"value\": \"@activity('Get Pipeline Status').output.Status\",\n \"type\": \"Expression\"\n },\n \"cases\": [\n {\n \"value\": \"Succeeded\",\n \"activities\": [\n {\n \"name\": \"Pipeline Status Succeeded\",\n \"description\": \"Updates the current execution table with a pipeline status of success if the function outcome is succeeded.\",\n \"type\": \"SqlServerStoredProcedure\",\n \"dependsOn\": [],\n \"policy\": {\n \"timeout\": \"7.00:00:00\",\n \"retry\": 0,\n \"retryIntervalInSeconds\": 30,\n \"secureOutput\": false,\n \"secureInput\": false\n },\n \"userProperties\": [],\n \"typeProperties\": {\n \"storedProcedureName\": \"[[procfwk].[SetLogPipelineSuccess]\",\n \"storedProcedureParameters\": {\n \"ExecutionId\": {\n \"value\": {\n \"value\": \"@pipeline().parameters.executionId\",\n \"type\": \"Expression\"\n },\n \"type\": \"Guid\"\n },\n \"PipelineId\": {\n \"value\": {\n \"value\": \"@pipeline().parameters.pipelineId\",\n \"type\": \"Expression\"\n },\n \"type\": \"Int32\"\n },\n \"StageId\": {\n \"value\": {\n \"value\": \"@pipeline().parameters.stageId\",\n \"type\": \"Expression\"\n },\n \"type\": \"Int32\"\n }\n }\n },\n \"linkedServiceName\": {\n \"referenceName\": \"SupportDatabase\",\n \"type\": \"LinkedServiceReference\"\n }\n }\n ]\n },\n {\n \"value\": \"Failed\",\n \"activities\": [\n {\n \"name\": \"Pipeline Status Failed\",\n \"description\": \"Updates the current execution table with a pipeline status of failed if the function outcome is failed. Also blocks pipelines in the downstream execution stage.\",\n \"type\": \"SqlServerStoredProcedure\",\n \"dependsOn\": [],\n \"policy\": {\n \"timeout\": \"7.00:00:00\",\n \"retry\": 0,\n \"retryIntervalInSeconds\": 30,\n \"secureOutput\": false,\n \"secureInput\": false\n },\n \"userProperties\": [],\n \"typeProperties\": {\n \"storedProcedureName\": \"[[procfwk].[SetLogPipelineFailed]\",\n \"storedProcedureParameters\": {\n \"ExecutionId\": {\n \"value\": {\n \"value\": \"@pipeline().parameters.executionId\",\n \"type\": \"Expression\"\n },\n \"type\": \"Guid\"\n },\n \"PipelineId\": {\n \"value\": {\n \"value\": \"@pipeline().parameters.pipelineId\",\n \"type\": \"Expression\"\n },\n \"type\": \"Int32\"\n },\n \"RunId\": {\n \"value\": {\n \"value\": \"@activity('Get Pipeline Status').output.RunId\",\n \"type\": \"Expression\"\n },\n \"type\": \"Guid\"\n },\n \"StageId\": {\n \"value\": {\n \"value\": \"@pipeline().parameters.stageId\",\n \"type\": \"Expression\"\n },\n \"type\": \"Int32\"\n }\n }\n },\n \"linkedServiceName\": {\n \"referenceName\": \"SupportDatabase\",\n \"type\": \"LinkedServiceReference\"\n }\n }\n ]\n }\n ],\n \"defaultActivities\": [\n {\n \"name\": \"Pipeline Status Unknown\",\n \"description\": \"Updates the current execution table with a pipeline status of unknown if the function returns an unexpected outcome.\",\n \"type\": \"SqlServerStoredProcedure\",\n \"dependsOn\": [],\n \"policy\": {\n \"timeout\": \"7.00:00:00\",\n \"retry\": 0,\n \"retryIntervalInSeconds\": 30,\n \"secureOutput\": false,\n \"secureInput\": false\n },\n \"userProperties\": [],\n \"typeProperties\": {\n \"storedProcedureName\": \"[[procfwk].[SetLogPipelineUnknown]\",\n \"storedProcedureParameters\": {\n \"ExecutionId\": {\n \"value\": {\n \"value\": \"@pipeline().parameters.executionId\",\n \"type\": \"Expression\"\n },\n \"type\": \"Guid\"\n },\n \"PipelineId\": {\n \"value\": {\n \"value\": \"@pipeline().parameters.pipelineId\",\n \"type\": \"Expression\"\n },\n \"type\": \"Int32\"\n },\n \"StageId\": {\n \"value\": {\n \"value\": \"@pipeline().parameters.stageId\",\n \"type\": \"Expression\"\n },\n \"type\": \"Int32\"\n }\n }\n },\n \"linkedServiceName\": {\n \"referenceName\": \"SupportDatabase\",\n \"type\": \"LinkedServiceReference\"\n }\n }\n ]\n }\n },\n {\n \"name\": \"Get Wait Duration\",\n \"type\": \"Lookup\",\n \"dependsOn\": [],\n \"policy\": {\n \"timeout\": \"7.00:00:00\",\n \"retry\": 0,\n \"retryIntervalInSeconds\": 30,\n \"secureOutput\": false,\n \"secureInput\": false\n },\n \"userProperties\": [],\n \"typeProperties\": {\n \"source\": {\n \"type\": \"AzureSqlSource\",\n \"sqlReaderStoredProcedureName\": \"[[procfwk].[GetPropertyValue]\",\n \"storedProcedureParameters\": {\n \"PropertyName\": {\n \"type\": \"String\",\n \"value\": \"PipelineStatusCheckDuration\"\n }\n },\n \"queryTimeout\": \"02:00:00\"\n },\n \"dataset\": {\n \"referenceName\": \"GetSetMetadata\",\n \"type\": \"DatasetReference\",\n \"parameters\": {}\n }\n }\n }\n ],\n \"parameters\": {\n \"tenantId\": {\n \"type\": \"string\"\n },\n \"applicationId\": {\n \"type\": \"string\"\n },\n \"authenticationKey\": {\n \"type\": \"string\"\n },\n \"subscriptionId\": {\n \"type\": \"string\"\n },\n \"resourceGroup\": {\n \"type\": \"string\"\n },\n \"factoryName\": {\n \"type\": \"string\"\n },\n \"pipelineName\": {\n \"type\": \"string\"\n },\n \"runId\": {\n \"type\": \"string\"\n },\n \"executionId\": {\n \"type\": \"string\"\n },\n \"stageId\": {\n \"type\": \"int\"\n },\n \"pipelineId\": {\n \"type\": \"int\"\n }\n },\n \"folder\": {\n \"name\": \"_ProcFwk\"\n },\n \"annotations\": [\n \"ADF.procfwk\"\n ]\n },\n \"dependsOn\": [\n \"[concat(variables('factoryId'), '/datasets/GetSetMetadata')]\",\n \"[concat(variables('factoryId'), '/linkedServices/PipelineExecutor')]\",\n \"[concat(variables('factoryId'), '/linkedServices/SupportDatabase')]\"\n ]\n },\n {\n \"name\": \"[concat(parameters('factoryName'), '/Intention Error')]\",\n \"type\": \"Microsoft.DataFactory/factories/pipelines\",\n \"apiVersion\": \"2018-06-01\",\n \"properties\": {\n \"description\": \"Used just so the ADF.procfwk has something to call during development.\",\n \"activities\": [\n {\n \"name\": \"Call Fail Procedure\",\n \"type\": \"SqlServerStoredProcedure\",\n \"dependsOn\": [],\n \"policy\": {\n \"timeout\": \"7.00:00:00\",\n \"retry\": 0,\n \"retryIntervalInSeconds\": 30,\n \"secureOutput\": false,\n \"secureInput\": false\n },\n \"userProperties\": [],\n \"typeProperties\": {\n \"storedProcedureName\": \"[[dbo].[FailProcedure]\"\n },\n \"linkedServiceName\": {\n \"referenceName\": \"SupportDatabase\",\n \"type\": \"LinkedServiceReference\"\n }\n }\n ],\n \"folder\": {\n \"name\": \"Processes\"\n },\n \"annotations\": []\n },\n \"dependsOn\": [\n \"[concat(variables('factoryId'), '/linkedServices/SupportDatabase')]\"\n ]\n },\n {\n \"name\": \"[concat(parameters('factoryName'), '/Wait 1')]\",\n \"type\": \"Microsoft.DataFactory/factories/pipelines\",\n \"apiVersion\": \"2018-06-01\",\n \"properties\": {\n \"description\": \"Used just so the ADF.procfwk has something to call during development.\",\n \"activities\": [\n {\n \"name\": \"Wait1\",\n \"type\": \"Wait\",\n \"dependsOn\": [],\n \"userProperties\": [],\n \"typeProperties\": {\n \"waitTimeInSeconds\": {\n \"value\": \"@pipeline().parameters.WaitTime\",\n \"type\": \"Expression\"\n }\n }\n }\n ],\n \"parameters\": {\n \"WaitTime\": {\n \"type\": \"int\",\n \"defaultValue\": 5\n }\n },\n \"folder\": {\n \"name\": \"Processes\"\n },\n \"annotations\": []\n },\n \"dependsOn\": []\n },\n {\n \"name\": \"[concat(parameters('factoryName'), '/Wait 10')]\",\n \"type\": \"Microsoft.DataFactory/factories/pipelines\",\n \"apiVersion\": \"2018-06-01\",\n \"properties\": {\n \"description\": \"Used just so the ADF.procfwk has something to call during development.\",\n \"activities\": [\n {\n \"name\": \"Wait10\",\n \"type\": \"Wait\",\n \"dependsOn\": [],\n \"userProperties\": [],\n \"typeProperties\": {\n \"waitTimeInSeconds\": {\n \"value\": \"@pipeline().parameters.WaitTime\",\n \"type\": \"Expression\"\n }\n }\n }\n ],\n \"parameters\": {\n \"WaitTime\": {\n \"type\": \"int\",\n \"defaultValue\": 5\n }\n },\n \"folder\": {\n \"name\": \"Processes\"\n },\n \"annotations\": []\n },\n \"dependsOn\": []\n },\n {\n \"name\": \"[concat(parameters('factoryName'), '/Wait 2')]\",\n \"type\": \"Microsoft.DataFactory/factories/pipelines\",\n \"apiVersion\": \"2018-06-01\",\n \"properties\": {\n \"description\": \"Used just so the ADF.procfwk has something to call during development.\",\n \"activities\": [\n {\n \"name\": \"Wait2\",\n \"type\": \"Wait\",\n \"dependsOn\": [],\n \"userProperties\": [],\n \"typeProperties\": {\n \"waitTimeInSeconds\": {\n \"value\": \"@pipeline().parameters.WaitTime\",\n \"type\": \"Expression\"\n }\n }\n }\n ],\n \"parameters\": {\n \"WaitTime\": {\n \"type\": \"int\",\n \"defaultValue\": 5\n }\n },\n \"folder\": {\n \"name\": \"Processes\"\n },\n \"annotations\": []\n },\n \"dependsOn\": []\n },\n {\n \"name\": \"[concat(parameters('factoryName'), '/Wait 3')]\",\n \"type\": \"Microsoft.DataFactory/factories/pipelines\",\n \"apiVersion\": \"2018-06-01\",\n \"properties\": {\n \"description\": \"Used just so the ADF.procfwk has something to call during development.\",\n \"activities\": [\n {\n \"name\": \"Wait3\",\n \"type\": \"Wait\",\n \"dependsOn\": [],\n \"userProperties\": [],\n \"typeProperties\": {\n \"waitTimeInSeconds\": {\n \"value\": \"@pipeline().parameters.WaitTime\",\n \"type\": \"Expression\"\n }\n }\n }\n ],\n \"parameters\": {\n \"WaitTime\": {\n \"type\": \"int\",\n \"defaultValue\": 5\n }\n },\n \"folder\": {\n \"name\": \"Processes\"\n },\n \"annotations\": []\n },\n \"dependsOn\": []\n },\n {\n \"name\": \"[concat(parameters('factoryName'), '/Wait 4')]\",\n \"type\": \"Microsoft.DataFactory/factories/pipelines\",\n \"apiVersion\": \"2018-06-01\",\n \"properties\": {\n \"description\": \"Used just so the ADF.procfwk has something to call during development.\",\n \"activities\": [\n {\n \"name\": \"Wait4\",\n \"type\": \"Wait\",\n \"dependsOn\": [],\n \"userProperties\": [],\n \"typeProperties\": {\n \"waitTimeInSeconds\": {\n \"value\": \"@pipeline().parameters.WaitTime\",\n \"type\": \"Expression\"\n }\n }\n }\n ],\n \"parameters\": {\n \"WaitTime\": {\n \"type\": \"int\",\n \"defaultValue\": 5\n }\n },\n \"folder\": {\n \"name\": \"Processes\"\n },\n \"annotations\": []\n },\n \"dependsOn\": []\n },\n {\n \"name\": \"[concat(parameters('factoryName'), '/Wait 5')]\",\n \"type\": \"Microsoft.DataFactory/factories/pipelines\",\n \"apiVersion\": \"2018-06-01\",\n \"properties\": {\n \"description\": \"Used just so the ADF.procfwk has something to call during development.\",\n \"activities\": [\n {\n \"name\": \"Wait5\",\n \"type\": \"Wait\",\n \"dependsOn\": [],\n \"userProperties\": [],\n \"typeProperties\": {\n \"waitTimeInSeconds\": {\n \"value\": \"@pipeline().parameters.WaitTime\",\n \"type\": \"Expression\"\n }\n }\n }\n ],\n \"parameters\": {\n \"WaitTime\": {\n \"type\": \"int\",\n \"defaultValue\": 5\n }\n },\n \"folder\": {\n \"name\": \"Processes\"\n },\n \"annotations\": []\n },\n \"dependsOn\": []\n },\n {\n \"name\": \"[concat(parameters('factoryName'), '/Wait 6')]\",\n \"type\": \"Microsoft.DataFactory/factories/pipelines\",\n \"apiVersion\": \"2018-06-01\",\n \"properties\": {\n \"description\": \"Used just so the ADF.procfwk has something to call during development.\",\n \"activities\": [\n {\n \"name\": \"Wait6\",\n \"type\": \"Wait\",\n \"dependsOn\": [],\n \"userProperties\": [],\n \"typeProperties\": {\n \"waitTimeInSeconds\": {\n \"value\": \"@pipeline().parameters.WaitTime\",\n \"type\": \"Expression\"\n }\n }\n }\n ],\n \"parameters\": {\n \"WaitTime\": {\n \"type\": \"int\",\n \"defaultValue\": 5\n }\n },\n \"folder\": {\n \"name\": \"Processes\"\n },\n \"annotations\": []\n },\n \"dependsOn\": []\n },\n {\n \"name\": \"[concat(parameters('factoryName'), '/Wait 7')]\",\n \"type\": \"Microsoft.DataFactory/factories/pipelines\",\n \"apiVersion\": \"2018-06-01\",\n \"properties\": {\n \"description\": \"Used just so the ADF.procfwk has something to call during development.\",\n \"activities\": [\n {\n \"name\": \"Wait7\",\n \"type\": \"Wait\",\n \"dependsOn\": [],\n \"userProperties\": [],\n \"typeProperties\": {\n \"waitTimeInSeconds\": {\n \"value\": \"@pipeline().parameters.WaitTime\",\n \"type\": \"Expression\"\n }\n }\n }\n ],\n \"parameters\": {\n \"WaitTime\": {\n \"type\": \"int\",\n \"defaultValue\": 5\n }\n },\n \"folder\": {\n \"name\": \"Processes\"\n },\n \"annotations\": []\n },\n \"dependsOn\": []\n },\n {\n \"name\": \"[concat(parameters('factoryName'), '/Wait 8')]\",\n \"type\": \"Microsoft.DataFactory/factories/pipelines\",\n \"apiVersion\": \"2018-06-01\",\n \"properties\": {\n \"description\": \"Used just so the ADF.procfwk has something to call during development.\",\n \"activities\": [\n {\n \"name\": \"Wait8\",\n \"type\": \"Wait\",\n \"dependsOn\": [],\n \"userProperties\": [],\n \"typeProperties\": {\n \"waitTimeInSeconds\": {\n \"value\": \"@pipeline().parameters.WaitTime\",\n \"type\": \"Expression\"\n }\n }\n }\n ],\n \"parameters\": {\n \"WaitTime\": {\n \"type\": \"int\",\n \"defaultValue\": 5\n }\n },\n \"folder\": {\n \"name\": \"Processes\"\n },\n \"annotations\": []\n },\n \"dependsOn\": []\n },\n {\n \"name\": \"[concat(parameters('factoryName'), '/Wait 9')]\",\n \"type\": \"Microsoft.DataFactory/factories/pipelines\",\n \"apiVersion\": \"2018-06-01\",\n \"properties\": {\n \"description\": \"Used just so the ADF.procfwk has something to call during development.\",\n \"activities\": [\n {\n \"name\": \"Wait9\",\n \"type\": \"Wait\",\n \"dependsOn\": [],\n \"userProperties\": [],\n \"typeProperties\": {\n \"waitTimeInSeconds\": {\n \"value\": \"@pipeline().parameters.WaitTime\",\n \"type\": \"Expression\"\n }\n }\n }\n ],\n \"parameters\": {\n \"WaitTime\": {\n \"type\": \"int\",\n \"defaultValue\": 5\n }\n },\n \"folder\": {\n \"name\": \"Processes\"\n },\n \"annotations\": []\n },\n \"dependsOn\": []\n },\n {\n \"name\": \"[concat(parameters('factoryName'), '/GetSetMetadata')]\",\n \"type\": \"Microsoft.DataFactory/factories/datasets\",\n \"apiVersion\": \"2018-06-01\",\n \"properties\": {\n \"linkedServiceName\": {\n \"referenceName\": \"SupportDatabase\",\n \"type\": \"LinkedServiceReference\"\n },\n \"annotations\": [\n \"ADF.procfwk\"\n ],\n \"type\": \"AzureSqlTable\",\n \"schema\": [],\n \"typeProperties\": {}\n },\n \"dependsOn\": [\n \"[concat(variables('factoryId'), '/linkedServices/SupportDatabase')]\"\n ]\n },\n {\n \"name\": \"[concat(parameters('factoryName'), '/Keys')]\",\n \"type\": \"Microsoft.DataFactory/factories/linkedServices\",\n \"apiVersion\": \"2018-06-01\",\n \"properties\": {\n \"description\": \"Connection to Key Vault for all other ADF linked service credentials required to run the processing framework.\",\n \"annotations\": [\n \"ADF.procfwk\"\n ],\n \"type\": \"AzureKeyVault\",\n \"typeProperties\": {\n \"baseUrl\": \"[parameters('Keys_properties_typeProperties_baseUrl')]\"\n }\n },\n \"dependsOn\": []\n },\n {\n \"name\": \"[concat(parameters('factoryName'), '/PipelineExecutor')]\",\n \"type\": \"Microsoft.DataFactory/factories/linkedServices\",\n \"apiVersion\": \"2018-06-01\",\n \"properties\": {\n \"description\": \"Connection to the Function App from ADF for calling the processing pipeline function within the orchestration framework.\",\n \"annotations\": [\n \"ADF.procfwk\"\n ],\n \"type\": \"AzureFunction\",\n \"typeProperties\": {\n \"functionAppUrl\": \"https://pipelineexecutor.azurewebsites.net\",\n \"functionKey\": {\n \"type\": \"AzureKeyVaultSecret\",\n \"store\": {\n \"referenceName\": \"Keys\",\n \"type\": \"LinkedServiceReference\"\n },\n \"secretName\": \"ExecutorFunctionAppKey\"\n }\n }\n },\n \"dependsOn\": [\n \"[concat(variables('factoryId'), '/linkedServices/Keys')]\"\n ]\n },\n {\n \"name\": \"[concat(parameters('factoryName'), '/SupportDatabase')]\",\n \"type\": \"Microsoft.DataFactory/factories/linkedServices\",\n \"apiVersion\": \"2018-06-01\",\n \"properties\": {\n \"description\": \"Connection between ADF and processing framework metadata SQLDB.\",\n \"annotations\": [\n \"ADF.procfwk\"\n ],\n \"type\": \"AzureSqlDatabase\",\n \"typeProperties\": {\n \"connectionString\": {\n \"type\": \"AzureKeyVaultSecret\",\n \"store\": {\n \"referenceName\": \"Keys\",\n \"type\": \"LinkedServiceReference\"\n },\n \"secretName\": \"[parameters('SupportDatabase_properties_typeProperties_connectionString_secretName')]\"\n }\n }\n },\n \"dependsOn\": [\n \"[concat(variables('factoryId'), '/linkedServices/Keys')]\"\n ]\n },\n {\n \"name\": \"[concat(parameters('factoryName'), '/FunctionalTestingTrigger')]\",\n \"type\": \"Microsoft.DataFactory/factories/triggers\",\n \"apiVersion\": \"2018-06-01\",\n \"properties\": {\n \"description\": \"Used for functional testing of the framework in a dedicated environment.\",\n \"annotations\": [\n \"ADF.procfwk\"\n ],\n \"runtimeState\": \"Stopped\",\n \"pipelines\": [\n {\n \"pipelineReference\": {\n \"referenceName\": \"01-Grandparent\",\n \"type\": \"PipelineReference\"\n },\n \"parameters\": {}\n }\n ],\n \"type\": \"ScheduleTrigger\",\n \"typeProperties\": {\n \"recurrence\": {\n \"frequency\": \"Hour\",\n \"interval\": 1,\n \"startTime\": \"2020-04-06T15:00:00.000Z\",\n \"timeZone\": \"UTC\"\n }\n }\n },\n \"dependsOn\": [\n \"[concat(variables('factoryId'), '/pipelines/01-Grandparent')]\"\n ]\n }\n ]\n}"
},
{
"path": "ARM Templates/Data Factory/v1.5 Export.json",
"content": "{\n \"$schema\": \"http://schema.management.azure.com/schemas/2015-01-01/deploymentTemplate.json#\",\n \"contentVersion\": \"1.0.0.0\",\n \"parameters\": {\n \"factoryName\": {\n \"type\": \"string\",\n \"metadata\": \"Data Factory name\",\n \"defaultValue\": \"\"\n },\n \"Keys_properties_typeProperties_baseUrl\": {\n \"type\": \"string\",\n \"defaultValue\": \"\"\n },\n \"SupportDatabase_properties_typeProperties_connectionString_secretName\": {\n \"type\": \"string\",\n \"defaultValue\": \"\"\n }\n },\n \"variables\": {\n \"factoryId\": \"[concat('Microsoft.DataFactory/factories/', parameters('factoryName'))]\"\n },\n \"resources\": [\n {\n \"name\": \"[concat(parameters('factoryName'), '/01-Grandparent')]\",\n \"type\": \"Microsoft.DataFactory/factories/pipelines\",\n \"apiVersion\": \"2018-06-01\",\n \"properties\": {\n \"description\": \"ADF.procfwk grandparent pipeline used optionally to bootstrap any wider processes in your Data Factory that then calls the processing framework.\",\n \"activities\": [\n {\n \"name\": \"Framework Processing\",\n \"type\": \"ExecutePipeline\",\n \"dependsOn\": [\n {\n \"activity\": \"Set Random Waits\",\n \"dependencyConditions\": [\n \"Succeeded\"\n ]\n }\n ],\n \"userProperties\": [],\n \"typeProperties\": {\n \"pipeline\": {\n \"referenceName\": \"02-Parent\",\n \"type\": \"PipelineReference\"\n },\n \"waitOnCompletion\": true,\n \"parameters\": {}\n }\n },\n {\n \"name\": \"Set Random Waits\",\n \"description\": \"For functional testing only.\",\n \"type\": \"SqlServerStoredProcedure\",\n \"dependsOn\": [],\n \"policy\": {\n \"timeout\": \"7.00:00:00\",\n \"retry\": 0,\n \"retryIntervalInSeconds\": 30,\n \"secureOutput\": false,\n \"secureInput\": false\n },\n \"userProperties\": [],\n \"typeProperties\": {\n \"storedProcedureName\": \"[[dbo].[SetRandomWaitValues]\"\n },\n \"linkedServiceName\": {\n \"referenceName\": \"SupportDatabase\",\n \"type\": \"LinkedServiceReference\"\n }\n }\n ],\n \"folder\": {\n \"name\": \"_ProcFwk\"\n },\n \"annotations\": [\n \"ADF.procfwk\"\n ]\n },\n \"dependsOn\": [\n \"[concat(variables('factoryId'), '/pipelines/02-Parent')]\",\n \"[concat(variables('factoryId'), '/linkedServices/SupportDatabase')]\"\n ]\n },\n {\n \"name\": \"[concat(parameters('factoryName'), '/02-Parent')]\",\n \"type\": \"Microsoft.DataFactory/factories/pipelines\",\n \"apiVersion\": \"2018-06-01\",\n \"properties\": {\n \"description\": \"ADF.procfwk parent pipeline used to bootstrap the orchestration framework in perform the first level ForEach calls in sequence for the metadata stages.\",\n \"activities\": [\n {\n \"name\": \"Get Stages\",\n \"description\": \"Returns a distinct list of execution stages within the framework metadata.\",\n \"type\": \"Lookup\",\n \"dependsOn\": [\n {\n \"activity\": \"Execution Wrapper\",\n \"dependencyConditions\": [\n \"Succeeded\"\n ]\n }\n ],\n \"policy\": {\n \"timeout\": \"7.00:00:00\",\n \"retry\": 0,\n \"retryIntervalInSeconds\": 30,\n \"secureOutput\": false,\n \"secureInput\": false\n },\n \"userProperties\": [],\n \"typeProperties\": {\n \"source\": {\n \"type\": \"AzureSqlSource\",\n \"sqlReaderStoredProcedureName\": \"[[procfwk].[GetStages]\",\n \"storedProcedureParameters\": {\n \"ExecutionId\": {\n \"type\": \"Guid\",\n \"value\": {\n \"value\": \"@activity('Execution Wrapper').output.firstRow.ExecutionId\",\n \"type\": \"Expression\"\n }\n }\n },\n \"queryTimeout\": \"02:00:00\"\n },\n \"dataset\": {\n \"referenceName\": \"GetSetMetadata\",\n \"type\": \"DatasetReference\",\n \"parameters\": {}\n },\n \"firstRowOnly\": false\n }\n },\n {\n \"name\": \"Execute Stages\",\n \"description\": \"Top level ForEach to sequentially call all processing stages within the framework metadata. Items for iteration passed from the Get Stages lookup activity.\",\n \"type\": \"ForEach\",\n \"dependsOn\": [\n {\n \"activity\": \"Get Stages\",\n \"dependencyConditions\": [\n \"Succeeded\"\n ]\n },\n {\n \"activity\": \"Get Tenant Id\",\n \"dependencyConditions\": [\n \"Succeeded\"\n ]\n },\n {\n \"activity\": \"Get Subscription Id\",\n \"dependencyConditions\": [\n \"Succeeded\"\n ]\n }\n ],\n \"userProperties\": [],\n \"typeProperties\": {\n \"items\": {\n \"value\": \"@activity('Get Stages').output.value\",\n \"type\": \"Expression\"\n },\n \"isSequential\": true,\n \"activities\": [\n {\n \"name\": \"Stage Executor\",\n \"description\": \"Call to the framework generic child pipeline for a given execution stage.\",\n \"type\": \"ExecutePipeline\",\n \"dependsOn\": [\n {\n \"activity\": \"Log Stage Preparing\",\n \"dependencyConditions\": [\n \"Succeeded\"\n ]\n }\n ],\n \"userProperties\": [],\n \"typeProperties\": {\n \"pipeline\": {\n \"referenceName\": \"03-Child\",\n \"type\": \"PipelineReference\"\n },\n \"waitOnCompletion\": true,\n \"parameters\": {\n \"StageId\": {\n \"value\": \"@item().StageId\",\n \"type\": \"Expression\"\n },\n \"ExecutionId\": {\n \"value\": \"@activity('Execution Wrapper').output.firstRow.ExecutionId\",\n \"type\": \"Expression\"\n },\n \"TenantId\": {\n \"value\": \"@activity('Get Tenant Id').output.firstRow.PropertyValue\",\n \"type\": \"Expression\"\n },\n \"SubscriptionId\": {\n \"value\": \"@activity('Get Subscription Id').output.firstRow.PropertyValue\",\n \"type\": \"Expression\"\n }\n }\n }\n },\n {\n \"name\": \"Log Stage Preparing\",\n \"description\": \"Update the current execution table flagging all pipelines within the stage as preparing.\",\n \"type\": \"SqlServerStoredProcedure\",\n \"dependsOn\": [\n {\n \"activity\": \"Check for Blockers\",\n \"dependencyConditions\": [\n \"Succeeded\"\n ]\n }\n ],\n \"policy\": {\n \"timeout\": \"7.00:00:00\",\n \"retry\": 0,\n \"retryIntervalInSeconds\": 30,\n \"secureOutput\": false,\n \"secureInput\": false\n },\n \"userProperties\": [],\n \"typeProperties\": {\n \"storedProcedureName\": \"[[procfwk].[SetLogStagePreparing]\",\n \"storedProcedureParameters\": {\n \"ExecutionId\": {\n \"value\": {\n \"value\": \"@activity('Execution Wrapper').output.firstRow.ExecutionId\",\n \"type\": \"Expression\"\n },\n \"type\": \"Guid\"\n },\n \"StageId\": {\n \"value\": {\n \"value\": \"@item().StageId\",\n \"type\": \"Expression\"\n },\n \"type\": \"Int32\"\n }\n }\n },\n \"linkedServiceName\": {\n \"referenceName\": \"SupportDatabase\",\n \"type\": \"LinkedServiceReference\"\n }\n },\n {\n \"name\": \"Check for Blockers\",\n \"description\": \"Used to double check and stop the next execution stage if failures and blockers have be incurred. Without this step processing would continue regardless of upstream failures.\",\n \"type\": \"SqlServerStoredProcedure\",\n \"dependsOn\": [],\n \"policy\": {\n \"timeout\": \"7.00:00:00\",\n \"retry\": 0,\n \"retryIntervalInSeconds\": 30,\n \"secureOutput\": false,\n \"secureInput\": false\n },\n \"userProperties\": [],\n \"typeProperties\": {\n \"storedProcedureName\": \"[[procfwk].[CheckForBlockedPipelines]\",\n \"storedProcedureParameters\": {\n \"StageId\": {\n \"value\": {\n \"value\": \"@item().StageId\",\n \"type\": \"Expression\"\n },\n \"type\": \"Int32\"\n }\n }\n },\n \"linkedServiceName\": {\n \"referenceName\": \"SupportDatabase\",\n \"type\": \"LinkedServiceReference\"\n }\n }\n ]\n }\n },\n {\n \"name\": \"Execution Wrapper\",\n \"description\": \"Wrapper to reset and restart processing or create a completely new execution instance of the framework metadata.\",\n \"type\": \"Lookup\",\n \"dependsOn\": [\n {\n \"activity\": \"Metadata Integrity Checks\",\n \"dependencyConditions\": [\n \"Succeeded\"\n ]\n }\n ],\n \"policy\": {\n \"timeout\": \"7.00:00:00\",\n \"retry\": 0,\n \"retryIntervalInSeconds\": 30,\n \"secureOutput\": false,\n \"secureInput\": false\n },\n \"userProperties\": [],\n \"typeProperties\": {\n \"source\": {\n \"type\": \"AzureSqlSource\",\n \"sqlReaderStoredProcedureName\": \"[[procfwk].[ExecutionWrapper]\",\n \"storedProcedureParameters\": {\n \"CallingDataFactory\": {\n \"type\": \"String\",\n \"value\": {\n \"value\": \"@pipeline().DataFactory\",\n \"type\": \"Expression\"\n }\n }\n },\n \"queryTimeout\": \"02:00:00\"\n },\n \"dataset\": {\n \"referenceName\": \"GetSetMetadata\",\n \"type\": \"DatasetReference\",\n \"parameters\": {}\n }\n }\n },\n {\n \"name\": \"Archive Execution Log\",\n \"description\": \"After a successful execution run the current execution metadata is moved to the long term logging table by this stored procedure call.\",\n \"type\": \"SqlServerStoredProcedure\",\n \"dependsOn\": [\n {\n \"activity\": \"Execute Stages\",\n \"dependencyConditions\": [\n \"Succeeded\"\n ]\n }\n ],\n \"policy\": {\n \"timeout\": \"7.00:00:00\",\n \"retry\": 0,\n \"retryIntervalInSeconds\": 30,\n \"secureOutput\": false,\n \"secureInput\": false\n },\n \"userProperties\": [],\n \"typeProperties\": {\n \"storedProcedureName\": \"[[procfwk].[UpdateExecutionLog]\"\n },\n \"linkedServiceName\": {\n \"referenceName\": \"SupportDatabase\",\n \"type\": \"LinkedServiceReference\"\n }\n },\n {\n \"name\": \"Get Tenant Id\",\n \"description\": \"Returning the Azure Tenant Id from the metadata properties table.\",\n \"type\": \"Lookup\",\n \"dependsOn\": [\n {\n \"activity\": \"Metadata Integrity Checks\",\n \"dependencyConditions\": [\n \"Succeeded\"\n ]\n }\n ],\n \"policy\": {\n \"timeout\": \"7.00:00:00\",\n \"retry\": 0,\n \"retryIntervalInSeconds\": 30,\n \"secureOutput\": false,\n \"secureInput\": false\n },\n \"userProperties\": [],\n \"typeProperties\": {\n \"source\": {\n \"type\": \"AzureSqlSource\",\n \"sqlReaderStoredProcedureName\": \"[[procfwk].[GetPropertyValue]\",\n \"storedProcedureParameters\": {\n \"PropertyName\": {\n \"type\": \"String\",\n \"value\": \"TenantId\"\n }\n },\n \"queryTimeout\": \"02:00:00\"\n },\n \"dataset\": {\n \"referenceName\": \"GetSetMetadata\",\n \"type\": \"DatasetReference\",\n \"parameters\": {}\n },\n \"firstRowOnly\": true\n }\n },\n {\n \"name\": \"Get Subscription Id\",\n \"description\": \"Returning the Azure Subscription Id from the metadata properties table.\",\n \"type\": \"Lookup\",\n \"dependsOn\": [\n {\n \"activity\": \"Metadata Integrity Checks\",\n \"dependencyConditions\": [\n \"Succeeded\"\n ]\n }\n ],\n \"policy\": {\n \"timeout\": \"7.00:00:00\",\n \"retry\": 0,\n \"retryIntervalInSeconds\": 30,\n \"secureOutput\": false,\n \"secureInput\": true\n },\n \"userProperties\": [],\n \"typeProperties\": {\n \"source\": {\n \"type\": \"AzureSqlSource\",\n \"sqlReaderStoredProcedureName\": \"[[procfwk].[GetPropertyValue]\",\n \"storedProcedureParameters\": {\n \"PropertyName\": {\n \"type\": \"String\",\n \"value\": \"SubscriptionId\"\n }\n },\n \"queryTimeout\": \"02:00:00\"\n },\n \"dataset\": {\n \"referenceName\": \"GetSetMetadata\",\n \"type\": \"DatasetReference\",\n \"parameters\": {}\n }\n }\n },\n {\n \"name\": \"Metadata Integrity Checks\",\n \"description\": \"Performs a series of checks on all metadata held in the framework SQLDB. This is intended to raise errors before an execution run even starts.\",\n \"type\": \"SqlServerStoredProcedure\",\n \"dependsOn\": [],\n \"policy\": {\n \"timeout\": \"7.00:00:00\",\n \"retry\": 0,\n \"retryIntervalInSeconds\": 30,\n \"secureOutput\": false,\n \"secureInput\": false\n },\n \"userProperties\": [],\n \"typeProperties\": {\n \"storedProcedureName\": \"[[procfwk].[CheckMetadataIntegrity]\",\n \"storedProcedureParameters\": {\n \"DebugMode\": {\n \"value\": \"false\",\n \"type\": \"Boolean\"\n }\n }\n },\n \"linkedServiceName\": {\n \"referenceName\": \"SupportDatabase\",\n \"type\": \"LinkedServiceReference\"\n }\n }\n ],\n \"folder\": {\n \"name\": \"_ProcFwk\"\n },\n \"annotations\": [\n \"ADF.procfwk\"\n ]\n },\n \"dependsOn\": [\n \"[concat(variables('factoryId'), '/datasets/GetSetMetadata')]\",\n \"[concat(variables('factoryId'), '/linkedServices/SupportDatabase')]\",\n \"[concat(variables('factoryId'), '/pipelines/03-Child')]\"\n ]\n },\n {\n \"name\": \"[concat(parameters('factoryName'), '/03-Child')]\",\n \"type\": \"Microsoft.DataFactory/factories/pipelines\",\n \"apiVersion\": \"2018-06-01\",\n \"properties\": {\n \"description\": \"ADF.procfwk child pipeline used to execute Worker pipelines within a given execution stage. This pipeline will be called once for each stage, then execute all Workers in parallel.\",\n \"activities\": [\n {\n \"name\": \"Get Pipelines\",\n \"description\": \"Returns all pipelines from the metadata to be executed within a given processing stage.\",\n \"type\": \"Lookup\",\n \"dependsOn\": [],\n \"policy\": {\n \"timeout\": \"7.00:00:00\",\n \"retry\": 0,\n \"retryIntervalInSeconds\": 30,\n \"secureOutput\": false,\n \"secureInput\": false\n },\n \"userProperties\": [],\n \"typeProperties\": {\n \"source\": {\n \"type\": \"AzureSqlSource\",\n \"sqlReaderStoredProcedureName\": \"[[procfwk].[GetPipelinesInStage]\",\n \"storedProcedureParameters\": {\n \"StageId\": {\n \"type\": \"Int32\",\n \"value\": {\n \"value\": \"@pipeline().parameters.StageId\",\n \"type\": \"Expression\"\n }\n }\n },\n \"queryTimeout\": \"02:00:00\"\n },\n \"dataset\": {\n \"referenceName\": \"GetSetMetadata\",\n \"type\": \"DatasetReference\",\n \"parameters\": {}\n },\n \"firstRowOnly\": false\n }\n },\n {\n \"name\": \"Execute Pipelines\",\n \"description\": \"Second level ForEach to run in parallel all pipelines within the stage. Items for iteration passed from the Get Pipelines lookup activity.\",\n \"type\": \"ForEach\",\n \"dependsOn\": [\n {\n \"activity\": \"Get Pipelines\",\n \"dependencyConditions\": [\n \"Succeeded\"\n ]\n }\n ],\n \"userProperties\": [],\n \"typeProperties\": {\n \"items\": {\n \"value\": \"@activity('Get Pipelines').output.value\",\n \"type\": \"Expression\"\n },\n \"isSequential\": false,\n \"activities\": [\n {\n \"name\": \"Execute Pipeline\",\n \"description\": \"The lowest level executor with the metadata framework to call existing processing pipelines within Data Factory. The function called will block processing and wait for an outcome.\",\n \"type\": \"AzureFunctionActivity\",\n \"dependsOn\": [\n {\n \"activity\": \"Log Pipeline Running\",\n \"dependencyConditions\": [\n \"Succeeded\"\n ]\n },\n {\n \"activity\": \"Get Pipeline Params\",\n \"dependencyConditions\": [\n \"Succeeded\"\n ]\n },\n {\n \"activity\": \"Get SPN Details\",\n \"dependencyConditions\": [\n \"Succeeded\"\n ]\n }\n ],\n \"policy\": {\n \"timeout\": \"7.00:00:00\",\n \"retry\": 0,\n \"retryIntervalInSeconds\": 30,\n \"secureOutput\": false,\n \"secureInput\": false\n },\n \"userProperties\": [],\n \"typeProperties\": {\n \"functionName\": \"ExecutePipelineV2\",\n \"method\": \"POST\",\n \"headers\": {},\n \"body\": {\n \"value\": \"@concat('\\n{\\n\\t\\\"tenantId\\\": \\\"',pipeline().parameters.TenantId,'\\\",\\n\\t\\\"applicationId\\\": \\\"',activity('Get SPN Details').output.firstRow.Id,'\\\",\\n\\t\\\"authenticationKey\\\": \\\"',activity('Get SPN Details').output.firstRow.Secret,'\\\",\\n\\t\\\"subscriptionId\\\": \\\"',pipeline().parameters.SubscriptionId,'\\\",\\n\\t\\\"resourceGroup\\\": \\\"',item().ResourceGroupName,'\\\",\\n\\t\\\"factoryName\\\": \\\"',item().DataFactoryName,'\\\",\\n\\t\\\"pipelineName\\\": \\\"',item().PipelineName,'\\\"',activity('Get Pipeline Params').output.firstRow.Params,'\\n}')\",\n \"type\": \"Expression\"\n }\n },\n \"linkedServiceName\": {\n \"referenceName\": \"PipelineExecutor\",\n \"type\": \"LinkedServiceReference\"\n }\n },\n {\n \"name\": \"Get Pipeline Params\",\n \"description\": \"Returns any parameters from metadata required for the processing pipeline being called. The output can be an empty string if no parameters are required.\",\n \"type\": \"Lookup\",\n \"dependsOn\": [],\n \"policy\": {\n \"timeout\": \"7.00:00:00\",\n \"retry\": 0,\n \"retryIntervalInSeconds\": 30,\n \"secureOutput\": false,\n \"secureInput\": false\n },\n \"userProperties\": [],\n \"typeProperties\": {\n \"source\": {\n \"type\": \"AzureSqlSource\",\n \"sqlReaderStoredProcedureName\": \"[[procfwk].[GetPipelineParameters]\",\n \"storedProcedureParameters\": {\n \"PipelineId\": {\n \"type\": \"Int32\",\n \"value\": {\n \"value\": \"@item().PipelineId\",\n \"type\": \"Expression\"\n }\n }\n },\n \"queryTimeout\": \"02:00:00\"\n },\n \"dataset\": {\n \"referenceName\": \"GetSetMetadata\",\n \"type\": \"DatasetReference\",\n \"parameters\": {}\n }\n }\n },\n {\n \"name\": \"Log Pipeline Running\",\n \"description\": \"Sets the current pipeline with a status of running within the current execution database table.\",\n \"type\": \"SqlServerStoredProcedure\",\n \"dependsOn\": [],\n \"policy\": {\n \"timeout\": \"7.00:00:00\",\n \"retry\": 0,\n \"retryIntervalInSeconds\": 30,\n \"secureOutput\": false,\n \"secureInput\": false\n },\n \"userProperties\": [],\n \"typeProperties\": {\n \"storedProcedureName\": \"[[procfwk].[SetLogPipelineRunning]\",\n \"storedProcedureParameters\": {\n \"ExecutionId\": {\n \"value\": {\n \"value\": \"@pipeline().parameters.ExecutionId\",\n \"type\": \"Expression\"\n },\n \"type\": \"Guid\"\n },\n \"PipelineId\": {\n \"value\": {\n \"value\": \"@item().PipelineId\",\n \"type\": \"Expression\"\n },\n \"type\": \"Int32\"\n },\n \"StageId\": {\n \"value\": {\n \"value\": \"@pipeline().parameters.StageId\",\n \"type\": \"Expression\"\n },\n \"type\": \"Int32\"\n }\n }\n },\n \"linkedServiceName\": {\n \"referenceName\": \"SupportDatabase\",\n \"type\": \"LinkedServiceReference\"\n }\n },\n {\n \"name\": \"Get SPN Details\",\n \"description\": \"Return the SPN ID and Secret for the processing pipeline being executed. Called at this level as each pipeline can have a different SPN.\",\n \"type\": \"Lookup\",\n \"dependsOn\": [],\n \"policy\": {\n \"timeout\": \"7.00:00:00\",\n \"retry\": 0,\n \"retryIntervalInSeconds\": 30,\n \"secureOutput\": false,\n \"secureInput\": false\n },\n \"userProperties\": [],\n \"typeProperties\": {\n \"source\": {\n \"type\": \"AzureSqlSource\",\n \"sqlReaderStoredProcedureName\": \"[[procfwk].[GetServicePrincipal]\",\n \"storedProcedureParameters\": {\n \"DataFactory\": {\n \"type\": \"String\",\n \"value\": {\n \"value\": \"@item().DataFactoryName\",\n \"type\": \"Expression\"\n }\n },\n \"PipelineName\": {\n \"type\": \"String\",\n \"value\": {\n \"value\": \"@item().PipelineName\",\n \"type\": \"Expression\"\n }\n }\n },\n \"queryTimeout\": \"02:00:00\"\n },\n \"dataset\": {\n \"referenceName\": \"GetSetMetadata\",\n \"type\": \"DatasetReference\",\n \"parameters\": {}\n }\n }\n },\n {\n \"name\": \"Log Activity Failure\",\n \"description\": \"Handle true failures from calling out to the Azure Function and update the current execution table accordingly so a restart can occur.\",\n \"type\": \"SqlServerStoredProcedure\",\n \"dependsOn\": [\n {\n \"activity\": \"Execute Pipeline\",\n \"dependencyConditions\": [\n \"Failed\"\n ]\n }\n ],\n \"policy\": {\n \"timeout\": \"7.00:00:00\",\n \"retry\": 0,\n \"retryIntervalInSeconds\": 30,\n \"secureOutput\": false,\n \"secureInput\": false\n },\n \"userProperties\": [],\n \"typeProperties\": {\n \"storedProcedureName\": \"[[procfwk].[SetLogActivityFailed]\",\n \"storedProcedureParameters\": {\n \"ExecutionId\": {\n \"value\": {\n \"value\": \"@pipeline().parameters.ExecutionId\",\n \"type\": \"Expression\"\n },\n \"type\": \"Guid\"\n },\n \"PipelineId\": {\n \"value\": {\n \"value\": \"@item().PipelineId\",\n \"type\": \"Expression\"\n },\n \"type\": \"Int32\"\n },\n \"RunId\": {\n \"value\": {\n \"value\": \"@pipeline().RunId\",\n \"type\": \"Expression\"\n },\n \"type\": \"Guid\"\n },\n \"StageId\": {\n \"value\": {\n \"value\": \"@pipeline().parameters.StageId\",\n \"type\": \"Expression\"\n },\n \"type\": \"Int32\"\n }\n }\n },\n \"linkedServiceName\": {\n \"referenceName\": \"SupportDatabase\",\n \"type\": \"LinkedServiceReference\"\n }\n },\n {\n \"name\": \"Running Pipeline Handler\",\n \"type\": \"ExecutePipeline\",\n \"dependsOn\": [\n {\n \"activity\": \"Execute Pipeline\",\n \"dependencyConditions\": [\n \"Succeeded\"\n ]\n }\n ],\n \"userProperties\": [],\n \"typeProperties\": {\n \"pipeline\": {\n \"referenceName\": \"04-Infant\",\n \"type\": \"PipelineReference\"\n },\n \"waitOnCompletion\": true,\n \"parameters\": {\n \"tenantId\": {\n \"value\": \"@pipeline().parameters.TenantId\",\n \"type\": \"Expression\"\n },\n \"applicationId\": {\n \"value\": \"@activity('Get SPN Details').output.firstRow.Id\",\n \"type\": \"Expression\"\n },\n \"authenticationKey\": {\n \"value\": \"@activity('Get SPN Details').output.firstRow.Secret\",\n \"type\": \"Expression\"\n },\n \"subscriptionId\": {\n \"value\": \"@pipeline().parameters.SubscriptionId\",\n \"type\": \"Expression\"\n },\n \"resourceGroup\": {\n \"value\": \"@item().ResourceGroupName\",\n \"type\": \"Expression\"\n },\n \"factoryName\": {\n \"value\": \"@item().DataFactoryName\",\n \"type\": \"Expression\"\n },\n \"pipelineName\": {\n \"value\": \"@item().PipelineName\",\n \"type\": \"Expression\"\n },\n \"runId\": {\n \"value\": \"@activity('Execute Pipeline').output.RunId\",\n \"type\": \"Expression\"\n },\n \"executionId\": {\n \"value\": \"@pipeline().parameters.ExecutionId\",\n \"type\": \"Expression\"\n },\n \"stageId\": {\n \"value\": \"@pipeline().parameters.StageId\",\n \"type\": \"Expression\"\n },\n \"pipelineId\": {\n \"value\": \"@item().PipelineId\",\n \"type\": \"Expression\"\n }\n }\n }\n },\n {\n \"name\": \"Set Run Id\",\n \"description\": \"Provide the actual ADF run ID back to the current execution table for long term logging and alignment between the metadata other Azure monitoring tools.\",\n \"type\": \"SqlServerStoredProcedure\",\n \"dependsOn\": [\n {\n \"activity\": \"Execute Pipeline\",\n \"dependencyConditions\": [\n \"Succeeded\"\n ]\n }\n ],\n \"policy\": {\n \"timeout\": \"7.00:00:00\",\n \"retry\": 0,\n \"retryIntervalInSeconds\": 30,\n \"secureOutput\": false,\n \"secureInput\": false\n },\n \"userProperties\": [],\n \"typeProperties\": {\n \"storedProcedureName\": \"[[procfwk].[SetLogPipelineRunId]\",\n \"storedProcedureParameters\": {\n \"ExecutionId\": {\n \"value\": {\n \"value\": \"@pipeline().parameters.ExecutionId\",\n \"type\": \"Expression\"\n },\n \"type\": \"Guid\"\n },\n \"PipelineId\": {\n \"value\": {\n \"value\": \"@item().PipelineId\",\n \"type\": \"Expression\"\n },\n \"type\": \"Int32\"\n },\n \"RunId\": {\n \"value\": {\n \"value\": \"@activity('Execute Pipeline').output.RunId\",\n \"type\": \"Expression\"\n },\n \"type\": \"Guid\"\n },\n \"StageId\": {\n \"value\": {\n \"value\": \"@pipeline().parameters.StageId\",\n \"type\": \"Expression\"\n },\n \"type\": \"Int32\"\n }\n }\n },\n \"linkedServiceName\": {\n \"referenceName\": \"SupportDatabase\",\n \"type\": \"LinkedServiceReference\"\n }\n }\n ]\n }\n }\n ],\n \"parameters\": {\n \"StageId\": {\n \"type\": \"int\"\n },\n \"ExecutionId\": {\n \"type\": \"string\"\n },\n \"TenantId\": {\n \"type\": \"string\"\n },\n \"SubscriptionId\": {\n \"type\": \"string\"\n }\n },\n \"variables\": {\n \"FunctionBody\": {\n \"type\": \"String\"\n }\n },\n \"folder\": {\n \"name\": \"_ProcFwk\"\n },\n \"annotations\": [\n \"ADF.procfwk\"\n ]\n },\n \"dependsOn\": [\n \"[concat(variables('factoryId'), '/datasets/GetSetMetadata')]\",\n \"[concat(variables('factoryId'), '/linkedServices/PipelineExecutor')]\",\n \"[concat(variables('factoryId'), '/linkedServices/SupportDatabase')]\",\n \"[concat(variables('factoryId'), '/pipelines/04-Infant')]\"\n ]\n },\n {\n \"name\": \"[concat(parameters('factoryName'), '/04-Infant')]\",\n \"type\": \"Microsoft.DataFactory/factories/pipelines\",\n \"apiVersion\": \"2018-06-01\",\n \"properties\": {\n \"description\": \"ADF.procfwk infant pipeline used to check when the processing pipeline called by the Child completes and passes the resulting status back to the metadata database.\",\n \"activities\": [\n {\n \"name\": \"Wait Until Pipeline Completes\",\n \"description\": \"Loops until the pipeline called completes.\\n\\nSimple status:\\n- Running = new iteration.\\n- Done = break.\",\n \"type\": \"Until\",\n \"dependsOn\": [\n {\n \"activity\": \"Get Wait Duration\",\n \"dependencyConditions\": [\n \"Succeeded\"\n ]\n }\n ],\n \"userProperties\": [],\n \"typeProperties\": {\n \"expression\": {\n \"value\": \"@equals('Done',activity('Get Pipeline Status').output.SimpleStatus)\",\n \"type\": \"Expression\"\n },\n \"activities\": [\n {\n \"name\": \"Get Pipeline Status\",\n \"description\": \"Checks the status of a given processing pipeline and provides the value for the downstream framework activities to act upon.\",\n \"type\": \"AzureFunctionActivity\",\n \"dependsOn\": [],\n \"policy\": {\n \"timeout\": \"7.00:00:00\",\n \"retry\": 0,\n \"retryIntervalInSeconds\": 30,\n \"secureOutput\": false,\n \"secureInput\": false\n },\n \"userProperties\": [],\n \"typeProperties\": {\n \"functionName\": \"CheckPipelineStatus\",\n \"method\": \"POST\",\n \"headers\": {},\n \"body\": {\n \"value\": \"@concat('\\n{\\n \\\"tenantId\\\": \\\"',pipeline().parameters.TenantId,'\\\",\\n \\\"applicationId\\\": \\\"',pipeline().parameters.applicationId,'\\\",\\n \\\"authenticationKey\\\": \\\"',pipeline().parameters.authenticationKey,'\\\",\\n \\\"subscriptionId\\\": \\\"',pipeline().parameters.subscriptionId,'\\\",\\n \\\"resourceGroup\\\": \\\"',pipeline().parameters.resourceGroup,'\\\",\\n \\\"factoryName\\\": \\\"',pipeline().parameters.factoryName,'\\\",\\n \\\"pipelineName\\\": \\\"',pipeline().parameters.pipelineName,'\\\",\\n \\\"runId\\\": \\\"',pipeline().parameters.runId,'\\\"\\n}')\",\n \"type\": \"Expression\"\n }\n },\n \"linkedServiceName\": {\n \"referenceName\": \"PipelineExecutor\",\n \"type\": \"LinkedServiceReference\"\n }\n },\n {\n \"name\": \"Wait If Running\",\n \"description\": \"True = Do nothing.\\nFalse = Wait.\",\n \"type\": \"IfCondition\",\n \"dependsOn\": [\n {\n \"activity\": \"Get Pipeline Status\",\n \"dependencyConditions\": [\n \"Succeeded\"\n ]\n }\n ],\n \"userProperties\": [],\n \"typeProperties\": {\n \"expression\": {\n \"value\": \"@equals('Done',activity('Get Pipeline Status').output.SimpleStatus)\",\n \"type\": \"Expression\"\n },\n \"ifFalseActivities\": [\n {\n \"name\": \"Wait for Pipeline\",\n \"description\": \"The processing pipeline is still running so Wait before checking its status again.\",\n \"type\": \"Wait\",\n \"dependsOn\": [],\n \"userProperties\": [],\n \"typeProperties\": {\n \"waitTimeInSeconds\": {\n \"value\": \"@activity('Get Wait Duration').output.firstRow.PropertyValue\",\n \"type\": \"Expression\"\n }\n }\n }\n ]\n }\n },\n {\n \"name\": \"Wait to Retry Function\",\n \"type\": \"Wait\",\n \"dependsOn\": [\n {\n \"activity\": \"Get Pipeline Status\",\n \"dependencyConditions\": [\n \"Failed\"\n ]\n }\n ],\n \"userProperties\": [],\n \"typeProperties\": {\n \"waitTimeInSeconds\": {\n \"value\": \"@activity('Get Wait Duration').output.firstRow.PropertyValue\",\n \"type\": \"Expression\"\n }\n }\n },\n {\n \"name\": \"Set Last Update\",\n \"type\": \"SqlServerStoredProcedure\",\n \"dependsOn\": [\n {\n \"activity\": \"Get Pipeline Status\",\n \"dependencyConditions\": [\n \"Succeeded\"\n ]\n }\n ],\n \"policy\": {\n \"timeout\": \"7.00:00:00\",\n \"retry\": 0,\n \"retryIntervalInSeconds\": 30,\n \"secureOutput\": false,\n \"secureInput\": false\n },\n \"userProperties\": [],\n \"typeProperties\": {\n \"storedProcedureName\": \"[[procfwk].[SetLogPipelineLastStatusCheck]\",\n \"storedProcedureParameters\": {\n \"ExecutionId\": {\n \"value\": {\n \"value\": \"@pipeline().parameters.executionId\",\n \"type\": \"Expression\"\n },\n \"type\": \"Guid\"\n },\n \"PipelineId\": {\n \"value\": {\n \"value\": \"@pipeline().parameters.pipelineId\",\n \"type\": \"Expression\"\n },\n \"type\": \"Int32\"\n },\n \"StageId\": {\n \"value\": {\n \"value\": \"@pipeline().parameters.stageId\",\n \"type\": \"Expression\"\n },\n \"type\": \"Int32\"\n }\n }\n },\n \"linkedServiceName\": {\n \"referenceName\": \"SupportDatabase\",\n \"type\": \"LinkedServiceReference\"\n }\n }\n ],\n \"timeout\": \"7.00:00:00\"\n }\n },\n {\n \"name\": \"Pipeline Result\",\n \"description\": \"Receives the outcome from the function execution for a given processing pipeline and updates the current execution table with different pipelines status values depending on the result (case).\",\n \"type\": \"Switch\",\n \"dependsOn\": [\n {\n \"activity\": \"Wait Until Pipeline Completes\",\n \"dependencyConditions\": [\n \"Completed\"\n ]\n }\n ],\n \"userProperties\": [],\n \"typeProperties\": {\n \"on\": {\n \"value\": \"@activity('Get Pipeline Status').output.Status\",\n \"type\": \"Expression\"\n },\n \"cases\": [\n {\n \"value\": \"Succeeded\",\n \"activities\": [\n {\n \"name\": \"Pipeline Status Succeeded\",\n \"description\": \"Updates the current execution table with a pipeline status of success if the function outcome is succeeded.\",\n \"type\": \"SqlServerStoredProcedure\",\n \"dependsOn\": [],\n \"policy\": {\n \"timeout\": \"7.00:00:00\",\n \"retry\": 0,\n \"retryIntervalInSeconds\": 30,\n \"secureOutput\": false,\n \"secureInput\": false\n },\n \"userProperties\": [],\n \"typeProperties\": {\n \"storedProcedureName\": \"[[procfwk].[SetLogPipelineSuccess]\",\n \"storedProcedureParameters\": {\n \"ExecutionId\": {\n \"value\": {\n \"value\": \"@pipeline().parameters.executionId\",\n \"type\": \"Expression\"\n },\n \"type\": \"Guid\"\n },\n \"PipelineId\": {\n \"value\": {\n \"value\": \"@pipeline().parameters.pipelineId\",\n \"type\": \"Expression\"\n },\n \"type\": \"Int32\"\n },\n \"StageId\": {\n \"value\": {\n \"value\": \"@pipeline().parameters.stageId\",\n \"type\": \"Expression\"\n },\n \"type\": \"Int32\"\n }\n }\n },\n \"linkedServiceName\": {\n \"referenceName\": \"SupportDatabase\",\n \"type\": \"LinkedServiceReference\"\n }\n }\n ]\n },\n {\n \"value\": \"Failed\",\n \"activities\": [\n {\n \"name\": \"Pipeline Status Failed\",\n \"description\": \"Updates the current execution table with a pipeline status of failed if the function outcome is failed. Also blocks pipelines in the downstream execution stage.\",\n \"type\": \"SqlServerStoredProcedure\",\n \"dependsOn\": [],\n \"policy\": {\n \"timeout\": \"7.00:00:00\",\n \"retry\": 0,\n \"retryIntervalInSeconds\": 30,\n \"secureOutput\": false,\n \"secureInput\": false\n },\n \"userProperties\": [],\n \"typeProperties\": {\n \"storedProcedureName\": \"[[procfwk].[SetLogPipelineFailed]\",\n \"storedProcedureParameters\": {\n \"ExecutionId\": {\n \"value\": {\n \"value\": \"@pipeline().parameters.executionId\",\n \"type\": \"Expression\"\n },\n \"type\": \"Guid\"\n },\n \"PipelineId\": {\n \"value\": {\n \"value\": \"@pipeline().parameters.pipelineId\",\n \"type\": \"Expression\"\n },\n \"type\": \"Int32\"\n },\n \"RunId\": {\n \"value\": {\n \"value\": \"@activity('Get Pipeline Status').output.RunId\",\n \"type\": \"Expression\"\n },\n \"type\": \"Guid\"\n },\n \"StageId\": {\n \"value\": {\n \"value\": \"@pipeline().parameters.stageId\",\n \"type\": \"Expression\"\n },\n \"type\": \"Int32\"\n }\n }\n },\n \"linkedServiceName\": {\n \"referenceName\": \"SupportDatabase\",\n \"type\": \"LinkedServiceReference\"\n }\n }\n ]\n }\n ],\n \"defaultActivities\": [\n {\n \"name\": \"Pipeline Status Unknown\",\n \"description\": \"Updates the current execution table with a pipeline status of unknown if the function returns an unexpected outcome.\",\n \"type\": \"SqlServerStoredProcedure\",\n \"dependsOn\": [],\n \"policy\": {\n \"timeout\": \"7.00:00:00\",\n \"retry\": 0,\n \"retryIntervalInSeconds\": 30,\n \"secureOutput\": false,\n \"secureInput\": false\n },\n \"userProperties\": [],\n \"typeProperties\": {\n \"storedProcedureName\": \"[[procfwk].[SetLogPipelineUnknown]\",\n \"storedProcedureParameters\": {\n \"ExecutionId\": {\n \"value\": {\n \"value\": \"@pipeline().parameters.executionId\",\n \"type\": \"Expression\"\n },\n \"type\": \"Guid\"\n },\n \"PipelineId\": {\n \"value\": {\n \"value\": \"@pipeline().parameters.pipelineId\",\n \"type\": \"Expression\"\n },\n \"type\": \"Int32\"\n },\n \"StageId\": {\n \"value\": {\n \"value\": \"@pipeline().parameters.stageId\",\n \"type\": \"Expression\"\n },\n \"type\": \"Int32\"\n }\n }\n },\n \"linkedServiceName\": {\n \"referenceName\": \"SupportDatabase\",\n \"type\": \"LinkedServiceReference\"\n }\n }\n ]\n }\n },\n {\n \"name\": \"Get Wait Duration\",\n \"type\": \"Lookup\",\n \"dependsOn\": [],\n \"policy\": {\n \"timeout\": \"7.00:00:00\",\n \"retry\": 0,\n \"retryIntervalInSeconds\": 30,\n \"secureOutput\": false,\n \"secureInput\": false\n },\n \"userProperties\": [],\n \"typeProperties\": {\n \"source\": {\n \"type\": \"AzureSqlSource\",\n \"sqlReaderStoredProcedureName\": \"[[procfwk].[GetPropertyValue]\",\n \"storedProcedureParameters\": {\n \"PropertyName\": {\n \"type\": \"String\",\n \"value\": \"PipelineStatusCheckDuration\"\n }\n },\n \"queryTimeout\": \"02:00:00\"\n },\n \"dataset\": {\n \"referenceName\": \"GetSetMetadata\",\n \"type\": \"DatasetReference\",\n \"parameters\": {}\n }\n }\n }\n ],\n \"parameters\": {\n \"tenantId\": {\n \"type\": \"string\"\n },\n \"applicationId\": {\n \"type\": \"string\"\n },\n \"authenticationKey\": {\n \"type\": \"string\"\n },\n \"subscriptionId\": {\n \"type\": \"string\"\n },\n \"resourceGroup\": {\n \"type\": \"string\"\n },\n \"factoryName\": {\n \"type\": \"string\"\n },\n \"pipelineName\": {\n \"type\": \"string\"\n },\n \"runId\": {\n \"type\": \"string\"\n },\n \"executionId\": {\n \"type\": \"string\"\n },\n \"stageId\": {\n \"type\": \"int\"\n },\n \"pipelineId\": {\n \"type\": \"int\"\n }\n },\n \"folder\": {\n \"name\": \"_ProcFwk\"\n },\n \"annotations\": [\n \"ADF.procfwk\"\n ]\n },\n \"dependsOn\": [\n \"[concat(variables('factoryId'), '/datasets/GetSetMetadata')]\",\n \"[concat(variables('factoryId'), '/linkedServices/PipelineExecutor')]\",\n \"[concat(variables('factoryId'), '/linkedServices/SupportDatabase')]\"\n ]\n },\n {\n \"name\": \"[concat(parameters('factoryName'), '/Intentional Error')]\",\n \"type\": \"Microsoft.DataFactory/factories/pipelines\",\n \"apiVersion\": \"2018-06-01\",\n \"properties\": {\n \"description\": \"Used just so the ADF.procfwk has something to call during development.\",\n \"activities\": [\n {\n \"name\": \"Call Fail Procedure\",\n \"type\": \"SqlServerStoredProcedure\",\n \"dependsOn\": [\n {\n \"activity\": \"Wait1\",\n \"dependencyConditions\": [\n \"Succeeded\"\n ]\n }\n ],\n \"policy\": {\n \"timeout\": \"7.00:00:00\",\n \"retry\": 0,\n \"retryIntervalInSeconds\": 30,\n \"secureOutput\": false,\n \"secureInput\": false\n },\n \"userProperties\": [],\n \"typeProperties\": {\n \"storedProcedureName\": \"[[dbo].[FailProcedure]\",\n \"storedProcedureParameters\": {\n \"RaiseError\": {\n \"value\": {\n \"value\": \"@pipeline().parameters.RaiseErrors\",\n \"type\": \"Expression\"\n },\n \"type\": \"String\"\n }\n }\n },\n \"linkedServiceName\": {\n \"referenceName\": \"SupportDatabase\",\n \"type\": \"LinkedServiceReference\"\n }\n },\n {\n \"name\": \"Wait1\",\n \"type\": \"Wait\",\n \"dependsOn\": [],\n \"userProperties\": [],\n \"typeProperties\": {\n \"waitTimeInSeconds\": 10\n }\n },\n {\n \"name\": \"Call Fail Notebook\",\n \"type\": \"DatabricksNotebook\",\n \"dependsOn\": [\n {\n \"activity\": \"Wait1\",\n \"dependencyConditions\": [\n \"Succeeded\"\n ]\n }\n ],\n \"policy\": {\n \"timeout\": \"7.00:00:00\",\n \"retry\": 0,\n \"retryIntervalInSeconds\": 30,\n \"secureOutput\": false,\n \"secureInput\": false\n },\n \"userProperties\": [],\n \"typeProperties\": {\n \"notebookPath\": \"/Playground/Throw Exception\",\n \"baseParameters\": {\n \"RaiseError\": {\n \"value\": \"@pipeline().parameters.RaiseErrors\",\n \"type\": \"Expression\"\n }\n }\n },\n \"linkedServiceName\": {\n \"referenceName\": \"BricksOfData\",\n \"type\": \"LinkedServiceReference\"\n }\n }\n ],\n \"parameters\": {\n \"RaiseErrors\": {\n \"type\": \"string\",\n \"defaultValue\": \"false\"\n }\n },\n \"folder\": {\n \"name\": \"_Workers\"\n },\n \"annotations\": [\n \"Worker\"\n ]\n },\n \"dependsOn\": [\n \"[concat(variables('factoryId'), '/linkedServices/SupportDatabase')]\",\n \"[concat(variables('factoryId'), '/linkedServices/BricksOfData')]\"\n ]\n },\n {\n \"name\": \"[concat(parameters('factoryName'), '/Wait 1')]\",\n \"type\": \"Microsoft.DataFactory/factories/pipelines\",\n \"apiVersion\": \"2018-06-01\",\n \"properties\": {\n \"description\": \"Used just so the ADF.procfwk has something to call during development.\",\n \"activities\": [\n {\n \"name\": \"Wait1\",\n \"type\": \"Wait\",\n \"dependsOn\": [],\n \"userProperties\": [],\n \"typeProperties\": {\n \"waitTimeInSeconds\": {\n \"value\": \"@pipeline().parameters.WaitTime\",\n \"type\": \"Expression\"\n }\n }\n }\n ],\n \"parameters\": {\n \"WaitTime\": {\n \"type\": \"int\",\n \"defaultValue\": 5\n }\n },\n \"folder\": {\n \"name\": \"_Workers\"\n },\n \"annotations\": [\n \"Worker\"\n ]\n },\n \"dependsOn\": []\n },\n {\n \"name\": \"[concat(parameters('factoryName'), '/Wait 10')]\",\n \"type\": \"Microsoft.DataFactory/factories/pipelines\",\n \"apiVersion\": \"2018-06-01\",\n \"properties\": {\n \"description\": \"Used just so the ADF.procfwk has something to call during development.\",\n \"activities\": [\n {\n \"name\": \"Wait10\",\n \"type\": \"Wait\",\n \"dependsOn\": [],\n \"userProperties\": [],\n \"typeProperties\": {\n \"waitTimeInSeconds\": {\n \"value\": \"@pipeline().parameters.WaitTime\",\n \"type\": \"Expression\"\n }\n }\n }\n ],\n \"parameters\": {\n \"WaitTime\": {\n \"type\": \"int\",\n \"defaultValue\": 5\n }\n },\n \"folder\": {\n \"name\": \"_Workers\"\n },\n \"annotations\": [\n \"Worker\"\n ]\n },\n \"dependsOn\": []\n },\n {\n \"name\": \"[concat(parameters('factoryName'), '/Wait 2')]\",\n \"type\": \"Microsoft.DataFactory/factories/pipelines\",\n \"apiVersion\": \"2018-06-01\",\n \"properties\": {\n \"description\": \"Used just so the ADF.procfwk has something to call during development.\",\n \"activities\": [\n {\n \"name\": \"Wait2\",\n \"type\": \"Wait\",\n \"dependsOn\": [],\n \"userProperties\": [],\n \"typeProperties\": {\n \"waitTimeInSeconds\": {\n \"value\": \"@pipeline().parameters.WaitTime\",\n \"type\": \"Expression\"\n }\n }\n }\n ],\n \"parameters\": {\n \"WaitTime\": {\n \"type\": \"int\",\n \"defaultValue\": 5\n }\n },\n \"folder\": {\n \"name\": \"_Workers\"\n },\n \"annotations\": [\n \"Worker\"\n ]\n },\n \"dependsOn\": []\n },\n {\n \"name\": \"[concat(parameters('factoryName'), '/Wait 3')]\",\n \"type\": \"Microsoft.DataFactory/factories/pipelines\",\n \"apiVersion\": \"2018-06-01\",\n \"properties\": {\n \"description\": \"Used just so the ADF.procfwk has something to call during development.\",\n \"activities\": [\n {\n \"name\": \"Wait3\",\n \"type\": \"Wait\",\n \"dependsOn\": [],\n \"userProperties\": [],\n \"typeProperties\": {\n \"waitTimeInSeconds\": {\n \"value\": \"@pipeline().parameters.WaitTime\",\n \"type\": \"Expression\"\n }\n }\n }\n ],\n \"parameters\": {\n \"WaitTime\": {\n \"type\": \"int\",\n \"defaultValue\": 5\n }\n },\n \"folder\": {\n \"name\": \"_Workers\"\n },\n \"annotations\": [\n \"Worker\"\n ]\n },\n \"dependsOn\": []\n },\n {\n \"name\": \"[concat(parameters('factoryName'), '/Wait 4')]\",\n \"type\": \"Microsoft.DataFactory/factories/pipelines\",\n \"apiVersion\": \"2018-06-01\",\n \"properties\": {\n \"description\": \"Used just so the ADF.procfwk has something to call during development.\",\n \"activities\": [\n {\n \"name\": \"Wait4\",\n \"type\": \"Wait\",\n \"dependsOn\": [],\n \"userProperties\": [],\n \"typeProperties\": {\n \"waitTimeInSeconds\": {\n \"value\": \"@pipeline().parameters.WaitTime\",\n \"type\": \"Expression\"\n }\n }\n }\n ],\n \"parameters\": {\n \"WaitTime\": {\n \"type\": \"int\",\n \"defaultValue\": 5\n }\n },\n \"folder\": {\n \"name\": \"_Workers\"\n },\n \"annotations\": [\n \"Worker\"\n ]\n },\n \"dependsOn\": []\n },\n {\n \"name\": \"[concat(parameters('factoryName'), '/Wait 5')]\",\n \"type\": \"Microsoft.DataFactory/factories/pipelines\",\n \"apiVersion\": \"2018-06-01\",\n \"properties\": {\n \"description\": \"Used just so the ADF.procfwk has something to call during development.\",\n \"activities\": [\n {\n \"name\": \"Wait5\",\n \"type\": \"Wait\",\n \"dependsOn\": [],\n \"userProperties\": [],\n \"typeProperties\": {\n \"waitTimeInSeconds\": {\n \"value\": \"@pipeline().parameters.WaitTime\",\n \"type\": \"Expression\"\n }\n }\n }\n ],\n \"parameters\": {\n \"WaitTime\": {\n \"type\": \"int\",\n \"defaultValue\": 5\n }\n },\n \"folder\": {\n \"name\": \"_Workers\"\n },\n \"annotations\": [\n \"Worker\"\n ]\n },\n \"dependsOn\": []\n },\n {\n \"name\": \"[concat(parameters('factoryName'), '/Wait 6')]\",\n \"type\": \"Microsoft.DataFactory/factories/pipelines\",\n \"apiVersion\": \"2018-06-01\",\n \"properties\": {\n \"description\": \"Used just so the ADF.procfwk has something to call during development.\",\n \"activities\": [\n {\n \"name\": \"Wait6\",\n \"type\": \"Wait\",\n \"dependsOn\": [],\n \"userProperties\": [],\n \"typeProperties\": {\n \"waitTimeInSeconds\": {\n \"value\": \"@pipeline().parameters.WaitTime\",\n \"type\": \"Expression\"\n }\n }\n }\n ],\n \"parameters\": {\n \"WaitTime\": {\n \"type\": \"int\",\n \"defaultValue\": 5\n }\n },\n \"folder\": {\n \"name\": \"_Workers\"\n },\n \"annotations\": [\n \"Worker\"\n ]\n },\n \"dependsOn\": []\n },\n {\n \"name\": \"[concat(parameters('factoryName'), '/Wait 7')]\",\n \"type\": \"Microsoft.DataFactory/factories/pipelines\",\n \"apiVersion\": \"2018-06-01\",\n \"properties\": {\n \"description\": \"Used just so the ADF.procfwk has something to call during development.\",\n \"activities\": [\n {\n \"name\": \"Wait7\",\n \"type\": \"Wait\",\n \"dependsOn\": [],\n \"userProperties\": [],\n \"typeProperties\": {\n \"waitTimeInSeconds\": {\n \"value\": \"@pipeline().parameters.WaitTime\",\n \"type\": \"Expression\"\n }\n }\n }\n ],\n \"parameters\": {\n \"WaitTime\": {\n \"type\": \"int\",\n \"defaultValue\": 5\n }\n },\n \"folder\": {\n \"name\": \"_Workers\"\n },\n \"annotations\": [\n \"Worker\"\n ]\n },\n \"dependsOn\": []\n },\n {\n \"name\": \"[concat(parameters('factoryName'), '/Wait 8')]\",\n \"type\": \"Microsoft.DataFactory/factories/pipelines\",\n \"apiVersion\": \"2018-06-01\",\n \"properties\": {\n \"description\": \"Used just so the ADF.procfwk has something to call during development.\",\n \"activities\": [\n {\n \"name\": \"Wait8\",\n \"type\": \"Wait\",\n \"dependsOn\": [],\n \"userProperties\": [],\n \"typeProperties\": {\n \"waitTimeInSeconds\": {\n \"value\": \"@pipeline().parameters.WaitTime\",\n \"type\": \"Expression\"\n }\n }\n }\n ],\n \"parameters\": {\n \"WaitTime\": {\n \"type\": \"int\",\n \"defaultValue\": 5\n }\n },\n \"folder\": {\n \"name\": \"_Workers\"\n },\n \"annotations\": [\n \"Worker\"\n ]\n },\n \"dependsOn\": []\n },\n {\n \"name\": \"[concat(parameters('factoryName'), '/Wait 9')]\",\n \"type\": \"Microsoft.DataFactory/factories/pipelines\",\n \"apiVersion\": \"2018-06-01\",\n \"properties\": {\n \"description\": \"Used just so the ADF.procfwk has something to call during development.\",\n \"activities\": [\n {\n \"name\": \"Wait9\",\n \"type\": \"Wait\",\n \"dependsOn\": [],\n \"userProperties\": [],\n \"typeProperties\": {\n \"waitTimeInSeconds\": {\n \"value\": \"@pipeline().parameters.WaitTime\",\n \"type\": \"Expression\"\n }\n }\n }\n ],\n \"parameters\": {\n \"WaitTime\": {\n \"type\": \"int\",\n \"defaultValue\": 5\n }\n },\n \"folder\": {\n \"name\": \"_Workers\"\n },\n \"annotations\": [\n \"Worker\"\n ]\n },\n \"dependsOn\": []\n },\n {\n \"name\": \"[concat(parameters('factoryName'), '/GetSetMetadata')]\",\n \"type\": \"Microsoft.DataFactory/factories/datasets\",\n \"apiVersion\": \"2018-06-01\",\n \"properties\": {\n \"linkedServiceName\": {\n \"referenceName\": \"SupportDatabase\",\n \"type\": \"LinkedServiceReference\"\n },\n \"annotations\": [\n \"ADF.procfwk\"\n ],\n \"type\": \"AzureSqlTable\",\n \"schema\": [],\n \"typeProperties\": {}\n },\n \"dependsOn\": [\n \"[concat(variables('factoryId'), '/linkedServices/SupportDatabase')]\"\n ]\n },\n {\n \"name\": \"[concat(parameters('factoryName'), '/BricksOfData')]\",\n \"type\": \"Microsoft.DataFactory/factories/linkedServices\",\n \"apiVersion\": \"2018-06-01\",\n \"properties\": {\n \"annotations\": [],\n \"type\": \"AzureDatabricks\",\n \"typeProperties\": {\n \"domain\": \"https://northeurope.azuredatabricks.net\",\n \"accessToken\": {\n \"type\": \"AzureKeyVaultSecret\",\n \"store\": {\n \"referenceName\": \"Keys\",\n \"type\": \"LinkedServiceReference\"\n },\n \"secretName\": \"BricksOfDataToken\",\n \"secretVersion\": \"\"\n },\n \"existingClusterId\": \"0422-090117-slots899\"\n }\n },\n \"dependsOn\": [\n \"[concat(variables('factoryId'), '/linkedServices/Keys')]\"\n ]\n },\n {\n \"name\": \"[concat(parameters('factoryName'), '/Keys')]\",\n \"type\": \"Microsoft.DataFactory/factories/linkedServices\",\n \"apiVersion\": \"2018-06-01\",\n \"properties\": {\n \"description\": \"Connection to Key Vault for all other ADF linked service credentials required to run the processing framework.\",\n \"annotations\": [\n \"ADF.procfwk\"\n ],\n \"type\": \"AzureKeyVault\",\n \"typeProperties\": {\n \"baseUrl\": \"[parameters('Keys_properties_typeProperties_baseUrl')]\"\n }\n },\n \"dependsOn\": []\n },\n {\n \"name\": \"[concat(parameters('factoryName'), '/PipelineExecutor')]\",\n \"type\": \"Microsoft.DataFactory/factories/linkedServices\",\n \"apiVersion\": \"2018-06-01\",\n \"properties\": {\n \"description\": \"Connection to the Function App from ADF for calling the processing pipeline function within the orchestration framework.\",\n \"annotations\": [\n \"ADF.procfwk\"\n ],\n \"type\": \"AzureFunction\",\n \"typeProperties\": {\n \"functionAppUrl\": \"https://pipelineexecutor.azurewebsites.net\",\n \"functionKey\": {\n \"type\": \"AzureKeyVaultSecret\",\n \"store\": {\n \"referenceName\": \"Keys\",\n \"type\": \"LinkedServiceReference\"\n },\n \"secretName\": \"ExecutorFunctionAppKey\"\n }\n }\n },\n \"dependsOn\": [\n \"[concat(variables('factoryId'), '/linkedServices/Keys')]\"\n ]\n },\n {\n \"name\": \"[concat(parameters('factoryName'), '/SupportDatabase')]\",\n \"type\": \"Microsoft.DataFactory/factories/linkedServices\",\n \"apiVersion\": \"2018-06-01\",\n \"properties\": {\n \"description\": \"Connection between ADF and processing framework metadata SQLDB.\",\n \"annotations\": [\n \"ADF.procfwk\"\n ],\n \"type\": \"AzureSqlDatabase\",\n \"typeProperties\": {\n \"connectionString\": {\n \"type\": \"AzureKeyVaultSecret\",\n \"store\": {\n \"referenceName\": \"Keys\",\n \"type\": \"LinkedServiceReference\"\n },\n \"secretName\": \"[parameters('SupportDatabase_properties_typeProperties_connectionString_secretName')]\"\n }\n }\n },\n \"dependsOn\": [\n \"[concat(variables('factoryId'), '/linkedServices/Keys')]\"\n ]\n },\n {\n \"name\": \"[concat(parameters('factoryName'), '/FunctionalTestingTrigger')]\",\n \"type\": \"Microsoft.DataFactory/factories/triggers\",\n \"apiVersion\": \"2018-06-01\",\n \"properties\": {\n \"description\": \"Used for functional testing of the framework in a dedicated environment.\",\n \"annotations\": [\n \"ADF.procfwk\"\n ],\n \"runtimeState\": \"Stopped\",\n \"pipelines\": [\n {\n \"pipelineReference\": {\n \"referenceName\": \"01-Grandparent\",\n \"type\": \"PipelineReference\"\n },\n \"parameters\": {}\n }\n ],\n \"type\": \"ScheduleTrigger\",\n \"typeProperties\": {\n \"recurrence\": {\n \"frequency\": \"Hour\",\n \"interval\": 2,\n \"startTime\": \"2020-04-06T15:00:00.000Z\",\n \"timeZone\": \"UTC\"\n }\n }\n },\n \"dependsOn\": [\n \"[concat(variables('factoryId'), '/pipelines/01-Grandparent')]\"\n ]\n }\n ]\n}"
},
{
"path": "ARM Templates/Data Factory/v1.6 Export.json",
"content": "{\n \"$schema\": \"http://schema.management.azure.com/schemas/2015-01-01/deploymentTemplate.json#\",\n \"contentVersion\": \"1.0.0.0\",\n \"parameters\": {\n \"factoryName\": {\n \"type\": \"string\",\n \"metadata\": \"Data Factory name\",\n \"defaultValue\": \"\"\n },\n \"Keys_properties_typeProperties_baseUrl\": {\n \"type\": \"string\",\n \"defaultValue\": \"\"\n },\n \"SupportDatabase_properties_typeProperties_connectionString_secretName\": {\n \"type\": \"string\",\n \"defaultValue\": \"\"\n }\n },\n \"variables\": {\n \"factoryId\": \"[concat('Microsoft.DataFactory/factories/', parameters('factoryName'))]\"\n },\n \"resources\": [\n {\n \"name\": \"[concat(parameters('factoryName'), '/01-Grandparent')]\",\n \"type\": \"Microsoft.DataFactory/factories/pipelines\",\n \"apiVersion\": \"2018-06-01\",\n \"properties\": {\n \"description\": \"ADF.procfwk grandparent pipeline used optionally to bootstrap any wider processes in your Data Factory that then calls the processing framework.\",\n \"activities\": [\n {\n \"name\": \"Framework Processing\",\n \"type\": \"ExecutePipeline\",\n \"dependsOn\": [\n {\n \"activity\": \"Set Random Waits\",\n \"dependencyConditions\": [\n \"Succeeded\"\n ]\n }\n ],\n \"userProperties\": [],\n \"typeProperties\": {\n \"pipeline\": {\n \"referenceName\": \"02-Parent\",\n \"type\": \"PipelineReference\"\n },\n \"waitOnCompletion\": true,\n \"parameters\": {}\n }\n },\n {\n \"name\": \"Set Random Waits\",\n \"description\": \"For functional testing only.\",\n \"type\": \"SqlServerStoredProcedure\",\n \"dependsOn\": [],\n \"policy\": {\n \"timeout\": \"7.00:00:00\",\n \"retry\": 0,\n \"retryIntervalInSeconds\": 30,\n \"secureOutput\": false,\n \"secureInput\": false\n },\n \"userProperties\": [],\n \"typeProperties\": {\n \"storedProcedureName\": \"[[dbo].[SetRandomWaitValues]\"\n },\n \"linkedServiceName\": {\n \"referenceName\": \"SupportDatabase\",\n \"type\": \"LinkedServiceReference\"\n }\n }\n ],\n \"folder\": {\n \"name\": \"_ProcFwk\"\n },\n \"annotations\": [\n \"ADF.procfwk\"\n ]\n },\n \"dependsOn\": [\n \"[concat(variables('factoryId'), '/pipelines/02-Parent')]\",\n \"[concat(variables('factoryId'), '/linkedServices/SupportDatabase')]\"\n ]\n },\n {\n \"name\": \"[concat(parameters('factoryName'), '/02-Parent')]\",\n \"type\": \"Microsoft.DataFactory/factories/pipelines\",\n \"apiVersion\": \"2018-06-01\",\n \"properties\": {\n \"description\": \"ADF.procfwk parent pipeline used to bootstrap the orchestration framework in perform the first level ForEach calls in sequence for the metadata stages.\",\n \"activities\": [\n {\n \"name\": \"Get Stages\",\n \"description\": \"Returns a distinct list of execution stages within the framework metadata.\",\n \"type\": \"Lookup\",\n \"dependsOn\": [\n {\n \"activity\": \"Execution Wrapper\",\n \"dependencyConditions\": [\n \"Succeeded\"\n ]\n }\n ],\n \"policy\": {\n \"timeout\": \"7.00:00:00\",\n \"retry\": 0,\n \"retryIntervalInSeconds\": 30,\n \"secureOutput\": false,\n \"secureInput\": false\n },\n \"userProperties\": [],\n \"typeProperties\": {\n \"source\": {\n \"type\": \"AzureSqlSource\",\n \"sqlReaderStoredProcedureName\": \"[[procfwk].[GetStages]\",\n \"storedProcedureParameters\": {\n \"ExecutionId\": {\n \"type\": \"Guid\",\n \"value\": {\n \"value\": \"@activity('Execution Wrapper').output.firstRow.ExecutionId\",\n \"type\": \"Expression\"\n }\n }\n },\n \"queryTimeout\": \"02:00:00\"\n },\n \"dataset\": {\n \"referenceName\": \"GetSetMetadata\",\n \"type\": \"DatasetReference\",\n \"parameters\": {}\n },\n \"firstRowOnly\": false\n }\n },\n {\n \"name\": \"Execute Stages\",\n \"description\": \"Top level ForEach to sequentially call all processing stages within the framework metadata. Items for iteration passed from the Get Stages lookup activity.\",\n \"type\": \"ForEach\",\n \"dependsOn\": [\n {\n \"activity\": \"Get Stages\",\n \"dependencyConditions\": [\n \"Succeeded\"\n ]\n },\n {\n \"activity\": \"Get Tenant Id\",\n \"dependencyConditions\": [\n \"Succeeded\"\n ]\n },\n {\n \"activity\": \"Get Subscription Id\",\n \"dependencyConditions\": [\n \"Succeeded\"\n ]\n }\n ],\n \"userProperties\": [],\n \"typeProperties\": {\n \"items\": {\n \"value\": \"@activity('Get Stages').output.value\",\n \"type\": \"Expression\"\n },\n \"isSequential\": true,\n \"activities\": [\n {\n \"name\": \"Stage Executor\",\n \"description\": \"Call to the framework generic child pipeline for a given execution stage.\",\n \"type\": \"ExecutePipeline\",\n \"dependsOn\": [\n {\n \"activity\": \"Log Stage Preparing\",\n \"dependencyConditions\": [\n \"Succeeded\"\n ]\n }\n ],\n \"userProperties\": [],\n \"typeProperties\": {\n \"pipeline\": {\n \"referenceName\": \"03-Child\",\n \"type\": \"PipelineReference\"\n },\n \"waitOnCompletion\": true,\n \"parameters\": {\n \"StageId\": {\n \"value\": \"@item().StageId\",\n \"type\": \"Expression\"\n },\n \"ExecutionId\": {\n \"value\": \"@activity('Execution Wrapper').output.firstRow.ExecutionId\",\n \"type\": \"Expression\"\n },\n \"TenantId\": {\n \"value\": \"@activity('Get Tenant Id').output.firstRow.PropertyValue\",\n \"type\": \"Expression\"\n },\n \"SubscriptionId\": {\n \"value\": \"@activity('Get Subscription Id').output.firstRow.PropertyValue\",\n \"type\": \"Expression\"\n }\n }\n }\n },\n {\n \"name\": \"Log Stage Preparing\",\n \"description\": \"Update the current execution table flagging all pipelines within the stage as preparing.\",\n \"type\": \"SqlServerStoredProcedure\",\n \"dependsOn\": [\n {\n \"activity\": \"Check for Blockers\",\n \"dependencyConditions\": [\n \"Succeeded\"\n ]\n }\n ],\n \"policy\": {\n \"timeout\": \"7.00:00:00\",\n \"retry\": 0,\n \"retryIntervalInSeconds\": 30,\n \"secureOutput\": false,\n \"secureInput\": false\n },\n \"userProperties\": [],\n \"typeProperties\": {\n \"storedProcedureName\": \"[[procfwk].[SetLogStagePreparing]\",\n \"storedProcedureParameters\": {\n \"ExecutionId\": {\n \"value\": {\n \"value\": \"@activity('Execution Wrapper').output.firstRow.ExecutionId\",\n \"type\": \"Expression\"\n },\n \"type\": \"Guid\"\n },\n \"StageId\": {\n \"value\": {\n \"value\": \"@item().StageId\",\n \"type\": \"Expression\"\n },\n \"type\": \"Int32\"\n }\n }\n },\n \"linkedServiceName\": {\n \"referenceName\": \"SupportDatabase\",\n \"type\": \"LinkedServiceReference\"\n }\n },\n {\n \"name\": \"Check for Blockers\",\n \"description\": \"Used to double check and stop the next execution stage if failures and blockers have be incurred. Without this step processing would continue regardless of upstream failures.\",\n \"type\": \"SqlServerStoredProcedure\",\n \"dependsOn\": [],\n \"policy\": {\n \"timeout\": \"7.00:00:00\",\n \"retry\": 0,\n \"retryIntervalInSeconds\": 30,\n \"secureOutput\": false,\n \"secureInput\": false\n },\n \"userProperties\": [],\n \"typeProperties\": {\n \"storedProcedureName\": \"[[procfwk].[CheckForBlockedPipelines]\",\n \"storedProcedureParameters\": {\n \"StageId\": {\n \"value\": {\n \"value\": \"@item().StageId\",\n \"type\": \"Expression\"\n },\n \"type\": \"Int32\"\n }\n }\n },\n \"linkedServiceName\": {\n \"referenceName\": \"SupportDatabase\",\n \"type\": \"LinkedServiceReference\"\n }\n }\n ]\n }\n },\n {\n \"name\": \"Execution Wrapper\",\n \"description\": \"Wrapper to reset and restart processing or create a completely new execution instance of the framework metadata.\",\n \"type\": \"Lookup\",\n \"dependsOn\": [\n {\n \"activity\": \"Metadata Integrity Checks\",\n \"dependencyConditions\": [\n \"Succeeded\"\n ]\n }\n ],\n \"policy\": {\n \"timeout\": \"7.00:00:00\",\n \"retry\": 0,\n \"retryIntervalInSeconds\": 30,\n \"secureOutput\": false,\n \"secureInput\": false\n },\n \"userProperties\": [],\n \"typeProperties\": {\n \"source\": {\n \"type\": \"AzureSqlSource\",\n \"sqlReaderStoredProcedureName\": \"[[procfwk].[ExecutionWrapper]\",\n \"storedProcedureParameters\": {\n \"CallingDataFactory\": {\n \"type\": \"String\",\n \"value\": {\n \"value\": \"@pipeline().DataFactory\",\n \"type\": \"Expression\"\n }\n }\n },\n \"queryTimeout\": \"02:00:00\"\n },\n \"dataset\": {\n \"referenceName\": \"GetSetMetadata\",\n \"type\": \"DatasetReference\",\n \"parameters\": {}\n }\n }\n },\n {\n \"name\": \"Archive Execution Log\",\n \"description\": \"After a successful execution run the current execution metadata is moved to the long term logging table by this stored procedure call.\",\n \"type\": \"SqlServerStoredProcedure\",\n \"dependsOn\": [\n {\n \"activity\": \"Execute Stages\",\n \"dependencyConditions\": [\n \"Succeeded\"\n ]\n }\n ],\n \"policy\": {\n \"timeout\": \"7.00:00:00\",\n \"retry\": 0,\n \"retryIntervalInSeconds\": 30,\n \"secureOutput\": false,\n \"secureInput\": false\n },\n \"userProperties\": [],\n \"typeProperties\": {\n \"storedProcedureName\": \"[[procfwk].[UpdateExecutionLog]\"\n },\n \"linkedServiceName\": {\n \"referenceName\": \"SupportDatabase\",\n \"type\": \"LinkedServiceReference\"\n }\n },\n {\n \"name\": \"Get Tenant Id\",\n \"description\": \"Returning the Azure Tenant Id from the metadata properties table.\",\n \"type\": \"Lookup\",\n \"dependsOn\": [\n {\n \"activity\": \"Metadata Integrity Checks\",\n \"dependencyConditions\": [\n \"Succeeded\"\n ]\n }\n ],\n \"policy\": {\n \"timeout\": \"7.00:00:00\",\n \"retry\": 0,\n \"retryIntervalInSeconds\": 30,\n \"secureOutput\": false,\n \"secureInput\": false\n },\n \"userProperties\": [],\n \"typeProperties\": {\n \"source\": {\n \"type\": \"AzureSqlSource\",\n \"sqlReaderStoredProcedureName\": \"[[procfwk].[GetPropertyValue]\",\n \"storedProcedureParameters\": {\n \"PropertyName\": {\n \"type\": \"String\",\n \"value\": \"TenantId\"\n }\n },\n \"queryTimeout\": \"02:00:00\"\n },\n \"dataset\": {\n \"referenceName\": \"GetSetMetadata\",\n \"type\": \"DatasetReference\",\n \"parameters\": {}\n },\n \"firstRowOnly\": true\n }\n },\n {\n \"name\": \"Get Subscription Id\",\n \"description\": \"Returning the Azure Subscription Id from the metadata properties table.\",\n \"type\": \"Lookup\",\n \"dependsOn\": [\n {\n \"activity\": \"Metadata Integrity Checks\",\n \"dependencyConditions\": [\n \"Succeeded\"\n ]\n }\n ],\n \"policy\": {\n \"timeout\": \"7.00:00:00\",\n \"retry\": 0,\n \"retryIntervalInSeconds\": 30,\n \"secureOutput\": false,\n \"secureInput\": true\n },\n \"userProperties\": [],\n \"typeProperties\": {\n \"source\": {\n \"type\": \"AzureSqlSource\",\n \"sqlReaderStoredProcedureName\": \"[[procfwk].[GetPropertyValue]\",\n \"storedProcedureParameters\": {\n \"PropertyName\": {\n \"type\": \"String\",\n \"value\": \"SubscriptionId\"\n }\n },\n \"queryTimeout\": \"02:00:00\"\n },\n \"dataset\": {\n \"referenceName\": \"GetSetMetadata\",\n \"type\": \"DatasetReference\",\n \"parameters\": {}\n }\n }\n },\n {\n \"name\": \"Metadata Integrity Checks\",\n \"description\": \"Performs a series of checks on all metadata held in the framework SQLDB. This is intended to raise errors before an execution run even starts.\",\n \"type\": \"SqlServerStoredProcedure\",\n \"dependsOn\": [],\n \"policy\": {\n \"timeout\": \"7.00:00:00\",\n \"retry\": 0,\n \"retryIntervalInSeconds\": 30,\n \"secureOutput\": false,\n \"secureInput\": false\n },\n \"userProperties\": [],\n \"typeProperties\": {\n \"storedProcedureName\": \"[[procfwk].[CheckMetadataIntegrity]\",\n \"storedProcedureParameters\": {\n \"DebugMode\": {\n \"value\": \"false\",\n \"type\": \"Boolean\"\n }\n }\n },\n \"linkedServiceName\": {\n \"referenceName\": \"SupportDatabase\",\n \"type\": \"LinkedServiceReference\"\n }\n }\n ],\n \"folder\": {\n \"name\": \"_ProcFwk\"\n },\n \"annotations\": [\n \"ADF.procfwk\"\n ]\n },\n \"dependsOn\": [\n \"[concat(variables('factoryId'), '/datasets/GetSetMetadata')]\",\n \"[concat(variables('factoryId'), '/linkedServices/SupportDatabase')]\",\n \"[concat(variables('factoryId'), '/pipelines/03-Child')]\"\n ]\n },\n {\n \"name\": \"[concat(parameters('factoryName'), '/03-Child')]\",\n \"type\": \"Microsoft.DataFactory/factories/pipelines\",\n \"apiVersion\": \"2018-06-01\",\n \"properties\": {\n \"description\": \"ADF.procfwk child pipeline used to execute Worker pipelines within a given execution stage. This pipeline will be called once for each stage, then execute all Workers in parallel.\",\n \"activities\": [\n {\n \"name\": \"Get Pipelines\",\n \"description\": \"Returns all pipelines from the metadata to be executed within a given processing stage.\",\n \"type\": \"Lookup\",\n \"dependsOn\": [],\n \"policy\": {\n \"timeout\": \"7.00:00:00\",\n \"retry\": 0,\n \"retryIntervalInSeconds\": 30,\n \"secureOutput\": false,\n \"secureInput\": false\n },\n \"userProperties\": [],\n \"typeProperties\": {\n \"source\": {\n \"type\": \"AzureSqlSource\",\n \"sqlReaderStoredProcedureName\": \"[[procfwk].[GetPipelinesInStage]\",\n \"storedProcedureParameters\": {\n \"StageId\": {\n \"type\": \"Int32\",\n \"value\": {\n \"value\": \"@pipeline().parameters.StageId\",\n \"type\": \"Expression\"\n }\n }\n },\n \"queryTimeout\": \"02:00:00\"\n },\n \"dataset\": {\n \"referenceName\": \"GetSetMetadata\",\n \"type\": \"DatasetReference\",\n \"parameters\": {}\n },\n \"firstRowOnly\": false\n }\n },\n {\n \"name\": \"Execute Pipelines\",\n \"description\": \"Second level ForEach to run in parallel all pipelines within the stage. Items for iteration passed from the Get Pipelines lookup activity.\",\n \"type\": \"ForEach\",\n \"dependsOn\": [\n {\n \"activity\": \"Get Pipelines\",\n \"dependencyConditions\": [\n \"Succeeded\"\n ]\n }\n ],\n \"userProperties\": [],\n \"typeProperties\": {\n \"items\": {\n \"value\": \"@activity('Get Pipelines').output.value\",\n \"type\": \"Expression\"\n },\n \"isSequential\": false,\n \"activities\": [\n {\n \"name\": \"Execute Pipeline\",\n \"description\": \"The lowest level executor with the metadata framework to call existing processing pipelines within Data Factory. The function called will block processing and wait for an outcome.\",\n \"type\": \"AzureFunctionActivity\",\n \"dependsOn\": [\n {\n \"activity\": \"Log Pipeline Running\",\n \"dependencyConditions\": [\n \"Succeeded\"\n ]\n },\n {\n \"activity\": \"Get Pipeline Params\",\n \"dependencyConditions\": [\n \"Succeeded\"\n ]\n },\n {\n \"activity\": \"Get SPN Details\",\n \"dependencyConditions\": [\n \"Succeeded\"\n ]\n }\n ],\n \"policy\": {\n \"timeout\": \"7.00:00:00\",\n \"retry\": 0,\n \"retryIntervalInSeconds\": 30,\n \"secureOutput\": false,\n \"secureInput\": false\n },\n \"userProperties\": [],\n \"typeProperties\": {\n \"functionName\": \"ExecutePipeline\",\n \"method\": \"POST\",\n \"headers\": {},\n \"body\": {\n \"value\": \"@concat('\\n{\\n\\t\\\"tenantId\\\": \\\"',pipeline().parameters.TenantId,'\\\",\\n\\t\\\"applicationId\\\": \\\"',activity('Get SPN Details').output.firstRow.Id,'\\\",\\n\\t\\\"authenticationKey\\\": \\\"',activity('Get SPN Details').output.firstRow.Secret,'\\\",\\n\\t\\\"subscriptionId\\\": \\\"',pipeline().parameters.SubscriptionId,'\\\",\\n\\t\\\"resourceGroup\\\": \\\"',item().ResourceGroupName,'\\\",\\n\\t\\\"factoryName\\\": \\\"',item().DataFactoryName,'\\\",\\n\\t\\\"pipelineName\\\": \\\"',item().PipelineName,'\\\"',activity('Get Pipeline Params').output.firstRow.Params,'\\n}')\",\n \"type\": \"Expression\"\n }\n },\n \"linkedServiceName\": {\n \"referenceName\": \"FrameworkFunctions\",\n \"type\": \"LinkedServiceReference\"\n }\n },\n {\n \"name\": \"Get Pipeline Params\",\n \"description\": \"Returns any parameters from metadata required for the processing pipeline being called. The output can be an empty string if no parameters are required.\",\n \"type\": \"Lookup\",\n \"dependsOn\": [],\n \"policy\": {\n \"timeout\": \"7.00:00:00\",\n \"retry\": 0,\n \"retryIntervalInSeconds\": 30,\n \"secureOutput\": false,\n \"secureInput\": false\n },\n \"userProperties\": [],\n \"typeProperties\": {\n \"source\": {\n \"type\": \"AzureSqlSource\",\n \"sqlReaderStoredProcedureName\": \"[[procfwk].[GetPipelineParameters]\",\n \"storedProcedureParameters\": {\n \"PipelineId\": {\n \"type\": \"Int32\",\n \"value\": {\n \"value\": \"@item().PipelineId\",\n \"type\": \"Expression\"\n }\n }\n },\n \"queryTimeout\": \"02:00:00\"\n },\n \"dataset\": {\n \"referenceName\": \"GetSetMetadata\",\n \"type\": \"DatasetReference\",\n \"parameters\": {}\n }\n }\n },\n {\n \"name\": \"Log Pipeline Running\",\n \"description\": \"Sets the current pipeline with a status of running within the current execution database table.\",\n \"type\": \"SqlServerStoredProcedure\",\n \"dependsOn\": [],\n \"policy\": {\n \"timeout\": \"7.00:00:00\",\n \"retry\": 0,\n \"retryIntervalInSeconds\": 30,\n \"secureOutput\": false,\n \"secureInput\": false\n },\n \"userProperties\": [],\n \"typeProperties\": {\n \"storedProcedureName\": \"[[procfwk].[SetLogPipelineRunning]\",\n \"storedProcedureParameters\": {\n \"ExecutionId\": {\n \"value\": {\n \"value\": \"@pipeline().parameters.ExecutionId\",\n \"type\": \"Expression\"\n },\n \"type\": \"Guid\"\n },\n \"PipelineId\": {\n \"value\": {\n \"value\": \"@item().PipelineId\",\n \"type\": \"Expression\"\n },\n \"type\": \"Int32\"\n },\n \"StageId\": {\n \"value\": {\n \"value\": \"@pipeline().parameters.StageId\",\n \"type\": \"Expression\"\n },\n \"type\": \"Int32\"\n }\n }\n },\n \"linkedServiceName\": {\n \"referenceName\": \"SupportDatabase\",\n \"type\": \"LinkedServiceReference\"\n }\n },\n {\n \"name\": \"Get SPN Details\",\n \"description\": \"Return the SPN ID and Secret for the processing pipeline being executed. Called at this level as each pipeline can have a different SPN.\",\n \"type\": \"Lookup\",\n \"dependsOn\": [],\n \"policy\": {\n \"timeout\": \"7.00:00:00\",\n \"retry\": 0,\n \"retryIntervalInSeconds\": 30,\n \"secureOutput\": false,\n \"secureInput\": false\n },\n \"userProperties\": [],\n \"typeProperties\": {\n \"source\": {\n \"type\": \"AzureSqlSource\",\n \"sqlReaderStoredProcedureName\": \"[[procfwk].[GetServicePrincipal]\",\n \"storedProcedureParameters\": {\n \"DataFactory\": {\n \"type\": \"String\",\n \"value\": {\n \"value\": \"@item().DataFactoryName\",\n \"type\": \"Expression\"\n }\n },\n \"PipelineName\": {\n \"type\": \"String\",\n \"value\": {\n \"value\": \"@item().PipelineName\",\n \"type\": \"Expression\"\n }\n }\n },\n \"queryTimeout\": \"02:00:00\"\n },\n \"dataset\": {\n \"referenceName\": \"GetSetMetadata\",\n \"type\": \"DatasetReference\",\n \"parameters\": {}\n }\n }\n },\n {\n \"name\": \"Log Activity Failure\",\n \"description\": \"Handle true failures from calling out to the Azure Function and update the current execution table accordingly so a restart can occur.\",\n \"type\": \"SqlServerStoredProcedure\",\n \"dependsOn\": [\n {\n \"activity\": \"Execute Pipeline\",\n \"dependencyConditions\": [\n \"Failed\"\n ]\n }\n ],\n \"policy\": {\n \"timeout\": \"7.00:00:00\",\n \"retry\": 0,\n \"retryIntervalInSeconds\": 30,\n \"secureOutput\": false,\n \"secureInput\": false\n },\n \"userProperties\": [],\n \"typeProperties\": {\n \"storedProcedureName\": \"[[procfwk].[SetLogActivityFailed]\",\n \"storedProcedureParameters\": {\n \"ExecutionId\": {\n \"value\": {\n \"value\": \"@pipeline().parameters.ExecutionId\",\n \"type\": \"Expression\"\n },\n \"type\": \"Guid\"\n },\n \"PipelineId\": {\n \"value\": {\n \"value\": \"@item().PipelineId\",\n \"type\": \"Expression\"\n },\n \"type\": \"Int32\"\n },\n \"StageId\": {\n \"value\": {\n \"value\": \"@pipeline().parameters.StageId\",\n \"type\": \"Expression\"\n },\n \"type\": \"Int32\"\n }\n }\n },\n \"linkedServiceName\": {\n \"referenceName\": \"SupportDatabase\",\n \"type\": \"LinkedServiceReference\"\n }\n },\n {\n \"name\": \"Running Pipeline Handler\",\n \"type\": \"ExecutePipeline\",\n \"dependsOn\": [\n {\n \"activity\": \"Execute Pipeline\",\n \"dependencyConditions\": [\n \"Succeeded\"\n ]\n }\n ],\n \"userProperties\": [],\n \"typeProperties\": {\n \"pipeline\": {\n \"referenceName\": \"04-Infant\",\n \"type\": \"PipelineReference\"\n },\n \"waitOnCompletion\": true,\n \"parameters\": {\n \"tenantId\": {\n \"value\": \"@pipeline().parameters.TenantId\",\n \"type\": \"Expression\"\n },\n \"applicationId\": {\n \"value\": \"@activity('Get SPN Details').output.firstRow.Id\",\n \"type\": \"Expression\"\n },\n \"authenticationKey\": {\n \"value\": \"@activity('Get SPN Details').output.firstRow.Secret\",\n \"type\": \"Expression\"\n },\n \"subscriptionId\": {\n \"value\": \"@pipeline().parameters.SubscriptionId\",\n \"type\": \"Expression\"\n },\n \"resourceGroup\": {\n \"value\": \"@item().ResourceGroupName\",\n \"type\": \"Expression\"\n },\n \"factoryName\": {\n \"value\": \"@item().DataFactoryName\",\n \"type\": \"Expression\"\n },\n \"pipelineName\": {\n \"value\": \"@item().PipelineName\",\n \"type\": \"Expression\"\n },\n \"runId\": {\n \"value\": \"@activity('Execute Pipeline').output.RunId\",\n \"type\": \"Expression\"\n },\n \"executionId\": {\n \"value\": \"@pipeline().parameters.ExecutionId\",\n \"type\": \"Expression\"\n },\n \"stageId\": {\n \"value\": \"@pipeline().parameters.StageId\",\n \"type\": \"Expression\"\n },\n \"pipelineId\": {\n \"value\": \"@item().PipelineId\",\n \"type\": \"Expression\"\n }\n }\n }\n },\n {\n \"name\": \"Set Run Id\",\n \"description\": \"Provide the actual ADF run ID back to the current execution table for long term logging and alignment between the metadata other Azure monitoring tools.\",\n \"type\": \"SqlServerStoredProcedure\",\n \"dependsOn\": [\n {\n \"activity\": \"Execute Pipeline\",\n \"dependencyConditions\": [\n \"Succeeded\"\n ]\n }\n ],\n \"policy\": {\n \"timeout\": \"7.00:00:00\",\n \"retry\": 0,\n \"retryIntervalInSeconds\": 30,\n \"secureOutput\": false,\n \"secureInput\": false\n },\n \"userProperties\": [],\n \"typeProperties\": {\n \"storedProcedureName\": \"[[procfwk].[SetLogPipelineRunId]\",\n \"storedProcedureParameters\": {\n \"ExecutionId\": {\n \"value\": {\n \"value\": \"@pipeline().parameters.ExecutionId\",\n \"type\": \"Expression\"\n },\n \"type\": \"Guid\"\n },\n \"PipelineId\": {\n \"value\": {\n \"value\": \"@item().PipelineId\",\n \"type\": \"Expression\"\n },\n \"type\": \"Int32\"\n },\n \"RunId\": {\n \"value\": {\n \"value\": \"@activity('Execute Pipeline').output.RunId\",\n \"type\": \"Expression\"\n },\n \"type\": \"Guid\"\n },\n \"StageId\": {\n \"value\": {\n \"value\": \"@pipeline().parameters.StageId\",\n \"type\": \"Expression\"\n },\n \"type\": \"Int32\"\n }\n }\n },\n \"linkedServiceName\": {\n \"referenceName\": \"SupportDatabase\",\n \"type\": \"LinkedServiceReference\"\n }\n }\n ]\n }\n }\n ],\n \"parameters\": {\n \"StageId\": {\n \"type\": \"int\"\n },\n \"ExecutionId\": {\n \"type\": \"string\"\n },\n \"TenantId\": {\n \"type\": \"string\"\n },\n \"SubscriptionId\": {\n \"type\": \"string\"\n }\n },\n \"variables\": {\n \"FunctionBody\": {\n \"type\": \"String\"\n }\n },\n \"folder\": {\n \"name\": \"_ProcFwk\"\n },\n \"annotations\": [\n \"ADF.procfwk\"\n ]\n },\n \"dependsOn\": [\n \"[concat(variables('factoryId'), '/datasets/GetSetMetadata')]\",\n \"[concat(variables('factoryId'), '/linkedServices/FrameworkFunctions')]\",\n \"[concat(variables('factoryId'), '/linkedServices/SupportDatabase')]\",\n \"[concat(variables('factoryId'), '/pipelines/04-Infant')]\"\n ]\n },\n {\n \"name\": \"[concat(parameters('factoryName'), '/04-Infant')]\",\n \"type\": \"Microsoft.DataFactory/factories/pipelines\",\n \"apiVersion\": \"2018-06-01\",\n \"properties\": {\n \"description\": \"ADF.procfwk infant pipeline used to check when the processing pipeline called by the Child completes and passes the resulting status back to the metadata database.\",\n \"activities\": [\n {\n \"name\": \"Wait Until Pipeline Completes\",\n \"description\": \"Loops until the pipeline called completes.\\n\\nSimple status:\\n- Running = new iteration.\\n- Done = break.\",\n \"type\": \"Until\",\n \"dependsOn\": [\n {\n \"activity\": \"Get Wait Duration\",\n \"dependencyConditions\": [\n \"Succeeded\"\n ]\n }\n ],\n \"userProperties\": [],\n \"typeProperties\": {\n \"expression\": {\n \"value\": \"@equals('Done',activity('Get Pipeline Status').output.SimpleStatus)\",\n \"type\": \"Expression\"\n },\n \"activities\": [\n {\n \"name\": \"Get Pipeline Status\",\n \"description\": \"Checks the status of a given processing pipeline and provides the value for the downstream framework activities to act upon.\",\n \"type\": \"AzureFunctionActivity\",\n \"dependsOn\": [],\n \"policy\": {\n \"timeout\": \"7.00:00:00\",\n \"retry\": 0,\n \"retryIntervalInSeconds\": 30,\n \"secureOutput\": false,\n \"secureInput\": false\n },\n \"userProperties\": [],\n \"typeProperties\": {\n \"functionName\": \"CheckPipelineStatus\",\n \"method\": \"POST\",\n \"headers\": {},\n \"body\": {\n \"value\": \"@concat('\\n{\\n \\\"tenantId\\\": \\\"',pipeline().parameters.TenantId,'\\\",\\n \\\"applicationId\\\": \\\"',pipeline().parameters.applicationId,'\\\",\\n \\\"authenticationKey\\\": \\\"',pipeline().parameters.authenticationKey,'\\\",\\n \\\"subscriptionId\\\": \\\"',pipeline().parameters.subscriptionId,'\\\",\\n \\\"resourceGroup\\\": \\\"',pipeline().parameters.resourceGroup,'\\\",\\n \\\"factoryName\\\": \\\"',pipeline().parameters.factoryName,'\\\",\\n \\\"pipelineName\\\": \\\"',pipeline().parameters.pipelineName,'\\\",\\n \\\"runId\\\": \\\"',pipeline().parameters.runId,'\\\"\\n}')\",\n \"type\": \"Expression\"\n }\n },\n \"linkedServiceName\": {\n \"referenceName\": \"FrameworkFunctions\",\n \"type\": \"LinkedServiceReference\"\n }\n },\n {\n \"name\": \"Wait If Running\",\n \"description\": \"True = Do nothing.\\nFalse = Wait.\",\n \"type\": \"IfCondition\",\n \"dependsOn\": [\n {\n \"activity\": \"Get Pipeline Status\",\n \"dependencyConditions\": [\n \"Succeeded\"\n ]\n }\n ],\n \"userProperties\": [],\n \"typeProperties\": {\n \"expression\": {\n \"value\": \"@equals('Done',activity('Get Pipeline Status').output.SimpleStatus)\",\n \"type\": \"Expression\"\n },\n \"ifFalseActivities\": [\n {\n \"name\": \"Wait for Pipeline\",\n \"description\": \"The processing pipeline is still running so Wait before checking its status again.\",\n \"type\": \"Wait\",\n \"dependsOn\": [],\n \"userProperties\": [],\n \"typeProperties\": {\n \"waitTimeInSeconds\": {\n \"value\": \"@activity('Get Wait Duration').output.firstRow.PropertyValue\",\n \"type\": \"Expression\"\n }\n }\n }\n ]\n }\n },\n {\n \"name\": \"Wait to Retry Function\",\n \"type\": \"Wait\",\n \"dependsOn\": [\n {\n \"activity\": \"Get Pipeline Status\",\n \"dependencyConditions\": [\n \"Failed\"\n ]\n }\n ],\n \"userProperties\": [],\n \"typeProperties\": {\n \"waitTimeInSeconds\": {\n \"value\": \"@activity('Get Wait Duration').output.firstRow.PropertyValue\",\n \"type\": \"Expression\"\n }\n }\n },\n {\n \"name\": \"Set Last Update\",\n \"type\": \"SqlServerStoredProcedure\",\n \"dependsOn\": [\n {\n \"activity\": \"Get Pipeline Status\",\n \"dependencyConditions\": [\n \"Succeeded\"\n ]\n }\n ],\n \"policy\": {\n \"timeout\": \"7.00:00:00\",\n \"retry\": 0,\n \"retryIntervalInSeconds\": 30,\n \"secureOutput\": false,\n \"secureInput\": false\n },\n \"userProperties\": [],\n \"typeProperties\": {\n \"storedProcedureName\": \"[[procfwk].[SetLogPipelineLastStatusCheck]\",\n \"storedProcedureParameters\": {\n \"ExecutionId\": {\n \"value\": {\n \"value\": \"@pipeline().parameters.executionId\",\n \"type\": \"Expression\"\n },\n \"type\": \"Guid\"\n },\n \"PipelineId\": {\n \"value\": {\n \"value\": \"@pipeline().parameters.pipelineId\",\n \"type\": \"Expression\"\n },\n \"type\": \"Int32\"\n },\n \"StageId\": {\n \"value\": {\n \"value\": \"@pipeline().parameters.stageId\",\n \"type\": \"Expression\"\n },\n \"type\": \"Int32\"\n }\n }\n },\n \"linkedServiceName\": {\n \"referenceName\": \"SupportDatabase\",\n \"type\": \"LinkedServiceReference\"\n }\n }\n ],\n \"timeout\": \"7.00:00:00\"\n }\n },\n {\n \"name\": \"Pipeline Result\",\n \"description\": \"Receives the outcome from the function execution for a given processing pipeline and updates the current execution table with different pipelines status values depending on the result (case).\",\n \"type\": \"Switch\",\n \"dependsOn\": [\n {\n \"activity\": \"Wait Until Pipeline Completes\",\n \"dependencyConditions\": [\n \"Completed\"\n ]\n }\n ],\n \"userProperties\": [],\n \"typeProperties\": {\n \"on\": {\n \"value\": \"@activity('Get Pipeline Status').output.Status\",\n \"type\": \"Expression\"\n },\n \"cases\": [\n {\n \"value\": \"Succeeded\",\n \"activities\": [\n {\n \"name\": \"Pipeline Status Succeeded\",\n \"description\": \"Updates the current execution table with a pipeline status of success if the function outcome is succeeded.\",\n \"type\": \"SqlServerStoredProcedure\",\n \"dependsOn\": [],\n \"policy\": {\n \"timeout\": \"7.00:00:00\",\n \"retry\": 0,\n \"retryIntervalInSeconds\": 30,\n \"secureOutput\": false,\n \"secureInput\": false\n },\n \"userProperties\": [],\n \"typeProperties\": {\n \"storedProcedureName\": \"[[procfwk].[SetLogPipelineSuccess]\",\n \"storedProcedureParameters\": {\n \"ExecutionId\": {\n \"value\": {\n \"value\": \"@pipeline().parameters.executionId\",\n \"type\": \"Expression\"\n },\n \"type\": \"Guid\"\n },\n \"PipelineId\": {\n \"value\": {\n \"value\": \"@pipeline().parameters.pipelineId\",\n \"type\": \"Expression\"\n },\n \"type\": \"Int32\"\n },\n \"StageId\": {\n \"value\": {\n \"value\": \"@pipeline().parameters.stageId\",\n \"type\": \"Expression\"\n },\n \"type\": \"Int32\"\n }\n }\n },\n \"linkedServiceName\": {\n \"referenceName\": \"SupportDatabase\",\n \"type\": \"LinkedServiceReference\"\n }\n }\n ]\n },\n {\n \"value\": \"Failed\",\n \"activities\": [\n {\n \"name\": \"Pipeline Status Failed\",\n \"description\": \"Updates the current execution table with a pipeline status of failed if the function outcome is failed. Also blocks pipelines in the downstream execution stage.\",\n \"type\": \"SqlServerStoredProcedure\",\n \"dependsOn\": [],\n \"policy\": {\n \"timeout\": \"7.00:00:00\",\n \"retry\": 0,\n \"retryIntervalInSeconds\": 30,\n \"secureOutput\": false,\n \"secureInput\": false\n },\n \"userProperties\": [],\n \"typeProperties\": {\n \"storedProcedureName\": \"[[procfwk].[SetLogPipelineFailed]\",\n \"storedProcedureParameters\": {\n \"ExecutionId\": {\n \"value\": {\n \"value\": \"@pipeline().parameters.executionId\",\n \"type\": \"Expression\"\n },\n \"type\": \"Guid\"\n },\n \"PipelineId\": {\n \"value\": {\n \"value\": \"@pipeline().parameters.pipelineId\",\n \"type\": \"Expression\"\n },\n \"type\": \"Int32\"\n },\n \"RunId\": {\n \"value\": {\n \"value\": \"@activity('Get Pipeline Status').output.RunId\",\n \"type\": \"Expression\"\n },\n \"type\": \"Guid\"\n },\n \"StageId\": {\n \"value\": {\n \"value\": \"@pipeline().parameters.stageId\",\n \"type\": \"Expression\"\n },\n \"type\": \"Int32\"\n }\n }\n },\n \"linkedServiceName\": {\n \"referenceName\": \"SupportDatabase\",\n \"type\": \"LinkedServiceReference\"\n }\n },\n {\n \"name\": \"Get Error Details\",\n \"description\": \"Get the activity error details for the run ID of the worker pipeline called. Returns an array of all errors.\",\n \"type\": \"AzureFunctionActivity\",\n \"dependsOn\": [],\n \"policy\": {\n \"timeout\": \"7.00:00:00\",\n \"retry\": 0,\n \"retryIntervalInSeconds\": 30,\n \"secureOutput\": false,\n \"secureInput\": false\n },\n \"userProperties\": [],\n \"typeProperties\": {\n \"functionName\": \"GetActivityErrors\",\n \"method\": \"POST\",\n \"headers\": {},\n \"body\": {\n \"value\": \"@concat('\\n{\\n \\\"tenantId\\\": \\\"',pipeline().parameters.TenantId,'\\\",\\n \\\"applicationId\\\": \\\"',pipeline().parameters.applicationId,'\\\",\\n \\\"authenticationKey\\\": \\\"',pipeline().parameters.authenticationKey,'\\\",\\n \\\"subscriptionId\\\": \\\"',pipeline().parameters.subscriptionId,'\\\",\\n \\\"resourceGroup\\\": \\\"',pipeline().parameters.resourceGroup,'\\\",\\n \\\"factoryName\\\": \\\"',pipeline().parameters.factoryName,'\\\",\\n \\\"pipelineName\\\": \\\"',pipeline().parameters.pipelineName,'\\\",\\n \\\"runId\\\": \\\"',pipeline().parameters.runId,'\\\"\\n}')\",\n \"type\": \"Expression\"\n }\n },\n \"linkedServiceName\": {\n \"referenceName\": \"FrameworkFunctions\",\n \"type\": \"LinkedServiceReference\"\n }\n },\n {\n \"name\": \"Log Error Details\",\n \"description\": \"Parses pipeline error details and persists them to the metadata database error log table.\",\n \"type\": \"SqlServerStoredProcedure\",\n \"dependsOn\": [\n {\n \"activity\": \"Get Error Details\",\n \"dependencyConditions\": [\n \"Succeeded\"\n ]\n }\n ],\n \"policy\": {\n \"timeout\": \"7.00:00:00\",\n \"retry\": 0,\n \"retryIntervalInSeconds\": 30,\n \"secureOutput\": false,\n \"secureInput\": false\n },\n \"userProperties\": [],\n \"typeProperties\": {\n \"storedProcedureName\": \"[[procfwk].[SetErrorLogDetails]\",\n \"storedProcedureParameters\": {\n \"JsonErrorDetails\": {\n \"value\": {\n \"value\": \"@string(activity('Get Error Details').output)\",\n \"type\": \"Expression\"\n },\n \"type\": \"String\"\n },\n \"LocalExecutionId\": {\n \"value\": {\n \"value\": \"@pipeline().parameters.executionId\",\n \"type\": \"Expression\"\n },\n \"type\": \"Guid\"\n }\n }\n },\n \"linkedServiceName\": {\n \"referenceName\": \"SupportDatabase\",\n \"type\": \"LinkedServiceReference\"\n }\n }\n ]\n }\n ],\n \"defaultActivities\": [\n {\n \"name\": \"Pipeline Status Unknown\",\n \"description\": \"Updates the current execution table with a pipeline status of unknown if the function returns an unexpected outcome.\",\n \"type\": \"SqlServerStoredProcedure\",\n \"dependsOn\": [],\n \"policy\": {\n \"timeout\": \"7.00:00:00\",\n \"retry\": 0,\n \"retryIntervalInSeconds\": 30,\n \"secureOutput\": false,\n \"secureInput\": false\n },\n \"userProperties\": [],\n \"typeProperties\": {\n \"storedProcedureName\": \"[[procfwk].[SetLogPipelineUnknown]\",\n \"storedProcedureParameters\": {\n \"ExecutionId\": {\n \"value\": {\n \"value\": \"@pipeline().parameters.executionId\",\n \"type\": \"Expression\"\n },\n \"type\": \"Guid\"\n },\n \"PipelineId\": {\n \"value\": {\n \"value\": \"@pipeline().parameters.pipelineId\",\n \"type\": \"Expression\"\n },\n \"type\": \"Int32\"\n },\n \"StageId\": {\n \"value\": {\n \"value\": \"@pipeline().parameters.stageId\",\n \"type\": \"Expression\"\n },\n \"type\": \"Int32\"\n }\n }\n },\n \"linkedServiceName\": {\n \"referenceName\": \"SupportDatabase\",\n \"type\": \"LinkedServiceReference\"\n }\n }\n ]\n }\n },\n {\n \"name\": \"Get Wait Duration\",\n \"type\": \"Lookup\",\n \"dependsOn\": [],\n \"policy\": {\n \"timeout\": \"7.00:00:00\",\n \"retry\": 0,\n \"retryIntervalInSeconds\": 30,\n \"secureOutput\": false,\n \"secureInput\": false\n },\n \"userProperties\": [],\n \"typeProperties\": {\n \"source\": {\n \"type\": \"AzureSqlSource\",\n \"sqlReaderStoredProcedureName\": \"[[procfwk].[GetPropertyValue]\",\n \"storedProcedureParameters\": {\n \"PropertyName\": {\n \"type\": \"String\",\n \"value\": \"PipelineStatusCheckDuration\"\n }\n },\n \"queryTimeout\": \"02:00:00\"\n },\n \"dataset\": {\n \"referenceName\": \"GetSetMetadata\",\n \"type\": \"DatasetReference\",\n \"parameters\": {}\n }\n }\n }\n ],\n \"parameters\": {\n \"tenantId\": {\n \"type\": \"string\"\n },\n \"applicationId\": {\n \"type\": \"string\"\n },\n \"authenticationKey\": {\n \"type\": \"string\"\n },\n \"subscriptionId\": {\n \"type\": \"string\"\n },\n \"resourceGroup\": {\n \"type\": \"string\"\n },\n \"factoryName\": {\n \"type\": \"string\"\n },\n \"pipelineName\": {\n \"type\": \"string\"\n },\n \"runId\": {\n \"type\": \"string\"\n },\n \"executionId\": {\n \"type\": \"string\"\n },\n \"stageId\": {\n \"type\": \"int\"\n },\n \"pipelineId\": {\n \"type\": \"int\"\n }\n },\n \"folder\": {\n \"name\": \"_ProcFwk\"\n },\n \"annotations\": [\n \"ADF.procfwk\"\n ]\n },\n \"dependsOn\": [\n \"[concat(variables('factoryId'), '/datasets/GetSetMetadata')]\",\n \"[concat(variables('factoryId'), '/linkedServices/FrameworkFunctions')]\",\n \"[concat(variables('factoryId'), '/linkedServices/SupportDatabase')]\"\n ]\n },\n {\n \"name\": \"[concat(parameters('factoryName'), '/Intentional Error')]\",\n \"type\": \"Microsoft.DataFactory/factories/pipelines\",\n \"apiVersion\": \"2018-06-01\",\n \"properties\": {\n \"description\": \"Used just so the ADF.procfwk has something to call during development.\",\n \"activities\": [\n {\n \"name\": \"Call Fail Procedure\",\n \"type\": \"SqlServerStoredProcedure\",\n \"dependsOn\": [\n {\n \"activity\": \"Wait1\",\n \"dependencyConditions\": [\n \"Succeeded\"\n ]\n }\n ],\n \"policy\": {\n \"timeout\": \"7.00:00:00\",\n \"retry\": 0,\n \"retryIntervalInSeconds\": 30,\n \"secureOutput\": false,\n \"secureInput\": false\n },\n \"userProperties\": [],\n \"typeProperties\": {\n \"storedProcedureName\": \"[[dbo].[FailProcedure]\",\n \"storedProcedureParameters\": {\n \"RaiseError\": {\n \"value\": {\n \"value\": \"@pipeline().parameters.RaiseErrors\",\n \"type\": \"Expression\"\n },\n \"type\": \"String\"\n }\n }\n },\n \"linkedServiceName\": {\n \"referenceName\": \"SupportDatabase\",\n \"type\": \"LinkedServiceReference\"\n }\n },\n {\n \"name\": \"Wait1\",\n \"type\": \"Wait\",\n \"dependsOn\": [],\n \"userProperties\": [],\n \"typeProperties\": {\n \"waitTimeInSeconds\": {\n \"value\": \"@pipeline().parameters.WaitTime\",\n \"type\": \"Expression\"\n }\n }\n },\n {\n \"name\": \"Call Fail Notebook\",\n \"type\": \"DatabricksNotebook\",\n \"dependsOn\": [\n {\n \"activity\": \"Wait1\",\n \"dependencyConditions\": [\n \"Succeeded\"\n ]\n }\n ],\n \"policy\": {\n \"timeout\": \"7.00:00:00\",\n \"retry\": 0,\n \"retryIntervalInSeconds\": 30,\n \"secureOutput\": false,\n \"secureInput\": false\n },\n \"userProperties\": [],\n \"typeProperties\": {\n \"notebookPath\": \"/Playground/Throw Exception\",\n \"baseParameters\": {\n \"RaiseError\": {\n \"value\": \"@pipeline().parameters.RaiseErrors\",\n \"type\": \"Expression\"\n }\n }\n },\n \"linkedServiceName\": {\n \"referenceName\": \"BricksOfData\",\n \"type\": \"LinkedServiceReference\"\n }\n }\n ],\n \"parameters\": {\n \"RaiseErrors\": {\n \"type\": \"string\",\n \"defaultValue\": \"false\"\n },\n \"WaitTime\": {\n \"type\": \"int\",\n \"defaultValue\": 5\n }\n },\n \"folder\": {\n \"name\": \"_Workers\"\n },\n \"annotations\": [\n \"Worker\"\n ]\n },\n \"dependsOn\": [\n \"[concat(variables('factoryId'), '/linkedServices/SupportDatabase')]\",\n \"[concat(variables('factoryId'), '/linkedServices/BricksOfData')]\"\n ]\n },\n {\n \"name\": \"[concat(parameters('factoryName'), '/Wait 1')]\",\n \"type\": \"Microsoft.DataFactory/factories/pipelines\",\n \"apiVersion\": \"2018-06-01\",\n \"properties\": {\n \"description\": \"Used just so the ADF.procfwk has something to call during development.\",\n \"activities\": [\n {\n \"name\": \"Wait1\",\n \"type\": \"Wait\",\n \"dependsOn\": [],\n \"userProperties\": [],\n \"typeProperties\": {\n \"waitTimeInSeconds\": {\n \"value\": \"@pipeline().parameters.WaitTime\",\n \"type\": \"Expression\"\n }\n }\n }\n ],\n \"parameters\": {\n \"WaitTime\": {\n \"type\": \"int\",\n \"defaultValue\": 5\n }\n },\n \"folder\": {\n \"name\": \"_Workers\"\n },\n \"annotations\": [\n \"Worker\"\n ]\n },\n \"dependsOn\": []\n },\n {\n \"name\": \"[concat(parameters('factoryName'), '/Wait 10')]\",\n \"type\": \"Microsoft.DataFactory/factories/pipelines\",\n \"apiVersion\": \"2018-06-01\",\n \"properties\": {\n \"description\": \"Used just so the ADF.procfwk has something to call during development.\",\n \"activities\": [\n {\n \"name\": \"Wait10\",\n \"type\": \"Wait\",\n \"dependsOn\": [],\n \"userProperties\": [],\n \"typeProperties\": {\n \"waitTimeInSeconds\": {\n \"value\": \"@pipeline().parameters.WaitTime\",\n \"type\": \"Expression\"\n }\n }\n }\n ],\n \"parameters\": {\n \"WaitTime\": {\n \"type\": \"int\",\n \"defaultValue\": 5\n }\n },\n \"folder\": {\n \"name\": \"_Workers\"\n },\n \"annotations\": [\n \"Worker\"\n ]\n },\n \"dependsOn\": []\n },\n {\n \"name\": \"[concat(parameters('factoryName'), '/Wait 2')]\",\n \"type\": \"Microsoft.DataFactory/factories/pipelines\",\n \"apiVersion\": \"2018-06-01\",\n \"properties\": {\n \"description\": \"Used just so the ADF.procfwk has something to call during development.\",\n \"activities\": [\n {\n \"name\": \"Wait2\",\n \"type\": \"Wait\",\n \"dependsOn\": [],\n \"userProperties\": [],\n \"typeProperties\": {\n \"waitTimeInSeconds\": {\n \"value\": \"@pipeline().parameters.WaitTime\",\n \"type\": \"Expression\"\n }\n }\n }\n ],\n \"parameters\": {\n \"WaitTime\": {\n \"type\": \"int\",\n \"defaultValue\": 5\n }\n },\n \"folder\": {\n \"name\": \"_Workers\"\n },\n \"annotations\": [\n \"Worker\"\n ]\n },\n \"dependsOn\": []\n },\n {\n \"name\": \"[concat(parameters('factoryName'), '/Wait 3')]\",\n \"type\": \"Microsoft.DataFactory/factories/pipelines\",\n \"apiVersion\": \"2018-06-01\",\n \"properties\": {\n \"description\": \"Used just so the ADF.procfwk has something to call during development.\",\n \"activities\": [\n {\n \"name\": \"Wait3\",\n \"type\": \"Wait\",\n \"dependsOn\": [],\n \"userProperties\": [],\n \"typeProperties\": {\n \"waitTimeInSeconds\": {\n \"value\": \"@pipeline().parameters.WaitTime\",\n \"type\": \"Expression\"\n }\n }\n }\n ],\n \"parameters\": {\n \"WaitTime\": {\n \"type\": \"int\",\n \"defaultValue\": 5\n }\n },\n \"folder\": {\n \"name\": \"_Workers\"\n },\n \"annotations\": [\n \"Worker\"\n ]\n },\n \"dependsOn\": []\n },\n {\n \"name\": \"[concat(parameters('factoryName'), '/Wait 4')]\",\n \"type\": \"Microsoft.DataFactory/factories/pipelines\",\n \"apiVersion\": \"2018-06-01\",\n \"properties\": {\n \"description\": \"Used just so the ADF.procfwk has something to call during development.\",\n \"activities\": [\n {\n \"name\": \"Wait4\",\n \"type\": \"Wait\",\n \"dependsOn\": [],\n \"userProperties\": [],\n \"typeProperties\": {\n \"waitTimeInSeconds\": {\n \"value\": \"@pipeline().parameters.WaitTime\",\n \"type\": \"Expression\"\n }\n }\n }\n ],\n \"parameters\": {\n \"WaitTime\": {\n \"type\": \"int\",\n \"defaultValue\": 5\n }\n },\n \"folder\": {\n \"name\": \"_Workers\"\n },\n \"annotations\": [\n \"Worker\"\n ]\n },\n \"dependsOn\": []\n },\n {\n \"name\": \"[concat(parameters('factoryName'), '/Wait 5')]\",\n \"type\": \"Microsoft.DataFactory/factories/pipelines\",\n \"apiVersion\": \"2018-06-01\",\n \"properties\": {\n \"description\": \"Used just so the ADF.procfwk has something to call during development.\",\n \"activities\": [\n {\n \"name\": \"Wait5\",\n \"type\": \"Wait\",\n \"dependsOn\": [],\n \"userProperties\": [],\n \"typeProperties\": {\n \"waitTimeInSeconds\": {\n \"value\": \"@pipeline().parameters.WaitTime\",\n \"type\": \"Expression\"\n }\n }\n }\n ],\n \"parameters\": {\n \"WaitTime\": {\n \"type\": \"int\",\n \"defaultValue\": 5\n }\n },\n \"folder\": {\n \"name\": \"_Workers\"\n },\n \"annotations\": [\n \"Worker\"\n ]\n },\n \"dependsOn\": []\n },\n {\n \"name\": \"[concat(parameters('factoryName'), '/Wait 6')]\",\n \"type\": \"Microsoft.DataFactory/factories/pipelines\",\n \"apiVersion\": \"2018-06-01\",\n \"properties\": {\n \"description\": \"Used just so the ADF.procfwk has something to call during development.\",\n \"activities\": [\n {\n \"name\": \"Wait6\",\n \"type\": \"Wait\",\n \"dependsOn\": [],\n \"userProperties\": [],\n \"typeProperties\": {\n \"waitTimeInSeconds\": {\n \"value\": \"@pipeline().parameters.WaitTime\",\n \"type\": \"Expression\"\n }\n }\n }\n ],\n \"parameters\": {\n \"WaitTime\": {\n \"type\": \"int\",\n \"defaultValue\": 5\n }\n },\n \"folder\": {\n \"name\": \"_Workers\"\n },\n \"annotations\": [\n \"Worker\"\n ]\n },\n \"dependsOn\": []\n },\n {\n \"name\": \"[concat(parameters('factoryName'), '/Wait 7')]\",\n \"type\": \"Microsoft.DataFactory/factories/pipelines\",\n \"apiVersion\": \"2018-06-01\",\n \"properties\": {\n \"description\": \"Used just so the ADF.procfwk has something to call during development.\",\n \"activities\": [\n {\n \"name\": \"Wait7\",\n \"type\": \"Wait\",\n \"dependsOn\": [],\n \"userProperties\": [],\n \"typeProperties\": {\n \"waitTimeInSeconds\": {\n \"value\": \"@pipeline().parameters.WaitTime\",\n \"type\": \"Expression\"\n }\n }\n }\n ],\n \"parameters\": {\n \"WaitTime\": {\n \"type\": \"int\",\n \"defaultValue\": 5\n }\n },\n \"folder\": {\n \"name\": \"_Workers\"\n },\n \"annotations\": [\n \"Worker\"\n ]\n },\n \"dependsOn\": []\n },\n {\n \"name\": \"[concat(parameters('factoryName'), '/Wait 8')]\",\n \"type\": \"Microsoft.DataFactory/factories/pipelines\",\n \"apiVersion\": \"2018-06-01\",\n \"properties\": {\n \"description\": \"Used just so the ADF.procfwk has something to call during development.\",\n \"activities\": [\n {\n \"name\": \"Wait8\",\n \"type\": \"Wait\",\n \"dependsOn\": [],\n \"userProperties\": [],\n \"typeProperties\": {\n \"waitTimeInSeconds\": {\n \"value\": \"@pipeline().parameters.WaitTime\",\n \"type\": \"Expression\"\n }\n }\n }\n ],\n \"parameters\": {\n \"WaitTime\": {\n \"type\": \"int\",\n \"defaultValue\": 5\n }\n },\n \"folder\": {\n \"name\": \"_Workers\"\n },\n \"annotations\": [\n \"Worker\"\n ]\n },\n \"dependsOn\": []\n },\n {\n \"name\": \"[concat(parameters('factoryName'), '/Wait 9')]\",\n \"type\": \"Microsoft.DataFactory/factories/pipelines\",\n \"apiVersion\": \"2018-06-01\",\n \"properties\": {\n \"description\": \"Used just so the ADF.procfwk has something to call during development.\",\n \"activities\": [\n {\n \"name\": \"Wait9\",\n \"type\": \"Wait\",\n \"dependsOn\": [],\n \"userProperties\": [],\n \"typeProperties\": {\n \"waitTimeInSeconds\": {\n \"value\": \"@pipeline().parameters.WaitTime\",\n \"type\": \"Expression\"\n }\n }\n }\n ],\n \"parameters\": {\n \"WaitTime\": {\n \"type\": \"int\",\n \"defaultValue\": 90\n }\n },\n \"folder\": {\n \"name\": \"_Workers\"\n },\n \"annotations\": [\n \"Worker\"\n ]\n },\n \"dependsOn\": []\n },\n {\n \"name\": \"[concat(parameters('factoryName'), '/GetSetMetadata')]\",\n \"type\": \"Microsoft.DataFactory/factories/datasets\",\n \"apiVersion\": \"2018-06-01\",\n \"properties\": {\n \"linkedServiceName\": {\n \"referenceName\": \"SupportDatabase\",\n \"type\": \"LinkedServiceReference\"\n },\n \"annotations\": [\n \"ADF.procfwk\"\n ],\n \"type\": \"AzureSqlTable\",\n \"schema\": [],\n \"typeProperties\": {}\n },\n \"dependsOn\": [\n \"[concat(variables('factoryId'), '/linkedServices/SupportDatabase')]\"\n ]\n },\n {\n \"name\": \"[concat(parameters('factoryName'), '/BricksOfData')]\",\n \"type\": \"Microsoft.DataFactory/factories/linkedServices\",\n \"apiVersion\": \"2018-06-01\",\n \"properties\": {\n \"annotations\": [],\n \"type\": \"AzureDatabricks\",\n \"typeProperties\": {\n \"domain\": \"https://northeurope.azuredatabricks.net\",\n \"accessToken\": {\n \"type\": \"AzureKeyVaultSecret\",\n \"store\": {\n \"referenceName\": \"Keys\",\n \"type\": \"LinkedServiceReference\"\n },\n \"secretName\": \"BricksOfDataToken\",\n \"secretVersion\": \"\"\n },\n \"existingClusterId\": \"0422-090117-slots899\"\n }\n },\n \"dependsOn\": [\n \"[concat(variables('factoryId'), '/linkedServices/Keys')]\"\n ]\n },\n {\n \"name\": \"[concat(parameters('factoryName'), '/FrameworkFunctions')]\",\n \"type\": \"Microsoft.DataFactory/factories/linkedServices\",\n \"apiVersion\": \"2018-06-01\",\n \"properties\": {\n \"annotations\": [\n \"ADF.procfwk\"\n ],\n \"type\": \"AzureFunction\",\n \"typeProperties\": {\n \"functionAppUrl\": \"https://frameworksupportfunctions.azurewebsites.net\",\n \"functionKey\": {\n \"type\": \"AzureKeyVaultSecret\",\n \"store\": {\n \"referenceName\": \"Keys\",\n \"type\": \"LinkedServiceReference\"\n },\n \"secretName\": \"FrameworkFunctionsKey\"\n }\n }\n },\n \"dependsOn\": [\n \"[concat(variables('factoryId'), '/linkedServices/Keys')]\"\n ]\n },\n {\n \"name\": \"[concat(parameters('factoryName'), '/Keys')]\",\n \"type\": \"Microsoft.DataFactory/factories/linkedServices\",\n \"apiVersion\": \"2018-06-01\",\n \"properties\": {\n \"description\": \"Connection to Key Vault for all other ADF linked service credentials required to run the processing framework.\",\n \"annotations\": [\n \"ADF.procfwk\"\n ],\n \"type\": \"AzureKeyVault\",\n \"typeProperties\": {\n \"baseUrl\": \"[parameters('Keys_properties_typeProperties_baseUrl')]\"\n }\n },\n \"dependsOn\": []\n },\n {\n \"name\": \"[concat(parameters('factoryName'), '/SupportDatabase')]\",\n \"type\": \"Microsoft.DataFactory/factories/linkedServices\",\n \"apiVersion\": \"2018-06-01\",\n \"properties\": {\n \"description\": \"Connection between ADF and processing framework metadata SQLDB.\",\n \"annotations\": [\n \"ADF.procfwk\"\n ],\n \"type\": \"AzureSqlDatabase\",\n \"typeProperties\": {\n \"connectionString\": {\n \"type\": \"AzureKeyVaultSecret\",\n \"store\": {\n \"referenceName\": \"Keys\",\n \"type\": \"LinkedServiceReference\"\n },\n \"secretName\": \"[parameters('SupportDatabase_properties_typeProperties_connectionString_secretName')]\"\n }\n }\n },\n \"dependsOn\": [\n \"[concat(variables('factoryId'), '/linkedServices/Keys')]\"\n ]\n },\n {\n \"name\": \"[concat(parameters('factoryName'), '/FunctionalTestingTrigger')]\",\n \"type\": \"Microsoft.DataFactory/factories/triggers\",\n \"apiVersion\": \"2018-06-01\",\n \"properties\": {\n \"description\": \"Used for functional testing of the framework in a dedicated environment.\",\n \"annotations\": [\n \"ADF.procfwk\"\n ],\n \"runtimeState\": \"Stopped\",\n \"pipelines\": [\n {\n \"pipelineReference\": {\n \"referenceName\": \"01-Grandparent\",\n \"type\": \"PipelineReference\"\n },\n \"parameters\": {}\n }\n ],\n \"type\": \"ScheduleTrigger\",\n \"typeProperties\": {\n \"recurrence\": {\n \"frequency\": \"Hour\",\n \"interval\": 2,\n \"startTime\": \"2020-04-06T15:00:00.000Z\",\n \"timeZone\": \"UTC\"\n }\n }\n },\n \"dependsOn\": [\n \"[concat(variables('factoryId'), '/pipelines/01-Grandparent')]\"\n ]\n }\n ]\n}"
},
{
"path": "ARM Templates/Data Factory/v1.7 Export.json",
"content": "{\n \"$schema\": \"http://schema.management.azure.com/schemas/2015-01-01/deploymentTemplate.json#\",\n \"contentVersion\": \"1.0.0.0\",\n \"parameters\": {\n \"factoryName\": {\n \"type\": \"string\",\n \"metadata\": \"Data Factory name\",\n \"defaultValue\": \"\"\n },\n \"Keys_properties_typeProperties_baseUrl\": {\n \"type\": \"string\",\n \"defaultValue\": \"\"\n },\n \"SupportDatabase_properties_typeProperties_connectionString_secretName\": {\n \"type\": \"string\",\n \"defaultValue\": \"\"\n }\n },\n \"variables\": {\n \"factoryId\": \"[concat('Microsoft.DataFactory/factories/', parameters('factoryName'))]\"\n },\n \"resources\": [\n {\n \"name\": \"[concat(parameters('factoryName'), '/01-Grandparent')]\",\n \"type\": \"Microsoft.DataFactory/factories/pipelines\",\n \"apiVersion\": \"2018-06-01\",\n \"properties\": {\n \"description\": \"ADF.procfwk grandparent pipeline used optionally to bootstrap any wider processes in your Data Factory that then calls the processing framework.\",\n \"activities\": [\n {\n \"name\": \"Framework Processing\",\n \"type\": \"ExecutePipeline\",\n \"dependsOn\": [\n {\n \"activity\": \"Set Random Waits\",\n \"dependencyConditions\": [\n \"Succeeded\"\n ]\n }\n ],\n \"userProperties\": [],\n \"typeProperties\": {\n \"pipeline\": {\n \"referenceName\": \"02-Parent\",\n \"type\": \"PipelineReference\"\n },\n \"waitOnCompletion\": true,\n \"parameters\": {}\n }\n },\n {\n \"name\": \"Set Random Waits\",\n \"description\": \"For functional testing only.\",\n \"type\": \"SqlServerStoredProcedure\",\n \"dependsOn\": [],\n \"policy\": {\n \"timeout\": \"7.00:00:00\",\n \"retry\": 0,\n \"retryIntervalInSeconds\": 30,\n \"secureOutput\": false,\n \"secureInput\": false\n },\n \"userProperties\": [],\n \"typeProperties\": {\n \"storedProcedureName\": \"[[dbo].[SetRandomWaitValues]\"\n },\n \"linkedServiceName\": {\n \"referenceName\": \"SupportDatabase\",\n \"type\": \"LinkedServiceReference\"\n }\n }\n ],\n \"folder\": {\n \"name\": \"_ProcFwk\"\n },\n \"annotations\": [\n \"ADF.procfwk\"\n ]\n },\n \"dependsOn\": [\n \"[concat(variables('factoryId'), '/pipelines/02-Parent')]\",\n \"[concat(variables('factoryId'), '/linkedServices/SupportDatabase')]\"\n ]\n },\n {\n \"name\": \"[concat(parameters('factoryName'), '/02-Parent')]\",\n \"type\": \"Microsoft.DataFactory/factories/pipelines\",\n \"apiVersion\": \"2018-06-01\",\n \"properties\": {\n \"description\": \"ADF.procfwk parent pipeline used to bootstrap the orchestration framework in perform the first level ForEach calls in sequence for the metadata stages.\",\n \"activities\": [\n {\n \"name\": \"Get Stages\",\n \"description\": \"Returns a distinct list of execution stages within the framework metadata.\",\n \"type\": \"Lookup\",\n \"dependsOn\": [\n {\n \"activity\": \"Execution Wrapper\",\n \"dependencyConditions\": [\n \"Succeeded\"\n ]\n }\n ],\n \"policy\": {\n \"timeout\": \"7.00:00:00\",\n \"retry\": 0,\n \"retryIntervalInSeconds\": 30,\n \"secureOutput\": false,\n \"secureInput\": false\n },\n \"userProperties\": [],\n \"typeProperties\": {\n \"source\": {\n \"type\": \"AzureSqlSource\",\n \"sqlReaderStoredProcedureName\": \"[[procfwk].[GetStages]\",\n \"storedProcedureParameters\": {\n \"ExecutionId\": {\n \"type\": \"Guid\",\n \"value\": {\n \"value\": \"@activity('Execution Wrapper').output.firstRow.ExecutionId\",\n \"type\": \"Expression\"\n }\n }\n },\n \"queryTimeout\": \"02:00:00\"\n },\n \"dataset\": {\n \"referenceName\": \"GetSetMetadata\",\n \"type\": \"DatasetReference\",\n \"parameters\": {}\n },\n \"firstRowOnly\": false\n }\n },\n {\n \"name\": \"Execute Stages\",\n \"description\": \"Top level ForEach to sequentially call all processing stages within the framework metadata. Items for iteration passed from the Get Stages lookup activity.\",\n \"type\": \"ForEach\",\n \"dependsOn\": [\n {\n \"activity\": \"Get Stages\",\n \"dependencyConditions\": [\n \"Succeeded\"\n ]\n },\n {\n \"activity\": \"Get Tenant Id\",\n \"dependencyConditions\": [\n \"Succeeded\"\n ]\n },\n {\n \"activity\": \"Get Subscription Id\",\n \"dependencyConditions\": [\n \"Succeeded\"\n ]\n }\n ],\n \"userProperties\": [],\n \"typeProperties\": {\n \"items\": {\n \"value\": \"@activity('Get Stages').output.value\",\n \"type\": \"Expression\"\n },\n \"isSequential\": true,\n \"activities\": [\n {\n \"name\": \"Stage Executor\",\n \"description\": \"Call to the framework generic child pipeline for a given execution stage.\",\n \"type\": \"ExecutePipeline\",\n \"dependsOn\": [\n {\n \"activity\": \"Log Stage Preparing\",\n \"dependencyConditions\": [\n \"Succeeded\"\n ]\n }\n ],\n \"userProperties\": [],\n \"typeProperties\": {\n \"pipeline\": {\n \"referenceName\": \"03-Child\",\n \"type\": \"PipelineReference\"\n },\n \"waitOnCompletion\": true,\n \"parameters\": {\n \"StageId\": {\n \"value\": \"@item().StageId\",\n \"type\": \"Expression\"\n },\n \"ExecutionId\": {\n \"value\": \"@activity('Execution Wrapper').output.firstRow.ExecutionId\",\n \"type\": \"Expression\"\n },\n \"TenantId\": {\n \"value\": \"@activity('Get Tenant Id').output.firstRow.PropertyValue\",\n \"type\": \"Expression\"\n },\n \"SubscriptionId\": {\n \"value\": \"@activity('Get Subscription Id').output.firstRow.PropertyValue\",\n \"type\": \"Expression\"\n }\n }\n }\n },\n {\n \"name\": \"Log Stage Preparing\",\n \"description\": \"Update the current execution table flagging all pipelines within the stage as preparing.\",\n \"type\": \"SqlServerStoredProcedure\",\n \"dependsOn\": [\n {\n \"activity\": \"Check for Blockers\",\n \"dependencyConditions\": [\n \"Succeeded\"\n ]\n }\n ],\n \"policy\": {\n \"timeout\": \"7.00:00:00\",\n \"retry\": 0,\n \"retryIntervalInSeconds\": 30,\n \"secureOutput\": false,\n \"secureInput\": false\n },\n \"userProperties\": [],\n \"typeProperties\": {\n \"storedProcedureName\": \"[[procfwk].[SetLogStagePreparing]\",\n \"storedProcedureParameters\": {\n \"ExecutionId\": {\n \"value\": {\n \"value\": \"@activity('Execution Wrapper').output.firstRow.ExecutionId\",\n \"type\": \"Expression\"\n },\n \"type\": \"Guid\"\n },\n \"StageId\": {\n \"value\": {\n \"value\": \"@item().StageId\",\n \"type\": \"Expression\"\n },\n \"type\": \"Int32\"\n }\n }\n },\n \"linkedServiceName\": {\n \"referenceName\": \"SupportDatabase\",\n \"type\": \"LinkedServiceReference\"\n }\n },\n {\n \"name\": \"Check for Blockers\",\n \"description\": \"Used to double check and stop the next execution stage if failures and blockers have be incurred. Without this step processing would continue regardless of upstream failures.\",\n \"type\": \"SqlServerStoredProcedure\",\n \"dependsOn\": [],\n \"policy\": {\n \"timeout\": \"7.00:00:00\",\n \"retry\": 0,\n \"retryIntervalInSeconds\": 30,\n \"secureOutput\": false,\n \"secureInput\": false\n },\n \"userProperties\": [],\n \"typeProperties\": {\n \"storedProcedureName\": \"[[procfwk].[CheckForBlockedPipelines]\",\n \"storedProcedureParameters\": {\n \"StageId\": {\n \"value\": {\n \"value\": \"@item().StageId\",\n \"type\": \"Expression\"\n },\n \"type\": \"Int32\"\n }\n }\n },\n \"linkedServiceName\": {\n \"referenceName\": \"SupportDatabase\",\n \"type\": \"LinkedServiceReference\"\n }\n }\n ]\n }\n },\n {\n \"name\": \"Execution Wrapper\",\n \"description\": \"Wrapper to reset and restart processing or create a completely new execution instance of the framework metadata.\",\n \"type\": \"Lookup\",\n \"dependsOn\": [\n {\n \"activity\": \"Metadata Integrity Checks\",\n \"dependencyConditions\": [\n \"Succeeded\"\n ]\n }\n ],\n \"policy\": {\n \"timeout\": \"7.00:00:00\",\n \"retry\": 0,\n \"retryIntervalInSeconds\": 30,\n \"secureOutput\": false,\n \"secureInput\": false\n },\n \"userProperties\": [],\n \"typeProperties\": {\n \"source\": {\n \"type\": \"AzureSqlSource\",\n \"sqlReaderStoredProcedureName\": \"[[procfwk].[ExecutionWrapper]\",\n \"storedProcedureParameters\": {\n \"CallingDataFactory\": {\n \"type\": \"String\",\n \"value\": {\n \"value\": \"@pipeline().DataFactory\",\n \"type\": \"Expression\"\n }\n }\n },\n \"queryTimeout\": \"02:00:00\"\n },\n \"dataset\": {\n \"referenceName\": \"GetSetMetadata\",\n \"type\": \"DatasetReference\",\n \"parameters\": {}\n }\n }\n },\n {\n \"name\": \"Archive Execution Log\",\n \"description\": \"After a successful execution run the current execution metadata is moved to the long term logging table by this stored procedure call.\",\n \"type\": \"SqlServerStoredProcedure\",\n \"dependsOn\": [\n {\n \"activity\": \"Execute Stages\",\n \"dependencyConditions\": [\n \"Succeeded\"\n ]\n }\n ],\n \"policy\": {\n \"timeout\": \"7.00:00:00\",\n \"retry\": 0,\n \"retryIntervalInSeconds\": 30,\n \"secureOutput\": false,\n \"secureInput\": false\n },\n \"userProperties\": [],\n \"typeProperties\": {\n \"storedProcedureName\": \"[[procfwk].[UpdateExecutionLog]\"\n },\n \"linkedServiceName\": {\n \"referenceName\": \"SupportDatabase\",\n \"type\": \"LinkedServiceReference\"\n }\n },\n {\n \"name\": \"Get Tenant Id\",\n \"description\": \"Returning the Azure Tenant Id from the metadata properties table.\",\n \"type\": \"Lookup\",\n \"dependsOn\": [\n {\n \"activity\": \"Metadata Integrity Checks\",\n \"dependencyConditions\": [\n \"Succeeded\"\n ]\n }\n ],\n \"policy\": {\n \"timeout\": \"7.00:00:00\",\n \"retry\": 0,\n \"retryIntervalInSeconds\": 30,\n \"secureOutput\": false,\n \"secureInput\": false\n },\n \"userProperties\": [],\n \"typeProperties\": {\n \"source\": {\n \"type\": \"AzureSqlSource\",\n \"sqlReaderStoredProcedureName\": \"[[procfwk].[GetPropertyValue]\",\n \"storedProcedureParameters\": {\n \"PropertyName\": {\n \"type\": \"String\",\n \"value\": \"TenantId\"\n }\n },\n \"queryTimeout\": \"02:00:00\"\n },\n \"dataset\": {\n \"referenceName\": \"GetSetMetadata\",\n \"type\": \"DatasetReference\",\n \"parameters\": {}\n },\n \"firstRowOnly\": true\n }\n },\n {\n \"name\": \"Get Subscription Id\",\n \"description\": \"Returning the Azure Subscription Id from the metadata properties table.\",\n \"type\": \"Lookup\",\n \"dependsOn\": [\n {\n \"activity\": \"Metadata Integrity Checks\",\n \"dependencyConditions\": [\n \"Succeeded\"\n ]\n }\n ],\n \"policy\": {\n \"timeout\": \"7.00:00:00\",\n \"retry\": 0,\n \"retryIntervalInSeconds\": 30,\n \"secureOutput\": false,\n \"secureInput\": true\n },\n \"userProperties\": [],\n \"typeProperties\": {\n \"source\": {\n \"type\": \"AzureSqlSource\",\n \"sqlReaderStoredProcedureName\": \"[[procfwk].[GetPropertyValue]\",\n \"storedProcedureParameters\": {\n \"PropertyName\": {\n \"type\": \"String\",\n \"value\": \"SubscriptionId\"\n }\n },\n \"queryTimeout\": \"02:00:00\"\n },\n \"dataset\": {\n \"referenceName\": \"GetSetMetadata\",\n \"type\": \"DatasetReference\",\n \"parameters\": {}\n }\n }\n },\n {\n \"name\": \"Metadata Integrity Checks\",\n \"description\": \"Performs a series of checks on all metadata held in the framework SQLDB. This is intended to raise errors before an execution run even starts.\",\n \"type\": \"SqlServerStoredProcedure\",\n \"dependsOn\": [],\n \"policy\": {\n \"timeout\": \"7.00:00:00\",\n \"retry\": 0,\n \"retryIntervalInSeconds\": 30,\n \"secureOutput\": false,\n \"secureInput\": false\n },\n \"userProperties\": [],\n \"typeProperties\": {\n \"storedProcedureName\": \"[[procfwk].[CheckMetadataIntegrity]\",\n \"storedProcedureParameters\": {\n \"DebugMode\": {\n \"value\": \"false\",\n \"type\": \"Boolean\"\n }\n }\n },\n \"linkedServiceName\": {\n \"referenceName\": \"SupportDatabase\",\n \"type\": \"LinkedServiceReference\"\n }\n }\n ],\n \"folder\": {\n \"name\": \"_ProcFwk\"\n },\n \"annotations\": [\n \"ADF.procfwk\"\n ]\n },\n \"dependsOn\": [\n \"[concat(variables('factoryId'), '/datasets/GetSetMetadata')]\",\n \"[concat(variables('factoryId'), '/linkedServices/SupportDatabase')]\",\n \"[concat(variables('factoryId'), '/pipelines/03-Child')]\"\n ]\n },\n {\n \"name\": \"[concat(parameters('factoryName'), '/03-Child')]\",\n \"type\": \"Microsoft.DataFactory/factories/pipelines\",\n \"apiVersion\": \"2018-06-01\",\n \"properties\": {\n \"description\": \"ADF.procfwk child pipeline used to execute Worker pipelines within a given execution stage. This pipeline will be called once for each stage, then execute all Workers in parallel.\",\n \"activities\": [\n {\n \"name\": \"Get Pipelines\",\n \"description\": \"Returns all pipelines from the metadata to be executed within a given processing stage.\",\n \"type\": \"Lookup\",\n \"dependsOn\": [],\n \"policy\": {\n \"timeout\": \"7.00:00:00\",\n \"retry\": 0,\n \"retryIntervalInSeconds\": 30,\n \"secureOutput\": false,\n \"secureInput\": false\n },\n \"userProperties\": [],\n \"typeProperties\": {\n \"source\": {\n \"type\": \"AzureSqlSource\",\n \"sqlReaderStoredProcedureName\": \"[[procfwk].[GetPipelinesInStage]\",\n \"storedProcedureParameters\": {\n \"StageId\": {\n \"type\": \"Int32\",\n \"value\": {\n \"value\": \"@pipeline().parameters.StageId\",\n \"type\": \"Expression\"\n }\n }\n },\n \"queryTimeout\": \"02:00:00\"\n },\n \"dataset\": {\n \"referenceName\": \"GetSetMetadata\",\n \"type\": \"DatasetReference\",\n \"parameters\": {}\n },\n \"firstRowOnly\": false\n }\n },\n {\n \"name\": \"Execute Pipelines\",\n \"description\": \"Second level ForEach to run in parallel all pipelines within the stage. Items for iteration passed from the Get Pipelines lookup activity.\",\n \"type\": \"ForEach\",\n \"dependsOn\": [\n {\n \"activity\": \"Get Pipelines\",\n \"dependencyConditions\": [\n \"Succeeded\"\n ]\n }\n ],\n \"userProperties\": [],\n \"typeProperties\": {\n \"items\": {\n \"value\": \"@activity('Get Pipelines').output.value\",\n \"type\": \"Expression\"\n },\n \"isSequential\": false,\n \"batchCount\": 40,\n \"activities\": [\n {\n \"name\": \"Execute Pipeline\",\n \"description\": \"The lowest level executor with the metadata framework to call existing processing pipelines within Data Factory. The function called will block processing and wait for an outcome.\",\n \"type\": \"AzureFunctionActivity\",\n \"dependsOn\": [\n {\n \"activity\": \"Log Pipeline Running\",\n \"dependencyConditions\": [\n \"Succeeded\"\n ]\n },\n {\n \"activity\": \"Get Pipeline Params\",\n \"dependencyConditions\": [\n \"Succeeded\"\n ]\n },\n {\n \"activity\": \"Get SPN Details\",\n \"dependencyConditions\": [\n \"Succeeded\"\n ]\n }\n ],\n \"policy\": {\n \"timeout\": \"7.00:00:00\",\n \"retry\": 0,\n \"retryIntervalInSeconds\": 30,\n \"secureOutput\": false,\n \"secureInput\": false\n },\n \"userProperties\": [],\n \"typeProperties\": {\n \"functionName\": \"ExecutePipeline\",\n \"method\": \"POST\",\n \"headers\": {},\n \"body\": {\n \"value\": \"@concat('\\n{\\n\\t\\\"tenantId\\\": \\\"',pipeline().parameters.TenantId,'\\\",\\n\\t\\\"applicationId\\\": \\\"',activity('Get SPN Details').output.firstRow.Id,'\\\",\\n\\t\\\"authenticationKey\\\": \\\"',activity('Get SPN Details').output.firstRow.Secret,'\\\",\\n\\t\\\"subscriptionId\\\": \\\"',pipeline().parameters.SubscriptionId,'\\\",\\n\\t\\\"resourceGroup\\\": \\\"',item().ResourceGroupName,'\\\",\\n\\t\\\"factoryName\\\": \\\"',item().DataFactoryName,'\\\",\\n\\t\\\"pipelineName\\\": \\\"',item().PipelineName,'\\\"',activity('Get Pipeline Params').output.firstRow.Params,'\\n}')\",\n \"type\": \"Expression\"\n }\n },\n \"linkedServiceName\": {\n \"referenceName\": \"FrameworkFunctions\",\n \"type\": \"LinkedServiceReference\"\n }\n },\n {\n \"name\": \"Get Pipeline Params\",\n \"description\": \"Returns any parameters from metadata required for the processing pipeline being called. The output can be an empty string if no parameters are required.\",\n \"type\": \"Lookup\",\n \"dependsOn\": [],\n \"policy\": {\n \"timeout\": \"7.00:00:00\",\n \"retry\": 0,\n \"retryIntervalInSeconds\": 30,\n \"secureOutput\": false,\n \"secureInput\": false\n },\n \"userProperties\": [],\n \"typeProperties\": {\n \"source\": {\n \"type\": \"AzureSqlSource\",\n \"sqlReaderStoredProcedureName\": \"[[procfwk].[GetPipelineParameters]\",\n \"storedProcedureParameters\": {\n \"PipelineId\": {\n \"type\": \"Int32\",\n \"value\": {\n \"value\": \"@item().PipelineId\",\n \"type\": \"Expression\"\n }\n }\n },\n \"queryTimeout\": \"02:00:00\"\n },\n \"dataset\": {\n \"referenceName\": \"GetSetMetadata\",\n \"type\": \"DatasetReference\",\n \"parameters\": {}\n }\n }\n },\n {\n \"name\": \"Log Pipeline Running\",\n \"description\": \"Sets the current pipeline with a status of running within the current execution database table.\",\n \"type\": \"SqlServerStoredProcedure\",\n \"dependsOn\": [],\n \"policy\": {\n \"timeout\": \"7.00:00:00\",\n \"retry\": 0,\n \"retryIntervalInSeconds\": 30,\n \"secureOutput\": false,\n \"secureInput\": false\n },\n \"userProperties\": [],\n \"typeProperties\": {\n \"storedProcedureName\": \"[[procfwk].[SetLogPipelineRunning]\",\n \"storedProcedureParameters\": {\n \"ExecutionId\": {\n \"value\": {\n \"value\": \"@pipeline().parameters.ExecutionId\",\n \"type\": \"Expression\"\n },\n \"type\": \"Guid\"\n },\n \"PipelineId\": {\n \"value\": {\n \"value\": \"@item().PipelineId\",\n \"type\": \"Expression\"\n },\n \"type\": \"Int32\"\n },\n \"StageId\": {\n \"value\": {\n \"value\": \"@pipeline().parameters.StageId\",\n \"type\": \"Expression\"\n },\n \"type\": \"Int32\"\n }\n }\n },\n \"linkedServiceName\": {\n \"referenceName\": \"SupportDatabase\",\n \"type\": \"LinkedServiceReference\"\n }\n },\n {\n \"name\": \"Get SPN Details\",\n \"description\": \"Return the SPN ID and Secret for the processing pipeline being executed. Called at this level as each pipeline can have a different SPN.\",\n \"type\": \"Lookup\",\n \"dependsOn\": [],\n \"policy\": {\n \"timeout\": \"7.00:00:00\",\n \"retry\": 0,\n \"retryIntervalInSeconds\": 30,\n \"secureOutput\": false,\n \"secureInput\": false\n },\n \"userProperties\": [],\n \"typeProperties\": {\n \"source\": {\n \"type\": \"AzureSqlSource\",\n \"sqlReaderStoredProcedureName\": \"[[procfwk].[GetServicePrincipal]\",\n \"storedProcedureParameters\": {\n \"DataFactory\": {\n \"type\": \"String\",\n \"value\": {\n \"value\": \"@item().DataFactoryName\",\n \"type\": \"Expression\"\n }\n },\n \"PipelineName\": {\n \"type\": \"String\",\n \"value\": {\n \"value\": \"@item().PipelineName\",\n \"type\": \"Expression\"\n }\n }\n },\n \"queryTimeout\": \"02:00:00\"\n },\n \"dataset\": {\n \"referenceName\": \"GetSetMetadata\",\n \"type\": \"DatasetReference\",\n \"parameters\": {}\n }\n }\n },\n {\n \"name\": \"Log Activity Failure\",\n \"description\": \"Handle true failures from calling out to the Azure Function and update the current execution table accordingly so a restart can occur.\",\n \"type\": \"SqlServerStoredProcedure\",\n \"dependsOn\": [\n {\n \"activity\": \"Execute Pipeline\",\n \"dependencyConditions\": [\n \"Failed\"\n ]\n }\n ],\n \"policy\": {\n \"timeout\": \"7.00:00:00\",\n \"retry\": 0,\n \"retryIntervalInSeconds\": 30,\n \"secureOutput\": false,\n \"secureInput\": false\n },\n \"userProperties\": [],\n \"typeProperties\": {\n \"storedProcedureName\": \"[[procfwk].[SetLogActivityFailed]\",\n \"storedProcedureParameters\": {\n \"ExecutionId\": {\n \"value\": {\n \"value\": \"@pipeline().parameters.ExecutionId\",\n \"type\": \"Expression\"\n },\n \"type\": \"Guid\"\n },\n \"PipelineId\": {\n \"value\": {\n \"value\": \"@item().PipelineId\",\n \"type\": \"Expression\"\n },\n \"type\": \"Int32\"\n },\n \"StageId\": {\n \"value\": {\n \"value\": \"@pipeline().parameters.StageId\",\n \"type\": \"Expression\"\n },\n \"type\": \"Int32\"\n },\n \"CallingActivity\": {\n \"value\": \"ExecutePipeline\",\n \"type\": \"String\"\n }\n }\n },\n \"linkedServiceName\": {\n \"referenceName\": \"SupportDatabase\",\n \"type\": \"LinkedServiceReference\"\n }\n },\n {\n \"name\": \"Running Pipeline Handler\",\n \"type\": \"ExecutePipeline\",\n \"dependsOn\": [\n {\n \"activity\": \"Execute Pipeline\",\n \"dependencyConditions\": [\n \"Succeeded\"\n ]\n }\n ],\n \"userProperties\": [],\n \"typeProperties\": {\n \"pipeline\": {\n \"referenceName\": \"04-Infant\",\n \"type\": \"PipelineReference\"\n },\n \"waitOnCompletion\": true,\n \"parameters\": {\n \"tenantId\": {\n \"value\": \"@pipeline().parameters.TenantId\",\n \"type\": \"Expression\"\n },\n \"applicationId\": {\n \"value\": \"@activity('Get SPN Details').output.firstRow.Id\",\n \"type\": \"Expression\"\n },\n \"authenticationKey\": {\n \"value\": \"@activity('Get SPN Details').output.firstRow.Secret\",\n \"type\": \"Expression\"\n },\n \"subscriptionId\": {\n \"value\": \"@pipeline().parameters.SubscriptionId\",\n \"type\": \"Expression\"\n },\n \"resourceGroup\": {\n \"value\": \"@item().ResourceGroupName\",\n \"type\": \"Expression\"\n },\n \"factoryName\": {\n \"value\": \"@item().DataFactoryName\",\n \"type\": \"Expression\"\n },\n \"pipelineName\": {\n \"value\": \"@item().PipelineName\",\n \"type\": \"Expression\"\n },\n \"runId\": {\n \"value\": \"@activity('Execute Pipeline').output.RunId\",\n \"type\": \"Expression\"\n },\n \"executionId\": {\n \"value\": \"@pipeline().parameters.ExecutionId\",\n \"type\": \"Expression\"\n },\n \"stageId\": {\n \"value\": \"@pipeline().parameters.StageId\",\n \"type\": \"Expression\"\n },\n \"pipelineId\": {\n \"value\": \"@item().PipelineId\",\n \"type\": \"Expression\"\n }\n }\n }\n },\n {\n \"name\": \"Set Run Id\",\n \"description\": \"Provide the actual ADF run ID back to the current execution table for long term logging and alignment between the metadata other Azure monitoring tools.\",\n \"type\": \"SqlServerStoredProcedure\",\n \"dependsOn\": [\n {\n \"activity\": \"Execute Pipeline\",\n \"dependencyConditions\": [\n \"Succeeded\"\n ]\n }\n ],\n \"policy\": {\n \"timeout\": \"7.00:00:00\",\n \"retry\": 0,\n \"retryIntervalInSeconds\": 30,\n \"secureOutput\": false,\n \"secureInput\": false\n },\n \"userProperties\": [],\n \"typeProperties\": {\n \"storedProcedureName\": \"[[procfwk].[SetLogPipelineRunId]\",\n \"storedProcedureParameters\": {\n \"ExecutionId\": {\n \"value\": {\n \"value\": \"@pipeline().parameters.ExecutionId\",\n \"type\": \"Expression\"\n },\n \"type\": \"Guid\"\n },\n \"PipelineId\": {\n \"value\": {\n \"value\": \"@item().PipelineId\",\n \"type\": \"Expression\"\n },\n \"type\": \"Int32\"\n },\n \"RunId\": {\n \"value\": {\n \"value\": \"@activity('Execute Pipeline').output.RunId\",\n \"type\": \"Expression\"\n },\n \"type\": \"Guid\"\n },\n \"StageId\": {\n \"value\": {\n \"value\": \"@pipeline().parameters.StageId\",\n \"type\": \"Expression\"\n },\n \"type\": \"Int32\"\n }\n }\n },\n \"linkedServiceName\": {\n \"referenceName\": \"SupportDatabase\",\n \"type\": \"LinkedServiceReference\"\n }\n },\n {\n \"name\": \"Check For Alerts\",\n \"description\": \"Checks the properties tables and if any recipients in the database require alerts sending for the current pipeline ID.\",\n \"type\": \"Lookup\",\n \"dependsOn\": [],\n \"policy\": {\n \"timeout\": \"7.00:00:00\",\n \"retry\": 0,\n \"retryIntervalInSeconds\": 30,\n \"secureOutput\": false,\n \"secureInput\": false\n },\n \"userProperties\": [],\n \"typeProperties\": {\n \"source\": {\n \"type\": \"AzureSqlSource\",\n \"sqlReaderStoredProcedureName\": \"[[procfwk].[CheckForEmailAlerts]\",\n \"storedProcedureParameters\": {\n \"PipelineId\": {\n \"type\": \"Int32\",\n \"value\": {\n \"value\": \"@item().PipelineId\",\n \"type\": \"Expression\"\n }\n }\n },\n \"queryTimeout\": \"02:00:00\"\n },\n \"dataset\": {\n \"referenceName\": \"GetSetMetadata\",\n \"type\": \"DatasetReference\",\n \"parameters\": {}\n },\n \"firstRowOnly\": true\n }\n },\n {\n \"name\": \"Send Alerts\",\n \"description\": \"True = alerts need sending.\\nFalse = do nothing.\",\n \"type\": \"IfCondition\",\n \"dependsOn\": [\n {\n \"activity\": \"Set Run Id\",\n \"dependencyConditions\": [\n \"Succeeded\"\n ]\n },\n {\n \"activity\": \"Check For Alerts\",\n \"dependencyConditions\": [\n \"Succeeded\"\n ]\n },\n {\n \"activity\": \"Running Pipeline Handler\",\n \"dependencyConditions\": [\n \"Completed\"\n ]\n }\n ],\n \"userProperties\": [],\n \"typeProperties\": {\n \"expression\": {\n \"value\": \"@activity('Check For Alerts').output.firstRow.SendAlerts\",\n \"type\": \"Expression\"\n },\n \"ifTrueActivities\": [\n {\n \"name\": \"Get Email Parts\",\n \"type\": \"Lookup\",\n \"dependsOn\": [],\n \"policy\": {\n \"timeout\": \"7.00:00:00\",\n \"retry\": 0,\n \"retryIntervalInSeconds\": 30,\n \"secureOutput\": false,\n \"secureInput\": false\n },\n \"userProperties\": [],\n \"typeProperties\": {\n \"source\": {\n \"type\": \"AzureSqlSource\",\n \"sqlReaderStoredProcedureName\": \"[[procfwk].[GetEmailAlertParts]\",\n \"storedProcedureParameters\": {\n \"PipelineId\": {\n \"type\": \"Int32\",\n \"value\": {\n \"value\": \"@item().PipelineId\",\n \"type\": \"Expression\"\n }\n }\n },\n \"queryTimeout\": \"02:00:00\"\n },\n \"dataset\": {\n \"referenceName\": \"GetSetMetadata\",\n \"type\": \"DatasetReference\",\n \"parameters\": {}\n },\n \"firstRowOnly\": true\n }\n },\n {\n \"name\": \"Send Email\",\n \"type\": \"AzureFunctionActivity\",\n \"dependsOn\": [\n {\n \"activity\": \"Get Email Parts\",\n \"dependencyConditions\": [\n \"Succeeded\"\n ]\n }\n ],\n \"policy\": {\n \"timeout\": \"7.00:00:00\",\n \"retry\": 0,\n \"retryIntervalInSeconds\": 30,\n \"secureOutput\": false,\n \"secureInput\": false\n },\n \"userProperties\": [],\n \"typeProperties\": {\n \"functionName\": \"SendEmail\",\n \"method\": \"POST\",\n \"headers\": {},\n \"body\": {\n \"value\": \"@activity('Get Email Parts').output.firstRow\",\n \"type\": \"Expression\"\n }\n },\n \"linkedServiceName\": {\n \"referenceName\": \"FrameworkFunctions\",\n \"type\": \"LinkedServiceReference\"\n }\n }\n ]\n }\n }\n ]\n }\n }\n ],\n \"parameters\": {\n \"StageId\": {\n \"type\": \"int\"\n },\n \"ExecutionId\": {\n \"type\": \"string\"\n },\n \"TenantId\": {\n \"type\": \"string\"\n },\n \"SubscriptionId\": {\n \"type\": \"string\"\n }\n },\n \"variables\": {\n \"FunctionBody\": {\n \"type\": \"String\"\n }\n },\n \"folder\": {\n \"name\": \"_ProcFwk\"\n },\n \"annotations\": [\n \"ADF.procfwk\"\n ]\n },\n \"dependsOn\": [\n \"[concat(variables('factoryId'), '/datasets/GetSetMetadata')]\",\n \"[concat(variables('factoryId'), '/linkedServices/FrameworkFunctions')]\",\n \"[concat(variables('factoryId'), '/linkedServices/SupportDatabase')]\",\n \"[concat(variables('factoryId'), '/pipelines/04-Infant')]\"\n ]\n },\n {\n \"name\": \"[concat(parameters('factoryName'), '/04-Infant')]\",\n \"type\": \"Microsoft.DataFactory/factories/pipelines\",\n \"apiVersion\": \"2018-06-01\",\n \"properties\": {\n \"description\": \"ADF.procfwk infant pipeline used to check when the processing pipeline called by the Child completes and passes the resulting status back to the metadata database.\",\n \"activities\": [\n {\n \"name\": \"Wait Until Pipeline Completes\",\n \"description\": \"Loops until the pipeline called completes.\\n\\nSimple status:\\n- Running = new iteration.\\n- Done = break.\",\n \"type\": \"Until\",\n \"dependsOn\": [\n {\n \"activity\": \"Get Wait Duration\",\n \"dependencyConditions\": [\n \"Succeeded\"\n ]\n }\n ],\n \"userProperties\": [],\n \"typeProperties\": {\n \"expression\": {\n \"value\": \"@equals('Done',activity('Get Pipeline Status').output.SimpleStatus)\",\n \"type\": \"Expression\"\n },\n \"activities\": [\n {\n \"name\": \"Get Pipeline Status\",\n \"description\": \"Checks the status of a given processing pipeline and provides the value for the downstream framework activities to act upon.\",\n \"type\": \"AzureFunctionActivity\",\n \"dependsOn\": [],\n \"policy\": {\n \"timeout\": \"7.00:00:00\",\n \"retry\": 0,\n \"retryIntervalInSeconds\": 30,\n \"secureOutput\": false,\n \"secureInput\": false\n },\n \"userProperties\": [],\n \"typeProperties\": {\n \"functionName\": \"CheckPipelineStatus\",\n \"method\": \"POST\",\n \"headers\": {},\n \"body\": {\n \"value\": \"@concat('\\n{\\n \\\"tenantId\\\": \\\"',pipeline().parameters.TenantId,'\\\",\\n \\\"applicationId\\\": \\\"',pipeline().parameters.applicationId,'\\\",\\n \\\"authenticationKey\\\": \\\"',pipeline().parameters.authenticationKey,'\\\",\\n \\\"subscriptionId\\\": \\\"',pipeline().parameters.subscriptionId,'\\\",\\n \\\"resourceGroup\\\": \\\"',pipeline().parameters.resourceGroup,'\\\",\\n \\\"factoryName\\\": \\\"',pipeline().parameters.factoryName,'\\\",\\n \\\"pipelineName\\\": \\\"',pipeline().parameters.pipelineName,'\\\",\\n \\\"runId\\\": \\\"',pipeline().parameters.runId,'\\\"\\n}')\",\n \"type\": \"Expression\"\n }\n },\n \"linkedServiceName\": {\n \"referenceName\": \"FrameworkFunctions\",\n \"type\": \"LinkedServiceReference\"\n }\n },\n {\n \"name\": \"Wait If Running\",\n \"description\": \"True = Do nothing.\\nFalse = Wait.\",\n \"type\": \"IfCondition\",\n \"dependsOn\": [\n {\n \"activity\": \"Get Pipeline Status\",\n \"dependencyConditions\": [\n \"Succeeded\"\n ]\n }\n ],\n \"userProperties\": [],\n \"typeProperties\": {\n \"expression\": {\n \"value\": \"@equals('Done',activity('Get Pipeline Status').output.SimpleStatus)\",\n \"type\": \"Expression\"\n },\n \"ifFalseActivities\": [\n {\n \"name\": \"Wait for Pipeline\",\n \"description\": \"The processing pipeline is still running so Wait before checking its status again.\",\n \"type\": \"Wait\",\n \"dependsOn\": [],\n \"userProperties\": [],\n \"typeProperties\": {\n \"waitTimeInSeconds\": {\n \"value\": \"@activity('Get Wait Duration').output.firstRow.PropertyValue\",\n \"type\": \"Expression\"\n }\n }\n }\n ]\n }\n },\n {\n \"name\": \"Set Last Update\",\n \"type\": \"SqlServerStoredProcedure\",\n \"dependsOn\": [\n {\n \"activity\": \"Get Pipeline Status\",\n \"dependencyConditions\": [\n \"Succeeded\"\n ]\n }\n ],\n \"policy\": {\n \"timeout\": \"7.00:00:00\",\n \"retry\": 0,\n \"retryIntervalInSeconds\": 30,\n \"secureOutput\": false,\n \"secureInput\": false\n },\n \"userProperties\": [],\n \"typeProperties\": {\n \"storedProcedureName\": \"[[procfwk].[SetLogPipelineLastStatusCheck]\",\n \"storedProcedureParameters\": {\n \"ExecutionId\": {\n \"value\": {\n \"value\": \"@pipeline().parameters.executionId\",\n \"type\": \"Expression\"\n },\n \"type\": \"Guid\"\n },\n \"PipelineId\": {\n \"value\": {\n \"value\": \"@pipeline().parameters.pipelineId\",\n \"type\": \"Expression\"\n },\n \"type\": \"Int32\"\n },\n \"StageId\": {\n \"value\": {\n \"value\": \"@pipeline().parameters.stageId\",\n \"type\": \"Expression\"\n },\n \"type\": \"Int32\"\n }\n }\n },\n \"linkedServiceName\": {\n \"referenceName\": \"SupportDatabase\",\n \"type\": \"LinkedServiceReference\"\n }\n },\n {\n \"name\": \"Log Activity Failure\",\n \"type\": \"SqlServerStoredProcedure\",\n \"dependsOn\": [\n {\n \"activity\": \"Get Pipeline Status\",\n \"dependencyConditions\": [\n \"Failed\"\n ]\n }\n ],\n \"policy\": {\n \"timeout\": \"7.00:00:00\",\n \"retry\": 0,\n \"retryIntervalInSeconds\": 30,\n \"secureOutput\": false,\n \"secureInput\": false\n },\n \"userProperties\": [],\n \"typeProperties\": {\n \"storedProcedureName\": \"[[procfwk].[SetLogActivityFailed]\",\n \"storedProcedureParameters\": {\n \"CallingActivity\": {\n \"value\": \"GetPipelineStatus\",\n \"type\": \"String\"\n },\n \"ExecutionId\": {\n \"value\": {\n \"value\": \"@pipeline().parameters.executionId\",\n \"type\": \"Expression\"\n },\n \"type\": \"Guid\"\n },\n \"PipelineId\": {\n \"value\": {\n \"value\": \"@pipeline().parameters.pipelineId\",\n \"type\": \"Expression\"\n },\n \"type\": \"Int32\"\n },\n \"StageId\": {\n \"value\": {\n \"value\": \"@pipeline().parameters.stageId\",\n \"type\": \"Expression\"\n },\n \"type\": \"Int32\"\n }\n }\n },\n \"linkedServiceName\": {\n \"referenceName\": \"SupportDatabase\",\n \"type\": \"LinkedServiceReference\"\n }\n }\n ],\n \"timeout\": \"7.00:00:00\"\n }\n },\n {\n \"name\": \"Pipeline Result\",\n \"description\": \"Receives the outcome from the function execution for a given processing pipeline and updates the current execution table with different pipelines status values depending on the result (case).\",\n \"type\": \"Switch\",\n \"dependsOn\": [\n {\n \"activity\": \"Wait Until Pipeline Completes\",\n \"dependencyConditions\": [\n \"Completed\"\n ]\n }\n ],\n \"userProperties\": [],\n \"typeProperties\": {\n \"on\": {\n \"value\": \"@activity('Get Pipeline Status').output.Status\",\n \"type\": \"Expression\"\n },\n \"cases\": [\n {\n \"value\": \"Succeeded\",\n \"activities\": [\n {\n \"name\": \"Pipeline Status Succeeded\",\n \"description\": \"Updates the current execution table with a pipeline status of success if the function outcome is succeeded.\",\n \"type\": \"SqlServerStoredProcedure\",\n \"dependsOn\": [],\n \"policy\": {\n \"timeout\": \"7.00:00:00\",\n \"retry\": 0,\n \"retryIntervalInSeconds\": 30,\n \"secureOutput\": false,\n \"secureInput\": false\n },\n \"userProperties\": [],\n \"typeProperties\": {\n \"storedProcedureName\": \"[[procfwk].[SetLogPipelineSuccess]\",\n \"storedProcedureParameters\": {\n \"ExecutionId\": {\n \"value\": {\n \"value\": \"@pipeline().parameters.executionId\",\n \"type\": \"Expression\"\n },\n \"type\": \"Guid\"\n },\n \"PipelineId\": {\n \"value\": {\n \"value\": \"@pipeline().parameters.pipelineId\",\n \"type\": \"Expression\"\n },\n \"type\": \"Int32\"\n },\n \"StageId\": {\n \"value\": {\n \"value\": \"@pipeline().parameters.stageId\",\n \"type\": \"Expression\"\n },\n \"type\": \"Int32\"\n }\n }\n },\n \"linkedServiceName\": {\n \"referenceName\": \"SupportDatabase\",\n \"type\": \"LinkedServiceReference\"\n }\n }\n ]\n },\n {\n \"value\": \"Failed\",\n \"activities\": [\n {\n \"name\": \"Pipeline Status Failed\",\n \"description\": \"Updates the current execution table with a pipeline status of failed if the function outcome is failed. Also blocks pipelines in the downstream execution stage.\",\n \"type\": \"SqlServerStoredProcedure\",\n \"dependsOn\": [],\n \"policy\": {\n \"timeout\": \"7.00:00:00\",\n \"retry\": 0,\n \"retryIntervalInSeconds\": 30,\n \"secureOutput\": false,\n \"secureInput\": false\n },\n \"userProperties\": [],\n \"typeProperties\": {\n \"storedProcedureName\": \"[[procfwk].[SetLogPipelineFailed]\",\n \"storedProcedureParameters\": {\n \"ExecutionId\": {\n \"value\": {\n \"value\": \"@pipeline().parameters.executionId\",\n \"type\": \"Expression\"\n },\n \"type\": \"Guid\"\n },\n \"PipelineId\": {\n \"value\": {\n \"value\": \"@pipeline().parameters.pipelineId\",\n \"type\": \"Expression\"\n },\n \"type\": \"Int32\"\n },\n \"RunId\": {\n \"value\": {\n \"value\": \"@activity('Get Pipeline Status').output.RunId\",\n \"type\": \"Expression\"\n },\n \"type\": \"Guid\"\n },\n \"StageId\": {\n \"value\": {\n \"value\": \"@pipeline().parameters.stageId\",\n \"type\": \"Expression\"\n },\n \"type\": \"Int32\"\n }\n }\n },\n \"linkedServiceName\": {\n \"referenceName\": \"SupportDatabase\",\n \"type\": \"LinkedServiceReference\"\n }\n },\n {\n \"name\": \"Get Error Details\",\n \"description\": \"Get the activity error details for the run ID of the worker pipeline called. Returns an array of all errors.\",\n \"type\": \"AzureFunctionActivity\",\n \"dependsOn\": [],\n \"policy\": {\n \"timeout\": \"7.00:00:00\",\n \"retry\": 0,\n \"retryIntervalInSeconds\": 30,\n \"secureOutput\": false,\n \"secureInput\": false\n },\n \"userProperties\": [],\n \"typeProperties\": {\n \"functionName\": \"GetActivityErrors\",\n \"method\": \"POST\",\n \"headers\": {},\n \"body\": {\n \"value\": \"@concat('\\n{\\n \\\"tenantId\\\": \\\"',pipeline().parameters.TenantId,'\\\",\\n \\\"applicationId\\\": \\\"',pipeline().parameters.applicationId,'\\\",\\n \\\"authenticationKey\\\": \\\"',pipeline().parameters.authenticationKey,'\\\",\\n \\\"subscriptionId\\\": \\\"',pipeline().parameters.subscriptionId,'\\\",\\n \\\"resourceGroup\\\": \\\"',pipeline().parameters.resourceGroup,'\\\",\\n \\\"factoryName\\\": \\\"',pipeline().parameters.factoryName,'\\\",\\n \\\"pipelineName\\\": \\\"',pipeline().parameters.pipelineName,'\\\",\\n \\\"runId\\\": \\\"',pipeline().parameters.runId,'\\\"\\n}')\",\n \"type\": \"Expression\"\n }\n },\n \"linkedServiceName\": {\n \"referenceName\": \"FrameworkFunctions\",\n \"type\": \"LinkedServiceReference\"\n }\n },\n {\n \"name\": \"Log Error Details\",\n \"description\": \"Parses pipeline error details and persists them to the metadata database error log table.\",\n \"type\": \"SqlServerStoredProcedure\",\n \"dependsOn\": [\n {\n \"activity\": \"Get Error Details\",\n \"dependencyConditions\": [\n \"Succeeded\"\n ]\n }\n ],\n \"policy\": {\n \"timeout\": \"7.00:00:00\",\n \"retry\": 0,\n \"retryIntervalInSeconds\": 30,\n \"secureOutput\": false,\n \"secureInput\": false\n },\n \"userProperties\": [],\n \"typeProperties\": {\n \"storedProcedureName\": \"[[procfwk].[SetErrorLogDetails]\",\n \"storedProcedureParameters\": {\n \"JsonErrorDetails\": {\n \"value\": {\n \"value\": \"@string(activity('Get Error Details').output)\",\n \"type\": \"Expression\"\n },\n \"type\": \"String\"\n },\n \"LocalExecutionId\": {\n \"value\": {\n \"value\": \"@pipeline().parameters.executionId\",\n \"type\": \"Expression\"\n },\n \"type\": \"Guid\"\n }\n }\n },\n \"linkedServiceName\": {\n \"referenceName\": \"SupportDatabase\",\n \"type\": \"LinkedServiceReference\"\n }\n }\n ]\n }\n ],\n \"defaultActivities\": [\n {\n \"name\": \"Pipeline Status Unknown\",\n \"description\": \"Updates the current execution table with a pipeline status of unknown if the function returns an unexpected outcome.\",\n \"type\": \"SqlServerStoredProcedure\",\n \"dependsOn\": [],\n \"policy\": {\n \"timeout\": \"7.00:00:00\",\n \"retry\": 0,\n \"retryIntervalInSeconds\": 30,\n \"secureOutput\": false,\n \"secureInput\": false\n },\n \"userProperties\": [],\n \"typeProperties\": {\n \"storedProcedureName\": \"[[procfwk].[SetLogPipelineUnknown]\",\n \"storedProcedureParameters\": {\n \"ExecutionId\": {\n \"value\": {\n \"value\": \"@pipeline().parameters.executionId\",\n \"type\": \"Expression\"\n },\n \"type\": \"Guid\"\n },\n \"PipelineId\": {\n \"value\": {\n \"value\": \"@pipeline().parameters.pipelineId\",\n \"type\": \"Expression\"\n },\n \"type\": \"Int32\"\n },\n \"StageId\": {\n \"value\": {\n \"value\": \"@pipeline().parameters.stageId\",\n \"type\": \"Expression\"\n },\n \"type\": \"Int32\"\n }\n }\n },\n \"linkedServiceName\": {\n \"referenceName\": \"SupportDatabase\",\n \"type\": \"LinkedServiceReference\"\n }\n }\n ]\n }\n },\n {\n \"name\": \"Get Wait Duration\",\n \"type\": \"Lookup\",\n \"dependsOn\": [],\n \"policy\": {\n \"timeout\": \"7.00:00:00\",\n \"retry\": 0,\n \"retryIntervalInSeconds\": 30,\n \"secureOutput\": false,\n \"secureInput\": false\n },\n \"userProperties\": [],\n \"typeProperties\": {\n \"source\": {\n \"type\": \"AzureSqlSource\",\n \"sqlReaderStoredProcedureName\": \"[[procfwk].[GetPropertyValue]\",\n \"storedProcedureParameters\": {\n \"PropertyName\": {\n \"type\": \"String\",\n \"value\": \"PipelineStatusCheckDuration\"\n }\n },\n \"queryTimeout\": \"02:00:00\"\n },\n \"dataset\": {\n \"referenceName\": \"GetSetMetadata\",\n \"type\": \"DatasetReference\",\n \"parameters\": {}\n }\n }\n }\n ],\n \"parameters\": {\n \"tenantId\": {\n \"type\": \"string\"\n },\n \"applicationId\": {\n \"type\": \"string\"\n },\n \"authenticationKey\": {\n \"type\": \"string\"\n },\n \"subscriptionId\": {\n \"type\": \"string\"\n },\n \"resourceGroup\": {\n \"type\": \"string\"\n },\n \"factoryName\": {\n \"type\": \"string\"\n },\n \"pipelineName\": {\n \"type\": \"string\"\n },\n \"runId\": {\n \"type\": \"string\"\n },\n \"executionId\": {\n \"type\": \"string\"\n },\n \"stageId\": {\n \"type\": \"int\"\n },\n \"pipelineId\": {\n \"type\": \"int\"\n }\n },\n \"folder\": {\n \"name\": \"_ProcFwk\"\n },\n \"annotations\": [\n \"ADF.procfwk\"\n ]\n },\n \"dependsOn\": [\n \"[concat(variables('factoryId'), '/datasets/GetSetMetadata')]\",\n \"[concat(variables('factoryId'), '/linkedServices/FrameworkFunctions')]\",\n \"[concat(variables('factoryId'), '/linkedServices/SupportDatabase')]\"\n ]\n },\n {\n \"name\": \"[concat(parameters('factoryName'), '/Intentional Error')]\",\n \"type\": \"Microsoft.DataFactory/factories/pipelines\",\n \"apiVersion\": \"2018-06-01\",\n \"properties\": {\n \"description\": \"Used just so the ADF.procfwk has something to call during development.\",\n \"activities\": [\n {\n \"name\": \"Wait1\",\n \"type\": \"Wait\",\n \"dependsOn\": [],\n \"userProperties\": [],\n \"typeProperties\": {\n \"waitTimeInSeconds\": {\n \"value\": \"@pipeline().parameters.WaitTime\",\n \"type\": \"Expression\"\n }\n }\n },\n {\n \"name\": \"Raise Errors or Not\",\n \"type\": \"IfCondition\",\n \"dependsOn\": [\n {\n \"activity\": \"Wait1\",\n \"dependencyConditions\": [\n \"Succeeded\"\n ]\n }\n ],\n \"userProperties\": [],\n \"typeProperties\": {\n \"expression\": {\n \"value\": \"@equals(pipeline().parameters.RaiseErrors,'true')\",\n \"type\": \"Expression\"\n },\n \"ifTrueActivities\": [\n {\n \"name\": \"Call Fail Procedure\",\n \"type\": \"SqlServerStoredProcedure\",\n \"dependsOn\": [],\n \"policy\": {\n \"timeout\": \"7.00:00:00\",\n \"retry\": 0,\n \"retryIntervalInSeconds\": 30,\n \"secureOutput\": false,\n \"secureInput\": false\n },\n \"userProperties\": [],\n \"typeProperties\": {\n \"storedProcedureName\": \"[[dbo].[FailProcedure]\",\n \"storedProcedureParameters\": {\n \"RaiseError\": {\n \"value\": {\n \"value\": \"@pipeline().parameters.RaiseErrors\",\n \"type\": \"Expression\"\n },\n \"type\": \"String\"\n }\n }\n },\n \"linkedServiceName\": {\n \"referenceName\": \"SupportDatabase\",\n \"type\": \"LinkedServiceReference\"\n }\n },\n {\n \"name\": \"Call Fail Notebook\",\n \"type\": \"DatabricksNotebook\",\n \"dependsOn\": [],\n \"policy\": {\n \"timeout\": \"7.00:00:00\",\n \"retry\": 0,\n \"retryIntervalInSeconds\": 30,\n \"secureOutput\": false,\n \"secureInput\": false\n },\n \"userProperties\": [],\n \"typeProperties\": {\n \"notebookPath\": \"/Playground/Throw Exception\",\n \"baseParameters\": {\n \"RaiseError\": {\n \"value\": \"@pipeline().parameters.RaiseErrors\",\n \"type\": \"Expression\"\n }\n }\n },\n \"linkedServiceName\": {\n \"referenceName\": \"BricksOfData\",\n \"type\": \"LinkedServiceReference\"\n }\n }\n ]\n }\n }\n ],\n \"parameters\": {\n \"RaiseErrors\": {\n \"type\": \"string\",\n \"defaultValue\": \"false\"\n },\n \"WaitTime\": {\n \"type\": \"int\",\n \"defaultValue\": 5\n }\n },\n \"folder\": {\n \"name\": \"_Workers\"\n },\n \"annotations\": [\n \"Worker\"\n ]\n },\n \"dependsOn\": [\n \"[concat(variables('factoryId'), '/linkedServices/SupportDatabase')]\",\n \"[concat(variables('factoryId'), '/linkedServices/BricksOfData')]\"\n ]\n },\n {\n \"name\": \"[concat(parameters('factoryName'), '/Wait 1')]\",\n \"type\": \"Microsoft.DataFactory/factories/pipelines\",\n \"apiVersion\": \"2018-06-01\",\n \"properties\": {\n \"description\": \"Used just so the ADF.procfwk has something to call during development.\",\n \"activities\": [\n {\n \"name\": \"Wait1\",\n \"type\": \"Wait\",\n \"dependsOn\": [],\n \"userProperties\": [],\n \"typeProperties\": {\n \"waitTimeInSeconds\": {\n \"value\": \"@pipeline().parameters.WaitTime\",\n \"type\": \"Expression\"\n }\n }\n }\n ],\n \"parameters\": {\n \"WaitTime\": {\n \"type\": \"int\",\n \"defaultValue\": 5\n }\n },\n \"folder\": {\n \"name\": \"_Workers\"\n },\n \"annotations\": [\n \"Worker\"\n ]\n },\n \"dependsOn\": []\n },\n {\n \"name\": \"[concat(parameters('factoryName'), '/Wait 10')]\",\n \"type\": \"Microsoft.DataFactory/factories/pipelines\",\n \"apiVersion\": \"2018-06-01\",\n \"properties\": {\n \"description\": \"Used just so the ADF.procfwk has something to call during development.\",\n \"activities\": [\n {\n \"name\": \"Wait10\",\n \"type\": \"Wait\",\n \"dependsOn\": [],\n \"userProperties\": [],\n \"typeProperties\": {\n \"waitTimeInSeconds\": {\n \"value\": \"@pipeline().parameters.WaitTime\",\n \"type\": \"Expression\"\n }\n }\n }\n ],\n \"parameters\": {\n \"WaitTime\": {\n \"type\": \"int\",\n \"defaultValue\": 5\n }\n },\n \"folder\": {\n \"name\": \"_Workers\"\n },\n \"annotations\": [\n \"Worker\"\n ]\n },\n \"dependsOn\": []\n },\n {\n \"name\": \"[concat(parameters('factoryName'), '/Wait 2')]\",\n \"type\": \"Microsoft.DataFactory/factories/pipelines\",\n \"apiVersion\": \"2018-06-01\",\n \"properties\": {\n \"description\": \"Used just so the ADF.procfwk has something to call during development.\",\n \"activities\": [\n {\n \"name\": \"Wait2\",\n \"type\": \"Wait\",\n \"dependsOn\": [],\n \"userProperties\": [],\n \"typeProperties\": {\n \"waitTimeInSeconds\": {\n \"value\": \"@pipeline().parameters.WaitTime\",\n \"type\": \"Expression\"\n }\n }\n }\n ],\n \"parameters\": {\n \"WaitTime\": {\n \"type\": \"int\",\n \"defaultValue\": 5\n }\n },\n \"folder\": {\n \"name\": \"_Workers\"\n },\n \"annotations\": [\n \"Worker\"\n ]\n },\n \"dependsOn\": []\n },\n {\n \"name\": \"[concat(parameters('factoryName'), '/Wait 3')]\",\n \"type\": \"Microsoft.DataFactory/factories/pipelines\",\n \"apiVersion\": \"2018-06-01\",\n \"properties\": {\n \"description\": \"Used just so the ADF.procfwk has something to call during development.\",\n \"activities\": [\n {\n \"name\": \"Wait3\",\n \"type\": \"Wait\",\n \"dependsOn\": [],\n \"userProperties\": [],\n \"typeProperties\": {\n \"waitTimeInSeconds\": {\n \"value\": \"@pipeline().parameters.WaitTime\",\n \"type\": \"Expression\"\n }\n }\n }\n ],\n \"parameters\": {\n \"WaitTime\": {\n \"type\": \"int\",\n \"defaultValue\": 5\n }\n },\n \"folder\": {\n \"name\": \"_Workers\"\n },\n \"annotations\": [\n \"Worker\"\n ]\n },\n \"dependsOn\": []\n },\n {\n \"name\": \"[concat(parameters('factoryName'), '/Wait 4')]\",\n \"type\": \"Microsoft.DataFactory/factories/pipelines\",\n \"apiVersion\": \"2018-06-01\",\n \"properties\": {\n \"description\": \"Used just so the ADF.procfwk has something to call during development.\",\n \"activities\": [\n {\n \"name\": \"Wait4\",\n \"type\": \"Wait\",\n \"dependsOn\": [],\n \"userProperties\": [],\n \"typeProperties\": {\n \"waitTimeInSeconds\": {\n \"value\": \"@pipeline().parameters.WaitTime\",\n \"type\": \"Expression\"\n }\n }\n }\n ],\n \"parameters\": {\n \"WaitTime\": {\n \"type\": \"int\",\n \"defaultValue\": 5\n }\n },\n \"folder\": {\n \"name\": \"_Workers\"\n },\n \"annotations\": [\n \"Worker\"\n ]\n },\n \"dependsOn\": []\n },\n {\n \"name\": \"[concat(parameters('factoryName'), '/Wait 5')]\",\n \"type\": \"Microsoft.DataFactory/factories/pipelines\",\n \"apiVersion\": \"2018-06-01\",\n \"properties\": {\n \"description\": \"Used just so the ADF.procfwk has something to call during development.\",\n \"activities\": [\n {\n \"name\": \"Wait5\",\n \"type\": \"Wait\",\n \"dependsOn\": [],\n \"userProperties\": [],\n \"typeProperties\": {\n \"waitTimeInSeconds\": {\n \"value\": \"@pipeline().parameters.WaitTime\",\n \"type\": \"Expression\"\n }\n }\n }\n ],\n \"parameters\": {\n \"WaitTime\": {\n \"type\": \"int\",\n \"defaultValue\": 5\n }\n },\n \"folder\": {\n \"name\": \"_Workers\"\n },\n \"annotations\": [\n \"Worker\"\n ]\n },\n \"dependsOn\": []\n },\n {\n \"name\": \"[concat(parameters('factoryName'), '/Wait 6')]\",\n \"type\": \"Microsoft.DataFactory/factories/pipelines\",\n \"apiVersion\": \"2018-06-01\",\n \"properties\": {\n \"description\": \"Used just so the ADF.procfwk has something to call during development.\",\n \"activities\": [\n {\n \"name\": \"Wait6\",\n \"type\": \"Wait\",\n \"dependsOn\": [],\n \"userProperties\": [],\n \"typeProperties\": {\n \"waitTimeInSeconds\": {\n \"value\": \"@pipeline().parameters.WaitTime\",\n \"type\": \"Expression\"\n }\n }\n }\n ],\n \"parameters\": {\n \"WaitTime\": {\n \"type\": \"int\",\n \"defaultValue\": 5\n }\n },\n \"folder\": {\n \"name\": \"_Workers\"\n },\n \"annotations\": [\n \"Worker\"\n ]\n },\n \"dependsOn\": []\n },\n {\n \"name\": \"[concat(parameters('factoryName'), '/Wait 7')]\",\n \"type\": \"Microsoft.DataFactory/factories/pipelines\",\n \"apiVersion\": \"2018-06-01\",\n \"properties\": {\n \"description\": \"Used just so the ADF.procfwk has something to call during development.\",\n \"activities\": [\n {\n \"name\": \"Wait7\",\n \"type\": \"Wait\",\n \"dependsOn\": [],\n \"userProperties\": [],\n \"typeProperties\": {\n \"waitTimeInSeconds\": {\n \"value\": \"@pipeline().parameters.WaitTime\",\n \"type\": \"Expression\"\n }\n }\n }\n ],\n \"parameters\": {\n \"WaitTime\": {\n \"type\": \"int\",\n \"defaultValue\": 5\n }\n },\n \"folder\": {\n \"name\": \"_Workers\"\n },\n \"annotations\": [\n \"Worker\"\n ]\n },\n \"dependsOn\": []\n },\n {\n \"name\": \"[concat(parameters('factoryName'), '/Wait 8')]\",\n \"type\": \"Microsoft.DataFactory/factories/pipelines\",\n \"apiVersion\": \"2018-06-01\",\n \"properties\": {\n \"description\": \"Used just so the ADF.procfwk has something to call during development.\",\n \"activities\": [\n {\n \"name\": \"Wait8\",\n \"type\": \"Wait\",\n \"dependsOn\": [],\n \"userProperties\": [],\n \"typeProperties\": {\n \"waitTimeInSeconds\": {\n \"value\": \"@pipeline().parameters.WaitTime\",\n \"type\": \"Expression\"\n }\n }\n }\n ],\n \"parameters\": {\n \"WaitTime\": {\n \"type\": \"int\",\n \"defaultValue\": 5\n }\n },\n \"folder\": {\n \"name\": \"_Workers\"\n },\n \"annotations\": [\n \"Worker\"\n ]\n },\n \"dependsOn\": []\n },\n {\n \"name\": \"[concat(parameters('factoryName'), '/Wait 9')]\",\n \"type\": \"Microsoft.DataFactory/factories/pipelines\",\n \"apiVersion\": \"2018-06-01\",\n \"properties\": {\n \"description\": \"Used just so the ADF.procfwk has something to call during development.\",\n \"activities\": [\n {\n \"name\": \"Wait9\",\n \"type\": \"Wait\",\n \"dependsOn\": [],\n \"userProperties\": [],\n \"typeProperties\": {\n \"waitTimeInSeconds\": {\n \"value\": \"@pipeline().parameters.WaitTime\",\n \"type\": \"Expression\"\n }\n }\n }\n ],\n \"parameters\": {\n \"WaitTime\": {\n \"type\": \"int\",\n \"defaultValue\": 90\n }\n },\n \"folder\": {\n \"name\": \"_Workers\"\n },\n \"annotations\": [\n \"Worker\"\n ]\n },\n \"dependsOn\": []\n },\n {\n \"name\": \"[concat(parameters('factoryName'), '/GetSetMetadata')]\",\n \"type\": \"Microsoft.DataFactory/factories/datasets\",\n \"apiVersion\": \"2018-06-01\",\n \"properties\": {\n \"linkedServiceName\": {\n \"referenceName\": \"SupportDatabase\",\n \"type\": \"LinkedServiceReference\"\n },\n \"annotations\": [\n \"ADF.procfwk\"\n ],\n \"type\": \"AzureSqlTable\",\n \"schema\": [],\n \"typeProperties\": {}\n },\n \"dependsOn\": [\n \"[concat(variables('factoryId'), '/linkedServices/SupportDatabase')]\"\n ]\n },\n {\n \"name\": \"[concat(parameters('factoryName'), '/BricksOfData')]\",\n \"type\": \"Microsoft.DataFactory/factories/linkedServices\",\n \"apiVersion\": \"2018-06-01\",\n \"properties\": {\n \"annotations\": [],\n \"type\": \"AzureDatabricks\",\n \"typeProperties\": {\n \"domain\": \"https://northeurope.azuredatabricks.net\",\n \"accessToken\": {\n \"type\": \"AzureKeyVaultSecret\",\n \"store\": {\n \"referenceName\": \"Keys\",\n \"type\": \"LinkedServiceReference\"\n },\n \"secretName\": \"BricksOfDataToken\",\n \"secretVersion\": \"\"\n },\n \"existingClusterId\": \"0422-090117-slots899\"\n }\n },\n \"dependsOn\": [\n \"[concat(variables('factoryId'), '/linkedServices/Keys')]\"\n ]\n },\n {\n \"name\": \"[concat(parameters('factoryName'), '/FrameworkFunctions')]\",\n \"type\": \"Microsoft.DataFactory/factories/linkedServices\",\n \"apiVersion\": \"2018-06-01\",\n \"properties\": {\n \"annotations\": [\n \"ADF.procfwk\"\n ],\n \"type\": \"AzureFunction\",\n \"typeProperties\": {\n \"functionAppUrl\": \"https://frameworksupportfunctions.azurewebsites.net\",\n \"functionKey\": {\n \"type\": \"AzureKeyVaultSecret\",\n \"store\": {\n \"referenceName\": \"Keys\",\n \"type\": \"LinkedServiceReference\"\n },\n \"secretName\": \"FrameworkFunctionsKey\"\n }\n }\n },\n \"dependsOn\": [\n \"[concat(variables('factoryId'), '/linkedServices/Keys')]\"\n ]\n },\n {\n \"name\": \"[concat(parameters('factoryName'), '/Keys')]\",\n \"type\": \"Microsoft.DataFactory/factories/linkedServices\",\n \"apiVersion\": \"2018-06-01\",\n \"properties\": {\n \"description\": \"Connection to Key Vault for all other ADF linked service credentials required to run the processing framework.\",\n \"annotations\": [\n \"ADF.procfwk\"\n ],\n \"type\": \"AzureKeyVault\",\n \"typeProperties\": {\n \"baseUrl\": \"[parameters('Keys_properties_typeProperties_baseUrl')]\"\n }\n },\n \"dependsOn\": []\n },\n {\n \"name\": \"[concat(parameters('factoryName'), '/SupportDatabase')]\",\n \"type\": \"Microsoft.DataFactory/factories/linkedServices\",\n \"apiVersion\": \"2018-06-01\",\n \"properties\": {\n \"description\": \"Connection between ADF and processing framework metadata SQLDB.\",\n \"annotations\": [\n \"ADF.procfwk\"\n ],\n \"type\": \"AzureSqlDatabase\",\n \"typeProperties\": {\n \"connectionString\": {\n \"type\": \"AzureKeyVaultSecret\",\n \"store\": {\n \"referenceName\": \"Keys\",\n \"type\": \"LinkedServiceReference\"\n },\n \"secretName\": \"[parameters('SupportDatabase_properties_typeProperties_connectionString_secretName')]\"\n }\n }\n },\n \"dependsOn\": [\n \"[concat(variables('factoryId'), '/linkedServices/Keys')]\"\n ]\n },\n {\n \"name\": \"[concat(parameters('factoryName'), '/FunctionalTestingTrigger')]\",\n \"type\": \"Microsoft.DataFactory/factories/triggers\",\n \"apiVersion\": \"2018-06-01\",\n \"properties\": {\n \"description\": \"Used for functional testing of the framework in a dedicated environment.\",\n \"annotations\": [\n \"ADF.procfwk\"\n ],\n \"runtimeState\": \"Stopped\",\n \"pipelines\": [\n {\n \"pipelineReference\": {\n \"referenceName\": \"01-Grandparent\",\n \"type\": \"PipelineReference\"\n },\n \"parameters\": {}\n }\n ],\n \"type\": \"ScheduleTrigger\",\n \"typeProperties\": {\n \"recurrence\": {\n \"frequency\": \"Hour\",\n \"interval\": 2,\n \"startTime\": \"2020-04-06T15:00:00.000Z\",\n \"timeZone\": \"UTC\"\n }\n }\n },\n \"dependsOn\": [\n \"[concat(variables('factoryId'), '/pipelines/01-Grandparent')]\"\n ]\n }\n ]\n}"
},
{
"path": "ARM Templates/Data Factory/v1.8 Export.json",
"content": "{\n \"$schema\": \"http://schema.management.azure.com/schemas/2015-01-01/deploymentTemplate.json#\",\n \"contentVersion\": \"1.0.0.0\",\n \"parameters\": {\n \"factoryName\": {\n \"type\": \"string\",\n \"metadata\": \"Data Factory name\",\n \"defaultValue\": \"\"\n },\n \"FrameworkFunctions_properties_typeProperties_functionAppUrl\": {\n \"type\": \"string\",\n \"defaultValue\": \"\"\n },\n \"Keys_properties_typeProperties_baseUrl\": {\n \"type\": \"string\",\n \"defaultValue\": \"\"\n },\n \"SupportDatabase_properties_typeProperties_connectionString_secretName\": {\n \"type\": \"string\",\n \"defaultValue\": \"\"\n }\n },\n \"variables\": {\n \"factoryId\": \"[concat('Microsoft.DataFactory/factories/', parameters('factoryName'))]\"\n },\n \"resources\": [\n {\n \"name\": \"[concat(parameters('factoryName'), '/01-Grandparent')]\",\n \"type\": \"Microsoft.DataFactory/factories/pipelines\",\n \"apiVersion\": \"2018-06-01\",\n \"properties\": {\n \"description\": \"ADF.procfwk grandparent pipeline used optionally to bootstrap any wider processes in your Data Factory that then calls the processing framework.\",\n \"activities\": [\n {\n \"name\": \"Framework Processing\",\n \"type\": \"ExecutePipeline\",\n \"dependsOn\": [\n {\n \"activity\": \"Set Random Waits\",\n \"dependencyConditions\": [\n \"Succeeded\"\n ]\n }\n ],\n \"userProperties\": [],\n \"typeProperties\": {\n \"pipeline\": {\n \"referenceName\": \"02-Parent\",\n \"type\": \"PipelineReference\"\n },\n \"waitOnCompletion\": true,\n \"parameters\": {}\n }\n },\n {\n \"name\": \"Set Random Waits\",\n \"description\": \"For functional testing only.\",\n \"type\": \"SqlServerStoredProcedure\",\n \"dependsOn\": [],\n \"policy\": {\n \"timeout\": \"7.00:00:00\",\n \"retry\": 0,\n \"retryIntervalInSeconds\": 30,\n \"secureOutput\": false,\n \"secureInput\": false\n },\n \"userProperties\": [],\n \"typeProperties\": {\n \"storedProcedureName\": \"[[dbo].[SetRandomWaitValues]\"\n },\n \"linkedServiceName\": {\n \"referenceName\": \"SupportDatabase\",\n \"type\": \"LinkedServiceReference\"\n }\n }\n ],\n \"folder\": {\n \"name\": \"_ProcFwk\"\n },\n \"annotations\": [\n \"ADF.procfwk\"\n ]\n },\n \"dependsOn\": [\n \"[concat(variables('factoryId'), '/pipelines/02-Parent')]\",\n \"[concat(variables('factoryId'), '/linkedServices/SupportDatabase')]\"\n ]\n },\n {\n \"name\": \"[concat(parameters('factoryName'), '/02-Parent')]\",\n \"type\": \"Microsoft.DataFactory/factories/pipelines\",\n \"apiVersion\": \"2018-06-01\",\n \"properties\": {\n \"description\": \"ADF.procfwk parent pipeline used to bootstrap the orchestration framework in perform the first level ForEach calls in sequence for the metadata stages.\",\n \"activities\": [\n {\n \"name\": \"Get Stages\",\n \"description\": \"Returns a distinct list of execution stages within the framework metadata.\",\n \"type\": \"Lookup\",\n \"dependsOn\": [\n {\n \"activity\": \"Execution Wrapper\",\n \"dependencyConditions\": [\n \"Succeeded\"\n ]\n }\n ],\n \"policy\": {\n \"timeout\": \"7.00:00:00\",\n \"retry\": 0,\n \"retryIntervalInSeconds\": 30,\n \"secureOutput\": false,\n \"secureInput\": false\n },\n \"userProperties\": [],\n \"typeProperties\": {\n \"source\": {\n \"type\": \"AzureSqlSource\",\n \"sqlReaderStoredProcedureName\": \"[[procfwk].[GetStages]\",\n \"storedProcedureParameters\": {\n \"ExecutionId\": {\n \"type\": \"Guid\",\n \"value\": {\n \"value\": \"@activity('Execution Wrapper').output.firstRow.ExecutionId\",\n \"type\": \"Expression\"\n }\n }\n },\n \"queryTimeout\": \"02:00:00\"\n },\n \"dataset\": {\n \"referenceName\": \"GetSetMetadata\",\n \"type\": \"DatasetReference\",\n \"parameters\": {}\n },\n \"firstRowOnly\": false\n }\n },\n {\n \"name\": \"Execute Stages\",\n \"description\": \"Top level ForEach to sequentially call all processing stages within the framework metadata. Items for iteration passed from the Get Stages lookup activity.\",\n \"type\": \"ForEach\",\n \"dependsOn\": [\n {\n \"activity\": \"Get Stages\",\n \"dependencyConditions\": [\n \"Succeeded\"\n ]\n },\n {\n \"activity\": \"Get Tenant Id\",\n \"dependencyConditions\": [\n \"Succeeded\"\n ]\n },\n {\n \"activity\": \"Get Subscription Id\",\n \"dependencyConditions\": [\n \"Succeeded\"\n ]\n }\n ],\n \"userProperties\": [],\n \"typeProperties\": {\n \"items\": {\n \"value\": \"@activity('Get Stages').output.value\",\n \"type\": \"Expression\"\n },\n \"isSequential\": true,\n \"activities\": [\n {\n \"name\": \"Stage Executor\",\n \"description\": \"Call to the framework generic child pipeline for a given execution stage.\",\n \"type\": \"ExecutePipeline\",\n \"dependsOn\": [\n {\n \"activity\": \"Log Stage Preparing\",\n \"dependencyConditions\": [\n \"Succeeded\"\n ]\n }\n ],\n \"userProperties\": [],\n \"typeProperties\": {\n \"pipeline\": {\n \"referenceName\": \"03-Child\",\n \"type\": \"PipelineReference\"\n },\n \"waitOnCompletion\": true,\n \"parameters\": {\n \"StageId\": {\n \"value\": \"@item().StageId\",\n \"type\": \"Expression\"\n },\n \"ExecutionId\": {\n \"value\": \"@activity('Execution Wrapper').output.firstRow.ExecutionId\",\n \"type\": \"Expression\"\n },\n \"TenantId\": {\n \"value\": \"@activity('Get Tenant Id').output.firstRow.PropertyValue\",\n \"type\": \"Expression\"\n },\n \"SubscriptionId\": {\n \"value\": \"@activity('Get Subscription Id').output.firstRow.PropertyValue\",\n \"type\": \"Expression\"\n }\n }\n }\n },\n {\n \"name\": \"Log Stage Preparing\",\n \"description\": \"Update the current execution table flagging all pipelines within the stage as preparing.\",\n \"type\": \"SqlServerStoredProcedure\",\n \"dependsOn\": [\n {\n \"activity\": \"Check and Update Blockers\",\n \"dependencyConditions\": [\n \"Succeeded\"\n ]\n }\n ],\n \"policy\": {\n \"timeout\": \"7.00:00:00\",\n \"retry\": 0,\n \"retryIntervalInSeconds\": 30,\n \"secureOutput\": false,\n \"secureInput\": false\n },\n \"userProperties\": [],\n \"typeProperties\": {\n \"storedProcedureName\": \"[[procfwk].[SetLogStagePreparing]\",\n \"storedProcedureParameters\": {\n \"ExecutionId\": {\n \"value\": {\n \"value\": \"@activity('Execution Wrapper').output.firstRow.ExecutionId\",\n \"type\": \"Expression\"\n },\n \"type\": \"Guid\"\n },\n \"StageId\": {\n \"value\": {\n \"value\": \"@item().StageId\",\n \"type\": \"Expression\"\n },\n \"type\": \"Int32\"\n }\n }\n },\n \"linkedServiceName\": {\n \"referenceName\": \"SupportDatabase\",\n \"type\": \"LinkedServiceReference\"\n }\n },\n {\n \"name\": \"Check and Update Blockers\",\n \"description\": \"Used to double check and stop the next execution stage if failures and blockers have be incurred. This also depends on the failure handling property value which defines the stored procedure behaviour.\",\n \"type\": \"SqlServerStoredProcedure\",\n \"dependsOn\": [],\n \"policy\": {\n \"timeout\": \"7.00:00:00\",\n \"retry\": 0,\n \"retryIntervalInSeconds\": 30,\n \"secureOutput\": false,\n \"secureInput\": false\n },\n \"userProperties\": [],\n \"typeProperties\": {\n \"storedProcedureName\": \"[[procfwk].[CheckForBlockedPipelines]\",\n \"storedProcedureParameters\": {\n \"StageId\": {\n \"value\": {\n \"value\": \"@item().StageId\",\n \"type\": \"Expression\"\n },\n \"type\": \"Int32\"\n }\n }\n },\n \"linkedServiceName\": {\n \"referenceName\": \"SupportDatabase\",\n \"type\": \"LinkedServiceReference\"\n }\n }\n ]\n }\n },\n {\n \"name\": \"Execution Wrapper\",\n \"description\": \"Wrapper to reset and restart processing or create a completely new execution instance of the framework metadata.\",\n \"type\": \"Lookup\",\n \"dependsOn\": [\n {\n \"activity\": \"Clean Up Previous Run\",\n \"dependencyConditions\": [\n \"Succeeded\"\n ]\n }\n ],\n \"policy\": {\n \"timeout\": \"7.00:00:00\",\n \"retry\": 0,\n \"retryIntervalInSeconds\": 30,\n \"secureOutput\": false,\n \"secureInput\": false\n },\n \"userProperties\": [],\n \"typeProperties\": {\n \"source\": {\n \"type\": \"AzureSqlSource\",\n \"sqlReaderStoredProcedureName\": \"[[procfwk].[ExecutionWrapper]\",\n \"storedProcedureParameters\": {\n \"CallingDataFactory\": {\n \"type\": \"String\",\n \"value\": {\n \"value\": \"@pipeline().DataFactory\",\n \"type\": \"Expression\"\n }\n }\n },\n \"queryTimeout\": \"02:00:00\"\n },\n \"dataset\": {\n \"referenceName\": \"GetSetMetadata\",\n \"type\": \"DatasetReference\",\n \"parameters\": {}\n }\n }\n },\n {\n \"name\": \"Check Outcome and Update Logs\",\n \"description\": \"After a successful execution run the current execution metadata is moved to the long term logging table by this stored procedure call. Otherwise an error will be raised.\",\n \"type\": \"SqlServerStoredProcedure\",\n \"dependsOn\": [\n {\n \"activity\": \"Execute Stages\",\n \"dependencyConditions\": [\n \"Succeeded\"\n ]\n }\n ],\n \"policy\": {\n \"timeout\": \"7.00:00:00\",\n \"retry\": 0,\n \"retryIntervalInSeconds\": 30,\n \"secureOutput\": false,\n \"secureInput\": false\n },\n \"userProperties\": [],\n \"typeProperties\": {\n \"storedProcedureName\": \"[[procfwk].[UpdateExecutionLog]\"\n },\n \"linkedServiceName\": {\n \"referenceName\": \"SupportDatabase\",\n \"type\": \"LinkedServiceReference\"\n }\n },\n {\n \"name\": \"Get Tenant Id\",\n \"description\": \"Returning the Azure Tenant Id from the metadata properties table.\",\n \"type\": \"Lookup\",\n \"dependsOn\": [\n {\n \"activity\": \"Metadata Integrity Checks\",\n \"dependencyConditions\": [\n \"Succeeded\"\n ]\n }\n ],\n \"policy\": {\n \"timeout\": \"7.00:00:00\",\n \"retry\": 0,\n \"retryIntervalInSeconds\": 30,\n \"secureOutput\": false,\n \"secureInput\": false\n },\n \"userProperties\": [],\n \"typeProperties\": {\n \"source\": {\n \"type\": \"AzureSqlSource\",\n \"sqlReaderStoredProcedureName\": \"[[procfwk].[GetPropertyValue]\",\n \"storedProcedureParameters\": {\n \"PropertyName\": {\n \"type\": \"String\",\n \"value\": \"TenantId\"\n }\n },\n \"queryTimeout\": \"02:00:00\"\n },\n \"dataset\": {\n \"referenceName\": \"GetSetMetadata\",\n \"type\": \"DatasetReference\",\n \"parameters\": {}\n },\n \"firstRowOnly\": true\n }\n },\n {\n \"name\": \"Get Subscription Id\",\n \"description\": \"Returning the Azure Subscription Id from the metadata properties table.\",\n \"type\": \"Lookup\",\n \"dependsOn\": [\n {\n \"activity\": \"Metadata Integrity Checks\",\n \"dependencyConditions\": [\n \"Succeeded\"\n ]\n }\n ],\n \"policy\": {\n \"timeout\": \"7.00:00:00\",\n \"retry\": 0,\n \"retryIntervalInSeconds\": 30,\n \"secureOutput\": false,\n \"secureInput\": true\n },\n \"userProperties\": [],\n \"typeProperties\": {\n \"source\": {\n \"type\": \"AzureSqlSource\",\n \"sqlReaderStoredProcedureName\": \"[[procfwk].[GetPropertyValue]\",\n \"storedProcedureParameters\": {\n \"PropertyName\": {\n \"type\": \"String\",\n \"value\": \"SubscriptionId\"\n }\n },\n \"queryTimeout\": \"02:00:00\"\n },\n \"dataset\": {\n \"referenceName\": \"GetSetMetadata\",\n \"type\": \"DatasetReference\",\n \"parameters\": {}\n }\n }\n },\n {\n \"name\": \"Metadata Integrity Checks\",\n \"description\": \"Performs a series of checks on all metadata held in the framework SQLDB. This is intended to raise errors before an execution run even starts.\",\n \"type\": \"Lookup\",\n \"dependsOn\": [],\n \"policy\": {\n \"timeout\": \"7.00:00:00\",\n \"retry\": 0,\n \"retryIntervalInSeconds\": 30,\n \"secureOutput\": false,\n \"secureInput\": false\n },\n \"userProperties\": [],\n \"typeProperties\": {\n \"source\": {\n \"type\": \"AzureSqlSource\",\n \"sqlReaderStoredProcedureName\": \"[[procfwk].[CheckMetadataIntegrity]\",\n \"storedProcedureParameters\": {\n \"DebugMode\": {\n \"type\": \"Boolean\",\n \"value\": \"false\"\n }\n },\n \"queryTimeout\": \"02:00:00\"\n },\n \"dataset\": {\n \"referenceName\": \"GetSetMetadata\",\n \"type\": \"DatasetReference\",\n \"parameters\": {}\n },\n \"firstRowOnly\": false\n }\n },\n {\n \"name\": \"Clean Up Previous Run\",\n \"description\": \"Handle Worker pipelines that are reported as Running when the parent pipeline is called again. Get what the actual status of those pipelines is.\",\n \"type\": \"ForEach\",\n \"dependsOn\": [\n {\n \"activity\": \"Metadata Integrity Checks\",\n \"dependencyConditions\": [\n \"Succeeded\"\n ]\n }\n ],\n \"userProperties\": [],\n \"typeProperties\": {\n \"items\": {\n \"value\": \"@activity('Metadata Integrity Checks').output.value\",\n \"type\": \"Expression\"\n },\n \"isSequential\": false,\n \"activities\": [\n {\n \"name\": \"Get SPN Details\",\n \"type\": \"Lookup\",\n \"dependsOn\": [],\n \"policy\": {\n \"timeout\": \"7.00:00:00\",\n \"retry\": 0,\n \"retryIntervalInSeconds\": 30,\n \"secureOutput\": false,\n \"secureInput\": false\n },\n \"userProperties\": [],\n \"typeProperties\": {\n \"source\": {\n \"type\": \"AzureSqlSource\",\n \"sqlReaderStoredProcedureName\": \"[[procfwk].[GetServicePrincipal]\",\n \"storedProcedureParameters\": {\n \"DataFactory\": {\n \"type\": \"String\",\n \"value\": {\n \"value\": \"@item().DataFactoryName\",\n \"type\": \"Expression\"\n }\n },\n \"PipelineName\": {\n \"type\": \"String\",\n \"value\": {\n \"value\": \"@item().PipelineName\",\n \"type\": \"Expression\"\n }\n }\n },\n \"queryTimeout\": \"02:00:00\"\n },\n \"dataset\": {\n \"referenceName\": \"GetSetMetadata\",\n \"type\": \"DatasetReference\",\n \"parameters\": {}\n }\n }\n },\n {\n \"name\": \"Log Pipeline Checking\",\n \"type\": \"SqlServerStoredProcedure\",\n \"dependsOn\": [],\n \"policy\": {\n \"timeout\": \"7.00:00:00\",\n \"retry\": 0,\n \"retryIntervalInSeconds\": 30,\n \"secureOutput\": false,\n \"secureInput\": false\n },\n \"userProperties\": [],\n \"typeProperties\": {\n \"storedProcedureName\": \"[[procfwk].[SetLogPipelineChecking]\",\n \"storedProcedureParameters\": {\n \"ExecutionId\": {\n \"value\": {\n \"value\": \"@item().LocalExecutionId\",\n \"type\": \"Expression\"\n },\n \"type\": \"Guid\"\n },\n \"PipelineId\": {\n \"value\": {\n \"value\": \"@item().PipelineId\",\n \"type\": \"Expression\"\n },\n \"type\": \"Int32\"\n },\n \"StageId\": {\n \"value\": {\n \"value\": \"@item().StageId\",\n \"type\": \"Expression\"\n },\n \"type\": \"Int32\"\n }\n }\n },\n \"linkedServiceName\": {\n \"referenceName\": \"SupportDatabase\",\n \"type\": \"LinkedServiceReference\"\n }\n },\n {\n \"name\": \"Get Pipeline Status\",\n \"type\": \"AzureFunctionActivity\",\n \"dependsOn\": [\n {\n \"activity\": \"Get SPN Details\",\n \"dependencyConditions\": [\n \"Succeeded\"\n ]\n },\n {\n \"activity\": \"Log Pipeline Checking\",\n \"dependencyConditions\": [\n \"Succeeded\"\n ]\n }\n ],\n \"policy\": {\n \"timeout\": \"7.00:00:00\",\n \"retry\": 0,\n \"retryIntervalInSeconds\": 30,\n \"secureOutput\": false,\n \"secureInput\": false\n },\n \"userProperties\": [],\n \"typeProperties\": {\n \"functionName\": \"CheckPipelineStatus\",\n \"method\": \"POST\",\n \"headers\": {},\n \"body\": {\n \"value\": \"@concat('\\n{\\n \\\"tenantId\\\": \\\"',item().TenantId,'\\\",\\n \\\"applicationId\\\": \\\"',activity('Get SPN Details').output.firstRow.Id,'\\\",\\n \\\"authenticationKey\\\": \\\"',activity('Get SPN Details').output.firstRow.Secret,'\\\",\\n \\\"subscriptionId\\\": \\\"',item().SubscriptionId,'\\\",\\n \\\"resourceGroup\\\": \\\"',item().ResourceGroupName,'\\\",\\n \\\"factoryName\\\": \\\"',item().DataFactoryName,'\\\",\\n \\\"pipelineName\\\": \\\"',item().PipelineName,'\\\",\\n \\\"runId\\\": \\\"',item().AdfPipelineRunId,'\\\"\\n}')\",\n \"type\": \"Expression\"\n }\n },\n \"linkedServiceName\": {\n \"referenceName\": \"FrameworkFunctions\",\n \"type\": \"LinkedServiceReference\"\n }\n },\n {\n \"name\": \"Set Pipeline Status\",\n \"type\": \"Switch\",\n \"dependsOn\": [\n {\n \"activity\": \"Get Pipeline Status\",\n \"dependencyConditions\": [\n \"Succeeded\"\n ]\n }\n ],\n \"userProperties\": [],\n \"typeProperties\": {\n \"on\": {\n \"value\": \"@activity('Get Pipeline Status').output.Status\",\n \"type\": \"Expression\"\n },\n \"cases\": [\n {\n \"value\": \"Failed\",\n \"activities\": [\n {\n \"name\": \"Pipeline Status Failed\",\n \"type\": \"SqlServerStoredProcedure\",\n \"dependsOn\": [],\n \"policy\": {\n \"timeout\": \"7.00:00:00\",\n \"retry\": 0,\n \"retryIntervalInSeconds\": 30,\n \"secureOutput\": false,\n \"secureInput\": false\n },\n \"userProperties\": [],\n \"typeProperties\": {\n \"storedProcedureName\": \"[[procfwk].[SetLogPipelineFailed]\",\n \"storedProcedureParameters\": {\n \"ExecutionId\": {\n \"value\": {\n \"value\": \"@item().LocalExecutionId\",\n \"type\": \"Expression\"\n },\n \"type\": \"Guid\"\n },\n \"PipelineId\": {\n \"value\": {\n \"value\": \"@item().PipelineId\",\n \"type\": \"Expression\"\n },\n \"type\": \"Int32\"\n },\n \"RunId\": {\n \"value\": null,\n \"type\": \"Guid\"\n },\n \"StageId\": {\n \"value\": {\n \"value\": \"@item().StageId\",\n \"type\": \"Expression\"\n },\n \"type\": \"Int32\"\n }\n }\n },\n \"linkedServiceName\": {\n \"referenceName\": \"SupportDatabase\",\n \"type\": \"LinkedServiceReference\"\n }\n }\n ]\n },\n {\n \"value\": \"Succeeded\",\n \"activities\": [\n {\n \"name\": \"Pipeline Status Succeeded\",\n \"type\": \"SqlServerStoredProcedure\",\n \"dependsOn\": [],\n \"policy\": {\n \"timeout\": \"7.00:00:00\",\n \"retry\": 0,\n \"retryIntervalInSeconds\": 30,\n \"secureOutput\": false,\n \"secureInput\": false\n },\n \"userProperties\": [],\n \"typeProperties\": {\n \"storedProcedureName\": \"[[procfwk].[SetLogPipelineSuccess]\",\n \"storedProcedureParameters\": {\n \"ExecutionId\": {\n \"value\": {\n \"value\": \"@item().LocalExecutionId\",\n \"type\": \"Expression\"\n },\n \"type\": \"Guid\"\n },\n \"PipelineId\": {\n \"value\": {\n \"value\": \"@item().PipelineId\",\n \"type\": \"Expression\"\n },\n \"type\": \"Int32\"\n },\n \"StageId\": {\n \"value\": {\n \"value\": \"@item().StageId\",\n \"type\": \"Expression\"\n },\n \"type\": \"Int32\"\n }\n }\n },\n \"linkedServiceName\": {\n \"referenceName\": \"SupportDatabase\",\n \"type\": \"LinkedServiceReference\"\n }\n }\n ]\n },\n {\n \"value\": \"Queued\",\n \"activities\": [\n {\n \"name\": \"Pipeline Status Queued - Running\",\n \"type\": \"SqlServerStoredProcedure\",\n \"dependsOn\": [],\n \"policy\": {\n \"timeout\": \"7.00:00:00\",\n \"retry\": 0,\n \"retryIntervalInSeconds\": 30,\n \"secureOutput\": false,\n \"secureInput\": false\n },\n \"userProperties\": [],\n \"typeProperties\": {\n \"storedProcedureName\": \"[[procfwk].[SetLogPipelineRunning]\",\n \"storedProcedureParameters\": {\n \"ExecutionId\": {\n \"value\": {\n \"value\": \"@item().LocalExecutionId\",\n \"type\": \"Expression\"\n },\n \"type\": \"Guid\"\n },\n \"PipelineId\": {\n \"value\": {\n \"value\": \"@item().PipelineId\",\n \"type\": \"Expression\"\n },\n \"type\": \"Int32\"\n },\n \"StageId\": {\n \"value\": {\n \"value\": \"@item().StageId\",\n \"type\": \"Expression\"\n },\n \"type\": \"Int32\"\n }\n }\n },\n \"linkedServiceName\": {\n \"referenceName\": \"SupportDatabase\",\n \"type\": \"LinkedServiceReference\"\n }\n }\n ]\n },\n {\n \"value\": \"InProgress\",\n \"activities\": [\n {\n \"name\": \"Pipeline Status InProgress - Running\",\n \"type\": \"SqlServerStoredProcedure\",\n \"dependsOn\": [],\n \"policy\": {\n \"timeout\": \"7.00:00:00\",\n \"retry\": 0,\n \"retryIntervalInSeconds\": 30,\n \"secureOutput\": false,\n \"secureInput\": false\n },\n \"userProperties\": [],\n \"typeProperties\": {\n \"storedProcedureName\": \"[[procfwk].[SetLogPipelineRunning]\",\n \"storedProcedureParameters\": {\n \"ExecutionId\": {\n \"value\": {\n \"value\": \"@item().LocalExecutionId\",\n \"type\": \"Expression\"\n },\n \"type\": \"Guid\"\n },\n \"PipelineId\": {\n \"value\": {\n \"value\": \"@item().PipelineId\",\n \"type\": \"Expression\"\n },\n \"type\": \"Int32\"\n },\n \"StageId\": {\n \"value\": {\n \"value\": \"@item().StageId\",\n \"type\": \"Expression\"\n },\n \"type\": \"Int32\"\n }\n }\n },\n \"linkedServiceName\": {\n \"referenceName\": \"SupportDatabase\",\n \"type\": \"LinkedServiceReference\"\n }\n }\n ]\n }\n ],\n \"defaultActivities\": [\n {\n \"name\": \"Pipeline Status Unknown\",\n \"type\": \"SqlServerStoredProcedure\",\n \"dependsOn\": [],\n \"policy\": {\n \"timeout\": \"7.00:00:00\",\n \"retry\": 0,\n \"retryIntervalInSeconds\": 30,\n \"secureOutput\": false,\n \"secureInput\": false\n },\n \"userProperties\": [],\n \"typeProperties\": {\n \"storedProcedureName\": \"[[procfwk].[SetLogPipelineUnknown]\",\n \"storedProcedureParameters\": {\n \"ExecutionId\": {\n \"value\": {\n \"value\": \"@item().LocalExecutionId\",\n \"type\": \"Expression\"\n },\n \"type\": \"Guid\"\n },\n \"PipelineId\": {\n \"value\": {\n \"value\": \"@item().PipelineId\",\n \"type\": \"Expression\"\n },\n \"type\": \"Int32\"\n },\n \"StageId\": {\n \"value\": {\n \"value\": \"@item().StageId\",\n \"type\": \"Expression\"\n },\n \"type\": \"Int32\"\n }\n }\n },\n \"linkedServiceName\": {\n \"referenceName\": \"SupportDatabase\",\n \"type\": \"LinkedServiceReference\"\n }\n }\n ]\n }\n },\n {\n \"name\": \"Set Last Check DateTime\",\n \"type\": \"SqlServerStoredProcedure\",\n \"dependsOn\": [\n {\n \"activity\": \"Get Pipeline Status\",\n \"dependencyConditions\": [\n \"Succeeded\"\n ]\n }\n ],\n \"policy\": {\n \"timeout\": \"7.00:00:00\",\n \"retry\": 0,\n \"retryIntervalInSeconds\": 30,\n \"secureOutput\": false,\n \"secureInput\": false\n },\n \"userProperties\": [],\n \"typeProperties\": {\n \"storedProcedureName\": \"[[procfwk].[SetLogPipelineLastStatusCheck]\",\n \"storedProcedureParameters\": {\n \"ExecutionId\": {\n \"value\": {\n \"value\": \"@item().LocalExecutionId\",\n \"type\": \"Expression\"\n },\n \"type\": \"Guid\"\n },\n \"PipelineId\": {\n \"value\": {\n \"value\": \"@item().PipelineId\",\n \"type\": \"Expression\"\n },\n \"type\": \"Int32\"\n },\n \"StageId\": {\n \"value\": {\n \"value\": \"@item().StageId\",\n \"type\": \"Expression\"\n },\n \"type\": \"Int32\"\n }\n }\n },\n \"linkedServiceName\": {\n \"referenceName\": \"SupportDatabase\",\n \"type\": \"LinkedServiceReference\"\n }\n }\n ]\n }\n }\n ],\n \"folder\": {\n \"name\": \"_ProcFwk\"\n },\n \"annotations\": [\n \"ADF.procfwk\"\n ]\n },\n \"dependsOn\": [\n \"[concat(variables('factoryId'), '/datasets/GetSetMetadata')]\",\n \"[concat(variables('factoryId'), '/linkedServices/SupportDatabase')]\",\n \"[concat(variables('factoryId'), '/pipelines/03-Child')]\",\n \"[concat(variables('factoryId'), '/linkedServices/FrameworkFunctions')]\"\n ]\n },\n {\n \"name\": \"[concat(parameters('factoryName'), '/03-Child')]\",\n \"type\": \"Microsoft.DataFactory/factories/pipelines\",\n \"apiVersion\": \"2018-06-01\",\n \"properties\": {\n \"description\": \"ADF.procfwk child pipeline used to execute Worker pipelines within a given execution stage. This pipeline will be called once for each stage, then execute all Workers in parallel.\",\n \"activities\": [\n {\n \"name\": \"Get Pipelines\",\n \"description\": \"Returns all pipelines from the metadata to be executed within a given processing stage.\",\n \"type\": \"Lookup\",\n \"dependsOn\": [],\n \"policy\": {\n \"timeout\": \"7.00:00:00\",\n \"retry\": 0,\n \"retryIntervalInSeconds\": 30,\n \"secureOutput\": false,\n \"secureInput\": false\n },\n \"userProperties\": [],\n \"typeProperties\": {\n \"source\": {\n \"type\": \"AzureSqlSource\",\n \"sqlReaderStoredProcedureName\": \"[[procfwk].[GetPipelinesInStage]\",\n \"storedProcedureParameters\": {\n \"StageId\": {\n \"type\": \"Int32\",\n \"value\": {\n \"value\": \"@pipeline().parameters.StageId\",\n \"type\": \"Expression\"\n }\n }\n },\n \"queryTimeout\": \"02:00:00\"\n },\n \"dataset\": {\n \"referenceName\": \"GetSetMetadata\",\n \"type\": \"DatasetReference\",\n \"parameters\": {}\n },\n \"firstRowOnly\": false\n }\n },\n {\n \"name\": \"Execute Pipelines\",\n \"description\": \"Second level ForEach to run in parallel all pipelines within the stage. Items for iteration passed from the Get Pipelines lookup activity.\",\n \"type\": \"ForEach\",\n \"dependsOn\": [\n {\n \"activity\": \"Get Pipelines\",\n \"dependencyConditions\": [\n \"Succeeded\"\n ]\n }\n ],\n \"userProperties\": [],\n \"typeProperties\": {\n \"items\": {\n \"value\": \"@activity('Get Pipelines').output.value\",\n \"type\": \"Expression\"\n },\n \"isSequential\": false,\n \"batchCount\": 40,\n \"activities\": [\n {\n \"name\": \"Execute Pipeline\",\n \"description\": \"The lowest level executor with the metadata framework to call existing processing pipelines within Data Factory. The function called will block processing and wait for an outcome.\",\n \"type\": \"AzureFunctionActivity\",\n \"dependsOn\": [\n {\n \"activity\": \"Log Pipeline Running\",\n \"dependencyConditions\": [\n \"Succeeded\"\n ]\n },\n {\n \"activity\": \"Get Pipeline Params\",\n \"dependencyConditions\": [\n \"Succeeded\"\n ]\n },\n {\n \"activity\": \"Get SPN Details\",\n \"dependencyConditions\": [\n \"Succeeded\"\n ]\n }\n ],\n \"policy\": {\n \"timeout\": \"7.00:00:00\",\n \"retry\": 0,\n \"retryIntervalInSeconds\": 30,\n \"secureOutput\": false,\n \"secureInput\": false\n },\n \"userProperties\": [],\n \"typeProperties\": {\n \"functionName\": \"ExecutePipeline\",\n \"method\": \"POST\",\n \"headers\": {},\n \"body\": {\n \"value\": \"@concat('\\n{\\n\\t\\\"tenantId\\\": \\\"',pipeline().parameters.TenantId,'\\\",\\n\\t\\\"applicationId\\\": \\\"',activity('Get SPN Details').output.firstRow.Id,'\\\",\\n\\t\\\"authenticationKey\\\": \\\"',activity('Get SPN Details').output.firstRow.Secret,'\\\",\\n\\t\\\"subscriptionId\\\": \\\"',pipeline().parameters.SubscriptionId,'\\\",\\n\\t\\\"resourceGroup\\\": \\\"',item().ResourceGroupName,'\\\",\\n\\t\\\"factoryName\\\": \\\"',item().DataFactoryName,'\\\",\\n\\t\\\"pipelineName\\\": \\\"',item().PipelineName,'\\\"',activity('Get Pipeline Params').output.firstRow.Params,'\\n}')\",\n \"type\": \"Expression\"\n }\n },\n \"linkedServiceName\": {\n \"referenceName\": \"FrameworkFunctions\",\n \"type\": \"LinkedServiceReference\"\n }\n },\n {\n \"name\": \"Get Pipeline Params\",\n \"description\": \"Returns any parameters from metadata required for the processing pipeline being called. The output can be an empty string if no parameters are required.\",\n \"type\": \"Lookup\",\n \"dependsOn\": [],\n \"policy\": {\n \"timeout\": \"7.00:00:00\",\n \"retry\": 0,\n \"retryIntervalInSeconds\": 30,\n \"secureOutput\": false,\n \"secureInput\": false\n },\n \"userProperties\": [],\n \"typeProperties\": {\n \"source\": {\n \"type\": \"AzureSqlSource\",\n \"sqlReaderStoredProcedureName\": \"[[procfwk].[GetPipelineParameters]\",\n \"storedProcedureParameters\": {\n \"PipelineId\": {\n \"type\": \"Int32\",\n \"value\": {\n \"value\": \"@item().PipelineId\",\n \"type\": \"Expression\"\n }\n }\n },\n \"queryTimeout\": \"02:00:00\"\n },\n \"dataset\": {\n \"referenceName\": \"GetSetMetadata\",\n \"type\": \"DatasetReference\",\n \"parameters\": {}\n }\n }\n },\n {\n \"name\": \"Log Pipeline Running\",\n \"description\": \"Sets the current pipeline with a status of running within the current execution database table.\",\n \"type\": \"SqlServerStoredProcedure\",\n \"dependsOn\": [],\n \"policy\": {\n \"timeout\": \"7.00:00:00\",\n \"retry\": 0,\n \"retryIntervalInSeconds\": 30,\n \"secureOutput\": false,\n \"secureInput\": false\n },\n \"userProperties\": [],\n \"typeProperties\": {\n \"storedProcedureName\": \"[[procfwk].[SetLogPipelineRunning]\",\n \"storedProcedureParameters\": {\n \"ExecutionId\": {\n \"value\": {\n \"value\": \"@pipeline().parameters.ExecutionId\",\n \"type\": \"Expression\"\n },\n \"type\": \"Guid\"\n },\n \"PipelineId\": {\n \"value\": {\n \"value\": \"@item().PipelineId\",\n \"type\": \"Expression\"\n },\n \"type\": \"Int32\"\n },\n \"StageId\": {\n \"value\": {\n \"value\": \"@pipeline().parameters.StageId\",\n \"type\": \"Expression\"\n },\n \"type\": \"Int32\"\n }\n }\n },\n \"linkedServiceName\": {\n \"referenceName\": \"SupportDatabase\",\n \"type\": \"LinkedServiceReference\"\n }\n },\n {\n \"name\": \"Get SPN Details\",\n \"description\": \"Return the SPN ID and Secret for the processing pipeline being executed. Called at this level as each pipeline can have a different SPN.\",\n \"type\": \"Lookup\",\n \"dependsOn\": [],\n \"policy\": {\n \"timeout\": \"7.00:00:00\",\n \"retry\": 0,\n \"retryIntervalInSeconds\": 30,\n \"secureOutput\": false,\n \"secureInput\": false\n },\n \"userProperties\": [],\n \"typeProperties\": {\n \"source\": {\n \"type\": \"AzureSqlSource\",\n \"sqlReaderStoredProcedureName\": \"[[procfwk].[GetServicePrincipal]\",\n \"storedProcedureParameters\": {\n \"DataFactory\": {\n \"type\": \"String\",\n \"value\": {\n \"value\": \"@item().DataFactoryName\",\n \"type\": \"Expression\"\n }\n },\n \"PipelineName\": {\n \"type\": \"String\",\n \"value\": {\n \"value\": \"@item().PipelineName\",\n \"type\": \"Expression\"\n }\n }\n },\n \"queryTimeout\": \"02:00:00\"\n },\n \"dataset\": {\n \"referenceName\": \"GetSetMetadata\",\n \"type\": \"DatasetReference\",\n \"parameters\": {}\n }\n }\n },\n {\n \"name\": \"Log Activity Failure\",\n \"description\": \"Handle true failures from calling out to the Azure Function and update the current execution table accordingly so a restart can occur.\",\n \"type\": \"SqlServerStoredProcedure\",\n \"dependsOn\": [\n {\n \"activity\": \"Execute Pipeline\",\n \"dependencyConditions\": [\n \"Failed\"\n ]\n }\n ],\n \"policy\": {\n \"timeout\": \"7.00:00:00\",\n \"retry\": 0,\n \"retryIntervalInSeconds\": 30,\n \"secureOutput\": false,\n \"secureInput\": false\n },\n \"userProperties\": [],\n \"typeProperties\": {\n \"storedProcedureName\": \"[[procfwk].[SetLogActivityFailed]\",\n \"storedProcedureParameters\": {\n \"ExecutionId\": {\n \"value\": {\n \"value\": \"@pipeline().parameters.ExecutionId\",\n \"type\": \"Expression\"\n },\n \"type\": \"Guid\"\n },\n \"PipelineId\": {\n \"value\": {\n \"value\": \"@item().PipelineId\",\n \"type\": \"Expression\"\n },\n \"type\": \"Int32\"\n },\n \"StageId\": {\n \"value\": {\n \"value\": \"@pipeline().parameters.StageId\",\n \"type\": \"Expression\"\n },\n \"type\": \"Int32\"\n },\n \"CallingActivity\": {\n \"value\": \"ExecutePipeline\",\n \"type\": \"String\"\n }\n }\n },\n \"linkedServiceName\": {\n \"referenceName\": \"SupportDatabase\",\n \"type\": \"LinkedServiceReference\"\n }\n },\n {\n \"name\": \"Running Pipeline Handler\",\n \"type\": \"ExecutePipeline\",\n \"dependsOn\": [\n {\n \"activity\": \"Execute Pipeline\",\n \"dependencyConditions\": [\n \"Succeeded\"\n ]\n }\n ],\n \"userProperties\": [],\n \"typeProperties\": {\n \"pipeline\": {\n \"referenceName\": \"04-Infant\",\n \"type\": \"PipelineReference\"\n },\n \"waitOnCompletion\": true,\n \"parameters\": {\n \"tenantId\": {\n \"value\": \"@pipeline().parameters.TenantId\",\n \"type\": \"Expression\"\n },\n \"applicationId\": {\n \"value\": \"@activity('Get SPN Details').output.firstRow.Id\",\n \"type\": \"Expression\"\n },\n \"authenticationKey\": {\n \"value\": \"@activity('Get SPN Details').output.firstRow.Secret\",\n \"type\": \"Expression\"\n },\n \"subscriptionId\": {\n \"value\": \"@pipeline().parameters.SubscriptionId\",\n \"type\": \"Expression\"\n },\n \"resourceGroup\": {\n \"value\": \"@item().ResourceGroupName\",\n \"type\": \"Expression\"\n },\n \"factoryName\": {\n \"value\": \"@item().DataFactoryName\",\n \"type\": \"Expression\"\n },\n \"pipelineName\": {\n \"value\": \"@item().PipelineName\",\n \"type\": \"Expression\"\n },\n \"runId\": {\n \"value\": \"@activity('Execute Pipeline').output.RunId\",\n \"type\": \"Expression\"\n },\n \"executionId\": {\n \"value\": \"@pipeline().parameters.ExecutionId\",\n \"type\": \"Expression\"\n },\n \"stageId\": {\n \"value\": \"@pipeline().parameters.StageId\",\n \"type\": \"Expression\"\n },\n \"pipelineId\": {\n \"value\": \"@item().PipelineId\",\n \"type\": \"Expression\"\n }\n }\n }\n },\n {\n \"name\": \"Set Run Id\",\n \"description\": \"Provide the actual ADF run ID back to the current execution table for long term logging and alignment between the metadata other Azure monitoring tools.\",\n \"type\": \"SqlServerStoredProcedure\",\n \"dependsOn\": [\n {\n \"activity\": \"Execute Pipeline\",\n \"dependencyConditions\": [\n \"Succeeded\"\n ]\n }\n ],\n \"policy\": {\n \"timeout\": \"7.00:00:00\",\n \"retry\": 0,\n \"retryIntervalInSeconds\": 30,\n \"secureOutput\": false,\n \"secureInput\": false\n },\n \"userProperties\": [],\n \"typeProperties\": {\n \"storedProcedureName\": \"[[procfwk].[SetLogPipelineRunId]\",\n \"storedProcedureParameters\": {\n \"ExecutionId\": {\n \"value\": {\n \"value\": \"@pipeline().parameters.ExecutionId\",\n \"type\": \"Expression\"\n },\n \"type\": \"Guid\"\n },\n \"PipelineId\": {\n \"value\": {\n \"value\": \"@item().PipelineId\",\n \"type\": \"Expression\"\n },\n \"type\": \"Int32\"\n },\n \"RunId\": {\n \"value\": {\n \"value\": \"@activity('Execute Pipeline').output.RunId\",\n \"type\": \"Expression\"\n },\n \"type\": \"Guid\"\n },\n \"StageId\": {\n \"value\": {\n \"value\": \"@pipeline().parameters.StageId\",\n \"type\": \"Expression\"\n },\n \"type\": \"Int32\"\n }\n }\n },\n \"linkedServiceName\": {\n \"referenceName\": \"SupportDatabase\",\n \"type\": \"LinkedServiceReference\"\n }\n },\n {\n \"name\": \"Check For Alerts\",\n \"description\": \"Checks the properties tables and if any recipients in the database require alerts sending for the current pipeline ID.\",\n \"type\": \"Lookup\",\n \"dependsOn\": [],\n \"policy\": {\n \"timeout\": \"7.00:00:00\",\n \"retry\": 0,\n \"retryIntervalInSeconds\": 30,\n \"secureOutput\": false,\n \"secureInput\": false\n },\n \"userProperties\": [],\n \"typeProperties\": {\n \"source\": {\n \"type\": \"AzureSqlSource\",\n \"sqlReaderStoredProcedureName\": \"[[procfwk].[CheckForEmailAlerts]\",\n \"storedProcedureParameters\": {\n \"PipelineId\": {\n \"type\": \"Int32\",\n \"value\": {\n \"value\": \"@item().PipelineId\",\n \"type\": \"Expression\"\n }\n }\n },\n \"queryTimeout\": \"02:00:00\"\n },\n \"dataset\": {\n \"referenceName\": \"GetSetMetadata\",\n \"type\": \"DatasetReference\",\n \"parameters\": {}\n },\n \"firstRowOnly\": true\n }\n },\n {\n \"name\": \"Send Alerts\",\n \"description\": \"True = alerts need sending.\\nFalse = do nothing.\",\n \"type\": \"IfCondition\",\n \"dependsOn\": [\n {\n \"activity\": \"Set Run Id\",\n \"dependencyConditions\": [\n \"Succeeded\"\n ]\n },\n {\n \"activity\": \"Check For Alerts\",\n \"dependencyConditions\": [\n \"Succeeded\"\n ]\n },\n {\n \"activity\": \"Running Pipeline Handler\",\n \"dependencyConditions\": [\n \"Completed\"\n ]\n }\n ],\n \"userProperties\": [],\n \"typeProperties\": {\n \"expression\": {\n \"value\": \"@activity('Check For Alerts').output.firstRow.SendAlerts\",\n \"type\": \"Expression\"\n },\n \"ifTrueActivities\": [\n {\n \"name\": \"Get Email Parts\",\n \"type\": \"Lookup\",\n \"dependsOn\": [],\n \"policy\": {\n \"timeout\": \"7.00:00:00\",\n \"retry\": 0,\n \"retryIntervalInSeconds\": 30,\n \"secureOutput\": false,\n \"secureInput\": false\n },\n \"userProperties\": [],\n \"typeProperties\": {\n \"source\": {\n \"type\": \"AzureSqlSource\",\n \"sqlReaderStoredProcedureName\": \"[[procfwk].[GetEmailAlertParts]\",\n \"storedProcedureParameters\": {\n \"PipelineId\": {\n \"type\": \"Int32\",\n \"value\": {\n \"value\": \"@item().PipelineId\",\n \"type\": \"Expression\"\n }\n }\n },\n \"queryTimeout\": \"02:00:00\"\n },\n \"dataset\": {\n \"referenceName\": \"GetSetMetadata\",\n \"type\": \"DatasetReference\",\n \"parameters\": {}\n },\n \"firstRowOnly\": true\n }\n },\n {\n \"name\": \"Send Email\",\n \"type\": \"AzureFunctionActivity\",\n \"dependsOn\": [\n {\n \"activity\": \"Get Email Parts\",\n \"dependencyConditions\": [\n \"Succeeded\"\n ]\n }\n ],\n \"policy\": {\n \"timeout\": \"7.00:00:00\",\n \"retry\": 0,\n \"retryIntervalInSeconds\": 30,\n \"secureOutput\": false,\n \"secureInput\": false\n },\n \"userProperties\": [],\n \"typeProperties\": {\n \"functionName\": \"SendEmail\",\n \"method\": \"POST\",\n \"headers\": {},\n \"body\": {\n \"value\": \"@activity('Get Email Parts').output.firstRow\",\n \"type\": \"Expression\"\n }\n },\n \"linkedServiceName\": {\n \"referenceName\": \"FrameworkFunctions\",\n \"type\": \"LinkedServiceReference\"\n }\n }\n ]\n }\n }\n ]\n }\n }\n ],\n \"parameters\": {\n \"StageId\": {\n \"type\": \"int\"\n },\n \"ExecutionId\": {\n \"type\": \"string\"\n },\n \"TenantId\": {\n \"type\": \"string\"\n },\n \"SubscriptionId\": {\n \"type\": \"string\"\n }\n },\n \"variables\": {\n \"FunctionBody\": {\n \"type\": \"String\"\n }\n },\n \"folder\": {\n \"name\": \"_ProcFwk\"\n },\n \"annotations\": [\n \"ADF.procfwk\"\n ]\n },\n \"dependsOn\": [\n \"[concat(variables('factoryId'), '/datasets/GetSetMetadata')]\",\n \"[concat(variables('factoryId'), '/linkedServices/FrameworkFunctions')]\",\n \"[concat(variables('factoryId'), '/linkedServices/SupportDatabase')]\",\n \"[concat(variables('factoryId'), '/pipelines/04-Infant')]\"\n ]\n },\n {\n \"name\": \"[concat(parameters('factoryName'), '/04-Infant')]\",\n \"type\": \"Microsoft.DataFactory/factories/pipelines\",\n \"apiVersion\": \"2018-06-01\",\n \"properties\": {\n \"description\": \"ADF.procfwk infant pipeline used to check when the processing pipeline called by the Child completes and passes the resulting status back to the metadata database.\",\n \"activities\": [\n {\n \"name\": \"Wait Until Pipeline Completes\",\n \"description\": \"Loops until the pipeline called completes.\\n\\nSimple status:\\n- Running = new iteration.\\n- Done = break.\",\n \"type\": \"Until\",\n \"dependsOn\": [\n {\n \"activity\": \"Get Wait Duration\",\n \"dependencyConditions\": [\n \"Succeeded\"\n ]\n }\n ],\n \"userProperties\": [],\n \"typeProperties\": {\n \"expression\": {\n \"value\": \"@equals('Done',activity('Get Pipeline Status').output.SimpleStatus)\",\n \"type\": \"Expression\"\n },\n \"activities\": [\n {\n \"name\": \"Get Pipeline Status\",\n \"description\": \"Checks the status of a given processing pipeline and provides the value for the downstream framework activities to act upon.\",\n \"type\": \"AzureFunctionActivity\",\n \"dependsOn\": [],\n \"policy\": {\n \"timeout\": \"7.00:00:00\",\n \"retry\": 0,\n \"retryIntervalInSeconds\": 30,\n \"secureOutput\": false,\n \"secureInput\": false\n },\n \"userProperties\": [],\n \"typeProperties\": {\n \"functionName\": \"CheckPipelineStatus\",\n \"method\": \"POST\",\n \"headers\": {},\n \"body\": {\n \"value\": \"@concat('\\n{\\n \\\"tenantId\\\": \\\"',pipeline().parameters.TenantId,'\\\",\\n \\\"applicationId\\\": \\\"',pipeline().parameters.applicationId,'\\\",\\n \\\"authenticationKey\\\": \\\"',pipeline().parameters.authenticationKey,'\\\",\\n \\\"subscriptionId\\\": \\\"',pipeline().parameters.subscriptionId,'\\\",\\n \\\"resourceGroup\\\": \\\"',pipeline().parameters.resourceGroup,'\\\",\\n \\\"factoryName\\\": \\\"',pipeline().parameters.factoryName,'\\\",\\n \\\"pipelineName\\\": \\\"',pipeline().parameters.pipelineName,'\\\",\\n \\\"runId\\\": \\\"',pipeline().parameters.runId,'\\\"\\n}')\",\n \"type\": \"Expression\"\n }\n },\n \"linkedServiceName\": {\n \"referenceName\": \"FrameworkFunctions\",\n \"type\": \"LinkedServiceReference\"\n }\n },\n {\n \"name\": \"Wait If Running\",\n \"description\": \"True = Do nothing.\\nFalse = Wait.\",\n \"type\": \"IfCondition\",\n \"dependsOn\": [\n {\n \"activity\": \"Get Pipeline Status\",\n \"dependencyConditions\": [\n \"Succeeded\"\n ]\n }\n ],\n \"userProperties\": [],\n \"typeProperties\": {\n \"expression\": {\n \"value\": \"@equals('Done',activity('Get Pipeline Status').output.SimpleStatus)\",\n \"type\": \"Expression\"\n },\n \"ifFalseActivities\": [\n {\n \"name\": \"Wait for Pipeline\",\n \"description\": \"The processing pipeline is still running so Wait before checking its status again.\",\n \"type\": \"Wait\",\n \"dependsOn\": [],\n \"userProperties\": [],\n \"typeProperties\": {\n \"waitTimeInSeconds\": {\n \"value\": \"@activity('Get Wait Duration').output.firstRow.PropertyValue\",\n \"type\": \"Expression\"\n }\n }\n }\n ]\n }\n },\n {\n \"name\": \"Set Last Check DateTime\",\n \"type\": \"SqlServerStoredProcedure\",\n \"dependsOn\": [\n {\n \"activity\": \"Get Pipeline Status\",\n \"dependencyConditions\": [\n \"Succeeded\"\n ]\n }\n ],\n \"policy\": {\n \"timeout\": \"7.00:00:00\",\n \"retry\": 0,\n \"retryIntervalInSeconds\": 30,\n \"secureOutput\": false,\n \"secureInput\": false\n },\n \"userProperties\": [],\n \"typeProperties\": {\n \"storedProcedureName\": \"[[procfwk].[SetLogPipelineLastStatusCheck]\",\n \"storedProcedureParameters\": {\n \"ExecutionId\": {\n \"value\": {\n \"value\": \"@pipeline().parameters.executionId\",\n \"type\": \"Expression\"\n },\n \"type\": \"Guid\"\n },\n \"PipelineId\": {\n \"value\": {\n \"value\": \"@pipeline().parameters.pipelineId\",\n \"type\": \"Expression\"\n },\n \"type\": \"Int32\"\n },\n \"StageId\": {\n \"value\": {\n \"value\": \"@pipeline().parameters.stageId\",\n \"type\": \"Expression\"\n },\n \"type\": \"Int32\"\n }\n }\n },\n \"linkedServiceName\": {\n \"referenceName\": \"SupportDatabase\",\n \"type\": \"LinkedServiceReference\"\n }\n },\n {\n \"name\": \"Log Activity Failure\",\n \"type\": \"SqlServerStoredProcedure\",\n \"dependsOn\": [\n {\n \"activity\": \"Get Pipeline Status\",\n \"dependencyConditions\": [\n \"Failed\"\n ]\n }\n ],\n \"policy\": {\n \"timeout\": \"7.00:00:00\",\n \"retry\": 0,\n \"retryIntervalInSeconds\": 30,\n \"secureOutput\": false,\n \"secureInput\": false\n },\n \"userProperties\": [],\n \"typeProperties\": {\n \"storedProcedureName\": \"[[procfwk].[SetLogActivityFailed]\",\n \"storedProcedureParameters\": {\n \"CallingActivity\": {\n \"value\": \"GetPipelineStatus\",\n \"type\": \"String\"\n },\n \"ExecutionId\": {\n \"value\": {\n \"value\": \"@pipeline().parameters.executionId\",\n \"type\": \"Expression\"\n },\n \"type\": \"Guid\"\n },\n \"PipelineId\": {\n \"value\": {\n \"value\": \"@pipeline().parameters.pipelineId\",\n \"type\": \"Expression\"\n },\n \"type\": \"Int32\"\n },\n \"StageId\": {\n \"value\": {\n \"value\": \"@pipeline().parameters.stageId\",\n \"type\": \"Expression\"\n },\n \"type\": \"Int32\"\n }\n }\n },\n \"linkedServiceName\": {\n \"referenceName\": \"SupportDatabase\",\n \"type\": \"LinkedServiceReference\"\n }\n }\n ],\n \"timeout\": \"7.00:00:00\"\n }\n },\n {\n \"name\": \"Set Pipeline Result\",\n \"description\": \"Receives the outcome from the function execution for a given processing pipeline and updates the current execution table with different pipelines status values depending on the result (case).\",\n \"type\": \"Switch\",\n \"dependsOn\": [\n {\n \"activity\": \"Wait Until Pipeline Completes\",\n \"dependencyConditions\": [\n \"Completed\"\n ]\n }\n ],\n \"userProperties\": [],\n \"typeProperties\": {\n \"on\": {\n \"value\": \"@activity('Get Pipeline Status').output.Status\",\n \"type\": \"Expression\"\n },\n \"cases\": [\n {\n \"value\": \"Succeeded\",\n \"activities\": [\n {\n \"name\": \"Pipeline Status Succeeded\",\n \"description\": \"Updates the current execution table with a pipeline status of success if the function outcome is succeeded.\",\n \"type\": \"SqlServerStoredProcedure\",\n \"dependsOn\": [],\n \"policy\": {\n \"timeout\": \"7.00:00:00\",\n \"retry\": 0,\n \"retryIntervalInSeconds\": 30,\n \"secureOutput\": false,\n \"secureInput\": false\n },\n \"userProperties\": [],\n \"typeProperties\": {\n \"storedProcedureName\": \"[[procfwk].[SetLogPipelineSuccess]\",\n \"storedProcedureParameters\": {\n \"ExecutionId\": {\n \"value\": {\n \"value\": \"@pipeline().parameters.executionId\",\n \"type\": \"Expression\"\n },\n \"type\": \"Guid\"\n },\n \"PipelineId\": {\n \"value\": {\n \"value\": \"@pipeline().parameters.pipelineId\",\n \"type\": \"Expression\"\n },\n \"type\": \"Int32\"\n },\n \"StageId\": {\n \"value\": {\n \"value\": \"@pipeline().parameters.stageId\",\n \"type\": \"Expression\"\n },\n \"type\": \"Int32\"\n }\n }\n },\n \"linkedServiceName\": {\n \"referenceName\": \"SupportDatabase\",\n \"type\": \"LinkedServiceReference\"\n }\n }\n ]\n },\n {\n \"value\": \"Failed\",\n \"activities\": [\n {\n \"name\": \"Pipeline Status Failed\",\n \"description\": \"Updates the current execution table with a pipeline status of failed if the function outcome is failed. Also blocks pipelines in the downstream execution stage.\",\n \"type\": \"SqlServerStoredProcedure\",\n \"dependsOn\": [],\n \"policy\": {\n \"timeout\": \"7.00:00:00\",\n \"retry\": 0,\n \"retryIntervalInSeconds\": 30,\n \"secureOutput\": false,\n \"secureInput\": false\n },\n \"userProperties\": [],\n \"typeProperties\": {\n \"storedProcedureName\": \"[[procfwk].[SetLogPipelineFailed]\",\n \"storedProcedureParameters\": {\n \"ExecutionId\": {\n \"value\": {\n \"value\": \"@pipeline().parameters.executionId\",\n \"type\": \"Expression\"\n },\n \"type\": \"Guid\"\n },\n \"PipelineId\": {\n \"value\": {\n \"value\": \"@pipeline().parameters.pipelineId\",\n \"type\": \"Expression\"\n },\n \"type\": \"Int32\"\n },\n \"RunId\": {\n \"value\": {\n \"value\": \"@activity('Get Pipeline Status').output.RunId\",\n \"type\": \"Expression\"\n },\n \"type\": \"Guid\"\n },\n \"StageId\": {\n \"value\": {\n \"value\": \"@pipeline().parameters.stageId\",\n \"type\": \"Expression\"\n },\n \"type\": \"Int32\"\n }\n }\n },\n \"linkedServiceName\": {\n \"referenceName\": \"SupportDatabase\",\n \"type\": \"LinkedServiceReference\"\n }\n },\n {\n \"name\": \"Get Error Details\",\n \"description\": \"Get the activity error details for the run ID of the worker pipeline called. Returns an array of all errors.\",\n \"type\": \"AzureFunctionActivity\",\n \"dependsOn\": [],\n \"policy\": {\n \"timeout\": \"7.00:00:00\",\n \"retry\": 0,\n \"retryIntervalInSeconds\": 30,\n \"secureOutput\": false,\n \"secureInput\": false\n },\n \"userProperties\": [],\n \"typeProperties\": {\n \"functionName\": \"GetActivityErrors\",\n \"method\": \"POST\",\n \"headers\": {},\n \"body\": {\n \"value\": \"@concat('\\n{\\n \\\"tenantId\\\": \\\"',pipeline().parameters.TenantId,'\\\",\\n \\\"applicationId\\\": \\\"',pipeline().parameters.applicationId,'\\\",\\n \\\"authenticationKey\\\": \\\"',pipeline().parameters.authenticationKey,'\\\",\\n \\\"subscriptionId\\\": \\\"',pipeline().parameters.subscriptionId,'\\\",\\n \\\"resourceGroup\\\": \\\"',pipeline().parameters.resourceGroup,'\\\",\\n \\\"factoryName\\\": \\\"',pipeline().parameters.factoryName,'\\\",\\n \\\"pipelineName\\\": \\\"',pipeline().parameters.pipelineName,'\\\",\\n \\\"runId\\\": \\\"',pipeline().parameters.runId,'\\\"\\n}')\",\n \"type\": \"Expression\"\n }\n },\n \"linkedServiceName\": {\n \"referenceName\": \"FrameworkFunctions\",\n \"type\": \"LinkedServiceReference\"\n }\n },\n {\n \"name\": \"Log Error Details\",\n \"description\": \"Parses pipeline error details and persists them to the metadata database error log table.\",\n \"type\": \"SqlServerStoredProcedure\",\n \"dependsOn\": [\n {\n \"activity\": \"Get Error Details\",\n \"dependencyConditions\": [\n \"Succeeded\"\n ]\n }\n ],\n \"policy\": {\n \"timeout\": \"7.00:00:00\",\n \"retry\": 0,\n \"retryIntervalInSeconds\": 30,\n \"secureOutput\": false,\n \"secureInput\": false\n },\n \"userProperties\": [],\n \"typeProperties\": {\n \"storedProcedureName\": \"[[procfwk].[SetErrorLogDetails]\",\n \"storedProcedureParameters\": {\n \"JsonErrorDetails\": {\n \"value\": {\n \"value\": \"@string(activity('Get Error Details').output)\",\n \"type\": \"Expression\"\n },\n \"type\": \"String\"\n },\n \"LocalExecutionId\": {\n \"value\": {\n \"value\": \"@pipeline().parameters.executionId\",\n \"type\": \"Expression\"\n },\n \"type\": \"Guid\"\n }\n }\n },\n \"linkedServiceName\": {\n \"referenceName\": \"SupportDatabase\",\n \"type\": \"LinkedServiceReference\"\n }\n }\n ]\n }\n ],\n \"defaultActivities\": [\n {\n \"name\": \"Pipeline Status Unknown\",\n \"description\": \"Updates the current execution table with a pipeline status of unknown if the function returns an unexpected outcome.\",\n \"type\": \"SqlServerStoredProcedure\",\n \"dependsOn\": [],\n \"policy\": {\n \"timeout\": \"7.00:00:00\",\n \"retry\": 0,\n \"retryIntervalInSeconds\": 30,\n \"secureOutput\": false,\n \"secureInput\": false\n },\n \"userProperties\": [],\n \"typeProperties\": {\n \"storedProcedureName\": \"[[procfwk].[SetLogPipelineUnknown]\",\n \"storedProcedureParameters\": {\n \"ExecutionId\": {\n \"value\": {\n \"value\": \"@pipeline().parameters.executionId\",\n \"type\": \"Expression\"\n },\n \"type\": \"Guid\"\n },\n \"PipelineId\": {\n \"value\": {\n \"value\": \"@pipeline().parameters.pipelineId\",\n \"type\": \"Expression\"\n },\n \"type\": \"Int32\"\n },\n \"StageId\": {\n \"value\": {\n \"value\": \"@pipeline().parameters.stageId\",\n \"type\": \"Expression\"\n },\n \"type\": \"Int32\"\n }\n }\n },\n \"linkedServiceName\": {\n \"referenceName\": \"SupportDatabase\",\n \"type\": \"LinkedServiceReference\"\n }\n }\n ]\n }\n },\n {\n \"name\": \"Get Wait Duration\",\n \"type\": \"Lookup\",\n \"dependsOn\": [],\n \"policy\": {\n \"timeout\": \"7.00:00:00\",\n \"retry\": 0,\n \"retryIntervalInSeconds\": 30,\n \"secureOutput\": false,\n \"secureInput\": false\n },\n \"userProperties\": [],\n \"typeProperties\": {\n \"source\": {\n \"type\": \"AzureSqlSource\",\n \"sqlReaderStoredProcedureName\": \"[[procfwk].[GetPropertyValue]\",\n \"storedProcedureParameters\": {\n \"PropertyName\": {\n \"type\": \"String\",\n \"value\": \"PipelineStatusCheckDuration\"\n }\n },\n \"queryTimeout\": \"02:00:00\"\n },\n \"dataset\": {\n \"referenceName\": \"GetSetMetadata\",\n \"type\": \"DatasetReference\",\n \"parameters\": {}\n }\n }\n }\n ],\n \"parameters\": {\n \"tenantId\": {\n \"type\": \"string\"\n },\n \"applicationId\": {\n \"type\": \"string\"\n },\n \"authenticationKey\": {\n \"type\": \"string\"\n },\n \"subscriptionId\": {\n \"type\": \"string\"\n },\n \"resourceGroup\": {\n \"type\": \"string\"\n },\n \"factoryName\": {\n \"type\": \"string\"\n },\n \"pipelineName\": {\n \"type\": \"string\"\n },\n \"runId\": {\n \"type\": \"string\"\n },\n \"executionId\": {\n \"type\": \"string\"\n },\n \"stageId\": {\n \"type\": \"int\"\n },\n \"pipelineId\": {\n \"type\": \"int\"\n }\n },\n \"folder\": {\n \"name\": \"_ProcFwk\"\n },\n \"annotations\": [\n \"ADF.procfwk\"\n ]\n },\n \"dependsOn\": [\n \"[concat(variables('factoryId'), '/datasets/GetSetMetadata')]\",\n \"[concat(variables('factoryId'), '/linkedServices/FrameworkFunctions')]\",\n \"[concat(variables('factoryId'), '/linkedServices/SupportDatabase')]\"\n ]\n },\n {\n \"name\": \"[concat(parameters('factoryName'), '/Intentional Error')]\",\n \"type\": \"Microsoft.DataFactory/factories/pipelines\",\n \"apiVersion\": \"2018-06-01\",\n \"properties\": {\n \"description\": \"Used just so the ADF.procfwk has something to call during development.\",\n \"activities\": [\n {\n \"name\": \"Wait1\",\n \"type\": \"Wait\",\n \"dependsOn\": [],\n \"userProperties\": [],\n \"typeProperties\": {\n \"waitTimeInSeconds\": {\n \"value\": \"@pipeline().parameters.WaitTime\",\n \"type\": \"Expression\"\n }\n }\n },\n {\n \"name\": \"Raise Errors or Not\",\n \"type\": \"IfCondition\",\n \"dependsOn\": [\n {\n \"activity\": \"Wait1\",\n \"dependencyConditions\": [\n \"Succeeded\"\n ]\n }\n ],\n \"userProperties\": [],\n \"typeProperties\": {\n \"expression\": {\n \"value\": \"@equals(pipeline().parameters.RaiseErrors,'true')\",\n \"type\": \"Expression\"\n },\n \"ifTrueActivities\": [\n {\n \"name\": \"Call Fail Procedure\",\n \"type\": \"SqlServerStoredProcedure\",\n \"dependsOn\": [],\n \"policy\": {\n \"timeout\": \"7.00:00:00\",\n \"retry\": 0,\n \"retryIntervalInSeconds\": 30,\n \"secureOutput\": false,\n \"secureInput\": false\n },\n \"userProperties\": [],\n \"typeProperties\": {\n \"storedProcedureName\": \"[[dbo].[FailProcedure]\",\n \"storedProcedureParameters\": {\n \"RaiseError\": {\n \"value\": {\n \"value\": \"@pipeline().parameters.RaiseErrors\",\n \"type\": \"Expression\"\n },\n \"type\": \"String\"\n }\n }\n },\n \"linkedServiceName\": {\n \"referenceName\": \"SupportDatabase\",\n \"type\": \"LinkedServiceReference\"\n }\n }\n ]\n }\n }\n ],\n \"parameters\": {\n \"RaiseErrors\": {\n \"type\": \"string\",\n \"defaultValue\": \"false\"\n },\n \"WaitTime\": {\n \"type\": \"int\",\n \"defaultValue\": 5\n }\n },\n \"folder\": {\n \"name\": \"_Workers\"\n },\n \"annotations\": [\n \"Worker\"\n ]\n },\n \"dependsOn\": [\n \"[concat(variables('factoryId'), '/linkedServices/SupportDatabase')]\"\n ]\n },\n {\n \"name\": \"[concat(parameters('factoryName'), '/Wait 1')]\",\n \"type\": \"Microsoft.DataFactory/factories/pipelines\",\n \"apiVersion\": \"2018-06-01\",\n \"properties\": {\n \"description\": \"Used just so the ADF.procfwk has something to call during development.\",\n \"activities\": [\n {\n \"name\": \"Wait1\",\n \"type\": \"Wait\",\n \"dependsOn\": [],\n \"userProperties\": [],\n \"typeProperties\": {\n \"waitTimeInSeconds\": {\n \"value\": \"@pipeline().parameters.WaitTime\",\n \"type\": \"Expression\"\n }\n }\n }\n ],\n \"parameters\": {\n \"WaitTime\": {\n \"type\": \"int\",\n \"defaultValue\": 5\n }\n },\n \"folder\": {\n \"name\": \"_Workers\"\n },\n \"annotations\": [\n \"Worker\"\n ]\n },\n \"dependsOn\": []\n },\n {\n \"name\": \"[concat(parameters('factoryName'), '/Wait 10')]\",\n \"type\": \"Microsoft.DataFactory/factories/pipelines\",\n \"apiVersion\": \"2018-06-01\",\n \"properties\": {\n \"description\": \"Used just so the ADF.procfwk has something to call during development.\",\n \"activities\": [\n {\n \"name\": \"Wait10\",\n \"type\": \"Wait\",\n \"dependsOn\": [],\n \"userProperties\": [],\n \"typeProperties\": {\n \"waitTimeInSeconds\": {\n \"value\": \"@pipeline().parameters.WaitTime\",\n \"type\": \"Expression\"\n }\n }\n }\n ],\n \"parameters\": {\n \"WaitTime\": {\n \"type\": \"int\",\n \"defaultValue\": 5\n }\n },\n \"folder\": {\n \"name\": \"_Workers\"\n },\n \"annotations\": [\n \"Worker\"\n ]\n },\n \"dependsOn\": []\n },\n {\n \"name\": \"[concat(parameters('factoryName'), '/Wait 2')]\",\n \"type\": \"Microsoft.DataFactory/factories/pipelines\",\n \"apiVersion\": \"2018-06-01\",\n \"properties\": {\n \"description\": \"Used just so the ADF.procfwk has something to call during development.\",\n \"activities\": [\n {\n \"name\": \"Wait2\",\n \"type\": \"Wait\",\n \"dependsOn\": [],\n \"userProperties\": [],\n \"typeProperties\": {\n \"waitTimeInSeconds\": {\n \"value\": \"@pipeline().parameters.WaitTime\",\n \"type\": \"Expression\"\n }\n }\n }\n ],\n \"parameters\": {\n \"WaitTime\": {\n \"type\": \"int\",\n \"defaultValue\": 5\n }\n },\n \"folder\": {\n \"name\": \"_Workers\"\n },\n \"annotations\": [\n \"Worker\"\n ]\n },\n \"dependsOn\": []\n },\n {\n \"name\": \"[concat(parameters('factoryName'), '/Wait 3')]\",\n \"type\": \"Microsoft.DataFactory/factories/pipelines\",\n \"apiVersion\": \"2018-06-01\",\n \"properties\": {\n \"description\": \"Used just so the ADF.procfwk has something to call during development.\",\n \"activities\": [\n {\n \"name\": \"Wait3\",\n \"type\": \"Wait\",\n \"dependsOn\": [],\n \"userProperties\": [],\n \"typeProperties\": {\n \"waitTimeInSeconds\": {\n \"value\": \"@pipeline().parameters.WaitTime\",\n \"type\": \"Expression\"\n }\n }\n }\n ],\n \"parameters\": {\n \"WaitTime\": {\n \"type\": \"int\",\n \"defaultValue\": 5\n }\n },\n \"folder\": {\n \"name\": \"_Workers\"\n },\n \"annotations\": [\n \"Worker\"\n ]\n },\n \"dependsOn\": []\n },\n {\n \"name\": \"[concat(parameters('factoryName'), '/Wait 4')]\",\n \"type\": \"Microsoft.DataFactory/factories/pipelines\",\n \"apiVersion\": \"2018-06-01\",\n \"properties\": {\n \"description\": \"Used just so the ADF.procfwk has something to call during development.\",\n \"activities\": [\n {\n \"name\": \"Wait4\",\n \"type\": \"Wait\",\n \"dependsOn\": [],\n \"userProperties\": [],\n \"typeProperties\": {\n \"waitTimeInSeconds\": {\n \"value\": \"@pipeline().parameters.WaitTime\",\n \"type\": \"Expression\"\n }\n }\n }\n ],\n \"parameters\": {\n \"WaitTime\": {\n \"type\": \"int\",\n \"defaultValue\": 5\n }\n },\n \"folder\": {\n \"name\": \"_Workers\"\n },\n \"annotations\": [\n \"Worker\"\n ]\n },\n \"dependsOn\": []\n },\n {\n \"name\": \"[concat(parameters('factoryName'), '/Wait 5')]\",\n \"type\": \"Microsoft.DataFactory/factories/pipelines\",\n \"apiVersion\": \"2018-06-01\",\n \"properties\": {\n \"description\": \"Used just so the ADF.procfwk has something to call during development.\",\n \"activities\": [\n {\n \"name\": \"Wait5\",\n \"type\": \"Wait\",\n \"dependsOn\": [],\n \"userProperties\": [],\n \"typeProperties\": {\n \"waitTimeInSeconds\": {\n \"value\": \"@pipeline().parameters.WaitTime\",\n \"type\": \"Expression\"\n }\n }\n }\n ],\n \"parameters\": {\n \"WaitTime\": {\n \"type\": \"int\",\n \"defaultValue\": 5\n }\n },\n \"folder\": {\n \"name\": \"_Workers\"\n },\n \"annotations\": [\n \"Worker\"\n ]\n },\n \"dependsOn\": []\n },\n {\n \"name\": \"[concat(parameters('factoryName'), '/Wait 6')]\",\n \"type\": \"Microsoft.DataFactory/factories/pipelines\",\n \"apiVersion\": \"2018-06-01\",\n \"properties\": {\n \"description\": \"Used just so the ADF.procfwk has something to call during development.\",\n \"activities\": [\n {\n \"name\": \"Wait6\",\n \"type\": \"Wait\",\n \"dependsOn\": [],\n \"userProperties\": [],\n \"typeProperties\": {\n \"waitTimeInSeconds\": {\n \"value\": \"@pipeline().parameters.WaitTime\",\n \"type\": \"Expression\"\n }\n }\n }\n ],\n \"parameters\": {\n \"WaitTime\": {\n \"type\": \"int\",\n \"defaultValue\": 5\n }\n },\n \"folder\": {\n \"name\": \"_Workers\"\n },\n \"annotations\": [\n \"Worker\"\n ]\n },\n \"dependsOn\": []\n },\n {\n \"name\": \"[concat(parameters('factoryName'), '/Wait 7')]\",\n \"type\": \"Microsoft.DataFactory/factories/pipelines\",\n \"apiVersion\": \"2018-06-01\",\n \"properties\": {\n \"description\": \"Used just so the ADF.procfwk has something to call during development.\",\n \"activities\": [\n {\n \"name\": \"Wait7\",\n \"type\": \"Wait\",\n \"dependsOn\": [],\n \"userProperties\": [],\n \"typeProperties\": {\n \"waitTimeInSeconds\": {\n \"value\": \"@pipeline().parameters.WaitTime\",\n \"type\": \"Expression\"\n }\n }\n }\n ],\n \"parameters\": {\n \"WaitTime\": {\n \"type\": \"int\",\n \"defaultValue\": 5\n }\n },\n \"folder\": {\n \"name\": \"_Workers\"\n },\n \"annotations\": [\n \"Worker\"\n ]\n },\n \"dependsOn\": []\n },\n {\n \"name\": \"[concat(parameters('factoryName'), '/Wait 8')]\",\n \"type\": \"Microsoft.DataFactory/factories/pipelines\",\n \"apiVersion\": \"2018-06-01\",\n \"properties\": {\n \"description\": \"Used just so the ADF.procfwk has something to call during development.\",\n \"activities\": [\n {\n \"name\": \"Wait8\",\n \"type\": \"Wait\",\n \"dependsOn\": [],\n \"userProperties\": [],\n \"typeProperties\": {\n \"waitTimeInSeconds\": {\n \"value\": \"@pipeline().parameters.WaitTime\",\n \"type\": \"Expression\"\n }\n }\n }\n ],\n \"parameters\": {\n \"WaitTime\": {\n \"type\": \"int\"\n }\n },\n \"folder\": {\n \"name\": \"_Workers\"\n },\n \"annotations\": [\n \"Worker\"\n ]\n },\n \"dependsOn\": []\n },\n {\n \"name\": \"[concat(parameters('factoryName'), '/Wait 9')]\",\n \"type\": \"Microsoft.DataFactory/factories/pipelines\",\n \"apiVersion\": \"2018-06-01\",\n \"properties\": {\n \"description\": \"Used just so the ADF.procfwk has something to call during development.\",\n \"activities\": [\n {\n \"name\": \"Wait9\",\n \"type\": \"Wait\",\n \"dependsOn\": [],\n \"userProperties\": [],\n \"typeProperties\": {\n \"waitTimeInSeconds\": {\n \"value\": \"@pipeline().parameters.WaitTime\",\n \"type\": \"Expression\"\n }\n }\n }\n ],\n \"parameters\": {\n \"WaitTime\": {\n \"type\": \"int\",\n \"defaultValue\": 15\n }\n },\n \"folder\": {\n \"name\": \"_Workers\"\n },\n \"annotations\": [\n \"Worker\"\n ]\n },\n \"dependsOn\": []\n },\n {\n \"name\": \"[concat(parameters('factoryName'), '/GetSetMetadata')]\",\n \"type\": \"Microsoft.DataFactory/factories/datasets\",\n \"apiVersion\": \"2018-06-01\",\n \"properties\": {\n \"linkedServiceName\": {\n \"referenceName\": \"SupportDatabase\",\n \"type\": \"LinkedServiceReference\"\n },\n \"annotations\": [\n \"ADF.procfwk\"\n ],\n \"type\": \"AzureSqlTable\",\n \"schema\": [],\n \"typeProperties\": {}\n },\n \"dependsOn\": [\n \"[concat(variables('factoryId'), '/linkedServices/SupportDatabase')]\"\n ]\n },\n {\n \"name\": \"[concat(parameters('factoryName'), '/FrameworkFunctions')]\",\n \"type\": \"Microsoft.DataFactory/factories/linkedServices\",\n \"apiVersion\": \"2018-06-01\",\n \"properties\": {\n \"annotations\": [\n \"ADF.procfwk\"\n ],\n \"type\": \"AzureFunction\",\n \"typeProperties\": {\n \"functionAppUrl\": \"[parameters('FrameworkFunctions_properties_typeProperties_functionAppUrl')]\",\n \"functionKey\": {\n \"type\": \"AzureKeyVaultSecret\",\n \"store\": {\n \"referenceName\": \"Keys\",\n \"type\": \"LinkedServiceReference\"\n },\n \"secretName\": \"FrameworkFunctionsKey\"\n }\n }\n },\n \"dependsOn\": [\n \"[concat(variables('factoryId'), '/linkedServices/Keys')]\"\n ]\n },\n {\n \"name\": \"[concat(parameters('factoryName'), '/Keys')]\",\n \"type\": \"Microsoft.DataFactory/factories/linkedServices\",\n \"apiVersion\": \"2018-06-01\",\n \"properties\": {\n \"description\": \"Connection to Key Vault for all other ADF linked service credentials required to run the processing framework.\",\n \"annotations\": [\n \"ADF.procfwk\"\n ],\n \"type\": \"AzureKeyVault\",\n \"typeProperties\": {\n \"baseUrl\": \"[parameters('Keys_properties_typeProperties_baseUrl')]\"\n }\n },\n \"dependsOn\": []\n },\n {\n \"name\": \"[concat(parameters('factoryName'), '/SupportDatabase')]\",\n \"type\": \"Microsoft.DataFactory/factories/linkedServices\",\n \"apiVersion\": \"2018-06-01\",\n \"properties\": {\n \"description\": \"Connection between ADF and processing framework metadata SQLDB.\",\n \"annotations\": [\n \"ADF.procfwk\"\n ],\n \"type\": \"AzureSqlDatabase\",\n \"typeProperties\": {\n \"connectionString\": {\n \"type\": \"AzureKeyVaultSecret\",\n \"store\": {\n \"referenceName\": \"Keys\",\n \"type\": \"LinkedServiceReference\"\n },\n \"secretName\": \"[parameters('SupportDatabase_properties_typeProperties_connectionString_secretName')]\"\n }\n }\n },\n \"dependsOn\": [\n \"[concat(variables('factoryId'), '/linkedServices/Keys')]\"\n ]\n },\n {\n \"name\": \"[concat(parameters('factoryName'), '/FunctionalTestingTrigger')]\",\n \"type\": \"Microsoft.DataFactory/factories/triggers\",\n \"apiVersion\": \"2018-06-01\",\n \"properties\": {\n \"description\": \"Used for functional testing of the framework in a dedicated environment.\",\n \"annotations\": [\n \"ADF.procfwk\"\n ],\n \"runtimeState\": \"Stopped\",\n \"pipelines\": [\n {\n \"pipelineReference\": {\n \"referenceName\": \"01-Grandparent\",\n \"type\": \"PipelineReference\"\n },\n \"parameters\": {}\n }\n ],\n \"type\": \"ScheduleTrigger\",\n \"typeProperties\": {\n \"recurrence\": {\n \"frequency\": \"Hour\",\n \"interval\": 2,\n \"startTime\": \"2020-04-06T15:00:00.000Z\",\n \"timeZone\": \"UTC\"\n }\n }\n },\n \"dependsOn\": [\n \"[concat(variables('factoryId'), '/pipelines/01-Grandparent')]\"\n ]\n }\n ]\n}"
},
{
"path": "ARM Templates/Data Factory/v1.8.3 Export.json",
"content": "{\n \"$schema\": \"http://schema.management.azure.com/schemas/2015-01-01/deploymentTemplate.json#\",\n \"contentVersion\": \"1.0.0.0\",\n \"parameters\": {\n \"factoryName\": {\n \"type\": \"string\",\n \"metadata\": \"Data Factory name\",\n \"defaultValue\": \"FrameworkFactory\"\n },\n \"FrameworkFunctions_properties_typeProperties_functionAppUrl\": {\n \"type\": \"string\",\n \"defaultValue\": \"\"\n },\n \"Keys_properties_typeProperties_baseUrl\": {\n \"type\": \"string\",\n \"defaultValue\": \"\"\n },\n \"SupportDatabase_properties_typeProperties_connectionString_secretName\": {\n \"type\": \"string\",\n \"defaultValue\": \"\"\n }\n },\n \"variables\": {\n \"factoryId\": \"[concat('Microsoft.DataFactory/factories/', parameters('factoryName'))]\"\n },\n \"resources\": [\n {\n \"name\": \"[concat(parameters('factoryName'), '/01-Grandparent')]\",\n \"type\": \"Microsoft.DataFactory/factories/pipelines\",\n \"apiVersion\": \"2018-06-01\",\n \"properties\": {\n \"description\": \"ADF.procfwk grandparent pipeline used optionally to bootstrap any wider processes in your Data Factory that then calls the processing framework.\",\n \"activities\": [\n {\n \"name\": \"Framework Processing\",\n \"type\": \"ExecutePipeline\",\n \"dependsOn\": [\n {\n \"activity\": \"Set Random Waits\",\n \"dependencyConditions\": [\n \"Succeeded\"\n ]\n }\n ],\n \"userProperties\": [],\n \"typeProperties\": {\n \"pipeline\": {\n \"referenceName\": \"02-Parent\",\n \"type\": \"PipelineReference\"\n },\n \"waitOnCompletion\": true,\n \"parameters\": {}\n }\n },\n {\n \"name\": \"Set Random Waits\",\n \"description\": \"For functional testing only.\",\n \"type\": \"SqlServerStoredProcedure\",\n \"dependsOn\": [],\n \"policy\": {\n \"timeout\": \"7.00:00:00\",\n \"retry\": 0,\n \"retryIntervalInSeconds\": 30,\n \"secureOutput\": false,\n \"secureInput\": false\n },\n \"userProperties\": [],\n \"typeProperties\": {\n \"storedProcedureName\": \"[[dbo].[SetRandomWaitValues]\"\n },\n \"linkedServiceName\": {\n \"referenceName\": \"SupportDatabase\",\n \"type\": \"LinkedServiceReference\"\n }\n }\n ],\n \"folder\": {\n \"name\": \"_ProcFwk\"\n },\n \"annotations\": [\n \"ADF.procfwk\"\n ]\n },\n \"dependsOn\": [\n \"[concat(variables('factoryId'), '/pipelines/02-Parent')]\",\n \"[concat(variables('factoryId'), '/linkedServices/SupportDatabase')]\"\n ]\n },\n {\n \"name\": \"[concat(parameters('factoryName'), '/02-Parent')]\",\n \"type\": \"Microsoft.DataFactory/factories/pipelines\",\n \"apiVersion\": \"2018-06-01\",\n \"properties\": {\n \"description\": \"ADF.procfwk parent pipeline used to bootstrap the orchestration framework in perform the first level ForEach calls in sequence for the metadata stages.\",\n \"activities\": [\n {\n \"name\": \"Get Stages\",\n \"description\": \"Returns a distinct list of execution stages within the framework metadata.\",\n \"type\": \"Lookup\",\n \"dependsOn\": [\n {\n \"activity\": \"Execution Wrapper\",\n \"dependencyConditions\": [\n \"Succeeded\"\n ]\n }\n ],\n \"policy\": {\n \"timeout\": \"7.00:00:00\",\n \"retry\": 0,\n \"retryIntervalInSeconds\": 30,\n \"secureOutput\": false,\n \"secureInput\": false\n },\n \"userProperties\": [],\n \"typeProperties\": {\n \"source\": {\n \"type\": \"AzureSqlSource\",\n \"sqlReaderStoredProcedureName\": \"[[procfwk].[GetStages]\",\n \"storedProcedureParameters\": {\n \"ExecutionId\": {\n \"type\": \"Guid\",\n \"value\": {\n \"value\": \"@activity('Execution Wrapper').output.firstRow.ExecutionId\",\n \"type\": \"Expression\"\n }\n }\n },\n \"queryTimeout\": \"02:00:00\"\n },\n \"dataset\": {\n \"referenceName\": \"GetSetMetadata\",\n \"type\": \"DatasetReference\",\n \"parameters\": {}\n },\n \"firstRowOnly\": false\n }\n },\n {\n \"name\": \"Execute Stages\",\n \"description\": \"Top level ForEach to sequentially call all processing stages within the framework metadata. Items for iteration passed from the Get Stages lookup activity.\",\n \"type\": \"ForEach\",\n \"dependsOn\": [\n {\n \"activity\": \"Get Stages\",\n \"dependencyConditions\": [\n \"Succeeded\"\n ]\n },\n {\n \"activity\": \"Get Tenant Id\",\n \"dependencyConditions\": [\n \"Succeeded\"\n ]\n },\n {\n \"activity\": \"Get Subscription Id\",\n \"dependencyConditions\": [\n \"Succeeded\"\n ]\n }\n ],\n \"userProperties\": [],\n \"typeProperties\": {\n \"items\": {\n \"value\": \"@activity('Get Stages').output.value\",\n \"type\": \"Expression\"\n },\n \"isSequential\": true,\n \"activities\": [\n {\n \"name\": \"Stage Executor\",\n \"description\": \"Call to the framework generic child pipeline for a given execution stage.\",\n \"type\": \"ExecutePipeline\",\n \"dependsOn\": [\n {\n \"activity\": \"Log Stage Preparing\",\n \"dependencyConditions\": [\n \"Succeeded\"\n ]\n }\n ],\n \"userProperties\": [],\n \"typeProperties\": {\n \"pipeline\": {\n \"referenceName\": \"03-Child\",\n \"type\": \"PipelineReference\"\n },\n \"waitOnCompletion\": true,\n \"parameters\": {\n \"StageId\": {\n \"value\": \"@item().StageId\",\n \"type\": \"Expression\"\n },\n \"ExecutionId\": {\n \"value\": \"@activity('Execution Wrapper').output.firstRow.ExecutionId\",\n \"type\": \"Expression\"\n },\n \"TenantId\": {\n \"value\": \"@activity('Get Tenant Id').output.firstRow.PropertyValue\",\n \"type\": \"Expression\"\n },\n \"SubscriptionId\": {\n \"value\": \"@activity('Get Subscription Id').output.firstRow.PropertyValue\",\n \"type\": \"Expression\"\n }\n }\n }\n },\n {\n \"name\": \"Log Stage Preparing\",\n \"description\": \"Update the current execution table flagging all pipelines within the stage as preparing.\",\n \"type\": \"SqlServerStoredProcedure\",\n \"dependsOn\": [\n {\n \"activity\": \"Check and Update Blockers\",\n \"dependencyConditions\": [\n \"Succeeded\"\n ]\n }\n ],\n \"policy\": {\n \"timeout\": \"7.00:00:00\",\n \"retry\": 0,\n \"retryIntervalInSeconds\": 30,\n \"secureOutput\": false,\n \"secureInput\": false\n },\n \"userProperties\": [],\n \"typeProperties\": {\n \"storedProcedureName\": \"[[procfwk].[SetLogStagePreparing]\",\n \"storedProcedureParameters\": {\n \"ExecutionId\": {\n \"value\": {\n \"value\": \"@activity('Execution Wrapper').output.firstRow.ExecutionId\",\n \"type\": \"Expression\"\n },\n \"type\": \"Guid\"\n },\n \"StageId\": {\n \"value\": {\n \"value\": \"@item().StageId\",\n \"type\": \"Expression\"\n },\n \"type\": \"Int32\"\n }\n }\n },\n \"linkedServiceName\": {\n \"referenceName\": \"SupportDatabase\",\n \"type\": \"LinkedServiceReference\"\n }\n },\n {\n \"name\": \"Check and Update Blockers\",\n \"description\": \"Used to double check and stop the next execution stage if failures and blockers have be incurred. This also depends on the failure handling property value which defines the stored procedure behaviour.\",\n \"type\": \"SqlServerStoredProcedure\",\n \"dependsOn\": [],\n \"policy\": {\n \"timeout\": \"7.00:00:00\",\n \"retry\": 0,\n \"retryIntervalInSeconds\": 30,\n \"secureOutput\": false,\n \"secureInput\": false\n },\n \"userProperties\": [],\n \"typeProperties\": {\n \"storedProcedureName\": \"[[procfwk].[CheckForBlockedPipelines]\",\n \"storedProcedureParameters\": {\n \"StageId\": {\n \"value\": {\n \"value\": \"@item().StageId\",\n \"type\": \"Expression\"\n },\n \"type\": \"Int32\"\n }\n }\n },\n \"linkedServiceName\": {\n \"referenceName\": \"SupportDatabase\",\n \"type\": \"LinkedServiceReference\"\n }\n }\n ]\n }\n },\n {\n \"name\": \"Execution Wrapper\",\n \"description\": \"Wrapper to reset and restart processing or create a completely new execution instance of the framework metadata.\",\n \"type\": \"Lookup\",\n \"dependsOn\": [\n {\n \"activity\": \"Clean Up Previous Run\",\n \"dependencyConditions\": [\n \"Succeeded\"\n ]\n }\n ],\n \"policy\": {\n \"timeout\": \"7.00:00:00\",\n \"retry\": 0,\n \"retryIntervalInSeconds\": 30,\n \"secureOutput\": false,\n \"secureInput\": false\n },\n \"userProperties\": [],\n \"typeProperties\": {\n \"source\": {\n \"type\": \"AzureSqlSource\",\n \"sqlReaderStoredProcedureName\": \"[[procfwk].[ExecutionWrapper]\",\n \"storedProcedureParameters\": {\n \"CallingDataFactory\": {\n \"type\": \"String\",\n \"value\": {\n \"value\": \"@pipeline().DataFactory\",\n \"type\": \"Expression\"\n }\n }\n },\n \"queryTimeout\": \"02:00:00\"\n },\n \"dataset\": {\n \"referenceName\": \"GetSetMetadata\",\n \"type\": \"DatasetReference\",\n \"parameters\": {}\n }\n }\n },\n {\n \"name\": \"Check Outcome and Update Logs\",\n \"description\": \"After a successful execution run the current execution metadata is moved to the long term logging table by this stored procedure call. Otherwise an error will be raised.\",\n \"type\": \"SqlServerStoredProcedure\",\n \"dependsOn\": [\n {\n \"activity\": \"Execute Stages\",\n \"dependencyConditions\": [\n \"Succeeded\"\n ]\n }\n ],\n \"policy\": {\n \"timeout\": \"7.00:00:00\",\n \"retry\": 0,\n \"retryIntervalInSeconds\": 30,\n \"secureOutput\": false,\n \"secureInput\": false\n },\n \"userProperties\": [],\n \"typeProperties\": {\n \"storedProcedureName\": \"[[procfwk].[UpdateExecutionLog]\"\n },\n \"linkedServiceName\": {\n \"referenceName\": \"SupportDatabase\",\n \"type\": \"LinkedServiceReference\"\n }\n },\n {\n \"name\": \"Get Tenant Id\",\n \"description\": \"Returning the Azure Tenant Id from the metadata properties table.\",\n \"type\": \"Lookup\",\n \"dependsOn\": [\n {\n \"activity\": \"Metadata Integrity Checks\",\n \"dependencyConditions\": [\n \"Succeeded\"\n ]\n }\n ],\n \"policy\": {\n \"timeout\": \"7.00:00:00\",\n \"retry\": 0,\n \"retryIntervalInSeconds\": 30,\n \"secureOutput\": false,\n \"secureInput\": false\n },\n \"userProperties\": [],\n \"typeProperties\": {\n \"source\": {\n \"type\": \"AzureSqlSource\",\n \"sqlReaderStoredProcedureName\": \"[[procfwk].[GetPropertyValue]\",\n \"storedProcedureParameters\": {\n \"PropertyName\": {\n \"type\": \"String\",\n \"value\": \"TenantId\"\n }\n },\n \"queryTimeout\": \"02:00:00\"\n },\n \"dataset\": {\n \"referenceName\": \"GetSetMetadata\",\n \"type\": \"DatasetReference\",\n \"parameters\": {}\n },\n \"firstRowOnly\": true\n }\n },\n {\n \"name\": \"Get Subscription Id\",\n \"description\": \"Returning the Azure Subscription Id from the metadata properties table.\",\n \"type\": \"Lookup\",\n \"dependsOn\": [\n {\n \"activity\": \"Metadata Integrity Checks\",\n \"dependencyConditions\": [\n \"Succeeded\"\n ]\n }\n ],\n \"policy\": {\n \"timeout\": \"7.00:00:00\",\n \"retry\": 0,\n \"retryIntervalInSeconds\": 30,\n \"secureOutput\": false,\n \"secureInput\": true\n },\n \"userProperties\": [],\n \"typeProperties\": {\n \"source\": {\n \"type\": \"AzureSqlSource\",\n \"sqlReaderStoredProcedureName\": \"[[procfwk].[GetPropertyValue]\",\n \"storedProcedureParameters\": {\n \"PropertyName\": {\n \"type\": \"String\",\n \"value\": \"SubscriptionId\"\n }\n },\n \"queryTimeout\": \"02:00:00\"\n },\n \"dataset\": {\n \"referenceName\": \"GetSetMetadata\",\n \"type\": \"DatasetReference\",\n \"parameters\": {}\n }\n }\n },\n {\n \"name\": \"Metadata Integrity Checks\",\n \"description\": \"Performs a series of checks on all metadata held in the framework SQLDB. This is intended to raise errors before an execution run even starts.\",\n \"type\": \"Lookup\",\n \"dependsOn\": [],\n \"policy\": {\n \"timeout\": \"7.00:00:00\",\n \"retry\": 0,\n \"retryIntervalInSeconds\": 30,\n \"secureOutput\": false,\n \"secureInput\": false\n },\n \"userProperties\": [],\n \"typeProperties\": {\n \"source\": {\n \"type\": \"AzureSqlSource\",\n \"sqlReaderStoredProcedureName\": \"[[procfwk].[CheckMetadataIntegrity]\",\n \"storedProcedureParameters\": {\n \"DebugMode\": {\n \"type\": \"Boolean\",\n \"value\": \"false\"\n }\n },\n \"queryTimeout\": \"02:00:00\"\n },\n \"dataset\": {\n \"referenceName\": \"GetSetMetadata\",\n \"type\": \"DatasetReference\",\n \"parameters\": {}\n },\n \"firstRowOnly\": false\n }\n },\n {\n \"name\": \"Clean Up Previous Run\",\n \"description\": \"Handle Worker pipelines that are reported as Running when the parent pipeline is called again. Get what the actual status of those pipelines is.\",\n \"type\": \"ForEach\",\n \"dependsOn\": [\n {\n \"activity\": \"Metadata Integrity Checks\",\n \"dependencyConditions\": [\n \"Succeeded\"\n ]\n }\n ],\n \"userProperties\": [],\n \"typeProperties\": {\n \"items\": {\n \"value\": \"@activity('Metadata Integrity Checks').output.value\",\n \"type\": \"Expression\"\n },\n \"isSequential\": false,\n \"activities\": [\n {\n \"name\": \"Get SPN Details\",\n \"type\": \"Lookup\",\n \"dependsOn\": [],\n \"policy\": {\n \"timeout\": \"7.00:00:00\",\n \"retry\": 0,\n \"retryIntervalInSeconds\": 30,\n \"secureOutput\": false,\n \"secureInput\": false\n },\n \"userProperties\": [],\n \"typeProperties\": {\n \"source\": {\n \"type\": \"AzureSqlSource\",\n \"sqlReaderStoredProcedureName\": \"[[procfwk].[GetServicePrincipal]\",\n \"storedProcedureParameters\": {\n \"DataFactory\": {\n \"type\": \"String\",\n \"value\": {\n \"value\": \"@item().DataFactoryName\",\n \"type\": \"Expression\"\n }\n },\n \"PipelineName\": {\n \"type\": \"String\",\n \"value\": {\n \"value\": \"@item().PipelineName\",\n \"type\": \"Expression\"\n }\n }\n },\n \"queryTimeout\": \"02:00:00\"\n },\n \"dataset\": {\n \"referenceName\": \"GetSetMetadata\",\n \"type\": \"DatasetReference\",\n \"parameters\": {}\n }\n }\n },\n {\n \"name\": \"Log Pipeline Checking\",\n \"type\": \"SqlServerStoredProcedure\",\n \"dependsOn\": [],\n \"policy\": {\n \"timeout\": \"7.00:00:00\",\n \"retry\": 0,\n \"retryIntervalInSeconds\": 30,\n \"secureOutput\": false,\n \"secureInput\": false\n },\n \"userProperties\": [],\n \"typeProperties\": {\n \"storedProcedureName\": \"[[procfwk].[SetLogPipelineChecking]\",\n \"storedProcedureParameters\": {\n \"ExecutionId\": {\n \"value\": {\n \"value\": \"@item().LocalExecutionId\",\n \"type\": \"Expression\"\n },\n \"type\": \"Guid\"\n },\n \"PipelineId\": {\n \"value\": {\n \"value\": \"@item().PipelineId\",\n \"type\": \"Expression\"\n },\n \"type\": \"Int32\"\n },\n \"StageId\": {\n \"value\": {\n \"value\": \"@item().StageId\",\n \"type\": \"Expression\"\n },\n \"type\": \"Int32\"\n }\n }\n },\n \"linkedServiceName\": {\n \"referenceName\": \"SupportDatabase\",\n \"type\": \"LinkedServiceReference\"\n }\n },\n {\n \"name\": \"Get Pipeline Status\",\n \"type\": \"AzureFunctionActivity\",\n \"dependsOn\": [\n {\n \"activity\": \"Get SPN Details\",\n \"dependencyConditions\": [\n \"Succeeded\"\n ]\n },\n {\n \"activity\": \"Log Pipeline Checking\",\n \"dependencyConditions\": [\n \"Succeeded\"\n ]\n }\n ],\n \"policy\": {\n \"timeout\": \"7.00:00:00\",\n \"retry\": 0,\n \"retryIntervalInSeconds\": 30,\n \"secureOutput\": false,\n \"secureInput\": false\n },\n \"userProperties\": [],\n \"typeProperties\": {\n \"functionName\": \"CheckPipelineStatus\",\n \"method\": \"POST\",\n \"headers\": {},\n \"body\": {\n \"value\": \"@concat('\\n{\\n \\\"tenantId\\\": \\\"',item().TenantId,'\\\",\\n \\\"applicationId\\\": \\\"',activity('Get SPN Details').output.firstRow.Id,'\\\",\\n \\\"authenticationKey\\\": \\\"',activity('Get SPN Details').output.firstRow.Secret,'\\\",\\n \\\"subscriptionId\\\": \\\"',item().SubscriptionId,'\\\",\\n \\\"resourceGroup\\\": \\\"',item().ResourceGroupName,'\\\",\\n \\\"factoryName\\\": \\\"',item().DataFactoryName,'\\\",\\n \\\"pipelineName\\\": \\\"',item().PipelineName,'\\\",\\n \\\"runId\\\": \\\"',item().AdfPipelineRunId,'\\\"\\n}')\",\n \"type\": \"Expression\"\n }\n },\n \"linkedServiceName\": {\n \"referenceName\": \"FrameworkFunctions\",\n \"type\": \"LinkedServiceReference\"\n }\n },\n {\n \"name\": \"Set Pipeline Status\",\n \"type\": \"Switch\",\n \"dependsOn\": [\n {\n \"activity\": \"Get Pipeline Status\",\n \"dependencyConditions\": [\n \"Succeeded\"\n ]\n }\n ],\n \"userProperties\": [],\n \"typeProperties\": {\n \"on\": {\n \"value\": \"@activity('Get Pipeline Status').output.Status\",\n \"type\": \"Expression\"\n },\n \"cases\": [\n {\n \"value\": \"Failed\",\n \"activities\": [\n {\n \"name\": \"Pipeline Status Failed\",\n \"type\": \"SqlServerStoredProcedure\",\n \"dependsOn\": [],\n \"policy\": {\n \"timeout\": \"7.00:00:00\",\n \"retry\": 0,\n \"retryIntervalInSeconds\": 30,\n \"secureOutput\": false,\n \"secureInput\": false\n },\n \"userProperties\": [],\n \"typeProperties\": {\n \"storedProcedureName\": \"[[procfwk].[SetLogPipelineFailed]\",\n \"storedProcedureParameters\": {\n \"ExecutionId\": {\n \"value\": {\n \"value\": \"@item().LocalExecutionId\",\n \"type\": \"Expression\"\n },\n \"type\": \"Guid\"\n },\n \"PipelineId\": {\n \"value\": {\n \"value\": \"@item().PipelineId\",\n \"type\": \"Expression\"\n },\n \"type\": \"Int32\"\n },\n \"RunId\": {\n \"value\": null,\n \"type\": \"Guid\"\n },\n \"StageId\": {\n \"value\": {\n \"value\": \"@item().StageId\",\n \"type\": \"Expression\"\n },\n \"type\": \"Int32\"\n }\n }\n },\n \"linkedServiceName\": {\n \"referenceName\": \"SupportDatabase\",\n \"type\": \"LinkedServiceReference\"\n }\n }\n ]\n },\n {\n \"value\": \"Succeeded\",\n \"activities\": [\n {\n \"name\": \"Pipeline Status Succeeded\",\n \"type\": \"SqlServerStoredProcedure\",\n \"dependsOn\": [],\n \"policy\": {\n \"timeout\": \"7.00:00:00\",\n \"retry\": 0,\n \"retryIntervalInSeconds\": 30,\n \"secureOutput\": false,\n \"secureInput\": false\n },\n \"userProperties\": [],\n \"typeProperties\": {\n \"storedProcedureName\": \"[[procfwk].[SetLogPipelineSuccess]\",\n \"storedProcedureParameters\": {\n \"ExecutionId\": {\n \"value\": {\n \"value\": \"@item().LocalExecutionId\",\n \"type\": \"Expression\"\n },\n \"type\": \"Guid\"\n },\n \"PipelineId\": {\n \"value\": {\n \"value\": \"@item().PipelineId\",\n \"type\": \"Expression\"\n },\n \"type\": \"Int32\"\n },\n \"StageId\": {\n \"value\": {\n \"value\": \"@item().StageId\",\n \"type\": \"Expression\"\n },\n \"type\": \"Int32\"\n }\n }\n },\n \"linkedServiceName\": {\n \"referenceName\": \"SupportDatabase\",\n \"type\": \"LinkedServiceReference\"\n }\n }\n ]\n },\n {\n \"value\": \"Queued\",\n \"activities\": [\n {\n \"name\": \"Pipeline Status Queued - Running\",\n \"type\": \"SqlServerStoredProcedure\",\n \"dependsOn\": [],\n \"policy\": {\n \"timeout\": \"7.00:00:00\",\n \"retry\": 0,\n \"retryIntervalInSeconds\": 30,\n \"secureOutput\": false,\n \"secureInput\": false\n },\n \"userProperties\": [],\n \"typeProperties\": {\n \"storedProcedureName\": \"[[procfwk].[SetLogPipelineRunning]\",\n \"storedProcedureParameters\": {\n \"ExecutionId\": {\n \"value\": {\n \"value\": \"@item().LocalExecutionId\",\n \"type\": \"Expression\"\n },\n \"type\": \"Guid\"\n },\n \"PipelineId\": {\n \"value\": {\n \"value\": \"@item().PipelineId\",\n \"type\": \"Expression\"\n },\n \"type\": \"Int32\"\n },\n \"StageId\": {\n \"value\": {\n \"value\": \"@item().StageId\",\n \"type\": \"Expression\"\n },\n \"type\": \"Int32\"\n }\n }\n },\n \"linkedServiceName\": {\n \"referenceName\": \"SupportDatabase\",\n \"type\": \"LinkedServiceReference\"\n }\n }\n ]\n },\n {\n \"value\": \"InProgress\",\n \"activities\": [\n {\n \"name\": \"Pipeline Status InProgress - Running\",\n \"type\": \"SqlServerStoredProcedure\",\n \"dependsOn\": [],\n \"policy\": {\n \"timeout\": \"7.00:00:00\",\n \"retry\": 0,\n \"retryIntervalInSeconds\": 30,\n \"secureOutput\": false,\n \"secureInput\": false\n },\n \"userProperties\": [],\n \"typeProperties\": {\n \"storedProcedureName\": \"[[procfwk].[SetLogPipelineRunning]\",\n \"storedProcedureParameters\": {\n \"ExecutionId\": {\n \"value\": {\n \"value\": \"@item().LocalExecutionId\",\n \"type\": \"Expression\"\n },\n \"type\": \"Guid\"\n },\n \"PipelineId\": {\n \"value\": {\n \"value\": \"@item().PipelineId\",\n \"type\": \"Expression\"\n },\n \"type\": \"Int32\"\n },\n \"StageId\": {\n \"value\": {\n \"value\": \"@item().StageId\",\n \"type\": \"Expression\"\n },\n \"type\": \"Int32\"\n }\n }\n },\n \"linkedServiceName\": {\n \"referenceName\": \"SupportDatabase\",\n \"type\": \"LinkedServiceReference\"\n }\n }\n ]\n },\n {\n \"value\": \"Cancelled\",\n \"activities\": [\n {\n \"name\": \"Pipeline Status Cancelled\",\n \"type\": \"SqlServerStoredProcedure\",\n \"dependsOn\": [],\n \"policy\": {\n \"timeout\": \"7.00:00:00\",\n \"retry\": 0,\n \"retryIntervalInSeconds\": 30,\n \"secureOutput\": false,\n \"secureInput\": false\n },\n \"userProperties\": [],\n \"typeProperties\": {\n \"storedProcedureName\": \"[[procfwk].[SetLogPipelineCancelled]\",\n \"storedProcedureParameters\": {\n \"ExecutionId\": {\n \"value\": {\n \"value\": \"@item().LocalExecutionId\",\n \"type\": \"Expression\"\n },\n \"type\": \"Guid\"\n },\n \"PipelineId\": {\n \"value\": {\n \"value\": \"@item().PipelineId\",\n \"type\": \"Expression\"\n },\n \"type\": \"Int32\"\n },\n \"StageId\": {\n \"value\": {\n \"value\": \"@item().StageId\",\n \"type\": \"Expression\"\n },\n \"type\": \"Int32\"\n }\n }\n },\n \"linkedServiceName\": {\n \"referenceName\": \"SupportDatabase\",\n \"type\": \"LinkedServiceReference\"\n }\n }\n ]\n }\n ],\n \"defaultActivities\": [\n {\n \"name\": \"Pipeline Status Unknown\",\n \"type\": \"SqlServerStoredProcedure\",\n \"dependsOn\": [],\n \"policy\": {\n \"timeout\": \"7.00:00:00\",\n \"retry\": 0,\n \"retryIntervalInSeconds\": 30,\n \"secureOutput\": false,\n \"secureInput\": false\n },\n \"userProperties\": [],\n \"typeProperties\": {\n \"storedProcedureName\": \"[[procfwk].[SetLogPipelineUnknown]\",\n \"storedProcedureParameters\": {\n \"ExecutionId\": {\n \"value\": {\n \"value\": \"@item().LocalExecutionId\",\n \"type\": \"Expression\"\n },\n \"type\": \"Guid\"\n },\n \"PipelineId\": {\n \"value\": {\n \"value\": \"@item().PipelineId\",\n \"type\": \"Expression\"\n },\n \"type\": \"Int32\"\n },\n \"StageId\": {\n \"value\": {\n \"value\": \"@item().StageId\",\n \"type\": \"Expression\"\n },\n \"type\": \"Int32\"\n }\n }\n },\n \"linkedServiceName\": {\n \"referenceName\": \"SupportDatabase\",\n \"type\": \"LinkedServiceReference\"\n }\n }\n ]\n }\n },\n {\n \"name\": \"Set Last Check DateTime\",\n \"type\": \"SqlServerStoredProcedure\",\n \"dependsOn\": [\n {\n \"activity\": \"Get Pipeline Status\",\n \"dependencyConditions\": [\n \"Succeeded\"\n ]\n }\n ],\n \"policy\": {\n \"timeout\": \"7.00:00:00\",\n \"retry\": 0,\n \"retryIntervalInSeconds\": 30,\n \"secureOutput\": false,\n \"secureInput\": false\n },\n \"userProperties\": [],\n \"typeProperties\": {\n \"storedProcedureName\": \"[[procfwk].[SetLogPipelineLastStatusCheck]\",\n \"storedProcedureParameters\": {\n \"ExecutionId\": {\n \"value\": {\n \"value\": \"@item().LocalExecutionId\",\n \"type\": \"Expression\"\n },\n \"type\": \"Guid\"\n },\n \"PipelineId\": {\n \"value\": {\n \"value\": \"@item().PipelineId\",\n \"type\": \"Expression\"\n },\n \"type\": \"Int32\"\n },\n \"StageId\": {\n \"value\": {\n \"value\": \"@item().StageId\",\n \"type\": \"Expression\"\n },\n \"type\": \"Int32\"\n }\n }\n },\n \"linkedServiceName\": {\n \"referenceName\": \"SupportDatabase\",\n \"type\": \"LinkedServiceReference\"\n }\n }\n ]\n }\n }\n ],\n \"folder\": {\n \"name\": \"_ProcFwk\"\n },\n \"annotations\": [\n \"ADF.procfwk\"\n ]\n },\n \"dependsOn\": [\n \"[concat(variables('factoryId'), '/datasets/GetSetMetadata')]\",\n \"[concat(variables('factoryId'), '/linkedServices/SupportDatabase')]\",\n \"[concat(variables('factoryId'), '/pipelines/03-Child')]\",\n \"[concat(variables('factoryId'), '/linkedServices/FrameworkFunctions')]\"\n ]\n },\n {\n \"name\": \"[concat(parameters('factoryName'), '/03-Child')]\",\n \"type\": \"Microsoft.DataFactory/factories/pipelines\",\n \"apiVersion\": \"2018-06-01\",\n \"properties\": {\n \"description\": \"ADF.procfwk child pipeline used to execute Worker pipelines within a given execution stage. This pipeline will be called once for each stage, then execute all Workers in parallel.\",\n \"activities\": [\n {\n \"name\": \"Get Pipelines\",\n \"description\": \"Returns all pipelines from the metadata to be executed within a given processing stage.\",\n \"type\": \"Lookup\",\n \"dependsOn\": [],\n \"policy\": {\n \"timeout\": \"7.00:00:00\",\n \"retry\": 0,\n \"retryIntervalInSeconds\": 30,\n \"secureOutput\": false,\n \"secureInput\": false\n },\n \"userProperties\": [],\n \"typeProperties\": {\n \"source\": {\n \"type\": \"AzureSqlSource\",\n \"sqlReaderStoredProcedureName\": \"[[procfwk].[GetPipelinesInStage]\",\n \"storedProcedureParameters\": {\n \"StageId\": {\n \"type\": \"Int32\",\n \"value\": {\n \"value\": \"@pipeline().parameters.StageId\",\n \"type\": \"Expression\"\n }\n }\n },\n \"queryTimeout\": \"02:00:00\"\n },\n \"dataset\": {\n \"referenceName\": \"GetSetMetadata\",\n \"type\": \"DatasetReference\",\n \"parameters\": {}\n },\n \"firstRowOnly\": false\n }\n },\n {\n \"name\": \"Execute Pipelines\",\n \"description\": \"Second level ForEach to run in parallel all pipelines within the stage. Items for iteration passed from the Get Pipelines lookup activity.\",\n \"type\": \"ForEach\",\n \"dependsOn\": [\n {\n \"activity\": \"Get Pipelines\",\n \"dependencyConditions\": [\n \"Succeeded\"\n ]\n }\n ],\n \"userProperties\": [],\n \"typeProperties\": {\n \"items\": {\n \"value\": \"@activity('Get Pipelines').output.value\",\n \"type\": \"Expression\"\n },\n \"isSequential\": false,\n \"batchCount\": 40,\n \"activities\": [\n {\n \"name\": \"Execute Pipeline\",\n \"description\": \"The lowest level executor with the metadata framework to call existing processing pipelines within Data Factory. The function called will block processing and wait for an outcome.\",\n \"type\": \"AzureFunctionActivity\",\n \"dependsOn\": [\n {\n \"activity\": \"Log Pipeline Running\",\n \"dependencyConditions\": [\n \"Succeeded\"\n ]\n },\n {\n \"activity\": \"Get Pipeline Params\",\n \"dependencyConditions\": [\n \"Succeeded\"\n ]\n },\n {\n \"activity\": \"Get SPN Details\",\n \"dependencyConditions\": [\n \"Succeeded\"\n ]\n }\n ],\n \"policy\": {\n \"timeout\": \"7.00:00:00\",\n \"retry\": 0,\n \"retryIntervalInSeconds\": 30,\n \"secureOutput\": false,\n \"secureInput\": false\n },\n \"userProperties\": [],\n \"typeProperties\": {\n \"functionName\": \"ExecutePipeline\",\n \"method\": \"POST\",\n \"headers\": {},\n \"body\": {\n \"value\": \"@concat('\\n{\\n\\t\\\"tenantId\\\": \\\"',pipeline().parameters.TenantId,'\\\",\\n\\t\\\"applicationId\\\": \\\"',activity('Get SPN Details').output.firstRow.Id,'\\\",\\n\\t\\\"authenticationKey\\\": \\\"',activity('Get SPN Details').output.firstRow.Secret,'\\\",\\n\\t\\\"subscriptionId\\\": \\\"',pipeline().parameters.SubscriptionId,'\\\",\\n\\t\\\"resourceGroup\\\": \\\"',item().ResourceGroupName,'\\\",\\n\\t\\\"factoryName\\\": \\\"',item().DataFactoryName,'\\\",\\n\\t\\\"pipelineName\\\": \\\"',item().PipelineName,'\\\"',activity('Get Pipeline Params').output.firstRow.Params,'\\n}')\",\n \"type\": \"Expression\"\n }\n },\n \"linkedServiceName\": {\n \"referenceName\": \"FrameworkFunctions\",\n \"type\": \"LinkedServiceReference\"\n }\n },\n {\n \"name\": \"Get Pipeline Params\",\n \"description\": \"Returns any parameters from metadata required for the processing pipeline being called. The output can be an empty string if no parameters are required.\",\n \"type\": \"Lookup\",\n \"dependsOn\": [],\n \"policy\": {\n \"timeout\": \"7.00:00:00\",\n \"retry\": 0,\n \"retryIntervalInSeconds\": 30,\n \"secureOutput\": false,\n \"secureInput\": false\n },\n \"userProperties\": [],\n \"typeProperties\": {\n \"source\": {\n \"type\": \"AzureSqlSource\",\n \"sqlReaderStoredProcedureName\": \"[[procfwk].[GetPipelineParameters]\",\n \"storedProcedureParameters\": {\n \"PipelineId\": {\n \"type\": \"Int32\",\n \"value\": {\n \"value\": \"@item().PipelineId\",\n \"type\": \"Expression\"\n }\n }\n },\n \"queryTimeout\": \"02:00:00\"\n },\n \"dataset\": {\n \"referenceName\": \"GetSetMetadata\",\n \"type\": \"DatasetReference\",\n \"parameters\": {}\n }\n }\n },\n {\n \"name\": \"Log Pipeline Running\",\n \"description\": \"Sets the current pipeline with a status of running within the current execution database table.\",\n \"type\": \"SqlServerStoredProcedure\",\n \"dependsOn\": [],\n \"policy\": {\n \"timeout\": \"7.00:00:00\",\n \"retry\": 0,\n \"retryIntervalInSeconds\": 30,\n \"secureOutput\": false,\n \"secureInput\": false\n },\n \"userProperties\": [],\n \"typeProperties\": {\n \"storedProcedureName\": \"[[procfwk].[SetLogPipelineRunning]\",\n \"storedProcedureParameters\": {\n \"ExecutionId\": {\n \"value\": {\n \"value\": \"@pipeline().parameters.ExecutionId\",\n \"type\": \"Expression\"\n },\n \"type\": \"Guid\"\n },\n \"PipelineId\": {\n \"value\": {\n \"value\": \"@item().PipelineId\",\n \"type\": \"Expression\"\n },\n \"type\": \"Int32\"\n },\n \"StageId\": {\n \"value\": {\n \"value\": \"@pipeline().parameters.StageId\",\n \"type\": \"Expression\"\n },\n \"type\": \"Int32\"\n }\n }\n },\n \"linkedServiceName\": {\n \"referenceName\": \"SupportDatabase\",\n \"type\": \"LinkedServiceReference\"\n }\n },\n {\n \"name\": \"Get SPN Details\",\n \"description\": \"Return the SPN ID and Secret for the processing pipeline being executed. Called at this level as each pipeline can have a different SPN.\",\n \"type\": \"Lookup\",\n \"dependsOn\": [],\n \"policy\": {\n \"timeout\": \"7.00:00:00\",\n \"retry\": 0,\n \"retryIntervalInSeconds\": 30,\n \"secureOutput\": false,\n \"secureInput\": false\n },\n \"userProperties\": [],\n \"typeProperties\": {\n \"source\": {\n \"type\": \"AzureSqlSource\",\n \"sqlReaderStoredProcedureName\": \"[[procfwk].[GetServicePrincipal]\",\n \"storedProcedureParameters\": {\n \"DataFactory\": {\n \"type\": \"String\",\n \"value\": {\n \"value\": \"@item().DataFactoryName\",\n \"type\": \"Expression\"\n }\n },\n \"PipelineName\": {\n \"type\": \"String\",\n \"value\": {\n \"value\": \"@item().PipelineName\",\n \"type\": \"Expression\"\n }\n }\n },\n \"queryTimeout\": \"02:00:00\"\n },\n \"dataset\": {\n \"referenceName\": \"GetSetMetadata\",\n \"type\": \"DatasetReference\",\n \"parameters\": {}\n }\n }\n },\n {\n \"name\": \"Log Activity Failure\",\n \"description\": \"Handle true failures from calling out to the Azure Function and update the current execution table accordingly so a restart can occur.\",\n \"type\": \"SqlServerStoredProcedure\",\n \"dependsOn\": [\n {\n \"activity\": \"Execute Pipeline\",\n \"dependencyConditions\": [\n \"Failed\"\n ]\n }\n ],\n \"policy\": {\n \"timeout\": \"7.00:00:00\",\n \"retry\": 0,\n \"retryIntervalInSeconds\": 30,\n \"secureOutput\": false,\n \"secureInput\": false\n },\n \"userProperties\": [],\n \"typeProperties\": {\n \"storedProcedureName\": \"[[procfwk].[SetLogActivityFailed]\",\n \"storedProcedureParameters\": {\n \"ExecutionId\": {\n \"value\": {\n \"value\": \"@pipeline().parameters.ExecutionId\",\n \"type\": \"Expression\"\n },\n \"type\": \"Guid\"\n },\n \"PipelineId\": {\n \"value\": {\n \"value\": \"@item().PipelineId\",\n \"type\": \"Expression\"\n },\n \"type\": \"Int32\"\n },\n \"StageId\": {\n \"value\": {\n \"value\": \"@pipeline().parameters.StageId\",\n \"type\": \"Expression\"\n },\n \"type\": \"Int32\"\n },\n \"CallingActivity\": {\n \"value\": \"ExecutePipeline\",\n \"type\": \"String\"\n }\n }\n },\n \"linkedServiceName\": {\n \"referenceName\": \"SupportDatabase\",\n \"type\": \"LinkedServiceReference\"\n }\n },\n {\n \"name\": \"Running Pipeline Handler\",\n \"type\": \"ExecutePipeline\",\n \"dependsOn\": [\n {\n \"activity\": \"Execute Pipeline\",\n \"dependencyConditions\": [\n \"Succeeded\"\n ]\n }\n ],\n \"userProperties\": [],\n \"typeProperties\": {\n \"pipeline\": {\n \"referenceName\": \"04-Infant\",\n \"type\": \"PipelineReference\"\n },\n \"waitOnCompletion\": true,\n \"parameters\": {\n \"tenantId\": {\n \"value\": \"@pipeline().parameters.TenantId\",\n \"type\": \"Expression\"\n },\n \"applicationId\": {\n \"value\": \"@activity('Get SPN Details').output.firstRow.Id\",\n \"type\": \"Expression\"\n },\n \"authenticationKey\": {\n \"value\": \"@activity('Get SPN Details').output.firstRow.Secret\",\n \"type\": \"Expression\"\n },\n \"subscriptionId\": {\n \"value\": \"@pipeline().parameters.SubscriptionId\",\n \"type\": \"Expression\"\n },\n \"resourceGroup\": {\n \"value\": \"@item().ResourceGroupName\",\n \"type\": \"Expression\"\n },\n \"factoryName\": {\n \"value\": \"@item().DataFactoryName\",\n \"type\": \"Expression\"\n },\n \"pipelineName\": {\n \"value\": \"@item().PipelineName\",\n \"type\": \"Expression\"\n },\n \"runId\": {\n \"value\": \"@activity('Execute Pipeline').output.RunId\",\n \"type\": \"Expression\"\n },\n \"executionId\": {\n \"value\": \"@pipeline().parameters.ExecutionId\",\n \"type\": \"Expression\"\n },\n \"stageId\": {\n \"value\": \"@pipeline().parameters.StageId\",\n \"type\": \"Expression\"\n },\n \"pipelineId\": {\n \"value\": \"@item().PipelineId\",\n \"type\": \"Expression\"\n }\n }\n }\n },\n {\n \"name\": \"Set Run Id\",\n \"description\": \"Provide the actual ADF run ID back to the current execution table for long term logging and alignment between the metadata other Azure monitoring tools.\",\n \"type\": \"SqlServerStoredProcedure\",\n \"dependsOn\": [\n {\n \"activity\": \"Execute Pipeline\",\n \"dependencyConditions\": [\n \"Succeeded\"\n ]\n }\n ],\n \"policy\": {\n \"timeout\": \"7.00:00:00\",\n \"retry\": 0,\n \"retryIntervalInSeconds\": 30,\n \"secureOutput\": false,\n \"secureInput\": false\n },\n \"userProperties\": [],\n \"typeProperties\": {\n \"storedProcedureName\": \"[[procfwk].[SetLogPipelineRunId]\",\n \"storedProcedureParameters\": {\n \"ExecutionId\": {\n \"value\": {\n \"value\": \"@pipeline().parameters.ExecutionId\",\n \"type\": \"Expression\"\n },\n \"type\": \"Guid\"\n },\n \"PipelineId\": {\n \"value\": {\n \"value\": \"@item().PipelineId\",\n \"type\": \"Expression\"\n },\n \"type\": \"Int32\"\n },\n \"RunId\": {\n \"value\": {\n \"value\": \"@activity('Execute Pipeline').output.RunId\",\n \"type\": \"Expression\"\n },\n \"type\": \"Guid\"\n },\n \"StageId\": {\n \"value\": {\n \"value\": \"@pipeline().parameters.StageId\",\n \"type\": \"Expression\"\n },\n \"type\": \"Int32\"\n }\n }\n },\n \"linkedServiceName\": {\n \"referenceName\": \"SupportDatabase\",\n \"type\": \"LinkedServiceReference\"\n }\n },\n {\n \"name\": \"Check For Alerts\",\n \"description\": \"Checks the properties tables and if any recipients in the database require alerts sending for the current pipeline ID.\",\n \"type\": \"Lookup\",\n \"dependsOn\": [\n {\n \"activity\": \"Set Run Id\",\n \"dependencyConditions\": [\n \"Succeeded\"\n ]\n },\n {\n \"activity\": \"Running Pipeline Handler\",\n \"dependencyConditions\": [\n \"Completed\"\n ]\n }\n ],\n \"policy\": {\n \"timeout\": \"7.00:00:00\",\n \"retry\": 0,\n \"retryIntervalInSeconds\": 30,\n \"secureOutput\": false,\n \"secureInput\": false\n },\n \"userProperties\": [],\n \"typeProperties\": {\n \"source\": {\n \"type\": \"AzureSqlSource\",\n \"sqlReaderStoredProcedureName\": \"[[procfwk].[CheckForEmailAlerts]\",\n \"storedProcedureParameters\": {\n \"PipelineId\": {\n \"type\": \"Int32\",\n \"value\": {\n \"value\": \"@item().PipelineId\",\n \"type\": \"Expression\"\n }\n }\n },\n \"queryTimeout\": \"02:00:00\"\n },\n \"dataset\": {\n \"referenceName\": \"GetSetMetadata\",\n \"type\": \"DatasetReference\",\n \"parameters\": {}\n },\n \"firstRowOnly\": true\n }\n },\n {\n \"name\": \"Send Alerts\",\n \"description\": \"True = alerts need sending.\\nFalse = do nothing.\",\n \"type\": \"IfCondition\",\n \"dependsOn\": [\n {\n \"activity\": \"Check For Alerts\",\n \"dependencyConditions\": [\n \"Succeeded\"\n ]\n }\n ],\n \"userProperties\": [],\n \"typeProperties\": {\n \"expression\": {\n \"value\": \"@activity('Check For Alerts').output.firstRow.SendAlerts\",\n \"type\": \"Expression\"\n },\n \"ifTrueActivities\": [\n {\n \"name\": \"Get Email Parts\",\n \"type\": \"Lookup\",\n \"dependsOn\": [],\n \"policy\": {\n \"timeout\": \"7.00:00:00\",\n \"retry\": 0,\n \"retryIntervalInSeconds\": 30,\n \"secureOutput\": false,\n \"secureInput\": false\n },\n \"userProperties\": [],\n \"typeProperties\": {\n \"source\": {\n \"type\": \"AzureSqlSource\",\n \"sqlReaderStoredProcedureName\": \"[[procfwk].[GetEmailAlertParts]\",\n \"storedProcedureParameters\": {\n \"PipelineId\": {\n \"type\": \"Int32\",\n \"value\": {\n \"value\": \"@item().PipelineId\",\n \"type\": \"Expression\"\n }\n }\n },\n \"queryTimeout\": \"02:00:00\"\n },\n \"dataset\": {\n \"referenceName\": \"GetSetMetadata\",\n \"type\": \"DatasetReference\",\n \"parameters\": {}\n },\n \"firstRowOnly\": true\n }\n },\n {\n \"name\": \"Send Email\",\n \"type\": \"AzureFunctionActivity\",\n \"dependsOn\": [\n {\n \"activity\": \"Get Email Parts\",\n \"dependencyConditions\": [\n \"Succeeded\"\n ]\n }\n ],\n \"policy\": {\n \"timeout\": \"7.00:00:00\",\n \"retry\": 0,\n \"retryIntervalInSeconds\": 30,\n \"secureOutput\": false,\n \"secureInput\": false\n },\n \"userProperties\": [],\n \"typeProperties\": {\n \"functionName\": \"SendEmail\",\n \"method\": \"POST\",\n \"headers\": {},\n \"body\": {\n \"value\": \"@activity('Get Email Parts').output.firstRow\",\n \"type\": \"Expression\"\n }\n },\n \"linkedServiceName\": {\n \"referenceName\": \"FrameworkFunctions\",\n \"type\": \"LinkedServiceReference\"\n }\n }\n ]\n }\n }\n ]\n }\n }\n ],\n \"parameters\": {\n \"StageId\": {\n \"type\": \"int\"\n },\n \"ExecutionId\": {\n \"type\": \"string\"\n },\n \"TenantId\": {\n \"type\": \"string\"\n },\n \"SubscriptionId\": {\n \"type\": \"string\"\n }\n },\n \"variables\": {\n \"FunctionBody\": {\n \"type\": \"String\"\n }\n },\n \"folder\": {\n \"name\": \"_ProcFwk\"\n },\n \"annotations\": [\n \"ADF.procfwk\"\n ]\n },\n \"dependsOn\": [\n \"[concat(variables('factoryId'), '/datasets/GetSetMetadata')]\",\n \"[concat(variables('factoryId'), '/linkedServices/FrameworkFunctions')]\",\n \"[concat(variables('factoryId'), '/linkedServices/SupportDatabase')]\",\n \"[concat(variables('factoryId'), '/pipelines/04-Infant')]\"\n ]\n },\n {\n \"name\": \"[concat(parameters('factoryName'), '/04-Infant')]\",\n \"type\": \"Microsoft.DataFactory/factories/pipelines\",\n \"apiVersion\": \"2018-06-01\",\n \"properties\": {\n \"description\": \"ADF.procfwk infant pipeline used to check when the processing pipeline called by the Child completes and passes the resulting status back to the metadata database.\",\n \"activities\": [\n {\n \"name\": \"Wait Until Pipeline Completes\",\n \"description\": \"Loops until the pipeline called completes.\\n\\nSimple status:\\n- Running = new iteration.\\n- Done = break.\",\n \"type\": \"Until\",\n \"dependsOn\": [\n {\n \"activity\": \"Get Wait Duration\",\n \"dependencyConditions\": [\n \"Succeeded\"\n ]\n }\n ],\n \"userProperties\": [],\n \"typeProperties\": {\n \"expression\": {\n \"value\": \"@equals('Done',activity('Get Pipeline Status').output.SimpleStatus)\",\n \"type\": \"Expression\"\n },\n \"activities\": [\n {\n \"name\": \"Get Pipeline Status\",\n \"description\": \"Checks the status of a given processing pipeline and provides the value for the downstream framework activities to act upon.\",\n \"type\": \"AzureFunctionActivity\",\n \"dependsOn\": [],\n \"policy\": {\n \"timeout\": \"7.00:00:00\",\n \"retry\": 0,\n \"retryIntervalInSeconds\": 30,\n \"secureOutput\": false,\n \"secureInput\": false\n },\n \"userProperties\": [],\n \"typeProperties\": {\n \"functionName\": \"CheckPipelineStatus\",\n \"method\": \"POST\",\n \"headers\": {},\n \"body\": {\n \"value\": \"@concat('\\n{\\n \\\"tenantId\\\": \\\"',pipeline().parameters.TenantId,'\\\",\\n \\\"applicationId\\\": \\\"',pipeline().parameters.applicationId,'\\\",\\n \\\"authenticationKey\\\": \\\"',pipeline().parameters.authenticationKey,'\\\",\\n \\\"subscriptionId\\\": \\\"',pipeline().parameters.subscriptionId,'\\\",\\n \\\"resourceGroup\\\": \\\"',pipeline().parameters.resourceGroup,'\\\",\\n \\\"factoryName\\\": \\\"',pipeline().parameters.factoryName,'\\\",\\n \\\"pipelineName\\\": \\\"',pipeline().parameters.pipelineName,'\\\",\\n \\\"runId\\\": \\\"',pipeline().parameters.runId,'\\\"\\n}')\",\n \"type\": \"Expression\"\n }\n },\n \"linkedServiceName\": {\n \"referenceName\": \"FrameworkFunctions\",\n \"type\": \"LinkedServiceReference\"\n }\n },\n {\n \"name\": \"Wait If Running\",\n \"description\": \"True = Do nothing.\\nFalse = Wait.\",\n \"type\": \"IfCondition\",\n \"dependsOn\": [\n {\n \"activity\": \"Get Pipeline Status\",\n \"dependencyConditions\": [\n \"Succeeded\"\n ]\n }\n ],\n \"userProperties\": [],\n \"typeProperties\": {\n \"expression\": {\n \"value\": \"@equals('Done',activity('Get Pipeline Status').output.SimpleStatus)\",\n \"type\": \"Expression\"\n },\n \"ifFalseActivities\": [\n {\n \"name\": \"Wait for Pipeline\",\n \"description\": \"The processing pipeline is still running so Wait before checking its status again.\",\n \"type\": \"Wait\",\n \"dependsOn\": [],\n \"userProperties\": [],\n \"typeProperties\": {\n \"waitTimeInSeconds\": {\n \"value\": \"@activity('Get Wait Duration').output.firstRow.PropertyValue\",\n \"type\": \"Expression\"\n }\n }\n }\n ]\n }\n },\n {\n \"name\": \"Set Last Check DateTime\",\n \"type\": \"SqlServerStoredProcedure\",\n \"dependsOn\": [\n {\n \"activity\": \"Get Pipeline Status\",\n \"dependencyConditions\": [\n \"Succeeded\"\n ]\n }\n ],\n \"policy\": {\n \"timeout\": \"7.00:00:00\",\n \"retry\": 0,\n \"retryIntervalInSeconds\": 30,\n \"secureOutput\": false,\n \"secureInput\": false\n },\n \"userProperties\": [],\n \"typeProperties\": {\n \"storedProcedureName\": \"[[procfwk].[SetLogPipelineLastStatusCheck]\",\n \"storedProcedureParameters\": {\n \"ExecutionId\": {\n \"value\": {\n \"value\": \"@pipeline().parameters.executionId\",\n \"type\": \"Expression\"\n },\n \"type\": \"Guid\"\n },\n \"PipelineId\": {\n \"value\": {\n \"value\": \"@pipeline().parameters.pipelineId\",\n \"type\": \"Expression\"\n },\n \"type\": \"Int32\"\n },\n \"StageId\": {\n \"value\": {\n \"value\": \"@pipeline().parameters.stageId\",\n \"type\": \"Expression\"\n },\n \"type\": \"Int32\"\n }\n }\n },\n \"linkedServiceName\": {\n \"referenceName\": \"SupportDatabase\",\n \"type\": \"LinkedServiceReference\"\n }\n },\n {\n \"name\": \"Log Activity Failure\",\n \"type\": \"SqlServerStoredProcedure\",\n \"dependsOn\": [\n {\n \"activity\": \"Get Pipeline Status\",\n \"dependencyConditions\": [\n \"Failed\"\n ]\n }\n ],\n \"policy\": {\n \"timeout\": \"7.00:00:00\",\n \"retry\": 0,\n \"retryIntervalInSeconds\": 30,\n \"secureOutput\": false,\n \"secureInput\": false\n },\n \"userProperties\": [],\n \"typeProperties\": {\n \"storedProcedureName\": \"[[procfwk].[SetLogActivityFailed]\",\n \"storedProcedureParameters\": {\n \"CallingActivity\": {\n \"value\": \"GetPipelineStatus\",\n \"type\": \"String\"\n },\n \"ExecutionId\": {\n \"value\": {\n \"value\": \"@pipeline().parameters.executionId\",\n \"type\": \"Expression\"\n },\n \"type\": \"Guid\"\n },\n \"PipelineId\": {\n \"value\": {\n \"value\": \"@pipeline().parameters.pipelineId\",\n \"type\": \"Expression\"\n },\n \"type\": \"Int32\"\n },\n \"StageId\": {\n \"value\": {\n \"value\": \"@pipeline().parameters.stageId\",\n \"type\": \"Expression\"\n },\n \"type\": \"Int32\"\n }\n }\n },\n \"linkedServiceName\": {\n \"referenceName\": \"SupportDatabase\",\n \"type\": \"LinkedServiceReference\"\n }\n }\n ],\n \"timeout\": \"7.00:00:00\"\n }\n },\n {\n \"name\": \"Set Pipeline Result\",\n \"description\": \"Receives the outcome from the function execution for a given processing pipeline and updates the current execution table with different pipelines status values depending on the result (case).\",\n \"type\": \"Switch\",\n \"dependsOn\": [\n {\n \"activity\": \"Wait Until Pipeline Completes\",\n \"dependencyConditions\": [\n \"Completed\"\n ]\n }\n ],\n \"userProperties\": [],\n \"typeProperties\": {\n \"on\": {\n \"value\": \"@activity('Get Pipeline Status').output.Status\",\n \"type\": \"Expression\"\n },\n \"cases\": [\n {\n \"value\": \"Succeeded\",\n \"activities\": [\n {\n \"name\": \"Pipeline Status Succeeded\",\n \"description\": \"Updates the current execution table with a pipeline status of success if the function outcome is succeeded.\",\n \"type\": \"SqlServerStoredProcedure\",\n \"dependsOn\": [],\n \"policy\": {\n \"timeout\": \"7.00:00:00\",\n \"retry\": 0,\n \"retryIntervalInSeconds\": 30,\n \"secureOutput\": false,\n \"secureInput\": false\n },\n \"userProperties\": [],\n \"typeProperties\": {\n \"storedProcedureName\": \"[[procfwk].[SetLogPipelineSuccess]\",\n \"storedProcedureParameters\": {\n \"ExecutionId\": {\n \"value\": {\n \"value\": \"@pipeline().parameters.executionId\",\n \"type\": \"Expression\"\n },\n \"type\": \"Guid\"\n },\n \"PipelineId\": {\n \"value\": {\n \"value\": \"@pipeline().parameters.pipelineId\",\n \"type\": \"Expression\"\n },\n \"type\": \"Int32\"\n },\n \"StageId\": {\n \"value\": {\n \"value\": \"@pipeline().parameters.stageId\",\n \"type\": \"Expression\"\n },\n \"type\": \"Int32\"\n }\n }\n },\n \"linkedServiceName\": {\n \"referenceName\": \"SupportDatabase\",\n \"type\": \"LinkedServiceReference\"\n }\n }\n ]\n },\n {\n \"value\": \"Failed\",\n \"activities\": [\n {\n \"name\": \"Pipeline Status Failed\",\n \"description\": \"Updates the current execution table with a pipeline status of failed if the function outcome is failed. Also blocks pipelines in the downstream execution stage.\",\n \"type\": \"SqlServerStoredProcedure\",\n \"dependsOn\": [],\n \"policy\": {\n \"timeout\": \"7.00:00:00\",\n \"retry\": 0,\n \"retryIntervalInSeconds\": 30,\n \"secureOutput\": false,\n \"secureInput\": false\n },\n \"userProperties\": [],\n \"typeProperties\": {\n \"storedProcedureName\": \"[[procfwk].[SetLogPipelineFailed]\",\n \"storedProcedureParameters\": {\n \"ExecutionId\": {\n \"value\": {\n \"value\": \"@pipeline().parameters.executionId\",\n \"type\": \"Expression\"\n },\n \"type\": \"Guid\"\n },\n \"PipelineId\": {\n \"value\": {\n \"value\": \"@pipeline().parameters.pipelineId\",\n \"type\": \"Expression\"\n },\n \"type\": \"Int32\"\n },\n \"RunId\": {\n \"value\": {\n \"value\": \"@activity('Get Pipeline Status').output.RunId\",\n \"type\": \"Expression\"\n },\n \"type\": \"Guid\"\n },\n \"StageId\": {\n \"value\": {\n \"value\": \"@pipeline().parameters.stageId\",\n \"type\": \"Expression\"\n },\n \"type\": \"Int32\"\n }\n }\n },\n \"linkedServiceName\": {\n \"referenceName\": \"SupportDatabase\",\n \"type\": \"LinkedServiceReference\"\n }\n },\n {\n \"name\": \"Get Error Details\",\n \"description\": \"Get the activity error details for the run ID of the worker pipeline called. Returns an array of all errors.\",\n \"type\": \"AzureFunctionActivity\",\n \"dependsOn\": [],\n \"policy\": {\n \"timeout\": \"7.00:00:00\",\n \"retry\": 0,\n \"retryIntervalInSeconds\": 30,\n \"secureOutput\": false,\n \"secureInput\": false\n },\n \"userProperties\": [],\n \"typeProperties\": {\n \"functionName\": \"GetActivityErrors\",\n \"method\": \"POST\",\n \"headers\": {},\n \"body\": {\n \"value\": \"@concat('\\n{\\n \\\"tenantId\\\": \\\"',pipeline().parameters.TenantId,'\\\",\\n \\\"applicationId\\\": \\\"',pipeline().parameters.applicationId,'\\\",\\n \\\"authenticationKey\\\": \\\"',pipeline().parameters.authenticationKey,'\\\",\\n \\\"subscriptionId\\\": \\\"',pipeline().parameters.subscriptionId,'\\\",\\n \\\"resourceGroup\\\": \\\"',pipeline().parameters.resourceGroup,'\\\",\\n \\\"factoryName\\\": \\\"',pipeline().parameters.factoryName,'\\\",\\n \\\"pipelineName\\\": \\\"',pipeline().parameters.pipelineName,'\\\",\\n \\\"runId\\\": \\\"',pipeline().parameters.runId,'\\\"\\n}')\",\n \"type\": \"Expression\"\n }\n },\n \"linkedServiceName\": {\n \"referenceName\": \"FrameworkFunctions\",\n \"type\": \"LinkedServiceReference\"\n }\n },\n {\n \"name\": \"Log Error Details\",\n \"description\": \"Parses pipeline error details and persists them to the metadata database error log table.\",\n \"type\": \"SqlServerStoredProcedure\",\n \"dependsOn\": [\n {\n \"activity\": \"Get Error Details\",\n \"dependencyConditions\": [\n \"Succeeded\"\n ]\n }\n ],\n \"policy\": {\n \"timeout\": \"7.00:00:00\",\n \"retry\": 0,\n \"retryIntervalInSeconds\": 30,\n \"secureOutput\": false,\n \"secureInput\": false\n },\n \"userProperties\": [],\n \"typeProperties\": {\n \"storedProcedureName\": \"[[procfwk].[SetErrorLogDetails]\",\n \"storedProcedureParameters\": {\n \"JsonErrorDetails\": {\n \"value\": {\n \"value\": \"@string(activity('Get Error Details').output)\",\n \"type\": \"Expression\"\n },\n \"type\": \"String\"\n },\n \"LocalExecutionId\": {\n \"value\": {\n \"value\": \"@pipeline().parameters.executionId\",\n \"type\": \"Expression\"\n },\n \"type\": \"Guid\"\n }\n }\n },\n \"linkedServiceName\": {\n \"referenceName\": \"SupportDatabase\",\n \"type\": \"LinkedServiceReference\"\n }\n }\n ]\n },\n {\n \"value\": \"Cancelled\",\n \"activities\": [\n {\n \"name\": \"Pipeline Status Cancelled\",\n \"description\": \"Updates the current execution table with a pipeline status of unknown if the function returns an unexpected outcome.\",\n \"type\": \"SqlServerStoredProcedure\",\n \"dependsOn\": [],\n \"policy\": {\n \"timeout\": \"7.00:00:00\",\n \"retry\": 0,\n \"retryIntervalInSeconds\": 30,\n \"secureOutput\": false,\n \"secureInput\": false\n },\n \"userProperties\": [],\n \"typeProperties\": {\n \"storedProcedureName\": \"[[procfwk].[SetLogPipelineCancelled]\",\n \"storedProcedureParameters\": {\n \"ExecutionId\": {\n \"value\": {\n \"value\": \"@pipeline().parameters.executionId\",\n \"type\": \"Expression\"\n },\n \"type\": \"Guid\"\n },\n \"PipelineId\": {\n \"value\": {\n \"value\": \"@pipeline().parameters.pipelineId\",\n \"type\": \"Expression\"\n },\n \"type\": \"Int32\"\n },\n \"StageId\": {\n \"value\": {\n \"value\": \"@pipeline().parameters.stageId\",\n \"type\": \"Expression\"\n },\n \"type\": \"Int32\"\n }\n }\n },\n \"linkedServiceName\": {\n \"referenceName\": \"SupportDatabase\",\n \"type\": \"LinkedServiceReference\"\n }\n }\n ]\n }\n ],\n \"defaultActivities\": [\n {\n \"name\": \"Pipeline Status Unknown\",\n \"description\": \"Updates the current execution table with a pipeline status of unknown if the function returns an unexpected outcome.\",\n \"type\": \"SqlServerStoredProcedure\",\n \"dependsOn\": [],\n \"policy\": {\n \"timeout\": \"7.00:00:00\",\n \"retry\": 0,\n \"retryIntervalInSeconds\": 30,\n \"secureOutput\": false,\n \"secureInput\": false\n },\n \"userProperties\": [],\n \"typeProperties\": {\n \"storedProcedureName\": \"[[procfwk].[SetLogPipelineUnknown]\",\n \"storedProcedureParameters\": {\n \"ExecutionId\": {\n \"value\": {\n \"value\": \"@pipeline().parameters.executionId\",\n \"type\": \"Expression\"\n },\n \"type\": \"Guid\"\n },\n \"PipelineId\": {\n \"value\": {\n \"value\": \"@pipeline().parameters.pipelineId\",\n \"type\": \"Expression\"\n },\n \"type\": \"Int32\"\n },\n \"StageId\": {\n \"value\": {\n \"value\": \"@pipeline().parameters.stageId\",\n \"type\": \"Expression\"\n },\n \"type\": \"Int32\"\n }\n }\n },\n \"linkedServiceName\": {\n \"referenceName\": \"SupportDatabase\",\n \"type\": \"LinkedServiceReference\"\n }\n }\n ]\n }\n },\n {\n \"name\": \"Get Wait Duration\",\n \"type\": \"Lookup\",\n \"dependsOn\": [],\n \"policy\": {\n \"timeout\": \"7.00:00:00\",\n \"retry\": 0,\n \"retryIntervalInSeconds\": 30,\n \"secureOutput\": false,\n \"secureInput\": false\n },\n \"userProperties\": [],\n \"typeProperties\": {\n \"source\": {\n \"type\": \"AzureSqlSource\",\n \"sqlReaderStoredProcedureName\": \"[[procfwk].[GetPropertyValue]\",\n \"storedProcedureParameters\": {\n \"PropertyName\": {\n \"type\": \"String\",\n \"value\": \"PipelineStatusCheckDuration\"\n }\n },\n \"queryTimeout\": \"02:00:00\"\n },\n \"dataset\": {\n \"referenceName\": \"GetSetMetadata\",\n \"type\": \"DatasetReference\",\n \"parameters\": {}\n }\n }\n }\n ],\n \"parameters\": {\n \"tenantId\": {\n \"type\": \"string\"\n },\n \"applicationId\": {\n \"type\": \"string\"\n },\n \"authenticationKey\": {\n \"type\": \"string\"\n },\n \"subscriptionId\": {\n \"type\": \"string\"\n },\n \"resourceGroup\": {\n \"type\": \"string\"\n },\n \"factoryName\": {\n \"type\": \"string\"\n },\n \"pipelineName\": {\n \"type\": \"string\"\n },\n \"runId\": {\n \"type\": \"string\"\n },\n \"executionId\": {\n \"type\": \"string\"\n },\n \"stageId\": {\n \"type\": \"int\"\n },\n \"pipelineId\": {\n \"type\": \"int\"\n }\n },\n \"folder\": {\n \"name\": \"_ProcFwk\"\n },\n \"annotations\": [\n \"ADF.procfwk\"\n ]\n },\n \"dependsOn\": [\n \"[concat(variables('factoryId'), '/datasets/GetSetMetadata')]\",\n \"[concat(variables('factoryId'), '/linkedServices/FrameworkFunctions')]\",\n \"[concat(variables('factoryId'), '/linkedServices/SupportDatabase')]\"\n ]\n },\n {\n \"name\": \"[concat(parameters('factoryName'), '/Intentional Error')]\",\n \"type\": \"Microsoft.DataFactory/factories/pipelines\",\n \"apiVersion\": \"2018-06-01\",\n \"properties\": {\n \"description\": \"Used just so the ADF.procfwk has something to call during development.\",\n \"activities\": [\n {\n \"name\": \"Wait1\",\n \"type\": \"Wait\",\n \"dependsOn\": [],\n \"userProperties\": [],\n \"typeProperties\": {\n \"waitTimeInSeconds\": {\n \"value\": \"@pipeline().parameters.WaitTime\",\n \"type\": \"Expression\"\n }\n }\n },\n {\n \"name\": \"Raise Errors or Not\",\n \"type\": \"IfCondition\",\n \"dependsOn\": [\n {\n \"activity\": \"Wait1\",\n \"dependencyConditions\": [\n \"Succeeded\"\n ]\n }\n ],\n \"userProperties\": [],\n \"typeProperties\": {\n \"expression\": {\n \"value\": \"@equals(pipeline().parameters.RaiseErrors,'true')\",\n \"type\": \"Expression\"\n },\n \"ifTrueActivities\": [\n {\n \"name\": \"Call Fail Procedure\",\n \"type\": \"SqlServerStoredProcedure\",\n \"dependsOn\": [],\n \"policy\": {\n \"timeout\": \"7.00:00:00\",\n \"retry\": 0,\n \"retryIntervalInSeconds\": 30,\n \"secureOutput\": false,\n \"secureInput\": false\n },\n \"userProperties\": [],\n \"typeProperties\": {\n \"storedProcedureName\": \"[[dbo].[FailProcedure]\",\n \"storedProcedureParameters\": {\n \"RaiseError\": {\n \"value\": {\n \"value\": \"@pipeline().parameters.RaiseErrors\",\n \"type\": \"Expression\"\n },\n \"type\": \"String\"\n }\n }\n },\n \"linkedServiceName\": {\n \"referenceName\": \"SupportDatabase\",\n \"type\": \"LinkedServiceReference\"\n }\n }\n ]\n }\n }\n ],\n \"parameters\": {\n \"RaiseErrors\": {\n \"type\": \"string\",\n \"defaultValue\": \"false\"\n },\n \"WaitTime\": {\n \"type\": \"int\",\n \"defaultValue\": 5\n }\n },\n \"folder\": {\n \"name\": \"_Workers\"\n },\n \"annotations\": [\n \"Worker\"\n ]\n },\n \"dependsOn\": [\n \"[concat(variables('factoryId'), '/linkedServices/SupportDatabase')]\"\n ]\n },\n {\n \"name\": \"[concat(parameters('factoryName'), '/Wait 1')]\",\n \"type\": \"Microsoft.DataFactory/factories/pipelines\",\n \"apiVersion\": \"2018-06-01\",\n \"properties\": {\n \"description\": \"Used just so the ADF.procfwk has something to call during development.\",\n \"activities\": [\n {\n \"name\": \"Wait1\",\n \"type\": \"Wait\",\n \"dependsOn\": [],\n \"userProperties\": [],\n \"typeProperties\": {\n \"waitTimeInSeconds\": {\n \"value\": \"@pipeline().parameters.WaitTime\",\n \"type\": \"Expression\"\n }\n }\n }\n ],\n \"parameters\": {\n \"WaitTime\": {\n \"type\": \"int\",\n \"defaultValue\": 5\n }\n },\n \"folder\": {\n \"name\": \"_Workers\"\n },\n \"annotations\": [\n \"Worker\"\n ]\n },\n \"dependsOn\": []\n },\n {\n \"name\": \"[concat(parameters('factoryName'), '/Wait 10')]\",\n \"type\": \"Microsoft.DataFactory/factories/pipelines\",\n \"apiVersion\": \"2018-06-01\",\n \"properties\": {\n \"description\": \"Used just so the ADF.procfwk has something to call during development.\",\n \"activities\": [\n {\n \"name\": \"Wait10\",\n \"type\": \"Wait\",\n \"dependsOn\": [],\n \"userProperties\": [],\n \"typeProperties\": {\n \"waitTimeInSeconds\": {\n \"value\": \"@pipeline().parameters.WaitTime\",\n \"type\": \"Expression\"\n }\n }\n }\n ],\n \"parameters\": {\n \"WaitTime\": {\n \"type\": \"int\",\n \"defaultValue\": 5\n }\n },\n \"folder\": {\n \"name\": \"_Workers\"\n },\n \"annotations\": [\n \"Worker\"\n ]\n },\n \"dependsOn\": []\n },\n {\n \"name\": \"[concat(parameters('factoryName'), '/Wait 2')]\",\n \"type\": \"Microsoft.DataFactory/factories/pipelines\",\n \"apiVersion\": \"2018-06-01\",\n \"properties\": {\n \"description\": \"Used just so the ADF.procfwk has something to call during development.\",\n \"activities\": [\n {\n \"name\": \"Wait2\",\n \"type\": \"Wait\",\n \"dependsOn\": [],\n \"userProperties\": [],\n \"typeProperties\": {\n \"waitTimeInSeconds\": {\n \"value\": \"@pipeline().parameters.WaitTime\",\n \"type\": \"Expression\"\n }\n }\n }\n ],\n \"parameters\": {\n \"WaitTime\": {\n \"type\": \"int\",\n \"defaultValue\": 5\n }\n },\n \"folder\": {\n \"name\": \"_Workers\"\n },\n \"annotations\": [\n \"Worker\"\n ]\n },\n \"dependsOn\": []\n },\n {\n \"name\": \"[concat(parameters('factoryName'), '/Wait 3')]\",\n \"type\": \"Microsoft.DataFactory/factories/pipelines\",\n \"apiVersion\": \"2018-06-01\",\n \"properties\": {\n \"description\": \"Used just so the ADF.procfwk has something to call during development.\",\n \"activities\": [\n {\n \"name\": \"Wait3\",\n \"type\": \"Wait\",\n \"dependsOn\": [],\n \"userProperties\": [],\n \"typeProperties\": {\n \"waitTimeInSeconds\": {\n \"value\": \"@pipeline().parameters.WaitTime\",\n \"type\": \"Expression\"\n }\n }\n }\n ],\n \"parameters\": {\n \"WaitTime\": {\n \"type\": \"int\",\n \"defaultValue\": 5\n }\n },\n \"folder\": {\n \"name\": \"_Workers\"\n },\n \"annotations\": [\n \"Worker\"\n ]\n },\n \"dependsOn\": []\n },\n {\n \"name\": \"[concat(parameters('factoryName'), '/Wait 4')]\",\n \"type\": \"Microsoft.DataFactory/factories/pipelines\",\n \"apiVersion\": \"2018-06-01\",\n \"properties\": {\n \"description\": \"Used just so the ADF.procfwk has something to call during development.\",\n \"activities\": [\n {\n \"name\": \"Wait4\",\n \"type\": \"Wait\",\n \"dependsOn\": [],\n \"userProperties\": [],\n \"typeProperties\": {\n \"waitTimeInSeconds\": {\n \"value\": \"@pipeline().parameters.WaitTime\",\n \"type\": \"Expression\"\n }\n }\n }\n ],\n \"parameters\": {\n \"WaitTime\": {\n \"type\": \"int\",\n \"defaultValue\": 5\n }\n },\n \"folder\": {\n \"name\": \"_Workers\"\n },\n \"annotations\": [\n \"Worker\"\n ]\n },\n \"dependsOn\": []\n },\n {\n \"name\": \"[concat(parameters('factoryName'), '/Wait 5')]\",\n \"type\": \"Microsoft.DataFactory/factories/pipelines\",\n \"apiVersion\": \"2018-06-01\",\n \"properties\": {\n \"description\": \"Used just so the ADF.procfwk has something to call during development.\",\n \"activities\": [\n {\n \"name\": \"Wait5\",\n \"type\": \"Wait\",\n \"dependsOn\": [],\n \"userProperties\": [],\n \"typeProperties\": {\n \"waitTimeInSeconds\": {\n \"value\": \"@pipeline().parameters.WaitTime\",\n \"type\": \"Expression\"\n }\n }\n }\n ],\n \"parameters\": {\n \"WaitTime\": {\n \"type\": \"int\",\n \"defaultValue\": 5\n }\n },\n \"folder\": {\n \"name\": \"_Workers\"\n },\n \"annotations\": [\n \"Worker\"\n ]\n },\n \"dependsOn\": []\n },\n {\n \"name\": \"[concat(parameters('factoryName'), '/Wait 6')]\",\n \"type\": \"Microsoft.DataFactory/factories/pipelines\",\n \"apiVersion\": \"2018-06-01\",\n \"properties\": {\n \"description\": \"Used just so the ADF.procfwk has something to call during development.\",\n \"activities\": [\n {\n \"name\": \"Wait6\",\n \"type\": \"Wait\",\n \"dependsOn\": [],\n \"userProperties\": [],\n \"typeProperties\": {\n \"waitTimeInSeconds\": {\n \"value\": \"@pipeline().parameters.WaitTime\",\n \"type\": \"Expression\"\n }\n }\n }\n ],\n \"parameters\": {\n \"WaitTime\": {\n \"type\": \"int\",\n \"defaultValue\": 5\n }\n },\n \"folder\": {\n \"name\": \"_Workers\"\n },\n \"annotations\": [\n \"Worker\"\n ]\n },\n \"dependsOn\": []\n },\n {\n \"name\": \"[concat(parameters('factoryName'), '/Wait 7')]\",\n \"type\": \"Microsoft.DataFactory/factories/pipelines\",\n \"apiVersion\": \"2018-06-01\",\n \"properties\": {\n \"description\": \"Used just so the ADF.procfwk has something to call during development.\",\n \"activities\": [\n {\n \"name\": \"Wait7\",\n \"type\": \"Wait\",\n \"dependsOn\": [],\n \"userProperties\": [],\n \"typeProperties\": {\n \"waitTimeInSeconds\": {\n \"value\": \"@pipeline().parameters.WaitTime\",\n \"type\": \"Expression\"\n }\n }\n }\n ],\n \"parameters\": {\n \"WaitTime\": {\n \"type\": \"int\",\n \"defaultValue\": 5\n }\n },\n \"folder\": {\n \"name\": \"_Workers\"\n },\n \"annotations\": [\n \"Worker\"\n ]\n },\n \"dependsOn\": []\n },\n {\n \"name\": \"[concat(parameters('factoryName'), '/Wait 8')]\",\n \"type\": \"Microsoft.DataFactory/factories/pipelines\",\n \"apiVersion\": \"2018-06-01\",\n \"properties\": {\n \"description\": \"Used just so the ADF.procfwk has something to call during development.\",\n \"activities\": [\n {\n \"name\": \"Wait8\",\n \"type\": \"Wait\",\n \"dependsOn\": [],\n \"userProperties\": [],\n \"typeProperties\": {\n \"waitTimeInSeconds\": {\n \"value\": \"@pipeline().parameters.WaitTime\",\n \"type\": \"Expression\"\n }\n }\n }\n ],\n \"parameters\": {\n \"WaitTime\": {\n \"type\": \"int\",\n \"defaultValue\": 5\n }\n },\n \"folder\": {\n \"name\": \"_Workers\"\n },\n \"annotations\": [\n \"Worker\"\n ]\n },\n \"dependsOn\": []\n },\n {\n \"name\": \"[concat(parameters('factoryName'), '/Wait 9')]\",\n \"type\": \"Microsoft.DataFactory/factories/pipelines\",\n \"apiVersion\": \"2018-06-01\",\n \"properties\": {\n \"description\": \"Used just so the ADF.procfwk has something to call during development.\",\n \"activities\": [\n {\n \"name\": \"Wait9\",\n \"type\": \"Wait\",\n \"dependsOn\": [],\n \"userProperties\": [],\n \"typeProperties\": {\n \"waitTimeInSeconds\": {\n \"value\": \"@pipeline().parameters.WaitTime\",\n \"type\": \"Expression\"\n }\n }\n }\n ],\n \"parameters\": {\n \"WaitTime\": {\n \"type\": \"int\",\n \"defaultValue\": 15\n }\n },\n \"folder\": {\n \"name\": \"_Workers\"\n },\n \"annotations\": [\n \"Worker\"\n ]\n },\n \"dependsOn\": []\n },\n {\n \"name\": \"[concat(parameters('factoryName'), '/GetSetMetadata')]\",\n \"type\": \"Microsoft.DataFactory/factories/datasets\",\n \"apiVersion\": \"2018-06-01\",\n \"properties\": {\n \"linkedServiceName\": {\n \"referenceName\": \"SupportDatabase\",\n \"type\": \"LinkedServiceReference\"\n },\n \"annotations\": [\n \"ADF.procfwk\"\n ],\n \"type\": \"AzureSqlTable\",\n \"schema\": [],\n \"typeProperties\": {}\n },\n \"dependsOn\": [\n \"[concat(variables('factoryId'), '/linkedServices/SupportDatabase')]\"\n ]\n },\n {\n \"name\": \"[concat(parameters('factoryName'), '/FrameworkFunctions')]\",\n \"type\": \"Microsoft.DataFactory/factories/linkedServices\",\n \"apiVersion\": \"2018-06-01\",\n \"properties\": {\n \"annotations\": [\n \"ADF.procfwk\"\n ],\n \"type\": \"AzureFunction\",\n \"typeProperties\": {\n \"functionAppUrl\": \"[parameters('FrameworkFunctions_properties_typeProperties_functionAppUrl')]\",\n \"functionKey\": {\n \"type\": \"AzureKeyVaultSecret\",\n \"store\": {\n \"referenceName\": \"Keys\",\n \"type\": \"LinkedServiceReference\"\n },\n \"secretName\": \"FrameworkFunctionsKey\"\n }\n }\n },\n \"dependsOn\": [\n \"[concat(variables('factoryId'), '/linkedServices/Keys')]\"\n ]\n },\n {\n \"name\": \"[concat(parameters('factoryName'), '/Keys')]\",\n \"type\": \"Microsoft.DataFactory/factories/linkedServices\",\n \"apiVersion\": \"2018-06-01\",\n \"properties\": {\n \"description\": \"Connection to Key Vault for all other ADF linked service credentials required to run the processing framework.\",\n \"annotations\": [\n \"ADF.procfwk\"\n ],\n \"type\": \"AzureKeyVault\",\n \"typeProperties\": {\n \"baseUrl\": \"[parameters('Keys_properties_typeProperties_baseUrl')]\"\n }\n },\n \"dependsOn\": []\n },\n {\n \"name\": \"[concat(parameters('factoryName'), '/SupportDatabase')]\",\n \"type\": \"Microsoft.DataFactory/factories/linkedServices\",\n \"apiVersion\": \"2018-06-01\",\n \"properties\": {\n \"description\": \"Connection between ADF and processing framework metadata SQLDB.\",\n \"annotations\": [\n \"ADF.procfwk\"\n ],\n \"type\": \"AzureSqlDatabase\",\n \"typeProperties\": {\n \"connectionString\": {\n \"type\": \"AzureKeyVaultSecret\",\n \"store\": {\n \"referenceName\": \"Keys\",\n \"type\": \"LinkedServiceReference\"\n },\n \"secretName\": \"[parameters('SupportDatabase_properties_typeProperties_connectionString_secretName')]\"\n }\n }\n },\n \"dependsOn\": [\n \"[concat(variables('factoryId'), '/linkedServices/Keys')]\"\n ]\n },\n {\n \"name\": \"[concat(parameters('factoryName'), '/FunctionalTestingTrigger')]\",\n \"type\": \"Microsoft.DataFactory/factories/triggers\",\n \"apiVersion\": \"2018-06-01\",\n \"properties\": {\n \"description\": \"Used for functional testing of the framework in a dedicated environment.\",\n \"annotations\": [\n \"ADF.procfwk\"\n ],\n \"runtimeState\": \"Stopped\",\n \"pipelines\": [\n {\n \"pipelineReference\": {\n \"referenceName\": \"01-Grandparent\",\n \"type\": \"PipelineReference\"\n },\n \"parameters\": {}\n }\n ],\n \"type\": \"ScheduleTrigger\",\n \"typeProperties\": {\n \"recurrence\": {\n \"frequency\": \"Hour\",\n \"interval\": 2,\n \"startTime\": \"2020-04-06T15:00:00.000Z\",\n \"timeZone\": \"UTC\"\n }\n }\n },\n \"dependsOn\": [\n \"[concat(variables('factoryId'), '/pipelines/01-Grandparent')]\"\n ]\n }\n ]\n}"
},
{
"path": "ARM Templates/Data Factory/v1.8.5 Export.json",
"content": "{\n \"$schema\": \"http://schema.management.azure.com/schemas/2015-01-01/deploymentTemplate.json#\",\n \"contentVersion\": \"1.0.0.0\",\n \"parameters\": {\n \"factoryName\": {\n \"type\": \"string\",\n \"metadata\": \"Data Factory name\",\n \"defaultValue\": \"\"\n },\n \"Keys_properties_typeProperties_baseUrl\": {\n \"type\": \"string\",\n \"defaultValue\": \"\"\n },\n \"FrameworkFunctions_properties_typeProperties_functionAppUrl\": {\n \"type\": \"string\",\n \"defaultValue\": \"\"\n },\n \"SupportDatabase_properties_typeProperties_connectionString_secretName\": {\n \"type\": \"string\",\n \"defaultValue\": \"\"\n }\n },\n \"variables\": {\n \"factoryId\": \"[concat('Microsoft.DataFactory/factories/', parameters('factoryName'))]\"\n },\n \"resources\": [\n {\n \"name\": \"[concat(parameters('factoryName'), '/GetSetMetadata')]\",\n \"type\": \"Microsoft.DataFactory/factories/datasets\",\n \"apiVersion\": \"2018-06-01\",\n \"properties\": {\n \"linkedServiceName\": {\n \"referenceName\": \"SupportDatabase\",\n \"type\": \"LinkedServiceReference\"\n },\n \"annotations\": [\n \"ADF.procfwk\"\n ],\n \"type\": \"AzureSqlTable\",\n \"schema\": [],\n \"typeProperties\": {}\n },\n \"dependsOn\": [\n \"[concat(variables('factoryId'), '/linkedServices/SupportDatabase')]\"\n ]\n },\n {\n \"name\": \"[concat(parameters('factoryName'), '/FunctionalTestingTrigger')]\",\n \"type\": \"Microsoft.DataFactory/factories/triggers\",\n \"apiVersion\": \"2018-06-01\",\n \"properties\": {\n \"description\": \"Used for functional testing of the framework in a dedicated environment.\",\n \"annotations\": [\n \"ADF.procfwk\"\n ],\n \"runtimeState\": \"Stopped\",\n \"pipelines\": [\n {\n \"pipelineReference\": {\n \"referenceName\": \"01-Grandparent\",\n \"type\": \"PipelineReference\"\n },\n \"parameters\": {}\n }\n ],\n \"type\": \"ScheduleTrigger\",\n \"typeProperties\": {\n \"recurrence\": {\n \"frequency\": \"Hour\",\n \"interval\": 2,\n \"startTime\": \"2020-04-06T15:00:00Z\",\n \"timeZone\": \"UTC\"\n }\n }\n },\n \"dependsOn\": [\n \"[concat(variables('factoryId'), '/pipelines/01-Grandparent')]\"\n ]\n },\n {\n \"name\": \"[concat(parameters('factoryName'), '/Keys')]\",\n \"type\": \"Microsoft.DataFactory/factories/linkedServices\",\n \"apiVersion\": \"2018-06-01\",\n \"properties\": {\n \"description\": \"Connection to Key Vault for all other ADF linked service credentials required to run the processing framework.\",\n \"annotations\": [\n \"ADF.procfwk\"\n ],\n \"type\": \"AzureKeyVault\",\n \"typeProperties\": {\n \"baseUrl\": \"[parameters('Keys_properties_typeProperties_baseUrl')]\"\n }\n },\n \"dependsOn\": []\n },\n {\n \"name\": \"[concat(parameters('factoryName'), '/FrameworkFunctions')]\",\n \"type\": \"Microsoft.DataFactory/factories/linkedServices\",\n \"apiVersion\": \"2018-06-01\",\n \"properties\": {\n \"annotations\": [\n \"ADF.procfwk\"\n ],\n \"type\": \"AzureFunction\",\n \"typeProperties\": {\n \"functionAppUrl\": \"[parameters('FrameworkFunctions_properties_typeProperties_functionAppUrl')]\",\n \"functionKey\": {\n \"type\": \"AzureKeyVaultSecret\",\n \"store\": {\n \"referenceName\": \"Keys\",\n \"type\": \"LinkedServiceReference\"\n },\n \"secretName\": \"FrameworkFunctionsKey\"\n }\n }\n },\n \"dependsOn\": [\n \"[concat(variables('factoryId'), '/linkedServices/Keys')]\"\n ]\n },\n {\n \"name\": \"[concat(parameters('factoryName'), '/SupportDatabase')]\",\n \"type\": \"Microsoft.DataFactory/factories/linkedServices\",\n \"apiVersion\": \"2018-06-01\",\n \"properties\": {\n \"description\": \"Connection between ADF and processing framework metadata SQLDB.\",\n \"annotations\": [\n \"ADF.procfwk\"\n ],\n \"type\": \"AzureSqlDatabase\",\n \"typeProperties\": {\n \"connectionString\": {\n \"type\": \"AzureKeyVaultSecret\",\n \"store\": {\n \"referenceName\": \"Keys\",\n \"type\": \"LinkedServiceReference\"\n },\n \"secretName\": \"[parameters('SupportDatabase_properties_typeProperties_connectionString_secretName')]\"\n }\n }\n },\n \"dependsOn\": [\n \"[concat(variables('factoryId'), '/linkedServices/Keys')]\"\n ]\n },\n {\n \"name\": \"[concat(parameters('factoryName'), '/04-Infant')]\",\n \"type\": \"Microsoft.DataFactory/factories/pipelines\",\n \"apiVersion\": \"2018-06-01\",\n \"properties\": {\n \"description\": \"ADF.procfwk infant pipeline used to check when the processing pipeline called by the Child completes and passes the resulting status back to the metadata database.\",\n \"activities\": [\n {\n \"name\": \"Wait Until Pipeline Completes\",\n \"description\": \"Loops until the pipeline called completes.\\n\\nSimple status:\\n- Running = new iteration.\\n- Done = break.\",\n \"type\": \"Until\",\n \"dependsOn\": [\n {\n \"activity\": \"Get Wait Duration\",\n \"dependencyConditions\": [\n \"Succeeded\"\n ]\n }\n ],\n \"userProperties\": [],\n \"typeProperties\": {\n \"expression\": {\n \"value\": \"@equals('Done',activity('Get Pipeline Status').output.SimpleStatus)\",\n \"type\": \"Expression\"\n },\n \"activities\": [\n {\n \"name\": \"Get Pipeline Status\",\n \"description\": \"Checks the status of a given processing pipeline and provides the value for the downstream framework activities to act upon.\",\n \"type\": \"AzureFunctionActivity\",\n \"dependsOn\": [],\n \"policy\": {\n \"timeout\": \"7.00:00:00\",\n \"retry\": 0,\n \"retryIntervalInSeconds\": 30,\n \"secureOutput\": false,\n \"secureInput\": false\n },\n \"userProperties\": [],\n \"typeProperties\": {\n \"functionName\": \"CheckPipelineStatus\",\n \"method\": \"POST\",\n \"headers\": {},\n \"body\": {\n \"value\": \"@concat('\\n{\\n \\\"tenantId\\\": \\\"',pipeline().parameters.TenantId,'\\\",\\n \\\"applicationId\\\": \\\"',pipeline().parameters.applicationId,'\\\",\\n \\\"authenticationKey\\\": \\\"',pipeline().parameters.authenticationKey,'\\\",\\n \\\"subscriptionId\\\": \\\"',pipeline().parameters.subscriptionId,'\\\",\\n \\\"resourceGroup\\\": \\\"',pipeline().parameters.resourceGroup,'\\\",\\n \\\"factoryName\\\": \\\"',pipeline().parameters.factoryName,'\\\",\\n \\\"pipelineName\\\": \\\"',pipeline().parameters.pipelineName,'\\\",\\n \\\"runId\\\": \\\"',pipeline().parameters.runId,'\\\"\\n}')\",\n \"type\": \"Expression\"\n }\n },\n \"linkedServiceName\": {\n \"referenceName\": \"FrameworkFunctions\",\n \"type\": \"LinkedServiceReference\"\n }\n },\n {\n \"name\": \"Wait If Running\",\n \"description\": \"True = Do nothing.\\nFalse = Wait.\",\n \"type\": \"IfCondition\",\n \"dependsOn\": [\n {\n \"activity\": \"Get Pipeline Status\",\n \"dependencyConditions\": [\n \"Succeeded\"\n ]\n }\n ],\n \"userProperties\": [],\n \"typeProperties\": {\n \"expression\": {\n \"value\": \"@equals('Done',activity('Get Pipeline Status').output.SimpleStatus)\",\n \"type\": \"Expression\"\n },\n \"ifFalseActivities\": [\n {\n \"name\": \"Wait for Pipeline\",\n \"description\": \"The processing pipeline is still running so Wait before checking its status again.\",\n \"type\": \"Wait\",\n \"dependsOn\": [],\n \"userProperties\": [],\n \"typeProperties\": {\n \"waitTimeInSeconds\": {\n \"value\": \"@activity('Get Wait Duration').output.firstRow.PropertyValue\",\n \"type\": \"Expression\"\n }\n }\n }\n ]\n }\n },\n {\n \"name\": \"Set Last Check DateTime\",\n \"type\": \"SqlServerStoredProcedure\",\n \"dependsOn\": [\n {\n \"activity\": \"Get Pipeline Status\",\n \"dependencyConditions\": [\n \"Succeeded\"\n ]\n }\n ],\n \"policy\": {\n \"timeout\": \"7.00:00:00\",\n \"retry\": 0,\n \"retryIntervalInSeconds\": 30,\n \"secureOutput\": false,\n \"secureInput\": false\n },\n \"userProperties\": [],\n \"typeProperties\": {\n \"storedProcedureName\": \"[[procfwk].[SetLogPipelineLastStatusCheck]\",\n \"storedProcedureParameters\": {\n \"ExecutionId\": {\n \"value\": {\n \"value\": \"@pipeline().parameters.executionId\",\n \"type\": \"Expression\"\n },\n \"type\": \"Guid\"\n },\n \"PipelineId\": {\n \"value\": {\n \"value\": \"@pipeline().parameters.pipelineId\",\n \"type\": \"Expression\"\n },\n \"type\": \"Int32\"\n },\n \"StageId\": {\n \"value\": {\n \"value\": \"@pipeline().parameters.stageId\",\n \"type\": \"Expression\"\n },\n \"type\": \"Int32\"\n }\n }\n },\n \"linkedServiceName\": {\n \"referenceName\": \"SupportDatabase\",\n \"type\": \"LinkedServiceReference\"\n }\n },\n {\n \"name\": \"Log Activity Failure\",\n \"type\": \"SqlServerStoredProcedure\",\n \"dependsOn\": [\n {\n \"activity\": \"Get Pipeline Status\",\n \"dependencyConditions\": [\n \"Failed\"\n ]\n }\n ],\n \"policy\": {\n \"timeout\": \"7.00:00:00\",\n \"retry\": 0,\n \"retryIntervalInSeconds\": 30,\n \"secureOutput\": false,\n \"secureInput\": false\n },\n \"userProperties\": [],\n \"typeProperties\": {\n \"storedProcedureName\": \"[[procfwk].[SetLogActivityFailed]\",\n \"storedProcedureParameters\": {\n \"CallingActivity\": {\n \"value\": \"GetPipelineStatus\",\n \"type\": \"String\"\n },\n \"ExecutionId\": {\n \"value\": {\n \"value\": \"@pipeline().parameters.executionId\",\n \"type\": \"Expression\"\n },\n \"type\": \"Guid\"\n },\n \"PipelineId\": {\n \"value\": {\n \"value\": \"@pipeline().parameters.pipelineId\",\n \"type\": \"Expression\"\n },\n \"type\": \"Int32\"\n },\n \"StageId\": {\n \"value\": {\n \"value\": \"@pipeline().parameters.stageId\",\n \"type\": \"Expression\"\n },\n \"type\": \"Int32\"\n }\n }\n },\n \"linkedServiceName\": {\n \"referenceName\": \"SupportDatabase\",\n \"type\": \"LinkedServiceReference\"\n }\n }\n ],\n \"timeout\": \"7.00:00:00\"\n }\n },\n {\n \"name\": \"Set Pipeline Result\",\n \"description\": \"Receives the outcome from the function execution for a given processing pipeline and updates the current execution table with different pipelines status values depending on the result (case).\",\n \"type\": \"Switch\",\n \"dependsOn\": [\n {\n \"activity\": \"Wait Until Pipeline Completes\",\n \"dependencyConditions\": [\n \"Completed\"\n ]\n }\n ],\n \"userProperties\": [],\n \"typeProperties\": {\n \"on\": {\n \"value\": \"@activity('Get Pipeline Status').output.Status\",\n \"type\": \"Expression\"\n },\n \"cases\": [\n {\n \"value\": \"Succeeded\",\n \"activities\": [\n {\n \"name\": \"Pipeline Status Succeeded\",\n \"description\": \"Updates the current execution table with a pipeline status of success if the function outcome is succeeded.\",\n \"type\": \"SqlServerStoredProcedure\",\n \"dependsOn\": [],\n \"policy\": {\n \"timeout\": \"7.00:00:00\",\n \"retry\": 0,\n \"retryIntervalInSeconds\": 30,\n \"secureOutput\": false,\n \"secureInput\": false\n },\n \"userProperties\": [],\n \"typeProperties\": {\n \"storedProcedureName\": \"[[procfwk].[SetLogPipelineSuccess]\",\n \"storedProcedureParameters\": {\n \"ExecutionId\": {\n \"value\": {\n \"value\": \"@pipeline().parameters.executionId\",\n \"type\": \"Expression\"\n },\n \"type\": \"Guid\"\n },\n \"PipelineId\": {\n \"value\": {\n \"value\": \"@pipeline().parameters.pipelineId\",\n \"type\": \"Expression\"\n },\n \"type\": \"Int32\"\n },\n \"StageId\": {\n \"value\": {\n \"value\": \"@pipeline().parameters.stageId\",\n \"type\": \"Expression\"\n },\n \"type\": \"Int32\"\n }\n }\n },\n \"linkedServiceName\": {\n \"referenceName\": \"SupportDatabase\",\n \"type\": \"LinkedServiceReference\"\n }\n }\n ]\n },\n {\n \"value\": \"Failed\",\n \"activities\": [\n {\n \"name\": \"Pipeline Status Failed\",\n \"description\": \"Updates the current execution table with a pipeline status of failed if the function outcome is failed. Also blocks pipelines in the downstream execution stage.\",\n \"type\": \"SqlServerStoredProcedure\",\n \"dependsOn\": [],\n \"policy\": {\n \"timeout\": \"7.00:00:00\",\n \"retry\": 0,\n \"retryIntervalInSeconds\": 30,\n \"secureOutput\": false,\n \"secureInput\": false\n },\n \"userProperties\": [],\n \"typeProperties\": {\n \"storedProcedureName\": \"[[procfwk].[SetLogPipelineFailed]\",\n \"storedProcedureParameters\": {\n \"ExecutionId\": {\n \"value\": {\n \"value\": \"@pipeline().parameters.executionId\",\n \"type\": \"Expression\"\n },\n \"type\": \"Guid\"\n },\n \"PipelineId\": {\n \"value\": {\n \"value\": \"@pipeline().parameters.pipelineId\",\n \"type\": \"Expression\"\n },\n \"type\": \"Int32\"\n },\n \"RunId\": {\n \"value\": {\n \"value\": \"@activity('Get Pipeline Status').output.RunId\",\n \"type\": \"Expression\"\n },\n \"type\": \"Guid\"\n },\n \"StageId\": {\n \"value\": {\n \"value\": \"@pipeline().parameters.stageId\",\n \"type\": \"Expression\"\n },\n \"type\": \"Int32\"\n }\n }\n },\n \"linkedServiceName\": {\n \"referenceName\": \"SupportDatabase\",\n \"type\": \"LinkedServiceReference\"\n }\n },\n {\n \"name\": \"Get Error Details\",\n \"description\": \"Get the activity error details for the run ID of the worker pipeline called. Returns an array of all errors.\",\n \"type\": \"AzureFunctionActivity\",\n \"dependsOn\": [],\n \"policy\": {\n \"timeout\": \"7.00:00:00\",\n \"retry\": 0,\n \"retryIntervalInSeconds\": 30,\n \"secureOutput\": false,\n \"secureInput\": false\n },\n \"userProperties\": [],\n \"typeProperties\": {\n \"functionName\": \"GetActivityErrors\",\n \"method\": \"POST\",\n \"headers\": {},\n \"body\": {\n \"value\": \"@concat('\\n{\\n \\\"tenantId\\\": \\\"',pipeline().parameters.TenantId,'\\\",\\n \\\"applicationId\\\": \\\"',pipeline().parameters.applicationId,'\\\",\\n \\\"authenticationKey\\\": \\\"',pipeline().parameters.authenticationKey,'\\\",\\n \\\"subscriptionId\\\": \\\"',pipeline().parameters.subscriptionId,'\\\",\\n \\\"resourceGroup\\\": \\\"',pipeline().parameters.resourceGroup,'\\\",\\n \\\"factoryName\\\": \\\"',pipeline().parameters.factoryName,'\\\",\\n \\\"pipelineName\\\": \\\"',pipeline().parameters.pipelineName,'\\\",\\n \\\"runId\\\": \\\"',pipeline().parameters.runId,'\\\"\\n}')\",\n \"type\": \"Expression\"\n }\n },\n \"linkedServiceName\": {\n \"referenceName\": \"FrameworkFunctions\",\n \"type\": \"LinkedServiceReference\"\n }\n },\n {\n \"name\": \"Log Error Details\",\n \"description\": \"Parses pipeline error details and persists them to the metadata database error log table.\",\n \"type\": \"SqlServerStoredProcedure\",\n \"dependsOn\": [\n {\n \"activity\": \"Get Error Details\",\n \"dependencyConditions\": [\n \"Succeeded\"\n ]\n }\n ],\n \"policy\": {\n \"timeout\": \"7.00:00:00\",\n \"retry\": 0,\n \"retryIntervalInSeconds\": 30,\n \"secureOutput\": false,\n \"secureInput\": false\n },\n \"userProperties\": [],\n \"typeProperties\": {\n \"storedProcedureName\": \"[[procfwk].[SetErrorLogDetails]\",\n \"storedProcedureParameters\": {\n \"JsonErrorDetails\": {\n \"value\": {\n \"value\": \"@string(activity('Get Error Details').output)\",\n \"type\": \"Expression\"\n },\n \"type\": \"String\"\n },\n \"LocalExecutionId\": {\n \"value\": {\n \"value\": \"@pipeline().parameters.executionId\",\n \"type\": \"Expression\"\n },\n \"type\": \"Guid\"\n }\n }\n },\n \"linkedServiceName\": {\n \"referenceName\": \"SupportDatabase\",\n \"type\": \"LinkedServiceReference\"\n }\n }\n ]\n },\n {\n \"value\": \"Cancelled\",\n \"activities\": [\n {\n \"name\": \"Pipeline Status Cancelled\",\n \"description\": \"Updates the current execution table with a pipeline status of unknown if the function returns an unexpected outcome.\",\n \"type\": \"SqlServerStoredProcedure\",\n \"dependsOn\": [],\n \"policy\": {\n \"timeout\": \"7.00:00:00\",\n \"retry\": 0,\n \"retryIntervalInSeconds\": 30,\n \"secureOutput\": false,\n \"secureInput\": false\n },\n \"userProperties\": [],\n \"typeProperties\": {\n \"storedProcedureName\": \"[[procfwk].[SetLogPipelineCancelled]\",\n \"storedProcedureParameters\": {\n \"ExecutionId\": {\n \"value\": {\n \"value\": \"@pipeline().parameters.executionId\",\n \"type\": \"Expression\"\n },\n \"type\": \"Guid\"\n },\n \"PipelineId\": {\n \"value\": {\n \"value\": \"@pipeline().parameters.pipelineId\",\n \"type\": \"Expression\"\n },\n \"type\": \"Int32\"\n },\n \"StageId\": {\n \"value\": {\n \"value\": \"@pipeline().parameters.stageId\",\n \"type\": \"Expression\"\n },\n \"type\": \"Int32\"\n }\n }\n },\n \"linkedServiceName\": {\n \"referenceName\": \"SupportDatabase\",\n \"type\": \"LinkedServiceReference\"\n }\n }\n ]\n }\n ],\n \"defaultActivities\": [\n {\n \"name\": \"Pipeline Status Unknown\",\n \"description\": \"Updates the current execution table with a pipeline status of unknown if the function returns an unexpected outcome.\",\n \"type\": \"SqlServerStoredProcedure\",\n \"dependsOn\": [],\n \"policy\": {\n \"timeout\": \"7.00:00:00\",\n \"retry\": 0,\n \"retryIntervalInSeconds\": 30,\n \"secureOutput\": false,\n \"secureInput\": false\n },\n \"userProperties\": [],\n \"typeProperties\": {\n \"storedProcedureName\": \"[[procfwk].[SetLogPipelineUnknown]\",\n \"storedProcedureParameters\": {\n \"ExecutionId\": {\n \"value\": {\n \"value\": \"@pipeline().parameters.executionId\",\n \"type\": \"Expression\"\n },\n \"type\": \"Guid\"\n },\n \"PipelineId\": {\n \"value\": {\n \"value\": \"@pipeline().parameters.pipelineId\",\n \"type\": \"Expression\"\n },\n \"type\": \"Int32\"\n },\n \"StageId\": {\n \"value\": {\n \"value\": \"@pipeline().parameters.stageId\",\n \"type\": \"Expression\"\n },\n \"type\": \"Int32\"\n }\n }\n },\n \"linkedServiceName\": {\n \"referenceName\": \"SupportDatabase\",\n \"type\": \"LinkedServiceReference\"\n }\n }\n ]\n }\n },\n {\n \"name\": \"Get Wait Duration\",\n \"type\": \"Lookup\",\n \"dependsOn\": [],\n \"policy\": {\n \"timeout\": \"7.00:00:00\",\n \"retry\": 0,\n \"retryIntervalInSeconds\": 30,\n \"secureOutput\": false,\n \"secureInput\": false\n },\n \"userProperties\": [],\n \"typeProperties\": {\n \"source\": {\n \"type\": \"AzureSqlSource\",\n \"sqlReaderStoredProcedureName\": \"[[procfwk].[GetPropertyValue]\",\n \"storedProcedureParameters\": {\n \"PropertyName\": {\n \"type\": \"String\",\n \"value\": \"PipelineStatusCheckDuration\"\n }\n },\n \"queryTimeout\": \"02:00:00\",\n \"partitionOption\": \"None\"\n },\n \"dataset\": {\n \"referenceName\": \"GetSetMetadata\",\n \"type\": \"DatasetReference\",\n \"parameters\": {}\n }\n }\n }\n ],\n \"parameters\": {\n \"tenantId\": {\n \"type\": \"string\"\n },\n \"applicationId\": {\n \"type\": \"string\"\n },\n \"authenticationKey\": {\n \"type\": \"string\"\n },\n \"subscriptionId\": {\n \"type\": \"string\"\n },\n \"resourceGroup\": {\n \"type\": \"string\"\n },\n \"factoryName\": {\n \"type\": \"string\"\n },\n \"pipelineName\": {\n \"type\": \"string\"\n },\n \"runId\": {\n \"type\": \"string\"\n },\n \"executionId\": {\n \"type\": \"string\"\n },\n \"stageId\": {\n \"type\": \"int\"\n },\n \"pipelineId\": {\n \"type\": \"int\"\n }\n },\n \"folder\": {\n \"name\": \"_ProcFwk\"\n },\n \"annotations\": [\n \"ADF.procfwk\"\n ],\n \"lastPublishTime\": \"2020-08-17T11:35:19Z\"\n },\n \"dependsOn\": [\n \"[concat(variables('factoryId'), '/datasets/GetSetMetadata')]\",\n \"[concat(variables('factoryId'), '/linkedServices/FrameworkFunctions')]\",\n \"[concat(variables('factoryId'), '/linkedServices/SupportDatabase')]\"\n ]\n },\n {\n \"name\": \"[concat(parameters('factoryName'), '/03-Child')]\",\n \"type\": \"Microsoft.DataFactory/factories/pipelines\",\n \"apiVersion\": \"2018-06-01\",\n \"properties\": {\n \"description\": \"ADF.procfwk child pipeline used to execute Worker pipelines within a given execution stage. This pipeline will be called once for each stage, then execute all Workers in parallel.\",\n \"activities\": [\n {\n \"name\": \"Get Pipelines\",\n \"description\": \"Returns all pipelines from the metadata to be executed within a given processing stage.\",\n \"type\": \"Lookup\",\n \"dependsOn\": [],\n \"policy\": {\n \"timeout\": \"7.00:00:00\",\n \"retry\": 0,\n \"retryIntervalInSeconds\": 30,\n \"secureOutput\": false,\n \"secureInput\": false\n },\n \"userProperties\": [],\n \"typeProperties\": {\n \"source\": {\n \"type\": \"AzureSqlSource\",\n \"sqlReaderStoredProcedureName\": \"[[procfwk].[GetPipelinesInStage]\",\n \"storedProcedureParameters\": {\n \"StageId\": {\n \"type\": \"Int32\",\n \"value\": {\n \"value\": \"@pipeline().parameters.StageId\",\n \"type\": \"Expression\"\n }\n }\n },\n \"queryTimeout\": \"02:00:00\",\n \"partitionOption\": \"None\"\n },\n \"dataset\": {\n \"referenceName\": \"GetSetMetadata\",\n \"type\": \"DatasetReference\",\n \"parameters\": {}\n },\n \"firstRowOnly\": false\n }\n },\n {\n \"name\": \"Execute Pipelines\",\n \"description\": \"Second level ForEach to run in parallel all pipelines within the stage. Items for iteration passed from the Get Pipelines lookup activity.\",\n \"type\": \"ForEach\",\n \"dependsOn\": [\n {\n \"activity\": \"Get Pipelines\",\n \"dependencyConditions\": [\n \"Succeeded\"\n ]\n }\n ],\n \"userProperties\": [],\n \"typeProperties\": {\n \"items\": {\n \"value\": \"@activity('Get Pipelines').output.value\",\n \"type\": \"Expression\"\n },\n \"isSequential\": false,\n \"batchCount\": 40,\n \"activities\": [\n {\n \"name\": \"Execute Pipeline\",\n \"description\": \"The lowest level executor with the metadata framework to call existing processing pipelines within Data Factory. The function called will block processing and wait for an outcome.\",\n \"type\": \"AzureFunctionActivity\",\n \"dependsOn\": [\n {\n \"activity\": \"Log Pipeline Running\",\n \"dependencyConditions\": [\n \"Succeeded\"\n ]\n },\n {\n \"activity\": \"Get Pipeline Params\",\n \"dependencyConditions\": [\n \"Succeeded\"\n ]\n },\n {\n \"activity\": \"Get SPN Details\",\n \"dependencyConditions\": [\n \"Succeeded\"\n ]\n }\n ],\n \"policy\": {\n \"timeout\": \"7.00:00:00\",\n \"retry\": 0,\n \"retryIntervalInSeconds\": 30,\n \"secureOutput\": false,\n \"secureInput\": false\n },\n \"userProperties\": [],\n \"typeProperties\": {\n \"functionName\": \"ExecutePipeline\",\n \"method\": \"POST\",\n \"headers\": {},\n \"body\": {\n \"value\": \"@concat('\\n{\\n\\t\\\"tenantId\\\": \\\"',pipeline().parameters.TenantId,'\\\",\\n\\t\\\"applicationId\\\": \\\"',activity('Get SPN Details').output.firstRow.Id,'\\\",\\n\\t\\\"authenticationKey\\\": \\\"',activity('Get SPN Details').output.firstRow.Secret,'\\\",\\n\\t\\\"subscriptionId\\\": \\\"',pipeline().parameters.SubscriptionId,'\\\",\\n\\t\\\"resourceGroup\\\": \\\"',item().ResourceGroupName,'\\\",\\n\\t\\\"factoryName\\\": \\\"',item().DataFactoryName,'\\\",\\n\\t\\\"pipelineName\\\": \\\"',item().PipelineName,'\\\"',activity('Get Pipeline Params').output.firstRow.Params,'\\n}')\",\n \"type\": \"Expression\"\n }\n },\n \"linkedServiceName\": {\n \"referenceName\": \"FrameworkFunctions\",\n \"type\": \"LinkedServiceReference\"\n }\n },\n {\n \"name\": \"Get Pipeline Params\",\n \"description\": \"Returns any parameters from metadata required for the processing pipeline being called. The output can be an empty string if no parameters are required.\",\n \"type\": \"Lookup\",\n \"dependsOn\": [],\n \"policy\": {\n \"timeout\": \"7.00:00:00\",\n \"retry\": 0,\n \"retryIntervalInSeconds\": 30,\n \"secureOutput\": false,\n \"secureInput\": false\n },\n \"userProperties\": [],\n \"typeProperties\": {\n \"source\": {\n \"type\": \"AzureSqlSource\",\n \"sqlReaderStoredProcedureName\": \"[[procfwk].[GetPipelineParameters]\",\n \"storedProcedureParameters\": {\n \"PipelineId\": {\n \"type\": \"Int32\",\n \"value\": {\n \"value\": \"@item().PipelineId\",\n \"type\": \"Expression\"\n }\n }\n },\n \"queryTimeout\": \"02:00:00\",\n \"partitionOption\": \"None\"\n },\n \"dataset\": {\n \"referenceName\": \"GetSetMetadata\",\n \"type\": \"DatasetReference\",\n \"parameters\": {}\n }\n }\n },\n {\n \"name\": \"Log Pipeline Running\",\n \"description\": \"Sets the current pipeline with a status of running within the current execution database table.\",\n \"type\": \"SqlServerStoredProcedure\",\n \"dependsOn\": [],\n \"policy\": {\n \"timeout\": \"7.00:00:00\",\n \"retry\": 0,\n \"retryIntervalInSeconds\": 30,\n \"secureOutput\": false,\n \"secureInput\": false\n },\n \"userProperties\": [],\n \"typeProperties\": {\n \"storedProcedureName\": \"[[procfwk].[SetLogPipelineRunning]\",\n \"storedProcedureParameters\": {\n \"ExecutionId\": {\n \"value\": {\n \"value\": \"@pipeline().parameters.ExecutionId\",\n \"type\": \"Expression\"\n },\n \"type\": \"Guid\"\n },\n \"PipelineId\": {\n \"value\": {\n \"value\": \"@item().PipelineId\",\n \"type\": \"Expression\"\n },\n \"type\": \"Int32\"\n },\n \"StageId\": {\n \"value\": {\n \"value\": \"@pipeline().parameters.StageId\",\n \"type\": \"Expression\"\n },\n \"type\": \"Int32\"\n }\n }\n },\n \"linkedServiceName\": {\n \"referenceName\": \"SupportDatabase\",\n \"type\": \"LinkedServiceReference\"\n }\n },\n {\n \"name\": \"Get SPN Details\",\n \"description\": \"Return the SPN ID and Secret for the processing pipeline being executed. Called at this level as each pipeline can have a different SPN.\",\n \"type\": \"Lookup\",\n \"dependsOn\": [],\n \"policy\": {\n \"timeout\": \"7.00:00:00\",\n \"retry\": 0,\n \"retryIntervalInSeconds\": 30,\n \"secureOutput\": false,\n \"secureInput\": false\n },\n \"userProperties\": [],\n \"typeProperties\": {\n \"source\": {\n \"type\": \"AzureSqlSource\",\n \"sqlReaderStoredProcedureName\": \"[[procfwk].[GetServicePrincipal]\",\n \"storedProcedureParameters\": {\n \"DataFactory\": {\n \"type\": \"String\",\n \"value\": {\n \"value\": \"@item().DataFactoryName\",\n \"type\": \"Expression\"\n }\n },\n \"PipelineName\": {\n \"type\": \"String\",\n \"value\": {\n \"value\": \"@item().PipelineName\",\n \"type\": \"Expression\"\n }\n }\n },\n \"queryTimeout\": \"02:00:00\",\n \"partitionOption\": \"None\"\n },\n \"dataset\": {\n \"referenceName\": \"GetSetMetadata\",\n \"type\": \"DatasetReference\",\n \"parameters\": {}\n }\n }\n },\n {\n \"name\": \"Log Activity Failure\",\n \"description\": \"Handle true failures from calling out to the Azure Function and update the current execution table accordingly so a restart can occur.\",\n \"type\": \"SqlServerStoredProcedure\",\n \"dependsOn\": [\n {\n \"activity\": \"Execute Pipeline\",\n \"dependencyConditions\": [\n \"Failed\"\n ]\n }\n ],\n \"policy\": {\n \"timeout\": \"7.00:00:00\",\n \"retry\": 0,\n \"retryIntervalInSeconds\": 30,\n \"secureOutput\": false,\n \"secureInput\": false\n },\n \"userProperties\": [],\n \"typeProperties\": {\n \"storedProcedureName\": \"[[procfwk].[SetLogActivityFailed]\",\n \"storedProcedureParameters\": {\n \"ExecutionId\": {\n \"value\": {\n \"value\": \"@pipeline().parameters.ExecutionId\",\n \"type\": \"Expression\"\n },\n \"type\": \"Guid\"\n },\n \"PipelineId\": {\n \"value\": {\n \"value\": \"@item().PipelineId\",\n \"type\": \"Expression\"\n },\n \"type\": \"Int32\"\n },\n \"StageId\": {\n \"value\": {\n \"value\": \"@pipeline().parameters.StageId\",\n \"type\": \"Expression\"\n },\n \"type\": \"Int32\"\n },\n \"CallingActivity\": {\n \"value\": \"ExecutePipeline\",\n \"type\": \"String\"\n }\n }\n },\n \"linkedServiceName\": {\n \"referenceName\": \"SupportDatabase\",\n \"type\": \"LinkedServiceReference\"\n }\n },\n {\n \"name\": \"Running Pipeline Handler\",\n \"type\": \"ExecutePipeline\",\n \"dependsOn\": [\n {\n \"activity\": \"Execute Pipeline\",\n \"dependencyConditions\": [\n \"Succeeded\"\n ]\n }\n ],\n \"userProperties\": [],\n \"typeProperties\": {\n \"pipeline\": {\n \"referenceName\": \"04-Infant\",\n \"type\": \"PipelineReference\"\n },\n \"waitOnCompletion\": true,\n \"parameters\": {\n \"tenantId\": {\n \"value\": \"@pipeline().parameters.TenantId\",\n \"type\": \"Expression\"\n },\n \"applicationId\": {\n \"value\": \"@activity('Get SPN Details').output.firstRow.Id\",\n \"type\": \"Expression\"\n },\n \"authenticationKey\": {\n \"value\": \"@activity('Get SPN Details').output.firstRow.Secret\",\n \"type\": \"Expression\"\n },\n \"subscriptionId\": {\n \"value\": \"@pipeline().parameters.SubscriptionId\",\n \"type\": \"Expression\"\n },\n \"resourceGroup\": {\n \"value\": \"@item().ResourceGroupName\",\n \"type\": \"Expression\"\n },\n \"factoryName\": {\n \"value\": \"@item().DataFactoryName\",\n \"type\": \"Expression\"\n },\n \"pipelineName\": {\n \"value\": \"@item().PipelineName\",\n \"type\": \"Expression\"\n },\n \"runId\": {\n \"value\": \"@activity('Execute Pipeline').output.RunId\",\n \"type\": \"Expression\"\n },\n \"executionId\": {\n \"value\": \"@pipeline().parameters.ExecutionId\",\n \"type\": \"Expression\"\n },\n \"stageId\": {\n \"value\": \"@pipeline().parameters.StageId\",\n \"type\": \"Expression\"\n },\n \"pipelineId\": {\n \"value\": \"@item().PipelineId\",\n \"type\": \"Expression\"\n }\n }\n }\n },\n {\n \"name\": \"Set Run Id\",\n \"description\": \"Provide the actual ADF run ID back to the current execution table for long term logging and alignment between the metadata other Azure monitoring tools.\",\n \"type\": \"SqlServerStoredProcedure\",\n \"dependsOn\": [\n {\n \"activity\": \"Execute Pipeline\",\n \"dependencyConditions\": [\n \"Succeeded\"\n ]\n }\n ],\n \"policy\": {\n \"timeout\": \"7.00:00:00\",\n \"retry\": 0,\n \"retryIntervalInSeconds\": 30,\n \"secureOutput\": false,\n \"secureInput\": false\n },\n \"userProperties\": [],\n \"typeProperties\": {\n \"storedProcedureName\": \"[[procfwk].[SetLogPipelineRunId]\",\n \"storedProcedureParameters\": {\n \"ExecutionId\": {\n \"value\": {\n \"value\": \"@pipeline().parameters.ExecutionId\",\n \"type\": \"Expression\"\n },\n \"type\": \"Guid\"\n },\n \"PipelineId\": {\n \"value\": {\n \"value\": \"@item().PipelineId\",\n \"type\": \"Expression\"\n },\n \"type\": \"Int32\"\n },\n \"RunId\": {\n \"value\": {\n \"value\": \"@activity('Execute Pipeline').output.RunId\",\n \"type\": \"Expression\"\n },\n \"type\": \"Guid\"\n },\n \"StageId\": {\n \"value\": {\n \"value\": \"@pipeline().parameters.StageId\",\n \"type\": \"Expression\"\n },\n \"type\": \"Int32\"\n }\n }\n },\n \"linkedServiceName\": {\n \"referenceName\": \"SupportDatabase\",\n \"type\": \"LinkedServiceReference\"\n }\n },\n {\n \"name\": \"Check For Alerts\",\n \"description\": \"Checks the properties tables and if any recipients in the database require alerts sending for the current pipeline ID.\",\n \"type\": \"Lookup\",\n \"dependsOn\": [\n {\n \"activity\": \"Set Run Id\",\n \"dependencyConditions\": [\n \"Succeeded\"\n ]\n },\n {\n \"activity\": \"Running Pipeline Handler\",\n \"dependencyConditions\": [\n \"Completed\"\n ]\n }\n ],\n \"policy\": {\n \"timeout\": \"7.00:00:00\",\n \"retry\": 0,\n \"retryIntervalInSeconds\": 30,\n \"secureOutput\": false,\n \"secureInput\": false\n },\n \"userProperties\": [],\n \"typeProperties\": {\n \"source\": {\n \"type\": \"AzureSqlSource\",\n \"sqlReaderStoredProcedureName\": \"[[procfwk].[CheckForEmailAlerts]\",\n \"storedProcedureParameters\": {\n \"PipelineId\": {\n \"type\": \"Int32\",\n \"value\": {\n \"value\": \"@item().PipelineId\",\n \"type\": \"Expression\"\n }\n }\n },\n \"queryTimeout\": \"02:00:00\",\n \"partitionOption\": \"None\"\n },\n \"dataset\": {\n \"referenceName\": \"GetSetMetadata\",\n \"type\": \"DatasetReference\",\n \"parameters\": {}\n },\n \"firstRowOnly\": true\n }\n },\n {\n \"name\": \"Send Alerts\",\n \"description\": \"True = alerts need sending.\\nFalse = do nothing.\",\n \"type\": \"IfCondition\",\n \"dependsOn\": [\n {\n \"activity\": \"Check For Alerts\",\n \"dependencyConditions\": [\n \"Succeeded\"\n ]\n }\n ],\n \"userProperties\": [],\n \"typeProperties\": {\n \"expression\": {\n \"value\": \"@activity('Check For Alerts').output.firstRow.SendAlerts\",\n \"type\": \"Expression\"\n },\n \"ifTrueActivities\": [\n {\n \"name\": \"Get Email Parts\",\n \"type\": \"Lookup\",\n \"dependsOn\": [],\n \"policy\": {\n \"timeout\": \"7.00:00:00\",\n \"retry\": 0,\n \"retryIntervalInSeconds\": 30,\n \"secureOutput\": false,\n \"secureInput\": false\n },\n \"userProperties\": [],\n \"typeProperties\": {\n \"source\": {\n \"type\": \"AzureSqlSource\",\n \"sqlReaderStoredProcedureName\": \"[[procfwk].[GetEmailAlertParts]\",\n \"storedProcedureParameters\": {\n \"PipelineId\": {\n \"type\": \"Int32\",\n \"value\": {\n \"value\": \"@item().PipelineId\",\n \"type\": \"Expression\"\n }\n }\n },\n \"queryTimeout\": \"02:00:00\",\n \"partitionOption\": \"None\"\n },\n \"dataset\": {\n \"referenceName\": \"GetSetMetadata\",\n \"type\": \"DatasetReference\",\n \"parameters\": {}\n },\n \"firstRowOnly\": true\n }\n },\n {\n \"name\": \"Send Email\",\n \"type\": \"AzureFunctionActivity\",\n \"dependsOn\": [\n {\n \"activity\": \"Get Email Parts\",\n \"dependencyConditions\": [\n \"Succeeded\"\n ]\n }\n ],\n \"policy\": {\n \"timeout\": \"7.00:00:00\",\n \"retry\": 0,\n \"retryIntervalInSeconds\": 30,\n \"secureOutput\": false,\n \"secureInput\": false\n },\n \"userProperties\": [],\n \"typeProperties\": {\n \"functionName\": \"SendEmail\",\n \"method\": \"POST\",\n \"headers\": {},\n \"body\": {\n \"value\": \"@activity('Get Email Parts').output.firstRow\",\n \"type\": \"Expression\"\n }\n },\n \"linkedServiceName\": {\n \"referenceName\": \"FrameworkFunctions\",\n \"type\": \"LinkedServiceReference\"\n }\n }\n ]\n }\n }\n ]\n }\n }\n ],\n \"parameters\": {\n \"StageId\": {\n \"type\": \"int\"\n },\n \"ExecutionId\": {\n \"type\": \"string\"\n },\n \"TenantId\": {\n \"type\": \"string\"\n },\n \"SubscriptionId\": {\n \"type\": \"string\"\n }\n },\n \"variables\": {\n \"FunctionBody\": {\n \"type\": \"String\"\n }\n },\n \"folder\": {\n \"name\": \"_ProcFwk\"\n },\n \"annotations\": [\n \"ADF.procfwk\"\n ],\n \"lastPublishTime\": \"2020-08-17T11:35:20Z\"\n },\n \"dependsOn\": [\n \"[concat(variables('factoryId'), '/datasets/GetSetMetadata')]\",\n \"[concat(variables('factoryId'), '/linkedServices/FrameworkFunctions')]\",\n \"[concat(variables('factoryId'), '/linkedServices/SupportDatabase')]\",\n \"[concat(variables('factoryId'), '/pipelines/04-Infant')]\"\n ]\n },\n {\n \"name\": \"[concat(parameters('factoryName'), '/02-Parent')]\",\n \"type\": \"Microsoft.DataFactory/factories/pipelines\",\n \"apiVersion\": \"2018-06-01\",\n \"properties\": {\n \"description\": \"ADF.procfwk parent pipeline used to bootstrap the orchestration framework in perform the first level ForEach calls in sequence for the metadata stages.\",\n \"activities\": [\n {\n \"name\": \"Get Stages\",\n \"description\": \"Returns a distinct list of execution stages within the framework metadata.\",\n \"type\": \"Lookup\",\n \"dependsOn\": [\n {\n \"activity\": \"Execution Wrapper\",\n \"dependencyConditions\": [\n \"Succeeded\"\n ]\n }\n ],\n \"policy\": {\n \"timeout\": \"7.00:00:00\",\n \"retry\": 0,\n \"retryIntervalInSeconds\": 30,\n \"secureOutput\": false,\n \"secureInput\": false\n },\n \"userProperties\": [],\n \"typeProperties\": {\n \"source\": {\n \"type\": \"AzureSqlSource\",\n \"sqlReaderStoredProcedureName\": \"[[procfwk].[GetStages]\",\n \"storedProcedureParameters\": {\n \"ExecutionId\": {\n \"type\": \"Guid\",\n \"value\": {\n \"value\": \"@activity('Execution Wrapper').output.firstRow.ExecutionId\",\n \"type\": \"Expression\"\n }\n }\n },\n \"queryTimeout\": \"02:00:00\",\n \"partitionOption\": \"None\"\n },\n \"dataset\": {\n \"referenceName\": \"GetSetMetadata\",\n \"type\": \"DatasetReference\",\n \"parameters\": {}\n },\n \"firstRowOnly\": false\n }\n },\n {\n \"name\": \"Execute Stages\",\n \"description\": \"Top level ForEach to sequentially call all processing stages within the framework metadata. Items for iteration passed from the Get Stages lookup activity.\",\n \"type\": \"ForEach\",\n \"dependsOn\": [\n {\n \"activity\": \"Get Stages\",\n \"dependencyConditions\": [\n \"Succeeded\"\n ]\n },\n {\n \"activity\": \"Get Tenant Id\",\n \"dependencyConditions\": [\n \"Succeeded\"\n ]\n },\n {\n \"activity\": \"Get Subscription Id\",\n \"dependencyConditions\": [\n \"Succeeded\"\n ]\n }\n ],\n \"userProperties\": [],\n \"typeProperties\": {\n \"items\": {\n \"value\": \"@activity('Get Stages').output.value\",\n \"type\": \"Expression\"\n },\n \"isSequential\": true,\n \"activities\": [\n {\n \"name\": \"Stage Executor\",\n \"description\": \"Call to the framework generic child pipeline for a given execution stage.\",\n \"type\": \"ExecutePipeline\",\n \"dependsOn\": [\n {\n \"activity\": \"Log Stage Preparing\",\n \"dependencyConditions\": [\n \"Succeeded\"\n ]\n }\n ],\n \"userProperties\": [],\n \"typeProperties\": {\n \"pipeline\": {\n \"referenceName\": \"03-Child\",\n \"type\": \"PipelineReference\"\n },\n \"waitOnCompletion\": true,\n \"parameters\": {\n \"StageId\": {\n \"value\": \"@item().StageId\",\n \"type\": \"Expression\"\n },\n \"ExecutionId\": {\n \"value\": \"@activity('Execution Wrapper').output.firstRow.ExecutionId\",\n \"type\": \"Expression\"\n },\n \"TenantId\": {\n \"value\": \"@activity('Get Tenant Id').output.firstRow.PropertyValue\",\n \"type\": \"Expression\"\n },\n \"SubscriptionId\": {\n \"value\": \"@activity('Get Subscription Id').output.firstRow.PropertyValue\",\n \"type\": \"Expression\"\n }\n }\n }\n },\n {\n \"name\": \"Log Stage Preparing\",\n \"description\": \"Update the current execution table flagging all pipelines within the stage as preparing.\",\n \"type\": \"SqlServerStoredProcedure\",\n \"dependsOn\": [\n {\n \"activity\": \"Check and Update Blockers\",\n \"dependencyConditions\": [\n \"Succeeded\"\n ]\n }\n ],\n \"policy\": {\n \"timeout\": \"7.00:00:00\",\n \"retry\": 0,\n \"retryIntervalInSeconds\": 30,\n \"secureOutput\": false,\n \"secureInput\": false\n },\n \"userProperties\": [],\n \"typeProperties\": {\n \"storedProcedureName\": \"[[procfwk].[SetLogStagePreparing]\",\n \"storedProcedureParameters\": {\n \"ExecutionId\": {\n \"value\": {\n \"value\": \"@activity('Execution Wrapper').output.firstRow.ExecutionId\",\n \"type\": \"Expression\"\n },\n \"type\": \"Guid\"\n },\n \"StageId\": {\n \"value\": {\n \"value\": \"@item().StageId\",\n \"type\": \"Expression\"\n },\n \"type\": \"Int32\"\n }\n }\n },\n \"linkedServiceName\": {\n \"referenceName\": \"SupportDatabase\",\n \"type\": \"LinkedServiceReference\"\n }\n },\n {\n \"name\": \"Check and Update Blockers\",\n \"description\": \"Used to double check and stop the next execution stage if failures and blockers have be incurred. This also depends on the failure handling property value which defines the stored procedure behaviour.\",\n \"type\": \"SqlServerStoredProcedure\",\n \"dependsOn\": [],\n \"policy\": {\n \"timeout\": \"7.00:00:00\",\n \"retry\": 0,\n \"retryIntervalInSeconds\": 30,\n \"secureOutput\": false,\n \"secureInput\": false\n },\n \"userProperties\": [],\n \"typeProperties\": {\n \"storedProcedureName\": \"[[procfwk].[CheckForBlockedPipelines]\",\n \"storedProcedureParameters\": {\n \"StageId\": {\n \"value\": {\n \"value\": \"@item().StageId\",\n \"type\": \"Expression\"\n },\n \"type\": \"Int32\"\n }\n }\n },\n \"linkedServiceName\": {\n \"referenceName\": \"SupportDatabase\",\n \"type\": \"LinkedServiceReference\"\n }\n }\n ]\n }\n },\n {\n \"name\": \"Execution Wrapper\",\n \"description\": \"Wrapper to reset and restart processing or create a completely new execution instance of the framework metadata.\",\n \"type\": \"Lookup\",\n \"dependsOn\": [\n {\n \"activity\": \"Clean Up Previous Run\",\n \"dependencyConditions\": [\n \"Succeeded\"\n ]\n }\n ],\n \"policy\": {\n \"timeout\": \"7.00:00:00\",\n \"retry\": 0,\n \"retryIntervalInSeconds\": 30,\n \"secureOutput\": false,\n \"secureInput\": false\n },\n \"userProperties\": [],\n \"typeProperties\": {\n \"source\": {\n \"type\": \"AzureSqlSource\",\n \"sqlReaderStoredProcedureName\": \"[[procfwk].[ExecutionWrapper]\",\n \"storedProcedureParameters\": {\n \"CallingDataFactory\": {\n \"type\": \"String\",\n \"value\": {\n \"value\": \"@pipeline().DataFactory\",\n \"type\": \"Expression\"\n }\n }\n },\n \"queryTimeout\": \"02:00:00\",\n \"partitionOption\": \"None\"\n },\n \"dataset\": {\n \"referenceName\": \"GetSetMetadata\",\n \"type\": \"DatasetReference\",\n \"parameters\": {}\n }\n }\n },\n {\n \"name\": \"Check Outcome and Update Logs\",\n \"description\": \"After a successful execution run the current execution metadata is moved to the long term logging table by this stored procedure call. Otherwise an error will be raised.\",\n \"type\": \"SqlServerStoredProcedure\",\n \"dependsOn\": [\n {\n \"activity\": \"Execute Stages\",\n \"dependencyConditions\": [\n \"Succeeded\"\n ]\n }\n ],\n \"policy\": {\n \"timeout\": \"7.00:00:00\",\n \"retry\": 0,\n \"retryIntervalInSeconds\": 30,\n \"secureOutput\": false,\n \"secureInput\": false\n },\n \"userProperties\": [],\n \"typeProperties\": {\n \"storedProcedureName\": \"[[procfwk].[UpdateExecutionLog]\",\n \"storedProcedureParameters\": {\n \"PerformErrorCheck\": {\n \"value\": {\n \"value\": \"@bool(1)\",\n \"type\": \"Expression\"\n },\n \"type\": \"Boolean\"\n }\n }\n },\n \"linkedServiceName\": {\n \"referenceName\": \"SupportDatabase\",\n \"type\": \"LinkedServiceReference\"\n }\n },\n {\n \"name\": \"Get Tenant Id\",\n \"description\": \"Returning the Azure Tenant Id from the metadata properties table.\",\n \"type\": \"Lookup\",\n \"dependsOn\": [\n {\n \"activity\": \"Metadata Integrity Checks\",\n \"dependencyConditions\": [\n \"Succeeded\"\n ]\n }\n ],\n \"policy\": {\n \"timeout\": \"7.00:00:00\",\n \"retry\": 0,\n \"retryIntervalInSeconds\": 30,\n \"secureOutput\": false,\n \"secureInput\": false\n },\n \"userProperties\": [],\n \"typeProperties\": {\n \"source\": {\n \"type\": \"AzureSqlSource\",\n \"sqlReaderStoredProcedureName\": \"[[procfwk].[GetPropertyValue]\",\n \"storedProcedureParameters\": {\n \"PropertyName\": {\n \"type\": \"String\",\n \"value\": \"TenantId\"\n }\n },\n \"queryTimeout\": \"02:00:00\",\n \"partitionOption\": \"None\"\n },\n \"dataset\": {\n \"referenceName\": \"GetSetMetadata\",\n \"type\": \"DatasetReference\",\n \"parameters\": {}\n },\n \"firstRowOnly\": true\n }\n },\n {\n \"name\": \"Get Subscription Id\",\n \"description\": \"Returning the Azure Subscription Id from the metadata properties table.\",\n \"type\": \"Lookup\",\n \"dependsOn\": [\n {\n \"activity\": \"Metadata Integrity Checks\",\n \"dependencyConditions\": [\n \"Succeeded\"\n ]\n }\n ],\n \"policy\": {\n \"timeout\": \"7.00:00:00\",\n \"retry\": 0,\n \"retryIntervalInSeconds\": 30,\n \"secureOutput\": false,\n \"secureInput\": true\n },\n \"userProperties\": [],\n \"typeProperties\": {\n \"source\": {\n \"type\": \"AzureSqlSource\",\n \"sqlReaderStoredProcedureName\": \"[[procfwk].[GetPropertyValue]\",\n \"storedProcedureParameters\": {\n \"PropertyName\": {\n \"type\": \"String\",\n \"value\": \"SubscriptionId\"\n }\n },\n \"queryTimeout\": \"02:00:00\",\n \"partitionOption\": \"None\"\n },\n \"dataset\": {\n \"referenceName\": \"GetSetMetadata\",\n \"type\": \"DatasetReference\",\n \"parameters\": {}\n }\n }\n },\n {\n \"name\": \"Metadata Integrity Checks\",\n \"description\": \"Performs a series of checks on all metadata held in the framework SQLDB. This is intended to raise errors before an execution run even starts.\",\n \"type\": \"Lookup\",\n \"dependsOn\": [\n {\n \"activity\": \"Execute Precursor\",\n \"dependencyConditions\": [\n \"Succeeded\"\n ]\n }\n ],\n \"policy\": {\n \"timeout\": \"7.00:00:00\",\n \"retry\": 0,\n \"retryIntervalInSeconds\": 30,\n \"secureOutput\": false,\n \"secureInput\": false\n },\n \"userProperties\": [],\n \"typeProperties\": {\n \"source\": {\n \"type\": \"AzureSqlSource\",\n \"sqlReaderStoredProcedureName\": \"[[procfwk].[CheckMetadataIntegrity]\",\n \"storedProcedureParameters\": {\n \"DebugMode\": {\n \"type\": \"Boolean\",\n \"value\": \"false\"\n }\n },\n \"queryTimeout\": \"02:00:00\",\n \"partitionOption\": \"None\"\n },\n \"dataset\": {\n \"referenceName\": \"GetSetMetadata\",\n \"type\": \"DatasetReference\",\n \"parameters\": {}\n },\n \"firstRowOnly\": false\n }\n },\n {\n \"name\": \"Clean Up Previous Run\",\n \"description\": \"Handle Worker pipelines that are reported as Running when the parent pipeline is called again. Get what the actual status of those pipelines is.\",\n \"type\": \"ForEach\",\n \"dependsOn\": [\n {\n \"activity\": \"Metadata Integrity Checks\",\n \"dependencyConditions\": [\n \"Succeeded\"\n ]\n }\n ],\n \"userProperties\": [],\n \"typeProperties\": {\n \"items\": {\n \"value\": \"@activity('Metadata Integrity Checks').output.value\",\n \"type\": \"Expression\"\n },\n \"isSequential\": false,\n \"activities\": [\n {\n \"name\": \"Get SPN Details\",\n \"type\": \"Lookup\",\n \"dependsOn\": [],\n \"policy\": {\n \"timeout\": \"7.00:00:00\",\n \"retry\": 0,\n \"retryIntervalInSeconds\": 30,\n \"secureOutput\": false,\n \"secureInput\": false\n },\n \"userProperties\": [],\n \"typeProperties\": {\n \"source\": {\n \"type\": \"AzureSqlSource\",\n \"sqlReaderStoredProcedureName\": \"[[procfwk].[GetServicePrincipal]\",\n \"storedProcedureParameters\": {\n \"DataFactory\": {\n \"type\": \"String\",\n \"value\": {\n \"value\": \"@item().DataFactoryName\",\n \"type\": \"Expression\"\n }\n },\n \"PipelineName\": {\n \"type\": \"String\",\n \"value\": {\n \"value\": \"@item().PipelineName\",\n \"type\": \"Expression\"\n }\n }\n },\n \"queryTimeout\": \"02:00:00\",\n \"partitionOption\": \"None\"\n },\n \"dataset\": {\n \"referenceName\": \"GetSetMetadata\",\n \"type\": \"DatasetReference\",\n \"parameters\": {}\n }\n }\n },\n {\n \"name\": \"Log Pipeline Checking\",\n \"type\": \"SqlServerStoredProcedure\",\n \"dependsOn\": [],\n \"policy\": {\n \"timeout\": \"7.00:00:00\",\n \"retry\": 0,\n \"retryIntervalInSeconds\": 30,\n \"secureOutput\": false,\n \"secureInput\": false\n },\n \"userProperties\": [],\n \"typeProperties\": {\n \"storedProcedureName\": \"[[procfwk].[SetLogPipelineChecking]\",\n \"storedProcedureParameters\": {\n \"ExecutionId\": {\n \"value\": {\n \"value\": \"@item().LocalExecutionId\",\n \"type\": \"Expression\"\n },\n \"type\": \"Guid\"\n },\n \"PipelineId\": {\n \"value\": {\n \"value\": \"@item().PipelineId\",\n \"type\": \"Expression\"\n },\n \"type\": \"Int32\"\n },\n \"StageId\": {\n \"value\": {\n \"value\": \"@item().StageId\",\n \"type\": \"Expression\"\n },\n \"type\": \"Int32\"\n }\n }\n },\n \"linkedServiceName\": {\n \"referenceName\": \"SupportDatabase\",\n \"type\": \"LinkedServiceReference\"\n }\n },\n {\n \"name\": \"Get Pipeline Status\",\n \"type\": \"AzureFunctionActivity\",\n \"dependsOn\": [\n {\n \"activity\": \"Get SPN Details\",\n \"dependencyConditions\": [\n \"Succeeded\"\n ]\n },\n {\n \"activity\": \"Log Pipeline Checking\",\n \"dependencyConditions\": [\n \"Succeeded\"\n ]\n }\n ],\n \"policy\": {\n \"timeout\": \"7.00:00:00\",\n \"retry\": 0,\n \"retryIntervalInSeconds\": 30,\n \"secureOutput\": false,\n \"secureInput\": false\n },\n \"userProperties\": [],\n \"typeProperties\": {\n \"functionName\": \"CheckPipelineStatus\",\n \"method\": \"POST\",\n \"headers\": {},\n \"body\": {\n \"value\": \"@concat('\\n{\\n \\\"tenantId\\\": \\\"',item().TenantId,'\\\",\\n \\\"applicationId\\\": \\\"',activity('Get SPN Details').output.firstRow.Id,'\\\",\\n \\\"authenticationKey\\\": \\\"',activity('Get SPN Details').output.firstRow.Secret,'\\\",\\n \\\"subscriptionId\\\": \\\"',item().SubscriptionId,'\\\",\\n \\\"resourceGroup\\\": \\\"',item().ResourceGroupName,'\\\",\\n \\\"factoryName\\\": \\\"',item().DataFactoryName,'\\\",\\n \\\"pipelineName\\\": \\\"',item().PipelineName,'\\\",\\n \\\"runId\\\": \\\"',item().AdfPipelineRunId,'\\\"\\n}')\",\n \"type\": \"Expression\"\n }\n },\n \"linkedServiceName\": {\n \"referenceName\": \"FrameworkFunctions\",\n \"type\": \"LinkedServiceReference\"\n }\n },\n {\n \"name\": \"Set Pipeline Status\",\n \"type\": \"Switch\",\n \"dependsOn\": [\n {\n \"activity\": \"Get Pipeline Status\",\n \"dependencyConditions\": [\n \"Succeeded\"\n ]\n }\n ],\n \"userProperties\": [],\n \"typeProperties\": {\n \"on\": {\n \"value\": \"@activity('Get Pipeline Status').output.Status\",\n \"type\": \"Expression\"\n },\n \"cases\": [\n {\n \"value\": \"Failed\",\n \"activities\": [\n {\n \"name\": \"Pipeline Status Failed\",\n \"type\": \"SqlServerStoredProcedure\",\n \"dependsOn\": [],\n \"policy\": {\n \"timeout\": \"7.00:00:00\",\n \"retry\": 0,\n \"retryIntervalInSeconds\": 30,\n \"secureOutput\": false,\n \"secureInput\": false\n },\n \"userProperties\": [],\n \"typeProperties\": {\n \"storedProcedureName\": \"[[procfwk].[SetLogPipelineFailed]\",\n \"storedProcedureParameters\": {\n \"ExecutionId\": {\n \"value\": {\n \"value\": \"@item().LocalExecutionId\",\n \"type\": \"Expression\"\n },\n \"type\": \"Guid\"\n },\n \"PipelineId\": {\n \"value\": {\n \"value\": \"@item().PipelineId\",\n \"type\": \"Expression\"\n },\n \"type\": \"Int32\"\n },\n \"RunId\": {\n \"value\": null,\n \"type\": \"Guid\"\n },\n \"StageId\": {\n \"value\": {\n \"value\": \"@item().StageId\",\n \"type\": \"Expression\"\n },\n \"type\": \"Int32\"\n }\n }\n },\n \"linkedServiceName\": {\n \"referenceName\": \"SupportDatabase\",\n \"type\": \"LinkedServiceReference\"\n }\n }\n ]\n },\n {\n \"value\": \"Succeeded\",\n \"activities\": [\n {\n \"name\": \"Pipeline Status Succeeded\",\n \"type\": \"SqlServerStoredProcedure\",\n \"dependsOn\": [],\n \"policy\": {\n \"timeout\": \"7.00:00:00\",\n \"retry\": 0,\n \"retryIntervalInSeconds\": 30,\n \"secureOutput\": false,\n \"secureInput\": false\n },\n \"userProperties\": [],\n \"typeProperties\": {\n \"storedProcedureName\": \"[[procfwk].[SetLogPipelineSuccess]\",\n \"storedProcedureParameters\": {\n \"ExecutionId\": {\n \"value\": {\n \"value\": \"@item().LocalExecutionId\",\n \"type\": \"Expression\"\n },\n \"type\": \"Guid\"\n },\n \"PipelineId\": {\n \"value\": {\n \"value\": \"@item().PipelineId\",\n \"type\": \"Expression\"\n },\n \"type\": \"Int32\"\n },\n \"StageId\": {\n \"value\": {\n \"value\": \"@item().StageId\",\n \"type\": \"Expression\"\n },\n \"type\": \"Int32\"\n }\n }\n },\n \"linkedServiceName\": {\n \"referenceName\": \"SupportDatabase\",\n \"type\": \"LinkedServiceReference\"\n }\n }\n ]\n },\n {\n \"value\": \"Queued\",\n \"activities\": [\n {\n \"name\": \"Pipeline Status Queued - Running\",\n \"type\": \"SqlServerStoredProcedure\",\n \"dependsOn\": [],\n \"policy\": {\n \"timeout\": \"7.00:00:00\",\n \"retry\": 0,\n \"retryIntervalInSeconds\": 30,\n \"secureOutput\": false,\n \"secureInput\": false\n },\n \"userProperties\": [],\n \"typeProperties\": {\n \"storedProcedureName\": \"[[procfwk].[SetLogPipelineRunning]\",\n \"storedProcedureParameters\": {\n \"ExecutionId\": {\n \"value\": {\n \"value\": \"@item().LocalExecutionId\",\n \"type\": \"Expression\"\n },\n \"type\": \"Guid\"\n },\n \"PipelineId\": {\n \"value\": {\n \"value\": \"@item().PipelineId\",\n \"type\": \"Expression\"\n },\n \"type\": \"Int32\"\n },\n \"StageId\": {\n \"value\": {\n \"value\": \"@item().StageId\",\n \"type\": \"Expression\"\n },\n \"type\": \"Int32\"\n }\n }\n },\n \"linkedServiceName\": {\n \"referenceName\": \"SupportDatabase\",\n \"type\": \"LinkedServiceReference\"\n }\n }\n ]\n },\n {\n \"value\": \"InProgress\",\n \"activities\": [\n {\n \"name\": \"Pipeline Status InProgress - Running\",\n \"type\": \"SqlServerStoredProcedure\",\n \"dependsOn\": [],\n \"policy\": {\n \"timeout\": \"7.00:00:00\",\n \"retry\": 0,\n \"retryIntervalInSeconds\": 30,\n \"secureOutput\": false,\n \"secureInput\": false\n },\n \"userProperties\": [],\n \"typeProperties\": {\n \"storedProcedureName\": \"[[procfwk].[SetLogPipelineRunning]\",\n \"storedProcedureParameters\": {\n \"ExecutionId\": {\n \"value\": {\n \"value\": \"@item().LocalExecutionId\",\n \"type\": \"Expression\"\n },\n \"type\": \"Guid\"\n },\n \"PipelineId\": {\n \"value\": {\n \"value\": \"@item().PipelineId\",\n \"type\": \"Expression\"\n },\n \"type\": \"Int32\"\n },\n \"StageId\": {\n \"value\": {\n \"value\": \"@item().StageId\",\n \"type\": \"Expression\"\n },\n \"type\": \"Int32\"\n }\n }\n },\n \"linkedServiceName\": {\n \"referenceName\": \"SupportDatabase\",\n \"type\": \"LinkedServiceReference\"\n }\n }\n ]\n },\n {\n \"value\": \"Cancelled\",\n \"activities\": [\n {\n \"name\": \"Pipeline Status Cancelled\",\n \"type\": \"SqlServerStoredProcedure\",\n \"dependsOn\": [],\n \"policy\": {\n \"timeout\": \"7.00:00:00\",\n \"retry\": 0,\n \"retryIntervalInSeconds\": 30,\n \"secureOutput\": false,\n \"secureInput\": false\n },\n \"userProperties\": [],\n \"typeProperties\": {\n \"storedProcedureName\": \"[[procfwk].[SetLogPipelineCancelled]\",\n \"storedProcedureParameters\": {\n \"ExecutionId\": {\n \"value\": {\n \"value\": \"@item().LocalExecutionId\",\n \"type\": \"Expression\"\n },\n \"type\": \"Guid\"\n },\n \"PipelineId\": {\n \"value\": {\n \"value\": \"@item().PipelineId\",\n \"type\": \"Expression\"\n },\n \"type\": \"Int32\"\n },\n \"StageId\": {\n \"value\": {\n \"value\": \"@item().StageId\",\n \"type\": \"Expression\"\n },\n \"type\": \"Int32\"\n }\n }\n },\n \"linkedServiceName\": {\n \"referenceName\": \"SupportDatabase\",\n \"type\": \"LinkedServiceReference\"\n }\n }\n ]\n }\n ],\n \"defaultActivities\": [\n {\n \"name\": \"Pipeline Status Unknown\",\n \"type\": \"SqlServerStoredProcedure\",\n \"dependsOn\": [],\n \"policy\": {\n \"timeout\": \"7.00:00:00\",\n \"retry\": 0,\n \"retryIntervalInSeconds\": 30,\n \"secureOutput\": false,\n \"secureInput\": false\n },\n \"userProperties\": [],\n \"typeProperties\": {\n \"storedProcedureName\": \"[[procfwk].[SetLogPipelineUnknown]\",\n \"storedProcedureParameters\": {\n \"ExecutionId\": {\n \"value\": {\n \"value\": \"@item().LocalExecutionId\",\n \"type\": \"Expression\"\n },\n \"type\": \"Guid\"\n },\n \"PipelineId\": {\n \"value\": {\n \"value\": \"@item().PipelineId\",\n \"type\": \"Expression\"\n },\n \"type\": \"Int32\"\n },\n \"StageId\": {\n \"value\": {\n \"value\": \"@item().StageId\",\n \"type\": \"Expression\"\n },\n \"type\": \"Int32\"\n }\n }\n },\n \"linkedServiceName\": {\n \"referenceName\": \"SupportDatabase\",\n \"type\": \"LinkedServiceReference\"\n }\n }\n ]\n }\n },\n {\n \"name\": \"Set Last Check DateTime\",\n \"type\": \"SqlServerStoredProcedure\",\n \"dependsOn\": [\n {\n \"activity\": \"Get Pipeline Status\",\n \"dependencyConditions\": [\n \"Succeeded\"\n ]\n }\n ],\n \"policy\": {\n \"timeout\": \"7.00:00:00\",\n \"retry\": 0,\n \"retryIntervalInSeconds\": 30,\n \"secureOutput\": false,\n \"secureInput\": false\n },\n \"userProperties\": [],\n \"typeProperties\": {\n \"storedProcedureName\": \"[[procfwk].[SetLogPipelineLastStatusCheck]\",\n \"storedProcedureParameters\": {\n \"ExecutionId\": {\n \"value\": {\n \"value\": \"@item().LocalExecutionId\",\n \"type\": \"Expression\"\n },\n \"type\": \"Guid\"\n },\n \"PipelineId\": {\n \"value\": {\n \"value\": \"@item().PipelineId\",\n \"type\": \"Expression\"\n },\n \"type\": \"Int32\"\n },\n \"StageId\": {\n \"value\": {\n \"value\": \"@item().StageId\",\n \"type\": \"Expression\"\n },\n \"type\": \"Int32\"\n }\n }\n },\n \"linkedServiceName\": {\n \"referenceName\": \"SupportDatabase\",\n \"type\": \"LinkedServiceReference\"\n }\n }\n ]\n }\n },\n {\n \"name\": \"Execute Precursor\",\n \"type\": \"SqlServerStoredProcedure\",\n \"dependsOn\": [],\n \"policy\": {\n \"timeout\": \"7.00:00:00\",\n \"retry\": 0,\n \"retryIntervalInSeconds\": 30,\n \"secureOutput\": false,\n \"secureInput\": false\n },\n \"userProperties\": [],\n \"typeProperties\": {\n \"storedProcedureName\": \"[[procfwk].[ExecutePrecursorProcedure]\"\n },\n \"linkedServiceName\": {\n \"referenceName\": \"SupportDatabase\",\n \"type\": \"LinkedServiceReference\"\n }\n }\n ],\n \"folder\": {\n \"name\": \"_ProcFwk\"\n },\n \"annotations\": [\n \"ADF.procfwk\"\n ],\n \"lastPublishTime\": \"2020-08-17T11:35:21Z\"\n },\n \"dependsOn\": [\n \"[concat(variables('factoryId'), '/datasets/GetSetMetadata')]\",\n \"[concat(variables('factoryId'), '/linkedServices/SupportDatabase')]\",\n \"[concat(variables('factoryId'), '/pipelines/03-Child')]\",\n \"[concat(variables('factoryId'), '/linkedServices/FrameworkFunctions')]\"\n ]\n },\n {\n \"name\": \"[concat(parameters('factoryName'), '/01-Grandparent')]\",\n \"type\": \"Microsoft.DataFactory/factories/pipelines\",\n \"apiVersion\": \"2018-06-01\",\n \"properties\": {\n \"description\": \"ADF.procfwk grandparent pipeline used optionally to bootstrap any wider processes in your Data Factory that then calls the processing framework.\",\n \"activities\": [\n {\n \"name\": \"Framework Processing\",\n \"type\": \"ExecutePipeline\",\n \"dependsOn\": [],\n \"userProperties\": [],\n \"typeProperties\": {\n \"pipeline\": {\n \"referenceName\": \"02-Parent\",\n \"type\": \"PipelineReference\"\n },\n \"waitOnCompletion\": true,\n \"parameters\": {}\n }\n }\n ],\n \"folder\": {\n \"name\": \"_ProcFwk\"\n },\n \"annotations\": [\n \"ADF.procfwk\"\n ],\n \"lastPublishTime\": \"2020-08-17T11:35:22Z\"\n },\n \"dependsOn\": [\n \"[concat(variables('factoryId'), '/pipelines/02-Parent')]\"\n ]\n }\n ]\n}"
},
{
"path": "ARM Templates/Data Factory/v1.8.6 Export.json",
"content": "{\n \"$schema\": \"http://schema.management.azure.com/schemas/2015-01-01/deploymentTemplate.json#\",\n \"contentVersion\": \"1.0.0.0\",\n \"parameters\": {\n \"factoryName\": {\n \"type\": \"string\",\n \"metadata\": \"Data Factory name\",\n \"defaultValue\": \"\"\n },\n \"FrameworkFunctions_properties_typeProperties_functionAppUrl\": {\n \"type\": \"string\",\n \"defaultValue\": \"\"\n },\n \"Keys_properties_typeProperties_baseUrl\": {\n \"type\": \"string\",\n \"defaultValue\": \"\"\n },\n \"SupportDatabase_properties_typeProperties_connectionString_secretName\": {\n \"type\": \"string\",\n \"defaultValue\": \"\"\n }\n },\n \"variables\": {\n \"factoryId\": \"[concat('Microsoft.DataFactory/factories/', parameters('factoryName'))]\"\n },\n \"resources\": [\n {\n \"name\": \"[concat(parameters('factoryName'), '/01-Grandparent')]\",\n \"type\": \"Microsoft.DataFactory/factories/pipelines\",\n \"apiVersion\": \"2018-06-01\",\n \"properties\": {\n \"description\": \"ADF.procfwk grandparent pipeline used optionally to bootstrap any wider processes in your Data Factory that then calls the processing framework.\",\n \"activities\": [\n {\n \"name\": \"Framework Processing\",\n \"type\": \"ExecutePipeline\",\n \"dependsOn\": [],\n \"userProperties\": [],\n \"typeProperties\": {\n \"pipeline\": {\n \"referenceName\": \"02-Parent\",\n \"type\": \"PipelineReference\"\n },\n \"waitOnCompletion\": true,\n \"parameters\": {}\n }\n }\n ],\n \"folder\": {\n \"name\": \"_ProcFwk\"\n },\n \"annotations\": [\n \"ADF.procfwk\",\n \"Grandparent\"\n ]\n },\n \"dependsOn\": [\n \"[concat(variables('factoryId'), '/pipelines/02-Parent')]\"\n ]\n },\n {\n \"name\": \"[concat(parameters('factoryName'), '/02-Parent')]\",\n \"type\": \"Microsoft.DataFactory/factories/pipelines\",\n \"apiVersion\": \"2018-06-01\",\n \"properties\": {\n \"description\": \"ADF.procfwk parent pipeline used to bootstrap the orchestration framework in perform the first level ForEach calls in sequence for the metadata stages.\",\n \"activities\": [\n {\n \"name\": \"Get Stages\",\n \"description\": \"Returns a distinct list of execution stages within the framework metadata.\",\n \"type\": \"Lookup\",\n \"dependsOn\": [\n {\n \"activity\": \"Set Execution Id\",\n \"dependencyConditions\": [\n \"Succeeded\"\n ]\n }\n ],\n \"policy\": {\n \"timeout\": \"7.00:00:00\",\n \"retry\": 0,\n \"retryIntervalInSeconds\": 30,\n \"secureOutput\": false,\n \"secureInput\": false\n },\n \"userProperties\": [],\n \"typeProperties\": {\n \"source\": {\n \"type\": \"AzureSqlSource\",\n \"sqlReaderStoredProcedureName\": \"[[procfwk].[GetStages]\",\n \"storedProcedureParameters\": {\n \"ExecutionId\": {\n \"type\": \"Guid\",\n \"value\": {\n \"value\": \"@variables('ExecutionId')\",\n \"type\": \"Expression\"\n }\n }\n },\n \"queryTimeout\": \"02:00:00\",\n \"partitionOption\": \"None\"\n },\n \"dataset\": {\n \"referenceName\": \"GetSetMetadata\",\n \"type\": \"DatasetReference\",\n \"parameters\": {}\n },\n \"firstRowOnly\": false\n }\n },\n {\n \"name\": \"Execute Stages\",\n \"description\": \"Top level ForEach to sequentially call all processing stages within the framework metadata. Items for iteration passed from the Get Stages lookup activity.\",\n \"type\": \"ForEach\",\n \"dependsOn\": [\n {\n \"activity\": \"Get Stages\",\n \"dependencyConditions\": [\n \"Succeeded\"\n ]\n },\n {\n \"activity\": \"Get Tenant Id\",\n \"dependencyConditions\": [\n \"Succeeded\"\n ]\n },\n {\n \"activity\": \"Get Subscription Id\",\n \"dependencyConditions\": [\n \"Succeeded\"\n ]\n }\n ],\n \"userProperties\": [],\n \"typeProperties\": {\n \"items\": {\n \"value\": \"@activity('Get Stages').output.value\",\n \"type\": \"Expression\"\n },\n \"isSequential\": true,\n \"activities\": [\n {\n \"name\": \"Stage Executor\",\n \"description\": \"Call to the framework generic child pipeline for a given execution stage.\",\n \"type\": \"ExecutePipeline\",\n \"dependsOn\": [\n {\n \"activity\": \"Log Stage Preparing\",\n \"dependencyConditions\": [\n \"Succeeded\"\n ]\n }\n ],\n \"userProperties\": [],\n \"typeProperties\": {\n \"pipeline\": {\n \"referenceName\": \"03-Child\",\n \"type\": \"PipelineReference\"\n },\n \"waitOnCompletion\": true,\n \"parameters\": {\n \"StageId\": {\n \"value\": \"@item().StageId\",\n \"type\": \"Expression\"\n },\n \"ExecutionId\": {\n \"value\": \"@variables('ExecutionId')\",\n \"type\": \"Expression\"\n },\n \"TenantId\": {\n \"value\": \"@activity('Get Tenant Id').output.firstRow.PropertyValue\",\n \"type\": \"Expression\"\n },\n \"SubscriptionId\": {\n \"value\": \"@activity('Get Subscription Id').output.firstRow.PropertyValue\",\n \"type\": \"Expression\"\n }\n }\n }\n },\n {\n \"name\": \"Log Stage Preparing\",\n \"description\": \"Update the current execution table flagging all pipelines within the stage as preparing.\",\n \"type\": \"SqlServerStoredProcedure\",\n \"dependsOn\": [\n {\n \"activity\": \"Check and Update Blockers\",\n \"dependencyConditions\": [\n \"Succeeded\"\n ]\n }\n ],\n \"policy\": {\n \"timeout\": \"7.00:00:00\",\n \"retry\": 0,\n \"retryIntervalInSeconds\": 30,\n \"secureOutput\": false,\n \"secureInput\": false\n },\n \"userProperties\": [],\n \"typeProperties\": {\n \"storedProcedureName\": \"[[procfwk].[SetLogStagePreparing]\",\n \"storedProcedureParameters\": {\n \"ExecutionId\": {\n \"value\": {\n \"value\": \"@variables('ExecutionId')\",\n \"type\": \"Expression\"\n },\n \"type\": \"Guid\"\n },\n \"StageId\": {\n \"value\": {\n \"value\": \"@item().StageId\",\n \"type\": \"Expression\"\n },\n \"type\": \"Int32\"\n }\n }\n },\n \"linkedServiceName\": {\n \"referenceName\": \"SupportDatabase\",\n \"type\": \"LinkedServiceReference\"\n }\n },\n {\n \"name\": \"Check and Update Blockers\",\n \"description\": \"Used to double check and stop the next execution stage if failures and blockers have be incurred. This also depends on the failure handling property value which defines the stored procedure behaviour.\",\n \"type\": \"SqlServerStoredProcedure\",\n \"dependsOn\": [],\n \"policy\": {\n \"timeout\": \"7.00:00:00\",\n \"retry\": 0,\n \"retryIntervalInSeconds\": 30,\n \"secureOutput\": false,\n \"secureInput\": false\n },\n \"userProperties\": [],\n \"typeProperties\": {\n \"storedProcedureName\": \"[[procfwk].[CheckForBlockedPipelines]\",\n \"storedProcedureParameters\": {\n \"StageId\": {\n \"value\": {\n \"value\": \"@item().StageId\",\n \"type\": \"Expression\"\n },\n \"type\": \"Int32\"\n }\n }\n },\n \"linkedServiceName\": {\n \"referenceName\": \"SupportDatabase\",\n \"type\": \"LinkedServiceReference\"\n }\n }\n ]\n }\n },\n {\n \"name\": \"Execution Wrapper\",\n \"description\": \"Wrapper to reset and restart processing or create a completely new execution instance of the framework metadata.\",\n \"type\": \"Lookup\",\n \"dependsOn\": [\n {\n \"activity\": \"Clean Up Previous Run\",\n \"dependencyConditions\": [\n \"Succeeded\"\n ]\n }\n ],\n \"policy\": {\n \"timeout\": \"7.00:00:00\",\n \"retry\": 0,\n \"retryIntervalInSeconds\": 30,\n \"secureOutput\": false,\n \"secureInput\": false\n },\n \"userProperties\": [],\n \"typeProperties\": {\n \"source\": {\n \"type\": \"AzureSqlSource\",\n \"sqlReaderStoredProcedureName\": \"[[procfwk].[ExecutionWrapper]\",\n \"storedProcedureParameters\": {\n \"CallingDataFactory\": {\n \"type\": \"String\",\n \"value\": {\n \"value\": \"@pipeline().DataFactory\",\n \"type\": \"Expression\"\n }\n }\n },\n \"queryTimeout\": \"02:00:00\",\n \"partitionOption\": \"None\"\n },\n \"dataset\": {\n \"referenceName\": \"GetSetMetadata\",\n \"type\": \"DatasetReference\",\n \"parameters\": {}\n }\n }\n },\n {\n \"name\": \"Check Outcome and Update Logs\",\n \"description\": \"After a successful execution run the current execution metadata is moved to the long term logging table by this stored procedure call. Otherwise an error will be raised.\",\n \"type\": \"SqlServerStoredProcedure\",\n \"dependsOn\": [\n {\n \"activity\": \"Execute Stages\",\n \"dependencyConditions\": [\n \"Succeeded\"\n ]\n }\n ],\n \"policy\": {\n \"timeout\": \"7.00:00:00\",\n \"retry\": 0,\n \"retryIntervalInSeconds\": 30,\n \"secureOutput\": false,\n \"secureInput\": false\n },\n \"userProperties\": [],\n \"typeProperties\": {\n \"storedProcedureName\": \"[[procfwk].[UpdateExecutionLog]\",\n \"storedProcedureParameters\": {\n \"PerformErrorCheck\": {\n \"value\": {\n \"value\": \"@bool(1)\",\n \"type\": \"Expression\"\n },\n \"type\": \"Boolean\"\n }\n }\n },\n \"linkedServiceName\": {\n \"referenceName\": \"SupportDatabase\",\n \"type\": \"LinkedServiceReference\"\n }\n },\n {\n \"name\": \"Get Tenant Id\",\n \"description\": \"Returning the Azure Tenant Id from the metadata properties table.\",\n \"type\": \"Lookup\",\n \"dependsOn\": [\n {\n \"activity\": \"Metadata Integrity Checks\",\n \"dependencyConditions\": [\n \"Succeeded\"\n ]\n }\n ],\n \"policy\": {\n \"timeout\": \"7.00:00:00\",\n \"retry\": 0,\n \"retryIntervalInSeconds\": 30,\n \"secureOutput\": false,\n \"secureInput\": false\n },\n \"userProperties\": [],\n \"typeProperties\": {\n \"source\": {\n \"type\": \"AzureSqlSource\",\n \"sqlReaderStoredProcedureName\": \"[[procfwk].[GetPropertyValue]\",\n \"storedProcedureParameters\": {\n \"PropertyName\": {\n \"type\": \"String\",\n \"value\": \"TenantId\"\n }\n },\n \"queryTimeout\": \"02:00:00\",\n \"partitionOption\": \"None\"\n },\n \"dataset\": {\n \"referenceName\": \"GetSetMetadata\",\n \"type\": \"DatasetReference\",\n \"parameters\": {}\n },\n \"firstRowOnly\": true\n }\n },\n {\n \"name\": \"Get Subscription Id\",\n \"description\": \"Returning the Azure Subscription Id from the metadata properties table.\",\n \"type\": \"Lookup\",\n \"dependsOn\": [\n {\n \"activity\": \"Metadata Integrity Checks\",\n \"dependencyConditions\": [\n \"Succeeded\"\n ]\n }\n ],\n \"policy\": {\n \"timeout\": \"7.00:00:00\",\n \"retry\": 0,\n \"retryIntervalInSeconds\": 30,\n \"secureOutput\": false,\n \"secureInput\": true\n },\n \"userProperties\": [],\n \"typeProperties\": {\n \"source\": {\n \"type\": \"AzureSqlSource\",\n \"sqlReaderStoredProcedureName\": \"[[procfwk].[GetPropertyValue]\",\n \"storedProcedureParameters\": {\n \"PropertyName\": {\n \"type\": \"String\",\n \"value\": \"SubscriptionId\"\n }\n },\n \"queryTimeout\": \"02:00:00\",\n \"partitionOption\": \"None\"\n },\n \"dataset\": {\n \"referenceName\": \"GetSetMetadata\",\n \"type\": \"DatasetReference\",\n \"parameters\": {}\n }\n }\n },\n {\n \"name\": \"Metadata Integrity Checks\",\n \"description\": \"Performs a series of checks on all metadata held in the framework SQLDB. This is intended to raise errors before an execution run even starts.\",\n \"type\": \"Lookup\",\n \"dependsOn\": [\n {\n \"activity\": \"Execute Precursor\",\n \"dependencyConditions\": [\n \"Succeeded\"\n ]\n }\n ],\n \"policy\": {\n \"timeout\": \"7.00:00:00\",\n \"retry\": 0,\n \"retryIntervalInSeconds\": 30,\n \"secureOutput\": false,\n \"secureInput\": false\n },\n \"userProperties\": [],\n \"typeProperties\": {\n \"source\": {\n \"type\": \"AzureSqlSource\",\n \"sqlReaderStoredProcedureName\": \"[[procfwk].[CheckMetadataIntegrity]\",\n \"storedProcedureParameters\": {\n \"DebugMode\": {\n \"type\": \"Boolean\",\n \"value\": \"false\"\n }\n },\n \"queryTimeout\": \"02:00:00\",\n \"partitionOption\": \"None\"\n },\n \"dataset\": {\n \"referenceName\": \"GetSetMetadata\",\n \"type\": \"DatasetReference\",\n \"parameters\": {}\n },\n \"firstRowOnly\": false\n }\n },\n {\n \"name\": \"Clean Up Previous Run\",\n \"description\": \"Handle Worker pipelines that are reported as Running when the parent pipeline is called again. Get what the actual status of those pipelines is.\",\n \"type\": \"ForEach\",\n \"dependsOn\": [\n {\n \"activity\": \"Metadata Integrity Checks\",\n \"dependencyConditions\": [\n \"Succeeded\"\n ]\n }\n ],\n \"userProperties\": [],\n \"typeProperties\": {\n \"items\": {\n \"value\": \"@activity('Metadata Integrity Checks').output.value\",\n \"type\": \"Expression\"\n },\n \"isSequential\": false,\n \"batchCount\": 20,\n \"activities\": [\n {\n \"name\": \"Get SPN Details\",\n \"type\": \"Lookup\",\n \"dependsOn\": [],\n \"policy\": {\n \"timeout\": \"7.00:00:00\",\n \"retry\": 0,\n \"retryIntervalInSeconds\": 30,\n \"secureOutput\": false,\n \"secureInput\": false\n },\n \"userProperties\": [],\n \"typeProperties\": {\n \"source\": {\n \"type\": \"AzureSqlSource\",\n \"sqlReaderStoredProcedureName\": \"[[procfwk].[GetServicePrincipal]\",\n \"storedProcedureParameters\": {\n \"DataFactory\": {\n \"type\": \"String\",\n \"value\": {\n \"value\": \"@item().DataFactoryName\",\n \"type\": \"Expression\"\n }\n },\n \"PipelineName\": {\n \"type\": \"String\",\n \"value\": {\n \"value\": \"@item().PipelineName\",\n \"type\": \"Expression\"\n }\n }\n },\n \"queryTimeout\": \"02:00:00\",\n \"partitionOption\": \"None\"\n },\n \"dataset\": {\n \"referenceName\": \"GetSetMetadata\",\n \"type\": \"DatasetReference\",\n \"parameters\": {}\n }\n }\n },\n {\n \"name\": \"Log Pipeline Checking\",\n \"type\": \"SqlServerStoredProcedure\",\n \"dependsOn\": [],\n \"policy\": {\n \"timeout\": \"7.00:00:00\",\n \"retry\": 0,\n \"retryIntervalInSeconds\": 30,\n \"secureOutput\": false,\n \"secureInput\": false\n },\n \"userProperties\": [],\n \"typeProperties\": {\n \"storedProcedureName\": \"[[procfwk].[SetLogPipelineChecking]\",\n \"storedProcedureParameters\": {\n \"ExecutionId\": {\n \"value\": {\n \"value\": \"@item().LocalExecutionId\",\n \"type\": \"Expression\"\n },\n \"type\": \"Guid\"\n },\n \"PipelineId\": {\n \"value\": {\n \"value\": \"@item().PipelineId\",\n \"type\": \"Expression\"\n },\n \"type\": \"Int32\"\n },\n \"StageId\": {\n \"value\": {\n \"value\": \"@item().StageId\",\n \"type\": \"Expression\"\n },\n \"type\": \"Int32\"\n }\n }\n },\n \"linkedServiceName\": {\n \"referenceName\": \"SupportDatabase\",\n \"type\": \"LinkedServiceReference\"\n }\n },\n {\n \"name\": \"Get Pipeline Status\",\n \"type\": \"AzureFunctionActivity\",\n \"dependsOn\": [\n {\n \"activity\": \"Get SPN Details\",\n \"dependencyConditions\": [\n \"Succeeded\"\n ]\n },\n {\n \"activity\": \"Log Pipeline Checking\",\n \"dependencyConditions\": [\n \"Succeeded\"\n ]\n }\n ],\n \"policy\": {\n \"timeout\": \"7.00:00:00\",\n \"retry\": 0,\n \"retryIntervalInSeconds\": 30,\n \"secureOutput\": false,\n \"secureInput\": false\n },\n \"userProperties\": [],\n \"typeProperties\": {\n \"functionName\": \"CheckPipelineStatus\",\n \"method\": \"POST\",\n \"headers\": {},\n \"body\": {\n \"value\": \"@concat('\\n{\\n \\\"tenantId\\\": \\\"',item().TenantId,'\\\",\\n \\\"applicationId\\\": \\\"',activity('Get SPN Details').output.firstRow.Id,'\\\",\\n \\\"authenticationKey\\\": \\\"',activity('Get SPN Details').output.firstRow.Secret,'\\\",\\n \\\"subscriptionId\\\": \\\"',item().SubscriptionId,'\\\",\\n \\\"resourceGroup\\\": \\\"',item().ResourceGroupName,'\\\",\\n \\\"factoryName\\\": \\\"',item().DataFactoryName,'\\\",\\n \\\"pipelineName\\\": \\\"',item().PipelineName,'\\\",\\n \\\"runId\\\": \\\"',item().AdfPipelineRunId,'\\\"\\n}')\",\n \"type\": \"Expression\"\n }\n },\n \"linkedServiceName\": {\n \"referenceName\": \"FrameworkFunctions\",\n \"type\": \"LinkedServiceReference\"\n }\n },\n {\n \"name\": \"Set Pipeline Status\",\n \"description\": \"Update the metadata depending on the actual pipeline outcome. Using the status as the case.\",\n \"type\": \"Switch\",\n \"dependsOn\": [\n {\n \"activity\": \"Get Pipeline Status\",\n \"dependencyConditions\": [\n \"Succeeded\"\n ]\n }\n ],\n \"userProperties\": [],\n \"typeProperties\": {\n \"on\": {\n \"value\": \"@activity('Get Pipeline Status').output.Status\",\n \"type\": \"Expression\"\n },\n \"cases\": [\n {\n \"value\": \"Failed\",\n \"activities\": [\n {\n \"name\": \"Pipeline Status Failed\",\n \"description\": \"Updates the current execution table with a pipeline status of failed if the function outcome is failed. Also blocks pipelines in the downstream execution stage.\",\n \"type\": \"SqlServerStoredProcedure\",\n \"dependsOn\": [],\n \"policy\": {\n \"timeout\": \"7.00:00:00\",\n \"retry\": 0,\n \"retryIntervalInSeconds\": 30,\n \"secureOutput\": false,\n \"secureInput\": false\n },\n \"userProperties\": [],\n \"typeProperties\": {\n \"storedProcedureName\": \"[[procfwk].[SetLogPipelineFailed]\",\n \"storedProcedureParameters\": {\n \"ExecutionId\": {\n \"value\": {\n \"value\": \"@item().LocalExecutionId\",\n \"type\": \"Expression\"\n },\n \"type\": \"Guid\"\n },\n \"PipelineId\": {\n \"value\": {\n \"value\": \"@item().PipelineId\",\n \"type\": \"Expression\"\n },\n \"type\": \"Int32\"\n },\n \"RunId\": {\n \"value\": null,\n \"type\": \"Guid\"\n },\n \"StageId\": {\n \"value\": {\n \"value\": \"@item().StageId\",\n \"type\": \"Expression\"\n },\n \"type\": \"Int32\"\n }\n }\n },\n \"linkedServiceName\": {\n \"referenceName\": \"SupportDatabase\",\n \"type\": \"LinkedServiceReference\"\n }\n }\n ]\n },\n {\n \"value\": \"Succeeded\",\n \"activities\": [\n {\n \"name\": \"Pipeline Status Succeeded\",\n \"description\": \"Updates the current execution table with a pipeline status of success if the function outcome is succeeded.\",\n \"type\": \"SqlServerStoredProcedure\",\n \"dependsOn\": [],\n \"policy\": {\n \"timeout\": \"7.00:00:00\",\n \"retry\": 0,\n \"retryIntervalInSeconds\": 30,\n \"secureOutput\": false,\n \"secureInput\": false\n },\n \"userProperties\": [],\n \"typeProperties\": {\n \"storedProcedureName\": \"[[procfwk].[SetLogPipelineSuccess]\",\n \"storedProcedureParameters\": {\n \"ExecutionId\": {\n \"value\": {\n \"value\": \"@item().LocalExecutionId\",\n \"type\": \"Expression\"\n },\n \"type\": \"Guid\"\n },\n \"PipelineId\": {\n \"value\": {\n \"value\": \"@item().PipelineId\",\n \"type\": \"Expression\"\n },\n \"type\": \"Int32\"\n },\n \"StageId\": {\n \"value\": {\n \"value\": \"@item().StageId\",\n \"type\": \"Expression\"\n },\n \"type\": \"Int32\"\n }\n }\n },\n \"linkedServiceName\": {\n \"referenceName\": \"SupportDatabase\",\n \"type\": \"LinkedServiceReference\"\n }\n }\n ]\n },\n {\n \"value\": \"Queued\",\n \"activities\": [\n {\n \"name\": \"Pipeline Status Queued - Running\",\n \"description\": \"Updates the current execution table with a pipeline status of running if the function outcome is queued.\",\n \"type\": \"SqlServerStoredProcedure\",\n \"dependsOn\": [],\n \"policy\": {\n \"timeout\": \"7.00:00:00\",\n \"retry\": 0,\n \"retryIntervalInSeconds\": 30,\n \"secureOutput\": false,\n \"secureInput\": false\n },\n \"userProperties\": [],\n \"typeProperties\": {\n \"storedProcedureName\": \"[[procfwk].[SetLogPipelineRunning]\",\n \"storedProcedureParameters\": {\n \"ExecutionId\": {\n \"value\": {\n \"value\": \"@item().LocalExecutionId\",\n \"type\": \"Expression\"\n },\n \"type\": \"Guid\"\n },\n \"PipelineId\": {\n \"value\": {\n \"value\": \"@item().PipelineId\",\n \"type\": \"Expression\"\n },\n \"type\": \"Int32\"\n },\n \"StageId\": {\n \"value\": {\n \"value\": \"@item().StageId\",\n \"type\": \"Expression\"\n },\n \"type\": \"Int32\"\n }\n }\n },\n \"linkedServiceName\": {\n \"referenceName\": \"SupportDatabase\",\n \"type\": \"LinkedServiceReference\"\n }\n }\n ]\n },\n {\n \"value\": \"InProgress\",\n \"activities\": [\n {\n \"name\": \"Pipeline Status InProgress - Running\",\n \"description\": \"Updates the current execution table with a pipeline status of running if the function outcome is in progress.\",\n \"type\": \"SqlServerStoredProcedure\",\n \"dependsOn\": [],\n \"policy\": {\n \"timeout\": \"7.00:00:00\",\n \"retry\": 0,\n \"retryIntervalInSeconds\": 30,\n \"secureOutput\": false,\n \"secureInput\": false\n },\n \"userProperties\": [],\n \"typeProperties\": {\n \"storedProcedureName\": \"[[procfwk].[SetLogPipelineRunning]\",\n \"storedProcedureParameters\": {\n \"ExecutionId\": {\n \"value\": {\n \"value\": \"@item().LocalExecutionId\",\n \"type\": \"Expression\"\n },\n \"type\": \"Guid\"\n },\n \"PipelineId\": {\n \"value\": {\n \"value\": \"@item().PipelineId\",\n \"type\": \"Expression\"\n },\n \"type\": \"Int32\"\n },\n \"StageId\": {\n \"value\": {\n \"value\": \"@item().StageId\",\n \"type\": \"Expression\"\n },\n \"type\": \"Int32\"\n }\n }\n },\n \"linkedServiceName\": {\n \"referenceName\": \"SupportDatabase\",\n \"type\": \"LinkedServiceReference\"\n }\n }\n ]\n },\n {\n \"value\": \"Cancelled\",\n \"activities\": [\n {\n \"name\": \"Pipeline Status Cancelled\",\n \"description\": \"Updates the current execution table with a pipeline status of cancelled if the function outcome is cancelled.\",\n \"type\": \"SqlServerStoredProcedure\",\n \"dependsOn\": [],\n \"policy\": {\n \"timeout\": \"7.00:00:00\",\n \"retry\": 0,\n \"retryIntervalInSeconds\": 30,\n \"secureOutput\": false,\n \"secureInput\": false\n },\n \"userProperties\": [],\n \"typeProperties\": {\n \"storedProcedureName\": \"[[procfwk].[SetLogPipelineCancelled]\",\n \"storedProcedureParameters\": {\n \"ExecutionId\": {\n \"value\": {\n \"value\": \"@item().LocalExecutionId\",\n \"type\": \"Expression\"\n },\n \"type\": \"Guid\"\n },\n \"PipelineId\": {\n \"value\": {\n \"value\": \"@item().PipelineId\",\n \"type\": \"Expression\"\n },\n \"type\": \"Int32\"\n },\n \"StageId\": {\n \"value\": {\n \"value\": \"@item().StageId\",\n \"type\": \"Expression\"\n },\n \"type\": \"Int32\"\n }\n }\n },\n \"linkedServiceName\": {\n \"referenceName\": \"SupportDatabase\",\n \"type\": \"LinkedServiceReference\"\n }\n }\n ]\n }\n ],\n \"defaultActivities\": [\n {\n \"name\": \"Pipeline Status Unknown\",\n \"description\": \"Updates the current execution table with a pipeline status of unknown if the function returns an unexpected outcome.\",\n \"type\": \"SqlServerStoredProcedure\",\n \"dependsOn\": [],\n \"policy\": {\n \"timeout\": \"7.00:00:00\",\n \"retry\": 0,\n \"retryIntervalInSeconds\": 30,\n \"secureOutput\": false,\n \"secureInput\": false\n },\n \"userProperties\": [],\n \"typeProperties\": {\n \"storedProcedureName\": \"[[procfwk].[SetLogPipelineUnknown]\",\n \"storedProcedureParameters\": {\n \"ExecutionId\": {\n \"value\": {\n \"value\": \"@item().LocalExecutionId\",\n \"type\": \"Expression\"\n },\n \"type\": \"Guid\"\n },\n \"PipelineId\": {\n \"value\": {\n \"value\": \"@item().PipelineId\",\n \"type\": \"Expression\"\n },\n \"type\": \"Int32\"\n },\n \"StageId\": {\n \"value\": {\n \"value\": \"@item().StageId\",\n \"type\": \"Expression\"\n },\n \"type\": \"Int32\"\n }\n }\n },\n \"linkedServiceName\": {\n \"referenceName\": \"SupportDatabase\",\n \"type\": \"LinkedServiceReference\"\n }\n }\n ]\n }\n },\n {\n \"name\": \"Set Last Check DateTime\",\n \"description\": \"Update the current execution table with a date time from when the function last checked the pipeline status.\",\n \"type\": \"SqlServerStoredProcedure\",\n \"dependsOn\": [\n {\n \"activity\": \"Get Pipeline Status\",\n \"dependencyConditions\": [\n \"Succeeded\"\n ]\n }\n ],\n \"policy\": {\n \"timeout\": \"7.00:00:00\",\n \"retry\": 0,\n \"retryIntervalInSeconds\": 30,\n \"secureOutput\": false,\n \"secureInput\": false\n },\n \"userProperties\": [],\n \"typeProperties\": {\n \"storedProcedureName\": \"[[procfwk].[SetLogPipelineLastStatusCheck]\",\n \"storedProcedureParameters\": {\n \"ExecutionId\": {\n \"value\": {\n \"value\": \"@item().LocalExecutionId\",\n \"type\": \"Expression\"\n },\n \"type\": \"Guid\"\n },\n \"PipelineId\": {\n \"value\": {\n \"value\": \"@item().PipelineId\",\n \"type\": \"Expression\"\n },\n \"type\": \"Int32\"\n },\n \"StageId\": {\n \"value\": {\n \"value\": \"@item().StageId\",\n \"type\": \"Expression\"\n },\n \"type\": \"Int32\"\n }\n }\n },\n \"linkedServiceName\": {\n \"referenceName\": \"SupportDatabase\",\n \"type\": \"LinkedServiceReference\"\n }\n }\n ]\n }\n },\n {\n \"name\": \"Execute Precursor\",\n \"type\": \"SqlServerStoredProcedure\",\n \"dependsOn\": [],\n \"policy\": {\n \"timeout\": \"7.00:00:00\",\n \"retry\": 0,\n \"retryIntervalInSeconds\": 30,\n \"secureOutput\": false,\n \"secureInput\": false\n },\n \"userProperties\": [],\n \"typeProperties\": {\n \"storedProcedureName\": \"[[procfwk].[ExecutePrecursorProcedure]\"\n },\n \"linkedServiceName\": {\n \"referenceName\": \"SupportDatabase\",\n \"type\": \"LinkedServiceReference\"\n }\n },\n {\n \"name\": \"Set Execution Id\",\n \"description\": \"Set the local execution Id to a pipeline variable for each in several downstream activities.\",\n \"type\": \"SetVariable\",\n \"dependsOn\": [\n {\n \"activity\": \"Execution Wrapper\",\n \"dependencyConditions\": [\n \"Succeeded\"\n ]\n }\n ],\n \"userProperties\": [],\n \"typeProperties\": {\n \"variableName\": \"ExecutionId\",\n \"value\": {\n \"value\": \"@activity('Execution Wrapper').output.firstRow.ExecutionId\",\n \"type\": \"Expression\"\n }\n }\n }\n ],\n \"variables\": {\n \"ExecutionId\": {\n \"type\": \"String\"\n }\n },\n \"folder\": {\n \"name\": \"_ProcFwk\"\n },\n \"annotations\": [\n \"ADF.procfwk\",\n \"Parent\"\n ]\n },\n \"dependsOn\": [\n \"[concat(variables('factoryId'), '/datasets/GetSetMetadata')]\",\n \"[concat(variables('factoryId'), '/linkedServices/SupportDatabase')]\",\n \"[concat(variables('factoryId'), '/pipelines/03-Child')]\",\n \"[concat(variables('factoryId'), '/linkedServices/FrameworkFunctions')]\"\n ]\n },\n {\n \"name\": \"[concat(parameters('factoryName'), '/03-Child')]\",\n \"type\": \"Microsoft.DataFactory/factories/pipelines\",\n \"apiVersion\": \"2018-06-01\",\n \"properties\": {\n \"description\": \"ADF.procfwk child pipeline used to execute Worker pipelines within a given execution stage. This pipeline will be called once for each stage, then execute all Workers in parallel.\",\n \"activities\": [\n {\n \"name\": \"Get Pipelines\",\n \"description\": \"Returns all pipelines from the metadata to be executed within a given processing stage.\",\n \"type\": \"Lookup\",\n \"dependsOn\": [],\n \"policy\": {\n \"timeout\": \"7.00:00:00\",\n \"retry\": 0,\n \"retryIntervalInSeconds\": 30,\n \"secureOutput\": false,\n \"secureInput\": false\n },\n \"userProperties\": [],\n \"typeProperties\": {\n \"source\": {\n \"type\": \"AzureSqlSource\",\n \"sqlReaderStoredProcedureName\": \"[[procfwk].[GetPipelinesInStage]\",\n \"storedProcedureParameters\": {\n \"StageId\": {\n \"type\": \"Int32\",\n \"value\": {\n \"value\": \"@pipeline().parameters.StageId\",\n \"type\": \"Expression\"\n }\n }\n },\n \"queryTimeout\": \"02:00:00\",\n \"partitionOption\": \"None\"\n },\n \"dataset\": {\n \"referenceName\": \"GetSetMetadata\",\n \"type\": \"DatasetReference\",\n \"parameters\": {}\n },\n \"firstRowOnly\": false\n }\n },\n {\n \"name\": \"Execute Pipelines\",\n \"description\": \"Second level ForEach to run in parallel all pipelines within the stage. Items for iteration passed from the Get Pipelines lookup activity.\",\n \"type\": \"ForEach\",\n \"dependsOn\": [\n {\n \"activity\": \"Get Pipelines\",\n \"dependencyConditions\": [\n \"Succeeded\"\n ]\n }\n ],\n \"userProperties\": [],\n \"typeProperties\": {\n \"items\": {\n \"value\": \"@activity('Get Pipelines').output.value\",\n \"type\": \"Expression\"\n },\n \"isSequential\": false,\n \"batchCount\": 50,\n \"activities\": [\n {\n \"name\": \"Execute Pipeline\",\n \"description\": \"The lowest level executor with the metadata framework to call existing processing pipelines within Data Factory. The function called will block processing and wait for an outcome.\",\n \"type\": \"AzureFunctionActivity\",\n \"dependsOn\": [\n {\n \"activity\": \"Log Pipeline Running\",\n \"dependencyConditions\": [\n \"Succeeded\"\n ]\n },\n {\n \"activity\": \"Get Pipeline Params\",\n \"dependencyConditions\": [\n \"Succeeded\"\n ]\n },\n {\n \"activity\": \"Get SPN Details\",\n \"dependencyConditions\": [\n \"Succeeded\"\n ]\n }\n ],\n \"policy\": {\n \"timeout\": \"7.00:00:00\",\n \"retry\": 0,\n \"retryIntervalInSeconds\": 30,\n \"secureOutput\": false,\n \"secureInput\": false\n },\n \"userProperties\": [],\n \"typeProperties\": {\n \"functionName\": \"ExecutePipeline\",\n \"method\": \"POST\",\n \"headers\": {},\n \"body\": {\n \"value\": \"@concat('\\n{\\n\\t\\\"tenantId\\\": \\\"',pipeline().parameters.TenantId,'\\\",\\n\\t\\\"applicationId\\\": \\\"',activity('Get SPN Details').output.firstRow.Id,'\\\",\\n\\t\\\"authenticationKey\\\": \\\"',activity('Get SPN Details').output.firstRow.Secret,'\\\",\\n\\t\\\"subscriptionId\\\": \\\"',pipeline().parameters.SubscriptionId,'\\\",\\n\\t\\\"resourceGroup\\\": \\\"',item().ResourceGroupName,'\\\",\\n\\t\\\"factoryName\\\": \\\"',item().DataFactoryName,'\\\",\\n\\t\\\"pipelineName\\\": \\\"',item().PipelineName,'\\\"',activity('Get Pipeline Params').output.firstRow.Params,'\\n}')\",\n \"type\": \"Expression\"\n }\n },\n \"linkedServiceName\": {\n \"referenceName\": \"FrameworkFunctions\",\n \"type\": \"LinkedServiceReference\"\n }\n },\n {\n \"name\": \"Get Pipeline Params\",\n \"description\": \"Returns any parameters from metadata required for the processing pipeline being called. The output can be an empty string if no parameters are required.\",\n \"type\": \"Lookup\",\n \"dependsOn\": [],\n \"policy\": {\n \"timeout\": \"7.00:00:00\",\n \"retry\": 0,\n \"retryIntervalInSeconds\": 30,\n \"secureOutput\": false,\n \"secureInput\": false\n },\n \"userProperties\": [],\n \"typeProperties\": {\n \"source\": {\n \"type\": \"AzureSqlSource\",\n \"sqlReaderStoredProcedureName\": \"[[procfwk].[GetPipelineParameters]\",\n \"storedProcedureParameters\": {\n \"PipelineId\": {\n \"type\": \"Int32\",\n \"value\": {\n \"value\": \"@item().PipelineId\",\n \"type\": \"Expression\"\n }\n }\n },\n \"queryTimeout\": \"02:00:00\",\n \"partitionOption\": \"None\"\n },\n \"dataset\": {\n \"referenceName\": \"GetSetMetadata\",\n \"type\": \"DatasetReference\",\n \"parameters\": {}\n }\n }\n },\n {\n \"name\": \"Log Pipeline Running\",\n \"description\": \"Sets the current pipeline with a status of running within the current execution database table.\",\n \"type\": \"SqlServerStoredProcedure\",\n \"dependsOn\": [],\n \"policy\": {\n \"timeout\": \"7.00:00:00\",\n \"retry\": 0,\n \"retryIntervalInSeconds\": 30,\n \"secureOutput\": false,\n \"secureInput\": false\n },\n \"userProperties\": [],\n \"typeProperties\": {\n \"storedProcedureName\": \"[[procfwk].[SetLogPipelineRunning]\",\n \"storedProcedureParameters\": {\n \"ExecutionId\": {\n \"value\": {\n \"value\": \"@pipeline().parameters.ExecutionId\",\n \"type\": \"Expression\"\n },\n \"type\": \"Guid\"\n },\n \"PipelineId\": {\n \"value\": {\n \"value\": \"@item().PipelineId\",\n \"type\": \"Expression\"\n },\n \"type\": \"Int32\"\n },\n \"StageId\": {\n \"value\": {\n \"value\": \"@pipeline().parameters.StageId\",\n \"type\": \"Expression\"\n },\n \"type\": \"Int32\"\n }\n }\n },\n \"linkedServiceName\": {\n \"referenceName\": \"SupportDatabase\",\n \"type\": \"LinkedServiceReference\"\n }\n },\n {\n \"name\": \"Get SPN Details\",\n \"description\": \"Return the SPN ID and Secret for the processing pipeline being executed. Called at this level as each pipeline can have a different SPN.\",\n \"type\": \"Lookup\",\n \"dependsOn\": [],\n \"policy\": {\n \"timeout\": \"7.00:00:00\",\n \"retry\": 0,\n \"retryIntervalInSeconds\": 30,\n \"secureOutput\": false,\n \"secureInput\": false\n },\n \"userProperties\": [],\n \"typeProperties\": {\n \"source\": {\n \"type\": \"AzureSqlSource\",\n \"sqlReaderStoredProcedureName\": \"[[procfwk].[GetServicePrincipal]\",\n \"storedProcedureParameters\": {\n \"DataFactory\": {\n \"type\": \"String\",\n \"value\": {\n \"value\": \"@item().DataFactoryName\",\n \"type\": \"Expression\"\n }\n },\n \"PipelineName\": {\n \"type\": \"String\",\n \"value\": {\n \"value\": \"@item().PipelineName\",\n \"type\": \"Expression\"\n }\n }\n },\n \"queryTimeout\": \"02:00:00\",\n \"partitionOption\": \"None\"\n },\n \"dataset\": {\n \"referenceName\": \"GetSetMetadata\",\n \"type\": \"DatasetReference\",\n \"parameters\": {}\n }\n }\n },\n {\n \"name\": \"Log Activity Failure\",\n \"description\": \"Handle true failures from calling out to the Azure Function and update the current execution table accordingly so a restart can occur.\",\n \"type\": \"SqlServerStoredProcedure\",\n \"dependsOn\": [\n {\n \"activity\": \"Execute Pipeline\",\n \"dependencyConditions\": [\n \"Failed\"\n ]\n }\n ],\n \"policy\": {\n \"timeout\": \"7.00:00:00\",\n \"retry\": 0,\n \"retryIntervalInSeconds\": 30,\n \"secureOutput\": false,\n \"secureInput\": false\n },\n \"userProperties\": [],\n \"typeProperties\": {\n \"storedProcedureName\": \"[[procfwk].[SetLogActivityFailed]\",\n \"storedProcedureParameters\": {\n \"ExecutionId\": {\n \"value\": {\n \"value\": \"@pipeline().parameters.ExecutionId\",\n \"type\": \"Expression\"\n },\n \"type\": \"Guid\"\n },\n \"PipelineId\": {\n \"value\": {\n \"value\": \"@item().PipelineId\",\n \"type\": \"Expression\"\n },\n \"type\": \"Int32\"\n },\n \"StageId\": {\n \"value\": {\n \"value\": \"@pipeline().parameters.StageId\",\n \"type\": \"Expression\"\n },\n \"type\": \"Int32\"\n },\n \"CallingActivity\": {\n \"value\": \"ExecutePipeline\",\n \"type\": \"String\"\n }\n }\n },\n \"linkedServiceName\": {\n \"referenceName\": \"SupportDatabase\",\n \"type\": \"LinkedServiceReference\"\n }\n },\n {\n \"name\": \"Running Pipeline Handler\",\n \"type\": \"ExecutePipeline\",\n \"dependsOn\": [\n {\n \"activity\": \"Execute Pipeline\",\n \"dependencyConditions\": [\n \"Succeeded\"\n ]\n }\n ],\n \"userProperties\": [],\n \"typeProperties\": {\n \"pipeline\": {\n \"referenceName\": \"04-Infant\",\n \"type\": \"PipelineReference\"\n },\n \"waitOnCompletion\": true,\n \"parameters\": {\n \"tenantId\": {\n \"value\": \"@pipeline().parameters.TenantId\",\n \"type\": \"Expression\"\n },\n \"applicationId\": {\n \"value\": \"@activity('Get SPN Details').output.firstRow.Id\",\n \"type\": \"Expression\"\n },\n \"authenticationKey\": {\n \"value\": \"@activity('Get SPN Details').output.firstRow.Secret\",\n \"type\": \"Expression\"\n },\n \"subscriptionId\": {\n \"value\": \"@pipeline().parameters.SubscriptionId\",\n \"type\": \"Expression\"\n },\n \"resourceGroup\": {\n \"value\": \"@item().ResourceGroupName\",\n \"type\": \"Expression\"\n },\n \"factoryName\": {\n \"value\": \"@item().DataFactoryName\",\n \"type\": \"Expression\"\n },\n \"pipelineName\": {\n \"value\": \"@item().PipelineName\",\n \"type\": \"Expression\"\n },\n \"runId\": {\n \"value\": \"@activity('Execute Pipeline').output.RunId\",\n \"type\": \"Expression\"\n },\n \"executionId\": {\n \"value\": \"@pipeline().parameters.ExecutionId\",\n \"type\": \"Expression\"\n },\n \"stageId\": {\n \"value\": \"@pipeline().parameters.StageId\",\n \"type\": \"Expression\"\n },\n \"pipelineId\": {\n \"value\": \"@item().PipelineId\",\n \"type\": \"Expression\"\n }\n }\n }\n },\n {\n \"name\": \"Set Run Id\",\n \"description\": \"Provide the actual ADF run ID back to the current execution table for long term logging and alignment between the metadata other Azure monitoring tools.\",\n \"type\": \"SqlServerStoredProcedure\",\n \"dependsOn\": [\n {\n \"activity\": \"Execute Pipeline\",\n \"dependencyConditions\": [\n \"Succeeded\"\n ]\n }\n ],\n \"policy\": {\n \"timeout\": \"7.00:00:00\",\n \"retry\": 0,\n \"retryIntervalInSeconds\": 30,\n \"secureOutput\": false,\n \"secureInput\": false\n },\n \"userProperties\": [],\n \"typeProperties\": {\n \"storedProcedureName\": \"[[procfwk].[SetLogPipelineRunId]\",\n \"storedProcedureParameters\": {\n \"ExecutionId\": {\n \"value\": {\n \"value\": \"@pipeline().parameters.ExecutionId\",\n \"type\": \"Expression\"\n },\n \"type\": \"Guid\"\n },\n \"PipelineId\": {\n \"value\": {\n \"value\": \"@item().PipelineId\",\n \"type\": \"Expression\"\n },\n \"type\": \"Int32\"\n },\n \"RunId\": {\n \"value\": {\n \"value\": \"@activity('Execute Pipeline').output.RunId\",\n \"type\": \"Expression\"\n },\n \"type\": \"Guid\"\n },\n \"StageId\": {\n \"value\": {\n \"value\": \"@pipeline().parameters.StageId\",\n \"type\": \"Expression\"\n },\n \"type\": \"Int32\"\n }\n }\n },\n \"linkedServiceName\": {\n \"referenceName\": \"SupportDatabase\",\n \"type\": \"LinkedServiceReference\"\n }\n },\n {\n \"name\": \"Check For Alerts\",\n \"description\": \"Checks the properties tables and if any recipients in the database require alerts sending for the current pipeline ID.\",\n \"type\": \"Lookup\",\n \"dependsOn\": [\n {\n \"activity\": \"Set Run Id\",\n \"dependencyConditions\": [\n \"Succeeded\"\n ]\n },\n {\n \"activity\": \"Running Pipeline Handler\",\n \"dependencyConditions\": [\n \"Completed\"\n ]\n }\n ],\n \"policy\": {\n \"timeout\": \"7.00:00:00\",\n \"retry\": 0,\n \"retryIntervalInSeconds\": 30,\n \"secureOutput\": false,\n \"secureInput\": false\n },\n \"userProperties\": [],\n \"typeProperties\": {\n \"source\": {\n \"type\": \"AzureSqlSource\",\n \"sqlReaderStoredProcedureName\": \"[[procfwk].[CheckForEmailAlerts]\",\n \"storedProcedureParameters\": {\n \"PipelineId\": {\n \"type\": \"Int32\",\n \"value\": {\n \"value\": \"@item().PipelineId\",\n \"type\": \"Expression\"\n }\n }\n },\n \"queryTimeout\": \"02:00:00\",\n \"partitionOption\": \"None\"\n },\n \"dataset\": {\n \"referenceName\": \"GetSetMetadata\",\n \"type\": \"DatasetReference\",\n \"parameters\": {}\n },\n \"firstRowOnly\": true\n }\n },\n {\n \"name\": \"Send Alerts\",\n \"description\": \"True = alerts need sending.\\nFalse = do nothing.\",\n \"type\": \"IfCondition\",\n \"dependsOn\": [\n {\n \"activity\": \"Check For Alerts\",\n \"dependencyConditions\": [\n \"Succeeded\"\n ]\n }\n ],\n \"userProperties\": [],\n \"typeProperties\": {\n \"expression\": {\n \"value\": \"@activity('Check For Alerts').output.firstRow.SendAlerts\",\n \"type\": \"Expression\"\n },\n \"ifTrueActivities\": [\n {\n \"name\": \"Get Email Parts\",\n \"type\": \"Lookup\",\n \"dependsOn\": [],\n \"policy\": {\n \"timeout\": \"7.00:00:00\",\n \"retry\": 0,\n \"retryIntervalInSeconds\": 30,\n \"secureOutput\": false,\n \"secureInput\": false\n },\n \"userProperties\": [],\n \"typeProperties\": {\n \"source\": {\n \"type\": \"AzureSqlSource\",\n \"sqlReaderStoredProcedureName\": \"[[procfwk].[GetEmailAlertParts]\",\n \"storedProcedureParameters\": {\n \"PipelineId\": {\n \"type\": \"Int32\",\n \"value\": {\n \"value\": \"@item().PipelineId\",\n \"type\": \"Expression\"\n }\n }\n },\n \"queryTimeout\": \"02:00:00\",\n \"partitionOption\": \"None\"\n },\n \"dataset\": {\n \"referenceName\": \"GetSetMetadata\",\n \"type\": \"DatasetReference\",\n \"parameters\": {}\n },\n \"firstRowOnly\": true\n }\n },\n {\n \"name\": \"Send Email\",\n \"type\": \"AzureFunctionActivity\",\n \"dependsOn\": [\n {\n \"activity\": \"Get Email Parts\",\n \"dependencyConditions\": [\n \"Succeeded\"\n ]\n }\n ],\n \"policy\": {\n \"timeout\": \"7.00:00:00\",\n \"retry\": 0,\n \"retryIntervalInSeconds\": 30,\n \"secureOutput\": false,\n \"secureInput\": false\n },\n \"userProperties\": [],\n \"typeProperties\": {\n \"functionName\": \"SendEmail\",\n \"method\": \"POST\",\n \"headers\": {},\n \"body\": {\n \"value\": \"@activity('Get Email Parts').output.firstRow\",\n \"type\": \"Expression\"\n }\n },\n \"linkedServiceName\": {\n \"referenceName\": \"FrameworkFunctions\",\n \"type\": \"LinkedServiceReference\"\n }\n }\n ]\n }\n }\n ]\n }\n }\n ],\n \"parameters\": {\n \"StageId\": {\n \"type\": \"int\"\n },\n \"ExecutionId\": {\n \"type\": \"string\"\n },\n \"TenantId\": {\n \"type\": \"string\"\n },\n \"SubscriptionId\": {\n \"type\": \"string\"\n }\n },\n \"folder\": {\n \"name\": \"_ProcFwk\"\n },\n \"annotations\": [\n \"ADF.procfwk\",\n \"Child\"\n ]\n },\n \"dependsOn\": [\n \"[concat(variables('factoryId'), '/datasets/GetSetMetadata')]\",\n \"[concat(variables('factoryId'), '/linkedServices/FrameworkFunctions')]\",\n \"[concat(variables('factoryId'), '/linkedServices/SupportDatabase')]\",\n \"[concat(variables('factoryId'), '/pipelines/04-Infant')]\"\n ]\n },\n {\n \"name\": \"[concat(parameters('factoryName'), '/04-Infant')]\",\n \"type\": \"Microsoft.DataFactory/factories/pipelines\",\n \"apiVersion\": \"2018-06-01\",\n \"properties\": {\n \"description\": \"ADF.procfwk infant pipeline used to check when the processing pipeline called by the Child completes and passes the resulting status back to the metadata database.\",\n \"activities\": [\n {\n \"name\": \"Wait Until Pipeline Completes\",\n \"description\": \"Loops until the Worker pipeline called completes.\\n\\nSimple status:\\n- Running = new iteration.\\n- Done = break.\",\n \"type\": \"Until\",\n \"dependsOn\": [\n {\n \"activity\": \"Get Wait Duration\",\n \"dependencyConditions\": [\n \"Succeeded\"\n ]\n }\n ],\n \"userProperties\": [],\n \"typeProperties\": {\n \"expression\": {\n \"value\": \"@variables('WorkerPipelineState')\",\n \"type\": \"Expression\"\n },\n \"activities\": [\n {\n \"name\": \"Get Pipeline Status\",\n \"description\": \"Checks the status of a given processing pipeline and provides the value for the downstream framework activities to act upon.\",\n \"type\": \"AzureFunctionActivity\",\n \"dependsOn\": [],\n \"policy\": {\n \"timeout\": \"7.00:00:00\",\n \"retry\": 0,\n \"retryIntervalInSeconds\": 30,\n \"secureOutput\": false,\n \"secureInput\": false\n },\n \"userProperties\": [],\n \"typeProperties\": {\n \"functionName\": \"CheckPipelineStatus\",\n \"method\": \"POST\",\n \"headers\": {},\n \"body\": {\n \"value\": \"@concat('\\n{\\n \\\"tenantId\\\": \\\"',pipeline().parameters.TenantId,'\\\",\\n \\\"applicationId\\\": \\\"',pipeline().parameters.applicationId,'\\\",\\n \\\"authenticationKey\\\": \\\"',pipeline().parameters.authenticationKey,'\\\",\\n \\\"subscriptionId\\\": \\\"',pipeline().parameters.subscriptionId,'\\\",\\n \\\"resourceGroup\\\": \\\"',pipeline().parameters.resourceGroup,'\\\",\\n \\\"factoryName\\\": \\\"',pipeline().parameters.factoryName,'\\\",\\n \\\"pipelineName\\\": \\\"',pipeline().parameters.pipelineName,'\\\",\\n \\\"runId\\\": \\\"',pipeline().parameters.runId,'\\\"\\n}')\",\n \"type\": \"Expression\"\n }\n },\n \"linkedServiceName\": {\n \"referenceName\": \"FrameworkFunctions\",\n \"type\": \"LinkedServiceReference\"\n }\n },\n {\n \"name\": \"Wait If Running\",\n \"description\": \"True = Do nothing.\\nFalse = Wait, before the next iteration.\",\n \"type\": \"IfCondition\",\n \"dependsOn\": [\n {\n \"activity\": \"Set Worker State\",\n \"dependencyConditions\": [\n \"Succeeded\"\n ]\n }\n ],\n \"userProperties\": [],\n \"typeProperties\": {\n \"expression\": {\n \"value\": \"@variables('WorkerPipelineState')\",\n \"type\": \"Expression\"\n },\n \"ifFalseActivities\": [\n {\n \"name\": \"Wait for Pipeline\",\n \"description\": \"The processing pipeline is still running so Wait before checking its status again.\",\n \"type\": \"Wait\",\n \"dependsOn\": [],\n \"userProperties\": [],\n \"typeProperties\": {\n \"waitTimeInSeconds\": {\n \"value\": \"@activity('Get Wait Duration').output.firstRow.PropertyValue\",\n \"type\": \"Expression\"\n }\n }\n }\n ]\n }\n },\n {\n \"name\": \"Set Last Check DateTime\",\n \"description\": \"Update the current execution table with a date time from when the Worker pipeline status was last checked as part of the Until iterations.\",\n \"type\": \"SqlServerStoredProcedure\",\n \"dependsOn\": [\n {\n \"activity\": \"Get Pipeline Status\",\n \"dependencyConditions\": [\n \"Succeeded\"\n ]\n }\n ],\n \"policy\": {\n \"timeout\": \"7.00:00:00\",\n \"retry\": 0,\n \"retryIntervalInSeconds\": 30,\n \"secureOutput\": false,\n \"secureInput\": false\n },\n \"userProperties\": [],\n \"typeProperties\": {\n \"storedProcedureName\": \"[[procfwk].[SetLogPipelineLastStatusCheck]\",\n \"storedProcedureParameters\": {\n \"ExecutionId\": {\n \"value\": {\n \"value\": \"@pipeline().parameters.executionId\",\n \"type\": \"Expression\"\n },\n \"type\": \"Guid\"\n },\n \"PipelineId\": {\n \"value\": {\n \"value\": \"@pipeline().parameters.pipelineId\",\n \"type\": \"Expression\"\n },\n \"type\": \"Int32\"\n },\n \"StageId\": {\n \"value\": {\n \"value\": \"@pipeline().parameters.stageId\",\n \"type\": \"Expression\"\n },\n \"type\": \"Int32\"\n }\n }\n },\n \"linkedServiceName\": {\n \"referenceName\": \"SupportDatabase\",\n \"type\": \"LinkedServiceReference\"\n }\n },\n {\n \"name\": \"Log Activity Failure\",\n \"description\": \"Report to the current execution table that the framework pipeline activity has failed. This failure is outside of the scope of the framework and is probably related to a wider platform problem.\",\n \"type\": \"SqlServerStoredProcedure\",\n \"dependsOn\": [\n {\n \"activity\": \"Get Pipeline Status\",\n \"dependencyConditions\": [\n \"Failed\"\n ]\n }\n ],\n \"policy\": {\n \"timeout\": \"7.00:00:00\",\n \"retry\": 0,\n \"retryIntervalInSeconds\": 30,\n \"secureOutput\": false,\n \"secureInput\": false\n },\n \"userProperties\": [],\n \"typeProperties\": {\n \"storedProcedureName\": \"[[procfwk].[SetLogActivityFailed]\",\n \"storedProcedureParameters\": {\n \"CallingActivity\": {\n \"value\": \"GetPipelineStatus\",\n \"type\": \"String\"\n },\n \"ExecutionId\": {\n \"value\": {\n \"value\": \"@pipeline().parameters.executionId\",\n \"type\": \"Expression\"\n },\n \"type\": \"Guid\"\n },\n \"PipelineId\": {\n \"value\": {\n \"value\": \"@pipeline().parameters.pipelineId\",\n \"type\": \"Expression\"\n },\n \"type\": \"Int32\"\n },\n \"StageId\": {\n \"value\": {\n \"value\": \"@pipeline().parameters.stageId\",\n \"type\": \"Expression\"\n },\n \"type\": \"Int32\"\n }\n }\n },\n \"linkedServiceName\": {\n \"referenceName\": \"SupportDatabase\",\n \"type\": \"LinkedServiceReference\"\n }\n },\n {\n \"name\": \"Set Worker State\",\n \"description\": \"Set the bool state of the Worker pipeline to be used by the Until and If expressions. True = Complete, False = Running.\",\n \"type\": \"SetVariable\",\n \"dependsOn\": [\n {\n \"activity\": \"Get Pipeline Status\",\n \"dependencyConditions\": [\n \"Succeeded\"\n ]\n }\n ],\n \"userProperties\": [],\n \"typeProperties\": {\n \"variableName\": \"WorkerPipelineState\",\n \"value\": {\n \"value\": \"@equals('Done',activity('Get Pipeline Status').output.SimpleStatus)\",\n \"type\": \"Expression\"\n }\n }\n }\n ],\n \"timeout\": \"7.00:00:00\"\n }\n },\n {\n \"name\": \"Set Pipeline Result\",\n \"description\": \"Receives the outcome from the function execution for a given processing pipeline and updates the current execution table with different pipelines status values depending on the result (case).\",\n \"type\": \"Switch\",\n \"dependsOn\": [\n {\n \"activity\": \"Wait Until Pipeline Completes\",\n \"dependencyConditions\": [\n \"Completed\"\n ]\n }\n ],\n \"userProperties\": [],\n \"typeProperties\": {\n \"on\": {\n \"value\": \"@activity('Get Pipeline Status').output.Status\",\n \"type\": \"Expression\"\n },\n \"cases\": [\n {\n \"value\": \"Succeeded\",\n \"activities\": [\n {\n \"name\": \"Pipeline Status Succeeded\",\n \"description\": \"Updates the current execution table with a pipeline status of success if the function outcome is succeeded.\",\n \"type\": \"SqlServerStoredProcedure\",\n \"dependsOn\": [],\n \"policy\": {\n \"timeout\": \"7.00:00:00\",\n \"retry\": 0,\n \"retryIntervalInSeconds\": 30,\n \"secureOutput\": false,\n \"secureInput\": false\n },\n \"userProperties\": [],\n \"typeProperties\": {\n \"storedProcedureName\": \"[[procfwk].[SetLogPipelineSuccess]\",\n \"storedProcedureParameters\": {\n \"ExecutionId\": {\n \"value\": {\n \"value\": \"@pipeline().parameters.executionId\",\n \"type\": \"Expression\"\n },\n \"type\": \"Guid\"\n },\n \"PipelineId\": {\n \"value\": {\n \"value\": \"@pipeline().parameters.pipelineId\",\n \"type\": \"Expression\"\n },\n \"type\": \"Int32\"\n },\n \"StageId\": {\n \"value\": {\n \"value\": \"@pipeline().parameters.stageId\",\n \"type\": \"Expression\"\n },\n \"type\": \"Int32\"\n }\n }\n },\n \"linkedServiceName\": {\n \"referenceName\": \"SupportDatabase\",\n \"type\": \"LinkedServiceReference\"\n }\n }\n ]\n },\n {\n \"value\": \"Failed\",\n \"activities\": [\n {\n \"name\": \"Pipeline Status Failed\",\n \"description\": \"Updates the current execution table with a pipeline status of failed if the function outcome is failed. Also blocks pipelines in the downstream execution stage.\",\n \"type\": \"SqlServerStoredProcedure\",\n \"dependsOn\": [],\n \"policy\": {\n \"timeout\": \"7.00:00:00\",\n \"retry\": 0,\n \"retryIntervalInSeconds\": 30,\n \"secureOutput\": false,\n \"secureInput\": false\n },\n \"userProperties\": [],\n \"typeProperties\": {\n \"storedProcedureName\": \"[[procfwk].[SetLogPipelineFailed]\",\n \"storedProcedureParameters\": {\n \"ExecutionId\": {\n \"value\": {\n \"value\": \"@pipeline().parameters.executionId\",\n \"type\": \"Expression\"\n },\n \"type\": \"Guid\"\n },\n \"PipelineId\": {\n \"value\": {\n \"value\": \"@pipeline().parameters.pipelineId\",\n \"type\": \"Expression\"\n },\n \"type\": \"Int32\"\n },\n \"RunId\": {\n \"value\": {\n \"value\": \"@activity('Get Pipeline Status').output.RunId\",\n \"type\": \"Expression\"\n },\n \"type\": \"Guid\"\n },\n \"StageId\": {\n \"value\": {\n \"value\": \"@pipeline().parameters.stageId\",\n \"type\": \"Expression\"\n },\n \"type\": \"Int32\"\n }\n }\n },\n \"linkedServiceName\": {\n \"referenceName\": \"SupportDatabase\",\n \"type\": \"LinkedServiceReference\"\n }\n },\n {\n \"name\": \"Get Error Details\",\n \"description\": \"Get the activity error details for the run ID of the worker pipeline called. Returns an array of all errors.\",\n \"type\": \"AzureFunctionActivity\",\n \"dependsOn\": [],\n \"policy\": {\n \"timeout\": \"7.00:00:00\",\n \"retry\": 0,\n \"retryIntervalInSeconds\": 30,\n \"secureOutput\": false,\n \"secureInput\": false\n },\n \"userProperties\": [],\n \"typeProperties\": {\n \"functionName\": \"GetActivityErrors\",\n \"method\": \"POST\",\n \"headers\": {},\n \"body\": {\n \"value\": \"@concat('\\n{\\n \\\"tenantId\\\": \\\"',pipeline().parameters.TenantId,'\\\",\\n \\\"applicationId\\\": \\\"',pipeline().parameters.applicationId,'\\\",\\n \\\"authenticationKey\\\": \\\"',pipeline().parameters.authenticationKey,'\\\",\\n \\\"subscriptionId\\\": \\\"',pipeline().parameters.subscriptionId,'\\\",\\n \\\"resourceGroup\\\": \\\"',pipeline().parameters.resourceGroup,'\\\",\\n \\\"factoryName\\\": \\\"',pipeline().parameters.factoryName,'\\\",\\n \\\"pipelineName\\\": \\\"',pipeline().parameters.pipelineName,'\\\",\\n \\\"runId\\\": \\\"',pipeline().parameters.runId,'\\\"\\n}')\",\n \"type\": \"Expression\"\n }\n },\n \"linkedServiceName\": {\n \"referenceName\": \"FrameworkFunctions\",\n \"type\": \"LinkedServiceReference\"\n }\n },\n {\n \"name\": \"Log Error Details\",\n \"description\": \"Parses pipeline error details and persists them to the metadata database error log table.\",\n \"type\": \"SqlServerStoredProcedure\",\n \"dependsOn\": [\n {\n \"activity\": \"Get Error Details\",\n \"dependencyConditions\": [\n \"Succeeded\"\n ]\n }\n ],\n \"policy\": {\n \"timeout\": \"7.00:00:00\",\n \"retry\": 0,\n \"retryIntervalInSeconds\": 30,\n \"secureOutput\": false,\n \"secureInput\": false\n },\n \"userProperties\": [],\n \"typeProperties\": {\n \"storedProcedureName\": \"[[procfwk].[SetErrorLogDetails]\",\n \"storedProcedureParameters\": {\n \"JsonErrorDetails\": {\n \"value\": {\n \"value\": \"@string(activity('Get Error Details').output)\",\n \"type\": \"Expression\"\n },\n \"type\": \"String\"\n },\n \"LocalExecutionId\": {\n \"value\": {\n \"value\": \"@pipeline().parameters.executionId\",\n \"type\": \"Expression\"\n },\n \"type\": \"Guid\"\n }\n }\n },\n \"linkedServiceName\": {\n \"referenceName\": \"SupportDatabase\",\n \"type\": \"LinkedServiceReference\"\n }\n }\n ]\n },\n {\n \"value\": \"Cancelled\",\n \"activities\": [\n {\n \"name\": \"Pipeline Status Cancelled\",\n \"description\": \"Updates the current execution table with a pipeline status of cancelled if the function outcome is cancelled.\",\n \"type\": \"SqlServerStoredProcedure\",\n \"dependsOn\": [],\n \"policy\": {\n \"timeout\": \"7.00:00:00\",\n \"retry\": 0,\n \"retryIntervalInSeconds\": 30,\n \"secureOutput\": false,\n \"secureInput\": false\n },\n \"userProperties\": [],\n \"typeProperties\": {\n \"storedProcedureName\": \"[[procfwk].[SetLogPipelineCancelled]\",\n \"storedProcedureParameters\": {\n \"ExecutionId\": {\n \"value\": {\n \"value\": \"@pipeline().parameters.executionId\",\n \"type\": \"Expression\"\n },\n \"type\": \"Guid\"\n },\n \"PipelineId\": {\n \"value\": {\n \"value\": \"@pipeline().parameters.pipelineId\",\n \"type\": \"Expression\"\n },\n \"type\": \"Int32\"\n },\n \"StageId\": {\n \"value\": {\n \"value\": \"@pipeline().parameters.stageId\",\n \"type\": \"Expression\"\n },\n \"type\": \"Int32\"\n }\n }\n },\n \"linkedServiceName\": {\n \"referenceName\": \"SupportDatabase\",\n \"type\": \"LinkedServiceReference\"\n }\n }\n ]\n }\n ],\n \"defaultActivities\": [\n {\n \"name\": \"Pipeline Status Unknown\",\n \"description\": \"Updates the current execution table with a pipeline status of unknown if the function returns an unexpected outcome.\",\n \"type\": \"SqlServerStoredProcedure\",\n \"dependsOn\": [],\n \"policy\": {\n \"timeout\": \"7.00:00:00\",\n \"retry\": 0,\n \"retryIntervalInSeconds\": 30,\n \"secureOutput\": false,\n \"secureInput\": false\n },\n \"userProperties\": [],\n \"typeProperties\": {\n \"storedProcedureName\": \"[[procfwk].[SetLogPipelineUnknown]\",\n \"storedProcedureParameters\": {\n \"ExecutionId\": {\n \"value\": {\n \"value\": \"@pipeline().parameters.executionId\",\n \"type\": \"Expression\"\n },\n \"type\": \"Guid\"\n },\n \"PipelineId\": {\n \"value\": {\n \"value\": \"@pipeline().parameters.pipelineId\",\n \"type\": \"Expression\"\n },\n \"type\": \"Int32\"\n },\n \"StageId\": {\n \"value\": {\n \"value\": \"@pipeline().parameters.stageId\",\n \"type\": \"Expression\"\n },\n \"type\": \"Int32\"\n }\n }\n },\n \"linkedServiceName\": {\n \"referenceName\": \"SupportDatabase\",\n \"type\": \"LinkedServiceReference\"\n }\n }\n ]\n }\n },\n {\n \"name\": \"Get Wait Duration\",\n \"description\": \"Return wait duration in seconds from database properties table to be used during each Until iteration when the Worker pipeline is still running.\",\n \"type\": \"Lookup\",\n \"dependsOn\": [],\n \"policy\": {\n \"timeout\": \"7.00:00:00\",\n \"retry\": 0,\n \"retryIntervalInSeconds\": 30,\n \"secureOutput\": false,\n \"secureInput\": false\n },\n \"userProperties\": [],\n \"typeProperties\": {\n \"source\": {\n \"type\": \"AzureSqlSource\",\n \"sqlReaderStoredProcedureName\": \"[[procfwk].[GetPropertyValue]\",\n \"storedProcedureParameters\": {\n \"PropertyName\": {\n \"type\": \"String\",\n \"value\": \"PipelineStatusCheckDuration\"\n }\n },\n \"queryTimeout\": \"02:00:00\",\n \"partitionOption\": \"None\"\n },\n \"dataset\": {\n \"referenceName\": \"GetSetMetadata\",\n \"type\": \"DatasetReference\",\n \"parameters\": {}\n }\n }\n }\n ],\n \"parameters\": {\n \"tenantId\": {\n \"type\": \"string\"\n },\n \"applicationId\": {\n \"type\": \"string\"\n },\n \"authenticationKey\": {\n \"type\": \"string\"\n },\n \"subscriptionId\": {\n \"type\": \"string\"\n },\n \"resourceGroup\": {\n \"type\": \"string\"\n },\n \"factoryName\": {\n \"type\": \"string\"\n },\n \"pipelineName\": {\n \"type\": \"string\"\n },\n \"runId\": {\n \"type\": \"string\"\n },\n \"executionId\": {\n \"type\": \"string\"\n },\n \"stageId\": {\n \"type\": \"int\"\n },\n \"pipelineId\": {\n \"type\": \"int\"\n }\n },\n \"variables\": {\n \"WorkerPipelineState\": {\n \"type\": \"Boolean\"\n }\n },\n \"folder\": {\n \"name\": \"_ProcFwk\"\n },\n \"annotations\": [\n \"ADF.procfwk\",\n \"Infant\"\n ]\n },\n \"dependsOn\": [\n \"[concat(variables('factoryId'), '/datasets/GetSetMetadata')]\",\n \"[concat(variables('factoryId'), '/linkedServices/FrameworkFunctions')]\",\n \"[concat(variables('factoryId'), '/linkedServices/SupportDatabase')]\"\n ]\n },\n {\n \"name\": \"[concat(parameters('factoryName'), '/Intentional Error')]\",\n \"type\": \"Microsoft.DataFactory/factories/pipelines\",\n \"apiVersion\": \"2018-06-01\",\n \"properties\": {\n \"description\": \"Used just so the ADF.procfwk has something to call during development.\",\n \"activities\": [\n {\n \"name\": \"Wait1\",\n \"type\": \"Wait\",\n \"dependsOn\": [],\n \"userProperties\": [],\n \"typeProperties\": {\n \"waitTimeInSeconds\": {\n \"value\": \"@pipeline().parameters.WaitTime\",\n \"type\": \"Expression\"\n }\n }\n },\n {\n \"name\": \"Raise Errors or Not\",\n \"type\": \"IfCondition\",\n \"dependsOn\": [\n {\n \"activity\": \"Wait1\",\n \"dependencyConditions\": [\n \"Succeeded\"\n ]\n }\n ],\n \"userProperties\": [],\n \"typeProperties\": {\n \"expression\": {\n \"value\": \"@equals(pipeline().parameters.RaiseErrors,'true')\",\n \"type\": \"Expression\"\n },\n \"ifTrueActivities\": [\n {\n \"name\": \"Call Fail Procedure\",\n \"type\": \"SqlServerStoredProcedure\",\n \"dependsOn\": [],\n \"policy\": {\n \"timeout\": \"7.00:00:00\",\n \"retry\": 0,\n \"retryIntervalInSeconds\": 30,\n \"secureOutput\": false,\n \"secureInput\": false\n },\n \"userProperties\": [],\n \"typeProperties\": {\n \"storedProcedureName\": \"[[dbo].[FailProcedure]\",\n \"storedProcedureParameters\": {\n \"RaiseError\": {\n \"value\": {\n \"value\": \"@pipeline().parameters.RaiseErrors\",\n \"type\": \"Expression\"\n },\n \"type\": \"String\"\n }\n }\n },\n \"linkedServiceName\": {\n \"referenceName\": \"SupportDatabase\",\n \"type\": \"LinkedServiceReference\"\n }\n }\n ]\n }\n }\n ],\n \"parameters\": {\n \"RaiseErrors\": {\n \"type\": \"string\",\n \"defaultValue\": \"false\"\n },\n \"WaitTime\": {\n \"type\": \"int\",\n \"defaultValue\": 5\n }\n },\n \"folder\": {\n \"name\": \"_Workers\"\n },\n \"annotations\": [\n \"Worker\"\n ]\n },\n \"dependsOn\": [\n \"[concat(variables('factoryId'), '/linkedServices/SupportDatabase')]\"\n ]\n },\n {\n \"name\": \"[concat(parameters('factoryName'), '/Wait 1')]\",\n \"type\": \"Microsoft.DataFactory/factories/pipelines\",\n \"apiVersion\": \"2018-06-01\",\n \"properties\": {\n \"description\": \"Used just so the ADF.procfwk has something to call during development.\",\n \"activities\": [\n {\n \"name\": \"Wait1\",\n \"type\": \"Wait\",\n \"dependsOn\": [],\n \"userProperties\": [],\n \"typeProperties\": {\n \"waitTimeInSeconds\": {\n \"value\": \"@pipeline().parameters.WaitTime\",\n \"type\": \"Expression\"\n }\n }\n }\n ],\n \"parameters\": {\n \"WaitTime\": {\n \"type\": \"int\",\n \"defaultValue\": 5\n }\n },\n \"folder\": {\n \"name\": \"_Workers\"\n },\n \"annotations\": [\n \"Worker\"\n ]\n },\n \"dependsOn\": []\n },\n {\n \"name\": \"[concat(parameters('factoryName'), '/Wait 10')]\",\n \"type\": \"Microsoft.DataFactory/factories/pipelines\",\n \"apiVersion\": \"2018-06-01\",\n \"properties\": {\n \"description\": \"Used just so the ADF.procfwk has something to call during development.\",\n \"activities\": [\n {\n \"name\": \"Wait10\",\n \"type\": \"Wait\",\n \"dependsOn\": [],\n \"userProperties\": [],\n \"typeProperties\": {\n \"waitTimeInSeconds\": {\n \"value\": \"@pipeline().parameters.WaitTime\",\n \"type\": \"Expression\"\n }\n }\n }\n ],\n \"parameters\": {\n \"WaitTime\": {\n \"type\": \"int\",\n \"defaultValue\": 5\n }\n },\n \"folder\": {\n \"name\": \"_Workers\"\n },\n \"annotations\": [\n \"Worker\"\n ]\n },\n \"dependsOn\": []\n },\n {\n \"name\": \"[concat(parameters('factoryName'), '/Wait 2')]\",\n \"type\": \"Microsoft.DataFactory/factories/pipelines\",\n \"apiVersion\": \"2018-06-01\",\n \"properties\": {\n \"description\": \"Used just so the ADF.procfwk has something to call during development.\",\n \"activities\": [\n {\n \"name\": \"Wait2\",\n \"type\": \"Wait\",\n \"dependsOn\": [],\n \"userProperties\": [],\n \"typeProperties\": {\n \"waitTimeInSeconds\": {\n \"value\": \"@pipeline().parameters.WaitTime\",\n \"type\": \"Expression\"\n }\n }\n }\n ],\n \"parameters\": {\n \"WaitTime\": {\n \"type\": \"int\",\n \"defaultValue\": 5\n }\n },\n \"folder\": {\n \"name\": \"_Workers\"\n },\n \"annotations\": [\n \"Worker\"\n ]\n },\n \"dependsOn\": []\n },\n {\n \"name\": \"[concat(parameters('factoryName'), '/Wait 3')]\",\n \"type\": \"Microsoft.DataFactory/factories/pipelines\",\n \"apiVersion\": \"2018-06-01\",\n \"properties\": {\n \"description\": \"Used just so the ADF.procfwk has something to call during development.\",\n \"activities\": [\n {\n \"name\": \"Wait3\",\n \"type\": \"Wait\",\n \"dependsOn\": [],\n \"userProperties\": [],\n \"typeProperties\": {\n \"waitTimeInSeconds\": {\n \"value\": \"@pipeline().parameters.WaitTime\",\n \"type\": \"Expression\"\n }\n }\n }\n ],\n \"parameters\": {\n \"WaitTime\": {\n \"type\": \"int\",\n \"defaultValue\": 5\n }\n },\n \"folder\": {\n \"name\": \"_Workers\"\n },\n \"annotations\": [\n \"Worker\"\n ]\n },\n \"dependsOn\": []\n },\n {\n \"name\": \"[concat(parameters('factoryName'), '/Wait 4')]\",\n \"type\": \"Microsoft.DataFactory/factories/pipelines\",\n \"apiVersion\": \"2018-06-01\",\n \"properties\": {\n \"description\": \"Used just so the ADF.procfwk has something to call during development.\",\n \"activities\": [\n {\n \"name\": \"Wait4\",\n \"type\": \"Wait\",\n \"dependsOn\": [],\n \"userProperties\": [],\n \"typeProperties\": {\n \"waitTimeInSeconds\": {\n \"value\": \"@pipeline().parameters.WaitTime\",\n \"type\": \"Expression\"\n }\n }\n }\n ],\n \"parameters\": {\n \"WaitTime\": {\n \"type\": \"int\",\n \"defaultValue\": 5\n }\n },\n \"folder\": {\n \"name\": \"_Workers\"\n },\n \"annotations\": [\n \"Worker\"\n ]\n },\n \"dependsOn\": []\n },\n {\n \"name\": \"[concat(parameters('factoryName'), '/Wait 5')]\",\n \"type\": \"Microsoft.DataFactory/factories/pipelines\",\n \"apiVersion\": \"2018-06-01\",\n \"properties\": {\n \"description\": \"Used just so the ADF.procfwk has something to call during development.\",\n \"activities\": [\n {\n \"name\": \"Wait5\",\n \"type\": \"Wait\",\n \"dependsOn\": [],\n \"userProperties\": [],\n \"typeProperties\": {\n \"waitTimeInSeconds\": {\n \"value\": \"@pipeline().parameters.WaitTime\",\n \"type\": \"Expression\"\n }\n }\n }\n ],\n \"parameters\": {\n \"WaitTime\": {\n \"type\": \"int\",\n \"defaultValue\": 5\n }\n },\n \"folder\": {\n \"name\": \"_Workers\"\n },\n \"annotations\": [\n \"Worker\"\n ]\n },\n \"dependsOn\": []\n },\n {\n \"name\": \"[concat(parameters('factoryName'), '/Wait 6')]\",\n \"type\": \"Microsoft.DataFactory/factories/pipelines\",\n \"apiVersion\": \"2018-06-01\",\n \"properties\": {\n \"description\": \"Used just so the ADF.procfwk has something to call during development.\",\n \"activities\": [\n {\n \"name\": \"Wait6\",\n \"type\": \"Wait\",\n \"dependsOn\": [],\n \"userProperties\": [],\n \"typeProperties\": {\n \"waitTimeInSeconds\": {\n \"value\": \"@pipeline().parameters.WaitTime\",\n \"type\": \"Expression\"\n }\n }\n }\n ],\n \"parameters\": {\n \"WaitTime\": {\n \"type\": \"int\",\n \"defaultValue\": 5\n }\n },\n \"folder\": {\n \"name\": \"_Workers\"\n },\n \"annotations\": [\n \"Worker\"\n ]\n },\n \"dependsOn\": []\n },\n {\n \"name\": \"[concat(parameters('factoryName'), '/Wait 7')]\",\n \"type\": \"Microsoft.DataFactory/factories/pipelines\",\n \"apiVersion\": \"2018-06-01\",\n \"properties\": {\n \"description\": \"Used just so the ADF.procfwk has something to call during development.\",\n \"activities\": [\n {\n \"name\": \"Wait7\",\n \"type\": \"Wait\",\n \"dependsOn\": [],\n \"userProperties\": [],\n \"typeProperties\": {\n \"waitTimeInSeconds\": {\n \"value\": \"@pipeline().parameters.WaitTime\",\n \"type\": \"Expression\"\n }\n }\n }\n ],\n \"parameters\": {\n \"WaitTime\": {\n \"type\": \"int\",\n \"defaultValue\": 5\n }\n },\n \"folder\": {\n \"name\": \"_Workers\"\n },\n \"annotations\": [\n \"Worker\"\n ]\n },\n \"dependsOn\": []\n },\n {\n \"name\": \"[concat(parameters('factoryName'), '/Wait 8')]\",\n \"type\": \"Microsoft.DataFactory/factories/pipelines\",\n \"apiVersion\": \"2018-06-01\",\n \"properties\": {\n \"description\": \"Used just so the ADF.procfwk has something to call during development.\",\n \"activities\": [\n {\n \"name\": \"Wait8\",\n \"type\": \"Wait\",\n \"dependsOn\": [],\n \"userProperties\": [],\n \"typeProperties\": {\n \"waitTimeInSeconds\": {\n \"value\": \"@pipeline().parameters.WaitTime\",\n \"type\": \"Expression\"\n }\n }\n }\n ],\n \"parameters\": {\n \"WaitTime\": {\n \"type\": \"int\",\n \"defaultValue\": 5\n }\n },\n \"folder\": {\n \"name\": \"_Workers\"\n },\n \"annotations\": [\n \"Worker\"\n ]\n },\n \"dependsOn\": []\n },\n {\n \"name\": \"[concat(parameters('factoryName'), '/Wait 9')]\",\n \"type\": \"Microsoft.DataFactory/factories/pipelines\",\n \"apiVersion\": \"2018-06-01\",\n \"properties\": {\n \"description\": \"Used just so the ADF.procfwk has something to call during development.\",\n \"activities\": [\n {\n \"name\": \"Wait9\",\n \"type\": \"Wait\",\n \"dependsOn\": [],\n \"userProperties\": [],\n \"typeProperties\": {\n \"waitTimeInSeconds\": {\n \"value\": \"@pipeline().parameters.WaitTime\",\n \"type\": \"Expression\"\n }\n }\n }\n ],\n \"parameters\": {\n \"WaitTime\": {\n \"type\": \"int\",\n \"defaultValue\": 15\n }\n },\n \"folder\": {\n \"name\": \"_Workers\"\n },\n \"annotations\": [\n \"Worker\"\n ]\n },\n \"dependsOn\": []\n },\n {\n \"name\": \"[concat(parameters('factoryName'), '/GetSetMetadata')]\",\n \"type\": \"Microsoft.DataFactory/factories/datasets\",\n \"apiVersion\": \"2018-06-01\",\n \"properties\": {\n \"linkedServiceName\": {\n \"referenceName\": \"SupportDatabase\",\n \"type\": \"LinkedServiceReference\"\n },\n \"annotations\": [\n \"ADF.procfwk\"\n ],\n \"type\": \"AzureSqlTable\",\n \"schema\": [],\n \"typeProperties\": {}\n },\n \"dependsOn\": [\n \"[concat(variables('factoryId'), '/linkedServices/SupportDatabase')]\"\n ]\n },\n {\n \"name\": \"[concat(parameters('factoryName'), '/FrameworkFunctions')]\",\n \"type\": \"Microsoft.DataFactory/factories/linkedServices\",\n \"apiVersion\": \"2018-06-01\",\n \"properties\": {\n \"annotations\": [\n \"ADF.procfwk\"\n ],\n \"type\": \"AzureFunction\",\n \"typeProperties\": {\n \"functionAppUrl\": \"[parameters('FrameworkFunctions_properties_typeProperties_functionAppUrl')]\",\n \"functionKey\": {\n \"type\": \"AzureKeyVaultSecret\",\n \"store\": {\n \"referenceName\": \"Keys\",\n \"type\": \"LinkedServiceReference\"\n },\n \"secretName\": \"FrameworkFunctionsKey\"\n }\n }\n },\n \"dependsOn\": [\n \"[concat(variables('factoryId'), '/linkedServices/Keys')]\"\n ]\n },\n {\n \"name\": \"[concat(parameters('factoryName'), '/Keys')]\",\n \"type\": \"Microsoft.DataFactory/factories/linkedServices\",\n \"apiVersion\": \"2018-06-01\",\n \"properties\": {\n \"description\": \"Connection to Key Vault for all other ADF linked service credentials required to run the processing framework.\",\n \"annotations\": [\n \"ADF.procfwk\"\n ],\n \"type\": \"AzureKeyVault\",\n \"typeProperties\": {\n \"baseUrl\": \"[parameters('Keys_properties_typeProperties_baseUrl')]\"\n }\n },\n \"dependsOn\": []\n },\n {\n \"name\": \"[concat(parameters('factoryName'), '/SupportDatabase')]\",\n \"type\": \"Microsoft.DataFactory/factories/linkedServices\",\n \"apiVersion\": \"2018-06-01\",\n \"properties\": {\n \"description\": \"Connection between ADF and processing framework metadata SQLDB.\",\n \"annotations\": [\n \"ADF.procfwk\"\n ],\n \"type\": \"AzureSqlDatabase\",\n \"typeProperties\": {\n \"connectionString\": {\n \"type\": \"AzureKeyVaultSecret\",\n \"store\": {\n \"referenceName\": \"Keys\",\n \"type\": \"LinkedServiceReference\"\n },\n \"secretName\": \"[parameters('SupportDatabase_properties_typeProperties_connectionString_secretName')]\"\n }\n }\n },\n \"dependsOn\": [\n \"[concat(variables('factoryId'), '/linkedServices/Keys')]\"\n ]\n },\n {\n \"name\": \"[concat(parameters('factoryName'), '/FunctionalTestingTrigger')]\",\n \"type\": \"Microsoft.DataFactory/factories/triggers\",\n \"apiVersion\": \"2018-06-01\",\n \"properties\": {\n \"description\": \"Used for functional testing of the framework in a dedicated environment.\",\n \"annotations\": [\n \"ADF.procfwk\"\n ],\n \"runtimeState\": \"Stopped\",\n \"pipelines\": [\n {\n \"pipelineReference\": {\n \"referenceName\": \"01-Grandparent\",\n \"type\": \"PipelineReference\"\n },\n \"parameters\": {}\n }\n ],\n \"type\": \"ScheduleTrigger\",\n \"typeProperties\": {\n \"recurrence\": {\n \"frequency\": \"Hour\",\n \"interval\": 2,\n \"startTime\": \"2020-04-06T15:00:00.000Z\",\n \"timeZone\": \"UTC\"\n }\n }\n },\n \"dependsOn\": [\n \"[concat(variables('factoryId'), '/pipelines/01-Grandparent')]\"\n ]\n }\n ]\n}"
},
{
"path": "ARM Templates/Data Factory/v1.9 Export.json",
"content": "{\n \"$schema\": \"http://schema.management.azure.com/schemas/2015-01-01/deploymentTemplate.json#\",\n \"contentVersion\": \"1.0.0.0\",\n \"parameters\": {\n \"factoryName\": {\n \"type\": \"string\",\n \"metadata\": \"Data Factory name\",\n \"defaultValue\": \"\"\n },\n \"FrameworkFunctions_properties_typeProperties_functionAppUrl\": {\n \"type\": \"string\",\n \"defaultValue\": \"\"\n },\n \"Keys_properties_typeProperties_baseUrl\": {\n \"type\": \"string\",\n \"defaultValue\": \"\"\n },\n \"SupportDatabase_properties_typeProperties_connectionString_secretName\": {\n \"type\": \"string\",\n \"defaultValue\": \"\"\n }\n },\n \"variables\": {\n \"factoryId\": \"[concat('Microsoft.DataFactory/factories/', parameters('factoryName'))]\"\n },\n \"resources\": [\n {\n \"name\": \"[concat(parameters('factoryName'), '/01-Grandparent')]\",\n \"type\": \"Microsoft.DataFactory/factories/pipelines\",\n \"apiVersion\": \"2018-06-01\",\n \"properties\": {\n \"description\": \"ADF.procfwk grandparent pipeline used optionally to bootstrap any wider processes in your Data Factory that then calls the processing framework.\",\n \"activities\": [\n {\n \"name\": \"Framework Processing\",\n \"type\": \"ExecutePipeline\",\n \"dependsOn\": [],\n \"userProperties\": [],\n \"typeProperties\": {\n \"pipeline\": {\n \"referenceName\": \"02-Parent\",\n \"type\": \"PipelineReference\"\n },\n \"waitOnCompletion\": true,\n \"parameters\": {}\n }\n }\n ],\n \"folder\": {\n \"name\": \"_ProcFwk\"\n },\n \"annotations\": [\n \"ADF.procfwk\",\n \"Grandparent\"\n ]\n },\n \"dependsOn\": [\n \"[concat(variables('factoryId'), '/pipelines/02-Parent')]\"\n ]\n },\n {\n \"name\": \"[concat(parameters('factoryName'), '/02-Parent')]\",\n \"type\": \"Microsoft.DataFactory/factories/pipelines\",\n \"apiVersion\": \"2018-06-01\",\n \"properties\": {\n \"description\": \"ADF.procfwk parent pipeline used to bootstrap the orchestration framework in perform the first level ForEach calls in sequence for the metadata stages.\",\n \"activities\": [\n {\n \"name\": \"Get Stages\",\n \"description\": \"Returns a distinct list of execution stages within the framework metadata.\",\n \"type\": \"Lookup\",\n \"dependsOn\": [\n {\n \"activity\": \"Set Execution Id\",\n \"dependencyConditions\": [\n \"Succeeded\"\n ]\n }\n ],\n \"policy\": {\n \"timeout\": \"7.00:00:00\",\n \"retry\": 0,\n \"retryIntervalInSeconds\": 30,\n \"secureOutput\": false,\n \"secureInput\": false\n },\n \"userProperties\": [],\n \"typeProperties\": {\n \"source\": {\n \"type\": \"AzureSqlSource\",\n \"sqlReaderStoredProcedureName\": \"[[procfwk].[GetStages]\",\n \"storedProcedureParameters\": {\n \"ExecutionId\": {\n \"type\": \"Guid\",\n \"value\": {\n \"value\": \"@variables('ExecutionId')\",\n \"type\": \"Expression\"\n }\n }\n },\n \"queryTimeout\": \"02:00:00\",\n \"partitionOption\": \"None\"\n },\n \"dataset\": {\n \"referenceName\": \"GetSetMetadata\",\n \"type\": \"DatasetReference\",\n \"parameters\": {}\n },\n \"firstRowOnly\": false\n }\n },\n {\n \"name\": \"Execute Stages\",\n \"description\": \"Top level ForEach to sequentially call all processing stages within the framework metadata. Items for iteration passed from the Get Stages lookup activity.\",\n \"type\": \"ForEach\",\n \"dependsOn\": [\n {\n \"activity\": \"Get Stages\",\n \"dependencyConditions\": [\n \"Succeeded\"\n ]\n }\n ],\n \"userProperties\": [],\n \"typeProperties\": {\n \"items\": {\n \"value\": \"@activity('Get Stages').output.value\",\n \"type\": \"Expression\"\n },\n \"isSequential\": true,\n \"activities\": [\n {\n \"name\": \"Stage Executor\",\n \"description\": \"Call to the framework generic child pipeline for a given execution stage.\",\n \"type\": \"ExecutePipeline\",\n \"dependsOn\": [\n {\n \"activity\": \"Log Stage Preparing\",\n \"dependencyConditions\": [\n \"Succeeded\"\n ]\n }\n ],\n \"userProperties\": [],\n \"typeProperties\": {\n \"pipeline\": {\n \"referenceName\": \"03-Child\",\n \"type\": \"PipelineReference\"\n },\n \"waitOnCompletion\": true,\n \"parameters\": {\n \"StageId\": {\n \"value\": \"@item().StageId\",\n \"type\": \"Expression\"\n },\n \"ExecutionId\": {\n \"value\": \"@variables('ExecutionId')\",\n \"type\": \"Expression\"\n }\n }\n }\n },\n {\n \"name\": \"Log Stage Preparing\",\n \"description\": \"Update the current execution table flagging all pipelines within the stage as preparing.\",\n \"type\": \"SqlServerStoredProcedure\",\n \"dependsOn\": [\n {\n \"activity\": \"Check and Update Blockers\",\n \"dependencyConditions\": [\n \"Succeeded\"\n ]\n }\n ],\n \"policy\": {\n \"timeout\": \"7.00:00:00\",\n \"retry\": 0,\n \"retryIntervalInSeconds\": 30,\n \"secureOutput\": false,\n \"secureInput\": false\n },\n \"userProperties\": [],\n \"typeProperties\": {\n \"storedProcedureName\": \"[[procfwk].[SetLogStagePreparing]\",\n \"storedProcedureParameters\": {\n \"ExecutionId\": {\n \"value\": {\n \"value\": \"@variables('ExecutionId')\",\n \"type\": \"Expression\"\n },\n \"type\": \"Guid\"\n },\n \"StageId\": {\n \"value\": {\n \"value\": \"@item().StageId\",\n \"type\": \"Expression\"\n },\n \"type\": \"Int32\"\n }\n }\n },\n \"linkedServiceName\": {\n \"referenceName\": \"SupportDatabase\",\n \"type\": \"LinkedServiceReference\"\n }\n },\n {\n \"name\": \"Check and Update Blockers\",\n \"description\": \"Used to double check and stop the next execution stage if failures and blockers have be incurred. This also depends on the failure handling property value which defines the stored procedure behaviour.\",\n \"type\": \"SqlServerStoredProcedure\",\n \"dependsOn\": [],\n \"policy\": {\n \"timeout\": \"7.00:00:00\",\n \"retry\": 0,\n \"retryIntervalInSeconds\": 30,\n \"secureOutput\": false,\n \"secureInput\": false\n },\n \"userProperties\": [],\n \"typeProperties\": {\n \"storedProcedureName\": \"[[procfwk].[CheckForBlockedPipelines]\",\n \"storedProcedureParameters\": {\n \"ExecutionId\": {\n \"value\": {\n \"value\": \"@variables('ExecutionId')\",\n \"type\": \"Expression\"\n },\n \"type\": \"Guid\"\n },\n \"StageId\": {\n \"value\": {\n \"value\": \"@item().StageId\",\n \"type\": \"Expression\"\n },\n \"type\": \"Int32\"\n }\n }\n },\n \"linkedServiceName\": {\n \"referenceName\": \"SupportDatabase\",\n \"type\": \"LinkedServiceReference\"\n }\n }\n ]\n }\n },\n {\n \"name\": \"Execution Wrapper\",\n \"description\": \"Wrapper to reset and restart processing or create a completely new execution instance of the framework metadata.\",\n \"type\": \"Lookup\",\n \"dependsOn\": [\n {\n \"activity\": \"Clean Up Previous Run\",\n \"dependencyConditions\": [\n \"Succeeded\"\n ]\n }\n ],\n \"policy\": {\n \"timeout\": \"7.00:00:00\",\n \"retry\": 0,\n \"retryIntervalInSeconds\": 30,\n \"secureOutput\": false,\n \"secureInput\": false\n },\n \"userProperties\": [],\n \"typeProperties\": {\n \"source\": {\n \"type\": \"AzureSqlSource\",\n \"sqlReaderStoredProcedureName\": \"[[procfwk].[ExecutionWrapper]\",\n \"storedProcedureParameters\": {\n \"CallingDataFactory\": {\n \"type\": \"String\",\n \"value\": {\n \"value\": \"@pipeline().DataFactory\",\n \"type\": \"Expression\"\n }\n }\n },\n \"queryTimeout\": \"02:00:00\",\n \"partitionOption\": \"None\"\n },\n \"dataset\": {\n \"referenceName\": \"GetSetMetadata\",\n \"type\": \"DatasetReference\",\n \"parameters\": {}\n }\n }\n },\n {\n \"name\": \"Check Outcome and Update Logs\",\n \"description\": \"After a successful execution run the current execution metadata is moved to the long term logging table by this stored procedure call. Otherwise an error will be raised.\",\n \"type\": \"SqlServerStoredProcedure\",\n \"dependsOn\": [\n {\n \"activity\": \"Execute Stages\",\n \"dependencyConditions\": [\n \"Succeeded\"\n ]\n }\n ],\n \"policy\": {\n \"timeout\": \"7.00:00:00\",\n \"retry\": 0,\n \"retryIntervalInSeconds\": 30,\n \"secureOutput\": false,\n \"secureInput\": false\n },\n \"userProperties\": [],\n \"typeProperties\": {\n \"storedProcedureName\": \"[[procfwk].[UpdateExecutionLog]\",\n \"storedProcedureParameters\": {\n \"PerformErrorCheck\": {\n \"value\": {\n \"value\": \"@bool(1)\",\n \"type\": \"Expression\"\n },\n \"type\": \"Boolean\"\n }\n }\n },\n \"linkedServiceName\": {\n \"referenceName\": \"SupportDatabase\",\n \"type\": \"LinkedServiceReference\"\n }\n },\n {\n \"name\": \"Metadata Integrity Checks\",\n \"description\": \"Performs a series of checks on all metadata held in the framework SQLDB. This is intended to raise errors before an execution run even starts.\",\n \"type\": \"Lookup\",\n \"dependsOn\": [\n {\n \"activity\": \"Execute Precursor\",\n \"dependencyConditions\": [\n \"Succeeded\"\n ]\n }\n ],\n \"policy\": {\n \"timeout\": \"7.00:00:00\",\n \"retry\": 0,\n \"retryIntervalInSeconds\": 30,\n \"secureOutput\": false,\n \"secureInput\": false\n },\n \"userProperties\": [],\n \"typeProperties\": {\n \"source\": {\n \"type\": \"AzureSqlSource\",\n \"sqlReaderStoredProcedureName\": \"[[procfwk].[CheckMetadataIntegrity]\",\n \"storedProcedureParameters\": {\n \"DebugMode\": {\n \"type\": \"Boolean\",\n \"value\": \"false\"\n }\n },\n \"queryTimeout\": \"02:00:00\",\n \"partitionOption\": \"None\"\n },\n \"dataset\": {\n \"referenceName\": \"GetSetMetadata\",\n \"type\": \"DatasetReference\",\n \"parameters\": {}\n },\n \"firstRowOnly\": false\n }\n },\n {\n \"name\": \"Clean Up Previous Run\",\n \"description\": \"Handle Worker pipelines that are reported as Running when the parent pipeline is called again. Get what the actual status of those pipelines is.\",\n \"type\": \"ForEach\",\n \"dependsOn\": [\n {\n \"activity\": \"Metadata Integrity Checks\",\n \"dependencyConditions\": [\n \"Succeeded\"\n ]\n }\n ],\n \"userProperties\": [],\n \"typeProperties\": {\n \"items\": {\n \"value\": \"@activity('Metadata Integrity Checks').output.value\",\n \"type\": \"Expression\"\n },\n \"isSequential\": false,\n \"batchCount\": 20,\n \"activities\": [\n {\n \"name\": \"Get SPN Details\",\n \"type\": \"Lookup\",\n \"dependsOn\": [],\n \"policy\": {\n \"timeout\": \"7.00:00:00\",\n \"retry\": 0,\n \"retryIntervalInSeconds\": 30,\n \"secureOutput\": false,\n \"secureInput\": false\n },\n \"userProperties\": [],\n \"typeProperties\": {\n \"source\": {\n \"type\": \"AzureSqlSource\",\n \"sqlReaderStoredProcedureName\": \"[[procfwk].[GetWorkerAuthDetails]\",\n \"storedProcedureParameters\": {\n \"ExecutionId\": {\n \"type\": \"Guid\",\n \"value\": {\n \"value\": \"@item().LocalExecutionId\",\n \"type\": \"Expression\"\n }\n },\n \"PipelineId\": {\n \"type\": \"Int32\",\n \"value\": {\n \"value\": \"@item().PipelineId\",\n \"type\": \"Expression\"\n }\n },\n \"StageId\": {\n \"type\": \"Int32\",\n \"value\": {\n \"value\": \"@item().StageId\",\n \"type\": \"Expression\"\n }\n }\n },\n \"queryTimeout\": \"02:00:00\",\n \"partitionOption\": \"None\"\n },\n \"dataset\": {\n \"referenceName\": \"GetSetMetadata\",\n \"type\": \"DatasetReference\",\n \"parameters\": {}\n }\n }\n },\n {\n \"name\": \"Log Pipeline Checking\",\n \"type\": \"SqlServerStoredProcedure\",\n \"dependsOn\": [],\n \"policy\": {\n \"timeout\": \"7.00:00:00\",\n \"retry\": 0,\n \"retryIntervalInSeconds\": 30,\n \"secureOutput\": false,\n \"secureInput\": false\n },\n \"userProperties\": [],\n \"typeProperties\": {\n \"storedProcedureName\": \"[[procfwk].[SetLogPipelineChecking]\",\n \"storedProcedureParameters\": {\n \"ExecutionId\": {\n \"value\": {\n \"value\": \"@item().LocalExecutionId\",\n \"type\": \"Expression\"\n },\n \"type\": \"Guid\"\n },\n \"PipelineId\": {\n \"value\": {\n \"value\": \"@item().PipelineId\",\n \"type\": \"Expression\"\n },\n \"type\": \"Int32\"\n },\n \"StageId\": {\n \"value\": {\n \"value\": \"@item().StageId\",\n \"type\": \"Expression\"\n },\n \"type\": \"Int32\"\n }\n }\n },\n \"linkedServiceName\": {\n \"referenceName\": \"SupportDatabase\",\n \"type\": \"LinkedServiceReference\"\n }\n },\n {\n \"name\": \"Get Pipeline Status\",\n \"type\": \"AzureFunctionActivity\",\n \"dependsOn\": [\n {\n \"activity\": \"Get SPN Details\",\n \"dependencyConditions\": [\n \"Succeeded\"\n ]\n },\n {\n \"activity\": \"Log Pipeline Checking\",\n \"dependencyConditions\": [\n \"Succeeded\"\n ]\n }\n ],\n \"policy\": {\n \"timeout\": \"7.00:00:00\",\n \"retry\": 0,\n \"retryIntervalInSeconds\": 30,\n \"secureOutput\": false,\n \"secureInput\": false\n },\n \"userProperties\": [],\n \"typeProperties\": {\n \"functionName\": \"CheckPipelineStatus\",\n \"method\": \"POST\",\n \"headers\": {},\n \"body\": {\n \"value\": \"@concat('\\n{\\n \\\"tenantId\\\": \\\"',activity('Get SPN Details').output.firstRow.TenantId,'\\\",\\n \\\"applicationId\\\": \\\"',activity('Get SPN Details').output.firstRow.AppId,'\\\",\\n \\\"authenticationKey\\\": \\\"',activity('Get SPN Details').output.firstRow.AppSecret,'\\\",\\n \\\"subscriptionId\\\": \\\"',activity('Get SPN Details').output.firstRow.SubscriptionId,'\\\",\\n \\\"resourceGroup\\\": \\\"',item().ResourceGroupName,'\\\",\\n \\\"factoryName\\\": \\\"',item().DataFactoryName,'\\\",\\n \\\"pipelineName\\\": \\\"',item().PipelineName,'\\\",\\n \\\"runId\\\": \\\"',item().AdfPipelineRunId,'\\\"\\n}')\",\n \"type\": \"Expression\"\n }\n },\n \"linkedServiceName\": {\n \"referenceName\": \"FrameworkFunctions\",\n \"type\": \"LinkedServiceReference\"\n }\n },\n {\n \"name\": \"Set Pipeline Status\",\n \"description\": \"Update the metadata depending on the actual pipeline outcome. Using the status as the case.\",\n \"type\": \"Switch\",\n \"dependsOn\": [\n {\n \"activity\": \"Get Pipeline Status\",\n \"dependencyConditions\": [\n \"Succeeded\"\n ]\n }\n ],\n \"userProperties\": [],\n \"typeProperties\": {\n \"on\": {\n \"value\": \"@activity('Get Pipeline Status').output.Status\",\n \"type\": \"Expression\"\n },\n \"cases\": [\n {\n \"value\": \"Failed\",\n \"activities\": [\n {\n \"name\": \"Pipeline Status Failed\",\n \"description\": \"Updates the current execution table with a pipeline status of failed if the function outcome is failed. Also blocks pipelines in the downstream execution stage.\",\n \"type\": \"SqlServerStoredProcedure\",\n \"dependsOn\": [],\n \"policy\": {\n \"timeout\": \"7.00:00:00\",\n \"retry\": 0,\n \"retryIntervalInSeconds\": 30,\n \"secureOutput\": false,\n \"secureInput\": false\n },\n \"userProperties\": [],\n \"typeProperties\": {\n \"storedProcedureName\": \"[[procfwk].[SetLogPipelineFailed]\",\n \"storedProcedureParameters\": {\n \"ExecutionId\": {\n \"value\": {\n \"value\": \"@item().LocalExecutionId\",\n \"type\": \"Expression\"\n },\n \"type\": \"Guid\"\n },\n \"PipelineId\": {\n \"value\": {\n \"value\": \"@item().PipelineId\",\n \"type\": \"Expression\"\n },\n \"type\": \"Int32\"\n },\n \"RunId\": {\n \"value\": null,\n \"type\": \"Guid\"\n },\n \"StageId\": {\n \"value\": {\n \"value\": \"@item().StageId\",\n \"type\": \"Expression\"\n },\n \"type\": \"Int32\"\n }\n }\n },\n \"linkedServiceName\": {\n \"referenceName\": \"SupportDatabase\",\n \"type\": \"LinkedServiceReference\"\n }\n }\n ]\n },\n {\n \"value\": \"Succeeded\",\n \"activities\": [\n {\n \"name\": \"Pipeline Status Succeeded\",\n \"description\": \"Updates the current execution table with a pipeline status of success if the function outcome is succeeded.\",\n \"type\": \"SqlServerStoredProcedure\",\n \"dependsOn\": [],\n \"policy\": {\n \"timeout\": \"7.00:00:00\",\n \"retry\": 0,\n \"retryIntervalInSeconds\": 30,\n \"secureOutput\": false,\n \"secureInput\": false\n },\n \"userProperties\": [],\n \"typeProperties\": {\n \"storedProcedureName\": \"[[procfwk].[SetLogPipelineSuccess]\",\n \"storedProcedureParameters\": {\n \"ExecutionId\": {\n \"value\": {\n \"value\": \"@item().LocalExecutionId\",\n \"type\": \"Expression\"\n },\n \"type\": \"Guid\"\n },\n \"PipelineId\": {\n \"value\": {\n \"value\": \"@item().PipelineId\",\n \"type\": \"Expression\"\n },\n \"type\": \"Int32\"\n },\n \"StageId\": {\n \"value\": {\n \"value\": \"@item().StageId\",\n \"type\": \"Expression\"\n },\n \"type\": \"Int32\"\n }\n }\n },\n \"linkedServiceName\": {\n \"referenceName\": \"SupportDatabase\",\n \"type\": \"LinkedServiceReference\"\n }\n }\n ]\n },\n {\n \"value\": \"Queued\",\n \"activities\": [\n {\n \"name\": \"Pipeline Status Queued - Running\",\n \"description\": \"Updates the current execution table with a pipeline status of running if the function outcome is queued.\",\n \"type\": \"SqlServerStoredProcedure\",\n \"dependsOn\": [],\n \"policy\": {\n \"timeout\": \"7.00:00:00\",\n \"retry\": 0,\n \"retryIntervalInSeconds\": 30,\n \"secureOutput\": false,\n \"secureInput\": false\n },\n \"userProperties\": [],\n \"typeProperties\": {\n \"storedProcedureName\": \"[[procfwk].[SetLogPipelineRunning]\",\n \"storedProcedureParameters\": {\n \"ExecutionId\": {\n \"value\": {\n \"value\": \"@item().LocalExecutionId\",\n \"type\": \"Expression\"\n },\n \"type\": \"Guid\"\n },\n \"PipelineId\": {\n \"value\": {\n \"value\": \"@item().PipelineId\",\n \"type\": \"Expression\"\n },\n \"type\": \"Int32\"\n },\n \"StageId\": {\n \"value\": {\n \"value\": \"@item().StageId\",\n \"type\": \"Expression\"\n },\n \"type\": \"Int32\"\n }\n }\n },\n \"linkedServiceName\": {\n \"referenceName\": \"SupportDatabase\",\n \"type\": \"LinkedServiceReference\"\n }\n }\n ]\n },\n {\n \"value\": \"InProgress\",\n \"activities\": [\n {\n \"name\": \"Pipeline Status InProgress - Running\",\n \"description\": \"Updates the current execution table with a pipeline status of running if the function outcome is in progress.\",\n \"type\": \"SqlServerStoredProcedure\",\n \"dependsOn\": [],\n \"policy\": {\n \"timeout\": \"7.00:00:00\",\n \"retry\": 0,\n \"retryIntervalInSeconds\": 30,\n \"secureOutput\": false,\n \"secureInput\": false\n },\n \"userProperties\": [],\n \"typeProperties\": {\n \"storedProcedureName\": \"[[procfwk].[SetLogPipelineRunning]\",\n \"storedProcedureParameters\": {\n \"ExecutionId\": {\n \"value\": {\n \"value\": \"@item().LocalExecutionId\",\n \"type\": \"Expression\"\n },\n \"type\": \"Guid\"\n },\n \"PipelineId\": {\n \"value\": {\n \"value\": \"@item().PipelineId\",\n \"type\": \"Expression\"\n },\n \"type\": \"Int32\"\n },\n \"StageId\": {\n \"value\": {\n \"value\": \"@item().StageId\",\n \"type\": \"Expression\"\n },\n \"type\": \"Int32\"\n }\n }\n },\n \"linkedServiceName\": {\n \"referenceName\": \"SupportDatabase\",\n \"type\": \"LinkedServiceReference\"\n }\n }\n ]\n },\n {\n \"value\": \"Cancelled\",\n \"activities\": [\n {\n \"name\": \"Pipeline Status Cancelled\",\n \"description\": \"Updates the current execution table with a pipeline status of cancelled if the function outcome is cancelled.\",\n \"type\": \"SqlServerStoredProcedure\",\n \"dependsOn\": [],\n \"policy\": {\n \"timeout\": \"7.00:00:00\",\n \"retry\": 0,\n \"retryIntervalInSeconds\": 30,\n \"secureOutput\": false,\n \"secureInput\": false\n },\n \"userProperties\": [],\n \"typeProperties\": {\n \"storedProcedureName\": \"[[procfwk].[SetLogPipelineCancelled]\",\n \"storedProcedureParameters\": {\n \"ExecutionId\": {\n \"value\": {\n \"value\": \"@item().LocalExecutionId\",\n \"type\": \"Expression\"\n },\n \"type\": \"Guid\"\n },\n \"PipelineId\": {\n \"value\": {\n \"value\": \"@item().PipelineId\",\n \"type\": \"Expression\"\n },\n \"type\": \"Int32\"\n },\n \"StageId\": {\n \"value\": {\n \"value\": \"@item().StageId\",\n \"type\": \"Expression\"\n },\n \"type\": \"Int32\"\n },\n \"CleanUpRun\": {\n \"value\": {\n \"value\": \"@bool(1)\",\n \"type\": \"Expression\"\n },\n \"type\": \"Boolean\"\n }\n }\n },\n \"linkedServiceName\": {\n \"referenceName\": \"SupportDatabase\",\n \"type\": \"LinkedServiceReference\"\n }\n }\n ]\n }\n ],\n \"defaultActivities\": [\n {\n \"name\": \"Pipeline Status Unknown\",\n \"description\": \"Updates the current execution table with a pipeline status of unknown if the function returns an unexpected outcome.\",\n \"type\": \"SqlServerStoredProcedure\",\n \"dependsOn\": [],\n \"policy\": {\n \"timeout\": \"7.00:00:00\",\n \"retry\": 0,\n \"retryIntervalInSeconds\": 30,\n \"secureOutput\": false,\n \"secureInput\": false\n },\n \"userProperties\": [],\n \"typeProperties\": {\n \"storedProcedureName\": \"[[procfwk].[SetLogPipelineUnknown]\",\n \"storedProcedureParameters\": {\n \"ExecutionId\": {\n \"value\": {\n \"value\": \"@item().LocalExecutionId\",\n \"type\": \"Expression\"\n },\n \"type\": \"Guid\"\n },\n \"PipelineId\": {\n \"value\": {\n \"value\": \"@item().PipelineId\",\n \"type\": \"Expression\"\n },\n \"type\": \"Int32\"\n },\n \"StageId\": {\n \"value\": {\n \"value\": \"@item().StageId\",\n \"type\": \"Expression\"\n },\n \"type\": \"Int32\"\n },\n \"CleanUpRun\": {\n \"value\": {\n \"value\": \"@bool(1)\",\n \"type\": \"Expression\"\n },\n \"type\": \"Boolean\"\n }\n }\n },\n \"linkedServiceName\": {\n \"referenceName\": \"SupportDatabase\",\n \"type\": \"LinkedServiceReference\"\n }\n }\n ]\n }\n },\n {\n \"name\": \"Set Last Check DateTime\",\n \"description\": \"Update the current execution table with a date time from when the function last checked the pipeline status.\",\n \"type\": \"SqlServerStoredProcedure\",\n \"dependsOn\": [\n {\n \"activity\": \"Get Pipeline Status\",\n \"dependencyConditions\": [\n \"Succeeded\"\n ]\n }\n ],\n \"policy\": {\n \"timeout\": \"7.00:00:00\",\n \"retry\": 0,\n \"retryIntervalInSeconds\": 30,\n \"secureOutput\": false,\n \"secureInput\": false\n },\n \"userProperties\": [],\n \"typeProperties\": {\n \"storedProcedureName\": \"[[procfwk].[SetLogPipelineLastStatusCheck]\",\n \"storedProcedureParameters\": {\n \"ExecutionId\": {\n \"value\": {\n \"value\": \"@item().LocalExecutionId\",\n \"type\": \"Expression\"\n },\n \"type\": \"Guid\"\n },\n \"PipelineId\": {\n \"value\": {\n \"value\": \"@item().PipelineId\",\n \"type\": \"Expression\"\n },\n \"type\": \"Int32\"\n },\n \"StageId\": {\n \"value\": {\n \"value\": \"@item().StageId\",\n \"type\": \"Expression\"\n },\n \"type\": \"Int32\"\n }\n }\n },\n \"linkedServiceName\": {\n \"referenceName\": \"SupportDatabase\",\n \"type\": \"LinkedServiceReference\"\n }\n }\n ]\n }\n },\n {\n \"name\": \"Execute Precursor\",\n \"type\": \"SqlServerStoredProcedure\",\n \"dependsOn\": [],\n \"policy\": {\n \"timeout\": \"7.00:00:00\",\n \"retry\": 0,\n \"retryIntervalInSeconds\": 30,\n \"secureOutput\": false,\n \"secureInput\": false\n },\n \"userProperties\": [],\n \"typeProperties\": {\n \"storedProcedureName\": \"[[procfwk].[ExecutePrecursorProcedure]\"\n },\n \"linkedServiceName\": {\n \"referenceName\": \"SupportDatabase\",\n \"type\": \"LinkedServiceReference\"\n }\n },\n {\n \"name\": \"Set Execution Id\",\n \"description\": \"Set the local execution Id to a pipeline variable for each in several downstream activities.\",\n \"type\": \"SetVariable\",\n \"dependsOn\": [\n {\n \"activity\": \"Execution Wrapper\",\n \"dependencyConditions\": [\n \"Succeeded\"\n ]\n }\n ],\n \"userProperties\": [],\n \"typeProperties\": {\n \"variableName\": \"ExecutionId\",\n \"value\": {\n \"value\": \"@activity('Execution Wrapper').output.firstRow.ExecutionId\",\n \"type\": \"Expression\"\n }\n }\n }\n ],\n \"variables\": {\n \"ExecutionId\": {\n \"type\": \"String\"\n }\n },\n \"folder\": {\n \"name\": \"_ProcFwk\"\n },\n \"annotations\": [\n \"ADF.procfwk\",\n \"Parent\"\n ]\n },\n \"dependsOn\": [\n \"[concat(variables('factoryId'), '/datasets/GetSetMetadata')]\",\n \"[concat(variables('factoryId'), '/linkedServices/SupportDatabase')]\",\n \"[concat(variables('factoryId'), '/pipelines/03-Child')]\",\n \"[concat(variables('factoryId'), '/linkedServices/FrameworkFunctions')]\"\n ]\n },\n {\n \"name\": \"[concat(parameters('factoryName'), '/03-Child')]\",\n \"type\": \"Microsoft.DataFactory/factories/pipelines\",\n \"apiVersion\": \"2018-06-01\",\n \"properties\": {\n \"description\": \"ADF.procfwk child pipeline used to execute Worker pipelines within a given execution stage. This pipeline will be called once for each stage, then execute all Workers in parallel.\",\n \"activities\": [\n {\n \"name\": \"Get Pipelines\",\n \"description\": \"Returns all pipelines from the metadata to be executed within a given processing stage.\",\n \"type\": \"Lookup\",\n \"dependsOn\": [],\n \"policy\": {\n \"timeout\": \"7.00:00:00\",\n \"retry\": 0,\n \"retryIntervalInSeconds\": 30,\n \"secureOutput\": false,\n \"secureInput\": false\n },\n \"userProperties\": [],\n \"typeProperties\": {\n \"source\": {\n \"type\": \"AzureSqlSource\",\n \"sqlReaderStoredProcedureName\": \"[[procfwk].[GetPipelinesInStage]\",\n \"storedProcedureParameters\": {\n \"ExecutionId\": {\n \"type\": \"Guid\",\n \"value\": {\n \"value\": \"@pipeline().parameters.ExecutionId\",\n \"type\": \"Expression\"\n }\n },\n \"StageId\": {\n \"type\": \"Int32\",\n \"value\": {\n \"value\": \"@pipeline().parameters.StageId\",\n \"type\": \"Expression\"\n }\n }\n },\n \"queryTimeout\": \"02:00:00\",\n \"partitionOption\": \"None\"\n },\n \"dataset\": {\n \"referenceName\": \"GetSetMetadata\",\n \"type\": \"DatasetReference\",\n \"parameters\": {}\n },\n \"firstRowOnly\": false\n }\n },\n {\n \"name\": \"Execute Pipelines\",\n \"description\": \"Second level ForEach to run in parallel all pipelines within the stage. Items for iteration passed from the Get Pipelines lookup activity.\",\n \"type\": \"ForEach\",\n \"dependsOn\": [\n {\n \"activity\": \"Get Pipelines\",\n \"dependencyConditions\": [\n \"Succeeded\"\n ]\n }\n ],\n \"userProperties\": [],\n \"typeProperties\": {\n \"items\": {\n \"value\": \"@activity('Get Pipelines').output.value\",\n \"type\": \"Expression\"\n },\n \"isSequential\": false,\n \"batchCount\": 50,\n \"activities\": [\n {\n \"name\": \"Worker Pipeline Executor\",\n \"description\": \"Run the required worker pipeline and wait for its completion. Update metadata once done.\",\n \"type\": \"ExecutePipeline\",\n \"dependsOn\": [],\n \"userProperties\": [],\n \"typeProperties\": {\n \"pipeline\": {\n \"referenceName\": \"04-Infant\",\n \"type\": \"PipelineReference\"\n },\n \"waitOnCompletion\": true,\n \"parameters\": {\n \"executionId\": {\n \"value\": \"@pipeline().parameters.ExecutionId\",\n \"type\": \"Expression\"\n },\n \"stageId\": {\n \"value\": \"@pipeline().parameters.StageId\",\n \"type\": \"Expression\"\n },\n \"pipelineId\": {\n \"value\": \"@item().PipelineId\",\n \"type\": \"Expression\"\n }\n }\n }\n }\n ]\n }\n }\n ],\n \"parameters\": {\n \"StageId\": {\n \"type\": \"int\"\n },\n \"ExecutionId\": {\n \"type\": \"string\"\n }\n },\n \"folder\": {\n \"name\": \"_ProcFwk\"\n },\n \"annotations\": [\n \"ADF.procfwk\",\n \"Child\"\n ]\n },\n \"dependsOn\": [\n \"[concat(variables('factoryId'), '/datasets/GetSetMetadata')]\",\n \"[concat(variables('factoryId'), '/pipelines/04-Infant')]\"\n ]\n },\n {\n \"name\": \"[concat(parameters('factoryName'), '/04-Infant')]\",\n \"type\": \"Microsoft.DataFactory/factories/pipelines\",\n \"apiVersion\": \"2018-06-01\",\n \"properties\": {\n \"description\": \"ADF.procfwk infant pipeline used to check when the processing pipeline called by the Child completes and passes the resulting status back to the metadata database.\",\n \"activities\": [\n {\n \"name\": \"Execute Worker Pipeline\",\n \"description\": \"The lowest level executor with the metadata framework to call existing processing pipelines within Data Factory. The function called will block processing and wait for an outcome.\",\n \"type\": \"AzureFunctionActivity\",\n \"dependsOn\": [\n {\n \"activity\": \"Log Pipeline Running\",\n \"dependencyConditions\": [\n \"Succeeded\"\n ]\n },\n {\n \"activity\": \"Get Pipeline Params\",\n \"dependencyConditions\": [\n \"Succeeded\"\n ]\n },\n {\n \"activity\": \"Set App Id\",\n \"dependencyConditions\": [\n \"Succeeded\"\n ]\n },\n {\n \"activity\": \"Set App Secret\",\n \"dependencyConditions\": [\n \"Succeeded\"\n ]\n },\n {\n \"activity\": \"Set Subscription Id\",\n \"dependencyConditions\": [\n \"Succeeded\"\n ]\n },\n {\n \"activity\": \"Set Tenant Id\",\n \"dependencyConditions\": [\n \"Succeeded\"\n ]\n },\n {\n \"activity\": \"Set Pipeline Name\",\n \"dependencyConditions\": [\n \"Succeeded\"\n ]\n },\n {\n \"activity\": \"Set Data Factory Name\",\n \"dependencyConditions\": [\n \"Succeeded\"\n ]\n },\n {\n \"activity\": \"Set Resource Group\",\n \"dependencyConditions\": [\n \"Succeeded\"\n ]\n }\n ],\n \"policy\": {\n \"timeout\": \"7.00:00:00\",\n \"retry\": 0,\n \"retryIntervalInSeconds\": 30,\n \"secureOutput\": false,\n \"secureInput\": false\n },\n \"userProperties\": [],\n \"typeProperties\": {\n \"functionName\": \"ExecutePipeline\",\n \"method\": \"POST\",\n \"headers\": {},\n \"body\": {\n \"value\": \"@concat('\\n{\\n\\t\\\"tenantId\\\": \\\"',variables('WorkerTenantId'),'\\\",\\n\\t\\\"applicationId\\\": \\\"',variables('WorkerAppId'),'\\\",\\n\\t\\\"authenticationKey\\\": \\\"',variables('WorkerAppSecret'),'\\\",\\n\\t\\\"subscriptionId\\\": \\\"',variables('WorkerSubscriptionId'),'\\\",\\n\\t\\\"resourceGroup\\\": \\\"',variables('WorkerResourceGroup'),'\\\",\\n\\t\\\"factoryName\\\": \\\"',variables('WorkerDataFactoryName'),'\\\",\\n\\t\\\"pipelineName\\\": \\\"',variables('WorkerPipelineName'),'\\\"',activity('Get Pipeline Params').output.firstRow.Params,'\\n}')\",\n \"type\": \"Expression\"\n }\n },\n \"linkedServiceName\": {\n \"referenceName\": \"FrameworkFunctions\",\n \"type\": \"LinkedServiceReference\"\n }\n },\n {\n \"name\": \"Get Pipeline Params\",\n \"description\": \"Returns any parameters from metadata required for the processing pipeline being called. The output can be an empty string if no parameters are required.\",\n \"type\": \"Lookup\",\n \"dependsOn\": [],\n \"policy\": {\n \"timeout\": \"7.00:00:00\",\n \"retry\": 0,\n \"retryIntervalInSeconds\": 30,\n \"secureOutput\": false,\n \"secureInput\": false\n },\n \"userProperties\": [],\n \"typeProperties\": {\n \"source\": {\n \"type\": \"AzureSqlSource\",\n \"sqlReaderStoredProcedureName\": \"[[procfwk].[GetPipelineParameters]\",\n \"storedProcedureParameters\": {\n \"PipelineId\": {\n \"type\": \"Int32\",\n \"value\": {\n \"value\": \"@pipeline().parameters.pipelineId\",\n \"type\": \"Expression\"\n }\n }\n },\n \"queryTimeout\": \"02:00:00\",\n \"partitionOption\": \"None\"\n },\n \"dataset\": {\n \"referenceName\": \"GetSetMetadata\",\n \"type\": \"DatasetReference\",\n \"parameters\": {}\n }\n }\n },\n {\n \"name\": \"Log Pipeline Running\",\n \"description\": \"Sets the current pipeline with a status of running within the current execution database table.\",\n \"type\": \"SqlServerStoredProcedure\",\n \"dependsOn\": [],\n \"policy\": {\n \"timeout\": \"7.00:00:00\",\n \"retry\": 0,\n \"retryIntervalInSeconds\": 30,\n \"secureOutput\": false,\n \"secureInput\": false\n },\n \"userProperties\": [],\n \"typeProperties\": {\n \"storedProcedureName\": \"[[procfwk].[SetLogPipelineRunning]\",\n \"storedProcedureParameters\": {\n \"ExecutionId\": {\n \"value\": {\n \"value\": \"@pipeline().parameters.ExecutionId\",\n \"type\": \"Expression\"\n },\n \"type\": \"Guid\"\n },\n \"PipelineId\": {\n \"value\": {\n \"value\": \"@pipeline().parameters.pipelineId\",\n \"type\": \"Expression\"\n },\n \"type\": \"Int32\"\n },\n \"StageId\": {\n \"value\": {\n \"value\": \"@pipeline().parameters.StageId\",\n \"type\": \"Expression\"\n },\n \"type\": \"Int32\"\n }\n }\n },\n \"linkedServiceName\": {\n \"referenceName\": \"SupportDatabase\",\n \"type\": \"LinkedServiceReference\"\n }\n },\n {\n \"name\": \"Get Worker Authentication Details\",\n \"description\": \"Return the SPN ID and Secret for the worker pipeline being executed. Called at this level as each pipeline can have a different SPN.\",\n \"type\": \"Lookup\",\n \"dependsOn\": [],\n \"policy\": {\n \"timeout\": \"7.00:00:00\",\n \"retry\": 0,\n \"retryIntervalInSeconds\": 30,\n \"secureOutput\": false,\n \"secureInput\": false\n },\n \"userProperties\": [],\n \"typeProperties\": {\n \"source\": {\n \"type\": \"AzureSqlSource\",\n \"sqlReaderStoredProcedureName\": \"[[procfwk].[GetWorkerAuthDetails]\",\n \"storedProcedureParameters\": {\n \"ExecutionId\": {\n \"type\": \"Guid\",\n \"value\": {\n \"value\": \"@pipeline().parameters.executionId\",\n \"type\": \"Expression\"\n }\n },\n \"PipelineId\": {\n \"type\": \"Int32\",\n \"value\": {\n \"value\": \"@pipeline().parameters.pipelineId\",\n \"type\": \"Expression\"\n }\n },\n \"StageId\": {\n \"type\": \"Int32\",\n \"value\": {\n \"value\": \"@pipeline().parameters.stageId\",\n \"type\": \"Expression\"\n }\n }\n },\n \"queryTimeout\": \"02:00:00\",\n \"partitionOption\": \"None\"\n },\n \"dataset\": {\n \"referenceName\": \"GetSetMetadata\",\n \"type\": \"DatasetReference\",\n \"parameters\": {}\n }\n }\n },\n {\n \"name\": \"Log Execute Function Activity Failure\",\n \"description\": \"Handle true failures from calling out to the Azure Function and update the current execution table accordingly so a restart can occur.\",\n \"type\": \"SqlServerStoredProcedure\",\n \"dependsOn\": [\n {\n \"activity\": \"Execute Worker Pipeline\",\n \"dependencyConditions\": [\n \"Failed\"\n ]\n }\n ],\n \"policy\": {\n \"timeout\": \"7.00:00:00\",\n \"retry\": 0,\n \"retryIntervalInSeconds\": 30,\n \"secureOutput\": false,\n \"secureInput\": false\n },\n \"userProperties\": [],\n \"typeProperties\": {\n \"storedProcedureName\": \"[[procfwk].[SetLogActivityFailed]\",\n \"storedProcedureParameters\": {\n \"ExecutionId\": {\n \"value\": {\n \"value\": \"@pipeline().parameters.ExecutionId\",\n \"type\": \"Expression\"\n },\n \"type\": \"Guid\"\n },\n \"PipelineId\": {\n \"value\": {\n \"value\": \"@pipeline().parameters.pipelineId\",\n \"type\": \"Expression\"\n },\n \"type\": \"Int32\"\n },\n \"StageId\": {\n \"value\": {\n \"value\": \"@pipeline().parameters.StageId\",\n \"type\": \"Expression\"\n },\n \"type\": \"Int32\"\n },\n \"CallingActivity\": {\n \"value\": \"ExecuteWorkerPipeline\",\n \"type\": \"String\"\n }\n }\n },\n \"linkedServiceName\": {\n \"referenceName\": \"SupportDatabase\",\n \"type\": \"LinkedServiceReference\"\n }\n },\n {\n \"name\": \"Update Run Id\",\n \"description\": \"Provide the actual ADF run ID back to the current execution table for long term logging and alignment between the metadata other Azure monitoring tools.\",\n \"type\": \"SqlServerStoredProcedure\",\n \"dependsOn\": [\n {\n \"activity\": \"Set Run Id\",\n \"dependencyConditions\": [\n \"Succeeded\"\n ]\n }\n ],\n \"policy\": {\n \"timeout\": \"7.00:00:00\",\n \"retry\": 0,\n \"retryIntervalInSeconds\": 30,\n \"secureOutput\": false,\n \"secureInput\": false\n },\n \"userProperties\": [],\n \"typeProperties\": {\n \"storedProcedureName\": \"[[procfwk].[SetLogPipelineRunId]\",\n \"storedProcedureParameters\": {\n \"ExecutionId\": {\n \"value\": {\n \"value\": \"@pipeline().parameters.ExecutionId\",\n \"type\": \"Expression\"\n },\n \"type\": \"Guid\"\n },\n \"PipelineId\": {\n \"value\": {\n \"value\": \"@pipeline().parameters.pipelineId\",\n \"type\": \"Expression\"\n },\n \"type\": \"Int32\"\n },\n \"RunId\": {\n \"value\": {\n \"value\": \"@variables('WorkerRunId')\",\n \"type\": \"Expression\"\n },\n \"type\": \"Guid\"\n },\n \"StageId\": {\n \"value\": {\n \"value\": \"@pipeline().parameters.StageId\",\n \"type\": \"Expression\"\n },\n \"type\": \"Int32\"\n }\n }\n },\n \"linkedServiceName\": {\n \"referenceName\": \"SupportDatabase\",\n \"type\": \"LinkedServiceReference\"\n }\n },\n {\n \"name\": \"Check For Alerts\",\n \"description\": \"Checks the properties tables and if any recipients in the database require alerts sending for the current pipeline ID.\",\n \"type\": \"Lookup\",\n \"dependsOn\": [\n {\n \"activity\": \"Update Run Id\",\n \"dependencyConditions\": [\n \"Succeeded\"\n ]\n },\n {\n \"activity\": \"Set Pipeline Result\",\n \"dependencyConditions\": [\n \"Completed\"\n ]\n }\n ],\n \"policy\": {\n \"timeout\": \"7.00:00:00\",\n \"retry\": 0,\n \"retryIntervalInSeconds\": 30,\n \"secureOutput\": false,\n \"secureInput\": false\n },\n \"userProperties\": [],\n \"typeProperties\": {\n \"source\": {\n \"type\": \"AzureSqlSource\",\n \"sqlReaderStoredProcedureName\": \"[[procfwk].[CheckForEmailAlerts]\",\n \"storedProcedureParameters\": {\n \"PipelineId\": {\n \"type\": \"Int32\",\n \"value\": {\n \"value\": \"@pipeline().parameters.pipelineId\",\n \"type\": \"Expression\"\n }\n }\n },\n \"queryTimeout\": \"02:00:00\",\n \"partitionOption\": \"None\"\n },\n \"dataset\": {\n \"referenceName\": \"GetSetMetadata\",\n \"type\": \"DatasetReference\",\n \"parameters\": {}\n },\n \"firstRowOnly\": true\n }\n },\n {\n \"name\": \"Send Alerts\",\n \"description\": \"True = alerts need sending.\\nFalse = do nothing.\",\n \"type\": \"IfCondition\",\n \"dependsOn\": [\n {\n \"activity\": \"Check For Alerts\",\n \"dependencyConditions\": [\n \"Succeeded\"\n ]\n }\n ],\n \"userProperties\": [],\n \"typeProperties\": {\n \"expression\": {\n \"value\": \"@activity('Check For Alerts').output.firstRow.SendAlerts\",\n \"type\": \"Expression\"\n },\n \"ifTrueActivities\": [\n {\n \"name\": \"Get Email Parts\",\n \"type\": \"Lookup\",\n \"dependsOn\": [],\n \"policy\": {\n \"timeout\": \"7.00:00:00\",\n \"retry\": 0,\n \"retryIntervalInSeconds\": 30,\n \"secureOutput\": false,\n \"secureInput\": false\n },\n \"userProperties\": [],\n \"typeProperties\": {\n \"source\": {\n \"type\": \"AzureSqlSource\",\n \"sqlReaderStoredProcedureName\": \"[[procfwk].[GetEmailAlertParts]\",\n \"storedProcedureParameters\": {\n \"PipelineId\": {\n \"type\": \"Int32\",\n \"value\": {\n \"value\": \"@pipeline().parameters.pipelineId\",\n \"type\": \"Expression\"\n }\n }\n },\n \"queryTimeout\": \"02:00:00\",\n \"partitionOption\": \"None\"\n },\n \"dataset\": {\n \"referenceName\": \"GetSetMetadata\",\n \"type\": \"DatasetReference\",\n \"parameters\": {}\n },\n \"firstRowOnly\": true\n }\n },\n {\n \"name\": \"Send Email\",\n \"type\": \"AzureFunctionActivity\",\n \"dependsOn\": [\n {\n \"activity\": \"Get Email Parts\",\n \"dependencyConditions\": [\n \"Succeeded\"\n ]\n }\n ],\n \"policy\": {\n \"timeout\": \"7.00:00:00\",\n \"retry\": 0,\n \"retryIntervalInSeconds\": 30,\n \"secureOutput\": false,\n \"secureInput\": false\n },\n \"userProperties\": [],\n \"typeProperties\": {\n \"functionName\": \"SendEmail\",\n \"method\": \"POST\",\n \"headers\": {},\n \"body\": {\n \"value\": \"@activity('Get Email Parts').output.firstRow\",\n \"type\": \"Expression\"\n }\n },\n \"linkedServiceName\": {\n \"referenceName\": \"FrameworkFunctions\",\n \"type\": \"LinkedServiceReference\"\n }\n }\n ]\n }\n },\n {\n \"name\": \"Wait Until Pipeline Completes\",\n \"description\": \"Loops until the Worker pipeline called completes.\\n\\nSimple status:\\n- Running = new iteration.\\n- Done = break.\",\n \"type\": \"Until\",\n \"dependsOn\": [\n {\n \"activity\": \"Get Wait Duration\",\n \"dependencyConditions\": [\n \"Succeeded\"\n ]\n },\n {\n \"activity\": \"Execute Worker Pipeline\",\n \"dependencyConditions\": [\n \"Succeeded\"\n ]\n },\n {\n \"activity\": \"Set Run Id\",\n \"dependencyConditions\": [\n \"Succeeded\"\n ]\n }\n ],\n \"userProperties\": [],\n \"typeProperties\": {\n \"expression\": {\n \"value\": \"@variables('WorkerPipelineState')\",\n \"type\": \"Expression\"\n },\n \"activities\": [\n {\n \"name\": \"Get Worker Pipeline Status\",\n \"description\": \"Checks the status of a given processing pipeline and provides the value for the downstream framework activities to act upon.\",\n \"type\": \"AzureFunctionActivity\",\n \"dependsOn\": [],\n \"policy\": {\n \"timeout\": \"7.00:00:00\",\n \"retry\": 0,\n \"retryIntervalInSeconds\": 30,\n \"secureOutput\": false,\n \"secureInput\": false\n },\n \"userProperties\": [],\n \"typeProperties\": {\n \"functionName\": \"CheckPipelineStatus\",\n \"method\": \"POST\",\n \"headers\": {},\n \"body\": {\n \"value\": \"@concat('\\n{\\n \\\"tenantId\\\": \\\"',variables('WorkerTenantId'),'\\\",\\n \\\"applicationId\\\": \\\"',variables('WorkerAppId'),'\\\",\\n \\\"authenticationKey\\\": \\\"',variables('WorkerAppSecret'),'\\\",\\n \\\"subscriptionId\\\": \\\"',variables('WorkerSubscriptionId'),'\\\",\\n \\\"resourceGroup\\\": \\\"',variables('WorkerResourceGroup'),'\\\",\\n \\\"factoryName\\\": \\\"',variables('WorkerDataFactoryName'),'\\\",\\n \\\"pipelineName\\\": \\\"',variables('WorkerPipelineName'),'\\\",\\n \\\"runId\\\": \\\"',variables('WorkerRunId'),'\\\"\\n}')\",\n \"type\": \"Expression\"\n }\n },\n \"linkedServiceName\": {\n \"referenceName\": \"FrameworkFunctions\",\n \"type\": \"LinkedServiceReference\"\n }\n },\n {\n \"name\": \"Wait If Running\",\n \"description\": \"True = Do nothing.\\nFalse = Wait, before the next iteration.\",\n \"type\": \"IfCondition\",\n \"dependsOn\": [\n {\n \"activity\": \"Set Worker State\",\n \"dependencyConditions\": [\n \"Succeeded\"\n ]\n }\n ],\n \"userProperties\": [],\n \"typeProperties\": {\n \"expression\": {\n \"value\": \"@variables('WorkerPipelineState')\",\n \"type\": \"Expression\"\n },\n \"ifFalseActivities\": [\n {\n \"name\": \"Wait for Pipeline\",\n \"description\": \"The processing pipeline is still running so Wait before checking its status again.\",\n \"type\": \"Wait\",\n \"dependsOn\": [],\n \"userProperties\": [],\n \"typeProperties\": {\n \"waitTimeInSeconds\": {\n \"value\": \"@activity('Get Wait Duration').output.firstRow.PropertyValue\",\n \"type\": \"Expression\"\n }\n }\n }\n ]\n }\n },\n {\n \"name\": \"Set Last Check DateTime\",\n \"description\": \"Update the current execution table with a date time from when the Worker pipeline status was last checked as part of the Until iterations.\",\n \"type\": \"SqlServerStoredProcedure\",\n \"dependsOn\": [\n {\n \"activity\": \"Get Worker Pipeline Status\",\n \"dependencyConditions\": [\n \"Succeeded\"\n ]\n }\n ],\n \"policy\": {\n \"timeout\": \"7.00:00:00\",\n \"retry\": 0,\n \"retryIntervalInSeconds\": 30,\n \"secureOutput\": false,\n \"secureInput\": false\n },\n \"userProperties\": [],\n \"typeProperties\": {\n \"storedProcedureName\": \"[[procfwk].[SetLogPipelineLastStatusCheck]\",\n \"storedProcedureParameters\": {\n \"ExecutionId\": {\n \"value\": {\n \"value\": \"@pipeline().parameters.executionId\",\n \"type\": \"Expression\"\n },\n \"type\": \"Guid\"\n },\n \"PipelineId\": {\n \"value\": {\n \"value\": \"@pipeline().parameters.pipelineId\",\n \"type\": \"Expression\"\n },\n \"type\": \"Int32\"\n },\n \"StageId\": {\n \"value\": {\n \"value\": \"@pipeline().parameters.stageId\",\n \"type\": \"Expression\"\n },\n \"type\": \"Int32\"\n }\n }\n },\n \"linkedServiceName\": {\n \"referenceName\": \"SupportDatabase\",\n \"type\": \"LinkedServiceReference\"\n }\n },\n {\n \"name\": \"Log Check Function Activity Failure\",\n \"description\": \"Report to the current execution table that the framework pipeline activity has failed. This failure is outside of the scope of the framework and is probably related to a wider platform problem.\",\n \"type\": \"SqlServerStoredProcedure\",\n \"dependsOn\": [\n {\n \"activity\": \"Get Worker Pipeline Status\",\n \"dependencyConditions\": [\n \"Failed\"\n ]\n }\n ],\n \"policy\": {\n \"timeout\": \"7.00:00:00\",\n \"retry\": 0,\n \"retryIntervalInSeconds\": 30,\n \"secureOutput\": false,\n \"secureInput\": false\n },\n \"userProperties\": [],\n \"typeProperties\": {\n \"storedProcedureName\": \"[[procfwk].[SetLogActivityFailed]\",\n \"storedProcedureParameters\": {\n \"CallingActivity\": {\n \"value\": \"GetWorkerPipelineStatus\",\n \"type\": \"String\"\n },\n \"ExecutionId\": {\n \"value\": {\n \"value\": \"@pipeline().parameters.executionId\",\n \"type\": \"Expression\"\n },\n \"type\": \"Guid\"\n },\n \"PipelineId\": {\n \"value\": {\n \"value\": \"@pipeline().parameters.pipelineId\",\n \"type\": \"Expression\"\n },\n \"type\": \"Int32\"\n },\n \"StageId\": {\n \"value\": {\n \"value\": \"@pipeline().parameters.stageId\",\n \"type\": \"Expression\"\n },\n \"type\": \"Int32\"\n }\n }\n },\n \"linkedServiceName\": {\n \"referenceName\": \"SupportDatabase\",\n \"type\": \"LinkedServiceReference\"\n }\n },\n {\n \"name\": \"Set Worker State\",\n \"description\": \"Set the bool state of the Worker pipeline to be used by the Until and If expressions. True = Complete, False = Running.\",\n \"type\": \"SetVariable\",\n \"dependsOn\": [\n {\n \"activity\": \"Get Worker Pipeline Status\",\n \"dependencyConditions\": [\n \"Succeeded\"\n ]\n }\n ],\n \"userProperties\": [],\n \"typeProperties\": {\n \"variableName\": \"WorkerPipelineState\",\n \"value\": {\n \"value\": \"@equals('Done',activity('Get Worker Pipeline Status').output.SimpleStatus)\",\n \"type\": \"Expression\"\n }\n }\n }\n ],\n \"timeout\": \"7.00:00:00\"\n }\n },\n {\n \"name\": \"Set Pipeline Result\",\n \"description\": \"Receives the outcome from the function execution for a given processing pipeline and updates the current execution table with different pipelines status values depending on the result (case).\",\n \"type\": \"Switch\",\n \"dependsOn\": [\n {\n \"activity\": \"Wait Until Pipeline Completes\",\n \"dependencyConditions\": [\n \"Completed\"\n ]\n }\n ],\n \"userProperties\": [],\n \"typeProperties\": {\n \"on\": {\n \"value\": \"@activity('Get Worker Pipeline Status').output.Status\",\n \"type\": \"Expression\"\n },\n \"cases\": [\n {\n \"value\": \"Succeeded\",\n \"activities\": [\n {\n \"name\": \"Pipeline Status Succeeded\",\n \"description\": \"Updates the current execution table with a pipeline status of success if the function outcome is succeeded.\",\n \"type\": \"SqlServerStoredProcedure\",\n \"dependsOn\": [],\n \"policy\": {\n \"timeout\": \"7.00:00:00\",\n \"retry\": 0,\n \"retryIntervalInSeconds\": 30,\n \"secureOutput\": false,\n \"secureInput\": false\n },\n \"userProperties\": [],\n \"typeProperties\": {\n \"storedProcedureName\": \"[[procfwk].[SetLogPipelineSuccess]\",\n \"storedProcedureParameters\": {\n \"ExecutionId\": {\n \"value\": {\n \"value\": \"@pipeline().parameters.executionId\",\n \"type\": \"Expression\"\n },\n \"type\": \"Guid\"\n },\n \"PipelineId\": {\n \"value\": {\n \"value\": \"@pipeline().parameters.pipelineId\",\n \"type\": \"Expression\"\n },\n \"type\": \"Int32\"\n },\n \"StageId\": {\n \"value\": {\n \"value\": \"@pipeline().parameters.stageId\",\n \"type\": \"Expression\"\n },\n \"type\": \"Int32\"\n }\n }\n },\n \"linkedServiceName\": {\n \"referenceName\": \"SupportDatabase\",\n \"type\": \"LinkedServiceReference\"\n }\n }\n ]\n },\n {\n \"value\": \"Failed\",\n \"activities\": [\n {\n \"name\": \"Pipeline Status Failed\",\n \"description\": \"Updates the current execution table with a pipeline status of failed if the function outcome is failed. Also blocks pipelines in the downstream execution stage.\",\n \"type\": \"SqlServerStoredProcedure\",\n \"dependsOn\": [],\n \"policy\": {\n \"timeout\": \"7.00:00:00\",\n \"retry\": 0,\n \"retryIntervalInSeconds\": 30,\n \"secureOutput\": false,\n \"secureInput\": false\n },\n \"userProperties\": [],\n \"typeProperties\": {\n \"storedProcedureName\": \"[[procfwk].[SetLogPipelineFailed]\",\n \"storedProcedureParameters\": {\n \"ExecutionId\": {\n \"value\": {\n \"value\": \"@pipeline().parameters.executionId\",\n \"type\": \"Expression\"\n },\n \"type\": \"Guid\"\n },\n \"PipelineId\": {\n \"value\": {\n \"value\": \"@pipeline().parameters.pipelineId\",\n \"type\": \"Expression\"\n },\n \"type\": \"Int32\"\n },\n \"RunId\": {\n \"value\": {\n \"value\": \"@variables('WorkerRunId')\",\n \"type\": \"Expression\"\n },\n \"type\": \"Guid\"\n },\n \"StageId\": {\n \"value\": {\n \"value\": \"@pipeline().parameters.stageId\",\n \"type\": \"Expression\"\n },\n \"type\": \"Int32\"\n }\n }\n },\n \"linkedServiceName\": {\n \"referenceName\": \"SupportDatabase\",\n \"type\": \"LinkedServiceReference\"\n }\n },\n {\n \"name\": \"Get Worker Pipeline Error Details\",\n \"description\": \"Get the activity error details for the run ID of the worker pipeline called. Returns an array of all errors.\",\n \"type\": \"AzureFunctionActivity\",\n \"dependsOn\": [],\n \"policy\": {\n \"timeout\": \"7.00:00:00\",\n \"retry\": 0,\n \"retryIntervalInSeconds\": 30,\n \"secureOutput\": false,\n \"secureInput\": false\n },\n \"userProperties\": [],\n \"typeProperties\": {\n \"functionName\": \"GetActivityErrors\",\n \"method\": \"POST\",\n \"headers\": {},\n \"body\": {\n \"value\": \"@concat('\\n{\\n \\\"tenantId\\\": \\\"',variables('WorkerTenantId'),'\\\",\\n \\\"applicationId\\\": \\\"',variables('WorkerAppId'),'\\\",\\n \\\"authenticationKey\\\": \\\"',variables('WorkerAppSecret'),'\\\",\\n \\\"subscriptionId\\\": \\\"',variables('WorkerSubscriptionId'),'\\\",\\n \\\"resourceGroup\\\": \\\"',variables('WorkerResourceGroup'),'\\\",\\n \\\"factoryName\\\": \\\"',variables('WorkerDataFactoryName'),'\\\",\\n \\\"pipelineName\\\": \\\"',variables('WorkerPipelineName'),'\\\",\\n \\\"runId\\\": \\\"',variables('WorkerRunId'),'\\\"\\n}')\",\n \"type\": \"Expression\"\n }\n },\n \"linkedServiceName\": {\n \"referenceName\": \"FrameworkFunctions\",\n \"type\": \"LinkedServiceReference\"\n }\n },\n {\n \"name\": \"Log Error Details\",\n \"description\": \"Parses pipeline error details and persists them to the metadata database error log table.\",\n \"type\": \"SqlServerStoredProcedure\",\n \"dependsOn\": [\n {\n \"activity\": \"Get Worker Pipeline Error Details\",\n \"dependencyConditions\": [\n \"Succeeded\"\n ]\n }\n ],\n \"policy\": {\n \"timeout\": \"7.00:00:00\",\n \"retry\": 0,\n \"retryIntervalInSeconds\": 30,\n \"secureOutput\": false,\n \"secureInput\": false\n },\n \"userProperties\": [],\n \"typeProperties\": {\n \"storedProcedureName\": \"[[procfwk].[SetErrorLogDetails]\",\n \"storedProcedureParameters\": {\n \"JsonErrorDetails\": {\n \"value\": {\n \"value\": \"@string(activity('Get Worker Pipeline Error Details').output)\",\n \"type\": \"Expression\"\n },\n \"type\": \"String\"\n },\n \"LocalExecutionId\": {\n \"value\": {\n \"value\": \"@pipeline().parameters.executionId\",\n \"type\": \"Expression\"\n },\n \"type\": \"Guid\"\n }\n }\n },\n \"linkedServiceName\": {\n \"referenceName\": \"SupportDatabase\",\n \"type\": \"LinkedServiceReference\"\n }\n }\n ]\n },\n {\n \"value\": \"Cancelled\",\n \"activities\": [\n {\n \"name\": \"Pipeline Status Cancelled\",\n \"description\": \"Updates the current execution table with a pipeline status of cancelled if the function outcome is cancelled.\",\n \"type\": \"SqlServerStoredProcedure\",\n \"dependsOn\": [],\n \"policy\": {\n \"timeout\": \"7.00:00:00\",\n \"retry\": 0,\n \"retryIntervalInSeconds\": 30,\n \"secureOutput\": false,\n \"secureInput\": false\n },\n \"userProperties\": [],\n \"typeProperties\": {\n \"storedProcedureName\": \"[[procfwk].[SetLogPipelineCancelled]\",\n \"storedProcedureParameters\": {\n \"ExecutionId\": {\n \"value\": {\n \"value\": \"@pipeline().parameters.executionId\",\n \"type\": \"Expression\"\n },\n \"type\": \"Guid\"\n },\n \"PipelineId\": {\n \"value\": {\n \"value\": \"@pipeline().parameters.pipelineId\",\n \"type\": \"Expression\"\n },\n \"type\": \"Int32\"\n },\n \"StageId\": {\n \"value\": {\n \"value\": \"@pipeline().parameters.stageId\",\n \"type\": \"Expression\"\n },\n \"type\": \"Int32\"\n }\n }\n },\n \"linkedServiceName\": {\n \"referenceName\": \"SupportDatabase\",\n \"type\": \"LinkedServiceReference\"\n }\n }\n ]\n }\n ],\n \"defaultActivities\": [\n {\n \"name\": \"Pipeline Status Unknown\",\n \"description\": \"Updates the current execution table with a pipeline status of unknown if the function returns an unexpected outcome.\",\n \"type\": \"SqlServerStoredProcedure\",\n \"dependsOn\": [],\n \"policy\": {\n \"timeout\": \"7.00:00:00\",\n \"retry\": 0,\n \"retryIntervalInSeconds\": 30,\n \"secureOutput\": false,\n \"secureInput\": false\n },\n \"userProperties\": [],\n \"typeProperties\": {\n \"storedProcedureName\": \"[[procfwk].[SetLogPipelineUnknown]\",\n \"storedProcedureParameters\": {\n \"ExecutionId\": {\n \"value\": {\n \"value\": \"@pipeline().parameters.executionId\",\n \"type\": \"Expression\"\n },\n \"type\": \"Guid\"\n },\n \"PipelineId\": {\n \"value\": {\n \"value\": \"@pipeline().parameters.pipelineId\",\n \"type\": \"Expression\"\n },\n \"type\": \"Int32\"\n },\n \"StageId\": {\n \"value\": {\n \"value\": \"@pipeline().parameters.stageId\",\n \"type\": \"Expression\"\n },\n \"type\": \"Int32\"\n }\n }\n },\n \"linkedServiceName\": {\n \"referenceName\": \"SupportDatabase\",\n \"type\": \"LinkedServiceReference\"\n }\n }\n ]\n }\n },\n {\n \"name\": \"Get Wait Duration\",\n \"description\": \"Return wait duration in seconds from database properties table to be used during each Until iteration when the Worker pipeline is still running.\",\n \"type\": \"Lookup\",\n \"dependsOn\": [],\n \"policy\": {\n \"timeout\": \"7.00:00:00\",\n \"retry\": 0,\n \"retryIntervalInSeconds\": 30,\n \"secureOutput\": false,\n \"secureInput\": false\n },\n \"userProperties\": [],\n \"typeProperties\": {\n \"source\": {\n \"type\": \"AzureSqlSource\",\n \"sqlReaderStoredProcedureName\": \"[[procfwk].[GetPropertyValue]\",\n \"storedProcedureParameters\": {\n \"PropertyName\": {\n \"type\": \"String\",\n \"value\": \"PipelineStatusCheckDuration\"\n }\n },\n \"queryTimeout\": \"02:00:00\",\n \"partitionOption\": \"None\"\n },\n \"dataset\": {\n \"referenceName\": \"GetSetMetadata\",\n \"type\": \"DatasetReference\",\n \"parameters\": {}\n }\n }\n },\n {\n \"name\": \"Set App Id\",\n \"type\": \"SetVariable\",\n \"dependsOn\": [\n {\n \"activity\": \"Get Worker Authentication Details\",\n \"dependencyConditions\": [\n \"Succeeded\"\n ]\n }\n ],\n \"userProperties\": [],\n \"typeProperties\": {\n \"variableName\": \"WorkerAppId\",\n \"value\": {\n \"value\": \"@activity('Get Worker Authentication Details').output.firstRow.AppId\",\n \"type\": \"Expression\"\n }\n }\n },\n {\n \"name\": \"Set App Secret\",\n \"type\": \"SetVariable\",\n \"dependsOn\": [\n {\n \"activity\": \"Get Worker Authentication Details\",\n \"dependencyConditions\": [\n \"Succeeded\"\n ]\n }\n ],\n \"userProperties\": [],\n \"typeProperties\": {\n \"variableName\": \"WorkerAppSecret\",\n \"value\": {\n \"value\": \"@activity('Get Worker Authentication Details').output.firstRow.AppSecret\",\n \"type\": \"Expression\"\n }\n }\n },\n {\n \"name\": \"Set Run Id\",\n \"type\": \"SetVariable\",\n \"dependsOn\": [\n {\n \"activity\": \"Execute Worker Pipeline\",\n \"dependencyConditions\": [\n \"Succeeded\"\n ]\n }\n ],\n \"userProperties\": [],\n \"typeProperties\": {\n \"variableName\": \"WorkerRunId\",\n \"value\": {\n \"value\": \"@activity('Execute Worker Pipeline').output.RunId\",\n \"type\": \"Expression\"\n }\n }\n },\n {\n \"name\": \"Get Worker Pipeline Details\",\n \"type\": \"Lookup\",\n \"dependsOn\": [],\n \"policy\": {\n \"timeout\": \"7.00:00:00\",\n \"retry\": 0,\n \"retryIntervalInSeconds\": 30,\n \"secureOutput\": false,\n \"secureInput\": false\n },\n \"userProperties\": [],\n \"typeProperties\": {\n \"source\": {\n \"type\": \"AzureSqlSource\",\n \"sqlReaderStoredProcedureName\": \"[[procfwk].[GetWorkerPipelineDetails]\",\n \"storedProcedureParameters\": {\n \"ExecutionId\": {\n \"type\": \"Guid\",\n \"value\": {\n \"value\": \"@pipeline().parameters.ExecutionId\",\n \"type\": \"Expression\"\n }\n },\n \"PipelineId\": {\n \"type\": \"Int32\",\n \"value\": {\n \"value\": \"@pipeline().parameters.pipelineId\",\n \"type\": \"Expression\"\n }\n },\n \"StageId\": {\n \"type\": \"Int32\",\n \"value\": {\n \"value\": \"@pipeline().parameters.stageId\",\n \"type\": \"Expression\"\n }\n }\n },\n \"queryTimeout\": \"02:00:00\",\n \"partitionOption\": \"None\"\n },\n \"dataset\": {\n \"referenceName\": \"GetSetMetadata\",\n \"type\": \"DatasetReference\",\n \"parameters\": {}\n }\n }\n },\n {\n \"name\": \"Set Tenant Id\",\n \"type\": \"SetVariable\",\n \"dependsOn\": [\n {\n \"activity\": \"Get Worker Authentication Details\",\n \"dependencyConditions\": [\n \"Succeeded\"\n ]\n }\n ],\n \"userProperties\": [],\n \"typeProperties\": {\n \"variableName\": \"WorkerTenantId\",\n \"value\": {\n \"value\": \"@activity('Get Worker Authentication Details').output.firstRow.TenantId\",\n \"type\": \"Expression\"\n }\n }\n },\n {\n \"name\": \"Set Subscription Id\",\n \"type\": \"SetVariable\",\n \"dependsOn\": [\n {\n \"activity\": \"Get Worker Authentication Details\",\n \"dependencyConditions\": [\n \"Succeeded\"\n ]\n }\n ],\n \"userProperties\": [],\n \"typeProperties\": {\n \"variableName\": \"WorkerSubscriptionId\",\n \"value\": {\n \"value\": \"@activity('Get Worker Authentication Details').output.firstRow.SubscriptionId\",\n \"type\": \"Expression\"\n }\n }\n },\n {\n \"name\": \"Set Pipeline Name\",\n \"type\": \"SetVariable\",\n \"dependsOn\": [\n {\n \"activity\": \"Get Worker Pipeline Details\",\n \"dependencyConditions\": [\n \"Succeeded\"\n ]\n }\n ],\n \"userProperties\": [],\n \"typeProperties\": {\n \"variableName\": \"WorkerPipelineName\",\n \"value\": {\n \"value\": \"@activity('Get Worker Pipeline Details').output.firstRow.PipelineName\",\n \"type\": \"Expression\"\n }\n }\n },\n {\n \"name\": \"Set Data Factory Name\",\n \"type\": \"SetVariable\",\n \"dependsOn\": [\n {\n \"activity\": \"Get Worker Pipeline Details\",\n \"dependencyConditions\": [\n \"Succeeded\"\n ]\n }\n ],\n \"userProperties\": [],\n \"typeProperties\": {\n \"variableName\": \"WorkerDataFactoryName\",\n \"value\": {\n \"value\": \"@activity('Get Worker Pipeline Details').output.firstRow.DataFactoryName\",\n \"type\": \"Expression\"\n }\n }\n },\n {\n \"name\": \"Set Resource Group\",\n \"type\": \"SetVariable\",\n \"dependsOn\": [\n {\n \"activity\": \"Get Worker Pipeline Details\",\n \"dependencyConditions\": [\n \"Succeeded\"\n ]\n }\n ],\n \"userProperties\": [],\n \"typeProperties\": {\n \"variableName\": \"WorkerResourceGroup\",\n \"value\": {\n \"value\": \"@activity('Get Worker Pipeline Details').output.firstRow.ResourceGroupName\",\n \"type\": \"Expression\"\n }\n }\n }\n ],\n \"parameters\": {\n \"executionId\": {\n \"type\": \"string\"\n },\n \"stageId\": {\n \"type\": \"int\"\n },\n \"pipelineId\": {\n \"type\": \"int\"\n }\n },\n \"variables\": {\n \"WorkerPipelineState\": {\n \"type\": \"Boolean\"\n },\n \"WorkerAppId\": {\n \"type\": \"String\"\n },\n \"WorkerAppSecret\": {\n \"type\": \"String\"\n },\n \"WorkerRunId\": {\n \"type\": \"String\"\n },\n \"WorkerTenantId\": {\n \"type\": \"String\"\n },\n \"WorkerSubscriptionId\": {\n \"type\": \"String\"\n },\n \"WorkerPipelineName\": {\n \"type\": \"String\"\n },\n \"WorkerDataFactoryName\": {\n \"type\": \"String\"\n },\n \"WorkerResourceGroup\": {\n \"type\": \"String\"\n }\n },\n \"folder\": {\n \"name\": \"_ProcFwk\"\n },\n \"annotations\": [\n \"ADF.procfwk\",\n \"Infant\"\n ]\n },\n \"dependsOn\": [\n \"[concat(variables('factoryId'), '/linkedServices/FrameworkFunctions')]\",\n \"[concat(variables('factoryId'), '/datasets/GetSetMetadata')]\",\n \"[concat(variables('factoryId'), '/linkedServices/SupportDatabase')]\"\n ]\n },\n {\n \"name\": \"[concat(parameters('factoryName'), '/Intentional Error')]\",\n \"type\": \"Microsoft.DataFactory/factories/pipelines\",\n \"apiVersion\": \"2018-06-01\",\n \"properties\": {\n \"description\": \"Used just so the ADF.procfwk has something to call during development.\",\n \"activities\": [\n {\n \"name\": \"Wait1\",\n \"type\": \"Wait\",\n \"dependsOn\": [],\n \"userProperties\": [],\n \"typeProperties\": {\n \"waitTimeInSeconds\": {\n \"value\": \"@pipeline().parameters.WaitTime\",\n \"type\": \"Expression\"\n }\n }\n },\n {\n \"name\": \"Raise Errors or Not\",\n \"type\": \"IfCondition\",\n \"dependsOn\": [\n {\n \"activity\": \"Wait1\",\n \"dependencyConditions\": [\n \"Succeeded\"\n ]\n }\n ],\n \"userProperties\": [],\n \"typeProperties\": {\n \"expression\": {\n \"value\": \"@equals(pipeline().parameters.RaiseErrors,'true')\",\n \"type\": \"Expression\"\n },\n \"ifTrueActivities\": [\n {\n \"name\": \"Call Fail Procedure\",\n \"type\": \"SqlServerStoredProcedure\",\n \"dependsOn\": [],\n \"policy\": {\n \"timeout\": \"7.00:00:00\",\n \"retry\": 0,\n \"retryIntervalInSeconds\": 30,\n \"secureOutput\": false,\n \"secureInput\": false\n },\n \"userProperties\": [],\n \"typeProperties\": {\n \"storedProcedureName\": \"[[dbo].[FailProcedure]\",\n \"storedProcedureParameters\": {\n \"RaiseError\": {\n \"value\": {\n \"value\": \"@pipeline().parameters.RaiseErrors\",\n \"type\": \"Expression\"\n },\n \"type\": \"String\"\n }\n }\n },\n \"linkedServiceName\": {\n \"referenceName\": \"SupportDatabase\",\n \"type\": \"LinkedServiceReference\"\n }\n }\n ]\n }\n }\n ],\n \"parameters\": {\n \"RaiseErrors\": {\n \"type\": \"string\",\n \"defaultValue\": \"false\"\n },\n \"WaitTime\": {\n \"type\": \"int\",\n \"defaultValue\": 5\n }\n },\n \"folder\": {\n \"name\": \"_Workers\"\n },\n \"annotations\": [\n \"Worker\"\n ]\n },\n \"dependsOn\": [\n \"[concat(variables('factoryId'), '/linkedServices/SupportDatabase')]\"\n ]\n },\n {\n \"name\": \"[concat(parameters('factoryName'), '/Wait 1')]\",\n \"type\": \"Microsoft.DataFactory/factories/pipelines\",\n \"apiVersion\": \"2018-06-01\",\n \"properties\": {\n \"description\": \"Used just so the ADF.procfwk has something to call during development.\",\n \"activities\": [\n {\n \"name\": \"Wait1\",\n \"type\": \"Wait\",\n \"dependsOn\": [],\n \"userProperties\": [],\n \"typeProperties\": {\n \"waitTimeInSeconds\": {\n \"value\": \"@pipeline().parameters.WaitTime\",\n \"type\": \"Expression\"\n }\n }\n }\n ],\n \"parameters\": {\n \"WaitTime\": {\n \"type\": \"int\",\n \"defaultValue\": 5\n }\n },\n \"folder\": {\n \"name\": \"_Workers\"\n },\n \"annotations\": [\n \"Worker\"\n ]\n },\n \"dependsOn\": []\n },\n {\n \"name\": \"[concat(parameters('factoryName'), '/Wait 10')]\",\n \"type\": \"Microsoft.DataFactory/factories/pipelines\",\n \"apiVersion\": \"2018-06-01\",\n \"properties\": {\n \"description\": \"Used just so the ADF.procfwk has something to call during development.\",\n \"activities\": [\n {\n \"name\": \"Wait10\",\n \"type\": \"Wait\",\n \"dependsOn\": [],\n \"userProperties\": [],\n \"typeProperties\": {\n \"waitTimeInSeconds\": {\n \"value\": \"@pipeline().parameters.WaitTime\",\n \"type\": \"Expression\"\n }\n }\n }\n ],\n \"parameters\": {\n \"WaitTime\": {\n \"type\": \"int\",\n \"defaultValue\": 5\n }\n },\n \"folder\": {\n \"name\": \"_Workers\"\n },\n \"annotations\": [\n \"Worker\"\n ]\n },\n \"dependsOn\": []\n },\n {\n \"name\": \"[concat(parameters('factoryName'), '/Wait 2')]\",\n \"type\": \"Microsoft.DataFactory/factories/pipelines\",\n \"apiVersion\": \"2018-06-01\",\n \"properties\": {\n \"description\": \"Used just so the ADF.procfwk has something to call during development.\",\n \"activities\": [\n {\n \"name\": \"Wait2\",\n \"type\": \"Wait\",\n \"dependsOn\": [],\n \"userProperties\": [],\n \"typeProperties\": {\n \"waitTimeInSeconds\": {\n \"value\": \"@pipeline().parameters.WaitTime\",\n \"type\": \"Expression\"\n }\n }\n }\n ],\n \"parameters\": {\n \"WaitTime\": {\n \"type\": \"int\",\n \"defaultValue\": 5\n }\n },\n \"folder\": {\n \"name\": \"_Workers\"\n },\n \"annotations\": [\n \"Worker\"\n ]\n },\n \"dependsOn\": []\n },\n {\n \"name\": \"[concat(parameters('factoryName'), '/Wait 3')]\",\n \"type\": \"Microsoft.DataFactory/factories/pipelines\",\n \"apiVersion\": \"2018-06-01\",\n \"properties\": {\n \"description\": \"Used just so the ADF.procfwk has something to call during development.\",\n \"activities\": [\n {\n \"name\": \"Wait3\",\n \"type\": \"Wait\",\n \"dependsOn\": [],\n \"userProperties\": [],\n \"typeProperties\": {\n \"waitTimeInSeconds\": {\n \"value\": \"@pipeline().parameters.WaitTime\",\n \"type\": \"Expression\"\n }\n }\n }\n ],\n \"parameters\": {\n \"WaitTime\": {\n \"type\": \"int\",\n \"defaultValue\": 5\n }\n },\n \"folder\": {\n \"name\": \"_Workers\"\n },\n \"annotations\": [\n \"Worker\"\n ]\n },\n \"dependsOn\": []\n },\n {\n \"name\": \"[concat(parameters('factoryName'), '/Wait 4')]\",\n \"type\": \"Microsoft.DataFactory/factories/pipelines\",\n \"apiVersion\": \"2018-06-01\",\n \"properties\": {\n \"description\": \"Used just so the ADF.procfwk has something to call during development.\",\n \"activities\": [\n {\n \"name\": \"Wait4\",\n \"type\": \"Wait\",\n \"dependsOn\": [],\n \"userProperties\": [],\n \"typeProperties\": {\n \"waitTimeInSeconds\": {\n \"value\": \"@pipeline().parameters.WaitTime\",\n \"type\": \"Expression\"\n }\n }\n }\n ],\n \"parameters\": {\n \"WaitTime\": {\n \"type\": \"int\",\n \"defaultValue\": 5\n }\n },\n \"folder\": {\n \"name\": \"_Workers\"\n },\n \"annotations\": [\n \"Worker\"\n ]\n },\n \"dependsOn\": []\n },\n {\n \"name\": \"[concat(parameters('factoryName'), '/Wait 5')]\",\n \"type\": \"Microsoft.DataFactory/factories/pipelines\",\n \"apiVersion\": \"2018-06-01\",\n \"properties\": {\n \"description\": \"Used just so the ADF.procfwk has something to call during development.\",\n \"activities\": [\n {\n \"name\": \"Wait5\",\n \"type\": \"Wait\",\n \"dependsOn\": [],\n \"userProperties\": [],\n \"typeProperties\": {\n \"waitTimeInSeconds\": {\n \"value\": \"@pipeline().parameters.WaitTime\",\n \"type\": \"Expression\"\n }\n }\n }\n ],\n \"parameters\": {\n \"WaitTime\": {\n \"type\": \"int\",\n \"defaultValue\": 5\n }\n },\n \"folder\": {\n \"name\": \"_Workers\"\n },\n \"annotations\": [\n \"Worker\"\n ]\n },\n \"dependsOn\": []\n },\n {\n \"name\": \"[concat(parameters('factoryName'), '/Wait 6')]\",\n \"type\": \"Microsoft.DataFactory/factories/pipelines\",\n \"apiVersion\": \"2018-06-01\",\n \"properties\": {\n \"description\": \"Used just so the ADF.procfwk has something to call during development.\",\n \"activities\": [\n {\n \"name\": \"Wait6\",\n \"type\": \"Wait\",\n \"dependsOn\": [],\n \"userProperties\": [],\n \"typeProperties\": {\n \"waitTimeInSeconds\": {\n \"value\": \"@pipeline().parameters.WaitTime\",\n \"type\": \"Expression\"\n }\n }\n }\n ],\n \"parameters\": {\n \"WaitTime\": {\n \"type\": \"int\",\n \"defaultValue\": 5\n }\n },\n \"folder\": {\n \"name\": \"_Workers\"\n },\n \"annotations\": [\n \"Worker\"\n ]\n },\n \"dependsOn\": []\n },\n {\n \"name\": \"[concat(parameters('factoryName'), '/Wait 7')]\",\n \"type\": \"Microsoft.DataFactory/factories/pipelines\",\n \"apiVersion\": \"2018-06-01\",\n \"properties\": {\n \"description\": \"Used just so the ADF.procfwk has something to call during development.\",\n \"activities\": [\n {\n \"name\": \"Wait7\",\n \"type\": \"Wait\",\n \"dependsOn\": [],\n \"userProperties\": [],\n \"typeProperties\": {\n \"waitTimeInSeconds\": {\n \"value\": \"@pipeline().parameters.WaitTime\",\n \"type\": \"Expression\"\n }\n }\n }\n ],\n \"parameters\": {\n \"WaitTime\": {\n \"type\": \"int\",\n \"defaultValue\": 5\n }\n },\n \"folder\": {\n \"name\": \"_Workers\"\n },\n \"annotations\": [\n \"Worker\"\n ]\n },\n \"dependsOn\": []\n },\n {\n \"name\": \"[concat(parameters('factoryName'), '/Wait 8')]\",\n \"type\": \"Microsoft.DataFactory/factories/pipelines\",\n \"apiVersion\": \"2018-06-01\",\n \"properties\": {\n \"description\": \"Used just so the ADF.procfwk has something to call during development.\",\n \"activities\": [\n {\n \"name\": \"Wait8\",\n \"type\": \"Wait\",\n \"dependsOn\": [],\n \"userProperties\": [],\n \"typeProperties\": {\n \"waitTimeInSeconds\": {\n \"value\": \"@pipeline().parameters.WaitTime\",\n \"type\": \"Expression\"\n }\n }\n }\n ],\n \"parameters\": {\n \"WaitTime\": {\n \"type\": \"int\",\n \"defaultValue\": 5\n }\n },\n \"folder\": {\n \"name\": \"_Workers\"\n },\n \"annotations\": [\n \"Worker\"\n ]\n },\n \"dependsOn\": []\n },\n {\n \"name\": \"[concat(parameters('factoryName'), '/Wait 9')]\",\n \"type\": \"Microsoft.DataFactory/factories/pipelines\",\n \"apiVersion\": \"2018-06-01\",\n \"properties\": {\n \"description\": \"Used just so the ADF.procfwk has something to call during development.\",\n \"activities\": [\n {\n \"name\": \"Wait9\",\n \"type\": \"Wait\",\n \"dependsOn\": [],\n \"userProperties\": [],\n \"typeProperties\": {\n \"waitTimeInSeconds\": {\n \"value\": \"@pipeline().parameters.WaitTime\",\n \"type\": \"Expression\"\n }\n }\n }\n ],\n \"parameters\": {\n \"WaitTime\": {\n \"type\": \"int\",\n \"defaultValue\": 15\n }\n },\n \"folder\": {\n \"name\": \"_Workers\"\n },\n \"annotations\": [\n \"Worker\"\n ]\n },\n \"dependsOn\": []\n },\n {\n \"name\": \"[concat(parameters('factoryName'), '/GetSetMetadata')]\",\n \"type\": \"Microsoft.DataFactory/factories/datasets\",\n \"apiVersion\": \"2018-06-01\",\n \"properties\": {\n \"linkedServiceName\": {\n \"referenceName\": \"SupportDatabase\",\n \"type\": \"LinkedServiceReference\"\n },\n \"annotations\": [\n \"ADF.procfwk\"\n ],\n \"type\": \"AzureSqlTable\",\n \"schema\": [],\n \"typeProperties\": {}\n },\n \"dependsOn\": [\n \"[concat(variables('factoryId'), '/linkedServices/SupportDatabase')]\"\n ]\n },\n {\n \"name\": \"[concat(parameters('factoryName'), '/FrameworkFunctions')]\",\n \"type\": \"Microsoft.DataFactory/factories/linkedServices\",\n \"apiVersion\": \"2018-06-01\",\n \"properties\": {\n \"annotations\": [\n \"ADF.procfwk\"\n ],\n \"type\": \"AzureFunction\",\n \"typeProperties\": {\n \"functionAppUrl\": \"[parameters('FrameworkFunctions_properties_typeProperties_functionAppUrl')]\",\n \"functionKey\": {\n \"type\": \"AzureKeyVaultSecret\",\n \"store\": {\n \"referenceName\": \"Keys\",\n \"type\": \"LinkedServiceReference\"\n },\n \"secretName\": \"FrameworkFunctionsKey\"\n }\n }\n },\n \"dependsOn\": [\n \"[concat(variables('factoryId'), '/linkedServices/Keys')]\"\n ]\n },\n {\n \"name\": \"[concat(parameters('factoryName'), '/Keys')]\",\n \"type\": \"Microsoft.DataFactory/factories/linkedServices\",\n \"apiVersion\": \"2018-06-01\",\n \"properties\": {\n \"description\": \"Connection to Key Vault for all other ADF linked service credentials required to run the processing framework.\",\n \"annotations\": [\n \"ADF.procfwk\"\n ],\n \"type\": \"AzureKeyVault\",\n \"typeProperties\": {\n \"baseUrl\": \"[parameters('Keys_properties_typeProperties_baseUrl')]\"\n }\n },\n \"dependsOn\": []\n },\n {\n \"name\": \"[concat(parameters('factoryName'), '/SupportDatabase')]\",\n \"type\": \"Microsoft.DataFactory/factories/linkedServices\",\n \"apiVersion\": \"2018-06-01\",\n \"properties\": {\n \"description\": \"Connection between ADF and processing framework metadata SQLDB.\",\n \"annotations\": [\n \"ADF.procfwk\"\n ],\n \"type\": \"AzureSqlDatabase\",\n \"typeProperties\": {\n \"connectionString\": {\n \"type\": \"AzureKeyVaultSecret\",\n \"store\": {\n \"referenceName\": \"Keys\",\n \"type\": \"LinkedServiceReference\"\n },\n \"secretName\": \"[parameters('SupportDatabase_properties_typeProperties_connectionString_secretName')]\"\n }\n }\n },\n \"dependsOn\": [\n \"[concat(variables('factoryId'), '/linkedServices/Keys')]\"\n ]\n },\n {\n \"name\": \"[concat(parameters('factoryName'), '/FunctionalTestingTrigger')]\",\n \"type\": \"Microsoft.DataFactory/factories/triggers\",\n \"apiVersion\": \"2018-06-01\",\n \"properties\": {\n \"description\": \"Used for functional testing of the framework in a dedicated environment.\",\n \"annotations\": [\n \"ADF.procfwk\"\n ],\n \"runtimeState\": \"Stopped\",\n \"pipelines\": [\n {\n \"pipelineReference\": {\n \"referenceName\": \"01-Grandparent\",\n \"type\": \"PipelineReference\"\n },\n \"parameters\": {}\n }\n ],\n \"type\": \"ScheduleTrigger\",\n \"typeProperties\": {\n \"recurrence\": {\n \"frequency\": \"Hour\",\n \"interval\": 2,\n \"startTime\": \"2020-04-06T15:00:00.000Z\",\n \"timeZone\": \"UTC\"\n }\n }\n },\n \"dependsOn\": [\n \"[concat(variables('factoryId'), '/pipelines/01-Grandparent')]\"\n ]\n }\n ]\n}"
},
{
"path": "ARM Templates/Data Factory/v1.9.1 Export.json",
"content": "{\n \"$schema\": \"http://schema.management.azure.com/schemas/2015-01-01/deploymentTemplate.json#\",\n \"contentVersion\": \"1.0.0.0\",\n \"parameters\": {\n \"factoryName\": {\n \"type\": \"string\",\n \"metadata\": \"Data Factory name\",\n \"defaultValue\": \"\"\n },\n \"FrameworkFunctions_properties_typeProperties_functionAppUrl\": {\n \"type\": \"string\",\n \"defaultValue\": \"\"\n },\n \"Keys_properties_typeProperties_baseUrl\": {\n \"type\": \"string\",\n \"defaultValue\": \"\"\n },\n \"SupportDatabase_properties_typeProperties_connectionString_secretName\": {\n \"type\": \"string\",\n \"defaultValue\": \"\"\n }\n },\n \"variables\": {\n \"factoryId\": \"[concat('Microsoft.DataFactory/factories/', parameters('factoryName'))]\"\n },\n \"resources\": [\n {\n \"name\": \"[concat(parameters('factoryName'), '/01-Grandparent')]\",\n \"type\": \"Microsoft.DataFactory/factories/pipelines\",\n \"apiVersion\": \"2018-06-01\",\n \"properties\": {\n \"description\": \"ADF.procfwk grandparent pipeline used optionally to bootstrap any wider processes in your Data Factory that then calls the processing framework.\",\n \"activities\": [\n {\n \"name\": \"Framework Processing\",\n \"description\": \"Call procfwk\",\n \"type\": \"ExecutePipeline\",\n \"dependsOn\": [],\n \"userProperties\": [],\n \"typeProperties\": {\n \"pipeline\": {\n \"referenceName\": \"02-Parent\",\n \"type\": \"PipelineReference\"\n },\n \"waitOnCompletion\": true,\n \"parameters\": {}\n }\n }\n ],\n \"folder\": {\n \"name\": \"_ProcFwk\"\n },\n \"annotations\": [\n \"ADF.procfwk\",\n \"Grandparent\"\n ]\n },\n \"dependsOn\": [\n \"[concat(variables('factoryId'), '/pipelines/02-Parent')]\"\n ]\n },\n {\n \"name\": \"[concat(parameters('factoryName'), '/02-Parent')]\",\n \"type\": \"Microsoft.DataFactory/factories/pipelines\",\n \"apiVersion\": \"2018-06-01\",\n \"properties\": {\n \"description\": \"ADF.procfwk parent pipeline used to bootstrap the orchestration framework in perform the first level ForEach calls in sequence for the metadata stages.\",\n \"activities\": [\n {\n \"name\": \"Get Stages\",\n \"description\": \"Returns a distinct list of execution stages within the framework metadata.\",\n \"type\": \"Lookup\",\n \"dependsOn\": [\n {\n \"activity\": \"Set Execution Id\",\n \"dependencyConditions\": [\n \"Succeeded\"\n ]\n }\n ],\n \"policy\": {\n \"timeout\": \"0.00:10:00\",\n \"retry\": 0,\n \"retryIntervalInSeconds\": 30,\n \"secureOutput\": false,\n \"secureInput\": false\n },\n \"userProperties\": [],\n \"typeProperties\": {\n \"source\": {\n \"type\": \"AzureSqlSource\",\n \"sqlReaderStoredProcedureName\": \"[[procfwk].[GetStages]\",\n \"storedProcedureParameters\": {\n \"ExecutionId\": {\n \"type\": \"Guid\",\n \"value\": {\n \"value\": \"@variables('ExecutionId')\",\n \"type\": \"Expression\"\n }\n }\n },\n \"queryTimeout\": \"02:00:00\",\n \"partitionOption\": \"None\"\n },\n \"dataset\": {\n \"referenceName\": \"GetSetMetadata\",\n \"type\": \"DatasetReference\",\n \"parameters\": {}\n },\n \"firstRowOnly\": false\n }\n },\n {\n \"name\": \"Execute Stages\",\n \"description\": \"Top level ForEach to sequentially call all processing stages within the framework metadata. Items for iteration passed from the Get Stages lookup activity.\",\n \"type\": \"ForEach\",\n \"dependsOn\": [\n {\n \"activity\": \"Get Stages\",\n \"dependencyConditions\": [\n \"Succeeded\"\n ]\n }\n ],\n \"userProperties\": [],\n \"typeProperties\": {\n \"items\": {\n \"value\": \"@activity('Get Stages').output.value\",\n \"type\": \"Expression\"\n },\n \"isSequential\": true,\n \"activities\": [\n {\n \"name\": \"Stage Executor\",\n \"description\": \"Call to the framework generic child pipeline for a given execution stage.\",\n \"type\": \"ExecutePipeline\",\n \"dependsOn\": [\n {\n \"activity\": \"Log Stage Preparing\",\n \"dependencyConditions\": [\n \"Succeeded\"\n ]\n }\n ],\n \"userProperties\": [],\n \"typeProperties\": {\n \"pipeline\": {\n \"referenceName\": \"03-Child\",\n \"type\": \"PipelineReference\"\n },\n \"waitOnCompletion\": true,\n \"parameters\": {\n \"StageId\": {\n \"value\": \"@item().StageId\",\n \"type\": \"Expression\"\n },\n \"ExecutionId\": {\n \"value\": \"@variables('ExecutionId')\",\n \"type\": \"Expression\"\n }\n }\n }\n },\n {\n \"name\": \"Log Stage Preparing\",\n \"description\": \"Update the current execution table flagging all pipelines within the stage as preparing.\",\n \"type\": \"SqlServerStoredProcedure\",\n \"dependsOn\": [\n {\n \"activity\": \"Check and Update Blockers\",\n \"dependencyConditions\": [\n \"Succeeded\"\n ]\n }\n ],\n \"policy\": {\n \"timeout\": \"0.00:10:00\",\n \"retry\": 0,\n \"retryIntervalInSeconds\": 30,\n \"secureOutput\": false,\n \"secureInput\": false\n },\n \"userProperties\": [],\n \"typeProperties\": {\n \"storedProcedureName\": \"[[procfwk].[SetLogStagePreparing]\",\n \"storedProcedureParameters\": {\n \"ExecutionId\": {\n \"value\": {\n \"value\": \"@variables('ExecutionId')\",\n \"type\": \"Expression\"\n },\n \"type\": \"Guid\"\n },\n \"StageId\": {\n \"value\": {\n \"value\": \"@item().StageId\",\n \"type\": \"Expression\"\n },\n \"type\": \"Int32\"\n }\n }\n },\n \"linkedServiceName\": {\n \"referenceName\": \"SupportDatabase\",\n \"type\": \"LinkedServiceReference\"\n }\n },\n {\n \"name\": \"Check and Update Blockers\",\n \"description\": \"Used to double check and stop the next execution stage if failures and blockers have be incurred. This also depends on the failure handling property value which defines the stored procedure behaviour.\",\n \"type\": \"SqlServerStoredProcedure\",\n \"dependsOn\": [],\n \"policy\": {\n \"timeout\": \"0.00:10:00\",\n \"retry\": 0,\n \"retryIntervalInSeconds\": 30,\n \"secureOutput\": false,\n \"secureInput\": false\n },\n \"userProperties\": [],\n \"typeProperties\": {\n \"storedProcedureName\": \"[[procfwk].[CheckForBlockedPipelines]\",\n \"storedProcedureParameters\": {\n \"ExecutionId\": {\n \"value\": {\n \"value\": \"@variables('ExecutionId')\",\n \"type\": \"Expression\"\n },\n \"type\": \"Guid\"\n },\n \"StageId\": {\n \"value\": {\n \"value\": \"@item().StageId\",\n \"type\": \"Expression\"\n },\n \"type\": \"Int32\"\n }\n }\n },\n \"linkedServiceName\": {\n \"referenceName\": \"SupportDatabase\",\n \"type\": \"LinkedServiceReference\"\n }\n }\n ]\n }\n },\n {\n \"name\": \"Execution Wrapper\",\n \"description\": \"Wrapper to reset and restart processing or create a completely new execution instance of the framework metadata.\",\n \"type\": \"Lookup\",\n \"dependsOn\": [\n {\n \"activity\": \"Clean Up Previous Run\",\n \"dependencyConditions\": [\n \"Succeeded\"\n ]\n }\n ],\n \"policy\": {\n \"timeout\": \"0.00:10:00\",\n \"retry\": 0,\n \"retryIntervalInSeconds\": 30,\n \"secureOutput\": false,\n \"secureInput\": false\n },\n \"userProperties\": [],\n \"typeProperties\": {\n \"source\": {\n \"type\": \"AzureSqlSource\",\n \"sqlReaderStoredProcedureName\": \"[[procfwk].[ExecutionWrapper]\",\n \"storedProcedureParameters\": {\n \"CallingDataFactory\": {\n \"type\": \"String\",\n \"value\": {\n \"value\": \"@pipeline().DataFactory\",\n \"type\": \"Expression\"\n }\n }\n },\n \"queryTimeout\": \"02:00:00\",\n \"partitionOption\": \"None\"\n },\n \"dataset\": {\n \"referenceName\": \"GetSetMetadata\",\n \"type\": \"DatasetReference\",\n \"parameters\": {}\n }\n }\n },\n {\n \"name\": \"Check Outcome and Update Logs\",\n \"description\": \"After a successful execution run the current execution metadata is moved to the long term logging table by this stored procedure call. Otherwise an error will be raised.\",\n \"type\": \"SqlServerStoredProcedure\",\n \"dependsOn\": [\n {\n \"activity\": \"Execute Stages\",\n \"dependencyConditions\": [\n \"Succeeded\"\n ]\n }\n ],\n \"policy\": {\n \"timeout\": \"0.00:10:00\",\n \"retry\": 0,\n \"retryIntervalInSeconds\": 30,\n \"secureOutput\": false,\n \"secureInput\": false\n },\n \"userProperties\": [],\n \"typeProperties\": {\n \"storedProcedureName\": \"[[procfwk].[UpdateExecutionLog]\",\n \"storedProcedureParameters\": {\n \"PerformErrorCheck\": {\n \"value\": {\n \"value\": \"@bool(1)\",\n \"type\": \"Expression\"\n },\n \"type\": \"Boolean\"\n }\n }\n },\n \"linkedServiceName\": {\n \"referenceName\": \"SupportDatabase\",\n \"type\": \"LinkedServiceReference\"\n }\n },\n {\n \"name\": \"Metadata Integrity Checks\",\n \"description\": \"Performs a series of checks on all metadata held in the framework SQLDB. This is intended to raise errors before an execution run even starts.\",\n \"type\": \"Lookup\",\n \"dependsOn\": [\n {\n \"activity\": \"Execute Precursor\",\n \"dependencyConditions\": [\n \"Succeeded\"\n ]\n }\n ],\n \"policy\": {\n \"timeout\": \"0.00:10:00\",\n \"retry\": 0,\n \"retryIntervalInSeconds\": 30,\n \"secureOutput\": false,\n \"secureInput\": false\n },\n \"userProperties\": [],\n \"typeProperties\": {\n \"source\": {\n \"type\": \"AzureSqlSource\",\n \"sqlReaderStoredProcedureName\": \"[[procfwk].[CheckMetadataIntegrity]\",\n \"storedProcedureParameters\": {\n \"DebugMode\": {\n \"type\": \"Boolean\",\n \"value\": \"false\"\n }\n },\n \"queryTimeout\": \"02:00:00\",\n \"partitionOption\": \"None\"\n },\n \"dataset\": {\n \"referenceName\": \"GetSetMetadata\",\n \"type\": \"DatasetReference\",\n \"parameters\": {}\n },\n \"firstRowOnly\": false\n }\n },\n {\n \"name\": \"Clean Up Previous Run\",\n \"description\": \"Handle Worker pipelines that are reported as Running when the parent pipeline is called again. Get what the actual status of those pipelines is.\",\n \"type\": \"ForEach\",\n \"dependsOn\": [\n {\n \"activity\": \"Metadata Integrity Checks\",\n \"dependencyConditions\": [\n \"Succeeded\"\n ]\n }\n ],\n \"userProperties\": [],\n \"typeProperties\": {\n \"items\": {\n \"value\": \"@activity('Metadata Integrity Checks').output.value\",\n \"type\": \"Expression\"\n },\n \"isSequential\": false,\n \"batchCount\": 50,\n \"activities\": [\n {\n \"name\": \"Get SPN Details\",\n \"type\": \"Lookup\",\n \"dependsOn\": [],\n \"policy\": {\n \"timeout\": \"0.00:10:00\",\n \"retry\": 0,\n \"retryIntervalInSeconds\": 30,\n \"secureOutput\": false,\n \"secureInput\": false\n },\n \"userProperties\": [],\n \"typeProperties\": {\n \"source\": {\n \"type\": \"AzureSqlSource\",\n \"sqlReaderStoredProcedureName\": \"[[procfwk].[GetWorkerAuthDetails]\",\n \"storedProcedureParameters\": {\n \"ExecutionId\": {\n \"type\": \"Guid\",\n \"value\": {\n \"value\": \"@item().LocalExecutionId\",\n \"type\": \"Expression\"\n }\n },\n \"PipelineId\": {\n \"type\": \"Int32\",\n \"value\": {\n \"value\": \"@item().PipelineId\",\n \"type\": \"Expression\"\n }\n },\n \"StageId\": {\n \"type\": \"Int32\",\n \"value\": {\n \"value\": \"@item().StageId\",\n \"type\": \"Expression\"\n }\n }\n },\n \"queryTimeout\": \"02:00:00\",\n \"partitionOption\": \"None\"\n },\n \"dataset\": {\n \"referenceName\": \"GetSetMetadata\",\n \"type\": \"DatasetReference\",\n \"parameters\": {}\n }\n }\n },\n {\n \"name\": \"Log Pipeline Checking\",\n \"type\": \"SqlServerStoredProcedure\",\n \"dependsOn\": [],\n \"policy\": {\n \"timeout\": \"0.00:10:00\",\n \"retry\": 0,\n \"retryIntervalInSeconds\": 30,\n \"secureOutput\": false,\n \"secureInput\": false\n },\n \"userProperties\": [],\n \"typeProperties\": {\n \"storedProcedureName\": \"[[procfwk].[SetLogPipelineChecking]\",\n \"storedProcedureParameters\": {\n \"ExecutionId\": {\n \"value\": {\n \"value\": \"@item().LocalExecutionId\",\n \"type\": \"Expression\"\n },\n \"type\": \"Guid\"\n },\n \"PipelineId\": {\n \"value\": {\n \"value\": \"@item().PipelineId\",\n \"type\": \"Expression\"\n },\n \"type\": \"Int32\"\n },\n \"StageId\": {\n \"value\": {\n \"value\": \"@item().StageId\",\n \"type\": \"Expression\"\n },\n \"type\": \"Int32\"\n }\n }\n },\n \"linkedServiceName\": {\n \"referenceName\": \"SupportDatabase\",\n \"type\": \"LinkedServiceReference\"\n }\n },\n {\n \"name\": \"Get Pipeline Status\",\n \"type\": \"AzureFunctionActivity\",\n \"dependsOn\": [\n {\n \"activity\": \"Get SPN Details\",\n \"dependencyConditions\": [\n \"Succeeded\"\n ]\n },\n {\n \"activity\": \"Log Pipeline Checking\",\n \"dependencyConditions\": [\n \"Succeeded\"\n ]\n }\n ],\n \"policy\": {\n \"timeout\": \"0.00:10:00\",\n \"retry\": 0,\n \"retryIntervalInSeconds\": 30,\n \"secureOutput\": false,\n \"secureInput\": true\n },\n \"userProperties\": [],\n \"typeProperties\": {\n \"functionName\": \"CheckPipelineStatus\",\n \"method\": \"POST\",\n \"headers\": {},\n \"body\": {\n \"value\": \"@concat('\\n{\\n \\\"tenantId\\\": \\\"',activity('Get SPN Details').output.firstRow.TenantId,'\\\",\\n \\\"applicationId\\\": \\\"',activity('Get SPN Details').output.firstRow.AppId,'\\\",\\n \\\"authenticationKey\\\": \\\"',activity('Get SPN Details').output.firstRow.AppSecret,'\\\",\\n \\\"subscriptionId\\\": \\\"',activity('Get SPN Details').output.firstRow.SubscriptionId,'\\\",\\n \\\"resourceGroup\\\": \\\"',item().ResourceGroupName,'\\\",\\n \\\"factoryName\\\": \\\"',item().DataFactoryName,'\\\",\\n \\\"pipelineName\\\": \\\"',item().PipelineName,'\\\",\\n \\\"runId\\\": \\\"',item().AdfPipelineRunId,'\\\"\\n}')\",\n \"type\": \"Expression\"\n }\n },\n \"linkedServiceName\": {\n \"referenceName\": \"FrameworkFunctions\",\n \"type\": \"LinkedServiceReference\"\n }\n },\n {\n \"name\": \"Set Pipeline Status\",\n \"description\": \"Update the metadata depending on the actual pipeline outcome. Using the status as the case.\",\n \"type\": \"Switch\",\n \"dependsOn\": [\n {\n \"activity\": \"Get Pipeline Status\",\n \"dependencyConditions\": [\n \"Succeeded\"\n ]\n }\n ],\n \"userProperties\": [],\n \"typeProperties\": {\n \"on\": {\n \"value\": \"@activity('Get Pipeline Status').output.Status\",\n \"type\": \"Expression\"\n },\n \"cases\": [\n {\n \"value\": \"Failed\",\n \"activities\": [\n {\n \"name\": \"Pipeline Status Failed\",\n \"description\": \"Updates the current execution table with a pipeline status of failed if the function outcome is failed. Also blocks pipelines in the downstream execution stage.\",\n \"type\": \"SqlServerStoredProcedure\",\n \"dependsOn\": [],\n \"policy\": {\n \"timeout\": \"0.00:10:00\",\n \"retry\": 0,\n \"retryIntervalInSeconds\": 30,\n \"secureOutput\": false,\n \"secureInput\": false\n },\n \"userProperties\": [],\n \"typeProperties\": {\n \"storedProcedureName\": \"[[procfwk].[SetLogPipelineFailed]\",\n \"storedProcedureParameters\": {\n \"ExecutionId\": {\n \"value\": {\n \"value\": \"@item().LocalExecutionId\",\n \"type\": \"Expression\"\n },\n \"type\": \"Guid\"\n },\n \"PipelineId\": {\n \"value\": {\n \"value\": \"@item().PipelineId\",\n \"type\": \"Expression\"\n },\n \"type\": \"Int32\"\n },\n \"RunId\": {\n \"value\": null,\n \"type\": \"Guid\"\n },\n \"StageId\": {\n \"value\": {\n \"value\": \"@item().StageId\",\n \"type\": \"Expression\"\n },\n \"type\": \"Int32\"\n }\n }\n },\n \"linkedServiceName\": {\n \"referenceName\": \"SupportDatabase\",\n \"type\": \"LinkedServiceReference\"\n }\n }\n ]\n },\n {\n \"value\": \"Succeeded\",\n \"activities\": [\n {\n \"name\": \"Pipeline Status Succeeded\",\n \"description\": \"Updates the current execution table with a pipeline status of success if the function outcome is succeeded.\",\n \"type\": \"SqlServerStoredProcedure\",\n \"dependsOn\": [],\n \"policy\": {\n \"timeout\": \"0.00:10:00\",\n \"retry\": 0,\n \"retryIntervalInSeconds\": 30,\n \"secureOutput\": false,\n \"secureInput\": false\n },\n \"userProperties\": [],\n \"typeProperties\": {\n \"storedProcedureName\": \"[[procfwk].[SetLogPipelineSuccess]\",\n \"storedProcedureParameters\": {\n \"ExecutionId\": {\n \"value\": {\n \"value\": \"@item().LocalExecutionId\",\n \"type\": \"Expression\"\n },\n \"type\": \"Guid\"\n },\n \"PipelineId\": {\n \"value\": {\n \"value\": \"@item().PipelineId\",\n \"type\": \"Expression\"\n },\n \"type\": \"Int32\"\n },\n \"StageId\": {\n \"value\": {\n \"value\": \"@item().StageId\",\n \"type\": \"Expression\"\n },\n \"type\": \"Int32\"\n }\n }\n },\n \"linkedServiceName\": {\n \"referenceName\": \"SupportDatabase\",\n \"type\": \"LinkedServiceReference\"\n }\n }\n ]\n },\n {\n \"value\": \"Queued\",\n \"activities\": [\n {\n \"name\": \"Pipeline Status Queued - Running\",\n \"description\": \"Updates the current execution table with a pipeline status of running if the function outcome is queued.\",\n \"type\": \"SqlServerStoredProcedure\",\n \"dependsOn\": [],\n \"policy\": {\n \"timeout\": \"0.00:10:00\",\n \"retry\": 0,\n \"retryIntervalInSeconds\": 30,\n \"secureOutput\": false,\n \"secureInput\": false\n },\n \"userProperties\": [],\n \"typeProperties\": {\n \"storedProcedureName\": \"[[procfwk].[SetLogPipelineRunning]\",\n \"storedProcedureParameters\": {\n \"ExecutionId\": {\n \"value\": {\n \"value\": \"@item().LocalExecutionId\",\n \"type\": \"Expression\"\n },\n \"type\": \"Guid\"\n },\n \"PipelineId\": {\n \"value\": {\n \"value\": \"@item().PipelineId\",\n \"type\": \"Expression\"\n },\n \"type\": \"Int32\"\n },\n \"StageId\": {\n \"value\": {\n \"value\": \"@item().StageId\",\n \"type\": \"Expression\"\n },\n \"type\": \"Int32\"\n }\n }\n },\n \"linkedServiceName\": {\n \"referenceName\": \"SupportDatabase\",\n \"type\": \"LinkedServiceReference\"\n }\n }\n ]\n },\n {\n \"value\": \"InProgress\",\n \"activities\": [\n {\n \"name\": \"Pipeline Status InProgress - Running\",\n \"description\": \"Updates the current execution table with a pipeline status of running if the function outcome is in progress.\",\n \"type\": \"SqlServerStoredProcedure\",\n \"dependsOn\": [],\n \"policy\": {\n \"timeout\": \"0.00:10:00\",\n \"retry\": 0,\n \"retryIntervalInSeconds\": 30,\n \"secureOutput\": false,\n \"secureInput\": false\n },\n \"userProperties\": [],\n \"typeProperties\": {\n \"storedProcedureName\": \"[[procfwk].[SetLogPipelineRunning]\",\n \"storedProcedureParameters\": {\n \"ExecutionId\": {\n \"value\": {\n \"value\": \"@item().LocalExecutionId\",\n \"type\": \"Expression\"\n },\n \"type\": \"Guid\"\n },\n \"PipelineId\": {\n \"value\": {\n \"value\": \"@item().PipelineId\",\n \"type\": \"Expression\"\n },\n \"type\": \"Int32\"\n },\n \"StageId\": {\n \"value\": {\n \"value\": \"@item().StageId\",\n \"type\": \"Expression\"\n },\n \"type\": \"Int32\"\n }\n }\n },\n \"linkedServiceName\": {\n \"referenceName\": \"SupportDatabase\",\n \"type\": \"LinkedServiceReference\"\n }\n }\n ]\n },\n {\n \"value\": \"Cancelled\",\n \"activities\": [\n {\n \"name\": \"Pipeline Status Cancelled\",\n \"description\": \"Updates the current execution table with a pipeline status of cancelled if the function outcome is cancelled.\",\n \"type\": \"SqlServerStoredProcedure\",\n \"dependsOn\": [],\n \"policy\": {\n \"timeout\": \"0.00:10:00\",\n \"retry\": 0,\n \"retryIntervalInSeconds\": 30,\n \"secureOutput\": false,\n \"secureInput\": false\n },\n \"userProperties\": [],\n \"typeProperties\": {\n \"storedProcedureName\": \"[[procfwk].[SetLogPipelineCancelled]\",\n \"storedProcedureParameters\": {\n \"ExecutionId\": {\n \"value\": {\n \"value\": \"@item().LocalExecutionId\",\n \"type\": \"Expression\"\n },\n \"type\": \"Guid\"\n },\n \"PipelineId\": {\n \"value\": {\n \"value\": \"@item().PipelineId\",\n \"type\": \"Expression\"\n },\n \"type\": \"Int32\"\n },\n \"StageId\": {\n \"value\": {\n \"value\": \"@item().StageId\",\n \"type\": \"Expression\"\n },\n \"type\": \"Int32\"\n },\n \"CleanUpRun\": {\n \"value\": {\n \"value\": \"@bool(1)\",\n \"type\": \"Expression\"\n },\n \"type\": \"Boolean\"\n }\n }\n },\n \"linkedServiceName\": {\n \"referenceName\": \"SupportDatabase\",\n \"type\": \"LinkedServiceReference\"\n }\n }\n ]\n }\n ],\n \"defaultActivities\": [\n {\n \"name\": \"Pipeline Status Unknown\",\n \"description\": \"Updates the current execution table with a pipeline status of unknown if the function returns an unexpected outcome.\",\n \"type\": \"SqlServerStoredProcedure\",\n \"dependsOn\": [],\n \"policy\": {\n \"timeout\": \"0.00:10:00\",\n \"retry\": 0,\n \"retryIntervalInSeconds\": 30,\n \"secureOutput\": false,\n \"secureInput\": false\n },\n \"userProperties\": [],\n \"typeProperties\": {\n \"storedProcedureName\": \"[[procfwk].[SetLogPipelineUnknown]\",\n \"storedProcedureParameters\": {\n \"ExecutionId\": {\n \"value\": {\n \"value\": \"@item().LocalExecutionId\",\n \"type\": \"Expression\"\n },\n \"type\": \"Guid\"\n },\n \"PipelineId\": {\n \"value\": {\n \"value\": \"@item().PipelineId\",\n \"type\": \"Expression\"\n },\n \"type\": \"Int32\"\n },\n \"StageId\": {\n \"value\": {\n \"value\": \"@item().StageId\",\n \"type\": \"Expression\"\n },\n \"type\": \"Int32\"\n },\n \"CleanUpRun\": {\n \"value\": {\n \"value\": \"@bool(1)\",\n \"type\": \"Expression\"\n },\n \"type\": \"Boolean\"\n }\n }\n },\n \"linkedServiceName\": {\n \"referenceName\": \"SupportDatabase\",\n \"type\": \"LinkedServiceReference\"\n }\n }\n ]\n }\n },\n {\n \"name\": \"Set Last Check DateTime\",\n \"description\": \"Update the current execution table with a date time from when the function last checked the pipeline status.\",\n \"type\": \"SqlServerStoredProcedure\",\n \"dependsOn\": [\n {\n \"activity\": \"Get Pipeline Status\",\n \"dependencyConditions\": [\n \"Succeeded\"\n ]\n }\n ],\n \"policy\": {\n \"timeout\": \"0.00:10:00\",\n \"retry\": 0,\n \"retryIntervalInSeconds\": 30,\n \"secureOutput\": false,\n \"secureInput\": false\n },\n \"userProperties\": [],\n \"typeProperties\": {\n \"storedProcedureName\": \"[[procfwk].[SetLogPipelineLastStatusCheck]\",\n \"storedProcedureParameters\": {\n \"ExecutionId\": {\n \"value\": {\n \"value\": \"@item().LocalExecutionId\",\n \"type\": \"Expression\"\n },\n \"type\": \"Guid\"\n },\n \"PipelineId\": {\n \"value\": {\n \"value\": \"@item().PipelineId\",\n \"type\": \"Expression\"\n },\n \"type\": \"Int32\"\n },\n \"StageId\": {\n \"value\": {\n \"value\": \"@item().StageId\",\n \"type\": \"Expression\"\n },\n \"type\": \"Int32\"\n }\n }\n },\n \"linkedServiceName\": {\n \"referenceName\": \"SupportDatabase\",\n \"type\": \"LinkedServiceReference\"\n }\n }\n ]\n }\n },\n {\n \"name\": \"Execute Precursor\",\n \"description\": \"Uses the database property value ExecutionPrecursorProc to run any custom logic against the metadata database before the execution run starts.\",\n \"type\": \"SqlServerStoredProcedure\",\n \"dependsOn\": [],\n \"policy\": {\n \"timeout\": \"0.00:10:00\",\n \"retry\": 0,\n \"retryIntervalInSeconds\": 30,\n \"secureOutput\": false,\n \"secureInput\": false\n },\n \"userProperties\": [],\n \"typeProperties\": {\n \"storedProcedureName\": \"[[procfwk].[ExecutePrecursorProcedure]\"\n },\n \"linkedServiceName\": {\n \"referenceName\": \"SupportDatabase\",\n \"type\": \"LinkedServiceReference\"\n }\n },\n {\n \"name\": \"Set Execution Id\",\n \"description\": \"Set the local execution Id to a pipeline variable for each in several downstream activities.\",\n \"type\": \"SetVariable\",\n \"dependsOn\": [\n {\n \"activity\": \"Execution Wrapper\",\n \"dependencyConditions\": [\n \"Succeeded\"\n ]\n }\n ],\n \"userProperties\": [],\n \"typeProperties\": {\n \"variableName\": \"ExecutionId\",\n \"value\": {\n \"value\": \"@activity('Execution Wrapper').output.firstRow.ExecutionId\",\n \"type\": \"Expression\"\n }\n }\n }\n ],\n \"variables\": {\n \"ExecutionId\": {\n \"type\": \"String\"\n }\n },\n \"folder\": {\n \"name\": \"_ProcFwk\"\n },\n \"annotations\": [\n \"ADF.procfwk\",\n \"Parent\"\n ]\n },\n \"dependsOn\": [\n \"[concat(variables('factoryId'), '/datasets/GetSetMetadata')]\",\n \"[concat(variables('factoryId'), '/linkedServices/SupportDatabase')]\",\n \"[concat(variables('factoryId'), '/pipelines/03-Child')]\",\n \"[concat(variables('factoryId'), '/linkedServices/FrameworkFunctions')]\"\n ]\n },\n {\n \"name\": \"[concat(parameters('factoryName'), '/03-Child')]\",\n \"type\": \"Microsoft.DataFactory/factories/pipelines\",\n \"apiVersion\": \"2018-06-01\",\n \"properties\": {\n \"description\": \"ADF.procfwk child pipeline used to execute Worker pipelines within a given execution stage. This pipeline will be called once for each stage, then execute all Workers in parallel.\",\n \"activities\": [\n {\n \"name\": \"Get Pipelines\",\n \"description\": \"Returns all pipelines from the metadata to be executed within a given processing stage.\",\n \"type\": \"Lookup\",\n \"dependsOn\": [],\n \"policy\": {\n \"timeout\": \"0.00:10:00\",\n \"retry\": 0,\n \"retryIntervalInSeconds\": 30,\n \"secureOutput\": false,\n \"secureInput\": false\n },\n \"userProperties\": [],\n \"typeProperties\": {\n \"source\": {\n \"type\": \"AzureSqlSource\",\n \"sqlReaderStoredProcedureName\": \"[[procfwk].[GetPipelinesInStage]\",\n \"storedProcedureParameters\": {\n \"ExecutionId\": {\n \"type\": \"Guid\",\n \"value\": {\n \"value\": \"@pipeline().parameters.ExecutionId\",\n \"type\": \"Expression\"\n }\n },\n \"StageId\": {\n \"type\": \"Int32\",\n \"value\": {\n \"value\": \"@pipeline().parameters.StageId\",\n \"type\": \"Expression\"\n }\n }\n },\n \"queryTimeout\": \"02:00:00\",\n \"partitionOption\": \"None\"\n },\n \"dataset\": {\n \"referenceName\": \"GetSetMetadata\",\n \"type\": \"DatasetReference\",\n \"parameters\": {}\n },\n \"firstRowOnly\": false\n }\n },\n {\n \"name\": \"Execute Pipelines\",\n \"description\": \"Second level ForEach to run in parallel all pipelines within the stage. Items for iteration passed from the Get Pipelines lookup activity.\",\n \"type\": \"ForEach\",\n \"dependsOn\": [\n {\n \"activity\": \"Get Pipelines\",\n \"dependencyConditions\": [\n \"Succeeded\"\n ]\n }\n ],\n \"userProperties\": [],\n \"typeProperties\": {\n \"items\": {\n \"value\": \"@activity('Get Pipelines').output.value\",\n \"type\": \"Expression\"\n },\n \"isSequential\": false,\n \"batchCount\": 50,\n \"activities\": [\n {\n \"name\": \"Worker Pipeline Executor\",\n \"description\": \"Run the required worker pipeline and wait for its completion. Update metadata once done.\",\n \"type\": \"ExecutePipeline\",\n \"dependsOn\": [],\n \"userProperties\": [],\n \"typeProperties\": {\n \"pipeline\": {\n \"referenceName\": \"04-Infant\",\n \"type\": \"PipelineReference\"\n },\n \"waitOnCompletion\": true,\n \"parameters\": {\n \"executionId\": {\n \"value\": \"@pipeline().parameters.ExecutionId\",\n \"type\": \"Expression\"\n },\n \"stageId\": {\n \"value\": \"@pipeline().parameters.StageId\",\n \"type\": \"Expression\"\n },\n \"pipelineId\": {\n \"value\": \"@item().PipelineId\",\n \"type\": \"Expression\"\n }\n }\n }\n }\n ]\n }\n }\n ],\n \"parameters\": {\n \"StageId\": {\n \"type\": \"int\"\n },\n \"ExecutionId\": {\n \"type\": \"string\"\n }\n },\n \"folder\": {\n \"name\": \"_ProcFwk\"\n },\n \"annotations\": [\n \"ADF.procfwk\",\n \"Child\"\n ]\n },\n \"dependsOn\": [\n \"[concat(variables('factoryId'), '/datasets/GetSetMetadata')]\",\n \"[concat(variables('factoryId'), '/pipelines/04-Infant')]\"\n ]\n },\n {\n \"name\": \"[concat(parameters('factoryName'), '/04-Infant')]\",\n \"type\": \"Microsoft.DataFactory/factories/pipelines\",\n \"apiVersion\": \"2018-06-01\",\n \"properties\": {\n \"description\": \"ADF.procfwk infant pipeline used to check when the processing pipeline called by the Child completes and passes the resulting status back to the metadata database.\",\n \"activities\": [\n {\n \"name\": \"Execute Worker Pipeline\",\n \"description\": \"The lowest level executor with the metadata framework to call existing processing pipelines within Data Factory. The function called will block processing and wait for an outcome.\",\n \"type\": \"AzureFunctionActivity\",\n \"dependsOn\": [\n {\n \"activity\": \"Log Pipeline Running\",\n \"dependencyConditions\": [\n \"Succeeded\"\n ]\n },\n {\n \"activity\": \"Get Pipeline Params\",\n \"dependencyConditions\": [\n \"Succeeded\"\n ]\n },\n {\n \"activity\": \"Set App Id\",\n \"dependencyConditions\": [\n \"Succeeded\"\n ]\n },\n {\n \"activity\": \"Set App Secret\",\n \"dependencyConditions\": [\n \"Succeeded\"\n ]\n },\n {\n \"activity\": \"Set Subscription Id\",\n \"dependencyConditions\": [\n \"Succeeded\"\n ]\n },\n {\n \"activity\": \"Set Tenant Id\",\n \"dependencyConditions\": [\n \"Succeeded\"\n ]\n },\n {\n \"activity\": \"Set Pipeline Name\",\n \"dependencyConditions\": [\n \"Succeeded\"\n ]\n },\n {\n \"activity\": \"Set Data Factory Name\",\n \"dependencyConditions\": [\n \"Succeeded\"\n ]\n },\n {\n \"activity\": \"Set Resource Group\",\n \"dependencyConditions\": [\n \"Succeeded\"\n ]\n }\n ],\n \"policy\": {\n \"timeout\": \"0.00:10:00\",\n \"retry\": 0,\n \"retryIntervalInSeconds\": 30,\n \"secureOutput\": false,\n \"secureInput\": true\n },\n \"userProperties\": [],\n \"typeProperties\": {\n \"functionName\": \"ExecutePipeline\",\n \"method\": \"POST\",\n \"headers\": {},\n \"body\": {\n \"value\": \"@concat('\\n{\\n\\t\\\"tenantId\\\": \\\"',variables('WorkerTenantId'),'\\\",\\n\\t\\\"applicationId\\\": \\\"',variables('WorkerAppId'),'\\\",\\n\\t\\\"authenticationKey\\\": \\\"',variables('WorkerAppSecret'),'\\\",\\n\\t\\\"subscriptionId\\\": \\\"',variables('WorkerSubscriptionId'),'\\\",\\n\\t\\\"resourceGroup\\\": \\\"',variables('WorkerResourceGroup'),'\\\",\\n\\t\\\"factoryName\\\": \\\"',variables('WorkerDataFactoryName'),'\\\",\\n\\t\\\"pipelineName\\\": \\\"',variables('WorkerPipelineName'),'\\\"',activity('Get Pipeline Params').output.firstRow.Params,'\\n}')\",\n \"type\": \"Expression\"\n }\n },\n \"linkedServiceName\": {\n \"referenceName\": \"FrameworkFunctions\",\n \"type\": \"LinkedServiceReference\"\n }\n },\n {\n \"name\": \"Get Pipeline Params\",\n \"description\": \"Returns any parameters from metadata required for the processing pipeline being called. The output can be an empty string if no parameters are required.\",\n \"type\": \"Lookup\",\n \"dependsOn\": [],\n \"policy\": {\n \"timeout\": \"0.00:10:00\",\n \"retry\": 0,\n \"retryIntervalInSeconds\": 30,\n \"secureOutput\": false,\n \"secureInput\": false\n },\n \"userProperties\": [],\n \"typeProperties\": {\n \"source\": {\n \"type\": \"AzureSqlSource\",\n \"sqlReaderStoredProcedureName\": \"[[procfwk].[GetPipelineParameters]\",\n \"storedProcedureParameters\": {\n \"PipelineId\": {\n \"type\": \"Int32\",\n \"value\": {\n \"value\": \"@pipeline().parameters.pipelineId\",\n \"type\": \"Expression\"\n }\n }\n },\n \"queryTimeout\": \"02:00:00\",\n \"partitionOption\": \"None\"\n },\n \"dataset\": {\n \"referenceName\": \"GetSetMetadata\",\n \"type\": \"DatasetReference\",\n \"parameters\": {}\n }\n }\n },\n {\n \"name\": \"Log Pipeline Running\",\n \"description\": \"Sets the current pipeline with a status of running within the current execution database table.\",\n \"type\": \"SqlServerStoredProcedure\",\n \"dependsOn\": [],\n \"policy\": {\n \"timeout\": \"0.00:10:00\",\n \"retry\": 0,\n \"retryIntervalInSeconds\": 30,\n \"secureOutput\": false,\n \"secureInput\": false\n },\n \"userProperties\": [],\n \"typeProperties\": {\n \"storedProcedureName\": \"[[procfwk].[SetLogPipelineRunning]\",\n \"storedProcedureParameters\": {\n \"ExecutionId\": {\n \"value\": {\n \"value\": \"@pipeline().parameters.ExecutionId\",\n \"type\": \"Expression\"\n },\n \"type\": \"Guid\"\n },\n \"PipelineId\": {\n \"value\": {\n \"value\": \"@pipeline().parameters.pipelineId\",\n \"type\": \"Expression\"\n },\n \"type\": \"Int32\"\n },\n \"StageId\": {\n \"value\": {\n \"value\": \"@pipeline().parameters.StageId\",\n \"type\": \"Expression\"\n },\n \"type\": \"Int32\"\n }\n }\n },\n \"linkedServiceName\": {\n \"referenceName\": \"SupportDatabase\",\n \"type\": \"LinkedServiceReference\"\n }\n },\n {\n \"name\": \"Get Worker Authentication Details\",\n \"description\": \"Return the SPN ID and Secret for the worker pipeline being executed. Called at this level as each pipeline can have a different SPN.\",\n \"type\": \"Lookup\",\n \"dependsOn\": [],\n \"policy\": {\n \"timeout\": \"0.00:10:00\",\n \"retry\": 0,\n \"retryIntervalInSeconds\": 30,\n \"secureOutput\": true,\n \"secureInput\": false\n },\n \"userProperties\": [],\n \"typeProperties\": {\n \"source\": {\n \"type\": \"AzureSqlSource\",\n \"sqlReaderStoredProcedureName\": \"[[procfwk].[GetWorkerAuthDetails]\",\n \"storedProcedureParameters\": {\n \"ExecutionId\": {\n \"type\": \"Guid\",\n \"value\": {\n \"value\": \"@pipeline().parameters.executionId\",\n \"type\": \"Expression\"\n }\n },\n \"PipelineId\": {\n \"type\": \"Int32\",\n \"value\": {\n \"value\": \"@pipeline().parameters.pipelineId\",\n \"type\": \"Expression\"\n }\n },\n \"StageId\": {\n \"type\": \"Int32\",\n \"value\": {\n \"value\": \"@pipeline().parameters.stageId\",\n \"type\": \"Expression\"\n }\n }\n },\n \"queryTimeout\": \"02:00:00\",\n \"partitionOption\": \"None\"\n },\n \"dataset\": {\n \"referenceName\": \"GetSetMetadata\",\n \"type\": \"DatasetReference\",\n \"parameters\": {}\n }\n }\n },\n {\n \"name\": \"Log Execute Function Activity Failure\",\n \"description\": \"Handle true failures from calling out to the Azure Function and update the current execution table accordingly so a restart can occur.\",\n \"type\": \"SqlServerStoredProcedure\",\n \"dependsOn\": [\n {\n \"activity\": \"Execute Worker Pipeline\",\n \"dependencyConditions\": [\n \"Failed\"\n ]\n }\n ],\n \"policy\": {\n \"timeout\": \"0.00:10:00\",\n \"retry\": 0,\n \"retryIntervalInSeconds\": 30,\n \"secureOutput\": false,\n \"secureInput\": false\n },\n \"userProperties\": [],\n \"typeProperties\": {\n \"storedProcedureName\": \"[[procfwk].[SetLogActivityFailed]\",\n \"storedProcedureParameters\": {\n \"ExecutionId\": {\n \"value\": {\n \"value\": \"@pipeline().parameters.ExecutionId\",\n \"type\": \"Expression\"\n },\n \"type\": \"Guid\"\n },\n \"PipelineId\": {\n \"value\": {\n \"value\": \"@pipeline().parameters.pipelineId\",\n \"type\": \"Expression\"\n },\n \"type\": \"Int32\"\n },\n \"StageId\": {\n \"value\": {\n \"value\": \"@pipeline().parameters.StageId\",\n \"type\": \"Expression\"\n },\n \"type\": \"Int32\"\n },\n \"CallingActivity\": {\n \"value\": \"ExecuteWorkerPipeline\",\n \"type\": \"String\"\n }\n }\n },\n \"linkedServiceName\": {\n \"referenceName\": \"SupportDatabase\",\n \"type\": \"LinkedServiceReference\"\n }\n },\n {\n \"name\": \"Update Run Id\",\n \"description\": \"Provide the actual ADF run ID back to the current execution table for long term logging and alignment between the metadata other Azure monitoring tools.\",\n \"type\": \"SqlServerStoredProcedure\",\n \"dependsOn\": [\n {\n \"activity\": \"Set Run Id\",\n \"dependencyConditions\": [\n \"Succeeded\"\n ]\n }\n ],\n \"policy\": {\n \"timeout\": \"0.00:10:00\",\n \"retry\": 0,\n \"retryIntervalInSeconds\": 30,\n \"secureOutput\": false,\n \"secureInput\": false\n },\n \"userProperties\": [],\n \"typeProperties\": {\n \"storedProcedureName\": \"[[procfwk].[SetLogPipelineRunId]\",\n \"storedProcedureParameters\": {\n \"ExecutionId\": {\n \"value\": {\n \"value\": \"@pipeline().parameters.ExecutionId\",\n \"type\": \"Expression\"\n },\n \"type\": \"Guid\"\n },\n \"PipelineId\": {\n \"value\": {\n \"value\": \"@pipeline().parameters.pipelineId\",\n \"type\": \"Expression\"\n },\n \"type\": \"Int32\"\n },\n \"RunId\": {\n \"value\": {\n \"value\": \"@variables('WorkerRunId')\",\n \"type\": \"Expression\"\n },\n \"type\": \"Guid\"\n },\n \"StageId\": {\n \"value\": {\n \"value\": \"@pipeline().parameters.StageId\",\n \"type\": \"Expression\"\n },\n \"type\": \"Int32\"\n }\n }\n },\n \"linkedServiceName\": {\n \"referenceName\": \"SupportDatabase\",\n \"type\": \"LinkedServiceReference\"\n }\n },\n {\n \"name\": \"Check For Alerts\",\n \"description\": \"Checks the properties tables and if any recipients in the database require alerts sending for the current pipeline ID.\",\n \"type\": \"Lookup\",\n \"dependsOn\": [\n {\n \"activity\": \"Update Run Id\",\n \"dependencyConditions\": [\n \"Succeeded\"\n ]\n },\n {\n \"activity\": \"Set Pipeline Result\",\n \"dependencyConditions\": [\n \"Completed\"\n ]\n }\n ],\n \"policy\": {\n \"timeout\": \"0.00:10:00\",\n \"retry\": 0,\n \"retryIntervalInSeconds\": 30,\n \"secureOutput\": false,\n \"secureInput\": false\n },\n \"userProperties\": [],\n \"typeProperties\": {\n \"source\": {\n \"type\": \"AzureSqlSource\",\n \"sqlReaderStoredProcedureName\": \"[[procfwk].[CheckForEmailAlerts]\",\n \"storedProcedureParameters\": {\n \"PipelineId\": {\n \"type\": \"Int32\",\n \"value\": {\n \"value\": \"@pipeline().parameters.pipelineId\",\n \"type\": \"Expression\"\n }\n }\n },\n \"queryTimeout\": \"02:00:00\",\n \"partitionOption\": \"None\"\n },\n \"dataset\": {\n \"referenceName\": \"GetSetMetadata\",\n \"type\": \"DatasetReference\",\n \"parameters\": {}\n },\n \"firstRowOnly\": true\n }\n },\n {\n \"name\": \"Send Alerts\",\n \"description\": \"True = alerts need sending.\\nFalse = do nothing.\",\n \"type\": \"IfCondition\",\n \"dependsOn\": [\n {\n \"activity\": \"Check For Alerts\",\n \"dependencyConditions\": [\n \"Succeeded\"\n ]\n }\n ],\n \"userProperties\": [],\n \"typeProperties\": {\n \"expression\": {\n \"value\": \"@activity('Check For Alerts').output.firstRow.SendAlerts\",\n \"type\": \"Expression\"\n },\n \"ifTrueActivities\": [\n {\n \"name\": \"Get Email Parts\",\n \"type\": \"Lookup\",\n \"dependsOn\": [],\n \"policy\": {\n \"timeout\": \"0.00:10:00\",\n \"retry\": 0,\n \"retryIntervalInSeconds\": 30,\n \"secureOutput\": true,\n \"secureInput\": false\n },\n \"userProperties\": [],\n \"typeProperties\": {\n \"source\": {\n \"type\": \"AzureSqlSource\",\n \"sqlReaderStoredProcedureName\": \"[[procfwk].[GetEmailAlertParts]\",\n \"storedProcedureParameters\": {\n \"PipelineId\": {\n \"type\": \"Int32\",\n \"value\": {\n \"value\": \"@pipeline().parameters.pipelineId\",\n \"type\": \"Expression\"\n }\n }\n },\n \"queryTimeout\": \"02:00:00\",\n \"partitionOption\": \"None\"\n },\n \"dataset\": {\n \"referenceName\": \"GetSetMetadata\",\n \"type\": \"DatasetReference\",\n \"parameters\": {}\n },\n \"firstRowOnly\": true\n }\n },\n {\n \"name\": \"Send Email\",\n \"type\": \"AzureFunctionActivity\",\n \"dependsOn\": [\n {\n \"activity\": \"Get Email Parts\",\n \"dependencyConditions\": [\n \"Succeeded\"\n ]\n }\n ],\n \"policy\": {\n \"timeout\": \"0.00:10:00\",\n \"retry\": 0,\n \"retryIntervalInSeconds\": 30,\n \"secureOutput\": false,\n \"secureInput\": true\n },\n \"userProperties\": [],\n \"typeProperties\": {\n \"functionName\": \"SendEmail\",\n \"method\": \"POST\",\n \"headers\": {},\n \"body\": {\n \"value\": \"@activity('Get Email Parts').output.firstRow\",\n \"type\": \"Expression\"\n }\n },\n \"linkedServiceName\": {\n \"referenceName\": \"FrameworkFunctions\",\n \"type\": \"LinkedServiceReference\"\n }\n }\n ]\n }\n },\n {\n \"name\": \"Wait Until Pipeline Completes\",\n \"description\": \"Loops until the Worker pipeline called completes.\\n\\nSimple status:\\n- Running = new iteration.\\n- Done = break.\",\n \"type\": \"Until\",\n \"dependsOn\": [\n {\n \"activity\": \"Get Wait Duration\",\n \"dependencyConditions\": [\n \"Succeeded\"\n ]\n },\n {\n \"activity\": \"Execute Worker Pipeline\",\n \"dependencyConditions\": [\n \"Succeeded\"\n ]\n },\n {\n \"activity\": \"Set Run Id\",\n \"dependencyConditions\": [\n \"Succeeded\"\n ]\n }\n ],\n \"userProperties\": [],\n \"typeProperties\": {\n \"expression\": {\n \"value\": \"@variables('WorkerPipelineState')\",\n \"type\": \"Expression\"\n },\n \"activities\": [\n {\n \"name\": \"Get Worker Pipeline Status\",\n \"description\": \"Checks the status of a given processing pipeline and provides the value for the downstream framework activities to act upon.\",\n \"type\": \"AzureFunctionActivity\",\n \"dependsOn\": [],\n \"policy\": {\n \"timeout\": \"0.00:10:00\",\n \"retry\": 0,\n \"retryIntervalInSeconds\": 30,\n \"secureOutput\": false,\n \"secureInput\": true\n },\n \"userProperties\": [],\n \"typeProperties\": {\n \"functionName\": \"CheckPipelineStatus\",\n \"method\": \"POST\",\n \"headers\": {},\n \"body\": {\n \"value\": \"@concat('\\n{\\n \\\"tenantId\\\": \\\"',variables('WorkerTenantId'),'\\\",\\n \\\"applicationId\\\": \\\"',variables('WorkerAppId'),'\\\",\\n \\\"authenticationKey\\\": \\\"',variables('WorkerAppSecret'),'\\\",\\n \\\"subscriptionId\\\": \\\"',variables('WorkerSubscriptionId'),'\\\",\\n \\\"resourceGroup\\\": \\\"',variables('WorkerResourceGroup'),'\\\",\\n \\\"factoryName\\\": \\\"',variables('WorkerDataFactoryName'),'\\\",\\n \\\"pipelineName\\\": \\\"',variables('WorkerPipelineName'),'\\\",\\n \\\"runId\\\": \\\"',variables('WorkerRunId'),'\\\"\\n}')\",\n \"type\": \"Expression\"\n }\n },\n \"linkedServiceName\": {\n \"referenceName\": \"FrameworkFunctions\",\n \"type\": \"LinkedServiceReference\"\n }\n },\n {\n \"name\": \"Wait If Running\",\n \"description\": \"True = Do nothing.\\nFalse = Wait, before the next iteration.\",\n \"type\": \"IfCondition\",\n \"dependsOn\": [\n {\n \"activity\": \"Set Worker State\",\n \"dependencyConditions\": [\n \"Succeeded\"\n ]\n }\n ],\n \"userProperties\": [],\n \"typeProperties\": {\n \"expression\": {\n \"value\": \"@variables('WorkerPipelineState')\",\n \"type\": \"Expression\"\n },\n \"ifFalseActivities\": [\n {\n \"name\": \"Wait for Pipeline\",\n \"description\": \"The processing pipeline is still running so Wait before checking its status again.\",\n \"type\": \"Wait\",\n \"dependsOn\": [],\n \"userProperties\": [],\n \"typeProperties\": {\n \"waitTimeInSeconds\": {\n \"value\": \"@activity('Get Wait Duration').output.firstRow.PropertyValue\",\n \"type\": \"Expression\"\n }\n }\n }\n ]\n }\n },\n {\n \"name\": \"Set Last Check DateTime\",\n \"description\": \"Update the current execution table with a date time from when the Worker pipeline status was last checked as part of the Until iterations.\",\n \"type\": \"SqlServerStoredProcedure\",\n \"dependsOn\": [\n {\n \"activity\": \"Get Worker Pipeline Status\",\n \"dependencyConditions\": [\n \"Succeeded\"\n ]\n }\n ],\n \"policy\": {\n \"timeout\": \"0.00:10:00\",\n \"retry\": 0,\n \"retryIntervalInSeconds\": 30,\n \"secureOutput\": false,\n \"secureInput\": false\n },\n \"userProperties\": [],\n \"typeProperties\": {\n \"storedProcedureName\": \"[[procfwk].[SetLogPipelineLastStatusCheck]\",\n \"storedProcedureParameters\": {\n \"ExecutionId\": {\n \"value\": {\n \"value\": \"@pipeline().parameters.executionId\",\n \"type\": \"Expression\"\n },\n \"type\": \"Guid\"\n },\n \"PipelineId\": {\n \"value\": {\n \"value\": \"@pipeline().parameters.pipelineId\",\n \"type\": \"Expression\"\n },\n \"type\": \"Int32\"\n },\n \"StageId\": {\n \"value\": {\n \"value\": \"@pipeline().parameters.stageId\",\n \"type\": \"Expression\"\n },\n \"type\": \"Int32\"\n }\n }\n },\n \"linkedServiceName\": {\n \"referenceName\": \"SupportDatabase\",\n \"type\": \"LinkedServiceReference\"\n }\n },\n {\n \"name\": \"Log Check Function Activity Failure\",\n \"description\": \"Report to the current execution table that the framework pipeline activity has failed. This failure is outside of the scope of the framework and is probably related to a wider platform problem.\",\n \"type\": \"SqlServerStoredProcedure\",\n \"dependsOn\": [\n {\n \"activity\": \"Get Worker Pipeline Status\",\n \"dependencyConditions\": [\n \"Failed\"\n ]\n }\n ],\n \"policy\": {\n \"timeout\": \"0.00:10:00\",\n \"retry\": 0,\n \"retryIntervalInSeconds\": 30,\n \"secureOutput\": false,\n \"secureInput\": false\n },\n \"userProperties\": [],\n \"typeProperties\": {\n \"storedProcedureName\": \"[[procfwk].[SetLogActivityFailed]\",\n \"storedProcedureParameters\": {\n \"CallingActivity\": {\n \"value\": \"GetWorkerPipelineStatus\",\n \"type\": \"String\"\n },\n \"ExecutionId\": {\n \"value\": {\n \"value\": \"@pipeline().parameters.executionId\",\n \"type\": \"Expression\"\n },\n \"type\": \"Guid\"\n },\n \"PipelineId\": {\n \"value\": {\n \"value\": \"@pipeline().parameters.pipelineId\",\n \"type\": \"Expression\"\n },\n \"type\": \"Int32\"\n },\n \"StageId\": {\n \"value\": {\n \"value\": \"@pipeline().parameters.stageId\",\n \"type\": \"Expression\"\n },\n \"type\": \"Int32\"\n }\n }\n },\n \"linkedServiceName\": {\n \"referenceName\": \"SupportDatabase\",\n \"type\": \"LinkedServiceReference\"\n }\n },\n {\n \"name\": \"Set Worker State\",\n \"description\": \"Set the bool state of the Worker pipeline to be used by the Until and If expressions. True = Complete, False = Running.\",\n \"type\": \"SetVariable\",\n \"dependsOn\": [\n {\n \"activity\": \"Get Worker Pipeline Status\",\n \"dependencyConditions\": [\n \"Succeeded\"\n ]\n }\n ],\n \"userProperties\": [],\n \"typeProperties\": {\n \"variableName\": \"WorkerPipelineState\",\n \"value\": {\n \"value\": \"@equals('Done',activity('Get Worker Pipeline Status').output.SimpleStatus)\",\n \"type\": \"Expression\"\n }\n }\n }\n ],\n \"timeout\": \"0.00:10:00\"\n }\n },\n {\n \"name\": \"Set Pipeline Result\",\n \"description\": \"Receives the outcome from the function execution for a given processing pipeline and updates the current execution table with different pipelines status values depending on the result (case).\",\n \"type\": \"Switch\",\n \"dependsOn\": [\n {\n \"activity\": \"Wait Until Pipeline Completes\",\n \"dependencyConditions\": [\n \"Completed\"\n ]\n }\n ],\n \"userProperties\": [],\n \"typeProperties\": {\n \"on\": {\n \"value\": \"@activity('Get Worker Pipeline Status').output.Status\",\n \"type\": \"Expression\"\n },\n \"cases\": [\n {\n \"value\": \"Succeeded\",\n \"activities\": [\n {\n \"name\": \"Pipeline Status Succeeded\",\n \"description\": \"Updates the current execution table with a pipeline status of success if the function outcome is succeeded.\",\n \"type\": \"SqlServerStoredProcedure\",\n \"dependsOn\": [],\n \"policy\": {\n \"timeout\": \"0.00:10:00\",\n \"retry\": 0,\n \"retryIntervalInSeconds\": 30,\n \"secureOutput\": false,\n \"secureInput\": false\n },\n \"userProperties\": [],\n \"typeProperties\": {\n \"storedProcedureName\": \"[[procfwk].[SetLogPipelineSuccess]\",\n \"storedProcedureParameters\": {\n \"ExecutionId\": {\n \"value\": {\n \"value\": \"@pipeline().parameters.executionId\",\n \"type\": \"Expression\"\n },\n \"type\": \"Guid\"\n },\n \"PipelineId\": {\n \"value\": {\n \"value\": \"@pipeline().parameters.pipelineId\",\n \"type\": \"Expression\"\n },\n \"type\": \"Int32\"\n },\n \"StageId\": {\n \"value\": {\n \"value\": \"@pipeline().parameters.stageId\",\n \"type\": \"Expression\"\n },\n \"type\": \"Int32\"\n }\n }\n },\n \"linkedServiceName\": {\n \"referenceName\": \"SupportDatabase\",\n \"type\": \"LinkedServiceReference\"\n }\n }\n ]\n },\n {\n \"value\": \"Failed\",\n \"activities\": [\n {\n \"name\": \"Pipeline Status Failed\",\n \"description\": \"Updates the current execution table with a pipeline status of failed if the function outcome is failed. Also blocks pipelines in the downstream execution stage.\",\n \"type\": \"SqlServerStoredProcedure\",\n \"dependsOn\": [],\n \"policy\": {\n \"timeout\": \"0.00:10:00\",\n \"retry\": 0,\n \"retryIntervalInSeconds\": 30,\n \"secureOutput\": false,\n \"secureInput\": false\n },\n \"userProperties\": [],\n \"typeProperties\": {\n \"storedProcedureName\": \"[[procfwk].[SetLogPipelineFailed]\",\n \"storedProcedureParameters\": {\n \"ExecutionId\": {\n \"value\": {\n \"value\": \"@pipeline().parameters.executionId\",\n \"type\": \"Expression\"\n },\n \"type\": \"Guid\"\n },\n \"PipelineId\": {\n \"value\": {\n \"value\": \"@pipeline().parameters.pipelineId\",\n \"type\": \"Expression\"\n },\n \"type\": \"Int32\"\n },\n \"RunId\": {\n \"value\": {\n \"value\": \"@variables('WorkerRunId')\",\n \"type\": \"Expression\"\n },\n \"type\": \"Guid\"\n },\n \"StageId\": {\n \"value\": {\n \"value\": \"@pipeline().parameters.stageId\",\n \"type\": \"Expression\"\n },\n \"type\": \"Int32\"\n }\n }\n },\n \"linkedServiceName\": {\n \"referenceName\": \"SupportDatabase\",\n \"type\": \"LinkedServiceReference\"\n }\n },\n {\n \"name\": \"Get Worker Pipeline Error Details\",\n \"description\": \"Get the activity error details for the run ID of the worker pipeline called. Returns an array of all errors.\",\n \"type\": \"AzureFunctionActivity\",\n \"dependsOn\": [],\n \"policy\": {\n \"timeout\": \"0.00:10:00\",\n \"retry\": 0,\n \"retryIntervalInSeconds\": 30,\n \"secureOutput\": false,\n \"secureInput\": true\n },\n \"userProperties\": [],\n \"typeProperties\": {\n \"functionName\": \"GetActivityErrors\",\n \"method\": \"POST\",\n \"headers\": {},\n \"body\": {\n \"value\": \"@concat('\\n{\\n \\\"tenantId\\\": \\\"',variables('WorkerTenantId'),'\\\",\\n \\\"applicationId\\\": \\\"',variables('WorkerAppId'),'\\\",\\n \\\"authenticationKey\\\": \\\"',variables('WorkerAppSecret'),'\\\",\\n \\\"subscriptionId\\\": \\\"',variables('WorkerSubscriptionId'),'\\\",\\n \\\"resourceGroup\\\": \\\"',variables('WorkerResourceGroup'),'\\\",\\n \\\"factoryName\\\": \\\"',variables('WorkerDataFactoryName'),'\\\",\\n \\\"pipelineName\\\": \\\"',variables('WorkerPipelineName'),'\\\",\\n \\\"runId\\\": \\\"',variables('WorkerRunId'),'\\\"\\n}')\",\n \"type\": \"Expression\"\n }\n },\n \"linkedServiceName\": {\n \"referenceName\": \"FrameworkFunctions\",\n \"type\": \"LinkedServiceReference\"\n }\n },\n {\n \"name\": \"Log Error Details\",\n \"description\": \"Parses pipeline error details and persists them to the metadata database error log table.\",\n \"type\": \"SqlServerStoredProcedure\",\n \"dependsOn\": [\n {\n \"activity\": \"Get Worker Pipeline Error Details\",\n \"dependencyConditions\": [\n \"Succeeded\"\n ]\n }\n ],\n \"policy\": {\n \"timeout\": \"0.00:10:00\",\n \"retry\": 0,\n \"retryIntervalInSeconds\": 30,\n \"secureOutput\": false,\n \"secureInput\": false\n },\n \"userProperties\": [],\n \"typeProperties\": {\n \"storedProcedureName\": \"[[procfwk].[SetErrorLogDetails]\",\n \"storedProcedureParameters\": {\n \"JsonErrorDetails\": {\n \"value\": {\n \"value\": \"@string(activity('Get Worker Pipeline Error Details').output)\",\n \"type\": \"Expression\"\n },\n \"type\": \"String\"\n },\n \"LocalExecutionId\": {\n \"value\": {\n \"value\": \"@pipeline().parameters.executionId\",\n \"type\": \"Expression\"\n },\n \"type\": \"Guid\"\n }\n }\n },\n \"linkedServiceName\": {\n \"referenceName\": \"SupportDatabase\",\n \"type\": \"LinkedServiceReference\"\n }\n }\n ]\n },\n {\n \"value\": \"Cancelled\",\n \"activities\": [\n {\n \"name\": \"Pipeline Status Cancelled\",\n \"description\": \"Updates the current execution table with a pipeline status of cancelled if the function outcome is cancelled.\",\n \"type\": \"SqlServerStoredProcedure\",\n \"dependsOn\": [],\n \"policy\": {\n \"timeout\": \"0.00:10:00\",\n \"retry\": 0,\n \"retryIntervalInSeconds\": 30,\n \"secureOutput\": false,\n \"secureInput\": false\n },\n \"userProperties\": [],\n \"typeProperties\": {\n \"storedProcedureName\": \"[[procfwk].[SetLogPipelineCancelled]\",\n \"storedProcedureParameters\": {\n \"ExecutionId\": {\n \"value\": {\n \"value\": \"@pipeline().parameters.executionId\",\n \"type\": \"Expression\"\n },\n \"type\": \"Guid\"\n },\n \"PipelineId\": {\n \"value\": {\n \"value\": \"@pipeline().parameters.pipelineId\",\n \"type\": \"Expression\"\n },\n \"type\": \"Int32\"\n },\n \"StageId\": {\n \"value\": {\n \"value\": \"@pipeline().parameters.stageId\",\n \"type\": \"Expression\"\n },\n \"type\": \"Int32\"\n }\n }\n },\n \"linkedServiceName\": {\n \"referenceName\": \"SupportDatabase\",\n \"type\": \"LinkedServiceReference\"\n }\n }\n ]\n }\n ],\n \"defaultActivities\": [\n {\n \"name\": \"Pipeline Status Unknown\",\n \"description\": \"Updates the current execution table with a pipeline status of unknown if the function returns an unexpected outcome.\",\n \"type\": \"SqlServerStoredProcedure\",\n \"dependsOn\": [],\n \"policy\": {\n \"timeout\": \"0.00:10:00\",\n \"retry\": 0,\n \"retryIntervalInSeconds\": 30,\n \"secureOutput\": false,\n \"secureInput\": false\n },\n \"userProperties\": [],\n \"typeProperties\": {\n \"storedProcedureName\": \"[[procfwk].[SetLogPipelineUnknown]\",\n \"storedProcedureParameters\": {\n \"ExecutionId\": {\n \"value\": {\n \"value\": \"@pipeline().parameters.executionId\",\n \"type\": \"Expression\"\n },\n \"type\": \"Guid\"\n },\n \"PipelineId\": {\n \"value\": {\n \"value\": \"@pipeline().parameters.pipelineId\",\n \"type\": \"Expression\"\n },\n \"type\": \"Int32\"\n },\n \"StageId\": {\n \"value\": {\n \"value\": \"@pipeline().parameters.stageId\",\n \"type\": \"Expression\"\n },\n \"type\": \"Int32\"\n }\n }\n },\n \"linkedServiceName\": {\n \"referenceName\": \"SupportDatabase\",\n \"type\": \"LinkedServiceReference\"\n }\n }\n ]\n }\n },\n {\n \"name\": \"Get Wait Duration\",\n \"description\": \"Return wait duration in seconds from database properties table to be used during each Until iteration when the Worker pipeline is still running.\",\n \"type\": \"Lookup\",\n \"dependsOn\": [],\n \"policy\": {\n \"timeout\": \"0.00:10:00\",\n \"retry\": 0,\n \"retryIntervalInSeconds\": 30,\n \"secureOutput\": false,\n \"secureInput\": false\n },\n \"userProperties\": [],\n \"typeProperties\": {\n \"source\": {\n \"type\": \"AzureSqlSource\",\n \"sqlReaderStoredProcedureName\": \"[[procfwk].[GetPropertyValue]\",\n \"storedProcedureParameters\": {\n \"PropertyName\": {\n \"type\": \"String\",\n \"value\": \"PipelineStatusCheckDuration\"\n }\n },\n \"queryTimeout\": \"02:00:00\",\n \"partitionOption\": \"None\"\n },\n \"dataset\": {\n \"referenceName\": \"GetSetMetadata\",\n \"type\": \"DatasetReference\",\n \"parameters\": {}\n }\n }\n },\n {\n \"name\": \"Set App Id\",\n \"description\": \"Set local variable from activity output once for value reuse in downstream activities.\",\n \"type\": \"SetVariable\",\n \"dependsOn\": [\n {\n \"activity\": \"Get Worker Authentication Details\",\n \"dependencyConditions\": [\n \"Succeeded\"\n ]\n }\n ],\n \"userProperties\": [],\n \"typeProperties\": {\n \"variableName\": \"WorkerAppId\",\n \"value\": {\n \"value\": \"@activity('Get Worker Authentication Details').output.firstRow.AppId\",\n \"type\": \"Expression\"\n }\n }\n },\n {\n \"name\": \"Set App Secret\",\n \"description\": \"Set local variable from activity output once for value reuse in downstream activities.\",\n \"type\": \"SetVariable\",\n \"dependsOn\": [\n {\n \"activity\": \"Get Worker Authentication Details\",\n \"dependencyConditions\": [\n \"Succeeded\"\n ]\n }\n ],\n \"userProperties\": [],\n \"typeProperties\": {\n \"variableName\": \"WorkerAppSecret\",\n \"value\": {\n \"value\": \"@activity('Get Worker Authentication Details').output.firstRow.AppSecret\",\n \"type\": \"Expression\"\n }\n }\n },\n {\n \"name\": \"Set Run Id\",\n \"description\": \"Set local variable from activity output once for value reuse in downstream activities.\",\n \"type\": \"SetVariable\",\n \"dependsOn\": [\n {\n \"activity\": \"Execute Worker Pipeline\",\n \"dependencyConditions\": [\n \"Succeeded\"\n ]\n }\n ],\n \"userProperties\": [],\n \"typeProperties\": {\n \"variableName\": \"WorkerRunId\",\n \"value\": {\n \"value\": \"@activity('Execute Worker Pipeline').output.RunId\",\n \"type\": \"Expression\"\n }\n }\n },\n {\n \"name\": \"Get Worker Pipeline Details\",\n \"description\": \"Return none sensitive worker pipeline information for metadata database. Including target data factory, pipeline name and resource group.\",\n \"type\": \"Lookup\",\n \"dependsOn\": [],\n \"policy\": {\n \"timeout\": \"0.00:10:00\",\n \"retry\": 0,\n \"retryIntervalInSeconds\": 30,\n \"secureOutput\": true,\n \"secureInput\": false\n },\n \"userProperties\": [],\n \"typeProperties\": {\n \"source\": {\n \"type\": \"AzureSqlSource\",\n \"sqlReaderStoredProcedureName\": \"[[procfwk].[GetWorkerPipelineDetails]\",\n \"storedProcedureParameters\": {\n \"ExecutionId\": {\n \"type\": \"Guid\",\n \"value\": {\n \"value\": \"@pipeline().parameters.ExecutionId\",\n \"type\": \"Expression\"\n }\n },\n \"PipelineId\": {\n \"type\": \"Int32\",\n \"value\": {\n \"value\": \"@pipeline().parameters.pipelineId\",\n \"type\": \"Expression\"\n }\n },\n \"StageId\": {\n \"type\": \"Int32\",\n \"value\": {\n \"value\": \"@pipeline().parameters.stageId\",\n \"type\": \"Expression\"\n }\n }\n },\n \"queryTimeout\": \"02:00:00\",\n \"partitionOption\": \"None\"\n },\n \"dataset\": {\n \"referenceName\": \"GetSetMetadata\",\n \"type\": \"DatasetReference\",\n \"parameters\": {}\n }\n }\n },\n {\n \"name\": \"Set Tenant Id\",\n \"description\": \"Set local variable from activity output once for value reuse in downstream activities.\",\n \"type\": \"SetVariable\",\n \"dependsOn\": [\n {\n \"activity\": \"Get Worker Authentication Details\",\n \"dependencyConditions\": [\n \"Succeeded\"\n ]\n }\n ],\n \"userProperties\": [],\n \"typeProperties\": {\n \"variableName\": \"WorkerTenantId\",\n \"value\": {\n \"value\": \"@activity('Get Worker Authentication Details').output.firstRow.TenantId\",\n \"type\": \"Expression\"\n }\n }\n },\n {\n \"name\": \"Set Subscription Id\",\n \"description\": \"Set local variable from activity output once for value reuse in downstream activities.\",\n \"type\": \"SetVariable\",\n \"dependsOn\": [\n {\n \"activity\": \"Get Worker Authentication Details\",\n \"dependencyConditions\": [\n \"Succeeded\"\n ]\n }\n ],\n \"userProperties\": [],\n \"typeProperties\": {\n \"variableName\": \"WorkerSubscriptionId\",\n \"value\": {\n \"value\": \"@activity('Get Worker Authentication Details').output.firstRow.SubscriptionId\",\n \"type\": \"Expression\"\n }\n }\n },\n {\n \"name\": \"Set Pipeline Name\",\n \"description\": \"Set local variable from activity output once for value reuse in downstream activities.\",\n \"type\": \"SetVariable\",\n \"dependsOn\": [\n {\n \"activity\": \"Get Worker Pipeline Details\",\n \"dependencyConditions\": [\n \"Succeeded\"\n ]\n }\n ],\n \"userProperties\": [],\n \"typeProperties\": {\n \"variableName\": \"WorkerPipelineName\",\n \"value\": {\n \"value\": \"@activity('Get Worker Pipeline Details').output.firstRow.PipelineName\",\n \"type\": \"Expression\"\n }\n }\n },\n {\n \"name\": \"Set Data Factory Name\",\n \"description\": \"Set local variable from activity output once for value reuse in downstream activities.\",\n \"type\": \"SetVariable\",\n \"dependsOn\": [\n {\n \"activity\": \"Get Worker Pipeline Details\",\n \"dependencyConditions\": [\n \"Succeeded\"\n ]\n }\n ],\n \"userProperties\": [],\n \"typeProperties\": {\n \"variableName\": \"WorkerDataFactoryName\",\n \"value\": {\n \"value\": \"@activity('Get Worker Pipeline Details').output.firstRow.DataFactoryName\",\n \"type\": \"Expression\"\n }\n }\n },\n {\n \"name\": \"Set Resource Group\",\n \"description\": \"Set local variable from activity output once for value reuse in downstream activities.\",\n \"type\": \"SetVariable\",\n \"dependsOn\": [\n {\n \"activity\": \"Get Worker Pipeline Details\",\n \"dependencyConditions\": [\n \"Succeeded\"\n ]\n }\n ],\n \"userProperties\": [],\n \"typeProperties\": {\n \"variableName\": \"WorkerResourceGroup\",\n \"value\": {\n \"value\": \"@activity('Get Worker Pipeline Details').output.firstRow.ResourceGroupName\",\n \"type\": \"Expression\"\n }\n }\n }\n ],\n \"parameters\": {\n \"executionId\": {\n \"type\": \"string\"\n },\n \"stageId\": {\n \"type\": \"int\"\n },\n \"pipelineId\": {\n \"type\": \"int\"\n }\n },\n \"variables\": {\n \"WorkerPipelineState\": {\n \"type\": \"Boolean\"\n },\n \"WorkerAppId\": {\n \"type\": \"String\"\n },\n \"WorkerAppSecret\": {\n \"type\": \"String\"\n },\n \"WorkerRunId\": {\n \"type\": \"String\"\n },\n \"WorkerTenantId\": {\n \"type\": \"String\"\n },\n \"WorkerSubscriptionId\": {\n \"type\": \"String\"\n },\n \"WorkerPipelineName\": {\n \"type\": \"String\"\n },\n \"WorkerDataFactoryName\": {\n \"type\": \"String\"\n },\n \"WorkerResourceGroup\": {\n \"type\": \"String\"\n }\n },\n \"folder\": {\n \"name\": \"_ProcFwk\"\n },\n \"annotations\": [\n \"ADF.procfwk\",\n \"Infant\"\n ]\n },\n \"dependsOn\": [\n \"[concat(variables('factoryId'), '/linkedServices/FrameworkFunctions')]\",\n \"[concat(variables('factoryId'), '/datasets/GetSetMetadata')]\",\n \"[concat(variables('factoryId'), '/linkedServices/SupportDatabase')]\"\n ]\n },\n {\n \"name\": \"[concat(parameters('factoryName'), '/Intentional Error')]\",\n \"type\": \"Microsoft.DataFactory/factories/pipelines\",\n \"apiVersion\": \"2018-06-01\",\n \"properties\": {\n \"description\": \"Used just so the ADF.procfwk has something to call during development.\",\n \"activities\": [\n {\n \"name\": \"Wait1\",\n \"description\": \"Framework development worker simulator.\",\n \"type\": \"Wait\",\n \"dependsOn\": [],\n \"userProperties\": [],\n \"typeProperties\": {\n \"waitTimeInSeconds\": {\n \"value\": \"@pipeline().parameters.WaitTime\",\n \"type\": \"Expression\"\n }\n }\n },\n {\n \"name\": \"Raise Errors or Not\",\n \"description\": \"Framework development worker simulator.\",\n \"type\": \"IfCondition\",\n \"dependsOn\": [\n {\n \"activity\": \"Wait1\",\n \"dependencyConditions\": [\n \"Succeeded\"\n ]\n }\n ],\n \"userProperties\": [],\n \"typeProperties\": {\n \"expression\": {\n \"value\": \"@equals(pipeline().parameters.RaiseErrors,'true')\",\n \"type\": \"Expression\"\n },\n \"ifTrueActivities\": [\n {\n \"name\": \"Call Fail Procedure\",\n \"type\": \"SqlServerStoredProcedure\",\n \"dependsOn\": [],\n \"policy\": {\n \"timeout\": \"0.00:10:00\",\n \"retry\": 0,\n \"retryIntervalInSeconds\": 30,\n \"secureOutput\": false,\n \"secureInput\": false\n },\n \"userProperties\": [],\n \"typeProperties\": {\n \"storedProcedureName\": \"[[dbo].[FailProcedure]\",\n \"storedProcedureParameters\": {\n \"RaiseError\": {\n \"value\": {\n \"value\": \"@pipeline().parameters.RaiseErrors\",\n \"type\": \"Expression\"\n },\n \"type\": \"String\"\n }\n }\n },\n \"linkedServiceName\": {\n \"referenceName\": \"SupportDatabase\",\n \"type\": \"LinkedServiceReference\"\n }\n }\n ]\n }\n }\n ],\n \"parameters\": {\n \"RaiseErrors\": {\n \"type\": \"string\",\n \"defaultValue\": \"false\"\n },\n \"WaitTime\": {\n \"type\": \"int\",\n \"defaultValue\": 5\n }\n },\n \"folder\": {\n \"name\": \"_Workers\"\n },\n \"annotations\": [\n \"Worker\"\n ]\n },\n \"dependsOn\": [\n \"[concat(variables('factoryId'), '/linkedServices/SupportDatabase')]\"\n ]\n },\n {\n \"name\": \"[concat(parameters('factoryName'), '/Wait 1')]\",\n \"type\": \"Microsoft.DataFactory/factories/pipelines\",\n \"apiVersion\": \"2018-06-01\",\n \"properties\": {\n \"description\": \"Used just so the ADF.procfwk has something to call during development.\",\n \"activities\": [\n {\n \"name\": \"Wait1\",\n \"description\": \"Framework development worker simulator.\",\n \"type\": \"Wait\",\n \"dependsOn\": [],\n \"userProperties\": [],\n \"typeProperties\": {\n \"waitTimeInSeconds\": {\n \"value\": \"@pipeline().parameters.WaitTime\",\n \"type\": \"Expression\"\n }\n }\n }\n ],\n \"parameters\": {\n \"WaitTime\": {\n \"type\": \"int\",\n \"defaultValue\": 5\n }\n },\n \"folder\": {\n \"name\": \"_Workers\"\n },\n \"annotations\": [\n \"Worker\"\n ]\n },\n \"dependsOn\": []\n },\n {\n \"name\": \"[concat(parameters('factoryName'), '/Wait 10')]\",\n \"type\": \"Microsoft.DataFactory/factories/pipelines\",\n \"apiVersion\": \"2018-06-01\",\n \"properties\": {\n \"description\": \"Used just so the ADF.procfwk has something to call during development.\",\n \"activities\": [\n {\n \"name\": \"Wait10\",\n \"description\": \"Framework development worker simulator.\",\n \"type\": \"Wait\",\n \"dependsOn\": [],\n \"userProperties\": [],\n \"typeProperties\": {\n \"waitTimeInSeconds\": {\n \"value\": \"@pipeline().parameters.WaitTime\",\n \"type\": \"Expression\"\n }\n }\n }\n ],\n \"parameters\": {\n \"WaitTime\": {\n \"type\": \"int\",\n \"defaultValue\": 5\n }\n },\n \"folder\": {\n \"name\": \"_Workers\"\n },\n \"annotations\": [\n \"Worker\"\n ]\n },\n \"dependsOn\": []\n },\n {\n \"name\": \"[concat(parameters('factoryName'), '/Wait 2')]\",\n \"type\": \"Microsoft.DataFactory/factories/pipelines\",\n \"apiVersion\": \"2018-06-01\",\n \"properties\": {\n \"description\": \"Used just so the ADF.procfwk has something to call during development.\",\n \"activities\": [\n {\n \"name\": \"Wait2\",\n \"description\": \"Framework development worker simulator.\",\n \"type\": \"Wait\",\n \"dependsOn\": [],\n \"userProperties\": [],\n \"typeProperties\": {\n \"waitTimeInSeconds\": {\n \"value\": \"@pipeline().parameters.WaitTime\",\n \"type\": \"Expression\"\n }\n }\n }\n ],\n \"parameters\": {\n \"WaitTime\": {\n \"type\": \"int\",\n \"defaultValue\": 5\n }\n },\n \"folder\": {\n \"name\": \"_Workers\"\n },\n \"annotations\": [\n \"Worker\"\n ]\n },\n \"dependsOn\": []\n },\n {\n \"name\": \"[concat(parameters('factoryName'), '/Wait 3')]\",\n \"type\": \"Microsoft.DataFactory/factories/pipelines\",\n \"apiVersion\": \"2018-06-01\",\n \"properties\": {\n \"description\": \"Used just so the ADF.procfwk has something to call during development.\",\n \"activities\": [\n {\n \"name\": \"Wait3\",\n \"description\": \"Framework development worker simulator.\",\n \"type\": \"Wait\",\n \"dependsOn\": [],\n \"userProperties\": [],\n \"typeProperties\": {\n \"waitTimeInSeconds\": {\n \"value\": \"@pipeline().parameters.WaitTime\",\n \"type\": \"Expression\"\n }\n }\n }\n ],\n \"parameters\": {\n \"WaitTime\": {\n \"type\": \"int\",\n \"defaultValue\": 5\n }\n },\n \"folder\": {\n \"name\": \"_Workers\"\n },\n \"annotations\": [\n \"Worker\"\n ]\n },\n \"dependsOn\": []\n },\n {\n \"name\": \"[concat(parameters('factoryName'), '/Wait 4')]\",\n \"type\": \"Microsoft.DataFactory/factories/pipelines\",\n \"apiVersion\": \"2018-06-01\",\n \"properties\": {\n \"description\": \"Used just so the ADF.procfwk has something to call during development.\",\n \"activities\": [\n {\n \"name\": \"Wait4\",\n \"description\": \"Framework development worker simulator.\",\n \"type\": \"Wait\",\n \"dependsOn\": [],\n \"userProperties\": [],\n \"typeProperties\": {\n \"waitTimeInSeconds\": {\n \"value\": \"@pipeline().parameters.WaitTime\",\n \"type\": \"Expression\"\n }\n }\n }\n ],\n \"parameters\": {\n \"WaitTime\": {\n \"type\": \"int\",\n \"defaultValue\": 5\n }\n },\n \"folder\": {\n \"name\": \"_Workers\"\n },\n \"annotations\": [\n \"Worker\"\n ]\n },\n \"dependsOn\": []\n },\n {\n \"name\": \"[concat(parameters('factoryName'), '/Wait 5')]\",\n \"type\": \"Microsoft.DataFactory/factories/pipelines\",\n \"apiVersion\": \"2018-06-01\",\n \"properties\": {\n \"description\": \"Used just so the ADF.procfwk has something to call during development.\",\n \"activities\": [\n {\n \"name\": \"Wait5\",\n \"description\": \"Framework development worker simulator.\",\n \"type\": \"Wait\",\n \"dependsOn\": [],\n \"userProperties\": [],\n \"typeProperties\": {\n \"waitTimeInSeconds\": {\n \"value\": \"@pipeline().parameters.WaitTime\",\n \"type\": \"Expression\"\n }\n }\n }\n ],\n \"parameters\": {\n \"WaitTime\": {\n \"type\": \"int\",\n \"defaultValue\": 5\n }\n },\n \"folder\": {\n \"name\": \"_Workers\"\n },\n \"annotations\": [\n \"Worker\"\n ]\n },\n \"dependsOn\": []\n },\n {\n \"name\": \"[concat(parameters('factoryName'), '/Wait 6')]\",\n \"type\": \"Microsoft.DataFactory/factories/pipelines\",\n \"apiVersion\": \"2018-06-01\",\n \"properties\": {\n \"description\": \"Used just so the ADF.procfwk has something to call during development.\",\n \"activities\": [\n {\n \"name\": \"Wait6\",\n \"description\": \"Framework development worker simulator.\",\n \"type\": \"Wait\",\n \"dependsOn\": [],\n \"userProperties\": [],\n \"typeProperties\": {\n \"waitTimeInSeconds\": {\n \"value\": \"@pipeline().parameters.WaitTime\",\n \"type\": \"Expression\"\n }\n }\n }\n ],\n \"parameters\": {\n \"WaitTime\": {\n \"type\": \"int\",\n \"defaultValue\": 5\n }\n },\n \"folder\": {\n \"name\": \"_Workers\"\n },\n \"annotations\": [\n \"Worker\"\n ]\n },\n \"dependsOn\": []\n },\n {\n \"name\": \"[concat(parameters('factoryName'), '/Wait 7')]\",\n \"type\": \"Microsoft.DataFactory/factories/pipelines\",\n \"apiVersion\": \"2018-06-01\",\n \"properties\": {\n \"description\": \"Used just so the ADF.procfwk has something to call during development.\",\n \"activities\": [\n {\n \"name\": \"Wait7\",\n \"description\": \"Framework development worker simulator.\",\n \"type\": \"Wait\",\n \"dependsOn\": [],\n \"userProperties\": [],\n \"typeProperties\": {\n \"waitTimeInSeconds\": {\n \"value\": \"@pipeline().parameters.WaitTime\",\n \"type\": \"Expression\"\n }\n }\n }\n ],\n \"parameters\": {\n \"WaitTime\": {\n \"type\": \"int\",\n \"defaultValue\": 5\n }\n },\n \"folder\": {\n \"name\": \"_Workers\"\n },\n \"annotations\": [\n \"Worker\"\n ]\n },\n \"dependsOn\": []\n },\n {\n \"name\": \"[concat(parameters('factoryName'), '/Wait 8')]\",\n \"type\": \"Microsoft.DataFactory/factories/pipelines\",\n \"apiVersion\": \"2018-06-01\",\n \"properties\": {\n \"description\": \"Used just so the ADF.procfwk has something to call during development.\",\n \"activities\": [\n {\n \"name\": \"Wait8\",\n \"description\": \"Framework development worker simulator.\",\n \"type\": \"Wait\",\n \"dependsOn\": [],\n \"userProperties\": [],\n \"typeProperties\": {\n \"waitTimeInSeconds\": {\n \"value\": \"@pipeline().parameters.WaitTime\",\n \"type\": \"Expression\"\n }\n }\n }\n ],\n \"parameters\": {\n \"WaitTime\": {\n \"type\": \"int\",\n \"defaultValue\": 5\n }\n },\n \"folder\": {\n \"name\": \"_Workers\"\n },\n \"annotations\": [\n \"Worker\"\n ]\n },\n \"dependsOn\": []\n },\n {\n \"name\": \"[concat(parameters('factoryName'), '/Wait 9')]\",\n \"type\": \"Microsoft.DataFactory/factories/pipelines\",\n \"apiVersion\": \"2018-06-01\",\n \"properties\": {\n \"description\": \"Used just so the ADF.procfwk has something to call during development.\",\n \"activities\": [\n {\n \"name\": \"Wait9\",\n \"description\": \"Framework development worker simulator.\",\n \"type\": \"Wait\",\n \"dependsOn\": [],\n \"userProperties\": [],\n \"typeProperties\": {\n \"waitTimeInSeconds\": {\n \"value\": \"@pipeline().parameters.WaitTime\",\n \"type\": \"Expression\"\n }\n }\n }\n ],\n \"parameters\": {\n \"WaitTime\": {\n \"type\": \"int\",\n \"defaultValue\": 15\n }\n },\n \"folder\": {\n \"name\": \"_Workers\"\n },\n \"annotations\": [\n \"Worker\"\n ]\n },\n \"dependsOn\": []\n },\n {\n \"name\": \"[concat(parameters('factoryName'), '/GetSetMetadata')]\",\n \"type\": \"Microsoft.DataFactory/factories/datasets\",\n \"apiVersion\": \"2018-06-01\",\n \"properties\": {\n \"description\": \"Single generic dataset used to get and set all database metadata.\",\n \"linkedServiceName\": {\n \"referenceName\": \"SupportDatabase\",\n \"type\": \"LinkedServiceReference\"\n },\n \"folder\": {\n \"name\": \"_ProcFwk\"\n },\n \"annotations\": [\n \"ADF.procfwk\"\n ],\n \"type\": \"AzureSqlTable\",\n \"schema\": [],\n \"typeProperties\": {}\n },\n \"dependsOn\": [\n \"[concat(variables('factoryId'), '/linkedServices/SupportDatabase')]\"\n ]\n },\n {\n \"name\": \"[concat(parameters('factoryName'), '/FrameworkFunctions')]\",\n \"type\": \"Microsoft.DataFactory/factories/linkedServices\",\n \"apiVersion\": \"2018-06-01\",\n \"properties\": {\n \"description\": \"Interact with the Azure Functions App used as middle ware when making requests to Worker pipelines. Authentication done at the Function App level.\",\n \"annotations\": [\n \"ADF.procfwk\"\n ],\n \"type\": \"AzureFunction\",\n \"typeProperties\": {\n \"functionAppUrl\": \"[parameters('FrameworkFunctions_properties_typeProperties_functionAppUrl')]\",\n \"functionKey\": {\n \"type\": \"AzureKeyVaultSecret\",\n \"store\": {\n \"referenceName\": \"Keys\",\n \"type\": \"LinkedServiceReference\"\n },\n \"secretName\": \"FrameworkFunctionsKey\"\n }\n }\n },\n \"dependsOn\": [\n \"[concat(variables('factoryId'), '/linkedServices/Keys')]\"\n ]\n },\n {\n \"name\": \"[concat(parameters('factoryName'), '/Keys')]\",\n \"type\": \"Microsoft.DataFactory/factories/linkedServices\",\n \"apiVersion\": \"2018-06-01\",\n \"properties\": {\n \"description\": \"Connection to Key Vault for all other ADF linked service credentials required to run the processing framework.\",\n \"annotations\": [\n \"ADF.procfwk\"\n ],\n \"type\": \"AzureKeyVault\",\n \"typeProperties\": {\n \"baseUrl\": \"[parameters('Keys_properties_typeProperties_baseUrl')]\"\n }\n },\n \"dependsOn\": []\n },\n {\n \"name\": \"[concat(parameters('factoryName'), '/SupportDatabase')]\",\n \"type\": \"Microsoft.DataFactory/factories/linkedServices\",\n \"apiVersion\": \"2018-06-01\",\n \"properties\": {\n \"description\": \"Connection between ADF and processing framework metadata SQLDB.\",\n \"annotations\": [\n \"ADF.procfwk\"\n ],\n \"type\": \"AzureSqlDatabase\",\n \"typeProperties\": {\n \"connectionString\": {\n \"type\": \"AzureKeyVaultSecret\",\n \"store\": {\n \"referenceName\": \"Keys\",\n \"type\": \"LinkedServiceReference\"\n },\n \"secretName\": \"[parameters('SupportDatabase_properties_typeProperties_connectionString_secretName')]\"\n }\n }\n },\n \"dependsOn\": [\n \"[concat(variables('factoryId'), '/linkedServices/Keys')]\"\n ]\n },\n {\n \"name\": \"[concat(parameters('factoryName'), '/FunctionalTestingTrigger')]\",\n \"type\": \"Microsoft.DataFactory/factories/triggers\",\n \"apiVersion\": \"2018-06-01\",\n \"properties\": {\n \"description\": \"Used for functional testing of the framework in a dedicated environment.\",\n \"annotations\": [\n \"ADF.procfwk\"\n ],\n \"runtimeState\": \"Stopped\",\n \"pipelines\": [\n {\n \"pipelineReference\": {\n \"referenceName\": \"01-Grandparent\",\n \"type\": \"PipelineReference\"\n },\n \"parameters\": {}\n }\n ],\n \"type\": \"ScheduleTrigger\",\n \"typeProperties\": {\n \"recurrence\": {\n \"frequency\": \"Hour\",\n \"interval\": 2,\n \"startTime\": \"2020-04-06T15:00:00Z\",\n \"timeZone\": \"UTC\"\n }\n }\n },\n \"dependsOn\": [\n \"[concat(variables('factoryId'), '/pipelines/01-Grandparent')]\"\n ]\n }\n ]\n}"
},
{
"path": "ARM Templates/Data Factory/v1.9.2 Export.json",
"content": "{\n \"$schema\": \"http://schema.management.azure.com/schemas/2015-01-01/deploymentTemplate.json#\",\n \"contentVersion\": \"1.0.0.0\",\n \"parameters\": {\n \"factoryName\": {\n \"type\": \"string\",\n \"metadata\": \"Data Factory name\",\n \"defaultValue\": \"\"\n },\n \"FrameworkFunctions_properties_typeProperties_functionAppUrl\": {\n \"type\": \"string\",\n \"defaultValue\": \"\"\n },\n \"Keys_properties_typeProperties_baseUrl\": {\n \"type\": \"string\",\n \"defaultValue\": \"\"\n },\n \"SupportDatabase_properties_typeProperties_connectionString_secretName\": {\n \"type\": \"string\",\n \"defaultValue\": \"\"\n }\n },\n \"variables\": {\n \"factoryId\": \"[concat('Microsoft.DataFactory/factories/', parameters('factoryName'))]\"\n },\n \"resources\": [\n {\n \"name\": \"[concat(parameters('factoryName'), '/01-Grandparent')]\",\n \"type\": \"Microsoft.DataFactory/factories/pipelines\",\n \"apiVersion\": \"2018-06-01\",\n \"properties\": {\n \"description\": \"ADF.procfwk grandparent pipeline used optionally to bootstrap any wider processes in your Data Factory that then calls the processing framework.\",\n \"activities\": [\n {\n \"name\": \"procfwk\",\n \"description\": \"Call procfwk\",\n \"type\": \"ExecutePipeline\",\n \"dependsOn\": [],\n \"userProperties\": [],\n \"typeProperties\": {\n \"pipeline\": {\n \"referenceName\": \"02-Parent\",\n \"type\": \"PipelineReference\"\n },\n \"waitOnCompletion\": true,\n \"parameters\": {\n \"BatchName\": {\n \"value\": \"@pipeline().parameters.BatchName\",\n \"type\": \"Expression\"\n }\n }\n }\n }\n ],\n \"parameters\": {\n \"BatchName\": {\n \"type\": \"string\",\n \"defaultValue\": \"NotUsed\"\n }\n },\n \"folder\": {\n \"name\": \"_ProcFwk\"\n },\n \"annotations\": [\n \"procfwk\",\n \"Grandparent\"\n ]\n },\n \"dependsOn\": [\n \"[concat(variables('factoryId'), '/pipelines/02-Parent')]\"\n ]\n },\n {\n \"name\": \"[concat(parameters('factoryName'), '/02-Parent')]\",\n \"type\": \"Microsoft.DataFactory/factories/pipelines\",\n \"apiVersion\": \"2018-06-01\",\n \"properties\": {\n \"description\": \"ADF.procfwk parent pipeline used to bootstrap the orchestration framework in perform the first level ForEach calls in sequence for the metadata stages.\",\n \"activities\": [\n {\n \"name\": \"Get Stages\",\n \"description\": \"Returns a distinct list of execution stages within the framework metadata.\",\n \"type\": \"Lookup\",\n \"dependsOn\": [\n {\n \"activity\": \"Set Execution Id\",\n \"dependencyConditions\": [\n \"Succeeded\"\n ]\n }\n ],\n \"policy\": {\n \"timeout\": \"0.00:10:00\",\n \"retry\": 0,\n \"retryIntervalInSeconds\": 30,\n \"secureOutput\": false,\n \"secureInput\": false\n },\n \"userProperties\": [],\n \"typeProperties\": {\n \"source\": {\n \"type\": \"AzureSqlSource\",\n \"sqlReaderStoredProcedureName\": \"[[procfwk].[GetStages]\",\n \"storedProcedureParameters\": {\n \"ExecutionId\": {\n \"type\": \"Guid\",\n \"value\": {\n \"value\": \"@variables('ExecutionId')\",\n \"type\": \"Expression\"\n }\n }\n },\n \"queryTimeout\": \"02:00:00\",\n \"partitionOption\": \"None\"\n },\n \"dataset\": {\n \"referenceName\": \"GetSetMetadata\",\n \"type\": \"DatasetReference\",\n \"parameters\": {}\n },\n \"firstRowOnly\": false\n }\n },\n {\n \"name\": \"Execute Stages\",\n \"description\": \"Top level ForEach to sequentially call all processing stages within the framework metadata. Items for iteration passed from the Get Stages lookup activity.\",\n \"type\": \"ForEach\",\n \"dependsOn\": [\n {\n \"activity\": \"Get Stages\",\n \"dependencyConditions\": [\n \"Succeeded\"\n ]\n }\n ],\n \"userProperties\": [],\n \"typeProperties\": {\n \"items\": {\n \"value\": \"@activity('Get Stages').output.value\",\n \"type\": \"Expression\"\n },\n \"isSequential\": true,\n \"activities\": [\n {\n \"name\": \"Stage Executor\",\n \"description\": \"Call to the framework generic child pipeline for a given execution stage.\",\n \"type\": \"ExecutePipeline\",\n \"dependsOn\": [\n {\n \"activity\": \"Log Stage Preparing\",\n \"dependencyConditions\": [\n \"Succeeded\"\n ]\n }\n ],\n \"userProperties\": [],\n \"typeProperties\": {\n \"pipeline\": {\n \"referenceName\": \"03-Child\",\n \"type\": \"PipelineReference\"\n },\n \"waitOnCompletion\": true,\n \"parameters\": {\n \"StageId\": {\n \"value\": \"@item().StageId\",\n \"type\": \"Expression\"\n },\n \"ExecutionId\": {\n \"value\": \"@variables('ExecutionId')\",\n \"type\": \"Expression\"\n }\n }\n }\n },\n {\n \"name\": \"Log Stage Preparing\",\n \"description\": \"Update the current execution table flagging all pipelines within the stage as preparing.\",\n \"type\": \"SqlServerStoredProcedure\",\n \"dependsOn\": [\n {\n \"activity\": \"Check and Update Blockers\",\n \"dependencyConditions\": [\n \"Succeeded\"\n ]\n }\n ],\n \"policy\": {\n \"timeout\": \"0.00:10:00\",\n \"retry\": 0,\n \"retryIntervalInSeconds\": 30,\n \"secureOutput\": false,\n \"secureInput\": false\n },\n \"userProperties\": [],\n \"typeProperties\": {\n \"storedProcedureName\": \"[[procfwk].[SetLogStagePreparing]\",\n \"storedProcedureParameters\": {\n \"ExecutionId\": {\n \"value\": {\n \"value\": \"@variables('ExecutionId')\",\n \"type\": \"Expression\"\n },\n \"type\": \"Guid\"\n },\n \"StageId\": {\n \"value\": {\n \"value\": \"@item().StageId\",\n \"type\": \"Expression\"\n },\n \"type\": \"Int32\"\n }\n }\n },\n \"linkedServiceName\": {\n \"referenceName\": \"SupportDatabase\",\n \"type\": \"LinkedServiceReference\"\n }\n },\n {\n \"name\": \"Check and Update Blockers\",\n \"description\": \"Used to double check and stop the next execution stage if failures and blockers have be incurred. This also depends on the failure handling property value which defines the stored procedure behaviour.\",\n \"type\": \"SqlServerStoredProcedure\",\n \"dependsOn\": [],\n \"policy\": {\n \"timeout\": \"0.00:10:00\",\n \"retry\": 0,\n \"retryIntervalInSeconds\": 30,\n \"secureOutput\": false,\n \"secureInput\": false\n },\n \"userProperties\": [],\n \"typeProperties\": {\n \"storedProcedureName\": \"[[procfwk].[CheckForBlockedPipelines]\",\n \"storedProcedureParameters\": {\n \"ExecutionId\": {\n \"value\": {\n \"value\": \"@variables('ExecutionId')\",\n \"type\": \"Expression\"\n },\n \"type\": \"Guid\"\n },\n \"StageId\": {\n \"value\": {\n \"value\": \"@item().StageId\",\n \"type\": \"Expression\"\n },\n \"type\": \"Int32\"\n }\n }\n },\n \"linkedServiceName\": {\n \"referenceName\": \"SupportDatabase\",\n \"type\": \"LinkedServiceReference\"\n }\n }\n ]\n }\n },\n {\n \"name\": \"Execution Wrapper\",\n \"description\": \"Wrapper to reset and restart processing or create a completely new execution instance of the framework metadata.\",\n \"type\": \"Lookup\",\n \"dependsOn\": [\n {\n \"activity\": \"Clean Up Previous Run\",\n \"dependencyConditions\": [\n \"Succeeded\"\n ]\n }\n ],\n \"policy\": {\n \"timeout\": \"0.00:10:00\",\n \"retry\": 0,\n \"retryIntervalInSeconds\": 30,\n \"secureOutput\": false,\n \"secureInput\": false\n },\n \"userProperties\": [],\n \"typeProperties\": {\n \"source\": {\n \"type\": \"AzureSqlSource\",\n \"sqlReaderStoredProcedureName\": \"[[procfwk].[ExecutionWrapper]\",\n \"storedProcedureParameters\": {\n \"CallingDataFactory\": {\n \"type\": \"String\",\n \"value\": {\n \"value\": \"@pipeline().DataFactory\",\n \"type\": \"Expression\"\n }\n },\n \"BatchName\": {\n \"type\": \"String\",\n \"value\": {\n \"value\": \"@pipeline().parameters.BatchName\",\n \"type\": \"Expression\"\n }\n }\n },\n \"queryTimeout\": \"02:00:00\",\n \"partitionOption\": \"None\"\n },\n \"dataset\": {\n \"referenceName\": \"GetSetMetadata\",\n \"type\": \"DatasetReference\",\n \"parameters\": {}\n }\n }\n },\n {\n \"name\": \"Check Outcome and Update Logs\",\n \"description\": \"After a successful execution run the current execution metadata is moved to the long term logging table by this stored procedure call. Otherwise an error will be raised.\",\n \"type\": \"SqlServerStoredProcedure\",\n \"dependsOn\": [\n {\n \"activity\": \"Execute Stages\",\n \"dependencyConditions\": [\n \"Succeeded\"\n ]\n }\n ],\n \"policy\": {\n \"timeout\": \"0.00:10:00\",\n \"retry\": 0,\n \"retryIntervalInSeconds\": 30,\n \"secureOutput\": false,\n \"secureInput\": false\n },\n \"userProperties\": [],\n \"typeProperties\": {\n \"storedProcedureName\": \"[[procfwk].[UpdateExecutionLog]\",\n \"storedProcedureParameters\": {\n \"PerformErrorCheck\": {\n \"value\": {\n \"value\": \"@bool(1)\",\n \"type\": \"Expression\"\n },\n \"type\": \"Boolean\"\n },\n \"ExecutionId\": {\n \"value\": {\n \"value\": \"@variables('ExecutionId')\",\n \"type\": \"Expression\"\n },\n \"type\": \"Guid\"\n }\n }\n },\n \"linkedServiceName\": {\n \"referenceName\": \"SupportDatabase\",\n \"type\": \"LinkedServiceReference\"\n }\n },\n {\n \"name\": \"Check Previous Execution\",\n \"description\": \"Query the current execution table for worker pipelines that require a clean up from the previous execution run.\",\n \"type\": \"Lookup\",\n \"dependsOn\": [\n {\n \"activity\": \"Execute Precursor\",\n \"dependencyConditions\": [\n \"Succeeded\"\n ]\n }\n ],\n \"policy\": {\n \"timeout\": \"0.00:10:00\",\n \"retry\": 0,\n \"retryIntervalInSeconds\": 30,\n \"secureOutput\": false,\n \"secureInput\": false\n },\n \"userProperties\": [],\n \"typeProperties\": {\n \"source\": {\n \"type\": \"AzureSqlSource\",\n \"sqlReaderStoredProcedureName\": \"[[procfwk].[CheckPreviousExeuction]\",\n \"storedProcedureParameters\": {\n \"BatchName\": {\n \"type\": \"String\",\n \"value\": {\n \"value\": \"@pipeline().parameters.BatchName\",\n \"type\": \"Expression\"\n }\n }\n },\n \"queryTimeout\": \"02:00:00\",\n \"partitionOption\": \"None\"\n },\n \"dataset\": {\n \"referenceName\": \"GetSetMetadata\",\n \"type\": \"DatasetReference\",\n \"parameters\": {}\n },\n \"firstRowOnly\": false\n }\n },\n {\n \"name\": \"Clean Up Previous Run\",\n \"description\": \"Handle Worker pipelines that are reported as Running when the parent pipeline is called again. Get what the actual status of those pipelines is.\",\n \"type\": \"ForEach\",\n \"dependsOn\": [\n {\n \"activity\": \"Check Previous Execution\",\n \"dependencyConditions\": [\n \"Succeeded\"\n ]\n },\n {\n \"activity\": \"Check Metadata Integrity\",\n \"dependencyConditions\": [\n \"Succeeded\"\n ]\n }\n ],\n \"userProperties\": [],\n \"typeProperties\": {\n \"items\": {\n \"value\": \"@activity('Check Previous Execution').output.value\",\n \"type\": \"Expression\"\n },\n \"isSequential\": false,\n \"batchCount\": 50,\n \"activities\": [\n {\n \"name\": \"Get SPN Details\",\n \"type\": \"Lookup\",\n \"dependsOn\": [],\n \"policy\": {\n \"timeout\": \"0.00:10:00\",\n \"retry\": 0,\n \"retryIntervalInSeconds\": 30,\n \"secureOutput\": true,\n \"secureInput\": false\n },\n \"userProperties\": [],\n \"typeProperties\": {\n \"source\": {\n \"type\": \"AzureSqlSource\",\n \"sqlReaderStoredProcedureName\": \"[[procfwk].[GetWorkerAuthDetails]\",\n \"storedProcedureParameters\": {\n \"ExecutionId\": {\n \"type\": \"Guid\",\n \"value\": {\n \"value\": \"@item().LocalExecutionId\",\n \"type\": \"Expression\"\n }\n },\n \"PipelineId\": {\n \"type\": \"Int32\",\n \"value\": {\n \"value\": \"@item().PipelineId\",\n \"type\": \"Expression\"\n }\n },\n \"StageId\": {\n \"type\": \"Int32\",\n \"value\": {\n \"value\": \"@item().StageId\",\n \"type\": \"Expression\"\n }\n }\n },\n \"queryTimeout\": \"02:00:00\",\n \"partitionOption\": \"None\"\n },\n \"dataset\": {\n \"referenceName\": \"GetSetMetadata\",\n \"type\": \"DatasetReference\",\n \"parameters\": {}\n }\n }\n },\n {\n \"name\": \"Log Pipeline Checking\",\n \"type\": \"SqlServerStoredProcedure\",\n \"dependsOn\": [],\n \"policy\": {\n \"timeout\": \"0.00:10:00\",\n \"retry\": 0,\n \"retryIntervalInSeconds\": 30,\n \"secureOutput\": false,\n \"secureInput\": false\n },\n \"userProperties\": [],\n \"typeProperties\": {\n \"storedProcedureName\": \"[[procfwk].[SetLogPipelineChecking]\",\n \"storedProcedureParameters\": {\n \"ExecutionId\": {\n \"value\": {\n \"value\": \"@item().LocalExecutionId\",\n \"type\": \"Expression\"\n },\n \"type\": \"Guid\"\n },\n \"PipelineId\": {\n \"value\": {\n \"value\": \"@item().PipelineId\",\n \"type\": \"Expression\"\n },\n \"type\": \"Int32\"\n },\n \"StageId\": {\n \"value\": {\n \"value\": \"@item().StageId\",\n \"type\": \"Expression\"\n },\n \"type\": \"Int32\"\n }\n }\n },\n \"linkedServiceName\": {\n \"referenceName\": \"SupportDatabase\",\n \"type\": \"LinkedServiceReference\"\n }\n },\n {\n \"name\": \"Get Pipeline Status\",\n \"type\": \"AzureFunctionActivity\",\n \"dependsOn\": [\n {\n \"activity\": \"Get SPN Details\",\n \"dependencyConditions\": [\n \"Succeeded\"\n ]\n },\n {\n \"activity\": \"Log Pipeline Checking\",\n \"dependencyConditions\": [\n \"Succeeded\"\n ]\n }\n ],\n \"policy\": {\n \"timeout\": \"0.00:10:00\",\n \"retry\": 0,\n \"retryIntervalInSeconds\": 30,\n \"secureOutput\": false,\n \"secureInput\": true\n },\n \"userProperties\": [],\n \"typeProperties\": {\n \"functionName\": \"CheckPipelineStatus\",\n \"method\": \"POST\",\n \"headers\": {},\n \"body\": {\n \"value\": \"@concat('\\n{\\n \\\"tenantId\\\": \\\"',activity('Get SPN Details').output.firstRow.TenantId,'\\\",\\n \\\"applicationId\\\": \\\"',activity('Get SPN Details').output.firstRow.AppId,'\\\",\\n \\\"authenticationKey\\\": \\\"',activity('Get SPN Details').output.firstRow.AppSecret,'\\\",\\n \\\"subscriptionId\\\": \\\"',activity('Get SPN Details').output.firstRow.SubscriptionId,'\\\",\\n \\\"resourceGroup\\\": \\\"',item().ResourceGroupName,'\\\",\\n \\\"factoryName\\\": \\\"',item().DataFactoryName,'\\\",\\n \\\"pipelineName\\\": \\\"',item().PipelineName,'\\\",\\n \\\"runId\\\": \\\"',item().AdfPipelineRunId,'\\\"\\n}')\",\n \"type\": \"Expression\"\n }\n },\n \"linkedServiceName\": {\n \"referenceName\": \"FrameworkFunctions\",\n \"type\": \"LinkedServiceReference\"\n }\n },\n {\n \"name\": \"Set Pipeline Status\",\n \"description\": \"Update the metadata depending on the actual pipeline outcome. Using the status as the case.\",\n \"type\": \"Switch\",\n \"dependsOn\": [\n {\n \"activity\": \"Get Pipeline Status\",\n \"dependencyConditions\": [\n \"Succeeded\"\n ]\n }\n ],\n \"userProperties\": [],\n \"typeProperties\": {\n \"on\": {\n \"value\": \"@activity('Get Pipeline Status').output.Status\",\n \"type\": \"Expression\"\n },\n \"cases\": [\n {\n \"value\": \"Failed\",\n \"activities\": [\n {\n \"name\": \"Pipeline Status Failed\",\n \"description\": \"Updates the current execution table with a pipeline status of failed if the function outcome is failed. Also blocks pipelines in the downstream execution stage.\",\n \"type\": \"SqlServerStoredProcedure\",\n \"dependsOn\": [],\n \"policy\": {\n \"timeout\": \"0.00:10:00\",\n \"retry\": 0,\n \"retryIntervalInSeconds\": 30,\n \"secureOutput\": false,\n \"secureInput\": false\n },\n \"userProperties\": [],\n \"typeProperties\": {\n \"storedProcedureName\": \"[[procfwk].[SetLogPipelineFailed]\",\n \"storedProcedureParameters\": {\n \"ExecutionId\": {\n \"value\": {\n \"value\": \"@item().LocalExecutionId\",\n \"type\": \"Expression\"\n },\n \"type\": \"Guid\"\n },\n \"PipelineId\": {\n \"value\": {\n \"value\": \"@item().PipelineId\",\n \"type\": \"Expression\"\n },\n \"type\": \"Int32\"\n },\n \"RunId\": {\n \"value\": null,\n \"type\": \"Guid\"\n },\n \"StageId\": {\n \"value\": {\n \"value\": \"@item().StageId\",\n \"type\": \"Expression\"\n },\n \"type\": \"Int32\"\n }\n }\n },\n \"linkedServiceName\": {\n \"referenceName\": \"SupportDatabase\",\n \"type\": \"LinkedServiceReference\"\n }\n }\n ]\n },\n {\n \"value\": \"Succeeded\",\n \"activities\": [\n {\n \"name\": \"Pipeline Status Succeeded\",\n \"description\": \"Updates the current execution table with a pipeline status of success if the function outcome is succeeded.\",\n \"type\": \"SqlServerStoredProcedure\",\n \"dependsOn\": [],\n \"policy\": {\n \"timeout\": \"0.00:10:00\",\n \"retry\": 0,\n \"retryIntervalInSeconds\": 30,\n \"secureOutput\": false,\n \"secureInput\": false\n },\n \"userProperties\": [],\n \"typeProperties\": {\n \"storedProcedureName\": \"[[procfwk].[SetLogPipelineSuccess]\",\n \"storedProcedureParameters\": {\n \"ExecutionId\": {\n \"value\": {\n \"value\": \"@item().LocalExecutionId\",\n \"type\": \"Expression\"\n },\n \"type\": \"Guid\"\n },\n \"PipelineId\": {\n \"value\": {\n \"value\": \"@item().PipelineId\",\n \"type\": \"Expression\"\n },\n \"type\": \"Int32\"\n },\n \"StageId\": {\n \"value\": {\n \"value\": \"@item().StageId\",\n \"type\": \"Expression\"\n },\n \"type\": \"Int32\"\n }\n }\n },\n \"linkedServiceName\": {\n \"referenceName\": \"SupportDatabase\",\n \"type\": \"LinkedServiceReference\"\n }\n }\n ]\n },\n {\n \"value\": \"Queued\",\n \"activities\": [\n {\n \"name\": \"Pipeline Status Queued - Running\",\n \"description\": \"Updates the current execution table with a pipeline status of running if the function outcome is queued.\",\n \"type\": \"SqlServerStoredProcedure\",\n \"dependsOn\": [],\n \"policy\": {\n \"timeout\": \"0.00:10:00\",\n \"retry\": 0,\n \"retryIntervalInSeconds\": 30,\n \"secureOutput\": false,\n \"secureInput\": false\n },\n \"userProperties\": [],\n \"typeProperties\": {\n \"storedProcedureName\": \"[[procfwk].[SetLogPipelineRunning]\",\n \"storedProcedureParameters\": {\n \"ExecutionId\": {\n \"value\": {\n \"value\": \"@item().LocalExecutionId\",\n \"type\": \"Expression\"\n },\n \"type\": \"Guid\"\n },\n \"PipelineId\": {\n \"value\": {\n \"value\": \"@item().PipelineId\",\n \"type\": \"Expression\"\n },\n \"type\": \"Int32\"\n },\n \"StageId\": {\n \"value\": {\n \"value\": \"@item().StageId\",\n \"type\": \"Expression\"\n },\n \"type\": \"Int32\"\n }\n }\n },\n \"linkedServiceName\": {\n \"referenceName\": \"SupportDatabase\",\n \"type\": \"LinkedServiceReference\"\n }\n }\n ]\n },\n {\n \"value\": \"InProgress\",\n \"activities\": [\n {\n \"name\": \"Pipeline Status InProgress - Running\",\n \"description\": \"Updates the current execution table with a pipeline status of running if the function outcome is in progress.\",\n \"type\": \"SqlServerStoredProcedure\",\n \"dependsOn\": [],\n \"policy\": {\n \"timeout\": \"0.00:10:00\",\n \"retry\": 0,\n \"retryIntervalInSeconds\": 30,\n \"secureOutput\": false,\n \"secureInput\": false\n },\n \"userProperties\": [],\n \"typeProperties\": {\n \"storedProcedureName\": \"[[procfwk].[SetLogPipelineRunning]\",\n \"storedProcedureParameters\": {\n \"ExecutionId\": {\n \"value\": {\n \"value\": \"@item().LocalExecutionId\",\n \"type\": \"Expression\"\n },\n \"type\": \"Guid\"\n },\n \"PipelineId\": {\n \"value\": {\n \"value\": \"@item().PipelineId\",\n \"type\": \"Expression\"\n },\n \"type\": \"Int32\"\n },\n \"StageId\": {\n \"value\": {\n \"value\": \"@item().StageId\",\n \"type\": \"Expression\"\n },\n \"type\": \"Int32\"\n }\n }\n },\n \"linkedServiceName\": {\n \"referenceName\": \"SupportDatabase\",\n \"type\": \"LinkedServiceReference\"\n }\n }\n ]\n },\n {\n \"value\": \"Cancelled\",\n \"activities\": [\n {\n \"name\": \"Pipeline Status Cancelled\",\n \"description\": \"Updates the current execution table with a pipeline status of cancelled if the function outcome is cancelled.\",\n \"type\": \"SqlServerStoredProcedure\",\n \"dependsOn\": [],\n \"policy\": {\n \"timeout\": \"0.00:10:00\",\n \"retry\": 0,\n \"retryIntervalInSeconds\": 30,\n \"secureOutput\": false,\n \"secureInput\": false\n },\n \"userProperties\": [],\n \"typeProperties\": {\n \"storedProcedureName\": \"[[procfwk].[SetLogPipelineCancelled]\",\n \"storedProcedureParameters\": {\n \"ExecutionId\": {\n \"value\": {\n \"value\": \"@item().LocalExecutionId\",\n \"type\": \"Expression\"\n },\n \"type\": \"Guid\"\n },\n \"PipelineId\": {\n \"value\": {\n \"value\": \"@item().PipelineId\",\n \"type\": \"Expression\"\n },\n \"type\": \"Int32\"\n },\n \"StageId\": {\n \"value\": {\n \"value\": \"@item().StageId\",\n \"type\": \"Expression\"\n },\n \"type\": \"Int32\"\n },\n \"CleanUpRun\": {\n \"value\": {\n \"value\": \"@bool(1)\",\n \"type\": \"Expression\"\n },\n \"type\": \"Boolean\"\n }\n }\n },\n \"linkedServiceName\": {\n \"referenceName\": \"SupportDatabase\",\n \"type\": \"LinkedServiceReference\"\n }\n }\n ]\n }\n ],\n \"defaultActivities\": [\n {\n \"name\": \"Pipeline Status Unknown\",\n \"description\": \"Updates the current execution table with a pipeline status of unknown if the function returns an unexpected outcome.\",\n \"type\": \"SqlServerStoredProcedure\",\n \"dependsOn\": [],\n \"policy\": {\n \"timeout\": \"0.00:10:00\",\n \"retry\": 0,\n \"retryIntervalInSeconds\": 30,\n \"secureOutput\": false,\n \"secureInput\": false\n },\n \"userProperties\": [],\n \"typeProperties\": {\n \"storedProcedureName\": \"[[procfwk].[SetLogPipelineUnknown]\",\n \"storedProcedureParameters\": {\n \"ExecutionId\": {\n \"value\": {\n \"value\": \"@item().LocalExecutionId\",\n \"type\": \"Expression\"\n },\n \"type\": \"Guid\"\n },\n \"PipelineId\": {\n \"value\": {\n \"value\": \"@item().PipelineId\",\n \"type\": \"Expression\"\n },\n \"type\": \"Int32\"\n },\n \"StageId\": {\n \"value\": {\n \"value\": \"@item().StageId\",\n \"type\": \"Expression\"\n },\n \"type\": \"Int32\"\n },\n \"CleanUpRun\": {\n \"value\": {\n \"value\": \"@bool(1)\",\n \"type\": \"Expression\"\n },\n \"type\": \"Boolean\"\n }\n }\n },\n \"linkedServiceName\": {\n \"referenceName\": \"SupportDatabase\",\n \"type\": \"LinkedServiceReference\"\n }\n }\n ]\n }\n },\n {\n \"name\": \"Set Last Check DateTime\",\n \"description\": \"Update the current execution table with a date time from when the function last checked the pipeline status.\",\n \"type\": \"SqlServerStoredProcedure\",\n \"dependsOn\": [\n {\n \"activity\": \"Get Pipeline Status\",\n \"dependencyConditions\": [\n \"Succeeded\"\n ]\n }\n ],\n \"policy\": {\n \"timeout\": \"0.00:10:00\",\n \"retry\": 0,\n \"retryIntervalInSeconds\": 30,\n \"secureOutput\": false,\n \"secureInput\": false\n },\n \"userProperties\": [],\n \"typeProperties\": {\n \"storedProcedureName\": \"[[procfwk].[SetLogPipelineLastStatusCheck]\",\n \"storedProcedureParameters\": {\n \"ExecutionId\": {\n \"value\": {\n \"value\": \"@item().LocalExecutionId\",\n \"type\": \"Expression\"\n },\n \"type\": \"Guid\"\n },\n \"PipelineId\": {\n \"value\": {\n \"value\": \"@item().PipelineId\",\n \"type\": \"Expression\"\n },\n \"type\": \"Int32\"\n },\n \"StageId\": {\n \"value\": {\n \"value\": \"@item().StageId\",\n \"type\": \"Expression\"\n },\n \"type\": \"Int32\"\n }\n }\n },\n \"linkedServiceName\": {\n \"referenceName\": \"SupportDatabase\",\n \"type\": \"LinkedServiceReference\"\n }\n }\n ]\n }\n },\n {\n \"name\": \"Execute Precursor\",\n \"description\": \"Uses the database property value ExecutionPrecursorProc to run any custom logic against the metadata database before the execution run starts.\",\n \"type\": \"SqlServerStoredProcedure\",\n \"dependsOn\": [\n {\n \"activity\": \"Is Parent Already Running\",\n \"dependencyConditions\": [\n \"Succeeded\"\n ]\n }\n ],\n \"policy\": {\n \"timeout\": \"0.00:10:00\",\n \"retry\": 0,\n \"retryIntervalInSeconds\": 30,\n \"secureOutput\": false,\n \"secureInput\": false\n },\n \"userProperties\": [],\n \"typeProperties\": {\n \"storedProcedureName\": \"[[procfwk].[ExecutePrecursorProcedure]\"\n },\n \"linkedServiceName\": {\n \"referenceName\": \"SupportDatabase\",\n \"type\": \"LinkedServiceReference\"\n }\n },\n {\n \"name\": \"Set Execution Id\",\n \"description\": \"Set the local execution Id to a pipeline variable for each in several downstream activities.\",\n \"type\": \"SetVariable\",\n \"dependsOn\": [\n {\n \"activity\": \"Execution Wrapper\",\n \"dependencyConditions\": [\n \"Succeeded\"\n ]\n }\n ],\n \"userProperties\": [],\n \"typeProperties\": {\n \"variableName\": \"ExecutionId\",\n \"value\": {\n \"value\": \"@activity('Execution Wrapper').output.firstRow.ExecutionId\",\n \"type\": \"Expression\"\n }\n }\n },\n {\n \"name\": \"Is Parent Already Running\",\n \"description\": \"Establish before anything else if the parent pipeline is already running. Batch execution aware.\",\n \"type\": \"ExecutePipeline\",\n \"dependsOn\": [],\n \"userProperties\": [],\n \"typeProperties\": {\n \"pipeline\": {\n \"referenceName\": \"Check For Running Pipeline\",\n \"type\": \"PipelineReference\"\n },\n \"waitOnCompletion\": true,\n \"parameters\": {\n \"BatchName\": {\n \"value\": \"@pipeline().parameters.BatchName\",\n \"type\": \"Expression\"\n },\n \"PipelineName\": {\n \"value\": \"@pipeline().Pipeline\",\n \"type\": \"Expression\"\n },\n \"ThisRunId\": {\n \"value\": \"@pipeline().RunId\",\n \"type\": \"Expression\"\n }\n }\n }\n },\n {\n \"name\": \"Check Metadata Integrity\",\n \"description\": \"Performs a series of checks on all metadata held in the framework SQLDB. This is intended to raise errors before an execution run even starts.\",\n \"type\": \"SqlServerStoredProcedure\",\n \"dependsOn\": [\n {\n \"activity\": \"Execute Precursor\",\n \"dependencyConditions\": [\n \"Succeeded\"\n ]\n }\n ],\n \"policy\": {\n \"timeout\": \"0.00:10:00\",\n \"retry\": 0,\n \"retryIntervalInSeconds\": 30,\n \"secureOutput\": false,\n \"secureInput\": false\n },\n \"userProperties\": [],\n \"typeProperties\": {\n \"storedProcedureName\": \"[[procfwk].[CheckMetadataIntegrity]\",\n \"storedProcedureParameters\": {\n \"BatchName\": {\n \"value\": {\n \"value\": \"@pipeline().parameters.BatchName\",\n \"type\": \"Expression\"\n },\n \"type\": \"String\"\n },\n \"DebugMode\": {\n \"value\": {\n \"value\": \"@bool(0)\",\n \"type\": \"Expression\"\n },\n \"type\": \"Boolean\"\n }\n }\n },\n \"linkedServiceName\": {\n \"referenceName\": \"SupportDatabase\",\n \"type\": \"LinkedServiceReference\"\n }\n }\n ],\n \"parameters\": {\n \"BatchName\": {\n \"type\": \"string\",\n \"defaultValue\": \"NotUsed\"\n }\n },\n \"variables\": {\n \"ExecutionId\": {\n \"type\": \"String\"\n }\n },\n \"folder\": {\n \"name\": \"_ProcFwk\"\n },\n \"annotations\": [\n \"procfwk\",\n \"Parent\"\n ]\n },\n \"dependsOn\": [\n \"[concat(variables('factoryId'), '/datasets/GetSetMetadata')]\",\n \"[concat(variables('factoryId'), '/linkedServices/SupportDatabase')]\",\n \"[concat(variables('factoryId'), '/pipelines/Check For Running Pipeline')]\",\n \"[concat(variables('factoryId'), '/pipelines/03-Child')]\",\n \"[concat(variables('factoryId'), '/linkedServices/FrameworkFunctions')]\"\n ]\n },\n {\n \"name\": \"[concat(parameters('factoryName'), '/03-Child')]\",\n \"type\": \"Microsoft.DataFactory/factories/pipelines\",\n \"apiVersion\": \"2018-06-01\",\n \"properties\": {\n \"description\": \"ADF.procfwk child pipeline used to execute Worker pipelines within a given execution stage. This pipeline will be called once for each stage, then execute all Workers in parallel.\",\n \"activities\": [\n {\n \"name\": \"Get Pipelines\",\n \"description\": \"Returns all pipelines from the metadata to be executed within a given processing stage.\",\n \"type\": \"Lookup\",\n \"dependsOn\": [],\n \"policy\": {\n \"timeout\": \"0.00:10:00\",\n \"retry\": 0,\n \"retryIntervalInSeconds\": 30,\n \"secureOutput\": false,\n \"secureInput\": false\n },\n \"userProperties\": [],\n \"typeProperties\": {\n \"source\": {\n \"type\": \"AzureSqlSource\",\n \"sqlReaderStoredProcedureName\": \"[[procfwk].[GetPipelinesInStage]\",\n \"storedProcedureParameters\": {\n \"ExecutionId\": {\n \"type\": \"Guid\",\n \"value\": {\n \"value\": \"@pipeline().parameters.ExecutionId\",\n \"type\": \"Expression\"\n }\n },\n \"StageId\": {\n \"type\": \"Int32\",\n \"value\": {\n \"value\": \"@pipeline().parameters.StageId\",\n \"type\": \"Expression\"\n }\n }\n },\n \"queryTimeout\": \"02:00:00\",\n \"partitionOption\": \"None\"\n },\n \"dataset\": {\n \"referenceName\": \"GetSetMetadata\",\n \"type\": \"DatasetReference\",\n \"parameters\": {}\n },\n \"firstRowOnly\": false\n }\n },\n {\n \"name\": \"Execute Pipelines\",\n \"description\": \"Second level ForEach to run in parallel all pipelines within the stage. Items for iteration passed from the Get Pipelines lookup activity.\",\n \"type\": \"ForEach\",\n \"dependsOn\": [\n {\n \"activity\": \"Get Pipelines\",\n \"dependencyConditions\": [\n \"Succeeded\"\n ]\n }\n ],\n \"userProperties\": [],\n \"typeProperties\": {\n \"items\": {\n \"value\": \"@activity('Get Pipelines').output.value\",\n \"type\": \"Expression\"\n },\n \"isSequential\": false,\n \"batchCount\": 50,\n \"activities\": [\n {\n \"name\": \"Worker Pipeline Executor\",\n \"description\": \"Run the required worker pipeline and wait for its completion. Update metadata once done.\",\n \"type\": \"ExecutePipeline\",\n \"dependsOn\": [],\n \"userProperties\": [],\n \"typeProperties\": {\n \"pipeline\": {\n \"referenceName\": \"04-Infant\",\n \"type\": \"PipelineReference\"\n },\n \"waitOnCompletion\": true,\n \"parameters\": {\n \"executionId\": {\n \"value\": \"@pipeline().parameters.ExecutionId\",\n \"type\": \"Expression\"\n },\n \"stageId\": {\n \"value\": \"@pipeline().parameters.StageId\",\n \"type\": \"Expression\"\n },\n \"pipelineId\": {\n \"value\": \"@item().PipelineId\",\n \"type\": \"Expression\"\n }\n }\n }\n }\n ]\n }\n }\n ],\n \"parameters\": {\n \"StageId\": {\n \"type\": \"int\"\n },\n \"ExecutionId\": {\n \"type\": \"string\"\n }\n },\n \"folder\": {\n \"name\": \"_ProcFwk\"\n },\n \"annotations\": [\n \"procfwk\",\n \"Child\"\n ]\n },\n \"dependsOn\": [\n \"[concat(variables('factoryId'), '/datasets/GetSetMetadata')]\",\n \"[concat(variables('factoryId'), '/pipelines/04-Infant')]\"\n ]\n },\n {\n \"name\": \"[concat(parameters('factoryName'), '/04-Infant')]\",\n \"type\": \"Microsoft.DataFactory/factories/pipelines\",\n \"apiVersion\": \"2018-06-01\",\n \"properties\": {\n \"description\": \"ADF.procfwk infant pipeline used to check when the processing pipeline called by the Child completes and passes the resulting status back to the metadata database.\",\n \"activities\": [\n {\n \"name\": \"Execute Worker Pipeline\",\n \"description\": \"The lowest level executor with the metadata framework to call existing processing pipelines within Data Factory. The function called will block processing and wait for an outcome.\",\n \"type\": \"AzureFunctionActivity\",\n \"dependsOn\": [\n {\n \"activity\": \"Log Pipeline Running\",\n \"dependencyConditions\": [\n \"Succeeded\"\n ]\n },\n {\n \"activity\": \"Get Pipeline Params\",\n \"dependencyConditions\": [\n \"Succeeded\"\n ]\n }\n ],\n \"policy\": {\n \"timeout\": \"0.00:10:00\",\n \"retry\": 0,\n \"retryIntervalInSeconds\": 30,\n \"secureOutput\": false,\n \"secureInput\": true\n },\n \"userProperties\": [],\n \"typeProperties\": {\n \"functionName\": \"ExecutePipeline\",\n \"method\": \"POST\",\n \"headers\": {},\n \"body\": {\n \"value\": \"@concat('\\n{\\n\\t\\\"tenantId\\\": \\\"',variables('WorkerTenantId'),'\\\",\\n\\t\\\"applicationId\\\": \\\"',variables('WorkerAppId'),'\\\",\\n\\t\\\"authenticationKey\\\": \\\"',variables('WorkerAppSecret'),'\\\",\\n\\t\\\"subscriptionId\\\": \\\"',variables('WorkerSubscriptionId'),'\\\",\\n\\t\\\"resourceGroup\\\": \\\"',variables('WorkerResourceGroup'),'\\\",\\n\\t\\\"factoryName\\\": \\\"',variables('WorkerDataFactoryName'),'\\\",\\n\\t\\\"pipelineName\\\": \\\"',variables('WorkerPipelineName'),'\\\"',activity('Get Pipeline Params').output.firstRow.Params,'\\n}')\",\n \"type\": \"Expression\"\n }\n },\n \"linkedServiceName\": {\n \"referenceName\": \"FrameworkFunctions\",\n \"type\": \"LinkedServiceReference\"\n }\n },\n {\n \"name\": \"Get Pipeline Params\",\n \"description\": \"Returns any parameters from metadata required for the processing pipeline being called. The output can be an empty string if no parameters are required.\",\n \"type\": \"Lookup\",\n \"dependsOn\": [],\n \"policy\": {\n \"timeout\": \"0.00:10:00\",\n \"retry\": 0,\n \"retryIntervalInSeconds\": 30,\n \"secureOutput\": false,\n \"secureInput\": false\n },\n \"userProperties\": [],\n \"typeProperties\": {\n \"source\": {\n \"type\": \"AzureSqlSource\",\n \"sqlReaderStoredProcedureName\": \"[[procfwk].[GetPipelineParameters]\",\n \"storedProcedureParameters\": {\n \"PipelineId\": {\n \"type\": \"Int32\",\n \"value\": {\n \"value\": \"@pipeline().parameters.pipelineId\",\n \"type\": \"Expression\"\n }\n }\n },\n \"queryTimeout\": \"02:00:00\",\n \"partitionOption\": \"None\"\n },\n \"dataset\": {\n \"referenceName\": \"GetSetMetadata\",\n \"type\": \"DatasetReference\",\n \"parameters\": {}\n }\n }\n },\n {\n \"name\": \"Log Pipeline Running\",\n \"description\": \"Sets the current pipeline with a status of running within the current execution database table.\",\n \"type\": \"SqlServerStoredProcedure\",\n \"dependsOn\": [\n {\n \"activity\": \"Is Target Worker Validate\",\n \"dependencyConditions\": [\n \"Succeeded\"\n ]\n }\n ],\n \"policy\": {\n \"timeout\": \"0.00:10:00\",\n \"retry\": 0,\n \"retryIntervalInSeconds\": 30,\n \"secureOutput\": false,\n \"secureInput\": false\n },\n \"userProperties\": [],\n \"typeProperties\": {\n \"storedProcedureName\": \"[[procfwk].[SetLogPipelineRunning]\",\n \"storedProcedureParameters\": {\n \"ExecutionId\": {\n \"value\": {\n \"value\": \"@pipeline().parameters.ExecutionId\",\n \"type\": \"Expression\"\n },\n \"type\": \"Guid\"\n },\n \"PipelineId\": {\n \"value\": {\n \"value\": \"@pipeline().parameters.pipelineId\",\n \"type\": \"Expression\"\n },\n \"type\": \"Int32\"\n },\n \"StageId\": {\n \"value\": {\n \"value\": \"@pipeline().parameters.StageId\",\n \"type\": \"Expression\"\n },\n \"type\": \"Int32\"\n }\n }\n },\n \"linkedServiceName\": {\n \"referenceName\": \"SupportDatabase\",\n \"type\": \"LinkedServiceReference\"\n }\n },\n {\n \"name\": \"Get Worker Authentication Details\",\n \"description\": \"Return the SPN ID and Secret for the worker pipeline being executed. Called at this level as each pipeline can have a different SPN.\",\n \"type\": \"Lookup\",\n \"dependsOn\": [],\n \"policy\": {\n \"timeout\": \"0.00:10:00\",\n \"retry\": 0,\n \"retryIntervalInSeconds\": 30,\n \"secureOutput\": true,\n \"secureInput\": false\n },\n \"userProperties\": [],\n \"typeProperties\": {\n \"source\": {\n \"type\": \"AzureSqlSource\",\n \"sqlReaderStoredProcedureName\": \"[[procfwk].[GetWorkerAuthDetails]\",\n \"storedProcedureParameters\": {\n \"ExecutionId\": {\n \"type\": \"Guid\",\n \"value\": {\n \"value\": \"@pipeline().parameters.executionId\",\n \"type\": \"Expression\"\n }\n },\n \"PipelineId\": {\n \"type\": \"Int32\",\n \"value\": {\n \"value\": \"@pipeline().parameters.pipelineId\",\n \"type\": \"Expression\"\n }\n },\n \"StageId\": {\n \"type\": \"Int32\",\n \"value\": {\n \"value\": \"@pipeline().parameters.stageId\",\n \"type\": \"Expression\"\n }\n }\n },\n \"queryTimeout\": \"02:00:00\",\n \"partitionOption\": \"None\"\n },\n \"dataset\": {\n \"referenceName\": \"GetSetMetadata\",\n \"type\": \"DatasetReference\",\n \"parameters\": {}\n }\n }\n },\n {\n \"name\": \"Log Execute Function Activity Failure\",\n \"description\": \"Handle true failures from calling out to the Azure Function and update the current execution table accordingly so a restart can occur.\",\n \"type\": \"SqlServerStoredProcedure\",\n \"dependsOn\": [\n {\n \"activity\": \"Execute Worker Pipeline\",\n \"dependencyConditions\": [\n \"Failed\"\n ]\n }\n ],\n \"policy\": {\n \"timeout\": \"0.00:10:00\",\n \"retry\": 0,\n \"retryIntervalInSeconds\": 30,\n \"secureOutput\": false,\n \"secureInput\": false\n },\n \"userProperties\": [],\n \"typeProperties\": {\n \"storedProcedureName\": \"[[procfwk].[SetLogActivityFailed]\",\n \"storedProcedureParameters\": {\n \"ExecutionId\": {\n \"value\": {\n \"value\": \"@pipeline().parameters.ExecutionId\",\n \"type\": \"Expression\"\n },\n \"type\": \"Guid\"\n },\n \"PipelineId\": {\n \"value\": {\n \"value\": \"@pipeline().parameters.pipelineId\",\n \"type\": \"Expression\"\n },\n \"type\": \"Int32\"\n },\n \"StageId\": {\n \"value\": {\n \"value\": \"@pipeline().parameters.StageId\",\n \"type\": \"Expression\"\n },\n \"type\": \"Int32\"\n },\n \"CallingActivity\": {\n \"value\": \"ExecuteWorkerPipeline\",\n \"type\": \"String\"\n }\n }\n },\n \"linkedServiceName\": {\n \"referenceName\": \"SupportDatabase\",\n \"type\": \"LinkedServiceReference\"\n }\n },\n {\n \"name\": \"Update Run Id\",\n \"description\": \"Provide the actual ADF run ID back to the current execution table for long term logging and alignment between the metadata other Azure monitoring tools.\",\n \"type\": \"SqlServerStoredProcedure\",\n \"dependsOn\": [\n {\n \"activity\": \"Set Run Id\",\n \"dependencyConditions\": [\n \"Succeeded\"\n ]\n }\n ],\n \"policy\": {\n \"timeout\": \"0.00:10:00\",\n \"retry\": 0,\n \"retryIntervalInSeconds\": 30,\n \"secureOutput\": false,\n \"secureInput\": false\n },\n \"userProperties\": [],\n \"typeProperties\": {\n \"storedProcedureName\": \"[[procfwk].[SetLogPipelineRunId]\",\n \"storedProcedureParameters\": {\n \"ExecutionId\": {\n \"value\": {\n \"value\": \"@pipeline().parameters.ExecutionId\",\n \"type\": \"Expression\"\n },\n \"type\": \"Guid\"\n },\n \"PipelineId\": {\n \"value\": {\n \"value\": \"@pipeline().parameters.pipelineId\",\n \"type\": \"Expression\"\n },\n \"type\": \"Int32\"\n },\n \"RunId\": {\n \"value\": {\n \"value\": \"@variables('WorkerRunId')\",\n \"type\": \"Expression\"\n },\n \"type\": \"Guid\"\n },\n \"StageId\": {\n \"value\": {\n \"value\": \"@pipeline().parameters.StageId\",\n \"type\": \"Expression\"\n },\n \"type\": \"Int32\"\n }\n }\n },\n \"linkedServiceName\": {\n \"referenceName\": \"SupportDatabase\",\n \"type\": \"LinkedServiceReference\"\n }\n },\n {\n \"name\": \"Check For Alerts\",\n \"description\": \"Checks the properties tables and if any recipients in the database require alerts sending for the current pipeline ID.\",\n \"type\": \"Lookup\",\n \"dependsOn\": [\n {\n \"activity\": \"Update Run Id\",\n \"dependencyConditions\": [\n \"Succeeded\"\n ]\n },\n {\n \"activity\": \"Set Pipeline Result\",\n \"dependencyConditions\": [\n \"Completed\"\n ]\n }\n ],\n \"policy\": {\n \"timeout\": \"0.00:10:00\",\n \"retry\": 0,\n \"retryIntervalInSeconds\": 30,\n \"secureOutput\": false,\n \"secureInput\": false\n },\n \"userProperties\": [],\n \"typeProperties\": {\n \"source\": {\n \"type\": \"AzureSqlSource\",\n \"sqlReaderStoredProcedureName\": \"[[procfwk].[CheckForEmailAlerts]\",\n \"storedProcedureParameters\": {\n \"PipelineId\": {\n \"type\": \"Int32\",\n \"value\": {\n \"value\": \"@pipeline().parameters.pipelineId\",\n \"type\": \"Expression\"\n }\n }\n },\n \"queryTimeout\": \"02:00:00\",\n \"partitionOption\": \"None\"\n },\n \"dataset\": {\n \"referenceName\": \"GetSetMetadata\",\n \"type\": \"DatasetReference\",\n \"parameters\": {}\n },\n \"firstRowOnly\": true\n }\n },\n {\n \"name\": \"Send Alerts\",\n \"description\": \"True = alerts need sending.\\nFalse = do nothing.\",\n \"type\": \"IfCondition\",\n \"dependsOn\": [\n {\n \"activity\": \"Check For Alerts\",\n \"dependencyConditions\": [\n \"Succeeded\"\n ]\n }\n ],\n \"userProperties\": [],\n \"typeProperties\": {\n \"expression\": {\n \"value\": \"@activity('Check For Alerts').output.firstRow.SendAlerts\",\n \"type\": \"Expression\"\n },\n \"ifTrueActivities\": [\n {\n \"name\": \"Get Email Parts\",\n \"description\": \"Return all required content from the metadata database to send an email alerting using the procfwk. The lookup returns the exact content for the function body request.\",\n \"type\": \"Lookup\",\n \"dependsOn\": [],\n \"policy\": {\n \"timeout\": \"0.00:10:00\",\n \"retry\": 0,\n \"retryIntervalInSeconds\": 30,\n \"secureOutput\": true,\n \"secureInput\": false\n },\n \"userProperties\": [],\n \"typeProperties\": {\n \"source\": {\n \"type\": \"AzureSqlSource\",\n \"sqlReaderStoredProcedureName\": \"[[procfwk].[GetEmailAlertParts]\",\n \"storedProcedureParameters\": {\n \"PipelineId\": {\n \"type\": \"Int32\",\n \"value\": {\n \"value\": \"@pipeline().parameters.pipelineId\",\n \"type\": \"Expression\"\n }\n }\n },\n \"queryTimeout\": \"02:00:00\",\n \"partitionOption\": \"None\"\n },\n \"dataset\": {\n \"referenceName\": \"GetSetMetadata\",\n \"type\": \"DatasetReference\",\n \"parameters\": {}\n },\n \"firstRowOnly\": true\n }\n },\n {\n \"name\": \"Send Email\",\n \"description\": \"Use an Azure Function to perform an SMTP client email send operation.\",\n \"type\": \"AzureFunctionActivity\",\n \"dependsOn\": [\n {\n \"activity\": \"Get Email Parts\",\n \"dependencyConditions\": [\n \"Succeeded\"\n ]\n }\n ],\n \"policy\": {\n \"timeout\": \"0.00:10:00\",\n \"retry\": 0,\n \"retryIntervalInSeconds\": 30,\n \"secureOutput\": false,\n \"secureInput\": true\n },\n \"userProperties\": [],\n \"typeProperties\": {\n \"functionName\": \"SendEmail\",\n \"method\": \"POST\",\n \"headers\": {},\n \"body\": {\n \"value\": \"@activity('Get Email Parts').output.firstRow\",\n \"type\": \"Expression\"\n }\n },\n \"linkedServiceName\": {\n \"referenceName\": \"FrameworkFunctions\",\n \"type\": \"LinkedServiceReference\"\n }\n }\n ]\n }\n },\n {\n \"name\": \"Wait Until Pipeline Completes\",\n \"description\": \"Loops until the Worker pipeline called completes.\\n\\nSimple status:\\n- Running = new iteration.\\n- Done = break.\",\n \"type\": \"Until\",\n \"dependsOn\": [\n {\n \"activity\": \"Get Wait Duration\",\n \"dependencyConditions\": [\n \"Succeeded\"\n ]\n },\n {\n \"activity\": \"Execute Worker Pipeline\",\n \"dependencyConditions\": [\n \"Succeeded\"\n ]\n },\n {\n \"activity\": \"Set Run Id\",\n \"dependencyConditions\": [\n \"Succeeded\"\n ]\n }\n ],\n \"userProperties\": [],\n \"typeProperties\": {\n \"expression\": {\n \"value\": \"@variables('WorkerPipelineState')\",\n \"type\": \"Expression\"\n },\n \"activities\": [\n {\n \"name\": \"Get Worker Pipeline Status\",\n \"description\": \"Checks the status of a given processing pipeline and provides the value for the downstream framework activities to act upon.\",\n \"type\": \"AzureFunctionActivity\",\n \"dependsOn\": [],\n \"policy\": {\n \"timeout\": \"0.00:10:00\",\n \"retry\": 0,\n \"retryIntervalInSeconds\": 30,\n \"secureOutput\": false,\n \"secureInput\": true\n },\n \"userProperties\": [],\n \"typeProperties\": {\n \"functionName\": \"CheckPipelineStatus\",\n \"method\": \"POST\",\n \"headers\": {},\n \"body\": {\n \"value\": \"@concat('\\n{\\n \\\"tenantId\\\": \\\"',variables('WorkerTenantId'),'\\\",\\n \\\"applicationId\\\": \\\"',variables('WorkerAppId'),'\\\",\\n \\\"authenticationKey\\\": \\\"',variables('WorkerAppSecret'),'\\\",\\n \\\"subscriptionId\\\": \\\"',variables('WorkerSubscriptionId'),'\\\",\\n \\\"resourceGroup\\\": \\\"',variables('WorkerResourceGroup'),'\\\",\\n \\\"factoryName\\\": \\\"',variables('WorkerDataFactoryName'),'\\\",\\n \\\"pipelineName\\\": \\\"',variables('WorkerPipelineName'),'\\\",\\n \\\"runId\\\": \\\"',variables('WorkerRunId'),'\\\"\\n}')\",\n \"type\": \"Expression\"\n }\n },\n \"linkedServiceName\": {\n \"referenceName\": \"FrameworkFunctions\",\n \"type\": \"LinkedServiceReference\"\n }\n },\n {\n \"name\": \"Wait If Running\",\n \"description\": \"True = Do nothing.\\nFalse = Wait, before the next iteration.\",\n \"type\": \"IfCondition\",\n \"dependsOn\": [\n {\n \"activity\": \"Set Worker State\",\n \"dependencyConditions\": [\n \"Succeeded\"\n ]\n }\n ],\n \"userProperties\": [],\n \"typeProperties\": {\n \"expression\": {\n \"value\": \"@variables('WorkerPipelineState')\",\n \"type\": \"Expression\"\n },\n \"ifFalseActivities\": [\n {\n \"name\": \"Wait for Pipeline\",\n \"description\": \"The processing pipeline is still running so Wait before checking its status again.\",\n \"type\": \"Wait\",\n \"dependsOn\": [],\n \"userProperties\": [],\n \"typeProperties\": {\n \"waitTimeInSeconds\": {\n \"value\": \"@activity('Get Wait Duration').output.firstRow.PropertyValue\",\n \"type\": \"Expression\"\n }\n }\n }\n ]\n }\n },\n {\n \"name\": \"Set Last Check DateTime\",\n \"description\": \"Update the current execution table with a date time from when the Worker pipeline status was last checked as part of the Until iterations.\",\n \"type\": \"SqlServerStoredProcedure\",\n \"dependsOn\": [\n {\n \"activity\": \"Get Worker Pipeline Status\",\n \"dependencyConditions\": [\n \"Succeeded\"\n ]\n }\n ],\n \"policy\": {\n \"timeout\": \"0.00:10:00\",\n \"retry\": 0,\n \"retryIntervalInSeconds\": 30,\n \"secureOutput\": false,\n \"secureInput\": false\n },\n \"userProperties\": [],\n \"typeProperties\": {\n \"storedProcedureName\": \"[[procfwk].[SetLogPipelineLastStatusCheck]\",\n \"storedProcedureParameters\": {\n \"ExecutionId\": {\n \"value\": {\n \"value\": \"@pipeline().parameters.executionId\",\n \"type\": \"Expression\"\n },\n \"type\": \"Guid\"\n },\n \"PipelineId\": {\n \"value\": {\n \"value\": \"@pipeline().parameters.pipelineId\",\n \"type\": \"Expression\"\n },\n \"type\": \"Int32\"\n },\n \"StageId\": {\n \"value\": {\n \"value\": \"@pipeline().parameters.stageId\",\n \"type\": \"Expression\"\n },\n \"type\": \"Int32\"\n }\n }\n },\n \"linkedServiceName\": {\n \"referenceName\": \"SupportDatabase\",\n \"type\": \"LinkedServiceReference\"\n }\n },\n {\n \"name\": \"Log Check Function Activity Failure\",\n \"description\": \"Report to the current execution table that the framework pipeline activity has failed. This failure is outside of the scope of the framework and is probably related to a wider platform problem.\",\n \"type\": \"SqlServerStoredProcedure\",\n \"dependsOn\": [\n {\n \"activity\": \"Get Worker Pipeline Status\",\n \"dependencyConditions\": [\n \"Failed\"\n ]\n }\n ],\n \"policy\": {\n \"timeout\": \"0.00:10:00\",\n \"retry\": 0,\n \"retryIntervalInSeconds\": 30,\n \"secureOutput\": false,\n \"secureInput\": false\n },\n \"userProperties\": [],\n \"typeProperties\": {\n \"storedProcedureName\": \"[[procfwk].[SetLogActivityFailed]\",\n \"storedProcedureParameters\": {\n \"CallingActivity\": {\n \"value\": \"GetWorkerPipelineStatus\",\n \"type\": \"String\"\n },\n \"ExecutionId\": {\n \"value\": {\n \"value\": \"@pipeline().parameters.executionId\",\n \"type\": \"Expression\"\n },\n \"type\": \"Guid\"\n },\n \"PipelineId\": {\n \"value\": {\n \"value\": \"@pipeline().parameters.pipelineId\",\n \"type\": \"Expression\"\n },\n \"type\": \"Int32\"\n },\n \"StageId\": {\n \"value\": {\n \"value\": \"@pipeline().parameters.stageId\",\n \"type\": \"Expression\"\n },\n \"type\": \"Int32\"\n }\n }\n },\n \"linkedServiceName\": {\n \"referenceName\": \"SupportDatabase\",\n \"type\": \"LinkedServiceReference\"\n }\n },\n {\n \"name\": \"Set Worker State\",\n \"description\": \"Set the bool state of the Worker pipeline to be used by the Until and If expressions. True = Complete, False = Running.\",\n \"type\": \"SetVariable\",\n \"dependsOn\": [\n {\n \"activity\": \"Get Worker Pipeline Status\",\n \"dependencyConditions\": [\n \"Succeeded\"\n ]\n }\n ],\n \"userProperties\": [],\n \"typeProperties\": {\n \"variableName\": \"WorkerPipelineState\",\n \"value\": {\n \"value\": \"@equals('Done',activity('Get Worker Pipeline Status').output.SimpleStatus)\",\n \"type\": \"Expression\"\n }\n }\n }\n ],\n \"timeout\": \"0.00:10:00\"\n }\n },\n {\n \"name\": \"Set Pipeline Result\",\n \"description\": \"Receives the outcome from the function execution for a given processing pipeline and updates the current execution table with different pipelines status values depending on the result (case).\",\n \"type\": \"Switch\",\n \"dependsOn\": [\n {\n \"activity\": \"Wait Until Pipeline Completes\",\n \"dependencyConditions\": [\n \"Completed\"\n ]\n }\n ],\n \"userProperties\": [],\n \"typeProperties\": {\n \"on\": {\n \"value\": \"@activity('Get Worker Pipeline Status').output.Status\",\n \"type\": \"Expression\"\n },\n \"cases\": [\n {\n \"value\": \"Succeeded\",\n \"activities\": [\n {\n \"name\": \"Pipeline Status Succeeded\",\n \"description\": \"Updates the current execution table with a pipeline status of success if the function outcome is succeeded.\",\n \"type\": \"SqlServerStoredProcedure\",\n \"dependsOn\": [],\n \"policy\": {\n \"timeout\": \"0.00:10:00\",\n \"retry\": 0,\n \"retryIntervalInSeconds\": 30,\n \"secureOutput\": false,\n \"secureInput\": false\n },\n \"userProperties\": [],\n \"typeProperties\": {\n \"storedProcedureName\": \"[[procfwk].[SetLogPipelineSuccess]\",\n \"storedProcedureParameters\": {\n \"ExecutionId\": {\n \"value\": {\n \"value\": \"@pipeline().parameters.executionId\",\n \"type\": \"Expression\"\n },\n \"type\": \"Guid\"\n },\n \"PipelineId\": {\n \"value\": {\n \"value\": \"@pipeline().parameters.pipelineId\",\n \"type\": \"Expression\"\n },\n \"type\": \"Int32\"\n },\n \"StageId\": {\n \"value\": {\n \"value\": \"@pipeline().parameters.stageId\",\n \"type\": \"Expression\"\n },\n \"type\": \"Int32\"\n }\n }\n },\n \"linkedServiceName\": {\n \"referenceName\": \"SupportDatabase\",\n \"type\": \"LinkedServiceReference\"\n }\n }\n ]\n },\n {\n \"value\": \"Failed\",\n \"activities\": [\n {\n \"name\": \"Pipeline Status Failed\",\n \"description\": \"Updates the current execution table with a pipeline status of failed if the function outcome is failed. Also blocks pipelines in the downstream execution stage.\",\n \"type\": \"SqlServerStoredProcedure\",\n \"dependsOn\": [],\n \"policy\": {\n \"timeout\": \"0.00:10:00\",\n \"retry\": 0,\n \"retryIntervalInSeconds\": 30,\n \"secureOutput\": false,\n \"secureInput\": false\n },\n \"userProperties\": [],\n \"typeProperties\": {\n \"storedProcedureName\": \"[[procfwk].[SetLogPipelineFailed]\",\n \"storedProcedureParameters\": {\n \"ExecutionId\": {\n \"value\": {\n \"value\": \"@pipeline().parameters.executionId\",\n \"type\": \"Expression\"\n },\n \"type\": \"Guid\"\n },\n \"PipelineId\": {\n \"value\": {\n \"value\": \"@pipeline().parameters.pipelineId\",\n \"type\": \"Expression\"\n },\n \"type\": \"Int32\"\n },\n \"RunId\": {\n \"value\": {\n \"value\": \"@variables('WorkerRunId')\",\n \"type\": \"Expression\"\n },\n \"type\": \"Guid\"\n },\n \"StageId\": {\n \"value\": {\n \"value\": \"@pipeline().parameters.stageId\",\n \"type\": \"Expression\"\n },\n \"type\": \"Int32\"\n }\n }\n },\n \"linkedServiceName\": {\n \"referenceName\": \"SupportDatabase\",\n \"type\": \"LinkedServiceReference\"\n }\n },\n {\n \"name\": \"Get Worker Pipeline Error Details\",\n \"description\": \"Get the activity error details for the run ID of the worker pipeline called. Returns an array of all errors.\",\n \"type\": \"AzureFunctionActivity\",\n \"dependsOn\": [],\n \"policy\": {\n \"timeout\": \"0.00:10:00\",\n \"retry\": 0,\n \"retryIntervalInSeconds\": 30,\n \"secureOutput\": false,\n \"secureInput\": true\n },\n \"userProperties\": [],\n \"typeProperties\": {\n \"functionName\": \"GetActivityErrors\",\n \"method\": \"POST\",\n \"headers\": {},\n \"body\": {\n \"value\": \"@concat('\\n{\\n \\\"tenantId\\\": \\\"',variables('WorkerTenantId'),'\\\",\\n \\\"applicationId\\\": \\\"',variables('WorkerAppId'),'\\\",\\n \\\"authenticationKey\\\": \\\"',variables('WorkerAppSecret'),'\\\",\\n \\\"subscriptionId\\\": \\\"',variables('WorkerSubscriptionId'),'\\\",\\n \\\"resourceGroup\\\": \\\"',variables('WorkerResourceGroup'),'\\\",\\n \\\"factoryName\\\": \\\"',variables('WorkerDataFactoryName'),'\\\",\\n \\\"pipelineName\\\": \\\"',variables('WorkerPipelineName'),'\\\",\\n \\\"runId\\\": \\\"',variables('WorkerRunId'),'\\\"\\n}')\",\n \"type\": \"Expression\"\n }\n },\n \"linkedServiceName\": {\n \"referenceName\": \"FrameworkFunctions\",\n \"type\": \"LinkedServiceReference\"\n }\n },\n {\n \"name\": \"Log Error Details\",\n \"description\": \"Parses pipeline error details and persists them to the metadata database error log table.\",\n \"type\": \"SqlServerStoredProcedure\",\n \"dependsOn\": [\n {\n \"activity\": \"Get Worker Pipeline Error Details\",\n \"dependencyConditions\": [\n \"Succeeded\"\n ]\n }\n ],\n \"policy\": {\n \"timeout\": \"0.00:10:00\",\n \"retry\": 0,\n \"retryIntervalInSeconds\": 30,\n \"secureOutput\": false,\n \"secureInput\": false\n },\n \"userProperties\": [],\n \"typeProperties\": {\n \"storedProcedureName\": \"[[procfwk].[SetErrorLogDetails]\",\n \"storedProcedureParameters\": {\n \"JsonErrorDetails\": {\n \"value\": {\n \"value\": \"@string(activity('Get Worker Pipeline Error Details').output)\",\n \"type\": \"Expression\"\n },\n \"type\": \"String\"\n },\n \"LocalExecutionId\": {\n \"value\": {\n \"value\": \"@pipeline().parameters.executionId\",\n \"type\": \"Expression\"\n },\n \"type\": \"Guid\"\n }\n }\n },\n \"linkedServiceName\": {\n \"referenceName\": \"SupportDatabase\",\n \"type\": \"LinkedServiceReference\"\n }\n }\n ]\n },\n {\n \"value\": \"Cancelled\",\n \"activities\": [\n {\n \"name\": \"Pipeline Status Cancelled\",\n \"description\": \"Updates the current execution table with a pipeline status of cancelled if the function outcome is cancelled.\",\n \"type\": \"SqlServerStoredProcedure\",\n \"dependsOn\": [],\n \"policy\": {\n \"timeout\": \"0.00:10:00\",\n \"retry\": 0,\n \"retryIntervalInSeconds\": 30,\n \"secureOutput\": false,\n \"secureInput\": false\n },\n \"userProperties\": [],\n \"typeProperties\": {\n \"storedProcedureName\": \"[[procfwk].[SetLogPipelineCancelled]\",\n \"storedProcedureParameters\": {\n \"ExecutionId\": {\n \"value\": {\n \"value\": \"@pipeline().parameters.executionId\",\n \"type\": \"Expression\"\n },\n \"type\": \"Guid\"\n },\n \"PipelineId\": {\n \"value\": {\n \"value\": \"@pipeline().parameters.pipelineId\",\n \"type\": \"Expression\"\n },\n \"type\": \"Int32\"\n },\n \"StageId\": {\n \"value\": {\n \"value\": \"@pipeline().parameters.stageId\",\n \"type\": \"Expression\"\n },\n \"type\": \"Int32\"\n }\n }\n },\n \"linkedServiceName\": {\n \"referenceName\": \"SupportDatabase\",\n \"type\": \"LinkedServiceReference\"\n }\n }\n ]\n }\n ],\n \"defaultActivities\": [\n {\n \"name\": \"Pipeline Status Unknown\",\n \"description\": \"Updates the current execution table with a pipeline status of unknown if the function returns an unexpected outcome.\",\n \"type\": \"SqlServerStoredProcedure\",\n \"dependsOn\": [],\n \"policy\": {\n \"timeout\": \"0.00:10:00\",\n \"retry\": 0,\n \"retryIntervalInSeconds\": 30,\n \"secureOutput\": false,\n \"secureInput\": false\n },\n \"userProperties\": [],\n \"typeProperties\": {\n \"storedProcedureName\": \"[[procfwk].[SetLogPipelineUnknown]\",\n \"storedProcedureParameters\": {\n \"ExecutionId\": {\n \"value\": {\n \"value\": \"@pipeline().parameters.executionId\",\n \"type\": \"Expression\"\n },\n \"type\": \"Guid\"\n },\n \"PipelineId\": {\n \"value\": {\n \"value\": \"@pipeline().parameters.pipelineId\",\n \"type\": \"Expression\"\n },\n \"type\": \"Int32\"\n },\n \"StageId\": {\n \"value\": {\n \"value\": \"@pipeline().parameters.stageId\",\n \"type\": \"Expression\"\n },\n \"type\": \"Int32\"\n }\n }\n },\n \"linkedServiceName\": {\n \"referenceName\": \"SupportDatabase\",\n \"type\": \"LinkedServiceReference\"\n }\n }\n ]\n }\n },\n {\n \"name\": \"Get Wait Duration\",\n \"description\": \"Return wait duration in seconds from database properties table to be used during each Until iteration when the Worker pipeline is still running.\",\n \"type\": \"Lookup\",\n \"dependsOn\": [],\n \"policy\": {\n \"timeout\": \"0.00:10:00\",\n \"retry\": 0,\n \"retryIntervalInSeconds\": 30,\n \"secureOutput\": false,\n \"secureInput\": false\n },\n \"userProperties\": [],\n \"typeProperties\": {\n \"source\": {\n \"type\": \"AzureSqlSource\",\n \"sqlReaderStoredProcedureName\": \"[[procfwk].[GetPropertyValue]\",\n \"storedProcedureParameters\": {\n \"PropertyName\": {\n \"type\": \"String\",\n \"value\": \"PipelineStatusCheckDuration\"\n }\n },\n \"queryTimeout\": \"02:00:00\",\n \"partitionOption\": \"None\"\n },\n \"dataset\": {\n \"referenceName\": \"GetSetMetadata\",\n \"type\": \"DatasetReference\",\n \"parameters\": {}\n }\n }\n },\n {\n \"name\": \"Set App Id\",\n \"description\": \"Set local variable from activity output once for value reuse in downstream activities.\",\n \"type\": \"SetVariable\",\n \"dependsOn\": [\n {\n \"activity\": \"Get Worker Authentication Details\",\n \"dependencyConditions\": [\n \"Succeeded\"\n ]\n }\n ],\n \"userProperties\": [],\n \"typeProperties\": {\n \"variableName\": \"WorkerAppId\",\n \"value\": {\n \"value\": \"@activity('Get Worker Authentication Details').output.firstRow.AppId\",\n \"type\": \"Expression\"\n }\n }\n },\n {\n \"name\": \"Set App Secret\",\n \"description\": \"Set local variable from activity output once for value reuse in downstream activities.\",\n \"type\": \"SetVariable\",\n \"dependsOn\": [\n {\n \"activity\": \"Get Worker Authentication Details\",\n \"dependencyConditions\": [\n \"Succeeded\"\n ]\n }\n ],\n \"userProperties\": [],\n \"typeProperties\": {\n \"variableName\": \"WorkerAppSecret\",\n \"value\": {\n \"value\": \"@activity('Get Worker Authentication Details').output.firstRow.AppSecret\",\n \"type\": \"Expression\"\n }\n }\n },\n {\n \"name\": \"Set Run Id\",\n \"description\": \"Set local variable from activity output once for value reuse in downstream activities.\",\n \"type\": \"SetVariable\",\n \"dependsOn\": [\n {\n \"activity\": \"Execute Worker Pipeline\",\n \"dependencyConditions\": [\n \"Succeeded\"\n ]\n }\n ],\n \"userProperties\": [],\n \"typeProperties\": {\n \"variableName\": \"WorkerRunId\",\n \"value\": {\n \"value\": \"@activity('Execute Worker Pipeline').output.RunId\",\n \"type\": \"Expression\"\n }\n }\n },\n {\n \"name\": \"Get Worker Pipeline Details\",\n \"description\": \"Return none sensitive worker pipeline information for metadata database. Including target data factory, pipeline name and resource group.\",\n \"type\": \"Lookup\",\n \"dependsOn\": [],\n \"policy\": {\n \"timeout\": \"0.00:10:00\",\n \"retry\": 0,\n \"retryIntervalInSeconds\": 30,\n \"secureOutput\": true,\n \"secureInput\": false\n },\n \"userProperties\": [],\n \"typeProperties\": {\n \"source\": {\n \"type\": \"AzureSqlSource\",\n \"sqlReaderStoredProcedureName\": \"[[procfwk].[GetWorkerPipelineDetails]\",\n \"storedProcedureParameters\": {\n \"ExecutionId\": {\n \"type\": \"Guid\",\n \"value\": {\n \"value\": \"@pipeline().parameters.ExecutionId\",\n \"type\": \"Expression\"\n }\n },\n \"PipelineId\": {\n \"type\": \"Int32\",\n \"value\": {\n \"value\": \"@pipeline().parameters.pipelineId\",\n \"type\": \"Expression\"\n }\n },\n \"StageId\": {\n \"type\": \"Int32\",\n \"value\": {\n \"value\": \"@pipeline().parameters.stageId\",\n \"type\": \"Expression\"\n }\n }\n },\n \"queryTimeout\": \"02:00:00\",\n \"partitionOption\": \"None\"\n },\n \"dataset\": {\n \"referenceName\": \"GetSetMetadata\",\n \"type\": \"DatasetReference\",\n \"parameters\": {}\n }\n }\n },\n {\n \"name\": \"Set Tenant Id\",\n \"description\": \"Set local variable from activity output once for value reuse in downstream activities.\",\n \"type\": \"SetVariable\",\n \"dependsOn\": [\n {\n \"activity\": \"Get Worker Authentication Details\",\n \"dependencyConditions\": [\n \"Succeeded\"\n ]\n }\n ],\n \"userProperties\": [],\n \"typeProperties\": {\n \"variableName\": \"WorkerTenantId\",\n \"value\": {\n \"value\": \"@activity('Get Worker Authentication Details').output.firstRow.TenantId\",\n \"type\": \"Expression\"\n }\n }\n },\n {\n \"name\": \"Set Subscription Id\",\n \"description\": \"Set local variable from activity output once for value reuse in downstream activities.\",\n \"type\": \"SetVariable\",\n \"dependsOn\": [\n {\n \"activity\": \"Get Worker Authentication Details\",\n \"dependencyConditions\": [\n \"Succeeded\"\n ]\n }\n ],\n \"userProperties\": [],\n \"typeProperties\": {\n \"variableName\": \"WorkerSubscriptionId\",\n \"value\": {\n \"value\": \"@activity('Get Worker Authentication Details').output.firstRow.SubscriptionId\",\n \"type\": \"Expression\"\n }\n }\n },\n {\n \"name\": \"Set Pipeline Name\",\n \"description\": \"Set local variable from activity output once for value reuse in downstream activities.\",\n \"type\": \"SetVariable\",\n \"dependsOn\": [\n {\n \"activity\": \"Get Worker Pipeline Details\",\n \"dependencyConditions\": [\n \"Succeeded\"\n ]\n }\n ],\n \"userProperties\": [],\n \"typeProperties\": {\n \"variableName\": \"WorkerPipelineName\",\n \"value\": {\n \"value\": \"@activity('Get Worker Pipeline Details').output.firstRow.PipelineName\",\n \"type\": \"Expression\"\n }\n }\n },\n {\n \"name\": \"Set Data Factory Name\",\n \"description\": \"Set local variable from activity output once for value reuse in downstream activities.\",\n \"type\": \"SetVariable\",\n \"dependsOn\": [\n {\n \"activity\": \"Get Worker Pipeline Details\",\n \"dependencyConditions\": [\n \"Succeeded\"\n ]\n }\n ],\n \"userProperties\": [],\n \"typeProperties\": {\n \"variableName\": \"WorkerDataFactoryName\",\n \"value\": {\n \"value\": \"@activity('Get Worker Pipeline Details').output.firstRow.DataFactoryName\",\n \"type\": \"Expression\"\n }\n }\n },\n {\n \"name\": \"Set Resource Group\",\n \"description\": \"Set local variable from activity output once for value reuse in downstream activities.\",\n \"type\": \"SetVariable\",\n \"dependsOn\": [\n {\n \"activity\": \"Get Worker Pipeline Details\",\n \"dependencyConditions\": [\n \"Succeeded\"\n ]\n }\n ],\n \"userProperties\": [],\n \"typeProperties\": {\n \"variableName\": \"WorkerResourceGroup\",\n \"value\": {\n \"value\": \"@activity('Get Worker Pipeline Details').output.firstRow.ResourceGroupName\",\n \"type\": \"Expression\"\n }\n }\n },\n {\n \"name\": \"Validate Pipeline\",\n \"description\": \"Query the target data factory and establish if the provided worker pipeline name is valid.\",\n \"type\": \"AzureFunctionActivity\",\n \"dependsOn\": [\n {\n \"activity\": \"Set Resource Group\",\n \"dependencyConditions\": [\n \"Succeeded\"\n ]\n },\n {\n \"activity\": \"Set Data Factory Name\",\n \"dependencyConditions\": [\n \"Succeeded\"\n ]\n },\n {\n \"activity\": \"Set Subscription Id\",\n \"dependencyConditions\": [\n \"Succeeded\"\n ]\n },\n {\n \"activity\": \"Set App Secret\",\n \"dependencyConditions\": [\n \"Succeeded\"\n ]\n },\n {\n \"activity\": \"Set App Id\",\n \"dependencyConditions\": [\n \"Succeeded\"\n ]\n },\n {\n \"activity\": \"Set Pipeline Name\",\n \"dependencyConditions\": [\n \"Succeeded\"\n ]\n },\n {\n \"activity\": \"Set Tenant Id\",\n \"dependencyConditions\": [\n \"Succeeded\"\n ]\n },\n {\n \"activity\": \"Log Pipeline Validating\",\n \"dependencyConditions\": [\n \"Succeeded\"\n ]\n }\n ],\n \"policy\": {\n \"timeout\": \"0.00:10:00\",\n \"retry\": 0,\n \"retryIntervalInSeconds\": 30,\n \"secureOutput\": false,\n \"secureInput\": true\n },\n \"userProperties\": [],\n \"typeProperties\": {\n \"functionName\": \"ValidatePipeline\",\n \"method\": \"POST\",\n \"headers\": {},\n \"body\": {\n \"value\": \"@concat('\\n{\\n\\t\\\"tenantId\\\": \\\"',variables('WorkerTenantId'),'\\\",\\n\\t\\\"applicationId\\\": \\\"',variables('WorkerAppId'),'\\\",\\n\\t\\\"authenticationKey\\\": \\\"',variables('WorkerAppSecret'),'\\\",\\n\\t\\\"subscriptionId\\\": \\\"',variables('WorkerSubscriptionId'),'\\\",\\n\\t\\\"resourceGroup\\\": \\\"',variables('WorkerResourceGroup'),'\\\",\\n\\t\\\"factoryName\\\": \\\"',variables('WorkerDataFactoryName'),'\\\",\\n\\t\\\"pipelineName\\\": \\\"',variables('WorkerPipelineName'),'\\\"\\n}')\",\n \"type\": \"Expression\"\n }\n },\n \"linkedServiceName\": {\n \"referenceName\": \"FrameworkFunctions\",\n \"type\": \"LinkedServiceReference\"\n }\n },\n {\n \"name\": \"Is Target Worker Validate\",\n \"description\": \"True = the worker pipeline name is valid.\\nFalse = the worker pipeline name is invalid. Raise an exception.\",\n \"type\": \"IfCondition\",\n \"dependsOn\": [\n {\n \"activity\": \"Validate Pipeline\",\n \"dependencyConditions\": [\n \"Succeeded\"\n ]\n }\n ],\n \"userProperties\": [],\n \"typeProperties\": {\n \"expression\": {\n \"value\": \"@bool(activity('Validate Pipeline').output.PipelineExists)\",\n \"type\": \"Expression\"\n },\n \"ifFalseActivities\": [\n {\n \"name\": \"Throw Exception - Invalid Infant\",\n \"description\": \"Throw an exception with details about the invalid worker pipeline name.\",\n \"type\": \"ExecutePipeline\",\n \"dependsOn\": [],\n \"userProperties\": [],\n \"typeProperties\": {\n \"pipeline\": {\n \"referenceName\": \"Throw Exception\",\n \"type\": \"PipelineReference\"\n },\n \"waitOnCompletion\": true,\n \"parameters\": {\n \"Message\": {\n \"value\": \"@concat('Worker pipeline [',variables('WorkerPipelineName'),'] is not valid in target Data Factory [',variables('WorkerDataFactoryName'),']')\",\n \"type\": \"Expression\"\n }\n }\n }\n },\n {\n \"name\": \"Update Execution With Invalid Worker\",\n \"description\": \"Update the current execution table with an informed status for the worker pipeline that couldn't be executed.\",\n \"type\": \"SqlServerStoredProcedure\",\n \"dependsOn\": [],\n \"policy\": {\n \"timeout\": \"7.00:00:00\",\n \"retry\": 0,\n \"retryIntervalInSeconds\": 30,\n \"secureOutput\": false,\n \"secureInput\": false\n },\n \"userProperties\": [],\n \"typeProperties\": {\n \"storedProcedureName\": \"[[procfwk].[SetLogActivityFailed]\",\n \"storedProcedureParameters\": {\n \"CallingActivity\": {\n \"value\": \"InvalidPipelineName\",\n \"type\": \"String\"\n },\n \"ExecutionId\": {\n \"value\": {\n \"value\": \"@pipeline().parameters.ExecutionId\",\n \"type\": \"Expression\"\n },\n \"type\": \"Guid\"\n },\n \"PipelineId\": {\n \"value\": {\n \"value\": \"@pipeline().parameters.pipelineId\",\n \"type\": \"Expression\"\n },\n \"type\": \"Int32\"\n },\n \"StageId\": {\n \"value\": {\n \"value\": \"@pipeline().parameters.StageId\",\n \"type\": \"Expression\"\n },\n \"type\": \"Int32\"\n }\n }\n },\n \"linkedServiceName\": {\n \"referenceName\": \"SupportDatabase\",\n \"type\": \"LinkedServiceReference\"\n }\n }\n ]\n }\n },\n {\n \"name\": \"Log Validate Function Activity Failure\",\n \"description\": \"Handle true failures from calling out to the Azure Function and update the current execution table accordingly so a restart can occur.\",\n \"type\": \"SqlServerStoredProcedure\",\n \"dependsOn\": [\n {\n \"activity\": \"Validate Pipeline\",\n \"dependencyConditions\": [\n \"Failed\"\n ]\n }\n ],\n \"policy\": {\n \"timeout\": \"0.00:10:00\",\n \"retry\": 0,\n \"retryIntervalInSeconds\": 30,\n \"secureOutput\": false,\n \"secureInput\": false\n },\n \"userProperties\": [],\n \"typeProperties\": {\n \"storedProcedureName\": \"[[procfwk].[SetLogActivityFailed]\",\n \"storedProcedureParameters\": {\n \"ExecutionId\": {\n \"value\": {\n \"value\": \"@pipeline().parameters.ExecutionId\",\n \"type\": \"Expression\"\n },\n \"type\": \"Guid\"\n },\n \"PipelineId\": {\n \"value\": {\n \"value\": \"@pipeline().parameters.pipelineId\",\n \"type\": \"Expression\"\n },\n \"type\": \"Int32\"\n },\n \"StageId\": {\n \"value\": {\n \"value\": \"@pipeline().parameters.StageId\",\n \"type\": \"Expression\"\n },\n \"type\": \"Int32\"\n },\n \"CallingActivity\": {\n \"value\": \"ValidatePipeline\",\n \"type\": \"String\"\n }\n }\n },\n \"linkedServiceName\": {\n \"referenceName\": \"SupportDatabase\",\n \"type\": \"LinkedServiceReference\"\n }\n },\n {\n \"name\": \"Log Pipeline Validating\",\n \"description\": \"Sets the current pipeline with a status of validating within the current execution database table.\",\n \"type\": \"SqlServerStoredProcedure\",\n \"dependsOn\": [],\n \"policy\": {\n \"timeout\": \"0.00:10:00\",\n \"retry\": 0,\n \"retryIntervalInSeconds\": 30,\n \"secureOutput\": false,\n \"secureInput\": false\n },\n \"userProperties\": [],\n \"typeProperties\": {\n \"storedProcedureName\": \"[[procfwk].[SetLogPipelineValidating]\",\n \"storedProcedureParameters\": {\n \"ExecutionId\": {\n \"value\": {\n \"value\": \"@pipeline().parameters.ExecutionId\",\n \"type\": \"Expression\"\n },\n \"type\": \"Guid\"\n },\n \"PipelineId\": {\n \"value\": {\n \"value\": \"@pipeline().parameters.pipelineId\",\n \"type\": \"Expression\"\n },\n \"type\": \"Int32\"\n },\n \"StageId\": {\n \"value\": {\n \"value\": \"@pipeline().parameters.StageId\",\n \"type\": \"Expression\"\n },\n \"type\": \"Int32\"\n }\n }\n },\n \"linkedServiceName\": {\n \"referenceName\": \"SupportDatabase\",\n \"type\": \"LinkedServiceReference\"\n }\n }\n ],\n \"parameters\": {\n \"executionId\": {\n \"type\": \"string\"\n },\n \"stageId\": {\n \"type\": \"int\"\n },\n \"pipelineId\": {\n \"type\": \"int\"\n }\n },\n \"variables\": {\n \"WorkerPipelineState\": {\n \"type\": \"Boolean\"\n },\n \"WorkerAppId\": {\n \"type\": \"String\"\n },\n \"WorkerAppSecret\": {\n \"type\": \"String\"\n },\n \"WorkerRunId\": {\n \"type\": \"String\"\n },\n \"WorkerTenantId\": {\n \"type\": \"String\"\n },\n \"WorkerSubscriptionId\": {\n \"type\": \"String\"\n },\n \"WorkerPipelineName\": {\n \"type\": \"String\"\n },\n \"WorkerDataFactoryName\": {\n \"type\": \"String\"\n },\n \"WorkerResourceGroup\": {\n \"type\": \"String\"\n }\n },\n \"folder\": {\n \"name\": \"_ProcFwk\"\n },\n \"annotations\": [\n \"procfwk\",\n \"Infant\"\n ]\n },\n \"dependsOn\": [\n \"[concat(variables('factoryId'), '/linkedServices/FrameworkFunctions')]\",\n \"[concat(variables('factoryId'), '/datasets/GetSetMetadata')]\",\n \"[concat(variables('factoryId'), '/linkedServices/SupportDatabase')]\",\n \"[concat(variables('factoryId'), '/pipelines/Throw Exception')]\"\n ]\n },\n {\n \"name\": \"[concat(parameters('factoryName'), '/Check For Running Pipeline')]\",\n \"type\": \"Microsoft.DataFactory/factories/pipelines\",\n \"apiVersion\": \"2018-06-01\",\n \"properties\": {\n \"description\": \"For a given pipeline and optional batch name establish if a pipeline run is already in progress. Throw an exception if it it.\",\n \"activities\": [\n {\n \"name\": \"Get Subscription\",\n \"description\": \"Use the Azure Management API to return the current subscription.\",\n \"type\": \"WebActivity\",\n \"dependsOn\": [],\n \"policy\": {\n \"timeout\": \"0.00:10:00\",\n \"retry\": 0,\n \"retryIntervalInSeconds\": 30,\n \"secureOutput\": false,\n \"secureInput\": false\n },\n \"userProperties\": [],\n \"typeProperties\": {\n \"url\": \"https://management.azure.com/subscriptions?api-version=2020-01-01\",\n \"method\": \"GET\",\n \"headers\": {},\n \"authentication\": {\n \"type\": \"MSI\",\n \"resource\": \"https://management.core.windows.net/\"\n }\n }\n },\n {\n \"name\": \"Get Pipeline Runs\",\n \"description\": \"Use the Azure Management API to return a list of pipeline runs within the given time window.\",\n \"type\": \"WebActivity\",\n \"dependsOn\": [\n {\n \"activity\": \"Get Query Run Days Value\",\n \"dependencyConditions\": [\n \"Succeeded\"\n ]\n },\n {\n \"activity\": \"Check for Valid Pipeline Name\",\n \"dependencyConditions\": [\n \"Succeeded\"\n ]\n }\n ],\n \"policy\": {\n \"timeout\": \"0.00:10:00\",\n \"retry\": 0,\n \"retryIntervalInSeconds\": 30,\n \"secureOutput\": false,\n \"secureInput\": false\n },\n \"userProperties\": [],\n \"typeProperties\": {\n \"url\": {\n \"value\": \"https://management.azure.com/subscriptions/@{variables('SubscriptionId')}/resourceGroups/@{activity('Get Resource Group').output.firstRow.PropertyValue}/providers/Microsoft.DataFactory/factories/@{pipeline().DataFactory}/queryPipelineRuns?api-version=2018-06-01\",\n \"type\": \"Expression\"\n },\n \"method\": \"POST\",\n \"headers\": {},\n \"body\": {\n \"value\": \"{\\n \\\"lastUpdatedAfter\\\": \\\"@{adddays(utcnow(),int(activity('Get Query Run Days Value').output.firstRow.PropertyValue))}\\\",\\n \\\"lastUpdatedBefore\\\": \\\"@{utcnow()}\\\",\\n \\\"filters\\\": [\\n {\\n \\\"operand\\\": \\\"PipelineName\\\",\\n \\\"operator\\\": \\\"Equals\\\",\\n \\\"values\\\": [\\n \\\"@{pipeline().parameters.PipelineName}\\\"\\n ]\\n }\\n ]\\n}\",\n \"type\": \"Expression\"\n },\n \"authentication\": {\n \"type\": \"MSI\",\n \"resource\": \"https://management.core.windows.net/\"\n }\n }\n },\n {\n \"name\": \"Set Parsed Subscription\",\n \"description\": \"Parse the subscription string value to return just the subscription ID.\",\n \"type\": \"SetVariable\",\n \"dependsOn\": [\n {\n \"activity\": \"Get Subscription\",\n \"dependencyConditions\": [\n \"Succeeded\"\n ]\n }\n ],\n \"userProperties\": [],\n \"typeProperties\": {\n \"variableName\": \"SubscriptionId\",\n \"value\": {\n \"value\": \"@replace(activity('Get Subscription').output.value[0].id,'/subscriptions/','')\",\n \"type\": \"Expression\"\n }\n }\n },\n {\n \"name\": \"Filter Running Pipelines\",\n \"description\": \"Filter the pipeline runs results for pipelines that exclude the current triggered run and that are currently running (in progress or queued).\",\n \"type\": \"Filter\",\n \"dependsOn\": [\n {\n \"activity\": \"Get Pipeline Runs\",\n \"dependencyConditions\": [\n \"Succeeded\"\n ]\n }\n ],\n \"userProperties\": [],\n \"typeProperties\": {\n \"items\": {\n \"value\": \"@activity('Get Pipeline Runs').output.value\",\n \"type\": \"Expression\"\n },\n \"condition\": {\n \"value\": \"@and(not(equals(item().runId,pipeline().parameters.ThisRunId)),or(equals(item().status,'InProgress'),equals(item().status,'Queued')))\",\n \"type\": \"Expression\"\n }\n }\n },\n {\n \"name\": \"Get Resource Group\",\n \"description\": \"Using the metadata properties table return the framework factory resource group name.\",\n \"type\": \"Lookup\",\n \"dependsOn\": [],\n \"policy\": {\n \"timeout\": \"0.00:10:00\",\n \"retry\": 0,\n \"retryIntervalInSeconds\": 30,\n \"secureOutput\": false,\n \"secureInput\": false\n },\n \"userProperties\": [],\n \"typeProperties\": {\n \"source\": {\n \"type\": \"AzureSqlSource\",\n \"sqlReaderStoredProcedureName\": \"[[procfwk].[GetPropertyValue]\",\n \"storedProcedureParameters\": {\n \"PropertyName\": {\n \"type\": \"String\",\n \"value\": \"FrameworkFactoryResourceGroup\"\n }\n },\n \"queryTimeout\": \"02:00:00\",\n \"partitionOption\": \"None\"\n },\n \"dataset\": {\n \"referenceName\": \"GetSetMetadata\",\n \"type\": \"DatasetReference\",\n \"parameters\": {}\n }\n }\n },\n {\n \"name\": \"Get Query Run Days Value\",\n \"description\": \"Using the metadata properties table return the run days value to provide the API request with a date range for pipeline executions.\",\n \"type\": \"Lookup\",\n \"dependsOn\": [],\n \"policy\": {\n \"timeout\": \"0.00:10:00\",\n \"retry\": 0,\n \"retryIntervalInSeconds\": 30,\n \"secureOutput\": false,\n \"secureInput\": false\n },\n \"userProperties\": [],\n \"typeProperties\": {\n \"source\": {\n \"type\": \"AzureSqlSource\",\n \"sqlReaderStoredProcedureName\": \"[[procfwk].[GetPropertyValue]\",\n \"storedProcedureParameters\": {\n \"PropertyName\": {\n \"type\": \"String\",\n \"value\": \"PreviousPipelineRunsQueryRange\"\n }\n },\n \"queryTimeout\": \"02:00:00\",\n \"partitionOption\": \"None\"\n },\n \"dataset\": {\n \"referenceName\": \"GetSetMetadata\",\n \"type\": \"DatasetReference\",\n \"parameters\": {}\n }\n }\n },\n {\n \"name\": \"If Pipeline Is Running\",\n \"description\": \"If the running pipeline count is greater than or equal to one.\\nTrue = raise an exception.\",\n \"type\": \"IfCondition\",\n \"dependsOn\": [\n {\n \"activity\": \"If Using Batch Executions\",\n \"dependencyConditions\": [\n \"Succeeded\"\n ]\n }\n ],\n \"userProperties\": [],\n \"typeProperties\": {\n \"expression\": {\n \"value\": \"@greaterOrEquals(int(variables('RunCount')),1)\",\n \"type\": \"Expression\"\n },\n \"ifTrueActivities\": [\n {\n \"name\": \"Throw Exceptiion - Pipeline Running\",\n \"description\": \"Using the utils pipeline raise an exception to stop the new trigger while a run is already in progress.\",\n \"type\": \"ExecutePipeline\",\n \"dependsOn\": [],\n \"userProperties\": [],\n \"typeProperties\": {\n \"pipeline\": {\n \"referenceName\": \"Throw Exception\",\n \"type\": \"PipelineReference\"\n },\n \"waitOnCompletion\": true,\n \"parameters\": {\n \"Message\": {\n \"value\": \"@concat('Provided pipeline name (',pipeline().parameters.PipelineName,') still has a run in progress or queued given the query range parameters set in the properties table.')\",\n \"type\": \"Expression\"\n }\n }\n }\n }\n ]\n }\n },\n {\n \"name\": \"Get Execution Batch Status\",\n \"description\": \"Using the metadata properties table return the flag to indicate if batch execution setting are enabled or disabled.\",\n \"type\": \"Lookup\",\n \"dependsOn\": [],\n \"policy\": {\n \"timeout\": \"0.00:10:00\",\n \"retry\": 0,\n \"retryIntervalInSeconds\": 30,\n \"secureOutput\": false,\n \"secureInput\": false\n },\n \"userProperties\": [],\n \"typeProperties\": {\n \"source\": {\n \"type\": \"AzureSqlSource\",\n \"sqlReaderStoredProcedureName\": \"[[procfwk].[GetPropertyValue]\",\n \"storedProcedureParameters\": {\n \"PropertyName\": {\n \"type\": \"String\",\n \"value\": \"UseExecutionBatches\"\n }\n },\n \"queryTimeout\": \"02:00:00\",\n \"partitionOption\": \"None\"\n },\n \"dataset\": {\n \"referenceName\": \"GetSetMetadata\",\n \"type\": \"DatasetReference\",\n \"parameters\": {}\n }\n }\n },\n {\n \"name\": \"If Using Batch Executions\",\n \"description\": \"True = batch executions are enabled.\\nFalse = batch execution are disabled.\",\n \"type\": \"IfCondition\",\n \"dependsOn\": [\n {\n \"activity\": \"Get Execution Batch Status\",\n \"dependencyConditions\": [\n \"Succeeded\"\n ]\n },\n {\n \"activity\": \"Filter Running Pipelines\",\n \"dependencyConditions\": [\n \"Succeeded\"\n ]\n }\n ],\n \"userProperties\": [],\n \"typeProperties\": {\n \"expression\": {\n \"value\": \"@equals(activity('Get Execution Batch Status').output.firstRow.PropertyValue,string(1))\",\n \"type\": \"Expression\"\n },\n \"ifFalseActivities\": [\n {\n \"name\": \"Set Run Count Without Batch\",\n \"description\": \"Set the pipelines running count variable to be tested later.\",\n \"type\": \"SetVariable\",\n \"dependsOn\": [],\n \"userProperties\": [],\n \"typeProperties\": {\n \"variableName\": \"RunCount\",\n \"value\": {\n \"value\": \"@string(activity('Filter Running Pipelines').output.FilteredItemsCount)\",\n \"type\": \"Expression\"\n }\n }\n }\n ],\n \"ifTrueActivities\": [\n {\n \"name\": \"Filter for Batch Name\",\n \"description\": \"Further filter the return pipeline runs for any running pipelines with the same batch name value.\",\n \"type\": \"Filter\",\n \"dependsOn\": [],\n \"userProperties\": [],\n \"typeProperties\": {\n \"items\": {\n \"value\": \"@activity('Filter Running Pipelines').output.value\",\n \"type\": \"Expression\"\n },\n \"condition\": {\n \"value\": \"@equals(item().parameters.BatchName,pipeline().parameters.BatchName)\",\n \"type\": \"Expression\"\n }\n }\n },\n {\n \"name\": \"Set Run Count for Batch\",\n \"description\": \"Set the resulting pipeline running count variable to be tested later.\",\n \"type\": \"SetVariable\",\n \"dependsOn\": [\n {\n \"activity\": \"Filter for Batch Name\",\n \"dependencyConditions\": [\n \"Succeeded\"\n ]\n }\n ],\n \"userProperties\": [],\n \"typeProperties\": {\n \"variableName\": \"RunCount\",\n \"value\": {\n \"value\": \"@string(activity('Filter for Batch Name').output.FilteredItemsCount)\",\n \"type\": \"Expression\"\n }\n }\n }\n ]\n }\n },\n {\n \"name\": \"Check for Valid Pipeline Name\",\n \"description\": \"Use the Azure Management API to return and establish if the framework pipeline exists in the target Data Factory, including being deployed.\",\n \"type\": \"WebActivity\",\n \"dependsOn\": [\n {\n \"activity\": \"Set Parsed Subscription\",\n \"dependencyConditions\": [\n \"Succeeded\"\n ]\n },\n {\n \"activity\": \"Get Resource Group\",\n \"dependencyConditions\": [\n \"Succeeded\"\n ]\n }\n ],\n \"policy\": {\n \"timeout\": \"0.00:10:00\",\n \"retry\": 0,\n \"retryIntervalInSeconds\": 30,\n \"secureOutput\": false,\n \"secureInput\": false\n },\n \"userProperties\": [],\n \"typeProperties\": {\n \"url\": {\n \"value\": \"https://management.azure.com/subscriptions/@{variables('SubscriptionId')}/resourceGroups/@{activity('Get Resource Group').output.firstRow.PropertyValue}/providers/Microsoft.DataFactory/factories/@{pipeline().DataFactory}/pipelines/@{pipeline().parameters.PipelineName}?api-version=2018-06-01\",\n \"type\": \"Expression\"\n },\n \"method\": \"GET\",\n \"headers\": {},\n \"authentication\": {\n \"type\": \"MSI\",\n \"resource\": \"https://management.core.windows.net/\"\n }\n }\n }\n ],\n \"parameters\": {\n \"BatchName\": {\n \"type\": \"string\",\n \"defaultValue\": \"NotUsed\"\n },\n \"PipelineName\": {\n \"type\": \"string\"\n },\n \"ThisRunId\": {\n \"type\": \"string\"\n }\n },\n \"variables\": {\n \"SubscriptionId\": {\n \"type\": \"String\"\n },\n \"RunCount\": {\n \"type\": \"String\"\n }\n },\n \"folder\": {\n \"name\": \"_ProcFwk/_ProcFwkUtils\"\n },\n \"annotations\": [\n \"procfwk\",\n \"Utils\"\n ]\n },\n \"dependsOn\": [\n \"[concat(variables('factoryId'), '/datasets/GetSetMetadata')]\",\n \"[concat(variables('factoryId'), '/pipelines/Throw Exception')]\"\n ]\n },\n {\n \"name\": \"[concat(parameters('factoryName'), '/Email Sender')]\",\n \"type\": \"Microsoft.DataFactory/factories/pipelines\",\n \"apiVersion\": \"2018-06-01\",\n \"properties\": {\n \"description\": \"Provide a simple abstract over the send email function with request body item exposed as pipeline parameters.\",\n \"activities\": [\n {\n \"name\": \"Send Email\",\n \"description\": \"Use an Azure Function to perform an SMTP client email send operation.\",\n \"type\": \"AzureFunctionActivity\",\n \"dependsOn\": [],\n \"policy\": {\n \"timeout\": \"0.00:10:00\",\n \"retry\": 0,\n \"retryIntervalInSeconds\": 30,\n \"secureOutput\": false,\n \"secureInput\": false\n },\n \"userProperties\": [],\n \"typeProperties\": {\n \"functionName\": \"SendEmail\",\n \"method\": \"POST\",\n \"headers\": {},\n \"body\": {\n \"value\": \"{\\n\\\"emailRecipients\\\": \\\"@{pipeline().parameters.Recipients}\\\",\\n\\\"emailCcRecipients\\\": \\\"@{pipeline().parameters.CcRecipients}\\\",\\n\\\"emailBccRecipients\\\": \\\"@{pipeline().parameters.BccRecipients}\\\",\\n\\\"emailSubject\\\": \\\"@{pipeline().parameters.Subject}\\\",\\n\\\"emailBody\\\": \\\"@{pipeline().parameters.Body}\\\",\\n\\\"emailImportance\\\": \\\"@{pipeline().parameters.Importance}\\\"\\n}\",\n \"type\": \"Expression\"\n }\n },\n \"linkedServiceName\": {\n \"referenceName\": \"FrameworkFunctions\",\n \"type\": \"LinkedServiceReference\"\n }\n }\n ],\n \"parameters\": {\n \"Recipients\": {\n \"type\": \"string\"\n },\n \"CcRecipients\": {\n \"type\": \"string\"\n },\n \"BccRecipients\": {\n \"type\": \"string\"\n },\n \"Subject\": {\n \"type\": \"string\"\n },\n \"Body\": {\n \"type\": \"string\"\n },\n \"Importance\": {\n \"type\": \"string\"\n }\n },\n \"folder\": {\n \"name\": \"_ProcFwk/_ProcFwkUtils\"\n },\n \"annotations\": [\n \"procfwk\",\n \"Utils\"\n ]\n },\n \"dependsOn\": [\n \"[concat(variables('factoryId'), '/linkedServices/FrameworkFunctions')]\"\n ]\n },\n {\n \"name\": \"[concat(parameters('factoryName'), '/Intentional Error')]\",\n \"type\": \"Microsoft.DataFactory/factories/pipelines\",\n \"apiVersion\": \"2018-06-01\",\n \"properties\": {\n \"description\": \"Used just so the ADF.procfwk has something to call during development.\",\n \"activities\": [\n {\n \"name\": \"Wait1\",\n \"description\": \"Framework development worker simulator.\",\n \"type\": \"Wait\",\n \"dependsOn\": [],\n \"userProperties\": [],\n \"typeProperties\": {\n \"waitTimeInSeconds\": {\n \"value\": \"@pipeline().parameters.WaitTime\",\n \"type\": \"Expression\"\n }\n }\n },\n {\n \"name\": \"Raise Errors or Not\",\n \"description\": \"Framework development worker simulator.\",\n \"type\": \"IfCondition\",\n \"dependsOn\": [\n {\n \"activity\": \"Wait1\",\n \"dependencyConditions\": [\n \"Succeeded\"\n ]\n }\n ],\n \"userProperties\": [],\n \"typeProperties\": {\n \"expression\": {\n \"value\": \"@equals(pipeline().parameters.RaiseErrors,'true')\",\n \"type\": \"Expression\"\n },\n \"ifTrueActivities\": [\n {\n \"name\": \"Call Fail Procedure\",\n \"type\": \"SqlServerStoredProcedure\",\n \"dependsOn\": [],\n \"policy\": {\n \"timeout\": \"0.00:10:00\",\n \"retry\": 0,\n \"retryIntervalInSeconds\": 30,\n \"secureOutput\": false,\n \"secureInput\": false\n },\n \"userProperties\": [],\n \"typeProperties\": {\n \"storedProcedureName\": \"[[dbo].[FailProcedure]\",\n \"storedProcedureParameters\": {\n \"RaiseError\": {\n \"value\": {\n \"value\": \"@pipeline().parameters.RaiseErrors\",\n \"type\": \"Expression\"\n },\n \"type\": \"String\"\n }\n }\n },\n \"linkedServiceName\": {\n \"referenceName\": \"SupportDatabase\",\n \"type\": \"LinkedServiceReference\"\n }\n }\n ]\n }\n }\n ],\n \"parameters\": {\n \"RaiseErrors\": {\n \"type\": \"string\",\n \"defaultValue\": \"false\"\n },\n \"WaitTime\": {\n \"type\": \"int\",\n \"defaultValue\": 5\n }\n },\n \"folder\": {\n \"name\": \"_Workers\"\n },\n \"annotations\": [\n \"Worker\"\n ]\n },\n \"dependsOn\": [\n \"[concat(variables('factoryId'), '/linkedServices/SupportDatabase')]\"\n ]\n },\n {\n \"name\": \"[concat(parameters('factoryName'), '/Throw Exception')]\",\n \"type\": \"Microsoft.DataFactory/factories/pipelines\",\n \"apiVersion\": \"2018-06-01\",\n \"properties\": {\n \"description\": \"Provide a simple way of throwing an exception within Data Factory using TSQL error handling.\",\n \"activities\": [\n {\n \"name\": \"Raise Error\",\n \"description\": \"Using a SQL database to raise an error/exception but wrapped up as a data factory pipeline. Error message information exposed as a pipeline parameter.\",\n \"type\": \"Lookup\",\n \"dependsOn\": [],\n \"policy\": {\n \"timeout\": \"0.00:10:00\",\n \"retry\": 0,\n \"retryIntervalInSeconds\": 30,\n \"secureOutput\": false,\n \"secureInput\": false\n },\n \"userProperties\": [],\n \"typeProperties\": {\n \"source\": {\n \"type\": \"AzureSqlSource\",\n \"sqlReaderQuery\": {\n \"value\": \"RAISERROR('@{pipeline().parameters.Message}',16,1);\",\n \"type\": \"Expression\"\n },\n \"queryTimeout\": \"02:00:00\",\n \"partitionOption\": \"None\"\n },\n \"dataset\": {\n \"referenceName\": \"GetSetMetadata\",\n \"type\": \"DatasetReference\",\n \"parameters\": {}\n },\n \"firstRowOnly\": false\n }\n }\n ],\n \"parameters\": {\n \"Message\": {\n \"type\": \"string\"\n }\n },\n \"folder\": {\n \"name\": \"_ProcFwk/_ProcFwkUtils\"\n },\n \"annotations\": [\n \"procfwk\",\n \"Utils\"\n ]\n },\n \"dependsOn\": [\n \"[concat(variables('factoryId'), '/datasets/GetSetMetadata')]\"\n ]\n },\n {\n \"name\": \"[concat(parameters('factoryName'), '/Wait 1')]\",\n \"type\": \"Microsoft.DataFactory/factories/pipelines\",\n \"apiVersion\": \"2018-06-01\",\n \"properties\": {\n \"description\": \"Used just so the ADF.procfwk has something to call during development.\",\n \"activities\": [\n {\n \"name\": \"Wait1\",\n \"description\": \"Framework development worker simulator.\",\n \"type\": \"Wait\",\n \"dependsOn\": [],\n \"userProperties\": [],\n \"typeProperties\": {\n \"waitTimeInSeconds\": {\n \"value\": \"@pipeline().parameters.WaitTime\",\n \"type\": \"Expression\"\n }\n }\n }\n ],\n \"parameters\": {\n \"WaitTime\": {\n \"type\": \"int\",\n \"defaultValue\": 5\n }\n },\n \"folder\": {\n \"name\": \"_Workers\"\n },\n \"annotations\": [\n \"Worker\"\n ]\n },\n \"dependsOn\": []\n },\n {\n \"name\": \"[concat(parameters('factoryName'), '/Wait 10')]\",\n \"type\": \"Microsoft.DataFactory/factories/pipelines\",\n \"apiVersion\": \"2018-06-01\",\n \"properties\": {\n \"description\": \"Used just so the ADF.procfwk has something to call during development.\",\n \"activities\": [\n {\n \"name\": \"Wait10\",\n \"description\": \"Framework development worker simulator.\",\n \"type\": \"Wait\",\n \"dependsOn\": [],\n \"userProperties\": [],\n \"typeProperties\": {\n \"waitTimeInSeconds\": {\n \"value\": \"@pipeline().parameters.WaitTime\",\n \"type\": \"Expression\"\n }\n }\n }\n ],\n \"parameters\": {\n \"WaitTime\": {\n \"type\": \"int\",\n \"defaultValue\": 5\n }\n },\n \"folder\": {\n \"name\": \"_Workers\"\n },\n \"annotations\": [\n \"Worker\"\n ]\n },\n \"dependsOn\": []\n },\n {\n \"name\": \"[concat(parameters('factoryName'), '/Wait 2')]\",\n \"type\": \"Microsoft.DataFactory/factories/pipelines\",\n \"apiVersion\": \"2018-06-01\",\n \"properties\": {\n \"description\": \"Used just so the ADF.procfwk has something to call during development.\",\n \"activities\": [\n {\n \"name\": \"Wait2\",\n \"description\": \"Framework development worker simulator.\",\n \"type\": \"Wait\",\n \"dependsOn\": [],\n \"userProperties\": [],\n \"typeProperties\": {\n \"waitTimeInSeconds\": {\n \"value\": \"@pipeline().parameters.WaitTime\",\n \"type\": \"Expression\"\n }\n }\n }\n ],\n \"parameters\": {\n \"WaitTime\": {\n \"type\": \"int\",\n \"defaultValue\": 5\n }\n },\n \"folder\": {\n \"name\": \"_Workers\"\n },\n \"annotations\": [\n \"Worker\"\n ]\n },\n \"dependsOn\": []\n },\n {\n \"name\": \"[concat(parameters('factoryName'), '/Wait 3')]\",\n \"type\": \"Microsoft.DataFactory/factories/pipelines\",\n \"apiVersion\": \"2018-06-01\",\n \"properties\": {\n \"description\": \"Used just so the ADF.procfwk has something to call during development.\",\n \"activities\": [\n {\n \"name\": \"Wait3\",\n \"description\": \"Framework development worker simulator.\",\n \"type\": \"Wait\",\n \"dependsOn\": [],\n \"userProperties\": [],\n \"typeProperties\": {\n \"waitTimeInSeconds\": {\n \"value\": \"@pipeline().parameters.WaitTime\",\n \"type\": \"Expression\"\n }\n }\n }\n ],\n \"parameters\": {\n \"WaitTime\": {\n \"type\": \"int\",\n \"defaultValue\": 5\n }\n },\n \"folder\": {\n \"name\": \"_Workers\"\n },\n \"annotations\": [\n \"Worker\"\n ]\n },\n \"dependsOn\": []\n },\n {\n \"name\": \"[concat(parameters('factoryName'), '/Wait 4')]\",\n \"type\": \"Microsoft.DataFactory/factories/pipelines\",\n \"apiVersion\": \"2018-06-01\",\n \"properties\": {\n \"description\": \"Used just so the ADF.procfwk has something to call during development.\",\n \"activities\": [\n {\n \"name\": \"Wait4\",\n \"description\": \"Framework development worker simulator.\",\n \"type\": \"Wait\",\n \"dependsOn\": [],\n \"userProperties\": [],\n \"typeProperties\": {\n \"waitTimeInSeconds\": {\n \"value\": \"@pipeline().parameters.WaitTime\",\n \"type\": \"Expression\"\n }\n }\n }\n ],\n \"parameters\": {\n \"WaitTime\": {\n \"type\": \"int\",\n \"defaultValue\": 5\n }\n },\n \"folder\": {\n \"name\": \"_Workers\"\n },\n \"annotations\": [\n \"Worker\"\n ]\n },\n \"dependsOn\": []\n },\n {\n \"name\": \"[concat(parameters('factoryName'), '/Wait 5')]\",\n \"type\": \"Microsoft.DataFactory/factories/pipelines\",\n \"apiVersion\": \"2018-06-01\",\n \"properties\": {\n \"description\": \"Used just so the ADF.procfwk has something to call during development.\",\n \"activities\": [\n {\n \"name\": \"Wait5\",\n \"description\": \"Framework development worker simulator.\",\n \"type\": \"Wait\",\n \"dependsOn\": [],\n \"userProperties\": [],\n \"typeProperties\": {\n \"waitTimeInSeconds\": {\n \"value\": \"@pipeline().parameters.WaitTime\",\n \"type\": \"Expression\"\n }\n }\n }\n ],\n \"parameters\": {\n \"WaitTime\": {\n \"type\": \"int\",\n \"defaultValue\": 5\n }\n },\n \"folder\": {\n \"name\": \"_Workers\"\n },\n \"annotations\": [\n \"Worker\"\n ]\n },\n \"dependsOn\": []\n },\n {\n \"name\": \"[concat(parameters('factoryName'), '/Wait 6')]\",\n \"type\": \"Microsoft.DataFactory/factories/pipelines\",\n \"apiVersion\": \"2018-06-01\",\n \"properties\": {\n \"description\": \"Used just so the ADF.procfwk has something to call during development.\",\n \"activities\": [\n {\n \"name\": \"Wait6\",\n \"description\": \"Framework development worker simulator.\",\n \"type\": \"Wait\",\n \"dependsOn\": [],\n \"userProperties\": [],\n \"typeProperties\": {\n \"waitTimeInSeconds\": {\n \"value\": \"@pipeline().parameters.WaitTime\",\n \"type\": \"Expression\"\n }\n }\n }\n ],\n \"parameters\": {\n \"WaitTime\": {\n \"type\": \"int\",\n \"defaultValue\": 5\n }\n },\n \"folder\": {\n \"name\": \"_Workers\"\n },\n \"annotations\": [\n \"Worker\"\n ]\n },\n \"dependsOn\": []\n },\n {\n \"name\": \"[concat(parameters('factoryName'), '/Wait 7')]\",\n \"type\": \"Microsoft.DataFactory/factories/pipelines\",\n \"apiVersion\": \"2018-06-01\",\n \"properties\": {\n \"description\": \"Used just so the ADF.procfwk has something to call during development.\",\n \"activities\": [\n {\n \"name\": \"Wait7\",\n \"description\": \"Framework development worker simulator.\",\n \"type\": \"Wait\",\n \"dependsOn\": [],\n \"userProperties\": [],\n \"typeProperties\": {\n \"waitTimeInSeconds\": {\n \"value\": \"@pipeline().parameters.WaitTime\",\n \"type\": \"Expression\"\n }\n }\n }\n ],\n \"parameters\": {\n \"WaitTime\": {\n \"type\": \"int\",\n \"defaultValue\": 5\n }\n },\n \"folder\": {\n \"name\": \"_Workers\"\n },\n \"annotations\": [\n \"Worker\"\n ]\n },\n \"dependsOn\": []\n },\n {\n \"name\": \"[concat(parameters('factoryName'), '/Wait 8')]\",\n \"type\": \"Microsoft.DataFactory/factories/pipelines\",\n \"apiVersion\": \"2018-06-01\",\n \"properties\": {\n \"description\": \"Used just so the ADF.procfwk has something to call during development.\",\n \"activities\": [\n {\n \"name\": \"Wait8\",\n \"description\": \"Framework development worker simulator.\",\n \"type\": \"Wait\",\n \"dependsOn\": [],\n \"userProperties\": [],\n \"typeProperties\": {\n \"waitTimeInSeconds\": {\n \"value\": \"@pipeline().parameters.WaitTime\",\n \"type\": \"Expression\"\n }\n }\n }\n ],\n \"parameters\": {\n \"WaitTime\": {\n \"type\": \"int\",\n \"defaultValue\": 5\n }\n },\n \"folder\": {\n \"name\": \"_Workers\"\n },\n \"annotations\": [\n \"Worker\"\n ]\n },\n \"dependsOn\": []\n },\n {\n \"name\": \"[concat(parameters('factoryName'), '/Wait 9')]\",\n \"type\": \"Microsoft.DataFactory/factories/pipelines\",\n \"apiVersion\": \"2018-06-01\",\n \"properties\": {\n \"description\": \"Used just so the ADF.procfwk has something to call during development.\",\n \"activities\": [\n {\n \"name\": \"Wait9\",\n \"description\": \"Framework development worker simulator.\",\n \"type\": \"Wait\",\n \"dependsOn\": [],\n \"userProperties\": [],\n \"typeProperties\": {\n \"waitTimeInSeconds\": {\n \"value\": \"@pipeline().parameters.WaitTime\",\n \"type\": \"Expression\"\n }\n }\n }\n ],\n \"parameters\": {\n \"WaitTime\": {\n \"type\": \"int\",\n \"defaultValue\": 15\n }\n },\n \"folder\": {\n \"name\": \"_Workers\"\n },\n \"annotations\": [\n \"Worker\"\n ]\n },\n \"dependsOn\": []\n },\n {\n \"name\": \"[concat(parameters('factoryName'), '/GetSetMetadata')]\",\n \"type\": \"Microsoft.DataFactory/factories/datasets\",\n \"apiVersion\": \"2018-06-01\",\n \"properties\": {\n \"description\": \"Single generic dataset used to get and set all database metadata.\",\n \"linkedServiceName\": {\n \"referenceName\": \"SupportDatabase\",\n \"type\": \"LinkedServiceReference\"\n },\n \"folder\": {\n \"name\": \"_ProcFwk\"\n },\n \"annotations\": [\n \"ADF.procfwk\"\n ],\n \"type\": \"AzureSqlTable\",\n \"schema\": [],\n \"typeProperties\": {}\n },\n \"dependsOn\": [\n \"[concat(variables('factoryId'), '/linkedServices/SupportDatabase')]\"\n ]\n },\n {\n \"name\": \"[concat(parameters('factoryName'), '/FrameworkFunctions')]\",\n \"type\": \"Microsoft.DataFactory/factories/linkedServices\",\n \"apiVersion\": \"2018-06-01\",\n \"properties\": {\n \"description\": \"Interact with the Azure Functions App used as middle ware when making requests to Worker pipelines. Authentication done at the Function App level.\",\n \"annotations\": [\n \"ADF.procfwk\"\n ],\n \"type\": \"AzureFunction\",\n \"typeProperties\": {\n \"functionAppUrl\": \"[parameters('FrameworkFunctions_properties_typeProperties_functionAppUrl')]\",\n \"functionKey\": {\n \"type\": \"AzureKeyVaultSecret\",\n \"store\": {\n \"referenceName\": \"Keys\",\n \"type\": \"LinkedServiceReference\"\n },\n \"secretName\": \"FrameworkFunctionsKey\"\n }\n }\n },\n \"dependsOn\": [\n \"[concat(variables('factoryId'), '/linkedServices/Keys')]\"\n ]\n },\n {\n \"name\": \"[concat(parameters('factoryName'), '/Keys')]\",\n \"type\": \"Microsoft.DataFactory/factories/linkedServices\",\n \"apiVersion\": \"2018-06-01\",\n \"properties\": {\n \"description\": \"Connection to Key Vault for all other ADF linked service credentials required to run the processing framework.\",\n \"annotations\": [\n \"ADF.procfwk\"\n ],\n \"type\": \"AzureKeyVault\",\n \"typeProperties\": {\n \"baseUrl\": \"[parameters('Keys_properties_typeProperties_baseUrl')]\"\n }\n },\n \"dependsOn\": []\n },\n {\n \"name\": \"[concat(parameters('factoryName'), '/SupportDatabase')]\",\n \"type\": \"Microsoft.DataFactory/factories/linkedServices\",\n \"apiVersion\": \"2018-06-01\",\n \"properties\": {\n \"description\": \"Connection between ADF and processing framework metadata SQLDB.\",\n \"annotations\": [\n \"ADF.procfwk\"\n ],\n \"type\": \"AzureSqlDatabase\",\n \"typeProperties\": {\n \"connectionString\": {\n \"type\": \"AzureKeyVaultSecret\",\n \"store\": {\n \"referenceName\": \"Keys\",\n \"type\": \"LinkedServiceReference\"\n },\n \"secretName\": \"[parameters('SupportDatabase_properties_typeProperties_connectionString_secretName')]\"\n }\n }\n },\n \"dependsOn\": [\n \"[concat(variables('factoryId'), '/linkedServices/Keys')]\"\n ]\n },\n {\n \"name\": \"[concat(parameters('factoryName'), '/FunctionalTestingTrigger')]\",\n \"type\": \"Microsoft.DataFactory/factories/triggers\",\n \"apiVersion\": \"2018-06-01\",\n \"properties\": {\n \"description\": \"Used for functional testing of the framework in a dedicated environment.\",\n \"annotations\": [\n \"ADF.procfwk\"\n ],\n \"runtimeState\": \"Stopped\",\n \"pipelines\": [\n {\n \"pipelineReference\": {\n \"referenceName\": \"01-Grandparent\",\n \"type\": \"PipelineReference\"\n },\n \"parameters\": {}\n }\n ],\n \"type\": \"ScheduleTrigger\",\n \"typeProperties\": {\n \"recurrence\": {\n \"frequency\": \"Hour\",\n \"interval\": 2,\n \"startTime\": \"2020-04-06T15:00:00Z\",\n \"timeZone\": \"UTC\"\n }\n }\n },\n \"dependsOn\": [\n \"[concat(variables('factoryId'), '/pipelines/01-Grandparent')]\"\n ]\n }\n ]\n}"
},
{
"path": "ARM Templates/Data Factory/v2.0 Export.json",
"content": "{\n \"$schema\": \"http://schema.management.azure.com/schemas/2015-01-01/deploymentTemplate.json#\",\n \"contentVersion\": \"1.0.0.0\",\n \"parameters\": {\n \"factoryName\": {\n \"type\": \"string\",\n \"metadata\": \"Data Factory name\",\n \"defaultValue\": \"\"\n },\n \"FrameworkFunctions_properties_typeProperties_functionAppUrl\": {\n \"type\": \"string\",\n \"defaultValue\": \"\"\n },\n \"Keys_properties_typeProperties_baseUrl\": {\n \"type\": \"string\",\n \"defaultValue\": \"\"\n },\n \"SupportDatabase_properties_typeProperties_connectionString_secretName\": {\n \"type\": \"string\",\n \"defaultValue\": \"\"\n }\n },\n \"variables\": {\n \"factoryId\": \"[concat('Microsoft.DataFactory/factories/', parameters('factoryName'))]\"\n },\n \"resources\": [\n {\n \"name\": \"[concat(parameters('factoryName'), '/01-Grandparent')]\",\n \"type\": \"Microsoft.DataFactory/factories/pipelines\",\n \"apiVersion\": \"2018-06-01\",\n \"properties\": {\n \"description\": \"procfwk grandparent pipeline used optionally to bootstrap any wider processes in your Data Factory that then calls the processing framework.\",\n \"activities\": [\n {\n \"name\": \"procfwk\",\n \"description\": \"Call procfwk\",\n \"type\": \"ExecutePipeline\",\n \"dependsOn\": [],\n \"userProperties\": [],\n \"typeProperties\": {\n \"pipeline\": {\n \"referenceName\": \"02-Parent\",\n \"type\": \"PipelineReference\"\n },\n \"waitOnCompletion\": true,\n \"parameters\": {\n \"BatchName\": {\n \"value\": \"@pipeline().parameters.BatchName\",\n \"type\": \"Expression\"\n }\n }\n }\n }\n ],\n \"parameters\": {\n \"BatchName\": {\n \"type\": \"string\",\n \"defaultValue\": \"NotUsed\"\n }\n },\n \"folder\": {\n \"name\": \"_ProcFwk\"\n },\n \"annotations\": [\n \"procfwk\",\n \"Grandparent\"\n ]\n },\n \"dependsOn\": [\n \"[concat(variables('factoryId'), '/pipelines/02-Parent')]\"\n ]\n },\n {\n \"name\": \"[concat(parameters('factoryName'), '/02-Parent')]\",\n \"type\": \"Microsoft.DataFactory/factories/pipelines\",\n \"apiVersion\": \"2018-06-01\",\n \"properties\": {\n \"description\": \"ADF.procfwk parent pipeline used to bootstrap the orchestration framework in perform the first level ForEach calls in sequence for the metadata stages.\",\n \"activities\": [\n {\n \"name\": \"Get Stages\",\n \"description\": \"Returns a distinct list of execution stages within the framework metadata.\",\n \"type\": \"Lookup\",\n \"dependsOn\": [\n {\n \"activity\": \"Set Execution Id\",\n \"dependencyConditions\": [\n \"Succeeded\"\n ]\n }\n ],\n \"policy\": {\n \"timeout\": \"0.00:10:00\",\n \"retry\": 0,\n \"retryIntervalInSeconds\": 30,\n \"secureOutput\": false,\n \"secureInput\": false\n },\n \"userProperties\": [],\n \"typeProperties\": {\n \"source\": {\n \"type\": \"AzureSqlSource\",\n \"sqlReaderStoredProcedureName\": \"[[procfwk].[GetStages]\",\n \"storedProcedureParameters\": {\n \"ExecutionId\": {\n \"type\": \"Guid\",\n \"value\": {\n \"value\": \"@variables('ExecutionId')\",\n \"type\": \"Expression\"\n }\n }\n },\n \"queryTimeout\": \"02:00:00\",\n \"partitionOption\": \"None\"\n },\n \"dataset\": {\n \"referenceName\": \"GetSetMetadata\",\n \"type\": \"DatasetReference\",\n \"parameters\": {}\n },\n \"firstRowOnly\": false\n }\n },\n {\n \"name\": \"Execute Stages\",\n \"description\": \"Top level ForEach to sequentially call all processing stages within the framework metadata. Items for iteration passed from the Get Stages lookup activity.\",\n \"type\": \"ForEach\",\n \"dependsOn\": [\n {\n \"activity\": \"Get Stages\",\n \"dependencyConditions\": [\n \"Succeeded\"\n ]\n }\n ],\n \"userProperties\": [],\n \"typeProperties\": {\n \"items\": {\n \"value\": \"@activity('Get Stages').output.value\",\n \"type\": \"Expression\"\n },\n \"isSequential\": true,\n \"activities\": [\n {\n \"name\": \"Stage Executor\",\n \"description\": \"Call to the framework generic child pipeline for a given execution stage.\",\n \"type\": \"ExecutePipeline\",\n \"dependsOn\": [\n {\n \"activity\": \"Log Stage Preparing\",\n \"dependencyConditions\": [\n \"Succeeded\"\n ]\n }\n ],\n \"userProperties\": [],\n \"typeProperties\": {\n \"pipeline\": {\n \"referenceName\": \"03-Child\",\n \"type\": \"PipelineReference\"\n },\n \"waitOnCompletion\": true,\n \"parameters\": {\n \"StageId\": {\n \"value\": \"@item().StageId\",\n \"type\": \"Expression\"\n },\n \"ExecutionId\": {\n \"value\": \"@variables('ExecutionId')\",\n \"type\": \"Expression\"\n }\n }\n }\n },\n {\n \"name\": \"Log Stage Preparing\",\n \"description\": \"Update the current execution table flagging all pipelines within the stage as preparing.\",\n \"type\": \"SqlServerStoredProcedure\",\n \"dependsOn\": [\n {\n \"activity\": \"Check and Update Blockers\",\n \"dependencyConditions\": [\n \"Succeeded\"\n ]\n }\n ],\n \"policy\": {\n \"timeout\": \"0.00:10:00\",\n \"retry\": 0,\n \"retryIntervalInSeconds\": 30,\n \"secureOutput\": false,\n \"secureInput\": false\n },\n \"userProperties\": [],\n \"typeProperties\": {\n \"storedProcedureName\": \"[[procfwk].[SetLogStagePreparing]\",\n \"storedProcedureParameters\": {\n \"ExecutionId\": {\n \"value\": {\n \"value\": \"@variables('ExecutionId')\",\n \"type\": \"Expression\"\n },\n \"type\": \"Guid\"\n },\n \"StageId\": {\n \"value\": {\n \"value\": \"@item().StageId\",\n \"type\": \"Expression\"\n },\n \"type\": \"Int32\"\n }\n }\n },\n \"linkedServiceName\": {\n \"referenceName\": \"SupportDatabase\",\n \"type\": \"LinkedServiceReference\"\n }\n },\n {\n \"name\": \"Check and Update Blockers\",\n \"description\": \"Used to double check and stop the next execution stage if failures and blockers have be incurred. This also depends on the failure handling property value which defines the stored procedure behaviour.\",\n \"type\": \"SqlServerStoredProcedure\",\n \"dependsOn\": [],\n \"policy\": {\n \"timeout\": \"0.00:10:00\",\n \"retry\": 0,\n \"retryIntervalInSeconds\": 30,\n \"secureOutput\": false,\n \"secureInput\": false\n },\n \"userProperties\": [],\n \"typeProperties\": {\n \"storedProcedureName\": \"[[procfwk].[CheckForBlockedPipelines]\",\n \"storedProcedureParameters\": {\n \"ExecutionId\": {\n \"value\": {\n \"value\": \"@variables('ExecutionId')\",\n \"type\": \"Expression\"\n },\n \"type\": \"Guid\"\n },\n \"StageId\": {\n \"value\": {\n \"value\": \"@item().StageId\",\n \"type\": \"Expression\"\n },\n \"type\": \"Int32\"\n }\n }\n },\n \"linkedServiceName\": {\n \"referenceName\": \"SupportDatabase\",\n \"type\": \"LinkedServiceReference\"\n }\n }\n ]\n }\n },\n {\n \"name\": \"Execution Wrapper\",\n \"description\": \"Wrapper to reset and restart processing or create a completely new execution instance of the framework metadata.\",\n \"type\": \"Lookup\",\n \"dependsOn\": [\n {\n \"activity\": \"Clean Up Previous Run\",\n \"dependencyConditions\": [\n \"Succeeded\"\n ]\n }\n ],\n \"policy\": {\n \"timeout\": \"0.00:10:00\",\n \"retry\": 0,\n \"retryIntervalInSeconds\": 30,\n \"secureOutput\": false,\n \"secureInput\": false\n },\n \"userProperties\": [],\n \"typeProperties\": {\n \"source\": {\n \"type\": \"AzureSqlSource\",\n \"sqlReaderStoredProcedureName\": \"[[procfwk].[ExecutionWrapper]\",\n \"storedProcedureParameters\": {\n \"CallingOrchestratorName\": {\n \"type\": \"String\",\n \"value\": {\n \"value\": \"@pipeline().DataFactory\",\n \"type\": \"Expression\"\n }\n },\n \"BatchName\": {\n \"type\": \"String\",\n \"value\": {\n \"value\": \"@pipeline().parameters.BatchName\",\n \"type\": \"Expression\"\n }\n }\n },\n \"queryTimeout\": \"02:00:00\",\n \"partitionOption\": \"None\"\n },\n \"dataset\": {\n \"referenceName\": \"GetSetMetadata\",\n \"type\": \"DatasetReference\",\n \"parameters\": {}\n }\n }\n },\n {\n \"name\": \"Check Outcome and Update Logs\",\n \"description\": \"After a successful execution run the current execution metadata is moved to the long term logging table by this stored procedure call. Otherwise an error will be raised.\",\n \"type\": \"SqlServerStoredProcedure\",\n \"dependsOn\": [\n {\n \"activity\": \"Execute Stages\",\n \"dependencyConditions\": [\n \"Succeeded\"\n ]\n }\n ],\n \"policy\": {\n \"timeout\": \"0.00:10:00\",\n \"retry\": 0,\n \"retryIntervalInSeconds\": 30,\n \"secureOutput\": false,\n \"secureInput\": false\n },\n \"userProperties\": [],\n \"typeProperties\": {\n \"storedProcedureName\": \"[[procfwk].[UpdateExecutionLog]\",\n \"storedProcedureParameters\": {\n \"PerformErrorCheck\": {\n \"value\": {\n \"value\": \"@bool(1)\",\n \"type\": \"Expression\"\n },\n \"type\": \"Boolean\"\n },\n \"ExecutionId\": {\n \"value\": {\n \"value\": \"@variables('ExecutionId')\",\n \"type\": \"Expression\"\n },\n \"type\": \"Guid\"\n }\n }\n },\n \"linkedServiceName\": {\n \"referenceName\": \"SupportDatabase\",\n \"type\": \"LinkedServiceReference\"\n }\n },\n {\n \"name\": \"Check Previous Execution\",\n \"description\": \"Query the current execution table for worker pipelines that require a clean up from the previous execution run.\",\n \"type\": \"Lookup\",\n \"dependsOn\": [\n {\n \"activity\": \"Execute Precursor\",\n \"dependencyConditions\": [\n \"Succeeded\"\n ]\n }\n ],\n \"policy\": {\n \"timeout\": \"0.00:10:00\",\n \"retry\": 0,\n \"retryIntervalInSeconds\": 30,\n \"secureOutput\": false,\n \"secureInput\": false\n },\n \"userProperties\": [],\n \"typeProperties\": {\n \"source\": {\n \"type\": \"AzureSqlSource\",\n \"sqlReaderStoredProcedureName\": \"[[procfwk].[CheckPreviousExeuction]\",\n \"storedProcedureParameters\": {\n \"BatchName\": {\n \"type\": \"String\",\n \"value\": {\n \"value\": \"@pipeline().parameters.BatchName\",\n \"type\": \"Expression\"\n }\n }\n },\n \"queryTimeout\": \"02:00:00\",\n \"partitionOption\": \"None\"\n },\n \"dataset\": {\n \"referenceName\": \"GetSetMetadata\",\n \"type\": \"DatasetReference\",\n \"parameters\": {}\n },\n \"firstRowOnly\": false\n }\n },\n {\n \"name\": \"Clean Up Previous Run\",\n \"description\": \"Handle Worker pipelines that are reported as Running when the parent pipeline is called again. Get what the actual status of those pipelines is.\",\n \"type\": \"ForEach\",\n \"dependsOn\": [\n {\n \"activity\": \"Check Previous Execution\",\n \"dependencyConditions\": [\n \"Succeeded\"\n ]\n },\n {\n \"activity\": \"Check Metadata Integrity\",\n \"dependencyConditions\": [\n \"Succeeded\"\n ]\n }\n ],\n \"userProperties\": [],\n \"typeProperties\": {\n \"items\": {\n \"value\": \"@activity('Check Previous Execution').output.value\",\n \"type\": \"Expression\"\n },\n \"isSequential\": false,\n \"batchCount\": 50,\n \"activities\": [\n {\n \"name\": \"Get SPN Details\",\n \"type\": \"Lookup\",\n \"dependsOn\": [],\n \"policy\": {\n \"timeout\": \"0.00:10:00\",\n \"retry\": 0,\n \"retryIntervalInSeconds\": 30,\n \"secureOutput\": true,\n \"secureInput\": false\n },\n \"userProperties\": [],\n \"typeProperties\": {\n \"source\": {\n \"type\": \"AzureSqlSource\",\n \"sqlReaderStoredProcedureName\": \"[[procfwk].[GetWorkerAuthDetails]\",\n \"storedProcedureParameters\": {\n \"ExecutionId\": {\n \"type\": \"Guid\",\n \"value\": {\n \"value\": \"@item().LocalExecutionId\",\n \"type\": \"Expression\"\n }\n },\n \"PipelineId\": {\n \"type\": \"Int32\",\n \"value\": {\n \"value\": \"@item().PipelineId\",\n \"type\": \"Expression\"\n }\n },\n \"StageId\": {\n \"type\": \"Int32\",\n \"value\": {\n \"value\": \"@item().StageId\",\n \"type\": \"Expression\"\n }\n }\n },\n \"queryTimeout\": \"02:00:00\",\n \"partitionOption\": \"None\"\n },\n \"dataset\": {\n \"referenceName\": \"GetSetMetadata\",\n \"type\": \"DatasetReference\",\n \"parameters\": {}\n }\n }\n },\n {\n \"name\": \"Log Pipeline Checking\",\n \"type\": \"SqlServerStoredProcedure\",\n \"dependsOn\": [],\n \"policy\": {\n \"timeout\": \"0.00:10:00\",\n \"retry\": 0,\n \"retryIntervalInSeconds\": 30,\n \"secureOutput\": false,\n \"secureInput\": false\n },\n \"userProperties\": [],\n \"typeProperties\": {\n \"storedProcedureName\": \"[[procfwk].[SetLogPipelineChecking]\",\n \"storedProcedureParameters\": {\n \"ExecutionId\": {\n \"value\": {\n \"value\": \"@item().LocalExecutionId\",\n \"type\": \"Expression\"\n },\n \"type\": \"Guid\"\n },\n \"PipelineId\": {\n \"value\": {\n \"value\": \"@item().PipelineId\",\n \"type\": \"Expression\"\n },\n \"type\": \"Int32\"\n },\n \"StageId\": {\n \"value\": {\n \"value\": \"@item().StageId\",\n \"type\": \"Expression\"\n },\n \"type\": \"Int32\"\n }\n }\n },\n \"linkedServiceName\": {\n \"referenceName\": \"SupportDatabase\",\n \"type\": \"LinkedServiceReference\"\n }\n },\n {\n \"name\": \"Get Pipeline Status\",\n \"type\": \"AzureFunctionActivity\",\n \"dependsOn\": [\n {\n \"activity\": \"Get SPN Details\",\n \"dependencyConditions\": [\n \"Succeeded\"\n ]\n },\n {\n \"activity\": \"Log Pipeline Checking\",\n \"dependencyConditions\": [\n \"Succeeded\"\n ]\n }\n ],\n \"policy\": {\n \"timeout\": \"0.00:10:00\",\n \"retry\": 0,\n \"retryIntervalInSeconds\": 30,\n \"secureOutput\": false,\n \"secureInput\": true\n },\n \"userProperties\": [],\n \"typeProperties\": {\n \"functionName\": \"CheckPipelineStatus\",\n \"method\": \"POST\",\n \"headers\": {},\n \"body\": {\n \"value\": \"@concat('\\n{\\n \\\"tenantId\\\": \\\"',activity('Get SPN Details').output.firstRow.TenantId,'\\\",\\n \\\"applicationId\\\": \\\"',activity('Get SPN Details').output.firstRow.AppId,'\\\",\\n \\\"authenticationKey\\\": \\\"',activity('Get SPN Details').output.firstRow.AppSecret,'\\\",\\n \\\"subscriptionId\\\": \\\"',activity('Get SPN Details').output.firstRow.SubscriptionId,'\\\",\\n \\\"resourceGroupName\\\": \\\"',item().ResourceGroupName,'\\\",\\n \\\"orchestratorName\\\": \\\"',item().OrchestratorName,'\\\",\\n \\\"orchestratorType\\\": \\\"',item().OrchestratorType,'\\\",\\n \\\"pipelineName\\\": \\\"',item().PipelineName,'\\\",\\n \\\"runId\\\": \\\"',item().PipelineRunId,'\\\"\\n}')\",\n \"type\": \"Expression\"\n }\n },\n \"linkedServiceName\": {\n \"referenceName\": \"FrameworkFunctions\",\n \"type\": \"LinkedServiceReference\"\n }\n },\n {\n \"name\": \"Set Pipeline Status\",\n \"description\": \"Update the metadata depending on the actual pipeline outcome. Using the status as the case.\",\n \"type\": \"Switch\",\n \"dependsOn\": [\n {\n \"activity\": \"Get Pipeline Status\",\n \"dependencyConditions\": [\n \"Succeeded\"\n ]\n }\n ],\n \"userProperties\": [],\n \"typeProperties\": {\n \"on\": {\n \"value\": \"@activity('Get Pipeline Status').output.ActualStatus\",\n \"type\": \"Expression\"\n },\n \"cases\": [\n {\n \"value\": \"Failed\",\n \"activities\": [\n {\n \"name\": \"Pipeline Status Failed\",\n \"description\": \"Updates the current execution table with a pipeline status of failed if the function outcome is failed. Also blocks pipelines in the downstream execution stage.\",\n \"type\": \"SqlServerStoredProcedure\",\n \"dependsOn\": [],\n \"policy\": {\n \"timeout\": \"0.00:10:00\",\n \"retry\": 0,\n \"retryIntervalInSeconds\": 30,\n \"secureOutput\": false,\n \"secureInput\": false\n },\n \"userProperties\": [],\n \"typeProperties\": {\n \"storedProcedureName\": \"[[procfwk].[SetLogPipelineFailed]\",\n \"storedProcedureParameters\": {\n \"ExecutionId\": {\n \"value\": {\n \"value\": \"@item().LocalExecutionId\",\n \"type\": \"Expression\"\n },\n \"type\": \"Guid\"\n },\n \"PipelineId\": {\n \"value\": {\n \"value\": \"@item().PipelineId\",\n \"type\": \"Expression\"\n },\n \"type\": \"Int32\"\n },\n \"RunId\": {\n \"value\": null,\n \"type\": \"Guid\"\n },\n \"StageId\": {\n \"value\": {\n \"value\": \"@item().StageId\",\n \"type\": \"Expression\"\n },\n \"type\": \"Int32\"\n }\n }\n },\n \"linkedServiceName\": {\n \"referenceName\": \"SupportDatabase\",\n \"type\": \"LinkedServiceReference\"\n }\n }\n ]\n },\n {\n \"value\": \"Succeeded\",\n \"activities\": [\n {\n \"name\": \"Pipeline Status Succeeded\",\n \"description\": \"Updates the current execution table with a pipeline status of success if the function outcome is succeeded.\",\n \"type\": \"SqlServerStoredProcedure\",\n \"dependsOn\": [],\n \"policy\": {\n \"timeout\": \"0.00:10:00\",\n \"retry\": 0,\n \"retryIntervalInSeconds\": 30,\n \"secureOutput\": false,\n \"secureInput\": false\n },\n \"userProperties\": [],\n \"typeProperties\": {\n \"storedProcedureName\": \"[[procfwk].[SetLogPipelineSuccess]\",\n \"storedProcedureParameters\": {\n \"ExecutionId\": {\n \"value\": {\n \"value\": \"@item().LocalExecutionId\",\n \"type\": \"Expression\"\n },\n \"type\": \"Guid\"\n },\n \"PipelineId\": {\n \"value\": {\n \"value\": \"@item().PipelineId\",\n \"type\": \"Expression\"\n },\n \"type\": \"Int32\"\n },\n \"StageId\": {\n \"value\": {\n \"value\": \"@item().StageId\",\n \"type\": \"Expression\"\n },\n \"type\": \"Int32\"\n }\n }\n },\n \"linkedServiceName\": {\n \"referenceName\": \"SupportDatabase\",\n \"type\": \"LinkedServiceReference\"\n }\n }\n ]\n },\n {\n \"value\": \"Queued\",\n \"activities\": [\n {\n \"name\": \"Pipeline Status Queued - Running\",\n \"description\": \"Updates the current execution table with a pipeline status of running if the function outcome is queued.\",\n \"type\": \"SqlServerStoredProcedure\",\n \"dependsOn\": [],\n \"policy\": {\n \"timeout\": \"0.00:10:00\",\n \"retry\": 0,\n \"retryIntervalInSeconds\": 30,\n \"secureOutput\": false,\n \"secureInput\": false\n },\n \"userProperties\": [],\n \"typeProperties\": {\n \"storedProcedureName\": \"[[procfwk].[SetLogPipelineRunning]\",\n \"storedProcedureParameters\": {\n \"ExecutionId\": {\n \"value\": {\n \"value\": \"@item().LocalExecutionId\",\n \"type\": \"Expression\"\n },\n \"type\": \"Guid\"\n },\n \"PipelineId\": {\n \"value\": {\n \"value\": \"@item().PipelineId\",\n \"type\": \"Expression\"\n },\n \"type\": \"Int32\"\n },\n \"StageId\": {\n \"value\": {\n \"value\": \"@item().StageId\",\n \"type\": \"Expression\"\n },\n \"type\": \"Int32\"\n }\n }\n },\n \"linkedServiceName\": {\n \"referenceName\": \"SupportDatabase\",\n \"type\": \"LinkedServiceReference\"\n }\n }\n ]\n },\n {\n \"value\": \"InProgress\",\n \"activities\": [\n {\n \"name\": \"Pipeline Status InProgress - Running\",\n \"description\": \"Updates the current execution table with a pipeline status of running if the function outcome is in progress.\",\n \"type\": \"SqlServerStoredProcedure\",\n \"dependsOn\": [],\n \"policy\": {\n \"timeout\": \"0.00:10:00\",\n \"retry\": 0,\n \"retryIntervalInSeconds\": 30,\n \"secureOutput\": false,\n \"secureInput\": false\n },\n \"userProperties\": [],\n \"typeProperties\": {\n \"storedProcedureName\": \"[[procfwk].[SetLogPipelineRunning]\",\n \"storedProcedureParameters\": {\n \"ExecutionId\": {\n \"value\": {\n \"value\": \"@item().LocalExecutionId\",\n \"type\": \"Expression\"\n },\n \"type\": \"Guid\"\n },\n \"PipelineId\": {\n \"value\": {\n \"value\": \"@item().PipelineId\",\n \"type\": \"Expression\"\n },\n \"type\": \"Int32\"\n },\n \"StageId\": {\n \"value\": {\n \"value\": \"@item().StageId\",\n \"type\": \"Expression\"\n },\n \"type\": \"Int32\"\n }\n }\n },\n \"linkedServiceName\": {\n \"referenceName\": \"SupportDatabase\",\n \"type\": \"LinkedServiceReference\"\n }\n }\n ]\n },\n {\n \"value\": \"Cancelled\",\n \"activities\": [\n {\n \"name\": \"Pipeline Status Cancelled\",\n \"description\": \"Updates the current execution table with a pipeline status of cancelled if the function outcome is cancelled.\",\n \"type\": \"SqlServerStoredProcedure\",\n \"dependsOn\": [],\n \"policy\": {\n \"timeout\": \"0.00:10:00\",\n \"retry\": 0,\n \"retryIntervalInSeconds\": 30,\n \"secureOutput\": false,\n \"secureInput\": false\n },\n \"userProperties\": [],\n \"typeProperties\": {\n \"storedProcedureName\": \"[[procfwk].[SetLogPipelineCancelled]\",\n \"storedProcedureParameters\": {\n \"ExecutionId\": {\n \"value\": {\n \"value\": \"@item().LocalExecutionId\",\n \"type\": \"Expression\"\n },\n \"type\": \"Guid\"\n },\n \"PipelineId\": {\n \"value\": {\n \"value\": \"@item().PipelineId\",\n \"type\": \"Expression\"\n },\n \"type\": \"Int32\"\n },\n \"StageId\": {\n \"value\": {\n \"value\": \"@item().StageId\",\n \"type\": \"Expression\"\n },\n \"type\": \"Int32\"\n },\n \"CleanUpRun\": {\n \"value\": {\n \"value\": \"@bool(1)\",\n \"type\": \"Expression\"\n },\n \"type\": \"Boolean\"\n }\n }\n },\n \"linkedServiceName\": {\n \"referenceName\": \"SupportDatabase\",\n \"type\": \"LinkedServiceReference\"\n }\n }\n ]\n }\n ],\n \"defaultActivities\": [\n {\n \"name\": \"Pipeline Status Unknown\",\n \"description\": \"Updates the current execution table with a pipeline status of unknown if the function returns an unexpected outcome.\",\n \"type\": \"SqlServerStoredProcedure\",\n \"dependsOn\": [],\n \"policy\": {\n \"timeout\": \"0.00:10:00\",\n \"retry\": 0,\n \"retryIntervalInSeconds\": 30,\n \"secureOutput\": false,\n \"secureInput\": false\n },\n \"userProperties\": [],\n \"typeProperties\": {\n \"storedProcedureName\": \"[[procfwk].[SetLogPipelineUnknown]\",\n \"storedProcedureParameters\": {\n \"ExecutionId\": {\n \"value\": {\n \"value\": \"@item().LocalExecutionId\",\n \"type\": \"Expression\"\n },\n \"type\": \"Guid\"\n },\n \"PipelineId\": {\n \"value\": {\n \"value\": \"@item().PipelineId\",\n \"type\": \"Expression\"\n },\n \"type\": \"Int32\"\n },\n \"StageId\": {\n \"value\": {\n \"value\": \"@item().StageId\",\n \"type\": \"Expression\"\n },\n \"type\": \"Int32\"\n },\n \"CleanUpRun\": {\n \"value\": {\n \"value\": \"@bool(1)\",\n \"type\": \"Expression\"\n },\n \"type\": \"Boolean\"\n }\n }\n },\n \"linkedServiceName\": {\n \"referenceName\": \"SupportDatabase\",\n \"type\": \"LinkedServiceReference\"\n }\n }\n ]\n }\n },\n {\n \"name\": \"Set Last Check DateTime\",\n \"description\": \"Update the current execution table with a date time from when the function last checked the pipeline status.\",\n \"type\": \"SqlServerStoredProcedure\",\n \"dependsOn\": [\n {\n \"activity\": \"Get Pipeline Status\",\n \"dependencyConditions\": [\n \"Succeeded\"\n ]\n }\n ],\n \"policy\": {\n \"timeout\": \"0.00:10:00\",\n \"retry\": 0,\n \"retryIntervalInSeconds\": 30,\n \"secureOutput\": false,\n \"secureInput\": false\n },\n \"userProperties\": [],\n \"typeProperties\": {\n \"storedProcedureName\": \"[[procfwk].[SetLogPipelineLastStatusCheck]\",\n \"storedProcedureParameters\": {\n \"ExecutionId\": {\n \"value\": {\n \"value\": \"@item().LocalExecutionId\",\n \"type\": \"Expression\"\n },\n \"type\": \"Guid\"\n },\n \"PipelineId\": {\n \"value\": {\n \"value\": \"@item().PipelineId\",\n \"type\": \"Expression\"\n },\n \"type\": \"Int32\"\n },\n \"StageId\": {\n \"value\": {\n \"value\": \"@item().StageId\",\n \"type\": \"Expression\"\n },\n \"type\": \"Int32\"\n }\n }\n },\n \"linkedServiceName\": {\n \"referenceName\": \"SupportDatabase\",\n \"type\": \"LinkedServiceReference\"\n }\n }\n ]\n }\n },\n {\n \"name\": \"Execute Precursor\",\n \"description\": \"Uses the database property value ExecutionPrecursorProc to run any custom logic against the metadata database before the execution run starts.\",\n \"type\": \"SqlServerStoredProcedure\",\n \"dependsOn\": [\n {\n \"activity\": \"Is Parent Already Running\",\n \"dependencyConditions\": [\n \"Succeeded\"\n ]\n }\n ],\n \"policy\": {\n \"timeout\": \"0.00:10:00\",\n \"retry\": 0,\n \"retryIntervalInSeconds\": 30,\n \"secureOutput\": false,\n \"secureInput\": false\n },\n \"userProperties\": [],\n \"typeProperties\": {\n \"storedProcedureName\": \"[[procfwk].[ExecutePrecursorProcedure]\"\n },\n \"linkedServiceName\": {\n \"referenceName\": \"SupportDatabase\",\n \"type\": \"LinkedServiceReference\"\n }\n },\n {\n \"name\": \"Set Execution Id\",\n \"description\": \"Set the local execution Id to a pipeline variable for each in several downstream activities.\",\n \"type\": \"SetVariable\",\n \"dependsOn\": [\n {\n \"activity\": \"Execution Wrapper\",\n \"dependencyConditions\": [\n \"Succeeded\"\n ]\n }\n ],\n \"userProperties\": [],\n \"typeProperties\": {\n \"variableName\": \"ExecutionId\",\n \"value\": {\n \"value\": \"@activity('Execution Wrapper').output.firstRow.ExecutionId\",\n \"type\": \"Expression\"\n }\n }\n },\n {\n \"name\": \"Is Parent Already Running\",\n \"description\": \"Establish before anything else if the parent pipeline is already running. Batch execution aware.\",\n \"type\": \"ExecutePipeline\",\n \"dependsOn\": [],\n \"userProperties\": [],\n \"typeProperties\": {\n \"pipeline\": {\n \"referenceName\": \"Check For Running Pipeline\",\n \"type\": \"PipelineReference\"\n },\n \"waitOnCompletion\": true,\n \"parameters\": {\n \"BatchName\": {\n \"value\": \"@pipeline().parameters.BatchName\",\n \"type\": \"Expression\"\n },\n \"PipelineName\": {\n \"value\": \"@pipeline().Pipeline\",\n \"type\": \"Expression\"\n },\n \"ThisRunId\": {\n \"value\": \"@pipeline().RunId\",\n \"type\": \"Expression\"\n }\n }\n }\n },\n {\n \"name\": \"Check Metadata Integrity\",\n \"description\": \"Performs a series of checks on all metadata held in the framework SQLDB. This is intended to raise errors before an execution run even starts.\",\n \"type\": \"SqlServerStoredProcedure\",\n \"dependsOn\": [\n {\n \"activity\": \"Execute Precursor\",\n \"dependencyConditions\": [\n \"Succeeded\"\n ]\n }\n ],\n \"policy\": {\n \"timeout\": \"0.00:10:00\",\n \"retry\": 0,\n \"retryIntervalInSeconds\": 30,\n \"secureOutput\": false,\n \"secureInput\": false\n },\n \"userProperties\": [],\n \"typeProperties\": {\n \"storedProcedureName\": \"[[procfwk].[CheckMetadataIntegrity]\",\n \"storedProcedureParameters\": {\n \"BatchName\": {\n \"value\": {\n \"value\": \"@pipeline().parameters.BatchName\",\n \"type\": \"Expression\"\n },\n \"type\": \"String\"\n },\n \"DebugMode\": {\n \"value\": {\n \"value\": \"@bool(0)\",\n \"type\": \"Expression\"\n },\n \"type\": \"Boolean\"\n }\n }\n },\n \"linkedServiceName\": {\n \"referenceName\": \"SupportDatabase\",\n \"type\": \"LinkedServiceReference\"\n }\n }\n ],\n \"parameters\": {\n \"BatchName\": {\n \"type\": \"string\",\n \"defaultValue\": \"NotUsed\"\n }\n },\n \"variables\": {\n \"ExecutionId\": {\n \"type\": \"String\"\n }\n },\n \"folder\": {\n \"name\": \"_ProcFwk\"\n },\n \"annotations\": [\n \"procfwk\",\n \"Parent\"\n ]\n },\n \"dependsOn\": [\n \"[concat(variables('factoryId'), '/datasets/GetSetMetadata')]\",\n \"[concat(variables('factoryId'), '/linkedServices/SupportDatabase')]\",\n \"[concat(variables('factoryId'), '/pipelines/Check For Running Pipeline')]\",\n \"[concat(variables('factoryId'), '/pipelines/03-Child')]\",\n \"[concat(variables('factoryId'), '/linkedServices/FrameworkFunctions')]\"\n ]\n },\n {\n \"name\": \"[concat(parameters('factoryName'), '/03-Child')]\",\n \"type\": \"Microsoft.DataFactory/factories/pipelines\",\n \"apiVersion\": \"2018-06-01\",\n \"properties\": {\n \"description\": \"procfwk child pipeline used to execute Worker pipelines within a given execution stage. This pipeline will be called once for each stage, then execute all Workers in parallel.\",\n \"activities\": [\n {\n \"name\": \"Get Pipelines\",\n \"description\": \"Returns all pipelines from the metadata to be executed within a given processing stage.\",\n \"type\": \"Lookup\",\n \"dependsOn\": [],\n \"policy\": {\n \"timeout\": \"0.00:10:00\",\n \"retry\": 0,\n \"retryIntervalInSeconds\": 30,\n \"secureOutput\": false,\n \"secureInput\": false\n },\n \"userProperties\": [],\n \"typeProperties\": {\n \"source\": {\n \"type\": \"AzureSqlSource\",\n \"sqlReaderStoredProcedureName\": \"[[procfwk].[GetPipelinesInStage]\",\n \"storedProcedureParameters\": {\n \"ExecutionId\": {\n \"type\": \"Guid\",\n \"value\": {\n \"value\": \"@pipeline().parameters.ExecutionId\",\n \"type\": \"Expression\"\n }\n },\n \"StageId\": {\n \"type\": \"Int32\",\n \"value\": {\n \"value\": \"@pipeline().parameters.StageId\",\n \"type\": \"Expression\"\n }\n }\n },\n \"queryTimeout\": \"02:00:00\",\n \"partitionOption\": \"None\"\n },\n \"dataset\": {\n \"referenceName\": \"GetSetMetadata\",\n \"type\": \"DatasetReference\",\n \"parameters\": {}\n },\n \"firstRowOnly\": false\n }\n },\n {\n \"name\": \"Execute Pipelines\",\n \"description\": \"Second level ForEach to run in parallel all pipelines within the stage. Items for iteration passed from the Get Pipelines lookup activity.\",\n \"type\": \"ForEach\",\n \"dependsOn\": [\n {\n \"activity\": \"Get Pipelines\",\n \"dependencyConditions\": [\n \"Succeeded\"\n ]\n }\n ],\n \"userProperties\": [],\n \"typeProperties\": {\n \"items\": {\n \"value\": \"@activity('Get Pipelines').output.value\",\n \"type\": \"Expression\"\n },\n \"isSequential\": false,\n \"batchCount\": 50,\n \"activities\": [\n {\n \"name\": \"Worker Pipeline Executor\",\n \"description\": \"Run the required worker pipeline and wait for its completion. Update metadata once done.\",\n \"type\": \"ExecutePipeline\",\n \"dependsOn\": [],\n \"userProperties\": [],\n \"typeProperties\": {\n \"pipeline\": {\n \"referenceName\": \"04-Infant\",\n \"type\": \"PipelineReference\"\n },\n \"waitOnCompletion\": true,\n \"parameters\": {\n \"executionId\": {\n \"value\": \"@pipeline().parameters.ExecutionId\",\n \"type\": \"Expression\"\n },\n \"stageId\": {\n \"value\": \"@pipeline().parameters.StageId\",\n \"type\": \"Expression\"\n },\n \"pipelineId\": {\n \"value\": \"@item().PipelineId\",\n \"type\": \"Expression\"\n }\n }\n }\n }\n ]\n }\n }\n ],\n \"parameters\": {\n \"StageId\": {\n \"type\": \"int\"\n },\n \"ExecutionId\": {\n \"type\": \"string\"\n }\n },\n \"folder\": {\n \"name\": \"_ProcFwk\"\n },\n \"annotations\": [\n \"procfwk\",\n \"Child\"\n ]\n },\n \"dependsOn\": [\n \"[concat(variables('factoryId'), '/datasets/GetSetMetadata')]\",\n \"[concat(variables('factoryId'), '/pipelines/04-Infant')]\"\n ]\n },\n {\n \"name\": \"[concat(parameters('factoryName'), '/04-Infant')]\",\n \"type\": \"Microsoft.DataFactory/factories/pipelines\",\n \"apiVersion\": \"2018-06-01\",\n \"properties\": {\n \"description\": \"procfwk infant pipeline used to check when the processing pipeline called by the Child completes and passes the resulting status back to the metadata database.\",\n \"activities\": [\n {\n \"name\": \"Execute Worker Pipeline\",\n \"description\": \"The lowest level executor with the metadata framework to call existing processing pipelines within Data Factory. The function called will block processing and wait for an outcome.\",\n \"type\": \"AzureFunctionActivity\",\n \"dependsOn\": [\n {\n \"activity\": \"Log Pipeline Running\",\n \"dependencyConditions\": [\n \"Succeeded\"\n ]\n },\n {\n \"activity\": \"Get Pipeline Params\",\n \"dependencyConditions\": [\n \"Succeeded\"\n ]\n }\n ],\n \"policy\": {\n \"timeout\": \"0.00:10:00\",\n \"retry\": 0,\n \"retryIntervalInSeconds\": 30,\n \"secureOutput\": false,\n \"secureInput\": true\n },\n \"userProperties\": [],\n \"typeProperties\": {\n \"functionName\": \"ExecutePipeline\",\n \"method\": \"POST\",\n \"headers\": {},\n \"body\": {\n \"value\": \"@concat('\\n{\\n \\\"tenantId\\\": \\\"',variables('WorkerCoreDetails')[0].tenantId,'\\\",\\n \\\"applicationId\\\": \\\"',variables('WorkerCoreDetails')[0].applicationId,'\\\",\\n \\\"authenticationKey\\\": \\\"',variables('WorkerCoreDetails')[0].authenticationKey,'\\\",\\n \\\"subscriptionId\\\": \\\"',variables('WorkerCoreDetails')[0].subscriptionId,'\\\",\\n \\\"resourceGroupName\\\": \\\"',variables('WorkerCoreDetails')[0].resourceGroupName,'\\\",\\n\\t\\\"orchestratorName\\\": \\\"',variables('WorkerCoreDetails')[0].orchestratorName,'\\\",\\n \\\"orchestratorType\\\": \\\"',variables('WorkerCoreDetails')[0].orchestratorType,'\\\",\\n \\\"pipelineName\\\": \\\"',variables('WorkerCoreDetails')[0].pipelineName,'\\\"',activity('Get Pipeline Params').output.firstRow.Params,'\\n}')\",\n \"type\": \"Expression\"\n }\n },\n \"linkedServiceName\": {\n \"referenceName\": \"FrameworkFunctions\",\n \"type\": \"LinkedServiceReference\"\n }\n },\n {\n \"name\": \"Get Pipeline Params\",\n \"description\": \"Returns any parameters from metadata required for the processing pipeline being called. The output can be an empty string if no parameters are required.\",\n \"type\": \"Lookup\",\n \"dependsOn\": [],\n \"policy\": {\n \"timeout\": \"0.00:10:00\",\n \"retry\": 0,\n \"retryIntervalInSeconds\": 30,\n \"secureOutput\": false,\n \"secureInput\": false\n },\n \"userProperties\": [],\n \"typeProperties\": {\n \"source\": {\n \"type\": \"AzureSqlSource\",\n \"sqlReaderStoredProcedureName\": \"[[procfwk].[GetPipelineParameters]\",\n \"storedProcedureParameters\": {\n \"PipelineId\": {\n \"type\": \"Int32\",\n \"value\": {\n \"value\": \"@pipeline().parameters.pipelineId\",\n \"type\": \"Expression\"\n }\n }\n },\n \"queryTimeout\": \"02:00:00\",\n \"partitionOption\": \"None\"\n },\n \"dataset\": {\n \"referenceName\": \"GetSetMetadata\",\n \"type\": \"DatasetReference\",\n \"parameters\": {}\n }\n }\n },\n {\n \"name\": \"Log Pipeline Running\",\n \"description\": \"Sets the current pipeline with a status of running within the current execution database table.\",\n \"type\": \"SqlServerStoredProcedure\",\n \"dependsOn\": [\n {\n \"activity\": \"Is Target Worker Validate\",\n \"dependencyConditions\": [\n \"Succeeded\"\n ]\n }\n ],\n \"policy\": {\n \"timeout\": \"0.00:10:00\",\n \"retry\": 0,\n \"retryIntervalInSeconds\": 30,\n \"secureOutput\": false,\n \"secureInput\": false\n },\n \"userProperties\": [],\n \"typeProperties\": {\n \"storedProcedureName\": \"[[procfwk].[SetLogPipelineRunning]\",\n \"storedProcedureParameters\": {\n \"ExecutionId\": {\n \"value\": {\n \"value\": \"@pipeline().parameters.ExecutionId\",\n \"type\": \"Expression\"\n },\n \"type\": \"Guid\"\n },\n \"PipelineId\": {\n \"value\": {\n \"value\": \"@pipeline().parameters.pipelineId\",\n \"type\": \"Expression\"\n },\n \"type\": \"Int32\"\n },\n \"StageId\": {\n \"value\": {\n \"value\": \"@pipeline().parameters.StageId\",\n \"type\": \"Expression\"\n },\n \"type\": \"Int32\"\n }\n }\n },\n \"linkedServiceName\": {\n \"referenceName\": \"SupportDatabase\",\n \"type\": \"LinkedServiceReference\"\n }\n },\n {\n \"name\": \"Log Execute Function Activity Failure\",\n \"description\": \"Handle true failures from calling out to the Azure Function and update the current execution table accordingly so a restart can occur.\",\n \"type\": \"SqlServerStoredProcedure\",\n \"dependsOn\": [\n {\n \"activity\": \"Execute Worker Pipeline\",\n \"dependencyConditions\": [\n \"Failed\"\n ]\n }\n ],\n \"policy\": {\n \"timeout\": \"0.00:10:00\",\n \"retry\": 0,\n \"retryIntervalInSeconds\": 30,\n \"secureOutput\": false,\n \"secureInput\": false\n },\n \"userProperties\": [],\n \"typeProperties\": {\n \"storedProcedureName\": \"[[procfwk].[SetLogActivityFailed]\",\n \"storedProcedureParameters\": {\n \"ExecutionId\": {\n \"value\": {\n \"value\": \"@pipeline().parameters.ExecutionId\",\n \"type\": \"Expression\"\n },\n \"type\": \"Guid\"\n },\n \"PipelineId\": {\n \"value\": {\n \"value\": \"@pipeline().parameters.pipelineId\",\n \"type\": \"Expression\"\n },\n \"type\": \"Int32\"\n },\n \"StageId\": {\n \"value\": {\n \"value\": \"@pipeline().parameters.StageId\",\n \"type\": \"Expression\"\n },\n \"type\": \"Int32\"\n },\n \"CallingActivity\": {\n \"value\": \"ExecuteWorkerPipeline\",\n \"type\": \"String\"\n }\n }\n },\n \"linkedServiceName\": {\n \"referenceName\": \"SupportDatabase\",\n \"type\": \"LinkedServiceReference\"\n }\n },\n {\n \"name\": \"Update Run Id\",\n \"description\": \"Provide the actual ADF run ID back to the current execution table for long term logging and alignment between the metadata other Azure monitoring tools.\",\n \"type\": \"SqlServerStoredProcedure\",\n \"dependsOn\": [\n {\n \"activity\": \"Set Run Id\",\n \"dependencyConditions\": [\n \"Succeeded\"\n ]\n }\n ],\n \"policy\": {\n \"timeout\": \"0.00:10:00\",\n \"retry\": 0,\n \"retryIntervalInSeconds\": 30,\n \"secureOutput\": false,\n \"secureInput\": false\n },\n \"userProperties\": [],\n \"typeProperties\": {\n \"storedProcedureName\": \"[[procfwk].[SetLogPipelineRunId]\",\n \"storedProcedureParameters\": {\n \"ExecutionId\": {\n \"value\": {\n \"value\": \"@pipeline().parameters.ExecutionId\",\n \"type\": \"Expression\"\n },\n \"type\": \"Guid\"\n },\n \"PipelineId\": {\n \"value\": {\n \"value\": \"@pipeline().parameters.pipelineId\",\n \"type\": \"Expression\"\n },\n \"type\": \"Int32\"\n },\n \"RunId\": {\n \"value\": {\n \"value\": \"@variables('WorkerRunId')\",\n \"type\": \"Expression\"\n },\n \"type\": \"Guid\"\n },\n \"StageId\": {\n \"value\": {\n \"value\": \"@pipeline().parameters.StageId\",\n \"type\": \"Expression\"\n },\n \"type\": \"Int32\"\n }\n }\n },\n \"linkedServiceName\": {\n \"referenceName\": \"SupportDatabase\",\n \"type\": \"LinkedServiceReference\"\n }\n },\n {\n \"name\": \"Check For Alerts\",\n \"description\": \"Checks the properties tables and if any recipients in the database require alerts sending for the current pipeline ID.\",\n \"type\": \"Lookup\",\n \"dependsOn\": [\n {\n \"activity\": \"Update Run Id\",\n \"dependencyConditions\": [\n \"Succeeded\"\n ]\n },\n {\n \"activity\": \"Set Pipeline Result\",\n \"dependencyConditions\": [\n \"Completed\"\n ]\n }\n ],\n \"policy\": {\n \"timeout\": \"0.00:10:00\",\n \"retry\": 0,\n \"retryIntervalInSeconds\": 30,\n \"secureOutput\": false,\n \"secureInput\": false\n },\n \"userProperties\": [],\n \"typeProperties\": {\n \"source\": {\n \"type\": \"AzureSqlSource\",\n \"sqlReaderStoredProcedureName\": \"[[procfwk].[CheckForEmailAlerts]\",\n \"storedProcedureParameters\": {\n \"PipelineId\": {\n \"type\": \"Int32\",\n \"value\": {\n \"value\": \"@pipeline().parameters.pipelineId\",\n \"type\": \"Expression\"\n }\n }\n },\n \"queryTimeout\": \"02:00:00\",\n \"partitionOption\": \"None\"\n },\n \"dataset\": {\n \"referenceName\": \"GetSetMetadata\",\n \"type\": \"DatasetReference\",\n \"parameters\": {}\n },\n \"firstRowOnly\": true\n }\n },\n {\n \"name\": \"Send Alerts\",\n \"description\": \"True = alerts need sending.\\nFalse = do nothing.\",\n \"type\": \"IfCondition\",\n \"dependsOn\": [\n {\n \"activity\": \"Check For Alerts\",\n \"dependencyConditions\": [\n \"Succeeded\"\n ]\n }\n ],\n \"userProperties\": [],\n \"typeProperties\": {\n \"expression\": {\n \"value\": \"@activity('Check For Alerts').output.firstRow.SendAlerts\",\n \"type\": \"Expression\"\n },\n \"ifTrueActivities\": [\n {\n \"name\": \"Get Email Parts\",\n \"description\": \"Return all required content from the metadata database to send an email alerting using the procfwk. The lookup returns the exact content for the function body request.\",\n \"type\": \"Lookup\",\n \"dependsOn\": [],\n \"policy\": {\n \"timeout\": \"0.00:10:00\",\n \"retry\": 0,\n \"retryIntervalInSeconds\": 30,\n \"secureOutput\": true,\n \"secureInput\": false\n },\n \"userProperties\": [],\n \"typeProperties\": {\n \"source\": {\n \"type\": \"AzureSqlSource\",\n \"sqlReaderStoredProcedureName\": \"[[procfwk].[GetEmailAlertParts]\",\n \"storedProcedureParameters\": {\n \"PipelineId\": {\n \"type\": \"Int32\",\n \"value\": {\n \"value\": \"@pipeline().parameters.pipelineId\",\n \"type\": \"Expression\"\n }\n }\n },\n \"queryTimeout\": \"02:00:00\",\n \"partitionOption\": \"None\"\n },\n \"dataset\": {\n \"referenceName\": \"GetSetMetadata\",\n \"type\": \"DatasetReference\",\n \"parameters\": {}\n },\n \"firstRowOnly\": true\n }\n },\n {\n \"name\": \"Call Email Sender\",\n \"description\": \"Pass off email request to Utils Send Email pipeline.\",\n \"type\": \"ExecutePipeline\",\n \"dependsOn\": [\n {\n \"activity\": \"Get Email Parts\",\n \"dependencyConditions\": [\n \"Succeeded\"\n ]\n }\n ],\n \"userProperties\": [],\n \"typeProperties\": {\n \"pipeline\": {\n \"referenceName\": \"Email Sender\",\n \"type\": \"PipelineReference\"\n },\n \"waitOnCompletion\": true,\n \"parameters\": {\n \"Recipients\": {\n \"value\": \"@activity('Get Email Parts').output.firstRow.emailRecipients\",\n \"type\": \"Expression\"\n },\n \"CcRecipients\": {\n \"value\": \"@activity('Get Email Parts').output.firstRow.emailCcRecipients\",\n \"type\": \"Expression\"\n },\n \"BccRecipients\": {\n \"value\": \"@activity('Get Email Parts').output.firstRow.emailBccRecipients\",\n \"type\": \"Expression\"\n },\n \"Subject\": {\n \"value\": \"@activity('Get Email Parts').output.firstRow.emailSubject\",\n \"type\": \"Expression\"\n },\n \"Body\": {\n \"value\": \"@activity('Get Email Parts').output.firstRow.emailBody\",\n \"type\": \"Expression\"\n },\n \"Importance\": {\n \"value\": \"@activity('Get Email Parts').output.firstRow.emailImportance\",\n \"type\": \"Expression\"\n }\n }\n }\n }\n ]\n }\n },\n {\n \"name\": \"Wait Until Pipeline Completes\",\n \"description\": \"Loops until the Worker pipeline called completes.\\n\\nSimple status:\\n- Running = new iteration.\\n- Done = break.\",\n \"type\": \"Until\",\n \"dependsOn\": [\n {\n \"activity\": \"Get Wait Duration\",\n \"dependencyConditions\": [\n \"Succeeded\"\n ]\n },\n {\n \"activity\": \"Execute Worker Pipeline\",\n \"dependencyConditions\": [\n \"Succeeded\"\n ]\n },\n {\n \"activity\": \"Set Run Id\",\n \"dependencyConditions\": [\n \"Succeeded\"\n ]\n }\n ],\n \"userProperties\": [],\n \"typeProperties\": {\n \"expression\": {\n \"value\": \"@variables('WorkerPipelineState')\",\n \"type\": \"Expression\"\n },\n \"activities\": [\n {\n \"name\": \"Get Worker Pipeline Status\",\n \"description\": \"Checks the status of a given processing pipeline and provides the value for the downstream framework activities to act upon.\",\n \"type\": \"AzureFunctionActivity\",\n \"dependsOn\": [],\n \"policy\": {\n \"timeout\": \"0.00:10:00\",\n \"retry\": 0,\n \"retryIntervalInSeconds\": 30,\n \"secureOutput\": false,\n \"secureInput\": true\n },\n \"userProperties\": [],\n \"typeProperties\": {\n \"functionName\": \"CheckPipelineStatus\",\n \"method\": \"POST\",\n \"headers\": {},\n \"body\": {\n \"value\": \"@concat('\\n{\\n \\\"tenantId\\\": \\\"',variables('WorkerCoreDetails')[0].tenantId,'\\\",\\n \\\"applicationId\\\": \\\"',variables('WorkerCoreDetails')[0].applicationId,'\\\",\\n \\\"authenticationKey\\\": \\\"',variables('WorkerCoreDetails')[0].authenticationKey,'\\\",\\n \\\"subscriptionId\\\": \\\"',variables('WorkerCoreDetails')[0].subscriptionId,'\\\",\\n \\\"resourceGroupName\\\": \\\"',variables('WorkerCoreDetails')[0].resourceGroupName,'\\\",\\n\\t\\\"orchestratorName\\\": \\\"',variables('WorkerCoreDetails')[0].orchestratorName,'\\\",\\n \\\"orchestratorType\\\": \\\"',variables('WorkerCoreDetails')[0].orchestratorType,'\\\",\\n \\\"pipelineName\\\": \\\"',variables('WorkerCoreDetails')[0].pipelineName,'\\\",\\n \\\"runId\\\": \\\"',variables('WorkerRunId'),'\\\"\\n}')\",\n \"type\": \"Expression\"\n }\n },\n \"linkedServiceName\": {\n \"referenceName\": \"FrameworkFunctions\",\n \"type\": \"LinkedServiceReference\"\n }\n },\n {\n \"name\": \"Wait If Running\",\n \"description\": \"True = Do nothing.\\nFalse = Wait, before the next iteration.\",\n \"type\": \"IfCondition\",\n \"dependsOn\": [\n {\n \"activity\": \"Set Worker State\",\n \"dependencyConditions\": [\n \"Succeeded\"\n ]\n }\n ],\n \"userProperties\": [],\n \"typeProperties\": {\n \"expression\": {\n \"value\": \"@variables('WorkerPipelineState')\",\n \"type\": \"Expression\"\n },\n \"ifFalseActivities\": [\n {\n \"name\": \"Wait for Pipeline\",\n \"description\": \"The processing pipeline is still running so Wait before checking its status again.\",\n \"type\": \"Wait\",\n \"dependsOn\": [],\n \"userProperties\": [],\n \"typeProperties\": {\n \"waitTimeInSeconds\": {\n \"value\": \"@activity('Get Wait Duration').output.firstRow.PropertyValue\",\n \"type\": \"Expression\"\n }\n }\n }\n ]\n }\n },\n {\n \"name\": \"Set Last Check DateTime\",\n \"description\": \"Update the current execution table with a date time from when the Worker pipeline status was last checked as part of the Until iterations.\",\n \"type\": \"SqlServerStoredProcedure\",\n \"dependsOn\": [\n {\n \"activity\": \"Get Worker Pipeline Status\",\n \"dependencyConditions\": [\n \"Succeeded\"\n ]\n }\n ],\n \"policy\": {\n \"timeout\": \"0.00:10:00\",\n \"retry\": 0,\n \"retryIntervalInSeconds\": 30,\n \"secureOutput\": false,\n \"secureInput\": false\n },\n \"userProperties\": [],\n \"typeProperties\": {\n \"storedProcedureName\": \"[[procfwk].[SetLogPipelineLastStatusCheck]\",\n \"storedProcedureParameters\": {\n \"ExecutionId\": {\n \"value\": {\n \"value\": \"@pipeline().parameters.executionId\",\n \"type\": \"Expression\"\n },\n \"type\": \"Guid\"\n },\n \"PipelineId\": {\n \"value\": {\n \"value\": \"@pipeline().parameters.pipelineId\",\n \"type\": \"Expression\"\n },\n \"type\": \"Int32\"\n },\n \"StageId\": {\n \"value\": {\n \"value\": \"@pipeline().parameters.stageId\",\n \"type\": \"Expression\"\n },\n \"type\": \"Int32\"\n }\n }\n },\n \"linkedServiceName\": {\n \"referenceName\": \"SupportDatabase\",\n \"type\": \"LinkedServiceReference\"\n }\n },\n {\n \"name\": \"Log Check Function Activity Failure\",\n \"description\": \"Report to the current execution table that the framework pipeline activity has failed. This failure is outside of the scope of the framework and is probably related to a wider platform problem.\",\n \"type\": \"SqlServerStoredProcedure\",\n \"dependsOn\": [\n {\n \"activity\": \"Get Worker Pipeline Status\",\n \"dependencyConditions\": [\n \"Failed\"\n ]\n }\n ],\n \"policy\": {\n \"timeout\": \"0.00:10:00\",\n \"retry\": 0,\n \"retryIntervalInSeconds\": 30,\n \"secureOutput\": false,\n \"secureInput\": false\n },\n \"userProperties\": [],\n \"typeProperties\": {\n \"storedProcedureName\": \"[[procfwk].[SetLogActivityFailed]\",\n \"storedProcedureParameters\": {\n \"CallingActivity\": {\n \"value\": \"GetWorkerPipelineStatus\",\n \"type\": \"String\"\n },\n \"ExecutionId\": {\n \"value\": {\n \"value\": \"@pipeline().parameters.executionId\",\n \"type\": \"Expression\"\n },\n \"type\": \"Guid\"\n },\n \"PipelineId\": {\n \"value\": {\n \"value\": \"@pipeline().parameters.pipelineId\",\n \"type\": \"Expression\"\n },\n \"type\": \"Int32\"\n },\n \"StageId\": {\n \"value\": {\n \"value\": \"@pipeline().parameters.stageId\",\n \"type\": \"Expression\"\n },\n \"type\": \"Int32\"\n }\n }\n },\n \"linkedServiceName\": {\n \"referenceName\": \"SupportDatabase\",\n \"type\": \"LinkedServiceReference\"\n }\n },\n {\n \"name\": \"Set Worker State\",\n \"description\": \"Set the bool state of the Worker pipeline to be used by the Until and If expressions. True = Complete, False = Running.\",\n \"type\": \"SetVariable\",\n \"dependsOn\": [\n {\n \"activity\": \"Get Worker Pipeline Status\",\n \"dependencyConditions\": [\n \"Succeeded\"\n ]\n }\n ],\n \"userProperties\": [],\n \"typeProperties\": {\n \"variableName\": \"WorkerPipelineState\",\n \"value\": {\n \"value\": \"@equals('Complete',activity('Get Worker Pipeline Status').output.SimpleStatus)\",\n \"type\": \"Expression\"\n }\n }\n }\n ],\n \"timeout\": \"0.00:10:00\"\n }\n },\n {\n \"name\": \"Set Pipeline Result\",\n \"description\": \"Receives the outcome from the function execution for a given processing pipeline and updates the current execution table with different pipelines status values depending on the result (case).\",\n \"type\": \"Switch\",\n \"dependsOn\": [\n {\n \"activity\": \"Wait Until Pipeline Completes\",\n \"dependencyConditions\": [\n \"Completed\"\n ]\n }\n ],\n \"userProperties\": [],\n \"typeProperties\": {\n \"on\": {\n \"value\": \"@activity('Get Worker Pipeline Status').output.ActualStatus\",\n \"type\": \"Expression\"\n },\n \"cases\": [\n {\n \"value\": \"Succeeded\",\n \"activities\": [\n {\n \"name\": \"Pipeline Status Succeeded\",\n \"description\": \"Updates the current execution table with a pipeline status of success if the function outcome is succeeded.\",\n \"type\": \"SqlServerStoredProcedure\",\n \"dependsOn\": [],\n \"policy\": {\n \"timeout\": \"0.00:10:00\",\n \"retry\": 0,\n \"retryIntervalInSeconds\": 30,\n \"secureOutput\": false,\n \"secureInput\": false\n },\n \"userProperties\": [],\n \"typeProperties\": {\n \"storedProcedureName\": \"[[procfwk].[SetLogPipelineSuccess]\",\n \"storedProcedureParameters\": {\n \"ExecutionId\": {\n \"value\": {\n \"value\": \"@pipeline().parameters.executionId\",\n \"type\": \"Expression\"\n },\n \"type\": \"Guid\"\n },\n \"PipelineId\": {\n \"value\": {\n \"value\": \"@pipeline().parameters.pipelineId\",\n \"type\": \"Expression\"\n },\n \"type\": \"Int32\"\n },\n \"StageId\": {\n \"value\": {\n \"value\": \"@pipeline().parameters.stageId\",\n \"type\": \"Expression\"\n },\n \"type\": \"Int32\"\n }\n }\n },\n \"linkedServiceName\": {\n \"referenceName\": \"SupportDatabase\",\n \"type\": \"LinkedServiceReference\"\n }\n }\n ]\n },\n {\n \"value\": \"Failed\",\n \"activities\": [\n {\n \"name\": \"Pipeline Status Failed\",\n \"description\": \"Updates the current execution table with a pipeline status of failed if the function outcome is failed. Also blocks pipelines in the downstream execution stage.\",\n \"type\": \"SqlServerStoredProcedure\",\n \"dependsOn\": [],\n \"policy\": {\n \"timeout\": \"0.00:10:00\",\n \"retry\": 0,\n \"retryIntervalInSeconds\": 30,\n \"secureOutput\": false,\n \"secureInput\": false\n },\n \"userProperties\": [],\n \"typeProperties\": {\n \"storedProcedureName\": \"[[procfwk].[SetLogPipelineFailed]\",\n \"storedProcedureParameters\": {\n \"ExecutionId\": {\n \"value\": {\n \"value\": \"@pipeline().parameters.executionId\",\n \"type\": \"Expression\"\n },\n \"type\": \"Guid\"\n },\n \"PipelineId\": {\n \"value\": {\n \"value\": \"@pipeline().parameters.pipelineId\",\n \"type\": \"Expression\"\n },\n \"type\": \"Int32\"\n },\n \"RunId\": {\n \"value\": {\n \"value\": \"@variables('WorkerRunId')\",\n \"type\": \"Expression\"\n },\n \"type\": \"Guid\"\n },\n \"StageId\": {\n \"value\": {\n \"value\": \"@pipeline().parameters.stageId\",\n \"type\": \"Expression\"\n },\n \"type\": \"Int32\"\n }\n }\n },\n \"linkedServiceName\": {\n \"referenceName\": \"SupportDatabase\",\n \"type\": \"LinkedServiceReference\"\n }\n },\n {\n \"name\": \"Get Worker Pipeline Error Details\",\n \"description\": \"Get the activity error details for the run ID of the worker pipeline called. Returns an array of all errors.\",\n \"type\": \"AzureFunctionActivity\",\n \"dependsOn\": [],\n \"policy\": {\n \"timeout\": \"0.00:10:00\",\n \"retry\": 0,\n \"retryIntervalInSeconds\": 30,\n \"secureOutput\": false,\n \"secureInput\": true\n },\n \"userProperties\": [],\n \"typeProperties\": {\n \"functionName\": \"GetActivityErrors\",\n \"method\": \"POST\",\n \"headers\": {},\n \"body\": {\n \"value\": \"@concat('\\n{\\n \\\"tenantId\\\": \\\"',variables('WorkerCoreDetails')[0].tenantId,'\\\",\\n \\\"applicationId\\\": \\\"',variables('WorkerCoreDetails')[0].applicationId,'\\\",\\n \\\"authenticationKey\\\": \\\"',variables('WorkerCoreDetails')[0].authenticationKey,'\\\",\\n \\\"subscriptionId\\\": \\\"',variables('WorkerCoreDetails')[0].subscriptionId,'\\\",\\n \\\"resourceGroupName\\\": \\\"',variables('WorkerCoreDetails')[0].resourceGroupName,'\\\",\\n\\t\\\"orchestratorName\\\": \\\"',variables('WorkerCoreDetails')[0].orchestratorName,'\\\",\\n \\\"orchestratorType\\\": \\\"',variables('WorkerCoreDetails')[0].orchestratorType,'\\\",\\n \\\"pipelineName\\\": \\\"',variables('WorkerCoreDetails')[0].pipelineName,'\\\",\\n \\\"runId\\\": \\\"',variables('WorkerRunId'),'\\\"\\n}')\",\n \"type\": \"Expression\"\n }\n },\n \"linkedServiceName\": {\n \"referenceName\": \"FrameworkFunctions\",\n \"type\": \"LinkedServiceReference\"\n }\n },\n {\n \"name\": \"Log Error Details\",\n \"description\": \"Parses pipeline error details and persists them to the metadata database error log table.\",\n \"type\": \"SqlServerStoredProcedure\",\n \"dependsOn\": [\n {\n \"activity\": \"Get Worker Pipeline Error Details\",\n \"dependencyConditions\": [\n \"Succeeded\"\n ]\n }\n ],\n \"policy\": {\n \"timeout\": \"0.00:10:00\",\n \"retry\": 0,\n \"retryIntervalInSeconds\": 30,\n \"secureOutput\": false,\n \"secureInput\": false\n },\n \"userProperties\": [],\n \"typeProperties\": {\n \"storedProcedureName\": \"[[procfwk].[SetErrorLogDetails]\",\n \"storedProcedureParameters\": {\n \"JsonErrorDetails\": {\n \"value\": {\n \"value\": \"@string(activity('Get Worker Pipeline Error Details').output)\",\n \"type\": \"Expression\"\n },\n \"type\": \"String\"\n },\n \"LocalExecutionId\": {\n \"value\": {\n \"value\": \"@pipeline().parameters.executionId\",\n \"type\": \"Expression\"\n },\n \"type\": \"Guid\"\n }\n }\n },\n \"linkedServiceName\": {\n \"referenceName\": \"SupportDatabase\",\n \"type\": \"LinkedServiceReference\"\n }\n }\n ]\n },\n {\n \"value\": \"Cancelled\",\n \"activities\": [\n {\n \"name\": \"Pipeline Status Cancelled\",\n \"description\": \"Updates the current execution table with a pipeline status of cancelled if the function outcome is cancelled.\",\n \"type\": \"SqlServerStoredProcedure\",\n \"dependsOn\": [],\n \"policy\": {\n \"timeout\": \"0.00:10:00\",\n \"retry\": 0,\n \"retryIntervalInSeconds\": 30,\n \"secureOutput\": false,\n \"secureInput\": false\n },\n \"userProperties\": [],\n \"typeProperties\": {\n \"storedProcedureName\": \"[[procfwk].[SetLogPipelineCancelled]\",\n \"storedProcedureParameters\": {\n \"ExecutionId\": {\n \"value\": {\n \"value\": \"@pipeline().parameters.executionId\",\n \"type\": \"Expression\"\n },\n \"type\": \"Guid\"\n },\n \"PipelineId\": {\n \"value\": {\n \"value\": \"@pipeline().parameters.pipelineId\",\n \"type\": \"Expression\"\n },\n \"type\": \"Int32\"\n },\n \"StageId\": {\n \"value\": {\n \"value\": \"@pipeline().parameters.stageId\",\n \"type\": \"Expression\"\n },\n \"type\": \"Int32\"\n }\n }\n },\n \"linkedServiceName\": {\n \"referenceName\": \"SupportDatabase\",\n \"type\": \"LinkedServiceReference\"\n }\n }\n ]\n }\n ],\n \"defaultActivities\": [\n {\n \"name\": \"Pipeline Status Unknown\",\n \"description\": \"Updates the current execution table with a pipeline status of unknown if the function returns an unexpected outcome.\",\n \"type\": \"SqlServerStoredProcedure\",\n \"dependsOn\": [],\n \"policy\": {\n \"timeout\": \"0.00:10:00\",\n \"retry\": 0,\n \"retryIntervalInSeconds\": 30,\n \"secureOutput\": false,\n \"secureInput\": false\n },\n \"userProperties\": [],\n \"typeProperties\": {\n \"storedProcedureName\": \"[[procfwk].[SetLogPipelineUnknown]\",\n \"storedProcedureParameters\": {\n \"ExecutionId\": {\n \"value\": {\n \"value\": \"@pipeline().parameters.executionId\",\n \"type\": \"Expression\"\n },\n \"type\": \"Guid\"\n },\n \"PipelineId\": {\n \"value\": {\n \"value\": \"@pipeline().parameters.pipelineId\",\n \"type\": \"Expression\"\n },\n \"type\": \"Int32\"\n },\n \"StageId\": {\n \"value\": {\n \"value\": \"@pipeline().parameters.stageId\",\n \"type\": \"Expression\"\n },\n \"type\": \"Int32\"\n }\n }\n },\n \"linkedServiceName\": {\n \"referenceName\": \"SupportDatabase\",\n \"type\": \"LinkedServiceReference\"\n }\n }\n ]\n }\n },\n {\n \"name\": \"Get Wait Duration\",\n \"description\": \"Return wait duration in seconds from database properties table to be used during each Until iteration when the Worker pipeline is still running.\",\n \"type\": \"Lookup\",\n \"dependsOn\": [],\n \"policy\": {\n \"timeout\": \"0.00:10:00\",\n \"retry\": 0,\n \"retryIntervalInSeconds\": 30,\n \"secureOutput\": false,\n \"secureInput\": false\n },\n \"userProperties\": [],\n \"typeProperties\": {\n \"source\": {\n \"type\": \"AzureSqlSource\",\n \"sqlReaderStoredProcedureName\": \"[[procfwk].[GetPropertyValue]\",\n \"storedProcedureParameters\": {\n \"PropertyName\": {\n \"type\": \"String\",\n \"value\": \"PipelineStatusCheckDuration\"\n }\n },\n \"queryTimeout\": \"02:00:00\",\n \"partitionOption\": \"None\"\n },\n \"dataset\": {\n \"referenceName\": \"GetSetMetadata\",\n \"type\": \"DatasetReference\",\n \"parameters\": {}\n }\n }\n },\n {\n \"name\": \"Set Run Id\",\n \"description\": \"Set local variable from activity output once for value reuse in downstream activities.\",\n \"type\": \"SetVariable\",\n \"dependsOn\": [\n {\n \"activity\": \"Execute Worker Pipeline\",\n \"dependencyConditions\": [\n \"Succeeded\"\n ]\n }\n ],\n \"userProperties\": [],\n \"typeProperties\": {\n \"variableName\": \"WorkerRunId\",\n \"value\": {\n \"value\": \"@activity('Execute Worker Pipeline').output.RunId\",\n \"type\": \"Expression\"\n }\n }\n },\n {\n \"name\": \"Validate Pipeline\",\n \"description\": \"Query the target data factory and establish if the provided worker pipeline name is valid.\",\n \"type\": \"AzureFunctionActivity\",\n \"dependsOn\": [\n {\n \"activity\": \"Log Pipeline Validating\",\n \"dependencyConditions\": [\n \"Succeeded\"\n ]\n },\n {\n \"activity\": \"Capture Worker Core Details as an Array\",\n \"dependencyConditions\": [\n \"Succeeded\"\n ]\n }\n ],\n \"policy\": {\n \"timeout\": \"0.00:10:00\",\n \"retry\": 0,\n \"retryIntervalInSeconds\": 30,\n \"secureOutput\": false,\n \"secureInput\": true\n },\n \"userProperties\": [],\n \"typeProperties\": {\n \"functionName\": \"ValidatePipeline\",\n \"method\": \"POST\",\n \"headers\": {},\n \"body\": {\n \"value\": \"@concat('\\n{\\n \\\"tenantId\\\": \\\"',variables('WorkerCoreDetails')[0].tenantId,'\\\",\\n \\\"applicationId\\\": \\\"',variables('WorkerCoreDetails')[0].applicationId,'\\\",\\n \\\"authenticationKey\\\": \\\"',variables('WorkerCoreDetails')[0].authenticationKey,'\\\",\\n \\\"subscriptionId\\\": \\\"',variables('WorkerCoreDetails')[0].subscriptionId,'\\\",\\n \\\"resourceGroupName\\\": \\\"',variables('WorkerCoreDetails')[0].resourceGroupName,'\\\",\\n\\t\\\"orchestratorName\\\": \\\"',variables('WorkerCoreDetails')[0].orchestratorName,'\\\",\\n \\\"orchestratorType\\\": \\\"',variables('WorkerCoreDetails')[0].orchestratorType,'\\\",\\n \\\"pipelineName\\\": \\\"',variables('WorkerCoreDetails')[0].pipelineName,'\\\"\\n}')\",\n \"type\": \"Expression\"\n }\n },\n \"linkedServiceName\": {\n \"referenceName\": \"FrameworkFunctions\",\n \"type\": \"LinkedServiceReference\"\n }\n },\n {\n \"name\": \"Is Target Worker Validate\",\n \"description\": \"True = the worker pipeline name is valid.\\nFalse = the worker pipeline name is invalid. Raise an exception.\",\n \"type\": \"IfCondition\",\n \"dependsOn\": [\n {\n \"activity\": \"Validate Pipeline\",\n \"dependencyConditions\": [\n \"Succeeded\"\n ]\n }\n ],\n \"userProperties\": [],\n \"typeProperties\": {\n \"expression\": {\n \"value\": \"@bool(activity('Validate Pipeline').output.PipelineExists)\",\n \"type\": \"Expression\"\n },\n \"ifFalseActivities\": [\n {\n \"name\": \"Throw Exception - Invalid Infant\",\n \"description\": \"Throw an exception with details about the invalid worker pipeline name.\",\n \"type\": \"ExecutePipeline\",\n \"dependsOn\": [],\n \"userProperties\": [],\n \"typeProperties\": {\n \"pipeline\": {\n \"referenceName\": \"Throw Exception\",\n \"type\": \"PipelineReference\"\n },\n \"waitOnCompletion\": true,\n \"parameters\": {\n \"Message\": {\n \"value\": \"@concat('Worker pipeline [',variables('WorkerCoreDetails')[0].pipelineName,'] is not valid in target Orchestrator [',variables('WorkerCoreDetails')[0].orchestratorName,']')\",\n \"type\": \"Expression\"\n }\n }\n }\n },\n {\n \"name\": \"Update Execution With Invalid Worker\",\n \"description\": \"Update the current execution table with an informed status for the worker pipeline that couldn't be executed.\",\n \"type\": \"SqlServerStoredProcedure\",\n \"dependsOn\": [],\n \"policy\": {\n \"timeout\": \"7.00:00:00\",\n \"retry\": 0,\n \"retryIntervalInSeconds\": 30,\n \"secureOutput\": false,\n \"secureInput\": false\n },\n \"userProperties\": [],\n \"typeProperties\": {\n \"storedProcedureName\": \"[[procfwk].[SetLogActivityFailed]\",\n \"storedProcedureParameters\": {\n \"CallingActivity\": {\n \"value\": \"InvalidPipelineName\",\n \"type\": \"String\"\n },\n \"ExecutionId\": {\n \"value\": {\n \"value\": \"@pipeline().parameters.ExecutionId\",\n \"type\": \"Expression\"\n },\n \"type\": \"Guid\"\n },\n \"PipelineId\": {\n \"value\": {\n \"value\": \"@pipeline().parameters.pipelineId\",\n \"type\": \"Expression\"\n },\n \"type\": \"Int32\"\n },\n \"StageId\": {\n \"value\": {\n \"value\": \"@pipeline().parameters.StageId\",\n \"type\": \"Expression\"\n },\n \"type\": \"Int32\"\n }\n }\n },\n \"linkedServiceName\": {\n \"referenceName\": \"SupportDatabase\",\n \"type\": \"LinkedServiceReference\"\n }\n }\n ]\n }\n },\n {\n \"name\": \"Log Validate Function Activity Failure\",\n \"description\": \"Handle true failures from calling out to the Azure Function and update the current execution table accordingly so a restart can occur.\",\n \"type\": \"SqlServerStoredProcedure\",\n \"dependsOn\": [\n {\n \"activity\": \"Validate Pipeline\",\n \"dependencyConditions\": [\n \"Failed\"\n ]\n }\n ],\n \"policy\": {\n \"timeout\": \"0.00:10:00\",\n \"retry\": 0,\n \"retryIntervalInSeconds\": 30,\n \"secureOutput\": false,\n \"secureInput\": false\n },\n \"userProperties\": [],\n \"typeProperties\": {\n \"storedProcedureName\": \"[[procfwk].[SetLogActivityFailed]\",\n \"storedProcedureParameters\": {\n \"ExecutionId\": {\n \"value\": {\n \"value\": \"@pipeline().parameters.ExecutionId\",\n \"type\": \"Expression\"\n },\n \"type\": \"Guid\"\n },\n \"PipelineId\": {\n \"value\": {\n \"value\": \"@pipeline().parameters.pipelineId\",\n \"type\": \"Expression\"\n },\n \"type\": \"Int32\"\n },\n \"StageId\": {\n \"value\": {\n \"value\": \"@pipeline().parameters.StageId\",\n \"type\": \"Expression\"\n },\n \"type\": \"Int32\"\n },\n \"CallingActivity\": {\n \"value\": \"ValidatePipeline\",\n \"type\": \"String\"\n }\n }\n },\n \"linkedServiceName\": {\n \"referenceName\": \"SupportDatabase\",\n \"type\": \"LinkedServiceReference\"\n }\n },\n {\n \"name\": \"Log Pipeline Validating\",\n \"description\": \"Sets the current pipeline with a status of validating within the current execution database table.\",\n \"type\": \"SqlServerStoredProcedure\",\n \"dependsOn\": [],\n \"policy\": {\n \"timeout\": \"0.00:10:00\",\n \"retry\": 0,\n \"retryIntervalInSeconds\": 30,\n \"secureOutput\": false,\n \"secureInput\": false\n },\n \"userProperties\": [],\n \"typeProperties\": {\n \"storedProcedureName\": \"[[procfwk].[SetLogPipelineValidating]\",\n \"storedProcedureParameters\": {\n \"ExecutionId\": {\n \"value\": {\n \"value\": \"@pipeline().parameters.ExecutionId\",\n \"type\": \"Expression\"\n },\n \"type\": \"Guid\"\n },\n \"PipelineId\": {\n \"value\": {\n \"value\": \"@pipeline().parameters.pipelineId\",\n \"type\": \"Expression\"\n },\n \"type\": \"Int32\"\n },\n \"StageId\": {\n \"value\": {\n \"value\": \"@pipeline().parameters.StageId\",\n \"type\": \"Expression\"\n },\n \"type\": \"Int32\"\n }\n }\n },\n \"linkedServiceName\": {\n \"referenceName\": \"SupportDatabase\",\n \"type\": \"LinkedServiceReference\"\n }\n },\n {\n \"name\": \"Get Worker Core Details\",\n \"description\": \"Return worker pipeline information for metadata database. Including target data factory, pipeline name and resource group. Return the SPN ID and Secret for the worker pipeline being executed. Called at this level as each pipeline can have a different SPN.\",\n \"type\": \"Lookup\",\n \"dependsOn\": [],\n \"policy\": {\n \"timeout\": \"0.00:10:00\",\n \"retry\": 0,\n \"retryIntervalInSeconds\": 30,\n \"secureOutput\": true,\n \"secureInput\": false\n },\n \"userProperties\": [],\n \"typeProperties\": {\n \"source\": {\n \"type\": \"AzureSqlSource\",\n \"sqlReaderStoredProcedureName\": \"[[procfwk].[GetWorkerDetailsWrapper]\",\n \"storedProcedureParameters\": {\n \"ExecutionId\": {\n \"type\": \"Guid\",\n \"value\": {\n \"value\": \"@pipeline().parameters.executionId\",\n \"type\": \"Expression\"\n }\n },\n \"PipelineId\": {\n \"type\": \"Int32\",\n \"value\": {\n \"value\": \"@pipeline().parameters.pipelineId\",\n \"type\": \"Expression\"\n }\n },\n \"StageId\": {\n \"type\": \"Int32\",\n \"value\": {\n \"value\": \"@pipeline().parameters.stageId\",\n \"type\": \"Expression\"\n }\n }\n },\n \"queryTimeout\": \"02:00:00\",\n \"partitionOption\": \"None\"\n },\n \"dataset\": {\n \"referenceName\": \"GetSetMetadata\",\n \"type\": \"DatasetReference\",\n \"parameters\": {}\n }\n }\n },\n {\n \"name\": \"Capture Worker Core Details as an Array\",\n \"description\": \"Add all worker pipeline details to a local variable array that can be accessed by each function call requiring the values.\",\n \"type\": \"SetVariable\",\n \"dependsOn\": [\n {\n \"activity\": \"Get Worker Core Details\",\n \"dependencyConditions\": [\n \"Succeeded\"\n ]\n }\n ],\n \"userProperties\": [],\n \"typeProperties\": {\n \"variableName\": \"WorkerCoreDetails\",\n \"value\": {\n \"value\": \"@array(activity('Get Worker Core Details').output.firstRow)\",\n \"type\": \"Expression\"\n }\n }\n }\n ],\n \"parameters\": {\n \"executionId\": {\n \"type\": \"string\"\n },\n \"stageId\": {\n \"type\": \"int\"\n },\n \"pipelineId\": {\n \"type\": \"int\"\n }\n },\n \"variables\": {\n \"WorkerPipelineState\": {\n \"type\": \"Boolean\"\n },\n \"WorkerRunId\": {\n \"type\": \"String\"\n },\n \"WorkerCoreDetails\": {\n \"type\": \"Array\"\n }\n },\n \"folder\": {\n \"name\": \"_ProcFwk\"\n },\n \"annotations\": [\n \"procfwk\",\n \"Infant\"\n ]\n },\n \"dependsOn\": [\n \"[concat(variables('factoryId'), '/linkedServices/FrameworkFunctions')]\",\n \"[concat(variables('factoryId'), '/datasets/GetSetMetadata')]\",\n \"[concat(variables('factoryId'), '/linkedServices/SupportDatabase')]\",\n \"[concat(variables('factoryId'), '/pipelines/Email Sender')]\",\n \"[concat(variables('factoryId'), '/pipelines/Throw Exception')]\"\n ]\n },\n {\n \"name\": \"[concat(parameters('factoryName'), '/Check For Running Pipeline')]\",\n \"type\": \"Microsoft.DataFactory/factories/pipelines\",\n \"apiVersion\": \"2018-06-01\",\n \"properties\": {\n \"description\": \"For a given pipeline and optional batch name establish if a pipeline run is already in progress. Throw an exception if it it.\",\n \"activities\": [\n {\n \"name\": \"Filter Running Pipelines\",\n \"description\": \"Filter the pipeline runs results for pipelines that exclude the current triggered run and that are currently running (in progress or queued).\",\n \"type\": \"Filter\",\n \"dependsOn\": [\n {\n \"activity\": \"Switch For Orchestrator Type\",\n \"dependencyConditions\": [\n \"Succeeded\"\n ]\n }\n ],\n \"userProperties\": [],\n \"typeProperties\": {\n \"items\": {\n \"value\": \"@variables('PipelineRuns')\",\n \"type\": \"Expression\"\n },\n \"condition\": {\n \"value\": \"@and(not(equals(item().runId,pipeline().parameters.ThisRunId)),or(equals(item().status,'InProgress'),equals(item().status,'Queued')))\",\n \"type\": \"Expression\"\n }\n }\n },\n {\n \"name\": \"Get Framework Orchestrator Details\",\n \"description\": \"Using the metadata orchestrators return details about the resource running the framework pipelines.\",\n \"type\": \"Lookup\",\n \"dependsOn\": [],\n \"policy\": {\n \"timeout\": \"0.00:10:00\",\n \"retry\": 0,\n \"retryIntervalInSeconds\": 30,\n \"secureOutput\": false,\n \"secureInput\": false\n },\n \"userProperties\": [],\n \"typeProperties\": {\n \"source\": {\n \"type\": \"AzureSqlSource\",\n \"sqlReaderStoredProcedureName\": \"[[procfwk].[GetFrameworkOrchestratorDetails]\",\n \"storedProcedureParameters\": {\n \"CallingOrchestratorName\": {\n \"type\": \"String\",\n \"value\": {\n \"value\": \"@pipeline().DataFactory\",\n \"type\": \"Expression\"\n }\n }\n },\n \"queryTimeout\": \"02:00:00\",\n \"partitionOption\": \"None\"\n },\n \"dataset\": {\n \"referenceName\": \"GetSetMetadata\",\n \"type\": \"DatasetReference\",\n \"parameters\": {}\n }\n }\n },\n {\n \"name\": \"Get Query Run Days Value\",\n \"description\": \"Using the metadata properties table return the run days value to provide the API request with a date range for pipeline executions.\",\n \"type\": \"Lookup\",\n \"dependsOn\": [],\n \"policy\": {\n \"timeout\": \"0.00:10:00\",\n \"retry\": 0,\n \"retryIntervalInSeconds\": 30,\n \"secureOutput\": false,\n \"secureInput\": false\n },\n \"userProperties\": [],\n \"typeProperties\": {\n \"source\": {\n \"type\": \"AzureSqlSource\",\n \"sqlReaderStoredProcedureName\": \"[[procfwk].[GetPropertyValue]\",\n \"storedProcedureParameters\": {\n \"PropertyName\": {\n \"type\": \"String\",\n \"value\": \"PreviousPipelineRunsQueryRange\"\n }\n },\n \"queryTimeout\": \"02:00:00\",\n \"partitionOption\": \"None\"\n },\n \"dataset\": {\n \"referenceName\": \"GetSetMetadata\",\n \"type\": \"DatasetReference\",\n \"parameters\": {}\n }\n }\n },\n {\n \"name\": \"If Pipeline Is Running\",\n \"description\": \"If the running pipeline count is greater than or equal to one.\\nTrue = raise an exception.\",\n \"type\": \"IfCondition\",\n \"dependsOn\": [\n {\n \"activity\": \"If Using Batch Executions\",\n \"dependencyConditions\": [\n \"Succeeded\"\n ]\n }\n ],\n \"userProperties\": [],\n \"typeProperties\": {\n \"expression\": {\n \"value\": \"@greaterOrEquals(int(variables('RunCount')),1)\",\n \"type\": \"Expression\"\n },\n \"ifTrueActivities\": [\n {\n \"name\": \"Throw Exception - Pipeline Running\",\n \"description\": \"Using the utils pipeline raise an exception to stop the new trigger while a run is already in progress.\",\n \"type\": \"ExecutePipeline\",\n \"dependsOn\": [],\n \"userProperties\": [],\n \"typeProperties\": {\n \"pipeline\": {\n \"referenceName\": \"Throw Exception\",\n \"type\": \"PipelineReference\"\n },\n \"waitOnCompletion\": true,\n \"parameters\": {\n \"Message\": {\n \"value\": \"@concat('Provided pipeline name (',pipeline().parameters.PipelineName,') still has a run in progress or queued given the query range parameters set in the properties table.')\",\n \"type\": \"Expression\"\n }\n }\n }\n }\n ]\n }\n },\n {\n \"name\": \"Get Execution Batch Status\",\n \"description\": \"Using the metadata properties table return the flag to indicate if batch execution setting are enabled or disabled.\",\n \"type\": \"Lookup\",\n \"dependsOn\": [],\n \"policy\": {\n \"timeout\": \"0.00:10:00\",\n \"retry\": 0,\n \"retryIntervalInSeconds\": 30,\n \"secureOutput\": false,\n \"secureInput\": false\n },\n \"userProperties\": [],\n \"typeProperties\": {\n \"source\": {\n \"type\": \"AzureSqlSource\",\n \"sqlReaderStoredProcedureName\": \"[[procfwk].[GetPropertyValue]\",\n \"storedProcedureParameters\": {\n \"PropertyName\": {\n \"type\": \"String\",\n \"value\": \"UseExecutionBatches\"\n }\n },\n \"queryTimeout\": \"02:00:00\",\n \"partitionOption\": \"None\"\n },\n \"dataset\": {\n \"referenceName\": \"GetSetMetadata\",\n \"type\": \"DatasetReference\",\n \"parameters\": {}\n }\n }\n },\n {\n \"name\": \"If Using Batch Executions\",\n \"description\": \"True = batch executions are enabled.\\nFalse = batch execution are disabled.\",\n \"type\": \"IfCondition\",\n \"dependsOn\": [\n {\n \"activity\": \"Get Execution Batch Status\",\n \"dependencyConditions\": [\n \"Succeeded\"\n ]\n },\n {\n \"activity\": \"Filter Running Pipelines\",\n \"dependencyConditions\": [\n \"Succeeded\"\n ]\n }\n ],\n \"userProperties\": [],\n \"typeProperties\": {\n \"expression\": {\n \"value\": \"@equals(activity('Get Execution Batch Status').output.firstRow.PropertyValue,string(1))\",\n \"type\": \"Expression\"\n },\n \"ifFalseActivities\": [\n {\n \"name\": \"Set Run Count Without Batch\",\n \"description\": \"Set the pipelines running count variable to be tested later.\",\n \"type\": \"SetVariable\",\n \"dependsOn\": [],\n \"userProperties\": [],\n \"typeProperties\": {\n \"variableName\": \"RunCount\",\n \"value\": {\n \"value\": \"@string(activity('Filter Running Pipelines').output.FilteredItemsCount)\",\n \"type\": \"Expression\"\n }\n }\n }\n ],\n \"ifTrueActivities\": [\n {\n \"name\": \"Filter for Batch Name\",\n \"description\": \"Further filter the return pipeline runs for any running pipelines with the same batch name value.\",\n \"type\": \"Filter\",\n \"dependsOn\": [],\n \"userProperties\": [],\n \"typeProperties\": {\n \"items\": {\n \"value\": \"@activity('Filter Running Pipelines').output.value\",\n \"type\": \"Expression\"\n },\n \"condition\": {\n \"value\": \"@equals(item().parameters.BatchName,pipeline().parameters.BatchName)\",\n \"type\": \"Expression\"\n }\n }\n },\n {\n \"name\": \"Set Run Count for Batch\",\n \"description\": \"Set the resulting pipeline running count variable to be tested later.\",\n \"type\": \"SetVariable\",\n \"dependsOn\": [\n {\n \"activity\": \"Filter for Batch Name\",\n \"dependencyConditions\": [\n \"Succeeded\"\n ]\n }\n ],\n \"userProperties\": [],\n \"typeProperties\": {\n \"variableName\": \"RunCount\",\n \"value\": {\n \"value\": \"@string(activity('Filter for Batch Name').output.FilteredItemsCount)\",\n \"type\": \"Expression\"\n }\n }\n }\n ]\n }\n },\n {\n \"name\": \"Set Subscription Id\",\n \"description\": \"Set the subscription Id value to a local variable for use in various downstream activities.\",\n \"type\": \"SetVariable\",\n \"dependsOn\": [\n {\n \"activity\": \"Get Framework Orchestrator Details\",\n \"dependencyConditions\": [\n \"Succeeded\"\n ]\n }\n ],\n \"userProperties\": [],\n \"typeProperties\": {\n \"variableName\": \"SubscriptionId\",\n \"value\": {\n \"value\": \"@activity('Get Framework Orchestrator Details').output.firstRow.SubscriptionId\",\n \"type\": \"Expression\"\n }\n }\n },\n {\n \"name\": \"Set Resource Group Name\",\n \"description\": \"Set the resource group name value to a local variable for use in various downstream activities.\",\n \"type\": \"SetVariable\",\n \"dependsOn\": [\n {\n \"activity\": \"Get Framework Orchestrator Details\",\n \"dependencyConditions\": [\n \"Succeeded\"\n ]\n }\n ],\n \"userProperties\": [],\n \"typeProperties\": {\n \"variableName\": \"ResourceGroupName\",\n \"value\": {\n \"value\": \"@activity('Get Framework Orchestrator Details').output.firstRow.ResourceGroupName\",\n \"type\": \"Expression\"\n }\n }\n },\n {\n \"name\": \"Set Orchestrator Type\",\n \"description\": \"Set the orchestrator type value to a local variable for use in various downstream activities.\",\n \"type\": \"SetVariable\",\n \"dependsOn\": [\n {\n \"activity\": \"Get Framework Orchestrator Details\",\n \"dependencyConditions\": [\n \"Succeeded\"\n ]\n }\n ],\n \"userProperties\": [],\n \"typeProperties\": {\n \"variableName\": \"OrchestratorType\",\n \"value\": {\n \"value\": \"@toUpper(activity('Get Framework Orchestrator Details').output.firstRow.OrchestratorType)\",\n \"type\": \"Expression\"\n }\n }\n },\n {\n \"name\": \"Switch For Orchestrator Type\",\n \"description\": \"Switch and handle requests for both Azure Data Factory (ADF) and Azure Synapse Analytics (SYN).\",\n \"type\": \"Switch\",\n \"dependsOn\": [\n {\n \"activity\": \"Set Orchestrator Type\",\n \"dependencyConditions\": [\n \"Succeeded\"\n ]\n },\n {\n \"activity\": \"Set Query Run Days\",\n \"dependencyConditions\": [\n \"Succeeded\"\n ]\n },\n {\n \"activity\": \"Set Resource Group Name\",\n \"dependencyConditions\": [\n \"Succeeded\"\n ]\n },\n {\n \"activity\": \"Set Subscription Id\",\n \"dependencyConditions\": [\n \"Succeeded\"\n ]\n }\n ],\n \"userProperties\": [],\n \"typeProperties\": {\n \"on\": {\n \"value\": \"@variables('OrchestratorType')\",\n \"type\": \"Expression\"\n },\n \"cases\": [\n {\n \"value\": \"ADF\",\n \"activities\": [\n {\n \"name\": \"Check for Valid ADF Pipeline Name\",\n \"description\": \"Use the Azure Management API to return and establish if the framework pipeline exists in the target Data Factory instance, including being deployed.\",\n \"type\": \"WebActivity\",\n \"dependsOn\": [],\n \"policy\": {\n \"timeout\": \"7.00:00:00\",\n \"retry\": 0,\n \"retryIntervalInSeconds\": 30,\n \"secureOutput\": false,\n \"secureInput\": false\n },\n \"userProperties\": [],\n \"typeProperties\": {\n \"url\": {\n \"value\": \"https://management.azure.com/subscriptions/@{variables('SubscriptionId')}/resourceGroups/@{variables('ResourceGroupName')}/providers/Microsoft.DataFactory/factories/@{pipeline().DataFactory}/pipelines/@{pipeline().parameters.PipelineName}?api-version=2018-06-01\",\n \"type\": \"Expression\"\n },\n \"method\": \"GET\",\n \"headers\": {},\n \"authentication\": {\n \"type\": \"MSI\",\n \"resource\": \"https://management.core.windows.net/\"\n }\n }\n },\n {\n \"name\": \"Get ADF Pipeline Runs\",\n \"description\": \"Use the Azure Management API to return a list of data factory pipeline runs within the given time window.\",\n \"type\": \"WebActivity\",\n \"dependsOn\": [\n {\n \"activity\": \"Check for Valid ADF Pipeline Name\",\n \"dependencyConditions\": [\n \"Succeeded\"\n ]\n }\n ],\n \"policy\": {\n \"timeout\": \"7.00:00:00\",\n \"retry\": 0,\n \"retryIntervalInSeconds\": 30,\n \"secureOutput\": false,\n \"secureInput\": false\n },\n \"userProperties\": [],\n \"typeProperties\": {\n \"url\": {\n \"value\": \"https://management.azure.com/subscriptions/@{variables('SubscriptionId')}/resourceGroups/@{variables('ResourceGroupName')}/providers/Microsoft.DataFactory/factories/@{pipeline().DataFactory}/queryPipelineRuns?api-version=2018-06-01\",\n \"type\": \"Expression\"\n },\n \"method\": \"POST\",\n \"headers\": {},\n \"body\": {\n \"value\": \"{\\n \\\"lastUpdatedAfter\\\": \\\"@{adddays(utcnow(),int(variables('QueryRunDays')))}\\\",\\n \\\"lastUpdatedBefore\\\": \\\"@{utcnow()}\\\",\\n \\\"filters\\\": [\\n {\\n \\\"operand\\\": \\\"PipelineName\\\",\\n \\\"operator\\\": \\\"Equals\\\",\\n \\\"values\\\": [\\n \\\"@{pipeline().parameters.PipelineName}\\\"\\n ]\\n }\\n ]\\n}\",\n \"type\": \"Expression\"\n },\n \"authentication\": {\n \"type\": \"MSI\",\n \"resource\": \"https://management.core.windows.net/\"\n }\n }\n },\n {\n \"name\": \"Set ADF Runs Output\",\n \"description\": \"Set output to local array for use in downstream filtering and pipeline checks. Use the same array output for both switch cases.\",\n \"type\": \"SetVariable\",\n \"dependsOn\": [\n {\n \"activity\": \"Get ADF Pipeline Runs\",\n \"dependencyConditions\": [\n \"Succeeded\"\n ]\n }\n ],\n \"userProperties\": [],\n \"typeProperties\": {\n \"variableName\": \"PipelineRuns\",\n \"value\": {\n \"value\": \"@activity('Get ADF Pipeline Runs').output.value\",\n \"type\": \"Expression\"\n }\n }\n }\n ]\n },\n {\n \"value\": \"SYN\",\n \"activities\": [\n {\n \"name\": \"Check for Valid SYN Pipeline Name\",\n \"description\": \"Use the Azure Management API to return and establish if the framework pipeline exists in the target Synapse instance, including being deployed.\\n\\nSee: https://docs.microsoft.com/en-us/rest/api/synapse/data-plane/pipeline/getpipeline\",\n \"type\": \"WebActivity\",\n \"dependsOn\": [],\n \"policy\": {\n \"timeout\": \"7.00:00:00\",\n \"retry\": 0,\n \"retryIntervalInSeconds\": 30,\n \"secureOutput\": false,\n \"secureInput\": false\n },\n \"userProperties\": [],\n \"typeProperties\": {\n \"url\": {\n \"value\": \"https://@{pipeline().DataFactory}.dev.azuresynapse.net/pipelines/@{pipeline().parameters.PipelineName}?api-version=2019-06-01-preview\",\n \"type\": \"Expression\"\n },\n \"method\": \"GET\",\n \"headers\": {},\n \"authentication\": {\n \"type\": \"MSI\",\n \"resource\": \"https://management.core.windows.net/\"\n }\n }\n },\n {\n \"name\": \"Get SYN Pipeline Runs\",\n \"description\": \"Use the Azure Management API to return a list of synapse pipeline runs within the given time window.\\n\\nSee: https://docs.microsoft.com/en-us/rest/api/synapse/data-plane/pipelinerun/querypipelinerunsbyworkspace\",\n \"type\": \"WebActivity\",\n \"dependsOn\": [\n {\n \"activity\": \"Check for Valid SYN Pipeline Name\",\n \"dependencyConditions\": [\n \"Succeeded\"\n ]\n }\n ],\n \"policy\": {\n \"timeout\": \"7.00:00:00\",\n \"retry\": 0,\n \"retryIntervalInSeconds\": 30,\n \"secureOutput\": false,\n \"secureInput\": false\n },\n \"userProperties\": [],\n \"typeProperties\": {\n \"url\": {\n \"value\": \"https://@{pipeline().DataFactory}.dev.azuresynapse.net/queryPipelineRuns?api-version=2019-06-01-preview\",\n \"type\": \"Expression\"\n },\n \"method\": \"POST\",\n \"headers\": {},\n \"body\": {\n \"value\": \"{\\n \\\"lastUpdatedAfter\\\": \\\"@{adddays(utcnow(),int(variables('QueryRunDays')))}\\\",\\n \\\"lastUpdatedBefore\\\": \\\"@{utcnow()}\\\",\\n \\\"filters\\\": [\\n {\\n \\\"operand\\\": \\\"PipelineName\\\",\\n \\\"operator\\\": \\\"Equals\\\",\\n \\\"values\\\": [\\n \\\"@{pipeline().parameters.PipelineName}\\\"\\n ]\\n }\\n ]\\n}\",\n \"type\": \"Expression\"\n },\n \"authentication\": {\n \"type\": \"MSI\",\n \"resource\": \"https://management.core.windows.net/\"\n }\n }\n },\n {\n \"name\": \"Set SYN Runs Output\",\n \"description\": \"Set output to local array for use in downstream filtering and pipeline checks. Use the same array output for both switch cases.\",\n \"type\": \"SetVariable\",\n \"dependsOn\": [\n {\n \"activity\": \"Get SYN Pipeline Runs\",\n \"dependencyConditions\": [\n \"Succeeded\"\n ]\n }\n ],\n \"userProperties\": [],\n \"typeProperties\": {\n \"variableName\": \"PipelineRuns\",\n \"value\": {\n \"value\": \"@activity('Get SYN Pipeline Runs').output.value\",\n \"type\": \"Expression\"\n }\n }\n }\n ]\n }\n ],\n \"defaultActivities\": [\n {\n \"name\": \"Throw Exception Invalid Orchestrator Type\",\n \"description\": \"Throw exception if switch cases are not met.\",\n \"type\": \"ExecutePipeline\",\n \"dependsOn\": [],\n \"userProperties\": [],\n \"typeProperties\": {\n \"pipeline\": {\n \"referenceName\": \"Throw Exception\",\n \"type\": \"PipelineReference\"\n },\n \"waitOnCompletion\": true,\n \"parameters\": {\n \"Message\": \"Invalid orchestrator type provided. Unable to check pipeline running state.\"\n }\n }\n }\n ]\n }\n },\n {\n \"name\": \"Set Query Run Days\",\n \"description\": \"Set the query run days value to a local variable for use in various downstream activities.\",\n \"type\": \"SetVariable\",\n \"dependsOn\": [\n {\n \"activity\": \"Get Query Run Days Value\",\n \"dependencyConditions\": [\n \"Succeeded\"\n ]\n }\n ],\n \"userProperties\": [],\n \"typeProperties\": {\n \"variableName\": \"QueryRunDays\",\n \"value\": {\n \"value\": \"@activity('Get Query Run Days Value').output.firstRow.PropertyValue\",\n \"type\": \"Expression\"\n }\n }\n }\n ],\n \"parameters\": {\n \"BatchName\": {\n \"type\": \"string\",\n \"defaultValue\": \"NotUsed\"\n },\n \"PipelineName\": {\n \"type\": \"string\"\n },\n \"ThisRunId\": {\n \"type\": \"string\"\n }\n },\n \"variables\": {\n \"SubscriptionId\": {\n \"type\": \"String\"\n },\n \"RunCount\": {\n \"type\": \"String\"\n },\n \"ResourceGroupName\": {\n \"type\": \"String\"\n },\n \"OrchestratorType\": {\n \"type\": \"String\"\n },\n \"QueryRunDays\": {\n \"type\": \"String\"\n },\n \"PipelineRuns\": {\n \"type\": \"Array\"\n }\n },\n \"folder\": {\n \"name\": \"_ProcFwk/_ProcFwkUtils\"\n },\n \"annotations\": [\n \"procfwk\",\n \"Utils\"\n ]\n },\n \"dependsOn\": [\n \"[concat(variables('factoryId'), '/datasets/GetSetMetadata')]\",\n \"[concat(variables('factoryId'), '/pipelines/Throw Exception')]\"\n ]\n },\n {\n \"name\": \"[concat(parameters('factoryName'), '/Email Sender')]\",\n \"type\": \"Microsoft.DataFactory/factories/pipelines\",\n \"apiVersion\": \"2018-06-01\",\n \"properties\": {\n \"description\": \"Provide a simple abstract over the send email function with request body item exposed as pipeline parameters.\",\n \"activities\": [\n {\n \"name\": \"Send Email\",\n \"description\": \"Use an Azure Function to perform an SMTP client email send operation.\",\n \"type\": \"AzureFunctionActivity\",\n \"dependsOn\": [],\n \"policy\": {\n \"timeout\": \"0.00:10:00\",\n \"retry\": 0,\n \"retryIntervalInSeconds\": 30,\n \"secureOutput\": false,\n \"secureInput\": false\n },\n \"userProperties\": [],\n \"typeProperties\": {\n \"functionName\": \"SendEmail\",\n \"method\": \"POST\",\n \"headers\": {},\n \"body\": {\n \"value\": \"{\\n\\\"emailRecipients\\\": \\\"@{pipeline().parameters.Recipients}\\\",\\n\\\"emailCcRecipients\\\": \\\"@{pipeline().parameters.CcRecipients}\\\",\\n\\\"emailBccRecipients\\\": \\\"@{pipeline().parameters.BccRecipients}\\\",\\n\\\"emailSubject\\\": \\\"@{pipeline().parameters.Subject}\\\",\\n\\\"emailBody\\\": \\\"@{pipeline().parameters.Body}\\\",\\n\\\"emailImportance\\\": \\\"@{pipeline().parameters.Importance}\\\"\\n}\",\n \"type\": \"Expression\"\n }\n },\n \"linkedServiceName\": {\n \"referenceName\": \"FrameworkFunctions\",\n \"type\": \"LinkedServiceReference\"\n }\n }\n ],\n \"parameters\": {\n \"Recipients\": {\n \"type\": \"string\"\n },\n \"CcRecipients\": {\n \"type\": \"string\"\n },\n \"BccRecipients\": {\n \"type\": \"string\"\n },\n \"Subject\": {\n \"type\": \"string\"\n },\n \"Body\": {\n \"type\": \"string\"\n },\n \"Importance\": {\n \"type\": \"string\"\n }\n },\n \"folder\": {\n \"name\": \"_ProcFwk/_ProcFwkUtils\"\n },\n \"annotations\": [\n \"procfwk\",\n \"Utils\"\n ]\n },\n \"dependsOn\": [\n \"[concat(variables('factoryId'), '/linkedServices/FrameworkFunctions')]\"\n ]\n },\n {\n \"name\": \"[concat(parameters('factoryName'), '/Intentional Error')]\",\n \"type\": \"Microsoft.DataFactory/factories/pipelines\",\n \"apiVersion\": \"2018-06-01\",\n \"properties\": {\n \"description\": \"Used just so the procfwk has something to call during development.\",\n \"activities\": [\n {\n \"name\": \"Wait1\",\n \"description\": \"Framework development worker simulator.\",\n \"type\": \"Wait\",\n \"dependsOn\": [],\n \"userProperties\": [],\n \"typeProperties\": {\n \"waitTimeInSeconds\": {\n \"value\": \"@pipeline().parameters.WaitTime\",\n \"type\": \"Expression\"\n }\n }\n },\n {\n \"name\": \"Raise Errors or Not\",\n \"description\": \"Framework development worker simulator.\",\n \"type\": \"IfCondition\",\n \"dependsOn\": [\n {\n \"activity\": \"Wait1\",\n \"dependencyConditions\": [\n \"Succeeded\"\n ]\n }\n ],\n \"userProperties\": [],\n \"typeProperties\": {\n \"expression\": {\n \"value\": \"@equals(pipeline().parameters.RaiseErrors,'true')\",\n \"type\": \"Expression\"\n },\n \"ifTrueActivities\": [\n {\n \"name\": \"Call Fail Procedure\",\n \"type\": \"SqlServerStoredProcedure\",\n \"dependsOn\": [],\n \"policy\": {\n \"timeout\": \"0.00:10:00\",\n \"retry\": 0,\n \"retryIntervalInSeconds\": 30,\n \"secureOutput\": false,\n \"secureInput\": false\n },\n \"userProperties\": [],\n \"typeProperties\": {\n \"storedProcedureName\": \"[[dbo].[FailProcedure]\",\n \"storedProcedureParameters\": {\n \"RaiseError\": {\n \"value\": {\n \"value\": \"@pipeline().parameters.RaiseErrors\",\n \"type\": \"Expression\"\n },\n \"type\": \"String\"\n }\n }\n },\n \"linkedServiceName\": {\n \"referenceName\": \"SupportDatabase\",\n \"type\": \"LinkedServiceReference\"\n }\n }\n ]\n }\n }\n ],\n \"parameters\": {\n \"RaiseErrors\": {\n \"type\": \"string\",\n \"defaultValue\": \"false\"\n },\n \"WaitTime\": {\n \"type\": \"int\",\n \"defaultValue\": 5\n }\n },\n \"folder\": {\n \"name\": \"_Workers\"\n },\n \"annotations\": [\n \"_ProcFwkWorker\"\n ]\n },\n \"dependsOn\": [\n \"[concat(variables('factoryId'), '/linkedServices/SupportDatabase')]\"\n ]\n },\n {\n \"name\": \"[concat(parameters('factoryName'), '/Throw Exception')]\",\n \"type\": \"Microsoft.DataFactory/factories/pipelines\",\n \"apiVersion\": \"2018-06-01\",\n \"properties\": {\n \"description\": \"Provide a simple way of throwing an exception within Data Factory using TSQL error handling.\",\n \"activities\": [\n {\n \"name\": \"Raise Error\",\n \"description\": \"Using a SQL database to raise an error/exception but wrapped up as a data factory pipeline. Error message information exposed as a pipeline parameter.\",\n \"type\": \"Lookup\",\n \"dependsOn\": [],\n \"policy\": {\n \"timeout\": \"0.00:10:00\",\n \"retry\": 0,\n \"retryIntervalInSeconds\": 30,\n \"secureOutput\": false,\n \"secureInput\": false\n },\n \"userProperties\": [],\n \"typeProperties\": {\n \"source\": {\n \"type\": \"AzureSqlSource\",\n \"sqlReaderQuery\": {\n \"value\": \"RAISERROR('@{pipeline().parameters.Message}',16,1);\",\n \"type\": \"Expression\"\n },\n \"queryTimeout\": \"02:00:00\",\n \"partitionOption\": \"None\"\n },\n \"dataset\": {\n \"referenceName\": \"GetSetMetadata\",\n \"type\": \"DatasetReference\",\n \"parameters\": {}\n },\n \"firstRowOnly\": false\n }\n }\n ],\n \"parameters\": {\n \"Message\": {\n \"type\": \"string\"\n }\n },\n \"folder\": {\n \"name\": \"_ProcFwk/_ProcFwkUtils\"\n },\n \"annotations\": [\n \"procfwk\",\n \"Utils\"\n ]\n },\n \"dependsOn\": [\n \"[concat(variables('factoryId'), '/datasets/GetSetMetadata')]\"\n ]\n },\n {\n \"name\": \"[concat(parameters('factoryName'), '/Wait 1')]\",\n \"type\": \"Microsoft.DataFactory/factories/pipelines\",\n \"apiVersion\": \"2018-06-01\",\n \"properties\": {\n \"description\": \"Used just so the procfwk has something to call during development.\",\n \"activities\": [\n {\n \"name\": \"Wait1\",\n \"description\": \"Framework development worker simulator.\",\n \"type\": \"Wait\",\n \"dependsOn\": [],\n \"userProperties\": [],\n \"typeProperties\": {\n \"waitTimeInSeconds\": {\n \"value\": \"@pipeline().parameters.WaitTime\",\n \"type\": \"Expression\"\n }\n }\n }\n ],\n \"parameters\": {\n \"WaitTime\": {\n \"type\": \"int\",\n \"defaultValue\": 5\n }\n },\n \"folder\": {\n \"name\": \"_Workers\"\n },\n \"annotations\": [\n \"_ProcFwkWorker\"\n ]\n },\n \"dependsOn\": []\n },\n {\n \"name\": \"[concat(parameters('factoryName'), '/Wait 10')]\",\n \"type\": \"Microsoft.DataFactory/factories/pipelines\",\n \"apiVersion\": \"2018-06-01\",\n \"properties\": {\n \"description\": \"Used just so the procfwk has something to call during development.\",\n \"activities\": [\n {\n \"name\": \"Wait10\",\n \"description\": \"Framework development worker simulator.\",\n \"type\": \"Wait\",\n \"dependsOn\": [],\n \"userProperties\": [],\n \"typeProperties\": {\n \"waitTimeInSeconds\": {\n \"value\": \"@pipeline().parameters.WaitTime\",\n \"type\": \"Expression\"\n }\n }\n }\n ],\n \"parameters\": {\n \"WaitTime\": {\n \"type\": \"int\",\n \"defaultValue\": 5\n }\n },\n \"folder\": {\n \"name\": \"_Workers\"\n },\n \"annotations\": [\n \"_ProcFwkWorker\"\n ]\n },\n \"dependsOn\": []\n },\n {\n \"name\": \"[concat(parameters('factoryName'), '/Wait 2')]\",\n \"type\": \"Microsoft.DataFactory/factories/pipelines\",\n \"apiVersion\": \"2018-06-01\",\n \"properties\": {\n \"description\": \"Used just so the procfwk has something to call during development.\",\n \"activities\": [\n {\n \"name\": \"Wait2\",\n \"description\": \"Framework development worker simulator.\",\n \"type\": \"Wait\",\n \"dependsOn\": [],\n \"userProperties\": [],\n \"typeProperties\": {\n \"waitTimeInSeconds\": {\n \"value\": \"@pipeline().parameters.WaitTime\",\n \"type\": \"Expression\"\n }\n }\n }\n ],\n \"parameters\": {\n \"WaitTime\": {\n \"type\": \"int\",\n \"defaultValue\": 5\n }\n },\n \"folder\": {\n \"name\": \"_Workers\"\n },\n \"annotations\": [\n \"_ProcFwkWorker\"\n ]\n },\n \"dependsOn\": []\n },\n {\n \"name\": \"[concat(parameters('factoryName'), '/Wait 3')]\",\n \"type\": \"Microsoft.DataFactory/factories/pipelines\",\n \"apiVersion\": \"2018-06-01\",\n \"properties\": {\n \"description\": \"Used just so the procfwk has something to call during development.\",\n \"activities\": [\n {\n \"name\": \"Wait3\",\n \"description\": \"Framework development worker simulator.\",\n \"type\": \"Wait\",\n \"dependsOn\": [],\n \"userProperties\": [],\n \"typeProperties\": {\n \"waitTimeInSeconds\": {\n \"value\": \"@pipeline().parameters.WaitTime\",\n \"type\": \"Expression\"\n }\n }\n }\n ],\n \"parameters\": {\n \"WaitTime\": {\n \"type\": \"int\",\n \"defaultValue\": 5\n }\n },\n \"folder\": {\n \"name\": \"_Workers\"\n },\n \"annotations\": [\n \"_ProcFwkWorker\"\n ]\n },\n \"dependsOn\": []\n },\n {\n \"name\": \"[concat(parameters('factoryName'), '/Wait 4')]\",\n \"type\": \"Microsoft.DataFactory/factories/pipelines\",\n \"apiVersion\": \"2018-06-01\",\n \"properties\": {\n \"description\": \"Used just so the procfwk has something to call during development.\",\n \"activities\": [\n {\n \"name\": \"Wait4\",\n \"description\": \"Framework development worker simulator.\",\n \"type\": \"Wait\",\n \"dependsOn\": [],\n \"userProperties\": [],\n \"typeProperties\": {\n \"waitTimeInSeconds\": {\n \"value\": \"@pipeline().parameters.WaitTime\",\n \"type\": \"Expression\"\n }\n }\n }\n ],\n \"parameters\": {\n \"WaitTime\": {\n \"type\": \"int\",\n \"defaultValue\": 5\n }\n },\n \"folder\": {\n \"name\": \"_Workers\"\n },\n \"annotations\": [\n \"_ProcFwkWorker\"\n ]\n },\n \"dependsOn\": []\n },\n {\n \"name\": \"[concat(parameters('factoryName'), '/Wait 5')]\",\n \"type\": \"Microsoft.DataFactory/factories/pipelines\",\n \"apiVersion\": \"2018-06-01\",\n \"properties\": {\n \"description\": \"Used just so the procfwk has something to call during development.\",\n \"activities\": [\n {\n \"name\": \"Wait5\",\n \"description\": \"Framework development worker simulator.\",\n \"type\": \"Wait\",\n \"dependsOn\": [],\n \"userProperties\": [],\n \"typeProperties\": {\n \"waitTimeInSeconds\": {\n \"value\": \"@pipeline().parameters.WaitTime\",\n \"type\": \"Expression\"\n }\n }\n }\n ],\n \"parameters\": {\n \"WaitTime\": {\n \"type\": \"int\",\n \"defaultValue\": 5\n }\n },\n \"folder\": {\n \"name\": \"_Workers\"\n },\n \"annotations\": [\n \"_ProcFwkWorker\"\n ]\n },\n \"dependsOn\": []\n },\n {\n \"name\": \"[concat(parameters('factoryName'), '/Wait 6')]\",\n \"type\": \"Microsoft.DataFactory/factories/pipelines\",\n \"apiVersion\": \"2018-06-01\",\n \"properties\": {\n \"description\": \"Used just so the procfwk has something to call during development.\",\n \"activities\": [\n {\n \"name\": \"Wait6\",\n \"description\": \"Framework development worker simulator.\",\n \"type\": \"Wait\",\n \"dependsOn\": [],\n \"userProperties\": [],\n \"typeProperties\": {\n \"waitTimeInSeconds\": {\n \"value\": \"@pipeline().parameters.WaitTime\",\n \"type\": \"Expression\"\n }\n }\n }\n ],\n \"parameters\": {\n \"WaitTime\": {\n \"type\": \"int\",\n \"defaultValue\": 5\n }\n },\n \"folder\": {\n \"name\": \"_Workers\"\n },\n \"annotations\": [\n \"_ProcFwkWorker\"\n ]\n },\n \"dependsOn\": []\n },\n {\n \"name\": \"[concat(parameters('factoryName'), '/Wait 7')]\",\n \"type\": \"Microsoft.DataFactory/factories/pipelines\",\n \"apiVersion\": \"2018-06-01\",\n \"properties\": {\n \"description\": \"Used just so the procfwk has something to call during development.\",\n \"activities\": [\n {\n \"name\": \"Wait7\",\n \"description\": \"Framework development worker simulator.\",\n \"type\": \"Wait\",\n \"dependsOn\": [],\n \"userProperties\": [],\n \"typeProperties\": {\n \"waitTimeInSeconds\": {\n \"value\": \"@pipeline().parameters.WaitTime\",\n \"type\": \"Expression\"\n }\n }\n }\n ],\n \"parameters\": {\n \"WaitTime\": {\n \"type\": \"int\",\n \"defaultValue\": 5\n }\n },\n \"folder\": {\n \"name\": \"_Workers\"\n },\n \"annotations\": [\n \"_ProcFwkWorker\"\n ]\n },\n \"dependsOn\": []\n },\n {\n \"name\": \"[concat(parameters('factoryName'), '/Wait 8')]\",\n \"type\": \"Microsoft.DataFactory/factories/pipelines\",\n \"apiVersion\": \"2018-06-01\",\n \"properties\": {\n \"description\": \"Used just so the procfwk has something to call during development.\",\n \"activities\": [\n {\n \"name\": \"Wait8\",\n \"description\": \"Framework development worker simulator.\",\n \"type\": \"Wait\",\n \"dependsOn\": [],\n \"userProperties\": [],\n \"typeProperties\": {\n \"waitTimeInSeconds\": {\n \"value\": \"@pipeline().parameters.WaitTime\",\n \"type\": \"Expression\"\n }\n }\n }\n ],\n \"parameters\": {\n \"WaitTime\": {\n \"type\": \"int\",\n \"defaultValue\": 5\n }\n },\n \"folder\": {\n \"name\": \"_Workers\"\n },\n \"annotations\": [\n \"_ProcFwkWorker\"\n ]\n },\n \"dependsOn\": []\n },\n {\n \"name\": \"[concat(parameters('factoryName'), '/Wait 9')]\",\n \"type\": \"Microsoft.DataFactory/factories/pipelines\",\n \"apiVersion\": \"2018-06-01\",\n \"properties\": {\n \"description\": \"Used just so the procfwk has something to call during development.\",\n \"activities\": [\n {\n \"name\": \"Wait9\",\n \"description\": \"Framework development worker simulator.\",\n \"type\": \"Wait\",\n \"dependsOn\": [],\n \"userProperties\": [],\n \"typeProperties\": {\n \"waitTimeInSeconds\": {\n \"value\": \"@pipeline().parameters.WaitTime\",\n \"type\": \"Expression\"\n }\n }\n }\n ],\n \"parameters\": {\n \"WaitTime\": {\n \"type\": \"int\",\n \"defaultValue\": 15\n }\n },\n \"folder\": {\n \"name\": \"_Workers\"\n },\n \"annotations\": [\n \"_ProcFwkWorker\"\n ]\n },\n \"dependsOn\": []\n },\n {\n \"name\": \"[concat(parameters('factoryName'), '/GetSetMetadata')]\",\n \"type\": \"Microsoft.DataFactory/factories/datasets\",\n \"apiVersion\": \"2018-06-01\",\n \"properties\": {\n \"description\": \"Single generic dataset used to get and set all database metadata.\",\n \"linkedServiceName\": {\n \"referenceName\": \"SupportDatabase\",\n \"type\": \"LinkedServiceReference\"\n },\n \"folder\": {\n \"name\": \"_ProcFwk\"\n },\n \"annotations\": [\n \"procfwk\"\n ],\n \"type\": \"AzureSqlTable\",\n \"schema\": [],\n \"typeProperties\": {}\n },\n \"dependsOn\": [\n \"[concat(variables('factoryId'), '/linkedServices/SupportDatabase')]\"\n ]\n },\n {\n \"name\": \"[concat(parameters('factoryName'), '/FrameworkFunctions')]\",\n \"type\": \"Microsoft.DataFactory/factories/linkedServices\",\n \"apiVersion\": \"2018-06-01\",\n \"properties\": {\n \"description\": \"Interact with the Azure Functions App used as middle ware when making requests to Worker pipelines. Authentication done at the Function App level.\",\n \"annotations\": [\n \"procfwk\"\n ],\n \"type\": \"AzureFunction\",\n \"typeProperties\": {\n \"functionAppUrl\": \"[parameters('FrameworkFunctions_properties_typeProperties_functionAppUrl')]\",\n \"functionKey\": {\n \"type\": \"AzureKeyVaultSecret\",\n \"store\": {\n \"referenceName\": \"Keys\",\n \"type\": \"LinkedServiceReference\"\n },\n \"secretName\": \"FrameworkFunctionsKey\"\n }\n }\n },\n \"dependsOn\": [\n \"[concat(variables('factoryId'), '/linkedServices/Keys')]\"\n ]\n },\n {\n \"name\": \"[concat(parameters('factoryName'), '/Keys')]\",\n \"type\": \"Microsoft.DataFactory/factories/linkedServices\",\n \"apiVersion\": \"2018-06-01\",\n \"properties\": {\n \"description\": \"Connection to Key Vault for all other ADF linked service credentials required to run the processing framework.\",\n \"annotations\": [\n \"procfwk\"\n ],\n \"type\": \"AzureKeyVault\",\n \"typeProperties\": {\n \"baseUrl\": \"[parameters('Keys_properties_typeProperties_baseUrl')]\"\n }\n },\n \"dependsOn\": []\n },\n {\n \"name\": \"[concat(parameters('factoryName'), '/SupportDatabase')]\",\n \"type\": \"Microsoft.DataFactory/factories/linkedServices\",\n \"apiVersion\": \"2018-06-01\",\n \"properties\": {\n \"description\": \"Connection between ADF and processing framework metadata SQLDB.\",\n \"annotations\": [\n \"procfwk\"\n ],\n \"type\": \"AzureSqlDatabase\",\n \"typeProperties\": {\n \"connectionString\": {\n \"type\": \"AzureKeyVaultSecret\",\n \"store\": {\n \"referenceName\": \"Keys\",\n \"type\": \"LinkedServiceReference\"\n },\n \"secretName\": \"[parameters('SupportDatabase_properties_typeProperties_connectionString_secretName')]\"\n }\n }\n },\n \"dependsOn\": [\n \"[concat(variables('factoryId'), '/linkedServices/Keys')]\"\n ]\n },\n {\n \"name\": \"[concat(parameters('factoryName'), '/FunctionalTestingTrigger')]\",\n \"type\": \"Microsoft.DataFactory/factories/triggers\",\n \"apiVersion\": \"2018-06-01\",\n \"properties\": {\n \"description\": \"Used for functional testing of the framework in a dedicated environment.\",\n \"annotations\": [\n \"procfwk\"\n ],\n \"runtimeState\": \"Stopped\",\n \"pipelines\": [\n {\n \"pipelineReference\": {\n \"referenceName\": \"01-Grandparent\",\n \"type\": \"PipelineReference\"\n },\n \"parameters\": {}\n }\n ],\n \"type\": \"ScheduleTrigger\",\n \"typeProperties\": {\n \"recurrence\": {\n \"frequency\": \"Hour\",\n \"interval\": 2,\n \"startTime\": \"2020-04-06T15:00:00Z\",\n \"timeZone\": \"UTC\"\n }\n }\n },\n \"dependsOn\": [\n \"[concat(variables('factoryId'), '/pipelines/01-Grandparent')]\"\n ]\n }\n ]\n}"
},
{
"path": "ARM Templates/Functions App/v1.6 Export.json",
"content": "{\n \"$schema\": \"https://schema.management.azure.com/schemas/2015-01-01/deploymentTemplate.json#\",\n \"contentVersion\": \"1.0.0.0\",\n \"parameters\": {\n \"FunctionAppName\": {\n \"defaultValue\": \"\",\n \"type\": \"String\"\n },\n \"SubscriptionAndRegionServerFarmLocation\": {\n \"defaultValue\": \"/subscriptions/{YourSubscriptionId}/resourceGroups/{YourResourceGroupName}/providers/Microsoft.Web/serverfarms/UKSouthPlan\",\n \"type\": \"String\"\n }\n },\n \"variables\": {},\n \"resources\": [\n {\n \"type\": \"Microsoft.Web/sites\",\n \"apiVersion\": \"2018-11-01\",\n \"name\": \"[parameters('FunctionAppName')]\",\n \"location\": \"UK South\",\n \"kind\": \"functionapp\",\n \"properties\": {\n \"enabled\": true,\n \"hostNameSslStates\": [\n {\n \"name\": \"[parameters('FunctionAppName')].azurewebsites.net\",\n \"sslState\": \"Disabled\",\n \"hostType\": \"Standard\"\n },\n {\n \"name\": \"[parameters('FunctionAppName')].scm.azurewebsites.net\",\n \"sslState\": \"Disabled\",\n \"hostType\": \"Repository\"\n }\n ],\n \"serverFarmId\": \"[parameters('SubscriptionAndRegionServerFarmLocation')]\",\n \"reserved\": false,\n \"isXenon\": false,\n \"hyperV\": false,\n \"siteConfig\": {},\n \"scmSiteAlsoStopped\": false,\n \"clientAffinityEnabled\": false,\n \"clientCertEnabled\": false,\n \"hostNamesDisabled\": false,\n \"containerSize\": 1536,\n \"dailyMemoryTimeQuota\": 0,\n \"httpsOnly\": true,\n \"redundancyMode\": \"None\"\n }\n },\n {\n \"type\": \"Microsoft.Web/sites/config\",\n \"apiVersion\": \"2018-11-01\",\n \"name\": \"[concat(parameters('FunctionAppName'), '/web')]\",\n \"location\": \"UK South\",\n \"dependsOn\": [\n \"[resourceId('Microsoft.Web/sites', parameters('FunctionAppName'))]\"\n ],\n \"properties\": {\n \"numberOfWorkers\": -1,\n \"defaultDocuments\": [\n \"Default.htm\",\n \"Default.html\",\n \"Default.asp\",\n \"index.htm\",\n \"index.html\",\n \"iisstart.htm\",\n \"default.aspx\",\n \"index.php\"\n ],\n \"netFrameworkVersion\": \"v4.0\",\n \"phpVersion\": \"5.6\",\n \"requestTracingEnabled\": false,\n \"remoteDebuggingEnabled\": false,\n \"httpLoggingEnabled\": false,\n \"logsDirectorySizeLimit\": 35,\n \"detailedErrorLoggingEnabled\": false,\n \"publishingUsername\": \"$[parameters('FunctionAppName')]\",\n \"scmType\": \"None\",\n \"use32BitWorkerProcess\": true,\n \"webSocketsEnabled\": false,\n \"alwaysOn\": false,\n \"managedPipelineMode\": \"Integrated\",\n \"virtualApplications\": [\n {\n \"virtualPath\": \"/\",\n \"physicalPath\": \"site\\\\wwwroot\",\n \"preloadEnabled\": false\n }\n ],\n \"loadBalancing\": \"LeastRequests\",\n \"experiments\": {\n \"rampUpRules\": []\n },\n \"autoHealEnabled\": false,\n \"cors\": {\n \"allowedOrigins\": [\n \"https://functions.azure.com\",\n \"https://functions-staging.azure.com\",\n \"https://functions-next.azure.com\"\n ],\n \"supportCredentials\": false\n },\n \"localMySqlEnabled\": false,\n \"ipSecurityRestrictions\": [\n {\n \"ipAddress\": \"Any\",\n \"action\": \"Allow\",\n \"priority\": 1,\n \"name\": \"Allow all\",\n \"description\": \"Allow all access\"\n }\n ],\n \"scmIpSecurityRestrictions\": [\n {\n \"ipAddress\": \"Any\",\n \"action\": \"Allow\",\n \"priority\": 1,\n \"name\": \"Allow all\",\n \"description\": \"Allow all access\"\n }\n ],\n \"scmIpSecurityRestrictionsUseMain\": false,\n \"http20Enabled\": false,\n \"minTlsVersion\": \"1.2\",\n \"ftpsState\": \"AllAllowed\",\n \"reservedInstanceCount\": 0\n }\n },\n {\n \"type\": \"Microsoft.Web/sites/deployments\",\n \"apiVersion\": \"2018-11-01\",\n \"name\": \"[concat(parameters('FunctionAppName'), '/5aee31e75b934c5f80fca0b26fe47dbb')]\",\n \"location\": \"UK South\",\n \"dependsOn\": [\n \"[resourceId('Microsoft.Web/sites', parameters('FunctionAppName'))]\"\n ],\n \"properties\": {\n \"status\": 4,\n \"author_email\": \"N/A\",\n \"author\": \"N/A\",\n \"deployer\": \"Push-Deployer\",\n \"message\": \"Created via a push deployment\",\n \"start_time\": \"2020-04-30T13:29:11.8890295Z\",\n \"end_time\": \"2020-04-30T13:29:16.4504076Z\",\n \"active\": false\n }\n },\n {\n \"type\": \"Microsoft.Web/sites/deployments\",\n \"apiVersion\": \"2018-11-01\",\n \"name\": \"[concat(parameters('FunctionAppName'), '/6b0808e64188414e88d75565027efc59')]\",\n \"location\": \"UK South\",\n \"dependsOn\": [\n \"[resourceId('Microsoft.Web/sites', parameters('FunctionAppName'))]\"\n ],\n \"properties\": {\n \"status\": 4,\n \"author_email\": \"N/A\",\n \"author\": \"N/A\",\n \"deployer\": \"Push-Deployer\",\n \"message\": \"Created via a push deployment\",\n \"start_time\": \"2020-04-30T16:17:52.965083Z\",\n \"end_time\": \"2020-04-30T16:17:58.0803657Z\",\n \"active\": false\n }\n },\n {\n \"type\": \"Microsoft.Web/sites/deployments\",\n \"apiVersion\": \"2018-11-01\",\n \"name\": \"[concat(parameters('FunctionAppName'), '/9ca87860d72f47049af7ab9bce50ccc1')]\",\n \"location\": \"UK South\",\n \"dependsOn\": [\n \"[resourceId('Microsoft.Web/sites', parameters('FunctionAppName'))]\"\n ],\n \"properties\": {\n \"status\": 4,\n \"author_email\": \"N/A\",\n \"author\": \"N/A\",\n \"deployer\": \"Push-Deployer\",\n \"message\": \"Created via a push deployment\",\n \"start_time\": \"2020-05-06T13:21:39.9815784Z\",\n \"end_time\": \"2020-05-06T13:21:45.8257955Z\",\n \"active\": true\n }\n },\n {\n \"type\": \"Microsoft.Web/sites/functions\",\n \"apiVersion\": \"2018-11-01\",\n \"name\": \"[concat(parameters('FunctionAppName'), '/CheckPipelineStatus')]\",\n \"location\": \"UK South\",\n \"dependsOn\": [\n \"[resourceId('Microsoft.Web/sites', parameters('FunctionAppName'))]\"\n ],\n \"properties\": {\n \"script_root_path_href\": \"https://[parameters('FunctionAppName')].azurewebsites.net/admin/vfs/site/wwwroot/CheckPipelineStatus/\",\n \"script_href\": \"https://[parameters('FunctionAppName')].azurewebsites.net/admin/vfs/site/wwwroot/bin/ADFprocfwk.dll\",\n \"config_href\": \"https://[parameters('FunctionAppName')].azurewebsites.net/admin/vfs/site/wwwroot/CheckPipelineStatus/function.json\",\n \"href\": \"https://[parameters('FunctionAppName')].azurewebsites.net/admin/functions/CheckPipelineStatus\",\n \"config\": {}\n }\n },\n {\n \"type\": \"Microsoft.Web/sites/functions\",\n \"apiVersion\": \"2018-11-01\",\n \"name\": \"[concat(parameters('FunctionAppName'), '/ExecutePipeline')]\",\n \"location\": \"UK South\",\n \"dependsOn\": [\n \"[resourceId('Microsoft.Web/sites', parameters('FunctionAppName'))]\"\n ],\n \"properties\": {\n \"script_root_path_href\": \"https://[parameters('FunctionAppName')].azurewebsites.net/admin/vfs/site/wwwroot/ExecutePipeline/\",\n \"script_href\": \"https://[parameters('FunctionAppName')].azurewebsites.net/admin/vfs/site/wwwroot/bin/ADFprocfwk.dll\",\n \"config_href\": \"https://[parameters('FunctionAppName')].azurewebsites.net/admin/vfs/site/wwwroot/ExecutePipeline/function.json\",\n \"href\": \"https://[parameters('FunctionAppName')].azurewebsites.net/admin/functions/ExecutePipeline\",\n \"config\": {}\n }\n },\n {\n \"type\": \"Microsoft.Web/sites/functions\",\n \"apiVersion\": \"2018-11-01\",\n \"name\": \"[concat(parameters('FunctionAppName'), '/GetActivityErrors')]\",\n \"location\": \"UK South\",\n \"dependsOn\": [\n \"[resourceId('Microsoft.Web/sites', parameters('FunctionAppName'))]\"\n ],\n \"properties\": {\n \"script_root_path_href\": \"https://[parameters('FunctionAppName')].azurewebsites.net/admin/vfs/site/wwwroot/GetActivityErrors/\",\n \"script_href\": \"https://[parameters('FunctionAppName')].azurewebsites.net/admin/vfs/site/wwwroot/bin/ADFprocfwk.dll\",\n \"config_href\": \"https://[parameters('FunctionAppName')].azurewebsites.net/admin/vfs/site/wwwroot/GetActivityErrors/function.json\",\n \"href\": \"https://[parameters('FunctionAppName')].azurewebsites.net/admin/functions/GetActivityErrors\",\n \"config\": {}\n }\n },\n {\n \"type\": \"Microsoft.Web/sites/functions\",\n \"apiVersion\": \"2018-11-01\",\n \"name\": \"[concat(parameters('FunctionAppName'), '/SendEmail')]\",\n \"location\": \"UK South\",\n \"dependsOn\": [\n \"[resourceId('Microsoft.Web/sites', parameters('FunctionAppName'))]\"\n ],\n \"properties\": {\n \"script_root_path_href\": \"https://[parameters('FunctionAppName')].azurewebsites.net/admin/vfs/site/wwwroot/SendEmail/\",\n \"script_href\": \"https://[parameters('FunctionAppName')].azurewebsites.net/admin/vfs/site/wwwroot/bin/ADFprocfwk.dll\",\n \"config_href\": \"https://[parameters('FunctionAppName')].azurewebsites.net/admin/vfs/site/wwwroot/SendEmail/function.json\",\n \"href\": \"https://[parameters('FunctionAppName')].azurewebsites.net/admin/functions/SendEmail\",\n \"config\": {}\n }\n },\n {\n \"type\": \"Microsoft.Web/sites/hostNameBindings\",\n \"apiVersion\": \"2018-11-01\",\n \"name\": \"[concat(parameters('FunctionAppName'), '/', parameters('FunctionAppName'), '.azurewebsites.net')]\",\n \"location\": \"UK South\",\n \"dependsOn\": [\n \"[resourceId('Microsoft.Web/sites', parameters('FunctionAppName'))]\"\n ],\n \"properties\": {\n \"siteName\": \"[parameters('FunctionAppName')]\",\n \"hostNameType\": \"Verified\"\n }\n }\n ]\n}"
},
{
"path": "ARM Templates/SQL Database/v1.6 Export.json",
"content": "{\n \"$schema\": \"https://schema.management.azure.com/schemas/2015-01-01/deploymentTemplate.json#\",\n \"contentVersion\": \"1.0.0.0\",\n \"parameters\": {\n \"servers_platformsupport01_name\": {\n \"defaultValue\": \"frameworksupport01\",\n \"type\": \"String\"\n }\n },\n \"variables\": {},\n \"resources\": [\n {\n \"type\": \"Microsoft.Sql/servers/databases\",\n \"apiVersion\": \"2019-06-01-preview\",\n \"name\": \"[concat(parameters('servers_platformsupport01_name'), '/OrchestrationSupport')]\",\n \"location\": \"uksouth\",\n \"sku\": {\n \"name\": \"Standard\",\n \"tier\": \"Standard\",\n \"capacity\": 20\n },\n \"kind\": \"v12.0,user\",\n \"properties\": {\n \"collation\": \"SQL_Latin1_General_CP1_CI_AS\",\n \"maxSizeBytes\": 10737418240,\n \"catalogCollation\": \"SQL_Latin1_General_CP1_CI_AS\",\n \"zoneRedundant\": false,\n \"readScale\": \"Disabled\",\n \"readReplicaCount\": 0,\n \"storageAccountType\": \"GRS\"\n }\n },\n {\n \"type\": \"Microsoft.Sql/servers/databases/advisors\",\n \"apiVersion\": \"2014-04-01\",\n \"name\": \"[concat(parameters('servers_platformsupport01_name'), '/OrchestrationSupport/CreateIndex')]\",\n \"dependsOn\": [\n \"[resourceId('Microsoft.Sql/servers/databases', parameters('servers_platformsupport01_name'), 'OrchestrationSupport')]\"\n ],\n \"properties\": {\n \"autoExecuteValue\": \"Disabled\"\n }\n },\n {\n \"type\": \"Microsoft.Sql/servers/databases/advisors\",\n \"apiVersion\": \"2014-04-01\",\n \"name\": \"[concat(parameters('servers_platformsupport01_name'), '/OrchestrationSupport/DbParameterization')]\",\n \"dependsOn\": [\n \"[resourceId('Microsoft.Sql/servers/databases', parameters('servers_platformsupport01_name'), 'OrchestrationSupport')]\"\n ],\n \"properties\": {\n \"autoExecuteValue\": \"Disabled\"\n }\n },\n {\n \"type\": \"Microsoft.Sql/servers/databases/advisors\",\n \"apiVersion\": \"2014-04-01\",\n \"name\": \"[concat(parameters('servers_platformsupport01_name'), '/OrchestrationSupport/DefragmentIndex')]\",\n \"dependsOn\": [\n \"[resourceId('Microsoft.Sql/servers/databases', parameters('servers_platformsupport01_name'), 'OrchestrationSupport')]\"\n ],\n \"properties\": {\n \"autoExecuteValue\": \"Disabled\"\n }\n },\n {\n \"type\": \"Microsoft.Sql/servers/databases/advisors\",\n \"apiVersion\": \"2014-04-01\",\n \"name\": \"[concat(parameters('servers_platformsupport01_name'), '/OrchestrationSupport/DropIndex')]\",\n \"dependsOn\": [\n \"[resourceId('Microsoft.Sql/servers/databases', parameters('servers_platformsupport01_name'), 'OrchestrationSupport')]\"\n ],\n \"properties\": {\n \"autoExecuteValue\": \"Disabled\"\n }\n },\n {\n \"type\": \"Microsoft.Sql/servers/databases/advisors\",\n \"apiVersion\": \"2014-04-01\",\n \"name\": \"[concat(parameters('servers_platformsupport01_name'), '/OrchestrationSupport/ForceLastGoodPlan')]\",\n \"dependsOn\": [\n \"[resourceId('Microsoft.Sql/servers/databases', parameters('servers_platformsupport01_name'), 'OrchestrationSupport')]\"\n ],\n \"properties\": {\n \"autoExecuteValue\": \"Enabled\"\n }\n },\n {\n \"type\": \"Microsoft.Sql/servers/databases/auditingPolicies\",\n \"apiVersion\": \"2014-04-01\",\n \"name\": \"[concat(parameters('servers_platformsupport01_name'), '/OrchestrationSupport/Default')]\",\n \"location\": \"UK South\",\n \"dependsOn\": [\n \"[resourceId('Microsoft.Sql/servers/databases', parameters('servers_platformsupport01_name'), 'OrchestrationSupport')]\"\n ],\n \"properties\": {\n \"auditingState\": \"Disabled\"\n }\n },\n {\n \"type\": \"Microsoft.Sql/servers/databases/auditingSettings\",\n \"apiVersion\": \"2017-03-01-preview\",\n \"name\": \"[concat(parameters('servers_platformsupport01_name'), '/OrchestrationSupport/Default')]\",\n \"dependsOn\": [\n \"[resourceId('Microsoft.Sql/servers/databases', parameters('servers_platformsupport01_name'), 'OrchestrationSupport')]\"\n ],\n \"properties\": {\n \"state\": \"Disabled\",\n \"retentionDays\": 0,\n \"storageAccountSubscriptionId\": \"00000000-0000-0000-0000-000000000000\",\n \"isAzureMonitorTargetEnabled\": false\n }\n },\n {\n \"type\": \"Microsoft.Sql/servers/databases/backupLongTermRetentionPolicies\",\n \"apiVersion\": \"2017-03-01-preview\",\n \"name\": \"[concat(parameters('servers_platformsupport01_name'), '/OrchestrationSupport/default')]\",\n \"dependsOn\": [\n \"[resourceId('Microsoft.Sql/servers/databases', parameters('servers_platformsupport01_name'), 'OrchestrationSupport')]\"\n ],\n \"properties\": {\n \"weeklyRetention\": \"PT0S\",\n \"monthlyRetention\": \"PT0S\",\n \"yearlyRetention\": \"PT0S\",\n \"weekOfYear\": 0\n }\n },\n {\n \"type\": \"Microsoft.Sql/servers/databases/backupShortTermRetentionPolicies\",\n \"apiVersion\": \"2017-10-01-preview\",\n \"name\": \"[concat(parameters('servers_platformsupport01_name'), '/OrchestrationSupport/default')]\",\n \"dependsOn\": [\n \"[resourceId('Microsoft.Sql/servers/databases', parameters('servers_platformsupport01_name'), 'OrchestrationSupport')]\"\n ],\n \"properties\": {\n \"retentionDays\": 7\n }\n },\n {\n \"type\": \"Microsoft.Sql/servers/databases/extendedAuditingSettings\",\n \"apiVersion\": \"2017-03-01-preview\",\n \"name\": \"[concat(parameters('servers_platformsupport01_name'), '/OrchestrationSupport/Default')]\",\n \"dependsOn\": [\n \"[resourceId('Microsoft.Sql/servers/databases', parameters('servers_platformsupport01_name'), 'OrchestrationSupport')]\"\n ],\n \"properties\": {\n \"state\": \"Disabled\",\n \"retentionDays\": 0,\n \"storageAccountSubscriptionId\": \"00000000-0000-0000-0000-000000000000\",\n \"isAzureMonitorTargetEnabled\": false\n }\n },\n {\n \"type\": \"Microsoft.Sql/servers/databases/geoBackupPolicies\",\n \"apiVersion\": \"2014-04-01\",\n \"name\": \"[concat(parameters('servers_platformsupport01_name'), '/OrchestrationSupport/Default')]\",\n \"location\": \"UK South\",\n \"dependsOn\": [\n \"[resourceId('Microsoft.Sql/servers/databases', parameters('servers_platformsupport01_name'), 'OrchestrationSupport')]\"\n ],\n \"properties\": {\n \"state\": \"Enabled\"\n }\n },\n {\n \"type\": \"Microsoft.Sql/servers/databases/securityAlertPolicies\",\n \"apiVersion\": \"2018-06-01-preview\",\n \"name\": \"[concat(parameters('servers_platformsupport01_name'), '/OrchestrationSupport/Default')]\",\n \"dependsOn\": [\n \"[resourceId('Microsoft.Sql/servers/databases', parameters('servers_platformsupport01_name'), 'OrchestrationSupport')]\"\n ],\n \"properties\": {\n \"state\": \"Disabled\",\n \"disabledAlerts\": [\n \"\"\n ],\n \"emailAddresses\": [\n \"\"\n ],\n \"emailAccountAdmins\": false,\n \"retentionDays\": 0\n }\n },\n {\n \"type\": \"Microsoft.Sql/servers/databases/transparentDataEncryption\",\n \"apiVersion\": \"2014-04-01\",\n \"name\": \"[concat(parameters('servers_platformsupport01_name'), '/OrchestrationSupport/current')]\",\n \"location\": \"UK South\",\n \"dependsOn\": [\n \"[resourceId('Microsoft.Sql/servers/databases', parameters('servers_platformsupport01_name'), 'OrchestrationSupport')]\"\n ],\n \"properties\": {\n \"status\": \"Enabled\"\n }\n },\n {\n \"type\": \"Microsoft.Sql/servers/databases/vulnerabilityAssessments\",\n \"apiVersion\": \"2017-03-01-preview\",\n \"name\": \"[concat(parameters('servers_platformsupport01_name'), '/OrchestrationSupport/Default')]\",\n \"dependsOn\": [\n \"[resourceId('Microsoft.Sql/servers/databases', parameters('servers_platformsupport01_name'), 'OrchestrationSupport')]\"\n ],\n \"properties\": {\n \"recurringScans\": {\n \"isEnabled\": false,\n \"emailSubscriptionAdmins\": true\n }\n }\n }\n ]\n}"
},
{
"path": "ARM Templates/Synapse/v2.0 Export.json",
"content": "{\n \"$schema\": \"http://schema.management.azure.com/schemas/2015-01-01/deploymentTemplate.json#\",\n \"contentVersion\": \"1.0.0.0\",\n \"parameters\": {\n \"workspaceName\": {\n \"type\": \"string\",\n \"metadata\": \"Workspace name\",\n \"defaultValue\": \"\"\n },\n \"FrameworkFunctions_properties_typeProperties_functionAppUrl\": {\n \"type\": \"string\",\n \"defaultValue\": \"\"\n },\n \"Keys_properties_typeProperties_baseUrl\": {\n \"type\": \"string\",\n \"defaultValue\": \"\"\n },\n \"SupportDatabase_properties_typeProperties_connectionString_secretName\": {\n \"type\": \"string\",\n \"defaultValue\": \"\"\n }\n },\n \"variables\": {\n \"workspaceId\": \"[concat('Microsoft.Synapse/workspaces/', parameters('workspaceName'))]\"\n },\n \"resources\": [\n {\n \"name\": \"[concat(parameters('workspaceName'), '/01-Grandparent')]\",\n \"type\": \"Microsoft.Synapse/workspaces/pipelines\",\n \"apiVersion\": \"2019-06-01-preview\",\n \"properties\": {\n \"description\": \"procfwk grandparent pipeline used optionally to bootstrap any wider processes in your Data Factory that then calls the processing framework.\",\n \"activities\": [\n {\n \"name\": \"procfwk\",\n \"description\": \"Call procfwk\",\n \"type\": \"ExecutePipeline\",\n \"dependsOn\": [],\n \"userProperties\": [],\n \"typeProperties\": {\n \"pipeline\": {\n \"referenceName\": \"02-Parent\",\n \"type\": \"PipelineReference\"\n },\n \"waitOnCompletion\": true,\n \"parameters\": {\n \"BatchName\": {\n \"value\": \"@pipeline().parameters.BatchName\",\n \"type\": \"Expression\"\n }\n }\n }\n }\n ],\n \"parameters\": {\n \"BatchName\": {\n \"type\": \"string\",\n \"defaultValue\": \"NotUsed\"\n }\n },\n \"folder\": {\n \"name\": \"_ProcFwk\"\n },\n \"annotations\": [\n \"procfwk\",\n \"Grandparent\"\n ]\n },\n \"dependsOn\": [\n \"[concat(variables('workspaceId'), '/pipelines/02-Parent')]\"\n ]\n },\n {\n \"name\": \"[concat(parameters('workspaceName'), '/02-Parent')]\",\n \"type\": \"Microsoft.Synapse/workspaces/pipelines\",\n \"apiVersion\": \"2019-06-01-preview\",\n \"properties\": {\n \"description\": \"ADF.procfwk parent pipeline used to bootstrap the orchestration framework in perform the first level ForEach calls in sequence for the metadata stages.\",\n \"activities\": [\n {\n \"name\": \"Get Stages\",\n \"description\": \"Returns a distinct list of execution stages within the framework metadata.\",\n \"type\": \"Lookup\",\n \"dependsOn\": [\n {\n \"activity\": \"Set Execution Id\",\n \"dependencyConditions\": [\n \"Succeeded\"\n ]\n }\n ],\n \"policy\": {\n \"timeout\": \"0.00:10:00\",\n \"retry\": 0,\n \"retryIntervalInSeconds\": 30,\n \"secureOutput\": false,\n \"secureInput\": false\n },\n \"userProperties\": [],\n \"typeProperties\": {\n \"source\": {\n \"type\": \"AzureSqlSource\",\n \"sqlReaderStoredProcedureName\": \"[[procfwk].[GetStages]\",\n \"storedProcedureParameters\": {\n \"ExecutionId\": {\n \"type\": \"Guid\",\n \"value\": {\n \"value\": \"@variables('ExecutionId')\",\n \"type\": \"Expression\"\n }\n }\n },\n \"queryTimeout\": \"02:00:00\",\n \"partitionOption\": \"None\"\n },\n \"dataset\": {\n \"referenceName\": \"GetSetMetadata\",\n \"type\": \"DatasetReference\",\n \"parameters\": {}\n },\n \"firstRowOnly\": false\n }\n },\n {\n \"name\": \"Execute Stages\",\n \"description\": \"Top level ForEach to sequentially call all processing stages within the framework metadata. Items for iteration passed from the Get Stages lookup activity.\",\n \"type\": \"ForEach\",\n \"dependsOn\": [\n {\n \"activity\": \"Get Stages\",\n \"dependencyConditions\": [\n \"Succeeded\"\n ]\n }\n ],\n \"userProperties\": [],\n \"typeProperties\": {\n \"items\": {\n \"value\": \"@activity('Get Stages').output.value\",\n \"type\": \"Expression\"\n },\n \"isSequential\": true,\n \"activities\": [\n {\n \"name\": \"Stage Executor\",\n \"description\": \"Call to the framework generic child pipeline for a given execution stage.\",\n \"type\": \"ExecutePipeline\",\n \"dependsOn\": [\n {\n \"activity\": \"Log Stage Preparing\",\n \"dependencyConditions\": [\n \"Succeeded\"\n ]\n }\n ],\n \"userProperties\": [],\n \"typeProperties\": {\n \"pipeline\": {\n \"referenceName\": \"03-Child\",\n \"type\": \"PipelineReference\"\n },\n \"waitOnCompletion\": true,\n \"parameters\": {\n \"StageId\": {\n \"value\": \"@item().StageId\",\n \"type\": \"Expression\"\n },\n \"ExecutionId\": {\n \"value\": \"@variables('ExecutionId')\",\n \"type\": \"Expression\"\n }\n }\n }\n },\n {\n \"name\": \"Log Stage Preparing\",\n \"description\": \"Update the current execution table flagging all pipelines within the stage as preparing.\",\n \"type\": \"SqlServerStoredProcedure\",\n \"dependsOn\": [\n {\n \"activity\": \"Check and Update Blockers\",\n \"dependencyConditions\": [\n \"Succeeded\"\n ]\n }\n ],\n \"policy\": {\n \"timeout\": \"0.00:10:00\",\n \"retry\": 0,\n \"retryIntervalInSeconds\": 30,\n \"secureOutput\": false,\n \"secureInput\": false\n },\n \"userProperties\": [],\n \"typeProperties\": {\n \"storedProcedureName\": \"[[procfwk].[SetLogStagePreparing]\",\n \"storedProcedureParameters\": {\n \"ExecutionId\": {\n \"value\": {\n \"value\": \"@variables('ExecutionId')\",\n \"type\": \"Expression\"\n },\n \"type\": \"Guid\"\n },\n \"StageId\": {\n \"value\": {\n \"value\": \"@item().StageId\",\n \"type\": \"Expression\"\n },\n \"type\": \"Int32\"\n }\n }\n },\n \"linkedServiceName\": {\n \"referenceName\": \"SupportDatabase\",\n \"type\": \"LinkedServiceReference\"\n }\n },\n {\n \"name\": \"Check and Update Blockers\",\n \"description\": \"Used to double check and stop the next execution stage if failures and blockers have be incurred. This also depends on the failure handling property value which defines the stored procedure behaviour.\",\n \"type\": \"SqlServerStoredProcedure\",\n \"dependsOn\": [],\n \"policy\": {\n \"timeout\": \"0.00:10:00\",\n \"retry\": 0,\n \"retryIntervalInSeconds\": 30,\n \"secureOutput\": false,\n \"secureInput\": false\n },\n \"userProperties\": [],\n \"typeProperties\": {\n \"storedProcedureName\": \"[[procfwk].[CheckForBlockedPipelines]\",\n \"storedProcedureParameters\": {\n \"ExecutionId\": {\n \"value\": {\n \"value\": \"@variables('ExecutionId')\",\n \"type\": \"Expression\"\n },\n \"type\": \"Guid\"\n },\n \"StageId\": {\n \"value\": {\n \"value\": \"@item().StageId\",\n \"type\": \"Expression\"\n },\n \"type\": \"Int32\"\n }\n }\n },\n \"linkedServiceName\": {\n \"referenceName\": \"SupportDatabase\",\n \"type\": \"LinkedServiceReference\"\n }\n }\n ]\n }\n },\n {\n \"name\": \"Execution Wrapper\",\n \"description\": \"Wrapper to reset and restart processing or create a completely new execution instance of the framework metadata.\",\n \"type\": \"Lookup\",\n \"dependsOn\": [\n {\n \"activity\": \"Clean Up Previous Run\",\n \"dependencyConditions\": [\n \"Succeeded\"\n ]\n }\n ],\n \"policy\": {\n \"timeout\": \"0.00:10:00\",\n \"retry\": 0,\n \"retryIntervalInSeconds\": 30,\n \"secureOutput\": false,\n \"secureInput\": false\n },\n \"userProperties\": [],\n \"typeProperties\": {\n \"source\": {\n \"type\": \"AzureSqlSource\",\n \"sqlReaderStoredProcedureName\": \"[[procfwk].[ExecutionWrapper]\",\n \"storedProcedureParameters\": {\n \"CallingOrchestratorName\": {\n \"type\": \"String\",\n \"value\": {\n \"value\": \"@pipeline().DataFactory\",\n \"type\": \"Expression\"\n }\n },\n \"BatchName\": {\n \"type\": \"String\",\n \"value\": {\n \"value\": \"@pipeline().parameters.BatchName\",\n \"type\": \"Expression\"\n }\n }\n },\n \"queryTimeout\": \"02:00:00\",\n \"partitionOption\": \"None\"\n },\n \"dataset\": {\n \"referenceName\": \"GetSetMetadata\",\n \"type\": \"DatasetReference\",\n \"parameters\": {}\n }\n }\n },\n {\n \"name\": \"Check Outcome and Update Logs\",\n \"description\": \"After a successful execution run the current execution metadata is moved to the long term logging table by this stored procedure call. Otherwise an error will be raised.\",\n \"type\": \"SqlServerStoredProcedure\",\n \"dependsOn\": [\n {\n \"activity\": \"Execute Stages\",\n \"dependencyConditions\": [\n \"Succeeded\"\n ]\n }\n ],\n \"policy\": {\n \"timeout\": \"0.00:10:00\",\n \"retry\": 0,\n \"retryIntervalInSeconds\": 30,\n \"secureOutput\": false,\n \"secureInput\": false\n },\n \"userProperties\": [],\n \"typeProperties\": {\n \"storedProcedureName\": \"[[procfwk].[UpdateExecutionLog]\",\n \"storedProcedureParameters\": {\n \"PerformErrorCheck\": {\n \"value\": {\n \"value\": \"@bool(1)\",\n \"type\": \"Expression\"\n },\n \"type\": \"Boolean\"\n },\n \"ExecutionId\": {\n \"value\": {\n \"value\": \"@variables('ExecutionId')\",\n \"type\": \"Expression\"\n },\n \"type\": \"Guid\"\n }\n }\n },\n \"linkedServiceName\": {\n \"referenceName\": \"SupportDatabase\",\n \"type\": \"LinkedServiceReference\"\n }\n },\n {\n \"name\": \"Check Previous Execution\",\n \"description\": \"Query the current execution table for worker pipelines that require a clean up from the previous execution run.\",\n \"type\": \"Lookup\",\n \"dependsOn\": [\n {\n \"activity\": \"Execute Precursor\",\n \"dependencyConditions\": [\n \"Succeeded\"\n ]\n }\n ],\n \"policy\": {\n \"timeout\": \"0.00:10:00\",\n \"retry\": 0,\n \"retryIntervalInSeconds\": 30,\n \"secureOutput\": false,\n \"secureInput\": false\n },\n \"userProperties\": [],\n \"typeProperties\": {\n \"source\": {\n \"type\": \"AzureSqlSource\",\n \"sqlReaderStoredProcedureName\": \"[[procfwk].[CheckPreviousExeuction]\",\n \"storedProcedureParameters\": {\n \"BatchName\": {\n \"type\": \"String\",\n \"value\": {\n \"value\": \"@pipeline().parameters.BatchName\",\n \"type\": \"Expression\"\n }\n }\n },\n \"queryTimeout\": \"02:00:00\",\n \"partitionOption\": \"None\"\n },\n \"dataset\": {\n \"referenceName\": \"GetSetMetadata\",\n \"type\": \"DatasetReference\",\n \"parameters\": {}\n },\n \"firstRowOnly\": false\n }\n },\n {\n \"name\": \"Clean Up Previous Run\",\n \"description\": \"Handle Worker pipelines that are reported as Running when the parent pipeline is called again. Get what the actual status of those pipelines is.\",\n \"type\": \"ForEach\",\n \"dependsOn\": [\n {\n \"activity\": \"Check Previous Execution\",\n \"dependencyConditions\": [\n \"Succeeded\"\n ]\n },\n {\n \"activity\": \"Check Metadata Integrity\",\n \"dependencyConditions\": [\n \"Succeeded\"\n ]\n }\n ],\n \"userProperties\": [],\n \"typeProperties\": {\n \"items\": {\n \"value\": \"@activity('Check Previous Execution').output.value\",\n \"type\": \"Expression\"\n },\n \"isSequential\": false,\n \"batchCount\": 50,\n \"activities\": [\n {\n \"name\": \"Get SPN Details\",\n \"type\": \"Lookup\",\n \"dependsOn\": [],\n \"policy\": {\n \"timeout\": \"0.00:10:00\",\n \"retry\": 0,\n \"retryIntervalInSeconds\": 30,\n \"secureOutput\": true,\n \"secureInput\": false\n },\n \"userProperties\": [],\n \"typeProperties\": {\n \"source\": {\n \"type\": \"AzureSqlSource\",\n \"sqlReaderStoredProcedureName\": \"[[procfwk].[GetWorkerAuthDetails]\",\n \"storedProcedureParameters\": {\n \"ExecutionId\": {\n \"type\": \"Guid\",\n \"value\": {\n \"value\": \"@item().LocalExecutionId\",\n \"type\": \"Expression\"\n }\n },\n \"PipelineId\": {\n \"type\": \"Int32\",\n \"value\": {\n \"value\": \"@item().PipelineId\",\n \"type\": \"Expression\"\n }\n },\n \"StageId\": {\n \"type\": \"Int32\",\n \"value\": {\n \"value\": \"@item().StageId\",\n \"type\": \"Expression\"\n }\n }\n },\n \"queryTimeout\": \"02:00:00\",\n \"partitionOption\": \"None\"\n },\n \"dataset\": {\n \"referenceName\": \"GetSetMetadata\",\n \"type\": \"DatasetReference\",\n \"parameters\": {}\n }\n }\n },\n {\n \"name\": \"Log Pipeline Checking\",\n \"type\": \"SqlServerStoredProcedure\",\n \"dependsOn\": [],\n \"policy\": {\n \"timeout\": \"0.00:10:00\",\n \"retry\": 0,\n \"retryIntervalInSeconds\": 30,\n \"secureOutput\": false,\n \"secureInput\": false\n },\n \"userProperties\": [],\n \"typeProperties\": {\n \"storedProcedureName\": \"[[procfwk].[SetLogPipelineChecking]\",\n \"storedProcedureParameters\": {\n \"ExecutionId\": {\n \"value\": {\n \"value\": \"@item().LocalExecutionId\",\n \"type\": \"Expression\"\n },\n \"type\": \"Guid\"\n },\n \"PipelineId\": {\n \"value\": {\n \"value\": \"@item().PipelineId\",\n \"type\": \"Expression\"\n },\n \"type\": \"Int32\"\n },\n \"StageId\": {\n \"value\": {\n \"value\": \"@item().StageId\",\n \"type\": \"Expression\"\n },\n \"type\": \"Int32\"\n }\n }\n },\n \"linkedServiceName\": {\n \"referenceName\": \"SupportDatabase\",\n \"type\": \"LinkedServiceReference\"\n }\n },\n {\n \"name\": \"Get Pipeline Status\",\n \"type\": \"AzureFunctionActivity\",\n \"dependsOn\": [\n {\n \"activity\": \"Get SPN Details\",\n \"dependencyConditions\": [\n \"Succeeded\"\n ]\n },\n {\n \"activity\": \"Log Pipeline Checking\",\n \"dependencyConditions\": [\n \"Succeeded\"\n ]\n }\n ],\n \"policy\": {\n \"timeout\": \"0.00:10:00\",\n \"retry\": 0,\n \"retryIntervalInSeconds\": 30,\n \"secureOutput\": false,\n \"secureInput\": true\n },\n \"userProperties\": [],\n \"typeProperties\": {\n \"functionName\": \"CheckPipelineStatus\",\n \"method\": \"POST\",\n \"headers\": {},\n \"body\": {\n \"value\": \"@concat('\\n{\\n \\\"tenantId\\\": \\\"',activity('Get SPN Details').output.firstRow.TenantId,'\\\",\\n \\\"applicationId\\\": \\\"',activity('Get SPN Details').output.firstRow.AppId,'\\\",\\n \\\"authenticationKey\\\": \\\"',activity('Get SPN Details').output.firstRow.AppSecret,'\\\",\\n \\\"subscriptionId\\\": \\\"',activity('Get SPN Details').output.firstRow.SubscriptionId,'\\\",\\n \\\"resourceGroupName\\\": \\\"',item().ResourceGroupName,'\\\",\\n \\\"orchestratorName\\\": \\\"',item().OrchestratorName,'\\\",\\n \\\"orchestratorType\\\": \\\"',item().OrchestratorType,'\\\",\\n \\\"pipelineName\\\": \\\"',item().PipelineName,'\\\",\\n \\\"runId\\\": \\\"',item().PipelineRunId,'\\\"\\n}')\",\n \"type\": \"Expression\"\n }\n },\n \"linkedServiceName\": {\n \"referenceName\": \"FrameworkFunctions\",\n \"type\": \"LinkedServiceReference\"\n }\n },\n {\n \"name\": \"Set Pipeline Status\",\n \"description\": \"Update the metadata depending on the actual pipeline outcome. Using the status as the case.\",\n \"type\": \"Switch\",\n \"dependsOn\": [\n {\n \"activity\": \"Get Pipeline Status\",\n \"dependencyConditions\": [\n \"Succeeded\"\n ]\n }\n ],\n \"userProperties\": [],\n \"typeProperties\": {\n \"on\": {\n \"value\": \"@activity('Get Pipeline Status').output.ActualStatus\",\n \"type\": \"Expression\"\n },\n \"cases\": [\n {\n \"value\": \"Failed\",\n \"activities\": [\n {\n \"name\": \"Pipeline Status Failed\",\n \"description\": \"Updates the current execution table with a pipeline status of failed if the function outcome is failed. Also blocks pipelines in the downstream execution stage.\",\n \"type\": \"SqlServerStoredProcedure\",\n \"dependsOn\": [],\n \"policy\": {\n \"timeout\": \"0.00:10:00\",\n \"retry\": 0,\n \"retryIntervalInSeconds\": 30,\n \"secureOutput\": false,\n \"secureInput\": false\n },\n \"userProperties\": [],\n \"typeProperties\": {\n \"storedProcedureName\": \"[[procfwk].[SetLogPipelineFailed]\",\n \"storedProcedureParameters\": {\n \"ExecutionId\": {\n \"value\": {\n \"value\": \"@item().LocalExecutionId\",\n \"type\": \"Expression\"\n },\n \"type\": \"Guid\"\n },\n \"PipelineId\": {\n \"value\": {\n \"value\": \"@item().PipelineId\",\n \"type\": \"Expression\"\n },\n \"type\": \"Int32\"\n },\n \"RunId\": {\n \"value\": null,\n \"type\": \"Guid\"\n },\n \"StageId\": {\n \"value\": {\n \"value\": \"@item().StageId\",\n \"type\": \"Expression\"\n },\n \"type\": \"Int32\"\n }\n }\n },\n \"linkedServiceName\": {\n \"referenceName\": \"SupportDatabase\",\n \"type\": \"LinkedServiceReference\"\n }\n }\n ]\n },\n {\n \"value\": \"Succeeded\",\n \"activities\": [\n {\n \"name\": \"Pipeline Status Succeeded\",\n \"description\": \"Updates the current execution table with a pipeline status of success if the function outcome is succeeded.\",\n \"type\": \"SqlServerStoredProcedure\",\n \"dependsOn\": [],\n \"policy\": {\n \"timeout\": \"0.00:10:00\",\n \"retry\": 0,\n \"retryIntervalInSeconds\": 30,\n \"secureOutput\": false,\n \"secureInput\": false\n },\n \"userProperties\": [],\n \"typeProperties\": {\n \"storedProcedureName\": \"[[procfwk].[SetLogPipelineSuccess]\",\n \"storedProcedureParameters\": {\n \"ExecutionId\": {\n \"value\": {\n \"value\": \"@item().LocalExecutionId\",\n \"type\": \"Expression\"\n },\n \"type\": \"Guid\"\n },\n \"PipelineId\": {\n \"value\": {\n \"value\": \"@item().PipelineId\",\n \"type\": \"Expression\"\n },\n \"type\": \"Int32\"\n },\n \"StageId\": {\n \"value\": {\n \"value\": \"@item().StageId\",\n \"type\": \"Expression\"\n },\n \"type\": \"Int32\"\n }\n }\n },\n \"linkedServiceName\": {\n \"referenceName\": \"SupportDatabase\",\n \"type\": \"LinkedServiceReference\"\n }\n }\n ]\n },\n {\n \"value\": \"Queued\",\n \"activities\": [\n {\n \"name\": \"Pipeline Status Queued - Running\",\n \"description\": \"Updates the current execution table with a pipeline status of running if the function outcome is queued.\",\n \"type\": \"SqlServerStoredProcedure\",\n \"dependsOn\": [],\n \"policy\": {\n \"timeout\": \"0.00:10:00\",\n \"retry\": 0,\n \"retryIntervalInSeconds\": 30,\n \"secureOutput\": false,\n \"secureInput\": false\n },\n \"userProperties\": [],\n \"typeProperties\": {\n \"storedProcedureName\": \"[[procfwk].[SetLogPipelineRunning]\",\n \"storedProcedureParameters\": {\n \"ExecutionId\": {\n \"value\": {\n \"value\": \"@item().LocalExecutionId\",\n \"type\": \"Expression\"\n },\n \"type\": \"Guid\"\n },\n \"PipelineId\": {\n \"value\": {\n \"value\": \"@item().PipelineId\",\n \"type\": \"Expression\"\n },\n \"type\": \"Int32\"\n },\n \"StageId\": {\n \"value\": {\n \"value\": \"@item().StageId\",\n \"type\": \"Expression\"\n },\n \"type\": \"Int32\"\n }\n }\n },\n \"linkedServiceName\": {\n \"referenceName\": \"SupportDatabase\",\n \"type\": \"LinkedServiceReference\"\n }\n }\n ]\n },\n {\n \"value\": \"InProgress\",\n \"activities\": [\n {\n \"name\": \"Pipeline Status InProgress - Running\",\n \"description\": \"Updates the current execution table with a pipeline status of running if the function outcome is in progress.\",\n \"type\": \"SqlServerStoredProcedure\",\n \"dependsOn\": [],\n \"policy\": {\n \"timeout\": \"0.00:10:00\",\n \"retry\": 0,\n \"retryIntervalInSeconds\": 30,\n \"secureOutput\": false,\n \"secureInput\": false\n },\n \"userProperties\": [],\n \"typeProperties\": {\n \"storedProcedureName\": \"[[procfwk].[SetLogPipelineRunning]\",\n \"storedProcedureParameters\": {\n \"ExecutionId\": {\n \"value\": {\n \"value\": \"@item().LocalExecutionId\",\n \"type\": \"Expression\"\n },\n \"type\": \"Guid\"\n },\n \"PipelineId\": {\n \"value\": {\n \"value\": \"@item().PipelineId\",\n \"type\": \"Expression\"\n },\n \"type\": \"Int32\"\n },\n \"StageId\": {\n \"value\": {\n \"value\": \"@item().StageId\",\n \"type\": \"Expression\"\n },\n \"type\": \"Int32\"\n }\n }\n },\n \"linkedServiceName\": {\n \"referenceName\": \"SupportDatabase\",\n \"type\": \"LinkedServiceReference\"\n }\n }\n ]\n },\n {\n \"value\": \"Cancelled\",\n \"activities\": [\n {\n \"name\": \"Pipeline Status Cancelled\",\n \"description\": \"Updates the current execution table with a pipeline status of cancelled if the function outcome is cancelled.\",\n \"type\": \"SqlServerStoredProcedure\",\n \"dependsOn\": [],\n \"policy\": {\n \"timeout\": \"0.00:10:00\",\n \"retry\": 0,\n \"retryIntervalInSeconds\": 30,\n \"secureOutput\": false,\n \"secureInput\": false\n },\n \"userProperties\": [],\n \"typeProperties\": {\n \"storedProcedureName\": \"[[procfwk].[SetLogPipelineCancelled]\",\n \"storedProcedureParameters\": {\n \"ExecutionId\": {\n \"value\": {\n \"value\": \"@item().LocalExecutionId\",\n \"type\": \"Expression\"\n },\n \"type\": \"Guid\"\n },\n \"PipelineId\": {\n \"value\": {\n \"value\": \"@item().PipelineId\",\n \"type\": \"Expression\"\n },\n \"type\": \"Int32\"\n },\n \"StageId\": {\n \"value\": {\n \"value\": \"@item().StageId\",\n \"type\": \"Expression\"\n },\n \"type\": \"Int32\"\n },\n \"CleanUpRun\": {\n \"value\": {\n \"value\": \"@bool(1)\",\n \"type\": \"Expression\"\n },\n \"type\": \"Boolean\"\n }\n }\n },\n \"linkedServiceName\": {\n \"referenceName\": \"SupportDatabase\",\n \"type\": \"LinkedServiceReference\"\n }\n }\n ]\n }\n ],\n \"defaultActivities\": [\n {\n \"name\": \"Pipeline Status Unknown\",\n \"description\": \"Updates the current execution table with a pipeline status of unknown if the function returns an unexpected outcome.\",\n \"type\": \"SqlServerStoredProcedure\",\n \"dependsOn\": [],\n \"policy\": {\n \"timeout\": \"0.00:10:00\",\n \"retry\": 0,\n \"retryIntervalInSeconds\": 30,\n \"secureOutput\": false,\n \"secureInput\": false\n },\n \"userProperties\": [],\n \"typeProperties\": {\n \"storedProcedureName\": \"[[procfwk].[SetLogPipelineUnknown]\",\n \"storedProcedureParameters\": {\n \"ExecutionId\": {\n \"value\": {\n \"value\": \"@item().LocalExecutionId\",\n \"type\": \"Expression\"\n },\n \"type\": \"Guid\"\n },\n \"PipelineId\": {\n \"value\": {\n \"value\": \"@item().PipelineId\",\n \"type\": \"Expression\"\n },\n \"type\": \"Int32\"\n },\n \"StageId\": {\n \"value\": {\n \"value\": \"@item().StageId\",\n \"type\": \"Expression\"\n },\n \"type\": \"Int32\"\n },\n \"CleanUpRun\": {\n \"value\": {\n \"value\": \"@bool(1)\",\n \"type\": \"Expression\"\n },\n \"type\": \"Boolean\"\n }\n }\n },\n \"linkedServiceName\": {\n \"referenceName\": \"SupportDatabase\",\n \"type\": \"LinkedServiceReference\"\n }\n }\n ]\n }\n },\n {\n \"name\": \"Set Last Check DateTime\",\n \"description\": \"Update the current execution table with a date time from when the function last checked the pipeline status.\",\n \"type\": \"SqlServerStoredProcedure\",\n \"dependsOn\": [\n {\n \"activity\": \"Get Pipeline Status\",\n \"dependencyConditions\": [\n \"Succeeded\"\n ]\n }\n ],\n \"policy\": {\n \"timeout\": \"0.00:10:00\",\n \"retry\": 0,\n \"retryIntervalInSeconds\": 30,\n \"secureOutput\": false,\n \"secureInput\": false\n },\n \"userProperties\": [],\n \"typeProperties\": {\n \"storedProcedureName\": \"[[procfwk].[SetLogPipelineLastStatusCheck]\",\n \"storedProcedureParameters\": {\n \"ExecutionId\": {\n \"value\": {\n \"value\": \"@item().LocalExecutionId\",\n \"type\": \"Expression\"\n },\n \"type\": \"Guid\"\n },\n \"PipelineId\": {\n \"value\": {\n \"value\": \"@item().PipelineId\",\n \"type\": \"Expression\"\n },\n \"type\": \"Int32\"\n },\n \"StageId\": {\n \"value\": {\n \"value\": \"@item().StageId\",\n \"type\": \"Expression\"\n },\n \"type\": \"Int32\"\n }\n }\n },\n \"linkedServiceName\": {\n \"referenceName\": \"SupportDatabase\",\n \"type\": \"LinkedServiceReference\"\n }\n }\n ]\n }\n },\n {\n \"name\": \"Execute Precursor\",\n \"description\": \"Uses the database property value ExecutionPrecursorProc to run any custom logic against the metadata database before the execution run starts.\",\n \"type\": \"SqlServerStoredProcedure\",\n \"dependsOn\": [],\n \"policy\": {\n \"timeout\": \"0.00:10:00\",\n \"retry\": 0,\n \"retryIntervalInSeconds\": 30,\n \"secureOutput\": false,\n \"secureInput\": false\n },\n \"userProperties\": [],\n \"typeProperties\": {\n \"storedProcedureName\": \"[[procfwk].[ExecutePrecursorProcedure]\"\n },\n \"linkedServiceName\": {\n \"referenceName\": \"SupportDatabase\",\n \"type\": \"LinkedServiceReference\"\n }\n },\n {\n \"name\": \"Set Execution Id\",\n \"description\": \"Set the local execution Id to a pipeline variable for each in several downstream activities.\",\n \"type\": \"SetVariable\",\n \"dependsOn\": [\n {\n \"activity\": \"Execution Wrapper\",\n \"dependencyConditions\": [\n \"Succeeded\"\n ]\n }\n ],\n \"userProperties\": [],\n \"typeProperties\": {\n \"variableName\": \"ExecutionId\",\n \"value\": {\n \"value\": \"@activity('Execution Wrapper').output.firstRow.ExecutionId\",\n \"type\": \"Expression\"\n }\n }\n },\n {\n \"name\": \"Check Metadata Integrity\",\n \"description\": \"Performs a series of checks on all metadata held in the framework SQLDB. This is intended to raise errors before an execution run even starts.\",\n \"type\": \"SqlServerStoredProcedure\",\n \"dependsOn\": [\n {\n \"activity\": \"Execute Precursor\",\n \"dependencyConditions\": [\n \"Succeeded\"\n ]\n }\n ],\n \"policy\": {\n \"timeout\": \"0.00:10:00\",\n \"retry\": 0,\n \"retryIntervalInSeconds\": 30,\n \"secureOutput\": false,\n \"secureInput\": false\n },\n \"userProperties\": [],\n \"typeProperties\": {\n \"storedProcedureName\": \"[[procfwk].[CheckMetadataIntegrity]\",\n \"storedProcedureParameters\": {\n \"BatchName\": {\n \"value\": {\n \"value\": \"@pipeline().parameters.BatchName\",\n \"type\": \"Expression\"\n },\n \"type\": \"String\"\n },\n \"DebugMode\": {\n \"value\": {\n \"value\": \"@bool(0)\",\n \"type\": \"Expression\"\n },\n \"type\": \"Boolean\"\n }\n }\n },\n \"linkedServiceName\": {\n \"referenceName\": \"SupportDatabase\",\n \"type\": \"LinkedServiceReference\"\n }\n }\n ],\n \"parameters\": {\n \"BatchName\": {\n \"type\": \"string\",\n \"defaultValue\": \"NotUsed\"\n }\n },\n \"variables\": {\n \"ExecutionId\": {\n \"type\": \"String\"\n }\n },\n \"folder\": {\n \"name\": \"_ProcFwk\"\n },\n \"annotations\": [\n \"procfwk\",\n \"Parent\"\n ]\n },\n \"dependsOn\": [\n \"[concat(variables('workspaceId'), '/datasets/GetSetMetadata')]\",\n \"[concat(variables('workspaceId'), '/linkedServices/SupportDatabase')]\",\n \"[concat(variables('workspaceId'), '/pipelines/03-Child')]\",\n \"[concat(variables('workspaceId'), '/linkedServices/FrameworkFunctions')]\"\n ]\n },\n {\n \"name\": \"[concat(parameters('workspaceName'), '/03-Child')]\",\n \"type\": \"Microsoft.Synapse/workspaces/pipelines\",\n \"apiVersion\": \"2019-06-01-preview\",\n \"properties\": {\n \"description\": \"procfwk child pipeline used to execute Worker pipelines within a given execution stage. This pipeline will be called once for each stage, then execute all Workers in parallel.\",\n \"activities\": [\n {\n \"name\": \"Get Pipelines\",\n \"description\": \"Returns all pipelines from the metadata to be executed within a given processing stage.\",\n \"type\": \"Lookup\",\n \"dependsOn\": [],\n \"policy\": {\n \"timeout\": \"0.00:10:00\",\n \"retry\": 0,\n \"retryIntervalInSeconds\": 30,\n \"secureOutput\": false,\n \"secureInput\": false\n },\n \"userProperties\": [],\n \"typeProperties\": {\n \"source\": {\n \"type\": \"AzureSqlSource\",\n \"sqlReaderStoredProcedureName\": \"[[procfwk].[GetPipelinesInStage]\",\n \"storedProcedureParameters\": {\n \"ExecutionId\": {\n \"type\": \"Guid\",\n \"value\": {\n \"value\": \"@pipeline().parameters.ExecutionId\",\n \"type\": \"Expression\"\n }\n },\n \"StageId\": {\n \"type\": \"Int32\",\n \"value\": {\n \"value\": \"@pipeline().parameters.StageId\",\n \"type\": \"Expression\"\n }\n }\n },\n \"queryTimeout\": \"02:00:00\",\n \"partitionOption\": \"None\"\n },\n \"dataset\": {\n \"referenceName\": \"GetSetMetadata\",\n \"type\": \"DatasetReference\",\n \"parameters\": {}\n },\n \"firstRowOnly\": false\n }\n },\n {\n \"name\": \"Execute Pipelines\",\n \"description\": \"Second level ForEach to run in parallel all pipelines within the stage. Items for iteration passed from the Get Pipelines lookup activity.\",\n \"type\": \"ForEach\",\n \"dependsOn\": [\n {\n \"activity\": \"Get Pipelines\",\n \"dependencyConditions\": [\n \"Succeeded\"\n ]\n }\n ],\n \"userProperties\": [],\n \"typeProperties\": {\n \"items\": {\n \"value\": \"@activity('Get Pipelines').output.value\",\n \"type\": \"Expression\"\n },\n \"isSequential\": false,\n \"batchCount\": 50,\n \"activities\": [\n {\n \"name\": \"Worker Pipeline Executor\",\n \"description\": \"Run the required worker pipeline and wait for its completion. Update metadata once done.\",\n \"type\": \"ExecutePipeline\",\n \"dependsOn\": [],\n \"userProperties\": [],\n \"typeProperties\": {\n \"pipeline\": {\n \"referenceName\": \"04-Infant\",\n \"type\": \"PipelineReference\"\n },\n \"waitOnCompletion\": true,\n \"parameters\": {\n \"executionId\": {\n \"value\": \"@pipeline().parameters.ExecutionId\",\n \"type\": \"Expression\"\n },\n \"stageId\": {\n \"value\": \"@pipeline().parameters.StageId\",\n \"type\": \"Expression\"\n },\n \"pipelineId\": {\n \"value\": \"@item().PipelineId\",\n \"type\": \"Expression\"\n }\n }\n }\n }\n ]\n }\n }\n ],\n \"parameters\": {\n \"StageId\": {\n \"type\": \"int\"\n },\n \"ExecutionId\": {\n \"type\": \"string\"\n }\n },\n \"folder\": {\n \"name\": \"_ProcFwk\"\n },\n \"annotations\": [\n \"procfwk\",\n \"Child\"\n ]\n },\n \"dependsOn\": [\n \"[concat(variables('workspaceId'), '/datasets/GetSetMetadata')]\",\n \"[concat(variables('workspaceId'), '/pipelines/04-Infant')]\"\n ]\n },\n {\n \"name\": \"[concat(parameters('workspaceName'), '/04-Infant')]\",\n \"type\": \"Microsoft.Synapse/workspaces/pipelines\",\n \"apiVersion\": \"2019-06-01-preview\",\n \"properties\": {\n \"description\": \"procfwk infant pipeline used to check when the processing pipeline called by the Child completes and passes the resulting status back to the metadata database.\",\n \"activities\": [\n {\n \"name\": \"Execute Worker Pipeline\",\n \"description\": \"The lowest level executor with the metadata framework to call existing processing pipelines within Data Factory. The function called will block processing and wait for an outcome.\",\n \"type\": \"AzureFunctionActivity\",\n \"dependsOn\": [\n {\n \"activity\": \"Log Pipeline Running\",\n \"dependencyConditions\": [\n \"Succeeded\"\n ]\n },\n {\n \"activity\": \"Get Pipeline Params\",\n \"dependencyConditions\": [\n \"Succeeded\"\n ]\n }\n ],\n \"policy\": {\n \"timeout\": \"0.00:10:00\",\n \"retry\": 0,\n \"retryIntervalInSeconds\": 30,\n \"secureOutput\": false,\n \"secureInput\": true\n },\n \"userProperties\": [],\n \"typeProperties\": {\n \"functionName\": \"ExecutePipeline\",\n \"method\": \"POST\",\n \"headers\": {},\n \"body\": {\n \"value\": \"@concat('\\n{\\n \\\"tenantId\\\": \\\"',variables('WorkerCoreDetails')[0].tenantId,'\\\",\\n \\\"applicationId\\\": \\\"',variables('WorkerCoreDetails')[0].applicationId,'\\\",\\n \\\"authenticationKey\\\": \\\"',variables('WorkerCoreDetails')[0].authenticationKey,'\\\",\\n \\\"subscriptionId\\\": \\\"',variables('WorkerCoreDetails')[0].subscriptionId,'\\\",\\n \\\"resourceGroupName\\\": \\\"',variables('WorkerCoreDetails')[0].resourceGroupName,'\\\",\\n\\t\\\"orchestratorName\\\": \\\"',variables('WorkerCoreDetails')[0].orchestratorName,'\\\",\\n \\\"orchestratorType\\\": \\\"',variables('WorkerCoreDetails')[0].orchestratorType,'\\\",\\n \\\"pipelineName\\\": \\\"',variables('WorkerCoreDetails')[0].pipelineName,'\\\"',activity('Get Pipeline Params').output.firstRow.Params,'\\n}')\",\n \"type\": \"Expression\"\n }\n },\n \"linkedServiceName\": {\n \"referenceName\": \"FrameworkFunctions\",\n \"type\": \"LinkedServiceReference\"\n }\n },\n {\n \"name\": \"Get Pipeline Params\",\n \"description\": \"Returns any parameters from metadata required for the processing pipeline being called. The output can be an empty string if no parameters are required.\",\n \"type\": \"Lookup\",\n \"dependsOn\": [],\n \"policy\": {\n \"timeout\": \"0.00:10:00\",\n \"retry\": 0,\n \"retryIntervalInSeconds\": 30,\n \"secureOutput\": false,\n \"secureInput\": false\n },\n \"userProperties\": [],\n \"typeProperties\": {\n \"source\": {\n \"type\": \"AzureSqlSource\",\n \"sqlReaderStoredProcedureName\": \"[[procfwk].[GetPipelineParameters]\",\n \"storedProcedureParameters\": {\n \"PipelineId\": {\n \"type\": \"Int32\",\n \"value\": {\n \"value\": \"@pipeline().parameters.pipelineId\",\n \"type\": \"Expression\"\n }\n }\n },\n \"queryTimeout\": \"02:00:00\",\n \"partitionOption\": \"None\"\n },\n \"dataset\": {\n \"referenceName\": \"GetSetMetadata\",\n \"type\": \"DatasetReference\",\n \"parameters\": {}\n }\n }\n },\n {\n \"name\": \"Log Pipeline Running\",\n \"description\": \"Sets the current pipeline with a status of running within the current execution database table.\",\n \"type\": \"SqlServerStoredProcedure\",\n \"dependsOn\": [\n {\n \"activity\": \"Is Target Worker Validate\",\n \"dependencyConditions\": [\n \"Succeeded\"\n ]\n }\n ],\n \"policy\": {\n \"timeout\": \"0.00:10:00\",\n \"retry\": 0,\n \"retryIntervalInSeconds\": 30,\n \"secureOutput\": false,\n \"secureInput\": false\n },\n \"userProperties\": [],\n \"typeProperties\": {\n \"storedProcedureName\": \"[[procfwk].[SetLogPipelineRunning]\",\n \"storedProcedureParameters\": {\n \"ExecutionId\": {\n \"value\": {\n \"value\": \"@pipeline().parameters.ExecutionId\",\n \"type\": \"Expression\"\n },\n \"type\": \"Guid\"\n },\n \"PipelineId\": {\n \"value\": {\n \"value\": \"@pipeline().parameters.pipelineId\",\n \"type\": \"Expression\"\n },\n \"type\": \"Int32\"\n },\n \"StageId\": {\n \"value\": {\n \"value\": \"@pipeline().parameters.StageId\",\n \"type\": \"Expression\"\n },\n \"type\": \"Int32\"\n }\n }\n },\n \"linkedServiceName\": {\n \"referenceName\": \"SupportDatabase\",\n \"type\": \"LinkedServiceReference\"\n }\n },\n {\n \"name\": \"Log Execute Function Activity Failure\",\n \"description\": \"Handle true failures from calling out to the Azure Function and update the current execution table accordingly so a restart can occur.\",\n \"type\": \"SqlServerStoredProcedure\",\n \"dependsOn\": [\n {\n \"activity\": \"Execute Worker Pipeline\",\n \"dependencyConditions\": [\n \"Failed\"\n ]\n }\n ],\n \"policy\": {\n \"timeout\": \"0.00:10:00\",\n \"retry\": 0,\n \"retryIntervalInSeconds\": 30,\n \"secureOutput\": false,\n \"secureInput\": false\n },\n \"userProperties\": [],\n \"typeProperties\": {\n \"storedProcedureName\": \"[[procfwk].[SetLogActivityFailed]\",\n \"storedProcedureParameters\": {\n \"ExecutionId\": {\n \"value\": {\n \"value\": \"@pipeline().parameters.ExecutionId\",\n \"type\": \"Expression\"\n },\n \"type\": \"Guid\"\n },\n \"PipelineId\": {\n \"value\": {\n \"value\": \"@pipeline().parameters.pipelineId\",\n \"type\": \"Expression\"\n },\n \"type\": \"Int32\"\n },\n \"StageId\": {\n \"value\": {\n \"value\": \"@pipeline().parameters.StageId\",\n \"type\": \"Expression\"\n },\n \"type\": \"Int32\"\n },\n \"CallingActivity\": {\n \"value\": \"ExecuteWorkerPipeline\",\n \"type\": \"String\"\n }\n }\n },\n \"linkedServiceName\": {\n \"referenceName\": \"SupportDatabase\",\n \"type\": \"LinkedServiceReference\"\n }\n },\n {\n \"name\": \"Update Run Id\",\n \"description\": \"Provide the actual ADF run ID back to the current execution table for long term logging and alignment between the metadata other Azure monitoring tools.\",\n \"type\": \"SqlServerStoredProcedure\",\n \"dependsOn\": [\n {\n \"activity\": \"Set Run Id\",\n \"dependencyConditions\": [\n \"Succeeded\"\n ]\n }\n ],\n \"policy\": {\n \"timeout\": \"0.00:10:00\",\n \"retry\": 0,\n \"retryIntervalInSeconds\": 30,\n \"secureOutput\": false,\n \"secureInput\": false\n },\n \"userProperties\": [],\n \"typeProperties\": {\n \"storedProcedureName\": \"[[procfwk].[SetLogPipelineRunId]\",\n \"storedProcedureParameters\": {\n \"ExecutionId\": {\n \"value\": {\n \"value\": \"@pipeline().parameters.ExecutionId\",\n \"type\": \"Expression\"\n },\n \"type\": \"Guid\"\n },\n \"PipelineId\": {\n \"value\": {\n \"value\": \"@pipeline().parameters.pipelineId\",\n \"type\": \"Expression\"\n },\n \"type\": \"Int32\"\n },\n \"RunId\": {\n \"value\": {\n \"value\": \"@variables('WorkerRunId')\",\n \"type\": \"Expression\"\n },\n \"type\": \"Guid\"\n },\n \"StageId\": {\n \"value\": {\n \"value\": \"@pipeline().parameters.StageId\",\n \"type\": \"Expression\"\n },\n \"type\": \"Int32\"\n }\n }\n },\n \"linkedServiceName\": {\n \"referenceName\": \"SupportDatabase\",\n \"type\": \"LinkedServiceReference\"\n }\n },\n {\n \"name\": \"Check For Alerts\",\n \"description\": \"Checks the properties tables and if any recipients in the database require alerts sending for the current pipeline ID.\",\n \"type\": \"Lookup\",\n \"dependsOn\": [\n {\n \"activity\": \"Update Run Id\",\n \"dependencyConditions\": [\n \"Succeeded\"\n ]\n },\n {\n \"activity\": \"Set Pipeline Result\",\n \"dependencyConditions\": [\n \"Completed\"\n ]\n }\n ],\n \"policy\": {\n \"timeout\": \"0.00:10:00\",\n \"retry\": 0,\n \"retryIntervalInSeconds\": 30,\n \"secureOutput\": false,\n \"secureInput\": false\n },\n \"userProperties\": [],\n \"typeProperties\": {\n \"source\": {\n \"type\": \"AzureSqlSource\",\n \"sqlReaderStoredProcedureName\": \"[[procfwk].[CheckForEmailAlerts]\",\n \"storedProcedureParameters\": {\n \"PipelineId\": {\n \"type\": \"Int32\",\n \"value\": {\n \"value\": \"@pipeline().parameters.pipelineId\",\n \"type\": \"Expression\"\n }\n }\n },\n \"queryTimeout\": \"02:00:00\",\n \"partitionOption\": \"None\"\n },\n \"dataset\": {\n \"referenceName\": \"GetSetMetadata\",\n \"type\": \"DatasetReference\",\n \"parameters\": {}\n },\n \"firstRowOnly\": true\n }\n },\n {\n \"name\": \"Send Alerts\",\n \"description\": \"True = alerts need sending.\\nFalse = do nothing.\",\n \"type\": \"IfCondition\",\n \"dependsOn\": [\n {\n \"activity\": \"Check For Alerts\",\n \"dependencyConditions\": [\n \"Succeeded\"\n ]\n }\n ],\n \"userProperties\": [],\n \"typeProperties\": {\n \"expression\": {\n \"value\": \"@activity('Check For Alerts').output.firstRow.SendAlerts\",\n \"type\": \"Expression\"\n },\n \"ifTrueActivities\": [\n {\n \"name\": \"Get Email Parts\",\n \"description\": \"Return all required content from the metadata database to send an email alerting using the procfwk. The lookup returns the exact content for the function body request.\",\n \"type\": \"Lookup\",\n \"dependsOn\": [],\n \"policy\": {\n \"timeout\": \"0.00:10:00\",\n \"retry\": 0,\n \"retryIntervalInSeconds\": 30,\n \"secureOutput\": true,\n \"secureInput\": false\n },\n \"userProperties\": [],\n \"typeProperties\": {\n \"source\": {\n \"type\": \"AzureSqlSource\",\n \"sqlReaderStoredProcedureName\": \"[[procfwk].[GetEmailAlertParts]\",\n \"storedProcedureParameters\": {\n \"PipelineId\": {\n \"type\": \"Int32\",\n \"value\": {\n \"value\": \"@pipeline().parameters.pipelineId\",\n \"type\": \"Expression\"\n }\n }\n },\n \"queryTimeout\": \"02:00:00\",\n \"partitionOption\": \"None\"\n },\n \"dataset\": {\n \"referenceName\": \"GetSetMetadata\",\n \"type\": \"DatasetReference\",\n \"parameters\": {}\n },\n \"firstRowOnly\": true\n }\n },\n {\n \"name\": \"Call Email Sender\",\n \"description\": \"Pass off email request to Utils Send Email pipeline.\",\n \"type\": \"ExecutePipeline\",\n \"dependsOn\": [\n {\n \"activity\": \"Get Email Parts\",\n \"dependencyConditions\": [\n \"Succeeded\"\n ]\n }\n ],\n \"userProperties\": [],\n \"typeProperties\": {\n \"pipeline\": {\n \"referenceName\": \"Email Sender\",\n \"type\": \"PipelineReference\"\n },\n \"waitOnCompletion\": true,\n \"parameters\": {\n \"Recipients\": {\n \"value\": \"@activity('Get Email Parts').output.firstRow.emailRecipients\",\n \"type\": \"Expression\"\n },\n \"CcRecipients\": {\n \"value\": \"@activity('Get Email Parts').output.firstRow.emailCcRecipients\",\n \"type\": \"Expression\"\n },\n \"BccRecipients\": {\n \"value\": \"@activity('Get Email Parts').output.firstRow.emailBccRecipients\",\n \"type\": \"Expression\"\n },\n \"Subject\": {\n \"value\": \"@activity('Get Email Parts').output.firstRow.emailSubject\",\n \"type\": \"Expression\"\n },\n \"Body\": {\n \"value\": \"@activity('Get Email Parts').output.firstRow.emailBody\",\n \"type\": \"Expression\"\n },\n \"Importance\": {\n \"value\": \"@activity('Get Email Parts').output.firstRow.emailImportance\",\n \"type\": \"Expression\"\n }\n }\n }\n }\n ]\n }\n },\n {\n \"name\": \"Wait Until Pipeline Completes\",\n \"description\": \"Loops until the Worker pipeline called completes.\\n\\nSimple status:\\n- Running = new iteration.\\n- Done = break.\",\n \"type\": \"Until\",\n \"dependsOn\": [\n {\n \"activity\": \"Get Wait Duration\",\n \"dependencyConditions\": [\n \"Succeeded\"\n ]\n },\n {\n \"activity\": \"Execute Worker Pipeline\",\n \"dependencyConditions\": [\n \"Succeeded\"\n ]\n },\n {\n \"activity\": \"Set Run Id\",\n \"dependencyConditions\": [\n \"Succeeded\"\n ]\n }\n ],\n \"userProperties\": [],\n \"typeProperties\": {\n \"expression\": {\n \"value\": \"@variables('WorkerPipelineState')\",\n \"type\": \"Expression\"\n },\n \"activities\": [\n {\n \"name\": \"Get Worker Pipeline Status\",\n \"description\": \"Checks the status of a given processing pipeline and provides the value for the downstream framework activities to act upon.\",\n \"type\": \"AzureFunctionActivity\",\n \"dependsOn\": [],\n \"policy\": {\n \"timeout\": \"0.00:10:00\",\n \"retry\": 0,\n \"retryIntervalInSeconds\": 30,\n \"secureOutput\": false,\n \"secureInput\": true\n },\n \"userProperties\": [],\n \"typeProperties\": {\n \"functionName\": \"CheckPipelineStatus\",\n \"method\": \"POST\",\n \"headers\": {},\n \"body\": {\n \"value\": \"@concat('\\n{\\n \\\"tenantId\\\": \\\"',variables('WorkerCoreDetails')[0].tenantId,'\\\",\\n \\\"applicationId\\\": \\\"',variables('WorkerCoreDetails')[0].applicationId,'\\\",\\n \\\"authenticationKey\\\": \\\"',variables('WorkerCoreDetails')[0].authenticationKey,'\\\",\\n \\\"subscriptionId\\\": \\\"',variables('WorkerCoreDetails')[0].subscriptionId,'\\\",\\n \\\"resourceGroupName\\\": \\\"',variables('WorkerCoreDetails')[0].resourceGroupName,'\\\",\\n\\t\\\"orchestratorName\\\": \\\"',variables('WorkerCoreDetails')[0].orchestratorName,'\\\",\\n \\\"orchestratorType\\\": \\\"',variables('WorkerCoreDetails')[0].orchestratorType,'\\\",\\n \\\"pipelineName\\\": \\\"',variables('WorkerCoreDetails')[0].pipelineName,'\\\",\\n \\\"runId\\\": \\\"',variables('WorkerRunId'),'\\\"\\n}')\",\n \"type\": \"Expression\"\n }\n },\n \"linkedServiceName\": {\n \"referenceName\": \"FrameworkFunctions\",\n \"type\": \"LinkedServiceReference\"\n }\n },\n {\n \"name\": \"Wait If Running\",\n \"description\": \"True = Do nothing.\\nFalse = Wait, before the next iteration.\",\n \"type\": \"IfCondition\",\n \"dependsOn\": [\n {\n \"activity\": \"Set Worker State\",\n \"dependencyConditions\": [\n \"Succeeded\"\n ]\n }\n ],\n \"userProperties\": [],\n \"typeProperties\": {\n \"expression\": {\n \"value\": \"@variables('WorkerPipelineState')\",\n \"type\": \"Expression\"\n },\n \"ifFalseActivities\": [\n {\n \"name\": \"Wait for Pipeline\",\n \"description\": \"The processing pipeline is still running so Wait before checking its status again.\",\n \"type\": \"Wait\",\n \"dependsOn\": [],\n \"userProperties\": [],\n \"typeProperties\": {\n \"waitTimeInSeconds\": {\n \"value\": \"@activity('Get Wait Duration').output.firstRow.PropertyValue\",\n \"type\": \"Expression\"\n }\n }\n }\n ]\n }\n },\n {\n \"name\": \"Set Last Check DateTime\",\n \"description\": \"Update the current execution table with a date time from when the Worker pipeline status was last checked as part of the Until iterations.\",\n \"type\": \"SqlServerStoredProcedure\",\n \"dependsOn\": [\n {\n \"activity\": \"Get Worker Pipeline Status\",\n \"dependencyConditions\": [\n \"Succeeded\"\n ]\n }\n ],\n \"policy\": {\n \"timeout\": \"0.00:10:00\",\n \"retry\": 0,\n \"retryIntervalInSeconds\": 30,\n \"secureOutput\": false,\n \"secureInput\": false\n },\n \"userProperties\": [],\n \"typeProperties\": {\n \"storedProcedureName\": \"[[procfwk].[SetLogPipelineLastStatusCheck]\",\n \"storedProcedureParameters\": {\n \"ExecutionId\": {\n \"value\": {\n \"value\": \"@pipeline().parameters.executionId\",\n \"type\": \"Expression\"\n },\n \"type\": \"Guid\"\n },\n \"PipelineId\": {\n \"value\": {\n \"value\": \"@pipeline().parameters.pipelineId\",\n \"type\": \"Expression\"\n },\n \"type\": \"Int32\"\n },\n \"StageId\": {\n \"value\": {\n \"value\": \"@pipeline().parameters.stageId\",\n \"type\": \"Expression\"\n },\n \"type\": \"Int32\"\n }\n }\n },\n \"linkedServiceName\": {\n \"referenceName\": \"SupportDatabase\",\n \"type\": \"LinkedServiceReference\"\n }\n },\n {\n \"name\": \"Log Check Function Activity Failure\",\n \"description\": \"Report to the current execution table that the framework pipeline activity has failed. This failure is outside of the scope of the framework and is probably related to a wider platform problem.\",\n \"type\": \"SqlServerStoredProcedure\",\n \"dependsOn\": [\n {\n \"activity\": \"Get Worker Pipeline Status\",\n \"dependencyConditions\": [\n \"Failed\"\n ]\n }\n ],\n \"policy\": {\n \"timeout\": \"0.00:10:00\",\n \"retry\": 0,\n \"retryIntervalInSeconds\": 30,\n \"secureOutput\": false,\n \"secureInput\": false\n },\n \"userProperties\": [],\n \"typeProperties\": {\n \"storedProcedureName\": \"[[procfwk].[SetLogActivityFailed]\",\n \"storedProcedureParameters\": {\n \"CallingActivity\": {\n \"value\": \"GetWorkerPipelineStatus\",\n \"type\": \"String\"\n },\n \"ExecutionId\": {\n \"value\": {\n \"value\": \"@pipeline().parameters.executionId\",\n \"type\": \"Expression\"\n },\n \"type\": \"Guid\"\n },\n \"PipelineId\": {\n \"value\": {\n \"value\": \"@pipeline().parameters.pipelineId\",\n \"type\": \"Expression\"\n },\n \"type\": \"Int32\"\n },\n \"StageId\": {\n \"value\": {\n \"value\": \"@pipeline().parameters.stageId\",\n \"type\": \"Expression\"\n },\n \"type\": \"Int32\"\n }\n }\n },\n \"linkedServiceName\": {\n \"referenceName\": \"SupportDatabase\",\n \"type\": \"LinkedServiceReference\"\n }\n },\n {\n \"name\": \"Set Worker State\",\n \"description\": \"Set the bool state of the Worker pipeline to be used by the Until and If expressions. True = Complete, False = Running.\",\n \"type\": \"SetVariable\",\n \"dependsOn\": [\n {\n \"activity\": \"Get Worker Pipeline Status\",\n \"dependencyConditions\": [\n \"Succeeded\"\n ]\n }\n ],\n \"userProperties\": [],\n \"typeProperties\": {\n \"variableName\": \"WorkerPipelineState\",\n \"value\": {\n \"value\": \"@equals('Complete',activity('Get Worker Pipeline Status').output.SimpleStatus)\",\n \"type\": \"Expression\"\n }\n }\n }\n ],\n \"timeout\": \"0.00:10:00\"\n }\n },\n {\n \"name\": \"Set Pipeline Result\",\n \"description\": \"Receives the outcome from the function execution for a given processing pipeline and updates the current execution table with different pipelines status values depending on the result (case).\",\n \"type\": \"Switch\",\n \"dependsOn\": [\n {\n \"activity\": \"Wait Until Pipeline Completes\",\n \"dependencyConditions\": [\n \"Completed\"\n ]\n }\n ],\n \"userProperties\": [],\n \"typeProperties\": {\n \"on\": {\n \"value\": \"@activity('Get Worker Pipeline Status').output.ActualStatus\",\n \"type\": \"Expression\"\n },\n \"cases\": [\n {\n \"value\": \"Succeeded\",\n \"activities\": [\n {\n \"name\": \"Pipeline Status Succeeded\",\n \"description\": \"Updates the current execution table with a pipeline status of success if the function outcome is succeeded.\",\n \"type\": \"SqlServerStoredProcedure\",\n \"dependsOn\": [],\n \"policy\": {\n \"timeout\": \"0.00:10:00\",\n \"retry\": 0,\n \"retryIntervalInSeconds\": 30,\n \"secureOutput\": false,\n \"secureInput\": false\n },\n \"userProperties\": [],\n \"typeProperties\": {\n \"storedProcedureName\": \"[[procfwk].[SetLogPipelineSuccess]\",\n \"storedProcedureParameters\": {\n \"ExecutionId\": {\n \"value\": {\n \"value\": \"@pipeline().parameters.executionId\",\n \"type\": \"Expression\"\n },\n \"type\": \"Guid\"\n },\n \"PipelineId\": {\n \"value\": {\n \"value\": \"@pipeline().parameters.pipelineId\",\n \"type\": \"Expression\"\n },\n \"type\": \"Int32\"\n },\n \"StageId\": {\n \"value\": {\n \"value\": \"@pipeline().parameters.stageId\",\n \"type\": \"Expression\"\n },\n \"type\": \"Int32\"\n }\n }\n },\n \"linkedServiceName\": {\n \"referenceName\": \"SupportDatabase\",\n \"type\": \"LinkedServiceReference\"\n }\n }\n ]\n },\n {\n \"value\": \"Failed\",\n \"activities\": [\n {\n \"name\": \"Pipeline Status Failed\",\n \"description\": \"Updates the current execution table with a pipeline status of failed if the function outcome is failed. Also blocks pipelines in the downstream execution stage.\",\n \"type\": \"SqlServerStoredProcedure\",\n \"dependsOn\": [],\n \"policy\": {\n \"timeout\": \"0.00:10:00\",\n \"retry\": 0,\n \"retryIntervalInSeconds\": 30,\n \"secureOutput\": false,\n \"secureInput\": false\n },\n \"userProperties\": [],\n \"typeProperties\": {\n \"storedProcedureName\": \"[[procfwk].[SetLogPipelineFailed]\",\n \"storedProcedureParameters\": {\n \"ExecutionId\": {\n \"value\": {\n \"value\": \"@pipeline().parameters.executionId\",\n \"type\": \"Expression\"\n },\n \"type\": \"Guid\"\n },\n \"PipelineId\": {\n \"value\": {\n \"value\": \"@pipeline().parameters.pipelineId\",\n \"type\": \"Expression\"\n },\n \"type\": \"Int32\"\n },\n \"RunId\": {\n \"value\": {\n \"value\": \"@variables('WorkerRunId')\",\n \"type\": \"Expression\"\n },\n \"type\": \"Guid\"\n },\n \"StageId\": {\n \"value\": {\n \"value\": \"@pipeline().parameters.stageId\",\n \"type\": \"Expression\"\n },\n \"type\": \"Int32\"\n }\n }\n },\n \"linkedServiceName\": {\n \"referenceName\": \"SupportDatabase\",\n \"type\": \"LinkedServiceReference\"\n }\n },\n {\n \"name\": \"Get Worker Pipeline Error Details\",\n \"description\": \"Get the activity error details for the run ID of the worker pipeline called. Returns an array of all errors.\",\n \"type\": \"AzureFunctionActivity\",\n \"dependsOn\": [],\n \"policy\": {\n \"timeout\": \"0.00:10:00\",\n \"retry\": 0,\n \"retryIntervalInSeconds\": 30,\n \"secureOutput\": false,\n \"secureInput\": true\n },\n \"userProperties\": [],\n \"typeProperties\": {\n \"functionName\": \"GetActivityErrors\",\n \"method\": \"POST\",\n \"headers\": {},\n \"body\": {\n \"value\": \"@concat('\\n{\\n \\\"tenantId\\\": \\\"',variables('WorkerCoreDetails')[0].tenantId,'\\\",\\n \\\"applicationId\\\": \\\"',variables('WorkerCoreDetails')[0].applicationId,'\\\",\\n \\\"authenticationKey\\\": \\\"',variables('WorkerCoreDetails')[0].authenticationKey,'\\\",\\n \\\"subscriptionId\\\": \\\"',variables('WorkerCoreDetails')[0].subscriptionId,'\\\",\\n \\\"resourceGroupName\\\": \\\"',variables('WorkerCoreDetails')[0].resourceGroupName,'\\\",\\n\\t\\\"orchestratorName\\\": \\\"',variables('WorkerCoreDetails')[0].orchestratorName,'\\\",\\n \\\"orchestratorType\\\": \\\"',variables('WorkerCoreDetails')[0].orchestratorType,'\\\",\\n \\\"pipelineName\\\": \\\"',variables('WorkerCoreDetails')[0].pipelineName,'\\\",\\n \\\"runId\\\": \\\"',variables('WorkerRunId'),'\\\"\\n}')\",\n \"type\": \"Expression\"\n }\n },\n \"linkedServiceName\": {\n \"referenceName\": \"FrameworkFunctions\",\n \"type\": \"LinkedServiceReference\"\n }\n },\n {\n \"name\": \"Log Error Details\",\n \"description\": \"Parses pipeline error details and persists them to the metadata database error log table.\",\n \"type\": \"SqlServerStoredProcedure\",\n \"dependsOn\": [\n {\n \"activity\": \"Get Worker Pipeline Error Details\",\n \"dependencyConditions\": [\n \"Succeeded\"\n ]\n }\n ],\n \"policy\": {\n \"timeout\": \"0.00:10:00\",\n \"retry\": 0,\n \"retryIntervalInSeconds\": 30,\n \"secureOutput\": false,\n \"secureInput\": false\n },\n \"userProperties\": [],\n \"typeProperties\": {\n \"storedProcedureName\": \"[[procfwk].[SetErrorLogDetails]\",\n \"storedProcedureParameters\": {\n \"JsonErrorDetails\": {\n \"value\": {\n \"value\": \"@string(activity('Get Worker Pipeline Error Details').output)\",\n \"type\": \"Expression\"\n },\n \"type\": \"String\"\n },\n \"LocalExecutionId\": {\n \"value\": {\n \"value\": \"@pipeline().parameters.executionId\",\n \"type\": \"Expression\"\n },\n \"type\": \"Guid\"\n }\n }\n },\n \"linkedServiceName\": {\n \"referenceName\": \"SupportDatabase\",\n \"type\": \"LinkedServiceReference\"\n }\n }\n ]\n },\n {\n \"value\": \"Cancelled\",\n \"activities\": [\n {\n \"name\": \"Pipeline Status Cancelled\",\n \"description\": \"Updates the current execution table with a pipeline status of cancelled if the function outcome is cancelled.\",\n \"type\": \"SqlServerStoredProcedure\",\n \"dependsOn\": [],\n \"policy\": {\n \"timeout\": \"0.00:10:00\",\n \"retry\": 0,\n \"retryIntervalInSeconds\": 30,\n \"secureOutput\": false,\n \"secureInput\": false\n },\n \"userProperties\": [],\n \"typeProperties\": {\n \"storedProcedureName\": \"[[procfwk].[SetLogPipelineCancelled]\",\n \"storedProcedureParameters\": {\n \"ExecutionId\": {\n \"value\": {\n \"value\": \"@pipeline().parameters.executionId\",\n \"type\": \"Expression\"\n },\n \"type\": \"Guid\"\n },\n \"PipelineId\": {\n \"value\": {\n \"value\": \"@pipeline().parameters.pipelineId\",\n \"type\": \"Expression\"\n },\n \"type\": \"Int32\"\n },\n \"StageId\": {\n \"value\": {\n \"value\": \"@pipeline().parameters.stageId\",\n \"type\": \"Expression\"\n },\n \"type\": \"Int32\"\n }\n }\n },\n \"linkedServiceName\": {\n \"referenceName\": \"SupportDatabase\",\n \"type\": \"LinkedServiceReference\"\n }\n }\n ]\n }\n ],\n \"defaultActivities\": [\n {\n \"name\": \"Pipeline Status Unknown\",\n \"description\": \"Updates the current execution table with a pipeline status of unknown if the function returns an unexpected outcome.\",\n \"type\": \"SqlServerStoredProcedure\",\n \"dependsOn\": [],\n \"policy\": {\n \"timeout\": \"0.00:10:00\",\n \"retry\": 0,\n \"retryIntervalInSeconds\": 30,\n \"secureOutput\": false,\n \"secureInput\": false\n },\n \"userProperties\": [],\n \"typeProperties\": {\n \"storedProcedureName\": \"[[procfwk].[SetLogPipelineUnknown]\",\n \"storedProcedureParameters\": {\n \"ExecutionId\": {\n \"value\": {\n \"value\": \"@pipeline().parameters.executionId\",\n \"type\": \"Expression\"\n },\n \"type\": \"Guid\"\n },\n \"PipelineId\": {\n \"value\": {\n \"value\": \"@pipeline().parameters.pipelineId\",\n \"type\": \"Expression\"\n },\n \"type\": \"Int32\"\n },\n \"StageId\": {\n \"value\": {\n \"value\": \"@pipeline().parameters.stageId\",\n \"type\": \"Expression\"\n },\n \"type\": \"Int32\"\n }\n }\n },\n \"linkedServiceName\": {\n \"referenceName\": \"SupportDatabase\",\n \"type\": \"LinkedServiceReference\"\n }\n }\n ]\n }\n },\n {\n \"name\": \"Get Wait Duration\",\n \"description\": \"Return wait duration in seconds from database properties table to be used during each Until iteration when the Worker pipeline is still running.\",\n \"type\": \"Lookup\",\n \"dependsOn\": [],\n \"policy\": {\n \"timeout\": \"0.00:10:00\",\n \"retry\": 0,\n \"retryIntervalInSeconds\": 30,\n \"secureOutput\": false,\n \"secureInput\": false\n },\n \"userProperties\": [],\n \"typeProperties\": {\n \"source\": {\n \"type\": \"AzureSqlSource\",\n \"sqlReaderStoredProcedureName\": \"[[procfwk].[GetPropertyValue]\",\n \"storedProcedureParameters\": {\n \"PropertyName\": {\n \"type\": \"String\",\n \"value\": \"PipelineStatusCheckDuration\"\n }\n },\n \"queryTimeout\": \"02:00:00\",\n \"partitionOption\": \"None\"\n },\n \"dataset\": {\n \"referenceName\": \"GetSetMetadata\",\n \"type\": \"DatasetReference\",\n \"parameters\": {}\n }\n }\n },\n {\n \"name\": \"Set Run Id\",\n \"description\": \"Set local variable from activity output once for value reuse in downstream activities.\",\n \"type\": \"SetVariable\",\n \"dependsOn\": [\n {\n \"activity\": \"Execute Worker Pipeline\",\n \"dependencyConditions\": [\n \"Succeeded\"\n ]\n }\n ],\n \"userProperties\": [],\n \"typeProperties\": {\n \"variableName\": \"WorkerRunId\",\n \"value\": {\n \"value\": \"@activity('Execute Worker Pipeline').output.RunId\",\n \"type\": \"Expression\"\n }\n }\n },\n {\n \"name\": \"Validate Pipeline\",\n \"description\": \"Query the target data factory and establish if the provided worker pipeline name is valid.\",\n \"type\": \"AzureFunctionActivity\",\n \"dependsOn\": [\n {\n \"activity\": \"Log Pipeline Validating\",\n \"dependencyConditions\": [\n \"Succeeded\"\n ]\n },\n {\n \"activity\": \"Capture Worker Core Details as an Array\",\n \"dependencyConditions\": [\n \"Succeeded\"\n ]\n }\n ],\n \"policy\": {\n \"timeout\": \"0.00:10:00\",\n \"retry\": 0,\n \"retryIntervalInSeconds\": 30,\n \"secureOutput\": false,\n \"secureInput\": true\n },\n \"userProperties\": [],\n \"typeProperties\": {\n \"functionName\": \"ValidatePipeline\",\n \"method\": \"POST\",\n \"headers\": {},\n \"body\": {\n \"value\": \"@concat('\\n{\\n \\\"tenantId\\\": \\\"',variables('WorkerCoreDetails')[0].tenantId,'\\\",\\n \\\"applicationId\\\": \\\"',variables('WorkerCoreDetails')[0].applicationId,'\\\",\\n \\\"authenticationKey\\\": \\\"',variables('WorkerCoreDetails')[0].authenticationKey,'\\\",\\n \\\"subscriptionId\\\": \\\"',variables('WorkerCoreDetails')[0].subscriptionId,'\\\",\\n \\\"resourceGroupName\\\": \\\"',variables('WorkerCoreDetails')[0].resourceGroupName,'\\\",\\n\\t\\\"orchestratorName\\\": \\\"',variables('WorkerCoreDetails')[0].orchestratorName,'\\\",\\n \\\"orchestratorType\\\": \\\"',variables('WorkerCoreDetails')[0].orchestratorType,'\\\",\\n \\\"pipelineName\\\": \\\"',variables('WorkerCoreDetails')[0].pipelineName,'\\\"\\n}')\",\n \"type\": \"Expression\"\n }\n },\n \"linkedServiceName\": {\n \"referenceName\": \"FrameworkFunctions\",\n \"type\": \"LinkedServiceReference\"\n }\n },\n {\n \"name\": \"Is Target Worker Validate\",\n \"description\": \"True = the worker pipeline name is valid.\\nFalse = the worker pipeline name is invalid. Raise an exception.\",\n \"type\": \"IfCondition\",\n \"dependsOn\": [\n {\n \"activity\": \"Validate Pipeline\",\n \"dependencyConditions\": [\n \"Succeeded\"\n ]\n }\n ],\n \"userProperties\": [],\n \"typeProperties\": {\n \"expression\": {\n \"value\": \"@bool(activity('Validate Pipeline').output.PipelineExists)\",\n \"type\": \"Expression\"\n },\n \"ifFalseActivities\": [\n {\n \"name\": \"Throw Exception - Invalid Infant\",\n \"description\": \"Throw an exception with details about the invalid worker pipeline name.\",\n \"type\": \"ExecutePipeline\",\n \"dependsOn\": [],\n \"userProperties\": [],\n \"typeProperties\": {\n \"pipeline\": {\n \"referenceName\": \"Throw Exception\",\n \"type\": \"PipelineReference\"\n },\n \"waitOnCompletion\": true,\n \"parameters\": {\n \"Message\": {\n \"value\": \"@concat('Worker pipeline [',variables('WorkerCoreDetails')[0].pipelineName,'] is not valid in target Orchestrator [',variables('WorkerCoreDetails')[0].orchestratorName,']')\",\n \"type\": \"Expression\"\n }\n }\n }\n },\n {\n \"name\": \"Update Execution With Invalid Worker\",\n \"description\": \"Update the current execution table with an informed status for the worker pipeline that couldn't be executed.\",\n \"type\": \"SqlServerStoredProcedure\",\n \"dependsOn\": [],\n \"policy\": {\n \"timeout\": \"7.00:00:00\",\n \"retry\": 0,\n \"retryIntervalInSeconds\": 30,\n \"secureOutput\": false,\n \"secureInput\": false\n },\n \"userProperties\": [],\n \"typeProperties\": {\n \"storedProcedureName\": \"[[procfwk].[SetLogActivityFailed]\",\n \"storedProcedureParameters\": {\n \"CallingActivity\": {\n \"value\": \"InvalidPipelineName\",\n \"type\": \"String\"\n },\n \"ExecutionId\": {\n \"value\": {\n \"value\": \"@pipeline().parameters.ExecutionId\",\n \"type\": \"Expression\"\n },\n \"type\": \"Guid\"\n },\n \"PipelineId\": {\n \"value\": {\n \"value\": \"@pipeline().parameters.pipelineId\",\n \"type\": \"Expression\"\n },\n \"type\": \"Int32\"\n },\n \"StageId\": {\n \"value\": {\n \"value\": \"@pipeline().parameters.StageId\",\n \"type\": \"Expression\"\n },\n \"type\": \"Int32\"\n }\n }\n },\n \"linkedServiceName\": {\n \"referenceName\": \"SupportDatabase\",\n \"type\": \"LinkedServiceReference\"\n }\n }\n ]\n }\n },\n {\n \"name\": \"Log Validate Function Activity Failure\",\n \"description\": \"Handle true failures from calling out to the Azure Function and update the current execution table accordingly so a restart can occur.\",\n \"type\": \"SqlServerStoredProcedure\",\n \"dependsOn\": [\n {\n \"activity\": \"Validate Pipeline\",\n \"dependencyConditions\": [\n \"Failed\"\n ]\n }\n ],\n \"policy\": {\n \"timeout\": \"0.00:10:00\",\n \"retry\": 0,\n \"retryIntervalInSeconds\": 30,\n \"secureOutput\": false,\n \"secureInput\": false\n },\n \"userProperties\": [],\n \"typeProperties\": {\n \"storedProcedureName\": \"[[procfwk].[SetLogActivityFailed]\",\n \"storedProcedureParameters\": {\n \"ExecutionId\": {\n \"value\": {\n \"value\": \"@pipeline().parameters.ExecutionId\",\n \"type\": \"Expression\"\n },\n \"type\": \"Guid\"\n },\n \"PipelineId\": {\n \"value\": {\n \"value\": \"@pipeline().parameters.pipelineId\",\n \"type\": \"Expression\"\n },\n \"type\": \"Int32\"\n },\n \"StageId\": {\n \"value\": {\n \"value\": \"@pipeline().parameters.StageId\",\n \"type\": \"Expression\"\n },\n \"type\": \"Int32\"\n },\n \"CallingActivity\": {\n \"value\": \"ValidatePipeline\",\n \"type\": \"String\"\n }\n }\n },\n \"linkedServiceName\": {\n \"referenceName\": \"SupportDatabase\",\n \"type\": \"LinkedServiceReference\"\n }\n },\n {\n \"name\": \"Log Pipeline Validating\",\n \"description\": \"Sets the current pipeline with a status of validating within the current execution database table.\",\n \"type\": \"SqlServerStoredProcedure\",\n \"dependsOn\": [],\n \"policy\": {\n \"timeout\": \"0.00:10:00\",\n \"retry\": 0,\n \"retryIntervalInSeconds\": 30,\n \"secureOutput\": false,\n \"secureInput\": false\n },\n \"userProperties\": [],\n \"typeProperties\": {\n \"storedProcedureName\": \"[[procfwk].[SetLogPipelineValidating]\",\n \"storedProcedureParameters\": {\n \"ExecutionId\": {\n \"value\": {\n \"value\": \"@pipeline().parameters.ExecutionId\",\n \"type\": \"Expression\"\n },\n \"type\": \"Guid\"\n },\n \"PipelineId\": {\n \"value\": {\n \"value\": \"@pipeline().parameters.pipelineId\",\n \"type\": \"Expression\"\n },\n \"type\": \"Int32\"\n },\n \"StageId\": {\n \"value\": {\n \"value\": \"@pipeline().parameters.StageId\",\n \"type\": \"Expression\"\n },\n \"type\": \"Int32\"\n }\n }\n },\n \"linkedServiceName\": {\n \"referenceName\": \"SupportDatabase\",\n \"type\": \"LinkedServiceReference\"\n }\n },\n {\n \"name\": \"Get Worker Core Details\",\n \"description\": \"Return worker pipeline information for metadata database. Including target data factory, pipeline name and resource group. Return the SPN ID and Secret for the worker pipeline being executed. Called at this level as each pipeline can have a different SPN.\",\n \"type\": \"Lookup\",\n \"dependsOn\": [],\n \"policy\": {\n \"timeout\": \"0.00:10:00\",\n \"retry\": 0,\n \"retryIntervalInSeconds\": 30,\n \"secureOutput\": true,\n \"secureInput\": false\n },\n \"userProperties\": [],\n \"typeProperties\": {\n \"source\": {\n \"type\": \"AzureSqlSource\",\n \"sqlReaderStoredProcedureName\": \"[[procfwk].[GetWorkerDetailsWrapper]\",\n \"storedProcedureParameters\": {\n \"ExecutionId\": {\n \"type\": \"Guid\",\n \"value\": {\n \"value\": \"@pipeline().parameters.executionId\",\n \"type\": \"Expression\"\n }\n },\n \"PipelineId\": {\n \"type\": \"Int32\",\n \"value\": {\n \"value\": \"@pipeline().parameters.pipelineId\",\n \"type\": \"Expression\"\n }\n },\n \"StageId\": {\n \"type\": \"Int32\",\n \"value\": {\n \"value\": \"@pipeline().parameters.stageId\",\n \"type\": \"Expression\"\n }\n }\n },\n \"queryTimeout\": \"02:00:00\",\n \"partitionOption\": \"None\"\n },\n \"dataset\": {\n \"referenceName\": \"GetSetMetadata\",\n \"type\": \"DatasetReference\",\n \"parameters\": {}\n }\n }\n },\n {\n \"name\": \"Capture Worker Core Details as an Array\",\n \"description\": \"Add all worker pipeline details to a local variable array that can be accessed by each function call requiring the values.\",\n \"type\": \"SetVariable\",\n \"dependsOn\": [\n {\n \"activity\": \"Get Worker Core Details\",\n \"dependencyConditions\": [\n \"Succeeded\"\n ]\n }\n ],\n \"userProperties\": [],\n \"typeProperties\": {\n \"variableName\": \"WorkerCoreDetails\",\n \"value\": {\n \"value\": \"@array(activity('Get Worker Core Details').output.firstRow)\",\n \"type\": \"Expression\"\n }\n }\n }\n ],\n \"parameters\": {\n \"executionId\": {\n \"type\": \"string\"\n },\n \"stageId\": {\n \"type\": \"int\"\n },\n \"pipelineId\": {\n \"type\": \"int\"\n }\n },\n \"variables\": {\n \"WorkerPipelineState\": {\n \"type\": \"Boolean\"\n },\n \"WorkerRunId\": {\n \"type\": \"String\"\n },\n \"WorkerCoreDetails\": {\n \"type\": \"Array\"\n }\n },\n \"folder\": {\n \"name\": \"_ProcFwk\"\n },\n \"annotations\": [\n \"procfwk\",\n \"Infant\"\n ]\n },\n \"dependsOn\": [\n \"[concat(variables('workspaceId'), '/linkedServices/FrameworkFunctions')]\",\n \"[concat(variables('workspaceId'), '/datasets/GetSetMetadata')]\",\n \"[concat(variables('workspaceId'), '/linkedServices/SupportDatabase')]\",\n \"[concat(variables('workspaceId'), '/pipelines/Email Sender')]\",\n \"[concat(variables('workspaceId'), '/pipelines/Throw Exception')]\"\n ]\n },\n {\n \"name\": \"[concat(parameters('workspaceName'), '/Check For Running Pipeline')]\",\n \"type\": \"Microsoft.Synapse/workspaces/pipelines\",\n \"apiVersion\": \"2019-06-01-preview\",\n \"properties\": {\n \"description\": \"For a given pipeline and optional batch name establish if a pipeline run is already in progress. Throw an exception if it it.\",\n \"activities\": [\n {\n \"name\": \"Filter Running Pipelines\",\n \"description\": \"Filter the pipeline runs results for pipelines that exclude the current triggered run and that are currently running (in progress or queued).\",\n \"type\": \"Filter\",\n \"dependsOn\": [\n {\n \"activity\": \"Switch For Orchestrator Type\",\n \"dependencyConditions\": [\n \"Succeeded\"\n ]\n }\n ],\n \"userProperties\": [],\n \"typeProperties\": {\n \"items\": {\n \"value\": \"@variables('PipelineRuns')\",\n \"type\": \"Expression\"\n },\n \"condition\": {\n \"value\": \"@and(not(equals(item().runId,pipeline().parameters.ThisRunId)),or(equals(item().status,'InProgress'),equals(item().status,'Queued')))\",\n \"type\": \"Expression\"\n }\n }\n },\n {\n \"name\": \"Get Framework Orchestrator Details\",\n \"description\": \"Using the metadata orchestrators return details about the resource running the framework pipelines.\",\n \"type\": \"Lookup\",\n \"dependsOn\": [],\n \"policy\": {\n \"timeout\": \"0.00:10:00\",\n \"retry\": 0,\n \"retryIntervalInSeconds\": 30,\n \"secureOutput\": false,\n \"secureInput\": false\n },\n \"userProperties\": [],\n \"typeProperties\": {\n \"source\": {\n \"type\": \"AzureSqlSource\",\n \"sqlReaderStoredProcedureName\": \"[[procfwk].[GetFrameworkOrchestratorDetails]\",\n \"storedProcedureParameters\": {\n \"CallingOrchestratorName\": {\n \"type\": \"String\",\n \"value\": {\n \"value\": \"@pipeline().DataFactory\",\n \"type\": \"Expression\"\n }\n }\n },\n \"queryTimeout\": \"02:00:00\",\n \"partitionOption\": \"None\"\n },\n \"dataset\": {\n \"referenceName\": \"GetSetMetadata\",\n \"type\": \"DatasetReference\",\n \"parameters\": {}\n }\n }\n },\n {\n \"name\": \"Get Query Run Days Value\",\n \"description\": \"Using the metadata properties table return the run days value to provide the API request with a date range for pipeline executions.\",\n \"type\": \"Lookup\",\n \"dependsOn\": [],\n \"policy\": {\n \"timeout\": \"0.00:10:00\",\n \"retry\": 0,\n \"retryIntervalInSeconds\": 30,\n \"secureOutput\": false,\n \"secureInput\": false\n },\n \"userProperties\": [],\n \"typeProperties\": {\n \"source\": {\n \"type\": \"AzureSqlSource\",\n \"sqlReaderStoredProcedureName\": \"[[procfwk].[GetPropertyValue]\",\n \"storedProcedureParameters\": {\n \"PropertyName\": {\n \"type\": \"String\",\n \"value\": \"PreviousPipelineRunsQueryRange\"\n }\n },\n \"queryTimeout\": \"02:00:00\",\n \"partitionOption\": \"None\"\n },\n \"dataset\": {\n \"referenceName\": \"GetSetMetadata\",\n \"type\": \"DatasetReference\",\n \"parameters\": {}\n }\n }\n },\n {\n \"name\": \"If Pipeline Is Running\",\n \"description\": \"If the running pipeline count is greater than or equal to one.\\nTrue = raise an exception.\",\n \"type\": \"IfCondition\",\n \"dependsOn\": [\n {\n \"activity\": \"If Using Batch Executions\",\n \"dependencyConditions\": [\n \"Succeeded\"\n ]\n }\n ],\n \"userProperties\": [],\n \"typeProperties\": {\n \"expression\": {\n \"value\": \"@greaterOrEquals(int(variables('RunCount')),1)\",\n \"type\": \"Expression\"\n },\n \"ifTrueActivities\": [\n {\n \"name\": \"Throw Exception - Pipeline Running\",\n \"description\": \"Using the utils pipeline raise an exception to stop the new trigger while a run is already in progress.\",\n \"type\": \"ExecutePipeline\",\n \"dependsOn\": [],\n \"userProperties\": [],\n \"typeProperties\": {\n \"pipeline\": {\n \"referenceName\": \"Throw Exception\",\n \"type\": \"PipelineReference\"\n },\n \"waitOnCompletion\": true,\n \"parameters\": {\n \"Message\": {\n \"value\": \"@concat('Provided pipeline name (',pipeline().parameters.PipelineName,') still has a run in progress or queued given the query range parameters set in the properties table.')\",\n \"type\": \"Expression\"\n }\n }\n }\n }\n ]\n }\n },\n {\n \"name\": \"Get Execution Batch Status\",\n \"description\": \"Using the metadata properties table return the flag to indicate if batch execution setting are enabled or disabled.\",\n \"type\": \"Lookup\",\n \"dependsOn\": [],\n \"policy\": {\n \"timeout\": \"0.00:10:00\",\n \"retry\": 0,\n \"retryIntervalInSeconds\": 30,\n \"secureOutput\": false,\n \"secureInput\": false\n },\n \"userProperties\": [],\n \"typeProperties\": {\n \"source\": {\n \"type\": \"AzureSqlSource\",\n \"sqlReaderStoredProcedureName\": \"[[procfwk].[GetPropertyValue]\",\n \"storedProcedureParameters\": {\n \"PropertyName\": {\n \"type\": \"String\",\n \"value\": \"UseExecutionBatches\"\n }\n },\n \"queryTimeout\": \"02:00:00\",\n \"partitionOption\": \"None\"\n },\n \"dataset\": {\n \"referenceName\": \"GetSetMetadata\",\n \"type\": \"DatasetReference\",\n \"parameters\": {}\n }\n }\n },\n {\n \"name\": \"If Using Batch Executions\",\n \"description\": \"True = batch executions are enabled.\\nFalse = batch execution are disabled.\",\n \"type\": \"IfCondition\",\n \"dependsOn\": [\n {\n \"activity\": \"Get Execution Batch Status\",\n \"dependencyConditions\": [\n \"Succeeded\"\n ]\n },\n {\n \"activity\": \"Filter Running Pipelines\",\n \"dependencyConditions\": [\n \"Succeeded\"\n ]\n }\n ],\n \"userProperties\": [],\n \"typeProperties\": {\n \"expression\": {\n \"value\": \"@equals(activity('Get Execution Batch Status').output.firstRow.PropertyValue,string(1))\",\n \"type\": \"Expression\"\n },\n \"ifFalseActivities\": [\n {\n \"name\": \"Set Run Count Without Batch\",\n \"description\": \"Set the pipelines running count variable to be tested later.\",\n \"type\": \"SetVariable\",\n \"dependsOn\": [],\n \"userProperties\": [],\n \"typeProperties\": {\n \"variableName\": \"RunCount\",\n \"value\": {\n \"value\": \"@string(activity('Filter Running Pipelines').output.FilteredItemsCount)\",\n \"type\": \"Expression\"\n }\n }\n }\n ],\n \"ifTrueActivities\": [\n {\n \"name\": \"Filter for Batch Name\",\n \"description\": \"Further filter the return pipeline runs for any running pipelines with the same batch name value.\",\n \"type\": \"Filter\",\n \"dependsOn\": [],\n \"userProperties\": [],\n \"typeProperties\": {\n \"items\": {\n \"value\": \"@activity('Filter Running Pipelines').output.value\",\n \"type\": \"Expression\"\n },\n \"condition\": {\n \"value\": \"@equals(item().parameters.BatchName,pipeline().parameters.BatchName)\",\n \"type\": \"Expression\"\n }\n }\n },\n {\n \"name\": \"Set Run Count for Batch\",\n \"description\": \"Set the resulting pipeline running count variable to be tested later.\",\n \"type\": \"SetVariable\",\n \"dependsOn\": [\n {\n \"activity\": \"Filter for Batch Name\",\n \"dependencyConditions\": [\n \"Succeeded\"\n ]\n }\n ],\n \"userProperties\": [],\n \"typeProperties\": {\n \"variableName\": \"RunCount\",\n \"value\": {\n \"value\": \"@string(activity('Filter for Batch Name').output.FilteredItemsCount)\",\n \"type\": \"Expression\"\n }\n }\n }\n ]\n }\n },\n {\n \"name\": \"Set Subscription Id\",\n \"description\": \"Set the subscription Id value to a local variable for use in various downstream activities.\",\n \"type\": \"SetVariable\",\n \"dependsOn\": [\n {\n \"activity\": \"Get Framework Orchestrator Details\",\n \"dependencyConditions\": [\n \"Succeeded\"\n ]\n }\n ],\n \"userProperties\": [],\n \"typeProperties\": {\n \"variableName\": \"SubscriptionId\",\n \"value\": {\n \"value\": \"@activity('Get Framework Orchestrator Details').output.firstRow.SubscriptionId\",\n \"type\": \"Expression\"\n }\n }\n },\n {\n \"name\": \"Set Resource Group Name\",\n \"description\": \"Set the resource group name value to a local variable for use in various downstream activities.\",\n \"type\": \"SetVariable\",\n \"dependsOn\": [\n {\n \"activity\": \"Get Framework Orchestrator Details\",\n \"dependencyConditions\": [\n \"Succeeded\"\n ]\n }\n ],\n \"userProperties\": [],\n \"typeProperties\": {\n \"variableName\": \"ResourceGroupName\",\n \"value\": {\n \"value\": \"@activity('Get Framework Orchestrator Details').output.firstRow.ResourceGroupName\",\n \"type\": \"Expression\"\n }\n }\n },\n {\n \"name\": \"Set Orchestrator Type\",\n \"description\": \"Set the orchestrator type value to a local variable for use in various downstream activities.\",\n \"type\": \"SetVariable\",\n \"dependsOn\": [\n {\n \"activity\": \"Get Framework Orchestrator Details\",\n \"dependencyConditions\": [\n \"Succeeded\"\n ]\n }\n ],\n \"userProperties\": [],\n \"typeProperties\": {\n \"variableName\": \"OrchestratorType\",\n \"value\": {\n \"value\": \"@toUpper(activity('Get Framework Orchestrator Details').output.firstRow.OrchestratorType)\",\n \"type\": \"Expression\"\n }\n }\n },\n {\n \"name\": \"Switch For Orchestrator Type\",\n \"description\": \"Switch and handle requests for both Azure Data Factory (ADF) and Azure Synapse Analytics (SYN).\",\n \"type\": \"Switch\",\n \"dependsOn\": [\n {\n \"activity\": \"Set Orchestrator Type\",\n \"dependencyConditions\": [\n \"Succeeded\"\n ]\n },\n {\n \"activity\": \"Set Query Run Days\",\n \"dependencyConditions\": [\n \"Succeeded\"\n ]\n },\n {\n \"activity\": \"Set Resource Group Name\",\n \"dependencyConditions\": [\n \"Succeeded\"\n ]\n },\n {\n \"activity\": \"Set Subscription Id\",\n \"dependencyConditions\": [\n \"Succeeded\"\n ]\n }\n ],\n \"userProperties\": [],\n \"typeProperties\": {\n \"on\": {\n \"value\": \"@variables('OrchestratorType')\",\n \"type\": \"Expression\"\n },\n \"cases\": [\n {\n \"value\": \"ADF\",\n \"activities\": [\n {\n \"name\": \"Check for Valid ADF Pipeline Name\",\n \"description\": \"Use the Azure Management API to return and establish if the framework pipeline exists in the target Data Factory instance, including being deployed.\",\n \"type\": \"WebActivity\",\n \"dependsOn\": [],\n \"policy\": {\n \"timeout\": \"7.00:00:00\",\n \"retry\": 0,\n \"retryIntervalInSeconds\": 30,\n \"secureOutput\": false,\n \"secureInput\": false\n },\n \"userProperties\": [],\n \"typeProperties\": {\n \"url\": {\n \"value\": \"https://management.azure.com/subscriptions/@{variables('SubscriptionId')}/resourceGroups/@{variables('ResourceGroupName')}/providers/Microsoft.DataFactory/factories/@{pipeline().DataFactory}/pipelines/@{pipeline().parameters.PipelineName}?api-version=2018-06-01\",\n \"type\": \"Expression\"\n },\n \"method\": \"GET\",\n \"headers\": {},\n \"authentication\": {\n \"type\": \"MSI\",\n \"resource\": \"https://management.core.windows.net/\"\n }\n }\n },\n {\n \"name\": \"Get ADF Pipeline Runs\",\n \"description\": \"Use the Azure Management API to return a list of data factory pipeline runs within the given time window.\",\n \"type\": \"WebActivity\",\n \"dependsOn\": [\n {\n \"activity\": \"Check for Valid ADF Pipeline Name\",\n \"dependencyConditions\": [\n \"Succeeded\"\n ]\n }\n ],\n \"policy\": {\n \"timeout\": \"7.00:00:00\",\n \"retry\": 0,\n \"retryIntervalInSeconds\": 30,\n \"secureOutput\": false,\n \"secureInput\": false\n },\n \"userProperties\": [],\n \"typeProperties\": {\n \"url\": {\n \"value\": \"https://management.azure.com/subscriptions/@{variables('SubscriptionId')}/resourceGroups/@{variables('ResourceGroupName')}/providers/Microsoft.DataFactory/factories/@{pipeline().DataFactory}/queryPipelineRuns?api-version=2018-06-01\",\n \"type\": \"Expression\"\n },\n \"method\": \"POST\",\n \"headers\": {},\n \"body\": {\n \"value\": \"{\\n \\\"lastUpdatedAfter\\\": \\\"@{adddays(utcnow(),int(variables('QueryRunDays')))}\\\",\\n \\\"lastUpdatedBefore\\\": \\\"@{utcnow()}\\\",\\n \\\"filters\\\": [\\n {\\n \\\"operand\\\": \\\"PipelineName\\\",\\n \\\"operator\\\": \\\"Equals\\\",\\n \\\"values\\\": [\\n \\\"@{pipeline().parameters.PipelineName}\\\"\\n ]\\n }\\n ]\\n}\",\n \"type\": \"Expression\"\n },\n \"authentication\": {\n \"type\": \"MSI\",\n \"resource\": \"https://management.core.windows.net/\"\n }\n }\n },\n {\n \"name\": \"Set ADF Runs Output\",\n \"description\": \"Set output to local array for use in downstream filtering and pipeline checks. Use the same array output for both switch cases.\",\n \"type\": \"SetVariable\",\n \"dependsOn\": [\n {\n \"activity\": \"Get ADF Pipeline Runs\",\n \"dependencyConditions\": [\n \"Succeeded\"\n ]\n }\n ],\n \"userProperties\": [],\n \"typeProperties\": {\n \"variableName\": \"PipelineRuns\",\n \"value\": {\n \"value\": \"@activity('Get ADF Pipeline Runs').output.value\",\n \"type\": \"Expression\"\n }\n }\n }\n ]\n },\n {\n \"value\": \"SYN\",\n \"activities\": [\n {\n \"name\": \"Check for Valid SYN Pipeline Name\",\n \"description\": \"Use the Azure Management API to return and establish if the framework pipeline exists in the target Synapse instance, including being deployed.\\n\\nSee: https://docs.microsoft.com/en-us/rest/api/synapse/data-plane/pipeline/getpipeline\",\n \"type\": \"WebActivity\",\n \"dependsOn\": [],\n \"policy\": {\n \"timeout\": \"7.00:00:00\",\n \"retry\": 0,\n \"retryIntervalInSeconds\": 30,\n \"secureOutput\": false,\n \"secureInput\": false\n },\n \"userProperties\": [],\n \"typeProperties\": {\n \"url\": {\n \"value\": \"https://@{pipeline().DataFactory}.dev.azuresynapse.net/pipelines/@{pipeline().parameters.PipelineName}?api-version=2019-06-01-preview\",\n \"type\": \"Expression\"\n },\n \"method\": \"GET\",\n \"headers\": {},\n \"authentication\": {\n \"type\": \"MSI\",\n \"resource\": \"https://management.core.windows.net/\"\n }\n }\n },\n {\n \"name\": \"Get SYN Pipeline Runs\",\n \"description\": \"Use the Azure Management API to return a list of synapse pipeline runs within the given time window.\\n\\nSee: https://docs.microsoft.com/en-us/rest/api/synapse/data-plane/pipelinerun/querypipelinerunsbyworkspace\",\n \"type\": \"WebActivity\",\n \"dependsOn\": [\n {\n \"activity\": \"Check for Valid SYN Pipeline Name\",\n \"dependencyConditions\": [\n \"Succeeded\"\n ]\n }\n ],\n \"policy\": {\n \"timeout\": \"7.00:00:00\",\n \"retry\": 0,\n \"retryIntervalInSeconds\": 30,\n \"secureOutput\": false,\n \"secureInput\": false\n },\n \"userProperties\": [],\n \"typeProperties\": {\n \"url\": {\n \"value\": \"https://@{pipeline().DataFactory}.dev.azuresynapse.net/queryPipelineRuns?api-version=2019-06-01-preview\",\n \"type\": \"Expression\"\n },\n \"method\": \"POST\",\n \"headers\": {},\n \"body\": {\n \"value\": \"{\\n \\\"lastUpdatedAfter\\\": \\\"@{adddays(utcnow(),int(variables('QueryRunDays')))}\\\",\\n \\\"lastUpdatedBefore\\\": \\\"@{utcnow()}\\\",\\n \\\"filters\\\": [\\n {\\n \\\"operand\\\": \\\"PipelineName\\\",\\n \\\"operator\\\": \\\"Equals\\\",\\n \\\"values\\\": [\\n \\\"@{pipeline().parameters.PipelineName}\\\"\\n ]\\n }\\n ]\\n}\",\n \"type\": \"Expression\"\n },\n \"authentication\": {\n \"type\": \"MSI\",\n \"resource\": \"https://management.core.windows.net/\"\n }\n }\n },\n {\n \"name\": \"Set SYN Runs Output\",\n \"description\": \"Set output to local array for use in downstream filtering and pipeline checks. Use the same array output for both switch cases.\",\n \"type\": \"SetVariable\",\n \"dependsOn\": [\n {\n \"activity\": \"Get SYN Pipeline Runs\",\n \"dependencyConditions\": [\n \"Succeeded\"\n ]\n }\n ],\n \"userProperties\": [],\n \"typeProperties\": {\n \"variableName\": \"PipelineRuns\",\n \"value\": {\n \"value\": \"@activity('Get SYN Pipeline Runs').output.value\",\n \"type\": \"Expression\"\n }\n }\n }\n ]\n }\n ],\n \"defaultActivities\": [\n {\n \"name\": \"Throw Exception Invalid Orchestrator Type\",\n \"description\": \"Throw exception if switch cases are not met.\",\n \"type\": \"ExecutePipeline\",\n \"dependsOn\": [],\n \"userProperties\": [],\n \"typeProperties\": {\n \"pipeline\": {\n \"referenceName\": \"Throw Exception\",\n \"type\": \"PipelineReference\"\n },\n \"waitOnCompletion\": true,\n \"parameters\": {\n \"Message\": \"Invalid orchestrator type provided. Unable to check pipeline running state.\"\n }\n }\n }\n ]\n }\n },\n {\n \"name\": \"Set Query Run Days\",\n \"description\": \"Set the query run days value to a local variable for use in various downstream activities.\",\n \"type\": \"SetVariable\",\n \"dependsOn\": [\n {\n \"activity\": \"Get Query Run Days Value\",\n \"dependencyConditions\": [\n \"Succeeded\"\n ]\n }\n ],\n \"userProperties\": [],\n \"typeProperties\": {\n \"variableName\": \"QueryRunDays\",\n \"value\": {\n \"value\": \"@activity('Get Query Run Days Value').output.firstRow.PropertyValue\",\n \"type\": \"Expression\"\n }\n }\n }\n ],\n \"parameters\": {\n \"BatchName\": {\n \"type\": \"string\",\n \"defaultValue\": \"NotUsed\"\n },\n \"PipelineName\": {\n \"type\": \"string\"\n },\n \"ThisRunId\": {\n \"type\": \"string\"\n }\n },\n \"variables\": {\n \"SubscriptionId\": {\n \"type\": \"String\"\n },\n \"RunCount\": {\n \"type\": \"String\"\n },\n \"ResourceGroupName\": {\n \"type\": \"String\"\n },\n \"OrchestratorType\": {\n \"type\": \"String\"\n },\n \"QueryRunDays\": {\n \"type\": \"String\"\n },\n \"PipelineRuns\": {\n \"type\": \"Array\"\n }\n },\n \"folder\": {\n \"name\": \"_ProcFwk/_ProcFwkUtils\"\n },\n \"annotations\": [\n \"procfwk\",\n \"Utils\"\n ]\n },\n \"dependsOn\": [\n \"[concat(variables('workspaceId'), '/datasets/GetSetMetadata')]\",\n \"[concat(variables('workspaceId'), '/pipelines/Throw Exception')]\"\n ]\n },\n {\n \"name\": \"[concat(parameters('workspaceName'), '/Email Sender')]\",\n \"type\": \"Microsoft.Synapse/workspaces/pipelines\",\n \"apiVersion\": \"2019-06-01-preview\",\n \"properties\": {\n \"description\": \"Provide a simple abstract over the send email function with request body item exposed as pipeline parameters.\",\n \"activities\": [\n {\n \"name\": \"Send Email\",\n \"description\": \"Use an Azure Function to perform an SMTP client email send operation.\",\n \"type\": \"AzureFunctionActivity\",\n \"dependsOn\": [],\n \"policy\": {\n \"timeout\": \"0.00:10:00\",\n \"retry\": 0,\n \"retryIntervalInSeconds\": 30,\n \"secureOutput\": false,\n \"secureInput\": false\n },\n \"userProperties\": [],\n \"typeProperties\": {\n \"functionName\": \"SendEmail\",\n \"method\": \"POST\",\n \"headers\": {},\n \"body\": {\n \"value\": \"{\\n\\\"emailRecipients\\\": \\\"@{pipeline().parameters.Recipients}\\\",\\n\\\"emailCcRecipients\\\": \\\"@{pipeline().parameters.CcRecipients}\\\",\\n\\\"emailBccRecipients\\\": \\\"@{pipeline().parameters.BccRecipients}\\\",\\n\\\"emailSubject\\\": \\\"@{pipeline().parameters.Subject}\\\",\\n\\\"emailBody\\\": \\\"@{pipeline().parameters.Body}\\\",\\n\\\"emailImportance\\\": \\\"@{pipeline().parameters.Importance}\\\"\\n}\",\n \"type\": \"Expression\"\n }\n },\n \"linkedServiceName\": {\n \"referenceName\": \"FrameworkFunctions\",\n \"type\": \"LinkedServiceReference\"\n }\n }\n ],\n \"parameters\": {\n \"Recipients\": {\n \"type\": \"string\"\n },\n \"CcRecipients\": {\n \"type\": \"string\"\n },\n \"BccRecipients\": {\n \"type\": \"string\"\n },\n \"Subject\": {\n \"type\": \"string\"\n },\n \"Body\": {\n \"type\": \"string\"\n },\n \"Importance\": {\n \"type\": \"string\"\n }\n },\n \"folder\": {\n \"name\": \"_ProcFwk/_ProcFwkUtils\"\n },\n \"annotations\": [\n \"procfwk\",\n \"Utils\"\n ]\n },\n \"dependsOn\": [\n \"[concat(variables('workspaceId'), '/linkedServices/FrameworkFunctions')]\"\n ]\n },\n {\n \"name\": \"[concat(parameters('workspaceName'), '/Intentional Error')]\",\n \"type\": \"Microsoft.Synapse/workspaces/pipelines\",\n \"apiVersion\": \"2019-06-01-preview\",\n \"properties\": {\n \"description\": \"Used just so the procfwk has something to call during development.\",\n \"activities\": [\n {\n \"name\": \"Wait1\",\n \"description\": \"Framework development worker simulator.\",\n \"type\": \"Wait\",\n \"dependsOn\": [],\n \"userProperties\": [],\n \"typeProperties\": {\n \"waitTimeInSeconds\": {\n \"value\": \"@pipeline().parameters.WaitTime\",\n \"type\": \"Expression\"\n }\n }\n },\n {\n \"name\": \"Raise Errors or Not\",\n \"description\": \"Framework development worker simulator.\",\n \"type\": \"IfCondition\",\n \"dependsOn\": [\n {\n \"activity\": \"Wait1\",\n \"dependencyConditions\": [\n \"Succeeded\"\n ]\n }\n ],\n \"userProperties\": [],\n \"typeProperties\": {\n \"expression\": {\n \"value\": \"@equals(pipeline().parameters.RaiseErrors,'true')\",\n \"type\": \"Expression\"\n },\n \"ifTrueActivities\": [\n {\n \"name\": \"Call Fail Procedure\",\n \"type\": \"SqlServerStoredProcedure\",\n \"dependsOn\": [],\n \"policy\": {\n \"timeout\": \"0.00:10:00\",\n \"retry\": 0,\n \"retryIntervalInSeconds\": 30,\n \"secureOutput\": false,\n \"secureInput\": false\n },\n \"userProperties\": [],\n \"typeProperties\": {\n \"storedProcedureName\": \"[[dbo].[FailProcedure]\",\n \"storedProcedureParameters\": {\n \"RaiseError\": {\n \"value\": {\n \"value\": \"@pipeline().parameters.RaiseErrors\",\n \"type\": \"Expression\"\n },\n \"type\": \"String\"\n }\n }\n },\n \"linkedServiceName\": {\n \"referenceName\": \"SupportDatabase\",\n \"type\": \"LinkedServiceReference\"\n }\n }\n ]\n }\n }\n ],\n \"parameters\": {\n \"RaiseErrors\": {\n \"type\": \"string\",\n \"defaultValue\": \"false\"\n },\n \"WaitTime\": {\n \"type\": \"int\",\n \"defaultValue\": 5\n }\n },\n \"folder\": {\n \"name\": \"_Workers\"\n },\n \"annotations\": [\n \"_ProcFwkWorker\"\n ]\n },\n \"dependsOn\": [\n \"[concat(variables('workspaceId'), '/linkedServices/SupportDatabase')]\"\n ]\n },\n {\n \"name\": \"[concat(parameters('workspaceName'), '/Throw Exception')]\",\n \"type\": \"Microsoft.Synapse/workspaces/pipelines\",\n \"apiVersion\": \"2019-06-01-preview\",\n \"properties\": {\n \"description\": \"Provide a simple way of throwing an exception within Data Factory using TSQL error handling.\",\n \"activities\": [\n {\n \"name\": \"Raise Error\",\n \"description\": \"Using a SQL database to raise an error/exception but wrapped up as a data factory pipeline. Error message information exposed as a pipeline parameter.\",\n \"type\": \"Lookup\",\n \"dependsOn\": [],\n \"policy\": {\n \"timeout\": \"0.00:10:00\",\n \"retry\": 0,\n \"retryIntervalInSeconds\": 30,\n \"secureOutput\": false,\n \"secureInput\": false\n },\n \"userProperties\": [],\n \"typeProperties\": {\n \"source\": {\n \"type\": \"AzureSqlSource\",\n \"sqlReaderQuery\": {\n \"value\": \"RAISERROR('@{pipeline().parameters.Message}',16,1);\",\n \"type\": \"Expression\"\n },\n \"queryTimeout\": \"02:00:00\",\n \"partitionOption\": \"None\"\n },\n \"dataset\": {\n \"referenceName\": \"GetSetMetadata\",\n \"type\": \"DatasetReference\",\n \"parameters\": {}\n },\n \"firstRowOnly\": false\n }\n }\n ],\n \"parameters\": {\n \"Message\": {\n \"type\": \"string\"\n }\n },\n \"folder\": {\n \"name\": \"_ProcFwk/_ProcFwkUtils\"\n },\n \"annotations\": [\n \"procfwk\",\n \"Utils\"\n ]\n },\n \"dependsOn\": [\n \"[concat(variables('workspaceId'), '/datasets/GetSetMetadata')]\"\n ]\n },\n {\n \"name\": \"[concat(parameters('workspaceName'), '/Wait 1')]\",\n \"type\": \"Microsoft.Synapse/workspaces/pipelines\",\n \"apiVersion\": \"2019-06-01-preview\",\n \"properties\": {\n \"description\": \"Used just so the procfwk has something to call during development.\",\n \"activities\": [\n {\n \"name\": \"Wait1\",\n \"description\": \"Framework development worker simulator.\",\n \"type\": \"Wait\",\n \"dependsOn\": [],\n \"userProperties\": [],\n \"typeProperties\": {\n \"waitTimeInSeconds\": {\n \"value\": \"@pipeline().parameters.WaitTime\",\n \"type\": \"Expression\"\n }\n }\n }\n ],\n \"parameters\": {\n \"WaitTime\": {\n \"type\": \"int\",\n \"defaultValue\": 5\n }\n },\n \"folder\": {\n \"name\": \"_Workers\"\n },\n \"annotations\": [\n \"_ProcFwkWorker\"\n ]\n },\n \"dependsOn\": []\n },\n {\n \"name\": \"[concat(parameters('workspaceName'), '/Wait 10')]\",\n \"type\": \"Microsoft.Synapse/workspaces/pipelines\",\n \"apiVersion\": \"2019-06-01-preview\",\n \"properties\": {\n \"description\": \"Used just so the procfwk has something to call during development.\",\n \"activities\": [\n {\n \"name\": \"Wait10\",\n \"description\": \"Framework development worker simulator.\",\n \"type\": \"Wait\",\n \"dependsOn\": [],\n \"userProperties\": [],\n \"typeProperties\": {\n \"waitTimeInSeconds\": {\n \"value\": \"@pipeline().parameters.WaitTime\",\n \"type\": \"Expression\"\n }\n }\n }\n ],\n \"parameters\": {\n \"WaitTime\": {\n \"type\": \"int\",\n \"defaultValue\": 5\n }\n },\n \"folder\": {\n \"name\": \"_Workers\"\n },\n \"annotations\": [\n \"_ProcFwkWorker\"\n ]\n },\n \"dependsOn\": []\n },\n {\n \"name\": \"[concat(parameters('workspaceName'), '/Wait 2')]\",\n \"type\": \"Microsoft.Synapse/workspaces/pipelines\",\n \"apiVersion\": \"2019-06-01-preview\",\n \"properties\": {\n \"description\": \"Used just so the procfwk has something to call during development.\",\n \"activities\": [\n {\n \"name\": \"Wait2\",\n \"description\": \"Framework development worker simulator.\",\n \"type\": \"Wait\",\n \"dependsOn\": [],\n \"userProperties\": [],\n \"typeProperties\": {\n \"waitTimeInSeconds\": {\n \"value\": \"@pipeline().parameters.WaitTime\",\n \"type\": \"Expression\"\n }\n }\n }\n ],\n \"parameters\": {\n \"WaitTime\": {\n \"type\": \"int\",\n \"defaultValue\": 5\n }\n },\n \"folder\": {\n \"name\": \"_Workers\"\n },\n \"annotations\": [\n \"_ProcFwkWorker\"\n ]\n },\n \"dependsOn\": []\n },\n {\n \"name\": \"[concat(parameters('workspaceName'), '/Wait 3')]\",\n \"type\": \"Microsoft.Synapse/workspaces/pipelines\",\n \"apiVersion\": \"2019-06-01-preview\",\n \"properties\": {\n \"description\": \"Used just so the procfwk has something to call during development.\",\n \"activities\": [\n {\n \"name\": \"Wait3\",\n \"description\": \"Framework development worker simulator.\",\n \"type\": \"Wait\",\n \"dependsOn\": [],\n \"userProperties\": [],\n \"typeProperties\": {\n \"waitTimeInSeconds\": {\n \"value\": \"@pipeline().parameters.WaitTime\",\n \"type\": \"Expression\"\n }\n }\n }\n ],\n \"parameters\": {\n \"WaitTime\": {\n \"type\": \"int\",\n \"defaultValue\": 5\n }\n },\n \"folder\": {\n \"name\": \"_Workers\"\n },\n \"annotations\": [\n \"_ProcFwkWorker\"\n ]\n },\n \"dependsOn\": []\n },\n {\n \"name\": \"[concat(parameters('workspaceName'), '/Wait 4')]\",\n \"type\": \"Microsoft.Synapse/workspaces/pipelines\",\n \"apiVersion\": \"2019-06-01-preview\",\n \"properties\": {\n \"description\": \"Used just so the procfwk has something to call during development.\",\n \"activities\": [\n {\n \"name\": \"Wait4\",\n \"description\": \"Framework development worker simulator.\",\n \"type\": \"Wait\",\n \"dependsOn\": [],\n \"userProperties\": [],\n \"typeProperties\": {\n \"waitTimeInSeconds\": {\n \"value\": \"@pipeline().parameters.WaitTime\",\n \"type\": \"Expression\"\n }\n }\n }\n ],\n \"parameters\": {\n \"WaitTime\": {\n \"type\": \"int\",\n \"defaultValue\": 5\n }\n },\n \"folder\": {\n \"name\": \"_Workers\"\n },\n \"annotations\": [\n \"_ProcFwkWorker\"\n ]\n },\n \"dependsOn\": []\n },\n {\n \"name\": \"[concat(parameters('workspaceName'), '/Wait 5')]\",\n \"type\": \"Microsoft.Synapse/workspaces/pipelines\",\n \"apiVersion\": \"2019-06-01-preview\",\n \"properties\": {\n \"description\": \"Used just so the procfwk has something to call during development.\",\n \"activities\": [\n {\n \"name\": \"Wait5\",\n \"description\": \"Framework development worker simulator.\",\n \"type\": \"Wait\",\n \"dependsOn\": [],\n \"userProperties\": [],\n \"typeProperties\": {\n \"waitTimeInSeconds\": {\n \"value\": \"@pipeline().parameters.WaitTime\",\n \"type\": \"Expression\"\n }\n }\n }\n ],\n \"parameters\": {\n \"WaitTime\": {\n \"type\": \"int\",\n \"defaultValue\": 5\n }\n },\n \"folder\": {\n \"name\": \"_Workers\"\n },\n \"annotations\": [\n \"_ProcFwkWorker\"\n ]\n },\n \"dependsOn\": []\n },\n {\n \"name\": \"[concat(parameters('workspaceName'), '/Wait 6')]\",\n \"type\": \"Microsoft.Synapse/workspaces/pipelines\",\n \"apiVersion\": \"2019-06-01-preview\",\n \"properties\": {\n \"description\": \"Used just so the procfwk has something to call during development.\",\n \"activities\": [\n {\n \"name\": \"Wait6\",\n \"description\": \"Framework development worker simulator.\",\n \"type\": \"Wait\",\n \"dependsOn\": [],\n \"userProperties\": [],\n \"typeProperties\": {\n \"waitTimeInSeconds\": {\n \"value\": \"@pipeline().parameters.WaitTime\",\n \"type\": \"Expression\"\n }\n }\n }\n ],\n \"parameters\": {\n \"WaitTime\": {\n \"type\": \"int\",\n \"defaultValue\": 5\n }\n },\n \"folder\": {\n \"name\": \"_Workers\"\n },\n \"annotations\": [\n \"_ProcFwkWorker\"\n ]\n },\n \"dependsOn\": []\n },\n {\n \"name\": \"[concat(parameters('workspaceName'), '/Wait 7')]\",\n \"type\": \"Microsoft.Synapse/workspaces/pipelines\",\n \"apiVersion\": \"2019-06-01-preview\",\n \"properties\": {\n \"description\": \"Used just so the procfwk has something to call during development.\",\n \"activities\": [\n {\n \"name\": \"Wait7\",\n \"description\": \"Framework development worker simulator.\",\n \"type\": \"Wait\",\n \"dependsOn\": [],\n \"userProperties\": [],\n \"typeProperties\": {\n \"waitTimeInSeconds\": {\n \"value\": \"@pipeline().parameters.WaitTime\",\n \"type\": \"Expression\"\n }\n }\n }\n ],\n \"parameters\": {\n \"WaitTime\": {\n \"type\": \"int\",\n \"defaultValue\": 5\n }\n },\n \"folder\": {\n \"name\": \"_Workers\"\n },\n \"annotations\": [\n \"_ProcFwkWorker\"\n ]\n },\n \"dependsOn\": []\n },\n {\n \"name\": \"[concat(parameters('workspaceName'), '/Wait 8')]\",\n \"type\": \"Microsoft.Synapse/workspaces/pipelines\",\n \"apiVersion\": \"2019-06-01-preview\",\n \"properties\": {\n \"description\": \"Used just so the procfwk has something to call during development.\",\n \"activities\": [\n {\n \"name\": \"Wait8\",\n \"description\": \"Framework development worker simulator.\",\n \"type\": \"Wait\",\n \"dependsOn\": [],\n \"userProperties\": [],\n \"typeProperties\": {\n \"waitTimeInSeconds\": {\n \"value\": \"@pipeline().parameters.WaitTime\",\n \"type\": \"Expression\"\n }\n }\n }\n ],\n \"parameters\": {\n \"WaitTime\": {\n \"type\": \"int\",\n \"defaultValue\": 5\n }\n },\n \"folder\": {\n \"name\": \"_Workers\"\n },\n \"annotations\": [\n \"_ProcFwkWorker\"\n ]\n },\n \"dependsOn\": []\n },\n {\n \"name\": \"[concat(parameters('workspaceName'), '/Wait 9')]\",\n \"type\": \"Microsoft.Synapse/workspaces/pipelines\",\n \"apiVersion\": \"2019-06-01-preview\",\n \"properties\": {\n \"description\": \"Used just so the procfwk has something to call during development.\",\n \"activities\": [\n {\n \"name\": \"Wait9\",\n \"description\": \"Framework development worker simulator.\",\n \"type\": \"Wait\",\n \"dependsOn\": [],\n \"userProperties\": [],\n \"typeProperties\": {\n \"waitTimeInSeconds\": {\n \"value\": \"@pipeline().parameters.WaitTime\",\n \"type\": \"Expression\"\n }\n }\n }\n ],\n \"parameters\": {\n \"WaitTime\": {\n \"type\": \"int\",\n \"defaultValue\": 15\n }\n },\n \"folder\": {\n \"name\": \"_Workers\"\n },\n \"annotations\": [\n \"_ProcFwkWorker\"\n ]\n },\n \"dependsOn\": []\n },\n {\n \"name\": \"[concat(parameters('workspaceName'), '/GetSetMetadata')]\",\n \"type\": \"Microsoft.Synapse/workspaces/datasets\",\n \"apiVersion\": \"2019-06-01-preview\",\n \"properties\": {\n \"description\": \"Single generic dataset used to get and set all database metadata.\",\n \"linkedServiceName\": {\n \"referenceName\": \"SupportDatabase\",\n \"type\": \"LinkedServiceReference\"\n },\n \"folder\": {\n \"name\": \"_ProcFwk\"\n },\n \"annotations\": [\n \"procfwk\"\n ],\n \"type\": \"AzureSqlTable\",\n \"schema\": [],\n \"typeProperties\": {}\n },\n \"dependsOn\": [\n \"[concat(variables('workspaceId'), '/linkedServices/SupportDatabase')]\"\n ]\n },\n {\n \"name\": \"[concat(parameters('workspaceName'), '/FrameworkFunctions')]\",\n \"type\": \"Microsoft.Synapse/workspaces/linkedServices\",\n \"apiVersion\": \"2019-06-01-preview\",\n \"properties\": {\n \"description\": \"Interact with the Azure Functions App used as middle ware when making requests to Worker pipelines. Authentication done at the Function App level.\",\n \"annotations\": [\n \"procfwk\"\n ],\n \"type\": \"AzureFunction\",\n \"typeProperties\": {\n \"functionAppUrl\": \"[parameters('FrameworkFunctions_properties_typeProperties_functionAppUrl')]\",\n \"functionKey\": {\n \"type\": \"AzureKeyVaultSecret\",\n \"store\": {\n \"referenceName\": \"Keys\",\n \"type\": \"LinkedServiceReference\"\n },\n \"secretName\": \"FrameworkFunctionsKey\"\n }\n }\n },\n \"dependsOn\": [\n \"[concat(variables('workspaceId'), '/linkedServices/Keys')]\"\n ]\n },\n {\n \"name\": \"[concat(parameters('workspaceName'), '/Keys')]\",\n \"type\": \"Microsoft.Synapse/workspaces/linkedServices\",\n \"apiVersion\": \"2019-06-01-preview\",\n \"properties\": {\n \"description\": \"Connection to Key Vault for all other ADF linked service credentials required to run the processing framework.\",\n \"annotations\": [\n \"procfwk\"\n ],\n \"type\": \"AzureKeyVault\",\n \"typeProperties\": {\n \"baseUrl\": \"[parameters('Keys_properties_typeProperties_baseUrl')]\"\n }\n },\n \"dependsOn\": []\n },\n {\n \"name\": \"[concat(parameters('workspaceName'), '/SupportDatabase')]\",\n \"type\": \"Microsoft.Synapse/workspaces/linkedServices\",\n \"apiVersion\": \"2019-06-01-preview\",\n \"properties\": {\n \"description\": \"Connection between ADF and processing framework metadata SQLDB.\",\n \"annotations\": [\n \"procfwk\"\n ],\n \"type\": \"AzureSqlDatabase\",\n \"typeProperties\": {\n \"connectionString\": {\n \"type\": \"AzureKeyVaultSecret\",\n \"store\": {\n \"referenceName\": \"Keys\",\n \"type\": \"LinkedServiceReference\"\n },\n \"secretName\": \"[parameters('SupportDatabase_properties_typeProperties_connectionString_secretName')]\"\n }\n }\n },\n \"dependsOn\": [\n \"[concat(variables('workspaceId'), '/linkedServices/Keys')]\"\n ]\n },\n {\n \"name\": \"[concat(parameters('workspaceName'), '/procfwkforsynapse-WorkspaceDefaultSqlServer')]\",\n \"type\": \"Microsoft.Synapse/workspaces/linkedServices\",\n \"apiVersion\": \"2019-06-01-preview\",\n \"properties\": {\n \"parameters\": {\n \"DBName\": {\n \"type\": \"String\"\n }\n },\n \"annotations\": [],\n \"type\": \"AzureSqlDW\",\n \"typeProperties\": {\n \"connectionString\": \"[parameters('procfwkforsynapse-WorkspaceDefaultSqlServer_connectionString')]\"\n },\n \"connectVia\": {\n \"referenceName\": \"AutoResolveIntegrationRuntime\",\n \"type\": \"IntegrationRuntimeReference\"\n }\n },\n \"dependsOn\": [\n \"[concat(variables('workspaceId'), '/integrationRuntimes/AutoResolveIntegrationRuntime')]\"\n ]\n },\n {\n \"name\": \"[concat(parameters('workspaceName'), '/procfwkforsynapse-WorkspaceDefaultStorage')]\",\n \"type\": \"Microsoft.Synapse/workspaces/linkedServices\",\n \"apiVersion\": \"2019-06-01-preview\",\n \"properties\": {\n \"annotations\": [],\n \"type\": \"AzureBlobFS\",\n \"typeProperties\": {\n \"url\": \"[parameters('procfwkforsynapse-WorkspaceDefaultStorage_properties_typeProperties_url')]\"\n },\n \"connectVia\": {\n \"referenceName\": \"AutoResolveIntegrationRuntime\",\n \"type\": \"IntegrationRuntimeReference\"\n }\n },\n \"dependsOn\": [\n \"[concat(variables('workspaceId'), '/integrationRuntimes/AutoResolveIntegrationRuntime')]\"\n ]\n },\n {\n \"name\": \"[concat(parameters('workspaceName'), '/FunctionalTestingTrigger')]\",\n \"type\": \"Microsoft.Synapse/workspaces/triggers\",\n \"apiVersion\": \"2019-06-01-preview\",\n \"properties\": {\n \"description\": \"Used for functional testing of the framework in a dedicated environment.\",\n \"annotations\": [\n \"procfwk\"\n ],\n \"runtimeState\": \"Stopped\",\n \"pipelines\": [\n {\n \"pipelineReference\": {\n \"referenceName\": \"01-Grandparent\",\n \"type\": \"PipelineReference\"\n },\n \"parameters\": {}\n }\n ],\n \"type\": \"ScheduleTrigger\",\n \"typeProperties\": {\n \"recurrence\": {\n \"frequency\": \"Hour\",\n \"interval\": 2,\n \"startTime\": \"2020-04-06T15:00:00Z\",\n \"timeZone\": \"UTC\"\n }\n }\n },\n \"dependsOn\": [\n \"[concat(variables('workspaceId'), '/pipelines/01-Grandparent')]\"\n ]\n },\n {\n \"name\": \"[concat(parameters('workspaceName'), '/AutoResolveIntegrationRuntime')]\",\n \"type\": \"Microsoft.Synapse/workspaces/integrationRuntimes\",\n \"apiVersion\": \"2019-06-01-preview\",\n \"properties\": {\n \"type\": \"Managed\",\n \"typeProperties\": {\n \"computeProperties\": {\n \"location\": \"AutoResolve\",\n \"dataFlowProperties\": {\n \"computeType\": \"General\",\n \"coreCount\": 8,\n \"timeToLive\": 0\n }\n }\n }\n },\n \"dependsOn\": []\n }\n ]\n}"
},
{
"path": "CODE_OF_CONDUCT.md",
"content": "# Contributor Covenant Code of Conduct\n\n## Our Pledge\n\nIn the interest of fostering an open and welcoming environment, we as\ncontributors and maintainers pledge to making participation in our project and\nour community a harassment-free experience for everyone, regardless of age, body\nsize, disability, ethnicity, sex characteristics, gender identity and expression,\nlevel of experience, education, socio-economic status, nationality, personal\nappearance, race, religion, or sexual identity and orientation.\n\n## Our Standards\n\nExamples of behavior that contributes to creating a positive environment\ninclude:\n\n* Using welcoming and inclusive language\n* Being respectful of differing viewpoints and experiences\n* Gracefully accepting constructive criticism\n* Focusing on what is best for the community\n* Showing empathy towards other community members\n\nExamples of unacceptable behavior by participants include:\n\n* The use of sexualized language or imagery and unwelcome sexual attention or\n advances\n* Trolling, insulting/derogatory comments, and personal or political attacks\n* Public or private harassment\n* Publishing others' private information, such as a physical or electronic\n address, without explicit permission\n* Other conduct which could reasonably be considered inappropriate in a\n professional setting\n\n## Our Responsibilities\n\nProject maintainers are responsible for clarifying the standards of acceptable\nbehavior and are expected to take appropriate and fair corrective action in\nresponse to any instances of unacceptable behavior.\n\nProject maintainers have the right and responsibility to remove, edit, or\nreject comments, commits, code, wiki edits, issues, and other contributions\nthat are not aligned to this Code of Conduct, or to ban temporarily or\npermanently any contributor for other behaviors that they deem inappropriate,\nthreatening, offensive, or harmful.\n\n## Scope\n\nThis Code of Conduct applies both within project spaces and in public spaces\nwhen an individual is representing the project or its community. Examples of\nrepresenting a project or community include using an official project e-mail\naddress, posting via an official social media account, or acting as an appointed\nrepresentative at an online or offline event. Representation of a project may be\nfurther defined and clarified by project maintainers.\n\n## Enforcement\n\nInstances of abusive, harassing, or otherwise unacceptable behavior may be\nreported by contacting the project team at paul@mrpaulandrew.com. All\ncomplaints will be reviewed and investigated and will result in a response that\nis deemed necessary and appropriate to the circumstances. The project team is\nobligated to maintain confidentiality with regard to the reporter of an incident.\nFurther details of specific enforcement policies may be posted separately.\n\nProject maintainers who do not follow or enforce the Code of Conduct in good\nfaith may face temporary or permanent repercussions as determined by other\nmembers of the project's leadership.\n\n## Attribution\n\nThis Code of Conduct is adapted from the [Contributor Covenant][homepage], version 1.4,\navailable at https://www.contributor-covenant.org/version/1/4/code-of-conduct.html\n\n[homepage]: https://www.contributor-covenant.org\n\nFor answers to common questions about this code of conduct, see\nhttps://www.contributor-covenant.org/faq\n"
},
{
"path": "CONTRIBUTING.md",
"content": "If you'd like to contribute towards this code project please contact me. I kindly request a minimum of 5 years experience working with Azure Data Platform services.\n\nThanks\n\nPaul\n"
},
{
"path": "DataFactory/dataset/GetSetMetadata.json",
"content": "{\n\t\"name\": \"GetSetMetadata\",\n\t\"properties\": {\n\t\t\"description\": \"Single generic dataset used to get and set all database metadata.\",\n\t\t\"linkedServiceName\": {\n\t\t\t\"referenceName\": \"SupportDatabase\",\n\t\t\t\"type\": \"LinkedServiceReference\"\n\t\t},\n\t\t\"folder\": {\n\t\t\t\"name\": \"_ProcFwk\"\n\t\t},\n\t\t\"annotations\": [\n\t\t\t\"procfwk\"\n\t\t],\n\t\t\"type\": \"AzureSqlTable\",\n\t\t\"schema\": []\n\t}\n}"
},
{
"path": "DataFactory/factory/FrameworkFactory.json",
"content": "{\n\t\"name\": \"FrameworkFactory\",\n\t\"properties\": {\n\t\t\"globalConfigurations\": {\n\t\t\t\"PipelineBillingEnabled\": \"true\"\n\t\t}\n\t},\n\t\"location\": \"uksouth\"\n}"
},
{
"path": "DataFactory/integrationRuntime/AzureIR-UKSouth.json",
"content": "{\n\t\"name\": \"AzureIR-UKSouth\",\n\t\"properties\": {\n\t\t\"type\": \"Managed\",\n\t\t\"typeProperties\": {\n\t\t\t\"computeProperties\": {\n\t\t\t\t\"location\": \"UK South\",\n\t\t\t\t\"dataFlowProperties\": {\n\t\t\t\t\t\"computeType\": \"General\",\n\t\t\t\t\t\"coreCount\": 8,\n\t\t\t\t\t\"timeToLive\": 10,\n\t\t\t\t\t\"cleanup\": false\n\t\t\t\t}\n\t\t\t}\n\t\t}\n\t}\n}"
},
{
"path": "DataFactory/linkedService/FrameworkFunctions.json",
"content": "{\n\t\"name\": \"FrameworkFunctions\",\n\t\"properties\": {\n\t\t\"annotations\": [\n\t\t\t\"procfwk\"\n\t\t],\n\t\t\"type\": \"AzureFunction\",\n\t\t\"typeProperties\": {\n\t\t\t\"functionAppUrl\": \"https://frameworksupportfunctions.azurewebsites.net\",\n\t\t\t\"functionKey\": {\n\t\t\t\t\"type\": \"AzureKeyVaultSecret\",\n\t\t\t\t\"store\": {\n\t\t\t\t\t\"referenceName\": \"Keys\",\n\t\t\t\t\t\"type\": \"LinkedServiceReference\"\n\t\t\t\t},\n\t\t\t\t\"secretName\": \"FrameworkFunctionsKey\"\n\t\t\t}\n\t\t},\n\t\t\"description\": \"Interact with the Azure Functions App used as middle ware when making requests to Worker pipelines. Authentication done at the Function App level.\"\n\t}\n}"
},
{
"path": "DataFactory/linkedService/Keys.json",
"content": "{\n\t\"name\": \"Keys\",\n\t\"properties\": {\n\t\t\"annotations\": [\n\t\t\t\"procfwk\"\n\t\t],\n\t\t\"type\": \"AzureKeyVault\",\n\t\t\"typeProperties\": {\n\t\t\t\"baseUrl\": \"https://FrameworkKeys.vault.azure.net/\"\n\t\t},\n\t\t\"description\": \"Connection to Key Vault for all other ADF linked service credentials required to run the processing framework.\"\n\t}\n}"
},
{
"path": "DataFactory/linkedService/SupportDatabase.json",
"content": "{\n\t\"name\": \"SupportDatabase\",\n\t\"properties\": {\n\t\t\"description\": \"Connection between ADF and processing framework metadata SQLDB.\",\n\t\t\"annotations\": [\n\t\t\t\"procfwk\"\n\t\t],\n\t\t\"type\": \"AzureSqlDatabase\",\n\t\t\"typeProperties\": {\n\t\t\t\"connectionString\": {\n\t\t\t\t\"type\": \"AzureKeyVaultSecret\",\n\t\t\t\t\"store\": {\n\t\t\t\t\t\"referenceName\": \"Keys\",\n\t\t\t\t\t\"type\": \"LinkedServiceReference\"\n\t\t\t\t},\n\t\t\t\t\"secretName\": \"FrameworkMetadataDev\"\n\t\t\t}\n\t\t}\n\t}\n}"
},
{
"path": "DataFactory/pipeline/01-Grandparent.json",
"content": "{\n\t\"name\": \"01-Grandparent\",\n\t\"properties\": {\n\t\t\"description\": \"procfwk grandparent pipeline used optionally to bootstrap any wider processes in your Data Factory that then calls the processing framework.\",\n\t\t\"activities\": [\n\t\t\t{\n\t\t\t\t\"name\": \"procfwk\",\n\t\t\t\t\"description\": \"Call procfwk\",\n\t\t\t\t\"type\": \"ExecutePipeline\",\n\t\t\t\t\"dependsOn\": [],\n\t\t\t\t\"userProperties\": [],\n\t\t\t\t\"typeProperties\": {\n\t\t\t\t\t\"pipeline\": {\n\t\t\t\t\t\t\"referenceName\": \"02-Parent\",\n\t\t\t\t\t\t\"type\": \"PipelineReference\"\n\t\t\t\t\t},\n\t\t\t\t\t\"waitOnCompletion\": true,\n\t\t\t\t\t\"parameters\": {\n\t\t\t\t\t\t\"BatchName\": {\n\t\t\t\t\t\t\t\"value\": \"@pipeline().parameters.BatchName\",\n\t\t\t\t\t\t\t\"type\": \"Expression\"\n\t\t\t\t\t\t}\n\t\t\t\t\t}\n\t\t\t\t}\n\t\t\t}\n\t\t],\n\t\t\"parameters\": {\n\t\t\t\"BatchName\": {\n\t\t\t\t\"type\": \"string\",\n\t\t\t\t\"defaultValue\": \"NotUsed\"\n\t\t\t}\n\t\t},\n\t\t\"folder\": {\n\t\t\t\"name\": \"_ProcFwk\"\n\t\t},\n\t\t\"annotations\": [\n\t\t\t\"procfwk\",\n\t\t\t\"Grandparent\"\n\t\t]\n\t}\n}"
},
{
"path": "DataFactory/pipeline/02-Parent.json",
"content": "{\n\t\"name\": \"02-Parent\",\n\t\"properties\": {\n\t\t\"description\": \"ADF.procfwk parent pipeline used to bootstrap the orchestration framework in perform the first level ForEach calls in sequence for the metadata stages.\",\n\t\t\"activities\": [\n\t\t\t{\n\t\t\t\t\"name\": \"Get Stages\",\n\t\t\t\t\"description\": \"Returns a distinct list of execution stages within the framework metadata.\",\n\t\t\t\t\"type\": \"Lookup\",\n\t\t\t\t\"dependsOn\": [\n\t\t\t\t\t{\n\t\t\t\t\t\t\"activity\": \"Set Execution Id\",\n\t\t\t\t\t\t\"dependencyConditions\": [\n\t\t\t\t\t\t\t\"Succeeded\"\n\t\t\t\t\t\t]\n\t\t\t\t\t}\n\t\t\t\t],\n\t\t\t\t\"policy\": {\n\t\t\t\t\t\"timeout\": \"0.00:10:00\",\n\t\t\t\t\t\"retry\": 0,\n\t\t\t\t\t\"retryIntervalInSeconds\": 30,\n\t\t\t\t\t\"secureOutput\": false,\n\t\t\t\t\t\"secureInput\": false\n\t\t\t\t},\n\t\t\t\t\"userProperties\": [],\n\t\t\t\t\"typeProperties\": {\n\t\t\t\t\t\"source\": {\n\t\t\t\t\t\t\"type\": \"AzureSqlSource\",\n\t\t\t\t\t\t\"sqlReaderStoredProcedureName\": \"[procfwk].[GetStages]\",\n\t\t\t\t\t\t\"storedProcedureParameters\": {\n\t\t\t\t\t\t\t\"ExecutionId\": {\n\t\t\t\t\t\t\t\t\"type\": \"Guid\",\n\t\t\t\t\t\t\t\t\"value\": {\n\t\t\t\t\t\t\t\t\t\"value\": \"@variables('ExecutionId')\",\n\t\t\t\t\t\t\t\t\t\"type\": \"Expression\"\n\t\t\t\t\t\t\t\t}\n\t\t\t\t\t\t\t}\n\t\t\t\t\t\t},\n\t\t\t\t\t\t\"queryTimeout\": \"02:00:00\",\n\t\t\t\t\t\t\"partitionOption\": \"None\"\n\t\t\t\t\t},\n\t\t\t\t\t\"dataset\": {\n\t\t\t\t\t\t\"referenceName\": \"GetSetMetadata\",\n\t\t\t\t\t\t\"type\": \"DatasetReference\"\n\t\t\t\t\t},\n\t\t\t\t\t\"firstRowOnly\": false\n\t\t\t\t}\n\t\t\t},\n\t\t\t{\n\t\t\t\t\"name\": \"Execute Stages\",\n\t\t\t\t\"description\": \"Top level ForEach to sequentially call all processing stages within the framework metadata. Items for iteration passed from the Get Stages lookup activity.\",\n\t\t\t\t\"type\": \"ForEach\",\n\t\t\t\t\"dependsOn\": [\n\t\t\t\t\t{\n\t\t\t\t\t\t\"activity\": \"Get Stages\",\n\t\t\t\t\t\t\"dependencyConditions\": [\n\t\t\t\t\t\t\t\"Succeeded\"\n\t\t\t\t\t\t]\n\t\t\t\t\t}\n\t\t\t\t],\n\t\t\t\t\"userProperties\": [],\n\t\t\t\t\"typeProperties\": {\n\t\t\t\t\t\"items\": {\n\t\t\t\t\t\t\"value\": \"@activity('Get Stages').output.value\",\n\t\t\t\t\t\t\"type\": \"Expression\"\n\t\t\t\t\t},\n\t\t\t\t\t\"isSequential\": true,\n\t\t\t\t\t\"activities\": [\n\t\t\t\t\t\t{\n\t\t\t\t\t\t\t\"name\": \"Stage Executor\",\n\t\t\t\t\t\t\t\"description\": \"Call to the framework generic child pipeline for a given execution stage.\",\n\t\t\t\t\t\t\t\"type\": \"ExecutePipeline\",\n\t\t\t\t\t\t\t\"dependsOn\": [\n\t\t\t\t\t\t\t\t{\n\t\t\t\t\t\t\t\t\t\"activity\": \"Log Stage Preparing\",\n\t\t\t\t\t\t\t\t\t\"dependencyConditions\": [\n\t\t\t\t\t\t\t\t\t\t\"Succeeded\"\n\t\t\t\t\t\t\t\t\t]\n\t\t\t\t\t\t\t\t}\n\t\t\t\t\t\t\t],\n\t\t\t\t\t\t\t\"userProperties\": [],\n\t\t\t\t\t\t\t\"typeProperties\": {\n\t\t\t\t\t\t\t\t\"pipeline\": {\n\t\t\t\t\t\t\t\t\t\"referenceName\": \"03-Child\",\n\t\t\t\t\t\t\t\t\t\"type\": \"PipelineReference\"\n\t\t\t\t\t\t\t\t},\n\t\t\t\t\t\t\t\t\"waitOnCompletion\": true,\n\t\t\t\t\t\t\t\t\"parameters\": {\n\t\t\t\t\t\t\t\t\t\"StageId\": {\n\t\t\t\t\t\t\t\t\t\t\"value\": \"@item().StageId\",\n\t\t\t\t\t\t\t\t\t\t\"type\": \"Expression\"\n\t\t\t\t\t\t\t\t\t},\n\t\t\t\t\t\t\t\t\t\"ExecutionId\": {\n\t\t\t\t\t\t\t\t\t\t\"value\": \"@variables('ExecutionId')\",\n\t\t\t\t\t\t\t\t\t\t\"type\": \"Expression\"\n\t\t\t\t\t\t\t\t\t}\n\t\t\t\t\t\t\t\t}\n\t\t\t\t\t\t\t}\n\t\t\t\t\t\t},\n\t\t\t\t\t\t{\n\t\t\t\t\t\t\t\"name\": \"Log Stage Preparing\",\n\t\t\t\t\t\t\t\"description\": \"Update the current execution table flagging all pipelines within the stage as preparing.\",\n\t\t\t\t\t\t\t\"type\": \"SqlServerStoredProcedure\",\n\t\t\t\t\t\t\t\"dependsOn\": [\n\t\t\t\t\t\t\t\t{\n\t\t\t\t\t\t\t\t\t\"activity\": \"Check and Update Blockers\",\n\t\t\t\t\t\t\t\t\t\"dependencyConditions\": [\n\t\t\t\t\t\t\t\t\t\t\"Succeeded\"\n\t\t\t\t\t\t\t\t\t]\n\t\t\t\t\t\t\t\t}\n\t\t\t\t\t\t\t],\n\t\t\t\t\t\t\t\"policy\": {\n\t\t\t\t\t\t\t\t\"timeout\": \"0.00:10:00\",\n\t\t\t\t\t\t\t\t\"retry\": 0,\n\t\t\t\t\t\t\t\t\"retryIntervalInSeconds\": 30,\n\t\t\t\t\t\t\t\t\"secureOutput\": false,\n\t\t\t\t\t\t\t\t\"secureInput\": false\n\t\t\t\t\t\t\t},\n\t\t\t\t\t\t\t\"userProperties\": [],\n\t\t\t\t\t\t\t\"typeProperties\": {\n\t\t\t\t\t\t\t\t\"storedProcedureName\": \"[procfwk].[SetLogStagePreparing]\",\n\t\t\t\t\t\t\t\t\"storedProcedureParameters\": {\n\t\t\t\t\t\t\t\t\t\"ExecutionId\": {\n\t\t\t\t\t\t\t\t\t\t\"value\": {\n\t\t\t\t\t\t\t\t\t\t\t\"value\": \"@variables('ExecutionId')\",\n\t\t\t\t\t\t\t\t\t\t\t\"type\": \"Expression\"\n\t\t\t\t\t\t\t\t\t\t},\n\t\t\t\t\t\t\t\t\t\t\"type\": \"Guid\"\n\t\t\t\t\t\t\t\t\t},\n\t\t\t\t\t\t\t\t\t\"StageId\": {\n\t\t\t\t\t\t\t\t\t\t\"value\": {\n\t\t\t\t\t\t\t\t\t\t\t\"value\": \"@item().StageId\",\n\t\t\t\t\t\t\t\t\t\t\t\"type\": \"Expression\"\n\t\t\t\t\t\t\t\t\t\t},\n\t\t\t\t\t\t\t\t\t\t\"type\": \"Int32\"\n\t\t\t\t\t\t\t\t\t}\n\t\t\t\t\t\t\t\t}\n\t\t\t\t\t\t\t},\n\t\t\t\t\t\t\t\"linkedServiceName\": {\n\t\t\t\t\t\t\t\t\"referenceName\": \"SupportDatabase\",\n\t\t\t\t\t\t\t\t\"type\": \"LinkedServiceReference\"\n\t\t\t\t\t\t\t}\n\t\t\t\t\t\t},\n\t\t\t\t\t\t{\n\t\t\t\t\t\t\t\"name\": \"Check and Update Blockers\",\n\t\t\t\t\t\t\t\"description\": \"Used to double check and stop the next execution stage if failures and blockers have be incurred. This also depends on the failure handling property value which defines the stored procedure behaviour.\",\n\t\t\t\t\t\t\t\"type\": \"SqlServerStoredProcedure\",\n\t\t\t\t\t\t\t\"dependsOn\": [],\n\t\t\t\t\t\t\t\"policy\": {\n\t\t\t\t\t\t\t\t\"timeout\": \"0.00:10:00\",\n\t\t\t\t\t\t\t\t\"retry\": 0,\n\t\t\t\t\t\t\t\t\"retryIntervalInSeconds\": 30,\n\t\t\t\t\t\t\t\t\"secureOutput\": false,\n\t\t\t\t\t\t\t\t\"secureInput\": false\n\t\t\t\t\t\t\t},\n\t\t\t\t\t\t\t\"userProperties\": [],\n\t\t\t\t\t\t\t\"typeProperties\": {\n\t\t\t\t\t\t\t\t\"storedProcedureName\": \"[procfwk].[CheckForBlockedPipelines]\",\n\t\t\t\t\t\t\t\t\"storedProcedureParameters\": {\n\t\t\t\t\t\t\t\t\t\"ExecutionId\": {\n\t\t\t\t\t\t\t\t\t\t\"value\": {\n\t\t\t\t\t\t\t\t\t\t\t\"value\": \"@variables('ExecutionId')\",\n\t\t\t\t\t\t\t\t\t\t\t\"type\": \"Expression\"\n\t\t\t\t\t\t\t\t\t\t},\n\t\t\t\t\t\t\t\t\t\t\"type\": \"Guid\"\n\t\t\t\t\t\t\t\t\t},\n\t\t\t\t\t\t\t\t\t\"StageId\": {\n\t\t\t\t\t\t\t\t\t\t\"value\": {\n\t\t\t\t\t\t\t\t\t\t\t\"value\": \"@item().StageId\",\n\t\t\t\t\t\t\t\t\t\t\t\"type\": \"Expression\"\n\t\t\t\t\t\t\t\t\t\t},\n\t\t\t\t\t\t\t\t\t\t\"type\": \"Int32\"\n\t\t\t\t\t\t\t\t\t}\n\t\t\t\t\t\t\t\t}\n\t\t\t\t\t\t\t},\n\t\t\t\t\t\t\t\"linkedServiceName\": {\n\t\t\t\t\t\t\t\t\"referenceName\": \"SupportDatabase\",\n\t\t\t\t\t\t\t\t\"type\": \"LinkedServiceReference\"\n\t\t\t\t\t\t\t}\n\t\t\t\t\t\t}\n\t\t\t\t\t]\n\t\t\t\t}\n\t\t\t},\n\t\t\t{\n\t\t\t\t\"name\": \"Execution Wrapper\",\n\t\t\t\t\"description\": \"Wrapper to reset and restart processing or create a completely new execution instance of the framework metadata.\",\n\t\t\t\t\"type\": \"Lookup\",\n\t\t\t\t\"dependsOn\": [\n\t\t\t\t\t{\n\t\t\t\t\t\t\"activity\": \"Clean Up Previous Run\",\n\t\t\t\t\t\t\"dependencyConditions\": [\n\t\t\t\t\t\t\t\"Succeeded\"\n\t\t\t\t\t\t]\n\t\t\t\t\t}\n\t\t\t\t],\n\t\t\t\t\"policy\": {\n\t\t\t\t\t\"timeout\": \"0.00:10:00\",\n\t\t\t\t\t\"retry\": 0,\n\t\t\t\t\t\"retryIntervalInSeconds\": 30,\n\t\t\t\t\t\"secureOutput\": false,\n\t\t\t\t\t\"secureInput\": false\n\t\t\t\t},\n\t\t\t\t\"userProperties\": [],\n\t\t\t\t\"typeProperties\": {\n\t\t\t\t\t\"source\": {\n\t\t\t\t\t\t\"type\": \"AzureSqlSource\",\n\t\t\t\t\t\t\"sqlReaderStoredProcedureName\": \"[procfwk].[ExecutionWrapper]\",\n\t\t\t\t\t\t\"storedProcedureParameters\": {\n\t\t\t\t\t\t\t\"CallingOrchestratorName\": {\n\t\t\t\t\t\t\t\t\"type\": \"String\",\n\t\t\t\t\t\t\t\t\"value\": {\n\t\t\t\t\t\t\t\t\t\"value\": \"@pipeline().DataFactory\",\n\t\t\t\t\t\t\t\t\t\"type\": \"Expression\"\n\t\t\t\t\t\t\t\t}\n\t\t\t\t\t\t\t},\n\t\t\t\t\t\t\t\"BatchName\": {\n\t\t\t\t\t\t\t\t\"type\": \"String\",\n\t\t\t\t\t\t\t\t\"value\": {\n\t\t\t\t\t\t\t\t\t\"value\": \"@pipeline().parameters.BatchName\",\n\t\t\t\t\t\t\t\t\t\"type\": \"Expression\"\n\t\t\t\t\t\t\t\t}\n\t\t\t\t\t\t\t}\n\t\t\t\t\t\t},\n\t\t\t\t\t\t\"queryTimeout\": \"02:00:00\",\n\t\t\t\t\t\t\"partitionOption\": \"None\"\n\t\t\t\t\t},\n\t\t\t\t\t\"dataset\": {\n\t\t\t\t\t\t\"referenceName\": \"GetSetMetadata\",\n\t\t\t\t\t\t\"type\": \"DatasetReference\"\n\t\t\t\t\t}\n\t\t\t\t}\n\t\t\t},\n\t\t\t{\n\t\t\t\t\"name\": \"Check Outcome and Update Logs\",\n\t\t\t\t\"description\": \"After a successful execution run the current execution metadata is moved to the long term logging table by this stored procedure call. Otherwise an error will be raised.\",\n\t\t\t\t\"type\": \"SqlServerStoredProcedure\",\n\t\t\t\t\"dependsOn\": [\n\t\t\t\t\t{\n\t\t\t\t\t\t\"activity\": \"Execute Stages\",\n\t\t\t\t\t\t\"dependencyConditions\": [\n\t\t\t\t\t\t\t\"Succeeded\"\n\t\t\t\t\t\t]\n\t\t\t\t\t}\n\t\t\t\t],\n\t\t\t\t\"policy\": {\n\t\t\t\t\t\"timeout\": \"0.00:10:00\",\n\t\t\t\t\t\"retry\": 0,\n\t\t\t\t\t\"retryIntervalInSeconds\": 30,\n\t\t\t\t\t\"secureOutput\": false,\n\t\t\t\t\t\"secureInput\": false\n\t\t\t\t},\n\t\t\t\t\"userProperties\": [],\n\t\t\t\t\"typeProperties\": {\n\t\t\t\t\t\"storedProcedureName\": \"[procfwk].[UpdateExecutionLog]\",\n\t\t\t\t\t\"storedProcedureParameters\": {\n\t\t\t\t\t\t\"PerformErrorCheck\": {\n\t\t\t\t\t\t\t\"value\": {\n\t\t\t\t\t\t\t\t\"value\": \"@bool(1)\",\n\t\t\t\t\t\t\t\t\"type\": \"Expression\"\n\t\t\t\t\t\t\t},\n\t\t\t\t\t\t\t\"type\": \"Boolean\"\n\t\t\t\t\t\t},\n\t\t\t\t\t\t\"ExecutionId\": {\n\t\t\t\t\t\t\t\"value\": {\n\t\t\t\t\t\t\t\t\"value\": \"@variables('ExecutionId')\",\n\t\t\t\t\t\t\t\t\"type\": \"Expression\"\n\t\t\t\t\t\t\t},\n\t\t\t\t\t\t\t\"type\": \"Guid\"\n\t\t\t\t\t\t}\n\t\t\t\t\t}\n\t\t\t\t},\n\t\t\t\t\"linkedServiceName\": {\n\t\t\t\t\t\"referenceName\": \"SupportDatabase\",\n\t\t\t\t\t\"type\": \"LinkedServiceReference\"\n\t\t\t\t}\n\t\t\t},\n\t\t\t{\n\t\t\t\t\"name\": \"Check Previous Execution\",\n\t\t\t\t\"description\": \"Query the current execution table for worker pipelines that require a clean up from the previous execution run.\",\n\t\t\t\t\"type\": \"Lookup\",\n\t\t\t\t\"dependsOn\": [\n\t\t\t\t\t{\n\t\t\t\t\t\t\"activity\": \"Execute Precursor\",\n\t\t\t\t\t\t\"dependencyConditions\": [\n\t\t\t\t\t\t\t\"Succeeded\"\n\t\t\t\t\t\t]\n\t\t\t\t\t}\n\t\t\t\t],\n\t\t\t\t\"policy\": {\n\t\t\t\t\t\"timeout\": \"0.00:10:00\",\n\t\t\t\t\t\"retry\": 0,\n\t\t\t\t\t\"retryIntervalInSeconds\": 30,\n\t\t\t\t\t\"secureOutput\": false,\n\t\t\t\t\t\"secureInput\": false\n\t\t\t\t},\n\t\t\t\t\"userProperties\": [],\n\t\t\t\t\"typeProperties\": {\n\t\t\t\t\t\"source\": {\n\t\t\t\t\t\t\"type\": \"AzureSqlSource\",\n\t\t\t\t\t\t\"sqlReaderStoredProcedureName\": \"[procfwk].[CheckPreviousExeuction]\",\n\t\t\t\t\t\t\"storedProcedureParameters\": {\n\t\t\t\t\t\t\t\"BatchName\": {\n\t\t\t\t\t\t\t\t\"type\": \"String\",\n\t\t\t\t\t\t\t\t\"value\": {\n\t\t\t\t\t\t\t\t\t\"value\": \"@pipeline().parameters.BatchName\",\n\t\t\t\t\t\t\t\t\t\"type\": \"Expression\"\n\t\t\t\t\t\t\t\t}\n\t\t\t\t\t\t\t}\n\t\t\t\t\t\t},\n\t\t\t\t\t\t\"queryTimeout\": \"02:00:00\",\n\t\t\t\t\t\t\"partitionOption\": \"None\"\n\t\t\t\t\t},\n\t\t\t\t\t\"dataset\": {\n\t\t\t\t\t\t\"referenceName\": \"GetSetMetadata\",\n\t\t\t\t\t\t\"type\": \"DatasetReference\"\n\t\t\t\t\t},\n\t\t\t\t\t\"firstRowOnly\": false\n\t\t\t\t}\n\t\t\t},\n\t\t\t{\n\t\t\t\t\"name\": \"Clean Up Previous Run\",\n\t\t\t\t\"description\": \"Handle Worker pipelines that are reported as Running when the parent pipeline is called again. Get what the actual status of those pipelines is.\",\n\t\t\t\t\"type\": \"ForEach\",\n\t\t\t\t\"dependsOn\": [\n\t\t\t\t\t{\n\t\t\t\t\t\t\"activity\": \"Check Previous Execution\",\n\t\t\t\t\t\t\"dependencyConditions\": [\n\t\t\t\t\t\t\t\"Succeeded\"\n\t\t\t\t\t\t]\n\t\t\t\t\t},\n\t\t\t\t\t{\n\t\t\t\t\t\t\"activity\": \"Check Metadata Integrity\",\n\t\t\t\t\t\t\"dependencyConditions\": [\n\t\t\t\t\t\t\t\"Succeeded\"\n\t\t\t\t\t\t]\n\t\t\t\t\t}\n\t\t\t\t],\n\t\t\t\t\"userProperties\": [],\n\t\t\t\t\"typeProperties\": {\n\t\t\t\t\t\"items\": {\n\t\t\t\t\t\t\"value\": \"@activity('Check Previous Execution').output.value\",\n\t\t\t\t\t\t\"type\": \"Expression\"\n\t\t\t\t\t},\n\t\t\t\t\t\"isSequential\": false,\n\t\t\t\t\t\"batchCount\": 50,\n\t\t\t\t\t\"activities\": [\n\t\t\t\t\t\t{\n\t\t\t\t\t\t\t\"name\": \"Get SPN Details\",\n\t\t\t\t\t\t\t\"type\": \"Lookup\",\n\t\t\t\t\t\t\t\"dependsOn\": [],\n\t\t\t\t\t\t\t\"policy\": {\n\t\t\t\t\t\t\t\t\"timeout\": \"0.00:10:00\",\n\t\t\t\t\t\t\t\t\"retry\": 0,\n\t\t\t\t\t\t\t\t\"retryIntervalInSeconds\": 30,\n\t\t\t\t\t\t\t\t\"secureOutput\": true,\n\t\t\t\t\t\t\t\t\"secureInput\": false\n\t\t\t\t\t\t\t},\n\t\t\t\t\t\t\t\"userProperties\": [],\n\t\t\t\t\t\t\t\"typeProperties\": {\n\t\t\t\t\t\t\t\t\"source\": {\n\t\t\t\t\t\t\t\t\t\"type\": \"AzureSqlSource\",\n\t\t\t\t\t\t\t\t\t\"sqlReaderStoredProcedureName\": \"[procfwk].[GetWorkerAuthDetails]\",\n\t\t\t\t\t\t\t\t\t\"storedProcedureParameters\": {\n\t\t\t\t\t\t\t\t\t\t\"ExecutionId\": {\n\t\t\t\t\t\t\t\t\t\t\t\"type\": \"Guid\",\n\t\t\t\t\t\t\t\t\t\t\t\"value\": {\n\t\t\t\t\t\t\t\t\t\t\t\t\"value\": \"@item().LocalExecutionId\",\n\t\t\t\t\t\t\t\t\t\t\t\t\"type\": \"Expression\"\n\t\t\t\t\t\t\t\t\t\t\t}\n\t\t\t\t\t\t\t\t\t\t},\n\t\t\t\t\t\t\t\t\t\t\"PipelineId\": {\n\t\t\t\t\t\t\t\t\t\t\t\"type\": \"Int32\",\n\t\t\t\t\t\t\t\t\t\t\t\"value\": {\n\t\t\t\t\t\t\t\t\t\t\t\t\"value\": \"@item().PipelineId\",\n\t\t\t\t\t\t\t\t\t\t\t\t\"type\": \"Expression\"\n\t\t\t\t\t\t\t\t\t\t\t}\n\t\t\t\t\t\t\t\t\t\t},\n\t\t\t\t\t\t\t\t\t\t\"StageId\": {\n\t\t\t\t\t\t\t\t\t\t\t\"type\": \"Int32\",\n\t\t\t\t\t\t\t\t\t\t\t\"value\": {\n\t\t\t\t\t\t\t\t\t\t\t\t\"value\": \"@item().StageId\",\n\t\t\t\t\t\t\t\t\t\t\t\t\"type\": \"Expression\"\n\t\t\t\t\t\t\t\t\t\t\t}\n\t\t\t\t\t\t\t\t\t\t}\n\t\t\t\t\t\t\t\t\t},\n\t\t\t\t\t\t\t\t\t\"queryTimeout\": \"02:00:00\",\n\t\t\t\t\t\t\t\t\t\"partitionOption\": \"None\"\n\t\t\t\t\t\t\t\t},\n\t\t\t\t\t\t\t\t\"dataset\": {\n\t\t\t\t\t\t\t\t\t\"referenceName\": \"GetSetMetadata\",\n\t\t\t\t\t\t\t\t\t\"type\": \"DatasetReference\"\n\t\t\t\t\t\t\t\t}\n\t\t\t\t\t\t\t}\n\t\t\t\t\t\t},\n\t\t\t\t\t\t{\n\t\t\t\t\t\t\t\"name\": \"Log Pipeline Checking\",\n\t\t\t\t\t\t\t\"type\": \"SqlServerStoredProcedure\",\n\t\t\t\t\t\t\t\"dependsOn\": [],\n\t\t\t\t\t\t\t\"policy\": {\n\t\t\t\t\t\t\t\t\"timeout\": \"0.00:10:00\",\n\t\t\t\t\t\t\t\t\"retry\": 0,\n\t\t\t\t\t\t\t\t\"retryIntervalInSeconds\": 30,\n\t\t\t\t\t\t\t\t\"secureOutput\": false,\n\t\t\t\t\t\t\t\t\"secureInput\": false\n\t\t\t\t\t\t\t},\n\t\t\t\t\t\t\t\"userProperties\": [],\n\t\t\t\t\t\t\t\"typeProperties\": {\n\t\t\t\t\t\t\t\t\"storedProcedureName\": \"[procfwk].[SetLogPipelineChecking]\",\n\t\t\t\t\t\t\t\t\"storedProcedureParameters\": {\n\t\t\t\t\t\t\t\t\t\"ExecutionId\": {\n\t\t\t\t\t\t\t\t\t\t\"value\": {\n\t\t\t\t\t\t\t\t\t\t\t\"value\": \"@item().LocalExecutionId\",\n\t\t\t\t\t\t\t\t\t\t\t\"type\": \"Expression\"\n\t\t\t\t\t\t\t\t\t\t},\n\t\t\t\t\t\t\t\t\t\t\"type\": \"Guid\"\n\t\t\t\t\t\t\t\t\t},\n\t\t\t\t\t\t\t\t\t\"PipelineId\": {\n\t\t\t\t\t\t\t\t\t\t\"value\": {\n\t\t\t\t\t\t\t\t\t\t\t\"value\": \"@item().PipelineId\",\n\t\t\t\t\t\t\t\t\t\t\t\"type\": \"Expression\"\n\t\t\t\t\t\t\t\t\t\t},\n\t\t\t\t\t\t\t\t\t\t\"type\": \"Int32\"\n\t\t\t\t\t\t\t\t\t},\n\t\t\t\t\t\t\t\t\t\"StageId\": {\n\t\t\t\t\t\t\t\t\t\t\"value\": {\n\t\t\t\t\t\t\t\t\t\t\t\"value\": \"@item().StageId\",\n\t\t\t\t\t\t\t\t\t\t\t\"type\": \"Expression\"\n\t\t\t\t\t\t\t\t\t\t},\n\t\t\t\t\t\t\t\t\t\t\"type\": \"Int32\"\n\t\t\t\t\t\t\t\t\t}\n\t\t\t\t\t\t\t\t}\n\t\t\t\t\t\t\t},\n\t\t\t\t\t\t\t\"linkedServiceName\": {\n\t\t\t\t\t\t\t\t\"referenceName\": \"SupportDatabase\",\n\t\t\t\t\t\t\t\t\"type\": \"LinkedServiceReference\"\n\t\t\t\t\t\t\t}\n\t\t\t\t\t\t},\n\t\t\t\t\t\t{\n\t\t\t\t\t\t\t\"name\": \"Get Pipeline Status\",\n\t\t\t\t\t\t\t\"type\": \"AzureFunctionActivity\",\n\t\t\t\t\t\t\t\"dependsOn\": [\n\t\t\t\t\t\t\t\t{\n\t\t\t\t\t\t\t\t\t\"activity\": \"Get SPN Details\",\n\t\t\t\t\t\t\t\t\t\"dependencyConditions\": [\n\t\t\t\t\t\t\t\t\t\t\"Succeeded\"\n\t\t\t\t\t\t\t\t\t]\n\t\t\t\t\t\t\t\t},\n\t\t\t\t\t\t\t\t{\n\t\t\t\t\t\t\t\t\t\"activity\": \"Log Pipeline Checking\",\n\t\t\t\t\t\t\t\t\t\"dependencyConditions\": [\n\t\t\t\t\t\t\t\t\t\t\"Succeeded\"\n\t\t\t\t\t\t\t\t\t]\n\t\t\t\t\t\t\t\t}\n\t\t\t\t\t\t\t],\n\t\t\t\t\t\t\t\"policy\": {\n\t\t\t\t\t\t\t\t\"timeout\": \"0.00:10:00\",\n\t\t\t\t\t\t\t\t\"retry\": 0,\n\t\t\t\t\t\t\t\t\"retryIntervalInSeconds\": 30,\n\t\t\t\t\t\t\t\t\"secureOutput\": false,\n\t\t\t\t\t\t\t\t\"secureInput\": true\n\t\t\t\t\t\t\t},\n\t\t\t\t\t\t\t\"userProperties\": [],\n\t\t\t\t\t\t\t\"typeProperties\": {\n\t\t\t\t\t\t\t\t\"functionName\": \"CheckPipelineStatus\",\n\t\t\t\t\t\t\t\t\"method\": \"POST\",\n\t\t\t\t\t\t\t\t\"body\": {\n\t\t\t\t\t\t\t\t\t\"value\": \"@concat('\\n{\\n \\\"tenantId\\\": \\\"',activity('Get SPN Details').output.firstRow.TenantId,'\\\",\\n \\\"applicationId\\\": \\\"',activity('Get SPN Details').output.firstRow.AppId,'\\\",\\n \\\"authenticationKey\\\": \\\"',activity('Get SPN Details').output.firstRow.AppSecret,'\\\",\\n \\\"subscriptionId\\\": \\\"',activity('Get SPN Details').output.firstRow.SubscriptionId,'\\\",\\n \\\"resourceGroupName\\\": \\\"',item().ResourceGroupName,'\\\",\\n \\\"orchestratorName\\\": \\\"',item().OrchestratorName,'\\\",\\n \\\"orchestratorType\\\": \\\"',item().OrchestratorType,'\\\",\\n \\\"pipelineName\\\": \\\"',item().PipelineName,'\\\",\\n \\\"runId\\\": \\\"',item().PipelineRunId,'\\\"\\n}')\",\n\t\t\t\t\t\t\t\t\t\"type\": \"Expression\"\n\t\t\t\t\t\t\t\t}\n\t\t\t\t\t\t\t},\n\t\t\t\t\t\t\t\"linkedServiceName\": {\n\t\t\t\t\t\t\t\t\"referenceName\": \"FrameworkFunctions\",\n\t\t\t\t\t\t\t\t\"type\": \"LinkedServiceReference\"\n\t\t\t\t\t\t\t}\n\t\t\t\t\t\t},\n\t\t\t\t\t\t{\n\t\t\t\t\t\t\t\"name\": \"Set Pipeline Status\",\n\t\t\t\t\t\t\t\"description\": \"Update the metadata depending on the actual pipeline outcome. Using the status as the case.\",\n\t\t\t\t\t\t\t\"type\": \"Switch\",\n\t\t\t\t\t\t\t\"dependsOn\": [\n\t\t\t\t\t\t\t\t{\n\t\t\t\t\t\t\t\t\t\"activity\": \"Get Pipeline Status\",\n\t\t\t\t\t\t\t\t\t\"dependencyConditions\": [\n\t\t\t\t\t\t\t\t\t\t\"Succeeded\"\n\t\t\t\t\t\t\t\t\t]\n\t\t\t\t\t\t\t\t}\n\t\t\t\t\t\t\t],\n\t\t\t\t\t\t\t\"userProperties\": [],\n\t\t\t\t\t\t\t\"typeProperties\": {\n\t\t\t\t\t\t\t\t\"on\": {\n\t\t\t\t\t\t\t\t\t\"value\": \"@activity('Get Pipeline Status').output.ActualStatus\",\n\t\t\t\t\t\t\t\t\t\"type\": \"Expression\"\n\t\t\t\t\t\t\t\t},\n\t\t\t\t\t\t\t\t\"cases\": [\n\t\t\t\t\t\t\t\t\t{\n\t\t\t\t\t\t\t\t\t\t\"value\": \"Failed\",\n\t\t\t\t\t\t\t\t\t\t\"activities\": [\n\t\t\t\t\t\t\t\t\t\t\t{\n\t\t\t\t\t\t\t\t\t\t\t\t\"name\": \"Pipeline Status Failed\",\n\t\t\t\t\t\t\t\t\t\t\t\t\"description\": \"Updates the current execution table with a pipeline status of failed if the function outcome is failed. Also blocks pipelines in the downstream execution stage.\",\n\t\t\t\t\t\t\t\t\t\t\t\t\"type\": \"SqlServerStoredProcedure\",\n\t\t\t\t\t\t\t\t\t\t\t\t\"dependsOn\": [],\n\t\t\t\t\t\t\t\t\t\t\t\t\"policy\": {\n\t\t\t\t\t\t\t\t\t\t\t\t\t\"timeout\": \"0.00:10:00\",\n\t\t\t\t\t\t\t\t\t\t\t\t\t\"retry\": 0,\n\t\t\t\t\t\t\t\t\t\t\t\t\t\"retryIntervalInSeconds\": 30,\n\t\t\t\t\t\t\t\t\t\t\t\t\t\"secureOutput\": false,\n\t\t\t\t\t\t\t\t\t\t\t\t\t\"secureInput\": false\n\t\t\t\t\t\t\t\t\t\t\t\t},\n\t\t\t\t\t\t\t\t\t\t\t\t\"userProperties\": [],\n\t\t\t\t\t\t\t\t\t\t\t\t\"typeProperties\": {\n\t\t\t\t\t\t\t\t\t\t\t\t\t\"storedProcedureName\": \"[procfwk].[SetLogPipelineFailed]\",\n\t\t\t\t\t\t\t\t\t\t\t\t\t\"storedProcedureParameters\": {\n\t\t\t\t\t\t\t\t\t\t\t\t\t\t\"ExecutionId\": {\n\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\"value\": {\n\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\"value\": \"@item().LocalExecutionId\",\n\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\"type\": \"Expression\"\n\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t},\n\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\"type\": \"Guid\"\n\t\t\t\t\t\t\t\t\t\t\t\t\t\t},\n\t\t\t\t\t\t\t\t\t\t\t\t\t\t\"PipelineId\": {\n\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\"value\": {\n\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\"value\": \"@item().PipelineId\",\n\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\"type\": \"Expression\"\n\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t},\n\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\"type\": \"Int32\"\n\t\t\t\t\t\t\t\t\t\t\t\t\t\t},\n\t\t\t\t\t\t\t\t\t\t\t\t\t\t\"RunId\": {\n\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\"value\": null,\n\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\"type\": \"Guid\"\n\t\t\t\t\t\t\t\t\t\t\t\t\t\t},\n\t\t\t\t\t\t\t\t\t\t\t\t\t\t\"StageId\": {\n\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\"value\": {\n\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\"value\": \"@item().StageId\",\n\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\"type\": \"Expression\"\n\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t},\n\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\"type\": \"Int32\"\n\t\t\t\t\t\t\t\t\t\t\t\t\t\t}\n\t\t\t\t\t\t\t\t\t\t\t\t\t}\n\t\t\t\t\t\t\t\t\t\t\t\t},\n\t\t\t\t\t\t\t\t\t\t\t\t\"linkedServiceName\": {\n\t\t\t\t\t\t\t\t\t\t\t\t\t\"referenceName\": \"SupportDatabase\",\n\t\t\t\t\t\t\t\t\t\t\t\t\t\"type\": \"LinkedServiceReference\"\n\t\t\t\t\t\t\t\t\t\t\t\t}\n\t\t\t\t\t\t\t\t\t\t\t}\n\t\t\t\t\t\t\t\t\t\t]\n\t\t\t\t\t\t\t\t\t},\n\t\t\t\t\t\t\t\t\t{\n\t\t\t\t\t\t\t\t\t\t\"value\": \"Succeeded\",\n\t\t\t\t\t\t\t\t\t\t\"activities\": [\n\t\t\t\t\t\t\t\t\t\t\t{\n\t\t\t\t\t\t\t\t\t\t\t\t\"name\": \"Pipeline Status Succeeded\",\n\t\t\t\t\t\t\t\t\t\t\t\t\"description\": \"Updates the current execution table with a pipeline status of success if the function outcome is succeeded.\",\n\t\t\t\t\t\t\t\t\t\t\t\t\"type\": \"SqlServerStoredProcedure\",\n\t\t\t\t\t\t\t\t\t\t\t\t\"dependsOn\": [],\n\t\t\t\t\t\t\t\t\t\t\t\t\"policy\": {\n\t\t\t\t\t\t\t\t\t\t\t\t\t\"timeout\": \"0.00:10:00\",\n\t\t\t\t\t\t\t\t\t\t\t\t\t\"retry\": 0,\n\t\t\t\t\t\t\t\t\t\t\t\t\t\"retryIntervalInSeconds\": 30,\n\t\t\t\t\t\t\t\t\t\t\t\t\t\"secureOutput\": false,\n\t\t\t\t\t\t\t\t\t\t\t\t\t\"secureInput\": false\n\t\t\t\t\t\t\t\t\t\t\t\t},\n\t\t\t\t\t\t\t\t\t\t\t\t\"userProperties\": [],\n\t\t\t\t\t\t\t\t\t\t\t\t\"typeProperties\": {\n\t\t\t\t\t\t\t\t\t\t\t\t\t\"storedProcedureName\": \"[procfwk].[SetLogPipelineSuccess]\",\n\t\t\t\t\t\t\t\t\t\t\t\t\t\"storedProcedureParameters\": {\n\t\t\t\t\t\t\t\t\t\t\t\t\t\t\"ExecutionId\": {\n\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\"value\": {\n\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\"value\": \"@item().LocalExecutionId\",\n\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\"type\": \"Expression\"\n\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t},\n\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\"type\": \"Guid\"\n\t\t\t\t\t\t\t\t\t\t\t\t\t\t},\n\t\t\t\t\t\t\t\t\t\t\t\t\t\t\"PipelineId\": {\n\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\"value\": {\n\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\"value\": \"@item().PipelineId\",\n\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\"type\": \"Expression\"\n\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t},\n\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\"type\": \"Int32\"\n\t\t\t\t\t\t\t\t\t\t\t\t\t\t},\n\t\t\t\t\t\t\t\t\t\t\t\t\t\t\"StageId\": {\n\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\"value\": {\n\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\"value\": \"@item().StageId\",\n\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\"type\": \"Expression\"\n\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t},\n\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\"type\": \"Int32\"\n\t\t\t\t\t\t\t\t\t\t\t\t\t\t}\n\t\t\t\t\t\t\t\t\t\t\t\t\t}\n\t\t\t\t\t\t\t\t\t\t\t\t},\n\t\t\t\t\t\t\t\t\t\t\t\t\"linkedServiceName\": {\n\t\t\t\t\t\t\t\t\t\t\t\t\t\"referenceName\": \"SupportDatabase\",\n\t\t\t\t\t\t\t\t\t\t\t\t\t\"type\": \"LinkedServiceReference\"\n\t\t\t\t\t\t\t\t\t\t\t\t}\n\t\t\t\t\t\t\t\t\t\t\t}\n\t\t\t\t\t\t\t\t\t\t]\n\t\t\t\t\t\t\t\t\t},\n\t\t\t\t\t\t\t\t\t{\n\t\t\t\t\t\t\t\t\t\t\"value\": \"Queued\",\n\t\t\t\t\t\t\t\t\t\t\"activities\": [\n\t\t\t\t\t\t\t\t\t\t\t{\n\t\t\t\t\t\t\t\t\t\t\t\t\"name\": \"Pipeline Status Queued - Running\",\n\t\t\t\t\t\t\t\t\t\t\t\t\"description\": \"Updates the current execution table with a pipeline status of running if the function outcome is queued.\",\n\t\t\t\t\t\t\t\t\t\t\t\t\"type\": \"SqlServerStoredProcedure\",\n\t\t\t\t\t\t\t\t\t\t\t\t\"dependsOn\": [],\n\t\t\t\t\t\t\t\t\t\t\t\t\"policy\": {\n\t\t\t\t\t\t\t\t\t\t\t\t\t\"timeout\": \"0.00:10:00\",\n\t\t\t\t\t\t\t\t\t\t\t\t\t\"retry\": 0,\n\t\t\t\t\t\t\t\t\t\t\t\t\t\"retryIntervalInSeconds\": 30,\n\t\t\t\t\t\t\t\t\t\t\t\t\t\"secureOutput\": false,\n\t\t\t\t\t\t\t\t\t\t\t\t\t\"secureInput\": false\n\t\t\t\t\t\t\t\t\t\t\t\t},\n\t\t\t\t\t\t\t\t\t\t\t\t\"userProperties\": [],\n\t\t\t\t\t\t\t\t\t\t\t\t\"typeProperties\": {\n\t\t\t\t\t\t\t\t\t\t\t\t\t\"storedProcedureName\": \"[procfwk].[SetLogPipelineRunning]\",\n\t\t\t\t\t\t\t\t\t\t\t\t\t\"storedProcedureParameters\": {\n\t\t\t\t\t\t\t\t\t\t\t\t\t\t\"ExecutionId\": {\n\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\"value\": {\n\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\"value\": \"@item().LocalExecutionId\",\n\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\"type\": \"Expression\"\n\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t},\n\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\"type\": \"Guid\"\n\t\t\t\t\t\t\t\t\t\t\t\t\t\t},\n\t\t\t\t\t\t\t\t\t\t\t\t\t\t\"PipelineId\": {\n\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\"value\": {\n\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\"value\": \"@item().PipelineId\",\n\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\"type\": \"Expression\"\n\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t},\n\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\"type\": \"Int32\"\n\t\t\t\t\t\t\t\t\t\t\t\t\t\t},\n\t\t\t\t\t\t\t\t\t\t\t\t\t\t\"StageId\": {\n\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\"value\": {\n\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\"value\": \"@item().StageId\",\n\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\"type\": \"Expression\"\n\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t},\n\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\"type\": \"Int32\"\n\t\t\t\t\t\t\t\t\t\t\t\t\t\t}\n\t\t\t\t\t\t\t\t\t\t\t\t\t}\n\t\t\t\t\t\t\t\t\t\t\t\t},\n\t\t\t\t\t\t\t\t\t\t\t\t\"linkedServiceName\": {\n\t\t\t\t\t\t\t\t\t\t\t\t\t\"referenceName\": \"SupportDatabase\",\n\t\t\t\t\t\t\t\t\t\t\t\t\t\"type\": \"LinkedServiceReference\"\n\t\t\t\t\t\t\t\t\t\t\t\t}\n\t\t\t\t\t\t\t\t\t\t\t}\n\t\t\t\t\t\t\t\t\t\t]\n\t\t\t\t\t\t\t\t\t},\n\t\t\t\t\t\t\t\t\t{\n\t\t\t\t\t\t\t\t\t\t\"value\": \"InProgress\",\n\t\t\t\t\t\t\t\t\t\t\"activities\": [\n\t\t\t\t\t\t\t\t\t\t\t{\n\t\t\t\t\t\t\t\t\t\t\t\t\"name\": \"Pipeline Status InProgress - Running\",\n\t\t\t\t\t\t\t\t\t\t\t\t\"description\": \"Updates the current execution table with a pipeline status of running if the function outcome is in progress.\",\n\t\t\t\t\t\t\t\t\t\t\t\t\"type\": \"SqlServerStoredProcedure\",\n\t\t\t\t\t\t\t\t\t\t\t\t\"dependsOn\": [],\n\t\t\t\t\t\t\t\t\t\t\t\t\"policy\": {\n\t\t\t\t\t\t\t\t\t\t\t\t\t\"timeout\": \"0.00:10:00\",\n\t\t\t\t\t\t\t\t\t\t\t\t\t\"retry\": 0,\n\t\t\t\t\t\t\t\t\t\t\t\t\t\"retryIntervalInSeconds\": 30,\n\t\t\t\t\t\t\t\t\t\t\t\t\t\"secureOutput\": false,\n\t\t\t\t\t\t\t\t\t\t\t\t\t\"secureInput\": false\n\t\t\t\t\t\t\t\t\t\t\t\t},\n\t\t\t\t\t\t\t\t\t\t\t\t\"userProperties\": [],\n\t\t\t\t\t\t\t\t\t\t\t\t\"typeProperties\": {\n\t\t\t\t\t\t\t\t\t\t\t\t\t\"storedProcedureName\": \"[procfwk].[SetLogPipelineRunning]\",\n\t\t\t\t\t\t\t\t\t\t\t\t\t\"storedProcedureParameters\": {\n\t\t\t\t\t\t\t\t\t\t\t\t\t\t\"ExecutionId\": {\n\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\"value\": {\n\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\"value\": \"@item().LocalExecutionId\",\n\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\"type\": \"Expression\"\n\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t},\n\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\"type\": \"Guid\"\n\t\t\t\t\t\t\t\t\t\t\t\t\t\t},\n\t\t\t\t\t\t\t\t\t\t\t\t\t\t\"PipelineId\": {\n\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\"value\": {\n\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\"value\": \"@item().PipelineId\",\n\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\"type\": \"Expression\"\n\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t},\n\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\"type\": \"Int32\"\n\t\t\t\t\t\t\t\t\t\t\t\t\t\t},\n\t\t\t\t\t\t\t\t\t\t\t\t\t\t\"StageId\": {\n\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\"value\": {\n\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\"value\": \"@item().StageId\",\n\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\"type\": \"Expression\"\n\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t},\n\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\"type\": \"Int32\"\n\t\t\t\t\t\t\t\t\t\t\t\t\t\t}\n\t\t\t\t\t\t\t\t\t\t\t\t\t}\n\t\t\t\t\t\t\t\t\t\t\t\t},\n\t\t\t\t\t\t\t\t\t\t\t\t\"linkedServiceName\": {\n\t\t\t\t\t\t\t\t\t\t\t\t\t\"referenceName\": \"SupportDatabase\",\n\t\t\t\t\t\t\t\t\t\t\t\t\t\"type\": \"LinkedServiceReference\"\n\t\t\t\t\t\t\t\t\t\t\t\t}\n\t\t\t\t\t\t\t\t\t\t\t}\n\t\t\t\t\t\t\t\t\t\t]\n\t\t\t\t\t\t\t\t\t},\n\t\t\t\t\t\t\t\t\t{\n\t\t\t\t\t\t\t\t\t\t\"value\": \"Cancelled\",\n\t\t\t\t\t\t\t\t\t\t\"activities\": [\n\t\t\t\t\t\t\t\t\t\t\t{\n\t\t\t\t\t\t\t\t\t\t\t\t\"name\": \"Pipeline Status Cancelled\",\n\t\t\t\t\t\t\t\t\t\t\t\t\"description\": \"Updates the current execution table with a pipeline status of cancelled if the function outcome is cancelled.\",\n\t\t\t\t\t\t\t\t\t\t\t\t\"type\": \"SqlServerStoredProcedure\",\n\t\t\t\t\t\t\t\t\t\t\t\t\"dependsOn\": [],\n\t\t\t\t\t\t\t\t\t\t\t\t\"policy\": {\n\t\t\t\t\t\t\t\t\t\t\t\t\t\"timeout\": \"0.00:10:00\",\n\t\t\t\t\t\t\t\t\t\t\t\t\t\"retry\": 0,\n\t\t\t\t\t\t\t\t\t\t\t\t\t\"retryIntervalInSeconds\": 30,\n\t\t\t\t\t\t\t\t\t\t\t\t\t\"secureOutput\": false,\n\t\t\t\t\t\t\t\t\t\t\t\t\t\"secureInput\": false\n\t\t\t\t\t\t\t\t\t\t\t\t},\n\t\t\t\t\t\t\t\t\t\t\t\t\"userProperties\": [],\n\t\t\t\t\t\t\t\t\t\t\t\t\"typeProperties\": {\n\t\t\t\t\t\t\t\t\t\t\t\t\t\"storedProcedureName\": \"[procfwk].[SetLogPipelineCancelled]\",\n\t\t\t\t\t\t\t\t\t\t\t\t\t\"storedProcedureParameters\": {\n\t\t\t\t\t\t\t\t\t\t\t\t\t\t\"ExecutionId\": {\n\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\"value\": {\n\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\"value\": \"@item().LocalExecutionId\",\n\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\"type\": \"Expression\"\n\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t},\n\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\"type\": \"Guid\"\n\t\t\t\t\t\t\t\t\t\t\t\t\t\t},\n\t\t\t\t\t\t\t\t\t\t\t\t\t\t\"PipelineId\": {\n\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\"value\": {\n\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\"value\": \"@item().PipelineId\",\n\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\"type\": \"Expression\"\n\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t},\n\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\"type\": \"Int32\"\n\t\t\t\t\t\t\t\t\t\t\t\t\t\t},\n\t\t\t\t\t\t\t\t\t\t\t\t\t\t\"StageId\": {\n\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\"value\": {\n\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\"value\": \"@item().StageId\",\n\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\"type\": \"Expression\"\n\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t},\n\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\"type\": \"Int32\"\n\t\t\t\t\t\t\t\t\t\t\t\t\t\t},\n\t\t\t\t\t\t\t\t\t\t\t\t\t\t\"CleanUpRun\": {\n\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\"value\": {\n\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\"value\": \"@bool(1)\",\n\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\"type\": \"Expression\"\n\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t},\n\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\"type\": \"Boolean\"\n\t\t\t\t\t\t\t\t\t\t\t\t\t\t}\n\t\t\t\t\t\t\t\t\t\t\t\t\t}\n\t\t\t\t\t\t\t\t\t\t\t\t},\n\t\t\t\t\t\t\t\t\t\t\t\t\"linkedServiceName\": {\n\t\t\t\t\t\t\t\t\t\t\t\t\t\"referenceName\": \"SupportDatabase\",\n\t\t\t\t\t\t\t\t\t\t\t\t\t\"type\": \"LinkedServiceReference\"\n\t\t\t\t\t\t\t\t\t\t\t\t}\n\t\t\t\t\t\t\t\t\t\t\t}\n\t\t\t\t\t\t\t\t\t\t]\n\t\t\t\t\t\t\t\t\t}\n\t\t\t\t\t\t\t\t],\n\t\t\t\t\t\t\t\t\"defaultActivities\": [\n\t\t\t\t\t\t\t\t\t{\n\t\t\t\t\t\t\t\t\t\t\"name\": \"Pipeline Status Unknown\",\n\t\t\t\t\t\t\t\t\t\t\"description\": \"Updates the current execution table with a pipeline status of unknown if the function returns an unexpected outcome.\",\n\t\t\t\t\t\t\t\t\t\t\"type\": \"SqlServerStoredProcedure\",\n\t\t\t\t\t\t\t\t\t\t\"dependsOn\": [],\n\t\t\t\t\t\t\t\t\t\t\"policy\": {\n\t\t\t\t\t\t\t\t\t\t\t\"timeout\": \"0.00:10:00\",\n\t\t\t\t\t\t\t\t\t\t\t\"retry\": 0,\n\t\t\t\t\t\t\t\t\t\t\t\"retryIntervalInSeconds\": 30,\n\t\t\t\t\t\t\t\t\t\t\t\"secureOutput\": false,\n\t\t\t\t\t\t\t\t\t\t\t\"secureInput\": false\n\t\t\t\t\t\t\t\t\t\t},\n\t\t\t\t\t\t\t\t\t\t\"userProperties\": [],\n\t\t\t\t\t\t\t\t\t\t\"typeProperties\": {\n\t\t\t\t\t\t\t\t\t\t\t\"storedProcedureName\": \"[procfwk].[SetLogPipelineUnknown]\",\n\t\t\t\t\t\t\t\t\t\t\t\"storedProcedureParameters\": {\n\t\t\t\t\t\t\t\t\t\t\t\t\"ExecutionId\": {\n\t\t\t\t\t\t\t\t\t\t\t\t\t\"value\": {\n\t\t\t\t\t\t\t\t\t\t\t\t\t\t\"value\": \"@item().LocalExecutionId\",\n\t\t\t\t\t\t\t\t\t\t\t\t\t\t\"type\": \"Expression\"\n\t\t\t\t\t\t\t\t\t\t\t\t\t},\n\t\t\t\t\t\t\t\t\t\t\t\t\t\"type\": \"Guid\"\n\t\t\t\t\t\t\t\t\t\t\t\t},\n\t\t\t\t\t\t\t\t\t\t\t\t\"PipelineId\": {\n\t\t\t\t\t\t\t\t\t\t\t\t\t\"value\": {\n\t\t\t\t\t\t\t\t\t\t\t\t\t\t\"value\": \"@item().PipelineId\",\n\t\t\t\t\t\t\t\t\t\t\t\t\t\t\"type\": \"Expression\"\n\t\t\t\t\t\t\t\t\t\t\t\t\t},\n\t\t\t\t\t\t\t\t\t\t\t\t\t\"type\": \"Int32\"\n\t\t\t\t\t\t\t\t\t\t\t\t},\n\t\t\t\t\t\t\t\t\t\t\t\t\"StageId\": {\n\t\t\t\t\t\t\t\t\t\t\t\t\t\"value\": {\n\t\t\t\t\t\t\t\t\t\t\t\t\t\t\"value\": \"@item().StageId\",\n\t\t\t\t\t\t\t\t\t\t\t\t\t\t\"type\": \"Expression\"\n\t\t\t\t\t\t\t\t\t\t\t\t\t},\n\t\t\t\t\t\t\t\t\t\t\t\t\t\"type\": \"Int32\"\n\t\t\t\t\t\t\t\t\t\t\t\t},\n\t\t\t\t\t\t\t\t\t\t\t\t\"CleanUpRun\": {\n\t\t\t\t\t\t\t\t\t\t\t\t\t\"value\": {\n\t\t\t\t\t\t\t\t\t\t\t\t\t\t\"value\": \"@bool(1)\",\n\t\t\t\t\t\t\t\t\t\t\t\t\t\t\"type\": \"Expression\"\n\t\t\t\t\t\t\t\t\t\t\t\t\t},\n\t\t\t\t\t\t\t\t\t\t\t\t\t\"type\": \"Boolean\"\n\t\t\t\t\t\t\t\t\t\t\t\t}\n\t\t\t\t\t\t\t\t\t\t\t}\n\t\t\t\t\t\t\t\t\t\t},\n\t\t\t\t\t\t\t\t\t\t\"linkedServiceName\": {\n\t\t\t\t\t\t\t\t\t\t\t\"referenceName\": \"SupportDatabase\",\n\t\t\t\t\t\t\t\t\t\t\t\"type\": \"LinkedServiceReference\"\n\t\t\t\t\t\t\t\t\t\t}\n\t\t\t\t\t\t\t\t\t}\n\t\t\t\t\t\t\t\t]\n\t\t\t\t\t\t\t}\n\t\t\t\t\t\t},\n\t\t\t\t\t\t{\n\t\t\t\t\t\t\t\"name\": \"Set Last Check DateTime\",\n\t\t\t\t\t\t\t\"description\": \"Update the current execution table with a date time from when the function last checked the pipeline status.\",\n\t\t\t\t\t\t\t\"type\": \"SqlServerStoredProcedure\",\n\t\t\t\t\t\t\t\"dependsOn\": [\n\t\t\t\t\t\t\t\t{\n\t\t\t\t\t\t\t\t\t\"activity\": \"Get Pipeline Status\",\n\t\t\t\t\t\t\t\t\t\"dependencyConditions\": [\n\t\t\t\t\t\t\t\t\t\t\"Succeeded\"\n\t\t\t\t\t\t\t\t\t]\n\t\t\t\t\t\t\t\t}\n\t\t\t\t\t\t\t],\n\t\t\t\t\t\t\t\"policy\": {\n\t\t\t\t\t\t\t\t\"timeout\": \"0.00:10:00\",\n\t\t\t\t\t\t\t\t\"retry\": 0,\n\t\t\t\t\t\t\t\t\"retryIntervalInSeconds\": 30,\n\t\t\t\t\t\t\t\t\"secureOutput\": false,\n\t\t\t\t\t\t\t\t\"secureInput\": false\n\t\t\t\t\t\t\t},\n\t\t\t\t\t\t\t\"userProperties\": [],\n\t\t\t\t\t\t\t\"typeProperties\": {\n\t\t\t\t\t\t\t\t\"storedProcedureName\": \"[procfwk].[SetLogPipelineLastStatusCheck]\",\n\t\t\t\t\t\t\t\t\"storedProcedureParameters\": {\n\t\t\t\t\t\t\t\t\t\"ExecutionId\": {\n\t\t\t\t\t\t\t\t\t\t\"value\": {\n\t\t\t\t\t\t\t\t\t\t\t\"value\": \"@item().LocalExecutionId\",\n\t\t\t\t\t\t\t\t\t\t\t\"type\": \"Expression\"\n\t\t\t\t\t\t\t\t\t\t},\n\t\t\t\t\t\t\t\t\t\t\"type\": \"Guid\"\n\t\t\t\t\t\t\t\t\t},\n\t\t\t\t\t\t\t\t\t\"PipelineId\": {\n\t\t\t\t\t\t\t\t\t\t\"value\": {\n\t\t\t\t\t\t\t\t\t\t\t\"value\": \"@item().PipelineId\",\n\t\t\t\t\t\t\t\t\t\t\t\"type\": \"Expression\"\n\t\t\t\t\t\t\t\t\t\t},\n\t\t\t\t\t\t\t\t\t\t\"type\": \"Int32\"\n\t\t\t\t\t\t\t\t\t},\n\t\t\t\t\t\t\t\t\t\"StageId\": {\n\t\t\t\t\t\t\t\t\t\t\"value\": {\n\t\t\t\t\t\t\t\t\t\t\t\"value\": \"@item().StageId\",\n\t\t\t\t\t\t\t\t\t\t\t\"type\": \"Expression\"\n\t\t\t\t\t\t\t\t\t\t},\n\t\t\t\t\t\t\t\t\t\t\"type\": \"Int32\"\n\t\t\t\t\t\t\t\t\t}\n\t\t\t\t\t\t\t\t}\n\t\t\t\t\t\t\t},\n\t\t\t\t\t\t\t\"linkedServiceName\": {\n\t\t\t\t\t\t\t\t\"referenceName\": \"SupportDatabase\",\n\t\t\t\t\t\t\t\t\"type\": \"LinkedServiceReference\"\n\t\t\t\t\t\t\t}\n\t\t\t\t\t\t}\n\t\t\t\t\t]\n\t\t\t\t}\n\t\t\t},\n\t\t\t{\n\t\t\t\t\"name\": \"Execute Precursor\",\n\t\t\t\t\"description\": \"Uses the database property value ExecutionPrecursorProc to run any custom logic against the metadata database before the execution run starts.\",\n\t\t\t\t\"type\": \"SqlServerStoredProcedure\",\n\t\t\t\t\"dependsOn\": [\n\t\t\t\t\t{\n\t\t\t\t\t\t\"activity\": \"Is Parent Already Running\",\n\t\t\t\t\t\t\"dependencyConditions\": [\n\t\t\t\t\t\t\t\"Succeeded\"\n\t\t\t\t\t\t]\n\t\t\t\t\t}\n\t\t\t\t],\n\t\t\t\t\"policy\": {\n\t\t\t\t\t\"timeout\": \"0.00:10:00\",\n\t\t\t\t\t\"retry\": 0,\n\t\t\t\t\t\"retryIntervalInSeconds\": 30,\n\t\t\t\t\t\"secureOutput\": false,\n\t\t\t\t\t\"secureInput\": false\n\t\t\t\t},\n\t\t\t\t\"userProperties\": [],\n\t\t\t\t\"typeProperties\": {\n\t\t\t\t\t\"storedProcedureName\": \"[procfwk].[ExecutePrecursorProcedure]\"\n\t\t\t\t},\n\t\t\t\t\"linkedServiceName\": {\n\t\t\t\t\t\"referenceName\": \"SupportDatabase\",\n\t\t\t\t\t\"type\": \"LinkedServiceReference\"\n\t\t\t\t}\n\t\t\t},\n\t\t\t{\n\t\t\t\t\"name\": \"Set Execution Id\",\n\t\t\t\t\"description\": \"Set the local execution Id to a pipeline variable for each in several downstream activities.\",\n\t\t\t\t\"type\": \"SetVariable\",\n\t\t\t\t\"dependsOn\": [\n\t\t\t\t\t{\n\t\t\t\t\t\t\"activity\": \"Execution Wrapper\",\n\t\t\t\t\t\t\"dependencyConditions\": [\n\t\t\t\t\t\t\t\"Succeeded\"\n\t\t\t\t\t\t]\n\t\t\t\t\t}\n\t\t\t\t],\n\t\t\t\t\"userProperties\": [],\n\t\t\t\t\"typeProperties\": {\n\t\t\t\t\t\"variableName\": \"ExecutionId\",\n\t\t\t\t\t\"value\": {\n\t\t\t\t\t\t\"value\": \"@activity('Execution Wrapper').output.firstRow.ExecutionId\",\n\t\t\t\t\t\t\"type\": \"Expression\"\n\t\t\t\t\t}\n\t\t\t\t}\n\t\t\t},\n\t\t\t{\n\t\t\t\t\"name\": \"Is Parent Already Running\",\n\t\t\t\t\"description\": \"Establish before anything else if the parent pipeline is already running. Batch execution aware.\",\n\t\t\t\t\"type\": \"ExecutePipeline\",\n\t\t\t\t\"dependsOn\": [],\n\t\t\t\t\"userProperties\": [],\n\t\t\t\t\"typeProperties\": {\n\t\t\t\t\t\"pipeline\": {\n\t\t\t\t\t\t\"referenceName\": \"Check For Running Pipeline\",\n\t\t\t\t\t\t\"type\": \"PipelineReference\"\n\t\t\t\t\t},\n\t\t\t\t\t\"waitOnCompletion\": true,\n\t\t\t\t\t\"parameters\": {\n\t\t\t\t\t\t\"BatchName\": {\n\t\t\t\t\t\t\t\"value\": \"@pipeline().parameters.BatchName\",\n\t\t\t\t\t\t\t\"type\": \"Expression\"\n\t\t\t\t\t\t},\n\t\t\t\t\t\t\"PipelineName\": {\n\t\t\t\t\t\t\t\"value\": \"@pipeline().Pipeline\",\n\t\t\t\t\t\t\t\"type\": \"Expression\"\n\t\t\t\t\t\t},\n\t\t\t\t\t\t\"ThisRunId\": {\n\t\t\t\t\t\t\t\"value\": \"@pipeline().RunId\",\n\t\t\t\t\t\t\t\"type\": \"Expression\"\n\t\t\t\t\t\t}\n\t\t\t\t\t}\n\t\t\t\t}\n\t\t\t},\n\t\t\t{\n\t\t\t\t\"name\": \"Check Metadata Integrity\",\n\t\t\t\t\"description\": \"Performs a series of checks on all metadata held in the framework SQLDB. This is intended to raise errors before an execution run even starts.\",\n\t\t\t\t\"type\": \"SqlServerStoredProcedure\",\n\t\t\t\t\"dependsOn\": [\n\t\t\t\t\t{\n\t\t\t\t\t\t\"activity\": \"Execute Precursor\",\n\t\t\t\t\t\t\"dependencyConditions\": [\n\t\t\t\t\t\t\t\"Succeeded\"\n\t\t\t\t\t\t]\n\t\t\t\t\t}\n\t\t\t\t],\n\t\t\t\t\"policy\": {\n\t\t\t\t\t\"timeout\": \"0.00:10:00\",\n\t\t\t\t\t\"retry\": 0,\n\t\t\t\t\t\"retryIntervalInSeconds\": 30,\n\t\t\t\t\t\"secureOutput\": false,\n\t\t\t\t\t\"secureInput\": false\n\t\t\t\t},\n\t\t\t\t\"userProperties\": [],\n\t\t\t\t\"typeProperties\": {\n\t\t\t\t\t\"storedProcedureName\": \"[procfwk].[CheckMetadataIntegrity]\",\n\t\t\t\t\t\"storedProcedureParameters\": {\n\t\t\t\t\t\t\"BatchName\": {\n\t\t\t\t\t\t\t\"value\": {\n\t\t\t\t\t\t\t\t\"value\": \"@pipeline().parameters.BatchName\",\n\t\t\t\t\t\t\t\t\"type\": \"Expression\"\n\t\t\t\t\t\t\t},\n\t\t\t\t\t\t\t\"type\": \"String\"\n\t\t\t\t\t\t},\n\t\t\t\t\t\t\"DebugMode\": {\n\t\t\t\t\t\t\t\"value\": {\n\t\t\t\t\t\t\t\t\"value\": \"@bool(0)\",\n\t\t\t\t\t\t\t\t\"type\": \"Expression\"\n\t\t\t\t\t\t\t},\n\t\t\t\t\t\t\t\"type\": \"Boolean\"\n\t\t\t\t\t\t}\n\t\t\t\t\t}\n\t\t\t\t},\n\t\t\t\t\"linkedServiceName\": {\n\t\t\t\t\t\"referenceName\": \"SupportDatabase\",\n\t\t\t\t\t\"type\": \"LinkedServiceReference\"\n\t\t\t\t}\n\t\t\t}\n\t\t],\n\t\t\"parameters\": {\n\t\t\t\"BatchName\": {\n\t\t\t\t\"type\": \"string\",\n\t\t\t\t\"defaultValue\": \"NotUsed\"\n\t\t\t}\n\t\t},\n\t\t\"variables\": {\n\t\t\t\"ExecutionId\": {\n\t\t\t\t\"type\": \"String\"\n\t\t\t}\n\t\t},\n\t\t\"folder\": {\n\t\t\t\"name\": \"_ProcFwk\"\n\t\t},\n\t\t\"annotations\": [\n\t\t\t\"procfwk\",\n\t\t\t\"Parent\"\n\t\t]\n\t}\n}"
},
{
"path": "DataFactory/pipeline/03-Child.json",
"content": "{\n\t\"name\": \"03-Child\",\n\t\"properties\": {\n\t\t\"description\": \"procfwk child pipeline used to execute Worker pipelines within a given execution stage. This pipeline will be called once for each stage, then execute all Workers in parallel.\",\n\t\t\"activities\": [\n\t\t\t{\n\t\t\t\t\"name\": \"Get Pipelines\",\n\t\t\t\t\"description\": \"Returns all pipelines from the metadata to be executed within a given processing stage.\",\n\t\t\t\t\"type\": \"Lookup\",\n\t\t\t\t\"dependsOn\": [],\n\t\t\t\t\"policy\": {\n\t\t\t\t\t\"timeout\": \"0.00:10:00\",\n\t\t\t\t\t\"retry\": 0,\n\t\t\t\t\t\"retryIntervalInSeconds\": 30,\n\t\t\t\t\t\"secureOutput\": false,\n\t\t\t\t\t\"secureInput\": false\n\t\t\t\t},\n\t\t\t\t\"userProperties\": [],\n\t\t\t\t\"typeProperties\": {\n\t\t\t\t\t\"source\": {\n\t\t\t\t\t\t\"type\": \"AzureSqlSource\",\n\t\t\t\t\t\t\"sqlReaderStoredProcedureName\": \"[procfwk].[GetPipelinesInStage]\",\n\t\t\t\t\t\t\"storedProcedureParameters\": {\n\t\t\t\t\t\t\t\"ExecutionId\": {\n\t\t\t\t\t\t\t\t\"type\": \"Guid\",\n\t\t\t\t\t\t\t\t\"value\": {\n\t\t\t\t\t\t\t\t\t\"value\": \"@pipeline().parameters.ExecutionId\",\n\t\t\t\t\t\t\t\t\t\"type\": \"Expression\"\n\t\t\t\t\t\t\t\t}\n\t\t\t\t\t\t\t},\n\t\t\t\t\t\t\t\"StageId\": {\n\t\t\t\t\t\t\t\t\"type\": \"Int32\",\n\t\t\t\t\t\t\t\t\"value\": {\n\t\t\t\t\t\t\t\t\t\"value\": \"@pipeline().parameters.StageId\",\n\t\t\t\t\t\t\t\t\t\"type\": \"Expression\"\n\t\t\t\t\t\t\t\t}\n\t\t\t\t\t\t\t}\n\t\t\t\t\t\t},\n\t\t\t\t\t\t\"queryTimeout\": \"02:00:00\",\n\t\t\t\t\t\t\"partitionOption\": \"None\"\n\t\t\t\t\t},\n\t\t\t\t\t\"dataset\": {\n\t\t\t\t\t\t\"referenceName\": \"GetSetMetadata\",\n\t\t\t\t\t\t\"type\": \"DatasetReference\"\n\t\t\t\t\t},\n\t\t\t\t\t\"firstRowOnly\": false\n\t\t\t\t}\n\t\t\t},\n\t\t\t{\n\t\t\t\t\"name\": \"Execute Pipelines\",\n\t\t\t\t\"description\": \"Second level ForEach to run in parallel all pipelines within the stage. Items for iteration passed from the Get Pipelines lookup activity.\",\n\t\t\t\t\"type\": \"ForEach\",\n\t\t\t\t\"dependsOn\": [\n\t\t\t\t\t{\n\t\t\t\t\t\t\"activity\": \"Get Pipelines\",\n\t\t\t\t\t\t\"dependencyConditions\": [\n\t\t\t\t\t\t\t\"Succeeded\"\n\t\t\t\t\t\t]\n\t\t\t\t\t}\n\t\t\t\t],\n\t\t\t\t\"userProperties\": [],\n\t\t\t\t\"typeProperties\": {\n\t\t\t\t\t\"items\": {\n\t\t\t\t\t\t\"value\": \"@activity('Get Pipelines').output.value\",\n\t\t\t\t\t\t\"type\": \"Expression\"\n\t\t\t\t\t},\n\t\t\t\t\t\"isSequential\": false,\n\t\t\t\t\t\"batchCount\": 50,\n\t\t\t\t\t\"activities\": [\n\t\t\t\t\t\t{\n\t\t\t\t\t\t\t\"name\": \"Worker Pipeline Executor\",\n\t\t\t\t\t\t\t\"description\": \"Run the required worker pipeline and wait for its completion. Update metadata once done.\",\n\t\t\t\t\t\t\t\"type\": \"ExecutePipeline\",\n\t\t\t\t\t\t\t\"dependsOn\": [],\n\t\t\t\t\t\t\t\"userProperties\": [],\n\t\t\t\t\t\t\t\"typeProperties\": {\n\t\t\t\t\t\t\t\t\"pipeline\": {\n\t\t\t\t\t\t\t\t\t\"referenceName\": \"04-Infant\",\n\t\t\t\t\t\t\t\t\t\"type\": \"PipelineReference\"\n\t\t\t\t\t\t\t\t},\n\t\t\t\t\t\t\t\t\"waitOnCompletion\": true,\n\t\t\t\t\t\t\t\t\"parameters\": {\n\t\t\t\t\t\t\t\t\t\"executionId\": {\n\t\t\t\t\t\t\t\t\t\t\"value\": \"@pipeline().parameters.ExecutionId\",\n\t\t\t\t\t\t\t\t\t\t\"type\": \"Expression\"\n\t\t\t\t\t\t\t\t\t},\n\t\t\t\t\t\t\t\t\t\"stageId\": {\n\t\t\t\t\t\t\t\t\t\t\"value\": \"@pipeline().parameters.StageId\",\n\t\t\t\t\t\t\t\t\t\t\"type\": \"Expression\"\n\t\t\t\t\t\t\t\t\t},\n\t\t\t\t\t\t\t\t\t\"pipelineId\": {\n\t\t\t\t\t\t\t\t\t\t\"value\": \"@item().PipelineId\",\n\t\t\t\t\t\t\t\t\t\t\"type\": \"Expression\"\n\t\t\t\t\t\t\t\t\t}\n\t\t\t\t\t\t\t\t}\n\t\t\t\t\t\t\t}\n\t\t\t\t\t\t}\n\t\t\t\t\t]\n\t\t\t\t}\n\t\t\t}\n\t\t],\n\t\t\"parameters\": {\n\t\t\t\"StageId\": {\n\t\t\t\t\"type\": \"int\"\n\t\t\t},\n\t\t\t\"ExecutionId\": {\n\t\t\t\t\"type\": \"string\"\n\t\t\t}\n\t\t},\n\t\t\"folder\": {\n\t\t\t\"name\": \"_ProcFwk\"\n\t\t},\n\t\t\"annotations\": [\n\t\t\t\"procfwk\",\n\t\t\t\"Child\"\n\t\t]\n\t}\n}"
},
{
"path": "DataFactory/pipeline/04-Infant.json",
"content": "{\n\t\"name\": \"04-Infant\",\n\t\"properties\": {\n\t\t\"description\": \"procfwk infant pipeline used to check when the processing pipeline called by the Child completes and passes the resulting status back to the metadata database.\",\n\t\t\"activities\": [\n\t\t\t{\n\t\t\t\t\"name\": \"Execute Worker Pipeline\",\n\t\t\t\t\"description\": \"The lowest level executor with the metadata framework to call existing processing pipelines within Data Factory. The function called will block processing and wait for an outcome.\",\n\t\t\t\t\"type\": \"AzureFunctionActivity\",\n\t\t\t\t\"dependsOn\": [\n\t\t\t\t\t{\n\t\t\t\t\t\t\"activity\": \"Log Pipeline Running\",\n\t\t\t\t\t\t\"dependencyConditions\": [\n\t\t\t\t\t\t\t\"Succeeded\"\n\t\t\t\t\t\t]\n\t\t\t\t\t},\n\t\t\t\t\t{\n\t\t\t\t\t\t\"activity\": \"Get Pipeline Params\",\n\t\t\t\t\t\t\"dependencyConditions\": [\n\t\t\t\t\t\t\t\"Succeeded\"\n\t\t\t\t\t\t]\n\t\t\t\t\t}\n\t\t\t\t],\n\t\t\t\t\"policy\": {\n\t\t\t\t\t\"timeout\": \"0.00:10:00\",\n\t\t\t\t\t\"retry\": 0,\n\t\t\t\t\t\"retryIntervalInSeconds\": 30,\n\t\t\t\t\t\"secureOutput\": false,\n\t\t\t\t\t\"secureInput\": true\n\t\t\t\t},\n\t\t\t\t\"userProperties\": [],\n\t\t\t\t\"typeProperties\": {\n\t\t\t\t\t\"functionName\": \"ExecutePipeline\",\n\t\t\t\t\t\"method\": \"POST\",\n\t\t\t\t\t\"body\": {\n\t\t\t\t\t\t\"value\": \"@concat('\\n{\\n \\\"tenantId\\\": \\\"',variables('WorkerCoreDetails')[0].tenantId,'\\\",\\n \\\"applicationId\\\": \\\"',variables('WorkerCoreDetails')[0].applicationId,'\\\",\\n \\\"authenticationKey\\\": \\\"',variables('WorkerCoreDetails')[0].authenticationKey,'\\\",\\n \\\"subscriptionId\\\": \\\"',variables('WorkerCoreDetails')[0].subscriptionId,'\\\",\\n \\\"resourceGroupName\\\": \\\"',variables('WorkerCoreDetails')[0].resourceGroupName,'\\\",\\n\\t\\\"orchestratorName\\\": \\\"',variables('WorkerCoreDetails')[0].orchestratorName,'\\\",\\n \\\"orchestratorType\\\": \\\"',variables('WorkerCoreDetails')[0].orchestratorType,'\\\",\\n \\\"pipelineName\\\": \\\"',variables('WorkerCoreDetails')[0].pipelineName,'\\\"',activity('Get Pipeline Params').output.firstRow.Params,'\\n}')\",\n\t\t\t\t\t\t\"type\": \"Expression\"\n\t\t\t\t\t}\n\t\t\t\t},\n\t\t\t\t\"linkedServiceName\": {\n\t\t\t\t\t\"referenceName\": \"FrameworkFunctions\",\n\t\t\t\t\t\"type\": \"LinkedServiceReference\"\n\t\t\t\t}\n\t\t\t},\n\t\t\t{\n\t\t\t\t\"name\": \"Get Pipeline Params\",\n\t\t\t\t\"description\": \"Returns any parameters from metadata required for the processing pipeline being called. The output can be an empty string if no parameters are required.\",\n\t\t\t\t\"type\": \"Lookup\",\n\t\t\t\t\"dependsOn\": [],\n\t\t\t\t\"policy\": {\n\t\t\t\t\t\"timeout\": \"0.00:10:00\",\n\t\t\t\t\t\"retry\": 0,\n\t\t\t\t\t\"retryIntervalInSeconds\": 30,\n\t\t\t\t\t\"secureOutput\": false,\n\t\t\t\t\t\"secureInput\": false\n\t\t\t\t},\n\t\t\t\t\"userProperties\": [],\n\t\t\t\t\"typeProperties\": {\n\t\t\t\t\t\"source\": {\n\t\t\t\t\t\t\"type\": \"AzureSqlSource\",\n\t\t\t\t\t\t\"sqlReaderStoredProcedureName\": \"[procfwk].[GetPipelineParameters]\",\n\t\t\t\t\t\t\"storedProcedureParameters\": {\n\t\t\t\t\t\t\t\"PipelineId\": {\n\t\t\t\t\t\t\t\t\"type\": \"Int32\",\n\t\t\t\t\t\t\t\t\"value\": {\n\t\t\t\t\t\t\t\t\t\"value\": \"@pipeline().parameters.pipelineId\",\n\t\t\t\t\t\t\t\t\t\"type\": \"Expression\"\n\t\t\t\t\t\t\t\t}\n\t\t\t\t\t\t\t}\n\t\t\t\t\t\t},\n\t\t\t\t\t\t\"queryTimeout\": \"02:00:00\",\n\t\t\t\t\t\t\"partitionOption\": \"None\"\n\t\t\t\t\t},\n\t\t\t\t\t\"dataset\": {\n\t\t\t\t\t\t\"referenceName\": \"GetSetMetadata\",\n\t\t\t\t\t\t\"type\": \"DatasetReference\"\n\t\t\t\t\t}\n\t\t\t\t}\n\t\t\t},\n\t\t\t{\n\t\t\t\t\"name\": \"Log Pipeline Running\",\n\t\t\t\t\"description\": \"Sets the current pipeline with a status of running within the current execution database table.\",\n\t\t\t\t\"type\": \"SqlServerStoredProcedure\",\n\t\t\t\t\"dependsOn\": [\n\t\t\t\t\t{\n\t\t\t\t\t\t\"activity\": \"Is Target Worker Validate\",\n\t\t\t\t\t\t\"dependencyConditions\": [\n\t\t\t\t\t\t\t\"Succeeded\"\n\t\t\t\t\t\t]\n\t\t\t\t\t}\n\t\t\t\t],\n\t\t\t\t\"policy\": {\n\t\t\t\t\t\"timeout\": \"0.00:10:00\",\n\t\t\t\t\t\"retry\": 0,\n\t\t\t\t\t\"retryIntervalInSeconds\": 30,\n\t\t\t\t\t\"secureOutput\": false,\n\t\t\t\t\t\"secureInput\": false\n\t\t\t\t},\n\t\t\t\t\"userProperties\": [],\n\t\t\t\t\"typeProperties\": {\n\t\t\t\t\t\"storedProcedureName\": \"[procfwk].[SetLogPipelineRunning]\",\n\t\t\t\t\t\"storedProcedureParameters\": {\n\t\t\t\t\t\t\"ExecutionId\": {\n\t\t\t\t\t\t\t\"value\": {\n\t\t\t\t\t\t\t\t\"value\": \"@pipeline().parameters.ExecutionId\",\n\t\t\t\t\t\t\t\t\"type\": \"Expression\"\n\t\t\t\t\t\t\t},\n\t\t\t\t\t\t\t\"type\": \"Guid\"\n\t\t\t\t\t\t},\n\t\t\t\t\t\t\"PipelineId\": {\n\t\t\t\t\t\t\t\"value\": {\n\t\t\t\t\t\t\t\t\"value\": \"@pipeline().parameters.pipelineId\",\n\t\t\t\t\t\t\t\t\"type\": \"Expression\"\n\t\t\t\t\t\t\t},\n\t\t\t\t\t\t\t\"type\": \"Int32\"\n\t\t\t\t\t\t},\n\t\t\t\t\t\t\"StageId\": {\n\t\t\t\t\t\t\t\"value\": {\n\t\t\t\t\t\t\t\t\"value\": \"@pipeline().parameters.StageId\",\n\t\t\t\t\t\t\t\t\"type\": \"Expression\"\n\t\t\t\t\t\t\t},\n\t\t\t\t\t\t\t\"type\": \"Int32\"\n\t\t\t\t\t\t}\n\t\t\t\t\t}\n\t\t\t\t},\n\t\t\t\t\"linkedServiceName\": {\n\t\t\t\t\t\"referenceName\": \"SupportDatabase\",\n\t\t\t\t\t\"type\": \"LinkedServiceReference\"\n\t\t\t\t}\n\t\t\t},\n\t\t\t{\n\t\t\t\t\"name\": \"Log Execute Function Activity Failure\",\n\t\t\t\t\"description\": \"Handle true failures from calling out to the Azure Function and update the current execution table accordingly so a restart can occur.\",\n\t\t\t\t\"type\": \"SqlServerStoredProcedure\",\n\t\t\t\t\"dependsOn\": [\n\t\t\t\t\t{\n\t\t\t\t\t\t\"activity\": \"Execute Worker Pipeline\",\n\t\t\t\t\t\t\"dependencyConditions\": [\n\t\t\t\t\t\t\t\"Failed\"\n\t\t\t\t\t\t]\n\t\t\t\t\t}\n\t\t\t\t],\n\t\t\t\t\"policy\": {\n\t\t\t\t\t\"timeout\": \"0.00:10:00\",\n\t\t\t\t\t\"retry\": 0,\n\t\t\t\t\t\"retryIntervalInSeconds\": 30,\n\t\t\t\t\t\"secureOutput\": false,\n\t\t\t\t\t\"secureInput\": false\n\t\t\t\t},\n\t\t\t\t\"userProperties\": [],\n\t\t\t\t\"typeProperties\": {\n\t\t\t\t\t\"storedProcedureName\": \"[procfwk].[SetLogActivityFailed]\",\n\t\t\t\t\t\"storedProcedureParameters\": {\n\t\t\t\t\t\t\"ExecutionId\": {\n\t\t\t\t\t\t\t\"value\": {\n\t\t\t\t\t\t\t\t\"value\": \"@pipeline().parameters.ExecutionId\",\n\t\t\t\t\t\t\t\t\"type\": \"Expression\"\n\t\t\t\t\t\t\t},\n\t\t\t\t\t\t\t\"type\": \"Guid\"\n\t\t\t\t\t\t},\n\t\t\t\t\t\t\"PipelineId\": {\n\t\t\t\t\t\t\t\"value\": {\n\t\t\t\t\t\t\t\t\"value\": \"@pipeline().parameters.pipelineId\",\n\t\t\t\t\t\t\t\t\"type\": \"Expression\"\n\t\t\t\t\t\t\t},\n\t\t\t\t\t\t\t\"type\": \"Int32\"\n\t\t\t\t\t\t},\n\t\t\t\t\t\t\"StageId\": {\n\t\t\t\t\t\t\t\"value\": {\n\t\t\t\t\t\t\t\t\"value\": \"@pipeline().parameters.StageId\",\n\t\t\t\t\t\t\t\t\"type\": \"Expression\"\n\t\t\t\t\t\t\t},\n\t\t\t\t\t\t\t\"type\": \"Int32\"\n\t\t\t\t\t\t},\n\t\t\t\t\t\t\"CallingActivity\": {\n\t\t\t\t\t\t\t\"value\": \"ExecuteWorkerPipeline\",\n\t\t\t\t\t\t\t\"type\": \"String\"\n\t\t\t\t\t\t}\n\t\t\t\t\t}\n\t\t\t\t},\n\t\t\t\t\"linkedServiceName\": {\n\t\t\t\t\t\"referenceName\": \"SupportDatabase\",\n\t\t\t\t\t\"type\": \"LinkedServiceReference\"\n\t\t\t\t}\n\t\t\t},\n\t\t\t{\n\t\t\t\t\"name\": \"Update Run Id\",\n\t\t\t\t\"description\": \"Provide the actual ADF run ID back to the current execution table for long term logging and alignment between the metadata other Azure monitoring tools.\",\n\t\t\t\t\"type\": \"SqlServerStoredProcedure\",\n\t\t\t\t\"dependsOn\": [\n\t\t\t\t\t{\n\t\t\t\t\t\t\"activity\": \"Set Run Id\",\n\t\t\t\t\t\t\"dependencyConditions\": [\n\t\t\t\t\t\t\t\"Succeeded\"\n\t\t\t\t\t\t]\n\t\t\t\t\t}\n\t\t\t\t],\n\t\t\t\t\"policy\": {\n\t\t\t\t\t\"timeout\": \"0.00:10:00\",\n\t\t\t\t\t\"retry\": 0,\n\t\t\t\t\t\"retryIntervalInSeconds\": 30,\n\t\t\t\t\t\"secureOutput\": false,\n\t\t\t\t\t\"secureInput\": false\n\t\t\t\t},\n\t\t\t\t\"userProperties\": [],\n\t\t\t\t\"typeProperties\": {\n\t\t\t\t\t\"storedProcedureName\": \"[procfwk].[SetLogPipelineRunId]\",\n\t\t\t\t\t\"storedProcedureParameters\": {\n\t\t\t\t\t\t\"ExecutionId\": {\n\t\t\t\t\t\t\t\"value\": {\n\t\t\t\t\t\t\t\t\"value\": \"@pipeline().parameters.ExecutionId\",\n\t\t\t\t\t\t\t\t\"type\": \"Expression\"\n\t\t\t\t\t\t\t},\n\t\t\t\t\t\t\t\"type\": \"Guid\"\n\t\t\t\t\t\t},\n\t\t\t\t\t\t\"PipelineId\": {\n\t\t\t\t\t\t\t\"value\": {\n\t\t\t\t\t\t\t\t\"value\": \"@pipeline().parameters.pipelineId\",\n\t\t\t\t\t\t\t\t\"type\": \"Expression\"\n\t\t\t\t\t\t\t},\n\t\t\t\t\t\t\t\"type\": \"Int32\"\n\t\t\t\t\t\t},\n\t\t\t\t\t\t\"RunId\": {\n\t\t\t\t\t\t\t\"value\": {\n\t\t\t\t\t\t\t\t\"value\": \"@variables('WorkerRunId')\",\n\t\t\t\t\t\t\t\t\"type\": \"Expression\"\n\t\t\t\t\t\t\t},\n\t\t\t\t\t\t\t\"type\": \"Guid\"\n\t\t\t\t\t\t},\n\t\t\t\t\t\t\"StageId\": {\n\t\t\t\t\t\t\t\"value\": {\n\t\t\t\t\t\t\t\t\"value\": \"@pipeline().parameters.StageId\",\n\t\t\t\t\t\t\t\t\"type\": \"Expression\"\n\t\t\t\t\t\t\t},\n\t\t\t\t\t\t\t\"type\": \"Int32\"\n\t\t\t\t\t\t}\n\t\t\t\t\t}\n\t\t\t\t},\n\t\t\t\t\"linkedServiceName\": {\n\t\t\t\t\t\"referenceName\": \"SupportDatabase\",\n\t\t\t\t\t\"type\": \"LinkedServiceReference\"\n\t\t\t\t}\n\t\t\t},\n\t\t\t{\n\t\t\t\t\"name\": \"Check For Alerts\",\n\t\t\t\t\"description\": \"Checks the properties tables and if any recipients in the database require alerts sending for the current pipeline ID.\",\n\t\t\t\t\"type\": \"Lookup\",\n\t\t\t\t\"dependsOn\": [\n\t\t\t\t\t{\n\t\t\t\t\t\t\"activity\": \"Update Run Id\",\n\t\t\t\t\t\t\"dependencyConditions\": [\n\t\t\t\t\t\t\t\"Succeeded\"\n\t\t\t\t\t\t]\n\t\t\t\t\t},\n\t\t\t\t\t{\n\t\t\t\t\t\t\"activity\": \"Set Pipeline Result\",\n\t\t\t\t\t\t\"dependencyConditions\": [\n\t\t\t\t\t\t\t\"Completed\"\n\t\t\t\t\t\t]\n\t\t\t\t\t}\n\t\t\t\t],\n\t\t\t\t\"policy\": {\n\t\t\t\t\t\"timeout\": \"0.00:00:30\",\n\t\t\t\t\t\"retry\": 3,\n\t\t\t\t\t\"retryIntervalInSeconds\": 30,\n\t\t\t\t\t\"secureOutput\": false,\n\t\t\t\t\t\"secureInput\": false\n\t\t\t\t},\n\t\t\t\t\"userProperties\": [],\n\t\t\t\t\"typeProperties\": {\n\t\t\t\t\t\"source\": {\n\t\t\t\t\t\t\"type\": \"AzureSqlSource\",\n\t\t\t\t\t\t\"sqlReaderStoredProcedureName\": \"[procfwk].[CheckForEmailAlerts]\",\n\t\t\t\t\t\t\"storedProcedureParameters\": {\n\t\t\t\t\t\t\t\"PipelineId\": {\n\t\t\t\t\t\t\t\t\"type\": \"Int32\",\n\t\t\t\t\t\t\t\t\"value\": {\n\t\t\t\t\t\t\t\t\t\"value\": \"@pipeline().parameters.pipelineId\",\n\t\t\t\t\t\t\t\t\t\"type\": \"Expression\"\n\t\t\t\t\t\t\t\t}\n\t\t\t\t\t\t\t}\n\t\t\t\t\t\t},\n\t\t\t\t\t\t\"queryTimeout\": \"02:00:00\",\n\t\t\t\t\t\t\"partitionOption\": \"None\"\n\t\t\t\t\t},\n\t\t\t\t\t\"dataset\": {\n\t\t\t\t\t\t\"referenceName\": \"GetSetMetadata\",\n\t\t\t\t\t\t\"type\": \"DatasetReference\"\n\t\t\t\t\t},\n\t\t\t\t\t\"firstRowOnly\": true\n\t\t\t\t}\n\t\t\t},\n\t\t\t{\n\t\t\t\t\"name\": \"Send Alerts\",\n\t\t\t\t\"description\": \"True = alerts need sending.\\nFalse = do nothing.\",\n\t\t\t\t\"type\": \"IfCondition\",\n\t\t\t\t\"dependsOn\": [\n\t\t\t\t\t{\n\t\t\t\t\t\t\"activity\": \"Check For Alerts\",\n\t\t\t\t\t\t\"dependencyConditions\": [\n\t\t\t\t\t\t\t\"Succeeded\"\n\t\t\t\t\t\t]\n\t\t\t\t\t}\n\t\t\t\t],\n\t\t\t\t\"userProperties\": [],\n\t\t\t\t\"typeProperties\": {\n\t\t\t\t\t\"expression\": {\n\t\t\t\t\t\t\"value\": \"@activity('Check For Alerts').output.firstRow.SendAlerts\",\n\t\t\t\t\t\t\"type\": \"Expression\"\n\t\t\t\t\t},\n\t\t\t\t\t\"ifTrueActivities\": [\n\t\t\t\t\t\t{\n\t\t\t\t\t\t\t\"name\": \"Get Email Parts\",\n\t\t\t\t\t\t\t\"description\": \"Return all required content from the metadata database to send an email alerting using the procfwk. The lookup returns the exact content for the function body request.\",\n\t\t\t\t\t\t\t\"type\": \"Lookup\",\n\t\t\t\t\t\t\t\"dependsOn\": [],\n\t\t\t\t\t\t\t\"policy\": {\n\t\t\t\t\t\t\t\t\"timeout\": \"0.00:10:00\",\n\t\t\t\t\t\t\t\t\"retry\": 0,\n\t\t\t\t\t\t\t\t\"retryIntervalInSeconds\": 30,\n\t\t\t\t\t\t\t\t\"secureOutput\": true,\n\t\t\t\t\t\t\t\t\"secureInput\": false\n\t\t\t\t\t\t\t},\n\t\t\t\t\t\t\t\"userProperties\": [],\n\t\t\t\t\t\t\t\"typeProperties\": {\n\t\t\t\t\t\t\t\t\"source\": {\n\t\t\t\t\t\t\t\t\t\"type\": \"AzureSqlSource\",\n\t\t\t\t\t\t\t\t\t\"sqlReaderStoredProcedureName\": \"[procfwk].[GetEmailAlertParts]\",\n\t\t\t\t\t\t\t\t\t\"storedProcedureParameters\": {\n\t\t\t\t\t\t\t\t\t\t\"PipelineId\": {\n\t\t\t\t\t\t\t\t\t\t\t\"type\": \"Int32\",\n\t\t\t\t\t\t\t\t\t\t\t\"value\": {\n\t\t\t\t\t\t\t\t\t\t\t\t\"value\": \"@pipeline().parameters.pipelineId\",\n\t\t\t\t\t\t\t\t\t\t\t\t\"type\": \"Expression\"\n\t\t\t\t\t\t\t\t\t\t\t}\n\t\t\t\t\t\t\t\t\t\t}\n\t\t\t\t\t\t\t\t\t},\n\t\t\t\t\t\t\t\t\t\"queryTimeout\": \"02:00:00\",\n\t\t\t\t\t\t\t\t\t\"partitionOption\": \"None\"\n\t\t\t\t\t\t\t\t},\n\t\t\t\t\t\t\t\t\"dataset\": {\n\t\t\t\t\t\t\t\t\t\"referenceName\": \"GetSetMetadata\",\n\t\t\t\t\t\t\t\t\t\"type\": \"DatasetReference\"\n\t\t\t\t\t\t\t\t},\n\t\t\t\t\t\t\t\t\"firstRowOnly\": true\n\t\t\t\t\t\t\t}\n\t\t\t\t\t\t},\n\t\t\t\t\t\t{\n\t\t\t\t\t\t\t\"name\": \"Call Email Sender\",\n\t\t\t\t\t\t\t\"description\": \"Pass off email request to Utils Send Email pipeline.\",\n\t\t\t\t\t\t\t\"type\": \"ExecutePipeline\",\n\t\t\t\t\t\t\t\"dependsOn\": [\n\t\t\t\t\t\t\t\t{\n\t\t\t\t\t\t\t\t\t\"activity\": \"Get Email Parts\",\n\t\t\t\t\t\t\t\t\t\"dependencyConditions\": [\n\t\t\t\t\t\t\t\t\t\t\"Succeeded\"\n\t\t\t\t\t\t\t\t\t]\n\t\t\t\t\t\t\t\t}\n\t\t\t\t\t\t\t],\n\t\t\t\t\t\t\t\"userProperties\": [],\n\t\t\t\t\t\t\t\"typeProperties\": {\n\t\t\t\t\t\t\t\t\"pipeline\": {\n\t\t\t\t\t\t\t\t\t\"referenceName\": \"Email Sender\",\n\t\t\t\t\t\t\t\t\t\"type\": \"PipelineReference\"\n\t\t\t\t\t\t\t\t},\n\t\t\t\t\t\t\t\t\"waitOnCompletion\": true,\n\t\t\t\t\t\t\t\t\"parameters\": {\n\t\t\t\t\t\t\t\t\t\"Recipients\": {\n\t\t\t\t\t\t\t\t\t\t\"value\": \"@activity('Get Email Parts').output.firstRow.emailRecipients\",\n\t\t\t\t\t\t\t\t\t\t\"type\": \"Expression\"\n\t\t\t\t\t\t\t\t\t},\n\t\t\t\t\t\t\t\t\t\"CcRecipients\": {\n\t\t\t\t\t\t\t\t\t\t\"value\": \"@activity('Get Email Parts').output.firstRow.emailCcRecipients\",\n\t\t\t\t\t\t\t\t\t\t\"type\": \"Expression\"\n\t\t\t\t\t\t\t\t\t},\n\t\t\t\t\t\t\t\t\t\"BccRecipients\": {\n\t\t\t\t\t\t\t\t\t\t\"value\": \"@activity('Get Email Parts').output.firstRow.emailBccRecipients\",\n\t\t\t\t\t\t\t\t\t\t\"type\": \"Expression\"\n\t\t\t\t\t\t\t\t\t},\n\t\t\t\t\t\t\t\t\t\"Subject\": {\n\t\t\t\t\t\t\t\t\t\t\"value\": \"@activity('Get Email Parts').output.firstRow.emailSubject\",\n\t\t\t\t\t\t\t\t\t\t\"type\": \"Expression\"\n\t\t\t\t\t\t\t\t\t},\n\t\t\t\t\t\t\t\t\t\"Body\": {\n\t\t\t\t\t\t\t\t\t\t\"value\": \"@activity('Get Email Parts').output.firstRow.emailBody\",\n\t\t\t\t\t\t\t\t\t\t\"type\": \"Expression\"\n\t\t\t\t\t\t\t\t\t},\n\t\t\t\t\t\t\t\t\t\"Importance\": {\n\t\t\t\t\t\t\t\t\t\t\"value\": \"@activity('Get Email Parts').output.firstRow.emailImportance\",\n\t\t\t\t\t\t\t\t\t\t\"type\": \"Expression\"\n\t\t\t\t\t\t\t\t\t}\n\t\t\t\t\t\t\t\t}\n\t\t\t\t\t\t\t}\n\t\t\t\t\t\t}\n\t\t\t\t\t]\n\t\t\t\t}\n\t\t\t},\n\t\t\t{\n\t\t\t\t\"name\": \"Wait Until Pipeline Completes\",\n\t\t\t\t\"description\": \"Loops until the Worker pipeline called completes.\\n\\nSimple status:\\n- Running = new iteration.\\n- Done = break.\",\n\t\t\t\t\"type\": \"Until\",\n\t\t\t\t\"dependsOn\": [\n\t\t\t\t\t{\n\t\t\t\t\t\t\"activity\": \"Get Wait Duration\",\n\t\t\t\t\t\t\"dependencyConditions\": [\n\t\t\t\t\t\t\t\"Succeeded\"\n\t\t\t\t\t\t]\n\t\t\t\t\t},\n\t\t\t\t\t{\n\t\t\t\t\t\t\"activity\": \"Execute Worker Pipeline\",\n\t\t\t\t\t\t\"dependencyConditions\": [\n\t\t\t\t\t\t\t\"Succeeded\"\n\t\t\t\t\t\t]\n\t\t\t\t\t},\n\t\t\t\t\t{\n\t\t\t\t\t\t\"activity\": \"Set Run Id\",\n\t\t\t\t\t\t\"dependencyConditions\": [\n\t\t\t\t\t\t\t\"Succeeded\"\n\t\t\t\t\t\t]\n\t\t\t\t\t}\n\t\t\t\t],\n\t\t\t\t\"userProperties\": [],\n\t\t\t\t\"typeProperties\": {\n\t\t\t\t\t\"expression\": {\n\t\t\t\t\t\t\"value\": \"@variables('WorkerPipelineState')\",\n\t\t\t\t\t\t\"type\": \"Expression\"\n\t\t\t\t\t},\n\t\t\t\t\t\"activities\": [\n\t\t\t\t\t\t{\n\t\t\t\t\t\t\t\"name\": \"Get Worker Pipeline Status\",\n\t\t\t\t\t\t\t\"description\": \"Checks the status of a given processing pipeline and provides the value for the downstream framework activities to act upon.\",\n\t\t\t\t\t\t\t\"type\": \"AzureFunctionActivity\",\n\t\t\t\t\t\t\t\"dependsOn\": [],\n\t\t\t\t\t\t\t\"policy\": {\n\t\t\t\t\t\t\t\t\"timeout\": \"0.03:59:59\",\n\t\t\t\t\t\t\t\t\"retry\": 0,\n\t\t\t\t\t\t\t\t\"retryIntervalInSeconds\": 30,\n\t\t\t\t\t\t\t\t\"secureOutput\": false,\n\t\t\t\t\t\t\t\t\"secureInput\": true\n\t\t\t\t\t\t\t},\n\t\t\t\t\t\t\t\"userProperties\": [],\n\t\t\t\t\t\t\t\"typeProperties\": {\n\t\t\t\t\t\t\t\t\"functionName\": \"CheckPipelineStatus\",\n\t\t\t\t\t\t\t\t\"method\": \"POST\",\n\t\t\t\t\t\t\t\t\"body\": {\n\t\t\t\t\t\t\t\t\t\"value\": \"@concat('\\n{\\n \\\"tenantId\\\": \\\"',variables('WorkerCoreDetails')[0].tenantId,'\\\",\\n \\\"applicationId\\\": \\\"',variables('WorkerCoreDetails')[0].applicationId,'\\\",\\n \\\"authenticationKey\\\": \\\"',variables('WorkerCoreDetails')[0].authenticationKey,'\\\",\\n \\\"subscriptionId\\\": \\\"',variables('WorkerCoreDetails')[0].subscriptionId,'\\\",\\n \\\"resourceGroupName\\\": \\\"',variables('WorkerCoreDetails')[0].resourceGroupName,'\\\",\\n\\t\\\"orchestratorName\\\": \\\"',variables('WorkerCoreDetails')[0].orchestratorName,'\\\",\\n \\\"orchestratorType\\\": \\\"',variables('WorkerCoreDetails')[0].orchestratorType,'\\\",\\n \\\"pipelineName\\\": \\\"',variables('WorkerCoreDetails')[0].pipelineName,'\\\",\\n \\\"runId\\\": \\\"',variables('WorkerRunId'),'\\\"\\n}')\",\n\t\t\t\t\t\t\t\t\t\"type\": \"Expression\"\n\t\t\t\t\t\t\t\t}\n\t\t\t\t\t\t\t},\n\t\t\t\t\t\t\t\"linkedServiceName\": {\n\t\t\t\t\t\t\t\t\"referenceName\": \"FrameworkFunctions\",\n\t\t\t\t\t\t\t\t\"type\": \"LinkedServiceReference\"\n\t\t\t\t\t\t\t}\n\t\t\t\t\t\t},\n\t\t\t\t\t\t{\n\t\t\t\t\t\t\t\"name\": \"Wait If Running\",\n\t\t\t\t\t\t\t\"description\": \"True = Do nothing.\\nFalse = Wait, before the next iteration.\",\n\t\t\t\t\t\t\t\"type\": \"IfCondition\",\n\t\t\t\t\t\t\t\"dependsOn\": [\n\t\t\t\t\t\t\t\t{\n\t\t\t\t\t\t\t\t\t\"activity\": \"Set Worker State\",\n\t\t\t\t\t\t\t\t\t\"dependencyConditions\": [\n\t\t\t\t\t\t\t\t\t\t\"Succeeded\"\n\t\t\t\t\t\t\t\t\t]\n\t\t\t\t\t\t\t\t}\n\t\t\t\t\t\t\t],\n\t\t\t\t\t\t\t\"userProperties\": [],\n\t\t\t\t\t\t\t\"typeProperties\": {\n\t\t\t\t\t\t\t\t\"expression\": {\n\t\t\t\t\t\t\t\t\t\"value\": \"@variables('WorkerPipelineState')\",\n\t\t\t\t\t\t\t\t\t\"type\": \"Expression\"\n\t\t\t\t\t\t\t\t},\n\t\t\t\t\t\t\t\t\"ifFalseActivities\": [\n\t\t\t\t\t\t\t\t\t{\n\t\t\t\t\t\t\t\t\t\t\"name\": \"Wait for Pipeline\",\n\t\t\t\t\t\t\t\t\t\t\"description\": \"The processing pipeline is still running so Wait before checking its status again.\",\n\t\t\t\t\t\t\t\t\t\t\"type\": \"Wait\",\n\t\t\t\t\t\t\t\t\t\t\"dependsOn\": [],\n\t\t\t\t\t\t\t\t\t\t\"userProperties\": [],\n\t\t\t\t\t\t\t\t\t\t\"typeProperties\": {\n\t\t\t\t\t\t\t\t\t\t\t\"waitTimeInSeconds\": {\n\t\t\t\t\t\t\t\t\t\t\t\t\"value\": \"@activity('Get Wait Duration').output.firstRow.PropertyValue\",\n\t\t\t\t\t\t\t\t\t\t\t\t\"type\": \"Expression\"\n\t\t\t\t\t\t\t\t\t\t\t}\n\t\t\t\t\t\t\t\t\t\t}\n\t\t\t\t\t\t\t\t\t}\n\t\t\t\t\t\t\t\t]\n\t\t\t\t\t\t\t}\n\t\t\t\t\t\t},\n\t\t\t\t\t\t{\n\t\t\t\t\t\t\t\"name\": \"Set Last Check DateTime\",\n\t\t\t\t\t\t\t\"description\": \"Update the current execution table with a date time from when the Worker pipeline status was last checked as part of the Until iterations.\",\n\t\t\t\t\t\t\t\"type\": \"SqlServerStoredProcedure\",\n\t\t\t\t\t\t\t\"dependsOn\": [\n\t\t\t\t\t\t\t\t{\n\t\t\t\t\t\t\t\t\t\"activity\": \"Get Worker Pipeline Status\",\n\t\t\t\t\t\t\t\t\t\"dependencyConditions\": [\n\t\t\t\t\t\t\t\t\t\t\"Succeeded\"\n\t\t\t\t\t\t\t\t\t]\n\t\t\t\t\t\t\t\t}\n\t\t\t\t\t\t\t],\n\t\t\t\t\t\t\t\"policy\": {\n\t\t\t\t\t\t\t\t\"timeout\": \"0.00:10:00\",\n\t\t\t\t\t\t\t\t\"retry\": 0,\n\t\t\t\t\t\t\t\t\"retryIntervalInSeconds\": 30,\n\t\t\t\t\t\t\t\t\"secureOutput\": false,\n\t\t\t\t\t\t\t\t\"secureInput\": false\n\t\t\t\t\t\t\t},\n\t\t\t\t\t\t\t\"userProperties\": [],\n\t\t\t\t\t\t\t\"typeProperties\": {\n\t\t\t\t\t\t\t\t\"storedProcedureName\": \"[procfwk].[SetLogPipelineLastStatusCheck]\",\n\t\t\t\t\t\t\t\t\"storedProcedureParameters\": {\n\t\t\t\t\t\t\t\t\t\"ExecutionId\": {\n\t\t\t\t\t\t\t\t\t\t\"value\": {\n\t\t\t\t\t\t\t\t\t\t\t\"value\": \"@pipeline().parameters.executionId\",\n\t\t\t\t\t\t\t\t\t\t\t\"type\": \"Expression\"\n\t\t\t\t\t\t\t\t\t\t},\n\t\t\t\t\t\t\t\t\t\t\"type\": \"Guid\"\n\t\t\t\t\t\t\t\t\t},\n\t\t\t\t\t\t\t\t\t\"PipelineId\": {\n\t\t\t\t\t\t\t\t\t\t\"value\": {\n\t\t\t\t\t\t\t\t\t\t\t\"value\": \"@pipeline().parameters.pipelineId\",\n\t\t\t\t\t\t\t\t\t\t\t\"type\": \"Expression\"\n\t\t\t\t\t\t\t\t\t\t},\n\t\t\t\t\t\t\t\t\t\t\"type\": \"Int32\"\n\t\t\t\t\t\t\t\t\t},\n\t\t\t\t\t\t\t\t\t\"StageId\": {\n\t\t\t\t\t\t\t\t\t\t\"value\": {\n\t\t\t\t\t\t\t\t\t\t\t\"value\": \"@pipeline().parameters.stageId\",\n\t\t\t\t\t\t\t\t\t\t\t\"type\": \"Expression\"\n\t\t\t\t\t\t\t\t\t\t},\n\t\t\t\t\t\t\t\t\t\t\"type\": \"Int32\"\n\t\t\t\t\t\t\t\t\t}\n\t\t\t\t\t\t\t\t}\n\t\t\t\t\t\t\t},\n\t\t\t\t\t\t\t\"linkedServiceName\": {\n\t\t\t\t\t\t\t\t\"referenceName\": \"SupportDatabase\",\n\t\t\t\t\t\t\t\t\"type\": \"LinkedServiceReference\"\n\t\t\t\t\t\t\t}\n\t\t\t\t\t\t},\n\t\t\t\t\t\t{\n\t\t\t\t\t\t\t\"name\": \"Log Check Function Activity Failure\",\n\t\t\t\t\t\t\t\"description\": \"Report to the current execution table that the framework pipeline activity has failed. This failure is outside of the scope of the framework and is probably related to a wider platform problem.\",\n\t\t\t\t\t\t\t\"type\": \"SqlServerStoredProcedure\",\n\t\t\t\t\t\t\t\"dependsOn\": [\n\t\t\t\t\t\t\t\t{\n\t\t\t\t\t\t\t\t\t\"activity\": \"Get Worker Pipeline Status\",\n\t\t\t\t\t\t\t\t\t\"dependencyConditions\": [\n\t\t\t\t\t\t\t\t\t\t\"Failed\"\n\t\t\t\t\t\t\t\t\t]\n\t\t\t\t\t\t\t\t}\n\t\t\t\t\t\t\t],\n\t\t\t\t\t\t\t\"policy\": {\n\t\t\t\t\t\t\t\t\"timeout\": \"0.00:10:00\",\n\t\t\t\t\t\t\t\t\"retry\": 0,\n\t\t\t\t\t\t\t\t\"retryIntervalInSeconds\": 30,\n\t\t\t\t\t\t\t\t\"secureOutput\": false,\n\t\t\t\t\t\t\t\t\"secureInput\": false\n\t\t\t\t\t\t\t},\n\t\t\t\t\t\t\t\"userProperties\": [],\n\t\t\t\t\t\t\t\"typeProperties\": {\n\t\t\t\t\t\t\t\t\"storedProcedureName\": \"[procfwk].[SetLogActivityFailed]\",\n\t\t\t\t\t\t\t\t\"storedProcedureParameters\": {\n\t\t\t\t\t\t\t\t\t\"CallingActivity\": {\n\t\t\t\t\t\t\t\t\t\t\"value\": \"GetWorkerPipelineStatus\",\n\t\t\t\t\t\t\t\t\t\t\"type\": \"String\"\n\t\t\t\t\t\t\t\t\t},\n\t\t\t\t\t\t\t\t\t\"ExecutionId\": {\n\t\t\t\t\t\t\t\t\t\t\"value\": {\n\t\t\t\t\t\t\t\t\t\t\t\"value\": \"@pipeline().parameters.executionId\",\n\t\t\t\t\t\t\t\t\t\t\t\"type\": \"Expression\"\n\t\t\t\t\t\t\t\t\t\t},\n\t\t\t\t\t\t\t\t\t\t\"type\": \"Guid\"\n\t\t\t\t\t\t\t\t\t},\n\t\t\t\t\t\t\t\t\t\"PipelineId\": {\n\t\t\t\t\t\t\t\t\t\t\"value\": {\n\t\t\t\t\t\t\t\t\t\t\t\"value\": \"@pipeline().parameters.pipelineId\",\n\t\t\t\t\t\t\t\t\t\t\t\"type\": \"Expression\"\n\t\t\t\t\t\t\t\t\t\t},\n\t\t\t\t\t\t\t\t\t\t\"type\": \"Int32\"\n\t\t\t\t\t\t\t\t\t},\n\t\t\t\t\t\t\t\t\t\"StageId\": {\n\t\t\t\t\t\t\t\t\t\t\"value\": {\n\t\t\t\t\t\t\t\t\t\t\t\"value\": \"@pipeline().parameters.stageId\",\n\t\t\t\t\t\t\t\t\t\t\t\"type\": \"Expression\"\n\t\t\t\t\t\t\t\t\t\t},\n\t\t\t\t\t\t\t\t\t\t\"type\": \"Int32\"\n\t\t\t\t\t\t\t\t\t}\n\t\t\t\t\t\t\t\t}\n\t\t\t\t\t\t\t},\n\t\t\t\t\t\t\t\"linkedServiceName\": {\n\t\t\t\t\t\t\t\t\"referenceName\": \"SupportDatabase\",\n\t\t\t\t\t\t\t\t\"type\": \"LinkedServiceReference\"\n\t\t\t\t\t\t\t}\n\t\t\t\t\t\t},\n\t\t\t\t\t\t{\n\t\t\t\t\t\t\t\"name\": \"Set Worker State\",\n\t\t\t\t\t\t\t\"description\": \"Set the bool state of the Worker pipeline to be used by the Until and If expressions. True = Complete, False = Running.\",\n\t\t\t\t\t\t\t\"type\": \"SetVariable\",\n\t\t\t\t\t\t\t\"dependsOn\": [\n\t\t\t\t\t\t\t\t{\n\t\t\t\t\t\t\t\t\t\"activity\": \"Get Worker Pipeline Status\",\n\t\t\t\t\t\t\t\t\t\"dependencyConditions\": [\n\t\t\t\t\t\t\t\t\t\t\"Succeeded\"\n\t\t\t\t\t\t\t\t\t]\n\t\t\t\t\t\t\t\t}\n\t\t\t\t\t\t\t],\n\t\t\t\t\t\t\t\"userProperties\": [],\n\t\t\t\t\t\t\t\"typeProperties\": {\n\t\t\t\t\t\t\t\t\"variableName\": \"WorkerPipelineState\",\n\t\t\t\t\t\t\t\t\"value\": {\n\t\t\t\t\t\t\t\t\t\"value\": \"@equals('Complete',activity('Get Worker Pipeline Status').output.SimpleStatus)\",\n\t\t\t\t\t\t\t\t\t\"type\": \"Expression\"\n\t\t\t\t\t\t\t\t}\n\t\t\t\t\t\t\t}\n\t\t\t\t\t\t}\n\t\t\t\t\t],\n\t\t\t\t\t\"timeout\": \"0.04:00:00\"\n\t\t\t\t}\n\t\t\t},\n\t\t\t{\n\t\t\t\t\"name\": \"Set Pipeline Result\",\n\t\t\t\t\"description\": \"Receives the outcome from the function execution for a given processing pipeline and updates the current execution table with different pipelines status values depending on the result (case).\",\n\t\t\t\t\"type\": \"Switch\",\n\t\t\t\t\"dependsOn\": [\n\t\t\t\t\t{\n\t\t\t\t\t\t\"activity\": \"Wait Until Pipeline Completes\",\n\t\t\t\t\t\t\"dependencyConditions\": [\n\t\t\t\t\t\t\t\"Completed\"\n\t\t\t\t\t\t]\n\t\t\t\t\t}\n\t\t\t\t],\n\t\t\t\t\"userProperties\": [],\n\t\t\t\t\"typeProperties\": {\n\t\t\t\t\t\"on\": {\n\t\t\t\t\t\t\"value\": \"@activity('Get Worker Pipeline Status').output.ActualStatus\",\n\t\t\t\t\t\t\"type\": \"Expression\"\n\t\t\t\t\t},\n\t\t\t\t\t\"cases\": [\n\t\t\t\t\t\t{\n\t\t\t\t\t\t\t\"value\": \"Succeeded\",\n\t\t\t\t\t\t\t\"activities\": [\n\t\t\t\t\t\t\t\t{\n\t\t\t\t\t\t\t\t\t\"name\": \"Pipeline Status Succeeded\",\n\t\t\t\t\t\t\t\t\t\"description\": \"Updates the current execution table with a pipeline status of success if the function outcome is succeeded.\",\n\t\t\t\t\t\t\t\t\t\"type\": \"SqlServerStoredProcedure\",\n\t\t\t\t\t\t\t\t\t\"dependsOn\": [],\n\t\t\t\t\t\t\t\t\t\"policy\": {\n\t\t\t\t\t\t\t\t\t\t\"timeout\": \"0.00:01:00\",\n\t\t\t\t\t\t\t\t\t\t\"retry\": 2,\n\t\t\t\t\t\t\t\t\t\t\"retryIntervalInSeconds\": 5,\n\t\t\t\t\t\t\t\t\t\t\"secureOutput\": false,\n\t\t\t\t\t\t\t\t\t\t\"secureInput\": false\n\t\t\t\t\t\t\t\t\t},\n\t\t\t\t\t\t\t\t\t\"userProperties\": [],\n\t\t\t\t\t\t\t\t\t\"typeProperties\": {\n\t\t\t\t\t\t\t\t\t\t\"storedProcedureName\": \"[procfwk].[SetLogPipelineSuccess]\",\n\t\t\t\t\t\t\t\t\t\t\"storedProcedureParameters\": {\n\t\t\t\t\t\t\t\t\t\t\t\"ExecutionId\": {\n\t\t\t\t\t\t\t\t\t\t\t\t\"value\": {\n\t\t\t\t\t\t\t\t\t\t\t\t\t\"value\": \"@pipeline().parameters.executionId\",\n\t\t\t\t\t\t\t\t\t\t\t\t\t\"type\": \"Expression\"\n\t\t\t\t\t\t\t\t\t\t\t\t},\n\t\t\t\t\t\t\t\t\t\t\t\t\"type\": \"Guid\"\n\t\t\t\t\t\t\t\t\t\t\t},\n\t\t\t\t\t\t\t\t\t\t\t\"PipelineId\": {\n\t\t\t\t\t\t\t\t\t\t\t\t\"value\": {\n\t\t\t\t\t\t\t\t\t\t\t\t\t\"value\": \"@pipeline().parameters.pipelineId\",\n\t\t\t\t\t\t\t\t\t\t\t\t\t\"type\": \"Expression\"\n\t\t\t\t\t\t\t\t\t\t\t\t},\n\t\t\t\t\t\t\t\t\t\t\t\t\"type\": \"Int32\"\n\t\t\t\t\t\t\t\t\t\t\t},\n\t\t\t\t\t\t\t\t\t\t\t\"StageId\": {\n\t\t\t\t\t\t\t\t\t\t\t\t\"value\": {\n\t\t\t\t\t\t\t\t\t\t\t\t\t\"value\": \"@pipeline().parameters.stageId\",\n\t\t\t\t\t\t\t\t\t\t\t\t\t\"type\": \"Expression\"\n\t\t\t\t\t\t\t\t\t\t\t\t},\n\t\t\t\t\t\t\t\t\t\t\t\t\"type\": \"Int32\"\n\t\t\t\t\t\t\t\t\t\t\t}\n\t\t\t\t\t\t\t\t\t\t}\n\t\t\t\t\t\t\t\t\t},\n\t\t\t\t\t\t\t\t\t\"linkedServiceName\": {\n\t\t\t\t\t\t\t\t\t\t\"referenceName\": \"SupportDatabase\",\n\t\t\t\t\t\t\t\t\t\t\"type\": \"LinkedServiceReference\"\n\t\t\t\t\t\t\t\t\t}\n\t\t\t\t\t\t\t\t}\n\t\t\t\t\t\t\t]\n\t\t\t\t\t\t},\n\t\t\t\t\t\t{\n\t\t\t\t\t\t\t\"value\": \"Failed\",\n\t\t\t\t\t\t\t\"activities\": [\n\t\t\t\t\t\t\t\t{\n\t\t\t\t\t\t\t\t\t\"name\": \"Pipeline Status Failed\",\n\t\t\t\t\t\t\t\t\t\"description\": \"Updates the current execution table with a pipeline status of failed if the function outcome is failed. Also blocks pipelines in the downstream execution stage.\",\n\t\t\t\t\t\t\t\t\t\"type\": \"SqlServerStoredProcedure\",\n\t\t\t\t\t\t\t\t\t\"dependsOn\": [],\n\t\t\t\t\t\t\t\t\t\"policy\": {\n\t\t\t\t\t\t\t\t\t\t\"timeout\": \"0.00:01:00\",\n\t\t\t\t\t\t\t\t\t\t\"retry\": 2,\n\t\t\t\t\t\t\t\t\t\t\"retryIntervalInSeconds\": 5,\n\t\t\t\t\t\t\t\t\t\t\"secureOutput\": false,\n\t\t\t\t\t\t\t\t\t\t\"secureInput\": false\n\t\t\t\t\t\t\t\t\t},\n\t\t\t\t\t\t\t\t\t\"userProperties\": [],\n\t\t\t\t\t\t\t\t\t\"typeProperties\": {\n\t\t\t\t\t\t\t\t\t\t\"storedProcedureName\": \"[procfwk].[SetLogPipelineFailed]\",\n\t\t\t\t\t\t\t\t\t\t\"storedProcedureParameters\": {\n\t\t\t\t\t\t\t\t\t\t\t\"ExecutionId\": {\n\t\t\t\t\t\t\t\t\t\t\t\t\"value\": {\n\t\t\t\t\t\t\t\t\t\t\t\t\t\"value\": \"@pipeline().parameters.executionId\",\n\t\t\t\t\t\t\t\t\t\t\t\t\t\"type\": \"Expression\"\n\t\t\t\t\t\t\t\t\t\t\t\t},\n\t\t\t\t\t\t\t\t\t\t\t\t\"type\": \"Guid\"\n\t\t\t\t\t\t\t\t\t\t\t},\n\t\t\t\t\t\t\t\t\t\t\t\"PipelineId\": {\n\t\t\t\t\t\t\t\t\t\t\t\t\"value\": {\n\t\t\t\t\t\t\t\t\t\t\t\t\t\"value\": \"@pipeline().parameters.pipelineId\",\n\t\t\t\t\t\t\t\t\t\t\t\t\t\"type\": \"Expression\"\n\t\t\t\t\t\t\t\t\t\t\t\t},\n\t\t\t\t\t\t\t\t\t\t\t\t\"type\": \"Int32\"\n\t\t\t\t\t\t\t\t\t\t\t},\n\t\t\t\t\t\t\t\t\t\t\t\"RunId\": {\n\t\t\t\t\t\t\t\t\t\t\t\t\"value\": {\n\t\t\t\t\t\t\t\t\t\t\t\t\t\"value\": \"@variables('WorkerRunId')\",\n\t\t\t\t\t\t\t\t\t\t\t\t\t\"type\": \"Expression\"\n\t\t\t\t\t\t\t\t\t\t\t\t},\n\t\t\t\t\t\t\t\t\t\t\t\t\"type\": \"Guid\"\n\t\t\t\t\t\t\t\t\t\t\t},\n\t\t\t\t\t\t\t\t\t\t\t\"StageId\": {\n\t\t\t\t\t\t\t\t\t\t\t\t\"value\": {\n\t\t\t\t\t\t\t\t\t\t\t\t\t\"value\": \"@pipeline().parameters.stageId\",\n\t\t\t\t\t\t\t\t\t\t\t\t\t\"type\": \"Expression\"\n\t\t\t\t\t\t\t\t\t\t\t\t},\n\t\t\t\t\t\t\t\t\t\t\t\t\"type\": \"Int32\"\n\t\t\t\t\t\t\t\t\t\t\t}\n\t\t\t\t\t\t\t\t\t\t}\n\t\t\t\t\t\t\t\t\t},\n\t\t\t\t\t\t\t\t\t\"linkedServiceName\": {\n\t\t\t\t\t\t\t\t\t\t\"referenceName\": \"SupportDatabase\",\n\t\t\t\t\t\t\t\t\t\t\"type\": \"LinkedServiceReference\"\n\t\t\t\t\t\t\t\t\t}\n\t\t\t\t\t\t\t\t},\n\t\t\t\t\t\t\t\t{\n\t\t\t\t\t\t\t\t\t\"name\": \"Get Worker Pipeline Error Details\",\n\t\t\t\t\t\t\t\t\t\"description\": \"Get the activity error details for the run ID of the worker pipeline called. Returns an array of all errors.\",\n\t\t\t\t\t\t\t\t\t\"type\": \"AzureFunctionActivity\",\n\t\t\t\t\t\t\t\t\t\"dependsOn\": [],\n\t\t\t\t\t\t\t\t\t\"policy\": {\n\t\t\t\t\t\t\t\t\t\t\"timeout\": \"0.00:10:00\",\n\t\t\t\t\t\t\t\t\t\t\"retry\": 0,\n\t\t\t\t\t\t\t\t\t\t\"retryIntervalInSeconds\": 30,\n\t\t\t\t\t\t\t\t\t\t\"secureOutput\": false,\n\t\t\t\t\t\t\t\t\t\t\"secureInput\": true\n\t\t\t\t\t\t\t\t\t},\n\t\t\t\t\t\t\t\t\t\"userProperties\": [],\n\t\t\t\t\t\t\t\t\t\"typeProperties\": {\n\t\t\t\t\t\t\t\t\t\t\"functionName\": \"GetActivityErrors\",\n\t\t\t\t\t\t\t\t\t\t\"method\": \"POST\",\n\t\t\t\t\t\t\t\t\t\t\"body\": {\n\t\t\t\t\t\t\t\t\t\t\t\"value\": \"@concat('\\n{\\n \\\"tenantId\\\": \\\"',variables('WorkerCoreDetails')[0].tenantId,'\\\",\\n \\\"applicationId\\\": \\\"',variables('WorkerCoreDetails')[0].applicationId,'\\\",\\n \\\"authenticationKey\\\": \\\"',variables('WorkerCoreDetails')[0].authenticationKey,'\\\",\\n \\\"subscriptionId\\\": \\\"',variables('WorkerCoreDetails')[0].subscriptionId,'\\\",\\n \\\"resourceGroupName\\\": \\\"',variables('WorkerCoreDetails')[0].resourceGroupName,'\\\",\\n\\t\\\"orchestratorName\\\": \\\"',variables('WorkerCoreDetails')[0].orchestratorName,'\\\",\\n \\\"orchestratorType\\\": \\\"',variables('WorkerCoreDetails')[0].orchestratorType,'\\\",\\n \\\"pipelineName\\\": \\\"',variables('WorkerCoreDetails')[0].pipelineName,'\\\",\\n \\\"runId\\\": \\\"',variables('WorkerRunId'),'\\\"\\n}')\",\n\t\t\t\t\t\t\t\t\t\t\t\"type\": \"Expression\"\n\t\t\t\t\t\t\t\t\t\t}\n\t\t\t\t\t\t\t\t\t},\n\t\t\t\t\t\t\t\t\t\"linkedServiceName\": {\n\t\t\t\t\t\t\t\t\t\t\"referenceName\": \"FrameworkFunctions\",\n\t\t\t\t\t\t\t\t\t\t\"type\": \"LinkedServiceReference\"\n\t\t\t\t\t\t\t\t\t}\n\t\t\t\t\t\t\t\t},\n\t\t\t\t\t\t\t\t{\n\t\t\t\t\t\t\t\t\t\"name\": \"Log Error Details\",\n\t\t\t\t\t\t\t\t\t\"description\": \"Parses pipeline error details and persists them to the metadata database error log table.\",\n\t\t\t\t\t\t\t\t\t\"type\": \"SqlServerStoredProcedure\",\n\t\t\t\t\t\t\t\t\t\"dependsOn\": [\n\t\t\t\t\t\t\t\t\t\t{\n\t\t\t\t\t\t\t\t\t\t\t\"activity\": \"Get Worker Pipeline Error Details\",\n\t\t\t\t\t\t\t\t\t\t\t\"dependencyConditions\": [\n\t\t\t\t\t\t\t\t\t\t\t\t\"Succeeded\"\n\t\t\t\t\t\t\t\t\t\t\t]\n\t\t\t\t\t\t\t\t\t\t}\n\t\t\t\t\t\t\t\t\t],\n\t\t\t\t\t\t\t\t\t\"policy\": {\n\t\t\t\t\t\t\t\t\t\t\"timeout\": \"0.00:01:00\",\n\t\t\t\t\t\t\t\t\t\t\"retry\": 2,\n\t\t\t\t\t\t\t\t\t\t\"retryIntervalInSeconds\": 5,\n\t\t\t\t\t\t\t\t\t\t\"secureOutput\": false,\n\t\t\t\t\t\t\t\t\t\t\"secureInput\": false\n\t\t\t\t\t\t\t\t\t},\n\t\t\t\t\t\t\t\t\t\"userProperties\": [],\n\t\t\t\t\t\t\t\t\t\"typeProperties\": {\n\t\t\t\t\t\t\t\t\t\t\"storedProcedureName\": \"[procfwk].[SetErrorLogDetails]\",\n\t\t\t\t\t\t\t\t\t\t\"storedProcedureParameters\": {\n\t\t\t\t\t\t\t\t\t\t\t\"JsonErrorDetails\": {\n\t\t\t\t\t\t\t\t\t\t\t\t\"value\": {\n\t\t\t\t\t\t\t\t\t\t\t\t\t\"value\": \"@string(activity('Get Worker Pipeline Error Details').output)\",\n\t\t\t\t\t\t\t\t\t\t\t\t\t\"type\": \"Expression\"\n\t\t\t\t\t\t\t\t\t\t\t\t},\n\t\t\t\t\t\t\t\t\t\t\t\t\"type\": \"String\"\n\t\t\t\t\t\t\t\t\t\t\t},\n\t\t\t\t\t\t\t\t\t\t\t\"LocalExecutionId\": {\n\t\t\t\t\t\t\t\t\t\t\t\t\"value\": {\n\t\t\t\t\t\t\t\t\t\t\t\t\t\"value\": \"@pipeline().parameters.executionId\",\n\t\t\t\t\t\t\t\t\t\t\t\t\t\"type\": \"Expression\"\n\t\t\t\t\t\t\t\t\t\t\t\t},\n\t\t\t\t\t\t\t\t\t\t\t\t\"type\": \"Guid\"\n\t\t\t\t\t\t\t\t\t\t\t}\n\t\t\t\t\t\t\t\t\t\t}\n\t\t\t\t\t\t\t\t\t},\n\t\t\t\t\t\t\t\t\t\"linkedServiceName\": {\n\t\t\t\t\t\t\t\t\t\t\"referenceName\": \"SupportDatabase\",\n\t\t\t\t\t\t\t\t\t\t\"type\": \"LinkedServiceReference\"\n\t\t\t\t\t\t\t\t\t}\n\t\t\t\t\t\t\t\t}\n\t\t\t\t\t\t\t]\n\t\t\t\t\t\t},\n\t\t\t\t\t\t{\n\t\t\t\t\t\t\t\"value\": \"Cancelled\",\n\t\t\t\t\t\t\t\"activities\": [\n\t\t\t\t\t\t\t\t{\n\t\t\t\t\t\t\t\t\t\"name\": \"Pipeline Status Cancelled\",\n\t\t\t\t\t\t\t\t\t\"description\": \"Updates the current execution table with a pipeline status of cancelled if the function outcome is cancelled.\",\n\t\t\t\t\t\t\t\t\t\"type\": \"SqlServerStoredProcedure\",\n\t\t\t\t\t\t\t\t\t\"dependsOn\": [],\n\t\t\t\t\t\t\t\t\t\"policy\": {\n\t\t\t\t\t\t\t\t\t\t\"timeout\": \"0.00:01:00\",\n\t\t\t\t\t\t\t\t\t\t\"retry\": 2,\n\t\t\t\t\t\t\t\t\t\t\"retryIntervalInSeconds\": 5,\n\t\t\t\t\t\t\t\t\t\t\"secureOutput\": false,\n\t\t\t\t\t\t\t\t\t\t\"secureInput\": false\n\t\t\t\t\t\t\t\t\t},\n\t\t\t\t\t\t\t\t\t\"userProperties\": [],\n\t\t\t\t\t\t\t\t\t\"typeProperties\": {\n\t\t\t\t\t\t\t\t\t\t\"storedProcedureName\": \"[procfwk].[SetLogPipelineCancelled]\",\n\t\t\t\t\t\t\t\t\t\t\"storedProcedureParameters\": {\n\t\t\t\t\t\t\t\t\t\t\t\"ExecutionId\": {\n\t\t\t\t\t\t\t\t\t\t\t\t\"value\": {\n\t\t\t\t\t\t\t\t\t\t\t\t\t\"value\": \"@pipeline().parameters.executionId\",\n\t\t\t\t\t\t\t\t\t\t\t\t\t\"type\": \"Expression\"\n\t\t\t\t\t\t\t\t\t\t\t\t},\n\t\t\t\t\t\t\t\t\t\t\t\t\"type\": \"Guid\"\n\t\t\t\t\t\t\t\t\t\t\t},\n\t\t\t\t\t\t\t\t\t\t\t\"PipelineId\": {\n\t\t\t\t\t\t\t\t\t\t\t\t\"value\": {\n\t\t\t\t\t\t\t\t\t\t\t\t\t\"value\": \"@pipeline().parameters.pipelineId\",\n\t\t\t\t\t\t\t\t\t\t\t\t\t\"type\": \"Expression\"\n\t\t\t\t\t\t\t\t\t\t\t\t},\n\t\t\t\t\t\t\t\t\t\t\t\t\"type\": \"Int32\"\n\t\t\t\t\t\t\t\t\t\t\t},\n\t\t\t\t\t\t\t\t\t\t\t\"StageId\": {\n\t\t\t\t\t\t\t\t\t\t\t\t\"value\": {\n\t\t\t\t\t\t\t\t\t\t\t\t\t\"value\": \"@pipeline().parameters.stageId\",\n\t\t\t\t\t\t\t\t\t\t\t\t\t\"type\": \"Expression\"\n\t\t\t\t\t\t\t\t\t\t\t\t},\n\t\t\t\t\t\t\t\t\t\t\t\t\"type\": \"Int32\"\n\t\t\t\t\t\t\t\t\t\t\t}\n\t\t\t\t\t\t\t\t\t\t}\n\t\t\t\t\t\t\t\t\t},\n\t\t\t\t\t\t\t\t\t\"linkedServiceName\": {\n\t\t\t\t\t\t\t\t\t\t\"referenceName\": \"SupportDatabase\",\n\t\t\t\t\t\t\t\t\t\t\"type\": \"LinkedServiceReference\"\n\t\t\t\t\t\t\t\t\t}\n\t\t\t\t\t\t\t\t}\n\t\t\t\t\t\t\t]\n\t\t\t\t\t\t}\n\t\t\t\t\t],\n\t\t\t\t\t\"defaultActivities\": [\n\t\t\t\t\t\t{\n\t\t\t\t\t\t\t\"name\": \"Pipeline Status Unknown\",\n\t\t\t\t\t\t\t\"description\": \"Updates the current execution table with a pipeline status of unknown if the function returns an unexpected outcome.\",\n\t\t\t\t\t\t\t\"type\": \"SqlServerStoredProcedure\",\n\t\t\t\t\t\t\t\"dependsOn\": [],\n\t\t\t\t\t\t\t\"policy\": {\n\t\t\t\t\t\t\t\t\"timeout\": \"0.00:01:00\",\n\t\t\t\t\t\t\t\t\"retry\": 2,\n\t\t\t\t\t\t\t\t\"retryIntervalInSeconds\": 5,\n\t\t\t\t\t\t\t\t\"secureOutput\": false,\n\t\t\t\t\t\t\t\t\"secureInput\": false\n\t\t\t\t\t\t\t},\n\t\t\t\t\t\t\t\"userProperties\": [],\n\t\t\t\t\t\t\t\"typeProperties\": {\n\t\t\t\t\t\t\t\t\"storedProcedureName\": \"[procfwk].[SetLogPipelineUnknown]\",\n\t\t\t\t\t\t\t\t\"storedProcedureParameters\": {\n\t\t\t\t\t\t\t\t\t\"ExecutionId\": {\n\t\t\t\t\t\t\t\t\t\t\"value\": {\n\t\t\t\t\t\t\t\t\t\t\t\"value\": \"@pipeline().parameters.executionId\",\n\t\t\t\t\t\t\t\t\t\t\t\"type\": \"Expression\"\n\t\t\t\t\t\t\t\t\t\t},\n\t\t\t\t\t\t\t\t\t\t\"type\": \"Guid\"\n\t\t\t\t\t\t\t\t\t},\n\t\t\t\t\t\t\t\t\t\"PipelineId\": {\n\t\t\t\t\t\t\t\t\t\t\"value\": {\n\t\t\t\t\t\t\t\t\t\t\t\"value\": \"@pipeline().parameters.pipelineId\",\n\t\t\t\t\t\t\t\t\t\t\t\"type\": \"Expression\"\n\t\t\t\t\t\t\t\t\t\t},\n\t\t\t\t\t\t\t\t\t\t\"type\": \"Int32\"\n\t\t\t\t\t\t\t\t\t},\n\t\t\t\t\t\t\t\t\t\"StageId\": {\n\t\t\t\t\t\t\t\t\t\t\"value\": {\n\t\t\t\t\t\t\t\t\t\t\t\"value\": \"@pipeline().parameters.stageId\",\n\t\t\t\t\t\t\t\t\t\t\t\"type\": \"Expression\"\n\t\t\t\t\t\t\t\t\t\t},\n\t\t\t\t\t\t\t\t\t\t\"type\": \"Int32\"\n\t\t\t\t\t\t\t\t\t}\n\t\t\t\t\t\t\t\t}\n\t\t\t\t\t\t\t},\n\t\t\t\t\t\t\t\"linkedServiceName\": {\n\t\t\t\t\t\t\t\t\"referenceName\": \"SupportDatabase\",\n\t\t\t\t\t\t\t\t\"type\": \"LinkedServiceReference\"\n\t\t\t\t\t\t\t}\n\t\t\t\t\t\t}\n\t\t\t\t\t]\n\t\t\t\t}\n\t\t\t},\n\t\t\t{\n\t\t\t\t\"name\": \"Get Wait Duration\",\n\t\t\t\t\"description\": \"Return wait duration in seconds from database properties table to be used during each Until iteration when the Worker pipeline is still running.\",\n\t\t\t\t\"type\": \"Lookup\",\n\t\t\t\t\"dependsOn\": [],\n\t\t\t\t\"policy\": {\n\t\t\t\t\t\"timeout\": \"0.00:10:00\",\n\t\t\t\t\t\"retry\": 0,\n\t\t\t\t\t\"retryIntervalInSeconds\": 30,\n\t\t\t\t\t\"secureOutput\": false,\n\t\t\t\t\t\"secureInput\": false\n\t\t\t\t},\n\t\t\t\t\"userProperties\": [],\n\t\t\t\t\"typeProperties\": {\n\t\t\t\t\t\"source\": {\n\t\t\t\t\t\t\"type\": \"AzureSqlSource\",\n\t\t\t\t\t\t\"sqlReaderStoredProcedureName\": \"[procfwk].[GetPropertyValue]\",\n\t\t\t\t\t\t\"storedProcedureParameters\": {\n\t\t\t\t\t\t\t\"PropertyName\": {\n\t\t\t\t\t\t\t\t\"type\": \"String\",\n\t\t\t\t\t\t\t\t\"value\": \"PipelineStatusCheckDuration\"\n\t\t\t\t\t\t\t}\n\t\t\t\t\t\t},\n\t\t\t\t\t\t\"queryTimeout\": \"02:00:00\",\n\t\t\t\t\t\t\"partitionOption\": \"None\"\n\t\t\t\t\t},\n\t\t\t\t\t\"dataset\": {\n\t\t\t\t\t\t\"referenceName\": \"GetSetMetadata\",\n\t\t\t\t\t\t\"type\": \"DatasetReference\"\n\t\t\t\t\t}\n\t\t\t\t}\n\t\t\t},\n\t\t\t{\n\t\t\t\t\"name\": \"Set Run Id\",\n\t\t\t\t\"description\": \"Set local variable from activity output once for value reuse in downstream activities.\",\n\t\t\t\t\"type\": \"SetVariable\",\n\t\t\t\t\"dependsOn\": [\n\t\t\t\t\t{\n\t\t\t\t\t\t\"activity\": \"Execute Worker Pipeline\",\n\t\t\t\t\t\t\"dependencyConditions\": [\n\t\t\t\t\t\t\t\"Succeeded\"\n\t\t\t\t\t\t]\n\t\t\t\t\t}\n\t\t\t\t],\n\t\t\t\t\"userProperties\": [],\n\t\t\t\t\"typeProperties\": {\n\t\t\t\t\t\"variableName\": \"WorkerRunId\",\n\t\t\t\t\t\"value\": {\n\t\t\t\t\t\t\"value\": \"@activity('Execute Worker Pipeline').output.RunId\",\n\t\t\t\t\t\t\"type\": \"Expression\"\n\t\t\t\t\t}\n\t\t\t\t}\n\t\t\t},\n\t\t\t{\n\t\t\t\t\"name\": \"Validate Pipeline\",\n\t\t\t\t\"description\": \"Query the target data factory and establish if the provided worker pipeline name is valid.\",\n\t\t\t\t\"type\": \"AzureFunctionActivity\",\n\t\t\t\t\"dependsOn\": [\n\t\t\t\t\t{\n\t\t\t\t\t\t\"activity\": \"Log Pipeline Validating\",\n\t\t\t\t\t\t\"dependencyConditions\": [\n\t\t\t\t\t\t\t\"Succeeded\"\n\t\t\t\t\t\t]\n\t\t\t\t\t},\n\t\t\t\t\t{\n\t\t\t\t\t\t\"activity\": \"Capture Worker Core Details as an Array\",\n\t\t\t\t\t\t\"dependencyConditions\": [\n\t\t\t\t\t\t\t\"Succeeded\"\n\t\t\t\t\t\t]\n\t\t\t\t\t}\n\t\t\t\t],\n\t\t\t\t\"policy\": {\n\t\t\t\t\t\"timeout\": \"0.00:10:00\",\n\t\t\t\t\t\"retry\": 0,\n\t\t\t\t\t\"retryIntervalInSeconds\": 30,\n\t\t\t\t\t\"secureOutput\": false,\n\t\t\t\t\t\"secureInput\": true\n\t\t\t\t},\n\t\t\t\t\"userProperties\": [],\n\t\t\t\t\"typeProperties\": {\n\t\t\t\t\t\"functionName\": \"ValidatePipeline\",\n\t\t\t\t\t\"method\": \"POST\",\n\t\t\t\t\t\"body\": {\n\t\t\t\t\t\t\"value\": \"@concat('\\n{\\n \\\"tenantId\\\": \\\"',variables('WorkerCoreDetails')[0].tenantId,'\\\",\\n \\\"applicationId\\\": \\\"',variables('WorkerCoreDetails')[0].applicationId,'\\\",\\n \\\"authenticationKey\\\": \\\"',variables('WorkerCoreDetails')[0].authenticationKey,'\\\",\\n \\\"subscriptionId\\\": \\\"',variables('WorkerCoreDetails')[0].subscriptionId,'\\\",\\n \\\"resourceGroupName\\\": \\\"',variables('WorkerCoreDetails')[0].resourceGroupName,'\\\",\\n\\t\\\"orchestratorName\\\": \\\"',variables('WorkerCoreDetails')[0].orchestratorName,'\\\",\\n \\\"orchestratorType\\\": \\\"',variables('WorkerCoreDetails')[0].orchestratorType,'\\\",\\n \\\"pipelineName\\\": \\\"',variables('WorkerCoreDetails')[0].pipelineName,'\\\"\\n}')\",\n\t\t\t\t\t\t\"type\": \"Expression\"\n\t\t\t\t\t}\n\t\t\t\t},\n\t\t\t\t\"linkedServiceName\": {\n\t\t\t\t\t\"referenceName\": \"FrameworkFunctions\",\n\t\t\t\t\t\"type\": \"LinkedServiceReference\"\n\t\t\t\t}\n\t\t\t},\n\t\t\t{\n\t\t\t\t\"name\": \"Is Target Worker Validate\",\n\t\t\t\t\"description\": \"True = the worker pipeline name is valid.\\nFalse = the worker pipeline name is invalid. Raise an exception.\",\n\t\t\t\t\"type\": \"IfCondition\",\n\t\t\t\t\"dependsOn\": [\n\t\t\t\t\t{\n\t\t\t\t\t\t\"activity\": \"Validate Pipeline\",\n\t\t\t\t\t\t\"dependencyConditions\": [\n\t\t\t\t\t\t\t\"Succeeded\"\n\t\t\t\t\t\t]\n\t\t\t\t\t}\n\t\t\t\t],\n\t\t\t\t\"userProperties\": [],\n\t\t\t\t\"typeProperties\": {\n\t\t\t\t\t\"expression\": {\n\t\t\t\t\t\t\"value\": \"@bool(activity('Validate Pipeline').output.PipelineExists)\",\n\t\t\t\t\t\t\"type\": \"Expression\"\n\t\t\t\t\t},\n\t\t\t\t\t\"ifFalseActivities\": [\n\t\t\t\t\t\t{\n\t\t\t\t\t\t\t\"name\": \"Throw Exception - Invalid Infant\",\n\t\t\t\t\t\t\t\"description\": \"Throw an exception with details about the invalid worker pipeline name.\",\n\t\t\t\t\t\t\t\"type\": \"ExecutePipeline\",\n\t\t\t\t\t\t\t\"dependsOn\": [],\n\t\t\t\t\t\t\t\"userProperties\": [],\n\t\t\t\t\t\t\t\"typeProperties\": {\n\t\t\t\t\t\t\t\t\"pipeline\": {\n\t\t\t\t\t\t\t\t\t\"referenceName\": \"Throw Exception\",\n\t\t\t\t\t\t\t\t\t\"type\": \"PipelineReference\"\n\t\t\t\t\t\t\t\t},\n\t\t\t\t\t\t\t\t\"waitOnCompletion\": true,\n\t\t\t\t\t\t\t\t\"parameters\": {\n\t\t\t\t\t\t\t\t\t\"Message\": {\n\t\t\t\t\t\t\t\t\t\t\"value\": \"@concat('Worker pipeline [',variables('WorkerCoreDetails')[0].pipelineName,'] is not valid in target Orchestrator [',variables('WorkerCoreDetails')[0].orchestratorName,']')\",\n\t\t\t\t\t\t\t\t\t\t\"type\": \"Expression\"\n\t\t\t\t\t\t\t\t\t}\n\t\t\t\t\t\t\t\t}\n\t\t\t\t\t\t\t}\n\t\t\t\t\t\t},\n\t\t\t\t\t\t{\n\t\t\t\t\t\t\t\"name\": \"Update Execution With Invalid Worker\",\n\t\t\t\t\t\t\t\"description\": \"Update the current execution table with an informed status for the worker pipeline that couldn't be executed.\",\n\t\t\t\t\t\t\t\"type\": \"SqlServerStoredProcedure\",\n\t\t\t\t\t\t\t\"dependsOn\": [],\n\t\t\t\t\t\t\t\"policy\": {\n\t\t\t\t\t\t\t\t\"timeout\": \"7.00:00:00\",\n\t\t\t\t\t\t\t\t\"retry\": 0,\n\t\t\t\t\t\t\t\t\"retryIntervalInSeconds\": 30,\n\t\t\t\t\t\t\t\t\"secureOutput\": false,\n\t\t\t\t\t\t\t\t\"secureInput\": false\n\t\t\t\t\t\t\t},\n\t\t\t\t\t\t\t\"userProperties\": [],\n\t\t\t\t\t\t\t\"typeProperties\": {\n\t\t\t\t\t\t\t\t\"storedProcedureName\": \"[procfwk].[SetLogActivityFailed]\",\n\t\t\t\t\t\t\t\t\"storedProcedureParameters\": {\n\t\t\t\t\t\t\t\t\t\"CallingActivity\": {\n\t\t\t\t\t\t\t\t\t\t\"value\": \"InvalidPipelineName\",\n\t\t\t\t\t\t\t\t\t\t\"type\": \"String\"\n\t\t\t\t\t\t\t\t\t},\n\t\t\t\t\t\t\t\t\t\"ExecutionId\": {\n\t\t\t\t\t\t\t\t\t\t\"value\": {\n\t\t\t\t\t\t\t\t\t\t\t\"value\": \"@pipeline().parameters.ExecutionId\",\n\t\t\t\t\t\t\t\t\t\t\t\"type\": \"Expression\"\n\t\t\t\t\t\t\t\t\t\t},\n\t\t\t\t\t\t\t\t\t\t\"type\": \"Guid\"\n\t\t\t\t\t\t\t\t\t},\n\t\t\t\t\t\t\t\t\t\"PipelineId\": {\n\t\t\t\t\t\t\t\t\t\t\"value\": {\n\t\t\t\t\t\t\t\t\t\t\t\"value\": \"@pipeline().parameters.pipelineId\",\n\t\t\t\t\t\t\t\t\t\t\t\"type\": \"Expression\"\n\t\t\t\t\t\t\t\t\t\t},\n\t\t\t\t\t\t\t\t\t\t\"type\": \"Int32\"\n\t\t\t\t\t\t\t\t\t},\n\t\t\t\t\t\t\t\t\t\"StageId\": {\n\t\t\t\t\t\t\t\t\t\t\"value\": {\n\t\t\t\t\t\t\t\t\t\t\t\"value\": \"@pipeline().parameters.StageId\",\n\t\t\t\t\t\t\t\t\t\t\t\"type\": \"Expression\"\n\t\t\t\t\t\t\t\t\t\t},\n\t\t\t\t\t\t\t\t\t\t\"type\": \"Int32\"\n\t\t\t\t\t\t\t\t\t}\n\t\t\t\t\t\t\t\t}\n\t\t\t\t\t\t\t},\n\t\t\t\t\t\t\t\"linkedServiceName\": {\n\t\t\t\t\t\t\t\t\"referenceName\": \"SupportDatabase\",\n\t\t\t\t\t\t\t\t\"type\": \"LinkedServiceReference\"\n\t\t\t\t\t\t\t}\n\t\t\t\t\t\t}\n\t\t\t\t\t]\n\t\t\t\t}\n\t\t\t},\n\t\t\t{\n\t\t\t\t\"name\": \"Log Validate Function Activity Failure\",\n\t\t\t\t\"description\": \"Handle true failures from calling out to the Azure Function and update the current execution table accordingly so a restart can occur.\",\n\t\t\t\t\"type\": \"SqlServerStoredProcedure\",\n\t\t\t\t\"dependsOn\": [\n\t\t\t\t\t{\n\t\t\t\t\t\t\"activity\": \"Validate Pipeline\",\n\t\t\t\t\t\t\"dependencyConditions\": [\n\t\t\t\t\t\t\t\"Failed\"\n\t\t\t\t\t\t]\n\t\t\t\t\t}\n\t\t\t\t],\n\t\t\t\t\"policy\": {\n\t\t\t\t\t\"timeout\": \"0.00:10:00\",\n\t\t\t\t\t\"retry\": 0,\n\t\t\t\t\t\"retryIntervalInSeconds\": 30,\n\t\t\t\t\t\"secureOutput\": false,\n\t\t\t\t\t\"secureInput\": false\n\t\t\t\t},\n\t\t\t\t\"userProperties\": [],\n\t\t\t\t\"typeProperties\": {\n\t\t\t\t\t\"storedProcedureName\": \"[procfwk].[SetLogActivityFailed]\",\n\t\t\t\t\t\"storedProcedureParameters\": {\n\t\t\t\t\t\t\"ExecutionId\": {\n\t\t\t\t\t\t\t\"value\": {\n\t\t\t\t\t\t\t\t\"value\": \"@pipeline().parameters.ExecutionId\",\n\t\t\t\t\t\t\t\t\"type\": \"Expression\"\n\t\t\t\t\t\t\t},\n\t\t\t\t\t\t\t\"type\": \"Guid\"\n\t\t\t\t\t\t},\n\t\t\t\t\t\t\"PipelineId\": {\n\t\t\t\t\t\t\t\"value\": {\n\t\t\t\t\t\t\t\t\"value\": \"@pipeline().parameters.pipelineId\",\n\t\t\t\t\t\t\t\t\"type\": \"Expression\"\n\t\t\t\t\t\t\t},\n\t\t\t\t\t\t\t\"type\": \"Int32\"\n\t\t\t\t\t\t},\n\t\t\t\t\t\t\"StageId\": {\n\t\t\t\t\t\t\t\"value\": {\n\t\t\t\t\t\t\t\t\"value\": \"@pipeline().parameters.StageId\",\n\t\t\t\t\t\t\t\t\"type\": \"Expression\"\n\t\t\t\t\t\t\t},\n\t\t\t\t\t\t\t\"type\": \"Int32\"\n\t\t\t\t\t\t},\n\t\t\t\t\t\t\"CallingActivity\": {\n\t\t\t\t\t\t\t\"value\": \"ValidatePipeline\",\n\t\t\t\t\t\t\t\"type\": \"String\"\n\t\t\t\t\t\t}\n\t\t\t\t\t}\n\t\t\t\t},\n\t\t\t\t\"linkedServiceName\": {\n\t\t\t\t\t\"referenceName\": \"SupportDatabase\",\n\t\t\t\t\t\"type\": \"LinkedServiceReference\"\n\t\t\t\t}\n\t\t\t},\n\t\t\t{\n\t\t\t\t\"name\": \"Log Pipeline Validating\",\n\t\t\t\t\"description\": \"Sets the current pipeline with a status of validating within the current execution database table.\",\n\t\t\t\t\"type\": \"SqlServerStoredProcedure\",\n\t\t\t\t\"dependsOn\": [],\n\t\t\t\t\"policy\": {\n\t\t\t\t\t\"timeout\": \"0.00:10:00\",\n\t\t\t\t\t\"retry\": 0,\n\t\t\t\t\t\"retryIntervalInSeconds\": 30,\n\t\t\t\t\t\"secureOutput\": false,\n\t\t\t\t\t\"secureInput\": false\n\t\t\t\t},\n\t\t\t\t\"userProperties\": [],\n\t\t\t\t\"typeProperties\": {\n\t\t\t\t\t\"storedProcedureName\": \"[procfwk].[SetLogPipelineValidating]\",\n\t\t\t\t\t\"storedProcedureParameters\": {\n\t\t\t\t\t\t\"ExecutionId\": {\n\t\t\t\t\t\t\t\"value\": {\n\t\t\t\t\t\t\t\t\"value\": \"@pipeline().parameters.ExecutionId\",\n\t\t\t\t\t\t\t\t\"type\": \"Expression\"\n\t\t\t\t\t\t\t},\n\t\t\t\t\t\t\t\"type\": \"Guid\"\n\t\t\t\t\t\t},\n\t\t\t\t\t\t\"PipelineId\": {\n\t\t\t\t\t\t\t\"value\": {\n\t\t\t\t\t\t\t\t\"value\": \"@pipeline().parameters.pipelineId\",\n\t\t\t\t\t\t\t\t\"type\": \"Expression\"\n\t\t\t\t\t\t\t},\n\t\t\t\t\t\t\t\"type\": \"Int32\"\n\t\t\t\t\t\t},\n\t\t\t\t\t\t\"StageId\": {\n\t\t\t\t\t\t\t\"value\": {\n\t\t\t\t\t\t\t\t\"value\": \"@pipeline().parameters.StageId\",\n\t\t\t\t\t\t\t\t\"type\": \"Expression\"\n\t\t\t\t\t\t\t},\n\t\t\t\t\t\t\t\"type\": \"Int32\"\n\t\t\t\t\t\t}\n\t\t\t\t\t}\n\t\t\t\t},\n\t\t\t\t\"linkedServiceName\": {\n\t\t\t\t\t\"referenceName\": \"SupportDatabase\",\n\t\t\t\t\t\"type\": \"LinkedServiceReference\"\n\t\t\t\t}\n\t\t\t},\n\t\t\t{\n\t\t\t\t\"name\": \"Get Worker Core Details\",\n\t\t\t\t\"description\": \"Return worker pipeline information for metadata database. Including target data factory, pipeline name and resource group. Return the SPN ID and Secret for the worker pipeline being executed. Called at this level as each pipeline can have a different SPN.\",\n\t\t\t\t\"type\": \"Lookup\",\n\t\t\t\t\"dependsOn\": [],\n\t\t\t\t\"policy\": {\n\t\t\t\t\t\"timeout\": \"0.00:10:00\",\n\t\t\t\t\t\"retry\": 0,\n\t\t\t\t\t\"retryIntervalInSeconds\": 30,\n\t\t\t\t\t\"secureOutput\": true,\n\t\t\t\t\t\"secureInput\": false\n\t\t\t\t},\n\t\t\t\t\"userProperties\": [],\n\t\t\t\t\"typeProperties\": {\n\t\t\t\t\t\"source\": {\n\t\t\t\t\t\t\"type\": \"AzureSqlSource\",\n\t\t\t\t\t\t\"sqlReaderStoredProcedureName\": \"[procfwk].[GetWorkerDetailsWrapper]\",\n\t\t\t\t\t\t\"storedProcedureParameters\": {\n\t\t\t\t\t\t\t\"ExecutionId\": {\n\t\t\t\t\t\t\t\t\"type\": \"Guid\",\n\t\t\t\t\t\t\t\t\"value\": {\n\t\t\t\t\t\t\t\t\t\"value\": \"@pipeline().parameters.executionId\",\n\t\t\t\t\t\t\t\t\t\"type\": \"Expression\"\n\t\t\t\t\t\t\t\t}\n\t\t\t\t\t\t\t},\n\t\t\t\t\t\t\t\"PipelineId\": {\n\t\t\t\t\t\t\t\t\"type\": \"Int32\",\n\t\t\t\t\t\t\t\t\"value\": {\n\t\t\t\t\t\t\t\t\t\"value\": \"@pipeline().parameters.pipelineId\",\n\t\t\t\t\t\t\t\t\t\"type\": \"Expression\"\n\t\t\t\t\t\t\t\t}\n\t\t\t\t\t\t\t},\n\t\t\t\t\t\t\t\"StageId\": {\n\t\t\t\t\t\t\t\t\"type\": \"Int32\",\n\t\t\t\t\t\t\t\t\"value\": {\n\t\t\t\t\t\t\t\t\t\"value\": \"@pipeline().parameters.stageId\",\n\t\t\t\t\t\t\t\t\t\"type\": \"Expression\"\n\t\t\t\t\t\t\t\t}\n\t\t\t\t\t\t\t}\n\t\t\t\t\t\t},\n\t\t\t\t\t\t\"queryTimeout\": \"02:00:00\",\n\t\t\t\t\t\t\"partitionOption\": \"None\"\n\t\t\t\t\t},\n\t\t\t\t\t\"dataset\": {\n\t\t\t\t\t\t\"referenceName\": \"GetSetMetadata\",\n\t\t\t\t\t\t\"type\": \"DatasetReference\"\n\t\t\t\t\t}\n\t\t\t\t}\n\t\t\t},\n\t\t\t{\n\t\t\t\t\"name\": \"Capture Worker Core Details as an Array\",\n\t\t\t\t\"description\": \"Add all worker pipeline details to a local variable array that can be accessed by each function call requiring the values.\",\n\t\t\t\t\"type\": \"SetVariable\",\n\t\t\t\t\"dependsOn\": [\n\t\t\t\t\t{\n\t\t\t\t\t\t\"activity\": \"Get Worker Core Details\",\n\t\t\t\t\t\t\"dependencyConditions\": [\n\t\t\t\t\t\t\t\"Succeeded\"\n\t\t\t\t\t\t]\n\t\t\t\t\t}\n\t\t\t\t],\n\t\t\t\t\"userProperties\": [],\n\t\t\t\t\"typeProperties\": {\n\t\t\t\t\t\"variableName\": \"WorkerCoreDetails\",\n\t\t\t\t\t\"value\": {\n\t\t\t\t\t\t\"value\": \"@array(activity('Get Worker Core Details').output.firstRow)\",\n\t\t\t\t\t\t\"type\": \"Expression\"\n\t\t\t\t\t}\n\t\t\t\t}\n\t\t\t}\n\t\t],\n\t\t\"parameters\": {\n\t\t\t\"executionId\": {\n\t\t\t\t\"type\": \"string\"\n\t\t\t},\n\t\t\t\"stageId\": {\n\t\t\t\t\"type\": \"int\"\n\t\t\t},\n\t\t\t\"pipelineId\": {\n\t\t\t\t\"type\": \"int\"\n\t\t\t}\n\t\t},\n\t\t\"variables\": {\n\t\t\t\"WorkerPipelineState\": {\n\t\t\t\t\"type\": \"Boolean\"\n\t\t\t},\n\t\t\t\"WorkerRunId\": {\n\t\t\t\t\"type\": \"String\"\n\t\t\t},\n\t\t\t\"WorkerCoreDetails\": {\n\t\t\t\t\"type\": \"Array\"\n\t\t\t}\n\t\t},\n\t\t\"folder\": {\n\t\t\t\"name\": \"_ProcFwk\"\n\t\t},\n\t\t\"annotations\": [\n\t\t\t\"procfwk\",\n\t\t\t\"Infant\"\n\t\t]\n\t}\n}\n"
},
{
"path": "DataFactory/pipeline/Check For Running Pipeline.json",
"content": "{\n\t\"name\": \"Check For Running Pipeline\",\n\t\"properties\": {\n\t\t\"description\": \"For a given pipeline and optional batch name establish if a pipeline run is already in progress. Throw an exception if it it.\",\n\t\t\"activities\": [\n\t\t\t{\n\t\t\t\t\"name\": \"Filter Running Pipelines\",\n\t\t\t\t\"description\": \"Filter the pipeline runs results for pipelines that exclude the current triggered run and that are currently running (in progress or queued).\",\n\t\t\t\t\"type\": \"Filter\",\n\t\t\t\t\"dependsOn\": [\n\t\t\t\t\t{\n\t\t\t\t\t\t\"activity\": \"Switch For Orchestrator Type\",\n\t\t\t\t\t\t\"dependencyConditions\": [\n\t\t\t\t\t\t\t\"Succeeded\"\n\t\t\t\t\t\t]\n\t\t\t\t\t}\n\t\t\t\t],\n\t\t\t\t\"userProperties\": [],\n\t\t\t\t\"typeProperties\": {\n\t\t\t\t\t\"items\": {\n\t\t\t\t\t\t\"value\": \"@variables('PipelineRuns')\",\n\t\t\t\t\t\t\"type\": \"Expression\"\n\t\t\t\t\t},\n\t\t\t\t\t\"condition\": {\n\t\t\t\t\t\t\"value\": \"@and(not(equals(item().runId,pipeline().parameters.ThisRunId)),or(equals(item().status,'InProgress'),equals(item().status,'Queued')))\",\n\t\t\t\t\t\t\"type\": \"Expression\"\n\t\t\t\t\t}\n\t\t\t\t}\n\t\t\t},\n\t\t\t{\n\t\t\t\t\"name\": \"Get Framework Orchestrator Details\",\n\t\t\t\t\"description\": \"Using the metadata orchestrators return details about the resource running the framework pipelines.\",\n\t\t\t\t\"type\": \"Lookup\",\n\t\t\t\t\"dependsOn\": [],\n\t\t\t\t\"policy\": {\n\t\t\t\t\t\"timeout\": \"0.00:10:00\",\n\t\t\t\t\t\"retry\": 0,\n\t\t\t\t\t\"retryIntervalInSeconds\": 30,\n\t\t\t\t\t\"secureOutput\": false,\n\t\t\t\t\t\"secureInput\": false\n\t\t\t\t},\n\t\t\t\t\"userProperties\": [],\n\t\t\t\t\"typeProperties\": {\n\t\t\t\t\t\"source\": {\n\t\t\t\t\t\t\"type\": \"AzureSqlSource\",\n\t\t\t\t\t\t\"sqlReaderStoredProcedureName\": \"[procfwk].[GetFrameworkOrchestratorDetails]\",\n\t\t\t\t\t\t\"storedProcedureParameters\": {\n\t\t\t\t\t\t\t\"CallingOrchestratorName\": {\n\t\t\t\t\t\t\t\t\"type\": \"String\",\n\t\t\t\t\t\t\t\t\"value\": {\n\t\t\t\t\t\t\t\t\t\"value\": \"@pipeline().DataFactory\",\n\t\t\t\t\t\t\t\t\t\"type\": \"Expression\"\n\t\t\t\t\t\t\t\t}\n\t\t\t\t\t\t\t}\n\t\t\t\t\t\t},\n\t\t\t\t\t\t\"queryTimeout\": \"02:00:00\",\n\t\t\t\t\t\t\"partitionOption\": \"None\"\n\t\t\t\t\t},\n\t\t\t\t\t\"dataset\": {\n\t\t\t\t\t\t\"referenceName\": \"GetSetMetadata\",\n\t\t\t\t\t\t\"type\": \"DatasetReference\"\n\t\t\t\t\t}\n\t\t\t\t}\n\t\t\t},\n\t\t\t{\n\t\t\t\t\"name\": \"Get Query Run Days Value\",\n\t\t\t\t\"description\": \"Using the metadata properties table return the run days value to provide the API request with a date range for pipeline executions.\",\n\t\t\t\t\"type\": \"Lookup\",\n\t\t\t\t\"dependsOn\": [],\n\t\t\t\t\"policy\": {\n\t\t\t\t\t\"timeout\": \"0.00:10:00\",\n\t\t\t\t\t\"retry\": 0,\n\t\t\t\t\t\"retryIntervalInSeconds\": 30,\n\t\t\t\t\t\"secureOutput\": false,\n\t\t\t\t\t\"secureInput\": false\n\t\t\t\t},\n\t\t\t\t\"userProperties\": [],\n\t\t\t\t\"typeProperties\": {\n\t\t\t\t\t\"source\": {\n\t\t\t\t\t\t\"type\": \"AzureSqlSource\",\n\t\t\t\t\t\t\"sqlReaderStoredProcedureName\": \"[procfwk].[GetPropertyValue]\",\n\t\t\t\t\t\t\"storedProcedureParameters\": {\n\t\t\t\t\t\t\t\"PropertyName\": {\n\t\t\t\t\t\t\t\t\"type\": \"String\",\n\t\t\t\t\t\t\t\t\"value\": \"PreviousPipelineRunsQueryRange\"\n\t\t\t\t\t\t\t}\n\t\t\t\t\t\t},\n\t\t\t\t\t\t\"queryTimeout\": \"02:00:00\",\n\t\t\t\t\t\t\"partitionOption\": \"None\"\n\t\t\t\t\t},\n\t\t\t\t\t\"dataset\": {\n\t\t\t\t\t\t\"referenceName\": \"GetSetMetadata\",\n\t\t\t\t\t\t\"type\": \"DatasetReference\"\n\t\t\t\t\t}\n\t\t\t\t}\n\t\t\t},\n\t\t\t{\n\t\t\t\t\"name\": \"If Pipeline Is Running\",\n\t\t\t\t\"description\": \"If the running pipeline count is greater than or equal to one.\\nTrue = raise an exception.\",\n\t\t\t\t\"type\": \"IfCondition\",\n\t\t\t\t\"dependsOn\": [\n\t\t\t\t\t{\n\t\t\t\t\t\t\"activity\": \"If Using Batch Executions\",\n\t\t\t\t\t\t\"dependencyConditions\": [\n\t\t\t\t\t\t\t\"Succeeded\"\n\t\t\t\t\t\t]\n\t\t\t\t\t}\n\t\t\t\t],\n\t\t\t\t\"userProperties\": [],\n\t\t\t\t\"typeProperties\": {\n\t\t\t\t\t\"expression\": {\n\t\t\t\t\t\t\"value\": \"@greaterOrEquals(int(variables('RunCount')),1)\",\n\t\t\t\t\t\t\"type\": \"Expression\"\n\t\t\t\t\t},\n\t\t\t\t\t\"ifTrueActivities\": [\n\t\t\t\t\t\t{\n\t\t\t\t\t\t\t\"name\": \"Throw Exception - Pipeline Running\",\n\t\t\t\t\t\t\t\"description\": \"Using the utils pipeline raise an exception to stop the new trigger while a run is already in progress.\",\n\t\t\t\t\t\t\t\"type\": \"ExecutePipeline\",\n\t\t\t\t\t\t\t\"dependsOn\": [],\n\t\t\t\t\t\t\t\"userProperties\": [],\n\t\t\t\t\t\t\t\"typeProperties\": {\n\t\t\t\t\t\t\t\t\"pipeline\": {\n\t\t\t\t\t\t\t\t\t\"referenceName\": \"Throw Exception\",\n\t\t\t\t\t\t\t\t\t\"type\": \"PipelineReference\"\n\t\t\t\t\t\t\t\t},\n\t\t\t\t\t\t\t\t\"waitOnCompletion\": true,\n\t\t\t\t\t\t\t\t\"parameters\": {\n\t\t\t\t\t\t\t\t\t\"Message\": {\n\t\t\t\t\t\t\t\t\t\t\"value\": \"@concat('Provided pipeline name (',pipeline().parameters.PipelineName,') still has a run in progress or queued given the query range parameters set in the properties table.')\",\n\t\t\t\t\t\t\t\t\t\t\"type\": \"Expression\"\n\t\t\t\t\t\t\t\t\t}\n\t\t\t\t\t\t\t\t}\n\t\t\t\t\t\t\t}\n\t\t\t\t\t\t}\n\t\t\t\t\t]\n\t\t\t\t}\n\t\t\t},\n\t\t\t{\n\t\t\t\t\"name\": \"Get Execution Batch Status\",\n\t\t\t\t\"description\": \"Using the metadata properties table return the flag to indicate if batch execution setting are enabled or disabled.\",\n\t\t\t\t\"type\": \"Lookup\",\n\t\t\t\t\"dependsOn\": [],\n\t\t\t\t\"policy\": {\n\t\t\t\t\t\"timeout\": \"0.00:10:00\",\n\t\t\t\t\t\"retry\": 0,\n\t\t\t\t\t\"retryIntervalInSeconds\": 30,\n\t\t\t\t\t\"secureOutput\": false,\n\t\t\t\t\t\"secureInput\": false\n\t\t\t\t},\n\t\t\t\t\"userProperties\": [],\n\t\t\t\t\"typeProperties\": {\n\t\t\t\t\t\"source\": {\n\t\t\t\t\t\t\"type\": \"AzureSqlSource\",\n\t\t\t\t\t\t\"sqlReaderStoredProcedureName\": \"[procfwk].[GetPropertyValue]\",\n\t\t\t\t\t\t\"storedProcedureParameters\": {\n\t\t\t\t\t\t\t\"PropertyName\": {\n\t\t\t\t\t\t\t\t\"type\": \"String\",\n\t\t\t\t\t\t\t\t\"value\": \"UseExecutionBatches\"\n\t\t\t\t\t\t\t}\n\t\t\t\t\t\t},\n\t\t\t\t\t\t\"queryTimeout\": \"02:00:00\",\n\t\t\t\t\t\t\"partitionOption\": \"None\"\n\t\t\t\t\t},\n\t\t\t\t\t\"dataset\": {\n\t\t\t\t\t\t\"referenceName\": \"GetSetMetadata\",\n\t\t\t\t\t\t\"type\": \"DatasetReference\"\n\t\t\t\t\t}\n\t\t\t\t}\n\t\t\t},\n\t\t\t{\n\t\t\t\t\"name\": \"If Using Batch Executions\",\n\t\t\t\t\"description\": \"True = batch executions are enabled.\\nFalse = batch execution are disabled.\",\n\t\t\t\t\"type\": \"IfCondition\",\n\t\t\t\t\"dependsOn\": [\n\t\t\t\t\t{\n\t\t\t\t\t\t\"activity\": \"Get Execution Batch Status\",\n\t\t\t\t\t\t\"dependencyConditions\": [\n\t\t\t\t\t\t\t\"Succeeded\"\n\t\t\t\t\t\t]\n\t\t\t\t\t},\n\t\t\t\t\t{\n\t\t\t\t\t\t\"activity\": \"Filter Running Pipelines\",\n\t\t\t\t\t\t\"dependencyConditions\": [\n\t\t\t\t\t\t\t\"Succeeded\"\n\t\t\t\t\t\t]\n\t\t\t\t\t}\n\t\t\t\t],\n\t\t\t\t\"userProperties\": [],\n\t\t\t\t\"typeProperties\": {\n\t\t\t\t\t\"expression\": {\n\t\t\t\t\t\t\"value\": \"@equals(activity('Get Execution Batch Status').output.firstRow.PropertyValue,string(1))\",\n\t\t\t\t\t\t\"type\": \"Expression\"\n\t\t\t\t\t},\n\t\t\t\t\t\"ifFalseActivities\": [\n\t\t\t\t\t\t{\n\t\t\t\t\t\t\t\"name\": \"Set Run Count Without Batch\",\n\t\t\t\t\t\t\t\"description\": \"Set the pipelines running count variable to be tested later.\",\n\t\t\t\t\t\t\t\"type\": \"SetVariable\",\n\t\t\t\t\t\t\t\"dependsOn\": [],\n\t\t\t\t\t\t\t\"userProperties\": [],\n\t\t\t\t\t\t\t\"typeProperties\": {\n\t\t\t\t\t\t\t\t\"variableName\": \"RunCount\",\n\t\t\t\t\t\t\t\t\"value\": {\n\t\t\t\t\t\t\t\t\t\"value\": \"@string(activity('Filter Running Pipelines').output.FilteredItemsCount)\",\n\t\t\t\t\t\t\t\t\t\"type\": \"Expression\"\n\t\t\t\t\t\t\t\t}\n\t\t\t\t\t\t\t}\n\t\t\t\t\t\t}\n\t\t\t\t\t],\n\t\t\t\t\t\"ifTrueActivities\": [\n\t\t\t\t\t\t{\n\t\t\t\t\t\t\t\"name\": \"Filter for Batch Name\",\n\t\t\t\t\t\t\t\"description\": \"Further filter the return pipeline runs for any running pipelines with the same batch name value.\",\n\t\t\t\t\t\t\t\"type\": \"Filter\",\n\t\t\t\t\t\t\t\"dependsOn\": [],\n\t\t\t\t\t\t\t\"userProperties\": [],\n\t\t\t\t\t\t\t\"typeProperties\": {\n\t\t\t\t\t\t\t\t\"items\": {\n\t\t\t\t\t\t\t\t\t\"value\": \"@activity('Filter Running Pipelines').output.value\",\n\t\t\t\t\t\t\t\t\t\"type\": \"Expression\"\n\t\t\t\t\t\t\t\t},\n\t\t\t\t\t\t\t\t\"condition\": {\n\t\t\t\t\t\t\t\t\t\"value\": \"@equals(item().parameters.BatchName,pipeline().parameters.BatchName)\",\n\t\t\t\t\t\t\t\t\t\"type\": \"Expression\"\n\t\t\t\t\t\t\t\t}\n\t\t\t\t\t\t\t}\n\t\t\t\t\t\t},\n\t\t\t\t\t\t{\n\t\t\t\t\t\t\t\"name\": \"Set Run Count for Batch\",\n\t\t\t\t\t\t\t\"description\": \"Set the resulting pipeline running count variable to be tested later.\",\n\t\t\t\t\t\t\t\"type\": \"SetVariable\",\n\t\t\t\t\t\t\t\"dependsOn\": [\n\t\t\t\t\t\t\t\t{\n\t\t\t\t\t\t\t\t\t\"activity\": \"Filter for Batch Name\",\n\t\t\t\t\t\t\t\t\t\"dependencyConditions\": [\n\t\t\t\t\t\t\t\t\t\t\"Succeeded\"\n\t\t\t\t\t\t\t\t\t]\n\t\t\t\t\t\t\t\t}\n\t\t\t\t\t\t\t],\n\t\t\t\t\t\t\t\"userProperties\": [],\n\t\t\t\t\t\t\t\"typeProperties\": {\n\t\t\t\t\t\t\t\t\"variableName\": \"RunCount\",\n\t\t\t\t\t\t\t\t\"value\": {\n\t\t\t\t\t\t\t\t\t\"value\": \"@string(activity('Filter for Batch Name').output.FilteredItemsCount)\",\n\t\t\t\t\t\t\t\t\t\"type\": \"Expression\"\n\t\t\t\t\t\t\t\t}\n\t\t\t\t\t\t\t}\n\t\t\t\t\t\t}\n\t\t\t\t\t]\n\t\t\t\t}\n\t\t\t},\n\t\t\t{\n\t\t\t\t\"name\": \"Set Subscription Id\",\n\t\t\t\t\"description\": \"Set the subscription Id value to a local variable for use in various downstream activities.\",\n\t\t\t\t\"type\": \"SetVariable\",\n\t\t\t\t\"dependsOn\": [\n\t\t\t\t\t{\n\t\t\t\t\t\t\"activity\": \"Get Framework Orchestrator Details\",\n\t\t\t\t\t\t\"dependencyConditions\": [\n\t\t\t\t\t\t\t\"Succeeded\"\n\t\t\t\t\t\t]\n\t\t\t\t\t}\n\t\t\t\t],\n\t\t\t\t\"userProperties\": [],\n\t\t\t\t\"typeProperties\": {\n\t\t\t\t\t\"variableName\": \"SubscriptionId\",\n\t\t\t\t\t\"value\": {\n\t\t\t\t\t\t\"value\": \"@activity('Get Framework Orchestrator Details').output.firstRow.SubscriptionId\",\n\t\t\t\t\t\t\"type\": \"Expression\"\n\t\t\t\t\t}\n\t\t\t\t}\n\t\t\t},\n\t\t\t{\n\t\t\t\t\"name\": \"Set Resource Group Name\",\n\t\t\t\t\"description\": \"Set the resource group name value to a local variable for use in various downstream activities.\",\n\t\t\t\t\"type\": \"SetVariable\",\n\t\t\t\t\"dependsOn\": [\n\t\t\t\t\t{\n\t\t\t\t\t\t\"activity\": \"Get Framework Orchestrator Details\",\n\t\t\t\t\t\t\"dependencyConditions\": [\n\t\t\t\t\t\t\t\"Succeeded\"\n\t\t\t\t\t\t]\n\t\t\t\t\t}\n\t\t\t\t],\n\t\t\t\t\"userProperties\": [],\n\t\t\t\t\"typeProperties\": {\n\t\t\t\t\t\"variableName\": \"ResourceGroupName\",\n\t\t\t\t\t\"value\": {\n\t\t\t\t\t\t\"value\": \"@activity('Get Framework Orchestrator Details').output.firstRow.ResourceGroupName\",\n\t\t\t\t\t\t\"type\": \"Expression\"\n\t\t\t\t\t}\n\t\t\t\t}\n\t\t\t},\n\t\t\t{\n\t\t\t\t\"name\": \"Set Orchestrator Type\",\n\t\t\t\t\"description\": \"Set the orchestrator type value to a local variable for use in various downstream activities.\",\n\t\t\t\t\"type\": \"SetVariable\",\n\t\t\t\t\"dependsOn\": [\n\t\t\t\t\t{\n\t\t\t\t\t\t\"activity\": \"Get Framework Orchestrator Details\",\n\t\t\t\t\t\t\"dependencyConditions\": [\n\t\t\t\t\t\t\t\"Succeeded\"\n\t\t\t\t\t\t]\n\t\t\t\t\t}\n\t\t\t\t],\n\t\t\t\t\"userProperties\": [],\n\t\t\t\t\"typeProperties\": {\n\t\t\t\t\t\"variableName\": \"OrchestratorType\",\n\t\t\t\t\t\"value\": {\n\t\t\t\t\t\t\"value\": \"@toUpper(activity('Get Framework Orchestrator Details').output.firstRow.OrchestratorType)\",\n\t\t\t\t\t\t\"type\": \"Expression\"\n\t\t\t\t\t}\n\t\t\t\t}\n\t\t\t},\n\t\t\t{\n\t\t\t\t\"name\": \"Switch For Orchestrator Type\",\n\t\t\t\t\"description\": \"Switch and handle requests for both Azure Data Factory (ADF) and Azure Synapse Analytics (SYN).\",\n\t\t\t\t\"type\": \"Switch\",\n\t\t\t\t\"dependsOn\": [\n\t\t\t\t\t{\n\t\t\t\t\t\t\"activity\": \"Set Orchestrator Type\",\n\t\t\t\t\t\t\"dependencyConditions\": [\n\t\t\t\t\t\t\t\"Succeeded\"\n\t\t\t\t\t\t]\n\t\t\t\t\t},\n\t\t\t\t\t{\n\t\t\t\t\t\t\"activity\": \"Set Query Run Days\",\n\t\t\t\t\t\t\"dependencyConditions\": [\n\t\t\t\t\t\t\t\"Succeeded\"\n\t\t\t\t\t\t]\n\t\t\t\t\t},\n\t\t\t\t\t{\n\t\t\t\t\t\t\"activity\": \"Set Resource Group Name\",\n\t\t\t\t\t\t\"dependencyConditions\": [\n\t\t\t\t\t\t\t\"Succeeded\"\n\t\t\t\t\t\t]\n\t\t\t\t\t},\n\t\t\t\t\t{\n\t\t\t\t\t\t\"activity\": \"Set Subscription Id\",\n\t\t\t\t\t\t\"dependencyConditions\": [\n\t\t\t\t\t\t\t\"Succeeded\"\n\t\t\t\t\t\t]\n\t\t\t\t\t}\n\t\t\t\t],\n\t\t\t\t\"userProperties\": [],\n\t\t\t\t\"typeProperties\": {\n\t\t\t\t\t\"on\": {\n\t\t\t\t\t\t\"value\": \"@variables('OrchestratorType')\",\n\t\t\t\t\t\t\"type\": \"Expression\"\n\t\t\t\t\t},\n\t\t\t\t\t\"cases\": [\n\t\t\t\t\t\t{\n\t\t\t\t\t\t\t\"value\": \"ADF\",\n\t\t\t\t\t\t\t\"activities\": [\n\t\t\t\t\t\t\t\t{\n\t\t\t\t\t\t\t\t\t\"name\": \"Check for Valid ADF Pipeline Name\",\n\t\t\t\t\t\t\t\t\t\"description\": \"Use the Azure Management API to return and establish if the framework pipeline exists in the target Data Factory instance, including being deployed.\",\n\t\t\t\t\t\t\t\t\t\"type\": \"WebActivity\",\n\t\t\t\t\t\t\t\t\t\"dependsOn\": [],\n\t\t\t\t\t\t\t\t\t\"policy\": {\n\t\t\t\t\t\t\t\t\t\t\"timeout\": \"7.00:00:00\",\n\t\t\t\t\t\t\t\t\t\t\"retry\": 0,\n\t\t\t\t\t\t\t\t\t\t\"retryIntervalInSeconds\": 30,\n\t\t\t\t\t\t\t\t\t\t\"secureOutput\": false,\n\t\t\t\t\t\t\t\t\t\t\"secureInput\": false\n\t\t\t\t\t\t\t\t\t},\n\t\t\t\t\t\t\t\t\t\"userProperties\": [],\n\t\t\t\t\t\t\t\t\t\"typeProperties\": {\n\t\t\t\t\t\t\t\t\t\t\"url\": {\n\t\t\t\t\t\t\t\t\t\t\t\"value\": \"https://management.azure.com/subscriptions/@{variables('SubscriptionId')}/resourceGroups/@{variables('ResourceGroupName')}/providers/Microsoft.DataFactory/factories/@{pipeline().DataFactory}/pipelines/@{pipeline().parameters.PipelineName}?api-version=2018-06-01\",\n\t\t\t\t\t\t\t\t\t\t\t\"type\": \"Expression\"\n\t\t\t\t\t\t\t\t\t\t},\n\t\t\t\t\t\t\t\t\t\t\"method\": \"GET\",\n\t\t\t\t\t\t\t\t\t\t\"authentication\": {\n\t\t\t\t\t\t\t\t\t\t\t\"type\": \"MSI\",\n\t\t\t\t\t\t\t\t\t\t\t\"resource\": \"https://management.core.windows.net/\"\n\t\t\t\t\t\t\t\t\t\t}\n\t\t\t\t\t\t\t\t\t}\n\t\t\t\t\t\t\t\t},\n\t\t\t\t\t\t\t\t{\n\t\t\t\t\t\t\t\t\t\"name\": \"Get ADF Pipeline Runs\",\n\t\t\t\t\t\t\t\t\t\"description\": \"Use the Azure Management API to return a list of data factory pipeline runs within the given time window.\",\n\t\t\t\t\t\t\t\t\t\"type\": \"WebActivity\",\n\t\t\t\t\t\t\t\t\t\"dependsOn\": [\n\t\t\t\t\t\t\t\t\t\t{\n\t\t\t\t\t\t\t\t\t\t\t\"activity\": \"Check for Valid ADF Pipeline Name\",\n\t\t\t\t\t\t\t\t\t\t\t\"dependencyConditions\": [\n\t\t\t\t\t\t\t\t\t\t\t\t\"Succeeded\"\n\t\t\t\t\t\t\t\t\t\t\t]\n\t\t\t\t\t\t\t\t\t\t}\n\t\t\t\t\t\t\t\t\t],\n\t\t\t\t\t\t\t\t\t\"policy\": {\n\t\t\t\t\t\t\t\t\t\t\"timeout\": \"7.00:00:00\",\n\t\t\t\t\t\t\t\t\t\t\"retry\": 0,\n\t\t\t\t\t\t\t\t\t\t\"retryIntervalInSeconds\": 30,\n\t\t\t\t\t\t\t\t\t\t\"secureOutput\": false,\n\t\t\t\t\t\t\t\t\t\t\"secureInput\": false\n\t\t\t\t\t\t\t\t\t},\n\t\t\t\t\t\t\t\t\t\"userProperties\": [],\n\t\t\t\t\t\t\t\t\t\"typeProperties\": {\n\t\t\t\t\t\t\t\t\t\t\"url\": {\n\t\t\t\t\t\t\t\t\t\t\t\"value\": \"https://management.azure.com/subscriptions/@{variables('SubscriptionId')}/resourceGroups/@{variables('ResourceGroupName')}/providers/Microsoft.DataFactory/factories/@{pipeline().DataFactory}/queryPipelineRuns?api-version=2018-06-01\",\n\t\t\t\t\t\t\t\t\t\t\t\"type\": \"Expression\"\n\t\t\t\t\t\t\t\t\t\t},\n\t\t\t\t\t\t\t\t\t\t\"method\": \"POST\",\n\t\t\t\t\t\t\t\t\t\t\"body\": {\n\t\t\t\t\t\t\t\t\t\t\t\"value\": \"{\\n \\\"lastUpdatedAfter\\\": \\\"@{adddays(utcnow(),int(variables('QueryRunDays')))}\\\",\\n \\\"lastUpdatedBefore\\\": \\\"@{utcnow()}\\\",\\n \\\"filters\\\": [\\n {\\n \\\"operand\\\": \\\"PipelineName\\\",\\n \\\"operator\\\": \\\"Equals\\\",\\n \\\"values\\\": [\\n \\\"@{pipeline().parameters.PipelineName}\\\"\\n ]\\n }\\n ]\\n}\",\n\t\t\t\t\t\t\t\t\t\t\t\"type\": \"Expression\"\n\t\t\t\t\t\t\t\t\t\t},\n\t\t\t\t\t\t\t\t\t\t\"authentication\": {\n\t\t\t\t\t\t\t\t\t\t\t\"type\": \"MSI\",\n\t\t\t\t\t\t\t\t\t\t\t\"resource\": \"https://management.core.windows.net/\"\n\t\t\t\t\t\t\t\t\t\t}\n\t\t\t\t\t\t\t\t\t}\n\t\t\t\t\t\t\t\t},\n\t\t\t\t\t\t\t\t{\n\t\t\t\t\t\t\t\t\t\"name\": \"Set ADF Runs Output\",\n\t\t\t\t\t\t\t\t\t\"description\": \"Set output to local array for use in downstream filtering and pipeline checks. Use the same array output for both switch cases.\",\n\t\t\t\t\t\t\t\t\t\"type\": \"SetVariable\",\n\t\t\t\t\t\t\t\t\t\"dependsOn\": [\n\t\t\t\t\t\t\t\t\t\t{\n\t\t\t\t\t\t\t\t\t\t\t\"activity\": \"Get ADF Pipeline Runs\",\n\t\t\t\t\t\t\t\t\t\t\t\"dependencyConditions\": [\n\t\t\t\t\t\t\t\t\t\t\t\t\"Succeeded\"\n\t\t\t\t\t\t\t\t\t\t\t]\n\t\t\t\t\t\t\t\t\t\t}\n\t\t\t\t\t\t\t\t\t],\n\t\t\t\t\t\t\t\t\t\"userProperties\": [],\n\t\t\t\t\t\t\t\t\t\"typeProperties\": {\n\t\t\t\t\t\t\t\t\t\t\"variableName\": \"PipelineRuns\",\n\t\t\t\t\t\t\t\t\t\t\"value\": {\n\t\t\t\t\t\t\t\t\t\t\t\"value\": \"@activity('Get ADF Pipeline Runs').output.value\",\n\t\t\t\t\t\t\t\t\t\t\t\"type\": \"Expression\"\n\t\t\t\t\t\t\t\t\t\t}\n\t\t\t\t\t\t\t\t\t}\n\t\t\t\t\t\t\t\t}\n\t\t\t\t\t\t\t]\n\t\t\t\t\t\t},\n\t\t\t\t\t\t{\n\t\t\t\t\t\t\t\"value\": \"SYN\",\n\t\t\t\t\t\t\t\"activities\": [\n\t\t\t\t\t\t\t\t{\n\t\t\t\t\t\t\t\t\t\"name\": \"Check for Valid SYN Pipeline Name\",\n\t\t\t\t\t\t\t\t\t\"description\": \"Use the Azure Management API to return and establish if the framework pipeline exists in the target Synapse instance, including being deployed.\\n\\nSee: https://docs.microsoft.com/en-us/rest/api/synapse/data-plane/pipeline/getpipeline\",\n\t\t\t\t\t\t\t\t\t\"type\": \"WebActivity\",\n\t\t\t\t\t\t\t\t\t\"dependsOn\": [],\n\t\t\t\t\t\t\t\t\t\"policy\": {\n\t\t\t\t\t\t\t\t\t\t\"timeout\": \"7.00:00:00\",\n\t\t\t\t\t\t\t\t\t\t\"retry\": 0,\n\t\t\t\t\t\t\t\t\t\t\"retryIntervalInSeconds\": 30,\n\t\t\t\t\t\t\t\t\t\t\"secureOutput\": false,\n\t\t\t\t\t\t\t\t\t\t\"secureInput\": false\n\t\t\t\t\t\t\t\t\t},\n\t\t\t\t\t\t\t\t\t\"userProperties\": [],\n\t\t\t\t\t\t\t\t\t\"typeProperties\": {\n\t\t\t\t\t\t\t\t\t\t\"url\": {\n\t\t\t\t\t\t\t\t\t\t\t\"value\": \"https://@{pipeline().DataFactory}.dev.azuresynapse.net/pipelines/@{pipeline().parameters.PipelineName}?api-version=2019-06-01-preview\",\n\t\t\t\t\t\t\t\t\t\t\t\"type\": \"Expression\"\n\t\t\t\t\t\t\t\t\t\t},\n\t\t\t\t\t\t\t\t\t\t\"method\": \"GET\",\n\t\t\t\t\t\t\t\t\t\t\"authentication\": {\n\t\t\t\t\t\t\t\t\t\t\t\"type\": \"MSI\",\n\t\t\t\t\t\t\t\t\t\t\t\"resource\": \"https://management.core.windows.net/\"\n\t\t\t\t\t\t\t\t\t\t}\n\t\t\t\t\t\t\t\t\t}\n\t\t\t\t\t\t\t\t},\n\t\t\t\t\t\t\t\t{\n\t\t\t\t\t\t\t\t\t\"name\": \"Get SYN Pipeline Runs\",\n\t\t\t\t\t\t\t\t\t\"description\": \"Use the Azure Management API to return a list of synapse pipeline runs within the given time window.\\n\\nSee: https://docs.microsoft.com/en-us/rest/api/synapse/data-plane/pipelinerun/querypipelinerunsbyworkspace\",\n\t\t\t\t\t\t\t\t\t\"type\": \"WebActivity\",\n\t\t\t\t\t\t\t\t\t\"dependsOn\": [\n\t\t\t\t\t\t\t\t\t\t{\n\t\t\t\t\t\t\t\t\t\t\t\"activity\": \"Check for Valid SYN Pipeline Name\",\n\t\t\t\t\t\t\t\t\t\t\t\"dependencyConditions\": [\n\t\t\t\t\t\t\t\t\t\t\t\t\"Succeeded\"\n\t\t\t\t\t\t\t\t\t\t\t]\n\t\t\t\t\t\t\t\t\t\t}\n\t\t\t\t\t\t\t\t\t],\n\t\t\t\t\t\t\t\t\t\"policy\": {\n\t\t\t\t\t\t\t\t\t\t\"timeout\": \"7.00:00:00\",\n\t\t\t\t\t\t\t\t\t\t\"retry\": 0,\n\t\t\t\t\t\t\t\t\t\t\"retryIntervalInSeconds\": 30,\n\t\t\t\t\t\t\t\t\t\t\"secureOutput\": false,\n\t\t\t\t\t\t\t\t\t\t\"secureInput\": false\n\t\t\t\t\t\t\t\t\t},\n\t\t\t\t\t\t\t\t\t\"userProperties\": [],\n\t\t\t\t\t\t\t\t\t\"typeProperties\": {\n\t\t\t\t\t\t\t\t\t\t\"url\": {\n\t\t\t\t\t\t\t\t\t\t\t\"value\": \"https://@{pipeline().DataFactory}.dev.azuresynapse.net/queryPipelineRuns?api-version=2019-06-01-preview\",\n\t\t\t\t\t\t\t\t\t\t\t\"type\": \"Expression\"\n\t\t\t\t\t\t\t\t\t\t},\n\t\t\t\t\t\t\t\t\t\t\"method\": \"POST\",\n\t\t\t\t\t\t\t\t\t\t\"body\": {\n\t\t\t\t\t\t\t\t\t\t\t\"value\": \"{\\n \\\"lastUpdatedAfter\\\": \\\"@{adddays(utcnow(),int(variables('QueryRunDays')))}\\\",\\n \\\"lastUpdatedBefore\\\": \\\"@{utcnow()}\\\",\\n \\\"filters\\\": [\\n {\\n \\\"operand\\\": \\\"PipelineName\\\",\\n \\\"operator\\\": \\\"Equals\\\",\\n \\\"values\\\": [\\n \\\"@{pipeline().parameters.PipelineName}\\\"\\n ]\\n }\\n ]\\n}\",\n\t\t\t\t\t\t\t\t\t\t\t\"type\": \"Expression\"\n\t\t\t\t\t\t\t\t\t\t},\n\t\t\t\t\t\t\t\t\t\t\"authentication\": {\n\t\t\t\t\t\t\t\t\t\t\t\"type\": \"MSI\",\n\t\t\t\t\t\t\t\t\t\t\t\"resource\": \"https://management.core.windows.net/\"\n\t\t\t\t\t\t\t\t\t\t}\n\t\t\t\t\t\t\t\t\t}\n\t\t\t\t\t\t\t\t},\n\t\t\t\t\t\t\t\t{\n\t\t\t\t\t\t\t\t\t\"name\": \"Set SYN Runs Output\",\n\t\t\t\t\t\t\t\t\t\"description\": \"Set output to local array for use in downstream filtering and pipeline checks. Use the same array output for both switch cases.\",\n\t\t\t\t\t\t\t\t\t\"type\": \"SetVariable\",\n\t\t\t\t\t\t\t\t\t\"dependsOn\": [\n\t\t\t\t\t\t\t\t\t\t{\n\t\t\t\t\t\t\t\t\t\t\t\"activity\": \"Get SYN Pipeline Runs\",\n\t\t\t\t\t\t\t\t\t\t\t\"dependencyConditions\": [\n\t\t\t\t\t\t\t\t\t\t\t\t\"Succeeded\"\n\t\t\t\t\t\t\t\t\t\t\t]\n\t\t\t\t\t\t\t\t\t\t}\n\t\t\t\t\t\t\t\t\t],\n\t\t\t\t\t\t\t\t\t\"userProperties\": [],\n\t\t\t\t\t\t\t\t\t\"typeProperties\": {\n\t\t\t\t\t\t\t\t\t\t\"variableName\": \"PipelineRuns\",\n\t\t\t\t\t\t\t\t\t\t\"value\": {\n\t\t\t\t\t\t\t\t\t\t\t\"value\": \"@activity('Get SYN Pipeline Runs').output.value\",\n\t\t\t\t\t\t\t\t\t\t\t\"type\": \"Expression\"\n\t\t\t\t\t\t\t\t\t\t}\n\t\t\t\t\t\t\t\t\t}\n\t\t\t\t\t\t\t\t}\n\t\t\t\t\t\t\t]\n\t\t\t\t\t\t}\n\t\t\t\t\t],\n\t\t\t\t\t\"defaultActivities\": [\n\t\t\t\t\t\t{\n\t\t\t\t\t\t\t\"name\": \"Throw Exception Invalid Orchestrator Type\",\n\t\t\t\t\t\t\t\"description\": \"Throw exception if switch cases are not met.\",\n\t\t\t\t\t\t\t\"type\": \"ExecutePipeline\",\n\t\t\t\t\t\t\t\"dependsOn\": [],\n\t\t\t\t\t\t\t\"userProperties\": [],\n\t\t\t\t\t\t\t\"typeProperties\": {\n\t\t\t\t\t\t\t\t\"pipeline\": {\n\t\t\t\t\t\t\t\t\t\"referenceName\": \"Throw Exception\",\n\t\t\t\t\t\t\t\t\t\"type\": \"PipelineReference\"\n\t\t\t\t\t\t\t\t},\n\t\t\t\t\t\t\t\t\"waitOnCompletion\": true,\n\t\t\t\t\t\t\t\t\"parameters\": {\n\t\t\t\t\t\t\t\t\t\"Message\": \"Invalid orchestrator type provided. Unable to check pipeline running state.\"\n\t\t\t\t\t\t\t\t}\n\t\t\t\t\t\t\t}\n\t\t\t\t\t\t}\n\t\t\t\t\t]\n\t\t\t\t}\n\t\t\t},\n\t\t\t{\n\t\t\t\t\"name\": \"Set Query Run Days\",\n\t\t\t\t\"description\": \"Set the query run days value to a local variable for use in various downstream activities.\",\n\t\t\t\t\"type\": \"SetVariable\",\n\t\t\t\t\"dependsOn\": [\n\t\t\t\t\t{\n\t\t\t\t\t\t\"activity\": \"Get Query Run Days Value\",\n\t\t\t\t\t\t\"dependencyConditions\": [\n\t\t\t\t\t\t\t\"Succeeded\"\n\t\t\t\t\t\t]\n\t\t\t\t\t}\n\t\t\t\t],\n\t\t\t\t\"userProperties\": [],\n\t\t\t\t\"typeProperties\": {\n\t\t\t\t\t\"variableName\": \"QueryRunDays\",\n\t\t\t\t\t\"value\": {\n\t\t\t\t\t\t\"value\": \"@activity('Get Query Run Days Value').output.firstRow.PropertyValue\",\n\t\t\t\t\t\t\"type\": \"Expression\"\n\t\t\t\t\t}\n\t\t\t\t}\n\t\t\t}\n\t\t],\n\t\t\"parameters\": {\n\t\t\t\"BatchName\": {\n\t\t\t\t\"type\": \"string\",\n\t\t\t\t\"defaultValue\": \"NotUsed\"\n\t\t\t},\n\t\t\t\"PipelineName\": {\n\t\t\t\t\"type\": \"string\"\n\t\t\t},\n\t\t\t\"ThisRunId\": {\n\t\t\t\t\"type\": \"string\"\n\t\t\t}\n\t\t},\n\t\t\"variables\": {\n\t\t\t\"SubscriptionId\": {\n\t\t\t\t\"type\": \"String\"\n\t\t\t},\n\t\t\t\"RunCount\": {\n\t\t\t\t\"type\": \"String\"\n\t\t\t},\n\t\t\t\"ResourceGroupName\": {\n\t\t\t\t\"type\": \"String\"\n\t\t\t},\n\t\t\t\"OrchestratorType\": {\n\t\t\t\t\"type\": \"String\"\n\t\t\t},\n\t\t\t\"QueryRunDays\": {\n\t\t\t\t\"type\": \"String\"\n\t\t\t},\n\t\t\t\"PipelineRuns\": {\n\t\t\t\t\"type\": \"Array\"\n\t\t\t}\n\t\t},\n\t\t\"folder\": {\n\t\t\t\"name\": \"_ProcFwk/_ProcFwkUtils\"\n\t\t},\n\t\t\"annotations\": [\n\t\t\t\"procfwk\",\n\t\t\t\"Utils\"\n\t\t]\n\t}\n}"
},
{
"path": "DataFactory/pipeline/Email Sender.json",
"content": "{\n\t\"name\": \"Email Sender\",\n\t\"properties\": {\n\t\t\"description\": \"Provide a simple abstract over the send email function with request body item exposed as pipeline parameters.\",\n\t\t\"activities\": [\n\t\t\t{\n\t\t\t\t\"name\": \"Send Email\",\n\t\t\t\t\"description\": \"Use an Azure Function to perform an SMTP client email send operation.\",\n\t\t\t\t\"type\": \"AzureFunctionActivity\",\n\t\t\t\t\"dependsOn\": [],\n\t\t\t\t\"policy\": {\n\t\t\t\t\t\"timeout\": \"0.00:10:00\",\n\t\t\t\t\t\"retry\": 0,\n\t\t\t\t\t\"retryIntervalInSeconds\": 30,\n\t\t\t\t\t\"secureOutput\": false,\n\t\t\t\t\t\"secureInput\": false\n\t\t\t\t},\n\t\t\t\t\"userProperties\": [],\n\t\t\t\t\"typeProperties\": {\n\t\t\t\t\t\"functionName\": \"SendEmail\",\n\t\t\t\t\t\"method\": \"POST\",\n\t\t\t\t\t\"body\": {\n\t\t\t\t\t\t\"value\": \"{\\n\\\"emailRecipients\\\": \\\"@{pipeline().parameters.Recipients}\\\",\\n\\\"emailCcRecipients\\\": \\\"@{pipeline().parameters.CcRecipients}\\\",\\n\\\"emailBccRecipients\\\": \\\"@{pipeline().parameters.BccRecipients}\\\",\\n\\\"emailSubject\\\": \\\"@{pipeline().parameters.Subject}\\\",\\n\\\"emailBody\\\": \\\"@{pipeline().parameters.Body}\\\",\\n\\\"emailImportance\\\": \\\"@{pipeline().parameters.Importance}\\\"\\n}\",\n\t\t\t\t\t\t\"type\": \"Expression\"\n\t\t\t\t\t}\n\t\t\t\t},\n\t\t\t\t\"linkedServiceName\": {\n\t\t\t\t\t\"referenceName\": \"FrameworkFunctions\",\n\t\t\t\t\t\"type\": \"LinkedServiceReference\"\n\t\t\t\t}\n\t\t\t}\n\t\t],\n\t\t\"parameters\": {\n\t\t\t\"Recipients\": {\n\t\t\t\t\"type\": \"string\"\n\t\t\t},\n\t\t\t\"CcRecipients\": {\n\t\t\t\t\"type\": \"string\"\n\t\t\t},\n\t\t\t\"BccRecipients\": {\n\t\t\t\t\"type\": \"string\"\n\t\t\t},\n\t\t\t\"Subject\": {\n\t\t\t\t\"type\": \"string\"\n\t\t\t},\n\t\t\t\"Body\": {\n\t\t\t\t\"type\": \"string\"\n\t\t\t},\n\t\t\t\"Importance\": {\n\t\t\t\t\"type\": \"string\"\n\t\t\t}\n\t\t},\n\t\t\"folder\": {\n\t\t\t\"name\": \"_ProcFwk/_ProcFwkUtils\"\n\t\t},\n\t\t\"annotations\": [\n\t\t\t\"procfwk\",\n\t\t\t\"Utils\"\n\t\t]\n\t}\n}"
},
{
"path": "DataFactory/pipeline/Intentional Error.json",
"content": "{\n\t\"name\": \"Intentional Error\",\n\t\"properties\": {\n\t\t\"description\": \"Used just so the procfwk has something to call during development.\",\n\t\t\"activities\": [\n\t\t\t{\n\t\t\t\t\"name\": \"Wait1\",\n\t\t\t\t\"description\": \"Framework development worker simulator.\",\n\t\t\t\t\"type\": \"Wait\",\n\t\t\t\t\"dependsOn\": [],\n\t\t\t\t\"userProperties\": [],\n\t\t\t\t\"typeProperties\": {\n\t\t\t\t\t\"waitTimeInSeconds\": {\n\t\t\t\t\t\t\"value\": \"@pipeline().parameters.WaitTime\",\n\t\t\t\t\t\t\"type\": \"Expression\"\n\t\t\t\t\t}\n\t\t\t\t}\n\t\t\t},\n\t\t\t{\n\t\t\t\t\"name\": \"Raise Errors or Not\",\n\t\t\t\t\"description\": \"Framework development worker simulator.\",\n\t\t\t\t\"type\": \"IfCondition\",\n\t\t\t\t\"dependsOn\": [\n\t\t\t\t\t{\n\t\t\t\t\t\t\"activity\": \"Wait1\",\n\t\t\t\t\t\t\"dependencyConditions\": [\n\t\t\t\t\t\t\t\"Succeeded\"\n\t\t\t\t\t\t]\n\t\t\t\t\t}\n\t\t\t\t],\n\t\t\t\t\"userProperties\": [],\n\t\t\t\t\"typeProperties\": {\n\t\t\t\t\t\"expression\": {\n\t\t\t\t\t\t\"value\": \"@equals(pipeline().parameters.RaiseErrors,'true')\",\n\t\t\t\t\t\t\"type\": \"Expression\"\n\t\t\t\t\t},\n\t\t\t\t\t\"ifTrueActivities\": [\n\t\t\t\t\t\t{\n\t\t\t\t\t\t\t\"name\": \"Call Fail Procedure\",\n\t\t\t\t\t\t\t\"type\": \"SqlServerStoredProcedure\",\n\t\t\t\t\t\t\t\"dependsOn\": [],\n\t\t\t\t\t\t\t\"policy\": {\n\t\t\t\t\t\t\t\t\"timeout\": \"0.00:10:00\",\n\t\t\t\t\t\t\t\t\"retry\": 0,\n\t\t\t\t\t\t\t\t\"retryIntervalInSeconds\": 30,\n\t\t\t\t\t\t\t\t\"secureOutput\": false,\n\t\t\t\t\t\t\t\t\"secureInput\": false\n\t\t\t\t\t\t\t},\n\t\t\t\t\t\t\t\"userProperties\": [],\n\t\t\t\t\t\t\t\"typeProperties\": {\n\t\t\t\t\t\t\t\t\"storedProcedureName\": \"[dbo].[FailProcedure]\",\n\t\t\t\t\t\t\t\t\"storedProcedureParameters\": {\n\t\t\t\t\t\t\t\t\t\"RaiseError\": {\n\t\t\t\t\t\t\t\t\t\t\"value\": {\n\t\t\t\t\t\t\t\t\t\t\t\"value\": \"@pipeline().parameters.RaiseErrors\",\n\t\t\t\t\t\t\t\t\t\t\t\"type\": \"Expression\"\n\t\t\t\t\t\t\t\t\t\t},\n\t\t\t\t\t\t\t\t\t\t\"type\": \"String\"\n\t\t\t\t\t\t\t\t\t}\n\t\t\t\t\t\t\t\t}\n\t\t\t\t\t\t\t},\n\t\t\t\t\t\t\t\"linkedServiceName\": {\n\t\t\t\t\t\t\t\t\"referenceName\": \"SupportDatabase\",\n\t\t\t\t\t\t\t\t\"type\": \"LinkedServiceReference\"\n\t\t\t\t\t\t\t}\n\t\t\t\t\t\t}\n\t\t\t\t\t]\n\t\t\t\t}\n\t\t\t}\n\t\t],\n\t\t\"parameters\": {\n\t\t\t\"RaiseErrors\": {\n\t\t\t\t\"type\": \"string\",\n\t\t\t\t\"defaultValue\": \"false\"\n\t\t\t},\n\t\t\t\"WaitTime\": {\n\t\t\t\t\"type\": \"int\",\n\t\t\t\t\"defaultValue\": 5\n\t\t\t}\n\t\t},\n\t\t\"folder\": {\n\t\t\t\"name\": \"_Workers\"\n\t\t},\n\t\t\"annotations\": [\n\t\t\t\"_ProcFwkWorker\"\n\t\t]\n\t}\n}"
},
{
"path": "DataFactory/pipeline/Throw Exception.json",
"content": "{\n\t\"name\": \"Throw Exception\",\n\t\"properties\": {\n\t\t\"description\": \"Provide a simple way of throwing an exception within Data Factory using TSQL error handling.\",\n\t\t\"activities\": [\n\t\t\t{\n\t\t\t\t\"name\": \"Raise Error Backup\",\n\t\t\t\t\"description\": \"Using a SQL database to raise an error/exception but wrapped up as a data factory pipeline. Error message information exposed as a pipeline parameter.\",\n\t\t\t\t\"type\": \"Lookup\",\n\t\t\t\t\"dependsOn\": [\n\t\t\t\t\t{\n\t\t\t\t\t\t\"activity\": \"Raise Error\",\n\t\t\t\t\t\t\"dependencyConditions\": [\n\t\t\t\t\t\t\t\"Failed\"\n\t\t\t\t\t\t]\n\t\t\t\t\t}\n\t\t\t\t],\n\t\t\t\t\"policy\": {\n\t\t\t\t\t\"timeout\": \"0.00:10:00\",\n\t\t\t\t\t\"retry\": 0,\n\t\t\t\t\t\"retryIntervalInSeconds\": 30,\n\t\t\t\t\t\"secureOutput\": false,\n\t\t\t\t\t\"secureInput\": false\n\t\t\t\t},\n\t\t\t\t\"userProperties\": [],\n\t\t\t\t\"typeProperties\": {\n\t\t\t\t\t\"source\": {\n\t\t\t\t\t\t\"type\": \"AzureSqlSource\",\n\t\t\t\t\t\t\"sqlReaderQuery\": {\n\t\t\t\t\t\t\t\"value\": \"RAISERROR('@{pipeline().parameters.Message}',16,1);\",\n\t\t\t\t\t\t\t\"type\": \"Expression\"\n\t\t\t\t\t\t},\n\t\t\t\t\t\t\"queryTimeout\": \"02:00:00\",\n\t\t\t\t\t\t\"partitionOption\": \"None\"\n\t\t\t\t\t},\n\t\t\t\t\t\"dataset\": {\n\t\t\t\t\t\t\"referenceName\": \"GetSetMetadata\",\n\t\t\t\t\t\t\"type\": \"DatasetReference\"\n\t\t\t\t\t},\n\t\t\t\t\t\"firstRowOnly\": false\n\t\t\t\t}\n\t\t\t},\n\t\t\t{\n\t\t\t\t\"name\": \"Raise Error\",\n\t\t\t\t\"description\": \"Using newer native activity raise an error/exception but wrapped up as a data factory pipeline. Error message information exposed as a pipeline parameter.\",\n\t\t\t\t\"type\": \"Fail\",\n\t\t\t\t\"dependsOn\": [],\n\t\t\t\t\"userProperties\": [],\n\t\t\t\t\"typeProperties\": {\n\t\t\t\t\t\"message\": {\n\t\t\t\t\t\t\"value\": \"@pipeline().parameters.Message\",\n\t\t\t\t\t\t\"type\": \"Expression\"\n\t\t\t\t\t},\n\t\t\t\t\t\"errorCode\": \"16\"\n\t\t\t\t}\n\t\t\t}\n\t\t],\n\t\t\"parameters\": {\n\t\t\t\"Message\": {\n\t\t\t\t\"type\": \"string\"\n\t\t\t}\n\t\t},\n\t\t\"folder\": {\n\t\t\t\"name\": \"_ProcFwk/_ProcFwkUtils\"\n\t\t},\n\t\t\"annotations\": [\n\t\t\t\"procfwk\",\n\t\t\t\"Utils\"\n\t\t]\n\t}\n}"
},
{
"path": "DataFactory/pipeline/Wait 1.json",
"content": "{\n\t\"name\": \"Wait 1\",\n\t\"properties\": {\n\t\t\"description\": \"Used just so the procfwk has something to call during development.\",\n\t\t\"activities\": [\n\t\t\t{\n\t\t\t\t\"name\": \"Wait1\",\n\t\t\t\t\"description\": \"Framework development worker simulator.\",\n\t\t\t\t\"type\": \"Wait\",\n\t\t\t\t\"dependsOn\": [],\n\t\t\t\t\"userProperties\": [],\n\t\t\t\t\"typeProperties\": {\n\t\t\t\t\t\"waitTimeInSeconds\": {\n\t\t\t\t\t\t\"value\": \"@pipeline().parameters.WaitTime\",\n\t\t\t\t\t\t\"type\": \"Expression\"\n\t\t\t\t\t}\n\t\t\t\t}\n\t\t\t}\n\t\t],\n\t\t\"parameters\": {\n\t\t\t\"WaitTime\": {\n\t\t\t\t\"type\": \"int\",\n\t\t\t\t\"defaultValue\": 5\n\t\t\t}\n\t\t},\n\t\t\"folder\": {\n\t\t\t\"name\": \"_Workers\"\n\t\t},\n\t\t\"annotations\": [\n\t\t\t\"_ProcFwkWorker\"\n\t\t]\n\t}\n}"
},
{
"path": "DataFactory/pipeline/Wait 10.json",
"content": "{\n\t\"name\": \"Wait 10\",\n\t\"properties\": {\n\t\t\"description\": \"Used just so the procfwk has something to call during development.\",\n\t\t\"activities\": [\n\t\t\t{\n\t\t\t\t\"name\": \"Wait10\",\n\t\t\t\t\"description\": \"Framework development worker simulator.\",\n\t\t\t\t\"type\": \"Wait\",\n\t\t\t\t\"dependsOn\": [],\n\t\t\t\t\"userProperties\": [],\n\t\t\t\t\"typeProperties\": {\n\t\t\t\t\t\"waitTimeInSeconds\": {\n\t\t\t\t\t\t\"value\": \"@pipeline().parameters.WaitTime\",\n\t\t\t\t\t\t\"type\": \"Expression\"\n\t\t\t\t\t}\n\t\t\t\t}\n\t\t\t}\n\t\t],\n\t\t\"parameters\": {\n\t\t\t\"WaitTime\": {\n\t\t\t\t\"type\": \"int\",\n\t\t\t\t\"defaultValue\": 5\n\t\t\t}\n\t\t},\n\t\t\"folder\": {\n\t\t\t\"name\": \"_Workers\"\n\t\t},\n\t\t\"annotations\": [\n\t\t\t\"_ProcFwkWorker\"\n\t\t]\n\t}\n}"
},
{
"path": "DataFactory/pipeline/Wait 2.json",
"content": "{\n\t\"name\": \"Wait 2\",\n\t\"properties\": {\n\t\t\"description\": \"Used just so the procfwk has something to call during development.\",\n\t\t\"activities\": [\n\t\t\t{\n\t\t\t\t\"name\": \"Wait2\",\n\t\t\t\t\"description\": \"Framework development worker simulator.\",\n\t\t\t\t\"type\": \"Wait\",\n\t\t\t\t\"dependsOn\": [],\n\t\t\t\t\"userProperties\": [],\n\t\t\t\t\"typeProperties\": {\n\t\t\t\t\t\"waitTimeInSeconds\": {\n\t\t\t\t\t\t\"value\": \"@pipeline().parameters.WaitTime\",\n\t\t\t\t\t\t\"type\": \"Expression\"\n\t\t\t\t\t}\n\t\t\t\t}\n\t\t\t}\n\t\t],\n\t\t\"parameters\": {\n\t\t\t\"WaitTime\": {\n\t\t\t\t\"type\": \"int\",\n\t\t\t\t\"defaultValue\": 5\n\t\t\t}\n\t\t},\n\t\t\"folder\": {\n\t\t\t\"name\": \"_Workers\"\n\t\t},\n\t\t\"annotations\": [\n\t\t\t\"_ProcFwkWorker\"\n\t\t]\n\t}\n}"
},
{
"path": "DataFactory/pipeline/Wait 3.json",
"content": "{\n\t\"name\": \"Wait 3\",\n\t\"properties\": {\n\t\t\"description\": \"Used just so the procfwk has something to call during development.\",\n\t\t\"activities\": [\n\t\t\t{\n\t\t\t\t\"name\": \"Wait3\",\n\t\t\t\t\"description\": \"Framework development worker simulator.\",\n\t\t\t\t\"type\": \"Wait\",\n\t\t\t\t\"dependsOn\": [],\n\t\t\t\t\"userProperties\": [],\n\t\t\t\t\"typeProperties\": {\n\t\t\t\t\t\"waitTimeInSeconds\": {\n\t\t\t\t\t\t\"value\": \"@pipeline().parameters.WaitTime\",\n\t\t\t\t\t\t\"type\": \"Expression\"\n\t\t\t\t\t}\n\t\t\t\t}\n\t\t\t}\n\t\t],\n\t\t\"parameters\": {\n\t\t\t\"WaitTime\": {\n\t\t\t\t\"type\": \"int\",\n\t\t\t\t\"defaultValue\": 5\n\t\t\t}\n\t\t},\n\t\t\"folder\": {\n\t\t\t\"name\": \"_Workers\"\n\t\t},\n\t\t\"annotations\": [\n\t\t\t\"_ProcFwkWorker\"\n\t\t]\n\t}\n}"
},
{
"path": "DataFactory/pipeline/Wait 4.json",
"content": "{\n\t\"name\": \"Wait 4\",\n\t\"properties\": {\n\t\t\"description\": \"Used just so the procfwk has something to call during development.\",\n\t\t\"activities\": [\n\t\t\t{\n\t\t\t\t\"name\": \"Wait4\",\n\t\t\t\t\"description\": \"Framework development worker simulator.\",\n\t\t\t\t\"type\": \"Wait\",\n\t\t\t\t\"dependsOn\": [],\n\t\t\t\t\"userProperties\": [],\n\t\t\t\t\"typeProperties\": {\n\t\t\t\t\t\"waitTimeInSeconds\": {\n\t\t\t\t\t\t\"value\": \"@pipeline().parameters.WaitTime\",\n\t\t\t\t\t\t\"type\": \"Expression\"\n\t\t\t\t\t}\n\t\t\t\t}\n\t\t\t}\n\t\t],\n\t\t\"parameters\": {\n\t\t\t\"WaitTime\": {\n\t\t\t\t\"type\": \"int\",\n\t\t\t\t\"defaultValue\": 5\n\t\t\t}\n\t\t},\n\t\t\"folder\": {\n\t\t\t\"name\": \"_Workers\"\n\t\t},\n\t\t\"annotations\": [\n\t\t\t\"_ProcFwkWorker\"\n\t\t]\n\t}\n}"
},
{
"path": "DataFactory/pipeline/Wait 5.json",
"content": "{\n\t\"name\": \"Wait 5\",\n\t\"properties\": {\n\t\t\"description\": \"Used just so the procfwk has something to call during development.\",\n\t\t\"activities\": [\n\t\t\t{\n\t\t\t\t\"name\": \"Wait5\",\n\t\t\t\t\"description\": \"Framework development worker simulator.\",\n\t\t\t\t\"type\": \"Wait\",\n\t\t\t\t\"dependsOn\": [],\n\t\t\t\t\"userProperties\": [],\n\t\t\t\t\"typeProperties\": {\n\t\t\t\t\t\"waitTimeInSeconds\": {\n\t\t\t\t\t\t\"value\": \"@pipeline().parameters.WaitTime\",\n\t\t\t\t\t\t\"type\": \"Expression\"\n\t\t\t\t\t}\n\t\t\t\t}\n\t\t\t}\n\t\t],\n\t\t\"parameters\": {\n\t\t\t\"WaitTime\": {\n\t\t\t\t\"type\": \"int\",\n\t\t\t\t\"defaultValue\": 5\n\t\t\t}\n\t\t},\n\t\t\"folder\": {\n\t\t\t\"name\": \"_Workers\"\n\t\t},\n\t\t\"annotations\": [\n\t\t\t\"_ProcFwkWorker\"\n\t\t]\n\t}\n}"
},
{
"path": "DataFactory/pipeline/Wait 6.json",
"content": "{\n\t\"name\": \"Wait 6\",\n\t\"properties\": {\n\t\t\"description\": \"Used just so the procfwk has something to call during development.\",\n\t\t\"activities\": [\n\t\t\t{\n\t\t\t\t\"name\": \"Wait6\",\n\t\t\t\t\"description\": \"Framework development worker simulator.\",\n\t\t\t\t\"type\": \"Wait\",\n\t\t\t\t\"dependsOn\": [],\n\t\t\t\t\"userProperties\": [],\n\t\t\t\t\"typeProperties\": {\n\t\t\t\t\t\"waitTimeInSeconds\": {\n\t\t\t\t\t\t\"value\": \"@pipeline().parameters.WaitTime\",\n\t\t\t\t\t\t\"type\": \"Expression\"\n\t\t\t\t\t}\n\t\t\t\t}\n\t\t\t}\n\t\t],\n\t\t\"parameters\": {\n\t\t\t\"WaitTime\": {\n\t\t\t\t\"type\": \"int\",\n\t\t\t\t\"defaultValue\": 5\n\t\t\t}\n\t\t},\n\t\t\"folder\": {\n\t\t\t\"name\": \"_Workers\"\n\t\t},\n\t\t\"annotations\": [\n\t\t\t\"_ProcFwkWorker\"\n\t\t]\n\t}\n}"
},
{
"path": "DataFactory/pipeline/Wait 7.json",
"content": "{\n\t\"name\": \"Wait 7\",\n\t\"properties\": {\n\t\t\"description\": \"Used just so the procfwk has something to call during development.\",\n\t\t\"activities\": [\n\t\t\t{\n\t\t\t\t\"name\": \"Wait7\",\n\t\t\t\t\"description\": \"Framework development worker simulator.\",\n\t\t\t\t\"type\": \"Wait\",\n\t\t\t\t\"dependsOn\": [],\n\t\t\t\t\"userProperties\": [],\n\t\t\t\t\"typeProperties\": {\n\t\t\t\t\t\"waitTimeInSeconds\": {\n\t\t\t\t\t\t\"value\": \"@pipeline().parameters.WaitTime\",\n\t\t\t\t\t\t\"type\": \"Expression\"\n\t\t\t\t\t}\n\t\t\t\t}\n\t\t\t}\n\t\t],\n\t\t\"parameters\": {\n\t\t\t\"WaitTime\": {\n\t\t\t\t\"type\": \"int\",\n\t\t\t\t\"defaultValue\": 5\n\t\t\t}\n\t\t},\n\t\t\"folder\": {\n\t\t\t\"name\": \"_Workers\"\n\t\t},\n\t\t\"annotations\": [\n\t\t\t\"_ProcFwkWorker\"\n\t\t]\n\t}\n}"
},
{
"path": "DataFactory/pipeline/Wait 8.json",
"content": "{\n\t\"name\": \"Wait 8\",\n\t\"properties\": {\n\t\t\"description\": \"Used just so the procfwk has something to call during development.\",\n\t\t\"activities\": [\n\t\t\t{\n\t\t\t\t\"name\": \"Wait8\",\n\t\t\t\t\"description\": \"Framework development worker simulator.\",\n\t\t\t\t\"type\": \"Wait\",\n\t\t\t\t\"dependsOn\": [],\n\t\t\t\t\"userProperties\": [],\n\t\t\t\t\"typeProperties\": {\n\t\t\t\t\t\"waitTimeInSeconds\": {\n\t\t\t\t\t\t\"value\": \"@pipeline().parameters.WaitTime\",\n\t\t\t\t\t\t\"type\": \"Expression\"\n\t\t\t\t\t}\n\t\t\t\t}\n\t\t\t}\n\t\t],\n\t\t\"parameters\": {\n\t\t\t\"WaitTime\": {\n\t\t\t\t\"type\": \"int\",\n\t\t\t\t\"defaultValue\": 5\n\t\t\t}\n\t\t},\n\t\t\"folder\": {\n\t\t\t\"name\": \"_Workers\"\n\t\t},\n\t\t\"annotations\": [\n\t\t\t\"_ProcFwkWorker\"\n\t\t]\n\t}\n}"
},
{
"path": "DataFactory/pipeline/Wait 9.json",
"content": "{\n\t\"name\": \"Wait 9\",\n\t\"properties\": {\n\t\t\"description\": \"Used just so the procfwk has something to call during development.\",\n\t\t\"activities\": [\n\t\t\t{\n\t\t\t\t\"name\": \"Wait9\",\n\t\t\t\t\"description\": \"Framework development worker simulator.\",\n\t\t\t\t\"type\": \"Wait\",\n\t\t\t\t\"dependsOn\": [],\n\t\t\t\t\"userProperties\": [],\n\t\t\t\t\"typeProperties\": {\n\t\t\t\t\t\"waitTimeInSeconds\": {\n\t\t\t\t\t\t\"value\": \"@pipeline().parameters.WaitTime\",\n\t\t\t\t\t\t\"type\": \"Expression\"\n\t\t\t\t\t}\n\t\t\t\t}\n\t\t\t}\n\t\t],\n\t\t\"parameters\": {\n\t\t\t\"WaitTime\": {\n\t\t\t\t\"type\": \"int\",\n\t\t\t\t\"defaultValue\": 15\n\t\t\t}\n\t\t},\n\t\t\"folder\": {\n\t\t\t\"name\": \"_Workers\"\n\t\t},\n\t\t\"annotations\": [\n\t\t\t\"_ProcFwkWorker\"\n\t\t]\n\t}\n}"
},
{
"path": "DataFactory/trigger/FunctionalTestingTrigger.json",
"content": "{\n\t\"name\": \"FunctionalTestingTrigger\",\n\t\"properties\": {\n\t\t\"description\": \"Used for functional testing of the framework in a dedicated environment.\",\n\t\t\"annotations\": [\n\t\t\t\"procfwk\"\n\t\t],\n\t\t\"runtimeState\": \"Stopped\",\n\t\t\"pipelines\": [\n\t\t\t{\n\t\t\t\t\"pipelineReference\": {\n\t\t\t\t\t\"referenceName\": \"01-Grandparent\",\n\t\t\t\t\t\"type\": \"PipelineReference\"\n\t\t\t\t}\n\t\t\t}\n\t\t],\n\t\t\"type\": \"ScheduleTrigger\",\n\t\t\"typeProperties\": {\n\t\t\t\"recurrence\": {\n\t\t\t\t\"frequency\": \"Hour\",\n\t\t\t\t\"interval\": 2,\n\t\t\t\t\"startTime\": \"2020-04-06T15:00:00Z\",\n\t\t\t\t\"timeZone\": \"UTC\"\n\t\t\t}\n\t\t}\n\t}\n}"
},
{
"path": "DeploymentTools/DataFactory/DeployProcFwkComponents.ps1",
"content": "function Publish-procfwkadf \n{\nParam(\n [Parameter(Mandatory)]\n [string]$resourceGroupName,\n [Parameter(Mandatory)]\n [string]$dataFactoryName,\n [Parameter(Mandatory)]\n [string]$region,\n [Parameter(Mandatory)]\n [string]$adfPath,\n [Parameter(Mandatory)]\n [string]$scriptPath\n)\n\n#SPN for deploying ADF:\n$tenantId = [System.Environment]::GetEnvironmentVariable('AZURE_TENANT_ID')\n$subscriptionId = [System.Environment]::GetEnvironmentVariable('AZURE_SUBSCRIPTION_ID')\n$spId = [System.Environment]::GetEnvironmentVariable('AZURE_CLIENT_ID')\n$spKey = [System.Environment]::GetEnvironmentVariable('AZURE_CLIENT_SECRET')\n\n#Modules\nImport-Module -Name \"Az\"\n#Update-Module -Name \"Az\"\n\nImport-Module -Name \"Az.DataFactory\"\n#Update-Module -Name \"Az.DataFactory\"\n\nImport-Module -Name \"azure.datafactory.tools\"\n#Update-Module -Name \"azure.datafactory.tools\"\n\nGet-Module -Name \"*DataFactory*\"\n\n# Login as a Service Principal\nif ($spId) {\n $passwd = ConvertTo-SecureString $spKey -AsPlainText -Force\n $pscredential = New-Object System.Management.Automation.PSCredential($spId, $passwd)\n Connect-AzAccount -ServicePrincipal -Credential $pscredential -TenantId $tenantId | Out-Null\n}\nGet-AzContext\n\n# Get Deployment Objects and Params files\n$deploymentFilePath = Join-Path -Path $scriptPath -ChildPath \"ProcFwkComponents.json\"\n$configFilePath = Join-Path -Path $scriptPath -ChildPath \"config-all.csv\"\n$Env:SQLDatabase = \"secretKeyToDbConnectionString\"\n\n$opt = New-AdfPublishOption\n$deploymentObject = (Get-Content $deploymentFilePath) | ConvertFrom-Json \n$objectsToInclude = $deploymentObject.datasets + $deploymentObject.linkedServices + $deploymentObject.pipelines + $deploymentObject.triggers\n$objectsToInclude | ForEach-Object { \n $objName = $_.substring(1).Replace('.json', '').Replace('/', '.') \n $opt.Includes.Add($objName, \"\")\n}\n\n# Deployment of ADF\n$opt.CreateNewInstance = $true\n$opt.DeleteNotInSource = $false\n$opt.StopStartTriggers = $true\nPublish-AdfV2FromJson -RootFolder $adfPath `\n -ResourceGroupName $resourceGroupName `\n -DataFactoryName $dataFactoryName `\n -Location $region `\n -Option $opt `\n -Stage $configFilePath\n\n}\n\n\n# Run function\n$VerbosePreference = 'Continue'\n$ErrorActionPreference = 'Stop'\n\n$scriptPath = Join-Path -Path (Get-Location) -ChildPath \"\\DeploymentTools\\DataFactory\"\n$AdfPath = Join-Path -Path (Get-Location) -ChildPath \"DataFactory\"\n\nPublish-procfwkadf -resourceGroupName 'rg-pademo' -dataFactoryName 'adf-metadata-driven-proc' -region 'uksouth' `\n -adfPath \"$AdfPath\" -scriptPath \"$scriptPath\"\n\n\n\n"
},
{
"path": "DeploymentTools/DataFactory/DeployProcFwkComponents_Old.ps1",
"content": "# Set global variables as required:\n$resourceGroupName = \"ADF.procfwk\"\n$dataFactoryName = \"FrameworkFactoryDev\"\n$region = \"uksouth\"\n\n#SPN for deploying ADF:\n$tenantId = [System.Environment]::GetEnvironmentVariable('AZURE_TENANT_ID')\n$subscriptionId = [System.Environment]::GetEnvironmentVariable('AZURE_SUBSCRIPTION_ID')\n$spId = [System.Environment]::GetEnvironmentVariable('AZURE_CLIENT_ID')\n$spKey = [System.Environment]::GetEnvironmentVariable('AZURE_CLIENT_SECRET')\n\n#Modules\nImport-Module -Name \"Az\"\n#Update-Module -Name \"Az\"\n\nImport-Module -Name \"Az.DataFactory\"\n#Update-Module -Name \"Az.DataFactory\"\n\n# Login as a Service Principal\n$passwd = ConvertTo-SecureString $spKey -AsPlainText -Force\n$pscredential = New-Object System.Management.Automation.PSCredential($spId, $passwd)\nConnect-AzAccount -ServicePrincipal -Credential $pscredential -TenantId $tenantId | Out-Null\n\n# Get Deployment Objects and Params files\n$scriptPath = \"C:\\Users\\PaulAndrew\\Source\\GitHub\\ADF.procfwk\\DeploymentTools\\DataFactory\\\"\n#$deploymentFilePath = $scriptPath + \"\\ProcFwkComponents.json\"\n$deploymentFilePath = \"C:\\Users\\PaulAndrew\\Source\\GitHub\\ADF.procfwk\\DeploymentTools\\DataFactory\\ProcFwkComponents.json\"\n\n#Write-Host $scriptPath\n\n$deploymentObject = (Get-Content $deploymentFilePath) | ConvertFrom-Json \n\n<#\nCreate Data Factory\n#>\nWrite-Host \"\"\nWrite-Host \"-----------------------Data Factory-------------------------\"\n\n$check = Get-AzDataFactoryV2 `\n -ResourceGroupName $resourceGroupName `\n -DataFactoryName $dataFactoryName `\n -ErrorAction SilentlyContinue\n\nif($check -eq $null)\n {\n Write-Host \"Creating Data Factory:\" $dataFactoryName\n \n try\n {\n Set-AzDataFactoryV2 `\n -ResourceGroupName $resourceGroupName `\n -DataFactoryName $dataFactoryName `\n -Location $region | Format-List | Out-Null\n\n Write-Host \"...done\" -ForegroundColor Green\n Write-Host \"\"\n }\n catch\n {\n Write-Host \"Failed to created data factory:\" $dataFactoryName -ForegroundColor Red\n Write-Host $_.Exception.Message\n Write-Host $_.Exception.ItemName\n Exit\n }\n }\nelse \n {\n Write-Host \"Data Factory $dataFactoryName already exists...\"\n }\n\n<#\nDeploy Linked Services\n#>\nWrite-Host \"\"\nWrite-Host \"----------------------Linked Services-----------------------\"\n\nForEach ($linkedService in $deploymentObject.linkedServices)\n {\n $componentPath = $scriptPath.Replace(\"DeploymentTools\\\",\"\") + $linkedService \n \n $linkedServiceFile = (Get-Content $componentPath) | ConvertFrom-Json\n $linkedServiceName = $linkedServiceFile.name\n Write-Host \"Deploying Linked Service:\" $linkedServiceName \n\n try\n {\n Set-AzDataFactoryV2LinkedService `\n -ResourceGroupName $resourceGroupName `\n -DataFactoryName $dataFactoryName `\n -Name $linkedServiceName `\n -DefinitionFile $componentPath `\n -Force | Format-List | Out-Null\n\n Write-Host \"...done\" -ForegroundColor Green\n Write-Host \"\"\n }\n catch\n {\n Write-Host \"Failed to deploy linked service:\" $linkedServiceName -ForegroundColor Green\n Write-Host $_.Exception.Message\n Write-Host $_.Exception.ItemName\n Exit\n }\n }\n\n<#\nDeploy Datasets\n#>\nWrite-Host \"\"\nWrite-Host \"--------------------------Datasets--------------------------\"\n\nForEach ($dataSet in $deploymentObject.datasets)\n {\n $componentPath = $scriptPath.Replace(\"DeploymentTools\\\",\"\") + $dataSet \n\n $datasetFile = (Get-Content $componentPath) | ConvertFrom-Json\n $datasetName = $datasetFile.name\n Write-Host \"Deploying Dataset:\" $datasetName\n \n try\n {\n Set-AzDataFactoryV2Dataset `\n -ResourceGroupName $resourceGroupName `\n -DataFactoryName $dataFactoryName `\n -Name $datasetName `\n -DefinitionFile $componentPath `\n -Force | Format-List | Out-Null\n\n Write-Host \"...done\" -ForegroundColor Green\n Write-Host \"\"\n }\n catch\n {\n Write-Host \"Failed to deploy dataset:\" $datasetName -ForegroundColor Red\n Write-Host $_.Exception.Message\n Write-Host $_.Exception.ItemName\n Exit\n } \n }\n\n<#\nDeploy Pipelines\n#>\nWrite-Host \"\"\nWrite-Host \"-------------------------Pipelines--------------------------\"\n\nForEach ($pipeline in $deploymentObject.pipelines)\n {\n $componentPath = $scriptPath.Replace(\"DeploymentTools\\\",\"\") + $pipeline\n\n $pipelineFile = (Get-Content $componentPath) | ConvertFrom-Json\n $pipelineName = $pipelineFile.name\n Write-Host \"Deploying Pipeline:\" $pipelineName\n \n try\n {\n ##Bug in SDK means this native cmdlet can't be used if pipeline contains a Wait activity expression.\n <#\n Set-AzDataFactoryV2Pipeline `\n -ResourceGroupName $resourceGroupName `\n -DataFactoryName $dataFactoryName `\n -Name $pipelineName `\n -DefinitionFile $componentPath `\n -Force | Format-List | Out-Null\n #>\n $body = (Get-Content -Path $componentPath | Out-String) \n $json = $body | ConvertFrom-Json\n\n New-AzResource `\n -ResourceType 'Microsoft.DataFactory/factories/pipelines' `\n -ResourceGroupName $resourceGroupName `\n -Name \"$dataFactoryName/$pipelineName\" `\n -ApiVersion \"2018-06-01\" `\n -Properties $json `\n -IsFullObject -Force | Out-Null\n\n Write-Host \"...done\" -ForegroundColor Green\n Write-Host \"\"\n }\n catch\n {\n Write-Host \"Failed to deploy pipeline:\" $pipelineName -ForegroundColor Red\n Write-Host $_.Exception.Message\n Write-Host $_.Exception.ItemName\n Exit\n }\n }\n\n\n<#\nDeploy Triggers\n#>\nWrite-Host \"\"\nWrite-Host \"-------------------------Triggers---------------------------\"\n\nForEach ($trigger in $deploymentObject.triggers)\n {\n $componentPath = $scriptPath.Replace(\"DeploymentTools\\\",\"\") + $trigger\n\n $triggerFile = (Get-Content $componentPath) | ConvertFrom-Json\n $triggerName = $triggerFile.name\n Write-Host \"Deploying Trigger:\" $triggerName\n \n $currentTrigger = Get-AzDataFactoryV2Trigger `\n -ResourceGroupName $resourceGroupName `\n -DataFactoryName $dataFactoryName `\n -Name $triggerName `\n -ErrorAction SilentlyContinue\n \n try\n {\n if($currentTrigger -ne $null)\n {\n #Stop trigger if already deployed as can't deploy over running trigger. \n Stop-AzDataFactoryV2Trigger `\n -ResourceGroupName $resourceGroupName `\n -DataFactoryName $dataFactoryName `\n -Name $triggerName -Force | Out-Null\n }\n \n Set-AzDataFactoryV2Trigger `\n -ResourceGroupName $resourceGroupName `\n -DataFactoryName $dataFactoryName `\n -Name $triggerName `\n -DefinitionFile $componentPath `\n -Force | Format-List | Out-Null\n\n Write-Host \"...done\" -ForegroundColor Green\n Write-Host \"\"\n }\n catch\n {\n Write-Host \"Failed to deploy trigger:\" $triggerName -ForegroundColor Red\n Write-Host $_.Exception.Message\n Write-Host $_.Exception.ItemName\n Exit\n }\n }"
},
{
"path": "DeploymentTools/DataFactory/Get Pipelines.ps1",
"content": "# Set global variables as required:\n$resourceGroupName = \"ADF.procfwk\"\n$dataFactoryName = \"FrameworkFactory\"\n$region = \"uksouth\"\n\n#SPN for deploying ADF:\n$tenantId = [System.Environment]::GetEnvironmentVariable('AZURE_TENANT_ID')\n$subscriptionId = [System.Environment]::GetEnvironmentVariable('AZURE_SUBSCRIPTION_ID')\n$spId = [System.Environment]::GetEnvironmentVariable('AZURE_CLIENT_ID')\n$spKey = [System.Environment]::GetEnvironmentVariable('AZURE_CLIENT_SECRET')\n\n#Modules\nImport-Module -Name \"Az\"\nImport-Module -Name \"Az.DataFactory\"\n\n# Login as a Service Principal\n$passwd = ConvertTo-SecureString $spKey -AsPlainText -Force\n$pscredential = New-Object System.Management.Automation.PSCredential($spId, $passwd)\nConnect-AzAccount -ServicePrincipal -Credential $pscredential -TenantId $tenantId | Out-Null\n\nGet-AzDataFactoryV2Pipeline -DataFactoryName $dataFactoryName -ResourceGroupName $resourceGroupName \n\n"
},
{
"path": "DeploymentTools/DataFactory/GlobalVars.ps1",
"content": "# Set global variables as required:\n$resourceGroupName = \"ADF.procfwk\"\n$dataFactoryName = \"FrameworkFactory\"\n$region = \"uksouth\"\n\n# ADF deployment from PS script\n.\\DeploymentTools\\DataFactory\\DeployProcFwkComponents.ps1 `\n -resourceGroupName \"$resourceGroupName\" `\n -dataFactoryName \"$dataFactoryName\" `\n -region \"$region\"\n\n\n.\\DeploymentTools\\DataFactory\\PopulatePipelinesInDb.ps1 `\n -SqlServerName '*****.database.windows.net' `\n -SqlDatabaseName 'adfprocfwk' `\n -SqlUser 'adm' `\n -SqlPass '******' `\n -resourceGroupName \"$resourceGroupName\" `\n -dataFactoryName \"$dataFactoryName\" `\n -region \"$region\"\n\n"
},
{
"path": "DeploymentTools/DataFactory/PopulatePipelinesInDb.ps1",
"content": "Param(\n [Parameter(Mandatory)]\n [string]$SqlServerName,\n [Parameter(Mandatory)]\n [string]$SqlDatabaseName,\n [Parameter(Mandatory)]\n [string]$SqlUser,\n [Parameter(Mandatory)]\n [string]$SqlPass,\n [Parameter(Mandatory)]\n [string]$resourceGroupName,\n [Parameter(Mandatory)]\n [string]$dataFactoryName,\n [Parameter(Mandatory)]\n [string]$region\n)\n\n$ErrorActionPreference = 'Stop'\n\n#Install-Module -Name SqlServer -AllowClobber\nImport-Module -Name SqlServer\n\n$pipelines = Get-AzDataFactoryV2Pipeline -DataFactoryName $dataFactoryName -ResourceGroupName $resourceGroupName \n\nforeach ($p in $pipelines) {\n Write-Host $p.name\n $query = \"EXEC [procfwkHelpers].[AddPipelineViaPowerShell] '$resourceGroupName', '$dataFactoryName', '$($p.Name)';\"\n Invoke-Sqlcmd -ServerInstance \"$SqlServerName\" -Database \"$SqlDatabaseName\" -Query \"$query\" -Username \"$SqlUser\" -Password \"$SqlPass\"\n}\n\nWrite-Host \"List of ADF pipelines has been populated into database.\"\n\n\n"
},
{
"path": "DeploymentTools/DataFactory/ProcFwkComponents.json",
"content": "{\n \"linkedServices\": [\n \"/linkedService/Keys.json\",\n \"/linkedService/FrameworkFunctions.json\",\n \"/linkedService/SupportDatabase.json\"\n ],\n \"datasets\": [\n \"/dataset/GetSetMetadata.json\"\n ],\n \"pipelines\": [\n \"/pipeline/Throw Exception.json\",\n \"/pipeline/Check For Running Pipeline.json\",\n \"/pipeline/Email Sender.json\",\n \"/pipeline/04-Infant.json\",\n \"/pipeline/03-Child.json\",\n \"/pipeline/02-Parent.json\",\n \"/pipeline/01-Grandparent.json\"\n ],\n \"triggers\": [\n \"/trigger/FunctionalTestingTrigger.json\"\n ]\n}\n"
},
{
"path": "DeploymentTools/DataFactory/config-all.csv",
"content": "type,name,path,value\nlinkedService,SupportDatabase,typeProperties.connectionString.secretName,$($Env:SQLDatabase)"
},
{
"path": "DeploymentTools/Deployment.targets",
"content": "\n\n \n Debug\n AnyCPU\n bin\\$(Configuration)\\\n false\n true\n false\n None\n obj\\\n $(BaseIntermediateOutputPath)\\\n $(BaseIntermediateOutputPath)$(Configuration)\\\n $(IntermediateOutputPath)ProjectReferences\n $(ProjectReferencesOutputPath)\\\n true\n \n\n \n false\n false\n \n\n \n \n \n \n \n\n \n \n Always\n \n \n Never\n \n \n false\n Build\n \n \n\n \n\n \n \n _GetDeploymentProjectContent;\n _CalculateContentOutputRelativePaths;\n _GetReferencedProjectsOutput;\n _CalculateArtifactStagingDirectory;\n _CopyOutputToArtifactStagingDirectory;\n \n \n\n \n \n \n \n\n \n \n \n \n \n\n \n \n Configuration=$(Configuration);Platform=$(Platform)\n \n\n \n\n \n \n $([System.IO.Path]::GetFileNameWithoutExtension('%(ProjectReference.Identity)'))\n \n \n \n\n \n \n $(OutDir)\n $(OutputPath)\n $(ArtifactStagingDirectory)\\\n $(ArtifactStagingDirectory)staging\\\n $(Build_StagingDirectory)\n \n \n\n \n \n \n <_OriginalIdentity>%(DeploymentProjectContentOutput.Identity)\n <_RelativePath>$(_OriginalIdentity.Replace('$(MSBuildProjectDirectory)', ''))\n \n\n \n \n $(_RelativePath)\n \n \n \n\n \n\n \n \n PrepareForRun\n \n \n\n \n\n \n \n \n \n \n\n"
},
{
"path": "DeploymentTools/DeploymentTools.deployproj",
"content": "\n\n \n \n Debug\n AnyCPU\n \n \n Release\n AnyCPU\n \n \n \n 3aead846-dbe4-45ad-97dd-37e94be009fd\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n False\n \n \n \n"
},
{
"path": "FactoryTesting/FactoryTesting.csproj",
"content": "\n\n \n netcoreapp3.1\n\n false\n \n\n \n \n \n\n \n \n \n\n \n \n \n \n \n \n \n \n all\n runtime; build; native; contentfiles; analyzers; buildtransitive\n \n \n \n \n\n\n"
},
{
"path": "FactoryTesting/Helpers/CoverageHelper.cs",
"content": "using FactoryTesting.Helpers;\nusing Microsoft.Azure.Management.DataFactory.Models;\nusing NUnit.Framework;\nusing System;\nusing System.Collections.Generic;\nusing System.Threading.Tasks;\n\nnamespace FactoryTesting.Helpers\n{\n public class CoverageHelper : DatabaseHelper where T : CoverageHelper\n {\n public new async Task RunPipeline(string pipelineName)\n {\n var callStack = new System.Diagnostics.StackTrace();\n await base.RunPipeline(pipelineName);\n if (ReportTestCoverage)\n foreach (var ar in await GetActivityRuns())\n RecordActivityRun(ar, callStack.ToString());\n }\n\n public new async Task RunPipelineAndCancel(string pipelineName)\n {\n var callStack = new System.Diagnostics.StackTrace();\n await base.RunPipelineAndCancel(pipelineName);\n }\n\n public bool ReportTestCoverage\n {\n get\n {\n try\n {\n var measure = GetSetting(\"ReportTestCoverage\");\n if (measure == \"true\")\n return true;\n }\n catch (Exception) { }\n return false;\n }\n }\n\n private void RecordActivityRun(ActivityRun ar, string context)\n {\n var parameters = new Dictionary\n {\n [\"@pipelineName\"] = ar.PipelineName,\n [\"@activityName\"] = ar.ActivityName,\n [\"@context\"] = context\n };\n ExecuteStoredProcedure(\"test.RecordActivityRun\", parameters);\n }\n }\n}\n\n[SetUpFixture]\npublic class CoverageHelperSetup : CoverageHelper\n{\n [OneTimeSetUp]\n public async Task SetupCoverageHelper()\n {\n if (ReportTestCoverage)\n {\n WithEmptyTable(\"test.ActivityRun\");\n WithEmptyTable(\"test.PipelineActivity\");\n\n foreach (var p in await GetPipelines())\n if (p.Activities != null)\n foreach (var a in p.Activities)\n RecordActivity(p.Name, a);\n }\n\n TearDown();\n }\n\n private void RecordActivity(string pipelineName, Activity act)\n {\n var parameters = new Dictionary\n {\n [\"@pipelineName\"] = pipelineName,\n [\"@activityName\"] = act.Name\n };\n ExecuteStoredProcedure(\"test.RecordActivity\", parameters);\n\n if (act is ForEachActivity)\n foreach (var a in ((ForEachActivity)act).Activities)\n RecordActivity(pipelineName, a);\n\n if (act is UntilActivity)\n foreach (var a in ((UntilActivity)act).Activities)\n RecordActivity(pipelineName, a);\n\n if (act is IfConditionActivity)\n {\n foreach (var a in ((IfConditionActivity)act).IfTrueActivities)\n RecordActivity(pipelineName, a);\n foreach (var a in ((IfConditionActivity)act).IfFalseActivities)\n RecordActivity(pipelineName, a);\n }\n\n if (act is SwitchActivity)\n foreach (var c in ((SwitchActivity)act).Cases)\n foreach (var a in c.Activities)\n RecordActivity(pipelineName, a);\n }\n}\n"
},
{
"path": "FactoryTesting/Helpers/DataFactoryHelper.cs",
"content": "using Microsoft.Azure.Management.DataFactory;\nusing Microsoft.Azure.Management.DataFactory.Models;\nusing Microsoft.IdentityModel.Clients.ActiveDirectory;\nusing Microsoft.Rest;\nusing System;\nusing System.Collections.Generic;\nusing System.Linq;\nusing System.Runtime.CompilerServices;\nusing System.Threading;\nusing System.Threading.Tasks;\n\nnamespace FactoryTesting.Helpers\n{\n public class DataFactoryHelper : SettingsHelper where T : DataFactoryHelper\n {\n private readonly string _adfName;\n private readonly string _rgName;\n private DataFactoryManagementClient _adfClient;\n\n private async Task InitialiseClient()\n {\n if (_adfClient != null)\n return;\n\n var context = new AuthenticationContext(\"https://login.windows.net/\" + GetSetting(\"AZURE_TENANT_ID\"));\n var cc = new ClientCredential(GetSetting(\"AZURE_CLIENT_ID\"), GetSetting(\"AZURE_CLIENT_SECRET\"));\n var authResult = await context.AcquireTokenAsync(\"https://management.azure.com/\", cc);\n\n var cred = new TokenCredentials(authResult.AccessToken);\n _adfClient = new DataFactoryManagementClient(cred) { SubscriptionId = GetSetting(\"AZURE_SUBSCRIPTION_ID\") };\n }\n\n public async Task TriggerPipeline(string pipelineName, IDictionary parameters)\n {\n await InitialiseClient();\n var response = await _adfClient.Pipelines.CreateRunWithHttpMessagesAsync(_rgName, _adfName, pipelineName, parameters: parameters);\n return response.Body.RunId;\n }\n\n public async Task> GetPipelines()\n {\n await InitialiseClient();\n\n var page = await _adfClient.Pipelines.ListByFactoryAsync(_rgName, _adfName);\n var pipelines = page.ToList();\n\n while (!string.IsNullOrWhiteSpace(page.NextPageLink))\n {\n page = await _adfClient.Pipelines.ListByFactoryNextAsync(page.NextPageLink);\n pipelines.AddRange(page.ToList());\n }\n\n return pipelines;\n }\n\n public async Task> GetActivityRuns(string pipelineRunId)\n {\n await InitialiseClient();\n\n var filter = new RunFilterParameters(DateTime.MinValue, DateTime.UtcNow);\n var arqr = await _adfClient.ActivityRuns.QueryByPipelineRunAsync(_rgName, _adfName, pipelineRunId, filter);\n var activityRuns = arqr.Value.ToList();\n\n while (!string.IsNullOrWhiteSpace(arqr.ContinuationToken))\n {\n filter.ContinuationToken = arqr.ContinuationToken;\n arqr = await _adfClient.ActivityRuns.QueryByPipelineRunAsync(_rgName, _adfName, pipelineRunId, filter);\n activityRuns.AddRange(arqr.Value);\n }\n\n return activityRuns;\n }\n public virtual void TearDown()\n {\n _adfClient?.Dispose();\n }\n\n public async Task GetRunStatus(string pipelineRunId)\n {\n await InitialiseClient();\n var run = await _adfClient.PipelineRuns.GetAsync(_rgName, _adfName, pipelineRunId);\n return run.Status;\n }\n public async Task GetRunStatus(string pipelineRunId, string adfName)\n {\n await InitialiseClient();\n var run = await _adfClient.PipelineRuns.GetAsync(_rgName, adfName, pipelineRunId);\n return run.Status;\n }\n\n public async Task IsInProgress(string pipelineRunId)\n {\n await InitialiseClient();\n var status = await GetRunStatus(pipelineRunId);\n return status == \"Queued\" || status == \"InProgress\" || status == \"Cancelling\";\n }\n public async Task IsInProgress(string pipelineRunId, string adfName)\n {\n await InitialiseClient();\n var status = await GetRunStatus(pipelineRunId, adfName);\n return status == \"Queued\" || status == \"InProgress\" || status == \"Cancelling\";\n }\n\n public async Task IsQueued(string pipelineRunId)\n {\n await InitialiseClient();\n var status = await GetRunStatus(pipelineRunId);\n return status == \"Queued\";\n }\n\n public async Task IsCancelling(string pipelineRunId)\n {\n await InitialiseClient();\n var status = await GetRunStatus(pipelineRunId);\n return status == \"Cancelling\" || status == \"Canceling\"; //microsoft typo\n }\n\n public async Task IsCancelling(string pipelineRunId, string adfName)\n {\n await InitialiseClient();\n var status = await GetRunStatus(pipelineRunId, adfName);\n return status == \"Cancelling\" || status == \"Canceling\"; //microsoft typo\n }\n\n public async Task CancelRunningPipeline(string pipelineRunId, bool recurseCancel = true)\n {\n await InitialiseClient();\n if (await IsInProgress(pipelineRunId))\n {\n _adfClient.PipelineRuns.Cancel(_rgName, _adfName, pipelineRunId, recurseCancel);\n\n while (await IsCancelling(pipelineRunId))\n Thread.Sleep(2000);\n }\n else\n {\n throw new Exception(\"Pipeline is not in a state that can be cancelled.\");\n }\n\n string status = await GetRunStatus(pipelineRunId);\n return status;\n }\n\n public async Task CancelRunningPipeline(string pipelineRunId, string adfName, bool recurseCancel = true)\n {\n await InitialiseClient();\n if (await IsInProgress(pipelineRunId, adfName))\n {\n _adfClient.PipelineRuns.Cancel(_rgName, adfName, pipelineRunId, recurseCancel);\n\n while (await IsCancelling(pipelineRunId, adfName))\n Thread.Sleep(2000);\n }\n else\n {\n throw new Exception(\"Pipeline is not in a state that can be cancelled.\");\n }\n\n string status = await GetRunStatus(pipelineRunId, adfName);\n return status;\n }\n\n public DataFactoryHelper()\n {\n _adfName = GetSetting(\"DataFactoryName\");\n _rgName = GetSetting(\"DataFactoryResourceGroup\");\n }\n }\n}\n"
},
{
"path": "FactoryTesting/Helpers/DatabaseHelper.cs",
"content": "using System.Collections.Generic;\nusing System.Data;\nusing System.Data.SqlClient;\n\nnamespace FactoryTesting.Helpers\n{\n public class DatabaseHelper : PipelineRunHelper where T : DatabaseHelper\n {\n public SqlConnection _conn;\n\n public DatabaseHelper()\n {\n _conn = new SqlConnection(GetSetting(GetSetting(\"MetadataDbConnectionStringSecretName\")));\n _conn.Open();\n }\n\n public T WithEmptyTable(string tableName)\n {\n using (var cmd = new SqlCommand($\"TRUNCATE TABLE {tableName}\", _conn))\n cmd.ExecuteNonQuery();\n return (T)this;\n }\n\n public int RowCount(string tableName, string where = \"\", string equals = \"\")\n {\n using (var cmd = new SqlCommand($\"SELECT COUNT(*) FROM {tableName}\"\n + (where?.Length == 0 ? \"\" : $\" WHERE {where} = '{equals.Replace(\"'\", \"''\")}'\")\n , _conn))\n using (var reader = cmd.ExecuteReader())\n {\n reader.Read();\n return reader.GetInt32(0);\n }\n }\n\n public string ColumnData(string tableName, string columnName, char separator = ',')\n {\n using (var cmd = new SqlCommand($\"SELECT STRING_AGG([{columnName}],'{separator}') FROM {tableName}\", _conn))\n using (var reader = cmd.ExecuteReader())\n {\n reader.Read();\n return reader.GetString(0);\n }\n }\n\n public void ExecuteStoredProcedure(string spName, Dictionary parameters = null)\n {\n using (var cmd = new SqlCommand(spName, _conn))\n {\n cmd.CommandType = CommandType.StoredProcedure;\n if (parameters != null)\n foreach (string parameterName in parameters.Keys)\n cmd.Parameters.Add(new SqlParameter(parameterName, parameters[parameterName]));\n cmd.ExecuteNonQuery();\n }\n }\n\n public void ExecuteNonQuery(string sql)\n {\n using (var cmd = new SqlCommand(sql, _conn))\n cmd.ExecuteNonQuery();\n }\n\n public void AddTenantAndSubscription(string tenantId = null, string subscriptionId = null)\n {\n if (string.IsNullOrEmpty(tenantId)) tenantId = GetSetting(\"AZURE_TENANT_ID\");\n if (string.IsNullOrEmpty(subscriptionId)) subscriptionId = GetSetting(\"AZURE_SUBSCRIPTION_ID\");\n\n ExecuteNonQuery($\"INSERT INTO[procfwk].[Tenants] ([TenantId],[Name],[Description]) VALUES ('{tenantId}', 'mrpaulandrew.com', NULL);\");\n ExecuteNonQuery($\"INSERT INTO [procfwk].[Subscriptions] ([SubscriptionId],[Name],[Description],[TenantId]) VALUES ('{subscriptionId}', 'Microsoft Sponsored Subscription', NULL, '{tenantId}');\");\n ExecuteNonQuery($\"UPDATE [procfwk].[Orchestrators] SET [SubscriptionId] = '{subscriptionId}';\");\n }\n\n public void AddWorkerSPNStoredInDatabase(string workerFactoryName, string orchestratorType = \"ADF\")\n {\n ExecuteNonQuery(\"UPDATE [procfwk].[Properties] SET [PropertyValue] = 'StoreInDatabase' WHERE [PropertyName] = 'SPNHandlingMethod';\");\n\n var parameters = new Dictionary\n {\n [\"@OrchestratorName\"] = workerFactoryName,\n [\"@OrchestratorType\"] = orchestratorType,\n [\"@PrincipalIdValue\"] = GetSetting(\"AZURE_CLIENT_ID\"),\n [\"@PrincipalSecretValue\"] = GetSetting(\"AZURE_CLIENT_SECRET\"),\n [\"@PrincipalName\"] = GetSetting(\"AZURE_CLIENT_NAME\")\n };\n\n ExecuteStoredProcedure(\"[procfwkHelpers].[AddServicePrincipalWrapper]\", parameters);\n }\n\n public void AddWorkerSPNStoredInKeyVault(string workerFactoryName, string orchestratorType = \"ADF\")\n {\n ExecuteNonQuery(\"UPDATE [procfwk].[Properties] SET [PropertyValue] = 'StoreInKeyVault' WHERE [PropertyName] = 'SPNHandlingMethod';\");\n\n var parameters = new Dictionary\n {\n [\"@OrchestratorName\"] = workerFactoryName,\n [\"@OrchestratorType\"] = orchestratorType,\n [\"@PrincipalIdValue\"] = GetSetting(\"AZURE_CLIENT_ID_URL\"),\n [\"@PrincipalSecretValue\"] = GetSetting(\"AZURE_CLIENT_SECRET_URL\"),\n [\"@PrincipalName\"] = GetSetting(\"AZURE_CLIENT_NAME\")\n };\n\n ExecuteStoredProcedure(\"[procfwkHelpers].[AddServicePrincipalWrapper]\", parameters);\n }\n\n public void AddBasicMetadata()\n {\n ExecuteStoredProcedure(\"[procfwkTesting].[ResetMetadata]\", null);\n ExecuteNonQuery(\"UPDATE [procfwk].[Orchestrators] SET [IsFrameworkOrchestrator] = '0';\");\n ExecuteNonQuery($\"UPDATE [procfwk].[Orchestrators] SET [IsFrameworkOrchestrator] = '1' WHERE [OrchestratorName] = '{GetSetting(\"DataFactoryName\")}';\");\n }\n\n public override void TearDown()\n {\n ExecuteStoredProcedure(\"[procfwkTesting].[CleanUpMetadata]\", null);\n\n _conn?.Dispose();\n base.TearDown();\n }\n }\n}\n"
},
{
"path": "FactoryTesting/Helpers/PipelineRunHelper.cs",
"content": "using Microsoft.Azure.Management.DataFactory.Models;\nusing Newtonsoft.Json.Linq;\nusing System;\nusing System.Collections.Generic;\nusing System.Linq;\nusing System.Text.RegularExpressions;\nusing System.Threading;\nusing System.Threading.Tasks;\n\nnamespace FactoryTesting.Helpers\n{\n public class PipelineRunHelper : DataFactoryHelper where T : PipelineRunHelper\n {\n private readonly Dictionary _parameters;\n private List _activityRuns;\n private bool _hasRun;\n\n public string RunId { get; private set; }\n\n public string RunOutcome { get; private set; }\n\n public PipelineRunHelper()\n {\n RunId = \"Unknown\";\n RunOutcome = \"Unknown\";\n _hasRun = false;\n _parameters = new Dictionary();\n }\n\n public T WithParameter(string name, object value)\n {\n _parameters[name] = value;\n return (T)this;\n }\n\n public async Task RunPipeline(string pipelineName)\n {\n if (_hasRun)\n throw new Exception(\"RunPipeline() can only be called once per instance lifetime\");\n _hasRun = true;\n\n RunId = await TriggerPipeline(pipelineName, _parameters);\n while (await IsInProgress(RunId))\n Thread.Sleep(2000);\n RunOutcome = await GetRunStatus(RunId);\n }\n\n public async Task RunPipelineAndCancel(string pipelineName)\n {\n if (_hasRun)\n throw new Exception(\"RunPipeline() can only be called once per instance lifetime\");\n _hasRun = true;\n\n RunId = await TriggerPipeline(pipelineName, _parameters);\n while (await IsQueued(RunId))\n Thread.Sleep(2000);\n\n await CancelRunningPipeline(RunId, true);\n RunOutcome = await GetRunStatus(RunId);\n }\n\n public async Task> GetActivityRuns()\n {\n await InitialiseActivityRuns();\n return _activityRuns;\n }\n\n public async Task GetActivityRunCount(string pattern = \".*\")\n {\n await InitialiseActivityRuns();\n Regex rgx = new Regex(pattern);\n return _activityRuns.Where(ar => rgx.IsMatch(ar.ActivityName)).Count();\n }\n\n private async Task InitialiseActivityRuns()\n {\n if (_activityRuns == null)\n _activityRuns = await GetActivityRuns(RunId);\n }\n\n public async Task GetActivityOutput(string activityName, string propertyPath = \"$\")\n {\n await InitialiseActivityRuns();\n string output = _activityRuns.Where(ar => ar.ActivityName == activityName).FirstOrDefault().Output.ToString();\n var obj = JObject.Parse(output);\n return obj.SelectToken(propertyPath).ToString();\n }\n }\n}\n"
},
{
"path": "FactoryTesting/Helpers/SettingsHelper.cs",
"content": "using Azure.Identity;\nusing Azure.Security.KeyVault.Secrets;\nusing NUnit.Framework;\nusing System;\n\nnamespace FactoryTesting.Helpers\n{\n public class SettingsHelper\n {\n public string GetSetting(string settingName)\n {\n // return environment variable \"settingName\", if present\n var value = Environment.GetEnvironmentVariable(settingName);\n if (value?.Length > 0)\n return value;\n\n // return value of runsettings parameter \"settingName\", if present\n value = TestContext.Parameters[settingName];\n if (value?.Length > 0)\n return value;\n\n // if a key vault is specified, return the value of secret \"settingName\", if present\n if (_keyVaultClient != null)\n {\n value = _keyVaultClient.GetSecret(settingName).Value.Value;\n if (value?.Length > 0)\n return value;\n }\n\n throw new Exception($\"Test setting '{settingName}' not found\");\n }\n\n private readonly SecretClient _keyVaultClient;\n\n public SettingsHelper()\n {\n var kvUrl = TestContext.Parameters[\"KeyVaultUrl\"];\n if (kvUrl?.Length > 0)\n _keyVaultClient = new SecretClient(new Uri(kvUrl), new DefaultAzureCredential());\n }\n }\n}"
},
{
"path": "FactoryTesting/Pipelines/01-Grandparent/Given300WorkerPipelines.cs",
"content": "using FluentAssertions;\nusing NUnit.Framework;\nusing NUnit.Framework.Internal;\nusing System.Threading.Tasks;\n\nnamespace FactoryTesting.Pipelines.Grandparent\n{\n public class Given300WorkerPipelines\n {\n private GrandparentHelper _helper;\n\n [OneTimeSetUp]\n public async Task WhenPipelineRun()\n {\n _helper = new GrandparentHelper()\n .WithBasicMetadata()\n .WithTenantAndSubscriptionIds()\n .With300WorkerPipelinesEnabled()\n .WithSPNInKeyVault(\"WorkersFactory\");\n\n await _helper.RunPipeline();\n }\n\n #region Integration tests\n\n [Test]\n public void ThenPipelineOutcomeIsSucceeded()\n {\n _helper.RunOutcome.Should().Be(\"Succeeded\");\n }\n [Test]\n public void ThenCurrentExecutionTableIsEmpty()\n {\n _helper.RowCount(\"procfwk.CurrentExecution\").Should().Be(0);\n }\n\n [Test]\n public void Then300ExecutionLogRecords()\n {\n _helper.RowCount(\"procfwk.ExecutionLog\").Should().Be(300);\n }\n\n #endregion\n\n [OneTimeTearDown]\n public void TearDown()\n {\n _helper?.TearDown();\n }\n }\n}\n"
},
{
"path": "FactoryTesting/Pipelines/01-Grandparent/Given600WorkerPipelineConcurrentBatches.cs",
"content": "using FluentAssertions;\nusing NUnit.Framework;\nusing NUnit.Framework.Internal;\nusing System.Threading.Tasks;\n\nnamespace FactoryTesting.Pipelines.Grandparent\n{\n public class Given600WorkerPipelineConcurrentBatches\n {\n private GrandparentHelper _helperFirstBatch;\n private GrandparentHelper _helperSecondBatch;\n\n [OneTimeSetUp]\n public async Task WhenPipelineRun()\n {\n _helperFirstBatch = new GrandparentHelper()\n .WithBasicMetadata()\n .WithTenantAndSubscriptionIds()\n .With300WorkerPipelinesEnabled()\n .WithBatchExecutionHandling()\n .With300WorkerPipelineBatches()\n .WithSPNInKeyVault(\"WorkersFactory\")\n .WithParameter(\"BatchName\", \"0to300\");\n\n _helperSecondBatch = new GrandparentHelper()\n .WithParameter(\"BatchName\", \"301to600\");\n\n var firstBatch = _helperFirstBatch.RunPipeline();\n var secondBatch = _helperSecondBatch.RunPipeline();\n\n await Task.WhenAll(firstBatch, secondBatch);\n }\n\n #region Integration tests\n\n [Test]\n public void ThenFirstBatchPipelineOutcomeIsSucceeded()\n {\n _helperFirstBatch.RunOutcome.Should().Be(\"Succeeded\");\n }\n\n [Test]\n public void ThenSecondBatchPipelineOutcomeIsSucceeded()\n {\n _helperSecondBatch.RunOutcome.Should().Be(\"Succeeded\");\n }\n \n [Test]\n public void ThenTwoBatchExecutionSuccessRecords()\n {\n _helperFirstBatch.RowCount(\"procfwk.BatchExecution\", where: \"BatchStatus\", equals: \"Success\").Should().Be(2);\n }\n\n [Test]\n public void ThenCurrentExecutionTableIsEmpty()\n {\n _helperFirstBatch.RowCount(\"procfwk.CurrentExecution\").Should().Be(0);\n }\n\n [Test]\n public void Then300ExecutionLogRecords()\n {\n _helperFirstBatch.RowCount(\"procfwk.ExecutionLog\").Should().Be(600);\n }\n\n #endregion\n\n [OneTimeTearDown]\n public void TearDown()\n {\n _helperFirstBatch?.TearDown();\n }\n }\n}\n"
},
{
"path": "FactoryTesting/Pipelines/01-Grandparent/GivenOneWorkerPipeline.cs",
"content": "using FluentAssertions;\nusing NUnit.Framework;\nusing NUnit.Framework.Internal;\nusing System.Threading.Tasks;\n\nnamespace FactoryTesting.Pipelines.Grandparent\n{\n public class GivenOneWorkerPipeline\n {\n private GrandparentHelper _helper;\n\n [OneTimeSetUp]\n public async Task WhenPipelineRun()\n {\n _helper = new GrandparentHelper()\n .WithBasicMetadata()\n .WithTenantAndSubscriptionIds()\n .WithSPNInDatabase(\"FrameworkFactory\")\n .WithEmptyExecutionTables()\n .WithSimpleFailureHandling()\n .WithOneWorkerPipelineEnabled();\n await _helper.RunPipeline();\n }\n\n #region Integration tests\n\n [Test]\n public void ThenPipelineOutcomeIsSucceeded()\n {\n _helper.RunOutcome.Should().Be(\"Succeeded\");\n }\n\n #endregion\n \n [OneTimeTearDown]\n public void TearDown()\n {\n _helper?.TearDown();\n }\n }\n}\n"
},
{
"path": "FactoryTesting/Pipelines/01-Grandparent/GrandparentHelper.cs",
"content": "using FactoryTesting.Helpers;\nusing System.Collections.Generic;\nusing System.Threading.Tasks;\n\nnamespace FactoryTesting.Pipelines.Grandparent\n{\n class GrandparentHelper : CoverageHelper\n {\n public async Task RunPipeline()\n {\n await RunPipeline(\"01-Grandparent\");\n }\n\n public GrandparentHelper WithTenantAndSubscriptionIds()\n {\n AddTenantAndSubscription();\n return this;\n }\n\n public GrandparentHelper WithSPNInDatabase(string workerFactoryName)\n {\n AddWorkerSPNStoredInDatabase(workerFactoryName);\n return this;\n }\n\n public GrandparentHelper WithSPNInKeyVault(string workerFactoryName)\n {\n AddWorkerSPNStoredInKeyVault(workerFactoryName);\n return this;\n }\n\n public GrandparentHelper WithBasicMetadata()\n {\n AddBasicMetadata();\n return this;\n }\n\n public GrandparentHelper WithEmptyExecutionTables()\n {\n WithEmptyTable(\"procfwk.CurrentExecution\");\n WithEmptyTable(\"procfwk.ExecutionLog\");\n WithEmptyTable(\"procfwk.ErrorLog\");\n\n return this;\n }\n\n public GrandparentHelper WithBatchExecutionHandling()\n {\n ExecuteNonQuery(@$\"UPDATE [procfwk].[Properties] \nSET [PropertyValue] = '1' \nWHERE [PropertyName] = 'UseExecutionBatches'\");\n return this;\n }\n\n public GrandparentHelper WithSimpleFailureHandling()\n {\n ExecuteNonQuery(\"UPDATE [procfwk].[Properties] SET [PropertyValue] = 'Simple' WHERE [PropertyName] = 'FailureHandling'\");\n return this;\n }\n public GrandparentHelper WithOneWorkerPipelineEnabled()\n {\n ExecuteNonQuery(\"UPDATE [procfwk].[Pipelines] SET [Enabled] = 0 WHERE [PipelineId] > 1\");\n return this;\n }\n\n public GrandparentHelper With300WorkerPipelinesEnabled()\n {\n ExecuteStoredProcedure(\"[procfwkTesting].[Add300WorkerPipelines]\", null);\n\n return this;\n }\n\n public GrandparentHelper With20BatchesFor1000WorkersEnabled()\n {\n ExecuteStoredProcedure(\"[procfwkTesting].[Add20BatchesFor1000Workers]\", null);\n\n return this;\n }\n\n public GrandparentHelper WithCustom()\n {\n ExecuteStoredProcedure(\"[dbo].[PaulTemp]\", null);\n\n return this;\n }\n\n public GrandparentHelper With300WorkerPipelineBatches()\n {\n ExecuteStoredProcedure(\"[procfwkTesting].[Add300WorkerPipelineBatches]\", null);\n\n return this;\n }\n }\n}\n"
},
{
"path": "FactoryTesting/Pipelines/02-Parent/Given20ConcurrentBatchesFor1000WorkerPipelines.cs",
"content": "using FluentAssertions;\nusing NUnit.Framework;\nusing NUnit.Framework.Internal;\nusing System.Threading.Tasks;\n\nnamespace FactoryTesting.Pipelines.Parent\n{\n public class Given20ConcurrentBatchesFor1000WorkerPipelines\n {\n private ParentHelper _helperBatchOne;\n private ParentHelper _helperBatchTwo;\n private ParentHelper _helperBatchEleven;\n private ParentHelper _helperBatchTwelve;\n private ParentHelper _helperBatchTwenty;\n private ParentHelper _helperBatchFifteen;\n private ParentHelper _helperBatchSeven;\n private ParentHelper _helperBatchNine;\n private ParentHelper _helperBatchEight;\n private ParentHelper _helperBatchSixteen;\n private ParentHelper _helperBatchFive;\n private ParentHelper _helperBatchSix;\n private ParentHelper _helperBatchThirteen;\n private ParentHelper _helperBatchNineteen;\n private ParentHelper _helperBatchFour;\n private ParentHelper _helperBatchEighteen;\n private ParentHelper _helperBatchThree;\n private ParentHelper _helperBatchFourteen;\n private ParentHelper _helperBatchTen;\n private ParentHelper _helperBatchSeventeen;\n\n [OneTimeSetUp]\n public async Task WhenPipelineRun()\n {\n /*\n _helperBatchOne = new ParentHelper()\n .WithBasicMetadata()\n .WithTenantAndSubscriptionIds()\n .WithBatchExecutionHandling()\n .With20BatchesFor1000WorkersEnabled()\n .WithSPNInKeyVault(\"WorkersFactory\")\n .WithParameter(\"BatchName\", \"One\");\n */\n _helperBatchOne = new ParentHelper()\n .WithCustom()\n .WithParameter(\"BatchName\", \"One\");\n \n _helperBatchTwo = new ParentHelper().WithParameter(\"BatchName\", \"Two\");\n _helperBatchEleven = new ParentHelper().WithParameter(\"BatchName\", \"Eleven\");\n _helperBatchTwelve = new ParentHelper().WithParameter(\"BatchName\", \"Twelve\");\n _helperBatchTwenty = new ParentHelper().WithParameter(\"BatchName\", \"Twenty\");\n _helperBatchFifteen = new ParentHelper().WithParameter(\"BatchName\", \"Fifteen\");\n _helperBatchSeven = new ParentHelper().WithParameter(\"BatchName\", \"Seven\");\n _helperBatchNine = new ParentHelper().WithParameter(\"BatchName\", \"Nine\");\n _helperBatchEight = new ParentHelper().WithParameter(\"BatchName\", \"Eight\");\n _helperBatchSixteen = new ParentHelper().WithParameter(\"BatchName\", \"Sixteen\");\n _helperBatchFive = new ParentHelper().WithParameter(\"BatchName\", \"Five\");\n _helperBatchSix = new ParentHelper().WithParameter(\"BatchName\", \"Six\");\n _helperBatchThirteen = new ParentHelper().WithParameter(\"BatchName\", \"Thirteen\");\n _helperBatchNineteen = new ParentHelper().WithParameter(\"BatchName\", \"Nineteen\");\n _helperBatchFour = new ParentHelper().WithParameter(\"BatchName\", \"Four\");\n _helperBatchEighteen = new ParentHelper().WithParameter(\"BatchName\", \"Eighteen\");\n _helperBatchThree = new ParentHelper().WithParameter(\"BatchName\", \"Three\");\n _helperBatchFourteen = new ParentHelper().WithParameter(\"BatchName\", \"Fourteen\");\n _helperBatchTen = new ParentHelper().WithParameter(\"BatchName\", \"Ten\");\n _helperBatchSeventeen = new ParentHelper().WithParameter(\"BatchName\", \"Seventeen\");\n \n var batchOne = _helperBatchOne.RunPipeline();\n var batchTwo = _helperBatchTwo.RunPipeline();\n var batchEleven = _helperBatchEleven.RunPipeline();\n var batchTwelve = _helperBatchTwelve.RunPipeline();\n var batchTwenty = _helperBatchTwenty.RunPipeline();\n var batchFifteen = _helperBatchFifteen.RunPipeline();\n var batchSeven = _helperBatchSeven.RunPipeline();\n var batchNine = _helperBatchNine.RunPipeline();\n var batchEight = _helperBatchEight.RunPipeline();\n var batchSixteen = _helperBatchSixteen.RunPipeline();\n var batchFive = _helperBatchFive.RunPipeline();\n var batchSix = _helperBatchSix.RunPipeline();\n var batchThirteen = _helperBatchThirteen.RunPipeline();\n var batchNineteen = _helperBatchNineteen.RunPipeline();\n var batchFour = _helperBatchFour.RunPipeline();\n var batchEighteen = _helperBatchEighteen.RunPipeline();\n var batchThree = _helperBatchThree.RunPipeline();\n var batchFourteen = _helperBatchFourteen.RunPipeline();\n var batchTen = _helperBatchTen.RunPipeline();\n var batchSeventeen = _helperBatchSeventeen.RunPipeline();\n \n //await _helperBatchOne.RunPipeline();\n \n await Task.WhenAll(\n batchOne,\n batchTwo,\n batchEleven,\n batchTwelve,\n batchTwenty,\n batchFifteen,\n batchSeven,\n batchNine,\n batchEight,\n batchSixteen,\n batchFive,\n batchSix,\n batchThirteen,\n batchNineteen,\n batchFour,\n batchEighteen,\n batchThree,\n batchFourteen,\n batchTen,\n batchSeventeen\n );\n }\n\n #region Integration tests\n\n [Test]\n public void ThenOneBatchPipelineOutcomeIsSucceeded()\n {\n _helperBatchOne.RunOutcome.Should().Be(\"Succeeded\");\n }\n \n [Test]\n public void ThenTwoBatchPipelineOutcomeIsSucceeded()\n {\n _helperBatchTwo.RunOutcome.Should().Be(\"Succeeded\");\n }\n [Test]\n public void ThenElevenBatchPipelineOutcomeIsSucceeded()\n {\n _helperBatchEleven.RunOutcome.Should().Be(\"Succeeded\");\n }\n [Test]\n public void ThenTwelveBatchPipelineOutcomeIsSucceeded()\n {\n _helperBatchTwelve.RunOutcome.Should().Be(\"Succeeded\");\n }\n [Test]\n public void ThenTwentyBatchPipelineOutcomeIsSucceeded()\n {\n _helperBatchTwenty.RunOutcome.Should().Be(\"Succeeded\");\n }\n [Test]\n public void ThenFifteenBatchPipelineOutcomeIsSucceeded()\n {\n _helperBatchFifteen.RunOutcome.Should().Be(\"Succeeded\");\n }\n [Test]\n public void ThenSevenBatchPipelineOutcomeIsSucceeded()\n {\n _helperBatchSeven.RunOutcome.Should().Be(\"Succeeded\");\n }\n [Test]\n public void ThenNineBatchPipelineOutcomeIsSucceeded()\n {\n _helperBatchNine.RunOutcome.Should().Be(\"Succeeded\");\n }\n [Test]\n public void ThenEightBatchPipelineOutcomeIsSucceeded()\n {\n _helperBatchEight.RunOutcome.Should().Be(\"Succeeded\");\n }\n [Test]\n public void ThenSixteenBatchPipelineOutcomeIsSucceeded()\n {\n _helperBatchSixteen.RunOutcome.Should().Be(\"Succeeded\");\n }\n [Test]\n public void ThenFiveBatchPipelineOutcomeIsSucceeded()\n {\n _helperBatchFive.RunOutcome.Should().Be(\"Succeeded\");\n }\n [Test]\n public void ThenSixBatchPipelineOutcomeIsSucceeded()\n {\n _helperBatchSix.RunOutcome.Should().Be(\"Succeeded\");\n }\n [Test]\n public void ThenThirteenBatchPipelineOutcomeIsSucceeded()\n {\n _helperBatchThirteen.RunOutcome.Should().Be(\"Succeeded\");\n }\n [Test]\n public void ThenNineteenBatchPipelineOutcomeIsSucceeded()\n {\n _helperBatchNineteen.RunOutcome.Should().Be(\"Succeeded\");\n }\n [Test]\n public void ThenFourBatchPipelineOutcomeIsSucceeded()\n {\n _helperBatchFour.RunOutcome.Should().Be(\"Succeeded\");\n }\n [Test]\n public void ThenEighteenBatchPipelineOutcomeIsSucceeded()\n {\n _helperBatchEighteen.RunOutcome.Should().Be(\"Succeeded\");\n }\n [Test]\n public void ThenThreeBatchPipelineOutcomeIsSucceeded()\n {\n _helperBatchThree.RunOutcome.Should().Be(\"Succeeded\");\n }\n [Test]\n public void ThenFourteenBatchPipelineOutcomeIsSucceeded()\n {\n _helperBatchFourteen.RunOutcome.Should().Be(\"Succeeded\");\n }\n [Test]\n public void ThenTenBatchPipelineOutcomeIsSucceeded()\n {\n _helperBatchTen.RunOutcome.Should().Be(\"Succeeded\");\n }\n [Test]\n public void ThenSeventeenBatchPipelineOutcomeIsSucceeded()\n {\n _helperBatchSeventeen.RunOutcome.Should().Be(\"Succeeded\");\n }\n\n [Test]\n public void Then1000ExecutionLogRecords()\n {\n _helperBatchOne.RowCount(\"procfwk.ExecutionLog\").Should().Be(1000);\n }\n \n #endregion\n /*\n [OneTimeTearDown]\n public void TearDown()\n {\n _helperBatchOne?.TearDown();\n }\n */\n }\n}\n"
},
{
"path": "FactoryTesting/Pipelines/02-Parent/GivenAlreadyRunning.cs",
"content": "using FluentAssertions;\nusing NUnit.Framework;\nusing NUnit.Framework.Internal;\nusing System.Threading.Tasks;\n\nnamespace FactoryTesting.Pipelines.Parent\n{\n public class GivenAlreadyRunning\n {\n private ParentHelper _helperRunOne;\n private ParentHelper _helperRunTwo;\n\n [OneTimeSetUp]\n public async Task WhenPipelineRun()\n {\n _helperRunOne = new ParentHelper()\n .WithBasicMetadata()\n .WithTenantAndSubscriptionIds()\n .WithSPNInDatabase(\"FrameworkFactory\")\n .WithEmptyExecutionTables()\n .WithoutSimulatedError()\n .WithFailureHandling(\"Simple\");\n\n _helperRunTwo = new ParentHelper();\n\n var firstRun = _helperRunOne.RunPipeline();\n var secondRun = _helperRunTwo.RunPipeline(15000); //15 second delay in run\n\n await Task.WhenAll(firstRun, secondRun);\n }\n\n [Test]\n public void ThenFirstPipelineOutcomeIsSucceeded()\n {\n _helperRunOne.RunOutcome.Should().Be(\"Succeeded\");\n }\n\n [Test]\n public void ThenSecondPipelineOutcomeIsFailed()\n {\n _helperRunTwo.RunOutcome.Should().Be(\"Failed\");\n }\n\n [Test]\n public void ThenCurrentExecutionTableIsEmpty()\n {\n _helperRunOne.RowCount(\"procfwk.CurrentExecution\").Should().Be(0);\n }\n\n [Test]\n public void ThenElevenExecutionLogRecords()\n {\n _helperRunOne.RowCount(\"procfwk.ExecutionLog\").Should().Be(11);\n }\n\n [OneTimeTearDown]\n public void TearDown()\n {\n _helperRunOne?.TearDown();\n }\n }\n}\n"
},
{
"path": "FactoryTesting/Pipelines/02-Parent/GivenBatchExecutionsForConcurrentBatches.cs",
"content": "using FluentAssertions;\nusing NUnit.Framework;\nusing NUnit.Framework.Internal;\nusing System.Threading.Tasks;\n\nnamespace FactoryTesting.Pipelines.Parent\n{\n public class GivenBatchExecutionsForConcurrentBatches\n {\n private ParentHelper _helperFirstBatch;\n private ParentHelper _helperSecondBatch;\n\n [OneTimeSetUp]\n public async Task WhenPipelineRun()\n {\n _helperFirstBatch = new ParentHelper()\n .WithBasicMetadata()\n .WithTenantAndSubscriptionIds()\n .WithSPNInDatabase(\"FrameworkFactory\")\n .WithEmptyExecutionTables()\n .WithBatchExecutionHandling()\n .WithStagesEnabled()\n .WithPipelinesEnabled()\n .WithParameter(\"BatchName\", \"Hourly\");\n\n _helperSecondBatch = new ParentHelper()\n .WithParameter(\"BatchName\", \"Daily\");\n\n var firstBatch = _helperFirstBatch.RunPipeline();\n var secondBatch = _helperSecondBatch.RunPipeline();\n\n await Task.WhenAll(firstBatch, secondBatch);\n }\n #region Integration tests\n\n [Test]\n public void ThenFirstBatchPipelineOutcomeIsSucceeded()\n {\n _helperFirstBatch.RunOutcome.Should().Be(\"Succeeded\");\n }\n\n [Test]\n public void ThenSecondBatchPipelineOutcomeIsSucceeded()\n {\n _helperSecondBatch.RunOutcome.Should().Be(\"Succeeded\");\n }\n\n [Test]\n public void ThenCurrentExecutionTableIsEmpty()\n {\n _helperFirstBatch.RowCount(\"procfwk.CurrentExecution\").Should().Be(0);\n }\n\n [Test]\n public void ThenFifteenExecutionLogSuccessRecords()\n {\n _helperFirstBatch.RowCount(\"procfwk.ExecutionLog\", where: \"PipelineStatus\", equals: \"Success\").Should().Be(15);\n }\n\n [Test]\n public void ThenTwoBatchExecutionSuccessRecords()\n {\n _helperFirstBatch.RowCount(\"procfwk.BatchExecution\", where: \"BatchStatus\", equals: \"Success\").Should().Be(2);\n }\n #endregion\n\n [OneTimeTearDown]\n public void TearDown()\n {\n _helperFirstBatch?.TearDown();\n }\n }\n}\n"
},
{
"path": "FactoryTesting/Pipelines/02-Parent/GivenBatchExecutionsForConcurrentBatchesAlreadyRunning.cs",
"content": "using FluentAssertions;\nusing NUnit.Framework;\nusing NUnit.Framework.Internal;\nusing System.Threading.Tasks;\n\nnamespace FactoryTesting.Pipelines.Parent\n{\n public class GivenBatchExecutionsForConcurrentBatchesAlreadyRunning\n {\n private ParentHelper _helperFirstBatch;\n private ParentHelper _helperSecondBatch;\n\n [OneTimeSetUp]\n public async Task WhenPipelineRun()\n {\n _helperFirstBatch = new ParentHelper()\n .WithBasicMetadata()\n .WithTenantAndSubscriptionIds()\n .WithSPNInDatabase(\"FrameworkFactory\")\n .WithEmptyExecutionTables()\n .WithBatchExecutionHandling()\n .WithStagesEnabled()\n .WithPipelinesEnabled()\n .WithParameter(\"BatchName\", \"Hourly\");\n\n _helperSecondBatch = new ParentHelper()\n .WithParameter(\"BatchName\", \"Hourly\");\n\n var firstBatch = _helperFirstBatch.RunPipeline();\n var secondBatch = _helperSecondBatch.RunPipeline(15000); //15 second delay in run\n\n await Task.WhenAll(firstBatch, secondBatch);\n }\n #region Integration tests\n\n [Test]\n public void ThenFirstBatchPipelineOutcomeIsSucceeded()\n {\n _helperFirstBatch.RunOutcome.Should().Be(\"Succeeded\");\n }\n \n [Test]\n public void ThenSecondBatchPipelineOutcomeIsFailed()\n {\n _helperSecondBatch.RunOutcome.Should().Be(\"Failed\");\n }\n\n [Test]\n public void ThenCurrentExecutionTableIsEmpty()\n {\n _helperFirstBatch.RowCount(\"procfwk.CurrentExecution\").Should().Be(0);\n }\n\n [Test]\n public void ThenFourExecutionLogSuccessRecords()\n {\n _helperFirstBatch.RowCount(\"procfwk.ExecutionLog\", where: \"PipelineStatus\", equals: \"Success\").Should().Be(4);\n }\n\n [Test]\n public void ThenOneBatchExecutionSuccessRecord()\n {\n _helperFirstBatch.RowCount(\"procfwk.BatchExecution\", where: \"BatchStatus\", equals: \"Success\").Should().Be(1);\n }\n #endregion\n\n [OneTimeTearDown]\n public void TearDown()\n {\n _helperFirstBatch?.TearDown();\n }\n }\n}\n"
},
{
"path": "FactoryTesting/Pipelines/02-Parent/GivenBatchExecutionsForConcurrentBatchesWithSimpleFailureHandling.cs",
"content": "using FluentAssertions;\nusing NUnit.Framework;\nusing NUnit.Framework.Internal;\nusing System.Threading.Tasks;\n\nnamespace FactoryTesting.Pipelines.Parent\n{\n public class GivenBatchExecutionsForConcurrentBatchesWithSimpleFailureHandling\n {\n private ParentHelper _helperFirstBatch;\n private ParentHelper _helperSecondBatch;\n\n [OneTimeSetUp]\n public async Task WhenPipelineRun()\n {\n _helperFirstBatch = new ParentHelper()\n .WithBasicMetadata()\n .WithTenantAndSubscriptionIds()\n .WithSPNInDatabase(\"FrameworkFactory\")\n .WithEmptyExecutionTables()\n .WithBatchExecutionHandling()\n .WithStagesEnabled()\n .WithPipelinesEnabled()\n .WithParameter(\"BatchName\", \"Hourly\");\n\n _helperSecondBatch = new ParentHelper()\n .WithSimulatedError()\n .WithFailureHandling(\"Simple\")\n .WithParameter(\"BatchName\", \"Daily\");\n\n var firstBatch = _helperFirstBatch.RunPipeline();\n var secondBatch = _helperSecondBatch.RunPipeline();\n\n await Task.WhenAll(firstBatch, secondBatch);\n }\n #region Integration tests\n\n //batch one - hourly:\n [Test]\n public void ThenFirstBatchPipelineOutcomeIsSucceeded()\n {\n _helperFirstBatch.RunOutcome.Should().Be(\"Succeeded\");\n }\n\n [Test]\n public void ThenOneBatchExecutionSuccessRecord()\n {\n _helperFirstBatch.RowCount(\"procfwk.BatchExecution\", where: \"BatchStatus\", equals: \"Success\").Should().Be(1);\n }\n\n [Test]\n public void ThenFourExecutionLogSuccessRecords()\n {\n _helperFirstBatch.RowCount(\"procfwk.ExecutionLog\", where: \"PipelineStatus\", equals: \"Success\").Should().Be(4);\n }\n\n //batch two - daily:\n [Test]\n public void ThenSecondBatchPipelineOutcomeIsFailed()\n {\n _helperSecondBatch.RunOutcome.Should().Be(\"Failed\");\n }\n\n [Test]\n public void ThenOneBatchExecutionStoppedRecord()\n {\n _helperSecondBatch.RowCount(\"procfwk.BatchExecution\", where: \"BatchStatus\", equals: \"Stopped\").Should().Be(1);\n }\n\n [Test]\n public void ThenThreeExecutionsSucceeded()\n {\n _helperSecondBatch.RowCount(\"procfwk.CurrentExecution\", where: \"PipelineStatus\", equals: \"Success\").Should().Be(3);\n }\n\n [Test]\n public void ThenOneExecutionFailed()\n {\n _helperSecondBatch.RowCount(\"procfwk.CurrentExecution\", where: \"PipelineStatus\", equals: \"Failed\").Should().Be(1);\n }\n\n [Test]\n public void ThenSevenExecutionsBlocked()\n {\n _helperSecondBatch.RowCount(\"procfwk.CurrentExecution\", where: \"PipelineStatus\", equals: \"Blocked\").Should().Be(7);\n }\n\n [Test]\n public void ThenOneExecutionLogRecord()\n {\n _helperSecondBatch.RowCount(\"procfwk.ExecutionLog\", where: \"PipelineStatus\", equals: \"Failed\").Should().Be(1);\n }\n\n [Test]\n public void ThenTwoErrorLogRecords()\n {\n _helperSecondBatch.RowCount(\"procfwk.ErrorLog\").Should().Be(2);\n }\n\n #endregion\n\n [OneTimeTearDown]\n public void TearDown()\n {\n _helperFirstBatch?.TearDown();\n }\n }\n}\n"
},
{
"path": "FactoryTesting/Pipelines/02-Parent/GivenBatchExecutionsForConcurrentBatchesWithSimpleFailureHandlingAndRestart.cs",
"content": "using FluentAssertions;\nusing NUnit.Framework;\nusing NUnit.Framework.Internal;\nusing System.Threading.Tasks;\n\nnamespace FactoryTesting.Pipelines.Parent\n{\n public class GivenBatchExecutionsForConcurrentBatchesWithSimpleFailureHandlingAndRestart\n {\n private ParentHelper _helperFirstBatch;\n private ParentHelper _helperSecondBatch;\n private ParentHelper _helperSecondBatchRestart;\n private ParentHelper _helperThirdBatch;\n\n [OneTimeSetUp]\n public async Task WhenPipelineRun()\n {\n //first\n _helperFirstBatch = new ParentHelper()\n .WithBasicMetadata()\n .WithTenantAndSubscriptionIds()\n .WithSPNInDatabase(\"FrameworkFactory\")\n .WithEmptyExecutionTables()\n .WithBatchExecutionHandling()\n .WithStagesEnabled()\n .WithPipelinesEnabled()\n .WithParameter(\"BatchName\", \"Hourly\");\n\n _helperSecondBatch = new ParentHelper()\n .WithSimulatedError()\n .WithFailureHandling(\"Simple\")\n .WithParameter(\"BatchName\", \"Daily\");\n\n var firstBatch = _helperFirstBatch.RunPipeline();\n var secondBatch = _helperSecondBatch.RunPipeline();\n\n await Task.WhenAll(firstBatch, secondBatch);\n\n //then\n _helperSecondBatchRestart = new ParentHelper()\n .WithoutSimulatedError()\n .WithParameter(\"BatchName\", \"Daily\");\n\n _helperThirdBatch = new ParentHelper()\n .WithParameter(\"BatchName\", \"Hourly\");\n\n var secondBatchRestart = _helperSecondBatchRestart.RunPipeline(10000); //10 second delay\n var thirdBatch = _helperThirdBatch.RunPipeline();\n \n await Task.WhenAll(secondBatchRestart, thirdBatch);\n }\n #region Integration tests\n\n //batch one - hourly:\n [Test]\n public void ThenFirstBatchPipelineOutcomeIsSucceeded()\n {\n _helperFirstBatch.RunOutcome.Should().Be(\"Succeeded\");\n }\n\n //batch two - daily:\n [Test]\n public void ThenSecondBatchPipelineOutcomeIsFailed()\n {\n _helperSecondBatch.RunOutcome.Should().Be(\"Failed\");\n }\n\n //batch two restart - daily:\n [Test]\n public void ThenSecondBatchRestartPipelineOutcomeIsSucceeded()\n {\n _helperSecondBatchRestart.RunOutcome.Should().Be(\"Succeeded\");\n }\n\n //batch three - hourly:\n [Test]\n public void ThenThirdBatchPipelineOutcomeIsSucceeded()\n {\n _helperThirdBatch.RunOutcome.Should().Be(\"Succeeded\");\n }\n #endregion\n\n #region Functional Tests\n [Test]\n public void ThenThreeBatchExecutionSuccessRecords()\n {\n _helperThirdBatch.RowCount(\"procfwk.BatchExecution\", where: \"BatchStatus\", equals: \"Success\").Should().Be(3);\n }\n\n [Test]\n public void ThenNineteenExecutionLogSuccessRecords()\n {\n _helperThirdBatch.RowCount(\"procfwk.ExecutionLog\", where: \"PipelineStatus\", equals: \"Success\").Should().Be(19);\n }\n\n [Test]\n public void ThenOneExecutionLogRecord()\n {\n _helperThirdBatch.RowCount(\"procfwk.ExecutionLog\", where: \"PipelineStatus\", equals: \"Failed\").Should().Be(1);\n }\n\n [Test]\n public void ThenTwoErrorLogRecords()\n {\n _helperThirdBatch.RowCount(\"procfwk.ErrorLog\").Should().Be(2);\n }\n\n [Test]\n public void ThenCurrentExecutionTableIsEmpty()\n {\n _helperThirdBatch.RowCount(\"procfwk.CurrentExecution\").Should().Be(0);\n }\n\n #endregion\n\n [OneTimeTearDown]\n public void TearDown()\n {\n _helperThirdBatch?.TearDown();\n }\n }\n}\n"
},
{
"path": "FactoryTesting/Pipelines/02-Parent/GivenBatchExecutionsForConcurrentBatchesWithSimpleFailureHandlingAndRestartOveride.cs",
"content": "using FluentAssertions;\nusing NUnit.Framework;\nusing NUnit.Framework.Internal;\nusing System.Threading.Tasks;\n\nnamespace FactoryTesting.Pipelines.Parent\n{\n public class GivenBatchExecutionsForConcurrentBatchesWithSimpleFailureHandlingAndRestartOveride\n {\n private ParentHelper _helperFirstBatch;\n private ParentHelper _helperSecondBatch;\n private ParentHelper _helperSecondBatchRestart;\n private ParentHelper _helperThirdBatch;\n\n [OneTimeSetUp]\n public async Task WhenPipelineRun()\n {\n //first\n _helperFirstBatch = new ParentHelper()\n .WithBasicMetadata()\n .WithTenantAndSubscriptionIds()\n .WithSPNInDatabase(\"FrameworkFactory\")\n .WithEmptyExecutionTables()\n .WithBatchExecutionHandling()\n .WithStagesEnabled()\n .WithPipelinesEnabled()\n .WithParameter(\"BatchName\", \"Hourly\");\n\n _helperSecondBatch = new ParentHelper()\n .WithSimulatedError()\n .WithFailureHandling(\"Simple\")\n .WithParameter(\"BatchName\", \"Daily\");\n\n var firstBatch = _helperFirstBatch.RunPipeline();\n var secondBatch = _helperSecondBatch.RunPipeline();\n\n await Task.WhenAll(firstBatch, secondBatch);\n\n //then\n _helperSecondBatchRestart = new ParentHelper()\n .WithoutSimulatedError()\n .WithOverideRestart(true)\n .WithParameter(\"BatchName\", \"Daily\");\n\n _helperThirdBatch = new ParentHelper()\n .WithParameter(\"BatchName\", \"Hourly\");\n\n var secondBatchRestart = _helperSecondBatchRestart.RunPipeline(10000); //10 second delay\n var thirdBatch = _helperThirdBatch.RunPipeline();\n\n await Task.WhenAll(secondBatchRestart, thirdBatch);\n }\n #region Integration tests\n\n //batch one - hourly:\n [Test]\n public void ThenFirstBatchPipelineOutcomeIsSucceeded()\n {\n _helperFirstBatch.RunOutcome.Should().Be(\"Succeeded\");\n }\n\n //batch two - daily:\n [Test]\n public void ThenSecondBatchPipelineOutcomeIsFailed()\n {\n _helperSecondBatch.RunOutcome.Should().Be(\"Failed\");\n }\n\n //batch two restart - daily:\n [Test]\n public void ThenSecondBatchRestartPipelineOutcomeIsSucceeded()\n {\n _helperSecondBatchRestart.RunOutcome.Should().Be(\"Succeeded\");\n }\n\n //batch three - hourly:\n [Test]\n public void ThenThirdBatchPipelineOutcomeIsSucceeded()\n {\n _helperThirdBatch.RunOutcome.Should().Be(\"Succeeded\");\n }\n #endregion\n\n #region Functional Tests\n [Test]\n public void ThenThreeBatchExecutionSuccessRecords()\n {\n _helperThirdBatch.RowCount(\"procfwk.BatchExecution\", where: \"BatchStatus\", equals: \"Success\").Should().Be(3);\n }\n\n [Test]\n public void ThenOneBatchExecutionAbandonedRecord()\n {\n _helperThirdBatch.RowCount(\"procfwk.BatchExecution\", where: \"BatchStatus\", equals: \"Abandoned\").Should().Be(1);\n }\n\n [Test]\n public void ThenTwentyTwoExecutionLogSuccessRecords()\n {\n _helperThirdBatch.RowCount(\"procfwk.ExecutionLog\", where: \"PipelineStatus\", equals: \"Success\").Should().Be(22);\n }\n\n [Test]\n public void ThenOneExecutionLogRecord()\n {\n _helperThirdBatch.RowCount(\"procfwk.ExecutionLog\", where: \"PipelineStatus\", equals: \"Failed\").Should().Be(2);\n }\n [Test]\n public void ThenSevenExecutionsBlocked()\n {\n _helperThirdBatch.RowCount(\"procfwk.ExecutionLog\", where: \"PipelineStatus\", equals: \"Blocked\").Should().Be(7);\n }\n [Test]\n public void ThenTwoErrorLogRecords()\n {\n _helperThirdBatch.RowCount(\"procfwk.ErrorLog\").Should().Be(2);\n }\n\n [Test]\n public void ThenCurrentExecutionTableIsEmpty()\n {\n _helperThirdBatch.RowCount(\"procfwk.CurrentExecution\").Should().Be(0);\n }\n\n #endregion\n\n [OneTimeTearDown]\n public void TearDown()\n {\n _helperThirdBatch?.TearDown();\n }\n }\n}\n"
},
{
"path": "FactoryTesting/Pipelines/02-Parent/GivenBatchExecutionsForSingleBatch.cs",
"content": "using FluentAssertions;\nusing NUnit.Framework;\nusing NUnit.Framework.Internal;\nusing System.Threading.Tasks;\n\nnamespace FactoryTesting.Pipelines.Parent\n{\n public class GivenBatchExecutionsForSingleBatch\n {\n private ParentHelper _helper;\n\n [OneTimeSetUp]\n public async Task WhenPipelineRun()\n {\n _helper = new ParentHelper()\n .WithBasicMetadata()\n .WithTenantAndSubscriptionIds()\n .WithSPNInDatabase(\"FrameworkFactory\")\n .WithEmptyExecutionTables()\n .WithBatchExecutionHandling()\n .WithStagesEnabled()\n .WithPipelinesEnabled()\n .WithParameter(\"BatchName\", \"Hourly\");\n await _helper.RunPipeline();\n }\n\n #region Functional tests\n\n [Test]\n public void ThenPipelineOutcomeIsSucceeded()\n {\n _helper.RunOutcome.Should().Be(\"Succeeded\");\n }\n\n [Test]\n public void ThenCurrentExecutionTableIsEmpty()\n {\n _helper.RowCount(\"procfwk.CurrentExecution\").Should().Be(0);\n }\n\n [Test]\n public void ThenFourExecutionLogSuccessRecords()\n {\n _helper.RowCount(\"procfwk.ExecutionLog\", where: \"PipelineStatus\", equals: \"Success\").Should().Be(4);\n }\n\n [Test]\n public void ThenOneBatchExecutionSuccessRecord()\n {\n _helper.RowCount(\"procfwk.BatchExecution\", where: \"BatchStatus\", equals: \"Success\").Should().Be(1);\n }\n #endregion\n\n [OneTimeTearDown]\n public void TearDown()\n {\n _helper?.TearDown();\n }\n }\n}\n"
},
{
"path": "FactoryTesting/Pipelines/02-Parent/GivenCancelledWorkerAndRestart.cs",
"content": "using FluentAssertions;\nusing NUnit.Framework;\nusing NUnit.Framework.Internal;\nusing System.Threading.Tasks;\n\nnamespace FactoryTesting.Pipelines.Parent\n{\n public class GivenCancelledWorkerAndRestart\n {\n private ParentHelper _helperFirstRun;\n private ParentHelper _helperRestartRun;\n\n [OneTimeSetUp]\n public async Task WhenPipelineRun()\n {\n _helperFirstRun = new ParentHelper()\n .WithBasicMetadata()\n .WithTenantAndSubscriptionIds()\n .WithSPNInDatabase(\"FrameworkFactory\")\n .WithEmptyExecutionTables()\n .WithoutPrecursorObject() //done to ensure 2min waits are used, not example precursor waits\n .With2MinWaitsOnWorkers() //to ensure the cancel call has enough time\n .WithCancelledWorkersBlock(true)\n .WithFailureHandling(\"Simple\");\n\n var runOrchestrator = _helperFirstRun.RunPipeline();\n var cancelWorker = _helperFirstRun.CancelAnyWorkerPipeline();\n\n await Task.WhenAll(runOrchestrator, cancelWorker);\n\n _helperRestartRun = new ParentHelper();\n await _helperRestartRun.RunPipeline();\n }\n #region Integration tests\n\n [Test]\n public void ThenPipelineOutcomeIsSucceeded()\n {\n _helperRestartRun.RunOutcome.Should().Be(\"Succeeded\");\n }\n\n [Test]\n public void ThenOneExecutionLogRecord()\n {\n _helperRestartRun.RowCount(\"procfwk.ExecutionLog\", where: \"PipelineStatus\", equals: \"Cancelled\").Should().Be(1);\n }\n [Test]\n public void ThenElevenExecutionsSucceeded()\n {\n _helperRestartRun.RowCount(\"procfwk.ExecutionLog\", where: \"PipelineStatus\", equals: \"Success\").Should().Be(11);\n }\n #endregion\n\n [OneTimeTearDown]\n public void TearDown()\n {\n _helperRestartRun?.TearDown();\n }\n }\n}\n"
},
{
"path": "FactoryTesting/Pipelines/02-Parent/GivenCancelledWorkerInOneExecutionStage.cs",
"content": "using FluentAssertions;\nusing NUnit.Framework;\nusing NUnit.Framework.Internal;\nusing System.Threading.Tasks;\n\nnamespace FactoryTesting.Pipelines.Parent\n{\n public class GivenCancelledWorkerInOneExecutionStage\n {\n private ParentHelper _helper;\n\n [OneTimeSetUp]\n public async Task WhenPipelineRun()\n {\n _helper = new ParentHelper()\n .WithBasicMetadata()\n .WithTenantAndSubscriptionIds()\n .WithSPNInDatabase(\"FrameworkFactory\")\n .WithEmptyExecutionTables()\n .WithoutPrecursorObject() //done to ensure 2min waits are used, not example precursor waits\n .With2MinWaitsOnWorkers() //to ensure the cancel call has enough time\n .WithCancelledWorkersBlock(false)\n .WithOnlyStageOneEnabled();\n \n var runOrchestrator = _helper.RunPipeline();\n var cancelWorker = _helper.CancelAnyWorkerPipeline(); //any worker\n\n await Task.WhenAll(runOrchestrator, cancelWorker);\n }\n #region Integration tests\n\n [Test]\n public void ThenPipelineOutcomeIsFailed()\n {\n _helper.RunOutcome.Should().Be(\"Failed\");\n }\n\n [Test]\n public void ThenOneExecutionsCancelled()\n {\n _helper.RowCount(\"procfwk.CurrentExecution\", where: \"PipelineStatus\", equals: \"Cancelled\").Should().Be(1);\n }\n \n [Test]\n public void ThenOneExecutionLogRecord()\n {\n _helper.RowCount(\"procfwk.ExecutionLog\", where: \"PipelineStatus\", equals: \"Cancelled\").Should().Be(1);\n }\n [Test]\n public void ThenThreeExecutionsSucceeded()\n {\n _helper.RowCount(\"procfwk.CurrentExecution\", where: \"PipelineStatus\", equals: \"Success\").Should().Be(3);\n }\n #endregion\n\n [OneTimeTearDown]\n public void TearDown()\n { \n _helper?.TearDown();\n }\n }\n}\n"
},
{
"path": "FactoryTesting/Pipelines/02-Parent/GivenCancelledWorkerThatBlocks.cs",
"content": "using FluentAssertions;\nusing NUnit.Framework;\nusing NUnit.Framework.Internal;\nusing System.Threading.Tasks;\n\nnamespace FactoryTesting.Pipelines.Parent\n{\n public class GivenCancelledWorkerThatBlocks\n {\n private ParentHelper _helper;\n\n [OneTimeSetUp]\n public async Task WhenPipelineRun()\n {\n _helper = new ParentHelper()\n .WithBasicMetadata()\n .WithTenantAndSubscriptionIds()\n .WithSPNInDatabase(\"FrameworkFactory\")\n .WithEmptyExecutionTables()\n .WithoutPrecursorObject() //done to ensure 2min waits are used, not example precursor waits\n .With2MinWaitsOnWorkers() //to ensure the cancel call has enough time\n .WithCancelledWorkersBlock(true)\n .WithFailureHandling(\"Simple\");\n\n var runOrchestrator = _helper.RunPipeline();\n var cancelWorker = _helper.CancelIntentionalErrorWorkerPipeline(); //specific worker\n\n await Task.WhenAll(runOrchestrator, cancelWorker);\n }\n #region Integration tests\n\n [Test]\n public void ThenPipelineOutcomeIsFailed()\n {\n _helper.RunOutcome.Should().Be(\"Failed\");\n }\n\n [Test]\n public void ThenOneExecutionsCancelled()\n {\n _helper.RowCount(\"procfwk.CurrentExecution\", where: \"PipelineStatus\", equals: \"Cancelled\").Should().Be(1);\n }\n\n [Test]\n public void ThenOneExecutionLogRecord()\n {\n _helper.RowCount(\"procfwk.ExecutionLog\", where: \"PipelineStatus\", equals: \"Cancelled\").Should().Be(1);\n }\n [Test]\n public void ThenThreeExecutionsSucceeded()\n {\n _helper.RowCount(\"procfwk.CurrentExecution\", where: \"PipelineStatus\", equals: \"Success\").Should().Be(3);\n }\n [Test]\n public void ThenSevenExecutionsBlocked()\n {\n _helper.RowCount(\"procfwk.CurrentExecution\", where: \"PipelineStatus\", equals: \"Blocked\").Should().Be(7);\n }\n #endregion\n\n [OneTimeTearDown]\n public void TearDown()\n {\n _helper?.TearDown();\n }\n }\n}\n"
},
{
"path": "FactoryTesting/Pipelines/02-Parent/GivenCancelledWorkerThatDoesntBlock.cs",
"content": "using FluentAssertions;\nusing NUnit.Framework;\nusing NUnit.Framework.Internal;\nusing System.Threading.Tasks;\n\nnamespace FactoryTesting.Pipelines.Parent\n{\n public class GivenCancelledWorkerThatDoesntBlock\n {\n private ParentHelper _helper;\n\n [OneTimeSetUp]\n public async Task WhenPipelineRun()\n {\n _helper = new ParentHelper()\n .WithBasicMetadata()\n .WithTenantAndSubscriptionIds()\n .WithSPNInDatabase(\"FrameworkFactory\")\n .WithEmptyExecutionTables()\n .WithoutPrecursorObject() //done to ensure 2min waits are used, not example precursor waits\n .With2MinWaitsOnWorkers() //to ensure the cancel call has enough time\n .WithCancelledWorkersBlock(false)\n .WithFailureHandling(\"Simple\");\n\n var runOrchestrator = _helper.RunPipeline();\n var cancelWorker = _helper.CancelIntentionalErrorWorkerPipeline(); //specific worker\n\n await Task.WhenAll(runOrchestrator, cancelWorker);\n }\n #region Integration tests\n\n [Test]\n public void ThenPipelineOutcomeIsFailed()\n {\n _helper.RunOutcome.Should().Be(\"Failed\");\n }\n\n [Test]\n public void ThenOneExecutionsCancelled()\n {\n _helper.RowCount(\"procfwk.CurrentExecution\", where: \"PipelineStatus\", equals: \"Cancelled\").Should().Be(1);\n }\n\n [Test]\n public void ThenOneExecutionLogRecord()\n {\n _helper.RowCount(\"procfwk.ExecutionLog\", where: \"PipelineStatus\", equals: \"Cancelled\").Should().Be(1);\n }\n [Test]\n public void ThenTenExecutionsSucceeded()\n {\n _helper.RowCount(\"procfwk.CurrentExecution\", where: \"PipelineStatus\", equals: \"Success\").Should().Be(10);\n }\n #endregion\n\n [OneTimeTearDown]\n public void TearDown()\n {\n _helper?.TearDown();\n }\n }\n}\n"
},
{
"path": "FactoryTesting/Pipelines/02-Parent/GivenCleanUpForCancelledWorkerAndRestart.cs",
"content": "using FluentAssertions;\nusing NUnit.Framework;\nusing NUnit.Framework.Internal;\nusing System.Threading.Tasks;\n\nnamespace FactoryTesting.Pipelines.Parent\n{\n public class GivenCleanUpForCancelledWorkerAndRestart\n {\n private ParentHelper _helperFirstRun;\n private ParentHelper _helperRestartRun;\n\n [OneTimeSetUp]\n public async Task WhenPipelineRun()\n {\n _helperFirstRun = new ParentHelper()\n .WithBasicMetadata()\n .WithTenantAndSubscriptionIds()\n .WithSPNInDatabase(\"FrameworkFactory\")\n .WithEmptyExecutionTables()\n .WithoutPrecursorObject() //done to ensure 2min waits are used, not example precursor waits\n .With2MinWaitsOnWorkers() //to ensure the cancel call has enough time\n .WithCancelledWorkersBlock(true)\n .WithFailureHandling(\"Simple\");\n\n var runOrchestrator = _helperFirstRun.RunPipeline();\n var cancelWorker = _helperFirstRun.CancelAnyWorkerPipeline();\n\n await Task.WhenAll(runOrchestrator, cancelWorker);\n\n _helperRestartRun = new ParentHelper()\n .WithPrecursorObject()\n .WithRunningPipelineStatusInPlaceOf(\"Cancelled\");\n \n await _helperRestartRun.RunPipeline();\n }\n #region Integration tests\n\n [Test]\n public void ThenPipelineOutcomeIsSucceeded()\n {\n _helperRestartRun.RunOutcome.Should().Be(\"Succeeded\");\n }\n\n [Test]\n public async Task ThenActivityShouldReturnOneRowForCleanUp()\n {\n var count = await _helperRestartRun.GetActivityOutput(\"Check Previous Execution\", \"$.count\");\n int.Parse(count).Should().Be(1);\n }\n\n [Test]\n public void ThenOneExecutionLogRecord()\n {\n _helperRestartRun.RowCount(\"procfwk.ExecutionLog\", where: \"PipelineStatus\", equals: \"Cancelled\").Should().Be(1);\n }\n [Test]\n public void ThenElevenExecutionsSucceeded()\n {\n _helperRestartRun.RowCount(\"procfwk.ExecutionLog\", where: \"PipelineStatus\", equals: \"Success\").Should().Be(11);\n }\n #endregion\n\n [OneTimeTearDown]\n public void TearDown()\n {\n _helperRestartRun?.TearDown();\n }\n }\n}\n"
},
{
"path": "FactoryTesting/Pipelines/02-Parent/GivenCleanUpForSuccessfulWorkersAndRestart.cs",
"content": "using FluentAssertions;\nusing NUnit.Framework;\nusing NUnit.Framework.Internal;\nusing System.Threading.Tasks;\n\nnamespace FactoryTesting.Pipelines.Parent\n{\n public class GivenCleanUpForSuccessfulWorkersAndRestart\n {\n private ParentHelper _helperFirstRun;\n private ParentHelper _helperRestartRun;\n\n [OneTimeSetUp]\n public async Task WhenPipelineRun()\n {\n _helperFirstRun = new ParentHelper()\n .WithBasicMetadata()\n .WithTenantAndSubscriptionIds()\n .WithSPNInDatabase(\"FrameworkFactory\")\n .WithEmptyExecutionTables()\n .WithoutPrecursorObject() //done to ensure 2min waits are used, not example precursor waits\n .With2MinWaitsOnWorkers() //to ensure the cancel call has enough time\n .WithCancelledWorkersBlock(true)\n .WithFailureHandling(\"Simple\");\n\n var runOrchestrator = _helperFirstRun.RunPipeline();\n var cancelWorker = _helperFirstRun.CancelAnyWorkerPipeline();\n\n await Task.WhenAll(runOrchestrator, cancelWorker);\n\n _helperRestartRun = new ParentHelper()\n .WithPrecursorObject()\n .WithRunningPipelineStatusInPlaceOf(\"Success\");\n\n await _helperRestartRun.RunPipeline();\n }\n\n [Test]\n public void ThenPipelineOutcomeIsSucceeded()\n {\n _helperRestartRun.RunOutcome.Should().Be(\"Succeeded\");\n }\n\n [Test]\n public async Task ThenActivityShouldReturnThreeRowsForCleanUp()\n {\n var count = await _helperRestartRun.GetActivityOutput(\"Check Previous Execution\", \"$.count\");\n int.Parse(count).Should().Be(3);\n }\n\n [Test]\n public void ThenOneExecutionLogRecord()\n {\n _helperRestartRun.RowCount(\"procfwk.ExecutionLog\", where: \"PipelineStatus\", equals: \"Cancelled\").Should().Be(1);\n }\n [Test]\n public void ThenElevenExecutionsSucceeded()\n {\n _helperRestartRun.RowCount(\"procfwk.ExecutionLog\", where: \"PipelineStatus\", equals: \"Success\").Should().Be(11);\n }\n\n [OneTimeTearDown]\n public void TearDown()\n {\n _helperRestartRun?.TearDown();\n }\n }\n}\n"
},
{
"path": "FactoryTesting/Pipelines/02-Parent/GivenDependencyChainFailureHandling.cs",
"content": "using FluentAssertions;\nusing NUnit.Framework;\nusing NUnit.Framework.Internal;\nusing System.Threading.Tasks;\n\nnamespace FactoryTesting.Pipelines.Parent\n{\n public class GivenDependencyChainFailureHandling\n {\n private ParentHelper _helper;\n\n [OneTimeSetUp]\n public async Task WhenPipelineRun()\n {\n _helper = new ParentHelper()\n .WithBasicMetadata()\n .WithTenantAndSubscriptionIds()\n .WithSPNInDatabase(\"FrameworkFactory\")\n .WithEmptyExecutionTables()\n .WithSimulatedError()\n .WithFailureHandling(\"DependencyChain\");\n await _helper.RunPipeline();\n }\n\n #region Functional tests\n\n [Test, Order(1)]\n public void ThenPipelineOutcomeIsFailed()\n {\n _helper.RunOutcome.Should().Be(\"Failed\");\n }\n \n [Test, Order(2)]\n public void ThenSixExecutionsSucceeded()\n {\n _helper.RowCount(\"procfwk.CurrentExecution\", where: \"PipelineStatus\", equals: \"Success\").Should().Be(6);\n }\n\n [Test, Order(3)]\n public void ThenOneExecutionFailed()\n {\n _helper.RowCount(\"procfwk.CurrentExecution\", where: \"PipelineStatus\", equals: \"Failed\").Should().Be(1);\n }\n\n [Test, Order(4)]\n public void ThenFourExecutionsBlocked()\n {\n _helper.RowCount(\"procfwk.CurrentExecution\", where: \"PipelineStatus\", equals: \"Blocked\").Should().Be(4);\n }\n\n [Test, Order(5)]\n public void ThenOneExecutionLogRecord()\n {\n _helper.RowCount(\"procfwk.ExecutionLog\", where: \"PipelineStatus\", equals: \"Failed\").Should().Be(1);\n }\n\n [Test, Order(6)]\n public void ThenTwoErrorLogRecords()\n {\n _helper.RowCount(\"procfwk.ErrorLog\").Should().Be(2);\n }\n #endregion\n\n [OneTimeTearDown]\n public void TearDown()\n {\n _helper?.TearDown();\n }\n }\n}\n"
},
{
"path": "FactoryTesting/Pipelines/02-Parent/GivenDependencyChainFailureHandlingAndRestart.cs",
"content": "using FluentAssertions;\nusing NUnit.Framework;\nusing NUnit.Framework.Internal;\nusing System.Threading.Tasks;\n\nnamespace FactoryTesting.Pipelines.Parent\n{\n public class GivenDependencyChainFailureHandlingAndRestart\n {\n private ParentHelper _helperFirstRun;\n private ParentHelper _helperRestartRun;\n\n [OneTimeSetUp]\n public async Task WhenPipelineRun()\n {\n _helperFirstRun = new ParentHelper()\n .WithBasicMetadata()\n .WithTenantAndSubscriptionIds()\n .WithSPNInDatabase(\"FrameworkFactory\")\n .WithEmptyExecutionTables()\n .WithSimulatedError()\n .WithFailureHandling(\"DependencyChain\");\n await _helperFirstRun.RunPipeline();\n\n _helperRestartRun = new ParentHelper()\n .WithoutSimulatedError();\n await _helperRestartRun.RunPipeline();\n }\n\n #region Functional tests\n\n [Test]\n public void ThenPipelineOutcomeIsSucceeded()\n {\n _helperRestartRun.RunOutcome.Should().Be(\"Succeeded\");\n }\n\n [Test]\n public void ThenOneExecutionLogFailedRecord()\n {\n _helperRestartRun.RowCount(\"procfwk.ExecutionLog\", where: \"PipelineStatus\", equals: \"Failed\").Should().Be(1);\n }\n [Test]\n public void ThenElevenExecutionLogSuccessRecord()\n {\n _helperRestartRun.RowCount(\"procfwk.ExecutionLog\", where: \"PipelineStatus\", equals: \"Success\").Should().Be(11);\n }\n\n [Test]\n public void ThenTwoErrorLogRecords()\n {\n _helperRestartRun.RowCount(\"procfwk.ErrorLog\").Should().Be(2);\n }\n #endregion\n\n [OneTimeTearDown]\n public void TearDown()\n {\n _helperFirstRun?.TearDown();\n }\n }\n}\n"
},
{
"path": "FactoryTesting/Pipelines/02-Parent/GivenDisabledBatches.cs",
"content": "using FluentAssertions;\nusing NUnit.Framework;\nusing NUnit.Framework.Internal;\nusing System.Threading.Tasks;\n\nnamespace FactoryTesting.Pipelines.Parent\n{\n public class GivenDisabledBatches\n {\n private ParentHelper _helper;\n\n [OneTimeSetUp]\n public async Task WhenPipelineRun()\n {\n _helper = new ParentHelper()\n .WithBasicMetadata()\n .WithTenantAndSubscriptionIds()\n .WithSPNInDatabase(\"FrameworkFactory\")\n .WithEmptyExecutionTables()\n .WithBatchExecutionHandling()\n .WithBatchesDisabled();\n await _helper.RunPipeline();\n }\n\n #region Integration tests\n\n [Test]\n public void ThenPipelineOutcomeIsFailed()\n {\n _helper.RunOutcome.Should().Be(\"Failed\");\n }\n\n #endregion\n\n [OneTimeTearDown]\n public void TearDown()\n {\n _helper?.TearDown();\n }\n }\n}\n"
},
{
"path": "FactoryTesting/Pipelines/02-Parent/GivenDisabledPipelines.cs",
"content": "using FluentAssertions;\nusing NUnit.Framework;\nusing NUnit.Framework.Internal;\nusing System.Threading.Tasks;\n\nnamespace FactoryTesting.Pipelines.Parent\n{\n class GivenDisabledPipelines\n {\n private ParentHelper _helper;\n\n [OneTimeSetUp]\n public async Task WhenPipelineRun()\n {\n _helper = new ParentHelper()\n .WithBasicMetadata()\n .WithTenantAndSubscriptionIds()\n .WithSPNInDatabase(\"FrameworkFactory\")\n .WithEmptyExecutionTables()\n .WithPipelinesDisabled();\n await _helper.RunPipeline();\n }\n\n #region Integration tests\n\n [Test, Order(1)]\n public void ThenPipelineOutcomeIsFailed()\n {\n _helper.RunOutcome.Should().Be(\"Failed\");\n }\n\n #endregion\n\n [OneTimeTearDown]\n public void TearDown()\n {\n _helper?.TearDown();\n }\n }\n}\n"
},
{
"path": "FactoryTesting/Pipelines/02-Parent/GivenDisabledStages.cs",
"content": "using FluentAssertions;\nusing NUnit.Framework;\nusing NUnit.Framework.Internal;\nusing System.Threading.Tasks;\n\nnamespace FactoryTesting.Pipelines.Parent\n{\n public class GivenDisabledStages\n {\n private ParentHelper _helper;\n\n [OneTimeSetUp]\n public async Task WhenPipelineRun()\n {\n _helper = new ParentHelper()\n .WithBasicMetadata()\n .WithTenantAndSubscriptionIds()\n .WithSPNInDatabase(\"FrameworkFactory\")\n .WithEmptyExecutionTables()\n .WithStagesDisabled();\n await _helper.RunPipeline();\n }\n\n #region Integration tests\n\n [Test, Order(1)]\n public void ThenPipelineOutcomeIsFailed()\n {\n _helper.RunOutcome.Should().Be(\"Failed\");\n }\n\n #endregion\n\n [OneTimeTearDown]\n public void TearDown()\n {\n _helper?.TearDown();\n }\n }\n}\n"
},
{
"path": "FactoryTesting/Pipelines/02-Parent/GivenNoErrorsAndSPNStoredInDatabase.cs",
"content": "using FluentAssertions;\nusing NUnit.Framework;\nusing NUnit.Framework.Internal;\nusing System.Threading.Tasks;\n\nnamespace FactoryTesting.Pipelines.Parent\n{\n public class GivenNoErrorsAndSPNStoredInDatabase\n {\n private ParentHelper _helper;\n\n [OneTimeSetUp]\n public async Task WhenPipelineRun()\n {\n _helper = new ParentHelper()\n .WithBasicMetadata()\n .WithTenantAndSubscriptionIds()\n .WithSPNInDatabase(\"FrameworkFactory\")\n .WithEmptyExecutionTables()\n .WithoutSimulatedError()\n .WithFailureHandling(\"Simple\");\n await _helper.RunPipeline();\n }\n\n #region Integration tests\n\n [Test]\n public void ThenPipelineOutcomeIsSucceeded()\n {\n _helper.RunOutcome.Should().Be(\"Succeeded\");\n }\n\n #endregion\n\n #region Functional tests\n\n [Test]\n public void ThenCurrentExecutionTableIsEmpty()\n {\n _helper.RowCount(\"procfwk.CurrentExecution\").Should().Be(0);\n }\n\n [Test]\n public void ThenElevenExecutionLogRecords()\n {\n _helper.RowCount(\"procfwk.ExecutionLog\").Should().Be(11);\n }\n\n #endregion\n\n [OneTimeTearDown]\n public void TearDown()\n {\n _helper?.TearDown();\n }\n }\n}\n"
},
{
"path": "FactoryTesting/Pipelines/02-Parent/GivenNoErrorsAndSPNStoredInKeyVault.cs",
"content": "using FluentAssertions;\nusing NUnit.Framework;\nusing NUnit.Framework.Internal;\nusing System.Threading.Tasks;\n\nnamespace FactoryTesting.Pipelines.Parent\n{\n public class GivenNoErrorsAndSPNStoredInKeyVault\n {\n private ParentHelper _helper;\n\n [OneTimeSetUp]\n public async Task WhenPipelineRun()\n {\n _helper = new ParentHelper()\n .WithBasicMetadata()\n .WithTenantAndSubscriptionIds()\n .WithSPNInKeyVault(\"FrameworkFactory\")\n .WithEmptyExecutionTables()\n .WithoutSimulatedError()\n .WithFailureHandling(\"Simple\");\n await _helper.RunPipeline();\n }\n\n #region Integration tests\n\n [Test, Order(1)]\n public void ThenPipelineOutcomeIsSucceeded()\n {\n _helper.RunOutcome.Should().Be(\"Succeeded\");\n }\n\n #endregion\n\n #region Functional tests\n\n [Test, Order(2)]\n public void ThenCurrentExecutionTableIsEmpty()\n {\n _helper.RowCount(\"procfwk.CurrentExecution\").Should().Be(0);\n }\n\n [Test, Order(3)]\n public void ThenElevenExecutionLogRecords()\n {\n _helper.RowCount(\"procfwk.ExecutionLog\").Should().Be(11);\n }\n\n #endregion\n\n [OneTimeTearDown]\n public void TearDown()\n {\n _helper?.TearDown();\n }\n }\n}\n"
},
{
"path": "FactoryTesting/Pipelines/02-Parent/GivenNoFailureHandling.cs",
"content": "using FluentAssertions;\nusing NUnit.Framework;\nusing NUnit.Framework.Internal;\nusing System.Threading.Tasks;\n\nnamespace FactoryTesting.Pipelines.Parent\n{\n public class GivenNoFailureHandling\n {\n private ParentHelper _helper;\n\n [OneTimeSetUp]\n public async Task WhenPipelineRun()\n {\n _helper = new ParentHelper()\n .WithBasicMetadata()\n .WithTenantAndSubscriptionIds()\n .WithSPNInDatabase(\"FrameworkFactory\")\n .WithEmptyExecutionTables()\n .WithSimulatedError()\n .WithFailureHandling(\"None\");\n await _helper.RunPipeline();\n }\n\n #region Functional tests\n\n [Test, Order(1)]\n public void ThenPipelineOutcomeIsFailed()\n {\n _helper.RunOutcome.Should().Be(\"Failed\");\n }\n \n [Test, Order(2)]\n public void ThenTenExecutionsSucceeded()\n {\n _helper.RowCount(\"procfwk.CurrentExecution\", where: \"PipelineStatus\", equals: \"Success\").Should().Be(10);\n }\n \n [Test, Order(3)]\n public void ThenOneExecutionFailed()\n {\n _helper.RowCount(\"procfwk.CurrentExecution\", where: \"PipelineStatus\", equals: \"Failed\").Should().Be(1);\n }\n\n [Test, Order(4)]\n public void ThenOneExecutionLogRecord()\n {\n _helper.RowCount(\"procfwk.ExecutionLog\", where: \"PipelineStatus\", equals: \"Failed\").Should().Be(1);\n }\n\n [Test, Order(5)]\n public void ThenTwoErrorLogRecords()\n {\n _helper.RowCount(\"procfwk.ErrorLog\").Should().Be(2);\n }\n\n #endregion\n\n [OneTimeTearDown]\n public void TearDown()\n {\n _helper?.TearDown();\n }\n }\n}\n"
},
{
"path": "FactoryTesting/Pipelines/02-Parent/GivenNoPipelineParameters.cs",
"content": "using FluentAssertions;\nusing NUnit.Framework;\nusing NUnit.Framework.Internal;\nusing System.Threading.Tasks;\n\nnamespace FactoryTesting.Pipelines.Parent\n{\n class GivenNoPipelineParameters\n {\n private ParentHelper _helper;\n\n [OneTimeSetUp]\n public async Task WhenPipelineRun()\n {\n _helper = new ParentHelper()\n .WithBasicMetadata()\n .WithTenantAndSubscriptionIds()\n .WithSPNInDatabase(\"FrameworkFactory\")\n .WithEmptyExecutionTables()\n .WithoutSimulatedError()\n .WithFailureHandling(\"Simple\"); ;\n await _helper.RunPipeline();\n }\n\n #region Integration tests\n\n [Test]\n public void ThenPipelineOutcomeIsSucceeded()\n {\n _helper.RunOutcome.Should().Be(\"Succeeded\");\n }\n\n #endregion\n\n #region Functional tests\n\n #endregion\n\n [OneTimeTearDown]\n public void TearDown()\n {\n _helper?.TearDown();\n }\n }\n}\n"
},
{
"path": "FactoryTesting/Pipelines/02-Parent/GivenOneExecutionStage.cs",
"content": "using FluentAssertions;\nusing NUnit.Framework;\nusing NUnit.Framework.Internal;\nusing System.Threading.Tasks;\n\nnamespace FactoryTesting.Pipelines.Parent\n{\n class GivenOneExecutionStage\n {\n private ParentHelper _helper;\n\n [OneTimeSetUp]\n public async Task WhenPipelineRun()\n {\n _helper = new ParentHelper()\n .WithBasicMetadata()\n .WithTenantAndSubscriptionIds()\n .WithSPNInDatabase(\"FrameworkFactory\")\n .WithEmptyExecutionTables()\n .WithoutSimulatedError()\n .WithFailureHandling(\"Simple\")\n .WithSingleExecutionStage();\n await _helper.RunPipeline();\n }\n\n #region Integration tests\n\n [Test]\n public void ThenPipelineOutcomeIsSucceeded()\n {\n _helper.RunOutcome.Should().Be(\"Succeeded\");\n }\n\n #endregion\n\n [OneTimeTearDown]\n public void TearDown()\n {\n _helper?.TearDown();\n }\n }\n}\n"
},
{
"path": "FactoryTesting/Pipelines/02-Parent/GivenSimpleFailureHandling.cs",
"content": "using FluentAssertions;\nusing NUnit.Framework;\nusing NUnit.Framework.Internal;\nusing System.Threading.Tasks;\n\nnamespace FactoryTesting.Pipelines.Parent\n{\n public class GivenSimpleFailureHandling\n {\n private ParentHelper _helper;\n\n [OneTimeSetUp]\n public async Task WhenPipelineRun()\n {\n _helper = new ParentHelper()\n .WithBasicMetadata()\n .WithTenantAndSubscriptionIds()\n .WithSPNInDatabase(\"FrameworkFactory\")\n .WithEmptyExecutionTables()\n .WithSimulatedError()\n .WithFailureHandling(\"Simple\");\n await _helper.RunPipeline();\n }\n\n #region Functional tests\n\n [Test, Order(1)]\n public void ThenPipelineOutcomeIsFailed()\n {\n _helper.RunOutcome.Should().Be(\"Failed\");\n }\n \n [Test, Order(2)]\n public void ThenThreeExecutionsSucceeded()\n {\n _helper.RowCount(\"procfwk.CurrentExecution\", where: \"PipelineStatus\", equals: \"Success\").Should().Be(3);\n }\n \n [Test, Order(3)]\n public void ThenOneExecutionFailed()\n {\n _helper.RowCount(\"procfwk.CurrentExecution\", where: \"PipelineStatus\", equals: \"Failed\").Should().Be(1);\n }\n\n [Test, Order(4)]\n public void ThenSevenExecutionsBlocked()\n {\n _helper.RowCount(\"procfwk.CurrentExecution\", where: \"PipelineStatus\", equals: \"Blocked\").Should().Be(7);\n }\n\n [Test, Order(5)]\n public void ThenOneExecutionLogRecord()\n {\n _helper.RowCount(\"procfwk.ExecutionLog\", where: \"PipelineStatus\", equals: \"Failed\").Should().Be(1);\n }\n \n [Test, Order(6)]\n public void ThenTwoErrorLogRecords()\n {\n _helper.RowCount(\"procfwk.ErrorLog\").Should().Be(2);\n }\n #endregion\n\n [OneTimeTearDown]\n public void TearDown()\n {\n _helper?.TearDown();\n }\n }\n}\n"
},
{
"path": "FactoryTesting/Pipelines/02-Parent/GivenSimpleFailureHandlingAndRestart.cs",
"content": "using FluentAssertions;\nusing NUnit.Framework;\nusing NUnit.Framework.Internal;\nusing System.Threading.Tasks;\n\nnamespace FactoryTesting.Pipelines.Parent\n{\n public class GivenSimpleFailureHandlingAndRestart\n {\n private ParentHelper _helperFirstRun;\n private ParentHelper _helperRestartRun;\n\n [OneTimeSetUp]\n public async Task WhenPipelineRun()\n {\n _helperFirstRun = new ParentHelper()\n .WithBasicMetadata()\n .WithTenantAndSubscriptionIds()\n .WithSPNInDatabase(\"FrameworkFactory\")\n .WithEmptyExecutionTables()\n .WithSimulatedError()\n .WithFailureHandling(\"Simple\");\n await _helperFirstRun.RunPipeline();\n\n _helperRestartRun = new ParentHelper()\n .WithoutSimulatedError();\n await _helperRestartRun.RunPipeline();\n }\n\n #region Functional tests\n\n [Test]\n public void ThenPipelineOutcomeIsSucceeded()\n {\n _helperRestartRun.RunOutcome.Should().Be(\"Succeeded\");\n }\n\n [Test]\n public void ThenOneExecutionLogFailedRecord()\n {\n _helperRestartRun.RowCount(\"procfwk.ExecutionLog\", where: \"PipelineStatus\", equals: \"Failed\").Should().Be(1);\n }\n [Test]\n public void ThenElevenExecutionLogSuccessRecord()\n {\n _helperRestartRun.RowCount(\"procfwk.ExecutionLog\", where: \"PipelineStatus\", equals: \"Success\").Should().Be(11);\n }\n\n [Test]\n public void ThenTwoErrorLogRecords()\n {\n _helperRestartRun.RowCount(\"procfwk.ErrorLog\").Should().Be(2);\n }\n #endregion\n\n [OneTimeTearDown]\n public void TearDown()\n {\n _helperRestartRun?.TearDown();\n }\n }\n}\n"
},
{
"path": "FactoryTesting/Pipelines/02-Parent/GivenSimpleFailureHandlingAndRestartOveride.cs",
"content": "using FluentAssertions;\nusing NUnit.Framework;\nusing NUnit.Framework.Internal;\nusing System.Threading.Tasks;\n\nnamespace FactoryTesting.Pipelines.Parent\n{\n public class GivenSimpleFailureHandlingAndRestartOveride\n {\n private ParentHelper _helperFirstRun;\n private ParentHelper _helperRestartRun;\n\n [OneTimeSetUp]\n public async Task WhenPipelineRun()\n {\n _helperFirstRun = new ParentHelper()\n .WithBasicMetadata()\n .WithTenantAndSubscriptionIds()\n .WithSPNInDatabase(\"FrameworkFactory\")\n .WithEmptyExecutionTables()\n .WithSimulatedError()\n .WithFailureHandling(\"Simple\");\n await _helperFirstRun.RunPipeline();\n\n _helperRestartRun = new ParentHelper()\n .WithoutSimulatedError()\n .WithOverideRestart(true);\n await _helperRestartRun.RunPipeline();\n }\n #region Functional tests\n\n [Test]\n public void ThenPipelineOutcomeIsSucceeded()\n {\n _helperRestartRun.RunOutcome.Should().Be(\"Succeeded\");\n }\n\n [Test]\n public void ThenOneExecutionLogFailedRecord()\n {\n _helperRestartRun.RowCount(\"procfwk.ExecutionLog\", where: \"PipelineStatus\", equals: \"Failed\").Should().Be(2);\n }\n [Test]\n public void ThenFourthteenExecutionLogSuccessRecord()\n {\n _helperRestartRun.RowCount(\"procfwk.ExecutionLog\", where: \"PipelineStatus\", equals: \"Success\").Should().Be(14);\n }\n [Test]\n public void ThenSevenExecutionsBlocked()\n {\n _helperRestartRun.RowCount(\"procfwk.ExecutionLog\", where: \"PipelineStatus\", equals: \"Blocked\").Should().Be(7);\n }\n [Test]\n public void ThenTwoErrorLogRecords()\n {\n _helperRestartRun.RowCount(\"procfwk.ErrorLog\").Should().Be(2);\n }\n #endregion\n\n [OneTimeTearDown]\n public void TearDown()\n {\n _helperRestartRun?.TearDown();\n }\n }\n}\n"
},
{
"path": "FactoryTesting/Pipelines/02-Parent/ParentHelper.cs",
"content": "using FactoryTesting.Helpers;\nusing System;\nusing System.Data.SqlClient;\nusing System.Threading;\nusing System.Threading.Tasks;\nusing System.Xml;\n\nnamespace FactoryTesting.Pipelines.Parent\n{\n class ParentHelper : CoverageHelper\n {\n public async Task RunPipeline()\n {\n await RunPipeline(\"02-Parent\");\n }\n public async Task RunPipeline(int fakeDelayMilliseconds)\n {\n Thread.Sleep(fakeDelayMilliseconds);\n await RunPipeline(\"02-Parent\");\n }\n\n public async Task CancelAnyWorkerPipeline()\n {\n await CancelRunningPipeline(GetWorkerRunId(), GetSetting(\"WorkersDataFactoryName\"));\n }\n public async Task CancelIntentionalErrorWorkerPipeline()\n {\n await CancelRunningPipeline(GetWorkerRunId(\"Intentional Error\"), GetSetting(\"WorkersDataFactoryName\"));\n }\n\n public virtual Task RunAsync()\n {\n return Task.CompletedTask;\n }\n public ParentHelper WithTenantAndSubscriptionIds()\n {\n AddTenantAndSubscription();\n return this;\n }\n public ParentHelper WithSPNInDatabase(string workerFactoryName)\n {\n AddWorkerSPNStoredInDatabase(workerFactoryName);\n return this;\n }\n\n public ParentHelper WithSPNInKeyVault(string workerFactoryName)\n {\n AddWorkerSPNStoredInKeyVault(workerFactoryName);\n return this;\n }\n\n public ParentHelper WithBasicMetadata()\n {\n AddBasicMetadata();\n return this;\n }\n\n public ParentHelper WithEmptyExecutionTables()\n {\n WithEmptyTable(\"procfwk.CurrentExecution\");\n WithEmptyTable(\"procfwk.ExecutionLog\");\n WithEmptyTable(\"procfwk.ErrorLog\");\n\n return this;\n }\n\n public ParentHelper WithRunningPipelineStatusInPlaceOf(string statusToOveride)\n {\n SetFalsePipelineStatus(\"Running\", \"PipelineStatus\", statusToOveride);\n return this;\n }\n\n public ParentHelper WithSimulatedError()\n {\n SimulateError(true);\n return this;\n }\n\n public ParentHelper WithoutSimulatedError()\n {\n SimulateError(false);\n return this;\n }\n\n public ParentHelper WithBatchesDisabled()\n {\n EnableDisableMetadata(\"Batches\", false);\n return this;\n }\n\n public ParentHelper WithStagesDisabled()\n {\n EnableDisableMetadata(\"Stages\", false);\n return this;\n }\n\n public ParentHelper WithStagesEnabled()\n {\n EnableDisableMetadata(\"Stages\", true);\n return this;\n }\n\n public ParentHelper WithOnlyStageOneEnabled()\n {\n EnableDisableMetadata(\"Stages\", false, \"StageId\", \"2\");\n EnableDisableMetadata(\"Stages\", false, \"StageId\", \"3\");\n EnableDisableMetadata(\"Stages\", false, \"StageId\", \"4\");\n EnableDisableMetadata(\"Stages\", false, \"StageId\", \"5\");\n return this;\n }\n\n public ParentHelper WithPipelinesDisabled()\n {\n EnableDisableMetadata(\"Pipelines\", false);\n return this;\n }\n public ParentHelper WithPipelinesEnabled()\n {\n EnableDisableMetadata(\"Pipelines\", true);\n return this;\n }\n public ParentHelper With2MinWaitsOnWorkers()\n {\n SetParameterValue(\"120\", \"ParameterName\", \"WaitTime\");\n return this;\n }\n\n public ParentHelper WithBatchExecutionHandling()\n {\n ExecuteNonQuery(@$\"UPDATE [procfwk].[Properties] \nSET [PropertyValue] = '1' \nWHERE [PropertyName] = 'UseExecutionBatches'\");\n return this;\n }\n\n public ParentHelper WithoutBatchExecutionHandling()\n {\n ExecuteNonQuery(@$\"UPDATE [procfwk].[Properties] \nSET [PropertyValue] = '0' \nWHERE [PropertyName] = 'UseExecutionBatches'\");\n return this;\n }\n\n public ParentHelper WithFailureHandling(string mode)\n {\n ExecuteNonQuery(@$\"UPDATE [procfwk].[Properties] \nSET [PropertyValue] = '{mode}' \nWHERE [PropertyName] = 'FailureHandling'\");\n return this;\n }\n\n public ParentHelper WithCancelledWorkersBlock(bool mode)\n {\n string modeString = mode ? \"1\" : \"0\";\n ExecuteNonQuery(@$\"UPDATE [procfwk].[Properties] \nSET [PropertyValue] = '{modeString}' \nWHERE [PropertyName] = 'CancelledWorkerResultBlocks'\");\n return this;\n }\n public ParentHelper WithOverideRestart(bool mode)\n {\n string modeString = mode ? \"1\" : \"0\";\n ExecuteNonQuery(@$\"UPDATE [procfwk].[Properties] \nSET [PropertyValue] = '{modeString}' \nWHERE [PropertyName] = 'OverideRestart'\");\n return this;\n }\n\n public ParentHelper WithoutPrecursorObject()\n {\n ExecuteNonQuery(@$\"UPDATE [procfwk].[Properties] \nSET [PropertyValue] = '[dbo].[none]' \nWHERE [PropertyName] = 'ExecutionPrecursorProc'\");\n return this;\n }\n\n public ParentHelper WithPrecursorObject()\n {\n ExecuteNonQuery(@$\"UPDATE [procfwk].[Properties] \nSET [PropertyValue] = '[dbo].[ExampleCustomExecutionPrecursor]' \nWHERE [PropertyName] = 'ExecutionPrecursorProc'\");\n return this;\n }\n public ParentHelper WithSingleExecutionStage()\n {\n ExecuteNonQuery(\"UPDATE [procfwk].[Pipelines] SET [StageId] = 1\");\n return this;\n }\n\n public ParentHelper WithCustom()\n {\n ExecuteStoredProcedure(\"[dbo].[PaulTemp]\", null);\n\n return this;\n }\n private ParentHelper SetFalsePipelineStatus(string falseStatus, string where, string equals)\n {\n ExecuteNonQuery($\"UPDATE [procfwk].[CurrentExecution] SET [PipelineStatus] = '{falseStatus}' WHERE {where} = '{equals.Replace(\"'\", \"''\")}'\");\n return this;\n }\n\n private string GetWorkerRunId(string pipelineName = null)\n {\n string PipelineRunId;\n\n using (var cmd = new SqlCommand(\"procfwkTesting.GetRunIdWhenAvailable\", _conn))\n {\n cmd.CommandTimeout = 600;\n cmd.CommandType = System.Data.CommandType.StoredProcedure;\n\n if (pipelineName != null) cmd.Parameters.Add(new SqlParameter(\"@PipelineName\", pipelineName));\n\n using var reader = cmd.ExecuteReader();\n reader.Read();\n PipelineRunId = reader.GetString(0).ToLower();\n }\n return PipelineRunId;\n }\n\n private void EnableDisableMetadata(string table, bool state)\n {\n string paramValue = state ? \"true\" : \"false\";\n ExecuteNonQuery(@$\"UPDATE [procfwk].[{table}] SET [Enabled] = '{paramValue}'\");\n }\n\n private void EnableDisableMetadata(string table, bool state, string where, string equals)\n {\n string paramValue = state ? \"true\" : \"false\";\n ExecuteNonQuery(@$\"UPDATE [procfwk].[{table}] SET [Enabled] = '{paramValue}' WHERE {where} = '{equals.Replace(\"'\", \"''\")}'\");\n }\n private void SetParameterValue(string value, string where, string equals)\n {\n string sqlStatement = @$\"UPDATE [procfwk].[PipelineParameters] SET [ParameterValue] = '{value.Replace(\"'\", \"''\")}' WHERE {where} = '{equals.Replace(\"'\", \"''\")}'\";\n ExecuteNonQuery(sqlStatement);\n }\n\n private void SimulateError(bool simulate)\n {\n string paramValue = simulate ? \"true\" : \"false\";\n ExecuteNonQuery(@$\"UPDATE pp \nSET [ParameterValue] = '{paramValue}' \nFROM [procfwk].[PipelineParameters] pp \n INNER JOIN [procfwk].[Pipelines] p ON pp.[PipelineId] = p.[PipelineId] \nWHERE p.[PipelineName] = 'Intentional Error' AND pp.[ParameterName] = 'RaiseErrors'\");\n }\n\n public override void TearDown()\n {\n base.TearDown();\n }\n }\n}\n"
},
{
"path": "FactoryTesting/Pipelines/Utilities/GivenCheckForRunningPipeline.cs",
"content": "using FluentAssertions;\nusing NUnit.Framework;\nusing NUnit.Framework.Internal;\nusing System.Threading.Tasks;\nusing FactoryTesting.Helpers;\n\nnamespace FactoryTesting.Pipelines.Utilities\n{\n public class GivenCheckForRunningPipeline\n {\n private UtilitiesHelper _helper;\n \n [OneTimeSetUp]\n public async Task WhenPipelineRun()\n {\n _helper = new UtilitiesHelper()\n .WithBasicMetadata()\n .WithTenantAndSubscriptionIds()\n .WithParameter(\"PipelineName\", \"Check For Running Pipeline\");\n\n await _helper.RunPipeline(\"Check For Running Pipeline\");\n }\n\n [Test]\n public void ThenPipelineOutcomeIsFailed()\n {\n _helper.RunOutcome.Should().Be(\"Failed\");\n }\n\n [Test]\n public async Task ThenActivityShouldReturnOneFilteredItemCount()\n {\n var filteredCount = await _helper.GetActivityOutput(\"Filter Running Pipelines\", \"$.FilteredItemsCount\");\n int.Parse(filteredCount).Should().Be(1);\n }\n\n [Test]\n public async Task ThenActivityShouldReturnMatchingSubscriptionId()\n {\n string subSetting = _helper.GetSetting(\"AZURE_SUBSCRIPTION_ID\");\n\n var subscriptionId = await _helper.GetActivityOutput(\"Set Subscription Id\", \"$.value\");\n subscriptionId.Should().Equals(subSetting.ToString());\n }\n\n [Test]\n public async Task ThenActivityShouldReturnMatchingResourceGroup()\n {\n string rgSsetting = _helper.GetSetting(\"DataFactoryResourceGroup\");\n\n var resourceGroupName = await _helper.GetActivityOutput(\"Set Resource Group Name\", \"$.value\");\n resourceGroupName.Should().Equals(rgSsetting.ToString());\n }\n\n [OneTimeTearDown]\n public void TearDown()\n {\n _helper?.TearDown();\n }\n }\n}\n"
},
{
"path": "FactoryTesting/Pipelines/Utilities/GivenEmailSender.cs",
"content": "using FluentAssertions;\nusing NUnit.Framework;\nusing NUnit.Framework.Internal;\nusing System.Threading.Tasks;\n\nnamespace FactoryTesting.Pipelines.Utilities\n{\n public class GivenEmailSender\n {\n private UtilitiesHelper _helper;\n\n [OneTimeSetUp]\n public async Task WhenPipelineRun()\n {\n _helper = new UtilitiesHelper()\n .WithParameter(\"Recipients\", \"paul@mrpaulandrew.com\")\n .WithParameter(\"Subject\", \"NUnit Test\")\n .WithParameter(\"Body\", \"NUnit Test\"); ;\n\n await _helper.RunPipeline(\"Email Sender\");\n }\n\n [Test]\n public void ThenPipelineOutcomeIsSucceeded()\n {\n _helper.RunOutcome.Should().Be(\"Succeeded\");\n }\n\n [Test]\n public async Task ThenActivityShouldReturnEmailSentTrue()\n {\n var sentState = await _helper.GetActivityOutput(\"Send Email\", \"$.EmailSent\");\n bool.Parse(sentState).Should().Be(true);\n }\n\n [OneTimeTearDown]\n public void TearDown()\n {\n _helper?.TearDown();\n }\n }\n}\n"
},
{
"path": "FactoryTesting/Pipelines/Utilities/GivenThrowException.cs",
"content": "using FluentAssertions;\nusing NUnit.Framework;\nusing NUnit.Framework.Internal;\nusing System.Threading.Tasks;\n\nnamespace FactoryTesting.Pipelines.Utilities\n{\n public class GivenThrowException\n {\n private UtilitiesHelper _helper;\n\n [OneTimeSetUp]\n public async Task WhenPipelineRun()\n {\n _helper = new UtilitiesHelper()\n .WithParameter(\"Message\",\"NUnit Test\");\n \n await _helper.RunPipeline(\"Throw Exception\");\n }\n\n [Test]\n public void ThenPipelineOutcomeIsFailed()\n {\n _helper.RunOutcome.Should().Be(\"Failed\");\n }\n\n [OneTimeTearDown]\n public void TearDown()\n {\n _helper?.TearDown();\n }\n }\n}\n"
},
{
"path": "FactoryTesting/Pipelines/Utilities/UtilitiesHelper.cs",
"content": "using FactoryTesting.Helpers;\nusing System;\nusing System.Data.SqlClient;\nusing System.Threading;\nusing System.Threading.Tasks;\nusing System.Xml;\n\nnamespace FactoryTesting.Pipelines.Utilities\n{\n class UtilitiesHelper : CoverageHelper \n {\n public UtilitiesHelper WithBasicMetadata()\n {\n AddBasicMetadata();\n return this;\n }\n\n public UtilitiesHelper WithTenantAndSubscriptionIds()\n {\n AddTenantAndSubscription();\n return this;\n }\n\n public override void TearDown()\n {\n base.TearDown();\n }\n }\n}\n"
},
{
"path": "FactoryTesting/dev.runsettings",
"content": "\n\n\n \n \n \n \n \n \n \n \n\n"
},
{
"path": "FactoryTesting/multi.runsettings",
"content": "\n\n\n \n \n \n \n \n \n \n \n\n"
},
{
"path": "FactoryTesting/test.runsettings",
"content": "\n\n\n \n \n \n \n \n \n \n \n\n"
},
{
"path": "Functions/.vscode/extensions.json",
"content": "{\n \"recommendations\": [\n \"ms-azuretools.vscode-azurefunctions\",\n \"ms-dotnettools.csharp\"\n ]\n}\n"
},
{
"path": "Functions/.vscode/launch.json",
"content": "{\n \"version\": \"0.2.0\",\n \"configurations\": [\n {\n \"name\": \"Attach to .NET Functions\",\n \"type\": \"coreclr\",\n \"request\": \"attach\",\n \"processId\": \"${command:azureFunctions.pickProcess}\"\n }\n ]\n}"
},
{
"path": "Functions/.vscode/settings.json",
"content": "{\n \"azureFunctions.deploySubpath\": \"bin/Release/netcoreapp3.1/publish\",\n \"azureFunctions.projectLanguage\": \"C#\",\n \"azureFunctions.projectRuntime\": \"~3\",\n \"debug.internalConsoleOptions\": \"neverOpen\",\n \"azureFunctions.preDeployTask\": \"publish\"\n}"
},
{
"path": "Functions/.vscode/tasks.json",
"content": "{\n\t\"version\": \"2.0.0\",\n\t\"tasks\": [\n\t\t{\n\t\t\t\"label\": \"clean\",\n\t\t\t\"command\": \"dotnet\",\n\t\t\t\"args\": [\n\t\t\t\t\"clean\",\n\t\t\t\t\"/property:GenerateFullPaths=true\",\n\t\t\t\t\"/consoleloggerparameters:NoSummary\"\n\t\t\t],\n\t\t\t\"type\": \"process\",\n\t\t\t\"problemMatcher\": \"$msCompile\"\n\t\t},\n\t\t{\n\t\t\t\"label\": \"build\",\n\t\t\t\"command\": \"dotnet\",\n\t\t\t\"args\": [\n\t\t\t\t\"build\",\n\t\t\t\t\"/property:GenerateFullPaths=true\",\n\t\t\t\t\"/consoleloggerparameters:NoSummary\"\n\t\t\t],\n\t\t\t\"type\": \"process\",\n\t\t\t\"dependsOn\": \"clean\",\n\t\t\t\"group\": {\n\t\t\t\t\"kind\": \"build\",\n\t\t\t\t\"isDefault\": true\n\t\t\t},\n\t\t\t\"problemMatcher\": \"$msCompile\"\n\t\t},\n\t\t{\n\t\t\t\"label\": \"clean release\",\n\t\t\t\"command\": \"dotnet\",\n\t\t\t\"args\": [\n\t\t\t\t\"clean\",\n\t\t\t\t\"--configuration\",\n\t\t\t\t\"Release\",\n\t\t\t\t\"/property:GenerateFullPaths=true\",\n\t\t\t\t\"/consoleloggerparameters:NoSummary\"\n\t\t\t],\n\t\t\t\"type\": \"process\",\n\t\t\t\"problemMatcher\": \"$msCompile\"\n\t\t},\n\t\t{\n\t\t\t\"label\": \"publish\",\n\t\t\t\"command\": \"dotnet\",\n\t\t\t\"args\": [\n\t\t\t\t\"publish\",\n\t\t\t\t\"--configuration\",\n\t\t\t\t\"Release\",\n\t\t\t\t\"/property:GenerateFullPaths=true\",\n\t\t\t\t\"/consoleloggerparameters:NoSummary\"\n\t\t\t],\n\t\t\t\"type\": \"process\",\n\t\t\t\"dependsOn\": \"clean release\",\n\t\t\t\"problemMatcher\": \"$msCompile\"\n\t\t},\n\t\t{\n\t\t\t\"type\": \"func\",\n\t\t\t\"dependsOn\": \"build\",\n\t\t\t\"options\": {\n\t\t\t\t\"cwd\": \"${workspaceFolder}/bin/Debug/netcoreapp3.1\"\n\t\t\t},\n\t\t\t\"command\": \"host start\",\n\t\t\t\"isBackground\": true,\n\t\t\t\"problemMatcher\": \"$func-watch\"\n\t\t}\n\t]\n}"
},
{
"path": "Functions/Functions/CancelPipeline.cs",
"content": "using System;\nusing System.IO;\nusing System.Threading.Tasks;\nusing Microsoft.AspNetCore.Mvc;\nusing Microsoft.Azure.WebJobs;\nusing Microsoft.Azure.WebJobs.Extensions.Http;\nusing Microsoft.AspNetCore.Http;\nusing Microsoft.Extensions.Logging;\nusing Newtonsoft.Json;\nusing mrpaulandrew.azure.procfwk.Helpers;\nusing mrpaulandrew.azure.procfwk.Services;\n\nnamespace mrpaulandrew.azure.procfwk\n{\n public static class CancelPipeline\n {\n [FunctionName(\"CancelPipeline\")]\n public static async Task Run(\n [HttpTrigger(AuthorizationLevel.Function, \"post\", Route = null)] HttpRequest httpRequest,\n ILogger logger)\n {\n logger.LogInformation(\"CancelPipeline Function triggered by HTTP request.\");\n\n logger.LogInformation(\"Parsing body from request.\");\n PipelineRunRequest request = await new BodyReader(httpRequest).GetRunRequestBodyAsync();\n request.Validate(logger);\n\n using (var service = PipelineService.GetServiceForRequest(request, logger))\n {\n PipelineRunStatus result = service.CancelPipeline(request);\n logger.LogInformation(\"CancelPipeline Function complete.\");\n return new OkObjectResult(JsonConvert.SerializeObject(result));\n }\n }\n }\n}\n"
},
{
"path": "Functions/Functions/CheckPipelineStatus.cs",
"content": "using System;\nusing System.IO;\nusing System.Threading.Tasks;\nusing Microsoft.AspNetCore.Mvc;\nusing Microsoft.Azure.WebJobs;\nusing Microsoft.Azure.WebJobs.Extensions.Http;\nusing Microsoft.AspNetCore.Http;\nusing Microsoft.Extensions.Logging;\nusing Newtonsoft.Json;\nusing mrpaulandrew.azure.procfwk.Helpers;\nusing mrpaulandrew.azure.procfwk.Services;\n\nnamespace mrpaulandrew.azure.procfwk\n{\n public static class CheckPipelineStatus\n {\n [FunctionName(\"CheckPipelineStatus\")]\n public static async Task Run(\n [HttpTrigger(AuthorizationLevel.Function, \"get\", \"post\", Route = null)] HttpRequest httpRequest,\n ILogger logger)\n {\n logger.LogInformation(\"CheckPipelineStatus Function triggered by HTTP request.\");\n\n logger.LogInformation(\"Parsing body from request.\");\n PipelineRunRequest request = await new BodyReader(httpRequest).GetRunRequestBodyAsync();\n request.Validate(logger);\n\n using (var service = PipelineService.GetServiceForRequest(request, logger))\n {\n PipelineRunStatus result = service.GetPipelineRunStatus(request);\n logger.LogInformation(\"CheckPipelineStatus Function complete.\");\n return new OkObjectResult(JsonConvert.SerializeObject(result));\n }\n }\n }\n}\n"
},
{
"path": "Functions/Functions/ExecutePipeline.cs",
"content": "using System;\nusing System.IO;\nusing System.Threading.Tasks;\nusing Microsoft.AspNetCore.Mvc;\nusing Microsoft.Azure.WebJobs;\nusing Microsoft.Azure.WebJobs.Extensions.Http;\nusing Microsoft.AspNetCore.Http;\nusing Microsoft.Extensions.Logging;\nusing Newtonsoft.Json;\nusing mrpaulandrew.azure.procfwk.Helpers;\nusing mrpaulandrew.azure.procfwk.Services;\n\nnamespace mrpaulandrew.azure.procfwk\n{\n public static class ExecutePipeline\n {\n [FunctionName(\"ExecutePipeline\")]\n public static async Task Run(\n [HttpTrigger(AuthorizationLevel.Function, \"post\", Route = null)] HttpRequest httpRequest,\n ILogger logger)\n {\n logger.LogInformation(\"ExecutePipeline Function triggered by HTTP request.\");\n\n logger.LogInformation(\"Parsing body from request.\");\n PipelineRequest request = await new BodyReader(httpRequest).GetRequestBodyAsync();\n request.Validate(logger);\n\n using (var service = PipelineService.GetServiceForRequest(request, logger))\n {\n PipelineRunStatus result = service.ExecutePipeline(request);\n logger.LogInformation(\"ExecutePipeline Function complete.\");\n return new OkObjectResult(JsonConvert.SerializeObject(result));\n }\n }\n }\n}\n"
},
{
"path": "Functions/Functions/GetActivityErrors.cs",
"content": "using System;\nusing System.IO;\nusing System.Threading.Tasks;\nusing Microsoft.AspNetCore.Mvc;\nusing Microsoft.Azure.WebJobs;\nusing Microsoft.Azure.WebJobs.Extensions.Http;\nusing Microsoft.AspNetCore.Http;\nusing Microsoft.Extensions.Logging;\nusing Newtonsoft.Json;\nusing mrpaulandrew.azure.procfwk.Helpers;\nusing mrpaulandrew.azure.procfwk.Services;\n\nnamespace mrpaulandrew.azure.procfwk\n{\n public static class GetActivityErrors\n {\n [FunctionName(\"GetActivityErrors\")]\n public static async Task Run(\n [HttpTrigger(AuthorizationLevel.Function, \"get\", \"post\", Route = null)] HttpRequest httpRequest,\n ILogger logger)\n {\n logger.LogInformation(\"GetActivityErrors Function triggered by HTTP request.\");\n\n logger.LogInformation(\"Parsing body from request.\");\n PipelineRunRequest request = await new BodyReader(httpRequest).GetRunRequestBodyAsync();\n request.Validate(logger);\n\n using (var service = PipelineService.GetServiceForRequest(request, logger))\n {\n PipelineErrorDetail result = service.GetPipelineRunActivityErrors(request);\n logger.LogInformation(\"GetActivityErrors Function complete.\");\n return new OkObjectResult(JsonConvert.SerializeObject(result));\n }\n }\n }\n}\n"
},
{
"path": "Functions/Functions/SendEmail.cs",
"content": "using System;\nusing System.IO;\nusing System.Threading.Tasks;\nusing System.Net.Mail;\nusing Microsoft.AspNetCore.Mvc;\nusing Microsoft.Azure.WebJobs;\nusing Microsoft.Azure.WebJobs.Extensions.Http;\nusing Microsoft.AspNetCore.Http;\nusing Microsoft.Extensions.Logging;\nusing Newtonsoft.Json;\nusing Newtonsoft.Json.Linq;\nusing mrpaulandrew.azure.procfwk.Helpers;\nusing System.Linq;\n\nnamespace mrpaulandrew.azure.procfwk\n{\n public static class SendEmail\n {\n [FunctionName(\"SendEmail\")]\n public static async Task Run(\n [HttpTrigger(AuthorizationLevel.Function, \"post\", Route = null)] HttpRequest req,\n ILogger log)\n {\n log.LogInformation(\"SendEmail Function triggered by HTTP request.\");\n\n #region ParseRequestBody\n log.LogInformation(\"Parsing body from request.\");\n\n string requestBody = await new StreamReader(req.Body).ReadToEndAsync();\n dynamic data = JsonConvert.DeserializeObject(requestBody);\n string outputString = string.Empty;\n JObject outputJson;\n\n string toRecipients = data?.emailRecipients;\n string ccRecipients = data?.emailCcRecipients;\n string bccRecipients = data?.emailBccRecipients;\n string subject = data?.emailSubject;\n string message = data?.emailBody;\n string passedImportance = data?.emailImportance ?? \"\"; //Set normal importance if not provided. \n #endregion\n\n #region ValidateRequestBody\n //Check for minimum mailing values in request body\n if (subject == null || message == null)\n {\n log.LogInformation(\"Invalid body - Subject/Body.\");\n \n outputString = \"{ \\\"EmailSent\\\": false, \\\"Details\\\": \\\"Email subject or body values missing.\\\"}\";\n outputJson = JObject.Parse(outputString);\n \n return new BadRequestObjectResult(outputJson);\n }\n\n if (\n (toRecipients == null && ccRecipients == null && bccRecipients == null) ||\n (string.IsNullOrEmpty(toRecipients) && string.IsNullOrEmpty(ccRecipients) && string.IsNullOrEmpty(bccRecipients))\n )\n {\n log.LogInformation(\"Invalid body - To/CC/BCC.\");\n \n outputString = \"{ \\\"EmailSent\\\": false, \\\"Details\\\": \\\"No email recipients provided as To/CC/BCC.\\\"}\";\n outputJson = JObject.Parse(outputString);\n \n return new BadRequestObjectResult(outputJson);\n }\n #endregion\n\n //Create email client\n log.LogInformation(\"Creating smtp client.\");\n\n using (var client = SMTPClient.CreateSMTPClient())\n {\n #region CreateMail \n MailAddress from = new MailAddress(SMTPClient.FromEmail);\n MailMessage mail = new MailMessage\n {\n From = from,\n IsBodyHtml = true,\n Subject = subject,\n Body = message\n };\n\n //Set mail importance\n if (passedImportance.ToUpper() == \"HIGH\")\n {\n mail.Priority = MailPriority.High;\n }\n else if (passedImportance.ToUpper() == \"LOW\")\n {\n mail.Priority = MailPriority.Low;\n }\n else\n {\n mail.Priority = MailPriority.Normal;\n }\n #endregion\n\n #region SetRecipients\n //to recipients\n if (!string.IsNullOrEmpty(toRecipients))\n {\n var allRecipients = toRecipients.Split(new[] { \",\" }, StringSplitOptions.RemoveEmptyEntries);\n \n foreach (var toAddress in allRecipients)\n {\n mail.To.Add(toAddress);\n }\n log.LogInformation(\"To Recipients Added: \" + allRecipients.Count().ToString());\n }\n else\n {\n log.LogInformation(\"To Recipients Added: 0\");\n }\n\n //cc recipients\n if (!string.IsNullOrEmpty(ccRecipients))\n {\n var allCcRecipients = ccRecipients.Split(new[] { \",\" }, StringSplitOptions.RemoveEmptyEntries);\n\n foreach (var ccAddress in allCcRecipients)\n {\n mail.CC.Add(ccAddress);\n }\n log.LogInformation(\"CC Recipients Added: \" + allCcRecipients.Count().ToString());\n }\n else\n {\n log.LogInformation(\"CC Recipients Added: 0\");\n }\n\n //bcc recipients\n if (!string.IsNullOrEmpty(bccRecipients))\n {\n var allBccRecipients = bccRecipients.Split(new[] { \",\" }, StringSplitOptions.RemoveEmptyEntries);\n\n foreach (var bccAddress in allBccRecipients)\n {\n mail.Bcc.Add(bccAddress);\n }\n log.LogInformation(\"BCC Recipients Added: \" + allBccRecipients.Count().ToString());\n }\n else\n {\n log.LogInformation(\"BCC Recipients Added: 0\");\n }\n #endregion\n\n #region SendEmail\n try\n {\n log.LogInformation(\"Sending email.\");\n\n client.Send(mail);\n outputString = \"{ \\\"EmailSent\\\": true }\";\n\n log.LogInformation(\"Sent email.\");\n }\n catch (SmtpException smtpEx)\n {\n outputString = \"{ \\\"EmailSent\\\": false, \\\"Details\\\": \\\"SMTP exception caught and logged to error output.\\\"}\";\n\n log.LogError(smtpEx.Message);\n log.LogInformation(\"Message has not been sent. Check Azure Function Logs for more information.\");\n }\n catch (Exception ex)\n {\n outputString = \"{ \\\"EmailSent\\\": false, \\\"Details\\\": \\\"Other exception caught and logged to error output.\\\"}\";\n\n log.LogError(ex.Message);\n log.LogInformation(\"Message has not been sent. Check Azure Function Logs for more information.\");\n }\n #endregion\n }\n\n outputJson = JObject.Parse(outputString);\n\n log.LogInformation(\"SendEmail Function complete.\");\n return new OkObjectResult(outputJson);\n }\n }\n}\n"
},
{
"path": "Functions/Functions/ValidatePipeline.cs",
"content": "using System;\nusing System.IO;\nusing System.Threading.Tasks;\nusing Microsoft.AspNetCore.Mvc;\nusing Microsoft.Azure.WebJobs;\nusing Microsoft.Azure.WebJobs.Extensions.Http;\nusing Microsoft.AspNetCore.Http;\nusing Microsoft.Extensions.Logging;\nusing Newtonsoft.Json;\nusing mrpaulandrew.azure.procfwk.Helpers;\nusing mrpaulandrew.azure.procfwk.Services;\n\nnamespace mrpaulandrew.azure.procfwk\n{\n public static class ValidatePipeline\n {\n [FunctionName(\"ValidatePipeline\")]\n public static async Task Run(\n [HttpTrigger(AuthorizationLevel.Function, \"get\", \"post\", Route = null)] HttpRequest httpRequest,\n ILogger logger)\n {\n logger.LogInformation(\"ValidatePipeline Function triggered by HTTP request.\");\n\n logger.LogInformation(\"Parsing body from request.\");\n PipelineRequest request = await new BodyReader(httpRequest).GetRequestBodyAsync();\n request.Validate(logger);\n\n using (var service = PipelineService.GetServiceForRequest(request, logger))\n {\n PipelineDescription result = service.ValidatePipeline(request);\n logger.LogInformation(\"ValidatePipeline Function complete.\");\n return new OkObjectResult(JsonConvert.SerializeObject(result));\n }\n }\n }\n}\n"
},
{
"path": "Functions/Functions.csproj",
"content": "\n \n netcoreapp3.1\n v3\n mrpaulandrew.azure.procfwk\n mrpaulandrew.azure.procfwk\n \n \n \n \n \n \n \n \n \n \n \n \n \n PreserveNewest\n \n \n Always\n \n \n"
},
{
"path": "Functions/Helpers/BodyReader.cs",
"content": "using System;\nusing System.Collections.Generic;\nusing System.IO;\nusing System.Text;\nusing System.Threading.Tasks;\nusing Microsoft.AspNetCore.Http;\nusing Microsoft.Extensions.Logging;\nusing Newtonsoft.Json;\n\nnamespace mrpaulandrew.azure.procfwk.Helpers\n{\n public class BodyReader\n {\n public string Body;\n public BodyReader(HttpRequest httpRequest)\n {\n Body = new StreamReader(httpRequest.Body).ReadToEnd();\n }\n\n public Task GetRequestBody()\n {\n PipelineRequest request = JsonConvert.DeserializeObject(Body);\n return Task.FromResult(request);\n }\n\n public async Task GetRequestBodyAsync()\n {\n PipelineRequest request = await GetRequestBody();\n return request;\n }\n\n public Task GetRunRequestBody()\n {\n PipelineRunRequest request = JsonConvert.DeserializeObject(Body);\n return Task.FromResult(request);\n }\n\n public async Task GetRunRequestBodyAsync()\n {\n PipelineRunRequest request = await GetRunRequestBody();\n return request;\n }\n }\n}\n"
},
{
"path": "Functions/Helpers/InvalidRequestException.cs",
"content": "using System;\nusing System.Runtime.Serialization;\n\nnamespace mrpaulandrew.azure.procfwk.Helpers\n{\n [Serializable]\n internal class InvalidRequestException : Exception\n {\n public InvalidRequestException()\n {\n }\n\n public InvalidRequestException(string message) : base(message)\n {\n }\n\n public InvalidRequestException(string message, Exception innerException) : base(message, innerException)\n {\n }\n\n protected InvalidRequestException(SerializationInfo info, StreamingContext context) : base(info, context)\n {\n }\n }\n}"
},
{
"path": "Functions/Helpers/KeyVaultClient.cs",
"content": "using Azure.Identity;\nusing Azure.Security.KeyVault.Secrets;\nusing System;\n\nnamespace mrpaulandrew.azure.procfwk.Helpers\n{\n internal class KeyVaultClient\n {\n private static readonly DefaultAzureCredential defaultCred = new DefaultAzureCredential();\n\n public static string GetSecretFromUri(string secretString)\n {\n return GetSecretFromUri(new Uri(secretString));\n }\n\n public static string GetSecretFromUri(Uri secretUri)\n {\n string keyVaultURL = \"https://\" + secretUri.Host.ToString();\n string secretName = secretUri.LocalPath.ToString().Replace(\"secrets/\", \"\").Replace(\"/\", \"\");\n return CreateKeyVaultClient(keyVaultURL).GetSecret(secretName).Value.Value;\n }\n\n public static string GetSecretFromName(string keyVaultURL, string secretName)\n {\n return CreateKeyVaultClient(keyVaultURL).GetSecret(secretName).Value.Value;\n }\n\n private static SecretClient CreateKeyVaultClient(string keyVaultURL)\n {\n return new SecretClient(new Uri(keyVaultURL), defaultCred);\n }\n }\n}\n"
},
{
"path": "Functions/Helpers/PipelineRequest.cs",
"content": "using Microsoft.Extensions.Logging;\nusing System;\nusing System.Collections.Generic;\n\nnamespace mrpaulandrew.azure.procfwk.Helpers\n{\n public class PipelineRequest\n {\n public string TenantId { get; set; }\n public string ApplicationId { get; set; }\n public string AuthenticationKey { get; set; }\n public string SubscriptionId { get; set; }\n public string ResourceGroupName { get; set; }\n public string OrchestratorName { get; set; }\n public string PipelineName { get; set; }\n public PipelineServiceType? OrchestratorType { get; set; }\n\n public Dictionary PipelineParameters;\n\n public virtual void Validate(ILogger logger)\n {\n // ensure properties not null\n if (\n TenantId == null ||\n ApplicationId == null ||\n AuthenticationKey == null ||\n SubscriptionId == null ||\n ResourceGroupName == null ||\n OrchestratorType == null ||\n OrchestratorName == null ||\n PipelineName == null\n )\n ReportInvalidBody(logger);\n\n //other validation\n if (!CheckGuid(TenantId)) ReportInvalidBody(logger, \"Expected Tenant Id to be a GUID.\");\n if (!CheckGuid(SubscriptionId)) ReportInvalidBody(logger, \"Expected Subscription Id to be a GUID.\");\n\n // resolve key vault values\n if (!CheckGuid(ApplicationId) && CheckUri(ApplicationId))\n {\n logger.LogInformation(\"Getting applicationId from Key Vault\");\n ApplicationId = KeyVaultClient.GetSecretFromUri(ApplicationId);\n }\n\n if (CheckUri(AuthenticationKey))\n {\n logger.LogInformation(\"Getting authenticationKey from Key Vault\");\n AuthenticationKey = KeyVaultClient.GetSecretFromUri(AuthenticationKey);\n }\n }\n\n private bool CheckUri(string uriValue)\n {\n bool result = Uri.TryCreate(uriValue, UriKind.Absolute, out Uri uriResult)\n && (uriResult.Scheme == Uri.UriSchemeHttp || uriResult.Scheme == Uri.UriSchemeHttps);\n\n return result;\n }\n\n public bool CheckGuid(string idValue)\n {\n bool result = Guid.TryParse(idValue, out _);\n\n return result;\n }\n\n protected void ReportInvalidBody(ILogger logger)\n {\n var msg = \"Invalid body.\";\n logger.LogError(msg);\n throw new InvalidRequestException(msg);\n }\n\n protected void ReportInvalidBody(ILogger logger, string additions)\n {\n var msg = \"Invalid body. \" + additions;\n logger.LogError(msg);\n throw new InvalidRequestException(msg);\n }\n\n public Dictionary ParametersAsObjects\n {\n get\n {\n if (PipelineParameters == null)\n return null;\n var dictionary = new Dictionary();\n foreach (var key in PipelineParameters.Keys)\n dictionary.Add(key, PipelineParameters[key]);\n return dictionary;\n }\n }\n }\n}"
},
{
"path": "Functions/Helpers/PipelineRunRequest.cs",
"content": "using System;\nusing Microsoft.Extensions.Logging;\n\nnamespace mrpaulandrew.azure.procfwk.Helpers\n{\n public class PipelineRunRequest : PipelineRequest\n {\n public string RunId { get; set; }\n\n public bool RecursivePipelineCancel = true; //might provide this as part of the request later\n\n public DateTime ActivityQueryStart = DateTime.Now.AddDays(-7); //max duration for eventual RunFilterParameters\n public DateTime ActivityQueryEnd = DateTime.Now;\n\n public override void Validate(ILogger logger)\n {\n base.Validate(logger);\n\n // ensure properties not null\n if (RunId == null)\n ReportInvalidBody(logger);\n\n //other validation\n if (!CheckGuid(RunId)) ReportInvalidBody(logger, \"Expected Run Id to be a GUID.\");\n }\n }\n}\n"
},
{
"path": "Functions/Helpers/SMTPClient.cs",
"content": "using System;\nusing System.Net.Mail;\n\nnamespace mrpaulandrew.azure.procfwk.Helpers\n{\n internal class SMTPClient\n {\n public static string FromEmail { get; set; }\n\n public static SmtpClient CreateSMTPClient()\n {\n string smtpHost = Environment.GetEnvironmentVariable(\"AppSettingSmtpHost\");\n int smtpPort = int.Parse(Environment.GetEnvironmentVariable(\"AppSettingSmtpPort\"));\n string smtpUser = Environment.GetEnvironmentVariable(\"AppSettingSmtpUser\");\n string smtpPass = Environment.GetEnvironmentVariable(\"AppSettingSmtpPass\");\n \n FromEmail = Environment.GetEnvironmentVariable(\"AppSettingFromEmail\");\n\n SmtpClient emailClient = new SmtpClient\n {\n EnableSsl = true,\n UseDefaultCredentials = false, //order properties are set is important\n Credentials = new System.Net.NetworkCredential(smtpUser, smtpPass),\n DeliveryMethod = SmtpDeliveryMethod.Network,\n Host = smtpHost,\n Port = smtpPort\n };\n\n return emailClient;\n }\n }\n}\n"
},
{
"path": "Functions/Properties/ServiceDependencies/FrameworkSupportFunctions - Zip Deploy/profile.arm.json",
"content": "{\n \"$schema\": \"https://schema.management.azure.com/schemas/2018-05-01/subscriptionDeploymentTemplate.json#\",\n \"contentVersion\": \"1.0.0.0\",\n \"metadata\": {\n \"_dependencyType\": \"function.windows.consumption\"\n },\n \"parameters\": {\n \"resourceGroupName\": {\n \"type\": \"string\",\n \"defaultValue\": \"ADF.procfwk\",\n \"metadata\": {\n \"description\": \"Name of the resource group for the resource. It is recommended to put resources under same resource group for better tracking.\"\n }\n },\n \"resourceGroupLocation\": {\n \"type\": \"string\",\n \"defaultValue\": \"uksouth\",\n \"metadata\": {\n \"description\": \"Location of the resource group. Resource groups could have different location than resources, however by default we use API versions from latest hybrid profile which support all locations for resource types we support.\"\n }\n },\n \"resourceName\": {\n \"type\": \"string\",\n \"defaultValue\": \"FrameworkSupportFunctions\",\n \"metadata\": {\n \"description\": \"Name of the main resource to be created by this template.\"\n }\n },\n \"resourceLocation\": {\n \"type\": \"string\",\n \"defaultValue\": \"[parameters('resourceGroupLocation')]\",\n \"metadata\": {\n \"description\": \"Location of the resource. By default use resource group's location, unless the resource provider is not supported there.\"\n }\n }\n },\n \"resources\": [\n {\n \"type\": \"Microsoft.Resources/resourceGroups\",\n \"name\": \"[parameters('resourceGroupName')]\",\n \"location\": \"[parameters('resourceGroupLocation')]\",\n \"apiVersion\": \"2019-10-01\"\n },\n {\n \"type\": \"Microsoft.Resources/deployments\",\n \"name\": \"[concat(parameters('resourceGroupName'), 'Deployment', uniqueString(concat(parameters('resourceName'), subscription().subscriptionId)))]\",\n \"resourceGroup\": \"[parameters('resourceGroupName')]\",\n \"apiVersion\": \"2019-10-01\",\n \"dependsOn\": [\n \"[parameters('resourceGroupName')]\"\n ],\n \"properties\": {\n \"mode\": \"Incremental\",\n \"expressionEvaluationOptions\": {\n \"scope\": \"inner\"\n },\n \"parameters\": {\n \"resourceGroupName\": {\n \"value\": \"[parameters('resourceGroupName')]\"\n },\n \"resourceGroupLocation\": {\n \"value\": \"[parameters('resourceGroupLocation')]\"\n },\n \"resourceName\": {\n \"value\": \"[parameters('resourceName')]\"\n },\n \"resourceLocation\": {\n \"value\": \"[parameters('resourceLocation')]\"\n }\n },\n \"template\": {\n \"$schema\": \"http://schema.management.azure.com/schemas/2015-01-01/deploymentTemplate.json#\",\n \"contentVersion\": \"1.0.0.0\",\n \"parameters\": {\n \"resourceGroupName\": {\n \"type\": \"string\"\n },\n \"resourceGroupLocation\": {\n \"type\": \"string\"\n },\n \"resourceName\": {\n \"type\": \"string\"\n },\n \"resourceLocation\": {\n \"type\": \"string\"\n }\n },\n \"variables\": {\n \"storage_name\": \"[toLower(concat('storage', uniqueString(concat(parameters('resourceName'), subscription().subscriptionId))))]\",\n \"storage_ResourceId\": \"[concat('/subscriptions/', subscription().subscriptionId, '/resourceGroups/', parameters('resourceGroupName'), '/providers/Microsoft.Storage/storageAccounts/', variables('storage_name'))]\",\n \"function_ResourceId\": \"[concat('/subscriptions/', subscription().subscriptionId, '/resourceGroups/', parameters('resourceGroupName'), '/providers/Microsoft.Web/sites/', parameters('resourceName'))]\"\n },\n \"resources\": [\n {\n \"location\": \"[parameters('resourceGroupLocation')]\",\n \"name\": \"[variables('storage_name')]\",\n \"type\": \"Microsoft.Storage/storageAccounts\",\n \"apiVersion\": \"2017-10-01\",\n \"tags\": {\n \"[concat('hidden-related:', concat('/providers/Microsoft.Web/sites/', parameters('resourceName')))]\": \"empty\"\n },\n \"properties\": {\n \"supportsHttpsTrafficOnly\": true\n },\n \"sku\": {\n \"name\": \"Standard_LRS\"\n },\n \"kind\": \"Storage\"\n },\n {\n \"location\": \"[parameters('resourceLocation')]\",\n \"name\": \"[parameters('resourceName')]\",\n \"type\": \"Microsoft.Web/sites\",\n \"apiVersion\": \"2015-08-01\",\n \"dependsOn\": [\n \"[variables('storage_ResourceId')]\"\n ],\n \"kind\": \"functionapp\",\n \"properties\": {\n \"name\": \"[parameters('resourceName')]\",\n \"kind\": \"functionapp\",\n \"httpsOnly\": true,\n \"reserved\": false\n },\n \"identity\": {\n \"type\": \"SystemAssigned\"\n },\n \"resources\": [\n {\n \"name\": \"appsettings\",\n \"type\": \"config\",\n \"apiVersion\": \"2015-08-01\",\n \"dependsOn\": [\n \"[variables('function_ResourceId')]\"\n ],\n \"properties\": {\n \"WEBSITE_CONTENTAZUREFILECONNECTIONSTRING\": \"[concat('DefaultEndpointsProtocol=https;AccountName=', variables('storage_name'), ';AccountKey=', listKeys(variables('storage_ResourceId'), '2017-10-01').keys[0].value, ';EndpointSuffix=', 'core.windows.net')]\",\n \"WEBSITE_CONTENTSHARE\": \"[toLower(parameters('resourceName'))]\",\n \"AzureWebJobsDashboard\": \"[concat('DefaultEndpointsProtocol=https;AccountName=', variables('storage_name'), ';AccountKey=', listKeys(variables('storage_ResourceId'), '2017-10-01').keys[0].value, ';EndpointSuffix=', 'core.windows.net')]\",\n \"AzureWebJobsStorage\": \"[concat('DefaultEndpointsProtocol=https;AccountName=', variables('storage_name'), ';AccountKey=', listKeys(variables('storage_ResourceId'), '2017-10-01').keys[0].value, ';EndpointSuffix=', 'core.windows.net')]\",\n \"FUNCTIONS_EXTENSION_VERSION\": \"~3\",\n \"FUNCTIONS_WORKER_RUNTIME\": \"dotnet\"\n }\n }\n ]\n }\n ]\n }\n }\n }\n ]\n}"
},
{
"path": "Functions/Properties/ServiceDependencies/FrameworkSupportFunctionsBig - Web Deploy/profile.arm.json",
"content": "{\n \"$schema\": \"https://schema.management.azure.com/schemas/2018-05-01/subscriptionDeploymentTemplate.json#\",\n \"contentVersion\": \"1.0.0.0\",\n \"metadata\": {\n \"_dependencyType\": \"appService.windows\"\n },\n \"parameters\": {\n \"resourceGroupName\": {\n \"type\": \"string\",\n \"defaultValue\": \"ADF.procfwkStressTest\",\n \"metadata\": {\n \"description\": \"Name of the resource group for the resource. It is recommended to put resources under same resource group for better tracking.\"\n }\n },\n \"resourceGroupLocation\": {\n \"type\": \"string\",\n \"defaultValue\": \"uksouth\",\n \"metadata\": {\n \"description\": \"Location of the resource group. Resource groups could have different location than resources, however by default we use API versions from latest hybrid profile which support all locations for resource types we support.\"\n }\n },\n \"resourceName\": {\n \"type\": \"string\",\n \"defaultValue\": \"FrameworkSupportFunctionsBig\",\n \"metadata\": {\n \"description\": \"Name of the main resource to be created by this template.\"\n }\n },\n \"resourceLocation\": {\n \"type\": \"string\",\n \"defaultValue\": \"[parameters('resourceGroupLocation')]\",\n \"metadata\": {\n \"description\": \"Location of the resource. By default use resource group's location, unless the resource provider is not supported there.\"\n }\n }\n },\n \"variables\": {\n \"appServicePlan_name\": \"[concat('Plan', uniqueString(concat(parameters('resourceName'), subscription().subscriptionId)))]\",\n \"appServicePlan_ResourceId\": \"[concat('/subscriptions/', subscription().subscriptionId, '/resourceGroups/', parameters('resourceGroupName'), '/providers/Microsoft.Web/serverFarms/', variables('appServicePlan_name'))]\"\n },\n \"resources\": [\n {\n \"type\": \"Microsoft.Resources/resourceGroups\",\n \"name\": \"[parameters('resourceGroupName')]\",\n \"location\": \"[parameters('resourceGroupLocation')]\",\n \"apiVersion\": \"2019-10-01\"\n },\n {\n \"type\": \"Microsoft.Resources/deployments\",\n \"name\": \"[concat(parameters('resourceGroupName'), 'Deployment', uniqueString(concat(parameters('resourceName'), subscription().subscriptionId)))]\",\n \"resourceGroup\": \"[parameters('resourceGroupName')]\",\n \"apiVersion\": \"2019-10-01\",\n \"dependsOn\": [\n \"[parameters('resourceGroupName')]\"\n ],\n \"properties\": {\n \"mode\": \"Incremental\",\n \"template\": {\n \"$schema\": \"http://schema.management.azure.com/schemas/2015-01-01/deploymentTemplate.json#\",\n \"contentVersion\": \"1.0.0.0\",\n \"resources\": [\n {\n \"location\": \"[parameters('resourceLocation')]\",\n \"name\": \"[parameters('resourceName')]\",\n \"type\": \"Microsoft.Web/sites\",\n \"apiVersion\": \"2015-08-01\",\n \"tags\": {\n \"[concat('hidden-related:', variables('appServicePlan_ResourceId'))]\": \"empty\"\n },\n \"dependsOn\": [\n \"[variables('appServicePlan_ResourceId')]\"\n ],\n \"kind\": \"app\",\n \"properties\": {\n \"name\": \"[parameters('resourceName')]\",\n \"kind\": \"app\",\n \"httpsOnly\": true,\n \"reserved\": false,\n \"serverFarmId\": \"[variables('appServicePlan_ResourceId')]\",\n \"siteConfig\": {\n \"metadata\": [\n {\n \"name\": \"CURRENT_STACK\",\n \"value\": \"dotnetcore\"\n }\n ]\n }\n },\n \"identity\": {\n \"type\": \"SystemAssigned\"\n }\n },\n {\n \"location\": \"[parameters('resourceLocation')]\",\n \"name\": \"[variables('appServicePlan_name')]\",\n \"type\": \"Microsoft.Web/serverFarms\",\n \"apiVersion\": \"2015-08-01\",\n \"sku\": {\n \"name\": \"S1\",\n \"tier\": \"Standard\",\n \"family\": \"S\",\n \"size\": \"S1\"\n },\n \"properties\": {\n \"name\": \"[variables('appServicePlan_name')]\"\n }\n }\n ]\n }\n }\n }\n ]\n}"
},
{
"path": "Functions/Services/AzureDataFactoryService.cs",
"content": "using System;\nusing System.Threading;\nusing Newtonsoft.Json;\nusing Microsoft.Rest;\nusing Microsoft.Extensions.Logging;\nusing Microsoft.IdentityModel.Clients.ActiveDirectory;\nusing Microsoft.Azure.Management.DataFactory;\nusing Microsoft.Azure.Management.DataFactory.Models;\nusing mrpaulandrew.azure.procfwk.Helpers;\n\nnamespace mrpaulandrew.azure.procfwk.Services\n{\n public class AzureDataFactoryService : PipelineService\n {\n private readonly DataFactoryManagementClient _adfManagementClient;\n private readonly ILogger _logger;\n\n public AzureDataFactoryService(PipelineRequest request, ILogger logger)\n {\n _logger = logger;\n _logger.LogInformation(\"Creating ADF connectivity clients.\");\n\n //Auth details\n var context = new AuthenticationContext(\"https://login.windows.net/\" + request.TenantId);\n var cc = new ClientCredential(request.ApplicationId, request.AuthenticationKey);\n var result = context.AcquireTokenAsync(\"https://management.azure.com/\", cc).Result;\n var cred = new TokenCredentials(result.AccessToken);\n\n //Management Client\n _adfManagementClient = new DataFactoryManagementClient(cred)\n {\n SubscriptionId = request.SubscriptionId\n };\n }\n\n public override PipelineDescription ValidatePipeline(PipelineRequest request)\n {\n _logger.LogInformation(\"Validating ADF pipeline.\");\n\n try\n {\n var pipelineResource = _adfManagementClient.Pipelines.Get\n (\n request.ResourceGroupName, \n request.OrchestratorName, \n request.PipelineName\n );\n \n _logger.LogInformation(pipelineResource.Id.ToString());\n\n return new PipelineDescription()\n {\n PipelineExists = \"True\",\n PipelineName = pipelineResource.Name,\n PipelineId = pipelineResource.Id,\n PipelineType = pipelineResource.Type,\n ActivityCount = pipelineResource.Activities.Count\n };\n }\n catch (Microsoft.Rest.Azure.CloudException) //expected exception when pipeline doesnt exist\n {\n return new PipelineDescription()\n {\n PipelineExists = \"False\",\n PipelineName = request.PipelineName,\n PipelineId = \"Unknown\",\n PipelineType = \"Unknown\",\n ActivityCount = 0\n };\n }\n catch (Exception ex) //other unknown issue\n {\n _logger.LogInformation(ex.Message);\n _logger.LogInformation(ex.GetType().ToString());\n throw new InvalidRequestException(\"Failed to validate pipeline. \", ex);\n }\n }\n\n public override PipelineRunStatus ExecutePipeline(PipelineRequest request)\n {\n if (request.PipelineParameters == null)\n _logger.LogInformation(\"Calling pipeline without parameters.\");\n else\n _logger.LogInformation(\"Calling pipeline with parameters.\");\n\n var runResponse = _adfManagementClient.Pipelines.CreateRunWithHttpMessagesAsync\n (\n request.ResourceGroupName, \n request.OrchestratorName, \n request.PipelineName, \n parameters: request.ParametersAsObjects\n ).Result.Body;\n\n _logger.LogInformation(\"Pipeline run ID: \" + runResponse.RunId);\n\n //Wait and check for pipeline to start...\n PipelineRun pipelineRun;\n _logger.LogInformation(\"Checking ADF pipeline status.\");\n while (true)\n {\n pipelineRun = _adfManagementClient.PipelineRuns.Get\n (\n request.ResourceGroupName, \n request.OrchestratorName,\n runResponse.RunId\n );\n\n _logger.LogInformation(\"Waiting for pipeline to start, current status: \" + pipelineRun.Status);\n\n if (pipelineRun.Status != \"Queued\")\n break;\n Thread.Sleep(internalWaitDuration);\n }\n\n return new PipelineRunStatus()\n {\n PipelineName = request.PipelineName,\n RunId = runResponse.RunId,\n ActualStatus = pipelineRun.Status\n };\n }\n\n public override PipelineRunStatus CancelPipeline(PipelineRunRequest request)\n {\n _logger.LogInformation(\"Getting ADF pipeline current status.\");\n\n PipelineRun pipelineRun;\n pipelineRun = _adfManagementClient.PipelineRuns.Get\n (\n request.ResourceGroupName,\n request.OrchestratorName,\n request.RunId\n );\n\n //Defensive check\n PipelineNameCheck(request.PipelineName, pipelineRun.PipelineName);\n\n if (pipelineRun.Status == \"InProgress\" || pipelineRun.Status == \"Queued\")\n {\n _logger.LogInformation(\"Attempting to cancel ADF pipeline.\");\n _adfManagementClient.PipelineRuns.Cancel\n (\n request.ResourceGroupName,\n request.OrchestratorName,\n request.RunId,\n isRecursive : request.RecursivePipelineCancel\n );\n }\n else\n {\n _logger.LogInformation(\"ADF pipeline status: \" + pipelineRun.Status);\n throw new InvalidRequestException(\"Target pipeline is not in a state that can be cancelled.\");\n }\n\n //wait for cancelled state\n _logger.LogInformation(\"Checking ADF pipeline status after cancel request.\");\n while (true)\n {\n pipelineRun = _adfManagementClient.PipelineRuns.Get\n (\n request.ResourceGroupName,\n request.OrchestratorName,\n request.RunId\n );\n\n _logger.LogInformation(\"Waiting for pipeline to cancel, current status: \" + pipelineRun.Status);\n\n if (pipelineRun.Status == \"Cancelled\")\n break;\n Thread.Sleep(internalWaitDuration);\n }\n\n //Final return detail\n return new PipelineRunStatus()\n {\n PipelineName = request.PipelineName,\n RunId = request.RunId,\n ActualStatus = pipelineRun.Status.Replace(\"Canceling\", \"Cancelling\") //microsoft typo\n };\n }\n\n public override PipelineRunStatus GetPipelineRunStatus(PipelineRunRequest request)\n {\n _logger.LogInformation(\"Checking ADF pipeline status.\");\n\n //Get pipeline status with provided run id\n PipelineRun pipelineRun;\n pipelineRun = _adfManagementClient.PipelineRuns.Get\n (\n request.ResourceGroupName, \n request.OrchestratorName, \n request.RunId\n );\n\n //Defensive check\n PipelineNameCheck(request.PipelineName, pipelineRun.PipelineName);\n\n _logger.LogInformation(\"ADF pipeline status: \" + pipelineRun.Status);\n\n //Defensive check\n PipelineNameCheck(request.PipelineName, pipelineRun.PipelineName);\n\n //Final return detail\n return new PipelineRunStatus()\n {\n PipelineName = request.PipelineName,\n RunId = pipelineRun.RunId,\n ActualStatus = pipelineRun.Status.Replace(\"Canceling\", \"Cancelling\") //microsoft typo\n };\n }\n\n public override PipelineErrorDetail GetPipelineRunActivityErrors(PipelineRunRequest request)\n {\n PipelineRun pipelineRun = _adfManagementClient.PipelineRuns.Get\n (\n request.ResourceGroupName, \n request.OrchestratorName, \n request.RunId\n );\n\n //Defensive check\n PipelineNameCheck(request.PipelineName, pipelineRun.PipelineName);\n\n _logger.LogInformation(\"Create pipeline Activity Runs query filters.\");\n RunFilterParameters filterParams = new RunFilterParameters\n (\n request.ActivityQueryStart, \n request.ActivityQueryEnd\n );\n\n _logger.LogInformation(\"Querying ADF pipeline for Activity Runs.\");\n ActivityRunsQueryResponse queryResponse = _adfManagementClient.ActivityRuns.QueryByPipelineRun\n (\n request.ResourceGroupName, \n request.OrchestratorName, \n request.RunId, \n filterParams\n );\n\n //Create initial output content\n PipelineErrorDetail output = new PipelineErrorDetail()\n {\n PipelineName = request.PipelineName,\n ActualStatus = pipelineRun.Status,\n RunId = request.RunId,\n ResponseCount = queryResponse.Value.Count\n };\n\n _logger.LogInformation(\"Pipeline status: \" + pipelineRun.Status);\n _logger.LogInformation(\"Activities found in pipeline response: \" + queryResponse.Value.Count.ToString());\n\n //Loop over activities in pipeline run\n foreach (ActivityRun activity in queryResponse.Value)\n {\n if (activity.Error == null)\n {\n continue; //only want errors\n }\n\n //Parse error output to customise output\n dynamic outputBlockInner = JsonConvert.DeserializeObject(activity.Error.ToString());\n string errorCode = outputBlockInner?.errorCode;\n string errorType = outputBlockInner?.failureType;\n string errorMessage = outputBlockInner?.message;\n\n _logger.LogInformation(\"Activity run id: \" + activity.ActivityRunId);\n _logger.LogInformation(\"Activity name: \" + activity.ActivityName);\n _logger.LogInformation(\"Activity type: \" + activity.ActivityType);\n _logger.LogInformation(\"Error message: \" + errorMessage);\n\n output.Errors.Add(new FailedActivity()\n {\n ActivityRunId = activity.ActivityRunId,\n ActivityName = activity.ActivityName,\n ActivityType = activity.ActivityType,\n ErrorCode = errorCode,\n ErrorType = errorType,\n ErrorMessage = errorMessage\n });\n }\n return output;\n }\n\n public override void Dispose()\n {\n _adfManagementClient?.Dispose();\n }\n }\n}"
},
{
"path": "Functions/Services/AzureSynapseService.cs",
"content": "using System;\nusing System.Threading;\nusing System.Collections.Generic;\nusing Newtonsoft.Json;\nusing Microsoft.Rest;\nusing Microsoft.Extensions.Logging;\nusing Microsoft.IdentityModel.Clients.ActiveDirectory;\nusing Microsoft.Azure.Management.Synapse;\nusing Azure.Core;\nusing Azure.Identity;\nusing Azure.Analytics.Synapse.Artifacts;\nusing Azure.Analytics.Synapse.Artifacts.Models;\nusing mrpaulandrew.azure.procfwk.Helpers;\n\nnamespace mrpaulandrew.azure.procfwk.Services\n{\n public class AzureSynapseService : PipelineService\n {\n private readonly SynapseManagementClient _synManagementClient;\n private readonly PipelineClient _pipelineClient;\n private readonly PipelineRunClient _pipelineRunClient;\n private readonly ILogger _logger;\n \n public AzureSynapseService(PipelineRequest request, ILogger logger)\n {\n _logger = logger;\n _logger.LogInformation(\"Creating SYN connectivity clients.\");\n\n //Auth details\n var context = new AuthenticationContext(\"https://login.windows.net/\" + request.TenantId);\n var cc = new ClientCredential(request.ApplicationId, request.AuthenticationKey);\n var result = context.AcquireTokenAsync(\"https://management.azure.com/\", cc).Result;\n var cred = new TokenCredentials(result.AccessToken);\n\n //Management Client\n _synManagementClient = new SynapseManagementClient(cred)\n {\n SubscriptionId = request.SubscriptionId\n };\n\n //Pipeline Clients\n Uri synapseDevEndpoint = new Uri(\"https://\" + request.OrchestratorName.ToLower() + \".dev.azuresynapse.net\");\n TokenCredential token = new ClientSecretCredential\n (\n request.TenantId,\n request.ApplicationId,\n request.AuthenticationKey\n );\n\n _pipelineClient = new PipelineClient(synapseDevEndpoint, token);\n _pipelineRunClient = new PipelineRunClient(synapseDevEndpoint, token);\n }\n\n public override PipelineDescription ValidatePipeline(PipelineRequest request)\n {\n _logger.LogInformation(\"Validating SYN pipeline.\");\n\n PipelineResource pipelineResource;\n\n try\n {\n pipelineResource = _pipelineClient.GetPipeline\n (\n request.PipelineName\n );\n\n _logger.LogInformation(pipelineResource.Id.ToString());\n\n return new PipelineDescription()\n {\n PipelineExists = \"True\",\n PipelineName = pipelineResource.Name,\n PipelineId = pipelineResource.Id,\n PipelineType = pipelineResource.Type,\n ActivityCount = pipelineResource.Activities.Count\n };\n }\n catch (System.InvalidOperationException) //for bug in underlying activity classes, pipeline does exist\n {\n return new PipelineDescription()\n {\n PipelineExists = \"True\",\n PipelineName = request.PipelineName,\n PipelineId = \"Unknown\",\n PipelineType = \"Unknown\",\n ActivityCount = 0\n }; \n }\n catch (Azure.RequestFailedException) //expected exception when pipeline doesnt exist\n {\n return new PipelineDescription()\n {\n PipelineExists = \"False\",\n PipelineName = request.PipelineName,\n PipelineId = \"Unknown\",\n PipelineType = \"Unknown\",\n ActivityCount = 0\n };\n }\n catch (Exception ex) //other unknown issue\n {\n _logger.LogInformation(ex.Message);\n _logger.LogInformation(ex.GetType().ToString());\n throw new InvalidRequestException(\"Failed to validate pipeline. \", ex);\n }\n }\n\n public override PipelineRunStatus ExecutePipeline(PipelineRequest request)\n {\n if (request.PipelineParameters == null)\n _logger.LogInformation(\"Calling pipeline without parameters.\");\n else\n _logger.LogInformation(\"Calling pipeline with parameters.\");\n\n CreateRunResponse runResponse;\n runResponse = _pipelineClient.CreatePipelineRun\n (\n request.PipelineName,\n parameters: request.ParametersAsObjects\n );\n\n _logger.LogInformation(\"Pipeline run ID: \" + runResponse.RunId);\n\n //Wait and check for pipeline to start...\n PipelineRun pipelineRun;\n _logger.LogInformation(\"Checking ADF pipeline status.\");\n while (true)\n {\n pipelineRun = _pipelineRunClient.GetPipelineRun\n (\n runResponse.RunId\n );\n\n _logger.LogInformation(\"Waiting for pipeline to start, current status: \" + pipelineRun.Status);\n\n if (pipelineRun.Status != \"Queued\")\n break;\n Thread.Sleep(internalWaitDuration);\n }\n\n return new PipelineRunStatus()\n {\n PipelineName = request.PipelineName,\n RunId = runResponse.RunId,\n ActualStatus = pipelineRun.Status\n };\n }\n\n public override PipelineRunStatus CancelPipeline(PipelineRunRequest request)\n {\n _logger.LogInformation(\"Getting SYN pipeline current status.\");\n\n PipelineRun pipelineRun;\n pipelineRun = _pipelineRunClient.GetPipelineRun\n (\n request.RunId\n );\n\n //Defensive check\n PipelineNameCheck(request.PipelineName, pipelineRun.PipelineName);\n\n if (pipelineRun.Status == \"InProgress\" || pipelineRun.Status == \"Queued\")\n {\n _logger.LogInformation(\"Attempting to cancel SYN pipeline.\");\n _pipelineRunClient.CancelPipelineRun\n (\n request.RunId,\n isRecursive: request.RecursivePipelineCancel\n );\n }\n else\n {\n _logger.LogInformation(\"ADF pipeline status: \" + pipelineRun.Status);\n throw new InvalidRequestException(\"Target pipeline is not in a state that can be cancelled.\");\n }\n\n //wait for cancelled state\n _logger.LogInformation(\"Checking ADF pipeline status after cancel request.\");\n while (true)\n {\n pipelineRun = _pipelineRunClient.GetPipelineRun\n (\n request.RunId\n );\n\n _logger.LogInformation(\"Waiting for pipeline to cancel, current status: \" + pipelineRun.Status);\n\n if (pipelineRun.Status == \"Cancelled\")\n break;\n Thread.Sleep(internalWaitDuration);\n }\n\n //Final return detail\n return new PipelineRunStatus()\n {\n PipelineName = request.PipelineName,\n RunId = request.RunId,\n ActualStatus = pipelineRun.Status\n };\n }\n\n public override PipelineRunStatus GetPipelineRunStatus(PipelineRunRequest request)\n {\n _logger.LogInformation(\"Getting SYN pipeline status.\");\n\n //Get pipeline status with provided run id\n PipelineRun pipelineRun;\n pipelineRun = _pipelineRunClient.GetPipelineRun\n (\n request.RunId\n );\n\n //Defensive check\n PipelineNameCheck(request.PipelineName, pipelineRun.PipelineName);\n\n _logger.LogInformation(\"SYN pipeline status: \" + pipelineRun.Status);\n\n //Final return detail\n return new PipelineRunStatus()\n {\n PipelineName = request.PipelineName,\n RunId = pipelineRun.RunId,\n ActualStatus = pipelineRun.Status.Replace(\"Canceling\", \"Cancelling\") //microsoft typo\n };\n }\n\n public override PipelineErrorDetail GetPipelineRunActivityErrors(PipelineRunRequest request)\n {\n PipelineRun pipelineRun = _pipelineRunClient.GetPipelineRun\n (\n request.RunId\n );\n\n //Defensive check\n PipelineNameCheck(request.PipelineName, pipelineRun.PipelineName);\n\n _logger.LogInformation(\"Create pipeline Activity Runs query filters.\");\n RunFilterParameters filterParams = new RunFilterParameters\n (\n request.ActivityQueryStart,\n request.ActivityQueryEnd\n );\n\n _logger.LogInformation(\"Querying SYN pipeline for Activity Runs.\");\n ActivityRunsQueryResponse queryResponse = _pipelineRunClient.QueryActivityRuns\n (\n request.PipelineName,\n request.RunId,\n filterParams\n );\n\n //Create initial output content\n PipelineErrorDetail output = new PipelineErrorDetail()\n {\n PipelineName = request.PipelineName,\n ActualStatus = pipelineRun.Status,\n RunId = request.RunId,\n ResponseCount = queryResponse.Value.Count\n };\n\n _logger.LogInformation(\"Pipeline status: \" + pipelineRun.Status);\n _logger.LogInformation(\"Activities found in pipeline response: \" + queryResponse.Value.Count.ToString());\n\n //Loop over activities in pipeline run\n foreach (ActivityRun activity in queryResponse.Value)\n {\n if (activity.Error == null)\n {\n continue; //only want errors\n }\n\n //Parse error output to customise output\n var json = JsonConvert.SerializeObject(activity.Error);\n Dictionary errorContent = JsonConvert.DeserializeObject>(json);\n\n _logger.LogInformation(\"Activity run id: \" + activity.ActivityRunId);\n _logger.LogInformation(\"Activity name: \" + activity.ActivityName);\n _logger.LogInformation(\"Activity type: \" + activity.ActivityType);\n _logger.LogInformation(\"Error message: \" + errorContent[\"message\"].ToString());\n\n output.Errors.Add(new FailedActivity()\n {\n ActivityRunId = activity.ActivityRunId,\n ActivityName = activity.ActivityName,\n ActivityType = activity.ActivityType,\n ErrorCode = errorContent[\"errorCode\"].ToString(),\n ErrorType = errorContent[\"failureType\"].ToString(),\n ErrorMessage = errorContent[\"message\"].ToString()\n });\n }\n return output;\n }\n\n public override void Dispose()\n {\n _synManagementClient?.Dispose();\n //_pipelineClient?.Dispose(); not yet supported\n //_pipelineRunClient?.Dispose(); not yet supported\n }\n }\n}\n"
},
{
"path": "Functions/Services/PipelineService.cs",
"content": "using mrpaulandrew.azure.procfwk.Helpers;\nusing Microsoft.Extensions.Logging;\nusing System;\n\nnamespace mrpaulandrew.azure.procfwk.Services\n{\n public abstract class PipelineService : IDisposable\n {\n public const int internalWaitDuration = 5000; //ms\n\n public static PipelineService GetServiceForRequest(PipelineRequest pr, ILogger logger)\n {\n if (pr.OrchestratorType == PipelineServiceType.ADF)\n return new AzureDataFactoryService(pr, logger);\n\n if (pr.OrchestratorType == PipelineServiceType.SYN)\n return new AzureSynapseService(pr, logger);\n\n throw new InvalidRequestException (\"Unsupported orchestrator type: \" + (pr.OrchestratorType?.ToString() ?? \"\"));\n }\n\n public abstract PipelineDescription ValidatePipeline(PipelineRequest request);\n\n public abstract PipelineRunStatus ExecutePipeline(PipelineRequest request);\n\n public abstract PipelineRunStatus CancelPipeline(PipelineRunRequest request);\n\n public abstract PipelineRunStatus GetPipelineRunStatus(PipelineRunRequest request);\n\n public abstract PipelineErrorDetail GetPipelineRunActivityErrors(PipelineRunRequest request);\n\n protected void PipelineNameCheck(string requestName, string foundName)\n {\n if (requestName.ToUpper() != foundName.ToUpper())\n {\n throw new InvalidRequestException($\"Pipeline name mismatch. Provided pipeline name does not match the provided Run Id. Expected name: {foundName}\");\n }\n }\n\n public abstract void Dispose();\n }\n}"
},
{
"path": "Functions/Services/PipelineServiceType.cs",
"content": "namespace mrpaulandrew.azure.procfwk\n{\n public enum PipelineServiceType\n {\n ADF, SYN\n }\n}"
},
{
"path": "Functions/Services/Returns/PipelineDescription.cs",
"content": "namespace mrpaulandrew.azure.procfwk.Services\n{\n public class PipelineDescription\n {\n public string PipelineExists { get; set; }\n public string PipelineName { get; set; }\n public string PipelineId { get; set; }\n public string PipelineType { get; set; }\n public int ActivityCount { get; set; }\n }\n}"
},
{
"path": "Functions/Services/Returns/PipelineErrorDetail.cs",
"content": "using System.Collections.Generic;\n\nnamespace mrpaulandrew.azure.procfwk.Services\n{\n public class PipelineErrorDetail : PipelineRunStatus\n {\n public PipelineErrorDetail()\n {\n Errors = new List();\n }\n public int ResponseCount { get; set; }\n public int ResponseErrorCount\n {\n get\n {\n return Errors.Count;\n }\n }\n public List Errors { get; }\n }\n\n public class FailedActivity\n {\n public string ActivityRunId { get; internal set; }\n public string ActivityName { get; internal set; }\n public string ActivityType { get; internal set; }\n public string ErrorCode { get; internal set; }\n public string ErrorType { get; internal set; }\n public string ErrorMessage { get; internal set; }\n }\n}\n"
},
{
"path": "Functions/Services/Returns/PipelineRunStatus.cs",
"content": "using System.Collections.Generic;\n\nnamespace mrpaulandrew.azure.procfwk.Services\n{\n public class PipelineRunStatus\n {\n private const string Running = \"Running\";\n private const string Complete = \"Complete\";\n\n public string PipelineName { get; set; }\n public string RunId { get; set; }\n public string ActualStatus { get; set; }\n\n public string SimpleStatus \n {\n get\n {\n return ConvertPipelineStatus(ActualStatus);\n }\n }\n\n private string ConvertPipelineStatus(string actualStatus)\n {\n string simpleStatus = actualStatus switch\n {\n \"Queued\" => Running,\n \"InProgress\" => Running,\n \"Canceling\" => Running, //microsoft typo\n \"Cancelling\" => Running,\n _ => Complete,\n };\n return simpleStatus;\n }\n }\n}"
},
{
"path": "Functions/host.json",
"content": "{\n \"version\": \"2.0\"\n}"
},
{
"path": "Functions/template_local.settings.json",
"content": "{\n \"IsEncrypted\": false,\n \"Values\": {\n \"AzureWebJobsStorage\": \"UseDevelopmentStorage=true\",\n \"FUNCTIONS_WORKER_RUNTIME\": \"dotnet\",\n \"AppSettingSmtpHost\": \"my.smtpservice.com\",\n \"AppSettingSmtpPort\": 1234,\n \"AppSettingSmtpUser\": \"AlertMailboxUser\",\n \"AppSettingSmtpPass\": \"Passw0rd123!\",\n \"AppSettingFromEmail\": \"noreply@adfprocfwk.com\"\n }\n}"
},
{
"path": "LICENSE",
"content": "MIT License\n\nCopyright (c) 2020 Paul Andrew (mrpaulandrew.com)\n\nPermission is hereby granted, free of charge, to any person obtaining a copy\nof this software and associated documentation files (the \"Software\"), to deal\nin the Software without restriction, including without limitation the rights\nto use, copy, modify, merge, publish, distribute, sublicense, and/or sell\ncopies of the Software, and to permit persons to whom the Software is\nfurnished to do so, subject to the following conditions:\n\nThe above copyright notice and this permission notice shall be included in all\ncopies or substantial portions of the Software.\n\nTHE SOFTWARE IS PROVIDED \"AS IS\", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR\nIMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,\nFITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE\nAUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER\nLIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,\nOUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE\nSOFTWARE.\n\nTHE SOFTWARE IS ALSO IN NO WAY ASOCIATED WITH OR SUPPORTED BY:\n- Altius Consulting Limited\n- Cloudpoint Limited\n- Farah Bidco Limited\n- Farah Midco Limited\n- Farah Topco Limited\n- Avanade UK Ltd\n- Avanade Europe Services Ltd\n- Avanade Europe Holdings Ltd\n- Avanade International Corporation"
},
{
"path": "MetadataDB/MetadataDB.refactorlog",
"content": "\n\n \n \n \n \n \n \n \n"
},
{
"path": "MetadataDB/MetadataDB.sqlproj",
"content": "\n\n \n Debug\n AnyCPU\n MetadataDB\n 2.0\n 4.1\n {202ebf84-a56b-4999-92a3-10f7ffe4ef25}\n Microsoft.Data.Tools.Schema.Sql.SqlAzureV12DatabaseSchemaProvider\n Database\n \n \n MetadataDB\n MetadataDB\n 1033,CI\n BySchemaAndSchemaType\n True\n v4.7.2\n CS\n Properties\n False\n True\n True\n SQL_Latin1_General_CP1_CI_AS\n PRIMARY\n procfwk\n \n \n \n bin\\Release\\\n $(MSBuildProjectName).sql\n False\n pdbonly\n true\n false\n true\n prompt\n 4\n \n \n bin\\Debug\\\n $(MSBuildProjectName).sql\n false\n true\n full\n false\n true\n true\n prompt\n 4\n \n \n 11.0\n \n True\n 11.0\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n $(DacPacRootPath)\\Extensions\\Microsoft\\SQLDB\\Extensions\\SqlServer\\AzureV12\\SqlSchemas\\master.dacpac\n False\n master\n \n \n \n \n \n"
},
{
"path": "MetadataDB/Scripts/Alter Database Scale.sql",
"content": "ALTER DATABASE [FrameworkMetadataDev] MODIFY (EDITION ='Standard', SERVICE_OBJECTIVE = 'S2', MAXSIZE = 50 GB);"
},
{
"path": "MetadataDB/Scripts/Handy Selects.sql",
"content": "SELECT * FROM [procfwk].[BatchExecution]\n\nSELECT * FROM [procfwk].[CurrentExecution]\n\nSELECT * FROM [procfwkReporting].[CurrentExecutionSummary]\n\nSELECT * FROM [procfwk].[CurrentProperties]\n\nSELECT * FROM [procfwk].[Batches]\n\nSELECT * FROM [procfwk].[BatchStageLink]\n\nSELECT * FROM [procfwk].[Orchestrators]\n\nSELECT * FROM [procfwk].[Stages]\n\nSELECT * FROM [procfwk].[Pipelines]\n\n--SELECT * FROM [procfwk].[PipelineParameters]\n\nSELECT * FROM [procfwk].[ExecutionLog]\n\nSELECT * FROM [procfwk].[ErrorLog]\n\n--SELECT * FROM [dbo].[ServicePrincipals]"
},
{
"path": "MetadataDB/Scripts/LogData/ErrorLogBackup.sql",
"content": "IF OBJECT_ID(N'[dbo].[ErrorLogBackup]') IS NOT NULL DROP TABLE [dbo].[ErrorLogBackup];\n\nIF OBJECT_ID(N'[procfwk].[ErrorLog]') IS NOT NULL --check for new deployments\nBEGIN\n\tSELECT \n\t\t*\n\tINTO\n\t\t[dbo].[ErrorLogBackup]\n\tFROM\n\t\t[procfwk].[ErrorLog];\nEND;"
},
{
"path": "MetadataDB/Scripts/LogData/ErrorLogRestore.sql",
"content": "DECLARE @ErrColumns VARCHAR(MAX) = '';\nDECLARE @ErrValues VARCHAR(MAX) = '';\nDECLARE @ErrSQL VARCHAR(MAX) = '';\n\nIF EXISTS\n\t(\n\tSELECT\n\t\t*\n\tFROM\n\t\tsys.objects o\n\t\tINNER JOIN sys.schemas s\n\t\t\tON o.[schema_id] = s.[schema_id]\n\tWHERE\n\t\to.[name] = 'ErrorLog'\n\t\tAND s.[name] = 'procfwk'\n\t)\n\tAND EXISTS\n\t(\n\tSELECT\n\t\t*\n\tFROM\n\t\tsys.objects o\n\t\tINNER JOIN sys.schemas s\n\t\t\tON o.[schema_id] = s.[schema_id]\n\tWHERE\n\t\to.[name] = 'ErrorLogBackup'\n\t\tAND s.[name] = 'dbo'\n\t)\n\tBEGIN\n\t\t;WITH oldTableColumns AS\n\t\t\t(\n\t\t\tSELECT\n\t\t\t\tc.[name] AS 'ColName'\n\t\t\tFROM\n\t\t\t\tsys.objects o\n\t\t\t\tINNER JOIN sys.schemas s\n\t\t\t\t\tON o.[schema_id] = s.[schema_id]\n\t\t\t\tINNER JOIN sys.columns c\n\t\t\t\t\tON o.[object_id] = c.[object_id]\n\t\t\tWHERE\n\t\t\t\ts.[name] = 'dbo'\n\t\t\t\tAND o.[name] = 'ErrorLogBackup'\n\t\t\t\tAND c.[name] != 'LogId'\n\t\t\t),\n\t\t\tnewTableColumns AS\n\t\t\t(\n\t\t\tSELECT\n\t\t\t\tc.[column_id] AS 'ColId',\n\t\t\t\tc.[name] AS 'ColName'\n\t\t\tFROM\n\t\t\t\tsys.objects o\n\t\t\t\tINNER JOIN sys.schemas s\n\t\t\t\t\tON o.[schema_id] = s.[schema_id]\n\t\t\t\tINNER JOIN sys.columns c\n\t\t\t\t\tON o.[object_id] = c.[object_id]\n\t\t\tWHERE\n\t\t\t\ts.[name] = 'procfwk'\n\t\t\t\tAND o.[name] = 'ErrorLog'\n\t\t\t\tAND c.[name] != 'LogId'\n\t\t\t)\n\t\tSELECT \n\t\t\t@ErrColumns += QUOTENAME(newTableColumns.[ColName]) + ',' + CHAR(13),\n\t\t\t@ErrValues += ISNULL(QUOTENAME(oldTableColumns.[ColName]),'NULL AS ''' + newTableColumns.[ColName] + '''' ) + ',' + CHAR(13)\n\t\tFROM\n\t\t\tnewTableColumns \n\t\t\tLEFT OUTER JOIN oldTableColumns\n\t\t\t\tON newTableColumns.[ColName] = oldTableColumns.[ColName];\n\n\t\tSET @ErrColumns = LEFT(@ErrColumns,LEN(@ErrColumns)-2);\n\t\tSET @ErrValues = LEFT(@ErrValues,LEN(@ErrValues)-2);\n\n\t\tSET @ErrSQL = \n\t\t'\n\t\tINSERT INTO [procfwk].[ErrorLog]\n\t\t(\n\t\t' + @ErrColumns + '\n\t\t)\n\t\tSELECT\n\t\t' + @ErrValues + '\n\t\tFROM\n\t\t\t[dbo].[ErrorLogBackup]\n\t\t';\n\n\t\tPRINT @ErrSQL;\n\t\tEXEC(@ErrSQL);\n\n\t\tDECLARE @ErrBefore INT = (SELECT COUNT(0) FROM [dbo].[ErrorLogBackup]);\n\t\tDECLARE @ErrAfter INT = (SELECT COUNT(0) FROM [procfwk].[ErrorLog]);\n\n\t\tIF (@ErrBefore = @ErrAfter)\n\t\tBEGIN\n\t\t\tDROP TABLE [dbo].[ErrorLogBackup]\n\t\tEND;\n\tEND"
},
{
"path": "MetadataDB/Scripts/LogData/ExecutionLogBackup.sql",
"content": "IF OBJECT_ID(N'[dbo].[ExecutionLogBackup]') IS NOT NULL DROP TABLE [dbo].[ExecutionLogBackup];\n\nIF OBJECT_ID(N'[procfwk].[ExecutionLog]') IS NOT NULL --check for new deployments\nBEGIN\n\tSELECT \n\t\t*\n\tINTO\n\t\t[dbo].[ExecutionLogBackup]\n\tFROM\n\t\t[procfwk].[ExecutionLog];\nEND;"
},
{
"path": "MetadataDB/Scripts/LogData/ExecutionLogRestore.sql",
"content": "DECLARE @Columns VARCHAR(MAX) = '';\nDECLARE @Values VARCHAR(MAX) = '';\nDECLARE @SQL VARCHAR(MAX) = '';\n\nIF EXISTS\n\t(\n\tSELECT\n\t\t*\n\tFROM\n\t\tsys.objects o\n\t\tINNER JOIN sys.schemas s\n\t\t\tON o.[schema_id] = s.[schema_id]\n\tWHERE\n\t\to.[name] = 'ExecutionLog'\n\t\tAND s.[name] = 'procfwk'\n\t)\n\tAND EXISTS\n\t(\n\tSELECT\n\t\t*\n\tFROM\n\t\tsys.objects o\n\t\tINNER JOIN sys.schemas s\n\t\t\tON o.[schema_id] = s.[schema_id]\n\tWHERE\n\t\to.[name] = 'ExecutionLogBackup'\n\t\tAND s.[name] = 'dbo'\n\t)\n\tBEGIN\n\t\t;WITH oldTableColumns AS\n\t\t\t(\n\t\t\tSELECT\n\t\t\t\tc.[name] AS ColName\n\t\t\tFROM\n\t\t\t\tsys.objects o\n\t\t\t\tINNER JOIN sys.schemas s\n\t\t\t\t\tON o.[schema_id] = s.[schema_id]\n\t\t\t\tINNER JOIN sys.columns c\n\t\t\t\t\tON o.[object_id] = c.[object_id]\n\t\t\tWHERE\n\t\t\t\ts.[name] = 'dbo'\n\t\t\t\tAND o.[name] = 'ExecutionLogBackup'\n\t\t\t\tAND c.[name] <> 'LogId'\n\t\t\t),\n\t\t\tnewTableColumns AS\n\t\t\t(\n\t\t\tSELECT\n\t\t\t\tc.[column_id] AS ColId,\n\t\t\t\tc.[name] AS ColName\n\t\t\tFROM\n\t\t\t\tsys.objects o\n\t\t\t\tINNER JOIN sys.schemas s\n\t\t\t\t\tON o.[schema_id] = s.[schema_id]\n\t\t\t\tINNER JOIN sys.columns c\n\t\t\t\t\tON o.[object_id] = c.[object_id]\n\t\t\tWHERE\n\t\t\t\ts.[name] = 'procfwk'\n\t\t\t\tAND o.[name] = 'ExecutionLog'\n\t\t\t\tAND c.[name] <> 'LogId'\n\t\t\t)\n\t\tSELECT \n\t\t\t@Columns += QUOTENAME(newTableColumns.[ColName]) + ',' + CHAR(13),\n\t\t\t@Values += ISNULL(QUOTENAME(oldTableColumns.[ColName]),'NULL AS ''' + newTableColumns.[ColName] + '''' ) + ',' + CHAR(13)\n\t\tFROM\n\t\t\tnewTableColumns \n\t\t\tLEFT OUTER JOIN oldTableColumns\n\t\t\t\tON newTableColumns.[ColName] = oldTableColumns.[ColName];\n\n\t\tSET @Columns = LEFT(@Columns,LEN(@Columns)-2);\n\t\tSET @Values = LEFT(@Values,LEN(@Values)-2);\n\n\t\tSET @SQL = \n\t\t'\n\t\tINSERT INTO [procfwk].[ExecutionLog]\n\t\t(\n\t\t' + @Columns + '\n\t\t)\n\t\tSELECT\n\t\t' + @Values + '\n\t\tFROM\n\t\t\t[dbo].[ExecutionLogBackup]\n\t\t';\n\n\t\tPRINT @SQL;\n\t\tEXEC(@SQL);\n\n\t\tDECLARE @Before INT = (SELECT COUNT(0) FROM [dbo].[ExecutionLogBackup]);\n\t\tDECLARE @After INT = (SELECT COUNT(0) FROM [procfwk].[ExecutionLog]);\n\n\t\tIF (@Before = @After)\n\t\tBEGIN\n\t\t\tDROP TABLE [dbo].[ExecutionLogBackup]\n\t\tEND;\n\tEND;"
},
{
"path": "MetadataDB/Scripts/Metadata/AlertOutcomes.sql",
"content": "EXEC [procfwkHelpers].[SetDefaultAlertOutcomes];"
},
{
"path": "MetadataDB/Scripts/Metadata/DeleteAll.sql",
"content": "\t--BatchExecution\n\tIF OBJECT_ID(N'[procfwk].[BatchExecution]') IS NOT NULL \n\t\tBEGIN\n\t\t\tTRUNCATE TABLE [procfwk].[BatchExecution];\n\t\tEND;\n\n\t--CurrentExecution\n\tIF OBJECT_ID(N'[procfwk].[CurrentExecution]') IS NOT NULL \n\t\tBEGIN\n\t\t\tTRUNCATE TABLE [procfwk].[CurrentExecution];\n\t\tEND;\n\n\t--ExecutionLog\n\tIF OBJECT_ID(N'[procfwk].[ExecutionLog]') IS NOT NULL \n\t\tBEGIN\n\t\t\tTRUNCATE TABLE [procfwk].[ExecutionLog];\n\t\tEND\n\n\t--ErrorLog\n\tIF OBJECT_ID(N'[procfwk].[ExecutionLog]') IS NOT NULL \n\t\tBEGIN\n\t\t\tTRUNCATE TABLE [procfwk].[ErrorLog];\n\t\tEND\n\n\t--BatchStageLink\n\tIF OBJECT_ID(N'[procfwk].[BatchStageLink]') IS NOT NULL \n\t\tBEGIN\n\t\t\tDELETE FROM [procfwk].[BatchStageLink];\n\t\tEND;\n\n\t--Batches\n\tIF OBJECT_ID(N'[procfwk].[Batches]') IS NOT NULL \n\t\tBEGIN\n\t\t\tDELETE FROM [procfwk].[Batches];\n\t\tEND;\n\n\t--PipelineDependencies\n\tIF OBJECT_ID(N'[procfwk].[PipelineDependencies]') IS NOT NULL \n\t\tBEGIN\n\t\t\tDELETE FROM [procfwk].[PipelineDependencies];\n\t\t\tDBCC CHECKIDENT ('[procfwk].[PipelineDependencies]', RESEED, 0);\n\t\tEND;\n\n\t--PipelineAlertLink\n\tIF OBJECT_ID(N'[procfwk].[PipelineAlertLink]') IS NOT NULL \n\t\tBEGIN\n\t\t\tDELETE FROM [procfwk].[PipelineAlertLink];\n\t\t\tDBCC CHECKIDENT ('[procfwk].[PipelineAlertLink]', RESEED, 0);\n\t\tEND;\n\n\t--Recipients\n\tIF OBJECT_ID(N'[procfwk].[Recipients]') IS NOT NULL \n\t\tBEGIN\n\t\t\tDELETE FROM [procfwk].[Recipients];\n\t\t\tDBCC CHECKIDENT ('[procfwk].[Recipients]', RESEED, 0);\n\t\tEND;\n\n\t--AlertOutcomes\n\tIF OBJECT_ID(N'[procfwk].[AlertOutcomes]') IS NOT NULL \n\t\tBEGIN\n\t\t\tTRUNCATE TABLE [procfwk].[AlertOutcomes];\n\t\tEND;\n\n\t--PipelineAuthLink\n\tIF OBJECT_ID(N'[procfwk].[PipelineAuthLink]') IS NOT NULL \n\t\tBEGIN\n\t\t\tDELETE FROM [procfwk].[PipelineAuthLink];\n\t\t\tDBCC CHECKIDENT ('[procfwk].[PipelineAuthLink]', RESEED, 0);\n\t\tEND;\n\n\t--ServicePrincipals\n\tIF OBJECT_ID(N'[dbo].[ServicePrincipals]') IS NOT NULL \n\t\tBEGIN\n\t\t\tDELETE FROM [dbo].[ServicePrincipals];\n\t\t\tDBCC CHECKIDENT ('[dbo].[ServicePrincipals]', RESEED, 0);\n\t\tEND;\n\n\t--Properties\n\tIF OBJECT_ID(N'[procfwk].[Properties]') IS NOT NULL \n\t\tBEGIN\n\t\t\tDELETE FROM [procfwk].[Properties];\n\t\t\tDBCC CHECKIDENT ('[procfwk].[Properties]', RESEED, 0);\n\t\tEND;\n\n\t--PipelineParameters\n\tIF OBJECT_ID(N'[procfwk].[PipelineParameters]') IS NOT NULL \n\t\tBEGIN\n\t\t\tDELETE FROM [procfwk].[PipelineParameters];\n\t\t\tDBCC CHECKIDENT ('[procfwk].[PipelineParameters]', RESEED, 0);\n\t\tEND;\n\n\t--Pipelines\n\tIF OBJECT_ID(N'[procfwk].[Pipelines]') IS NOT NULL \n\t\tBEGIN\n\t\t\tDELETE FROM [procfwk].[Pipelines];\n\t\t\tDBCC CHECKIDENT ('[procfwk].[Pipelines]', RESEED, 0);\n\t\tEND;\n\n\t--Orchestrators\n\tIF EXISTS \n\t\t(\n\t\tSELECT\n\t\t\t* \n\t\tFROM\n\t\t\tsys.objects o\n\t\t\tINNER JOIN sys.schemas s\n\t\t\t\tON o.[schema_id] = s.[schema_id]\n\t\tWHERE\n\t\t\to.[name] = 'DataFactorys'\n\t\t\tAND s.[name] = 'procfwk'\n\t\t\tAND o.[type] = 'U' --Check for tables as created synonyms to support backwards compatability\n\t\t)\n\t\tBEGIN\n\t\t\tDELETE FROM [procfwk].[DataFactorys];\n\t\tEND;\n\n\tIF OBJECT_ID(N'[procfwk].[Orchestrators]') IS NOT NULL \n\t\tBEGIN\n\t\t\tDELETE FROM [procfwk].[Orchestrators];\n\t\t\tDBCC CHECKIDENT ('[procfwk].[Orchestrators]', RESEED, 0);\n\t\tEND;\n\n\t--Stages\n\tIF OBJECT_ID(N'[procfwk].[Stages]') IS NOT NULL \n\t\tBEGIN\n\t\t\tDELETE FROM [procfwk].[Stages];\n\t\t\tDBCC CHECKIDENT ('[procfwk].[Stages]', RESEED, 0);\n\t\tEND;\n\n\t--Subscriptions\n\tIF OBJECT_ID(N'[procfwk].[Subscriptions]') IS NOT NULL \n\t\tBEGIN\n\t\t\tDELETE FROM [procfwk].[Subscriptions];\n\t\tEND;\n\t\n\t--Tenants\n\tIF OBJECT_ID(N'[procfwk].[Tenants]') IS NOT NULL \n\t\tBEGIN\n\t\t\tDELETE FROM [procfwk].[Tenants];\n\t\tEND;"
},
{
"path": "MetadataDB/Scripts/Metadata/DropLegacyObjects.sql",
"content": "IF OBJECT_ID(N'tempdb..#DropLegacyObjects') IS NOT NULL DROP PROCEDURE #DropLegacyObjects;\nGO\n\nCREATE PROCEDURE #DropLegacyObjects\n\t(\n\t@ObjectName NVARCHAR(128),\n\t@ObjectType CHAR(2)\n\t)\nAS\nBEGIN\n\tDECLARE @DDLType NVARCHAR(128)\n\n\tIF EXISTS\n\t\t(\n\t\tSELECT\n\t\t\t*\n\t\tFROM\n\t\t\tsys.objects o\n\t\t\tINNER JOIN sys.schemas s\n\t\t\t\tON o.[schema_id] = s.[schema_id]\n\t\tWHERE\n\t\t\to.[Name] = @ObjectName\n\t\t\tAND s.[name] = 'procfwk'\n\t\t\tAND o.[type] = @ObjectType\n\t\t)\n\t\tBEGIN\t\t\t\n\t\t\tSELECT\n\t\t\t\t@DDLType = CASE @ObjectType\n\t\t\t\t\t\t\t\tWHEN 'P' THEN 'PROCEDURE'\n\t\t\t\t\t\t\t\tWHEN 'V' THEN 'VIEW'\n\t\t\t\t\t\t\t\tWHEN 'FN' THEN 'FUNCTION'\n\t\t\t\t\t\t\tEND;\n\n\t\t\tEXEC('DROP ' + @DDLType + ' [procfwk].[' + @ObjectName + '];')\n\t\tEND;\nEND;\nGO\n\nEXEC #DropLegacyObjects 'AddProperty', 'P';\nEXEC #DropLegacyObjects 'GetExecutionDetails', 'P';\nEXEC #DropLegacyObjects 'AddRecipientPipelineAlerts', 'P';\nEXEC #DropLegacyObjects 'DeleteRecipientAlerts', 'P';\nEXEC #DropLegacyObjects 'CheckStageAndPiplineIntegrity', 'P';\nEXEC #DropLegacyObjects 'AddPipelineDependant', 'P';\nEXEC #DropLegacyObjects 'AddServicePrincipalWrapper', 'P';\nEXEC #DropLegacyObjects 'AddServicePrincipalUrls', 'P';\nEXEC #DropLegacyObjects 'AddServicePrincipal', 'P';\nEXEC #DropLegacyObjects 'DeleteServicePrincipal', 'P';\nEXEC #DropLegacyObjects 'CheckForValidURL', 'FN';\nEXEC #DropLegacyObjects 'PipelineDependencyChains', 'V';\nEXEC #DropLegacyObjects 'AverageStageDuration', 'V';\nEXEC #DropLegacyObjects 'CompleteExecutionErrorLog', 'V';\nEXEC #DropLegacyObjects 'CompleteExecutionLog', 'V';\nEXEC #DropLegacyObjects 'CurrentExecutionSummary', 'V';\nEXEC #DropLegacyObjects 'LastExecution', 'V';\nEXEC #DropLegacyObjects 'LastExecutionSummary', 'V';\nEXEC #DropLegacyObjects 'WorkerParallelismOverTime', 'V';\n\n--replaced with new precursor concept in v1.8.5:\nIF OBJECT_ID(N'[dbo].[SetRandomWaitValues]') IS NOT NULL DROP PROCEDURE [dbo].[SetRandomWaitValues];"
},
{
"path": "MetadataDB/Scripts/Metadata/DropLegacyTables.sql",
"content": "--PipelineProcesses\nIF EXISTS \n\t(\n\tSELECT\n\t\t* \n\tFROM\n\t\tsys.objects o\n\t\tINNER JOIN sys.schemas s\n\t\t\tON o.[schema_id] = s.[schema_id]\n\tWHERE\n\t\to.[name] = 'PipelineProcesses'\n\t\tAND s.[name] = 'procfwk'\n\t\tAND o.[type] = 'U' --Check for tables as created synonyms to support backwards compatability\n\t)\n\tBEGIN\n\t\t--drop just to avoid constraints\n\t\tIF OBJECT_ID(N'[procfwk].[PipelineParameters]') IS NOT NULL DROP TABLE [procfwk].[PipelineParameters];\n\t\tIF OBJECT_ID(N'[procfwk].[PipelineAuthLink]') IS NOT NULL DROP TABLE [procfwk].[PipelineAuthLink];\n\n\t\tSELECT * INTO [dbo].[zz_PipelineProcesses] FROM [procfwk].[PipelineProcesses];\n\n\t\tDROP TABLE [procfwk].[PipelineProcesses];\n\tEND\n\n--ProcessingStageDetails\nIF EXISTS \n\t(\n\tSELECT\n\t\t* \n\tFROM\n\t\tsys.objects o\n\t\tINNER JOIN sys.schemas s\n\t\t\tON o.[schema_id] = s.[schema_id]\n\tWHERE\n\t\to.[name] = 'ProcessingStageDetails'\n\t\tAND s.[name] = 'procfwk'\n\t\tAND o.[type] = 'U' --Check for tables as created synonyms to support backwards compatability\n\t)\n\tBEGIN\n\t\tSELECT * INTO [dbo].[zz_ProcessingStageDetails] FROM [procfwk].[ProcessingStageDetails];\n\t\t\n\t\tDROP TABLE [procfwk].[ProcessingStageDetails];\n\tEND;\n\n--DataFactoryDetails\nIF EXISTS \n\t(\n\tSELECT\n\t\t* \n\tFROM\n\t\tsys.objects o\n\t\tINNER JOIN sys.schemas s\n\t\t\tON o.[schema_id] = s.[schema_id]\n\tWHERE\n\t\to.[name] = 'DataFactoryDetails'\n\t\tAND s.[name] = 'procfwk'\n\t\tAND o.[type] = 'U' --Check for tables as created synonyms to support backwards compatability\n\t)\n\tBEGIN\n\t\tSELECT * INTO [dbo].[zz_DataFactoryDetails] FROM [procfwk].[DataFactoryDetails];\n\t\t\n\t\tDROP TABLE [procfwk].[DataFactoryDetails];\n\tEND;\n\n--DataFactorys\nIF EXISTS \n\t(\n\tSELECT\n\t\t* \n\tFROM\n\t\tsys.objects o\n\t\tINNER JOIN sys.schemas s\n\t\t\tON o.[schema_id] = s.[schema_id]\n\tWHERE\n\t\to.[name] = 'DataFactorys'\n\t\tAND s.[name] = 'procfwk'\n\t\tAND o.[type] = 'U' --Check for tables as created synonyms to support backwards compatability\n\t)\n\tBEGIN\n\t\tSELECT * INTO [dbo].[zz_DataFactorys] FROM [procfwk].[DataFactorys];\n\tEND;"
},
{
"path": "MetadataDB/Scripts/Metadata/Orchestrators.sql",
"content": "EXEC [procfwkHelpers].[SetDefaultTenant];\nEXEC [procfwkHelpers].[SetDefaultSubscription];\nEXEC [procfwkHelpers].[SetDefaultOrchestrators];"
},
{
"path": "MetadataDB/Scripts/Metadata/PipelineDependencies.sql",
"content": "EXEC [procfwkHelpers].[SetDefaultPipelineDependants];"
},
{
"path": "MetadataDB/Scripts/Metadata/PipelineParams.sql",
"content": "EXEC [procfwkHelpers].[SetDefaultPipelineParameters];"
},
{
"path": "MetadataDB/Scripts/Metadata/Pipelines.sql",
"content": "EXEC [procfwkHelpers].[SetDefaultPipelines];"
},
{
"path": "MetadataDB/Scripts/Metadata/Properties.sql",
"content": "EXEC [procfwkHelpers].[SetDefaultProperties];"
},
{
"path": "MetadataDB/Scripts/Metadata/RecipientAlertsLink.sql",
"content": "EXEC [procfwkHelpers].[SetDefaultRecipientPipelineAlerts]"
},
{
"path": "MetadataDB/Scripts/Metadata/Recipients.sql",
"content": "EXEC [procfwkHelpers].[SetDefaultRecipients];"
},
{
"path": "MetadataDB/Scripts/Metadata/ReplaceDataFactorys.sql",
"content": "IF EXISTS \n\t(\n\tSELECT\n\t\t* \n\tFROM\n\t\tsys.objects o\n\t\tINNER JOIN sys.schemas s\n\t\t\tON o.[schema_id] = s.[schema_id]\n\tWHERE\n\t\to.[name] = 'DataFactorys'\n\t\tAND s.[name] = 'procfwk'\n\t\tAND o.[type] = 'U' --Check for tables as created synonyms to support backwards compatability\n\t)\n\tBEGIN\n\t\tDROP TABLE [procfwk].[DataFactorys];\n\n\t\tEXEC('CREATE VIEW [procfwk].[DataFactorys]\nAS\nSELECT\n\t[OrchestratorId] AS DataFactoryId,\n\t[OrchestratorName] AS DataFactoryName,\n\t[ResourceGroupName],\n\t[SubscriptionId],\n\t[Description]\nFROM\n\t[procfwk].[Orchestrators]\nWHERE\n\t[OrchestratorType] = ''ADF'';')\n\tEND;"
},
{
"path": "MetadataDB/Scripts/Metadata/Stages.sql",
"content": "EXEC [procfwkHelpers].[SetDefaultBatches];\nEXEC [procfwkHelpers].[SetDefaultStages];\nEXEC [procfwkHelpers].[SetDefaultBatchStageLink];"
},
{
"path": "MetadataDB/Scripts/Metadata/TransferHelperObjects.sql",
"content": "IF OBJECT_ID(N'tempdb..#TransferHelperObjects') IS NOT NULL DROP PROCEDURE #TransferHelperObjects;\nGO\n\nCREATE PROCEDURE #TransferHelperObjects\n\t(\n\t@ObjectName NVARCHAR(128),\n\t@ObjectType CHAR(2)\n\t)\nAS\nBEGIN\n\tIF EXISTS\n\t\t(\n\t\tSELECT\n\t\t\t*\n\t\tFROM\n\t\t\tsys.objects o\n\t\t\tINNER JOIN sys.schemas s\n\t\t\t\tON o.[schema_id] = s.[schema_id]\n\t\tWHERE\n\t\t\to.[Name] = @ObjectName\n\t\t\tAND s.[name] = 'procfwk'\n\t\t\tAND o.[type] = @ObjectType\n\t\t)\n\t\tBEGIN\n\t\t\tPRINT 'Transferring: ' + @ObjectName;\n\t\t\tEXEC('ALTER SCHEMA [procfwkHelpers] TRANSFER [procfwk].[' + @ObjectName + '];')\n\t\tEND;\nEND;\nGO\n\nEXEC #TransferHelperObjects 'AddProperty', 'P';\nEXEC #TransferHelperObjects 'GetExecutionDetails', 'P';\nEXEC #TransferHelperObjects 'AddRecipientPipelineAlerts', 'P';\nEXEC #TransferHelperObjects 'DeleteRecipientAlerts', 'P';\nEXEC #TransferHelperObjects 'CheckStageAndPiplineIntegrity', 'P';\nEXEC #TransferHelperObjects 'AddPipelineDependant', 'P';\nEXEC #TransferHelperObjects 'AddServicePrincipalWrapper', 'P';\nEXEC #TransferHelperObjects 'AddServicePrincipalUrls', 'P';\nEXEC #TransferHelperObjects 'AddServicePrincipal', 'P';\nEXEC #TransferHelperObjects 'DeleteServicePrincipal', 'P';\nEXEC #TransferHelperObjects 'CheckForValidURL', 'FN';\nEXEC #TransferHelperObjects 'PipelineDependencyChains', 'V';"
},
{
"path": "MetadataDB/Scripts/Metadata/TransferReportingObjects.sql",
"content": "IF OBJECT_ID(N'tempdb..#TransferReportingObjects') IS NOT NULL DROP PROCEDURE #TransferReportingObjects;\nGO\n\nCREATE PROCEDURE #TransferReportingObjects\n\t(\n\t@ObjectName NVARCHAR(128),\n\t@ObjectType CHAR(2)\n\t)\nAS\nBEGIN\n\tIF EXISTS\n\t\t(\n\t\tSELECT\n\t\t\t*\n\t\tFROM\n\t\t\tsys.objects o\n\t\t\tINNER JOIN sys.schemas s\n\t\t\t\tON o.[schema_id] = s.[schema_id]\n\t\tWHERE\n\t\t\to.[Name] = @ObjectName\n\t\t\tAND s.[name] = 'procfwk'\n\t\t\tAND o.[type] = @ObjectType\n\t\t)\n\t\tBEGIN\n\t\t\tPRINT 'Transferring: ' + @ObjectName;\n\t\t\tEXEC('ALTER SCHEMA [procfwkHelpers] TRANSFER [procfwk].[' + @ObjectName + '];')\n\t\tEND;\nEND;\nGO\n\nEXEC #TransferReportingObjects 'PipelineDependencyChains', 'V';\n"
},
{
"path": "MetadataDB/Scripts/Script.PostDeployment.sql",
"content": "/*\nPost-Deployment Script Template\t\t\t\t\t\t\t\n--------------------------------------------------------------------------------------\n This file contains SQL statements that will be appended to the build script.\t\t\n Use SQLCMD syntax to include a file in the post-deployment script.\t\t\t\n Example: :r .\\myfile.sql\t\t\t\t\t\t\t\t\n Use SQLCMD syntax to reference a variable in the post-deployment script.\t\t\n Example: :setvar TableName MyTable\t\t\t\t\t\t\t\n SELECT * FROM [$(TableName)]\t\t\t\t\t\n--------------------------------------------------------------------------------------\n*/\n--load default metadata\n:r .\\Metadata\\Properties.sql\n:r .\\Metadata\\Orchestrators.sql\n:r .\\Metadata\\Stages.sql\n:r .\\Metadata\\Pipelines.sql\n:r .\\Metadata\\PipelineParams.sql\n:r .\\Metadata\\PipelineDependencies.sql\n:r .\\Metadata\\Recipients.sql\n:r .\\Metadata\\AlertOutcomes.sql\n:r .\\Metadata\\RecipientAlertsLink.sql\n\n--restore log data\n:r .\\LogData\\ExecutionLogRestore.sql\n:r .\\LogData\\ErrorLogRestore.sql\n\n--object transfers\n:r .\\Metadata\\TransferHelperObjects.sql\n:r .\\Metadata\\TransferReportingObjects.sql\n\n--replace old objects\n:r .\\Metadata\\ReplaceDataFactorys.sql"
},
{
"path": "MetadataDB/Scripts/Script.PreDeployment.sql",
"content": "/*\n Pre-Deployment Script Template\t\t\t\t\t\t\t\n--------------------------------------------------------------------------------------\n This file contains SQL statements that will be executed before the build script.\t\n Use SQLCMD syntax to include a file in the pre-deployment script.\t\t\t\n Example: :r .\\myfile.sql\t\t\t\t\t\t\t\t\n Use SQLCMD syntax to reference a variable in the pre-deployment script.\t\t\n Example: :setvar TableName MyTable\t\t\t\t\t\t\t\n SELECT * FROM [$(TableName)]\t\t\t\t\t\n--------------------------------------------------------------------------------------\n*/\n\n--data\n:r .\\LogData\\ExecutionLogBackup.sql\n:r .\\LogData\\ErrorLogBackup.sql\n\n--delete all\n:r .\\Metadata\\DropLegacyTables.sql\n:r .\\Metadata\\DropLegacyObjects.sql\n:r .\\Metadata\\DeleteAll.sql"
},
{
"path": "MetadataDB/Scripts/Script.SetLocalAuthenticationDetails.sql",
"content": "--Run in SQLCMD mode to use environment variables.\n\n--Optional script used to setup function testing for processing framework parts post deployment.\n\nPRINT 'TenantId: ' + '$(AZURE_TENANT_ID)'\nPRINT 'SubscriptionId: ' + '$(AZURE_SUBSCRIPTION_ID)'\n\n--add my tenant\nINSERT INTO [procfwk].[Tenants]\n\t(\n\t[TenantId],\n\t[Name],\n\t[Description]\n\t)\nVALUES\n\t('$(AZURE_TENANT_ID)', 'mrpaulandrew.com', NULL);\n\n--add my subscription\nINSERT INTO [procfwk].[Subscriptions]\n\t(\n\t[SubscriptionId],\n\t[Name],\n\t[Description],\n\t[TenantId]\n\t)\nVALUES\n\t('$(AZURE_SUBSCRIPTION_ID)', 'Microsoft Sponsored Subscription', NULL, '$(AZURE_TENANT_ID)');\n\n--update Orchestrator with new subscription\nUPDATE\n\t[procfwk].[Orchestrators]\nSET\n\t[SubscriptionId] = '$(AZURE_SUBSCRIPTION_ID)';\n\n--remove default values\nDELETE FROM [procfwk].[Subscriptions] WHERE [SubscriptionId] = '12345678-1234-1234-1234-012345678910';\nDELETE FROM [procfwk].[Tenants] WHERE [TenantId] = '12345678-1234-1234-1234-012345678910';\n\n/*\nEXEC [procfwkHelpers].[AddProperty]\n\t@PropertyName = N'SPNHandlingMethod',\n\t@PropertyValue = N'StoreInKeyVault',\n\t@Description = N'Accepted values: StoreInDatabase, StoreInKeyVault. See v1.8.2 release notes for full details.';\n*/\n\nIF (SELECT [procfwk].[GetPropertyValueInternal]('SPNHandlingMethod')) = 'StoreInDatabase'\n\tBEGIN\n\t\t--Add SPN for execution of all worker pipelines using database to store SPN values\n\t\tEXEC [procfwkHelpers].[AddServicePrincipalWrapper]\n\t\t\t@OrchestratorName = N'FrameworkFactory',\n\t\t\t@OrchestratorType = 'ADF',\n\t\t\t@PrincipalIdValue = '$(AZURE_CLIENT_ID)',\n\t\t\t@PrincipalSecretValue = '$(AZURE_CLIENT_SECRET)',\n\t\t\t@PrincipalName = '$(AZURE_CLIENT_NAME)';\n\t\t\n\t\tEXEC [procfwkHelpers].[AddServicePrincipalWrapper]\n\t\t\t@OrchestratorName = N'procfwkforsynapse',\n\t\t\t@OrchestratorType = 'SYN',\n\t\t\t@PrincipalIdValue = '$(AZURE_CLIENT_ID)',\n\t\t\t@PrincipalSecretValue = '$(AZURE_CLIENT_SECRET)',\n\t\t\t@PrincipalName = '$(AZURE_CLIENT_NAME)';\n\n\t\t/*\n\t\t--Add specific SPN for execution of Wait 1 pipeline (functional testing)\t\n\t\tEXEC [procfwkHelpers].[DeleteServicePrincipal]\n\t\t\t@OrchestratorName = N'FrameworkFactory',\n\t\t\t@OrchestratorType = 'ADF',\n\t\t\t@PrincipalIdValue = '$(AZURE_CLIENT_ID)',\n\t\t\t@SpecificPipelineName = N'Wait 1';\n\n\t\tEXEC [procfwkHelpers].[AddServicePrincipalWrapper]\n\t\t\t@OrchestratorName = N'FrameworkFactory',\n\t\t\t@OrchestratorType = 'ADF',\n\t\t\t@PrincipalIdValue = '$(AZURE_CLIENT_ID_2)',\n\t\t\t@PrincipalSecretValue = '$(AZURE_CLIENT_SECRET_2)',\n\t\t\t@PrincipalName = '$(AZURE_CLIENT_NAME_2)',\n\t\t\t@SpecificPipelineName = N'Wait 1';\n\t\t*/\n\tEND\nELSE IF (SELECT [procfwk].[GetPropertyValueInternal]('SPNHandlingMethod')) = 'StoreInKeyVault'\n\tBEGIN\n\t\t--Add SPN for execution of all worker pipelines using database to store key vault URLs\n\t\tEXEC [procfwkHelpers].[AddServicePrincipalWrapper]\n\t\t\t@OrchestratorName = N'FrameworkFactory',\n\t\t\t@OrchestratorType = 'ADF',\n\t\t\t@PrincipalIdValue = '$(AZURE_CLIENT_ID_URL)',\n\t\t\t@PrincipalSecretValue = '$(AZURE_CLIENT_SECRET_URL)',\n\t\t\t@PrincipalName = '$(AZURE_CLIENT_NAME)';\n\tEND\nELSE\n\tBEGIN\n\t\tRAISERROR('Unknown SPN insert method.',16,1);\n\tEND;\n\nSELECT * FROM [dbo].[ServicePrincipals];"
},
{
"path": "MetadataDB/Security/procfwk.sql",
"content": "CREATE SCHEMA [procfwk]\nAUTHORIZATION [dbo];\n\n"
},
{
"path": "MetadataDB/Security/procfwkHelpers.sql",
"content": "CREATE SCHEMA [procfwkHelpers]\nAUTHORIZATION [dbo];"
},
{
"path": "MetadataDB/Security/procfwkReporting.sql",
"content": "CREATE SCHEMA [procfwkReporting]\nAUTHORIZATION [dbo];"
},
{
"path": "MetadataDB/Security/procfwkTesting.sql",
"content": "CREATE SCHEMA [procfwkTesting]\nAUTHORIZATION [dbo];"
},
{
"path": "MetadataDB/Security/procfwkuser Role.sql",
"content": "/*\nCREATE USER [##Data Factory Name (Managed Identity)##] \nFROM EXTERNAL PROVIDER;\n*/\n\nCREATE ROLE [procfwkuser]\nGO\n\nGRANT \n\tEXECUTE, \n\tSELECT,\n\tCONTROL,\n\tALTER\nON SCHEMA::[procfwk] TO [procfwkuser]\nGO\n\n/*\nALTER ROLE [procfwkuser] \nADD MEMBER [##Data Factory Name (Managed Identity)##];\n*/"
},
{
"path": "MetadataDB/dbo/Stored Procedures/DemoModePrecursor.sql",
"content": "CREATE PROCEDURE [dbo].[DemoModePrecursor]\nAS\nBEGIN\n\n\t--quick win\n\tIF ([procfwk].[GetPropertyValueInternal]('ExecutionPrecursorProc')) <> '[dbo].[DemoModePrecursor]'\n\tBEGIN\n\t\tEXEC [procfwkHelpers].[AddProperty]\n\t\t\t@PropertyName = N'ExecutionPrecursorProc',\n\t\t\t@PropertyValue = N'[dbo].[DemoModePrecursor]';\n\tEND;\n\n\t--reduce wait times\n\t;WITH cte AS\n\t\t(\n\t\tSELECT \n\t\t\t[PipelineId],\n\t\t\tLEFT(ABS(CAST(CAST(NEWID() AS VARBINARY(192)) AS INT)),1) AS NewValue\n\t\tFROM \n\t\t\t[procfwk].[PipelineParameters]\n\t\t)\n\tUPDATE\n\t\tpp\n\tSET\n\t\tpp.[ParameterValue] = cte.[NewValue]\n\tFROM\n\t\t[procfwk].[PipelineParameters] pp\n\t\tINNER JOIN cte\n\t\t\tON pp.[PipelineId] = cte.[PipelineId]\n\t\tINNER JOIN [procfwk].[Pipelines] p\n\t\t\tON pp.[PipelineId] = p.[PipelineId]\n\tWHERE\n\t\tpp.[ParameterName] LIKE 'Wait%'\n\t\tAND p.[Enabled] = 1;\n\n\n\t--for intentional error\n\tIF NOT EXISTS\n\t\t(\n\t\tSELECT * FROM [procfwk].[CurrentExecution]\n\t\t)\n\t\tBEGIN\n\t\t\tUPDATE\n\t\t\t\tpp\n\t\t\tSET\n\t\t\t\tpp.[ParameterValue] = 'true'\n\t\t\tFROM\n\t\t\t\t[procfwk].[PipelineParameters] pp\n\t\t\t\tINNER JOIN [procfwk].[Pipelines] p\n\t\t\t\t\tON pp.[PipelineId] = p.[PipelineId]\n\t\t\tWHERE\n\t\t\t\tp.[PipelineName] = 'Intentional Error'\n\t\t\t\tAND pp.[ParameterName] = 'RaiseErrors';\n\t\tEND;\n\t\tELSE\n\t\tBEGIN\n\t\t\tUPDATE\n\t\t\t\tpp\n\t\t\tSET\n\t\t\t\tpp.[ParameterValue] = 'false'\n\t\t\tFROM\n\t\t\t\t[procfwk].[PipelineParameters] pp\n\t\t\t\tINNER JOIN [procfwk].[Pipelines] p\n\t\t\t\t\tON pp.[PipelineId] = p.[PipelineId]\n\t\t\tWHERE\n\t\t\t\tp.[PipelineName] = 'Intentional Error'\n\t\t\t\tAND pp.[ParameterName] = 'RaiseErrors';\n\t\tEND;\n\n\t--dependency chain failure handling\n\tIF ([procfwk].[GetPropertyValueInternal]('FailureHandling')) <> 'DependencyChain'\n\tBEGIN\n\t\tEXEC [procfwkHelpers].[AddProperty]\n\t\t\t@PropertyName = N'FailureHandling',\n\t\t\t@PropertyValue = N'DependencyChain';\n\tEND;\n\n\n\t--short infant iterations\n\tIF ([procfwk].[GetPropertyValueInternal]('PipelineStatusCheckDuration')) <> '5'\n\tBEGIN\n\t\tEXEC [procfwkHelpers].[AddProperty]\n\t\t\t@PropertyName = N'PipelineStatusCheckDuration',\n\t\t\t@PropertyValue = N'5';\n\t\tEND;\nEND;"
},
{
"path": "MetadataDB/dbo/Stored Procedures/ExampleCustomExecutionPrecursor.sql",
"content": "CREATE PROCEDURE [dbo].[ExampleCustomExecutionPrecursor]\nAS\nBEGIN\n\t\n\t--set random Worker pipeline parameter wait times for development environment\n\t;WITH cte AS\n\t\t(\n\t\tSELECT \n\t\t\t[PipelineId],\n\t\t\tLEFT(ABS(CAST(CAST(NEWID() AS VARBINARY(192)) AS INT)),2) AS NewValue\n\t\tFROM \n\t\t\t[procfwk].[PipelineParameters]\n\t\t)\n\tUPDATE\n\t\tpp\n\tSET\n\t\tpp.[ParameterValue] = cte.[NewValue]\n\tFROM\n\t\t[procfwk].[PipelineParameters] pp\n\t\tINNER JOIN cte\n\t\t\tON pp.[PipelineId] = cte.[PipelineId]\n\t\tINNER JOIN [procfwk].[Pipelines] p\n\t\t\tON pp.[PipelineId] = p.[PipelineId]\n\tWHERE\n\t\tpp.[ParameterName] LIKE 'Wait%'\n\t\tAND p.[Enabled] = 1;\n\n\n\t--disable certain Workers if running at the weekend...\n\t-- YOUR CODE HERE\n\n\t--enable certain Workers if running on the 10th day of the month...\n\t-- YOUR CODE HERE\n\n\t--disable certain Stages if running on Friday...\n\t-- YOUR CODE HERE\n\n\t--set Worker pipeline parameters to new value based on ______ ....\n\t-- YOUR CODE HERE\n\t\n\t--etc\nEND;"
},
{
"path": "MetadataDB/dbo/Stored Procedures/FailProcedure.sql",
"content": "CREATE PROCEDURE [dbo].[FailProcedure]\n\t(\n\t@RaiseError VARCHAR(50)\n\t)\nAS\nBEGIN\n\tIF(@RaiseError = 'true')\n\tBEGIN\n\t\tRAISERROR('The Stored Procedure intentionally failed.',16,1);\n\t\tRETURN 0;\n\tEND\nEND;"
},
{
"path": "MetadataDB/dbo/Tables/ServicePrincipals.sql",
"content": "CREATE TABLE [dbo].[ServicePrincipals](\n\t[CredentialId] INT IDENTITY(1,1) NOT NULL,\n\t[PrincipalName] NVARCHAR(256) NULL,\n\t[PrincipalId] UNIQUEIDENTIFIER NULL,\n\t[PrincipalSecret] VARBINARY(256) NULL,\n\t[PrincipalIdUrl] NVARCHAR(MAX) NULL, \n [PrincipalSecretUrl] NVARCHAR(MAX) NULL, \n CONSTRAINT [PK_ServicePrincipals] PRIMARY KEY CLUSTERED ([CredentialId] ASC)\n\t)\nGO\n"
},
{
"path": "MetadataDB/procfwk/Functions/GetPropertyValueInternal.sql",
"content": "CREATE FUNCTION [procfwk].[GetPropertyValueInternal]\n\t(\n\t@PropertyName VARCHAR(128)\n\t)\nRETURNS NVARCHAR(MAX)\nAS\nBEGIN\n\tDECLARE @PropertyValue NVARCHAR(MAX)\n\n\tSELECT\n\t\t@PropertyValue = ISNULL([PropertyValue],'')\n\tFROM\n\t\t[procfwk].[CurrentProperties]\n\tWHERE\n\t\t[PropertyName] = @PropertyName\n\n RETURN @PropertyValue\nEND;\n"
},
{
"path": "MetadataDB/procfwk/Stored Procedures/BatchWrapper.sql",
"content": "CREATE PROCEDURE [procfwk].[BatchWrapper]\n\t(\n\t@BatchId UNIQUEIDENTIFIER,\n\t@LocalExecutionId UNIQUEIDENTIFIER OUTPUT\n\t)\nAS\nBEGIN\n\tSET NOCOUNT ON;\n\n\tDECLARE @RestartStatus BIT\n\n\t--get restart overide property\t\n\tSELECT @RestartStatus = [procfwk].[GetPropertyValueInternal]('OverideRestart')\n\n\t--check for running batch execution\n\tIF EXISTS\n\t\t(\n\t\tSELECT 1 FROM [procfwk].[BatchExecution] WHERE [BatchId] = @BatchId AND ISNULL([BatchStatus],'') = 'Running'\n\t\t)\n\t\tBEGIN\n\t\t\tSELECT\n\t\t\t\t@LocalExecutionId = [ExecutionId]\n\t\t\tFROM\n\t\t\t\t[procfwk].[BatchExecution]\n\t\t\tWHERE\n\t\t\t\t[BatchId] = @BatchId;\n\t\t\t\n\t\t\t--should never actually be called as handled within Orchestrator pipelines using the Pipeline Run Check utility\n\t\t\tRAISERROR('There is already an batch execution run in progress. Stop the related parent pipeline via the Orchestrator first.',16,1);\n\t\t\tRETURN 0;\n\t\tEND\n\tELSE IF EXISTS\n\t\t(\n\t\tSELECT 1 FROM [procfwk].[BatchExecution] WHERE [BatchId] = @BatchId AND ISNULL([BatchStatus],'') = 'Stopped'\n\t\t)\n\t\tAND @RestartStatus = 0\n\t\tBEGIN\n\t\t\tSELECT\n\t\t\t\t@LocalExecutionId = [ExecutionId]\n\t\t\tFROM\n\t\t\t\t[procfwk].[BatchExecution]\n\t\t\tWHERE\n\t\t\t\t[BatchId] = @BatchId\n\t\t\t\tAND ISNULL([BatchStatus],'') = 'Stopped';\n\n\t\t\tRETURN 0;\n\t\tEND\n\tELSE IF EXISTS\n\t\t(\n\t\tSELECT 1 FROM [procfwk].[BatchExecution] WHERE [BatchId] = @BatchId AND ISNULL([BatchStatus],'') = 'Stopped'\n\t\t)\n\t\tAND @RestartStatus = 1\n\t\tBEGIN\n\t\t\t--clean up current execution table and abandon batch\n\t\t\tSELECT\n\t\t\t\t@LocalExecutionId = [ExecutionId]\n\t\t\tFROM\n\t\t\t\t[procfwk].[BatchExecution]\n\t\t\tWHERE\n\t\t\t\t[BatchId] = @BatchId\n\t\t\t\tAND ISNULL([BatchStatus],'') = 'Stopped';\n\t\t\t\n\t\t\tEXEC [procfwk].[UpdateExecutionLog]\n\t\t\t\t@PerformErrorCheck = 0, --Special case when OverideRestart = 1;\n\t\t\t\t@ExecutionId = @LocalExecutionId;\n\t\t\t\n\t\t\t--abandon previous batch execution\n\t\t\tUPDATE\n\t\t\t\t[procfwk].[BatchExecution]\n\t\t\tSET\n\t\t\t\t[BatchStatus] = 'Abandoned'\n\t\t\tWHERE\n\t\t\t\t[BatchId] = @BatchId \n\t\t\t\tAND ISNULL([BatchStatus],'') = 'Stopped';\n\t\n\t\t\tSET @LocalExecutionId = NEWID();\n\n\t\t\t--create new batch run record\n\t\t\tINSERT INTO [procfwk].[BatchExecution]\n\t\t\t\t(\n\t\t\t\t[BatchId],\n\t\t\t\t[ExecutionId],\n\t\t\t\t[BatchName],\n\t\t\t\t[BatchStatus],\n\t\t\t\t[StartDateTime]\n\t\t\t\t)\n\t\t\tSELECT\n\t\t\t\t[BatchId],\n\t\t\t\t@LocalExecutionId,\n\t\t\t\t[BatchName],\n\t\t\t\t'Running',\n\t\t\t\tGETUTCDATE()\n\t\t\tFROM\n\t\t\t\t[procfwk].[Batches]\n\t\t\tWHERE\n\t\t\t\t[BatchId] = @BatchId;\t\t\t\n\t\tEND\n\tELSE\n\t\tBEGIN\n\t\t\tSET @LocalExecutionId = NEWID();\n\n\t\t\t--create new batch run record\n\t\t\tINSERT INTO [procfwk].[BatchExecution]\n\t\t\t\t(\n\t\t\t\t[BatchId],\n\t\t\t\t[ExecutionId],\n\t\t\t\t[BatchName],\n\t\t\t\t[BatchStatus],\n\t\t\t\t[StartDateTime]\n\t\t\t\t)\n\t\t\tSELECT\n\t\t\t\t[BatchId],\n\t\t\t\t@LocalExecutionId,\n\t\t\t\t[BatchName],\n\t\t\t\t'Running',\n\t\t\t\tGETUTCDATE()\n\t\t\tFROM\n\t\t\t\t[procfwk].[Batches]\n\t\t\tWHERE\n\t\t\t\t[BatchId] = @BatchId;\n\t\tEND;\nEND;"
},
{
"path": "MetadataDB/procfwk/Stored Procedures/CheckForBlockedPipelines.sql",
"content": "CREATE PROCEDURE [procfwk].[CheckForBlockedPipelines]\n\t(\n\t@ExecutionId UNIQUEIDENTIFIER,\n\t@StageId INT\n\t)\nAS\nBEGIN\n\tSET NOCOUNT ON;\n\n\t-- If any pipelines still have a status of running, mark as failed to block downstream processing, and add an error log\n\tIF EXISTS\n\t\t(\n\t\tSELECT \n\t\t\t*\n\t\tFROM \n\t\t\t[procfwk].[CurrentExecution]\n\t\tWHERE \n\t\t\t[LocalExecutionId] = @ExecutionId\n\t\t\tAND [StageId] < @StageId\n\t\t\tAND [PipelineStatus] = 'Running'\n\t\t)\n\t\tBEGIN\t\t\n\t\t\tDECLARE @RunningPipelineId INT;\n\t\t\tDECLARE @RunningPipelineStageId INT;\n\t\t\tDECLARE @RunId UNIQUEIDENTIFIER;\n\t\t\tDECLARE @ErrorJson NVARCHAR(MAX);\n\t\t\tDECLARE @RunningCursor CURSOR ;\n\n\t\t\tSET @RunningCursor = CURSOR FOR \n\t\t\t\t\t\t\t\t\t SELECT \n\t\t\t\t\t\t\t\t\t\t [PipelineId],\n\t\t\t\t\t\t\t\t\t\t [StageId],\n\t\t\t\t\t\t\t\t\t\t [PipelineRunId]\n\t\t\t\t\t\t\t\t\t FROM \n\t\t\t\t\t\t\t\t\t\t [procfwk].[CurrentExecution] \n\t\t\t\t\t\t\t\t\t WHERE \n\t\t\t\t\t\t\t\t\t\t [LocalExecutionId] = @ExecutionId\n\t\t\t\t\t\t\t\t\t\t AND [StageId] < @StageId\n\t\t\t\t\t\t\t\t\t\t AND [PipelineStatus] = 'Running'\n\n\t\t\tOPEN @RunningCursor\n\t\t\tFETCH NEXT FROM @RunningCursor INTO @RunningPipelineId, @RunningPipelineStageId, @RunId\n\t\t\t\t\t\n\t\t\tWHILE @@FETCH_STATUS = 0\n\t\t\tBEGIN \n\t\t\t\tEXEC [procfwk].SetLogPipelineFailed \n\t\t\t\t\t@ExecutionId = @ExecutionId,\n\t\t\t\t\t@StageId = @RunningPipelineStageId,\n\t\t\t\t\t@PipelineId = @RunningPipelineId,\n\t\t\t\t\t@RunId = @RunId;\n\n\t\t\t\tSET @ErrorJson = '{ \"RunId\": \"' + Cast(@RunId AS CHAR(36)) + '\", \"Errors\": [ { \"ActivityRunId\": \"00000000-0000-0000-0000-000000000000\", \"ActivityName\": \"Set Pipeline Result\", \"ActivityType\": \"Switch\", \"ErrorCode\": \"Unknown\", \"ErrorType\": \"Framework Error\", \"ErrorMessage\": \"Framework pipeline ''04-Infant'' failed to set the pipeline result, most likely due to a timeout or azure connectivity issue. Check the framework Data Factory monitor for more information.\" } ] }'\n\t\t\t\tEXEC procfwk.SetErrorLogDetails @LocalExecutionId = @ExecutionId,\n @JsonErrorDetails = @ErrorJson;\n\t\t\t\t\t\t\n\t\t\t\tFETCH NEXT FROM @RunningCursor INTO @RunningPipelineId, @RunningPipelineStageId, @RunId;\n\t\t\tEND;\n\t\t\tCLOSE @RunningCursor;\n\t\t\tDEALLOCATE @RunningCursor;\n\t\tEND;\n\n\n\tIF ([procfwk].[GetPropertyValueInternal]('FailureHandling')) = 'None'\n\t\tBEGIN\n\t\t\t--do nothing allow processing to carry on regardless\n\t\t\tRETURN 0;\n\t\tEND;\n\t\t\n\tELSE IF ([procfwk].[GetPropertyValueInternal]('FailureHandling')) = 'Simple'\n\t\tBEGIN\n\t\t\tIF EXISTS\n\t\t\t\t(\n\t\t\t\tSELECT \n\t\t\t\t\t*\n\t\t\t\tFROM \n\t\t\t\t\t[procfwk].[CurrentExecution]\n\t\t\t\tWHERE \n\t\t\t\t\t[LocalExecutionId] = @ExecutionId\n\t\t\t\t\tAND [StageId] = @StageId\n\t\t\t\t\tAND [IsBlocked] = 1\n\t\t\t\t)\n\t\t\t\tBEGIN\t\t\n\t\t\t\t\tUPDATE\n\t\t\t\t\t\t[procfwk].[BatchExecution]\n\t\t\t\t\tSET\n\t\t\t\t\t\t[EndDateTime] = GETUTCDATE(),\n\t\t\t\t\t\t[BatchStatus] = 'Stopped'\n\t\t\t\t\tWHERE\n\t\t\t\t\t\t[ExecutionId] = @ExecutionId;\n\t\t\t\t\t\n\t\t\t\t\t--Saves the infant pipeline and activities being called throwing the exception at this level.\n\t\t\t\t\tRAISERROR('All pipelines are blocked. Stopping processing.',16,1); \n\t\t\t\t\t--If not thrown here, the proc [procfwk].[UpdateExecutionLog] would eventually throw an exception.\n\t\t\t\t\tRETURN 0;\n\t\t\t\tEND\t\t\t\n\t\tEND;\n\t\n\tELSE IF ([procfwk].[GetPropertyValueInternal]('FailureHandling')) = 'DependencyChain'\n\t\tBEGIN\n\t\t\tIF EXISTS\n\t\t\t\t(\n\t\t\t\tSELECT \n\t\t\t\t\t*\n\t\t\t\tFROM \n\t\t\t\t\t[procfwk].[CurrentExecution]\n\t\t\t\tWHERE \n\t\t\t\t\t[LocalExecutionId] = @ExecutionId\n\t\t\t\t\tAND [StageId] = @StageId\n\t\t\t\t\tAND [IsBlocked] = 1\n\t\t\t\t)\n\t\t\t\tBEGIN\t\t\n\t\t\t\t\tDECLARE @PipelineId INT;\n\t\t\t\t\tDECLARE @Cursor CURSOR ;\n\n\t\t\t\t\tSET @Cursor = CURSOR FOR \n\t\t\t\t\t\t\t\t\t\t\tSELECT \n\t\t\t\t\t\t\t\t\t\t\t\t[PipelineId] \n\t\t\t\t\t\t\t\t\t\t\tFROM \n\t\t\t\t\t\t\t\t\t\t\t\t[procfwk].[CurrentExecution] \n\t\t\t\t\t\t\t\t\t\t\tWHERE \n\t\t\t\t\t\t\t\t\t\t\t\t[LocalExecutionId] = @ExecutionId\n\t\t\t\t\t\t\t\t\t\t\t\tAND [StageId] = @StageId \n\t\t\t\t\t\t\t\t\t\t\t\tAND [IsBlocked] = 1\n\n\t\t\t\t\tOPEN @Cursor\n\t\t\t\t\tFETCH NEXT FROM @Cursor INTO @PipelineId\n\t\t\t\t\t\n\t\t\t\t\tWHILE @@FETCH_STATUS = 0\n\t\t\t\t\tBEGIN \n\t\t\t\t\t\tEXEC [procfwk].[SetExecutionBlockDependants]\n\t\t\t\t\t\t\t@ExecutionId = @ExecutionId,\n\t\t\t\t\t\t\t@PipelineId = @PipelineId;\n\t\t\t\t\t\t\n\t\t\t\t\t\tFETCH NEXT FROM @Cursor INTO @PipelineId;\n\t\t\t\t\tEND;\n\t\t\t\t\tCLOSE @Cursor;\n\t\t\t\t\tDEALLOCATE @Cursor;\n\t\t\t\tEND;\n\t\tEND;\n\tELSE\n\t\tBEGIN\n\t\t\tRAISERROR('Unknown failure handling state.',16,1);\n\t\t\tRETURN 0;\n\t\tEND;\nEND;\n"
},
{
"path": "MetadataDB/procfwk/Stored Procedures/CheckForEmailAlerts.sql",
"content": "CREATE PROCEDURE [procfwk].[CheckForEmailAlerts]\n\t(\n\t@PipelineId INT\n\t)\nAS\nBEGIN\n\tSET NOCOUNT ON;\n\n\tDECLARE @SendAlerts BIT\n\tDECLARE @AlertingEnabled BIT\n\n\t--get property\n\tSELECT\n\t\t@AlertingEnabled = [procfwk].[GetPropertyValueInternal]('UseFrameworkEmailAlerting');\n\n\t--based on global property\n\tIF (@AlertingEnabled = 1)\n\t\tBEGIN\n\t\t\t--based on piplines to recipients link\n\t\t\tIF EXISTS\n\t\t\t\t(\n\t\t\t\tSELECT pal.AlertId\n\t\t\t\tFROM procfwk.CurrentExecution AS ce\n\t\t\t\tINNER JOIN procfwk.AlertOutcomes AS ao\n\t\t\t\t\tON ao.PipelineOutcomeStatus = ce.PipelineStatus\n\t\t\t\tINNER JOIN procfwk.PipelineAlertLink AS pal\n\t\t\t\t\tON pal.PipelineId = ce.PipelineId\n\t\t\t\tINNER JOIN procfwk.Recipients AS r\n\t\t\t\t\tON r.RecipientId = pal.RecipientId\n\t\t\t\tWHERE ce.PipelineId = @PipelineId\n\t\t\t\t\t AND (\n\t\t\t\t\t\tao.BitValue & pal.OutcomesBitValue <> 0\n\t\t\t\t\t\tOR pal.OutcomesBitValue & 1 <> 0 --all\n\t\t\t\t\t\t)\n\t\t\t\t\t AND pal.[Enabled] = 1\n\t\t\t\t\t AND r.[Enabled] = 1\n\t\t\t\t)\n\t\t\t\tBEGIN\n\t\t\t\t\tSET @SendAlerts = 1;\n\t\t\t\tEND;\n\t\t\tELSE\n\t\t\t\tBEGIN\n\t\t\t\t\tSET @SendAlerts = 0;\n\t\t\t\tEND;\n\t\tEND\n\tELSE\n\t\tBEGIN\n\t\t\tSET @SendAlerts = 0;\n\t\tEND;\n\n\tSELECT @SendAlerts AS SendAlerts\nEND;"
},
{
"path": "MetadataDB/procfwk/Stored Procedures/CheckMetadataIntegrity.sql",
"content": "CREATE PROCEDURE [procfwk].[CheckMetadataIntegrity]\n\t(\n\t@DebugMode BIT = 0,\n\t@BatchName VARCHAR(255) = NULL\n\t)\nAS\nBEGIN\n\tSET NOCOUNT ON;\n\t\n\t/*\n\tCheck 1 - Are there execution stages enabled in the metadata?\n\tCheck 2 - Are there pipelines enabled in the metadata?\n\tCheck 3 - Are there any service principals available to run the processing pipelines?\n\tCheck 4 - Is there at least one TenantId available?\n\tCheck 5 - Is there at least one SubscriptionId available?\n\tCheck 6 - Is there a current OverideRestart property available?\n\tCheck 7 - Are there any enabled pipelines configured without a service principal?\n\tCheck 8 - Are any Orchestrators set to use the default subscription value?\n\tCheck 9 - Are any Subscription set to use the default tenant value?\n\tCheck 10 - Is there a current PipelineStatusCheckDuration property available?\n\tCheck 11 - Is there a current UseFrameworkEmailAlerting property available?\n\tCheck 12 - Is there a current EmailAlertBodyTemplate property available?\n\tCheck 13 - Does the total size of the request body for the pipeline parameters added exceed the Azure Functions size limit when the Worker execute pipeline body is created?\n\tCheck 14 - Is there a current FailureHandling property available?\n\tCheck 15 - Does the FailureHandling property have a valid value?\n\tCheck 16 - When using DependencyChain failure handling, are there any dependants in the same execution stage of the predecessor?\n\tCheck 17 - Does the SPNHandlingMethod property have a valid value?\n\tCheck 18 - Does the Service Principal table contain both types of SPN handling for a single credential?\n\tCheck 19 - Is there a current UseExecutionBatches property available?\n\tCheck 20 - Is there a current FrameworkFactoryResourceGroup property available?\n\tCheck 21 - Is there a current PreviousPipelineRunsQueryRange property available?\n\t\n\t--Batch execution checks:\n\tCheck 22 - If using batch executions, is the requested batch name enabled?\n\tCheck 23 - If using batch executions, does the requested batch have links to execution stages?\n\tCheck 24 - Have batch executions been enabled after a none batch execution run?\n\n\tCheck 25 - Has the execution failed due to an invalid pipeline name? If so, attend to update this before the next run.\n\tCheck 26 - Is there more than one framework orchestrator set?\n\tCheck 27 - Has a framework orchestrator been set for any orchestrators?\n\t*/\n\n\tDECLARE @BatchId UNIQUEIDENTIFIER\n\tDECLARE @ErrorDetails VARCHAR(500)\n\tDECLARE @MetadataIntegrityIssues TABLE\n\t\t(\n\t\t[CheckNumber] INT NOT NULL,\n\t\t[IssuesFound] VARCHAR(MAX) NOT NULL\n\t\t)\n\n\t/*\n\tChecks:\n\t*/\n\n\t--Check 1:\n\tIF NOT EXISTS\n\t\t(\n\t\tSELECT 1 FROM [procfwk].[Stages] WHERE [Enabled] = 1\n\t\t)\n\t\tBEGIN\n\t\t\tINSERT INTO @MetadataIntegrityIssues\n\t\t\tVALUES\n\t\t\t\t( \n\t\t\t\t1,\n\t\t\t\t'No execution stages are enabled within the metadatabase. Orchestrator has nothing to run.'\n\t\t\t\t)\n\t\tEND;\n\n\t--Check 2:\n\tIF NOT EXISTS\n\t\t(\n\t\tSELECT 1 FROM [procfwk].[Pipelines] WHERE [Enabled] = 1\n\t\t)\n\t\tBEGIN\n\t\t\tINSERT INTO @MetadataIntegrityIssues\n\t\t\tVALUES\n\t\t\t\t( \n\t\t\t\t2,\n\t\t\t\t'No execution pipelines are enabled within the metadatabase. Orchestrator has nothing to run.'\n\t\t\t\t)\n\t\tEND;\n\n\t--Check 3:\n\tIF NOT EXISTS \n\t\t(\n\t\tSELECT 1 FROM [dbo].[ServicePrincipals]\n\t\t)\n\t\tBEGIN\n\t\t\tINSERT INTO @MetadataIntegrityIssues\n\t\t\tVALUES\n\t\t\t\t( \n\t\t\t\t3,\n\t\t\t\t'No service principal details have been added to the metadata. Orchestrator cannot authorise pipeline executions.'\n\t\t\t\t)\t\t\n\t\tEND;\n\n\t--Check 4:\n\tIF NOT EXISTS\n\t\t(\n\t\tSELECT * FROM [procfwk].[Tenants]\n\t\t)\n\t\tBEGIN\n\t\t\tINSERT INTO @MetadataIntegrityIssues\n\t\t\tVALUES\n\t\t\t\t( \n\t\t\t\t4,\n\t\t\t\t'TenantId value is missing from the [procfwk].[Tenants] table.'\n\t\t\t\t)\t\t\n\t\tEND;\n\n\t--Check 5:\n\tIF NOT EXISTS\n\t\t(\n\t\tSELECT * FROM [procfwk].[Subscriptions]\n\t\t)\n\t\tBEGIN\n\t\t\tINSERT INTO @MetadataIntegrityIssues\n\t\t\tVALUES\n\t\t\t\t( \n\t\t\t\t5,\n\t\t\t\t'SubscriptionId value is missing from the [procfwk].[Subscriptions] table.'\n\t\t\t\t)\t\t\n\t\tEND;\n\n\t--Check 6:\n\tIF NOT EXISTS\n\t\t(\n\t\tSELECT * FROM [procfwk].[CurrentProperties] WHERE [PropertyName] = 'OverideRestart'\n\t\t)\n\t\tBEGIN\n\t\t\tINSERT INTO @MetadataIntegrityIssues\n\t\t\tVALUES\n\t\t\t\t( \n\t\t\t\t6,\n\t\t\t\t'A current OverideRestart value is missing from the properties table.'\n\t\t\t\t)\t\t\n\t\tEND;\n\n\t--Check 7:\n\tIF EXISTS\n\t\t( \n\t\tSELECT \n\t\t\t* \n\t\tFROM \n\t\t\t[procfwk].[Pipelines] p \n\t\t\tLEFT OUTER JOIN [procfwk].[PipelineAuthLink] al \n\t\t\t\tON p.[PipelineId] = al.[PipelineId]\n\t\tWHERE\n\t\t\tp.[Enabled] = 1\n\t\t\tAND al.[PipelineId] IS NULL\n\t\t)\n\t\tBEGIN\n\t\t\tINSERT INTO @MetadataIntegrityIssues\n\t\t\tVALUES\n\t\t\t\t( \n\t\t\t\t7,\n\t\t\t\t'Enabled pipelines are missing a valid Service Principal link.'\n\t\t\t\t)\t\t\n\t\tEND;\n\n\t--Check 8:\n\tIF EXISTS\n\t\t(\n\t\tSELECT * FROM [procfwk].[Orchestrators] WHERE [SubscriptionId] = '12345678-1234-1234-1234-012345678910'\n\t\t)\n\t\tBEGIN\n\t\t\tINSERT INTO @MetadataIntegrityIssues\n\t\t\tVALUES\n\t\t\t\t( \n\t\t\t\t8,\n\t\t\t\t'Orchestrator still set to use the default subscription value of 12345678-1234-1234-1234-012345678910.'\n\t\t\t\t)\t\t\n\t\tEND;\n\n\t--Check 9:\n\tIF EXISTS\n\t\t(\n\t\tSELECT * FROM [procfwk].[Subscriptions] WHERE [TenantId] = '12345678-1234-1234-1234-012345678910' AND [SubscriptionId] <> '12345678-1234-1234-1234-012345678910'\n\t\t)\n\t\tBEGIN\n\t\t\tINSERT INTO @MetadataIntegrityIssues\n\t\t\tVALUES\n\t\t\t\t( \n\t\t\t\t9,\n\t\t\t\t'None default subscription still set to use the default tenant value of 12345678-1234-1234-1234-012345678910.'\n\t\t\t\t)\t\t\n\t\tEND;\n\n\t--Check 10:\n\tIF NOT EXISTS\n\t\t(\n\t\tSELECT * FROM [procfwk].[CurrentProperties] WHERE [PropertyName] = 'PipelineStatusCheckDuration'\n\t\t)\n\t\tBEGIN\n\t\t\tINSERT INTO @MetadataIntegrityIssues\n\t\t\tVALUES\n\t\t\t\t( \n\t\t\t\t10,\n\t\t\t\t'A current PipelineStatusCheckDuration value is missing from the properties table.'\n\t\t\t\t)\t\t\n\t\tEND;\n\n\t--Check 11:\n\tIF NOT EXISTS\n\t\t(\n\t\tSELECT * FROM [procfwk].[CurrentProperties] WHERE [PropertyName] = 'UseFrameworkEmailAlerting'\n\t\t)\n\t\tBEGIN\n\t\t\tINSERT INTO @MetadataIntegrityIssues\n\t\t\tVALUES\n\t\t\t\t( \n\t\t\t\t11,\n\t\t\t\t'A current UseFrameworkEmailAlerting value is missing from the properties table.'\n\t\t\t\t)\t\t\n\t\tEND;\n\n\t--Check 12:\n\tIF (\n\t\tSELECT\n\t\t\t[PropertyValue]\n\t\tFROM\n\t\t\t[procfwk].[CurrentProperties]\n\t\tWHERE\n\t\t\t[PropertyName] = 'UseFrameworkEmailAlerting'\n\t\t) = 1\n\t\tBEGIN\n\t\t\tIF NOT EXISTS\n\t\t\t\t(\n\t\t\t\tSELECT * FROM [procfwk].[CurrentProperties] WHERE [PropertyName] = 'EmailAlertBodyTemplate'\n\t\t\t\t)\n\t\t\t\tBEGIN\n\t\t\t\t\tINSERT INTO @MetadataIntegrityIssues\n\t\t\t\t\tVALUES\n\t\t\t\t\t\t( \n\t\t\t\t\t\t12,\n\t\t\t\t\t\t'A current EmailAlertBodyTemplate value is missing from the properties table.'\n\t\t\t\t\t\t)\t\t\n\t\t\t\tEND;\n\t\tEND;\n\n\t--Check 13:\n\tIF EXISTS\n\t\t(\n\t\tSELECT * FROM [procfwk].[PipelineParameterDataSizes] WHERE [Size] > 9\n\t\t/*\n\t\tAzure Function request limit is 10MB.\n\t\thttps://docs.microsoft.com/en-us/azure/azure-functions/functions-scale\n\t\t9MB to allow for other content in execute pipeline body request.\n\t\t*/\n\t\t)\n\t\tBEGIN\n\t\t\tINSERT INTO @MetadataIntegrityIssues\n\t\t\tVALUES\n\t\t\t\t( \n\t\t\t\t13,\n\t\t\t\t'The pipeline parameters entered exceed the Azure Function request body maximum of 10MB. Query view [procfwk].[PipelineParameterDataSizes] for details.'\n\t\t\t\t)\t\n\t\tEND;\n\n\t--Check 14:\n\tIF NOT EXISTS\n\t\t(\n\t\tSELECT * FROM [procfwk].[CurrentProperties] WHERE [PropertyName] = 'FailureHandling'\n\t\t)\n\t\tBEGIN\n\t\t\tINSERT INTO @MetadataIntegrityIssues\n\t\t\tVALUES\n\t\t\t\t( \n\t\t\t\t14,\n\t\t\t\t'A current FailureHandling value is missing from the properties table.'\n\t\t\t\t)\t\t\n\t\tEND;\n\n\t--Check 15:\n\tIF NOT EXISTS\n\t\t(\n\t\tSELECT \n\t\t\t*\n\t\tFROM\n\t\t\t[procfwk].[CurrentProperties] \n\t\tWHERE \n\t\t\t[PropertyName] = 'FailureHandling' \n\t\t\tAND [PropertyValue] IN ('None','Simple','DependencyChain')\n\t\t)\n\t\tBEGIN\n\t\t\tINSERT INTO @MetadataIntegrityIssues\n\t\t\tVALUES\n\t\t\t\t( \n\t\t\t\t15,\n\t\t\t\t'The property FailureHandling does not have a supported value.'\n\t\t\t\t)\t\n\t\tEND;\n\n\t--Check 16:\n\tIF ([procfwk].[GetPropertyValueInternal]('FailureHandling')) = 'DependencyChain'\n\tBEGIN\n\t\tIF EXISTS\n\t\t(\n\t\tSELECT \n\t\t\tpd.[DependencyId]\n\t\tFROM \n\t\t\t[procfwk].[PipelineDependencies] pd\n\t\t\tINNER JOIN [procfwk].[Pipelines] pp\n\t\t\t\tON pd.[PipelineId] = pp.[PipelineId]\n\t\t\tINNER JOIN [procfwk].[Pipelines] dp\n\t\t\t\tON pd.[DependantPipelineId] = dp.[PipelineId]\n\t\tWHERE\n\t\t\tpp.[StageId] = dp.[StageId]\n\t\t)\t\n\t\tBEGIN\n\t\t\tINSERT INTO @MetadataIntegrityIssues\n\t\t\tVALUES\n\t\t\t\t( \n\t\t\t\t16,\n\t\t\t\t'A dependant pipeline and its upstream predecessor exist in the same execution stage. Fix this dependency chain to allow correct failure handling.'\n\t\t\t\t)\t\n\t\tEND;\n\tEND;\n\n\t--Check 17:\n\tIF NOT EXISTS\n\t\t(\n\t\tSELECT \n\t\t\t*\n\t\tFROM\n\t\t\t[procfwk].[CurrentProperties] \n\t\tWHERE \n\t\t\t[PropertyName] = 'SPNHandlingMethod' \n\t\t\tAND [PropertyValue] IN ('StoreInDatabase','StoreInKeyVault')\n\t\t)\n\t\tBEGIN\n\t\t\tINSERT INTO @MetadataIntegrityIssues\n\t\t\tVALUES\n\t\t\t\t( \n\t\t\t\t17,\n\t\t\t\t'The property SPNHandlingMethod does not have a supported value.'\n\t\t\t\t)\t\n\t\tEND;\n\n\t--Check 18:\n\tIF EXISTS\n\t\t(\n\t\tSELECT\n\t\t\t*\n\t\tFROM\n\t\t\t[dbo].[ServicePrincipals]\n\t\tWHERE\n\t\t\t(\n\t\t\t[PrincipalId] IS NOT NULL\n\t\t\tOR [PrincipalSecret] IS NOT NULL\n\t\t\t)\n\t\t\tAND \n\t\t\t(\n\t\t\t[PrincipalIdUrl] IS NOT NULL\n\t\t\tOR [PrincipalSecretUrl] IS NOT NULL\n\t\t\t)\n\t\t)\n\t\tBEGIN\n\t\t\tINSERT INTO @MetadataIntegrityIssues\n\t\t\tVALUES\n\t\t\t\t( \n\t\t\t\t18,\n\t\t\t\t'The table [dbo].[ServicePrincipals] can only have one method of SPN details sorted per credential ID.'\n\t\t\t\t)\t\n\t\tEND;\n\t\n\t--Check 19:\n\tIF NOT EXISTS\n\t\t(\n\t\tSELECT * FROM [procfwk].[CurrentProperties] WHERE [PropertyName] = 'UseExecutionBatches'\n\t\t)\n\t\tBEGIN\n\t\t\tINSERT INTO @MetadataIntegrityIssues\n\t\t\tVALUES\n\t\t\t\t( \n\t\t\t\t19,\n\t\t\t\t'A current UseExecutionBatches value is missing from the properties table.'\n\t\t\t\t)\t\t\n\t\tEND;\n\n\t--Check 20:\n\tIF NOT EXISTS\n\t\t(\n\t\tSELECT * FROM [procfwk].[CurrentProperties] WHERE [PropertyName] = 'FrameworkFactoryResourceGroup'\n\t\t)\n\t\tBEGIN\n\t\t\tINSERT INTO @MetadataIntegrityIssues\n\t\t\tVALUES\n\t\t\t\t( \n\t\t\t\t20,\n\t\t\t\t'A current FrameworkFactoryResourceGroup value is missing from the properties table.'\n\t\t\t\t)\t\t\n\t\tEND;\n\n\t--Check 21:\n\tIF NOT EXISTS\n\t\t(\n\t\tSELECT * FROM [procfwk].[CurrentProperties] WHERE [PropertyName] = 'PreviousPipelineRunsQueryRange'\n\t\t)\n\t\tBEGIN\n\t\t\tINSERT INTO @MetadataIntegrityIssues\n\t\t\tVALUES\n\t\t\t\t( \n\t\t\t\t21,\n\t\t\t\t'A current PreviousPipelineRunsQueryRange value is missing from the properties table.'\n\t\t\t\t)\t\t\n\t\tEND;\n\n\t--batch execution checks\n\tIF ([procfwk].[GetPropertyValueInternal]('UseExecutionBatches')) = '1'\n\t\tBEGIN\t\t\t\n\t\t\tIF @BatchName IS NULL\n\t\t\t\tBEGIN\n\t\t\t\t\tRAISERROR('A NULL batch name cannot be passed when the UseExecutionBatches property is set to 1 (true).',16,1);\n\t\t\t\t\tRETURN 0;\n\t\t\t\tEND\n\n\t\t\tSELECT \n\t\t\t\t@BatchId = [BatchId]\n\t\t\tFROM\n\t\t\t\t[procfwk].[Batches]\n\t\t\tWHERE\n\t\t\t\t[BatchName] = @BatchName;\n\n\t\t\t--Check 22:\n\t\t\tIF EXISTS\n\t\t\t\t(\n\t\t\t\tSELECT 1 FROM [procfwk].[Batches] WHERE [BatchId] = @BatchId AND [Enabled] = 0\n\t\t\t\t)\n\t\t\t\tBEGIN\n\t\t\t\t\tINSERT INTO @MetadataIntegrityIssues\n\t\t\t\t\tVALUES\n\t\t\t\t\t\t( \n\t\t\t\t\t\t22,\n\t\t\t\t\t\t'The requested execution batch is currently disabled. Enable the batch before proceeding.'\n\t\t\t\t\t\t)\n\t\t\t\tEND;\n\n\t\t\t--Check 23:\n\t\t\tIF NOT EXISTS\n\t\t\t\t(\n\t\t\t\tSELECT 1 FROM [procfwk].[BatchStageLink] WHERE [BatchId] = @BatchId\n\t\t\t\t)\n\t\t\t\tBEGIN\n\t\t\t\t\tINSERT INTO @MetadataIntegrityIssues\n\t\t\t\t\tVALUES\n\t\t\t\t\t\t( \n\t\t\t\t\t\t23,\n\t\t\t\t\t\t'The requested execution batch does not have any linked execution stages. See table [procfwk].[BatchStageLink] for details.'\n\t\t\t\t\t\t)\n\t\t\t\tEND;\n\n\t\t\t--Check 24:\n\t\t\tIF EXISTS\n\t\t\t\t(\n\t\t\t\tSELECT\n\t\t\t\t\t*\n\t\t\t\tFROM\n\t\t\t\t\t[procfwk].[CurrentExecution] c\n\t\t\t\t\tLEFT OUTER JOIN [procfwk].[BatchExecution] b\n\t\t\t\t\t\tON c.[LocalExecutionId] = b.[ExecutionId]\n\t\t\t\tWHERE\n\t\t\t\t\tb.[ExecutionId] IS NULL\n\t\t\t\t)\n\t\t\t\tBEGIN\n\t\t\t\t\tINSERT INTO @MetadataIntegrityIssues\n\t\t\t\t\tVALUES\n\t\t\t\t\t\t( \n\t\t\t\t\t\t24,\n\t\t\t\t\t\t'Execution records exist in the [procfwk].[CurrentExecution] table that do not have a record in [procfwk].[BatchExecution] table. Has batch excutions been enabed after an incomplete none batch run?'\n\t\t\t\t\t\t)\n\t\t\t\tEND;\t\t\t\n\t\tEND; --end batch checks\n\t\n\t--Check 25: \n\tIF EXISTS\n\t\t(\n\t\tSELECT 1 FROM [procfwk].[CurrentExecution] WHERE [PipelineStatus] = 'InvalidPipelineNameError'\n\t\t)\n\t\tBEGIN\n\t\t\tUPDATE\n\t\t\t\tce\n\t\t\tSET\n\t\t\t\tce.[PipelineName] = p.[PipelineName]\n\t\t\tFROM\n\t\t\t\t[procfwk].[CurrentExecution] ce\n\t\t\t\tINNER JOIN [procfwk].[Pipelines] p\n\t\t\t\t\tON ce.[PipelineId] = p.[PipelineId]\n\t\t\t\t\t\tAND ce.[StageId] = p.[StageId]\n\t\t\tWHERE\n\t\t\t\tce.[PipelineStatus] = 'InvalidPipelineNameError'\n\t\tEND;\n\t\n\t--Check 26:\n\tIF (SELECT COUNT(0) FROM [procfwk].[Orchestrators] WHERE [IsFrameworkOrchestrator] = 1) > 1\n\tBEGIN\n\t\tINSERT INTO @MetadataIntegrityIssues\n\t\tVALUES\n\t\t\t( \n\t\t\t26,\n\t\t\t'There is more than one FrameworkOrchestrator set in the table [procfwk].[Orchestrators]. Only one is supported.'\n\t\t\t)\t\t\n\tEND\n\n\t--Check 27:\n\tIF NOT EXISTS\n\t\t(\n\t\tSELECT 1 FROM [procfwk].[Orchestrators] WHERE [IsFrameworkOrchestrator] = 1\n\t\t)\n\t\tBEGIN\n\t\t\tINSERT INTO @MetadataIntegrityIssues\n\t\t\tVALUES\n\t\t\t\t( \n\t\t\t\t27,\n\t\t\t\t'A FrameworkOrchestrator has not been set in the table [procfwk].[Orchestrators]. Only one is supported.'\n\t\t\t\t)\t\t\n\t\tEND\n\n\t/*\n\tIntegrity Checks Outcome:\n\t*/\n\t\n\t--throw runtime error if checks fail\n\tIF EXISTS\n\t\t(\n\t\tSELECT * FROM @MetadataIntegrityIssues\n\t\t)\n\t\tAND @DebugMode = 0\n\t\tBEGIN\n\t\t\tSET @ErrorDetails = 'Metadata integrity checks failed. Run EXEC [procfwk].[CheckMetadataIntegrity] @DebugMode = 1; for details.'\n\n\t\t\tRAISERROR(@ErrorDetails, 16, 1);\n\t\t\tRETURN 0;\n\t\tEND;\n\n\t--report issues when in debug mode\n\tIF @DebugMode = 1\n\tBEGIN\n\t\tIF NOT EXISTS\n\t\t\t(\n\t\t\tSELECT * FROM @MetadataIntegrityIssues\n\t\t\t)\n\t\t\tBEGIN\n\t\t\t\tPRINT 'No data integrity issues found in metadata.'\n\t\t\t\tRETURN 0;\n\t\t\tEND\n\t\tELSE\t\t\n\t\t\tBEGIN\n\t\t\t\tSELECT * FROM @MetadataIntegrityIssues;\n\t\t\tEND;\n\tEND;\nEND;\n"
},
{
"path": "MetadataDB/procfwk/Stored Procedures/CheckPreviousExeuction.sql",
"content": "CREATE PROCEDURE [procfwk].[CheckPreviousExeuction]\n\t(\n\t@BatchName VARCHAR(255) = NULL\n\t)\nAS\nBEGIN\n\tSET NOCOUNT ON;\n\t/*\n\tCheck A: - Are there any Running pipelines that need to be cleaned up?\n\t*/\n\n\tDECLARE @BatchId UNIQUEIDENTIFIER\n\tDECLARE @LocalExecutionId UNIQUEIDENTIFIER\n\n\t--Check A:\n\tIF ([procfwk].[GetPropertyValueInternal]('UseExecutionBatches')) = '0'\n\t\tBEGIN\n\t\t\tIF EXISTS\n\t\t\t\t(\n\t\t\t\tSELECT \n\t\t\t\t\t1 \n\t\t\t\tFROM \n\t\t\t\t\t[procfwk].[CurrentExecution] \n\t\t\t\tWHERE \n\t\t\t\t\t[PipelineStatus] NOT IN ('Success','Failed','Blocked', 'Cancelled') \n\t\t\t\t\tAND [PipelineRunId] IS NOT NULL\n\t\t\t\t)\n\t\t\t\tBEGIN\n\t\t\t\t\t--return pipelines details that require a clean up\n\t\t\t\t\tSELECT \n\t\t\t\t\t\t[ResourceGroupName],\n\t\t\t\t\t\t[OrchestratorType],\n\t\t\t\t\t\t[OrchestratorName],\n\t\t\t\t\t\t[PipelineName],\n\t\t\t\t\t\t[PipelineRunId],\n\t\t\t\t\t\t[LocalExecutionId],\n\t\t\t\t\t\t[StageId],\n\t\t\t\t\t\t[PipelineId]\n\t\t\t\t\tFROM \n\t\t\t\t\t\t[procfwk].[CurrentExecution]\n\t\t\t\t\tWHERE \n\t\t\t\t\t\t[PipelineStatus] NOT IN ('Success','Failed','Blocked','Cancelled') \n\t\t\t\t\t\tAND [PipelineRunId] IS NOT NULL\n\t\t\t\tEND;\n\t\t\tELSE\n\t\t\t\tGOTO LookUpReturnEmptyResult;\n\t\tEND\n\tELSE IF ([procfwk].[GetPropertyValueInternal]('UseExecutionBatches')) = '1'\n\t\tBEGIN\n\t\t\tIF @BatchName IS NULL\n\t\t\t\tBEGIN\n\t\t\t\t\tRAISERROR('A NULL batch name cannot be passed when the UseExecutionBatches property is set to 1 (true).',16,1);\n\t\t\t\t\tRETURN 0;\n\t\t\t\tEND\n\t\t\t\n\t\t\tIF EXISTS\n\t\t\t\t(\n\t\t\t\tSELECT \n\t\t\t\t\t1 \n\t\t\t\tFROM \n\t\t\t\t\t[procfwk].[CurrentExecution] ce\n\t\t\t\t\tINNER JOIN [procfwk].[BatchExecution] be\n\t\t\t\t\t\tON ce.[LocalExecutionId] = be.[ExecutionId]\n\t\t\t\t\tINNER JOIN [procfwk].[Batches] b\n\t\t\t\t\t\tON be.[BatchId] = b.[BatchId]\n\t\t\t\tWHERE \n\t\t\t\t\tb.[BatchName] = @BatchName\n\t\t\t\t\tAND ce.[PipelineStatus] NOT IN ('Success','Failed','Blocked','Cancelled') \n\t\t\t\t\tAND ce.[PipelineRunId] IS NOT NULL\n\t\t\t\t)\n\t\t\t\tBEGIN\n\t\t\t\t\t--return pipelines details that require a clean up\n\t\t\t\t\tSELECT \n\t\t\t\t\t\tce.[ResourceGroupName],\n\t\t\t\t\t\tce.[OrchestratorType],\n\t\t\t\t\t\tce.[OrchestratorName],\n\t\t\t\t\t\tce.[PipelineName],\n\t\t\t\t\t\tce.[PipelineRunId],\n\t\t\t\t\t\tce.[LocalExecutionId],\n\t\t\t\t\t\tce.[StageId],\n\t\t\t\t\t\tce.[PipelineId]\n\t\t\t\t\tFROM \n\t\t\t\t\t\t[procfwk].[CurrentExecution] ce\n\t\t\t\t\t\tINNER JOIN [procfwk].[BatchExecution] be\n\t\t\t\t\t\t\tON ce.[LocalExecutionId] = be.[ExecutionId]\n\t\t\t\t\t\tINNER JOIN [procfwk].[Batches] b\n\t\t\t\t\t\t\tON be.[BatchId] = b.[BatchId]\n\t\t\t\t\tWHERE \n\t\t\t\t\t\tb.[BatchName] = @BatchName\n\t\t\t\t\t\tAND ce.[PipelineStatus] NOT IN ('Success','Failed','Blocked','Cancelled') \n\t\t\t\t\t\tAND ce.[PipelineRunId] IS NOT NULL\n\t\t\t\tEND;\n\t\t\tELSE\n\t\t\t\tGOTO LookUpReturnEmptyResult;\n\t\tEND\n\t\n\tLookUpReturnEmptyResult:\n\t--lookup activity must return something, even if just an empty dataset\n\tSELECT \n\t\tNULL AS ResourceGroupName,\n\t\tNULL AS OrchestratorType,\n\t\tNULL AS OrchestratorName,\n\t\tNULL AS PipelineName,\n\t\tNULL AS PipelineRunId,\n\t\tNULL AS LocalExecutionId,\n\t\tNULL AS StageId,\n\t\tNULL AS PipelineId\n\tFROM\n\t\t[procfwk].[CurrentExecution]\n\tWHERE\n\t\t1 = 2; --ensure no results\nEND;"
},
{
"path": "MetadataDB/procfwk/Stored Procedures/CreateNewExecution.sql",
"content": "CREATE PROCEDURE [procfwk].[CreateNewExecution]\n\t(\n\t@CallingOrchestratorName NVARCHAR(200),\n\t@LocalExecutionId UNIQUEIDENTIFIER = NULL\n\t)\nAS\nBEGIN\n\tSET NOCOUNT ON;\n\n\tDECLARE @BatchId UNIQUEIDENTIFIER;\n\n\tIF([procfwk].[GetPropertyValueInternal]('UseExecutionBatches')) = '0'\n\t\tBEGIN\n\t\t\tSET @LocalExecutionId = NEWID();\n\n\t\t\tTRUNCATE TABLE [procfwk].[CurrentExecution];\n\n\t\t\t--defensive check\n\t\t\tIF NOT EXISTS\n\t\t\t\t(\n\t\t\t\tSELECT\n\t\t\t\t\t1\n\t\t\t\tFROM\n\t\t\t\t\t[procfwk].[Pipelines] p\n\t\t\t\t\tINNER JOIN [procfwk].[Stages] s\n\t\t\t\t\t\tON p.[StageId] = s.[StageId]\n\t\t\t\t\tINNER JOIN [procfwk].[Orchestrators] d\n\t\t\t\t\t\tON p.[OrchestratorId] = d.[OrchestratorId]\n\t\t\t\tWHERE\n\t\t\t\t\tp.[Enabled] = 1\n\t\t\t\t\tAND s.[Enabled] = 1\n\t\t\t\t)\n\t\t\t\tBEGIN\n\t\t\t\t\tRAISERROR('Requested execution run does not contain any enabled stages/pipelines.',16,1);\n\t\t\t\t\tRETURN 0;\n\t\t\t\tEND;\n\n\t\t\tINSERT INTO [procfwk].[CurrentExecution]\n\t\t\t\t(\n\t\t\t\t[LocalExecutionId],\n\t\t\t\t[StageId],\n\t\t\t\t[PipelineId],\n\t\t\t\t[CallingOrchestratorName],\n\t\t\t\t[ResourceGroupName],\n\t\t\t\t[OrchestratorType],\n\t\t\t\t[OrchestratorName],\n\t\t\t\t[PipelineName]\n\t\t\t\t)\n\t\t\tSELECT\n\t\t\t\t@LocalExecutionId,\n\t\t\t\tp.[StageId],\n\t\t\t\tp.[PipelineId],\n\t\t\t\t@CallingOrchestratorName,\n\t\t\t\td.[ResourceGroupName],\n\t\t\t\td.[OrchestratorType],\n\t\t\t\td.[OrchestratorName],\n\t\t\t\tp.[PipelineName]\n\t\t\tFROM\n\t\t\t\t[procfwk].[Pipelines] p\n\t\t\t\tINNER JOIN [procfwk].[Stages] s\n\t\t\t\t\tON p.[StageId] = s.[StageId]\n\t\t\t\tINNER JOIN [procfwk].[Orchestrators] d\n\t\t\t\t\tON p.[OrchestratorId] = d.[OrchestratorId]\n\t\t\tWHERE\n\t\t\t\tp.[Enabled] = 1\n\t\t\t\tAND s.[Enabled] = 1;\n\n\t\t\tSELECT\n\t\t\t\t@LocalExecutionId AS ExecutionId;\n\t\tEND\n\tELSE IF ([procfwk].[GetPropertyValueInternal]('UseExecutionBatches')) = '1'\n\t\tBEGIN\n\t\t\tDELETE FROM \n\t\t\t\t[procfwk].[CurrentExecution]\n\t\t\tWHERE\n\t\t\t\t[LocalExecutionId] = @LocalExecutionId;\n\n\t\t\tSELECT\n\t\t\t\t@BatchId = [BatchId]\n\t\t\tFROM\n\t\t\t\t[procfwk].[BatchExecution]\n\t\t\tWHERE\n\t\t\t\t[ExecutionId] = @LocalExecutionId;\n\t\t\t\n\t\t\t--defensive check\n\t\t\tIF NOT EXISTS\n\t\t\t\t(\n\t\t\t\tSELECT\n\t\t\t\t\t1\n\t\t\t\tFROM\n\t\t\t\t\t[procfwk].[Pipelines] p\n\t\t\t\t\tINNER JOIN [procfwk].[Stages] s\n\t\t\t\t\t\tON p.[StageId] = s.[StageId]\n\t\t\t\t\tINNER JOIN [procfwk].[Orchestrators] d\n\t\t\t\t\t\tON p.[OrchestratorId] = d.[OrchestratorId]\n\t\t\t\t\tINNER JOIN [procfwk].[BatchStageLink] b\n\t\t\t\t\t\tON b.[StageId] = s.[StageId]\n\t\t\t\tWHERE\n\t\t\t\t\tb.[BatchId] = @BatchId\n\t\t\t\t\tAND p.[Enabled] = 1\n\t\t\t\t\tAND s.[Enabled] = 1\n\t\t\t\t)\n\t\t\t\tBEGIN\n\t\t\t\t\tRAISERROR('Requested execution run does not contain any enabled stages/pipelines.',16,1);\n\t\t\t\t\tRETURN 0;\n\t\t\t\tEND;\n\n\t\t\tINSERT INTO [procfwk].[CurrentExecution]\n\t\t\t\t(\n\t\t\t\t[LocalExecutionId],\n\t\t\t\t[StageId],\n\t\t\t\t[PipelineId],\n\t\t\t\t[CallingOrchestratorName],\n\t\t\t\t[ResourceGroupName],\n\t\t\t\t[OrchestratorType],\n\t\t\t\t[OrchestratorName],\n\t\t\t\t[PipelineName]\n\t\t\t\t)\n\t\t\tSELECT\n\t\t\t\t@LocalExecutionId,\n\t\t\t\tp.[StageId],\n\t\t\t\tp.[PipelineId],\n\t\t\t\t@CallingOrchestratorName,\n\t\t\t\td.[ResourceGroupName],\n\t\t\t\td.[OrchestratorType],\n\t\t\t\td.[OrchestratorName],\n\t\t\t\tp.[PipelineName]\n\t\t\tFROM\n\t\t\t\t[procfwk].[Pipelines] p\n\t\t\t\tINNER JOIN [procfwk].[Stages] s\n\t\t\t\t\tON p.[StageId] = s.[StageId]\n\t\t\t\tINNER JOIN [procfwk].[Orchestrators] d\n\t\t\t\t\tON p.[OrchestratorId] = d.[OrchestratorId]\n\t\t\t\tINNER JOIN [procfwk].[BatchStageLink] b\n\t\t\t\t\tON b.[StageId] = s.[StageId]\n\t\t\tWHERE\n\t\t\t\tb.[BatchId] = @BatchId\n\t\t\t\tAND p.[Enabled] = 1\n\t\t\t\tAND s.[Enabled] = 1;\n\t\t\t\t\n\t\t\tSELECT\n\t\t\t\t@LocalExecutionId AS ExecutionId;\n\t\tEND;\n\n\tALTER INDEX [IDX_GetPipelinesInStage] ON [procfwk].[CurrentExecution]\n\tREBUILD;\nEND;"
},
{
"path": "MetadataDB/procfwk/Stored Procedures/ExecutePrecursorProcedure.sql",
"content": "CREATE PROCEDURE [procfwk].[ExecutePrecursorProcedure]\nAS\nBEGIN\n\tDECLARE @SQL VARCHAR(MAX) \n\tDECLARE @ErrorDetail NVARCHAR(MAX)\n\n\tIF OBJECT_ID([procfwk].[GetPropertyValueInternal]('ExecutionPrecursorProc')) IS NOT NULL\n\t\tBEGIN\n\t\t\tBEGIN TRY\n\t\t\t\tSET @SQL = [procfwk].[GetPropertyValueInternal]('ExecutionPrecursorProc');\n\t\t\t\tEXEC(@SQL);\n\t\t\tEND TRY\n\t\t\tBEGIN CATCH\n\t\t\t\tSELECT\n\t\t\t\t\t@ErrorDetail = 'Precursor procedure failed with error: ' + ERROR_MESSAGE();\n\n\t\t\t\tRAISERROR(@ErrorDetail,16,1);\n\t\t\tEND CATCH\n\t\tEND;\n\tELSE\n\t\tBEGIN\n\t\t\tPRINT 'Precursor object not found in database.';\n\t\tEND;\nEND;"
},
{
"path": "MetadataDB/procfwk/Stored Procedures/ExecutionWrapper.sql",
"content": "CREATE PROCEDURE [procfwk].[ExecutionWrapper]\n\t(\n\t@CallingOrchestratorName NVARCHAR(200) = NULL,\n\t@BatchName VARCHAR(255) = NULL\n\t)\nAS\nBEGIN\n\tSET NOCOUNT ON;\n\n\tDECLARE @RestartStatus BIT\n\tDECLARE @BatchId UNIQUEIDENTIFIER\n\tDECLARE @BLocalExecutionId UNIQUEIDENTIFIER --declared here for batches\n\n\tIF @CallingOrchestratorName IS NULL\n\t\tSET @CallingOrchestratorName = 'Unknown';\n\n\t--get restart overide property\t\n\tSELECT @RestartStatus = [procfwk].[GetPropertyValueInternal]('OverideRestart')\n\n\tIF([procfwk].[GetPropertyValueInternal]('UseExecutionBatches')) = '0'\n\t\tBEGIN\n\t\t\tSET @BatchId = NULL;\n\n\t\t\t--check for running execution\n\t\t\tIF EXISTS\n\t\t\t\t(\n\t\t\t\tSELECT * FROM [procfwk].[CurrentExecution] WHERE ISNULL([PipelineStatus],'') = 'Running'\n\t\t\t\t)\n\t\t\t\tBEGIN\n\t\t\t\t\tRAISERROR('There is already an execution run in progress. Stop this via the Orchestrator before restarting.',16,1);\n\t\t\t\t\tRETURN 0;\n\t\t\t\tEND;\t\n\n\t\t\t--reset and restart execution\n\t\t\tIF EXISTS\n\t\t\t\t(\n\t\t\t\tSELECT 1 FROM [procfwk].[CurrentExecution] WHERE ISNULL([PipelineStatus],'') <> 'Success'\n\t\t\t\t) \n\t\t\t\tAND @RestartStatus = 0\n\t\t\t\tBEGIN\n\t\t\t\t\tEXEC [procfwk].[ResetExecution]\n\t\t\t\tEND\n\t\t\t--capture failed execution and run new anyway\n\t\t\tELSE IF EXISTS\n\t\t\t\t(\n\t\t\t\tSELECT 1 FROM [procfwk].[CurrentExecution]\n\t\t\t\t)\n\t\t\t\tAND @RestartStatus = 1\n\t\t\t\tBEGIN\n\t\t\t\t\tEXEC [procfwk].[UpdateExecutionLog]\n\t\t\t\t\t\t@PerformErrorCheck = 0; --Special case when OverideRestart = 1;\n\n\t\t\t\t\tEXEC [procfwk].[CreateNewExecution] \n\t\t\t\t\t\t@CallingOrchestratorName = @CallingOrchestratorName\n\t\t\t\tEND\n\t\t\t--no restart considerations, just create new execution\n\t\t\tELSE\n\t\t\t\tBEGIN\n\t\t\t\t\tEXEC [procfwk].[CreateNewExecution] \n\t\t\t\t\t\t@CallingOrchestratorName = @CallingOrchestratorName\n\t\t\t\tEND\n\t\tEND\n\tELSE IF ([procfwk].[GetPropertyValueInternal]('UseExecutionBatches')) = '1'\n\t\tBEGIN\t\t\t\t\t\t\n\t\t\tIF @BatchName IS NULL\n\t\t\t\tBEGIN\n\t\t\t\t\tRAISERROR('A NULL batch name cannot be passed when the UseExecutionBatches property is set to 1 (true).',16,1);\n\t\t\t\t\tRETURN 0;\n\t\t\t\tEND;\n\t\t\t\n\t\t\tSELECT \n\t\t\t\t@BatchId = [BatchId]\n\t\t\tFROM\n\t\t\t\t[procfwk].[Batches]\n\t\t\tWHERE\n\t\t\t\t[BatchName] = @BatchName;\n\t\t\t\n\t\t\t--create local execution id now for the batch\n\t\t\tEXEC [procfwk].[BatchWrapper]\n\t\t\t\t@BatchId = @BatchId,\n\t\t\t\t@LocalExecutionId = @BLocalExecutionId OUTPUT;\n\t\t\t\t\n\t\t\t--reset and restart execution\n\t\t\tIF EXISTS\n\t\t\t\t(\n\t\t\t\tSELECT 1 \n\t\t\t\tFROM \n\t\t\t\t\t[procfwk].[CurrentExecution] \n\t\t\t\tWHERE \n\t\t\t\t\t[LocalExecutionId] = @BLocalExecutionId \n\t\t\t\t\tAND ISNULL([PipelineStatus],'') <> 'Success'\n\t\t\t\t) \n\t\t\t\tAND @RestartStatus = 0\n\t\t\t\tBEGIN\n\t\t\t\t\tEXEC [procfwk].[ResetExecution]\n\t\t\t\t\t\t@LocalExecutionId = @BLocalExecutionId;\n\t\t\t\tEND\n\t\t\t--capture failed execution and run new anyway\n\t\t\tELSE IF EXISTS\n\t\t\t\t(\n\t\t\t\tSELECT 1 \n\t\t\t\tFROM \n\t\t\t\t\t[procfwk].[CurrentExecution]\n\t\t\t\tWHERE\n\t\t\t\t\t[LocalExecutionId] = @BLocalExecutionId \n\t\t\t\t)\n\t\t\t\tAND @RestartStatus = 1\n\t\t\t\tBEGIN\n\t\t\t\t\tEXEC [procfwk].[UpdateExecutionLog]\n\t\t\t\t\t\t@PerformErrorCheck = 0, --Special case when OverideRestart = 1;\n\t\t\t\t\t\t@ExecutionId = @BLocalExecutionId;\n\n\t\t\t\t\tEXEC [procfwk].[CreateNewExecution]\n\t\t\t\t\t\t@CallingOrchestratorName = @CallingOrchestratorName,\n\t\t\t\t\t\t@LocalExecutionId = @BLocalExecutionId;\n\t\t\t\tEND\n\t\t\t--no restart considerations, just create new execution\n\t\t\tELSE\n\t\t\t\tBEGIN\n\t\t\t\t\tEXEC [procfwk].[CreateNewExecution] \n\t\t\t\t\t\t@CallingOrchestratorName = @CallingOrchestratorName,\n\t\t\t\t\t\t@LocalExecutionId = @BLocalExecutionId;\n\t\t\t\tEND\n\t\t\t\n\t\tEND\n\tELSE\n\t\tBEGIN\n\t\t\t--metadata integrity checks should mean this condition is never hit\n\t\t\tRAISERROR('Unknown batch handling configuration. Update properties with UseExecutionBatches value and try again.',16,1);\n\t\t\tRETURN 0;\n\t\tEND\nEND;"
},
{
"path": "MetadataDB/procfwk/Stored Procedures/GetEmailAlertParts.sql",
"content": "CREATE PROCEDURE [procfwk].[GetEmailAlertParts]\n\t(\n\t@PipelineId INT\n\t)\nAS\nBEGIN\n\tSET NOCOUNT ON;\n\n\tDECLARE @ToRecipients NVARCHAR(MAX) = ''\n\tDECLARE @CcRecipients NVARCHAR(MAX) = ''\n\tDECLARE @BccRecipients NVARCHAR(MAX) = ''\n\tDECLARE @EmailSubject NVARCHAR(500)\n\tDECLARE\t@EmailBody NVARCHAR(MAX)\n\tDECLARE @EmailImportance VARCHAR(5)\n\tDECLARE @OutcomeBitValue INT\n\n\t--map pipeline status to alert outcome bit value\n\tSELECT\n\t\t@OutcomeBitValue = ao.[BitValue]\n\tFROM\n\t\t[procfwk].[CurrentExecution] ce\n\t\tINNER JOIN [procfwk].[AlertOutcomes] ao\n\t\t\tON ce.[PipelineStatus] = ao.[PipelineOutcomeStatus]\n\tWHERE\n\t\tce.[PipelineId] = @PipelineId;\n\n\t--get to recipients\n\tSELECT\n\t\t@ToRecipients += r.[EmailAddress] + ','\n\tFROM\n\t\t[procfwk].[PipelineAlertLink] al\n\t\tINNER JOIN [procfwk].[Recipients] r\n\t\t\tON al.[RecipientId] = r.[RecipientId]\n\tWHERE\n\t\tal.[PipelineId] = @PipelineId\n\t\tAND al.[Enabled] = 1\n\t\tAND r.[Enabled] = 1\n\t\tAND UPPER(r.[MessagePreference]) = 'TO'\n\t\tAND (\n\t\t\tal.[OutcomesBitValue] & @OutcomeBitValue <> 0\n\t\t\tOR al.[OutcomesBitValue] & 1 <> 0 --all\n\t\t\t);\n\n\tIF (@ToRecipients <> '') SET @ToRecipients = LEFT(@ToRecipients,LEN(@ToRecipients)-1);\n\n\t--get cc recipients\n\tSELECT\n\t\t@CcRecipients += r.[EmailAddress] + ','\n\tFROM\n\t\t[procfwk].[PipelineAlertLink] al\n\t\tINNER JOIN [procfwk].[Recipients] r\n\t\t\tON al.[RecipientId] = r.[RecipientId]\n\tWHERE\n\t\tal.[PipelineId] = @PipelineId\n\t\tAND al.[Enabled] = 1\n\t\tAND r.[Enabled] = 1\n\t\tAND UPPER(r.[MessagePreference]) = 'CC'\n\t\tAND (\n\t\t\tal.[OutcomesBitValue] & @OutcomeBitValue <> 0\n\t\t\tOR al.[OutcomesBitValue] & 1 <> 0 --all\n\t\t\t);\n\t\n\tIF (@CcRecipients <> '') SET @CcRecipients = LEFT(@CcRecipients,LEN(@CcRecipients)-1);\n\n\t--get bcc recipients\n\tSELECT\n\t\t@BccRecipients += r.[EmailAddress] + ','\n\tFROM\n\t\t[procfwk].[PipelineAlertLink] al\n\t\tINNER JOIN [procfwk].[Recipients] r\n\t\t\tON al.[RecipientId] = r.[RecipientId]\n\tWHERE\n\t\tal.[PipelineId] = @PipelineId\n\t\tAND al.[Enabled] = 1\n\t\tAND r.[Enabled] = 1\n\t\tAND UPPER(r.[MessagePreference]) = 'BCC'\n\t\tAND (\n\t\t\tal.[OutcomesBitValue] & @OutcomeBitValue <> 0\n\t\t\tOR al.[OutcomesBitValue] & 1 <> 0 --all\n\t\t\t);\n\n\tIF (@BccRecipients <> '') SET @BccRecipients = LEFT(@BccRecipients,LEN(@BccRecipients)-1);\n\t\n\t--get email template\n\tSELECT\n\t\t@EmailBody = [PropertyValue]\n\tFROM\n\t\t[procfwk].[CurrentProperties]\n\tWHERE\n\t\t[PropertyName] = 'EmailAlertBodyTemplate';\n\n\t--set subject, body and importance\n\tSELECT TOP (1)\n\t\t--subject\n\t\t@EmailSubject = 'ProcFwk Alert: ' + [PipelineName] + ' - ' + [PipelineStatus],\n\t\n\t\t--body\n\t\t@EmailBody = REPLACE(@EmailBody,'##PipelineName###',[PipelineName]),\n\t\t@EmailBody = REPLACE(@EmailBody,'##Status###',[PipelineStatus]),\n\t\t@EmailBody = REPLACE(@EmailBody,'##ExecId###',CAST([LocalExecutionId] AS VARCHAR(36))),\n\t\t@EmailBody = REPLACE(@EmailBody,'##RunId###',CAST([PipelineRunId] AS VARCHAR(36))),\n\t\t@EmailBody = REPLACE(@EmailBody,'##StartDateTime###',CONVERT(VARCHAR(30), [StartDateTime], 120)),\n\t\t@EmailBody = CASE\n\t\t\t\t\t\tWHEN [EndDateTime] IS NULL THEN REPLACE(@EmailBody,'##EndDateTime###','N/A')\n\t\t\t\t\t\tELSE REPLACE(@EmailBody,'##EndDateTime###',CONVERT(VARCHAR(30), [EndDateTime], 120))\n\t\t\t\t\tEND,\n\t\t@EmailBody = CASE\n\t\t\t\t\t\tWHEN [EndDateTime] IS NULL THEN REPLACE(@EmailBody,'##Duration###','N/A')\n\t\t\t\t\t\tELSE REPLACE(@EmailBody,'##Duration###',CAST(DATEDIFF(MINUTE, [StartDateTime], [EndDateTime]) AS VARCHAR(30)))\n\t\t\t\t\tEND,\n\t\t@EmailBody = REPLACE(@EmailBody,'##CalledByOrc###',[CallingOrchestratorName]),\n\t\t@EmailBody = REPLACE(@EmailBody,'##ExecutedByOrcType###',[OrchestratorType]),\n\t\t@EmailBody = REPLACE(@EmailBody,'##ExecutedByOrc###',[OrchestratorName]),\n\n\t\t--importance\n\t\t@EmailImportance = \n\t\t\tCASE [PipelineStatus] \n\t\t\t\tWHEN 'Success' THEN 'Low'\n\t\t\t\tWHEN 'Failed' THEN 'High'\n\t\t\t\tELSE 'Normal'\n\t\t\tEND\n\tFROM\n\t\t[procfwk].[CurrentExecution]\n\tWHERE\n\t\t[PipelineId] = @PipelineId\n\tORDER BY\n\t\t[StartDateTime] DESC;\n\t\n\t--precaution\n\tIF @EmailBody IS NULL\n\t\tSET @EmailBody = 'Internal error. Failed to create profwk email alert body. Execute procedure [procfwk].[GetEmailAlertParts] with pipeline Id: ' + CAST(@PipelineId AS VARCHAR(30)) + ' to debug.';\n\n\t--return email parts\n\tSELECT\n\t\t@ToRecipients AS emailRecipients,\n\t\t@CcRecipients AS emailCcRecipients,\n\t\t@BccRecipients AS emailBccRecipients,\n\t\t@EmailSubject AS emailSubject,\n\t\t@EmailBody AS emailBody,\n\t\t@EmailImportance AS emailImportance;\nEND;"
},
{
"path": "MetadataDB/procfwk/Stored Procedures/GetFrameworkOrchestratorDetails.sql",
"content": "CREATE PROCEDURE [procfwk].[GetFrameworkOrchestratorDetails]\n\t(\n\t@CallingOrchestratorName NVARCHAR(200)\n\t)\nAS\nBEGIN\n\tSET NOCOUNT ON;\n\n\tDECLARE @FrameworkOrchestrator NVARCHAR(200)\n\n\t--defensive check\n\tSELECT\n\t\t@FrameworkOrchestrator = UPPER([OrchestratorName]),\n\t\t@CallingOrchestratorName = UPPER(@CallingOrchestratorName)\n\tFROM\n\t\t[procfwk].[Orchestrators]\n\tWHERE\n\t\t[IsFrameworkOrchestrator] = 1;\n\n\tIF(@FrameworkOrchestrator <> @CallingOrchestratorName)\n\tBEGIN\n\t\tRAISERROR('Orchestrator mismatch. Calling orchestrator does not match expected IsFrameworkOrchestrator name.',16,1);\n\t\tRETURN 0;\n\tEND\n\n\t--orchestrator detials\n\tSELECT\n\t\t[SubscriptionId],\n\t\t[ResourceGroupName],\n\t\t[OrchestratorName],\n\t\t[OrchestratorType]\n\tFROM\n\t\t[procfwk].[Orchestrators]\n\tWHERE\n\t\t[IsFrameworkOrchestrator] = 1;\nEND;"
},
{
"path": "MetadataDB/procfwk/Stored Procedures/GetPipelineParameters.sql",
"content": "CREATE PROCEDURE [procfwk].[GetPipelineParameters]\n\t(\n\t@PipelineId INT\n\t)\nAS\nBEGIN\n\tSET NOCOUNT ON;\n\n\tDECLARE @Json VARCHAR(MAX) = ''\n\n\t--get parameters if required for worker pipeline\n\tIF NOT EXISTS\n\t\t(\n\t\tSELECT \n\t\t\t[ParameterId] \n\t\tFROM \n\t\t\t[procfwk].[PipelineParameters] \n\t\tWHERE \n\t\t\t[PipelineId] = @PipelineId\n\t\t)\n\t\tBEGIN\n\t\t\tSET @Json = '' --Can't return NULL. Would break ADF expression.\n\t\tEND\n\tELSE\n\t\tBEGIN\n\t\t\tSELECT\n\t\t\t\t@Json += \n\t\t\t\t\t\tCASE\n\t\t\t\t\t\t\tWHEN [ParameterValue] IS NULL THEN '' --don't add pair so ADF uses default\n\t\t\t\t\t\t\tELSE '\"' + [ParameterName] + '\": \"' + STRING_ESCAPE([ParameterValue],'json') + '\",'\n\t\t\t\t\t\tEND\n\t\t\tFROM\n\t\t\t\t[procfwk].[PipelineParameters]\n\t\t\tWHERE\n\t\t\t\t[PipelineId] = @PipelineId;\n\t\t\t\n\t\t\t--handle parameter(s) with a NULL values\n\t\t\tIF LEN(@Json) > 0\n\t\t\tBEGIN\n\t\t\t\t--JSON snippet gets injected into Azure Function body request via Orchestrator expressions.\n\t\t\t\t--Comma used to support Orchestrator expression.\n\t\t\t\tSET @Json = ',\"pipelineParameters\": {' + LEFT(@Json,LEN(@Json)-1) + '}'\n\n\t\t\t\t--update current execution log if this is a runtime request\n\t\t\t\tUPDATE\n\t\t\t\t\t[procfwk].[CurrentExecution]\n\t\t\t\tSET\n\t\t\t\t\t--add extra braces to make JSON string valid in logs\n\t\t\t\t\t[PipelineParamsUsed] = '{ ' + RIGHT(@Json,LEN(@Json)-1) + ' }'\n\t\t\t\tWHERE\n\t\t\t\t\t[PipelineId] = @PipelineId;\n\n\t\t\t\t--set last values values\n\t\t\t\tUPDATE\n\t\t\t\t\t[procfwk].[PipelineParameters]\n\t\t\t\tSET\n\t\t\t\t\t[ParameterValueLastUsed] = [ParameterValue]\n\t\t\t\tWHERE\n\t\t\t\t\t[PipelineId] = @PipelineId;\n\t\t\tEND;\n\t\tEND;\n\n\t--return JSON snippet\n\tSELECT @Json AS Params\nEND;"
},
{
"path": "MetadataDB/procfwk/Stored Procedures/GetPipelinesInStage.sql",
"content": "CREATE PROCEDURE [procfwk].[GetPipelinesInStage]\n\t(\n\t@ExecutionId UNIQUEIDENTIFIER,\n\t@StageId INT\n\t)\nAS\nBEGIN\n\tSET NOCOUNT ON;\n\n\tSELECT \n\t\t[PipelineId]\n\tFROM \n\t\t[procfwk].[CurrentExecution]\n\tWHERE \n\t\t[LocalExecutionId] = @ExecutionId\n\t\tAND [StageId] = @StageId\n\t\tAND ISNULL([PipelineStatus],'') <> 'Success'\n\t\tAND [IsBlocked] <> 1\n\tORDER BY\n\t\t[PipelineId] ASC;\nEND;"
},
{
"path": "MetadataDB/procfwk/Stored Procedures/GetPropertyValue.sql",
"content": "CREATE PROCEDURE [procfwk].[GetPropertyValue]\n\t(\n\t@PropertyName VARCHAR(128)\n\t)\nAS\nBEGIN\t\n\tDECLARE @ErrorDetail NVARCHAR(4000) = ''\n\n\t--defensive checks\n\tIF NOT EXISTS\n\t\t(\n\t\tSELECT * FROM [procfwk].[Properties] WHERE [PropertyName] = @PropertyName\n\t\t)\n\t\tBEGIN\n\t\t\tSET @ErrorDetail = 'Invalid property name provided. Property does not exist.'\n\t\t\tRAISERROR(@ErrorDetail, 16, 1);\n\t\t\tRETURN 0;\n\t\tEND\n\tELSE IF NOT EXISTS\n\t\t(\n\t\tSELECT * FROM [procfwk].[Properties] WHERE [PropertyName] = @PropertyName AND [ValidTo] IS NULL\n\t\t)\n\t\tBEGIN\n\t\t\tSET @ErrorDetail = 'Property name provided does not have a current valid version of the required value.'\n\t\t\tRAISERROR(@ErrorDetail, 16, 1);\n\t\t\tRETURN 0;\n\t\tEND\n\t--get valid property value\n\tELSE\n\t\tBEGIN\n\t\t\tSELECT\n\t\t\t\t[PropertyValue]\n\t\t\tFROM\n\t\t\t\t[procfwk].[CurrentProperties]\n\t\t\tWHERE\n\t\t\t\t[PropertyName] = @PropertyName\n\t\tEND\nEND;"
},
{
"path": "MetadataDB/procfwk/Stored Procedures/GetStages.sql",
"content": "CREATE PROCEDURE [procfwk].[GetStages]\n\t(\n\t@ExecutionId UNIQUEIDENTIFIER\n\t)\nAS\nBEGIN\n\tSET NOCOUNT ON;\n\n\t--defensive check\n\tIF NOT EXISTS \n\t\t( \n\t\tSELECT\n\t\t\t1\n\t\tFROM \n\t\t\t[procfwk].[CurrentExecution]\n\t\tWHERE\n\t\t\t[LocalExecutionId] = @ExecutionId\n\t\t\tAND ISNULL([PipelineStatus],'') <> 'Success'\n\t\t)\n\t\tBEGIN\n\t\t\tRAISERROR('Requested execution run does not contain any enabled stages/pipelines.',16,1);\n\t\t\tRETURN 0;\n\t\tEND;\n\n\tSELECT DISTINCT \n\t\t[StageId] \n\tFROM \n\t\t[procfwk].[CurrentExecution]\n\tWHERE\n\t\t[LocalExecutionId] = @ExecutionId\n\t\tAND ISNULL([PipelineStatus],'') <> 'Success'\n\tORDER BY \n\t\t[StageId] ASC\nEND;"
},
{
"path": "MetadataDB/procfwk/Stored Procedures/GetWorkerAuthDetails.sql",
"content": "CREATE PROCEDURE [procfwk].[GetWorkerAuthDetails]\n\t(\n\t@ExecutionId UNIQUEIDENTIFIER,\n\t@StageId INT,\n\t@PipelineId INT\n\t)\nAS\nBEGIN\n\tSET NOCOUNT ON;\n\n\tDECLARE @TenId NVARCHAR(MAX)\n\tDECLARE @SubId NVARCHAR(MAX)\n\tDECLARE @AppId NVARCHAR(MAX)\n\tDECLARE @AppSecret NVARCHAR(MAX)\n\n\tDECLARE @OrchestratorName NVARCHAR(200)\n\tDECLARE @OrchestratorType CHAR(3)\n\tDECLARE @PipelineName NVARCHAR(200)\n\n\tSELECT \n\t\t@PipelineName = [PipelineName],\n\t\t@OrchestratorName = [OrchestratorName],\n\t\t@OrchestratorType = [OrchestratorType]\n\tFROM \n\t\t[procfwk].[CurrentExecution]\n\tWHERE \n\t\t[LocalExecutionId] = @ExecutionId\n\t\tAND [StageId] = @StageId\n\t\tAND [PipelineId] = @PipelineId;\n\t\t\n\n\tIF ([procfwk].[GetPropertyValueInternal]('SPNHandlingMethod')) = 'StoreInDatabase'\n\t\tBEGIN\n\t\t\t--get auth details regardless of being pipeline specific and regardless of a pipeline param being passed\n\t\t\t;WITH cte AS\n\t\t\t\t(\n\t\t\t\tSELECT DISTINCT\n\t\t\t\t\tSub.[TenantId],\n\t\t\t\t\tSub.[SubscriptionId],\n\t\t\t\t\tS.[PrincipalId] AS AppId,\n\t\t\t\t\tCAST(DECRYPTBYPASSPHRASE(CONCAT(@OrchestratorName, @OrchestratorType, @PipelineName), S.[PrincipalSecret]) AS NVARCHAR(MAX)) AS AppSecret\n\t\t\t\tFROM\n\t\t\t\t\t[dbo].[ServicePrincipals] S\n\t\t\t\t\tINNER JOIN [procfwk].[PipelineAuthLink] L\n\t\t\t\t\t\tON S.[CredentialId] = L.[CredentialId]\n\t\t\t\t\tINNER JOIN [procfwk].[Pipelines] P\n\t\t\t\t\t\tON L.[PipelineId] = P.[PipelineId]\n\t\t\t\t\tINNER JOIN [procfwk].[Orchestrators] D\n\t\t\t\t\t\tON P.[OrchestratorId] = D.[OrchestratorId]\n\t\t\t\t\t\t\tAND L.[OrchestratorId] = D.[OrchestratorId]\n\t\t\t\t\tINNER JOIN [procfwk].[Subscriptions] Sub\n\t\t\t\t\t\tON D.[SubscriptionId] = Sub.[SubscriptionId]\n\t\t\t\tWHERE\n\t\t\t\t\tP.[PipelineName] = @PipelineName\n\t\t\t\t\tAND D.[OrchestratorName] = @OrchestratorName\n\t\t\t\t\tAND D.[OrchestratorType] = @OrchestratorType\n\t\t\t\n\t\t\t\tUNION\n\n\t\t\t\tSELECT DISTINCT\n\t\t\t\t\tSub.[TenantId],\n\t\t\t\t\tSub.[SubscriptionId],\t\t\t\t\t\n\t\t\t\t\tS.[PrincipalId] AS AppId,\n\t\t\t\t\tCAST(DECRYPTBYPASSPHRASE(CONCAT(@OrchestratorName, @OrchestratorType), S.[PrincipalSecret]) AS NVARCHAR(MAX)) AS AppSecret\n\t\t\t\tFROM\n\t\t\t\t\t[dbo].[ServicePrincipals] S\n\t\t\t\t\tINNER JOIN [procfwk].[PipelineAuthLink] L\n\t\t\t\t\t\tON S.[CredentialId] = L.[CredentialId]\n\t\t\t\t\tINNER JOIN [procfwk].[Orchestrators] D\n\t\t\t\t\t\tON L.[OrchestratorId] = D.[OrchestratorId]\n\t\t\t\t\tINNER JOIN [procfwk].[Subscriptions] Sub\n\t\t\t\t\t\tON D.[SubscriptionId] = Sub.[SubscriptionId]\n\t\t\t\tWHERE\n\t\t\t\t\tD.[OrchestratorName] = @OrchestratorName\n\t\t\t\t\tAND D.[OrchestratorType] = @OrchestratorType\n\t\t\t\t)\n\t\t\tSELECT TOP 1\n\t\t\t\t@TenId = [TenantId],\n\t\t\t\t@SubId = [SubscriptionId],\n\t\t\t\t@AppId = [AppId],\n\t\t\t\t@AppSecret = [AppSecret]\n\t\t\tFROM\n\t\t\t\tcte\n\t\t\tWHERE\n\t\t\t\t[AppSecret] IS NOT NULL\n\t\tEND\n\tELSE IF ([procfwk].[GetPropertyValueInternal]('SPNHandlingMethod')) = 'StoreInKeyVault'\n\t\tBEGIN\n\t\t\t\n\t\t\t--get auth details regardless of being pipeline specific and regardless of a pipeline param being passed\n\t\t\t;WITH cte AS\n\t\t\t\t(\n\t\t\t\tSELECT DISTINCT\n\t\t\t\t\tSub.[TenantId],\n\t\t\t\t\tSub.[SubscriptionId],\t\t\t\t\t\t\n\t\t\t\t\tS.[PrincipalIdUrl] AS AppId,\n\t\t\t\t\tS.[PrincipalSecretUrl] AS AppSecret\n\t\t\t\tFROM\n\t\t\t\t\t[dbo].[ServicePrincipals] S\n\t\t\t\t\tINNER JOIN [procfwk].[PipelineAuthLink] L\n\t\t\t\t\t\tON S.[CredentialId] = L.[CredentialId]\n\t\t\t\t\tINNER JOIN [procfwk].[Pipelines] P\n\t\t\t\t\t\tON L.[PipelineId] = P.[PipelineId]\n\t\t\t\t\tINNER JOIN [procfwk].[Orchestrators] D\n\t\t\t\t\t\tON P.[OrchestratorId] = D.[OrchestratorId]\n\t\t\t\t\t\t\tAND L.[OrchestratorId] = D.[OrchestratorId]\n\t\t\t\t\tINNER JOIN [procfwk].[Subscriptions] Sub\n\t\t\t\t\t\tON D.[SubscriptionId] = Sub.[SubscriptionId]\n\t\t\t\tWHERE\n\t\t\t\t\tP.[PipelineName] = @PipelineName\n\t\t\t\t\tAND D.[OrchestratorName] = @OrchestratorName\n\t\t\t\t\tAND D.[OrchestratorType] = @OrchestratorType\n\t\t\t\n\t\t\t\tUNION\n\n\t\t\t\tSELECT DISTINCT\n\t\t\t\t\tSub.[TenantId],\n\t\t\t\t\tSub.[SubscriptionId],\t\t\t\t\t\n\t\t\t\t\tS.[PrincipalIdUrl] AS AppId,\n\t\t\t\t\tS.[PrincipalSecretUrl] AS AppSecret\n\t\t\t\tFROM\n\t\t\t\t\t[dbo].[ServicePrincipals] S\n\t\t\t\t\tINNER JOIN [procfwk].[PipelineAuthLink] L\n\t\t\t\t\t\tON S.[CredentialId] = L.[CredentialId]\n\t\t\t\t\tINNER JOIN [procfwk].[Orchestrators] D\n\t\t\t\t\t\tON L.[OrchestratorId] = D.[OrchestratorId]\n\t\t\t\t\tINNER JOIN [procfwk].[Subscriptions] Sub\n\t\t\t\t\t\tON D.[SubscriptionId] = Sub.[SubscriptionId]\n\t\t\t\tWHERE\n\t\t\t\t\tD.[OrchestratorName] = @OrchestratorName\n\t\t\t\t\tAND D.[OrchestratorType] = @OrchestratorType\n\t\t\t\t)\n\t\t\tSELECT TOP 1\n\t\t\t\t@TenId = [TenantId],\n\t\t\t\t@SubId = [SubscriptionId],\n\t\t\t\t@AppId = [AppId],\n\t\t\t\t@AppSecret = [AppSecret]\n\t\t\tFROM\n\t\t\t\tcte\n\t\t\tWHERE\n\t\t\t\t[AppSecret] IS NOT NULL\n\t\tEND\n\tELSE\n\t\tBEGIN\n\t\t\tRAISERROR('Unknown SPN retrieval method.',16,1);\n\t\t\tRETURN 0;\n\t\tEND\n\n\t--return usable values\n\tSELECT\n\t\t@TenId AS TenantId,\n\t\t@SubId AS SubscriptionId,\n\t\t@AppId AS AppId,\n\t\t@AppSecret AS AppSecret\nEND;"
},
{
"path": "MetadataDB/procfwk/Stored Procedures/GetWorkerDetailsWrapper.sql",
"content": "CREATE PROCEDURE [procfwk].[GetWorkerDetailsWrapper]\n\t(\n\t@ExecutionId UNIQUEIDENTIFIER,\n\t@StageId INT,\n\t@PipelineId INT\n\t)\nAS\nBEGIN\n\t/*\n\tCreated this proc just to reduce and refactor the number of pipeline activity \n\tcalls needed due to the Microsoft enforced limit of 40 activities per pipeline.\n\t*/\n\tSET NOCOUNT ON;\n\n\tDECLARE @WorkerAuthDetails TABLE\n\t\t(\n\t\t[tenantId] UNIQUEIDENTIFIER NULL,\n\t\t[applicationId] NVARCHAR(MAX) NULL,\n\t\t[authenticationKey] NVARCHAR(MAX) NULL,\n\t\t[subscriptionId] UNIQUEIDENTIFIER NULL\n\t\t)\n\n\tDECLARE @WorkerDetails TABLE\n\t\t(\n\t\t[resourceGroupName] NVARCHAR(200) NULL,\n\t\t[orchestratorName] NVARCHAR(200) NULL,\n\t\t[orchestratorType] CHAR(3) NULL,\n\t\t[pipelineName] NVARCHAR(200) NULL\n\t\t)\n\n\t--get work auth details\n\tINSERT INTO @WorkerAuthDetails\n\t\t(\n\t\t[tenantId],\n\t\t[subscriptionId],\n\t\t[applicationId],\n\t\t[authenticationKey]\n\t\t)\n\tEXEC [procfwk].[GetWorkerAuthDetails]\n\t\t@ExecutionId = @ExecutionId,\n\t\t@StageId = @StageId,\n\t\t@PipelineId = @PipelineId;\n\t\n\t--get main worker details\n\tINSERT INTO @WorkerDetails\n\t\t(\n\t\t[pipelineName],\n\t\t[orchestratorName],\n\t\t[orchestratorType],\n\t\t[resourceGroupName]\n\t\t)\n\tEXEC [procfwk].[GetWorkerPipelineDetails]\n\t\t@ExecutionId = @ExecutionId,\n\t\t@StageId = @StageId,\n\t\t@PipelineId = @PipelineId;\t\t\n\t\n\t--return all details\n\tSELECT \n\t\tad.[tenantId],\n\t\tad.[applicationId],\n\t\tad.[authenticationKey],\t\t\n\t\tad.[subscriptionId],\n\t\td.[resourceGroupName],\n\t\td.[orchestratorName],\n\t\td.[orchestratorType],\n\t\td.[pipelineName]\n\tFROM \n\t\t@WorkerDetails d \n\t\tCROSS JOIN @WorkerAuthDetails ad;\nEND;"
},
{
"path": "MetadataDB/procfwk/Stored Procedures/GetWorkerPipelineDetails.sql",
"content": "CREATE PROCEDURE [procfwk].[GetWorkerPipelineDetails]\n\t(\n\t@ExecutionId UNIQUEIDENTIFIER,\n\t@StageId INT,\n\t@PipelineId INT\n\t)\nAS\nBEGIN\n\tSET NOCOUNT ON;\n\n\tSELECT \n\t\t[PipelineName],\n\t\t[OrchestratorName],\n\t\t[OrchestratorType],\n\t\t[ResourceGroupName]\n\tFROM \n\t\t[procfwk].[CurrentExecution]\n\tWHERE \n\t\t[LocalExecutionId] = @ExecutionId\n\t\tAND [StageId] = @StageId\n\t\tAND [PipelineId] = @PipelineId;\nEND;"
},
{
"path": "MetadataDB/procfwk/Stored Procedures/ResetExecution.sql",
"content": "CREATE PROCEDURE [procfwk].[ResetExecution]\n\t(\n\t@LocalExecutionId UNIQUEIDENTIFIER = NULL\n\t)\nAS\nBEGIN \n\tSET NOCOUNT\tON;\n\n\tIF([procfwk].[GetPropertyValueInternal]('UseExecutionBatches')) = '0'\n\t\tBEGIN\n\t\t\t--capture any pipelines that might be in an unexpected state\n\t\t\tINSERT INTO [procfwk].[ExecutionLog]\n\t\t\t\t(\n\t\t\t\t[LocalExecutionId],\n\t\t\t\t[StageId],\n\t\t\t\t[PipelineId],\n\t\t\t\t[CallingOrchestratorName],\n\t\t\t\t[ResourceGroupName],\n\t\t\t\t[OrchestratorType],\n\t\t\t\t[OrchestratorName],\n\t\t\t\t[PipelineName],\n\t\t\t\t[StartDateTime],\n\t\t\t\t[PipelineStatus],\n\t\t\t\t[EndDateTime]\n\t\t\t\t)\n\t\t\tSELECT\n\t\t\t\t[LocalExecutionId],\n\t\t\t\t[StageId],\n\t\t\t\t[PipelineId],\n\t\t\t\t[CallingOrchestratorName],\n\t\t\t\t[ResourceGroupName],\n\t\t\t\t[OrchestratorType],\n\t\t\t\t[OrchestratorName],\n\t\t\t\t[PipelineName],\n\t\t\t\t[StartDateTime],\n\t\t\t\t'Unknown',\n\t\t\t\t[EndDateTime]\n\t\t\tFROM\n\t\t\t\t[procfwk].[CurrentExecution]\n\t\t\tWHERE\n\t\t\t\t--these are predicted states\n\t\t\t\t[PipelineStatus] NOT IN\n\t\t\t\t\t(\n\t\t\t\t\t'Success',\n\t\t\t\t\t'Failed',\n\t\t\t\t\t'Blocked',\n\t\t\t\t\t'Cancelled'\n\t\t\t\t\t);\n\t\t\n\t\t\t--reset status ready for next attempt\n\t\t\tUPDATE\n\t\t\t\t[procfwk].[CurrentExecution]\n\t\t\tSET\n\t\t\t\t[StartDateTime] = NULL,\n\t\t\t\t[EndDateTime] = NULL,\n\t\t\t\t[PipelineStatus] = NULL,\n\t\t\t\t[LastStatusCheckDateTime] = NULL,\n\t\t\t\t[PipelineRunId] = NULL,\n\t\t\t\t[PipelineParamsUsed] = NULL,\n\t\t\t\t[IsBlocked] = 0\n\t\t\tWHERE\n\t\t\t\tISNULL([PipelineStatus],'') <> 'Success'\n\t\t\t\tOR [IsBlocked] = 1;\n\t\n\t\t\t--return current execution id\n\t\t\tSELECT DISTINCT\n\t\t\t\t[LocalExecutionId] AS ExecutionId\n\t\t\tFROM\n\t\t\t\t[procfwk].[CurrentExecution];\n\t\tEND\n\tELSE IF ([procfwk].[GetPropertyValueInternal]('UseExecutionBatches')) = '1'\n\t\tBEGIN\n\t\t\t--capture any pipelines that might be in an unexpected state\n\t\t\tINSERT INTO [procfwk].[ExecutionLog]\n\t\t\t\t(\n\t\t\t\t[LocalExecutionId],\n\t\t\t\t[StageId],\n\t\t\t\t[PipelineId],\n\t\t\t\t[CallingOrchestratorName],\n\t\t\t\t[ResourceGroupName],\n\t\t\t\t[OrchestratorType],\n\t\t\t\t[OrchestratorName],\n\t\t\t\t[PipelineName],\n\t\t\t\t[StartDateTime],\n\t\t\t\t[PipelineStatus],\n\t\t\t\t[EndDateTime]\n\t\t\t\t)\n\t\t\tSELECT\n\t\t\t\t[LocalExecutionId],\n\t\t\t\t[StageId],\n\t\t\t\t[PipelineId],\n\t\t\t\t[CallingOrchestratorName],\n\t\t\t\t[ResourceGroupName],\n\t\t\t\t[OrchestratorType],\n\t\t\t\t[OrchestratorName],\n\t\t\t\t[PipelineName],\n\t\t\t\t[StartDateTime],\n\t\t\t\t'Unknown',\n\t\t\t\t[EndDateTime]\n\t\t\tFROM\n\t\t\t\t[procfwk].[CurrentExecution]\n\t\t\tWHERE\n\t\t\t\t[LocalExecutionId] = @LocalExecutionId\n\t\t\t\t--these are predicted states\n\t\t\t\tAND [PipelineStatus] NOT IN\n\t\t\t\t\t(\n\t\t\t\t\t'Success',\n\t\t\t\t\t'Failed',\n\t\t\t\t\t'Blocked',\n\t\t\t\t\t'Cancelled'\n\t\t\t\t\t);\n\t\t\n\t\t\t--reset status ready for next attempt\n\t\t\tUPDATE\n\t\t\t\t[procfwk].[CurrentExecution]\n\t\t\tSET\n\t\t\t\t[StartDateTime] = NULL,\n\t\t\t\t[EndDateTime] = NULL,\n\t\t\t\t[PipelineStatus] = NULL,\n\t\t\t\t[LastStatusCheckDateTime] = NULL,\n\t\t\t\t[PipelineRunId] = NULL,\n\t\t\t\t[PipelineParamsUsed] = NULL,\n\t\t\t\t[IsBlocked] = 0\n\t\t\tWHERE\n\t\t\t\t[LocalExecutionId] = @LocalExecutionId\n\t\t\t\tAND ISNULL([PipelineStatus],'') <> 'Success'\n\t\t\t\tOR [IsBlocked] = 1;\n\t\t\t\t\n\t\t\tUPDATE\n\t\t\t\t[procfwk].[BatchExecution]\n\t\t\tSET\n\t\t\t\t[EndDateTime] = NULL,\n\t\t\t\t[BatchStatus] = 'Running'\n\t\t\tWHERE\n\t\t\t\t[ExecutionId] = @LocalExecutionId;\n\n\t\t\tSELECT \n\t\t\t\t@LocalExecutionId AS ExecutionId\n\t\tEND;\nEND;"
},
{
"path": "MetadataDB/procfwk/Stored Procedures/SetErrorLogDetails.sql",
"content": "CREATE PROCEDURE [procfwk].[SetErrorLogDetails]\n\t(\n\t@LocalExecutionId UNIQUEIDENTIFIER,\n\t@JsonErrorDetails VARCHAR(MAX)\n\t)\nAS\nBEGIN\n\tSET NOCOUNT ON;\n\n\tINSERT INTO [procfwk].[ErrorLog]\n\t\t(\n\t\t[LocalExecutionId],\n\t\t[PipelineRunId],\n\t\t[ActivityRunId],\n\t\t[ActivityName],\n\t\t[ActivityType],\n\t\t[ErrorCode],\n\t\t[ErrorType],\n\t\t[ErrorMessage]\n\t\t)\n\tSELECT\n\t\t@LocalExecutionId,\n\t\tBase.[RunId],\n\t\tErrorDetail.[ActivityRunId],\n\t\tErrorDetail.[ActivityName],\n\t\tErrorDetail.[ActivityType],\n\t\tErrorDetail.[ErrorCode],\n\t\tErrorDetail.[ErrorType],\n\t\tErrorDetail.[ErrorMessage]\n\tFROM \n\t\tOPENJSON(@JsonErrorDetails) WITH\n\t\t\t( \n\t\t\t[RunId] UNIQUEIDENTIFIER,\n\t\t\t[Errors] NVARCHAR(MAX) AS JSON\n\t\t\t) AS Base\n\t\tCROSS APPLY OPENJSON (Base.[Errors]) WITH\n\t\t\t(\n\t\t\t[ActivityRunId] UNIQUEIDENTIFIER,\n\t\t\t[ActivityName] VARCHAR(100),\n\t\t\t[ActivityType] VARCHAR(100),\n\t\t\t[ErrorCode] VARCHAR(100),\n\t\t\t[ErrorType] VARCHAR(100),\n\t\t\t[ErrorMessage] VARCHAR(MAX)\n\t\t\t) AS ErrorDetail\nEND;"
},
{
"path": "MetadataDB/procfwk/Stored Procedures/SetExecutionBlockDependants.sql",
"content": "CREATE PROCEDURE [procfwk].[SetExecutionBlockDependants]\n\t(\n\t@ExecutionId UNIQUEIDENTIFIER = NULL,\n\t@PipelineId INT\n\t)\nAS\nBEGIN\n\t--update dependents status\n\tUPDATE\n\t\tce\n\tSET\n\t\tce.[PipelineStatus] = 'Blocked',\n\t\tce.[IsBlocked] = 1\n\tFROM\n\t\t[procfwk].[PipelineDependencies] pe\n\t\tINNER JOIN [procfwk].[CurrentExecution] ce\n\t\t\tON pe.[DependantPipelineId] = ce.[PipelineId]\n\tWHERE\n\t\tce.[LocalExecutionId] = @ExecutionId\n\t\tAND pe.[PipelineId] = @PipelineId\nEND;"
},
{
"path": "MetadataDB/procfwk/Stored Procedures/SetLogActivityFailed.sql",
"content": "CREATE PROCEDURE [procfwk].[SetLogActivityFailed]\n\t(\n\t@ExecutionId UNIQUEIDENTIFIER,\n\t@StageId INT,\n\t@PipelineId INT,\n\t@CallingActivity VARCHAR(255)\n\t)\nAS\n\nBEGIN\n\tSET NOCOUNT ON;\n\t\n\t--mark specific failure pipeline\n\tUPDATE\n\t\t[procfwk].[CurrentExecution]\n\tSET\n\t\t[PipelineStatus] = @CallingActivity + 'Error'\n\tWHERE\n\t\t[LocalExecutionId] = @ExecutionId\n\t\tAND [StageId] = @StageId\n\t\tAND [PipelineId] = @PipelineId\n\n\t--persist failed pipeline records to long term log\n\tINSERT INTO [procfwk].[ExecutionLog]\n\t\t(\n\t\t[LocalExecutionId],\n\t\t[StageId],\n\t\t[PipelineId],\n\t\t[CallingOrchestratorName],\n\t\t[ResourceGroupName],\n\t\t[OrchestratorType],\n\t\t[OrchestratorName],\n\t\t[PipelineName],\n\t\t[StartDateTime],\n\t\t[PipelineStatus],\n\t\t[EndDateTime],\n\t\t[PipelineRunId],\n\t\t[PipelineParamsUsed]\n\t\t)\n\tSELECT\n\t\t[LocalExecutionId],\n\t\t[StageId],\n\t\t[PipelineId],\n\t\t[CallingOrchestratorName],\n\t\t[ResourceGroupName],\n\t\t[OrchestratorType],\n\t\t[OrchestratorName],\n\t\t[PipelineName],\n\t\t[StartDateTime],\n\t\t[PipelineStatus],\n\t\t[EndDateTime],\n\t\t[PipelineRunId],\n\t\t[PipelineParamsUsed]\n\tFROM\n\t\t[procfwk].[CurrentExecution]\n\tWHERE\n\t\t[LocalExecutionId] = @ExecutionId\n\t\tAND [PipelineStatus] = @CallingActivity + 'Error'\n\t\tAND [StageId] = @StageId\n\t\tAND [PipelineId] = @PipelineId\n\t\n\t--decide how to proceed with error/failure depending on framework property configuration\n\tIF ([procfwk].[GetPropertyValueInternal]('FailureHandling')) = 'None'\n\t\tBEGIN\n\t\t\t--do nothing allow processing to carry on regardless\n\t\t\tRETURN 0;\n\t\tEND;\n\t\t\n\tELSE IF ([procfwk].[GetPropertyValueInternal]('FailureHandling')) = 'Simple'\n\t\tBEGIN\n\t\t\t--flag all downstream stages as blocked\n\t\t\tUPDATE\n\t\t\t\t[procfwk].[CurrentExecution]\n\t\t\tSET\n\t\t\t\t[PipelineStatus] = 'Blocked',\n\t\t\t\t[IsBlocked] = 1\n\t\t\tWHERE\n\t\t\t\t[LocalExecutionId] = @ExecutionId\n\t\t\t\tAND [StageId] > @StageId;\n\n\t\t\t--update batch if applicable\n\t\t\tIF ([procfwk].[GetPropertyValueInternal]('UseExecutionBatches')) = '1'\n\t\t\t\tBEGIN\n\t\t\t\t\tUPDATE\n\t\t\t\t\t\t[procfwk].[BatchExecution]\n\t\t\t\t\tSET\n\t\t\t\t\t\t[BatchStatus] = 'Stopping' --special case when its an activity failure to call stop ready for restart\n\t\t\t\t\tWHERE\n\t\t\t\t\t\t[ExecutionId] = @ExecutionId\n\t\t\t\t\t\tAND [BatchStatus] = 'Running';\n\t\t\t\tEND;\t\t\t\n\t\tEND;\n\t\n\tELSE IF ([procfwk].[GetPropertyValueInternal]('FailureHandling')) = 'DependencyChain'\n\t\tBEGIN\n\t\t\tEXEC [procfwk].[SetExecutionBlockDependants]\n\t\t\t\t@ExecutionId = @ExecutionId,\n\t\t\t\t@PipelineId = @PipelineId\n\t\tEND;\n\tELSE\n\t\tBEGIN\n\t\t\tRAISERROR('Unknown failure handling state.',16,1);\n\t\t\tRETURN 0;\n\t\tEND;\nEND;"
},
{
"path": "MetadataDB/procfwk/Stored Procedures/SetLogPipelineCancelled.sql",
"content": "CREATE PROCEDURE [procfwk].[SetLogPipelineCancelled]\n\t(\n\t@ExecutionId UNIQUEIDENTIFIER,\n\t@StageId INT,\n\t@PipelineId INT,\n\t@CleanUpRun BIT = 0\n\t)\nAS\nBEGIN\n\tSET NOCOUNT ON;\n\n\tDECLARE @ErrorDetail VARCHAR(500);\n\n\t--mark specific failure pipeline\n\tUPDATE\n\t\t[procfwk].[CurrentExecution]\n\tSET\n\t\t[PipelineStatus] = 'Cancelled'\n\tWHERE\n\t\t[LocalExecutionId] = @ExecutionId\n\t\tAND [StageId] = @StageId\n\t\tAND [PipelineId] = @PipelineId\n\t\n\t--no need to block and log if done during a clean up cycle\n\tIF @CleanUpRun = 1 RETURN 0;\n\n\t--persist cancelled pipeline records to long term log\n\tINSERT INTO [procfwk].[ExecutionLog]\n\t\t(\n\t\t[LocalExecutionId],\n\t\t[StageId],\n\t\t[PipelineId],\n\t\t[CallingOrchestratorName],\n\t\t[ResourceGroupName],\n\t\t[OrchestratorType],\n\t\t[OrchestratorName],\n\t\t[PipelineName],\n\t\t[StartDateTime],\n\t\t[PipelineStatus],\n\t\t[EndDateTime],\n\t\t[PipelineRunId],\n\t\t[PipelineParamsUsed]\n\t\t)\n\tSELECT\n\t\t[LocalExecutionId],\n\t\t[StageId],\n\t\t[PipelineId],\n\t\t[CallingOrchestratorName],\n\t\t[ResourceGroupName],\n\t\t[OrchestratorType],\n\t\t[OrchestratorName],\n\t\t[PipelineName],\n\t\t[StartDateTime],\n\t\t[PipelineStatus],\n\t\t[EndDateTime],\n\t\t[PipelineRunId],\n\t\t[PipelineParamsUsed]\n\tFROM\n\t\t[procfwk].[CurrentExecution]\n\tWHERE\n\t\t[LocalExecutionId] = @ExecutionId\n\t\tAND [PipelineStatus] = 'Cancelled'\n\t\tAND [StageId] = @StageId\n\t\tAND [PipelineId] = @PipelineId;\n\n\t--block down stream stages?\n\tIF ([procfwk].[GetPropertyValueInternal]('CancelledWorkerResultBlocks')) = 1\n\tBEGIN\t\n\t\t--decide how to proceed with error/failure depending on framework property configuration\n\t\tIF ([procfwk].[GetPropertyValueInternal]('FailureHandling')) = 'None'\n\t\t\tBEGIN\n\t\t\t\t--do nothing allow processing to carry on regardless\n\t\t\t\tRETURN 0;\n\t\t\tEND;\n\n\t\tELSE IF ([procfwk].[GetPropertyValueInternal]('FailureHandling')) = 'Simple'\n\t\t\tBEGIN\n\t\t\t\t--flag all downstream stages as blocked\n\t\t\t\tUPDATE\n\t\t\t\t\t[procfwk].[CurrentExecution]\n\t\t\t\tSET\n\t\t\t\t\t[PipelineStatus] = 'Blocked',\n\t\t\t\t\t[IsBlocked] = 1\n\t\t\t\tWHERE\n\t\t\t\t\t[LocalExecutionId] = @ExecutionId\n\t\t\t\t\tAND [StageId] > @StageId\n\t\t\t\t\n\t\t\t\t--update batch if applicable\n\t\t\t\tIF ([procfwk].[GetPropertyValueInternal]('UseExecutionBatches')) = '1'\n\t\t\t\t\tBEGIN\n\t\t\t\t\t\tUPDATE\n\t\t\t\t\t\t\t[procfwk].[BatchExecution]\n\t\t\t\t\t\tSET\n\t\t\t\t\t\t\t[BatchStatus] = 'Stopping'\n\t\t\t\t\t\tWHERE\n\t\t\t\t\t\t\t[ExecutionId] = @ExecutionId\n\t\t\t\t\t\t\tAND [BatchStatus] = 'Running';\n\t\t\t\t\tEND;\n\n\t\t\t\tSET @ErrorDetail = 'Pipeline execution has a cancelled status. Blocking downstream stages as a precaution.'\n\n\t\t\t\tRAISERROR(@ErrorDetail,16,1);\n\t\t\t\tRETURN 0;\n\t\t\tEND;\n\t\tELSE IF ([procfwk].[GetPropertyValueInternal]('FailureHandling')) = 'DependencyChain'\n\t\t\tBEGIN\n\t\t\t\tEXEC [procfwk].[SetExecutionBlockDependants]\n\t\t\t\t\t@ExecutionId = @ExecutionId,\n\t\t\t\t\t@PipelineId = @PipelineId\n\t\t\tEND;\n\t\tELSE\n\t\t\tBEGIN\n\t\t\t\tRAISERROR('Cancelled execution failure handling state.',16,1);\n\t\t\t\tRETURN 0;\n\t\t\tEND;\n\tEND;\nEND;"
},
{
"path": "MetadataDB/procfwk/Stored Procedures/SetLogPipelineChecking.sql",
"content": "CREATE PROCEDURE [procfwk].[SetLogPipelineChecking]\n\t(\n\t@ExecutionId UNIQUEIDENTIFIER,\n\t@StageId INT,\n\t@PipelineId INT\n\t)\nAS\nBEGIN\n\tSET NOCOUNT ON;\n\n\tUPDATE\n\t\t[procfwk].[CurrentExecution]\n\tSET\n\t\t[PipelineStatus] = 'Checking'\n\tWHERE\n\t\t[LocalExecutionId] = @ExecutionId\n\t\tAND [StageId] = @StageId\n\t\tAND [PipelineId] = @PipelineId\nEND;\n"
},
{
"path": "MetadataDB/procfwk/Stored Procedures/SetLogPipelineFailed.sql",
"content": "CREATE PROCEDURE [procfwk].[SetLogPipelineFailed]\n\t(\n\t@ExecutionId UNIQUEIDENTIFIER,\n\t@StageId INT,\n\t@PipelineId INT,\n\t@RunId UNIQUEIDENTIFIER = NULL\n\t)\nAS\nBEGIN\n\tSET NOCOUNT ON;\n\n\tDECLARE @ErrorDetail VARCHAR(500)\n\n\t--mark specific failure pipeline\n\tUPDATE\n\t\t[procfwk].[CurrentExecution]\n\tSET\n\t\t[PipelineStatus] = 'Failed'\n\tWHERE\n\t\t[LocalExecutionId] = @ExecutionId\n\t\tAND [StageId] = @StageId\n\t\tAND [PipelineId] = @PipelineId\n\n\t--persist failed pipeline records to long term log\n\tINSERT INTO [procfwk].[ExecutionLog]\n\t\t(\n\t\t[LocalExecutionId],\n\t\t[StageId],\n\t\t[PipelineId],\n\t\t[CallingOrchestratorName],\n\t\t[ResourceGroupName],\n\t\t[OrchestratorType],\n\t\t[OrchestratorName],\n\t\t[PipelineName],\n\t\t[StartDateTime],\n\t\t[PipelineStatus],\n\t\t[EndDateTime],\n\t\t[PipelineRunId],\n\t\t[PipelineParamsUsed]\n\t\t)\n\tSELECT\n\t\t[LocalExecutionId],\n\t\t[StageId],\n\t\t[PipelineId],\n\t\t[CallingOrchestratorName],\n\t\t[ResourceGroupName],\n\t\t[OrchestratorType],\n\t\t[OrchestratorName],\n\t\t[PipelineName],\n\t\t[StartDateTime],\n\t\t[PipelineStatus],\n\t\t[EndDateTime],\n\t\t[PipelineRunId],\n\t\t[PipelineParamsUsed]\n\tFROM\n\t\t[procfwk].[CurrentExecution]\n\tWHERE\n\t\t[LocalExecutionId] = @ExecutionId\n\t\tAND [PipelineStatus] = 'Failed'\n\t\tAND [StageId] = @StageId\n\t\tAND [PipelineId] = @PipelineId;\n\t\n\tIF ([procfwk].[GetPropertyValueInternal]('FailureHandling')) = 'None'\n\t\tBEGIN\n\t\t\t--do nothing allow processing to carry on regardless\n\t\t\tRETURN 0;\n\t\tEND;\n\t\t\n\tELSE IF ([procfwk].[GetPropertyValueInternal]('FailureHandling')) = 'Simple'\n\t\tBEGIN\n\t\t\t--flag all downstream stages as blocked\n\t\t\tUPDATE\n\t\t\t\t[procfwk].[CurrentExecution]\n\t\t\tSET\n\t\t\t\t[PipelineStatus] = 'Blocked',\n\t\t\t\t[IsBlocked] = 1\n\t\t\tWHERE\n\t\t\t\t[LocalExecutionId] = @ExecutionId\n\t\t\t\tAND [StageId] > @StageId\n\t\t\t\n\t\t\t--raise error to stop processing\n\t\t\tIF @RunId IS NOT NULL\n\t\t\t\tBEGIN\n\t\t\t\t\tSET @ErrorDetail = 'Pipeline execution failed. Check Run ID: ' + CAST(@RunId AS CHAR(36)) + ' in ADF monitoring for details.'\n\t\t\t\tEND;\n\t\t\tELSE\n\t\t\t\tBEGIN\n\t\t\t\t\tSET @ErrorDetail = 'Pipeline execution failed. See ADF monitoring for details.'\n\t\t\t\tEND;\n\n\t\t\t--update batch if applicable\n\t\t\tIF ([procfwk].[GetPropertyValueInternal]('UseExecutionBatches')) = '1'\n\t\t\t\tBEGIN\n\t\t\t\t\tUPDATE\n\t\t\t\t\t\t[procfwk].[BatchExecution]\n\t\t\t\t\tSET\n\t\t\t\t\t\t[BatchStatus] = 'Stopping'\n\t\t\t\t\tWHERE\n\t\t\t\t\t\t[ExecutionId] = @ExecutionId\n\t\t\t\t\t\tAND [BatchStatus] = 'Running';\n\t\t\t\tEND;\n\n\t\t\tRAISERROR(@ErrorDetail,16,1);\n\t\t\tRETURN 0;\n\t\tEND;\n\t\n\tELSE IF ([procfwk].[GetPropertyValueInternal]('FailureHandling')) = 'DependencyChain'\n\t\tBEGIN\n\t\t\tEXEC [procfwk].[SetExecutionBlockDependants]\n\t\t\t\t@ExecutionId = @ExecutionId,\n\t\t\t\t@PipelineId = @PipelineId\n\t\tEND;\n\tELSE\n\t\tBEGIN\n\t\t\tRAISERROR('Unknown failure handling state.',16,1);\n\t\t\tRETURN 0;\n\t\tEND;\nEND;"
},
{
"path": "MetadataDB/procfwk/Stored Procedures/SetLogPipelineLastStatusCheck.sql",
"content": "CREATE PROCEDURE [procfwk].[SetLogPipelineLastStatusCheck]\n\t(\n\t@ExecutionId UNIQUEIDENTIFIER,\n\t@StageId INT,\n\t@PipelineId INT\n\t)\nAS\nBEGIN\n\tSET NOCOUNT ON;\n\n\tUPDATE\n\t\t[procfwk].[CurrentExecution]\n\tSET\n\t\t[LastStatusCheckDateTime] = GETUTCDATE()\n\tWHERE\n\t\t[LocalExecutionId] = @ExecutionId\n\t\tAND [StageId] = @StageId\n\t\tAND [PipelineId] = @PipelineId\nEND;"
},
{
"path": "MetadataDB/procfwk/Stored Procedures/SetLogPipelineRunId.sql",
"content": "CREATE PROCEDURE [procfwk].[SetLogPipelineRunId]\n\t(\n\t@ExecutionId UNIQUEIDENTIFIER,\n\t@StageId INT,\n\t@PipelineId INT,\n\t@RunId UNIQUEIDENTIFIER = NULL\n\t)\nAS\nBEGIN\n\tSET NOCOUNT ON;\n\n\tUPDATE\n\t\t[procfwk].[CurrentExecution]\n\tSET\n\t\t[PipelineRunId] = LOWER(@RunId)\n\tWHERE\n\t\t[LocalExecutionId] = @ExecutionId\n\t\tAND [StageId] = @StageId\n\t\tAND [PipelineId] = @PipelineId\nEND;"
},
{
"path": "MetadataDB/procfwk/Stored Procedures/SetLogPipelineRunning.sql",
"content": "CREATE PROCEDURE procfwk.SetLogPipelineRunning\n\t(\n\t@ExecutionId UNIQUEIDENTIFIER,\n\t@StageId INT,\n\t@PipelineId INT\n\t)\nAS\nBEGIN\n\tSET NOCOUNT ON;\n\n\tUPDATE\n\t\t[procfwk].[CurrentExecution]\n\tSET\n\t\t--case for clean up runs\n\t\t[StartDateTime] = CASE WHEN [StartDateTime] IS NULL THEN GETUTCDATE() ELSE [StartDateTime] END,\n\t\t[PipelineStatus] = 'Running'\n\tWHERE\n\t\t[LocalExecutionId] = @ExecutionId\n\t\tAND [StageId] = @StageId\n\t\tAND [PipelineId] = @PipelineId\nEND;\n"
},
{
"path": "MetadataDB/procfwk/Stored Procedures/SetLogPipelineSuccess.sql",
"content": "CREATE PROCEDURE procfwk.SetLogPipelineSuccess\n\t(\n\t@ExecutionId UNIQUEIDENTIFIER,\n\t@StageId INT,\n\t@PipelineId INT\n\t)\nAS\nBEGIN\n\tSET NOCOUNT ON;\n\n\tUPDATE\n\t\t[procfwk].[CurrentExecution]\n\tSET\n\t\t--case for clean up runs\n\t\t[EndDateTime] = CASE WHEN [EndDateTime] IS NULL THEN GETUTCDATE() ELSE [EndDateTime] END,\n\t\t[PipelineStatus] = 'Success'\n\tWHERE\n\t\t[LocalExecutionId] = @ExecutionId\n\t\tAND [StageId] = @StageId\n\t\tAND [PipelineId] = @PipelineId\nEND;\n"
},
{
"path": "MetadataDB/procfwk/Stored Procedures/SetLogPipelineUnknown.sql",
"content": "CREATE PROCEDURE [procfwk].[SetLogPipelineUnknown]\n\t(\n\t@ExecutionId UNIQUEIDENTIFIER,\n\t@StageId INT,\n\t@PipelineId INT,\n\t@CleanUpRun BIT = 0\n\t)\nAS\nBEGIN\n\tSET NOCOUNT ON;\n\n\tDECLARE @ErrorDetail VARCHAR(500);\n\n\t--mark specific failure pipeline\n\tUPDATE\n\t\t[procfwk].[CurrentExecution]\n\tSET\n\t\t[PipelineStatus] = 'Unknown'\n\tWHERE\n\t\t[LocalExecutionId] = @ExecutionId\n\t\tAND [StageId] = @StageId\n\t\tAND [PipelineId] = @PipelineId\n\n\t--no need to block and log if done during a clean up cycle\n\tIF @CleanUpRun = 1 RETURN 0;\n\n\t--persist unknown pipeline records to long term log\n\tINSERT INTO [procfwk].[ExecutionLog]\n\t\t(\n\t\t[LocalExecutionId],\n\t\t[StageId],\n\t\t[PipelineId],\n\t\t[CallingOrchestratorName],\n\t\t[ResourceGroupName],\n\t\t[OrchestratorType],\n\t\t[OrchestratorName],\n\t\t[PipelineName],\n\t\t[StartDateTime],\n\t\t[PipelineStatus],\n\t\t[EndDateTime],\n\t\t[PipelineRunId],\n\t\t[PipelineParamsUsed]\n\t\t)\n\tSELECT\n\t\t[LocalExecutionId],\n\t\t[StageId],\n\t\t[PipelineId],\n\t\t[CallingOrchestratorName],\n\t\t[ResourceGroupName],\n\t\t[OrchestratorType],\n\t\t[OrchestratorName],\n\t\t[PipelineName],\n\t\t[StartDateTime],\n\t\t[PipelineStatus],\n\t\t[EndDateTime],\n\t\t[PipelineRunId],\n\t\t[PipelineParamsUsed]\n\tFROM\n\t\t[procfwk].[CurrentExecution]\n\tWHERE\n\t\t[PipelineStatus] = 'Unknown'\n\t\tAND [StageId] = @StageId\n\t\tAND [PipelineId] = @PipelineId;\n\n\t--block down stream stages?\n\tIF ([procfwk].[GetPropertyValueInternal]('UnknownWorkerResultBlocks')) = 1\n\tBEGIN\t\n\t\t--decide how to proceed with error/failure depending on framework property configuration\n\t\tIF ([procfwk].[GetPropertyValueInternal]('FailureHandling')) = 'None'\n\t\t\tBEGIN\n\t\t\t\t--do nothing allow processing to carry on regardless\n\t\t\t\tRETURN 0;\n\t\t\tEND;\n\t\t\n\t\tELSE IF ([procfwk].[GetPropertyValueInternal]('FailureHandling')) = 'Simple'\n\t\t\tBEGIN\n\t\t\t\t--flag all downstream stages as blocked\n\t\t\t\tUPDATE\n\t\t\t\t\t[procfwk].[CurrentExecution]\n\t\t\t\tSET\n\t\t\t\t\t[PipelineStatus] = 'Blocked',\n\t\t\t\t\t[IsBlocked] = 1\n\t\t\t\tWHERE\n\t\t\t\t\t[LocalExecutionId] = @ExecutionId\n\t\t\t\t\tAND [StageId] > @StageId\n\n\t\t\t\tUPDATE\n\t\t\t\t\t[procfwk].[BatchExecution]\n\t\t\t\tSET\n\t\t\t\t\t[BatchStatus] = 'Stopping'\n\t\t\t\tWHERE\n\t\t\t\t\t[ExecutionId] = @ExecutionId\n\t\t\t\t\tAND [BatchStatus] = 'Running';\n\n\t\t\t\tSET @ErrorDetail = 'Pipeline execution has an unknown status. Blocking downstream stages as a precaution.'\n\n\t\t\t\tRAISERROR(@ErrorDetail,16,1);\n\t\t\t\tRETURN 0;\n\t\t\tEND;\n\t\tELSE IF ([procfwk].[GetPropertyValueInternal]('FailureHandling')) = 'DependencyChain'\n\t\t\tBEGIN\n\t\t\t\tEXEC [procfwk].[SetExecutionBlockDependants]\n\t\t\t\t\t@ExecutionId = @ExecutionId,\n\t\t\t\t\t@PipelineId = @PipelineId\n\t\t\tEND;\n\t\tELSE\n\t\t\tBEGIN\n\t\t\t\tRAISERROR('Unknown failure handling state.',16,1);\n\t\t\t\tRETURN 0;\n\t\t\tEND;\n\tEND;\nEND;"
},
{
"path": "MetadataDB/procfwk/Stored Procedures/SetLogPipelineValidating.sql",
"content": "CREATE PROCEDURE [procfwk].[SetLogPipelineValidating]\n\t(\n\t@ExecutionId UNIQUEIDENTIFIER,\n\t@StageId INT,\n\t@PipelineId INT\n\t)\nAS\nBEGIN\n\tSET NOCOUNT ON;\n\n\tUPDATE\n\t\t[procfwk].[CurrentExecution]\n\tSET\n\t\t[PipelineStatus] = 'Validating'\n\tWHERE\n\t\t[LocalExecutionId] = @ExecutionId\n\t\tAND [StageId] = @StageId\n\t\tAND [PipelineId] = @PipelineId\nEND;\n"
},
{
"path": "MetadataDB/procfwk/Stored Procedures/SetLogStagePreparing.sql",
"content": "CREATE PROCEDURE [procfwk].[SetLogStagePreparing]\n\t(\n\t@ExecutionId UNIQUEIDENTIFIER,\n\t@StageId INT\n\t)\nAS\nBEGIN\n\tSET NOCOUNT ON;\n\t\n\tUPDATE\n\t\t[procfwk].[CurrentExecution]\n\tSET\n\t\t[PipelineStatus] = 'Preparing'\n\tWHERE\n\t\t[LocalExecutionId] = @ExecutionId\n\t\tAND [StageId] = @StageId\n\t\tAND [StartDateTime] IS NULL\n\t\tAND [IsBlocked] <> 1;\nEND;\n"
},
{
"path": "MetadataDB/procfwk/Stored Procedures/UpdateExecutionLog.sql",
"content": "CREATE PROCEDURE [procfwk].[UpdateExecutionLog]\n\t(\n\t@PerformErrorCheck BIT = 1,\n\t@ExecutionId UNIQUEIDENTIFIER = NULL\n\t)\nAS\nBEGIN\n\tSET NOCOUNT ON;\n\t\t\n\tDECLARE @AllCount INT\n\tDECLARE @SuccessCount INT\n\n\tIF([procfwk].[GetPropertyValueInternal]('UseExecutionBatches')) = '0'\n\t\tBEGIN\n\t\t\tIF @PerformErrorCheck = 1\n\t\t\tBEGIN\n\t\t\t\t--Check current execution\n\t\t\t\tSELECT @AllCount = COUNT(0) FROM [procfwk].[CurrentExecution]\n\t\t\t\tSELECT @SuccessCount = COUNT(0) FROM [procfwk].[CurrentExecution] WHERE [PipelineStatus] = 'Success'\n\n\t\t\t\tIF @AllCount <> @SuccessCount\n\t\t\t\t\tBEGIN\n\t\t\t\t\t\tRAISERROR('Framework execution complete but not all Worker pipelines succeeded. See the [procfwk].[CurrentExecution] table for details',16,1);\n\t\t\t\t\t\tRETURN 0;\n\t\t\t\t\tEND;\n\t\t\tEND;\n\n\t\t\t--Do this if no error raised and when called by the execution wrapper (OverideRestart = 1).\n\t\t\tINSERT INTO [procfwk].[ExecutionLog]\n\t\t\t\t(\n\t\t\t\t[LocalExecutionId],\n\t\t\t\t[StageId],\n\t\t\t\t[PipelineId],\n\t\t\t\t[CallingOrchestratorName],\n\t\t\t\t[ResourceGroupName],\n\t\t\t\t[OrchestratorType],\n\t\t\t\t[OrchestratorName],\n\t\t\t\t[PipelineName],\n\t\t\t\t[StartDateTime],\n\t\t\t\t[PipelineStatus],\n\t\t\t\t[EndDateTime],\n\t\t\t\t[PipelineRunId],\n\t\t\t\t[PipelineParamsUsed]\n\t\t\t\t)\n\t\t\tSELECT\n\t\t\t\t[LocalExecutionId],\n\t\t\t\t[StageId],\n\t\t\t\t[PipelineId],\n\t\t\t\t[CallingOrchestratorName],\n\t\t\t\t[ResourceGroupName],\n\t\t\t\t[OrchestratorType],\n\t\t\t\t[OrchestratorName],\n\t\t\t\t[PipelineName],\n\t\t\t\t[StartDateTime],\n\t\t\t\t[PipelineStatus],\n\t\t\t\t[EndDateTime],\n\t\t\t\t[PipelineRunId],\n\t\t\t\t[PipelineParamsUsed]\n\t\t\tFROM\n\t\t\t\t[procfwk].[CurrentExecution];\n\n\t\t\tTRUNCATE TABLE [procfwk].[CurrentExecution];\n\t\tEND\n\tELSE IF ([procfwk].[GetPropertyValueInternal]('UseExecutionBatches')) = '1'\n\t\tBEGIN\n\t\t\tIF @PerformErrorCheck = 1\n\t\t\tBEGIN\n\t\t\t\t--Check current execution\n\t\t\t\tSELECT \n\t\t\t\t\t@AllCount = COUNT(0) \n\t\t\t\tFROM \n\t\t\t\t\t[procfwk].[CurrentExecution] \n\t\t\t\tWHERE \n\t\t\t\t\t[LocalExecutionId] = @ExecutionId;\n\t\t\t\t\n\t\t\t\tSELECT \n\t\t\t\t\t@SuccessCount = COUNT(0) \n\t\t\t\tFROM \n\t\t\t\t\t[procfwk].[CurrentExecution] \n\t\t\t\tWHERE \n\t\t\t\t\t[LocalExecutionId] = @ExecutionId \n\t\t\t\t\tAND [PipelineStatus] = 'Success';\n\n\t\t\t\tIF @AllCount <> @SuccessCount\n\t\t\t\t\tBEGIN\n\t\t\t\t\t\tUPDATE\n\t\t\t\t\t\t\t[procfwk].[BatchExecution]\n\t\t\t\t\t\tSET\n\t\t\t\t\t\t\t[BatchStatus] = 'Stopped',\n\t\t\t\t\t\t\t[EndDateTime] = GETUTCDATE()\n\t\t\t\t\t\tWHERE\n\t\t\t\t\t\t\t[ExecutionId] = @ExecutionId;\n\t\t\t\t\t\t\n\t\t\t\t\t\tRAISERROR('Framework execution complete for batch but not all Worker pipelines succeeded. See the [procfwk].[CurrentExecution] table for details',16,1);\n\t\t\t\t\t\tRETURN 0;\n\t\t\t\t\tEND;\n\t\t\t\tELSE\n\t\t\t\t\tBEGIN\n\t\t\t\t\t\tUPDATE\n\t\t\t\t\t\t\t[procfwk].[BatchExecution]\n\t\t\t\t\t\tSET\n\t\t\t\t\t\t\t[BatchStatus] = 'Success',\n\t\t\t\t\t\t\t[EndDateTime] = GETUTCDATE()\n\t\t\t\t\t\tWHERE\n\t\t\t\t\t\t\t[ExecutionId] = @ExecutionId;\n\t\t\t\t\tEND;\n\t\t\tEND; --end check\n\n\t\t\t--Do this if no error raised and when called by the execution wrapper (OverideRestart = 1).\n\t\t\tINSERT INTO [procfwk].[ExecutionLog]\n\t\t\t\t(\n\t\t\t\t[LocalExecutionId],\n\t\t\t\t[StageId],\n\t\t\t\t[PipelineId],\n\t\t\t\t[CallingOrchestratorName],\n\t\t\t\t[ResourceGroupName],\n\t\t\t\t[OrchestratorType],\n\t\t\t\t[OrchestratorName],\n\t\t\t\t[PipelineName],\n\t\t\t\t[StartDateTime],\n\t\t\t\t[PipelineStatus],\n\t\t\t\t[EndDateTime],\n\t\t\t\t[PipelineRunId],\n\t\t\t\t[PipelineParamsUsed]\n\t\t\t\t)\n\t\t\tSELECT\n\t\t\t\t[LocalExecutionId],\n\t\t\t\t[StageId],\n\t\t\t\t[PipelineId],\n\t\t\t\t[CallingOrchestratorName],\n\t\t\t\t[ResourceGroupName],\n\t\t\t\t[OrchestratorType],\n\t\t\t\t[OrchestratorName],\n\t\t\t\t[PipelineName],\n\t\t\t\t[StartDateTime],\n\t\t\t\t[PipelineStatus],\n\t\t\t\t[EndDateTime],\n\t\t\t\t[PipelineRunId],\n\t\t\t\t[PipelineParamsUsed]\n\t\t\tFROM\n\t\t\t\t[procfwk].[CurrentExecution]\n\t\t\tWHERE \n\t\t\t\t[LocalExecutionId] = @ExecutionId;\n\n\t\t\tDELETE FROM\n\t\t\t\t[procfwk].[CurrentExecution]\n\t\t\tWHERE\n\t\t\t\t[LocalExecutionId] = @ExecutionId;\n\t\tEND;\nEND;"
},
{
"path": "MetadataDB/procfwk/Synonyms/AverageStageDuration.sql",
"content": "CREATE SYNONYM [procfwk].[AverageStageDuration] \nFOR [procfwkReporting].[AverageStageDuration];"
},
{
"path": "MetadataDB/procfwk/Synonyms/CompleteExecutionErrorLog.sql",
"content": "CREATE SYNONYM [procfwk].[CompleteExecutionErrorLog] \nFOR [procfwkReporting].[CompleteExecutionErrorLog];"
},
{
"path": "MetadataDB/procfwk/Synonyms/CompleteExecutionLog.sql",
"content": "CREATE SYNONYM [procfwk].[CompleteExecutionLog] \nFOR [procfwkReporting].[CompleteExecutionLog];"
},
{
"path": "MetadataDB/procfwk/Synonyms/CurrentExecutionSummary.sql",
"content": "CREATE SYNONYM [procfwk].[CurrentExecutionSummary] \nFOR [procfwkReporting].[CurrentExecutionSummary];"
},
{
"path": "MetadataDB/procfwk/Synonyms/DataFactoryDetails.sql",
"content": "CREATE SYNONYM [procfwk].[DataFactoryDetails]\nFOR [procfwk].[Orchestrators];"
},
{
"path": "MetadataDB/procfwk/Synonyms/LastExecution.sql",
"content": "CREATE SYNONYM [procfwk].[LastExecution] \nFOR [procfwkReporting].[LastExecution];"
},
{
"path": "MetadataDB/procfwk/Synonyms/LastExecutionSummary.sql",
"content": "CREATE SYNONYM [procfwk].[LastExecutionSummary] \nFOR [procfwkReporting].[LastExecutionSummary];"
},
{
"path": "MetadataDB/procfwk/Synonyms/PipelineDependencyChains.sql",
"content": "CREATE SYNONYM [procfwk].[PipelineDependencyChains]\nFOR [procfwkHelpers].[PipelineDependencyChains];\n"
},
{
"path": "MetadataDB/procfwk/Synonyms/PipelineProcesses.sql",
"content": "CREATE SYNONYM [procfwk].[PipelineProcesses]\nFOR [procfwk].[Pipelines];"
},
{
"path": "MetadataDB/procfwk/Synonyms/ProcessingStageDetails.sql",
"content": "CREATE SYNONYM [procfwk].[ProcessingStageDetails]\nFOR [procfwk].[Stages];\n"
},
{
"path": "MetadataDB/procfwk/Synonyms/WorkerParallelismOverTime.sql",
"content": "CREATE SYNONYM [procfwk].[WorkerParallelismOverTime] \nFOR [procfwkReporting].[WorkerParallelismOverTime];"
},
{
"path": "MetadataDB/procfwk/Tables/AlertOutcomes.sql",
"content": "CREATE TABLE [procfwk].[AlertOutcomes]\n\t(\n\t[OutcomeBitPosition] INT IDENTITY(0,1) NOT NULL,\n\t[PipelineOutcomeStatus] NVARCHAR(200) NOT NULL,\n\t[BitValue] AS (POWER((2),[OutcomeBitPosition])),\n\tCONSTRAINT [PK_AlertOutcomes] PRIMARY KEY CLUSTERED ([OutcomeBitPosition] ASC),\n\tCONSTRAINT [UK_PipelineOutcomeStatus] UNIQUE ([PipelineOutcomeStatus])\n\t)"
},
{
"path": "MetadataDB/procfwk/Tables/BatchExecution.sql",
"content": "CREATE TABLE [procfwk].[BatchExecution](\n\t[BatchId] [UNIQUEIDENTIFIER] NOT NULL,\n\t[ExecutionId] [UNIQUEIDENTIFIER] NOT NULL,\n\t[BatchName] VARCHAR(255) NOT NULL,\n\t[BatchStatus] [NVARCHAR](200) NOT NULL,\n\t[StartDateTime] [DATETIME] NOT NULL,\n\t[EndDateTime] [DATETIME] NULL,\n\tCONSTRAINT [PK_BatchExecution] PRIMARY KEY CLUSTERED \n\t(\n\t\t[BatchId] ASC,\n\t\t[ExecutionId] ASC\n\t)\n)\n\n\n"
},
{
"path": "MetadataDB/procfwk/Tables/BatchStageLink.sql",
"content": "CREATE TABLE [procfwk].[BatchStageLink]\n\t(\n\t[BatchId] [UNIQUEIDENTIFIER] NOT NULL,\n\t[StageId] [INT] NOT NULL,\n\tCONSTRAINT [PK_BatchStageLink] PRIMARY KEY CLUSTERED \n\t\t(\n\t\t[BatchId] ASC,\n\t\t[StageId] ASC\n\t\t),\n\tCONSTRAINT [FK_BatchStageLink_Batches] FOREIGN KEY([BatchId]) REFERENCES [procfwk].[Batches] ([BatchId]),\n\tCONSTRAINT [FK_BatchStageLink_Stages] FOREIGN KEY([StageId]) REFERENCES [procfwk].[Stages] ([StageId])\n)\n"
},
{
"path": "MetadataDB/procfwk/Tables/Batches.sql",
"content": "CREATE TABLE [procfwk].[Batches](\n\t[BatchId] UNIQUEIDENTIFIER DEFAULT(NEWID()) NOT NULL,\n\t[BatchName] [VARCHAR](255) NOT NULL,\n\t[BatchDescription] [VARCHAR](4000) NULL,\n\t[Enabled] [BIT]\tDEFAULT(0) NOT NULL,\n\t CONSTRAINT [PK_Batches] PRIMARY KEY CLUSTERED \n\t(\n\t\t[BatchId] ASC\n\t)\n)"
},
{
"path": "MetadataDB/procfwk/Tables/CurrentExecution.sql",
"content": "CREATE TABLE [procfwk].[CurrentExecution] (\n [LocalExecutionId] UNIQUEIDENTIFIER NOT NULL,\n [StageId] INT NOT NULL,\n [PipelineId] INT NOT NULL,\n [CallingOrchestratorName] NVARCHAR(200) NOT NULL,\n [ResourceGroupName]NVARCHAR (200) NOT NULL,\n [OrchestratorType] CHAR(3) NOT NULL,\n [OrchestratorName] NVARCHAR (200) NOT NULL,\n [PipelineName] NVARCHAR (200) NOT NULL,\n [StartDateTime] DATETIME NULL,\n [PipelineStatus] NVARCHAR (200) NULL,\n [LastStatusCheckDateTime] DATETIME NULL,\n [EndDateTime] DATETIME NULL,\n [IsBlocked] BIT NOT NULL DEFAULT 0,\n [PipelineRunId] UNIQUEIDENTIFIER NULL,\n [PipelineParamsUsed] NVARCHAR(MAX) NULL, \n CONSTRAINT [PK_CurrentExecution] PRIMARY KEY CLUSTERED ([LocalExecutionId] ASC, [StageId] ASC, [PipelineId] ASC)\n);\nGO\n\nCREATE NONCLUSTERED INDEX [IDX_GetPipelinesInStage] ON [procfwk].[CurrentExecution]\n (\n [LocalExecutionId],\n [StageId],\n [PipelineStatus]\n )\nINCLUDE\n (\n [PipelineId],\n [PipelineName],\n [OrchestratorType],\n [OrchestratorName],\n [ResourceGroupName]\n )\nGO\n"
},
{
"path": "MetadataDB/procfwk/Tables/ErrorLog.sql",
"content": "CREATE TABLE [procfwk].[ErrorLog]\n\t(\n\t[LogId] [int] IDENTITY(1,1) NOT NULL,\n\t[LocalExecutionId] [uniqueidentifier] NOT NULL,\n\t[PipelineRunId] [uniqueidentifier] NOT NULL,\n\t[ActivityRunId] [uniqueidentifier] NOT NULL,\n\t[ActivityName] [varchar](100) NOT NULL,\n\t[ActivityType] [varchar](100) NOT NULL,\n\t[ErrorCode] VARCHAR(100) NOT NULL,\n\t[ErrorType] [varchar](100) NOT NULL,\n\t[ErrorMessage] [nvarchar](MAX) NULL,\n\t CONSTRAINT [PK_ErrorLog] PRIMARY KEY CLUSTERED \n\t\t(\n\t\t[LogId] ASC\n\t\t)\n\t)"
},
{
"path": "MetadataDB/procfwk/Tables/ExecutionLog.sql",
"content": "CREATE TABLE [procfwk].[ExecutionLog] \n (\n [LogId] INT IDENTITY (1, 1) NOT NULL,\n [LocalExecutionId] UNIQUEIDENTIFIER NOT NULL,\n [StageId] INT NOT NULL,\n [PipelineId] INT NOT NULL,\n [CallingOrchestratorName] NVARCHAR(200) NOT NULL DEFAULT ('Unknown'),\n [ResourceGroupName] NVARCHAR(200) NOT NULL DEFAULT ('Unknown'),\n\t[OrchestratorType] CHAR(3) NOT NULL DEFAULT('N/A'),\n [OrchestratorName] NVARCHAR(200) NOT NULL DEFAULT ('Unknown'),\n [PipelineName] NVARCHAR (200) NOT NULL,\n [StartDateTime] DATETIME NULL,\n [PipelineStatus] NVARCHAR (200) NULL,\n [EndDateTime] DATETIME NULL,\n [PipelineRunId] UNIQUEIDENTIFIER NULL,\n [PipelineParamsUsed] NVARCHAR(MAX) NULL DEFAULT ('None'), \n CONSTRAINT [PK_ExecutionLog] PRIMARY KEY CLUSTERED ([LogId] ASC)\n );\n\n"
},
{
"path": "MetadataDB/procfwk/Tables/Orchestrators.sql",
"content": "CREATE TABLE [procfwk].[Orchestrators]\n\t(\n\t[OrchestratorId] [int] IDENTITY(1,1) NOT NULL,\n\t[OrchestratorName] NVARCHAR(200) NOT NULL,\n\t[OrchestratorType] CHAR(3) NOT NULL,\n\t[IsFrameworkOrchestrator] BIT NOT NULL DEFAULT(0),\n\t[ResourceGroupName] NVARCHAR(200) NOT NULL, \n\t[SubscriptionId] UNIQUEIDENTIFIER NOT NULL,\n\t[Description] NVARCHAR(MAX) NULL,\n\tCONSTRAINT [OrchestratorType] CHECK ([OrchestratorType] IN ('ADF','SYN')),\n CONSTRAINT [FK_Orchestrators_Subscriptions] FOREIGN KEY([SubscriptionId]) REFERENCES [procfwk].[Subscriptions] ([SubscriptionId]),\n\tCONSTRAINT [PK_Orchestrators] PRIMARY KEY CLUSTERED ([OrchestratorId] ASC)\n\t)"
},
{
"path": "MetadataDB/procfwk/Tables/PipelineAlertLink.sql",
"content": "CREATE TABLE [procfwk].[PipelineAlertLink]\n\t(\n\t[AlertId] INT IDENTITY(1,1) NOT NULL,\n\t[PipelineId] INT NOT NULL,\n\t[RecipientId] INT NOT NULL,\n\t[OutcomesBitValue] INT NOT NULL,\n\t[Enabled] BIT NOT NULL DEFAULT 1,\n\tCONSTRAINT [PK_PipelineAlertLink] PRIMARY KEY CLUSTERED ([AlertId] ASC),\n\tCONSTRAINT [FK_PipelineAlertLink_Pipelines] FOREIGN KEY([PipelineId]) REFERENCES [procfwk].[Pipelines] ([PipelineId]),\n\tCONSTRAINT [FK_PipelineAlertLink_Recipients] FOREIGN KEY([RecipientId]) REFERENCES [procfwk].[Recipients] ([RecipientId])\n\t);"
},
{
"path": "MetadataDB/procfwk/Tables/PipelineAuthLink.sql",
"content": "CREATE TABLE [procfwk].[PipelineAuthLink]\n\t(\n\t[AuthId] [int] IDENTITY(1,1) NOT NULL,\n\t[PipelineId] [int] NOT NULL,\n\t[OrchestratorId] [int] NOT NULL,\n\t[CredentialId] [int] NOT NULL,\n\tCONSTRAINT [PK_PipelineAuthLink] PRIMARY KEY CLUSTERED ([AuthId] ASC),\n\tCONSTRAINT [FK_PipelineAuthLink_Orchestrators] FOREIGN KEY([OrchestratorId]) REFERENCES [procfwk].[Orchestrators] ([OrchestratorId]),\n\tCONSTRAINT [FK_PipelineAuthLink_Pipelines] FOREIGN KEY([PipelineId]) REFERENCES [procfwk].[Pipelines] ([PipelineId]),\n\tCONSTRAINT [FK_PipelineAuthLink_ServicePrincipals] FOREIGN KEY([CredentialId]) REFERENCES [dbo].[ServicePrincipals] ([CredentialId])\n\t);"
},
{
"path": "MetadataDB/procfwk/Tables/PipelineDependencies.sql",
"content": "CREATE TABLE [procfwk].[PipelineDependencies]\n\t(\n\t[DependencyId] [INT] IDENTITY(1,1) NOT NULL,\n\t[PipelineId] [INT] NOT NULL,\n\t[DependantPipelineId] [INT] NOT NULL,\n\tCONSTRAINT [PK_PipelineDependencies] PRIMARY KEY CLUSTERED ([DependencyId] ASC),\n\tCONSTRAINT [FK_PipelineDependencies_Pipelines] FOREIGN KEY([PipelineId]) REFERENCES [procfwk].[Pipelines] ([PipelineId]),\n\tCONSTRAINT [FK_PipelineDependencies_Pipelines1] FOREIGN KEY([DependantPipelineId]) REFERENCES [procfwk].[Pipelines] ([PipelineId]),\n\tCONSTRAINT [UK_PipelinesToDependantPipelines] UNIQUE ([PipelineId],[DependantPipelineId]),\n\tCONSTRAINT [EQ_PipelineIdDependantPipelineId] CHECK ([PipelineId] <> [DependantPipelineId])\n\t)"
},
{
"path": "MetadataDB/procfwk/Tables/PipelineParameters.sql",
"content": "CREATE TABLE [procfwk].[PipelineParameters] (\n [ParameterId] INT IDENTITY (1, 1) NOT NULL,\n [PipelineId] INT NOT NULL,\n [ParameterName] VARCHAR (128) NOT NULL,\n [ParameterValue] NVARCHAR(MAX) NULL,\n [ParameterValueLastUsed] NVARCHAR(MAX) NULL,\n CONSTRAINT [PK_PipelineParameters] PRIMARY KEY CLUSTERED ([ParameterId] ASC),\n CONSTRAINT [FK_PipelineParameters_Pipelines] FOREIGN KEY ([PipelineId]) REFERENCES [procfwk].[Pipelines] ([PipelineId])\n);\n\n"
},
{
"path": "MetadataDB/procfwk/Tables/Pipelines.sql",
"content": "CREATE TABLE [procfwk].[Pipelines] (\n [PipelineId] INT IDENTITY (1, 1) NOT NULL,\n [OrchestratorId] INT NOT NULL,\n [StageId] INT NOT NULL,\n [PipelineName] NVARCHAR (200) NOT NULL,\n [LogicalPredecessorId] INT NULL,\n [Enabled] BIT CONSTRAINT [DF_Pipelines_Enabled] DEFAULT ((1)) NOT NULL,\n CONSTRAINT [PK_Pipelines] PRIMARY KEY CLUSTERED ([PipelineId] ASC),\n CONSTRAINT [FK_Pipelines_Stages] FOREIGN KEY ([StageId]) REFERENCES [procfwk].[Stages] ([StageId]),\n CONSTRAINT [FK_Pipelines_Orchestrators] FOREIGN KEY([OrchestratorId]) REFERENCES [procfwk].[Orchestrators] ([OrchestratorId]),\n CONSTRAINT [FK_Pipelines_Pipelines] FOREIGN KEY([LogicalPredecessorId]) REFERENCES [procfwk].[Pipelines] ([PipelineId])\n);\n\n"
},
{
"path": "MetadataDB/procfwk/Tables/Properties.sql",
"content": "CREATE TABLE [procfwk].[Properties]\n\t(\n\t[PropertyId] [int] IDENTITY (1, 1) NOT NULL,\n\t[PropertyName] [varchar](128) NOT NULL,\n\t[PropertyValue] [nvarchar](MAX) NOT NULL,\n\t[Description] [nvarchar](MAX) NULL,\n\t[ValidFrom] [datetime] CONSTRAINT [DF_Properties_ValidFrom] DEFAULT (GETDATE()) NOT NULL,\n\t[ValidTo] [datetime] NULL,\n\tCONSTRAINT [PK_Properties] PRIMARY KEY CLUSTERED ([PropertyId] ASC, [PropertyName] ASC)\n\t)\nGO"
},
{
"path": "MetadataDB/procfwk/Tables/Recipients.sql",
"content": "CREATE TABLE [procfwk].[Recipients]\n\t(\n\t[RecipientId] INT IDENTITY(1,1) NOT NULL,\n\t[Name] VARCHAR(255) NULL,\n\t[EmailAddress] NVARCHAR(500) NOT NULL,\n\t[MessagePreference] CHAR(3) NOT NULL DEFAULT ('TO'),\n\tCONSTRAINT [MessagePreferenceValue] CHECK ([MessagePreference] IN ('TO','CC','BCC')),\n\t[Enabled] BIT NOT NULL DEFAULT 1,\n\tCONSTRAINT [PK_Recipients] PRIMARY KEY CLUSTERED ([RecipientId] ASC),\n\tCONSTRAINT [UK_EmailAddressMessagePreference] UNIQUE ([EmailAddress],[MessagePreference])\n\t);\n"
},
{
"path": "MetadataDB/procfwk/Tables/Stages.sql",
"content": "CREATE TABLE [procfwk].[Stages] (\n [StageId] INT IDENTITY (1, 1) NOT NULL,\n [StageName] VARCHAR (225) NOT NULL,\n [StageDescription] VARCHAR (4000) NULL,\n [Enabled] BIT CONSTRAINT [DF_Stages_Enabled] DEFAULT ((1)) NOT NULL,\n CONSTRAINT [PK_Stages] PRIMARY KEY CLUSTERED ([StageId] ASC)\n);\n\n"
},
{
"path": "MetadataDB/procfwk/Tables/Subscriptions.sql",
"content": "CREATE TABLE [procfwk].[Subscriptions]\n\t(\n\t[SubscriptionId] UNIQUEIDENTIFIER NOT NULL,\n\t[Name] NVARCHAR(200) NOT NULL,\n\t[Description] NVARCHAR(MAX) NULL,\n\t[TenantId] UNIQUEIDENTIFIER NOT NULL,\n\tCONSTRAINT [PK_Subscriptions] PRIMARY KEY CLUSTERED ([SubscriptionId] ASC),\n\tCONSTRAINT [FK_Subscriptions_Tenants] FOREIGN KEY([TenantId]) REFERENCES [procfwk].[Tenants] ([TenantId])\n\t)"
},
{
"path": "MetadataDB/procfwk/Tables/Tenants.sql",
"content": "CREATE TABLE [procfwk].[Tenants]\n\t(\n\t[TenantId] [UNIQUEIDENTIFIER] NOT NULL,\n\t[Name] [NVARCHAR](200) NOT NULL,\n\t[Description] [NVARCHAR](MAX) NULL,\n\tCONSTRAINT [PK_Tenants] PRIMARY KEY CLUSTERED ([TenantId] ASC)\n\t)"
},
{
"path": "MetadataDB/procfwk/Views/CurrentProperties.sql",
"content": "CREATE VIEW [procfwk].[CurrentProperties]\nAS\n\nSELECT\n\t[PropertyName],\n\t[PropertyValue]\nFROM\n\t[procfwk].[Properties]\nWHERE\n\t[ValidTo] IS NULL;"
},
{
"path": "MetadataDB/procfwk/Views/DataFactorys.sql",
"content": "CREATE VIEW [procfwk].[DataFactorys]\nAS\nSELECT\n\t[OrchestratorId] AS DataFactoryId,\n\t[OrchestratorName] AS DataFactoryName,\n\t[ResourceGroupName],\n\t[SubscriptionId],\n\t[Description]\nFROM\n\t[procfwk].[Orchestrators]\nWHERE\n\t[OrchestratorType] = 'ADF';"
},
{
"path": "MetadataDB/procfwk/Views/PipelineParameterDataSizes.sql",
"content": "CREATE VIEW [procfwk].[PipelineParameterDataSizes]\nAS\n\nSELECT \n\t[PipelineId],\n\tSUM(\n\t\t(CAST(\n\t\t\tDATALENGTH(\n\t\t\t\tSTRING_ESCAPE([ParameterName] + [ParameterValue],'json')) AS DECIMAL)\n\t\t\t/1024) --KB\n\t\t\t/1024 --MB\n\t\t) AS Size\nFROM \n\t[procfwk].[PipelineParameters]\nGROUP BY\n\t[PipelineId];"
},
{
"path": "MetadataDB/procfwkHelpers/Functions/CheckForValidURL.sql",
"content": "CREATE FUNCTION [procfwkHelpers].[CheckForValidURL] (@Url NVARCHAR(MAX))\nRETURNS INT\nAS\nBEGIN\n DECLARE @Ending VARCHAR(50)\n\tDECLARE @TempString VARCHAR(50)\n\n\t--check URL start\n\tIF CHARINDEX('https://', @Url) <> 1\n BEGIN\n RETURN 0;\n END\n\n\t--check for expected sub domains\n IF CHARINDEX('vault.azure.net', @Url) = 0\n BEGIN\n RETURN 0;\n END\n\n\t--check for expected value type\n IF CHARINDEX('secrets', @Url) = 0\n BEGIN\n\t\tRETURN 0;\n END\n \n\t--attempt to check for secret version \n\tSELECT \n\t\t@Ending = \n\t\t\tCASE\n\t\t\t\tWHEN RIGHT(@Url,1) = '/' THEN REVERSE(LEFT(@Url,LEN(@Url)-1))\n\t\t\t\tELSE REVERSE(@Url)\n\t\t\tEND,\n\t\t@Ending = REVERSE(LEFT(@Ending,CHARINDEX('/',@Ending)-1))\n\n\t\tIF LEN(@Ending) = 32 \n\t\tBEGIN \n \t\tSET @TempString = \n\t\t\t\tSUBSTRING(@Ending, 1, 8) + '-' + \n\t\t\t\tSUBSTRING(@Ending, 9, 4) + '-' + \n \t\t\tSUBSTRING(@Ending, 13, 4) + '-' + \n\t\t\t\tSUBSTRING(@Ending, 13, 4) + '-' + \n\t\t\t\tSUBSTRING(@Ending, 20, 12) \n\t\tEND\n\t\n\t\tIF TRY_CAST(@TempString AS UNIQUEIDENTIFIER) IS NOT NULL \n\t\tBEGIN\n\t\t\tRETURN 0;\n\t\tEND;\n\n\t-- It is a valid URL\n RETURN 1;\nEND;"
},
{
"path": "MetadataDB/procfwkHelpers/Stored Procedures/AddPipelineDependant.sql",
"content": "CREATE PROCEDURE [procfwkHelpers].[AddPipelineDependant]\n\t(\n\t@PipelineName NVARCHAR(200),\n\t@DependantPipelineName NVARCHAR(200)\n\t)\nAS\nBEGIN\n\tSET NOCOUNT ON;\n\n\tDECLARE @PipelineId INT;\n\tDECLARE @DependantPipelineId INT;\n\n\t--get pipeline ids\n\tSELECT\n\t\t@PipelineId = [PipelineId]\n\tFROM\n\t\t[procfwk].[Pipelines]\n\tWHERE\n\t\t[PipelineName] = @PipelineName;\n\n\tSELECT\n\t\t@DependantPipelineId = [PipelineId]\n\tFROM\n\t\t[procfwk].[Pipelines]\n\tWHERE\n\t\t[PipelineName] = @DependantPipelineName;\n\n\t--defensive checks\n\tIF @PipelineId IS NULL\n\tBEGIN\n\t\tRAISERROR('Pipeline not found in pipelines table.', 16,1);\n\t\tRETURN 0;\n\tEND;\n\n\tIF @DependantPipelineId IS NULL\n\tBEGIN\n\t\tRAISERROR('Dependant pipeline not found in pipelines table.', 16,1);\n\t\tRETURN 0;\n\tEND;\n\n\tIF @PipelineId = @DependantPipelineId\n\tBEGIN\n\t\tRAISERROR('Pipeline cannot be dependant on itself.', 16,1);\n\t\tRETURN 0;\n\tEND;\n\n\tIF EXISTS\n\t\t(\n\t\tSELECT \n\t\t\t*\n\t\tFROM \n\t\t\t[procfwk].[Pipelines] pp\n\t\t\tINNER JOIN [procfwk].[Pipelines] dp\n\t\t\t\tON dp.[PipelineId] = @DependantPipelineId\n\t\tWHERE\n\t\t\tpp.[PipelineId] = @PipelineId\n\t\t\tAND pp.[StageId] = dp.[StageId]\n\t\t)\n\t\tBEGIN\n\t\t\tRAISERROR('Pipeline and dependent pipeline cannot be in the same execution stage.', 16,1);\n\t\t\tRETURN 0;\n\t\tEND;\n\n\t--final soft check and insert\n\tIF EXISTS\n\t\t(\n\t\tSELECT \n\t\t\t* \n\t\tFROM \n\t\t\t[procfwk].[PipelineDependencies] \n\t\tWHERE\n\t\t\t[PipelineId] = @PipelineId\n\t\t\tAND [DependantPipelineId] = @DependantPipelineId\n\t\t)\n\t\tBEGIN\n\t\t\tPRINT 'Dependency already exists. Nothing added.'\n\t\t\tRETURN 0;\n\t\tEND\n\tELSE\n\t\tBEGIN\n\t\t\tINSERT INTO [procfwk].[PipelineDependencies]\n\t\t\t\t(\n\t\t\t\t[PipelineId],\n\t\t\t\t[DependantPipelineId]\n\t\t\t\t)\n\t\t\tVALUES\n\t\t\t\t(\n\t\t\t\t@PipelineId,\n\t\t\t\t@DependantPipelineId\n\t\t\t\t)\n\t\tEND;\nEND;\n"
},
{
"path": "MetadataDB/procfwkHelpers/Stored Procedures/AddPipelineViaPowerShell.sql",
"content": "CREATE PROCEDURE [procfwkHelpers].[AddPipelineViaPowerShell]\n\t(\n\t@ResourceGroup NVARCHAR(200),\n\t@OrchestratorName NVARCHAR(200),\n\t@OrchestratorType CHAR(3) = 'ADF',\n\t@PipelineName NVARCHAR(200)\n\t)\nAS\nBEGIN\n\tSET NOCOUNT ON;\n\n\tDECLARE @OrchestratorId INT\n\tDECLARE @StageId INT\n\tDECLARE @StageName VARCHAR(255) = 'PoShAdded'\n\n\t--get/set orchestrator\n\tIF EXISTS\n\t\t(\n\t\tSELECT * FROM [procfwk].[Orchestrators] WHERE [OrchestratorName] = @OrchestratorName AND [ResourceGroupName] = @ResourceGroup AND [OrchestratorType] = @OrchestratorType\n\t\t)\n\t\tBEGIN\n\t\t\tSELECT @OrchestratorId = [OrchestratorId] FROM [procfwk].[Orchestrators] WHERE [OrchestratorName] = @OrchestratorName AND [ResourceGroupName] = @ResourceGroup AND [OrchestratorType] = @OrchestratorType;\n\t\tEND\n\tELSE\n\t\tBEGIN\n\t\t\tINSERT INTO [procfwk].[Orchestrators]\n\t\t\t\t(\n\t\t\t\t[OrchestratorName],\n\t\t\t\t[OrchestratorType],\n\t\t\t\t[ResourceGroupName],\n\t\t\t\t[Description],\n\t\t\t\t[SubscriptionId]\n\t\t\t\t)\n\t\t\tVALUES\n\t\t\t\t(\n\t\t\t\t@OrchestratorName,\n\t\t\t\t@OrchestratorType,\n\t\t\t\t@ResourceGroup,\n\t\t\t\t'Added via PowerShell.',\n\t\t\t\t'12345678-1234-1234-1234-012345678910'\n\t\t\t\t)\n\n\t\t\tSELECT\n\t\t\t\t@OrchestratorId = SCOPE_IDENTITY();\n\t\tEND\n\n\t--get/set stage\n\tIF EXISTS\n\t\t(\n\t\tSELECT * FROM [procfwk].[Stages] WHERE [StageName] = @StageName\n\t\t)\n\t\tBEGIN\n\t\t\tSELECT @StageId = [StageId] FROM [procfwk].[Stages] WHERE [StageName] = @StageName;\n\t\tEND;\n\tELSE\n\t\tBEGIN\n\t\t\tINSERT INTO [procfwk].[Stages]\n\t\t\t\t(\n\t\t\t\t[StageName],\n\t\t\t\t[StageDescription],\n\t\t\t\t[Enabled]\n\t\t\t\t)\n\t\t\tVALUES\n\t\t\t\t(\n\t\t\t\t@StageName,\n\t\t\t\t'Added via PowerShell.',\n\t\t\t\t1\n\t\t\t\t);\n\n\t\t\tSELECT\n\t\t\t\t@StageId = SCOPE_IDENTITY();\n\t\tEND;\n\n\t--upsert pipeline\n\t;WITH sourceData AS\n\t\t(\n\t\tSELECT\n\t\t\t@OrchestratorId AS OrchestratorId,\n\t\t\t@PipelineName AS PipelineName,\n\t\t\t@StageId AS StageId,\n\t\t\tNULL AS LogicalPredecessorId,\n\t\t\t1 AS [Enabled]\n\t\t)\n\tMERGE INTO [procfwk].[Pipelines] AS tgt\n\tUSING \n\t\tsourceData AS src\n\t\t\tON tgt.[OrchestratorId] = src.[OrchestratorId]\n\t\t\t\tAND tgt.[PipelineName] = src.[PipelineName]\n\tWHEN MATCHED THEN\n\t\tUPDATE\n\t\tSET\n\t\t\ttgt.[StageId] = src.[StageId],\n\t\t\ttgt.[LogicalPredecessorId] = src.[LogicalPredecessorId],\n\t\t\ttgt.[Enabled] = src.[Enabled]\n\tWHEN NOT MATCHED BY TARGET THEN\n\t\tINSERT\n\t\t\t(\n\t\t\t[OrchestratorId],\n\t\t\t[StageId],\n\t\t\t[PipelineName], \n\t\t\t[LogicalPredecessorId],\n\t\t\t[Enabled]\n\t\t\t)\n\t\tVALUES\n\t\t\t(\n\t\t\tsrc.[OrchestratorId],\n\t\t\tsrc.[StageId],\n\t\t\tsrc.[PipelineName], \n\t\t\tsrc.[LogicalPredecessorId],\n\t\t\tsrc.[Enabled]\n\t\t\t);\nEND;\n"
},
{
"path": "MetadataDB/procfwkHelpers/Stored Procedures/AddProperty.sql",
"content": "CREATE PROCEDURE [procfwkHelpers].[AddProperty]\n\t(\n\t@PropertyName VARCHAR(128),\n\t@PropertyValue NVARCHAR(MAX),\n\t@Description NVARCHAR(MAX) = NULL\n\t)\nAS\nBEGIN\n\t\n\tSET NOCOUNT ON;\n\n\t--defensive check\n\tIF EXISTS\n\t\t(\n\t\tSELECT * FROM [procfwk].[Properties] WHERE [PropertyName] = @PropertyName AND [ValidTo] IS NOT NULL\n\t\t)\n\t\tAND NOT EXISTS\n\t\t(\n\t\tSELECT * FROM [procfwk].[Properties] WHERE [PropertyName] = @PropertyName AND [ValidTo] IS NULL\n\t\t)\n\t\tBEGIN\n\t\t\tWITH lastValue AS\n\t\t\t\t(\n\t\t\t\tSELECT\n\t\t\t\t\t[PropertyId],\n\t\t\t\t\tROW_NUMBER() OVER (PARTITION BY [PropertyName] ORDER BY [ValidTo] ASC) AS Rn\n\t\t\t\tFROM\n\t\t\t\t\t[procfwk].[Properties]\n\t\t\t\tWHERE\n\t\t\t\t\t[PropertyName] = @PropertyName\n\t\t\t\t)\n\t\t\t--reset property if valid to date has been incorrectly set\n\t\t\tUPDATE\n\t\t\t\tprop\n\t\t\tSET\n\t\t\t\t[ValidTo] = NULL\n\t\t\tFROM\n\t\t\t\t[procfwk].[Properties] prop\n\t\t\t\tINNER JOIN lastValue\n\t\t\t\t\tON prop.[PropertyId] = lastValue.[PropertyId]\n\t\t\tWHERE\n\t\t\t\tlastValue.[Rn] = 1\n\t\tEND\n\t\n\n\t--upsert property\n\t;WITH sourceTable AS\n\t\t(\n\t\tSELECT\n\t\t\t@PropertyName AS PropertyName,\n\t\t\t@PropertyValue AS PropertyValue,\n\t\t\t@Description AS [Description],\n\t\t\tGETUTCDATE() AS StartEndDate\n\t\t)\n\t--insert new version of existing property from MERGE OUTPUT\n\tINSERT INTO [procfwk].[Properties]\n\t\t(\n\t\t[PropertyName],\n\t\t[PropertyValue],\n\t\t[Description],\n\t\t[ValidFrom]\n\t\t)\n\tSELECT\n\t\t[PropertyName],\n\t\t[PropertyValue],\n\t\t[Description],\n\t\tGETUTCDATE()\n\tFROM\n\t\t(\n\t\tMERGE INTO\n\t\t\t[procfwk].[Properties] targetTable\n\t\tUSING\n\t\t\tsourceTable\n\t\t\t\tON sourceTable.[PropertyName] = targetTable.[PropertyName]\t\n\t\t--set valid to date on existing property\n\t\tWHEN MATCHED AND [ValidTo] IS NULL THEN \n\t\t\tUPDATE\n\t\t\tSET\n\t\t\t\ttargetTable.[ValidTo] = sourceTable.[StartEndDate]\n\t\t--add new property\n\t\tWHEN NOT MATCHED BY TARGET THEN\n\t\t\tINSERT\n\t\t\t\t(\n\t\t\t\t[PropertyName],\n\t\t\t\t[PropertyValue],\n\t\t\t\t[Description],\n\t\t\t\t[ValidFrom]\n\t\t\t\t)\n\t\t\tVALUES\n\t\t\t\t(\n\t\t\t\tsourceTable.[PropertyName],\n\t\t\t\tsourceTable.[PropertyValue],\n\t\t\t\tsourceTable.[Description],\n\t\t\t\tsourceTable.[StartEndDate]\n\t\t\t\t)\n\t\t\t--for new entry of existing record\n\t\t\tOUTPUT\n\t\t\t\t$action AS [Action],\n\t\t\t\tsourceTable.*\n\t\t\t) AS MergeOutput\n\t\tWHERE\n\t\t\tMergeOutput.[Action] = 'UPDATE';\nEND;"
},
{
"path": "MetadataDB/procfwkHelpers/Stored Procedures/AddRecipientPipelineAlerts.sql",
"content": "CREATE PROCEDURE [procfwkHelpers].[AddRecipientPipelineAlerts]\n\t(\n\t@RecipientName VARCHAR(255),\n\t@PipelineName NVARCHAR(200) = NULL,\n\t@AlertForStatus NVARCHAR(500) = 'All'\n\t)\nAS\nBEGIN\n\tSET NOCOUNT ON;\n\n\tDECLARE @ActualBitValue INT\n\tDECLARE @SQL NVARCHAR(MAX) = ''\n\tDECLARE @BitValue TABLE ([TotalBitValue] INT NOT NULL);\n\n\t--get alert status bit value\n\tSET @AlertForStatus = LTRIM(RTRIM(@AlertForStatus))\n\tSET @AlertForStatus = REPLACE(@AlertForStatus,' ','')\n\tSET @AlertForStatus = '''' + REPLACE(@AlertForStatus,',',''',''') + ''''\n\n\tSET @SQL = \n\t\t'\n\t\tSELECT\n\t\t\tSUM([BitValue]) AS ''TotalBitValue''\n\t\tFROM\n\t\t\t[procfwk].[AlertOutcomes]\n\t\tWHERE\n\t\t\t[PipelineOutcomeStatus] IN (' + @AlertForStatus + ')\n\t\t'\n\n\tINSERT INTO @BitValue ([TotalBitValue]) EXECUTE(@SQL)\n\tSELECT @ActualBitValue = [TotalBitValue] FROM @BitValue\n\t\n\t--set link table\n\tIF @PipelineName IS NOT NULL\n\t\tBEGIN\n\t\t\t--add alert for specific pipeline if doesn't exist\n\t\t\tINSERT INTO [procfwk].[PipelineAlertLink]\n\t\t\t\t(\n\t\t\t\t[PipelineId],\n\t\t\t\t[RecipientId],\n\t\t\t\t[OutcomesBitValue]\n\t\t\t\t)\t\t\t\n\t\t\tSELECT\n\t\t\t\tp.[PipelineId],\n\t\t\t\tr.[RecipientId],\n\t\t\t\t@ActualBitValue\n\t\t\tFROM\n\t\t\t\t[procfwk].[Pipelines] p\n\t\t\t\tINNER JOIN [procfwk].[Recipients] r\n\t\t\t\t\tON r.[Name] = @RecipientName\n\t\t\t\tLEFT OUTER JOIN [procfwk].[PipelineAlertLink] al\n\t\t\t\t\tON p.[PipelineId] = al.[PipelineId]\n\t\t\t\t\t\tAND r.[RecipientId] = al.[RecipientId]\n\t\t\tWHERE\n\t\t\t\tp.[PipelineName] = @PipelineName\n\t\t\t\tAND al.[PipelineId] IS NULL\n\t\t\t\tAND al.[RecipientId] IS NULL;\n\t\tEND\n\tELSE IF @PipelineName IS NULL\n\t\tBEGIN\n\t\t\t--remove and re-add alerts for all pipelines\n\t\t\tDELETE \n\t\t\t\tal\n\t\t\tFROM \n\t\t\t\t[procfwk].[PipelineAlertLink] al\n\t\t\t\tINNER JOIN [procfwk].[Recipients] r\n\t\t\t\t\tON al.[RecipientId] = r.[RecipientId]\n\t\t\tWHERE\n\t\t\t\tr.[Name] = @RecipientName;\n\t\t\t\t\t\t\n\t\t\tINSERT INTO [procfwk].[PipelineAlertLink]\n\t\t\t\t(\n\t\t\t\t[PipelineId],\n\t\t\t\t[RecipientId],\n\t\t\t\t[OutcomesBitValue]\n\t\t\t\t)\n\t\t\tSELECT\n\t\t\t\tp.[PipelineId],\n\t\t\t\tr.[RecipientId],\n\t\t\t\t@ActualBitValue\n\t\t\tFROM\n\t\t\t\t[procfwk].[Recipients] r\n\t\t\t\tCROSS JOIN [procfwk].[Pipelines] p\n\t\t\tWHERE\n\t\t\t\tr.[Name] = @RecipientName;\n\t\tEND;\nEND"
},
{
"path": "MetadataDB/procfwkHelpers/Stored Procedures/AddServicePrincipal.sql",
"content": "CREATE PROCEDURE [procfwkHelpers].[AddServicePrincipal]\n\t(\n\t@OrchestratorName NVARCHAR(200),\n\t@OrchestratorType CHAR(3),\n\t@PrincipalId NVARCHAR(256),\n\t@PrincipalSecret NVARCHAR(MAX),\n\t@SpecificPipelineName NVARCHAR(200) = NULL,\n\t@PrincipalName NVARCHAR(256) = NULL\n\t)\nAS\nBEGIN\n\n\tSET NOCOUNT ON;\n\n\tDECLARE @ErrorDetails NVARCHAR(500) = ''\n\tDECLARE @CredentialId INT\n\tDECLARE @LocalPrincipalId UNIQUEIDENTIFIER\n\n\t--defensive checks\n\tBEGIN TRY\n\t\tSELECT --assigned to variable just to supress output of SELECT\n\t\t\t@LocalPrincipalId = CAST(@PrincipalId AS UNIQUEIDENTIFIER)\n\tEND TRY\n\tBEGIN CATCH\n\t\t\tSET @ErrorDetails = 'Invalid @PrincipalId provided. The format must be a UNIQUEIDENTIFIER.'\n\t\t\tRAISERROR(@ErrorDetails, 16, 1);\n\t\t\tRETURN 0;\n\tEND CATCH\n\n\tIF NOT EXISTS\n\t\t(\n\t\tSELECT [OrchestratorName] FROM [procfwk].[Orchestrators] WHERE [OrchestratorName] = @OrchestratorName AND [OrchestratorType] = @OrchestratorType\n\t\t)\n\t\tBEGIN\n\t\t\tSET @ErrorDetails = 'Invalid Orchestrator name. Please ensure the Orchestrator metadata exists before trying to add authentication for it.'\n\t\t\tRAISERROR(@ErrorDetails, 16, 1);\n\t\t\tRETURN 0;\n\t\tEND\n\t\n\tIF EXISTS\n\t\t(\n\t\tSELECT\n\t\t\t*\n\t\tFROM\n\t\t\t[procfwk].[PipelineAuthLink] AL\n\t\t\tINNER JOIN [procfwk].[Orchestrators] DF\n\t\t\t\tON AL.[OrchestratorId] = DF.[OrchestratorId]\n\t\t\tINNER JOIN [procfwk].[Pipelines] PP\n\t\t\t\tON AL.[PipelineId] = PP.[PipelineId]\n\t\tWHERE\n\t\t\tDF.[OrchestratorName] = @OrchestratorName\n\t\t\tAND DF.[OrchestratorType] = @OrchestratorType\n\t\t\tAND PP.[PipelineName] = @SpecificPipelineName\n\t\t)\n\t\tBEGIN\n\t\t\tSET @ErrorDetails = 'The provided Pipeline or Orchestrator combination already have a Service Principal. Delete the existing record using the procedure [procfwk].[DeleteServicePrincipal].'\n\t\t\tRAISERROR(@ErrorDetails, 16, 1);\n\t\t\tRETURN 0;\n\t\tEND\n\t\t\n\t--add SPN for specific pipeline\n\tIF @SpecificPipelineName IS NOT NULL\n\t\tBEGIN\n\t\t\t--secondary defensive check for pipeline optional param\n\t\t\tIF NOT EXISTS\n\t\t\t\t( \n\t\t\t\tSELECT [PipelineName] FROM [procfwk].[Pipelines] WHERE [PipelineName] = @SpecificPipelineName\n\t\t\t\t)\n\t\t\t\tBEGIN\n\t\t\t\t\tSET @ErrorDetails = 'Invalid Pipeline name. Please ensure the Pipeline metadata exists before trying to add authentication for it.'\n\t\t\t\t\tRAISERROR(@ErrorDetails, 16, 1);\n\t\t\t\t\tRETURN 0;\n\t\t\t\tEND\n\t\t\t\n\t\t\t--spn may already exist for other pipelines\n\t\t\tIF NOT EXISTS\n\t\t\t\t(\n\t\t\t\tSELECT [PrincipalId] FROM [dbo].[ServicePrincipals] WHERE [PrincipalId] = @PrincipalId\n\t\t\t\t)\n\t\t\t\tBEGIN\n\t\t\t\t\t--add service principal\n\t\t\t\t\tINSERT INTO [dbo].[ServicePrincipals]\n\t\t\t\t\t\t( \n\t\t\t\t\t\t[PrincipalName],\n\t\t\t\t\t\t[PrincipalId],\n\t\t\t\t\t\t[PrincipalSecret]\n\t\t\t\t\t\t)\n\t\t\t\t\tSELECT\n\t\t\t\t\t\tISNULL(@PrincipalName, 'Unknown'),\n\t\t\t\t\t\t@PrincipalId,\n\t\t\t\t\t\tENCRYPTBYPASSPHRASE(CONCAT(@OrchestratorName, @OrchestratorType, @SpecificPipelineName), @PrincipalSecret)\n\n\t\t\t\t\tSET @CredentialId = SCOPE_IDENTITY()\n\t\t\t\tEND\n\t\t\tELSE\n\t\t\t\tBEGIN\n\t\t\t\t\tSELECT @CredentialId = [CredentialId] FROM [dbo].[ServicePrincipals] WHERE [PrincipalId] = @PrincipalId\n\t\t\t\tEND\n\n\t\t\t--add single pipeline to SPN link\n\t\t\tINSERT INTO [procfwk].[PipelineAuthLink]\n\t\t\t\t(\n\t\t\t\t[PipelineId],\n\t\t\t\t[OrchestratorId],\n\t\t\t\t[CredentialId]\n\t\t\t\t)\n\t\t\tSELECT\n\t\t\t\tP.[PipelineId],\n\t\t\t\tD.[OrchestratorId],\n\t\t\t\t@CredentialId\n\t\t\tFROM\n\t\t\t\t[procfwk].[Pipelines] P\n\t\t\t\tINNER JOIN [procfwk].[Orchestrators] D\n\t\t\t\t\tON P.[OrchestratorId] = D.[OrchestratorId]\n\t\t\tWHERE\n\t\t\t\tP.[PipelineName] = @SpecificPipelineName\n\t\t\t\tAND D.[OrchestratorType] = @OrchestratorType\n\t\t\t\tAND D.[OrchestratorName] = @OrchestratorName;\n\t\tEND\n\tELSE\n\t\t--add SPN for all pipelines in Orchestrator\n\t\tBEGIN\n\t\t\t--add service principal\n\t\t\tINSERT INTO [dbo].[ServicePrincipals]\n\t\t\t\t( \n\t\t\t\t[PrincipalName],\n\t\t\t\t[PrincipalId],\n\t\t\t\t[PrincipalSecret]\n\t\t\t\t)\n\t\t\tSELECT\n\t\t\t\tISNULL(@PrincipalName, 'Unknown'),\n\t\t\t\t@PrincipalId,\n\t\t\t\tENCRYPTBYPASSPHRASE(CONCAT(@OrchestratorName, @OrchestratorType), @PrincipalSecret)\n\n\t\t\tSET @CredentialId = SCOPE_IDENTITY()\n\n\t\t\t--add link\n\t\t\tINSERT INTO [procfwk].[PipelineAuthLink]\n\t\t\t\t(\n\t\t\t\t[PipelineId],\n\t\t\t\t[OrchestratorId],\n\t\t\t\t[CredentialId]\n\t\t\t\t)\n\t\t\tSELECT\n\t\t\t\tP.[PipelineId],\n\t\t\t\tD.[OrchestratorId],\n\t\t\t\t@CredentialId\n\t\t\tFROM\n\t\t\t\t[procfwk].[Pipelines] P\n\t\t\t\tINNER JOIN [procfwk].[Orchestrators] D\n\t\t\t\t\tON P.[OrchestratorId] = D.[OrchestratorId]\n\t\t\t\tLEFT OUTER JOIN [procfwk].[PipelineAuthLink] L\n\t\t\t\t\tON P.[PipelineId] = L.[PipelineId]\n\t\t\tWHERE\n\t\t\t\tD.[OrchestratorName] = @OrchestratorName\n\t\t\t\tAND D.[OrchestratorType] = @OrchestratorType\n\t\t\t\tAND L.[PipelineId] IS NULL;\n\t\tEND\nEND;"
},
{
"path": "MetadataDB/procfwkHelpers/Stored Procedures/AddServicePrincipalUrls.sql",
"content": "CREATE PROCEDURE [procfwkHelpers].[AddServicePrincipalUrls]\n\t(\n\t@OrchestratorName NVARCHAR(200),\n\t@OrchestratorType CHAR(3),\n\t@PrincipalIdUrl NVARCHAR(MAX),\n\t@PrincipalSecretUrl NVARCHAR(MAX),\n\t@SpecificPipelineName NVARCHAR(200) = NULL,\n\t@PrincipalName NVARCHAR(256) = NULL\n\t)\nAS\nBEGIN\n\n\tSET NOCOUNT ON;\n\n\tDECLARE @ErrorDetails NVARCHAR(500) = ''\n\tDECLARE @CredentialId INT\n\n\t--defensive checks\n\tIF NOT EXISTS\n\t\t(\n\t\tSELECT [OrchestratorName] FROM [procfwk].[Orchestrators] WHERE [OrchestratorName] = @OrchestratorName AND [OrchestratorType] = @OrchestratorType\n\t\t)\n\t\tBEGIN\n\t\t\tSET @ErrorDetails = 'Invalid Orchestrator name. Please ensure the Orchestrator metadata exists before trying to add authentication for it.'\n\t\t\tRAISERROR(@ErrorDetails, 16, 1);\n\t\t\tRETURN 0;\n\t\tEND\n\t\n\tIF EXISTS\n\t\t(\n\t\tSELECT\n\t\t\t*\n\t\tFROM\n\t\t\t[procfwk].[PipelineAuthLink] AL\n\t\t\tINNER JOIN [procfwk].[Orchestrators] DF\n\t\t\t\tON AL.[OrchestratorId] = DF.[OrchestratorId]\n\t\t\tINNER JOIN [procfwk].[Pipelines] PP\n\t\t\t\tON AL.[PipelineId] = PP.[PipelineId]\n\t\tWHERE\n\t\t\tDF.[OrchestratorName] = @OrchestratorName\n\t\t\tAND DF.[OrchestratorType] = @OrchestratorType\n\t\t\tAND PP.[PipelineName] = @SpecificPipelineName\n\t\t)\n\t\tBEGIN\n\t\t\tSET @ErrorDetails = 'The provided Pipeline or Orchestrator combination already have a Service Principal. Delete the existing record using the procedure [procfwk].[DeleteServicePrincipal].'\n\t\t\tRAISERROR(@ErrorDetails, 16, 1);\n\t\t\tRETURN 0;\n\t\tEND\n\t\n\tIF ([procfwkHelpers].[CheckForValidURL](@PrincipalIdUrl)) = 0\n\tBEGIN\n\t\tSET @ErrorDetails = 'PrincipalIdUrl value is not in the expected format. . Please confirm the URL follows the structure https://{YourKeyVaultName}.vault.azure.net/secrets/{YourSecretName} and does not include the secret version guid.'\n\t\tPRINT @ErrorDetails;\n\tEND\n\n\tIF ([procfwkHelpers].[CheckForValidURL](@PrincipalSecretUrl)) = 0\n\tBEGIN\n\t\tSET @ErrorDetails = 'PrincipalSecretUrl value is not in the expected format. Please confirm the URL follows the structure https://{YourKeyVaultName}.vault.azure.net/secrets/{YourSecretName} and does not include the secret version guid.'\n\t\tPRINT @ErrorDetails;\t\t\n\tEND\n\n\t--add SPN for specific pipeline\n\tIF @SpecificPipelineName IS NOT NULL\n\t\tBEGIN\n\t\t\t--secondary defensive check for pipeline optional param\n\t\t\tIF NOT EXISTS\n\t\t\t\t( \n\t\t\t\tSELECT [PipelineName] FROM [procfwk].[Pipelines] WHERE [PipelineName] = @SpecificPipelineName\n\t\t\t\t)\n\t\t\t\tBEGIN\n\t\t\t\t\tSET @ErrorDetails = 'Invalid Pipeline name. Please ensure the Pipeline metadata exists before trying to add authentication for it.'\n\t\t\t\t\tRAISERROR(@ErrorDetails, 16, 1);\n\t\t\t\t\tRETURN 0;\n\t\t\t\tEND\n\t\t\t\n\t\t\t--spn may already exist for other pipelines\n\t\t\tIF NOT EXISTS\n\t\t\t\t(\t\t\t\t\n\t\t\t\tSELECT [PrincipalIdUrl] FROM [dbo].[ServicePrincipals] WHERE [PrincipalIdUrl] = @PrincipalIdUrl\n\t\t\t\t)\n\t\t\t\tBEGIN\n\t\t\t\t\t--add service principal\n\t\t\t\t\tINSERT INTO [dbo].[ServicePrincipals]\n\t\t\t\t\t\t( \n\t\t\t\t\t\t[PrincipalName],\n\t\t\t\t\t\t[PrincipalIdUrl],\n\t\t\t\t\t\t[PrincipalSecretUrl]\n\t\t\t\t\t\t)\n\t\t\t\t\tSELECT\n\t\t\t\t\t\tISNULL(@PrincipalName, 'Unknown'),\n\t\t\t\t\t\t@PrincipalIdUrl,\n\t\t\t\t\t\t@PrincipalSecretUrl\n\n\t\t\t\t\tSET @CredentialId = SCOPE_IDENTITY()\n\t\t\t\tEND\n\t\t\tELSE\n\t\t\t\tBEGIN\n\t\t\t\t\tSELECT @CredentialId = [CredentialId] FROM [dbo].[ServicePrincipals] WHERE [PrincipalIdUrl] = @PrincipalIdUrl\n\t\t\t\tEND\n\n\t\t\t--add single pipeline to SPN link\n\t\t\tINSERT INTO [procfwk].[PipelineAuthLink]\n\t\t\t\t(\n\t\t\t\t[PipelineId],\n\t\t\t\t[OrchestratorId],\n\t\t\t\t[CredentialId]\n\t\t\t\t)\n\t\t\tSELECT\n\t\t\t\tP.[PipelineId],\n\t\t\t\tD.[OrchestratorId],\n\t\t\t\t@CredentialId\n\t\t\tFROM\n\t\t\t\t[procfwk].[Pipelines] P\n\t\t\t\tINNER JOIN [procfwk].[Orchestrators] D\n\t\t\t\t\tON P.[OrchestratorId] = D.[OrchestratorId]\n\t\t\tWHERE\n\t\t\t\tP.[PipelineName] = @SpecificPipelineName\n\t\t\t\tAND D.[OrchestratorType] = @OrchestratorType\n\t\t\t\tAND D.[OrchestratorName] = @OrchestratorName;\n\t\tEND\n\tELSE\n\t\t--add SPN for all pipelines in Orchestrator\n\t\tBEGIN\n\t\t\t--add service principal\n\t\t\tINSERT INTO [dbo].[ServicePrincipals]\n\t\t\t\t( \n\t\t\t\t[PrincipalName],\n\t\t\t\t[PrincipalIdUrl],\n\t\t\t\t[PrincipalSecretUrl]\n\t\t\t\t)\n\t\t\tSELECT\n\t\t\t\tISNULL(@PrincipalName, 'Unknown'),\n\t\t\t\t@PrincipalIdUrl,\n\t\t\t\t@PrincipalSecretUrl\n\n\t\t\tSET @CredentialId = SCOPE_IDENTITY()\n\n\t\t\t--add link\n\t\t\tINSERT INTO [procfwk].[PipelineAuthLink]\n\t\t\t\t(\n\t\t\t\t[PipelineId],\n\t\t\t\t[OrchestratorId],\n\t\t\t\t[CredentialId]\n\t\t\t\t)\n\t\t\tSELECT\n\t\t\t\tP.[PipelineId],\n\t\t\t\tD.[OrchestratorId],\n\t\t\t\t@CredentialId\n\t\t\tFROM\n\t\t\t\t[procfwk].[Pipelines] P\n\t\t\t\tINNER JOIN [procfwk].[Orchestrators] D\n\t\t\t\t\tON P.[OrchestratorId] = D.[OrchestratorId]\n\t\t\t\tLEFT OUTER JOIN [procfwk].[PipelineAuthLink] L\n\t\t\t\t\tON P.[PipelineId] = L.[PipelineId]\n\t\t\tWHERE\n\t\t\t\tD.[OrchestratorName] = @OrchestratorName\n\t\t\t\tAND D.[OrchestratorType] = @OrchestratorType\n\t\t\t\tAND L.[PipelineId] IS NULL;\n\t\tEND\nEND;"
},
{
"path": "MetadataDB/procfwkHelpers/Stored Procedures/AddServicePrincipalWrapper.sql",
"content": "CREATE PROCEDURE [procfwkHelpers].[AddServicePrincipalWrapper]\n\t(\n\t@OrchestratorName NVARCHAR(200),\n\t@OrchestratorType CHAR(3),\n\t@PrincipalIdValue NVARCHAR(MAX),\n\t@PrincipalSecretValue NVARCHAR(MAX),\n\t@SpecificPipelineName NVARCHAR(200) = NULL,\n\t@PrincipalName NVARCHAR(256) = NULL\n\t)\nAS\nBEGIN\n\t\n\tIF ([procfwk].[GetPropertyValueInternal]('SPNHandlingMethod')) = 'StoreInDatabase'\n\t\tBEGIN\n\t\t\tEXEC [procfwk].[AddServicePrincipal]\n\t\t\t\t@OrchestratorName = @OrchestratorName,\n\t\t\t\t@OrchestratorType = @OrchestratorType,\n\t\t\t\t@PrincipalId = @PrincipalIdValue,\n\t\t\t\t@PrincipalSecret = @PrincipalSecretValue,\n\t\t\t\t@PrincipalName = @PrincipalName,\n\t\t\t\t@SpecificPipelineName = @SpecificPipelineName\t\t\t\n\t\tEND\n\tELSE IF ([procfwk].[GetPropertyValueInternal]('SPNHandlingMethod')) = 'StoreInKeyVault'\n\t\tBEGIN\n\t\t\tEXEC [procfwk].[AddServicePrincipalUrls]\n\t\t\t\t@OrchestratorName = @OrchestratorName,\n\t\t\t\t@OrchestratorType = @OrchestratorType,\n\t\t\t\t@PrincipalIdUrl = @PrincipalIdValue,\n\t\t\t\t@PrincipalSecretUrl = @PrincipalSecretValue,\n\t\t\t\t@PrincipalName = @PrincipalName,\n\t\t\t\t@SpecificPipelineName = @SpecificPipelineName\t\t\n\t\tEND\n\tELSE\n\t\tBEGIN\n\t\t\tRAISERROR('Unknown SPN insert method.',16,1);\n\t\t\tRETURN 0;\n\t\tEND\nEND;\n"
},
{
"path": "MetadataDB/procfwkHelpers/Stored Procedures/CheckStageAndPiplineIntegrity.sql",
"content": "CREATE PROCEDURE [procfwkHelpers].[CheckStageAndPiplineIntegrity]\nAS\n\nBEGIN\n\n\tDECLARE @TempCheckStageAndPiplineIntegrity TABLE\n\t\t(\n\t\t[ResourceGroupName] NVARCHAR(200) NOT NULL,\n\t\t[OrchestratorName] NVARCHAR(200) NOT NULL,\n\t\t[StageId] INT NOT NULL,\n\t\t[StageName] VARCHAR(225) NOT NULL,\n\t\t[PipelineId] INT NOT NULL,\n\t\t[PipelineName] NVARCHAR(200) NOT NULL,\n\t\t[Enabled] BIT NOT NULL,\n\t\t[SuccessorStageId] INT NULL,\n\t\t[SuccessorStage] VARCHAR(225) NULL,\n\t\t[SuccessorId] INT NULL,\n\t\t[SuccessorName] NVARCHAR(200) NULL,\n\t\t[Information] VARCHAR(92) NULL\n\t\t)\t\n\t\n\t--get min execution stage\n\t;WITH firstStage AS\n\t\t(\n\t\tSELECT\n\t\t\tMIN([StageId]) AS firstStageId\n\t\tFROM\n\t\t\t[procfwk].[Stages]\n\t\t)\n\t--query metadata\n\tINSERT INTO @TempCheckStageAndPiplineIntegrity\n\tSELECT\n\t\tadf.[ResourceGroupName],\n\t\tadf.[OrchestratorName],\t\n\t\tbase.[StageId],\n\t\tbaseStage.[StageName],\n\t\tbase.[PipelineId],\n\t\tbase.[PipelineName],\n\t\tbase.[Enabled],\n\t\tpreds.[StageId] AS SuccessorStageId,\n\t\tpredsStage.[StageName] AS SuccessorStage,\n\t\tpreds.[PipelineId] AS SuccessorId,\n\t\tpreds.[PipelineName] AS SuccessorName,\n\t\tCASE\n\t\t\tWHEN preds.[StageId] > base.[StageId] +1 THEN 'Successor pipeline could be moved to an earlier stage.'\n\t\t\tWHEN preds.[StageId] = base.[StageId] THEN 'Dependency issue, predeccessor pipeline is currently running in the same stage as successor.'\n\t\t\tWHEN preds.[PipelineId] IS NOT NULL AND base.[Enabled] = 0 THEN 'Disabled pipeline has downstream successors.'\n\t\t\tWHEN preds.[PipelineId] IS NOT NULL AND baseStage.[Enabled] = 0 THEN 'Disabled stage has downstream successors.'\n\t\t\tWHEN base.[LogicalPredecessorId] IS NULL AND base.[StageId] <> firstStage.[firstStageId] THEN 'Pipeline could be moved to an earlier stage.'\n\t\t\tELSE NULL\n\t\tEND AS Information\n\tFROM \n\t\t--get base pipeline details\n\t\t[procfwk].[Pipelines] base\n\t\tINNER JOIN [procfwk].[Orchestrators] adf\n\t\t\tON base.[OrchestratorId] = adf.[OrchestratorId]\n\t\tINNER JOIN [procfwk].[Stages] baseStage\n\t\t\tON base.[StageId] = baseStage.[StageId]\t\n\t\t--get successor details\n\t\tLEFT OUTER JOIN [procfwk].[Pipelines] preds\n\t\t\tON base.[PipelineId] = preds.[LogicalPredecessorId]\n\t\tLEFT OUTER JOIN [procfwk].[Stages] predsStage\n\t\t\tON preds.[StageId] = predsStage.[StageId]\n\t\t--other details for checking\n\t\tCROSS JOIN firstStage\n\n\t--provide outcome\n\tIF EXISTS\n\t\t(\n\t\tSELECT [Information] FROM @TempCheckStageAndPiplineIntegrity\n\t\t)\n\t\tBEGIN\n\t\t\tSELECT * FROM @TempCheckStageAndPiplineIntegrity\n\t\tEND\n\tELSE\n\t\tBEGIN\n\t\t\tPRINT 'No pipeline integrity issues to report. Nice work! :-)'\n\t\tEND\nEND;\n\n"
},
{
"path": "MetadataDB/procfwkHelpers/Stored Procedures/DeleteMetadataWithIntegrity.sql",
"content": "CREATE PROCEDURE [procfwkHelpers].[DeleteMetadataWithIntegrity]\nAS\nBEGIN\n\t/*\n\tDELETE ORDER IMPORTANT FOR REFERENTIAL INTEGRITY\n\t*/\n\n\t--BatchExecution\n\tIF OBJECT_ID(N'[procfwk].[BatchExecution]') IS NOT NULL \n\t\tBEGIN\n\t\t\tTRUNCATE TABLE [procfwk].[BatchExecution];\n\t\tEND;\n\n\t--CurrentExecution\n\tIF OBJECT_ID(N'[procfwk].[CurrentExecution]') IS NOT NULL \n\t\tBEGIN\n\t\t\tTRUNCATE TABLE [procfwk].[CurrentExecution];\n\t\tEND;\n\n\t--ExecutionLog\n\tIF OBJECT_ID(N'[procfwk].[ExecutionLog]') IS NOT NULL \n\t\tBEGIN\n\t\t\tTRUNCATE TABLE [procfwk].[ExecutionLog];\n\t\tEND\n\n\t--ErrorLog\n\tIF OBJECT_ID(N'[procfwk].[ExecutionLog]') IS NOT NULL \n\t\tBEGIN\n\t\t\tTRUNCATE TABLE [procfwk].[ErrorLog];\n\t\tEND\n\n\t--BatchStageLink\n\tIF OBJECT_ID(N'[procfwk].[BatchStageLink]') IS NOT NULL \n\t\tBEGIN\n\t\t\tDELETE FROM [procfwk].[BatchStageLink];\n\t\tEND;\n\n\t--Batches\n\tIF OBJECT_ID(N'[procfwk].[Batches]') IS NOT NULL \n\t\tBEGIN\n\t\t\tDELETE FROM [procfwk].[Batches];\n\t\tEND;\n\n\t--PipelineDependencies\n\tIF OBJECT_ID(N'[procfwk].[PipelineDependencies]') IS NOT NULL \n\t\tBEGIN\n\t\t\tDELETE FROM [procfwk].[PipelineDependencies];\n\t\t\tDBCC CHECKIDENT ('[procfwk].[PipelineDependencies]', RESEED, 0);\n\t\tEND;\n\n\t--PipelineAlertLink\n\tIF OBJECT_ID(N'[procfwk].[PipelineAlertLink]') IS NOT NULL \n\t\tBEGIN\n\t\t\tDELETE FROM [procfwk].[PipelineAlertLink];\n\t\t\tDBCC CHECKIDENT ('[procfwk].[PipelineAlertLink]', RESEED, 0);\n\t\tEND;\n\n\t--Recipients\n\tIF OBJECT_ID(N'[procfwk].[Recipients]') IS NOT NULL \n\t\tBEGIN\n\t\t\tDELETE FROM [procfwk].[Recipients];\n\t\t\tDBCC CHECKIDENT ('[procfwk].[Recipients]', RESEED, 0);\n\t\tEND;\n\n\t--AlertOutcomes\n\tIF OBJECT_ID(N'[procfwk].[AlertOutcomes]') IS NOT NULL \n\t\tBEGIN\n\t\t\tTRUNCATE TABLE [procfwk].[AlertOutcomes];\n\t\tEND;\n\n\t--PipelineAuthLink\n\tIF OBJECT_ID(N'[procfwk].[PipelineAuthLink]') IS NOT NULL \n\t\tBEGIN\n\t\t\tDELETE FROM [procfwk].[PipelineAuthLink];\n\t\t\tDBCC CHECKIDENT ('[procfwk].[PipelineAuthLink]', RESEED, 0);\n\t\tEND;\n\n\t--ServicePrincipals\n\tIF OBJECT_ID(N'[dbo].[ServicePrincipals]') IS NOT NULL \n\t\tBEGIN\n\t\t\tDELETE FROM [dbo].[ServicePrincipals];\n\t\t\tDBCC CHECKIDENT ('[dbo].[ServicePrincipals]', RESEED, 0);\n\t\tEND;\n\n\t--Properties\n\tIF OBJECT_ID(N'[procfwk].[Properties]') IS NOT NULL \n\t\tBEGIN\n\t\t\tDELETE FROM [procfwk].[Properties];\n\t\t\tDBCC CHECKIDENT ('[procfwk].[Properties]', RESEED, 0);\n\t\tEND;\n\n\t--PipelineParameters\n\tIF OBJECT_ID(N'[procfwk].[PipelineParameters]') IS NOT NULL \n\t\tBEGIN\n\t\t\tDELETE FROM [procfwk].[PipelineParameters];\n\t\t\tDBCC CHECKIDENT ('[procfwk].[PipelineParameters]', RESEED, 0);\n\t\tEND;\n\n\t--Pipelines\n\tIF OBJECT_ID(N'[procfwk].[Pipelines]') IS NOT NULL \n\t\tBEGIN\n\t\t\tDELETE FROM [procfwk].[Pipelines];\n\t\t\tDBCC CHECKIDENT ('[procfwk].[Pipelines]', RESEED, 0);\n\t\tEND;\n\n\t--Orchestrators\n\tIF OBJECT_ID(N'[procfwk].[Orchestrators]') IS NOT NULL \n\t\tBEGIN\n\t\t\tDELETE FROM [procfwk].[Orchestrators];\n\t\t\tDBCC CHECKIDENT ('[procfwk].[Orchestrators]', RESEED, 0);\n\t\tEND;\n\n\t--Stages\n\tIF OBJECT_ID(N'[procfwk].[Stages]') IS NOT NULL \n\t\tBEGIN\n\t\t\tDELETE FROM [procfwk].[Stages];\n\t\t\tDBCC CHECKIDENT ('[procfwk].[Stages]', RESEED, 0);\n\t\tEND;\n\n\t--Subscriptions\n\tIF OBJECT_ID(N'[procfwk].[Subscriptions]') IS NOT NULL \n\t\tBEGIN\n\t\t\tDELETE FROM [procfwk].[Subscriptions];\n\t\tEND;\n\t\n\t--Tenants\n\tIF OBJECT_ID(N'[procfwk].[Tenants]') IS NOT NULL \n\t\tBEGIN\n\t\t\tDELETE FROM [procfwk].[Tenants];\n\t\tEND;\nEND;"
},
{
"path": "MetadataDB/procfwkHelpers/Stored Procedures/DeleteMetadataWithoutIntegrity.sql",
"content": "CREATE PROCEDURE [procfwkHelpers].[DeleteMetadataWithoutIntegrity]\nAS\nBEGIN\n\n\tDECLARE @SQL NVARCHAR(MAX) = ''\n\n\t;WITH procfwkTables AS\n\t\t(\n\t\tSELECT\n\t\t\tQUOTENAME(s.[name]) + '.' + QUOTENAME(o.[name]) AS FullName\n\t\tFROM\n\t\t\tsys.objects o\n\t\t\tINNER JOIN sys.schemas s\n\t\t\t\tON o.[schema_id] = s.[schema_id]\n\t\tWHERE\n\t\t\ts.[name] LIKE 'procfwk%'\n\t\t\tAND o.[type] = 'U'\n\n\t\tUNION \n\n\t\tSELECT\n\t\t\tQUOTENAME(s.[name]) + '.' + QUOTENAME(o.[name]) AS FullName\n\t\tFROM\n\t\t\tsys.objects o\n\t\t\tINNER JOIN sys.schemas s\n\t\t\t\tON o.[schema_id] = s.[schema_id]\n\t\tWHERE\n\t\t\to.[name] = 'ServicePrincipals'\n\t\t\tAND o.[type] = 'U'\n\t\t)\n\n\tSELECT --tables must exist or wouldnt appear in sys.objects query\n\t\t@SQL += 'DELETE FROM ' + [FullName] + ';' + CHAR(13)\n\tFROM \n\t\tprocfwkTables\n\n\tEXEC(@SQL);\nEND;"
},
{
"path": "MetadataDB/procfwkHelpers/Stored Procedures/DeleteRecipientAlerts.sql",
"content": "CREATE PROCEDURE [procfwkHelpers].[DeleteRecipientAlerts]\n\t(\n\t@EmailAddress NVARCHAR(500),\n\t@SoftDeleteOnly BIT = 1\n\t)\nAS\nBEGIN\n\tSET NOCOUNT ON;\n\n\t--defensive check\n\tIF NOT EXISTS\n\t\t(\n\t\tSELECT [RecipientId] FROM [procfwk].[Recipients] WHERE [EmailAddress] = @EmailAddress\n\t\t)\n\t\tBEGIN\n\t\t\tRAISERROR('Recipient email address does not exists in [procfwk].[Recipients] table.',16,1);\n\t\t\tRETURN 0;\n\t\tEND;\n\n\t--update/delete\n\tIF @SoftDeleteOnly = 1\n\t\tBEGIN\n\t\t\t--disable links\n\t\t\tUPDATE\n\t\t\t\tal\n\t\t\tSET\n\t\t\t\tal.[Enabled] = 0\n\t\t\tFROM\n\t\t\t\t[procfwk].[PipelineAlertLink] al\n\t\t\t\tINNER JOIN [procfwk].[Recipients] r\n\t\t\t\t\tON al.[RecipientId] = r.[RecipientId]\n\t\t\tWHERE\n\t\t\t\tr.[EmailAddress] = @EmailAddress;\n\t\n\t\t\t--disable recipient(s)\n\t\t\tUPDATE\n\t\t\t\t[procfwk].[Recipients]\n\t\t\tSET\n\t\t\t\t[Enabled] = 0\n\t\t\tWHERE\n\t\t\t\t[EmailAddress] = @EmailAddress;\n\n\t\tEND\n\tELSE\n\t\tBEGIN\n\t\t\t--delete links\n\t\t\tDELETE\t\t\n\t\t\t\tal\n\t\t\tFROM\n\t\t\t\t[procfwk].[PipelineAlertLink] al\n\t\t\t\tINNER JOIN [procfwk].[Recipients] r\n\t\t\t\t\tON al.[RecipientId] = r.[RecipientId]\n\t\t\tWHERE\n\t\t\t\tr.[EmailAddress] = @EmailAddress;\n\n\t\t\t--delete recipient(s)\n\t\t\tDELETE FROM\n\t\t\t\t[procfwk].[Recipients]\n\t\t\tWHERE\n\t\t\t\t[EmailAddress] = @EmailAddress;\n\t\tEND;\nEND;"
},
{
"path": "MetadataDB/procfwkHelpers/Stored Procedures/DeleteServicePrincipal.sql",
"content": "CREATE PROCEDURE [procfwkHelpers].[DeleteServicePrincipal]\n\t(\n\t@OrchestratorName NVARCHAR(200),\n\t@OrchestratorType CHAR(3),\n\t@PrincipalIdValue NVARCHAR(256),\n\t@SpecificPipelineName NVARCHAR(200) = NULL\n\t)\nAS\nBEGIN\n\tSET NOCOUNT ON;\n\n\tDECLARE @ErrorDetails NVARCHAR(500) = ''\n\tDECLARE @CredentialId INT\n\n\t--resolve principal Id or Url to credential Id\n\tIF ([procfwk].[GetPropertyValueInternal]('SPNHandlingMethod')) = 'StoreInDatabase'\n\t\tBEGIN\n\t\t\t--defensive checks\n\t\t\tBEGIN TRY\n\t\t\t\tDECLARE @LocalPrincipalId UNIQUEIDENTIFIER\n\n\t\t\t\tSELECT --assigned to variable just to supress output of SELECT\n\t\t\t\t\t@LocalPrincipalId = CAST(@PrincipalIdValue AS UNIQUEIDENTIFIER)\n\t\t\tEND TRY\n\t\t\tBEGIN CATCH\n\t\t\t\t\tSET @ErrorDetails = 'Invalid @PrincipalId provided. The format must be a UNIQUEIDENTIFIER.'\n\t\t\t\t\tRAISERROR(@ErrorDetails, 16, 1);\n\t\t\t\t\tRETURN 0;\n\t\t\tEND CATCH\t\n\t\t\t\n\t\t\t--get cred id using principal id\n\t\t\tSELECT\n\t\t\t\t@CredentialId = [CredentialId]\n\t\t\tFROM\n\t\t\t\t[dbo].[ServicePrincipals]\n\t\t\tWHERE\n\t\t\t\t[PrincipalId] = @PrincipalIdValue\n\t\tEND\n\tELSE IF ([procfwk].[GetPropertyValueInternal]('SPNHandlingMethod')) = 'StoreInKeyVault'\n\t\tBEGIN\n\t\t\t--get cred id using principal id url\n\t\t\tSELECT\n\t\t\t\t@CredentialId = [CredentialId]\n\t\t\tFROM\n\t\t\t\t[dbo].[ServicePrincipals]\n\t\t\tWHERE\n\t\t\t\t[PrincipalIdUrl] = @PrincipalIdValue\n\t\tEND;\n\tELSE\n\t\tBEGIN\n\t\t\tRAISERROR('Unknown SPN deletion method.',16,1);\n\t\t\tRETURN 0;\n\t\tEND;\n\n\t--secondary defensive checks\n\tIF NOT EXISTS\n\t\t(\n\t\tSELECT [OrchestratorName] FROM [procfwk].[Orchestrators] WHERE [OrchestratorName] = @OrchestratorName AND [OrchestratorType] = @OrchestratorType\n\t\t)\n\t\tBEGIN\n\t\t\tSET @ErrorDetails = 'Invalid Orchestrator name. Please ensure the Orchestrator metadata exists.'\n\t\t\tRAISERROR(@ErrorDetails, 16, 1);\n\t\t\tRETURN 0;\n\t\tEND\n\n\tIF @CredentialId IS NULL\n\t\tBEGIN\n\t\t\tSET @ErrorDetails = 'Invalid Service Principal Id Value provided. Please ensure the Service Principal exists.'\n\t\t\tRAISERROR(@ErrorDetails, 16, 1);\n\t\t\tRETURN 0;\n\t\tEND\n\n\t--delete SPN for specific pipeline\n\tIF @SpecificPipelineName IS NOT NULL\n\t\tBEGIN\n\t\t\tIF NOT EXISTS\n\t\t\t\t( \n\t\t\t\tSELECT [PipelineName] FROM [procfwk].[Pipelines] WHERE [PipelineName] = @SpecificPipelineName\n\t\t\t\t)\n\t\t\t\tBEGIN\n\t\t\t\t\tSET @ErrorDetails = 'Invalid Pipeline name. Please ensure the Pipeline metadata exists.'\n\t\t\t\t\tRAISERROR(@ErrorDetails, 16, 1);\n\t\t\t\t\tRETURN 0;\n\t\t\t\tEND\n\t\t\t\n\t\t\t--delete links\n\t\t\tDELETE\n\t\t\t\tL\n\t\t\tFROM\n\t\t\t\t[procfwk].[PipelineAuthLink] L\n\t\t\t\tINNER JOIN [procfwk].[Pipelines] P\n\t\t\t\t\tON L.[PipelineId] = P.[PipelineId]\n\t\t\t\tINNER JOIN [procfwk].[Orchestrators] D\n\t\t\t\t\tON P.[OrchestratorId] = D.[OrchestratorId]\n\t\t\t\t\t\tAND L.[OrchestratorId] = D.[OrchestratorId]\n\t\t\t\tINNER JOIN [dbo].[ServicePrincipals] S\n\t\t\t\t\tON L.[CredentialId] = S.[CredentialId]\n\t\t\tWHERE\n\t\t\t\tP.[PipelineName] = @SpecificPipelineName\n\t\t\t\tAND D.[OrchestratorName] = @OrchestratorName\n\t\t\t\tAND D.[OrchestratorType] = @OrchestratorType\n\t\t\t\tAND S.[CredentialId] = @CredentialId;\n\t\tEND\n\tELSE\n\t\tBEGIN\n\t\t\t--delete links\n\t\t\tDELETE\n\t\t\t\tL\n\t\t\tFROM\n\t\t\t\t[procfwk].[PipelineAuthLink] L\n\t\t\t\tINNER JOIN [procfwk].[Orchestrators] D\n\t\t\t\t\tON L.[OrchestratorId] = D.[OrchestratorId]\n\t\t\t\tINNER JOIN [dbo].[ServicePrincipals] S\n\t\t\t\t\tON L.[CredentialId] = S.[CredentialId]\n\t\t\tWHERE\n\t\t\t\tD.[OrchestratorName] = @OrchestratorName\n\t\t\t\tAND D.[OrchestratorType] = @OrchestratorType\n\t\t\t\tAND S.[CredentialId] = @CredentialId;\n\t\tEND\n\n\t--finall, delete principal only if not still used by other pipelines\n\tDELETE \n\t\tSP\n\tFROM \n\t\t[dbo].[ServicePrincipals] SP\n\t\tLEFT OUTER JOIN [procfwk].[PipelineAuthLink] AL\n\t\t\tON SP.[CredentialId] = AL.[CredentialId]\n\tWHERE \n\t\tSP.[CredentialId] = @CredentialId\n\t\tAND AL.[CredentialId] IS NULL;\nEND;"
},
{
"path": "MetadataDB/procfwkHelpers/Stored Procedures/GetExecutionDetails.sql",
"content": "CREATE PROCEDURE [procfwkHelpers].[GetExecutionDetails]\n\t(\n\t@LocalExecutionId UNIQUEIDENTIFIER = NULL\n\t)\nAS\nBEGIN\n\n\t--Get last execution ID\n\tIF @LocalExecutionId IS NULL\n\tBEGIN\n\t\tWITH maxLog AS\n\t\t\t(\n\t\t\tSELECT\n\t\t\t\tMAX([LogId]) AS MaxLogId\n\t\t\tFROM\n\t\t\t\t[procfwk].[ExecutionLog]\n\t\t\t)\n\t\tSELECT\n\t\t\t@LocalExecutionId = el1.[LocalExecutionId]\n\t\tFROM\n\t\t\t[procfwk].[ExecutionLog] el1\n\t\t\tINNER JOIN maxLog\n\t\t\t\tON maxLog.[MaxLogId] = el1.[LogId];\n\tEND;\n\n\t--Execution Summary\n\tSELECT\n\t\tCAST(el2.[StageId] AS VARCHAR(5)) + ' - ' + stgs.[StageName] AS Stage,\n\t\tCOUNT(0) AS RecordCount,\n\t\tDATEDIFF(MINUTE, MIN(el2.[StartDateTime]), MAX(el2.[EndDateTime])) DurationMinutes\n\tFROM\n\t\t[procfwk].[ExecutionLog] el2\n\t\tINNER JOIN [procfwk].[Stages] stgs\n\t\t\tON el2.[StageId] = stgs.[StageId]\n\tWHERE\n\t\tel2.[LocalExecutionId] = @LocalExecutionId\n\tGROUP BY\n\t\tCAST(el2.[StageId] AS VARCHAR(5)) + ' - ' + stgs.[StageName]\n\tORDER BY\n\t\tCAST(el2.[StageId] AS VARCHAR(5)) + ' - ' + stgs.[StageName];\n\n\t--Full execution details\n\tSELECT\n\t\tel3.[LogId],\n\t\tel3.[LocalExecutionId],\n\t\tel3.[OrchestratorType],\n\t\tel3.[OrchestratorName],\n\t\tel3.[StageId],\n\t\tstgs.[StageName],\n\t\tel3.[PipelineId],\n\t\tel3.[PipelineName],\n\t\tel3.[StartDateTime],\n\t\tel3.[EndDateTime],\n\t\tISNULL(DATEDIFF(MINUTE, el3.[StartDateTime], el3.[EndDateTime]),0) AS DurationMinutes,\n\t\tel3.[PipelineStatus],\n\t\tel3.[PipelineRunId],\n\t\tel3.[PipelineParamsUsed],\n\t\terrLog.[ActivityRunId],\n\t\terrLog.[ActivityName],\n\t\terrLog.[ActivityType],\n\t\terrLog.[ErrorCode],\n\t\terrLog.[ErrorType],\n\t\terrLog.[ErrorMessage]\n\tFROM \n\t\t[procfwk].[ExecutionLog] el3\n\t\tLEFT OUTER JOIN [procfwk].[ErrorLog] errLog\n\t\t\tON el3.[LocalExecutionId] = errLog.[LocalExecutionId]\n\t\t\t\tAND el3.[PipelineRunId] = errLog.[PipelineRunId]\n\t\tINNER JOIN [procfwk].[Stages] stgs\n\t\t\tON el3.[StageId] = stgs.[StageId]\n\tWHERE\n\t\tel3.[LocalExecutionId] = @LocalExecutionId\n\tORDER BY\n\t\tel3.[PipelineId],\n\t\tel3.[StartDateTime];\nEND;"
},
{
"path": "MetadataDB/procfwkHelpers/Stored Procedures/GetServicePrincipal.sql",
"content": "CREATE PROCEDURE [procfwkHelpers].[GetServicePrincipal]\n\t(\n\t@OrchestratorName NVARCHAR(200),\n\t@OrchestratorType CHAR(3),\n\t@PipelineName NVARCHAR(200) = NULL\n\t)\nAS\nBEGIN\n\tSET NOCOUNT ON;\n\n\tDECLARE @Id NVARCHAR(MAX)\n\tDECLARE @Secret NVARCHAR(MAX)\n\n\tIF ([procfwk].[GetPropertyValueInternal]('SPNHandlingMethod')) = 'StoreInDatabase'\n\t\tBEGIN\n\t\t\t--get auth details regardless of being pipeline specific and regardless of a pipeline param being passed\n\t\t\t;WITH cte AS\n\t\t\t\t(\n\t\t\t\tSELECT DISTINCT\n\t\t\t\t\tS.[PrincipalId] AS Id,\n\t\t\t\t\tCAST(DECRYPTBYPASSPHRASE(CONCAT(@OrchestratorName, @PipelineName), S.[PrincipalSecret]) AS NVARCHAR(MAX)) AS [Secret]\n\t\t\t\tFROM\n\t\t\t\t\t[dbo].[ServicePrincipals] S\n\t\t\t\t\tINNER JOIN [procfwk].[PipelineAuthLink] L\n\t\t\t\t\t\tON S.[CredentialId] = L.[CredentialId]\n\t\t\t\t\tINNER JOIN [procfwk].[Pipelines] P\n\t\t\t\t\t\tON L.[PipelineId] = P.[PipelineId]\n\t\t\t\t\tINNER JOIN [procfwk].[Orchestrators] D\n\t\t\t\t\t\tON P.[OrchestratorId] = D.[OrchestratorId]\n\t\t\t\t\t\t\tAND L.[OrchestratorId] = D.[OrchestratorId]\n\t\t\t\tWHERE\n\t\t\t\t\tP.[PipelineName] = @PipelineName\n\t\t\t\t\tAND D.[OrchestratorName] = @OrchestratorName\n\t\t\t\t\tAND D.[OrchestratorType] = @OrchestratorType\n\t\t\t\n\t\t\t\tUNION\n\n\t\t\t\tSELECT DISTINCT\n\t\t\t\t\tS.[PrincipalId] AS Id,\n\t\t\t\t\tCAST(DECRYPTBYPASSPHRASE(@OrchestratorName, S.[PrincipalSecret]) AS NVARCHAR(MAX)) AS [Secret]\n\t\t\t\tFROM\n\t\t\t\t\t[dbo].[ServicePrincipals] S\n\t\t\t\t\tINNER JOIN [procfwk].[PipelineAuthLink] L\n\t\t\t\t\t\tON S.[CredentialId] = L.[CredentialId]\n\t\t\t\t\tINNER JOIN [procfwk].[Orchestrators] D\n\t\t\t\t\t\tON L.[OrchestratorId] = D.[OrchestratorId]\n\t\t\t\tWHERE\n\t\t\t\t\tD.[OrchestratorName] = @OrchestratorName\n\t\t\t\t\tAND D.[OrchestratorType] = @OrchestratorType\n\t\t\t\t)\n\t\t\tSELECT TOP 1\n\t\t\t\t@Id = [Id],\n\t\t\t\t@Secret = [Secret]\n\t\t\tFROM\n\t\t\t\tcte\n\t\t\tWHERE\n\t\t\t\t[Secret] IS NOT NULL\n\t\tEND\n\tELSE IF ([procfwk].[GetPropertyValueInternal]('SPNHandlingMethod')) = 'StoreInKeyVault'\n\t\tBEGIN\n\t\t\t--get auth details regardless of being pipeline specific and regardless of a pipeline param being passed\n\t\t\t;WITH cte AS\n\t\t\t\t(\n\t\t\t\tSELECT DISTINCT\n\t\t\t\t\tS.[PrincipalIdUrl] AS Id,\n\t\t\t\t\tS.[PrincipalSecretUrl] AS [Secret]\n\t\t\t\tFROM\n\t\t\t\t\t[dbo].[ServicePrincipals] S\n\t\t\t\t\tINNER JOIN [procfwk].[PipelineAuthLink] L\n\t\t\t\t\t\tON S.[CredentialId] = L.[CredentialId]\n\t\t\t\t\tINNER JOIN [procfwk].[Pipelines] P\n\t\t\t\t\t\tON L.[PipelineId] = P.[PipelineId]\n\t\t\t\t\tINNER JOIN [procfwk].[Orchestrators] D\n\t\t\t\t\t\tON P.[OrchestratorId] = D.[OrchestratorId]\n\t\t\t\t\t\t\tAND L.[OrchestratorId] = D.[OrchestratorId]\n\t\t\t\tWHERE\n\t\t\t\t\tP.[PipelineName] = @PipelineName\n\t\t\t\t\tAND D.[OrchestratorName] = @OrchestratorName\n\t\t\t\t\tAND D.[OrchestratorType] = @OrchestratorType\n\t\t\t\n\t\t\t\tUNION\n\n\t\t\t\tSELECT DISTINCT\n\t\t\t\t\tS.[PrincipalIdUrl] AS Id,\n\t\t\t\t\tS.[PrincipalSecretUrl] AS [Secret]\n\t\t\t\tFROM\n\t\t\t\t\t[dbo].[ServicePrincipals] S\n\t\t\t\t\tINNER JOIN [procfwk].[PipelineAuthLink] L\n\t\t\t\t\t\tON S.[CredentialId] = L.[CredentialId]\n\t\t\t\t\tINNER JOIN [procfwk].[Orchestrators] D\n\t\t\t\t\t\tON L.[OrchestratorId] = D.[OrchestratorId]\n\t\t\t\tWHERE\n\t\t\t\t\tD.[OrchestratorName] = @OrchestratorName\n\t\t\t\t\tAND D.[OrchestratorType] = @OrchestratorType\n\t\t\t\t)\n\t\t\tSELECT TOP 1\n\t\t\t\t@Id = [Id],\n\t\t\t\t@Secret = [Secret]\n\t\t\tFROM\n\t\t\t\tcte\n\t\t\tWHERE\n\t\t\t\t[Secret] IS NOT NULL\n\t\tEND\n\tELSE\n\t\tBEGIN\n\t\t\tRAISERROR('Unknown SPN retrieval method.',16,1);\n\t\t\tRETURN 0;\n\t\tEND\n\n\t--return usable values\n\tSELECT\n\t\t@Id AS Id,\n\t\t@Secret AS [Secret]\nEND;"
},
{
"path": "MetadataDB/procfwkHelpers/Stored Procedures/SetDefaultAlertOutcomes.sql",
"content": "CREATE PROCEDURE [procfwkHelpers].[SetDefaultAlertOutcomes]\nAS\nBEGIN\n\tTRUNCATE TABLE [procfwk].[AlertOutcomes];\n\n\tINSERT INTO [procfwk].[AlertOutcomes] \n\t\t(\n\t\t[PipelineOutcomeStatus]\n\t\t)\n\tVALUES \n\t\t('All'),\n\t\t('Success'),\n\t\t('Failed'),\n\t\t('Unknown'),\n\t\t('Cancelled');\nEND;"
},
{
"path": "MetadataDB/procfwkHelpers/Stored Procedures/SetDefaultBatchStageLink.sql",
"content": "CREATE PROCEDURE [procfwkHelpers].[SetDefaultBatchStageLink]\nAS\nBEGIN\n\tTRUNCATE TABLE [procfwk].[BatchStageLink]\n\n\tINSERT INTO [procfwk].[BatchStageLink]\n\t\t(\n\t\t[BatchId],\n\t\t[StageId]\n\t\t)\n\tSELECT\n\t\tb.[BatchId],\n\t\ts.[StageId]\n\tFROM\n\t\t[procfwk].[Batches] b\n\t\tINNER JOIN [procfwk].[Stages] s\n\t\t\tON s.[StageName] <> 'Speed'\n\tWHERE\n\t\tb.[BatchName] = 'Daily'\n\n\tUNION ALL\n\n\tSELECT\n\t\tb.[BatchId],\n\t\ts.[StageId]\n\tFROM\n\t\t[procfwk].[Batches] b\n\t\tINNER JOIN [procfwk].[Stages] s\n\t\t\tON s.[StageName] = 'Speed'\n\tWHERE\n\t\tb.[BatchName] = 'Hourly'\nEND;"
},
{
"path": "MetadataDB/procfwkHelpers/Stored Procedures/SetDefaultBatches.sql",
"content": "CREATE PROCEDURE [procfwkHelpers].[SetDefaultBatches]\nAS\nBEGIN\n\tDECLARE @Batches TABLE\n\t\t(\n\t\t[BatchName] [VARCHAR](225) NOT NULL,\n\t\t[BatchDescription] [VARCHAR](4000) NULL,\n\t\t[Enabled] [BIT] NOT NULL\n\t\t)\n\t\n\tINSERT @Batches\n\t\t(\n\t\t[BatchName], \n\t\t[BatchDescription], \n\t\t[Enabled]\n\t\t) \n\tVALUES \n\t\t('Daily', N'Daily Worker Pipelines.', 1),\n\t\t('Hourly', N'Hourly Worker Pipelines.', 1);\t\n\n\tMERGE INTO [procfwk].[Batches] AS tgt\n\tUSING \n\t\t@Batches AS src\n\t\t\tON tgt.[BatchName] = src.[BatchName]\n\tWHEN MATCHED THEN\n\t\tUPDATE\n\t\tSET\n\t\t\ttgt.[BatchDescription] = src.[BatchDescription],\n\t\t\ttgt.[Enabled] = src.[Enabled]\n\tWHEN NOT MATCHED BY TARGET THEN\n\t\tINSERT\n\t\t\t(\n\t\t\t[BatchName],\n\t\t\t[BatchDescription],\n\t\t\t[Enabled]\n\t\t\t)\n\t\tVALUES\n\t\t\t(\n\t\t\tsrc.[BatchName],\n\t\t\tsrc.[BatchDescription],\n\t\t\tsrc.[Enabled]\n\t\t\t)\n\tWHEN NOT MATCHED BY SOURCE THEN\n\t\tDELETE;\t\nEND;"
},
{
"path": "MetadataDB/procfwkHelpers/Stored Procedures/SetDefaultOrchestrators.sql",
"content": "CREATE PROCEDURE [procfwkHelpers].[SetDefaultOrchestrators]\nAS\nBEGIN\n\tDECLARE @Orchestrators TABLE \n\t\t(\n\t\t[OrchestratorName] NVARCHAR(200) NOT NULL,\n\t\t[OrchestratorType] CHAR(3) NOT NULL,\n\t\t[IsFrameworkOrchestrator] BIT NOT NULL,\n\t\t[ResourceGroupName] NVARCHAR(200) NOT NULL,\n\t\t[SubscriptionId] UNIQUEIDENTIFIER NOT NULL,\n\t\t[Description] NVARCHAR(MAX) NULL\n\t\t)\n\t\n\tINSERT INTO @Orchestrators\n\t\t(\n\t\t[OrchestratorName],\n\t\t[OrchestratorType],\n\t\t[IsFrameworkOrchestrator],\n\t\t[Description],\n\t\t[ResourceGroupName],\n\t\t[SubscriptionId]\n\t\t)\n\tVALUES\n\t\t('FrameworkFactory','ADF',1,'Example Data Factory used for development.','ADF.procfwk','12345678-1234-1234-1234-012345678910'),\n\t\t('FrameworkFactoryDev','ADF',0,'Example Data Factory used for development deployments.','ADF.procfwk','12345678-1234-1234-1234-012345678910'),\n\t\t('FrameworkFactoryTest','ADF',0,'Example Data Factory used for testing.','ADF.procfwk','12345678-1234-1234-1234-012345678910'),\n\t\t('WorkersFactory','ADF',0,'Example Data Factory used to house worker pipelines.','ADF.procfwk','12345678-1234-1234-1234-012345678910'),\n\t\t('procfwkforsynapse','SYN',0,'Example Synapse instance used to house all pipelines.','ADF.procfwk','12345678-1234-1234-1234-012345678910');\n\n\tMERGE INTO [procfwk].[Orchestrators] AS tgt\n\tUSING \n\t\t@Orchestrators AS src\n\t\t\tON tgt.[OrchestratorName] = src.[OrchestratorName]\n\t\t\t\tAND tgt.[OrchestratorType] = src.[OrchestratorType]\n\tWHEN MATCHED THEN\n\t\tUPDATE\n\t\tSET\n\t\t\ttgt.[IsFrameworkOrchestrator] = src.[IsFrameworkOrchestrator],\n\t\t\ttgt.[Description] = src.[Description],\n\t\t\ttgt.[ResourceGroupName] = src.[ResourceGroupName],\n\t\t\ttgt.[SubscriptionId] = src.[SubscriptionId]\n\tWHEN NOT MATCHED BY TARGET THEN\n\t\tINSERT\n\t\t\t(\n\t\t\t[OrchestratorName],\n\t\t\t[OrchestratorType],\n\t\t\t[IsFrameworkOrchestrator],\n\t\t\t[Description],\n\t\t\t[ResourceGroupName],\n\t\t\t[SubscriptionId]\n\t\t\t)\n\t\tVALUES\n\t\t\t(\n\t\t\tsrc.[OrchestratorName],\n\t\t\tsrc.[OrchestratorType],\n\t\t\tsrc.[IsFrameworkOrchestrator],\n\t\t\tsrc.[Description],\n\t\t\tsrc.[ResourceGroupName],\n\t\t\tsrc.[SubscriptionId]\n\t\t\t)\n\tWHEN NOT MATCHED BY SOURCE THEN\n\t\tDELETE;\nEND;"
},
{
"path": "MetadataDB/procfwkHelpers/Stored Procedures/SetDefaultPipelineDependants.sql",
"content": "CREATE PROCEDURE [procfwkHelpers].[SetDefaultPipelineDependants]\nAS\nBEGIN\n\tEXEC [procfwkHelpers].[AddPipelineDependant]\n\t\t@PipelineName = 'Intentional Error',\n\t\t@DependantPipelineName = 'Wait 5';\n\n\tEXEC [procfwkHelpers].[AddPipelineDependant]\n\t\t@PipelineName = 'Intentional Error',\n\t\t@DependantPipelineName = 'Wait 6';\n\n\tEXEC [procfwkHelpers].[AddPipelineDependant]\n\t\t@PipelineName = 'Wait 6',\n\t\t@DependantPipelineName = 'Wait 9';\n\n\tEXEC [procfwkHelpers].[AddPipelineDependant]\n\t\t@PipelineName = 'Wait 9',\n\t\t@DependantPipelineName = 'Wait 10';\nEND;"
},
{
"path": "MetadataDB/procfwkHelpers/Stored Procedures/SetDefaultPipelineParameters.sql",
"content": "CREATE PROCEDURE [procfwkHelpers].[SetDefaultPipelineParameters]\nAS\nBEGIN\n\tDECLARE @PipelineParameters TABLE\n\t\t(\n\t\t[PipelineId] [INT] NOT NULL,\n\t\t[ParameterName] [VARCHAR](128) NOT NULL,\n\t\t[ParameterValue] [NVARCHAR](MAX) NULL\n\t\t)\n\n\tINSERT @PipelineParameters\n\t\t(\n\t\t[PipelineId], \n\t\t[ParameterName], \n\t\t[ParameterValue]\n\t\t) \n\tVALUES \n\t\t(1, 'WaitTime', '3'),\n\t\t(2, 'WaitTime', '6'),\n\t\t(6, 'WaitTime', '9'),\n\t\t(4, 'WaitTime', '5'),\n\t\t(5, 'WaitTime', '2'),\n\t\t(3, 'RaiseErrors', 'false'),\n\t\t(3, 'WaitTime', '10'),\n\t\t(7, 'WaitTime', '3'),\n\t\t(8, 'WaitTime', '5'),\n\t\t(9, 'WaitTime', '7'),\n\t\t(11, 'WaitTime', '10');\n\n\tMERGE INTO [procfwk].[PipelineParameters] AS tgt\n\tUSING \n\t\t@PipelineParameters AS src\n\t\t\tON tgt.[PipelineId] = src.[PipelineId]\n\t\t\t\tAND tgt.[ParameterName] = src.[ParameterName]\n\tWHEN MATCHED THEN\n\t\tUPDATE\n\t\tSET\n\t\t\ttgt.[ParameterValue] = src.[ParameterValue]\n\tWHEN NOT MATCHED BY TARGET THEN\n\t\tINSERT\n\t\t\t(\n\t\t\t[PipelineId], \n\t\t\t[ParameterName], \n\t\t\t[ParameterValue]\n\t\t\t) \n\t\tVALUES\n\t\t\t(\n\t\t\tsrc.[PipelineId], \n\t\t\tsrc.[ParameterName], \n\t\t\tsrc.[ParameterValue]\n\t\t\t) \n\tWHEN NOT MATCHED BY SOURCE THEN\n\t\tDELETE;\t\nEND;"
},
{
"path": "MetadataDB/procfwkHelpers/Stored Procedures/SetDefaultPipelines.sql",
"content": "CREATE PROCEDURE [procfwkHelpers].[SetDefaultPipelines]\nAS\nBEGIN\n\tDECLARE @Pipelines TABLE\n\t\t(\n\t\t[OrchestratorId] [INT] NOT NULL,\n\t\t[StageId] [INT] NOT NULL,\n\t\t[PipelineName] [NVARCHAR](200) NOT NULL,\n\t\t[LogicalPredecessorId] [INT] NULL,\n\t\t[Enabled] [BIT] NOT NULL\n\t\t)\n\n\tINSERT @Pipelines\n\t\t(\n\t\t[OrchestratorId],\n\t\t[StageId],\n\t\t[PipelineName], \n\t\t[LogicalPredecessorId],\n\t\t[Enabled]\n\t\t) \n\tVALUES \n\t\t(1,1\t,'Wait 1'\t\t\t\t,NULL\t\t,1),\n\t\t(1,1\t,'Wait 2'\t\t\t\t,NULL\t\t,1),\n\t\t(1,1\t,'Intentional Error'\t,NULL\t\t,1),\n\t\t(1,1\t,'Wait 3'\t\t\t\t,NULL\t\t,1),\n\t\t(1,2\t,'Wait 4'\t\t\t\t,NULL\t\t,1),\n\t\t(1,2\t,'Wait 5'\t\t\t\t,1\t\t\t,1),\n\t\t(1,2\t,'Wait 6'\t\t\t\t,1\t\t\t,1),\n\t\t(1,2\t,'Wait 7'\t\t\t\t,NULL\t\t,1),\n\t\t(1,3\t,'Wait 8'\t\t\t\t,1\t\t\t,1),\n\t\t(1,3\t,'Wait 9'\t\t\t\t,6\t\t\t,1),\n\t\t(1,4\t,'Wait 10'\t\t\t\t,9\t\t\t,1),\n\t\t--speed\n\t\t(1,5\t,'Wait 1'\t\t\t\t,NULL\t\t,0),\n\t\t(1,5\t,'Wait 2'\t\t\t\t,NULL\t\t,0),\n\t\t(1,5\t,'Wait 3'\t\t\t\t,NULL\t\t,0),\n\t\t(1,5\t,'Wait 4'\t\t\t\t,NULL\t\t,0),\n\t\t--synapse\n\t\t(5,1\t,'Wait 1'\t\t\t\t,NULL\t\t,1),\n\t\t(5,1\t,'Wait 2'\t\t\t\t,NULL\t\t,1),\n\t\t(5,1\t,'Wait 3'\t\t\t\t,NULL\t\t,1),\n\t\t(5,1\t,'Wait 4'\t\t\t\t,NULL\t\t,1);\n\n\tMERGE INTO [procfwk].[Pipelines] AS tgt\n\tUSING \n\t\t@Pipelines AS src\n\t\t\tON tgt.[OrchestratorId] = src.[OrchestratorId]\n\t\t\t\tAND tgt.[PipelineName] = src.[PipelineName]\n\t\t\t\tAND tgt.[StageId] = src.[StageId]\n\tWHEN MATCHED THEN\n\t\tUPDATE\n\t\tSET\n\t\t\ttgt.[LogicalPredecessorId] = src.[LogicalPredecessorId],\n\t\t\ttgt.[Enabled] = src.[Enabled]\n\tWHEN NOT MATCHED BY TARGET THEN\n\t\tINSERT\n\t\t\t(\n\t\t\t[OrchestratorId],\n\t\t\t[StageId],\n\t\t\t[PipelineName], \n\t\t\t[LogicalPredecessorId],\n\t\t\t[Enabled]\n\t\t\t)\n\t\tVALUES\n\t\t\t(\n\t\t\tsrc.[OrchestratorId],\n\t\t\tsrc.[StageId],\n\t\t\tsrc.[PipelineName], \n\t\t\tsrc.[LogicalPredecessorId],\n\t\t\tsrc.[Enabled]\n\t\t\t)\n\tWHEN NOT MATCHED BY SOURCE THEN\n\t\tDELETE;\t\nEND;"
},
{
"path": "MetadataDB/procfwkHelpers/Stored Procedures/SetDefaultProperties.sql",
"content": "CREATE PROCEDURE [procfwkHelpers].[SetDefaultProperties]\nAS\nBEGIN\n\tEXEC [procfwkHelpers].[AddProperty]\n\t\t@PropertyName = N'OverideRestart',\n\t\t@PropertyValue = N'0',\n\t\t@Description = N'Should processing not be restarted from the point of failure or should a new execution will be created regardless. 1 = Start New, 0 = Restart. ';\n\n\tEXEC [procfwkHelpers].[AddProperty]\n\t\t@PropertyName = N'PipelineStatusCheckDuration',\n\t\t@PropertyValue = N'30',\n\t\t@Description = N'Duration applied to the Wait activity within the Infant pipeline Until iterations.';\n\n\tEXEC [procfwkHelpers].[AddProperty]\n\t\t@PropertyName = N'UnknownWorkerResultBlocks',\n\t\t@PropertyValue = N'1',\n\t\t@Description = N'If a worker pipeline returns an unknown status. Should this block and fail downstream pipeline? 1 = Yes, 0 = No.';\n\n\tEXEC [procfwkHelpers].[AddProperty]\n\t\t@PropertyName = N'CancelledWorkerResultBlocks',\n\t\t@PropertyValue = N'1',\n\t\t@Description = N'If a worker pipeline returns an cancelled status. Should this block and fail downstream pipeline? 1 = Yes, 0 = No.';\n\n\tEXEC [procfwkHelpers].[AddProperty]\n\t\t@PropertyName = N'UseFrameworkEmailAlerting',\n\t\t@PropertyValue = N'0',\n\t\t@Description = N'Do you want the framework to handle pipeline email alerts via the database metadata? 1 = Yes, 0 = No.';\n\n\tEXEC [procfwkHelpers].[AddProperty]\n\t\t@PropertyName = N'EmailAlertBodyTemplate',\n\t\t@PropertyValue = \n\t\tN'
Pipeline Name: ##PipelineName###
\n\tStatus: ##Status###
\n\tExecution ID: ##ExecId###
\n\tRun ID: ##RunId###
\n\tStart Date Time: ##StartDateTime###
\n\tEnd Date Time: ##EndDateTime###
\n\tDuration (Minutes): ##Duration###
\n\tCalled by Orchestrator: ##CalledByOrc###
\n\tExecuted by Orchestrator Type: ##ExecutedByOrcType###
\n\tExecuted by Orchestrator: ##ExecutedByOrc###
',\n\t\t@Description = N'Basic HTML template of execution information used as the eventual body in email alerts sent.';\n\n\tEXEC [procfwkHelpers].[AddProperty]\n\t\t@PropertyName = N'FailureHandling',\n\t\t@PropertyValue = N'Simple',\n\t\t@Description = N'Accepted values: None, Simple, DependencyChain. Controls processing bahaviour in the event of Worker failures. See v1.8 release notes for full details.';\n\n\tEXEC [procfwkHelpers].[AddProperty]\n\t\t@PropertyName = N'SPNHandlingMethod',\n\t\t@PropertyValue = N'StoreInDatabase',\n\t\t@Description = N'Accepted values: StoreInDatabase, StoreInKeyVault. See v1.8.2 release notes for full details.';\n\n\tEXEC [procfwkHelpers].[AddProperty]\n\t\t@PropertyName = N'ExecutionPrecursorProc',\n\t\t@PropertyValue = N'[dbo].[ExampleCustomExecutionPrecursor]',\n\t\t@Description = N'This procedure will be called first in the parent pipeline and can be used to perform/update any required custom behaviour in the framework execution. For example, enable/disable Worker pipelines given a certain run time/day. Invalid proc name values will be ignored.'\n\n\tEXEC [procfwkHelpers].[AddProperty]\n\t\t@PropertyName = N'UseExecutionBatches',\n\t\t@PropertyValue = N'0',\n\t\t@Description = N'Establishes if execution batches are used as a level above execution stages within the framework. 1 = True, 0 = False.';\n\n\tEXEC [procfwkHelpers].[AddProperty]\n\t\t@PropertyName = N'FrameworkFactoryResourceGroup',\n\t\t@PropertyValue = N'ADF.procfwk',\n\t\t@Description = N'Supports various queries where the framework factory is inspecting itself and the resource group cant be inferred.';\n\n\tEXEC [procfwkHelpers].[AddProperty]\n\t\t@PropertyName = N'PreviousPipelineRunsQueryRange',\n\t\t@PropertyValue = N'-1',\n\t\t@Description = N'Used as a date range, today +- this value, when checking for if an execution for a given pipeline is already running. Must include +- symbol in value.';\nEND;"
},
{
"path": "MetadataDB/procfwkHelpers/Stored Procedures/SetDefaultRecipientPipelineAlerts.sql",
"content": "CREATE PROCEDURE [procfwkHelpers].[SetDefaultRecipientPipelineAlerts]\nAS\nBEGIN\n\tEXEC [procfwkHelpers].[AddRecipientPipelineAlerts]\n\t\t@RecipientName = N'Test User 1',\n\t\t@AlertForStatus = 'All';\n\n\tEXEC [procfwkHelpers].[AddRecipientPipelineAlerts]\n\t\t@RecipientName = N'Test User 2',\n\t\t@PipelineName = 'Intentional Error',\n\t\t@AlertForStatus = 'Failed';\n\n\tEXEC [procfwkHelpers].[AddRecipientPipelineAlerts]\n\t\t@RecipientName = N'Test User 3',\n\t\t@PipelineName = 'Wait 1',\n\t\t@AlertForStatus = 'Success, Failed, Cancelled';\t\nEND;"
},
{
"path": "MetadataDB/procfwkHelpers/Stored Procedures/SetDefaultRecipients.sql",
"content": "CREATE PROCEDURE [procfwkHelpers].[SetDefaultRecipients]\nAS\nBEGIN\n\tDECLARE @Recipients TABLE\n\t\t(\n\t\t[Name] [VARCHAR](255) NULL,\n\t\t[EmailAddress] [NVARCHAR](500) NOT NULL,\n\t\t[MessagePreference] [CHAR](3) NOT NULL,\n\t\t[Enabled] [BIT] NOT NULL\n\t\t)\n\n\tINSERT INTO @Recipients\n\t\t(\n\t\t[Name],\n\t\t[EmailAddress],\n\t\t[MessagePreference],\n\t\t[Enabled]\n\t\t)\n\tVALUES\n\t\t('Test User 1','test.user1@adfprocfwk.com', 'TO', 1),\n\t\t('Test User 2','test.user2@adfprocfwk.com', 'CC', 1),\n\t\t('Test User 3','test.user3@adfprocfwk.com', 'BCC', 1);\n\n\tMERGE INTO [procfwk].[Recipients] AS tgt\n\tUSING \n\t\t@Recipients AS src\n\t\t\tON tgt.[Name] = src.[Name]\n\tWHEN MATCHED THEN\n\t\tUPDATE\n\t\tSET\n\t\t\ttgt.[EmailAddress] = src.[EmailAddress],\n\t\t\ttgt.[MessagePreference] = src.[MessagePreference],\n\t\t\ttgt.[Enabled] = src.[Enabled]\n\tWHEN NOT MATCHED BY TARGET THEN\n\t\tINSERT\n\t\t\t(\n\t\t\t[Name],\n\t\t\t[EmailAddress],\n\t\t\t[MessagePreference],\n\t\t\t[Enabled]\n\t\t\t)\n\t\tVALUES\n\t\t\t(\n\t\t\tsrc.[Name],\n\t\t\tsrc.[EmailAddress],\n\t\t\tsrc.[MessagePreference],\n\t\t\tsrc.[Enabled]\n\t\t\t)\n\tWHEN NOT MATCHED BY SOURCE THEN\n\t\tDELETE;\t\nEND;"
},
{
"path": "MetadataDB/procfwkHelpers/Stored Procedures/SetDefaultStages.sql",
"content": "CREATE PROCEDURE [procfwkHelpers].[SetDefaultStages]\nAS\nBEGIN\n\tDECLARE @Stages TABLE\n\t\t(\n\t\t[StageName] [VARCHAR](225) NOT NULL,\n\t\t[StageDescription] [VARCHAR](4000) NULL,\n\t\t[Enabled] [BIT] NOT NULL\n\t\t)\n\t\n\tINSERT @Stages\n\t\t(\n\t\t[StageName], \n\t\t[StageDescription], \n\t\t[Enabled]\n\t\t) \n\tVALUES \n\t\t('Extract', N'Ingest all data from source systems.', 1),\n\t\t('Transform', N'Transform ingested data and apply business logic.', 1),\n\t\t('Load', N'Load transformed data into data warehouse layer.', 1),\n\t\t('Serve', N'Load transformed data into semantic layer.', 1),\n\t\t('Speed', N'Regular loading of frequently used data.', 0);\n\n\tMERGE INTO [procfwk].[Stages] AS tgt\n\tUSING \n\t\t@Stages AS src\n\t\t\tON tgt.[StageName] = src.[StageName]\n\tWHEN MATCHED THEN\n\t\tUPDATE\n\t\tSET\n\t\t\ttgt.[StageDescription] = src.[StageDescription],\n\t\t\ttgt.[Enabled] = src.[Enabled]\n\tWHEN NOT MATCHED BY TARGET THEN\n\t\tINSERT\n\t\t\t(\n\t\t\t[StageName],\n\t\t\t[StageDescription],\n\t\t\t[Enabled]\n\t\t\t)\n\t\tVALUES\n\t\t\t(\n\t\t\tsrc.[StageName],\n\t\t\tsrc.[StageDescription],\n\t\t\tsrc.[Enabled]\n\t\t\t)\n\tWHEN NOT MATCHED BY SOURCE THEN\n\t\tDELETE;\t\nEND;"
},
{
"path": "MetadataDB/procfwkHelpers/Stored Procedures/SetDefaultSubscription.sql",
"content": "CREATE PROCEDURE [procfwkHelpers].[SetDefaultSubscription]\nAS\nBEGIN\n\tDECLARE @Subscriptions TABLE\n\t\t(\n\t\t[SubscriptionId] UNIQUEIDENTIFIER NOT NULL,\n\t\t[Name] NVARCHAR(200) NOT NULL,\n\t\t[Description] NVARCHAR(MAX) NULL,\n\t\t[TenantId] UNIQUEIDENTIFIER NOT NULL\n\t\t)\n\n\tINSERT INTO @Subscriptions\n\t\t(\n\t\t[SubscriptionId],\n\t\t[Name],\n\t\t[Description],\n\t\t[TenantId]\n\t\t)\n\tVALUES\n\t\t('12345678-1234-1234-1234-012345678910', 'Default', 'Example value for development environment.', '12345678-1234-1234-1234-012345678910');\n\n\tMERGE INTO [procfwk].[Subscriptions] AS tgt\n\tUSING \n\t\t@Subscriptions AS src\n\t\t\tON tgt.[SubscriptionId] = src.[SubscriptionId]\n\tWHEN MATCHED THEN\n\t\tUPDATE\n\t\tSET\n\t\t\ttgt.[Name] = src.[Name],\n\t\t\ttgt.[Description] = src.[Description],\n\t\t\ttgt.[TenantId] = src.[TenantId]\n\tWHEN NOT MATCHED BY TARGET THEN\n\t\tINSERT\n\t\t\t(\n\t\t\t[SubscriptionId],\n\t\t\t[Name],\n\t\t\t[Description],\n\t\t\t[TenantId]\n\t\t\t)\n\t\tVALUES\n\t\t\t(\n\t\t\tsrc.[SubscriptionId],\n\t\t\tsrc.[Name],\n\t\t\tsrc.[Description],\n\t\t\tsrc.[TenantId]\n\t\t\t)\n\tWHEN NOT MATCHED BY SOURCE THEN\n\t\tDELETE;\nEND;"
},
{
"path": "MetadataDB/procfwkHelpers/Stored Procedures/SetDefaultTenant.sql",
"content": "CREATE PROCEDURE [procfwkHelpers].[SetDefaultTenant]\nAS\nBEGIN\n\tDECLARE @Tenants TABLE\n\t\t(\n\t\t[TenantId] UNIQUEIDENTIFIER NOT NULL,\n\t\t[Name] NVARCHAR(200) NOT NULL,\n\t\t[Description] NVARCHAR(MAX) NULL\n\t\t)\n\n\tINSERT INTO @Tenants\n\t\t(\n\t\t[TenantId],\n\t\t[Name],\n\t\t[Description]\n\t\t)\n\tVALUES\n\t\t('12345678-1234-1234-1234-012345678910', 'Default', 'Example value for development environment.');\n\n\tMERGE INTO [procfwk].[Tenants] AS tgt\n\tUSING \n\t\t@Tenants AS src\n\t\t\tON tgt.[TenantId] = src.[TenantId]\n\tWHEN MATCHED THEN\n\t\tUPDATE\n\t\tSET\n\t\t\ttgt.[Name] = src.[Name],\n\t\t\ttgt.[Description] = src.[Description]\t\t\n\tWHEN NOT MATCHED BY TARGET THEN\n\t\tINSERT\n\t\t\t(\n\t\t\t[TenantId],\n\t\t\t[Name],\n\t\t\t[Description]\n\t\t\t)\n\t\tVALUES\n\t\t\t(\n\t\t\tsrc.[TenantId],\n\t\t\tsrc.[Name],\n\t\t\tsrc.[Description]\n\t\t\t)\n\tWHEN NOT MATCHED BY SOURCE THEN\n\t\tDELETE;\nEND;"
},
{
"path": "MetadataDB/procfwkHelpers/Views/PipelineDependencyChains.sql",
"content": "CREATE VIEW [procfwkHelpers].[PipelineDependencyChains]\nAS\n\nSELECT \n\tps.[StageName] AS PredecessorStage,\n\tpp.[PipelineName] AS PredecessorPipeline,\n\tds.[StageName] AS DependantStage,\n\tdp.[PipelineName] AS DependantPipeline\nFROM \n\t[procfwk].[PipelineDependencies]\t\t\t\t\tpd --pipeline dependencies\n\tINNER JOIN [procfwk].[Pipelines]\t\t\t\t\tpp --predecessor pipelines\n\t\tON pd.[PipelineId] = pp.[PipelineId]\n\tINNER JOIN [procfwk].[Pipelines]\t\t\t\t\tdp --dependant pipelines\n\t\tON pd.[DependantPipelineId] = dp.[PipelineId]\n\tINNER JOIN [procfwk].[Stages]\t\t\t\t\t\tps --predecessor stage\n\t\tON pp.[StageId] = ps.[StageId]\n\tINNER JOIN [procfwk].[Stages]\t\t\t\t\t\tds --dependant stage\n\t\tON dp.[StageId] = ds.[StageId];"
},
{
"path": "MetadataDB/procfwkReporting/Views/AverageStageDuration.sql",
"content": "CREATE VIEW [procfwkReporting].[AverageStageDuration]\nAS\n\nWITH stageStartEnd AS\n\t(\n\tSELECT\n\t\t[LocalExecutionId],\n\t\t[StageId],\n\t\tMIN([StartDateTime]) AS 'StageStart',\n\t\tMAX([EndDateTime]) AS 'StageEnd'\n\tFROM\n\t\t[procfwk].[ExecutionLog]\n\tGROUP BY\n\t\t[LocalExecutionId],\n\t\t[StageId]\n\t)\n\nSELECT\n\ts.[StageId],\n\ts.[StageName],\n\ts.[StageDescription],\n\tAVG(DATEDIFF(MINUTE, stageStartEnd.[StageStart], stageStartEnd.[StageEnd])) 'AvgStageRunDurationMinutes'\nFROM\n\tstageStartEnd\n\tINNER JOIN [procfwk].[Stages] s\n\t\tON stageStartEnd.[StageId] = s.[StageId]\nGROUP BY\n\ts.[StageId],\n\ts.[StageName],\n\ts.[StageDescription]"
},
{
"path": "MetadataDB/procfwkReporting/Views/CompleteExecutionErrorLog.sql",
"content": "CREATE VIEW [procfwkReporting].[CompleteExecutionErrorLog]\nAS\n\nSELECT\n\texeLog.[LogId] AS ExecutionLogId,\n\terrLog.[LogId] AS ErrorLogId,\n\texeLog.[LocalExecutionId],\n\texeLog.[StartDateTime] AS ProcessingDateTime,\n\texeLog.[CallingOrchestratorName],\n\texeLog.[OrchestratorType] AS WorkerOrchestartorType,\n\texeLog.[OrchestratorName] AS WorkerOrchestrator,\n\texeLog.[PipelineName] AS WorkerPipelineName,\n\texeLog.[PipelineStatus],\n\terrLog.[ActivityRunId],\n\terrLog.[ActivityName],\n\terrLog.[ActivityType],\n\terrLog.[ErrorCode],\n\terrLog.[ErrorType],\n\terrLog.[ErrorMessage]\nFROM\n\t[procfwk].[ExecutionLog] exeLog\n\tINNER JOIN [procfwk].[ErrorLog] errLog\n\t\tON exeLog.[LocalExecutionId] = errLog.[LocalExecutionId]\n\t\t\tAND exeLog.[PipelineRunId] = errLog.[PipelineRunId]\n\tINNER JOIN [procfwk].[Stages] stgs\n\t\tON exeLog.[StageId] = stgs.[StageId]\n;"
},
{
"path": "MetadataDB/procfwkReporting/Views/CompleteExecutionLog.sql",
"content": "CREATE VIEW [procfwkReporting].[CompleteExecutionLog]\nAS\n\nSELECT\n\t[LogId],\n\t[LocalExecutionId],\n\t[StageId],\n\t[PipelineId],\n\t[CallingOrchestratorName],\n\t[ResourceGroupName],\n\t[OrchestratorType],\n\t[OrchestratorName],\n\t[PipelineName],\n\t[StartDateTime],\n\t[PipelineStatus],\n\t[EndDateTime],\n\tDATEDIFF(MINUTE, [StartDateTime], [EndDateTime]) 'RunDurationMinutes'\nFROM \n\t[procfwk].[ExecutionLog]"
},
{
"path": "MetadataDB/procfwkReporting/Views/CurrentExecutionSummary.sql",
"content": "CREATE VIEW [procfwkReporting].[CurrentExecutionSummary]\nAS\n\nSELECT \n\tISNULL([PipelineStatus], 'Not Started') AS 'PipelineStatus',\n\tCOUNT(0) AS 'RecordCount'\nFROM \n\t[procfwk].[CurrentExecution]\nGROUP BY\n\t[PipelineStatus]"
},
{
"path": "MetadataDB/procfwkReporting/Views/LastExecution.sql",
"content": "CREATE VIEW [procfwkReporting].[LastExecution]\nAS\n\nWITH maxLog AS\n\t(\n\tSELECT\n\t\tMAX([LogId]) AS 'MaxLogId'\n\tFROM\n\t\t[procfwk].[ExecutionLog]\n\t),\n\tlastExecutionId AS\n\t(\n\tSELECT\n\t\t[LocalExecutionId]\n\tFROM\n\t\t[procfwk].[ExecutionLog] el1\n\t\tINNER JOIN maxLog\n\t\t\tON maxLog.[MaxLogId] = el1.[LogId]\n\t)\nSELECT\n\tel2.[LogId],\n\tel2.[StageId],\n\tel2.[PipelineId],\n\tel2.[PipelineName],\n\tel2.[StartDateTime],\n\tel2.[PipelineStatus],\n\tel2.[EndDateTime],\n\tDATEDIFF(MINUTE, el2.[StartDateTime], el2.[EndDateTime]) AS RunDurationMinutes\nFROM \n\t[procfwk].[ExecutionLog] el2\n\tINNER JOIN lastExecutionId\n\t\tON el2.[LocalExecutionId] = lastExecutionId.[LocalExecutionId]\nWHERE\n\tel2.[EndDateTime] IS NOT NULL;"
},
{
"path": "MetadataDB/procfwkReporting/Views/LastExecutionSummary.sql",
"content": "CREATE VIEW [procfwkReporting].[LastExecutionSummary]\nAS\n\nWITH maxLog AS\n\t(\n\tSELECT\n\t\tMAX([LogId]) AS 'MaxLogId'\n\tFROM\n\t\t[procfwk].[ExecutionLog]\n\t),\n\tlastExecutionId AS\n\t(\n\tSELECT\n\t\t[LocalExecutionId]\n\tFROM\n\t\t[procfwk].[ExecutionLog] el1\n\t\tINNER JOIN maxLog\n\t\t\tON maxLog.[MaxLogId] = el1.[LogId]\n\t)\nSELECT\n\tel2.[LocalExecutionId],\n\tDATEDIFF(MINUTE, MIN(el2.[StartDateTime]), MAX(el2.[EndDateTime])) 'RunDurationMinutes'\nFROM \n\t[procfwk].[ExecutionLog] el2\n\tINNER JOIN lastExecutionId\n\t\tON el2.[LocalExecutionId] = lastExecutionId.[LocalExecutionId]\nGROUP BY\n\tel2.[LocalExecutionId]"
},
{
"path": "MetadataDB/procfwkReporting/Views/WorkerParallelismOverTime.sql",
"content": "CREATE VIEW [procfwkReporting].[WorkerParallelismOverTime]\nAS\n\nWITH numbers AS\n\t(\n\tSELECT TOP 500\n\t\tROW_NUMBER() OVER (ORDER BY s1.[object_id]) - 1 AS 'Number'\n\tFROM \n\t\tsys.all_columns AS s1\n\t\tCROSS JOIN sys.all_columns AS s2\n\t),\n\texecutionBoundaries AS\n\t(\n\tSELECT\n\t\t[LocalExecutionId],\n\t\tCAST(CONVERT(VARCHAR(16), MIN([StartDateTime]), 120) AS DATETIME) AS 'ExecutionStart',\n\t\tCAST(CONVERT(VARCHAR(16), MAX([EndDateTime]), 120) AS DATETIME) AS 'ExecutionEnd'\n\tFROM\n\t\t[procfwk].[ExecutionLog]\n\t--WHERE\n\t--\t[LocalExecutionId] = '2BB02783-2A2C-4970-9BEA-0543013BFD5E'\n\tGROUP BY\n\t\t[LocalExecutionId]\n\t),\n\twallclockRunning AS\n\t(\n\tSELECT\n\t\tCAST(DATEADD(MINUTE, n.[Number], eB.[ExecutionStart]) AS DATE) AS 'WallclockDate',\n\t\tCAST(DATEADD(MINUTE, n.[Number], eB.[ExecutionStart]) AS TIME) AS 'WallclockTime',\n\t\tel.[LocalExecutionId],\n\t\tel.[PipelineId],\n\t\tel.[PipelineName],\n\t\ts.[StageName]\n\tFROM\n\t\texecutionBoundaries eB\n\t\tCROSS JOIN numbers n\n\t\tINNER JOIN [procfwk].[ExecutionLog] eL\n\t\t\tON eB.[LocalExecutionId] = eL.[LocalExecutionId]\n\t\t\t\tAND DATEADD(MINUTE, n.[Number], eB.[ExecutionStart]) \n\t\t\t\t\tBETWEEN eL.[StartDateTime] AND eL.[EndDateTime]\n\t\tINNER JOIN [procfwk].[Stages] s\n\t\t\tON eL.[StageId] = s.[StageId]\n\t)\n\nSELECT\n\t[WallclockDate],\n\t[WallclockTime],\n\t[LocalExecutionId],\n\t[StageName],\n\tSTRING_AGG(ISNULL([PipelineName],' '),', ') As 'PipelineName',\n\tCOUNT([PipelineId]) AS 'WorkerCount'\nFROM\n\twallclockRunning\nGROUP BY\n\t[WallclockDate],\n\t[WallclockTime],\n\t[LocalExecutionId],\n\t[StageName]\nGO\n\n\n"
},
{
"path": "MetadataDB/procfwkTesting/Stored Procedures/Add20BatchesFor1000Workers.sql",
"content": "CREATE PROCEDURE [procfwkTesting].[Add20BatchesFor1000Workers]\nAS\nBEGIN\n\tSET NOCOUNT ON;\n\n\t--clear default metadata\n\tDELETE FROM [procfwk].[BatchStageLink];\n\t\n\tDELETE FROM [procfwk].[Batches];\n\n\tDELETE FROM [procfwk].[PipelineDependencies];\n\tDBCC CHECKIDENT ('[procfwk].[PipelineDependencies]', RESEED, 0);\n\n\tDELETE FROM [procfwk].[PipelineAlertLink];\n\tDBCC CHECKIDENT ('[procfwk].[PipelineAlertLink]', RESEED, 0);\n\n\tDELETE FROM [procfwk].[Recipients];\n\tDBCC CHECKIDENT ('[procfwk].[Recipients]', RESEED, 0);\n\n\tDELETE FROM [procfwk].[PipelineAuthLink];\n\tDBCC CHECKIDENT ('[procfwk].[PipelineAuthLink]', RESEED, 0);\n\n\tDELETE FROM [dbo].[ServicePrincipals];\n\tDBCC CHECKIDENT ('[dbo].[ServicePrincipals]', RESEED, 0);\n\n\tDELETE FROM [procfwk].[PipelineParameters];\n\tDBCC CHECKIDENT ('[procfwk].[PipelineParameters]', RESEED, 0);\n\n\tDELETE FROM [procfwk].[Pipelines];\n\tDBCC CHECKIDENT ('[procfwk].[Pipelines]', RESEED, 0);\n\n\t--get Orchestrator id\n\tDECLARE @OrcId INT\n\t\n\tSELECT \n\t\t@OrcId = [OrchestratorId] \n\tFROM \n\t\t[procfwk].[Orchestrators] \n\tWHERE \n\t\t[OrchestratorName] = 'WorkersFactory'\n\t\tAND [OrchestratorType] = 'ADF';\n\n\t--tweak properties\n\tUPDATE\n\t\t[procfwk].[Properties]\n\tSET\n\t\t[PropertyValue] = '[dbo].[None]'\n\tWHERE\n\t\t[PropertyName] = 'ExecutionPrecursorProc';\n\n\tUPDATE\n\t\t[procfwk].[Properties]\n\tSET\n\t\t[PropertyValue] = '30'\n\tWHERE\n\t\t[PropertyName] = 'PipelineStatusCheckDuration';\n\n\tUPDATE\n\t\t[procfwk].[Properties]\n\tSET\n\t\t[PropertyValue] = '0'\n\tWHERE\n\t\t[PropertyName] = 'UseFrameworkEmailAlerting';\n\n\tUPDATE\n\t\t[procfwk].[Properties]\n\tSET\n\t\t[PropertyValue] = '1'\n\tWHERE\n\t\t[PropertyName] = 'UseExecutionBatches';\n\n\t--insert 200 pipelines\n\t;WITH cte AS\n\t\t(\n\t\tSELECT TOP 200\n\t\t\tROW_NUMBER() OVER (ORDER BY s1.[object_id]) AS Number\n\t\tFROM \n\t\t\tsys.all_columns AS s1\n\t\t\tCROSS JOIN sys.all_columns AS s2\n\t\t)\n\tINSERT INTO [procfwk].[Pipelines]\n\t\t(\n\t\t[OrchestratorId],\n\t\t[StageId],\n\t\t[PipelineName],\n\t\t[LogicalPredecessorId],\n\t\t[Enabled]\n\t\t)\n\tSELECT\n\t\t@OrcId,\n\t\tCASE\n\t\t\tWHEN [Number] <= 50 THEN 1\n\t\t\tWHEN [Number] > 50 AND [Number] <= 100 THEN 2\n\t\t\tWHEN [Number] > 100 AND [Number] <= 150 THEN 3\n\t\t\tWHEN [Number] > 150 AND [Number] <= 200 THEN 4\n\t\tEND,\n\t\t'Wait ' + CAST([Number] AS VARCHAR),\n\t\tNULL,\n\t\t1\n\tFROM\n\t\tcte;\n\n\t--disable other execution stages if exist\n\tUPDATE [procfwk].[Stages] SET [Enabled] = 1;\n\tUPDATE [procfwk].[Stages] SET [Enabled] = 0 WHERE [StageId] > 4;\n\n\t--insert 200 pipeline parameters\n\tINSERT INTO [procfwk].[PipelineParameters]\t\n\t\t(\n\t\t[PipelineId],\n\t\t[ParameterName],\n\t\t[ParameterValue]\n\t\t)\n\tSELECT\n\t\t[PipelineId],\n\t\t'WaitTime',\n\t\tLEFT(ABS(CAST(CAST(NEWID() AS VARBINARY) AS INT)),1)\n\tFROM\n\t\t[procfwk].[Pipelines];\n\t\n\t--insert batches\n\t;WITH batchNames AS\n\t\t(\n\t\tSELECT 'One' AS [Name]\n\t\tUNION SELECT 'Two'\n\t\tUNION SELECT 'Three'\n\t\tUNION SELECT 'Four'\n\t\tUNION SELECT 'Five'\n\t\tUNION SELECT 'Six'\n\t\tUNION SELECT 'Seven'\n\t\tUNION SELECT 'Eight'\n\t\tUNION SELECT 'Nine'\n\t\tUNION SELECT 'Ten'\n\t\tUNION SELECT 'Eleven'\n\t\tUNION SELECT 'Twelve'\n\t\tUNION SELECT 'Thirteen'\n\t\tUNION SELECT 'Fourteen'\n\t\tUNION SELECT 'Fifteen'\n\t\tUNION SELECT 'Sixteen'\n\t\tUNION SELECT 'Seventeen'\n\t\tUNION SELECT 'Eighteen'\n\t\tUNION SELECT 'Nineteen'\n\t\tUNION SELECT 'Twenty'\n\t\t)\n\tINSERT INTO [procfwk].[Batches]\n\t\t(\n\t\t[BatchName],\n\t\t[BatchDescription],\n\t\t[Enabled]\n\t\t)\n\tSELECT\n\t\t[Name],\n\t\t'1000 Workers Test',\n\t\t1\n\tFROM\n\t\tbatchNames\n\t\n\t--allocation batches to stages evenly\n\t;WITH maxStages AS\n\t\t(\n\t\tSELECT \n\t\t\tMAX([StageId]) AS Id\n\t\tFROM\n\t\t\t[procfwk].[Stages]\n\t\tWHERE\n\t\t\t[Enabled] = 1\n\t\t)\n\tINSERT INTO [procfwk].[BatchStageLink]\n\tSELECT \n\t\tb.[BatchId],\n\t\tCASE\n\t\t\tWHEN (ROW_NUMBER() OVER (ORDER BY b.[BatchName] DESC) * 1) % maxStages.[Id] = 0 THEN maxStages.[Id] \n\t\t\tELSE (ROW_NUMBER() OVER (ORDER BY b.[BatchName] DESC) * 1) % maxStages.[Id] \n\t\tEND AS Stage\n\tFROM \n\t\t[procfwk].[Batches] b\n\t\tCROSS JOIN maxStages;\n\n\t/*\n\t--generate the c# for the NUnit test:\n\n\tSELECT\n\t[BatchName],\n\t'private GrandparentHelper _helperBatch' + [BatchName] + ';',\n '_helperBatch' + [BatchName] + ' = new GrandparentHelper().WithParameter(\"BatchName\", \"' + [BatchName] + '\");',\n\t'var batch' + [BatchName] + ' = _helperBatch' + [BatchName] + '.RunPipeline();',\n\t'batch' + [BatchName] + ',',\n'[Test]' + CHAR(13) +\n'public void Then' + [BatchName] + 'BatchPipelineOutcomeIsSucceeded()' + CHAR(13) +\n'{' + CHAR(13) +\n' _helperBatch' + [BatchName] + '.RunOutcome.Should().Be(\"Succeeded\");' + CHAR(13) +\n'}'\nFROM \n\t[procfwk].[Batches]\n\t*/\nEND;"
},
{
"path": "MetadataDB/procfwkTesting/Stored Procedures/Add300WorkerPipelineBatches.sql",
"content": "CREATE PROCEDURE [procfwkTesting].[Add300WorkerPipelineBatches]\nAS\nBEGIN\n\tSET NOCOUNT ON;\n\n\t--clear default metadata\n\tDELETE FROM [procfwk].[BatchStageLink];\n\tDELETE FROM [procfwk].[Batches];\n\n\t--add batch details\n\t;WITH sourceData AS\n\t\t(\n\t\tSELECT\n\t\t\t'0to300' AS BatchName, \n\t\t\t'The first 300.' AS BatchDescription,\n\t\t\t1 AS [Enabled]\n\t\tUNION SELECT\n\t\t\t'301to600',\n\t\t\t'The second 300.',\n\t\t\t1\t\n\t\t)\n\tMERGE INTO [procfwk].[Batches] AS tgt\n\tUSING \n\t\tsourceData AS src\n\t\t\tON tgt.[BatchName] = src.[BatchName]\n\tWHEN MATCHED THEN\n\t\tUPDATE\n\t\tSET\n\t\t\ttgt.[BatchDescription] = src.[BatchDescription],\n\t\t\ttgt.[Enabled] = src.[Enabled]\n\tWHEN NOT MATCHED BY TARGET THEN\n\t\tINSERT\n\t\t\t(\n\t\t\t[BatchName],\n\t\t\t[BatchDescription],\n\t\t\t[Enabled]\n\t\t\t)\n\t\tVALUES\n\t\t\t(\n\t\t\tsrc.[BatchName],\n\t\t\tsrc.[BatchDescription],\n\t\t\tsrc.[Enabled]\n\t\t\t)\n\tWHEN NOT MATCHED BY SOURCE THEN\n\t\tDELETE;\t\n\t\n\t--link batches to stages\n\tINSERT INTO [procfwk].[BatchStageLink]\n\t\t(\n\t\t[BatchId],\n\t\t[StageId]\n\t\t)\n\tSELECT\n\t\tb.[BatchId],\n\t\ts.[StageId]\n\tFROM\n\t\t[procfwk].[Batches] b\n\t\tINNER JOIN [procfwk].[Stages] s\n\t\t\tON s.[Enabled] = 1;\nEND;\n"
},
{
"path": "MetadataDB/procfwkTesting/Stored Procedures/Add300WorkerPipelines.sql",
"content": "CREATE PROCEDURE [procfwkTesting].[Add300WorkerPipelines]\nAS\nBEGIN\n\tSET NOCOUNT ON;\n\n\t--clear default metadata\n\tDELETE FROM [procfwk].[PipelineDependencies];\n\tDBCC CHECKIDENT ('[procfwk].[PipelineDependencies]', RESEED, 0);\n\n\tDELETE FROM [procfwk].[PipelineAlertLink];\n\tDBCC CHECKIDENT ('[procfwk].[PipelineAlertLink]', RESEED, 0);\n\n\tDELETE FROM [procfwk].[Recipients];\n\tDBCC CHECKIDENT ('[procfwk].[Recipients]', RESEED, 0);\n\n\tDELETE FROM [procfwk].[PipelineAuthLink];\n\tDBCC CHECKIDENT ('[procfwk].[PipelineAuthLink]', RESEED, 0);\n\n\tDELETE FROM [dbo].[ServicePrincipals];\n\tDBCC CHECKIDENT ('[dbo].[ServicePrincipals]', RESEED, 0);\n\n\tDELETE FROM [procfwk].[PipelineParameters];\n\tDBCC CHECKIDENT ('[procfwk].[PipelineParameters]', RESEED, 0);\n\n\tDELETE FROM [procfwk].[Pipelines];\n\tDBCC CHECKIDENT ('[procfwk].[Pipelines]', RESEED, 0);\n\n\t--get Orchestrator id\n\tDECLARE @OrcId INT\n\t\n\tSELECT \n\t\t@OrcId = [OrchestratorId] \n\tFROM \n\t\t[procfwk].[Orchestrators] \n\tWHERE \n\t\t[OrchestratorName] = 'WorkersFactory'\n\t\tAND [OrchestratorType] = 'ADF';\n\n\t--insert 300 pipelines\n\t;WITH cte AS\n\t\t(\n\t\tSELECT TOP 300\n\t\t\tROW_NUMBER() OVER (ORDER BY s1.[object_id]) AS Number\n\t\tFROM \n\t\t\tsys.all_columns AS s1\n\t\t\tCROSS JOIN sys.all_columns AS s2\n\t\t)\n\tINSERT INTO [procfwk].[Pipelines]\n\t\t(\n\t\t[OrchestratorId],\n\t\t[StageId],\n\t\t[PipelineName],\n\t\t[LogicalPredecessorId],\n\t\t[Enabled]\n\t\t)\n\tSELECT\n\t\t@OrcId,\n\t\tCASE\n\t\t\tWHEN [Number] <= 100 THEN 1\n\t\t\tWHEN [Number] > 100 AND [Number] <= 200 THEN 2\n\t\t\tWHEN [Number] > 200 AND [Number] <= 300 THEN 3\n\t\tEND,\n\t\t'Wait ' + CAST([Number] AS VARCHAR),\n\t\tNULL,\n\t\t1\n\tFROM\n\t\tcte;\n\n\t--disable other execution stages if exist\n\tUPDATE [procfwk].[Stages] SET [Enabled] = 0 WHERE [StageId] > 3;\n\n\t--insert 300 pipeline parameters\n\tINSERT INTO [procfwk].[PipelineParameters]\t\n\t\t(\n\t\t[PipelineId],\n\t\t[ParameterName],\n\t\t[ParameterValue]\n\t\t)\n\tSELECT\n\t\t[PipelineId],\n\t\t'WaitTime',\n\t\tLEFT(ABS(CAST(CAST(NEWID() AS VARBINARY) AS INT)),2)\n\tFROM\n\t\t[procfwk].[Pipelines];\nEND;"
},
{
"path": "MetadataDB/procfwkTesting/Stored Procedures/CleanUpMetadata.sql",
"content": "CREATE PROCEDURE [procfwkTesting].[CleanUpMetadata]\nAS\nBEGIN\n\tEXEC [procfwkHelpers].[DeleteMetadataWithIntegrity];\n\tEXEC [procfwkHelpers].[DeleteMetadataWithoutIntegrity];\nEND;"
},
{
"path": "MetadataDB/procfwkTesting/Stored Procedures/GetRunIdWhenAvailable.sql",
"content": "CREATE PROCEDURE [procfwkTesting].[GetRunIdWhenAvailable]\n\t(\n\t@PipelineName NVARCHAR(200) = NULL\n\t)\nAS\nBEGIN\n\tIF @PipelineName IS NULL\n\t\tBEGIN\n\t\t\tWHILE 1=1\n\t\t\tBEGIN\n\t\t\t\tIF EXISTS\n\t\t\t\t\t(\n\t\t\t\t\tSELECT TOP 1 \n\t\t\t\t\t\t[PipelineRunId] \n\t\t\t\t\tFROM \n\t\t\t\t\t\t[procfwk].[CurrentExecution] \n\t\t\t\t\tWHERE \n\t\t\t\t\t\t[PipelineRunId] IS NOT NULL\n\t\t\t\t\t)\n\t\t\t\tBEGIN\n\t\t\t\t\tBREAK;\n\t\t\t\tEND\n\n\t\t\t\tWAITFOR DELAY '00:00:10';\n\t\t\tEND;\n\n\t\t\tSELECT TOP 1 \n\t\t\t\tCAST([PipelineRunId] AS VARCHAR(36)) AS RunId \n\t\t\tFROM \n\t\t\t\t[procfwk].[CurrentExecution] \n\t\t\tWHERE \n\t\t\t\t[PipelineRunId] IS NOT NULL\n\t\tEND\n\tELSE IF @PipelineName IS NOT NULL\n\t\tBEGIN\n\t\t\tWHILE 1=1\n\t\t\tBEGIN\n\t\t\t\tIF EXISTS\n\t\t\t\t\t(\n\t\t\t\t\tSELECT TOP 1 \n\t\t\t\t\t\t[PipelineRunId] \n\t\t\t\t\tFROM \n\t\t\t\t\t\t[procfwk].[CurrentExecution] \n\t\t\t\t\tWHERE \n\t\t\t\t\t\t[PipelineRunId] IS NOT NULL \n\t\t\t\t\t\tAND [PipelineName] = @PipelineName\n\t\t\t\t\t)\n\t\t\t\tBEGIN\n\t\t\t\t\tBREAK;\n\t\t\t\tEND\n\n\t\t\t\tWAITFOR DELAY '00:00:10';\n\t\t\tEND;\n\n\t\t\tSELECT TOP 1 \n\t\t\t\tCAST([PipelineRunId] AS VARCHAR(36)) AS RunId \n\t\t\tFROM \n\t\t\t\t[procfwk].[CurrentExecution] \n\t\t\tWHERE \n\t\t\t\t[PipelineRunId] IS NOT NULL\n\t\t\t\tAND [PipelineName] = @PipelineName\n\t\tEND\n\tELSE\n\t\tBEGIN\n\t\t\tRAISERROR('Unknown use of testing procedure.',16,1);\n\t\tEND\nEND;"
},
{
"path": "MetadataDB/procfwkTesting/Stored Procedures/ResetMetadata.sql",
"content": "CREATE PROCEDURE [procfwkTesting].[ResetMetadata]\nAS\nBEGIN\t\n\tEXEC [procfwkHelpers].[SetDefaultProperties];\n\tEXEC [procfwkHelpers].[SetDefaultTenant];\n\tEXEC [procfwkHelpers].[SetDefaultSubscription];\n\tEXEC [procfwkHelpers].[SetDefaultOrchestrators];\n\tEXEC [procfwkHelpers].[SetDefaultBatches];\n\tEXEC [procfwkHelpers].[SetDefaultStages];\n\tEXEC [procfwkHelpers].[SetDefaultBatchStageLink];\n\tEXEC [procfwkHelpers].[SetDefaultPipelines];\n\tEXEC [procfwkHelpers].[SetDefaultPipelineParameters];\n\tEXEC [procfwkHelpers].[SetDefaultPipelineDependants];\n\tEXEC [procfwkHelpers].[SetDefaultRecipients];\n\tEXEC [procfwkHelpers].[SetDefaultAlertOutcomes];\n\tEXEC [procfwkHelpers].[SetDefaultRecipientPipelineAlerts];\nEND;"
},
{
"path": "MetadataDBTests/MetadataDBTests.sqlproj",
"content": "\n\n \n Debug\n AnyCPU\n MetadataDBTests\n 2.0\n 4.1\n {32d54b90-932d-44a3-911d-007c4b90bc55}\n Microsoft.Data.Tools.Schema.Sql.SqlAzureV12DatabaseSchemaProvider\n Database\n \n \n MetadataDBTests\n MetadataDBTests\n 1033, CI\n BySchemaAndSchemaType\n True\n v4.7.2\n CS\n Properties\n False\n True\n True\n \n \n \n bin\\Release\\\n $(MSBuildProjectName).sql\n False\n pdbonly\n true\n false\n true\n prompt\n 4\n \n \n bin\\Debug\\\n $(MSBuildProjectName).sql\n false\n true\n full\n false\n true\n true\n prompt\n 4\n \n \n 11.0\n \n True\n 11.0\n \n \n \n \n \n \n \n \n \n \n \n \n MetadataDB\n {202ebf84-a56b-4999-92a3-10f7ffe4ef25}\n True\n False\n \n \n \n \n $(DacPacRootPath)\\Extensions\\Microsoft\\SQLDB\\Extensions\\SqlServer\\AzureV12\\SqlSchemas\\master.dacpac\n False\n master\n \n \n tSQLt.dacpac\n False\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n"
},
{
"path": "MetadataDBTests/Scripts/Script.PostDeployment.sql",
"content": "/*\nPost-Deployment Script Template \n--------------------------------------------------------------------------------------\n This file contains SQL statements that will be appended to the build script. \n Use SQLCMD syntax to include a file in the post-deployment script. \n Example: :r .\\myfile.sql \n Use SQLCMD syntax to reference a variable in the post-deployment script. \n Example: :setvar TableName MyTable \n SELECT * FROM [$(TableName)] \n--------------------------------------------------------------------------------------\n*/\n\n:r ..\\..\\MetadataDB\\Scripts\\Script.PostDeployment.sql"
},
{
"path": "MetadataDBTests/Scripts/Script.PreDeployment.sql",
"content": "/*\n Pre-Deployment Script Template \n--------------------------------------------------------------------------------------\n This file contains SQL statements that will be executed before the build script. \n Use SQLCMD syntax to include a file in the pre-deployment script. \n Example: :r .\\myfile.sql \n Use SQLCMD syntax to reference a variable in the pre-deployment script. \n Example: :setvar TableName MyTable \n SELECT * FROM [$(TableName)] \n--------------------------------------------------------------------------------------\n*/\n\n:r ..\\..\\MetadataDB\\Scripts\\Script.PreDeployment.sql\n\n"
},
{
"path": "MetadataDBTests/_TestClasses/procfwk_GetPropertyValue.sql",
"content": "CREATE SCHEMA procfwk_GetPropertyValue\n AUTHORIZATION [dbo];\nGO\n\nEXEC sp_addextendedproperty @name = N'tSQLt.TestClass', @value = 1, @level0type = N'SCHEMA', @level0name = N'procfwk_GetPropertyValue'\n"
},
{
"path": "MetadataDBTests/_TestClasses/procfwk_GetPropertyValueInternal.sql",
"content": "CREATE SCHEMA procfwk_GetPropertyValueInternal\n AUTHORIZATION [dbo];\nGO\n\nEXEC sp_addextendedproperty @name = N'tSQLt.TestClass', @value = 1, @level0type = N'SCHEMA', @level0name = N'procfwk_GetPropertyValueInternal'\n"
},
{
"path": "MetadataDBTests/procfwk_GetPropertyValue/setup.sql",
"content": "CREATE PROCEDURE procfwk_GetPropertyValue.[setup]\nAS\n\nDECLARE @propertyName NVARCHAR(128) = 'TestProperty'\n\nEXEC tSQLt.FakeTable 'procfwk.CurrentProperties'\n\nINSERT INTO procfwk.CurrentProperties (\n [PropertyName]\n, [PropertyValue]\n) VALUES (\n @propertyName\n, 'TestPropertyValue'\n)\n"
},
{
"path": "MetadataDBTests/procfwk_GetPropertyValue/test WHEN property does not exist THEN error raised.sql",
"content": "CREATE PROCEDURE procfwk_GetPropertyValue.[test WHEN property does not exist THEN error raised]\nAS\n\n-- ARRANGE\nDECLARE @propertyName NVARCHAR(128) = 'OtherProperty'\n\nEXEC tSQLt.FakeTable 'procfwk.Properties'\n\nINSERT INTO procfwk.Properties (\n [PropertyName]\n, [PropertyValue]\n) VALUES (\n 'TestProperty'\n, 'TestPropertyValue'\n)\n\n-- EXPECT\nEXEC tSQLt.ExpectException @ExpectedMessagePattern = '%Invalid property name %'\n\n-- ACT\nCREATE TABLE #actual (\n PropertyValue NVARCHAR(4000)\n)\n\nINSERT INTO #actual (\n PropertyValue \n)\nEXEC procfwk.GetPropertyValue @propertyName\n"
},
{
"path": "MetadataDBTests/procfwk_GetPropertyValue/test WHEN property exists THEN property value returned.sql",
"content": "CREATE PROCEDURE procfwk_GetPropertyValue.[test WHEN property exists THEN property value returned]\nAS\n\n-- ARRANGE\nDECLARE @propertyName NVARCHAR(128) = 'TestProperty'\n\nEXEC tSQLt.FakeTable 'procfwk.Properties'\n\nINSERT INTO procfwk.Properties (\n [PropertyName]\n, [PropertyValue]\n) VALUES (\n 'TestProperty'\n, 'TestPropertyValue'\n)\n\nSELECT N'TestPropertyValue' AS PropertyValue\nINTO #expected\n\n-- ACT\nCREATE TABLE #actual (\n PropertyValue NVARCHAR(4000)\n)\n\nINSERT INTO #actual (\n PropertyValue \n)\nEXEC procfwk.GetPropertyValue @propertyName\n\n-- ASSERT\nEXEC tSQLt.AssertEqualsTable\n @Expected = '#expected'\n, @Actual = '#actual'\n"
},
{
"path": "MetadataDBTests/procfwk_GetPropertyValue/test WHEN property invalidated THEN error raised.sql",
"content": "CREATE PROCEDURE procfwk_GetPropertyValue.[test WHEN property invalidated THEN error raised]\nAS\n\n-- ARRANGE\nDECLARE @propertyName NVARCHAR(128) = 'TestProperty'\n\nEXEC tSQLt.FakeTable 'procfwk.Properties'\n\nINSERT INTO procfwk.Properties (\n [PropertyName]\n, [PropertyValue]\n, ValidTo\n) VALUES (\n 'TestProperty'\n, 'TestPropertyValue'\n, GETDATE()\n)\n\n-- EXPECT\nEXEC tSQLt.ExpectException @ExpectedMessagePattern = '% does not have a current valid version %'\n\n-- ACT\nCREATE TABLE #actual (\n PropertyValue NVARCHAR(4000)\n)\n\nINSERT INTO #actual (\n PropertyValue \n)\nEXEC procfwk.GetPropertyValue @propertyName\n"
},
{
"path": "MetadataDBTests/procfwk_GetPropertyValue/test WHEN property name is null THEN error raised.sql",
"content": "CREATE PROCEDURE procfwk_GetPropertyValue.[test WHEN property name is null THEN error raised]\nAS\n\n-- ARRANGE\nDECLARE @propertyName NVARCHAR(128) = NULL\n\nEXEC tSQLt.FakeTable 'procfwk.Properties'\n\nINSERT INTO procfwk.Properties (\n [PropertyName]\n, [PropertyValue]\n) VALUES (\n 'TestProperty'\n, 'TestPropertyValue'\n)\n\n-- EXPECT\nEXEC tSQLt.ExpectException @ExpectedMessagePattern = '%Invalid property name %'\n\n-- ACT\nCREATE TABLE #actual (\n PropertyValue NVARCHAR(4000)\n)\n\nINSERT INTO #actual (\n PropertyValue \n)\nEXEC procfwk.GetPropertyValue @propertyName\n"
},
{
"path": "MetadataDBTests/procfwk_GetPropertyValueInternal/test WHEN property does not exist THEN empty string returned.sql",
"content": "CREATE PROCEDURE procfwk_GetPropertyValueInternal.[test WHEN property name does not exist THEN empty string returned]\nAS\n\n-- ARRANGE\nDECLARE @propertyName NVARCHAR(128) = 'TestProperty'\n\nEXEC tSQLt.FakeTable 'procfwk.CurrentProperties'\n\nDECLARE @expected NVARCHAR(4000) = ''\n\n-- ACT\nDECLARE @actual NVARCHAR(4000) = procfwk.GetPropertyValueInternal(@propertyName)\n\n-- ASSERT\nEXEC tSQLt.AssertEquals \n @Expected = @expected\n, @Actual = @actual\n"
},
{
"path": "MetadataDBTests/procfwk_GetPropertyValueInternal/test WHEN property exists THEN property value returned.sql",
"content": "CREATE PROCEDURE procfwk_GetPropertyValueInternal.[test WHEN property exists THEN property value returned]\nAS\n\n-- ARRANGE\nDECLARE @propertyName NVARCHAR(128) = 'TestProperty'\n\nEXEC tSQLt.FakeTable 'procfwk.CurrentProperties'\n\nINSERT INTO procfwk.CurrentProperties (\n [PropertyName]\n, [PropertyValue]\n) VALUES (\n 'TestProperty'\n, 'TestPropertyValue'\n)\n\nDECLARE @expected NVARCHAR(4000) = 'TestPropertyValue'\n\n-- ACT\nDECLARE @actual NVARCHAR(4000) = procfwk.GetPropertyValueInternal(@propertyName)\n\n-- ASSERT\nEXEC tSQLt.AssertEquals \n @Expected = @expected\n, @Actual = @actual\n"
},
{
"path": "MetadataDBTests/procfwk_GetPropertyValueInternal/test WHEN property name is null THEN empty string returned.sql",
"content": "CREATE PROCEDURE procfwk_GetPropertyValueInternal.[test WHEN property name is null THEN empty string returned]\nAS\n\n-- ARRANGE\nDECLARE @propertyName NVARCHAR(128) = NULL\n\nEXEC tSQLt.FakeTable 'procfwk.CurrentProperties'\n\nINSERT INTO procfwk.CurrentProperties (\n [PropertyName]\n, [PropertyValue]\n) VALUES (\n 'TestProperty'\n, 'TestPropertyValue'\n)\n\nDECLARE @expected NVARCHAR(4000) = ''\n\n-- ACT\nDECLARE @actual NVARCHAR(4000) = procfwk.GetPropertyValueInternal(@propertyName)\n\n-- ASSERT\nEXEC tSQLt.AssertEquals \n @Expected = @expected\n, @Actual = @actual\n"
},
{
"path": "Notebooks/Databricks - Throw Exception.scala",
"content": "// Databricks notebook source\nimport scala.util.Try\n\ndbutils.widgets.text(\"RaiseError\", \"\",\"\")\n\n\n// COMMAND ----------\n\nval raiseError = Try(dbutils.widgets.get(\"RaiseError\").toBoolean).getOrElse(false)\n\nif(raiseError)\n{\n throw new Exception(\"The Notebook intentionally failed.\")\n}\n"
},
{
"path": "Notebooks/Metadata Guide and Handy Code Snippets.ipynb",
"content": "{\n \"metadata\": {\n \"kernelspec\": {\n \"name\": \"SQL\",\n \"display_name\": \"SQL\",\n \"language\": \"sql\"\n },\n \"language_info\": {\n \"name\": \"sql\",\n \"version\": \"\"\n }\n },\n \"nbformat_minor\": 2,\n \"nbformat\": 4,\n \"cells\": [\n {\n \"cell_type\": \"markdown\",\n \"source\": [\n \"# Welcome to the ADF.procfwk Handy User Guide Notebook\\r\\n\",\n \"\\r\\n\",\n \"This Notebook provides a series of short code snippets and narrative to help developers work with processing framework metadata.\\r\\n\",\n \"\\r\\n\",\n \"![alt text](https://mrpaulandrew.files.wordpress.com/2020/03/adfprocfwk-icon.png \\\"ADF.procfwk Icon\\\")\\r\\n\",\n \"\\r\\n\",\n \"Note: do not just run all cells in this Notebook. This may make unwanted changes to your deployed solution. The EXEC code is written with example values for you to update.\"\n ],\n \"metadata\": {\n \"azdata_cell_guid\": \"be267c69-3b61-4daf-980e-0adf85f85108\"\n }\n },\n {\n \"cell_type\": \"code\",\n \"source\": [\n \"SET NOCOUNT ON;\"\n ],\n \"metadata\": {\n \"azdata_cell_guid\": \"4b561a97-5508-4001-9bb3-4d2ed7fe3a2f\"\n },\n \"outputs\": [],\n \"execution_count\": null\n },\n {\n \"cell_type\": \"markdown\",\n \"source\": [\n \"***\\r\\n\",\n \"# Getting Current & Previous Execution Details\\r\\n\",\n \"\\r\\n\",\n \"* The [CurrentExecution] table only contains data for processing runs that our in progress or in an incomplete state.\\r\\n\",\n \"\\r\\n\",\n \"* The [ExecutionLog] table and [CompleteExecutionLog] views contains all other log details for execution runs.\"\n ],\n \"metadata\": {\n \"azdata_cell_guid\": \"1f574591-3c81-4fa4-ad6a-54a06aa75e2b\"\n }\n },\n {\n \"cell_type\": \"code\",\n \"source\": [\n \"SELECT * FROM [procfwk].[CurrentExecution];\\r\\n\",\n \"\\r\\n\",\n \"SELECT * FROM [procfwk].[CompleteExecutionLog];\\r\\n\",\n \"\\r\\n\",\n \"SELECT * FROM [procfwk].[LastExecution];\\r\\n\",\n \"\\r\\n\",\n \"SELECT TOP 100 * FROM [procfwk].[ExecutionLog] ORDER BY [LocalExecutionId], [StageId], [PipelineId], [StartDateTime];\\r\\n\",\n \"\\r\\n\",\n \"EXEC [procfwk].[GetExecutionDetails];\"\n ],\n \"metadata\": {\n \"azdata_cell_guid\": \"ad517e7c-9517-404b-9e4d-c58e4b7dbd0f\"\n },\n \"outputs\": [],\n \"execution_count\": null\n },\n {\n \"cell_type\": \"markdown\",\n \"source\": [\n \"***\\r\\n\",\n \"# Getting Error Details\\r\\n\",\n \"\\r\\n\",\n \"* The [CurrentExecution] table and the [CompleteExecutionErrorLog] view contain details of error logged for failed pipeline activities. A single pipeline can contain multiple activities that if executed in parallel may generate multiple errrors for a single pipeline Run ID.\"\n ],\n \"metadata\": {\n \"azdata_cell_guid\": \"f6362034-9ff7-4a6d-a39f-789787d445df\"\n }\n },\n {\n \"cell_type\": \"code\",\n \"source\": [\n \"SELECT * FROM [procfwk].[ErrorLog];\\r\\n\",\n \"\\r\\n\",\n \"SELECT * FROM [procfwk].[CompleteExecutionErrorLog];\\r\\n\",\n \"\\r\\n\",\n \"EXEC [procfwk].[GetExecutionDetails];\"\n ],\n \"metadata\": {\n \"azdata_cell_guid\": \"5a3106a2-c3c8-4985-923e-79410022ff90\"\n },\n \"outputs\": [],\n \"execution_count\": null\n },\n {\n \"cell_type\": \"markdown\",\n \"source\": [\n \"***\\r\\n\",\n \"# Reviewing Basic Processing Metadata\\r\\n\",\n \"The following tables have been ordered as per there level within the processing framework. A logical hierarchy exists between these tables and is enforced via database constraints connecting primary and foreign keys.\\r\\n\",\n \"* Data Factory's\\r\\n\",\n \"* Stages\\r\\n\",\n \"* Pipelines\\r\\n\",\n \"* Pipeline Parameters\"\n ],\n \"metadata\": {\n \"azdata_cell_guid\": \"88506809-f0a5-422d-bf7f-6e6e5082a8d0\"\n }\n },\n {\n \"cell_type\": \"code\",\n \"source\": [\n \"SELECT * FROM [procfwk].[DataFactorys];\\r\\n\",\n \"\\r\\n\",\n \"SELECT * FROM [procfwk].[Stages];\\r\\n\",\n \"\\r\\n\",\n \"SELECT * FROM [procfwk].[Pipelines];\\r\\n\",\n \"\\r\\n\",\n \"SELECT * FROM [procfwk].[PipelineParameters];\"\n ],\n \"metadata\": {\n \"azdata_cell_guid\": \"1a77b2ec-ffa9-4d67-91db-a1fb3acb1693\"\n },\n \"outputs\": [],\n \"execution_count\": null\n },\n {\n \"cell_type\": \"markdown\",\n \"source\": [\n \"***\\r\\n\",\n \"# Review and Check Logical Pipeline Metadata Integrity\\r\\n\",\n \"\\r\\n\",\n \"The following procedure uses the optional attribute [LogicalPredecessorId] within the table [procfwk].[Pipelines] to create a chain of dependencies between processes. This chain has zero affect of the execution of the framework, but does allow data lineage chains to be reviewed and created without impacting processing.\\r\\n\",\n \"\\r\\n\",\n \"The procedure also implements a series of case statement checks against pipeline chains to advise where metadata issues may exist. For example:\\r\\n\",\n \"\\r\\n\",\n \"* Pipeline could be moved to an earlier stage if it has no predecessors and/or isn't in an earlier stage.\\r\\n\",\n \"* Dependency issue, predeccessor pipeline is currently running in the same stage as successor.\\r\\n\",\n \"* Disabled pipeline has downstream successors.\\r\\n\",\n \"* Disabled stage has downstream successors.\\r\\n\",\n \"\\r\\n\",\n \"It is recommended that these advisory points are reviewed and the logical chain of pipelines updated to inform better framework execution.\\r\\n\",\n \"\\r\\n\",\n \"\"\n ],\n \"metadata\": {\n \"azdata_cell_guid\": \"8fae870b-a02f-4b4b-8122-bd94e2fbd39b\"\n }\n },\n {\n \"cell_type\": \"code\",\n \"source\": [\n \"EXEC [procfwk].[CheckStageAndPiplineIntegrity];\"\n ],\n \"metadata\": {\n \"azdata_cell_guid\": \"eeded0ba-002d-4c6f-b4ff-1047aa38ee62\"\n },\n \"outputs\": [],\n \"execution_count\": null\n },\n {\n \"cell_type\": \"markdown\",\n \"source\": [\n \"***\\r\\n\",\n \"# Checking and Adding Framework Properties\\r\\n\",\n \"* Use the [CurrentProperties] view to check only the latest version of property values.\\r\\n\",\n \"* Use the complete [Properties] table to see all versions of framework properties with valid from and to dates.\\r\\n\",\n \"* Use the stored procedure [AddProperty] to add a new property/value or update an existing property/value.\\r\\n\",\n \"* Use the stored procedure [GetPropertyValue] to get the latest version of a single property value.\"\n ],\n \"metadata\": {\n \"azdata_cell_guid\": \"ae575f21-e47d-4544-96c5-44e1af85c9d9\"\n }\n },\n {\n \"cell_type\": \"code\",\n \"source\": [\n \"SELECT * FROM [procfwk].[CurrentProperties];\\r\\n\",\n \"\\r\\n\",\n \"SELECT * FROM [procfwk].[Properties];\\r\\n\",\n \"\\r\\n\",\n \"EXEC [procfwk].[AddProperty] \\r\\n\",\n \"\\t@PropertyName = 'TenantId',\\r\\n\",\n \"\\t@PropertyValue = '1234-1234-1234-1234-1234',\\r\\n\",\n \"\\t@Description = 'Used to provide authentication throughout the framework execution.';\\r\\n\",\n \"\\r\\n\",\n \"EXEC [procfwk].[GetPropertyValue]\\r\\n\",\n \"\\t@PropertyName = N'TenantId';\"\n ],\n \"metadata\": {\n \"azdata_cell_guid\": \"e0e57080-aa45-49d6-a994-7fb83dfe4a08\"\n },\n \"outputs\": [],\n \"execution_count\": null\n },\n {\n \"cell_type\": \"markdown\",\n \"source\": [\n \"***\\r\\n\",\n \"# Checking, Getting, Adding and Deleting Service Principals\\r\\n\",\n \"\\r\\n\",\n \"To **check** which SPN details are being used follow the guidance in the framework for pipeline authentication:\\r\\n\",\n \"\\r\\n\",\n \"* The [ServicePrincipals] table in the [dbo] schema details which credentials are being used by the framework. These details are specific to an Azure Tenant. However, this table also needs to be joined with table [procfwk].[PipelineAuthLink] to better understand which credentials are being used by which Data Factory/Pipeline.\\r\\n\",\n \"\\r\\n\",\n \"To **get** actual unencrypted SPN details the stored procedure [procfwk].[GetServicePrincipal].\\r\\n\",\n \"\\r\\n\",\n \"Be careful not to change the Tenant ID property after Service Principals have been added.\\r\\n\",\n \"\\r\\n\",\n \"To **add** Service Principals use the stored procedure [procfwk].[AddServicePrincipal]. This will handle the creation of the links between Data Factory's and Pipelines.\\r\\n\",\n \"\\r\\n\",\n \"To **delete** Service Principals use the stored procedure [procfwk].[DeleteServicePrincipal]. This will also handle the removal of the links between Data Factory's and Pipelines.\\r\\n\",\n \"\\r\\n\",\n \"Also, be aware that as part of the new metadata integrity checks introduced in v1.3 of the framework. This will ensure all enabled pipelines have a valid SPN before the execution run starts.\"\n ],\n \"metadata\": {\n \"azdata_cell_guid\": \"86d98611-0f43-4c65-bf91-ea8d3542e705\"\n }\n },\n {\n \"cell_type\": \"code\",\n \"source\": [\n \"--Checking:\\r\\n\",\n \"SELECT * FROM [dbo].[ServicePrincipals];\\r\\n\",\n \"\\r\\n\",\n \"SELECT\\r\\n\",\n \"\\tSP.[PrincipalName],\\r\\n\",\n \"\\tDF.[ResourceGroupName],\\r\\n\",\n \"\\tDF.[DataFactoryName],\\r\\n\",\n \"\\tPP.[PipelineName]\\r\\n\",\n \"FROM\\r\\n\",\n \"\\t[procfwk].[PipelineAuthLink] AL\\r\\n\",\n \"\\tINNER JOIN [procfwk].[DataFactorys] DF\\r\\n\",\n \"\\t\\tON AL.[DataFactoryId] = DF.[DataFactoryId]\\r\\n\",\n \"\\tINNER JOIN [procfwk].[Pipelines] PP\\r\\n\",\n \"\\t\\tON AL.[PipelineId] = PP.[PipelineId]\\r\\n\",\n \"\\tINNER JOIN [dbo].[ServicePrincipals] SP\\r\\n\",\n \"\\t\\tON AL.[CredentialId] = SP.[CredentialId];\\r\\n\",\n \"\\r\\n\",\n \"--Getting:\\r\\n\",\n \"EXEC [procfwk].[GetServicePrincipal]\\r\\n\",\n \"\\t@DataFactory = 'FrameworkFactory',\\r\\n\",\n \"\\t@PipelineName = 'Wait 1';\\r\\n\",\n \"\\r\\n\",\n \"--Adding:\\r\\n\",\n \"EXEC [procfwk].[AddServicePrincipal]\\r\\n\",\n \"\\t@DataFactory = N'FrameworkFactory',\\r\\n\",\n \"\\t@PrincipalId = N'1234-1234-1234-1234-1234',\\r\\n\",\n \"\\t@PrincipalSecret = N'Passw0rd123!',\\r\\n\",\n \"\\t@PrincipalName = N'ADFFrameworkExecutor',\\r\\n\",\n \"\\t@SpecificPipelineName = N'Wait 1' --Optional parameter\\r\\n\",\n \"\\r\\n\",\n \"--Deleteing:\\r\\n\",\n \"EXEC [procfwk].[DeleteServicePrincipal]\\r\\n\",\n \"\\t@DataFactory = N'FrameworkFactory',\\r\\n\",\n \"\\t@PrincipalId = N'1234-1234-1234-1234-1234',\\r\\n\",\n \"\\t@SpecificPipelineName = N'Wait 1' --Optional parameter\"\n ],\n \"metadata\": {\n \"azdata_cell_guid\": \"ac0c404d-5a8a-4c39-96ff-66e198fffd9f\"\n },\n \"outputs\": [],\n \"execution_count\": null\n },\n {\n \"cell_type\": \"markdown\",\n \"source\": [\n \"***\\r\\n\",\n \"# Pre Execution Metadata Integrity Checks\\r\\n\",\n \"\\r\\n\",\n \"This procedure now acts as a pass/fail check before a given framework exeuction runs starts. If any of the checks fail the Data Factory pipelines will not start and the execution run will be stalled.\\r\\n\",\n \"\\r\\n\",\n \"In debug mode details of the checks that failed can be reviewed and fixed.\\r\\n\",\n \"\"\n ],\n \"metadata\": {\n \"azdata_cell_guid\": \"6698a16a-026d-4ab0-bae0-5a3b0a672649\"\n }\n },\n {\n \"cell_type\": \"code\",\n \"source\": [\n \"EXEC [procfwk].[CheckMetadataIntegrity]\\r\\n\",\n \"\\t@DebugMode = 1\"\n ],\n \"metadata\": {\n \"azdata_cell_guid\": \"f4247341-87a7-46d3-b55e-1371a48f6d81\"\n },\n \"outputs\": [],\n \"execution_count\": null\n },\n {\n \"cell_type\": \"markdown\",\n \"source\": [\n \"***\\r\\n\",\n \"# Email Alerting\\r\\n\",\n \"\\r\\n\",\n \"The following snippets provide examples on setting up alerting and adding users to the metadata.\\r\\n\",\n \"\\r\\n\",\n \"Note: SMTP details need to be provided in the Application Settings of the Azure Function App for email to be sent.\\r\\n\",\n \"\"\n ],\n \"metadata\": {\n \"azdata_cell_guid\": \"f0e2fc3b-b6d9-49be-874f-d8fd59d8c7bc\"\n }\n },\n {\n \"cell_type\": \"code\",\n \"source\": [\n \"--Is framework email alerting enabled?\\r\\n\",\n \"SELECT CASE [PropertyValue] WHEN 1 THEN 'Yes' WHEN 0 THEN 'No' ELSE 'Unknown' END AS AlertingStatus FROM [procfwk].[CurrentProperties] WHERE [PropertyName] = 'UseFrameworkEmailAlerting';\\r\\n\",\n \"\\r\\n\",\n \"--Enable framework alerting:\\r\\n\",\n \"EXEC [procfwk].[AddProperty]\\r\\n\",\n \"\\t@PropertyName = N'UseFrameworkEmailAlerting',\\r\\n\",\n \"\\t@PropertyValue = N'1',\\r\\n\",\n \"\\t@Description = N'Do you want the framework to handle pipeline email alerts via the database metadata? 1 = Yes, 0 = No.';\\r\\n\",\n \"\\r\\n\",\n \"--List current recipients:\\r\\n\",\n \"SELECT [RecipientId], [Name], [EmailAddress], [MessagePreference], [Enabled] FROM [procfwk].[Recipients];\\r\\n\",\n \"\\r\\n\",\n \"--Add a new recipient:\\r\\n\",\n \"INSERT INTO [procfwk].[Recipients] VALUES ('Test User 3','test.user3@adfprocfwk.com', 'BCC', 1);\\r\\n\",\n \"\\r\\n\",\n \"--Add an alert link for a recipient and pipeline:\\r\\n\",\n \"EXEC [procfwk].[AddRecipientPipelineAlerts]\\r\\n\",\n \"\\t@RecipientName = N'Test User 3',\\r\\n\",\n \" @PipelineName = 'Wait 1', --optional, default = all\\r\\n\",\n \"\\t@AlertForStatus = 'Success, Failed, Cancelled'; --optional, default = all\\r\\n\",\n \"\\r\\n\",\n \"--Remove recipient alerts:\\r\\n\",\n \"EXEC [procfwk].[DeleteRecipientAlerts]\\r\\n\",\n \"\\t@EmailAddress = N'test.user1@adfprocfwk.com',\\r\\n\",\n \"\\t@SoftDeleteOnly = 1; --soft delete sets Enabled = 0\\r\\n\",\n \"\\r\\n\",\n \"--Check a recipients alert subscriptions and preferences for all pipelines:\\r\\n\",\n \"SELECT\\r\\n\",\n \"\\tr.[Name] AS RecipientName,\\r\\n\",\n \"\\tp.[PipelineName],\\r\\n\",\n \"\\tr.[MessagePreference],\\r\\n\",\n \"\\tao.[PipelineOutcomeStatus],\\r\\n\",\n \"\\tp.[Enabled] AS PipelineEnabled,\\r\\n\",\n \"\\tr.[Enabled] AS RecipientEnabled,\\r\\n\",\n \"\\tal.[Enabled] AS AlertLinkEnabled\\r\\n\",\n \"FROM\\r\\n\",\n \"\\t[procfwk].[PipelineAlertLink] al\\r\\n\",\n \"\\tINNER JOIN [procfwk].[Recipients] r\\r\\n\",\n \"\\t\\tON al.[RecipientId] = r.[RecipientId]\\r\\n\",\n \"\\tINNER JOIN [procfwk].[Pipelines] p\\r\\n\",\n \"\\t\\tON al.[PipelineId] = p.[PipelineId]\\r\\n\",\n \"\\tINNER JOIN [procfwk].[AlertOutcomes] ao\\r\\n\",\n \"\\t\\tON al.[OutcomesBitValue] & ao.[BitValue] <> 0\\r\\n\",\n \"WHERE\\r\\n\",\n \"\\tr.[EmailAddress] = 'test.user3@adfprocfwk.com';\\r\\n\",\n \"\\r\\n\",\n \"--Customise the email body HTML template:\\r\\n\",\n \"DECLARE @BodyTemplate NVARCHAR(MAX) =\\r\\n\",\n \"N'
Pipeline Name: ##PipelineName###
\\r\\n\",\n \"Status: ##Status###
\\r\\n\",\n \"Execution ID: ##ExecId###
\\r\\n\",\n \"Run ID: ##RunId###
\\r\\n\",\n \"Start Date Time: ##StartDateTime###
\\r\\n\",\n \"End Date Time: ##EndDateTime###
\\r\\n\",\n \"Duration (Minutes): ##Duration###
\\r\\n\",\n \"Called by Data Factory: ##CalledByADF###
\\r\\n\",\n \"Executed by Data Factory: ##ExecutedByADF###
'\\r\\n\",\n \"\\r\\n\",\n \"EXEC [procfwk].[AddProperty]\\r\\n\",\n \"\\t@PropertyName = N'EmailAlertBodyTemplate',\\r\\n\",\n \"\\t@PropertyValue = @BodyTemplate,\\r\\n\",\n \"\\t@Description = N'Custom HTML template of execution information used as the eventual body in email alerts sent.';\"\n ],\n \"metadata\": {\n \"azdata_cell_guid\": \"63ffc489-41e7-4ff7-8fd8-4accc2429cf2\"\n },\n \"outputs\": [],\n \"execution_count\": null\n },\n {\n \"cell_type\": \"markdown\",\n \"source\": [\n \"***\\r\\n\",\n \"# Resources and Content\\r\\n\",\n \"\\r\\n\",\n \"| ![alt text](https://mrpaulandrew.files.wordpress.com/2020/03/azure-square-logo.png?w=75 \\\"Blog Icon\\\") | Blogs |[mrpaulandrew.com/ADF.procfwk](https://mrpaulandrew.com/category/azure/data-factory/adf-procfwk/)|\\r\\n\",\n \"|:----:|:----:|:----:|\\r\\n\",\n \"| ![alt text](https://mrpaulandrew.files.wordpress.com/2018/11/github-icon.png?w=75 \\\"GitHub Icon\\\") | **GitHub** |**[github.com/mrpaulandrew/ADF.procfwk](https://github.com/mrpaulandrew/ADF.procfwk)** |\\r\\n\",\n \"| ![alt text](https://mrpaulandrew.files.wordpress.com/2020/03/twitterlogo.png?w=75 \\\"Twitter Icon\\\") | **Twitter** |**[#ADFprocfwk](https://twitter.com/search?q=%23ADFprocfwk&src=hashtag_click)** |\"\n ],\n \"metadata\": {\n \"azdata_cell_guid\": \"0ffd1628-cef6-41a4-b37c-133d37a01bcf\"\n }\n }\n ]\n}"
},
{
"path": "ProcessingFramework.sln",
"content": "\nMicrosoft Visual Studio Solution File, Format Version 12.00\n# Visual Studio Version 16\nVisualStudioVersion = 16.0.29728.190\nMinimumVisualStudioVersion = 10.0.40219.1\nProject(\"{9A19103F-16F7-4668-BE54-9A1E7A4F7556}\") = \"Functions\", \"Functions\\Functions.csproj\", \"{491422B7-393B-4421-BE9E-5B70E7746CC1}\"\nEndProject\nProject(\"{00D1A9C2-B5F0-4AF3-8072-F6C62B433612}\") = \"MetadataDB\", \"MetadataDB\\MetadataDB.sqlproj\", \"{202EBF84-A56B-4999-92A3-10F7FFE4EF25}\"\nEndProject\nProject(\"{151D2E53-A2C4-4D7D-83FE-D05416EBD58E}\") = \"DeploymentTools\", \"DeploymentTools\\DeploymentTools.deployproj\", \"{3AEAD846-DBE4-45AD-97DD-37E94BE009FD}\"\nEndProject\nProject(\"{2150E333-8FDC-42A3-9474-1A3956D46DE8}\") = \"Markdown\", \"Markdown\", \"{C7297C57-3403-473B-AA2A-31012A4A9ADF}\"\n\tProjectSection(SolutionItems) = preProject\n\t\t.github\\ISSUE_TEMPLATE\\bug-found.md = .github\\ISSUE_TEMPLATE\\bug-found.md\n\t\tCODE_OF_CONDUCT.md = CODE_OF_CONDUCT.md\n\t\tCONTRIBUTING.md = CONTRIBUTING.md\n\t\t.github\\ISSUE_TEMPLATE\\feature-request.md = .github\\ISSUE_TEMPLATE\\feature-request.md\n\t\tLICENSE = LICENSE\n\t\tREADME.md = README.md\n\tEndProjectSection\nEndProject\nProject(\"{2150E333-8FDC-42A3-9474-1A3956D46DE8}\") = \"DataFactory\", \"DataFactory\", \"{E081CE7B-3DE2-4AB7-8C3A-E58A075BD6F5}\"\nEndProject\nProject(\"{2150E333-8FDC-42A3-9474-1A3956D46DE8}\") = \"dataset\", \"dataset\", \"{997E1D61-E47F-48B0-8F67-09ED4AE92E0E}\"\n\tProjectSection(SolutionItems) = preProject\n\t\tDataFactory\\dataset\\GetSetMetadata.json = DataFactory\\dataset\\GetSetMetadata.json\n\tEndProjectSection\nEndProject\nProject(\"{2150E333-8FDC-42A3-9474-1A3956D46DE8}\") = \"linkedService\", \"linkedService\", \"{6B27406C-2C36-4BA1-AFC8-CEFC960D8EE6}\"\n\tProjectSection(SolutionItems) = preProject\n\t\tDataFactory\\linkedService\\FrameworkFunctions.json = DataFactory\\linkedService\\FrameworkFunctions.json\n\t\tDataFactory\\linkedService\\Keys.json = DataFactory\\linkedService\\Keys.json\n\t\tDataFactory\\linkedService\\SupportDatabase.json = DataFactory\\linkedService\\SupportDatabase.json\n\tEndProjectSection\nEndProject\nProject(\"{2150E333-8FDC-42A3-9474-1A3956D46DE8}\") = \"pipeline\", \"pipeline\", \"{9CEBE9B6-F1E4-43A8-B84D-FF7D43D5CB20}\"\nEndProject\nProject(\"{2150E333-8FDC-42A3-9474-1A3956D46DE8}\") = \"_ProcFwk\", \"_ProcFwk\", \"{17586826-8BE9-4622-B12E-1683F2A20352}\"\n\tProjectSection(SolutionItems) = preProject\n\t\tDataFactory\\pipeline\\01-Grandparent.json = DataFactory\\pipeline\\01-Grandparent.json\n\t\tDataFactory\\pipeline\\02-Parent.json = DataFactory\\pipeline\\02-Parent.json\n\t\tDataFactory\\pipeline\\03-Child.json = DataFactory\\pipeline\\03-Child.json\n\t\tDataFactory\\pipeline\\04-Infant.json = DataFactory\\pipeline\\04-Infant.json\n\tEndProjectSection\nEndProject\nProject(\"{2150E333-8FDC-42A3-9474-1A3956D46DE8}\") = \"Workers\", \"Workers\", \"{04142829-7936-4C27-9891-FB086BEAE1AD}\"\n\tProjectSection(SolutionItems) = preProject\n\t\tDataFactory\\pipeline\\Intentional Error.json = DataFactory\\pipeline\\Intentional Error.json\n\t\tDataFactory\\pipeline\\Wait 1.json = DataFactory\\pipeline\\Wait 1.json\n\t\tDataFactory\\pipeline\\Wait 10.json = DataFactory\\pipeline\\Wait 10.json\n\t\tDataFactory\\pipeline\\Wait 2.json = DataFactory\\pipeline\\Wait 2.json\n\t\tDataFactory\\pipeline\\Wait 3.json = DataFactory\\pipeline\\Wait 3.json\n\t\tDataFactory\\pipeline\\Wait 4.json = DataFactory\\pipeline\\Wait 4.json\n\t\tDataFactory\\pipeline\\Wait 5.json = DataFactory\\pipeline\\Wait 5.json\n\t\tDataFactory\\pipeline\\Wait 6.json = DataFactory\\pipeline\\Wait 6.json\n\t\tDataFactory\\pipeline\\Wait 7.json = DataFactory\\pipeline\\Wait 7.json\n\t\tDataFactory\\pipeline\\Wait 8.json = DataFactory\\pipeline\\Wait 8.json\n\t\tDataFactory\\pipeline\\Wait 9.json = DataFactory\\pipeline\\Wait 9.json\n\tEndProjectSection\nEndProject\nProject(\"{2150E333-8FDC-42A3-9474-1A3956D46DE8}\") = \"Images\", \"Images\", \"{16E32CF7-361C-4A58-B0E1-5C8C46181ADE}\"\n\tProjectSection(SolutionItems) = preProject\n\t\tImages\\Activity Chain.png = Images\\Activity Chain.png\n\t\tImages\\Database Diagram.png = Images\\Database Diagram.png\n\t\tImages\\procfwk Designs.vsdx = Images\\procfwk Designs.vsdx\n\t\tImages\\Repo Social Media Image.png = Images\\Repo Social Media Image.png\n\tEndProjectSection\nEndProject\nProject(\"{2150E333-8FDC-42A3-9474-1A3956D46DE8}\") = \"Notebooks\", \"Notebooks\", \"{19F99FD2-9170-42DA-B4FE-A52A358B5473}\"\n\tProjectSection(SolutionItems) = preProject\n\t\tNotebooks\\Databricks - Throw Exception.scala = Notebooks\\Databricks - Throw Exception.scala\n\t\tNotebooks\\Metadata Guide and Handy Code Snippets.ipynb = Notebooks\\Metadata Guide and Handy Code Snippets.ipynb\n\tEndProjectSection\nEndProject\nProject(\"{2150E333-8FDC-42A3-9474-1A3956D46DE8}\") = \"ARM Templates\", \"ARM Templates\", \"{CC7B607B-8C84-4E07-A6C8-326923752F92}\"\nEndProject\nProject(\"{2150E333-8FDC-42A3-9474-1A3956D46DE8}\") = \"DataFactory\", \"DataFactory\", \"{C8F387D0-13CC-4AA5-892C-91D2F3403B55}\"\n\tProjectSection(SolutionItems) = preProject\n\t\tARM Templates\\Data Factory\\v1.0 Export.json = ARM Templates\\Data Factory\\v1.0 Export.json\n\t\tARM Templates\\Data Factory\\v1.1 Export.json = ARM Templates\\Data Factory\\v1.1 Export.json\n\t\tARM Templates\\Data Factory\\v1.2 Export.json = ARM Templates\\Data Factory\\v1.2 Export.json\n\t\tARM Templates\\Data Factory\\v1.3 Export.json = ARM Templates\\Data Factory\\v1.3 Export.json\n\t\tARM Templates\\Data Factory\\v1.4 Export.json = ARM Templates\\Data Factory\\v1.4 Export.json\n\t\tARM Templates\\Data Factory\\v1.5 Export.json = ARM Templates\\Data Factory\\v1.5 Export.json\n\t\tARM Templates\\Data Factory\\v1.6 Export.json = ARM Templates\\Data Factory\\v1.6 Export.json\n\t\tARM Templates\\Data Factory\\v1.7 Export.json = ARM Templates\\Data Factory\\v1.7 Export.json\n\t\tARM Templates\\Data Factory\\v1.8 Export.json = ARM Templates\\Data Factory\\v1.8 Export.json\n\t\tARM Templates\\Data Factory\\v1.8.3 Export.json = ARM Templates\\Data Factory\\v1.8.3 Export.json\n\t\tARM Templates\\Data Factory\\v1.8.5 Export.json = ARM Templates\\Data Factory\\v1.8.5 Export.json\n\t\tARM Templates\\Data Factory\\v1.8.6 Export.json = ARM Templates\\Data Factory\\v1.8.6 Export.json\n\t\tARM Templates\\Data Factory\\v1.9 Export.json = ARM Templates\\Data Factory\\v1.9 Export.json\n\t\tARM Templates\\Data Factory\\v1.9.1 Export.json = ARM Templates\\Data Factory\\v1.9.1 Export.json\n\t\tARM Templates\\Data Factory\\v1.9.2 Export.json = ARM Templates\\Data Factory\\v1.9.2 Export.json\n\t\tARM Templates\\Data Factory\\v2.0 Export.json = ARM Templates\\Data Factory\\v2.0 Export.json\n\tEndProjectSection\nEndProject\nProject(\"{2150E333-8FDC-42A3-9474-1A3956D46DE8}\") = \"trigger\", \"trigger\", \"{FC7877EF-110F-4AD7-B203-EEECE6F08A86}\"\n\tProjectSection(SolutionItems) = preProject\n\t\tDataFactory\\trigger\\FunctionalTestingTrigger.json = DataFactory\\trigger\\FunctionalTestingTrigger.json\n\tEndProjectSection\nEndProject\nProject(\"{2150E333-8FDC-42A3-9474-1A3956D46DE8}\") = \"Reporting\", \"Reporting\", \"{5A56A63A-351D-467D-A2DA-FDB3C9CBD252}\"\nEndProject\nProject(\"{2150E333-8FDC-42A3-9474-1A3956D46DE8}\") = \"PowerBI\", \"PowerBI\", \"{EBC230DF-D45D-43E4-A74E-32ADA3DDAE7A}\"\n\tProjectSection(SolutionItems) = preProject\n\t\tReporting\\PowerBI\\Executions Overview.pbit = Reporting\\PowerBI\\Executions Overview.pbit\n\t\tReporting\\PowerBI\\Executions Overview.pbix = Reporting\\PowerBI\\Executions Overview.pbix\n\tEndProjectSection\nEndProject\nProject(\"{2150E333-8FDC-42A3-9474-1A3956D46DE8}\") = \"SQL Database\", \"SQL Database\", \"{20313ECA-118E-408D-BB42-8094A7A9CD20}\"\n\tProjectSection(SolutionItems) = preProject\n\t\tARM Templates\\SQL Database\\v1.6 Export.json = ARM Templates\\SQL Database\\v1.6 Export.json\n\tEndProjectSection\nEndProject\nProject(\"{2150E333-8FDC-42A3-9474-1A3956D46DE8}\") = \"Functions App\", \"Functions App\", \"{463330CD-189F-42B4-984A-8D3C1385645B}\"\n\tProjectSection(SolutionItems) = preProject\n\t\tARM Templates\\Functions App\\v1.6 Export.json = ARM Templates\\Functions App\\v1.6 Export.json\n\tEndProjectSection\nEndProject\nProject(\"{9A19103F-16F7-4668-BE54-9A1E7A4F7556}\") = \"FactoryTesting\", \"FactoryTesting\\FactoryTesting.csproj\", \"{777BB424-9D89-49E5-BE29-E73E0A0E159A}\"\nEndProject\nProject(\"{00D1A9C2-B5F0-4AF3-8072-F6C62B433612}\") = \"MetadataDBTests\", \"MetadataDBTests\\MetadataDBTests.sqlproj\", \"{32D54B90-932D-44A3-911D-007C4B90BC55}\"\nEndProject\nProject(\"{2150E333-8FDC-42A3-9474-1A3956D46DE8}\") = \"_ProcFwkUtils\", \"_ProcFwkUtils\", \"{6809EDBD-125D-440E-8486-9CFAB9BB5B7D}\"\n\tProjectSection(SolutionItems) = preProject\n\t\tDataFactory\\pipeline\\Check For Running Pipeline.json = DataFactory\\pipeline\\Check For Running Pipeline.json\n\t\tDataFactory\\pipeline\\Email Sender.json = DataFactory\\pipeline\\Email Sender.json\n\t\tDataFactory\\pipeline\\Intentional Error.json = DataFactory\\pipeline\\Intentional Error.json\n\t\tDataFactory\\pipeline\\Throw Exception.json = DataFactory\\pipeline\\Throw Exception.json\n\tEndProjectSection\nEndProject\nProject(\"{2150E333-8FDC-42A3-9474-1A3956D46DE8}\") = \"Synapse\", \"Synapse\", \"{6B8CC8C2-C4AA-4BC4-85EC-719348FC3B07}\"\nEndProject\nProject(\"{2150E333-8FDC-42A3-9474-1A3956D46DE8}\") = \"pipeline\", \"pipeline\", \"{398CB5BE-3687-4F95-8EAA-54EE0DAC4E6C}\"\nEndProject\nProject(\"{2150E333-8FDC-42A3-9474-1A3956D46DE8}\") = \"_ProcFwk\", \"_ProcFwk\", \"{2181D7DA-8237-418F-8CD9-CD1BF9843524}\"\n\tProjectSection(SolutionItems) = preProject\n\t\tSynapse\\pipeline\\01-Grandparent.json = Synapse\\pipeline\\01-Grandparent.json\n\t\tSynapse\\pipeline\\02-Parent.json = Synapse\\pipeline\\02-Parent.json\n\t\tSynapse\\pipeline\\03-Child.json = Synapse\\pipeline\\03-Child.json\n\t\tSynapse\\pipeline\\04-Infant.json = Synapse\\pipeline\\04-Infant.json\n\tEndProjectSection\nEndProject\nProject(\"{2150E333-8FDC-42A3-9474-1A3956D46DE8}\") = \"_ProcFwkUtils\", \"_ProcFwkUtils\", \"{6E660431-66E0-4B1A-BCFC-8FCBF1C646F2}\"\n\tProjectSection(SolutionItems) = preProject\n\t\tSynapse\\pipeline\\Check For Running Pipeline.json = Synapse\\pipeline\\Check For Running Pipeline.json\n\t\tSynapse\\pipeline\\Email Sender.json = Synapse\\pipeline\\Email Sender.json\n\t\tSynapse\\pipeline\\Throw Exception.json = Synapse\\pipeline\\Throw Exception.json\n\tEndProjectSection\nEndProject\nProject(\"{2150E333-8FDC-42A3-9474-1A3956D46DE8}\") = \"Workers\", \"Workers\", \"{21058901-FBEE-4D40-83E0-21B05636417B}\"\n\tProjectSection(SolutionItems) = preProject\n\t\tSynapse\\pipeline\\Intentional Error.json = Synapse\\pipeline\\Intentional Error.json\n\t\tSynapse\\pipeline\\Wait 1.json = Synapse\\pipeline\\Wait 1.json\n\t\tSynapse\\pipeline\\Wait 10.json = Synapse\\pipeline\\Wait 10.json\n\t\tSynapse\\pipeline\\Wait 2.json = Synapse\\pipeline\\Wait 2.json\n\t\tSynapse\\pipeline\\Wait 3.json = Synapse\\pipeline\\Wait 3.json\n\t\tSynapse\\pipeline\\Wait 4.json = Synapse\\pipeline\\Wait 4.json\n\t\tSynapse\\pipeline\\Wait 5.json = Synapse\\pipeline\\Wait 5.json\n\t\tSynapse\\pipeline\\Wait 6.json = Synapse\\pipeline\\Wait 6.json\n\t\tSynapse\\pipeline\\Wait 7.json = Synapse\\pipeline\\Wait 7.json\n\t\tSynapse\\pipeline\\Wait 8.json = Synapse\\pipeline\\Wait 8.json\n\t\tSynapse\\pipeline\\Wait 9.json = Synapse\\pipeline\\Wait 9.json\n\tEndProjectSection\nEndProject\nProject(\"{2150E333-8FDC-42A3-9474-1A3956D46DE8}\") = \"dataset\", \"dataset\", \"{7F17593A-2CEF-4741-A4B7-C1CC79F12EB7}\"\n\tProjectSection(SolutionItems) = preProject\n\t\tSynapse\\dataset\\GetSetMetadata.json = Synapse\\dataset\\GetSetMetadata.json\n\tEndProjectSection\nEndProject\nProject(\"{2150E333-8FDC-42A3-9474-1A3956D46DE8}\") = \"linkedservice\", \"linkedservice\", \"{F495E5FB-B908-4F76-B8BB-6C83CC5D2337}\"\n\tProjectSection(SolutionItems) = preProject\n\t\tSynapse\\linkedService\\FrameworkFunctions.json = Synapse\\linkedService\\FrameworkFunctions.json\n\t\tSynapse\\linkedService\\Keys.json = Synapse\\linkedService\\Keys.json\n\t\tSynapse\\linkedService\\SupportDatabase.json = Synapse\\linkedService\\SupportDatabase.json\n\tEndProjectSection\nEndProject\nProject(\"{2150E333-8FDC-42A3-9474-1A3956D46DE8}\") = \"trigger\", \"trigger\", \"{29DBE5FC-D1FE-4C49-9DA2-4115C7F830B4}\"\n\tProjectSection(SolutionItems) = preProject\n\t\tSynapse\\trigger\\FunctionalTestingTrigger.json = Synapse\\trigger\\FunctionalTestingTrigger.json\n\tEndProjectSection\nEndProject\nProject(\"{2150E333-8FDC-42A3-9474-1A3956D46DE8}\") = \"Synapse\", \"Synapse\", \"{4A3176F1-B8E0-4D2B-861B-9A1655EB322E}\"\n\tProjectSection(SolutionItems) = preProject\n\t\tARM Templates\\Synapse\\v2.0 Export.json = ARM Templates\\Synapse\\v2.0 Export.json\n\tEndProjectSection\nEndProject\nGlobal\n\tGlobalSection(SolutionConfigurationPlatforms) = preSolution\n\t\tDebug|Any CPU = Debug|Any CPU\n\t\tRelease|Any CPU = Release|Any CPU\n\tEndGlobalSection\n\tGlobalSection(ProjectConfigurationPlatforms) = postSolution\n\t\t{491422B7-393B-4421-BE9E-5B70E7746CC1}.Debug|Any CPU.ActiveCfg = Debug|Any CPU\n\t\t{491422B7-393B-4421-BE9E-5B70E7746CC1}.Debug|Any CPU.Build.0 = Debug|Any CPU\n\t\t{491422B7-393B-4421-BE9E-5B70E7746CC1}.Release|Any CPU.ActiveCfg = Release|Any CPU\n\t\t{491422B7-393B-4421-BE9E-5B70E7746CC1}.Release|Any CPU.Build.0 = Release|Any CPU\n\t\t{202EBF84-A56B-4999-92A3-10F7FFE4EF25}.Debug|Any CPU.ActiveCfg = Debug|Any CPU\n\t\t{202EBF84-A56B-4999-92A3-10F7FFE4EF25}.Debug|Any CPU.Build.0 = Debug|Any CPU\n\t\t{202EBF84-A56B-4999-92A3-10F7FFE4EF25}.Debug|Any CPU.Deploy.0 = Debug|Any CPU\n\t\t{202EBF84-A56B-4999-92A3-10F7FFE4EF25}.Release|Any CPU.ActiveCfg = Release|Any CPU\n\t\t{202EBF84-A56B-4999-92A3-10F7FFE4EF25}.Release|Any CPU.Build.0 = Release|Any CPU\n\t\t{202EBF84-A56B-4999-92A3-10F7FFE4EF25}.Release|Any CPU.Deploy.0 = Release|Any CPU\n\t\t{3AEAD846-DBE4-45AD-97DD-37E94BE009FD}.Debug|Any CPU.ActiveCfg = Debug|Any CPU\n\t\t{3AEAD846-DBE4-45AD-97DD-37E94BE009FD}.Debug|Any CPU.Build.0 = Debug|Any CPU\n\t\t{3AEAD846-DBE4-45AD-97DD-37E94BE009FD}.Release|Any CPU.ActiveCfg = Release|Any CPU\n\t\t{3AEAD846-DBE4-45AD-97DD-37E94BE009FD}.Release|Any CPU.Build.0 = Release|Any CPU\n\t\t{777BB424-9D89-49E5-BE29-E73E0A0E159A}.Debug|Any CPU.ActiveCfg = Debug|Any CPU\n\t\t{777BB424-9D89-49E5-BE29-E73E0A0E159A}.Debug|Any CPU.Build.0 = Debug|Any CPU\n\t\t{777BB424-9D89-49E5-BE29-E73E0A0E159A}.Release|Any CPU.ActiveCfg = Release|Any CPU\n\t\t{777BB424-9D89-49E5-BE29-E73E0A0E159A}.Release|Any CPU.Build.0 = Release|Any CPU\n\t\t{32D54B90-932D-44A3-911D-007C4B90BC55}.Debug|Any CPU.ActiveCfg = Debug|Any CPU\n\t\t{32D54B90-932D-44A3-911D-007C4B90BC55}.Debug|Any CPU.Build.0 = Debug|Any CPU\n\t\t{32D54B90-932D-44A3-911D-007C4B90BC55}.Debug|Any CPU.Deploy.0 = Debug|Any CPU\n\t\t{32D54B90-932D-44A3-911D-007C4B90BC55}.Release|Any CPU.ActiveCfg = Release|Any CPU\n\t\t{32D54B90-932D-44A3-911D-007C4B90BC55}.Release|Any CPU.Build.0 = Release|Any CPU\n\t\t{32D54B90-932D-44A3-911D-007C4B90BC55}.Release|Any CPU.Deploy.0 = Release|Any CPU\n\tEndGlobalSection\n\tGlobalSection(SolutionProperties) = preSolution\n\t\tHideSolutionNode = FALSE\n\tEndGlobalSection\n\tGlobalSection(NestedProjects) = preSolution\n\t\t{997E1D61-E47F-48B0-8F67-09ED4AE92E0E} = {E081CE7B-3DE2-4AB7-8C3A-E58A075BD6F5}\n\t\t{6B27406C-2C36-4BA1-AFC8-CEFC960D8EE6} = {E081CE7B-3DE2-4AB7-8C3A-E58A075BD6F5}\n\t\t{9CEBE9B6-F1E4-43A8-B84D-FF7D43D5CB20} = {E081CE7B-3DE2-4AB7-8C3A-E58A075BD6F5}\n\t\t{17586826-8BE9-4622-B12E-1683F2A20352} = {9CEBE9B6-F1E4-43A8-B84D-FF7D43D5CB20}\n\t\t{04142829-7936-4C27-9891-FB086BEAE1AD} = {9CEBE9B6-F1E4-43A8-B84D-FF7D43D5CB20}\n\t\t{C8F387D0-13CC-4AA5-892C-91D2F3403B55} = {CC7B607B-8C84-4E07-A6C8-326923752F92}\n\t\t{FC7877EF-110F-4AD7-B203-EEECE6F08A86} = {E081CE7B-3DE2-4AB7-8C3A-E58A075BD6F5}\n\t\t{EBC230DF-D45D-43E4-A74E-32ADA3DDAE7A} = {5A56A63A-351D-467D-A2DA-FDB3C9CBD252}\n\t\t{20313ECA-118E-408D-BB42-8094A7A9CD20} = {CC7B607B-8C84-4E07-A6C8-326923752F92}\n\t\t{463330CD-189F-42B4-984A-8D3C1385645B} = {CC7B607B-8C84-4E07-A6C8-326923752F92}\n\t\t{6809EDBD-125D-440E-8486-9CFAB9BB5B7D} = {17586826-8BE9-4622-B12E-1683F2A20352}\n\t\t{398CB5BE-3687-4F95-8EAA-54EE0DAC4E6C} = {6B8CC8C2-C4AA-4BC4-85EC-719348FC3B07}\n\t\t{2181D7DA-8237-418F-8CD9-CD1BF9843524} = {398CB5BE-3687-4F95-8EAA-54EE0DAC4E6C}\n\t\t{6E660431-66E0-4B1A-BCFC-8FCBF1C646F2} = {2181D7DA-8237-418F-8CD9-CD1BF9843524}\n\t\t{21058901-FBEE-4D40-83E0-21B05636417B} = {398CB5BE-3687-4F95-8EAA-54EE0DAC4E6C}\n\t\t{7F17593A-2CEF-4741-A4B7-C1CC79F12EB7} = {6B8CC8C2-C4AA-4BC4-85EC-719348FC3B07}\n\t\t{F495E5FB-B908-4F76-B8BB-6C83CC5D2337} = {6B8CC8C2-C4AA-4BC4-85EC-719348FC3B07}\n\t\t{29DBE5FC-D1FE-4C49-9DA2-4115C7F830B4} = {6B8CC8C2-C4AA-4BC4-85EC-719348FC3B07}\n\t\t{4A3176F1-B8E0-4D2B-861B-9A1655EB322E} = {CC7B607B-8C84-4E07-A6C8-326923752F92}\n\tEndGlobalSection\n\tGlobalSection(ExtensibilityGlobals) = postSolution\n\t\tSolutionGuid = {170F5302-BB9C-4569-8F43-D859C7678EF0}\n\tEndGlobalSection\nEndGlobal\n"
},
{
"path": "README.md",
"content": "# Read Me - Orchestrate.[procfwk](http://procfwk.com/)\n\nFor complete documentation on this solution see [procfwk.com](http://procfwk.com/).\n\n## ProcFwk Has Become CF.Cumulus.Control\n\nSee blog: [mrpaulandrew.com](https://mrpaulandrew.com/2024/01/07/procfwk-is-getting-an-upgrade-to-cf-cumulus/)\n\nSee new product page: [cloudformations.org/cumulus](https://www.cloudformations.org/cumulus?utm_source=pa&utm_medium=github&utm_campaign=cumulus&utm_content=l2)\n\n[  ](https://mrpaulandrew.github.io/procfwk/procfwk-to-cumulus.png)\n\nProcFwk will receive no further development beyond December 2023.\n\n## Framework Capabilities\n\n * Granular metadata control.\n * Metadata integrity checking.\n * Global properties.\n * Complete pipeline dependency chains.\n * Concurrent batch executions (hourly/daily/monthly).\n * Execution restart-ability.\n * Parallel pipeline execution.\n * Full execution and error logs.\n * Operational dashboards.\n * Low cost orchestration.\n * Disconnection between framework and Worker pipelines.\n * Cross Tenant/Subscription/Data Factory control flows.\n * Pipeline parameter support.\n * Simple troubleshooting.\n * Easy deployment.\n * Email alerting.\n * Automated testing.\n * Azure Key Vault integration.\n * Is pipeline already running checks.\n\n## Complete Data Factory Activity Chain\n\n[  ](https://mrpaulandrew.github.io/procfwk/activitychain-full.png)\n\n## Issues\n\nIf you've found a bug or have a new feature request please log the details using the repository issues.\n\nGo to... [Issues](https://github.com/mrpaulandrew/procfwk/issues)\n\n## Projects\nGo to... [External Requests](https://github.com/mrpaulandrew/procfwk/projects/2)\n\nGo to... [Internal Backlog](https://github.com/mrpaulandrew/procfwk/projects/1)\n\n## Release Details\n\n| Version | Overview | Version Details & Release Notes |\n|:----:|--------------|--------|\n| 2.0 |Azure Synapse Analytics fully supported as an interchangeable orchestrator of pipelines within the procfwk.|GitHub Pages: [Orchestrators](https://mrpaulandrew.github.io/procfwk/orchestrators)[Orchestrator Types](https://mrpaulandrew.github.io/procfwk/orchestratortypes) Release Summary Video: [YouTube - procfwk Playlist](https://www.youtube.com/c/mrpaulandrew)
GitHub Issues:[procfwk #95](https://github.com/mrpaulandrew/procfwk/issues/95) |\n| 2.0-beta |Azure Synapse Analytics **Beta** support added.
Development of Azure Functions App completed using the Synapse namespace: _Azure.Analytics.Synapse.Artifacts_ with version **1.0.0-beta.1** of the NuGet package.|GitHub Issues:[procfwk #21](https://github.com/mrpaulandrew/procfwk/issues/21) |\n| 1.9.2 |Batch Executions added, plus:- Exception Pipeline
- Running Pipeline Check
- Pipeline Parameter Last Values
- Worker Pipeline Validation
|GitHub Pages: [Batch Executions](https://mrpaulandrew.github.io/procfwk/executionbatches) Release Demo Summary Video: [YouTube - procfwk Playlist](https://www.youtube.com/c/mrpaulandrew)
GitHub Issues:[procfwk #78](https://github.com/mrpaulandrew/procfwk/issues/78)
[procfwk #77](https://github.com/mrpaulandrew/procfwk/issues/77)
[procfwk #71](https://github.com/mrpaulandrew/procfwk/issues/71)
[procfwk #73](https://github.com/mrpaulandrew/procfwk/issues/73)
[procfwk #80](https://github.com/mrpaulandrew/procfwk/issues/80)
[procfwk #72](https://github.com/mrpaulandrew/procfwk/issues/72) |\n| 1.9.1 |Activity Policy Update, plus:- Secure Activity Inputs/Outputs.
- Execution Wrapper Hardening.
- New Activity Icons and Framework Factory Cosmetics.
|GitHub Issues:[procfwk #65](https://github.com/mrpaulandrew/procfwk/issues/65)
[procfwk #66](https://github.com/mrpaulandrew/procfwk/issues/66)
[procfwk #67](https://github.com/mrpaulandrew/procfwk/issues/67)
[procfwk #69](https://github.com/mrpaulandrew/procfwk/issues/69) |\n| 1.9.0 |Cross Tenant & Subscription Support added, plus:- New integration tests created.
- Infant pipeline refactoring.
- tSQLt project added.
|GitHub Issues:[procfwk #34](https://github.com/mrpaulandrew/procfwk/issues/34)
[procfwk #35](https://github.com/mrpaulandrew/procfwk/issues/35)
[procfwk #46](https://github.com/mrpaulandrew/procfwk/issues/46)
[procfwk #55](https://github.com/mrpaulandrew/procfwk/issues/55)
[procfwk #56](https://github.com/mrpaulandrew/procfwk/issues/56)
[procfwk #59](https://github.com/mrpaulandrew/procfwk/issues/59) |\n| 1.8.6 |Pipeline Expressions Refactored to Use Variables added, plus:- New integration tests created.
- Complete activity chain redrawn in Visio.
|GitHub Issues:[procfwk #51](https://github.com/mrpaulandrew/procfwk/issues/51)
[procfwk #52](https://github.com/mrpaulandrew/procfwk/issues/52) |\n| 1.8.5 |Execution Precursor added, plus:- PowerShell helper to add initial Worker metadata.
|[procfwk v1.8.5 - Execution Precursor](https://mrpaulandrew.com/2020/08/17/adf-procfwk-v1-8-5-execution-precursor/) |\n| 1.8.4 |Database Schema Reorganise and Restructuring |[procfwk v1.8.4 - Database Schema Reorganise and Restructuring](https://mrpaulandrew.com/2020/07/23/adf-procfwk-v1-8-4-database-schema-reorganise-and-restructuring/) |\n| 1.8.3 |Bug Fixes from the Community, including:- Email alerts sent to blank email addresses due to wrong flow in Child pipeline.
- Worker pipelines cancelled during an execution fail when the framework is restarted due to missing Parent pipeline clean up condition.
|GitHub Issues:[procfwk #38](https://github.com/mrpaulandrew/procfwk/issues/38)
[procfwk #37](https://github.com/mrpaulandrew/procfwk/issues/37) |\n| 1.8.2 |Optionally Store SPN Details in Azure Key Vault |[procfwk v1.8.2 - Optionally Store SPN Details in Azure Key Vault](https://mrpaulandrew.com/2020/07/22/adf-procfwk-v1-8-2-optionally-store-spn-details-in-azure-key-vault/) |\n| 1.8.1 |Automated Framework Pipeline Testing added, including tests for:- A simple grandparent run.
- All types of failure dependency handling.
- Metadata checks when pipelines and staged are disabled.
- No pipeline parameters provided.
|Blog Series:
- [Set up automated testing for Azure Data Factory](https://richardswinbank.net/adf/set_up_automated_testing_for_azure_data_factory)
- [Automate integration tests in Azure Data Factory](https://richardswinbank.net/adf/automate_integration_tests_in_azure_data_factory)
- [Isolated functional tests for Azure Data Factory](https://richardswinbank.net/adf/isolated_functional_tests_for_azure_data_factory)
- [Testing Azure Data Factory in your CI/CD pipeline](https://richardswinbank.net/adf/testing_azure_data_factory_in_your_cicd_pipeline)
- [Unit testing Azure Data Factory pipelines](https://richardswinbank.net/adf/unit_testing_azure_data_factory_pipelines)
- [Calculating Azure Data Factory test coverage](https://richardswinbank.net/adf/calculating_azure_data_factory_test_coverage)
|\n| 1.8.0 |Complete Pipeline Dependency Chains For Failure Handling added, plus:- Clean up of a previous execution run if Workers appear as running.
- New metadata integrity checks.
- Internal get property value function added.
|[procfwk v1.8 - Complete Pipeline Dependency Chains For Failure Handling](https://mrpaulandrew.com/2020/07/01/adf-procfwk-v1-8-complete-pipeline-dependency-chains-for-failure-handling/) |\n| 1.7.3 |Data Factory Deployment Updated To Use azure.datafactory.tools PowerShell Module |[SQLPlayer/azure.datafactory.tools](https://github.com/SQLPlayer/azure.datafactory.tools) |\n| 1.7.2 |Pipeline Parameter NULL Handling added, plus:- Worker pipelines with a status of 'Running' protected from a new execution start/restart.
|[procfwk v1.7.2 - NULL Pipeline Parameters Handled](https://mrpaulandrew.com/2020/06/22/adf-procfwk-v1-7-2-null-pipeline-parameters-handled/) |\n| 1.7.1 |Alerting Check Bug Fix added, plus:- Pipeline parameter value size limit removed.
|[procfwk v1.7.1 - Alerting Bug Fix And Pipeline Parameter Size Limit Removed](https://mrpaulandrew.com/2020/06/12/adf-procfwk-v1-7-1-alerting-bug-fix-and-pipeline-parameter-size-limit-removed/) |\n| 1.7.0 |Pipleline EMail Alerting added, plus:- Send email Function implemented and hardened.
- Handy Notebook updates.
- Activity failure paths improved.
- MIT license and code of conduct added.
- Error table bug fix. Error code attribute; INT to VARCHAR
|[procfwk v1.7 - Pipeline Email Alerting](https://mrpaulandrew.com/2020/06/08/adf-procfwk-v1-7-pipeline-email-alerting/) |\n| 1.6.0 |Error Details for Failed Activities Captured, plus:- Pipeline parameters used at runtime captured in execution logs.
- Emailing Function added, not yet implemented.
- Unknown Worker outcomes optionally blocks downstream stages.
- Solution housekeeping.
|[procfwk v1.6 - Error Details for Failed Activities Captured](https://mrpaulandrew.com/2020/05/19/adf-procfwk-v1-6-error-details-for-failed-activities-captured/) |\n| 1.5.0 |Power BI Dashboard for Framework Executions, plus:- Worker Parallelism View.
- Pipeline Run ID now logged.
- Logging Attributes Bug Fix.
|[procfwk v1.5 - Power BI Dashboard for Framework Executions](https://mrpaulandrew.com/2020/05/01/adf-procfwk-v1-5-power-bi-dashboard-for-framework-executions/) |\n| 1.4.0 |Enhancements for Long Running Pipelines, plus:- Pipeline check status function added.
- Function Data Factory client moved to internal class.
- SQL GETDATE() changed to GETUTCDATE().
- Glossary created, [here](https://github.com/mrpaulandrew/procfwk/blob/master/Glossary.md).
- Updated database views.
|[procfwk v1.4 - Enhancements for Long Running Pipelines](https://mrpaulandrew.com/2020/04/20/adf-procfwk-v1-4-enhancements-for-long-running-pipelines/) |\n| 1.3.0 |Metadata Integrity Checks, plus: - Logical pipeline predecessors.
- Data Factory Powershell deployment script.
- Helper Notebook.
- Database objects renames and solution tidy up.
|[procfwk v1.3 - Metadata Integrity Checks](https://mrpaulandrew.com/2020/04/07/adf-procfwk-v1-3-metadata-integrity-checks/) |\n| 1.2.0 |Execution Restartability, plus: - Data Factory annotations and descriptions.
- Database covering indexes.
- Pipeline log status changed from 'Started' to 'Preparing'.
- Pipeline log start date/time now set in child pipeline.
|[procfwk v1.2 - Execution Restartability](https://mrpaulandrew.com/2020/03/24/adf-procfwk-v1-2-execution-restartability/) |\n| 1.1.0 |Service Principal Handling via Metadata, plus: - Data Factory table.
- Properties table and view.
- Function body bug fix.
- New sample data.
|[procfwk v1.1 - Service Principal Handling via Metadata](https://mrpaulandrew.com/2020/03/17/adf-procfwk-v1-1-service-principal-handling-via-metadata/) |\n| 1.0.0 |Simple framework designed and base compontents built.
- Part 1 - Design, concepts, service coupling, caveats, problems.
- Part 2 - Database build and metadata.
- Part 3 - Data Factory build.
- Part 4 - Execution, conclusions, enhancements.
|Blog Series:
[Creating a Simple Staged Metadata Driven Processing Framework for Azure Data Factory Pipelines](https://mrpaulandrew.com/2020/02/25/creating-a-simple-staged-metadata-driven-processing-framework-for-azure-data-factory-pipelines-part-1-of-4/) |\n"
},
{
"path": "Synapse/dataflow/MDF _Order_Count.json",
"content": "{\n\t\"name\": \"MDF _Order_Count\",\n\t\"properties\": {\n\t\t\"type\": \"MappingDataFlow\",\n\t\t\"typeProperties\": {\n\t\t\t\"sources\": [],\n\t\t\t\"sinks\": [],\n\t\t\t\"transformations\": [],\n\t\t\t\"script\": \"\"\n\t\t}\n\t}\n}"
},
{
"path": "Synapse/dataset/GetSetMetadata.json",
"content": "{\n\t\"name\": \"GetSetMetadata\",\n\t\"properties\": {\n\t\t\"description\": \"Single generic dataset used to get and set all database metadata.\",\n\t\t\"linkedServiceName\": {\n\t\t\t\"referenceName\": \"SupportDatabase\",\n\t\t\t\"type\": \"LinkedServiceReference\"\n\t\t},\n\t\t\"folder\": {\n\t\t\t\"name\": \"_ProcFwk\"\n\t\t},\n\t\t\"annotations\": [\n\t\t\t\"procfwk\"\n\t\t],\n\t\t\"type\": \"AzureSqlTable\",\n\t\t\"schema\": []\n\t}\n}"
},
{
"path": "Synapse/factory/FrameworkFactory.json",
"content": "{\n\t\"name\": \"FrameworkFactory\"\n}"
},
{
"path": "Synapse/integrationRuntime/AutoResolveIntegrationRuntime.json",
"content": "{\n\t\"name\": \"AutoResolveIntegrationRuntime\",\n\t\"properties\": {\n\t\t\"type\": \"Managed\",\n\t\t\"typeProperties\": {\n\t\t\t\"computeProperties\": {\n\t\t\t\t\"location\": \"AutoResolve\",\n\t\t\t\t\"dataFlowProperties\": {\n\t\t\t\t\t\"computeType\": \"General\",\n\t\t\t\t\t\"coreCount\": 8,\n\t\t\t\t\t\"timeToLive\": 0\n\t\t\t\t}\n\t\t\t}\n\t\t}\n\t}\n}"
},
{
"path": "Synapse/linkedService/FrameworkFunctions.json",
"content": "{\n\t\"name\": \"FrameworkFunctions\",\n\t\"properties\": {\n\t\t\"annotations\": [\n\t\t\t\"procfwk\"\n\t\t],\n\t\t\"type\": \"AzureFunction\",\n\t\t\"typeProperties\": {\n\t\t\t\"functionAppUrl\": \"https://frameworksupportfunctions.azurewebsites.net\",\n\t\t\t\"functionKey\": {\n\t\t\t\t\"type\": \"AzureKeyVaultSecret\",\n\t\t\t\t\"store\": {\n\t\t\t\t\t\"referenceName\": \"Keys\",\n\t\t\t\t\t\"type\": \"LinkedServiceReference\"\n\t\t\t\t},\n\t\t\t\t\"secretName\": \"FrameworkFunctionsKey\"\n\t\t\t}\n\t\t},\n\t\t\"description\": \"Interact with the Azure Functions App used as middle ware when making requests to Worker pipelines. Authentication done at the Function App level.\"\n\t}\n}"
},
{
"path": "Synapse/linkedService/Keys.json",
"content": "{\n\t\"name\": \"Keys\",\n\t\"properties\": {\n\t\t\"annotations\": [\n\t\t\t\"procfwk\"\n\t\t],\n\t\t\"type\": \"AzureKeyVault\",\n\t\t\"typeProperties\": {\n\t\t\t\"baseUrl\": \"https://FrameworkKeys.vault.azure.net/\"\n\t\t},\n\t\t\"description\": \"Connection to Key Vault for all other ADF linked service credentials required to run the processing framework.\"\n\t}\n}"
},
{
"path": "Synapse/linkedService/SupportDatabase.json",
"content": "{\n\t\"name\": \"SupportDatabase\",\n\t\"properties\": {\n\t\t\"description\": \"Connection between ADF and processing framework metadata SQLDB.\",\n\t\t\"annotations\": [\n\t\t\t\"procfwk\"\n\t\t],\n\t\t\"type\": \"AzureSqlDatabase\",\n\t\t\"typeProperties\": {\n\t\t\t\"connectionString\": {\n\t\t\t\t\"type\": \"AzureKeyVaultSecret\",\n\t\t\t\t\"store\": {\n\t\t\t\t\t\"referenceName\": \"Keys\",\n\t\t\t\t\t\"type\": \"LinkedServiceReference\"\n\t\t\t\t},\n\t\t\t\t\"secretName\": \"FrameworkMetadataDev\"\n\t\t\t}\n\t\t}\n\t}\n}"
},
{
"path": "Synapse/linkedService/covid-tracking.json",
"content": "{\n\t\"name\": \"covid-tracking\",\n\t\"properties\": {\n\t\t\"annotations\": [],\n\t\t\"type\": \"AzureBlobStorage\",\n\t\t\"typeProperties\": {\n\t\t\t\"sasUri\": \"https://pandemicdatalake.blob.core.windows.net/public/curated/covid-19/covid_tracking/latest/covid_tracking.parquet?\\\"\\\"\"\n\t\t}\n\t}\n}"
},
{
"path": "Synapse/linkedService/procfwkforsynapse-WorkspaceDefaultSqlServer.json",
"content": "{\n\t\"name\": \"procfwkforsynapse-WorkspaceDefaultSqlServer\",\n\t\"type\": \"Microsoft.Synapse/workspaces/linkedservices\",\n\t\"properties\": {\n\t\t\"typeProperties\": {\n\t\t\t\"connectionString\": \"Data Source=tcp:procfwkforsynapse.sql.azuresynapse.net,1433;Initial Catalog=@{linkedService().DBName}\"\n\t\t},\n\t\t\"parameters\": {\n\t\t\t\"DBName\": {\n\t\t\t\t\"type\": \"String\"\n\t\t\t}\n\t\t},\n\t\t\"type\": \"AzureSqlDW\",\n\t\t\"connectVia\": {\n\t\t\t\"referenceName\": \"AutoResolveIntegrationRuntime\",\n\t\t\t\"type\": \"IntegrationRuntimeReference\"\n\t\t},\n\t\t\"annotations\": []\n\t}\n}"
},
{
"path": "Synapse/linkedService/procfwkforsynapse-WorkspaceDefaultStorage.json",
"content": "{\n\t\"name\": \"procfwkforsynapse-WorkspaceDefaultStorage\",\n\t\"type\": \"Microsoft.Synapse/workspaces/linkedservices\",\n\t\"properties\": {\n\t\t\"typeProperties\": {\n\t\t\t\"url\": \"https://frameworkdatalake01.dfs.core.windows.net\"\n\t\t},\n\t\t\"type\": \"AzureBlobFS\",\n\t\t\"connectVia\": {\n\t\t\t\"referenceName\": \"AutoResolveIntegrationRuntime\",\n\t\t\t\"type\": \"IntegrationRuntimeReference\"\n\t\t},\n\t\t\"annotations\": []\n\t}\n}"
},
{
"path": "Synapse/notebook/Getting Started with Delta Lake.json",
"content": "{\n\t\"name\": \"Getting Started with Delta Lake\",\n\t\"properties\": {\n\t\t\"nbformat\": 4,\n\t\t\"nbformat_minor\": 2,\n\t\t\"bigDataPool\": {\n\t\t\t\"referenceName\": \"SampleSpark\",\n\t\t\t\"type\": \"BigDataPoolReference\"\n\t\t},\n\t\t\"sessionProperties\": {\n\t\t\t\"driverMemory\": \"28g\",\n\t\t\t\"driverCores\": 4,\n\t\t\t\"executorMemory\": \"28g\",\n\t\t\t\"executorCores\": 4,\n\t\t\t\"numExecutors\": 2,\n\t\t\t\"conf\": {\n\t\t\t\t\"spark.dynamicAllocation.enabled\": \"false\",\n\t\t\t\t\"spark.dynamicAllocation.minExecutors\": \"2\",\n\t\t\t\t\"spark.dynamicAllocation.maxExecutors\": \"2\"\n\t\t\t}\n\t\t},\n\t\t\"metadata\": {\n\t\t\t\"saveOutput\": true,\n\t\t\t\"kernelspec\": {\n\t\t\t\t\"name\": \"synapse_sparkdotnet\",\n\t\t\t\t\"display_name\": \"Synapse SparkDotNet\"\n\t\t\t},\n\t\t\t\"language_info\": {\n\t\t\t\t\"name\": \"csharp\"\n\t\t\t},\n\t\t\t\"a365ComputeOptions\": {\n\t\t\t\t\"id\": \"/subscriptions/450eaf4d-1124-4b6d-b490-95dedc991c1e/resourceGroups/ADF.procfwk/providers/Microsoft.Synapse/workspaces/procfwkforsynapse/bigDataPools/SampleSpark\",\n\t\t\t\t\"name\": \"SampleSpark\",\n\t\t\t\t\"type\": \"Spark\",\n\t\t\t\t\"endpoint\": \"https://procfwkforsynapse.dev.azuresynapse.net/livyApi/versions/2019-11-01-preview/sparkPools/SampleSpark\",\n\t\t\t\t\"auth\": {\n\t\t\t\t\t\"type\": \"AAD\",\n\t\t\t\t\t\"authResource\": \"https://dev.azuresynapse.net\"\n\t\t\t\t},\n\t\t\t\t\"sparkVersion\": \"2.4\",\n\t\t\t\t\"nodeCount\": 3,\n\t\t\t\t\"cores\": 8,\n\t\t\t\t\"memory\": 56,\n\t\t\t\t\"automaticScaleJobs\": false\n\t\t\t},\n\t\t\t\"sessionKeepAliveTimeout\": 30\n\t\t},\n\t\t\"cells\": [\n\t\t\t{\n\t\t\t\t\"cell_type\": \"markdown\",\n\t\t\t\t\"source\": [\n\t\t\t\t\t\"# Hitchhiker's Guide to Delta Lake (.NET for Spark C#)\\n\",\n\t\t\t\t\t\"\\n\",\n\t\t\t\t\t\"This tutorial has been adapted for more clarity from its original counterpart [here](https://docs.delta.io/latest/quick-start.html). This notebook helps you quickly explore the main features of [Delta Lake](https://github.com/delta-io/delta). It provides code snippets that show how to read from and write to Delta Lake tables from interactive, batch, and streaming queries.\\n\",\n\t\t\t\t\t\"\\n\",\n\t\t\t\t\t\"Here's what we will cover:\\n\",\n\t\t\t\t\t\"* Create a table\\n\",\n\t\t\t\t\t\"* Understanding meta-data\\n\",\n\t\t\t\t\t\"* Read data\\n\",\n\t\t\t\t\t\"* Update table data\\n\",\n\t\t\t\t\t\"* Overwrite table data\\n\",\n\t\t\t\t\t\"* Conditional update without overwrite\\n\",\n\t\t\t\t\t\"* Read older versions of data using Time Travel\\n\",\n\t\t\t\t\t\"* Write a stream of data to a table\\n\",\n\t\t\t\t\t\"* Read a stream of changes from a table\"\n\t\t\t\t]\n\t\t\t},\n\t\t\t{\n\t\t\t\t\"cell_type\": \"markdown\",\n\t\t\t\t\"source\": [\n\t\t\t\t\t\"## Configuration\\n\",\n\t\t\t\t\t\"Make sure you modify this as appropriate.\"\n\t\t\t\t]\n\t\t\t},\n\t\t\t{\n\t\t\t\t\"cell_type\": \"code\",\n\t\t\t\t\"source\": [\n\t\t\t\t\t\"var sessionId = (new Random()).Next(10000000);\\n\",\n\t\t\t\t\t\"var deltaTablePath = $\\\"/delta/delta-table-{sessionId}\\\";\\n\",\n\t\t\t\t\t\"\\n\",\n\t\t\t\t\t\"deltaTablePath\"\n\t\t\t\t],\n\t\t\t\t\"execution_count\": 1\n\t\t\t},\n\t\t\t{\n\t\t\t\t\"cell_type\": \"markdown\",\n\t\t\t\t\"source\": [\n\t\t\t\t\t\"## Create a table\\n\",\n\t\t\t\t\t\"To create a Delta Lake table, write a DataFrame out in the **delta** format. You can use existing Spark SQL code and change the format from parquet, csv, json, and so on, to delta.\\n\",\n\t\t\t\t\t\"\\n\",\n\t\t\t\t\t\"These operations create a new Delta Lake table using the schema that was inferred from your DataFrame. For the full set of options available when you create a new Delta Lake table, see Create a table and Write to a table (subsequent cells in this notebook).\"\n\t\t\t\t]\n\t\t\t},\n\t\t\t{\n\t\t\t\t\"cell_type\": \"code\",\n\t\t\t\t\"source\": [\n\t\t\t\t\t\"var data = spark.Range(0,5);\\n\",\n\t\t\t\t\t\"data.Show();\\n\",\n\t\t\t\t\t\"data.Write().Format(\\\"delta\\\").Save(deltaTablePath);\"\n\t\t\t\t],\n\t\t\t\t\"execution_count\": 2\n\t\t\t},\n\t\t\t{\n\t\t\t\t\"cell_type\": \"markdown\",\n\t\t\t\t\"source\": [\n\t\t\t\t\t\"## Understanding Meta-data\\n\",\n\t\t\t\t\t\"\\n\",\n\t\t\t\t\t\"In Delta Lake, meta-data is no different from data i.e., it is stored next to the data. Therefore, an interesting side-effect here is that you can peek into meta-data using regular Spark APIs. \"\n\t\t\t\t]\n\t\t\t},\n\t\t\t{\n\t\t\t\t\"cell_type\": \"code\",\n\t\t\t\t\"source\": [\n\t\t\t\t\t\"using System.Linq;\\n\",\n\t\t\t\t\t\"spark.Read().Text($\\\"{deltaTablePath}/_delta_log/\\\").Collect().ToList().ForEach(x => Console.WriteLine(x.GetAs(\\\"value\\\")));\"\n\t\t\t\t],\n\t\t\t\t\"execution_count\": 3\n\t\t\t},\n\t\t\t{\n\t\t\t\t\"cell_type\": \"markdown\",\n\t\t\t\t\"source\": [\n\t\t\t\t\t\"## Read data\\n\",\n\t\t\t\t\t\"\\n\",\n\t\t\t\t\t\"You read data in your Delta Lake table by specifying the path to the files.\"\n\t\t\t\t]\n\t\t\t},\n\t\t\t{\n\t\t\t\t\"cell_type\": \"code\",\n\t\t\t\t\"source\": [\n\t\t\t\t\t\"var df = spark.Read().Format(\\\"delta\\\").Load(deltaTablePath);\\n\",\n\t\t\t\t\t\"df.Show()\"\n\t\t\t\t],\n\t\t\t\t\"execution_count\": 4\n\t\t\t},\n\t\t\t{\n\t\t\t\t\"cell_type\": \"markdown\",\n\t\t\t\t\"source\": [\n\t\t\t\t\t\"## Update table data\\n\",\n\t\t\t\t\t\"\\n\",\n\t\t\t\t\t\"Delta Lake supports several operations to modify tables using standard DataFrame APIs. This example runs a batch job to overwrite the data in the table.\\n\",\n\t\t\t\t\t\"\"\n\t\t\t\t]\n\t\t\t},\n\t\t\t{\n\t\t\t\t\"cell_type\": \"code\",\n\t\t\t\t\"source\": [\n\t\t\t\t\t\"var data = spark.Range(5,10);\\n\",\n\t\t\t\t\t\"data.Write().Format(\\\"delta\\\").Mode(\\\"overwrite\\\").Save(deltaTablePath);\\n\",\n\t\t\t\t\t\"df.Show();\"\n\t\t\t\t],\n\t\t\t\t\"execution_count\": 5\n\t\t\t},\n\t\t\t{\n\t\t\t\t\"cell_type\": \"markdown\",\n\t\t\t\t\"source\": [\n\t\t\t\t\t\"When you now inspect the meta-data, what you will notice is that the original data is over-written. Well, not in a true sense but appropriate entries are added to Delta's transaction log so it can provide an \\\"illusion\\\" that the original data was deleted. We can verify this by re-inspecting the meta-data. You will see several entries indicating reference removal to the original data.\"\n\t\t\t\t]\n\t\t\t},\n\t\t\t{\n\t\t\t\t\"cell_type\": \"code\",\n\t\t\t\t\"source\": [\n\t\t\t\t\t\"spark.Read().Text($\\\"{deltaTablePath}/_delta_log/\\\").Collect().ToList().ForEach(x => Console.WriteLine(x.GetAs(\\\"value\\\")));\"\n\t\t\t\t],\n\t\t\t\t\"execution_count\": 6\n\t\t\t},\n\t\t\t{\n\t\t\t\t\"cell_type\": \"markdown\",\n\t\t\t\t\"source\": [\n\t\t\t\t\t\"## Save as catalog tables\\n\",\n\t\t\t\t\t\"\\n\",\n\t\t\t\t\t\"Delta Lake can write to managed or external catalog tables.\\n\",\n\t\t\t\t\t\"\"\n\t\t\t\t]\n\t\t\t},\n\t\t\t{\n\t\t\t\t\"cell_type\": \"code\",\n\t\t\t\t\"source\": [\n\t\t\t\t\t\"// Write data to a new managed catalog table.\\n\",\n\t\t\t\t\t\"data.Write().Format(\\\"delta\\\").SaveAsTable(\\\"ManagedDeltaTable\\\");\"\n\t\t\t\t],\n\t\t\t\t\"execution_count\": 7\n\t\t\t},\n\t\t\t{\n\t\t\t\t\"cell_type\": \"code\",\n\t\t\t\t\"source\": [\n\t\t\t\t\t\"// Define an external catalog table that points to the existing Delta Lake data in storage.\\n\",\n\t\t\t\t\t\"spark.Sql($\\\"CREATE TABLE ExternalDeltaTable USING DELTA LOCATION '{deltaTablePath}'\\\");\"\n\t\t\t\t],\n\t\t\t\t\"execution_count\": 8\n\t\t\t},\n\t\t\t{\n\t\t\t\t\"cell_type\": \"code\",\n\t\t\t\t\"source\": [\n\t\t\t\t\t\"// List the 2 new tables.\\n\",\n\t\t\t\t\t\"spark.Sql(\\\"SHOW TABLES\\\").Show();\\n\",\n\t\t\t\t\t\"\\n\",\n\t\t\t\t\t\"// Explore their properties.\\n\",\n\t\t\t\t\t\"spark.Sql(\\\"DESCRIBE EXTENDED ManagedDeltaTable\\\").Show(truncate: 0);\\n\",\n\t\t\t\t\t\"spark.Sql(\\\"DESCRIBE EXTENDED ExternalDeltaTable\\\").Show(truncate: 0);\"\n\t\t\t\t],\n\t\t\t\t\"execution_count\": 9\n\t\t\t},\n\t\t\t{\n\t\t\t\t\"cell_type\": \"markdown\",\n\t\t\t\t\"source\": [\n\t\t\t\t\t\"## Conditional update without overwrite\\n\",\n\t\t\t\t\t\"\\n\",\n\t\t\t\t\t\"Delta Lake provides programmatic APIs to conditional update, delete, and merge (upsert) data into tables. For more information on these operations, see [Table Deletes, Updates, and Merges](https://docs.delta.io/latest/delta-update.html).\"\n\t\t\t\t]\n\t\t\t},\n\t\t\t{\n\t\t\t\t\"cell_type\": \"code\",\n\t\t\t\t\"source\": [\n\t\t\t\t\t\"using Microsoft.Spark.Extensions.Delta;\\n\",\n\t\t\t\t\t\"using Microsoft.Spark.Extensions.Delta.Tables;\\n\",\n\t\t\t\t\t\"using Microsoft.Spark.Sql;\\n\",\n\t\t\t\t\t\"using static Microsoft.Spark.Sql.Functions;\\n\",\n\t\t\t\t\t\"\\n\",\n\t\t\t\t\t\"var deltaTable = DeltaTable.ForPath(deltaTablePath);\"\n\t\t\t\t],\n\t\t\t\t\"execution_count\": 10\n\t\t\t},\n\t\t\t{\n\t\t\t\t\"cell_type\": \"code\",\n\t\t\t\t\"source\": [\n\t\t\t\t\t\"// Update every even value by adding 100 to it\\n\",\n\t\t\t\t\t\"deltaTable.Update(\\n\",\n\t\t\t\t\t\" condition: Expr(\\\"id % 2 == 0\\\"),\\n\",\n\t\t\t\t\t\" set: new Dictionary(){{ \\\"id\\\", Expr(\\\"id + 100\\\") }});\\n\",\n\t\t\t\t\t\"deltaTable.ToDF().Show();\"\n\t\t\t\t],\n\t\t\t\t\"execution_count\": 11\n\t\t\t},\n\t\t\t{\n\t\t\t\t\"cell_type\": \"code\",\n\t\t\t\t\"source\": [\n\t\t\t\t\t\"// Delete every even value\\n\",\n\t\t\t\t\t\"deltaTable.Delete(condition: Expr(\\\"id % 2 == 0\\\"));\\n\",\n\t\t\t\t\t\"deltaTable.ToDF().Show();\"\n\t\t\t\t],\n\t\t\t\t\"execution_count\": 12\n\t\t\t},\n\t\t\t{\n\t\t\t\t\"cell_type\": \"code\",\n\t\t\t\t\"source\": [\n\t\t\t\t\t\"// Upsert (merge) new data\\n\",\n\t\t\t\t\t\"var newData = spark.Range(20).As(\\\"newData\\\");\\n\",\n\t\t\t\t\t\"\\n\",\n\t\t\t\t\t\"deltaTable\\n\",\n\t\t\t\t\t\" .As(\\\"oldData\\\")\\n\",\n\t\t\t\t\t\" .Merge(newData, \\\"oldData.id = newData.id\\\")\\n\",\n\t\t\t\t\t\" .WhenMatched()\\n\",\n\t\t\t\t\t\" .Update(new Dictionary() {{\\\"id\\\", Lit(\\\"-1\\\")}})\\n\",\n\t\t\t\t\t\" .WhenNotMatched()\\n\",\n\t\t\t\t\t\" .Insert(new Dictionary() {{\\\"id\\\", Col(\\\"newData.id\\\")}})\\n\",\n\t\t\t\t\t\" .Execute();\\n\",\n\t\t\t\t\t\"\\n\",\n\t\t\t\t\t\"deltaTable.ToDF().Show(100);\"\n\t\t\t\t],\n\t\t\t\t\"execution_count\": 13\n\t\t\t},\n\t\t\t{\n\t\t\t\t\"cell_type\": \"markdown\",\n\t\t\t\t\"source\": [\n\t\t\t\t\t\"## History\\n\",\n\t\t\t\t\t\"Delta's most powerful feature is the ability to allow looking into history i.e., the changes that were made to the underlying Delta Table. The cell below shows how simple it is to inspect the history.\"\n\t\t\t\t]\n\t\t\t},\n\t\t\t{\n\t\t\t\t\"cell_type\": \"code\",\n\t\t\t\t\"source\": [\n\t\t\t\t\t\"deltaTable.History().Show(20, 1000, false);\"\n\t\t\t\t],\n\t\t\t\t\"execution_count\": 14\n\t\t\t},\n\t\t\t{\n\t\t\t\t\"cell_type\": \"markdown\",\n\t\t\t\t\"source\": [\n\t\t\t\t\t\"## Read older versions of data using Time Travel\\n\",\n\t\t\t\t\t\"\\n\",\n\t\t\t\t\t\"You can query previous snapshots of your Delta Lake table by using a feature called Time Travel. If you want to access the data that you overwrote, you can query a snapshot of the table before you overwrote the first set of data using the versionAsOf option.\\n\",\n\t\t\t\t\t\"\\n\",\n\t\t\t\t\t\"Once you run the cell below, you should see the first set of data, from before you overwrote it. Time Travel is an extremely powerful feature that takes advantage of the power of the Delta Lake transaction log to access data that is no longer in the table. Removing the version 0 option (or specifying version 1) would let you see the newer data again. For more information, see [Query an older snapshot of a table (time travel)](https://docs.delta.io/latest/delta-batch.html#deltatimetravel).\"\n\t\t\t\t]\n\t\t\t},\n\t\t\t{\n\t\t\t\t\"cell_type\": \"code\",\n\t\t\t\t\"source\": [\n\t\t\t\t\t\"var df = spark.Read().Format(\\\"delta\\\").Option(\\\"versionAsOf\\\", 0).Load(deltaTablePath);\\n\",\n\t\t\t\t\t\"df.Show();\"\n\t\t\t\t],\n\t\t\t\t\"execution_count\": 15\n\t\t\t},\n\t\t\t{\n\t\t\t\t\"cell_type\": \"markdown\",\n\t\t\t\t\"source\": [\n\t\t\t\t\t\"## Write a stream of data to a table\\n\",\n\t\t\t\t\t\"\\n\",\n\t\t\t\t\t\"You can also write to a Delta Lake table using Spark's Structured Streaming. The Delta Lake transaction log guarantees exactly-once processing, even when there are other streams or batch queries running concurrently against the table. By default, streams run in append mode, which adds new records to the table.\\n\",\n\t\t\t\t\t\"\\n\",\n\t\t\t\t\t\"For more information about Delta Lake integration with Structured Streaming, see [Table Streaming Reads and Writes](https://docs.delta.io/latest/delta-streaming.html).\\n\",\n\t\t\t\t\t\"\\n\",\n\t\t\t\t\t\"In the cells below, here's what we are doing:\\n\",\n\t\t\t\t\t\"\\n\",\n\t\t\t\t\t\"1. *Cell 28* Setup a simple Spark Structured Streaming job to generate a sequence and make the job write into our Delta Table\\n\",\n\t\t\t\t\t\"2. *Cell 30* Show the newly appended data\\n\",\n\t\t\t\t\t\"3. *Cell 31* Inspect history\\n\",\n\t\t\t\t\t\"4. *Cell 32* Stop the structured streaming job\\n\",\n\t\t\t\t\t\"5. *Cell 33* Inspect history <-- You'll notice appends have stopped\"\n\t\t\t\t]\n\t\t\t},\n\t\t\t{\n\t\t\t\t\"cell_type\": \"code\",\n\t\t\t\t\"source\": [\n\t\t\t\t\t\"var streamingDf = spark.ReadStream().Format(\\\"rate\\\").Load();\\n\",\n\t\t\t\t\t\"var stream = streamingDf.SelectExpr(\\\"value as id\\\").WriteStream().Format(\\\"delta\\\").Option(\\\"checkpointLocation\\\", $\\\"/tmp/checkpoint-{sessionId}\\\").Start(deltaTablePath);\"\n\t\t\t\t],\n\t\t\t\t\"execution_count\": 16\n\t\t\t},\n\t\t\t{\n\t\t\t\t\"cell_type\": \"markdown\",\n\t\t\t\t\"source\": [\n\t\t\t\t\t\"## Read a stream of changes from a table\\n\",\n\t\t\t\t\t\"\\n\",\n\t\t\t\t\t\"While the stream is writing to the Delta Lake table, you can also read from that table as streaming source. For example, you can start another streaming query that prints all the changes made to the Delta Lake table.\"\n\t\t\t\t]\n\t\t\t},\n\t\t\t{\n\t\t\t\t\"cell_type\": \"code\",\n\t\t\t\t\"source\": [\n\t\t\t\t\t\"deltaTable.ToDF().Sort(Col(\\\"id\\\").Desc()).Show(100);\"\n\t\t\t\t],\n\t\t\t\t\"execution_count\": 17\n\t\t\t},\n\t\t\t{\n\t\t\t\t\"cell_type\": \"code\",\n\t\t\t\t\"source\": [\n\t\t\t\t\t\"deltaTable.History().Drop(\\\"userId\\\", \\\"userName\\\", \\\"job\\\", \\\"notebook\\\", \\\"clusterId\\\", \\\"isolationLevel\\\", \\\"isBlindAppend\\\").Show(20, 1000, false);\"\n\t\t\t\t],\n\t\t\t\t\"execution_count\": 18\n\t\t\t},\n\t\t\t{\n\t\t\t\t\"cell_type\": \"code\",\n\t\t\t\t\"source\": [\n\t\t\t\t\t\"stream.Stop();\"\n\t\t\t\t],\n\t\t\t\t\"execution_count\": 19\n\t\t\t},\n\t\t\t{\n\t\t\t\t\"cell_type\": \"code\",\n\t\t\t\t\"source\": [\n\t\t\t\t\t\"deltaTable.History().Drop(\\\"userId\\\", \\\"userName\\\", \\\"job\\\", \\\"notebook\\\", \\\"clusterId\\\", \\\"isolationLevel\\\", \\\"isBlindAppend\\\").Show(100, 1000, false);\"\n\t\t\t\t],\n\t\t\t\t\"execution_count\": 20\n\t\t\t},\n\t\t\t{\n\t\t\t\t\"cell_type\": \"markdown\",\n\t\t\t\t\"source\": [\n\t\t\t\t\t\"## Compaction\\n\",\n\t\t\t\t\t\"\\n\",\n\t\t\t\t\t\"If a Delta Table is growing too large, you can compact it by repartitioning into a smaller number of files.\\n\",\n\t\t\t\t\t\"\\n\",\n\t\t\t\t\t\"The option `dataChange = false` is an optimization that tells Delta Lake to do the repartition without marking the underlying data as \\\"modified\\\". This ensures that any other concurrent operations (such as streaming reads/writes) aren't negatively impacted.\\n\",\n\t\t\t\t\t\"\"\n\t\t\t\t]\n\t\t\t},\n\t\t\t{\n\t\t\t\t\"cell_type\": \"code\",\n\t\t\t\t\"source\": [\n\t\t\t\t\t\"int partitionCount = 2;\\n\",\n\t\t\t\t\t\"\\n\",\n\t\t\t\t\t\"spark.Read()\\n\",\n\t\t\t\t\t\" .Format(\\\"delta\\\")\\n\",\n\t\t\t\t\t\" .Load(deltaTablePath)\\n\",\n\t\t\t\t\t\" .Repartition(partitionCount)\\n\",\n\t\t\t\t\t\" .Write()\\n\",\n\t\t\t\t\t\" .Option(\\\"dataChange\\\", \\\"false\\\")\\n\",\n\t\t\t\t\t\" .Format(\\\"delta\\\")\\n\",\n\t\t\t\t\t\" .Mode(\\\"overwrite\\\")\\n\",\n\t\t\t\t\t\" .Save(deltaTablePath);\"\n\t\t\t\t],\n\t\t\t\t\"execution_count\": 21\n\t\t\t},\n\t\t\t{\n\t\t\t\t\"cell_type\": \"markdown\",\n\t\t\t\t\"source\": [\n\t\t\t\t\t\"## Convert Parquet to Delta\\n\",\n\t\t\t\t\t\"You can do an in-place conversion from the Parquet format to Delta.\"\n\t\t\t\t]\n\t\t\t},\n\t\t\t{\n\t\t\t\t\"cell_type\": \"code\",\n\t\t\t\t\"source\": [\n\t\t\t\t\t\"var parquetPath = $\\\"/parquet/parquet-table-{sessionId}\\\";\\n\",\n\t\t\t\t\t\"\\n\",\n\t\t\t\t\t\"var data = spark.Range(0,5);\\n\",\n\t\t\t\t\t\"data.Write().Parquet(parquetPath);\\n\",\n\t\t\t\t\t\"\\n\",\n\t\t\t\t\t\"// Confirm that the data isn't in the Delta format\\n\",\n\t\t\t\t\t\"DeltaTable.IsDeltaTable(parquetPath)\"\n\t\t\t\t],\n\t\t\t\t\"execution_count\": 22\n\t\t\t},\n\t\t\t{\n\t\t\t\t\"cell_type\": \"code\",\n\t\t\t\t\"source\": [\n\t\t\t\t\t\"DeltaTable.ConvertToDelta(spark, $\\\"parquet.`{parquetPath}`\\\");\\n\",\n\t\t\t\t\t\"\\n\",\n\t\t\t\t\t\"//Confirm that the converted data is now in the Delta format\\n\",\n\t\t\t\t\t\"DeltaTable.IsDeltaTable(parquetPath)\"\n\t\t\t\t],\n\t\t\t\t\"execution_count\": 23\n\t\t\t},\n\t\t\t{\n\t\t\t\t\"cell_type\": \"markdown\",\n\t\t\t\t\"source\": [\n\t\t\t\t\t\"## SQL Support\\n\",\n\t\t\t\t\t\"Delta supports table utility commands through SQL. You can use SQL to:\\n\",\n\t\t\t\t\t\"* Get a DeltaTable's history\\n\",\n\t\t\t\t\t\"* Vacuum a DeltaTable\\n\",\n\t\t\t\t\t\"* Convert a Parquet file to Delta\\n\",\n\t\t\t\t\t\"\"\n\t\t\t\t]\n\t\t\t},\n\t\t\t{\n\t\t\t\t\"cell_type\": \"code\",\n\t\t\t\t\"source\": [\n\t\t\t\t\t\"spark.Sql($\\\"DESCRIBE HISTORY delta.`{deltaTablePath}`\\\").Show();\"\n\t\t\t\t],\n\t\t\t\t\"execution_count\": 24\n\t\t\t},\n\t\t\t{\n\t\t\t\t\"cell_type\": \"code\",\n\t\t\t\t\"source\": [\n\t\t\t\t\t\"spark.Sql($\\\"VACUUM delta.`{deltaTablePath}`\\\").Show();\"\n\t\t\t\t],\n\t\t\t\t\"execution_count\": 25\n\t\t\t},\n\t\t\t{\n\t\t\t\t\"cell_type\": \"code\",\n\t\t\t\t\"source\": [\n\t\t\t\t\t\"var parquetId = (new Random()).Next(10000000);\\n\",\n\t\t\t\t\t\"var parquetPath = $\\\"/parquet/parquet-table-{sessionId}-{parquetId}\\\";\\n\",\n\t\t\t\t\t\"\\n\",\n\t\t\t\t\t\"var data = spark.Range(0,5);\\n\",\n\t\t\t\t\t\"data.Write().Parquet(parquetPath);\\n\",\n\t\t\t\t\t\"\\n\",\n\t\t\t\t\t\"// Confirm that the data isn't in the Delta format\\n\",\n\t\t\t\t\t\"DeltaTable.IsDeltaTable(parquetPath);\\n\",\n\t\t\t\t\t\"\\n\",\n\t\t\t\t\t\"// Use SQL to convert the parquet table to Delta\\n\",\n\t\t\t\t\t\"spark.Sql($\\\"CONVERT TO DELTA parquet.`{parquetPath}`\\\");\\n\",\n\t\t\t\t\t\"\\n\",\n\t\t\t\t\t\"DeltaTable.IsDeltaTable(parquetPath);\"\n\t\t\t\t],\n\t\t\t\t\"execution_count\": 26\n\t\t\t}\n\t\t]\n\t}\n}"
},
{
"path": "Synapse/pipeline/01-Grandparent.json",
"content": "{\n\t\"name\": \"01-Grandparent\",\n\t\"properties\": {\n\t\t\"description\": \"procfwk grandparent pipeline used optionally to bootstrap any wider processes in your Data Factory that then calls the processing framework.\",\n\t\t\"activities\": [\n\t\t\t{\n\t\t\t\t\"name\": \"procfwk\",\n\t\t\t\t\"description\": \"Call procfwk\",\n\t\t\t\t\"type\": \"ExecutePipeline\",\n\t\t\t\t\"dependsOn\": [],\n\t\t\t\t\"userProperties\": [],\n\t\t\t\t\"typeProperties\": {\n\t\t\t\t\t\"pipeline\": {\n\t\t\t\t\t\t\"referenceName\": \"02-Parent\",\n\t\t\t\t\t\t\"type\": \"PipelineReference\"\n\t\t\t\t\t},\n\t\t\t\t\t\"waitOnCompletion\": true,\n\t\t\t\t\t\"parameters\": {\n\t\t\t\t\t\t\"BatchName\": {\n\t\t\t\t\t\t\t\"value\": \"@pipeline().parameters.BatchName\",\n\t\t\t\t\t\t\t\"type\": \"Expression\"\n\t\t\t\t\t\t}\n\t\t\t\t\t}\n\t\t\t\t}\n\t\t\t}\n\t\t],\n\t\t\"parameters\": {\n\t\t\t\"BatchName\": {\n\t\t\t\t\"type\": \"string\",\n\t\t\t\t\"defaultValue\": \"NotUsed\"\n\t\t\t}\n\t\t},\n\t\t\"folder\": {\n\t\t\t\"name\": \"_ProcFwk\"\n\t\t},\n\t\t\"annotations\": [\n\t\t\t\"procfwk\",\n\t\t\t\"Grandparent\"\n\t\t]\n\t}\n}"
},
{
"path": "Synapse/pipeline/02-Parent.json",
"content": "{\n\t\"name\": \"02-Parent\",\n\t\"properties\": {\n\t\t\"description\": \"ADF.procfwk parent pipeline used to bootstrap the orchestration framework in perform the first level ForEach calls in sequence for the metadata stages.\",\n\t\t\"activities\": [\n\t\t\t{\n\t\t\t\t\"name\": \"Get Stages\",\n\t\t\t\t\"description\": \"Returns a distinct list of execution stages within the framework metadata.\",\n\t\t\t\t\"type\": \"Lookup\",\n\t\t\t\t\"dependsOn\": [\n\t\t\t\t\t{\n\t\t\t\t\t\t\"activity\": \"Set Execution Id\",\n\t\t\t\t\t\t\"dependencyConditions\": [\n\t\t\t\t\t\t\t\"Succeeded\"\n\t\t\t\t\t\t]\n\t\t\t\t\t}\n\t\t\t\t],\n\t\t\t\t\"policy\": {\n\t\t\t\t\t\"timeout\": \"0.00:10:00\",\n\t\t\t\t\t\"retry\": 0,\n\t\t\t\t\t\"retryIntervalInSeconds\": 30,\n\t\t\t\t\t\"secureOutput\": false,\n\t\t\t\t\t\"secureInput\": false\n\t\t\t\t},\n\t\t\t\t\"userProperties\": [],\n\t\t\t\t\"typeProperties\": {\n\t\t\t\t\t\"source\": {\n\t\t\t\t\t\t\"type\": \"AzureSqlSource\",\n\t\t\t\t\t\t\"sqlReaderStoredProcedureName\": \"[procfwk].[GetStages]\",\n\t\t\t\t\t\t\"storedProcedureParameters\": {\n\t\t\t\t\t\t\t\"ExecutionId\": {\n\t\t\t\t\t\t\t\t\"type\": \"Guid\",\n\t\t\t\t\t\t\t\t\"value\": {\n\t\t\t\t\t\t\t\t\t\"value\": \"@variables('ExecutionId')\",\n\t\t\t\t\t\t\t\t\t\"type\": \"Expression\"\n\t\t\t\t\t\t\t\t}\n\t\t\t\t\t\t\t}\n\t\t\t\t\t\t},\n\t\t\t\t\t\t\"queryTimeout\": \"02:00:00\",\n\t\t\t\t\t\t\"partitionOption\": \"None\"\n\t\t\t\t\t},\n\t\t\t\t\t\"dataset\": {\n\t\t\t\t\t\t\"referenceName\": \"GetSetMetadata\",\n\t\t\t\t\t\t\"type\": \"DatasetReference\"\n\t\t\t\t\t},\n\t\t\t\t\t\"firstRowOnly\": false\n\t\t\t\t}\n\t\t\t},\n\t\t\t{\n\t\t\t\t\"name\": \"Execute Stages\",\n\t\t\t\t\"description\": \"Top level ForEach to sequentially call all processing stages within the framework metadata. Items for iteration passed from the Get Stages lookup activity.\",\n\t\t\t\t\"type\": \"ForEach\",\n\t\t\t\t\"dependsOn\": [\n\t\t\t\t\t{\n\t\t\t\t\t\t\"activity\": \"Get Stages\",\n\t\t\t\t\t\t\"dependencyConditions\": [\n\t\t\t\t\t\t\t\"Succeeded\"\n\t\t\t\t\t\t]\n\t\t\t\t\t}\n\t\t\t\t],\n\t\t\t\t\"userProperties\": [],\n\t\t\t\t\"typeProperties\": {\n\t\t\t\t\t\"items\": {\n\t\t\t\t\t\t\"value\": \"@activity('Get Stages').output.value\",\n\t\t\t\t\t\t\"type\": \"Expression\"\n\t\t\t\t\t},\n\t\t\t\t\t\"isSequential\": true,\n\t\t\t\t\t\"activities\": [\n\t\t\t\t\t\t{\n\t\t\t\t\t\t\t\"name\": \"Stage Executor\",\n\t\t\t\t\t\t\t\"description\": \"Call to the framework generic child pipeline for a given execution stage.\",\n\t\t\t\t\t\t\t\"type\": \"ExecutePipeline\",\n\t\t\t\t\t\t\t\"dependsOn\": [\n\t\t\t\t\t\t\t\t{\n\t\t\t\t\t\t\t\t\t\"activity\": \"Log Stage Preparing\",\n\t\t\t\t\t\t\t\t\t\"dependencyConditions\": [\n\t\t\t\t\t\t\t\t\t\t\"Succeeded\"\n\t\t\t\t\t\t\t\t\t]\n\t\t\t\t\t\t\t\t}\n\t\t\t\t\t\t\t],\n\t\t\t\t\t\t\t\"userProperties\": [],\n\t\t\t\t\t\t\t\"typeProperties\": {\n\t\t\t\t\t\t\t\t\"pipeline\": {\n\t\t\t\t\t\t\t\t\t\"referenceName\": \"03-Child\",\n\t\t\t\t\t\t\t\t\t\"type\": \"PipelineReference\"\n\t\t\t\t\t\t\t\t},\n\t\t\t\t\t\t\t\t\"waitOnCompletion\": true,\n\t\t\t\t\t\t\t\t\"parameters\": {\n\t\t\t\t\t\t\t\t\t\"StageId\": {\n\t\t\t\t\t\t\t\t\t\t\"value\": \"@item().StageId\",\n\t\t\t\t\t\t\t\t\t\t\"type\": \"Expression\"\n\t\t\t\t\t\t\t\t\t},\n\t\t\t\t\t\t\t\t\t\"ExecutionId\": {\n\t\t\t\t\t\t\t\t\t\t\"value\": \"@variables('ExecutionId')\",\n\t\t\t\t\t\t\t\t\t\t\"type\": \"Expression\"\n\t\t\t\t\t\t\t\t\t}\n\t\t\t\t\t\t\t\t}\n\t\t\t\t\t\t\t}\n\t\t\t\t\t\t},\n\t\t\t\t\t\t{\n\t\t\t\t\t\t\t\"name\": \"Log Stage Preparing\",\n\t\t\t\t\t\t\t\"description\": \"Update the current execution table flagging all pipelines within the stage as preparing.\",\n\t\t\t\t\t\t\t\"type\": \"SqlServerStoredProcedure\",\n\t\t\t\t\t\t\t\"dependsOn\": [\n\t\t\t\t\t\t\t\t{\n\t\t\t\t\t\t\t\t\t\"activity\": \"Check and Update Blockers\",\n\t\t\t\t\t\t\t\t\t\"dependencyConditions\": [\n\t\t\t\t\t\t\t\t\t\t\"Succeeded\"\n\t\t\t\t\t\t\t\t\t]\n\t\t\t\t\t\t\t\t}\n\t\t\t\t\t\t\t],\n\t\t\t\t\t\t\t\"policy\": {\n\t\t\t\t\t\t\t\t\"timeout\": \"0.00:10:00\",\n\t\t\t\t\t\t\t\t\"retry\": 0,\n\t\t\t\t\t\t\t\t\"retryIntervalInSeconds\": 30,\n\t\t\t\t\t\t\t\t\"secureOutput\": false,\n\t\t\t\t\t\t\t\t\"secureInput\": false\n\t\t\t\t\t\t\t},\n\t\t\t\t\t\t\t\"userProperties\": [],\n\t\t\t\t\t\t\t\"typeProperties\": {\n\t\t\t\t\t\t\t\t\"storedProcedureName\": \"[procfwk].[SetLogStagePreparing]\",\n\t\t\t\t\t\t\t\t\"storedProcedureParameters\": {\n\t\t\t\t\t\t\t\t\t\"ExecutionId\": {\n\t\t\t\t\t\t\t\t\t\t\"value\": {\n\t\t\t\t\t\t\t\t\t\t\t\"value\": \"@variables('ExecutionId')\",\n\t\t\t\t\t\t\t\t\t\t\t\"type\": \"Expression\"\n\t\t\t\t\t\t\t\t\t\t},\n\t\t\t\t\t\t\t\t\t\t\"type\": \"Guid\"\n\t\t\t\t\t\t\t\t\t},\n\t\t\t\t\t\t\t\t\t\"StageId\": {\n\t\t\t\t\t\t\t\t\t\t\"value\": {\n\t\t\t\t\t\t\t\t\t\t\t\"value\": \"@item().StageId\",\n\t\t\t\t\t\t\t\t\t\t\t\"type\": \"Expression\"\n\t\t\t\t\t\t\t\t\t\t},\n\t\t\t\t\t\t\t\t\t\t\"type\": \"Int32\"\n\t\t\t\t\t\t\t\t\t}\n\t\t\t\t\t\t\t\t}\n\t\t\t\t\t\t\t},\n\t\t\t\t\t\t\t\"linkedServiceName\": {\n\t\t\t\t\t\t\t\t\"referenceName\": \"SupportDatabase\",\n\t\t\t\t\t\t\t\t\"type\": \"LinkedServiceReference\"\n\t\t\t\t\t\t\t}\n\t\t\t\t\t\t},\n\t\t\t\t\t\t{\n\t\t\t\t\t\t\t\"name\": \"Check and Update Blockers\",\n\t\t\t\t\t\t\t\"description\": \"Used to double check and stop the next execution stage if failures and blockers have be incurred. This also depends on the failure handling property value which defines the stored procedure behaviour.\",\n\t\t\t\t\t\t\t\"type\": \"SqlServerStoredProcedure\",\n\t\t\t\t\t\t\t\"dependsOn\": [],\n\t\t\t\t\t\t\t\"policy\": {\n\t\t\t\t\t\t\t\t\"timeout\": \"0.00:10:00\",\n\t\t\t\t\t\t\t\t\"retry\": 0,\n\t\t\t\t\t\t\t\t\"retryIntervalInSeconds\": 30,\n\t\t\t\t\t\t\t\t\"secureOutput\": false,\n\t\t\t\t\t\t\t\t\"secureInput\": false\n\t\t\t\t\t\t\t},\n\t\t\t\t\t\t\t\"userProperties\": [],\n\t\t\t\t\t\t\t\"typeProperties\": {\n\t\t\t\t\t\t\t\t\"storedProcedureName\": \"[procfwk].[CheckForBlockedPipelines]\",\n\t\t\t\t\t\t\t\t\"storedProcedureParameters\": {\n\t\t\t\t\t\t\t\t\t\"ExecutionId\": {\n\t\t\t\t\t\t\t\t\t\t\"value\": {\n\t\t\t\t\t\t\t\t\t\t\t\"value\": \"@variables('ExecutionId')\",\n\t\t\t\t\t\t\t\t\t\t\t\"type\": \"Expression\"\n\t\t\t\t\t\t\t\t\t\t},\n\t\t\t\t\t\t\t\t\t\t\"type\": \"Guid\"\n\t\t\t\t\t\t\t\t\t},\n\t\t\t\t\t\t\t\t\t\"StageId\": {\n\t\t\t\t\t\t\t\t\t\t\"value\": {\n\t\t\t\t\t\t\t\t\t\t\t\"value\": \"@item().StageId\",\n\t\t\t\t\t\t\t\t\t\t\t\"type\": \"Expression\"\n\t\t\t\t\t\t\t\t\t\t},\n\t\t\t\t\t\t\t\t\t\t\"type\": \"Int32\"\n\t\t\t\t\t\t\t\t\t}\n\t\t\t\t\t\t\t\t}\n\t\t\t\t\t\t\t},\n\t\t\t\t\t\t\t\"linkedServiceName\": {\n\t\t\t\t\t\t\t\t\"referenceName\": \"SupportDatabase\",\n\t\t\t\t\t\t\t\t\"type\": \"LinkedServiceReference\"\n\t\t\t\t\t\t\t}\n\t\t\t\t\t\t}\n\t\t\t\t\t]\n\t\t\t\t}\n\t\t\t},\n\t\t\t{\n\t\t\t\t\"name\": \"Execution Wrapper\",\n\t\t\t\t\"description\": \"Wrapper to reset and restart processing or create a completely new execution instance of the framework metadata.\",\n\t\t\t\t\"type\": \"Lookup\",\n\t\t\t\t\"dependsOn\": [\n\t\t\t\t\t{\n\t\t\t\t\t\t\"activity\": \"Clean Up Previous Run\",\n\t\t\t\t\t\t\"dependencyConditions\": [\n\t\t\t\t\t\t\t\"Succeeded\"\n\t\t\t\t\t\t]\n\t\t\t\t\t}\n\t\t\t\t],\n\t\t\t\t\"policy\": {\n\t\t\t\t\t\"timeout\": \"0.00:10:00\",\n\t\t\t\t\t\"retry\": 0,\n\t\t\t\t\t\"retryIntervalInSeconds\": 30,\n\t\t\t\t\t\"secureOutput\": false,\n\t\t\t\t\t\"secureInput\": false\n\t\t\t\t},\n\t\t\t\t\"userProperties\": [],\n\t\t\t\t\"typeProperties\": {\n\t\t\t\t\t\"source\": {\n\t\t\t\t\t\t\"type\": \"AzureSqlSource\",\n\t\t\t\t\t\t\"sqlReaderStoredProcedureName\": \"[procfwk].[ExecutionWrapper]\",\n\t\t\t\t\t\t\"storedProcedureParameters\": {\n\t\t\t\t\t\t\t\"CallingOrchestratorName\": {\n\t\t\t\t\t\t\t\t\"type\": \"String\",\n\t\t\t\t\t\t\t\t\"value\": {\n\t\t\t\t\t\t\t\t\t\"value\": \"@pipeline().DataFactory\",\n\t\t\t\t\t\t\t\t\t\"type\": \"Expression\"\n\t\t\t\t\t\t\t\t}\n\t\t\t\t\t\t\t},\n\t\t\t\t\t\t\t\"BatchName\": {\n\t\t\t\t\t\t\t\t\"type\": \"String\",\n\t\t\t\t\t\t\t\t\"value\": {\n\t\t\t\t\t\t\t\t\t\"value\": \"@pipeline().parameters.BatchName\",\n\t\t\t\t\t\t\t\t\t\"type\": \"Expression\"\n\t\t\t\t\t\t\t\t}\n\t\t\t\t\t\t\t}\n\t\t\t\t\t\t},\n\t\t\t\t\t\t\"queryTimeout\": \"02:00:00\",\n\t\t\t\t\t\t\"partitionOption\": \"None\"\n\t\t\t\t\t},\n\t\t\t\t\t\"dataset\": {\n\t\t\t\t\t\t\"referenceName\": \"GetSetMetadata\",\n\t\t\t\t\t\t\"type\": \"DatasetReference\"\n\t\t\t\t\t}\n\t\t\t\t}\n\t\t\t},\n\t\t\t{\n\t\t\t\t\"name\": \"Check Outcome and Update Logs\",\n\t\t\t\t\"description\": \"After a successful execution run the current execution metadata is moved to the long term logging table by this stored procedure call. Otherwise an error will be raised.\",\n\t\t\t\t\"type\": \"SqlServerStoredProcedure\",\n\t\t\t\t\"dependsOn\": [\n\t\t\t\t\t{\n\t\t\t\t\t\t\"activity\": \"Execute Stages\",\n\t\t\t\t\t\t\"dependencyConditions\": [\n\t\t\t\t\t\t\t\"Succeeded\"\n\t\t\t\t\t\t]\n\t\t\t\t\t}\n\t\t\t\t],\n\t\t\t\t\"policy\": {\n\t\t\t\t\t\"timeout\": \"0.00:10:00\",\n\t\t\t\t\t\"retry\": 0,\n\t\t\t\t\t\"retryIntervalInSeconds\": 30,\n\t\t\t\t\t\"secureOutput\": false,\n\t\t\t\t\t\"secureInput\": false\n\t\t\t\t},\n\t\t\t\t\"userProperties\": [],\n\t\t\t\t\"typeProperties\": {\n\t\t\t\t\t\"storedProcedureName\": \"[procfwk].[UpdateExecutionLog]\",\n\t\t\t\t\t\"storedProcedureParameters\": {\n\t\t\t\t\t\t\"PerformErrorCheck\": {\n\t\t\t\t\t\t\t\"value\": {\n\t\t\t\t\t\t\t\t\"value\": \"@bool(1)\",\n\t\t\t\t\t\t\t\t\"type\": \"Expression\"\n\t\t\t\t\t\t\t},\n\t\t\t\t\t\t\t\"type\": \"Boolean\"\n\t\t\t\t\t\t},\n\t\t\t\t\t\t\"ExecutionId\": {\n\t\t\t\t\t\t\t\"value\": {\n\t\t\t\t\t\t\t\t\"value\": \"@variables('ExecutionId')\",\n\t\t\t\t\t\t\t\t\"type\": \"Expression\"\n\t\t\t\t\t\t\t},\n\t\t\t\t\t\t\t\"type\": \"Guid\"\n\t\t\t\t\t\t}\n\t\t\t\t\t}\n\t\t\t\t},\n\t\t\t\t\"linkedServiceName\": {\n\t\t\t\t\t\"referenceName\": \"SupportDatabase\",\n\t\t\t\t\t\"type\": \"LinkedServiceReference\"\n\t\t\t\t}\n\t\t\t},\n\t\t\t{\n\t\t\t\t\"name\": \"Check Previous Execution\",\n\t\t\t\t\"description\": \"Query the current execution table for worker pipelines that require a clean up from the previous execution run.\",\n\t\t\t\t\"type\": \"Lookup\",\n\t\t\t\t\"dependsOn\": [\n\t\t\t\t\t{\n\t\t\t\t\t\t\"activity\": \"Execute Precursor\",\n\t\t\t\t\t\t\"dependencyConditions\": [\n\t\t\t\t\t\t\t\"Succeeded\"\n\t\t\t\t\t\t]\n\t\t\t\t\t}\n\t\t\t\t],\n\t\t\t\t\"policy\": {\n\t\t\t\t\t\"timeout\": \"0.00:10:00\",\n\t\t\t\t\t\"retry\": 0,\n\t\t\t\t\t\"retryIntervalInSeconds\": 30,\n\t\t\t\t\t\"secureOutput\": false,\n\t\t\t\t\t\"secureInput\": false\n\t\t\t\t},\n\t\t\t\t\"userProperties\": [],\n\t\t\t\t\"typeProperties\": {\n\t\t\t\t\t\"source\": {\n\t\t\t\t\t\t\"type\": \"AzureSqlSource\",\n\t\t\t\t\t\t\"sqlReaderStoredProcedureName\": \"[procfwk].[CheckPreviousExeuction]\",\n\t\t\t\t\t\t\"storedProcedureParameters\": {\n\t\t\t\t\t\t\t\"BatchName\": {\n\t\t\t\t\t\t\t\t\"type\": \"String\",\n\t\t\t\t\t\t\t\t\"value\": {\n\t\t\t\t\t\t\t\t\t\"value\": \"@pipeline().parameters.BatchName\",\n\t\t\t\t\t\t\t\t\t\"type\": \"Expression\"\n\t\t\t\t\t\t\t\t}\n\t\t\t\t\t\t\t}\n\t\t\t\t\t\t},\n\t\t\t\t\t\t\"queryTimeout\": \"02:00:00\",\n\t\t\t\t\t\t\"partitionOption\": \"None\"\n\t\t\t\t\t},\n\t\t\t\t\t\"dataset\": {\n\t\t\t\t\t\t\"referenceName\": \"GetSetMetadata\",\n\t\t\t\t\t\t\"type\": \"DatasetReference\"\n\t\t\t\t\t},\n\t\t\t\t\t\"firstRowOnly\": false\n\t\t\t\t}\n\t\t\t},\n\t\t\t{\n\t\t\t\t\"name\": \"Clean Up Previous Run\",\n\t\t\t\t\"description\": \"Handle Worker pipelines that are reported as Running when the parent pipeline is called again. Get what the actual status of those pipelines is.\",\n\t\t\t\t\"type\": \"ForEach\",\n\t\t\t\t\"dependsOn\": [\n\t\t\t\t\t{\n\t\t\t\t\t\t\"activity\": \"Check Previous Execution\",\n\t\t\t\t\t\t\"dependencyConditions\": [\n\t\t\t\t\t\t\t\"Succeeded\"\n\t\t\t\t\t\t]\n\t\t\t\t\t},\n\t\t\t\t\t{\n\t\t\t\t\t\t\"activity\": \"Check Metadata Integrity\",\n\t\t\t\t\t\t\"dependencyConditions\": [\n\t\t\t\t\t\t\t\"Succeeded\"\n\t\t\t\t\t\t]\n\t\t\t\t\t}\n\t\t\t\t],\n\t\t\t\t\"userProperties\": [],\n\t\t\t\t\"typeProperties\": {\n\t\t\t\t\t\"items\": {\n\t\t\t\t\t\t\"value\": \"@activity('Check Previous Execution').output.value\",\n\t\t\t\t\t\t\"type\": \"Expression\"\n\t\t\t\t\t},\n\t\t\t\t\t\"isSequential\": false,\n\t\t\t\t\t\"batchCount\": 50,\n\t\t\t\t\t\"activities\": [\n\t\t\t\t\t\t{\n\t\t\t\t\t\t\t\"name\": \"Get SPN Details\",\n\t\t\t\t\t\t\t\"type\": \"Lookup\",\n\t\t\t\t\t\t\t\"dependsOn\": [],\n\t\t\t\t\t\t\t\"policy\": {\n\t\t\t\t\t\t\t\t\"timeout\": \"0.00:10:00\",\n\t\t\t\t\t\t\t\t\"retry\": 0,\n\t\t\t\t\t\t\t\t\"retryIntervalInSeconds\": 30,\n\t\t\t\t\t\t\t\t\"secureOutput\": true,\n\t\t\t\t\t\t\t\t\"secureInput\": false\n\t\t\t\t\t\t\t},\n\t\t\t\t\t\t\t\"userProperties\": [],\n\t\t\t\t\t\t\t\"typeProperties\": {\n\t\t\t\t\t\t\t\t\"source\": {\n\t\t\t\t\t\t\t\t\t\"type\": \"AzureSqlSource\",\n\t\t\t\t\t\t\t\t\t\"sqlReaderStoredProcedureName\": \"[procfwk].[GetWorkerAuthDetails]\",\n\t\t\t\t\t\t\t\t\t\"storedProcedureParameters\": {\n\t\t\t\t\t\t\t\t\t\t\"ExecutionId\": {\n\t\t\t\t\t\t\t\t\t\t\t\"type\": \"Guid\",\n\t\t\t\t\t\t\t\t\t\t\t\"value\": {\n\t\t\t\t\t\t\t\t\t\t\t\t\"value\": \"@item().LocalExecutionId\",\n\t\t\t\t\t\t\t\t\t\t\t\t\"type\": \"Expression\"\n\t\t\t\t\t\t\t\t\t\t\t}\n\t\t\t\t\t\t\t\t\t\t},\n\t\t\t\t\t\t\t\t\t\t\"PipelineId\": {\n\t\t\t\t\t\t\t\t\t\t\t\"type\": \"Int32\",\n\t\t\t\t\t\t\t\t\t\t\t\"value\": {\n\t\t\t\t\t\t\t\t\t\t\t\t\"value\": \"@item().PipelineId\",\n\t\t\t\t\t\t\t\t\t\t\t\t\"type\": \"Expression\"\n\t\t\t\t\t\t\t\t\t\t\t}\n\t\t\t\t\t\t\t\t\t\t},\n\t\t\t\t\t\t\t\t\t\t\"StageId\": {\n\t\t\t\t\t\t\t\t\t\t\t\"type\": \"Int32\",\n\t\t\t\t\t\t\t\t\t\t\t\"value\": {\n\t\t\t\t\t\t\t\t\t\t\t\t\"value\": \"@item().StageId\",\n\t\t\t\t\t\t\t\t\t\t\t\t\"type\": \"Expression\"\n\t\t\t\t\t\t\t\t\t\t\t}\n\t\t\t\t\t\t\t\t\t\t}\n\t\t\t\t\t\t\t\t\t},\n\t\t\t\t\t\t\t\t\t\"queryTimeout\": \"02:00:00\",\n\t\t\t\t\t\t\t\t\t\"partitionOption\": \"None\"\n\t\t\t\t\t\t\t\t},\n\t\t\t\t\t\t\t\t\"dataset\": {\n\t\t\t\t\t\t\t\t\t\"referenceName\": \"GetSetMetadata\",\n\t\t\t\t\t\t\t\t\t\"type\": \"DatasetReference\"\n\t\t\t\t\t\t\t\t}\n\t\t\t\t\t\t\t}\n\t\t\t\t\t\t},\n\t\t\t\t\t\t{\n\t\t\t\t\t\t\t\"name\": \"Log Pipeline Checking\",\n\t\t\t\t\t\t\t\"type\": \"SqlServerStoredProcedure\",\n\t\t\t\t\t\t\t\"dependsOn\": [],\n\t\t\t\t\t\t\t\"policy\": {\n\t\t\t\t\t\t\t\t\"timeout\": \"0.00:10:00\",\n\t\t\t\t\t\t\t\t\"retry\": 0,\n\t\t\t\t\t\t\t\t\"retryIntervalInSeconds\": 30,\n\t\t\t\t\t\t\t\t\"secureOutput\": false,\n\t\t\t\t\t\t\t\t\"secureInput\": false\n\t\t\t\t\t\t\t},\n\t\t\t\t\t\t\t\"userProperties\": [],\n\t\t\t\t\t\t\t\"typeProperties\": {\n\t\t\t\t\t\t\t\t\"storedProcedureName\": \"[procfwk].[SetLogPipelineChecking]\",\n\t\t\t\t\t\t\t\t\"storedProcedureParameters\": {\n\t\t\t\t\t\t\t\t\t\"ExecutionId\": {\n\t\t\t\t\t\t\t\t\t\t\"value\": {\n\t\t\t\t\t\t\t\t\t\t\t\"value\": \"@item().LocalExecutionId\",\n\t\t\t\t\t\t\t\t\t\t\t\"type\": \"Expression\"\n\t\t\t\t\t\t\t\t\t\t},\n\t\t\t\t\t\t\t\t\t\t\"type\": \"Guid\"\n\t\t\t\t\t\t\t\t\t},\n\t\t\t\t\t\t\t\t\t\"PipelineId\": {\n\t\t\t\t\t\t\t\t\t\t\"value\": {\n\t\t\t\t\t\t\t\t\t\t\t\"value\": \"@item().PipelineId\",\n\t\t\t\t\t\t\t\t\t\t\t\"type\": \"Expression\"\n\t\t\t\t\t\t\t\t\t\t},\n\t\t\t\t\t\t\t\t\t\t\"type\": \"Int32\"\n\t\t\t\t\t\t\t\t\t},\n\t\t\t\t\t\t\t\t\t\"StageId\": {\n\t\t\t\t\t\t\t\t\t\t\"value\": {\n\t\t\t\t\t\t\t\t\t\t\t\"value\": \"@item().StageId\",\n\t\t\t\t\t\t\t\t\t\t\t\"type\": \"Expression\"\n\t\t\t\t\t\t\t\t\t\t},\n\t\t\t\t\t\t\t\t\t\t\"type\": \"Int32\"\n\t\t\t\t\t\t\t\t\t}\n\t\t\t\t\t\t\t\t}\n\t\t\t\t\t\t\t},\n\t\t\t\t\t\t\t\"linkedServiceName\": {\n\t\t\t\t\t\t\t\t\"referenceName\": \"SupportDatabase\",\n\t\t\t\t\t\t\t\t\"type\": \"LinkedServiceReference\"\n\t\t\t\t\t\t\t}\n\t\t\t\t\t\t},\n\t\t\t\t\t\t{\n\t\t\t\t\t\t\t\"name\": \"Get Pipeline Status\",\n\t\t\t\t\t\t\t\"type\": \"AzureFunctionActivity\",\n\t\t\t\t\t\t\t\"dependsOn\": [\n\t\t\t\t\t\t\t\t{\n\t\t\t\t\t\t\t\t\t\"activity\": \"Get SPN Details\",\n\t\t\t\t\t\t\t\t\t\"dependencyConditions\": [\n\t\t\t\t\t\t\t\t\t\t\"Succeeded\"\n\t\t\t\t\t\t\t\t\t]\n\t\t\t\t\t\t\t\t},\n\t\t\t\t\t\t\t\t{\n\t\t\t\t\t\t\t\t\t\"activity\": \"Log Pipeline Checking\",\n\t\t\t\t\t\t\t\t\t\"dependencyConditions\": [\n\t\t\t\t\t\t\t\t\t\t\"Succeeded\"\n\t\t\t\t\t\t\t\t\t]\n\t\t\t\t\t\t\t\t}\n\t\t\t\t\t\t\t],\n\t\t\t\t\t\t\t\"policy\": {\n\t\t\t\t\t\t\t\t\"timeout\": \"0.00:10:00\",\n\t\t\t\t\t\t\t\t\"retry\": 0,\n\t\t\t\t\t\t\t\t\"retryIntervalInSeconds\": 30,\n\t\t\t\t\t\t\t\t\"secureOutput\": false,\n\t\t\t\t\t\t\t\t\"secureInput\": true\n\t\t\t\t\t\t\t},\n\t\t\t\t\t\t\t\"userProperties\": [],\n\t\t\t\t\t\t\t\"typeProperties\": {\n\t\t\t\t\t\t\t\t\"functionName\": \"CheckPipelineStatus\",\n\t\t\t\t\t\t\t\t\"method\": \"POST\",\n\t\t\t\t\t\t\t\t\"body\": {\n\t\t\t\t\t\t\t\t\t\"value\": \"@concat('\\n{\\n \\\"tenantId\\\": \\\"',activity('Get SPN Details').output.firstRow.TenantId,'\\\",\\n \\\"applicationId\\\": \\\"',activity('Get SPN Details').output.firstRow.AppId,'\\\",\\n \\\"authenticationKey\\\": \\\"',activity('Get SPN Details').output.firstRow.AppSecret,'\\\",\\n \\\"subscriptionId\\\": \\\"',activity('Get SPN Details').output.firstRow.SubscriptionId,'\\\",\\n \\\"resourceGroupName\\\": \\\"',item().ResourceGroupName,'\\\",\\n \\\"orchestratorName\\\": \\\"',item().OrchestratorName,'\\\",\\n \\\"orchestratorType\\\": \\\"',item().OrchestratorType,'\\\",\\n \\\"pipelineName\\\": \\\"',item().PipelineName,'\\\",\\n \\\"runId\\\": \\\"',item().PipelineRunId,'\\\"\\n}')\",\n\t\t\t\t\t\t\t\t\t\"type\": \"Expression\"\n\t\t\t\t\t\t\t\t}\n\t\t\t\t\t\t\t},\n\t\t\t\t\t\t\t\"linkedServiceName\": {\n\t\t\t\t\t\t\t\t\"referenceName\": \"FrameworkFunctions\",\n\t\t\t\t\t\t\t\t\"type\": \"LinkedServiceReference\"\n\t\t\t\t\t\t\t}\n\t\t\t\t\t\t},\n\t\t\t\t\t\t{\n\t\t\t\t\t\t\t\"name\": \"Set Pipeline Status\",\n\t\t\t\t\t\t\t\"description\": \"Update the metadata depending on the actual pipeline outcome. Using the status as the case.\",\n\t\t\t\t\t\t\t\"type\": \"Switch\",\n\t\t\t\t\t\t\t\"dependsOn\": [\n\t\t\t\t\t\t\t\t{\n\t\t\t\t\t\t\t\t\t\"activity\": \"Get Pipeline Status\",\n\t\t\t\t\t\t\t\t\t\"dependencyConditions\": [\n\t\t\t\t\t\t\t\t\t\t\"Succeeded\"\n\t\t\t\t\t\t\t\t\t]\n\t\t\t\t\t\t\t\t}\n\t\t\t\t\t\t\t],\n\t\t\t\t\t\t\t\"userProperties\": [],\n\t\t\t\t\t\t\t\"typeProperties\": {\n\t\t\t\t\t\t\t\t\"on\": {\n\t\t\t\t\t\t\t\t\t\"value\": \"@activity('Get Pipeline Status').output.ActualStatus\",\n\t\t\t\t\t\t\t\t\t\"type\": \"Expression\"\n\t\t\t\t\t\t\t\t},\n\t\t\t\t\t\t\t\t\"cases\": [\n\t\t\t\t\t\t\t\t\t{\n\t\t\t\t\t\t\t\t\t\t\"value\": \"Failed\",\n\t\t\t\t\t\t\t\t\t\t\"activities\": [\n\t\t\t\t\t\t\t\t\t\t\t{\n\t\t\t\t\t\t\t\t\t\t\t\t\"name\": \"Pipeline Status Failed\",\n\t\t\t\t\t\t\t\t\t\t\t\t\"description\": \"Updates the current execution table with a pipeline status of failed if the function outcome is failed. Also blocks pipelines in the downstream execution stage.\",\n\t\t\t\t\t\t\t\t\t\t\t\t\"type\": \"SqlServerStoredProcedure\",\n\t\t\t\t\t\t\t\t\t\t\t\t\"dependsOn\": [],\n\t\t\t\t\t\t\t\t\t\t\t\t\"policy\": {\n\t\t\t\t\t\t\t\t\t\t\t\t\t\"timeout\": \"0.00:10:00\",\n\t\t\t\t\t\t\t\t\t\t\t\t\t\"retry\": 0,\n\t\t\t\t\t\t\t\t\t\t\t\t\t\"retryIntervalInSeconds\": 30,\n\t\t\t\t\t\t\t\t\t\t\t\t\t\"secureOutput\": false,\n\t\t\t\t\t\t\t\t\t\t\t\t\t\"secureInput\": false\n\t\t\t\t\t\t\t\t\t\t\t\t},\n\t\t\t\t\t\t\t\t\t\t\t\t\"userProperties\": [],\n\t\t\t\t\t\t\t\t\t\t\t\t\"typeProperties\": {\n\t\t\t\t\t\t\t\t\t\t\t\t\t\"storedProcedureName\": \"[procfwk].[SetLogPipelineFailed]\",\n\t\t\t\t\t\t\t\t\t\t\t\t\t\"storedProcedureParameters\": {\n\t\t\t\t\t\t\t\t\t\t\t\t\t\t\"ExecutionId\": {\n\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\"value\": {\n\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\"value\": \"@item().LocalExecutionId\",\n\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\"type\": \"Expression\"\n\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t},\n\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\"type\": \"Guid\"\n\t\t\t\t\t\t\t\t\t\t\t\t\t\t},\n\t\t\t\t\t\t\t\t\t\t\t\t\t\t\"PipelineId\": {\n\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\"value\": {\n\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\"value\": \"@item().PipelineId\",\n\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\"type\": \"Expression\"\n\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t},\n\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\"type\": \"Int32\"\n\t\t\t\t\t\t\t\t\t\t\t\t\t\t},\n\t\t\t\t\t\t\t\t\t\t\t\t\t\t\"RunId\": {\n\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\"value\": null,\n\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\"type\": \"Guid\"\n\t\t\t\t\t\t\t\t\t\t\t\t\t\t},\n\t\t\t\t\t\t\t\t\t\t\t\t\t\t\"StageId\": {\n\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\"value\": {\n\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\"value\": \"@item().StageId\",\n\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\"type\": \"Expression\"\n\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t},\n\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\"type\": \"Int32\"\n\t\t\t\t\t\t\t\t\t\t\t\t\t\t}\n\t\t\t\t\t\t\t\t\t\t\t\t\t}\n\t\t\t\t\t\t\t\t\t\t\t\t},\n\t\t\t\t\t\t\t\t\t\t\t\t\"linkedServiceName\": {\n\t\t\t\t\t\t\t\t\t\t\t\t\t\"referenceName\": \"SupportDatabase\",\n\t\t\t\t\t\t\t\t\t\t\t\t\t\"type\": \"LinkedServiceReference\"\n\t\t\t\t\t\t\t\t\t\t\t\t}\n\t\t\t\t\t\t\t\t\t\t\t}\n\t\t\t\t\t\t\t\t\t\t]\n\t\t\t\t\t\t\t\t\t},\n\t\t\t\t\t\t\t\t\t{\n\t\t\t\t\t\t\t\t\t\t\"value\": \"Succeeded\",\n\t\t\t\t\t\t\t\t\t\t\"activities\": [\n\t\t\t\t\t\t\t\t\t\t\t{\n\t\t\t\t\t\t\t\t\t\t\t\t\"name\": \"Pipeline Status Succeeded\",\n\t\t\t\t\t\t\t\t\t\t\t\t\"description\": \"Updates the current execution table with a pipeline status of success if the function outcome is succeeded.\",\n\t\t\t\t\t\t\t\t\t\t\t\t\"type\": \"SqlServerStoredProcedure\",\n\t\t\t\t\t\t\t\t\t\t\t\t\"dependsOn\": [],\n\t\t\t\t\t\t\t\t\t\t\t\t\"policy\": {\n\t\t\t\t\t\t\t\t\t\t\t\t\t\"timeout\": \"0.00:10:00\",\n\t\t\t\t\t\t\t\t\t\t\t\t\t\"retry\": 0,\n\t\t\t\t\t\t\t\t\t\t\t\t\t\"retryIntervalInSeconds\": 30,\n\t\t\t\t\t\t\t\t\t\t\t\t\t\"secureOutput\": false,\n\t\t\t\t\t\t\t\t\t\t\t\t\t\"secureInput\": false\n\t\t\t\t\t\t\t\t\t\t\t\t},\n\t\t\t\t\t\t\t\t\t\t\t\t\"userProperties\": [],\n\t\t\t\t\t\t\t\t\t\t\t\t\"typeProperties\": {\n\t\t\t\t\t\t\t\t\t\t\t\t\t\"storedProcedureName\": \"[procfwk].[SetLogPipelineSuccess]\",\n\t\t\t\t\t\t\t\t\t\t\t\t\t\"storedProcedureParameters\": {\n\t\t\t\t\t\t\t\t\t\t\t\t\t\t\"ExecutionId\": {\n\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\"value\": {\n\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\"value\": \"@item().LocalExecutionId\",\n\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\"type\": \"Expression\"\n\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t},\n\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\"type\": \"Guid\"\n\t\t\t\t\t\t\t\t\t\t\t\t\t\t},\n\t\t\t\t\t\t\t\t\t\t\t\t\t\t\"PipelineId\": {\n\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\"value\": {\n\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\"value\": \"@item().PipelineId\",\n\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\"type\": \"Expression\"\n\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t},\n\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\"type\": \"Int32\"\n\t\t\t\t\t\t\t\t\t\t\t\t\t\t},\n\t\t\t\t\t\t\t\t\t\t\t\t\t\t\"StageId\": {\n\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\"value\": {\n\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\"value\": \"@item().StageId\",\n\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\"type\": \"Expression\"\n\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t},\n\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\"type\": \"Int32\"\n\t\t\t\t\t\t\t\t\t\t\t\t\t\t}\n\t\t\t\t\t\t\t\t\t\t\t\t\t}\n\t\t\t\t\t\t\t\t\t\t\t\t},\n\t\t\t\t\t\t\t\t\t\t\t\t\"linkedServiceName\": {\n\t\t\t\t\t\t\t\t\t\t\t\t\t\"referenceName\": \"SupportDatabase\",\n\t\t\t\t\t\t\t\t\t\t\t\t\t\"type\": \"LinkedServiceReference\"\n\t\t\t\t\t\t\t\t\t\t\t\t}\n\t\t\t\t\t\t\t\t\t\t\t}\n\t\t\t\t\t\t\t\t\t\t]\n\t\t\t\t\t\t\t\t\t},\n\t\t\t\t\t\t\t\t\t{\n\t\t\t\t\t\t\t\t\t\t\"value\": \"Queued\",\n\t\t\t\t\t\t\t\t\t\t\"activities\": [\n\t\t\t\t\t\t\t\t\t\t\t{\n\t\t\t\t\t\t\t\t\t\t\t\t\"name\": \"Pipeline Status Queued - Running\",\n\t\t\t\t\t\t\t\t\t\t\t\t\"description\": \"Updates the current execution table with a pipeline status of running if the function outcome is queued.\",\n\t\t\t\t\t\t\t\t\t\t\t\t\"type\": \"SqlServerStoredProcedure\",\n\t\t\t\t\t\t\t\t\t\t\t\t\"dependsOn\": [],\n\t\t\t\t\t\t\t\t\t\t\t\t\"policy\": {\n\t\t\t\t\t\t\t\t\t\t\t\t\t\"timeout\": \"0.00:10:00\",\n\t\t\t\t\t\t\t\t\t\t\t\t\t\"retry\": 0,\n\t\t\t\t\t\t\t\t\t\t\t\t\t\"retryIntervalInSeconds\": 30,\n\t\t\t\t\t\t\t\t\t\t\t\t\t\"secureOutput\": false,\n\t\t\t\t\t\t\t\t\t\t\t\t\t\"secureInput\": false\n\t\t\t\t\t\t\t\t\t\t\t\t},\n\t\t\t\t\t\t\t\t\t\t\t\t\"userProperties\": [],\n\t\t\t\t\t\t\t\t\t\t\t\t\"typeProperties\": {\n\t\t\t\t\t\t\t\t\t\t\t\t\t\"storedProcedureName\": \"[procfwk].[SetLogPipelineRunning]\",\n\t\t\t\t\t\t\t\t\t\t\t\t\t\"storedProcedureParameters\": {\n\t\t\t\t\t\t\t\t\t\t\t\t\t\t\"ExecutionId\": {\n\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\"value\": {\n\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\"value\": \"@item().LocalExecutionId\",\n\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\"type\": \"Expression\"\n\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t},\n\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\"type\": \"Guid\"\n\t\t\t\t\t\t\t\t\t\t\t\t\t\t},\n\t\t\t\t\t\t\t\t\t\t\t\t\t\t\"PipelineId\": {\n\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\"value\": {\n\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\"value\": \"@item().PipelineId\",\n\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\"type\": \"Expression\"\n\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t},\n\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\"type\": \"Int32\"\n\t\t\t\t\t\t\t\t\t\t\t\t\t\t},\n\t\t\t\t\t\t\t\t\t\t\t\t\t\t\"StageId\": {\n\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\"value\": {\n\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\"value\": \"@item().StageId\",\n\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\"type\": \"Expression\"\n\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t},\n\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\"type\": \"Int32\"\n\t\t\t\t\t\t\t\t\t\t\t\t\t\t}\n\t\t\t\t\t\t\t\t\t\t\t\t\t}\n\t\t\t\t\t\t\t\t\t\t\t\t},\n\t\t\t\t\t\t\t\t\t\t\t\t\"linkedServiceName\": {\n\t\t\t\t\t\t\t\t\t\t\t\t\t\"referenceName\": \"SupportDatabase\",\n\t\t\t\t\t\t\t\t\t\t\t\t\t\"type\": \"LinkedServiceReference\"\n\t\t\t\t\t\t\t\t\t\t\t\t}\n\t\t\t\t\t\t\t\t\t\t\t}\n\t\t\t\t\t\t\t\t\t\t]\n\t\t\t\t\t\t\t\t\t},\n\t\t\t\t\t\t\t\t\t{\n\t\t\t\t\t\t\t\t\t\t\"value\": \"InProgress\",\n\t\t\t\t\t\t\t\t\t\t\"activities\": [\n\t\t\t\t\t\t\t\t\t\t\t{\n\t\t\t\t\t\t\t\t\t\t\t\t\"name\": \"Pipeline Status InProgress - Running\",\n\t\t\t\t\t\t\t\t\t\t\t\t\"description\": \"Updates the current execution table with a pipeline status of running if the function outcome is in progress.\",\n\t\t\t\t\t\t\t\t\t\t\t\t\"type\": \"SqlServerStoredProcedure\",\n\t\t\t\t\t\t\t\t\t\t\t\t\"dependsOn\": [],\n\t\t\t\t\t\t\t\t\t\t\t\t\"policy\": {\n\t\t\t\t\t\t\t\t\t\t\t\t\t\"timeout\": \"0.00:10:00\",\n\t\t\t\t\t\t\t\t\t\t\t\t\t\"retry\": 0,\n\t\t\t\t\t\t\t\t\t\t\t\t\t\"retryIntervalInSeconds\": 30,\n\t\t\t\t\t\t\t\t\t\t\t\t\t\"secureOutput\": false,\n\t\t\t\t\t\t\t\t\t\t\t\t\t\"secureInput\": false\n\t\t\t\t\t\t\t\t\t\t\t\t},\n\t\t\t\t\t\t\t\t\t\t\t\t\"userProperties\": [],\n\t\t\t\t\t\t\t\t\t\t\t\t\"typeProperties\": {\n\t\t\t\t\t\t\t\t\t\t\t\t\t\"storedProcedureName\": \"[procfwk].[SetLogPipelineRunning]\",\n\t\t\t\t\t\t\t\t\t\t\t\t\t\"storedProcedureParameters\": {\n\t\t\t\t\t\t\t\t\t\t\t\t\t\t\"ExecutionId\": {\n\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\"value\": {\n\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\"value\": \"@item().LocalExecutionId\",\n\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\"type\": \"Expression\"\n\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t},\n\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\"type\": \"Guid\"\n\t\t\t\t\t\t\t\t\t\t\t\t\t\t},\n\t\t\t\t\t\t\t\t\t\t\t\t\t\t\"PipelineId\": {\n\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\"value\": {\n\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\"value\": \"@item().PipelineId\",\n\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\"type\": \"Expression\"\n\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t},\n\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\"type\": \"Int32\"\n\t\t\t\t\t\t\t\t\t\t\t\t\t\t},\n\t\t\t\t\t\t\t\t\t\t\t\t\t\t\"StageId\": {\n\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\"value\": {\n\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\"value\": \"@item().StageId\",\n\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\"type\": \"Expression\"\n\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t},\n\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\"type\": \"Int32\"\n\t\t\t\t\t\t\t\t\t\t\t\t\t\t}\n\t\t\t\t\t\t\t\t\t\t\t\t\t}\n\t\t\t\t\t\t\t\t\t\t\t\t},\n\t\t\t\t\t\t\t\t\t\t\t\t\"linkedServiceName\": {\n\t\t\t\t\t\t\t\t\t\t\t\t\t\"referenceName\": \"SupportDatabase\",\n\t\t\t\t\t\t\t\t\t\t\t\t\t\"type\": \"LinkedServiceReference\"\n\t\t\t\t\t\t\t\t\t\t\t\t}\n\t\t\t\t\t\t\t\t\t\t\t}\n\t\t\t\t\t\t\t\t\t\t]\n\t\t\t\t\t\t\t\t\t},\n\t\t\t\t\t\t\t\t\t{\n\t\t\t\t\t\t\t\t\t\t\"value\": \"Cancelled\",\n\t\t\t\t\t\t\t\t\t\t\"activities\": [\n\t\t\t\t\t\t\t\t\t\t\t{\n\t\t\t\t\t\t\t\t\t\t\t\t\"name\": \"Pipeline Status Cancelled\",\n\t\t\t\t\t\t\t\t\t\t\t\t\"description\": \"Updates the current execution table with a pipeline status of cancelled if the function outcome is cancelled.\",\n\t\t\t\t\t\t\t\t\t\t\t\t\"type\": \"SqlServerStoredProcedure\",\n\t\t\t\t\t\t\t\t\t\t\t\t\"dependsOn\": [],\n\t\t\t\t\t\t\t\t\t\t\t\t\"policy\": {\n\t\t\t\t\t\t\t\t\t\t\t\t\t\"timeout\": \"0.00:10:00\",\n\t\t\t\t\t\t\t\t\t\t\t\t\t\"retry\": 0,\n\t\t\t\t\t\t\t\t\t\t\t\t\t\"retryIntervalInSeconds\": 30,\n\t\t\t\t\t\t\t\t\t\t\t\t\t\"secureOutput\": false,\n\t\t\t\t\t\t\t\t\t\t\t\t\t\"secureInput\": false\n\t\t\t\t\t\t\t\t\t\t\t\t},\n\t\t\t\t\t\t\t\t\t\t\t\t\"userProperties\": [],\n\t\t\t\t\t\t\t\t\t\t\t\t\"typeProperties\": {\n\t\t\t\t\t\t\t\t\t\t\t\t\t\"storedProcedureName\": \"[procfwk].[SetLogPipelineCancelled]\",\n\t\t\t\t\t\t\t\t\t\t\t\t\t\"storedProcedureParameters\": {\n\t\t\t\t\t\t\t\t\t\t\t\t\t\t\"ExecutionId\": {\n\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\"value\": {\n\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\"value\": \"@item().LocalExecutionId\",\n\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\"type\": \"Expression\"\n\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t},\n\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\"type\": \"Guid\"\n\t\t\t\t\t\t\t\t\t\t\t\t\t\t},\n\t\t\t\t\t\t\t\t\t\t\t\t\t\t\"PipelineId\": {\n\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\"value\": {\n\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\"value\": \"@item().PipelineId\",\n\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\"type\": \"Expression\"\n\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t},\n\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\"type\": \"Int32\"\n\t\t\t\t\t\t\t\t\t\t\t\t\t\t},\n\t\t\t\t\t\t\t\t\t\t\t\t\t\t\"StageId\": {\n\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\"value\": {\n\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\"value\": \"@item().StageId\",\n\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\"type\": \"Expression\"\n\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t},\n\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\"type\": \"Int32\"\n\t\t\t\t\t\t\t\t\t\t\t\t\t\t},\n\t\t\t\t\t\t\t\t\t\t\t\t\t\t\"CleanUpRun\": {\n\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\"value\": {\n\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\"value\": \"@bool(1)\",\n\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\"type\": \"Expression\"\n\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t},\n\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\"type\": \"Boolean\"\n\t\t\t\t\t\t\t\t\t\t\t\t\t\t}\n\t\t\t\t\t\t\t\t\t\t\t\t\t}\n\t\t\t\t\t\t\t\t\t\t\t\t},\n\t\t\t\t\t\t\t\t\t\t\t\t\"linkedServiceName\": {\n\t\t\t\t\t\t\t\t\t\t\t\t\t\"referenceName\": \"SupportDatabase\",\n\t\t\t\t\t\t\t\t\t\t\t\t\t\"type\": \"LinkedServiceReference\"\n\t\t\t\t\t\t\t\t\t\t\t\t}\n\t\t\t\t\t\t\t\t\t\t\t}\n\t\t\t\t\t\t\t\t\t\t]\n\t\t\t\t\t\t\t\t\t}\n\t\t\t\t\t\t\t\t],\n\t\t\t\t\t\t\t\t\"defaultActivities\": [\n\t\t\t\t\t\t\t\t\t{\n\t\t\t\t\t\t\t\t\t\t\"name\": \"Pipeline Status Unknown\",\n\t\t\t\t\t\t\t\t\t\t\"description\": \"Updates the current execution table with a pipeline status of unknown if the function returns an unexpected outcome.\",\n\t\t\t\t\t\t\t\t\t\t\"type\": \"SqlServerStoredProcedure\",\n\t\t\t\t\t\t\t\t\t\t\"dependsOn\": [],\n\t\t\t\t\t\t\t\t\t\t\"policy\": {\n\t\t\t\t\t\t\t\t\t\t\t\"timeout\": \"0.00:10:00\",\n\t\t\t\t\t\t\t\t\t\t\t\"retry\": 0,\n\t\t\t\t\t\t\t\t\t\t\t\"retryIntervalInSeconds\": 30,\n\t\t\t\t\t\t\t\t\t\t\t\"secureOutput\": false,\n\t\t\t\t\t\t\t\t\t\t\t\"secureInput\": false\n\t\t\t\t\t\t\t\t\t\t},\n\t\t\t\t\t\t\t\t\t\t\"userProperties\": [],\n\t\t\t\t\t\t\t\t\t\t\"typeProperties\": {\n\t\t\t\t\t\t\t\t\t\t\t\"storedProcedureName\": \"[procfwk].[SetLogPipelineUnknown]\",\n\t\t\t\t\t\t\t\t\t\t\t\"storedProcedureParameters\": {\n\t\t\t\t\t\t\t\t\t\t\t\t\"ExecutionId\": {\n\t\t\t\t\t\t\t\t\t\t\t\t\t\"value\": {\n\t\t\t\t\t\t\t\t\t\t\t\t\t\t\"value\": \"@item().LocalExecutionId\",\n\t\t\t\t\t\t\t\t\t\t\t\t\t\t\"type\": \"Expression\"\n\t\t\t\t\t\t\t\t\t\t\t\t\t},\n\t\t\t\t\t\t\t\t\t\t\t\t\t\"type\": \"Guid\"\n\t\t\t\t\t\t\t\t\t\t\t\t},\n\t\t\t\t\t\t\t\t\t\t\t\t\"PipelineId\": {\n\t\t\t\t\t\t\t\t\t\t\t\t\t\"value\": {\n\t\t\t\t\t\t\t\t\t\t\t\t\t\t\"value\": \"@item().PipelineId\",\n\t\t\t\t\t\t\t\t\t\t\t\t\t\t\"type\": \"Expression\"\n\t\t\t\t\t\t\t\t\t\t\t\t\t},\n\t\t\t\t\t\t\t\t\t\t\t\t\t\"type\": \"Int32\"\n\t\t\t\t\t\t\t\t\t\t\t\t},\n\t\t\t\t\t\t\t\t\t\t\t\t\"StageId\": {\n\t\t\t\t\t\t\t\t\t\t\t\t\t\"value\": {\n\t\t\t\t\t\t\t\t\t\t\t\t\t\t\"value\": \"@item().StageId\",\n\t\t\t\t\t\t\t\t\t\t\t\t\t\t\"type\": \"Expression\"\n\t\t\t\t\t\t\t\t\t\t\t\t\t},\n\t\t\t\t\t\t\t\t\t\t\t\t\t\"type\": \"Int32\"\n\t\t\t\t\t\t\t\t\t\t\t\t},\n\t\t\t\t\t\t\t\t\t\t\t\t\"CleanUpRun\": {\n\t\t\t\t\t\t\t\t\t\t\t\t\t\"value\": {\n\t\t\t\t\t\t\t\t\t\t\t\t\t\t\"value\": \"@bool(1)\",\n\t\t\t\t\t\t\t\t\t\t\t\t\t\t\"type\": \"Expression\"\n\t\t\t\t\t\t\t\t\t\t\t\t\t},\n\t\t\t\t\t\t\t\t\t\t\t\t\t\"type\": \"Boolean\"\n\t\t\t\t\t\t\t\t\t\t\t\t}\n\t\t\t\t\t\t\t\t\t\t\t}\n\t\t\t\t\t\t\t\t\t\t},\n\t\t\t\t\t\t\t\t\t\t\"linkedServiceName\": {\n\t\t\t\t\t\t\t\t\t\t\t\"referenceName\": \"SupportDatabase\",\n\t\t\t\t\t\t\t\t\t\t\t\"type\": \"LinkedServiceReference\"\n\t\t\t\t\t\t\t\t\t\t}\n\t\t\t\t\t\t\t\t\t}\n\t\t\t\t\t\t\t\t]\n\t\t\t\t\t\t\t}\n\t\t\t\t\t\t},\n\t\t\t\t\t\t{\n\t\t\t\t\t\t\t\"name\": \"Set Last Check DateTime\",\n\t\t\t\t\t\t\t\"description\": \"Update the current execution table with a date time from when the function last checked the pipeline status.\",\n\t\t\t\t\t\t\t\"type\": \"SqlServerStoredProcedure\",\n\t\t\t\t\t\t\t\"dependsOn\": [\n\t\t\t\t\t\t\t\t{\n\t\t\t\t\t\t\t\t\t\"activity\": \"Get Pipeline Status\",\n\t\t\t\t\t\t\t\t\t\"dependencyConditions\": [\n\t\t\t\t\t\t\t\t\t\t\"Succeeded\"\n\t\t\t\t\t\t\t\t\t]\n\t\t\t\t\t\t\t\t}\n\t\t\t\t\t\t\t],\n\t\t\t\t\t\t\t\"policy\": {\n\t\t\t\t\t\t\t\t\"timeout\": \"0.00:10:00\",\n\t\t\t\t\t\t\t\t\"retry\": 0,\n\t\t\t\t\t\t\t\t\"retryIntervalInSeconds\": 30,\n\t\t\t\t\t\t\t\t\"secureOutput\": false,\n\t\t\t\t\t\t\t\t\"secureInput\": false\n\t\t\t\t\t\t\t},\n\t\t\t\t\t\t\t\"userProperties\": [],\n\t\t\t\t\t\t\t\"typeProperties\": {\n\t\t\t\t\t\t\t\t\"storedProcedureName\": \"[procfwk].[SetLogPipelineLastStatusCheck]\",\n\t\t\t\t\t\t\t\t\"storedProcedureParameters\": {\n\t\t\t\t\t\t\t\t\t\"ExecutionId\": {\n\t\t\t\t\t\t\t\t\t\t\"value\": {\n\t\t\t\t\t\t\t\t\t\t\t\"value\": \"@item().LocalExecutionId\",\n\t\t\t\t\t\t\t\t\t\t\t\"type\": \"Expression\"\n\t\t\t\t\t\t\t\t\t\t},\n\t\t\t\t\t\t\t\t\t\t\"type\": \"Guid\"\n\t\t\t\t\t\t\t\t\t},\n\t\t\t\t\t\t\t\t\t\"PipelineId\": {\n\t\t\t\t\t\t\t\t\t\t\"value\": {\n\t\t\t\t\t\t\t\t\t\t\t\"value\": \"@item().PipelineId\",\n\t\t\t\t\t\t\t\t\t\t\t\"type\": \"Expression\"\n\t\t\t\t\t\t\t\t\t\t},\n\t\t\t\t\t\t\t\t\t\t\"type\": \"Int32\"\n\t\t\t\t\t\t\t\t\t},\n\t\t\t\t\t\t\t\t\t\"StageId\": {\n\t\t\t\t\t\t\t\t\t\t\"value\": {\n\t\t\t\t\t\t\t\t\t\t\t\"value\": \"@item().StageId\",\n\t\t\t\t\t\t\t\t\t\t\t\"type\": \"Expression\"\n\t\t\t\t\t\t\t\t\t\t},\n\t\t\t\t\t\t\t\t\t\t\"type\": \"Int32\"\n\t\t\t\t\t\t\t\t\t}\n\t\t\t\t\t\t\t\t}\n\t\t\t\t\t\t\t},\n\t\t\t\t\t\t\t\"linkedServiceName\": {\n\t\t\t\t\t\t\t\t\"referenceName\": \"SupportDatabase\",\n\t\t\t\t\t\t\t\t\"type\": \"LinkedServiceReference\"\n\t\t\t\t\t\t\t}\n\t\t\t\t\t\t}\n\t\t\t\t\t]\n\t\t\t\t}\n\t\t\t},\n\t\t\t{\n\t\t\t\t\"name\": \"Execute Precursor\",\n\t\t\t\t\"description\": \"Uses the database property value ExecutionPrecursorProc to run any custom logic against the metadata database before the execution run starts.\",\n\t\t\t\t\"type\": \"SqlServerStoredProcedure\",\n\t\t\t\t\"dependsOn\": [],\n\t\t\t\t\"policy\": {\n\t\t\t\t\t\"timeout\": \"0.00:10:00\",\n\t\t\t\t\t\"retry\": 0,\n\t\t\t\t\t\"retryIntervalInSeconds\": 30,\n\t\t\t\t\t\"secureOutput\": false,\n\t\t\t\t\t\"secureInput\": false\n\t\t\t\t},\n\t\t\t\t\"userProperties\": [],\n\t\t\t\t\"typeProperties\": {\n\t\t\t\t\t\"storedProcedureName\": \"[procfwk].[ExecutePrecursorProcedure]\"\n\t\t\t\t},\n\t\t\t\t\"linkedServiceName\": {\n\t\t\t\t\t\"referenceName\": \"SupportDatabase\",\n\t\t\t\t\t\"type\": \"LinkedServiceReference\"\n\t\t\t\t}\n\t\t\t},\n\t\t\t{\n\t\t\t\t\"name\": \"Set Execution Id\",\n\t\t\t\t\"description\": \"Set the local execution Id to a pipeline variable for each in several downstream activities.\",\n\t\t\t\t\"type\": \"SetVariable\",\n\t\t\t\t\"dependsOn\": [\n\t\t\t\t\t{\n\t\t\t\t\t\t\"activity\": \"Execution Wrapper\",\n\t\t\t\t\t\t\"dependencyConditions\": [\n\t\t\t\t\t\t\t\"Succeeded\"\n\t\t\t\t\t\t]\n\t\t\t\t\t}\n\t\t\t\t],\n\t\t\t\t\"userProperties\": [],\n\t\t\t\t\"typeProperties\": {\n\t\t\t\t\t\"variableName\": \"ExecutionId\",\n\t\t\t\t\t\"value\": {\n\t\t\t\t\t\t\"value\": \"@activity('Execution Wrapper').output.firstRow.ExecutionId\",\n\t\t\t\t\t\t\"type\": \"Expression\"\n\t\t\t\t\t}\n\t\t\t\t}\n\t\t\t},\n\t\t\t{\n\t\t\t\t\"name\": \"Check Metadata Integrity\",\n\t\t\t\t\"description\": \"Performs a series of checks on all metadata held in the framework SQLDB. This is intended to raise errors before an execution run even starts.\",\n\t\t\t\t\"type\": \"SqlServerStoredProcedure\",\n\t\t\t\t\"dependsOn\": [\n\t\t\t\t\t{\n\t\t\t\t\t\t\"activity\": \"Execute Precursor\",\n\t\t\t\t\t\t\"dependencyConditions\": [\n\t\t\t\t\t\t\t\"Succeeded\"\n\t\t\t\t\t\t]\n\t\t\t\t\t}\n\t\t\t\t],\n\t\t\t\t\"policy\": {\n\t\t\t\t\t\"timeout\": \"0.00:10:00\",\n\t\t\t\t\t\"retry\": 0,\n\t\t\t\t\t\"retryIntervalInSeconds\": 30,\n\t\t\t\t\t\"secureOutput\": false,\n\t\t\t\t\t\"secureInput\": false\n\t\t\t\t},\n\t\t\t\t\"userProperties\": [],\n\t\t\t\t\"typeProperties\": {\n\t\t\t\t\t\"storedProcedureName\": \"[procfwk].[CheckMetadataIntegrity]\",\n\t\t\t\t\t\"storedProcedureParameters\": {\n\t\t\t\t\t\t\"BatchName\": {\n\t\t\t\t\t\t\t\"value\": {\n\t\t\t\t\t\t\t\t\"value\": \"@pipeline().parameters.BatchName\",\n\t\t\t\t\t\t\t\t\"type\": \"Expression\"\n\t\t\t\t\t\t\t},\n\t\t\t\t\t\t\t\"type\": \"String\"\n\t\t\t\t\t\t},\n\t\t\t\t\t\t\"DebugMode\": {\n\t\t\t\t\t\t\t\"value\": {\n\t\t\t\t\t\t\t\t\"value\": \"@bool(0)\",\n\t\t\t\t\t\t\t\t\"type\": \"Expression\"\n\t\t\t\t\t\t\t},\n\t\t\t\t\t\t\t\"type\": \"Boolean\"\n\t\t\t\t\t\t}\n\t\t\t\t\t}\n\t\t\t\t},\n\t\t\t\t\"linkedServiceName\": {\n\t\t\t\t\t\"referenceName\": \"SupportDatabase\",\n\t\t\t\t\t\"type\": \"LinkedServiceReference\"\n\t\t\t\t}\n\t\t\t}\n\t\t],\n\t\t\"parameters\": {\n\t\t\t\"BatchName\": {\n\t\t\t\t\"type\": \"string\",\n\t\t\t\t\"defaultValue\": \"NotUsed\"\n\t\t\t}\n\t\t},\n\t\t\"variables\": {\n\t\t\t\"ExecutionId\": {\n\t\t\t\t\"type\": \"String\"\n\t\t\t}\n\t\t},\n\t\t\"folder\": {\n\t\t\t\"name\": \"_ProcFwk\"\n\t\t},\n\t\t\"annotations\": [\n\t\t\t\"procfwk\",\n\t\t\t\"Parent\"\n\t\t]\n\t}\n}"
},
{
"path": "Synapse/pipeline/03-Child.json",
"content": "{\n\t\"name\": \"03-Child\",\n\t\"properties\": {\n\t\t\"description\": \"procfwk child pipeline used to execute Worker pipelines within a given execution stage. This pipeline will be called once for each stage, then execute all Workers in parallel.\",\n\t\t\"activities\": [\n\t\t\t{\n\t\t\t\t\"name\": \"Get Pipelines\",\n\t\t\t\t\"description\": \"Returns all pipelines from the metadata to be executed within a given processing stage.\",\n\t\t\t\t\"type\": \"Lookup\",\n\t\t\t\t\"dependsOn\": [],\n\t\t\t\t\"policy\": {\n\t\t\t\t\t\"timeout\": \"0.00:10:00\",\n\t\t\t\t\t\"retry\": 0,\n\t\t\t\t\t\"retryIntervalInSeconds\": 30,\n\t\t\t\t\t\"secureOutput\": false,\n\t\t\t\t\t\"secureInput\": false\n\t\t\t\t},\n\t\t\t\t\"userProperties\": [],\n\t\t\t\t\"typeProperties\": {\n\t\t\t\t\t\"source\": {\n\t\t\t\t\t\t\"type\": \"AzureSqlSource\",\n\t\t\t\t\t\t\"sqlReaderStoredProcedureName\": \"[procfwk].[GetPipelinesInStage]\",\n\t\t\t\t\t\t\"storedProcedureParameters\": {\n\t\t\t\t\t\t\t\"ExecutionId\": {\n\t\t\t\t\t\t\t\t\"type\": \"Guid\",\n\t\t\t\t\t\t\t\t\"value\": {\n\t\t\t\t\t\t\t\t\t\"value\": \"@pipeline().parameters.ExecutionId\",\n\t\t\t\t\t\t\t\t\t\"type\": \"Expression\"\n\t\t\t\t\t\t\t\t}\n\t\t\t\t\t\t\t},\n\t\t\t\t\t\t\t\"StageId\": {\n\t\t\t\t\t\t\t\t\"type\": \"Int32\",\n\t\t\t\t\t\t\t\t\"value\": {\n\t\t\t\t\t\t\t\t\t\"value\": \"@pipeline().parameters.StageId\",\n\t\t\t\t\t\t\t\t\t\"type\": \"Expression\"\n\t\t\t\t\t\t\t\t}\n\t\t\t\t\t\t\t}\n\t\t\t\t\t\t},\n\t\t\t\t\t\t\"queryTimeout\": \"02:00:00\",\n\t\t\t\t\t\t\"partitionOption\": \"None\"\n\t\t\t\t\t},\n\t\t\t\t\t\"dataset\": {\n\t\t\t\t\t\t\"referenceName\": \"GetSetMetadata\",\n\t\t\t\t\t\t\"type\": \"DatasetReference\"\n\t\t\t\t\t},\n\t\t\t\t\t\"firstRowOnly\": false\n\t\t\t\t}\n\t\t\t},\n\t\t\t{\n\t\t\t\t\"name\": \"Execute Pipelines\",\n\t\t\t\t\"description\": \"Second level ForEach to run in parallel all pipelines within the stage. Items for iteration passed from the Get Pipelines lookup activity.\",\n\t\t\t\t\"type\": \"ForEach\",\n\t\t\t\t\"dependsOn\": [\n\t\t\t\t\t{\n\t\t\t\t\t\t\"activity\": \"Get Pipelines\",\n\t\t\t\t\t\t\"dependencyConditions\": [\n\t\t\t\t\t\t\t\"Succeeded\"\n\t\t\t\t\t\t]\n\t\t\t\t\t}\n\t\t\t\t],\n\t\t\t\t\"userProperties\": [],\n\t\t\t\t\"typeProperties\": {\n\t\t\t\t\t\"items\": {\n\t\t\t\t\t\t\"value\": \"@activity('Get Pipelines').output.value\",\n\t\t\t\t\t\t\"type\": \"Expression\"\n\t\t\t\t\t},\n\t\t\t\t\t\"isSequential\": false,\n\t\t\t\t\t\"batchCount\": 50,\n\t\t\t\t\t\"activities\": [\n\t\t\t\t\t\t{\n\t\t\t\t\t\t\t\"name\": \"Worker Pipeline Executor\",\n\t\t\t\t\t\t\t\"description\": \"Run the required worker pipeline and wait for its completion. Update metadata once done.\",\n\t\t\t\t\t\t\t\"type\": \"ExecutePipeline\",\n\t\t\t\t\t\t\t\"dependsOn\": [],\n\t\t\t\t\t\t\t\"userProperties\": [],\n\t\t\t\t\t\t\t\"typeProperties\": {\n\t\t\t\t\t\t\t\t\"pipeline\": {\n\t\t\t\t\t\t\t\t\t\"referenceName\": \"04-Infant\",\n\t\t\t\t\t\t\t\t\t\"type\": \"PipelineReference\"\n\t\t\t\t\t\t\t\t},\n\t\t\t\t\t\t\t\t\"waitOnCompletion\": true,\n\t\t\t\t\t\t\t\t\"parameters\": {\n\t\t\t\t\t\t\t\t\t\"executionId\": {\n\t\t\t\t\t\t\t\t\t\t\"value\": \"@pipeline().parameters.ExecutionId\",\n\t\t\t\t\t\t\t\t\t\t\"type\": \"Expression\"\n\t\t\t\t\t\t\t\t\t},\n\t\t\t\t\t\t\t\t\t\"stageId\": {\n\t\t\t\t\t\t\t\t\t\t\"value\": \"@pipeline().parameters.StageId\",\n\t\t\t\t\t\t\t\t\t\t\"type\": \"Expression\"\n\t\t\t\t\t\t\t\t\t},\n\t\t\t\t\t\t\t\t\t\"pipelineId\": {\n\t\t\t\t\t\t\t\t\t\t\"value\": \"@item().PipelineId\",\n\t\t\t\t\t\t\t\t\t\t\"type\": \"Expression\"\n\t\t\t\t\t\t\t\t\t}\n\t\t\t\t\t\t\t\t}\n\t\t\t\t\t\t\t}\n\t\t\t\t\t\t}\n\t\t\t\t\t]\n\t\t\t\t}\n\t\t\t}\n\t\t],\n\t\t\"parameters\": {\n\t\t\t\"StageId\": {\n\t\t\t\t\"type\": \"int\"\n\t\t\t},\n\t\t\t\"ExecutionId\": {\n\t\t\t\t\"type\": \"string\"\n\t\t\t}\n\t\t},\n\t\t\"folder\": {\n\t\t\t\"name\": \"_ProcFwk\"\n\t\t},\n\t\t\"annotations\": [\n\t\t\t\"procfwk\",\n\t\t\t\"Child\"\n\t\t]\n\t}\n}"
},
{
"path": "Synapse/pipeline/04-Infant.json",
"content": "{\n\t\"name\": \"04-Infant\",\n\t\"properties\": {\n\t\t\"description\": \"procfwk infant pipeline used to check when the processing pipeline called by the Child completes and passes the resulting status back to the metadata database.\",\n\t\t\"activities\": [\n\t\t\t{\n\t\t\t\t\"name\": \"Execute Worker Pipeline\",\n\t\t\t\t\"description\": \"The lowest level executor with the metadata framework to call existing processing pipelines within Data Factory. The function called will block processing and wait for an outcome.\",\n\t\t\t\t\"type\": \"AzureFunctionActivity\",\n\t\t\t\t\"dependsOn\": [\n\t\t\t\t\t{\n\t\t\t\t\t\t\"activity\": \"Log Pipeline Running\",\n\t\t\t\t\t\t\"dependencyConditions\": [\n\t\t\t\t\t\t\t\"Succeeded\"\n\t\t\t\t\t\t]\n\t\t\t\t\t},\n\t\t\t\t\t{\n\t\t\t\t\t\t\"activity\": \"Get Pipeline Params\",\n\t\t\t\t\t\t\"dependencyConditions\": [\n\t\t\t\t\t\t\t\"Succeeded\"\n\t\t\t\t\t\t]\n\t\t\t\t\t}\n\t\t\t\t],\n\t\t\t\t\"policy\": {\n\t\t\t\t\t\"timeout\": \"0.00:10:00\",\n\t\t\t\t\t\"retry\": 0,\n\t\t\t\t\t\"retryIntervalInSeconds\": 30,\n\t\t\t\t\t\"secureOutput\": false,\n\t\t\t\t\t\"secureInput\": true\n\t\t\t\t},\n\t\t\t\t\"userProperties\": [],\n\t\t\t\t\"typeProperties\": {\n\t\t\t\t\t\"functionName\": \"ExecutePipeline\",\n\t\t\t\t\t\"method\": \"POST\",\n\t\t\t\t\t\"body\": {\n\t\t\t\t\t\t\"value\": \"@concat('\\n{\\n \\\"tenantId\\\": \\\"',variables('WorkerCoreDetails')[0].tenantId,'\\\",\\n \\\"applicationId\\\": \\\"',variables('WorkerCoreDetails')[0].applicationId,'\\\",\\n \\\"authenticationKey\\\": \\\"',variables('WorkerCoreDetails')[0].authenticationKey,'\\\",\\n \\\"subscriptionId\\\": \\\"',variables('WorkerCoreDetails')[0].subscriptionId,'\\\",\\n \\\"resourceGroupName\\\": \\\"',variables('WorkerCoreDetails')[0].resourceGroupName,'\\\",\\n\\t\\\"orchestratorName\\\": \\\"',variables('WorkerCoreDetails')[0].orchestratorName,'\\\",\\n \\\"orchestratorType\\\": \\\"',variables('WorkerCoreDetails')[0].orchestratorType,'\\\",\\n \\\"pipelineName\\\": \\\"',variables('WorkerCoreDetails')[0].pipelineName,'\\\"',activity('Get Pipeline Params').output.firstRow.Params,'\\n}')\",\n\t\t\t\t\t\t\"type\": \"Expression\"\n\t\t\t\t\t}\n\t\t\t\t},\n\t\t\t\t\"linkedServiceName\": {\n\t\t\t\t\t\"referenceName\": \"FrameworkFunctions\",\n\t\t\t\t\t\"type\": \"LinkedServiceReference\"\n\t\t\t\t}\n\t\t\t},\n\t\t\t{\n\t\t\t\t\"name\": \"Get Pipeline Params\",\n\t\t\t\t\"description\": \"Returns any parameters from metadata required for the processing pipeline being called. The output can be an empty string if no parameters are required.\",\n\t\t\t\t\"type\": \"Lookup\",\n\t\t\t\t\"dependsOn\": [],\n\t\t\t\t\"policy\": {\n\t\t\t\t\t\"timeout\": \"0.00:10:00\",\n\t\t\t\t\t\"retry\": 0,\n\t\t\t\t\t\"retryIntervalInSeconds\": 30,\n\t\t\t\t\t\"secureOutput\": false,\n\t\t\t\t\t\"secureInput\": false\n\t\t\t\t},\n\t\t\t\t\"userProperties\": [],\n\t\t\t\t\"typeProperties\": {\n\t\t\t\t\t\"source\": {\n\t\t\t\t\t\t\"type\": \"AzureSqlSource\",\n\t\t\t\t\t\t\"sqlReaderStoredProcedureName\": \"[procfwk].[GetPipelineParameters]\",\n\t\t\t\t\t\t\"storedProcedureParameters\": {\n\t\t\t\t\t\t\t\"PipelineId\": {\n\t\t\t\t\t\t\t\t\"type\": \"Int32\",\n\t\t\t\t\t\t\t\t\"value\": {\n\t\t\t\t\t\t\t\t\t\"value\": \"@pipeline().parameters.pipelineId\",\n\t\t\t\t\t\t\t\t\t\"type\": \"Expression\"\n\t\t\t\t\t\t\t\t}\n\t\t\t\t\t\t\t}\n\t\t\t\t\t\t},\n\t\t\t\t\t\t\"queryTimeout\": \"02:00:00\",\n\t\t\t\t\t\t\"partitionOption\": \"None\"\n\t\t\t\t\t},\n\t\t\t\t\t\"dataset\": {\n\t\t\t\t\t\t\"referenceName\": \"GetSetMetadata\",\n\t\t\t\t\t\t\"type\": \"DatasetReference\"\n\t\t\t\t\t}\n\t\t\t\t}\n\t\t\t},\n\t\t\t{\n\t\t\t\t\"name\": \"Log Pipeline Running\",\n\t\t\t\t\"description\": \"Sets the current pipeline with a status of running within the current execution database table.\",\n\t\t\t\t\"type\": \"SqlServerStoredProcedure\",\n\t\t\t\t\"dependsOn\": [\n\t\t\t\t\t{\n\t\t\t\t\t\t\"activity\": \"Is Target Worker Validate\",\n\t\t\t\t\t\t\"dependencyConditions\": [\n\t\t\t\t\t\t\t\"Succeeded\"\n\t\t\t\t\t\t]\n\t\t\t\t\t}\n\t\t\t\t],\n\t\t\t\t\"policy\": {\n\t\t\t\t\t\"timeout\": \"0.00:10:00\",\n\t\t\t\t\t\"retry\": 0,\n\t\t\t\t\t\"retryIntervalInSeconds\": 30,\n\t\t\t\t\t\"secureOutput\": false,\n\t\t\t\t\t\"secureInput\": false\n\t\t\t\t},\n\t\t\t\t\"userProperties\": [],\n\t\t\t\t\"typeProperties\": {\n\t\t\t\t\t\"storedProcedureName\": \"[procfwk].[SetLogPipelineRunning]\",\n\t\t\t\t\t\"storedProcedureParameters\": {\n\t\t\t\t\t\t\"ExecutionId\": {\n\t\t\t\t\t\t\t\"value\": {\n\t\t\t\t\t\t\t\t\"value\": \"@pipeline().parameters.ExecutionId\",\n\t\t\t\t\t\t\t\t\"type\": \"Expression\"\n\t\t\t\t\t\t\t},\n\t\t\t\t\t\t\t\"type\": \"Guid\"\n\t\t\t\t\t\t},\n\t\t\t\t\t\t\"PipelineId\": {\n\t\t\t\t\t\t\t\"value\": {\n\t\t\t\t\t\t\t\t\"value\": \"@pipeline().parameters.pipelineId\",\n\t\t\t\t\t\t\t\t\"type\": \"Expression\"\n\t\t\t\t\t\t\t},\n\t\t\t\t\t\t\t\"type\": \"Int32\"\n\t\t\t\t\t\t},\n\t\t\t\t\t\t\"StageId\": {\n\t\t\t\t\t\t\t\"value\": {\n\t\t\t\t\t\t\t\t\"value\": \"@pipeline().parameters.StageId\",\n\t\t\t\t\t\t\t\t\"type\": \"Expression\"\n\t\t\t\t\t\t\t},\n\t\t\t\t\t\t\t\"type\": \"Int32\"\n\t\t\t\t\t\t}\n\t\t\t\t\t}\n\t\t\t\t},\n\t\t\t\t\"linkedServiceName\": {\n\t\t\t\t\t\"referenceName\": \"SupportDatabase\",\n\t\t\t\t\t\"type\": \"LinkedServiceReference\"\n\t\t\t\t}\n\t\t\t},\n\t\t\t{\n\t\t\t\t\"name\": \"Log Execute Function Activity Failure\",\n\t\t\t\t\"description\": \"Handle true failures from calling out to the Azure Function and update the current execution table accordingly so a restart can occur.\",\n\t\t\t\t\"type\": \"SqlServerStoredProcedure\",\n\t\t\t\t\"dependsOn\": [\n\t\t\t\t\t{\n\t\t\t\t\t\t\"activity\": \"Execute Worker Pipeline\",\n\t\t\t\t\t\t\"dependencyConditions\": [\n\t\t\t\t\t\t\t\"Failed\"\n\t\t\t\t\t\t]\n\t\t\t\t\t}\n\t\t\t\t],\n\t\t\t\t\"policy\": {\n\t\t\t\t\t\"timeout\": \"0.00:10:00\",\n\t\t\t\t\t\"retry\": 0,\n\t\t\t\t\t\"retryIntervalInSeconds\": 30,\n\t\t\t\t\t\"secureOutput\": false,\n\t\t\t\t\t\"secureInput\": false\n\t\t\t\t},\n\t\t\t\t\"userProperties\": [],\n\t\t\t\t\"typeProperties\": {\n\t\t\t\t\t\"storedProcedureName\": \"[procfwk].[SetLogActivityFailed]\",\n\t\t\t\t\t\"storedProcedureParameters\": {\n\t\t\t\t\t\t\"ExecutionId\": {\n\t\t\t\t\t\t\t\"value\": {\n\t\t\t\t\t\t\t\t\"value\": \"@pipeline().parameters.ExecutionId\",\n\t\t\t\t\t\t\t\t\"type\": \"Expression\"\n\t\t\t\t\t\t\t},\n\t\t\t\t\t\t\t\"type\": \"Guid\"\n\t\t\t\t\t\t},\n\t\t\t\t\t\t\"PipelineId\": {\n\t\t\t\t\t\t\t\"value\": {\n\t\t\t\t\t\t\t\t\"value\": \"@pipeline().parameters.pipelineId\",\n\t\t\t\t\t\t\t\t\"type\": \"Expression\"\n\t\t\t\t\t\t\t},\n\t\t\t\t\t\t\t\"type\": \"Int32\"\n\t\t\t\t\t\t},\n\t\t\t\t\t\t\"StageId\": {\n\t\t\t\t\t\t\t\"value\": {\n\t\t\t\t\t\t\t\t\"value\": \"@pipeline().parameters.StageId\",\n\t\t\t\t\t\t\t\t\"type\": \"Expression\"\n\t\t\t\t\t\t\t},\n\t\t\t\t\t\t\t\"type\": \"Int32\"\n\t\t\t\t\t\t},\n\t\t\t\t\t\t\"CallingActivity\": {\n\t\t\t\t\t\t\t\"value\": \"ExecuteWorkerPipeline\",\n\t\t\t\t\t\t\t\"type\": \"String\"\n\t\t\t\t\t\t}\n\t\t\t\t\t}\n\t\t\t\t},\n\t\t\t\t\"linkedServiceName\": {\n\t\t\t\t\t\"referenceName\": \"SupportDatabase\",\n\t\t\t\t\t\"type\": \"LinkedServiceReference\"\n\t\t\t\t}\n\t\t\t},\n\t\t\t{\n\t\t\t\t\"name\": \"Update Run Id\",\n\t\t\t\t\"description\": \"Provide the actual ADF run ID back to the current execution table for long term logging and alignment between the metadata other Azure monitoring tools.\",\n\t\t\t\t\"type\": \"SqlServerStoredProcedure\",\n\t\t\t\t\"dependsOn\": [\n\t\t\t\t\t{\n\t\t\t\t\t\t\"activity\": \"Set Run Id\",\n\t\t\t\t\t\t\"dependencyConditions\": [\n\t\t\t\t\t\t\t\"Succeeded\"\n\t\t\t\t\t\t]\n\t\t\t\t\t}\n\t\t\t\t],\n\t\t\t\t\"policy\": {\n\t\t\t\t\t\"timeout\": \"0.00:10:00\",\n\t\t\t\t\t\"retry\": 0,\n\t\t\t\t\t\"retryIntervalInSeconds\": 30,\n\t\t\t\t\t\"secureOutput\": false,\n\t\t\t\t\t\"secureInput\": false\n\t\t\t\t},\n\t\t\t\t\"userProperties\": [],\n\t\t\t\t\"typeProperties\": {\n\t\t\t\t\t\"storedProcedureName\": \"[procfwk].[SetLogPipelineRunId]\",\n\t\t\t\t\t\"storedProcedureParameters\": {\n\t\t\t\t\t\t\"ExecutionId\": {\n\t\t\t\t\t\t\t\"value\": {\n\t\t\t\t\t\t\t\t\"value\": \"@pipeline().parameters.ExecutionId\",\n\t\t\t\t\t\t\t\t\"type\": \"Expression\"\n\t\t\t\t\t\t\t},\n\t\t\t\t\t\t\t\"type\": \"Guid\"\n\t\t\t\t\t\t},\n\t\t\t\t\t\t\"PipelineId\": {\n\t\t\t\t\t\t\t\"value\": {\n\t\t\t\t\t\t\t\t\"value\": \"@pipeline().parameters.pipelineId\",\n\t\t\t\t\t\t\t\t\"type\": \"Expression\"\n\t\t\t\t\t\t\t},\n\t\t\t\t\t\t\t\"type\": \"Int32\"\n\t\t\t\t\t\t},\n\t\t\t\t\t\t\"RunId\": {\n\t\t\t\t\t\t\t\"value\": {\n\t\t\t\t\t\t\t\t\"value\": \"@variables('WorkerRunId')\",\n\t\t\t\t\t\t\t\t\"type\": \"Expression\"\n\t\t\t\t\t\t\t},\n\t\t\t\t\t\t\t\"type\": \"Guid\"\n\t\t\t\t\t\t},\n\t\t\t\t\t\t\"StageId\": {\n\t\t\t\t\t\t\t\"value\": {\n\t\t\t\t\t\t\t\t\"value\": \"@pipeline().parameters.StageId\",\n\t\t\t\t\t\t\t\t\"type\": \"Expression\"\n\t\t\t\t\t\t\t},\n\t\t\t\t\t\t\t\"type\": \"Int32\"\n\t\t\t\t\t\t}\n\t\t\t\t\t}\n\t\t\t\t},\n\t\t\t\t\"linkedServiceName\": {\n\t\t\t\t\t\"referenceName\": \"SupportDatabase\",\n\t\t\t\t\t\"type\": \"LinkedServiceReference\"\n\t\t\t\t}\n\t\t\t},\n\t\t\t{\n\t\t\t\t\"name\": \"Check For Alerts\",\n\t\t\t\t\"description\": \"Checks the properties tables and if any recipients in the database require alerts sending for the current pipeline ID.\",\n\t\t\t\t\"type\": \"Lookup\",\n\t\t\t\t\"dependsOn\": [\n\t\t\t\t\t{\n\t\t\t\t\t\t\"activity\": \"Update Run Id\",\n\t\t\t\t\t\t\"dependencyConditions\": [\n\t\t\t\t\t\t\t\"Succeeded\"\n\t\t\t\t\t\t]\n\t\t\t\t\t},\n\t\t\t\t\t{\n\t\t\t\t\t\t\"activity\": \"Set Pipeline Result\",\n\t\t\t\t\t\t\"dependencyConditions\": [\n\t\t\t\t\t\t\t\"Completed\"\n\t\t\t\t\t\t]\n\t\t\t\t\t}\n\t\t\t\t],\n\t\t\t\t\"policy\": {\n\t\t\t\t\t\"timeout\": \"0.00:10:00\",\n\t\t\t\t\t\"retry\": 0,\n\t\t\t\t\t\"retryIntervalInSeconds\": 30,\n\t\t\t\t\t\"secureOutput\": false,\n\t\t\t\t\t\"secureInput\": false\n\t\t\t\t},\n\t\t\t\t\"userProperties\": [],\n\t\t\t\t\"typeProperties\": {\n\t\t\t\t\t\"source\": {\n\t\t\t\t\t\t\"type\": \"AzureSqlSource\",\n\t\t\t\t\t\t\"sqlReaderStoredProcedureName\": \"[procfwk].[CheckForEmailAlerts]\",\n\t\t\t\t\t\t\"storedProcedureParameters\": {\n\t\t\t\t\t\t\t\"PipelineId\": {\n\t\t\t\t\t\t\t\t\"type\": \"Int32\",\n\t\t\t\t\t\t\t\t\"value\": {\n\t\t\t\t\t\t\t\t\t\"value\": \"@pipeline().parameters.pipelineId\",\n\t\t\t\t\t\t\t\t\t\"type\": \"Expression\"\n\t\t\t\t\t\t\t\t}\n\t\t\t\t\t\t\t}\n\t\t\t\t\t\t},\n\t\t\t\t\t\t\"queryTimeout\": \"02:00:00\",\n\t\t\t\t\t\t\"partitionOption\": \"None\"\n\t\t\t\t\t},\n\t\t\t\t\t\"dataset\": {\n\t\t\t\t\t\t\"referenceName\": \"GetSetMetadata\",\n\t\t\t\t\t\t\"type\": \"DatasetReference\"\n\t\t\t\t\t},\n\t\t\t\t\t\"firstRowOnly\": true\n\t\t\t\t}\n\t\t\t},\n\t\t\t{\n\t\t\t\t\"name\": \"Send Alerts\",\n\t\t\t\t\"description\": \"True = alerts need sending.\\nFalse = do nothing.\",\n\t\t\t\t\"type\": \"IfCondition\",\n\t\t\t\t\"dependsOn\": [\n\t\t\t\t\t{\n\t\t\t\t\t\t\"activity\": \"Check For Alerts\",\n\t\t\t\t\t\t\"dependencyConditions\": [\n\t\t\t\t\t\t\t\"Succeeded\"\n\t\t\t\t\t\t]\n\t\t\t\t\t}\n\t\t\t\t],\n\t\t\t\t\"userProperties\": [],\n\t\t\t\t\"typeProperties\": {\n\t\t\t\t\t\"expression\": {\n\t\t\t\t\t\t\"value\": \"@activity('Check For Alerts').output.firstRow.SendAlerts\",\n\t\t\t\t\t\t\"type\": \"Expression\"\n\t\t\t\t\t},\n\t\t\t\t\t\"ifTrueActivities\": [\n\t\t\t\t\t\t{\n\t\t\t\t\t\t\t\"name\": \"Get Email Parts\",\n\t\t\t\t\t\t\t\"description\": \"Return all required content from the metadata database to send an email alerting using the procfwk. The lookup returns the exact content for the function body request.\",\n\t\t\t\t\t\t\t\"type\": \"Lookup\",\n\t\t\t\t\t\t\t\"dependsOn\": [],\n\t\t\t\t\t\t\t\"policy\": {\n\t\t\t\t\t\t\t\t\"timeout\": \"0.00:10:00\",\n\t\t\t\t\t\t\t\t\"retry\": 0,\n\t\t\t\t\t\t\t\t\"retryIntervalInSeconds\": 30,\n\t\t\t\t\t\t\t\t\"secureOutput\": true,\n\t\t\t\t\t\t\t\t\"secureInput\": false\n\t\t\t\t\t\t\t},\n\t\t\t\t\t\t\t\"userProperties\": [],\n\t\t\t\t\t\t\t\"typeProperties\": {\n\t\t\t\t\t\t\t\t\"source\": {\n\t\t\t\t\t\t\t\t\t\"type\": \"AzureSqlSource\",\n\t\t\t\t\t\t\t\t\t\"sqlReaderStoredProcedureName\": \"[procfwk].[GetEmailAlertParts]\",\n\t\t\t\t\t\t\t\t\t\"storedProcedureParameters\": {\n\t\t\t\t\t\t\t\t\t\t\"PipelineId\": {\n\t\t\t\t\t\t\t\t\t\t\t\"type\": \"Int32\",\n\t\t\t\t\t\t\t\t\t\t\t\"value\": {\n\t\t\t\t\t\t\t\t\t\t\t\t\"value\": \"@pipeline().parameters.pipelineId\",\n\t\t\t\t\t\t\t\t\t\t\t\t\"type\": \"Expression\"\n\t\t\t\t\t\t\t\t\t\t\t}\n\t\t\t\t\t\t\t\t\t\t}\n\t\t\t\t\t\t\t\t\t},\n\t\t\t\t\t\t\t\t\t\"queryTimeout\": \"02:00:00\",\n\t\t\t\t\t\t\t\t\t\"partitionOption\": \"None\"\n\t\t\t\t\t\t\t\t},\n\t\t\t\t\t\t\t\t\"dataset\": {\n\t\t\t\t\t\t\t\t\t\"referenceName\": \"GetSetMetadata\",\n\t\t\t\t\t\t\t\t\t\"type\": \"DatasetReference\"\n\t\t\t\t\t\t\t\t},\n\t\t\t\t\t\t\t\t\"firstRowOnly\": true\n\t\t\t\t\t\t\t}\n\t\t\t\t\t\t},\n\t\t\t\t\t\t{\n\t\t\t\t\t\t\t\"name\": \"Call Email Sender\",\n\t\t\t\t\t\t\t\"description\": \"Pass off email request to Utils Send Email pipeline.\",\n\t\t\t\t\t\t\t\"type\": \"ExecutePipeline\",\n\t\t\t\t\t\t\t\"dependsOn\": [\n\t\t\t\t\t\t\t\t{\n\t\t\t\t\t\t\t\t\t\"activity\": \"Get Email Parts\",\n\t\t\t\t\t\t\t\t\t\"dependencyConditions\": [\n\t\t\t\t\t\t\t\t\t\t\"Succeeded\"\n\t\t\t\t\t\t\t\t\t]\n\t\t\t\t\t\t\t\t}\n\t\t\t\t\t\t\t],\n\t\t\t\t\t\t\t\"userProperties\": [],\n\t\t\t\t\t\t\t\"typeProperties\": {\n\t\t\t\t\t\t\t\t\"pipeline\": {\n\t\t\t\t\t\t\t\t\t\"referenceName\": \"Email Sender\",\n\t\t\t\t\t\t\t\t\t\"type\": \"PipelineReference\"\n\t\t\t\t\t\t\t\t},\n\t\t\t\t\t\t\t\t\"waitOnCompletion\": true,\n\t\t\t\t\t\t\t\t\"parameters\": {\n\t\t\t\t\t\t\t\t\t\"Recipients\": {\n\t\t\t\t\t\t\t\t\t\t\"value\": \"@activity('Get Email Parts').output.firstRow.emailRecipients\",\n\t\t\t\t\t\t\t\t\t\t\"type\": \"Expression\"\n\t\t\t\t\t\t\t\t\t},\n\t\t\t\t\t\t\t\t\t\"CcRecipients\": {\n\t\t\t\t\t\t\t\t\t\t\"value\": \"@activity('Get Email Parts').output.firstRow.emailCcRecipients\",\n\t\t\t\t\t\t\t\t\t\t\"type\": \"Expression\"\n\t\t\t\t\t\t\t\t\t},\n\t\t\t\t\t\t\t\t\t\"BccRecipients\": {\n\t\t\t\t\t\t\t\t\t\t\"value\": \"@activity('Get Email Parts').output.firstRow.emailBccRecipients\",\n\t\t\t\t\t\t\t\t\t\t\"type\": \"Expression\"\n\t\t\t\t\t\t\t\t\t},\n\t\t\t\t\t\t\t\t\t\"Subject\": {\n\t\t\t\t\t\t\t\t\t\t\"value\": \"@activity('Get Email Parts').output.firstRow.emailSubject\",\n\t\t\t\t\t\t\t\t\t\t\"type\": \"Expression\"\n\t\t\t\t\t\t\t\t\t},\n\t\t\t\t\t\t\t\t\t\"Body\": {\n\t\t\t\t\t\t\t\t\t\t\"value\": \"@activity('Get Email Parts').output.firstRow.emailBody\",\n\t\t\t\t\t\t\t\t\t\t\"type\": \"Expression\"\n\t\t\t\t\t\t\t\t\t},\n\t\t\t\t\t\t\t\t\t\"Importance\": {\n\t\t\t\t\t\t\t\t\t\t\"value\": \"@activity('Get Email Parts').output.firstRow.emailImportance\",\n\t\t\t\t\t\t\t\t\t\t\"type\": \"Expression\"\n\t\t\t\t\t\t\t\t\t}\n\t\t\t\t\t\t\t\t}\n\t\t\t\t\t\t\t}\n\t\t\t\t\t\t}\n\t\t\t\t\t]\n\t\t\t\t}\n\t\t\t},\n\t\t\t{\n\t\t\t\t\"name\": \"Wait Until Pipeline Completes\",\n\t\t\t\t\"description\": \"Loops until the Worker pipeline called completes.\\n\\nSimple status:\\n- Running = new iteration.\\n- Done = break.\",\n\t\t\t\t\"type\": \"Until\",\n\t\t\t\t\"dependsOn\": [\n\t\t\t\t\t{\n\t\t\t\t\t\t\"activity\": \"Get Wait Duration\",\n\t\t\t\t\t\t\"dependencyConditions\": [\n\t\t\t\t\t\t\t\"Succeeded\"\n\t\t\t\t\t\t]\n\t\t\t\t\t},\n\t\t\t\t\t{\n\t\t\t\t\t\t\"activity\": \"Execute Worker Pipeline\",\n\t\t\t\t\t\t\"dependencyConditions\": [\n\t\t\t\t\t\t\t\"Succeeded\"\n\t\t\t\t\t\t]\n\t\t\t\t\t},\n\t\t\t\t\t{\n\t\t\t\t\t\t\"activity\": \"Set Run Id\",\n\t\t\t\t\t\t\"dependencyConditions\": [\n\t\t\t\t\t\t\t\"Succeeded\"\n\t\t\t\t\t\t]\n\t\t\t\t\t}\n\t\t\t\t],\n\t\t\t\t\"userProperties\": [],\n\t\t\t\t\"typeProperties\": {\n\t\t\t\t\t\"expression\": {\n\t\t\t\t\t\t\"value\": \"@variables('WorkerPipelineState')\",\n\t\t\t\t\t\t\"type\": \"Expression\"\n\t\t\t\t\t},\n\t\t\t\t\t\"activities\": [\n\t\t\t\t\t\t{\n\t\t\t\t\t\t\t\"name\": \"Get Worker Pipeline Status\",\n\t\t\t\t\t\t\t\"description\": \"Checks the status of a given processing pipeline and provides the value for the downstream framework activities to act upon.\",\n\t\t\t\t\t\t\t\"type\": \"AzureFunctionActivity\",\n\t\t\t\t\t\t\t\"dependsOn\": [],\n\t\t\t\t\t\t\t\"policy\": {\n\t\t\t\t\t\t\t\t\"timeout\": \"0.00:10:00\",\n\t\t\t\t\t\t\t\t\"retry\": 0,\n\t\t\t\t\t\t\t\t\"retryIntervalInSeconds\": 30,\n\t\t\t\t\t\t\t\t\"secureOutput\": false,\n\t\t\t\t\t\t\t\t\"secureInput\": true\n\t\t\t\t\t\t\t},\n\t\t\t\t\t\t\t\"userProperties\": [],\n\t\t\t\t\t\t\t\"typeProperties\": {\n\t\t\t\t\t\t\t\t\"functionName\": \"CheckPipelineStatus\",\n\t\t\t\t\t\t\t\t\"method\": \"POST\",\n\t\t\t\t\t\t\t\t\"body\": {\n\t\t\t\t\t\t\t\t\t\"value\": \"@concat('\\n{\\n \\\"tenantId\\\": \\\"',variables('WorkerCoreDetails')[0].tenantId,'\\\",\\n \\\"applicationId\\\": \\\"',variables('WorkerCoreDetails')[0].applicationId,'\\\",\\n \\\"authenticationKey\\\": \\\"',variables('WorkerCoreDetails')[0].authenticationKey,'\\\",\\n \\\"subscriptionId\\\": \\\"',variables('WorkerCoreDetails')[0].subscriptionId,'\\\",\\n \\\"resourceGroupName\\\": \\\"',variables('WorkerCoreDetails')[0].resourceGroupName,'\\\",\\n\\t\\\"orchestratorName\\\": \\\"',variables('WorkerCoreDetails')[0].orchestratorName,'\\\",\\n \\\"orchestratorType\\\": \\\"',variables('WorkerCoreDetails')[0].orchestratorType,'\\\",\\n \\\"pipelineName\\\": \\\"',variables('WorkerCoreDetails')[0].pipelineName,'\\\",\\n \\\"runId\\\": \\\"',variables('WorkerRunId'),'\\\"\\n}')\",\n\t\t\t\t\t\t\t\t\t\"type\": \"Expression\"\n\t\t\t\t\t\t\t\t}\n\t\t\t\t\t\t\t},\n\t\t\t\t\t\t\t\"linkedServiceName\": {\n\t\t\t\t\t\t\t\t\"referenceName\": \"FrameworkFunctions\",\n\t\t\t\t\t\t\t\t\"type\": \"LinkedServiceReference\"\n\t\t\t\t\t\t\t}\n\t\t\t\t\t\t},\n\t\t\t\t\t\t{\n\t\t\t\t\t\t\t\"name\": \"Wait If Running\",\n\t\t\t\t\t\t\t\"description\": \"True = Do nothing.\\nFalse = Wait, before the next iteration.\",\n\t\t\t\t\t\t\t\"type\": \"IfCondition\",\n\t\t\t\t\t\t\t\"dependsOn\": [\n\t\t\t\t\t\t\t\t{\n\t\t\t\t\t\t\t\t\t\"activity\": \"Set Worker State\",\n\t\t\t\t\t\t\t\t\t\"dependencyConditions\": [\n\t\t\t\t\t\t\t\t\t\t\"Succeeded\"\n\t\t\t\t\t\t\t\t\t]\n\t\t\t\t\t\t\t\t}\n\t\t\t\t\t\t\t],\n\t\t\t\t\t\t\t\"userProperties\": [],\n\t\t\t\t\t\t\t\"typeProperties\": {\n\t\t\t\t\t\t\t\t\"expression\": {\n\t\t\t\t\t\t\t\t\t\"value\": \"@variables('WorkerPipelineState')\",\n\t\t\t\t\t\t\t\t\t\"type\": \"Expression\"\n\t\t\t\t\t\t\t\t},\n\t\t\t\t\t\t\t\t\"ifFalseActivities\": [\n\t\t\t\t\t\t\t\t\t{\n\t\t\t\t\t\t\t\t\t\t\"name\": \"Wait for Pipeline\",\n\t\t\t\t\t\t\t\t\t\t\"description\": \"The processing pipeline is still running so Wait before checking its status again.\",\n\t\t\t\t\t\t\t\t\t\t\"type\": \"Wait\",\n\t\t\t\t\t\t\t\t\t\t\"dependsOn\": [],\n\t\t\t\t\t\t\t\t\t\t\"userProperties\": [],\n\t\t\t\t\t\t\t\t\t\t\"typeProperties\": {\n\t\t\t\t\t\t\t\t\t\t\t\"waitTimeInSeconds\": {\n\t\t\t\t\t\t\t\t\t\t\t\t\"value\": \"@activity('Get Wait Duration').output.firstRow.PropertyValue\",\n\t\t\t\t\t\t\t\t\t\t\t\t\"type\": \"Expression\"\n\t\t\t\t\t\t\t\t\t\t\t}\n\t\t\t\t\t\t\t\t\t\t}\n\t\t\t\t\t\t\t\t\t}\n\t\t\t\t\t\t\t\t]\n\t\t\t\t\t\t\t}\n\t\t\t\t\t\t},\n\t\t\t\t\t\t{\n\t\t\t\t\t\t\t\"name\": \"Set Last Check DateTime\",\n\t\t\t\t\t\t\t\"description\": \"Update the current execution table with a date time from when the Worker pipeline status was last checked as part of the Until iterations.\",\n\t\t\t\t\t\t\t\"type\": \"SqlServerStoredProcedure\",\n\t\t\t\t\t\t\t\"dependsOn\": [\n\t\t\t\t\t\t\t\t{\n\t\t\t\t\t\t\t\t\t\"activity\": \"Get Worker Pipeline Status\",\n\t\t\t\t\t\t\t\t\t\"dependencyConditions\": [\n\t\t\t\t\t\t\t\t\t\t\"Succeeded\"\n\t\t\t\t\t\t\t\t\t]\n\t\t\t\t\t\t\t\t}\n\t\t\t\t\t\t\t],\n\t\t\t\t\t\t\t\"policy\": {\n\t\t\t\t\t\t\t\t\"timeout\": \"0.00:10:00\",\n\t\t\t\t\t\t\t\t\"retry\": 0,\n\t\t\t\t\t\t\t\t\"retryIntervalInSeconds\": 30,\n\t\t\t\t\t\t\t\t\"secureOutput\": false,\n\t\t\t\t\t\t\t\t\"secureInput\": false\n\t\t\t\t\t\t\t},\n\t\t\t\t\t\t\t\"userProperties\": [],\n\t\t\t\t\t\t\t\"typeProperties\": {\n\t\t\t\t\t\t\t\t\"storedProcedureName\": \"[procfwk].[SetLogPipelineLastStatusCheck]\",\n\t\t\t\t\t\t\t\t\"storedProcedureParameters\": {\n\t\t\t\t\t\t\t\t\t\"ExecutionId\": {\n\t\t\t\t\t\t\t\t\t\t\"value\": {\n\t\t\t\t\t\t\t\t\t\t\t\"value\": \"@pipeline().parameters.executionId\",\n\t\t\t\t\t\t\t\t\t\t\t\"type\": \"Expression\"\n\t\t\t\t\t\t\t\t\t\t},\n\t\t\t\t\t\t\t\t\t\t\"type\": \"Guid\"\n\t\t\t\t\t\t\t\t\t},\n\t\t\t\t\t\t\t\t\t\"PipelineId\": {\n\t\t\t\t\t\t\t\t\t\t\"value\": {\n\t\t\t\t\t\t\t\t\t\t\t\"value\": \"@pipeline().parameters.pipelineId\",\n\t\t\t\t\t\t\t\t\t\t\t\"type\": \"Expression\"\n\t\t\t\t\t\t\t\t\t\t},\n\t\t\t\t\t\t\t\t\t\t\"type\": \"Int32\"\n\t\t\t\t\t\t\t\t\t},\n\t\t\t\t\t\t\t\t\t\"StageId\": {\n\t\t\t\t\t\t\t\t\t\t\"value\": {\n\t\t\t\t\t\t\t\t\t\t\t\"value\": \"@pipeline().parameters.stageId\",\n\t\t\t\t\t\t\t\t\t\t\t\"type\": \"Expression\"\n\t\t\t\t\t\t\t\t\t\t},\n\t\t\t\t\t\t\t\t\t\t\"type\": \"Int32\"\n\t\t\t\t\t\t\t\t\t}\n\t\t\t\t\t\t\t\t}\n\t\t\t\t\t\t\t},\n\t\t\t\t\t\t\t\"linkedServiceName\": {\n\t\t\t\t\t\t\t\t\"referenceName\": \"SupportDatabase\",\n\t\t\t\t\t\t\t\t\"type\": \"LinkedServiceReference\"\n\t\t\t\t\t\t\t}\n\t\t\t\t\t\t},\n\t\t\t\t\t\t{\n\t\t\t\t\t\t\t\"name\": \"Log Check Function Activity Failure\",\n\t\t\t\t\t\t\t\"description\": \"Report to the current execution table that the framework pipeline activity has failed. This failure is outside of the scope of the framework and is probably related to a wider platform problem.\",\n\t\t\t\t\t\t\t\"type\": \"SqlServerStoredProcedure\",\n\t\t\t\t\t\t\t\"dependsOn\": [\n\t\t\t\t\t\t\t\t{\n\t\t\t\t\t\t\t\t\t\"activity\": \"Get Worker Pipeline Status\",\n\t\t\t\t\t\t\t\t\t\"dependencyConditions\": [\n\t\t\t\t\t\t\t\t\t\t\"Failed\"\n\t\t\t\t\t\t\t\t\t]\n\t\t\t\t\t\t\t\t}\n\t\t\t\t\t\t\t],\n\t\t\t\t\t\t\t\"policy\": {\n\t\t\t\t\t\t\t\t\"timeout\": \"0.00:10:00\",\n\t\t\t\t\t\t\t\t\"retry\": 0,\n\t\t\t\t\t\t\t\t\"retryIntervalInSeconds\": 30,\n\t\t\t\t\t\t\t\t\"secureOutput\": false,\n\t\t\t\t\t\t\t\t\"secureInput\": false\n\t\t\t\t\t\t\t},\n\t\t\t\t\t\t\t\"userProperties\": [],\n\t\t\t\t\t\t\t\"typeProperties\": {\n\t\t\t\t\t\t\t\t\"storedProcedureName\": \"[procfwk].[SetLogActivityFailed]\",\n\t\t\t\t\t\t\t\t\"storedProcedureParameters\": {\n\t\t\t\t\t\t\t\t\t\"CallingActivity\": {\n\t\t\t\t\t\t\t\t\t\t\"value\": \"GetWorkerPipelineStatus\",\n\t\t\t\t\t\t\t\t\t\t\"type\": \"String\"\n\t\t\t\t\t\t\t\t\t},\n\t\t\t\t\t\t\t\t\t\"ExecutionId\": {\n\t\t\t\t\t\t\t\t\t\t\"value\": {\n\t\t\t\t\t\t\t\t\t\t\t\"value\": \"@pipeline().parameters.executionId\",\n\t\t\t\t\t\t\t\t\t\t\t\"type\": \"Expression\"\n\t\t\t\t\t\t\t\t\t\t},\n\t\t\t\t\t\t\t\t\t\t\"type\": \"Guid\"\n\t\t\t\t\t\t\t\t\t},\n\t\t\t\t\t\t\t\t\t\"PipelineId\": {\n\t\t\t\t\t\t\t\t\t\t\"value\": {\n\t\t\t\t\t\t\t\t\t\t\t\"value\": \"@pipeline().parameters.pipelineId\",\n\t\t\t\t\t\t\t\t\t\t\t\"type\": \"Expression\"\n\t\t\t\t\t\t\t\t\t\t},\n\t\t\t\t\t\t\t\t\t\t\"type\": \"Int32\"\n\t\t\t\t\t\t\t\t\t},\n\t\t\t\t\t\t\t\t\t\"StageId\": {\n\t\t\t\t\t\t\t\t\t\t\"value\": {\n\t\t\t\t\t\t\t\t\t\t\t\"value\": \"@pipeline().parameters.stageId\",\n\t\t\t\t\t\t\t\t\t\t\t\"type\": \"Expression\"\n\t\t\t\t\t\t\t\t\t\t},\n\t\t\t\t\t\t\t\t\t\t\"type\": \"Int32\"\n\t\t\t\t\t\t\t\t\t}\n\t\t\t\t\t\t\t\t}\n\t\t\t\t\t\t\t},\n\t\t\t\t\t\t\t\"linkedServiceName\": {\n\t\t\t\t\t\t\t\t\"referenceName\": \"SupportDatabase\",\n\t\t\t\t\t\t\t\t\"type\": \"LinkedServiceReference\"\n\t\t\t\t\t\t\t}\n\t\t\t\t\t\t},\n\t\t\t\t\t\t{\n\t\t\t\t\t\t\t\"name\": \"Set Worker State\",\n\t\t\t\t\t\t\t\"description\": \"Set the bool state of the Worker pipeline to be used by the Until and If expressions. True = Complete, False = Running.\",\n\t\t\t\t\t\t\t\"type\": \"SetVariable\",\n\t\t\t\t\t\t\t\"dependsOn\": [\n\t\t\t\t\t\t\t\t{\n\t\t\t\t\t\t\t\t\t\"activity\": \"Get Worker Pipeline Status\",\n\t\t\t\t\t\t\t\t\t\"dependencyConditions\": [\n\t\t\t\t\t\t\t\t\t\t\"Succeeded\"\n\t\t\t\t\t\t\t\t\t]\n\t\t\t\t\t\t\t\t}\n\t\t\t\t\t\t\t],\n\t\t\t\t\t\t\t\"userProperties\": [],\n\t\t\t\t\t\t\t\"typeProperties\": {\n\t\t\t\t\t\t\t\t\"variableName\": \"WorkerPipelineState\",\n\t\t\t\t\t\t\t\t\"value\": {\n\t\t\t\t\t\t\t\t\t\"value\": \"@equals('Complete',activity('Get Worker Pipeline Status').output.SimpleStatus)\",\n\t\t\t\t\t\t\t\t\t\"type\": \"Expression\"\n\t\t\t\t\t\t\t\t}\n\t\t\t\t\t\t\t}\n\t\t\t\t\t\t}\n\t\t\t\t\t],\n\t\t\t\t\t\"timeout\": \"0.00:10:00\"\n\t\t\t\t}\n\t\t\t},\n\t\t\t{\n\t\t\t\t\"name\": \"Set Pipeline Result\",\n\t\t\t\t\"description\": \"Receives the outcome from the function execution for a given processing pipeline and updates the current execution table with different pipelines status values depending on the result (case).\",\n\t\t\t\t\"type\": \"Switch\",\n\t\t\t\t\"dependsOn\": [\n\t\t\t\t\t{\n\t\t\t\t\t\t\"activity\": \"Wait Until Pipeline Completes\",\n\t\t\t\t\t\t\"dependencyConditions\": [\n\t\t\t\t\t\t\t\"Completed\"\n\t\t\t\t\t\t]\n\t\t\t\t\t}\n\t\t\t\t],\n\t\t\t\t\"userProperties\": [],\n\t\t\t\t\"typeProperties\": {\n\t\t\t\t\t\"on\": {\n\t\t\t\t\t\t\"value\": \"@activity('Get Worker Pipeline Status').output.ActualStatus\",\n\t\t\t\t\t\t\"type\": \"Expression\"\n\t\t\t\t\t},\n\t\t\t\t\t\"cases\": [\n\t\t\t\t\t\t{\n\t\t\t\t\t\t\t\"value\": \"Succeeded\",\n\t\t\t\t\t\t\t\"activities\": [\n\t\t\t\t\t\t\t\t{\n\t\t\t\t\t\t\t\t\t\"name\": \"Pipeline Status Succeeded\",\n\t\t\t\t\t\t\t\t\t\"description\": \"Updates the current execution table with a pipeline status of success if the function outcome is succeeded.\",\n\t\t\t\t\t\t\t\t\t\"type\": \"SqlServerStoredProcedure\",\n\t\t\t\t\t\t\t\t\t\"dependsOn\": [],\n\t\t\t\t\t\t\t\t\t\"policy\": {\n\t\t\t\t\t\t\t\t\t\t\"timeout\": \"0.00:10:00\",\n\t\t\t\t\t\t\t\t\t\t\"retry\": 0,\n\t\t\t\t\t\t\t\t\t\t\"retryIntervalInSeconds\": 30,\n\t\t\t\t\t\t\t\t\t\t\"secureOutput\": false,\n\t\t\t\t\t\t\t\t\t\t\"secureInput\": false\n\t\t\t\t\t\t\t\t\t},\n\t\t\t\t\t\t\t\t\t\"userProperties\": [],\n\t\t\t\t\t\t\t\t\t\"typeProperties\": {\n\t\t\t\t\t\t\t\t\t\t\"storedProcedureName\": \"[procfwk].[SetLogPipelineSuccess]\",\n\t\t\t\t\t\t\t\t\t\t\"storedProcedureParameters\": {\n\t\t\t\t\t\t\t\t\t\t\t\"ExecutionId\": {\n\t\t\t\t\t\t\t\t\t\t\t\t\"value\": {\n\t\t\t\t\t\t\t\t\t\t\t\t\t\"value\": \"@pipeline().parameters.executionId\",\n\t\t\t\t\t\t\t\t\t\t\t\t\t\"type\": \"Expression\"\n\t\t\t\t\t\t\t\t\t\t\t\t},\n\t\t\t\t\t\t\t\t\t\t\t\t\"type\": \"Guid\"\n\t\t\t\t\t\t\t\t\t\t\t},\n\t\t\t\t\t\t\t\t\t\t\t\"PipelineId\": {\n\t\t\t\t\t\t\t\t\t\t\t\t\"value\": {\n\t\t\t\t\t\t\t\t\t\t\t\t\t\"value\": \"@pipeline().parameters.pipelineId\",\n\t\t\t\t\t\t\t\t\t\t\t\t\t\"type\": \"Expression\"\n\t\t\t\t\t\t\t\t\t\t\t\t},\n\t\t\t\t\t\t\t\t\t\t\t\t\"type\": \"Int32\"\n\t\t\t\t\t\t\t\t\t\t\t},\n\t\t\t\t\t\t\t\t\t\t\t\"StageId\": {\n\t\t\t\t\t\t\t\t\t\t\t\t\"value\": {\n\t\t\t\t\t\t\t\t\t\t\t\t\t\"value\": \"@pipeline().parameters.stageId\",\n\t\t\t\t\t\t\t\t\t\t\t\t\t\"type\": \"Expression\"\n\t\t\t\t\t\t\t\t\t\t\t\t},\n\t\t\t\t\t\t\t\t\t\t\t\t\"type\": \"Int32\"\n\t\t\t\t\t\t\t\t\t\t\t}\n\t\t\t\t\t\t\t\t\t\t}\n\t\t\t\t\t\t\t\t\t},\n\t\t\t\t\t\t\t\t\t\"linkedServiceName\": {\n\t\t\t\t\t\t\t\t\t\t\"referenceName\": \"SupportDatabase\",\n\t\t\t\t\t\t\t\t\t\t\"type\": \"LinkedServiceReference\"\n\t\t\t\t\t\t\t\t\t}\n\t\t\t\t\t\t\t\t}\n\t\t\t\t\t\t\t]\n\t\t\t\t\t\t},\n\t\t\t\t\t\t{\n\t\t\t\t\t\t\t\"value\": \"Failed\",\n\t\t\t\t\t\t\t\"activities\": [\n\t\t\t\t\t\t\t\t{\n\t\t\t\t\t\t\t\t\t\"name\": \"Pipeline Status Failed\",\n\t\t\t\t\t\t\t\t\t\"description\": \"Updates the current execution table with a pipeline status of failed if the function outcome is failed. Also blocks pipelines in the downstream execution stage.\",\n\t\t\t\t\t\t\t\t\t\"type\": \"SqlServerStoredProcedure\",\n\t\t\t\t\t\t\t\t\t\"dependsOn\": [],\n\t\t\t\t\t\t\t\t\t\"policy\": {\n\t\t\t\t\t\t\t\t\t\t\"timeout\": \"0.00:10:00\",\n\t\t\t\t\t\t\t\t\t\t\"retry\": 0,\n\t\t\t\t\t\t\t\t\t\t\"retryIntervalInSeconds\": 30,\n\t\t\t\t\t\t\t\t\t\t\"secureOutput\": false,\n\t\t\t\t\t\t\t\t\t\t\"secureInput\": false\n\t\t\t\t\t\t\t\t\t},\n\t\t\t\t\t\t\t\t\t\"userProperties\": [],\n\t\t\t\t\t\t\t\t\t\"typeProperties\": {\n\t\t\t\t\t\t\t\t\t\t\"storedProcedureName\": \"[procfwk].[SetLogPipelineFailed]\",\n\t\t\t\t\t\t\t\t\t\t\"storedProcedureParameters\": {\n\t\t\t\t\t\t\t\t\t\t\t\"ExecutionId\": {\n\t\t\t\t\t\t\t\t\t\t\t\t\"value\": {\n\t\t\t\t\t\t\t\t\t\t\t\t\t\"value\": \"@pipeline().parameters.executionId\",\n\t\t\t\t\t\t\t\t\t\t\t\t\t\"type\": \"Expression\"\n\t\t\t\t\t\t\t\t\t\t\t\t},\n\t\t\t\t\t\t\t\t\t\t\t\t\"type\": \"Guid\"\n\t\t\t\t\t\t\t\t\t\t\t},\n\t\t\t\t\t\t\t\t\t\t\t\"PipelineId\": {\n\t\t\t\t\t\t\t\t\t\t\t\t\"value\": {\n\t\t\t\t\t\t\t\t\t\t\t\t\t\"value\": \"@pipeline().parameters.pipelineId\",\n\t\t\t\t\t\t\t\t\t\t\t\t\t\"type\": \"Expression\"\n\t\t\t\t\t\t\t\t\t\t\t\t},\n\t\t\t\t\t\t\t\t\t\t\t\t\"type\": \"Int32\"\n\t\t\t\t\t\t\t\t\t\t\t},\n\t\t\t\t\t\t\t\t\t\t\t\"RunId\": {\n\t\t\t\t\t\t\t\t\t\t\t\t\"value\": {\n\t\t\t\t\t\t\t\t\t\t\t\t\t\"value\": \"@variables('WorkerRunId')\",\n\t\t\t\t\t\t\t\t\t\t\t\t\t\"type\": \"Expression\"\n\t\t\t\t\t\t\t\t\t\t\t\t},\n\t\t\t\t\t\t\t\t\t\t\t\t\"type\": \"Guid\"\n\t\t\t\t\t\t\t\t\t\t\t},\n\t\t\t\t\t\t\t\t\t\t\t\"StageId\": {\n\t\t\t\t\t\t\t\t\t\t\t\t\"value\": {\n\t\t\t\t\t\t\t\t\t\t\t\t\t\"value\": \"@pipeline().parameters.stageId\",\n\t\t\t\t\t\t\t\t\t\t\t\t\t\"type\": \"Expression\"\n\t\t\t\t\t\t\t\t\t\t\t\t},\n\t\t\t\t\t\t\t\t\t\t\t\t\"type\": \"Int32\"\n\t\t\t\t\t\t\t\t\t\t\t}\n\t\t\t\t\t\t\t\t\t\t}\n\t\t\t\t\t\t\t\t\t},\n\t\t\t\t\t\t\t\t\t\"linkedServiceName\": {\n\t\t\t\t\t\t\t\t\t\t\"referenceName\": \"SupportDatabase\",\n\t\t\t\t\t\t\t\t\t\t\"type\": \"LinkedServiceReference\"\n\t\t\t\t\t\t\t\t\t}\n\t\t\t\t\t\t\t\t},\n\t\t\t\t\t\t\t\t{\n\t\t\t\t\t\t\t\t\t\"name\": \"Get Worker Pipeline Error Details\",\n\t\t\t\t\t\t\t\t\t\"description\": \"Get the activity error details for the run ID of the worker pipeline called. Returns an array of all errors.\",\n\t\t\t\t\t\t\t\t\t\"type\": \"AzureFunctionActivity\",\n\t\t\t\t\t\t\t\t\t\"dependsOn\": [],\n\t\t\t\t\t\t\t\t\t\"policy\": {\n\t\t\t\t\t\t\t\t\t\t\"timeout\": \"0.00:10:00\",\n\t\t\t\t\t\t\t\t\t\t\"retry\": 0,\n\t\t\t\t\t\t\t\t\t\t\"retryIntervalInSeconds\": 30,\n\t\t\t\t\t\t\t\t\t\t\"secureOutput\": false,\n\t\t\t\t\t\t\t\t\t\t\"secureInput\": true\n\t\t\t\t\t\t\t\t\t},\n\t\t\t\t\t\t\t\t\t\"userProperties\": [],\n\t\t\t\t\t\t\t\t\t\"typeProperties\": {\n\t\t\t\t\t\t\t\t\t\t\"functionName\": \"GetActivityErrors\",\n\t\t\t\t\t\t\t\t\t\t\"method\": \"POST\",\n\t\t\t\t\t\t\t\t\t\t\"body\": {\n\t\t\t\t\t\t\t\t\t\t\t\"value\": \"@concat('\\n{\\n \\\"tenantId\\\": \\\"',variables('WorkerCoreDetails')[0].tenantId,'\\\",\\n \\\"applicationId\\\": \\\"',variables('WorkerCoreDetails')[0].applicationId,'\\\",\\n \\\"authenticationKey\\\": \\\"',variables('WorkerCoreDetails')[0].authenticationKey,'\\\",\\n \\\"subscriptionId\\\": \\\"',variables('WorkerCoreDetails')[0].subscriptionId,'\\\",\\n \\\"resourceGroupName\\\": \\\"',variables('WorkerCoreDetails')[0].resourceGroupName,'\\\",\\n\\t\\\"orchestratorName\\\": \\\"',variables('WorkerCoreDetails')[0].orchestratorName,'\\\",\\n \\\"orchestratorType\\\": \\\"',variables('WorkerCoreDetails')[0].orchestratorType,'\\\",\\n \\\"pipelineName\\\": \\\"',variables('WorkerCoreDetails')[0].pipelineName,'\\\",\\n \\\"runId\\\": \\\"',variables('WorkerRunId'),'\\\"\\n}')\",\n\t\t\t\t\t\t\t\t\t\t\t\"type\": \"Expression\"\n\t\t\t\t\t\t\t\t\t\t}\n\t\t\t\t\t\t\t\t\t},\n\t\t\t\t\t\t\t\t\t\"linkedServiceName\": {\n\t\t\t\t\t\t\t\t\t\t\"referenceName\": \"FrameworkFunctions\",\n\t\t\t\t\t\t\t\t\t\t\"type\": \"LinkedServiceReference\"\n\t\t\t\t\t\t\t\t\t}\n\t\t\t\t\t\t\t\t},\n\t\t\t\t\t\t\t\t{\n\t\t\t\t\t\t\t\t\t\"name\": \"Log Error Details\",\n\t\t\t\t\t\t\t\t\t\"description\": \"Parses pipeline error details and persists them to the metadata database error log table.\",\n\t\t\t\t\t\t\t\t\t\"type\": \"SqlServerStoredProcedure\",\n\t\t\t\t\t\t\t\t\t\"dependsOn\": [\n\t\t\t\t\t\t\t\t\t\t{\n\t\t\t\t\t\t\t\t\t\t\t\"activity\": \"Get Worker Pipeline Error Details\",\n\t\t\t\t\t\t\t\t\t\t\t\"dependencyConditions\": [\n\t\t\t\t\t\t\t\t\t\t\t\t\"Succeeded\"\n\t\t\t\t\t\t\t\t\t\t\t]\n\t\t\t\t\t\t\t\t\t\t}\n\t\t\t\t\t\t\t\t\t],\n\t\t\t\t\t\t\t\t\t\"policy\": {\n\t\t\t\t\t\t\t\t\t\t\"timeout\": \"0.00:10:00\",\n\t\t\t\t\t\t\t\t\t\t\"retry\": 0,\n\t\t\t\t\t\t\t\t\t\t\"retryIntervalInSeconds\": 30,\n\t\t\t\t\t\t\t\t\t\t\"secureOutput\": false,\n\t\t\t\t\t\t\t\t\t\t\"secureInput\": false\n\t\t\t\t\t\t\t\t\t},\n\t\t\t\t\t\t\t\t\t\"userProperties\": [],\n\t\t\t\t\t\t\t\t\t\"typeProperties\": {\n\t\t\t\t\t\t\t\t\t\t\"storedProcedureName\": \"[procfwk].[SetErrorLogDetails]\",\n\t\t\t\t\t\t\t\t\t\t\"storedProcedureParameters\": {\n\t\t\t\t\t\t\t\t\t\t\t\"JsonErrorDetails\": {\n\t\t\t\t\t\t\t\t\t\t\t\t\"value\": {\n\t\t\t\t\t\t\t\t\t\t\t\t\t\"value\": \"@string(activity('Get Worker Pipeline Error Details').output)\",\n\t\t\t\t\t\t\t\t\t\t\t\t\t\"type\": \"Expression\"\n\t\t\t\t\t\t\t\t\t\t\t\t},\n\t\t\t\t\t\t\t\t\t\t\t\t\"type\": \"String\"\n\t\t\t\t\t\t\t\t\t\t\t},\n\t\t\t\t\t\t\t\t\t\t\t\"LocalExecutionId\": {\n\t\t\t\t\t\t\t\t\t\t\t\t\"value\": {\n\t\t\t\t\t\t\t\t\t\t\t\t\t\"value\": \"@pipeline().parameters.executionId\",\n\t\t\t\t\t\t\t\t\t\t\t\t\t\"type\": \"Expression\"\n\t\t\t\t\t\t\t\t\t\t\t\t},\n\t\t\t\t\t\t\t\t\t\t\t\t\"type\": \"Guid\"\n\t\t\t\t\t\t\t\t\t\t\t}\n\t\t\t\t\t\t\t\t\t\t}\n\t\t\t\t\t\t\t\t\t},\n\t\t\t\t\t\t\t\t\t\"linkedServiceName\": {\n\t\t\t\t\t\t\t\t\t\t\"referenceName\": \"SupportDatabase\",\n\t\t\t\t\t\t\t\t\t\t\"type\": \"LinkedServiceReference\"\n\t\t\t\t\t\t\t\t\t}\n\t\t\t\t\t\t\t\t}\n\t\t\t\t\t\t\t]\n\t\t\t\t\t\t},\n\t\t\t\t\t\t{\n\t\t\t\t\t\t\t\"value\": \"Cancelled\",\n\t\t\t\t\t\t\t\"activities\": [\n\t\t\t\t\t\t\t\t{\n\t\t\t\t\t\t\t\t\t\"name\": \"Pipeline Status Cancelled\",\n\t\t\t\t\t\t\t\t\t\"description\": \"Updates the current execution table with a pipeline status of cancelled if the function outcome is cancelled.\",\n\t\t\t\t\t\t\t\t\t\"type\": \"SqlServerStoredProcedure\",\n\t\t\t\t\t\t\t\t\t\"dependsOn\": [],\n\t\t\t\t\t\t\t\t\t\"policy\": {\n\t\t\t\t\t\t\t\t\t\t\"timeout\": \"0.00:10:00\",\n\t\t\t\t\t\t\t\t\t\t\"retry\": 0,\n\t\t\t\t\t\t\t\t\t\t\"retryIntervalInSeconds\": 30,\n\t\t\t\t\t\t\t\t\t\t\"secureOutput\": false,\n\t\t\t\t\t\t\t\t\t\t\"secureInput\": false\n\t\t\t\t\t\t\t\t\t},\n\t\t\t\t\t\t\t\t\t\"userProperties\": [],\n\t\t\t\t\t\t\t\t\t\"typeProperties\": {\n\t\t\t\t\t\t\t\t\t\t\"storedProcedureName\": \"[procfwk].[SetLogPipelineCancelled]\",\n\t\t\t\t\t\t\t\t\t\t\"storedProcedureParameters\": {\n\t\t\t\t\t\t\t\t\t\t\t\"ExecutionId\": {\n\t\t\t\t\t\t\t\t\t\t\t\t\"value\": {\n\t\t\t\t\t\t\t\t\t\t\t\t\t\"value\": \"@pipeline().parameters.executionId\",\n\t\t\t\t\t\t\t\t\t\t\t\t\t\"type\": \"Expression\"\n\t\t\t\t\t\t\t\t\t\t\t\t},\n\t\t\t\t\t\t\t\t\t\t\t\t\"type\": \"Guid\"\n\t\t\t\t\t\t\t\t\t\t\t},\n\t\t\t\t\t\t\t\t\t\t\t\"PipelineId\": {\n\t\t\t\t\t\t\t\t\t\t\t\t\"value\": {\n\t\t\t\t\t\t\t\t\t\t\t\t\t\"value\": \"@pipeline().parameters.pipelineId\",\n\t\t\t\t\t\t\t\t\t\t\t\t\t\"type\": \"Expression\"\n\t\t\t\t\t\t\t\t\t\t\t\t},\n\t\t\t\t\t\t\t\t\t\t\t\t\"type\": \"Int32\"\n\t\t\t\t\t\t\t\t\t\t\t},\n\t\t\t\t\t\t\t\t\t\t\t\"StageId\": {\n\t\t\t\t\t\t\t\t\t\t\t\t\"value\": {\n\t\t\t\t\t\t\t\t\t\t\t\t\t\"value\": \"@pipeline().parameters.stageId\",\n\t\t\t\t\t\t\t\t\t\t\t\t\t\"type\": \"Expression\"\n\t\t\t\t\t\t\t\t\t\t\t\t},\n\t\t\t\t\t\t\t\t\t\t\t\t\"type\": \"Int32\"\n\t\t\t\t\t\t\t\t\t\t\t}\n\t\t\t\t\t\t\t\t\t\t}\n\t\t\t\t\t\t\t\t\t},\n\t\t\t\t\t\t\t\t\t\"linkedServiceName\": {\n\t\t\t\t\t\t\t\t\t\t\"referenceName\": \"SupportDatabase\",\n\t\t\t\t\t\t\t\t\t\t\"type\": \"LinkedServiceReference\"\n\t\t\t\t\t\t\t\t\t}\n\t\t\t\t\t\t\t\t}\n\t\t\t\t\t\t\t]\n\t\t\t\t\t\t}\n\t\t\t\t\t],\n\t\t\t\t\t\"defaultActivities\": [\n\t\t\t\t\t\t{\n\t\t\t\t\t\t\t\"name\": \"Pipeline Status Unknown\",\n\t\t\t\t\t\t\t\"description\": \"Updates the current execution table with a pipeline status of unknown if the function returns an unexpected outcome.\",\n\t\t\t\t\t\t\t\"type\": \"SqlServerStoredProcedure\",\n\t\t\t\t\t\t\t\"dependsOn\": [],\n\t\t\t\t\t\t\t\"policy\": {\n\t\t\t\t\t\t\t\t\"timeout\": \"0.00:10:00\",\n\t\t\t\t\t\t\t\t\"retry\": 0,\n\t\t\t\t\t\t\t\t\"retryIntervalInSeconds\": 30,\n\t\t\t\t\t\t\t\t\"secureOutput\": false,\n\t\t\t\t\t\t\t\t\"secureInput\": false\n\t\t\t\t\t\t\t},\n\t\t\t\t\t\t\t\"userProperties\": [],\n\t\t\t\t\t\t\t\"typeProperties\": {\n\t\t\t\t\t\t\t\t\"storedProcedureName\": \"[procfwk].[SetLogPipelineUnknown]\",\n\t\t\t\t\t\t\t\t\"storedProcedureParameters\": {\n\t\t\t\t\t\t\t\t\t\"ExecutionId\": {\n\t\t\t\t\t\t\t\t\t\t\"value\": {\n\t\t\t\t\t\t\t\t\t\t\t\"value\": \"@pipeline().parameters.executionId\",\n\t\t\t\t\t\t\t\t\t\t\t\"type\": \"Expression\"\n\t\t\t\t\t\t\t\t\t\t},\n\t\t\t\t\t\t\t\t\t\t\"type\": \"Guid\"\n\t\t\t\t\t\t\t\t\t},\n\t\t\t\t\t\t\t\t\t\"PipelineId\": {\n\t\t\t\t\t\t\t\t\t\t\"value\": {\n\t\t\t\t\t\t\t\t\t\t\t\"value\": \"@pipeline().parameters.pipelineId\",\n\t\t\t\t\t\t\t\t\t\t\t\"type\": \"Expression\"\n\t\t\t\t\t\t\t\t\t\t},\n\t\t\t\t\t\t\t\t\t\t\"type\": \"Int32\"\n\t\t\t\t\t\t\t\t\t},\n\t\t\t\t\t\t\t\t\t\"StageId\": {\n\t\t\t\t\t\t\t\t\t\t\"value\": {\n\t\t\t\t\t\t\t\t\t\t\t\"value\": \"@pipeline().parameters.stageId\",\n\t\t\t\t\t\t\t\t\t\t\t\"type\": \"Expression\"\n\t\t\t\t\t\t\t\t\t\t},\n\t\t\t\t\t\t\t\t\t\t\"type\": \"Int32\"\n\t\t\t\t\t\t\t\t\t}\n\t\t\t\t\t\t\t\t}\n\t\t\t\t\t\t\t},\n\t\t\t\t\t\t\t\"linkedServiceName\": {\n\t\t\t\t\t\t\t\t\"referenceName\": \"SupportDatabase\",\n\t\t\t\t\t\t\t\t\"type\": \"LinkedServiceReference\"\n\t\t\t\t\t\t\t}\n\t\t\t\t\t\t}\n\t\t\t\t\t]\n\t\t\t\t}\n\t\t\t},\n\t\t\t{\n\t\t\t\t\"name\": \"Get Wait Duration\",\n\t\t\t\t\"description\": \"Return wait duration in seconds from database properties table to be used during each Until iteration when the Worker pipeline is still running.\",\n\t\t\t\t\"type\": \"Lookup\",\n\t\t\t\t\"dependsOn\": [],\n\t\t\t\t\"policy\": {\n\t\t\t\t\t\"timeout\": \"0.00:10:00\",\n\t\t\t\t\t\"retry\": 0,\n\t\t\t\t\t\"retryIntervalInSeconds\": 30,\n\t\t\t\t\t\"secureOutput\": false,\n\t\t\t\t\t\"secureInput\": false\n\t\t\t\t},\n\t\t\t\t\"userProperties\": [],\n\t\t\t\t\"typeProperties\": {\n\t\t\t\t\t\"source\": {\n\t\t\t\t\t\t\"type\": \"AzureSqlSource\",\n\t\t\t\t\t\t\"sqlReaderStoredProcedureName\": \"[procfwk].[GetPropertyValue]\",\n\t\t\t\t\t\t\"storedProcedureParameters\": {\n\t\t\t\t\t\t\t\"PropertyName\": {\n\t\t\t\t\t\t\t\t\"type\": \"String\",\n\t\t\t\t\t\t\t\t\"value\": \"PipelineStatusCheckDuration\"\n\t\t\t\t\t\t\t}\n\t\t\t\t\t\t},\n\t\t\t\t\t\t\"queryTimeout\": \"02:00:00\",\n\t\t\t\t\t\t\"partitionOption\": \"None\"\n\t\t\t\t\t},\n\t\t\t\t\t\"dataset\": {\n\t\t\t\t\t\t\"referenceName\": \"GetSetMetadata\",\n\t\t\t\t\t\t\"type\": \"DatasetReference\"\n\t\t\t\t\t}\n\t\t\t\t}\n\t\t\t},\n\t\t\t{\n\t\t\t\t\"name\": \"Set Run Id\",\n\t\t\t\t\"description\": \"Set local variable from activity output once for value reuse in downstream activities.\",\n\t\t\t\t\"type\": \"SetVariable\",\n\t\t\t\t\"dependsOn\": [\n\t\t\t\t\t{\n\t\t\t\t\t\t\"activity\": \"Execute Worker Pipeline\",\n\t\t\t\t\t\t\"dependencyConditions\": [\n\t\t\t\t\t\t\t\"Succeeded\"\n\t\t\t\t\t\t]\n\t\t\t\t\t}\n\t\t\t\t],\n\t\t\t\t\"userProperties\": [],\n\t\t\t\t\"typeProperties\": {\n\t\t\t\t\t\"variableName\": \"WorkerRunId\",\n\t\t\t\t\t\"value\": {\n\t\t\t\t\t\t\"value\": \"@activity('Execute Worker Pipeline').output.RunId\",\n\t\t\t\t\t\t\"type\": \"Expression\"\n\t\t\t\t\t}\n\t\t\t\t}\n\t\t\t},\n\t\t\t{\n\t\t\t\t\"name\": \"Validate Pipeline\",\n\t\t\t\t\"description\": \"Query the target data factory and establish if the provided worker pipeline name is valid.\",\n\t\t\t\t\"type\": \"AzureFunctionActivity\",\n\t\t\t\t\"dependsOn\": [\n\t\t\t\t\t{\n\t\t\t\t\t\t\"activity\": \"Log Pipeline Validating\",\n\t\t\t\t\t\t\"dependencyConditions\": [\n\t\t\t\t\t\t\t\"Succeeded\"\n\t\t\t\t\t\t]\n\t\t\t\t\t},\n\t\t\t\t\t{\n\t\t\t\t\t\t\"activity\": \"Capture Worker Core Details as an Array\",\n\t\t\t\t\t\t\"dependencyConditions\": [\n\t\t\t\t\t\t\t\"Succeeded\"\n\t\t\t\t\t\t]\n\t\t\t\t\t}\n\t\t\t\t],\n\t\t\t\t\"policy\": {\n\t\t\t\t\t\"timeout\": \"0.00:10:00\",\n\t\t\t\t\t\"retry\": 0,\n\t\t\t\t\t\"retryIntervalInSeconds\": 30,\n\t\t\t\t\t\"secureOutput\": false,\n\t\t\t\t\t\"secureInput\": true\n\t\t\t\t},\n\t\t\t\t\"userProperties\": [],\n\t\t\t\t\"typeProperties\": {\n\t\t\t\t\t\"functionName\": \"ValidatePipeline\",\n\t\t\t\t\t\"method\": \"POST\",\n\t\t\t\t\t\"body\": {\n\t\t\t\t\t\t\"value\": \"@concat('\\n{\\n \\\"tenantId\\\": \\\"',variables('WorkerCoreDetails')[0].tenantId,'\\\",\\n \\\"applicationId\\\": \\\"',variables('WorkerCoreDetails')[0].applicationId,'\\\",\\n \\\"authenticationKey\\\": \\\"',variables('WorkerCoreDetails')[0].authenticationKey,'\\\",\\n \\\"subscriptionId\\\": \\\"',variables('WorkerCoreDetails')[0].subscriptionId,'\\\",\\n \\\"resourceGroupName\\\": \\\"',variables('WorkerCoreDetails')[0].resourceGroupName,'\\\",\\n\\t\\\"orchestratorName\\\": \\\"',variables('WorkerCoreDetails')[0].orchestratorName,'\\\",\\n \\\"orchestratorType\\\": \\\"',variables('WorkerCoreDetails')[0].orchestratorType,'\\\",\\n \\\"pipelineName\\\": \\\"',variables('WorkerCoreDetails')[0].pipelineName,'\\\"\\n}')\",\n\t\t\t\t\t\t\"type\": \"Expression\"\n\t\t\t\t\t}\n\t\t\t\t},\n\t\t\t\t\"linkedServiceName\": {\n\t\t\t\t\t\"referenceName\": \"FrameworkFunctions\",\n\t\t\t\t\t\"type\": \"LinkedServiceReference\"\n\t\t\t\t}\n\t\t\t},\n\t\t\t{\n\t\t\t\t\"name\": \"Is Target Worker Validate\",\n\t\t\t\t\"description\": \"True = the worker pipeline name is valid.\\nFalse = the worker pipeline name is invalid. Raise an exception.\",\n\t\t\t\t\"type\": \"IfCondition\",\n\t\t\t\t\"dependsOn\": [\n\t\t\t\t\t{\n\t\t\t\t\t\t\"activity\": \"Validate Pipeline\",\n\t\t\t\t\t\t\"dependencyConditions\": [\n\t\t\t\t\t\t\t\"Succeeded\"\n\t\t\t\t\t\t]\n\t\t\t\t\t}\n\t\t\t\t],\n\t\t\t\t\"userProperties\": [],\n\t\t\t\t\"typeProperties\": {\n\t\t\t\t\t\"expression\": {\n\t\t\t\t\t\t\"value\": \"@bool(activity('Validate Pipeline').output.PipelineExists)\",\n\t\t\t\t\t\t\"type\": \"Expression\"\n\t\t\t\t\t},\n\t\t\t\t\t\"ifFalseActivities\": [\n\t\t\t\t\t\t{\n\t\t\t\t\t\t\t\"name\": \"Throw Exception - Invalid Infant\",\n\t\t\t\t\t\t\t\"description\": \"Throw an exception with details about the invalid worker pipeline name.\",\n\t\t\t\t\t\t\t\"type\": \"ExecutePipeline\",\n\t\t\t\t\t\t\t\"dependsOn\": [],\n\t\t\t\t\t\t\t\"userProperties\": [],\n\t\t\t\t\t\t\t\"typeProperties\": {\n\t\t\t\t\t\t\t\t\"pipeline\": {\n\t\t\t\t\t\t\t\t\t\"referenceName\": \"Throw Exception\",\n\t\t\t\t\t\t\t\t\t\"type\": \"PipelineReference\"\n\t\t\t\t\t\t\t\t},\n\t\t\t\t\t\t\t\t\"waitOnCompletion\": true,\n\t\t\t\t\t\t\t\t\"parameters\": {\n\t\t\t\t\t\t\t\t\t\"Message\": {\n\t\t\t\t\t\t\t\t\t\t\"value\": \"@concat('Worker pipeline [',variables('WorkerCoreDetails')[0].pipelineName,'] is not valid in target Orchestrator [',variables('WorkerCoreDetails')[0].orchestratorName,']')\",\n\t\t\t\t\t\t\t\t\t\t\"type\": \"Expression\"\n\t\t\t\t\t\t\t\t\t}\n\t\t\t\t\t\t\t\t}\n\t\t\t\t\t\t\t}\n\t\t\t\t\t\t},\n\t\t\t\t\t\t{\n\t\t\t\t\t\t\t\"name\": \"Update Execution With Invalid Worker\",\n\t\t\t\t\t\t\t\"description\": \"Update the current execution table with an informed status for the worker pipeline that couldn't be executed.\",\n\t\t\t\t\t\t\t\"type\": \"SqlServerStoredProcedure\",\n\t\t\t\t\t\t\t\"dependsOn\": [],\n\t\t\t\t\t\t\t\"policy\": {\n\t\t\t\t\t\t\t\t\"timeout\": \"7.00:00:00\",\n\t\t\t\t\t\t\t\t\"retry\": 0,\n\t\t\t\t\t\t\t\t\"retryIntervalInSeconds\": 30,\n\t\t\t\t\t\t\t\t\"secureOutput\": false,\n\t\t\t\t\t\t\t\t\"secureInput\": false\n\t\t\t\t\t\t\t},\n\t\t\t\t\t\t\t\"userProperties\": [],\n\t\t\t\t\t\t\t\"typeProperties\": {\n\t\t\t\t\t\t\t\t\"storedProcedureName\": \"[procfwk].[SetLogActivityFailed]\",\n\t\t\t\t\t\t\t\t\"storedProcedureParameters\": {\n\t\t\t\t\t\t\t\t\t\"CallingActivity\": {\n\t\t\t\t\t\t\t\t\t\t\"value\": \"InvalidPipelineName\",\n\t\t\t\t\t\t\t\t\t\t\"type\": \"String\"\n\t\t\t\t\t\t\t\t\t},\n\t\t\t\t\t\t\t\t\t\"ExecutionId\": {\n\t\t\t\t\t\t\t\t\t\t\"value\": {\n\t\t\t\t\t\t\t\t\t\t\t\"value\": \"@pipeline().parameters.ExecutionId\",\n\t\t\t\t\t\t\t\t\t\t\t\"type\": \"Expression\"\n\t\t\t\t\t\t\t\t\t\t},\n\t\t\t\t\t\t\t\t\t\t\"type\": \"Guid\"\n\t\t\t\t\t\t\t\t\t},\n\t\t\t\t\t\t\t\t\t\"PipelineId\": {\n\t\t\t\t\t\t\t\t\t\t\"value\": {\n\t\t\t\t\t\t\t\t\t\t\t\"value\": \"@pipeline().parameters.pipelineId\",\n\t\t\t\t\t\t\t\t\t\t\t\"type\": \"Expression\"\n\t\t\t\t\t\t\t\t\t\t},\n\t\t\t\t\t\t\t\t\t\t\"type\": \"Int32\"\n\t\t\t\t\t\t\t\t\t},\n\t\t\t\t\t\t\t\t\t\"StageId\": {\n\t\t\t\t\t\t\t\t\t\t\"value\": {\n\t\t\t\t\t\t\t\t\t\t\t\"value\": \"@pipeline().parameters.StageId\",\n\t\t\t\t\t\t\t\t\t\t\t\"type\": \"Expression\"\n\t\t\t\t\t\t\t\t\t\t},\n\t\t\t\t\t\t\t\t\t\t\"type\": \"Int32\"\n\t\t\t\t\t\t\t\t\t}\n\t\t\t\t\t\t\t\t}\n\t\t\t\t\t\t\t},\n\t\t\t\t\t\t\t\"linkedServiceName\": {\n\t\t\t\t\t\t\t\t\"referenceName\": \"SupportDatabase\",\n\t\t\t\t\t\t\t\t\"type\": \"LinkedServiceReference\"\n\t\t\t\t\t\t\t}\n\t\t\t\t\t\t}\n\t\t\t\t\t]\n\t\t\t\t}\n\t\t\t},\n\t\t\t{\n\t\t\t\t\"name\": \"Log Validate Function Activity Failure\",\n\t\t\t\t\"description\": \"Handle true failures from calling out to the Azure Function and update the current execution table accordingly so a restart can occur.\",\n\t\t\t\t\"type\": \"SqlServerStoredProcedure\",\n\t\t\t\t\"dependsOn\": [\n\t\t\t\t\t{\n\t\t\t\t\t\t\"activity\": \"Validate Pipeline\",\n\t\t\t\t\t\t\"dependencyConditions\": [\n\t\t\t\t\t\t\t\"Failed\"\n\t\t\t\t\t\t]\n\t\t\t\t\t}\n\t\t\t\t],\n\t\t\t\t\"policy\": {\n\t\t\t\t\t\"timeout\": \"0.00:10:00\",\n\t\t\t\t\t\"retry\": 0,\n\t\t\t\t\t\"retryIntervalInSeconds\": 30,\n\t\t\t\t\t\"secureOutput\": false,\n\t\t\t\t\t\"secureInput\": false\n\t\t\t\t},\n\t\t\t\t\"userProperties\": [],\n\t\t\t\t\"typeProperties\": {\n\t\t\t\t\t\"storedProcedureName\": \"[procfwk].[SetLogActivityFailed]\",\n\t\t\t\t\t\"storedProcedureParameters\": {\n\t\t\t\t\t\t\"ExecutionId\": {\n\t\t\t\t\t\t\t\"value\": {\n\t\t\t\t\t\t\t\t\"value\": \"@pipeline().parameters.ExecutionId\",\n\t\t\t\t\t\t\t\t\"type\": \"Expression\"\n\t\t\t\t\t\t\t},\n\t\t\t\t\t\t\t\"type\": \"Guid\"\n\t\t\t\t\t\t},\n\t\t\t\t\t\t\"PipelineId\": {\n\t\t\t\t\t\t\t\"value\": {\n\t\t\t\t\t\t\t\t\"value\": \"@pipeline().parameters.pipelineId\",\n\t\t\t\t\t\t\t\t\"type\": \"Expression\"\n\t\t\t\t\t\t\t},\n\t\t\t\t\t\t\t\"type\": \"Int32\"\n\t\t\t\t\t\t},\n\t\t\t\t\t\t\"StageId\": {\n\t\t\t\t\t\t\t\"value\": {\n\t\t\t\t\t\t\t\t\"value\": \"@pipeline().parameters.StageId\",\n\t\t\t\t\t\t\t\t\"type\": \"Expression\"\n\t\t\t\t\t\t\t},\n\t\t\t\t\t\t\t\"type\": \"Int32\"\n\t\t\t\t\t\t},\n\t\t\t\t\t\t\"CallingActivity\": {\n\t\t\t\t\t\t\t\"value\": \"ValidatePipeline\",\n\t\t\t\t\t\t\t\"type\": \"String\"\n\t\t\t\t\t\t}\n\t\t\t\t\t}\n\t\t\t\t},\n\t\t\t\t\"linkedServiceName\": {\n\t\t\t\t\t\"referenceName\": \"SupportDatabase\",\n\t\t\t\t\t\"type\": \"LinkedServiceReference\"\n\t\t\t\t}\n\t\t\t},\n\t\t\t{\n\t\t\t\t\"name\": \"Log Pipeline Validating\",\n\t\t\t\t\"description\": \"Sets the current pipeline with a status of validating within the current execution database table.\",\n\t\t\t\t\"type\": \"SqlServerStoredProcedure\",\n\t\t\t\t\"dependsOn\": [],\n\t\t\t\t\"policy\": {\n\t\t\t\t\t\"timeout\": \"0.00:10:00\",\n\t\t\t\t\t\"retry\": 0,\n\t\t\t\t\t\"retryIntervalInSeconds\": 30,\n\t\t\t\t\t\"secureOutput\": false,\n\t\t\t\t\t\"secureInput\": false\n\t\t\t\t},\n\t\t\t\t\"userProperties\": [],\n\t\t\t\t\"typeProperties\": {\n\t\t\t\t\t\"storedProcedureName\": \"[procfwk].[SetLogPipelineValidating]\",\n\t\t\t\t\t\"storedProcedureParameters\": {\n\t\t\t\t\t\t\"ExecutionId\": {\n\t\t\t\t\t\t\t\"value\": {\n\t\t\t\t\t\t\t\t\"value\": \"@pipeline().parameters.ExecutionId\",\n\t\t\t\t\t\t\t\t\"type\": \"Expression\"\n\t\t\t\t\t\t\t},\n\t\t\t\t\t\t\t\"type\": \"Guid\"\n\t\t\t\t\t\t},\n\t\t\t\t\t\t\"PipelineId\": {\n\t\t\t\t\t\t\t\"value\": {\n\t\t\t\t\t\t\t\t\"value\": \"@pipeline().parameters.pipelineId\",\n\t\t\t\t\t\t\t\t\"type\": \"Expression\"\n\t\t\t\t\t\t\t},\n\t\t\t\t\t\t\t\"type\": \"Int32\"\n\t\t\t\t\t\t},\n\t\t\t\t\t\t\"StageId\": {\n\t\t\t\t\t\t\t\"value\": {\n\t\t\t\t\t\t\t\t\"value\": \"@pipeline().parameters.StageId\",\n\t\t\t\t\t\t\t\t\"type\": \"Expression\"\n\t\t\t\t\t\t\t},\n\t\t\t\t\t\t\t\"type\": \"Int32\"\n\t\t\t\t\t\t}\n\t\t\t\t\t}\n\t\t\t\t},\n\t\t\t\t\"linkedServiceName\": {\n\t\t\t\t\t\"referenceName\": \"SupportDatabase\",\n\t\t\t\t\t\"type\": \"LinkedServiceReference\"\n\t\t\t\t}\n\t\t\t},\n\t\t\t{\n\t\t\t\t\"name\": \"Get Worker Core Details\",\n\t\t\t\t\"description\": \"Return worker pipeline information for metadata database. Including target data factory, pipeline name and resource group. Return the SPN ID and Secret for the worker pipeline being executed. Called at this level as each pipeline can have a different SPN.\",\n\t\t\t\t\"type\": \"Lookup\",\n\t\t\t\t\"dependsOn\": [],\n\t\t\t\t\"policy\": {\n\t\t\t\t\t\"timeout\": \"0.00:10:00\",\n\t\t\t\t\t\"retry\": 0,\n\t\t\t\t\t\"retryIntervalInSeconds\": 30,\n\t\t\t\t\t\"secureOutput\": true,\n\t\t\t\t\t\"secureInput\": false\n\t\t\t\t},\n\t\t\t\t\"userProperties\": [],\n\t\t\t\t\"typeProperties\": {\n\t\t\t\t\t\"source\": {\n\t\t\t\t\t\t\"type\": \"AzureSqlSource\",\n\t\t\t\t\t\t\"sqlReaderStoredProcedureName\": \"[procfwk].[GetWorkerDetailsWrapper]\",\n\t\t\t\t\t\t\"storedProcedureParameters\": {\n\t\t\t\t\t\t\t\"ExecutionId\": {\n\t\t\t\t\t\t\t\t\"type\": \"Guid\",\n\t\t\t\t\t\t\t\t\"value\": {\n\t\t\t\t\t\t\t\t\t\"value\": \"@pipeline().parameters.executionId\",\n\t\t\t\t\t\t\t\t\t\"type\": \"Expression\"\n\t\t\t\t\t\t\t\t}\n\t\t\t\t\t\t\t},\n\t\t\t\t\t\t\t\"PipelineId\": {\n\t\t\t\t\t\t\t\t\"type\": \"Int32\",\n\t\t\t\t\t\t\t\t\"value\": {\n\t\t\t\t\t\t\t\t\t\"value\": \"@pipeline().parameters.pipelineId\",\n\t\t\t\t\t\t\t\t\t\"type\": \"Expression\"\n\t\t\t\t\t\t\t\t}\n\t\t\t\t\t\t\t},\n\t\t\t\t\t\t\t\"StageId\": {\n\t\t\t\t\t\t\t\t\"type\": \"Int32\",\n\t\t\t\t\t\t\t\t\"value\": {\n\t\t\t\t\t\t\t\t\t\"value\": \"@pipeline().parameters.stageId\",\n\t\t\t\t\t\t\t\t\t\"type\": \"Expression\"\n\t\t\t\t\t\t\t\t}\n\t\t\t\t\t\t\t}\n\t\t\t\t\t\t},\n\t\t\t\t\t\t\"queryTimeout\": \"02:00:00\",\n\t\t\t\t\t\t\"partitionOption\": \"None\"\n\t\t\t\t\t},\n\t\t\t\t\t\"dataset\": {\n\t\t\t\t\t\t\"referenceName\": \"GetSetMetadata\",\n\t\t\t\t\t\t\"type\": \"DatasetReference\"\n\t\t\t\t\t}\n\t\t\t\t}\n\t\t\t},\n\t\t\t{\n\t\t\t\t\"name\": \"Capture Worker Core Details as an Array\",\n\t\t\t\t\"description\": \"Add all worker pipeline details to a local variable array that can be accessed by each function call requiring the values.\",\n\t\t\t\t\"type\": \"SetVariable\",\n\t\t\t\t\"dependsOn\": [\n\t\t\t\t\t{\n\t\t\t\t\t\t\"activity\": \"Get Worker Core Details\",\n\t\t\t\t\t\t\"dependencyConditions\": [\n\t\t\t\t\t\t\t\"Succeeded\"\n\t\t\t\t\t\t]\n\t\t\t\t\t}\n\t\t\t\t],\n\t\t\t\t\"userProperties\": [],\n\t\t\t\t\"typeProperties\": {\n\t\t\t\t\t\"variableName\": \"WorkerCoreDetails\",\n\t\t\t\t\t\"value\": {\n\t\t\t\t\t\t\"value\": \"@array(activity('Get Worker Core Details').output.firstRow)\",\n\t\t\t\t\t\t\"type\": \"Expression\"\n\t\t\t\t\t}\n\t\t\t\t}\n\t\t\t}\n\t\t],\n\t\t\"parameters\": {\n\t\t\t\"executionId\": {\n\t\t\t\t\"type\": \"string\"\n\t\t\t},\n\t\t\t\"stageId\": {\n\t\t\t\t\"type\": \"int\"\n\t\t\t},\n\t\t\t\"pipelineId\": {\n\t\t\t\t\"type\": \"int\"\n\t\t\t}\n\t\t},\n\t\t\"variables\": {\n\t\t\t\"WorkerPipelineState\": {\n\t\t\t\t\"type\": \"Boolean\"\n\t\t\t},\n\t\t\t\"WorkerRunId\": {\n\t\t\t\t\"type\": \"String\"\n\t\t\t},\n\t\t\t\"WorkerCoreDetails\": {\n\t\t\t\t\"type\": \"Array\"\n\t\t\t}\n\t\t},\n\t\t\"folder\": {\n\t\t\t\"name\": \"_ProcFwk\"\n\t\t},\n\t\t\"annotations\": [\n\t\t\t\"procfwk\",\n\t\t\t\"Infant\"\n\t\t]\n\t}\n}"
},
{
"path": "Synapse/pipeline/Check For Running Pipeline.json",
"content": "{\n\t\"name\": \"Check For Running Pipeline\",\n\t\"properties\": {\n\t\t\"description\": \"For a given pipeline and optional batch name establish if a pipeline run is already in progress. Throw an exception if it it.\",\n\t\t\"activities\": [\n\t\t\t{\n\t\t\t\t\"name\": \"Filter Running Pipelines\",\n\t\t\t\t\"description\": \"Filter the pipeline runs results for pipelines that exclude the current triggered run and that are currently running (in progress or queued).\",\n\t\t\t\t\"type\": \"Filter\",\n\t\t\t\t\"dependsOn\": [\n\t\t\t\t\t{\n\t\t\t\t\t\t\"activity\": \"Switch For Orchestrator Type\",\n\t\t\t\t\t\t\"dependencyConditions\": [\n\t\t\t\t\t\t\t\"Succeeded\"\n\t\t\t\t\t\t]\n\t\t\t\t\t}\n\t\t\t\t],\n\t\t\t\t\"userProperties\": [],\n\t\t\t\t\"typeProperties\": {\n\t\t\t\t\t\"items\": {\n\t\t\t\t\t\t\"value\": \"@variables('PipelineRuns')\",\n\t\t\t\t\t\t\"type\": \"Expression\"\n\t\t\t\t\t},\n\t\t\t\t\t\"condition\": {\n\t\t\t\t\t\t\"value\": \"@and(not(equals(item().runId,pipeline().parameters.ThisRunId)),or(equals(item().status,'InProgress'),equals(item().status,'Queued')))\",\n\t\t\t\t\t\t\"type\": \"Expression\"\n\t\t\t\t\t}\n\t\t\t\t}\n\t\t\t},\n\t\t\t{\n\t\t\t\t\"name\": \"Get Framework Orchestrator Details\",\n\t\t\t\t\"description\": \"Using the metadata orchestrators return details about the resource running the framework pipelines.\",\n\t\t\t\t\"type\": \"Lookup\",\n\t\t\t\t\"dependsOn\": [],\n\t\t\t\t\"policy\": {\n\t\t\t\t\t\"timeout\": \"0.00:10:00\",\n\t\t\t\t\t\"retry\": 0,\n\t\t\t\t\t\"retryIntervalInSeconds\": 30,\n\t\t\t\t\t\"secureOutput\": false,\n\t\t\t\t\t\"secureInput\": false\n\t\t\t\t},\n\t\t\t\t\"userProperties\": [],\n\t\t\t\t\"typeProperties\": {\n\t\t\t\t\t\"source\": {\n\t\t\t\t\t\t\"type\": \"AzureSqlSource\",\n\t\t\t\t\t\t\"sqlReaderStoredProcedureName\": \"[procfwk].[GetFrameworkOrchestratorDetails]\",\n\t\t\t\t\t\t\"storedProcedureParameters\": {\n\t\t\t\t\t\t\t\"CallingOrchestratorName\": {\n\t\t\t\t\t\t\t\t\"type\": \"String\",\n\t\t\t\t\t\t\t\t\"value\": {\n\t\t\t\t\t\t\t\t\t\"value\": \"@pipeline().DataFactory\",\n\t\t\t\t\t\t\t\t\t\"type\": \"Expression\"\n\t\t\t\t\t\t\t\t}\n\t\t\t\t\t\t\t}\n\t\t\t\t\t\t},\n\t\t\t\t\t\t\"queryTimeout\": \"02:00:00\",\n\t\t\t\t\t\t\"partitionOption\": \"None\"\n\t\t\t\t\t},\n\t\t\t\t\t\"dataset\": {\n\t\t\t\t\t\t\"referenceName\": \"GetSetMetadata\",\n\t\t\t\t\t\t\"type\": \"DatasetReference\"\n\t\t\t\t\t}\n\t\t\t\t}\n\t\t\t},\n\t\t\t{\n\t\t\t\t\"name\": \"Get Query Run Days Value\",\n\t\t\t\t\"description\": \"Using the metadata properties table return the run days value to provide the API request with a date range for pipeline executions.\",\n\t\t\t\t\"type\": \"Lookup\",\n\t\t\t\t\"dependsOn\": [],\n\t\t\t\t\"policy\": {\n\t\t\t\t\t\"timeout\": \"0.00:10:00\",\n\t\t\t\t\t\"retry\": 0,\n\t\t\t\t\t\"retryIntervalInSeconds\": 30,\n\t\t\t\t\t\"secureOutput\": false,\n\t\t\t\t\t\"secureInput\": false\n\t\t\t\t},\n\t\t\t\t\"userProperties\": [],\n\t\t\t\t\"typeProperties\": {\n\t\t\t\t\t\"source\": {\n\t\t\t\t\t\t\"type\": \"AzureSqlSource\",\n\t\t\t\t\t\t\"sqlReaderStoredProcedureName\": \"[procfwk].[GetPropertyValue]\",\n\t\t\t\t\t\t\"storedProcedureParameters\": {\n\t\t\t\t\t\t\t\"PropertyName\": {\n\t\t\t\t\t\t\t\t\"type\": \"String\",\n\t\t\t\t\t\t\t\t\"value\": \"PreviousPipelineRunsQueryRange\"\n\t\t\t\t\t\t\t}\n\t\t\t\t\t\t},\n\t\t\t\t\t\t\"queryTimeout\": \"02:00:00\",\n\t\t\t\t\t\t\"partitionOption\": \"None\"\n\t\t\t\t\t},\n\t\t\t\t\t\"dataset\": {\n\t\t\t\t\t\t\"referenceName\": \"GetSetMetadata\",\n\t\t\t\t\t\t\"type\": \"DatasetReference\"\n\t\t\t\t\t}\n\t\t\t\t}\n\t\t\t},\n\t\t\t{\n\t\t\t\t\"name\": \"If Pipeline Is Running\",\n\t\t\t\t\"description\": \"If the running pipeline count is greater than or equal to one.\\nTrue = raise an exception.\",\n\t\t\t\t\"type\": \"IfCondition\",\n\t\t\t\t\"dependsOn\": [\n\t\t\t\t\t{\n\t\t\t\t\t\t\"activity\": \"If Using Batch Executions\",\n\t\t\t\t\t\t\"dependencyConditions\": [\n\t\t\t\t\t\t\t\"Succeeded\"\n\t\t\t\t\t\t]\n\t\t\t\t\t}\n\t\t\t\t],\n\t\t\t\t\"userProperties\": [],\n\t\t\t\t\"typeProperties\": {\n\t\t\t\t\t\"expression\": {\n\t\t\t\t\t\t\"value\": \"@greaterOrEquals(int(variables('RunCount')),1)\",\n\t\t\t\t\t\t\"type\": \"Expression\"\n\t\t\t\t\t},\n\t\t\t\t\t\"ifTrueActivities\": [\n\t\t\t\t\t\t{\n\t\t\t\t\t\t\t\"name\": \"Throw Exception - Pipeline Running\",\n\t\t\t\t\t\t\t\"description\": \"Using the utils pipeline raise an exception to stop the new trigger while a run is already in progress.\",\n\t\t\t\t\t\t\t\"type\": \"ExecutePipeline\",\n\t\t\t\t\t\t\t\"dependsOn\": [],\n\t\t\t\t\t\t\t\"userProperties\": [],\n\t\t\t\t\t\t\t\"typeProperties\": {\n\t\t\t\t\t\t\t\t\"pipeline\": {\n\t\t\t\t\t\t\t\t\t\"referenceName\": \"Throw Exception\",\n\t\t\t\t\t\t\t\t\t\"type\": \"PipelineReference\"\n\t\t\t\t\t\t\t\t},\n\t\t\t\t\t\t\t\t\"waitOnCompletion\": true,\n\t\t\t\t\t\t\t\t\"parameters\": {\n\t\t\t\t\t\t\t\t\t\"Message\": {\n\t\t\t\t\t\t\t\t\t\t\"value\": \"@concat('Provided pipeline name (',pipeline().parameters.PipelineName,') still has a run in progress or queued given the query range parameters set in the properties table.')\",\n\t\t\t\t\t\t\t\t\t\t\"type\": \"Expression\"\n\t\t\t\t\t\t\t\t\t}\n\t\t\t\t\t\t\t\t}\n\t\t\t\t\t\t\t}\n\t\t\t\t\t\t}\n\t\t\t\t\t]\n\t\t\t\t}\n\t\t\t},\n\t\t\t{\n\t\t\t\t\"name\": \"Get Execution Batch Status\",\n\t\t\t\t\"description\": \"Using the metadata properties table return the flag to indicate if batch execution setting are enabled or disabled.\",\n\t\t\t\t\"type\": \"Lookup\",\n\t\t\t\t\"dependsOn\": [],\n\t\t\t\t\"policy\": {\n\t\t\t\t\t\"timeout\": \"0.00:10:00\",\n\t\t\t\t\t\"retry\": 0,\n\t\t\t\t\t\"retryIntervalInSeconds\": 30,\n\t\t\t\t\t\"secureOutput\": false,\n\t\t\t\t\t\"secureInput\": false\n\t\t\t\t},\n\t\t\t\t\"userProperties\": [],\n\t\t\t\t\"typeProperties\": {\n\t\t\t\t\t\"source\": {\n\t\t\t\t\t\t\"type\": \"AzureSqlSource\",\n\t\t\t\t\t\t\"sqlReaderStoredProcedureName\": \"[procfwk].[GetPropertyValue]\",\n\t\t\t\t\t\t\"storedProcedureParameters\": {\n\t\t\t\t\t\t\t\"PropertyName\": {\n\t\t\t\t\t\t\t\t\"type\": \"String\",\n\t\t\t\t\t\t\t\t\"value\": \"UseExecutionBatches\"\n\t\t\t\t\t\t\t}\n\t\t\t\t\t\t},\n\t\t\t\t\t\t\"queryTimeout\": \"02:00:00\",\n\t\t\t\t\t\t\"partitionOption\": \"None\"\n\t\t\t\t\t},\n\t\t\t\t\t\"dataset\": {\n\t\t\t\t\t\t\"referenceName\": \"GetSetMetadata\",\n\t\t\t\t\t\t\"type\": \"DatasetReference\"\n\t\t\t\t\t}\n\t\t\t\t}\n\t\t\t},\n\t\t\t{\n\t\t\t\t\"name\": \"If Using Batch Executions\",\n\t\t\t\t\"description\": \"True = batch executions are enabled.\\nFalse = batch execution are disabled.\",\n\t\t\t\t\"type\": \"IfCondition\",\n\t\t\t\t\"dependsOn\": [\n\t\t\t\t\t{\n\t\t\t\t\t\t\"activity\": \"Get Execution Batch Status\",\n\t\t\t\t\t\t\"dependencyConditions\": [\n\t\t\t\t\t\t\t\"Succeeded\"\n\t\t\t\t\t\t]\n\t\t\t\t\t},\n\t\t\t\t\t{\n\t\t\t\t\t\t\"activity\": \"Filter Running Pipelines\",\n\t\t\t\t\t\t\"dependencyConditions\": [\n\t\t\t\t\t\t\t\"Succeeded\"\n\t\t\t\t\t\t]\n\t\t\t\t\t}\n\t\t\t\t],\n\t\t\t\t\"userProperties\": [],\n\t\t\t\t\"typeProperties\": {\n\t\t\t\t\t\"expression\": {\n\t\t\t\t\t\t\"value\": \"@equals(activity('Get Execution Batch Status').output.firstRow.PropertyValue,string(1))\",\n\t\t\t\t\t\t\"type\": \"Expression\"\n\t\t\t\t\t},\n\t\t\t\t\t\"ifFalseActivities\": [\n\t\t\t\t\t\t{\n\t\t\t\t\t\t\t\"name\": \"Set Run Count Without Batch\",\n\t\t\t\t\t\t\t\"description\": \"Set the pipelines running count variable to be tested later.\",\n\t\t\t\t\t\t\t\"type\": \"SetVariable\",\n\t\t\t\t\t\t\t\"dependsOn\": [],\n\t\t\t\t\t\t\t\"userProperties\": [],\n\t\t\t\t\t\t\t\"typeProperties\": {\n\t\t\t\t\t\t\t\t\"variableName\": \"RunCount\",\n\t\t\t\t\t\t\t\t\"value\": {\n\t\t\t\t\t\t\t\t\t\"value\": \"@string(activity('Filter Running Pipelines').output.FilteredItemsCount)\",\n\t\t\t\t\t\t\t\t\t\"type\": \"Expression\"\n\t\t\t\t\t\t\t\t}\n\t\t\t\t\t\t\t}\n\t\t\t\t\t\t}\n\t\t\t\t\t],\n\t\t\t\t\t\"ifTrueActivities\": [\n\t\t\t\t\t\t{\n\t\t\t\t\t\t\t\"name\": \"Filter for Batch Name\",\n\t\t\t\t\t\t\t\"description\": \"Further filter the return pipeline runs for any running pipelines with the same batch name value.\",\n\t\t\t\t\t\t\t\"type\": \"Filter\",\n\t\t\t\t\t\t\t\"dependsOn\": [],\n\t\t\t\t\t\t\t\"userProperties\": [],\n\t\t\t\t\t\t\t\"typeProperties\": {\n\t\t\t\t\t\t\t\t\"items\": {\n\t\t\t\t\t\t\t\t\t\"value\": \"@activity('Filter Running Pipelines').output.value\",\n\t\t\t\t\t\t\t\t\t\"type\": \"Expression\"\n\t\t\t\t\t\t\t\t},\n\t\t\t\t\t\t\t\t\"condition\": {\n\t\t\t\t\t\t\t\t\t\"value\": \"@equals(item().parameters.BatchName,pipeline().parameters.BatchName)\",\n\t\t\t\t\t\t\t\t\t\"type\": \"Expression\"\n\t\t\t\t\t\t\t\t}\n\t\t\t\t\t\t\t}\n\t\t\t\t\t\t},\n\t\t\t\t\t\t{\n\t\t\t\t\t\t\t\"name\": \"Set Run Count for Batch\",\n\t\t\t\t\t\t\t\"description\": \"Set the resulting pipeline running count variable to be tested later.\",\n\t\t\t\t\t\t\t\"type\": \"SetVariable\",\n\t\t\t\t\t\t\t\"dependsOn\": [\n\t\t\t\t\t\t\t\t{\n\t\t\t\t\t\t\t\t\t\"activity\": \"Filter for Batch Name\",\n\t\t\t\t\t\t\t\t\t\"dependencyConditions\": [\n\t\t\t\t\t\t\t\t\t\t\"Succeeded\"\n\t\t\t\t\t\t\t\t\t]\n\t\t\t\t\t\t\t\t}\n\t\t\t\t\t\t\t],\n\t\t\t\t\t\t\t\"userProperties\": [],\n\t\t\t\t\t\t\t\"typeProperties\": {\n\t\t\t\t\t\t\t\t\"variableName\": \"RunCount\",\n\t\t\t\t\t\t\t\t\"value\": {\n\t\t\t\t\t\t\t\t\t\"value\": \"@string(activity('Filter for Batch Name').output.FilteredItemsCount)\",\n\t\t\t\t\t\t\t\t\t\"type\": \"Expression\"\n\t\t\t\t\t\t\t\t}\n\t\t\t\t\t\t\t}\n\t\t\t\t\t\t}\n\t\t\t\t\t]\n\t\t\t\t}\n\t\t\t},\n\t\t\t{\n\t\t\t\t\"name\": \"Set Subscription Id\",\n\t\t\t\t\"description\": \"Set the subscription Id value to a local variable for use in various downstream activities.\",\n\t\t\t\t\"type\": \"SetVariable\",\n\t\t\t\t\"dependsOn\": [\n\t\t\t\t\t{\n\t\t\t\t\t\t\"activity\": \"Get Framework Orchestrator Details\",\n\t\t\t\t\t\t\"dependencyConditions\": [\n\t\t\t\t\t\t\t\"Succeeded\"\n\t\t\t\t\t\t]\n\t\t\t\t\t}\n\t\t\t\t],\n\t\t\t\t\"userProperties\": [],\n\t\t\t\t\"typeProperties\": {\n\t\t\t\t\t\"variableName\": \"SubscriptionId\",\n\t\t\t\t\t\"value\": {\n\t\t\t\t\t\t\"value\": \"@activity('Get Framework Orchestrator Details').output.firstRow.SubscriptionId\",\n\t\t\t\t\t\t\"type\": \"Expression\"\n\t\t\t\t\t}\n\t\t\t\t}\n\t\t\t},\n\t\t\t{\n\t\t\t\t\"name\": \"Set Resource Group Name\",\n\t\t\t\t\"description\": \"Set the resource group name value to a local variable for use in various downstream activities.\",\n\t\t\t\t\"type\": \"SetVariable\",\n\t\t\t\t\"dependsOn\": [\n\t\t\t\t\t{\n\t\t\t\t\t\t\"activity\": \"Get Framework Orchestrator Details\",\n\t\t\t\t\t\t\"dependencyConditions\": [\n\t\t\t\t\t\t\t\"Succeeded\"\n\t\t\t\t\t\t]\n\t\t\t\t\t}\n\t\t\t\t],\n\t\t\t\t\"userProperties\": [],\n\t\t\t\t\"typeProperties\": {\n\t\t\t\t\t\"variableName\": \"ResourceGroupName\",\n\t\t\t\t\t\"value\": {\n\t\t\t\t\t\t\"value\": \"@activity('Get Framework Orchestrator Details').output.firstRow.ResourceGroupName\",\n\t\t\t\t\t\t\"type\": \"Expression\"\n\t\t\t\t\t}\n\t\t\t\t}\n\t\t\t},\n\t\t\t{\n\t\t\t\t\"name\": \"Set Orchestrator Type\",\n\t\t\t\t\"description\": \"Set the orchestrator type value to a local variable for use in various downstream activities.\",\n\t\t\t\t\"type\": \"SetVariable\",\n\t\t\t\t\"dependsOn\": [\n\t\t\t\t\t{\n\t\t\t\t\t\t\"activity\": \"Get Framework Orchestrator Details\",\n\t\t\t\t\t\t\"dependencyConditions\": [\n\t\t\t\t\t\t\t\"Succeeded\"\n\t\t\t\t\t\t]\n\t\t\t\t\t}\n\t\t\t\t],\n\t\t\t\t\"userProperties\": [],\n\t\t\t\t\"typeProperties\": {\n\t\t\t\t\t\"variableName\": \"OrchestratorType\",\n\t\t\t\t\t\"value\": {\n\t\t\t\t\t\t\"value\": \"@toUpper(activity('Get Framework Orchestrator Details').output.firstRow.OrchestratorType)\",\n\t\t\t\t\t\t\"type\": \"Expression\"\n\t\t\t\t\t}\n\t\t\t\t}\n\t\t\t},\n\t\t\t{\n\t\t\t\t\"name\": \"Switch For Orchestrator Type\",\n\t\t\t\t\"description\": \"Switch and handle requests for both Azure Data Factory (ADF) and Azure Synapse Analytics (SYN).\",\n\t\t\t\t\"type\": \"Switch\",\n\t\t\t\t\"dependsOn\": [\n\t\t\t\t\t{\n\t\t\t\t\t\t\"activity\": \"Set Orchestrator Type\",\n\t\t\t\t\t\t\"dependencyConditions\": [\n\t\t\t\t\t\t\t\"Succeeded\"\n\t\t\t\t\t\t]\n\t\t\t\t\t},\n\t\t\t\t\t{\n\t\t\t\t\t\t\"activity\": \"Set Query Run Days\",\n\t\t\t\t\t\t\"dependencyConditions\": [\n\t\t\t\t\t\t\t\"Succeeded\"\n\t\t\t\t\t\t]\n\t\t\t\t\t},\n\t\t\t\t\t{\n\t\t\t\t\t\t\"activity\": \"Set Resource Group Name\",\n\t\t\t\t\t\t\"dependencyConditions\": [\n\t\t\t\t\t\t\t\"Succeeded\"\n\t\t\t\t\t\t]\n\t\t\t\t\t},\n\t\t\t\t\t{\n\t\t\t\t\t\t\"activity\": \"Set Subscription Id\",\n\t\t\t\t\t\t\"dependencyConditions\": [\n\t\t\t\t\t\t\t\"Succeeded\"\n\t\t\t\t\t\t]\n\t\t\t\t\t}\n\t\t\t\t],\n\t\t\t\t\"userProperties\": [],\n\t\t\t\t\"typeProperties\": {\n\t\t\t\t\t\"on\": {\n\t\t\t\t\t\t\"value\": \"@variables('OrchestratorType')\",\n\t\t\t\t\t\t\"type\": \"Expression\"\n\t\t\t\t\t},\n\t\t\t\t\t\"cases\": [\n\t\t\t\t\t\t{\n\t\t\t\t\t\t\t\"value\": \"ADF\",\n\t\t\t\t\t\t\t\"activities\": [\n\t\t\t\t\t\t\t\t{\n\t\t\t\t\t\t\t\t\t\"name\": \"Check for Valid ADF Pipeline Name\",\n\t\t\t\t\t\t\t\t\t\"description\": \"Use the Azure Management API to return and establish if the framework pipeline exists in the target Data Factory instance, including being deployed.\",\n\t\t\t\t\t\t\t\t\t\"type\": \"WebActivity\",\n\t\t\t\t\t\t\t\t\t\"dependsOn\": [],\n\t\t\t\t\t\t\t\t\t\"policy\": {\n\t\t\t\t\t\t\t\t\t\t\"timeout\": \"7.00:00:00\",\n\t\t\t\t\t\t\t\t\t\t\"retry\": 0,\n\t\t\t\t\t\t\t\t\t\t\"retryIntervalInSeconds\": 30,\n\t\t\t\t\t\t\t\t\t\t\"secureOutput\": false,\n\t\t\t\t\t\t\t\t\t\t\"secureInput\": false\n\t\t\t\t\t\t\t\t\t},\n\t\t\t\t\t\t\t\t\t\"userProperties\": [],\n\t\t\t\t\t\t\t\t\t\"typeProperties\": {\n\t\t\t\t\t\t\t\t\t\t\"url\": {\n\t\t\t\t\t\t\t\t\t\t\t\"value\": \"https://management.azure.com/subscriptions/@{variables('SubscriptionId')}/resourceGroups/@{variables('ResourceGroupName')}/providers/Microsoft.DataFactory/factories/@{pipeline().DataFactory}/pipelines/@{pipeline().parameters.PipelineName}?api-version=2018-06-01\",\n\t\t\t\t\t\t\t\t\t\t\t\"type\": \"Expression\"\n\t\t\t\t\t\t\t\t\t\t},\n\t\t\t\t\t\t\t\t\t\t\"method\": \"GET\",\n\t\t\t\t\t\t\t\t\t\t\"authentication\": {\n\t\t\t\t\t\t\t\t\t\t\t\"type\": \"MSI\",\n\t\t\t\t\t\t\t\t\t\t\t\"resource\": \"https://management.core.windows.net/\"\n\t\t\t\t\t\t\t\t\t\t}\n\t\t\t\t\t\t\t\t\t}\n\t\t\t\t\t\t\t\t},\n\t\t\t\t\t\t\t\t{\n\t\t\t\t\t\t\t\t\t\"name\": \"Get ADF Pipeline Runs\",\n\t\t\t\t\t\t\t\t\t\"description\": \"Use the Azure Management API to return a list of data factory pipeline runs within the given time window.\",\n\t\t\t\t\t\t\t\t\t\"type\": \"WebActivity\",\n\t\t\t\t\t\t\t\t\t\"dependsOn\": [\n\t\t\t\t\t\t\t\t\t\t{\n\t\t\t\t\t\t\t\t\t\t\t\"activity\": \"Check for Valid ADF Pipeline Name\",\n\t\t\t\t\t\t\t\t\t\t\t\"dependencyConditions\": [\n\t\t\t\t\t\t\t\t\t\t\t\t\"Succeeded\"\n\t\t\t\t\t\t\t\t\t\t\t]\n\t\t\t\t\t\t\t\t\t\t}\n\t\t\t\t\t\t\t\t\t],\n\t\t\t\t\t\t\t\t\t\"policy\": {\n\t\t\t\t\t\t\t\t\t\t\"timeout\": \"7.00:00:00\",\n\t\t\t\t\t\t\t\t\t\t\"retry\": 0,\n\t\t\t\t\t\t\t\t\t\t\"retryIntervalInSeconds\": 30,\n\t\t\t\t\t\t\t\t\t\t\"secureOutput\": false,\n\t\t\t\t\t\t\t\t\t\t\"secureInput\": false\n\t\t\t\t\t\t\t\t\t},\n\t\t\t\t\t\t\t\t\t\"userProperties\": [],\n\t\t\t\t\t\t\t\t\t\"typeProperties\": {\n\t\t\t\t\t\t\t\t\t\t\"url\": {\n\t\t\t\t\t\t\t\t\t\t\t\"value\": \"https://management.azure.com/subscriptions/@{variables('SubscriptionId')}/resourceGroups/@{variables('ResourceGroupName')}/providers/Microsoft.DataFactory/factories/@{pipeline().DataFactory}/queryPipelineRuns?api-version=2018-06-01\",\n\t\t\t\t\t\t\t\t\t\t\t\"type\": \"Expression\"\n\t\t\t\t\t\t\t\t\t\t},\n\t\t\t\t\t\t\t\t\t\t\"method\": \"POST\",\n\t\t\t\t\t\t\t\t\t\t\"body\": {\n\t\t\t\t\t\t\t\t\t\t\t\"value\": \"{\\n \\\"lastUpdatedAfter\\\": \\\"@{adddays(utcnow(),int(variables('QueryRunDays')))}\\\",\\n \\\"lastUpdatedBefore\\\": \\\"@{utcnow()}\\\",\\n \\\"filters\\\": [\\n {\\n \\\"operand\\\": \\\"PipelineName\\\",\\n \\\"operator\\\": \\\"Equals\\\",\\n \\\"values\\\": [\\n \\\"@{pipeline().parameters.PipelineName}\\\"\\n ]\\n }\\n ]\\n}\",\n\t\t\t\t\t\t\t\t\t\t\t\"type\": \"Expression\"\n\t\t\t\t\t\t\t\t\t\t},\n\t\t\t\t\t\t\t\t\t\t\"authentication\": {\n\t\t\t\t\t\t\t\t\t\t\t\"type\": \"MSI\",\n\t\t\t\t\t\t\t\t\t\t\t\"resource\": \"https://management.core.windows.net/\"\n\t\t\t\t\t\t\t\t\t\t}\n\t\t\t\t\t\t\t\t\t}\n\t\t\t\t\t\t\t\t},\n\t\t\t\t\t\t\t\t{\n\t\t\t\t\t\t\t\t\t\"name\": \"Set ADF Runs Output\",\n\t\t\t\t\t\t\t\t\t\"description\": \"Set output to local array for use in downstream filtering and pipeline checks. Use the same array output for both switch cases.\",\n\t\t\t\t\t\t\t\t\t\"type\": \"SetVariable\",\n\t\t\t\t\t\t\t\t\t\"dependsOn\": [\n\t\t\t\t\t\t\t\t\t\t{\n\t\t\t\t\t\t\t\t\t\t\t\"activity\": \"Get ADF Pipeline Runs\",\n\t\t\t\t\t\t\t\t\t\t\t\"dependencyConditions\": [\n\t\t\t\t\t\t\t\t\t\t\t\t\"Succeeded\"\n\t\t\t\t\t\t\t\t\t\t\t]\n\t\t\t\t\t\t\t\t\t\t}\n\t\t\t\t\t\t\t\t\t],\n\t\t\t\t\t\t\t\t\t\"userProperties\": [],\n\t\t\t\t\t\t\t\t\t\"typeProperties\": {\n\t\t\t\t\t\t\t\t\t\t\"variableName\": \"PipelineRuns\",\n\t\t\t\t\t\t\t\t\t\t\"value\": {\n\t\t\t\t\t\t\t\t\t\t\t\"value\": \"@activity('Get ADF Pipeline Runs').output.value\",\n\t\t\t\t\t\t\t\t\t\t\t\"type\": \"Expression\"\n\t\t\t\t\t\t\t\t\t\t}\n\t\t\t\t\t\t\t\t\t}\n\t\t\t\t\t\t\t\t}\n\t\t\t\t\t\t\t]\n\t\t\t\t\t\t},\n\t\t\t\t\t\t{\n\t\t\t\t\t\t\t\"value\": \"SYN\",\n\t\t\t\t\t\t\t\"activities\": [\n\t\t\t\t\t\t\t\t{\n\t\t\t\t\t\t\t\t\t\"name\": \"Check for Valid SYN Pipeline Name\",\n\t\t\t\t\t\t\t\t\t\"description\": \"Use the Azure Management API to return and establish if the framework pipeline exists in the target Synapse instance, including being deployed.\\n\\nSee: https://docs.microsoft.com/en-us/rest/api/synapse/data-plane/pipeline/getpipeline\",\n\t\t\t\t\t\t\t\t\t\"type\": \"WebActivity\",\n\t\t\t\t\t\t\t\t\t\"dependsOn\": [],\n\t\t\t\t\t\t\t\t\t\"policy\": {\n\t\t\t\t\t\t\t\t\t\t\"timeout\": \"7.00:00:00\",\n\t\t\t\t\t\t\t\t\t\t\"retry\": 0,\n\t\t\t\t\t\t\t\t\t\t\"retryIntervalInSeconds\": 30,\n\t\t\t\t\t\t\t\t\t\t\"secureOutput\": false,\n\t\t\t\t\t\t\t\t\t\t\"secureInput\": false\n\t\t\t\t\t\t\t\t\t},\n\t\t\t\t\t\t\t\t\t\"userProperties\": [],\n\t\t\t\t\t\t\t\t\t\"typeProperties\": {\n\t\t\t\t\t\t\t\t\t\t\"url\": {\n\t\t\t\t\t\t\t\t\t\t\t\"value\": \"https://@{pipeline().DataFactory}.dev.azuresynapse.net/pipelines/@{pipeline().parameters.PipelineName}?api-version=2019-06-01-preview\",\n\t\t\t\t\t\t\t\t\t\t\t\"type\": \"Expression\"\n\t\t\t\t\t\t\t\t\t\t},\n\t\t\t\t\t\t\t\t\t\t\"method\": \"GET\",\n\t\t\t\t\t\t\t\t\t\t\"authentication\": {\n\t\t\t\t\t\t\t\t\t\t\t\"type\": \"MSI\",\n\t\t\t\t\t\t\t\t\t\t\t\"resource\": \"https://management.core.windows.net/\"\n\t\t\t\t\t\t\t\t\t\t}\n\t\t\t\t\t\t\t\t\t}\n\t\t\t\t\t\t\t\t},\n\t\t\t\t\t\t\t\t{\n\t\t\t\t\t\t\t\t\t\"name\": \"Get SYN Pipeline Runs\",\n\t\t\t\t\t\t\t\t\t\"description\": \"Use the Azure Management API to return a list of synapse pipeline runs within the given time window.\\n\\nSee: https://docs.microsoft.com/en-us/rest/api/synapse/data-plane/pipelinerun/querypipelinerunsbyworkspace\",\n\t\t\t\t\t\t\t\t\t\"type\": \"WebActivity\",\n\t\t\t\t\t\t\t\t\t\"dependsOn\": [\n\t\t\t\t\t\t\t\t\t\t{\n\t\t\t\t\t\t\t\t\t\t\t\"activity\": \"Check for Valid SYN Pipeline Name\",\n\t\t\t\t\t\t\t\t\t\t\t\"dependencyConditions\": [\n\t\t\t\t\t\t\t\t\t\t\t\t\"Succeeded\"\n\t\t\t\t\t\t\t\t\t\t\t]\n\t\t\t\t\t\t\t\t\t\t}\n\t\t\t\t\t\t\t\t\t],\n\t\t\t\t\t\t\t\t\t\"policy\": {\n\t\t\t\t\t\t\t\t\t\t\"timeout\": \"7.00:00:00\",\n\t\t\t\t\t\t\t\t\t\t\"retry\": 0,\n\t\t\t\t\t\t\t\t\t\t\"retryIntervalInSeconds\": 30,\n\t\t\t\t\t\t\t\t\t\t\"secureOutput\": false,\n\t\t\t\t\t\t\t\t\t\t\"secureInput\": false\n\t\t\t\t\t\t\t\t\t},\n\t\t\t\t\t\t\t\t\t\"userProperties\": [],\n\t\t\t\t\t\t\t\t\t\"typeProperties\": {\n\t\t\t\t\t\t\t\t\t\t\"url\": {\n\t\t\t\t\t\t\t\t\t\t\t\"value\": \"https://@{pipeline().DataFactory}.dev.azuresynapse.net/queryPipelineRuns?api-version=2019-06-01-preview\",\n\t\t\t\t\t\t\t\t\t\t\t\"type\": \"Expression\"\n\t\t\t\t\t\t\t\t\t\t},\n\t\t\t\t\t\t\t\t\t\t\"method\": \"POST\",\n\t\t\t\t\t\t\t\t\t\t\"body\": {\n\t\t\t\t\t\t\t\t\t\t\t\"value\": \"{\\n \\\"lastUpdatedAfter\\\": \\\"@{adddays(utcnow(),int(variables('QueryRunDays')))}\\\",\\n \\\"lastUpdatedBefore\\\": \\\"@{utcnow()}\\\",\\n \\\"filters\\\": [\\n {\\n \\\"operand\\\": \\\"PipelineName\\\",\\n \\\"operator\\\": \\\"Equals\\\",\\n \\\"values\\\": [\\n \\\"@{pipeline().parameters.PipelineName}\\\"\\n ]\\n }\\n ]\\n}\",\n\t\t\t\t\t\t\t\t\t\t\t\"type\": \"Expression\"\n\t\t\t\t\t\t\t\t\t\t},\n\t\t\t\t\t\t\t\t\t\t\"authentication\": {\n\t\t\t\t\t\t\t\t\t\t\t\"type\": \"MSI\",\n\t\t\t\t\t\t\t\t\t\t\t\"resource\": \"https://management.core.windows.net/\"\n\t\t\t\t\t\t\t\t\t\t}\n\t\t\t\t\t\t\t\t\t}\n\t\t\t\t\t\t\t\t},\n\t\t\t\t\t\t\t\t{\n\t\t\t\t\t\t\t\t\t\"name\": \"Set SYN Runs Output\",\n\t\t\t\t\t\t\t\t\t\"description\": \"Set output to local array for use in downstream filtering and pipeline checks. Use the same array output for both switch cases.\",\n\t\t\t\t\t\t\t\t\t\"type\": \"SetVariable\",\n\t\t\t\t\t\t\t\t\t\"dependsOn\": [\n\t\t\t\t\t\t\t\t\t\t{\n\t\t\t\t\t\t\t\t\t\t\t\"activity\": \"Get SYN Pipeline Runs\",\n\t\t\t\t\t\t\t\t\t\t\t\"dependencyConditions\": [\n\t\t\t\t\t\t\t\t\t\t\t\t\"Succeeded\"\n\t\t\t\t\t\t\t\t\t\t\t]\n\t\t\t\t\t\t\t\t\t\t}\n\t\t\t\t\t\t\t\t\t],\n\t\t\t\t\t\t\t\t\t\"userProperties\": [],\n\t\t\t\t\t\t\t\t\t\"typeProperties\": {\n\t\t\t\t\t\t\t\t\t\t\"variableName\": \"PipelineRuns\",\n\t\t\t\t\t\t\t\t\t\t\"value\": {\n\t\t\t\t\t\t\t\t\t\t\t\"value\": \"@activity('Get SYN Pipeline Runs').output.value\",\n\t\t\t\t\t\t\t\t\t\t\t\"type\": \"Expression\"\n\t\t\t\t\t\t\t\t\t\t}\n\t\t\t\t\t\t\t\t\t}\n\t\t\t\t\t\t\t\t}\n\t\t\t\t\t\t\t]\n\t\t\t\t\t\t}\n\t\t\t\t\t],\n\t\t\t\t\t\"defaultActivities\": [\n\t\t\t\t\t\t{\n\t\t\t\t\t\t\t\"name\": \"Throw Exception Invalid Orchestrator Type\",\n\t\t\t\t\t\t\t\"description\": \"Throw exception if switch cases are not met.\",\n\t\t\t\t\t\t\t\"type\": \"ExecutePipeline\",\n\t\t\t\t\t\t\t\"dependsOn\": [],\n\t\t\t\t\t\t\t\"userProperties\": [],\n\t\t\t\t\t\t\t\"typeProperties\": {\n\t\t\t\t\t\t\t\t\"pipeline\": {\n\t\t\t\t\t\t\t\t\t\"referenceName\": \"Throw Exception\",\n\t\t\t\t\t\t\t\t\t\"type\": \"PipelineReference\"\n\t\t\t\t\t\t\t\t},\n\t\t\t\t\t\t\t\t\"waitOnCompletion\": true,\n\t\t\t\t\t\t\t\t\"parameters\": {\n\t\t\t\t\t\t\t\t\t\"Message\": \"Invalid orchestrator type provided. Unable to check pipeline running state.\"\n\t\t\t\t\t\t\t\t}\n\t\t\t\t\t\t\t}\n\t\t\t\t\t\t}\n\t\t\t\t\t]\n\t\t\t\t}\n\t\t\t},\n\t\t\t{\n\t\t\t\t\"name\": \"Set Query Run Days\",\n\t\t\t\t\"description\": \"Set the query run days value to a local variable for use in various downstream activities.\",\n\t\t\t\t\"type\": \"SetVariable\",\n\t\t\t\t\"dependsOn\": [\n\t\t\t\t\t{\n\t\t\t\t\t\t\"activity\": \"Get Query Run Days Value\",\n\t\t\t\t\t\t\"dependencyConditions\": [\n\t\t\t\t\t\t\t\"Succeeded\"\n\t\t\t\t\t\t]\n\t\t\t\t\t}\n\t\t\t\t],\n\t\t\t\t\"userProperties\": [],\n\t\t\t\t\"typeProperties\": {\n\t\t\t\t\t\"variableName\": \"QueryRunDays\",\n\t\t\t\t\t\"value\": {\n\t\t\t\t\t\t\"value\": \"@activity('Get Query Run Days Value').output.firstRow.PropertyValue\",\n\t\t\t\t\t\t\"type\": \"Expression\"\n\t\t\t\t\t}\n\t\t\t\t}\n\t\t\t}\n\t\t],\n\t\t\"parameters\": {\n\t\t\t\"BatchName\": {\n\t\t\t\t\"type\": \"string\",\n\t\t\t\t\"defaultValue\": \"NotUsed\"\n\t\t\t},\n\t\t\t\"PipelineName\": {\n\t\t\t\t\"type\": \"string\"\n\t\t\t},\n\t\t\t\"ThisRunId\": {\n\t\t\t\t\"type\": \"string\"\n\t\t\t}\n\t\t},\n\t\t\"variables\": {\n\t\t\t\"SubscriptionId\": {\n\t\t\t\t\"type\": \"String\"\n\t\t\t},\n\t\t\t\"RunCount\": {\n\t\t\t\t\"type\": \"String\"\n\t\t\t},\n\t\t\t\"ResourceGroupName\": {\n\t\t\t\t\"type\": \"String\"\n\t\t\t},\n\t\t\t\"OrchestratorType\": {\n\t\t\t\t\"type\": \"String\"\n\t\t\t},\n\t\t\t\"QueryRunDays\": {\n\t\t\t\t\"type\": \"String\"\n\t\t\t},\n\t\t\t\"PipelineRuns\": {\n\t\t\t\t\"type\": \"Array\"\n\t\t\t}\n\t\t},\n\t\t\"folder\": {\n\t\t\t\"name\": \"_ProcFwk/_ProcFwkUtils\"\n\t\t},\n\t\t\"annotations\": [\n\t\t\t\"procfwk\",\n\t\t\t\"Utils\"\n\t\t]\n\t}\n}"
},
{
"path": "Synapse/pipeline/Email Sender.json",
"content": "{\n\t\"name\": \"Email Sender\",\n\t\"properties\": {\n\t\t\"description\": \"Provide a simple abstract over the send email function with request body item exposed as pipeline parameters.\",\n\t\t\"activities\": [\n\t\t\t{\n\t\t\t\t\"name\": \"Send Email\",\n\t\t\t\t\"description\": \"Use an Azure Function to perform an SMTP client email send operation.\",\n\t\t\t\t\"type\": \"AzureFunctionActivity\",\n\t\t\t\t\"dependsOn\": [],\n\t\t\t\t\"policy\": {\n\t\t\t\t\t\"timeout\": \"0.00:10:00\",\n\t\t\t\t\t\"retry\": 0,\n\t\t\t\t\t\"retryIntervalInSeconds\": 30,\n\t\t\t\t\t\"secureOutput\": false,\n\t\t\t\t\t\"secureInput\": false\n\t\t\t\t},\n\t\t\t\t\"userProperties\": [],\n\t\t\t\t\"typeProperties\": {\n\t\t\t\t\t\"functionName\": \"SendEmail\",\n\t\t\t\t\t\"method\": \"POST\",\n\t\t\t\t\t\"body\": {\n\t\t\t\t\t\t\"value\": \"{\\n\\\"emailRecipients\\\": \\\"@{pipeline().parameters.Recipients}\\\",\\n\\\"emailCcRecipients\\\": \\\"@{pipeline().parameters.CcRecipients}\\\",\\n\\\"emailBccRecipients\\\": \\\"@{pipeline().parameters.BccRecipients}\\\",\\n\\\"emailSubject\\\": \\\"@{pipeline().parameters.Subject}\\\",\\n\\\"emailBody\\\": \\\"@{pipeline().parameters.Body}\\\",\\n\\\"emailImportance\\\": \\\"@{pipeline().parameters.Importance}\\\"\\n}\",\n\t\t\t\t\t\t\"type\": \"Expression\"\n\t\t\t\t\t}\n\t\t\t\t},\n\t\t\t\t\"linkedServiceName\": {\n\t\t\t\t\t\"referenceName\": \"FrameworkFunctions\",\n\t\t\t\t\t\"type\": \"LinkedServiceReference\"\n\t\t\t\t}\n\t\t\t}\n\t\t],\n\t\t\"parameters\": {\n\t\t\t\"Recipients\": {\n\t\t\t\t\"type\": \"string\"\n\t\t\t},\n\t\t\t\"CcRecipients\": {\n\t\t\t\t\"type\": \"string\"\n\t\t\t},\n\t\t\t\"BccRecipients\": {\n\t\t\t\t\"type\": \"string\"\n\t\t\t},\n\t\t\t\"Subject\": {\n\t\t\t\t\"type\": \"string\"\n\t\t\t},\n\t\t\t\"Body\": {\n\t\t\t\t\"type\": \"string\"\n\t\t\t},\n\t\t\t\"Importance\": {\n\t\t\t\t\"type\": \"string\"\n\t\t\t}\n\t\t},\n\t\t\"folder\": {\n\t\t\t\"name\": \"_ProcFwk/_ProcFwkUtils\"\n\t\t},\n\t\t\"annotations\": [\n\t\t\t\"procfwk\",\n\t\t\t\"Utils\"\n\t\t]\n\t}\n}"
},
{
"path": "Synapse/pipeline/Intentional Error.json",
"content": "{\n\t\"name\": \"Intentional Error\",\n\t\"properties\": {\n\t\t\"description\": \"Used just so the procfwk has something to call during development.\",\n\t\t\"activities\": [\n\t\t\t{\n\t\t\t\t\"name\": \"Wait1\",\n\t\t\t\t\"description\": \"Framework development worker simulator.\",\n\t\t\t\t\"type\": \"Wait\",\n\t\t\t\t\"dependsOn\": [],\n\t\t\t\t\"userProperties\": [],\n\t\t\t\t\"typeProperties\": {\n\t\t\t\t\t\"waitTimeInSeconds\": {\n\t\t\t\t\t\t\"value\": \"@pipeline().parameters.WaitTime\",\n\t\t\t\t\t\t\"type\": \"Expression\"\n\t\t\t\t\t}\n\t\t\t\t}\n\t\t\t},\n\t\t\t{\n\t\t\t\t\"name\": \"Raise Errors or Not\",\n\t\t\t\t\"description\": \"Framework development worker simulator.\",\n\t\t\t\t\"type\": \"IfCondition\",\n\t\t\t\t\"dependsOn\": [\n\t\t\t\t\t{\n\t\t\t\t\t\t\"activity\": \"Wait1\",\n\t\t\t\t\t\t\"dependencyConditions\": [\n\t\t\t\t\t\t\t\"Succeeded\"\n\t\t\t\t\t\t]\n\t\t\t\t\t}\n\t\t\t\t],\n\t\t\t\t\"userProperties\": [],\n\t\t\t\t\"typeProperties\": {\n\t\t\t\t\t\"expression\": {\n\t\t\t\t\t\t\"value\": \"@equals(pipeline().parameters.RaiseErrors,'true')\",\n\t\t\t\t\t\t\"type\": \"Expression\"\n\t\t\t\t\t},\n\t\t\t\t\t\"ifTrueActivities\": [\n\t\t\t\t\t\t{\n\t\t\t\t\t\t\t\"name\": \"Call Fail Procedure\",\n\t\t\t\t\t\t\t\"type\": \"SqlServerStoredProcedure\",\n\t\t\t\t\t\t\t\"dependsOn\": [],\n\t\t\t\t\t\t\t\"policy\": {\n\t\t\t\t\t\t\t\t\"timeout\": \"0.00:10:00\",\n\t\t\t\t\t\t\t\t\"retry\": 0,\n\t\t\t\t\t\t\t\t\"retryIntervalInSeconds\": 30,\n\t\t\t\t\t\t\t\t\"secureOutput\": false,\n\t\t\t\t\t\t\t\t\"secureInput\": false\n\t\t\t\t\t\t\t},\n\t\t\t\t\t\t\t\"userProperties\": [],\n\t\t\t\t\t\t\t\"typeProperties\": {\n\t\t\t\t\t\t\t\t\"storedProcedureName\": \"[dbo].[FailProcedure]\",\n\t\t\t\t\t\t\t\t\"storedProcedureParameters\": {\n\t\t\t\t\t\t\t\t\t\"RaiseError\": {\n\t\t\t\t\t\t\t\t\t\t\"value\": {\n\t\t\t\t\t\t\t\t\t\t\t\"value\": \"@pipeline().parameters.RaiseErrors\",\n\t\t\t\t\t\t\t\t\t\t\t\"type\": \"Expression\"\n\t\t\t\t\t\t\t\t\t\t},\n\t\t\t\t\t\t\t\t\t\t\"type\": \"String\"\n\t\t\t\t\t\t\t\t\t}\n\t\t\t\t\t\t\t\t}\n\t\t\t\t\t\t\t},\n\t\t\t\t\t\t\t\"linkedServiceName\": {\n\t\t\t\t\t\t\t\t\"referenceName\": \"SupportDatabase\",\n\t\t\t\t\t\t\t\t\"type\": \"LinkedServiceReference\"\n\t\t\t\t\t\t\t}\n\t\t\t\t\t\t}\n\t\t\t\t\t]\n\t\t\t\t}\n\t\t\t}\n\t\t],\n\t\t\"parameters\": {\n\t\t\t\"RaiseErrors\": {\n\t\t\t\t\"type\": \"string\",\n\t\t\t\t\"defaultValue\": \"false\"\n\t\t\t},\n\t\t\t\"WaitTime\": {\n\t\t\t\t\"type\": \"int\",\n\t\t\t\t\"defaultValue\": 5\n\t\t\t}\n\t\t},\n\t\t\"folder\": {\n\t\t\t\"name\": \"_Workers\"\n\t\t},\n\t\t\"annotations\": [\n\t\t\t\"_ProcFwkWorker\"\n\t\t]\n\t}\n}"
},
{
"path": "Synapse/pipeline/Throw Exception.json",
"content": "{\n\t\"name\": \"Throw Exception\",\n\t\"properties\": {\n\t\t\"description\": \"Provide a simple way of throwing an exception within Data Factory using TSQL error handling.\",\n\t\t\"activities\": [\n\t\t\t{\n\t\t\t\t\"name\": \"Raise Error\",\n\t\t\t\t\"description\": \"Using a SQL database to raise an error/exception but wrapped up as a data factory pipeline. Error message information exposed as a pipeline parameter.\",\n\t\t\t\t\"type\": \"Lookup\",\n\t\t\t\t\"dependsOn\": [],\n\t\t\t\t\"policy\": {\n\t\t\t\t\t\"timeout\": \"0.00:10:00\",\n\t\t\t\t\t\"retry\": 0,\n\t\t\t\t\t\"retryIntervalInSeconds\": 30,\n\t\t\t\t\t\"secureOutput\": false,\n\t\t\t\t\t\"secureInput\": false\n\t\t\t\t},\n\t\t\t\t\"userProperties\": [],\n\t\t\t\t\"typeProperties\": {\n\t\t\t\t\t\"source\": {\n\t\t\t\t\t\t\"type\": \"AzureSqlSource\",\n\t\t\t\t\t\t\"sqlReaderQuery\": {\n\t\t\t\t\t\t\t\"value\": \"RAISERROR('@{pipeline().parameters.Message}',16,1);\",\n\t\t\t\t\t\t\t\"type\": \"Expression\"\n\t\t\t\t\t\t},\n\t\t\t\t\t\t\"queryTimeout\": \"02:00:00\",\n\t\t\t\t\t\t\"partitionOption\": \"None\"\n\t\t\t\t\t},\n\t\t\t\t\t\"dataset\": {\n\t\t\t\t\t\t\"referenceName\": \"GetSetMetadata\",\n\t\t\t\t\t\t\"type\": \"DatasetReference\"\n\t\t\t\t\t},\n\t\t\t\t\t\"firstRowOnly\": false\n\t\t\t\t}\n\t\t\t}\n\t\t],\n\t\t\"parameters\": {\n\t\t\t\"Message\": {\n\t\t\t\t\"type\": \"string\"\n\t\t\t}\n\t\t},\n\t\t\"folder\": {\n\t\t\t\"name\": \"_ProcFwk/_ProcFwkUtils\"\n\t\t},\n\t\t\"annotations\": [\n\t\t\t\"procfwk\",\n\t\t\t\"Utils\"\n\t\t]\n\t}\n}"
},
{
"path": "Synapse/pipeline/Wait 1.json",
"content": "{\n\t\"name\": \"Wait 1\",\n\t\"properties\": {\n\t\t\"description\": \"Used just so the procfwk has something to call during development.\",\n\t\t\"activities\": [\n\t\t\t{\n\t\t\t\t\"name\": \"Wait1\",\n\t\t\t\t\"description\": \"Framework development worker simulator.\",\n\t\t\t\t\"type\": \"Wait\",\n\t\t\t\t\"dependsOn\": [],\n\t\t\t\t\"userProperties\": [],\n\t\t\t\t\"typeProperties\": {\n\t\t\t\t\t\"waitTimeInSeconds\": {\n\t\t\t\t\t\t\"value\": \"@pipeline().parameters.WaitTime\",\n\t\t\t\t\t\t\"type\": \"Expression\"\n\t\t\t\t\t}\n\t\t\t\t}\n\t\t\t}\n\t\t],\n\t\t\"parameters\": {\n\t\t\t\"WaitTime\": {\n\t\t\t\t\"type\": \"int\",\n\t\t\t\t\"defaultValue\": 5\n\t\t\t}\n\t\t},\n\t\t\"folder\": {\n\t\t\t\"name\": \"_Workers\"\n\t\t},\n\t\t\"annotations\": [\n\t\t\t\"_ProcFwkWorker\"\n\t\t]\n\t}\n}"
},
{
"path": "Synapse/pipeline/Wait 10.json",
"content": "{\n\t\"name\": \"Wait 10\",\n\t\"properties\": {\n\t\t\"description\": \"Used just so the procfwk has something to call during development.\",\n\t\t\"activities\": [\n\t\t\t{\n\t\t\t\t\"name\": \"Wait10\",\n\t\t\t\t\"description\": \"Framework development worker simulator.\",\n\t\t\t\t\"type\": \"Wait\",\n\t\t\t\t\"dependsOn\": [],\n\t\t\t\t\"userProperties\": [],\n\t\t\t\t\"typeProperties\": {\n\t\t\t\t\t\"waitTimeInSeconds\": {\n\t\t\t\t\t\t\"value\": \"@pipeline().parameters.WaitTime\",\n\t\t\t\t\t\t\"type\": \"Expression\"\n\t\t\t\t\t}\n\t\t\t\t}\n\t\t\t}\n\t\t],\n\t\t\"parameters\": {\n\t\t\t\"WaitTime\": {\n\t\t\t\t\"type\": \"int\",\n\t\t\t\t\"defaultValue\": 5\n\t\t\t}\n\t\t},\n\t\t\"folder\": {\n\t\t\t\"name\": \"_Workers\"\n\t\t},\n\t\t\"annotations\": [\n\t\t\t\"_ProcFwkWorker\"\n\t\t]\n\t}\n}"
},
{
"path": "Synapse/pipeline/Wait 2.json",
"content": "{\n\t\"name\": \"Wait 2\",\n\t\"properties\": {\n\t\t\"description\": \"Used just so the procfwk has something to call during development.\",\n\t\t\"activities\": [\n\t\t\t{\n\t\t\t\t\"name\": \"Wait2\",\n\t\t\t\t\"description\": \"Framework development worker simulator.\",\n\t\t\t\t\"type\": \"Wait\",\n\t\t\t\t\"dependsOn\": [],\n\t\t\t\t\"userProperties\": [],\n\t\t\t\t\"typeProperties\": {\n\t\t\t\t\t\"waitTimeInSeconds\": {\n\t\t\t\t\t\t\"value\": \"@pipeline().parameters.WaitTime\",\n\t\t\t\t\t\t\"type\": \"Expression\"\n\t\t\t\t\t}\n\t\t\t\t}\n\t\t\t}\n\t\t],\n\t\t\"parameters\": {\n\t\t\t\"WaitTime\": {\n\t\t\t\t\"type\": \"int\",\n\t\t\t\t\"defaultValue\": 5\n\t\t\t}\n\t\t},\n\t\t\"folder\": {\n\t\t\t\"name\": \"_Workers\"\n\t\t},\n\t\t\"annotations\": [\n\t\t\t\"_ProcFwkWorker\"\n\t\t]\n\t}\n}"
},
{
"path": "Synapse/pipeline/Wait 3.json",
"content": "{\n\t\"name\": \"Wait 3\",\n\t\"properties\": {\n\t\t\"description\": \"Used just so the procfwk has something to call during development.\",\n\t\t\"activities\": [\n\t\t\t{\n\t\t\t\t\"name\": \"Wait3\",\n\t\t\t\t\"description\": \"Framework development worker simulator.\",\n\t\t\t\t\"type\": \"Wait\",\n\t\t\t\t\"dependsOn\": [],\n\t\t\t\t\"userProperties\": [],\n\t\t\t\t\"typeProperties\": {\n\t\t\t\t\t\"waitTimeInSeconds\": {\n\t\t\t\t\t\t\"value\": \"@pipeline().parameters.WaitTime\",\n\t\t\t\t\t\t\"type\": \"Expression\"\n\t\t\t\t\t}\n\t\t\t\t}\n\t\t\t}\n\t\t],\n\t\t\"parameters\": {\n\t\t\t\"WaitTime\": {\n\t\t\t\t\"type\": \"int\",\n\t\t\t\t\"defaultValue\": 5\n\t\t\t}\n\t\t},\n\t\t\"folder\": {\n\t\t\t\"name\": \"_Workers\"\n\t\t},\n\t\t\"annotations\": [\n\t\t\t\"_ProcFwkWorker\"\n\t\t]\n\t}\n}"
},
{
"path": "Synapse/pipeline/Wait 4.json",
"content": "{\n\t\"name\": \"Wait 4\",\n\t\"properties\": {\n\t\t\"description\": \"Used just so the procfwk has something to call during development.\",\n\t\t\"activities\": [\n\t\t\t{\n\t\t\t\t\"name\": \"Wait4\",\n\t\t\t\t\"description\": \"Framework development worker simulator.\",\n\t\t\t\t\"type\": \"Wait\",\n\t\t\t\t\"dependsOn\": [],\n\t\t\t\t\"userProperties\": [],\n\t\t\t\t\"typeProperties\": {\n\t\t\t\t\t\"waitTimeInSeconds\": {\n\t\t\t\t\t\t\"value\": \"@pipeline().parameters.WaitTime\",\n\t\t\t\t\t\t\"type\": \"Expression\"\n\t\t\t\t\t}\n\t\t\t\t}\n\t\t\t}\n\t\t],\n\t\t\"parameters\": {\n\t\t\t\"WaitTime\": {\n\t\t\t\t\"type\": \"int\",\n\t\t\t\t\"defaultValue\": 5\n\t\t\t}\n\t\t},\n\t\t\"folder\": {\n\t\t\t\"name\": \"_Workers\"\n\t\t},\n\t\t\"annotations\": [\n\t\t\t\"_ProcFwkWorker\"\n\t\t]\n\t}\n}"
},
{
"path": "Synapse/pipeline/Wait 5.json",
"content": "{\n\t\"name\": \"Wait 5\",\n\t\"properties\": {\n\t\t\"description\": \"Used just so the procfwk has something to call during development.\",\n\t\t\"activities\": [\n\t\t\t{\n\t\t\t\t\"name\": \"Wait5\",\n\t\t\t\t\"description\": \"Framework development worker simulator.\",\n\t\t\t\t\"type\": \"Wait\",\n\t\t\t\t\"dependsOn\": [],\n\t\t\t\t\"userProperties\": [],\n\t\t\t\t\"typeProperties\": {\n\t\t\t\t\t\"waitTimeInSeconds\": {\n\t\t\t\t\t\t\"value\": \"@pipeline().parameters.WaitTime\",\n\t\t\t\t\t\t\"type\": \"Expression\"\n\t\t\t\t\t}\n\t\t\t\t}\n\t\t\t}\n\t\t],\n\t\t\"parameters\": {\n\t\t\t\"WaitTime\": {\n\t\t\t\t\"type\": \"int\",\n\t\t\t\t\"defaultValue\": 5\n\t\t\t}\n\t\t},\n\t\t\"folder\": {\n\t\t\t\"name\": \"_Workers\"\n\t\t},\n\t\t\"annotations\": [\n\t\t\t\"_ProcFwkWorker\"\n\t\t]\n\t}\n}"
},
{
"path": "Synapse/pipeline/Wait 6.json",
"content": "{\n\t\"name\": \"Wait 6\",\n\t\"properties\": {\n\t\t\"description\": \"Used just so the procfwk has something to call during development.\",\n\t\t\"activities\": [\n\t\t\t{\n\t\t\t\t\"name\": \"Wait6\",\n\t\t\t\t\"description\": \"Framework development worker simulator.\",\n\t\t\t\t\"type\": \"Wait\",\n\t\t\t\t\"dependsOn\": [],\n\t\t\t\t\"userProperties\": [],\n\t\t\t\t\"typeProperties\": {\n\t\t\t\t\t\"waitTimeInSeconds\": {\n\t\t\t\t\t\t\"value\": \"@pipeline().parameters.WaitTime\",\n\t\t\t\t\t\t\"type\": \"Expression\"\n\t\t\t\t\t}\n\t\t\t\t}\n\t\t\t}\n\t\t],\n\t\t\"parameters\": {\n\t\t\t\"WaitTime\": {\n\t\t\t\t\"type\": \"int\",\n\t\t\t\t\"defaultValue\": 5\n\t\t\t}\n\t\t},\n\t\t\"folder\": {\n\t\t\t\"name\": \"_Workers\"\n\t\t},\n\t\t\"annotations\": [\n\t\t\t\"_ProcFwkWorker\"\n\t\t]\n\t}\n}"
},
{
"path": "Synapse/pipeline/Wait 7.json",
"content": "{\n\t\"name\": \"Wait 7\",\n\t\"properties\": {\n\t\t\"description\": \"Used just so the procfwk has something to call during development.\",\n\t\t\"activities\": [\n\t\t\t{\n\t\t\t\t\"name\": \"Wait7\",\n\t\t\t\t\"description\": \"Framework development worker simulator.\",\n\t\t\t\t\"type\": \"Wait\",\n\t\t\t\t\"dependsOn\": [],\n\t\t\t\t\"userProperties\": [],\n\t\t\t\t\"typeProperties\": {\n\t\t\t\t\t\"waitTimeInSeconds\": {\n\t\t\t\t\t\t\"value\": \"@pipeline().parameters.WaitTime\",\n\t\t\t\t\t\t\"type\": \"Expression\"\n\t\t\t\t\t}\n\t\t\t\t}\n\t\t\t}\n\t\t],\n\t\t\"parameters\": {\n\t\t\t\"WaitTime\": {\n\t\t\t\t\"type\": \"int\",\n\t\t\t\t\"defaultValue\": 5\n\t\t\t}\n\t\t},\n\t\t\"folder\": {\n\t\t\t\"name\": \"_Workers\"\n\t\t},\n\t\t\"annotations\": [\n\t\t\t\"_ProcFwkWorker\"\n\t\t]\n\t}\n}"
},
{
"path": "Synapse/pipeline/Wait 8.json",
"content": "{\n\t\"name\": \"Wait 8\",\n\t\"properties\": {\n\t\t\"description\": \"Used just so the procfwk has something to call during development.\",\n\t\t\"activities\": [\n\t\t\t{\n\t\t\t\t\"name\": \"Wait8\",\n\t\t\t\t\"description\": \"Framework development worker simulator.\",\n\t\t\t\t\"type\": \"Wait\",\n\t\t\t\t\"dependsOn\": [],\n\t\t\t\t\"userProperties\": [],\n\t\t\t\t\"typeProperties\": {\n\t\t\t\t\t\"waitTimeInSeconds\": {\n\t\t\t\t\t\t\"value\": \"@pipeline().parameters.WaitTime\",\n\t\t\t\t\t\t\"type\": \"Expression\"\n\t\t\t\t\t}\n\t\t\t\t}\n\t\t\t}\n\t\t],\n\t\t\"parameters\": {\n\t\t\t\"WaitTime\": {\n\t\t\t\t\"type\": \"int\",\n\t\t\t\t\"defaultValue\": 5\n\t\t\t}\n\t\t},\n\t\t\"folder\": {\n\t\t\t\"name\": \"_Workers\"\n\t\t},\n\t\t\"annotations\": [\n\t\t\t\"_ProcFwkWorker\"\n\t\t]\n\t}\n}"
},
{
"path": "Synapse/pipeline/Wait 9.json",
"content": "{\n\t\"name\": \"Wait 9\",\n\t\"properties\": {\n\t\t\"description\": \"Used just so the procfwk has something to call during development.\",\n\t\t\"activities\": [\n\t\t\t{\n\t\t\t\t\"name\": \"Wait9\",\n\t\t\t\t\"description\": \"Framework development worker simulator.\",\n\t\t\t\t\"type\": \"Wait\",\n\t\t\t\t\"dependsOn\": [],\n\t\t\t\t\"userProperties\": [],\n\t\t\t\t\"typeProperties\": {\n\t\t\t\t\t\"waitTimeInSeconds\": {\n\t\t\t\t\t\t\"value\": \"@pipeline().parameters.WaitTime\",\n\t\t\t\t\t\t\"type\": \"Expression\"\n\t\t\t\t\t}\n\t\t\t\t}\n\t\t\t}\n\t\t],\n\t\t\"parameters\": {\n\t\t\t\"WaitTime\": {\n\t\t\t\t\"type\": \"int\",\n\t\t\t\t\"defaultValue\": 15\n\t\t\t}\n\t\t},\n\t\t\"folder\": {\n\t\t\t\"name\": \"_Workers\"\n\t\t},\n\t\t\"annotations\": [\n\t\t\t\"_ProcFwkWorker\"\n\t\t]\n\t}\n}"
},
{
"path": "Synapse/trigger/FunctionalTestingTrigger.json",
"content": "{\n\t\"name\": \"FunctionalTestingTrigger\",\n\t\"properties\": {\n\t\t\"description\": \"Used for functional testing of the framework in a dedicated environment.\",\n\t\t\"annotations\": [\n\t\t\t\"procfwk\"\n\t\t],\n\t\t\"runtimeState\": \"Stopped\",\n\t\t\"pipelines\": [\n\t\t\t{\n\t\t\t\t\"pipelineReference\": {\n\t\t\t\t\t\"referenceName\": \"01-Grandparent\",\n\t\t\t\t\t\"type\": \"PipelineReference\"\n\t\t\t\t}\n\t\t\t}\n\t\t],\n\t\t\"type\": \"ScheduleTrigger\",\n\t\t\"typeProperties\": {\n\t\t\t\"recurrence\": {\n\t\t\t\t\"frequency\": \"Hour\",\n\t\t\t\t\"interval\": 2,\n\t\t\t\t\"startTime\": \"2020-04-06T15:00:00Z\",\n\t\t\t\t\"timeZone\": \"UTC\"\n\t\t\t}\n\t\t}\n\t}\n}"
}
]