Describe the solution you'd like
Add subscription ID in description of All-in-one.

Starting from 27 September 2021 at 15:30 UTC, portal private endpoint for Azure Purview requires a separate Azure Private DNS Zone (privatelink.purviewstudio.azure.com).

For ESA, this means we should deploy a new Azure Private DNS zones in Data Management subscription:

Purview account name A record should be added to privatelink.purview.azure.com zone and Web A record must be added to privatelink.purviewstudio.azure.com zone.

in general, 5 Azure Private DNS zones are required for Azure Purview for account, portal and ingestion PEs:

Describe the solution you'd like
Add Outputs for managed Purview resources to further enhance the Purview integration in Data Landing Zone and Data Products. This will allow to automatically host private endpoints for Purview on managed vnets for ADF within the Data Landing Zone and potentially also within the Data Products.

Some deployments may fail with GHA due to a issue in the Azure ARM Action for GHA: Azure/arm-deploy#33

Not urgent. Logging this for post-vacation analysis, @marvinbuss

Since when you select the above option, it uses the existing firewall and does not create a new one, but throws an error after deployment (dependency issue?).

The issue:

When deploying to using an existing Firewall inside of an existing ESLZ setup, if during ESLZ deployment the DNS proxy was not enabled, the data management landing zone deployment fails as it requires the DNS proxy to be enabled. It might be an idea to include a warning here that please ensure DNS proxy is enabled in your existing firewall policy as seen below otherwise deployment will fail.

Describe the bug
Some users are seeing the following error message when deploying ESA through the Portal experience:

"Failed to get resource provider Microsoft.EventHub, requestId: <ID>. Exception: (Exception) ErrorCode:AuthorizationFailed. Message:The client '<clientId>' with object id '<objectId>' does not have authorization to perform action 'Microsoft.Resources/subscriptions/providers/read' over scope '/subscriptions/<subscriptionId>' or the scope is invalid. If access was recently granted, please refresh your credentials.. Target:.."

Steps to reproduce

1. Deploy ESA through all-in-one deployment or deploy Data Management Zone and make sure that Microsoft.EventHub and Microsoft.Storage are not registered.
2. Deploy them through the Deploy to Azure Button

The policy name and description on this doesn't seem to be right:

https://github.com/Azure/data-management-zone/blob/main/infra/Policies/SqlManagedInstance/params.policyDefinition.Deny-SqlManagedInstance-Sku.json

It's identical to another policy.

I guess the name should be "Deny-SqlManagedInstance-Sku" and description something along the lines of enforcing certain SKUs.

Describe the solution you'd like
From a security standpoint it would be better to rely on a base64 encoding for the custom script installation instead of installing the script from a public Storage Account. This reduces the attack surface and also simplifies this template.

Given the fact that when Key Vault resources are deleted, they are retained for x amount of days because of soft-delete capability in the product, you cannot remove the existing deployment and create new resources with the same name until the soft-delete period. Please consider randomizing the resources names so customers can redeploy the environment if they ever need to start over again.

correlation id: f874e407-556b-44d6-8af0-a5409fb27f70

error regarding purview deployment, no more purview instances allowed:

{ "status": "Failed", "error": { "code": "DeploymentFailed", "message": "At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/DeployOperations for usage details.", "details": [ { "code": "BadRequest", "message": "{\r\n \"error\": {\r\n \"code\": \"InvalidTemplateDeployment\",\r\n \"message\": \"The template deployment 'purview001' is not valid according to the validation procedure. The tracking id is 'c307c7c9-4e94-4d31-8c13-90179a34698a'. See inner errors for details.\",\r\n \"details\": [\r\n {\r\n \"code\": \"2005\",\r\n \"message\": \"Tenant 72f988bf-86f1-41af-91ab-2d7cd011db47 with 100 accounts has surpassed its resource quota for southeastasia location. Please try creating in other available locations or contact support.\"\r\n }\r\n ]\r\n }\r\n}" } ] } }

Region: South East Asia

I'm seeing this error when I try to deploy into a Enterprise-scale environment:

{
  "code": "SubscriptionNotFound",
  "message": "The subscription 'grp-azfwpolicy-eastus2' could not be found."
}

Screenshot of the configuration:

Wrong path to the Vnet Peering deploy and param file in the networkDeployment.yml in the step #deploy vnet peering 001

The deployment of the dataManagement zone was successful, but not the one for the dataLandingZone.
Correlation ID: f953f92e-3406-4414-99c2-bec526a01554

Describe the solution you'd like
Add a warning in the ESA Portal deployment to make sure users check the RP registration for various services like EventHub, Storage, etc.
Alternatively, register RP via POST Call.

Generate public ssh key in workflows for Linux DNS forwarder

Describe the solution you'd like
Make the Private DNS ID an optional parameter for sub deployments (RG level deployments).

When using the "Deploy to Azure" button, the custom template asks for a "Data Management Zone Prefix" in the second step. In the description of the field or in the info tooltip, it might be helpful to mention that this prefix should be globally unique.

In the workshops we have run, one of the common errors that users encounter is that they try to use a very common prefix like dmz or dataman. I know we don't have a way for the UI to check for global uniqueness, but if we put a note on the screen, it might prompt the user to think a little harder about the prefix they select. This could help prevent a very common issue.

Add custom policy definitions to Enterprise-Scale Landing Zones and automatically enforce necessary policies:

• Synapse (Azure/Enterprise-Scale#756)
• Databricks (Azure/Enterprise-Scale#757)
• Search (Azure/Enterprise-Scale#758)
• Data Factory (Azure/Enterprise-Scale#759)
• Stream Analytics (Azure/Enterprise-Scale#760)
• IoTHub (Azure/Enterprise-Scale#761)
• EventHub (Azure/Enterprise-Scale#762)
• Data Explorer (Azure/Enterprise-Scale#763)
• Cosmos DB (Azure/Enterprise-Scale#764)
• HDInsight (Azure/Enterprise-Scale#765)
• Storage
• Batch
• Diagnostics
• KeyVault
• MariaDB
• MySQL
• PostgreSQL
• PrivateDNSZoneGroups
• Private Endpoint
• SQL
• SQL InstancePools
• SQL Managed Instance
• Storage

It will be really useful and productive to have a pre-provisioned Jump Box within Data Landing Shared Product Virtual Network for quicker adoption and get cracking straightaway with any labs / workshops or Demo purposes.

• Remove NSG rules for public access.
• Update comment in Update Parameters Process docs for ADO. ADO users were thinking that they only have to update the ADO workflow file. @xigyenge
• Update Data Landing Zone Name to something else in data domain and data product
• Separate workflows into separate files: This requires discussions and is a question of simplicity vs best practice
• Be clear on the subnet resource ID in domains and products. @xigyenge
• Look into Purview issue
• Create Parameter for Vnet CIDR range
• Add supported regions to Docs @xigyenge
• Be clear on the Synapse Storage Account and filesystem variable @xigyenge
• Error purview in known issues - Data Hub repo only@xigyenge

Describe the bug
I used the "Deploy to Azure" button to deploy the "Enterprise-Scale Analytics" template. On the fourth step, I tried to add tags to my deployment. When I clicked the "Create" button, I received a template validation error.

When I removed my tags, the process ran correctly.

Steps to reproduce

1. Use the portal.enterpriseScaleAnalytics.json template
2. Add tags to the deployment
3. Get validation error when creating the resources

Error message

{"code":"InvalidTemplate","message":"Deployment template parse failed: 'Error reading string. Unexpected token: StartObject. Path 'EnterpriseScaleAnalytics'.'."}

Screenshots

• Power BI Gateway
• SelfHostetAgent GitHub
• DNS Forwarder

Describe the solution you'd like
Bundle the Bastion host deployment with the simple E2E deployment button, to make it even easier for users to get started inside the Data Landing Zone and Data Management Zone.

Propblematic policies from Enterprise Scale:

• Deny Vnet peering
• https://github.com/Azure/enterprise-scale-csyou/blob/main/azopsreference/3fc1081d-6105-4e19-b60c-1ec1252cf560%20(3fc1081d-6105-4e19-b60c-1ec1252cf560)/contoso%20(contoso)/.AzState/Microsoft.Authorization_policyDefinitions-Deny-ERPeering.parameters.json

IoTHub:

• Microsoft.Security/IoTSecuritySolutions does not work when deploying via Policy

Synapse:

• No effect with Microsoft.Synapse/workspaces/managedVirtualNetworkSettings.linkedAccessCheckOnTargetResource
• No effect with Microsoft.Synapse/workspaces/bigDataPools/defaultSparkLogFolder
• Internal Server Error with Microsoft.Synapse/workspaces/sqlPools/metadataSync

{
    "status": "Failed",
    "error": {
        "code": "HttpWrapOperationAsyncFailed",
        "message": "System.InvalidOperationException : The stream was already consumed. It cannot be read again.\r\n   at System.Net.Http.StreamContent.PrepareContent()\r\n   at System.Net.Http.StreamContent.SerializeToStreamAsync(Stream stream, TransportContext context)\r\n   at System.Net.Http.HttpContent.CopyToAsync(Stream stream, TransportContext context)\r\n   at System.Net.Http.HttpClientHandler.GetRequestStreamCallback(IAsyncResult ar)\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Analytics.Common.Http.RequestResponseLoggingHandler.<SendAsync>d__6.MoveNext() in S:\\src\\common\\Microsoft.Analytics.Common\\Http\\RequestResponseLoggingHandler.cs:line 44\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Analytics.Common.PartnerApi.SubscriptionIdBasedAuthorizationHeaderAppenderForArm.<SendAsync>d__8.MoveNext() in S:\\src\\common\\Microsoft.Analytics.Common\\PartnerApi\\SubscriptionIdBasedAuthorizationHeaderAppenderForArm.cs:line 67\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Analytics.Common.Http.RetryWebRequestHandler.<>c__DisplayClass12_0.<<SendAsync>b__0>d.MoveNext() in S:\\src\\common\\Microsoft.Analytics.Common\\Http\\RetryWebRequestHandler.cs:line 150\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Analytics.Common.Extensions.RetryExtensions.<>c__DisplayClass4_0`2.<<ExecuteFuncWithRetry>b__0>d.MoveNext() in S:\\src\\common\\Microsoft.Analytics.Common\\Extensions\\RetryExtensions.cs:line 0\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Analytics.Common.Extensions.RetryExtensions.<ExecuteFuncWithRetry>d__4`2.MoveNext() in S:\\src\\common\\Microsoft.Analytics.Common\\Extensions\\RetryExtensions.cs:line 0\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Analytics.Common.PartnerApi.SqlResourceProviderClient.<Microsoft-Analytics-Common-PartnerApi-ISqlResourceProviderClient-ProxyExecutePutActionAsync>d__26.MoveNext() in S:\\src\\common\\Microsoft.Analytics.Common\\PartnerApi\\SqlResourceProviderClient.cs:line 293\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Analytics.Common.PartnerApi.SqlManagedResourceGroupResourceProviderClient.<Microsoft-Analytics-Common-PartnerApi-ISqlResourceProviderClient-ProxyExecutePutActionAsync>d__26.MoveNext() in S:\\src\\common\\Microsoft.Analytics.Common\\PartnerApi\\SqlManagedResourceGroupResourceProviderClient.cs:line 204\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Analytics.RpApiSvc.Controllers.SqlCompute.ExecutePutAction.<PerformSqlRpOperation>d__1.MoveNext() in S:\\src\\services\\rpc\\Microsoft.Analytics.RpApiSvc\\Controllers\\SqlCompute\\ExecutePutAction.cs:line 35\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Analytics.RpApiSvc.Controllers.SqlCompute.SqlComputeAsyncProxyOperationBase.<Microsoft-Analytics-RpApiSvc-Controllers-IRpRequest-Process>d__15.MoveNext() in S:\\src\\services\\rpc\\Microsoft.Analytics.RpApiSvc\\Controllers\\SqlCompute\\SqlComputeAsyncProxyOperationBase.cs:line 88\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Analytics.Common.Http.BaseController.<HttpWrapOperationAsync>d__9.MoveNext() in S:\\src\\common\\Microsoft.Analytics.Common\\Http\\BaseController.cs:line 53"
    }
}

DataFactory:

• No effect with Microsoft.DataFactory/factories/publicNetworkAccess

EventHub:

• Microsoft.EventHub/namespaces/maximumThroughputUnits does not work porperly

All Repos:

• Fixed that the prefix for the DLZ can be larger than 11 characters. Previously, it has taken max. 12 chars.
• Added additional guidance to the instructions in the PR
• Role Assignment section requires more guidance and explanation
• Tags

Data Hub:

• continue-on-error for Purview in ADO and GHA
• Fix type in README dataNodeDeployment.yml
• Reduced prefix length to 30 of public IP prefix resource
• Global DNS error: GHA is failing even though Deployment successfully completes - Possible Root Cause identified by engineers and working now on the fix
• Purview Issue: Authentication and missing claims 'puid' or 'altsecid' or 'oid'

Data Node:

• Removed role assignments from Synapse and ADF
• Removed linkedServices, which requires role assignment
• continue-on-error for SHIR sharing in ADO and GHA
• more details added on the Role Assignment section from readme file
• added comment on updateParameter.yml file regarding purview_Id value to be left empty string in case none is deployed
• Regional issues: Discuss issue with team

Data Domain - Batch:

Data Domain - Streaming:

Data Product - Analytics:

Data Product - Reporting:

There are open compliance tasks that need to be reviewed for your data-hub repo.

Action required: 4 compliance tasks

To bring this repository to the standard required for 2021, we require administrators of this and all Microsoft GitHub repositories to complete a small set of tasks within the next 60 days. This is critical work to ensure the compliance and security of your Azure GitHub organization.

Please take a few minutes to complete the tasks at: https://repos.opensource.microsoft.com/orgs/Azure/repos/data-hub/compliance

• The GitHub AE (GitHub inside Microsoft) migration survey has not been completed for this private repository
• No Service Tree mapping has been set for this repo. If this team does not use Service Tree, they can also opt-out of providing Service Tree data in the Compliance tab.
• No repository maintainers are set. The Open Source Maintainers are the decision-makers and actionable owners of the repository, irrespective of administrator permission grants on GitHub.
• Classification of the repository as production/non-production is missing in the Compliance tab.

You can close this work item once you have completed the compliance tasks, or it will automatically close within a day of taking action.

If you no longer need this repository, it might be quickest to delete the repo, too.

GitHub inside Microsoft program information

More information about GitHub inside Microsoft and the new GitHub AE product can be found at https://aka.ms/gim or by contacting [email protected]

FYI: current admins at Microsoft include @marvinbuss, @daltondhcp, @esbran

• Add Firewall rules
• Add Route Tables and Route Table rules
• Add Public IP Prefixes for Azure Firewall
• Add NSG base rules (Overwrite baseline rules etc.)
• Add ANM

Change Self hosted agents from VMSS to VMs with availability sets, if the behavior stays the same.

Describe the solution you'd like
Add a script for cleanup of all resources after testing the environment.

$filter = 'prefix-'
Get-AzResourceGroup | ? ResourceGroupName -match $filter | Select-Object ResourceGroupName

$filter = 'prefix-'
Get-AzResourceGroup | ? ResourceGroupName -match $filter | Remove-AzResourceGroup -AsJob -Force

@marvinbuss - in the networkDeployment.yml, we should deploy the storage account in the integration resource group, not in the Management, right? Please confirm if so :)
Currently in the ado workflow the value from the storage rg is
resourceGroupName: '$(AZURE_RESOURCE_GROUP_NAME_MANAGEMENT)'

Describe the bug
In the "Update Parameters" process, many of the script input parameters and variables have names that refer to a Data Landing Zone when they actually control the Data Management Zone that is being created.

Describe the solution you'd like
Improve the network range validation in the portal for unexperienced users. Check the following:

• Overlap of address ranges (Subnet within Vnet)
• Overlap of address ranges (Overlapping Subnets)

It would be nice if registering required Resource Providers for deploying Azure Purview can be included in the deployment templates:

• Microsoft.Purview
• Microsoft.Storage
• Microsoft.EventHub

note: in case you consider including this feature SPN may require additional role.

Data Hub:

• Deployment automation for new Domains, Products and Landing Zones - In Progress
  Clone repo (find solution)
• Network Manager Config
• Update self-hosted Agents - In Progress
• Move templates to Node (SHIR, ADF, )
• Add automation to workflows (LogicApp, Automation, Scripts, etc.)
• Power BI Gateway Template
• Add Key Vault access for Purview

Data Node

• ADB Hive to Atlas Synch (on time schedule) - In Progress
• Add Purview souces through Powershell - In Progress
• Data Tokenization
• Databricks automation - blocked
• Databricks SCIM automation - blocked
• Databricks Private Link - In Progress, testing required
• Automated Ingestion Framework
• Add Share SHIR Template
• Evaluate new storage feature
• Remove Triggers again from Node
• Add Synapse PW to KV (all templates)

Data Domain Batch

Data Domain Streaming:

• IoT Hub tests - Done

Data Product Analytics:

• Test AML Private Clusters and get them working - In Progress/Blocked (Support)
• Add Databricks Connection Template

Other tasks:

• Create Purview presentation
• Documentation for Domains and Products
• Testing of Domains and Products
• Domain and Product: Convert Subnet to private link subnet
• Check RG names in Products and domains and check for //

• Add private DNS zone for dfs

• Use Key Vault backed secret scope (mspnp/spark-monitoring#96)
• Upgrade to Spark 3.0.0 and 3.0.1 support (mspnp/spark-monitoring#92)
  https://github.com/mspnp/spark-monitoring

Discussed in #137

Describe the solution you'd like
Add some guidance describing how users can host a self-hosted agent in their Data Management Zone or Data Landing Zone. This should include:

• Short doc describing the steps that need to be followed
• Automation of Setup via ARM/Bicep for ADO
• Portal UI for Self-hosted Agent for ADO

Describe the bug
I am not able to deploy Enterprise Scale Analytics ("ESA") in Azure Gov. The deployment template does not allow me to choose any region.

Question: Is ESA supported in Azure Gov? If not, is there an ETA?

Steps to reproduce

1. Navigate to https://portal.azure.us/#blade/Microsoft_Azure_CreateUIDef/CustomDeploymentBlade/uri/https%3A%2F%2Fraw.githubusercontent.com%2FAzure%2Fdata-management-zone%2Fmain%2Fdocs%2Freference%2FenterpriseScaleAnalytics.json/uiFormDefinitionUri/https%3A%2F%2Fraw.githubusercontent.com%2FAzure%2Fdata-management-zone%2Fmain%2Fdocs%2Freference%2Fportal.enterpriseScaleAnalytics.json

• Note that I had to change from portal.azure.com to portal.azure.us

2. Proceed to "Data Management Zone" tab
3. Expand "Location" dropdown

Screenshots

Describe the bug
all 5 Private Endpoints for Azure Purview are deployed inside the governance resource group, however, only account and portal private endpoints are linked to Azure Purview. the ingestion private endpoints are not linked. we need to use them to enable network isolation for end-to-end scan scenarios.

Steps to reproduce

1. Deploy Azure Purview using ARM template or deploy the entire Data Management Landing Zone.
2. From Azure Portal open Azure Purview blade \ Networking/. Validate if private links are listed in Ingestion private endpoint connections tab.

Screenshots

Describe the solution you'd like
Please create contribution guidelines for users to help them get started contributing to the repository.

• Technical details wrt how to make changes, engineering principles, contributing guidelines (@sasever)
• Improve PR Templates (@marvinbuss)

Discussed in #133

I wasn't sure if the policy descriptions for these align with the policies?

• Deny-Storage-RoutingPreference

• Deny-Storage-Kind

• Deny-Storage-NetworkAclsDefaultAction

• Deny-Synapse-AllowedAadTenantIdsForLinking

Update reference ARM template with Resource Group deployment template.

Describe the solution you'd like
When deploying the Data Management Zone through GH Actions or Azure DevOps, the deployment is executed through a Service Principle. This has the effect that only the Service Principle gets added as Root Collection Admin in Purview. To overcome this, we need to add the option to add additional Users as Root Collection Admins in Purview.

Missing Policies:

• Create DNS A records
• Create retention policies for SQL DB etc.
• Share SHIR
• SQL Auditing settings (Azure SQL, Synapse, etc.)
• Azure Defender for SQL (Azure SQL, Synapse, etc.)
• Azure Synapse encryption with cmk
• Storage encryption with cmk
• AML encryption with cmk
• TDE synapse SQL pools
• Append to deploy for managedIdentitySqlControlSettings
• dataFlowProperties and other computeProperties for Microsoft.Synapse/workspaces/integrationRuntimes/Managed.typeProperties.
• Databricks: requireInfrastructureEncryption, prepareEncryption, encryption, ...
• Cosmos Network settings: IPs and virtual networks --> only PEs
• Cosmos Customer managed Keys
  ...

Add Key Vault and store Log Analytics Workspace secrets in it, such as Log Analytics Workspace ID and Log Analytics Workspace Key.

Describe the bug
I used the "Deploy to Azure" button to deploy the "Enterprise-Scale Analytics" template. My first deployment ran successfully. I wanted to run the template again, but I couldn't because the deployment name dataManagementZoneDeployment was already in use.

Can we add a random string to the end of the deployment name so the names are unique?

Steps to reproduce

1. Use the portal.enterpriseScaleAnalytics.json template to run a deployment
2. Use the same template again to create a second deployment
3. Get validation error

Error message

{"code":"InvalidDeploymentLocation","message":"Invalid deployment location 'eastus2'. The deployment 'dataManagementZoneDeployment' already exists in location 'centralus'."}

Screenshots