Git Product home page Git Product logo

mcw-migrate-edw-to-azure-sql-data-warehouse's Introduction

Migrate EDW to Azure SQL Data Warehouse

This workshop is archived and is no longer being maintained. Content is read-only.

Coho, a retail company focusing on consumer electronics, is modernizing their data architecture. Critical to this effort is migrating their existing enterprise data warehouse to the cloud for better integration with their cloud native customer 360 project and self-service business intelligence for their people in the field.

October 2019

Target audience

  • Database Administrators
  • Database Developers
  • Data Architects

Abstracts

Workshop

In this workshop, you will look at the process of migrating an on-premises data warehouse to Azure SQL Data Warehouse. Throughout the whiteboard design session and hands-on lab, you will look at the planning process for data warehouse migration, identifying schema and data incompatibilities, efficiently migrating data from on-premises databases to the cloud, data distribution in Azure SQL Data Warehouse, migrating ETL jobs to Azure Data Factory, and supporting ad-hoc workloads in an Azure SQL Data Warehouse through Azure Analysis Services.

At the end of this workshop, you will be better able to plan and implement a migration of your existing on-premises enterprise data warehouse to Azure SQL Data Warehouse and integrating it with both cloud-based and on-premises services and data sources.

Whiteboard Design Session

This whiteboard design session will look at the process of migrating an on-premises data warehouse to Azure SQL Data Warehouse. The design session will cover planning for a data warehouse migration, data and schema preparation, data loading, optimizing the data distribution, building a solution to support ad-hoc queries, migrating existing ETL packages and visualizing data with Power BI.

At the end of this whiteboard design session, you will be better able to plan and implement a migration of your existing on-premises enterprise data warehouse to Azure SQL Data Warehouse and integrating it with both cloud-based and on-premises services and data sources.

Hands-on Lab

In this hands-on lab you will migrate an existing on-premises enterprise data warehouse to the cloud. You will investigate the current data warehouse to identify any incompatibilities, export the data from the on-premises data warehouse, and transfer it to an Azure Blob Storage. You will then load the data into the warehouse using Polybase. Finally, you will integrate the warehouse by migrating ETL to Azure Data Factory and supporting ad-hoc access by implementing Azure Analysis Services.

At the end of this hands-on lab, you will be better able to plan and implement a migration of your existing on-premises enterprise data warehouse to Azure SQL Data Warehouse and integrating it with both cloud-based and on-premises services and data sources.

Azure services and related products

  • Azure SQL Data Warehouse
  • Azure Data Factory v2
  • Azure Analysis Services
  • Azure Storage
  • Power BI

Azure solutions

Cloud Scale Analytics

Related references

Help & Support

We welcome feedback and comments from Microsoft SMEs & learning partners who deliver MCWs.

Having trouble?

  • First, verify you have followed all written lab instructions (including the Before the Hands-on lab document).
  • Next, submit an issue with a detailed description of the problem.
  • Do not submit pull requests. Our content authors will make all changes and submit pull requests for approval.

If you are planning to present a workshop, review and test the materials early! We recommend at least two weeks prior.

Please allow 5 - 10 business days for review and resolution of issues.

mcw-migrate-edw-to-azure-sql-data-warehouse's People

Contributors

apaul2101gh avatar burhandodhy avatar dawnmariedesjardins avatar microsoftopensource avatar msftgits avatar mwasham avatar orcelik avatar paulopsgility avatar zobayer098 avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

mcw-migrate-edw-to-azure-sql-data-warehouse's Issues

Before the HOL - Missing sections

Hello,
I updated the Before the lab document to reflect the correct header & license info but it's missing the TOC and Requirements sections that are in the template. I don't know the coding to add the TOC, sorry. Please update

HOL Ex 1 Task 4

Task 4 asks user to use a region that was previously used. However, we've provisioned 3-4 resources at this point in the lab and each could have been to a different region. We need to give clearer guidance here.

Before the lab - Ex 2, Step 2

Please repeat user name and password from previous steps so that users don't have to scroll around for this info.

Better script for Azure Data Warehouse than hacking the script

In the lab an on-prem database gets scripted for a SQL Server on-prem destination. Afterwards users need to remove the filegroup statements, the use statements, some go statements... But in Management Studio an on-prem database can be scripted for an Azure Data Warehouse destination, which does all this cleanup automatically, so I would suggest to alter Exercise 2, task 2, step 6 in using the Advanced option of scripting, and configure the options Script for the database engine type Microsoft Azure SQL Database, Script for the database engine edition: Microsoft Azure SQL Data Warehouse Edition:
image
Then the next step where the script needs to altered can be removed!

HOL Ex 1

We are creating resources in 3-4 resource groups. We should use one to make clean up easier.

HOL - Ex 2, Task 1, Step 9

Correct screenshot to show ON [PRIMARY] . Current screenshot does not reflect square brackets. Square brackets required for successful completion of step.

HOL - Ex 4, Task 2, Task 13

Provide more details to direct users to Triggers item in left-hand navigation (instead of the Triggers button on the current pane).

HOL - Ex 5, Task 3, Step 5

Users should change the processing to Process Full. Because the AS model is in a fully processed state following restore, Process Default does not trigger any data processing. Process Full will force AS to connect to the Azure DW to reprocess itself. Our users will know enough about AS to catch this.

Exercise 5, step 7: twice the same file is mentioned

The instruction reads: Select the C:\LabFiles\CustomerInfoData.csv, C:\LabFiles\CustomerMrktResearchData.csv and C:\LabFiles\CustomerInfoData.csv files and click the Upload button.

The C:\LabFiles\CustomerInfoData.csv is mentioned twice, probably another filename is needed

HOL - Ex 4, Task 1, Step 2

Please provide the Resource Group name within which this is created. By this point we have a lot of RGs to navigate, esp if we are using a Subscription used for other work.

VM is too low on memory

The VM has too little memory to function properly.

  • In around 33% of the cases, the post-deployment script fails (at least in region westeurope)
  • During the workshopk Google chrome crashes regularly while working in the Azure portal inside the VM

Suggestion: assign a VM with more memory during the deployment in the ARM script.

December Update

  • Updated screenshots
  • Simplified Deployment (template)
  • Simplified resource groups

HOL - Ex 3, Task 3, Step 6

Add a step to create a master key in the EDW. Without this step, you get this error message:

Msg 15581, Level 16, State 6, Line 1
Please create a master key in the database or open the master key in the session before performing this operation.

HOL - step-by-step

Several images missing alt-text.

Usually, when I hover over an image in GitHub, I see the alt-text title. Then, when it's in HTML we see the full alt-text. On your workshop, I don't see the title either....can you please look into why? Thanks!

No database roles largerc or xlargerc

Hi,

I'm trying to execute the script below to change the user's resource class but getting an error.
EXEC sp_addrolemember 'xlargerc', 'dataloader';

Error: Cannot alter the role 'xlargerc', because it does not exist or you do not have permission.
Then I checked the sys.database_principals view and the workload management database roles don't exist.

Please advise.

Workshop title correction

Hello,
Please correct the workshop title from Migrating and Enterprise Data Warehouse to Azure SQL Database on the README.md to Migrate EDW to Azure SQL Data Warehouse. Also correct the repo title and all other documents/links to match if needed.
Thanks,
Dawnmarie

Inconsistency in Power BI download

Small inconsistency: Near the end of the lab students have to download Power BI. In the first screenshot one has to download the 32 bit version, in the remainder of the screenshots you're using the 64 bit version

January 2020 - content update

This workshop is scheduled for a January 2020 content update. Please review open issues #48 & #54 and give your suggestions for further changes recommended for SME review.

NLA

Dear all. I have problems with the prerequisites of the lab. The VM that comes with the template always gives the same error (regarding NLA). I've taken several steps to solve this problem but don't manage to establish a connection.

"your windows DC cannot be contacted to perform NLA."

HOL - Ex 2, Step 3

Question marks after table name cause syntax error from SQL Server

USE CohoDW
GO
SELECT t.[name] as [Table],
c.[name] as [Column],
c.[system_type_id],
c.[user_type_id],
y.[is_user_defined],
y.[name]
FROM sys.tables?? t JOIN sys.columns c ON t.[object_id] = c.[object_id]
JOIN sys.types y ON c.[user_type_id] = y.[user_type_id]
WHERE y.[name] IN ('geography', 'geometry', 'hierarchyid', 'image', 'ntext',
'numeric', 'sql_variant', 'sysname', 'text', 'timestamp', 'uniqueidentifier', 'xml')
OR (y.[name] IN ('varchar', 'varbinary') AND ((c.[max_length] = -1) or
(c.max_length > 8000)))
OR (y.[name] IN ('nvarchar') AND ((c.[max_length] = -1) or
(c.max_length > 4000))) OR y.[is_user_defined] = 1;

HOL - Ex 5, Task 3, Step 2

Users need to change both the Data Source and Initial Catalog parameters in the connection string. The restored connection string points to the AdventureWorksDW2014 initial catalog (which users must modify to be CohoDW).

HOL - step-by-step & unguided

Requirements sections of the labs do not match each other. One has 1 requirement, the other has 2. Is that right?

Passwords sometimes contain backslashes

Passwords in the Hands-On Labs sometimes contain a backslash:

Password: Demo@pass123

However, later on when referring to the same password, it is displayed without a backslash:

  1. Open a remote desktop session to the SQLCohoDW virtual machine that you created before the lab using the demouser login and Demo@pass123 password.

HOL Ex1 Task 1

Task does not indicate which Subscription to select. Other tasks do. We should be consistent.

HOL - Setup

There are just too many resources being deployed across too many resource groups. Having completed the setup steps, I have a SQL VM, two SQL DBs, a SQL DW, Azure AS, and an Azure Data Factory with its own VM/Runtime. These are spread across 5 resource groups making clean up difficult. This needs to be simplified.

HOL - Ex 5, Task 2, Step 6

Need to provide clearer guidance on which authentication method to use. For organizations such as Microsoft which have implemented MFA, you must choose the Azure AD - MFA option.

CRITICAL: HOL Ex 1 Task 6

This line:

From the SQLCohoDW virtual machine, open Internet Explorer, and connect to the Azure Portal

When did we ever create a virtual machine named SQLCohoDW. I'm lost.

Links to migration docs end up at wrong article

In the Whiteboard Design Session, participants are expected to review migration overview and high-level checklists. However, this documentation seems to have been removed from Azure, and all links about migration (in general, about schema, about data, and about code) redirect to the overview of developing Azure SQL DW (without any migration tips) : https://docs.microsoft.com/en-us/azure/sql-data-warehouse/sql-data-warehouse-overview-develop

In order to complete the WDS, participants should either be pointed to other docs (containing the same information), or the suggested answers in the training guide (at the text "Answer: The data preparation phase or the pre-migration phase is when you plan your migration. Here is a minimal list of or research to get you started:") have to be updated.

There is some documentation in the Azure Database Migration Guide (https://datamigration.microsoft.com/scenario/sql-to-sqldw?step=1), although this guide also contains links to the (now disappeared) SQLDW Migration docs.

August 2019 - Content update

This MCW is scheduled for content update.
Opsgility - please review current content and any existing open issues and provide your recommended updates for SME review.

Before the lab

Please add a note referencing the Before the Lab.md in the Step-by-Step HOL instructions. Otherwise, folks will miss this.

HOL Ex1 Task 2

Task 2 asks to specify region for ADF v2. These regions do not cover all the regions associated with the ADF RunTime. Users may be confused. We should add a note to indicate that the RunTime does not need to be in the same region as the ADF.

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.