Git Product home page Git Product logo

mcw-real-time-data-with-azure-database-for-postgresql-hyperscale's Introduction

This workshop is archived and is no longer being maintained. Content is read-only.

For additional Data and AI content, please go to https://microsoft.github.io/sqlworkshops/.

Real-time data with Azure Database for PostgreSQL Hyperscale

Wide World Importers (WWI) is a traditional brick and mortar business with a long track record of success, generating profits through strong retail store sales of their unique offering of affordable products from around the world and a strong focus on customer relationships. WWI modernized their business by expanding to online storefronts and experimenting with various marketing tactics to drive online sales. This expansion has made it more challenging to analyze user clickstream data, online ad performance, and other marketing campaigns at scale, and to provide insights to the marketing team in real-time.

WWI is interested in learning to use advanced features of the managed PostgreSQL PaaS service on Azure to make their database more scalable and able to handle the rapid ingest of streaming data while simultaneously generating and serving pre-aggregated data for reports.

June 2020

Target audience

  • Database Administrator
  • Data Engineer
  • Data Scientist
  • Database Developer
  • Solution Architect

Abstracts

Workshop

In this workshop, you will learn how to use advanced features of the managed PostgreSQL PaaS service on Azure to make your database more scalable and able to handle the rapid ingest of streaming data while simultaneously generating and serving pre-aggregated data for reports. You will design a resilient stream processing pipeline to ingest, process, and save real-time data to Postgres. Next, you will learn how to explore and gain insights from flat files at scale, perform aggregates, and save the valuable information to your database. Finally, you will create complex reports containing advanced visualizations, using a drag-and-drop interface, and use them to build a customizable dashboard that gets updated in real-time.

At the end of this workshop, you will be better able to implement a highly scalable, managed open source database solution that can simultaneously handle real-time data and roll-up and serve data for advanced visualizations.

Whiteboard design session

In this whiteboard design session, you will work with a group to design a solution for using advanced features of the managed PostgreSQL PaaS service on Azure to make your database more scalable and able to handle the rapid ingest of streaming data while simultaneously generating and serving pre-aggregated data for reports. You will provide guidance for designing a resilient stream processing pipeline to ingest, process, and save real-time data to Postgres. Next, you will design a solution for exploring and gaining insights from flat files at scale, perform aggregates, and saving the valuable information to your database. Finally, you will provide guidance on how to create complex reports containing advanced visualizations, using a drag-and-drop interface, and use them to build a customizable dashboard that gets updated in real-time.

At the end of this whiteboard design session, you will be better able to design a highly scalable, managed open source database solution that can simultaneously handle real-time data and roll-up and serve data for advanced visualizations.

Hands-on lab

In this hands-on lab, you will implement a proof-of-concept (PoC) for using advanced features of the managed PostgreSQL PaaS service on Azure. These features help make your database more scalable and able to handle the rapid ingest of streaming data while simultaneously generating and serving pre-aggregated data for reports. You will create a resilient stream processing pipeline to ingest, process, and save real-time data to Postgres. Next, you will implement a solution for exploring and gaining insights from flat files at scale, perform aggregates, and saving the valuable information to your database. Finally, you will create complex reports containing advanced visualizations, using a drag-and-drop interface, and use them to build a customizable dashboard that gets updated in real-time.

At the end of this hands-on-lab, you will be better able to implement a highly scalable, managed open source database solution that can simultaneously handle real-time data and roll-up and serve data for advanced visualizations.

Azure services and related products

  • Azure Database for PostgreSQL
  • Azure Event Hubs with Kafka
  • Azure Databricks
  • Power BI
  • Azure Cloud Shell
  • pgAdmin

Related references

mcw-real-time-data-with-azure-database-for-postgresql-hyperscale's People

Contributors

codingbandit avatar dawnmariedesjardins avatar joelhulen avatar microsoftopensource avatar msftgits avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

mcw-real-time-data-with-azure-database-for-postgresql-hyperscale's Issues

September 2020 update suggestions

I suggest the following updates. Please provide your feedback:

  • The latest Npgsql provider now supports DirectQuery with Power BI. Update the WDS and, if appropriate, the lab, to reflect this change.
  • Update lab instructions with the latest UI changes for deploying Hyperscale (Citus).
  • Add high availability scenario to the WDS that prompts students to enable the high availability option for a Hyperscale cluster.
  • Update the data generator and libraries used for the lab to the latest, stable versions.
  • Add exercise in the lab to use Azure Data Studio to create a PostgreSQL database health check SQL notebook, which monitors cache hit ratio, monitors dead tuples, monitors unused indexes, and monitors query performance.

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.