ashishkr007 Goto Github PK

azure-data-factory-v2-hands-on

azure-eventhub-kafka-streaming

In this repository, have posted some sample notebook for spark streaming with EventHub. I have kept my focus on EventHub integration rather than stream operation. If you are looking to set up your streaming application with EventHub or Kafka. This is all that you need. you will get example for both producer and consumer.

csvdatavalidator

CSV Data Validator is a tool to validate csv file. It parse csv and validate the data with .hdr(csv meta data) before ingestion to Data Lake. It checks data file availability for every day load and validate data with respective meta data like File Size, Checksum, Delimiter, Record count etc. It ensure landed data conformity before give go ahead for data ingestion to Data Lake. It generate complete stats or error log.

csvparser

CSVParser is a tool to parse csv file using univocity and commons csv parsers. It cleans new line (\n) character & special characters between data. It also handle various garbage data like odd no of quotes or delimiters in side quotes. It validate each record with specified delimiter count and separate it out to _GoodRecords.CSV and _BadRecords.CSV file. This is a Data Cleaner tool to run before ingestion to Data Lake. It make sure data is in right csv format to build table on it.

data-cleaner

If you have some csv file and having CRLF, LF in between data and you want to create some table (Hive table). You will face issue that some of column have null value. It’s because line terminator in hive is \n and if and \n or \r coming between data it treating as line terminator before actual line terminator and rest for column is getting null value. I tried multiple option like spark, hive serde and many more but I found good with perl. Today I a sharing my Perl script to remove all newline and special characters.

data-purge-script

Data purging is powershell script to purge older data based upon a data-retention policy.

databricks-group-api

The Groups API allows you to manage groups of users. This code will allow to access databricks group api from notebook. I written this code snipped for one of my requirement, where i have to send workspace security events/workspace event to security team. This is one of example to access Databricks Api but with this code you can access any databricks Api. The same this can be done with normal curl command also.

delta-lake-concurrency-control

Delta Lake uses optimistic concurrency control to provide transactional guarantees between writes. optimistic concurrency control is ways to deal with concurrent transaction that making changes to table.

facebook_insight_streaming

Facebook Insight Streaming

java-properties-reader

Java properties file is used to store project configuration data or settings. In this projects will show you how to load properties file.

spark-sqlserver-scala

Spark-SQLServer-Scala

sparkkafkaintegration

Apache Spark and Apache Kafka integration and Spark Analytics over DataFrame.