This library contains the source code for Kusto Data Source and Data Sink Connector for Apache Spark.

Kusto (A.K.A. Azure Data Explorer) is a lightning-fast indexing and querying service.

Spark is a unified analytics engine for large-scale data processing.

Making Kusto and Spark work together enables our users to build fast and scalable applications, targeting a variety of Machine Learning, Extract-Transform-Load, Log Analytics and other data driven scenarios.

This is a beta release, serving as the baseline version of Kusto connector for Spark. It exposes Kusto as a valid Data Store for standard Spark source and sink operations such as write, read and writeStream. It can be used for experimentation, and for applications where the size of data read from Kusto is fairly small.

Future releases will extend Kusto connector capabilities in the following areas:

• Allow large-scale data transport to and from Kusto
• Add capabilities: will be specified in the version CHANGELIST
• Improve fault tolerance and resilience

For Scala/Java applications using Maven project definitions, link your application with the artifact below.

groupId = com.microsoft.azure
artifactId = spark-kusto-connector
version = 1.0.0-Beta-01 

In Maven:

  <dependency>
    <groupId>com.microsoft.azure</groupId>
    <artifactId>spark-kusto-connector</artifactId>
    <version>1.0.0-Beta-01</version>
  </dependency>

In order to use the connector, you need to have:

• Java 1.8 SDK installed

• Maven 3.x installed

• Spark version 2.4.0 or higher

  Note: 2.3.x versions are also supported, but require some changes in pom.xml dependencies. For details, refer to CHANGELIST

// Builds jar and runs all tests
mvn clean package

// Builds jar, runs all tests, and installs jar to your local maven repository
mvn clean install

In order to facilitate ramp-up on platforms such as Azure Databricks, pre-compiled libraries can be found here

Detailed documentation can be found here. Usage examples can be found here

Contributions and suggestions to this product will be possible, and most welcome, starting with next release