This is the source code of EventHubsReceiver for Spark Streaming.
Here are the examples that use this library to process streaming data from Azure Eventhubs.
For latest integration of EventHubs and Spark Streaming, the document can be found here.
For help with Spark Cluster setup and running Spark-EventHubs applications in your cluster, checkout our Getting Started page.
We will have the official release in the maven central repo, you can add the following dependency to your project to reference Spark-EventHubs
<!-- https://mvnrepository.com/artifact/com.microsoft.azure/spark-streaming-eventhubs_[2.10 for spark 1.6. 2.11 for spark 2.0.x or spark 2.1.x] -->
<dependency>
<groupId>com.microsoft.azure</groupId>
<artifactId>spark-streaming-eventhubs_[2.10 for spark 1.6. 2.11 for spark 2.0.x or spark 2.1.x]</artifactId>
<version>[change it to latest version]</version>
</dependency>
// https://mvnrepository.com/artifact/com.microsoft.azure/spark-streaming-eventhubs_2.11
libraryDependencies += "com.microsoft.azure" % "spark-streaming-eventhubs_[2.10 for spark 1.6. 2.11 for spark 2.0.x or spark 2.1.x]" % "[change it to latest version]"
https://mvnrepository.com/artifact/com.microsoft.azure/spark-streaming-eventhubs_[2.10 for spark 1.6. 2.11 for spark 2.0.x or spark 2.1.x]/[change it to latest version]
We will also publish the staging version of Spark-EventHubs in GitHub. To use the staging version of Spark-EventHubs, the first step is to indicate GitHub as the source repo by adding the following entry to pom.xml:
<repository>
<id>spark-eventhubs</id>
<url>https://raw.github.com/hdinsight/spark-eventhubs/maven-repo/</url>
<snapshots>
<enabled>true</enabled>
<updatePolicy>always</updatePolicy>
</snapshots>
</repository>You can then add the following dependency to your project to take the pre-released version.
<!-- https://mvnrepository.com/artifact/com.microsoft.azure/spark-streaming-eventhubs_2.11 -->
<dependency>
<groupId>com.microsoft.azure</groupId>
<artifactId>spark-streaming-eventhubs_[2.10 for spark 1.6. 2.11 for spark 2.0.x or spark 2.1.x]</artifactId>
<version>2.1.0-SNAPSHOT</version>
</dependency>
// https://mvnrepository.com/artifact/com.microsoft.azure/spark-streaming-eventhubs_2.11
libraryDependencies += "com.microsoft.azure" % "spark-streaming-eventhubs_2.11" % "2.1.0-SNAPSHOT"
In order to build and run the examples, you need to have:
- Java 1.8 SDK.
- Maven 3.x
- Scala 2.11
mvn clean
mvn install
This command builds and installs spark-streaming-eventhubs jar to local maven cache. Subsequently you can build any Spark Streaming application that references this jar.
React
Typescript
Django
Laravel
Facebook
Microsoft
Google
Alibaba
D3
Tencent