Comments (3)
yamin-msft
commented on June 9, 2024
Currently, we only support checkpoint location which is a path in an HDFS compatible file system. We don't have a firm deadline for this feature yet.
from azure-event-hubs-spark.
ddevidence
commented on June 9, 2024
• Access that is compatible with Hadoop: In Azure Data Lake Storage Gen2, you can manage and access data just as you would with a Hadoop Distributed File System (HDFS). The Azure Blob File System (ABFS) driver is available within all Apache Hadoop environments, including Azure HDInsight and Azure Databricks. Use ABFS to access data stored in Data Lake Storage Gen2.
– this ref seems to indicate ADLS Gen2 supports Hadoop operations which lead to the assumption the library would support checkpoint write as part of the spark hadoopConfiguration.
Can you provide insight ? if this would be a significant feature update or perhaps something on the lower-end.. just want to have this available on a priority for the use-case at hand.
from azure-event-hubs-spark.
yamin-msft
commented on June 9, 2024
This will require some works and I am not able to provide a deadline for it
from azure-event-hubs-spark.
Related Issues (20)
- Spark starts crashing when some of the eventhub partitions go down.
- Structured streaming job hangs after a while
- EH - Trigger once HOT 1
- This library or kafka
- Using Kafka driver for Spark the throughput is 50-80 times faster
- AAD Authentication is terminated after running for a couple of minutes HOT 1
- Job for consumption of Event Hub messages aborts on Databricks (request seqNo less than received seqNo) HOT 3
- Package Support for Scala 2.13 or 3+ HOT 4
- PySpark job doesnt stop on stopping query
- Batch read from eventhubs throws duration format error
- Missing tag for v2.3.22
- Spark-scala api references for Azure eventhub schema registry
- maxEventsPerTrigger is not working
- Spark streaming kubernetes - Fails to recover from chechpoint. Cannot find endpoint: spark://PartitionPerformanceReceiver
- EventHub Writer fails due to Throttling of EventHub, configuration settings have no impact. HOT 1
- ReceiverDisconnectedException even if using different consumer groups HOT 1
- The auth docs are wrong - causing service unavailability issues
- Azure EventHub - PySpark Failed to configure SaslClientAuthenticator works when using Confluent cloud
- Support for Spark v3.3.0 HOT 1
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from azure-event-hubs-spark.