deep-bi / spark-druid-segment-reader Goto Github PK

Currently only supports spark2.3.4, spark2.4.5 does not support， can you help me?

: java.lang.NoClassDefFoundError: org/apache/spark/sql/sources/v2/reader/DataReaderFactory
at bi.deep.DruidSource.createReader(DruidSource.scala:12)
at org.apache.spark.sql.execution.datasources.v2.DataSourceV2Relation$SourceHelpers.createReader(DataSourceV2Relation.scala:155)
at org.apache.spark.sql.execution.datasources.v2.DataSourceV2Relation$.create(DataSourceV2Relation.scala:172)
at org.apache.spark.sql.DataFrameReader.load(DataFrameReader.scala:204)
at org.apache.spark.sql.DataFrameReader.load(DataFrameReader.scala:167)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:244)
at py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:357)
at py4j.Gateway.invoke(Gateway.java:282)
at py4j.commands.AbstractCommand.invokeMethod(AbstractCommand.java:132)
at py4j.commands.CallCommand.execute(CallCommand.java:79)
at py4j.GatewayConnection.run(GatewayConnection.java:238)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.ClassNotFoundException: org.apache.spark.sql.sources.v2.reader.DataReaderFactory
at java.net.URLClassLoader.findClass(URLClassLoader.java:382)
at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:349)
at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
... 16 more

Hi guys,
I'm not sure if the question makes sense, but we are trying to use your library and we noticed that only Dimensions are processes and no measures, just curious if it's possible to extract measures as well?
Regards,
Pavel

Hello,

using this segment reader I'm running pyspark code in k8s. I have passed the jar to Spark. jars like it's said in the Read Me file at the time of creating the spark session. I'm using apache/spark-py:v3.1.1 docker image at first it threw DataSourcev2 class not found exception then I passed spark-sql_2.11-2.4.8 jar to spark jars it cleared the exception though now I'm facing druid-segment data source is not valid..

Can someone please guide me on how I can proceed further?

Just wondering if you have look/review apache/druid#10920 before?