Unable to execute df.action in synapse. about azure-kusto-spark HOT 4 CLOSED

Glad that helped @msft33333. If this works can we close the issue

from azure-kusto-spark.

Hello @msft33333
Apologies for this inconvenience, this is a known issue because a new native implementation of Parquet writer that uses new encoding schemes was rolled out on the ADX side. This uses delta byte array for strings and other byte array-based Parquet types (default in Parquet V2 which most modern parquet readers support by default)

Spark 2.x has the issue (with Delta encodings on Parquet V2) as mentioned here https://issues.apache.org/jira/si/jira.issueviews:issue-html/SPARK-26509/SPARK-26509.html

Parquet V2 which has Delta encodings was to be an improvement over the Parquet V1 (plain encoding). Based on documentation, delta encoding provided advantages in reading rows in batches (Ref : https://issues.apache.org/jira/browse/SPARK-12854). These flags are provided as workaround as you can see in the previous ticket on Spark Jira

Mitigation: To mitigate this a new version of the library has been rolled out that will be deployed on Synapse (Spark library version 3.1.10).

Will triage it to the right team and get you a roll out as soon as possible

Once again apologize for the inconvenience

from azure-kusto-spark.

Also, to overcome this you can try setting the following spark options and this should fix the issue immediately. We will work parallelly to get a new revision to Synapse

spark.conf.set("spark.sql.parquet.enableVectorizedReader", "false")
spark.conf.set("parquet.split.files", "false")

from azure-kusto-spark.

Hi @ag-ramachandran ,
Thanks for the detailed reply!The workaround works fine for me,thanks again!

from azure-kusto-spark.