ThrottleExceptions when writing data to ADX/Kusto about azure-kusto-spark HOT 12 OPEN

@ag-ramachandran may be the message is truncated. I sent you a new email with Exception stack trace

from azure-kusto-spark.

Hello @ravikiransharvirala

More than a connector issue, this is an underlying issue on the capacity and not a connector issue that will require investigation.
All connector things remaining same, it could quite be that there are limits hit on parallel runs (or) other workloads using capacity
As this will need looking into the cluster, would be great if you can raise a support request (or) alternatively send me details of the cluster on my microsoft handle (ramacg) so that i can take it up further

cc: @asaharn

from azure-kusto-spark.

@ag-ramachandran Thanks for responding. I shared cluster details via email. Please let me know if you haven't received it.

from azure-kusto-spark.

@ravikiransharvirala Not yet!

from azure-kusto-spark.

@ag-ramachandran interesting. I sent it to ramacg at microsoft.

I sent it again without links and images.

from azure-kusto-spark.

@ravikiransharvirala got it now. Will check

from azure-kusto-spark.

@ravikiransharvirala , Please share the logs from STDOUT/ERR/LOG4j so that i can look at time correlation of these errors and capacity too

from azure-kusto-spark.

@ag-ramachandran Sure, will do that.

from azure-kusto-spark.

@ag-ramachandran Sent it as text. Let me know if you haven't received it.

from azure-kusto-spark.

@ravikiransharvirala no exceptions in that log though!

from azure-kusto-spark.

@ag-ramachandran Do you recommend persisting the dataframe after reading the data from the Kusto connector coz after reading the data from the database and performing transformations on it, I notice the connector making calls to the database throughout the job's execution.

These are the two queries I noticed while running the job (the job needs entire data from the table)

<table_name> | count
<table_name> | evaluate estimate_rows_count()

from azure-kusto-spark.

@ravikiransharvirala , Need more specifics. If are trying to read the same data again and again, it makes good reading to cache it.
These queries are used to determine how data is read, the internals of reading are different in ForceSingle and ForceDistributed modes. In your case you can set the readMode as ForceDistributed and i think some of these queries would go away.

If in force distributed mode (parquet export) you want to reuse the same file use the transient cache to true.

KUSTO_READ_MODE 'readMode' - Override the connector heuristic to choose between 'Single' and 'Distributed' mode. Options are - 'ForceSingleMode', 'ForceDistributedMode'. Scala and Java users may take these options from com.microsoft.kusto.spark.datasource.ReadMode.

KUSTO_DISTRIBUTED_READ_MODE_TRANSIENT_CACHE When 'Distributed' read mode is used and this is set to 'true', the request query is exported only once and exported data is reused.

Read up more on : https://github.com/Azure/azure-kusto-spark/blob/master/docs/KustoSource.md

P.S. It may not be related to this issue, but are good options to set and try for optimized reads

from azure-kusto-spark.