Data from subsequent batches are skipped after an BlobAlreadyReceived_BlobAlreadyFoundInBatch error about azure-kusto-spark HOT 4 OPEN

@lgo-solytic , Please provide the write options you are using and the cluster name. Please provide the approx time of these runs as well

from azure-kusto-spark.

@ag-ramachandran,

Please provide the write options you are using and the cluster name. Please provide the approx time of these runs as well
Options:
KustoSinkOptions.KUSTO_TABLE_CREATE_OPTIONS -> "CreateIfNotExist",
KustoSinkOptions.KUSTO_STAGING_RESOURCE_AUTO_CLEANUP_TIMEOUT -> "5",
KustoSinkOptions.KUSTO_WRITE_ENABLE_ASYNC -> "true",
KustoSinkOptions.KUSTO_WRITE_MODE -> "Queued"

Nothing additional in SparkIngestionProperties

Both jobs were running (streaming) for longer than 48h uninterrupted.

from azure-kusto-spark.

@lgo-solytic : BlobAlreadyReceived_BlobAlreadyFoundInBatch is a Kusto warning from : https://learn.microsoft.com/en-us/azure/data-explorer/error-codes#category-blobalreadyreceived
(it correlates to a similar case of 2 blobs getting ingested as well)

Is there any other Connector logs you see from this failure ? The logs start with KustoConnector for the Spark connector. If you find anything in that correlates, that'd be useful

There is no correlation inference we can draw between the drop in volumes related to this error. You need not use the option

KustoSinkOptions.KUSTO_WRITE_ENABLE_ASYNC -> "true", , exceptions in tasks are not propagated to driver if this is used.

from azure-kusto-spark.

@ag-ramachandran
thanks for your answer.
No unfortunately no logs that could help. Is it possible to configure the KustoConnector to log to application insights?

The same thing happened again last night. But this time only one job was running so at least we can exclude the case of two jobs colliding.

from azure-kusto-spark.