Running into issue with .option("kustoAADAuthorityID","72f988bf-86f1-41af-91ab-2d7cd011db47") about azure-kusto-spark HOT 6 CLOSED

hey
If you haven't upgraded the connector than it couldn't have changed ..
KUSTO_AAD_AUTHORITY_ID is the property of the object KustoSourceOptions that is holding the string value : "kustoAADAuthorityID"
please add the error that you get

from azure-kusto-spark.

An error occurred while calling o2981.load.
: com.microsoft.azure.kusto.data.exceptions.DataServiceException: Error in acquiring ApplicationAccessToken
at com.microsoft.azure.kusto.data.AadAuthenticationHelper.acquireToken(AadAuthenticationHelper.java:247)
at com.microsoft.azure.kusto.data.AadAuthenticationHelper.acquireAccessToken(AadAuthenticationHelper.java:90)
at com.microsoft.azure.kusto.data.ClientImpl.initHeaders(ClientImpl.java:142)
at com.microsoft.azure.kusto.data.ClientImpl.execute(ClientImpl.java:89)
at com.microsoft.azure.kusto.data.ClientImpl.execute(ClientImpl.java:50)
at com.microsoft.kusto.spark.utils.KustoDataSourceUtils$.getSchema(KustoDataSourceUtils.scala:87)
at com.microsoft.kusto.spark.datasource.KustoRelation.getSchema(KustoRelation.scala:101)
at com.microsoft.kusto.spark.datasource.KustoRelation.schema(KustoRelation.scala:47)
at org.apache.spark.sql.execution.datasources.DataSource.resolveRelation(DataSource.scala:458)
at org.apache.spark.sql.DataFrameReader.loadV1Source(DataFrameReader.scala:311)
at org.apache.spark.sql.DataFrameReader.load(DataFrameReader.scala:297)
at org.apache.spark.sql.DataFrameReader.load(DataFrameReader.scala:203)
at sun.reflect.GeneratedMethodAccessor849.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:244)
at py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:380)
at py4j.Gateway.invoke(Gateway.java:295)
at py4j.commands.AbstractCommand.invokeMethod(AbstractCommand.java:132)
at py4j.commands.CallCommand.execute(CallCommand.java:79)
at py4j.GatewayConnection.run(GatewayConnection.java:251)
at java.lang.Thread.run(Thread.java:748)
Caused by: com.microsoft.azure.kusto.data.exceptions.DataClientException: Error in acquiring ApplicationAccessToken
at com.microsoft.azure.kusto.data.AadAuthenticationHelper.acquireAadApplicationAccessToken(AadAuthenticationHelper.java:142)
at com.microsoft.azure.kusto.data.AadAuthenticationHelper.acquireToken(AadAuthenticationHelper.java:231)
... 21 more
Caused by: java.util.concurrent.ExecutionException: com.microsoft.aad.adal4j.AuthenticationException: {"error_description":"AADSTS7000215: Invalid client secret is provided.\r\nTrace ID: 0151fff4-9b05-4448-a779-e46b0c917d00\r\nCorrelation ID: 120ac7d4-5c8b-43d1-9d8c-e6e8c54c3ef4\r\nTimestamp: 2020-08-07 23:08:27Z","error":"invalid_client","error_uri":"https://login.microsoftonline.com/error?code=7000215"}
at java.util.concurrent.FutureTask.report(FutureTask.java:122)
at java.util.concurrent.FutureTask.get(FutureTask.java:192)
at com.microsoft.azure.kusto.data.AadAuthenticationHelper.acquireAadApplicationAccessToken(AadAuthenticationHelper.java:140)
... 22 more
Caused by: com.microsoft.aad.adal4j.AuthenticationException: {"error_description":"AADSTS7000215: Invalid client secret is provided.\r\nTrace ID: 0151fff4-9b05-4448-a779-e46b0c917d00\r\nCorrelation ID: 120ac7d4-5c8b-43d1-9d8c-e6e8c54c3ef4\r\nTimestamp: 2020-08-07 23:08:27Z","error":"invalid_client","error_uri":"https://login.microsoftonline.com/error?code=7000215"}
at com.microsoft.aad.adal4j.AdalTokenRequest.executeOAuthRequestAndProcessResponse(AdalTokenRequest.java:128)
at com.microsoft.aad.adal4j.AuthenticationContext.acquireTokenCommon(AuthenticationContext.java:928)
at com.microsoft.aad.adal4j.AcquireTokenCallable.execute(AcquireTokenCallable.java:70)
at com.microsoft.aad.adal4j.AcquireTokenCallable.execute(AcquireTokenCallable.java:38)
at com.microsoft.aad.adal4j.AdalCallable.call(AdalCallable.java:47)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
... 1 more

from azure-kusto-spark.

I compared one of the successful scripts with this one, and the only difference I see is that this one (failing one) uses this:

crp = sc._jvm.com.microsoft.azure.kusto.data.ClientRequestProperties()

I noticed that https://github.com/Azure/azure-kusto-spark/releases notes for v2.0.0 have this part:

Could this be it?
We also updated the service principal secret that we are using -- but it is updated in the scoped keyvault as well and that wouldn't explain why other scripts succeed and why this one succeeds if we use "KUSTO_AAD_AUTHORITY_ID" instead of "kustoAADAuthorityID" or "microsoft.com" instead of guid.

from azure-kusto-spark.

as i said it cant be it as u havent updated the client
The error states that the client secret is invalid - meaning that the tenant you provided is correct - the application was found for authentication and only the secret is wrong - please check its value

btw when u used "KUSTO_AAD_AUTHORITY_ID" as a string the connector couldnt find it and used its default - microsoft.com

from azure-kusto-spark.

The secret is pulled from keyvault -- and if it was invalid, wouldn't default "microsoft.com" also fail?
As I said, this happened soon after the secret was updated for service principal, so this was first thing I checked. Keyvault contains the new version; it is also used by other scripts, which do not fail.

from azure-kusto-spark.

hey
if you want we can have a call on teams although i'm pretty sure the problem is not from the connector's side, its really weird that authenticating with "microsoft.com" has different results from "72f988bf-86f1-41af-91ab-2d7cd011db47" - i would debug this locally if you can - just use our java sdk library (https://github.com/Azure/azure-kusto-java) which we use here for authentication, quering and ingestion.

from azure-kusto-spark.