sql connection using service principal to specific table about databricks-sdk-py HOT 2 OPEN

Does the service principal itself work? If you take your config and do

w = WorkspaceClient(config)
w.current_user.me()

does it succeed?

from databricks-sdk-py.

I'm not sure if this is related or not, but it might help others that are facing a similar issue.

We started getting the same error (invalid_client: Client authentication failed) a few days ago, nothing has changed on our code base/infrastructure and it was working perfectly before.

For context, we're using Azure Databricks and connecting via service principal. We've been using the client id and secret from Microsoft Entra ID.

After digging a little bit on the stack trace, we confirmed the issue was coming from the call to get an access token, which should be something like this, according to the docs:

curl -X POST \
<per-workspace-url>/oidc/v1/token \
-d "grant_type=client_credentials" \
-d "scope=all-apis" \
-u "<service-principal-id>:<oauth-secret>"

We've then tried the same request using the same credentials from Microsoft Entra ID and confirmed the error was coming from this call.

Reading a bit more the docs we saw on Step 6: Create an Azure Databricks OAuth secret for the service principal that we should be using a Databricks OAuth Secret (apparently something new, since we've been using only the service principal credentials from Microsoft Entra ID). So we have it a try... I created a new token, sent it on the same request as before and it worked fine.

We believe this was introduced by this change, however we're not sure if this was an intended (and miscommunicated) breaking change or a regression bug, because it did work fine before 🤷🏼

from databricks-sdk-py.