No possibility to use Service Principal when Workload Identity is enabled in the Pod about kubelogin HOT 7 OPEN

@daleksandrowiczgd can you share what your kubeconfig looks like after kubelogin convert-kubeconfig -l spn? you only need to capture the exec plugin part:

      exec:
        apiVersion: client.authentication.k8s.io/v1beta1
        command: kubelogin
        args:
          - get-token
          - --environment
          - AzurePublicCloud
          - --server-id
          - <AAD server app ID>
          - --client-id
          - <AAD client app ID>
          - --tenant-id
          - <AAD tenant ID>

besides, in your repro step you seem to miss -l spn? not sure if this is intentional or not?

Convert kubeconfig with kubelogin convert-kubeconfig command and Service Principal parameters specified as arguments --client-id <AZURE_CLIENT_ID> --client-secret <AZURE_CLIENT_SECRET> --tenant-id <AZURE_TENANT_ID>

can you also share the actual relevant environment variables in your runner environment? There is no reference to AZURE_FEDERATED_CREDENTIALS in any repo I can find.

from kubelogin.

besides, in your repro step you seem to miss -l spn? not sure if this is intentional or not?

Non intentional, I missed it this in my message, so please just assume it should be there.

can you also share the actual relevant environment variables in your runner environment? There is no reference to AZURE_FEDERATED_CREDENTIALS in any repo I can find.

Actually, I just mentioned all environment variables set by Workload Identity, but it seems like only the AZURE_CLIENT_ID variable generates conflicts for kubelogin tool.

can you share what your kubeconfig looks like after kubelogin convert-kubeconfig -l spn? you only need to capture the exec plugin part:

Yes, let me even share you few cases with exact commands I run and results of the kubeconfig file after running each of them.

In all cases the following env variables will be set (I will use the same name to identify the same values in kubeconfig):

export SP_CLIENT_ID="<SP_CLIENT_ID>"                    # Client (Application) ID of the Service Principal
export SP_CLIENT_SECRET="<SP_CLIENT_SECRET>"            # Client secret of the Service Principal
export SP_TENANT_ID="<SP_TENANT_ID>"                    # Tenant ID of the Service Principal
export CLUSTER_RG="<CLUSTER_RG>"                        # AKS cluster resource group
export CLUSTER_NAME="<CLUSTER_NAME>"                    # AKS cluster name
export CUSTOM_KUBECONFIG_PATH="${HOME}/.kube/config"    # Path to the kubeconfig

Additionally, for better understanding of the output I will specify WI_CLIENT_ID (Workload Identity client ID), instead of the real value.

Pass Service Principal Client ID as the CLI argument in the kubelogin command (desired usage)

az login --service-principal -u "${SP_CLIENT_ID}" -p "${SP_CLIENT_SECRET}" -t "${SP_TENANT_ID}"
az aks get-credentials --resource-group "${CLUSTER_RG}" --name "${CLUSTER_NAME}" --file "${CUSTOM_KUBECONFIG_PATH}"
kubelogin convert-kubeconfig --login "spn" --kubeconfig "${CUSTOM_KUBECONFIG_PATH}" --client-id "${SP_CLIENT_ID}" --client-secret "${SP_CLIENT_SECRET}" --tenant-id "${SP_TENANT_ID}"

Kubeconfig:

exec:
  apiVersion: client.authentication.k8s.io/v1beta1
  args:
  - get-token
  - --login
  - spn
  - --server-id
  - 6dae42f8-4368-4678-94ff-3960e28e3630
  - --client-id
  - ${WI_CLIENT_ID}
  - --tenant-id
  - ${SP_TENANT_ID}
  - --environment
  - AzurePublicCloud
  - --client-secret
  - ${SP_CLIENT_SECRET}
  command: kubelogin
  env: null
  installHint: |2

    kubelogin is not installed which is required to connect to AAD enabled cluster.

    To learn more, please go to https://aka.ms/aks/kubelogin
  provideClusterInfo: false

$ kubectl get pod --kubeconfig ${CUSTOM_KUBECONFIG_PATH}
# RESPONSE 401 Unauthorized
# --------------------------------------------------------------------------------
# {
#   "error": "invalid_client",
#   "error_description": "AADSTS7000215: Invalid client secret provided. Ensure the secret being sent in the request is the client secret value, not the client secret ID, for a secret added to app '${WI_CLIENT_ID}'. Trace ID: 1d7b8101-9172-4b1e-8ac6-ac840f666301 Correlation ID: 9d204593-95f3-460e-81e6-fb6a94e4d959 Timestamp: 2024-01-03 10:45:10Z",
#   "error_codes": [
#     7000215
#   ],
#   "timestamp": "2024-01-03 10:45:10Z",
#   "trace_id": "1d7b8101-9172-4b1e-8ac6-ac840f666301",
#   "correlation_id": "9d204593-95f3-460e-81e6-fb6a94e4d959",
#   "error_uri": "https://login.microsoftonline.com/error?code=7000215"
# }
# --------------------------------------------------------------------------------
# To troubleshoot, visit https://aka.ms/azsdk/go/identity/troubleshoot#client-secret
# Unable to connect to the server: getting credentials: exec: executable kubelogin failed with exit code 1

Conclusion: --client-id argument in kubeconfig was overriden by Workload Identity. kubectl command doesn't work correctly - return 401 error.

Override AZURE_CLIENT_ID environment variable only for the kubelogin command

az login --service-principal -u "${SP_CLIENT_ID}" -p "${SP_CLIENT_SECRET}" -t "${SP_TENANT_ID}"
az aks get-credentials --resource-group "${CLUSTER_RG}" --name "${CLUSTER_NAME}" --file "${CUSTOM_KUBECONFIG_PATH}"
AZURE_CLIENT_ID="${SP_CLIENT_ID}" kubelogin convert-kubeconfig --login "spn" --kubeconfig "${CUSTOM_KUBECONFIG_PATH}" --client-id "${SP_CLIENT_ID}" --client-secret "${SP_CLIENT_SECRET}" --tenant-id "${SP_TENANT_ID}"

Kubeconfig:

exec:
  apiVersion: client.authentication.k8s.io/v1beta1
  args:
  - get-token
  - --login
  - spn
  - --server-id
  - 6dae42f8-4368-4678-94ff-3960e28e3630
  - --client-id
  - ${SP_CLIENT_ID}
  - --tenant-id
  - ${SP_TENANT_ID}
  - --environment
  - AzurePublicCloud
  - --client-secret
  - ${SP_CLIENT_SECRET}
  command: kubelogin
  env: null
  installHint: |2

    kubelogin is not installed which is required to connect to AAD enabled cluster.

    To learn more, please go to https://aka.ms/aks/kubelogin
  provideClusterInfo: false

$ kubectl get pod --kubeconfig ${CUSTOM_KUBECONFIG_PATH}
RESPONSE 401 Unauthorized
--------------------------------------------------------------------------------
{
  "error": "invalid_client",
  "error_description": "AADSTS7000215: Invalid client secret provided. Ensure the secret being sent in the request is the client secret value, not the client secret ID, for a secret added to app '${WI_CLIENT_ID}'. Trace ID: 854ccce1-bdf3-4122-8453-8e3ba73f9700 Correlation ID: 1371d7e8-053f-4f32-8f7b-9859bc27f635 Timestamp: 2024-01-03 11:00:37Z",
  "error_codes": [
    7000215
  ],
  "timestamp": "2024-01-03 11:00:37Z",
  "trace_id": "854ccce1-bdf3-4122-8453-8e3ba73f9700",
  "correlation_id": "1371d7e8-053f-4f32-8f7b-9859bc27f635",
  "error_uri": "https://login.microsoftonline.com/error?code=7000215"
}
--------------------------------------------------------------------------------
To troubleshoot, visit https://aka.ms/azsdk/go/identity/troubleshoot#client-secret
Unable to connect to the server: getting credentials: exec: executable kubelogin failed with exit code 1

Conclusion: --client-id argument in kubeconfig got correct Client ID of the Service Principal. kubectl command doesn't work correctly - return 401 error, again with the Workload Identity client ID in the error (WI_CLIENT_ID).

Override AZURE_CLIENT_ID environment variable only for kubelogin and kubectl commands

az login --service-principal -u "${SP_CLIENT_ID}" -p "${SP_CLIENT_SECRET}" -t "${SP_TENANT_ID}"
az aks get-credentials --resource-group "${CLUSTER_RG}" --name "${CLUSTER_NAME}" --file "${CUSTOM_KUBECONFIG_PATH}"
AZURE_CLIENT_ID="${SP_CLIENT_ID}" kubelogin convert-kubeconfig --login "spn" --kubeconfig "${CUSTOM_KUBECONFIG_PATH}" --client-id "${SP_CLIENT_ID}" --client-secret "${SP_CLIENT_SECRET}" --tenant-id "${SP_TENANT_ID}"

Kubeconfig:

exec:
  apiVersion: client.authentication.k8s.io/v1beta1
  args:
  - get-token
  - --login
  - spn
  - --server-id
  - 6dae42f8-4368-4678-94ff-3960e28e3630
  - --client-id
  - ${SP_CLIENT_ID}
  - --tenant-id
  - ${SP_TENANT_ID}
  - --environment
  - AzurePublicCloud
  - --client-secret
  - ${SP_CLIENT_SECRET}
  command: kubelogin
  env: null
  installHint: |2

    kubelogin is not installed which is required to connect to AAD enabled cluster.

    To learn more, please go to https://aka.ms/aks/kubelogin
  provideClusterInfo: false

$ AZURE_CLIENT_ID="${SP_CLIENT_ID}" kubectl get pod --kubeconfig ${CUSTOM_KUBECONFIG_PATH}
No resources found in default namespace.

Conclusion: --client-id argument in kubeconfig got correct Client ID of the Service Principal. kubectl works correctly, because AZURE_CLIENT_ID was overriden only during running this command.

Override AZURE_CLIENT_ID environment variable only for the kubectl command

az login --service-principal -u "${SP_CLIENT_ID}" -p "${SP_CLIENT_SECRET}" -t "${SP_TENANT_ID}"
az aks get-credentials --resource-group "${CLUSTER_RG}" --name "${CLUSTER_NAME}" --file "${CUSTOM_KUBECONFIG_PATH}"
kubelogin convert-kubeconfig --login "spn" --kubeconfig "${CUSTOM_KUBECONFIG_PATH}" --client-id "${SP_CLIENT_ID}" --client-secret "${SP_CLIENT_SECRET}" --tenant-id "${SP_TENANT_ID}"

Kubeconfig:

exec:
  apiVersion: client.authentication.k8s.io/v1beta1
  args:
  - get-token
  - --login
  - spn
  - --server-id
  - 6dae42f8-4368-4678-94ff-3960e28e3630
  - --client-id
  - ${WI_CLIENT_ID}
  - --tenant-id
  - ${SP_TENANT_ID}
  - --environment
  - AzurePublicCloud
  - --client-secret
  - ${SP_CLIENT_SECRET}
  command: kubelogin
  env: null
  installHint: |2

    kubelogin is not installed which is required to connect to AAD enabled cluster.

    To learn more, please go to https://aka.ms/aks/kubelogin
  provideClusterInfo: false

$ AZURE_CLIENT_ID="${SP_CLIENT_ID}" kubectl get pod --kubeconfig ${CUSTOM_KUBECONFIG_PATH}
No resources found in default namespace.

Conclusion: --client-id argument in kubeconfig was overriden by Workload Identity. kubectl works correctly, because AZURE_CLIENT_ID was overriden only during running this command.

So as you can see, always the AZURE_CLIENT_ID variable takes precedence over CLI arguments defined in the kubeconfig:

• if the wrong --client-id was put inside the kubeconfig (Workload Identity one), you can override AZURE_CLIENT_ID variable only during running kubectl/helm command, but this is not a deal.
• if the correct --client-id was put inside the kubeconfig (Service Principal one), the kubectl/helm command will fail with 401 error, what gives no possibility to use Service Principal, when there is already Workload Identity injected into the Pod.

Hope everything is understandable.

from kubelogin.

@daleksandrowiczgd does directly overriding the env field help as a workaround?

exec:
  apiVersion: client.authentication.k8s.io/v1beta1
  args:
  - get-token
  - ...
  command: kubelogin
  env:
  - name: "AZURE_CLIENT_ID"
    value: "value for client ID"

from kubelogin.

what is your environment? do you know how AZURE_CLIENT_ID is set?

from kubelogin.

@weinong, our environment:

• Self-managed runners created via Gitlab runner helm chart
• Workload Identity associated with Gitlab runner Pod's Service Account

We migrated from deprecated AAD Pod Identity to the Workload Identity. In our Gitlab CI jobs we want to give us the opportunity to switch between Service Principal or Managed Identity credentials, if needed (e.g. sometimes we have issues with Azure API throttling, so we need to have the option to use SP in such cases). After migration to Workload Identity, we encountered the problem that I described in the issue description.

In general, how the runners config related to the Workload Identity looks like and how those variables are injected:

• Custom ServiceAccount with the annotation azure.workload.identity/client-id: <SP_CLIENT_ID> (<SP_CLIENT_ID> - client ID of the Service Principal)
• Gitlab runner config modified with two additional fields:
  • serviceAccount: <SA_NAME> (<SA_NAME> - the name of the above ServiceAccount)
  • Label azure.workload.identity/use: "true"
• With the above configuration, the newly created runner Pod that runs the Gitlab job will have injected the following 4 variables (but I think only AZURE_CLIENT_ID interfere with kubelogin) :
  • AZURE_CLIENT_ID
  • AZURE_TENANT_ID
  • AZURE_FEDERATED_TOKEN_FILE
  • AZURE_AUTHORITY_HOST

Right now after migration to Workload Identity, it's impossible to use Service Principal nor Managed Identity with different client ID, because AZURE_CLIENT_ID environment variable injected by Workload Identity always takes precedence in the kubelogin commands.

from kubelogin.

@enj yes, this workaround works, thanks for an idea. But we still need to modify every pipeline and script to manually update the kubeconfig file everywhere we want to use the Service Principal credentials.

Maybe it will be good to add AZURE_CLIENT_ID env variable to this exec part in the kubeconfig file, when Service Principal is chosen in the kubelogin convert-kubeconfig command (--login "spn"), wdyt?

Or even better will be to use the --client-id CLI argument that is passed to the get-token command inside the kubeconfig.

from kubelogin.