No container id found by container id providers after AKS upgrade 1.23->1.25 about applicationinsights-kubernetes HOT 15 CLOSED

Pod can be recreate with the same name, but there can only be one with the same name at a time. Object Names and IDs

If I tested it right... if Container Id missing then Application Insight not showing "Cloud Role Name".
But is show and populate this custom properties

Kubernetes.Node.Name
Kubernetes.Pod.Name
Kubernetes.Pod.Namespace
Kubernetes.Node.ID
Kubernetes.Deployment.Name
Kubernetes.Pod.ID
Kubernetes.ReplicaSet.Name
Kubernetes.Pod.Labels

from applicationinsights-kubernetes.

@TTmaister your change has been released:

https://www.nuget.org/packages/Microsoft.ApplicationInsights.Kubernetes/6.1.1-beta2

Let me know if that unblocks you. Thank you for your contribution!

Works as expected.

1. One container no need for environment variable to get "Cloud Role Name" visible.
2. More than one container and ContainerName environment variable is not present "Cloud Role Name" not visible.
3. Multiple containers and ContainerName environment variable is preset "Cloud Role Name" visible.

@xiaomi7732 Maybe this should be documented to project readme.

from applicationinsights-kubernetes.

Hey @Ismael-Pep, thanks for the inquiry. Here's the new package is released today:
https://www.nuget.org/packages/Microsoft.ApplicationInsights.Kubernetes/6.1.1

from applicationinsights-kubernetes.

Hi @TTmaister, this is a known issue because of the implementations of docker engines. There has been a related issue #290.

Please check out the design doc here for the container id providers: https://github.com/microsoft/ApplicationInsights-Kubernetes/wiki/Design-for-ContainerIdProviders

That said, 2 questions for you:

1. Is the container id anywhere in the mountinfo you shared? If it is, we might be able to append another container id providers for your case.

2. Is it possible for you to set up the environment variable for container id as mentioned in the design doc to workaround the issue? For example: ContainerId=my-container-id.

Any proposal for how to make this more adaptive is super-welcome!

BTW, just FYI, here's a lengthy stackoverflow discussion on the topic:
https://stackoverflow.com/questions/20995351/how-can-i-get-docker-linux-container-information-from-within-the-container-itsel/72565733#72565733

And an open standard issue:

opencontainers/runtime-spec#1105

from applicationinsights-kubernetes.

1. Container ID was 34bc656c1739fe713a867e18cc307f96e6fcba3cf89755db31b3c837f9d137f5. The answer is that there is no container id visible in mountinfo.

2. Possibly I can set the container id in the environment variables, but I don't know how to do it in the Kubernetes Deployment yaml so that each pod has its own ID. Does it have to be the same as what the Kubernetes Api provides.

from applicationinsights-kubernetes.

Hey @TTmaister, thanks for providing the information. Unfortunately, without container id anywhere in the container, we won't have the magic to somehow make it available.

Manually set the environment would probably be the only way that will provide full enhancement.
I don't know what will happen if you set container id to a mismatched value. That is an interesting idea. I assume you could set the environment variable to a random value for identification of the instances, but will it provide too much value? (pods comes and goes, and if the container id is random, why bother?)

That said, could you please help me understand what the telemetry looks like on your end? I assume container id missing but everything else, like podName, node info should be there on the events, does that align with what you see?

from applicationinsights-kubernetes.

Application Insight is missing all the data brought by the plugin. I can see in Application Insight that the POD requests information about the POD from the Kubernetes API and gets the response http200.

from applicationinsights-kubernetes.

Hey @xiaomi7732

Does it make any sense to change the ContainerIdHolder class to be able to parse the ContainerName variable from the environment variables.

And then filtter ContainerStatus by ContainerName

 if (containerStatuses is not null && containerStatuses.Count > 1)
        {
            string? containerName = Environment.GetEnvironmentVariable("ContainerName");
            containerStatus = containerStatuses.FirstOrDefault(c => c.Name == containerName);
            _logger.LogInformation(FormattableString.Invariant($"Use the only container inside the pod for container id: {containerStatus.ContainerID}"));

            using (IServiceScope scope = _serviceScopeFactory.CreateScope())
            {
                IContainerIdNormalizer normalizer = scope.ServiceProvider.GetRequiredService<IContainerIdNormalizer>();
                if (normalizer.TryNormalize(containerStatus.ContainerID, out string? normalizedContainerId))
                {
                    _containerId = normalizedContainerId;
                    return true;
                }
            }

            _logger.LogError(FormattableString.Invariant($"Normalization failed for container id: {containerStatus.ContainerID}"));
        }

ContinerName can be defined as environment variable Kubernetes Deployment yaml.

env:
  - name: ContainerName
     valueFrom:
       fieldRef:
         fieldPath: metadata.labels['app']
         

Or some other label. We have set "app" label for all deployments.

from applicationinsights-kubernetes.

Hi @TTmaister, using ContainerName for matching seem like an approach. I think we should peruse it when time permits. Do you know is pod name always unique?

With regarding what is going on, I think I see it. Container id was supposed to be optional, but it is required - in between several iterations to support .NET 6 and the latest K8s SDK. See below for the details:

It was like this, there had been a note to allow empty container id.

// Notes: It is still possible for the optional container id to be empty at this point, the following method needs to handle the case.
if (!await SpinWaitContainerReadyAsync(timeoutAt, queryClient, myPod, containerId, cancellationToken).ConfigureAwait(false))
{
    _logger.LogError(Invariant($"Kubernetes info is not available before the timeout at {timeoutAt}."));
    return null;
}

And in the implementation, this is the logic to allow empty container id:

if (!string.IsNullOrEmpty(myContainerId))
{
    // Check targeted container status
    readyToGo = IsContainerReady(podInfo.GetContainerStatus(myContainerId));
}
else
{
    _logger.LogWarning("No container id available. Fallback to use the any container for status checking.");
    readyToGo = podInfo.GetAllContainerStatus().Any(s => IsContainerReady(s));
}

And it is like this now, meaning optional container id is not allowed:

public async Task<bool> IsContainerReadyAsync(CancellationToken cancellationToken)
{
    V1ContainerStatus? myContainerStatus = await GetMyContainerStatusAsync(cancellationToken).ConfigureAwait(false);
    if (myContainerStatus is not null)
    {
        return IsContainerStatusReady(myContainerStatus);
    }

    return false;
}

That doesn't agree with the design that I shared earlier and shall be treated as a bug. Will it unblock you if I make it optional again?

from applicationinsights-kubernetes.

That is correct. FYI, the line of code:

from applicationinsights-kubernetes.

I think there could be 2 fixes:

1. Make container id optional (PR #345 )
2. Allows user the specify Container Name. (#346)

And I inspected various options to back fill container id, and I think using container name is a very good idea. It might worth a bit documentation but could be useful as an alternative to container ids.

@TTmaister are you interested to submit a PR for it? Or I can prepare a PR if you don't have the time.

BTW, if you are interested, please fork this repository, and branching off your own fork since this repository has been locked down and nobody could PR into it directly. :-)

from applicationinsights-kubernetes.

@TTmaister your change has been released:

https://www.nuget.org/packages/Microsoft.ApplicationInsights.Kubernetes/6.1.1-beta2

Let me know if that unblocks you. Thank you for your contribution!

from applicationinsights-kubernetes.

@TTmaister, thanks for the verification! I will update the wiki. I'll keep this issue open until the stable version of 6.1.1 got released.

from applicationinsights-kubernetes.

A FAQ is added:
https://github.com/microsoft/ApplicationInsights-Kubernetes/wiki/FAQ, pointing to the wiki: How to set container name manually.

from applicationinsights-kubernetes.

Is there an Eta on when the version 6.1.1 will be released?, Thx!

from applicationinsights-kubernetes.