Correct correlation for Microsoft.Azure.Storage.DataMovement about applicationinsights-aspnetcore HOT 12 CLOSED

@Planche95 Correct me if I understood the question wrong. ParentID for all blob operations are same because they are all done within the scope of 'StartOperation'(). So all those calls are treated as child of the same operation. From your screenshot of E2E, all are correctly shown correlated as well.

I may have misunderstood your question here - if you expected to see a different behaviour, can you describe what that behaviour is?

from applicationinsights-aspnetcore.

I have started operation twice, so I expect that parentId of blob operations will be different in each "StartOperation()" Scope, just like in http calls to "bing.com"

from applicationinsights-aspnetcore.

Yes that's expected behavior. All calls inside 1st operation will be parented to that operation, and the ones from 2nd operation to the 2nd operation.

Can you run the following query for each operation (i.e for operationid1 and operationid2) and paste results. Its lot easier for investigation to see all items are correctly parented.

union *
| project timestamp, itemType , operation_Id , operation_ParentId
| where operation_Id == "operationid"

from applicationinsights-aspnetcore.

For the first operation:

For second:

CustomEvent, Warning from log and Get to bing.com was correlated correctly, only DataMovement operations don't have proper parent-id. It looks like it set up parent-id once and never change it. e.g. if I do something like this:

await DownloadFile("HubbleUltraDeepField.png");

using (telemetryClient.StartOperation<RequestTelemetry>("operation"))
{
  _logger.LogWarning("Warning!");
  var res = await new HttpClient().GetAsync("https://bing.com");
  telemetryClient.TrackEvent("Bing call event completed");

  await DownloadFile("HubbleUltraDeepField.png");
}

For the first "DownloadFile" I don't startedOperation so parentId was not assigned, and when I look into metrics from second "DownloadFile" inside startedOperation the situation doesn't change and it still doesn't have parent-id!

from applicationinsights-aspnetcore.

Any updates? Are u able to reproduce the issue?

from applicationinsights-aspnetcore.

I couldn't repro with regular storage client. (but you mentioned this in your original post as well).
Will try with datamovement api now.

from applicationinsights-aspnetcore.

I got this repro-ed with datamovement library.

from applicationinsights-aspnetcore.

@lmolkova Do you know about DataMovement library, and any special cases with that?

from applicationinsights-aspnetcore.

Any update how to fix that? :)

from applicationinsights-aspnetcore.

Looking into the implementation of DataMovement, they do put requests into the internal queue and then process requests from the queue.

This does not work well with .NET async context flow (that we leverage for correlation) and Application Insights cannot do anything to fix this. DataMovement library would need to flow execution context to work items to allow .NET correlation mechanisms to work.

So I'm closing this as there is not much we can do here. Feel free to reopen.

from applicationinsights-aspnetcore.

@lmolkova
This is Emma from DataMovement team. I'd need some background about the issue. Could you share mote details on "flow execution context to work items to allow .NET correlation mechanisms to work"?

from applicationinsights-aspnetcore.

@EmmaZhu

.NET (and ApplicationInsights uses Activity based on AsyncLocal for implicit context propagation in-process.

This works over the execution context supported by .NET and works nicely with async scenarios. It does not work with processing things in queues in background thread.

The way to work it around is to capture Activity.Current when item is added to the queue and resetting it when you process item.

Activity.Current is only present when there is a tracing system that lights up instrumentation in .NET,. So if there is no tracing system, you check it for null and do no extra work to propagate/restore it.

Let me know if it helps and if you want to learn more about it.

from applicationinsights-aspnetcore.