datadog / datadog-aas-extension Goto Github PK

As far as I can tell, the continuous profiling described here https://docs.datadoghq.com/profiler/enabling/dotnet/ only works on dedicated hosts/VMs/etc. It doesn't work with azure App Services via the extension even if the environment variables to turn the profiler on are set.

https://github.com/DataDog/datadog-aas-extension/tree/master/linux

i read the above document
Applied to azure linux app service.

I am using node js

While looking at the datadog service tab, I confirmed that logging was stopped in the middle.

So I checked the azure log,
It turns out that there is a problem in the middle.

How do I solve this?

===

Below are some of the Azure AppService logs. From that point on, logging started not working.

2023-01-30T03:15:45.485309977Z 2023-01-30 03:15:45 UTC | TRACE | INFO | (pkg/tagger/remote/tagger.go:330 in func1) | unable to establish stream, will possibly retry: unable to fetch authentication token: unable to read authentication token file: open /opt/datadog-agent/etc/auth_token: no such file or directory
2023-01-30T03:15:45.486246300Z 2023-01-30 03:15:45 UTC | TRACE | INFO | (run.go:160 in Run) | starting remote tagger failed. falling back to local tagger: unable to fetch authentication token: unable to read authentication token file: open /opt/datadog-agent/etc/auth_token: no such file or directory
2023-01-30T03:15:45.486286001Z 2023-01-30 03:15:45 UTC | TRACE | INFO | (pkg/workloadmeta/store.go:340 in startCandidates) | workloadmeta collector "cloudfoundry-container" could not start. error: component workloadmeta-cloudfoundry-container is disabled: Agent is not running on CloudFoundry
2023-01-30T03:15:45.486294801Z 2023-01-30 03:15:45 UTC | TRACE | INFO | (pkg/workloadmeta/store.go:340 in startCandidates) | workloadmeta collector "cloudfoundry-vm" could not start. error: component workloadmeta-cloudfoundry-vm is disabled: Agent is not running on CloudFoundry
2023-01-30T03:15:45.486301501Z 2023-01-30 03:15:45 UTC | TRACE | INFO | (pkg/workloadmeta/store.go:340 in startCandidates) | workloadmeta collector "docker" could not start. error: component workloadmeta-docker is disabled: Agent is not running on Docker
2023-01-30T03:15:45.486822713Z 2023-01-30 03:15:45 UTC | TRACE | INFO | (pkg/workloadmeta/store.go:340 in startCandidates) | workloadmeta collector "ecs_fargate" could not start. error: component workloadmeta-ecs_fargate is disabled: Agent is not running on Fargate
2023-01-30T03:15:45.486841114Z 2023-01-30 03:15:45 UTC | TRACE | INFO | (pkg/workloadmeta/store.go:340 in startCandidates) | workloadmeta collector "podman" could not start. error: component workloadmeta-podman is disabled: Podman not detected
2023-01-30T03:15:45.486848014Z 2023-01-30 03:15:45 UTC | TRACE | INFO | (pkg/workloadmeta/store.go:340 in startCandidates) | workloadmeta collector "ecs" could not start. error: component workloadmeta-ecs is disabled: Agent is not running on Docker
2023-01-30T03:15:45.487778736Z 2023-01-30 03:15:45 UTC | TRACE | INFO | (pkg/workloadmeta/store.go:340 in startCandidates) | workloadmeta collector "kube_metadata" could not start. error: component workloadmeta-kube_metadata is disabled: Agent is not running on Kubernetes
2023-01-30T03:15:45.487800437Z 2023-01-30 03:15:45 UTC | TRACE | INFO | (pkg/workloadmeta/store.go:340 in startCandidates) | workloadmeta collector "containerd" could not start. error: component workloadmeta-containerd is disabled: Agent is not running on containerd
2023-01-30T03:15:45.487807937Z 2023-01-30 03:15:45 UTC | TRACE | INFO | (pkg/workloadmeta/store.go:340 in startCandidates) | workloadmeta collector "kubelet" could not start. error: component workloadmeta-kubelet is disabled: Agent is not running on Kubernetes
2023-01-30T03:15:45.487814237Z 2023-01-30 03:15:45 UTC | TRACE | INFO | (pkg/workloadmeta/store.go:154 in Start) | workloadmeta store initialized successfully
2023-01-30T03:15:45.487940140Z 2023-01-30 03:15:45 UTC | TRACE | INFO | (pkg/tagger/collectors/workloadmeta_main.go:115 in stream) | workloadmeta tagger collector started
2023-01-30T03:15:45.490209394Z 2023-01-30 03:15:45 UTC | TRACE | INFO | (run.go:207 in Run) | Trace agent running on host 6bd25e3eb171
2023-01-30T03:15:45.491858033Z 2023-01-30 03:15:45 UTC | TRACE | INFO | (run.go:254 in Infof) | Listening for traces at http://localhost:8126

Extension is installed on Azure app service it the installation looks correct but its not collecting any metric, found this error on one of the logs

.NET Datadog APM 2.13.0

'C:\home\SiteExtensions\Datadog.AzureAppServices.DotNet\v2_13_000\Agent\auth_token': No such file or directory

Hi!

When we deploy the Datadog AAS extension into our web and application API app services we have noticed a dramatic increase in overall application response times, latency, and also what appears to be a memory/handle/thread leak which ultimately leads to extremely poor overall application performance and often requires a restart of said applications. Once we remove the extension response times flatten out to expected norms. We have removed profiling and ASM features and we continue to see the same behavior. I am at a loss to explain the nature of these behaviors when the Datadog AAS extension is introduced. It doesn't matter which version of the AAS extension we use, they all exhibit the same application impacts. We do not see similar patterns when using the Azure Application Insights APM extension. Any guidance/help would be appreciated.

Like this, I also change the hostname
I want to do what I want, but what should I do?
Is it possible with AAS app configuration (environment variable)?
If not, can I change it myself?

https://github.com/DataDog/datadog-agent/releases/tag/7.32.2

Hi guys, hope you're well!

At FlipDish we're avid users of DataDog, many of the integrations you offer and the aas-extension! We'd really like to combine all 3 and make use of the ASP.NET integration with our app services, is there anyway we can do this?

I have read the configuration instructions for the integration and I am familiar with the integration configuration process, having configured the SQL Server integration on a standalone agent we host. However, from the docs and having a root about myself I don't see any way to do this configuration while using the Azure App Service extension. I know that the extension runs an agent instance on the app service hosts so I'm just wondering if there is a way to configure it to perform integration checks?

The other option, as I see it, would be to configure a stand alone agent to remote into the app service host and perform the checks - I can see there are configuration options for this in the aspnet.d conf example. However, I don't think Azure users have this level of access to the underlying app service hosts, but would be glad to be proved wrong on this! 😄

Keen to get your thoughts and advice on how best to proceed with this.

Cheers, James.

Description

When setting up the site extension Datadog.AzureAppServices.DotNet for an Azure Function v4, the extension installs successfully, however there are errors on site transform and no metrics are sent to Datadog.

Context

Datadog.AzureAppServices.DotNet: v2.11.0
Azure Functions: v4
.NET: 6.0

Logging

%AzureAppServiceHomeDirectory%\LogFiles\datadog

[(v2.11.000) 2022-07-07 00:50 +00:00] Beginning install
[(v2.11.000) 2022-07-07 00:50 +00:00] Setting uniqueStatsPipeId to ECCFB9EB-E296-4BB9-930E-88EFCA34A745 in .\applicationHost.xdt
[(v2.11.000) 2022-07-07 00:50 +00:00] Setting uniqueTracePipeId to 7F8F4DAB-CD6F-493E-AAE1-DE0068B0F411 in .\applicationHost.xdt
[(v2.11.000) 2022-07-07 00:50 +00:00] Setting uniqueStatsPipeId to ECCFB9EB-E296-4BB9-930E-88EFCA34A745 in .\scmApplicationHost.xdt
[(v2.11.000) 2022-07-07 00:50 +00:00] Setting uniqueTracePipeId to 7F8F4DAB-CD6F-493E-AAE1-DE0068B0F411 in .\scmApplicationHost.xdt
[(v2.11.000) 2022-07-07 00:50 +00:00] Install complete

%AzureAppServiceHomeDirectory%\LogFiles\Transform

2022-07-07T01:00:30 sandboxproc.exe D:\DWASFiles\Sites#1fn-xxxx-api\Temp\applicationhost.config False True
2022-07-07T01:00:30 env XPROC_TYPENAME=Microsoft.Web.Hosting.Transformers.ApplicationHost.SiteExtensionHelper, Microsoft.Web.Hosting, Version=7.1.0.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35
2022-07-07T01:00:30 env XPROC_METHODNAME=Transform
2022-07-07T01:00:31 Start 'Datadog.AzureAppServices.DotNet' site extension transform
2022-07-07T01:00:31 sandboxproc.exe Elapsed = 485.00 ms failed with System.Reflection.TargetInvocationException: Exception has been thrown by the target of an invocation. ---> System.Xml.XmlException: The 'configuration' start tag on line 2 position 2 does not match the end tag of 'runtime'. Line 25, position 7.
at System.Xml.XmlTextReaderImpl.Throw(Exception e)
at System.Xml.XmlTextReaderImpl.ThrowTagMismatch(NodeData startTag)
at System.Xml.XmlTextReaderImpl.ParseEndElement()
at System.Xml.XmlTextReaderImpl.ParseElementContent()
at System.Xml.XmlLoader.LoadNode(Boolean skipOverWhitespace)
at System.Xml.XmlLoader.LoadDocSequence(XmlDocument parentDoc)
at System.Xml.XmlDocument.Load(XmlReader reader)
at Microsoft.Web.XmlTransform.XmlFileInfoDocument.Load(XmlReader reader)
at System.Xml.XmlDocument.LoadXml(String xml)
at Microsoft.Web.XmlTransform.XmlTransformation..ctor(String transform, Boolean isTransformAFile, IXmlTransformationLogger logger)
at Microsoft.Web.Hosting.Transformers.ApplicationHost.SiteExtensionDefinition.Transform(XmlDocument document, IDictionary2 environments, Boolean isMainSite, Boolean separateProcessForScm) at Microsoft.Web.Hosting.Transformers.ApplicationHost.SiteExtensionHelper.Transform(XmlDocument doc, List1 definitions, IDictionary`2 environments, Boolean isMainSite, Boolean separateProcessForScm)
at Microsoft.Web.Hosting.Transformers.ApplicationHost.SiteExtensionHelper.Transform(String appHostConfig, String isMainSiteStr, String separateProcessForScmStr)
--- End of inner exception stack trace ---
at System.RuntimeMethodHandle.InvokeMethod(Object target, Object[] arguments, Signature sig, Boolean constructor)
at System.Reflection.RuntimeMethodInfo.UnsafeInvokeInternal(Object obj, Object[] parameters, Object[] arguments)
at System.Reflection.RuntimeMethodInfo.Invoke(Object obj, BindingFlags invokeAttr, Binder binder, Object[] parameters, CultureInfo culture)
at Microsoft.Web.Hosting.ProcessModel.Program.Main(String[] args)

DD_TRACE_DEBUG has been set to true, but no other logs are present.

Possible Solution

https://github.com/DataDog/datadog-aas-extension/blob/master/dotnet/content/scmApplicationHost.xdt contains a mismatched runtime tag, as referenced by the error. This seems to be fixed in #111 so it is possible that merging this PR will solve the issue.

Additional Information

I have an open ticket with Datadog here: https://help.datadoghq.com/hc/en-us/requests/855745

Hey Team ,

if we are using AAS extension for tracking automating tracing data, how can we track the deployments when these apps are deployed via Azure pipeline stages ?

More interms of https://docs.datadoghq.com/tracing/services/deployment_tracking/

Hello! The current version that the Datadog AAS Extension has published includes the DD Agent version 7.45.0 (released on June 5th). The current version is 7.48.1 which represents 3 "big" updates since. Is there a particular blocker to updating to a more recent version? My main impetous for wanting this is that https://docs.datadoghq.com/database_monitoring/connect_dbm_and_apm/?tab=net is listed as supporting only version 7.46+.

When enabled, we see the jars loaded through the JAVA_OPTS as expected.

With Site Extensions Enabled:
JAVA_OPTS -javaagent:D:\home\SiteExtensions\DevelopmentVerification.DdJava.Apm\v0_1_4-prerelease\Tracer\dd-java-agent.jar -Ddatadog.slf4j.simpleLogger.logFile=D:\home\LogFiles\datadog\java\v0_1_4-prerelease\dd-java-agent.log -Dsite.logdir=D:/home/LogFiles -Dsite.home=D:/home -Dsite.libs=D:/home/site/libs -Dsite.appbase=D:/home/site/wwwroot/webapps -Dsite.xmlbase=D:/home/site/wwwroot -Dsite.unpackwars=true -Dsite.tempdir=D:\local\Temp -Dcatalina.instance.name=ff951f732a5191dd5c5a023ee55ef0c677b50ec7eb042339d18fa4c73cf79781 -Dport.http=2560 -Djava.net.preferIPv4Stack=true -noverify -Dsite.logRetentionDays=0 -Dsite.logRotatable=true -Dsite.connectionTimeout=240000 -Djdk.tls.ephemeralDHKeySize=2048 -Djava.protocol.handler.pkgs=org.apache.catalina.webresources

The conflict arises if the user passes arguments to the JAVA_OPTS through the app settings. Any existing arguments passed through the app settings using JAVA_OPTS look to be ignored when this site extension was enabled.

Issue may be around the xdt:Transform if we can try inserting.
https://github.com/DataDog/datadog-aas-extension/blob/master/java/content/applicationHost.xdt#L54

I have two Azure app services with the same .NET Framework 4.8 app. The first has .NET Datadog APM extension v. 1.6.0 installed and works fine for a while, on the second I've just installed the latest version 2.6.0 and I could not get the site working anymore getting the following exception on each request:

System.Data.Entity.Core.EntityCommandExecutionException: An error occurred while executing the command definition. See the inner exception for details. ---> System.InvalidProgramException: Common Language Runtime detected an invalid program.
   at EFCache.CachingCommand.ExecuteDbDataReader(CommandBehavior behavior)
   at System.Data.Entity.Infrastructure.Interception.InternalDispatcher`1.Dispatch[TTarget,TInterceptionContext,TResult](TTarget target, Func`3 operation, TInterceptionContext interceptionContext, Action`3 executing, Action`3 executed)
   at System.Data.Entity.Infrastructure.Interception.DbCommandDispatcher.Reader(DbCommand command, DbCommandInterceptionContext interceptionContext)
   at System.Data.Entity.Internal.InterceptableDbCommand.ExecuteDbDataReader(CommandBehavior behavior)
   at System.Data.Entity.Core.EntityClient.Internal.EntityCommandDefinition.ExecuteStoreCommands(EntityCommand entityCommand, CommandBehavior behavior)
   --- End of inner exception stack trace ---
   at System.Data.Entity.Core.EntityClient.Internal.EntityCommandDefinition.ExecuteStoreCommands(EntityCommand entityCommand, CommandBehavior behavior)
   at System.Data.Entity.Core.Objects.Internal.ObjectQueryExecutionPlan.Execute[TResultType](ObjectContext context, ObjectParameterCollection parameterValues)
   at System.Data.Entity.Core.Objects.ObjectContext.ExecuteInTransaction[T](Func`1 func, IDbExecutionStrategy executionStrategy, Boolean startLocalTransaction, Boolean releaseConnectionOnSuccess)
   at System.Data.Entity.Core.Objects.ObjectQuery`1.<>c__DisplayClass7.<GetResults>b__5()
   at System.Data.Entity.SqlServer.DefaultSqlExecutionStrategy.Execute[TResult](Func`1 operation)
   at System.Data.Entity.Core.Objects.ObjectQuery`1.GetResults(Nullable`1 forMergeOption)
   at System.Data.Entity.Core.Objects.ObjectQuery`1.<System.Collections.Generic.IEnumerable<T>.GetEnumerator>b__0()
   at System.Data.Entity.Internal.LazyEnumerator`1.MoveNext()
   at System.Linq.Enumerable.FirstOrDefault[TSource](IEnumerable`1 source)

When I installed v. 1.6.0 on the second app service it started working fine.

Why no custom log agent collector for app service? We were forced to use agentless method. https://docs.datadoghq.com/logs/log_collection/csharp/?tab=serilog#agentless-logging

It would be great to included the log agent into the App Service Extension as well. (where we can use environment variables to enable the log agent and customize the log path.)

example:

• DD_LOG_ENABLED: "true",
• DD_LOG_PATH: "D:\Home\LogFiles\myapp\web-.log" # Log path wildcard

After installing this extension on .NET 6 azure app service application.
There is error in dogstatsd log file ( LogFiles/datadog/v2_14_000/dogstatsd.txt )

2022-09-19 19:04:43 GMT | DSD | ERROR | (pkg/util/log/log.go:612 in func1) | Error creating expvar server on port 5000: listen tcp 127.0.0.1:5000: bind: Only one usage of each socket address (protocol/network address/port) is normally permitted.
2022-09-19 19:04:43 GMT | DSD | WARN | (pkg/util/log/log.go:592 in func1) | Unknown environment variable: DD_AAS_DOTNET_EXTENSION_VERSION
2022-09-19 19:04:43 GMT | DSD | WARN | (pkg/util/log/log.go:592 in func1) | Unknown environment variable: DD_AGENT_PIPE_NAME
2022-09-19 19:04:43 GMT | DSD | WARN | (pkg/util/log/log.go:592 in func1) | Unknown environment variable: DD_AZURE_APP_SERVICES
2022-09-19 19:04:43 GMT | DSD | WARN | (pkg/util/log/log.go:592 in func1) | Unknown environment variable: DD_DOGSTATSD_ARGS
2022-09-19 19:04:43 GMT | DSD | WARN | (pkg/util/log/log.go:592 in func1) | Unknown environment variable: DD_DOGSTATSD_PATH
2022-09-19 19:04:43 GMT | DSD | WARN | (pkg/util/log/log.go:592 in func1) | Unknown environment variable: DD_DOGSTATSD_WINDOWS_PIPE_NAME
2022-09-19 19:04:43 GMT | DSD | WARN | (pkg/util/log/log.go:592 in func1) | Unknown environment variable: DD_DOTNET_TRACER_HOME
2022-09-19 19:04:43 GMT | DSD | WARN | (pkg/util/log/log.go:592 in func1) | Unknown environment variable: DD_INTEGRATIONS
2022-09-19 19:04:43 GMT | DSD | WARN | (pkg/util/log/log.go:592 in func1) | Unknown environment variable: DD_INTERNAL_NATIVE_LOADER_PATH
2022-09-19 19:04:43 GMT | DSD | WARN | (pkg/util/log/log.go:592 in func1) | Unknown environment variable: DD_INTERNAL_PROFILING_NATIVE_ENGINE_PATH
2022-09-19 19:04:43 GMT | DSD | WARN | (pkg/util/log/log.go:592 in func1) | Unknown environment variable: DD_INTERNAL_TRACE_NATIVE_ENGINE_PATH
2022-09-19 19:04:43 GMT | DSD | WARN | (pkg/util/log/log.go:592 in func1) | Unknown environment variable: DD_PROFILER_EXCLUDE_PROCESSES
2022-09-19 19:04:43 GMT | DSD | WARN | (pkg/util/log/log.go:592 in func1) | Unknown environment variable: DD_TRACE_AGENT_ARGS
2022-09-19 19:04:43 GMT | DSD | WARN | (pkg/util/log/log.go:592 in func1) | Unknown environment variable: DD_TRACE_AGENT_PATH
2022-09-19 19:04:43 GMT | DSD | WARN | (pkg/util/log/log.go:592 in func1) | Unknown environment variable: DD_TRACE_LOG_PATH
2022-09-19 19:04:43 GMT | DSD | WARN | (pkg/util/log/log.go:592 in func1) | Unknown environment variable: DD_TRACE_METRICS_ENABLED
2022-09-19 19:04:43 GMT | DSD | WARN | (pkg/util/log/log.go:592 in func1) | Unknown environment variable: DD_TRACE_PIPE_NAME
2022-09-19 19:04:43 GMT | DSD | WARN | (pkg/util/log/log.go:592 in func1) | Unknown environment variable: DD_TRACE_TRANSPORT
2022-09-19 19:04:43 GMT | DSD | WARN | (pkg/util/log/log.go:592 in func1) | Unknown environment variable: DD_PROFILING_CPU_ENABLED
2022-09-19 19:04:43 GMT | DSD | WARN | (pkg/util/log/log.go:592 in func1) | Unknown environment variable: DD_PROFILING_ENABLED
2022-09-19 19:04:43 GMT | DSD | WARN | (pkg/util/log/log.go:592 in func1) | Unknown environment variable: DD_PROFILING_EXCEPTION_ENABLED
2022-09-19 19:04:43 GMT | DSD | WARN | (pkg/util/log/log.go:592 in func1) | Unknown environment variable: DD_SERVICE
2022-09-19 19:04:43 GMT | DSD | WARN | (pkg/util/log/log.go:592 in func1) | Unknown environment variable: DD_TRACE_DEBUG
2022-09-19 19:04:43 GMT | DSD | WARN | (pkg/metadata/helper.go:99 in SetupMetadataCollection) | Metadata collection disabled, only do that if another agent/dogstatsd is running on this host
2022-09-19 19:04:43 GMT | DSD | WARN | (pkg/config/viper.go:220 in GetStringSlice) | failed to get configuration value for key "proxy.no_proxy": unable to cast <nil> of type <nil> to []string

I have been trying to make profiling for hot code paths work, could this be related?

We are using version 1.6.0. Unauthenticated bogus path request is logged as 200 OK instead of 404.

Hi, we installed the extension in Azure AppService, it worked for App Services to emit custom metrics, however, when we trying to emit custom metrics in a web job hosted in the same App Service, there was an error saying config.StatsdServerName and DD_AGENT_HOST environment variable not set. Upon further investigation, it appears the Datadog Azure App Service is using named pipe for the client SDK to connect to the DogStatsd agent. But not sure if how to configure the client in an App Service web job.

Hi all,

We are using datadog extension on azure app service to collect logs by agentless APM, it is missing "}" at the end of logs when the logs is deeply nested, the problem happen when json nested more than 3 layers.

The debug process as below:

1. I try to send logs to datadog by agentless serilog sink, the logs don't missing "}" at the end, the datadog give me a MessageTemplate event attributes, it doesn't missing "}" in this event attribute.
2. I try to write logs to Azure console, i check it in azure app service console , it doesn't missing "}" also.

All of our app services have default publishing credentials, so downloading the publishing profile for each is a pain. Our build scripts get the publishing creds and then use them to deploy our code, so I made a added some code to one of the install scripts and got something working.

Between the original check for "ambient" and creating the BASE64 encoded creds, I added the following block which should still support the use of the ambient (env variables).

if($Username -eq "ambient" -or $Username -eq "") {
	Write-Output "No username provided. Fetching publish crendentials from Kudu..."
	$publishProfile = (az webapp deployment list-publishing-credentials --name ${SiteName} --resource-group ${ResourceGroup} --subscription ${SubscriptionId} | ConvertFrom-Json)
	$Username = $publishProfile.publishingUserName
	$Password = $publishProfile.publishingPassword
}