Custom instrumentation? about dd-trace-java HOT 14 CLOSED

We allow methods to be instrumented via configuration, similar to adding @Trace to the method, but that only does simple method timing. We don't currently have a way to dynamically load external instrumentation.

Can you give some additional background on what you're trying to accomplish?

from dd-trace-java.

Ideally it'd be something like this: https://github.com/opentracing-contrib/java-agent#creating-custom-rules

I have the need to get spans around code that I do not control the source of. So being able to add traces to that code, and the code it calls, would be fantastically helpful.

from dd-trace-java.

have you seen the trace.methods configuration in the docs? I think that might solve the problem to some degree with the existing tracer.

from dd-trace-java.

That didn't seem to work. I've setup the DD_TRACE_METHODS value to 3 fully qualified methods in the form org.springframework.jms.core.JmsTemplate[send] and nothing gets traced even though that method is certainly called. The other 2 methods are not traced either.

from dd-trace-java.

I'm thinking perhaps there might be something else going on. Might be a good idea to open a support request where we can dig in deeper.

Thanks!

from dd-trace-java.

have you seen the trace.methods configuration in the docs? I think that might solve the problem to some degree with the existing tracer.

trace.methods do not offer any wildcard capability. As the result it looks very unmanageable to instrument many classes. Look how long it looks just for 2 classes from Quartz scheduler:

            - name: DD_TRACE_METHODS
              value: "org.quartz.impl.StdScheduler[addCalendar,addJob,addJob,checkExists,checkExists,clear,deleteCalendar,deleteJob,deleteJobs,interrupt,interrupt,isInStandbyMode,isShutdown,isStarted,pauseAll,pauseJob,pauseJobs,pauseTrigger,pauseTriggers,rescheduleJob,resumeAll,resumeJob,resumeJobs,resumeTrigger,resumeTriggers,scheduleJob,scheduleJob,scheduleJob,scheduleJobs,shutdown,shutdown,standby,start,startDelayed,triggerJob,triggerJob,unscheduleJob,unscheduleJobs];org.quartz.impl.RemoteScheduler[addCalendar,addJob,addJob,checkExists,checkExists,clear,deleteCalendar,deleteJob,deleteJobs,interrupt,interrupt,pauseAll,pauseJob,pauseJobs,pauseTrigger,pauseTriggers,rescheduleJob,resumeAll,resumeJob,resumeJobs,resumeTrigger,resumeTriggers,scheduleJob,scheduleJob,scheduleJob,scheduleJobs,shutdown,shutdown,standby,start,startDelayed,triggerJob,triggerJob,unscheduleJob,unscheduleJobs];denver.commerce.backoffice.scheduler.TaskedJobRunShell[initialize,run,notifyListenersBeginning,notifyJobListenersComplete,notifyTriggerListenersComplete];org.quartz.impl.jdbcjobstore.JobStoreSupport$ClusterManager[initialize,run,shutdown];org.quartz.impl.jdbcjobstore.JobStoreSupport[acquireNextTriggers,canUseProperties,checkExists,checkExists,clearAllSchedulingData,equals,executeWithoutLock,initialize,notify,notifyAll,pauseAll,pauseAll,pauseJob,pauseJobs,pauseTrigger,pauseTrigger,pauseTriggerGroup,pauseTriggers,releaseAcquiredTrigger,removeCalendar,removeJob,removeJobs,removeTrigger,removeTriggers,replaceTrigger,resumeAll,resumeAll,resumeJob,resumeJobs,resumeTrigger,resumeTrigger,resumeTriggerGroup,resumeTriggers,retrieveCalendar,retrieveJob,retrieveTrigger,schedulerPaused,schedulerResumed,schedulerStarted,shutdown,storeCalendar,storeJob,storeJobAndTrigger,storeJobsAndTriggers,storeTrigger,supportsPersistence,toString,triggeredJobComplete,triggersFired,wait,wait,wait,acquireNextTrigger,calcFailedIfAfter,calendarExists,checkBlockedState,checkExists,checkExists,cleanupConnection,clearAllSchedulingData,clearAndGetSignalSchedulingChangeOnTxCompletion,clone,closeConnection,clusterCheckIn,clusterRecover,commitConnection,doCheckin,doRecoverMisfires,executeInLock,executeInNonManagedTXLock,finalize,findFailedInstances,jobExists,logWarnIfNonZero,recoverJobs,recoverJobs,recoverMisfiredJobs,releaseAcquiredTrigger,releaseLock,removeCalendar,removeJob,removeTrigger,replaceTrigger,retrieveCalendar,retrieveJob,retrieveTrigger,retryExecuteInNonManagedTXLock,rollbackConnection,signalSchedulingChangeImmediately,signalSchedulingChangeOnTxCompletion,storeCalendar,storeJob,storeTrigger,triggeredJobComplete,triggerExists,triggerFired,updateMisfiredTrigger]"

from dd-trace-java.

@vgarmash-chewy we are evaluating a file based option that will hopefully be more manageable. Stay tuned.

from dd-trace-java.

I hope that "file" solution would also include AOP style.

from dd-trace-java.

@vgarmash-chewy would you be interested in submitting a PR to automatically instrument quartz? That might be the best option...

from dd-trace-java.

I believe that instrumentation of the library without full knowledge of all it's functions is very hard task. It will take a lot of time and a lot of iterations to get it right.

I think the best option would be having universal simple way to instrument any java library without accessing it's source code. Since DD Java agent has access to wrap any Java class it should be possible to define pointcuts with less explicit syntax than it is now. It can be in AspectJ style, or your custom, but it should include wildcards in some form. It is not practical to expect full path to the class and it's methods in 1 environment variable or 1 system property as it is done currently.

from dd-trace-java.

@vgarmash-chewy if you're looking for something AOP style, I suggest you use something like Spring AOP or AspectJ and add the custom instrumentation yourself. If the AOP framework allows adding annotations, you can add @Trace, otherwise adding the OpenTracing logic shouldn't be difficult with AOP. I believe some AOP frameworks even allow you to make the changes at compile time. This would give you far more flexibility and control than anything we come up with.

from dd-trace-java.

@vgarmash-chewy I don't mean to hijack this conversation but have you tried BTrace? You can specify the instrumentation externally in pure annotated Java and then attach the agent on JVM startup or dynamically. It does not have OpenTracing support OOTB but should not be too difficult to do quick hack to see whether it is usable or not.

from dd-trace-java.

WE

@vgarmash-chewy I don't mean to hijack this conversation but have you tried BTrace? You can specify the instrumentation externally in pure annotated Java and then attach the agent on JVM startup or dynamically. It does not have OpenTracing support OOTB but should not be too difficult to do quick hack to see whether it is usable or not.

You are right - this conversation is about capabilities of Datadog Tracing agent which are not up to the level with other popular tracing solutions. The goal here is to provide feedback to the Datadog team so they can improve their agent.

from dd-trace-java.

I created 2 separate issues to track the feature requests. Closing this issue.

from dd-trace-java.