Increasing CPU usage about kamon HOT 14 CLOSED

Here's a view of the heap, in case it may be useful:

And the GC activity (that frees about 300MB every time it's started - every 2 to 5 seconds):

from kamon.

Here's the CPU call tree 5 minutes after the application started:

And the GC activity (idle):

from kamon.

It seems like you have a increasing number of different traces over time, did you include the traceName directive in your route to group requests that effectively represent the same functionality?

Currently Kamon does not evict unused trace names from the metrics collection registry, so over time you might be getting many snapshots with Zero measurements that eventually makes the buffer cry.

from kamon.

No right now it uses the URLs as trace names, that may contain variables. So it may totally be the cause of the issue, thanks for that.

I've put custom trace names on the next update that I want to push to the server, but I can't deploy it until the issue discussed here is fixed: #37 (comment)

from kamon.

Diego created a new issue for that item (#38) that has been resolved.. let me publish a new snapshot including those changes for you.

from kamon.

Perfect, thanks.
I'll try to quickly review the change and deploy it in the next 20-30 minutes.

from kamon.

0.3.1-b0ec8c43a4fa4f7546eabdf141726fab61db48b7 is in the snapshots repo already! Another thing that you can do to alleviate this problem until you can include the traceName directive in your routes is to increase the kamon.metrics.tick-interval setting to a higher value. To give you a little intro on that, Kamon flushes the metrics data to all registered subscribers (as the New Relic module) periodically, in intervals controlled by the kamon.metrics.tick-interval setting.. in your case, since you are using the New Relic module which reports the data every 60 seconds and (i guess) no other reporting module is being used, there is no sense in having the tick-interval so low.

from kamon.

Thanks @ivantopo. So I've deployed the update (with Scala 2.11, latest screenshot, and custom trace names) and it works well, but it doesn't seem to report anymore to New Relic.
I've tried restarting the app several times.
I also believe that there were several times something like Kamon(NewRelic) extension loaded logged at startup, that I don't see anymore.

from kamon.

common questions:

• are you sure that the AspectJ weaver is in place?
• maybe you put some specific filters under the kamon.metrics.filters section that no longer match the new trace names?
• are you seeing the usual output from the New Relic module at application startup that tells you to which collector it got connected?

from kamon.

• Yes, AspectJ 1.8.0, I haven't changed anything on this side
• I've never used that property and just checked that it was not in our configuration
• No I don't see any log related to New Relic, and I'm pretty sure there were some in the last snapshot version (the one that produced NPE with multiple actor systems)

from kamon.

give me a few minutes to review this and get back to you.

from kamon.

Cool, thanks

from kamon.

So thanks to an awesome 1-to-1 support by @ivantopo (thank you again for that!), we solved the issue I was talking about here: #39 (comment).

Turned out that I moved the Akka configuration related to Kamon from the root of the configuration to a configuration block specific to the actor system used by Spray. This block:

akka {
  event-handlers = ["akka.event.slf4j.Slf4jEventHandler", "kamon.newrelic.NewRelicErrorLogger"]
  extensions = ["kamon.newrelic.NewRelic"]
}

But left the block related to New Relic at the root of the configuration. This block:

kamon {
  newrelic {
    app-name="<application name>"
    license-key="<license key>"
  }
}

So when Kamon got the configuration from Akka, it couldn't find the New Relic configuration and wasn't reporting correctly for that reason.

Thanks for the awesome work and support guys! Looking forward for your next iterations :)

from kamon.

Thanks to you for sharing, testing and reporting!

from kamon.