Comments (11)
Hi @pjvds thanks a lot for your input!
This is the actually the first time I've heard of people who've been running into serious performance issues with the logger in production. Performance hasn't been a hardcore goal (e.g. in some cases we've favoured nice API and usability over performance, although obviously harvesting any performance benefits we could) because the time spent in logrus in the applications I have has been negligible, likely because I log much less than you do. However, if you're seeing problems, let's discuss how we can best solve them.
How much are you logging?
Can you track down more specifically which codepath is slow (you're mentioning the low-level calls which is great)? It might be a hook or a specific part of the formatter.
I have a hunch it has to do with the formatter, e.g. it's doing a sort by default which is notoriously slow (but hasn't been a big deal in my cases).
from logrus.
@pjvds do you have a branch with your current changes? I'm just curious :)
from logrus.
I would be interested in seeing the proposed differences to the surface API this might entail. I love how easy to use and extend logrus messages are, but with an app that has very tight performance constraints, I would additionally love to know that I'm not losing time in my logging framework of choice.
from logrus.
As a consumer I am willing to refactor code to gain a performance increase assuming the numbers you are quoting are realistic. However logrus is imported by 702 packages. Breaking changes are pretty rough, any way to offer a new version or different API for the faster stuff?
from logrus.
I as well have seem performance issues w/ logrus related to the garbage creation. I solved this by removing the WithField() calls for debug logs by using the following function.
// Debug to logger a key/value pair and message. Intended to save the mem alloc that WithField creates
func Debug(l *logrus.Logger, key string, val interface{}, msg string) {
if l.Level >= logrus.DebugLevel {
l.WithField(key, val).Debug(msg)
}
}
from logrus.
@cep21 would being able to pass a function that's lazily evaluated to WithField{,s}
work for you? Or is it simply because you're doing so many map
allocations?
from logrus.
For me, on this issue, it was the large number of allocs and the GC pressure I was trying to reduce. l.WithField(key, val).Debug(msg) requires a few allocs even when I'm not at debug level.
The only I can see to get around this is to reverse the order.
l.Debug(msg).WithField(key, val).Fire()
With this order of operations, l.Debug(msg) could return a no-op singleton and make any future calls no-op non memory allocs. With the order logrus currently has, l.WithField(key, val) requires an alloc because it doesn't know that Debug is coming next.
from logrus.
Good to find more people discussing this!
@cep21 Actually I am submitting a PR to fix the first issue you talked about.
Regarding the logger.WithField / WithFields problem, I am also proposing a similar approach to yours.
Please check #154
I am basically proposing either
- inverted logic: log.Info(...).WithFields(...)
- like your Fire but trying to be semver compatible:
log.WithLevel(Info).WithFields(...).Log(...)
from logrus.
Rel: PERF: use buffer pool in json formatter #755
from logrus.
Hi, WithFields over-allocates memory even when it doesn't have to. In this particular example (see attached) the log level is set to 'warn', but a call to Trace allocated ~32Mb. Has anybody found a reliable workaround for this issue? I haven't profiled the cep21 solution above. Does it work? Thanks!
from logrus.
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
from logrus.
Related Issues (20)
- Info requested: How to buffer messages and release at the end HOT 2
- Vulnerability of dependency "gopkg.in/yaml.v3" HOT 2
- Logrus potential DOS - Please open a CVE for issue #1370 to encourage users to patch HOT 3
- Feature: Add slog compatibility layer HOT 5
- logrus+lumberjackrus, After the old log is deleted, the hard drive will not be released, HOT 2
- Vulnerability: gopkg.in/yaml.v3 CVE-2022-28948 HOT 2
- Bazel compilation failling HOT 2
- Question: Any way to view the value of a field from a logrus.FieldLogger? HOT 2
- How to apply colours for a custom log format? HOT 2
- How to safely change Logger.Out on the fly? HOT 2
- How should I position my problem? HOT 2
- Cut a new release with wasip1 support? HOT 4
- github.com/interactive-solutions/go-logrus-elasticsearch cannot find HOT 2
- Bump version of stretchr/testify? HOT 3
- Unclear on documentation regarding Hook interface HOT 3
- Please maintain the CHANGELOG HOT 3
- logger.Out.Write's mutex may be unnecessary, when Out is a file fd HOT 2
- Use for range to output two identical logs
- Update stretchr/testify to 1.9.0 to resolve 2 High sev Vulnerabilities found in go yaml lib from current version HOT 4
- Please reopen and fix https://github.com/sirupsen/logrus/issues/972, skipping caller frames HOT 2
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from logrus.