Comments (5)
Assuming it's all strings and only one level deep (that's how JSON log lines are usually structured), logfmt is 1:1 with JSON log lines, so it's only a matter of output.
It is not uncommon for JSON log lines to have multiple nested levels when using observability tools, so if logfmt is supported, I suggest making JSON the primary focus and gracefully degrading to logfmt.
Semantic Logger has a nice approach of allowing the user to choose among a list of formatters for each appender that receives the logs. In fact, if you take a look at their documentation for appenders (a concept that I think would be fantastic to have as part of this work), you'll find an example payload that shows a nested payload structure. 😁
from rails.
Huge +1 to having something out-of-the-box for this; we've managed to grow several different implementations of structured logging at Zendesk and I would very much like there to be a clear gold standard here to switch to!
One thing I would like to see addressed in a built-in structured logging solution is what to do about multiple independent processes writing to stdout.
It's very common, in my experience, for apps in container environments to be expected to write their logs to stdout, where it gets collected by the container engine and then forwarded to a centralised logging solution. It's also very common to run Rails apps in a multiprocess app server like Unicorn or Puma's cluster mode, so there might be several independent Rails processes running in the one container. The problem is, by default, all of those processes will share the single stdout stream, leading to issues where structured log messages might be interleaved and thus their structure broken.
I've been sketching out a way to solve this for us at Zendesk (https://github.com/zendesk/multiproc_log_device). Essentially, it runs a parent process above Unicorn which exposes a unix socket, and reopens stdout in each process to connect to that socket. Then, the parent process can merge the logs together in a structure aware way.
I would be very interested in hearing how other Rails users have dealt with this issue, and very very interested in trying to hammer out a way this problem can be addressed in a built-in Rails structured logging solution.
from rails.
API would be something like this:
logger.struct my_message: "Something happened"
But this isn't set and I'm not planning to work in the first half of the year.
from rails.
we should add structured json logging on a per-request basis
Good initiative!
Maybe logfmt
(https://brandur.org/logfmt) could be supported as an output alternative to JSON?
Assuming it's all strings and only one level deep (that's how JSON log lines are usually structured), logfmt is 1:1 with JSON log lines, so it's only a matter of output.
Also, maybe this is an area where an RFC light might be useful.
Just write down a rough plan on the interface and output structure, and ask a few large Rails users (Shopify, Zendesk, Github, etc) to provide feedback.
from rails.
Is there any indication what the rough shape of the structured logging interfaces/APIs might look like? Or, what's the best way to provide input on this?
We’ve cooked up a bunch of home-grown interfaces for this in Zendesk:
- making
Rails.logger.tagged(attr: 'value') { ... }
accept a hash of structured attributes - passing attributes to
Rails.logger.info "message", extra: 'attributes'
- entirely separate methods like
Rails.logger.append_attributes({attribute_for: 'remaining lifetime of this request'})
(I think the semantic logger gem mentioned above supports the first two methods, but we're not actually using that implementation).
I’m presently trying to consolidate all of these various interfaces in our code. I’d ideally want to “just do what rails does”, but of course rails doesn’t do anything related to structured logging… yet. If I had a rough idea what direction Rails is planning to go in, it would greatly help me out with deciding which interfaces to keep and which to refactor 🙏
from rails.
Related Issues (20)
- Queries generated incorrectly in class methods HOT 2
- Order of operations for saving nested associations has changed in Rails v7.2 with automatically_invert_plural_associations enabled HOT 7
- rescue_from not working with subscribed method on ActionCable channel HOT 3
- Production rails server does not boot if skipping action cable in Rails 8.0.0.alpha HOT 2
- Generate Migration - Rename Table HOT 6
- Rails 7.1 Transactional tests, threaded code and schema caching can cause deadlocks HOT 1
- /Users/amol/.rvm/gems/ruby-2.5.1/gems/activerecord-5.0.2/lib/active_record/connection_adapters/mysql2_adapter.rb:68: [BUG] Segmentation fault at 0x94000f78d65f0300 ruby 2.5.1p57 (2018-03-29 revision 63029) [x86_64-darwin23] HOT 1
- Support enum in tableless model HOT 4
- `ActiveStorage::Attachment` - Unhandled case where `named_variants` can be `nil`
- `ActiveSupport::BroadcastLogger` with `ActiveSupport::TaggedLogging` throws `no implicit conversion of String into Integer (TypeError)` HOT 3
- Guides: Explain reset_column_information in Chapter "Migrations and Seed Data"
- One of `ApplicationTests::ConfigurationTest` fails with sqlite3 gem that bundles sqlite v3.46.0 HOT 3
- Add a datetime_local method on Time and DateTime classes - typical rails convenience HOT 1
- Deferred route drawing causes problems with some tests
- Incorrect documentation for authenticate_or_request_with_http_token HOT 1
- Regression in URL generation in Rails 7.1 HOT 13
- ActionCable tests fail when encrypted cookie is set with options
- Turbo depends on ActiveJob::Base but no error when I start a project with `rails new . --skip-active-job` HOT 1
- ActionPack 7.1.3.3 Selenium DriverFinder.path(options, service_class) deprecation is NOT fixed HOT 2
- warning: the block passed to 'Arel::Collectors::Bind#add_bind' defined at /rails/activerecord/lib/arel/collectors/bind.rb:16 may be ignored
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from rails.