Git Product home page Git Product logo

semantic_logger's Introduction

Semantic Logger

Gem Version Build Status Downloads License

Semantic Logger is a feature rich logging framework, and replacement for existing Ruby & Rails loggers.

Documentation

Semantic Logger Guide

Logging Destinations

Logging to the following destinations are all supported "out-of-the-box":

  • File
  • Screen
  • ElasticSearch. (Use with Kibana for Dashboards and Visualizations)
  • Graylog
  • BugSnag
  • NewRelic
  • Splunk
  • MongoDB
  • Honeybadger
  • Sentry (both with legacy sentry-raven and modern sentry-ruby gem)
  • HTTP
  • TCP
  • UDP
  • Syslog
  • Add any existing Ruby logger as another destination.
  • Roll-your-own

Semantic Logger is capable of logging thousands of lines per second without slowing down the application. Traditional logging systems make the application wait while the log information is being saved. Semantic Logger avoids this slowdown by pushing log events to an in-memory queue that is serviced by a separate thread that only handles saving log information to multiple destinations / appenders.

Rails

When running Rails, use rails_semantic_logger instead of Semantic Logger directly since it will automatically replace the Rails default logger with Semantic Logger.

Rocket Job

Checkout the sister project Rocket Job: Ruby's missing batch system.

Fully supports Semantic Logger when running jobs in the background. Complete support for job metrics sent via Semantic Logger to your favorite dashboards.

Optional Dependencies

The following gems are only required when their corresponding appenders are being used, and are therefore not automatically included by this gem:

  • Bugsnag Appender: gem 'bugsnag'
  • MongoDB Appender: gem 'mongo' 1.9.2 or above
  • NewRelic Appender: gem 'newrelic_rpm'
  • NewRelicLogs Appender: gem 'newrelic_rpm'
  • Syslog Appender: gem 'syslog_protocol' 0.9.2 or above
  • Syslog Appender to a remote syslogng server over TCP or UDP: gem 'net_tcp_client'
  • Splunk Appender: gem 'splunk-sdk-ruby'
  • Elasticsearch Appender: gem 'elasticsearch'
  • Kafka Appender: gem 'ruby-kafka'
  • Legacy Sentry Appender: gem 'sentry-raven' (deprecated)
  • Sentry Appender: gem 'sentry-ruby'

Upgrading to Semantic Logger v4.9

These changes should not be noticeable by the majority of users of Semantic Logger, since they are to the internal API. It is possible that advanced users may be using these internal API's directly.

This does not affect any calls to the public api SemanticLogger.add_appender.

File and IO are now separate appenders. When creating the File appender explicitly, its arguments have changed. For example, when requesting an IO stream, it needs to be changed from:

SemanticLogger::Appender::File.new(io: $stderr)

to:

SemanticLogger::Appender::IO.new($stderr)

Additionally, this needs to be changed from:

SemanticLogger::Appender::File.new(file_name: "file.log")

to:

SemanticLogger::Appender::File.new("file.log")

Rails Semantic Logger, if used, needs to be upgraded to v4.9 when upgrading to Semantic Logger v4.9.

Upgrading to Semantic Logger v4.4

With some forking frameworks it is necessary to call reopen after the fork. With v4.4 the workaround for Ruby 2.5 crashes is no longer needed. I.e. Please remove the following line if being called anywhere:

SemanticLogger::Processor.instance.instance_variable_set(:@queue, Queue.new)

Upgrading to Semantic Logger v4.0

The following changes need to be made when upgrading to V4:

  • Ruby V2.3 / JRuby V9.1 is now the minimum runtime version.
  • Replace calls to Logger#with_payload with SemanticLogger.named_tagged.
  • Replace calls to Logger#payload with SemanticLogger.named_tags.
  • MongoDB Appender requires Mongo Ruby Client V2 or greater.
  • Appenders now write payload data in a seperate :payload tag instead of mixing them directly into the root elements to avoid name clashes.

As a result any calls like the following:

logger.debug foo: 'foo', bar: 'bar'

Must be replaced with the following in v4:

logger.debug payload: {foo: 'foo', bar: 'bar'}

Similarly, for measure blocks:

logger.measure_info('How long is the sleep', foo: 'foo', bar: 'bar') { sleep 1 } 

Must be replaced with the following in v4:

logger.measure_info('How long is the sleep', payload: {foo: 'foo', bar: 'bar'}) { sleep 1 } 

The common log call has not changed, and the payload is still logged directly:

logger.debug('log this', foo: 'foo', bar: 'bar')

Install

gem install semantic_logger

To configure a stand-alone application for Semantic Logger:

require 'semantic_logger'

# Set the global default log level
SemanticLogger.default_level = :trace

# Log to a file, and use the colorized formatter
SemanticLogger.add_appender(file_name: 'development.log', formatter: :color)

If running rails, see: Semantic Logger Rails

Author

Reid Morrison

Contributors

Versioning

This project uses Semantic Versioning.

semantic_logger's People

Contributors

amanfredi avatar anotherregulardude avatar arielvalentin avatar bduc avatar bedol avatar chriscz avatar havenwood avatar johnathanludwig avatar joker-777 avatar kamil-gwozdz avatar katafrakt avatar keithrbennett avatar lambcr avatar lonre avatar marc avatar olleolleolle avatar petergoldstein avatar picandocodigo avatar pnomolos avatar qstearns avatar refractalize avatar reidmorrison avatar rewritten avatar seanthingee avatar silex avatar syedsanahassan avatar thg303 avatar treacher avatar xrl avatar znz avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

semantic_logger's Issues

log.to_h and log.to_json yield different results (3.2.1)

pry(#<SemanticLogger::Log>):1> to_h
=> {:host=>"tims-mac.local",
 :application=>"Semantic Logger",
 :name=>"Rails",
 :pid=>16094,
 :thread=>"70245767569880",
 :time=>2016-04-11 14:13:50 +1200,
 :level=>:warn,
 :level_index=>3,
 :file=>"(irb)",
 :line=>3,
 :message=>"test"}
[37] pry(#<SemanticLogger::Log>):1> to_json
=> "{\"level\":\"warn\",\"thread_name\":\"70245767569880\",\"name\":\"Rails\",\"message\":\"test\",\"payload\":null,\"time\":\"2016-04-11T14:13:50.529+12:00\",\"duration\":null,\"tags\":[],\"level_index\":3,\"exception\":null,\"metric\":null,\"backtrace\":[\"(irb):3:in `irb_binding'\",\"/Users/tim/.rubies/ruby-2.2.3/lib/ruby/2.2.0/irb/workspace.rb:86:in `eval'\",\"/Users/tim/.rubies/ruby-2.2.3/lib/ruby/2.2.0/irb/workspace.rb:86:in `evaluate'\",\"/Users/tim/.rubies/ruby-2.2.3/lib/ruby/2.2.0/irb/context.rb:379:in `evaluate'\",\"/Users/tim/.rubies/ruby-2.2.3/lib/ruby/2.2.0/irb.rb:489:in `block (2 levels) in eval_input'\",\"/Users/tim/.rubies/ruby-2.2.3/lib/ruby/2.2.0/irb.rb:623:in `signal_status'\",\"/Users/tim/.rubies/ruby-2.2.3/lib/ruby/2.2.0/irb.rb:486:in `block in eval_input'\",\"/Users/tim/.rubies/ruby-2.2.3/lib/ruby/2.2.0/irb/ruby-lex.rb:245:in `block (2 levels) in each_top_level_statement'\",\"/Users/tim/.rubies/ruby-2.2.3/lib/ruby/2.2.0/irb/ruby-lex.rb:231:in `loop'\",\"/Users/tim/.rubies/ruby-2.2.3/lib/ruby/2.2.0/irb/ruby-lex.rb:231:in `block in each_top_level_statement'\",\"/Users/tim/.rubies/ruby-2.2.3/lib/ruby/2.2.0/irb/ruby-lex.rb:230:in `catch'\",\"/Users/tim/.rubies/ruby-2.2.3/lib/ruby/2.2.0/irb/ruby-lex.rb:230:in `each_top_level_statement'\",\"/Users/tim/.rubies/ruby-2.2.3/lib/ruby/2.2.0/irb.rb:485:in `eval_input'\",\"/Users/tim/.rubies/ruby-2.2.3/lib/ruby/2.2.0/irb.rb:395:in `block in start'\",\"/Users/tim/.rubies/ruby-2.2.3/lib/ruby/2.2.0/irb.rb:394:in `catch'\",\"/Users/tim/.rubies/ruby-2.2.3/lib/ruby/2.2.0/irb.rb:394:in `start'\",\"/Users/tim/.gem/ruby/2.2.3/gems/railties-4.2.5/lib/rails/commands/console.rb:110:in `start'\",\"/Users/tim/.gem/ruby/2.2.3/gems/railties-4.2.5/lib/rails/commands/console.rb:9:in `start'\",\"/Users/tim/.gem/ruby/2.2.3/gems/railties-4.2.5/lib/rails/commands/commands_tasks.rb:68:in `console'\",\"/Users/tim/.gem/ruby/2.2.3/gems/railties-4.2.5/lib/rails/commands/commands_tasks.rb:39:in `run_command!'\",\"/Users/tim/.gem/ruby/2.2.3/gems/railties-4.2.5/lib/rails/commands.rb:17:in `\\u003ctop (required)\\u003e'\",\"bin/rails:4:in `require'\",\"bin/rails:4:in `\\u003cmain\\u003e'\"],\"metric_amount\":null}"

Use signals to increase or decrease log level

Use signals to increase or decrease the current log level. Useful in production to increase the log level for problem determination purposes.
This would only affect logger instances that do not have an explicit log level set and are using the global default_level.
The signals must be user configurable. By default signal use should be off so as not to interfere with existing gems that may be using signals.
Consider creating and supporting a config file. rails_semantic_logger gem should auto-detect and load this file.

Rack::CommonLogger

Im working on a small sinatra application I wanted to switch out the Rack::CommonLogger with semantic logger so sinatra will log with semantic_logger. This does not seem possible, and I think its just semantic_logger missing some things to make it compatible but wanted to see if you could confirm

Example:
Required gems: sinatra, sinatra-contrib, semantic_logger

In this example the log statement in the main end point logs just fine. However, nothing from Rack::CommonLogger is making it into the log.

require 'sinatra'
require 'sinatra/custom_logger'
require 'semantic_logger'

configure do
  register Sinatra::CustomLogger
  SemanticLogger.default_level = :info
  SemanticLogger.add_appender("#{settings.root}/log/#{settings.environment}.log", &SemanticLogger::Appender::Base.colorized_formatter)
  logger = SemanticLogger['TestLogger']
  set :raise_errors, true
  set :logger, logger
  use Rack::CommonLogger, logger
end

get '/' do
  # This makes it into the log file. But nothing from Rack::CommonLogger
  logger.warn("Test")
  'Logger testing'
end

In this example the log statement and the Rack::CommonLogger logs are making it into the log file which leads me to believe that semantic_logger is not compatible with Rack::CommonLogger

require 'sinatra'
require 'sinatra/custom_logger'
require 'logger'

configure do
  register Sinatra::CustomLogger
  logger = Logger.new("#{settings.root}/log/#{settings.environment}.log")
  set :raise_errors, true
  set :logger, logger
  use Rack::CommonLogger, logger
end

get '/' do
  # This makes it into the log file and also from Rack::CommonLogger
  logger.warn("Test")
  'Logger testing'
end

The only difference between the two scenarios is one uses the logger class from ruby which is compatible with Rack::CommonLogger and the other is using SemanticLogger which does not seem compatible with Rack::CommonLogger.

Thanks!

logging into different files

Sorry if this is a silly question, but I might have missed something very obvious...

My Sidekiq is currently set to log to a separate log file, set inside the config.yml. When SemanticLogger replaces the logger, then all log output goes to the same file as the rails log, and it doesn't use this config directive any more.

Even if it ignores the config, is there a way to tell SemanticLogger to log sidekiq log output into one file, and other rails log output into another? I tried to probe SemanticLogger[Sidekiq] but can't work out a way to assign it with a different appender (which I thought would be the right thing to do?)

What am I missing? :)

Allow Syslog appender to be reopened with a new identifier.

Currently SemanticLogger has the method for reopen which will run reopen on the appenders that respond to it. However, Syslog allows you to pass in options to reopen which allow one to change the identifier, options, and facility.

It'd be a good feature to pass through these options down to Syslog reopen.

JSON log format

Is it possible to have it output purely JSON without and prefix strings?

Disambiguating logger from mixin from Rails / config logger - New feature: Appender specific class filtering

Hello,

It is tempting to use the Loggable module in e.g. Rails controllers, but there may be confusion between the logger method and the logger provided by Rails / config.

Could you provide an example of defining a custom logger for a particular controller (with custom appender/s) in a singleton fashion and so that it does not collide with other loggers possibly used within the same controller?

Thank you

Incompatible with honeybadger?

Adding semantic logger to an existing project yields the following result:

/Users/mhenrixon/.rvm/gems/ruby-2.3.1/gems/honeybadger-2.7.2/lib/honeybadger/logging.rb:29:in `<': comparison of Fixnum with :debug failed (ArgumentError)
  from /Users/mhenrixon/.rvm/gems/ruby-2.3.1/gems/honeybadger-2.7.2/lib/honeybadger/logging.rb:29:in `warn'
  from /Users/mhenrixon/.rvm/gems/ruby-2.3.1/gems/honeybadger-2.7.2/lib/honeybadger/agent.rb:124:in `initialize'
  from /Users/mhenrixon/.rvm/gems/ruby-2.3.1/gems/honeybadger-2.7.2/lib/honeybadger/agent.rb:69:in `new'
  from /Users/mhenrixon/.rvm/gems/ruby-2.3.1/gems/honeybadger-2.7.2/lib/honeybadger/agent.rb:69:in `start'
  from /Users/mhenrixon/.rvm/gems/ruby-2.3.1/gems/honeybadger-2.7.2/lib/honeybadger.rb:30:in `start'
  from /Users/mhenrixon/.rvm/gems/ruby-2.3.1/gems/honeybadger-2.7.2/lib/honeybadger/init/rails.rb:20:in `block in <class:Railtie>'
  from /Users/mhenrixon/.rvm/gems/ruby-2.3.1/gems/railties-4.2.7.1/lib/rails/initializable.rb:30:in `instance_exec'
  from /Users/mhenrixon/.rvm/gems/ruby-2.3.1/gems/railties-4.2.7.1/lib/rails/initializable.rb:30:in `run'
  from /Users/mhenrixon/.rvm/gems/ruby-2.3.1/gems/railties-4.2.7.1/lib/rails/initializable.rb:55:in `block in run_initializers'
  from /Users/mhenrixon/.rvm/rubies/ruby-2.3.1/lib/ruby/2.3.0/tsort.rb:228:in `block in tsort_each'
  from /Users/mhenrixon/.rvm/rubies/ruby-2.3.1/lib/ruby/2.3.0/tsort.rb:350:in `block (2 levels) in each_strongly_connected_component'
  from /Users/mhenrixon/.rvm/rubies/ruby-2.3.1/lib/ruby/2.3.0/tsort.rb:431:in `each_strongly_connected_component_from'
  from /Users/mhenrixon/.rvm/rubies/ruby-2.3.1/lib/ruby/2.3.0/tsort.rb:349:in `block in each_strongly_connected_component'
  from /Users/mhenrixon/.rvm/rubies/ruby-2.3.1/lib/ruby/2.3.0/tsort.rb:347:in `each'
  from /Users/mhenrixon/.rvm/rubies/ruby-2.3.1/lib/ruby/2.3.0/tsort.rb:347:in `call'
  from /Users/mhenrixon/.rvm/rubies/ruby-2.3.1/lib/ruby/2.3.0/tsort.rb:347:in `each_strongly_connected_component'
  from /Users/mhenrixon/.rvm/rubies/ruby-2.3.1/lib/ruby/2.3.0/tsort.rb:226:in `tsort_each'
  from /Users/mhenrixon/.rvm/rubies/ruby-2.3.1/lib/ruby/2.3.0/tsort.rb:205:in `tsort_each'
  from /Users/mhenrixon/.rvm/gems/ruby-2.3.1/gems/railties-4.2.7.1/lib/rails/initializable.rb:54:in `run_initializers'
  from /Users/mhenrixon/.rvm/gems/ruby-2.3.1/gems/railties-4.2.7.1/lib/rails/application.rb:352:in `initialize!'
  from /Users/mhenrixon/.rvm/gems/ruby-2.3.1/gems/railties-4.2.7.1/lib/rails/railtie.rb:194:in `public_send'
  from /Users/mhenrixon/.rvm/gems/ruby-2.3.1/gems/railties-4.2.7.1/lib/rails/railtie.rb:194:in `method_missing'
  from /Users/mhenrixon/Code/project_name/config/environment.rb:5:in `<top (required)>'
  from /Users/mhenrixon/.rvm/gems/ruby-2.3.1/gems/activesupport-4.2.7.1/lib/active_support/dependencies.rb:274:in `require'
  from /Users/mhenrixon/.rvm/gems/ruby-2.3.1/gems/activesupport-4.2.7.1/lib/active_support/dependencies.rb:274:in `block in require'
  from /Users/mhenrixon/.rvm/gems/ruby-2.3.1/gems/activesupport-4.2.7.1/lib/active_support/dependencies.rb:240:in `load_dependency'
  from /Users/mhenrixon/.rvm/gems/ruby-2.3.1/gems/activesupport-4.2.7.1/lib/active_support/dependencies.rb:274:in `require'
  from /Users/mhenrixon/Code/project_name/config.ru:3:in `block in <main>'
  from /Users/mhenrixon/.rvm/gems/ruby-2.3.1/gems/rack-1.6.5/lib/rack/builder.rb:55:in `instance_eval'
  from /Users/mhenrixon/.rvm/gems/ruby-2.3.1/gems/rack-1.6.5/lib/rack/builder.rb:55:in `initialize'
  from /Users/mhenrixon/Code/project_name/config.ru:in `new'
  from /Users/mhenrixon/Code/project_name/config.ru:in `<main>'
  from /Users/mhenrixon/.rvm/gems/ruby-2.3.1/gems/rack-1.6.5/lib/rack/builder.rb:49:in `eval'
  from /Users/mhenrixon/.rvm/gems/ruby-2.3.1/gems/rack-1.6.5/lib/rack/builder.rb:49:in `new_from_string'
  from /Users/mhenrixon/.rvm/gems/ruby-2.3.1/gems/rack-1.6.5/lib/rack/builder.rb:40:in `parse_file'
  from /Users/mhenrixon/.rvm/gems/ruby-2.3.1/gems/rack-1.6.5/lib/rack/server.rb:299:in `build_app_and_options_from_config'
  from /Users/mhenrixon/.rvm/gems/ruby-2.3.1/gems/rack-1.6.5/lib/rack/server.rb:208:in `app'
  from /Users/mhenrixon/.rvm/gems/ruby-2.3.1/gems/railties-4.2.7.1/lib/rails/commands/server.rb:61:in `app'
  from /Users/mhenrixon/.rvm/gems/ruby-2.3.1/gems/rack-1.6.5/lib/rack/server.rb:336:in `wrapped_app'
  from /Users/mhenrixon/.rvm/gems/ruby-2.3.1/gems/rack-1.6.5/lib/rack/server.rb:272:in `start'
  from /Users/mhenrixon/.rvm/gems/ruby-2.3.1/gems/railties-4.2.7.1/lib/rails/commands/server.rb:80:in `start'
  from /Users/mhenrixon/.rvm/gems/ruby-2.3.1/gems/railties-4.2.7.1/lib/rails/commands/commands_tasks.rb:80:in `block in server'
  from /Users/mhenrixon/.rvm/gems/ruby-2.3.1/gems/railties-4.2.7.1/lib/rails/commands/commands_tasks.rb:75:in `tap'
  from /Users/mhenrixon/.rvm/gems/ruby-2.3.1/gems/railties-4.2.7.1/lib/rails/commands/commands_tasks.rb:75:in `server'
  from /Users/mhenrixon/.rvm/gems/ruby-2.3.1/gems/railties-4.2.7.1/lib/rails/commands/commands_tasks.rb:39:in `run_command!'
  from /Users/mhenrixon/.rvm/gems/ruby-2.3.1/gems/railties-4.2.7.1/lib/rails/commands.rb:17:in `<top (required)>'
  from bin/rails:4:in `require'
  from bin/rails:4:in `<main>'

Anyone have any ideas why that is happening?

some bug about SemanticLogger::Loggable

Hi, I'm using SemanticLogger now in my project, I use SemanticLogger::Loggable to write log in each class, but I found that, if I write log in method B of module A, as I include SemanticLogger::Loggable, the log run success, but when another class C include module A, and call method B, there is error that the logger method not in class C.
I think the reason is module B include SemanticLogger::Loggable, so module B has extend a method of logger, but if class C include module B, class C doesn't have method logger because C didn't include SemanticLogger::Loggable.
I saw the source of Loggable, I think it's hard to slove this problem.

overriding level_to_s

I have a quick question about the "official" way to get the log level to show fully (e.g. ERROR instead of E). I can see that this is done in level_to_s method, and used in the default formatter.

Would it be best to monkey-path this method? (how?) or to define a custom formatter?

Intuitively I would use a custom formatter, but feels like a lot of copy&paste for changing a small fraction of the default format.

Or maybe there's a different option altogether?

missing logs when running in multiple docker

configuring stdout appender and running 2 dockers, one for main rails other for sidekiq, only rails logs are being logged, sidekiq logs are lost.
It works as expected when running both proccesses in local dev environment

thread_name wrong when not using java

In Logger#thread_name, your returning Thread.object_id when I think you want to return Thread.current.object_id otherwise your just returning the object_id of the Thread class.

Doesn't log anything on production

For some reason it doesn't log anything on our production environment but it works fine on development environment. These are the lines that I added on both /config/environment/production.rb and /config/environment/development.rb:

  config.rails_semantic_logger.add_file_appender = false # disable logging to a file
  config.after_initialize do
    config.semantic_logger.add_appender(appender: :syslog, formatter: :json)
  end

It logs the active record queries in our rake tasks though. Any ideas on what I'm doing wrong?

Rails 4.2.6
ruby 2.2.5p319 (2016-04-26 revision 54774) [x86_64-linux]
rails_semantic_logger (3.4.1)
semantic_logger (~> 3.3)

Graylog appender - ArgumentError: short_message is missing

I have a code (RAILS_ENV=production)

Rails.logger.error StandardError.new("hello world")

And result exception:

Rails -- Exception: StandardError: hello world
SemanticLogger -- Appender thread: Failed to log to appender: <DELETED> -- Exception: ArgumentError: short_message is missing. Options version, short_message and host must be set.

Thread dump does not dump current thread

When a program adds a signal handler using SemanticLogger.add_signal_handler and the signal handler is executed (e.g. by sending SIGTTIN), semantic_logger prints a thread dump.

Expected result

The thread dump covers all threads.

Actual result

The thread dump covers all threads except the current thread. This may mean that the thread dump covers no thread at all (except threads not spawned by the program, but by semantic_logger itself).

Analysis

The code line https://github.com/rocketjob/semantic_logger/blob/5bff1c5dae0a3f965fa2c0edb2fe50f5dfbe8060/lib/semantic_logger/semantic_logger.rb#L269 seems to be the cause of this. It should probably be replaced or removed.

Impact

This bug makes the thread dump feature less useful, as one typical application is to determine in which part of the code the current process spends time in. For a single-threaded program, the dump of the single thread is not printed and thus contains no useful information.

Support log format specifications directly from the configuration, without having to create a new formatter.

Currently using Log4R - which allows a Pattern Based Formatter - eg

class PatternFormatter < BasicFormatter

# Arguments to sprintf keyed to directive letters<br>
# %c - event short name<br>
# %C - event fullname<br>
# %d - date<br>
# %g - Global Diagnostic Context (GDC)<br>
# %t - trace<br>
# %m - message<br>
# %h - thread name<br>
# %p - process ID aka PID<br>
# %M - formatted message<br>
# %l - Level in string form<br>
# %x - Nested Diagnostic Context (NDC)<br>
# %X - Mapped Diagnostic Context (MDC), syntax is "%X{key}"<br>
# %% - Insert a %<br>
DirectiveTable = {
  "c" => 'event.name',
  "C" => 'event.fullname',
  "d" => 'format_date',
  "g" => 'Log4r::GDC.get()',
  "t" => '(event.tracer.nil? ? "no trace" : event.tracer[0])',
  "T" => '(event.tracer.nil? ? "no trace" : event.tracer[0].split(File::SEPARATOR)[-1])',
  "m" => 'event.data',
  "h" => '(Thread.current[:name] or Thread.current.to_s)',
  "p" => 'Process.pid.to_s',
  "M" => 'format_object(event.data)',
  "l" => 'LNAMES[event.level]',
  "x" => 'Log4r::NDC.get()',
  "X" => 'Log4r::MDC.get("DTR_REPLACE")',
  "%" => '"%"'
}

This is useful to me because I log to stdout (a simple message format for users) AND I log a timestamped message to a file as well (that may be at a different log level)

Is there a way to get this with Semantic Logger ? My first thought is to write a new Appender, but, maybe other appenders could use the 'custom formatter' as well

Cheers & Thanks

Make tags accessible from other threads ?

Hi, Is there any plan to make tags accessible from other threads?

The case that tag information is related with current thread(not with a logger instance) can hardly make sense.

If you guys have the same thought, I can make a PR to solve this issue.

Re-open text log files after every 1000 messages

If someone deletes or moves a log file, all file logging will stop. Although copy-truncate is the correct way to clear out log files, it would be useful if the log file was closed and then re-opened after approximately every 1000 lines.

Even though some log entries would be lost, it is a good compromise between performance and reliability in this circumstance.

Sidekiq logging is missing context information

Following the readme I`ve added semantic logger to Sidekiq.

Sidekiq::Logging.logger = SemanticLogger[Sidekiq] if defined?(Sidekiq)

And it occurred that i`ve lost the most valuable data - worker class name.

The log before:

2017-02-12T19:18:43.908Z 22548 TID-osbolpr4g SomeWorker JID- INFO: start 2017-02-12T19:18:43.912Z 22548 TID-osbolpr4g SomeWorker JID- INFO: done: 0.003 sec

The log after:

{"name":"Sidekiq","pid":19857,"thread":"47328718486600","level":"info","level_index":2,"host":"web01","application":"Semantic Logger","message":"start","timestamp":"2017-02-12T19:32:00.599277Z"} {"name":"Sidekiq","pid":19857,"thread":"47328718486600","level":"info","level_index":2,"host":"web01","application":"Semantic Logger","message":"done: 0.002 sec","timestamp":"2017-02-12T19:32:00.601137Z"}

Is there a known way to capture the name of the calling worker class. Or maybe we should provider some facilities for such a broad use-case?

Problem rails 5.1 and ruby 2.4.1

E [18506:puma 005 log_subscriber.rb:85] ActiveRecord::Base -- Could not log "sql.active_record" event. ArgumentError: wrong number of arguments (given 1, expected 2)

Better formatting options for ActiveRecord::Base?

In semantic_logger v1.8 I had logging output that looked like this (with a custom formatter):
screen shot 2016-08-01 at 5 20 23 pm

I just upgraded to v3.3 and the same formatter (adapted to new AwesomePrint format) looks like this:
screen shot 2016-08-01 at 10 12 38 pm

Unfortunately, the 'payload' portion of the log feels like a step backwards. Any way to get the old log formatting back?

Sentry Appender: NoMethodError: undefined method `error_class='

I noticed this error in our logs recently:

2017-05-08 11:12:39.674917 E [25083:SemanticLogger::Processor] SemanticLogger -- Appender thread: Failed to log to appender: #<SemanticLogger::Appender::Sentry:0x00000002aa6b10 @formatter=#<SemanticLogger::Formatters::Raw:0x00000002aa3050 @precision=6, @time_format="%Y-%m-%d %H:%M:%S.%6N", @log_host=true, @log_application=true>, @application=nil, @host=nil, @filter=nil, @name="SemanticLogger::Appender::Sentry", @level_index=4, @level=:error> -- Exception: NoMethodError: undefined method `error_class=' for #<Raven::Event:0x007f3fb2048b40>
/var/local/kenhub/kenhub.d/28/vendor/bundle/ruby/2.3.0/gems/sentry-raven-2.4.0/lib/raven/event.rb:67:in `public_send'
/var/local/kenhub/kenhub.d/28/vendor/bundle/ruby/2.3.0/gems/sentry-raven-2.4.0/lib/raven/event.rb:67:in `block in initialize'
/var/local/kenhub/kenhub.d/28/vendor/bundle/ruby/2.3.0/gems/sentry-raven-2.4.0/lib/raven/event.rb:67:in `each_pair'
/var/local/kenhub/kenhub.d/28/vendor/bundle/ruby/2.3.0/gems/sentry-raven-2.4.0/lib/raven/event.rb:67:in `initialize'
/var/local/kenhub/kenhub.d/28/vendor/bundle/ruby/2.3.0/gems/sentry-raven-2.4.0/lib/raven/event.rb:122:in `new'
/var/local/kenhub/kenhub.d/28/vendor/bundle/ruby/2.3.0/gems/sentry-raven-2.4.0/lib/raven/event.rb:122:in `from_message'
/var/local/kenhub/kenhub.d/28/vendor/bundle/ruby/2.3.0/gems/sentry-raven-2.4.0/lib/raven/instance.rb:114:in `capture_type'
/var/local/kenhub/kenhub.d/28/vendor/bundle/ruby/2.3.0/gems/semantic_logger-4.0.0/lib/semantic_logger/appender/sentry.rb:60:in `log'
/var/local/kenhub/kenhub.d/28/vendor/bundle/ruby/2.3.0/gems/semantic_logger-4.0.0/lib/semantic_logger/processor.rb:201:in `block in call_appenders'
/var/local/kenhub/kenhub.d/28/vendor/bundle/ruby/2.3.0/gems/semantic_logger-4.0.0/lib/semantic_logger/processor.rb:199:in `each'
/var/local/kenhub/kenhub.d/28/vendor/bundle/ruby/2.3.0/gems/semantic_logger-4.0.0/lib/semantic_logger/processor.rb:199:in `call_appenders'
/var/local/kenhub/kenhub.d/28/vendor/bundle/ruby/2.3.0/gems/semantic_logger-4.0.0/lib/semantic_logger/processor.rb:141:in `process_requests'
/var/local/kenhub/kenhub.d/28/vendor/bundle/ruby/2.3.0/gems/semantic_logger-4.0.0/lib/semantic_logger/processor.rb:7:in `block in start'

(another odd thing is that the formatter of this error is different from our regular formatter, that uses ERROR instead of E for log level, but that's not the main problem obviously).

How to design a new SemanticLogger::Loggable

module Foo
  include SemanticLogger::Loggable

  def foo
    logger.info 'foo in Foo'
  end
end


class Bar
  include Foo

  def bar
    foo
  end
end

Bar.new.bar will throw NoMethodError: undefined method 'logger' for Bar:Class

I have some cases like this, Bar does not care about logger, it just do a method call.

Any suggest about this case? or design a new SemanticLogger::Loggable? Thanks

Log statement silently failing since 2.16 due to AwesomePrint

Please take a look at this line of code here:
https://github.com/rocketjob/semantic_logger/blob/master/lib/semantic_logger/appender/base.rb#L103

It seems that while AwesomePrint is defined, calling its ai method is causing a quiet failure which means the logging statement is not produced.

TBH: I have no idea what this library is, and it's being included by another gem in my project (v1.2.0). I managed to grab the error through testing: uninitialized constant Moped::BSON.

So, not sure what the solution is here. If that special call is wrapped in a rescue block then at least any failures of the library can be caught and the normal inspect can be used as a fallback.

Here's an example of how I'm handling this right now via fallback: https://github.com/UnquietCode/semantic_logger/blob/02f328f5d819e41ecead4a3b864d85b2d170ee06/lib/semantic_logger/appender/base.rb#L104

Bug on

Environment

ruby 2.3.1
Puma, Rails
Rails Semantic Logger Version: last
No additional rails configuration

Bug in the code,

I'm getting next issue:

 #<ArgumentError: wrong number of arguments (given 1, expected 0)>
/usr/local/bundle/gems/semantic_logger-4.1.1/lib/semantic_logger/log.rb:68:in `assign'
/usr/local/bundle/gems/semantic_logger-4.1.1/lib/semantic_logger/log.rb:136:in `assign_positional'

This issue on current master and last released version (didn't check older versions):
https://github.com/rocketjob/semantic_logger/blob/master/lib/semantic_logger/log.rb#L147

def assign_positional(message = nil, payload = nil, exception = nil)
...
elsif message.nil? && result.is_a?(Hash)
          assign(result)   # line:147
...
end

But actually it can't since method assign accept only hash like arguments:

def assign(message: nil,
               payload: nil,
               min_duration: 0.0,
               exception: nil,
               metric: nil,
               metric_amount: nil,
               duration: nil,
               backtrace: nil,
               log_exception: :full,
               on_exception_level: nil)

Possible bug

In passed Hash you should have all keys as Symbol type. Any other will be filtered just as usual argument

Documentation for Logentries

Hello!

Thanks for all your work on Semantic Logger, it's by far the best logging library for Ruby. The TCP logging feature is working great. I was able to get it working with logentries.com by doing the following:

module Example
  module Formatters
    class Logentries < SemanticLogger::Formatters::Json
      def call(log, logger)
        "#{LOG_ENTRIES_TOKEN} #{super(log, logger)}"
      end
    end
  end
end


formatter = Example::Formatters::Logentries.new
SemanticLogger.add_appender(appender: :tcp, server: 'api.logentries.com:20000', ssl: true, formatter: formatter)

which uses the TCP method documented here: https://logentries.com/doc/input-token/

I thought you may want to add this information to the Semantic Logger documentation.

Thanks!

Breaks quiet_assets

I have added semantic_logger to a project that already had the quiet_assets gem and now am getting lots of output like:

2017-01-17 08:52:35.401833 D [20492:puma 005] Rails --
2017-01-17 08:52:35.401917 D [20492:puma 005] Rails --
2017-01-17 08:52:35.402101 D [20492:puma 005] Rails -- Started -- { :method => "GET", :path => "/assets/ckeditor/init.self-2f3e80b764bc18c9362e6f3326e1f32d4a8bdceff67345a3453a93b111626ff3.js?body=1", :ip => "::1" }

Quiet assets is intended to suppress this. Is this a known conflict? Is there any way to configure semantic logger to also suppress this output or could it be made compatible with quiet assets?

read/write fragment logs

We've finally deployed semantic logger in production (it was a backengine task that kept being delayed). ๐Ÿพ

One thing that jumps out is that we see far more entries of this sort:

2017-05-03 11:48:26.122200 INFO [12752:19706120] (0.644ms) Rails -- Write fragment views/middlebar/...
2017-05-03 11:48:26.156359 INFO [12752:19706120] (0.358ms) Rails -- Read fragment views/...

(redacted to avoid noise). Looks like cache fragment read/writes are being logged. How can we silence these? we didn't have these things logged before.

Appender thread: Failed to log to appender

Hello,

I am trying out semantic logger with elasticsearch and kibana. Followed the guides, but I am getting an error saying I can't write to appender due to URI which is wrong, but I am not sure how to fix it.

2017-02-14 10:53:42.266409 E [11094:SemanticLogger::AppenderThread] SemanticLogger -- Appender thread: Failed to log to appender: #<SemanticLogger::Appender::Elasticsearch:0x007fb61dcc10e8 @index="semantic_logger", @type="log", @url="http://localhost:9200/", @ssl_options=nil, @username=nil, @password=nil, @compress=false, @open_timeout=2.0, @read_timeout=1.0, @continue_timeout=1.0, @header={"Accept"=>"application/json", "Content-Type"=>"application/json", "Connection"=>"keep-alive", "Keep-Alive"=>"300"}, @server="localhost", @port=9200, @path="/", @http=#<Net::HTTP localhost:9200 open=true>, @formatter=#<SemanticLogger::Formatters::Json:0x007fb61dcfe1f0 @precision=6, @time_format=:iso_8601, @log_host=true, @log_application=true>, @application=nil, @host=nil, @filter=nil, @name="SemanticLogger::Appender::Elasticsearch", @level_index=nil, @level=nil> -- Exception: URI::InvalidURIError: bad URI(is not URI?): http://localhost:9200semantic_logger-2017.02.14/log
Using semantic_logger 3.4.1
Using rails_semantic_logger 3.4.1

I am using Rails and my semantic_logger.rb inside initializers folder looks like:

SemanticLogger.add_appender(
  appender: :elasticsearch,
  url:      'http://localhost:9200/'
)

Seems like the slash is missing in the URI, but I've added it in the config, but still, didn't help.

Anyone else is having same issue?

Thanks

EDIT: seems like this line loses the slash at the end: https://github.com/rocketjob/semantic_logger/blob/v3.4.1/lib/semantic_logger/appender/elasticsearch.rb#L60

EDIT 2: ok, apparently it has been fixed in the release candidate.

sentry appender potential error

There seems to be a small problem with the sentry appender when it uses capture_exception

See getsentry/sentry-ruby#678

I suspect that the context isn't passed correctly and should be set inside the extra key. It's just a suspicion at this stage, I didn't get a chance to dig deeper... I'll try to play around and maybe submit a PR, but I'm going on holiday soon, so not sure when I'll have a chance to look into it.

not logging authenticated controllers

I am using knock for authentication, rails_semantic_controller is loging by default any controller that skips authentication, but is ignoring authenticated controllers

log rotation

Hi๏ผŒ
I am a absolutely beginner of ruby.
I am looking for a log lib, and I like this lib. It is so cool.
But I have a question now.
Do I need implement log rotation myself ?

SemanticLogger.add_signal_handler is unusable

When SemanticLogger.add_signal_handler is called, the signal handler is installed. When the actual signal is sent and the signal handler is executed, the signal handler fails. It fails because it tries to autoload code. Autoloading code is not allowed during the execution of a signal handler.

Testcase

require 'semantic_logger'

SemanticLogger.add_signal_handler

our_pid = Process.pid

Process.kill("TTIN",our_pid)

Expected result

A stack trace should be printed.

Actual result

A ThreadError can't be called from trap context is raised and printed.

Additional comments

This bug can be easily obscured by using other features of semantic_logger before the signal handler is executed, as the other features may also trigger autoloading, which succeeds in this case and thus will not be performed again when the signal handler is finally executed. Hence, in these cases, the bug will not be revealed.

Solution idea

Preload all code (use require instead of autoload), at least when SemanticLogger.add_signal_handler is called.

Tag logs with current Thread context information

I would like to be able to tag logs with information retrieved from the context of the Thread that called the log method. Similar to how the Thread.current_name is appended but with custom information.

A use case for this would be tagging anything logged in a rack request context. For example, I use Thread.current to store a request's Id and it'd be useful to specify a configuration to ensure that request id gets tagged in all the logs.

Right now this is possible if we wrap the log calls in the tagged method block and specify the request id. However, this is not possible in all instances.

This is also not possible using custom formatters since these get run in the context of the appender Thread which wouldn't have the information we need.

Discuss: should the payload be merged when `to_h`?

As the source code shows, when payload is a hash, it will be merged into the hash output.
As I can see, there is a potential problem: When payload contains an duplicate field(say: message or line), it will override the log message, or the file line number. Users need to go to the source code to find out the problem.

Error on Rails 5 API - ActiveModelSerializers

I am using Rails 5 to build an API, I tried to use SemanticLogger along with my application but when the controller tries to respond to the caller the following error is thrown:

2016-10-17 11:38:47.192437 I 27060:47357639085680 Api::V1::TestController -- Completed #index -- { :controller => "Api::V1::TestController", :action => "index", :params => { "array" => {} }, :format => "/", :method => "GET", :path => "/api/v1/test", :view_runtime => 0.0, :db_runtime => 19.61, :exception_object => #<NoMethodError: undefined method logger' for ActiveModelSerializers::SerializableResource:Class>, :status => 500, :status_message => "Internal Server Error" } 2016-10-17 11:38:47.192551 F [27060:47357639085680 debug_exceptions.rb:7] Rails -- Exception: NoMethodError: undefined method logger' for ActiveModelSerializers::SerializableResource:Class
/home/user/.rbenv/versions/2.3.1/lib/ruby/gems/2.3.0/gems/semantic_logger-3.3.0/lib/semantic_logger/loggable.rb:44:inlogger' /home/user/.rbenv/versions/2.3.1/lib/ruby/gems/2.3.0/gems/rails_semantic_logger-3.3.1/lib/rails_semantic_logger/extensions/active_model_serializers/logging.rb:9:in tag_logger'
/home/user/.rbenv/versions/2.3.1/lib/ruby/gems/2.3.0/gems/active_model_serializers-0.10.2/lib/active_model_serializers/logging.rb:20:in`block in instrument_rendering'

Somehow when I install the gem the ActiveModelSerializer loses the logger.

error with sentry appender

I'm looking at moving from logging-rails + lograge to semantic logger. It looks great so far.

I'm trying to add the sentry appender. I added in my config:

SemanticLogger.add_appender(:appender => :sentry, :level => :error)

When I log an error, I bump into this error:

2016-09-07 09:41:59.750979 E [11695:SemanticLogger::AppenderThread] SemanticLogger -- Appender thread: Failed to log to appender: #<SemanticLogger::Appender::Sentry:0x00000003b3f738 @formatter=#<SemanticLogger::Formatters::Raw:0x00000003b3cc68 @precision=6, @time_format="%Y-%m-%d %H:%M:%S.%6N", @log_host=true, @log_application=true>, @application=nil, @host=nil, @filter=nil, @name="SemanticLogger::Appender::Sentry", @level_index=4, @level=:error> -- Exception: NoMethodError: undefined method `user' for #<Hash:0x0000000287c718>
/usr/lib/ruby/gems/2.2.0/gems/sentry-raven-1.2.0/lib/raven/event.rb:63:in `initialize'
/usr/lib/ruby/gems/2.2.0/gems/sentry-raven-1.2.0/lib/raven/event.rb:102:in `new'
/usr/lib/ruby/gems/2.2.0/gems/sentry-raven-1.2.0/lib/raven/event.rb:102:in `from_message'
/usr/lib/ruby/gems/2.2.0/gems/sentry-raven-1.2.0/lib/raven/instance.rb:118:in `capture_type'
/usr/lib/ruby/gems/2.2.0/bundler/gems/semantic_logger-4278b62a1445/lib/semantic_logger/appender/sentry.rb:60:in `log'
/usr/lib/ruby/gems/2.2.0/bundler/gems/semantic_logger-4278b62a1445/lib/semantic_logger/logger.rb:191:in `block in appender_thread'
/usr/lib/ruby/gems/2.2.0/bundler/gems/semantic_logger-4278b62a1445/lib/semantic_logger/logger.rb:189:in `each'
/usr/lib/ruby/gems/2.2.0/bundler/gems/semantic_logger-4278b62a1445/lib/semantic_logger/logger.rb:189:in `appender_thread'
/usr/lib/ruby/gems/2.2.0/bundler/gems/semantic_logger-4278b62a1445/lib/semantic_logger/logger.rb:165:in `block in start_appender_thread'

Note that this log is from Sidekiq, since my sentry is set to report asynchronously.

If I manually report a message to sentry from within a Sidekiq job, it does work. e.g.

class SomeWorker
  include Sidekiq::Worker
  def perform
    Raven.capture_message('this error goes to sentry')
    logger.error('this error does not')
  end
end

logger.error fails to report to sentry also if I call it outside the sidekiq job.

Any clues / tips on how to resolve this? did I set up something wrong?

Prettify stacktrace on exceptions / 404

Hello,

What is the rationale of semantic_logger reguarding 404 or exceptions?

Right now it looks like the error is captured by semantic_logger (good!), but then the stack trace is blurted out like normal.

Is there a way to "jsonize" / prettify / suppress the stack trace?
Or maybe simply store it as is but in another log?

I'd really like to have "one line = one request" in the log for easier reading of what happened.

For information, I'm trying to find a replacement for lograge (see roidrage/lograge#48)

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.