ariegato / serilog-sinks-rabbitmq Goto Github PK

BatchPostingLimit and Period are not propagated thru LoggerConfigurationRabbitMQExtension, causing the sink not to log anything!

I have a fix ready and will issue a PR.

Please consider if there are changes in the Serilog implementation that we need to consider

Is your feature request related to a problem? Please describe.
Current code is not following RabbitMQ client's recommended usage. There is a risk that in high concurrency scenario, the logs will get corrupted. However, it is unclear whether Serilog itself has any built-in synchronization mechanism that makes synchronizing usage of RabbitMQ client unneeded

Describe the solution you'd like
Create a stress test docker environment to show that the current code breaks, or doesn't

Describe alternatives you've considered
Just follow the recomended RabbitMQ client pattern without test

Additional context
https://www.rabbitmq.com/dotnet-api-guide.html#model-sharing

https://github.com/sonicjolt/serilog-sinks-rabbitmq/blob/master/src/Serilog.Sink.RabbitMq/Sinks/RabbitMq/RabbitMqMessage.cs#L22

Even if JSON formatting is used, this will end up being JSON-string-within-JSON, so not super consumable - in general most sinks choose to either support JSON only or text only. It's not incorrect to pass an ITextFormatter through from the configuration method, but where this is done it's generally the flat output formats like console/file that do it.

It should be possible for a use for the sink to publish straight to a queue, and otherwise configure exchange, and exchange type.

We need to create a test-environment using docker containers to be able to run tests and prove known issues with the sink.

If I understand correctly, RabbitMQSink and RabbitMQClient should implement IDisposable and be closed correctly when the user calls Log.CloseAndFlush() from the client application. I have a simple console application, which keeps hanging and cannot exit even after the call to Log.CloseAndFlush. On my machine, I tried implementing Dispose in said classes, and this seems to fix the hanging issue.

It must be possible to specify how the RabbitMQ sink should react to a missing rabbitmq instance.

We need to upgrade the included RabbitMQ.Client to the latest version. I'm unsure if there are any breaking changes in the client, so we need to have the dockerized setup created before we are able to ensure that upgrading RabbitMq.Client is successful

Hi,
I'm going to collect all the logs of each request (from the action method to maybe multiple services and classes) and at the end, send them all to RabbitMQ at once!

Is it possible? what is your suggestion?

Thanks alot

(I couldn't chose Question as label, there were just Bug and New Feature)

After upgrading to version 2.0 we notice that there is not longer a property called Queue on RabbitMQConfiguration. What is the correct property to use instead?

Hi Steffen,
1) I had an issue with logging using serilog to rabbitmq.
while publishing messages from serilog, rabbitmq receives a lot of unwanted messages with the actual messages.
how to store only the log message in rabbitmq.
Currently it saves like below:
{"Timestamp":"2018-08-27T12:11:30.4682719+05:30","Level":"Error","MessageTemplate":"Startup Logging!!"}
{"Timestamp":"2018-08-27T12:11:32.9674578+05:30","Level":"Information","MessageTemplate":"User profile is available. Using '{FullName}' as key repository and Windows DPAPI to encrypt keys at rest.","Properties":{"FullName":"C:\Users\Administrator\AppData\Local\ASP.NET\DataProtection-Keys","SourceContext":"Microsoft.AspNetCore.DataProtection.KeyManagement.XmlKeyManager"}}
{"Timestamp":"2018-08-27T12:11:33.4844303+05:30","Level":"Information","MessageTemplate":"Entity Framework Core {version} initialized '{contextType}' using provider '{provider}' with options: {options}","Properties":{"version":"2.1.1-rtm-30846","contextType":"ConnectionsStoreModelContext","provider":"Microsoft.EntityFrameworkCore.Sqlite","options":"None","EventId":{"Id":10403,"Name":"Microsoft.EntityFrameworkCore.Infrastructure.ContextInitialized"},"SourceContext":"Microsoft.EntityFrameworkCore.Infrastructure"}}
{"Timestamp":"2018-08-27T12:11:33.6766514+05:30","Level":"Information","MessageTemplate":"Executed DbCommand ({elapsed}ms) [Parameters=[{parameters}], CommandType='{commandType}', CommandTimeout='{commandTimeout}']{newLine}{commandText}","Properties":{"elapsed":"8","parameters":"","commandType":"Text","commandTimeout":30,"newLine":"\r\n","commandText":"PRAGMA foreign_keys=ON;","EventId":{"Id":20101,"Name":"Microsoft.EntityFrameworkCore.Database.Command.CommandExecuted"},"SourceContext":"Microsoft.EntityFrameworkCore.Database.Command"}}

2) my 2nd problem was i want to format my payload with additional params as below:
{
"AppId" : "18c57f8a-a8f5-4bd1",
"AppName" : "some applicationName",
"MessageType" : "Information",
"Message" : "New Dashboard named 'dash 1' has been created by user dinesh.",
"TimeStamp" : "2018-08-17T17:04:56",
}
but my currrent payload does't have id, name etc.

Readme.md should contain usage documentation

Greetings! 👋

The LICENSE here on the repository was changed to GPL v3 a short while ago. The package on NuGet, (because of this line in the CSPROJ), is still listed as Apache 2.0.

There's a big difference between these licenses, so it would be great to get some clarity/make sure that the package information is correct.

Many users who are able to use the package under the earlier Apache 2.0 licensing possibly can't use it under GPL v3, so an announcement of the change somehow (README or issue tracker FAQ) might avoid some misunderstandings/confusion.

Hope this helps,
Nick

Is there any way of handling the exception differently when there is an error connecting to the server at bootstrap level?

Thanks a lot

At the moment RabbitMQ only supports UTF-8 encoding. This can quite easily be extended to support other encodings.

.

I am using a .net core web application when updating the serilog.sink.rabitmq to v3.0.0 the sink is not picking the configuration from the appsettings, I tried to adjust the setting with the new(breaking) change in version 3, I could not find any explicit guide how version 3 should be set up in appsettings.json so I tried the nested configuration according to https://github.com/serilog/serilog-settings-configuration#nested-configuration-sections
but get an exception, is there any way to add configuration settings in the appsettings.json or it should be manually set up in the code?

working settings with version 2

"WriteTo": [{ "Name": "RabbitMQ", "Args": { "Hostname": "", "Username": "", "Password": "", "Exchange": "", "ExchangeType": "", "DeliveryMode": "", "Port": 5672, "Formatter": "Serilog.Formatting.Compact.RenderedCompactJsonFormatter, Serilog.Formatting.Compact", "restrictedToMinimumLevel": "Warning" } }]

new setting according to version 3 changes

"WriteTo": [{ "Name": "RabbitMQ", "Args": { "configure": [ { "Name": "RabbitMQClientConfiguration", "Args": { "Hostnames": [ "" ], "Username": "", "Password": "", "Exchange": "", "ExchangeType": "", "DeliveryMode": "", "Port": } }, { "Name": "RabbitMQSinkConfiguration", "Args": { "textFormatter": "Serilog.Formatting.Compact.RenderedCompactJsonFormatter, Serilog.Formatting.Compact", "restrictedToMinimumLevel": "Warning" } } ] } }]

Startup.cs
loggerConfiguration.Enrich.FromLogContext().ReadFrom.Configuration(Configuration);

the exception
Cannot create instance of type 'System.Action2[Serilog.Sinks.RabbitMQ.Sinks.RabbitMQ.RabbitMQClientConfiguration,Serilog.Sinks.RabbitMQ.Sinks.RabbitMQ.RabbitMQSinkConfiguration]' because it is missing a public parameterless constructor.

Stack trace

at Microsoft.Extensions.Configuration.ConfigurationBinder.CreateInstance(Type type)
at Microsoft.Extensions.Configuration.ConfigurationBinder.BindInstance(Type type, Object instance, IConfiguration config, BinderOptions options)
at Microsoft.Extensions.Configuration.ConfigurationBinder.Get(IConfiguration configuration, Type type, Action1 configureOptions) at Microsoft.Extensions.Configuration.ConfigurationBinder.Get(IConfiguration configuration, Type type) at Serilog.Settings.Configuration.ConfigurationReader.<>c__DisplayClass20_0.<CallConfigurationMethods>b__3(<>f__AnonymousType72 <>h__TransparentIdentifier0)
at System.Linq.Utilities.<>c__DisplayClass2_03.<CombineSelectors>b__0(TSource x) at System.Linq.Enumerable.SelectListPartitionIterator2.ToList()
at System.Linq.Enumerable.ToList[TSource](IEnumerable1 source) at Serilog.Settings.Configuration.ConfigurationReader.CallConfigurationMethods(ILookup2 methods, IList1 configurationMethods, Object receiver, IReadOnlyDictionary2 declaredLevelSwitches)
at Serilog.Settings.Configuration.ConfigurationReader.Configure(LoggerConfiguration loggerConfiguration)
at Serilog.Configuration.LoggerSettingsConfiguration.Settings(ILoggerSettings settings) in C:\projects\serilog\src\Serilog\Configuration\LoggerSettingsConfiguration.cs:line 44
at Nuvei.App.MD.WebApi.Program.<>c.b__1_1(WebHostBuilderContext hostingContext, LoggerConfiguration loggerConfiguration) in

Describe the bug
Unable to overwrite the TextFormatter via appsettings.json

To Reproduce
Add configuration in appsetting.json:

"WriteTo": [
    {
      "Name": "RabbitMQ",
      "Args": {
        "sinkConfiguration": {
          "TextFormatter": "Serilog.Formatting.Compact.CompactJsonFormatter, Serilog.Formatting.Compact",
          "BatchPostingLimit": 300,
          "RestrictedToMinimumLevel": "Information"
        },
        "clientConfiguration": {
          "Hostnames": [
            "<hostname>"
          ],
          "Exchange": "log_events",
          "ExchangeType": "direct",
          "DeliveryMode": "Durable",
          "RouteKey": "logs",
          "Port": 5672,
          "Password": "<password>",
          "Username": "<user>"
        }
      }
    }
]

Expected behavior
Use specified formatter

Operating System and Software

• OS: Windows 10
• Nuget version: 3.0.3
• Assembly version: 0.0.0.0 (this is strange but not related to this problem)

Additional context
All attributes specified in both sinkConfiguration and clientConfiguration are used except for the TextFormatter attribute. It always reverts back to the default: RawFormatter
If tried "Serilog.Formatting.Json.JsonFormatter, Serilog" as well, but it just does not work.
It looks like you can only use an assembly reference as a parameter to the extension method (like in version 2.0.2) but not as a property in a class parameter.

I am not sure if I am missing something or if this is a bug.

The current configuration api is cumbersome with a lot of overload methods and large number of parameters. Using configuration via lambda expression will be more intuitive and easy to use

Log.Logger = new LoggerConfiguration()
        .Enrich.FromLogContext()
        .WriteTo.RabbitMQ((clientConfiguration, sinkConfiguration) => {
            clientConfiguration.Hostnames     = _config["RABBITMQ_HOSTS"];
            clientConfiguration.Username     = _config["RABBITMQ_USER"];
            clientConfiguration.Password     = _config["RABBITMQ_PASSWORD"];
            clientConfiguration.Exchange     = _config["RABBITMQ_EXCHANGE"];
            clientConfiguration.ExchangeType = _config["RABBITMQ_EXCHANGE_TYPE"];
            clientConfiguration.DeliveryMode = RabbitMQDeliveryMode.Durable;
            clientConfiguration.RouteKey     = "Logs";
            clientConfiguration.Port         = 5672;
            sinkConfiguration.TextFormatter = new JsonFormatter();
        })
        .CreateLogger();

In order to support configuration via app settings binding the following api can be added

var config= new RabbitMQConfiguration
        {
            Hostnames       = _config["RABBITMQ_HOSTS"],
            Username        = _config["RABBITMQ_USER"],
            Password        = _config["RABBITMQ_PASSWORD"],
            Exchange        = _config["RABBITMQ_EXCHANGE"],
            ExchangeType    = _config["RABBITMQ_EXCHANGE_TYPE"],
            DeliveryMode    = RabbitMQDeliveryMode.Durable,
            RouteKey        = "Logs",
            Port            = 5672
        };

        Log.Logger = new LoggerConfiguration()
        .Enrich.FromLogContext()
        .WriteTo.RabbitMQ((clientConfiguration, sinkConfiguration) => {
            clientConfiguration.From(config);
            sinkConfiguration.TextFormatter = new JsonFormatter();
        })
        .CreateLogger();

https://github.com/sonicjolt/serilog-sinks-rabbitmq/blob/master/src/Serilog.Sink.RabbitMq/LoggerConfigurationRabbitMqExtension.cs#L21

Serilog's app settings support needs an overload of this method that accepts each parameter individually.

https://github.com/sonicjolt/serilog-sinks-rabbitmq/blob/master/src/Serilog.Sink.RabbitMq/Serilog.Sinks.RabbitMq.csproj#L10

If you set the root namespace to Serilog then the namespacing and folder structure will line up (avoiding things like Serilog.Sinks.RabbitMQ.Sinks.RabbitMQ appearing in a few files).

Currently depends on Serilog (>= 1.5.14 && < 2.0.0).
NuGet package won't install with the Serilog 2.2.1.

The sink configuration only allows for specifying a single hostname (at least as far as I could see). We have a setup with multiple RabbitMQ servers where we'd like to connect to the first available.

RawRabbit supports this (to take another library around RabbitMQ as an example), by taking a list of Hostnames to connect to. Instead of setting the HostName property on ConnectionFactory, it passes that list of hostnames to RabbitMQ's CreateConnection method.

Would it be possible to add support for this in the sink? Perhaps a separate configuration property named in plural (Hostnames), which if specified makes the sink use multiple endpoints instead of a single one.

https://github.com/sonicjolt/serilog-sinks-rabbitmq/blob/master/src/Serilog.Sink.RabbitMq/Sinks/RabbitMq/RabbitMqClient.cs#L104

Is System.Text.Encoding.UTF8.GetBytes() an option here?

Is your feature request related to a problem? Please describe.
We can only use signed packages.
When using Serilog.Sinks.RabbitMq we've got following exception:

FileLoadException: Could not load file or assembly 'Serilog.Sinks.RabbitMQ, Version=0.0.0.0, Culture=neutral, PublicKeyToken=null' or one of its dependencies. A strongly-named assembly is required. (Exception from HRESULT: 0x80131044)

Describe the solution you'd like
Provide a signed package

Hello, is there any way to set up the sink with a TLS version (eg: TLS12) and Policies, instead of always using the default?

Thanks a lot
AR

Hi,

Yesterday after my rabbitmq server outage, my applications stopped logging completely.
Here is the stacktrace:

at RabbitMQ.Client.Impl.InboundFrame.ReadFrom(NetworkBinaryReader reader)
at RabbitMQ.Client.Impl.SocketFrameHandler.ReadFrame()
at RabbitMQ.Client.Framing.Impl.Connection.MainLoopIteration()
at RabbitMQ.Client.Framing.Impl.Connection.MainLoop()
at RabbitMQ.Client.Impl.SessionBase.Transmit(Command cmd)
at RabbitMQ.Client.Impl.ModelBase.ModelSend(MethodBase method, ContentHeaderBase header, Byte[] body)
at RabbitMQ.Client.Framing.Impl.Model._Private_BasicPublish(String exchange, String routingKey, Boolean mandatory, IBasicProperties basicProperties, Byte[] body)
at RabbitMQ.Client.Impl.ModelBase.BasicPublish(String exchange, String routingKey, Boolean mandatory, IBasicProperties basicProperties, Byte[] body)
at RabbitMQ.Client.Impl.AutorecoveringModel.BasicPublish(String exchange, String routingKey, Boolean mandatory, IBasicProperties basicProperties, Byte[] body)
at Serilog.Sinks.RabbitMQ.RabbitMQSink.EmitBatch(IEnumerable1 events) at Serilog.Sinks.PeriodicBatching.PeriodicBatchingSink.<EmitBatchAsync>d__15.MoveNext() --- End of stack trace from previous location where exception was thrown --- at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task) at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) at Serilog.Sinks.PeriodicBatching.PeriodicBatchingSink.<OnTick>d__16.MoveNext()RabbitMQ.Client.Exceptions.BrokerUnreachableException: None of the specified endpoints were reachable ---> System.AggregateException: One or more errors occurred. ---> RabbitMQ.Client.Exceptions.ConnectFailureException: Connection failed ---> System.Net.Sockets.SocketException: No connection could be made because the target machine actively refused it 10.132.87.118:5672 at System.Net.Sockets.Socket.DoConnect(EndPoint endPointSnapshot, SocketAddress socketAddress) at System.Net.Sockets.Socket.Connect(EndPoint remoteEP) at System.Net.Sockets.Socket.Connect(IPAddress address, Int32 port) at RabbitMQ.Client.TcpClientAdapter.<ConnectAsync>d__2.MoveNext() --- End of stack trace from previous location where exception was thrown --- at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task) at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) at RabbitMQ.Client.Impl.TaskExtensions.<TimeoutAfter>d__1.MoveNext() --- End of stack trace from previous location where exception was thrown --- at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task) at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) at RabbitMQ.Client.Impl.SocketFrameHandler.ConnectOrFail(ITcpClient socket, AmqpTcpEndpoint endpoint, Int32 timeout) --- End of inner exception stack trace --- at RabbitMQ.Client.Impl.SocketFrameHandler.ConnectUsingAddressFamily(AmqpTcpEndpoint endpoint, Func2 socketFactory, Int32 timeout, AddressFamily family)
at RabbitMQ.Client.Impl.SocketFrameHandler..ctor(AmqpTcpEndpoint endpoint, Func2 socketFactory, Int32 connectionTimeout, Int32 readTimeout, Int32 writeTimeout) at RabbitMQ.Client.ConnectionFactory.CreateFrameHandler(AmqpTcpEndpoint endpoint) at RabbitMQ.Client.EndpointResolverExtensions.SelectOne[T](IEndpointResolver resolver, Func2 selector)
--- End of inner exception stack trace ---
at RabbitMQ.Client.EndpointResolverExtensions.SelectOne[T](IEndpointResolver resolver, Func2 selector) at RabbitMQ.Client.Framing.Impl.AutorecoveringConnection.Init(IEndpointResolver endpoints) at RabbitMQ.Client.ConnectionFactory.CreateConnection(IEndpointResolver endpointResolver, String clientProvidedName) --- End of inner exception stack trace --- at RabbitMQ.Client.ConnectionFactory.CreateConnection(IEndpointResolver endpointResolver, String clientProvidedName) at RabbitMQ.Client.ConnectionFactory.CreateConnection(IList1 hostnames, String clientProvidedName)
at RabbitMQ.Client.ConnectionFactory.CreateConnection(IList1 hostnames) at Serilog.Sinks.RabbitMQ.RabbitMQClient.InitializeEndpoint() at Serilog.Sinks.RabbitMQ.RabbitMQClient..ctor(RabbitMQClientConfiguration configuration) at Serilog.Sinks.RabbitMQ.RabbitMQSink..ctor(RabbitMQClientConfiguration configuration, RabbitMQSinkConfiguration rabbitMQSinkConfiguration) at Serilog.LoggerConfigurationRabbitMqExtension.RabbitMQ(LoggerSinkConfiguration loggerConfiguration, Action2 configure)

Steps to reproduce the behavior:

1. Setup serilog with rabbitMQ
2. Close rabbitMq
3. See error

Ignore and keep logging to file

Additional context
here are the logs from my program:
Topshelf.Logging.SerilogLogWriter.Fatal() - The service threw an unhandled exception
System.IO.FileNotFoundException: Could not load file or assembly 'Microsoft.Diagnostics.Tracing.EventSource, Version=1.1.28.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a' or one of its dependencies. The system cannot find the file specified.
File name: 'Microsoft.Diagnostics.Tracing.EventSource, Version=1.1.28.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a'
at RabbitMQ.Client.ESLog.Error(String message, Exception ex)
at RabbitMQ.Client.Framing.Impl.Connection.LogCloseError(String error, Exception ex)
at RabbitMQ.Client.Framing.Impl.Connection.ClosingLoop()
at RabbitMQ.Client.Framing.Impl.Connection.MainLoop()
at System.Threading.ThreadHelper.ThreadStart_Context(Object state)
at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx)
at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx)
at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state)
at System.Threading.ThreadHelper.ThreadStart()

WRN: Assembly binding logging is turned OFF.
To enable assembly bind failure logging, set the registry value [HKLM\Software\Microsoft\Fusion!EnableLog] (DWORD) to 1.
Note: There is some performance penalty associated with assembly bind failure logging.
To turn this feature off, remove the registry value [HKLM\Software\Microsoft\Fusion!EnableLog].

Currently routing key configuration support only constants assigned at the initialization time. In case of topic exchange it would be beneficial to build different routing keys at a runtime.

For instance routing key specified in configuration as logs.{level} might be resolved at runtime to logs.warning, logs.debug, etc. depending on actual log entry.

How can i deserialize the LogEvent that was serialize to the rabbit queue?

In the queue, i have a message just like this:

{"Timestamp":"2017-09-28T20:34:04.6883220-03:00","Level":"Information","MessageTemplate":"{HostingRequestStarting:l}","RenderedMessage":"Request starting HTTP/1.1 GET http://localhost:5100/teste application/json ","Properties":{"Protocol":"HTTP/1.1","Method":"GET","ContentType":"application/json","ContentLength":null,"Scheme":"http","Host":"localhost:5100","PathBase":"","Path":"/teste","QueryString":"","HostingRequestStarting":"Request starting HTTP/1.1 GET http://localhost:5100/teste application/json ","EventId":{"Id":1},"SourceContext":"Microsoft.AspNetCore.Hosting.Internal.WebHost","RequestId":"0HL86P5K139GQ","RequestPath":"/teste"},"Renderings":{"HostingRequestStarting":[{"Format":"l","Rendering":"Request starting HTTP/1.1 GET http://localhost:5100/teste application/json "}]}}

When i try to deserialize i got some exceptions.

_model.QueueDeclare(_config.QueueName,
durable: (int)_config.DeliveryMode == 2,
exclusive: false,
autoDelete: false,
arguments: null);

Hi, Im going to use this project in my for example console App.
I want to send logs to rabbitmq and receive in another App.
what are the steps?
Thanks

Currently the sink only supports UTF-8. It might be a good idea to support other encodings.

Hi there, in our case we dont use username and password, but the library make us use one.
It shouldn't be necessary to specify one.

I could contribute if you wish

thanks!

https://github.com/serilog/serilog-sinks-email/blob/master/src/Serilog.Sinks.Email/Properties/AssemblyInfo.cs

As far as I can tell the queue name validation is entirely unnecessary since we're publishing logs. I don't see where the queue name is used in the RabbitMQClient

This results in an unnecessary error for an unnecessary configuration value.

The plugin generally works great. However, if the broker is unavailable during logging initialization or during operation, it seems to break the application. (Exception from the RabbitMQ client is not handled inside the sink). Is there any advise on how to handle this?

There is no code examples or at least basic wiki page.

Nitpick, but the FDGs suggest this should be MQ, not Mq:

Do capitalize both characters of two-character acronyms

https://msdn.microsoft.com/en-us/library/vstudio/ms229043(v=vs.100).aspx

Please add an infrastructure for unit tests. Create at least one unit test with desired framework and configure infrastructure, so community will be able to add the rest