mahdighorbanpour / serilog-sinks-azurecosmosdb Goto Github PK

A Serilog sink that writes to Azure CosmosDB and supports PartitionKey for better performance.

License: Apache License 2.0

C# 98.21% JavaScript 1.79%

serilog-sinks-azurecosmosdb's Introduction

Serilog.Sinks.AzureCosmosDB

A Serilog sink that writes to Azure CosmosDB and supports PartitionKey for better performance. This code is based on serilog-sinks-azuredocumentdb and adapted to use the latest Microsof.Azure.Cosmos SDK version 3.12.0 and uses a custom JsonSerializationSettings which helps to keep the sink alive when serialization fails sometimes specially for Exceptions!

Getting started

You can start by installing the NuGet package.

Configure logger by calling WriteTo.AzureCosmosDB(<uri>, <key>)

var logger = new LoggerConfiguration()
    .WriteTo.AzureCosmosDB(<uri>, <secure-key>)
    .CreateLogger();

PartitionKey

Default partition key name is /UtcDate althought it can be overrided using parameter like below

Log.Logger = new LoggerConfiguration()
                .WriteTo.AzureCosmosDB(
                    endpointUri: <uri>,
                    authorizationKey: <secure-key>,
                    partitionKey: "MyCustomKeyName"
                )
                .CreateLogger();

IPartitionKeyProvider

The DefaultPartitionkeyProvide will generate a utc date string with the format "dd.MM.yyyy". If you want to override it, you need to define a class and implement IPartitionKeyProvider interface and pass an instance of it in the arguments list.

TTL (Time-to-live)

Azure CosmosDB is making easier to prune old data with support of Time To Live (TTL) so does Sink. AzureCosmosDB Sink offers TTL at two levels.

Enable TTL at collection level.

Sink supports TTL at collection level, if collection does not already exist.

To enable TTL at collection level, set timeToLive parameter in code.

.WriteTo.AzureCosmosDB(<uri>, <secure-key>, timeToLive: TimeSpan.FromDays(7))

If collection in CosmosDB doesn't exists, it will create one and set TTL on collection level causing all logs messages purge older than 7 days.

Enable TTL at inidividual log message level.

Sink do support TTL at individual message level. This allows developer to retian log message of high importance longer than of lesser importance.

logger.Information("This message will expire and purge automatically after {@_ttl} seconds", 60);

logger.Information("Log message will be retained for 30 days {@_ttl}", 2592000); // 30*24*60*60

logger.Information("Messages of high importance will never expire {@_ttl}", -1);

See TTL behavior in CosmosDB documentation for in depth explianation.

Note: {@_ttl} is a reserved expression for TTL.

XML configuration

To use the AzureCosmosDB sink with the Serilog.Settings.AppSettings package, first install that package if you haven't already done so:

Install-Package Serilog.Settings.AppSettings

In your code, call ReadFrom.AppSettings()

var logger = new LoggerConfiguration()
    .ReadFrom.AppSettings()
    .CreateLogger();

In your application's App.config or Web.config file, specify the CosmosDB sink assembly and required endpointUrl and authorizationKey parameters under the <appSettings>

<appSettings>
  <add key="serilog:using:AzureCosmosDB" value="Serilog.Sinks.AzureCosmosDB" />
  <add key="serilog:write-to:AzureCosmosDB.endpointUrl" value="https://****.cosmos.azure.com:443" />
  <add key="serilog:write-to:AzureCosmosDB.authorizationKey" value="****" />
    
  <!-- Liefspan of log messages in CosmosDB in seconds, leave empty to disable expiration. -->
  <add key="serilog:write-to:AzureCosmosDB.timeToLive" value="60" />
</appSettings>

Performance

Sink buffers log internally and flush to Azure CosmosDB in batches using dedicated thread. However, it highly depends on type of Azure CosmosDB subscription you have.

serilog-sinks-azurecosmosdb's People

Contributors

Stargazers

Watchers

Forkers

tghamm arcalise08 irwin08 janaks09 kisoft-tech

serilog-sinks-azurecosmosdb's Issues

Pushing a property to the root of log event object

I'm having an issue figuring out if it's possible to pull a property of of the Properties object and push it into the root of the object, in effort to set this property as my partition key.

I would like this because EF Core currently doesn't support the nested partition key
(dotnet/efcore#26978)

{ "Properties": { "CorrelationId": "d26b41bd-d107-4f29-955a-68ea5c572f45" }, "UtcDate": "01.07.2022" }
(snipped down for brevity)

Where UtcDate is the current partition key, by default. I would like to bring CorrelationId out of the properties object, and into the root so that I can tell EF Core that CorrelationId is the new partition key. I cannot find anything in the Serilog or Cosmos sink docs that may allow me to do this.

Thank you!

Open to a PR with some improvements?

Hi, thank you for your work on this sink. I came across it, but noticed a few issues, and have made some changes that I think will make it more stable and work like some of the more popular sinks. Would you be open to a PR if I submitted it for review?

Thanks so much!

Grant

Would you consider an additional maintainer?

Hi @mahdighorbanpour would you consider the help of an additional project maintainer for this project? I'd be happy to lend a hand if there is any interest. I maintain another project that has had a little bit of use. Please see: https://github.com/tghamm/dynamic-linq-query-builder. Thanks for considering, but no problem if you're not interested.

Getting 400 While inserting data

Hi when I try to insert 10 records using postman runner. Out of 10 only 2 or 3 records are getting added in cosmos db rest are getting 400 bad request.
Here is my code.

Log.Logger = new LoggerConfiguration()    
                    .WriteTo.Console()    
                    .WriteTo.AzureCosmosDB(
                   endpointUri: new Uri(cosmosDbConfig.Account),    
                    authorizationKey: cosmosDbConfig.Key,    
                    partitionKey: "PartitionKey",    
                    partitionKeyProvider: new PartitionKeyProvider(),     
                    databaseName: cosmosDbConfig.DatabaseName,      
                    collectionName: cosmosDbConfig.ContainerName,      
                    batchSize: 100,     
                    timeToLive: TimeSpan.FromDays(7)     
                    ).CreateLogger();

I have simple post api.

[HttpPost]
public IActionResult Post([FromBody] LogInfoModel logInfo)	
{	
    try	
    {	
        if (logInfo!= null)	
        { 	
            Log.ForContext("logInfo", logInfo, true).Information($"xyz");	
            return Ok();	
        }	
    }	
    catch (Exception ex)	
    {	
        return StatusCode(500, ex.Message);	
    }	
}

Is there any configuration or setting which i missed over here?

Can you also guide me how I can enable detail level logging for capturing more information about failure.

This package violates CosmosClient singleton rules

There should be only one instance of CosmosClient per application. By not allowing us to pass in our own client, using this package means essentially, you are maintaining two CosmosClients or more. In a microservice environment this can really balloon up. (7 services, 7 clients, using this package =14 clients).

I make the following recommendations
-Allow the ability to pass in your own CosmosClient
-Allow the ability to pass in your own options to the CosmosBuilder
-At the very minimum, SSL to the CosmosBuilder should be togglable. Local networked CosmosDB testing environments don't have -good solutions for SSL see here Microsoft recommends turning off SSL validation at the application level in these instances. Since that is not an option for this package, Its impossible to test it in these situations

I am unsure if this package is being maintained. I will fork these changes to my own branch for the moment. However if the owner of this repo would like me to make a pull request I will do so.