stemarie / log4net.azure Goto Github PK

Is it possible to Create and Log data to Custom columns on Azure tables ?

Provide a mechanism to set table name for each web request or thread.

I have a multitenant webapp, and it will be nice to have logs for each tenant in different tables so it get easy to lookup.

I think it would be better if the RowKey and PartitionKey (in DateReverse mode) had been generated from UTC rather than from local time.

If you like the feature, I can probably do it myself and create a PR.

Thanks for a great utility, I really needed this one when I found it!

I use a web.config contain the log4net configuration.
but I check the blob container, and it has nothing yet.
I was searching the idea for a long time as a beginner.
and I don't know why.

web.config

<?xml version="1.0" encoding="utf-8"?>
<configuration>
  
  <configSections>
    <section name="log4net" type="log4net.Config.Log4NetConfigurationSectionHandler, log4net" />
  </configSections>

  <appSettings>
    <add key="log4net.Config" value="log4net.Config" />
    <add key="log4net.Config.watch" value="true" />
    <add key="log4net.Internal.Debug" value="true" />
  </appSettings>

  <system.diagnostics>
    <trace autoflush="true">
      <listeners>
        <add
            name="TextWriterTraceListener"
            type="System.Diagnostics.TextWriterTraceListener"
            initializeData="D:\srcCSharp\SampleApp\WebApplicationTest\logs\log4net.txt" />
      </listeners>
    </trace>
  </system.diagnostics>
  
  <log4net>

  
    <!-- Azure AppendBlob Appender, uncomment, set proper BucketName and AWS credentials (appSettings) to try it out -->
    <appender name="BlobAppender" type="log4net.Appender.AzureAppendBlobAppender, log4net.Appender.Azure">
      <!-- You can either specify a connection string or use the ConnectionStringName property instead -->
      <param name="ConnectionString" value="UseDevelopmentStorage=true" />
      <param name="ConnectionString" value="DefaultEndpointsProtocol=https;AccountName=logstorage;AccountKey=3r90Axf+UO;EndpointSuffix=core.windows.net" />
      <param name="ContainerName" value="logs" />
      <param name="DirectoryName" value="%date{yyyyMM}" />
      <!--<param name="ConnectionStringName" value="GlobalConfigurationString" />-->
      <layout type="log4net.Layout.PatternLayout">
        <conversionPattern value="Log At %date{yyyy/MM/dd HH:mm:ss, fff} [%thread] %-5level %logger  - %message%newline" />
      </layout>
    </appender>

    <appender name="ConsoleAppender" type="log4net.Appender.ConsoleAppender">
      <layout type="log4net.Layout.PatternLayout">
        <conversionPattern value="Log At %date{yyyy/MM/dd HH:mm:ss,fff} [%thread][%-5level]%logger  - %message%newline" />
      </layout>
    </appender>

    <root>
      <level value="ALL" />
      <appender-ref ref="BlobAppender" />
      <appender-ref ref="ConsoleAppender" />
    </root>
  </log4net>

  <connectionStrings>
    <!-- Provided as an example for specifying a global connection string for multiple appenders -->
    <add name="GlobalConfigurationString" connectionString="UseDevelopmentStorage=true" />
  </connectionStrings>
  
  <startup>
    <supportedRuntime version="v4.0" sku=".NETFramework,Version=v4.5" />
  </startup>
  <runtime>
    <assemblyBinding xmlns="urn:schemas-microsoft-com:asm.v1">
      <dependentAssembly>
        <assemblyIdentity name="Microsoft.Data.Edm" publicKeyToken="31bf3856ad364e35" culture="neutral" />
        <bindingRedirect oldVersion="0.0.0.0-5.6.0.0" newVersion="5.6.0.0" />
      </dependentAssembly>
      <dependentAssembly>
        <assemblyIdentity name="Microsoft.Data.Services.Client" publicKeyToken="31bf3856ad364e35" culture="neutral" />
        <bindingRedirect oldVersion="0.0.0.0-5.6.0.0" newVersion="5.6.0.0" />
      </dependentAssembly>
      <dependentAssembly>
        <assemblyIdentity name="Microsoft.Data.OData" publicKeyToken="31bf3856ad364e35" culture="neutral" />
        <bindingRedirect oldVersion="0.0.0.0-5.6.0.0" newVersion="5.6.0.0" />
      </dependentAssembly>
      <dependentAssembly>
        <assemblyIdentity name="Newtonsoft.Json" publicKeyToken="30ad4fe6b2a6aeed" culture="neutral" />
        <bindingRedirect oldVersion="0.0.0.0-7.0.0.0" newVersion="7.0.0.0" />
      </dependentAssembly>
    </assemblyBinding>
  </runtime>

</configuration>

Startup.cs

// public void Configure(IApplicationBuilder app, IHostingEnvironment env) 
XmlDocument XmlDoc = new XmlDocument();
string configFile = "web.config";
XmlDoc.Load(File.OpenRead(configFile));
var repo = log4net.LogManager.CreateRepository(Assembly.GetEntryAssembly(), typeof(log4net.Repository.Hierarchy.Hierarchy));
XmlNodeList elemList = XmlDoc.GetElementsByTagName("log4net");
log4net.Config.XmlConfigurator.Configure(repo, (XmlElement)elemList[0]);

ILog Logger = LogManager.GetLogger(System.Reflection.MethodBase.GetCurrentMethod().DeclaringType);
if(Logger.IsDebugEnabled)
{
    Logger.Debug(" ---- All is well !");
}

As the ConsoleAppender is working well , the BlobAppender works nothing .
My Azure Storage Explorer has no file created.
Are there Any ideas for me .
@matt-lethargic
@dstj
@stemarie
@mryderie
@Thumann

Is there any plan to add DirectoryName for Azure Blob Appender in the near future?
It would be really helpful if the DirectoryName could be specified as a date format from the configuration file.

Something like yyyy-MM-dd, so that we end up with one folder per day.

Thanks for the great work 👍

There is no license set for this project's source code. There is not tons of code, and it's relatively straight forward, but could you please set a license to avoid any ambiguity. I suggest MIT or Apache2

I'm trying to use azure table logging an MVC app and the logs aren't being sent to azure. I have a textappender and consoleappender that work fine.

Currently logging to blob storage is done via adding a separate xml file to the specified blob container. However, Azure storage now supports AppendBlobs, which are optimized for logging scenarios.

http://blogs.msdn.com/b/windowsazurestorage/archive/2015/04/13/introducing-azure-storage-append-blob.aspx

Is there any interest in accepting a PR that implements this functionality?

Table storage is a pain to delete row from so it would be a good idea to roll the table name that is written to like the RollingFileAppender.

I'd suggest just supporting daily/weekly/monthly rolling as that provides enough granularity to work out the wipe/archival strategy you might want for logs

Don't mind having a go writing it, just let me know if anyone else has any requirements before I start

Hi,

When the project is built in debug mode the logging works fine. However when deployed in release mode it doesn't log. Even on the dev machine doesn't log when mode is release.

Do you have any ideas why is that ?

Thanks

Precision is up to the second and may overwrite previous entries

I have installed the package. The appender creates the logging table, but never actually writes anything to it. Below is the setup for confirmation:

<log4net> <appender name="AzureTableAppender" type="log4net.Appender.AzureTableAppender, log4net.Appender.Azure"> <param name="TableName" value="MobileLog"/> <param name="ConnectionStringName" value="Azure_Logging" /> <!-- You can specify this to make each LogProperty as separate Column in TableStorage, Default: all Custom Properties were logged into one single field --> <param name="PropAsColumn" value="true" /> <param name="PartitionKeyType" value="LoggerName" /> </appender> <root> <level value="INFO" /> <appender-ref ref="AzureTableAppender" /> </root> </log4net>

when i use ConnectionStringName like below, it doesn't work
<connectionStrings>
<add name="Log4Net.WindowsAzureStorage.ConnectionString" connectionString=****" />
</connectionStrings>

`
...

but if i use the ConnectionString directly like below, it works, why? is it because a lack of ? but i don't know how to write the ...
`
...

I suggest introducing a property Columns which can be used to specify what to log to Azure. Eg.

<param name="Columns" value="Level,Message,EventTimeStamp,ThreadName,MyProperty" />

If you like the feature, I can probably do it myself and create a PR.

The exception happened 🥇

System.MissingMethodException: Method not found: 'Boolean Microsoft.WindowsAzure.Storage.Blob.CloudBlobContainer.CreateIfNotExists(Microsoft.WindowsAzure.Storage.Blob.BlobRequestOptions, Microsoft.WindowsAzure.Storage.OperationContext)'.
at log4net.Appender.AzureAppendBlobAppender.ActivateOptions()
at log4net.Repository.Hierarchy.XmlHierarchyConfigurator.ParseAppender(XmlElement appenderElement)
log4net:ERROR Appender named [BlobAppender] not found.

My env happened the issue by using log4net.Appender.Azure.
I named my appender with [BlobAppender] on the type of AzureAppendBlobAppender.
I spended one and a half day trying to sovle it as a beginner.
Finally I change the CreateIfNotExists() method to CreateIfNotExistsAsync(), and it solved.

before:------------------------
_cloudBlobContainer.CreateIfNotExists();
after:--------------------------
_cloudBlobContainer.CreateIfNotExistsAsync();

Just for share~

Thank you for this great appender. Will there be a .NET Standard version soon?

It is a pity that this appender cannot be used in .NET Core, otherwise I can run a Linux service on Azure and then also log to Azure instead to files.

I am getting the following warning from MSBuild:

CSC: Referenced assembly 'log4net.Appender.Azure, Version=1.2.1.23255, Culture=neutral, PublicKeyToken=null' does not have a strong name.

This is apparently because the assembly is not signed.

Can we have a flexibility to override method to have our own way of storing for PartitionKey and RowKey ?

If PartionKeyType=DataReverse, it will split the data up into many more batches than necessary.

Log entries made with, for example, log4net.ILog.Error(msg, exc) are missing the Exception details from the log message.

Any suggestions on how to effectively query using Row key(ticks+guid) ?

I'm using the following for now :

    <appender name="AzureBlobAppender" type="log4net.Appender.AzureBlobAppender, log4net.Appender.Azure">
      <param name="ContainerName" value="testblobcontainer" />
      <param name="DirectoryName" value="logs/" />
      <!-- You can either specify a connection string or use the ConnectionStringName property instead -->
      <param name="ConnectionString" value="UseDevelopmentStorage=true;" />
      <!--<param name="ConnectionStringName" value="GlobalConfigurationString" />-->
      <bufferSize value="5" />
    </appender>

But as expected I'm ending up with MANY xml files. So my question is, can I log into 1 text file? If not, can I at least end up with 1 xml file instead of 1 for each log message?

PS: I tried using the following but didnt work

<appender name="AzureAppendBlobAppender" type="log4net.Appender.AzureAppendBlobAppender, log4net.Appender.Azure">
  <param name="ContainerName" value="testblobcontainer"/>
  <param name="DirectoryName" value="logs"/>
  <!-- You can either specify a connection string or use the ConnectionStringName property instead -->
  <param name="ConnectionString" value="UseDevelopmentStorage=true"/>
  <!--<param name="ConnectionStringName" value="GlobalConfigurationString" />-->
  <bufferSize value="5" />
</appender>

Somehow, the .csproj contains the following xml:

<Reference Include="Microsoft.WindowsAzure.Configuration">
  <Reference Include="Microsoft.Data.Services.Client">
    <HintPath>..\packages\Microsoft.Data.Services.Client.5.6.4\lib\net40\Microsoft.Data.Services.Client.dll</HintPath> 
    <Private>True</Private>
  </Reference>

The <Reference ... WIndowsAzure.Configuration is never closed. Later on, the entry is correctly repeated, but the XML is already broken.

The azure table storage entry is missing a column for the Log Level information.

Hi,

I have a issue when i use the library with asp.net mvc 5. Only the RollingLogFileAppender is known. The appenders using your library is not known! With a console application, that's work!

I publish my code sample application using asp.net mvc and the log4net appender using it with azure storage.
https://github.com/jducobu/log4net.Appender.Azure-test/tree/master

When i debug the code of the library, i have an http exception 400 - The remote server returned an error: (400) Bad Request trying to execute CreateIfNotExists method with a status ProtocoError

Do you have have an idea ?

Jérôme

Hi @stemarie,

On this Repo there are some issues with possible solutions so can you continue with integrate or discussing the solutions?
If no, can you add another contributor to continue Developing and integrating Features? And updating the Nuget Packages.

I can take this role, but if anyone else want to do this so they are welcome and just add a comment to this issue.

Thanks Erik

I'm new to Azure table storage so I'm guessing that this issue is just due to something I'm missing on my end. I've set up a new general Resource Manager storage account with otherwise typical configuration. I've formulated my connection string as so.

<appender name="AzureTableAppender" type="log4net.Appender.AzureTableAppender, log4net.Appender.Azure">
  <param name="TableName" value="MY_TABLE"/>
  <param name="ConnectionString" value="DefaultEndpointsProtocol=https;AccountName=AccountName;AccountKey=MYKEY;"/>
  <param name="PropAsColumn" value="true" />
  <param name="PartitionKeyType" value="LoggerName" />
</appender>

But when logging using AzureTableAppender I can see entries logged to the other appenders but nothing shows up in my storage account tables but I found the following in my Debug Output window in Visual Studio.

log4net:ERROR Could not create Appender [AzureTableAppender] of type [log4net.Appender.AzureTableAppender, log4net.Appender.Azure]. Reported error follows.
Microsoft.WindowsAzure.Storage.StorageException: The remote server returned an error: (400) Bad Request. ---> System.Net.WebException: The remote server returned an error: (400) Bad Request.
   at System.Net.HttpWebRequest.GetResponse()
   at Microsoft.WindowsAzure.Storage.Core.Executor.Executor.ExecuteSync[T](RESTCommand`1 cmd, IRetryPolicy policy, OperationContext operationContext) in c:\Program Files (x86)\Jenkins\workspace\release_dotnet_master\Lib\ClassLibraryCommon\Core\Executor\Executor.cs:line 677
   --- End of inner exception stack trace ---
   at Microsoft.WindowsAzure.Storage.Core.Executor.Executor.ExecuteSync[T](RESTCommand`1 cmd, IRetryPolicy policy, OperationContext operationContext) in c:\Program Files (x86)\Jenkins\workspace\release_dotnet_master\Lib\ClassLibraryCommon\Core\Executor\Executor.cs:line 604
   at Microsoft.WindowsAzure.Storage.Table.TableOperation.Execute(CloudTableClient client, CloudTable table, TableRequestOptions requestOptions, OperationContext operationContext) in c:\Program Files (x86)\Jenkins\workspace\release_dotnet_master\Lib\ClassLibraryCommon\Table\TableOperation.cs:line 41
   at Microsoft.WindowsAzure.Storage.Table.CloudTable.Create(TableRequestOptions requestOptions, OperationContext operationContext) in c:\Program Files (x86)\Jenkins\workspace\release_dotnet_master\Lib\ClassLibraryCommon\Table\CloudTable.cs:line 910
   at Microsoft.WindowsAzure.Storage.Table.CloudTable.CreateIfNotExists(TableRequestOptions requestOptions, OperationContext operationContext) in c:\Program Files (x86)\Jenkins\workspace\release_dotnet_master\Lib\ClassLibraryCommon\Table\CloudTable.cs:line 1051
   at log4net.Appender.AzureTableAppender.ActivateOptions()
   at log4net.Repository.Hierarchy.XmlHierarchyConfigurator.ParseAppender(XmlElement appenderElement)
Request Information
RequestID:ba55bf8d-0002-00b3-427c-8cbb4d000000
RequestDate:Tue, 21 Feb 2017 19:58:41 GMT
StatusMessage:Bad Request
ErrorCode:InvalidResourceName

log4net:ERROR Appender named [AzureTableAppender] not found.

Am I missing something? Do I have to create some sort of table name or a schema ahead of time?

Hi,

can you explain why you have used the LoggerName Property as PartitionKey?
most of the Logging examples always use a combination of year month day ...
And for Scaling in a table this make more sence or am I wrong?

regards,
Erik

I was trying to use the table appender, but looks like it fails because the AzureLoggingEventEntity has a property called "Properties". Very weird, but I am able to consistently repro the issue.
Steps:
Just ensure some value is being set to the Properties property. If no value is set, issue doesn't repro. I changed the property name from Properties to PropertiesDictionary and everything works fine!

Our azure website is generating warnings on start up "Role environment . FAILED TO INITIALIZE. hr: -2147024891" which are traced down to dependency on WindowsAzure.Configuration assembly.

for more info check:
http://stackoverflow.com/questions/35495505/azure-websites-complaining-role-environment-failed-to-initialize-on-startup

Azure cloud services, roles are basically deprecated.

consider getting rid of this block code in Util.cs

 // Fallback to CloudConfigurationManager in case we're running as a worker/web role
            var azConfig = CloudConfigurationManager.GetSetting(connectionStringName);
            if (!string.IsNullOrWhiteSpace(azConfig))
            {
                return azConfig;
            }

I think it would be a nice feature if it were possible to specify that the actual should take place on a background thread.

If you like the feature, I can probably do it myself and create a PR.

The official log4net.Appender.Azure NuGet package doesn't extract the library file. As a result, the DLL is not being referenced in the project upon package installation.

Hi @stemarie ,

I'm using log4net.Appender.AzureAppendBlobAppender to log my web app's info & errors. Sometimes I'm getting the "BlockCountExceedsLimit" exception. It is due to the append blob accepts only 50,000 block commits after that it through the exception (Conflict (409)). I have checked the code and found that it waits for the 512 log events and flush each log entry separately to the append blob. So, we can log only 50,000 log entries in a day.

Do you have any solution for this? Can we flush the 512 log entries in a single flush?

Thanks,
Karthik

How can i "commit/flush" an entry immediately after logging it ?

Kind regrads
Markus

I've been experimenting with this project, and rather than provide one of the default keys for the storage account as the connection string, I'd like to provide one with limited permissions. Specifically, I'd like to use a connection string that only allows for access to the specific container where the blob appender will be storing logs.

I can create a connection string from a stored access policy that I create that just provides the required permissions for my target container, but ActivateOptions in the AzureBlobAppender throws an exception due to a 403 returned by the remote service. The problem is due to the call to CreateIfNotExists, which cannot be performed without the ability to create containers.

I don't think that providing the ability to create containers should be necessary for logging into one. Would you consider either removing this call or wrapping it in a try/catch, to prevent failure when this permission is not available?

When deploying on Azure, it often makes sense not to put storage connection strings into source code. It is possible to configure them as part of the portal service configuration.
They can then be access via ConfigurationManager.ConnectionStrings.

The following patch allows to use

<appender name="AzureTableAppender" ....>
  <param name="ConnectionStringName" value="XXX" />

when configuring AzureTableAppender, and use the same connection definition XXX that is used in other places in the code, anyway.

--- a/log4net.Azure/AzureTableAppender.cs
+++ b/log4net.Azure/AzureTableAppender.cs
@@ -1,4 +1,5 @@
 using System;
+using System.Configuration;
 using System.Linq;
 using log4net.Appender.Extensions;
 using log4net.Appender.Language;
@@ -14,12 +15,19 @@ public class AzureTableAppender : BufferingAppenderSkeleton
         private CloudTableClient _client;
         private CloudTable _table;

+        public string ConnectionStringName { get; set; }
         private string _connectionString;

         public string ConnectionString
         {
             get
             {
+                if (!string.IsNullOrWhiteSpace(ConnectionStringName))
+                {
+                    var config = ConfigurationManager.ConnectionStrings[ConnectionStringName];
+                    if (config != null)
+                        return config.ConnectionString;
+                }
                 if (String.IsNullOrEmpty(_connectionString))
                     throw new ApplicationException(Resources.AzureConnectionStringNotSpecified);
                 return _connectionString;

Unit tests failing for all three types of loggers

• Azure Table
• Azure Dynamic Table
• Azure Blob

Hi,

I'm trying to use the Azure Table storage appender but nothing gets added.
I've added

<add key="log4net.Internal.Debug" value="true"/>

and

<system.diagnostics>
    <trace>
     <listeners>
  <add
       name="textWriterTraceListener"
       type="System.Diagnostics.TextWriterTraceListener"
       initializeData="C:\dev\log4net.txt" />
       </listeners>
    </trace>
  </system.diagnostics>

and the error that I'm getting is this one

log4net:ERROR Could not create Appender [AzureTableAppender] of type [log4net.Appender.AzureTableAppender, log4net.Appender.Azure]. Reported error follows.
Microsoft.WindowsAzure.Storage.StorageException: The remote server returned an error: (501) Not Implemented. ---> System.Net.WebException: The remote server returned an error: (501) Not Implemented.

Any ideas ?

Thanks

When using the PropAsColumn setting with the Table Storage appended I am seeing duplicate columns: there is a column for Identity & log4net_Identity both containing the same values, UserName & log4net_UserName also contain the same values.

AzureDynamicLoggingEventEntity is writing all avaialble members of the LoggingEvent along with any items in the Propertes collection which contains the duplicate values.

I'm getting "FormatException: The string '11/18/2016 15:55:45' is not a valid AllXsd value" when I'm trying to deserialize LogEntry to object

I'm using the appender in our projects. As log tables can get very large we would like to create a daily log table, e.g. AzureTable20170113, AzureTable20170114...
We could then delete old logs by droping the table, which is faster then deleting individual rows.

My understanding is that this can be accomplished by using extending the classes, but it would be great to have a property which will handle it natively, [TableNamePattern] = "AzureTable%date{yyyyMMdd}".

In our situation, the connection string needs to be injected from external resources (Azure DocumentDB, App Configuration Settings, Azure KeyVault). The idea is to remove AccountKeys from the source code.
Our first idea was to not provide the Connection string in the config and to call ActiveOptions() only if there is a connection string.

We created our own classes, but would prefer a native solution. Let me know if I can provide samples

Please add in the .nuspec file a dependency to Microsoft.WindowsAzure.ConfigurationManager (≥ 3.1.0)

See the attached log for reference

log4net.txt

Natively, Azure Table Storage creates a "Timestamp" column with the date the row was created. The AzureLoggingEventEntity also has a TimeStamp column (but with a capital S). Confusion is easy between the two.

DateTime values missing milliseconds which are very useful in tracing events