features.. about log4j-s3-search HOT 3 CLOSED

Hello,

Append logs to an existing file in S3

The selection of the key name in S3 currently takes into account the date/time, host name, and an application-specific ID to avoid name clash and nasty concurrency issues when multiple hosts/processes are appending to the same destination (just imagine two threads writing to a file). There is also a possible performance hit (unless there is an S3 feature to append that I don't know about) of having to download the content of the key, append to it, then upload it back into the same key.
Honestly I can't think of a good solution to these challenges off of the top of my head. Perhaps if you can elaborate on what problem you're trying to solve by having these logs go into a preexisting file, I can try to figure something out.

compress output before storing in S3

This is definitely doable. I will look into it. I believe there is already support in Java for this.

from log4j-s3-search.

re appending logs, the motivation is mainly to reduce the amount of files that get generated in the bucket. Just googling now it's clear s3 does not support in place file appending so I guess it's not an option. Downloading & overwriting would work but inefficient I guess.

Thanks anyway!

from log4j-s3-search.

Oh I see what you mean. You can mitigate the number of keys used somewhat by increasing the log4j.appender.S3Appender.stagingBufferSize or log4j.appender.S3Appender.stagingBufferAge.

But it won't solve the problem completely because at some threshold value the buffer will take on too much memory (and the eventual flush/publish will be more risky due to potential of network errors). That may be mitigated further by using local files instead of an in-memory buffer as it does now, but it is still susceptible to loss of content if something goes wrong while uploading a "huge" file to S3. Maybe with compression we can push the envelope a bit more.

So perhaps out of this are two features worth looking into:

• Using a file in the local file system instead of in-memory buffer. Slightly more cumbersome since now we have to worry about IO errors, but nothing is ever free, is it? :-)
• Compressing log data

from log4j-s3-search.