omrikiei / aws_logging_handlers Goto Github PK
View Code? Open in Web Editor NEWAWS integration for python logging handlers(S3, Kinesis)
Home Page: https://omrikiei.github.io/aws_logging_handlers/
License: MIT License
AWS integration for python logging handlers(S3, Kinesis)
Home Page: https://omrikiei.github.io/aws_logging_handlers/
License: MIT License
import logging
from aws_logging_handlers.S3 import S3Handler
import boto3
s3 = boto3.client("s3")
bucket="sing-dev-chumbucket1" # The bucket should already exist
s3_handler = S3Handler("error/output/", bucket, workers=3)
formatter = logging.Formatter('[%(asctime)s] %(filename)s:%(lineno)d} %(levelname)s - %(message)s')
s3_handler.setFormatter(formatter)
logger = logging.getLogger('root')
logger.setLevel(logging.INFO)
logger.addHandler(s3_handler)
for i in range(0, 100):
logger.info("test info message")
logger.warning("test warning message")
logger.error("test error message")
logging.shutdown()
getting an error
File "C:/Users/Administrator/PycharmProjects/baselineloader/venv/Scripts/testarg.py", line 7, in <module>
s3_handler = S3Handler("error/output/", bucket, workers=3)
File "C:\Users\Administrator\PycharmProjects\baselineloader\venv\lib\site-packages\aws_logging_handlers\S3\__init__.py", line 329, in __init__
self.stream = S3Stream(self.bucket, key, chunk_size=chunk_size, max_file_log_time=time_rotation,
TypeError: ABCMeta object got multiple values for keyword argument 'workers'
kindly help me .
Hi,
When building KinesisHandler
/S3Handler
, it asks for a key_id
, a secret
and a region
.
However, boto3
takes that automatically from .aws/credentials
and $AWS_DEFAULT_REGION
I suggest to make these parameters optionals for the simplicity of the users (no need to read the file and read the environment ourselves).
Allow initialization of the relevant parameters:
DEFAULT_CHUNK_SIZE = 5 * 1024 ** 2 # 5 MB
DEFAULT_ROTATION_TIME_SECS = 12 * 60 * 60 # 12 hours
MAX_FILE_SIZE_BYTES = 100 * 1024 ** 2 # 100 MB
Hello and thanks for the package. I get an error when I run the example code in README.
I am using Python 3.6.7.
import logging
from aws_logging_handlers.S3 import S3Handler
bucket="test_bucket" # The bucket should already exist
s3_handler = S3Handler("test_log", bucket)
I get:
Traceback (most recent call last):
File "", line 1, in
File "C:\Users\username\python_venvs\env_name\lib\site-packages\aws_logging_handlers\S3_init_.py", line 331, in init
compress=compress, **boto_session_kwargs)
File "C:\Users\username\python_venvs\env_name\lib\site-packages\aws_logging_handlers\S3_init_.py", line 98, in init
self.start_time = int(datetime.utcnow().strftime('%s'))
ValueError: Invalid format string
I tried to replace strftime('%s') timestamp() , see:
https://stackoverflow.com/questions/11743019/convert-python-datetime-to-epoch-with-strftime
But then I got another error:
Traceback (most recent call last):
File "", line 1, in
File "C:\Users\username\python_venvs\env_name\lib\site-packages\aws_logging_handlers\S3_init_.py", line 336, in init
signal.signal(signal.SIGQUIT, self._teardown)
AttributeError: module 'signal' has no attribute 'SIGQUIT'
partition=
can be specified when creating a KinesisHandler
, however it accepts only a string.
The implementation can differ and we may want to generate a random unique number to be sure the logs are spread across multiple shards.
To let the implementation's developer decide, I suggest to change the argument's type to be also a function.
e.g. pseudo code
# my_app.py changes:
k = KinesisHandler(.., partition=uuid4.uuid)
# aws_logging_handlers/KinesisStreamHandler.py changes:
if callable(self.partition):
..put_record(.., PartitionKey=self.partition())
else:
..put_record(.., PartitionKey=self.partition)
using FROM python:3.8-slim-buster
docker and latest aws-logging-handlers>=2.0.3
segfaults with;
File "/usr/local/lib/python3.8/site-packages/boto3/utils.py", line 52 in import_module
File "/usr/local/lib/python3.8/site-packages/boto3/utils.py", line 61 in _handler
File "/usr/local/lib/python3.8/site-packages/botocore/hooks.py", line 211 in _emit
File "/usr/local/lib/python3.8/site-packages/botocore/hooks.py", line 228 in emit
File "/usr/local/lib/python3.8/site-packages/botocore/hooks.py", line 356 in emit
File "/usr/local/lib/python3.8/site-packages/botocore/client.py", line 106 in _create_client_class
File "/usr/local/lib/python3.8/site-packages/botocore/client.py", line 79 in create_client
File "/usr/local/lib/python3.8/site-packages/botocore/session.py", line 831 in create_client
File "/usr/local/lib/python3.8/site-packages/boto3/session.py", line 258 in client
File "/usr/local/lib/python3.8/site-packages/boto3/session.py", line 384 in resource
File "/usr/local/lib/python3.8/site-packages/aws_logging_handlers/S3/__init__.py", line 97 in __init__
File "/usr/local/lib/python3.8/site-packages/aws_logging_handlers/S3/__init__.py", line 329 in __init__
To see the verbose output run with flags python -d -X dev app.py
script runs fine on the laptop running;
python3 --version
Python 3.8.2
uname -a
Linux 5.4.0-33-generic #37-Ubuntu SMP Thu May 21 12:53:59 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux
Writing to a buffer is cool and nice, however sometime we want to write into Kinesis the quickest way as possible.
For instance, if we write crit
and those are plugged into a Kinesis
stream to be shown in a operator's dashboard, we may don't want to wait.
Any ideas about that ?
Solutions that I have in mind:
Please change
from aws_logging_handlers import S3Handler --> from aws_logging_handlers.S3 import S3Handler
from aws_logging_handlers import KinesisHandler --> from aws_logging_handlers.Kinesis import KinesisHandler
Also, even though it uploads logs to S3 bucket, the whole program (in readme) hangs!
Using this handler prevents python from exiting - tested on 3.7.3
and 3.6.8
This could be caused by the teardown signal handler for the S3 logging handler, which is wrong. The documentation says that the signal handlers take 2 arguments
The handler is called with two arguments: the signal number and the current stack frame (None or a frame object; for a description of frame objects, see the description in the type hierarchy or see the attribute descriptions in the inspect module).
the _teardown
function takes a single arg.
I tried fixing that locally but the exit still hangs.
Closing the log handler explicitly before exit
solves the issue but this really shouldn't be necessary.
Hitting ctrl+c
once the program is frozen gives the following traceback
Traceback (most recent call last):
File "/usr/local/lib/python3.7/threading.py", line 1281, in _shutdown
t.join()
File "/usr/local/lib/python3.7/threading.py", line 1032, in join
self._wait_for_tstate_lock()
File "/usr/local/lib/python3.7/threading.py", line 1048, in _wait_for_tstate_lock
elif lock.acquire(block, timeout):
KeyboardInterrupt```
I got this error when logging inside a dask worker: signal only works in main thread
Could you make signals optional to allow loggin in dask workers?
Hi,
I'm dealing with a situation that this library does not cover. I want to structure my logging using Hive partition on S3 to be consumed by other applications. So I need a way to build the S3 file path dynamically like that:
def get_dynamic_path(date):
# Will return /year=2019/month=06/day=19/20190619.json
return date.strftime('/year=%Y/month=%m/day=%d/%s')
s3_handler = S3Handler(dynamic_filename=get_dynamic_path)
Would be nice if I could pass my own function to S3Handler
to build dynamically the path.
`import logging
from aws_logging_handlers.S3 import S3Handler
from aws_logging_handlers.Kinesis import KinesisHandler
bucket = "services-s3-log"
s3_handler = S3Handler("test_log/log.log", bucket, workers=1)
formatter = logging.Formatter('[%(asctime)s] %(filename)s:%(lineno)d} %(levelname)s - %(message)s')
s3_handler.setFormatter(formatter)
logger = logging.getLogger('root')
logger.setLevel(logging.INFO)
logger.addHandler(s3_handler)
for i in range(0, 100000):
logger.info("test info message")
logger.warning("test warning message")
logger.error("test error message")
logging.shutdown()
version: 2.0.5
I hit the following error:
self = <aws_logging_handlers.S3.StreamObject object at 0x7f3dc744fdf0>, s3_resource = s3.ServiceResource(), bucket_name = 'batch-detect-bucket', filename = 'sid=source1/module/function-o365-detection_1606755959', buffer_queue = <queue.Queue object at 0x7f3de595e9d0>, encryption = None
def __init__(self, s3_resource, bucket_name, filename, buffer_queue, encryption):
self.object = s3_resource.Object(bucket_name, filename)
> self.uploader = self.object.initiate_multipart_upload(**encryption)
E TypeError: initiate_multipart_upload() argument after ** must be a mapping, not NoneType
../local/lib/python3.8/site-packages/aws_logging_handlers/S3/__init__.py:34: TypeError
I think I've narrowed down the change that causes this issue.
1ad2c42#diff-915075834d3035bf217f6a9dedfa0afd57882e221e9ca41b6c44b4e88281ddd1
The issue is encryption_options is default initialized as None
. Which can't be used with **.
I don't have this issue with version 2.0.3.
s3_handler = S3Handler("logs", bucket, workers=3)
Traceback (most recent call last):
File "", line 1, in
File "C:\Users\admin\Documents\crash_bandicoot\api.localgovai\venv\Lib\site-packages\aws_logging_handlers\S3_init_.py", line 329, in init
self.stream = S3Stream(self.bucket, key, chunk_size=chunk_size, max_file_log_time=time_rotation,
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\admin\Documents\crash_bandicoot\api.localgovai\venv\Lib\site-packages\aws_logging_handlers\S3_init_.py", line 97, in init
self.start_time = int(datetime.utcnow().strftime('%s'))
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ValueError: Invalid format string
How can I provide the access_key_id and secret_access_key for the logger?
Add support to asyncio so that initialization can determine wether to use the event loop or threads
I cannot see my logs before handler shutdown
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.