pklaus / wsgi-request-logger Goto Github PK

View Code? Open in Web Editor NEW

48.0 6.0 19.0 45 KB

A WSGI HTTP-Request Logger

Home Page: http://go.klaus.pw/wsgi-request-logger

License: BSD 3-Clause "New" or "Revised" License

Python 100.00%

wsgi-middleware python request-logging flask bottlepy

wsgi-request-logger's Introduction

Request Logging for WSGI Web Applications

This is a middleware which you can use to log requests to your WSGI based site. It's even imitating Apache's combined log format to allow you to use any of the many tools for Apache log file analysis.

By making use of Python's standard library logging facilities, you can easily set it up to log to STDOUT, time rotated log files, email, syslog, etc.

Installation

Simply install this Python module via

pip install wsgi-request-logger

Usage

To add this middleware to your WSGI application and log to the file access.log, do:

from requestlogger import WSGILogger, ApacheFormatter
from logging.handlers import TimedRotatingFileHandler

def application(environ, start_response):
    response_body = 'The request method was %s' % environ['REQUEST_METHOD']
    response_body = response_body.encode('utf-8')
    response_headers = [('Content-Type', 'text/plain'),
                        ('Content-Length', str(len(response_body)))]
    start_response('200 OK', response_headers)
    return [response_body]

handlers = [ TimedRotatingFileHandler('access.log', 'd', 7) , ]
loggingapp = WSGILogger(application, handlers, ApacheFormatter())

if __name__ == '__main__':
    from wsgiref.simple_server import make_server
    http = make_server('', 8080, loggingapp)
    http.serve_forever()

The Authors

This WSGI middleware was originally developed under the name wsgilog by L. C. Rees. It was forked by Philipp Klaus in 2013 to build a WSGI middleware for request logging rather than exception handling and logging.

License

This software, wsgi-request-logger, is published under a 3-clause BSD license.

Developers' Resources

To read about your options for the logging handler, you may want to read Python's Logging Cookbook.
Documentation on Apache's log format can be found here.
The WSGI - Web Server Gateway Interface - is defined in PEP 333 with an update for Python 3 in PEP 3333.

General References

PyPI's listing of wsgi-request-logger
The source code for this Python module is hosted on Github.
A blog post with more background information and usage examples: wsgi-request-logger – Logging HTTP Requests With Any WSGI Web Application like Flask, Bottle or Django

wsgi-request-logger's People

Contributors

Stargazers

Watchers

Forkers

lrowe nod martinrm77 rrva schlitzered luccasmenezes sahilagarwal321 alonisser marfire ivdelchev cnutshell nightmare-rg neumachen jonnywaffles sgml bepro-company tonysepia clncmbl eliadh

wsgi-request-logger's Issues

allow to set self.logger.propagate=False

hey,

first thank you for this small handy package.

second:

could you please add a parameter to WSGILogger called "propergate" that accepts True/False, and pass this parameter to self.logger.propagate?

with this i can disable the default logger, because i do not like to get spammed on stdout, when i have a file appender enabled.

Thank You,

Stephan

Escaping for user agent and referer

If the user agent or referer include double quotes ("), the log entries become unparsable because the quotes are not escaped and will look like
127.0.0.1 - - [16/Mar/2017:15:39:18 +0000] "GET / HTTP/1.1" 200 233 "ref" "erer" "user"agent"
Some simple backslash escaping (" and \) should be fine

The same issue might apply for requests, but i didn't check.

Keep a CHANGELOG // Upgrade Gracefully

Looks like you made a breaking change in the latest version that caused our builds to fail. It'd be nice if you could keep a changelog so that we could see what's changing, or if you could make changes happen more gracefully across versions.

Py3 Support

I get the following Traceback with py3

Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/waitress/channel.py", line 356, in service
    task.service()
  File "/usr/local/lib/python3.8/site-packages/waitress/task.py", line 172, in service
    self.execute()
  File "/usr/local/lib/python3.8/site-packages/waitress/task.py", line 440, in execute
    app_iter = self.channel.server.application(environ, start_response)
  File "/usr/local/lib/python3.8/site-packages/requestlogger/__init__.py", line 55, in __call__
    content_length = content_lengths[0] if content_lengths else len(b''.join(retval))
TypeError: sequence item 0: expected a bytes-like object, str found

NOT PRODUCTION READY - This project makes the application respond with empty responses on some cases

This makes the responses empty on some cases. I cannot give you the code to reproduce but this is not production ready.

ERR_CONTENT_LENGTH_MISMATCH

Using 'yield' in bottle gets an exception

Exception happened during processing of request from ('127.0.0.1', 50163)
Traceback (most recent call last):
File "D:\python35\lib\site-packages\paste\httpserver.py", line 1085, in process_request_in_thread
self.finish_request(request, client_address)
File "D:\python35\lib\socketserver.py", line 357, in finish_request
self.RequestHandlerClass(request, client_address, self)
File "D:\python35\lib\socketserver.py", line 684, in init
self.handle()
File "D:\python35\lib\site-packages\paste\httpserver.py", line 459, in handle
BaseHTTPRequestHandler.handle(self)
File "D:\python35\lib\http\server.py", line 415, in handle
self.handle_one_request()
File "D:\python35\lib\site-packages\paste\httpserver.py", line 454, in handle_one_request
self.wsgi_execute()
File "D:\python35\lib\site-packages\paste\httpserver.py", line 309, in wsgi_execute
self.wsgi_write_chunk('')
File "D:\python35\lib\site-packages\paste\httpserver.py", line 167, in wsgi_write_chunk
self.wfile.write(chunk)
File "D:\python35\lib\socket.py", line 589, in write
return self._sock.send(b)

TypeError: a bytes-like object is required, not 'str'

Sorry to bother you again, this error occurs once on the old repository "bottlelog" too, my source code contains "yield" instead of "return".

Then I made a try,

from bottle import get, run, Bottle

app = Bottle()

@app.get('/index')
def aa():
    return '''bla bla'''

@app.get('/index1')
def aa():
    yield '''bla bla'''
    return
# from requestlogger import WSGILogger, ApacheFormatter
# from logging.handlers import TimedRotatingFileHandler
# handlers = [ TimedRotatingFileHandler('access.log', 'd', 7) , ]
# app = WSGILogger(app, handlers, ApacheFormatter())

run(app, host='0.0.0.0', port=8081)

The same code, as long as I del the code contains requestlogger, 'index1' goes well, if retains them, 'index1' can not show anything .

Empty files when request logger enabled.

I'm serving dynamically generated files with my WSGI application and are streaming them to the client. When I apply the wsgi-request-logger middleware the files are empty. The reason is that when you are calculating the content type you are calling len on the retval which on a file-like object sets the internal pointer to the end of the file.

# file /__init__.py
51 content_length = content_lengths[0] if content_lengths else len(b''.join(retval))

A minimal example with flask, where the middleware breaks the application:

# file  /main.py
import logging
from flask import Flask, send_file
import requestlogger

app = Flask(__name__)

handler = logging.StreamHandler()
app.wsgi_app = requestlogger.WSGILogger(app.wsgi_app, [handler], requestlogger.ApacheFormatter())

@app.route('/file', methods=['GET'])
def serve_file():
    f = open('data', 'r')

    return send_file(f, mimetype="application/ocet-stream")

if __name__ == '__main__':
    app.run(port=4500, debug=True)

#file /data
Some data...

This application will serves an empty file. Commenting out the middleware serves the file properly.

ZERO not defined in timehacks

I was using wsgi-request-logger in an environment with the timezone (/etc/timezone) set to 'etc/UTC', then the timehacks.LocalTimezone.dst() may tried to return ZERO.

ZERO is not defined.

I'm not actually sure what conditions cause this dst() to try to return ZERO, but I suspect that not defining ZERO is only minor oversight ;)

Python3, Ubuntu 13.04, virtualenv

Request time accuracy to low

The times for my request are not correct. I have many request that are being log as taking no time. I see that the timer is using time.clock(). Apparently it's accuracy is not very good.

http://stackoverflow.com/questions/85451/python-time-clock-vs-time-time-accuracy

time.time() would be a better method to use.

wsgi-request-logger fails to pip install if pypandoc is already installed

I ran into an issue whereby I could not get wsgi-request-logger to install if pypandoc was already installed in my virtual environment (python 3.5).

Expected behavior

The following should work:

$ pip install pypandoc
$ pip install wsgi-request-logger

Actual behavior

Executing the above results in the following error:

Collecting wsgi-request-logger
  Using cached wsgi-request-logger-0.4.5.tar.gz
    Complete output from command python setup.py egg_info:
    Traceback (most recent call last):
      File "<string>", line 1, in <module>
      File "/private/var/folders/4m/k137dqwd7hzftnctvnys5fxmkk84cx/T/pip-build-tqs7ddm_/wsgi-request-logger/setup.py", line 13, in <module>
        LDESC = pypandoc.convert('README.md', 'rst')
      File "<VIRTUALENV_ROOT>/lib/python3.5/site-packages/pypandoc/__init__.py", line 66, in convert
        raise RuntimeError("Format missing, but need one (identified source as text as no "
    RuntimeError: Format missing, but need one (identified source as text as no file with that name was found).

Possible fixes

Check for the existence of README.md before attempting to use pypandoc
Modify setup.py to catch a RuntimeError from the pypandoc block

HTTP 5xx responses not logged to access log

It seems that 5xx responses are not logged to access log.

Example below does not log a request at all, when the server responded with HTTP 500:

from requestlogger import WSGILogger, ApacheFormatter
from logging.handlers import TimedRotatingFileHandler
from wsgiref.simple_server import make_server

def hello_world_app(environ, start_response):
    raise Exception("broken")

handlers = [ TimedRotatingFileHandler('access.log', 'd', 7) , ]
loggingapp = WSGILogger(hello_world_app, handlers, ApacheFormatter())

if __name__ == '__main__':
    port = 8000
    from wsgiref.simple_server import make_server
    http = make_server('', port, loggingapp)
    print "Listening on %s" % port
    http.serve_forever()

REQUEST_URI instead of PATH_INFO

environ.get('PATH_INFO', ''),
should use REQUEST_URI.

path_info does not include the query string, which is normally included in NCSA logs like apache
applications like werkzeug.wsgi.DispatcherMiddleware modify PATH_INFO and will lead to wrong logged values