resurfaceio / logger-python Goto Github PK

View Code? Open in Web Editor NEW

22.0 5.0 6.0 2.9 MB

Log API calls with Python

License: Apache License 2.0

Python 99.76% Makefile 0.24%

http-logger django requests-module python aiohttp flask logger-python api-logger

logger-python's People

Contributors

Stargazers

Watchers

Forkers

d80ep08th claudiunicolaa vvixi vaishnavimudaliar noviicee hmercer105

logger-python's Issues

Custom fields

Add support for custom fields. This introduces breaking changes since the signature for the HttpMessage.send method must be modified to accept custom fields. The custom fields themselves should be added to the message after rules have been applied in HttpLogger.submit_if_passing. See the logger-go implementation.

Add custom User-Agent to POST requests

In the past, we didn't set a custom User-Agent on POST requests made by the logger to the database -- and we're seeing cases where these default User-Agents are getting blocked or misinterpreted. (Cloudflare is aggressive about blocking suspicious agents)

Please change BaseLogger.submit so that a User-Agent header is set like this:

Resurface/2.x.y (Python)

Here's an example of how this is done with the Java logger for comparison:

url_connection.setRequestProperty("User-Agent", "Resurface/" + version + " (Java)");

Please note that this does NOT affect the request_user_agent field in the database, or the "request_header:user-agent" details found in logger messages -- these are still intended to be the User-Agent from the original request/response.

Flask middleware breaking

Seeing this error with the test-flask app:

robfromboulder@radware3 test-flask % make ping
curl http://localhost/ping
<html>
  <head>
    <title>Internal Server Error</title>
  </head>
  <body>
    <h1><p>Internal Server Error</p></h1>
    
  </body>
</html>
robfromboulder@radware3 test-flask % make logs
[2021-04-21 03:59:38 +0000] [1] [INFO] Starting gunicorn 20.1.0
[2021-04-21 03:59:38 +0000] [1] [INFO] Listening at: http://0.0.0.0:5000 (1)
[2021-04-21 03:59:38 +0000] [1] [INFO] Using worker: sync
[2021-04-21 03:59:38 +0000] [9] [INFO] Booting worker with pid: 9
[2021-04-21 03:59:50 +0000] [9] [ERROR] Error handling request /ping
Traceback (most recent call last):
  File "/usr/local/lib/python3.7/site-packages/gunicorn/workers/sync.py", line 136, in handle
    self.handle_request(listener, req, client, addr)
  File "/usr/local/lib/python3.7/site-packages/gunicorn/workers/sync.py", line 179, in handle_request
    respiter = self.wsgi(environ, resp.start_response)
  File "/usr/local/lib/python3.7/site-packages/flask/app.py", line 2309, in __call__
    return self.wsgi_app(environ, start_response)
  File "/usr/local/lib/python3.7/site-packages/usagelogger/flask.py", line 58, in __call__
    body__ = self.request_body(environ)
  File "/usr/local/lib/python3.7/site-packages/usagelogger/flask.py", line 54, in request_body
    return body.decode("utf-8")
AttributeError: 'str' object has no attribute 'decode'
q^Cmake: *** [logs] Error 255

Middleware url/rules params should be optional

All middleware -- aiohttp, django, and requests -- should declare url and rules to be optional. If these are None, default values will be loaded by BaseLogger/UsageLoggers, without any other coding required.

Middleware should use the same signatures as HttpLogger:

    def __init__(
        url: Optional[str] = None,
        rules: Optional[str] = None,
    )

This will allow the url to be left out, as in the following example.

    app = web.Application(
        middlewares=[
            HttpLoggerForAIOHTTP(
                rules="include debug",
            )
        ]
    )

Add interval tracking for Django logger

he “interval” field is the approximate response time for the API call from the server side. This is an optional field that we’d like to have filled in by default for Django apps.

You can find examples of interval tracking in our Express and Rack loggers.

Change django.py to track intervals, like this:

def __call__(self, request):
    start = current_millis()
    response = self.get_response(request)
    now = current_millis()
    interval = now - start
    HttpMessage.send(self.logger, request=request, response=response, None, None, now, interval)
    return response

🏅 To complete this task, you should see values for the internal column in the database. The interval column will be NULL if this is not being provided properly.

Configure timeout for BaseLogger.submit

BaseLogger init should take a timeout argument that is set to a low value by default (5 sec).
This timeout value should be passed into self.conn.post.

Compression support

BaseLogger.submit should implement deflate compression, like the Node and Ruby loggers.

🏅There are standard tests to be enabled here for submitting with and without compression.

Background submission with dedicated thread

Send POST requests on a separate thread with bounded queue. Introduce BaseLogger.maxQueueDepth (with default value of 128) to control the depth of the bounded queue before the response is blocked.

BaseLogger.submit is where the handoff to the background thread should be done. (Rules engine and JSON conversion will continue to be done in the calling thread)

BaseLogger needs only a single background thread for performing POST requests since a single CPU core is normally capable of saturating the network.

copy_session_field rules not implemented

These are not required for a logger to work properly, but are helpful in specific cases.

Middleware support for tornado web framework

Middleware support for FastAPI web framework

Connection pooling

The current implementation of BaseLogger.submit opens and closes the network connection each time. Obviously this is very slow. Use urllib3 (or make an alternative recommendation) to implement connection pooling for both HTTP and HTTPS traffic.

🏅 To complete this task, all unit and integration tests should pass.

Flask middleware

Support monitoring APIs built on Flask, using a middleware plugin similar to Django.

Define GraphQL example application using Flask
Add Flask middleware class to logger package
Include documentation

Decode error on multipart/form-data request body

There seems to be an issue in all middleware with handling the multipart/form-data content-type.

Serverless middleware

Create middleware for Python lambdas on AWS and other serverless environments.

aiohttp middleware

Support monitoring APIs built on aiohttp (https://docs.aiohttp.org/en/stable/index.html), using a middleware plugin similar to Django.

Define GraphQL example application using aiohttp
Add aiohttp middleware class to logger package
Include documentation

Missing legal notices

I just realized that with some of the recent changes, we're missing a few entries in the NOTICES file.

Please add entries for mypy, flake8 and black libraries

Handle the RawPostDataException comming from the django middleware.

Django only allows reading POST body once. We cannot access the body after reading from the request's data stream. So try to find a workaround for this issue.

Requests client middleware

Research and create middleware that works from the client side, to record calls made to a remote API.

# standard session
s = requests.Session()
s.get('https://httpbin.org/cookies/set/sessioncookie/123456789')

# logging session
s = resurface.Session(url, rules)
s.get('https://httpbin.org/cookies/set/sessioncookie/123456789')

Client address

Log client IP address as a custom X-FORWARDED-FOR header. See similar issue over at logger-nodejs.

Restructure the project tree with better sub-packages and module management

Currently, all the python modules are in the main directory. We have to manage all those modules inside the sub-packages and update the setup.py to be able to find those sub-packages.

Phase out use of demo.resurface.io

demo.resurface.io is no longer available -- please use a local Pilot database (localhost:4001) for submission tests instead

Remove auto filtering by code & type

There is some filtering logic that predates our logging rules facility, and this old filtering can be safely removed.

In django.py, remove the status code filtering and content type check here:

def __call__(self, request):
    response = self.get_response(request)
    status = response.status_code
    if (status < 300 or status == 302) and HttpLogger.is_string_content_type(response['Content-Type']):
        HttpMessage.send(self.logger, request=request, response=response)
    return response

Next remove the HttpLogger.is_string_content_type and supporting variables (___STRING_TYPES and ___STRING_TYPES_REGEX). Then remove the related tests. You get the idea — any of this old logic that checks response codes or string types should be removed.

🏅 To complete this task, the integration test for Django should run to completion. This has a section where it checks whether 404 responses are captured.

pytest failed

Ran pytest on Windows 10, using Python 3.9.1, from the virtual environment indicated in the guidelines. It failed on 18% (1 of 96 tests):

============================= test session starts =============================
platform win32 -- Python 3.9.1, pytest-6.2.2, py-1.10.0, pluggy-0.13.1
rootdir: C:\Users\reimo\resurfaceio-logger-python, configfile: pyproject.toml
collected 96 items

tests/test_base_logger.py::test_creates_instance PASSED                  [  1%]
tests/test_base_logger.py::test_creates_multiple_instances PASSED        [  2%]
tests/test_base_logger.py::test_has_valid_host PASSED                    [  3%]
tests/test_base_logger.py::test_has_valid_version PASSED                 [  4%]
tests/test_base_logger.py::test_performs_enabling_when_expected PASSED   [  5%]
tests/test_base_logger.py::test_skips_enabling_for_invalid_urls PASSED   [  6%]
tests/test_base_logger.py::test_skips_enabling_for_missing_url PASSED    [  7%]
tests/test_base_logger.py::test_skips_enabling_for_undefined_url PASSED  [  8%]
tests/test_base_logger.py::test_submits_to_demo_url PASSED               [  9%]
tests/test_base_logger.py::test_submits_to_demo_url_via_http PASSED      [ 10%]
tests/test_base_logger.py::test_submits_to_demo_url_without_compression PASSED [ 11%]
tests/test_base_logger.py::test_submits_to_denied_url PASSED             [ 12%]
tests/test_base_logger.py::test_submits_to_queue PASSED                  [ 13%]
tests/test_base_logger.py::test_silently_ignores_unexpected_option_classes PASSED [ 14%]
tests/test_base_logger.py::test_uses_skip_options PASSED                 [ 15%]
tests/test_base_logger.py::test_good_json PASSED                         [ 16%]
tests/test_base_logger.py::test_invalid_json PASSED                      [ 17%]
tests/test_connection_pooling.py::TestBaseLogger::test_requests_keep_alive FAILED [ 18%]
tests/test_helper.py::test_good_json PASSED                              [ 19%]
tests/test_helper.py::test_invalid_json PASSED                           [ 20%]
tests/test_http_logger.py::test_creates_instance PASSED                  [ 21%]
tests/test_http_logger.py::test_creates_multiple_instances PASSED        [ 22%]
tests/test_http_logger.py::test_has_valid_agent PASSED                   [ 23%]
tests/test_http_logger.py::test_silently_ignores_unexpected_option_classes PASSED [ 25%]
tests/test_http_logger.py::test_good_json PASSED                         [ 26%]
tests/test_http_logger.py::test_invalid_json PASSED                      [ 27%]
tests/test_http_logger_rules.py::test_overrides_default_rules PASSED     [ 28%]
tests/test_http_logger_rules.py::test_uses_allow_http_url_rules PASSED   [ 29%]
tests/test_http_logger_rules.py::test_uses_copy_session_field_rules PASSED [ 30%]
tests/test_http_logger_rules.py::test_uses_copy_session_field_and_remove_rules PASSED [ 31%]
tests/test_http_logger_rules.py::test_uses_copy_session_field_and_stop_rules PASSED [ 32%]
tests/test_http_logger_rules.py::test_uses_remove_rules PASSED           [ 33%]
tests/test_http_logger_rules.py::test_uses_remove_if_rules PASSED        [ 34%]
tests/test_http_logger_rules.py::test_uses_remove_if_found_rules PASSED  [ 35%]
tests/test_http_logger_rules.py::test_uses_remove_unless_rules PASSED    [ 36%]
tests/test_http_logger_rules.py::test_uses_remove_unless_found_rules PASSED [ 37%]
tests/test_http_logger_rules.py::test_uses_replace_rules PASSED          [ 38%]
tests/test_http_logger_rules.py::test_uses_replace_rules_with_complex_expressions PASSED [ 39%]
tests/test_http_logger_rules.py::test_uses_sample_rules PASSED           [ 40%]
tests/test_http_logger_rules.py::test_uses_skip_compression_rules PASSED [ 41%]
tests/test_http_logger_rules.py::test_uses_skip_submission_rules PASSED  [ 42%]
tests/test_http_logger_rules.py::test_uses_stop_rules PASSED             [ 43%]
tests/test_http_logger_rules.py::test_uses_stop_if_rules PASSED          [ 44%]
tests/test_http_logger_rules.py::test_uses_stop_if_found_rules PASSED    [ 45%]
tests/test_http_logger_rules.py::test_uses_stop_unless_rules PASSED      [ 46%]
tests/test_http_logger_rules.py::test_uses_stop_unless_found_rules PASSED [ 47%]
tests/test_http_logger_rules.py::test_good_json PASSED                   [ 48%]
tests/test_http_logger_rules.py::test_invalid_json PASSED                [ 50%]
tests/test_http_message.py::test_formats_request PASSED                  [ 51%]
tests/test_http_message.py::test_formats_request_with_body PASSED        [ 52%]
tests/test_http_message.py::test_formats_request_with_empty_body PASSED  [ 53%]
tests/test_http_message.py::test_formats_request_with_missing_details PASSED [ 54%]
tests/test_http_message.py::test_formats_response PASSED                 [ 55%]
tests/test_http_message.py::test_formats_response_with_body PASSED       [ 56%]
tests/test_http_message.py::test_formats_response_with_empty_body PASSED [ 57%]
tests/test_http_message.py::test_formats_response_with_missing_details PASSED [ 58%]
tests/test_http_message.py::test_good_json PASSED                        [ 59%]
tests/test_http_message.py::test_invalid_json PASSED                     [ 60%]
tests/test_http_request_impl.py::test_body PASSED                        [ 61%]
tests/test_http_request_impl.py::test_headers PASSED                     [ 62%]
tests/test_http_request_impl.py::test_method PASSED                      [ 63%]
tests/test_http_request_impl.py::test_params PASSED                      [ 64%]
tests/test_http_request_impl.py::test_url PASSED                         [ 65%]
tests/test_http_request_impl.py::test_good_json PASSED                   [ 66%]
tests/test_http_request_impl.py::test_invalid_json PASSED                [ 67%]
tests/test_http_response_impl.py::test_body PASSED                       [ 68%]
tests/test_http_response_impl.py::test_headers PASSED                    [ 69%]
tests/test_http_response_impl.py::test_url PASSED                        [ 70%]
tests/test_http_response_impl.py::test_good_json PASSED                  [ 71%]
tests/test_http_response_impl.py::test_invalid_json PASSED               [ 72%]
tests/test_http_rules.py::test_changes_default_rules PASSED              [ 73%]
tests/test_http_rules.py::test_includes_debug_rules PASSED               [ 75%]
tests/test_http_rules.py::test_includes_standard_rules PASSED            [ 76%]
tests/test_http_rules.py::test_includes_strict_rules PASSED              [ 77%]
tests/test_http_rules.py::test_load_rules_from_file PASSED               [ 78%]
tests/test_http_rules.py::test_parses_empty_rules PASSED                 [ 79%]
tests/test_http_rules.py::test_parses_rules_with_bad_verbs PASSED        [ 80%]
tests/test_http_rules.py::test_parses_rules_with_invalid_scopes PASSED   [ 81%]
tests/test_http_rules.py::test_parses_allow_http_url_rules PASSED        [ 82%]
tests/test_http_rules.py::test_parses_copy_session_field_rules PASSED    [ 83%]
tests/test_http_rules.py::test_parses_remove_rules PASSED                [ 84%]
tests/test_http_rules.py::test_parses_remove_if_rules PASSED             [ 85%]
tests/test_http_rules.py::test_parses_remove_if_found_rules PASSED       [ 86%]
tests/test_http_rules.py::test_parses_remove_unless_rules PASSED         [ 87%]
tests/test_http_rules.py::test_parses_remove_unless_found_rules PASSED   [ 88%]
tests/test_http_rules.py::test_parses_replace_rules PASSED               [ 89%]
tests/test_http_rules.py::test_parses_sample_rules PASSED                [ 90%]
tests/test_http_rules.py::test_parses_skip_compression_rules PASSED      [ 91%]
tests/test_http_rules.py::test_parses_skip_submission_rules PASSED       [ 92%]
tests/test_http_rules.py::test_parses_stop_rules PASSED                  [ 93%]
tests/test_http_rules.py::test_parses_stop_if_rules PASSED               [ 94%]
tests/test_http_rules.py::test_parses_stop_if_found_rules PASSED         [ 95%]
tests/test_http_rules.py::test_parses_stop_unless_rules PASSED           [ 96%]
tests/test_http_rules.py::test_parses_stop_unless_found_rules PASSED     [ 97%]
tests/test_http_rules.py::test_raises_expected_errors PASSED             [ 98%]
tests/test_usage_loggers.py::test_provides_default_url PASSED            [100%]

================================== FAILURES ===================================
___________________ TestBaseLogger.test_requests_keep_alive ___________________

self = <test_connection_pooling.TestBaseLogger testMethod=test_requests_keep_alive>

    def tearDown(self) -> None:
>       os.remove("test_debug.log")
E       PermissionError: [WinError 32] The process cannot access the file because it is being used by another process: 'test_debug.log'

tests\test_connection_pooling.py:18: PermissionError
------------------------------ Captured log call ------------------------------
DEBUG    urllib3.connectionpool:connectionpool.py:971 Starting new HTTPS connection (1): demo.resurface.io:443
DEBUG    urllib3.connectionpool:connectionpool.py:452 https://demo.resurface.io:443 "POST /ping HTTP/1.1" 204 0
DEBUG    urllib3.connectionpool:connectionpool.py:452 https://demo.resurface.io:443 "POST /ping HTTP/1.1" 204 0
DEBUG    urllib3.connectionpool:connectionpool.py:452 https://demo.resurface.io:443 "POST /ping HTTP/1.1" 204 0
=========================== short test summary info ===========================
FAILED tests/test_connection_pooling.py::TestBaseLogger::test_requests_keep_alive
======================== 1 failed, 95 passed in 4.58s =========================

I tried testing tests/test_connection_pooling.py by itself and also failed, but this time by an AssertionError:

============================= test session starts =============================
platform win32 -- Python 3.9.1, pytest-6.2.2, py-1.10.0, pluggy-0.13.1
rootdir: C:\Users\reimo\resurfaceio-logger-python, configfile: pyproject.toml
collected 1 item

tests/test_connection_pooling.py::TestBaseLogger::test_requests_keep_alive FAILED [100%]
tests/test_connection_pooling.py::TestBaseLogger::test_requests_keep_alive ERROR [100%]

=================================== ERRORS ====================================
________ ERROR at teardown of TestBaseLogger.test_requests_keep_alive _________

self = <test_connection_pooling.TestBaseLogger testMethod=test_requests_keep_alive>

    def tearDown(self) -> None:
>       os.remove("test_debug.log")
E       PermissionError: [WinError 32] The process cannot access the file because it is being used by another process: 'test_debug.log'

tests\test_connection_pooling.py:18: PermissionError
------------------------------ Captured log call ------------------------------
DEBUG    urllib3.connectionpool:connectionpool.py:971 Starting new HTTPS connection (1): demo.resurface.io:443
DEBUG    urllib3.connectionpool:connectionpool.py:452 https://demo.resurface.io:443 "POST /ping HTTP/1.1" 204 0
DEBUG    urllib3.connectionpool:connectionpool.py:452 https://demo.resurface.io:443 "POST /ping HTTP/1.1" 204 0
DEBUG    urllib3.connectionpool:connectionpool.py:452 https://demo.resurface.io:443 "POST /ping HTTP/1.1" 204 0
================================== FAILURES ===================================
___________________ TestBaseLogger.test_requests_keep_alive ___________________

self = <test_connection_pooling.TestBaseLogger testMethod=test_requests_keep_alive>

    def test_requests_keep_alive(self) -> None:
        for _ in range(3):
            message: List[List[str]] = [
                ["agent", self.logger.agent],
                ["version", self.logger.version],
                ["now", "1455908640173"],
                ["prototol", "https"],
            ]
            msg = json.dumps(message, separators=(",", ":"))
            self.logger.submit(msg)
    
        with open("test_debug.log", "r") as f:
>           self.assertEqual(
                sum(
                    [
                        "new https connection (1): demo.resurface.io:443" in x.lower()
                        for x in f.readlines()
                    ]
                ),
                1 + 1,  # One for earlier test's http connection
            )
E           AssertionError: 1 != 2

tests\test_connection_pooling.py:32: AssertionError
------------------------------ Captured log call ------------------------------
DEBUG    urllib3.connectionpool:connectionpool.py:971 Starting new HTTPS connection (1): demo.resurface.io:443
DEBUG    urllib3.connectionpool:connectionpool.py:452 https://demo.resurface.io:443 "POST /ping HTTP/1.1" 204 0
DEBUG    urllib3.connectionpool:connectionpool.py:452 https://demo.resurface.io:443 "POST /ping HTTP/1.1" 204 0
DEBUG    urllib3.connectionpool:connectionpool.py:452 https://demo.resurface.io:443 "POST /ping HTTP/1.1" 204 0
=========================== short test summary info ===========================
FAILED tests/test_connection_pooling.py::TestBaseLogger::test_requests_keep_alive
ERROR tests/test_connection_pooling.py::TestBaseLogger::test_requests_keep_alive
========================= 1 failed, 1 error in 1.46s ==========================

The second one is expected since there was only one http connection when running this test alone. So I modified the test_connection_pooling.py file to assertEqual to 1. Ran pytest tests\test_connection_pooling.py again and got the same PermissionError when trying to remove the test_debug.log file:

============================= test session starts =============================
platform win32 -- Python 3.9.1, pytest-6.2.2, py-1.10.0, pluggy-0.13.1
rootdir: C:\Users\reimo\resurfaceio-logger-python, configfile: pyproject.toml
collected 1 item

tests/test_connection_pooling.py::TestBaseLogger::test_requests_keep_alive FAILED [100%]

================================== FAILURES ===================================
___________________ TestBaseLogger.test_requests_keep_alive ___________________

self = <test_connection_pooling.TestBaseLogger testMethod=test_requests_keep_alive>

    def tearDown(self) -> None:
>       os.remove("test_debug.log")
E       PermissionError: [WinError 32] The process cannot access the file because it is being used by another process: 'test_debug.log'

tests\test_connection_pooling.py:18: PermissionError
------------------------------ Captured log call ------------------------------
DEBUG    urllib3.connectionpool:connectionpool.py:971 Starting new HTTPS connection (1): demo.resurface.io:443
DEBUG    urllib3.connectionpool:connectionpool.py:452 https://demo.resurface.io:443 "POST /ping HTTP/1.1" 204 0
DEBUG    urllib3.connectionpool:connectionpool.py:452 https://demo.resurface.io:443 "POST /ping HTTP/1.1" 204 0
DEBUG    urllib3.connectionpool:connectionpool.py:452 https://demo.resurface.io:443 "POST /ping HTTP/1.1" 204 0
=========================== short test summary info ===========================
FAILED tests/test_connection_pooling.py::TestBaseLogger::test_requests_keep_alive
============================== 1 failed in 3.11s ==============================

Certainly, now the complete pytest will fail because of the additional AssertionError:

============================= test session starts =============================
platform win32 -- Python 3.9.1, pytest-6.2.2, py-1.10.0, pluggy-0.13.1
rootdir: C:\Users\reimo\resurfaceio-logger-python, configfile: pyproject.toml
collected 96 items

tests/test_base_logger.py::test_creates_instance PASSED                  [  1%]
tests/test_base_logger.py::test_creates_multiple_instances PASSED        [  2%]
tests/test_base_logger.py::test_has_valid_host PASSED                    [  3%]
tests/test_base_logger.py::test_has_valid_version PASSED                 [  4%]
tests/test_base_logger.py::test_performs_enabling_when_expected PASSED   [  5%]
tests/test_base_logger.py::test_skips_enabling_for_invalid_urls PASSED   [  6%]
tests/test_base_logger.py::test_skips_enabling_for_missing_url PASSED    [  7%]
tests/test_base_logger.py::test_skips_enabling_for_undefined_url PASSED  [  8%]
tests/test_base_logger.py::test_submits_to_demo_url PASSED               [  9%]
tests/test_base_logger.py::test_submits_to_demo_url_via_http PASSED      [ 10%]
tests/test_base_logger.py::test_submits_to_demo_url_without_compression PASSED [ 11%]
tests/test_base_logger.py::test_submits_to_denied_url PASSED             [ 12%]
tests/test_base_logger.py::test_submits_to_queue PASSED                  [ 13%]
tests/test_base_logger.py::test_silently_ignores_unexpected_option_classes PASSED [ 14%]
tests/test_base_logger.py::test_uses_skip_options PASSED                 [ 15%]
tests/test_base_logger.py::test_good_json PASSED                         [ 16%]
tests/test_base_logger.py::test_invalid_json PASSED                      [ 17%]
tests/test_connection_pooling.py::TestBaseLogger::test_requests_keep_alive FAILED [ 18%]
tests/test_connection_pooling.py::TestBaseLogger::test_requests_keep_alive ERROR [ 18%]
tests/test_helper.py::test_good_json PASSED                              [ 19%]
tests/test_helper.py::test_invalid_json PASSED                           [ 20%]
tests/test_http_logger.py::test_creates_instance PASSED                  [ 21%]
tests/test_http_logger.py::test_creates_multiple_instances PASSED        [ 22%]
tests/test_http_logger.py::test_has_valid_agent PASSED                   [ 23%]
tests/test_http_logger.py::test_silently_ignores_unexpected_option_classes PASSED [ 25%]
tests/test_http_logger.py::test_good_json PASSED                         [ 26%]
tests/test_http_logger.py::test_invalid_json PASSED                      [ 27%]
tests/test_http_logger_rules.py::test_overrides_default_rules PASSED     [ 28%]
tests/test_http_logger_rules.py::test_uses_allow_http_url_rules PASSED   [ 29%]
tests/test_http_logger_rules.py::test_uses_copy_session_field_rules PASSED [ 30%]
tests/test_http_logger_rules.py::test_uses_copy_session_field_and_remove_rules PASSED [ 31%]
tests/test_http_logger_rules.py::test_uses_copy_session_field_and_stop_rules PASSED [ 32%]
tests/test_http_logger_rules.py::test_uses_remove_rules PASSED           [ 33%]
tests/test_http_logger_rules.py::test_uses_remove_if_rules PASSED        [ 34%]
tests/test_http_logger_rules.py::test_uses_remove_if_found_rules PASSED  [ 35%]
tests/test_http_logger_rules.py::test_uses_remove_unless_rules PASSED    [ 36%]
tests/test_http_logger_rules.py::test_uses_remove_unless_found_rules PASSED [ 37%]
tests/test_http_logger_rules.py::test_uses_replace_rules PASSED          [ 38%]
tests/test_http_logger_rules.py::test_uses_replace_rules_with_complex_expressions PASSED [ 39%]
tests/test_http_logger_rules.py::test_uses_sample_rules PASSED           [ 40%]
tests/test_http_logger_rules.py::test_uses_skip_compression_rules PASSED [ 41%]
tests/test_http_logger_rules.py::test_uses_skip_submission_rules PASSED  [ 42%]
tests/test_http_logger_rules.py::test_uses_stop_rules PASSED             [ 43%]
tests/test_http_logger_rules.py::test_uses_stop_if_rules PASSED          [ 44%]
tests/test_http_logger_rules.py::test_uses_stop_if_found_rules PASSED    [ 45%]
tests/test_http_logger_rules.py::test_uses_stop_unless_rules PASSED      [ 46%]
tests/test_http_logger_rules.py::test_uses_stop_unless_found_rules PASSED [ 47%]
tests/test_http_logger_rules.py::test_good_json PASSED                   [ 48%]
tests/test_http_logger_rules.py::test_invalid_json PASSED                [ 50%]
tests/test_http_message.py::test_formats_request PASSED                  [ 51%]
tests/test_http_message.py::test_formats_request_with_body PASSED        [ 52%]
tests/test_http_message.py::test_formats_request_with_empty_body PASSED  [ 53%]
tests/test_http_message.py::test_formats_request_with_missing_details PASSED [ 54%]
tests/test_http_message.py::test_formats_response PASSED                 [ 55%]
tests/test_http_message.py::test_formats_response_with_body PASSED       [ 56%]
tests/test_http_message.py::test_formats_response_with_empty_body PASSED [ 57%]
tests/test_http_message.py::test_formats_response_with_missing_details PASSED [ 58%]
tests/test_http_message.py::test_good_json PASSED                        [ 59%]
tests/test_http_message.py::test_invalid_json PASSED                     [ 60%]
tests/test_http_request_impl.py::test_body PASSED                        [ 61%]
tests/test_http_request_impl.py::test_headers PASSED                     [ 62%]
tests/test_http_request_impl.py::test_method PASSED                      [ 63%]
tests/test_http_request_impl.py::test_params PASSED                      [ 64%]
tests/test_http_request_impl.py::test_url PASSED                         [ 65%]
tests/test_http_request_impl.py::test_good_json PASSED                   [ 66%]
tests/test_http_request_impl.py::test_invalid_json PASSED                [ 67%]
tests/test_http_response_impl.py::test_body PASSED                       [ 68%]
tests/test_http_response_impl.py::test_headers PASSED                    [ 69%]
tests/test_http_response_impl.py::test_url PASSED                        [ 70%]
tests/test_http_response_impl.py::test_good_json PASSED                  [ 71%]
tests/test_http_response_impl.py::test_invalid_json PASSED               [ 72%]
tests/test_http_rules.py::test_changes_default_rules PASSED              [ 73%]
tests/test_http_rules.py::test_includes_debug_rules PASSED               [ 75%]
tests/test_http_rules.py::test_includes_standard_rules PASSED            [ 76%]
tests/test_http_rules.py::test_includes_strict_rules PASSED              [ 77%]
tests/test_http_rules.py::test_load_rules_from_file PASSED               [ 78%]
tests/test_http_rules.py::test_parses_empty_rules PASSED                 [ 79%]
tests/test_http_rules.py::test_parses_rules_with_bad_verbs PASSED        [ 80%]
tests/test_http_rules.py::test_parses_rules_with_invalid_scopes PASSED   [ 81%]
tests/test_http_rules.py::test_parses_allow_http_url_rules PASSED        [ 82%]
tests/test_http_rules.py::test_parses_copy_session_field_rules PASSED    [ 83%]
tests/test_http_rules.py::test_parses_remove_rules PASSED                [ 84%]
tests/test_http_rules.py::test_parses_remove_if_rules PASSED             [ 85%]
tests/test_http_rules.py::test_parses_remove_if_found_rules PASSED       [ 86%]
tests/test_http_rules.py::test_parses_remove_unless_rules PASSED         [ 87%]
tests/test_http_rules.py::test_parses_remove_unless_found_rules PASSED   [ 88%]
tests/test_http_rules.py::test_parses_replace_rules PASSED               [ 89%]
tests/test_http_rules.py::test_parses_sample_rules PASSED                [ 90%]
tests/test_http_rules.py::test_parses_skip_compression_rules PASSED      [ 91%]
tests/test_http_rules.py::test_parses_skip_submission_rules PASSED       [ 92%]
tests/test_http_rules.py::test_parses_stop_rules PASSED                  [ 93%]
tests/test_http_rules.py::test_parses_stop_if_rules PASSED               [ 94%]
tests/test_http_rules.py::test_parses_stop_if_found_rules PASSED         [ 95%]
tests/test_http_rules.py::test_parses_stop_unless_rules PASSED           [ 96%]
tests/test_http_rules.py::test_parses_stop_unless_found_rules PASSED     [ 97%]
tests/test_http_rules.py::test_raises_expected_errors PASSED             [ 98%]
tests/test_usage_loggers.py::test_provides_default_url PASSED            [100%]

=================================== ERRORS ====================================
________ ERROR at teardown of TestBaseLogger.test_requests_keep_alive _________

self = <test_connection_pooling.TestBaseLogger testMethod=test_requests_keep_alive>

    def tearDown(self) -> None:
>       os.remove("test_debug.log")
E       PermissionError: [WinError 32] The process cannot access the file because it is being used by another process: 'test_debug.log'

tests\test_connection_pooling.py:18: PermissionError
------------------------------ Captured log call ------------------------------
DEBUG    urllib3.connectionpool:connectionpool.py:971 Starting new HTTPS connection (1): demo.resurface.io:443
DEBUG    urllib3.connectionpool:connectionpool.py:452 https://demo.resurface.io:443 "POST /ping HTTP/1.1" 204 0
DEBUG    urllib3.connectionpool:connectionpool.py:452 https://demo.resurface.io:443 "POST /ping HTTP/1.1" 204 0
DEBUG    urllib3.connectionpool:connectionpool.py:452 https://demo.resurface.io:443 "POST /ping HTTP/1.1" 204 0
================================== FAILURES ===================================
___________________ TestBaseLogger.test_requests_keep_alive ___________________

self = <test_connection_pooling.TestBaseLogger testMethod=test_requests_keep_alive>

    def test_requests_keep_alive(self) -> None:
        for _ in range(3):
            message: List[List[str]] = [
                ["agent", self.logger.agent],
                ["version", self.logger.version],
                ["now", "1455908640173"],
                ["prototol", "https"],
            ]
            msg = json.dumps(message, separators=(",", ":"))
            self.logger.submit(msg)
    
        with open("test_debug.log", "r") as f:
>           self.assertEqual(
                sum(
                    [
                        "new https connection (1): demo.resurface.io:443" in x.lower()
                        for x in f.readlines()
                    ]
                ),
                1,  # One for earlier test's http connection
            )
E           AssertionError: 2 != 1

tests\test_connection_pooling.py:32: AssertionError
------------------------------ Captured log call ------------------------------
DEBUG    urllib3.connectionpool:connectionpool.py:971 Starting new HTTPS connection (1): demo.resurface.io:443
DEBUG    urllib3.connectionpool:connectionpool.py:452 https://demo.resurface.io:443 "POST /ping HTTP/1.1" 204 0
DEBUG    urllib3.connectionpool:connectionpool.py:452 https://demo.resurface.io:443 "POST /ping HTTP/1.1" 204 0
DEBUG    urllib3.connectionpool:connectionpool.py:452 https://demo.resurface.io:443 "POST /ping HTTP/1.1" 204 0
=========================== short test summary info ===========================
FAILED tests/test_connection_pooling.py::TestBaseLogger::test_requests_keep_alive
ERROR tests/test_connection_pooling.py::TestBaseLogger::test_requests_keep_alive
==================== 1 failed, 95 passed, 1 error in 5.78s ====================

The issue then is the PermissionError when trying to remove the test_debug.log file. I verified that the context manager in test_connection_pooling.py was closing the file object (as it should), but now I'm not sure which process doesn't let the tearDown method remove the test_debug.log file

Recommend Projects

React

A declarative, efficient, and flexible JavaScript library for building user interfaces.
Vue.js

🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
Typescript

TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
TensorFlow

An Open Source Machine Learning Framework for Everyone
Django

The Web framework for perfectionists with deadlines.
Laravel

A PHP framework for web artisans
D3

Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

javascript

JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
web

Some thing interesting about web. New door for the world.
server

A server is a program made to process requests and deliver data to clients.
Machine learning

Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Visualization

Some thing interesting about visualization, use data art
Game

Some thing interesting about game, make everyone happy.

Recommend Org

Facebook

We are working to build community through open source technology. NB: members must have two-factor auth.
Microsoft

Open source projects and samples from Microsoft.
Google

Google ❤️ Open Source for everyone.
Alibaba

Alibaba Open Source for everyone
D3

Data-Driven Documents codes.
Tencent

China tencent open source team.