vovanec / supervisor_checks Goto Github PK

View Code? Open in Web Editor NEW

82.0 82.0 20.0 106 KB

Framework to build health checks for Supervisor-based services.

License: MIT License

Python 100.00%

supervisor_checks's People

Contributors

Stargazers

Watchers

supervisor_checks's Issues

HTTP 503 response bypasses num_retries

When num_retries is specified, regardless of whether the number of retries has been met, the service will be restarted if the http check gets a 503 back. The following log is from a check with -r 10

2024/04/23 22:39:06 [mysql5306_check] Received TICK_5 event from supervisor
2024/04/23 22:39:07 [mysql5306_check] No processes in state RUNNING found for process mysql5306
2024/04/23 22:39:11 [mysql5306_check] Received TICK_5 event from supervisor
2024/04/23 22:39:11 [mysql5306_check] Performing `http` check for process name mysql5306
2024/04/23 22:39:11 [mysql5306_check] HTTPCheck: Querying URL http://127.0.0.1:7035/check for process mysql5306
2024/04/23 22:39:11 [mysql5306_check] HTTPCheck: Status contacting URL http://127.0.0.1:7035/check for process mysql5306: 503 Service Unavailable
2024/04/23 22:39:11 [mysql5306_check] HTTPCheck: Check failed: Bad HTTP status code: 503
2024/04/23 22:39:11 [mysql5306_check] `http` check failed for process mysql5306. Trying to restart.
2024/04/23 22:39:11 [mysql5306_check] Trying to stop process mysql5306
2024/04/23 22:39:11 [mysql5306_check] Stopped process mysql5306
2024/04/23 22:39:11 [mysql5306_check] Starting process mysql5306
2024/04/23 22:39:15 [mysql5306_check] Received TICK_5 event from supervisor
2024/04/23 22:39:16 [mysql5306_check] No processes in state RUNNING found for process mysql5306
2024/04/23 22:39:21 [mysql5306_check] Received TICK_5 event from supervisor
2024/04/23 22:39:22 [mysql5306_check] Performing `http` check for process name mysql5306
2024/04/23 22:39:22 [mysql5306_check] HTTPCheck: Querying URL http://127.0.0.1:7035/check for process mysql5306
2024/04/23 22:39:22 [mysql5306_check] HTTPCheck: Status contacting URL http://127.0.0.1:7035/check for process mysql5306: 503 Service Unavailable
2024/04/23 22:39:22 [mysql5306_check] HTTPCheck: Check failed: Bad HTTP status code: 503
2024/04/23 22:39:22 [mysql5306_check] `http` check failed for process mysql5306. Trying to restart.
2024/04/23 22:39:22 [mysql5306_check] Trying to stop process mysql5306

My expectation here is that even with a 503 or connection refused response, the check would honor the retries.

run supervisor_tcp_check failure

[root@NESP-WEB ~]# /opt/deps/env/bin/supervisor_tcp_check -g hdfs-over-ftp -n hdfs-over-ftp -t 30 -r 3 -p 2222
Traceback (most recent call last):
File "/opt/deps/env/bin/supervisor_tcp_check", line 11, in
load_entry_point('supervisor-checks==0.7.0', 'console_scripts', 'supervisor_tcp_check')()
File "build/bdist.linux-x86_64/egg/supervisor_checks/bin/tcp_check.py", line 59, in main
File "build/bdist.linux-x86_64/egg/supervisor_checks/check_runner.py", line 60, in init
File "/opt/deps/env/lib/python2.7/site-packages/supervisor/childutils.py", line 17, in getRPCInterface
return xmlrpclib.ServerProxy('http://127.0.0.1', getRPCTransport(env))
File "/opt/deps/env/lib/python2.7/site-packages/supervisor/childutils.py", line 11, in getRPCTransport
return SupervisorTransport(u, p, env['SUPERVISOR_SERVER_URL'])
File "/opt/deps/env/lib/python2.7/UserDict.py", line 23, in getitem
raise KeyError(key)
KeyError: 'SUPERVISOR_SERVER_URL'

Deprecation warning due to invalid escape sequences

Deprecation warnings are raised due to invalid escape sequences. This can be fixed by using raw strings or escaping the literals. pyupgrade also helps in automatic conversion : https://github.com/asottile/pyupgrade/

find . -iname '*.py' | grep -Ev 'test.py' | xargs -P4 -I{} python3.8 -Wall -m py_compile {}
./supervisor_checks/bin/complex_check.py:3: DeprecationWarning: invalid escape sequence \/
  """Example configuration:

Illegal seek | KeyError: 'SUPERVISOR_SERVER_URL'

Using image python:3.10-slim
curling localhost:4180/ping in the container returns an OK

root@008a65d1d6b8:/# curl localhost:4180/ping
OK

oauth.sh works fine without running the event listener.

[program:oauth]
command                 = /opt/oauth-proxy/oauth.sh
user                    = application
process_name            = %(program_name)s

[eventlistener:oauth_check]
command=/usr/local/bin/supervisor_http_check -n oauth -u /ping -t 30 -r 3 -p 4180
events=TICK_60
stdout_logfile          = /dev/stdout
stderr_logfile          = /dev/stderr

container_instance  | Traceback (most recent call last):
container_instance  |   File "/usr/local/bin/supervisor_http_check", line 8, in <module>
container_instance  | 2022-02-13 18:45:05,447 CRIT uncaptured python exception, closing channel <POutputDispatcher at 139876582247632 for <Subprocess at 139876582250656 with name oauth_check in state STARTING> (stderr)> (<class 'OSError'>:[Errno 29] Illegal seek [/usr/local/lib/python3.10/site-packages/supervisor/supervisord.py|runforever|218] [/usr/local/lib/python3.10/site-packages/supervisor/dispatchers.py|handle_read_event|281] [/usr/local/lib/python3.10/site-packages/supervisor/dispatchers.py|record_output|215] [/usr/local/lib/python3.10/site-packages/supervisor/dispatchers.py|_log|184] [/usr/local/lib/python3.10/site-packages/supervisor/loggers.py|info|327] [/usr/local/lib/python3.10/site-packages/supervisor/loggers.py|log|345] [/usr/local/lib/python3.10/site-packages/supervisor/loggers.py|emit|227] [/usr/local/lib/python3.10/site-packages/supervisor/loggers.py|doRollover|264])
container_instance  | 2022-02-13 18:45:05,447 CRIT uncaptured python exception, closing channel <POutputDispatcher at 139876582247632 for <Subprocess at 139876582250656 with name oauth_check in state STARTING> (stderr)> (<class 'OSError'>:[Errno 29] Illegal seek [/usr/local/lib/python3.10/site-packages/supervisor/supervisord.py|runforever|218] [/usr/local/lib/python3.10/site-packages/supervisor/dispatchers.py|handle_read_event|281] [/usr/local/lib/python3.10/site-packages/supervisor/dispatchers.py|record_output|215] [/usr/local/lib/python3.10/site-packages/supervisor/dispatchers.py|_log|184] [/usr/local/lib/python3.10/site-packages/supervisor/loggers.py|info|327] [/usr/local/lib/python3.10/site-packages/supervisor/loggers.py|log|345] [/usr/local/lib/python3.10/site-packages/supervisor/loggers.py|emit|227] [/usr/local/lib/python3.10/site-packages/supervisor/loggers.py|doRollover|264])
container_instance  |     sys.exit(main())
container_instance  |   File "/usr/local/lib/python3.10/site-packages/supervisor_checks/bin/http_check.py", line 86, in main
container_instance  |     return check_runner.CheckRunner(
container_instance  |   File "/usr/local/lib/python3.10/site-packages/supervisor_checks/check_runner.py", line 62, in __init__
container_instance  |     self._rpc_client = childutils.getRPCInterface(self._environment)
container_instance  |   File "/usr/local/lib/python3.10/site-packages/supervisor/childutils.py", line 21, in getRPCInterface
container_instance  |     return xmlrpclib.ServerProxy('http://127.0.0.1', getRPCTransport(env))
container_instance  |   File "/usr/local/lib/python3.10/site-packages/supervisor/childutils.py", line 15, in getRPCTransport
container_instance  | 2022-02-13 18:45:05,447 CRIT uncaptured python exception, closing channel <POutputDispatcher at 139876582247632 for <Subprocess at 139876582250656 with name oauth_check in state STARTING> (stderr)> (<class 'OSError'>:[Errno 29] Illegal seek [/usr/local/lib/python3.10/site-packages/supervisor/supervisord.py|runforever|218] [/usr/local/lib/python3.10/site-packages/supervisor/dispatchers.py|handle_read_event|281] [/usr/local/lib/python3.10/site-packages/supervisor/dispatchers.py|record_output|215] [/usr/local/lib/python3.10/site-packages/supervisor/dispatchers.py|_log|184] [/usr/local/lib/python3.10/site-packages/supervisor/loggers.py|info|327] [/usr/local/lib/python3.10/site-packages/supervisor/loggers.py|log|345] [/usr/local/lib/python3.10/site-packages/supervisor/loggers.py|emit|227] [/usr/local/lib/python3.10/site-packages/supervisor/loggers.py|doRollover|264])
container_instance  | 2022-02-13 18:45:05,447 CRIT uncaptured python exception, closing channel <POutputDispatcher at 139876582247632 for <Subprocess at 139876582250656 with name oauth_check in state STARTING> (stderr)> (<class 'OSError'>:[Errno 29] Illegal seek [/usr/local/lib/python3.10/site-packages/supervisor/supervisord.py|runforever|218] [/usr/local/lib/python3.10/site-packages/supervisor/dispatchers.py|handle_read_event|281] [/usr/local/lib/python3.10/site-packages/supervisor/dispatchers.py|record_output|215] [/usr/local/lib/python3.10/site-packages/supervisor/dispatchers.py|_log|184] [/usr/local/lib/python3.10/site-packages/supervisor/loggers.py|info|327] [/usr/local/lib/python3.10/site-packages/supervisor/loggers.py|log|345] [/usr/local/lib/python3.10/site-packages/supervisor/loggers.py|emit|227] [/usr/local/lib/python3.10/site-packages/supervisor/loggers.py|doRollover|264])
container_instance  |     return SupervisorTransport(u, p, env['SUPERVISOR_SERVER_URL'])
container_instance  |   File "/usr/local/lib/python3.10/os.py", line 679, in __getitem__
container_instance  | 2022-02-13 18:45:05,447 CRIT uncaptured python exception, closing channel <POutputDispatcher at 139876582247632 for <Subprocess at 139876582250656 with name oauth_check in state STARTING> (stderr)> (<class 'OSError'>:[Errno 29] Illegal seek [/usr/local/lib/python3.10/site-packages/supervisor/supervisord.py|runforever|218] [/usr/local/lib/python3.10/site-packages/supervisor/dispatchers.py|handle_read_event|281] [/usr/local/lib/python3.10/site-packages/supervisor/dispatchers.py|record_output|215] [/usr/local/lib/python3.10/site-packages/supervisor/dispatchers.py|_log|184] [/usr/local/lib/python3.10/site-packages/supervisor/loggers.py|info|327] [/usr/local/lib/python3.10/site-packages/supervisor/loggers.py|log|345] [/usr/local/lib/python3.10/site-packages/supervisor/loggers.py|emit|227] [/usr/local/lib/python3.10/site-packages/supervisor/loggers.py|doRollover|264])
container_instance  | 2022-02-13 18:45:05,447 CRIT uncaptured python exception, closing channel <POutputDispatcher at 139876582247632 for <Subprocess at 139876582250656 with name oauth_check in state STARTING> (stderr)> (<class 'OSError'>:[Errno 29] Illegal seek [/usr/local/lib/python3.10/site-packages/supervisor/supervisord.py|runforever|218] [/usr/local/lib/python3.10/site-packages/supervisor/dispatchers.py|handle_read_event|281] [/usr/local/lib/python3.10/site-packages/supervisor/dispatchers.py|record_output|215] [/usr/local/lib/python3.10/site-packages/supervisor/dispatchers.py|_log|184] [/usr/local/lib/python3.10/site-packages/supervisor/loggers.py|info|327] [/usr/local/lib/python3.10/site-packages/supervisor/loggers.py|log|345] [/usr/local/lib/python3.10/site-packages/supervisor/loggers.py|emit|227] [/usr/local/lib/python3.10/site-packages/supervisor/loggers.py|doRollover|264])
container_instance  |     raise KeyError(key) from None
container_instance  | KeyError: 'SUPERVISOR_SERVER_URL'
container_instance  | 2022-02-13 18:45:05,448 CRIT uncaptured python exception, closing channel <POutputDispatcher at 139876582247632 for <Subprocess at 139876582250656 with name oauth_check in state STARTING> (stderr)> (<class 'OSError'>:[Errno 29] Illegal seek [/usr/local/lib/python3.10/site-packages/supervisor/supervisord.py|runforever|218] [/usr/local/lib/python3.10/site-packages/supervisor/dispatchers.py|handle_read_event|281] [/usr/local/lib/python3.10/site-packages/supervisor/dispatchers.py|record_output|215] [/usr/local/lib/python3.10/site-packages/supervisor/dispatchers.py|_log|184] [/usr/local/lib/python3.10/site-packages/supervisor/loggers.py|info|327] [/usr/local/lib/python3.10/site-packages/supervisor/loggers.py|log|345] [/usr/local/lib/python3.10/site-packages/supervisor/loggers.py|emit|227] [/usr/local/lib/python3.10/site-packages/supervisor/loggers.py|doRollover|264])
container_instance  | 2022-02-13 18:45:05,448 CRIT uncaptured python exception, closing channel <POutputDispatcher at 139876582247632 for <Subprocess at 139876582250656 with name oauth_check in state STARTING> (stderr)> (<class 'OSError'>:[Errno 29] Illegal seek [/usr/local/lib/python3.10/site-packages/supervisor/supervisord.py|runforever|218] [/usr/local/lib/python3.10/site-packages/supervisor/dispatchers.py|handle_read_event|281] [/usr/local/lib/python3.10/site-packages/supervisor/dispatchers.py|record_output|215] [/usr/local/lib/python3.10/site-packages/supervisor/dispatchers.py|_log|184] [/usr/local/lib/python3.10/site-packages/supervisor/loggers.py|info|327] [/usr/local/lib/python3.10/site-packages/supervisor/loggers.py|log|345] [/usr/local/lib/python3.10/site-packages/supervisor/loggers.py|emit|227] [/usr/local/lib/python3.10/site-packages/supervisor/loggers.py|doRollover|264])
container_instance  | 2022-02-13 18:45:05,470 INFO exited: oauth_check (exit status 1; not expected)
container_instance  | 2022-02-13 18:45:05,470 INFO exited: oauth_check (exit status 1; not expected)
container_instance  | 2022-02-13 18:45:06,470 INFO gave up: oauth_check entered FATAL state, too many start retries too quickly
container_instance  | 2022-02-13 18:45:06,470 INFO gave up: oauth_check entered FATAL state, too many start retries too quickly

No module named compat

compat.py is not included into the latest version of supervisor installed via pip causing checks to fail:

Traceback (most recent call last):
  File "/home/user/work/tst/bin/supervisor_http_check", line 7, in <module>
    from supervisor_checks.bin.http_check import main
  File "/home/user/work/tst/lib/python2.7/site-packages/supervisor_checks/bin/http_check.py", line 14, in <module>
    from supervisor_checks.check_modules import http
  File "/home/user/work/tst/lib/python2.7/site-packages/supervisor_checks/check_modules/http.py", line 4, in <module>
    from supervisor.compat import httplib
ImportError: No module named compat

Is it possible to not use this file so far for compatibility with current supervisor versions?

KeyError: 'SUPERVISOR_SERVER_URL'

I think the error is not quite esplicative ... can you help me understand what is happening here?

$ supervisor_memory_check -N garanteasy-mail-import -n test -m 102400
Traceback (most recent call last):
  File "/home/lrkwz/.local/bin/supervisor_memory_check", line 11, in <module>
    sys.exit(main())
  File "/home/lrkwz/.local/lib/python3.6/site-packages/supervisor_checks/bin/memory_check.py", line 55, in main
    args.check_name, args.process_group, args.process_name, checks_config).run()
  File "/home/lrkwz/.local/lib/python3.6/site-packages/supervisor_checks/check_runner.py", line 62, in __init__
    self._rpc_client = childutils.getRPCInterface(self._environment)
  File "/home/lrkwz/.local/lib/python3.6/site-packages/supervisor/childutils.py", line 21, in getRPCInterface
    return xmlrpclib.ServerProxy('http://127.0.0.1', getRPCTransport(env))
  File "/home/lrkwz/.local/lib/python3.6/site-packages/supervisor/childutils.py", line 15, in getRPCTransport
    return SupervisorTransport(u, p, env['SUPERVISOR_SERVER_URL'])
  File "/usr/lib/python3.6/os.py", line 669, in __getitem__
    raise KeyError(key) from None
KeyError: 'SUPERVISOR_SERVER_URL'

BTW supervisord is running on python2.x (default on ubuntu 18.04)

Recommend Projects

React

A declarative, efficient, and flexible JavaScript library for building user interfaces.
Vue.js

🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
Typescript

TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
TensorFlow

An Open Source Machine Learning Framework for Everyone
Django

The Web framework for perfectionists with deadlines.
Laravel

A PHP framework for web artisans
D3

Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

javascript

JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
web

Some thing interesting about web. New door for the world.
server

A server is a program made to process requests and deliver data to clients.
Machine learning

Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Visualization

Some thing interesting about visualization, use data art
Game

Some thing interesting about game, make everyone happy.

Recommend Org

Facebook

We are working to build community through open source technology. NB: members must have two-factor auth.
Microsoft

Open source projects and samples from Microsoft.
Google

Google ❤️ Open Source for everyone.
Alibaba

Alibaba Open Source for everyone
D3

Data-Driven Documents codes.
Tencent

China tencent open source team.