Comments (6)
balbirthomas
commented on September 23, 2024
Hi @Abuelodelanada I have repeated local run of these (still work in progress) integration tests about 10 times today and a similar number of times on Github previously. I was not able to reproduce the above failure. One thing to try is to replace the following line in src/charm.py in the Prometheus repository
self.unit.status = BlockedStatus("Failed to load Prometheus config")
with
container.restart(self._name)
Let me know (with a comment here) if that fixes the issue for you. I suspect the problem you are seeing may be related to this issue. If so then I see no option other than accepting the penalty of sometimes restarting Prometheus rather than always reloading config if service layer definitions have not changed. ATTN: @mmanciop @simskij .
from prometheus-k8s-operator.
Abuelodelanada
commented on September 23, 2024
With this change:
if current_services == new_layer.services:
reloaded = self._prometheus_server.reload_configuration()
if not reloaded:
- self.unit.status = BlockedStatus("Failed to load Prometheus config")
+ # self.unit.status = BlockedStatus("Failed to load Prometheus config")
+ container.restart(self._name)
else:
self.unit.status = ActiveStatus()
logger.info("Prometheus configuration reloaded")and I think, I get the same issue:
➜ prometheus-operator git:(main) ✗ tox -e integration
integration installed: attrs==21.2.0,backcall==0.2.0,bcrypt==3.2.0,cachetools==4.2.4,certifi==2021.10.8,cffi==1.15.0,charset-normalizer==2.0.7,cryptography==35.0.0,decorator==5.1.0,google-auth==2.3.0,idna==3.3,iniconfig==1.1.1,ipdb==0.13.9,ipython==7.28.0,jedi==0.18.0,Jinja2==3.0.2,juju==2.9.4,jujubundlelib==0.5.6,kubernetes==18.20.0,macaroonbakery==1.3.1,MarkupSafe==2.0.1,matplotlib-inline==0.1.3,mypy-extensions==0.4.3,oauthlib==3.1.1,packaging==21.0,paramiko==2.8.0,parso==0.8.2,pexpect==4.8.0,pickleshare==0.7.5,pluggy==1.0.0,prompt-toolkit==3.0.21,protobuf==3.19.0,ptyprocess==0.7.0,py==1.10.0,pyasn1==0.4.8,pyasn1-modules==0.2.8,pycparser==2.20,Pygments==2.10.0,pymacaroons==0.13.0,PyNaCl==1.4.0,pyparsing==2.4.7,pyRFC3339==1.1,pytest==6.2.5,pytest-asyncio==0.16.0,pytest-operator==0.8.4,python-dateutil==2.8.2,pytz==2021.3,PyYAML==6.0,requests==2.26.0,requests-oauthlib==1.3.0,rsa==4.7.2,six==1.16.0,theblues==0.5.2,toml==0.10.2,toposort==1.7,traitlets==5.1.0,typing-extensions==3.10.0.2,typing-inspect==0.7.1,urllib3==1.26.7,wcwidth==0.2.5,websocket-client==1.2.1,websockets==7.0
integration run-test-pre: PYTHONHASHSEED='712029368'
integration run-test: commands[0] | pytest -v --tb native --log-cli-level=INFO -s /home/jose/trabajos/canonical/repos/prometheus-operator/tests/integration
====================================================================================== test session starts ======================================================================================
platform linux -- Python 3.8.10, pytest-6.2.5, py-1.10.0, pluggy-1.0.0 -- /home/jose/trabajos/canonical/repos/prometheus-operator/.tox/integration/bin/python
cachedir: .tox/integration/.pytest_cache
rootdir: /home/jose/trabajos/canonical/repos/prometheus-operator
plugins: asyncio-0.16.0, operator-0.8.4
collected 1 item
tests/integration/test_charm.py::test_build_and_deploy_with_ubuntu_image /snap/bin/juju
/snap/bin/charmcraft
---------------------------------------------------------------------------------------- live log setup -----------------------------------------------------------------------------------------
INFO pytest_operator.plugin:plugin.py:160 Using tmp_path: /home/jose/trabajos/canonical/repos/prometheus-operator/.tox/integration/tmp/pytest/test-charm-vd5l0
INFO pytest_operator.plugin:plugin.py:222 Adding model mk8s:test-charm-vd5l
WARNING juju.client.connection:connection.py:729 unknown facade CAASModelConfigManager
WARNING juju.client.connection:connection.py:753 unexpected facade CAASModelConfigManager found, unable to decipher version to use
WARNING juju.client.connection:connection.py:729 unknown facade RaftLease
WARNING juju.client.connection:connection.py:753 unexpected facade RaftLease found, unable to decipher version to use
WARNING juju.client.connection:connection.py:729 unknown facade Secrets
WARNING juju.client.connection:connection.py:753 unexpected facade Secrets found, unable to decipher version to use
WARNING juju.client.connection:connection.py:729 unknown facade SecretsManager
WARNING juju.client.connection:connection.py:753 unexpected facade SecretsManager found, unable to decipher version to use
WARNING juju.client.connection:connection.py:729 unknown facade SecretsRotationWatcher
WARNING juju.client.connection:connection.py:753 unexpected facade SecretsRotationWatcher found, unable to decipher version to use
----------------------------------------------------------------------------------------- live log call -----------------------------------------------------------------------------------------
INFO pytest_operator.plugin:plugin.py:338 Building charm prometheus-k8s
INFO juju.model:model.py:1873 Deploying local:focal/prometheus-k8s-0
INFO juju.model:model.py:2627 Waiting for model:
prometheus/0 [allocating] waiting: installing agent
FAILED
--------------------------------------------------------------------------------------- live log teardown ---------------------------------------------------------------------------------------
INFO pytest_operator.plugin:plugin.py:260 Model status:
Unit Machine Status Message
prometheus/0* no-machine error hook failed: "config-changed"
Machine Series Status
INFO pytest_operator.plugin:plugin.py:280 Juju error logs:
controller-0: 10:55:38 ERROR juju.worker.caasapplicationprovisioner.runner exited "prometheus": Operation cannot be fulfilled on pods "prometheus-0": the object has been modified; please apply your changes to the latest version and try again
unit-prometheus-0: 10:55:47 ERROR unit.prometheus/0.juju-log Uncaught exception while in charm code:
Traceback (most recent call last):
File "/var/lib/juju/agents/unit-prometheus-0/charm/venv/ops/pebble.py", line 1103, in _request_raw
response = self.opener.open(request, timeout=self.timeout)
File "/usr/lib/python3.8/urllib/request.py", line 531, in open
response = meth(req, response)
File "/usr/lib/python3.8/urllib/request.py", line 640, in http_response
response = self.parent.error(
File "/usr/lib/python3.8/urllib/request.py", line 569, in error
return self._call_chain(*args)
File "/usr/lib/python3.8/urllib/request.py", line 502, in _call_chain
result = func(*args)
File "/usr/lib/python3.8/urllib/request.py", line 649, in http_error_default
raise HTTPError(req.full_url, code, msg, hdrs, fp)
urllib.error.HTTPError: HTTP Error 400: Bad Request
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/var/lib/juju/agents/unit-prometheus-0/charm/venv/ops/model.py", line 1133, in restart
self._pebble.restart_services(service_names)
File "/var/lib/juju/agents/unit-prometheus-0/charm/venv/ops/pebble.py", line 1250, in restart_services
return self._services_action('restart', services, timeout, delay)
File "/var/lib/juju/agents/unit-prometheus-0/charm/venv/ops/pebble.py", line 1263, in _services_action
resp = self._request('POST', '/v1/services', body=body)
File "/var/lib/juju/agents/unit-prometheus-0/charm/venv/ops/pebble.py", line 1074, in _request
response = self._request_raw(method, path, query, headers, data)
File "/var/lib/juju/agents/unit-prometheus-0/charm/venv/ops/pebble.py", line 1114, in _request_raw
raise APIError(body, code, status, message)
ops.pebble.APIError: action "restart" is unsupported
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "./src/charm.py", line 312, in <module>
main(PrometheusCharm)
File "/var/lib/juju/agents/unit-prometheus-0/charm/venv/ops/main.py", line 408, in main
_emit_charm_event(charm, dispatcher.event_name)
File "/var/lib/juju/agents/unit-prometheus-0/charm/venv/ops/main.py", line 142, in _emit_charm_event
event_to_emit.emit(*args, **kwargs)
File "/var/lib/juju/agents/unit-prometheus-0/charm/venv/ops/framework.py", line 275, in emit
framework._emit(event)
File "/var/lib/juju/agents/unit-prometheus-0/charm/venv/ops/framework.py", line 735, in _emit
self._reemit(event_path)
File "/var/lib/juju/agents/unit-prometheus-0/charm/venv/ops/framework.py", line 782, in _reemit
custom_handler(event)
File "./src/charm.py", line 121, in _configure
container.restart(self._name)
File "/var/lib/juju/agents/unit-prometheus-0/charm/venv/ops/model.py", line 1138, in restart
for svc in self.get_services(service_names):
File "/var/lib/juju/agents/unit-prometheus-0/charm/venv/ops/model.py", line 1179, in get_services
services = self._pebble.get_services(service_names)
File "/var/lib/juju/agents/unit-prometheus-0/charm/venv/ops/pebble.py", line 1398, in get_services
query = {'names': ','.join(names)}
TypeError: sequence item 0: expected str instance, tuple found
unit-prometheus-0: 10:55:48 ERROR juju.worker.uniter.operation hook "prometheus-pebble-ready" (via hook dispatching script: dispatch) failed: exit status 1
unit-prometheus-0: 10:55:48 ERROR juju.worker.uniter pebble poll failed for container "prometheus": hook failed
unit-prometheus-0: 10:55:48 ERROR unit.prometheus/0.juju-log Uncaught exception while in charm code:
Traceback (most recent call last):
File "/var/lib/juju/agents/unit-prometheus-0/charm/venv/ops/pebble.py", line 1103, in _request_raw
response = self.opener.open(request, timeout=self.timeout)
File "/usr/lib/python3.8/urllib/request.py", line 531, in open
response = meth(req, response)
File "/usr/lib/python3.8/urllib/request.py", line 640, in http_response
response = self.parent.error(
File "/usr/lib/python3.8/urllib/request.py", line 569, in error
return self._call_chain(*args)
File "/usr/lib/python3.8/urllib/request.py", line 502, in _call_chain
result = func(*args)
File "/usr/lib/python3.8/urllib/request.py", line 649, in http_error_default
raise HTTPError(req.full_url, code, msg, hdrs, fp)
urllib.error.HTTPError: HTTP Error 400: Bad Request
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/var/lib/juju/agents/unit-prometheus-0/charm/venv/ops/model.py", line 1133, in restart
self._pebble.restart_services(service_names)
File "/var/lib/juju/agents/unit-prometheus-0/charm/venv/ops/pebble.py", line 1250, in restart_services
return self._services_action('restart', services, timeout, delay)
File "/var/lib/juju/agents/unit-prometheus-0/charm/venv/ops/pebble.py", line 1263, in _services_action
resp = self._request('POST', '/v1/services', body=body)
File "/var/lib/juju/agents/unit-prometheus-0/charm/venv/ops/pebble.py", line 1074, in _request
response = self._request_raw(method, path, query, headers, data)
File "/var/lib/juju/agents/unit-prometheus-0/charm/venv/ops/pebble.py", line 1114, in _request_raw
raise APIError(body, code, status, message)
ops.pebble.APIError: action "restart" is unsupported
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "./src/charm.py", line 312, in <module>
main(PrometheusCharm)
File "/var/lib/juju/agents/unit-prometheus-0/charm/venv/ops/main.py", line 408, in main
_emit_charm_event(charm, dispatcher.event_name)
File "/var/lib/juju/agents/unit-prometheus-0/charm/venv/ops/main.py", line 142, in _emit_charm_event
event_to_emit.emit(*args, **kwargs)
File "/var/lib/juju/agents/unit-prometheus-0/charm/venv/ops/framework.py", line 275, in emit
framework._emit(event)
File "/var/lib/juju/agents/unit-prometheus-0/charm/venv/ops/framework.py", line 735, in _emit
self._reemit(event_path)
File "/var/lib/juju/agents/unit-prometheus-0/charm/venv/ops/framework.py", line 782, in _reemit
custom_handler(event)
File "./src/charm.py", line 115, in _configure
container.restart(self._name)
File "/var/lib/juju/agents/unit-prometheus-0/charm/venv/ops/model.py", line 1138, in restart
for svc in self.get_services(service_names):
File "/var/lib/juju/agents/unit-prometheus-0/charm/venv/ops/model.py", line 1179, in get_services
services = self._pebble.get_services(service_names)
File "/var/lib/juju/agents/unit-prometheus-0/charm/venv/ops/pebble.py", line 1398, in get_services
query = {'names': ','.join(names)}
TypeError: sequence item 0: expected str instance, tuple found
unit-prometheus-0: 10:55:49 ERROR juju.worker.uniter.operation hook "prometheus-pebble-ready" (via hook dispatching script: dispatch) failed: exit status 1
unit-prometheus-0: 10:55:49 ERROR juju.worker.uniter pebble poll failed for container "prometheus": hook failed
unit-prometheus-0: 10:55:51 ERROR unit.prometheus/0.juju-log Uncaught exception while in charm code:
Traceback (most recent call last):
File "/var/lib/juju/agents/unit-prometheus-0/charm/venv/ops/pebble.py", line 1103, in _request_raw
response = self.opener.open(request, timeout=self.timeout)
File "/usr/lib/python3.8/urllib/request.py", line 531, in open
response = meth(req, response)
File "/usr/lib/python3.8/urllib/request.py", line 640, in http_response
response = self.parent.error(
File "/usr/lib/python3.8/urllib/request.py", line 569, in error
return self._call_chain(*args)
File "/usr/lib/python3.8/urllib/request.py", line 502, in _call_chain
result = func(*args)
File "/usr/lib/python3.8/urllib/request.py", line 649, in http_error_default
raise HTTPError(req.full_url, code, msg, hdrs, fp)
urllib.error.HTTPError: HTTP Error 400: Bad Request
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/var/lib/juju/agents/unit-prometheus-0/charm/venv/ops/model.py", line 1133, in restart
self._pebble.restart_services(service_names)
File "/var/lib/juju/agents/unit-prometheus-0/charm/venv/ops/pebble.py", line 1250, in restart_services
return self._services_action('restart', services, timeout, delay)
File "/var/lib/juju/agents/unit-prometheus-0/charm/venv/ops/pebble.py", line 1263, in _services_action
resp = self._request('POST', '/v1/services', body=body)
File "/var/lib/juju/agents/unit-prometheus-0/charm/venv/ops/pebble.py", line 1074, in _request
response = self._request_raw(method, path, query, headers, data)
File "/var/lib/juju/agents/unit-prometheus-0/charm/venv/ops/pebble.py", line 1114, in _request_raw
raise APIError(body, code, status, message)
ops.pebble.APIError: action "restart" is unsupported
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "./src/charm.py", line 312, in <module>
main(PrometheusCharm)
File "/var/lib/juju/agents/unit-prometheus-0/charm/venv/ops/main.py", line 408, in main
_emit_charm_event(charm, dispatcher.event_name)
File "/var/lib/juju/agents/unit-prometheus-0/charm/venv/ops/main.py", line 142, in _emit_charm_event
event_to_emit.emit(*args, **kwargs)
File "/var/lib/juju/agents/unit-prometheus-0/charm/venv/ops/framework.py", line 275, in emit
framework._emit(event)
File "/var/lib/juju/agents/unit-prometheus-0/charm/venv/ops/framework.py", line 735, in _emit
self._reemit(event_path)
File "/var/lib/juju/agents/unit-prometheus-0/charm/venv/ops/framework.py", line 782, in _reemit
custom_handler(event)
File "./src/charm.py", line 115, in _configure
container.restart(self._name)
File "/var/lib/juju/agents/unit-prometheus-0/charm/venv/ops/model.py", line 1138, in restart
for svc in self.get_services(service_names):
File "/var/lib/juju/agents/unit-prometheus-0/charm/venv/ops/model.py", line 1179, in get_services
services = self._pebble.get_services(service_names)
File "/var/lib/juju/agents/unit-prometheus-0/charm/venv/ops/pebble.py", line 1398, in get_services
query = {'names': ','.join(names)}
TypeError: sequence item 0: expected str instance, tuple found
unit-prometheus-0: 10:55:51 ERROR juju.worker.uniter.operation hook "config-changed" (via hook dispatching script: dispatch) failed: exit status 1
INFO pytest_operator.plugin:plugin.py:294 Destroying model test-charm-vd5l
ERROR asyncio:base_events.py:1707 Task exception was never retrieved
future: <Task finished name='Task-292' coro=<WebSocketCommonProtocol.recv() done, defined at /home/jose/trabajos/canonical/repos/prometheus-operator/.tox/integration/lib/python3.8/site-packages/websockets/protocol.py:369> exception=ConnectionClosed('WebSocket connection is closed: code = 1000 (OK), no reason')>
Traceback (most recent call last):
File "/home/jose/trabajos/canonical/repos/prometheus-operator/.tox/integration/lib/python3.8/site-packages/websockets/protocol.py", line 434, in recv
yield from self.ensure_open()
File "/home/jose/trabajos/canonical/repos/prometheus-operator/.tox/integration/lib/python3.8/site-packages/websockets/protocol.py", line 644, in ensure_open
raise ConnectionClosed(
websockets.exceptions.ConnectionClosed: WebSocket connection is closed: code = 1000 (OK), no reason
=========================================================================================== FAILURES ============================================================================================
____________________________________________________________________________ test_build_and_deploy_with_ubuntu_image ____________________________________________________________________________
Traceback (most recent call last):
File "/home/jose/trabajos/canonical/repos/prometheus-operator/tests/integration/test_charm.py", line 21, in test_build_and_deploy_with_ubuntu_image
await ops_test.model.wait_for_idle(apps=["prometheus"], status="active")
File "/home/jose/trabajos/canonical/repos/prometheus-operator/.tox/integration/lib/python3.8/site-packages/juju/model.py", line 2619, in wait_for_idle
_raise_for_status(errors, "error")
File "/home/jose/trabajos/canonical/repos/prometheus-operator/.tox/integration/lib/python3.8/site-packages/juju/model.py", line 2571, in _raise_for_status
raise error_type("{}{} in {}: {}".format(
juju.errors.JujuUnitError: Unit in error: prometheus/0
-------------------------------------------------------------------------------------- Captured log setup ---------------------------------------------------------------------------------------
INFO pytest_operator.plugin:plugin.py:160 Using tmp_path: /home/jose/trabajos/canonical/repos/prometheus-operator/.tox/integration/tmp/pytest/test-charm-vd5l0
INFO pytest_operator.plugin:plugin.py:222 Adding model mk8s:test-charm-vd5l
WARNING juju.client.connection:connection.py:729 unknown facade CAASModelConfigManager
WARNING juju.client.connection:connection.py:753 unexpected facade CAASModelConfigManager found, unable to decipher version to use
WARNING juju.client.connection:connection.py:729 unknown facade RaftLease
WARNING juju.client.connection:connection.py:753 unexpected facade RaftLease found, unable to decipher version to use
WARNING juju.client.connection:connection.py:729 unknown facade Secrets
WARNING juju.client.connection:connection.py:753 unexpected facade Secrets found, unable to decipher version to use
WARNING juju.client.connection:connection.py:729 unknown facade SecretsManager
WARNING juju.client.connection:connection.py:753 unexpected facade SecretsManager found, unable to decipher version to use
WARNING juju.client.connection:connection.py:729 unknown facade SecretsRotationWatcher
WARNING juju.client.connection:connection.py:753 unexpected facade SecretsRotationWatcher found, unable to decipher version to use
--------------------------------------------------------------------------------------- Captured log call ---------------------------------------------------------------------------------------
INFO pytest_operator.plugin:plugin.py:338 Building charm prometheus-k8s
INFO juju.model:model.py:1873 Deploying local:focal/prometheus-k8s-0
INFO juju.model:model.py:2627 Waiting for model:
prometheus/0 [allocating] waiting: installing agent
------------------------------------------------------------------------------------- Captured log teardown -------------------------------------------------------------------------------------
INFO pytest_operator.plugin:plugin.py:260 Model status:
Unit Machine Status Message
prometheus/0* no-machine error hook failed: "config-changed"
Machine Series Status
INFO pytest_operator.plugin:plugin.py:280 Juju error logs:
controller-0: 10:55:38 ERROR juju.worker.caasapplicationprovisioner.runner exited "prometheus": Operation cannot be fulfilled on pods "prometheus-0": the object has been modified; please apply your changes to the latest version and try again
unit-prometheus-0: 10:55:47 ERROR unit.prometheus/0.juju-log Uncaught exception while in charm code:
Traceback (most recent call last):
File "/var/lib/juju/agents/unit-prometheus-0/charm/venv/ops/pebble.py", line 1103, in _request_raw
response = self.opener.open(request, timeout=self.timeout)
File "/usr/lib/python3.8/urllib/request.py", line 531, in open
response = meth(req, response)
File "/usr/lib/python3.8/urllib/request.py", line 640, in http_response
response = self.parent.error(
File "/usr/lib/python3.8/urllib/request.py", line 569, in error
return self._call_chain(*args)
File "/usr/lib/python3.8/urllib/request.py", line 502, in _call_chain
result = func(*args)
File "/usr/lib/python3.8/urllib/request.py", line 649, in http_error_default
raise HTTPError(req.full_url, code, msg, hdrs, fp)
urllib.error.HTTPError: HTTP Error 400: Bad Request
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/var/lib/juju/agents/unit-prometheus-0/charm/venv/ops/model.py", line 1133, in restart
self._pebble.restart_services(service_names)
File "/var/lib/juju/agents/unit-prometheus-0/charm/venv/ops/pebble.py", line 1250, in restart_services
return self._services_action('restart', services, timeout, delay)
File "/var/lib/juju/agents/unit-prometheus-0/charm/venv/ops/pebble.py", line 1263, in _services_action
resp = self._request('POST', '/v1/services', body=body)
File "/var/lib/juju/agents/unit-prometheus-0/charm/venv/ops/pebble.py", line 1074, in _request
response = self._request_raw(method, path, query, headers, data)
File "/var/lib/juju/agents/unit-prometheus-0/charm/venv/ops/pebble.py", line 1114, in _request_raw
raise APIError(body, code, status, message)
ops.pebble.APIError: action "restart" is unsupported
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "./src/charm.py", line 312, in <module>
main(PrometheusCharm)
File "/var/lib/juju/agents/unit-prometheus-0/charm/venv/ops/main.py", line 408, in main
_emit_charm_event(charm, dispatcher.event_name)
File "/var/lib/juju/agents/unit-prometheus-0/charm/venv/ops/main.py", line 142, in _emit_charm_event
event_to_emit.emit(*args, **kwargs)
File "/var/lib/juju/agents/unit-prometheus-0/charm/venv/ops/framework.py", line 275, in emit
framework._emit(event)
File "/var/lib/juju/agents/unit-prometheus-0/charm/venv/ops/framework.py", line 735, in _emit
self._reemit(event_path)
File "/var/lib/juju/agents/unit-prometheus-0/charm/venv/ops/framework.py", line 782, in _reemit
custom_handler(event)
File "./src/charm.py", line 121, in _configure
container.restart(self._name)
File "/var/lib/juju/agents/unit-prometheus-0/charm/venv/ops/model.py", line 1138, in restart
for svc in self.get_services(service_names):
File "/var/lib/juju/agents/unit-prometheus-0/charm/venv/ops/model.py", line 1179, in get_services
services = self._pebble.get_services(service_names)
File "/var/lib/juju/agents/unit-prometheus-0/charm/venv/ops/pebble.py", line 1398, in get_services
query = {'names': ','.join(names)}
TypeError: sequence item 0: expected str instance, tuple found
unit-prometheus-0: 10:55:48 ERROR juju.worker.uniter.operation hook "prometheus-pebble-ready" (via hook dispatching script: dispatch) failed: exit status 1
unit-prometheus-0: 10:55:48 ERROR juju.worker.uniter pebble poll failed for container "prometheus": hook failed
unit-prometheus-0: 10:55:48 ERROR unit.prometheus/0.juju-log Uncaught exception while in charm code:
Traceback (most recent call last):
File "/var/lib/juju/agents/unit-prometheus-0/charm/venv/ops/pebble.py", line 1103, in _request_raw
response = self.opener.open(request, timeout=self.timeout)
File "/usr/lib/python3.8/urllib/request.py", line 531, in open
response = meth(req, response)
File "/usr/lib/python3.8/urllib/request.py", line 640, in http_response
response = self.parent.error(
File "/usr/lib/python3.8/urllib/request.py", line 569, in error
return self._call_chain(*args)
File "/usr/lib/python3.8/urllib/request.py", line 502, in _call_chain
result = func(*args)
File "/usr/lib/python3.8/urllib/request.py", line 649, in http_error_default
raise HTTPError(req.full_url, code, msg, hdrs, fp)
urllib.error.HTTPError: HTTP Error 400: Bad Request
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/var/lib/juju/agents/unit-prometheus-0/charm/venv/ops/model.py", line 1133, in restart
self._pebble.restart_services(service_names)
File "/var/lib/juju/agents/unit-prometheus-0/charm/venv/ops/pebble.py", line 1250, in restart_services
return self._services_action('restart', services, timeout, delay)
File "/var/lib/juju/agents/unit-prometheus-0/charm/venv/ops/pebble.py", line 1263, in _services_action
resp = self._request('POST', '/v1/services', body=body)
File "/var/lib/juju/agents/unit-prometheus-0/charm/venv/ops/pebble.py", line 1074, in _request
response = self._request_raw(method, path, query, headers, data)
File "/var/lib/juju/agents/unit-prometheus-0/charm/venv/ops/pebble.py", line 1114, in _request_raw
raise APIError(body, code, status, message)
ops.pebble.APIError: action "restart" is unsupported
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "./src/charm.py", line 312, in <module>
main(PrometheusCharm)
File "/var/lib/juju/agents/unit-prometheus-0/charm/venv/ops/main.py", line 408, in main
_emit_charm_event(charm, dispatcher.event_name)
File "/var/lib/juju/agents/unit-prometheus-0/charm/venv/ops/main.py", line 142, in _emit_charm_event
event_to_emit.emit(*args, **kwargs)
File "/var/lib/juju/agents/unit-prometheus-0/charm/venv/ops/framework.py", line 275, in emit
framework._emit(event)
File "/var/lib/juju/agents/unit-prometheus-0/charm/venv/ops/framework.py", line 735, in _emit
self._reemit(event_path)
File "/var/lib/juju/agents/unit-prometheus-0/charm/venv/ops/framework.py", line 782, in _reemit
custom_handler(event)
File "./src/charm.py", line 115, in _configure
container.restart(self._name)
File "/var/lib/juju/agents/unit-prometheus-0/charm/venv/ops/model.py", line 1138, in restart
for svc in self.get_services(service_names):
File "/var/lib/juju/agents/unit-prometheus-0/charm/venv/ops/model.py", line 1179, in get_services
services = self._pebble.get_services(service_names)
File "/var/lib/juju/agents/unit-prometheus-0/charm/venv/ops/pebble.py", line 1398, in get_services
query = {'names': ','.join(names)}
TypeError: sequence item 0: expected str instance, tuple found
unit-prometheus-0: 10:55:49 ERROR juju.worker.uniter.operation hook "prometheus-pebble-ready" (via hook dispatching script: dispatch) failed: exit status 1
unit-prometheus-0: 10:55:49 ERROR juju.worker.uniter pebble poll failed for container "prometheus": hook failed
unit-prometheus-0: 10:55:51 ERROR unit.prometheus/0.juju-log Uncaught exception while in charm code:
Traceback (most recent call last):
File "/var/lib/juju/agents/unit-prometheus-0/charm/venv/ops/pebble.py", line 1103, in _request_raw
response = self.opener.open(request, timeout=self.timeout)
File "/usr/lib/python3.8/urllib/request.py", line 531, in open
response = meth(req, response)
File "/usr/lib/python3.8/urllib/request.py", line 640, in http_response
response = self.parent.error(
File "/usr/lib/python3.8/urllib/request.py", line 569, in error
return self._call_chain(*args)
File "/usr/lib/python3.8/urllib/request.py", line 502, in _call_chain
result = func(*args)
File "/usr/lib/python3.8/urllib/request.py", line 649, in http_error_default
raise HTTPError(req.full_url, code, msg, hdrs, fp)
urllib.error.HTTPError: HTTP Error 400: Bad Request
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/var/lib/juju/agents/unit-prometheus-0/charm/venv/ops/model.py", line 1133, in restart
self._pebble.restart_services(service_names)
File "/var/lib/juju/agents/unit-prometheus-0/charm/venv/ops/pebble.py", line 1250, in restart_services
return self._services_action('restart', services, timeout, delay)
File "/var/lib/juju/agents/unit-prometheus-0/charm/venv/ops/pebble.py", line 1263, in _services_action
resp = self._request('POST', '/v1/services', body=body)
File "/var/lib/juju/agents/unit-prometheus-0/charm/venv/ops/pebble.py", line 1074, in _request
response = self._request_raw(method, path, query, headers, data)
File "/var/lib/juju/agents/unit-prometheus-0/charm/venv/ops/pebble.py", line 1114, in _request_raw
raise APIError(body, code, status, message)
ops.pebble.APIError: action "restart" is unsupported
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "./src/charm.py", line 312, in <module>
main(PrometheusCharm)
File "/var/lib/juju/agents/unit-prometheus-0/charm/venv/ops/main.py", line 408, in main
_emit_charm_event(charm, dispatcher.event_name)
File "/var/lib/juju/agents/unit-prometheus-0/charm/venv/ops/main.py", line 142, in _emit_charm_event
event_to_emit.emit(*args, **kwargs)
File "/var/lib/juju/agents/unit-prometheus-0/charm/venv/ops/framework.py", line 275, in emit
framework._emit(event)
File "/var/lib/juju/agents/unit-prometheus-0/charm/venv/ops/framework.py", line 735, in _emit
self._reemit(event_path)
File "/var/lib/juju/agents/unit-prometheus-0/charm/venv/ops/framework.py", line 782, in _reemit
custom_handler(event)
File "./src/charm.py", line 115, in _configure
container.restart(self._name)
File "/var/lib/juju/agents/unit-prometheus-0/charm/venv/ops/model.py", line 1138, in restart
for svc in self.get_services(service_names):
File "/var/lib/juju/agents/unit-prometheus-0/charm/venv/ops/model.py", line 1179, in get_services
services = self._pebble.get_services(service_names)
File "/var/lib/juju/agents/unit-prometheus-0/charm/venv/ops/pebble.py", line 1398, in get_services
query = {'names': ','.join(names)}
TypeError: sequence item 0: expected str instance, tuple found
unit-prometheus-0: 10:55:51 ERROR juju.worker.uniter.operation hook "config-changed" (via hook dispatching script: dispatch) failed: exit status 1
INFO pytest_operator.plugin:plugin.py:294 Destroying model test-charm-vd5l
ERROR asyncio:base_events.py:1707 Task exception was never retrieved
future: <Task finished name='Task-292' coro=<WebSocketCommonProtocol.recv() done, defined at /home/jose/trabajos/canonical/repos/prometheus-operator/.tox/integration/lib/python3.8/site-packages/websockets/protocol.py:369> exception=ConnectionClosed('WebSocket connection is closed: code = 1000 (OK), no reason')>
Traceback (most recent call last):
File "/home/jose/trabajos/canonical/repos/prometheus-operator/.tox/integration/lib/python3.8/site-packages/websockets/protocol.py", line 434, in recv
yield from self.ensure_open()
File "/home/jose/trabajos/canonical/repos/prometheus-operator/.tox/integration/lib/python3.8/site-packages/websockets/protocol.py", line 644, in ensure_open
raise ConnectionClosed(
websockets.exceptions.ConnectionClosed: WebSocket connection is closed: code = 1000 (OK), no reason
==================================================================================== short test summary info ====================================================================================
FAILED tests/integration/test_charm.py::test_build_and_deploy_with_ubuntu_image - juju.errors.JujuUnitError: Unit in error: prometheus/0
================================================================================= 1 failed in 133.53s (0:02:13) =================================================================================
ERROR: InvocationError for command /home/jose/trabajos/canonical/repos/prometheus-operator/.tox/integration/bin/pytest -v --tb native --log-cli-level=INFO -s tests/integration (exited with code 1)
____________________________________________________________________________________________ summary ____________________________________________________________________________________________
ERROR: integration: commands failedfrom prometheus-k8s-operator.
balbirthomas
commented on September 23, 2024
The first error message showed that the charm managed to set a valid status
Unit Machine Status Message
prometheus/0* no-machine blocked Failed to load Prometheus config
This implied charm failed to make a HTTP request to the Prometheus API. Hence I suggested restarting the container in such a case. The second error message is quite strange and is coming from Pebble itself
controller-0: 10:55:38 ERROR juju.worker.caasapplicationprovisioner.runner exited "prometheus": Operation cannot be fulfilled on pods "prometheus-0": the object has been modified; please apply your changes to the latest version and try again
unit-prometheus-0: 10:55:47 ERROR unit.prometheus/0.juju-log Uncaught exception while in charm code:
Traceback (most recent call last):
File "/var/lib/juju/agents/unit-prometheus-0/charm/venv/ops/pebble.py", line 1103, in _request_raw
response = self.opener.open(request, timeout=self.timeout)
from prometheus-k8s-operator.
balbirthomas
commented on September 23, 2024
@Abuelodelanada I am still not able to reproduce this issue. Is this still happening for you using the latest pull request branch ? Have you tried purging Juju, Microk8s, removing the directories ~/.cache/juju, ~/.local/share/juju, then reinstalling everthing and checking again ?
from prometheus-k8s-operator.
Abuelodelanada
commented on September 23, 2024
Hello @balbirthomas
With this PR, the integration tests passed! There is only one ERROR message:
➜ prometheus-operator git:(balbirthomas-integ) ✗ tox -e integration
integration installed: attrs==21.2.0,backcall==0.2.0,bcrypt==3.2.0,cachetools==4.2.4,certifi==2021.10.8,cffi==1.15.0,charset-normalizer==2.0.7,cryptography==35.0.0,decorator==5.1.0,google-auth==2.3.0,idna==3.3,iniconfig==1.1.1,ipdb==0.13.9,ipython==7.28.0,jedi==0.18.0,Jinja2==3.0.2,juju==2.9.4,jujubundlelib==0.5.6,kubernetes==18.20.0,macaroonbakery==1.3.1,MarkupSafe==2.0.1,matplotlib-inline==0.1.3,mypy-extensions==0.4.3,oauthlib==3.1.1,packaging==21.0,paramiko==2.8.0,parso==0.8.2,pexpect==4.8.0,pickleshare==0.7.5,pluggy==1.0.0,prompt-toolkit==3.0.21,protobuf==3.19.0,ptyprocess==0.7.0,py==1.10.0,pyasn1==0.4.8,pyasn1-modules==0.2.8,pycparser==2.20,Pygments==2.10.0,pymacaroons==0.13.0,PyNaCl==1.4.0,pyparsing==2.4.7,pyRFC3339==1.1,pytest==6.2.5,pytest-asyncio==0.16.0,pytest-operator==0.8.4,python-dateutil==2.8.2,pytz==2021.3,PyYAML==6.0,requests==2.26.0,requests-oauthlib==1.3.0,rsa==4.7.2,six==1.16.0,theblues==0.5.2,toml==0.10.2,toposort==1.7,traitlets==5.1.0,typing-extensions==3.10.0.2,typing-inspect==0.7.1,urllib3==1.26.7,wcwidth==0.2.5,websocket-client==1.2.1,websockets==7.0
integration run-test-pre: PYTHONHASHSEED='3122953279'
integration run-test: commands[0] | pytest -v --tb native --log-cli-level=INFO -s /home/jose/trabajos/canonical/repos/prometheus-operator/tests/integration
===================================================================================== test session starts =====================================================================================
platform linux -- Python 3.8.10, pytest-6.2.5, py-1.10.0, pluggy-1.0.0 -- /home/jose/trabajos/canonical/repos/prometheus-operator/.tox/integration/bin/python
cachedir: .tox/integration/.pytest_cache
rootdir: /home/jose/trabajos/canonical/repos/prometheus-operator
plugins: asyncio-0.16.0, operator-0.8.4
collected 3 items
tests/integration/test_charm.py::test_build_and_deploy_with_alternative_images[ubuntu/prometheus:latest] /snap/bin/juju
/snap/bin/charmcraft
--------------------------------------------------------------------------------------- live log setup ----------------------------------------------------------------------------------------
INFO pytest_operator.plugin:plugin.py:160 Using tmp_path: /home/jose/trabajos/canonical/repos/prometheus-operator/.tox/integration/tmp/pytest/test-charm-ldv00
INFO pytest_operator.plugin:plugin.py:222 Adding model mk8s:test-charm-ldv0
WARNING juju.client.connection:connection.py:729 unknown facade CAASModelConfigManager
WARNING juju.client.connection:connection.py:753 unexpected facade CAASModelConfigManager found, unable to decipher version to use
WARNING juju.client.connection:connection.py:729 unknown facade RaftLease
WARNING juju.client.connection:connection.py:753 unexpected facade RaftLease found, unable to decipher version to use
WARNING juju.client.connection:connection.py:729 unknown facade Secrets
WARNING juju.client.connection:connection.py:753 unexpected facade Secrets found, unable to decipher version to use
WARNING juju.client.connection:connection.py:729 unknown facade SecretsManager
WARNING juju.client.connection:connection.py:753 unexpected facade SecretsManager found, unable to decipher version to use
WARNING juju.client.connection:connection.py:729 unknown facade SecretsRotationWatcher
WARNING juju.client.connection:connection.py:753 unexpected facade SecretsRotationWatcher found, unable to decipher version to use
INFO pytest_operator.plugin:plugin.py:338 Building charm prometheus-k8s
---------------------------------------------------------------------------------------- live log call ----------------------------------------------------------------------------------------
INFO juju.model:model.py:1873 Deploying local:focal/prometheus-k8s-0
INFO juju.model:model.py:2627 Waiting for model:
prometheus-ubuntu/0 [allocating] waiting: installing agent
INFO juju.model:model.py:2627 Waiting for model:
prometheus-ubuntu/0 [idle] waiting: Waiting for Pebble ready
PASSED
tests/integration/test_charm.py::test_build_and_deploy_with_alternative_images[prom/prometheus:latest]
---------------------------------------------------------------------------------------- live log call ----------------------------------------------------------------------------------------
INFO juju.model:model.py:1873 Deploying local:focal/prometheus-k8s-1
INFO juju.model:model.py:2627 Waiting for model:
prometheus-prom/0 [allocating] waiting: installing agent
INFO juju.model:model.py:2627 Waiting for model:
prometheus-prom/0 [idle] active:
PASSED
tests/integration/test_charm.py::test_build_and_deploy_prometheus_tester
--------------------------------------------------------------------------------------- live log setup ----------------------------------------------------------------------------------------
INFO pytest_operator.plugin:plugin.py:338 Building charm prometheus-tester
---------------------------------------------------------------------------------------- live log call ----------------------------------------------------------------------------------------
INFO juju.model:model.py:1873 Deploying local:focal/prometheus-tester-0
INFO juju.model:model.py:2627 Waiting for model:
prometheus-tester/0 [allocating] waiting: installing agent
INFO juju.model:model.py:2627 Waiting for model:
prometheus-tester/0 [idle] active:
PASSED
-------------------------------------------------------------------------------------- live log teardown --------------------------------------------------------------------------------------
INFO pytest_operator.plugin:plugin.py:260 Model status:
Unit Machine Status Message
prometheus-tester/0* no-machine active
Machine Series Status
INFO pytest_operator.plugin:plugin.py:280 Juju error logs:
unit-prometheus-ubuntu-0: 10:55:35 ERROR juju.worker.uniter pebble poll failed for container "prometheus": failed to get pebble info: cannot obtain system details: cannot communicate with server: Get "http://localhost/v1/system-info": dial unix /charm/containers/prometheus/pebble.socket: connect: no such file or directory
unit-prometheus-ubuntu-0: 10:55:40 ERROR juju.worker.uniter pebble poll failed for container "prometheus": failed to get pebble info: cannot obtain system details: cannot communicate with server: Get "http://localhost/v1/system-info": dial unix /charm/containers/prometheus/pebble.socket: connect: no such file or directory
unit-prometheus-ubuntu-0: 10:55:45 ERROR juju.worker.uniter pebble poll failed for container "prometheus": failed to get pebble info: cannot obtain system details: cannot communicate with server: Get "http://localhost/v1/system-info": dial unix /charm/containers/prometheus/pebble.socket: connect: no such file or directory
unit-prometheus-ubuntu-0: 10:55:50 ERROR juju.worker.uniter pebble poll failed for container "prometheus": failed to get pebble info: cannot obtain system details: cannot communicate with server: Get "http://localhost/v1/system-info": dial unix /charm/containers/prometheus/pebble.socket: connect: no such file or directory
unit-prometheus-prom-0: 10:56:35 ERROR juju.worker.uniter pebble poll failed for container "prometheus": failed to get pebble info: cannot obtain system details: cannot communicate with server: Get "http://localhost/v1/system-info": dial unix /charm/containers/prometheus/pebble.socket: connect: no such file or directory
unit-prometheus-prom-0: 10:57:06 ERROR juju.worker.uniter pebble poll failed for container "prometheus": failed to get pebble info: cannot obtain system details: cannot communicate with server: Get "http://localhost/v1/system-info": dial unix /charm/containers/prometheus/pebble.socket: connect: no such file or directory
unit-prometheus-prom-0: 10:57:11 ERROR juju.worker.uniter pebble poll failed for container "prometheus": failed to get pebble info: cannot obtain system details: cannot communicate with server: Get "http://localhost/v1/system-info": dial unix /charm/containers/prometheus/pebble.socket: connect: no such file or directory
unit-prometheus-prom-0: 10:57:16 ERROR juju.worker.uniter pebble poll failed for container "prometheus": failed to get pebble info: cannot obtain system details: cannot communicate with server: Get "http://localhost/v1/system-info": dial unix /charm/containers/prometheus/pebble.socket: connect: no such file or directory
controller-0: 10:58:03 ERROR juju.worker.caasapplicationprovisioner.runner exited "prometheus-tester": Operation cannot be fulfilled on pods "prometheus-tester-0": the object has been modified; please apply your changes to the latest version and try again
INFO pytest_operator.plugin:plugin.py:294 Destroying model test-charm-ldv0
ERROR asyncio:base_events.py:1707 Task exception was never retrieved
future: <Task finished name='Task-839' coro=<WebSocketCommonProtocol.recv() done, defined at /home/jose/trabajos/canonical/repos/prometheus-operator/.tox/integration/lib/python3.8/site-packages/websockets/protocol.py:369> exception=ConnectionClosed('WebSocket connection is closed: code = 1000 (OK), no reason')>
Traceback (most recent call last):
File "/home/jose/trabajos/canonical/repos/prometheus-operator/.tox/integration/lib/python3.8/site-packages/websockets/protocol.py", line 434, in recv
yield from self.ensure_open()
File "/home/jose/trabajos/canonical/repos/prometheus-operator/.tox/integration/lib/python3.8/site-packages/websockets/protocol.py", line 644, in ensure_open
raise ConnectionClosed(
websockets.exceptions.ConnectionClosed: WebSocket connection is closed: code = 1000 (OK), no reason
================================================================================ 3 passed in 275.65s (0:04:35) ================================================================================
___________________________________________________________________________________________ summary ___________________________________________________________________________________________
integration: commands succeeded
congratulations :)from prometheus-k8s-operator.
balbirthomas
commented on September 23, 2024
Thanks. Closing this ticket for now. There will be additional pushes to that PR. There are lot of spurious errors that are currently generated by libjuju/pytest-operator and we may ignore them for now.
from prometheus-k8s-operator.
Related Issues (20)
- The charm size got 10x bigger due to cygrpc.cpython-38-x86_64-linux-gnu.so with debug symbols HOT 2
- Non-functional deployment with latest/edge - /cos-prometheus-0: 404 Not Found HOT 3
- After removing the relation with scrape-inteval-config old alert rules are still in Prometheus HOT 5
- Alert rules from different sources can overwrite each other HOT 13
- Broken scrape jobs can get past our checks HOT 1
- Stuck on BlockedStatus even after `maximum_retention_size` config option changed to valid
- Out of order sample from remote write HOT 3
- [MetricsEndpointAggregator] Group name should be rendered using unit name, not app name HOT 2
- Topology labels are overridden in MetricsEndpointProvider HOT 1
- "certificates-relation-changed" hook failed during deployment HOT 4
- Set external URL as a Grafana Data Source HOT 4
- cannot stop service: promtail service does not exist HOT 1
- Internal url is rendered as the "source" in alerts HOT 2
- Charm reports permission denied when setting relation data HOT 2
- Remove the check `if alert_rules_as_dict` in `MetricsEndpointProvider`
- chore: update libraries to new major versions
- `prometheus_scrape` relation should block charm for cross-model relations HOT 1
- charm tracing fails over tls
- Alerts gone after pod reschedule HOT 1
- Error in scrape job definition pass silently HOT 1
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from prometheus-k8s-operator.