cockpit-project / console.dot Goto Github PK

It's not packaged in any distribution. So far the best idea that we had is to replace it with a little Python script that uses the websockets module.

So far we are still using my hacked container from cockpit-project/cockpit#17473 , which includes websocat. But we don't need the two cockpit patches there after all.

We should either add python3 and python3-websocket to the cockpit/ws container, or install cockpit-ws into the appservice container and use the same container both for the appservice and for the sessions. I would prefer the second variant, as that's a bit more efficient on k8s (shared container image) and we retain more control over that. The only downside is that cockpit-ws isn't available in UBI, so we'd need to pull it from our COPR or CentOS Stream.

#3 (comment)

The running pods and sessions map in redis can go out of sync for any number of reasons: session pods crashing and the "websocket EOF" message not making it through, or API pods crashing at the wrong time, or new sessions in state wait_target which were never connected to. In the app service

• track the time of session creation in SESSIONS
• do a regular sweep over:
  • all running session pods, and delete the ones which are not in SESSIONS
  • all SESSIONS entries, and delete the wait_target ones which are older than 1 hour, and the closed ones, and the running ones which are older than, say, one day?

look at the X-RH-Identity header from both the browser (user type) and the target machine (system type), check that the org IDs match. Mention the org ID in the /status reply.

This requires changing the bridge connector to connect with a TLS certificate instead of basic auth.

We need to do some mocking to be able to do that with podman and in the tests. We can just generate another cert pair for the connected host, and add the header in our 3scale container.

Implement a "landing" page which is shown when a user clicks on "open cockpit" and shows the current status and replaces the page with Cockpit once the pod is launched and the bridge is connected.

make run fails to run the test containers locally as non-root user on my system (Fedora 36, podman 4.2.0). The containers are unable to open some files from their volume mounts:

$ podman pod logs webconsoleapp
27c97aa74d14 2022/10/11 10:15:58 [emerg] 1#1: open() "/etc/nginx/nginx.conf" failed (13: Permission denied)
27c97aa74d14 nginx: [emerg] open() "/etc/nginx/nginx.conf" failed (13: Permission denied)
5f6fece4a9e4 python3: can't open file '/usr/local/bin/multiplexer.py': [Errno 13] Permission denied

The problem seems to be caused by SELinux violations. The files have wrong SELinux labels:

type=AVC msg=audit(1665483358.464:2468): avc:  denied  { read } for  pid=792821 comm="nginx" name="nginx.conf" dev="dm-3" ino=9177030 scontext=system_u:system_r:container_t:s0:c375,c920 tcontext=unconfined_u:object_r:user_home_t:s0 tclass=file permissive=0

type=AVC msg=audit(1665483358.584:2469): avc:  denied  { read } for  pid=792885 comm="python3" name="multiplexer.py" dev="dm-3" ino=9177035 scontext=system_u:system_r:container_t:s0:c375,c920 tcontext=unconfined_u:object_r:user_home_t:s0 tclass=file permissive=0

I tried to set the volume option SELinuxRelabel: true, but that doesn't fix the issue.

We can run websocat | bridge right inside the session pod for this, so that we don't have to spawn VMs.

We currently have a sleep(1) there, because otherwise nginx immediately crashes half of the time. My initial suspicion is that we are sending a SIGHUP too fast, and nginx did not yet set up its signal handlers at this point. We need to wait until nginx is running before we start to change the config and HUP.

I have been playing to deploy it in the ephemeral environment (wip) and I suggest this ticket to read configurations provided by clowder operator.

When it is deployed with an openshift template using clowderapp resource, it injects into the pod a secret with several configurations; in this particular scenario, the redis configuration (host, port) are provided into a json file which is pointed out by ACG_CONFIG environment variable. Currently exists app-common-python to help to read this parameters.

at appservice/config.py it could be something like:

from app_common_python import LoadedConfig, isClowderEnabled

# https://github.com/RedHatInsights/app-common-python
if isClowderEnabled():
    REDIS_SERVICE_HOST = LoadedConfig.inMemoryDb.hostname
    REDIS_SERVICE_PORT = int(LoadedConfig.inMemoryDb.port)
else:
    REDIS_SERVICE_HOST = environ.get('REDIS_SERVICE_HOST', 'localhost')
    REDIS_SERVICE_PORT = int(environ.get('REDIS_SERVICE_PORT', '6379'))

and use it at appservice/multiplexor.py by addding the app-common-python package:

   REDIS = redis.asyncio.Redis(host=config.REDIS_SERVICE_HOST,
                               port=config.REDIS_SERVICE_PORT)

Finally at appservice/Containerfile I added the dependency.

RUN pip3 install redis starlette httpx websockets uvicorn app-common-python

Additional Information:

• I have tried to deploy with this: https://github.com/avisiedo/cockpit-ephemeral (wip)
• https://github.com/RedHatInsights/app-common-python
• https://pypi.org/project/app-common-python/

This triggers an assertion in cockpit-ws and makes it crash.

See #3 (comment)

@marusak had the idea to make these two cases spit out the same page. This would allow us to clean up the SESSIONS map aggressively, right after a session is gone. See issue #30 and #54.

It still feels like these are two different cases -- trying a random (nonexisting) UUID shouldn't claim that "the session has ended" -- that would be equally confusing.

I didn't quite make up my mind about this, but wanted to note it down.

bind-mount it into the container on make run-devel or so

curl magic

See "HMSIDM Webconsole Cloud Service" doc

It's not packaged in any distribution. So far the best idea that we had is to replace it with a little Python script that uses the websockets module, and deliver that as a .pyz app so that the only requirement on the target machine is python3 (which we'll soon need anyway due to the ported bridge).

We need to keep track of the session state: "new", "pod ready", "target machine connected", and possibly "abandoned" or so.
POSTing to /api/webconsole/v1/sessions/<session_id>/status should update the status, and GETing it should retrieve it.

This API endpoint should not be exposed publicly, just internally from the appservice to the session pod.

Also wire this up so that connecting the target machine updates the status. This is not super-trivial, as currently that port is forwarded directly to cockpit-ws and its socat-session.sh, and it's not obvious where to slip in the status update. We may need to modify cockpit for that, or replace socat-session.sh+websocat with something more customized.

If for whatever reason cockpit-bridge-connector.pyz dies, cockpit will detect this and show a special page:

The reconnect option will however not work, maybe we should just remove it?

handle_session_new uses socket.gethostbyname to resolve a hostname to IP address. Hostname lookup is blocking operation. The correct API is https://docs.python.org/3/library/asyncio-eventloop.html#asyncio.loop.getaddrinfo

When adding this to our testSession:

        # crash container for s2
        subprocess.call(['podman', 'rm', '--time=0', '--force', f'session-{s2}'])
        # first session still works
        self.checkSession(s1)

        # can create a new session
        s3 = self.newSession()
        self.checkSession(s3)

Then nginx freaks out because it cannot DNS-resolve the second session pod any more. This can happen all the time due to crashes, idle timeouts, networking flakes, etc. This must not break the routing to other sessions.

This is a well-known problem, and there are workarounds. I tried to applied that one:

--- appservice/multiplexer.py
+++ appservice/multiplexer.py
@@ -50,6 +50,8 @@ http {{
 
         server_name localhost;
 
+        resolver 10.89.0.1 valid=5s;
+
         {routes}
 
         location {route_control}/ping {{
@@ -92,7 +94,8 @@ def write_routes(sessions):
     for sessionid in sessions:
         routes += f"""
 location {config.ROUTE_WSS}/sessions/{sessionid}/web {{
-    proxy_pass http://session-{sessionid}{SESSION_INSTANCE_DOMAIN}:9090;
+    set $fwd http://session-{sessionid}{SESSION_INSTANCE_DOMAIN}:9090;
+    proxy_pass $fwd;
 
     # Required to proxy the connection to Cockpit
     proxy_set_header Host $host;
@@ -109,7 +112,8 @@ location {config.ROUTE_WSS}/sessions/{sessionid}/web {{
     gzip off;
 }}
 location {config.ROUTE_WSS}/sessions/{sessionid}/ws {{
-    proxy_pass http://session-{sessionid}{SESSION_INSTANCE_DOMAIN}:8080;
+    set $fwd http://session-{sessionid}{SESSION_INSTANCE_DOMAIN}:8080;
+    proxy_pass $fwd;
 
     # Required to proxy the connection to Cockpit
     proxy_set_header Host $host;

That is the resolver IP for podman. Of course k8s has a different one (172.30.0.10), and it should just be read from /etc/resolv.conf, but that's not the point. This still does not work. I've spent over two hours on this already.

Frankly, I think this is the point where our nginx PoC just breaks down. It won't be the final solution anyway, as we cannot implement proper session lifetime API/handling (#28, #29, #30). We need more control. So I suppose this will only be fixed by a rewrite.

GET requests should not change any state. The spec already specifies that this must be a POST request instead.

If the /ws websocket gets closed (which usually happens on logout or when the user moves away from the web page), the connector should notice that and exit. If it doesn't, fix it. Add a test that this happens.

I think I saw this crash in cockpit-bridge-websocket-connector the other day, but didn't pay much attention to it. Let's move server/Containerfile to CentOS 8 and test with that.

Configure c-ws to set a session idle time of 15 minutes by default.

Detect browser disconnects or explicit logouts, and clean up the session container and session table accordingly. The latter must keep the session as "closed" at least for a while, to get a correct UI. See issue #60

The cleanup can also happen in a regular "garbage collection sweep", that is handled in issue #54. For this issue, just consider the session container. Ideally cockpit-ws would time out and stop itself, then we can mark the session containers as auto-delete. But if c-ws does not do that, we need to kill the containers explicitly.

Right now these are very simple and ugly stubs. They have the required functionality, but look nothing like a real product 😁

In our setup the login page cannot work, due to our usage of --local-session. Instead, after the session ends we should just show some "Nothing to see here, please close" page without any actions.

This will have to be changed in cockpit-ws itself and thus reassigned. But filing it here first for tracking the outstanding issues.