Comments (8)
thomascube
commented on July 1, 2024
Update: I see this works when starting kubedock server from my local machine with full permissions on the target namespace. The lease is successfully released with holderIdentity: ''. This however does not happen when running kubedock from within a container (Tekton pipeline) with the privileges suggested in the README (I added the "update" verb for leases).
from kubedock.
joyrex2001
commented on July 1, 2024
Fixed the documentation with regards to RBAC required for locking. After adding the 'update' verb, I could not reproduce with kubectl run and kubectl delete pod. Removed the other verbs as well, as the lease implementation doesn't require these.
from kubedock.
thomascube
commented on July 1, 2024
Thanks for updating the docs. I can confirm that the lease is returned correctly (having holderIdentity: '') when running kubedock as a pod and after deleting it. However, we're running kubedock inside a Tekton task, which essentially also is a pod, but for some reason, the lease remains unchanged once the process ends. This just leads to a delay when the next run starts and has to wait until it can aquire the lease. I'll keep on digging...
from kubedock.
joyrex2001
commented on July 1, 2024
Can you share the logs during start & exit of kubedock?
from kubedock.
thomascube
commented on July 1, 2024
There's a difference in the shutdown behavior, depending how we run kubedock and how we terminate it.
When running as individual Pod (or Sidecar container inside a Tekton pipeline), the lease is correctly released.
When we start kubedock inside a running container and then stop it explicitly with kill $(pidof kubedock), the lease remains unchanged. I tried different term signals with the kill command (none, 1, 2, 4) but no difference. Here are the logs for both cases just described:
Logs when stopped with kill $(pidof kubedock)
$ kubedock server --lock -n tekton-dev --reverse-proxy --unix-socket /tmp/kubedock.sock -v 4 &
I0724 10:28:26.456072 703 main.go:28] kubedock 0.11.0 (20230524-110633)
I0724 10:28:26.546374 703 main.go:105] kubernetes config: namespace=tekton-dev, initimage=joyrex2001/kubedock:0.11.0, ready timeout=2m0s
I0724 10:28:26.547537 703 leaderelection.go:245] attempting to acquire leader lease tekton-dev/kubedock-lock...
I0724 10:28:26.562013 703 leaderelection.go:255] successfully acquired lease tekton-dev/kubedock-lock
I0724 10:28:26.562089 703 main.go:85] new leader elected: 4dec02674662
I0724 10:28:26.562333 703 main.go:129] reaper started with max container age 1h0m0s
I0724 10:28:26.562426 703 main.go:75] enabled reverse-proxy services via 0.0.0.0 on the kubedock host
I0724 10:28:26.562517 703 main.go:102] default image pull policy: ifnotpresent
I0724 10:28:26.562945 703 main.go:107] using namespace: tekton-dev
...
I0724 10:30:10.191123 703 main.go:175] exit signal recieved, removing pods, configmaps and services
Lease after kill
kind: Lease
apiVersion: coordination.k8s.io/v1
metadata:
name: kubedock-lock
namespace: tekton-dev
spec:
holderIdentity: 4dec02674662
leaseDurationSeconds: 60
acquireTime: '2023-07-24T08:28:26.547551Z'
renewTime: '2023-07-24T08:30:06.744343Z'
leaseTransitions: 12
Logs when running as Pod (Tekton Sidecar)
... or when starting kubedock from my workstation and stopping with ctrl-c)
$ kubedock server --lock -n tekton-dev --reverse-proxy --unix-socket /tmp/kubedock.sock -v 4
I0724 10:36:11.109712 82406 main.go:28] kubedock 0.11.0 (20230524-110633)
I0724 10:36:11.117669 82406 main.go:105] kubernetes config: namespace=tekton-dev, initimage=joyrex2001/kubedock:0.11.0, ready timeout=1m0s
I0724 10:36:11.119303 82406 leaderelection.go:245] attempting to acquire leader lease tekton-dev/kubedock-lock...
I0724 10:36:11.474071 82406 main.go:85] new leader elected: 4dec02674662
I0724 10:37:15.758226 82406 main.go:85] new leader elected: 377cd6884941
I0724 10:37:15.758261 82406 leaderelection.go:255] successfully acquired lease tekton-dev/kubedock-lock
I0724 10:37:15.761447 82406 main.go:129] reaper started with max container age 1h0m0s
I0724 10:37:15.762991 82406 main.go:75] enabled reverse-proxy services via 0.0.0.0 on the kubedock host
I0724 10:37:15.763410 82406 main.go:102] default image pull policy: ifnotpresent
I0724 10:37:15.763520 82406 main.go:105] service account used in deployments: default
I0724 10:37:15.763547 82406 main.go:107] using namespace: tekton-dev
...
I0724 10:38:32.251400 82406 main.go:175] exit signal recieved, removing pods, configmaps and services
I0724 10:38:32.432919 82406 main.go:82] lost lock on namespace tekton-dev
Lease after pod termination
kind: Lease
apiVersion: coordination.k8s.io/v1
metadata:
name: kubedock-lock
namespace: tekton-dev
spec:
holderIdentity: ''
leaseDurationSeconds: 1
acquireTime: '2023-07-24T08:38:32.371780Z'
renewTime: '2023-07-24T08:38:32.371780Z'
leaseTransitions: 13
from kubedock.
joyrex2001
commented on July 1, 2024
I worked a bit on the shutdown code, which might fixed your issue as well. Can you retest?
from kubedock.
thomascube
commented on July 1, 2024
Great! I can confirm that with the latest version from Git master (tested with 24f9ef4, the lease is now released as expected when terminating the kubedock process as described earlier.
from kubedock.
joyrex2001
commented on July 1, 2024
Awesome, thanks for confirming. I will close this issue, and the fix will be included in the next release.
from kubedock.
Related Issues (20)
- Providing Kubedock executable binaries as a Jar File HOT 2
- Invalid state.status = "Up"
- Port forward not working with full URL HOT 2
- docker-in-docker (ryuk support) doesn't work testcontainers-dotnet HOT 2
- Localstack container fails to start due to missing container id HOT 7
- Allow to set spec.activeDeadlineSeconds HOT 8
- Could not start kafka container HOT 1
- Passing environment variables, podman HOT 5
- Regression on 0.15.4 for .NET testcontainer HOT 4
- get container logs: "tail" parameter not valid HOT 3
- Error logged for services, configmaps and pods deletion with an 'Unauthorized' message HOT 8
- Lease hangs indefinitely when Kubedock quits unexpectedly HOT 6
- Change prefix-name for the orchestrated pods via an environment variable/cli argument HOT 5
- Verbosity switch is ignored with --pod-template option HOT 6
- Not able to work with testcontainer node 10.8.1 (was working on 10.6.0) HOT 5
- Reasoning for mount HOT 2
- readOnlyRootFilesystem breaks postgres.WithConfigFile HOT 1
- Support container creation when resource quotas are hit HOT 2
- Bug with concurrent `docker run` calls
- Files with --pre-archive are only copied if there are Volumes/Bind Mounts HOT 1
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from kubedock.