Comments (8)
/sig node
from kubernetes.
/triage needs-information
Can you please include a reproduction case?
Its not clear to me how one can test this.
It sounds like you are asking why are these logs not deleted? I would check the logs for kubelet log rotation and see if there are any failures.
from kubernetes.
/triage accepted
from kubernetes.
It sounds like you are asking why are these logs not deleted?
If the container isn't deleted by kubelet, it appears that the dangling link of the log to the deleted container isn't handled, causing the issue.
Here are the steps to reproduce
- Create a pod
$ cat <<EOF | kubectl apply -f -
apiVersion: v1
kind: Pod
metadata:
name: busybox-sleep
spec:
containers:
- name: busybox
image: busybox:1.28
args:
- sleep
- "1000000"
EOF
- Now we have container up, delete it manually
$ docker ps -a
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
7a35cb0df654 busybox "sleep 1000000" About a minute ago Up About a minute k8s_busybox_busybox-sleep_default_635d57dd-b5bd-421b-90e9-66750be5b260_0
$ docker rm -f 7a35cb0df654
- Error log appears
cri_stats_provider.go:694] "Unable to fetch container log stats" err="failed to get fsstats for \"/var/log/pods/default_busybox-sleep_635d57dd-b5bd-421b-90e9-66750be5b260/busybox/0.log\": no such file or directory" containerName="busybox"
The container is gone, the log file no longer exists, and a new one is up, but the link persists.
$ ls -al /var/log/pods/default_busybox-sleep_635d57dd-b5bd-421b-90e9-66750be5b260/busybox/0.log
lrwxrwxrwx 1 root root 165 Feb 21 23:47 /var/log/pods/default_busybox-sleep_635d57dd-b5bd-421b-90e9-66750be5b260/busybox/0.log -> /var/lib/docker/containers/7a35cb0df6543424d26661f6a0dcacdd7a10c5ec1d55fe20d9be3582bc4ef7dd/7a35cb0df6543424d26661f6a0dcacdd7a10c5ec1d55fe20d9be3582bc4ef7dd-json.log
$ ls -al /var/lib/docker/containers/7a35cb0df6543424d26661f6a0dcacdd7a10c5ec1d55fe20d9be3582bc4ef7dd/7a35cb0df6543424d26661f6a0dcacdd7a10c5ec1d55fe20d9be3582bc4ef7dd-json.log
ls: cannot access '/var/lib/docker/containers/7a35cb0df6543424d26661f6a0dcacdd7a10c5ec1d55fe20d9be3582bc4ef7dd/7a35cb0df6543424d26661f6a0dcacdd7a10c5ec1d55fe20d9be3582bc4ef7dd-json.log': No such file or directory
from kubernetes.
Could you try reproducing it using containerd? maybe try kind and see if you see the same behavior.
from kubernetes.
cc @haircommander @mikebrow
tagging you all for cri-stats-provider.
from kubernetes.
TBH I don't really understand if this is supported.
Code is related to CRIStatsProvider.
from kubernetes.
yeah this sounds like a bug in cri-dockerd, as it's responsible for cleaning up that link AFAIU. If you cannot reproduce with cri-o or containerd, please open an issue in cri-dockerd repo
from kubernetes.
Related Issues (20)
- Not able to install kubectl,kubelet and kubeadm in kubernates HOT 4
- KEP: Support for Cloud Native Confidential Computing: Integrity Measurement and Attestation Services HOT 3
- Job "should apply changes to a job status" fails for suites with enabled alpha features HOT 7
- Soft eviction of pods with long grace periods blocks hard evictions when under resource pressure HOT 8
- The efficiency of synchronizing Pod status too slow. HOT 5
- [Flaking Test] master-integration TestPolicyAdmission HOT 3
- [Flaking Test] [sig-storage] PersistentVolumes-local HOT 6
- [Failing Test] e2e-ci-kubernetes-e2e-[cos-gce|al2023-aws]-disruptive-canary HOT 2
- [Flaking Test] [kubetest2 ec2] [sig-network] Services should preserve source pod IP for traffic thru service cluster IP [LinuxOnly] HOT 2
- pod preemptionPolicy can not set HOT 7
- [Failing Test] ci-kubernetes-kubemark-gce-scale-scheduler | kubemark-5000-scheduler HOT 4
- HPA doesn't work because HPA reads cronjobs resources with a "completed" status. HOT 4
- DRA: CDI: maximum annotation length HOT 1
- [Failing Test] [sig-scheduling] [Feature:GPUDevicePlugin] run Nvidia GPU Device Plugin tests in ci-kubernetes-e2e-gce-device-plugin-gpu-canary HOT 6
- [ja] translation issue in "kubectl config delete-context -h" HOT 3
- Troubleshooting kubeadm (docker has enough images but k8s init still pulls the image) HOT 8
- Issues with runtime handler supported feature lookup HOT 3
- PDB views pods in Terminating state as available HOT 5
- Failure cluster [c1aa76ec...] pull-kubernetes-e2e-gce-canary consistently failing HOT 4
- `kubectl exec -i` echoes passwords HOT 3
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from kubernetes.