TestMakeUserNsManagerFailsPodRecord from pkg/kubelet/userns/userns_manager_test.go is failing about kubernetes HOT 26 CLOSED

This issue is currently awaiting triage.

If a SIG or subproject determines this is a relevant issue, they will accept it by applying the triage/accepted label and provide further guidance.

The triage/accepted label can be added by org members by writing /triage accepted in a comment.

from kubernetes.

@rata ^^

from kubernetes.

/sig node

from kubernetes.

don't see anything on linux for sure (4 hits on windows): https://storage.googleapis.com/k8s-triage/index.html?pr=1&test=TestMakeUserNsManagerFailsPodRecord

from kubernetes.

I’m wondering how to reproduce this locally. 🤔

from kubernetes.

I’m wondering how to reproduce this locally. 🤔

Steps to reproduce:

1. git clone https://github.com/kubernetes/kubernetes
2. cd kubernetes
3. go test ./pkg/kubelet/userns/ -race

from kubernetes.

@Rajalakshmi-Girish where did you run it?

Works just fine for me on Ubuntu x86_64:

$ go test ./pkg/kubelet/userns/ -race
ok  	k8s.io/kubernetes/pkg/kubelet/userns	1.241s

from kubernetes.

I can't repro locally, I don't think there is any arch specific issue either. Based on @dims comment and link I guess for some reason this test is running on Windows and os.Chmod there works differently.

This feature is linux-only, so I'll rename the test the include '_linux' and that should solve this issue.

from kubernetes.

Isn't there any other way to avoid windows picking this up? Because lot of things are linux only (not just in userns), it would reduce the noise considerably if windows was opt-in.

from kubernetes.

I don't see any signs of windows here: https://storage.googleapis.com/ppc64le-kubernetes/logs/periodic-kubernetes-unit-test-ppc64le/1759608942486360064/build-log.txt
It's not a x86_64 Linux, but not a windows for sure.

from kubernetes.

@bart0sh the link @dims shared points to windows failures.

from kubernetes.

@rata The test grid this issue has in description points to the same failure on ppc.

from kubernetes.

Let's see if this fixes it, otherwise we can remove that test too: #123388

from kubernetes.

@Rajalakshmi-Girish where did you run it?

I ran it in one of our local VMs with centos.

cat /etc/os-release
NAME="CentOS Stream"
VERSION="8"
ID="centos"
ID_LIKE="rhel fedora"
VERSION_ID="8"
PLATFORM_ID="platform:el8"
PRETTY_NAME="CentOS Stream 8"
ANSI_COLOR="0;31"
CPE_NAME="cpe:/o:centos:centos:8"
HOME_URL="https://centos.org/"
BUG_REPORT_URL="https://bugzilla.redhat.com/"
REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 8"
REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream"
[root@maxwells1 kubernetes]#

from kubernetes.

What I don't understand is why this never fail when the PR was open.

from kubernetes.

@rata

Let's see if this fixes it, otherwise we can remove that test too: #123388

How this suppose to fix a failure on PPC linux? and on x86_64 linux (I guess) mentioned by @Rajalakshmi-Girish above?

from kubernetes.

@Rajalakshmi-Girish

I ran it in one of our local VMs with centos.

which hw architecture do you run it on? Can you show the output of the arch on that machine?

from kubernetes.

Ohh, I got it! The CI is running the "go test" as root, with all the capabilities. So all test that rely on not having permissions to write somewhere, fail, as they have capabilities to bypass that.

It seems completely weird that this didn't fail before. This smells like:

• Not all of our CI runs on open PRs
• The CI was changed to run as root now

Both things can cause lot of noise and failures. If it wasn't any of those, I don't know what could have caused it.

from kubernetes.

@Rajalakshmi-Girish

I ran it in one of our local VMs with centos.

which hw architecture do you run it on? Can you show the output of the arch on that machine?

[root@maxwells1 kubernetes]# uname -a
Linux maxwells1.fyre.ibm.com 4.18.0-500.el8.x86_64 #1 SMP Wed Jun 28 00:07:07 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux
[root@maxwells1 kubernetes]#

from kubernetes.

@bart0sh @Rajalakshmi-Girish

it is not arch nor distro, it is that just some are running as root. IMHO, if not all are running as root or unprivileged, it is a bug in our CI configuration.

from kubernetes.

I have opened PR #123388 to fix the issue, but the root cause seems to be a bug in our CI.

How can we report that we started running tests on some architectures as root, with all the capabilities, and therefore tests that exercise a permission check failure can't never work in those setups?

from kubernetes.

@rata you're right, the test fails when running as root. I'd suggest to fix the test then. In shouldn't fail even if it runs with root privileges.

from kubernetes.

@bart0sh with root AND all the capabilities, you bypass lot of permission checks. I open a PR to just remove this test, as it wasn't adding a LOT of value either.

I'm most worried about the rest of the unit test (outside the userns package), as if this change was done recently, I expect several others to fail too. How can we give this feedback to the relevant people?

from kubernetes.

@rata I think @dims should know more, but suspect that it was not done recently.

from kubernetes.

If it wasn't done recently, those workers running as root didn't run when the PR was open. So something changed recently, otherwise the test would have failed (it is not flaky, it always passes as unprivileged and it always fails as root).

from kubernetes.

Ohh, I got it! The CI is running the "go test" as root, with all the capabilities. So all test that rely on not having permissions to write somewhere, fail, as they have capabilities to bypass that.

It seems completely weird that this didn't fail before. This smells like:

• Not all of our CI runs on open PRs
• The CI was changed to run as root now

Both things can cause lot of noise and failures. If it wasn't any of those, I don't know what could have caused it.

Yes, the majority of the node related tests are not run on open PRs. We usually delegate feature testing to the author of the PR by triggering presubmits. The node tests are much longer than the required jobs on the PR and they usually filter out alpha jobs.

from kubernetes.