SELinux blocking disk operations on a hostPath volume about local-path-provisioner HOT 7 OPEN

We create a selinux policy to make it work. I don't remember how we figured this out, but we have our ansible do it automatically now.

1. Create /root/localpathpoicy.te

   module localpathpolicy 1.0;
   
   require {
       type usr_t;
       type init_t;
       type container_t;
       type container_var_lib_t;
       class dir { search write add_name create remove_name rmdir setattr getattr };
       class file { create open write append read unlink setattr getattr };
   }
   
   #============= container_t ==============
   allow container_t container_var_lib_t:file { create open write append read setattr getattr unlink };
   allow container_t container_var_lib_t:dir { add_name create remove_name rmdir setattr write search };
   allow container_t init_t:dir search;
   allow container_t usr_t:dir { add_name create remove_name rmdir setattr getattr write };
   allow container_t usr_t:file { create unlink write setattr getattr };
   allow container_t init_t:file { read open };
   

2. checkmodule -M -m -o /root/localpathpolicy.mod /root/localpathpolicy.te
3. semodule_package -o /root/localpathpolicy.pp -m /root/localpathpolicy.mod
4. semodule -i /root/localpathpolicy.pp
5. semanage fcontext -a -t container_file_t "/var/lib/localpath(/.*)?" (replace /var/lib/localpath with your path on disk)
6. restorecon -R -v /var/lib/localpath

from local-path-provisioner.

This did not, unfortunately, solve the problem:

sudo chcon -R -t svirt_sandbox_file_t -l s0 /opt/anaconda/storage

from local-path-provisioner.

I was able to get the full application running this command repeatedly as the containers started up.

sudo chcon -R -t svirt_sandbox_file_t -l s0 /opt/anaconda/storage

Because three different pods are interacting with this directory, and Postgres is particularly feisty about ownership, the various operations they do to confirm ownership and set permissions causes the pods to lose access. Here's the audit log:

----
type=PROCTITLE msg=audit(11/06/2023 15:26:59.770:300) : proctitle=chmod 00700 /mnt/pgdata 
type=SYSCALL msg=audit(11/06/2023 15:26:59.770:300) : arch=x86_64 syscall=fchmodat success=no exit=EACCES(Permission denied) a0=AT_FDCWD a1=0x558dc11ee4d0 a2=0700 a3=0x7f075ab07f98 items=0 ppid=14261 pid=14317 auid=unset uid=ec2-user gid=root euid=ec2-user suid=ec2-user fsuid=ec2-user egid=root sgid=root fsgid=root tty=(none) ses=unset comm=chmod exe=/usr/bin/chmod subj=system_u:system_r:container_t:s0:c813,c975 key=(null) 
type=AVC msg=audit(11/06/2023 15:26:59.770:300) : avc:  denied  { setattr } for  pid=14317 comm=chmod name=pgdata dev="nvme0n1p2" ino=511705289 scontext=system_u:system_r:container_t:s0:c813,c975 tcontext=system_u:object_r:container_file_t:s0:c79,c834 tclass=dir permissive=0 
----
type=PROCTITLE msg=audit(11/06/2023 15:40:56.187:543) : proctitle=postgres: walwriter    
type=SYSCALL msg=audit(11/06/2023 15:40:56.187:543) : arch=x86_64 syscall=pwrite success=no exit=EACCES(Permission denied) a0=0x3 a1=0x7f86a2678000 a2=0x2000 a3=0x866000 items=0 ppid=36573 pid=36596 auid=unset uid=ec2-user gid=root euid=ec2-user suid=ec2-user fsuid=ec2-user egid=root sgid=root fsgid=root tty=(none) ses=unset comm=postgres exe=/usr/lib/postgresql/12/bin/postgres subj=system_u:system_r:container_t:s0:c250,c1019 key=(null) 
type=AVC msg=audit(11/06/2023 15:40:56.187:543) : avc:  denied  { write } for  pid=36596 comm=postgres path=/mnt/pgdata/pg_wal/000000010000000000000001 dev="nvme0n1p2" ino=58720526 scontext=system_u:system_r:container_t:s0:c250,c1019 tcontext=system_u:object_r:container_file_t:s0:c428,c438 tclass=file permissive=0

from local-path-provisioner.

OK, here's what worked for me.

1. Run sudo chcon -R -t svirt_sandbox_file_t -l s0 /opt/anaconda/storage
2. Launch one of the deployments in isolation; let it stabilize and interact with the volume.
3. Examine ls -Zd /opt/anaconda/storage to obtain the auto-assigned MCS label for the container: s0:c248,c366, for isntance
4. Add this label to the securityContext for the deployments, including the one just launched.

   securityContext:
       runAsUser: 1000
       seLinuxOptions:
         level: s0:c248,c366
   

5. Bring up the rest of the deployments.

from local-path-provisioner.

Sorry, it's me again. I've been comparing k3s and rke2 here, which ostensibly use the same local path provisioner, and k3s is not having the same issue. And it seems to be because k3s is not applying the same ownership semantics on the volume. In particular, here's what ls -lZd gives me for the Postgres volume:

drwx------. 19 ec2-user root system_u:object_r:usr_t:s0 4096 Nov  8 14:23 pgdata

Note the lack of container labels.

I'm just going to check myself that perhaps k3s doesn't enable full SELinux enforcement by default; I will see. If that's the case, no need to reply, I'll figure that out soon enough :-)

EDIT: installing k3s with --selinux produces the same behavior, so it's definitely generic

from local-path-provisioner.