Hi,
I'm having a weird issue that appeared only this week,
used to work without issue..
with a cPanel server on cloudlinux and cagefs enabled,
I'm sharing a directory to multiple users.
This actually seems to still work as expected..
but when the user is to be deleted, after the umount, the directory stays "busy"..
and the user's home directory cannot be removed completely unless the server is rebooted.
Here's what my fstab looks like:
/home/share/files /home/test9/files fuse.bindfs map=1015/1060:@1017/@1062 0 0
/home/share/files /home/test10/files fuse.bindfs map=1015/1061:@1017/@1063 0 0
/home/share/files /home/test11/files fuse.bindfs map=1015/1062:@1017/@1064 0 0
/home/share/files /home/test12/files fuse.bindfs map=1015/1063:@1017/@1065 0 0
/home/share/files /home/test13/files fuse.bindfs map=1015/1064:@1017/@1066 0 0
/home/share/files /home/test14/files fuse.bindfs map=1015/1065:@1017/@1067 0 0
/home/share/files /home/test15/files fuse.bindfs map=1015/1066:@1017/@1068 0 0
/home/share/files /home/test16/files fuse.bindfs map=1015/1067:@1017/@1069 0 0
#/home/share/files /home/test16/files fuse.bindfs nodev,nosuid,noatime,force-user=test16,force-group=test16 0 0
the last line is me trying different options without any success so far..
Let's try to remove the user 'test13':
1. while it is still mounted and working as expected, fuser will show:
# fuser -vm /home/test13/files
USER PID ACCESS COMMAND
/home/test13/files:
root kernel mount /home/test13/files
2. proceed with the umount
# umount /home/test13/files
I also tried those, but I'm getting the same 'busy' error at the end:
# umount -f /home/test13/files
# umount -l /home/test13/files
# fusermount -u /home/test13/files
# fusermount -z -u /home/test13/files
3. there's an extra step with cagefs enabled
at this point, root sees the mount point empty ( with a simple # ls -la /home/test13/files
)
but for the user to see it, we need to:
# cagefsctl --remount test13
or
# cagefsctl --remount-all
4. and this is where it all goes down.. removing the mount point:
# rm -rf /home/test13/files
rm: cannot remove ‘/home/test13/files’: Device or resource busy
5. the mount point is now empty, but to confirm I ran:
# umount /home/test13/files
umount: /home/test13/files: not mounted
# cat /proc/mounts | grep test13
( no output )
6. now the reason I created this issue here is because of the new output from fuser:
# fuser -vm /home/test13/files
USER PID ACCESS COMMAND
/home/test13/files:
root kernel mount /home
root 3586 f.c.. bindfs
root 3588 f.c.. bindfs
root 3590 f.c.. bindfs
root 3592 f.c.. bindfs
root 3596 f.c.. bindfs
root 3598 f.c.. bindfs
root 3600 f.c.. bindfs
these are the bindfs' PID of all the other mounts still active.
# ps aux | grep bindfs
root 3586 0.0 0.0 13616 492 ? Ss 10:33 0:00 bindfs /home/share/files /home/test9/files -o rw,map=1015/1060:@1017/@1062,dev,suid
root 3588 0.0 0.0 13616 492 ? Ss 10:33 0:00 bindfs /home/share/files /home/test10/files -o rw,map=1015/1061:@1017/@1063,dev,suid
root 3590 0.0 0.0 13616 492 ? Ss 10:33 0:00 bindfs /home/share/files /home/test11/files -o rw,map=1015/1062:@1017/@1064,dev,suid
root 3592 0.0 0.0 13616 492 ? Ss 10:33 0:00 bindfs /home/share/files /home/test12/files -o rw,map=1015/1063:@1017/@1065,dev,suid
root 3596 0.0 0.0 13616 492 ? Ss 10:33 0:00 bindfs /home/share/files /home/test14/files -o rw,map=1015/1065:@1017/@1067,dev,suid
root 3598 0.0 0.0 13616 492 ? Ss 10:33 0:00 bindfs /home/share/files /home/test15/files -o rw,map=1015/1066:@1017/@1068,dev,suid
root 3600 0.0 0.0 13616 696 ? Ss 10:33 0:00 bindfs /home/share/files /home/test16/files -o rw,map=1015/1067:@1017/@1069,dev,suid
lsof has similar output too:
# lsof | grep 'share/files'
bindfs 3586 root cwd DIR 253,2 4096 1073752629 /home/share/files
bindfs 3586 root 3r DIR 253,2 4096 1073752629 /home/share/files
bindfs 3588 root cwd DIR 253,2 4096 1073752629 /home/share/files
bindfs 3588 root 3r DIR 253,2 4096 1073752629 /home/share/files
bindfs 3590 root cwd DIR 253,2 4096 1073752629 /home/share/files
bindfs 3590 root 3r DIR 253,2 4096 1073752629 /home/share/files
bindfs 3592 root cwd DIR 253,2 4096 1073752629 /home/share/files
bindfs 3592 root 3r DIR 253,2 4096 1073752629 /home/share/files
bindfs 3596 root cwd DIR 253,2 4096 1073752629 /home/share/files
bindfs 3596 root 3r DIR 253,2 4096 1073752629 /home/share/files
bindfs 3598 root cwd DIR 253,2 4096 1073752629 /home/share/files
bindfs 3598 root 3r DIR 253,2 4096 1073752629 /home/share/files
bindfs 3600 root cwd DIR 253,2 4096 1073752629 /home/share/files
bindfs 3600 root 3r DIR 253,2 4096 1073752629 /home/share/files
7. i'm still not convinced bindfs is part of the issue
I tried unmounting them all.. fuser now shows:
fuser -vm /home/test13/files/
USER PID ACCESS COMMAND
/home/test13/files
root kernel mount /home
and yet i'm still unable to rm the mount point.. unless there's an ugly reboot involved
# rm -rf /home/test13/files
rm: cannot remove ‘/home/test13/files’: Device or resource busy
and now...
it used to work perfectly last week...
bindfs hasn't been updated or changed whatsoever since..
and the kernel hasn't been updated either for the path month.
there has been an update to cpanel.. ( 60.0.29 -> 60.0.30 ),
but I cannot see anything that looks suspicious in the changelog yet..
is there anything fishy in my steps above ? am I missing something obvious ?
at this point, even just a confirmation that bindfs has nothing to do with it would be helpful.
a tiny bit more info:
# bindfs --version
bindfs 1.13.8
# uname -a
Linux example.com 3.10.0-714.10.2.lve1.4.79.el7.x86_64 #1 SMP Thu Jan 4 13:30:50 EST 2018 x86_64 x86_64 x86_64 GNU/Linux