loadavg and swap about lxcfs HOT 66 CLOSED

from lxcfs.

#237 :)

from lxcfs.

Any progress on the loadavg-feature?

I would say swapping in general has become kinda obsolete anyway since memory is dirt cheap and swapping has too much of a performance malus anyway.

But seeing correct loadavg is needed to monitor guests properly in order to see "whats wrong where" if the host behaves unusually. Not even lxc-info will get me these numbers for guests at this moment.

So I would love to see support for guest-specific loadavg implemented soon :)

from lxcfs.

This is so much needed

from lxcfs.

The loadavg part is implemented and opt-in now.

from lxcfs.

Would this also fix the issue that you see the CPU cores beeing busy in htop although the PID that cause the load on the CPU are not from the container where htop runs (ie; i have a container that does NOTHING but still sees a lot of load on the CPU)? Than this would be the most awesome feature; it would make LXC much more like a true hypervisor (as stated as a goal of LXD).

Even without this, this proposed fix for loadavg and the amount of available SWAP would be a great contribution. We are willing to help with this! Maybe devide the work that has to be done?

from lxcfs.

@Harvie the logic could be re-used, but the fact is that OpenVZ was doing all this at the kernel level where all the needed information is available. It's been made clear multiple times that the upstream kernel community doesn't want the files in /proc as exposed by the kernel to reflect container limits.

So the difficulty we have now is that we need to do this in lxcfs which is a userspace program (fuse filesystem) which can only see what's available to userspace. This makes it very expensive to iterate through big set of process information because that may mean opening hundreds or thousands of files (one or more per process), then parsing all of them and doing math on top of it to come up with the load number.

It's certainly not impossible, but we have to find a reasonably cheap way of doing it because having to constantly poll the process list of every container and do math on the process scheduled time is going to cause a much bigger load on your system than the container itself does :)

from lxcfs.

awesome :)

from lxcfs.

This has been merged downstream to Proxmox VE. See this thread on how to enable it:
https://forum.proxmox.com/threads/lxc-loadavg.44463/

from lxcfs.

from lxcfs.

I have implemented /proc/swaps. Please see commit tssge@50b75e3 for more information.

It needs testing, I haven't even tried compiling yet, but it should work. Maybe someone else would like to test it? Maybe @devonblzx ?

from lxcfs.

Great job! Stupid question probably, but whats the best way of testing this branch? I'm using ubuntu 15.04.

from lxcfs.

I think the best way is to just clone my branch, then follow the compilation instructions for lxcfs. Basically just clone my branch, make and make install and it should be up and running.

I'll try my commit myself in a few days, but currently I do not have the time necessary for testing.

One thing I forgot to mention: this will not work if you do not have swap accounting on for your kernel. Please use swapaccount=1 in kernel command line to enable the kernel support. Also, if memory.memsw.limit_in_bytes is not set for your LXC container, this might give very strange readings.

from lxcfs.

Thanks. I tried and get stuck here while doing the ./configure script (according to https://linuxcontainers.org/lxcfs/getting-started/ while using your fork)

checking for main in -lpthread... yes
./configure: line 12341: syntax error near unexpected token GLIB,' ./configure: line 12341:PKG_CHECK_MODULES(GLIB, glib-2.0)'

Tried installing every glib/dev/ package available but to no avail:

apt-get install make automake gcc autotools-dev libtool libglib2.0-dev intltool

Any ideas?

from lxcfs.

I'll work on the code, it is quite broken actually (I tested it). I was able to compile it however, using the instructions, not sure how to help with that.

from lxcfs.

I don' t know of any progress although it is a very important feature IMHO (especially the loadavg).

Swapping would also be nice because sometimes it simply is needed and clients do expect to have it. A nice solution would be to place swapspace on a seperate SSD to prevent system impact and to use block I/O cgroup to make sure everybody gets its share. Or in a less ideal (non seperate swapdrive) system; cap the usage of swap altogether to prevent extreme system slowdown. This is relatively simple with cgroups i guess. But still, without a way to see per container the swapusage, it would not really add value and we cannot monitor in a good way.

from lxcfs.

I have already implemented swap usage counting on htop and top, please see commit 4622ad7 It has already been merged quite a while ago.

This however does not mean that /proc/swaps works, but that is only used for things like swapon etc. I have implemented that as well, but not perfected.

Also, I have no idea how to implement load average. In my honest opinion it needs kernel support. I could program that as well, but I saw some older patches which were not liked and actually that led to creating lxcfs I think so yeah... hard problem.

from lxcfs.

@tssge Good work! Thanks. I see that it is not in any distribution yet, thats why i didn't see it. I'll give it a try using https://linuxcontainers.org/lxcfs/getting-started/

For the loadavg, would we need some sort of /proc/stat emulation?? Couldn't this be done also with FUSE approach? I really wonder how Canonical is planning to do this. Without loadavg it is simply not workable if you plan on giving clients full server access. They would not know how heavy their VPS is used. Although through /sys/fs/cgroup/cpu/lxc/my-container/cpuacct.usage and such we could make a webbased overview for them.

I tried https://linuxcontainers.org/lxd/try-it/ to see if LXD somehow got around this, but is doesn't seem so. Although i must say the demo with nested containers is cool!

from lxcfs.

from lxcfs.

@hallyn I had sent my patch to the mailing list.

from lxcfs.

What patches are you talking about? Am I missing something? Something I should sign?

I can sign, just tell what.

from lxcfs.

@tssge Could you please rework on the patch at https://lists.linuxcontainers.org/pipermail/lxc-devel/2016-January/013326.html and send it again with an added Signed-off-By?

from lxcfs.

I signed it off and sent to the mailing list.

from lxcfs.

Thanks,

@nehaljwani, are you willing to take that and push a new PR against the current lxfs source?

from lxcfs.

@hallyn I was just about to send the PR, when I saw 70dcc12. Missed by by an hour :)

from lxcfs.

On Tue, Feb 09, 2016 at 12:52:51PM -0800, Nehal J Wani wrote:

@hallyn I was just about to send the PR, when I saw 70dcc12. Missed by by an hour :)

Oh, sorry. Please do double-check that I didn't get anything wrong.
(I fixed a fd and memory leak inline as well)

from lxcfs.

Now that swaps seems to be merged, any idea on how the loadavg should be done? I don't know where to start really.

from lxcfs.

On Tue, Feb 09, 2016 at 09:55:39PM -0800, Teemu Grönqvist wrote:

Now that swaps seems to be merged, any idea on how the loadavg should be done? I don't know where to start really.

I've heard no good ideas so far.

from lxcfs.

Disclaimer: I have no real experience hacking lxc / kernel stuff.

As far as I understand, load average is (i simplify) calculated using the count of processes that wait on I/O and how long they do so. It actually is quite simple to calculate, if I believe https://en.wikipedia.org/wiki/Load_%28computing%29

Cant you just use that to calculate the same load average, but just for processes in the given lxc guest namespace and then overriding /proc/loadavg? The latter is the hard part I guess, but until we have a solution for that, we could just use /proc/loadavg_lxc or that kind. That way, actually could find the culprit guest that caused loadavg spikes on the host system (because we can monitor /proc/loadavg_lxc inside guests) which is currently next to impossible after the fact.

from lxcfs.

Re: loadavg

I haven't really been involved with this thread much since I opened it about a year ago, but the only idea I could think of is constant polling with some sort of database, as suggested in the OP. The load average only needs to consist of the last 600 seconds (10 minutes), so depending on the poll frequency, we can generate a pretty good estimate by storing the number of running processes in the container and generating an average of the polls from the last 60 seconds, 300 seconds, and 600 seconds everytime loadavg is read inside the container.

I don't have a system in front of me at the moment, but is there a cgroup param for processes in a running or i/o wait state?

The problem with this solution, of course being that if you have lots of containers, there is a lot of polling going on, but without constant polling, I don't see of a way to find out how many running processes there were x seconds ago to generate an average.

from lxcfs.

from lxcfs.

What about kernel support? I myself have done some kernel code before and might be able to rip this off of OpenVZ. Any idea if they'd like a patch that counts cgroup loadavg in the kernel?

from lxcfs.

Quoting Teemu Grönqvist ([email protected]):

What about kernel support? I myself have done some kernel code before and might be able to rip this off of OpenVZ. Any idea if they'd like a patch that counts cgroup loadavg in the kernel?

Since there is effectively no good cheap way to do it in userspace, it may be doable.

I'd encourage you to proceed :)

Thanks!

from lxcfs.

Hi, what is the status of this feature ?

from lxcfs.

Would also know if someone is working on it?

from lxcfs.

@TheNuclearCat - not that I know of. Are you considering it?

from lxcfs.

@hallyn I would, but unfortunately I have not enough knowledge yet :)

from lxcfs.

This is issue as our clients have 1-2 CPU containers on our 48 CPU server. On such server it's perfectly OK to have loadavg of 30 (because it's lower than processor core count), but it looks really bad in 1 or 2 CPU container, which might make clients doubt about quality of our service... For example on OpenVZ they calculate loadavg individualy for each container, which looks much better and is even useful.

from lxcfs.

We know the reason for wanting this, we just need someone to find a way to implement the logic needed in lxcfs :)

from lxcfs.

Is this something that Docket/RKT has overcome, or do they share the same issue because of the underlying use of cgroups/namespaces?

from lxcfs.

@stgraber is it possible to partialy reuse OpenVZ code? It should be GPL as it's part of kernel.

from lxcfs.

I'm also looking forward for this problem to be fixed. It is important that the statistics (cpu info, ram, swap, average load) should be isolated for each containers, so that it doesn't confuse users

from lxcfs.

@hallyn Hey, but openvz has this implemeted. Would it be hard to port from openvz to lxc?
But it seems that they probably handle this in kernel space...

IRC guys have given me some clues, but i think it needs further research...

https://github.com/OpenVZ/vzkernel/blob/branch-rh7-3.10.0-123.1.2-ovz/kernel/ve/ve.c
https://github.com/OpenVZ/vzkernel/blob/branch-rh7-3.10.0-123.1.2-ovz/fs/proc/loadavg.c
https://github.com/OpenVZ/vzkernel/blob/branch-rh7-3.10.0-123.1.2-ovz/kernel/fairsched.c

especially the function fairsched_show_loadavg() in kernel/fairsched.c might point you to code that calculates loadavg per cgroup.

from lxcfs.

@stgraber they don't want to change behaviour of /proc/loadavg or they don't want to expose it at all?
Like they have some reasons not to add per cgroup statistics to completely new files like /proc/cgroup/lxc/214/ns/stats ? Is there some good reason not to have this in kernel? Is it possible to solve this problem easily using kernel module?

from lxcfs.

New files under /proc are heavily frowned upon.

Actually what might be doable is a new cgroup subsystem, which just tracks the load for each cgroup.

I'm also not sure whether anyone has ever considered just adding a "cgroup.loadavg" type file, allowing the kernel to give us the info for any cgroup (and its descendents).

I think that might be the best place to start here.

from lxcfs.

from lxcfs.

from lxcfs.

Hi guys. Is there an update on how to proceed with this? I've seen comments regarding this issue in other places (lxcfs @hallyn #40) as well. Is there any workaround for this in the meantime, maybe installing some package in the containers? In my case specifically the problem seems to be CPU usage being reported by top / htop. Is there any replacement for top / htop that would account for that for usage within LXC containers?

The problem we are having is that containers that "share" the same cpu cores are reporting increased CPU usage when we start processes in another Container which seems erroneous since independent users should not have access to CPU usage from the host.

from lxcfs.

@amostech Not erroneous, just the difference between a container and virtual machine (containers share the same kernel). LXCFS tries to emulate various counters but if you read through this thread, you'll see there is no efficient way to calculate load numbers for containers at this time.

from lxcfs.

My conclusion after using containers for a decade is that containers (docker,lxc, openvz etc) are simply not a "true"-VM. It is an amazing technology for developing and offering a SAAS product though! Unbeatable in CPU and MEM efficiency and ease of deployment. And within a SAAS solution these problems discussed here are far less problematic and cgroups should give perfect way of isolating things.

So IMHO containers not suitable for true VM "hosting". When clients expect a actual VM, always go for KVM/Xen/VMware.

And don't forget that all containers have the problem of missing sys_rawio permisison in kernel because of security (you can enable this, but at a serious security cost). This means that the max IOPS generally are about 1000 (HDD speed). Maybe with a good CPU you can boost this to maybe 10.000 by wasting CPU time on checking permissions. But you will never get 100.000 (SSD) - 400.000 (NVME PCIe) with low CPU directly via the controller. Only solution is probably integrating the (Intel VT or AMD-V) features.

This in mind, maybe it is not very useful to try to go to deep in something that is not going to work anyway. Maybe just accept the downfalls and enjoy the good sides as well and use accordingly.

from lxcfs.

And don't forget that all containers have the problem of missing sys_rawio permisison in kernel because of

That's not true. Containers can absolutely have CAP_SYS_RAWIO.

from lxcfs.

You are right. Sorry for the bad info. This was only for privileged old containers as it seems (https://www.nccgroup.trust/globalassets/our-research/us/whitepapers/2016/june/container_whitepaper.pdf). Still strange that CAP_SYS_RAWIO is disabled on OpenVZ and that it causes low IOPS. But this is probably not the place to discuss this.

from lxcfs.

in htop i see some basic stats (CPU%, CPU time, etc..) for indivudal processes. you can take these numbers for all processes of lxc container sum them up and present them. Does not have to be really precise, still better than nothing. Eg.:

Host is running 20 containers and has loadavg of 10:

One of the containers has following processes:

apache 3% CPU
postfix 5% CPU
SUM 3+5 = 8% CPU

Loadavg shown in container can be host load * SUM of %CPU, which is 8% of 10 = loadavg of 0.8.

These are values that anyone can easily get from unmodified kernel to do the math (at least htop can do it no problem).

This is not very accurate, but the output would make much more sense to the customers who don't have any idea about activity in 19 other containers. (They will not see huge load while not having any processes running).

With some modifications you would be able to apply this calculation to calculate % usage of single CPU core, but you have to calculate it just from the processes running on that core (that's different from loadavg, which is global for all cores).

This will allow to easily tell which container is loading the server. Right now i have nagios checking loadavg on all of our containers (along with other nrpe checks). Once host get overloaded all of the containers start flashing red in nagios. That really sux, because you don't know which of the containers caused the problem.

This solution is more of heuristic, but can kinda solve the problem. Later it can be replaced by proper algorithm once we'll be able to get the right counters from the kernel.

from lxcfs.

@brauner cool! which commit?
update: b04c865

from lxcfs.

Excuse me, how do I get loadavg support? I have the latest version of lxcfs from the official repository ubuntu 18.04.1, but it does not have loadavg

from lxcfs.

@PowerMakc how do you run your lxcfs? #237 states you must specify the -l key specifically.

from lxcfs.

I do not run it separately, I'm guided by the lxcfs description line - "use LXC 1.1 in which case it should work automatically". My version of LXC 3.0.1. Perhaps I'm mistaken that it should work automatically.

from lxcfs.

Perhaps I'm mistaken that it should work automatically.

That phrase means lxc supports systemd-based containers starting v1.1

from lxcfs.

in man lxcfs, the -l switch is not described, and attempts to use lxcfs -l /var/lib/lxcfs do not lead to anything. Please point me to the manual to use loadavg in LXC 3.

from lxcfs.

What lxcfs version are you running?

from lxcfs.

lxcfs --version
3.0.1
and cont. config:

lxc.log.file = /var/log/lxd/day/lxc.log
lxc.log.level = warn
lxc.console.buffer.size = auto
lxc.console.size = auto
lxc.console.logfile = /var/log/lxd/day/console.log
lxc.mount.auto = proc:rw sys:rw
lxc.autodev = 1
lxc.pty.max = 1024
lxc.mount.entry = /dev/fuse dev/fuse none bind,create=file,optional
lxc.mount.entry = /dev/net/tun dev/net/tun none bind,create=file,optional
lxc.mount.entry = /proc/sys/fs/binfmt_misc proc/sys/fs/binfmt_misc none rbind,create=dir,optional
lxc.mount.entry = /sys/fs/fuse/connections sys/fs/fuse/connections none rbind,create=dir,optional
lxc.mount.entry = /sys/fs/pstore sys/fs/pstore none rbind,create=dir,optional
lxc.mount.entry = /sys/kernel/debug sys/kernel/debug none rbind,create=dir,optional
lxc.mount.entry = /sys/kernel/security sys/kernel/security none rbind,create=dir,optional
lxc.mount.entry = /dev/mqueue dev/mqueue none rbind,create=dir,optional
lxc.include = /usr/share/lxc/config/common.conf.d/
lxc.arch = linux64
lxc.hook.pre-start = /usr/lib/lxd/lxd callhook /var/lib/lxd 18 start
lxc.hook.post-stop = /usr/lib/lxd/lxd callhook /var/lib/lxd 18 stop
lxc.tty.max = 0
lxc.uts.name = day
lxc.mount.entry = /var/lib/lxd/devlxd dev/lxd none bind,create=dir 0 0
lxc.apparmor.profile = lxd-day_</var/lib/lxd>//&:lxd-day_<var-lib-lxd>:
lxc.seccomp.profile = /var/lib/lxd/security/seccomp/day
lxc.idmap = u 0 296608 65536
lxc.idmap = g 0 296608 65536
lxc.environment = http_proxy=
lxc.cgroup.memory.soft_limit_in_bytes = 8589934592
lxc.cgroup.cpu.cfs_period_us = 100000
lxc.rootfs.path = dir:/var/lib/lxd/containers/day/rootfs
lxc.net.0.type = veth
lxc.net.0.flags = up
lxc.net.0.link = lxdbr0
lxc.net.0.hwaddr = 00:16:3e:d5:03:1e
lxc.net.0.name = eth0
lxc.mount.entry = /var/lib/lxd/shmounts/day dev/.lxd-mounts none bind,create=dir 0 0

from lxcfs.

if you have the right version you should see the -l switch in the output of lxcfs -h

b04c865

-	fprintf(stderr, "lxcfs [-f|-d] [-p pidfile] mountpoint\n");
+	fprintf(stderr, "lxcfs [-f|-d] -l [-p pidfile] mountpoint\n");

from lxcfs.

I do not understand :) I used this https://linuxcontainers.org/lxcfs/getting-started/#building-and-running-lxcfs-from-git, downloaded the git version, compiled and ./lxcfs --version shows me 3.0.0 and in output lxcfs -h is "-l use loadavg", but now I have 3.0.1, which was installed with LXC and in lxcfs -h there is no "-l use loadavg".
Maybe this is due to the recent update from 16.04 to 18.04.1?
If I try to run git version 3.0.0
sudo ./lxcfs -s -f -o -l allow_other /var/lib/lxcfs/
then I get
lxcfs.c: 84: start_loadavg: load_daemon fails: /usr/local/lib/lxcfs/liblxcfs.so: undefined symbol: load_daemon

from lxcfs.

Because 3.0.0 from git is not the same as 3.0.0 release.

from lxcfs.

I need to delete the standard lxcfs and use the only git version?
How to fix lxcfs.c: 84: start_loadavg: load_daemon fails: /usr/local/lib/lxcfs/liblxcfs.so: undefined symbol: load_daemon?

from lxcfs.

I am running 3.18 from snap and lxcf does not show the -l parameter ? What am I doing wrong ?

$ lxd --version
3.18
$ lxcfs -h
Usage:

lxcfs [-f|-d] [-p pidfile] mountpoint
  -f running foreground by default; -d enable debug output 
  Default pidfile is /run/lxcfs.pid
lxcfs -h

from lxcfs.