koalaman / linuxatemyram.com Goto Github PK

I'm running SuSE Linux Enterprise Server 12 SP1.

Your post "Linux ate my RAM! is MOSTLY true/correct/accurate with a couple of notable exceptions.

But before I get into that - my system specs:
Supermicro 6027TR-HTRF
4 dual socket nodes, with each node:
2x Intel Xeon E5-2960 (8-core, 2.9 GHz stock, full load TurboBoost 3.3 GHz, max. TurboBoost 3.6 GHz, HTT disabled)
8x Saumsung 16 GB DDR3-1333 ECC Registered 4R RAM (running at DDR3-800 because it's 4R)
1x Intel 540 Series 1 TB SATA 6 Gbps SSD
1x HGST 3 TB SATA 6 Gbps 7200 rpm HDD

(total memory per node: 128 GB, total memory for the entire system: 512 GB)

Linux caching pagecache and slab objects can have a detrimental impact on application performance because it sees that memory that has been used by pagecache and slab objects = used as you pointed out.

# echo 200 > /proc/sys/vm/vfs_cache_pressure

which is the setting that I am testing right now, after have executed:

# echo 3 > /proc/sys/vm/drop_caches

and on two of the nodes, they're currently caching between 82 GiB to 94 GiB to RAM right now.

So - long story short, that REALLY sucks when you have an application that needs about 73 GiB of RAM to run.

This site is just a very good information.
But do you have more links to add to the "When should I start to worry?" section ?

I'm in the situation where available memory is low, but the processes showed in "top -o %MEM" do not seem to use all the memory ...

So I just spent a good while trying to figure out why I was missing ~700MB+ of RAM and wasn't having any luck. Annoyingly almost every forum post or StackOverflow post would scream "it's buffers+cache!" or point users to this website, even when the user posting had clearly taken those figures into account. None of these posts were useful, it was just the same comments parroted over and over.

My situation was this: The RSS field of "ps aux" summed came to around 230MB at worst. Free completely disagreed with this:

 free -m
              total        used        free      shared  buff/cache   available
Mem:           7746         323        6686         647         737        6598
Swap:          8165          24        8141

Even adding up the fields in TOP came nowhere close to what free was claiming. The "323 used" in free was close, but why are free and available showing a deficit of over 1GB?

The answer, was tmpfs. For anyone unfamiliar, tmpfs is used to making RAM disks. Virtual media where the data is stored in RAM. On most distros, the /tmp folder of the system is a RAM disk.

Anyway, sure enough, there was a nearly 650MB file in the /tmp folder which accounted for most of this missing RAM. Once deleted, I now see...

free -m
              total        used        free      shared  buff/cache   available
Mem:           7746         323        7331           9          92        7239
Swap:          8165          12        8153

(ignore the buff/cache change, I dropped caches between runs which is why that suddenly changed)
The 323MB used reported by free still doesn't match up with the 220MB used I'm seeing by summing the RSS field of "ps aux", but at least it's a lot closer now.

TL;DR: Missing RAM? Check your RAM disk(s).

Hi, web cettificate expired as per today (https) - 2022-06-20

Title says it all.

In short, there are platforms with shared GPU/CPU RAM (like Nvidia Jetson) where this becomes problematic, because the GPU drivers won't allocate GPU memory decreasing the buffers/cached data but instead will report lower GPU memory available.
And who uses CUDA for DNN will notice that his model might fail to load in certain cases (after the system has been up for enough time to allocate more cache buffers).
The magic stanza below is often given as solution:
sync; echo 3 > /proc/sys/vm/drop_caches
but it doesn't seem to make the buffers/cached memory report close to 0....

I found that total does not equal to used+free+shared+buff/cache

and free+shared does not equal to available

so how does free calculate my RAM?

free -m    
              total        used        free      shared  buff/cache   available
Mem:           7875         889        5903         388        1082        6357
Swap:             0           0           0

On the other hand, my computer has 8GB RAM but it only says total=7875 not 8192

See https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=34e431b0ae398fc54ea69ff85ec700722c9da773

Greetings, kindly perhaps consider specifying a version number instead of a year? or say (MIGHT be on)/etc.

for instance, the version of SLES 12 SP3 i'm running still has -/+ buffers/cache row showing.
cat /etc/os-release NAME="SLES" VERSION="12-SP3" VERSION_ID="12.3" PRETTY_NAME="SUSE Linux Enterprise Server 12 SP3" ID="sles" ANSI_COLOR="0;32" CPE_NAME="cpe:/o:suse:sles:12:sp3"
SLES 12.3
free -V
free from procps-ng 3.3.9

There is some wrong information in the section: "How do I see how much free ram I really have?"

It says:
"This is your answer in megabytes."
When it actually should say:
"This is your answer in mebibytes."

Source: man free, Megabyte and Mebibyte wikipedia pages.

Hi,

Have you thought of translating this site? I would happily contribute a french translation.

Come on, its 2017 already, https the website!

https://www.ssllabs.com/ssltest/analyze.html?d=linuxatemyram.com&hideResults=on

Thanks for this website! It was really helpful in learning how the disk cache works and how to interpret memory metrics.

Could you please add the basic specifications of the hard drive and memory you used on the page with real measurements? I'm curious if those measurements were made on a spinning hard disk or an SSD, for example.

Some good specifications to share might be:

• SSD or HDD?
• SATA or NVMe connection?
• RAM type (DDR4, etc)

Older version of free dont show available ram, so this command can be used:
cat /proc/meminfo | grep '^MemAvailable:' | awk '{print $2/1024}'

Maybe you want to add it.

this is a funny website. my free in ubuntu14.04 shows an extra line of -/+ buffers/cache:.
this shows the real free memory I think.

I'd like to fork this project and create zfsatemyram but, apart from #6, I can't find any reference to project's license.

Could you add a license?

Thanks!

In the "What's going on?" section, you can read the following sentence :

This makes it looks like you are low on memory, but you are not!

However, it shouldn't be "This makes it looks like [...]" but "This makes it look like [...]"

Here are some complementary posts for clarification :

• https://ell.stackexchange.com/questions/95484/i-want-to-make-my-house-look-looks-more-beautiful
• https://www.quora.com/Is-it-correct-to-say-it-makes-something-look-bad-in-English

I have been (very) half-ass fighting this issue for a year and every time I come back to this site only to get very mad that this article is the common sentiment all over the web.

There are scenarios where the kernel will not reclaim ram from the buff/cache and quite frankly I'm not sure why the kernel (or tmpfs) considers it buff/cached ram in the first place.

tmpfs

• systemd (maybe Arch Linux specific?) automatically mounts /tmp (and some others) as tmpfs (I assumed this was still on disk...)
• tmpfs registers under buff/cache ram which IMO is misleading
• Since the kernel doesn't reclaim this ram, this leads to OOM even with several GB considered "available"

page cache

There is also another case with the page cache. I don't quite understand the scope but here's a link - https://serverfault.com/questions/288319/linux-not-freeing-large-disk-cache-when-memory-demand-goes-up

I saw issue #3 and appreciate that you put back the mention of the older format of 'free'.

However, I'm stuck using a system with the older format, and actually showing it on the web page, as it was in the first version of the site, is really helpful!

Could you please bring back the "-/+ buffers/cache:" version? It'll help those who are stuck with older versions of Linux.

Thanks!

Thank you for this website! It's really helpful since it's such a common source of confusion. One thing that's even more confusing is how this works inside a docker container. I think the principles are the same, but tools like top and free are no longer as helpful. With docker becoming more popular, it might make sense to add a section to explain how to check "How do I see how much free ram I really have?" inside a docker container.

If your applications want more memory, they just take back a chunk that the disk cache borrowed. Disk cache can always be given back to applications immediately! You are not low on ram!

This isn't true. Something is eating the RAM, and won't give it back. The use case is Chromium or Firefox with lots of tabs, then quit them. There will be additional RAM taken up than before the browser was run.

A few numbers to show what I mean:
On bootup: <2GB used, usually ~1.7GB
Open a bunch of tabs in FF/Chromium, eventually more than 8GB are used
Quit FF/Chromium, and 3.5GB are used.

What is the 3.5-1.7= 1.8GB? Let's call it X.

Is X a memory cache? A memory leak? Why do both Firefox and Chromium reproduce the exact seem results?

Opening tabs eventually completely fills my 8GB RAM, system freezes, then eventually tabs die/freeze/crash, freeing just enough memory to continue.

The correct function, that I wish Ubuntu had, would be giving whatever the hell X is back to applications when they want RAM to run. However, when I launch an app, and most of the RAM has been eaten up by X, whether it was created by either of the 2 most mainstream browsers, no RAM is reclaimed from X. It remains until I reboot.

Use of "free" and other tools has not helped me diagnose WTF X is.

Any ideas?