Goofys consumes all memory with extended concurrent transfers about goofys HOT 35 CLOSED

quoting @felixbuenemann:

a fixed buffer size of 4 uploading a 1GB file requires ~280 MB RSS, while startup RSS is ~9 MB

Effectively that's the same as the proposed --max-threads=4, but we are using way more memory than expected and I would like to understand why. Something like --max-threads may still be necessary for #66 but I think that only partially overlaps with this issue.

Anyways, I did a bit more digging tonight. Looks like Go's GC behavior partially explains this. If I cap the buffer size to 1, RSS is around ~50MB, 4 -> ~120MB, 8 -> ~200MB. If I let goofys idle for a while memory size goes back down to close to initial RSS, which looks to be expected behavior: http://stackoverflow.com/a/14586361/3879341 .

Since goofys is mostly IO bound and memory allocation performance is rarely a concern, I will make it more aggressively return memory to the OS on file close and also before we call umount, which should address some of the issues here. After that, as far as this issue is concerned, is to improve the available memory detection. Are we on the same page or are there other issues that I missed?

from goofys.

There's always the option of having automatic tuning as the default and a config switch to override it, when needed. I don't thin it's impossible to get the auto-tuning right, I just think it's pretty hard to implement correctly and have it react to changing memory conditions in a timely matter.

Btw. goofys has by far the best performance and latency of all the s3 filesystems I tried out. So if the memory issues are sorted out, it's a great alternative to existing solutions like s3fs.

from goofys.

do you know approximately how many transfers it took for this to happen? goofys rely on Go's GC although it's possible that we have some dangling references somewhere. Do the clients always read the complete files and close the files afterwards?

goofys tries to detect amount of available memory to use for read prefetch. --debug_fuse will now dump how much memory goofys thinks there is. We re-calculate that on every 100 10MB allocations.

from goofys.

Regarding the number of transfers, I think it's about 200 files.
The client is a bash script that uses curl to upload files from s3 to a local web app, so I have no direct control over the close aspect.
Our QA is looking for this problem now. I'll add the --debug-fuse into the default settings until we get some more data on this. - thanks

from goofys.

note that we only dump the memory log lines in master and not in the release.

I am a little confused, is goofys used for reads or writes?

from goofys.

Yep, I saw the master commit. Will build from source & retry.
Only for reads. We read from the goofys mounted s3 bucket and write to a local spring webapp using curl.

from goofys.

@felixbuenemann hit a similar problem #57 (comment) . The original assumption is that most of the memory is used by buffer pool (which despite its name, actually no longer keeps references to allocated buffers, and is only used to keep track of maximum amount of memory to use at once). Adding a maximum concurrent upload/download may not help unless we know where the extra allocations are coming from. Agreed that the detection heuristics can use some improvements.

from goofys.

One option could be to provide a --max-threads= option to the executable to limit the number of concurrent transfers.
This would mitigate buffering issues by making clients to block if too many requests were still pending.
It may actually help the overall throughput by limiting the number of connections sharing the bandwidth.

from goofys.

I'm not sure that the current behavior of grabbing as much available memory as possible for buffers is viable for a filesystem, which is not something you expect to eat up lots of memory. I think the memory usage should be very predictable or you could easily run into issues where your systems seems to work fine and then you upload or download a big file and suddenly most of the memory gets allocated to goofys and the kernel starts dropping caches to free up memory.

Instead of auto-tuning concurrency based on available memory I would much prefer a manual setting and a README entry that tells me the formula to calculate memory usage at max concurrency (something like multi-part size * concurrent connections). That way I could trade-off between low memory usage and saturating a 10 GBps link.

I think this would also greatly simplify the current logic around how many buffers to allocate and when to grow or shrink them.

from goofys.

Most filesystems indirectly use as much memory as possible via the page cache. Any time you process a big file via any other filesystem kernel would drop caches to make space for the big file (O_DIRECT/POSIX_FADV_DONTNEED notwithstanding).

My experience is that people rarely apply the correct tunables and even if I allow tunables, it should still by default do something reasonable. Agreed that this is not easy to get right especially in the fuse context, but goofys is all of 6 months old and I want to give this some more thoughts before giving up.

from goofys.

Completely agree with the @felixbuenemann, goofys is by far the best option. I tried it as an alternative to s3fs. Despite some issues, no one wants to use s3fs anymore.

I was able to get the debug logging setup on one of our QA VMs. Anything specific I should look for in the logs?

from goofys.

you should see groups of these 3 logs:

        log.Debugf("amount of available memory: %v", m.Available)
        log.Debugf("amount of allocated memory: %v", ms.Alloc)
        log.Debugf("using up to %v %vMB buffers", maxBuffersGlobal, BUF_SIZE/1024/1024)

As @felixbuenemann pointed out in #57 (comment), ms.Alloc is not a good approximation of how much memory we are using. Maybe it should be ms.Sys instead? Also, to factor in go's overhead, maybe the heuristic should be something like:

max := uint64(m.Available+ms.Sys) / 2
overheadPerBuf := ms.Sys / pool.numBuffers
maxBuffersGlobal := MaxUInt64(max/overheadPerBuf, 1)

from goofys.

I am having trouble reproducing this, could one of you take a look at

Note that the test case requires a version of bash that has wait -n, which is in ubuntu 14.04 but not on the default amazon ami.

from goofys.

Looks like that should have replicated the situation. I haven't heard any new feedback from our QA on this either.
Just a random thought, could this be exacerbated by network errors? IOW, can this only be replicated when the network layer less than 100% reliable?

from goofys.

Which ec2 machine type are you using, if you are using ec2?

Network errors should manifest in slower read speed, which means effectively we will need to hold on to the buffers for longer. For reference, the test I did read 300 * 300MB in ~17 minutes, which means the effective bandwidth was 86MB/s.

from goofys.

Also, what version of go are you building goofys with?

from goofys.

m4.xlarge
I was using the pre-built binaries but I had to build my own to get the logging update. This was using: go version go1.5.2 linux/amd64

from goofys.

I was using the Ubuntu Xenial packages of go 1.6 for amd64 (golang-1.6-0ubuntu3) on a t2.nano instance running Ubuntu Trusty. The t2.nano instances seem to have ~400MBps bandwidth at least for traffic bursts. I don't think there are any network layer problems given both the instance and the bucket where in the eu-central-1 region. Network latency to the S3 endpoint was consistently below 1 ms.

@kahing Maybe you should try on a memory constrained instance like a t2.nano instead of the huge c4.xlarge. The t2.nano has ~450 MB available memory after booting trusty.

from goofys.

@kahing hello :) first of all thank you for your great works !
recently, i have same problem with @bedge
Out of memory: Kill process 2723 (goofys) score 845 or sacrifice child
Killed process 2723 (goofys) total-vm:1492856kB, anon-rss:886804kB, file-rss:0kB
when i was used s3fs, i had same problem. i solved this problem using "stat_cache_expire=300"
i use t2.micro ec2 server. so i can use only 400600MB real memory.
i want to keep using goofys (it is amazing fast!) so, i need to make clear this problem.
which information do you need more ? sorry i'm not good in english.

from goofys.

Sorry, work's been getting busy. I will try to look at this again this weekend

from goofys.

There's definitely some memory funkiness going on. Using t2.nano and the linked testcase goofys would run out of memory very soon even if I hardcode the max buffers to 4. no obvious leak from GODEBUG=allocfreetrace=1. Setting GOGC=10 helps but only if max buffers is set to a smallish value. I am inclined to blame Go's GC here.

from goofys.

@kahing Do you have any new ideas how to fix this?

I tried current master on a t2.small and it still eats up all the memory in a short time.
I tried riofs as an alternative, but it totally failed on "directories" with many keys.

from goofys.

I haven't looked into this in a while. Have you tried to play with GOGC and hard coding max buffers values?

from goofys.

Some refactoring I am doing for #20 will lower the per buffer size to 5MB which may help as well. How big are the files you need to create?

from goofys.

In my last test file sizes were between a few KB and 2.5 GB, but the majority of files were below 100 MB.

from goofys.

@felixbuenemann or @influz do you want to give master a try?

from goofys.

probably fixed. reopen if it still happens

from goofys.

Sorry, I'm busy working on other stuff right now, but I'll check the next time I have some time to experiments with goofys.

from goofys.

@kahing could you tag a new release with the improvements available only on master at the moment? :D

from goofys.

I will cut a new release soon but 0.0.7 already has the fix for this issue

from goofys.

Oh, nice to know, thanks!

from goofys.

Hi I just installed the goofys v0.0.16 yesterday, and although it was working great yesterday, this morning I came to find my mount point was dead and I see following message on syslog.

ug 30 12:34:20 brain-life kernel: [429916.481373] [13043]     0 13043  3497207  3044140    6575      18        0             0 goofys
Aug 30 12:34:20 brain-life kernel: [429916.481430] Out of memory: Kill process 13043 (goofys) score 720 or sacrifice child
Aug 30 12:34:20 brain-life kernel: [429916.482113] Killed process 13043 (goofys) total-vm:13988828kB, anon-rss:12176560kB, file-rss:0kB

The bucket that I am mounting has a huge number of files, and I see someone has suggested to add ""stat_cache_expire=300" somewhere. Is this still necessary?

from goofys.

How many files do you have in the biggest directory? What operations were you doing? (Create new files, read, etc)

from goofys.

I am really not sure how many files are there in this repository. It's publicly accessible read-only S3 bucket with the name of fcp-indi. When it crashed, I believe I was trying to read a file from it.

from goofys.

What was the workload like? Were you going through the files and reading all of them?

from goofys.