goofys gets stuck when free memory used by filesystem cache about goofys HOT 8 CLOSED

ctrl-\ shows the stuck goroutine:

goroutine 34894 [semacquire]:
sync.runtime_Syncsemacquire(0xc820055810)
        /usr/local/go/src/runtime/sema.go:237 +0x201
sync.(*Cond).Wait(0xc820055800)
        /usr/local/go/src/sync/cond.go:62 +0x9b
github.com/kahing/goofys/internal.(*BufferPool).requestBuffer(0xc8200557c0, 0x0, 0x0, 0x0)
        /home/khc/Code/go/src/github.com/kahing/goofys/internal/buffer_pool.go:111 +0x137
github.com/kahing/goofys/internal.(*BufferPoolHandle).Request(0xc833752330, 0x0, 0x0, 0x0)
        /home/khc/Code/go/src/github.com/kahing/goofys/internal/buffer_pool.go:136 +0xb6
github.com/kahing/goofys/internal.MBuf.Init(0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0xc833752330, 0x1400000, ...)
        /home/khc/Code/go/src/github.com/kahing/goofys/internal/buffer_pool.go:185 +0xa0
github.com/kahing/goofys/internal.S3ReadBuffer.Init(0x0, 0x0, 0x0, 0x0, 0xc82009a500, 0xc828a10750, 0x8210000, 0x1400000, 0x4)
        /home/khc/Code/go/src/github.com/kahing/goofys/internal/handles.go:561 +0x11e
github.com/kahing/goofys/internal.(*FileHandle).readAhead(0xc828a10750, 0xc82009a500, 0x3210000, 0x20000)
        /home/khc/Code/go/src/github.com/kahing/goofys/internal/handles.go:653 +0x30d
github.com/kahing/goofys/internal.(*FileHandle).ReadFile(0xc828a10750, 0xc82009a500, 0x3210000, 0xc820516018, 0x20000, 0x20000, 0x0, 0x0, 0x0)
        /home/khc/Code/go/src/github.com/kahing/goofys/internal/handles.go:713 +0x762
github.com/kahing/goofys/internal.(*Goofys).ReadFile(0xc82009a500, 0x7fece9a0d648, 0xc8204bfa70, 0xc8203cf5c0, 0x0, 0x0)
        /home/khc/Code/go/src/github.com/kahing/goofys/internal/goofys.go:729 +0xe7
github.com/jacobsa/fuse/fuseutil.(*fileSystemServer).handleOp(0xc820110640, 0xc820135380, 0x7fece9a0d648, 0xc8204bfa70, 0x7ebba0, 0xc8203cf5c0)
        /home/khc/Code/go/src/github.com/jacobsa/fuse/fuseutil/file_system.go:171 +0x9df
created by github.com/jacobsa/fuse/fuseutil.(*fileSystemServer).ServeOps
        /home/khc/Code/go/src/github.com/jacobsa/fuse/fuseutil/file_system.go:109 +0x18f

It's related to the multi-thread read optimization. Looking into this. In the mean time, this can disable that feature:

diff --git a/internal/handles.go b/internal/handles.go
index f2586ff..a0af1e8 100644
--- a/internal/handles.go
+++ b/internal/handles.go
@@ -703,7 +703,7 @@ func (fh *FileHandle) ReadFile(fs *Goofys, offset int64, buf []byte) (bytesRead
                fh.buffers = nil
        }

-       if fh.seqReadAmount >= BUF_SIZE {
+       if false {
                if fh.reader != nil {
                        fh.inode.logFuse("cutover to the parallel algorithm")
                        fh.reader.Close()

from goofys.

so e0b1dad and 00e3ffb introduced a feature to autotune goofys' memory usage. When memory is tight the minimum is one goofys page (10MB) but that failed to take prefetch into account, which can use up to 10 pages (100MB) at a time. We will just hang if those requests can't be satisfied.

2016/02/20 07:28:36.054503 main.DEBUG amount of free memory: 28164096
2016/02/20 07:28:36.054548 main.DEBUG amount of allocated memory: 44302672
2016/02/20 07:28:36.054575 main.DEBUG using up to 4 10MB buffers

There are two bugs here, one is that the minimum should be higher. The other is that the free memory calculation looks off to me, we think that goofys is using 44MB but top says its RES is 361MB. Additionally we should take cached into account when we try to figure out how much we can use.

Getting a bit sleepy and my internet is a bit slow atm. Will probe it some more tomorrow.

from goofys.

Additionally we should take cached into account when we try to figure out how much we can use.

I've patched this in #58 to use Available rather than Free memory. This avoids the problem for me, though the underlying deadlock is still there and would surface under real memory pressure.

from goofys.

goofys should now fallback to serial read when there isn't enough memory. I will cut another release this week. Thanks for the report!

from goofys.

Thanks! Current master seems to be working well now for me.

from goofys.

I'm still having issues with memory usage on 0.0.5: Copying a 1GB file on a t2.nano with ~450MB free memory allocates all memory so forking new processes fails with out of memory errors after the copy process. Maybe there should be a commandline option to set the size of the buffer cache.

from goofys.

There are several problems:

• maxMemToUse() calculates existing buffer memory usage using ms.Alloc, which is wrong, because that's all memory currently used by the whole program, not just the buffer pool.
• ms.Alloc is the memory currently allocated, but this is not equal to the RSS of the process, so there might be lot's of available process memory that is not currently used, but has not yet been released to the OS by go, so this memory would not be listed as available memory.
• Max buffers is only calculated each 100 buffer allocations (or 1GB of memory), which is to infrequent, something like checking every 10 allocations or 100MB of memory should be much safer.
• For some reason it divides the available memory and the allocated memory by 2, I guess the intention was to use only half of the available memory.
• Lot's of memory is allocated outside the buffer pool (I see much higher RSS than number of buffers * buffer size, even if I edit the code to limit to a single buffer). If buffers are increased a lot more memory is allocated per parallel upload that is not accounted for by the additional buffer memory. For example with a fixed buffer size of 4 uploading a 1GB file requires ~280 MB RSS, while startup RSS is ~9 MB.

It would also be good if goofys tried to deallocate memory before exiting, because otherwise forking the unmount process will fail in OOM situations:

^C2016/03/10 01:23:49.567051 main.INFO Received interrupt, attempting to unmount...
2016/03/10 01:23:49.567546 main.ERROR Failed to unmount in response to interrupt: fork/exec /bin/fusermount: cannot allocate memory

I think overall a configuration setting to set max concurrent uploads and downloads would make more sense then trying to autodetect this and potentially allocating a lot more memory/connections than required to saturate the network link.

from goofys.

Understood that buffer pool detection is still not ideal, but seems like that is more relevant under #64 than this issue. I will respond there

from goofys.