Comments (8)
ctrl-\ shows the stuck goroutine:
goroutine 34894 [semacquire]:
sync.runtime_Syncsemacquire(0xc820055810)
/usr/local/go/src/runtime/sema.go:237 +0x201
sync.(*Cond).Wait(0xc820055800)
/usr/local/go/src/sync/cond.go:62 +0x9b
github.com/kahing/goofys/internal.(*BufferPool).requestBuffer(0xc8200557c0, 0x0, 0x0, 0x0)
/home/khc/Code/go/src/github.com/kahing/goofys/internal/buffer_pool.go:111 +0x137
github.com/kahing/goofys/internal.(*BufferPoolHandle).Request(0xc833752330, 0x0, 0x0, 0x0)
/home/khc/Code/go/src/github.com/kahing/goofys/internal/buffer_pool.go:136 +0xb6
github.com/kahing/goofys/internal.MBuf.Init(0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0xc833752330, 0x1400000, ...)
/home/khc/Code/go/src/github.com/kahing/goofys/internal/buffer_pool.go:185 +0xa0
github.com/kahing/goofys/internal.S3ReadBuffer.Init(0x0, 0x0, 0x0, 0x0, 0xc82009a500, 0xc828a10750, 0x8210000, 0x1400000, 0x4)
/home/khc/Code/go/src/github.com/kahing/goofys/internal/handles.go:561 +0x11e
github.com/kahing/goofys/internal.(*FileHandle).readAhead(0xc828a10750, 0xc82009a500, 0x3210000, 0x20000)
/home/khc/Code/go/src/github.com/kahing/goofys/internal/handles.go:653 +0x30d
github.com/kahing/goofys/internal.(*FileHandle).ReadFile(0xc828a10750, 0xc82009a500, 0x3210000, 0xc820516018, 0x20000, 0x20000, 0x0, 0x0, 0x0)
/home/khc/Code/go/src/github.com/kahing/goofys/internal/handles.go:713 +0x762
github.com/kahing/goofys/internal.(*Goofys).ReadFile(0xc82009a500, 0x7fece9a0d648, 0xc8204bfa70, 0xc8203cf5c0, 0x0, 0x0)
/home/khc/Code/go/src/github.com/kahing/goofys/internal/goofys.go:729 +0xe7
github.com/jacobsa/fuse/fuseutil.(*fileSystemServer).handleOp(0xc820110640, 0xc820135380, 0x7fece9a0d648, 0xc8204bfa70, 0x7ebba0, 0xc8203cf5c0)
/home/khc/Code/go/src/github.com/jacobsa/fuse/fuseutil/file_system.go:171 +0x9df
created by github.com/jacobsa/fuse/fuseutil.(*fileSystemServer).ServeOps
/home/khc/Code/go/src/github.com/jacobsa/fuse/fuseutil/file_system.go:109 +0x18f
It's related to the multi-thread read optimization. Looking into this. In the mean time, this can disable that feature:
diff --git a/internal/handles.go b/internal/handles.go
index f2586ff..a0af1e8 100644
--- a/internal/handles.go
+++ b/internal/handles.go
@@ -703,7 +703,7 @@ func (fh *FileHandle) ReadFile(fs *Goofys, offset int64, buf []byte) (bytesRead
fh.buffers = nil
}
- if fh.seqReadAmount >= BUF_SIZE {
+ if false {
if fh.reader != nil {
fh.inode.logFuse("cutover to the parallel algorithm")
fh.reader.Close()
from goofys.
so e0b1dad and 00e3ffb introduced a feature to autotune goofys' memory usage. When memory is tight the minimum is one goofys page (10MB) but that failed to take prefetch into account, which can use up to 10 pages (100MB) at a time. We will just hang if those requests can't be satisfied.
2016/02/20 07:28:36.054503 main.DEBUG amount of free memory: 28164096
2016/02/20 07:28:36.054548 main.DEBUG amount of allocated memory: 44302672
2016/02/20 07:28:36.054575 main.DEBUG using up to 4 10MB buffers
There are two bugs here, one is that the minimum should be higher. The other is that the free memory calculation looks off to me, we think that goofys is using 44MB but top says its RES is 361MB. Additionally we should take cached into account when we try to figure out how much we can use.
Getting a bit sleepy and my internet is a bit slow atm. Will probe it some more tomorrow.
from goofys.
Additionally we should take cached into account when we try to figure out how much we can use.
I've patched this in #58 to use Available rather than Free memory. This avoids the problem for me, though the underlying deadlock is still there and would surface under real memory pressure.
from goofys.
goofys should now fallback to serial read when there isn't enough memory. I will cut another release this week. Thanks for the report!
from goofys.
Thanks! Current master seems to be working well now for me.
from goofys.
I'm still having issues with memory usage on 0.0.5: Copying a 1GB file on a t2.nano with ~450MB free memory allocates all memory so forking new processes fails with out of memory errors after the copy process. Maybe there should be a commandline option to set the size of the buffer cache.
from goofys.
There are several problems:
maxMemToUse()
calculates existing buffer memory usage usingms.Alloc
, which is wrong, because that's all memory currently used by the whole program, not just the buffer pool.ms.Alloc
is the memory currently allocated, but this is not equal to the RSS of the process, so there might be lot's of available process memory that is not currently used, but has not yet been released to the OS by go, so this memory would not be listed as available memory.- Max buffers is only calculated each 100 buffer allocations (or 1GB of memory), which is to infrequent, something like checking every 10 allocations or 100MB of memory should be much safer.
- For some reason it divides the available memory and the allocated memory by 2, I guess the intention was to use only half of the available memory.
- Lot's of memory is allocated outside the buffer pool (I see much higher RSS than number of buffers * buffer size, even if I edit the code to limit to a single buffer). If buffers are increased a lot more memory is allocated per parallel upload that is not accounted for by the additional buffer memory. For example with a fixed buffer size of 4 uploading a 1GB file requires ~280 MB RSS, while startup RSS is ~9 MB.
It would also be good if goofys
tried to deallocate memory before exiting, because otherwise forking the unmount process will fail in OOM situations:
^C2016/03/10 01:23:49.567051 main.INFO Received interrupt, attempting to unmount...
2016/03/10 01:23:49.567546 main.ERROR Failed to unmount in response to interrupt: fork/exec /bin/fusermount: cannot allocate memory
I think overall a configuration setting to set max concurrent uploads and downloads would make more sense then trying to autodetect this and potentially allocating a lot more memory/connections than required to saturate the network link.
from goofys.
Understood that buffer pool detection is still not ideal, but seems like that is more relevant under #64 than this issue. I will respond there
from goofys.
Related Issues (20)
- Can't access subdir from top dir even as a root user
- Deprecated Azure API
- main.ERROR Unable to access '<bucket_name>': permission denied HOT 3
- Feature Request: Support s3 new checksums crc32/crc32c/sha1/sha256
- Goofys work with contabo object-storage HOT 2
- Operation to mounted Goofys S3 does not respond
- ReadDir inode leak causes memory leak
- mount with ceph rgw got error ,ceph version is 18.2.1 HOT 1
- Improvement request: set op.Handle as point to FileHandle/DirHandle
- can not compile with go1.16
- Building leads to [email protected]: retracted by module author
- certificate has expired or is not yet valid HOT 3
- Goofys mount is not reloading inside container
- Disable multipart upload HOT 1
- mounting, listing and reading file works, writing doesn't HOT 1
- In kubernetes when killing pod with goffys container, pod stuck in terminating. exit code 137. HOT 2
- Process exits with error during copy of file under goofys mount point
- Failed deleting files HOT 1
- Any luck building this on mac osx? Brew and go have both moved on HOT 2
- How to mount the Blob/Data Lake containers with different Azure Storage Accounts via /etc/fstab ? HOT 1
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from goofys.