Comments (24)
lrowe
commented on August 23, 2024
+1. I think this could make a big difference for working with multi-gigabyte files.
from goofys.
kahing
commented on August 23, 2024
Is goofys currently not fast enough? Could you elaborate on your environment and use case so I can prioritize better?
from goofys.
lrowe
commented on August 23, 2024
I'm experimenting with goofyfs and riofs for running various validation utilities on genome sequence files uploaded to our S3 bucket. These files can be anything from a 1-60 GB. Some of the validation utilities use seek() so I can't simply stream from curl / aws s3 cp.
On a t2.large (a c4.xlarge performs similarly), md5sum maxes out at 5% cpu while some of the other utilities we run max out at around 25% cpu. The bounding factor is obviously the speed that s3 serves data in a single response.
(During these tests top shows riofs using ~7 cpu % and goofyfs ~30%. The support for using instance role credentials makes goofyfs more easily deployable for me.)
For us, the speed is not a showstopper, though faster would obviously be great! This is batch processing so I can probably get some more efficiency by running parallel jobs on the worker machines.
from goofys.
kahing
commented on August 23, 2024
Could you do a strace on a smaller job and post the result? Do you know what's the actual throughput you get?
from goofys.
lrowe
commented on August 23, 2024
Here's some more timings along with excerpts of the strace for cat and md5sum.
$ time curl "http://<bucket>.s3.amazonaws.com/TEST.fastq.gz" > /dev/null
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 1551M 100 1551M 0 0 51.2M 0 0:00:30 0:00:30 --:--:-- 49.0M
real 0m30.362s
user 0m0.001s
sys 0m0.861s
# Downloads in parallel using many parts.
$ time aws --region us-west-2 s3 cp s3://<bucket>/TEST.fastq.gz TEST.fastq.gz
download: s3://<bucket>/TEST.fastq.gz to ./TEST.fastq.gz
real 0m16.876s
user 0m7.916s
sys 0m8.695s
$ time cat /mnt/TEST.fastq.gz > /dev/null
real 0m50.482s
user 0m0.000s
sys 0m0.623s
# Another run, remounted in between.
$ time cat /mnt/TEST.fastq.gz > /dev/null
real 1m6.399s
user 0m0.000s
sys 0m0.639s
$ strace cat /mnt/TEST.fastq.gz > /dev/null
...
open("/mnt/TEST.fastq.gz", O_RDONLY) = 3
fstat(3, {st_mode=S_IFREG|0444, st_size=1627202877, ...}) = 0
fadvise64(3, 0, 0, POSIX_FADV_SEQUENTIAL) = 0
read(3, "\37\213\10\10\2B\377U\2\377DS39386_GTGGCC_L005_R2"..., 65536) = 65536
write(1, "\37\213\10\10\2B\377U\2\377DS39386_GTGGCC_L005_R2"..., 65536) = 65536
read(3, "\243\260&\202n&\211j\313}\267b|~\3612\267o,\311\227Xs\331Q\0I\3013\27\34|"..., 65536) = 65536
write(1, "\243\260&\202n&\211j\313}\267b|~\3612\267o,\311\227Xs\331Q\0I\3013\27\34|"..., 65536) = 65536
...
$ time md5sum /mnt/TEST.fastq.gz
458add84ee7ecfb38d399d8002c2f3aa /mnt/TEST.fastq.gz
real 1m22.392s
user 0m3.068s
sys 0m0.579s
$ strace md5sum /mnt/TEST.fastq.gz
...
open("/mnt/TEST.fastq.gz", O_RDONLY) = 3
fadvise64(3, 0, 0, POSIX_FADV_SEQUENTIAL) = 0
fstat(3, {st_mode=S_IFREG|0444, st_size=1627202877, ...}) = 0
mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7ff5ddd81000
read(3, "\37\213\10\10\2B\377U\2\377DS39386_GTGGCC_L005_R2"..., 32768) = 32768
read(3, "\376\361\1-\255\324\275A_\346\21\276a\354e\240\363\204\235\6\10\324\34\222\215\276?\302n\237\356e"..., 32768) = 32768
read(3, "\243\260&\202n&\211j\313}\267b|~\3612\267o,\311\227Xs\331Q\0I\3013\27\34|"..., 32768) = 32768
...
$ time cp /mnt/TEST.fastq.gz TEST.fastq.gz
real 0m46.084s
user 0m0.000s
sys 0m1.560s
$ sudo su
# echo 3 > /proc/sys/vm/drop_caches
# exit
$ time md5sum TEST.fastq.gz
e1eade9f7e9419bd8c2655a61d59486a TEST.fastq.gz
real 0m16.389s
user 0m2.947s
sys 0m0.244s
This is on a c4.xlarge running Ubuntu 14.04.
from goofys.
kahing
commented on August 23, 2024
I was more curious about the IO patterns for your other utilities
from goofys.
lrowe
commented on August 23, 2024
As well as verifying the md5sum and the gunzipped md5sum, we have two cases of custom validation on large files:
- Gzipped line data (fastq). The validation utility simply opens a gunzip stream. This one will also open stdin so I can potentially pipe in from
aws s3 cpwhich saves about half the time, 55s vs 1m50s. (Piping from curl is also ~55s.) - A chunked gzipped format (bam). This can seek around to read from the specific blocks, however our current usage simply reads the header block so speed is not an issue.
from goofys.
kahing
commented on August 23, 2024
The curious thing that stands out now is that according to my own benchmarks, I can get 58MB/s read with dd bs=1MB which is similar to your curl number. Can you try to use the same block size to see if you can get around the same speed? If not there maybe bigger difference between our setups.
from goofys.
kahing
commented on August 23, 2024
has #26 improved your speed here?
from goofys.
lrowe
commented on August 23, 2024
#26 fixes the small difference (~10%) between 32K and 128K block reads. (1M blocked reads are almost the same as 128K reads as they get split by the kernel down to 128K.) I'll try running on a larger instance and in another region (I''m currently using us-west-2, Oregon) to see if that makes a difference.
from goofys.
lrowe
commented on August 23, 2024
Testing with latest master goofyfs@bda497bc7862ddbad25cc79b4964fb61cc126a86 I'm seeing almost double the read speed compared to goofyfs@0a1959b702382a9a488a7f13309df7db193b28ee on a c4.8xlarge instance running Ubuntu Wily. Awesome!
from goofys.
lrowe
commented on August 23, 2024
The gap with curl is definitely closed now, with both goofyfs and curl taking ~18s to download a 1GB file, though there is still some potential in using multiple concurrent downloads as aws s3 cp s3://encodedcc-speed-test-oregon/1GB - | dd of=/dev/null bs=128K takes ~8s.
There is https://github.com/aws/aws-sdk-go/wiki/S3Manager-Downloader but that requires the ability to write to a position in a buffer rather than just a stream. The python aws-cli deals with that problem by queuing up the pending parts before writing to the stream: https://github.com/aws/aws-cli/blob/1.9.2/awscli/customizations/s3/tasks.py#L392
However that functionality probably belongs in aws-sdk-go rather than here so maybe it is worth opening a ticket there to ask them to implement it.
from goofys.
kahing
commented on August 23, 2024
I was getting ~18s before (see the benchmark result in README) so I am not sure why you only see this just now. 19ed6f9 could improve things but probably only in high latency cases.
The aws-sdk-go api is fine for concurrent read, I just haven't gotten to it yet.
from goofys.
lrowe
commented on August 23, 2024
Which region, instance type, and ami are you running on? Both my s3 bucket and ec2 instance were in the same region (Oregon).
from goofys.
kahing
commented on August 23, 2024
The setup and test scripts are all in README and in the bench directory
On Oct 29, 2015 22:46, Laurence Rowe [email protected] wrote:Which region, instance type, and ami are you running on? Both my s3 bucket and ec2 instance were in the same region (Oregon).
—Reply to this email directly or view it on GitHub.
from goofys.
kahing
commented on August 23, 2024
also regarding cpu usage, could this be because riofs uses http by default and goofys uses https?
from goofys.
lrowe
commented on August 23, 2024
http vs https seems a likely culprit for the difference in cpu usage.
from goofys.
kahing
commented on August 23, 2024
in case you care, you can disable ssl by specifying a http endpoint manually (--endpoint "http://s3-us-west-2.amazonaws.com")
from goofys.
kahing
commented on August 23, 2024
@lrowe could you try out #38 to see how well it works for you? In my tests I get roughly 100MB/s
from goofys.
lrowe
commented on August 23, 2024
After a couple of warm-up runs I'm seeing ~100MB/s with #38 too. (That's about double the speed of current master.)
However, I've seen goofys completely stall about 1/5 or 1/10 times. My dd process just seems to hang. The log seems to show three response headers but no data being read after that: https://gist.github.com/lrowe/a68b37583494597f26e7
Perhaps goofys needs to set some socket timeouts so that the responses error after a while and then retry?
from goofys.
kahing
commented on August 23, 2024
I am not able to reproduce this yet. I created a file that's the same size as yours, and then ran:
$ for i in $(seq 1 10); do goofys -f --debug_s3 --debug_fuse goofys bench-mnt/ >& xout & true; PID=$!; sleep 3; dd if=bench-mnt/ENCFF531KNS.fastq.gz of=/dev/null bs=131072; kill $PID; done
tried a couple times and there's no stall. Next time it happens for you, could you try to send SIGQUIT to goofys to get the stacktraces?
from goofys.
lrowe
commented on August 23, 2024
This is a different error, to reproduce:
In the first terminal, run goofys.
In a second terminal:
ddthe file normally.ddthe same file again, this will be fast as it is now in the fs cache.
In the first terminal notice that goofys has crashed:
2015/11/16 22:51:15.656505 fuse.DEBUG Op 0x000056ea connection.go:395] <- OpenFile (inode 6)
2015/11/16 22:51:15.656865 fuse.DEBUG OpenFile 6 2015/06/19/67365aaf-b53d-46a4-b26f-8bd61dbabd25/ENCFF531KNS.fastq.gz []
2015/11/16 22:51:15.656895 fuse.DEBUG Op 0x000056ea connection.go:474] -> OK
2015/11/16 22:51:15.656980 fuse.DEBUG Op 0x000056eb connection.go:395] <- FlushFile (inode 6)
2015/11/16 22:51:15.657376 fuse.DEBUG FlushFile 6 2015/06/19/67365aaf-b53d-46a4-b26f-8bd61dbabd25/ENCFF531KNS.fastq.gz []
2015/11/16 22:51:15.657408 fuse.DEBUG Op 0x000056eb connection.go:474] -> OK
2015/11/16 22:51:16.071473 fuse.DEBUG Op 0x000056ec connection.go:395] <- FlushFile (inode 6)
2015/11/16 22:51:16.071808 fuse.DEBUG FlushFile 6 2015/06/19/67365aaf-b53d-46a4-b26f-8bd61dbabd25/ENCFF531KNS.fastq.gz []
2015/11/16 22:51:16.071842 fuse.DEBUG Op 0x000056ec connection.go:474] -> OK
2015/11/16 22:51:16.071919 fuse.DEBUG Op 0x000056ed connection.go:395] <- ReleaseFileHandle
panic: runtime error: invalid memory address or nil pointer dereference
[signal 0xb code=0x1 addr=0x10 pc=0x4715f2]
goroutine 27364 [running]:
github.com/kahing/goofys/internal.(*FileHandle).Release(0xc830a4ac30)
/opt/goofys/src/github.com/kahing/goofys/internal/handles.go:730 +0xd2
github.com/kahing/goofys/internal.(*Goofys).ReleaseFileHandle(0xc82011a1e0, 0x7fd475d27c30, 0xc82acf41e0, 0xc82ace5990, 0x0, 0x0)
/opt/goofys/src/github.com/kahing/goofys/internal/goofys.go:716 +0xf4
github.com/jacobsa/fuse/fuseutil.(*fileSystemServer).handleOp(0xc8204e7460, 0xc820199800, 0x7fd475d27c30, 0xc82acf41e0, 0x7dd540, 0xc82ace5990)
/opt/goofys/src/github.com/jacobsa/fuse/fuseutil/file_system.go:181 +0x862
created by github.com/jacobsa/fuse/fuseutil.(*fileSystemServer).ServeOps
/opt/goofys/src/github.com/jacobsa/fuse/fuseutil/file_system.go:107 +0x18f
goroutine 1 [select]:
github.com/jacobsa/fuse.(*MountedFileSystem).Join(0xc8204f4690, 0x7fd472e86500, 0xc820012340, 0x0, 0x0)
/opt/goofys/src/github.com/jacobsa/fuse/mounted_file_system.go:42 +0x167
main.main.func1(0xc8200f2120)
/opt/goofys/src/github.com/kahing/goofys/main.go:194 +0x8a4
github.com/codegangsta/cli.(*App).Run(0xc8200f2000, 0xc82000a2a0, 0xe, 0xe, 0x0, 0x0)
/opt/goofys/src/github.com/codegangsta/cli/app.go:180 +0x1028
main.main()
/opt/goofys/src/github.com/kahing/goofys/main.go:203 +0xf6
goroutine 17 [syscall, locked to thread]:
runtime.goexit()
/usr/lib/go/src/runtime/asm_amd64.s:1696 +0x1
goroutine 5 [syscall]:
os/signal.loop()
/usr/lib/go/src/os/signal/signal_unix.go:22 +0x18
created by os/signal.init.1
/usr/lib/go/src/os/signal/signal_unix.go:28 +0x37
goroutine 23 [select, locked to thread]:
runtime.gopark(0xa7e220, 0xc820028f28, 0x98cff8, 0x6, 0x42e518, 0x2)
/usr/lib/go/src/runtime/proc.go:185 +0x163
runtime.selectgoImpl(0xc820028f28, 0x0, 0x18)
/usr/lib/go/src/runtime/select.go:392 +0xa64
runtime.selectgo(0xc820028f28)
/usr/lib/go/src/runtime/select.go:212 +0x12
runtime.ensureSigM.func1()
/usr/lib/go/src/runtime/signal1_unix.go:227 +0x353
runtime.goexit()
/usr/lib/go/src/runtime/asm_amd64.s:1696 +0x1
goroutine 12 [semacquire]:
sync.runtime_Syncsemacquire(0xc8201996c0)
/usr/lib/go/src/runtime/sema.go:237 +0x201
sync.(*Cond).Wait(0xc8201996b0)
/usr/lib/go/src/sync/cond.go:62 +0x9b
io.(*pipe).read(0xc820199680, 0xc8201c5000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
/usr/lib/go/src/io/pipe.go:52 +0x2d2
io.(*PipeReader).Read(0xc82002a0c0, 0xc8201c5000, 0x1000, 0x1000, 0x43c454, 0x0, 0x0)
/usr/lib/go/src/io/pipe.go:134 +0x50
bufio.(*Scanner).Scan(0xc820028730, 0x1000)
/usr/lib/go/src/bufio/scan.go:180 +0x877
github.com/Sirupsen/logrus.(*Logger).writerScanner(0xc8204dd630, 0xc82002a0c0)
/opt/goofys/src/github.com/Sirupsen/logrus/writer.go:20 +0x129
created by github.com/Sirupsen/logrus.(*Logger).Writer
/opt/goofys/src/github.com/Sirupsen/logrus/writer.go:12 +0x17b
goroutine 24 [chan receive]:
main.registerSIGINTHandler.func1(0xc82011e840, 0x7ffe40df688a, 0x10)
/opt/goofys/src/github.com/kahing/goofys/main.go:52 +0x71
created by main.registerSIGINTHandler
/opt/goofys/src/github.com/kahing/goofys/main.go:63 +0xe7
goroutine 13 [semacquire]:
sync.runtime_Syncsemacquire(0xc820199780)
/usr/lib/go/src/runtime/sema.go:237 +0x201
sync.(*Cond).Wait(0xc820199770)
/usr/lib/go/src/sync/cond.go:62 +0x9b
io.(*pipe).read(0xc820199740, 0xc820197145, 0xebb, 0xebb, 0x0, 0x0, 0x0)
/usr/lib/go/src/io/pipe.go:52 +0x2d2
io.(*PipeReader).Read(0xc82002a0d0, 0xc820197145, 0xebb, 0xebb, 0x0, 0x0, 0x0)
/usr/lib/go/src/io/pipe.go:134 +0x50
bufio.(*Scanner).Scan(0xc8201bff10, 0xc8201bfeb8)
/usr/lib/go/src/bufio/scan.go:180 +0x877
github.com/Sirupsen/logrus.(*Logger).writerScanner(0xc820010820, 0xc82002a0d0)
/opt/goofys/src/github.com/Sirupsen/logrus/writer.go:20 +0x129
created by github.com/Sirupsen/logrus.(*Logger).Writer
/opt/goofys/src/github.com/Sirupsen/logrus/writer.go:12 +0x17b
goroutine 22 [syscall]:
syscall.Syscall(0x0, 0x8, 0xc820512018, 0x21000, 0xc81fa98d2b, 0x0, 0x411566)
/usr/lib/go/src/syscall/asm_linux_amd64.s:18 +0x5
syscall.read(0x8, 0xc820512018, 0x21000, 0x21000, 0xc820174480, 0x0, 0x0)
/usr/lib/go/src/syscall/zsyscall_linux_amd64.go:783 +0x5f
syscall.Read(0x8, 0xc820512018, 0x21000, 0x21000, 0x0, 0x0, 0x0)
/usr/lib/go/src/syscall/syscall_unix.go:160 +0x4d
os.(*File).read(0xc82002a138, 0xc820512018, 0x21000, 0x21000, 0xa7ab7c, 0x0, 0x0)
/usr/lib/go/src/os/file_unix.go:211 +0x53
os.(*File).Read(0xc82002a138, 0xc820512018, 0x21000, 0x21000, 0x411566, 0x0, 0x0)
/usr/lib/go/src/os/file.go:95 +0x8a
github.com/jacobsa/fuse/internal/buffer.(*InMessage).Init(0xc820512000, 0x7fd475d1a270, 0xc82002a138, 0x0, 0x0)
/opt/goofys/src/github.com/jacobsa/fuse/internal/buffer/in_message.go:53 +0xa2
github.com/jacobsa/fuse.(*Connection).readMessage(0xc820199800, 0xc820512000, 0x0, 0x0)
/opt/goofys/src/github.com/jacobsa/fuse/connection.go:317 +0x7a
github.com/jacobsa/fuse.(*Connection).ReadOp(0xc820199800, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0)
/opt/goofys/src/github.com/jacobsa/fuse/connection.go:379 +0x81
github.com/jacobsa/fuse/fuseutil.(*fileSystemServer).ServeOps(0xc8204e7460, 0xc820199800)
/opt/goofys/src/github.com/jacobsa/fuse/fuseutil/file_system.go:97 +0x53
github.com/jacobsa/fuse.Mount.func1(0x7fd475d27ab8, 0xc8204e7460, 0xc820199800, 0xc82002a0e0)
/opt/goofys/src/github.com/jacobsa/fuse/mount.go:89 +0x35
created by github.com/jacobsa/fuse.Mount
/opt/goofys/src/github.com/jacobsa/fuse/mount.go:92 +0x827
from goofys.
kahing
commented on August 23, 2024
I think this was already fixed in the parallel_read branch
from goofys.
kahing
commented on August 23, 2024
closed via b1d50fc
from goofys.
Related Issues (20)
- Can't access subdir from top dir even as a root user
- Deprecated Azure API
- main.ERROR Unable to access '<bucket_name>': permission denied HOT 3
- Feature Request: Support s3 new checksums crc32/crc32c/sha1/sha256
- Goofys work with contabo object-storage HOT 2
- Operation to mounted Goofys S3 does not respond
- ReadDir inode leak causes memory leak
- mount with ceph rgw got error ,ceph version is 18.2.1 HOT 1
- Improvement request: set op.Handle as point to FileHandle/DirHandle
- can not compile with go1.16
- Building leads to [email protected]: retracted by module author
- certificate has expired or is not yet valid HOT 3
- Goofys mount is not reloading inside container
- Disable multipart upload HOT 1
- mounting, listing and reading file works, writing doesn't HOT 1
- In kubernetes when killing pod with goffys container, pod stuck in terminating. exit code 137. HOT 2
- Process exits with error during copy of file under goofys mount point
- Failed deleting files HOT 1
- Any luck building this on mac osx? Brew and go have both moved on HOT 2
- How to mount the Blob/Data Lake containers with different Azure Storage Accounts via /etc/fstab ? HOT 1
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from goofys.