Do you think that you could replicate this tool using Python?

There doesn't appear to be a flag/option to test against insecure (self-signed or otherwise untrusted) endpoints. Am I simply missing something? I'll offer a preemptive mea culpa here...I've inherited a lab in a crappy state and I'm trying to get that straightened out, including valid certs. from an internal CA for everything. In the meantime I've got a need to do some S3 benchmarking/testing.

Would be nice to benchmark write performance as well. S3 is used for backups a lot.

An interesting benchmark and if I understand it correctly it measures latencies and throughput when being accessed from S3 instances. I think it would be interesting to see how it behaves when accessed from outside Amazon network, e.g. downloading object with browser.

It would be fascinating to see how S3's SSE (both KMS and the old-fashioned flavor) affect throughput and latencies. Might require some finagling to figure out how/if per-object data keys are getting cached somewhere in S3 after getting KMS to decrypt the data key.

It says:

The minimum object size to test, with 1 = 1 KB

I had to look through the code to understand that "payload" is a power of 2 which is then multiplied by 1KB. So, in short:

# assuming payload is > 0
multiplier = 2 ^ (payload - 1)
object_min_size = 1KB * multiplier

So, you see, you have to pass -payload-min=16 to get 32 MB object, and -payload-min=17 to get 64 MB object while the parameter clearly says it is an object size. Why is it encoded in such a weird way then?

Same applies to payload-max. I suggest either renaming this parameter to something better corresponding to the actual meaning, or - even better - allow user to define object min and max size like:
-payload-min=1000 - meaning 1 MB

Howdy,

Great tool, thanks so much. I'm also interested in how you built the Analysis XLS. Would you be able to share that?

The provided documentation doesn't include instructions on how to build this from source, and the provided binaries don't include the -endpoint option necessary to test alternative S3 providers.

(Looking to run some simple tests of @minio on @packethost across a few machine type combinations, will be happy to share results).

There isn't any mentioning of EBS in this benchmark. Given that a gp2 volume is limited to 250 MB/s, the bottleneck on a CPU that supports 10-gigabit network is more likely to be the hard-drive I/O.

@dvassallo Do you have information about the EBS volume types you've used? Is hard drive I/O not a factor?

Can I configure to benchmark https://aws.amazon.com/de/s3/storage-classes/express-one-zone/ ?

was trying to test this with swiftstack s3 and it failed - we should probably add a way to add the -signatureVersion flag. Something like -signatureVersion v1. Aws by default uses v4 i guess

Hey,

Are you considering implementing other s3 providers?

Hi,

Environment setup:
testvm => 2x harpoxy-s3 => 3x ceph radosgw

Rados gw runs on dedicated hardware while testvm and haproxy-s3 runs on openstack. This setup works good since i was able to use rclone to POST/GET objects on this custom s3. Note that radosgw are completly standalone and not integrated with openstack keystone at all.

Problem definition:
As a second step i wanted to run s3-benchmark just to check overall long term performance.

This is how my credentials are set:

[fedora@brtest ~]$ cat ~/.aws/credentials 
[brtest]
aws_access_key_id = xxx
aws_secret_access_key = xxxx

I get something like that:

[fedora@brtest ~]$ ./s3-benchmark --endpoint s3.ourdomain.io --region custom-signing-region --bucket-name benchmark

--- SETUP --------------------------------------------------------------------------------------------------------------------

panic: Filed to create S3 bucket: EC2RoleRequestError: no EC2 instance role found
caused by: EC2MetadataError: failed to make EC2Metadata request
caused by: 404 Not Found

The resource could not be found.

   

goroutine 1 [running]:
main.setup()
	/Users/daniel/go/src/github.com/dvassallo/s3-benchmark/main.go:229 +0xa35
main.main()
	/Users/daniel/go/src/github.com/dvassallo/s3-benchmark/main.go:113 +0x44

I believe the issue is caused by fact that aws-go-sdk recognizes our openstack api as a valid aws api and tries to obtain s3 credentials this way.

fedora@brtest ~]$ curl http://169.254.169.254/latest/meta-data/
ami-id
ami-launch-index
ami-manifest-path
block-device-mapping/
hostname
instance-action
instance-id
instance-type
local-hostname
local-ipv4
placement/
public-hostname
public-ipv4
public-keys/
reservation-id

Would it possible to force s3-benchmark to skip aws autodetection ??

Regards,
Bart

Trying to run the benchmark tool from an EC2 instance on an existing bucket:

[ec2-user@ip-172-31-2-167 ~]$ ./s3-benchmark    -bucket-name dhavivresearch -full -region us-east-1

--- SETUP --------------------------------------------------------------------------------------------------------------------

panic: Filed to create S3 bucket: InvalidLocationConstraint: The specified location-constraint is not valid
        status code: 400, request id: 41997BE73BC03A06, host id: ZNEaOvY5TyFs0PR3xh73aUAPVg6y18y9iV/bzJH4TJYiQ8/sKPk+wIqReWkRtpH3/wV+oV53Sns=

goroutine 1 [running]:
main.setup()
        /Users/daniel/go/src/github.com/dvassallo/s3-benchmark/main.go:229 +0xa35
main.main()
        /Users/daniel/go/src/github.com/dvassallo/s3-benchmark/main.go:113 +0x44

It looks like -samples 1000000 or, eg. -samples 100 makes no difference. Shouldn't 1000000 samples take longer than 100?

Hello, it would be nice to have a --profile option to indicate which credentials to use in .aws/credentials

Hi,
Very nice project. Thanks for open sourcing it.
Please consider adding S3 upload benchmarks as well.

Thanks for building this! It would be great if there was an option to report upload performance instead of download with everything the same otherwise.

Hi @dvassallo,

Great utility!

I'm attempting to run dep ensure but the process fails. Any guidance is appreciated!

...
	v0.2.0: Could not introduce github.com/schollz/[email protected], as its subpackage github.com/schollz/progressbar/v2 is missing. (Package is required by (root).)
	v0.1.0: Could not introduce github.com/schollz/[email protected], as its subpackage github.com/schollz/progressbar/v2 is missing. (Package is required by (root).)
	master: Could not introduce github.com/schollz/progressbar@master, as its subpackage github.com/schollz/progressbar/v2 is missing. (Package is required by (root).)
	finish: Could not introduce github.com/schollz/progressbar@finish, as its subpackage github.com/schollz/progressbar/v2 is missing. (Package is required by (root).)
	time: Could not introduce github.com/schollz/progressbar@time, as its subpackage github.com/schollz/progressbar/v2 is missing. (Package is required by (root).)