Git Product home page Git Product logo

fasts3's Introduction

FastS3

Fast s3 utility is a faster version of s3cmd's ls, get, and cp functions ideal for buckets containing millions of keys.

#Installation

go get github.com/TuneOSS/fasts3

#Configuration

Use aws configure command from the aws cli tool (https://aws.amazon.com/cli/) which will create the necessary config files in ~/.aws/credentials.

Alternatively you can set these environment variables which will take precedence over the credentials file:

export AWS_ACCESS_KEY_ID=<access_key>
export AWS_SECRET_ACCESS_KEY=<secret_key>

#Usage Use:

fasts3 --help
fasts3 <cmd> --help

###Using search depth to go faster Many times you know the structure of your s3 bucket, and this can be used to optimize listings. Say you have a structure like so:

fasts3 ls s3://mybuck/logs/

DIR s3://mybuck/logs/2011/
DIR s3://mybuck/logs/2012/
DIR s3://mybuck/logs/2013/
DIR s3://mybuck/logs/2014/
DIR s3://mybuck/logs/2015/

Doing a fasts3 ls -r s3://mybuck/logs/ will read all keys under logs sequentially. We can make this faster by adding a --search-depth 1 flag to the command which gives each of the underlying directories its own thread, increasing throughput.

###Concurrency The concurrency level of s3 command execution can be tweaked based on your usage needs. By default, 4*NumCPU s3 commands will be executed concurrently, which is ideal based on our benchmarks. If you want to override this value, set GOMAXPROCS in your environment to set the concurrency level: GOMAXPROCS=64 fasts3 ls -r s3://mybuck/logs/ will execute 64 s3 commands concurrently.

###Examples

# ls
fasts3 ls s3://mybucket/ # lists top level directories and keys
fasts3 ls -r s3://mybucket/ # lists all keys in the bucket
fasts3 ls -r --search-depth 1 s3://mybucket/ # lists all keys in the bucket using the directories 1 level down to thread
fasts3 ls -r s3://mybucket/ | awk '{s += $1}END{print s}' # sum sizes of all objects in the bucket

# get
fasts3 get s3://mybuck/logs/ # fetches all logs in the prefix

# stream
fasts3 stream s3://mybuck/logs/ # streams all logs under prefix to stdout
fasts3 stream --key-regex ".*2015-01-01" s3://mybuck/logs/ # streams all logs with 2015-01-01 in the key name stdout

# cp
fasts3 cp -r s3://mybuck/logs/ s3://otherbuck/ # copies all subdirectories to another bucket
fasts3 cp -r -f s3://mybuck/logs/ s3://otherbuck/all-logs/ # copies all source files into the same destination directory

###Completion Bash and ZSH completion are available.

To install for bash:

source completion.sh

For zsh:

source completion.zsh

Demo:

autocomplete demo for zsh

fasts3's People

Contributors

hasjohnb avatar hastyler avatar rushton avatar

Watchers

 avatar

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.