Git Product home page Git Product logo

aspu's Introduction

Azure Storage Parallel Uploader (ASPU)

Get Started

Install

1. Install requirement tools

  $ sudo pip install blobxfer -y

If you got errors that did not find pip, try below as,

Ubuntu
  $ sudo apt-cache update
  $ sudo apt-get install python-pip
CentOS 7 or later
  $ sudo yum groupinstall "Development Tools"
  $ sudo yum install python-pip

If you want to use mongodb on local

Ubuntu
  $ sudo apt-cache update
  $ sudo apt-get install mongodb-server mongodb -y
CentOS 7 or later
  $ sudo yum install epel-release -y
  $ sudo yum install mongodb-server mongodb -y

2. Get sources from GitHub

  $ git clone https://bitbucket.org/shkawan/aspu.git

3. Setup

You need a superuser power. You might be asked sudo password.

  $ cd aspu/
  $ sh ./setup.sh

If you got some messages, you should follow them.

4. Test

  $ perl -c ./aspu

You saw a message "Syntax OK", everything would be working !

Configure

$HOME/config.yaml

Edit your config.yaml placed your home directory

  $ cp sample-config.yaml $HOME/config.yaml

Minimum config:

  1. Set your storage accounts
  2. Select a logic to select storage
    For now, it can support logics dispatch_by_filename and roundrobin

Usage

Pre requirement

You need to prepare a list to upload.

/mnt/backup/GST/C/readblock.jpg
/mnt/backup/GST/C/writeblock.jpg
/mnt/backup/GST/C/deadlock.jpg
/mnt/backup/GST/C/rwlocks1.jpg
/mnt/backup/GST/C/simplelock.jpg
/mnt/backup/GST/JAVA/readblock.jpg
/mnt/backup/GST/JAVA/writeblock.jpg
  :

Run

To run the script, You have to set environment value AZURE_STORAGE_SELECT_CONFIG_PATH that is used to config path. if not, config path is your HOME directory.

  $ export AZURE_STORAGE_SELECT_CONFIG_PATH=$HOME/aspu

For simply start, You can copy dispatch.json and define-storage.yaml to $HOME or AZURE_STORAGE_SELECT_CONFIG_PATH, and edit for your Azure system environment.

Example1

Parallelism is Single.(Default)

  $ ./aspu < list.txt

Example2

Parallelism is 10.

  $ ./aspu -p 10 < list.txt

Writing to DB

MongoDB(or You can use Azure DocumentDB with MongoDB compatible feature)

Default

  $ mongo azure_storage
  > db.store.find()
  { "_id" : ObjectId("57e0059c135f4e5189c344fd"), "container" : "foo", "filename" : "test1pix.jpg", "path" : "/mnt/backup/new-managed000/usr/local/src/php-5.3.14/ext/standard/tests/image/test1pix.jpg", "storage" : "storage9391" }
  { "_id" : ObjectId("57e0059c135f4e5189c344fe"), "container" : "bar", "filename" : "image025.jpg", "path" : "/mnt/backup/new-managed000/usr/local/src/php-5.3.14/ext/exif/tests/image025.jpg", "storage" : "storage2101" }
    :

You can use Azure DocumentDB with MongoDB compatible instead of original MongoDB.

Logging and Notification

Syslog

Facility is local0, Level is info

Example(Default on Ubuntu)
  $ tail -f /var/log/syslog

to Slack

At first, you need to get incoming api webhook uri, and set it to your environment value MY_SLACK_API

  $ export MY_SLACK_API=https://hooks.slack.com/services/T1DF00000/B2DP00000/y7kqe88JsXrOwP0000000000
  $ ./aspu --slack channelname -p 5

to mail

  $ ./aspu --mail [email protected] -p 5

aspu's People

Contributors

shin5ok avatar

Watchers

James Cloos avatar  avatar  avatar

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.