Git Product home page Git Product logo

circleci-parallel's Introduction

Gem Version Dependency Status CircleCI Coverage Status Code Climate

CircleCI::Parallel

CircleCI::Parallel provides simple Ruby APIs for syncing CircleCI parallel nodes and transferring all node data to a single master node.

Installation

Add this line to your application's Gemfile:

gem 'circleci-parallel'

And then execute:

$ bundle install

Usage

Before using CircleCI::Parallel:

CircleCI::Parallel uses SSH for syncing nodes and transferring data between nodes. All data will be transferred to the master node (where CIRCLE_NODE_INDEX is 0).

# circle.yml
test:
  override:
    - ruby test.rb:
        parallel: true
# test.rb
require 'circleci/parallel'

CircleCI::Parallel.configure do |config|
  # This hook will be invoked on every node.
  # The current working directory in this hook is set to the local data directory
  # where node specific data should be saved in.
  config.on_every_node.before_sync do
    data = { 'foo' => ENV['CIRCLE_NODE_INDEX'].to_i * 10 }
    json = JSON.generate(data)
    File.write('data.json', json)
  end

  # This hook will be invoked only on the master node after downloading all data from slave nodes.
  # The current working directory in this hook is set to the download data directory
  # where all node data are gathered into.
  # The directory structure on the master node will be the following:
  #
  #     .
  #     ├── node0
  #     │   └── node_specific_data_you_saved_on_node0.txt
  #     ├── node1
  #     │   └── node_specific_data_you_saved_on_node1.txt
  #     └── node2
  #         └── node_specific_data_you_saved_on_node2.txt
  config.on_master_node.after_download do
    merged_data = {}

    Dir.glob('*/data.json') do |path|
      json = File.read(path)
      data = JSON.parse(json)
      node_name = File.dirname(path)
      merged_data[node_name] = data
    end

    p merged_data
    # {
    #   "node0" => { "foo" => 0 },
    #   "node1" => { "foo" => 10 },
    #   "node2" => { "foo" => 20 }
    # }
  end
end

# Sync all nodes in the same build and gather all node data into the master node.
# Invoking this method blocks until the sync and the data transfer is complete.
CircleCI::Parallel.sync

See API docs for more details.

License

The gem is available as open source under the terms of the MIT License.

circleci-parallel's People

Contributors

yujinakayama avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar

Forkers

akiray03

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.