Git Product home page Git Product logo

GoCollaborate

Build Status Go Report Card alt text

What is GoCollaborate?

GoCollaborate is an universal framework for stream computing and distributed services management that you can easily program with, build extension on, and on top of which you can create your own high performance distributed applications.

The Idea Behind

GoCollaborate absorbs the best practice experience and improves from the popular distributed computing frameworks including✨Hadoop, ✨Spark, ✨ZooKeeper, ✨Dubbo and ✨Kite that helps to ideally provision the computability for large scale data sets with an easy-to-launch setups.

Am I Free to Use GoCollaborate?

Yes! Please check out the terms of the BSD License.

Contribution

This project is currently under development, please feel free to fork it and report issues!

Please check out most recent API document for more information.

Relative Links

Quick Start

Installation

go get -u github.com/GoCollaborate/src

Create Project

mkdir Your_Project_Name
cd Your_Project_Name
mkdir core
touch case.json
touch main.go
cd ./core
touch example.go

The project structure now looks something like this:

[Your_Project_Name]
┬
├ [core]
	┬
	└ example.go
├ case.json
└ main.go

Configure file case.json:

{
	"caseid": "GoCollaborateStandardCase",
	"cards": {
		"localhost:57851": {
			"ip": "localhost",
			"port": 57851,
			"alive": false,
			"seed": false
		},
		"localhost:57852": {
			"ip": "localhost",
			"port": 57852,
			"alive": true,
			"seed": true
		}
	},
	"timestamp": 1508619931,
	"local": {
		"ip": "localhost",
		"port": 57852,
		"alive": true,
		"seed": true
	},
	"coordinator": {
		"ip": "localhost",
		"port": 0,
		"alive": true,
		"seed": false
	}
}

Entry

package main

import (
	"./core"
	"github.com/GoCollaborate/src"
)

func main() {
	mp := new(core.SimpleMapper)
	rd := new(core.SimpleReducer)
	collaborate.Set("Function", core.ExampleFunc, "exampleFunc")
	collaborate.Set("Mapper", mp, "core.ExampleTask.Mapper")
	collaborate.Set("Reducer", rd, "core.ExampleTask.Reducer")
	collaborate.Set("Shared", []string{"GET", "POST"}, core.ExampleJobHandler)
	collaborate.Run()
}

Map-Reduce

package core

import (
	"fmt"
	"github.com/GoCollaborate/src/artifacts/task"
	"github.com/GoCollaborate/src/wrappers/taskHelper"
	"net/http"
)

func ExampleJobHandler(w http.ResponseWriter, r *http.Request, bg *task.Background) {
	job := task.MakeJob()

	job.Tasks(
		&task.Task{
			task.SHORT,
			task.BASE,
			"exampleFunc",
			task.Collection{1, 2, 3, 4, 1, 2, 3, 4, 1, 2, 3, 4},
			task.Collection{0},
			task.NewTaskContext(struct{}{}),
			0,
		},
	)

	job.Stacks("core.ExampleTask.Mapper", "core.ExampleTask.Reducer")

	bg.Mount(job)
}

func ExampleFunc(source *task.Collection,
	result *task.Collection,
	context *task.TaskContext) bool {
	// deal with passed in request
	fmt.Println("Example Task Executed...")
	var total int
	// the function will calculate the sum of source data
	for _, n := range *source {
		total += n.(int)
	}
	result.Append(total)
	return true
}

type SimpleMapper int

func (m *SimpleMapper) Map(inmaps map[int]*task.Task) (map[int]*task.Task, error) {
	// slice the data source of the map into 3 separate segments
	return taskHelper.Slice(inmaps, 3), nil
}

type SimpleReducer int

func (r *SimpleReducer) Reduce(maps map[int]*task.Task) (map[int]*task.Task, error) {
	var sum int
	for _, s := range maps {
		for _, r := range (*s).Result {
			sum += r.(int)
		}
	}
	fmt.Printf("The sum of numbers is: %v \n", sum)
	fmt.Printf("The task set is: %v", maps)
	return maps, nil
}

Run

Here we create the entry file and a simple implementation of map-reduce interface, and next we will run with std arguments:

go run main.go -mode=clbt

The task is now up and running at:

http://localhost:8080/core/ExampleJobHandler

Collaborate

  1. Copy your project directory:
cp Your_Project_Name Your_Project_Name_Copy
  1. Enter the copied project:
cd Your_Project_Name_Copy
  1. Edit local ip address in case.json:
{
	"caseid": "GoCollaborateStandardCase",
	"cards": {
		"localhost:57852": {
			"ip": "localhost",
			"port": 57852,
			"alive": true,
			"seed": true
		}
	},
	"timestamp": 1508619931,
	"local": {
		"ip": "localhost",
		"port": 57851,
		"alive": true,
		"seed": false
	},
	"coordinator": {
		"ip": "localhost",
		"port": 0,
		"alive": true,
		"seed": false
	}
}
  1. Run the copied project, don't forget to change your port number if you are running locally:
go run main.go -mode=clbt -port=8081
  1. Now the distributed servers are available at:
http://localhost:8080/core/ExampleJobHandler
// and 
http://localhost:8081/core/ExampleJobHandler
  1. Alternatively, access the GoCollaborate UI for more infomation:
http://localhost:8080

Acknowledgement

gocollaborate's Projects

examples icon examples

Code snippets of distributed framework GoCollaborate

src icon src

A light-weight distributed stream computing framework for Golang

tests icon tests

Test scripts of distributed framework GoCollaborate

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.