Git Product home page Git Product logo

pipe's Introduction

Pipes - an utility to create streamable workers

As sometimes we are bound to IO blocks this will help to create workers to stream data

A Proc:

  • only have one consumer and can consume from several sources
  • can have more than one sender and each sender can send to several consumers
  • should not have a cyclic links

Example:

func main() {
	// Create an origin that produces data, consumes from an DB/API or etc
	origin := pipe.NewProc(
		pipe.WithFunc(func(_ pipe.Consumer, ints pipe.Sender) error {
			for i := 0; i < 10; i++ {
				if err := ints.Send(i); err !=nil {
					return err
				}
			}
			return nil
		}),
	)

	evenodd := pipe.NewProc(
		pipe.WithWorkers(4),        // use 4 go routines
		pipe.WithSource(0, origin), // consumes output 0 from origin
		pipe.WithFunc(func(c pipe.Consumer, odds, evens pipe.Sender) error {
			return c.Consume(func(vv interface{}) error {
				v := c.Value().(int)
				target := odds
				if v&1 == 0 {
					target = evens
				}
				return target.Send(v)
			}
			return nil
		}),
	)

	res := []int{}
	// consumes data produced by evenodd and write it to result slice
	// could be an API endpoint/file/db
	pipe.NewProc(
		pipe.WithBuffer(10),          // buffer size of the consumer
		pipe.WithSource(0, evenodd),  // consumes output 0 (odds) from evenodd
		pipe.WithSource(1, evenodd),  // consumes output 1 (evens) from evenodd
		pipe.WithFunc(func(c pipe.Consumer) error {
			return c.Consume(func(vv interface{}) error {
				v := c.Value().(string) // we expect strings
				res = append(res, v)
				return nil
			})
		}),
	)

	// Run will start the procs binded to `origin` and wait until all finishes
	// if an error is returned in any proc func the context will be canceled
	// and the first error will be returned here
	if err := origin.Run(); err != nil {
		log.Fatal(err)
	}
}

pipe's People

Contributors

stdiopt avatar

Watchers

 avatar

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.