Git Product home page Git Product logo

gbatchy's Introduction

gbatchy

PyPI version Build Status Coverage Status

A small library inspired by batchy, but using gevent greenlets instead of yield to transfer control.

For example:

from gbatchy import spawn, pget, batched, batch_context

@batched()
def batch_fn(arg_list):
    print 'In batch function with args:'
    results = []
    for args, kwargs in arg_list:
        print '\t', args[0]
        results.append(args[0] + 1)
    print 'Batch function done'
    return results

@batch_context
def fetcher():
    results = pget(batch_fn(i, as_future=True) for i in xrange(3))
    print results

@batch_context
def test():
    pget(spawn(fetcher) for i in xrange(2))
    
test()

would print

In batch function with args:
	0
	1
	2
	0
	1
	2
Batch function done
[1, 2, 3]
[1, 2, 3]

Mini docs:

  • @batch_context: Ensures that the function is running in a batch context (i.e. all concurrent calls to @batched functions will be coalesced)
  • spawn(fn, *args, **kwargs): start a new greenlet that will run fn(*args, **kwargs). This creates a batch context or uses the current one.
  • spawn_proxy(fn, *args, **kwargs): same as spawn(), but returns a proxy type instead of a greenlet. This should help get rid of .get() around a lot of your code.
  • @batched(accepts_kwargs=True) and @class_batched(): marks this function as a batch function. All batch functions take just one arg: args_list: [(args, kwargs), ...] (or [args, ...] if accepts_kwargs=False)
  • pget(iterable): a quick way to .get() all the arguments passed.
  • pmap(fn, iterable): same as map(fn, iterable), except runs in parallel. Note: keyword arguments to pmap are passed through to fn for each element.
  • pfilter(fn, iterable): same as filter(fn, iterable) except runs in parallel.
  • Pool(size): same as gevent.pool.Pool - a way to limit the maximum concurrent amount of work.
  • iwait(greenlets): same as gevent.iwait, but works with batch greenlets. Using gevent.iwait with batch greenlets is strongly discouraged and will lead to mysterious hangs.
  • wait(greenlets, timeout, count): same as gevent.wait.
  • immediate(v): returns an AsyncResult-like object that is immediately ready and immediate(v).get() is v == True.
  • immediate_exception(exc): same as immediate, but raises exc.
  • with may_block(): a low-level primitive when you need to use a gevent-native blocking call between calls to @batched functions (e.g. gevent.queue).
  • transform(pending, fn): a somewhat low-level, but performant way to take an AsyncResult-like object and run immediate(fn(pending.get())). Note that fn must be pure - it cannot interact with greenlets. Any extra kwargs will be passed to fn.

gbatchy's People

Contributors

mikekap avatar

Watchers

Felix Fung avatar James Cloos avatar

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.