Git Product home page Git Product logo

garden's Introduction

https://travis-ci.org/iffy/garden.png?branch=master

Builds | Docs

Garden

Jorge Luis Borges wrote The Garden of Forking Paths. This project lets you create gardens of computing, forking paths.

Install it

git clone https://github.com/iffy/garden garden.git && cd garden.git
python setup.py install

How it works, through an example

You are a teacher, and want to compute students' grades. Assignments are worth 40% of the grade and exams 60%. You want letter grades too. We'll use decimal.Decimal (imported as D) instead of floats to avoid precision problems:

>>> from decimal import Decimal as D
>>> def compute_percent(assignments, exams):
...    assignments = D(assignments)
...    exams = D(exams)
...    return str((D('0.4') * assignments) + (D('0.6') * exams))
...
>>> def compute_letter(percent):
...     percent = D(percent)
...     if percent > D('0.9'):
...         return 'A'
...     elif percent > D('0.7'):
...         return 'B'
...     elif percent > D('0.6'):
...         return 'C'
...     elif percent > D('0.5'):
...         return 'D'
...     else:
...         return 'F'

Define how functions relate to each other by putting them in a Garden. This code adds a path for 'percent' that depends on 'assignments' and 'exams'. The code also adds a path for 'letter' which depends on 'percent' (gloss over the 'v1' strings for now -- they will be important later):

>>> from garden.path import Garden
>>> garden = Garden()
>>> garden.addPath('percent', 'v1', inputs=[
...     ('assignments', 'v1'),
...     ('exams', 'v1'),
... ])
>>> garden.addPath('letter', 'v1', inputs=[
...     ('percent', 'v1'),
... ])

Get a Worker ready to do the computations by telling it which functions correspond to which pieces of data:

>>> from garden.worker import BlockingWorker
>>> worker = BlockingWorker()
>>> worker.registerFunction('percent', 'v1', compute_percent)
>>> worker.registerFunction('letter', 'v1', compute_letter)

Create a place to store the results:

>>> from garden.store import InMemoryStore
>>> store = InMemoryStore()

Create a Gardener to coordinate work for the worker and tell them about each other:

>>> from garden.gardener import Gardener
>>> gardener = Gardener(garden, store)
>>> gardener.subscribe(worker)
[...]
>>> worker.subscribe(gardener)
[...]

Now give the Gardener some data about Frodo's progress in the class:

>>> from garden.data import Input
>>> gardener.inputReceived(Input('Frodo', 'assignments', 'v1', '0.5'))
<Deferred...>
>>> gardener.inputReceived(Input('Frodo', 'exams', 'v1', '0.9'))
<Deferred...>

And see that the grade was computed:

>>> store.get('Frodo', 'percent', 'v1').result
[Data(entity='Frodo', name='percent', version='v1', ... value='0.74')]
>>> store.get('Frodo', 'letter', 'v1').result
[Data(entity='Frodo', name='letter', version='v1', ... value='B')]

Are you kidding me?

That was way too much work. Why would anyone want to use such a complicated system to do what is essentially two function calls?

Because this is no ordinary garden,

In all fictional works, each time a man is confronted with several alternatives, he chooses one and eliminates the others; in the fiction of Ts’ui Pên, he chooses -- simultaneously -- all of them. ...In the work of Ts’ui Pên, all possible outcomes occur.

(The Garden of Forking Paths by Jorge Luis Borges)

Versions

There's a mistake in the compute_letter function above: the cut-off for B, C and D grades are 10% too low. We want to fix this, but want to be able to test our fix before we replace the buggy function. Here's our fixed function:

>>> def compute_letter_v2(percent):
...     percent = D(percent)
...     if percent > D('0.9'):
...         return 'A'
...     elif percent > D('0.8'):
...         return 'B'
...     elif percent > D('0.7'):
...         return 'C'
...     elif percent > D('0.6'):
...         return 'D'
...     else:
...         return 'F'

Add the new function spec to the Garden, with a distinct version:

>>> garden.addPath('letter', 'v2', inputs=[
...     ('percent', 'v1'),
... ])

Tell the worker about the new function:

>>> worker.registerFunction('letter', 'v2', compute_letter_v2)

Compute the result:

>>> gardener.doPossibleWork('Frodo', 'letter', 'v2')
<Deferred...>

And see that Frodo now has two 'letter' values:

>>> store.get('Frodo', 'letter', 'v1').result
[Data(entity='Frodo', name='letter', version='v1', ... value='B')]
>>> store.get('Frodo', 'letter', 'v2').result
[Data(entity='Frodo', name='letter', version='v2', ... value='C')]

More Versions

Suppose we are a terrible teacher and want to change the grade weighting half way through the semester so that exams are 90% and assignments are 10%. We make a new version of compute_percent, add it to the Garden and tell the worker about it as before. We also indicate that both 'letter' functions can use this new 'percent' as an input:

>>> def compute_percent_v2(assignments, exams):
...    assignments = D(assignments)
...    exams = D(exams)
...    return str((D('0.1') * assignments) + (D('0.9') * exams))
...
>>> garden.addPath('percent', 'v2', inputs=[
...     ('assignments', 'v1'),
...     ('exams', 'v1'),
... ])
>>> garden.addPath('letter', 'v1', inputs=[
...     ('percent', 'v2'),
... ])
>>> garden.addPath('letter', 'v2', inputs=[
...     ('percent', 'v2'),
... ])
>>> worker.registerFunction('percent', 'v2', compute_percent_v2)
>>> gardener.doPossibleWork('Frodo', 'percent', 'v2')
<Deferred...>

As you may expect, Frodo now has two versions of 'percent':

>>> store.get('Frodo', 'percent', 'v1').result
[Data(entity='Frodo', name='percent', version='v1', ... value='0.74')]
>>> store.get('Frodo', 'percent', 'v2').result
[Data(entity='Frodo', name='percent', version='v2', ... value='0.86')]

And Frodo now has four versions of 'letter':

>>> store.get('Frodo', 'letter', 'v1').result
[Data(entity='Frodo', name='letter', version='v1', ... value='B'),
 Data(entity='Frodo', name='letter', version='v1', ... value='B')]
>>> store.get('Frodo', 'letter', 'v2').result
[Data(entity='Frodo', name='letter', version='v2', ... value='C'),
 Data(entity='Frodo', name='letter', version='v2', ... value='B')]

Confused? Enlightened?

Using/Deploying

There are many ways to deploy components of the Garden. Here are some:

Single Combination Process

You can start a single process containing both a Gardener and a single Worker pretty easily. Write a python module containing getWorker() and getGarden() functions, which return an IWorker and a Garden respectively. Save the following as sample.py:

# sample.py
from garden.worker import ThreadedWorker
from garden.path import Garden

def cake(eggs, flour, flavor):
    words = []
    if flour == 'wheat':
        words.append('gross')
    return ' '.join(words + [flavor, 'cake'])

def getGarden():
    garden = Garden()
    garden.addPath('cake', '1', [
        ('eggs', '1'),
        ('flour', '1'),
        ('flavor', '1'),
    ])
    return garden

def getWorker():
    worker = ThreadedWorker()
    worker.registerFunction('cake', '1', cake)
    return worker

And then spawn a combo process with twistd:

twistd -n garden combo -m sample --sqlite-db=/tmp/data.sqlite -w tcp:9990

Data will be saved in /tmp/data.sqlite. New data values can be sent using HTTP on port 9990. (You can manually add data by visiting http://127.0.0.1:9990/ and you can view a live feed of the results at http://127.0.0.1:9990/feed).

Load some data with curl:

curl -d 'entity=Gandalf' -d 'name=eggs' -d 'version=1' -d 'value=good' http://127.0.0.1:9990
curl -d 'entity=Gandalf' -d 'name=flour' -d 'version=1' -d 'value=wheat' http://127.0.0.1:9990
curl -d 'entity=Gandalf' -d 'name=flavor' -d 'version=1' -d 'value=hobbit' http://127.0.0.1:9990

And see the result:

$ sqlite3 /tmp/data.sqlite "select value from data where name='cake';"
value
--------------------
gross hobbit cake

Use better flour, and see the data change:

curl -d 'entity=Gandalf' -d 'name=flour' -d 'version=1' -d 'value=white' http://127.0.0.1:9990
$ sqlite3 /tmp/data.sqlite "select value from data where name='cake';"
value
--------------------
hobbit cake

garden's People

Contributors

iffy avatar

Watchers

avatar avatar avatar

garden's Issues

I*Receiver and IGardener need different definitions

Right now it's:

class IResultReceiver:
    gardener = Attribute()

class IWorkReceiver:
    worker = Attribute()

class IGardener:
    def inputReceived(...):
        pass
    def workReceived(...):
        pass
    def workErrorReceived(...):
        pass

When I think it really ought to be:

class IInputSource:
    input_receiver = Attribute()

class IResultSource:
    result_receiver = Attribute()

class IWorkSource:
    worker = Attribute()

class IResultReceiver:
    def workReceived(...):
        pass
    def workErrorReceived(...):
        pass

class IInputReceiver:
    def inputReceived(...):
        pass

And maybe those attributes (input_receiver, result_receiver and worker) ought to be setter methods? The argument for making them setters is so that if an I*Source has pending data for the receiver/worker, it could start producing when the setter method is called.

twistd garden worker

This is done when we can do the following to start a worker:

twistd garden worker --gardener-endpoint 'tcp:127.0.0.1:9990' --module 'my.module'

Add concept of default values

I'm not sure what component is responsible for it, but we need default values for some things.

Suppose, for instance, we need to flag entities for having too many violations. Something like:

def is_flagged(violation_count):
    if int(violation_count) > 5:
        return 'flagged'
    return 'not flagged'

By default, entities have '0' violations, and the is_flagged function should be computed for them. Is it the data store's job to do that? Or the Garden's? Or is it the input data provider's job to set it (in which case no default value support is needed)?

Add database store

Right now, there's only garden.local.InMemoryStore. This issue is done when there's a database backed store.

twistd garden gardener

We want to be able to run a gardener instance quickly with twistd. This issue is done when the following is supported:

twistd garden gardener --worker-endpoint=tcp:9990 --input-endpoint=tcp:9991 --module='my.module'

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.