Git Product home page Git Product logo

microbial's People

Contributors

darsee avatar pelger avatar

Stargazers

avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar

Watchers

avatar avatar avatar

Forkers

darsee nherment polyma

microbial's Issues

Handling consumer groups

Peter. Some months back you were considering re-jigging microbial to emphasise consumer groups as a means of partitioning. Are you any closer in implementing that? I'm trying to get logstash working with Zookeeper/Kafka as an input (https://github.com/joekiller/logstash-kafka) and it appears to rely on having a group_id (consumer group) defined. Does the current version of microbial handle this? In fact, that plugin for logstash to interface with Kafka is now merged into the core as of a few days ago (elastic/logstash#1472). I can't seem to get it to work with request/response pattern the microbial applies to Zookeeper though ... and perhaps this is because no consumer group is set. Would you have any thoughts on a fix? Thanks. Colum

Forward requests

I'd like to be able to forward a request to a new service maintaining the original respond channel so that the next service can reply to the client who initiated the request.

Maybe the request could accept respond to info such as:

{
    topic: 'web.res',
    partition: 0
}

If this information isn't provided then microbial could default to the response channel that was provided when calling microbial's run function.

A request that has a property with a dot in the value doesn't get a response (tested with postal only)

In basicTest.js, this:

mcb.request({request: 'say'}, function(res) {...});

gets a response corresponding to match: { request: 'say' }.

Output from postal.diagnostics:

{
    "channel": "postal",
    "topic": "subscription.created",
    "data": {
        "event": "subscription.created",
        "channel": "request",
        "topic": "*"
    },
    "timeStamp": "2014-02-24T12:19:22.472Z"
}
{
    "channel": "postal",
    "topic": "subscription.created",
    "data": {
        "event": "subscription.created",
        "channel": "6c5b14b9-2447-41d1-95c0-723381a69f0e",
        "topic": "*"
    },
    "timeStamp": "2014-02-24T12:19:22.476Z"
}
{
    "topic": "{\"request\":\"say\"}",
    "data": {
        "request": {
            "request": "say",
            "$id": "ab0f9654-d4d7-4f2f-82c9-a318e4cb7792"
        },
        "$inf": {
            "type": "request",
            "req": {
                "topic": {
                    "type": "request",
                    "name": "request"
                },
                "channel": {
                    "channel": "request"
                }
            },
            "res": {
                "topic": {
                    "type": "response",
                    "name": "6c5b14b9-2447-41d1-95c0-723381a69f0e"
                },
                "channel": {
                    "channel": "6c5b14b9-2447-41d1-95c0-723381a69f0e"
                }
            }
        }
    },
    "channel": "request",
    "timeStamp": "2014-02-24T12:19:22.497Z"
}
{
    "topic": "{\"request\":{\"request\":\"say\",\"$id\":\"ab0f9654-d4d7-4f2f-82c9-a318e4cb7792\"},\"$inf\":{\"type\":\"request\",\"req\":{\"topic\":{\"type\":\"request\",\"name\":\"request\"},\"channel\":{\"channel\":\"request\"}},\"res\":{\"topic\":{\"type\":\"response\",\"name\":\"6c5b14b9-2447-41d1-95c0-723381a69f0e\"},\"channel\":{\"channel\":\"6c5b14b9-2447-41d1-95c0-723381a69f0e\"}}}}",
    "data": {
        "response": {
            "say": "whatever",
            "$id": "ab0f9654-d4d7-4f2f-82c9-a318e4cb7792"
        },
        "$inf": {
            "type": "response",
            "req": {
                "topic": {
                    "type": "response",
                    "name": "6c5b14b9-2447-41d1-95c0-723381a69f0e"
                },
                "channel": {
                    "channel": "6c5b14b9-2447-41d1-95c0-723381a69f0e"
                }
            },
            "res": null
        }
    },
    "channel": "6c5b14b9-2447-41d1-95c0-723381a69f0e",
    "timeStamp": "2014-02-24T12:19:22.499Z"
}

This also gets the same response:

mcb.request({request: 'say', what: 'hi'}, function(res) {...});

but if a property in the request contains a dot, no response is forthcoming:

mcb.request({request: 'say', what: '.'}, function(res) {...});

{
    "channel": "postal",
    "topic": "subscription.created",
    "data": {
        "event": "subscription.created",
        "channel": "request",
        "topic": "*"
    },
    "timeStamp": "2014-02-24T12:20:30.588Z"
}
{
    "channel": "postal",
    "topic": "subscription.created",
    "data": {
        "event": "subscription.created",
        "channel": "70b41418-9987-4649-b125-f72037747986",
        "topic": "*"
    },
    "timeStamp": "2014-02-24T12:20:30.592Z"
}
{
    "topic": "{\"request\":\"say\",\"what\":\".\"}",
    "data": {
        "request": {
            "request": "say",
            "what": ".",
            "$id": "1b2cf1f5-ec24-4a31-8feb-f0fc2c79bc13"
        },
        "$inf": {
            "type": "request",
            "req": {
                "topic": {
                    "type": "request",
                    "name": "request"
                },
                "channel": {
                    "channel": "request"
                }
            },
            "res": {
                "topic": {
                    "type": "response",
                    "name": "70b41418-9987-4649-b125-f72037747986"
                },
                "channel": {
                    "channel": "70b41418-9987-4649-b125-f72037747986"
                }
            }
        }
    },
    "channel": "request",
    "timeStamp": "2014-02-24T12:20:30.613Z"
}

Pass error to the request callback

Currently the request function takes a callback that only has a res parameter, it would be better if this also accepted an error object reporting errors in microbial or kafka.

Fire and forget requests

For some situation such as logging it'd be nice to have a fire and forget pattern for requests. The code would be similar to standard request.

Cannot read property 'host' of undefined

@pelger
In our usage of microbial, I think we're experiencing the same problem as described here:
rjrodger/seneca-kafka-transport#1
There, you recommend running deregister.js to clear out slots in zookeeper, although that solution seems to be seneca-kafka-transport specific ...
Is there any other way for microbial to directly flush out zookeeper that might help us resolve?
Thanks.
Colum

error: uncaughtException: Cannot read property 'host' of undefined
error: TypeError: Cannot read property 'host' of undefined
at _makeBrokerKey (/home/me/Development/Tools/Eolas/eolas/node_modules/microbial/node_modules/kafkaesque/lib/kafkaesque.js:48:18)
at /home/me/Development/Tools/Eolas/eolas/node_modules/microbial/node_modules/kafkaesque/lib/kafkaesque.js:121:23
at Object.callback (/home/me/Development/Tools/Eolas/eolas/node_modules/microbial/node_modules/kafkaesque/lib/kafkaesque.js:56:85)
at Socket. (/home/me/Development/Tools/Eolas/eolas/node_modules/microbial/node_modules/kafkaesque/lib/api.js:246:17)
at Socket.emit (events.js:95:17)
at Socket. (stream_readable.js:764:14)
at Socket.emit (events.js:92:17)
at emitReadable (_stream_readable.js:426:10)
at emitReadable (_stream_readable.js:422:5)
at readableAddChunk (_stream_readable.js:165:9)
mccoole@ubuntu:~/Development/Tools/Eolas/eolas/services$

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.