Git Product home page Git Product logo

ndkale's People

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

ndkale's Issues

Errors when running example code

I'm trying to run the example code and am running into a couple of issues. I'm trying to keep the environment as clean as possible and am using all the defaults/have not made any changes to any files. Is there something else I need to configure? Any help would be greatly appreciated -- thanks!

Here's what I'm running:

docker pull pakohan/elasticmq
docker run -p 9324:9324 pakohan/elasticmq

docker run -it --entrypoint bash python -c 'git clone https://github.com/Nextdoor/ndkale && cd ndkale && python setup.py install && cd example && ./run_worker.sh'

docker run -it --entrypoint bash python -c 'git clone https://github.com/Nextdoor/ndkale && cd ndkale && python setup.py install && cd example && ./run_publisher.sh'

Here are the issues I'm running into:

  1. run_publisher.sh seems to have an invalid param passed to it:
Using /usr/local/lib/python3.6/site-packages
Finished processing dependencies for ndkale==0.3.0
usage: publisher.py [-h] n
publisher.py: error: unrecognized arguments: -n
  1. elasticMQ seems to be running, but run_worker.sh seems to have difficulty connecting to it (even though the correct port is passed through on the docker command line):
Traceback (most recent call last):
  File "worker.py", line 9, in <module>
    worker.Worker().run()
  File "/usr/local/lib/python3.6/site-packages/ndkale-0.3.0-py3.6.egg/kale/worker.py", line 182, in run
  File "/usr/local/lib/python3.6/site-packages/ndkale-0.3.0-py3.6.egg/kale/queue_selector.py", line 165, in get_queue
  File "/usr/local/lib/python3.6/site-packages/ndkale-0.3.0-py3.6.egg/kale/queue_info.py", line 174, in does_queue_need_work
  File "/usr/local/lib/python3.6/site-packages/ndkale-0.3.0-py3.6.egg/kale/queue_info.py", line 158, in is_queue_empty
  File "/usr/local/lib/python3.6/site-packages/ndkale-0.3.0-py3.6.egg/kale/sqs.py", line 70, in _get_or_create_queue
  File "/usr/local/lib/python3.6/site-packages/boto-2.48.0-py3.6.egg/boto/sqs/connection.py", line 521, in get_queue
    return self.get_object('GetQueueUrl', params, Queue)
  File "/usr/local/lib/python3.6/site-packages/boto-2.48.0-py3.6.egg/boto/connection.py", line 1192, in get_object
    response = self.make_request(action, params, path, verb)
  File "/usr/local/lib/python3.6/site-packages/boto-2.48.0-py3.6.egg/boto/connection.py", line 1116, in make_request
    return self._mexe(http_request)
  File "/usr/local/lib/python3.6/site-packages/boto-2.48.0-py3.6.egg/boto/connection.py", line 1030, in _mexe
    raise ex
  File "/usr/local/lib/python3.6/site-packages/boto-2.48.0-py3.6.egg/boto/connection.py", line 943, in _mexe
    request.body, request.headers)
  File "/usr/local/lib/python3.6/http/client.py", line 1239, in request
    self._send_request(method, url, body, headers, encode_chunked)
  File "/usr/local/lib/python3.6/http/client.py", line 1285, in _send_request
    self.endheaders(body, encode_chunked=encode_chunked)
  File "/usr/local/lib/python3.6/http/client.py", line 1234, in endheaders
    self._send_output(message_body, encode_chunked=encode_chunked)
  File "/usr/local/lib/python3.6/http/client.py", line 1026, in _send_output
    self.send(msg)
  File "/usr/local/lib/python3.6/http/client.py", line 964, in send
    self.connect()
  File "/usr/local/lib/python3.6/http/client.py", line 936, in connect
    (self.host,self.port), self.timeout, self.source_address)
  File "/usr/local/lib/python3.6/socket.py", line 724, in create_connection
    raise err
  File "/usr/local/lib/python3.6/socket.py", line 713, in create_connection
    sock.connect(sa)
ConnectionRefusedError: [Errno 111] Connection refused

Broken

I don't think this works anymore. At least the example is broken.

MESSAGE_QUEUE_USE_PROXY = True
MESSAGE_QUEUE_PROXY_PORT = 9324
MESSAGE_QUEUE_PROXY_HOST = os.getenv('MESSAGE_QUEUE_PROXY_HOST', '0.0.0.0')

Are not even used anywhere in the code.

Setting MESSAGE_QUEUE_ENDPOINT_URL = 'http://localhost:9324' seems to work for ElasticMQ connectivity.

This also seems broken

./run_publisher.sh
usage: publisher.py [-h] n
publisher.py: error: unrecognized arguments: -n

DelaySeconds parameter not compatible with FIFO Queue.

This error can also be found at ndkale/example/kale/publisher.py: line 60,

Fix could be initializing first the standard parameters into a dict object

params = {
    "MessageBody":kale_msg.encode()
}

then appropriating the different requirements for each queue types into it

if someFunctionToGetQueueType == 'FIFO':
    params[MessageGroupId] =  group_id or 'default',
elif someFunctionToGetQueueType ==== 'Standard':
    params[DelaySeconds] = delay_sec or 1

which then could be passed to the send_message() function as

sqs_queue.send_message(**params)

MessageGroupId should be included as a variable in kale.publisher

When trying to connect to a AWS SQS queue with an enabled content-based deduplication it returns error stated below:

ndkale-master/example/src/ndkale/kale/publisher.py", line 60:
botocore.exceptions.ClientError: An error occurred (MissingParameter) when calling the SendMessage operation: The request must contain the parameter MessageGroupId.

This issue can be easily fix by adding the MessageGroupId paramter on the sqs_queue.send_message() function on the publisher.py

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.