nextdoor / ndkale Goto Github PK

View Code? Open in Web Editor NEW

203.0 120.0 37.0 201 KB

Kale is a python task worker library that supports priority queues on Amazon SQS

Home Page: https://engblog.nextdoor.com/ac4f7886957b

License: BSD 2-Clause "Simplified" License

Python 99.84% Shell 0.16%

ndkale's People

Stargazers

Watchers

ndkale's Issues

Q: Why are you using multiple queues?

Why are you using multiple queues instead of a single queue with priorities set per task?

Thanks in advance!

YAMLLoadWarning: calling yaml.load() without Loader=... is deprecated, as the default Loader is unsafe

Hi folks,

I've the following error when running taskwork.

python3.7/site-packages/kale/queue_info.py:192: YAMLLoadWarning: calling yaml.load() without Loader=... is deprecated, as the default Loader is unsafe. Please read https://msg.pyyaml.org/load for full details.
queues_from_config = yaml.load(fp)

kind regards,

MCaputo.

fixed:
commit

Errors when running example code

I'm trying to run the example code and am running into a couple of issues. I'm trying to keep the environment as clean as possible and am using all the defaults/have not made any changes to any files. Is there something else I need to configure? Any help would be greatly appreciated -- thanks!

Here's what I'm running:

docker pull pakohan/elasticmq
docker run -p 9324:9324 pakohan/elasticmq

docker run -it --entrypoint bash python -c 'git clone https://github.com/Nextdoor/ndkale && cd ndkale && python setup.py install && cd example && ./run_worker.sh'

docker run -it --entrypoint bash python -c 'git clone https://github.com/Nextdoor/ndkale && cd ndkale && python setup.py install && cd example && ./run_publisher.sh'

Here are the issues I'm running into:

run_publisher.sh seems to have an invalid param passed to it:

Using /usr/local/lib/python3.6/site-packages
Finished processing dependencies for ndkale==0.3.0
usage: publisher.py [-h] n
publisher.py: error: unrecognized arguments: -n

elasticMQ seems to be running, but run_worker.sh seems to have difficulty connecting to it (even though the correct port is passed through on the docker command line):

Traceback (most recent call last):
  File "worker.py", line 9, in <module>
    worker.Worker().run()
  File "/usr/local/lib/python3.6/site-packages/ndkale-0.3.0-py3.6.egg/kale/worker.py", line 182, in run
  File "/usr/local/lib/python3.6/site-packages/ndkale-0.3.0-py3.6.egg/kale/queue_selector.py", line 165, in get_queue
  File "/usr/local/lib/python3.6/site-packages/ndkale-0.3.0-py3.6.egg/kale/queue_info.py", line 174, in does_queue_need_work
  File "/usr/local/lib/python3.6/site-packages/ndkale-0.3.0-py3.6.egg/kale/queue_info.py", line 158, in is_queue_empty
  File "/usr/local/lib/python3.6/site-packages/ndkale-0.3.0-py3.6.egg/kale/sqs.py", line 70, in _get_or_create_queue
  File "/usr/local/lib/python3.6/site-packages/boto-2.48.0-py3.6.egg/boto/sqs/connection.py", line 521, in get_queue
    return self.get_object('GetQueueUrl', params, Queue)
  File "/usr/local/lib/python3.6/site-packages/boto-2.48.0-py3.6.egg/boto/connection.py", line 1192, in get_object
    response = self.make_request(action, params, path, verb)
  File "/usr/local/lib/python3.6/site-packages/boto-2.48.0-py3.6.egg/boto/connection.py", line 1116, in make_request
    return self._mexe(http_request)
  File "/usr/local/lib/python3.6/site-packages/boto-2.48.0-py3.6.egg/boto/connection.py", line 1030, in _mexe
    raise ex
  File "/usr/local/lib/python3.6/site-packages/boto-2.48.0-py3.6.egg/boto/connection.py", line 943, in _mexe
    request.body, request.headers)
  File "/usr/local/lib/python3.6/http/client.py", line 1239, in request
    self._send_request(method, url, body, headers, encode_chunked)
  File "/usr/local/lib/python3.6/http/client.py", line 1285, in _send_request
    self.endheaders(body, encode_chunked=encode_chunked)
  File "/usr/local/lib/python3.6/http/client.py", line 1234, in endheaders
    self._send_output(message_body, encode_chunked=encode_chunked)
  File "/usr/local/lib/python3.6/http/client.py", line 1026, in _send_output
    self.send(msg)
  File "/usr/local/lib/python3.6/http/client.py", line 964, in send
    self.connect()
  File "/usr/local/lib/python3.6/http/client.py", line 936, in connect
    (self.host,self.port), self.timeout, self.source_address)
  File "/usr/local/lib/python3.6/socket.py", line 724, in create_connection
    raise err
  File "/usr/local/lib/python3.6/socket.py", line 713, in create_connection
    sock.connect(sa)
ConnectionRefusedError: [Errno 111] Connection refused

Images

worker.py does not execute in example

The ./run_worker.sh fails with exit code 1(seemingly from _check_process_resources) when I try to run the included example.

https://rollbar.com/nextdoor/nextdoor/items/237418/

We get a ton of these rollbars. Can we get rid of them?

would this just work?:

if message in self._incomplete_messages:

self._incomplete_messages.remove(message)

Broken

I don't think this works anymore. At least the example is broken.

MESSAGE_QUEUE_USE_PROXY = True
MESSAGE_QUEUE_PROXY_PORT = 9324
MESSAGE_QUEUE_PROXY_HOST = os.getenv('MESSAGE_QUEUE_PROXY_HOST', '0.0.0.0')

Are not even used anywhere in the code.

Setting MESSAGE_QUEUE_ENDPOINT_URL = 'http://localhost:9324' seems to work for ElasticMQ connectivity.

This also seems broken

./run_publisher.sh
usage: publisher.py [-h] n
publisher.py: error: unrecognized arguments: -n

DelaySeconds parameter not compatible with FIFO Queue.

This error can also be found at ndkale/example/kale/publisher.py: line 60,

Fix could be initializing first the standard parameters into a dict object

params = {
    "MessageBody":kale_msg.encode()
}

then appropriating the different requirements for each queue types into it

if someFunctionToGetQueueType == 'FIFO':
    params[MessageGroupId] =  group_id or 'default',
elif someFunctionToGetQueueType ==== 'Standard':
    params[DelaySeconds] = delay_sec or 1

which then could be passed to the send_message() function as

sqs_queue.send_message(**params)

MessageGroupId should be included as a variable in kale.publisher

When trying to connect to a AWS SQS queue with an enabled content-based deduplication it returns error stated below:

ndkale-master/example/src/ndkale/kale/publisher.py", line 60:
botocore.exceptions.ClientError: An error occurred (MissingParameter) when calling the SendMessage operation: The request must contain the parameter MessageGroupId.

This issue can be easily fix by adding the MessageGroupId paramter on the sqs_queue.send_message() function on the publisher.py

Not available in pypi

It looks like this was pushed to pypi recently but is no longer available, e.g. https://pypi.org/project/ndkale/ gives a 404. Are you planning to publish this to pypi?

nextdoor / ndkale Goto Github PK

ndkale's People

Stargazers

Watchers

Forkers

ndkale's Issues

Q: Why are you using multiple queues?

YAMLLoadWarning: calling yaml.load() without Loader=... is deprecated, as the default Loader is unsafe

Errors when running example code

Images

worker.py does not execute in example

https://rollbar.com/nextdoor/nextdoor/items/237418/

Broken

DelaySeconds parameter not compatible with FIFO Queue.

MessageGroupId should be included as a variable in kale.publisher

Not available in pypi

Recommend Projects

React

Vue.js

Typescript

TensorFlow

Django

Laravel

D3

Recommend Topics

javascript

web

server

Machine learning

Visualization

Game

Recommend Org

Facebook

Microsoft

Google

Alibaba

D3

Tencent