nextdoor / ndkale Goto Github PK

Kale is a python task worker library that supports priority queues on Amazon SQS

Home Page: https://engblog.nextdoor.com/ac4f7886957b

License: BSD 2-Clause "Simplified" License

Python 99.84% Shell 0.16%

ndkale's Issues

Errors when running example code

I'm trying to run the example code and am running into a couple of issues. I'm trying to keep the environment as clean as possible and am using all the defaults/have not made any changes to any files. Is there something else I need to configure? Any help would be greatly appreciated -- thanks!

Here's what I'm running:

docker pull pakohan/elasticmq
docker run -p 9324:9324 pakohan/elasticmq

docker run -it --entrypoint bash python -c 'git clone https://github.com/Nextdoor/ndkale && cd ndkale && python setup.py install && cd example && ./run_worker.sh'

docker run -it --entrypoint bash python -c 'git clone https://github.com/Nextdoor/ndkale && cd ndkale && python setup.py install && cd example && ./run_publisher.sh'

Here are the issues I'm running into:

run_publisher.sh seems to have an invalid param passed to it:

Using /usr/local/lib/python3.6/site-packages
Finished processing dependencies for ndkale==0.3.0
usage: publisher.py [-h] n
publisher.py: error: unrecognized arguments: -n

elasticMQ seems to be running, but run_worker.sh seems to have difficulty connecting to it (even though the correct port is passed through on the docker command line):

Traceback (most recent call last):
  File "worker.py", line 9, in <module>
    worker.Worker().run()
  File "/usr/local/lib/python3.6/site-packages/ndkale-0.3.0-py3.6.egg/kale/worker.py", line 182, in run
  File "/usr/local/lib/python3.6/site-packages/ndkale-0.3.0-py3.6.egg/kale/queue_selector.py", line 165, in get_queue
  File "/usr/local/lib/python3.6/site-packages/ndkale-0.3.0-py3.6.egg/kale/queue_info.py", line 174, in does_queue_need_work
  File "/usr/local/lib/python3.6/site-packages/ndkale-0.3.0-py3.6.egg/kale/queue_info.py", line 158, in is_queue_empty
  File "/usr/local/lib/python3.6/site-packages/ndkale-0.3.0-py3.6.egg/kale/sqs.py", line 70, in _get_or_create_queue
  File "/usr/local/lib/python3.6/site-packages/boto-2.48.0-py3.6.egg/boto/sqs/connection.py", line 521, in get_queue
    return self.get_object('GetQueueUrl', params, Queue)
  File "/usr/local/lib/python3.6/site-packages/boto-2.48.0-py3.6.egg/boto/connection.py", line 1192, in get_object
    response = self.make_request(action, params, path, verb)
  File "/usr/local/lib/python3.6/site-packages/boto-2.48.0-py3.6.egg/boto/connection.py", line 1116, in make_request
    return self._mexe(http_request)
  File "/usr/local/lib/python3.6/site-packages/boto-2.48.0-py3.6.egg/boto/connection.py", line 1030, in _mexe
    raise ex
  File "/usr/local/lib/python3.6/site-packages/boto-2.48.0-py3.6.egg/boto/connection.py", line 943, in _mexe
    request.body, request.headers)
  File "/usr/local/lib/python3.6/http/client.py", line 1239, in request
    self._send_request(method, url, body, headers, encode_chunked)
  File "/usr/local/lib/python3.6/http/client.py", line 1285, in _send_request
    self.endheaders(body, encode_chunked=encode_chunked)
  File "/usr/local/lib/python3.6/http/client.py", line 1234, in endheaders
    self._send_output(message_body, encode_chunked=encode_chunked)
  File "/usr/local/lib/python3.6/http/client.py", line 1026, in _send_output
    self.send(msg)
  File "/usr/local/lib/python3.6/http/client.py", line 964, in send
    self.connect()
  File "/usr/local/lib/python3.6/http/client.py", line 936, in connect
    (self.host,self.port), self.timeout, self.source_address)
  File "/usr/local/lib/python3.6/socket.py", line 724, in create_connection
    raise err
  File "/usr/local/lib/python3.6/socket.py", line 713, in create_connection
    sock.connect(sa)
ConnectionRefusedError: [Errno 111] Connection refused

https://rollbar.com/nextdoor/nextdoor/items/237418/

We get a ton of these rollbars. Can we get rid of them?

would this just work?:

if message in self._incomplete_messages:

self._incomplete_messages.remove(message)

Images

Broken

I don't think this works anymore. At least the example is broken.

MESSAGE_QUEUE_USE_PROXY = True
MESSAGE_QUEUE_PROXY_PORT = 9324
MESSAGE_QUEUE_PROXY_HOST = os.getenv('MESSAGE_QUEUE_PROXY_HOST', '0.0.0.0')

Are not even used anywhere in the code.

Setting MESSAGE_QUEUE_ENDPOINT_URL = 'http://localhost:9324' seems to work for ElasticMQ connectivity.

This also seems broken

./run_publisher.sh
usage: publisher.py [-h] n
publisher.py: error: unrecognized arguments: -n

MessageGroupId should be included as a variable in kale.publisher

When trying to connect to a AWS SQS queue with an enabled content-based deduplication it returns error stated below:

ndkale-master/example/src/ndkale/kale/publisher.py", line 60:
botocore.exceptions.ClientError: An error occurred (MissingParameter) when calling the SendMessage operation: The request must contain the parameter MessageGroupId.

This issue can be easily fix by adding the MessageGroupId paramter on the sqs_queue.send_message() function on the publisher.py

YAMLLoadWarning: calling yaml.load() without Loader=... is deprecated, as the default Loader is unsafe

Hi folks,

I've the following error when running taskwork.

python3.7/site-packages/kale/queue_info.py:192: YAMLLoadWarning: calling yaml.load() without Loader=... is deprecated, as the default Loader is unsafe. Please read https://msg.pyyaml.org/load for full details.
queues_from_config = yaml.load(fp)

kind regards,

MCaputo.

fixed:
commit

Q: Why are you using multiple queues?

Why are you using multiple queues instead of a single queue with priorities set per task?

Thanks in advance!

DelaySeconds parameter not compatible with FIFO Queue.

This error can also be found at ndkale/example/kale/publisher.py: line 60,

Fix could be initializing first the standard parameters into a dict object

params = {
    "MessageBody":kale_msg.encode()
}

then appropriating the different requirements for each queue types into it

if someFunctionToGetQueueType == 'FIFO':
    params[MessageGroupId] =  group_id or 'default',
elif someFunctionToGetQueueType ==== 'Standard':
    params[DelaySeconds] = delay_sec or 1

which then could be passed to the send_message() function as

sqs_queue.send_message(**params)

Not available in pypi

It looks like this was pushed to pypi recently but is no longer available, e.g. https://pypi.org/project/ndkale/ gives a 404. Are you planning to publish this to pypi?

worker.py does not execute in example

The ./run_worker.sh fails with exit code 1(seemingly from _check_process_resources) when I try to run the included example.

Recommend Projects

React

A declarative, efficient, and flexible JavaScript library for building user interfaces.
Vue.js

🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
Typescript

TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
TensorFlow

An Open Source Machine Learning Framework for Everyone
Django

The Web framework for perfectionists with deadlines.
Laravel

A PHP framework for web artisans
D3

Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

javascript

JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
web

Some thing interesting about web. New door for the world.
server

A server is a program made to process requests and deliver data to clients.
Machine learning

Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Visualization

Some thing interesting about visualization, use data art
Game

Some thing interesting about game, make everyone happy.

Recommend Org

Facebook

We are working to build community through open source technology. NB: members must have two-factor auth.
Microsoft

Open source projects and samples from Microsoft.
Google

Google ❤️ Open Source for everyone.
Alibaba

Alibaba Open Source for everyone
D3

Data-Driven Documents codes.
Tencent

China tencent open source team.