nextdoor / ndkale Goto Github PK
View Code? Open in Web Editor NEWKale is a python task worker library that supports priority queues on Amazon SQS
Home Page: https://engblog.nextdoor.com/ac4f7886957b
License: BSD 2-Clause "Simplified" License
Kale is a python task worker library that supports priority queues on Amazon SQS
Home Page: https://engblog.nextdoor.com/ac4f7886957b
License: BSD 2-Clause "Simplified" License
Why are you using multiple queues instead of a single queue with priorities set per task?
Thanks in advance!
Hi folks,
I've the following error when running taskwork.
python3.7/site-packages/kale/queue_info.py:192: YAMLLoadWarning: calling yaml.load() without Loader=... is deprecated, as the default Loader is unsafe. Please read https://msg.pyyaml.org/load for full details.
queues_from_config = yaml.load(fp)
kind regards,
MCaputo.
fixed:
commit
I'm trying to run the example code and am running into a couple of issues. I'm trying to keep the environment as clean as possible and am using all the defaults/have not made any changes to any files. Is there something else I need to configure? Any help would be greatly appreciated -- thanks!
Here's what I'm running:
docker pull pakohan/elasticmq
docker run -p 9324:9324 pakohan/elasticmq
docker run -it --entrypoint bash python -c 'git clone https://github.com/Nextdoor/ndkale && cd ndkale && python setup.py install && cd example && ./run_worker.sh'
docker run -it --entrypoint bash python -c 'git clone https://github.com/Nextdoor/ndkale && cd ndkale && python setup.py install && cd example && ./run_publisher.sh'
Here are the issues I'm running into:
Using /usr/local/lib/python3.6/site-packages
Finished processing dependencies for ndkale==0.3.0
usage: publisher.py [-h] n
publisher.py: error: unrecognized arguments: -n
Traceback (most recent call last):
File "worker.py", line 9, in <module>
worker.Worker().run()
File "/usr/local/lib/python3.6/site-packages/ndkale-0.3.0-py3.6.egg/kale/worker.py", line 182, in run
File "/usr/local/lib/python3.6/site-packages/ndkale-0.3.0-py3.6.egg/kale/queue_selector.py", line 165, in get_queue
File "/usr/local/lib/python3.6/site-packages/ndkale-0.3.0-py3.6.egg/kale/queue_info.py", line 174, in does_queue_need_work
File "/usr/local/lib/python3.6/site-packages/ndkale-0.3.0-py3.6.egg/kale/queue_info.py", line 158, in is_queue_empty
File "/usr/local/lib/python3.6/site-packages/ndkale-0.3.0-py3.6.egg/kale/sqs.py", line 70, in _get_or_create_queue
File "/usr/local/lib/python3.6/site-packages/boto-2.48.0-py3.6.egg/boto/sqs/connection.py", line 521, in get_queue
return self.get_object('GetQueueUrl', params, Queue)
File "/usr/local/lib/python3.6/site-packages/boto-2.48.0-py3.6.egg/boto/connection.py", line 1192, in get_object
response = self.make_request(action, params, path, verb)
File "/usr/local/lib/python3.6/site-packages/boto-2.48.0-py3.6.egg/boto/connection.py", line 1116, in make_request
return self._mexe(http_request)
File "/usr/local/lib/python3.6/site-packages/boto-2.48.0-py3.6.egg/boto/connection.py", line 1030, in _mexe
raise ex
File "/usr/local/lib/python3.6/site-packages/boto-2.48.0-py3.6.egg/boto/connection.py", line 943, in _mexe
request.body, request.headers)
File "/usr/local/lib/python3.6/http/client.py", line 1239, in request
self._send_request(method, url, body, headers, encode_chunked)
File "/usr/local/lib/python3.6/http/client.py", line 1285, in _send_request
self.endheaders(body, encode_chunked=encode_chunked)
File "/usr/local/lib/python3.6/http/client.py", line 1234, in endheaders
self._send_output(message_body, encode_chunked=encode_chunked)
File "/usr/local/lib/python3.6/http/client.py", line 1026, in _send_output
self.send(msg)
File "/usr/local/lib/python3.6/http/client.py", line 964, in send
self.connect()
File "/usr/local/lib/python3.6/http/client.py", line 936, in connect
(self.host,self.port), self.timeout, self.source_address)
File "/usr/local/lib/python3.6/socket.py", line 724, in create_connection
raise err
File "/usr/local/lib/python3.6/socket.py", line 713, in create_connection
sock.connect(sa)
ConnectionRefusedError: [Errno 111] Connection refused
The ./run_worker.sh
fails with exit code 1(seemingly from _check_process_resources
) when I try to run the included example.
We get a ton of these rollbars. Can we get rid of them?
would this just work?:
if message in self._incomplete_messages:
self._incomplete_messages.remove(message)
I don't think this works anymore. At least the example is broken.
MESSAGE_QUEUE_USE_PROXY = True
MESSAGE_QUEUE_PROXY_PORT = 9324
MESSAGE_QUEUE_PROXY_HOST = os.getenv('MESSAGE_QUEUE_PROXY_HOST', '0.0.0.0')
Are not even used anywhere in the code.
Setting MESSAGE_QUEUE_ENDPOINT_URL = 'http://localhost:9324' seems to work for ElasticMQ connectivity.
This also seems broken
./run_publisher.sh
usage: publisher.py [-h] n
publisher.py: error: unrecognized arguments: -n
This error can also be found at ndkale/example/kale/publisher.py: line 60,
Fix could be initializing first the standard parameters into a dict object
params = {
"MessageBody":kale_msg.encode()
}
then appropriating the different requirements for each queue types into it
if someFunctionToGetQueueType == 'FIFO':
params[MessageGroupId] = group_id or 'default',
elif someFunctionToGetQueueType ==== 'Standard':
params[DelaySeconds] = delay_sec or 1
which then could be passed to the send_message() function as
sqs_queue.send_message(**params)
When trying to connect to a AWS SQS queue with an enabled content-based deduplication it returns error stated below:
ndkale-master/example/src/ndkale/kale/publisher.py", line 60:
botocore.exceptions.ClientError: An error occurred (MissingParameter) when calling the SendMessage operation: The request must contain the parameter MessageGroupId.
This issue can be easily fix by adding the MessageGroupId paramter on the sqs_queue.send_message() function on the publisher.py
It looks like this was pushed to pypi recently but is no longer available, e.g. https://pypi.org/project/ndkale/ gives a 404. Are you planning to publish this to pypi?
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.