Any plans to add an SQS hook and SQS Sensor? about airflow HOT 3 CLOSED

Wouldn't SQS make most sense as a broker for Celery? Seems like it may doable from this old article here:
https://www.caktusgroup.com/blog/2011/12/19/using-django-and-celery-amazon-sqs/

from airflow.

It does make sense for use with Celery, but that is a different use-case than what I have in mind.

Our data pipeline uses SQS : Agari's Data Pipeline

We use SNS+SQS actually - we publish S3 object-created notifications over SNS+SQS. SNS has push-based topics with 0 day message retention and SQS has pull-based queues with max 14 day message retention. When we publish to SNS, SNS pushes to multiple SQS queues. We have several data importers that load this data into different DBs.

So, as part of my data pipeline, I would like to detect that a queue receives a message (it signals that it is receiving data). If that sensor returns true, I advance to the next stage : checking whether my db is receiving data. If that passes, I advance to checking if the SQS queue is drained (end of data load). If any of these fail, I want email notification. If the last step succeeds, I would like to send a "Data Load Successfully Completed" email notification.

from airflow.

Interesting! An alternative would be to check on whether the file exists in s3 (you may have to use a trigger file to signify the file is fully loaded, or load into a tmp key and rename it).

But I know nothing about your setup so I'm sure you have a much better understanding of the components you need. BTW it's very easy to create hooks and operators, and we're excited for the community to extend the portfolio of external systems we integrate with.

from airflow.