Comments (10)
@JasonBarnabe Correct me if I'm missing something, but hasn't Sidekiq already effectively made this async for you? It sounds like you've already done the right thing by moving mail delivery to a background job.
If mail
had its own async queue, then if the Ruby process dies or is shutdown for a deploy, you'd lose email. For reliable delivery, mail
would need to persist those messages somewhere, handle retries, inform you of permanent failures, and make it all observable with metrics. That sounds quite a bit like a background queue system. 😄
Sidekiq does a great job of handling retries and exceptions and has a persistence layer, so it sounds like a good choice for all of that. (It is worth remembering though that Sidekiq's free edition doesn't do particularly well with handling Ruby shutdowns. Other queues address that better, but usually at a cost of lower overall throughput.)
As you've already noted, you could also switch to a sendmail-compatible interface that queues mail itself. (Or reconfigure postfix to do so, if possible.)
from mail.
It's async from the perspective of the original web request that triggered the action. It's synchronous from the perspective of a sidekiq job: the job will not complete until it's gone (in our case) to postfix and gotten a response from the remote server. Like I said, we send a lot of email and we have a significant percentage of sidekiq time waiting on the I/O from the remote server.
I'm not familiar with the behaviour of the various ways to run a system command. If sendmail was called with spawn/fork/detach, does the Ruby process dying stop the sendmail command? Would a sidekiq shutdown affect it? A system shutdown?
from mail.
Note I'm not proposing/asking about mail having its own queue - just that it spawns async threads/processes whatever for the sendmail calls.
from mail.
The mail
gem is responsible for delivering a message to an MTA. It makes sense for this delivery to be synchronous so that you know if there are errors sending data to the MTA. The MTA is responsible for handling mail delivery (queueing etc). Is Sendmail's delivery mode set to queue? See man page on delivery mode.
from mail.
While some sendmail interfaces are asynchronous (queuing the message to be delivered later), others like postfix are not, so the process only completes once the mail has been received by the remote server.
It seems like it would not be able to raise an exception if the command fails. Are there any other drawbacks?
from mail.
@JasonBarnabe Are you positive postfix cannot enable an outbound mail queue? A super quick search shows articles discussing how to unstick a message in the outbound queue, which suggests that postfix has one.
Otherwise, if you can afford the extra processes that spawn/fork would require, why not just bump up the number of Sidekiq threads? Those would be lighter-weight on your servers than forking anyway.
from mail.
I am not positive that postfix cannot run asynchronously, but from what I've read, the normal sendmail way of doing it (using the DeliveryMode argument/setting) does not work with postfix. I have also added logging to this gem around the call to sendmail which shows this can often take several seconds, but I'm not 100% sure it's the normal operation of sendmail that is causing the delay and not something else.
Our bottleneck is memory usage. I would assume that spawning/forking the sendgrid command would be less memory intensive then a whole other sidekiq thread.
from mail.
Seems to me that this is out of scope for this Gem.
from mail.
@JasonBarnabe Spawning/forking would likely make your memory usage worse, not better. It would also reduce observability into what's going on, which I'd suggest is a net negative when you're already resource constrained.
I think your best bet is to address this by one of: a) changing your Postfix config to queue first, deliver second; b) replacing Postfix with another SMTP server that can take on the queuing; c) moving to a 3rd party service like AWS SES, Sendgrid, Sparkpost, Mailgun, or Postmark; or (d) upgrading your servers.
BTW, some of the multi-second delays in sending you're seeing are, some of the time, intentional on the part of the destination SMTP servers. It's a common tactic to make spamming more expensive by consuming sender resources. 😄 For legit mail, this is just a cost of doing business.
As others have mentioned, this is out of scope for the mail
gem, so perhaps you'd consider closing this Issue?
from mail.
I don't understand how considering the way mail invokes sendmail is "out of scope" given that it already invokes sendmail today, but I can understand how this could be a bad idea at least in the general sense, so closing.
from mail.
Related Issues (20)
- Please tag 2.8.1.rc2 HOT 1
- Mail on MacOS Ventura 13.02 Issue HOT 4
- Problem folding non-ASCII header HOT 3
- Test string is 1 char short of 78
- JRuby 9.3+ fails with Java::JavaLang::ArrayIndexOutOfBoundsException HOT 1
- Mail calls non-existent Net::SMTP disable_starttls_auto HOT 3
- Two warnings in date_time_parser.rb
- Error when send email with Round bracket In "From" HOT 1
- Various questions regarding built-in capabilities HOT 4
- Re-encoding a multipart message introduces extra newline characters
- Address list with comment before address parses incorrectly if no space after comma
- Gmail SMTP relay issues after upgrading to 2.8.1 HOT 2
- Forward email doesn't encode the attachment correctly HOT 1
- Running into an issue while sending an SMTP mail with v2.7.1 on Ruby 3.0.0 on container
- Encoding
- Parsing a `Mail::Message` from a string ignores the string's encoding
- Net::IMAP connection uses insecure deprecated options HOT 7
- Delivery with RCPT / NOTIFY not achievable
- Where to Report Vulnerabilities? HOT 2
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from mail.