kumocorp / kumomta Goto Github PK
View Code? Open in Web Editor NEWThe first Open-Source high-performance MTA developed from the ground-up for high-volume email sending environments.
Home Page: https://kumomta.com
License: Apache License 2.0
The first Open-Source high-performance MTA developed from the ground-up for high-volume email sending environments.
Home Page: https://kumomta.com
License: Apache License 2.0
As a DevOps engineer, I need to be able to integrate my MTA's event data against internal systems without having to poll and read the logs. Our platform currently has webhook ingestion and I want to use it with my KumoMTA servers.
As a cloud relay provider, I need to provide my tenants with webhook event data, so that they don't have to constantly visit my API. I need to dynamically configure a large quantity of webhooks on a per-tenant basis, with each tenant able to select which events they receive in their webhook.
When using a "smarthost" route, it seems required to add a trailing dot to prevent resolving to the local cloud host.
In my test, i needed to resolve this issue by adding a training dot to the domain. The domain has an A record but not an MX record, but I thought that would be fine since A would be a fallback from a missing MX anyway.
This is in the config:
kumo.on('smtp_server_message_received', function(msg)
msg:set_meta('queue', 'az.ksink.aasland.com')
end)
and I can resolve the address with a ping:
PING az.ksink.aasland.com (20.9.144.182) 56(84) bytes of data.
But...
DEBUG localset-4 insert{name="az.ksink.aasland.com"}:insert: kumod::queue: insert_ready: failed to resolve az.ksink.aasland.com: no record found for Query { name: Name("az.ksink.aasland.com.qjcierrxmllupc5mn3fhg4m1uf.xx.internal.cloudapp.net."), query_type: A, query_class: IN }
In order to fix this I needed to do this: (notice the training dot)
msg:set_meta('queue', 'az.ksink.aasland.com.')
Users should not need to add the dot.
While HAProxy support is a good step toward our users being able to leverage more IPs than can typically be allocated on a single instance, it remains limited as a solution due to how it handles connection issues.
We will improve on the current functionality available via HAProxy by implementing a more tightly integrated proxy using SOCKS5.
Consider building off:
As a Deliverability Engineer, there are numerous responses I can encounter that are best addressed by suspending my traffic on a given pathway for a period of time.
One example of this is the common Yahoo TS04 error:
421 4.7.0 [TSS04] Messages from 0.0.0.0 temporarily deferred due to user complaints - 4.16.55.1; see https://postmaster.yahooinc.com/error-codes
When such an error is encountered, the MTA needs to stop sending over that IP to the Yahoo MX group for two hours, as per Yahoo's recommendation.
Depends On:
As a Deliverability Engineer, I need to be able to define the events I'm interested in automating, based on the response returned by the remote MTA.
For this to work, I need to be able to define the response string of interest using expressions, because remote servers can return customized responses that are personalized to the individual interaction.
For example, I may receive a response such as:
451 4.2.0 68.69.14.27 Throttled - try again later. SyEWdGKFD38hOSyEbdIFU1 Please see [http://postmaster.comcast.net/smtp-error-codes.php#RL000002](http://postmaster.comcast.net/smtp-error-codes.php#RL000002%C2%A0)
In this case, the random string is not of interest, and the IP address is specific to the individual connection. My real interest is in the initial IANA response code and the RL000002 category code.
MTA-STS: https://datatracker.ietf.org/doc/html/rfc8461
The gist for us is:
enable_mta_sts
option in kumo.make_egress_path
. This should default to true
. Setting it to false
will skip resolving the MTA STS policy, and is present as a kill-switch in case something goes wrong.enable_tls
setting, but additionally applying MX host validation to the MX names as described in https://datatracker.ietf.org/doc/html/rfc8461#section-4.1. Failed hosts are treated as transient connection failures and allow the dispatcher to continue with the next candidate MX.Related:
Depends On:
We should add these logging items
The TLS version that was negotiated with the remote end during STARTTLS
The TLS cipher that was negotiated with the remote end during STARTTLS
We should report on the IP address of the egress source. Currently can report on the name only.
Logging currently includes egress_source. We should be able to report egress_source.name and egress_source.ip
Depends on supporting each of these:
When shutting down the service, logs should write out any buffered entries before allowing the service to actually shut down.
Ubuntu 20
Linux kdev2.kumomta.com 5.15.0-1039-azure #46~20.04.1-Ubuntu SMP Mon May 22 19:42:46 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux
NAME="Ubuntu"
VERSION="20.04.6 LTS (Focal Fossa)"
ID=ubuntu
ID_LIKE=debian
PRETTY_NAME="Ubuntu 20.04.6 LTS"
VERSION_ID="20.04"
HOME_URL="https://www.ubuntu.com/"
SUPPORT_URL="https://help.ubuntu.com/"
BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
VERSION_CODENAME=focal
UBUNTU_CODENAME=focal
Azure Standard D8lds v5 (8 vcpus, 16 GiB memory)
kumod 2023.06.13-2992e421
Yes, and I updated the version box above to show the version that I tried
Mail not sending and a debug trace shows no source assignment and this trace message:
stack traceback:
[C]: in function 'kumo.make_egress_pool'
/opt/kumomta/share/policy-extras/sources.lua:93: in function </opt/kumomta/share/policy-extras/sources.lua:91>
caused by: deserialize error: missing field `name`
Adding a ` print(kumo.json_encode(data.pools[pool_name]))` on line 92 showed that the required poll name was not in the data.
To fix, you need to add "name = pool," to the data:
@@ -14,6 +14,7 @@ local function load_data_from_file(file_name, target)
for pool_source, params in pairs(pool_def) do
target.pools[pool] = target.pools[pool]
or {
+ name = pool,
entries = {},
}
Note: after fixing the issue, I noticed the same thing in the source data and a similar fix was needed.
The final fix with both changes looks like this:
local function load_data_from_file(file_name, target)
local data = utils.load_json_or_toml_file(file_name)
for source, params in pairs(data.source) do
target.sources[source] = target.sources[source]
or {
name = source,
entries = {},
}
utils.merge_into(params, target.sources[source])
end
for pool, pool_def in pairs(data.pool) do
for pool_source, params in pairs(pool_def) do
target.pools[pool] = target.pools[pool]
or {
name = pool,
entries = {},
}
params.name = pool_source
table.insert(target.pools[pool].entries, params)
end
end
end
Configure to use the sources helper (https://docs.kumomta.com/userguide/configuration/sendingips/?h=helper#using-the-sourceslua-policy-helper)
Add a debug trace to the config in the init section:
kumo.set_diagnostic_log_filter 'kumod=trace'
restart and tail the system log:
sudo systemctl stop kumomta.service
sudo systemctl start kumomta.service
journalctl -f -n100 -u kumomta.service
In egress_sources.toml:
# Pool Definitions
[pool.'default']
[pool.'default'.'tomtest']
# Egress IP Definitions
[source.'tomtest']
source_address = '10.0.0.9'
ehlo_domain = 'kdev2.aasland.com'
In init.lua:
local sources = require 'policy-extras.sources'
sources:setup { '/opt/kumomta/etc/policy/egress_sources.toml' }
assign a source and pool and send mail
See above
As a Deliverability Engineer, I need to be able to record the rules I want used for my traffic shaping automation.
I would like to know that I can start from a default set of rules that are updated by the community, but can then augment/replace existing rules with logic I have created from my own research and experience.
While I would like to be able to manage those rules through a UI that is either provided or developed myself, I typically expect that the rules do not change on a very regular basis and therefore am fine with manually editing a file with my rules contained in it.
I want to be able to define rules based on a site name that represents a rollup of all domains that deliver to the same MX servers, but also may need to explicitly define rules for a domain in spite of it being a member of a given site name.
For each rule, I want to be able to specify the response in question and one or more actions to take as a result of the response.
I should be able to leave comments in the rules file so that I can reference those comments in the future to determine the logic used by myself, my peers, or my predecessors.
Depends On:
As a Deliverability Engineer, I need to comply with the feedback being provided by the remote MTAs I connect to in order to maintain deliverability and sender reputation.
For example, if the server received the following response:
451 4.7.650 The mail server [x.xx.xx.xx] has been temporarily rate limited due to IP reputation.
The server would want to lower the rate of sending on the IP address for a window of time.
This feature is composed of a number of elements as serves as the master ticket.
Depends On:
DANE for SMTP: https://datatracker.ietf.org/doc/html/rfc7672
DANE in General: https://datatracker.ietf.org/doc/html/rfc6698
TLSA resolution psuedocode: https://datatracker.ietf.org/doc/html/rfc6698#appendix-B.2
Implementing this requires solid DNSSEC support, and I'm not sure whether it currently exists.
Relevant issues in the upstream resolver library:
In addition, rustls doesn't have first-class support for DANE in general:
and it isn't clear how/if we can adapt the data and feed it into it verifier implementation
Depends On:
KumoMTA is for all use cases, which means having wide support for all RFCs pertaining to data transmission.
Lack of RFC2549 support is a glaring omission in all other MTAs, whether commercial or Open Source.
As a Deliverability Engineer I have to handle responses from remote MTAs that indicate our server is sending too quickly.
For example:
451 4.7.650 The mail server [x.xx.xx.xx] has been temporarily rate limited due to IP reputation. For e-mail delivery information, see https://postmaster.live.com (S843)[#.prod.protection.outlook.com]
I need to throttle traffic down, wait an appropriate period of time to see if the response stops being returned, then try and throttle down again.
While there are ISP responses to say I need to throttle down, there are no anticipated responses to tell me to throttle up, so I either need to make the throttle down a temporary measure, or be able to define a cadence at which the server should attempt to throttle back up, likely based on age of the IP or time since the last throttle down response.
As an admin using deferred spooling, I need a way to flag a message for immediate fsync, so that I don't lose critical messages in the event of a server failure.
While RocksDB gives the safety of a disk spool with the performance of memory spooling, many will opt for disk spooling with deferred spooling, but will have high-priority messages that must be spooled immediately. We need a way to flag those messages on injection to ensure an immediate spool to disk.
Depends On:
Depends On:
We should add these logging items
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.