Make message ID a function of {sender, subject, body…} about email HOT 6 OPEN

Thanks for explaining the ratio!

If everyone copied from a bad source, the bad implementations will propagate and their predominance will be (recursively) used as »proof« that they are doing it right.

Message-IDs are supposed to be unique for every unique message. That is, the same message, generated twice, should carry the same ID. Yes. ;-)

Some anecotes:

• Google Mail
• https://lists.andrew.cmu.edu/pipermail/cyrus-devel/2011-April/001815.html
• https://tools.ietf.org/html/rfc7352
• https://technet.microsoft.com/en-us/library/dd577073(v=exchg.80).aspx
• https://docs.cyrus.foundation/imap/faqs/feature-duplicate-delivery.html

from email.

Hey @wmark - thanks for the suggestion! Just curious - what benefit would this give us as opposed to the existing implementation?

How it's currently implemented should avoid duplicates, and generates an ID that is similar to that found in other mail clients. It seems like introducing a hash between the different parts of the email (assuming they exist - we'd have to check that) might be a bit overkill?

from email.

The version from 2016-01-31 is practically using a random number by rand.Reader. Everything else can be removed (!) because it has less entropy.

Therefore, the current implementation is not suitable for duplicate suppression.

Just use data obtained from struct Email as described above for the Message ID. This will result in an even more elegant implementation with less dependencies.

Don't confuse this with the mime-type delimiter. The latter just needs to be a string that is no substring of any of the (encoded) attachments.

from email.

I see what you're saying, but I based my function on this RFC draft which appears to closely mirror how other clients such as Swift mailer generate message ids. As another example, this is how Python's email.utils.make_msgid function works

def make_msgid(idstring=None):
    """Returns a string suitable for RFC 2822 compliant Message-ID, e.g:

    <142480216486.20800.16526388040877946887@nightshade.la.mastaler.com>

    Optional idstring if given is a string used to strengthen the
    uniqueness of the message id.
    """
    timeval = int(time.time()*100)
    pid = os.getpid()
    randint = random.getrandbits(64)
    if idstring is None:
        idstring = ''
    else:
        idstring = '.' + idstring
    idhost = socket.getfqdn()
    msgid = '<%d.%d.%d%s@%s>' % (timeval, pid, randint, idstring, idhost)
    return msgid

Look familiar? 😄

With this being the case, we appear to be matching how other libraries generate message Id's, so I'm not really inclined to change them. The only dependency I would be removing is in the stdlib so I don't see much benefit in removing it.

Since message-ids are supposed to be unique to each message, it seems dangerous to try and incorporate elements that are the same across an entire thread (or multiple threads) in the ID. If, however, you can find an RFC that supports doing things this way - I'm good to re-consider!

from email.

Interesting - thanks for the links! It'll take me a bit to look over these, but I'll give them a look over and will decide how best to proceed.

Thanks again!

from email.

Entropy is scarce at servers. And, the security community is moving away from randomness. So, for example, this would even be suitable for an email message before encryption (but then +a per-recipient secret):

const dateTimeRounding = 5 * time.Minute
var msg *mail.Message

    var dt time.Time
    if dt, err = msg.Header.Date(); err != nil {
        dt = time.Now().UTC()
        msg.Header["Date"] = []string{dt.Format(time.RFC1123Z)}
    }
    // Including a rounded datetime will result in suppressing otherwise identical messages within a timespan.
    // This prevents, for example, ∞ alerts within a second for a re-cycled fan (and the sender from being blacklisted due to hammering).
    dtFrac := dt.Round(dateTimeRounding).Format(time.RFC3339)

    // deterministic Message-ID (replace sha256 by sth faster if you like)
    hasher := sha256.New()
    io.Copy(hasher, msg.Body)
    hasher.Write([]byte(msg.Header.Get("from")))
    hasher.Write([]byte(dtFrac))
    hasher.Write([]byte(msg.Header.Get("subject")))
    // arbitrarily chosen 16 chars as ID are plenty; a high-radix base makes this shorter
    idLeft := base64.RawStdEncoding.EncodeToString(hasher.Sum(nil))[4:20]
    messageID := strings.Join([]string{idLeft, hostname}, "@")
    msg.Header["Message-ID"] = []string{strings.Join([]string{"<", messageID, ">"}, "")}

from email.