imap2json

License: MIT

Acknowledgements: Maxim Khitrov

WARNING: DOES NOT WORK WITH GMAIL IMAPD SINCE IT DOES NOT IMPLEMENT THE STANDARD RFC5256

imap2json converts an imap URL of a folder to a HTML conversation view (Organised by thread)

Dependencies:

• (Dovecot) IMAP server for http://tools.ietf.org/html/rfc5256
• Golang for sanitising and serialising IMAP response to JSON
• Web interface to render the JSON, between familiar index and expanded conversation views

Example use case:

Ruth and Kai have an email exchange arranging a movie. A friend Jamie wants to catch up on the conversation and perhaps join. Kai emails Jamie the link to the start of the conversation.

The URL points to the Web email archive produced imap2json of the entire conversation. Now Jamie knows all the details, in a familiar Web conversation view, that is mails of the same topic are grouped together, of the email exchange. There is no need for Kai or Ruth to use the clumsy "Forward all" mechanism on their email client.

Example usage:

1. go get github.com/kaihendry/imap2json

• $GOPATH/bin/imap2json imap://imap.dabase.com

This will generate mail.json from our test repo imap://imap.dabase.com

imap.dabase.com is a documented working read only IMAP server, serving an example Maildir.

For a simple Web interface (json2html) for mail.json see serve our html directory from your Webserver.

Why use Dovecot?

Dovecot implements RFC5256 threading which can be used to group emails into conversations like Gmail does.

This saves us from having to code this tricky part ourselves by tracking Message-ID: and References: / In-Reply-To:, which can be rather complex.

IMAP also gives us a uniform interface at an IMAP URL, e.g. imap://$USER[@]imap.dabase.com[/folder] to generate an archive from.

Raw dovecot output

On a local dovecot instance using Maildir and minimal Dovecot configuration:

printf "1 select inbox\n2 UID THREAD references us-ascii all\n3 fetch 1:* envelope\n4 logout\n" |
/usr/lib/dovecot/imap

Or using netcat with our read only test IMAP service, so you don't need to setup Dovecot:

nc imap.dabase.com imap << EOF
1 authenticate anonymous foobar
2 select inbox
3 UID THREAD references UTF-8 all
4 fetch 1:* envelope
5 logout
EOF

Dovecot's THREAD rfc5256

THREAD (1 2 3 4)(5)

We can see messages 1,2,3,4 are a single conversation about a "Movie".

5 is a separate conversation about a Dentist appointment.

-- 1
   \-- 2
	   |-- 3
		   \-- 4
-- 5

Mutt's view

Mutt is an excellent MUA and reference point.

mutt -f Maildir

Mutt conjures a great threaded conversation view as a reference independent of Dovecot's THREAD rfc5256 support. So it would be interesting to eventually compare the threading structure.

Using mutt to debug IMAP commands

mutt needs to be compiled with --enable-debug.

$ mutt -v | grep -i debug
Configure options: '--prefix=/usr' '--sysconfdir=/etc' '--enable-imap' '--enable-debug' '--enable-smtp' '--enable-hcache' '--with-curses=/usr' '--with-regex' '--with-gss=/usr' '--with-ssl=/usr' '--with-sasl' '--with-idn' 'CFLAGS=-march=x86-64 -mtune=generic -O2 -pipe -fstack-protector --param=ssp-buffer-size=4 -D_FORTIFY_SOURCE=2' 'LDFLAGS=-Wl,-O1,--sort-common,--as-needed,-z,relro,--hash-style=gnu'
+DEBUG

We need a mutt configuration for testing:

$ cat mutt-imap2json
set spoolfile=imap://[email protected]
set folder=imap://[email protected]
set sort=threads
set sort_aux=reverse-last-date-received

Now run mutt in debug mode with the above configuration:

mutt -d2 -F mutt-imap2json

Look at .muttdebug0 for a complete log

grep '4>' .muttdebug0 # show the IMAP commands issued by mutt

Golang

Going to use Golang to interface with Dovecot IMAP using http://godoc.org/code.google.com/p/go-imap

http://golang.org/pkg/net/mail/ can help parse/sanitise addresses, text bodies and convert Dates to Epoch.

Then json.MarshalIndent the sanitised email metadata & body into JSON.

Hoping to statically generate a large JSON file per mailbox. This will be compressed on delivery to the Web interface.

JSON to Web interface

Javascript needs to be written to AJAX the JSON and produce the output. Initially as close to Gmail's or Fastmail's conversation view as possible.

Perhaps a NodeJS can be written to statically generate the HTML too, so no Javascript is required, making it Web crawler friendly.

The index & expanded views are probably most important.

Conversation URL structure

Gmail uses 16 character alphanumeric string as a conversation id, e.g. https://mail.google.com/mail/u/0/#inbox/1424243f629686c1

Fastmail uses two 16 character alphanumeric strings with a dash in between, e.g. https://www.fastmail.fm/mail/Inbox/5cf2a643c23b82dc-f65945369u632?u=148c411e

The sha1sum of the first message (\r\n line endings) of a conversation could be used as the id of the conversation.

$ unix2dos 'Maildir/cur/1386050966.M540929P13587.sg.webconverger.com,S=315,W=328:2,S'
unix2dos: converting file Maildir/cur/1386050966.M540929P13587.sg.webconverger.com,S=315,W=328:2,S to DOS format ...
$ sha1sum 'Maildir/cur/1386050966.M540929P13587.sg.webconverger.com,S=315,W=328:2,S'
5048d370149a7a5d25dc17869cb1404cf747b6bb  Maildir/cur/1386050966.M540929P13587.sg.webconverger.com,S=315,W=328:2,S

So the "movie" conversation would be identified by 5048d370149a7a5d25dc17869cb1404cf747b6bb

http://imap2json.dabase.com/#5048d370149a7a5d25dc17869cb1404cf747b6bb or rather some short, easily sharable version like: http://imap2json.dabase.com/#5048d

Other mail archive projects

Mailman's hellokitty demo

• http://play.golang.org/p/kqroEKVxRD