jmapio / jmap-perl Goto Github PK

Does the proxy have one yet? I don't think so, but that's now a problem because I need to discover the download endpoint to use a message blobId.

This had some setup. I have four messages in a FastMail account. All were unread. Via JMAP, I can see that they were all unread. All good.

In FastMail, I read two of them. FastMail does the right thing and I can confirm it worked via IMAP:

. SELECT INBOX
* 4 EXISTS
* 0 RECENT
* FLAGS (\Answered \Flagged \Draft \Deleted \Seen $X-ME-Annot-2 $IsMailingList $IsNotification $HasAttachment $HasTD $IsTrusted $NotJunk $Forwarded $Junk $X-FM-Device-Push)
* OK [PERMANENTFLAGS (\Answered \Flagged \Draft \Deleted \Seen $X-ME-Annot-2 $IsMailingList $IsNotification $HasAttachment $HasTD $IsTrusted $NotJunk $Forwarded $Junk $X-FM-Device-Push \*)] Ok
* OK [UNSEEN 3] Ok
* OK [UIDVALIDITY 1386137841] Ok
* OK [UIDNEXT 3129] Ok
* OK [HIGHESTMODSEQ 17365878007025304772] Ok
* OK [URLMECH INTERNAL] Ok
* OK [ANNOTATIONS 65536] Ok
. OK [READ-WRITE] Completed
. FETCH 1:* (FLAGS)
* 1 FETCH (FLAGS (\Seen $X-ME-Annot-2))
* 2 FETCH (FLAGS (\Seen $X-ME-Annot-2))
* 3 FETCH (FLAGS ($X-ME-Annot-2))
* 4 FETCH (FLAGS ($X-ME-Annot-2))
. OK Completed (0.000 sec)

The flag changes aren't reflected via the proxy, including in the client.

In the getMailboxes response:

            {
               "id" : "9",
               "mayAddItems" : true,
               "mayCreateChild" : true,
               "mayDelete" : false,
               "mayReadItems" : true,
               "mayRemoveItems" : true,
               "mayRename" : false,
               "mustBeOnlyMailbox" : true,
               "name" : "Inbox",
               "parentId" : null,
               "role" : "inbox",
               "sortOrder" : 1,
               "totalMessages" : 4,
               "totalThreads" : 4,
               "unreadMessages" : 4,
               "unreadThreads" : 4
            },

The individual messages all have isUnread: true as well (not showing here, its loooooong).

Further, changing the isUnread flag (by opening a message) doesn't do the right thing:

[
   [
      "setMessages",
      {
         "create" : {},
         "destroy" : [],
         "state" : "328",
         "update" : {
            "m76fdbd6d5" : {
               "isUnread" : false
            }
         }
      },
      "0"
   ],
   [
      "getMailboxes",
      {
         "ids" : [
            "9"
         ],
         "properties" : [
            "totalMessages",
            "unreadMessages",
            "totalThreads",
            "unreadThreads"
         ]
      },
      "1"
   ]
]

[
   [
      "messagesSet",
      {
         "accountId" : "9e2e4652-3191-11e5-847c-0026b9fac7aa",
         "created" : {},
         "destroyed" : [],
         "newState" : null,
         "notCreated" : {},
         "notDestroyed" : {},
         "notUpdated" : {},
         "oldState" : null,
         "updated" : [
            "m76fdbd6d5"
         ]
      },
      "0"
   ],
   [
      "mailboxes",
      {
         "accountId" : "9e2e4652-3191-11e5-847c-0026b9fac7aa",
         "list" : [
            {
               "id" : "9",
               "totalMessages" : 4,
               "totalThreads" : 4,
               "unreadMessages" : 4,
               "unreadThreads" : 4
            }
         ],
         "notFound" : null,
         "state" : "328"
      },
      "1"
   ]
]

So it claims the update has occurred, but that's not reflected in the mailboxes response, and there's no change on the backend IMAP server either.

Step to reproduce:

1. create a new mailbox with setMailboxes
2. upload and import a message to that mailbox with importMessages
3. observe that the message has appeared in the backing IMAP store
4. observe that the totalMessages in the created mailbox is still 0
5. observe that getMessages and getThreads can't get the message or thread by ids provided in step 2

If I stop and restart the proxy server (server.pl):

• fetching the message by id does not work
• …but fetching the mailbox now shows the right message count (often only on the second try)
• …and once it does, I can fetch the message by id

Spec:

If an id given cannot be found, the update or destroy MUST be rejected with a notFound set error.

I ask for all the ids of my mailboxes:

~/code/topicbox$ ./jc getMailboxes | jq '.[0][1].list | .[] | .id'
"1"
"2"
"3"
"4"
"5"
"6"
"7"
"8"
"10"
"11"
"12"

So, I have mailboxes with ids 1 .. 12, but not 9. I'll try updating the name on 13.

~/code/topicbox$ ./jc setMailboxes '{update:{13:{name:"YourFace"}}}'
[
   [
      "mailboxesSet",
      {
         "accountId" : "11049eb6-1dcc-11e6-82c1-f23c91556942",
         "created" : {},
         "destroyed" : [],
         "newState" : null,
         "notCreated" : {},
         "notDestroyed" : {},
         "notUpdated" : {},
         "oldState" : null,
         "updated" : [
            "13"
         ]
      },
      "a"
   ]
]

Seems to work, but:

~/code/topicbox$ ./jc getMailboxes '{ids:["13"]}'
[
   [
      "mailboxes",
      {
         "accountId" : "11049eb6-1dcc-11e6-82c1-f23c91556942",
         "list" : [],
         "notFound" : [
            "13"
         ],
         "state" : "16"
      },
      "a"
   ]
]

For several reasons, but the main one is that the data gets stored in a TEXT column in the sqlite DB.

It's because of the BINARY fetch command the proxy is sending.
I don't understand exactly why as I'm no IMAP expert, but using BODY instead works fine.

Lots of things in ImapDB aren't really IMAP specfic, and mean it's not so easy to plug in a "purelocal" server version

getThreadUpdates is returning message ids instead of thread ids.

This also means if you say 'fetchRecords => true', it claims the items are not found (since its looking up thread.id using message.id).

Trying to use the live version of the proxy (https://proxy.jmap.io/) gives a "Sign in with Google temporarily disabled for this app" error for Gmail. Using an IMAP connection gives a "Account {id} does not exist, please re-create" error.

Trying to run the project locally fails during the install instructions with an error about the libemail-address-xs-perl package. Changing this to libemail-address-perl (which appears to provide the same packages, but could be wrong) gives the following much later in the build:

Use of uninitialized value $Net::CardDAVTalk::VERSION in concatenation (.) or string at t/00-load.t line 13.

(Tested on both Debian Jessie and Ubuntu 16.04.)

Am I doing something wrong here? Was hoping to be able to test out JMAP!

I accidentally sent a bogus setMailboxes request and surprisingly got a completely bizarre response back:

~/code/topicbox$ ./jc setMailboxes '{create:{foo:{}}}'
[
   [
      "mailboxesSet",
      {
         "accountId" : "11049eb6-1dcc-11e6-82c1-f23c91556942",
         "created" : {
            "foo" : {
               "id" : null
            }
         },
         "destroyed" : [],
         "newState" : null,
         "notCreated" : {},
         "notDestroyed" : {},
         "notUpdated" : {},
         "oldState" : null,
         "updated" : []
      },
      "a"
   ]
]

I'd expect some kind of error, not a "I created it, and its id is null".

If you try to sign up with gmail through the proxy UI and you have no config file (or it's incomplete) the errors the server gives in the logs aren't clear.

bin/server.pl needs MIME::Base64::URLSafe but it doesn't seem to get installed by the Makefile

I found two issues:

• Array references are not correctly dereferenced
• AND matching logic is wrong (it should return false is any condition returns false)

It would be great to have better input/output validation in a separate spec file for typed data so the code is simpler and we're sure each field is tested against a schema - both in what we read, but most importantly in what we produce, so we're always talking valid JMAP to clients.

It should be fetching from a wrapper that allows us to sub out sqlite with another DB engine more easily

Even if it succeed, mailbox creation returns an error like

 [["error",{"message":"Can't use string (\"20\") as a HASH ref while \"strict refs\" in use at /home/jmap/jmap-perl/JMAP/API.pm line 374.\n","type":"serverError"},"#0"]]

More a feature request than a bug: It would be great if you could add a Vagrant file // Dockerfile to get the proxy up and running locally if people - like me - just want to test it out.

It seems that the public proxy running at proxy.jmap.io returns these values in the Attachment object:

"isInline":"1"
"isInline":""

Neither of them is valid JSON Boolean value as per the JMAP and JSON specifications.

The last update is Jul 15, 2018:
https://github.com/jmapio/jmap-perl/commits/master

Spec has changed quite a bit since then:
https://github.com/jmapio/jmap/commits/master

I am hoping to use along with https://github.com/virtualmin/virtualmin-gpl (also in Perl) which uses Dovecot: https://doxfer.webmin.com/Webmin/Dovecot_IMAP/POP3_Server

Thanks!

Marc

Folder names with non-ascii characters are listed by the IMAP server with UTF-7 encoding. The JMAP spec says for the mailbox name property: "This may be any UTF-8 string of at least 1 character in length".

So converting UTF-7 folder names into UTF-8 is required here.

Example IMAP folder listing:

C: A0005 LIST (SUBSCRIBED) "" "*"
S: * LIST (\Noinferiors \Subscribed) "/" INBOX
S: * LIST (\Subscribed) "/" Archive
S: * LIST (\Subscribed) "/" Drafts
S: * LIST (\Subscribed) "/" Junk
S: * LIST (\Subscribed) "/" Sent
S: * LIST (\Subscribed) "/" Spam
S: * LIST (\Subscribed) "/" Test-&A5EDwAOsA70DxAO3A8MDtw-
S: * LIST (\Subscribed) "/" Trash
S: * LIST (\Subscribed) "/" "Shared Folders/shared/Project Planning"
S: * LIST (\Subscribed) "/" "Shared Folders/shared/testing"
S: A0005 OK Completed

JMAP response for getMailboxes:

[
    [
        "mailboxes",
        {
            "accountId": "ce69e3d4-2430-11e6-810b-0242ac120007",
            "list": [
                ...
                {
                    "id": "50",
                    "mayAddItems": true,
                    "mayCreateChild": true,
                    "mayDelete": true,
                    "mayReadItems": true,
                    "mayRemoveItems": true,
                    "mayRename": true,
                    "mustBeOnlyMailbox": true,
                    "name": "Test-&A5EDwAOsA70DxAO3A8MDtw-",
                    "parentId": null,
                    "role": null,
                    "sortOrder": 3,
                    "totalMessages": 0,
                    "totalThreads": 0,
                    "unreadMessages": 0,
                    "unreadThreads": 0
                },
                ...
            ],
            "notFound": null,
            "state": "24"
        },
        "#1"
    ]
]

When accessing messages with UTF-8 quoted-printable encoded headers, the values exposed in the JSON response by the JMAP proxy contain wrongly encoded unicode characters.

The following subject header in the original IMAP message

Subject: =?UTF-8?Q?Bern:=20Vari=C3=A9t=C3=A9,=20Juliette=20&=20Meeressause?=

results in a response

[
    "messages",
    {
        "accountId": "xxxxx",
        "list": [
            {
                "headers": {
                    "Subject": "Bern: Variété, Juliette & Meeressause",
                    ...
                },
                ...
                "subject": "Bern: Variété, Juliette & Meeressause",
                ...
            }
        ]
    },
    "#1"
]

It appears that the decoded header value is treated as iso-8859-1 encoded and is then converted to utf-8 resulting in a double encoding.

The sort criteria and filter criteria for messagelists are different for no good reason - they should have the same name.

If a mailbox is deleted at the remote (IMAP) end, IMAP syncing will fail in several ways, and no new syncs will occur. Here's the report I had originally given via private email, here for easy future reference:

I noted this error:

ERROR Not a HASH reference at /home/jmap/jmap-perl/JMAP/ImapDB.pm line 672.
on c4de019e-1c5e-11e6-b873-f23c91556942:sync (dropping backend)

It's one place where the problem can occur. Here's that line, reformatted:

  next if (
    $status->{$row->{imapname}}{uidvalidity} == $row->{uidvalidity}
    and $status->{$row->{imapname}}{highestmodseq}
    and $status->{$row->{imapname}}{highestmodseq} == $row->{highestmodseq}
  );

The undef being hash-dereferenced is $status->{$row->{imapname}}. We got the status for every folder, but the status entry for a deleted folder is \'no' and not a hash ref. (\$string)->{...} is fatal.

I started by just adding another "next"-ing condition, but this comes up elsewehere, too. Many of the imap_* methods in JMAP::Sync::Common rely on being able EXAMINE folders that may have been deleted.

This is what we do in Cyrus, and it makes things heaps faster when getting threads.

Just a reminder to either me or @brong to get it up there sometime. Makes installing the proxy rather difficult :)

I look at this page: https://proxy.jmap.io/

This is the first text I read:

The JMAP proxy is a work in a progress. It is currently stable enough to test out and get a feel for JMAP in action. All the methods in the spec are implemented, though some atomic guarantees are not possible with other users accessing your servers at the same time

Please start with an introduction. What is JMAP proxy all about? Then the details about the state (work in progress).

Related: https://github.com/guettli/programming-guidelines#love-your-docs

Instead, create an intermediate state with JUST the current changes, then sync later and bring things fully into sync. This means JMAP clients may see things out of order, but they're still going to be valid state transitions that bring you to the same point, and they mean that setFoos won't be as expensive.

According to the spec, both "from" and "to" are lists of objects:

  from: Emailer[]|null An array of name/email objects (see below) representing the parsed From header of the email, in the same order as they appear in the header. If the email  doesn’t have a From header, this is null. If the header exists but does not have any content, the response is an array of zero length.

  to: Emailer[]|null An array of name/email objects (see below) representing the parsed To header of the email, in the same order as they appear in the header. If the email doesn’t have a To header, this is null. If the header exists but does not have any content, the response is an array of zero length.

However, JMAP Proxy only accepts an object for "from" when you call setMessages:

([
  [
    setMessages => {
      create => {
        1 => {
          mailboxIds => [8], # outbox
          from => { email => "test\@example.com" },
          to => [ { email => "test2\@example.com" } ],
          subject => "Test mail",
          textBody => "Hello there!",
        },
      }
    },
  ],
]

@rjbs points out that the spec changed; at one point from used to be an object and not a list of objects:

jmapio/jmap@7b45eb3

I suspect JMAP Proxy was just never updated to account for this.

Thanks!

When I try send email from Drafts to Outbox:

[ [ "setMessages", { "update": { "md5a07f187":{ "mailboxIds":["5"] } } }, "#1" ] ]

, I get error:

[ [ "error", { "message": "isa check for \"helo\" failed: undef is not a string! at (eval 310) line 108.\n\teval {...} called at (eval 310) line 107\n\tEmail::Sender::Transport::SMTPS::new(\"Email::Sender::Transport::SMTPS\", HASH(0x2bee918)) called at /home/jmap/jmap-perl/JMAP/Sync/Standard.pm line 103\n\tJMAP::Sync::Standard::send_email(JMAP::Sync::Standard=HASH(0x66c9b00), \"From: bob\\@bicsh.com\\x{d}\\x{a}To: alice\\@bicsh.com\\x{d}\\x{a}Cc: \\x{d}\\x{a}Bcc: \\x{d}\\x{a}Subjec\"...) called at /home/jmap/jmap-perl/JMAP/ImapDB.pm line 135\n\tJMAP::ImapDB::backend_cmd(JMAP::ImapDB=HASH(0x5969338), \"send_email\", \"From: bob\\@bicsh.com\\x{d}\\x{a}To: alice\\@bicsh.com\\x{d}\\x{a}Cc: \\x{d}\\x{a}Bcc: \\x{d}\\x{a}Subjec\"...) called at /home/jmap/jmap-perl/JMAP/ImapDB.pm line 1109\n\tJMAP::ImapDB::update_messages(JMAP::ImapDB=HASH(0x5969338), HASH(0x23b21c8), CODE(0x66f0160)) called at /home/jmap/jmap-perl/JMAP/API.pm line 1178\n\tJMAP::API::setMessages(JMAP::API=HASH(0x66c9a28), HASH(0x23b21f8), \"#1\") called at /home/jmap/jmap-perl/bin/apiendpoint.pl line 601\n\teval {...} called at /home/jmap/jmap-perl/bin/apiendpoint.pl line 601\n\tJMAP::Backend::handle_jmap(JMAP::ImapDB=HASH(0x5969338), ARRAY(0x23be420), \"#0\") called at /home/jmap/jmap-perl/bin/apiendpoint.pl line 201\n\teval {...} called at /home/jmap/jmap-perl/bin/apiendpoint.pl line 187\n\tJMAP::Backend::__ANON__(AnyEvent::Handle=HASH(0x6489048), ARRAY(0x23eb1f0)) called at /usr/lib/x86_64-linux-gnu/perl5/5.20/AnyEvent/Handle.pm line 1735\n\tAnyEvent::Handle::__ANON__(AnyEvent::Handle=HASH(0x6489048)) called at /usr/lib/x86_64-linux-gnu/perl5/5.20/AnyEvent/Handle.pm line 1310\n\tAnyEvent::Handle::_drain_rbuf(AnyEvent::Handle=HASH(0x6489048)) called at /usr/lib/x86_64-linux-gnu/perl5/5.20/AnyEvent/Handle.pm line 2009\n\tAnyEvent::Handle::__ANON__(EV::IO=SCALAR(0x23b2168), 1) called at /home/jmap/jmap-perl/bin/apiendpoint.pl line 120\n\teval {...} called at /home/jmap/jmap-perl/bin/apiendpoint.pl line 120\n\tJMAP::Backend::process_request(JMAP::Backend=HASH(0x1d76a60), Net::Server::Proto::TCP=GLOB(0x63d0060)) called at /usr/local/share/perl/5.20.2/Net/Server.pm line 74\n\tNet::Server::run_client_connection(JMAP::Backend=HASH(0x1d76a60)) called at /usr/local/share/perl/5.20.2/Net/Server/Fork.pm line 199\n\tNet::Server::Fork::run_client_connection(JMAP::Backend=HASH(0x1d76a60)) called at /usr/local/share/perl/5.20.2/Net/Server/Fork.pm line 144\n\tNet::Server::Fork::loop(JMAP::Backend=HASH(0x1d76a60)) called at /usr/local/share/perl/5.20.2/Net/Server.pm line 60\n\tNet::Server::run(\"JMAP::Backend\", \"host\", \"127.0.0.1\", \"port\", 5000) called at /home/jmap/jmap-perl/bin/apiendpoint.pl line 124\n", "type": "serverError" }, "#1" ] ]

When I connect to this SMTP server with different client, it works. Can somebody help me, where can be problem.

My "A & B" folder isn't coming through correctly

Not only can we split locking so that calendar / contact fetches don't lock mail and vice versa, but we can track deletedModseq separately so we don't invalidate as much on clients.

The README says:

NOTE: this library does not ship with the cloned FastMail web interface which
is running on https://proxy.jmap.io/.  The licensing situation for that code is
not yet resolved.

Is that what https://github.com/jmapio/jmap-demo-webmail is? It's README says:

There's a hosted version at http://proxy.jmap.io, which uses the JMAP proxy to provide a JMAP 
interface to a Gmail or IMAP account.