nylas / sync-engine Goto Github PK

:incoming_envelope: IMAP/SMTP sync system with modern APIs

Home Page: https://nylas.com/docs/platform

License: GNU Affero General Public License v3.0

Ruby 0.03% PHP 0.44% Python 99.09% Shell 0.42% Mako 0.02%

sync-engine's Introduction

Nylas Sync Engine

The Nylas Sync Engine provides a RESTful API on top of a powerful email sync platform, making it easy to build apps on top of email. See the full API documentation for more details.

Need help? Join our Slack channel Slack Invite Button

Installation and Setup

Install the latest versions of VirtualBox and Install Vagrant.
git clone https://github.com/nylas/sync-engine.git
cd sync-engine
vagrant up

Feel free to check out the Vagrantfile while this starts up. It creates a host-only network for the VM at 192.168.10.200.
vagrant ssh
cd /vagrant
NYLAS_ENV=dev bin/inbox-start

And voilà! Auth an account via the commandline to start syncing:

bin/inbox-auth [email protected]

The inbox-auth command will walk you through the process of obtaining an authorization token from Google or another service for syncing your mail. In the open-source version of the sync engine, your credentials are stored to the local MySQL database for simplicity. The open-source Nylas Sync Engine does not support Exchange, but the hosted version does.

The sync engine will automatically begin syncing your account with the underlying provider. The inbox-sync command allows you to manually stop or restart the sync by running inbox-sync stop [YOUR_ACCOUNT]@example.com or inbox-sync start [YOUR_ACCOUNT]@example.com. Note that an initial sync can take quite a while depending on how much mail you have.

Nylas API Service

The Nylas API service provides a REST API for interacting with your data. To start it in your development environment, run command below from the /vagrant folder within your VM:

$ bin/inbox-api

This will start the API Server on port 5555. At this point You're now ready to make requests! If you're using VirtualBox or VMWare fusion with Vagrant, port 5555 has already been forwarded to your host machine, so you can hit the API from your regular web browser.

You can get a list of all connected accounts by requesting http://localhost:5555/accounts. This endpoint requires no authentication.

For subsequent requests to retreive mail, contacts, and calendar data, your app should pass the account_id value from the previous step as the "username" parameter in HTTP Basic auth. For example:

curl --user 'ACCOUNT_ID_VALUE_HERE:' http://localhost:5555/threads

If you are using a web browser and would like to clear your cached HTTP Basic Auth values, simply visit http://localhost:5555/logout and click "Cancel".

Now you can start writing your own application on top of the Nylas API! For more information about the internals of the Nylas Sync Engine, see the Nylas API Documentation.

Production Support

We provide a fully managed and supported version of the Nylas sync engine for production apps. Read more at https://nylas.com

Pull Requests

We'd love your help making Nylas better! Please sign-up for a developer account for project updates and the latest news. Feel free to create issues or pull requests to start discussions.

We require all authors sign our Contributor License Agreement when submitting pull requests. (It's similar to other projects, like NodeJS or Meteor.)

Security

For the sake of simplicity and setup speed, the development VM does not include any authentication or permission. For developing with sensitive data, we encourage developers to add their own protection, such as only running Nylas on a local machine or behind a controlled firewall. Note that passwords and OAuth tokens are stored unencrypted in the local MySQL data store on disk. This is intentional, for the same reason as above.

License

This code is free software, licensed under the The GNU Affero General Public License (AGPL). See the LICENSE file for more details.

sync-engine's People

Contributors

Stargazers

Watchers

Forkers

nvasilakis cenkalti nvdnkpr caitp christienlomax prodigeni bictorman yuklai metaware jayakumark elevateops tchen0123 jeffbaumes trestletech rosco5 jollen iusr bsorin martin-martinez manithnuon ivanfoong amyweiner waytai afthill hanxue jaishankarm sashka ericol dougpuchalski rmasters cosmocommerce tanwanirahul prabhuvishnumurthy loftindenny agamdua abhishekgahlot utkarshx anilkabobo nodraak kimroro systemizer chris0422 rianspeed paulreiber sunnycmf fijiwebdesign rmoorman bekatom thatchristoph ghotiv inanny chengjunjian rayleyva ictdcapilot marius-t okaiji michaeldarius jayceecam 0xcd03 ggnall admix nixon1333 calinrada jjayyoung lucidntr mediapreneur siwuxie mishravikas alexchan dlitz mdeng1024 xarg frictionlesscoin alihalabyah tatoalo octopusinfo abusimbele rmcnairn frankcarey gsh45 wmv dhootha cyberatz partlab robertodijo dracophoenix1 tianboyand neuroradiology itspradhan roryosiochain iamalexander mobileworks rht klaudit jprice athiyaman vjeantet geverding aidanhs igs013

sync-engine's Issues

Namespace object typo

When listing authenticated namespaces with http://localhost:5555/n/ the account objects returned contain a misspelled object type "namepace".
{ "object": "namepace", "provider": "gmail" }

Unexpected query parameters order_by=rank

The order_by=rank filter seems be not supported in the last master release:

http://localhost:5555/n/<namespace>/contacts?order_by=rank
returns:

{
"message": "Unexpected query parameters set(['order_by'])",
"type": "api_error"
}

Admin Dashboard - Sync Error

State (running/ stopped/ killed): killed

Sync Error:

Traceback (most recent call last): File "/vagrant/inbox/util/concurrency.py", line 69, in wrapped return func(*args, **kwargs) File "/vagrant/inbox/mailsync/backends/imap/imap.py", line 258, in _run_impl self.folder_name, self.shared_state) File "/vagrant/inbox/util/concurrency.py", line 69, in wrapped return func(*args, **kwargs) File "/vagrant/inbox/mailsync/backends/gmail.py", line 89, in initial_sync create_gmail_message) File "/vagrant/inbox/mailsync/backends/imap/imap.py", line 321, in base_initial_sync msg_create_fn) File "/vagrant/inbox/mailsync/backends/gmail.py", line 167, in gmail_initial_sync gmail_download_and_commit_uids, msg_create_fn) File "/vagrant/inbox/mailsync/backends/imap/imap.py", line 724, in download_queued_uids msg_create_fn, syncmanager_lock) File "/vagrant/inbox/mailsync/backends/gmail.py", line 264, in gmail_download_and_commit_uids msg_create_fn) File "/vagrant/inbox/mailsync/backends/base.py", line 120, in create_db_objects uid = msg_create_fn(db_session, log, acc, folder, msg) File "/vagrant/inbox/mailsync/backends/gmail.py", line 660, in create_gmail_message update_contacts(db_session, acct.id, new_uid.message) File "/vagrant/inbox/contacts/process_mail.py", line 25, in update_contacts contacts = get_contact_objects(db_session, account_id, addresses) File "/vagrant/inbox/contacts/process_mail.py", line 58, in get_contact_objects Contact.account_id == account_id).all() File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/query.py", line 2293, in all return list(self) File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/query.py", line 2404, in __iter__ self.session._autoflush() File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/session.py", line 1198, in _autoflush util.raise_from_cause(e) File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/util/compat.py", line 189, in raise_from_cause reraise(type(exception), exception, tb=exc_tb) File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/session.py", line 1188, in _autoflush self.flush() File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/session.py", line 1907, in flush self._flush(objects) File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/session.py", line 2025, in _flush transaction.rollback(_capture_exception=True) File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/util/langhelpers.py", line 57, in __exit__ compat.reraise(exc_type, exc_value, exc_tb) File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/session.py", line 1989, in _flush flush_context.execute() File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/unitofwork.py", line 371, in execute rec.execute(self) File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/unitofwork.py", line 524, in execute uow File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/persistence.py", line 64, in save_obj mapper, table, insert) File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/persistence.py", line 600, in _emit_insert_statements execute(statement, params) File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 727, in execute return meth(self, multiparams, params) File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/sql/elements.py", line 322, in _execute_on_connection return connection._execute_clauseelement(self, multiparams, params) File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 824, in _execute_clauseelement compiled_sql, distilled_params File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 954, in _execute_context context) File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 1116, in _handle_dbapi_exception exc_info File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/util/compat.py", line 189, in raise_from_cause reraise(type(exception), exception, tb=exc_tb) File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 947, in _execute_context context) File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/engine/default.py", line 435, in do_execute cursor.execute(statement, parameters) File "/usr/local/lib/python2.7/dist-packages/pymysql/cursors.py", line 132, in execute result = self._query(query) File "/usr/local/lib/python2.7/dist-packages/pymysql/cursors.py", line 271, in _query conn.query(q) File "/usr/local/lib/python2.7/dist-packages/pymysql/connections.py", line 726, in query self._affected_rows = self._read_query_result(unbuffered=unbuffered) File "/usr/local/lib/python2.7/dist-packages/pymysql/connections.py", line 861, in _read_query_result result.read() File "/usr/local/lib/python2.7/dist-packages/pymysql/connections.py", line 1064, in read first_packet = self.connection._read_packet() File "/usr/local/lib/python2.7/dist-packages/pymysql/connections.py", line 826, in _read_packet packet.check_error() File "/usr/local/lib/python2.7/dist-packages/pymysql/connections.py", line 370, in check_error raise_mysql_exception(self._data) File "/usr/local/lib/python2.7/dist-packages/pymysql/err.py", line 116, in raise_mysql_exception _check_mysql_exception(errinfo) File "/usr/local/lib/python2.7/dist-packages/pymysql/err.py", line 109, in _check_mysql_exception raise errorclass(errno,errorvalue) IntegrityError: (raised as a result of Query-invoked autoflush; consider using a session.no_autoflush block if this flush is occurring prematurely) (IntegrityError) (1452, u'Cannot add or update a child row: a foreign key constraint fails (`inbox`.`folderitem`, CONSTRAINT `folderitem_ibfk_2` FOREIGN KEY (`folder_id`) REFERENCES `folder` (`id`) ON DELETE CASCADE)') 'INSERT INTO folderitem (created_at, updated_at, deleted_at, thread_id, folder_id) VALUES (%s, %s, %s, %s, %s)' (datetime.datetime(2014, 7, 22, 2, 9, 31, 477284), datetime.datetime(2014, 7, 22, 2, 9, 31, 477293), None, 13165, 80)

Vagrant can't forward port

Hi!

I don't know if this is the best place for this, but I'll try. I installed VirtualBox and Vagrant on my 10.7.4 Mac. I downloaded this repo and executed vagrant up, but then I got this error:

$ vagrant up
Bringing machine 'default' up with 'virtualbox' provider...
Vagrant cannot forward the specified ports on this VM, since they
would collide with some other application that is already listening
on these ports. The forwarded port to 9200 is already in use
on the host machine.

To fix this, modify your current projects Vagrantfile to use another
port. Example, where '1234' would be replaced by a unique host port:

  config.vm.network :forwarded_port, guest: 9200, host: 1234

Sometimes, Vagrant will attempt to auto-correct this for you. In this
case, Vagrant was unable to. This is usually because the guest machine
is in a state which doesn't allow modifying port forwarding.

I don't know which process is using the port 9200, but this makes vagrant up fail and thus can't execute vagrant ssh. Any ideas?

Thank you! :)

Running ./setup.sh fails on Digital Ocean

As per Issue #5 I tried simply running $ sudo ./setup.sh on my DO instance. It installed a bunch of dependencies, but then error'd out with the following:

Successfully installed inbox
Cleaning up...
[InboxApp] Finished installing dependencies.
140401 11:20:53 mysqld_safe Can't log to error log and syslog at the same time.  Remove all --log-error configuration options for --syslog to take effect.
140401 11:20:53 mysqld_safe Logging to '/var/log/mysql/error.log'.
140401 11:20:53 mysqld_safe A mysqld process already exists
Creating database: test
/usr/local/lib/python2.7/dist-packages/sqlalchemy/engine/default.py:324: Warning: Can't create database 'test'; database exists
  cursor.execute(statement, parameters)
Creating database: inbox
/usr/local/lib/python2.7/dist-packages/sqlalchemy/engine/default.py:324: Warning: Can't create database 'inbox'; database exists
  cursor.execute(statement, parameters)
Stamping with alembic revision
Traceback (most recent call last):
  File "tools/create_db.py", line 33, in <module>
    alembic_command.stamp(alembic_cfg, "head")
  File "/usr/local/lib/python2.7/dist-packages/alembic/command.py", line 257, in stamp
    script.run_env()
  File "/usr/local/lib/python2.7/dist-packages/alembic/script.py", line 199, in run_env
    util.load_python_file(self.dir, 'env.py')
  File "/usr/local/lib/python2.7/dist-packages/alembic/util.py", line 205, in load_python_file
    module = load_module_py(module_id, path)
  File "/usr/local/lib/python2.7/dist-packages/alembic/compat.py", line 58, in load_module_py
    mod = imp.load_source(module_id, path, fp)
  File "migrations/env.py", line 7, in <module>
    from inbox.server.config import load_config
ImportError: No module named server.config
bnvk@inbox:~/inbox$ ./inbox debug
Removing pyc files...
removing ./src/inbox-util/inbox/__init__.pyc
Traceback (most recent call last):
  File "./inbox", line 22, in <module>
    os.remove(full_path)
OSError: [Errno 13] Permission denied: './src/inbox-util/inbox/__init__.pyc'

Webhook does not seem to call the callback url

Received email shows up when i list threads/messages, but callback url specified in the webhook does not seem to be called. This is what my webhook looks like (I'm using Parse).

[
    {
        "active": true,
        "any_email": null,
        "bcc": null,
        "callback_url": "https://somethingsomething.parseapp.com/notify_email",
        "cc": null,
        "failure_notify_url": "https://somethingsomething.parseapp.com/notify_failure",
        "filename": null,
        "from": null,
        "id": "3tbsjnlqj7kclt1jrtq7w43pk",
        "include_body": true,
        "last_message_after": null,
        "last_message_before": null,
        "namespace": "yv2pkn5fasxiiuucvmbrugqu",
        "object": "webhook",
        "started_after": null,
        "started_before": null,
        "subject": null,
        "thread": null,
        "to": null
    }
]

NoResultFound: No row was found for one()

{"exception": "Traceback (most recent call last):\n File "/vagrant/inbox/util/concurrency.py", line 69, in wrapped\n return func(_args, *_kwargs)\n File "/vagrant/inbox/mailsync/backends/imap/imap.py", line 235, in _run_impl\n self.folder_name, self.shared_state)\n File "/vagrant/inbox/util/concurrency.py", line 69, in wrapped\n return func(_args, *_kwargs)\n File "/vagrant/inbox/mailsync/backends/gmail.py", line 84, in initial_sync\n create_gmail_message)\n File "/vagrant/inbox/mailsync/backends/imap/imap.py", line 298, in base_initial_sync\n msg_create_fn)\n File "/vagrant/inbox/mailsync/backends/gmail.py", line 147, in gmail_initial_sync\n shared_state['syncmanager_lock'])\n File "/vagrant/inbox/mailsync/backends/gmail.py", line 400, in download_queued_threads\n add_new_imapuid(db_session, log, msg, folder_name, acc)\n File "/vagrant/inbox/mailsync/backends/gmail.py", line 496, in add_new_imapuid\n g_msgid=gmessage.g_metadata.msgid).one()\n File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/query.py", line 2368, in one\n raise orm_exc.NoResultFound("No row was found for one()")\nNoResultFound: No row was found for one()", "account_id": 1, "level": "error", "timestamp": "2014-07-17T20:32:52.247532Z", "module": "inbox.util.concurrency:96", "folder": "Inbox", "event": "Uncaught error"}
Traceback (most recent call last):
File "/usr/local/lib/python2.7/dist-packages/gevent/greenlet.py", line 327, in run
result = self._run(_self.args, *_self.kwargs)
File "/vagrant/inbox/mailsync/backends/imap/imap.py", line 203, in _run
fail_classes=self.retry_fail_classes)
File "/vagrant/inbox/util/concurrency.py", line 100, in retry_and_report_killed
fail_classes=fail_classes)()
File "/vagrant/inbox/util/concurrency.py", line 69, in wrapped
return func(_args, *_kwargs)
File "/vagrant/inbox/mailsync/backends/imap/imap.py", line 235, in _run_impl
self.folder_name, self.shared_state)
File "/vagrant/inbox/util/concurrency.py", line 69, in wrapped
return func(_args, *_kwargs)
File "/vagrant/inbox/mailsync/backends/gmail.py", line 84, in initial_sync
create_gmail_message)
File "/vagrant/inbox/mailsync/backends/imap/imap.py", line 298, in base_initial_sync
msg_create_fn)
File "/vagrant/inbox/mailsync/backends/gmail.py", line 147, in gmail_initial_sync
shared_state['syncmanager_lock'])
File "/vagrant/inbox/mailsync/backends/gmail.py", line 400, in download_queued_threads
add_new_imapuid(db_session, log, msg, folder_name, acc)
File "/vagrant/inbox/mailsync/backends/gmail.py", line 496, in add_new_imapuid
g_msgid=gmessage.g_metadata.msgid).one()
File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/query.py", line 2368, in one
raise orm_exc.NoResultFound("No row was found for one()")
NoResultFound: No row was found for one()
<ImapFolderSyncMonitor at 0x49eec30> failed with NoResultFound

DecodeError

[ERROR] {"timestamp": "2014-07-17T20:32:46.892964Z", "event": "DecodeError, msg logged to /var/log/inboxapp/1/errors/[Gmail]/All Mail/118082", "module": "inbox.models.message:161", "level": "error"}

The 'vagrant up' fails on Digital Ocean

Howdi,

Just tried spinning up an install of Inbox on a fresh Digital Ocean account. After installing Vagrant via apt-get and cloning the repo, upon typing $ vagrant up I am presented with the following error:

bnvk@inbox:~/inbox$ vagrant up
There is a syntax error in the following Vagrantfile. The syntax error
message is reproduced below for convenience:

/home/bnvk/inbox/Vagrantfile:24: syntax error, unexpected ':', expecting kEND
  config.vm.network "private_network", ip: "192.168.10.200"

Perhaps it is just my lack of familiarity with vagrant (which is next to none), is there some other step required?

RUD operations for Tags

For consistency with other models, I think it would be good to implement RUD operations here.

Retrieving a singular tag model may be wasteful/unnecessary, but it allows a clientside API to have fewer special cases where unexpected things happen.

Updating tag names may be desirable, for instance suppose I want to rename "Coworkers" to "Cow-orkers", or some similar silliness. These could be changed client-side, but usually you expect these changes to propagate to all of your clients.

Deletion is iffy because most things in this service appear to be immutable, but suppose you wanted to "deactivate" a tag, so that it didn't appear in the returned set of tags when querying /n/<namespace_id>/tags --- This could be useful for user interfaces which want to render the set of tags, for whatever reason (tag cloud? menu of tag names?)

Anyways, the primary reason for wanting this is for consistency, so that there are fewer special cases in the client APIs

Cannot send a draft

I cannot send an email using the iOS SDK. The way the SDK works is that it always creates a draft and sends it. The responses look good to me, but the email is not delivered.

PS. I've checked my email and I can see the draft in my Drafts folder, but there is nothing in my sent folder.

PPS. I also tried to send an email (not a draft) right after I deployed Inboxapp to my VM and the email was delivered. I've tried the same after it's been running for a couple of hours and didn't go through this time. (Inbox sync seems to be running though)

Create a draft

Request

/n/7u88e1hxy3idi2cb9p8tbqp4o/drafts

x-request-start: 1406288519326
x-request-id: 72db6735-6065-4817-a409-a2553de8414d
user-agent: MailClient/0.1 (iPhone Simulator; iOS 8.0; Scale/2.00)
host: posty.apiary-proxy.com
content-type: application/json; charset=utf-8
content-length: 487
connection: close
authorization: Basic bm8tb3Blbi1zb3VyY2UtYXV0aDo=
accept-language: en;q=1, fr;q=0.9, de;q=0.8, zh-Hans;q=0.7, zh-Hant;q=0.6, ja;q=0.5
accept-encoding: gzip, deflate
accept: */*

{
  "subject" : "New message composed",
  "body" : "2014-07-25 11:41:57 +0000",
  "thread" : null,
  "state" : null,
  "created_at" : null,
  "namespace" : "7u88e1hxy3idi2cb9p8tbqp4o",
  "from" : [
    {
      "email" : "[email protected]",
      "name" : "[email protected]"
    }
  ],
  "files" : null,
  "date" : 1406288517.88533,
  "updated_at" : null,
  "unread" : false,
  "to" : [
    {
      "name" : "Petr Pavlik",
      "email" : "[email protected]"
    }
  ]
}

Response

content-type: application/json
content-length: 583
date: Fri, 25 Jul 2014 11:41:59 GMT
x-apiary-ratelimit-limit: 120
x-apiary-ratelimit-remaining: 119
x-apiary-transaction-id: 53d2428709d8ee0200001455

{
    "bcc": [],
    "body": "2014-07-25 11:41:57 +0000",
    "cc": [],
    "date": 1406288519,
    "files": [],
    "from": [
        {
            "email": "[email protected]",
            "name": "Inbox App"
        }
    ],
    "id": "3qvoezxh6s43tohstjhvs9ijx",
    "namespace": "7u88e1hxy3idi2cb9p8tbqp4o",
    "object": "draft",
    "state": "draft",
    "subject": "New message composed",
    "thread": "d381m8xwyazzrrsx0qlpm0vi",
    "to": [
        {
            "email": "[email protected]",
            "name": "Petr Pavlik"
        }
    ],
    "unread": false
}

Send the draft

Request

/n/7u88e1hxy3idi2cb9p8tbqp4o/send

x-request-start: 1406288519639
x-request-id: e567b5b7-33f9-4ee2-a4cf-063d7058727d
user-agent: MailClient/0.1 (iPhone Simulator; iOS 8.0; Scale/2.00)
host: posty.apiary-proxy.com
content-type: application/json; charset=utf-8
content-length: 46
connection: close
authorization: Basic bm8tb3Blbi1zb3VyY2UtYXV0aDo=
accept-language: en;q=1, fr;q=0.9, de;q=0.8, zh-Hans;q=0.7, zh-Hant;q=0.6, ja;q=0.5
accept-encoding: gzip, deflate
accept: */*

{
  "draft_id" : "3qvoezxh6s43tohstjhvs9ijx"
}

Response

content-type: application/json
content-length: 585
date: Fri, 25 Jul 2014 11:42:00 GMT
x-apiary-ratelimit-limit: 120
x-apiary-ratelimit-remaining: 118
x-apiary-transaction-id: 53d242879662420200001428

{
    "bcc": [],
    "body": "2014-07-25 11:41:57 +0000",
    "cc": [],
    "date": 1406288519,
    "files": [],
    "from": [
        {
            "email": "[email protected]",
            "name": "Inbox App"
        }
    ],
    "id": "3qvoezxh6s43tohstjhvs9ijx",
    "namespace": "7u88e1hxy3idi2cb9p8tbqp4o",
    "object": "draft",
    "state": "sending",
    "subject": "New message composed",
    "thread": "d381m8xwyazzrrsx0qlpm0vi",
    "to": [
        {
            "email": "[email protected]",
            "name": "Petr Pavlik"
        }
    ],
    "unread": false
}

inbox-sync error on slow query

I'm just starting to sync my gmail account with Inbox and keep on seeing an error whenever a message takes too long to download. I'm not sure if this is the expected behavior but I thought I'd share it anyways.

inbox.general        [W 2014-07-08 14:19:06,693 util    :44  ] Slow query took 265.69ms: 
 SELECT message.g_msgid AS message_g_msgid FROM message INNER JOIN imapuid ON message.id = imapuid.message_id WHERE imapuid.account_id = %s  
Params:  (1,)  
Trace:    File "/vagrant/inbox/mailsync/backends/imap/imap.py", line 200, in _run
    folder_name=self.folder_name)
  File "/vagrant/inbox/util/concurrency.py", line 97, in retry_and_report_killed
    fail_callback=fail_callback)()
  File "/vagrant/inbox/util/concurrency.py", line 70, in wrapped
    return func(*args, **kwargs)
  File "/vagrant/inbox/mailsync/backends/imap/imap.py", line 232, in _run_impl
    self.folder_name, self.shared_state)
  File "/vagrant/inbox/util/concurrency.py", line 70, in wrapped
    return func(*args, **kwargs)
  File "/vagrant/inbox/mailsync/backends/gmail.py", line 82, in initial_sync
    create_gmail_message)
  File "/vagrant/inbox/mailsync/backends/imap/imap.py", line 296, in base_initial_sync
    msg_create_fn)
  File "/vagrant/inbox/mailsync/backends/gmail.py", line 146, in gmail_initial_sync
    shared_state['syncmanager_lock'])
  File "/vagrant/inbox/mailsync/backends/gmail.py", line 393, in download_queued_threads
    message.g_metadata.thrid, thread_uids)
  File "/vagrant/inbox/mailsync/backends/gmail.py", line 429, in download_thread
    thread_g_metadata, thread_uids)
  File "/vagrant/inbox/mailsync/backends/gmail.py", line 463, in deduplicate_message_download
    uid in uids if uid in
  File "/vagrant/inbox/mailsync/backends/imap/account.py", line 60, in g_msgids
    .filter(ImapUid.account_id == account_id).all()

Ruby Gem

Would you guys be interested in someone helping to develop a Ruby Gem? I've suffered from the pains of email for quite some time, so this project has really grabbed my attention.

I'd be happy to get the ball rolling so this could easily be pulled into something like a Rails app.

Less coarse-grained threading information?

Right now, the Inbox API for messages / threads only returns which thread a message belongs to, and not the actual relationship between messages. Some clients (gnus, cough) might want to expose the thread's underlying tree structure.

It would be really useful if inbox sync could use jwz's threading algorithm to figure out which message is a reply to which other message.

Snippet property of a thread contains a subject of a message when using a Gmail account

I haven't tried to this with other provider except for Gmail, but snippet property of each thread is prefixed with a subject of an email.

"snippet": "Stay more organized with Gmails inbox Hi InboxGmails inbox puts you in control Meet the inbox Gmail's inbox sorts your email into categories so you can see what's new at a glance, decide whic",
"subject": "Stay more organized with Gmail's inbox",

OAuth Error: invalid_client

[ERROR] {"exception": "Traceback (most recent call last):\n File "/vagrant/inbox/util/concurrency.py", line 69, in wrapped\n return func(_args, *_kwargs)\n File "/vagrant/inbox/mailsync/backends/imap/imap.py", line 145, in sync\n with connection_pool(self.account_id).get() as crispin_client:\n File "/vagrant/inbox/crispin.py", line 62, in connection_pool\n readonly=True)\n File "/vagrant/inbox/crispin.py", line 107, in init\n self._set_account_info()\n File "/vagrant/inbox/crispin.py", line 119, in _set_account_info\n self.access_token = account.access_token\n File "/vagrant/inbox/models/backends/gmail.py", line 60, in access_token\n tok, expires = new_token(self.refresh_token)\n File "/vagrant/inbox/oauth.py", line 88, in new_token\n raise OAuthError(session_dict['error'])\nOAuthError: invalid_client", "account_id": 1, "level": "error", "timestamp": "2014-07-17T20:32:00.346783Z", "component": "mail sync", "module": "inbox.util.concurrency:96", "event": "Uncaught error"}

Multi database type (not only MySQL)

This using sqlalchemy why not let flexible for other database (for example PostgreSQL)?

https://github.com/inboxapp/inbox/blob/9930020d90d491ee62f0dc5ad6fd51604c5db191/inbox/config.py#L36:L42

Feature - Account Sync Progress and Status

It would be nice to have additional information about synchronization process and account status. Example:

[
    {
        "account": "bticijruzg2f7z0ornnzcrfe5",
        "email_address": "[email protected]",
        "id": "6ka52om2ene9mclduf1d1b4jn",
        "namespace": "6ka55pm3enw8mclduf1d1b4jn",
        "object": "namepace",
        "provider": "gmail",
        "messages": 12800,
        "folders": [
            {
                "Inbox": {
                    "total_messages": 34,
                    "synced_messages": 34,
                    "state": "running"
                }
            },
            {
                "[Gmail]/Spam": {
                    "total_messages": 203,
                    "synced_messages": 203,
                    "state": "running"
                }
            },
            {
                "[Gmail]/Trash": {
                    "total_messages": 3,
                    "synced_messages": 3,
                    "state": "running"
                }
            },
            {
                "[Gmail]/All Mail": {
                    "total_messages": 12560,
                    "synced_messages": 5097,
                    "state": "running"
                }
            }
        ]
    }
]

Dev branch vagrant up fails

I just checked out the dev branch to try the fix from earlier and vagrant up has problems.

vagrant up
Bringing machine 'default' up with 'virtualbox' provider...
==> default: Resuming suspended VM...
==> default: Booting VM...
==> default: Waiting for machine to boot. This may take a few minutes...
    default: SSH address: 127.0.0.1:2222
    default: SSH username: vagrant
    default: SSH auth method: private key
    default: Warning: Connection refused. Retrying...
    default: Warning: Connection timeout. Retrying...
    default: Warning: Connection timeout. Retrying...
    default: Warning: Connection timeout. Retrying...
    default: Warning: Connection timeout. Retrying...
    default: Warning: Connection timeout. Retrying...
    default: Warning: Connection timeout. Retrying...
    default: Warning: Connection timeout. Retrying...
    default: Warning: Connection timeout. Retrying...
    default: Warning: Connection timeout. Retrying...
    default: Warning: Connection timeout. Retrying...
    default: Warning: Connection timeout. Retrying...
    default: Warning: Connection timeout. Retrying...
    default: Warning: Connection timeout. Retrying...
    default: Warning: Connection timeout. Retrying...

Migration problem

Migration failed

vagrant@precise64:/vagrant$ git branch -v
  dev    610aa50 Set db pool size in config, and remove unused config values.
* master f327b12 Merge pull request #47 from lheyberger/master

alembic upgrade head

INFO  [alembic.migration] Context impl MySQLImpl.
INFO  [alembic.migration] Will assume non-transactional DDL.
INFO  [alembic.migration] Running upgrade 29217fad3f46 -> 1925c535a52d, store secrets in local vault
Traceback (most recent call last):
  File "/usr/local/bin/alembic", line 9, in <module>
    load_entry_point('alembic==0.6.4', 'console_scripts', 'alembic')()
  File "/usr/local/lib/python2.7/dist-packages/alembic/config.py", line 298, in main
    CommandLine(prog=prog).main(argv=argv)
  File "/usr/local/lib/python2.7/dist-packages/alembic/config.py", line 293, in main
    self.run_cmd(cfg, options)
  File "/usr/local/lib/python2.7/dist-packages/alembic/config.py", line 279, in run_cmd
    **dict((k, getattr(options, k)) for k in kwarg)
  File "/usr/local/lib/python2.7/dist-packages/alembic/command.py", line 125, in upgrade
    script.run_env()
  File "/usr/local/lib/python2.7/dist-packages/alembic/script.py", line 203, in run_env
    util.load_python_file(self.dir, 'env.py')
  File "/usr/local/lib/python2.7/dist-packages/alembic/util.py", line 212, in load_python_file
    module = load_module_py(module_id, path)
  File "/usr/local/lib/python2.7/dist-packages/alembic/compat.py", line 58, in load_module_py
    mod = imp.load_source(module_id, path, fp)
  File "migrations/env.py", line 81, in <module>
    run_migrations_online()
  File "migrations/env.py", line 74, in run_migrations_online
    context.run_migrations()
  File "<string>", line 7, in run_migrations
  File "/usr/local/lib/python2.7/dist-packages/alembic/environment.py", line 681, in run_migrations
    self.get_context().run_migrations(**kw)
  File "/usr/local/lib/python2.7/dist-packages/alembic/migration.py", line 225, in run_migrations
    change(**kw)
  File "migrations/versions/051_store_secrets_in_local_vault.py", line 31, in upgrade
    sa.PrimaryKeyConstraint('id'))
  File "<string>", line 7, in create_table
  File "/usr/local/lib/python2.7/dist-packages/alembic/operations.py", line 713, in create_table
    self._table(name, *columns, **kw)
  File "/usr/local/lib/python2.7/dist-packages/alembic/ddl/impl.py", line 149, in create_table
    self._exec(schema.CreateTable(table))
  File "/usr/local/lib/python2.7/dist-packages/alembic/ddl/impl.py", line 76, in _exec
    conn.execute(construct, *multiparams, **params)
  File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 727, in execute
    return meth(self, multiparams, params)
  File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/sql/ddl.py", line 67, in _execute_on_connection
    return connection._execute_ddl(self, multiparams, params)
  File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 781, in _execute_ddl
    compiled
  File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 954, in _execute_context
    context)
  File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 1116, in _handle_dbapi_exception
    exc_info
  File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/util/compat.py", line 189, in raise_from_cause
    reraise(type(exception), exception, tb=exc_tb)
  File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 947, in _execute_context
    context)
  File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/engine/default.py", line 435, in do_execute
    cursor.execute(statement, parameters)
  File "/usr/local/lib/python2.7/dist-packages/pymysql/cursors.py", line 132, in execute
    result = self._query(query)
  File "/usr/local/lib/python2.7/dist-packages/pymysql/cursors.py", line 271, in _query
    conn.query(q)
  File "/usr/local/lib/python2.7/dist-packages/pymysql/connections.py", line 726, in query
    self._affected_rows = self._read_query_result(unbuffered=unbuffered)
  File "/usr/local/lib/python2.7/dist-packages/pymysql/connections.py", line 861, in _read_query_result
    result.read()
  File "/usr/local/lib/python2.7/dist-packages/pymysql/connections.py", line 1064, in read
    first_packet = self.connection._read_packet()
  File "/usr/local/lib/python2.7/dist-packages/pymysql/connections.py", line 826, in _read_packet
    packet.check_error()
  File "/usr/local/lib/python2.7/dist-packages/pymysql/connections.py", line 370, in check_error
    raise_mysql_exception(self._data)
  File "/usr/local/lib/python2.7/dist-packages/pymysql/err.py", line 116, in raise_mysql_exception
    _check_mysql_exception(errinfo)
  File "/usr/local/lib/python2.7/dist-packages/pymysql/err.py", line 112, in _check_mysql_exception
    raise InternalError(errno, errorvalue)
sqlalchemy.exc.InternalError: (InternalError) (1050, u"Table 'secret' already exists") '\nCREATE TABLE secret (\n\tcreated_at DATETIME, \n\tupdated_at DATETIME, \n\tdeleted_at DATETIME, \n\tid INTEGER NOT NULL AUTO_INCREMENT, \n\tacl_id INTEGER NOT NULL, \n\ttype INTEGER NOT NULL, \n\tsecret VARCHAR(512), \n\tPRIMARY KEY (id)\n)\n\n' ()

With offset beyond 1652, the api returns and empty list

The server logs show it got over 72K emails but while
http://localhost:5555/n/xxxxxxxxxxxxx/messages?limit=1&offset=1652
returns one email.
http://localhost:5555/n/xxxxxxxxxxxxx/messages?limit=1&offset=1653
returns []

Access-Control-Allow-Headers

Trying to modify the tags of a thread via PUT causes Chrome to first fire off an OPTIONS request like this.

OPTIONS /n/ud340xuwvl1hzqhens9xy237/threads/a2y5h6zn6yzxk5n7u4zp3nmqf HTTP/1.1
Host: localhost:5555
Connection: keep-alive
Access-Control-Request-Method: PUT
Origin: http://localhost
User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_0) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/36.0.1985.125 Safari/537.36
Access-Control-Request-Headers: accept, content-type
Accept: */*
Referer: http://localhost/~nickclaw/inbox-email-client/build/
Accept-Encoding: gzip,deflate,sdch
Accept-Language: en-US,en;q=0.8,fil;q=0.6

The inbox api returns:

HTTP/1.1 200 OK
Content-Type: text/html; charset=utf-8
Allow: HEAD, GET, PUT, OPTIONS, DELETE
Access-Control-Allow-Origin: http://localhost
Access-Control-Allow-Headers: Authorization
Access-Control-Allow-Methods: GET,PUT,POST,DELETE,OPTIONS
Access-Control-Allow-Credentials: true
Content-Length: 0
Date: Fri, 25 Jul 2014 02:40:22 GMT

which stops Chrome from successfully making the request because the Access-Control-Allow-Headers from the inbox-api doesn't allow content-type. Giving me this error:

XMLHttpRequest cannot load http://localhost:5555/n/ud340xuwvl1hzqhens9xy237/threads/a2y5h6zn6yzxk5n7u4zp3nmqf. Request header field Content-Type is not allowed by Access-Control-Allow-Headers.

It looks like the headers returned need to be changed to allow CORS requests, or maybe I'm just doing something wrong on my end.

GMail Provider Requires "Important" to be IMAP-Visible

In GMail settings, you're allowed to suppress some "key" folders from IMAP visibility: https://mail.google.com/mail/u/1/#settings/labels

For instance, I prefer to have "Important" not be a visible folder in my IMAP clients, so I disable it. This seems to cause problems here when you go to do the sync -- I presume because an expected folder doesn't exist.

Steps to repro: Uncheck "Show in IMAP" next to your "Important" folder then try to do an initial sync.

Uncaught NoResultFound exception

I'm seeing this error in the logs:

inbox.general        �[36m[D 2014-07-08 21:50:47,221 startup :89  ]�[39;49;0m Removing pyc files...�[39;49;0m
inbox.general        �[36m[D 2014-07-08 21:50:47,436 startup :96  ]�[39;49;0m Not writing pyc bytecode for this execution�[39;49;0m
inbox.general        �[32m[I 2014-07-08 21:50:48,462 startup :69  ]�[39;49;0m Head database revision: 29217fad3f46�[39;49;0m
inbox.general        �[32m[I 2014-07-08 21:50:48,462 startup :70  ]�[39;49;0m Current database revision: 29217fad3f46�[39;49;0m
inbox.general        �[32m[I 2014-07-08 21:50:48,462 startup :76  ]�[39;49;0m [OK] Database scheme matches latest�[39;49;0m
inbox.general        �[36m[D 2014-07-08 21:50:48,483 startup :138 ]�[39;49;0m Current git revision: 2764644�[39;49;0m
 * Restarting with reloader
inbox.general        �[36m[D 2014-07-08 21:50:49,101 startup :89  ]�[39;49;0m Removing pyc files...�[39;49;0m
inbox.general        �[36m[D 2014-07-08 21:50:49,321 startup :96  ]�[39;49;0m Not writing pyc bytecode for this execution�[39;49;0m
inbox.general        �[32m[I 2014-07-08 21:50:50,372 startup :69  ]�[39;49;0m Head database revision: 29217fad3f46�[39;49;0m
inbox.general        �[32m[I 2014-07-08 21:50:50,372 startup :70  ]�[39;49;0m Current database revision: 29217fad3f46�[39;49;0m
inbox.general        �[32m[I 2014-07-08 21:50:50,373 startup :76  ]�[39;49;0m [OK] Database scheme matches latest�[39;49;0m
inbox.general        �[36m[D 2014-07-08 21:50:50,392 startup :138 ]�[39;49;0m Current git revision: 2764644�[39;49;0m
inbox.general        �[32m[I 2014-07-08 21:50:50,914 inbox-st:76  ]�[39;49;0m Recursion limit: 1000�[39;49;0m
inbox.general        �[32m[I 2014-07-08 21:50:50,915 inbox-st:79  ]�[39;49;0m Providers currently supported: imap, yahoo, gmail�[39;49;0m
inbox.general        �[32m[I 2014-07-08 21:50:50,915 session :184 ]�[39;49;0m Don't yet have engine... creating default from ignition�[39;49;0m
inbox.general        �[32m[I 2014-07-08 21:50:51,101 service :55  ]�[39;49;0m Starting sync for account [email protected]�[39;49;0m
inbox.contacts.1     �[32m[I 2014-07-08 21:50:51,114 remote_s:37  ]�[39;49;0m Begin syncing contacts...�[39;49;0m
inbox.general        �[32m[I 2014-07-08 21:50:51,129 concurre:107 ]�[39;49;0m ZeroRPC: Starting WebhookService at tcp://0.0.0.0:9994�[39;49;0m
inbox.general        �[32m[I 2014-07-08 21:50:51,141 concurre:107 ]�[39;49;0m ZeroRPC: Starting SyncService at tcp://0.0.0.0:9997�[39;49;0m
inbox.general        �[32m[I 2014-07-08 21:50:51,148 crispin :99  ]�[39;49;0m Creating Crispin connection pool for account 1 with 8 connections�[39;49;0m
inbox.general        �[32m[I 2014-07-08 21:50:51,152 oauth   :64  ]�[39;49;0m Getting new oauth token...�[39;49;0m
inbox.general        �[32m[I 2014-07-08 21:50:51,166 oauth   :64  ]�[39;49;0m Getting new oauth token...�[39;49;0m
inbox.general        �[32m[I 2014-07-08 21:50:51,446 oauth   :40  ]�[39;49;0m Validating oauth token...�[39;49;0m
inbox.general        �[32m[I 2014-07-08 21:50:51,451 oauth   :40  ]�[39;49;0m Validating oauth token...�[39;49;0m
inbox.general        �[32m[I 2014-07-08 21:50:52,156 remote_s:135 ]�[39;49;0m Added 0 contacts.�[39;49;0m
inbox.general        �[32m[I 2014-07-08 21:50:52,157 remote_s:136 ]�[39;49;0m Updated 0 contacts.�[39;49;0m
inbox.general        �[32m[I 2014-07-08 21:50:52,157 remote_s:137 ]�[39;49;0m Deleted 0 contacts.�[39;49;0m
inbox.mailsync.1     �[32m[I 2014-07-08 21:50:52,597 base    :101 ]�[39;49;0m Folders were deleted from the remote: []�[39;49;0m
inbox.mailsync.1     �[32m[I 2014-07-08 21:50:52,920 imap    :154 ]�[39;49;0m Initializing folder sync for Inbox�[39;49;0m
inbox.mailsync.1     �[32m[I 2014-07-08 21:50:52,925 imap    :283 ]�[39;49;0m Starting initial sync for Inbox�[39;49;0m
inbox.mailsync.1     �[32m[I 2014-07-08 21:50:54,050 crispin :251 ]�[39;49;0m Selected folder Inbox with 72675 messages.�[39;49;0m
inbox.mailsync.1     �[32m[I 2014-07-08 21:50:54,686 gmail   :566 ]�[39;49;0m Attempting to retrieve remote_g_metadata from cache�[39;49;0m
inbox.general        �[32m[I 2014-07-08 21:50:54,686 cache   :45  ]�[39;49;0m Loading cache to /var/lib/inboxapp/cache/1/Inbox/remote_g_metadata�[39;49;0m
inbox.mailsync.1     �[32m[I 2014-07-08 21:50:54,845 gmail   :578 ]�[39;49;0m Successfully retrieved remote_g_metadata cache with 72656 objects�[39;49;0m
inbox.mailsync.1     �[32m[I 2014-07-08 21:50:54,846 gmail   :600 ]�[39;49;0m Updating cache with latest changes�[39;49;0m
inbox.mailsync.1     �[32m[I 2014-07-08 21:52:15,663 misc    :89  ]�[39;49;0m [timer] <function new_and_updated_uids at 0x500e6e0> took 80.816 seconds.�[39;49;0m
inbox.mailsync.1     �[32m[I 2014-07-08 21:52:15,664 gmail   :608 ]�[39;49;0m Found 0 modified�[39;49;0m
inbox.mailsync.1     �[32m[I 2014-07-08 21:52:15,664 gmail   :610 ]�[39;49;0m 0 new and 0 updated UIDs�[39;49;0m
inbox.mailsync.1     �[32m[I 2014-07-08 21:52:15,664 gmail   :615 ]�[39;49;0m No new messages to update metadata for�[39;49;0m
inbox.general        �[32m[I 2014-07-08 21:52:15,975 cache   :28  ]�[39;49;0m Saving cache to /var/lib/inboxapp/cache/1/Inbox�[39;49;0m
inbox.mailsync.1     �[32m[I 2014-07-08 21:52:16,000 gmail   :637 ]�[39;49;0m No modified messages to update metadata for�[39;49;0m
inbox.mailsync.1     �[32m[I 2014-07-08 21:52:16,015 gmail   :101 ]�[39;49;0m Found 72656 UIDs for folder Inbox�[39;49;0m
inbox.mailsync.1     �[36m[D 2014-07-08 21:52:16,140 gmail   :106 ]�[39;49;0m gmail_initial_sync grabbed syncmanager_lock�[39;49;0m
inbox.mailsync.1     �[32m[I 2014-07-08 21:52:59,153 gmail   :357 ]�[39;49;0m 71074 messages found initially (unsorted by thread)�[39;49;0m
inbox.mailsync.1     �[32m[I 2014-07-08 21:52:59,154 gmail   :359 ]�[39;49;0m Expanding threads and downloading messages.�[39;49;0m
inbox.mailsync.1     �[32m[I 2014-07-08 21:53:00,434 crispin :251 ]�[39;49;0m Selected folder [Gmail]/All Mail with 96503 messages.�[39;49;0m
inbox.mailsync.1     �[32m[I 2014-07-08 21:53:01,242 crispin :251 ]�[39;49;0m Selected folder Inbox with 72675 messages.�[39;49;0m
inbox.mailsync.1     �[36m[D 2014-07-08 21:53:01,246 crispin :533 ]�[39;49;0m Fetching X-GM-MSGID and X-GM-THRID for 1 uids.�[39;49;0m
inbox.mailsync.1     �[32m[I 2014-07-08 21:53:01,249 gmail   :290 ]�[39;49;0m Checking for new/deleted messages during initial sync.�[39;49;0m
inbox.mailsync.1     �[36m[D 2014-07-08 21:53:01,350 gmail   :426 ]�[39;49;0m Downloading thread 1471251894581484616 with 1 messages.�[39;49;0m
inbox.mailsync.1     �[36m[D 2014-07-08 21:53:01,404 gmail   :430 ]�[39;49;0m 1 deduplicated messages to download.�[39;49;0m
inbox.mailsync.1     �[36m[D 2014-07-08 21:53:01,405 gmail   :247 ]�[39;49;0m Downloading uids (176111L,)�[39;49;0m
inbox.mailsync.1     �[36m[D 2014-07-08 21:53:01,692 gmail   :295 ]�[39;49;0m check_new_g_thrids acquired syncmanager_lock�[39;49;0m
inbox.mailsync.1     �[32m[I 2014-07-08 21:53:01,746 gmail   :306 ]�[39;49;0m Removed 0 deleted UIDs from Inbox�[39;49;0m
inbox.mailsync.1     �[36m[D 2014-07-08 21:53:01,882 crispin :533 ]�[39;49;0m Fetching X-GM-MSGID and X-GM-THRID for 20 uids.�[39;49;0m
inbox.mailsync.1     �[32m[I 2014-07-08 21:53:01,992 gmail   :320 ]�[39;49;0m Adding 20 new messages to the download queue for Inbox�[39;49;0m
inbox.mailsync.1     �[32m[I 2014-07-08 21:53:02,011 gmail   :332 ]�[39;49;0m Idling on Inbox with 300 timeout�[39;49;0m
inbox.mailsync.1     �[36m[D 2014-07-08 21:53:02,012 gmail   :250 ]�[39;49;0m gmail_download_and_commit_uids acquired syncmanager_lock�[39;49;0m
inbox.mailsync.1     �[32m[I 2014-07-08 21:53:02,013 gmail   :441 ]�[39;49;0m Deduplicating message object creation.�[39;49;0m
inbox.mailsync.1     �[36m[D 2014-07-08 21:53:02,135 gmail   :255 ]�[39;49;0m Have 1 unsaved messages objects�[39;49;0m
inbox.general        �[31m[E 2014-07-08 21:53:02,592 message :161 ]�[39;49;0m DecodeError, msg logged to /var/log/inboxapp/1/errors/[Gmail]/All Mail/176111�[39;49;0m
inbox.mailsync.1     �[32m[I 2014-07-08 21:53:02,662 base    :168 ]�[39;49;0m Committing 0 UIDs�[39;49;0m
inbox.mailsync.1     �[36m[D 2014-07-08 21:53:02,808 gmail   :260 ]�[39;49;0m Committed 0 new messages�[39;49;0m
inbox.mailsync.1     �[36m[D 2014-07-08 21:53:02,808 gmail   :398 ]�[39;49;0m download_queued_threads acquired syncmanager_lock�[39;49;0m
inbox.mailsync.1     �[36m[D 2014-07-08 21:53:02,808 gmail   :403 ]�[39;49;0m Adding Inbox imapuid rows for 1 processed messages�[39;49;0m
inbox.general        �[32m[I 2014-07-08 21:53:07,824 crispin :140 ]�[39;49;0m Connection broken with error (<class 'sqlalchemy.orm.exc.NoResultFound'>, NoResultFound('No row was found for one()',), <traceback object at 0x452c4d0>); retrying with new connection�[39;49;0m
inbox.general        �[31m[E 2014-07-08 21:53:07,825 crispin :145 ]�[39;49;0m Max retries reached. Aborting. Error: (<class 'sqlalchemy.orm.exc.NoResultFound'>, NoResultFound('No row was found for one()',), <traceback object at 0x452c4d0>)�[39;49;0m
inbox.mailsync.1     �[31m[E 2014-07-08 21:53:07,826 log     :162 ]�[39;49;0m Uncaught error
Traceback (most recent call last):
  File "/vagrant/inbox/util/concurrency.py", line 70, in wrapped
    return func(*args, **kwargs)
  File "/vagrant/inbox/mailsync/backends/imap/imap.py", line 232, in _run_impl
    self.folder_name, self.shared_state)
  File "/vagrant/inbox/util/concurrency.py", line 70, in wrapped
    return func(*args, **kwargs)
  File "/vagrant/inbox/mailsync/backends/gmail.py", line 82, in initial_sync
    create_gmail_message)
  File "/vagrant/inbox/mailsync/backends/imap/imap.py", line 296, in base_initial_sync
    msg_create_fn)
  File "/vagrant/inbox/mailsync/backends/gmail.py", line 146, in gmail_initial_sync
    shared_state['syncmanager_lock'])
  File "/vagrant/inbox/mailsync/backends/gmail.py", line 405, in download_queued_threads
    add_new_imapuid(db_session, log, msg, folder_name, acc)
  File "/vagrant/inbox/mailsync/backends/gmail.py", line 502, in add_new_imapuid
    g_msgid=gmessage.g_metadata.msgid).one()
  File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/query.py", line 2368, in one
    raise orm_exc.NoResultFound("No row was found for one()")
NoResultFound: No row was found for one()�[39;49;0m
inbox.mailsync.1     �[32m[I 2014-07-08 21:53:07,839 imap    :283 ]�[39;49;0m Starting initial sync for Inbox�[39;49;0m
inbox.mailsync.1     �[32m[I 2014-07-08 21:53:08,859 crispin :251 ]�[39;49;0m Selected folder Inbox with 72675 messages.�[39;49;0m
inbox.mailsync.1     �[32m[I 2014-07-08 21:53:09,375 gmail   :566 ]�[39;49;0m Attempting to retrieve remote_g_metadata from cache�[39;49;0m
inbox.general        �[32m[I 2014-07-08 21:53:09,375 cache   :45  ]�[39;49;0m Loading cache to /var/lib/inboxapp/cache/1/Inbox/remote_g_metadata�[39;49;0m
inbox.mailsync.1     �[32m[I 2014-07-08 21:53:09,542 gmail   :578 ]�[39;49;0m Successfully retrieved remote_g_metadata cache with 72656 objects�[39;49;0m
inbox.mailsync.1     �[32m[I 2014-07-08 21:53:09,543 gmail   :600 ]�[39;49;0m Updating cache with latest changes�[39;49;0m
inbox.mailsync.1     �[32m[I 2014-07-08 21:54:21,167 misc    :89  ]�[39;49;0m [timer] <function new_and_updated_uids at 0x500e6e0> took 71.623 seconds.�[39;49;0m
inbox.mailsync.1     �[32m[I 2014-07-08 21:54:21,167 gmail   :608 ]�[39;49;0m Found 0 modified�[39;49;0m
inbox.mailsync.1     �[32m[I 2014-07-08 21:54:21,168 gmail   :610 ]�[39;49;0m 0 new and 0 updated UIDs�[39;49;0m
inbox.mailsync.1     �[32m[I 2014-07-08 21:54:21,168 gmail   :615 ]�[39;49;0m No new messages to update metadata for�[39;49;0m
inbox.general        �[32m[I 2014-07-08 21:54:21,462 cache   :28  ]�[39;49;0m Saving cache to /var/lib/inboxapp/cache/1/Inbox�[39;49;0m
inbox.mailsync.1     �[32m[I 2014-07-08 21:54:21,491 gmail   :637 ]�[39;49;0m No modified messages to update metadata for�[39;49;0m
inbox.mailsync.1     �[32m[I 2014-07-08 21:54:21,508 gmail   :101 ]�[39;49;0m Found 72656 UIDs for folder Inbox�[39;49;0m
inbox.mailsync.1     �[36m[D 2014-07-08 21:54:21,665 gmail   :106 ]�[39;49;0m gmail_initial_sync grabbed syncmanager_lock�[39;49;0m
inbox.mailsync.1     �[32m[I 2014-07-08 21:55:04,873 gmail   :357 ]�[39;49;0m 71074 messages found initially (unsorted by thread)�[39;49;0m
inbox.mailsync.1     �[32m[I 2014-07-08 21:55:04,873 gmail   :359 ]�[39;49;0m Expanding threads and downloading messages.�[39;49;0m
inbox.mailsync.1     �[32m[I 2014-07-08 21:55:06,137 crispin :251 ]�[39;49;0m Selected folder [Gmail]/All Mail with 96503 messages.�[39;49;0m
inbox.mailsync.1     �[32m[I 2014-07-08 21:55:06,963 crispin :251 ]�[39;49;0m Selected folder Inbox with 72675 messages.�[39;49;0m
inbox.mailsync.1     �[36m[D 2014-07-08 21:55:06,966 crispin :533 ]�[39;49;0m Fetching X-GM-MSGID and X-GM-THRID for 1 uids.�[39;49;0m
inbox.mailsync.1     �[32m[I 2014-07-08 21:55:06,970 gmail   :290 ]�[39;49;0m Checking for new/deleted messages during initial sync.�[39;49;0m
inbox.mailsync.1     �[36m[D 2014-07-08 21:55:07,073 gmail   :426 ]�[39;49;0m Downloading thread 1471251894581484616 with 1 messages.�[39;49;0m
inbox.mailsync.1     �[36m[D 2014-07-08 21:55:07,126 gmail   :430 ]�[39;49;0m 1 deduplicated messages to download.�[39;49;0m
inbox.mailsync.1     �[36m[D 2014-07-08 21:55:07,126 gmail   :247 ]�[39;49;0m Downloading uids (176111L,)�[39;49;0m
inbox.mailsync.1     �[36m[D 2014-07-08 21:55:07,536 gmail   :250 ]�[39;49;0m gmail_download_and_commit_uids acquired syncmanager_lock�[39;49;0m
inbox.mailsync.1     �[32m[I 2014-07-08 21:55:07,537 gmail   :441 ]�[39;49;0m Deduplicating message object creation.�[39;49;0m
inbox.mailsync.1     �[36m[D 2014-07-08 21:55:07,585 gmail   :255 ]�[39;49;0m Have 1 unsaved messages objects�[39;49;0m
inbox.general        �[31m[E 2014-07-08 21:55:08,143 message :161 ]�[39;49;0m DecodeError, msg logged to /var/log/inboxapp/1/errors/[Gmail]/All Mail/176111�[39;49;0m
inbox.mailsync.1     �[32m[I 2014-07-08 21:55:08,238 base    :168 ]�[39;49;0m Committing 0 UIDs�[39;49;0m
inbox.mailsync.1     �[36m[D 2014-07-08 21:55:08,240 gmail   :260 ]�[39;49;0m Committed 0 new messages�[39;49;0m
inbox.mailsync.1     �[36m[D 2014-07-08 21:55:08,241 gmail   :398 ]�[39;49;0m download_queued_threads acquired syncmanager_lock�[39;49;0m
inbox.mailsync.1     �[36m[D 2014-07-08 21:55:08,241 gmail   :403 ]�[39;49;0m Adding Inbox imapuid rows for 1 processed messages�[39;49;0m
inbox.mailsync.1     �[36m[D 2014-07-08 21:55:08,246 gmail   :295 ]�[39;49;0m check_new_g_thrids acquired syncmanager_lock�[39;49;0m
inbox.mailsync.1     �[32m[I 2014-07-08 21:55:08,304 gmail   :306 ]�[39;49;0m Removed 0 deleted UIDs from Inbox�[39;49;0m
inbox.mailsync.1     �[36m[D 2014-07-08 21:55:08,598 crispin :533 ]�[39;49;0m Fetching X-GM-MSGID and X-GM-THRID for 20 uids.�[39;49;0m
inbox.mailsync.1     �[32m[I 2014-07-08 21:55:08,722 gmail   :320 ]�[39;49;0m Adding 20 new messages to the download queue for Inbox�[39;49;0m
inbox.mailsync.1     �[32m[I 2014-07-08 21:55:08,741 gmail   :332 ]�[39;49;0m Idling on Inbox with 300 timeout�[39;49;0m
inbox.general        �[32m[I 2014-07-08 21:55:13,247 crispin :140 ]�[39;49;0m Connection broken with error (<class 'sqlalchemy.orm.exc.NoResultFound'>, NoResultFound('No row was found for one()',), <traceback object at 0xb4f8908>); retrying with new connection�[39;49;0m
inbox.general        �[31m[E 2014-07-08 21:55:13,248 crispin :145 ]�[39;49;0m Max retries reached. Aborting. Error: (<class 'sqlalchemy.orm.exc.NoResultFound'>, NoResultFound('No row was found for one()',), <traceback object at 0xb4f8908>)�[39;49;0m
inbox.mailsync.1     �[31m[E 2014-07-08 21:55:13,256 log     :162 ]�[39;49;0m Uncaught error
Traceback (most recent call last):
  File "/vagrant/inbox/util/concurrency.py", line 70, in wrapped
    return func(*args, **kwargs)
  File "/vagrant/inbox/mailsync/backends/imap/imap.py", line 232, in _run_impl
    self.folder_name, self.shared_state)
  File "/vagrant/inbox/util/concurrency.py", line 70, in wrapped
    return func(*args, **kwargs)
  File "/vagrant/inbox/mailsync/backends/gmail.py", line 82, in initial_sync
    create_gmail_message)
  File "/vagrant/inbox/mailsync/backends/imap/imap.py", line 296, in base_initial_sync
    msg_create_fn)
  File "/vagrant/inbox/mailsync/backends/gmail.py", line 146, in gmail_initial_sync
    shared_state['syncmanager_lock'])
  File "/vagrant/inbox/mailsync/backends/gmail.py", line 405, in download_queued_threads
    add_new_imapuid(db_session, log, msg, folder_name, acc)
  File "/vagrant/inbox/mailsync/backends/gmail.py", line 502, in add_new_imapuid
    g_msgid=gmessage.g_metadata.msgid).one()
  File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/query.py", line 2368, in one
    raise orm_exc.NoResultFound("No row was found for one()")
NoResultFound: No row was found for one()�[39;49;0m
inbox.mailsync.1     �[32m[I 2014-07-08 21:55:13,265 imap    :283 ]�[39;49;0m Starting initial sync for Inbox�[39;49;0m
inbox.mailsync.1     �[32m[I 2014-07-08 21:55:14,353 crispin :251 ]�[39;49;0m Selected folder Inbox with 72675 messages.�[39;49;0m
inbox.mailsync.1     �[32m[I 2014-07-08 21:55:14,859 gmail   :566 ]�[39;49;0m Attempting to retrieve remote_g_metadata from cache�[39;49;0m
inbox.general        �[32m[I 2014-07-08 21:55:14,859 cache   :45  ]�[39;49;0m Loading cache to /var/lib/inboxapp/cache/1/Inbox/remote_g_metadata�[39;49;0m
inbox.mailsync.1     �[32m[I 2014-07-08 21:55:15,021 gmail   :578 ]�[39;49;0m Successfully retrieved remote_g_metadata cache with 72656 objects�[39;49;0m
inbox.mailsync.1     �[32m[I 2014-07-08 21:55:15,022 gmail   :600 ]�[39;49;0m Updating cache with latest changes�[39;49;0m

Failure for setting unread property on a message

After spinning up the Vagrant VM following the documentation, attempting to set unread to true gives the following error:

$ curl -X PUT -H "Content-Type: application/json" -d '{"unread": true}' "http://localhost:5555/n/2ccozq2j92i7c38aour8ls0p/messages/b2jy9qkn0okdvdmaddfb9xosi
{
  "message": "'builtin_function_or_method' object has no attribute '__getitem__'",
  "type": "api_error"
}

Removing inconsequential parts of an address

I was unable to filter messages in format: [email protected], but I was able to do so for [email protected]. Gmail supports booth and it is hard to catch it user use the first variant.

Adding an imap account to sync

I have successfully added a gmail account (using the command line) to sync and it works wonderfully.

I cannot see how to add an imap account. Ideally, I would like to add it via the api and also know how to add a gmail account via the api but I cannot see anything in the docs for this. I have looked through the code also but cannot see anything. It looks like it can sync but I cannot see a resource I can use to add.

Log is not appearing and error is yielding

After updating the inbox app to the latest version (f327b12) the app stopped to log properly and some weird errors started to appear:

[ERROR] {"exception": "Traceback (most recent call last):\
  File \"/vagrant/inbox/util/concurrency.py\", line 69, in wrapped\
    return func(*args, **kwargs)\
  File \"/vagrant/inbox/mailsync/backends/imap/imap.py\", line 156, in sync\
    folder_id_for[folder_name],\
KeyError: u'Inbox'", "account_id": 1, "level": "error", "timestamp": "2014-07-21T19:32:24.743533Z", "component": "mail sync", "module": "inbox.util.concurrency:96", "event": "Uncaught error"}

We decided to implode the current vagrant and start a new one from scrach (vagrant box remove precise64 virtualbox) and the error persisted.

In both cases, the API was consistently return 501.

The commit 50274a1 is good, but we could not do any bisect because we lacked time. =/

The env is:

Linux themachine 3.11.0-23-generic #40-Ubuntu SMP Wed Jun 4 21:05:23 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
cat /etc/debian_version #=> wheezy/sid

[WIKI] Custom Google Applications

I added a page to the new Wiki @spang so graciously setup. Unfortunately, GH doesn't seem to support PRs on Wikis yet, so this issue is an old-fashioned merge request.

You can preview the page here: https://github.com/trestletech/inbox/wiki/Custom-Google-Application and the associated Git repo is https://github.com/trestletech/inbox.wiki.git (which would be merged into [email protected]:inboxapp/inbox.wiki.git if the changes were accepted).

If you're willing to start giving out Wiki modification privileges, I'd be happy to make the changes on the real wiki myself.

feature req: Allow sending mail through 3rd party provider like Mailgun

Copied from mail as suggested:

I'd like to know if, in time, it becomes possible to build CRM like applications like Contactually with InboxApp.

For this it's crucial to be able to register Mailgun or similar as the provider to actually send mail with, because of limitations in the nr of mails/ day to be send through, say, gmail.

Is this something that is supported / in the pipeline?

File metadata endpoints should return the same/consistent structure

https://gunks.inboxapp.com/n/6rhcf8db14sq3i8aixskd8n1z/files and https://gunks.inboxapp.com/n/6rhcf8db14sq3i8aixskd8n1z/files/7op1lcla3h4vy2hslg05y59bn yield two different structures:

When querying for multiple files, we get something like this:

[
    {
        "content_type": null,
        "filename": null,
        "id": "7op1lcla3h4vy2hslg05y59bn",
        "is_embedded": false,
        "message": "6ygq9a7qjhke87jud5wxc7967",
        "namespace": "6rhcf8db14sq3i8aixskd8n1z",
        "object": "file",
        "size": 4407
    }
]

Whereas when querying for a single file, we get something like this:

{
    "content_type": null,
    "id": "7op1lcla3h4vy2hslg05y59bn",
    "namespace": "6rhcf8db14sq3i8aixskd8n1z",
    "object": "file",
    "size": 4407
}

What this is telling me is that we actually have two structures: files which are bound to Messages (Attachments), and files which are just separate independent entities living on the server.

I think it would make sense to either make this separation clear in the endpoint names (eg /attachments vs /files), or otherwise find some other way to unify them. Consistency here is pretty important.

Docker image?

ImportError: No module named inbox.config

When I run bin/inbox-start this is returned:

File "bin/inbox-start", line 8, in
from inbox.config import config
ImportError: No module named inbox.config

is this is an issue or is something else causing this?

Not able to run inbox-start

Hi Inbox,

Getting below error in running inbox-start command.

Traceback (most recent call last):
File "bin/inbox-start", line 92, in
main()
File "/usr/local/lib/python2.7/dist-packages/click/core.py", line 488, in call
return self.main(_args, *_kwargs)
File "/usr/local/lib/python2.7/dist-packages/click/core.py", line 474, in main
self.invoke(ctx)
File "/usr/local/lib/python2.7/dist-packages/click/core.py", line 659, in invoke
ctx.invoke(self.callback, *_ctx.params)
File "/usr/local/lib/python2.7/dist-packages/click/core.py", line 325, in invoke
return callback(_args, **kwargs)
File "bin/inbox-start", line 42, in main
preflight()
File "/home/user/INBOX/inbox/inbox/util/startup.py", line 132, in preflight
check_db()
File "/home/user/INBOX/inbox/inbox/util/startup.py", line 74, in check_db
'Outdated database! Migrate using alembic upgrade head')
Exception: Outdated database! Migrate using alembic upgrade head

Also I tried alembic upgrade head, but then gets below error.

File "/usr/local/lib/python2.7/dist-packages/pymysql/err.py", line 112, in _check_mysql_exception
raise InternalError(errno, errorvalue)
sqlalchemy.exc.InternalError: (InternalError) (1050, u"Table 'actionlog' already exists") '\nCREATE TABLE actionlog (\n\tcreated_at DATETIME NOT NULL, \n\tupdated_at DATETIME NOT NULL, \n\tdeleted_at DATETIME, \n\tid INTEGER NOT NULL AUTO_INCREMENT, \n\tnamespace_id INTEGER NOT NULL, \n\taction TEXT(40) NOT NULL, \n\trecord_id INTEGER NOT NULL, \n\ttable_name TEXT(40) NOT NULL, \n\tPRIMARY KEY (id), \n\tFOREIGN KEY(namespace_id) REFERENCES namespace (id)\n)\n\n' ()

document Thread#subject_date

What is this? Date the subject was set / updated on? Date the thread was started on?

It's not really self-documenting, so it would be good to update the api docs with this info.

1 of 33 tests failing; new mail hitting gmail is only getting processed on restarting ./inbox start

inbox app team: Thanks for a great project. I had been looking for projects using zerorpc. I have been playing around with inbox today and have it mostly working except for a single test. I wasn't sure if this was broken for you too? Part of the error msg was "No handlers could be found for logger "inbox.server.general"" and I was wondering if this might explain why new mails hitting my gmail inbox aren't getting picked up and pushed across to mysql. I set up a workbench GUI to try to understand the tables in mysql a bit better ... the message and thread tables only appear to update when I stop and restart ./inbox start. As per the installation readme, I had set up ./inbox sync start [email protected] ... and wasn't sure if this was hindering new mails being picked up automatically. Thanks. Colum

runtest output is here: https://gist.github.com/Analect/f98543c05c39d763b307

Update Instructions

It would be nice to have detailed update instructions. Like:

vagrant halt
git pull
vagrant up
vagrant ssh
cd /vagrant/
sudo sh setup.sh
...

I am unable to start server again after the update.

./setup.sh failing on zerorpc setup ... some guidance needed!

Hi,
I've run through the README to set up a virtualbox, which I'm running under parallels on mac osx 10.9. I modified the Vagrantfile. I ended up commenting out some of the config.vm.customize items, which parallels doesn't support, but that didn't appear to impact the virtualbox booting up. I ran ./setup.sh ... which all seemed to run ok ... except for this part towards the end related to zerorpc ... it's not altogether clear to me what might be the issue here ... so just posting the output to /home/vagrant/.pip/pip.log to see if anyone there might have suggestions.

Thanks very much,
Colum

Obtaining zerorpc from git+git://github.com/inboxapp/zerorpc-python.git@master#egg=zerorpc (from -r requirements.txt (line 35))
Cloning git://github.com/inboxapp/zerorpc-python.git (to master) to ./src/zerorpc
Found command 'git' at '/usr/bin/git'
Running command /usr/bin/git clone -q git://github.com/inboxapp/zerorpc-python.git /media/psf/vagrant/src/zerorpc
Complete output from command /usr/bin/git clone -q git://github.com/inboxapp/zerorpc-python.git /media/psf/vagrant/src/zerorpc:

Command /usr/bin/git clone -q git://github.com/inboxapp/zerorpc-python.git /media/psf/vagrant/src/zerorpc failed with error code 128
Exception information:
Traceback (most recent call last):
File "/usr/lib/python2.7/dist-packages/pip/basecommand.py", line 126, in main
self.run(options, args)
File "/usr/lib/python2.7/dist-packages/pip/commands/install.py", line 223, in run
requirement_set.prepare_files(finder, force_root_egg_info=self.bundle, bundle=self.bundle)
File "/usr/lib/python2.7/dist-packages/pip/req.py", line 930, in prepare_files
req_to_install.update_editable(not self.is_download)
File "/usr/lib/python2.7/dist-packages/pip/req.py", line 378, in update_editable
vcs_backend.obtain(self.source_dir)
File "/usr/lib/python2.7/dist-packages/pip/vcs/git.py", line 107, in obtain
call_subprocess([self.cmd, 'clone', '-q', url, dest])
File "/usr/lib/python2.7/dist-packages/pip/init.py", line 255, in call_subprocess
% (command_desc, proc.returncode))
InstallationError: Command /usr/bin/git clone -q git://github.com/inboxapp/zerorpc-python.git /media/psf/vagrant/src/zerorpc failed with error code 128

Sending API does not seem to work correctly.

Working with dev branch (pulled this morning, 12 hours ago), I'm noticing that messages sent with reply_to_thread in the payload are not actually attached to the replied thread. Instead, a new thread is created (actually, multiple new threads are created over the course of the reply's state changes --- it ends up being 3 new threads every time).

I've created an example video which hopefully github will let me upload, but if not I'll send that to vimeo momentarily.

Anyhow, the request payload being sent in this video is the following (the actual formatting does not have this whitespace, and obviously the comments are just informative):

// POST http://localhost:5555/n/i7bpj8skzz2647f5qs0myoqp/send
{
  "body": "This is a reply to a test message.",
  "namespace": "i7bpj8skzz2647f5qs0myoqp",
  "reply_to_thread": "3dsedatcbzwzg4j2qag1qhf64",
  "to": [
    {
      "email": "[email protected]",
      "name": ""
    }
  ]
}

My expectation would be that to should be optional and inferred from the thread being replied to, if I am claiming to be "replying to a thread", but more importantly, if I'm replying to a thread, the new message should be added to that same thread.

If threads are entirely immutable and a new thread must be created whenever a message is attached, then that's understandable --- however, if that is the case, the newly created thread should contain all of the original messages from the old thread.

Add snippet to message object

Have you considered adding "snippet" property from thread object to message object? I would find it very helpful being able to use it as a preview of a message instead of parsing the entire body and generating a snippet myself.

Running inbox does not load configuration from config.cfg

Running sudo ./inbox debug returns:

Couldn't load configuration from config.cfg

Interestingly, the previous command (sudo ./setup.sh) ends with:

Successfully installed inbox
Cleaning up...
[InboxApp] Finished installing dependencies.
140204 00:38:18 mysqld_safe Can't log to error log and syslog at the same time.  Remove all --log-error configuration options for --syslog to take effect.
140204 00:38:18 mysqld_safe Logging to '/var/log/mysql/error.log'.
140204 00:38:18 mysqld_safe A mysqld process already exists
ERROR 1007 (HY000) at line 1: Can't create database 'test'; database exists

Thanks!

Start and stop inbox-sync for all accounts by one command

It would be nice to have an option to stop inbox-sync for all linked accounts at once.
./bin/inbox-sync stop --all

Same for start as well:

./bin/inbox-sync start --all

Unable To Start Inbox Server (master)

git branch -v
  dev    2217f01 Strip <title>, <script> and <style> tags from HTML emails.
* master b6e7815 Moving to latest version of Pygments for Swift code formatting support

Used commands:

git pull origin master
alembic upgrade head
cp etc/config-dev.json /etc/inboxapp
./bin/inbox-start

Error log: https://gist.github.com/pavelbinar/81917a8f3e36765102c4#file-error-log-L75

Snippet should not contain HTML/CSS

The snippet contained in the thread object often contains HTML/CSS, frequently so much that even with a regex removing the HTML no meaningful text-only thread preview could be displayed without loading the most recent message. Some form of regex should remove HTML tags and CSS from the message body before the snippet is generated.

AWS Sync Engine

Is there anything to watch out for or recommendations when running the sync engine on Amazon Web Services. Has anyone tried with success?

If it has been done, any chance we could add a little bit about it in the documentation as a quicker way to start up?

vagrant directory not in PYTHONPATH by default

Each of the binaries are unable to import the inbox.submodule.... packages without adding the root directory to PYTHONPATH manually.

Each script could add the path on its own, or alternatively setup.sh or the Vagrantfile could export it.

README.md Broken Link

Problem

In file README.md, the provided link to the Contributor License Agreement is broken. The aforementioned link just redirects to the Inbox home page.

I wanted to sign it but can't seem to find where it is located.

Line endings in Windows

I am running Windows 7 on a Windows 64-bit machine.

I needed to make the following changes to the Getting Started procedure:

Changes I made to Vagrantfile:

if File.exist?("../inbox-eas") (deleted)
if File.exist?("../inbox") (inserted)
config.vm.synced_folder "../inbox-eas", "/inbox-eas" (deleted)
config.vm.synced_folder "../inbox", "/vagrant" (inserted)

When running ./setup.sh and ./inbox:
Error message: ... /bin/sh^M: bad interpreter: ...
Solution: Copy each file to a new file; save with line endings set to "unix" format.

With these changes, the Getting Started procedure worked.

Sync is Killed

I am having trouble to sync up my Gmail account.

Each time I start the Inbox server it start syncing automatically nad after the moment the sync is killed (status from Monocle dashboard: State (running/ stopped/ killed)).

If I stop the sync by command inbox-sync stop and than start again by inbox-sync start it starts the sync but again after the moment (about a minute) it is killed.

Here is the log:
https://gist.github.com/pavelbinar/d9f727930ca16f636733#file-killed-sync-log-L519

(! 505)->  git branch -v
* dev    422d262 [behind 4] X-GM-MSGID is unique per account, but not globally.
  master f327b12 Merge pull request #47 from lheyberger/master

api for email header path

It would be great to have an api for the email header field.

Thanks
/tesh