rgladwell / imap-upload Goto Github PK
View Code? Open in Web Editor NEWPython script for uploading a local mbox file to IMAP4 server.
License: Other
Python script for uploading a local mbox file to IMAP4 server.
License: Other
Hi,
how can Import from MBOX Google Take Out Gmail to IMAP4 server?
I think that ImapUploadMessage.as_string function should be used also for non Google Takeout uploads to fix encoding problems of some messages.
Hi,
Here is my local structure :
When I try : python imap_upload.py --host=imap.myhost.com --port=143 --user=[email protected] --password=password -r /FOLDER/MASTER
Inbox go to Inbox
MASTER folder is create
But FOLDER1 is not created, either SUBFOLDERB and all mails are going to MASTER.
Any ideas ?
Thanks for the work !
Great work!
I am trying to use this tool to upload my emails from Thunderbird (68.10) on Ubuntu 18.04 to Gmail (Google Worksapce). While setting up Thunderbird for Gmail (IMAP), I realized that Gmail nowadays only allows authentication via OAuth2. Because of this reason, when I try this script with --username and --password it fails with the following error:
Connecting to imap.gmail.com:993. IMAP4 error: [AUTHENTICATIONFAILED] Invalid credentials (Failure)
Is there a workaround?
Hi,
I have an error if my folder contain a name with accent: Foncière.mbox
Error message:
An unknown error has occurred [493]: 'ascii' codec can't decode byte 0xc3 in position 51: ordinal not in range(128)
As --google-takeout-language has a default 'en', the code suspects, that the option was set by the user and complains, that --google-takeout was not supplied. So currently (684cd23) --google-takeout is a mandatory option.
Do you know how to check if the user had supplied --google-takeout-language on cmdline or if it is just the default?
We might could just ignore google-takeout-language, if google-takeout was not supplied... suggestions?
On the CLI, I saw the following error after python imap_upload.py --gmail --box imported mbox_file.mbox
:
Username: [email protected]
Password: ........
Connecting to imap.gmail.com:993.
IMAP4 error: [AUTHENTICATIONFAILED] Invalid credentials (Failure)
The error is related to what Google calls "Less secure apps". I fixed it by going to My Account -> Security and then choosing "Turn on access (not recommended)":
Tried pointing it at Local Folders and various .sbd folders, it connects to gmail, then there's no output, and it stops in about 2 seconds. Problem is that Thunderbird doesn't add .mbox to the end of the file names. When this is fixed, it works recursively, but doesn't seem to understand the folder hierarchy beyond one level, dumping a bunch of messages into parent folders.
If you're reading this and trying to use this script on linux, here's a helpful bash one-liner to rename all the files without extensions to .mbox. If it renames a file that isn't an mbox file, no big deal, the script will just skip it. Obviously change path to your directory
find /path -type f -not -name "." -exec mv "{}" "{}".mbox ;
Thanks for the script, it's really useful for large mbox files, but not so much for complex hierarchies.
The new -r option introduced in v1.3 does recurse into local directories but not recreate the same structure on the imap server, if I'm not mistaking. Is this intentional?
Thanks for the great work!
There doesn't seem to be a way to import emails to google workspace to a specific sub-label. I've tried using the --box command which creates the label, but the emails end up in the main inbox/drafts/important/sent mail folders. Any way to avoid this and just upload all to a label and have it create the sub-labels under that based on structure? Example structure of Mac Mail mbox eport below:
Just list mail boxes!
"INBOX""
"[Gmail]"" #Noselect
+- "All Mail"" #All
+- "Drafts"" #Drafts
+- "Important"" #Important
+- "Sent Mail"" #Sent
+- "Spam"" #Junk
+- "Starred"" #Flagged
+- "Trash"" #Trash
When an mbox
or directory name in a scanned path contains a non-ASCII character the script fails with the following error:
An unknown error has occurred [493]: 'ascii' codec can't decode byte 0xcc in position 62: ordinal not in range(128)
Tested with this folder structure:
Archiv
└── testing
├── Drehbücher
│ ├── Drehbücher\ 2008.mbox
│ ├── Drehbücher\ 2009.mbox
│ ├── Drehbücher\ 2010.mbox
│ ├── Drehbücher\ 2011.mbox
│ ├── Drehbücher\ 2012.mbox
│ ├── Drehbücher\ 2013.mbox
│ ├── Drehbücher\ 2014.mbox
│ ├── Drehbücher\ 2015.mbox
│ ├── Drehbücher\ 2016.mbox
│ ├── Drehbücher\ 2017.mbox
│ ├── Drehbücher\ 2018.mbox
│ ├── Drehbücher\ 2019.mbox
│ ├── Drehbücher\ 2020.mbox
└── Drehbücher.mbox
Happy to test further and provide more feedback if needed.
I'm using recursive, but I see in line 475 for recursive uploads, it sets the "BOX" to "" - is this by design? For my needs, I've changed this to do the same as non-recursive; options["box"] instead of ""
Thanks your project is working great for me so far!
When trying to send emails to an account that uses cpanel, the following error occurs.
(Mailbox name should probably be prefixed with: INBOX.)
I tried the script and, even though I disabled the Less secure app access, I get this error,
python imap_upload.py --gmail --list_boxes
IMAP4 error: [ALERT] Please log in via your web browser: https://support.google.com/mail/accounts/answer/78754 (Failure)
What is going on?
Hello!
I backed up an mbox (via Google Takeout) to migrate 89,000 emails from one account to another.
Due to an issue, I interrupted the script's process, and now I want to know if running it again will duplicate the emails.
My command is as follows:
python imap_upload.py --ssl [email protected] --password="password" --host=imap.gmail.com --port=993 --dry-run --error="errores.mbox" --google-takeout "original.mbox"
Thank you in advance for your help, and for building this great tool!
The only tagged release is 1.0.
If I import single file without extension it works however recursive option not work. Thunderbird stores mailboxes on Windows without giving them extension so with large number of mailboxes it can be pain to change them all manually. Is it any way for a program to detect mboxes anyway?
However even with that option script seems to stop after processing first folder
322/322 5.3 kB Nowe wiadomości w skrzynce OK (1 sec)
Done. (OK: 322, WARNING: 0, ERROR: 0)
Found mailbox at /Volumes/T5/LocalFolders/Archives/2011.mbox...
I want to use this script to import mbox
file created via Google Takeout service. They export all mail as a single mbox file. However, each message has an additional header specifying to what folder the message belongs to. E.g:
X-Gmail-Labels: Archived,Sent
It would be great if this script can handle these.
Thanks for this tool!
I just ran the script with the following command, on an .mbox file from Google Takeout containing approximately 7,000 emails:
$ python3 imap_upload.py --gmail --box imported takeout.mbox
It seems to have got stuck on an email with a subject line containing an ð ("eth") character. The full subject line is "FW: Youtube Job Wants You 👉 $20K/Month Potential! 80272150" (yes, it appears to be a spam message).
Is there anything I can do to recover from this? If I run the script a second time, will it upload duplicate emails?
On a mailcow mail setup, folders are located parallel to INBOX:
Just list mail boxes!
"Test" #UnMarked
"Archive" #UnMarked #Archive
"Trash" #Marked #Trash
"Junk" #UnMarked #Junk
"Sent" #UnMarked #Sent
"Drafts" #UnMarked #Drafts
"INBOX"
on a Hetzner Mailserver (Level 9 Webhosting) all folders are below INBOX
"INBOX"
+- "test" #Marked
+- "Trash" #UnMarked #Trash
+- "spambucket" #UnMarked #Junk
+- "Sent" #Marked #Sent
+- "Drafts" #UnMarked #Drafts
+- "Templates" #UnMarked
On the latter, creation of folders and append fails (using --google-takeout) with
Traceback (most recent call last):
File "/home/leonard/src/imap-upload-poldy79/imap_upload.py", line 426, in upload
raise Exception(r2[0]) # FIXME: Should use custom class
Exception: b'Client tried to access nonexistent namespace. (Mailbox name should probably be prefixed with: INBOX.) (0.001 + 0.000 secs).'
)
I am not sure if this is already covered by imap-upload.py and I have overseen something or if this is not yet supported.
If I prefix the path for imap.create_folder and imap.append with "INBOX" everything works fine.
I could provide a solution, might be a switch --everything-below-inbox ? What do you think?
Hello,
at first I want to thank you for this great tool - it works great!
...Well, until it meets an exception.
Configuration: I have Thunderbird 52.2.1 installed on Windows. I come from the German speaking world, so Umlauts (Ä / Ö / Ü / ß) are quite common in my e-mails. The mails go back to 2011 - I used Office Outlook back then, but imported all of them (they work well in Thunderbird).
My folder-structure is something along:
and so on.
I tried both branches of your tool, with thunderbird-support
working at least for greater time-spans.
As I thought it would be easier, I used my Ubuntu (16.04) VM, copied the thunderbird-folder to it and used the following code:
python imap_upload.py --gmail --box=test [email protected] --password=mypassword --retry=10 -r /custom_thunderbird_folder/Profiles/default/Mail/POP-client/Inbox.sbd
That will (successfully!) upload most of my e-mails - however, some certain subfolders seem to make troubles:
After finishing one subfolder, it almost immediately states:
Found mailbox at /custom_thunderbird_folder/Profiles/default/Mail/POP-client/Inbox.sbd/[subfolder-name].mozmsgs/md5[...].wdseml...
Uploading to [subfolder-name]...
Counting the mailbox (it could take a while for the large one).
Done. (OK: 0, NG: 0)
An unknown error has occurred: 'ascii' codec can't decode byte 0xc3 in position 116: ordinal not in range(128)
(Things in [ ]
brackets were changed by me.)
I can reproduce the error, and it always stays the same (and on the same spot). If I delete the subfolder, it may find another subfolder that throws the same error.
I tried to see if I can fix it myself by looking for mentions of "ascii", "utf", "code" and "ordinal" in your script and replacing them with "utf-8" where it seemed to make sense. However, that did not make any difference, so I would assume that it has to do with one of the imported modules.
So I'm out of ideas. Do you have any idea about what the problem might be?
I will gladly upload any sample here IF there is a way to first delete sensitive data - However, I fear that manipulating the files will lead to other errors... Of course, I will do what I can to give you all the information you need - just tell me what it is.
Any help is appreciated - thanks in advance!
https://github.com/rgladwell/imap-upload/blob/master/imap_upload.py#L414
reference msg.boxes
but msg.boxes
is assigned at
https://github.com/rgladwell/imap-upload/blob/master/imap_upload.py#L394
only for google_takeout usage
it looks like based on this snippet of code and confirmed by my server logs, that on each email upload it tries to create the dest folder (either INBOX or --box) i presume. Can the code be modified to check for the existence of the folder first and if it doesn't exit then create it
383 try: 384 self.open() 385 self.imap.create(box) 386 return self.imap.append(box, [], delivery_time, message)
Hello and thanks for this tool.
I keep getting this error:
NG (Client tried to access nonexistent namespace. (Mailbox name should probably be prefixed with: INBOX.)
I'm trying this with an empty mailbox that I set up for testing purposes. I'm positive that INBOX
is the correct IMAP prefix to use. --list_boxes
gives me this result:
"INBOX"
+- "Sent Messages"" #UnMarked
+- "Drafts" #UnMarked #Drafts
Any idea on what the problem might be?
Using current master (684cd23) with python 3.9.2 on debian bullseye with IMAPClient==2.2.0 option --list_boxes requires --google-takeout. Furthermore it then fails with
IMAP Upload (v2.0.0)
Connecting to mail.your-server.de:993.
Just list mail boxes!
An unknown error has occurred [679]: cannot use a string pattern on a bytes-like object
It seems that uploader.list_boxes() returns the list of boxes as a byte stream, which has to be decoded to a string.
I suspect, that this is because of a change in IMAPClient. You might add a requirements.txt which states the latest version known to work.
I am running python 3.8 in a Windows 10 machine.
I have changed the headers to
#!/usr/bin/env python3.8
but I keep on getting this error:
Python .\imap_upload.py
File ".\imap_upload.py", line 213
except Exception, e:
^
SyntaxError: invalid syntax
Any way to solve it?
This version prematurely exists without any error message if I give it the right credentials to gmail. If I purposely give it wrong login data it displays an error message. (tried with --gmail and with imaps://imap.gmail.com/ ). Not tested with other providers.
I was stumped for awhile, thinking the script was not grabbing the dates for my emails correctly, but it was actually an issue with how google handles multiple copies of the same email (apparently it keeps the metadata from the first one for all copies).
What happened was:
from
time field.time-fields=date,received
parameter.The issue was that the first/wrong message was still in my Trash on google, and not actually deleted. Soo, when I was trying the re-upload with the correct date, I think the google servers see the duplicate and simply un-trash it (complete with the original incorrect date).
Anyway, maybe you could add a mention of this behavior to your docs? Maybe it'll help other people in who encounter the same situation.
Thanks for the AWESOME work!
See https://github.com/chriskiehl/Gooey. I tried to get a build working locally, but failed when I got to the pyinstaller -F --windowed build.spec
part under https://github.com/chriskiehl/Gooey#packaging. Not too familiar with Python, but thought that you could benefit from adding Gooey to this project so that it would have widespread ease of use (and users wouldn't need to download python/etc).
I have 20GB .MBOX file exported from Google g-suite, but importing using roundcube or horde not possible for files bigger than 2GB.
Can I using this tool to import MBOX to cPanel?
I have tried a variety of different options to upload my Google Takeout mbox to my IMAP account. No matter what I do I get:
to [Opened,Category Updates] NG (b'Client tried to access nonexistent namespace. (Mailbox name should probably be prefixed with: INBOX
Here is the command line I am using. Any suggestions would be greatly appreciated...
python3 imap_upload.py --ssl --user=[email protected] --password='test' --host=imap.dreamhost.com --port=993 --error='All mail Including Spam and Trash_errors.mbox' --box='INBOX' --google-takeout --google-takeout-box-as-base-folder 'All mail Including Spam and Trash-002.mbox'
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.