Git Product home page Git Product logo

getmail's Introduction

Getmail

Getmail is a small Python script to retrieve emails from IMAP accounts (e.g. gmx.de, gmail.com) and deliver these emails to the Mailcow/Dovecot mailbox. I wrote to Getmail because I couldn't find any other solution with IMAP-IDLE in 2018 mailcow/mailcow-dockerized#1554. Emails are retrieved using IMAP IDLE, so emails are retrieved immediately and not after a fixed interval (as with imapsync from Mailcow). The transfer to Mailcow is done via LMTP interface (simplified SNMP) from Dovecot, so you can also use the sieve rules from Dovecoat (imapsync from Mailcow only allows a fixed folder). The goal of Getmail is to empty the entire INBOX. If the source IMAP account contains emails, you should copy/move them to another folder for testing (e.g. with an email programm or webmail).

Install:

  • Clone getmail
    cd /opt
    git clone https://github.com/christianbur/getmail.git
    cd /opt/getmail
    
  • Copy the docker-compose.override.yml file to the mailcow-dockerized folder. Please check if there is already a docker-compose.override.yml in the mailcow directory!!
    cp /opt/getmail/mailcow-dockerized_docker-compose.override.yml /opt/mailcow-dockerized/docker-compose.override.yml
    
  • Because the docker network "network-getmail" is used independently in mailcow and getmail, the network "network-getmail" must be created externally (i.e. not in the docker-compose.yml). Any other IP range can be used, it should only not be already in use (test: ip show -6 route show, ip show route show).
     docker network create --driver=bridge --ipv6 --subnet=fdcb:9c90:23:11::/64 --gateway=fdcb:9c90:23:11::1 --subnet=172.23.11.0/24 --gateway=172.23.11.1 -o "com.docker.network.bridge.name"="br-getmail" network-getmail
    
  • Config file must be customized
     cp /opt/getmail/settings.ini.example  /opt/getmail/settings.ini
     vi /opt/getmail/settings.ini
    
  • Start mailcow and getmail.
     cd /opt/getmail
     docker compose build 
     docker compose  up -d
    
  • Now check the logs from getmail
    docker compose logs
    

Config

Getmail is configured with the configuration file .getmail/settings.ini. Everything under [DEFAULT] applies to all IMAP accounts. Mostly only imap_hostname:, imap_username:, imap_password: have to be customized. In the source IMAP account only one folder is monitored (default = imap_sync_folder: INBOX), if the junk folder should also be monitored, two accounts must be created.

[INBOX_test_gmx.de]
imap_hostname:     imap.gmx.net
imap_username:     [email protected]
imap_password:     xxx
# INFO: "imap_sync_folder: INBOX" is default

[JUNK_test_gmx.de]
imap_hostname:     imap.gmx.net
imap_username:     [email protected]
imap_password:     xxx
imap_sync_folder:  Junk

Normally retrieved emails are deleted from the source IMAP account (imap_move_enable: False), the goal of the script is to completely empty the monitored folder (e.g. INBOX). It is also possible to move the emails to a folder in the IMAP source account; the emails are then available in both the IMAP source account and the IMAP target account.

imap_move_enable: True
imap_move_folder: getmail
# "getmail" "getmail" is an existing folder in the IMAP source account

With 'lmtp_recipient:' you specify the destination imap account in mailcow.

Sieve filter: In every retrieved email, two header (X-getmail-retrieved-from-mailbox-user, X-getmail-retrieved-from-mailbox-folder) are added, with this information you can filter with sieve (Mailcow: Mail Setup -> Filters -> Add Filter)

Example:

require "fileinto";
require "regex";
require "body";

...
if header :contains ["X-getmail-retrieved-from-mailbox-user"] ["[email protected]", "[email protected]"]
{
    fileinto "INBOX/Getmail_GMX";
}
elsif header :contains ["X-getmail-retrieved-from-mailbox-user"] ["[email protected]", "[email protected]"]
{
    fileinto "INBOX/Getmail_Outlook";
}
else
{
  # The rest goes into INBOX
  # default is "implicit keep", we do it explicitly here
  keep;
}

TZ variable:

The TZ vairable defines the time zone. I have defined the variable in the file /etc/default/locale (reboot may be necessary afterwards)

# cat /etc/default/locale
LANG="en_US.UTF-8"
LANGUAGE="en_US:en"
TZ="Europe/Berlin"

Alternatively, you can replace TZ=${TZ} with TZ="Europe/Berlin" in the docker-compose.yml.

If the TZ variable is not defined, the following error message appears when starting getmail: "WARNING: The TZ variable is not set. Defaulting to a blank string."

getmail's People

Contributors

christianbur avatar psychowood avatar

Stargazers

 avatar  avatar Matthias Hester avatar SalimTerryLi avatar  avatar Juan Luis Francés avatar peroksid5 avatar Lea Waller avatar Alexander Dunkel avatar axute avatar Rasel Hossain avatar André avatar  avatar  avatar Tobias Strobel avatar  avatar

Watchers

Petros Koutsolampros avatar André avatar Tim Eisele avatar  avatar  avatar

getmail's Issues

Max. message size

Hi,

Is there a size limitation in getmail ? : i received today a mail with attachements (av 32Mo) in my FAI mailbox and it seems this mail blocked all the getmail process. No download for the following mails.

I tried message_size_limit = 3932160 in "/opt/mailcow-dockerized/data/conf/postfix/extra.cf" but it changes nothing.

So is there a max message size setting for getmail ? or anything else to do ?

Thanks,
Phil

LMTP-Server is not reachable

Hey there,
I was using getmail without problems for a longer time. Now it does not work. I have an idea why, but no clue how to solve it.

First to my structure:
My mailcow is located at "/opt/containers/mailcow".
"getmail" is located at "/opt/containers/getmail".

My mailcow-override-file looks like this:

version: '2.1'

services:
    dovecot-mailcow:
      networks:
        - getmail

networks:
  getmail:
    external: true

And my getmail-compose-file looks like this:

version: '2.4'

services:
    getmail:
      image: cb/getmail
      container_name: getmail
      build: ./Dockerfiles
      volumes:
        - ./conf/settings.ini:/app/settings.ini:ro
        - /etc/localtime:/etc/localtime:ro
        - /etc/timezone:/etc/timezone:ro
      restart: always
      mem_limit: 100m
      mem_reservation: 20m
      labels:
        - "com.centurylinklabs.watchtower.enable=false"
      networks:
        getmail:

networks:
    getmail:
      external: false
      name: getmail
      driver: bridge
      driver_opts:
        com.docker.network.bridge.name: br-getmail

The error message in the logs looks like this - for every thread:

2021-02-16 09:23:47,975 - Thread-XYZ - ERROR: LMTP deliver (LMTP-Server is not reachable): [Errno -3] Try again
2021-02-16 09:23:47,976 - Thread-XYZ - INFO: LMTP deliver: start -- LMTP host: dovecot-mailcow:24

Normally I start getmail before mailcow. getmail creates the network and then I start mailcow.
But, dovecot does not join the getmail-network. This is my theory why it is not working - I do not know if this is correct.
I then added the dovecot container manually (via Portainer) to the getmail network, but the error persists.

I have no clue why suddenly it does not work.
mailcow and getmail are and the latest versions.

Can you help me with my problem?
It would be greatly appreciated. :)

Spam filtering on lmtp email

I use getmail to sync a number of imap accounts. Several of those are from hotmail using application passwords (2fa enabled).

The syncing part works, but given microsoft's poor spam filtering, it also grabs a number of spam messages daily.

It appears the way getmail works bypasses mailcow's own spam filters.

Is there a way to process sync'd email via mailcow's spam filters?

Thank you

Pending email at imap target during mailcow reboot not retrieved

Scenario.

  • Mailcow server is rebooting
  • new mail has arrived at defined imap target

Issue - Due to timing, mailcow has not finished initializing while getmail is already up and running. It polls the imap target, sees new mail and retrieves it. Because mailcow not entirely up, delivery fails. This situation arises when your email volume is low. Otherwise all are grabbed when the next message arrives.

Solution. - A single line delay added to getmail_imap2lmtp.py

Add time.sleep (30) under import time

This will introduce a 30s pause at getmail start up which should allow mailcow to fully finish loading. 30s Is probably overkill, but this will ensure getmail can fully communicate with mailcow.

Rebuild with docker-compose build to apply the change.

Container keeps rebooting (... and a little explanation how getmail works)

Hi,

Thanks for providing this great solution. I need sync to a bunch of outlook and gmail accounts to my mailcow box and getmail seems the only solution to accomplish this efficiently

Tried to implement your getmail solution. Both through the instructions in the mailcow issue, aswell as the updated version on your own repo.
For some reason the container keeps on rebooting and it appears no e-mail message are synced to my mailcow mailbox. I enabled debug for both imap as well as lmtp, but can't find any debug logs.

FYI.. I had to declare the value "imap_sync_folder" in my settings.ini to get getmail starting without any issues. I set this to " inbox". but need to sync other folders like " junk" aswell as i will use spamd and sieve rules to take care of the filtering.

[DEFAULT]
imap_hostname:     MUST_BE_DEFINED_INDIVIDUALLY
imap_port:         993
imap_username:     MUST_BE_DEFINED_INDIVIDUALLY
imap_password:     MUST_BE_DEFINED_INDIVIDUALLY
imap_debug:        True
imap_move_enable:  False
imap_move_folder:  getmail
imap_sync_folder:  inbox
lmtp_hostname:     dovecot-mailcow
lmtp_port:         24
lmtp_recipient:    [email protected]
lmtp_debug:        True

[EMAIL_outlook.com]
imap_hostname:     outlook.office365.com
imap_username:     [email protected]
imap_password:     pass

[EMAIL_outlook.de]
imap_hostname:     outlook.office365.com
imap_username:     [email protected]
imap_password:     pass
2021-01-26 22:04:03,111 - MainThread - INFO: use config file: ./settings.ini
2021-01-26 22:04:08,114 - Thread-EMAIL_outlook.com - INFO: Start Getmail - server: outlook.office365.com:993, username: [email protected]
2021-01-26 22:04:08,114 - Thread-EMAIL_outlook.de - INFO: Start Getmail - server: outlook.office365.com:993, username: [email protected]
2021-01-26 22:04:26,770 - MainThread - INFO: use config file: ./settings.ini
2021-01-26 22:04:31,773 - Thread-EMAIL_outlook.com - INFO: Start Getmail - server: outlook.office365.com:993, username: [email protected]
2021-01-26 22:04:31,774 - Thread-EMAIL_outlook.de - INFO: Start Getmail - server: outlook.office365.com:993, username: [email protected]
2021-01-26 22:04:59,411 - MainThread - INFO: use config file: ./settings.ini
2021-01-26 22:05:04,414 - Thread-EMAIL_outlook.com - INFO: Start Getmail - server: outlook.office365.com:993, username: [email protected]
2021-01-26 22:05:04,415 - Thread-EMAIL_outlook.de - INFO: Start Getmail - server: outlook.office365.com:993, username: [email protected]
2021-01-26 22:05:46,893 - MainThread - INFO: use config file: ./settings.ini
2021-01-26 22:05:51,897 - Thread-EMAIL_outlook.com - INFO: Start Getmail - server: outlook.office365.com:993, username: [email protected]
2021-01-26 22:05:51,898 - Thread-EMAIL_outlook.de - INFO: Start Getmail - server: outlook.office365.com:993, username: [email protected]
2021-01-26 22:06:11,412 - MainThread - INFO: use config file: ./settings.ini
2021-01-26 22:06:16,415 - Thread-EMAIL_outlook.com - INFO: Start Getmail - server: outlook.office365.com:993, username: [email protected]
2021-01-26 22:06:16,416 - Thread-EMAIL_outlook.de - INFO: Start Getmail - server: outlook.office365.com:993, username: [email protected]
2021-01-26 22:07:07,663 - MainThread - INFO: use config file: ./settings.ini
2021-01-26 22:07:12,666 - Thread-EMAIL_outlook.com - INFO: Start Getmail - server: outlook.office365.com:993, username: [email protected]
2021-01-26 22:07:12,667 - Thread-EMAIL_outlook.de - INFO: Start Getmail - server: outlook.office365.com:993, username: [email protected]
2021-01-26 22:07:56,549 - MainThread - INFO: use config file: ./settings.ini
2021-01-26 22:08:01,551 - Thread-EMAIL_outlook.com - INFO: Start Getmail - server: outlook.office365.com:993, username: [email protected]
2021-01-26 22:08:01,552 - Thread-EMAIL_outlook.de - INFO: Start Getmail - server: outlook.office365.com:993, username: [email protected]
2021-01-26 22:08:31,823 - MainThread - INFO: use config file: ./settings.ini
2021-01-26 22:08:36,827 - Thread-EMAIL_outlook.com - INFO: Start Getmail - server: outlook.office365.com:993, username: [email protected]
2021-01-26 22:08:36,828 - Thread-EMAIL_outlook.de - INFO: Start Getmail - server: outlook.office365.com:993, username: [email protected]

Unable to fetch mail

Hi,

I followed the discussion here (mailcow/mailcow-dockerized#1554) and your install.txt.
I have a running mailcow server for about 5 years and need to integrate Imap fetching, was earlier using fetchmail.
I see the networks connected, but the fetch doesn't work as far as I can see.
from a cli in the container (using portainer) I am able to ping dovecot-mailcow and I am able to ping the external IMAP server.

Still this is what I get in the logs (even enabled debugging):

getmail-getmail-1 | 2022-10-16 11:17:42,014 - MainThread - INFO: use config file: ./settings.ini
getmail-getmail-1 | 2022-10-16 11:24:49,337 - MainThread - INFO: Caught signal 15
getmail-getmail-1 | 2022-10-16 11:24:49,337 - MainThread - INFO: START: shutdown all IMAP connections
getmail-getmail-1 | 2022-10-16 11:24:49,337 - MainThread - INFO: END: shutdown all IMAP connections
getmail-getmail-1 | 2022-10-16 11:25:07,087 - MainThread - INFO: use config file: ./settings.ini

Where to start troubleshooting?

Thanks for your help!

Problems with special character

Hi!

I'm just trying to set up your get mail solution! Actually, I do not understand why the mallow team is not integrating it. But thank you for your work!
During configuration of the mail accounts there was a problem with "%" in the password string. I changed my password, to get around the problem. Did I miss something or are special characters not escaped?

thanx
Michael

Sync and deliver multiple folders for the same account into respective folders

First off thank you for this tool, it provides a function missing from the main project.

My question is, say I want to sync Inbox, Sent, and Trash for the same account. What is the format of the .cfg file so content from each folder is delivered to the same respective folder in mailcow?

In testing, it appears everything is delivered to Inbox, regardless where it came from.

Thank you

yahoo email is not showing login success

[my_yahoo]
imap_hostname: IMAP.mail.yahoo.com
imap_username: name****@yahoo.com
imap_password: password

docker compose logs
the only log showing yahoo
getmail-1 | 2024-03-25 13:03:33,643 - Thread-my_yahoo - INFO: Start Getmail - server: IMAP.mail.yahoo.com:993, username: name****@yahoo.com, ssl: True

Please help. Thank you!!

getmail restarting every 15 seconds

Hello,
it seems that there is a problem with the getmail implementation, that is not working with the native IPv6 implementation in docker - which is used by mailcow since recently (mailcow/mailcow-dockerized@f0309c5).
The effect is, that the container restarts every about 15 seconds. I did not change anything, only updated mailcow.
Any idea how the fix this? Thank you very much for your help!

Junk mails in a different lmtp recipient

Hi,

I followed the instructions in the install.txt in order to getmail from INBOX + Junk.

It works fine on the provider side : imap_move_folder: INBOX/getmail-SPAM and imap_sync_folder: INBOX/QUARANTAINE are OK.
However, i receive all mails in the same lmtp_recipient: [email protected] (i.e. INBOX)
Is it possible to receive only Junk mails in a different lmtp_recipient like [email protected]/Junk ?

Thanks for your help,
Phil

email recovery issue with certain attached files => Special Characters in file name ? / Special encoding character file ?

Hi,

I've been using getmail for at least one year and it does the job except when I receive certain emails with files attached.
I thought this was due to the size of the attached files, but that's not the case.

So I tried to analyze this problem differently and each time there is at least one attached file (pdf or zip or pptx...) and I suspect a problem with special characters like accents i.e. é. è [blank space]... used directly in the attached filename and/or the encoding character file used : recovery most often fails with Unix LF ANSI or Macintosh CR ANSI files while it seems to work better with Unix LF UTF8.

When this hangs, I move all pending emails from the inbox to a subfolder in the ISP's webmail client, then put all emails back into the inbox except the email with the problematic attachment, and getmail works instantly!

When I replace the first unrecovered email with getmail (the one that blocked the system) in the ISP webmail inbox it blocks again !

Thank you in advance for your help in solving this problem which I think comes from character sets.

Phil

Problems with IMAP IDLE

Hi, i'm using your solution with mailcow as a lot of time without problems with an Italian ISP (aruba.it) that manage my private domain but from (araund) the 20th of february, the getmail can't sync correctly. The firts login works correctly and it gets the lastest received emails but the imap idle seems that it can't say to the daemon that a new email has arrived so it don't download nothing and i can't see the notification of the new email.
I've tried many test and i'm pretty sure that it could be an incorrect or a missing feature of the implementation of the imap idle protocol, i've also contacted the (payment) support of the ISP but without success.

The mailbox(es) in your getmail implementation remain in the status "INFO: Join infinite loop and wait for new mails, cancel with Ctrl-c" without receiving nothing. The same result occours using my private domain but also using a free and generic domain @aruba.it
I've also tried with another free email provider (eg: libero.it) and in this case getmail it works correctly. Enabling the imap debug does not have any effect and i can't see any type of error.
The same problem (obviously only for test or comparison) occours using the email in Outlook 2019, that, connected directly to the imap of the ISP, it can't see the new email reveived if it remain in inbox, changing the folder (eg: from INBOX to Draft ad back to INBOX) it "renew" the connection and it can receive the email.

I've also tried with or without the SSL but without success. I've also tried using the openssl login to the email to emulating and manual enabling the imap idle connection and apparently it seems that the feature is enabled but the only signal that i receive when an email is delivered to the mailbox is a generic "* OK Still here"

The only difference that i've find between the two previous examples aruba.it and libero.it is that aruba.it use a TLS 1.3 and libero.it a TLS 1.2

If you have an idea it would be appreciated. For now the only solution for me is to enable the sync job integrated in mailcow.

Thank you very much for your support and sorry for my bad english

Best regards

vi /opt/getmail/settings.ini, ERROR settings.ini not found!

I did these below steps and i was getting ERROR settings.ini not found! . Did i do something wrong ?

Thank you!!!

vi /opt/getmail/settings.ini (i double checked this file is there)

cd /opt/getmail

docker compose build
docker compose up -d

docker compose logs

WARN[0000] The "TZ" variable is not set. Defaulting to a blank string.
getmail-1 | 2024-03-23 20:38:58,260 - MainThread - ERROR: ERROR settings.ini not found!

Virtual environment pip WARNING

Hello Christianbur,

Thanks for your package which is very useful.

During installation i have a warning message concerning virtual environment :

command line : docker-compose built getmail
Messages :
.../...
Successfully installed pip-22.1.2
Collecting imapclient
Downloading IMAPClient-2.2.0-py2.py3-none-any.whl (179 kB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 179.2/179.2 kB 9.9 MB/s eta 0:00:00
Collecting configparser
Downloading configparser-5.2.0-py3-none-any.whl (19 kB)
Requirement already satisfied: six in /usr/lib/python3.8/site-packages (from imapclient) (1.15.0)
Installing collected packages: imapclient, configparser
Successfully installed configparser-5.2.0 imapclient-2.2.0
WARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv
.../...

I tried on another fresh installation right now => same warning !?
Tfis warning appears only during the first "built getmail"
I'm not a specialist and don't know if this warning is important.

It occurs both on OVH-VPS (Debian 11) and on self-hosting NAS (Proxmox + Debian 11)
With mailcow-dockerized last update.

Docker version : 20.10.16
Docker-compose version: 1.29.2

Motivation

Including virtual environment could prevent conflicts
I had an issue with my mailcow-dockerized installation.
Getmail was ko so maybe this new feature could avoid conflicts.

Thanks!
Phil

Installation instruction

Hello there,
I found your script, which seems very helpful for me - as I am using a mailcow myself.
Can you provide a little installation instruction, so that I can use it too?
Thank you very much!

mailcow did not connect to network-getmail

I needed to run this command: "docker network connect network-getmail mailcowdockerized_dovecot-mailcow_1" to get mailcow to connect to network-getmail (verified connection with: "docker network inspect network-getmail"

And of course then needed to make sure settings.ini specified "lmtp_hostname: mailcowdockerized_dovecot-mailcow_1" in the default section.

Everything worked then, just testing it out now.

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.