sdgathman / pyspf Goto Github PK

Please fix url for check spf record.
Domain http://openspf.net don't work.

I believe that the example below should error out due to the '|' in the included domain before attempting a DNS lookup:

$ pyspf "v=spf1 include:a.b.com|c.d.com|e.f.com ~all" 1.1.1.1 [email protected] test.kitterman.com
result: ('permerror', 550, 'SPF Permanent Error: No valid SPF record for included domain: a.b.com|c.d.com|e.f.com: include:a.b.com|c.d.com|e.f.com') ~all

Since neither ALPHA nor DIGIT have "|", I think it's invalid per the ABNF.

So this is based on a report I got for my SPF validator because someone had an SPF record that really did this.

Imagine the following record set:

example.com IN TXT "v=spf1 include:a.example.com include:b.example.com -all
a.example.com IN TXT "v=spf1 a -all"
b.example.com IN CNAME a.example.com

What appears to happen is that pyspf follows the CNAME and thinks there are two SPF records at a.example.com. I've set up test records in kitterman.org to demonstrate the problem:

pyspf 1.1.1.1 [email protected] test.kitterman.org
result: ('permerror', 550, 'SPF Permanent Error: Two or more type TXT spf records found.') None

Any SPF test of parallel.kitterman.org raises this error because of the CNAME following issue.

Please, add option to use custom DNS server address.

I have split horizon DNS with SMTP server in my local infrastructure, thus i get local IP for A/MX records from system resolver and i cannot use this project to verify public IP with own SPF... Using custom DNS server will allow to bypass this restriction.

Hi,

I'd like to use this tool to validate an spf record. It's not really clear to me the best and easiest way to do this. It seems like a useful feature to support easily and perhaps document in the README.md.

I did look at the Kitterman code which seems to validate spf records, but didn't quite figure out their invocation yet.

If using pyspf on python2.7.5 (though think that all python2 is affected) the execution fails if spf.py finds ipaddress package and imports it. In my case ipaddress came as dependency of salt-minion

import spf
spf.check2('XX.YY.ZZ.YY', '[email protected]', 'example.com')
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/lib/python2.7/site-packages/spf.py", line 297, in check2
    receiver=receiver,timeout=timeout,verbose=verbose,querytime=querytime).check()
  File "/usr/lib/python2.7/site-packages/spf.py", line 378, in __init__
    self.set_ip(i)
  File "/usr/lib/python2.7/site-packages/spf.py", line 405, in set_ip
    self.ipaddr = ipaddress.ip_address(i)
  File "/usr/lib/python2.7/site-packages/ipaddress.py", line 163, in ip_address
    ' a unicode object?' % address)
ipaddress.AddressValueError: 'XX.YY.ZZ.YY' does not appear to be an IPv4 or IPv6 address. Did you pass in a bytes (str in Python 2) instead of a unicode object?

I tried to make it unicode

import spf
spf.check2(unicode('XX.YY.ZZ.YY'), '[email protected]', 'example.com')
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/lib/python2.7/site-packages/spf.py", line 297, in check2
    receiver=receiver,timeout=timeout,verbose=verbose,querytime=querytime).check()
  File "/usr/lib/python2.7/site-packages/spf.py", line 547, in check
    rc = self.check1(spf, self.d, 0)
  File "/usr/lib/python2.7/site-packages/spf.py", line 586, in check1
    return self.check0(spf, recursion)
  File "/usr/lib/python2.7/site-packages/spf.py", line 871, in check0
    res, code, txt = self.check1(d,arg, recursion + 1)
  File "/usr/lib/python2.7/site-packages/spf.py", line 586, in check1
    return self.check0(spf, recursion)
  File "/usr/lib/python2.7/site-packages/spf.py", line 895, in check0
    if self.cidrmatch(self.dns_a(arg,self.A), cidrlength):
  File "/usr/lib/python2.7/site-packages/spf.py", line 1348, in cidrmatch
    for netwrk in [ipaddress.ip_network(ip) for ip in ipaddrs]:
  File "/usr/lib/python2.7/site-packages/ipaddress.py", line 199, in ip_network
    ' a unicode object?' % address)
ipaddress.AddressValueError: 'XX.XX.XX.XX' does not appear to be an IPv4 or IPv6 address. Did you pass in a bytes (str in Python 2) instead of a unicode object?

note that using unicode() the reported IP address is NOT the one tested by spf.check2() but an ipv4 address from the senders domain spf record.

After trying some I found that if I use an inport like this

try:
    import ipaddr as ipadress

in /usr/lib/python2.7/site-packages/spf.py "solves" the issue and spf.check2() works as expected again

The HELO field is ignored by Big Email, and rDNS demanded instead - mostly because so few admins could manage to actually put the hostname in HELO. I never understood why.

There have been some demands to remove +all - because some admins are using it dishonestly or incorrectly. Having examples of how to use it effectively (and honestly) might help forestall +all meeting the same fate as HELO.

Greetings

This is a suggestion/question, not a bug report.

I just wonder if pyspf could not add some special (optional) handling for SPF records like this:

v=spf1 +a +mx +a:vpr60.prodius.be +all

Basically it would just ignore them.

this is the current behaviour,

(mypy3) gerard@q1900:~/pyspf$ python spf.py "v=spf1 +a +mx +a:vpr60.prodius.be +all" 70.184.247.44 [email protected] mail.gathman.org
result: ('pass', 250, 'sender SPF authorized') +all

A formally correct result, but a bit of a let down in the spirit of the standard (it's supposed to fight spam, right ? this SPF record is one among many other like that I have seen in my logs)

So the changed output could be - with an additional command switch (corresponding to a class parameter / parameter to check/check2 global functions):

(mypy3) gerard@q1900:~/pyspf$ python spf.py -k "v=spf1 +a +mx +a:vpr60.prodius.be +all" 70.184.247.44 [email protected] mail.gathman.org
result: ('fail', 550, 'SPF fail - not authorized') -all](url)

I could do a PR along these lines, but before doing it I prefer to know if it would be rejected out of hand as not standard conformant.

The default behaviour would stay of course standard conformant.

I'm well aware that this could be handled at the mail server level, but it would be easier to do it at the library level.

In addition to "v=spf1 +all", any sequence of positive matches ending in +all will always return Pass. Admins may wish to impose additional restrictive treatment on such worse than non-existent sender policies. @DanielAbrecht

Stack trace:

File "/opt/homebrew/lib/python3.10/site-packages/spf.py", line 595, in check
  spf = self.dns_spf(self.d)
File "/opt/homebrew/lib/python3.10/site-packages/spf.py", line 1164, in dns_spf
  a = [t for t in self.dns_txt(domain) if RE_SPF.match(t)]
File "/opt/homebrew/lib/python3.10/site-packages/spf.py", line 1219, in dns_txt
  if isinstance(a[0],bytes):
IndexError: list index out of range

Steps to reproduce:

- Python 3.10.4
- dnspython 2.2.1
- pyspf 2.0.14

python spf.py 0.0.0.0 [email protected] openprocurement.io

DiG 9.10.6 output:

dig openprocurement.io TXT
;; Warning: Message parser reports malformed message packet.
;; WARNING: Message has 11 extra bytes at end

As noted in the readme, cvs2git chose the wrong master. While the normal checkout/commit/push/pull of git is pretty intuitive, some of the more subversive operations are not easy to understand. I'm getting some lessons from dkg in sdgathman/pymilter#27

When using check2 in python3 using the builtin ipaddress, with i='list', if the spf record has a network of prefix 32 or more, the code crashes.
my code:
`q = spf.query(i='list', s='[email protected]', h=None, local=None, receiver=None, timeout=spf.MAX_PER_LOOKUP_TIME,
verbose=False, querytime=20)

print(q.check())`

Traceback of the error:

Traceback (most recent call last):
File "/home/shay/PycharmProjects/testing/venv/lib/python3.8/site-packages/spf.py", line 1433, in cidrmatch
self.iplist.append(network.ip)
AttributeError: 'IPv4Network' object has no attribute 'ip'

During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/home/shay/PycharmProjects/testing/sp.py", line 7, in
print(q.check())
File "/home/shay/PycharmProjects/testing/venv/lib/python3.8/site-packages/spf.py", line 600, in check
rc = self.check1(spf, self.d, 0)
File "/home/shay/PycharmProjects/testing/venv/lib/python3.8/site-packages/spf.py", line 639, in check1
return self.check0(spf, recursion)
File "/home/shay/PycharmProjects/testing/venv/lib/python3.8/site-packages/spf.py", line 924, in check0
res, code, txt = self.check1(d,arg, recursion + 1)
File "/home/shay/PycharmProjects/testing/venv/lib/python3.8/site-packages/spf.py", line 639, in check1
return self.check0(spf, recursion)
File "/home/shay/PycharmProjects/testing/venv/lib/python3.8/site-packages/spf.py", line 959, in check0
if self.cidrmatch([arg], cidrlength): break
File "/home/shay/PycharmProjects/testing/venv/lib/python3.8/site-packages/spf.py", line 1435, in cidrmatch
for netwrk in [ipaddress.IPNetwork(ip,strict=False) for ip in ipaddrs]:
File "/home/shay/PycharmProjects/testing/venv/lib/python3.8/site-packages/spf.py", line 1435, in
for netwrk in [ipaddress.IPNetwork(ip,strict=False) for ip in ipaddrs]:
AttributeError: module 'ipaddress' has no attribute 'IPNetwork'

This is because of line 1433 in spf.py:
self.iplist.append(network.ip)
which uses non existing ip attirbute for netwrok in the ipaddress python3 library.

proposed fix:
change to network.exploded

(This is a proposed solution to the issue that I encountered and described in SpamExperts/OrangeAssassin#81)

While I was trying to get OrangeAssassin running on my OSX machine I noted that there were some strange namespace issues, caused by the fact that both py3dns and dnspython were installed in parallel. py3dns installs itself in top level module DNS and dnspython in dns, which turns out to be the same directory on a default install OSX machine due to filesystem case insensitivity.

I understand that pyspf has not seen a lot of development in the last few years, and my understanding is that dnspython is by far the more widely used library these days. It also happens to be the dns library used by other parts of OrangeAssassin.

I would be happy to create a diff that changes the dns library dependency in pyspf from py3dns to dnspython (along the lines of the code present in master) and also introduces a dependency declaration on dnspython (so that pip will handle the dependency for you) if you were interested in accepting it and releasing a new pyspf. What do you think?

spf.py:1827: FutureWarning: split() requires a non-empty pattern match.
ln, reverse, delimiters = RE_ARGS.split(str)[1:4]

I would like to contribute to this project by making the code PEP8 compliant:
https://www.python.org/dev/peps/pep-0008/

I have never contributed to an open source project before so I'm hoping this is the right place to post and simple thing to start on.

I'm using pyspf to validate a list of domains include our servers in their SPF and it gives unknown instead of permerror for badly formed SPF records

I've seen this in the following instances:

• too many includes (over 10)
• multiple SPF records
• too many void records
• too many invalid policies (see example)

def check_spf(address, domain):
    """ Check a domain SPF record for the given IPs """
    chk = spf.check(i=address, s=domain, h=domain)
    if __name__ == '__main__':
        print("{}:{}".format(address, chk[0]))
    return chk[0]

check_spf("pyspf.clancs.co.uk", "10.10.10.10") // or any IP address, 

I would expect the above to return permerror, not unknown... This doesn't make it unusable but is a confusing result as the RFC doesn't support an unknown result.

Sorry this is not really an issue.. just a question.
Looking at the code that perform dns queries, am not seeing how a CNAME response is detected and handled! Could someone explain please?

Looking at the code of DNSLookup_dnspython() function, the call to dns.resolver.query() does not include raise_on_no_answer=False, without which a CNAME response would come as NoAnswer exception, no?

So how does this code catch the CNAME responses?

➜  pyspf git:(master) ✗ python3 spf.py 1.2.3.4
Traceback (most recent call last):
  File "spf.py", line 95, in DNSLookup
    from SPF.pydns import DNSLookup
  File "/path/pyspf/SPF/pydns.py", line 28
    raise spf.TempError, 'DNS: TCP Fallback error: ' + str(x)
                       ^
SyntaxError: invalid syntax

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "spf.py", line 1886, in <module>
    print(q.dns_spf(argv[0]))
  File "spf.py", line 1086, in dns_spf
    a = [t for t in self.dns_txt(domain) if RE_SPF.match(t)]
  File "spf.py", line 1135, in dns_txt
    dns_list = self.dns(domainname, rr,ignore_void=ignore_void)
  File "spf.py", line 1266, in dns
    for k, v in DNSLookup(name, qtype, self.strict, timeout):
  File "spf.py", line 97, in DNSLookup
    from SPF.dnspython import DNSLookup
  File "/path/pyspf/SPF/dnspython.py", line 27
    except dns.exception.DNSException,x:
                                     ^
SyntaxError: invalid syntax

Changed: raise spf.AmbiguityWarning, 'DNS: Truncated UDP Reply, SPF records should fit in a UDP packet, retrying TCP'

To
raise spf.AmbiguityWarning('DNS: Truncated UDP Reply, SPF records should fit in a UDP packet, retrying TCP')

To make it work on python3 on my setup.

Hi,

When DNSLookup_dnspython() is called with qtype option including lowercase letter, empty list is always returned as if there's no record, regardless of the actual server response.

dnspython accepts both upper/lowercase rdtype and internally normalizes to uppercase.
I suppose letting DNSLookup_dnspython() behave alike would be the best fix.

This one-line patch works well in my environment:

--- a/spf.py
+++ b/spf.py
@@ -120,6 +120,7 @@ def DNSLookup_pydns(name, qtype, strict=True, timeout=20):
 def DNSLookup_dnspython(name, qtype, tcpfallback=True, timeout=30):
     retVal = []
     try:
+        qtype = qtype.upper()
         # FIXME: how to disable TCP fallback in dnspython if not tcpfallback?
         dns_version = dns.version.MAJOR+dns.version.MINOR/100
         if dns_version<1.16:

Thanks.

pyspf is "Python Software Foundation License". That doesn't specify the version, but I think I should copy the latest PSL.

Several checks fails with this traceback:

: Traceback (most recent call last):
:   File "/var/policyd-spf/env/bin/policyd-spf", line 809, in <module>
:     instance_dict, configData, peruser, peruserconfigData)
:   File "/var/policyd-spf/env/bin/policyd-spf", line 623, in _spfcheck
:     mres = mfromquery.check()
:   File "/var/policyd-spf/env/lib/python3.5/site-packages/spf.py", line 591, in check
:     spf = self.dns_spf(self.d)
:   File "/var/policyd-spf/env/lib/python3.5/site-packages/spf.py", line 1160, in dns_spf
:     a = [t for t in self.dns_txt(domain) if RE_SPF.match(t)]
:   File "/var/policyd-spf/env/lib/python3.5/site-packages/spf.py", line 1210, in dns_txt
:     dns_list = self.dns(domainname, rr,ignore_void=ignore_void)
:   File "/var/policyd-spf/env/lib/python3.5/site-packages/spf.py", line 1354, in dns
:     for k, v in DNSLookup(name, qtype, self.strict, timeout):
:   File "/var/policyd-spf/env/lib/python3.5/site-packages/spf.py", line 106, in DNSLookup_pydns
:     if strict > 1:
: NameError: name 'strict' is not defined

Output:
warning: command /var/policyd-spf/env/bin/policyd-spf exit status 1
warning: premature end-of-input on private/policy-spf while reading input attribute name

More info:
I'm using this module from policyd-spf filter for postfix. Current installed version is: pyspf==2.0.13

under python3, with systems having both dnspython and python3 dns libraries installed, the SPF code fails due to how the resolver dependencies are walked.

The specific bug raised is that the snd.resolver.resolve(name,qtype,lifetime) args are incompatible with python3-dns, and cause pyspf to crash.

Something like this (as seen online elsewhere) seems to resolve the issue:

spf.py @@ -120,10 
 def DNSLookup_dnspython(name, qtype, tcpfallback=True, timeout=30):
+    import dns.version #for compatibility testing
     retVal = []
     try:
         # FIXME: how to disable TCP fallback in dnspython if not tcpfallback?
-        answers = dns.resolver.query(name, qtype, lifetime=timeout)
+        dns_version = dns.version.MAJOR+dns.version.MINOR/100
+        if dns_version<1.16:
+          answers = dns.resolver.query(name, qtype)
+        elif dns_version<2.0:
+          answers = dns.resolver.query(name, qtype, lifetime=timeout)
+        else: # >=2.0
+          answers = dns.resolver.resolve(name, qtype, lifetime=timeout)
         for rdata in answers:
             if qtype == 'A' or qtype == 'AAAA':
                 retVal.append(((name, qtype), rdata.address))

(this is working on my local system, though i have some other local patches which may or may not intertwine with this, but not listed here)

Hello

pip install pyspf download 2.0.11 version
looking at the pypi data it seems that 2.0.12t is present 2 times, maybe that's the reason and removing the extraneous version could fix it. Thanks!

Initially submitted to https://bugs.launchpad.net/pypolicyd-spf/+bug/1842005

Aug 30 06:09:43 central1 policyd-spf[29530]: Traceback (most recent call last):
Aug 30 06:09:43 central1 policyd-spf[29530]: File "/usr/bin/policyd-spf", line 809, in
Aug 30 06:09:43 central1 policyd-spf[29530]: instance_dict, configData, peruser, peruserconfigData)
Aug 30 06:09:43 central1 policyd-spf[29530]: File "/usr/bin/policyd-spf", line 623, in _spfcheck
Aug 30 06:09:43 central1 policyd-spf[29530]: mres = mfromquery.check()
Aug 30 06:09:43 central1 policyd-spf[29530]: File "/usr/lib/python3.4/site-packages/spf.py", line 547, in check
Aug 30 06:09:43 central1 policyd-spf[29530]: rc = self.check1(spf, self.d, 0)
Aug 30 06:09:43 central1 policyd-spf[29530]: File "/usr/lib/python3.4/site-packages/spf.py", line 586, in check1
Aug 30 06:09:43 central1 policyd-spf[29530]: return self.check0(spf, recursion)
Aug 30 06:09:43 central1 policyd-spf[29530]: File "/usr/lib/python3.4/site-packages/spf.py", line 906, in check0
Aug 30 06:09:43 central1 policyd-spf[29530]: if self.cidrmatch([arg], cidrlength): break
Aug 30 06:09:43 central1 policyd-spf[29530]: File "/usr/lib/python3.4/site-packages/spf.py", line 1348, in cidrmatch
Aug 30 06:09:43 central1 policyd-spf[29530]: for netwrk in [ipaddress.ip_network(ip) for ip in ipaddrs]:
Aug 30 06:09:43 central1 policyd-spf[29530]: File "/usr/lib/python3.4/site-packages/spf.py", line 1348, in
Aug 30 06:09:43 central1 policyd-spf[29530]: for netwrk in [ipaddress.ip_network(ip) for ip in ipaddrs]:
Aug 30 06:09:43 central1 policyd-spf[29530]: File "/usr/lib64/python3.4/ipaddress.py", line 84, in ip_network
Aug 30 06:09:43 central1 policyd-spf[29530]: address)
Aug 30 06:09:43 central1 policyd-spf[29530]: ValueError: '119.18.39.164\n' does not appear to be an IPv4 or IPv6 network

The txt record triggers the crash:
"v=spf1 ip4:203.143.89.146 ip4:203.27.138.188 ip4:119.18.39.164\010 include:spf.protection.outlook.com ~all"

RFC7208 requests that the "Received-SPF" header provides details about the reason the SPF result was what it was. See https://tools.ietf.org/html/rfc7208#section-9.1 . I don't see any API call in pyspf able to provide that kind of detail.

Since I am using pyspf from a MILTER filter and I can add an appropriate "Received-SPF" header to incoming emails, it would be very nice to be able to get the SPF tests details.

Thanks.

hi all,

when I test this SPF "v=spf1 v=spf1 redirect=spf.protection.outlook.com -all" with IP 40.92.1.1 in MXtool box, it return pass. However, when I do the same thing with this module, it returns Fail.

I did some extra test on this case, look like because it end with -all, so that the redirect part become useless.

Search path contains gathman.org. Gathman.org has a wildcard that returns "v=spf1 -all" for any otherwise unspecified TXT queries.

$ python spf.py aws.telekom153.com
using pydns
None
*** install dnspython
$ python -m spf aws.telekom153.com
using dnspython
v=spf1 -all
$ python -m spf aws.telekom153.com.
using dnspython
None
$ host -t txt aws.telekom153.com
aws.telekom153.com.gathman.org descriptive text "v=spf1 -all"
$ host -t txt aws.telekom153.com.
Host aws.telekom153.com. not found: 3(NXDOMAIN)

One solution may be to always append a dot for the dnspython driver.

If we're going to leave ipaddr (and possibly python2) behind, it might be a good idea to bump the version to 2.1. I'd suggest renaming the existing 2.1 branch for clarity, creating a new pysfp_2.0 branch as of the last release. Then master would be for new development towards the new 2.1 (I think the old one is dead at this point).

Github wants markdown, PyPi wants RST. My attempt to make README.md compatible with both failed. Do we maintain both README.md and README.rst ? Is there a convert ? Is there a previewer (web or otherwise) for RST? Can I reupload versions to debug README to test.pypi.org? So many questions.

When the spf record includes methods like a or mx and I do a query against an ipv6 address and the a or mx records do not include ipv6 adresses this the void counter is increased. I consider this a bug and the expected behavior would be to not increase the void counter
Example SPF:
"v=spf1 a mx include:spf.example.com ~all"

Example query:

>>> import spf
>>> spf.check2(i='2a01:aa:bb:cc::2',s='j,[email protected]',h='mx1.example.com')
('permerror', 'SPF Permanent Error: Void lookup limit of 2 exceeded')

I totally agree that methods a or mx should not be used, but on received mails I do not have control over dns

Last night our SPF daemon crashed a few hundred times because of the invalid ip4 entry (note the \010) in the following record:

$ dig +short kruisecloud.com txt
"v=spf1 +a +mx +ip4:79.124.76.140 +ip4:107.175.172.4 +ip4:107.175.172.41\010 +ip4:107.174.15.33 +ip4:107.174.15.34 +ip4:62.108.40.57 +ip4:62.108.40.58 ~all"

Scott Kitterman fixed this bug two years ago, but there hasn't been a release since =)

It'd make everything a lot easier if there was a new version we could just push out, rather than trying to patch over top of the distro package.