keltia / dmarc-cat Goto Github PK

View Code? Open in Web Editor NEW

61.0 7.0 13.0 120 KB

Small utility to decode the report sent by various email providers following the DMARC spec

License: Other

Go 97.82% Makefile 2.18%

golang dmarc-reports dmarc-parser

dmarc-cat's Introduction

README.md

Status

Summary

dmarc-cat is a small command-line utility to analyze and display in a usable manner the content of the DMARC XML reports sent by the various email providers around the globe. Should work properly on UNIX (FreeBSD, Linux, etc.) and now Windows systems.

Installation

As with many Go utilities, a simple

go get github.com/keltia/dmarc-cat

is enough to fetch, build and install. On some systems you may need to add some environment variables to enable the Go and C compilers to find the gpgme include files and libraries.

CGO_CFLAGS="-I/usr/local/include" CGO_LDFLAGS="-L/usr/local/lib" go get ...

On Windows systems, GPG support is disabled in the archive module so you don't need to compile any non-Go code and the above go get command should work directly in a Powershell window.

Linux

Arch Linux

Dmarc-cat is available on the AUR:

dmarc-cat (release package)
dmarc-cat-git (git package)

You can install it using your AUR helper of choice.

Example:

$ yay -Sy dmarc-cat

Dependencies

Aside from the standard library, I use github.com/intel/tfortools to generate tables.

go get -u github.com/intel/tfortools

It also use my own module github.com/keltia/archive to handle the various archive types.

If you use Go modules, it should all work automatically.

Usage

SYNOPSIS

dmarc-cat -hvDN [-j N] [-t type] [-S sort] [-version] <zipfile|xmlfile>

Usage of ./dmarc-cat:
  -D	Debug mode
  -N	Do not resolve IPs
  -S string
    	Sort results (default "\"Count\" \"dsc\"")
  -j int
    	Parallel jobs (default 8)
  -t string
    	File type for stdin mode
  -v	Verbose mode
  -version
    	Display version
    	
Example:

$ dmarc-cat /tmp/yahoo.com\!keltia.net\!1518912000\!1518998399.xml

Reporting by: Yahoo! Inc. — [email protected]
From 2018-02-18 01:00:00 +0100 CET to 2018-02-19 00:59:59 +0100 CET

Domain: keltia.net
Policy: p=none; dkim=r; spf=r

Reports(1):
IP            Count   From       RFrom      RDKIM   RSPF
88.191.250.24 1       keltia.net keltia.net neutral pass

Columns

The full XML grammar is available here

The report has several columns:

IP is matching IP address
Count is the number of times this IP was present
From is the From: header value
RFrom is the envelope From value
RDKIM is the result from DKIM checking
RSPF is the result from SPF checking

Supported formats

The file sent by MTAs can differ in format, some providers send zip files with both csv and XML files, some directly send compressed XML files. The archive module should support all these, please open an issue if not.

Tests

Getting close to 90% coverage.

License

This is released under the BSD 2-Clause license. See LICENSE.md.

References

Contributing

I use Git Flow for this package so please use something similar or the usual github workflow.

Fork it ( https://github.com/keltia/dmarc-cat/fork )
Checkout the develop branch (git checkout develop)
Create your feature branch (git checkout -b my-new-feature)
Commit your changes (git commit -am 'Add some feature')
Push to the branch (git push origin my-new-feature)
Create a new Pull Request

dmarc-cat's People

Contributors

Stargazers

Watchers

Forkers

rhamzeh kenmoini steakunderscore sanjaymsh yebowhatsay themactep bfabio barbuk guss77 bkahly iq-scm bitterkoekje

dmarc-cat's Issues

Names missing when resolving

When one runs dmarc-cat with many IPs, some will be missing in the report even though they were correctly resolved.

Submitted by: Michel Arbois

IP field in wrong order relativ to other values for this host

I am using dmarc-cat 0.14 installed via debian bullseye/testing 1.

I currently have a very strange error with dmarc-cat.The first column (IP) is sometimes (not always) displayed in the wrong order, the other columns are correct. The behaviour is the same, no matter from whom the report comes. It behaves the same whether dns ptr requests are active or not.

The video below shows the behaviour. The part of the IP, FROM and RFROM fields are displayed correctly, I removed them for privacy reasons.

20210728_dmarcat_randomlist_blured.mp4

'Bad Filename' if it contains a unique ID

May I draw your attention to debian bug 923551?

https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=923551

https://tools.ietf.org/html/rfc7489#section-7.2.1.1 says:

 filename = receiver "!" policy-domain "!" begin-timestamp
            "!" end-timestamp [ "!" unique-id ] "." extension

 unique-id = 1*(ALPHA / DIGIT)

-Benoît-

Feature request: Document report columns

With dmarc-cat I see IP, Count, From, RFrom, RDKIM, RSPF columns for a Rua report, but I don't exactly understand the meaning of each one; Receiver MTA? Sender MTA/SMTP? Envelope From:? RSPF=none what means?

Thank you.

bad filename on XML file

I'm getting an error while trying to make dmarc-cat parse an XML file from Google (as opposed to a ZIP) file:

anarcat@curie:Downloads(master)$ dmarc-cat 'google.com!orangeseeds.org!1548201600!1548287999.xml'
2019/01/24 15:28:14 error handling google.com!orangeseeds.org!1548201600!1548287999.xml: bad filename

I first thought it might not have liked the exclamation marks there, so I tried renaming the file:

anarcat@curie:Downloads(master)$ cp 'google.com!orangeseeds.org!1548201600!1548287999.xml' d.xml
anarcat@curie:Downloads(master)$ dmarc-cat d.xml
2019/01/24 15:28:24 error handling d.xml: bad filename

No luck there.

The README does say the usage is dmarc-cat -hvD <zipfile|xmlfile> - maybe that's an error and only ZIP files are supported? Those work correctly:

anarcat@curie:Downloads(master)$ dmarc-cat 'google.com!orangeseeds.org!1548201600!1548287999.zip'
dmarc-cat 0.9.1/j4 by Ollivier Robert

Reporting by: google.com — [email protected]
From 2019-01-22 19:00:00 -0500 EST to 2019-01-23 18:59:59 -0500 EST

[...]

Thanks for the nice tool! It makes those reports much more useful...

types.go: some struct are misinterpreted from RFC-7489

While writing the Rust port of dmarc-cat, it becomes clear that I misinterpreted some structs.

Handle gzipped XML files

outlook.com sends gzipped XML instead of plain XML or a ZIP file like protection.outlook.com!example.com!1634515200!1634601600.xml.gz

This is currently not supported:

$ dmarc-cat -v /tmp/protection.outlook.com\!example.com\!1634515200\!1634601600.xml.gz
2021/10/20 10:09:55 Analyzing /tmp/protection.outlook.com!example.com!1634515200!1634601600.xml.gz
2021/10/20 10:09:55 Error: file /tmp/protection.outlook.com!example.com!1634515200!1634601600.xml.gz:: unmarshall: XML syntax error on line 2: invalid character entity &9 (no semicolon)

These should be relatively easy to support as well.

please provide (and maintain) a manual page

Hi!

It would be nice if dmarc-cat would ship with a manpage. This is somewhat of a standard on Debian's side of things, and has been reported as a bug there. There's a suggested manpage in that bug report as well, but I have found a few issues with it.

Before going any further (e.g. a PR here), I figured I would first ask if this is something you're interested in maintaining in the long term, as it does mean duplicating documentation with the README file, or moving some of it out of there...

Thanks!

Accept files on STDIN

It would be useful if dmac-cat accepted input on STDIN so that it could be used in a pipe.

This would allow dmarc reports to be read directly from an e-mail message without having to perform busy work with temporary files.

I'd like to be able to do this:

mhstore -noverbose -outfile - | funzip | dmarc-cat

gpgme.h missing on MacOs

When attempting to install on MacOS Catalina after brew install go, the following occurs:

warwick@Warwicks-MacBook-Pro ~ % go get github.com/keltia/dmarc-cat
# github.com/proglottis/gpgme
go/src/github.com/proglottis/gpgme/data.go:4:11: fatal error: 'gpgme.h' file not found
 #include <gpgme.h>
          ^~~~~~~~~
1 error generated.

Cannot compile on Centos 7

After installating gpgme and gpgme-devel, dependencies the
go get github.com/keltia/dmarc-cat command failed:

# get github.com/keltia/dmarc-cat  
# github.com/proglottis/gpgme
go/src/github.com/proglottis/gpgme/data.go:184:12: could not determine kind of name for C.gogpgme_data_seek
go/src/github.com/proglottis/gpgme/data.go:53:53: could not determine kind of name for C.gpgme_off_t
cgo: 
gcc errors for preamble:
In file included from go/src/github.com/proglottis/gpgme/data.go:6:0:
./go_gpgme.h:15:1: error: unknown type name 'gpgme_off_t'
 extern gpgme_off_t gogpgme_data_seek(gpgme_data_t dh, gpgme_off_t offset, int whence);
 ^
./go_gpgme.h:15:55: error: unknown type name 'gpgme_off_t'
 extern gpgme_off_t gogpgme_data_seek(gpgme_data_t dh, gpgme_off_t offset, int whence);

# uname -rms                  
Linux 3.10.0-957.27.2.el7.x86_64 x86_64

# go version
go version go1.11.5 linux/amd64

# yum gpgme info
Installed Packages
Name        : gpgme
Arch        : x86_64
Version     : 1.3.2
Release     : 5.el7
Size        : 535 k
Repo        : installed
From repo   : anaconda
Summary     : GnuPG Made Easy - high level crypto API
URL         : http://www.gnupg.org/related_software/gpgme/
License     : LGPLv2+
Description : GnuPG Made Easy (GPGME) is a library designed to make access to
            : GnuPG easier for applications.  It provides a high-level crypto
            : API for encryption, decryption, signing, signature verification
            : and key management.

# yum gpgme-devel info
Name        : gpgme-devel
Arch        : x86_64
Version     : 1.3.2
Release     : 5.el7
Size        : 160 k
Repo        : installed
From repo   : base
Summary     : Development headers and libraries for gpgme
URL         : http://www.gnupg.org/related_software/gpgme/
License     : LGPLv2+
Description : Development headers and libraries for gpgme

Display the type and comment fields for a given entry.

For each Row, there is a Policy field with Type and Comment. We could display it but that require a different template, maybe in an extended or more verbose mode.

Submitted by: Michel

`-t` required in non-stdin mode

dmarc-cat currently always uses -t/fType to decide what kind of compression is in use whenever it isn't in the non-standard .zip format, even when you specify a filename.

Installation instructions in README are outdated?

~ $ go version
go version go1.19.2 linux/amd64

Following the install instructions from the README, I got this:

~ $ go get github.com/keltia/dmarc-cat
go: go.mod file not found in current directory or any parent directory.
        'go get' is no longer supported outside a module.
        To build and install a command, use 'go install' with a version,
        like 'go install example.com/cmd@latest'
        For more information, see https://golang.org/doc/go-get-install-deprecation
        or run 'go help get' or 'go help install'.

Then

~ $ go install github.com/keltia/dmarc-cat
go: 'go install' requires a version when current directory is not in a module
        Try 'go install github.com/keltia/dmarc-cat@latest' to install the latest version
~ $ go install github.com/keltia/dmarc-cat@latest
go: downloading github.com/keltia/dmarc-cat v0.15.0
go: downloading github.com/intel/tfortools v0.2.0
go: downloading github.com/keltia/archive v0.9.1
go: downloading github.com/pkg/errors v0.9.1
go: downloading github.com/stretchr/testify v1.3.0
go: downloading github.com/klauspost/compress v1.10.10
go: downloading github.com/proglottis/gpgme v0.1.1
go: downloading github.com/davecgh/go-spew v1.1.1
go: downloading github.com/pmezard/go-difflib v1.0.0
# github.com/proglottis/gpgme
.local/share/go/pkg/mod/github.com/proglottis/[email protected]/data.go:4:11: fatal error: gpgme.h: No such file or directory
    4 | // #include <gpgme.h>
      |           ^~~~~~~~~
compilation terminated.
~ $ sudo apt install libgpgme-dev
### ...
~ $ go install github.com/keltia/dmarc-cat@latest
~ $ dmarc-cat --version
dmarc-cat version 0.15.0,parallel/j8 archive/0.9.1

The missing gpgme dep is kind of on me - you did mention something about it in the README, but still - it could have been more explicit: even a just having "Dependencies" section with a list would have made me realize that I need to install something first.

IP resolving is sequential

I should move to a pool based approach and store IPs.

Handle multiple files on the command-line

Looks like it could be useful.