gourmetproject / gourmet Goto Github PK

In order to implement more sophisticated analyzers like JA3, self-signed cert detection, etc., we first need a basic TLS analyzer.

gopacket has a TLS Layer already defined, but it does not currently decode a lot of the data we care about, such as Handshake messages or ChangeCipherSpec messages. Ultimately, you should ignore it, as the maintainers are not planning to use the layers package for TCP-specific protocols anyways, and it shouldn't have been merged in the first place.

In order to implement:

1. Look at the DecodeFromBytes source code at https://github.com/google/gopacket/blob/master/layers/tls.go#L130 to get an idea for an idiomatic way in Go to take raw bytes and decode them into structs.
2. Take the full application payload of a reassembled TCP stream, which will be the payload field into a Connection object, and, if the TCP payload is TLS, decode it into structs that store more information about handshakes, cipher specs, etc.
3. Make sure we only store unencrypted data in these structs. Encrypted data is not useful.
4. See the DNS analyzer and how the Analyze function just uses layers.DNS from gopacket because gopacket already has a robust DNS decoder. You will need to write your own TLS decoder since gopacket's layers.TLS doesn't decode enough for us.

Because we are lazily reading in the entire JSON log file each time we want to add a new entry, the amount of memory we use also grows with the file.

Instead, Logger.Log(c Connection) should append a new JSON record to the file without reading the entire file in. ioutil.ReadFile(l.fileName) is bad.

Steps to fix:

1. Remove ioutil.ReadFile(l.fileName)
2. For each call to Logger.Log, append the JSON as bytes to the file
3. Make sure to close the file, use mutexes correctly, etc.

If a user only wants to log Connection objects that have at least one analyzer, and ignore everything else, they should be able to.

For example, if someone wants to write an analyzer that filters for DNS traffic that contains domains ending in .io, they should be able to configure Gourmet to only log Connections that meet this filter, and ignore/drop everything else.

To implement this:

1. Create a new config.yml option called capture_mode with three options: minimal, normal, and payloads.
2. If minimal is set, only log connections that have met one or filters for the loaded analyzers
3. If normal is set, log all connections objects and any analyzer results (default, current mode)
4. If verbose is set, log all connection objects, a base64 dump of the connection payload, and any analyzer results

Got a cool analyzer idea? Submit a pull request updating the README.md with a link to your analyzer repo!

Right now, as of writing this issue, Gourmet has no unit testing or code coverage.

For a brief overview of Go testing, here are some resources

This issue will stay open as long as there is less than 90% code coverage. If you are looking to help on this effort for Hacktoberfest, try to improve coverage by ~5-10% (or more!) before submitting a pull request.

Create CI/CD pipeline using GitHub Actions to validate pull requests

Go's built-in plugin package, part of the standard library as of Go 1.8, is very finicky with dependency versions across plugins. If a plugin is built using Go Modules, and two plugins use the same package with different versions, then the build process gets angry. This will naturally create a lot of headaches.

Hashicorp's go-plugin package uses an RPC model instead of shared object, which should (hopefully) fix this problem. Migrating is going to be a pain in the ass, but its better to do it now then later.