activecm / rita-blacklist Goto Github PK

User should be able to pass in an io.Writer object to set log output in rita-blacklist

Enhance the current blacklisting module by finding more quality sources.
Ideas:

• Lists in this readme described as malware or spam: https://github.com/notracking/hosts-blocklists/
• Lists in this readme described as malware or spam: https://github.com/StevenBlack/hosts/blob/master/readme.md
• https://mirror1.malwaredomains.com/files/justdomains
• https://zeustracker.abuse.ch/blocklist.php?download=domainblocklist
• https://github.com/A3sal0n/FalconGate/blob/master/intel-sources.md

Criteria

• Looking for lists containing either domains, IPs, or URLs
• Only looking for malicious lists, not ads, tracking, etc.

Ideally, create a new file for each list using hostlist/myipms.go as a template.

For people who want to add their own blacklists, add a module (or two) that let them configure the location of the blacklist at runtime.

I say possibly two modules because it would be good to let the location be either a local path (e.g. /usr/local/blacklist.txt) or a custom URL (e.g. http://192.168.1.100/blacklist.txt). If this can be done easily in one module then awesome. If it's cleaner code to do it in two modules then that is fine too.

You are free to define the file format you expect in the blacklist, just be sure to document it. It could be a CSV that has all the fields you need in predefined columns. Be sure to allow empty values for each of the columns where values are optional.

Example file format:

domain, ip_address, country
google.com, 24.220.112.177, usa

Currently when adding a new blacklist the output is:
"Blacklist: parsed x of y lines in file."
where x is the number of lines not put into the database (for example, didn't have enough fields, was a comment or a blank line) and y is the total number of lines in the file. This is confusing and could be read as only x number of lines were added to the database.
Change to show number of lines removed and number of lines successfully added

If rita-blacklist fails during a blacklist update, it still writes to the metadatabase causing rita to think that the blacklist is up to date. This leads to false negatives due to the blacklist not being populated correctly.

This has happened to me twice. Once where the update appeared to hang for far too long so I killed it. Running again did not cause another update and there were no blacklist entries. The second time there was a segfault that I believe was caused by a network issue (my firewall blocked the update attempt).

2017/06/17 22:58:27 Updating blacklist source:  myip-ms . This may take a few minutes...
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x40 pc=0x1356ce6]

goroutine 32 [running]:
github.com/ocmdev/rita-blacklist/hostlist.(*MyIpMs).downloadFile(0x18069c8, 0x158151b, 0x14, 0x0, 0x0)
	$GOPATH/src/github.com/ocmdev/rita-blacklist/hostlist/myipms.go:49 +0xb6
github.com/ocmdev/rita-blacklist/hostlist.(*MyIpMs).UpdateList(0x18069c8, 0xc4203b6d20, 0x0, 0x0)
	$GOPATH/src/github.com/ocmdev/rita-blacklist/hostlist/myipms.go:146 +0x58
github.com/ocmdev/rita-blacklist.(*BlackList).Init.func1(0xc4201af170, 0xc4203b6d20, 0xc4201af9c0)
	$GOPATH/src/github.com/ocmdev/rita-blacklist/blacklist.go:58 +0x3d
created by github.com/ocmdev/rita-blacklist.(*BlackList).Init
	$GOPATH/src/github.com/ocmdev/rita-blacklist/blacklist.go:60 +0x474

I'd suggest that the blacklist loader detects if no new entries are added (e.g. if the database is uninitialized, has 0 entries, or is the same size as before the update) and to not modify the metadatabase.

Would like to provide support for custom blacklists. I'm creating this issue to track what still needs to be done before merging staging into master.

Write an install script to download the rita-blacklist config file