activecm / rita-blacklist Goto Github PK
View Code? Open in Web Editor NEWReal Intelligence Threat Analytics -- Blacklist Database
License: GNU General Public License v3.0
Real Intelligence Threat Analytics -- Blacklist Database
License: GNU General Public License v3.0
User should be able to pass in an io.Writer object to set log output in rita-blacklist
Enhance the current blacklisting module by finding more quality sources.
Ideas:
Criteria
Ideally, create a new file for each list using hostlist/myipms.go as a template.
For people who want to add their own blacklists, add a module (or two) that let them configure the location of the blacklist at runtime.
I say possibly two modules because it would be good to let the location be either a local path (e.g. /usr/local/blacklist.txt) or a custom URL (e.g. http://192.168.1.100/blacklist.txt). If this can be done easily in one module then awesome. If it's cleaner code to do it in two modules then that is fine too.
You are free to define the file format you expect in the blacklist, just be sure to document it. It could be a CSV that has all the fields you need in predefined columns. Be sure to allow empty values for each of the columns where values are optional.
Example file format:
domain, ip_address, country
google.com, 24.220.112.177, usa
Currently when adding a new blacklist the output is:
"Blacklist: parsed x of y lines in file."
where x is the number of lines not put into the database (for example, didn't have enough fields, was a comment or a blank line) and y is the total number of lines in the file. This is confusing and could be read as only x number of lines were added to the database.
Change to show number of lines removed and number of lines successfully added
If rita-blacklist fails during a blacklist update, it still writes to the metadatabase causing rita to think that the blacklist is up to date. This leads to false negatives due to the blacklist not being populated correctly.
This has happened to me twice. Once where the update appeared to hang for far too long so I killed it. Running again did not cause another update and there were no blacklist entries. The second time there was a segfault that I believe was caused by a network issue (my firewall blocked the update attempt).
2017/06/17 22:58:27 Updating blacklist source: myip-ms . This may take a few minutes...
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x40 pc=0x1356ce6]
goroutine 32 [running]:
github.com/ocmdev/rita-blacklist/hostlist.(*MyIpMs).downloadFile(0x18069c8, 0x158151b, 0x14, 0x0, 0x0)
$GOPATH/src/github.com/ocmdev/rita-blacklist/hostlist/myipms.go:49 +0xb6
github.com/ocmdev/rita-blacklist/hostlist.(*MyIpMs).UpdateList(0x18069c8, 0xc4203b6d20, 0x0, 0x0)
$GOPATH/src/github.com/ocmdev/rita-blacklist/hostlist/myipms.go:146 +0x58
github.com/ocmdev/rita-blacklist.(*BlackList).Init.func1(0xc4201af170, 0xc4203b6d20, 0xc4201af9c0)
$GOPATH/src/github.com/ocmdev/rita-blacklist/blacklist.go:58 +0x3d
created by github.com/ocmdev/rita-blacklist.(*BlackList).Init
$GOPATH/src/github.com/ocmdev/rita-blacklist/blacklist.go:60 +0x474
I'd suggest that the blacklist loader detects if no new entries are added (e.g. if the database is uninitialized, has 0 entries, or is the same size as before the update) and to not modify the metadatabase.
Would like to provide support for custom blacklists. I'm creating this issue to track what still needs to be done before merging staging into master.
Write an install script to download the rita-blacklist config file
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.