alphasoc / nfr Goto Github PK

We need to be able to continuously process dns.log and other files which are running on IDS sensors and appliances processing traffic. If we are using the "read" command to read a file once and process, we should establish a "monitor" command which is similar but doing the equivalent of tail -f for the file and monitoring it continuously.

In the same way we pick up Bro IP data for scoring, we need to do the same for Suricata. Take a look at #49 and #50 for pointers.

If we can process the Bro dns.log format (by reading the file from the local filesystem) we can deploy NFR onto Bro IDS sensors and Corelight appliances to submit data to our API for scoring. The schema is described here and we just need to pick up the ts, id.orig_h, query, and qtype_name values for each query.

One of arguments for /v1/key/request is platform name and version.

How do you handle versioning? Hardcoding?

as in title

I'd provide a more verbose bug report, but namescore listen just sits there silently and doesn't produce any logs.

namescore status looks really good, but we can make some improvements:

1. Capitalize "Namescore status" and "Network interface to use" to make them consistent with the rest of the lines
2. Show configuration path in the 2nd line:
   Configuration status: /etc/alphasoc/namescore.toml, OK

A question about network interface should list all interfaces and suggest the best matching one, i.e. this:
Network interface to bind with:
should look like this:

Available network interfaces: eth0, lo, docker0
A network interface to bind to [eth0]:

The codebase should pass a standard golang linter tooling without errors and warnings (no uncommented exported methods, etc.).

I'm running a daemon and it keeps adding empty lines to its log file:

[root@biuro phob0s/alphasoc]# ls -la
total 24
drwx------ 3 root root 4096 Feb 28 12:01 .
drwx------ 3 root root 4096 Feb 28 11:48 ..
drwx------ 2 root root 4096 Feb 28 11:48 backup
-rw-r----- 1 root root    3 Feb 28 12:01 follow
-rw------- 1 root root    3 Feb 28 12:01 namescore.log           <-- size 3
-rw-r----- 1 root root   87 Feb 28 11:48 namescore.toml
[root@biuro phob0s/alphasoc]# ls -la
total 24
drwx------ 3 root root 4096 Feb 28 12:01 .
drwx------ 3 root root 4096 Feb 28 11:48 ..
drwx------ 2 root root 4096 Feb 28 11:48 backup
-rw-r----- 1 root root    3 Feb 28 12:01 follow
-rw------- 1 root root    4 Feb 28 12:01 namescore.log           <-- size 4
-rw-r----- 1 root root   87 Feb 28 11:48 namescore.toml
[root@biuro phob0s/alphasoc]# ls -la
total 24
drwx------ 3 root root 4096 Feb 28 12:02 .
drwx------ 3 root root 4096 Feb 28 11:48 ..
drwx------ 2 root root 4096 Feb 28 11:48 backup
-rw-r----- 1 root root    3 Feb 28 12:02 follow
-rw------- 1 root root    5 Feb 28 12:02 namescore.log           <-- size 5
-rw-r----- 1 root root   87 Feb 28 11:48 namescore.toml
[root@biuro phob0s/alphasoc]# cat namescore.log 





[root@biuro phob0s/alphasoc]# 

When API key is set but not registered:
a) send queries to AlphaSoc
b) exit
c) any other idea?

If I remember correctly, using a promiscuous mode on an interface may be enabled for unpriviledged users by a kernel option (or was it ACL? don't remember). Please research it if you can and if my statement is true -- remove that root check.

We need a circleci.yml file to run tests and make binaries for rpm/deb packages.

Remove .'s, namescore -> Namescore, and exitting -> exiting.

INFO[04-04|20:45:18] Configuration was successfully read. 
INFO[04-04|20:45:18] DNS sniffer was created.                 iface=eth0
INFO[04-04|20:45:18] Whitelist notice                         err="open /etc/alphasoc/whitelist.toml: no such file or directory"
INFO[04-04|20:45:18] namescore daemon started                 version=0.1
INFO[04-04|20:45:18] Handlers are started. 
DBUG[04-04|20:45:38] Sniffed:                                 FQDN=google.com IP=10.0.2.15
DBUG[04-04|20:45:39] Sniffed:                                 FQDN=14.8.217.172.in-addr.arpa IP=10.0.2.15
DBUG[04-04|20:45:42] Sniffed:                                 FQDN=whatever.com IP=10.0.2.15
DBUG[04-04|20:45:42] Sniffed:                                 FQDN=250.151.57.198.in-addr.arpa IP=10.0.2.15
DBUG[04-04|20:45:43] Sniffed:                                 FQDN=250.151.57.198.in-addr.arpa IP=10.0.2.15
^CINFO[04-04|20:45:48] Stopped sending queries. 
INFO[04-04|20:45:48] Stopped sending queries. 
INFO[04-04|20:45:48] Stopped retrieving alerts. 
INFO[04-04|20:45:50] namescore exitting                       signal=interrupt

Clap and riswiz log to stdout/stderr (not sure), which makes them easily runnable in a container and easily fed to journald. That's what namescore should do as well.

Make sure that default locations of config and log files make sense. This one, for example, doesn't:
https://github.com/alphasoc/namescore/blob/develop/config/config.go#L18

Currently namescore runs only on Linux, but we need to support MS Windows as well.

i.e. provide a path to a PCAP file from the command line, which then extracts the DNS query events, sends and sends them to the API for scoring. This will be useful with regard to incident response data and artifacts that consultants and security teams wish to score.

There should be introduced buffering mechanism for queries, for example when alphasoc server will be down for some time.

My proposal:
All unsuccessful attepts to send would be stored in:
/tmp/asoc/chunk_ID

Create worker gorutine, which would every time_interval check them and try to resend.

I installed namescore, didn't configure it and tried to run listen. It returned without any message, which left me unsure what happened:

master ➜  namescore git:(master) namescore
namescore is application which captures DNS requests and provides
deep analysis and alerting of suspicious events,
identifying gaps in your security controls and highlighting targeted attacks.

Usage:
  namescore [command]

Available Commands:
  listen      daemon mode
  register    Acquire and register API key.
  status      Shows status of namescore

Use "namescore [command] --help" for more information about a command.
master ➜  namescore git:(master) namescore listen                              <--- HERE
master ➜  namescore git:(master) namescore status
namescore version:        0.1
Configuration status:     config file does not exist

When run, if Namescore doesn't find a configuration file, it should prompt the user:

# /home/vagrant/go/bin/namescore
AlphaSOC Namescore Setup and API Key Generation

Select a network interface to monitor for DNS traffic
Detected interfaces:
  - lo
  - eth0

Interface to monitor: eth0

Interface to monitor: eth0
Provide your details to generate an API key and complete setup. A valid email
address is required to activate the key. By performing this request you agree to
our Terms of Service and Privacy Policy (https://www.alphasoc.com/terms-of-service)

Full name: Joey Bag O'Donuts
Organization: AlphaSOC
Email: [email protected]

Success! Check your email and click the verification link to activate your API key

Lol, namescore just created /home/phob0s/ to keep its files. Make yourself at home (on the Finesti server), phob0s! ;-)

Do you think namescore should analyze /v1/queries response

Lets say I would receive:

{
  "received": 1000,
  "accepted": 80,
  "rejected": {
    "bad_names": 20,
    "ignored_domains": 800,
    "duplicates": 100
  }
}

Should some mechanism be introduced ( eg. print warning to syslog about every 1000 rejected queries ) ? Or new milestone ?

This shall not pass:

Network interface to bind with: no idea         <-- i typed "no idea" and it went through
Provide necessary data for API key registration.
Name: ^C

We need to support sending of alerts using GELF over TCP, as per http://docs.graylog.org/en/2.4/pages/gelf.html#. Once we have the data coming into GELF (likely encoding our content within full_message or using _field1, _field2, _fieldn to communicate threat, severity, flags, and so on) we need to put together a "content pack" which describes the data and how it should be rendered in Graylog, e.g.

https://marketplace.graylog.org/addons/9e13e6bd-5439-48ac-8065-73b24e6ca027
https://github.com/colin-stubbs/graylog-cb-defense/blob/master/content_pack.json

Just FYI, other output formats will later include CEF (IBM ArcSight) and LEEF (IBM QRadar). I'll write these up another time. They are far lower priority than Graylog.

Off the top of my head we need to add these to config.yml and support processing of different files in different locations and then sending of alerts to different destinations / services.

• Data inputs (e.g. monitor this file and it's in this format)
• Data outputs (e.g. send alerts here using this format)

Regarding inputs, we have monitoring of dns or ip data but also need to add http and then we need to tell NFR which file to monitor and tail -f (or equivalent within Windows) and its format (e.g. msdns, named, bro, suricata)

Regarding outputs, the idea is that we generate alerts in formats that can easily be consumed by other software. Currently we use our own proprietary JSON format and we need to move towards CEF, LEEF, and other formats that are used by SIEM platforms (e.g. HP ArcSight, IBM QRadar, and Graylog).

I'll add additional inputs and outputs as issues here, but let's set up the config.yml so that its possible to monitor multiple files in different formats and then send alerts to different locations (e.g. send policy violations to ServiceNow, send threats high severity and above to Graylog, and send all threats to Slack).

namescore depends on libpcap which has different versions between centos and ubuntu. Linking statically solves the problem in one go (that's what clap and riswiz do).

I will fix dat

Add --version and make proper releases (with builds). We could also add homebrew formula. goreleaser might be useful, unless there are better tools.

This is our trademark, we can't afford to misspell it ;-)

Please refactor asoc.Entry into a struct instead of []string. I'd suggest using net.IP and time.Time for endpoint and timestamp information respectively.

You can then implement a function which unpacks layers.DNSQuestion into it (along with timestamp and endpoint info) and methods for JSON marshalling, whitelist matching and anything else required.

Similar to #48, we need to support local pickup and processing of DNS events from Suricata eve.json. The schema is described here and we should look for "type": "query" events and then pull timestamp, source (IP), rrname, and rrtype values to send to the API for scoring.

        server := os.Getenv("ASOC_TEST_SERVER")
	if server == "" {
		return
	}

	key := os.Getenv("ASOC_API_KEY")
	if key == "" {
		return
	}

I have such a code in tests of API functions, because I am running clap meanwhile:

ASOC_API_KEY=2d18a990c0587b2078fbab5faa84be02 ./clap --sick --mock mock.toml

Is this way that is okay for you?

as in title

As discussed today -- have a think about using goroutines and channels where it makes sense, e.g. on blocking operations like interface sniffing, API communication and on scheduled tasks, like retrieving alerts, etc.

Currently format of events is as following:
timestamp;ip;record_type;domain;severity;threat_definition;flags

However according to API there can be more that one threat_definition and severity.

Format need to be re-specified, for example separate them with colon:
timestamp;ip;record_type;domain;severity1,severity2;threat_definition1,threat_definition2;flags

Update README with a more comprehensive installation guide.

I suggest using log15 instead of a custom wrapper. We're using log15 in our software and I think it has everything we need. No need to duplicate work which was already done by someone else.

To send IP events to the API for scoring, we need to pick up the following from conn.log (full schema here): ts, id.orig_h, id.orig_p, id.resp_h, id.resp_p, proto, orig_bytes, and resp_bytes. NFR should only send IP data for Internet-bound destinations to the API for processing, so RFC 3330 IPv4 and RFC 5156 IPv6 destinations should be ignored.

Also make sure that the default API address doesn't point to localhost:

Daemon is running, but status reports it's not:

I ran the daemon, it started silently and apparently no data is pushed to AlphaSOC. I can check it in the backend, but a normal use won't be able to do it. Currently a user can't know if the daemon is working correctly or not. I suggest doing two things:

1. Add a --verbose flag to the listen command
2. Log some stats to namescore.log once every few minutes (captured packet stats, etc.)

This function is overly complicated and uses a custom FileExists() which doesn't handle errors correctly (i.e. it doesn't fail and returns true on errors other than IsNotExist). That function may be removed anyway, and instead you can just handle errors returned by os.MkdirAll()

e.g. the --help text needs to change, mentions of Namescore throughout, and the /etc/namescore/ path. Each of these need to change to NFR, as per the commit I just made to the README with the new details (3ea9579)