activecm / rita-legacy Goto Github PK

We have code to do a cross reference analysis which alerts which hosts have appeared in multiple modules. However, we do not have the ability to display the results to the CLI.

I'm doing a first time setup, and the "Importing Data, Option 2" section says "Say you have two sets of logs to analyze"

What if you don't? Can I comment the DirectoryMap entries, or should I put one entry and comment the other?

Need to update the documentation to indicate that the bro/networks.cfg needs to be configured with the user's IP address space.

Also, when importing pcaps into Bro the command should be bro -r file.pcap local to enable marking local network connections in the Bro output. This command should be tested first to verify it works as expected.

We allow the creation of databases based on dates. However, in this mode, we do not know the name of the database we wish to parse the file into until we open the file. We would like to make a record of which files were parsed into each database and when in the files collection in the meta db. We create this collection before parse time in the hopes of recovering from a failed parse. However, if we create an entry in metadb for the file before the parse, we will not know the database the file was parsed into because the time portion of the database name was determined at parse time.

The solution I have come up with is to create a separate collection for each parse attempt. Create the files collection before the parse, and if the parse succeeds update the parse attempt's table and copy the updated records into the main files collection.

Line 60 of the install script fails:

[+] Transferring files
cp: cannot stat ‘bin’: No such file or directory

I have commented that out and then line 62 fails with this error:

[+] Transferring files
cp: cannot stat ‘usr’: No such file or directory

When I comment that out it seems to then succeed, but I don't know if there are missing files that need copied over.

Thank you

Currently, the socket time-out is hard coded at 2 hours. This should be moved a configuration value since large datasets may need to be analysed. A setting of 0 disable time-outs.

Line in util.go causes seg fault when list is empty:
last := sortedIn[0]

Traced back to "newInput" variable containing an empty timestamp list:
Code

For version 0.9.2

The blacklisted module should have an output that shows what local machines connected to the blacklisted addresses, the blacklisted address' score, and the blacklisted address.

CSV can easily be imported into Microsoft Excel, LibreOffice, etc. However, column headers are necessary in order to take advantage of the most useful features in these products. RITA should display column headers in CSV format.

Move the code that handles the Safebrowsing functionality in https://github.com/ocmdev/rita/blob/master/analysis/blacklisted/blacklisted.go and write a new hostlist module here https://github.com/ocmdev/rita-blacklist/tree/master/hostlist

Since bro is not registered with apt/ dpkg on SecurityOnion, installation blows up.

I am sure I am doing something wrong. Your instructions are great now. But where do I go for help after I have everything installed?

I tried running this :
[bin]$ ./rita import INFO[0000] Starting run start_time=2016-10-06 22:55:41 ERRO[0000] aborting file parsing for this file error=Did not find a match in directory map file=/usr/local/bro/logs/2016-10-06/app_stats.22:04:24-22:53:38.log ERRO[0000] aborting file parsing for this file error=Did not find a match in directory map file=/usr/local/bro/logs/2016-10-06/communication.21:49:24-22:00:00.log ERRO[0000] aborting file parsing for this file error=Did not find a match in directory map file=/usr/local/bro/logs/2016-10-06/conn-summary.21:49:29-22:00:00.log ERRO[0000] aborting file parsing for this file error=Did not find a match in directory map file=/usr/local/bro/logs/2016-10-06/conn-summary.22:00:00-22:53:38.log ERRO[0000] aborting file parsing for this file error=Did not find a match in directory map file=/usr/local/bro/logs/2016-10-06/conn.21:49:29-22:00:00.log

It first was compressed bro files, but I uncompressed them to see if I got something different.

Any video tutorials, besides John's old one?

Thank you,
Brian

• Remove anything in the readme that is covered by the install script
• Include instructions for using install script.

When a bro log contains extra fields, the parser crashes with

ERRO[0000] the log contains a field with no candidate in the data structure  error="unmatched field in log" missing_field="FIELD" path=conn.log 

Extra fields may present themselves when users run additional bro scripts through bro such as:

redef record Conn::Info += {
  orig_cc: string &optional &log;
  resp_cc: string &optional &log;
};

event connection_state_remove(c: connection)
  {
    local orig_loc = lookup_location(c$id$orig_h);
    if ( orig_loc?$country_code )
      c$conn$orig_cc = orig_loc$country_code;
    local resp_loc = lookup_location(c$id$resp_h);
    if ( resp_loc?$country_code )
      c$conn$resp_cc = resp_loc$country_code;
  }

This happens because https://github.com/ocmdev/rita/blob/master/parser/parser.go#L379 enforces more restrictions than necessary while parsing. Having enough fields to fill the struct should be sufficient. Extra data should not cause a crash.

To facilitate easier testing and evaluate performance for production deployments.

Current brainstormed options:

• Add docker files to current rita repo.
• Create new repo under ocmdev for files to live.
• Deploy on Docker Hub

Also, create a readme or add to the readme pointing to the right place and giving the command(s) that sets everything up.

For version: 0.9.2

1. The UserAgent module should have an output that allows a user to view the machines that had that UserAgent string.
2. The analyzer should also look for machines with the most unique UserAgent strings and the least unique UserAgent strings.
3. The analyzer should identify which UserAgent strings were associated with the most and least number of bytes transferred.

The hostname table maps domain names to ip addresses. This critical piece of analysis infrastructure is currently being built off of the urls table rather than the dns table. We need to base this table off of the dns table.

Bro IDS possesses an extensible architecture allowing for new code to be distributed and ran along side it. In particular, it allows plugins to define new log writers. In theory, a plugin to write Bro data to MongoDB directly could be written.

Unfortunately, there isn't great documentation on how to go about doing this.

http://supbrosup.blogspot.com/2014/09/bro-plugins.html describes the process of making a C++ function available to Bro via the plugin system.

https://github.com/0xxon/bro-postgresql is an example of a Bro plugin which writes to an SQL server.

https://github.com/bro/bro/blob/master/src/logging is the source code for the logging plugin framework. It appears to be well documented from a code perspective.

Allow hooking into the analysis command to run custom commands afterwards.

Option 1 - Config file option:

• Create a new option in the configuration file that allows specifying a shell script
• Modify the backend so that after the analysis command finishes it reads the config option and executes the script

Option 2 - Command line switch:

• Add a new command line switch & argument to the analysis command that allows specifying a shell script via command line
• Modify the backend so that after the analysis command finishes it executes the command line argument

Option 3 - Log file watcher:

• Ensure that there is a log file entry that corresponds to a successful analysis run.
• The next step would be to have a separate service that triggers when it detects the log entry. This wouldn't involve Rita source modification.

This feature is to support post processing of the analysis data. E.g. A script that uploads the analysis data to an off-site Mongo instance. This is useful for frontend deployments, but could also be utilized for alerting.

Many of the collections have the _id field enabled when they do not need it. This means that the code spends time indexing a field it doesn't need.

Logs are only written to the RitaLogDirectory when the import command is run. They should be written during other commands as well except for show-* command output.

We should document a recommended upgrade procedure. The following commands should be a reasonable attempt at updating a RITA installation.

# This removes a dependency issue caused by a third party
rm -rf $GOPATH/src/github.com/*irupsen/logrus
echo "Updating to the latest release of RITA."
# This fetches the latest code from Github
go get -u github.com/ocmdev/rita
# This will compile the latest code and install the binary into $GOPATH/bin
cd $GOPATH/src/github.com/ocmdev/rita
make install
# Using old config files in RITA will cause issues so this updates to the newest config and backs up the old one
echo "Backing up your RITA config to ~/.rita/config.yaml-orig"
mv ~/.rita/config.yaml ~/.rita/config.yaml-orig
cp $GOPATH/src/github.com/ocmdev/rita/etc/rita.yaml ~/.rita/config.yaml
echo "RITA has been updated to:" $(rita --version)
echo "You must copy any custom settings from ~/.rita/config.yaml-orig to ~/.rita/config.yaml"

The go get is redundant from make install and could be replaced by changing to the directory and running git pull. The one thing go get -u will do though is update dependencies. This may or may not be a good thing. We probably should look at vendoring again to decide if that's the right way to go.

The current implementation of the import feature uses individual Mongo insert commands for each data point that goes into the database. This area is highly threaded, but it is worth researching using bulk inserts instead.

The current proposal that would require the least messing with the current threading code (a large undertaking) is to add in a buffer of some sort that sits between the import code and the Mongo database. Instead of inserting directly into Mongo, the threads would place the data into an in-memory buffer. When the buffer is full, additional code will take that data and perform a bulk insert into Mongo.

https://github.com/ocmdev/rita/blob/staging-26-2017/parser/watch.go#L116

Currently, a thread is created for reach file we want to read from.
We need to create a fixed number of threads for reading from files.
Ideally this would be the same number of threads as was passed to the doc writer. (or evenly split the number of threads)

DNS has become a common protocol for use in covert channels. We would like to automate the discovery of these channels. This should be implemented using the research here: https://www.sans.org/reading-room/whitepapers/dns/detecting-dns-tunneling-34152

Currently, the parser needs a little cleanup for vector types which the bro dns log uses heavily.
Particularly, we need to assign a type in the parser types file to the type strings currently being matched in the parser, and we need to split the input string into a slice of data.

Bro DNS log example:

1486944317.772209	C3KlvK1CLFn9utAI87	192.168.0.112	56556	8.8.8.8	53	udp	3054	0.040023	google.com	1	C_INTERNET	1	A	0	NOERROR	F	F	T	T	0	216.58.217.14	9.000000	F
14:1486944317.852245	CTiEoV2krojvWy6Xr	192.168.0.112	38979	8.8.8.8	53	udp	10063	0.055996	14.217.58.216.in-addr.arpa	1	C_INTERNET	12	PTR	0	NOERROR	F	F	T	T	0	den03s09-in-f14.1e100.net,den03s09-in-f14.1e100.net,den03s09-in-f14.1e100.net,den03s09-in-f14.1e100.net	82723.000000,82723.000000,82723.000000,82723.000000	F

After fixing the parser, we need to convert the hostnames table to use the dns table rather than urls/http.

Finally, we need to implement the logic as described in the research paper in order to identify domains with large amounts of subdomains. In the future, we want to be able to support whitelisting domains in this process such as ad companies.

Other analysis that could be done on DNS logs include query type stats, abnormality detection (for example the qclass and z fields), and long domain analysis.

Helpful links:
https://en.wikipedia.org/wiki/Domain_Name_System
https://www.bro.org/sphinx/scripts/base/protocols/dns/main.bro.html
http://www.iana.org/assignments/dns-parameters/dns-parameters.xhtml
https://www.sans.org/reading-room/whitepapers/dns/detecting-dns-tunneling-34152

Bro collects x509 data for every certificate it sees. https://www.bro.org/sphinx/scripts/base/init-bare.bro.html#type-X509::Certificate

We can use this data to find new certificates and identify riskier certificate chains.

Connections which have been open for a very long time could be hosting command and control channels. Doing analysis on the length of connections may be worth while.

The query to get the first/last timestamps is probably timing out and causing there to be 0 beaconing results.

The following log output is characteristic of this issue.

[-] Running beacon analysis
INFO[3779] Building collection: beacon                  
INFO[3779] Running beacon hunt                          
DEBU[3779] Looking for first connection timestamp       
DEBU[3839] Looking for last connection timestamp        
DEBU[3839] First and last timestamps found               first=0 last=0 time_elapsed=1m0.000551352s

When file logging is enabled and the rita log dir option is set, but the directory does not exist, Rita should make it.

For version: 0.9.2

Url analysis should have an output that optionally shows which machines connected to a given URL. It should also optionally shorten the URL to a given length.

Any chance this project will later have a feature to pull the Bro logs directly from Elasticsearch rather than from files? I thought a previous version of RITA did just this but I may be mistaken.

I would love to incorporate this in use cases where individuals are collecting bro logs with ELK and then use RITA to analyze them.

Can we move the Mongo table settings to a separate configuration file (e.g. ~/.rita/database-structure.yaml)? This is more of an open question/discussion that needs to be decided before implementing.

# NOTE: DO NOT CHANGE THE SETTINGS BELOW UNLESS YOU ARE FAMILIAR WITH THE CODE #
Structure:
    ConnectionTable: conn
    HttpTable: http
    DnsTable: dns
    UniqueConnectionTable: uconn
    HostTable: host

BlackListed:
    ThreadCount: 2
    ChannelSize: 1000
    BlackListTable: blacklisted
    Database: rita-blacklist

Dns:
    ExplodedDnsTable: explodedDns
    HostnamesTable: hostnames

Crossref:
    InternalTable: internXREF
    ExternalTable: externXREF
    BeaconThreshold: .7

Scanning:
    ScanThreshold: 50
    ScanTable: scan

Beacon:
    DefaultConnectionThresh: 24
    BeaconTable: beacon

Urls:
    UrlsTable: urls

UserAgent:
    UserAgentTable: useragent

MetaTables:
    FilesTable: files
    DatabasesTable: databases


# Adjusting batchsize and prefetch may help speed up certain database queries
BatchSize: 300
Prefetch: 0.33

This would allow the following improvements:

• When RITA code updates that adds or changes these fields, using an old config will cause errors. However, this means the user needs to update their config file and possibly either overwrite their modifications or manually copy them over. Having a separate file would allow changes to database structure without forcing users to update their config.
• This would declutter the user's config file and make it less confusing to edit.

Update commands like "show beacons" to allow the user to dump results into static HTML files. (No web server... just static files.)

Perhaps there should be a flag to control the name of the directory the html files are placed in. Or, perhaps the program should name the folder after the databases being ran.

When UseDates is set, it seems that Rita chooses or assumes a timezone which to split the datasets on.

A TimeZone config value would be useful to allow localization.

Logs statements can currently disappear if the logger is used with a level other than

log.DebugLevel: logPath + "/debug-" + time.Now().Format(util.TimeFormat) + ".log",
log.InfoLevel:  logPath + "/info-" + time.Now().Format(util.TimeFormat) + ".log",
log.WarnLevel:  logPath + "/warn-" + time.Now().Format(util.TimeFormat) + ".log",
log.ErrorLevel: logPath + "/error-" + time.Now().Format(util.TimeFormat) + ".log",

Adding additional levels, making sure time.Now() is consistent across the different files, and separating the logs into subfolders will help troubleshoot crashes with RITA.

Currently, there is an externally referenced image in the HTML report.
https://github.com/ocmdev/rita/blob/dev/reporting/templates/templates.go#L21
https://github.com/ocmdev/rita/blob/dev/reporting/templates/templates.go#L50

This needs to be embedded in the generated file so that the HTML report is self-contained. These lines should be changed to:

<img src="data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAFoAAAAtCAMAAAAZUYxJAAAABGdBTUEAALGPC/xhBQAAAAFzUkdCAK7OHOkAAAIBUExURQAAACEJAP10APtzAP91AA0AACIhIP94AP////95AAMAAB4eHQkAABcDABsaGhAQEBYWFcxcAHgAAP+GHP98ACAAAAsLCt0AAAYGBrazsa+trBsDAP+tZgMDA4qKiv7CjMXFxeLi4rGwsNzb2xIDAO9tAI+Pj/v7+9liAP+ZQNFeAMdaAH59fP/z4/+dQv/66nQAAOzs7MFXAZiYmNjX1tRgAP/Hkv/btmYAAMrJyaSko9LS0q5OAF5dXeFmAKVLBLxUAP/WsTwAAJOTk4g8Aff3928AAHQyAN1kAPVwAJdEA342AISDg25ubf+ya7i4t80AAOtrAJ6cnOjo6KlLAJ1FAGVlZTQWBEZFRZA+ALZRAKmpqVAmCP/u2fuBG/3Srl0AAHRzc5UAAIEAANcAAOdpAKFJAvuVPjk5Oc3NzfqoYlRUVNumeb+/vywAAFUAAKCgoK4AAPX19UsAAIoAALsAAJsAACAVDGgrAP7Pp6BjL91xFy8vLxkQCcQAAPHx8fbKpKOCZctnFNF8M71vLeGXWBELB0UaAHJDHPy7hX1YOZ11VMGSasGghN+5msODTfK5iOjWx9OwkikpKZ8AAKMAAOZ1FsnAt4lNGtONUp2TitrNwrJ2RLeKZdLBst6DNuawgqQAALikkv/q1YVsWGdTQ0o6K+eIOOp5Gvvp2jkeByNTNI8AAAfdSURBVEjHvVaJW1pJEi/hhdcccigSEEUOeSDiwaVyeeMJGkXwnBgPjMYzaIxJzBgn5yaTY5LMTDI5Z3dmZ3f+yq1+gGAcN5lvv536eFXdXb/6dXV1v9cA/AUiqa6WfSlW9iewICs6g7qoGhu86Ksztrg4Y6uplYAeR6C4iGaC2OocoJhv6nPYY8y0Nz6OSv/5LIr+AFt8gi0nmPFYq9Waah+HPRUVx9Sgg1rf2ICPt4NjPkc7SNodjiWYcaes1tb2+9DtUDlCIVRjgHCHO6GqRVF1F8wkkYFDzKKIUzAoRmHVqTY1S63djZZl1Ut2tZjdg36xWgXWHDakZtV9DlStMIXaWpuJmeIXllviACtg+kcYoXhqjxEIBEidQMsgdTsrpLbNwQrZEIwwbHubWMhYrYidTOCYI8QKmRS4sembymCPU9eyQnFokBWwfYMMwwhVqlCCTlbrSLjFAnbErer1iYVsH1KL3W5sdQ+IheKxBCofZsX2gwqbtVNiAWOt9U0WFEQPPqRe6kVQqhup+3FWpGZb0VdLY9D24VTCvRQrdiML2zupFqineOpemst9O80thH1VZuvy+0s9bZ+jFrChVjaTtX1msHvyfhtG2e8LMaK3FZsDOer8mZcUUI/w1GNje3Sd1sTYDCVqHWgbt4uRui/FiNuxFogbxMAxli8SrmcyhfDBEMVOJmbyaZ+gZhh1+yTWDbckMcVb8aCPWqFQgEtI0VlY1XiWGtfDhkZYhpnJYifz23iSmj2FmmEEuOKZPjGDc/Tx1K1YMgHrtuJa4Quohfa+Af44+voG2+mu++x7PjHTL6TUtPCJflYgVC8laAEghEh7P8OOAMX221t78+8pUt8oPV/a1ovqh+7S0tLzVwGWaAed36D9Bu2N0tIb+GBnpm3gVfcPtNWG6gYMYMS/zqMb3FmsrPCE3K65WH/9Qc3Fmt9e1dTUXJyphuv1F+t/g3G4g/Y22g/19bfv1NfU1N+h+hXFfqCYD/CARqDjdg57r+ATchNeV1yquP4A1d2HFRUVl2YArg9VDN2Fq/B06NLQU8TcHRp6TX1DT99S7O+XKiruvuUxM9ikjrc89jViK/PUV+Hbqr9VvXuEau1hU1XTs38CPEL7kXpof+3ju7WmpjV4hp1vKezwIcUeYncN4GNTFUrTI4qtera29g7uFVD/0jF77data9eaD3/tmO24hdSHaJ+g53Hz7GxHR/Pz583Yfd482/z41xz28SF2nwAdRen4JYd9AjclR9xK+Ps/vqLyHpx1dXX/ngD4qaGu4TuQwo8NOFDX8P5NA3bp4Psc9rvfR3kM6HjIV+Fso+ENKPPUWgDnj16v/yeAVXO5ObAPENOV60ZBAVFdOYqO43Q6D0xgy8BjLW8CyzCHGA9QU84HRYfNOp1u2AOago3U3syeQ2V2QJOdUlF4e2Q6iiwWl8RjFMfdR8FHIpfeOyOXakEjEomkuMUKtHJaKxEvcjntUqcyg1WKFDlMBoJBOeyJO6+y8guvfh4r+eJ/ChKJDOXMZ0SWfU5zoyAT/EUi0W90dm5vb25eaGnpaezBp6SkpLGnp7GxBH8lqGmXH6QPHaOmp6enpeUCRvGyub3dibJRVv0p9fYmMi7szH9/+fLlK1e+PpeTxcXFs2cXF7Pm7CeyeO7c14i+cvnnl7vz8zsLjS3bnfrTSiKjK+jcbOEF815Y2Nnd3VkoKZl/iaHz87sv6eQ/Y2MXKb/fWVjoudDY0rmxsVFWVlYs+YNtlMkym/C/F/cEh7Yyf1Jws6uLc6LX6zEd1EVlZUX4FIi+mHcUI/iM7LTjoZFrFSdPukapKTz2VP70IrKvdaVSqsRZFAq5RqlUyhU8u1ReqZUrlUczKKRyLWjliDwGV2g1Uqkm4zjGLNKZvBzwL3bB+IvVfcgNhb0mc9hiMmcCtPSTof0EnvuMFH53pFBOgi6ShCQXhonYxGpsOZwMR2GaDMMql4T92OoLJyGWLkKCMGFwYsgctwoF8GR4YhmiUSl1TBRyS2GFOP3E0GWLp6MeUm4kBlMkSIaDxMQduNKBmM1lMURIXEeIf9nlOvDDetoVmfNE4oTzkGmEx9NxLph2WcAciduikN82OVJ3TZvmTIQjwVGi8xPOFekiwQAxrxMufRAlxOS0HbhMLmJEKi+J2UwcMa+QmGV6lJgRHifpAPEP05Ripq3s5zZHjZ90iEcMxItYPg0DpcaMLOm4gbggRrxxskKMAWLChJMeLxkeJZYkFq0c1xsnSSdxmUiXjgRXCzeAz9qzEjPZOEqNWRtcGWqsjdkQdRIvzJEVC9ER4xYxOj3LOleQGIFLky4+a4MrAk5cmScMAZvNUFAQWuvoCuFMESelns5nHTCSZNAVReoYMfrJFs26y2xz2kxRMuxZDxPLFg93RRROsh4lnmkd5lBQEBGMElM6su8nQUySWOKk6wAXYPEQ7zQOeeeIiWaNpSTBJLEcpCcOIitkfZr4iZGjcO6ALC9HsCBhI47pkLDgfvGsGGOwXO4P4BE3j5aHRwPhck60ZZ7g/LoXE+YuWPV7nMNh4xY4jetJMKx3maMw6jcvS4cz8H1Irhud8CLgnxaB5tO7S/5f3ldF/n/FKUjlkaJ/EAq4+duSXqNaLd6gIoVUpBHJsamRikQKLfq0IqWCjoGU3rxyxCjRTX0ZOH/pZi5mzf/96voPSPjPP+4MrGwAAAAASUVORK5CYII=" alt="Offensive Countermeasures" style="width:90px; float:left" />

Bro collects how many bytes were sent with each connection. We can use this to aid in our detection of beacons. A beacon should have a small, regular payload.

I can cause a segfault when running html-report when either:

1. a database is imported but not analyzed
2. a database does not exist

These could be from the same root cause but I wanted to make sure I mentioned both.

In the output below the "rita" and "reporting" databases are imported but not analyzed and the "doesnoteexist" database, well, does not exist.

➜  rita git:(f5024db) ✗ ./rita show-databases
Empire
rita
reporting
➜  rita git:(f5024db) ✗ ./rita html-report
[-] Writing: $GOPATH/src/github.com/ocmdev/rita/rita-html-report/Empire
[-] Writing: $GOPATH/src/github.com/ocmdev/rita/rita-html-report/rita
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0x11c4e4a]

goroutine 1 [running]:
gopkg.in/mgo%2ev2.(*Iter).Next(0x0, 0x14a7dc0, 0xc420076980, 0x14a7dc0)
        $GOPATH/src/gopkg.in/mgo.v2/session.go:3684 +0x3a
gopkg.in/mgo%2ev2.(*Iter).All(0x0, 0x14a3980, 0xc420198980, 0x0, 0x0)
        $GOPATH/src/gopkg.in/mgo.v2/session.go:3800 +0x2b6
github.com/ocmdev/rita/reporting.printBeacons(0xc4201f4234, 0x4, 0xc420198460, 0x0, 0x0)
        $GOPATH/src/github.com/ocmdev/rita/reporting/report-beacons.go:29 +0x370
github.com/ocmdev/rita/reporting.writeDB(0xc4201f4234, 0x4, 0xc4201550c0, 0x3b, 0xc420198460, 0x0, 0x0)
        $GOPATH/src/github.com/ocmdev/rita/reporting/report.go:154 +0x291
github.com/ocmdev/rita/reporting.PrintHTML(0xc420155080, 0x3, 0x4, 0xc420198460, 0x0, 0x0)
        $GOPATH/src/github.com/ocmdev/rita/reporting/report.go:67 +0x4ab
github.com/ocmdev/rita/commands.init.4.func1(0xc42001b180, 0x0, 0xc42001b180)
        $GOPATH/src/github.com/ocmdev/rita/commands/reporting.go:31 +0x111
github.com/urfave/cli.HandleAction(0x14d0ca0, 0x1598c40, 0xc42001b180, 0xc42006c900, 0x0)
        $GOPATH/src/github.com/urfave/cli/app.go:485 +0xd4
github.com/urfave/cli.Command.Run(0x157c697, 0xb, 0x0, 0x0, 0x0, 0x0, 0x0, 0x158d40a, 0x29, 0x0, ...)
        $GOPATH/src/github.com/urfave/cli/command.go:207 +0xb6e
github.com/urfave/cli.(*App).Run(0xc420098d00, 0xc42000c280, 0x2, 0x2, 0x0, 0x0)
        $GOPATH/src/github.com/urfave/cli/app.go:250 +0x7d0
main.main()
        $GOPATH/src/github.com/ocmdev/rita/rita.go:26 +0x110

➜  rita git:(f5024db) ✗ ./rita html-report -d doesnotexist
[-] Writing: $GOPATH/src/github.com/ocmdev/rita/doesnotexist/doesnotexist
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0x11c4e4a]

goroutine 1 [running]:
gopkg.in/mgo%2ev2.(*Iter).Next(0x0, 0x14a7dc0, 0xc420182d00, 0x14a7dc0)
        $GOPATH/src/gopkg.in/mgo.v2/session.go:3684 +0x3a
gopkg.in/mgo%2ev2.(*Iter).All(0x0, 0x14a3980, 0xc420185540, 0x0, 0x0)
        $GOPATH/src/gopkg.in/mgo.v2/session.go:3800 +0x2b6
github.com/ocmdev/rita/reporting.printBeacons(0x7fff5fbffa06, 0xc, 0xc42015f540, 0x0, 0x0)
        $GOPATH/src/github.com/ocmdev/rita/reporting/report-beacons.go:29 +0x370
github.com/ocmdev/rita/reporting.writeDB(0x7fff5fbffa06, 0xc, 0xc420153540, 0x37, 0xc42015f540, 0xc, 0xc420186c90)
        $GOPATH/src/github.com/ocmdev/rita/reporting/report.go:154 +0x291
github.com/ocmdev/rita/reporting.PrintHTML(0xc420186c90, 0x1, 0x1, 0xc42015f540, 0x0, 0x0)
        $GOPATH/src/github.com/ocmdev/rita/reporting/report.go:67 +0x4ab
github.com/ocmdev/rita/commands.init.4.func1(0xc42001b180, 0x0, 0xc42001b180)
        $GOPATH/src/github.com/ocmdev/rita/commands/reporting.go:31 +0x111
github.com/urfave/cli.HandleAction(0x14d0ca0, 0x1598c40, 0xc42001b180, 0xc42006c900, 0x0)
        $GOPATH/src/github.com/urfave/cli/app.go:485 +0xd4
github.com/urfave/cli.Command.Run(0x157c697, 0xb, 0x0, 0x0, 0x0, 0x0, 0x0, 0x158d40a, 0x29, 0x0, ...)
        $GOPATH/src/github.com/urfave/cli/command.go:207 +0xb6e
github.com/urfave/cli.(*App).Run(0xc4200989c0, 0xc420010300, 0x4, 0x4, 0x0, 0x0)
        $GOPATH/src/github.com/urfave/cli/app.go:250 +0x7d0
main.main()
        $GOPATH/src/github.com/ocmdev/rita/rita.go:26 +0x110

Bro has an intel log documented here.

One notable service that integrates with this log is Critical Stack which acts as a sort of blacklist marketplace/aggregator.

The suggestion I have is reading intel.log as a blacklist source. All the matching will already be done and we will just have to report the entries in this file. Users would be responsible for configuring Bro to populate the intel.log as they see fit.

For version: 0.9.2

The scanning module is quite broken. It will find port scanning, but the output is very noisy. Specifically anything that does NAT or Proxy etc. will come up as a port scan. The module needs to be rewritten to use something like the port popularity list from nmap to score the likelihood that a machine which is seen accessing multiple ports on another machine is actually scanning.

Add in a confirmation prompt (e.g. "Are you sure you want to reset analysis on 'X' database?") when using the rita reset-analysis command. The caveat here is that if you don't specify the database it will reset all databases and the prompt needs to reflect that (e.g. "Are you sure you want to reset analysis on ALL databases?")

Similar to issue #46. And the implementation will likely be similar.

A variable is still present in the config structure for the GNU Netcat path. It is unused.

Add in a confirmation prompt (e.g. "Are you sure you want to delete 'X' database?") when using the rita delete-database command.

@bglebrun

RITA should be able to communicate with MongoDB over encrypted channels, and properly authenticate to MongoDB. Note: this should be an optional configuration option.

Not sure what is causing this error, everything else worked up until this point. Any help would be appreciated.

Rita1:~/go/bin$ ./rita show-beacons -d beacons
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0x5aa53a]

goroutine 1 [running]:
panic(0x8798e0, 0xc4200100f0)
/usr/local/go/src/runtime/panic.go:500 +0x1a1
gopkg.in/mgo%2ev2.(*Iter).Next(0x0, 0x83ac60, 0xc420062b80, 0x83ac60)
/home/XXXX/go/src/gopkg.in/mgo.v2/session.go:3684 +0x3a
gopkg.in/mgo%2ev2.(*Iter).All(0x0, 0x836ea0, 0xc4201301a0, 0x0, 0x7)
/home/XXXX/go/src/gopkg.in/mgo.v2/session.go:3800 +0x27f
github.com/ocmdev/rita/commands.showBeacons(0xc4200852c0, 0x0, 0xc4200852c0)
/home/XXXX/go/src/github.com/ocmdev/rita/commands/show-beacons.go:48 +0x22a
github.com/urfave/cli.HandleAction(0x863360, 0x938ed8, 0xc4200852c0, 0xc420054900, 0x0)
/home/XXXX/go/src/github.com/urfave/cli/app.go:485 +0xd4
github.com/urfave/cli.Command.Run(0x8fae36, 0xc, 0x0, 0x0, 0x0, 0x0, 0x0, 0x908ca2, 0x28, 0x0, ...)
/home/XXXX/go/src/github.com/urfave/cli/command.go:207 +0xb96
github.com/urfave/cli.(*App).Run(0xc42007f380, 0xc42000c340, 0x4, 0x4, 0x0, 0x0)
/home/XXXX/go/src/github.com/urfave/cli/app.go:250 +0x812
main.main()
/home/XXXX/go/src/github.com/ocmdev/rita/rita.go:26 +0xf0

A long time ago we added a new module called tbd... because we didn't know what to call it. It's now known as beacon (or beacons or beaconing...) analysis. We should rename every instance of the word TBD to beacon(ing/s).

This line is also causing an error, is it too not needed?

What is suppose to be in the usr folder?

    cp -r usr $_RITADIR/usr

rita$ sudo ./install.sh
[sudo] password for rita:

 _ \ _ _| __ __|  \
   /   |     |   _ \
_|_\ ___|   _| _/  _\

Brought to you by the Offensive CounterMeasures

[+] Transferring files
cp: cannot stat ‘usr’: No such file or directory

Thank you,
Brian

Rita version: v1.0.0-alpha-126-gc13c02d

rita show-beacons -d db-does-not-exist
DEBU[0000] entering                                      function=isBuilt module=meta package=database
INFO[0000] Failed aggregation: (Source collection: beacon doesn't exist)
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0x5c2e5a]

goroutine 1 [running]:
panic(0x8b3120, 0xc420010100)
        /usr/local/go/src/runtime/panic.go:500 +0x1a1
gopkg.in/mgo%2ev2.(*Iter).Next(0x0, 0x8710a0, 0xc4201ee180, 0x8710a0)
        /home/bhisht/go/src/gopkg.in/mgo.v2/session.go:3684 +0x3a
gopkg.in/mgo%2ev2.(*Iter).All(0x0, 0x86d1e0, 0xc4201ca740, 0x0, 0x11)
        /home/bhisht/go/src/gopkg.in/mgo.v2/session.go:3800 +0x27f
github.com/ocmdev/rita/commands.showBeacons(0xc4200c3540, 0x0, 0xc4200c3540)
        /home/bhisht/go/src/github.com/ocmdev/rita/commands/show-beacons.go:45 +0x26a
github.com/urfave/cli.HandleAction(0x89be20, 0x980560, 0xc4200c3540, 0xc42008c900, 0x0)
        /home/bhisht/go/src/github.com/urfave/cli/app.go:485 +0xd4
github.com/urfave/cli.Command.Run(0x93ce6e, 0xc, 0x0, 0x0, 0x0, 0x0, 0x0, 0x94c4f2, 0x28, 0x0, ...)
        /home/bhisht/go/src/github.com/urfave/cli/command.go:207 +0xb92
github.com/urfave/cli.(*App).Run(0xc4200b7860, 0xc42000c180, 0x4, 0x4, 0x0, 0x0)
        /home/bhisht/go/src/github.com/urfave/cli/app.go:250 +0x812
main.main()
        /home/bhisht/go/src/github.com/ocmdev/rita/rita.go:26 +0xf0

Looks like this (and possibly other) exception just needs to be caught and handled with a user-friendly error message.