From a user report in our public Slack forum: When they tried to ingest a pcap, they saw this error message ensureSpawnedProcessTermination failed: Access is denied. That's from our zeek-launcher that was added in #9 . We've asked for a systeminfo dump to see their Windows version.

While the new Zeek artifacts starting with v3.2.0-dev-brim1 are generating logs ok, we've noticed a warning that shows up fairly often when processing pcaps:

1521835102.636608 warning in <command line>, line 7: calc_next_rotate(): can't parse rotation base time

We've observed that these generate events in the reporter.log which causes an integration test failure and may also confuse users. Rather than just filter them out, @nwt has confirmed we should be able to fix this instead.

As described in the Brim v0 demo video, JA3/JA3S and HASSH are well-known scripts in the Zeek community for gaining insight into encrypted traffic. As their licenses appear to allow their use in tools such as Brim, they should be included as part of the default configuration in its embedded Zeek. This way the additional logs/fields they generate will be part of the Zeek events generated when Brim processes a pcap file that was opened in the app by the user.

Scope of the implementation includes:

1. Including the licenses in the Acknowledgments that ship with the app
2. Ensuring the scripts are packaged correctly into artifacts on all supported Brim platforms
3. Ensuring Zeek logs generated on all supported Brim platforms include the expected additional logs/fields

In a Slack conversation a community user reported a possible memory leak when observing Brim's Zeek process running on Windows. I attempted to reproduce this and it does seem likely. Importing the 4 GB all.pcap we often test with into Brim v0.17.0 (so, using the Zeek artifact v3.2.0-dev-brim9) and watching in Task Manager, the reported memory use started around 80 MB and gradually climbed throughout the import to a peak of about 570 MB as it finished. Repeating the same on macOS, the Activity Monitor reported that Zeek's memory use ramped to 108.5 MB by the middle of the import process, and the usage remained flat throughout the remainder of import. While I know that Zeek maintains some state during its runs and I'm not surprised to see some ramp, clearly the slope on the Windows side is suspiciously high.

We now have the pipeline to create a Cygwin-based Zeek binary as described at https://github.com/brimsec/zeek/blob/master/brim/windows/README.md. We still need a zipfile artifact that has the expected structure as the MacOS artifact, and a script/batch file that handles the ZEEK* environment variables.

@alfred-landrum already merged #9 that's related to this. There will be additional PR’s to automatically build & put that utility into the windows Zeek ZIP artifact.

In a normal Zeek environment, adding Community ID support has the potential to be simple, since one could use the Zeek Package Manager to install the package https://github.com/corelight/zeek-community-id. However, in the Zeek artifact that's bundled with Brim, there's a couple challenges due to the fact that this package is a plugin that includes C++ code, rather than just a collection of Zeek scripts.

Known challenges include:

1. We've been minimizing the size of the artifact, such that some pieces needed to compile such code are not currently present. The zeek-config script and headers are two things we're aware of offhand that we think will be needed, but there may be others.

2. @nwt has confirmed that zeek-config didn't work when he was last working on our Windows build process, and that there's additional complications to this because the plugin framework simply doesn’t work on Windows yet.

Therefore the high-level goal is to enhance our fork of Zeek and our build system (both in general and for Windows in particular) such that a plugin like the Community ID one can be added like we currently do for other script-based Zeek Packages.

As identified in #59, Brim's Zeek port now experiences build failures on Windows. Through some digging I've confirmed that this fix addresses the issue. Since the code for Brim's Zeek port and the hash-based pointers to underlying submodule dependencies have not changed, it's a bit of a mystery to me why it started failing. The fix linked above unfortunately seems to lack any PR or other context about when they stumbled across the problem. My best guess is that maybe the tooling on the GitHub Actions Windows 2019 hosted runners may have undergone changes that caused this failure during the long period since Brim's Zeek was last built.

In a world of infinite resources, perhaps the ideal solution would be to begin using the much-newer C++ Actor Framework (CAF) that includes the fix as well as the much-newer Zeek Broker and Zeek itself that depend on it. However, assuming Brim Dev the team lacks the time or patience to revisit the complex porting work involved, I've instead investigated the minimal steps necessary to only cherry-pick the CAF fix and then wire up the necessary submodule pointers to leverage it. Because of how Zeek uses multiple levels of submodules, this was a bit of a pain to wire up, but now that I've gone through it and seen it build successfully using personal forks of the necessary repos, here I can lay out the steps so someone on the Brim team with necessary GitHub permissions can carry out the steps with forks in the Brimdata GitHub org.

The steps:

1. In the GitHub UI, do a "fetch upstream" of https://github.com/brimdata/broker. As you can see here, while this Broker fork has existed for some time, the submodule dependency in Brim's Zeek port hasn't been pointing at it. However, we'll need to start pointing at a new Broker commit that'll point to a CAF fork that contains the ultimate fix, and Brim's Broker fork is currently way behind the Broker submodule pointer to commit 5edfc77 shown here, so the fork needs to be brought current as a prerequisite.

2. In the GitHub UI, delete the fork https://github.com/brimdata/actor-framework and replace it with a fork of https://github.com/zeek/actor-framework. We established in the last step that Brim's Zeek port has been pointing at a commit in the Zeek project's Broker repo this whole time, and as shown here, the Zeek Broker at that commit has been pointing to Zeek's CAF fork at commit f7d4fc7. Therefore, we'll want to start from having a current Zeek CAF fork onto which we can cherry-pick the fix.

3. Clone the Brim CAF fork at the commit that was used to build successfully in the past (which we can do via a tag), cherry-pick the fix atop that, and commit to a branch.

export ORG="brimdata"
cd $TMPDIR
git clone -b 0.17.6 https://github.com/$ORG/actor-framework
cd actor-framework
git checkout -b inc-limits-fix
git cherry-pick a080891412b028bf89801cd8db959f3478352e89
git push origin inc-limits-fix

4. Clone the Brim Broker fork at the commit that was used to build successfully in the past (which we can do via a tag), change the CAF submodule pointer to the CAF branch with the fix, and commit to a branch.

export ORG="brimdata"
cd $TMPDIR
git clone -b v1.4.0 https://github.com/$ORG/broker.git
cd broker
git checkout -b point-at-fixed-caf
git config --file=.gitmodules submodule.caf.url https://github.com/$ORG/actor-framework.git
git config --file=.gitmodules submodule.caf.branch inc-limits-fix
git submodule update --init --remote caf
git add .gitmodules caf
git commit -m "Point at caf fork/branch with limit fix"
git push origin point-at-fixed-caf

5. Clone the Brim Zeek fork at master and change the Broker submodule pointer to the Broker branch that points to the CAF branch with the fix. The change from #59 is also included.

export ORG="brimdata"
cd $TMPDIR
git clone https://github.com/$ORG/zeek.git
cd zeek
git checkout -b point-at-own-broker
perl -pi -e 's/git:/https:/' .github/workflows/brim.yml
git config --file=.gitmodules submodule.auxil/broker.url https://github.com/$ORG/broker.git
git config --file=.gitmodules submodule.auxil/broker.branch point-at-fixed-caf
git submodule update --init --remote auxil/broker
git add .gitmodules auxil/broker .github/workflows/brim.yml
git commit -m "Point at broker fork/branch that points to caf fork/branch with limits fix"
git push origin point-at-own-broker

At this point a PR can be made for the Brim Zeek point-at-own-broker branch and it should build successfully in Actions. You can see an example of this working in my personal forks (i.e., the above commands run with export ORG="philrz") in https://github.com/philrz/zeek/pull/1 and its accompanying successful Actions Run here.

Assuming a seamless implementation of #37, adding https://github.com/corelight/zeek-community-id to our Zeek artifacts should be a trivial change to this section of the release script similar to how we install other Zeek Packages.

When testing a Brim v0.19.0 release candidate that includes the v3.2.1-brim3 Zeek artifact, an attempt to import a pcap failed with the following error:

A community user inquired on a Slack thread:

It looks like Brim/Zui doesn’t process/store the TLS Client Hello records as I don’t see the array list of Client Cipher suites. Is my assumption correct here?

Per the docs at https://docs.zeek.org/en/master/scripts/policy/protocols/ssl/ssl-log-ext.zeek.html, this is one of the additional fields that's disabled by default. I confirmed with a separate Zeek install that adding the config @load policy/protocols/ssl/ssl-log-ext to local.zeek added these additional fields.

When informed of this, the community user said:

TBH with you at any day, I’d absolutely LOVE to have this TLS client_hello cipher suites parsing support natively for the Zui instance. It is a life saver feature if you ask me. 😛

When we get to updating the bundled Zeek, we could consider enabling this by default.

This is a known limitation that is documented in print-types.zeek, but hadn't made its way over into this bug tracking system. The short summary is that the print-types script doesn't handle more than two levels of nesting. I had initially run into this with the openflow log, and here @philrz notes something similar with the smb_cmd log.

Repro is with the print-types.zeek script in Brim's fork of Zeek at commit e35de70 and zq commit 0397e42.

As of Zeek release v3.1.2, the smb_cmd.log is not generated by default. However, it can be brought to life by adding this line to local.zeek on a "stock" config:

@load policy/protocols/smb/log-cmds

Now we generate a typing config (attached):

# ZEEK_ALLOW_INIT_ERRORS=1 /usr/local/zeek-3.1.2/bin/zeek print-types.zeek /usr/local/zeek-3.1.2/share/zeek/site/local.zeek | tail +2 | jq | python -m json.tool > ~/work/zq/zeek/types-with-smb-cmd.txt

If I attempt to use it with zq commit 0397e42, it is rejected.

# zq -j ~/work/zq/zeek/types-with-smb-cmd.txt *
syntax error parsing type string

@henridf and I had discussed this at some point in the past at which point we noted that the cause of the error seems to lie within the referenced_file portion, as if I remove this it works:

# diff types-with-smb-cmd.txt types-with-smb-cmd-fixed.txt
2554,2598d2553
<                 "name": "referenced_file",
<                 "type": [
<                     {
<                         "name": "ts",
<                         "type": "time"
<                     },
<                     {
<                         "name": "uid",
<                         "type": "bstring"
<                     },
<                     {
<                         "name": "id",
<                         "type": "record conn_id"
<                     },
<                     {
<                         "name": "fuid",
<                         "type": "bstring"
<                     },
<                     {
<                         "name": "action",
<                         "type": "zenum"
<                     },
<                     {
<                         "name": "path",
<                         "type": "bstring"
<                     },
<                     {
<                         "name": "name",
<                         "type": "bstring"
<                     },
<                     {
<                         "name": "size",
<                         "type": "uint64"
<                     },
<                     {
<                         "name": "prev_name",
<                         "type": "bstring"
<                     },
<                     {
<                         "name": "times",
<                         "type": "record SMB::MACTimes"
<                     }
<                 ]
<             },
<             {

# zq -j ~/work/zq/zeek/types-with-smb-cmd-fixed.txt *
#0:record[_path:string,ts:time,peer:bstring,mem:uint64,pkts_proc:uint64,bytes_recv:uint64,pkts_dropped:uint64,pkts_link:uint64,pkt_lag:duration,events_proc:uint64,events_queued:uint64,active_tcp_conns:uint64,active_udp_conns:uint64,active_icmp_conns:uint64,tcp_conns:uint64,udp_conns:uint64,icmp_conns:uint64,timers:uint64,active_timers:uint64,files:uint64,active_files:uint64,dns_requests:uint64,active_dns_requests:uint64,reassem_tcp_size:uint64,reassem_file_size:uint64,reassem_frag_size:uint64,reassem_unknown_size:uint64,_write_ts:time]
0:[stats;1425565512.943615;bro;87;58;20905;-;-;-;410;13;0;0;1;0;0;1;36;31;0;0;0;0;0;0;0;0;1425565512.943615;]
...

The comments in print-types.zeek imply there's limits to how it deals with recursion, so perhaps the nested records like id inside referenced_file are a source of trouble.

Since this particular Zeek log isn't even enabled as default, this probably needn't be a high priority. However, since this same symptom might be lurking among other Zeek logs that aren't on by default, we may want to give it some consideration before too long, since use of zq with Zeek JSON is likely to start soon, and we expect to guide users through running print-types.zeek as needed to generate their own custom schemas.

We'd like to enhance the Zeek artifact that's bundled with Brim to add the Community ID field to conn records. This will allow for joining with other data sources, such as Suricata (brimdata/zed#1211).

Corelight is a major contributor to open source Zeek, and users of their Sensor product are likely to also be users of the Brim app. It may be helpful for the Zeek included with Brim to invoke scripts commonly found in a Corelight Sensor so the experience will be familiar to Corelight users and enriched for Zeek users in general.

Our current plan is to get JA3/HASSH into the Zeek artifact that's included with Brim (#11) and as users (at Corelight, or otherwise) tell us of other scripts they wish they'd had, we'll add them over time.

If/when we add such scripts, scope of the implementation includes:

1. Including the licenses in the Acknowledgments that ship with the app
2. Ensuring the scripts are packaged correctly into artifacts on all supported Brim platforms
3. Ensuring Zeek logs generated on all supported Brim platforms include the expected additional logs/fields

The scripts in https://github.com/brimsec/zeek/tree/master/brim/windows all still point to branches in @henridf's personal GitHub and other select gists/URLs. While we much appreciate @henridf's contribution in getting Brim to the current level of Windows support [ 😄 ] , to ensure cleanliness and stability we should migrate the referenced branches/code to be under Brim's GitHub org.

In a public Slack thread a community user recently inquired about LDAP parsing. I did some searching and learned that https://github.com/zeek/spicy-ldap is appears to be the way this is currently done in Zeek. I installed it via zkg in my local Zeek 5.0.2 and it worked fine. Therefore the next time we're assembling an updated Zeek artifact to bundle with Brimcap/Brim (which I expect will become feasible when the new Windows port is complete) we can look at including this package so the parsing will happen by default.

A community user asked in a public Slack message:

How can I get the remote location information to show up?

These are fields on Zeek's notice logs. The geoip-conn package that's enabled in the Zeek that's included with Brimcap does provide some geolocation data. However, a closer look at the Zeek docs explains that these fields are only populated in the notice logs if that https://docs.zeek.org/en/master/scripts/base/frameworks/notice/actions/add-geodata.zeek.html is loaded, and it looks like that's not loaded by default in our Zeek. This should be pretty easy to enable once we create our next round of Zeek artifacts. While we're at it, it's probably worthwhile to look around for any other non-conn logs that might have geolocation fields that are disabled by default that we can toggle.

In the meantime, as shown in the following video taken with Brim v0.31.0, it's pretty easy for users to pivot from a notice record to the corresponding conn record and see if the geo fields are populated, such as they are in this example with a wrccdc pcap.

This problem seems to be unique to Linux, as I've tested with this same pcapng file on both MacOS and Windows may times without complaint. Repro'ed via:

1. Create a scratch Linux VM in Google Cloud running Ubuntu 18.04 LTS
2. Download and unpack the Zeek v3.0.2-brim2 release artifact
3. Unpack hello.pcap.zip and attempt to process it with Zeek

# zeek/zeek -C -r hello.pcapng 
fatal error: problem with trace file hello.pcapng (invalid interface capture length 524288, bigger than maximum of 262144)

Since we're going to replace the Cygwin based windows zeek package with one build with MSYS2-MingW, we should be able to build it in much the same way as the Linux/Mac artifact. To do this, we'll need to create a Github Actions workflow for windows that prepares the msys2 build environment & creates the release artifact.