cisco / mercury Goto Github PK

Mercury: network metadata capture and analysis

License: Other

Makefile 0.97% M4 0.12% Python 6.63% C 5.10% C++ 84.38% Shell 0.49% GLSL 0.58% Emacs Lisp 0.01% Dockerfile 0.04% Cython 1.69%

packet-capture-and-analysis packet-capture network-security fingerprinting linux

mercury's Issues

pmercury UnicodeDecodeError: HTTP user-agent (\x99)

{
	'src_ip': 'x.x.x.x',
	'dst_ip': 'x.x.x.x',
	'src_port': 10630,
	'dst_port': 80,
	'protocol': 6,
	'event_start': 1295981648.965332,
	'fingerprints': {
		'http': '(474554)(485454502f312e31)(557365722d4167656e74)(486f7374)(43616368652d436f6e74726f6c3a206e6f2d6361636865)'
	},
	'http': {
		'user_agent': b'Skype\x99 5.0',
		'host': b'ui.skype.com'
	}
}
 
 File "/usr/local/bin/pmercury", line 368, in write_record
    self.out_file_pointer.write('%s\n' % json.dumps(flow_repr))
UnicodeDecodeError: 'utf-8' codec can't decode byte 0x99 in position 290: invalid start byte

Support Python 3.9

Looks like due to pypcap not supporting it, it can't be compiled with python3.9:

pynetwork/pypcap#102

Is there any plans to work around this to get python3.9 support for pmercury?

Mercury in capture mode can have high CPU utilization after suspend

After resuming from a suspend, mercury may cause high CPU utilization, in cases where it was running in capture mode before the suspend operation. Workarounds:

Don't suspend while mercury is running, or
Restart mercury; if you are using systemctl, as with the default install, you can perform 'sudo systemctl restart mercury'.

Speculation about the root cause: it is possible that the mercury code that uses the AF_PACKET ring buffer(s) shared with the kernel are getting out of sync during the suspend/resume process, and the application code keeps reading through the circular buffer in an attempt to catch up with the kernel.

Big Sur Mac Support Documentation

Any guidance compiling this for Big Sur, I get compiling errors, but I was able to compile and install Cisco Joy just fine, what are the dependencies I am missing for Mercury?

Fingerprints Database update

Up until 8 September 2020 there were frequent updates to the fingerprint database as may be seen here: https://github.com/cisco/mercury/commits/main/resources/fingerprint_db.json.gz

It looks like there were new much smaller database published in 2022 with 'tls/1' fingerprints https://github.com/cisco/mercury/blob/main/resources/resources.tgz

There were no updates since then since then as there are new devices and new OS versions. Are there any plans to publish an update to the database?

adduser not available for some distro

the makefile uses adduser --system --no-create-home --group mercury to add a new user mercury and create a same-named group.

adduser is an actual alias for useradd and may not be available in some linux distros (eg. OpenSUSE). This will cause make install to fail.

The equivalent command should be useradd --system --user-group mercury in these distros.

Definition update (what is the functional difference between "mercury" and "pmercury")?

It is not clear (at least to me) after looking at the output from both tools what the major difference is, can you elaborate in the documentation please?

Extracting payload hash from network traffic

I was wondering whether it would be possible to extract payload or the payload hash from network traffic along with the fingerprints using mercury. Are there any options for it? We can do it with tcpdump but it does not give fingerprints. Any pointers will be helpful. Thanks.

"Null/Loopback" protocol captures not supported in "pmercury" or "mercury"

When capturing on a tunnel interface (at least on a MAC), the L2 header information is set to Null (more details below)...

https://wiki.wireshark.org/NullLoopback

See example below...
"null": {
"null.family": "2"
}

% tshark -T json -i utun0
Capturing on 'USB 10/100/1000 LAN: en7'
[
  {
    "_index": "packets-2021-05-12",
    "_type": "doc",
    "_score": null,
    "_source": {
      "layers": {
        "frame": {
          "frame.interface_id": "0",
          "frame.interface_id_tree": {
            "frame.interface_name": "en7",
            "frame.interface_description": "USB 10/100/1000 LAN"
          },
          "frame.encap_type": "1",
          "frame.time": "May 12, 2021 11:54:07.695295000 MST",
          "frame.offset_shift": "0.000000000",
          "frame.time_epoch": "1620845647.695295000",
          "frame.time_delta": "0.000000000",
          "frame.time_delta_displayed": "0.000000000",
          "frame.time_relative": "0.000000000",
          "frame.number": "1",
          "frame.len": "1514",
          "frame.cap_len": "1514",
          "frame.marked": "0",
          "frame.ignored": "0",
          "frame.protocols": "null:ip:tcp:data"
        },
        "null": {
          "null.family": "2"
        },
        "ip": {
          "ip.version": "4",
          "ip.hdr_len": "20",
          "ip.dsfield": "0x00000002",
          "ip.dsfield_tree": {
            "ip.dsfield.dscp": "0",
            "ip.dsfield.ecn": "2"
          },
          "ip.len": "1500",
          "ip.id": "0x00006391",
          "ip.flags": "0x00000040",
          "ip.flags_tree": {
            "ip.flags.rb": "0",
            "ip.flags.df": "1",
            "ip.flags.mf": "0"
          },
          ...

[Patch] Support for overriding default compilers

Summary

I have been working on building Mercury for Amazon Linux 2 (AL2). AL2 has GCC, but is an older version that doesn't support C++17 standards. This requires me to install GCC10 instead:

yum install gcc10 gcc10-c++

The problem that emerges is that GCC10 for AL2 (and likely other RHEL based distros) doesn't install GCC10 as /usr/bin/gcc, but rather /usr/bin/gcc10-gcc (also applies to g++) as well.

The solution for this is to override CC, CPP, and CXX during make, however there are a handful of places in checked in Makefiles which assumes uses g++ directly rather than the CXX variable.

make CC=/usr/bin/gcc10-gcc CPP=/usr/bin/gcc10-cpp CXX=/usr/bin/gcc10-c++ V=s

Solution

I have include the patch for updating Makefiles to use the build time compiler and not static paths. Also uploaded here.

diff --git a/src/Makefile.in b/src/Makefile.in
index 85c1bb2..9154890 100644
--- a/src/Makefile.in
+++ b/src/Makefile.in
@@ -225,7 +225,7 @@ libmerc_driver:
 #
 
 intercept.so: intercept.cc libmerc.a
-	g++ -std=c++17 -Wall -Wno-narrowing intercept.cc libmerc/pkt_proc.cc -D_GNU_SOURCE -I/usr/include/nspr/ -fPIC -shared -lssl -lnspr4 -lgnutls libmerc/libmerc.a -o intercept.so
+	$(CXX) -std=c++17 -Wall -Wno-narrowing intercept.cc libmerc/pkt_proc.cc -D_GNU_SOURCE -I/usr/include/nspr/ -fPIC -shared -lssl -lnspr4 -lgnutls libmerc/libmerc.a -o intercept.so
 
 # special targets
 #
diff --git a/src/libmerc/lctrie/Makefile b/src/libmerc/lctrie/Makefile
index bbc45e7..5bc2b5a 100644
--- a/src/libmerc/lctrie/Makefile
+++ b/src/libmerc/lctrie/Makefile
@@ -12,7 +12,6 @@ clean:
 	rm -f lctrie_test
 	rm -f liblctrie.a
 
-CC     = g++
 CFLAGS = -g -ggdb -std=c++11 -Wall -O3 -fPIC # -Wno-sign-compare
 LDFLAGS = -g -ggdb -O3
 LDLIBS =
@@ -23,7 +22,7 @@ DEPDIR := .d
 $(shell mkdir -p $(DEPDIR) >/dev/null)
 DEPFLAGS = -MT $@ -MMD -MP -MF $(DEPDIR)/$*.Td
 
-COMPILE.c = $(CC) $(DEPFLAGS) $(CFLAGS) $(CPPFLAGS) $(TARGET_ARCH) -c
+COMPILE.c = $(CXX) $(DEPFLAGS) $(CFLAGS) $(CPPFLAGS) $(TARGET_ARCH) -c
 COMPILE.cc = $(CXX) $(DEPFLAGS) $(CXXFLAGS) $(CPPFLAGS) $(TARGET_ARCH) -c
 POSTCOMPILE = mv -f $(DEPDIR)/$*.Td $(DEPDIR)/$*.d

Please let me know if you have any questions.

Inputs into pmercury.protocols.tls.fingerprint()

Hello,

I'm attempting to use the fingerprint function from pmercury.protocols.tls in a python script. Can more documentation be provided about the data, offset, and data_len parameters of the fingerprint functions, e.g. data type? From what I can tell, offset and data_len are integers and data is a list, but I can't tell what's in this list. Are they strings and integers? If so, how do we map what's parsed from a PCAP into inputs to the fingerprint function? Better yet, map string input fields from sources like Zeek's conn.log, ssl.log, or x509.log? Ideally, I would understand what fields and their formats the fingerprint function is appending to c and context. My goal is to pipeline input into this function, but without more documentation, I can't anticipate what the format of the data input needs to be to ensure the function doesn't output a faulty fingerprint.

buffer overflow when accessing std::vector

Hi,

While running AddressSanitizer on a real traffic I hit the following crash:

READ of size 8 at 0x6030036242c8 thread T0 (ntdp-builtin)                                                                                                                                                            
    #0 0x7f379598c744 in std::vector<os_information, std::allocator<os_information> >::data() /usr/include/c++/8/bits/stl_vector.h:1056
    #1 0x7f379597db7e in fingerprint_data::perform_analysis(char const*, char const*, unsigned short) ../../src/engine/features/mercury/dp/analysis.h:433
    #2 0x7f379597f194 in classifier::perform_analysis(char const*, char const*, char const*, unsigned short) ../../src/engine/features/mercury/dp/analysis.h:832
    #3 0x7f379597f595 in classifier::analyze_fingerprint_and_destination_context(fingerprint const&, destination_context const&, analysis_result&) ../../src/engine/features/mercury/dp/analysis.h:842
    #4 0x7f3795988e89 in do_analysis::operator()(tls_client_hello&) ../../src/engine/features/mercury/dp/pkt_proc.cc:716
    #5 0x7f3795993701 in bool std::__invoke_impl<bool, do_analysis, tls_client_hello&>(std::__invoke_other, do_analysis&&, tls_client_hello&) /usr/include/c++/8/bits/invoke.h:60
    #6 0x7f379598e8c2 in std::__invoke_result<do_analysis, tls_client_hello&>::type std::__invoke<do_analysis, tls_client_hello&>(do_analysis&&, tls_client_hello&) /usr/include/c++/8/bits/invoke.h:95
    #7 0x7f379598ae78 in std::__detail::__variant::__gen_vtable_impl<std::__detail::__variant::_Multi_array<bool (*)(do_analysis&&, std::variant<std::monostate, http_request, http_response, tls_client_hello, tls_server_hello_and_certificate, ssh_init_packet, ssh_kex_init, smtp_client, smtp_server, unknown_initial_packet>&)>, std::tuple<std::variant<std::monostate, http_request, http_response, tls_client_hello, tls_server_hello_and_certificate, ssh_init_packet, ssh_kex_init, smtp_client, smtp_server, unknown_initial_packet>&>, std::integer_sequence<unsigned long, 3ul> >::__visit_invoke(do_analysis&&, std::variant<std::monostate, http_request, http_response, tls_client_hello, tls_server_hello_and_certificate, ssh_init_packet, ssh_kex_init, smtp_client, smtp_server, unknown_initial_packet>&) /usr/include/c++/8/variant:830
    #8 0x7f379598b0f7 in decltype(auto) std::visit<do_analysis, std::variant<std::monostate, http_request, http_response, tls_client_hello, tls_server_hello_and_certificate, ssh_init_packet, ssh_kex_init, smtp_client, smtp_server, unknown_initial_packet>&>(do_analysis&&, std::variant<std::monostate, http_request, http_response, tls_client_hello, tls_server_hello_and_certificate, ssh_init_packet, ssh_kex_init, smtp_client, smtp_server, unknown_initial_packet>&) /usr/include/c++/8/variant:1386

Memory was allocated here:

0x6030036242c8 is located 0 bytes to the right of 24-byte region [0x6030036242b0,0x6030036242c8)
allocated by thread T1 here:                                                                                                                                                                                         
    #0 0x7f3795f0fd30 in operator new(unsigned long) (/lib/x86_64-linux-gnu/libasan.so.5+0xead30)
    #1 0x7f379585d5f3 in __gnu_cxx::new_allocator<std::vector<os_information, std::allocator<os_information> > >::allocate(unsigned long, void const*) /usr/include/c++/8/ext/new_allocator.h:111
    #2 0x7f3795855ac1 in std::allocator_traits<std::allocator<std::vector<os_information, std::allocator<os_information> > > >::allocate(std::allocator<std::vector<os_information, std::allocator<os_information> > >&, unsigned long) (../lib/libntdp.so+0x368ac1)
    #3 0x7f379584a323 in std::_Vector_base<std::vector<os_information, std::allocator<os_information> >, std::allocator<std::vector<os_information, std::allocator<os_information> > > >::_M_allocate(unsigned long) (../lib/libntdp.so+0x35d323)
    #4 0x7f379583c85c in std::vector<os_information, std::allocator<os_information> >* std::vector<std::vector<os_information, std::allocator<os_information> >, std::allocator<std::vector<os_information, std::allocator<os_information> > > >::_M_allocate_and_copy<__gnu_cxx::__normal_iterator<std::vector<os_information, std::allocator<os_information> > const*, std::vector<std::vector<os_information, std::allocator<os_information> >, std::allocator<std::vector<os_information, std::allocator<os_information> > > > > >(unsigned long, __gnu_cxx::__normal_iterator<std::vector<os_information, std::allocator<os_information> > const*, std::vector<std::vector<os_information, std::allocator<os_information> >, std::allocator<std::vector<os_information, std::allocator<os_information> > > > >, __gnu_cxx::__normal_iterator<std::vector<os_information, std::allocator<os_information> > const*, std::vector<std::vector<os_information, std::allocator<os_information> >, std::allocator<std::vector<os_information, std::allocator<os_information> > > > >) (../lib/libntdp.so+0x34f85c)
    #5 0x7f379582b67d in std::vector<std::vector<os_information, std::allocator<os_information> >, std::allocator<std::vector<os_information, std::allocator<os_information> > > >::operator=(std::vector<std::vector<os_information, std::allocator<os_information> >, std::allocator<std::vector<os_information, std::allocator<os_information> > > > const&) (../lib/libntdp.so+0x33e67d)
    #6 0x7f379581b8a3 in fingerprint_data::operator=(fingerprint_data const&) (../lib/libntdp.so+0x32e8a3)
    #7 0x7f379581e763 in classifier::process_fp_db_line(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >&, float, float, bool) (../lib/libntdp.so+0x331763)
    #8 0x7f379581fe3a in classifier::classifier(encrypted_compressed_archive&, float, float, bool) (../lib/libntdp.so+0x332e3a)
    #9 0x7f379580ef4b in analysis_init_from_archive(int, char const*, unsigned char const*, enc_key_type, float, float, bool) ../../src/engine/features/mercury/dp/analysis.cc:49
    #10 0x7f379589e092 in mercury::mercury(libmerc_config const*, int) ../../src/engine/features/mercury/dp/pkt_proc.h:42
    #11 0x7f3795893bbb in mercury_init ../../src/engine/features/mercury/dp/libmerc.cc:49

Seems like the issue hits here:
os_info_data = process_os_info_vector[index_max].data();

where index_max is hitting just at the end:
0x6030036242c8 is located 0 bytes to the right of 24-byte region

dns_name::parse() can cause a segmentation fault in 2.3.0

This issue only occurs when --dns-json is in effect, and then only in rare circumstances. Fix is in progress.

concern regarding TLS extension sorting

I came across this today:

The newer one, "tls/1", sorts the extensions into lexicographic order, to compensate for the randomization of those fields introduced by some TLS clients.

https://github.com/cisco/mercury/blob/main/doc/npf.md#tls

which is concerning. some servers, such as android.googleapis.com, are sensitive to the extension order. This is not theoretical. for example, using these extensions:

&RenegotiationInfoExtension{},
&SNIExtension{},
&UtlsExtendedMasterSecretExtension{},
&SessionTicketExtension{},
&SignatureAlgorithmsExtension{},
&StatusRequestExtension{},
&ALPNExtension{},
&SupportedPointsExtension{},
&SupportedCurvesExtension{},

I get this result:

200 OK

if I swap the first two, I get this:

403 Forbidden

So if extensions are sorted, then an NPF cannot be used to reconstruct the original client hello, which would make it useless for actually making network requests. So imagine this conversation:

A: hello I cannot connect to this server, can anyone help?
B: sure just use tls/1...
A: it doesn't work...
B: ...

Also, this is confusing:

The "tls/1" fingerprint format is

   "tls/1" (TLS_Version) (TLS_Ciphersuite) [ QUIC_Extension* ]

and the older "tls" fingerprint format is

   "tls/" (TLS_Version) (TLS_Ciphersuite) ((TLS_Extension)*)

supposedly tls/1 sorts the extensions, but it also lists QUIC_Extension. does that mean only QUIC hellos can be sorted?

Unable to install latest pypi versions of pmercury on ARM platforms

Looks like the way the latest versions were uploaded to pypi don't make it compatible with other platforms like it did it earlier versions:

#21 [linux/arm64 5/5] RUN pip3 install --no-cache-dir -r requirements.txt
#21 11.99 ERROR: Could not find a version that satisfies the requirement pmercury==0.4.3.16 (from -r requirements.txt (line 2)) (from versions: 0.1.0.0, 0.1.1.0, 0.1.3.1, 0.2.0.1, 0.2.1.2, 0.2.1.3)

It would useful to be able to install this on ARM like it used to be.

reassembled PDU Parse error

Hello, in the PMercury project when TLS1.2 is parsed, the method in the readable field will be retrieved incorrectly.

Please support "Linux Netfilter NFLOG"

I captured packets through following codes on Android devices
`
iptables -A OUTPUT -p udp --dport 53 -j CONNMARK --set-mark ${userId}

iptables -A OUTPUT -m owner --uid-owner ${userId} -j CONNMARK --set-mark ${userId}

iptables -A INPUT -m connmark --mark ${userId} -j NFLOG --nflog-group ${userId}

iptables -A OUTPUT -m connmark --mark ${userId} -j NFLOG --nflog-group ${userId}

nohup tcpdump -i nflog:${userId} -w ${pcapPath} > /dev/null 2>&1 &
`
but I cannot analyse this pcap, because the packet wrapped by "Linux Netfilter NFLOG".

TLS fingerprint format

in doc/npf.md it is stated that:

The "tls/1" fingerprint format is

"tls/1" (TLS_Version) (TLS_Ciphersuite) [ QUIC_Extension* ]

and the older "tls" fingerprint format is

"tls/" (TLS_Version) (TLS_Ciphersuite) ((TLS_Extension)*)

but the mercury tool provides fingerprints of these formats:

"tls/1/(0303)...

tls/(0303)...

which means that either is the npf.md is wrong, and the format of the tls/1 should be changed to:

"tls/1/" (TLS_Version) (TLS_Ciphersuite) [ QUIC_Extension* ]

or that the resulting fingerprint from the mercury tool is incorrect and should be:

"tls/1(0303)...

I offer to fix the issue in case someone can confirm which is the incorrect one.
(i believe it is the first case, and the npf.md tls/1 format should be changed to "tls/1/" (TLS_Version) ...)

cert_analyze

cert_analyze command line
What command should I enter to extract the certificate characteristics of the pcap file？
Thank you.

Configure shouldn't be doing installs

If you just run ./configure as a regular user it fails because:

Could not install packages due to an EnvironmentError: [Errno 13] Permission denied: '/usr/local/lib64/python3.7'
Consider using the `--user` option or check the permissions.

In general, configure scripts should actually modify the system or other envornments. Maybe move that to make?

cisco / mercury Goto Github PK

mercury's Issues

Summary

Solution

Recommend Projects

Recommend Topics

Recommend Org