This is a traffic analysis tool complete with a Command Line Interface (CLI) developed in order to streamline the analysis of traffic captures.

At the moment of this report, Sentinel allows highly specific and precise metric measurements, such as:

• Search for possible exfiltration.
• Search for possible botnet and C&C communication.
• Volume of connection per destination.
• Hourly connection volume.

This tool was developed in Python version 3.11.2 and Poetry, which is a tool for dependency management and packaging. It allows for the declaration of libraries in a project and it will manage (install/update) this dependencies automatically. Poetry offers a lockfile to ensure repeatable installs, and can build a project for distribution.

Sentinel is publicly available at PyPi and it is recommended that the installation is done using Pipx, however, it is NOT absolutely necessary.

pipx install sentinel-analysis

After installation, the tool can be called from any terminal session with the sentinel command.

$ sentinel -h
usage: sentinel [-h] --data DATA -gi GI -gin GIN [-dx DATAEXFILTRATION] [-bn BOTNET] [-cv] [-td] [-q] [-p]

Traffic analysis tool.

options:
  -h, --help            show this help message and exit
  --data DATA           Path to datafile to analyze
  -gi GI                Geolocation IP database
  -gin GIN              Geolocation IP database with nameserver
  -dx DATAEXFILTRATION, --dataExfiltration DATAEXFILTRATION
                        Threshold percentage for outliers in upload/download byte amount. Example '-dx 90' will return the top 10%
  -bn BOTNET, --botnet BOTNET
                        Detecting possible botnet communication in specified networks, or all networks. Example '-bn 192.168.1.0/24' will search P2P communication within that network
  -cv, --countryVolume  Measures the connection volume per country
  -td, --trafficDistribution
                        Generates a graph illustrating the hourly distribution of the volume of connections during a day
  -q, --quiet           Suppress stdout
  -o, --outfileMd       Export to markdown file

There are multiple options for processing, and each one is only executed if explicitly called. There are, however, three mandatory parameters that must be provided.

• data - Required - Path to the paquet file containing the traffic capture to be analyses.
• gi - Required - Path to the dat file containing the IP geolocation database.
• gin - Required - PAth to the dat file containing the IP geolocation of nameservers database.

Some anomalous behavior can be searched for by using the following options:

• dx - By specifying an integer between 0 and 100, the user is filtering for flows above the download over the upload percentile of all the connections in the session.
• bn - The user can specify in which subnet to look for P2P communication, which can indicate the presence of a botnet. If 0.0.0.0/0 is used, it will search in all subnets found.

For getting a general overview of the traffic, two commands are also available:

• cv - Measures the number of connections per country of destination.
• td - Measures the traffic hourly volume distribution.

Now we will perform an overall analysis of a traffic capture file with all of the options selected and analyze the data recovered, as well as inspect how the calculations for each of the metrics is performed.

The code presented next demonstrates a comprehensive set of SIEM rules specifically tailored to detect and combat two major security threats: data exfiltration and botnets. These rules have been meticulously crafted to analyze network traffic patterns, monitor system logs, and identify anomalous behaviors indicative of data exfiltration attempts. By leveraging these SIEM rules, organizations can effectively safeguard their sensitive data from unauthorized exfiltration and combat the growing menace of botnets, ensuring a resilient and secure computing environment.

Sentinel will be called in the following manner:

sentinel --data ../dataset4/data4.parquet -gi ../GeoIP_DBs/GeoIP.dat -gin ../GeoIP_DBs/GeoIPASNum.dat -cv -td -dx 90 -bn 192.168.104.0/24 -o

Note that the only option not selected is --quiet as it would suppress all stdout.

Note also that the flag -o is selected, what this does is creates a markdown file at the end of the analysis with all of the generated data. This can be useful for posterior archiving.

The objective of this measurement is simply to understand which countries are more targeted as connection destinations and which country has downloaded more data. When executed, it produces the following output at the console,

[!] Calculate the country of origin of the destination IPs and the connection volume for each country. Generating a PNG with the bar graph.

         dst_ip   down_bytes
          count          sum
country                     
US       334778  39621538092
PT       274777  26423827803
         274208  15754974807
NL        17948   1728071763
NA        13613   1310943454
GB         7460    717314962
BR         5335    512859364
ES         2019    196188082
IE         1281    118558649
...

and generates the following graph as a PNG image.

def countryVolume():
    ...
    # Analyses of destination countries' connection volume
    countries = data
    countries["country"] = countries["dst_ip"].apply(
        lambda ip: gi.country_code_by_addr(ip)
    )

    # Associates each country with its number of connections and the total
    #  data downloads by it. For normal flows they should be directly proportional
    volume = countries.groupby(["country"]).agg(
        {"dst_ip": ["count"], "down_bytes": ["sum"]}
    )
    
    # Sorting the DataFrame by dst_ip column in descending order
    volume = volume.sort_values(("dst_ip", "count"), ascending=False)

    printq(volume)
    ...

The objective of this measurement is to understand the time of the day in which there is a higher volume of connections.

hen executed, it produces the following output at the console,

[!] Distribution of connection volume during the day.

hour
4      2439
5      6845
6     18117
7     29313
8     55071
9     76117
10    91050
11    96017
12    97340
13    99522
14    92054
15    77853
16    64196
...

and generates the following graph as a PNG image.

def trafficDistribution():
    ...
    traffic = data
    traffic["hour"] = pd.to_datetime(traffic["timestamp"] / 100.0, unit="s").dt.hour
    hourly = traffic["hour"].value_counts().sort_index()
    fig, ax = plt.subplots(figsize=(10, 6))
    ax.bar(hourly.index, hourly.values)
    ax.set_xlabel("Hours")
    ax.set_ylabel("Number of connections")
    ax.set_title("Connections per hour during a day")
    ax.set_xticks(range(24))

    filepath = "connectionsxhour"
    fig.savefig(fname=filepath)

    printq(hourly)
    ...

For data exfiltration analysis we focused on the fact that although the amount of downloaded data is usually larger than the uploaded data if the service is the same for all of the clients during the session, the ratio of this two values should (if normal behaviour is considered) be similar for all the clients.

If a particular client demonstrates a particular large ratio of downloads over uploads, this might indicate that it is extracting a large amount of data, possibly with malicious intent.

This is the generated output at the console,

[!] IPs above the 90 percentile of the downloads over uploads in this session.

              src_ip  down_bytes  up_bytes      ratio
110  192.168.104.200    14778895   1415961  10.437360
116  192.168.104.206    53495023   5195056  10.297295
64   192.168.104.159   324226107  32356733  10.020360
144   192.168.104.44    98180581   9977423   9.840274
131   192.168.104.31   259857369  26500446   9.805773
174   192.168.104.74   385717683  39392101   9.791752
184   192.168.104.84   193031076  19880947   9.709350
90   192.168.104.182   609527835  63089225   9.661362
...

def dataExfiltration(percentage):
    percentage = 1 - (percentage / 100)

    # Calculate the total downloaded and uploaded bytes per IP
    ratios = data.groupby(["src_ip"], as_index=False)[["down_bytes", "up_bytes"]].sum()

    # Calculate the ratio between downloaded and uploaded bytes
    ratios["ratio"] = ratios["down_bytes"] / ratios["up_bytes"]

    # Sort by ratio
    ratios = ratios.sort_values(["ratio"], ascending=False)

    # Top outliers based on percentage
    outliers = ratios.head(int(len(ratios) * percentage))

    printq(outliers)

It is not usual for devices within the same network in a corporate environment to engage in peer-to-peer communication, if so, this may indicate that botnets, C&C, or horizontal traversal may be taking place within the network.

This is the generated output at the console when the botnet detection algorithm is executed,

[!] Flows between internal hosts to common destinations in the same subnet (192.168.104.0/24), may indicate the existence of an internal botnet.

            dst_ip                                             src_ip  frequency
1  192.168.104.224  [192.168.104.82, 192.168.104.63, 192.168.104.2...   0.008091
3  192.168.104.231  [192.168.104.82, 192.168.104.63, 192.168.104.2...   0.007870
2  192.168.104.230  [192.168.104.82, 192.168.104.63, 192.168.104.2...   0.005934
5  192.168.104.234  [192.168.104.82, 192.168.104.63, 192.168.104.2...   0.005923
0  192.168.104.222  [192.168.104.82, 192.168.104.63, 192.168.104.2...   0.005816
4  192.168.104.232  [192.168.104.82, 192.168.104.63, 192.168.104.2...   0.005724

def botnet(subnet):
    subnet = ipaddress.IPv4Network(subnet)

    # Filter to src_ip for desired subnet
    same_subnet = data
    same_subnet["src_ip"] = same_subnet["src_ip"].astype(str)

    # Filter to src_ip and dst_ip in the same desired subnet
    same_subnet = same_subnet.loc[
        (data["src_ip"].apply(lambda x: ipaddress.IPv4Address(x) in subnet))
        & (data["dst_ip"].apply(lambda x: ipaddress.IPv4Address(x) in subnet))
    ]

    # Associates each dst_ip with a set of IPs and a set of timestamps
    same_subnet = same_subnet.groupby(["dst_ip"], as_index=False).agg(
        {"src_ip": list, "timestamp": list}
    )

    # Calculate the frequency of the communications
    def frequency(comms, period):
        return len(comms) / (period[-1] - period[0])

    same_subnet["frequency"] = same_subnet.apply(
        lambda x: frequency(x.src_ip, x.timestamp), axis=1
    )

    # Clean non relevant series
    same_subnet = same_subnet.drop(["timestamp"], axis=1)

    # Clean repeated IPs
    same_subnet["src_ip"] = same_subnet["src_ip"].apply(lambda x: list(set(x)))

    # Order
    same_subnet = same_subnet.sort_values(["frequency"], ascending=False)

    printq(same_subnet)
    ...