A Go program to fetch the USOM (TR-CERT) OSINT Feed in XML format and convert it to PANOS EDL format.

This program's existence is due to the following reasons:

• PANOS can only read EDLs from plain TXT files. It does not support other formats like XML, RSS, STIX, or TAXII. It is sensitive to data format and only accepts one value per line.
• PANOS has no regex capability for EDLs to parse different types. It cannot segregate different types like IP, domain, and URL from a single EDL source. Each type needs to be defined separately.
• PANOS has capacity limits for EDLs. Higher end platforms have a 250k URL limit, and lower end platforms have a 100k URL limit. Other limits exist for IPs and domains depending on the platform.
• USOM does not provide separate feeds for IP, domain, and URL IOC types, unlike other CERTs. They only provide a single source file with a mixture of each IOC type.
• USOM provides that single source in two formats: TXT and XML. Even in XML format, there is no metadata included for IOCs to identify their type. So regular expression operations are a must to identify types.
• USOM contains different and multiple syntaxes for each IOC type. They have no standard definitions for IOCs. Some sort of normalization is needed before use.
• USOM feed contains a huge number of IOCs. The number exceeds the PANOS capacity limits even for the highest plarforms. As they never remove a record from the list, it contains ancient and questionable records.
• Minemeld could be used to address some of these downsides formerly, but it is archived and not developed by PAN anymore. Also, it produces another link in the security toolchain on its own, which needs to be learned and maintained.

To overcome these challenges, PAN-USOM-XML2EDL needs to be used as a middleware. This program:

• Fetches USOM IOC Feed in XML format.
• Marshalls the XML data to match metadata for each IOCs,
• Filters IOCs by date using metadata information.
• Limits IOCs by count to stay within platform limits.
• Identifies and extracts different types of IOCs from a single feed.
• Parses multiple syntaxes and normalizes IOCs for each type.
• Compacts and de-duplicates recurring IOCs for each type.
• Segregates different types of IOCs and presents a seperate feed for each.

Pre-compiled binaries can be downloaded directly from the latest release (Link here: Latest Release). These binaries can readily be used on the systems for which they were compiled. Neither re-compiling any source code nor installing GO is not needed. In case there is no pre-compiled binary presented for your system, you can refer to the Compilation section.

This program only requires the appsett.env file to determine which settings it will run with. Even if it can run with its default settings without any parameters set in the appsett.env file, this file must be present and accessible.

By default, the program searches for the appsett.env file in the working directory. The working directory can be altered by passing -dir [PATH] argument. (PATH value for -dir can be ablsolute or relative.) The working directory is also determines where the generated EDL file(s) will be placed. If you need to change the output directory without changing the working directory, or if you have a scenario where you need to read the appsett.env file from one directory but create EDL files in a different directory, you can use the -out [PATH] argument for that purpose. The -out [PATH] argument overrides the working directory setting for the EDL file(s) that will be created. (PATH value for -out can be ablsolute or relative. If a relative PATH value for -out used with -dir option, it is based on former working directory.) These options can be explored by passing the -usage argument to the program.

There are multiple settings controlled in environment variable format in the appsett.env file. These settings are explained in the Settings section.

How to schedule this program and how to serve generated EDL file(s) are not within the scope of this program or this documentation. Nevertheless, some hints are shared in the Hints section.

All setting options are provided with the sample appsett.env file, along with short descriptions and default values for each. Note that all lines are commented-out in the sample. If any of the options are to be used simply clear the comment token (#) and set the preferred setting.

# Basic & Optional Settings:
PANUSOMXML2EDL_FEED_URL={Enter URL of USOM XML OSINT Feed, Default: https://www.usom.gov.tr/url-list.xml}
PANUSOMXML2EDL_DAYS_OLD={Enter Max Day Old to Filter Entries, Default: 0 (Unlimited)}
PANUSOMXML2EDL_LIMIT_COUNT={Enter Limit Count to Filter Entries, Default: 0 (Unlimited)}
PANUSOMXML2EDL_NO_SORT={Enter Either TRUE or FALSE to Skip Sorting XML Entries, Default: FALSE}
PANUSOMXML2EDL_SINGLE_OUTPUT={Enter Either TRUE or FALSE to Create Single Output Without Any Parsing, Default: FALSE}

# Advanced & Expert Options:
PANUSOMXML2EDL_LOG_SILENCE={Enter Either TRUE or FALSE to Enable Verbose Log Suppression, Default: FALSE}
PANUSOMXML2EDL_LOG_UNPARSABLE={Enter Either TRUE or FALSE to Log Each Unparsable Record, Default: FALSE}
PANUSOMXML2EDL_SKIP_VERIFY={Enter Either TRUE or FALSE to Skip TLS Certificate Verification, Default: FALSE}
PANUSOMXML2EDL_FILE_TEST={Enter Either TRUE or FALSE to Run Offline Test from url-list.xml File, Default: FALSE}

When using this program under Unix-like OSes like Linux or macOS, Cron can be used to schedule periodic execution of the program. Here is an example of Crontab file:

# /etc/crontab
# To run the program every hour:
0 * * * * /path_to_binary/PAN-USOM-XML2EDL -dir /path_to_appsett.env_file/ -out /path_for_edl_files/

Under Windows OSes, the Task Scheduler tool can be used for the same purpose.

For serving the generated EDL file(s), Apache, Nginx, or IIS can be used to handle incoming HTTP(S) requests. For short-term testing purposes, http.server module of Python 3 can be used. Here is an example of how to run this:

# Serves content of the current working directory:
python3 -m http.server 8080

If none of the pre-compiled binaries covers your environment, you can choose to compile from source by yourself. Here are the instructions for that:

git clone https://github.com/enginy88/PAN-USOM-XML2EDL.git
cd PAN-USOM-XML2EDL
go mod init
go mod tidy
make local # To compile for your own environment.
make # To compile for pre-selected environments. 

NOTE: To compile from the source code, GO must be installed in the environment. However, it is not necessary for to run compiled binaries! Please consult the GO website for installation instructions: Installing Go

Because it (cross) compiles into machine code! You can directly run ready-to-go binaries on Windows, Linux, and macOS. No installation, no libraries, no dependencies, no prerequisites... Unlike Bash/PowerShell/Python it is not interpreted at runtime, which drastically reduces runtime overhead compared to scripting languages. The decision to use a compiled language makes it run lightning fast with lower memory usage. Also, due to the statically typed nature of the Go language, it is more error-proof against possible bugs/typos.