mitre-attack / attack-scripts Goto Github PK

Update taxii2client in requirements and in all scripts to use version 2.0.0.

This will require changing the import statement to enable backwards compatability with stix2.0 content. The new import statement is:

from taxii2client.v20 import Collection

Both #1 and #2 should validate input and output layers to make sure they match the schema. The validation function should be both generalized and flexible to multiple use cases.

• An optional flag should be provided for allowing layers to include additional fields in the schema (non-strict checking). If the flag is not present the script will report a break from the schema as soon as an unexpected field is present.
• The script should be able to return multiple kinds of reports.
  • whether or not the layer is valid (boolean)
  • list of ways the layer does not conform to the schema
  • human readable report on how the layer does not match the schema, for printing to the console

The format for sub-techniques in changelogs generated by diff_stix.py is incorrect, and missing the /techniques prefix. This is presumably a bug in getUrlFromStix(self, datum)

As a user of diff_stix, I want the script to be able to summarize changes in data sources and data components.

Update diff_stix to include data sources and data components. Data components should be nested beneath their data sources using the same logic as sub-techniques and techniques.

As a user of ATT&CK Layers in Python, I want to be able to convert a layer into an easily readable/editable CSV file.

Each row should be a technique.
Each column should be a technique property/annotation, e.g color, score, comment.

Further considerations should be taken on how to encode the non-technique layer data (sorting, domain, name, description) in the CSV.

As a maintainer of ATT&CK release notes, I want to be able to determine who contributed to the most recent release of ATT&CK using diff_stix.

Note: this is not simply subtracting the set of contributors on the old version from the new. A contributor listing who existed on an object before, but also was added to a new/updated object, should be considered an addition for the sake of the output of this comparison since they contributed to content added/updated in the new version.

Track changes to contributors on an object-by-object basis. The following characteristics constitute a contributor to a release:

• Contributors who were not previously present on a given object which has been updated in any way
• Anyone who contributed to a new object

The following characteristics DO NOT constitute a contributor for a release:

• If the order of contributors on an object are changed
• If a contributor is removed from an object and (not re-added on a different object)

• The markdown report should include a section, contributors, which lists the contributors to the release in a bullet-point list.
• detected contributor changes should be available on the DiffStix object in a field after data has been parsed.
• An argument should be given to opt-in to listing contributors in the output markdown. If the argument is not specified, the contributor data should still be parsed but not written in the markdown report.

The current content of CONTRIBUTING.md is overly generic and doesn't offer sufficient information on what sorts of scripts we're hoping to solicit. We should update it to provide better guidance regarding our implementation standards and intentions for this repository.

See also #27.

As a user of ATT&CK Layers in python, I want to be able to import a layer from a CSV file in the same format as defined in #27.

Line 10 in attack-scripts/layers/exporters/to_svg.py:
from layers.core import Layer as topLayer # alternative import for typechecking

throws the error that it cannot find the layer module. I am running attack-scripts as a submodule in a project (not sure if that changes anything). Removing that line fixes the issue on my end.

Add support in scripts for the newly stixified ics-attack domain.

To reproduce:

python3 attackToExcel.py -output ~/Documents/attack -version v1.0

Expected result:

A set of XLS files with the data for V1.0

Actual result:

Conversion fails with an error and no output is produced.

Traceback (most recent call last):
  File "attackToExcel.py", line 175, in <module>
    main(args.domain, args.version, args.output)
  File "attackToExcel.py", line 151, in main
    dataframes = build_dataframes(get_data_from_version(domain, version), domain)
  File "attackToExcel.py", line 33, in build_dataframes
    "tactics": stixToDf.tacticsToDf(src, domain),
  File "/home/vonglasowm/src/attack-scripts/attackToExcel/stixToDf.py", line 166, in tacticsToDf
    "tactics": pd.DataFrame(tactic_rows).sort_values("name"),
  File "/home/vonglasowm/src/attack-scripts/env/lib/python3.6/site-packages/pandas/core/frame.py", line 5291, in sort_values
    k = self._get_label_or_level_values(by, axis=axis)
  File "/home/vonglasowm/src/attack-scripts/env/lib/python3.6/site-packages/pandas/core/generic.py", line 1560, in _get_label_or_level_values
    raise KeyError(key)
KeyError: 'name'

Versions used:

6347904 on Ubuntu 18.04

Additional information:

Exporting the latest version (without specifying a --version argument) works as expected as of today. V2.0 gives an error which looks like the above, V3.0 and V4.0 also error out, but with somewhat different messages. V5.0, V6.0, V7.0 and V8.0 complete without errors.

As a user of ATT&CK Layers, I want to be able to convert layers to excel spreadsheets just like in the ATT&CK Navigator's "export to Excel" feature. I want a script with an easy-to-use CLI and/or API so that I can convert multiple layers simultaneously without having to open them all in the Navigator independently.

As a user, I want to be able to access a table of contents for the diff_stix's Markdown output because the output can be very long.

• The TOC should include anchor links to scroll to the relevant sections.
• The TOC should adapt to differing contents of the markdown sections, so if the user builds with --contributors it shows up properly
• The TOC should be opt-in through a flag (like --contributors)

Really small issue, currently the first script link is broken.

techniques_from_data_src.py should actually be techniques_from_data_source.py.

The redirect goes to https://github.com/mitre-attack/attack-scripts/blob/master/scripts/techniques_from_data_src.py

As a user of ATT&CK, I want to be able to create a diff between two arbitrary versions which I can further process electronically, e.g. in a relational database.

The stix_diff script is a start but falls short of this requirement in two points:

• it does not allow for an exact comparison of old/new (cf. #11)
• the output is not easily machine-parseable

We are in the process of implementing the MaGMa framework in our organization. MaGMa implements a top-down model of an adversary’s actions; a major source for the bottom layer are techniques taken from ATT&CK. In order to reflect changes in ATT&CK in our MaGMa implementation, we would need a detailed diff.

As we maintain the information in a relational database, we would need this information in a compatible format (CSV, XLS or ODS would work, in a manner similar to #45).

Our focus is mainly on techniques, although similar use cases with similar considerations might apply to other object types.

A diff would need to contain information such as:

• T2370 has been removed from the new version
• T2380 has been removed from TA0001 (but is still part of the new version and has other associated tactics)
• T2390 has been added to TA0002 (i.e. the technique itself was already present in a previous version and has not changed)
• T3170 had its title changed from Rubber ducky placement to Placement of water fowl-shaped rubber toy
• T6580 has been added (quoting all its data fields, only those related to version control may be omitted)
• T8038 is now T8031/008 (changed into a sub-technique)

requests.exceptions.ConnectionError: HTTPSConnectionPool(host='cti-taxii.mitre.org', port=443): Max retries exceeded with url: /stix/collections/95ecc380-afe9-11e4-9b6c-751b66dd541e/ (Caused by NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x1042280a0>: Failed to establish a new connection: [Errno 61] Connection refused'))

Just want to check if these scripts are still being maintained?

The detection training added in #8 is now outdated and should be removed from this repository.

ATT&CK Navigator layer files generated or ingested by scripts in this repository need to use the most updated schema to support the upcoming v4.0 of the Navigator.

This issue will be updated once the new schema is finalized.

As a visitor to the ATT&CK Scripts repository, I want the root README to give me an overview of the different functionalities present in the repo.

Currently the root README doesn't include any mention the layers folder. We should update it to point users towards the layers folder README.

The ATT&CK Navigator defines a layer format which can be used to describe annotations on ATT&CK matrixes in JSON format. The principal functionality of the ATT&CK Navigator is viewing and editing these matrixes, but it also includes the ability to merge (combine) the annotations from multiple layers. Unfortunately, these layer operations are only available in the GUI and therefore unsuited to scripting.

This issue proposes the creation of a layer-operations script to handling merging of layer annotations. It would allow complex manipulations of scores, comments, enabled/disabled states, colors, and metadata.

The script would feature two interfaces, and evaluation of the technique annotations would be performed by passing functions as named arguments.

list interface

In this interface, the layers would be provided in a list. The annotation-evaluation function would then be passed a list of that annotation in the same order as the layers specified. For example, to average the scores of three layers:

avglayer = layerops([layer1, layer2, layer3], score=lambda t: sum(t)/len(t))

To double the score of a single input layer:

doublelayer = layerops([layer1], score=lambda t: t[0] * 2)

To combine the comments of three layers:

commentlayer = layerops([layer1, layer2, layer3], comment=lambda t: ", ".join(t))

Another example where both score and enabled are set

layer = layerops([layer1, layer2], score=lambda t: t[0] * t[1], enabled=lambda t: t[0] or t[1])

dict interface

In the case where the user is working with complex formulas, remembering the index of a layer may be difficult. In this case, the user could provide a dict of layers as input. The input to the annotation-evaluation function would therefore also be a dict:

layer = layerops({a: layer1, b: layer2, c: layer3}, score=lambda t: (t["a"] + t["b"]) / t["c"])

layer = layerops({a: layer1, b: layer2}, comment=lambda t: "comment for layer1 is {a}, comment for layer2 is {b}".replace("{a}", t["a"]).replace("{b}", t["b"]))

As a user of ATT&CK, I want to be able to access the dataset in excel instead of STIX. The excel spreadsheets should use ATT&CK terminology instead of STIX terminology (e.g "technique" instead of "attack-pattern").

Part of attack-website's Making data more accessible milestone is providing spreadsheets of ATT&CK data on the ATT&CK website. We should build the STIX => Excel converters here first, and then use the ATT&CK website to automate the distribution of the resulting files to users who can't/won't run the Python converter themselves.

Revoked and deprecated objects, and revoked-by relationships, should not be included unless a flag has been set in the CLI.

The script(s) should generate the following files:

• {domain}-attack-{version}.xlsx, which contains the entire dataset divided into sheets by type:
  • techniques
  • tactics
  • software
  • groups
  • mitigations
  • matrices (laid out like the layer to excel format of the matrix, but with the matrix name and description included as well)
  • relationships
• {domain}-attack-{version}-techniques.xlsx, which contains techniques and all relationships with techniques, as well as a representation of the technique->tactic mappings which are not strictly relationships in the attack data. Relationships should be kept in one sheet per relationship type.
• {domain}-attack-{version}-tactics.xlsx, which contains tactics and all relationships with tactics, as well as a representation of the technique->tactic mappings which are not strictly relationships in the attack data. Relationships should be kept in one sheet per relationship type.
• {domain}-attack-{version}-software.xlsx, which contains software and all relationships with software. Relationships should be kept in one sheet per relationship type.
• {domain}-attack-{version}-groups.xlsx, which contains groups and all relationships with groups. Relationships should be kept in one sheet per relationship type.
• {domain}-attack-{version}-mitigations.xlsx, which contains mitigations and all relationships with mitigations. Relationships should be kept in one sheet per relationship type.

As a user of diff_stix, I want to be able to see subtechniques inside of the techniques section.

Add support for sub-techniques. Sub-techniques should be shown inside of the techniques section, as child lists of their parent technique's list item.

For example:

added:
- technique 1
    - subtechnique 1.1
    - subtechnique 1.2
    - subtechnique 1.3
- technique 2
    - subtechnique 2.1
    - subtechnique 2.2

If a parent technique is relevant to a section (e.g changed, added, etc), and the subtechnique is not, omit the subtechnique from the list.

added:
- technique 1
- technique 2

If a parent technique is relevant to a section, and the subtechnique is too, show both in the list.

added:
- technique 1
    - subtechnique 1.1
- technique 2

If a parent technique is not relevant to a section, but the subtechnique is, than show both, but mark the parent technique such that its inclusion is obviously only to show the hierarchy.

added:
- (technique 1)    ⟸ italics
     - subtechnique 1.1
- technique 2

As a user of the ATT&CK scripts repo, I want to be able to see the correct trademark status (registered) when ATT&CK is mentioned.

For scripts like
https://github.com/mitre-attack/attack-scripts/blob/master/scripts/technique_mappings_to_csv.py ,the collection URL is not working anymore.

    collection_map = {
        "enterprise_attack": "95ecc380-afe9-11e4-9b6c-751b66dd541e",
        "mobile_attack": "2f669986-b40b-4423-b720-4396ca6a462b"
    }
    collection_url = "https://cti-taxii.mitre.org/stix/collections/" + collection_map[collection_name] + "/"
    collection = taxii2client.Collection(collection_url)
    taxii_ds = stix2.TAXIICollectionSource(collection)

The error shows as: 406 Client Error: Not Acceptable for url:

  File "/tmp/stix_code/py3env/lib/python3.6/site-packages/stix2/datastore/taxii.py", line 166, in __init__
    " Collection object provided could not be reached. Recieved error:", e,
stix2.datastore.DataSourceError: The underlying TAXII Collection resource defined in the supplied TAXII Collection object provided could not be reached. Recieved error: "406 Client Error: Not Acceptable for url: https://cti-taxii.mitre.org/stix/collections/95ecc380-afe9-11e4-9b6c-751b66dd541e/"

As a user of ATT&CK and the diff_stix script, I want to be able to see granular details about what changed for each object in an update.

Add a --detailed-changes flag which reports on which fields have changed between updates (though not the contents of those changes). It should also report on whether or not relationships have been added, changed, and removed.

As a user of the ATT&CK Navigator, I want to learn how to write scripts to generate layers from the STIX data on MITRE/CTI.

The ATT&CK Navigator includes sample layers in /layers/data/samples. There are scripts corresponding to these layers. We should release those scripts on this repository and create references to them from the ATT&CK Navigator repository.

With the next release of ATT&CK, the pre-ATT&CK domain is being migrated into new tactics under the enterprise domain. Therefore the pre-ATT&CK domain itself should be removed from the scripts entirely in preparation for the next release.

To reproduce:

1. Obtain enterprise_attack.json from the git releases tagged ATT&CK_v1.0 and ATT&CK_v8.2 respectively, and save them locally.
2. Run python3 diff_stix.py -old ~/Documents/attack/V1.0 -new ~/Documents/attack/V8.2 -types technique -domains enterprise-attack -markdown ~/Documents/attack/diff-v1_0-v8_2.md.

Expected output:

The script completes and produces a markdown file at the specified location.

Actual output:

An error is thrown and no output is produced.

ERROR: cannot get old version for object: attack-pattern--7dd95ff6-712e-4056-9626-312ea4ab4c5e
Traceback (most recent call last):
  File "diff_stix.py", line 657, in <module>
    verbose=args.verbose
  File "diff_stix.py", line 124, in __init__
    self.load_data()
  File "diff_stix.py", line 275, in load_data
    if new_version > old_version:
UnboundLocalError: local variable 'old_version' referenced before assignment

Versions used:

c544101 on Ubuntu 18.04

Additional information:

If I use V5.0 as old, the script completes successfully. I didn’t investigate which versions are affected, but suspect it might be related to #55.

I forked this repository and added the capability to dump all mitigations for each technique and sub-technique. If the technique/sub-technique does not have one listed, I just populate with "No Mitigation".

If I put in a PR, is there any interest? The main use case for me was for data science and generating statistics for each technique, sub-technique, and mitigation.

Thank you!

The ATT&CK Navigator defines a layer format which can be used to describe annotations on ATT&CK matrixes in JSON format. The principal functionality of the ATT&CK Navigator is viewing and editing these matrixes, but it also includes an export functionality to render and save the matrix to an SVG image. Unfortunately, the layer renderer is only available through the webapp UI and therefore unsuited for many use cases involving programmatic layer generation.

This issue proposes the creation of a layer renderer script. It would take as input a layer file and optional parameters for the output file destination and format, dimensions, fields to display in the header, and so on. The script would exist as a python function for importing into larger workflows, and would include a CLI. The script would be implemented in python3.