fortify / fcli Goto Github PK

Support polling/waiting of SC-DAST scan to complete to auto retrieve scan status

This issue holds ideas and proposals for running workflows and composite commands.

• Investigate possibility to run multiple subcommands in one go:
  • For example: fcli login ssc --url ... get applications -o applications.json get versions -o versions.json logout ssc
  • See picocli subcommandsRepeatable = true configuration option
• Allow shell-based chaining of commands by providing suitable output and functionality for parsing input; some examples:
  • FCLI_FMT=json fcli create ssc version app/version … | fcli … --someOption=${{ stdin.json.project.id }}
  • for v in $(fcli get versions); do fcli download ssc artifact --from ${v}

Idea for workflow syntax:

# Define environment variables for use during workflow run, inheriting from parent environment
env:    
  name: value

# Should we have a separate element for logging in? This would open transient session and would auto-logout
# after workflow completion 
login:   
  ssc:
    url: ...

steps:
  - id: <id used for referencing output of this step in other steps>
    name: Name to display when running this step
    if: ${{ steps.previousStepId.output.json.criticalIssueCount > 0 }}
    forEach: ${{ steps.previousStepId.output.json.data[].id }}
    fcli: <fcli command to run>
    composite:
      async: true|false # Run each step in a separate thread, for example allowing to spawn both sast and dast scans
      maxThreads: 5   # Maximum number of threads to spawn
      joinThreads: true|false # If async==true, wait until all steps defined below have completed
      steps:
        - id, name, fcli, ...

Regarding the fcli: entry:

• Can use expressions like ${{ env.xyz }}, ${{ steps.abc.output.plain }}, ${{ steps.abc.output.json.someProperty}}, ...
• Can use run cmd subcommand to run arbitrary commands (run cmd should be disabled by default for security reasons, can be enabled through fcli config run-cmd enabled true|false or something similar
• Can use run workflow to run other workflows

Looking at debugging output, it seems like a logout seems to trigger another login if --allow-renew has been specified. This requires further research.

Authentication session data, which may include passwords and tokens, is currently being stored as follows:

• In plain text
• In files located under the Fortify home directory
• Various mechanisms are used in an attempt to make these files accessible to only the current user

This data should be stored in a more secure way, while still allowing all existing functionality like listing login sessions. Below are some possible approaches.

1. Static encryption key bundled with fcli
Define a static encryption key in source code or accompanying resource file. Session data files will not be readable without knowing the encryption key. Anyone with access to the source code or resource file will be able to decrypt session data.

2. Generated encryption key stored in home directory
Generate an encryption key in the Fortify home directory. Session data files will not be readable without knowing the encryption key. Anyone with access to the secret key stored in the Fortify home directory will be able to decrypt session data. Since encryption key is located right next to the data being protected, this is probably one of the least secure options.

3. Combination of #⁠1 & #⁠2 above
Combine a static encryption key with a dynamically generated key, adding a level of security. Anyone with access to both source code and generated key will be able to decrypt session data.

4. Automatically start and use a daemon
Similar to git-credential-cache--daemon (maybe even reusing that), fcli would start a daemon that stores session data in memory. In this scenario, session data is never stored on disk, and potentially the daemon could even take measures to protect the in-memory data. For shared systems, we should be very careful that session data is only accessible by the user that performed the login. Note that the git-credential-cache is really just a specific type of credential helper as discussed in #⁠5 below, so if we implement #⁠5 we can also have support for this daemon-based approach.

5. Use credential helpers
Similar to how Docker and Git store their credentials. We can compile our own fcli-specific helpers (potentially based on the source code of Git or Docker helpers), or we can interact with Git or Docker credentials helpers if installed.

The latter is probably easier, but we need to assume that these helpers are safe and cannot easily fix any security vulnerabilities in these helpers. On the other hand, these are used on a much larger scale than fcli, so it is more likely that any security vulnerabilities are discovered and fixed in a timely fashion.

Note that fcli allows for multiple login sessions to a single URL, and allows for listing those sessions. The Docker and Git credential helpers do not natively support this; they can store a single set of credentials for specific URL's, and do not provide functionality for listing the stored credentials. As such, we'll need some work-around.

Probably by far the easiest approach is to use these credential helpers to store and retrieve a generated private key that is then used to encrypt session data on disk. This approach is very similar to approach #⁠2, but with the encryption key being stored using Git or Docker credential helpers instead of on-disk in the Fortify home directory. So the store request to one of the Docker helpers would for example look like this:

{
  "ServerURL": "fortify://fcli/v1/secret",
  "Username": "dummy",
  "Secret": "<Encryption key>"
}

6. Combine #⁠1, #⁠2 & #⁠5
Use a combined encryption key that is made up of a static and generated encryption key, with the generated encryption key being stored using a credentials helper if available, or on disk in the Fortify home directory if no credentials helper is available.

The fcli dast-scan command tree is meant for preparing, running, and managing DAST scans. This includes functionality like waiting for scan completion, checking scan status, acting on scan results, ...

Currently some of this functionality is in the fcli get tree, which is meant to get arbitrary data independent of what scans are currently running (although the 'get' tree could have commands for listing all scans including scan status, it's not meant to wait for scan completion and such),

Generic output formatting functionality yet to be completed

• Add support for doing string replacements in field names, i.e. project->application to manage inconsistencies between user interface and API when outputting in human-readable/text format
• Perform transformation before actually invoking the writer, such that not every writer needs to explicitly call the transformer?
• Implement --output-file / -o command line options
• Review structure of IJsonNodeTransformerSupplier and AbstractJsonNodeTransformerSupplier (can we avoid having to do a getJsonNodeTransformer call in the command class)?
• Add subclass/mixee for OutputWriterMixin that contains write(HttpRequest<?>) and/or write(HttpResponse<?>) methods. The method taking an HttpRequest parameter would then execute that request, check HTTP response code, and write either success or error data in the requested format. Also see #15

Calls to fcli get <entity> <resource> --format <format> should support a format option to output data/results in table format.

Below is an example output:

C:\workspace>.\fcli.exe get ssc versions --fmt table
 #  Name     Diameter  Mass  Atmosphere
 1  Mercury   0.382    0.06  minimal
 2  Venus     0.949    0.82  Carbon dioxide, Nitrogen
 3  Earth     1.000    1.00  Nitrogen, Oxygen, Argon
 4  Mars      0.532    0.11  Carbon dioxide, Nitrogen, Argon

When using fcli tool <ToolName> download or fcli tool <ToolName> install, fcli should perform some kind of integrity check of the package or tool being downloaded. Ideally validating a sha256 or higher checksum should be performed.

SSC login fails when the url has a trailing "/" character.

Calls to fcli get <entity> <resource> --format <format> should support a format option to output data/results in csv format.

Add the following actions:

• rebranding a parser plugin jar by changing the engine type and logo/icon for an existing plugin jar. For example, rebrand CycloneDX plugin to Debricked plugin.
• Install a parser plugin jar to SSC (possibly with rebranding options to override engine type and logo/icon with a single action)
• enable/disable/delete a parser plugin in SSC
• list installed/enabled/disabled parser plugins/engine types
• upload 3rd-party artifacts, specifying the engine type as CLI option (no need for user to manually create a zip-file containing scan.info file)

• Research what report commands should be provided
• Implement those commands

Decide on how to manage documentation for fcli. This includes:

• General information about fcli
• Possibly man-style page(s) that list all sub-commands and options
• Workflow yaml format description
• Sample use cases, workflows, ...

Preferably this documentation should be managed alongside the source code, following the same version numbering. This will allow users of older versions of the utility to still be able to access the appropriate documentation. As such, the GitHub wiki is probably not the best place to manage documentation for multiple versions of fcli (unless those wiki pages are somehow auto-generated).

FortifyVulnerabilityExporter uses the fortify-client-api project, which uses a different (Apache) HTTP client and probably comes with many dependencies that may not be compatible with GraalVM/native binaries

Effectively it is just retrieving vulnerability data and transforming that to various formats, so potentially this could be rewritten and incorporated into fcli.

Need to think about how the command structure should look, especially since FortifyVulnerabiltityExporter is very configurable; see https://github.com/fortify/FortifyVulnerabilityExporter/tree/main/config and https://github.com/fortify/FortifyVulnerabilityExporter/tree/main/FortifyVulnerabilityExporter-plugin-to-json/src/main/resources/pluginConfig

Oher notes:
The 'transformation library' consists of just a couple of classes, mostly in https://github.com/fortify/FortifyVulnerabilityExporter/tree/main/FortifyVulnerabilityExporter-plugin-to-json/src/main/java/com/fortify/vulnexport/to/json/vuln, but as mentioned it heavily depends on Spring EL of which I'm not sure it will work well with native binaries. The transformation library is based on Spring Expression Language.

Other than that, it would be mostly replicating some of the functionality from my fortify-client-api to get and filter data from SSC and FoD, but then using the HTTP/REST client used by fcli.

In particular, based on configuration it should be possible to get additional data from SSC/FoD for a particular application version/release or vulnerability, like calling the details endpoint for every SSC vuln.

• Research what operations can be performed on sensor pools
• Implement those operations

Tool download should show download progress bar or some kind of other information on the status of the tool or package being downloaded.

• Research what subcommands should be supported by the sc-sast sensor command
• Implement those subcommands

Native binaries for Windows, Linux, and MacOS are rather large (currently about 47MB.
We can probably compress the binaries down a bit. After testing this with UPX and the Windows binary, I was able to get the size down to about 12.

I'm submitting a pull request to update the build workflow for Windows. This can be used as a reference for the other binaries. Please note that currently the PR does not delete the uncompressed version of the native binary, so you may wish to modify the PR before merge.
#2

• Add a command for listing sensors, including status if available
• Any other CRUD commands for sensors?

• Artifact upload and download should be mostly complete
• Upload: Add support for uploading 3rd-party artifacts, both as zip-file or the raw artifact by specifying --engine-type or --type CLI option
• Download: Implement options for downloading application state FPR (current behavior), specific artifact id, or latest artifact of a specific type
• Implement purge and delete commands

Should be mostly implemented now; review functionality before closing this issue

Support polling/waiting of SC-DAST scan to complete to auto retrieve scan results

Support starting ScanCentral DAST.

Have a command or help option that prints the full command tree, such that users can easily see what commands are available, how they are structured. ...

With subcommands or options that accept an application version ID, those same subcommands and options should also allow the user to specify the application version using the following syntax "appName:verName". Also, the delimiter should be customizable in the event that the default seperator character (:) is used in either the application or app-version name.

• Decide where in the command tree these commands should be located
• Implement these commands

The SCDastScanActionsHandler class contains a UnirestInstance field annotated with @Inject, however it seems like this field is not meant to be injected by Micronaut's dependency injection mechanism, but rather is supposed to be provided as a constructor parameter by the caller of this class. As such, the @Inject annotation is not applicable here and should be removed. It should be verified whether there are any inappropriate annotations in any of the other classes.

• Implement the install command for all available tools
• If tool is already installed, user should have the option to skip or reinstall (CLI options and/or interactive prompts)
• Install command should cache the install location for use by uninstall

Not sure what this should do. @wtfacoconut Can you add a proper description, describing the feature that you had in mind?

rsenden@NLsendenr04:/mnt/c/Users/sendenr/Downloads$ ./fcli dast-scan sc-dast start
java.lang.NullPointerException
        at com.fortify.cli.sc_dast.command.dast_scan.SCDastScanCommands$Start.runWithUnirest(SCDastScanCommands.java:39)
        at com.fortify.cli.sc_dast.rest.unirest.runner.SCDastUnirestRunner.lambda$runWithUnirest$1(SCDastUnirestRunner.java:57)
        at com.fortify.cli.common.rest.unirest.runner.UnirestRunner.runWithUnirest(UnirestRunner.java:56)
        at com.fortify.cli.sc_dast.rest.unirest.runner.SCDastUnirestRunner.lambda$runWithUnirest$2(SCDastUnirestRunner.java:51)
        at com.fortify.cli.common.rest.unirest.runner.AbstractAuthSessionUnirestRunner.lambda$runWithUnirest$0(AbstractAuthSessionUnirestRunner.java:63)
        at com.fortify.cli.common.rest.unirest.runner.ConnectionConfigUnirestRunner.lambda$runWithUnirest$0(ConnectionConfigUnirestRunner.java:62)
        at com.fortify.cli.common.rest.unirest.runner.UnirestRunner.runWithUnirest(UnirestRunner.java:56)
        at com.fortify.cli.common.rest.unirest.runner.ConnectionConfigUnirestRunner.runWithUnirest(ConnectionConfigUnirestRunner.java:60)
        at com.fortify.cli.common.rest.unirest.runner.AbstractAuthSessionUnirestRunner.runWithUnirest(AbstractAuthSessionUnirestRunner.java:61)
        at com.fortify.cli.common.rest.unirest.runner.AbstractAuthSessionUnirestRunner.runWithUnirest(AbstractAuthSessionUnirestRunner.java:81)
        at com.fortify.cli.sc_dast.rest.unirest.runner.SCDastUnirestRunner.runWithUnirest(SCDastUnirestRunner.java:48)
        at com.fortify.cli.sc_dast.command.AbstractSCDastUnirestRunnerCommand.run(AbstractSCDastUnirestRunnerCommand.java:57)
        at picocli.CommandLine.executeUserObject(CommandLine.java:1939)
        at picocli.CommandLine.access$1300(CommandLine.java:145)
        at picocli.CommandLine$RunLast.executeUserObjectOfLastSubcommandWithSameParent(CommandLine.java:2352)
        at picocli.CommandLine$RunLast.handle(CommandLine.java:2346)
        at picocli.CommandLine$RunLast.handle(CommandLine.java:2311)
        at picocli.CommandLine$AbstractParseResultHandler.execute(CommandLine.java:2179)
        at picocli.CommandLine.execute(CommandLine.java:2078)
        at com.fortify.cli.common.picocli.executor.CommandLineExecutor.execute(CommandLineExecutor.java:74)
        at com.fortify.cli.app.FortifyCLI.execute(FortifyCLI.java:72)
        at com.fortify.cli.app.FortifyCLI.main(FortifyCLI.java:59)

Null-checks need to be improved, and most likely some of the options should be listed as required options as this command can't function if for example no scan settings id has been specified.

• Any other subcommands?

Support login to SSC & FoD with the additional capabilities:

• auto login renewal
• auto encryption of sensitive data. (eg: credentials, api tokens)
• switch login sessions

This is a feature request to add support for running Sonatype scans. In addition to just running the Sonatype scan, maybe we can also incorporate some logic from the SourceAndLibScanner so that we can get Susceptibility Analysis results as well?

Tasks:

• HTTP response codes:

  • Properly handle non-20x HTTP responses
  • Non-20x responses should result in non-zero exit code
  • Error data (body, status code, ...) should be returned in the format specified by --format option if possible

• Exception handling:

  • Review use of @SneakyThrows, do we want to use this?
  • Is there nay need to have a picocli exception handler, for example to provide more user-friendly error messages?
  • Make sure that exceptions result in non-zero exit code
  • Make sure that exceptions are also written to any logs

List should show which version of a tool is installed.

If there's any error deserializing or processing the auth session data (including auth session expired and --allow-renew not specified), then this error will prevent the session data to be removed, thereby making it impossible to perform logout or login operations for that session name; the only work-around is to manually remove the session data file from .fortify/fcli/authSessions/...

• List command already available, but could possibly be enhanced, providing more output
• Delete command should delete all application versions under an application
• Any useful data that could be updated through an update command? App description, ...?

Based on @AlphaFeature and @RequiresProduct annotations, some commands may be disabled. The following improvements still need to be implemented:

• Currently when a subcommand is disabled, picocli simply doesn't recognize it. It would be better to make those commands hidden, and maybe replace the root node of a disabled tree with a hidden runnable command that generates an appropriate error message stating that the command is not available due to being disabled
• If a disabled command doesn't have any siblings, then it's parent command should also be disabled. This should be done recursively, potentially resulting in a top-level command disappearing.
• Commands that we currently haven't implemented should be annotated with @AlphaFeature (although most of those have no runnable children so would automatically disappear once the task listed above has been implemented)

as seen in https://github.com/fortify-ps/fcli/blob/3279ccec8bb97c8af43caa5ece0d2f8ab2bc9a73/fcli-ssc/src/main/java/com/fortify/cli/ssc/picocli/command/crud/get/version/SSCGetVersionAttributesCommand.java#L55

• List and delete commands already available
• Single command for both local and LDAP users, or separate commands?

• Listing scan settings is already possible
• Need the remaining operations to be implemented
• Any others apart from create/delete/update?

• List command already available
• Create and update commands can reuse some functionality from appversion-attribute and appversion-user to update attributes and assigned users

Support starting ScanCentral SAST with package and MBS.

This issue is low priority.
Tested the Linux native binary (and jar) and it appears that the output from the table output format needs an additional line break. See screenshot:

Not experiencing this behavior on windows.

• Status command is already partially available, but should also support the new endpoint for showing SSC processing status
• Other commands yet to be implemented