checkmarx-ltd / cx-flow Goto Github PK

JIRA severity based due dates

Investigate and implement additional capabilities within GitLab Pull Request feedback

• Block Merge after scan completes, based on filter/policy criteria
• Provide diff view, to help determine if current commit/pull has introduced the finding
• Source snippet highlights as a comment. This would need to occur at the commit level (i.e. Violation comments as a reviewer on the file/commit). Checkmarx GL Service account would be a "reviewer" of the code. References:

https://wiki.jenkins.io/display/JENKINS/Violation+Comments+to+Bitbucket+Server+Plugin

Reflect scan details in Pull/Merge feedback within already existing comments if a scan has triggered a second time. This is applicable to all repository types.

Sample implementation:
https://github.com/danger/danger/blob/56286aab4d3b9644eb6da5e81d0ab5b32575e9d4/lib/danger/comment_generators/github_inline.md.erb#L3

In addition, I think there should be some options for configuring how additional commits added to an existing PR are handled. Right now, if a PR is created, every time additional commit(s) are pushed to the PR HEAD, a new "scan submitted" comment is added and, more importantly, a new scan is queued. The latter means that the scan queue can very quickly become completely consumed by scan requests for a single PR of a single project, especially if the project/PR consists of a large code base and/or requires a full scan. In our current production environment, a single project has one PR that has resulted in 30 additional scans in the queue; this is made worse in our specific case because this particular project has > 1M lines of code and can only be handled by a small subset of our scan engines.

I'd propose handling this by updating CxFlow's PR processing behavior to do one or more of the following:

If a scan request exceeds the configured timeout in CxFlow, optionally cancel the actual scan itself.
Allow setting a configurable per-project webhook queue limit. Behavior might be that the queue is handled such that existing scan closest to completion/greatest progress % is left in queue, and then remaining queue size is allocated with most recent scan request prioritized. In other words, attempt to minimize time-to-results (by ensuring first/closest to completion scan is not canceled) while also ensuring most recently committed changes are added to the scan queue and older scans that have been queued but not started are canceled to make room for newest commits.

Create pipeline of automated integration testing for validating various 3rd party APIs are working as desired

• GitHub
• GitLab
• BitBucket
• AzureDevOps
• JIRA

Introduce a UI for managing the configuration of CxFlow in a user-friendly way

Issue link from the previous repo: https://github.com/checkmarx-ts/cx-flow/issues/39

Introduce the capability of externalizing credential injection. Some examples are:

• HashiCorp's Vault
• KeyStore
• AWS SSM Parameter Store
• Many more

Issue link from previous repo : https://github.com/checkmarx-ts/cx-flow/issues/40

When the scan completes and 0 vulnerabilites of any severity are found, IF the breakbuild flag is set to true in the config file, we get an error message (Code 10).

If breakbuild=false, then we get past that error message. Please fix so when breakbuild=true AND there are 0 results, the process completes successfully.

Investigate project cleanup strategy. Possibilities include:

• CLI driven
• WebHook event drive (Branch delete event)
• Time based

Checkmarx version: 8.9.0 HF13 (SDLC)
CX-flow version: 1.5.4

When having a project already created in Checkmarx SAST, and CXflow is setup to generate scans on Pull Requests, the Checkmarx preset specified in the Pull Request Webhook parameter is never used. Instead, the preset defined in the application.yml is always used.

Use case:
Say that we have a CXFlow instance listening to Github PRs that triggers scans. The "scan-preset" property in application.yml not specified (intentionally), so that one can override the preset in the WebHook Parameter (as specified in GitHubController.Java in method:
public ResponseEntity pullRequest

Observations:

the "override" is never used (greyed out by the IDE)
No matter what preset I specify in the WebHook parameter, the Checkmarx Default is always used
While debugging through the code, I noticed that in file FlowService.java, method
private Integer determinePresetAndProjectId(ScanRequest request, String ownerId, String projectName ) {

it seems that the check is wrongly done (unless I understood #the code logic wrong), and the method fetches and sets the project's preset if the request has the scanPresetOverride as false.
image

By removing the negation before the "request.isScanPresetOverride()", it works as expected and the scan will use the project's defined Preset (that was set from within the Web Interface).

I might have not understood the code logic, or maybe this is just a bug and the negation needs to be removed.

Currently CxFlow publishes/updates findings in JIRA. The ticket that is generated however does not get reflected in Checkmarx.

CxFlow should associate the ticket with the finding(s) on the Checkmarx side to allow for reconciliation and for a reference point within the Checkmarx console.

Investigate IAST Integration Flows.

Introduce Configuration as Code capability to override CxFlow settings

• GitHub - Done
• GitLab - Done
• Azure DevOps - https://dev.azure.com/{org}/_apis/git/repositories/{repoid}/items?scopePath={file}&download=true&api-version=5.1
• BitBucket Server
• BitBucket Cloud

Issue link from previous repo : https://github.com/checkmarx-ts/cx-flow/issues/36

When running Instance Batch mode, the file name substitution logic does not allow for Checkmarx team to be used in filename with the [TEAM] format.

Note: CxFlow supports 3 batch modes through CLI

Project based: The results for a single project is retrieved and published as per the defined feedback channel (file based, ticket/JIRA based).
Team based: The latest results for each project for a given team are returned and published as per the defined feedback channel)
Instance based: The latest results for each project for the entire instance of Checkmarx is returned and published as per the defined feedback channel.
For Team and Instance using a file based feedback (i.e. Json/CxXml/Csv) there is the ability to substitute parts of the filename per project ([REPO], [PROJECT]). The team ([TEAM] block is only substituted when Team batch mode is used, the team is not available to reference (at the moment) to substitute in the Instance batch mode).

Example [TEAM] substitution for Json/Xml/Csv outputs. Team is not currently available when running against the entire instance of checkmarx.

`json:
file-name-format: "[NAMESPACE]-[REPO]-[BRANCH]-[TIME].json"
data-folder: "D:\tmp"

cx-xml:
file-name-format: "[NAMESPACE]-[REPO]-[BRANCH]-[TIME].xml"
data-folder: "D:\tmp"

csv:
file-name-format: "[TEAM]-[PROJECT]-[TIME].csv"
data-folder: "D:\tmp"`

Allow for automated project creation to include assigning a Policy from M&O

Kerberos authentication support would be nice for organisations that don't allow username and passwords to be used to authenticate to JIRA.

Allow for the ability to configure defect tracking against policy violations in M&O

Add ServiceNow Support for Defect Tracking.

Add capability to scan repository for code content and finger print the application to help determine a preset/scan ruleset to use automatically

• GitHub - Done
• GitLab - Done
• Azure DevOps - https://dev.azure.com/{acct}/{project}/_apis/projectanalysis/languagemetrics
• BitBucket Server
• BitBucket Cloud

Issue link from previous repo : https://github.com/checkmarx-ts/cx-flow/issues/37

Add Dashboard Integration Support for Result Ingestion

• Splunk
• ELK
• Other

The logic used to determine the link to the exact line of source in within the repository will break for BitBucket when the branch contains a forward slash (i.e. feature/branch).

Add the ability to block merge based on scan results filter for:

• GitHub - Done
• GitLab
• Azure DevOps

Investigate and implement additional capabilities within BitBucket Pull Request feedback

• Block Merge after scan completes, based on filter/policy criteria
• Provide diff view, to help determine if current commit/pull has introduced the finding
• Source snippet highlights as a comment. This would need to occur at the commit level (i.e. Violation comments as a reviewer on the file/commit). Checkmarx BB Service account would be a "reviewer" of the code. References:

https://wiki.jenkins.io/display/JENKINS/Violation+Comments+to+Bitbucket+Server+Plugin

Archer is a commonly used Risk Management Platform that we could push vulnerabilities into natively from CxFlow. I'm unfamiliar with the customization's that the platform has but if organisations have different requirements we should be able to support them.

Introduce Checkmarx Project Branching capability within CxFlow

Currently, a new project is created for every repo/branch combination unless project naming overrides are leveraged. This Enhancement is to leverage a base project and branch it within Checkmarx instead of creating new each time.

Issue link from previous repo: https://github.com/checkmarx-ts/cx-flow/issues/31

->We have integrated Cx-Flow scan with Gitlab CI/CD pipeline. Even after marking few HIGH vulnerabilities as Not exploitable in Cx Console, Gitlab CI/CD pipeline fails. We are using Cx-Flow version of 1.5.4 (latest version).
->We also added flag break-build=true (as suggested by Support Engineer) and tested it again. But its still failing.

Publish compiled versions of CxFlow via standard locations (such as Maven Central).

i.e. OSS Repository Hosting provided by Sonatype:
https://central.sonatype.org/pages/ossrh-guide.html

Add the ability to leverage templating for Defect / Issue description

Investigate the use of Velocity template language:
https://velocity.apache.org/

Investigate and implement additional capabilities within ADO Pull Request feedback

• Block Merge after scan completes, based on filter/policy criteria
• Provide diff view, to help determine if current commit/pull has introduced the finding
• Source snippet highlights as a comment. This would need to occur at the commit level (i.e. Violation comments as a reviewer on the file/commit). Checkmarx ADO Service account would be a "reviewer" of the code. References:

https://wiki.jenkins.io/display/JENKINS/Violation+Comments+to+Bitbucket+Server+Plugin

Allow for external script (Groovy) for Issue Descriptions

Where do we envision the configuration for this Groovy script to be?

I suggest under cx-flow or/and per bug tracker implementation to allow different issue descriptions per service (the second would take priority). This would be an optional property.

The per implementation is more for cases where organisations have multiple different services which require possibly their own description format for each service and don't need two separate CxFlow instances to achieve this. This could also be achieved if we pass in the bug tracking system into the Groovy data binding as run-time.

Proposed Example:

cx-flow:
// ...
// The name of the property is subject to change
script-issueDescription: "IssueDescription.groovy"

Add ability to link issues in JIRA

Leverage the GitHub Wiki to document CxFlow with public-facing details.

Allow for external Script (Groovy) for JIRA custom fields

Describe the problem

Customers using CxFlow in CLI mode through GitLab CI integration would like to leverage the Security Dashboard capabilities.

Proposed solution

Create bug tracker implementation forGitLabDashboard, which is a specific JSON output format.

Additional details

https://docs.gitlab.com/ee/user/application_security/security_dashboard/

When ADO PR Payload is sent to the Push (/ado/push) endpoint, CxFlow has an NPE.

2020-02-19 13:27:31.565 ERROR 2276 --- [io-11111-exec-7] o.a.c.c.C.[.[.[/].[dispatcherServlet] : Servlet.service() for servlet [dispatcherServlet] in context with path [] threw exception [Request processing failed; nested exception is java.lang.NullPointerException] with root cause

java.lang.NullPointerException: null
at com.checkmarx.flow.controller.ADOController.pushRequest(ADOController.java:322) ~[classes!/:na]
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[na:1.8.0_211]
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[na:1.8.0_211]
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:1.8.0_211]
at java.lang.reflect.Method.invoke(Method.java:498) ~[na:1.8.0_211]
at org.springframework.web.method.support.InvocableHandlerMethod.doInvoke(InvocableHandlerMethod.java:190) ~[spring-web-5.2.3.RELEASE.jar!/:5.2.3.RELEASE]
at org.springframework.web.method.support.InvocableHandlerMethod.invokeForRequest(InvocableHandlerMethod.java:138) ~[spring-web-5.2.3.RELEASE.jar!/:5.2.3.RELEASE]
at org.springframework.web.servlet.mvc.method.annotation.ServletInvocableHandlerMethod.invokeAndHandle(ServletInvocableHandlerMethod.java:106) ~[spring-webmvc-5.2.3.RELEASE.jar!/:5.2.3.RELEASE]
at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.invokeHandlerMethod(RequestMappingHandlerAdapter.java:888) ~[spring-webmvc-5.2.3.RELEASE.jar!/:5.2.3.RELEASE]
at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.handleInternal(RequestMappingHandlerAdapter.java:793) ~[spring-webmvc-5.2.3.RELEASE.jar!/:5.2.3.RELEASE]
at org.springframework.web.servlet.mvc.method.AbstractHandlerMethodAdapter.handle(AbstractHandlerMethodAdapter.java:87) ~[spring-webmvc-5.2.3.RELEASE.jar!/:5.2.3.RELEASE]
at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:1040) ~[spring-webmvc-5.2.3.RELEASE.jar!/:5.2.3.RELEASE]
at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:943) ~[spring-webmvc-5.2.3.RELEASE.jar!/:5.2.3.RELEASE]
at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:1006) ~[spring-webmvc-5.2.3.RELEASE.jar!/:5.2.3.RELEASE]
at org.springframework.web.servlet.FrameworkServlet.doPost(FrameworkServlet.java:909) ~[spring-webmvc-5.2.3.RELEASE.jar!/:5.2.3.RELEASE]
at javax.servlet.http.HttpServlet.service(HttpServlet.java:660) ~[tomcat-embed-core-9.0.30.jar!/:9.0.30]
at org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:883) ~[spring-webmvc-5.2.3.RELEASE.jar!/:5.2.3.RELEASE]
at javax.servlet.http.HttpServlet.service(HttpServlet.java:741) ~[tomcat-embed-core-9.0.30.jar!/:9.0.30]
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:231) ~[tomcat-embed-core-9.0.30.jar!/:9.0.30]
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) ~[tomcat-embed-core-9.0.30.jar!/:9.0.30]
at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:53) ~[tomcat-embed-websocket-9.0.30.jar!/:9.0.30]
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) ~[tomcat-embed-core-9.0.30.jar!/:9.0.30]
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) ~[tomcat-embed-core-9.0.30.jar!/:9.0.30]
at org.springframework.web.filter.RequestContextFilter.doFilterInternal(RequestContextFilter.java:100) ~[spring-web-5.2.3.RELEASE.jar!/:5.2.3.RELEASE]
at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:119) ~[spring-web-5.2.3.RELEASE.jar!/:5.2.3.RELEASE]
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) ~[tomcat-embed-core-9.0.30.jar!/:9.0.30]
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) ~[tomcat-embed-core-9.0.30.jar!/:9.0.30]
at org.springframework.web.filter.FormContentFilter.doFilterInternal(FormContentFilter.java:93) ~[spring-web-5.2.3.RELEASE.jar!/:5.2.3.RELEASE]
at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:119) ~[spring-web-5.2.3.RELEASE.jar!/:5.2.3.RELEASE]
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) ~[tomcat-embed-core-9.0.30.jar!/:9.0.30]
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) ~[tomcat-embed-core-9.0.30.jar!/:9.0.30]
at org.springframework.boot.actuate.metrics.web.servlet.WebMvcMetricsFilter.doFilterInternal(WebMvcMetricsFilter.java:108) ~[spring-boot-actuator-2.2.4.RELEASE.jar!/:2.2.4.RELEASE]
at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:119) ~[spring-web-5.2.3.RELEASE.jar!/:5.2.3.RELEASE]
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) ~[tomcat-embed-core-9.0.30.jar!/:9.0.30]
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) ~[tomcat-embed-core-9.0.30.jar!/:9.0.30]
at org.springframework.web.filter.CharacterEncodingFilter.doFilterInternal(CharacterEncodingFilter.java:201) ~[spring-web-5.2.3.RELEASE.jar!/:5.2.3.RELEASE]
at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:119) ~[spring-web-5.2.3.RELEASE.jar!/:5.2.3.RELEASE]
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) ~[tomcat-embed-core-9.0.30.jar!/:9.0.30]
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) ~[tomcat-embed-core-9.0.30.jar!/:9.0.30]
at com.checkmarx.flow.filter.SecurityFilter.doFilter(SecurityFilter.java:41) ~[classes!/:na]
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) ~[tomcat-embed-core-9.0.30.jar!/:9.0.30]
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) ~[tomcat-embed-core-9.0.30.jar!/:9.0.30]
at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:202) ~[tomcat-embed-core-9.0.30.jar!/:9.0.30]
at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:96) [tomcat-embed-core-9.0.30.jar!/:9.0.30]
at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:541) [tomcat-embed-core-9.0.30.jar!/:9.0.30]
at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:139) [tomcat-embed-core-9.0.30.jar!/:9.0.30]
at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:92) [tomcat-embed-core-9.0.30.jar!/:9.0.30]
at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:74) [tomcat-embed-core-9.0.30.jar!/:9.0.30]
at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:343) [tomcat-embed-core-9.0.30.jar!/:9.0.30]
at org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:367) [tomcat-embed-core-9.0.30.jar!/:9.0.30]
at org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:65) [tomcat-embed-core-9.0.30.jar!/:9.0.30]
at org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:860) [tomcat-embed-core-9.0.30.jar!/:9.0.30]
at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1598) [tomcat-embed-core-9.0.30.jar!/:9.0.30]
at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49) [tomcat-embed-core-9.0.30.jar!/:9.0.30]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [na:1.8.0_211]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [na:1.8.0_211]
at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61) [tomcat-embed-core-9.0.30.jar!/:9.0.30]
at java.lang.Thread.run(Thread.java:748) [na:1.8.0_211]

Introduce support for direct links to CB lessons based on findings.

Currentl CxFlow allows for a global training link that is referenced within the body of a ticket/issue, which can be the high level URL of Codebashing if desired. CxFlow should allow for deeper integration with Codebashing to introduce a link to the exact lesson that would aid in developers fixing/learning about the issue.

Allow for external Script (Groovy) for JIRA custom fields

Currently ADO Workitems only support issue description.

Enhance functionality to allow for custom fields in ADO WorkItems - introduce the same functionality as JIRA.

Enhance CxFlow to integrate with Jira in a 2-way manner. i.e. Change events within Jira are then reflected in Jira.

This would need to be built from WebHook integration in Jira that reflect changes to the issues in Checkmarx.

A listener would need to be developed for the Issue Updated webhook within JIRA:
https://developer.atlassian.com/server/jira/platform/webhooks/

Things to consider:

What can be updated? State / transitions, comments, labels?
How much can/should be configurable/customizable?