Git Product home page Git Product logo

deepfence / threatmapper Goto Github PK

View Code? Open in Web Editor NEW
4.6K 58.0 568.0 221.04 MB

Open source cloud native security observability platform. Linux, K8s, AWS Fargate and more.

Home Page: https://deepfence.io

License: Apache License 2.0

Shell 0.79% Dockerfile 0.18% Go 42.81% HTML 0.18% HCL 0.09% Mustache 0.03% Makefile 0.20% JavaScript 0.03% Python 0.06% C 0.03% CSS 0.05% Smarty 0.05% TypeScript 55.45% PLpgSQL 0.06%
cloud-native vulnerability-management threat-analysis devsecops secops registry-scanning security-tools cwpp observability cloudsecurity

threatmapper's Introduction

Deepfence Logo

GitHub license GitHub stars Hacktoberfest GitHub issues Documentation Demo Docker pulls Slack Twitter

๐ŸŽ‰ Announcing ThreatMapper v2

ThreatMapper versions v1.x are depreciated. Please upgrade to the latest version.

ThreatMapper - Runtime Threat Management and Attack Path Enumeration for Cloud Native

Deepfence ThreatMapper hunts for threats in your production platforms, and ranks these threats based on their risk-of-exploit. It uncovers vulnerable software components, exposed secrets and deviations from good security practice. ThreatMapper uses a combination of agent-based inspection and agent-less monitoring to provide the widest possible coverage to detect threats.

With ThreatMapper's ThreatGraph visualization, you can then identify the issues that present the greatest risk to the security of your applications, and prioritize these for planned protection or remediation.

When to use ThreatMapper

ThreatMapper carries on the good 'shift left' security practices that you already employ in your development pipelines. It continues to monitor running applications against emerging software vulnerabilities, and monitors the host and cloud configuration against industry-expert benchmarks.

Use ThreatMapper to provide security observability for your production workloads and infrastructure, across cloud, kubernetes, serverless (Fargate) and on-prem platforms.

Planning your Deployment

ThreatMapper consists of two components:

  • The ThreatMapper Management Console is a container-based application that can be deployed on a single docker host or in a Kubernetes cluster.
  • ThreatMapper monitors running infrastructure using agentless Cloud Scanner tasks and agent-based Sensor Agents

The Management Console

You deploy the Management Console first, on a suitable docker host or Kubernetes cluster. For example, on Docker:

# Docker installation process for ThreatMapper Management Console

wget https://github.com/deepfence/ThreatMapper/raw/release-2.2/deployment-scripts/docker-compose.yml
docker-compose -f docker-compose.yml up --detach

Once the Management Console is up and running, you can register an admin account and obtain an API key.

Cloud Scanner tasks

ThreatMapper Cloud Scanner tasks are responsible for querying the cloud provider APIs to gather configuration and identify deviations from compliance benchmarks.

The task is deployed using a Terraform module. The ThreatMapper Management Console will present a basic configuration that may be deployed with Terraform, or you can refer to the expert configurations to fine-tune the deployment (AWS, Azure, GCP).

Sensor Agents

Install the sensor agents on your production or development platforms. The sensors report to the Management Console; they tell it what services they discover, provide telemetry and generate manifests of software dependencies.

The following production platforms are supported by ThreatMapper sensor agents:

  • Kubernetes: ThreatMapper sensors are deployed as a daemonset in the Kubernetes cluster, using a helm chart.
  • Docker: ThreatMapper sensors are deployed as a lightweight container.
  • Amazon ECS: ThreatMapper sensors are deployed as a daemon service using a task definition.
  • AWS Fargate: ThreatMapper sensors are deployed as a sidecar container, using a task definition.
  • Bare-Metal or Virtual Machines: ThreatMapper sensors are deployed within a lightweight Docker runtime.

For example, run the following command to start the ThreatMapper sensor on a Docker host:

docker run -dit \
    --cpus=".2" \
    --name=deepfence-agent \
    --restart on-failure \
    --pid=host \
    --net=host \
    --log-driver json-file \
    --log-opt max-size=50m \
    --privileged=true \
    -v /sys/kernel/debug:/sys/kernel/debug:rw \
    -v /var/log/fenced \
    -v /var/run/docker.sock:/var/run/docker.sock \
    -v /:/fenced/mnt/host/:ro \
    -e USER_DEFINED_TAGS="" \
    -e MGMT_CONSOLE_URL="---CONSOLE-IP---" \
    -e MGMT_CONSOLE_PORT="443" \
    -e DEEPFENCE_KEY="---DEEPFENCE-API-KEY---" \
    quay.io/deepfenceio/deepfence_agent_ce:2.2.0

Note: Image tag quay.io/deepfenceio/deepfence_agent_ce:2.2.0-multiarch is supported in amd64 and arm64/v8 architectures.

On a Kubernetes platform, the sensors are installed using helm chart

Next Steps

Visit the Deepfence ThreatMapper Documentation, to learn how to get started and how to use ThreatMapper.

Get in touch

Thank you for using ThreatMapper. Please feel welcome to participate in the ThreatMapper Community.

Get ThreatStryker for Enterprise

ThreatStryker is the enterprise version of ThreatMapper, with additional features for enterprise security teams. ThreatStryker is available as a cloud service or for on-premises deployment.

Security and Support

For any security-related issues in the ThreatMapper project, contact productsecurity at deepfence dot io.

Please file GitHub issues as needed, and join the Deepfence Community Slack channel.

License

The Deepfence ThreatMapper project (this repository) is offered under the Apache2 license.

Contributions to Deepfence ThreatMapper project are similarly accepted under the Apache2 license, as per GitHub's inbound=outbound policy.

Performance Stats of deepfence/ThreatMapper - Last 28 days

Performance Stats of deepfence/ThreatMapper - Last 28 days

threatmapper's People

Contributors

akumars1 avatar ansalamdaniel avatar cupcakesprinkle3 avatar dependabot[bot] avatar deviprasad303 avatar fferrann avatar github-actions[bot] avatar gnmahanth avatar harishshg avatar ibreakthecloud avatar j771 avatar jatin-baweja avatar jo19in1 avatar juneezee avatar manv avatar milan-deepfence avatar mukuldeepfence avatar noboruma avatar ogarrett avatar pandyamarut avatar ramanan-ravi avatar saiprasanth1303 avatar sandman137 avatar sangam14 avatar saurabh2253 avatar shahpratikr avatar shubhamjain32 avatar shyam-dev avatar sidd0529 avatar varunsharma0286 avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

threatmapper's Issues

Unable to navigate from 'Topology'

Hi,

I've setup the management console on an AWS instance using the repository instructions. I've registered an account and accessed the console but I can't navigate to any other pages (registries, vulnerabilities, settings etc.). It seems to be in a loop of 'resuming the live state' and is stuck 'Optimizing hosts'.

I've given it over 60 minutes as the readme mentioned, but there's been no change.

I can't find any obvious problems looking through the various container logs. I wondered if someone here could offer any guidance?

Update README with console sizing requirement

Now that we are removing host node counts + also adding some optimisations on UI side, we need to add the console size requirements for various scale levels assuming user is always using 1 node kubernetes cluster or docker compose.

Need following numbers:

Console size

8 core - 16G RAM - how many hosts, unlimited number of containers
16 core - 16G RAM - how many hosts, unlimited number of containers
8 core - 32G RAM - how many hosts, unlimited number of containers
16 core - 32G RAM - how many hosts, unlimited number of containers

Issue scanning Management node and registries

Unable to run vulnerability scans on management console node and registries. Vulnerability scans start but hang after a while with error "Scan was interrupted". There is no info in the docker-compose logs to investigate or viewable logs from the console itself.

ServiceNow integration

Got a query on community channel on feasibility of adding this, should be straight forward based on my cursory analysis.

Trying to get in touch regarding a security issue

Hey there!

I'd like to report a security issue but cannot find contact instructions on your repository.

If not a hassle, might you kindly add a SECURITY.md file with an email, or another contact method? GitHub recommends this best practice to ensure security issues are responsibly disclosed, and it would serve as a simple instruction for security researchers in the future.

Thank you for your consideration, and I look forward to hearing from you!

(cc @huntr-helper)

How to modify admin user or create other one.

Intalled deepfence and create the first registration... with an error in the email. Try to change but it doesn't have any button to modify only delete but if you try to delete admin you cant do that. If I want to create other admin user is not possible or tranfers de admin role to other user is not possible.

web UI up but no 80 or 443 binding after server reboot

Hi I installed the community edition and work fine. After a linux upgrade and patch I had to reboot the server (including put the server certs) and after that reboot docker ps shows that deepfence-ui hasn't any 80 or 443 binding. I run docker-compose down and up again and same result no web ui is bind to 80 or 443 ports

pulling deepfence_agent_ce:latest is failing

Problem:

getting error pulling image configuration: unknown blob when trying to pull latest version

sudo docker pull deepfenceio/deepfence_agent_ce:latest
Trying to pull repository docker.io/deepfenceio/deepfence_agent_ce ... 
sha256:ef2bfb042c1947d7fa1521c92d80d48b427ec1921a7a8bf719a4da1bfe41a2d5: Pulling from docker.io/deepfenceio/deepfence_agent_ce
c9b1b535fdd9: Pulling fs layer 
a467aaeebe51: Pulling fs layer 
4972601ee06f: Pulling fs layer 
50fea360ba80: Waiting 
12cb344c186e: Waiting 
7fcfa6d4777d: Waiting 
88e4e5d70d35: Waiting 
09111211c518: Waiting 
02367567fd52: Waiting 
14db001a3bcd: Waiting 
b3f9fa29ccb1: Waiting 
9f2c31a673b2: Waiting 
37a51eba9536: Waiting 
d58fe0077a61: Waiting 
5007810f217c: Waiting 
f3da2746ca1d: Waiting 
4d0e5e4a4a1a: Waiting 
Trying to pull repository registry.fedoraproject.org/deepfenceio/deepfence_agent_ce ... 
Trying to pull repository registry.access.redhat.com/deepfenceio/deepfence_agent_ce ... 
Trying to pull repository registry.centos.org/deepfenceio/deepfence_agent_ce ... 
Trying to pull repository quay.io/deepfenceio/deepfence_agent_ce ... 
Trying to pull repository docker.io/deepfenceio/deepfence_agent_ce ... 
sha256:ef2bfb042c1947d7fa1521c92d80d48b427ec1921a7a8bf719a4da1bfe41a2d5: Pulling from docker.io/deepfenceio/deepfence_agent_ce
c9b1b535fdd9: Pulling fs layer 
a467aaeebe51: Pulling fs layer 
4972601ee06f: Pulling fs layer 
50fea360ba80: Waiting 
12cb344c186e: Waiting 
7fcfa6d4777d: Waiting 
88e4e5d70d35: Waiting 
09111211c518: Waiting 
02367567fd52: Waiting 
14db001a3bcd: Waiting 
b3f9fa29ccb1: Waiting 
9f2c31a673b2: Waiting 
37a51eba9536: Waiting 
d58fe0077a61: Waiting 
5007810f217c: Waiting 
f3da2746ca1d: Waiting 
4d0e5e4a4a1a: Waiting 
error pulling image configuration: unknown blob

pulling bottlerocket version works as expected

sudo docker pull deepfenceio/deepfence_agent_ce:bottlerocket
Trying to pull repository docker.io/deepfenceio/deepfence_agent_ce ... 
sha256:1caea67efebbebd83dc09f869ac66c0bd99f2b4a7ec2d80bfe80042fe1f1786a: Pulling from docker.io/deepfenceio/deepfence_agent_ce
c9b1b535fdd9: Pull complete 
b6d83a030696: Pull complete 
f6a06778061f: Pull complete 
4346f63f6540: Pull complete 
9621f77de183: Pull complete 
ea774dd0a492: Pull complete 
bc61a45c72b5: Pull complete 
3d194fa3baa1: Pull complete 
3ab4871f49df: Pull complete 
58a3160a4149: Pull complete 
737c91d96cd4: Pull complete 
37db6d33e67d: Pull complete 
25ff9d31fb02: Pull complete 
19c622447144: Pull complete 
2dd6c587aacb: Pull complete 
28634e02c7dd: Pull complete 
d06993b63779: Pull complete 
Digest: sha256:1caea67efebbebd83dc09f869ac66c0bd99f2b4a7ec2d80bfe80042fe1f1786a
Status: Downloaded newer image for docker.io/deepfenceio/deepfence_agent_ce:bottlerocket

Most exploitable vulnerabilities & nodes heatmap

Currently we show a ranked list most exploitable vulnerabilities across running hosts and containers as a table. Would be good to add a heat map or a split bubble chart showing the same with added filters.

Remove traces of enterprise edition from ThreatMapper

  1. Remove host node count limit, on UI as well as backend

  2. No realtime node count tracking for hosts. Containers and images are already unlimited.

  3. Remove work email requirement while registering

  4. Remove menu items which do not belong in threatmapper like compliance etc

Enable runtime scanning at cluster, namespace and user defined tags level

Currently only container, container images and host level scanning is support. With this feature user should be able to scan at following granularities with a single click

  • an entire kubernetes cluster

  • a certain namespace

  • user defined tags which could contains multiple hosts e.g. dev/staging/prod etc

Router does start without operating mode environment variable

Hi.
I had the control working a few days ago using the docker-compose file in this repository but I tried to create a control node today and the router fails with the following message.

deepfence-router                  | Set operating_mode to either 'docker' or 'k8s'. Exiting...
deepfence-router                  | Set operating_mode to either 'docker' or 'k8s'. Exiting...
deepfence-router                  | Set operating_mode to either 'docker' or 'k8s'. Exiting...

I had to add the following to the docker-compose deepfence-router block file

   environment:
      OPERATING_MODE: "docker"

I am deploying on a GCE instance with Debian 10 via a startup script

New feature request - GitHub Actions plugin

In addition to Jenkins, CircleCI and GitLab plugins, I would like one for GitHub Actions. Maybe it's better to make Threatmapper available in the GitHub Actions marketplace, then developing a plugin, I'm not sure. Could someone make Threatmapper compatible with GitHub Actions?

deepfence-api and deepfence-ui problem

Problem:
cannot switch tabs inside deepfence-ui, ui stuck at loading topology page with "verifying topologies message"

Expected:
to be able to browse the ui normally

deepfence-api_logs.txt.gz
2020-10-07_10-56

truncated deepfence-api container logs below

[2020-10-07 07:27:41 +0000] [97] [ERROR] Exception on /deepfence/v1.5/node_status [GET]
Traceback (most recent call last):
  File "/usr/local/lib/python3.6/site-packages/flask/app.py", line 2447, in wsgi_app
    response = self.full_dispatch_request()
  File "/usr/local/lib/python3.6/site-packages/flask/app.py", line 1952, in full_dispatch_request
    rv = self.handle_user_exception(e)
  File "/usr/local/lib/python3.6/site-packages/flask_restful/__init__.py", line 272, in error_router
    return original_handler(e)
  File "/usr/local/lib/python3.6/site-packages/flask_restful/__init__.py", line 272, in error_router
    return original_handler(e)
  File "/usr/local/lib/python3.6/site-packages/flask_restful/__init__.py", line 272, in error_router
    return original_handler(e)
  File "/usr/local/lib/python3.6/site-packages/flask_cors/extension.py", line 161, in wrapped_function
    return cors_after_request(app.make_response(f(*args, **kwargs)))
  File "/usr/local/lib/python3.6/site-packages/flask/app.py", line 1821, in handle_user_exception
    reraise(exc_type, exc_value, tb)
  File "/usr/local/lib/python3.6/site-packages/flask/_compat.py", line 39, in reraise
    raise value
  File "/usr/local/lib/python3.6/site-packages/flask/app.py", line 1950, in full_dispatch_request
    rv = self.dispatch_request()
  File "/usr/local/lib/python3.6/site-packages/flask/app.py", line 1936, in dispatch_request
    return self.view_functions[rule.endpoint](**req.view_args)
  File "api/common_api.py", line 1424, in api.common_api.get_node_status
  File "api/common_api.py", line 1433, in api.common_api.determine_node_status
  File "api/common_api.py", line 1552, in api.common_api.get_topology_cve_status
  File "utils/esconn.py", line 2823, in utils.esconn.ESConn.group_by
  File "utils/esconn.py", line 503, in utils.esconn.ESConn.search
  File "/usr/local/lib/python3.6/site-packages/elasticsearch/client/utils.py", line 92, in _wrapped
    return func(*args, params=params, headers=headers, **kwargs)
  File "/usr/local/lib/python3.6/site-packages/elasticsearch/client/__init__.py", line 1623, in search
    body=body,
  File "/usr/local/lib/python3.6/site-packages/elasticsearch/transport.py", line 362, in perform_request
    timeout=timeout,
  File "/usr/local/lib/python3.6/site-packages/elasticsearch/connection/http_urllib3.py", line 252, in perform_request
    self._raise_error(response.status, raw_data)
  File "/usr/local/lib/python3.6/site-packages/elasticsearch/connection/base.py", line 282, in _raise_error
    status_code, error_message, additional_info
elasticsearch.exceptions.RequestError: RequestError(400, 'illegal_argument_exception', 'field name is null or empty')
ERROR:config.df_app:Exception on /deepfence/v1.5/node_status [GET]
Traceback (most recent call last):

Enable threat mapping for AWS Fargate

Add instructions to deploy threatmapper as a sidecar + embedding. This depends on #43 as all binaries must be agnostic to underlying distro i.e. static binaries only.

Feature -Request integration with OpenCTI

Additional context
ThreatMapper is missing features used to connect to an open-source Cyber Threat Intelligence Platform, such as OpenCTI. This feature request can be used used to map vulnerabilities exploited by known malicious actors (i.e APT). Additionally,

Is your feature request related to a problem? Please describe.
Similar to Anomali, or ThreatConnect, a novice CTI analyst will have difficulties mapping CRITICAL/HIGH vulnerabilities to a malicious actor (i.e. APT, or UNC) without integration between ThreatMapper and a CTI platform.

Describe the solution you'd like
This feature enhancement would allow CTI analysts to prioritize and rank vulnerabilities exploited, and security advisories warning to patch vulnerabilities commonly propagated by malicious actors

Describe alternatives you've considered

MITRE ATT&CK
ThreatConnect
Anomali ThreatStrem
Securonix

Components/Services

  • UI/Frontend
  • API/Backend
  • Agent
  • Deployment/YAMLs
  • CI/CD Integration
  • Other (specify) - Support for OpenCTI Connectors

Additional context
https://www.opencti.io/en/
https://github.com/OpenCTI-Platform
https://github.com/OpenCTI-Platform/connectors

Cannot compose the docker yaml file

root@ubuntu-s-6vcpu-16gb-nyc1-01:~# sudo docker-compose -f ./docker-compose.yml up -d
ERROR: yaml.scanner.ScannerError: mapping values are not allowed here
in "././docker-compose.yml", line 119, column 73

Installation of Management Console Fails

Provided the docer-compose.yml file, when executing the following command:

docker-compose -f docker-compose.yml up -d

I am unable to browse to the management console as not all containers are started, specifically the deepfence-es-master container which presents the following errors:

Exception in thread "main" java.lang.RuntimeException: starting java failed with [137]
output:

error:
OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
	at org.elasticsearch.tools.launchers.JvmErgonomics.flagsFinal(JvmErgonomics.java:123)
	at org.elasticsearch.tools.launchers.JvmErgonomics.finalJvmOptions(JvmErgonomics.java:88)
	at org.elasticsearch.tools.launchers.JvmErgonomics.choose(JvmErgonomics.java:59)
	at org.elasticsearch.tools.launchers.JvmOptionsParser.main(JvmOptionsParser.java:95)

Would appreciate any guidance.

Allow using self signed certificates

Currently we use a self-signed certificate for Deepfence UI. User might want to use their own self signed certificates due to various reasons.

If user specifies path to a dir containing certs, we should volume mount and use those certs instead of deepfence self signed certs.

Image and tag search capability per registry

If a certain container registry has > 1K images, it gets tricky to search, sort and scan images due to current rigid table structure. This gets amplified when more registries are added and each has more than a few hundred images.

Need to:

  1. Add a per registry search box (free text search with autocomplete) so users can search per based on container image name AND/OR image tag

  2. While we are at this, lets also make tables dynamically paginated based on number of images and make all columns sortable

Pre package vulnerability DB in container images

Problem:

Currently, the initial vulnerability db update takes anywhere between 15-60 minutes and only after this first full update threatmapper is usable.

  • Every 24 hours we can push pre-populated vulnerability DB images to threatmapper registry so when a user uses a fresh installation of threatmapper, at the very worst a 24 hr old database is being referred for scanning. This db anyway will get updated over next few hours but reduced the initial wait required.

JIRA integration

Need to enable JIRA integration and document supported JIRA versions.

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.