Git Product home page Git Product logo

dotnetrussell / ensemble Goto Github PK

View Code? Open in Web Editor NEW
44.0 3.0 6.0 8.36 MB

A Bug Bounty Platform that allows hunters to issue commands over a geo-distributed cluster. The ideal user is someone who is attempting to scan multiple bug bounty programs simultaneously, on a recurring basis.

Home Page: https://DotNetRussell.com

License: MIT License

Dockerfile 0.31% Shell 0.06% Python 58.26% CSS 0.35% JavaScript 3.28% HTML 37.74%
blueteam bug-bounty bug-bounty-hunting bug-bounty-recon bug-bounty-tools bugbounty hacking red-team red-team-tools red-teaming

ensemble's Introduction

Ensemble

A Bug Bounty Platform that allows hunters to issue commands over a geo-distributed cluster. The ideal user is someone who is attempting to scan multiple bug bounty programs simultaneously, on a recurring basis

Usage

For every Ensemble cluster to function you will need to have at least one director and one agent.


Installing Ensemble

Ensemble is a cluster of machines. So at a minimum you're going to want at least two. One machine to be the director and web portal that you access and another machine that is a node in your cluster. Ideally though you would have a node for every region in the world. I used Digital Ocean for setting up a global cluster but you can use any VPS provider you'd like.

This is the least amount of commands to run to start an ensemble server. The server doesn't require any extra tooling as all commands are run on the agents. Your server can and should be very light weight.

apt-get update
apt install git -y;
apt install python3
apt install pipx
git clone https://github.com/DotNetRussell/Ensemble.git;
python3 -m venv .venv
source .venv/bin/activate
python3 -m pip install -r requirements.txt

NOTE Some users have experienced an issue where flask is installed but when you run ensemble it says it's not installed. I that happens just use the command below.

apt install python3-flask


Starting an Ensemble Director Server

Ensemble Director is the master node of your cluster. It will not only be the web portal that you connect to and control your cluster with but it will also be the node that all other nodes in the cluster communicate with.

To start a director once fully installed, run the following command ./ensemble_director --config-file <see-sample-confilg>

Next, visit the IP of your director on port 5000 and create your admin account. DO THIS IMMEDIATELY AFTER STARTING THE DIRECTOR
https://127.0.0.1:5000

NOTE You will need to use https protocol. It will say it's an insecure connection because the ensemble_director just generated a new unique certificate for you and it's not registered with a certificate authority.


Creating an Ensemble Agent

Creating an ensemble agent is relatively easy. The director has generated a new symmetric key for you and the command you need to run your agent. Just visit your Ensemble Director settings page and you will find the command you need to run.

apt-get update;
apt install git -y;
apt install python3;
apt install python3-psutil;
git clone https://github.com/DotNetRussell/Ensemble.git;
cd Ensemble;

./ensemble_agent --connection-string '{"HOST":"<your host ip>","PORT":"5680","ENCRYPTION_KEY":"<your symmetric key>"}'

NOTE As soon as your agent is running, it will appear on your director web portal.


Navigating the Application

Dashboard Page

Alt text

  • The Dashboard is where you can easily see where your active agents are distributed in the world.
  • It also displays statistics about the jobs that are running and that have completed.
  • This is also where you create and switch workspaces. This is helpful for separating out bug bounties

Agents Page

Alt text

  • The agents page shows you statistics about your agents. This includes their health, if they're active or offline, and how many jobs they're running.

Agent Health Page

Alt text

  • The agent health page shows you detailed information about each agent.
  • This includes memory consumption, remaining storage, CPU usage, log file size, job history and running processes
  • You also have some control over the agent on this page

Jobs Page

Alt text

  • The jobs page shows you running and completed jobs as well as details about the jobs.
  • Clicking the magnified glass will navigate to the Job Results page which is an aggregation of all of the job output from all agents

Job Results Page

Alt text

  • The job results page shows detailed information about the job results from each agent the job ran on

Scheduled Jobs Page

Alt text

  • The scheduled jobs page shows you the jobs that have been scheduled to run both recurringly as well as at a specific date and time.
  • Jobs that run recurringly daily/weekly/monthly, will appear under the completed scheduled jobs section and you'll be able to diff the results between runs. This is nice for finding changes in a attack surface over time.

Create New Job Page

Alt text

  • Create new job page allows users to create new jobs to run instantly, run on a recurring basis, or schedule the job to run at a specific date and time
  • Running a load balanced command is best used when you expect the results will be the same regardless what region you run the command in. This will take your command and distribute it across the cluster evenly based on how many targets you have. For example, if you have 4 nodes, and only one target defined, it will issue the job only to the first node. If you have 4 nodes and two targets defined, then it'll issue one target to the first node and one target to the second node.
  • Not running a command with the load balanced flag on means that your command and all targets will be issued to every node equally. For example, if you have 4 nodes, and one target. Then each node will run the same command against that one target.
  • Run as a single command will run identical to what is described previously except that it will dump all targets into a temporary file, then put the file into the command where you defined {{target}}
  • Templates have been added to the application by default but you can run whatever commands you'd like in the command input. You can also create your own templates for future use.

Event Stream Page

Alt text

  • Displays the last 100 events to have taken place on the server

Settings Page

Alt text

  • Lets users update their password as well as add and remove command templates
  • This is also where you will be able to retrieve the command to run your agent

ensemble's People

Contributors

bostonpaola avatar dotnetrussell avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar

ensemble's Issues

Separate the Webclient and Director Processes

The webclient process is currently started by the director process. Instead we should have a watchdog process that controls starting both webclient and director as well as restarting them if they crash.

Looking for some data / stats on the geo distributed results

Hello Russell,

Just saw your defcon talk, really nice tool you're building here.

I guess you did it at first as a POC but would you mind sharing some compared results between regions?

It could be simple metrics on the same list of domains, like :

  • with httpx, how many alived domains per regions?
  • where the domain is alived, is there a content difference?
  • with nmap, how many open ports per regions per domain, etc

Regards!

Create Watchdog Process for Agents

Create a watchdog process for agents that communicates with the director.

It will

  • Notify director when agent goes down and up
  • Restart agent when agent dies automatically
  • Restart agent at when director issues command

Feature: Push Notifications

Feature Summary: Users should be able to subscribe to push notifications for director notifications. Users should be able to choose what type of notifications they wish to receive and these notifications should either pop up as a toast in windows or as a push notification on the phone.

Acceptance Criteria:
Given a user wants to setup push notifications
When they navigate to the settings page
Then they will have the ability to activate push notifications and select what notifications they want to receive such as job status, and agent status

Given a user has enabled push notifications
When a notification is received by the browser
Then a push notification will be sent to the user

Agent Restore State

Currently if an agent dies mid job, the agent will not resume the job when the agent restarts. This is going to take some investigation to figure out a way to do this.

  • Agents start a subprocess, wait for the process to finish, and then dump standard out to a temp file which is then processed and returned when requested. The issue with state is that if the agent dies mid job, the job still continues but now no longer belongs to the agent. So if the agent restarts there's no way for the agent to retrieve the output.

One possible solution to this could be running all commands with "tee" and then dumping the results to a results file named with the job id as it runs. Then when the agent restarts it could check some flat file that contains running job ids and retrieve the job results

Another possible solution (untested) is to track the pid of the process and then continuing to read from STDOUT by tailing /proc/pid/fd/1 but I believe this won't continue to capture the output between the restart and the tail. This is probably a backup solution

Feature: Easy way to install new tools on agents

Feature Summary: Users should have a nice interface for installing new tools on agents, seeing what tools are installed, and uninstalling tools from agents. Currently users can install tools by creating a job but this is not the correct use of that interface.

Acceptance Criteria:
Given a user wants to install a new tool on the cluster or modify an existing one
When the user navigates to the settings page
Then the user will be presented with an interface that allows the user to install or delete existing apps in the cluster

Use a WSGI enabled server instead of flask

Do not use flask as the primary server. It's not appropriate for production. Instead use a web server able to communicate with Flask through a WSGI protocol such as waitress. Waitress doesn't currently support TLS natively so another approach will need to be created for that also.

Feature: Better Scheduled Job Result Diffing

Feature Summary: The job diffing that's currently enabled doesn't fully work as intended. The output should be sorted in some repeatable way prior to diffing and the diff output should have colors to help identify changes more easily.

Acceptance Criteria:
Given a user has run a scheduled job more than once
When the user views the scheduled job results
Then the user will see a diff of the results that are color coded (green for new red for removed) and sorted

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.