Currently, whenever the execution of an attack leads to a failure (e.g. exit status not 0), gauntlt raises an ExecutionFailed error. This can include undefined step definitions as well as mismatches in output. These should be differentiated. Not sure if cucumber returns different exit statuses that would allow for this to be easily done.

The main motivation for this is to isolate the case when a security assertion is not satisfied from typos and other errors. We want to clearly distinguish between a situation where there is a problem with the attack file and a situation where a vulnerability is detected.

Just throwing this on the queue for tracking. Feel free to assign to me.

the Dirb attack adapter has some env variables it wants set, so document how to do that

I am new to gauntlt and so far it has been great. I started with the base nmap file and am using it to test open ports on our web and database servers. I made a new file for each host that I wanted to attack. It would be nice to be able to include a list of host so I could run the attack on multiple servers from one attack file.

There are a lot of step definitions in gauntlt, which fall into 2 categories:

1. Those defined by gauntlt
2. Those defined by aruba

This would look like:

$ gauntlt --steps

This feature will help users understand what they can do with gauntlt. It will also help us to pare down the number of steps to the number required and add an abstraction layer between gauntlt and aruba so that we can more easily drop our dependency on cucumber (either by using a replacement for aruba, or forking aruba to create a version of it that does not require cucumber).

We are not doing anything meaningful with the -n option.

We should switch from:

$ gauntlt attack -n nmap -a nmap.attack

to:

$ gauntlt attack nmap.attack

This will also allow easier inter-mingling of attack adapters within the same attack. Need to deal with undefined steps more rigorously though to prevent users from getting themselves into trouble.

I noticed that I needed to provide the hostname in each attack script. This of course makes sense until you want to target various hosts. As a traditional software vendor, I would target various hosts since different build servers would build different versions of the same product for Linux/Windows. Thus, the same attack script would need to support different builds. This is challenging because the current attack scripts seem to require me to hard code the hostname.

Use Case

Avoid hard-coding values within attack scripts through parameterization

Change Suggestion

In researching whether this functionality exists (does not appear to), I believe I have a change suggestion that can accomplish this:

• Use Cucumber profiles
  ** Cucumber already supports profiles. The caller for gauntlt just needs to:
  ** Create a folder called config in whatever directory it is in
  ** Create a file called cucumber.yaml in config/
  ** "cucumber.yaml" contents:
  This is up to the user but I have:

  [root@steph-lnx config]# cat cucumber.yml
  default: TARGET_HOSTNAME=target-machine01.internal.local SSLYZE_PATH=/root/sslyze-0.5_src/sslyze.py

Modify add_to_profile function (gauntlt-0.1.2/lib/gauntlt/attack_adapters/support/profile_helper.rb):

      def add_to_profile(k,v)
        puts "Overwriting profile value for #{k}" if gauntlt_profile.has_key?(k)    
        if v =~ /^ENV\['.+'\]$/
          puts eval(v)
          gauntlt_profile[k] = eval(v)
        else
          gauntlt_profile[k] = v
        end
      end

By doing the above, I would be able create an attack file with the following profile to leverage values from cucumber profile if I need to. Otherwise, if I still want to hard code the hostname, the above code change suggestion would still work.

  And the following profile:
    | name     | value      |
    | hostname | ENV['TARGET_HOSTNAME'] |

This also makes transparent the SSLYZE path since it wouldn't need to be defined in the .bash_profile and the sslyze.rb file would automatically be able to retrieve this.

Using version 0.1.0 I get this error message when running with a clean directory with no attack files. This should instead give a message "No Attack files found. Download some examples and get started."

$ mkdir test-dir
$ cd test-dir
$ gauntlt
~/.rvm/gems/ruby-1.9.3-head/gems/gauntlt-0.1.0/lib/gauntlt/attack.rb:16:in initialize': No files found in path: ./**/*.attack (Gauntlt::Attack::NoFilesFound) from /Users/jwickett/.rvm/gems/ruby-1.9.3-head/gems/gauntlt-0.1.0/lib/gauntlt.rb:32:innew'
from /Users/jwickett/.rvm/gems/ruby-1.9.3-head/gems/gauntlt-0.1.0/lib/gauntlt.rb:32:in attack' from /Users/jwickett/.rvm/gems/ruby-1.9.3-head/gems/gauntlt-0.1.0/bin/gauntlt:37:in<top (required)>'
from /Users/jwickett/.rvm/gems/ruby-1.9.3-head/bin/gauntlt:19:in load' from /Users/jwickett/.rvm/gems/ruby-1.9.3-head/bin/gauntlt:19:in

Add nikto 2.1.4

Jonas Nicklas, the man behind Capybara and Carrierwave, has created a new gem called turnip which is an implementation of cucumber directly on top of rspec. It also includes some nice additions to cucumber which allow for easier definition and re-use of step definitions.

It's worth checking out turnip to see if it would work well as the cucumber engine for gauntlt.

Its open source, cross platform, and supports JSON & XML via a REST API.
Under very active development, and I'm very happy to enhance it to make it easier to integrate with gauntlt or any other security tool.
It scores 100% XSS detection on wavsep.

• Simon

Add:

• OS detection
• SYN scan
• Checks to make sure certain ports are not available
• Output matching checks

The usage would follow cucumber:

$ gauntlt --tags @important

# or exclude tags
$ gauntlt --tags ~@optional

using --format junit doesnt work because underneath gauntlt, cucumber wants a --dir passed to it.

Currently, an attack is specified in a defensive way:

Given I try to breach the system's security
Then I should fail

This has 2 big problems:

1. It encourages the writing of useless attacks that don't verify anything (e.g. "port 443 should be open") since you do not have to specify what a successful attack looks like or means
2. It makes each attack more complex since you have to specify the failure criteria

A better alternative is to specify the attack from the attacker's point of view:

Given I try to breach the system's security
Then I should succeed

This approach means that a successful attack will trigger an exit status of 1 since a breach in security was successful. It will be up to the creators of attacks to come up with meaningful attack scenarios that specify how a breach occurs. This will allow for better communication about the tactics of an attack as well as the potential losses from a successful attack.

Cucumber has built-in support for reversing the exit statuses so technically this is not hard to do. The biggest task here will be to improve our attack examples. We will have to rewrite them from the attacker's point of view.

Let me know what you all think and if any clarification is needed.

Maybe I'm just really missing something, but even running the example seems to only work with google.

Am I missing something?

➜  gauntlt git:(master) ✗ bundle exec gauntlt verify --test nmap --host yahoo.com
Feature: Run nmap against a target and pass the value of the hostname from the profile.xml.

  Background:               # /tmp/gauntlt/lib/gauntlt/cucumber/nmap.feature:3
    Given nmap is installed # lib/gauntlt/cucumber/step_definitions/nmap_steps.rb:1

  Scenario: Verify server is available on standard web ports # /tmp/gauntlt/lib/gauntlt/cucumber/nmap.feature:6
    Given the target hostname is "google.com"                # lib/gauntlt/cucumber/step_definitions/profile_steps.rb:1
    When I run nmap against the following ports:             # lib/gauntlt/cucumber/step_definitions/nmap_steps.rb:11
      | port_number |
      | 80          |
      | 443         |
    Then the output should contain:                          # vendor/ruby/1.9.1/gems/aruba-0.4.11/lib/aruba/cucumber.rb:98
      """
      80/tcp  open  http
      443/tcp open  https
      """

1 scenario (1 passed)
4 steps (4 passed)
0m0.742s
false
➜  gauntlt git:(master) ✗ 

Awesome foundations on the tool though. Thanks!

Allow gauntlt to distinguish against between crawl, walk, run tests. Using 5 seconds, 30 seconds, unlimited timeout values.

Logging, Installation, How-Tos

Output from many tools like nmap, nikto and sslyze is not valuable on a single run. The change in the output can indicate issues and rugged professionals are especially interested in knowing when the signature of a particular scan changes.

To that end, gauntlt needs a feature where full or partial output can be tracked. Changes in output from one run to another should trigger an alert. Raw and formatted output should be handled as well (e.g. XML).

Here is a rough example of what the XML formatted diff might look like:

And the following XML should not change from the previous run:
 | css                                       |
 | ports port[protocol="tcp"][portid="80"]   |

There are at least 2 significant challenges to overcome:

1. How and where to store the output from previous runs
2. How to deal with changes in the attack file that may result in changed output

This feature is not a simple one to implement, but it promises to deliver great value, so it deserves priority. Before we rush to write code, we should refine how we would like it to work and we could implement it.

Implement sqlmap for gauntlt so that it can be called via gauntlt steps.

After you clone the repo and run the specs and cukes, they fail because of sqlmap, sslyze and nmap are not installed. I found sslyze and sqlmap in vendor directory but it asks to set environment variables for each tool.

What if we set up paths in rspec and cucumber config to load sslyze and sqlmap from vendor folder for developer convenience?

or

Is there a better alternative to install these tools?

Tool for evaluating SSL configurations: http://code.google.com/p/sslyze/wiki/QuickStart

This would allow for an arbitrary command to be executed and its output read. We can use this as a first step when adding new attack adapters. Also, this will provide an easy way for someone to use custom attack adapters. They can use an existing command line tool or write their own wrapper script and execute it within gauntlt.

In the future, we can use this generic command line adapter as the base for the other existing command line adapters.

@jeremiahshirk had the brilliant idea to compile gauntlt into a jar file using JRuyb that could be downloaded without having to install anything. We could also create a mac binary using MacRuby (since many of our potential users will be on OS X).

This makes a lot of sense and would speed up the adoption of gauntlt.

Lets add rails goat into gauntlt as a target for us to run all the tests against. Could be a real easy win for both projects.

http://railsgoat.cktricky.com

https://github.com/OWASP/railsgoat

e.g.

$ gauntlt attack -n nmap -a nmap.attack nmap2.attack

or

$ gauntlt attack -n nmap -a *.attack

This should be pretty easy, as the attack files and/or glob can be passed through to Cucumber::Cli::Main.execute here:

• https://github.com/thegauntlet/gauntlt/blob/master/lib/gauntlt/attack.rb#L29

Just running a simple sqlmap attack causes gauntlt to lock up. Not sure if something changed with sqlmap or what...

@jeremiahshirk can you help test this out? I moved features/attacks/sqlmap.attack to .broken for now so that it is skipped in the CI.

Using the command gauntlt bootstrap webgoat (or something similar) a webgoat target should be setup for testing. This will help people know that they tests they are building work and it will also help our module builder community to test the new tools out that they want to put into gauntlt

http://arachni-scanner.com/

Right now CI takes a long time to run. Some of that is due to the really long install of railsgoat. We could cache the dependent gems and a sundry, however it might be better to just use a small python web app. Leaning toward using Gruyere.

Showing progress and verifying that certain tests were run or not run

Some gauntlt tests might want to have the .audit extension instead of the .attack extension. It seems that someone would want to write network.attack and network.audit or website-main.attack and website-main.audit. It is just a naming scheme but it makes sense to offer that as an option.

Currently, the attack adapters are all available all the time. Also, their step definitions and support methods are in the global namespace. This makes clashes a real danger and also makes it very difficult to do any kind of inheritance or code-sharing between attack adapters in a clean way.

I tried (and failed) to address this a few months back, but now that our attack adapter structure has become more stable, I think it's worth another try. Basically, the idea is to hook into cucumber's StepMother to add and remove step definitions.

Then the user would have to activate attack adapters explicitly with tags, like this:

@activate-nmap
Feature: nmap attacks
  Scenario:
    Given I launch an nmap attack with ...

I think in real-world usage, we want to make it easy to use various attack adapters together without dumping all of them into the global namespace. Advanced users could easily create a cucumber hook to activate all adapters, but that should be discouraged.

When using cucumber 1.3.1, this is the output from running cucumber on gauntlt. Our "Bad or undefined attack! is not getting raised.

Scenario: Run attack with undefined steps # features/attack.feature:71
Given an attack "nmap" exists # features/step_definitions/config_steps.rb:1
And a file named "nmap.attack" with: # aruba-0.5.1/lib/aruba/cucumber.rb:19
"""
Feature: my non-existent attack
Scenario: Fail on undefined step definition
Given this_attack_would_never_exist
"""
When I run gauntlt # aruba-0.5.1/lib/aruba/cucumber.rb:60
Then it should fail with: # aruba-0.5.1/lib/aruba/cucumber.rb:162
"""
Bad or undefined attack!
"""
expected "Feature: my non-existent attack\n\n Scenario: Fail on undefined step definition # ./nmap.attack:2\n Given this_attack_would_never_exist # ./nmap.attack:3\n Undefined step: "this_attack_would_never_exist" (Cucumber::Undefined)\n ./nmap.attack:3:in Given this_attack_would_never_exist'\n\n1 scenario (1 undefined)\n1 step (1 undefined)\n0m0.003s\n\nYou can implement step definitions for undefined steps with these snippets:\n\nGiven(/^this_attack_would_never_exist$/) do\n pending # express the regexp above with the code you wish you had\nend\n\n" to include "Bad or undefined attack!" Diff: @@ -1,2 +1,17 @@ -Bad or undefined attack! +Feature: my non-existent attack + + Scenario: Fail on undefined step definition # ./nmap.attack:2 + Given this_attack_would_never_exist # ./nmap.attack:3 + Undefined step: "this_attack_would_never_exist" (Cucumber::Undefined) + ./nmap.attack:3:inGiven this_attack_would_never_exist'
+
+1 scenario (1 undefined)
+1 step (1 undefined)
+0m0.003s
+
+You can implement step definitions for undefined steps with these snippets:
+
+Given(/^this_attack_would_never_exist$/) do
+ pending # express the regexp above with the code you wish you had
+end
(RSpec::Expectations::ExpectationNotMetError)
features/attack.feature:80:in `Then it should fail with:'

It seems that any useful penetration test will use multiple tools to put together a successful attack. For example, nmap can be used to find open database ports and then sqlmap can be targeted at those ports, as in this tutorial:

• http://hackonadime.blogspot.com/2011/07/sqlmap-introduction-sql-injection.html

Starting from my completely ignorant perspective, I just managed to completely fail to figure out how to install and use gauntlt. Installation instructions need some work; making it easier to know if I've got all my requirements in order would help.

There is no installation of sqlmap and sslyze. To run these tools, one downloads the code from github and then executes the main python script, (e.g. $ python sslyze.py). For gauntlt, I decided to make things a lot simpler for us and a little more complicated for users by requiring the user to have a wrapper binary on their path so that the tool can be executed like a normal command-line binary (e.g. $ sslyze). This is the error message a user gets if sslyze is not installed:

sslyze.py not installed or sslyze alias not set!

1. Download sslyze from: https://github.com/iSECPartners/sslyze
2. Create a file named 'sslyze' with the following content:

   #! /usr/bin/env bash
   python /path/to/sslyze.py

3. Make sure the file is in your path:

   $ which sslyze

We need to figure out a way to pull the sslyze.py repo into Travis and create the wrapper executable. Might need a rake task and/or custom script to handle this. This mechanism will be useful for testing locally as well, and at some point, we may offer the end user the ability to install sslyze via a gauntlt tool (maybe $ gauntlt install sslyze ?).

Add the License for gauntlt to github repo

It is not that hard to add new attacks to gauntlt now. A .feature file and a corresponding step definition file need to be added to lib/cucumber. For example, the nmap tests are defined in:

• lib/gauntlt/cucumber/nmap.feature
• lib/gauntlt/cucumber/step_definitions/nmap_steps.rb

This needs to be made more systematic to allow for new built-in tests and also for user-created tests. Profile settings and dependency checks need to be take into account in a standard way.

The goal is to make it very easy to add new tests to gauntlt and to make every test completely modular, interfacing with the main gauntlt code only through public API methods.

After looking into Turnip as an alternative engine for gauntlt (#10), eventually we came to the conclusion that building our own custom cucumber implementation on top of Gherkin and RSpec would give us maximal control over features and behaviour. Also, this would allow us to think about what we want gauntlt to be, rather than focus on what it is easy to do given cucumber's existing features and code structure.

Once we consider gauntlt ready we need to push it rubygems

This problem occurs on mac and ubuntu. There seems to be a problem with the way gems are being loaded, or with the gauntlt gemspec.

To reproduce the bug take the following steps:

1. install gauntlt as a gem:

   $ gem install gauntlt
   

2. Create a temp folder and cd into it.

3. Try to run gauntlt:

   $ gauntlt
   
   fatal: Not a git repository (or any of the parent directories): .git
   fatal: Not a git repository (or any of the parent directories): .git
   fatal: Not a git repository (or any of the parent directories): .git
   
         -h, --help
             Help outputs available command options
         -v, --version
             Version of app
   
     Command 'attack':
   
         -h, --help
             Help outputs available command options
         -l, --list
             List of available attacks
         -n, --name
             Name of attack to launch
         -a, --attack-file
             Name of file with attack definition
   

4. Now initialize the folder as a git repo

   $ git init .
   

5. Try running gauntlt again and errors will be gone:

   $ gauntlt                                                                     
   
         -h, --help
             Help outputs available command options
         -v, --version
             Version of app
   
     Command 'attack':
   
         -h, --help
             Help outputs available command options
         -l, --list
             List of available attacks
         -n, --name
             Name of attack to launch
         -a, --attack-file
             Name of file with attack definition
   

The feature files are confusing and big because they contain the sample attacks. Also, the sample attacks in the examples directory can get out of sync (and thus not work) with those in the cukes. We should use aruba to copy the attacks in the examples directory for use in cukes.

Write a test for gauntlt that would quickly scan a single page for XSS.