Git Product home page Git Product logo

pipal's Introduction

Pipal, Password Analyser

Copyright(c) 2022, Robin Wood [email protected]

On most internal pen-tests I do, I generally manage to get a password dump from the DC. To do some basic analysis on this I wrote Counter and since I originally released it I've made quite a few mods to it to generate extra stats that are useful when doing reports to management.

Recently a good friend, n00bz, asked on Twitter if anyone had a tool that he could use to analyse some passwords he had. I pointed him to Counter and said if he had any suggestions for additions to let me know. He did just that and over the last month between us we have come up with a load of new features which we both think will help anyone with a large dump of cracked passwords to analyse. We also got some input from well known password analysts Matt Weir and Martin Bos who I'd like to give a big thanks to.

I have to point out before going on, all this tool does is to give you the stats and the information to help you analyse the passwords. The real work is done by you in interpreting the results, I give you the numbers, you tell the story.

Seeing as there have been so many changes to the underlying code I also decided to change the name (see below) and do a full new release.

So, what does this new version do? The best way to describe it is to see some examples so go to the Pipal project page for a full walk through of a sample analysis.

Install / Usage

The app will only work with Ruby 1.9.x and newer, if you try to run it in any previous versions you will get a warning and the app will close.

Pipal is completely self contained and requires no gems installing so should work on any vanilla Ruby install.

Usage is fairly simple, -? will give you full instructions:

$ ./pipal.rb -?
pipal 2.0 Robin Wood (robin@digi.ninja) (http://digi.ninja)

Usage: pipal [OPTION] ... FILENAME
        --help, -h: show help
        --top, -t X: show the top X results (default 10)
        --output, -o : output to file
        --external, -e : external file to compare words against
		--gkey <Google Maps API key>: to allow zip code lookups (optional)

        FILENAME: The file to count

When you run the app you'll get a nice progress bar which gives you a rough idea of how long the app will take to run. If you want to stop it at any point hitting Ctrl-c will stop the parsing and will dump out the stats generated so far.

The progress bar is based on a line count from the file which it gets this using the wc command. If it can't find wc it will make a guess at the number of lines based on the file size and an average line length of 8 bytes so the progress bar may not be fully accurate but should still give you an idea.

The Google Maps API key is supposed to be used by Google to only allow access to their API to registered users. I assumed this was true and registered for a key but in putting together this release I found that it will take any value and still do the look up. This may be a bug at the Google end or deliberate and may change any any time so I'd suggest grabbing a key just in case. To use it you can either edit the script and put the key into the constant on line 35 or you can pass it on the command line every time. If you are going to hope that you don't need a valid key then just put X in as the value as without something Pipal won't try to perform a look up.

Enabling Checkers

Checkers are the scripts that do the actual work, to understand how these work, see the README_modular.md file.

Version History

Version 2 - Two big changes, the first a massive speed increase. This patch was submitted by Stefan Venken who said a small mention would be good enough, I want to give him a big mention. Running through the LinkedIn lists would have taken many many hours on version 1, version 2 went through 3.5 million records in about 15 minutes. Thank you.

Second change is the addition of US area and zip code lookups. This little feature gives some interesting geographical data when ran across password lists originating in the US. The best example I've seen of this is the dump from the Military Singles site where some passwords could be obviously seen to be grouped around US military bases. People in the UK don't have the same relationship with phone numbers so I know this won't work here but if anyone can suggest any other areas where this might be useful then I'll look at building in some kind of location awareness feature so you can specify the source of the list and get results customized to the correct area or just run every area and see if a pattern emerges.

A non-code-base change is for version 2 is the move from hosting the code myself to github. This is my first github hosted project so I may get things wrong, if I do, sorry. A number of people asked how they could submit patches so this seems like the best way to do it, lets hope it works out.

Version 1 - Was a proof of concept, written fairly in a fairly verbose way so not very optimised. Took off way more than I expected it would and gathered a lot of community support.

Feedback/Todo

If you have a read through the source for Pipal you'll notice that it isn't very efficient at the moment. The way I built it was to try to keep each chunk of stats together as a distinct group so that if I wanted to add a new, similar, group then it was easy to just copy and paste the group. Now I've got a working app and I know roughly what I need in the different group types I've got an idea on how to rewrite the main parser to make it much more efficient and hopefully multi-threaded which should speed up the processing by a lot for large lists.

I could have made these changes before releasing version 1.0 but I figured before I do I want to get as much feedback as possible from users about the features already implemented and about any new features they would like to see so that I can bundle all these together into version 2. So, please get in touch if there is a set of stats that you'd like to see included.

One other thing I know needs fixing, Pipal doesn't handle certain character encodings very well. If anyone knows how to correctly deal with different encoding types, especially with regards to regular expressions, please let me know.

Licence

This project released under the Creative Commons Attribution-Share Alike 2.0 UK: England & Wales

pipal's People

Contributors

0xc0da avatar clem9669 avatar console avatar digininja avatar dirkjanm avatar dnucna avatar enderax avatar maggick avatar michaeldim02 avatar odd-meta avatar webbreacher avatar whoot avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

pipal's Issues

Feature request - create base word list & de-1337

It would be nice to have a feature for creating wordlists based on the base words. A 'de-1337'-function could help creating these wordlists. The password '1945un1c0rn01' could result in the base word 'unicorn' by stripping the begin/end non-alpha's and replacing the inner 1 and 0 with the alpha character (i/o).

[OS X] Not Working Properly

Running ruby 1.9.3 :

09:35:27 connection@GLaDOS:[~/pipal]$ ruby -v
ruby 1.9.3p448 (2013-06-27 revision 41675) [x86_64-darwin12.5.0]

And pipal 3.0 :

09:36:49 connection@GLaDOS:[~/pipal]$ ruby pipal.rb
pipal 3.0 alpha Robin Wood ([email protected]) (www.digininja.org)

I get the following error :

No Checkers enabled, please read README_modular for more information

Even when running --list-checkers I am not able to run it and options such as -h and -? do not even show usage instructrions

"Dates" aren't sorted by frequency [bug?]

I'm not sure if this was intentional, a bug, or just an artifact of the behavior of this checker. However, "date_checker.rb" does not sort the resulting lists by frequency like every other checker. It appears that everything else uses "list_checker.rb" which sorts the output, date_checker does not.

For example, vehicles are sorted in descending order:

Vehicle
car = 101125 (0.31%)
ford = 9976 (0.03%)
bmw = 4829 (0.01%)
audi = 4653 (0.01%)

Here's a sample from Dates/Months, where March should come first (but doesn't), followed by January and February.

Dates

Months
january = 3259 (0.01%)
february = 1273 (0.0%)
march = 8481 (0.03%)

Masks

Pipal is my go-to password analyzer. Love it. The one additional thing I wish it had is the hashcat mask generator that PACK has. I always run both analyzers, but PACK just to get the masks. If pipal had that, that would be pretty awesome. I thought of just merging the two, but PACK is python and pipal is ruby. I've never tried ruby myself, otherwise I'd jump in and try to create a version myself. Sorry...

Here is the source code for PACK: https://thesprawl.org/projects/pack/

Error with module

I try to use some module and I get an error :

pipal-modular/checkers_available/FR_area_codes.rb:145:in get_results': undefined methodlength' for nil:NilClass (NoMethodError)
from ./pipal.rb:717:in block (2 levels) in <main>' from ./pipal.rb:716:ineach'
from ./pipal.rb:716:in block in <main>' from /usr/lib/ruby/1.9.1/benchmark.rb:280:inmeasure'
from ./pipal.rb:75:in `

'

I do not have this error with the FR_color_checker module !

Nop Such File or Directory

Hi

This is the first time I have used Ruby or Pipal and I'm getting the following error:

C:/Security/Ruby/Pipal/pipal.rb:261:in ``': No such file or directory - wc -
l 'passwords.txt' (Errno::ENOENT)
from C:/Security/Ruby/Pipal/pipal.rb:261:in`

'

This is probably user error but any help would be much appreciated. I'm running the following command and both pipal and the passwords.txt file are in same directory:

pipal.rb passwords.txt -o passout.txt

Thanks

list_checker.rb colour refs

It looks like you pulled the colour checking class out from list_checker.rb but there are still variable references/names in that file referencing the colour variables.

For instance, the following excerpt shows the "colour" variables. Should they be more generically named now?

def process_word (word, extras = nil)
        @list.each_pair do |colour, count|
            if /#{colour}/i.match word
                @list[colour] += 1

Base words (feature request)

Hi @digininja First of all thank you for all the hard work you put in this tool. Pipal does a great job analyzing passwords. What I noticed is that the top 10 base words are not really the "base" words. For example, it sometimes happens that "p@ssw0rd" and "p@ssword' are (yes also separately) mentioned as a base word, while you would think in the basis it should be "password" (the others are variations).

Perhaps an idea to merge the functionality of deleet with pipal (https://github.com/digininja/deleet)? It could really improve the value of the output.

syntax error ?!!!

i had tried ruby1.9.3 and higher versoin and got same error
[root@localhost pipal-master]# ./pipal.ry -? -bash: ./pipal.ry: No such file or directory [root@localhost pipal-master]# ./pipal.rb -? <internal:/usr/local/lib/ruby/3.0.0/rubygems/core_ext/kernel_require.rb>:85:in require': /opt/pipal-master/checkers_enabled/01basic
.rb:1: unknown regexp options - bac (SyntaxError)
../checkers_available/basic.rb
^~~~~~
from internal:/usr/local/lib/ruby/3.0.0/rubygems/core_ext/kernel_require.rb:85:in require' from ./pipal.rb:184:in block in

'
from ./pipal.rb:179:in each' from ./pipal.rb:179:in '
[root@localhost pipal-master]#
`
how can i fix this?

See found strings inline

I'd like to see a flag/option that shows the found strings to be able to quickly highlight ones of interest. I picture a --highlight or --show flag to be passed to the main script. It'll turn output like:

Seasons
winter = 15 (0.83%)
summer = 9 (0.5%)
fall = 3 (0.17%)

to

Seasons
winter = 15 (0.83%) - winteriscoming123, winterwinter!
summer = 9 (0.5%) - ILovesummer, summertimeR0CK$
fall = 3 (0.17%) - fallinginluv, snowfallsnow, fallismyFAVoriteseason

This is important for things like the new "violent" and "explicit" checkers that look for possibly disgruntled people. It'd make it easy to see the difference from a password of 'badWeatherSucks' and 'myf-ingjobsucks' at a quick glance without having to search the password list. Kind of a method of easily false positive viewing.

--external option gives error

Using pipal with the '--external' option gives the following errror:

undefined local variable or method `external_list' for main:Object

NameError

Backtrace:
./pipal.rb:236:in `block (2 levels) in <main>'
./pipal.rb:235:in `each_line'
./pipal.rb:235:in `block in <main>'
/usr/lib/ruby/1.9.1/getoptlong.rb:604:in `block in each'
/usr/lib/ruby/1.9.1/getoptlong.rb:601:in `loop'
/usr/lib/ruby/1.9.1/getoptlong.rb:601:in `each'
./pipal.rb:211:in `<main>'

I am using the github current version of pipal. I tried both ruby1.9.1 and ruby1.9.3.

Change branding/wording

This tool is really a string analyzer. Most of the time, people pass passwords to it to analyze. But you can pass any word list to it. I suggest changing all the output and help text to move from "password" to "string" to help people understand it can be used in wider cases.

unknown regexp options - bac (SyntaxError)

So, i tried installing ruby on a new OS, Debian 11, Latest linux and on Windows

None of them did work and all throwing the same error

root@20012-42031:~/test# ./pipal.rb --help Traceback (most recent call last): 4: from ./pipal.rb:179:in

'
3: from ./pipal.rb:179:in each' 2: from ./pipal.rb:184:in block in '
1: from /usr/lib/ruby/vendor_ruby/rubygems/core_ext/kernel_require.rb:85:in require' /usr/lib/ruby/vendor_ruby/rubygems/core_ext/kernel_require.rb:85:in require': /root/test/checkers_enabled/01basic.rb:1: unknown regexp options - bac (SyntaxError)
../checkers_available/basic.rb`

ruby versions used:
windows (3.0)
debian (2.7)
linux (2.5)

Mangling

Pipal is awesome at searching for specific strings in passwords. Does the password have the word "pink"? It does? Ok, increment the count of pink +1.

What I find is that many of my users use mangled forms of the words in the pipal checkers. So instead of "pink" there may be "p!nk", "p1nk", "pInk", etc. From what I'm seeing pipal doesn't have mangle rules that can be applied to each string it is looking for. Since this would be something that could be used across all checkers and not just within a certain one, wanted to see what @digininja thought about adding something to the core pipal.rb script to allow for --mangle rules to be applied to strings searched for.

Passing email ids [question]

I have a dump of cleartext email id and password pairs. I have used the basic checkers like basic.rb, date_checker.rb, etc. by extracting all passwords in a txt file and passing that. But I can't figure out how am I supposed to pass the email ids for using the email_names.rb file. I have tried passing the file as colon-separated lines of pairs but that doesn't seem to work...

The --external param (mentioned in the readme) is also removed.

Thanks!

unknown regexp option s - bac (SyntaxError)

C:/Tools/Coding/Ruby193/lib/ruby/site_ruby/1.9.1/rubygems/custom_require.rb:36:i
n `require': C:/Tools/pipal/checkers_enabled/01basic.rb:1: unknown regexp option
s - bac (SyntaxError)
        from C:/Tools/Coding/Ruby193/lib/ruby/site_ruby/1.9.1/rubygems/custom_re
quire.rb:36:in `require'
        from pipal.rb:184:in `block in <main>'
        from pipal.rb:179:in `each'
        from pipal.rb:179:in `<main>'

incompatible encoding regexp match error

While running pipal against the passwords included in the newseasims.com dump, I ran into an error:

$ ./pipal.rb -o 2013.04.20-newseasims_com-pipal 2013.04.20-newseasims_com-pwd 
Generating stats, hit CTRL-C to finish early and dump stats on words already processed.
Please wait...
Something went wrong, please report it to [email protected] along with these messages:

incompatible encoding regexp match (UTF-8 regexp with ASCII-8BIT string)

Encoding::CompatibilityError

Backtrace:
./pipal.rb:401:in `=~'
./pipal.rb:401:in `block (3 levels) in <main>'
./pipal.rb:331:in `each_line'
./pipal.rb:331:in `block (2 levels) in <main>'
./pipal.rb:329:in `catch'
./pipal.rb:329:in `block in <main>'
/Users/adamcaudill/.rvm/rubies/ruby-2.0.0-p0/lib/ruby/2.0.0/benchmark.rb:281:in `measure'
./pipal.rb:45:in `<main>'

I was able to correct the error by changing line pipal.rb:340 from

line.force_encoding("ASCII-8BIT")

to:

line.force_encoding("UTF-8")

Running on OSX:

$ ruby -v
ruby 2.0.0p0 (2013-02-24 revision 39474) [x86_64-darwin12.2.1]

Not sure if it's because I'm running Ruby 2, or something about my setup, I know the ASCII vs UTF8 issue with Ruby can be a nightmare to get right.

Output as markdown

@Lexus89 As a user of the tool, how would you feel if I update the output to produce markdown rather than basic text? I'm thinking about it as it would then let me do markdown to HTML for some nicer reports.

Make it as a gem

It would be great if you can make it as a gem so pipal command can be used in anywhere in the command line. Also, it makes it easier to maintain your code from the user perspective.

Pipal lag and failure to complete analysis (RAM limitations?)

Hey @digininja , I would like to feature pipal on Null Byte, but I'm experiencing an issue.

The wordlist being analyzed is 92,400,000 lines, 1.1Gb. Pipal seems to lag after ~2 hours of processing. At that point my CPU usage drops down to normal levels and the ETA starts to slowly increase until it stops. This was done in a Kali VM with 4 cores and 6Gb RAM. Tried again on a dedicated Ubuntu machine with an i7 and 16Gb RAM -- same issue.

Does pipal have limitations? What might be causing the lag and how can I debug?

EDIT:

I just realized Ruby v1.9 is recommended in the README.

> apt-cache policy ruby
ruby:
  Installed: 1:2.5.0
  Candidate: 1:2.5.0
  Version table:
 *** 1:2.5.0 500
	500 http://http.kali.org/kali kali-rolling/main amd64 Packages
	100 /var/lib/dpkg/status

But pipal works fine when analyzing smaller wordlists using v1.2. Should I try manually installing v1.9.x in Kali? There's an APT candidate for v2.3 and v2.5, will those suffice?

P.S. I'm not a ruby coder

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.