Find web directories without bruteforce
License: MIT License
[200] http://www.xxxxxxx.ru/resources/Image/ (Index Of)
Interesting extension files: tj.php, tj.php, tj.php, tj.php, tj.php, tj.php, tj.php, tj.php, tj.php, tj.php, tj.php, tj.php
Real Index Of:
tj.php(10).gif 2017-12-08 20:57 1.2K
[IMG] tj.php(11).gif 2017-12-09 02:28 1.2K
[IMG] tj.php.gif 2017-10-02 16:26 1.2K
[IMG] tj.php;(1).jpg 2017-10-02 23:28 1.2K
[IMG] tj.php;(2).jpg 2017-10-03 05:54 1.2K
[IMG] tj.php;(3).jpg 2017-10-20 08:23 1.2K
[IMG] tj.php;(4).jpg 2017-10-20 16:53 1.2K
[IMG] tj.php;(5).jpg 2017-10-20 21:55 1.2K
[IMG] tj.php;(6).jpg 2017-11-09 15:51 1.2K
[IMG] tj.php;(7).jpg 2017-11-10 04:02 1.2K
[IMG] tj.php;(8).jpg 2017-11-10 14:53 1.2K
[IMG] tj.php;(9).jpg 2017-11-11 12:09 1.2K
[IMG] tj.php;(10).jpg 2017-12-08 20:57 1.2K
[IMG] tj.php;(11).jpg 2017-12-09 02:28 1.2K
Reported by jmesa
For future releases and improve results, would be necessary to extract dates from files. It could be useful for detecting those "new files" in the server comparing the dates present in the folder results.
For example:
| Name | Last modified | Size | Description | |
|---|---|---|---|---|
| Parent Directory | - | |||
| wpml-config.xml | 2017-01-15 00:04 | 213 | ||
| wpcom-test-backup/ | 2017-01-15 00:04 | - | ||
| webpack.config.js | 2017-01-15 00:04 | 3.1K | ||
| views/ | 2017-01-15 00:04 | - | ||
| uninstall.php | 2017-01-15 00:04 | 830 | ||
| sync/ | 2017-01-15 00:04 | - | ||
| scss/ | 2017-01-15 00:04 | - | ||
| sal/ | 2017-01-15 00:04 | - | ||
| rest-api.md | 2017-01-15 00:04 | 6.1K | ||
| require-libX.php | 2018-03-15 00:04 | 1.0K | ||
| readme.txt | 2017-01-15 00:04 | 15K | ||
| modules/ | 2017-01-15 00:04 | - |
It's obvious all files are dated from 2017-01-15 but:
| require-libX.php | 2018-03-15 00:04 | 1.0K |
That file has a different date file, so could be represented in the results in a another way ;)
Pentest box cannot install dirhunt command directly after installing dirhunt, it needs to manually add script to run, and can not find script location.
Is there any switch to use dirhunt with a file to crawl from list of domains ?
[200] http://xxxxxxx.in/admin/upload/ (Index Of)
Interesting name files: <map object at 0x7fe6885032e8>
python.exe: can't find 'main' module in 'dirhunt', how solve it ?
([SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:645))
Detect recursion loops for:
Continuous repeated dir:
/sub/A/B
/sub/A/A/B
/sub/A/A/A/B
Repeated same path:
/sub/A/B/C
/sub/A/B/C/A/B/C
/sub/A/B/C/A/B/C/A/B/C
They should not be considered loop:
Different start path:
/sub1/A/B
/sub2/A/A/B
/sub3/A/A/A/B
Middle directories:
/sub/A/B
/sub/A/C/A/B
/sub/A/A/C/A/B
Middle directories:
/sub/A/B/C
/sub/A/B/C/D/A/B/C
/sub/D/A/B/C/A/B/C/A/B/C
Related PR: #43
--dictionary <file.txt>
To improve file detection, would be great to compare the file name results with an external dictionary.
Possible representation:
[200] https://domain.com/wp-content/plugins/woocommerce/includes/gateways/simplify-commerce/includes/Simplify/ (Index Of)
Interesting extension files: AccessToken.php, Authentication.php, wp-link.php, ...
Dictionary matches: wp-link.php
or, if --dictionary <file.txt> is selected, to show results that match with the dictionary, only.
after starting "huntng" on some site where i working now, and in beginning all go be ok, but after 2 minutes of "hunting' i see a cycle redirects on site pages like: example.com/page/page/page/page/page and this "pages" count gonna be raise on next url "hunt" in current run, what could go wrong?
You are running Dirhunt v0.4.1 using Python 3.6.5.
Related PR: #43
For example:
dirhunt myawesomewebsite.com
Command not found issue after instalation according README
$pip install dirhunt
....
Running setup.py install for humanize ... done
Running setup.py install for dirhunt ... done
Successfully installed beautifulsoup4-4.6.0 certifi-2018.4.16 chardet-3.0.4 click-6.7 colorama-0.3.9 dirhunt-0.4.0 humanize-0.5.1 idna-2.7 requests-2.19.1 requests-mock-1.5.2 six-1.11.0 urllib3-1.23
$dirhunt http://website.com/
"dirhunt" command not found
Windows10 64bit, python 3.7.0
Original asset:
<link rel="stylesheet" href="css:custom.css">
Traceback (most recent call last):
File "/home/nekmo/.virtualenvs/dirhunt/lib/python3.6/site-packages/requests/models.py", line 371, in prepare_url
scheme, auth, host, port, path, query, fragment = parse_url(url)
File "/home/nekmo/.virtualenvs/dirhunt/lib/python3.6/site-packages/urllib3/util/url.py", line 199, in parse_url
raise LocationParseError(url)
urllib3.exceptions.LocationParseError: Failed to parse: website.comcss:custom.css
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/home/nekmo/Workspace/dirhunt/dirhunt/crawler.py", line 67, in start
resp = session.get(self.url.url, stream=True, timeout=TIMEOUT, allow_redirects=False)
File "/home/nekmo/Workspace/dirhunt/dirhunt/crawler.py", line 119, in get
response = self.session.get(url, **kwargs)
File "/home/nekmo/.virtualenvs/dirhunt/lib/python3.6/site-packages/requests/sessions.py", line 521, in get
return self.request('GET', url, **kwargs)
File "/home/nekmo/.virtualenvs/dirhunt/lib/python3.6/site-packages/requests/sessions.py", line 494, in request
prep = self.prepare_request(req)
File "/home/nekmo/.virtualenvs/dirhunt/lib/python3.6/site-packages/requests/sessions.py", line 437, in prepare_request
hooks=merge_hooks(request.hooks, self.hooks),
File "/home/nekmo/.virtualenvs/dirhunt/lib/python3.6/site-packages/requests/models.py", line 305, in prepare
self.prepare_url(url, params)
File "/home/nekmo/.virtualenvs/dirhunt/lib/python3.6/site-packages/requests/models.py", line 373, in prepare_url
raise InvalidURL(*e.args)
requests.exceptions.InvalidURL: Failed to parse: domain.comcss:custom.css
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/home/nekmo/Workspace/dirhunt/dirhunt/crawler.py", line 29, in wrapped
return func(*args, **kwargs)
File "/home/nekmo/Workspace/dirhunt/dirhunt/crawler.py", line 70, in start
self.close()
File "/home/nekmo/Workspace/dirhunt/dirhunt/crawler.py", line 110, in close
del self.crawler.processing[self.url.url]
Traceback (most recent call last):
File "/usr/lib/python3.6/site-packages/dirhunt/crawler.py", line 29, in wrapped
return func(*args, **kwargs)
File "/usr/lib/python3.6/site-packages/dirhunt/crawler.py", line 83, in start
processor.process(text, soup)
File "/usr/lib/python3.6/site-packages/dirhunt/processors.py", line 202, in process
self.links(soup)
File "/usr/lib/python3.6/site-packages/dirhunt/crawler.py", line 29, in wrapped
return func(*args, **kwargs)
File "/usr/lib/python3.6/site-packages/dirhunt/processors.py", line 210, in links
if not url.is_valid():
File "/usr/lib/python3.6/site-packages/dirhunt/url.py", line 124, in is_valid
return bool(self.urlparsed)
File "/usr/lib/python3.6/site-packages/dirhunt/crawler.py", line 83, in start
processor.process(text, soup)
File "/usr/lib/python3.6/site-packages/dirhunt/url.py", line 25, in urlparsed
address = address.url
File "/usr/lib/python3.6/site-packages/dirhunt/url.py", line 113, in url
return self.urlparsed[0] + '://' + self.urlparsed[1] + self.full_path
TypeError: 'NoneType' object is not subscriptable
I just ran dirhunt http://gravitytales.com to see what would happen on a site with many links...
Looks like dirhunt ran into some non ascii characters:
Traceback (most recent call last):
File "/usr/local/bin/dirhunt", line 11, in <module>
sys.exit(main())
File "/usr/local/lib/python2.7/dist-packages/dirhunt/management.py", line 160, in main
catch(hunt)()
File "/usr/local/lib/python2.7/dist-packages/dirhunt/exceptions.py", line 33, in wrap
fn(*args, **kwargs)
File "/home/user/.local/lib/python2.7/site-packages/click/core.py", line 722, in __call__
return self.main(*args, **kwargs)
File "/home/user/.local/lib/python2.7/site-packages/click/core.py", line 697, in main
rv = self.invoke(ctx)
File "/home/user/.local/lib/python2.7/site-packages/click/core.py", line 895, in invoke
return ctx.invoke(self.callback, **ctx.params)
File "/home/user/.local/lib/python2.7/site-packages/click/core.py", line 535, in invoke
return callback(*args, **kwargs)
File "/usr/local/lib/python2.7/dist-packages/dirhunt/management.py", line 151, in hunt
catch_keyboard_interrupt(crawler.print_results, crawler.restart)(set(exclude_flags), set(include_flags))
File "/usr/local/lib/python2.7/dist-packages/dirhunt/utils.py", line 37, in wrap
return fn(*args, **kwargs)
File "/usr/local/lib/python2.7/dist-packages/dirhunt/crawler.py", line 149, in print_results
self.echo(result)
File "/usr/local/lib/python2.7/dist-packages/dirhunt/crawler.py", line 115, in echo
self.std.write(str(body))
File "/usr/local/lib/python2.7/dist-packages/dirhunt/processors.py", line 194, in __str__
body = self.url_line()
File "/usr/local/lib/python2.7/dist-packages/dirhunt/processors.py", line 94, in url_line
body += ' {} '.format(self.crawler_url.url.url)
UnicodeEncodeError: 'ascii' codec can't encode characters in position 251-252: ordinal not in range(128)
I don't know which exact site causes this as the last url before the stacktrace changes... so just run it on the base domain and wait until it crashes...
Currently only supported:
On the future:
dirhunt http://domain1/blog/ --timeout15
-->no such option --timeout15
Dirhunt is not able to report certain false 404 errors because the program still does not have all the information during the analysis. A solution is an option to obtain a false 404 report.
Traceback (most recent call last):
File "/usr/lib/python3.6/site-packages/dirhunt/crawler.py", line 29, in wrapped
return func(*args, **kwargs)
File "/usr/lib/python3.6/site-packages/dirhunt/crawler.py", line 83, in start
processor.process(text, soup)
File "/usr/lib/python3.6/site-packages/dirhunt/processors.py", line 201, in process
self.assets(soup)
File "/usr/lib/python3.6/site-packages/dirhunt/processors.py", line 228, in assets
self.analyze_asset(asset)
File "/usr/lib/python3.6/site-packages/dirhunt/processors.py", line 237, in analyze_asset
if 'wordpress' not in self.crawler_url.flags and 'wp-content' in asset.path:
File "/usr/lib/python3.6/site-packages/dirhunt/url.py", line 85, in path
return self.urlparsed[2]
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
Django
The Web framework for perfectionists with deadlines.
Laravel
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
Microsoft
Open source projects and samples from Microsoft.
Google
Google ❤️ Open Source for everyone.
Alibaba
Alibaba Open Source for everyone
D3
Data-Driven Documents codes.
Tencent
China tencent open source team.