shieldersec / webtech Goto Github PK

View Code? Open in Web Editor NEW

268.0 14.0 44.0 285 KB

Identify technologies used on websites.

Home Page: https://www.shielder.it/

License: GNU General Public License v3.0

Python 100.00%

webtech burpsuite

webtech's Introduction

WebTech

Identify technologies used on websites. More info on the release's blogpost.

CLI Installation

WebTech is available on pip:

pip install webtech

It can be also installed via setup.py:

python setup.py install --user

Burp Integration

Download Jython 2.7.0 standalone and install it into Burp.

In "Extender" > "Options" > "Python Environment":

Select the Jython jar location

Finally, in "Extender" > "Extension":

Click "Add"
Select "py" or "Python" as extension format
Select the Burp-WebTech.py file in this folder

Usage

Scan a website:

$ webtech -u https://example.com/

Target URL: https://example.com
...

$ webtech -u file://response.txt

Target URL:
...

Full usage:

$ webtech -h

Usage: webtech [options]

Options:
  -h, --help            show this help message and exit
  -u URLS, --urls=URLS  url(s) to scan
  --ul=URLS_FILE, --urls-file=URLS_FILE
                        url(s) list file to scan
  --ua=USER_AGENT, --user-agent=USER_AGENT
                        use this user agent
  --rua, --random-user-agent
                        use a random user agent
  --db=DB_FILE, --database-file=DB_FILE
                        custom database file
  --oj, --json          output json-encoded report
  --og, --grep          output grepable report
  --udb, --update-db    force update of remote db files

Use WebTech as a library

import webtech

# you can use options, same as from the command line
wt = webtech.WebTech(options={'json': True})

# scan a single website
try:
  report = wt.start_from_url('https://shielder.it')
  print(report)
except webtech.utils.ConnectionException:
  print("Connection error")

For more examples see webtech_example.py.

Resources for database matching

HTTP Headers information - http://netinfo.link/http/headers.html
Cookie names - https://webcookies.org/top-cookie-names

webtech's People

Contributors

Stargazers

Watchers

webtech's Issues

Database can not be updated because apps.json no longer exists.

The following URL, defined in database.py, does not exist anymore:

https://raw.githubusercontent.com/AliasIO/Wappalyzer/master/src/apps.json

Therefore database can not be updated

Fedora recently added webtech to their repositories, but webtech can not used because the included database is older than 30 days and program tries and fails to update database before doing anything.

Replacing https://raw.githubusercontent.com/AliasIO/Wappalyzer/master/src/apps.json with https://raw.githubusercontent.com/ShielderSec/webtech/master/webtech/apps.json in webtech/database.py helps solve the issue.

support system wide installation

I'm thinking to push your software (thanks for hard work!) to Pentoo Linux.
We install all files into system location so that a regular user would run it later.

I noticed that webtech has an update feature which will try to modify certain files.
That basically will fail because a regular user does not have a write access.

DATABASE_FILE = os.path.join(INSTALLATION_DIR, "webtech.json")
WAPPALYZER_DATABASE_FILE = os.path.join(INSTALLATION_DIR, "apps.json")

What you should do instead, is to create a local ~./webtech folder use it for any custom config files and updates.

Please consider.

Add ascii art of webtech #12

I have just contributed ascii art to webtech ,check PR#12 .
review #12 & add it. if it is useful

The Wappalyzer database seems offline.

No such file or directory: '/usr/local/lib/python3.8/dist-packages/webtech/apps.json'

Just started getting this error on a fresh Kali 2020.3 image. Any ideas?

kali@kali:~$ pip3 install -U webtech
Requirement already up-to-date: webtech in /usr/local/lib/python3.8/dist-packages (1.2.8)
Requirement already satisfied, skipping upgrade: requests in /usr/local/lib/python3.8/dist-packages (from webtech) (2.20.0)
Requirement already satisfied, skipping upgrade: idna<2.8,>=2.5 in /usr/local/lib/python3.8/dist-packages (from requests->webtech) (2.7)
Requirement already satisfied, skipping upgrade: urllib3<1.25,>=1.21.1 in /usr/local/lib/python3.8/dist-packages (from requests->webtech) (1.24.3)
Requirement already satisfied, skipping upgrade: certifi>=2017.4.17 in /usr/lib/python3/dist-packages (from requests->webtech) (2020.4.5.1)
Requirement already satisfied, skipping upgrade: chardet<3.1.0,>=3.0.2 in /usr/lib/python3/dist-packages (from requests->webtech) (3.0.4)

kali@kali:~$ webtech -u https://xerosecurity.com
Wappalyzer Database file not present.
Updating database...
Unable to update database, check your internet connection and Github.com availability.
Traceback (most recent call last):
  File "/usr/local/bin/webtech", line 8, in <module>
    sys.exit(main())
  File "/usr/local/lib/python3.8/dist-packages/webtech/__main__.py", line 53, in main
    wt = WebTech(options)
  File "/usr/local/lib/python3.8/dist-packages/webtech/webtech.py", line 69, in __init__
    with open(database.WAPPALYZER_DATABASE_FILE) as f:
FileNotFoundError: [Errno 2] No such file or directory: '/usr/local/lib/python3.8/dist-packages/webtech/apps.json'

The Wappalyzer database seems offline

It seems the Wappalyzer database is down for some reason.

regards
-imran

Getting RemoteDisconnected while downloading the database (during DB update)

Getting RemoteDisconnected error while updating the database.
webtech --version
webtech 1.3.2

ubuntu@ubuntu:~$ webtech --udb

Force update of Database file
Downloading database: https://raw.githubusercontent.com/ShielderSec/webtech/master/webtech/webtech.json
Downloading database: https://raw.githubusercontent.com/AliasIO/wappalyzer/master/src/technologies/a.json
The Wappalyzer database seems offline. Report this issue to: https://github.com/ShielderSec/webtech/
Downloading database: https://raw.githubusercontent.com/AliasIO/wappalyzer/master/src/technologies/b.json
Traceback (most recent call last):
  File "/usr/bin/webtech", line 8, in <module>
    sys.exit(main())
  File "/home/ubuntu/.local/lib/python3.10/site-packages/webtech/__main__.py", line 53, in main
    wt = WebTech(options)
  File "/home/ubuntu/.local/lib/python3.10/site-packages/webtech/webtech.py", line 68, in __init__
    database.update_database(force=update)
  File "/home/ubuntu/.local/lib/python3.10/site-packages/webtech/database.py", line 79, in update_database
    download()
  File "/home/ubuntu/.local/lib/python3.10/site-packages/webtech/database.py", line 55, in download
    download_database_file("{}{}.json".format(WAPPALYZER_DATABASE_URL_BASE,c), os.path.join(DATA_DIR,"temp.json"))
  File "/home/ubuntu/.local/lib/python3.10/site-packages/webtech/database.py", line 37, in download_database_file
    response = urlopen(url)
  File "/usr/lib/python3.10/urllib/request.py", line 216, in urlopen
    return opener.open(url, data, timeout)
  File "/usr/lib/python3.10/urllib/request.py", line 519, in open
    response = self._open(req, data)
  File "/usr/lib/python3.10/urllib/request.py", line 536, in _open
    result = self._call_chain(self.handle_open, protocol, protocol +
  File "/usr/lib/python3.10/urllib/request.py", line 496, in _call_chain
    result = func(*args)
  File "/usr/lib/python3.10/urllib/request.py", line 1391, in https_open
    return self.do_open(http.client.HTTPSConnection, req,
  File "/usr/lib/python3.10/urllib/request.py", line 1352, in do_open
    r = h.getresponse()
  File "/usr/lib/python3.10/http/client.py", line 1374, in getresponse
    response.begin()
  File "/usr/lib/python3.10/http/client.py", line 318, in begin
    version, status, reason = self._read_status()
  File "/usr/lib/python3.10/http/client.py", line 287, in _read_status
    raise RemoteDisconnected("Remote end closed connection without"
http.client.RemoteDisconnected: Remote end closed connection without response

First of all, thank you for this awesome tool. I've been testing it for a while and it's being doing a great job and very stable. Awesome that it exists as python SDK, can customize what I want and very simple API.

Here's a bug I've encountered when I was batch-scanning some URLs.

Reproduce:

url = 'http://news.co.th/news.co.th.zip'
wt = webtech.WebTech(options={
  'json': True,
})
report = wt.start_from_url(url)

Thrown error: unknown status keyword 'K' in marked section

Debug logs:

2022-07-26 17:57:44,470 [DEBUG] Generated user agent: Mozilla/5.0 (iPad; CPU OS 8_4_1 like Mac OS X) AppleWebKit/600.1.4 (KHTML, like Gecko) CriOS/43.0.2357.61 Mobile/12H321 Safari/600.1.4
2022-07-26 17:57:44,482 [DEBUG] Starting new HTTP connection (1): news.co.th:80
2022-07-26 17:57:45,094 [DEBUG] http://news.co.th:80 "GET /news.co.th.zip HTTP/1.1" 200 16314266
2022-07-26 18:01:52,014 [DEBUG] SHIFT_JIS Japanese prober hit error at byte 11
2022-07-26 18:01:52,014 [DEBUG] EUC-JP Japanese prober hit error at byte 10
2022-07-26 18:01:52,014 [DEBUG] GB2312 Chinese prober hit error at byte 11
2022-07-26 18:01:52,014 [DEBUG] EUC-KR Korean prober hit error at byte 10
2022-07-26 18:01:52,014 [DEBUG] CP949 Korean prober hit error at byte 11
2022-07-26 18:01:52,015 [DEBUG] Big5 Chinese prober hit error at byte 10
2022-07-26 18:01:52,015 [DEBUG] EUC-TW Taiwan prober hit error at byte 10
2022-07-26 18:01:57,974 [DEBUG] windows-1251 confidence = 0.04284181968104123, below negative shortcut threshhold 0.05
2022-07-26 18:02:03,923 [DEBUG] KOI8-R confidence = 0.041309005742054414, below negative shortcut threshhold 0.05
2022-07-26 18:02:09,873 [DEBUG] ISO-8859-5 confidence = 0.04158455218939259, below negative shortcut threshhold 0.05
2022-07-26 18:02:15,734 [DEBUG] MacCyrillic confidence = 0.043316827603991484, below negative shortcut threshhold 0.05
2022-07-26 18:02:21,768 [DEBUG] IBM866 confidence = 0.04203255622482825, below negative shortcut threshhold 0.05
2022-07-26 18:02:27,898 [DEBUG] IBM855 confidence = 0.04124558620798153, below negative shortcut threshhold 0.05
2022-07-26 18:02:45,389 [DEBUG] ISO-8859-5 confidence = 0.04065749192599313, below negative shortcut threshhold 0.05
2022-07-26 18:02:51,299 [DEBUG] windows-1251 confidence = 0.041915259363694175, below negative shortcut threshhold 0.05
2022-07-26 18:02:57,212 [DEBUG] TIS-620 confidence = 0.04620826074805007, below negative shortcut threshhold 0.05
2022-07-26 18:03:11,205 [DEBUG] windows-1255 confidence = 0.04420672943568373, below negative shortcut threshhold 0.05
2022-07-26 18:03:17,123 [DEBUG] windows-1255 confidence = 0.04426233160412784, below negative shortcut threshhold 0.05
2022-07-26 18:03:21,357 [DEBUG] windows-1251 not active
2022-07-26 18:03:21,357 [DEBUG] KOI8-R not active
2022-07-26 18:03:21,357 [DEBUG] ISO-8859-5 not active
2022-07-26 18:03:21,357 [DEBUG] MacCyrillic not active
2022-07-26 18:03:21,357 [DEBUG] IBM866 not active
2022-07-26 18:03:21,357 [DEBUG] IBM855 not active
2022-07-26 18:03:21,357 [DEBUG] ISO-8859-7 Greek confidence = 0.07158825613241834
2022-07-26 18:03:21,357 [DEBUG] windows-1253 Greek confidence = 0.0728303862329963
2022-07-26 18:03:21,357 [DEBUG] ISO-8859-5 not active
2022-07-26 18:03:21,357 [DEBUG] windows-1251 not active
2022-07-26 18:03:21,357 [DEBUG] TIS-620 not active
2022-07-26 18:03:21,357 [DEBUG] ISO-8859-9 Turkish confidence = 0.06570938840808227
2022-07-26 18:03:21,357 [DEBUG] windows-1255 Hebrew confidence = 0.0
2022-07-26 18:03:21,357 [DEBUG] windows-1255 not active
2022-07-26 18:03:21,357 [DEBUG] windows-1255 not active
2022-07-26 18:03:21,358 [DEBUG] no probers hit minimum threshold
2022-07-26 18:03:21,358 [DEBUG] utf-8  confidence = 0.010000000000000009
2022-07-26 18:03:21,358 [DEBUG] SHIFT_JIS Japanese confidence = 0.01
2022-07-26 18:03:21,358 [DEBUG] EUC-JP Japanese confidence = 0.01
2022-07-26 18:03:21,358 [DEBUG] GB2312 Chinese confidence = 0.01
2022-07-26 18:03:21,358 [DEBUG] EUC-KR Korean confidence = 0.01
2022-07-26 18:03:21,358 [DEBUG] CP949 Korean confidence = 0.01
2022-07-26 18:03:21,358 [DEBUG] Big5 Chinese confidence = 0.01
2022-07-26 18:03:21,358 [DEBUG] EUC-TW Taiwan confidence = 0.01
2022-07-26 18:03:21,358 [DEBUG] windows-1251 Russian confidence = 0.04284181968104123
2022-07-26 18:03:21,358 [DEBUG] KOI8-R Russian confidence = 0.041309005742054414
2022-07-26 18:03:21,358 [DEBUG] ISO-8859-5 Russian confidence = 0.04158455218939259
2022-07-26 18:03:21,358 [DEBUG] MacCyrillic Russian confidence = 0.043316827603991484
2022-07-26 18:03:21,358 [DEBUG] IBM866 Russian confidence = 0.04203255622482825
2022-07-26 18:03:21,358 [DEBUG] IBM855 Russian confidence = 0.04124558620798153
2022-07-26 18:03:21,358 [DEBUG] ISO-8859-7 Greek confidence = 0.07158825613241834
2022-07-26 18:03:21,358 [DEBUG] windows-1253 Greek confidence = 0.0728303862329963
2022-07-26 18:03:21,358 [DEBUG] ISO-8859-5 Bulgarian confidence = 0.04065749192599313
2022-07-26 18:03:21,358 [DEBUG] windows-1251 Bulgarian confidence = 0.041915259363694175
2022-07-26 18:03:21,358 [DEBUG] TIS-620 Thai confidence = 0.04620826074805007
2022-07-26 18:03:21,358 [DEBUG] ISO-8859-9 Turkish confidence = 0.06570938840808227
2022-07-26 18:03:21,358 [DEBUG] windows-1255 Hebrew confidence = 0.0
2022-07-26 18:03:21,358 [DEBUG] windows-1255 Hebrew confidence = 0.04420672943568373
2022-07-26 18:03:21,358 [DEBUG] windows-1255 Hebrew confidence = 0.04426233160412784
2022-07-26 18:03:21,358 [DEBUG] ISO-8859-1  confidence = 0.01

The Wappalyzer database seems offline.

Downloading database: https://raw.githubusercontent.com/AliasIO/wappalyzer/master/src/technologies/w.json
The Wappalyzer database seems offline. Report this issue to: https://github.com/ShielderSec/webtech/
Downloading database: https://raw.githubusercontent.com/AliasIO/wappalyzer/master/src/technologies/x.json
The Wappalyzer database seems offline. Report this issue to: https://github.com/ShielderSec/webtech/
Downloading database: https://raw.githubusercontent.com/AliasIO/wappalyzer/master/src/technologies/y.json
The Wappalyzer database seems offline. Report this issue to: https://github.com/ShielderSec/webtech/
Downloading database: https://raw.githubusercontent.com/AliasIO/wappalyzer/master/src/technologies/z.json
The Wappalyzer database seems offline. Report this issue to: https://github.com/ShielderSec/webtech/
Downloading database: https://raw.githubusercontent.com/AliasIO/wappalyzer/master/src/technologies/_.json
The Wappalyzer database seems offline. Report this issue to: https://github.com/ShielderSec/webtech/

❯ webtech --version
webtech 1.3.3

Failed with Json Output

The "Output Json Format" (--oj) doesn't seem to work well (please, don't print a json data... write into a file).

root@kali:~/test# webtech -u http://127.0.0.1:8000 --oj demo.json
{'http://127.0.0.1:8000': {'tech': [{'name': 'SimpleHTTP', 'version': '0.6'}, {'name': 'Python', 'version': '2.7.16'}], 'headers': []}}
root@kali:~/test# webtech -u http://127.0.0.1:8000 --oj demo.json | jq
parse error: Invalid numeric literal at line 1, column 7
root@kali:~/test# ls demo.json
ls: no se puede acceder a 'demo.json': No existe el fichero o el directorio

self.USER_AGENT doesn't exist in this class

webtech/webtech/target.py

Line 111 in 13f56ab

 return self.scrape_url("https://" + path, headers={'User-Agent': self.USER_AGENT}, cookies={}) 

argument of type 'NoneType' is not iterable

https://github.com/ShielderSec/webtech/blob/master/webtech/target.py#L101

ct = response.headers.get('Content-Type', None)
crash if ct is None

raise source.error("invalid group reference %d" % group, len(escape) - 1) re.error: invalid group reference 1 at position 130

Webtech was working fine for me for a while, but now I'm getting the following error.

Any ideas?

 root@kali2019  /pentest/mobile/testing  pip3 install webtech                                                                                                            SIGINT(2) ↵  ⚡  1144  13:50:39 
Requirement already satisfied: webtech in /usr/local/lib/python3.7/dist-packages (1.2.7)
Requirement already satisfied: requests in /usr/local/lib/python3.7/dist-packages (from webtech) (2.18.4)
Requirement already satisfied: urllib3<1.23,>=1.21.1 in /usr/local/lib/python3.7/dist-packages (from requests->webtech) (1.22)
Requirement already satisfied: certifi>=2017.4.17 in /usr/lib/python3/dist-packages (from requests->webtech) (2018.8.24)
Requirement already satisfied: idna<2.7,>=2.5 in /usr/lib/python3/dist-packages (from requests->webtech) (2.6)
Requirement already satisfied: chardet<3.1.0,>=3.0.2 in /usr/lib/python3/dist-packages (from requests->webtech) (3.0.4)

 root@kali2019  ~  webtech -u https://xerosecurity.com                                                                                                                             ✔  ⚡  1151  13:46:17 
Traceback (most recent call last):
  File "/usr/local/bin/webtech", line 8, in <module>
    sys.exit(main())
  File "/usr/local/lib/python3.7/dist-packages/webtech/__main__.py", line 54, in main
    wt.start()
  File "/usr/local/lib/python3.7/dist-packages/webtech/webtech.py", line 132, in start
    temp_output = self.start_from_url(url)
  File "/usr/local/lib/python3.7/dist-packages/webtech/webtech.py", line 172, in start_from_url
    return self.perform(target)
  File "/usr/local/lib/python3.7/dist-packages/webtech/webtech.py", line 222, in perform
    target.check_html(tech, html)
  File "/usr/local/lib/python3.7/dist-packages/webtech/target.py", line 221, in check_html
    matches = re.search(source, self.data['html'], re.IGNORECASE)
  File "/usr/lib/python3.7/re.py", line 183, in search
    return _compile(pattern, flags).search(string)
  File "/usr/lib/python3.7/re.py", line 286, in _compile
    p = sre_compile.compile(pattern, flags)
  File "/usr/lib/python3.7/sre_compile.py", line 764, in compile
    p = sre_parse.parse(p, flags)
  File "/usr/lib/python3.7/sre_parse.py", line 924, in parse
    p = _parse_sub(source, pattern, flags & SRE_FLAG_VERBOSE, 0)
  File "/usr/lib/python3.7/sre_parse.py", line 420, in _parse_sub
    not nested and not items))
  File "/usr/lib/python3.7/sre_parse.py", line 501, in _parse
    code = _escape(source, this, state)
  File "/usr/lib/python3.7/sre_parse.py", line 399, in _escape
    raise source.error("invalid group reference %d" % group, len(escape) - 1)
re.error: invalid group reference 1 at position 130

Failed with html file parse

When I try to read an html file (obtained with wget) I get an error:

root@kali:~/test# webtech -u file:///root/test/data.html
Traceback (most recent call last):
  File "/usr/local/bin/webtech", line 11, in <module>
    load_entry_point('webtech==1.2.6', 'console_scripts', 'webtech')()
  File "/usr/local/lib/python3.7/dist-packages/webtech-1.2.6-py3.7.egg/webtech/__main__.py", line 54, in main
    wt.start()
  File "/usr/local/lib/python3.7/dist-packages/webtech-1.2.6-py3.7.egg/webtech/webtech.py", line 132, in start
    temp_output = self.start_from_url(url)
  File "/usr/local/lib/python3.7/dist-packages/webtech-1.2.6-py3.7.egg/webtech/webtech.py", line 168, in start_from_url
    target.parse_http_file(url)
  File "/usr/local/lib/python3.7/dist-packages/webtech-1.2.6-py3.7.egg/webtech/target.py", line 120, in parse_http_file
    return self.parse_http_request(data)
  File "/usr/local/lib/python3.7/dist-packages/webtech-1.2.6-py3.7.egg/webtech/target.py", line 182, in parse_http_request
    replay_url = "https://" + host + replay_uri
UnboundLocalError: local variable 'host' referenced before assignment

I'm using like the example in the readme file:

$ webtech -u file://response.txt

join forces with wappalyzer

wappalyzer been around for much longer, so I suspect that they have a bigger database.

Apparently, both databases in JSON format and it may be make some sense to join forces and use one database?
https://github.com/AliasIO/wappalyzer/blob/master/src/apps.json

For your consideration please

thezero@sesamo:~$ webtech --udb
Force update of Wappalyzer Database file
Updating database...
Database updated successfully!
Force update of WebTech Database file
Updating database...
Database updated successfully!
Traceback (most recent call last):
  File "/home/thezero/.local/bin/webtech", line 10, in <module>
    sys.exit(main())
  File "/home/thezero/.local/lib/python3.6/site-packages/webtech/__main__.py", line 54, in main
    wt.start()
  File "/home/thezero/.local/lib/python3.6/site-packages/webtech/webtech.py", line 130, in start
    for url in self.urls:
TypeError: 'NoneType' object is not iterable