cern-cert / wad Goto Github PK
View Code? Open in Web Editor NEWWeb application/technology detection tool
License: GNU General Public License v3.0
Web application/technology detection tool
License: GNU General Public License v3.0
is there an option to suppress/skip " [Errno 104] connection reset by peer" as to avoid the script crashing?
Hey guys,
What's the most recent version of Python this supports?
Still might be useful to perform detection, regardless of certificate errors.
User should be able to extend clues with their files. Sometime user wants to merge your rules and their rules. But wad does not support this.
For example;
class Detector(object):
def __init__(self, extra_clues_file=None):
self.apps, self.categories = Clues.get_clues(extra_clues)
Happened with:
https://web.de/
https://www.spin.de/
Would be nice to have an option to update the rules to the latest version. I just submitted a PR upstream and it would be nice to use it with wad
if approved.
It seems that there is an issue with the 0.4.2 release which was published.
$ pip3 install wad
Collecting wad
Downloading https://files.pythonhosted.org/packages/e8/31/e27ce4d2fd36183b3df34b5764ecc671855de01be782a92e2bbfb26d1fc4/wad-0.4.2.tar.gz (109kB)
|████████████████████████████████| 112kB 1.5MB/s
ERROR: Complete output from command python setup.py egg_info:
ERROR: Traceback (most recent call last):
File "<string>", line 1, in <module>
File "/usr/lib64/python3.7/tokenize.py", line 447, in open
buffer = _builtin_open(filename, 'rb')
FileNotFoundError: [Errno 2] No such file or directory: '/tmp/pip-install-hwowozpm/wad/setup.py'
----------------------------------------
ERROR: Command "python setup.py egg_info" failed with error code 1 in /tmp/pip-install-hwowozpm/wad/
WARNING: You are using pip version 19.1.1, however version 20.1 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.
The archive doesn't looks properly created.
Hi,
i found this bug on using wad with python version 3.6.5 and wad 0.3.4
It happen on any url i try to check.
Traceback (most recent call last):
File "/usr/local/bin/wad", line 11, in
sys.exit(main())
File "/usr/local/lib/python3.6/dist-packages/wad/main.py", line 92, in main
Clues.get_clues(options.clues_file)
File "/usr/local/lib/python3.6/dist-packages/wad/clues.py", line 33, in get_clues
self.compile_clues()
File "/usr/local/lib/python3.6/dist-packages/wad/clues.py", line 124, in compile_clues
regexps[key + "_re"] = list(six.moves.map(self.compile_clue, self.apps[app][key]))
File "/usr/local/lib/python3.6/dist-packages/wad/clues.py", line 108, in compile_clue
regex_dict = {"re": re.compile(values[0], flags=re.IGNORECASE)}
File "/usr/lib/python3.6/re.py", line 233, in compile
return _compile(pattern, flags)
File "/usr/lib/python3.6/re.py", line 301, in _compile
p = sre_compile.compile(pattern, flags)
File "/usr/lib/python3.6/sre_compile.py", line 562, in compile
p = sre_parse.parse(p, flags)
File "/usr/lib/python3.6/sre_parse.py", line 855, in parse
p = _parse_sub(source, pattern, flags & SRE_FLAG_VERBOSE, 0)
File "/usr/lib/python3.6/sre_parse.py", line 416, in _parse_sub
not nested and not items))
File "/usr/lib/python3.6/sre_parse.py", line 502, in _parse
code = _escape(source, this, state)
File "/usr/lib/python3.6/sre_parse.py", line 401, in _escape
raise source.error("bad escape %s" % escape, len(escape))
sre_constants.error: bad escape \c at position 10
I've installed WAD using pip install wad
.
Now I am looking for any executable, but there's nothing to run. wad
is not on classpath, and pip show wad
shows me a dir with some scripts but no executable or help how to run it.
I suggest to either document where the wad script is or fix the installation.
win10 pythonx32 3.8.8 wad0.4.6
C:\WINDOWS\system32>wad -u 'https://example.com/'
(b7484665):clues:ERROR Error while reading JSON file, terminating: 'gbk' codec can't decode byte 0xa1 in position 2638: illegal multibyte sequence
Traceback (most recent call last):
File "c:\program files (x86)\python38-32\lib\runpy.py", line 194, in _run_module_as_main
return _run_code(code, main_globals, None,
File "c:\program files (x86)\python38-32\lib\runpy.py", line 87, in _run_code
exec(code, run_globals)
File "C:\Program Files (x86)\Python38-32\Scripts\wad.exe\__main__.py", line 7, in <module>
File "c:\program files (x86)\python38-32\lib\site-packages\wad\__main__.py", line 91, in main
Clues.get_clues(options.clues_file)
File "c:\program files (x86)\python38-32\lib\site-packages\wad\clues.py", line 38, in get_clues
self.load_clues(filename)
File "c:\program files (x86)\python38-32\lib\site-packages\wad\clues.py", line 94, in load_clues
self.apps, self.categories = self.read_clues_from_file(filename)
File "c:\program files (x86)\python38-32\lib\site-packages\wad\clues.py", line 55, in read_clues_from_file
clues = json.load(json_data)
File "c:\program files (x86)\python38-32\lib\json\__init__.py", line 293, in load
return loads(fp.read(),
UnicodeDecodeError: 'gbk' codec can't decode byte 0xa1 in position 2638: illegal multibyte sequence
After I edit WAD/wad/clues.py
Line 46 :
json_data = open(filename, encoding='utf-8')
But:
C:\WINDOWS\system32>wad -u 'https://example.com/'
(b7484665):detection:WARNING Error opening 'https://example.com/', terminating: <urlopen error unknown url type: 'https>
{}
That may because Python's built-in urllib module does not support HTTPS protocol, but;
C:\WINDOWS\system32>wad -u 'http://example.com/'
(a28d461b):detection:WARNING Error opening 'http://example.com/', terminating: <urlopen error unknown url type: 'http>
{}
???
funny little bug
how to reproduce:
wad.Detector().detect("https://webradio.divsi.unimi.it")
How about to implement python 3 support?
I found this bug, the packages are installed with pip
.
(venv) ➜ python --version
Python 3.5.0
(venv) ➜ pip freeze
six==1.10.0
wad==0.3.4
(venv) ➜ wad -u http://designfund.dk
Traceback (most recent call last):
File "/Users/brunsgaard/clerk/leadsland/venv/bin/wad", line 9, in <module>
load_entry_point('wad==0.3.4', 'console_scripts', 'wad')()
File "/Users/brunsgaard/clerk/leadsland/venv/lib/python3.5/site-packages/wad/__main__.py", line 94, in main
results = Detector().detect_multiple(urls, limit=options.limit, exclude=options.exclude, timeout=timeout)
File "/Users/brunsgaard/clerk/leadsland/venv/lib/python3.5/site-packages/wad/detection.py", line 78, in detect_multiple
res = self.detect(url, limit, exclude, timeout)
File "/Users/brunsgaard/clerk/leadsland/venv/lib/python3.5/site-packages/wad/detection.py", line 54, in detect
content = content.decode()
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xe6 in position 799: invalid continuation byte
All the logs are using the root logger.
Given this is a third party package, root logger should be avoided as it will interfere with the configurations of the main app running it.
An article about it: https://bubtaylor.com/stop-using-the-root-logger-in-python-1183bd89f4dd
Hello.
I´m launch WAD from subprocess.check_output.
The response have the logging messages, and corrupt the JSON parsed. Example:
b'**2020/09/25-09:44:08 (6a31b360):detection:WARNING Error opening https://elconsentimiento.es\n**{\n "https://elconsentimiento.es/": [\n {\n "app": "Apache",\n "ver": "2.4.10",\n "type": "Web Servers"\n },\n {\n "app": "Debian",\n "ver": null,\n "type": "Operating Systems"\n }\n ]\n}'
My fix is write all logging info in log files:
Example in detection.py:
logging.basicConfig(level=logging.ERROR, filename='{ABSOLUTE_PATH_TO_LOG_FILES}/logs/error.log')
logging.basicConfig(level=logging.WARNING, filename='{ABSOLUTE_PATH_TO_LOG_FILES}/logs/warning.log')
logging.basicConfig(level=logging.INFO, filename='{ABSOLUTE_PATH_TO_LOG_FILES}/logs/info.log')
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.