cern-cert / wad Goto Github PK

View Code? Open in Web Editor NEW

205.0 205.0 48.0 195 KB

Web application/technology detection tool

License: GNU General Public License v3.0

Python 100.00%

wad's People

Contributors

Stargazers

Watchers

wad's Issues

encoding parameter of json.load has been removed in Python 3.9

WAD/wad/clues.py

Line 52 in 6ef12ab

clues = json.load(json_data, encoding='utf-8')

https://bugzilla.redhat.com/show_bug.cgi?id=1834183

connection reset by peer

is there an option to suppress/skip " [Errno 104] connection reset by peer" as to avoid the script crashing?

Supported Python versions?

Hey guys,

What's the most recent version of Python this supports?

Crash when ssl cert doesn't match hostname

Still might be useful to perform detection, regardless of certificate errors.

[FEATURE REQUEST] Extendable clues

User should be able to extend clues with their files. Sometime user wants to merge your rules and their rules. But wad does not support this.

For example;

class Detector(object):
    def __init__(self, extra_clues_file=None):
        self.apps, self.categories = Clues.get_clues(extra_clues)

'utf-8' codec can't decode byte 0x8b in position 1: invalid start byte

Happened with:
https://web.de/
https://www.spin.de/

Add command to update rules

Would be nice to have an option to update the rules to the latest version. I just submitted a PR upstream and it would be nice to use it with wad if approved.

Issue with 0.4.2

It seems that there is an issue with the 0.4.2 release which was published.

$ pip3 install wad
Collecting wad
  Downloading https://files.pythonhosted.org/packages/e8/31/e27ce4d2fd36183b3df34b5764ecc671855de01be782a92e2bbfb26d1fc4/wad-0.4.2.tar.gz (109kB)
     |████████████████████████████████| 112kB 1.5MB/s 
    ERROR: Complete output from command python setup.py egg_info:
    ERROR: Traceback (most recent call last):
      File "<string>", line 1, in <module>
      File "/usr/lib64/python3.7/tokenize.py", line 447, in open
        buffer = _builtin_open(filename, 'rb')
    FileNotFoundError: [Errno 2] No such file or directory: '/tmp/pip-install-hwowozpm/wad/setup.py'
    ----------------------------------------
ERROR: Command "python setup.py egg_info" failed with error code 1 in /tmp/pip-install-hwowozpm/wad/
WARNING: You are using pip version 19.1.1, however version 20.1 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.

The archive doesn't looks properly created.

sre_constants.error

Hi,
i found this bug on using wad with python version 3.6.5 and wad 0.3.4
It happen on any url i try to check.

Traceback (most recent call last):
File "/usr/local/bin/wad", line 11, in
sys.exit(main())
File "/usr/local/lib/python3.6/dist-packages/wad/main.py", line 92, in main
Clues.get_clues(options.clues_file)
File "/usr/local/lib/python3.6/dist-packages/wad/clues.py", line 33, in get_clues
self.compile_clues()
File "/usr/local/lib/python3.6/dist-packages/wad/clues.py", line 124, in compile_clues
regexps[key + "_re"] = list(six.moves.map(self.compile_clue, self.apps[app][key]))
File "/usr/local/lib/python3.6/dist-packages/wad/clues.py", line 108, in compile_clue
regex_dict = {"re": re.compile(values[0], flags=re.IGNORECASE)}
File "/usr/lib/python3.6/re.py", line 233, in compile
return _compile(pattern, flags)
File "/usr/lib/python3.6/re.py", line 301, in _compile
p = sre_compile.compile(pattern, flags)
File "/usr/lib/python3.6/sre_compile.py", line 562, in compile
p = sre_parse.parse(p, flags)
File "/usr/lib/python3.6/sre_parse.py", line 855, in parse
p = _parse_sub(source, pattern, flags & SRE_FLAG_VERBOSE, 0)
File "/usr/lib/python3.6/sre_parse.py", line 416, in _parse_sub
not nested and not items))
File "/usr/lib/python3.6/sre_parse.py", line 502, in _parse
code = _escape(source, this, state)
File "/usr/lib/python3.6/sre_parse.py", line 401, in _escape
raise source.error("bad escape %s" % escape, len(escape))
sre_constants.error: bad escape \c at position 10

`pip install wad` does not put any executable on path

I've installed WAD using pip install wad.
Now I am looking for any executable, but there's nothing to run. wad is not on classpath, and pip show wad shows me a dir with some scripts but no executable or help how to run it.

I suggest to either document where the wad script is or fix the installation.

Fail to run properly under Windows

win10 pythonx32 3.8.8 wad0.4.6

C:\WINDOWS\system32>wad -u 'https://example.com/'
 (b7484665):clues:ERROR Error while reading JSON file, terminating: 'gbk' codec can't decode byte 0xa1 in position 2638: illegal multibyte sequence
Traceback (most recent call last):
  File "c:\program files (x86)\python38-32\lib\runpy.py", line 194, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "c:\program files (x86)\python38-32\lib\runpy.py", line 87, in _run_code
    exec(code, run_globals)
  File "C:\Program Files (x86)\Python38-32\Scripts\wad.exe\__main__.py", line 7, in <module>
  File "c:\program files (x86)\python38-32\lib\site-packages\wad\__main__.py", line 91, in main
    Clues.get_clues(options.clues_file)
  File "c:\program files (x86)\python38-32\lib\site-packages\wad\clues.py", line 38, in get_clues
    self.load_clues(filename)
  File "c:\program files (x86)\python38-32\lib\site-packages\wad\clues.py", line 94, in load_clues
    self.apps, self.categories = self.read_clues_from_file(filename)
  File "c:\program files (x86)\python38-32\lib\site-packages\wad\clues.py", line 55, in read_clues_from_file
    clues = json.load(json_data)
  File "c:\program files (x86)\python38-32\lib\json\__init__.py", line 293, in load
    return loads(fp.read(),
UnicodeDecodeError: 'gbk' codec can't decode byte 0xa1 in position 2638: illegal multibyte sequence

After I edit WAD/wad/clues.py Line 46 :

cjh0613@2eb201f

json_data = open(filename, encoding='utf-8')

But:

C:\WINDOWS\system32>wad -u 'https://example.com/'
(b7484665):detection:WARNING Error opening 'https://example.com/', terminating: <urlopen error unknown url type: 'https>
{}

That may because Python's built-in urllib module does not support HTTPS protocol, but;

C:\WINDOWS\system32>wad -u 'http://example.com/'
(a28d461b):detection:WARNING Error opening 'http://example.com/', terminating: <urlopen error unknown url type: 'http>
{}

???

(venv) ➜  python --version
Python 3.5.0
(venv) ➜  pip freeze
six==1.10.0
wad==0.3.4
(venv) ➜  wad -u http://designfund.dk
Traceback (most recent call last):
  File "/Users/brunsgaard/clerk/leadsland/venv/bin/wad", line 9, in <module>
    load_entry_point('wad==0.3.4', 'console_scripts', 'wad')()
  File "/Users/brunsgaard/clerk/leadsland/venv/lib/python3.5/site-packages/wad/__main__.py", line 94, in main
    results = Detector().detect_multiple(urls, limit=options.limit, exclude=options.exclude, timeout=timeout)
  File "/Users/brunsgaard/clerk/leadsland/venv/lib/python3.5/site-packages/wad/detection.py", line 78, in detect_multiple
    res = self.detect(url, limit, exclude, timeout)
  File "/Users/brunsgaard/clerk/leadsland/venv/lib/python3.5/site-packages/wad/detection.py", line 54, in detect
    content = content.decode()
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xe6 in position 799: invalid continuation byte

Declare and use loggers at module level

All the logs are using the root logger.
Given this is a third party package, root logger should be avoided as it will interfere with the configurations of the main app running it.

An article about it: https://bubtaylor.com/stop-using-the-root-logger-in-python-1183bd89f4dd

Logging info

Hello.

I´m launch WAD from subprocess.check_output.

The response have the logging messages, and corrupt the JSON parsed. Example:

b'**2020/09/25-09:44:08 (6a31b360):detection:WARNING Error opening https://elconsentimiento.es\n**{\n    "https://elconsentimiento.es/": [\n        {\n            "app": "Apache",\n            "ver": "2.4.10",\n            "type": "Web Servers"\n        },\n        {\n            "app": "Debian",\n            "ver": null,\n            "type": "Operating Systems"\n        }\n    ]\n}'

My fix is write all logging info in log files:

Example in detection.py:

logging.basicConfig(level=logging.ERROR, filename='{ABSOLUTE_PATH_TO_LOG_FILES}/logs/error.log')
logging.basicConfig(level=logging.WARNING, filename='{ABSOLUTE_PATH_TO_LOG_FILES}/logs/warning.log')
logging.basicConfig(level=logging.INFO, filename='{ABSOLUTE_PATH_TO_LOG_FILES}/logs/info.log')

cern-cert / wad Goto Github PK

wad's People

Contributors

Stargazers

Watchers

Forkers

wad's Issues

Recommend Projects

Recommend Topics

Recommend Org