I'd like to use whisper similar to trivy

whisper --exit-code 1 --severity CRITICAL
in this case, whisper should only list CRITICAL findings and end its process with exit code 1

This tool is ideal for a CI pipeline and would be great if we have a whispers github action in the github action marketplace.

Context

I am manually running the CLI to find sensitive data in my project and I hope to automate it in a pipeline later. In the /code folder I have the structure of a react project

Problem

After running the command I find a problem similar to the issue #33.

whispers --rules aws-id,aws-secret,apikey,file-contains-password,file-contains-secret ./code

Prints

Python Version:

Command error output:

Currently setup.py uses simple find_packages() which includes 'tests', 'tests.unit', 'tests.unit.plugins' and 'tests.unit.rules' in a list of packages.
This results in tests being installed under site-packages, outside of whispers folder.

$ ls lib/python3.9/site-packages   
tests  whispers  whispers-1.4.7-py3.9.egg-info

The solution would be to use find_packages() with additional exclude parameter, like so:

find_packages(exclude=['tests', 'tests.*'])

it would be nice if whisper can scan remote git repositories.

Hey! Nice tool, just having a play and running into a couple of issues.

I'm trying to scan some code, if I use whispers with python3, I get parse errors when the code I'm trying to scan is python2, for example:

Traceback (most recent call last):
  File "/Users/user/.virtualenvs/path/lib/python3.9/site-packages/whispers/plugins/python.py", line 17, in pairs
    tree = astroid.parse(filepath.read_text())
  File "/Users/user/.virtualenvs/path/lib/python3.9/site-packages/astroid/builder.py", line 279, in parse
    return builder.string_build(code, modname=module_name, path=path)
  File "/Users/user/.virtualenvs/path/lib/python3.9/site-packages/astroid/builder.py", line 142, in string_build
    module = self._data_build(data, modname, path)
  File "/Users/user/.virtualenvs/path/lib/python3.9/site-packages/astroid/builder.py", line 170, in _data_build
    raise exceptions.AstroidSyntaxError(
astroid.exceptions.AstroidSyntaxError: Parsing Python code failed:
Missing parentheses in call to 'print'. Did you mean print(header)? (<unknown>, line 38)

I tried running whispers with python2, and I get the following build error:

ollecting whispers
  Using cached https://pypi.apple.com/packages/packages/db/12/9630c48c409c12cedb116d28e5e6d0402db81ec82ac6366dae9f7cfc02a1/whispers-1.3.12.tar.gz (26 kB)
  Installing build dependencies ... done
  Getting requirements to build wheel ... error
  ERROR: Command errored out with exit status 1:
   command: /Users/user/.virtualenvs/path/bin/python /Users/user/.virtualenvs/path/lib/python2.7/site-packages/pip/_vendor/pep517/_in_process.py get_requires_for_build_wheel /var/folders/kp/rn0d0qbj25gfj9sn5wg23pth0000gp/T/tmpGHkCim
       cwd: /private/var/folders/kp/rn0d0qbj25gfj9sn5wg23pth0000gp/T/pip-install-dfI8Zr/whispers
  Complete output (18 lines):
  Traceback (most recent call last):
    File "/Users/user/.virtualenvs/path/lib/python2.7/site-packages/pip/_vendor/pep517/_in_process.py", line 280, in <module>
      main()
    File "/Users/user/.virtualenvs/path/lib/python2.7/site-packages/pip/_vendor/pep517/_in_process.py", line 263, in main
      json_out['return_val'] = hook(**hook_input['kwargs'])
    File "/Users/user/.virtualenvs/path/lib/python2.7/site-packages/pip/_vendor/pep517/_in_process.py", line 114, in get_requires_for_build_wheel
      return hook(config_settings)
    File "/private/var/folders/kp/rn0d0qbj25gfj9sn5wg23pth0000gp/T/pip-build-env-kr5PVa/overlay/lib/python2.7/site-packages/setuptools/build_meta.py", line 146, in get_requires_for_build_wheel
      return self._get_build_requires(config_settings, requirements=['wheel'])
    File "/private/var/folders/kp/rn0d0qbj25gfj9sn5wg23pth0000gp/T/pip-build-env-kr5PVa/overlay/lib/python2.7/site-packages/setuptools/build_meta.py", line 127, in _get_build_requires
      self.run_setup()
    File "/private/var/folders/kp/rn0d0qbj25gfj9sn5wg23pth0000gp/T/pip-build-env-kr5PVa/overlay/lib/python2.7/site-packages/setuptools/build_meta.py", line 243, in run_setup
      self).run_setup(setup_script=setup_script)
    File "/private/var/folders/kp/rn0d0qbj25gfj9sn5wg23pth0000gp/T/pip-build-env-kr5PVa/overlay/lib/python2.7/site-packages/setuptools/build_meta.py", line 142, in run_setup
      exec(compile(code, __file__, 'exec'), locals())
    File "setup.py", line 3, in <module>
      from pathlib import Path
  ImportError: No module named pathlib

So currently I cannot use whispers if any of the target code is python2. Is there a way in the meantime to disable python parsing and treat it as a standard file?

Let's say I have a file that contains:

password: $ecret
And I use the following rule:

password:
  description: Variable names referring to passwords
  message: Password
  severity: CRITICAL
  key:
    regex: ^\S*(passwords?|passwd|pass|pwd)_?(hash)?[0-9]*$
    ignorecase: True
    isAscii: True
  value:
    isUri: False

Why does it not detect $secret? If it's s$cret it will be detected.

The maintained fork is available on: https://github.com/adeptex/whispers.

This repository should be switched has read-only and tagged as deprecated.

I have a simple config.yaml:

rules:
  oauth: 
    description: OAuth attributes
    message: OAuth
    severity: MAJOR
    key:
      regex: .*(client_id|client_secret).*$
      ignorecase: True

My Python code:

import whispers

print("Scan file file.txt")
for secret in whispers.secrets(f"--config config.yml --rules oauth file.txt"):
    print(secret)

The scanned file:

client_secret=test

No secret is detected / displayed.

Maybe I do not clearly understand how to use config file ?

Thanks

something like isEnv: False when the password is read from env? e.g. for php

{"file": "config/packages/doctrine.yaml", "line": 26, "key": "password", "value": "%env(resolve:DATABASE_PASSWORD)%", "message": "Password", "severity": "CRITICAL"}

but I guess that's hard to implement

Firstly, thank you for creating this tool. It is very helpful.

During a recent scan I noticed that folders with sensitive names (e.g. password, debugKeystore, etc) are flagged as problematic. I think that is a false positive that it should be safe to rule out? What's more, in the process of flagging these as issues whispers crashes with:

File "/usr/local/lib/python3.9/site-packages/whispers/core.py", line 88, in run
  for secret in whispers.scan(filename):
File "/usr/local/lib/python3.9/site-packages/whispers/secrets.py", line 90, in scan
  yield self.detect_secrets("file", plugin.filepath.as_posix(), plugin.filepath)
File "/usr/local/lib/python3.9/site-packages/whispers/secrets.py", line 83, in detect_secrets
  return self.rules.check(key, value, filepath, self.foundlines[filepath.as_posix()])
File "/usr/local/lib/python3.9/site-packages/whispers/rules/__init__.py", line 114, in check
  find_line_number(filepath, key, value, foundlines),
File "/usr/local/lib/python3.9/site-packages/whispers/utils.py", line 104, in find_line_number
  for line_number, line in enumerate(filepath.open().readlines(), 1):
File "/usr/local/Cellar/[email protected]/3.9.1/Frameworks/Python.framework/Versions/3.9/lib/python3.9/pathlib.py", line 1241, in open
  return io.open(self, mode, buffering, encoding, errors, newline,
IsADirectoryError: [Errno 21] Is a directory: ...

This seems to stem from whispers expecting a file for which it can determine a line number but actually it is dealing with a folder. See

A temporary workaround for anyone with this issue is to simply rename the problematic folder to something that won't cause a false positive.

File "/usr/local/lib/python3.9/dist-packages/whispers/utils.py", line 104, in find_line_number
for line_number, line in enumerate(filepath.open().readlines(), 1):
File "/usr/lib/python3.9/codecs.py", line 322, in decode
(result, consumed) = self._buffer_decode(data, self.errors, final)
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xfe in position 0: invalid start byte

File "/usr/local/lib/python3.9/dist-packages/whispers/plugins/init.py", line 60, in load_plugin
if self.filepath.open("r").readline().startswith("<?xml "):
File "/usr/lib/python3.9/codecs.py", line 322, in decode
(result, consumed) = self._buffer_decode(data, self.errors, final)
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xbc in position 21: invalid start byte

Fix:
utils.py:
for line_number, line in enumerate(filepath.open(errors='ignore').readlines(), 1):

whispers/plugins/init.py:
if self.filepath.open("r", errors='ignore').readline().startswith("<?xml "):

java、php、golang etc, these languages just use "=" to analyze hardcode. I think python use "=" to analyze is Enough. I don't know why author use astroid turn python to ast.
@adeptex

{
"errorMessage": "module 'typing' has no attribute '_ClassVar'",
"errorType": "AttributeError",
"requestId": "ac8561f5-dc64-42e5-af44-9f3c942ef80d",
"stackTrace": [
" File "/var/task/lambda_function.py", line 13, in lambda_handler\n for secret in whispers.secrets(args):\n",
" File "/opt/python/whispers/init.py", line 14, in secrets\n from whispers.core.args import parse_args\n",
" File "/opt/python/whispers/core/args.py", line 7, in \n from whispers.core.utils import DEFAULT_PATH, default_rules, load_regex\n",
" File "/opt/python/whispers/core/utils.py", line 12, in \n from whispers.models.pair import KeyValuePair\n",
" File "/opt/python/whispers/models/pair.py", line 6, in \n class KeyValuePair:\n",
" File "/opt/python/dataclasses.py", line 958, in dataclass\n return wrap(_cls)\n",
" File "/opt/python/dataclasses.py", line 950, in wrap\n return _process_class(cls, init, repr, eq, order, unsafe_hash, frozen)\n",
" File "/opt/python/dataclasses.py", line 800, in _process_class\n cls_fields = [_get_field(cls, name, type)\n",
" File "/opt/python/dataclasses.py", line 800, in \n cls_fields = [_get_field(cls, name, type)\n",
" File "/opt/python/dataclasses.py", line 659, in _get_field\n if (_is_classvar(a_type, typing)\n",
" File "/opt/python/dataclasses.py", line 550, in _is_classvar\n return type(a_type) is typing._ClassVar\n"
]
}

This involves a similar message to #33, but isn't the same.

Running whispers v1.3.7 on Windows, default config.

Encountering errors due to non-ASCII characters in the codebase, such as:

Traceback (most recent call last):
  File "C:\Python36_64\Lib\runpy.py", line 193, in _run_module_as_main
    "__main__", mod_spec)
  File "C:\Python36_64\Lib\runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "C:\Envs\whispers\Scripts\whispers.exe\__main__.py", line 7, in <module>
  File "c:\envs\whispers\lib\site-packages\whispers\cli.py", line 50, in cli
    for secret in run(args):
  File "c:\envs\whispers\lib\site-packages\whispers\core.py", line 90, in run
    for secret in whispers.scan(filename):
  File "c:\envs\whispers\lib\site-packages\whispers\secrets.py", line 92, in scan
    for ret in plugin.pairs():
  File "c:\envs\whispers\lib\site-packages\whispers\plugins\__init__.py", line 83, in pairs
    yield from self.plugin.pairs(self.filepath)
  File "c:\envs\whispers\lib\site-packages\whispers\plugins\javascript.py", line 8, in pairs
    for line in filepath.open("r").readlines():
  File "c:\envs\whispers\lib\encodings\cp1252.py", line 23, in decode
    return codecs.charmap_decode(input,self.errors,decoding_table)[0]
UnicodeDecodeError: 'charmap' codec can't decode byte 0x8d in position 87: character maps to <undefined>

This is due to the default encoding on Windows when reading from text files using file.open being cp1252, which has a few gaps (e.g. 0x8d).

I was able to fix this by changing line 8 of javascript.py to:

for line in filepath.open("r", encoding="utf8").readlines():

Also made similar changes to similar calls in json.py and plaintext.py.

Probably makes sense to make sure that whispers is using UTF-8 throughout on Windows.

Of course I don't know that this won't mess something up elsewhere, but it certainly allowed my run to complete (and found some useful things, thanks!)

Following error appeared while running the scan in CLI mode against a java application used the default config.yaml file provided.

ERROR

File "/usr/local/bin/whispers", line 8, in
sys.exit(cli())
File "/usr/local/lib/python3.8/dist-packages/whispers/cli.py", line 50, in cli
for secret in run(args):
File "/usr/local/lib/python3.8/dist-packages/whispers/core.py", line 88, in run
for secret in whispers.scan(filename):
File "/usr/local/lib/python3.8/dist-packages/whispers/secrets.py", line 91, in scan
for ret in plugin.pairs():
File "/usr/local/lib/python3.8/dist-packages/whispers/plugins/init.py", line 83, in pairs
yield from self.plugin.pairs(self.filepath)
File "/usr/local/lib/python3.8/dist-packages/whispers/plugins/javascript.py", line 8, in pairs
for line in filepath.open("r").readlines():
File "/usr/lib/python3.8/codecs.py", line 322, in decode
(result, consumed) = self._buffer_decode(data, self.errors, final)
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xb4 in position 100: invalid start byte