Git Product home page Git Product logo

ochronasec / ochrona-cli Goto Github PK

View Code? Open in Web Editor NEW
51.0 4.0 8.0 3.34 MB

A command line tool for detecting vulnerabilities in Python dependencies and doing safe package installs

Home Page: https://ochrona.dev

License: MIT License

Python 95.51% Dockerfile 0.04% Jinja 4.45%
security security-tools vulnerabilities vulnerability-scanners dependency-analysis supply-chain python developer-tools pipfile pip requirements devsecops

ochrona-cli's Introduction

Ochrona PyPI Versions License Downloads Vuln DB Version codecov Code style: black


NOTICE

Ochrona is no longer under active development. The accompanying Python Vulnerability Repository is also no longer receiving regular updates.

Overview

Ochrona is a free solution for securing the dependencies used in Python projects. Ochrona also includes support for policies which give you additional control over what aspects of your dependency usage you'd like to be alerted on.

The Ochrona maintainers care deeply about Developer Experience (DX), if you have any feedback or run into issues please open an issue here.

Vulnerability Data

Ochrona maintains its own database of vulnerabilities impacting Python packages. You're welcome to check out the database here. This database is updated frequently using data from NVD, Github, and other sources. Ochrona will update its local copy of the database if a new version exists.

Supported file types

  • *requirements*.txt
  • Pipfile.lock
  • poetry.lock
  • setup.py
  • *constraints*.txt
  • environment.yml
  • tox.ini

Installation

via pip

pip install ochrona

via pipenv

pipenv install --dev ochrona

via poetry

poetry add -D ochrona

Configuration

via command line args

Arg Description Type Example
--dir Directory to recursively search for dependencies files to scan [.] path /User/me/my_project
--exclude_dir Directory names that should be excluded from recursive search. Comma separated str build,dev
--file Single dependency file to scan file /User/me/my_project/requirements.txt
--debug Enable debug logging [False] bool True
--silent Silent mode [False] bool True
--report_type The report type that's desired [BASIC] str XML
--output Location for report output path /User/me/my_project/logs
--exit Exit with Code 0 regardless of vulnerability findings. [False] bool True
--ignore Ignore a CVE or package str requests
--include_dev Include develop dependencies from Pipfile.lock [False] bool True
--sbom Must be provided, along with --output to generate an SBOM [False] bool True
--sbom_format Whether the generated SBOM should be a JSON or XML file. [JSON] str JSON
--enable_sast Whether SAST checks should be run [False] bool True
--sast_id_exclude Whether a SAST check should be ignored str O001
--sast_dir Which directory should SAST checks run against path /User/me/my_project/src

via environment variables

Variable Name Corresponding Arg
OCHRONA_DEBUG_LOGGING --debug
OCHRONA_IGNORED_VULNS --ignore

via .ochrona.yml

There is an empty .ochrona.yml file included in the repo.

Key Description Type Example
dir Directory to recursively search for dependencies files to scan [.] path /User/me/my_project
exclude_dir Directory names that should be excluded from recursive search. array build
file Single dependency file to scan file /User/me/my_project/requirements.txt
debug Enable debug logging [false] bool true
silent Silent mode [false] bool true
report_type The report type that's desired [BASIC] str XML
report_location Location for report output [.] path /User/me/my_project/logs
exit Exit with Code 0 regardless of vulnerability findings [false] bool true
ignore Ignore a CVE or package name. Supports multiple values. str requests
include_dev Include develop dependencies from files that support dev/required dependencies [false] bool true
color_output Whether or not std out text should use color. Note: this is enabled by default when running in a non-Windows environment [true] bool false
policies Policies are a way of defining additional checks against your dependencies. See here for more details array details
sbom SBOM will only be generated if this argument is supplied. report_location also must be specified to generate an SBOM. See here for more details about SBOMs [false] bool false
sbom_format Whether the generated SBOM should be a JSON or XML file. [JSON] str JSON
enable_sast Whether SAST checks should be run [False] bool true
sast_id_exclude Whether a SAST check should be ignored. See list of SAST IDs here. Supports multiple values. str O001
sast_dir Which directory should SAST checks run against. [.] path /User/me/my_project/src

Example:

# debug: true
# silent: false
# dir: .
# report_type: JSON
# report_location: .
# ignore: 
# - requests
# include_dev: false
# color_output: false
# policies:
#  - license_type NIN APSL,GPL-PA,JSON
# sbom: true
# sbom_format: JSON
# enable_sast: true
# sast_id_exclude:
#  - O001
# sast_dir: ./src

Policies

Policies are a way to add additional check to your Python dependency usage. Policies can be defined using conditional and logical syntax. These generic policy definitions allow you to define unique, custom policies to fit your need, and they can be extensible as new fields and capabilities are added.

Policy vioations are not the same as vulnerabilities, however violations will cause Ochrona to emit a failure exit code and the output will include details about the policy violation.

Policy Syntax

At their most basic, policies are defined using conditional statements and logical operators. A conditional statement is structured as <field><operator><value>, for example, license_type == MIT. Whitespace is always ignored during policy evaluation.

Allowed Fields

Name Description
name The name of a package.
license_type An SPDX license type for a package.
latest_version The most recent version of a package.
latest_update The timestamp for when a package was last updated. ISO-8601 Format YYYY-MM-DDTHH:MM-SS.ffffffZ
release_count The number of releases a package has on Pypi.

Allowed Conditional Operators

Operator Description
== An equals operator for comparing exact string matches.
!= A NOT equals operators for non-matching strings.
< Less than, for comparing numerical or string values.
<= Less than or equal to.
> Greater than, for comparing numerical or string values.
>= Greater than or equal to.
IN For checking whether a value exists within a set. Values used with IN should be comma separated.
NIN For checking that a value does not exist within a set. Values used with NIN should be comma separated.

Allowed Logical Operators

Operator Description
AND For checking that both adjacent conditions are true.
OR For checking that at least one adjacent condition is true.

Special Values

Value Description
NOW-N Shorthand for an ISO 8601 formatted date in the past. N will be an integer number of days.
* Match ANY value, only works with == operator.

Precedence

Policies allow the usage of ( and ) to indicate evaluation precedence and grouping. Example (license_type==Apache-2.0 OR license_type==MIT) AND latest_update < NOW-30

Policy Examples

# Policy to check that a license type is in my aproved list
license_type IN MIT,ISC,Apache-2.0

# Policy to check that all packages have been updated this year
latest_update >= NOW-365

SBOM

Software Bill-of-Materials (SBOM) are a list of the components used to build a piece of software. They aim to make the delivery and composition of software components more transparent. These documents can be useful for understanding software supply chains and ensuring license complaince.

Ochrona has opted to support (CycloneDX)[https://cyclonedx.org/] as our SBOM standard of choice. CycloneDX is a lightweight software bill of materials (SBOM) standard designed for use in application security contexts and supply chain component analysis. CycloneDX is developed and support by (OWASP)[https://owasp.org/www-project-cyclonedx/].

To enable sbom generation, the --sbom and --output arguements needs to be provided.

Static Analysis Beta

Ochrona has introduced limited Static Application Security Testing (SAST) in version 2.0.0 as a beta feature. These check are run if the --enable_sast argument is provided. It will run recursively against all .py files found in the current working directory.

SAST Checks

ID Description
O001 Checks for use of "exec" from the Standard Library.
O002 Checks for use of "eval" from the Standard Library.
O003 Checks for use of the "assert" keyword.
O004 Checks for use of "tarfile.extractall" from the Standard Library.
O005 Checks for use of "pickle.loads" from the Standard Library.
O006 Checks for use of "xml.etree.ElementTree.parse" from the Standard Library.
O101 Checks for use of "load" from PyYAML.
O102 Checks for usage of Requests with verify=False.
O103 Checks for usage of Flask with debug=True.

Usage Examples

Default Mode

$ ochrona

This will search for any supported dependency files recursively from the run location. It will output rules in the BASIC format to stdout. The program will exit with an error exit code if any confirmed vulnerabilities are found.

Standard error code with Junit XML reporting saved to file

$ ochrona --exit --report_type XML --output ./output

Safe Import Mode

In this mode ochrona acts as a safe wrapper around standard pip installs to ensure that a package and it's dependencies are safe before installing. This action preemptively checks a package and only imports if no vulnerabilities are found. It can be used with a base package (i.e. requests), or with a package pinned to an exact version (i.e. requests==2.21.0). It also supports importing a requirements.txt style, the pip equivalent of pip install -r <file>.

$ ochrona --install <package_name>|<requirements.txt>

stdin Support

Ochrona supports supplying dependencies via stdin and can accept a PEP-508 complaint (i.e. requirements.txt) formated string, or a single dependency. Single dependencies can be supplied as the first argument or piped.

Single dependency via stdin

$ ochrona urllib3==1.26.4
$ echo "urllib3==1.26.4" | ochrona

Multi-dependency via stdin

$ pip freeze | ochrona
$ pipenv lock -r | ochrona
$ cat requirements.txt | ochrona

Docker Support

Ochrona can be run via Docker. This is useful for the paranoid who may worry that an installed module could have modified the Python package namespace and allow malicious packages to bypass Ochrona's security checks. We've added this support in response to CVE-2020-5252 which was disclosed prior to Ochrona and affects several other similar tools.

Dockerized Ochrona

$ pip freeze | docker run -i --rm ochrona/ochrona ochrona

Dockerized Ochrona with environment variables

$ pip freeze | docker run -i -e OCHRONA_IGNORED_VULNS=requests --rm ochrona/ochrona ochrona

Output Formats

Ochrona supports several built in output options include a BASIC and FULL plaintext reports, a Junit style XML report, a JSON style report for incorporating with other tools, and and HTML summarized report.

Basic

Full

XML (Junit)

JSON

HTML

Represent!

Ochrona

Let the world know you're keeping your project safe with Ochrona. Add our shield to your README.md by adding the following line.

[![Ochrona](https://img.shields.io/badge/secured_by-ochrona-blue)](https://ochrona.dev)

ochrona-cli's People

Contributors

beatsbears avatar mathbou avatar ochronasec avatar pepastach avatar psibre avatar ttw225 avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar

ochrona-cli's Issues

Ochrona can't parse version_value for tornado

Describe the bug
When evaluation tornado the line to get the vuln_specifiers fails because the version value is "-". This raises an InvalidSpecifier error from the packaging module.

To Reproduce
Running orchrona tornado on version 2.0.2

Expected behavior
An error not to be raised and ochrona to detect potential vulnerabilities.

Background (please complete the following information):

  • OS: WSL2 Ubuntu 22.04
  • Python Version: 3.10.6
  • Ochrona Version: 2.0.2
  • Ochrona Arguments: tornado
  • Debug output:
ochrona tornado --debug

,---.     |                        
|   |,---.|---.,---.,---.,---.,---.
|   ||    |   ||    |   ||   |,---|
`---'`---'`   '`    `---'`   '`---^
                           v. 2.0.2 
                           https://ochrona.dev   
                           
Processing 0 Files... ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━   0% -:--:--
(DEBUG) Discovered dependencies: 
--- tornado
(DEBUG) DB instance found: 2022.07.14
(DEBUG) Found 4 vulnerabilities potentially affecting package: tornado
Traceback (most recent call last):
 File "/home/arichardson/experiments/ochrona-cli/ochrona/eval/vuln/evaluate.py", line 64, in evaluate
   vuln_specifiers &= f"=={versions['version_value']}"
 File "/home/arichardson/experiments/ochrona-cli/.venv/lib/python3.10/site-packages/packaging/specifiers.py", line 778, in __and__
   other = SpecifierSet(other)
 File "/home/arichardson/experiments/ochrona-cli/.venv/lib/python3.10/site-packages/packaging/specifiers.py", line 700, in __init__
   parsed.add(Specifier(specifier))
 File "/home/arichardson/experiments/ochrona-cli/.venv/lib/python3.10/site-packages/packaging/specifiers.py", line 234, in __init__
   raise InvalidSpecifier(f"Invalid specifier: '{spec}'")
packaging.specifiers.InvalidSpecifier: Invalid specifier: '==-'

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
 File "/home/arichardson/experiments/ochrona-cli/.venv/bin/ochrona", line 33, in <module>
   sys.exit(load_entry_point('ochrona', 'console_scripts', 'ochrona')())
 File "/home/arichardson/experiments/ochrona-cli/.venv/lib/python3.10/site-packages/click/core.py", line 1130, in __call__
   return self.main(*args, **kwargs)
 File "/home/arichardson/experiments/ochrona-cli/.venv/lib/python3.10/site-packages/click/core.py", line 1055, in main
   rv = self.invoke(ctx)
 File "/home/arichardson/experiments/ochrona-cli/.venv/lib/python3.10/site-packages/click/core.py", line 1404, in invoke
   return ctx.invoke(self.callback, **ctx.params)
 File "/home/arichardson/experiments/ochrona-cli/.venv/lib/python3.10/site-packages/click/core.py", line 760, in invoke
   return __callback(*args, **kwargs)
 File "/home/arichardson/experiments/ochrona-cli/ochrona/cli/cli.py", line 177, in run
   results.append(resolve(**payload))
 File "/home/arichardson/experiments/ochrona-cli/ochrona/eval/eval.py", line 30, in resolve
   resp.confirmed_vulnerabilities = evaluate(vulns, resp.flat_list)
 File "/home/arichardson/experiments/ochrona-cli/ochrona/eval/vuln/evaluate.py", line 104, in evaluate
   raise Exception("evaluate exception") from ex
Exception: evaluate exception

Auto-publish new images to docker hub

Feature Overview
A github action should be added to auto-publish to docker hub.

Is your feature request related to a problem?
Reduced manual release steps

Additional context
N/A

Error when run ochrona command using a .ochrona.yml without defining a policies key.

Describe the bug
I need to execute the ochrona validation using a .ochrona.yml with only ignore key, but if I don't define a policies configuration with some parameter or even though an empty police I receive this error.

  File ".venv/bin/ochrona", line 8, in <module>
    sys.exit(run())
  File ".venv/lib/python3.8/site-packages/click/core.py", line 1130, in __call__
    return self.main(*args, **kwargs)
  File ".venv/lib/python3.8/site-packages/click/core.py", line 1055, in main
    rv = self.invoke(ctx)
  File ".venv/lib/python3.8/site-packages/click/core.py", line 1404, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File ".venv/lib/python3.8/site-packages/click/core.py", line 760, in invoke
    return __callback(*args, **kwargs)
  File ".venv/lib/python3.8/site-packages/ochrona/cli/cli.py", line 107, in run
    config = OchronaConfig(
  File ".venv/lib/python3.8/site-packages/ochrona/config/config.py", line 46, in __init__
    valid = self._validate()
  File ".venv/lib/python3.8/site-packages/ochrona/config/config.py", line 175, in _validate
    if len(self._policies) > 0:
TypeError: object of type 'NoneType' has no len()

But, If I use the .ochrona.yml with the follow content the ochrona validation work normally:

ignore:
  - requests
policies: []

But, adding empty keys for a configuration that I won't use doesn't seem to be necessary.

Apparently the error can occur due to this line because if the polices key does not exist in .ochrona.yml a None value will be returned to self._policies.

To Reproduce
Steps to reproduce the behavior:

  • Create .ochrona.yml file.
  • Add the follow content in .ochrona.yml file:
  ignore: 
    - requests
  • Run ochrona command:
ochrona

Expected behavior
Run the vulnerability validation using only ignore key inside .ochrona.yml.

Background (please complete the following information):

  • OS: Ubuntu 20.04.4 LTS
  • Python Version: 3.8.10
  • Ochrona Version: 2.0.0
  • Ochrona Arguments: None or -r requirements.txt --report_type FULL

Auto-publish to Pypi on push to master

Feature Overview
ochrona package should auto-publish to pypi and cut a new release on push to master.

Is your feature request related to a problem?
Reduces manual effort for new releases

Additional context
N/A

Unexpected Response from https://api.ochrona.dev/python/analyze - Internal server error

Running the latest version of Ochrona v0.0.17 via pipenv run ochrona I get the following message:

[DEBUG] Unexpected Response from https://api.ochrona.dev/python/analyze - {"message": "Internal server error"}
[!] Unexpected result from analysis, please try again later. If this persists, please report this at https://github.com/ochronasec/ochrona-cli/issues
make: *** [check] Error 1

MacOS Big Sur: 11.2.1
Python Version: 3.8.7

Pipfile

[[source]]
name = "pypi"
url = "https://pypi.org/simple"
verify_ssl = true

[dev-packages]
bandit = "*"
black = "20.8b1"
mypy = "*"
ochrona = "*"

[packages]
flask = "1.1.2"

[requires]
python_version = "3.8.6"

ochrona.yml

---
debug: true
silent: false
dir: .
report_type: JSON
report_location: ./output
include_dev: true
project_name: pythonsecuritytools

Full debug output

[DEBUG] Found matching requirements*.txt file at .venv/lib/python3.8/site-packages/pbr/tests/testpackage/test-requirements.txt
[DEBUG] Found matching pipfile.lock file at Pipfile.lock
[DEBUG] Found matching setup.py file at .venv/lib/python3.8/site-packages/mypyc/lib-rt/setup.py
[DEBUG] Found matching setup.py file at .venv/lib/python3.8/site-packages/pbr/tests/test_setup.py
[DEBUG] Found matching setup.py file at .venv/lib/python3.8/site-packages/pbr/tests/testpackage/setup.py
[DEBUG] Found matching setup.py file at .venv/lib/python3.8/site-packages/stevedore/example2/setup.py
[DEBUG] Found matching setup.py file at .venv/lib/python3.8/site-packages/stevedore/example/setup.py
[DEBUG] Discovered dependencies: ["ordereddictpython_version=='2.6'", 'requests-mock']
[DEBUG] Discovered dependencies: ['click==7.1.2', 'flask==1.1.2', 'itsdangerous==1.1.0', 'jinja2==2.11.3', 'markupsafe==1.1.1', 'werkzeug==1.0.1', 'appdirs==1.4.4', 'bandit==1.7.0', 'black==20.8b1', 'certifi==2020.12.5', 'chardet==4.0.0', 'click==7.1.2', 'gitdb==4.0.5', 'gitpython==3.1.13', 'idna==2.10', 'mypy==0.812', 'mypy-extensions==0.4.3', 'ochrona==0.0.17', 'pathspec==0.8.1', 'pbr==5.5.1', 'pyyaml==5.4.1', 'regex==2020.11.13', 'requests==2.25.1', 'six==1.15.0', 'smmap==3.0.5', 'stevedore==3.3.0', 'toml==0.10.2', 'typed-ast==1.4.2', 'typing-extensions==3.7.4.3', 'urllib3==1.26.3']
[DEBUG] Discovered dependencies: []
[DEBUG] Discovered dependencies: ['', 'from', '', 'import', '', 'try:', '', '', 'except', '', '', '', 'import', '', 'from', 'from', 'from', 'from', '', '', 'class', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '_changelog_content', '04316fe\\x00Make', '378261a\\x00Add', '3c373ac\\x00Merge', '182feb3\\x00Fix', 'fa4f46e\\x00Remove', 'd1c53dd\\x00Use', 'a793ea1\\x00Merge', '6c27ce7\\x00Skip', '451e513\\x00Bug', '4c8cfe4\\x00Improve', 'd7e6167\\x00Bug', 'c47ec15\\x00Consider', '8696fbd\\x00Improve', 'f0440f8\\x00Improve', '04984a5\\x00Refactor', 'a65e8ee\\x00Remove', '"""', '', '', 'def', '', '', '', '', '', '', '', '', '', '_old_git_changelog_content', '', '', '', '', 'class', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', 'class', '', '', '', 'class', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', 'class', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', 'class', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '']
[DEBUG] Unexpected Response from https://api.ochrona.dev/python/analyze - {"message": "Internal server error"}

Directory with one Python file error

On Windows 10 and in Powershell I ran ochrona --api_key hidden_key --dir directory_with_one_python_file and received the following error.

,---.     |
|   |,---.|---.,---.,---.,---.,---.
|   ||    |   ||    |   ||   |,---|
`---'`---'`   '`    `---'`   '`---^
                            v. 0.0.17
                            https://ochrona.dev

Traceback (most recent call last):
  File "c:\python39\lib\runpy.py", line 197, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "c:\python39\lib\runpy.py", line 87, in _run_code
    exec(code, run_globals)
  File "C:\Python39\Scripts\ochrona.exe\__main__.py", line 7, in <module>
  File "c:\python39\lib\site-packages\click\core.py", line 829, in __call__
    return self.main(*args, **kwargs)
  File "c:\python39\lib\site-packages\click\core.py", line 782, in main
    rv = self.invoke(ctx)
  File "c:\python39\lib\site-packages\click\core.py", line 1066, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "c:\python39\lib\site-packages\click\core.py", line 610, in invoke
    return callback(*args, **kwargs)
  File "c:\python39\lib\site-packages\ochrona\cli.py", line 112, in run
    files = rfind_all_dependencies_files(
  File "c:\python39\lib\site-packages\ochrona\file_handler.py", line 107, in rfind_all_dependencies_files
    raise OchronaFileException("No dependencies files found")
ochrona.exceptions.OchronaFileException: No dependencies files found

unbalanced parenthesis error

I ran into this error when running Ochrona in a venv:

Traceback (most recent call last): File "/home/user/project/.venv/bin/ochrona", line 8, in <module> sys.exit(run()) File "/home/user/project/.venv/lib/python3.8/site-packages/click/core.py", line 1128, in __call__ return self.main(*args, **kwargs) File "/home/user/project/.venv/lib/python3.8/site-packages/click/core.py", line 1053, in main rv = self.invoke(ctx) File "/home/user/project/.venv/lib/python3.8/site-packages/click/core.py", line 1395, in invoke return ctx.invoke(self.callback, **ctx.params) File "/home/user/project/.venv/lib/python3.8/site-packages/click/core.py", line 754, in invoke return __callback(*args, **kwargs) File "/home/user/project/.venv/lib/python3.8/site-packages/ochrona/cli/cli.py", line 144, in run results.append(resolve(**payload)) File "/home/user/project/.venv/lib/python3.8/site-packages/ochrona/eval/eval.py", line 23, in resolve vulns += db.lookup_by_name(_safe_query_name(dep)) File "/home/user/project/.venv/lib/python3.8/site-packages/ochrona/db/db.py", line 95, in lookup_by_name if re.match(VULN_PATTERN.format(name), vuln.name): File "/usr/lib/python3.8/re.py", line 191, in match return _compile(pattern, flags).match(string) File "/usr/lib/python3.8/re.py", line 304, in _compile p = sre_compile.compile(pattern, flags) File "/usr/lib/python3.8/sre_compile.py", line 764, in compile p = sre_parse.parse(p, flags) File "/usr/lib/python3.8/sre_parse.py", line 962, in parse raise source.error("unbalanced parenthesis") re.error: unbalanced parenthesis at position 13

exclude_dir is ignored when specified as list in yaml config

Describe the bug
exclude_dir configuration property is ignored when specified as a list in the .ochrona.yml. It works as expected if specified as a comma separated string, e.g.:

exclude_dir: exclude_dir1,exclude_dir2

To Reproduce

  1. Create the following .ochrona.yml configuration:
---
dir: root_dir
exclude_dir:
  - exclude_dir1
  - exclude_dir2
debug: true
  1. Run ochrona

Expected behavior
exclude_dir1 and exclude_dir2 should be excluded from the recursive search.

Background:

  • OS: OSX 12.5
  • Python Version: python 3.8.7
  • Ochrona Version: 2.0.1
  • Ochrona Arguments: no arguments, using .ochrona.yml instead
  • Debug output: running with debug flag just lists all dependency files including those in exclude_dir

Additional context
I thought the problem was with the yaml parsing, but it looks like ochrona.config.OchronaConfig works as expected and correctly returns a list for exclude_dir. Also ignore configuration property works as list without any problems. So it is probably something else, but I didn't do a more detailed investigation. Thank you!

Ochrona fails to run due to Paramiko dependency

Describe the bug
Looks like ochrona fails to run when there is the paramiko dependency defined in the python project.

To Reproduce
Steps to reproduce the behavior:
Use paramiko dependency in the python project and run poetry run ochrona --report_type XML --output .

Expected behavior
Exception

Background (please complete the following information):

  • OS: macOS 11.6.1
  • Python Version: 3.7.12
  • Ochrona Version: 1.2.1
  • Ochrona Arguments: --report_type XML --output .
  • Debug output:
$ poetry run ochrona --report_type XML --debug --output .

,---.     |                        
|   |,---.|---.,---.,---.,---.,---.
|   ||    |   ||    |   ||   |,---|
`---'`---'`   '`    `---'`   '`---^
                            v. 1.2.1 
                            https://ochrona.dev   
                            
[DEBUG] Found matching poetry.lock file at /Users/ivanim/rbs/src/function-iam-uer-feed/poetry.lock
[DEBUG] Discovered dependencies: ['bcrypt==3.2.0', 'boto3==1.20.46', 'botocore==1.23.46', 'certifi==2021.10.8', 'cffi==1.15.0', 'charset-normalizer==2.0.11', 'cryptography==3.4.8', 'idna==3.3', 'jmespath==0.10.0', 'paramiko==2.9.2', 'pycparser==2.21', 'pynacl==1.5.0', 'python-dateutil==2.8.2', 'requests==2.27.1', 's3transfer==0.5.0', 'six==1.16.0', 'urllib3==1.25.10']
[DEBUG] DB instance found: 2022.01.25
[DEBUG] Found 0 vulnerabilities potentially affecting package: bcrypt
[DEBUG] Found 0 vulnerabilities potentially affecting package: boto3
[DEBUG] Found 0 vulnerabilities potentially affecting package: botocore
[DEBUG] Found 0 vulnerabilities potentially affecting package: certifi
[DEBUG] Found 0 vulnerabilities potentially affecting package: cffi
[DEBUG] Found 0 vulnerabilities potentially affecting package: charset-normalizer
[DEBUG] Found 4 vulnerabilities potentially affecting package: cryptography
[DEBUG] Found 0 vulnerabilities potentially affecting package: idna
[DEBUG] Found 0 vulnerabilities potentially affecting package: jmespath
[DEBUG] Found 3 vulnerabilities potentially affecting package: paramiko
[DEBUG] Found 0 vulnerabilities potentially affecting package: pycparser
[DEBUG] Found 0 vulnerabilities potentially affecting package: pynacl
[DEBUG] Found 0 vulnerabilities potentially affecting package: python-dateutil
[DEBUG] Found 5 vulnerabilities potentially affecting package: requests
[DEBUG] Found 0 vulnerabilities potentially affecting package: s3transfer
[DEBUG] Found 0 vulnerabilities potentially affecting package: six
[DEBUG] Found 8 vulnerabilities potentially affecting package: urllib3
Traceback (most recent call last):
  File "/Users/ivanim/Library/Caches/pypoetry/virtualenvs/function-iam-uer-feed-BDK6hZgi-py3.7/lib/python3.7/site-packages/ochrona/eval/vuln/vuln_evaluator.py", line 37, in evaluate
    for v in vulnerability["affected_versions"]
  File "/Users/ivanim/Library/Caches/pypoetry/virtualenvs/function-iam-uer-feed-BDK6hZgi-py3.7/lib/python3.7/site-packages/ochrona/eval/vuln/vuln_evaluator.py", line 38, in <listcomp>
    if v["operator"] == "="
KeyError: 'version_value'

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/Users/ivanim/Library/Caches/pypoetry/virtualenvs/function-iam-uer-feed-BDK6hZgi-py3.7/bin/ochrona", line 8, in <module>
    sys.exit(run())
  File "/Users/ivanim/Library/Caches/pypoetry/virtualenvs/function-iam-uer-feed-BDK6hZgi-py3.7/lib/python3.7/site-packages/click/core.py", line 1128, in __call__
    return self.main(*args, **kwargs)
  File "/Users/ivanim/Library/Caches/pypoetry/virtualenvs/function-iam-uer-feed-BDK6hZgi-py3.7/lib/python3.7/site-packages/click/core.py", line 1053, in main
    rv = self.invoke(ctx)
  File "/Users/ivanim/Library/Caches/pypoetry/virtualenvs/function-iam-uer-feed-BDK6hZgi-py3.7/lib/python3.7/site-packages/click/core.py", line 1395, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/Users/ivanim/Library/Caches/pypoetry/virtualenvs/function-iam-uer-feed-BDK6hZgi-py3.7/lib/python3.7/site-packages/click/core.py", line 754, in invoke
    return __callback(*args, **kwargs)
  File "/Users/ivanim/Library/Caches/pypoetry/virtualenvs/function-iam-uer-feed-BDK6hZgi-py3.7/lib/python3.7/site-packages/ochrona/cli/cli.py", line 144, in run
    results.append(resolve(**payload))
  File "/Users/ivanim/Library/Caches/pypoetry/virtualenvs/function-iam-uer-feed-BDK6hZgi-py3.7/lib/python3.7/site-packages/ochrona/eval/eval.py", line 26, in resolve
    resp.confirmed_vulnerabilities = evaluate(vulns, resp.flat_list)
  File "/Users/ivanim/Library/Caches/pypoetry/virtualenvs/function-iam-uer-feed-BDK6hZgi-py3.7/lib/python3.7/site-packages/ochrona/eval/vuln/vuln_evaluator.py", line 80, in evaluate
    raise Exception("evauluate exception") from ex
Exception: evauluate exception

Store cache files in temp location

Feature Overview

Change the cache files (db_cache, pypi_cache) location into a temporary location, instead of the current working dir.

Is your feature request related to a problem?

If ochrona is called inside the project folder, cache files are created in the project and pollutes it.

Many tests fail on Windows

Describe the bug

A lot a test fail on Windows, most of them due to the absence of path normalization (slash/anti-slash problem) and line breaks normalization.

Background (please complete the following information):

  • OS: Windows 10
  • Python Version: 3.10.8
  • Ochrona Version: 2.0.1

Html report creation fails on windows

Describe the bug

On Windows, the html report creation fails due to an enconding problem.

Traceback
Traceback (most recent call last):
  File "C:\Users\mathieu.bouzard\AppData\Local\Programs\Python\Python310\lib\runpy.py", line 196, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "C:\Users\mathieu.bouzard\AppData\Local\Programs\Python\Python310\lib\runpy.py", line 86, in _run_code
    exec(code, run_globals)
  File "C:\Users\mathieu.bouzard\Documents\pipx\bin\ochrona.exe\__main__.py", line 7, in <module>
    sys.exit(run())
  File "C:\Users\mathieu.bouzard\Documents\pipx\venvs\ochrona\lib\site-packages\click\core.py", line 1130, in __call__
    return self.main(*args, **kwargs)
  File "C:\Users\mathieu.bouzard\Documents\pipx\venvs\ochrona\lib\site-packages\click\core.py", line 1055, in main
    rv = self.invoke(ctx)
  File "C:\Users\mathieu.bouzard\Documents\pipx\venvs\ochrona\lib\site-packages\click\core.py", line 1404, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "C:\Users\mathieu.bouzard\Documents\pipx\venvs\ochrona\lib\site-packages\click\core.py", line 760, in invoke
    return __callback(*args, **kwargs)
  File "C:\Users\mathieu.bouzard\Documents\pipx\venvs\ochrona\lib\site-packages\ochrona\cli\cli.py", line 190, in run
    reporter.report_collector(files, results)
  File "C:\Users\mathieu.bouzard\Documents\pipx\venvs\ochrona\lib\site-packages\ochrona\reporter\reporter.py", line 54, in report_collector
    reports.append(self.generate_report(source, result, index, len(sources)))
  File "C:\Users\mathieu.bouzard\Documents\pipx\venvs\ochrona\lib\site-packages\ochrona\reporter\reporter.py", line 80, in generate_report
    report_dict[self._report_type](
  File "C:\Users\mathieu.bouzard\Documents\pipx\venvs\ochrona\lib\site-packages\ochrona\reporter\reports\html.py", line 41, in generate
    HTMLReport.save_report_to_file(
  File "C:\Users\mathieu.bouzard\Documents\pipx\venvs\ochrona\lib\site-packages\ochrona\reporter\reports\html.py", line 51, in save_report_to_file
    f.write(HTMLReport.generate_report_body(result, source, index, total))
  File "C:\Users\mathieu.bouzard\AppData\Local\Programs\Python\Python310\lib\encodings\cp1252.py", line 19, in encode
    return codecs.charmap_encode(input,self.errors,encoding_table)[0]
UnicodeEncodeError: 'charmap' codec can't encode character '\U0001f517' in position 5496: character maps to <undefined>

To Reproduce

On Windows:

ochrona --report_type html --output .

Background (please complete the following information):

  • OS: Windows 10
  • Python Version: 3.9.13
  • Ochrona Version: 2.0.1

Invalid specifier: '05.0.0'

Describe the bug
Exception is thrown and dependencies are not checked.

To Reproduce
Run ochrona with the provided poetry.lock file

Expected behavior
What did you expected to happen.

Background (please complete the following information):

  • OS: Lubuntu 22.04
  • Python Version: 3.10.4
  • Ochrona Version: 2.0.0
  • Ochrona Arguments: none
  • Debug output:
                                                                                                                                                                                                
,---.     |                                                                                     
|   |,---.|---.,---.,---.,---.,---.                                                                                                                                                             
|   ||    |   ||    |   ||   |,---|                                                             
`---'`---'`   '`    `---'`   '`---^                                                                                                                                                             
                            v. 2.0.0                                                                                                                                                            
                            https://ochrona.dev                                                                                                                                                 
                                                                                                                                                                                                
(DEBUG) Found matching poetry.lock file at /home/username/repos/dwh/poetry.lock                 
(DEBUG) Discovered dependencies:                                                                
 --- aiofiles==0.8.0                                                                                                                                                                            
 --- asyncpg==0.25.0                                                                            
 --- bcrypt==3.2.2                                                                              
 --- certifi==2022.6.15                                                                         
 --- cffi==1.15.1                                                                                                                                                                               
 --- charset-normalizer==2.1.0                                                                  
 --- click==8.1.3                                                                                                                                                                               
 --- colorama==0.4.5                                                                            
 --- cryptography==37.0.3                                                                                                                                                                       
 --- cx-oracle==8.3.0                                                                                                                                                                           
 --- fabric==2.7.0                                                                                                                                                                              
 --- greenlet==1.1.2                                                                            
 --- httptools==0.4.0                                                                                                                                                                           
 --- idna==3.3                                                                                  
 --- invoke==1.7.1                                                                              
 --- multidict==6.0.2
 --- orjson==3.7.6
 --- paramiko==2.11.0
 --- pathlib2==2.3.7.post1
 --- psycopg2==2.9.3
 --- pycparser==2.21
 --- pynacl==1.5.0
 --- python-json-logger==2.0.2
 --- requests==2.28.1
 --- sanic==22.6.0
 --- sanic-routing==22.3.0
 --- six==1.16.0
 --- sqlalchemy==1.4.39
 --- ujson==5.4.0
 --- urllib3==1.26.9
 --- uvloop==0.16.0
 --- websockets==10.3
 --- xlsxwriter==3.0.3
(DEBUG) DB instance found: 2022.06.30
(DEBUG) Found 0 vulnerabilities potentially affecting package: aiofiles
(DEBUG) Found 1 vulnerabilities potentially affecting package: asyncpg
(DEBUG) Found 0 vulnerabilities potentially affecting package: bcrypt
(DEBUG) Found 0 vulnerabilities potentially affecting package: certifi
(DEBUG) Found 0 vulnerabilities potentially affecting package: cffi
(DEBUG) Found 0 vulnerabilities potentially affecting package: charset-normalizer
(DEBUG) Found 0 vulnerabilities potentially affecting package: click
(DEBUG) Found 0 vulnerabilities potentially affecting package: colorama
(DEBUG) Found 4 vulnerabilities potentially affecting package: cryptography
(DEBUG) Found 0 vulnerabilities potentially affecting package: cx-oracle
(DEBUG) Found 0 vulnerabilities potentially affecting package: fabric
(DEBUG) Found 0 vulnerabilities potentially affecting package: greenlet
(DEBUG) Found 0 vulnerabilities potentially affecting package: httptools
(DEBUG) Found 0 vulnerabilities potentially affecting package: idna
(DEBUG) Found 0 vulnerabilities potentially affecting package: invoke
(DEBUG) Found 0 vulnerabilities potentially affecting package: multidict
(DEBUG) Found 0 vulnerabilities potentially affecting package: orjson
(DEBUG) Found 3 vulnerabilities potentially affecting package: paramiko
(DEBUG) Found 0 vulnerabilities potentially affecting package: pathlib2
(DEBUG) Found 0 vulnerabilities potentially affecting package: psycopg2
(DEBUG) Found 0 vulnerabilities potentially affecting package: pycparser
(DEBUG) Found 0 vulnerabilities potentially affecting package: pynacl
(DEBUG) Found 0 vulnerabilities potentially affecting package: python-json-logger
(DEBUG) Found 5 vulnerabilities potentially affecting package: requests
(DEBUG) Found 1 vulnerabilities potentially affecting package: sanic
(DEBUG) Found 0 vulnerabilities potentially affecting package: sanic-routing
(DEBUG) Found 0 vulnerabilities potentially affecting package: six
(DEBUG) Found 3 vulnerabilities potentially affecting package: sqlalchemy
(DEBUG) Found 1 vulnerabilities potentially affecting package: ujson
(DEBUG) Found 8 vulnerabilities potentially affecting package: urllib3
(DEBUG) Found 0 vulnerabilities potentially affecting package: uvloop
(DEBUG) Found 2 vulnerabilities potentially affecting package: websockets
(DEBUG) Found 0 vulnerabilities potentially affecting package: xlsxwriter
Processing 1 Files... ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━   0% -:--:--
Traceback (most recent call last):
  File "/home/username/.cache/pypoetry/virtualenvs/dwh-CmEpGaUJ-py3.10/lib/python3.10/site-packages/packaging/specifiers.py", line 634, in __init__
    parsed.add(Specifier(specifier))
  File "/home/username/.cache/pypoetry/virtualenvs/dwh-CmEpGaUJ-py3.10/lib/python3.10/site-packages/packaging/specifiers.py", line 98, in __init__
    raise InvalidSpecifier(f"Invalid specifier: '{spec}'")
packaging.specifiers.InvalidSpecifier: Invalid specifier: '05.0.0'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/username/.cache/pypoetry/virtualenvs/dwh-CmEpGaUJ-py3.10/lib/python3.10/site-packages/ochrona/eval/vuln/evaluate.py", line 66, in evaluate
    vuln_specifiers &= (
  File "/home/username/.cache/pypoetry/virtualenvs/dwh-CmEpGaUJ-py3.10/lib/python3.10/site-packages/packaging/specifiers.py", line 662, in __and__
    other = SpecifierSet(other)
  File "/home/username/.cache/pypoetry/virtualenvs/dwh-CmEpGaUJ-py3.10/lib/python3.10/site-packages/packaging/specifiers.py", line 636, in __init__
    parsed.add(LegacySpecifier(specifier))
  File "/home/username/.cache/pypoetry/virtualenvs/dwh-CmEpGaUJ-py3.10/lib/python3.10/site-packages/packaging/specifiers.py", line 253, in __init__
    super().__init__(spec, prereleases)
  File "/home/username/.cache/pypoetry/virtualenvs/dwh-CmEpGaUJ-py3.10/lib/python3.10/site-packages/packaging/specifiers.py", line 98, in __init__
    raise InvalidSpecifier(f"Invalid specifier: '{spec}'")
packaging.specifiers.InvalidSpecifier: Invalid specifier: '05.0.0'

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/home/username/.cache/pypoetry/virtualenvs/dwh-CmEpGaUJ-py3.10/bin/ochrona", line 8, in <module>                                                                              [69/56080]
    sys.exit(run())
  File "/home/username/.cache/pypoetry/virtualenvs/dwh-CmEpGaUJ-py3.10/lib/python3.10/site-packages/click/core.py", line 1130, in __call__
    return self.main(*args, **kwargs)
  File "/home/username/.cache/pypoetry/virtualenvs/dwh-CmEpGaUJ-py3.10/lib/python3.10/site-packages/click/core.py", line 1055, in main
    rv = self.invoke(ctx)
  File "/home/username/.cache/pypoetry/virtualenvs/dwh-CmEpGaUJ-py3.10/lib/python3.10/site-packages/click/core.py", line 1404, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/home/username/.cache/pypoetry/virtualenvs/dwh-CmEpGaUJ-py3.10/lib/python3.10/site-packages/click/core.py", line 760, in invoke
    return __callback(*args, **kwargs)
  File "/home/username/.cache/pypoetry/virtualenvs/dwh-CmEpGaUJ-py3.10/lib/python3.10/site-packages/ochrona/cli/cli.py", line 162, in run
    results.append(resolve(**payload))
  File "/home/username/.cache/pypoetry/virtualenvs/dwh-CmEpGaUJ-py3.10/lib/python3.10/site-packages/ochrona/eval/eval.py", line 30, in resolve
    resp.confirmed_vulnerabilities = evaluate(vulns, resp.flat_list)
  File "/home/username/.cache/pypoetry/virtualenvs/dwh-CmEpGaUJ-py3.10/lib/python3.10/site-packages/ochrona/eval/vuln/evaluate.py", line 104, in evaluate
    raise Exception("evaluate exception") from ex
Exception: evaluate exception

Additional context
poetry.lock

Pre-Commit and/or GitHub Action Integration

Feature Overview
What would you like to see changed/added?

It would be nice to be able to use Ochrona in a pre-commit routine (anytime requirements change then ochrona runs).

An alternative would be to have a GitHub action that has similar functionality.

Is your feature request related to a problem?
Having a way to automate the runs of Ochrona whenever the underlying requirements files change would be useful when maintaining a Python package.

Additional context
Documentation mentions CI/CD integration but doesn't provide any examples. It is possible this is straight-forward to accomplish, but not documented (or I couldn't find documentation).

Readme markdown format

Describe the bug
Minor issue of README.md, which has some little format bugs that cause markdown parsing error.

Section via .ochrona.yml at line 105, GitHub shows no error, but PyPi doc page cannot correctly render the table.

To Reproduce
Steps to reproduce the behavior:
PyPi doc page directly shows the markdown code of the table.

Expected behavior
What did you expected to happen.
Rendering the table.

Background (please complete the following information):

  • OS: [e.g. OSX 10.15]
  • Python Version: [e.g. python 3.8.1]
  • Ochrona Version: 1.1.0
  • Ochrona Arguments: [e.g. urllib3==1.26.2 --report_type FULL]
  • Debug output - if possible please reproduce the issue with the --debug flag

Additional context
Add any other context about the issue here.
ochrona-pypi

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.