As part of closing #1, we need to address di/pip-api#61, since real-world requirements.txts will likely contain environment markers.

Doing so should be pretty straightforward, since marker parsing is already present -- the dependencies just need to be correctly uniqued based on the satisfied markers.

When running pip-audit against a large environment or requirements file, querying the vulnerability service can take a decent amount of time. We should consider adding some sort of progress spinner or bar to the CLI so that the user knows that pip-audit hasn't hung or frozen.

Some considerations:

• We should only render the progress indicator if we know for certain that we're on a TTY
• The progress indicator's output must not interfere with the contents of stdout

As part of the handoff, we should deliver documentation that explains our core design decisions:

• Key architectural components (vuln service, dep collection, formatting interfaces) and how to use them (examples of implementing each)
• An explanation of our data model (each dependency has multiple potential vulnerabilities, etc.)

pip-audit -r <FILE> should parse the supplied requirements file instead of scanning the local environment.

In this comment thread, there's a discussion about whether we need to filter out requirements in dependency resolution due to their markers.

As of now, we think that it's not necessary due to this change to pip-api to evaluate markers during requirements parsing. But once we get requirements parsing done end to end, we should come back and confirm that this is the case.

The WIP patch (not working as expected) I had to do the filtering looks like this. If we need to filter, we can use this as a starting point and debug it.

modified   pip_audit/dependency_source/resolvelib/pypi_wheel_provider.py
@@ -123,6 +123,12 @@ class PyPIProvider(AbstractProvider):

         bad_versions = {c.version for c in incompatibilities[identifier]}

+        requirements = [
+            req
+            for req in requirements
+            if req.marker is None or req.marker.evaluate({"extra": req.extras})
+        ]
+
         # Accumulate extras
         extras: Set[str] = set()
         for r in requirements:
modified   test/dependency_source/test_resolvelib.py
@@ -73,6 +73,31 @@ def test_resolvelib_extras():
     assert expected_deps == set(resolved_deps[req])


+def test_resolvelib_conditional_extras():
+    resolver = resolvelib.ResolveLibResolver()
+
+    # First check the dependencies without extras and as a basis for comparison
+    reqs = [
+        Requirement("requests>=2.8.1 ; python_version >= '2.7'"),
+        Requirement("requests[socks]>=2.8.1 ; python_version < '2.7'"),
+    ]
+    resolved_deps = dict(resolver.resolve_all(reqs))
+    print(resolved_deps)
+    assert len(resolved_deps) == 1
+    expected_deps = set(
+        [
+            Dependency("requests", Version("2.26.0")),
+            Dependency("charset-normalizer", Version("2.0.6")),
+            Dependency("idna", Version("3.2")),
+            Dependency("certifi", Version("2021.5.30")),
+            Dependency("urllib3", Version("1.26.7")),
+        ]
+    )
+    assert reqs[0] in resolved_deps
+    assert reqs[1] not in resolved_deps
+    assert expected_deps == set(resolved_deps[reqs[0]])
+
+
 def test_resolvelib_patched(monkeypatch):
     # In the following unit tests, we'll be mocking certain function calls to test corner cases in
     # the resolver. Before doing that, use the mocks to exercise the happy path to ensure that

The current provider implementation only supports wheels, whereas we'll need to support source distributions. The relevant check is here.

@woodruffw had a useful comment here outlining what this will entail.

The current plan is to extend the release endpoint to provide vulnerability information for a given release. We should provide an example schema that has all the information that we need for pip-audit.

Currently, the CLI can block without any indicators for a decent period while collecting dependency information from requirements files.

We should figure out a way to make it responsive (or provide a working indicator) during that initial period.

In the medium-to-long-term, it would be great to make pip-audit a subcommand of pip, i.e. pip audit.

This will involve coordination with pip as the upstream, and requires us to figure a few things out, including but not limited to:

1. If we merge into pip, should we drop our pip-api dependency and use pips internal APIs directly?
2. Does it make sense to maintain a parallel "standalone" pip-audit that has functionality pip might not want (e.g. container scanning)?

This tool should be configured to cache HTTP requests based on ETags. Once #11 is resolved, that API includes ETags and this may speed up repeated audits.

As a project maintainer, I'd like to be able to use pip-audit to audit the sub-dependencies of my project (likely by somehow evaluating my local source tree prior to building a distribution artifact).

E.g., I maintain https://github.com/pypa/sampleproject, which depends on peppercorn. A CVE is released for some version of peppercorn, and I need to adjust my sub-dependency specification to avoid installing affected versions.

Currently it's not clear what the default value is from the help message:

  -r REQUIREMENTS, --requirement REQUIREMENTS
                        audit the given requirements file; this option can be used multiple times (default: [])

This should probably be something like ./requirements.txt.

For each of the available inputs, support adding a --fix flag which will automatically update the dependency specification or environment in question to exclude any found vulnerabilities.

For example:

• if a vulnerability is found in a local environment, uninstall it and install a version with the fix
• if a vulnerability is found in a requirements.txt file, update the version specification for the affected project to exclude the vulnerable version

This should be a meta-issue for determining the UX around this feature, and we should create sub-issues for each of the potential ways a fix can be applied.

• #212
• #215

In order to isolate concerns and ensure that pip-audit can be used both as a CLI and as an API, we should create something like a pip_audit.Auditor class.

That Auditor class should be instantiated the appropriate state:

• A list of dependencies to audit
• The vulnerability service to use
• Any additional configurable options (e.g. whether to do a dry run, ignore system dependencies, etc.)

Emitting a well-known SBOM format is a lower priority, but it's something we specified in the proposal.

Two good options are SPDX and CycloneDX; we should determine:

• Which one(s) have good, maintained Python APIs
• Which one(s) have community adoption

Now that we've evaluated osv.dev (#2), we can start designing an abstract interface that individual vulnerability services can satisfy.

From discussion on Slack: we probably want something like this:

class VulnerabilityService(ABC):
    def query(spec: Dependency) -> List[VulnerabilityResult]:
        ...

    def query_all(specs: List[Dependency]) -> Dict[Dependency, List[VulnerabilityResult]]:
        # naive implementation that can be overridden if
        # a particular service supports bulk queries
        for spec in specs:
            ...

Similarly to how we've developed interfaces for vulnerability services and reporting formats: we should have an interface that individual sources of dependencies should implement.

That, in turn, will make our Auditor (#18) implementation sufficiently generic and extensible.

Outside of requirements.txt, there are a few other common Python packaging files:

• A few different tools store requirements in pyproject.toml (#83)
• poetry puts locked (i.e., frozen) dependencies in poetry.lock (#84)
• pipenv uses Pipfile and Pipfile.lock (#85)

Each of these functionally boils down to a RequirementsSource, but with a bit of pre-processing to get them out of their dedicated formats.

There are (at least) two scenarios in which we'll want to report a vulnerability in a package, but won't be (naively) able to offer upgrade advice:

• The package is on its latest release, and as such cannot be upgraded any further
• The package can be upgraded to a newer release that fixes a vulnerability, but not without introducing another known vulnerability

In the first case, we should probably emit a special value indicating that we're incapable of offering upgrade advice.

The second case is tricker; we could:

• Not offer upgrade advice at all
• Offer upgrade advice if and only if the newer version is "less" vulnerable than the older
• Offer upgrade advice unconditionally, but warn the user that they're just trading one vulnerability for another

#22 will implement DependencySource for pip list; we need a corresponding implementation for parsing and concretizing requirement files.

The CI should test pip-audit against Python 3.10.

pip-api will be our source of ground truth regarding the Python packaging environment; we'll use it both to determine what's currently installed and to parse any explicitly specified requirements files.

We need to determine:

• Whether pip-api returns sufficient information for our use cases
• Whether it has any known bugs related to either the environment's packages or parsing requirement files
• Performance characteristics, compatibility with our expected development environment
  • From local testing, listing installed distributions is reasonably fast (especially since we should only need to do it at most once per pip-audit invocation).
  • pip-api supports Python 3.5 and newer, so we're fine on that front.

There are various URLs in the README that will need to be rewritten when this repository is transfered.

We currently display vulnerability descriptions in the pip-audit output. However, these descriptions can be quite long and make the output hard to read.

In the case of columns (output designed to be read by humans), we should hide descriptions unless explicitly requested.

These failures seem to occur intermittently on all unit tests that use resolvelib:

____________________ test_requirement_source_multiple_files ____________________

monkeypatch = <_pytest.monkeypatch.MonkeyPatch object at 0x7fccbeeebb80>

    def test_requirement_source_multiple_files(monkeypatch):
        file1 = "requirements1.txt"
        file2 = "requirements2.txt"
        file3 = "requirements3.txt"
    
        source = requirement.RequirementSource(
            [Path(file1), Path(file2), Path(file3)],
            ResolveLibResolver(),
        )
    
        def read_file_mock(f):
            filename = f.name
            if filename == file1:
                return ["flask==2.0.1"]
            elif filename == file2:
                return ["requests==2.8.1"]
            else:
                assert filename == file3
                return ["pip-api==0.0.22\n", "packaging==21.0"]
    
        monkeypatch.setattr(_parse_requirements, "_read_file", read_file_mock)
    
>       specs = list(source.collect())

test/dependency_source/test_requirement.py:50: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
pip_audit/dependency_source/requirement.py:33: in collect
    for _, deps in self.resolver.resolve_all(iter(req_values)):
pip_audit/dependency_source/interface.py:55: in resolve_all
    yield (req, self.resolve(req))
pip_audit/dependency_source/resolvelib/resolvelib.py:27: in resolve
    result = self.resolver.resolve([req])
env/lib/python3.8/site-packages/resolvelib/resolvers.py:482: in resolve
    state = resolution.resolve(requirements, max_rounds=max_rounds)
env/lib/python3.8/site-packages/resolvelib/resolvers.py:373: in resolve
    name = min(unsatisfied_names, key=self._get_preference)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

self = <resolvelib.resolvers.Resolution object at 0x7fccbee6bf40>
name = 'flask'

    def _get_preference(self, name):
>       return self._p.get_preference(
            identifier=name,
            resolutions=self.state.mapping,
            candidates=IteratorMapping(
                self.state.criteria,
                operator.attrgetter("candidates"),
            ),
            information=IteratorMapping(
                self.state.criteria,
                operator.attrgetter("information"),
            ),
            backtrack_causes=self.state.backtrack_causes,
        )
E       TypeError: get_preference() got an unexpected keyword argument 'backtrack_causes'

env/lib/python3.8/site-packages/resolvelib/resolvers.py:178: TypeError

In #48, I introduced a dependency to virtualenv-api to install source distributions in a virtual environment and figure out what their dependencies are.

We should remove this dependency and instead rely on builtin modules. The builtin venv module has less functionality (it's more about creating virtual environments than modifying or introspecting them) so we'll have to do some legwork to replace it. Specifically:

• Creating the virtual environment with venv.
• Shell out to Python and do something like:

source env/bin/activate
pip install -e sdist_dir/
pip install pip-api
python -c "import pip_api; print(pip_api.installed_packages)"

• Parse the shell output into package name and version pairs.
• Filter out pip_api dependencies since those will turn up in the output also.

In "dry-run" mode, pip-audit should do everything it would normally do up to hitting the vulnerability service.

Blocked on #3.

As a parallel or perhaps complementary track to #29: resolvelib provides an abstract interface for resolving dependencies that are produced by a client-supplied "provider" (presumably something like us querying the PyPI APIs).

This might be too generic of a library for our use case, but it might be useful if we go further into reworking pip-tools to supply a Python API.

PyPI Link: https://pypi.org/project/resolvelib/

For the time being, our MVP is scoped to just support for PyPI. However, it's worth considering what we'd require in order to support custom package indices (whether PyPI mirrors or entirely separate private indexes).

Some potential points of issue:

• Our #23 adaptor will need to be sufficiently generic, adapting either PyPI or another index under the hood as configured.
• We might need custom index support from pip-api? This is unclear, since our only use of pip-api in these contexts would be for requirements parsing, and the requirements file itself shouldn't make any references to the index.

Title says it all: this will improve our confidence in our use of pip-api's APIs and will allow us to enable more MyPy features in our own code.

Per @di: The final deliverable version of pip-audit will not use osv.dev, but instead should use a (hitherto unimplemented) REST API provided by PyPI.

Since we'll need to consume that API, we should evaluate osv.dev and determine what we'd like to be different.

Related to #72: the PyPI vulnerability API, which is part of Warehouse's codebase, is not currently schematized.

Schematizing it would increase our confidence in pip-audit's reliability, and would us to write more type/construction friendly interactions with the API, e.g. with Pydantic.

Right now, our dependency collection process (via resolvelib looks like this):

1. Start at top, iteratively walk the dependency list
2. If the dependency has a wheel, grab its dependencies from the wheel metadata for the next iteration
3. If the dependency only has an sdist, install that sdist in a temporary venv to compute its dependencies for the next iteration

(3) is slow, since at each iterative step we might have N source distributions that each get their own virtual environment (and virtual environment setup is pretty slow). We could probably speed it up a decent amount by using a thread pool (with a reasonable cap) to create multiple environments and collect them as they complete.

Downsides: thread pools mean more complex error and interrupt (i.e. ^C) handling. We should figure out the tradeoffs for this before committing to this direction.

Related to #56, since doing this will (hopefully) make the CLI more responsive.

Hi!

I noticed that you're using the old OSV format fields in https://github.com/trailofbits/pip-audit/blob/b3fd732bf5f70311dcd9a32e15e26889597e3060/pip_audit/service/osv.py#L42.

This field is deprecated and will be removed soon, along with the top level "package" field. The new field which replaces this is "affected". Could you please use that instead? The 1.0 spec is described at https://ossf.github.io/osv-schema/.

We're probably going to need pip-tools to fully resolve any requirements files that we're passed, since we'll need to both resolve the entire dependency tree and concretize each dependency down to a specific version.

Link: https://pypi.org/project/pip-tools/

Link: https://github.com/jazzband/pip-tools/

At the moment, the provider that got from the resolvelib example, doesn't support extras in the package requirement. We should make this work.

Once we add support, we should ensure that this unit test works end-to-end.

pip-api does a valiant job of wrapping pip for us, but certain functionality fundamentally degrades beyond a few versions:

• Earlier than 10.0.0b0: pip list -v --format=json doesn't include location information, which hampers our ability to filter "system" dependencies (#7)

We should still try our hardest with these older pips, but also emit a warning similar to the one emitted by pip itself when it's out of date.

Our test suite currently includes "online" tests, i.e. tests that require network connectivity and access to services for their functionality.

We should mark these tests as "online," so that a user can disable them for local-only testing.

Pytest supports CLI extensions for this purpose, like in this SO post.

OSV and PyPI both provide JSON APIs, both of which are (probably?) schematized.

We should embed their schemas and generate models (maybe pydantic ones) from them, to give ourselves more confidence about the shape of the responses we expect and to better conform to correctness by construction.

Once pip-audit is handed off, the PYPI_TOKEN should be updated to someone who isn't me or ToB.

pip_api.installed_distributions() returns every visible distribution, which can potentially include distributions provided by the system/system package manager.

Issuing messages for these might not be desirable default behavior, for a few reasons:

• System-installed dependencies might be required by the system, and thus cannot be safely upgraded
• System-installed dependencies might be installed via a mechanism other than pip, so issued guidance might not always be applicable.
• System-installed dependencies might be patched by distribution maintainers to remove known vulnerabilities, without updating the version number.

As such, it probably makes sense for the CLI to have option(s) that allow the user to enable (or disable) filtering of dependencies that look like they're supplied by the system. This, in turn, requires us to come up with a reliable way of determining whether a given dependency is a "system" one.

Currently it's OSV:

  -s {osv,pypi}, --vulnerability-service {osv,pypi}
                        the vulnerability service to audit dependencies against (default: osv)

Following #9 and #10: once it's available, we should support PyPI's vulnerability service via our adaptor.

The syft tool supports generating a SBOM for a container image and has support for Python packages. We should check to see if we can leverage this to support container images in pip-audit.

cc: @di

Apart from supporting a standard SBOM format (#3), pip-audit should have at least two output formats for the MVP:

• A human-readable format for display on a terminal or logs
• A JSON format for interpretation by other programs

In the long term, we'd like to support providing container images a dependency source.

We should start by evaluating https://github.com/anchore/syft to see if it helps us since syft does supposedly report a list of Python packages from a container image.

As part of/following #9: we should support osv.dev via our adaptor.

The README.md should include the output of pip-audit --help as well as provide some of the most common examples of how to use pip-audit.

The development guide should be moved to CONTRIBUTING.md.