For pip, wheel and setuptools.

Look at install_wheel, along with WheelFilename and compatible_tags from Monotrail.

Right now, we're dropping the derivation tree.

This might be difficult since we're just using an HTTP cache, but e.g., we don't need the full descriptions the metadata responses.

Can we make it as efficient as the naive solver? (In other words, using PubGrub should be zero cost.)

Right now, we use the normalized name everywhere. But if the user or package author writes Jinja2, we should show that instead.

Replace the ad hoc parser I wrote.

I actually don't know how to do this, but the resolver needs to fail if those requests fail.

Run tests, formatting, linting.

This is especially important for enabling backwards-compatible changes to the cached representations in the future.

We could either do something npm- and yarn-like (show a progress bar), or something pip-like (show output for every package, for everything we do).

Like pip-compile, the resultant lockfile should contain comments telling you why each dependency was included.

To create a virtual environment via gourgeist.

Orogene, Bun, pnpm, etc. all have per-platform optimizations for file-copying.

https://twitter.com/zkat__/status/1701289288296702464

Store the (zipped, and later unzipped?) wheels in a content-address store via cache using the known SHA.

Could we provide our own endpoint that's closer to the Crates sparse index? That is: a single endpoint for each package, which includes all the dependencies for all versions, thereby limiting the number of fetches to the depth of the tree. (Right now, the resolver needs to perform a fetch for every package-version pair.)

Can we develop benchmarks that don't depend on hitting PyPI?

Right now, these steps are all parallelized to some degree, but the phases themselves are sequential. So, e.g., we don’t start unzipping (the slowest step) until we’ve downloaded everything.

We could instead use channels to ensure that we can schedule all work efficiently across the phases.

This would be a little more complex and makes debugging and logging harder (or more confusing). And it also may not have a significant impact, since the largest wheels are gonna be the bottleneck in each step anyway. (Actually, I think the bottleneck is largest file in any given wheel, like Ruff, which just has one huge executable.) But it’s worth a try.

Turns out this is really expensive (at least on a relative basis), since it's bounded by Python startup time.

I like Bun's output, as an example:

They show a loading spinner while installing, then show a summary of the resolved versions for the first-party packages at the end.

From the Bun documentation:

Bun strives to avoid re-downloading packages multiple times. When installing a package, if the cache already contains a version in the range specified by package.json, Bun will use the cached package instead of downloading it again.

If the semver version has pre-release suffix (1.0.0-beta.0) or a build suffix (1.0.0+20220101), it is replaced with a hash of that value instead, to reduce the chances of errors associated with long file paths.

When the node_modules folder exists, before installing, Bun checks that node_modules contains all expected packages with appropriate versions. If so bun install completes. Bun uses a custom JSON parser which stops parsing as soon as it finds "name" and "version".

If a package is missing or has a version incompatible with the package.json, Bun checks for a compatible module in the cache. If found, it is installed into node_modules. Otherwise, the package will be downloaded from the registry then installed.

We're sorely lacking test coverage here, but at least some end-to-end integration tests would be useful.