Changing the lockfile format to reduce duplication about pnpm HOT 12 CLOSED

Regarding append only and dividing the lockfile into two files, I am also not sure about these ideas. I was just thinking about some additional benefits that this change could provide. I don't think we'll enable append only and splitting the lockfile by default. We could add that feature as an experimental feature that should be enabled.

EDIT:

If we would move the "resolutions" to a separate file, we could in theory store it somewhere else, outside of Git. But that's a whole different discussion. As of now, in v9 I just suggest to split the packages section to two sections. No append-only and no two files (at least by default).

from pnpm.

@pnpm/collaborators how do you feel about this?

from pnpm.

I like it. I think, importantly, it will make it clearer what versions have changed when reviewing a PR by removing a lot of the noise.

Lockfile size is not an issue for us at least (it already has 56k lines).

Another downside is that third parties will have to do a bit more work too—not just pnpm. Things like the Rushstack pnpm lockfile explorer will have to do some lookups (not a big deal if you ask me).

from pnpm.

My only thoughts are that I'm not a fan of the "append only" idea, as I really like the current property that I can delete the lockfile and recreate it and it is (in my experience) the same as it was before removal (which gives me confidence in pnpm's behavior), and that I wouldn't want to split the file given the churn of having to re-exclude this file from formatters or similar.

But I'm not totally sure how much splitting things apart will help; presumably both "packages" and "resolutions" will be of similar lengths, and quite long?

from pnpm.

Oh agreed with Jake—not a fan of the idea of splitting. Personally haven't seen merge conflicts be a big issue.

from pnpm.

I also like it! Definitely a cool idea that normalizes data a bit better.

from pnpm.

There's one issue that prevents us from this change. The requiresBuild field. We currently set it to true for any optional dependency because we don't know if the package needs to be built without extracting it. But we don't download optional dependencies.

I guess the only way would be to remove requiresBuild from the lockfile.

Maybe we could just move the list of packages that require build to node_modules/.modules.yaml.

Created an issue: #7707

from pnpm.

The change is ready for review: #7700

from pnpm.

Hi! As you’re considering this, I’m wondering if you’ve given thought to feedback like this:

the developers are using wrong version of pnpm and most of the developers are outside contributors of our OSS project(Rspack), so we can’t just tell them use specific version of pnpm and we need a force mechanism, Currently we use engines.pnpm and packageManager together, we use packageManager to auto switch to right pnpm version and use engines.pnpm to do double check, so actually we have different choices here:

• engines.pnpm alone: this is our first choice, then people ask why need to globally switch to a specific pnpm version when develop Rspack, I personally also dislike use engines.pnpm alone because I’m also member of Infra team of Bytedance, and deals with tons of oncall every day, and every project has different engines.pnpm version, so I need to globally switch pnpm version for every project, which really sucks.
  …

Does pnpm not use a consistent lockfile format between minor and patch releases on the 8.x line? Like if one team member is running 8.15.4 and another is running 8.15.3, and there’s no engines or packageManager involved, would the lockfile be different or inconsistent if one user did a lockfile-effecting operation and then the other user did?

Or put another way, I would think that if there was some way to define in a project “every developer of this project should be using pnpm 8.x”, that should be enough to avoid lockfile issues, correct? (As opposed to “every developer of this project must be on exactly pnpm 8.15.4.”) Or if not, could it be so in pnpm 9?

Related: openjs-foundation/package-metadata-interoperability-collab-space#15

from pnpm.

I am not sure how this is related to the current issue. The change in this issue will be a breaking change in pnpm v9.

Changes to the lockfile format are breaking changes in most cases. We do have the possibility to make a minor bump to the lockfile version and we used it in the past. This can happen if we add some new fields to the lockfile that are just ignored by older versions of pnpm. In this case, you can keep an older version of pnpm on CI and it will still be able to correctly install everything.

from pnpm.

I am not sure how this is related to the current issue.

It might not be, my apologies. I can open a new issue if you’d like. Basically, I’m trying to resolve the debate in that thread on the Node side, which stems from this user needing to lock pnpm to exact versions. I’m wondering why pnpm users feel the need to lock to exact versions, as opposed to major versions, in order to avoid lockfile issues. I would assume that pnpm tries to keep lockfile versions consistent within major versions, in which case users shouldn’t be having this issue; yet they are.

Does pnpm have a position on whether it wants users locking exact versions of pnpm for projects?

from pnpm.

I have merged the change. It will be released in an alpha version, so if someone will have additional comments or objections, there's still time to discuss and revise the change before stable v9 is released.

from pnpm.