Is there any way to not rely on pinning a specific parsl version? about funcx HOT 4 CLOSED

Quite a bit of our code relies on the Parsl API, but Parsl does not provide a way to recognize backward-compatible versions. If we do not pin the version, any new release could potentially break things. However, in most cases, you should not have an issue installing the latest version because the aspects of the Parsl API that we utilize are generally stable.

from funcx.

@rjmello Isn't that what the CI and things like dependabot are for? The CI would rely on a pinned version, as is standard practice. Why also force the restriction on the end user?

To me, it seems wise to adopt the general best practice of having a requirements.txt file with specific versions used in CI and dependabot to continuously evaluate the stability of new Parsl releases. The setup.py should only have limits on versions when the CI indicates a break in compatability. We shouldn't rely on semantic versioning anyway; there is no strict guarantee that breaking changes don't happen in a given release.

from funcx.

Andrew, the purpose of pinning is to ensure that when an end user does pip install globus-compute-endpoint they, by default, end up with a functioning installation. CI and dependabot do not ensure compatibility of future versions, instead, at best, they can warn when a new version of parsl breaks something. Parsl has adopted date versioning and makes no explicit guarantees about backwards compatibility. In fact, the official recommendation is for all projects that use parsl to pin the version they expect. That is a well reasoned stance from the Parsl team given the resources they have to put toward maintenance and backward compatibility support, but does mean that to ensure Globus Compute users have a smooth experience, pinning the Parsl version is the best option.

That said, you can override the version installed by either providing your own constraints file to pip, or installing parsl separately after installing globus compute. Also, as standard practice with new releases of the Globus Compute endpoint, we do bump the pinned Parsl version if we can do that safely.

from funcx.

Thanks for the reply, @joshbryan-globus.

I understand now, although I disagree that this recommendation from Parsl team is the way to go. It seems like using a <= to bound the current Parsl version at the time of release would be more generous than a strict pin, also (optionally) using a >= for the lower bound as appropriate. However, I can understand that perhaps there are hesitations about updating Globus Compute and not realizing that a change is incompatible with an older Parsl version.

Regarding overriding, this is true for users but not for developers. If I develop and ship a package that relies on Parsl and Globus Compute as dependencies, there is no mechanism for me to have my pyproject.toml use any Parsl version other than the one dictated by Globus Compute. There would be a conflict otherwise.

In general, the Parsl recommendation of pinning versions seems quite limiting for developers. Perhaps that is something for me to discuss with the team as a whole.

from funcx.