Advantages over `--build_python_zip` about subpar HOT 13 OPEN

@duggelz Regarding (4), do you have a link to where that discussion is ongoing?

from subpar.

After spending some time reviewing subpar vs --build_python_zip, here's a summary of the main differences I've found:

1. The standard rules don't give you a way to specify whether or not you want a zip on a per-target basis (bazelbuild/bazel#3530). This shouldn't be hard to fix.

2. The standard rules' zip file is self-extracting, so it presents the same view of runfiles as any other standard ruleset. In contrast, subpar is only self-extracting when zip_safe is false, so by default you can't use the standard runfiles libraries with subpar and have to use pkg_resources instead.

3. The standard rules use a stub file, executed by #!/usr/bin/env python, which unpacks the runfiles, locates the second-stage interpreter, and uses it to execute the payload. Subpar bypasses the stub file and uses the same interpreter for both its own initialization and the payload. This means that subpar can't be used with in-build runtimes (#98). This might be fixed by adding a bootstrapping stage to subpar's initialization or unifying its init logic with the standard rules' stub script.

From my point of view, the most compelling path forward is to add zips as implicit outputs in the standard rules and deprecate subpar. I do wonder if there's anyone really depending on features in subpar they can't otherwise get.

from subpar.

Yes. We have the option in our repo bazelrc.

from subpar.

I'd be quite happy if Bazel adds official support for Python Zip Applications that renders subpar obsolete, but so far the .zip support is incomplete, only available to Windows by default, and makes some interesting design choices.

from subpar.

I'm using --build_python_zip in Linux and I didn't notice any issues yet. Can you please elaborate in which way the .zip support is incomplete?

from subpar.

I actually find it useful to be able to specify the .par file as a dependency in other rules, such as docker_build. IIUC, that's not possible with the --build_python_zip flag.

from subpar.

As per Bazel devs:

1. Feature request for per-target zip functionality on Linux is tracked here: bazelbuild/bazel#3530
2. No renaming of targets or extensions planned, Linux and Windows will continue to be slightly different
3. runfiles will not be added to Windows or removed from Linux, but work is underway to add helpers to smooth out differences.
4. The discussion of google/subpar vs --build_python_zip is ongoing.

from subpar.

To the point of @hwright, we have been using the zipped binaries as file dependencies in docker_build rules without any problems.

from subpar.

@siddharthab Do you have an example where you use rules_docker and the output of --build_python_zip?

@brandjon I agree, I feel like if py_binary had zipped output, then subpar would no longer need to be supported.

from subpar.

@groodt Not sure how detailed an example you want because it's a simple point. But something like,

py_binary(
    name = "bin",
    ...
)

container_image(
    name = "img",
    ...
    files = [":bin"],
)

works just fine for us.

Just the above is enough to package python binaries with their runfiles and our python toolchain (wrappers to ensure correct minor versions of python, etc.) into the docker image as a single zip file.

Repeated invocations of the self-extracting zip however is slow (e.g. calling the binary in a loop), and I wish that there was some documentation that gave instructions or guidance on how to extract once and then call the stub multiple times.

from subpar.

Thanks @siddharthab

So to make sure I understand, do you push the image with:
bazel run //example:img --build_python_zip?

from subpar.

Yes. We have the option in our repo bazelrc.

Interesting. Thanks! I'll have to try that. The only reason we're currently using subpar is because I couldn't get py_binary working reliably when added as files with rules_docker.

from subpar.

The problem I have with this argument is that your comparing subpar to zip when subpar is shell of what it was suppose to be. Subpar does not support the features that are needed. Anything not full python breaks it. (psutil, netaddress). Its unfortunate that it looks like python is going to get the shaft on this one which of course will make bazel fall out of favor for serious python.

from subpar.