google / subpar Goto Github PK

View Code? Open in Web Editor NEW

566.0 17.0 77.0 175 KB

Subpar is a utility for creating self-contained python executables. It is designed to work well with Bazel.

License: Apache License 2.0

Python 71.21% Shell 7.37% Dockerfile 0.75% Starlark 20.67%

bazel python

subpar's Introduction

Subpar (deprecated)

Subpar is a utility for creating self-contained python executables. It is designed to work well with Bazel.

Status

This project is unmaintained and considered deprecated. Historically, subpar was the only way to produce a deployable Python artifact in Bazel. This is no longer true; --build_python_zip and the python_zip_file output_group allows you to create executable Python zip artifacts with the standard py_binary rule. rules_docker can also be used to build container images that launch py_binary.

Setup

Add the following to your WORKSPACE file:

git_repository(
    name = "subpar",
    remote = "https://github.com/google/subpar",
    tag = "1.0.0",
)

Add the following to the top of any BUILD files that declare par_binary() rules:

load("@subpar//:subpar.bzl", "par_binary")

Usage

par_binary() is a drop-in replacement for py_binary() in your BUILD files that also builds a self-contained, single-file executable for the application, with a .par file extension.

To build the .par file associated with a par_binary(name=myname) rule, do

bazel build //my/package:myname.par

The .par file is created alongside the python stub and .runfiles directories that py_binary() creates, but is independent of them. It can be copied to other directories or machines, and executed directly without needing the .runfiles directory. The body of the .par file contains all the srcs, deps, and data files listed.

Limitations:

C extension modules in 'deps' is not yet supported
Automatic re-extraction of '.runfiles' is not yet supported
Does not include a copy of the Python interpreter ('hermetic .par')

Example

Given a BUILD file with the following:

load("@subpar//:subpar.bzl", "par_binary")

par_binary(
    name = 'foo',
    srcs = ['foo.py', 'bar.py'],
    deps = ['//baz:some_py_lib'],
    data = ['quux.dat'],
)

Run the following build command:

bazel build //package:foo.par

This results in the following files being created by bazel build:

bazel-bin/
    package/
        foo
        foo.par
        foo.runfiles/
            ...

The .par file can be copied, moved, or renamed, and still run like a compiled executable file:

$ scp bazel-bin/package/foo.par my-other-machine:foo.par
$ ssh my-other-machine ./foo.par

System Requirements

Python Versions: CPython versions 2.7.6+
Operating Systems: Debian-derived Linux, including Ubuntu and Goobuntu.

DISCLAIMER

This is not an official Google product, it is just code that happens to be owned by Google.

subpar's People

Contributors

Stargazers

Watchers

subpar's Issues

PAR picks up system-installed version instead of bundled version

In my PR to add PIP support to Bazel you can reproduce this as shown below.

On a system with:

$ pip --version
pip 1.5.4 from /usr/lib/python2.7/dist-packages (python 2.7)

(I must have gotten this through apt-get or some other super-slow release channel)

If I bazel run the py_binary form, no warning:

rules_python$ bazel run rules_python:piptool --  --name foo --input $PWD/examples/helloworld/requirements.txt --directory /tmp/pip/ --output /tmp/requirements.bzl
INFO: Analysed target //rules_python:piptool (0 packages loaded).
INFO: Found 1 target...
Target //rules_python:piptool up-to-date:
  bazel-bin/rules_python/piptool
INFO: Elapsed time: 0.184s, Critical Path: 0.00s
INFO: Build completed successfully, 1 total action

INFO: Running command line: bazel-bin/rules_python/piptool --name foo --input /home/mattmoor/rules_python/examples/helloworld/requirements.txt --directory /tmp/pip/ --output /tmp/requirements.bzl
Collecting futures>=3.1 (from -r /home/mattmoor/rules_python/examples/helloworld/requirements.txt (line 1))
  File was already downloaded /tmp/pip/futures-3.1.1-py2-none-any.whl
Skipping futures, due to already being wheel.

If I bazel run the .par target, I get a warning:

rules_python$ bazel run rules_python:piptool.par --  --name foo --input $PWD/examples/helloworld/requirements.txt --
directory /tmp/pip/ --output /tmp/requirements.bzl
INFO: Analysed target //rules_python:piptool.par (0 packages loaded).
INFO: Found 1 target...
Target //rules_python:piptool.par up-to-date:
  bazel-bin/rules_python/piptool.par
INFO: Elapsed time: 0.204s, Critical Path: 0.01s
INFO: Build completed successfully, 1 total action

INFO: Running command line: bazel-bin/rules_python/piptool.par --name foo --input /home/mattmoor/rules_python/examples/helloworld/requirements.txt --directory /tmp/pip/ --output /tmp/requirements.bzl
Collecting futures>=3.1 (from -r /home/mattmoor/rules_python/examples/helloworld/requirements.txt (line 1))
  File was already downloaded /tmp/pip/futures-3.1.1-py2-none-any.whl
Skipping futures, due to already being wheel.
You are using pip version 1.5.4, however version 9.0.1 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.

Happy to provide more info as needed.

par_binary doesn't respect visibility attribute

If I have a par_binary with visibility = ["//visibility:public"] Bazel fails with an error about the visibility of the :foo.par target.

If I set the containing package's default_visibility, this goes away.

par file sources have incorrect path when included in workspace

With the par_binary from this repo which is defined as:

package(default_visibility = ["//visibility:public"])

load("@subpar//:subpar.bzl", "par_binary")

py_library(
    name = 'shared',
    srcs = glob(['shared/*.py']),
)

py_library(
    name = 'simulator',
    srcs = glob(['simulator_control/*.py']),
    deps = [
        ':shared',
    ],
)

par_binary(
    name = 'ios_test_runner',
    srcs = glob(
        ['test_runner/*.py'],
        exclude = ['test_runner/TestProject/**']
    ),
    main = 'test_runner/ios_test_runner.py',
    deps = [
        ':shared',
        ':simulator',
    ],
    data = glob(['test_runner/TestProject/**']),
)

When building normally in that repo with bazel build xctestrunner:ios_test_runner.par, you end up with a valid par file with these SOURCES being used:

__main__/xctestrunner/__init__.py 
__main__/xctestrunner/ios_test_runner bazel-out/darwin-fastbuild/bin/xctestrunner/ios_test_runner
__main__/xctestrunner/shared/__init__.py xctestrunner/shared/__init__.py
__main__/xctestrunner/shared/bundle_util.py xctestrunner/shared/bundle_util.py
__main__/xctestrunner/shared/ios_constants.py xctestrunner/shared/ios_constants.py
__main__/xctestrunner/shared/ios_errors.py xctestrunner/shared/ios_errors.py
__main__/xctestrunner/shared/plist_util.py xctestrunner/shared/plist_util.py
__main__/xctestrunner/shared/provisioning_profile.py xctestrunner/shared/provisioning_profile.py
__main__/xctestrunner/shared/xcode_info_util.py xctestrunner/shared/xcode_info_util.py
__main__/xctestrunner/simulator_control/__init__.py xctestrunner/simulator_control/__init__.py
__main__/xctestrunner/simulator_control/simtype_profile.py xctestrunner/simulator_control/simtype_profile.py
__main__/xctestrunner/simulator_control/simulator_util.py xctestrunner/simulator_control/simulator_util.py
__main__/xctestrunner/test_runner/TestProject/TestProject.xcodeproj/project.pbxproj xctestrunner/test_runner/TestProject/TestProject.xcodeproj/project.pbxproj
__main__/xctestrunner/test_runner/TestProject/TestProject.xcodeproj/xcshareddata/xcschemes/TestProjectXctest.xcscheme xctestrunner/test_runner/TestProject/TestProject.xcodeproj/xcshareddata/xcschemes/TestProjectXctest.xcscheme
__main__/xctestrunner/test_runner/TestProject/TestProject.xcodeproj/xcshareddata/xcschemes/TestProjectXcuitest.xcscheme xctestrunner/test_runner/TestProject/TestProject.xcodeproj/xcshareddata/xcschemes/TestProjectXcuitest.xcscheme
__main__/xctestrunner/test_runner/__init__.py xctestrunner/test_runner/__init__.py
__main__/xctestrunner/test_runner/dummy_project.py xctestrunner/test_runner/dummy_project.py
__main__/xctestrunner/test_runner/ios_test_runner.py xctestrunner/test_runner/ios_test_runner.py
__main__/xctestrunner/test_runner/logic_test_util.py xctestrunner/test_runner/logic_test_util.py
__main__/xctestrunner/test_runner/runner_exit_codes.py xctestrunner/test_runner/runner_exit_codes.py
__main__/xctestrunner/test_runner/test_summaries_util.py xctestrunner/test_runner/test_summaries_util.py
__main__/xctestrunner/test_runner/xcodebuild_test_executor.py xctestrunner/test_runner/xcodebuild_test_executor.py
__main__/xctestrunner/test_runner/xctest_session.py xctestrunner/test_runner/xctest_session.py
__main__/xctestrunner/test_runner/xctestrun.py xctestrunner/test_runner/xctestrun.py

When including this repository in your WORKSPACE (instead of using the par files they vendor on GitHub releases) like this:

git_repository(
    name = "subpar",
    remote = "https://github.com/google/subpar",
    tag = "1.3.0",
)

git_repository(
    name = "xctestrunner",
    remote = "https://github.com/google/xctestrunner.git",
    tag = "0.2.5",
)

The SOURCES used for building with bazel build @xctestrunner//xctestrunner:ios_test_runner.par are:

__init__.py 
xctestrunner/__init__.py 
xctestrunner/xctestrunner/__init__.py 
xctestrunner/xctestrunner/ios_test_runner bazel-out/darwin-fastbuild/bin/external/xctestrunner/xctestrunner/ios_test_runner
xctestrunner/xctestrunner/shared/__init__.py external/xctestrunner/xctestrunner/shared/__init__.py
xctestrunner/xctestrunner/shared/bundle_util.py external/xctestrunner/xctestrunner/shared/bundle_util.py
xctestrunner/xctestrunner/shared/ios_constants.py external/xctestrunner/xctestrunner/shared/ios_constants.py
xctestrunner/xctestrunner/shared/ios_errors.py external/xctestrunner/xctestrunner/shared/ios_errors.py
xctestrunner/xctestrunner/shared/plist_util.py external/xctestrunner/xctestrunner/shared/plist_util.py
xctestrunner/xctestrunner/shared/provisioning_profile.py external/xctestrunner/xctestrunner/shared/provisioning_profile.py
xctestrunner/xctestrunner/shared/xcode_info_util.py external/xctestrunner/xctestrunner/shared/xcode_info_util.py
xctestrunner/xctestrunner/simulator_control/__init__.py external/xctestrunner/xctestrunner/simulator_control/__init__.py
xctestrunner/xctestrunner/simulator_control/simtype_profile.py external/xctestrunner/xctestrunner/simulator_control/simtype_profile.py
xctestrunner/xctestrunner/simulator_control/simulator_util.py external/xctestrunner/xctestrunner/simulator_control/simulator_util.py
xctestrunner/xctestrunner/test_runner/TestProject/TestProject.xcodeproj/project.pbxproj external/xctestrunner/xctestrunner/test_runner/TestProject/TestProject.xcodeproj/project.pbxproj
xctestrunner/xctestrunner/test_runner/TestProject/TestProject.xcodeproj/xcshareddata/xcschemes/TestProjectXctest.xcscheme external/xctestrunner/xctestrunner/test_runner/TestProject/TestProject.xcodeproj/xcshareddata/xcschemes/TestProjectXctest.xcscheme
xctestrunner/xctestrunner/test_runner/TestProject/TestProject.xcodeproj/xcshareddata/xcschemes/TestProjectXcuitest.xcscheme external/xctestrunner/xctestrunner/test_runner/TestProject/TestProject.xcodeproj/xcshareddata/xcschemes/TestProjectXcuitest.xcscheme
xctestrunner/xctestrunner/test_runner/__init__.py external/xctestrunner/xctestrunner/test_runner/__init__.py
xctestrunner/xctestrunner/test_runner/dummy_project.py external/xctestrunner/xctestrunner/test_runner/dummy_project.py
xctestrunner/xctestrunner/test_runner/ios_test_runner.py external/xctestrunner/xctestrunner/test_runner/ios_test_runner.py
xctestrunner/xctestrunner/test_runner/logic_test_util.py external/xctestrunner/xctestrunner/test_runner/logic_test_util.py
xctestrunner/xctestrunner/test_runner/runner_exit_codes.py external/xctestrunner/xctestrunner/test_runner/runner_exit_codes.py
xctestrunner/xctestrunner/test_runner/test_summaries_util.py external/xctestrunner/xctestrunner/test_runner/test_summaries_util.py
xctestrunner/xctestrunner/test_runner/xcodebuild_test_executor.py external/xctestrunner/xctestrunner/test_runner/xcodebuild_test_executor.py
xctestrunner/xctestrunner/test_runner/xctest_session.py external/xctestrunner/xctestrunner/test_runner/xctest_session.py
xctestrunner/xctestrunner/test_runner/xctestrun.py external/xctestrunner/xctestrunner/test_runner/xctestrun.py

As you can see there's an extra xctestrunner prefix. This leads to a runtime crash because the imports are no longer valid:

% bazel-bin/external/xctestrunner/xctestrunner/ios_test_runner.par
Traceback (most recent call last):
  File "/Users/ksmiley/.pyenv/versions/2.7.15/lib/python2.7/runpy.py", line 174, in _run_module_as_main
    "__main__", fname, loader, pkg_name)
  File "/Users/ksmiley/.pyenv/versions/2.7.15/lib/python2.7/runpy.py", line 72, in _run_code
    exec code in run_globals
  File "bazel-bin/external/xctestrunner/xctestrunner/ios_test_runner.par/__main__.py", line 32, in <module>
ImportError: No module named shared

The difference seems to be the result calculated from this function

subpar/subpar.bzl

Lines 97 to 111 in 07ff5fe

 def _prepend_workspace(path, ctx): 

 """Given a path, prepend the workspace name as the parent directory""" 

 # It feels like there should be an easier, less fragile way. 

 if path.startswith('../'): 

 # External workspace, for example 

 # '../protobuf/python/google/protobuf/any_pb2.py' 

 stored_path = path[len('../'):] 

 elif path.startswith('external/'): 

 # External workspace, for example 

 # 'external/protobuf/python/__init__.py' 

 stored_path = path[len('external/'):] 

 else: 

 # Main workspace, for example 'mypackage/main.py' 

 stored_path = ctx.workspace_name + '/' + path 

 return stored_path

If I hack this function to remove ../xctestrunner/ from the beginning of the path, I get a valid par.

Is something incorrect with my configuration? Is this use case supported?

Sass dependency is broken

$ bazel build //...
ERROR: /usr/local/google/home/dgreiman/.cache/bazel/_bazel_dgreiman/e5a26b4e88a22444ade4e9e5494d8dc3/external/io_bazel_rules_sass/sass/sass.bzl:81:19: Traceback (most recent call last):
	File "/usr/local/google/home/dgreiman/.cache/bazel/_bazel_dgreiman/e5a26b4e88a22444ade4e9e5494d8dc3/external/io_bazel_rules_sass/sass/sass.bzl", line 72
		rule(attrs = {"src": attr.label(allow...)}, <2 more arguments>)
	File "/usr/local/google/home/dgreiman/.cache/bazel/_bazel_dgreiman/e5a26b4e88a22444ade4e9e5494d8dc3/external/io_bazel_rules_sass/sass/sass.bzl", line 81, in rule
		attr.label(default = Label("//sass:sassc"), e..., ...)
cfg parameter is mandatory when executable=True is provided. Please see https://www.bazel.build/versions/master/docs/skylark/rules.html#configurations for more details.
ERROR: error loading package '': Extension file 'sass/sass.bzl' has errors.

Default compiler label broken

Attempting to build a par file gives this:

$ bazel build //...
ERROR: /path/BUILD:3:1: no such package 'compiler': BUILD file not found on package path and referenced by '//:foo.par'
ERROR: Analysis of target '//:foo.par' failed; build aborted: no such package 'compiler': BUILD file not found on package path
INFO: Elapsed time: 0.271s
FAILED: Build did NOT complete successfully (0 packages loaded)

Cause: the default value of the compiler attribute is being evaluated in the wrong context. See #70

OS X support

Hey,
Are you considering supporting OS X?
cc @Capibara-

parfile_test should use test size/timeout

When running a par_test, the native py_test uses size/timeout as specified in kwargs. But the corresponding parfile_test ignores those, defaulting to size="moderate". Not a major issue, but impacts e.g. smoke testing when I want to run small/short tests first: in a par_test marked as size="small" the native py_test gets run but the corresponding parfile_test gets filtered out due to its larger size. Expected instead: The parfile_test gets run using the size/timeout as specified in the par_test.

Windows support

Opening this as a general tracking issue for PAR support for Bazel on Windows.

The first issue I'm hitting seems to be where the PAR compiler parses the stub file.

`--action_env` not passed when building `//compiler:compiler.par`

I'd like to pass PYTHONNOUSERSITE=1 while compiling subpar.

As shown in the log below, the env variable is passed to compiler.par when building bazel_test_runner.par.

However, the env variable is not passed to compiler when compiling compiler.par itself for internal use. (Seems like the issue is somewhat related to bazelbuild/bazel#3320 (comment))

This is causing problems when building subpar binaries from our CI server running under docker where python site.py fails while finding python user base dir.

BTW, is there any reason the default compiler is //compiler:compiler.par, not //compiler:compiler?

Log:

$ bazel build --action_env PYTHONNOUSERSITE=1 //build_tools:bazel_test_runner.par
INFO: Build options have changed, discarding analysis cache.
INFO: Analysed 5 targets (1 packages loaded).
INFO: Found 5 targets...
SUBCOMMAND: # @subpar//compiler:compiler.par [action 'Building par file @subpar//compiler:compiler.par [for host]']
(cd /private/var/tmp/_bazel_sjcho/63181ab111ac20e668ef72c8d40fea5b/execroot/__main__ && \
  exec env - \
    PATH=/usr/local/bin:/usr/bin:/bin \
  bazel-out/host/bin/external/subpar/compiler/compiler --manifest_file bazel-out/host/bin/external/subpar/compiler/compiler.par_SOURCES --outputpar bazel-out/host/bin/external/subpar/compiler/compiler.par --stub_file bazel-out/host/bin/external/subpar/compiler/compiler --zip_safe True external/subpar/compiler/compiler.py)
SUBCOMMAND: # //build_tools:bazel_test_runner.par [action 'Building par file //build_tools:bazel_test_runner.par']
(cd /private/var/tmp/_bazel_sjcho/63181ab111ac20e668ef72c8d40fea5b/execroot/__main__ && \
  exec env - \
    PATH=/usr/local/bin:/usr/bin:/bin \
    PYTHONNOUSERSITE=1 \
  bazel-out/host/bin/external/subpar/compiler/compiler.par --manifest_file bazel-out/darwin-opt/bin/build_tools/bazel_test_runner.par_SOURCES --outputpar bazel-out/darwin-opt/bin/build_tools/bazel_test_runner.par --stub_file bazel-out/darwin-opt/bin/build_tools/bazel_test_runner --zip_safe True build_tools/bazel_test_runner.py)
INFO: Elapsed time: 5.345s, Critical Path: 1.07s
INFO: 2 processes: 2 darwin-sandbox.
INFO: Build completed successfully, 13 total actions

My bazel version is 0.18.0, and subpar 1.3.0.

Python 3 par_binary does not work if main source file is converted via 2to3

A par_binary must set srcs_version to "PY2AND3" or "PY3", otherwise it fails with an error message like:

attribute main: Main entry point [Artifact:[/usr/local/google/home/dgreiman/code/github/subpar[source]]tests/package_f/f_PY3.py] not listed in srcs.

Support extracting the archive before running

Sometimes we have a resource file that needs to be extracted (e.g. a bash script that needs to be executed).

I wonder if it makes sense for subpar to support the mode where we extract the archive to a specific location and runs the python code.

In pex, there's zip_safe=False that achieves this goal.

Do tagged releases of subpar

The git_repository() rule in Bazel no longer accepts "HEAD", and I shouldn't make people use HEAD that anyway.

Boilerplate shebang question

I've come to realise that the Bazel Python support is not hermetic and that there is lots of work happening around py_runtime and py_toolchain to possibly address some of the issues.

One thing that I've noticed with subpar specifically, is the code that generates the boilerplate shebang line in the .par executable file, is obtained by sniffing the PYTHON_BINARY from the stub_file.

The implications of this for me, are that the target runtime environment will need to have the python interpreter installed at the same location as the environment used to build the subpar archive.

This is currently causing me some problems, because I'm trying to use Nix to setup a hermetic build environment. The regular build and test is working reliably on Nix for developers and CI machines, but unfortunately, the .par files that are generated are not portable. This is because the location of the Python3 interpreter within Nix becomes something like: /nix/store/qbp8hp47y9xbk7065q9rkzhfbwaym6cy-python3-3.7.0/bin/python3
and the subpar shebang then becomes:
#!/nix/store/qbp8hp47y9xbk7065q9rkzhfbwaym6cy-python3-3.7.0/bin/python3

When this is then bundled into a distroless or alpine Python3 Docker container, the subpar file cannot be executed, because the Python interpreter is not located in the same location.

Now, I guess that I could make the target runtime environment skip the shebang line by invoking it like python3 file.par.

I'm just wondering if this is the best way to be doing things, or if there should be an additional way of setting the interpreter in the shebang.

Build fails with bazel 0.25 --incompatible_use_python_toolchains

It seems like the first line in the par has an incomplete path to the wrapper script:

$ head bazel-bin/compiler/compiler.par
#!bazel_tools/tools/python/py2wrapper.sh
PK

parfile depends upon non-deterministic runfiles_manifest

The runfiles_manifest for src contains paths that are relative to the output_base of a bazel WORKSPACE.

Neither the rule implementation nor the action use this file as an input.

Why was it included in the first place in the extra_inputs set? This prevents reliable shared remote caching and results in unnecessary remote executions.

Cannot execute par containing grpcio

When I run .par file built with par_binary, I receive following error

$ bazel-bin/client.par
Traceback (most recent call last):
  File "/home/pjachowi/anaconda3/lib/python3.6/runpy.py", line 193, in _run_module_as_main
    "__main__", mod_spec)
  File "/home/pjachowi/anaconda3/lib/python3.6/runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "bazel-bin/client.par/__main__.py", line 6, in <module>
  File "bazel-bin/client.par/pypi__grpcio_1_6_0/grpc/__init__.py", line 22, in <module>
ImportError: cannot import name 'cygrpc'

Here is my hello-world project:

$ cat WORKSPACE 
#load("@bazel_tools//tools/build_defs/repo:git.bzl", "git_repository")
git_repository(
  name = "org_pubref_rules_protobuf",
  remote = "https://github.com/pubref/rules_protobuf",
  tag = "v0.8.2",
)

load("@org_pubref_rules_protobuf//python:rules.bzl", "py_proto_repositories")
py_proto_repositories()


# ================================================================
# Python GRPC support requires rules_python
# ================================================================
load("@org_pubref_rules_protobuf//protobuf:rules.bzl", "github_archive")
github_archive(
    name = "io_bazel_rules_python",
    commit = "8b5d0683a7d878b28fffe464779c8a53659fc645",
    org = "bazelbuild",
    repo = "rules_python",
    sha256 = "40499c0a9d55f0c5deb245ed24733da805f05aaf6085cb39027ba486faf1d2e1",
)

load("@io_bazel_rules_python//python:pip.bzl", "pip_repositories", "pip_import")

pip_repositories()

pip_import(
   name = "pip_grpcio",
   # requirements = "//third_party:grpcio_requirements.txt",
   requirements = "@org_pubref_rules_protobuf//python:requirements.txt",
)

load("@pip_grpcio//:requirements.bzl", pip_grpcio_install = "pip_install")

pip_grpcio_install()


# ================================================================
# Support for par - python archives
# ================================================================
git_repository(
    name = "subpar",
    remote = "https://github.com/google/subpar",
    tag = "1.0.0",
)

$ cat BUILD 
package(default_visibility = ["//visibility:public"])

load("@org_pubref_rules_protobuf//python:rules.bzl", "py_proto_library")
load("@subpar//:subpar.bzl", "par_binary")
load("@pip_grpcio//:requirements.bzl", "requirement")

proto_library(
  name = "hello_proto",
  srcs = ["hello.proto"],
)

py_proto_library(
  name = "hello_py_proto",
  protos = ["hello.proto"],
  py_proto_deps = [
    "hello_proto",
  ],
)

par_binary(
  name = "client",
  srcs = ["client.py"],
  deps = [":hello_py_proto",
        requirement("grpcio"),
        requirement("protobuf"),
    ],
)

$ cat hello.proto 
syntax = "proto3";

// The greeting service definition.
service Greeter {
  // Sends a greeting
  rpc SayHello (HelloRequest) returns (HelloReply) {}
}

// The request message containing the user's name.
message HelloRequest {
  string name = 1;
}

// The response message containing the greetings
message HelloReply {
  string message = 1;
}

$ cat client.py 
import grpc
import hello_pb2
import hello_pb2_grpc

class Greeter(hello_pb2_grpc.GreeterServicer):

  def SayHello(self, request, context):
    return hello_pb2.HelloReply(message='Hello, %s!' % request.name)

  def SayHelloAgain(self, request, context):
    return hello_pb2.HelloReply(message='Hello again, %s!' % request.name)

To my surprise, when I unzip client.par and invoke PYTHONPATH=. python ./__main__.py it works fine.

subpar not working with psycopg2 dependency.

I've made a par_binary() called batch-check that has psycopg2 as a dependency. The par bundle seems to be alright but the main import cannot work out the dependencies of psycopg. This is the error I get:

File "batch-check.par/__main__.py", line 6, in <module>
  File "batch-check.par/pypi__psycopg2_2_7_4/psycopg2/__init__.py", line 50, in <module>
ImportError: No module named _psycopg

I can see pypi__psycopg2_2_7_4/psycopg2/_psycopg.so inside the par file, but it appears to be a .so shared library file. I've attached the par file as a zip here because the par file extension is not allowed on github.

Thank you for your help.

kpi-check.zip

Advantages over `--build_python_zip`

Bazel has a --build_python_zip flag for build that also produces a self-contained executable binary. Could someone update the README.md to clearly articulate the advantages the .par files provide over that?

Reference: https://bazel.build/designs/2016/09/05/build-python-on-windows.html

[Cleanup] Rename all tests for consistency

Rename everything in the tests/ directory

Rename 'package_a' -> 'package_human_readable_name' for [a-g]
Rename a.py -> human_readable_name.py for [a-g]
Remove all import nastiness out of 'package_a/a.py' into 'package_circular_import/circular_import.py'
Rename package_shadow/main.py to package_shadow/shadow.py

Back to back Bazel commands fail with strange error

You can see this in my initial attempt at a .travis.yml for google/containerregistry:

ERROR: /home/travis/build/google/containerregistry/BUILD.bazel:28:1: Building par file //:puller.par failed: Process exited with status 1 [sandboxed].
Traceback (most recent call last):
  File "/usr/lib/python2.7/runpy.py", line 162, in _run_module_as_main
    "__main__", fname, loader, pkg_name)
  File "/usr/lib/python2.7/runpy.py", line 72, in _run_code
    exec code in run_globals
  File "bazel-out/host/bin/external/subpar/compiler/compiler.par/__main__.py", line 28, in <module>
  File "bazel-out/host/bin/external/subpar/compiler/compiler.par/subpar/compiler/cli.py", line 114, in main
  File "bazel-out/host/bin/external/subpar/compiler/compiler.par/subpar/compiler/python_archive.py", line 113, in create
  File "bazel-out/host/bin/external/subpar/compiler/compiler.par/subpar/compiler/python_archive.py", line 250, in write_zip_data
  File "bazel-out/host/bin/external/subpar/compiler/compiler.par/subpar/compiler/stored_resource.py", line 51, in store
  File "/usr/lib/python2.7/zipfile.py", line 1118, in write
    st = os.stat(filename)
OSError: [Errno 2] No such file or directory: 'bazel-bin/puller.runfiles/__init__.py'
Use --strategy=PythonCompile=standalone to disable sandboxing for the failing actions.

Readme needs to show the actual build command

If I wasn't an ex-googler, I wouldn't know that in order to build a *.par you need to add .par to the name of the target. I.e. do

bazel build //bar/baz/quux.par

Even though the target is just quux.

Subpar ignores Bazel imports

Subpar ignores Bazel py_library imports attributes. This is needed by any workspace with interdependent Python setuptools packages.

The TODO currently references a Buganizer issue: https://github.com/google/subpar/blob/master/subpar.bzl#L62

No support for purelib data directory

According to PEP491, part of the wheel spread involves:

Move each subtree of distribution-1.0.data/ onto its destination path. Each subdirectory of distribution-1.0.data/ is a key into a dict of destination directories, such as distribution-1.0.data/(purelib|platlib|headers|scripts|data).

The purelib directory should resolve to the root of the package.

xgboost is currently unsupported, however my PR overcomes the SO issue. As a next step, we should be moving the purelib and other directories to where they belong. I wrote a sample implementation here:
jac-stripe@50e9b37

I think the best way to reproduce this is to just get the latest master of xgboost building and working (not just 0.6a2 on pypi - thats quite outdated) as an import, as that package seems to exhibit a few issues in subpar.

I don't know enough about python and subpar to know if this is the best solution going forward.

par files are already perl archives

Perl Archive Toolkit already creates .par files since 2002 at least. Perhaps go with PYAR/.pyar?

Non-determinism in PAR output

See: bazelbuild/rules_python#10 (comment)

Add the exact bazel build command in README

We need to run bazel build <name>.par in order to build .par.

Can't use HEAD in git_repository

git_repository(
    name = "subpar",
    remote = "https://github.com/google/subpar",
    commit = "HEAD",
)

It errors out with:

ERROR: error loading package '': Encountered error while reading extension file 'subpar.bzl': no such package '@subpar//': Invalid branch, tag, or commit: Ref HEAD can not be resolved.

Here's my version:

$ bazel version
Build label: 0.4.5
Build target: bazel-out/local-fastbuild/bin/src/main/java/com/google/devtools/build/lib/bazel/BazelServer_deploy.jar
Build time: Thu Mar 16 12:19:38 2017 (1489666778)
Build timestamp: 1489666778
Build timestamp as int: 1489666778

C extension modules in 'deps' is not yet supported

Lots of Python packages out there depend on C modules. In my case my code depends on python-apt (https://anonscm.debian.org/gitweb/?p=apt/python-apt.git). Any plans or time line on when this limitation will be lifted?

'from future import' in main source file doesn't work

This must be first in any Python source file, which conflicts with the boostrapping code that subpar prepends to main.py.

The boostrapping code also breaks coding directives like:

#!/usr/bin/python
# -*- coding: latin-1 -*-

that must be at the beginning of the file.

[Cleanup] Remove skydoc from workspace dependencies

And/or clean up the warnings:

WARNING: e5a26b4e88a22444ade4e9e5494d8dc3/external/io_bazel_rules_sass/sass/sass.bzl:81:19: Argument `cfg = "host"` or `cfg = "data"` is required if `executable = True` is provided for a label.
WARNING: e5a26b4e88a22444ade4e9e5494d8dc3/external/io_bazel_skydoc/skylark/skylark.bzl:154:15: Variables HOST_CFG and DATA_CFG are deprecated in favor of strings "host" and "data" correspondingly.

Migrate documentation generation from Skydoc to Stardoc

Skydoc is legacy and will be removed in future versions of @io_bazel_skydoc..

C extension modules

Wondering if there is any roadmap/ETA for supporting C extension modules.

Tests may fail when run via `bazel test`

After the fix to tests for #98, tests may fail when run directly via bazel test as opposed to through the runner script run_tests.sh. I see two modes of failure:

When --incompatible_use_python_toolchains is disabled, //tests:version_test may fail if /usr/bin/env python is a Python 2 interpreter, due to bazelbuild/bazel#4815.
When --incompatible_use_python_toolchains is enabled, the default toolchain's py[2|3]wrapper.sh script is added to runfiles, causing brittle assertions in many tests to fail. This doesn't appear under run_tests.sh because it has its own custom toolchain that adds no runfiles.

This isn't an immediate problem because CI still passes, but it is annoying for the developer workflow. For 1), I'd just wait for the toolchain flag flip. For 2), the test assertions should be made more robust. In the meantime, use run_tests.sh instead of bazel test //....

compiler can't find python binary

On some systems I see errors like the following:

benley@alnitak:~/pm/pi-k8s$ bazel build  @subpar//compiler:compiler.par --verbose_failures --strategy=PythonCompile=standalone                               
INFO: Found 1 target...           
ERROR: /home/benley/.cache/bazel/_bazel_benley/9e259f4d6c3d097d1daa32f0372be769/external/subpar/compiler/BUILD:40:1: Building par file @subpar//compiler:compiler.par failed: compiler failed: error executing command                                                                
  (cd /home/benley/.cache/bazel/_bazel_benley/9e259f4d6c3d097d1daa32f0372be769/execroot/pi-k8s && \                                        
  exec env - \                    
  bazel-out/host/bin/external/subpar/compiler/compiler --manifest_file bazel-out/local-fastbuild/bin/external/subpar/compiler/compiler.par_SOURCES --outputpar bazel-out/local-fastbuild/bin/external/subpar/compiler/compiler.par --stub_file bazel-out/local-fastbuild/bin/external/subpar/compiler/compiler external/subpar/compiler/compiler.py): com.google.devtools.build.lib.shell.BadExitStatusException: Process exited with status 1.                    
Traceback (most recent call last):
  File "bazel-out/host/bin/external/subpar/compiler/compiler", line 168, in <module>                                                       
    Main()                        
  File "bazel-out/host/bin/external/subpar/compiler/compiler", line 148, in Main                                                           
    raise AssertionError('Could not find python binary: ' + PYTHON_BINARY)                                                                 
AssertionError: Could not find python binary: python                 
Target @subpar//compiler:compiler.par failed to build                
INFO: Elapsed time: 0.103s, Critical Path: 0.03s

On others it works.

The difference appears to be in the behaviour of env - across linux distributions. On Ubuntu and Debian, the resulting "empty" environment still has $PATH set to a default value:

benley@aram:~$ env - bash -c set
BASH=/bin/bash
BASHOPTS=cmdhist:complete_fullquote:extquote:force_fignore:hostcomplete:interactive_comments:progcomp:promptvars:sourcepath
BASH_ALIASES=()
BASH_ARGC=()
BASH_ARGV=()
BASH_CMDS=()
BASH_EXECUTION_STRING=set
BASH_LINENO=()
BASH_SOURCE=()
BASH_VERSINFO=([0]="4" [1]="3" [2]="30" [3]="1" [4]="release" [5]="i586-pc-linux-gnu")
BASH_VERSION='4.3.30(1)-release'
DIRSTACK=()
EUID=1009
GROUPS=()
HOSTNAME=aram
HOSTTYPE=i586
IFS=$' \t\n'
MACHTYPE=i586-pc-linux-gnu
OPTERR=1
OPTIND=1
OSTYPE=linux-gnu
PATH=/usr/local/bin:/usr/local/sbin:/usr/bin:/usr/sbin:/bin:/sbin:.
PPID=6844
PS4='+ '
PWD=/home/benley
SHELL=/bin/bash
SHELLOPTS=braceexpand:hashall:interactive-comments
SHLVL=1
TERM=dumb
UID=1009
_=bash

(same output on ubuntu 16.04)

On at least one other distro (NixOS linux), there seems to be no such default PATH:

benley@alnitak:~$ env - bash -c set
BASH=/run/current-system/sw/bin/bash
BASHOPTS=cmdhist:complete_fullquote:extquote:force_fignore:hostcomplete:interactive_comments:progcomp:promptvars:sourcepath
BASH_ALIASES=()
BASH_ARGC=()
BASH_ARGV=()
BASH_CMDS=()
BASH_EXECUTION_STRING=set
BASH_LINENO=()
BASH_SOURCE=()
BASH_VERSINFO=([0]="4" [1]="4" [2]="12" [3]="1" [4]="release" [5]="x86_64-unknown-linux-gnu")
BASH_VERSION='4.4.12(1)-release'
DIRSTACK=()
EUID=1000
GROUPS=()
HOSTNAME=alnitak
HOSTTYPE=x86_64
IFS=$' \t\n'
MACHTYPE=x86_64-unknown-linux-gnu
OPTERR=1
OPTIND=1
OSTYPE=linux-gnu
PATH=/no-such-path
PPID=3931
PS4='+ '
PWD=/home/benley/pm/pi-k8s
SHELL=/run/current-system/sw/bin/bash
SHELLOPTS=braceexpand:hashall:interactive-comments
SHLVL=1
TERM=dumb
UID=1000
_=bash

It's not entirely clear to me why PATH gets set like that, but it certainly breaks the compiler's build.

Import list for .par file does not match Bazel stub script

See issue 10 for larger context.

When a workspace has no name, its files get added to the runfiles tree and .par file under the directory 'main'. The Bazel produced stub script adds this directory to sys.path automatically. The .par file code in subpar/runtime/support.py should as well.

Development Best Practices

I'm working on writing Bazel rules that are executed by subpar python packages. When I specify the .par target as the executable label for the rule I keep getting an error from bazel saying Not a regular file: /private/var/tmp/_bazel_.../a256311e92234013bb6c7cb2b2b6cad9/external/distroless/package_manager/dpkg_parser.par

What is the best way to rapidly develop the python code and test against bazel rules?

Thanks for your patience.

Subpar tests fail under Linux when Bazel invoked with --build_python_zip

$ bazel test --build_python_zip //...
INFO: Analysed 57 targets (0 packages loaded).
INFO: Found 30 targets and 27 test targets...
ERROR: /xxx/code/github/subpar/compiler/BUILD:40:1: Building par file //compiler:compiler.par failed (Exit 1)
Traceback (most recent call last):
  File "/tmp/Bazel.runfiles_HNt40m/runfiles/subpar/compiler/compiler.py", line 23, in <module>
    sys.exit(cli.main(sys.argv))
  File "/tmp/Bazel.runfiles_HNt40m/runfiles/subpar/compiler/cli.py", line 104, in main
    import_roots, interpreter = parse_stub(args.stub_file)
  File "/tmp/Bazel.runfiles_HNt40m/runfiles/subpar/compiler/cli.py", line 71, in parse_stub
    for line in stub_file:
  File "/usr/lib/python2.7/codecs.py", line 296, in decode
    (result, consumed) = self._buffer_decode(data, self.errors, final)
UnicodeDecodeError: 'utf8' codec can't decode byte 0x95 in position 63: invalid start byte
INFO: Elapsed time: 0.700s, Critical Path: 0.18s

Incompatible search for main files

Here is my py_binary rule:

py_binary(
    name = "testing",
    srcs = ["bin/testing.py"],
)

The py_binary rule finds the main file just fine even though it is in the bin directory. However, the following rule fails:

par_binary(
    name = "testing",
    srcs = ["bin/testing.py"],
)

I'm using Bazel version 0.26.1 and subpar 2.0.0

Tests fail for centos7 configuration

This error occurs in a clean checkout locally but not in travis.

Testing virtualenv tests/requirements-test-centos7.txt
Already using interpreter /usr/bin/python3
Using base prefix '/usr'
New python executable in /usr/local/google/home/brandjon/src/blah/subpar/.env/bin/python3
Also creating executable in /usr/local/google/home/brandjon/src/blah/subpar/.env/bin/python
Installing setuptools, pkg_resources, pip, wheel...done.
Collecting setuptools==0.9.8 (from -r tests/requirements-test-centos7.txt (line 2))
  Using cached https://files.pythonhosted.org/packages/51/20/7b9e9cc363f176d9c3f1c4570e8ed8654ceba5cf3b0022d06add2f0e36ae/setuptools-0.9.8-py2.py3-none-any.whl
Installing collected packages: setuptools
  Found existing installation: setuptools 41.0.1
    Uninstalling setuptools-41.0.1:
      Successfully uninstalled setuptools-41.0.1
Successfully installed setuptools-0.9.8
INFO: Starting clean (this may take a while). Consider using --async if the clean takes more than several minutes.
INFO: Analyzed 62 targets (32 packages loaded, 687 targets configured).
INFO: Found 32 targets and 30 test targets...
ERROR: /usr/local/google/home/brandjon/.cache/bazel/_bazel_brandjon/f0064f73a7ea36f3e834106b5eca1fe4/external/test_workspace/BUILD:32:1: Building par file @test_workspace//:test_compiler_label.par failed (Exit 1) compiler.par failed: error executing command bazel-out/host/bin/external/subpar/compiler/compiler.par --manifest_file bazel-out/k8-fastbuild/bin/external/test_workspace/test_compiler_label.par_SOURCES --output_par ... (remaining 6 argument(s) skipped)

Use --sandbox_debug to see verbose messages from the sandbox
Traceback (most recent call last):
  File "/usr/lib/python3.6/runpy.py", line 193, in _run_module_as_main
    "__main__", mod_spec)
  File "/usr/lib/python3.6/runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "bazel-out/host/bin/external/subpar/compiler/compiler.par/__main__.py", line 19, in <module>
  File "bazel-out/host/bin/external/subpar/compiler/compiler.par/subpar/runtime/support.py", line 326, in setup
  File "bazel-out/host/bin/external/subpar/compiler/compiler.par/subpar/runtime/support.py", line 157, in _setup_pkg_resources
  File "/usr/local/google/home/brandjon/src/blah/subpar/.env/lib/python3.6/site-packages/pkg_resources.py", line 1545, in <module>
    register_loader_type(importlib_bootstrap.SourceFileLoader, DefaultProvider)
AttributeError: module 'importlib._bootstrap' has no attribute 'SourceFileLoader'

Implicitly created init.py files cause grief

For example, in //tests:package_pkg_resources/main.par, the following files shouldn't exist (but do):

__init__.py
pypi__portpicker_1_2_0/__init__.py
pypi__yapf_0_19_0/__init__.py

`from future` in the main file causes SyntaxError

If the main python script contains from __future__ import ... statements, running par file causes the following error:

$ ./train.par
Traceback (most recent call last):
  File "/usr/lib/python2.7/runpy.py", line 165, in _run_module_as_main
    mod_name, loader, code, fname = _get_main_module_details(_Error)
  File "/usr/lib/python2.7/runpy.py", line 133, in _get_main_module_details
    return _get_module_details(main_name)
  File "/usr/lib/python2.7/runpy.py", line 119, in _get_module_details
    code = loader.get_code(mod_name)
  File "./train.par/__main__.py", line 74
SyntaxError: from __future__ imports must occur at the beginning of the file

This seems to be due to the boilerplate added by subpar compiler before the __future statements.:

# Boilerplate added by subpar/compiler/python_archive.py
from subpar.runtime import support as _
_.setup(import_roots=[u'__main__', u'six_archive'])
del _
# End boilerplate

Inconsistent import behaviour between py_binary and par_binary

Description

I encountered some unexpected incompatibility between py_binary and par_binary, that belies this line "par_binary() is a drop-in replacement for py_binary() in your BUILD file".

How to Reproduce

Subpar version:

# Import the rules to build Python subpar packages.
git_repository(
    name = "subpar",
    remote = "https://github.com/google/subpar.git",
    tag = "1.3.0",
)

Python Rules Version:

# TODO: switch back to upstream once https://github.com/bazelbuild/rules_python/pull/82
# and https://github.com/bazelbuild/rules_python/pull/136 are merged.
http_archive(
    name = "io_bazel_rules_python",
    sha256 = "9bd76f014ae4239f0d3ed0c29900a7e3d7307e5fa00506c287dc658cdfe3840a",
    strip_prefix = "rules_python-be39b7906da9c924b09ca78ce3fa152a10f782ae",
    url = "https://github.com/uri-canva/rules_python/archive/be39b7906da9c924b09ca78ce3fa152a10f782ae.tar.gz",
)

Given a WORKSPACE root of /example, create a buggy directory inside that, with the following:

A.py

import B


if __name__ == '__main__':
    b = B.Thing()
    print(b.speak())

B.py

class Thing:
    def speak(self):
        return "Hi I'm B.Thing!"

BUILD.bazel

load("@io_bazel_rules_python//python:python.bzl", "py_binary", "py_test")
load("@subpar//:subpar.bzl", "par_binary")

SRCS = [
    "A.py",
    "B.py",
]

# Simple py_binary
py_binary(
    name = "main",
    srcs = SRCS,
    default_python_version = "PY3",
    main = "A.py",
    srcs_version = "PY3",
    deps = [],
)

# Google subpar self-contained monoarchive executable
par_binary(
    name = "buggy",
    srcs = SRCS,
    default_python_version = "PY3",
#    imports = [""], <-- Uncommenting this changes our fortunes
    main = "A.py",
    srcs_version = "PY3",
    deps = [],
)

Executing bazel run --force_python=PY3 --python_path=$(which python3) //buggy:main gives the following:

INFO: Invocation ID: fa858959-917a-4240-b2ba-bd6e099c928f
INFO: Build options --force_python and --python_path have changed, discarding analysis cache.
INFO: Analysed target //buggy:main (1 packages loaded, 116 targets configured).
INFO: Found 1 target...
Target //buggy:main up-to-date:
  bazel-bin/buggy/main
INFO: Elapsed time: 0.224s, Critical Path: 0.04s
INFO: 0 processes.
INFO: Build completed successfully, 4 total actions
INFO: Build completed successfully, 4 total actions
Hi I'm B.Thing!

But executing bazel run --force_python=PY3 --python_path=$(which python3) //buggy:buggy.par gives this:

INFO: Invocation ID: 2f0582ac-c075-49b7-be27-76ad4e20a455
INFO: Analysed target //buggy:buggy.par (0 packages loaded, 16 targets configured).
INFO: Found 1 target...
Target //buggy:buggy.par up-to-date:
  bazel-bin/buggy/buggy.par
INFO: Elapsed time: 0.387s, Critical Path: 0.28s
INFO: 2 processes: 2 processwrapper-sandbox.
INFO: Build completed successfully, 10 total actions
INFO: Build completed successfully, 10 total actions
Traceback (most recent call last):
  File "/nix/store/d74rl714a32wbv5i5j5qblcl169k1vs4-python3-3.7.2/lib/python3.7/runpy.py", line 193, in _run_module_as_main
    "__main__", mod_spec)
  File "/nix/store/d74rl714a32wbv5i5j5qblcl169k1vs4-python3-3.7.2/lib/python3.7/runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "/private/var/tmp/_bazel_jbelotti/6d46c1a8d79be3c21fd51c25cfdc932c/execroot/data_eng/bazel-out/darwin-py3-fastbuild/bin/buggy/buggy.par/__main__.py", line 6, in <module>
ModuleNotFoundError: No module named 'B'

The Solution

The solution is to add imports = [""], to the par_binary rule, which is fine, but in my opinion this should be unnecessary as subpar should by default behave the same as py_binary.

Interested to hear your thoughts, and open to submitting a patch for this if you think it appropriate.

Close this issue

Just wondering if there is a technical reason for this. Otherwise I may try to add support for this.

Subpar not generating a correct PAR file

When I build my "binary" with bazel build some_app.par, it builds a self-contained PAR file but doesn't set the correct sys.path through the boilerplate code prepended to my actual application.

When I try to run some_app.par, it complains that it's unable to import its dependencies:

%  ../../bazel-bin/code/some_app/some_app.par
Traceback (most recent call last):
File "/usr/lib/python2.7/runpy.py", line 162, in _run_module_as_main
"__main__", fname, loader, pkg_name)
File "/usr/lib/python2.7/runpy.py", line 72, in _run_code
exec code in run_globals
File "../../bazel-bin/code/some_app/some_app.par/__main__.py", line 23, in <module>
ImportError: No module named code.util

Some details

In the workspace definition, I defined an external GIT repository that provides the extension required to build PAR files:

~/workspace $ cat WORKSPACE
git_repository(
name = "subpar",
remote = "https://github.com/google/subpar.git",
# Commit on 2016-08-06.
commit = "e77b91cc277fa8239596eff94b326af60185b62e",
)

The some_app's BUILD file is fairly simple:

~/workspace/some_app $ cat BUILD
package(default_visibility = ["//visibility:public"])
load("@subpar//:subpar.bzl", "par_binary")

par_binary(
  name = "some_app",
  srcs = ["some_app.py"],
  deps = [
    "//code/util:some_library",
    "//code/util:log",
  ],
)

The util BUILD file isn't rocket science either:

~/workspace/some_app $ cat ../util/BUILD
package(default_visibility = ["//visibility:public"])

py_library(
  name = "some_library",
  srcs = ["some_library.py"],
)

py_library(
  name = "some_log",
  srcs = ["log.py"],
)

Imports from some_app.py:

$ grep import some_app.py
import os
import time
from code.util import some_library
from code.util import log

When I unzipped the PAR file, it looks like all dependencies are there but sys.path isn't getting modified by the boilerplate code for them to be found:

% find /tmp/unzipped_parfile

__main__
__main__/code
__main__/code/some_app
__main__/code/some_app/some_app.py
__main__/code/some_app/__init__.py
__main__/code/some_app/some_app
__main__/code/util
__main__/code/util/__init__.py
__main__/code/util/some_library.py
__main__/code/util/log.py
__main__/code/__init__.py
subpar
subpar/__init__.py
subpar/runtime
subpar/runtime/__init__.py
subpar/runtime/support.py
__main__.py

Par executables can't find data

I'm using subpar to generate an independent python executable with Bazel. Unfortunately, it doesn't appear that the resulting par file can find data files referenced by the par_binary() rule.

For example, consider this trivial workspace: foo.tar.gz

The par file is built and runs fine:

$ tar -xvzf foo.tar.gz
x foo/
x foo/BUILD
x foo/dir/
x foo/foo.py
x foo/WORKSPACE
x foo/dir/file.txt
$ cd foo
$ bazel build :foo.par
INFO: Found 1 target...
Target //:foo.par up-to-date:
  bazel-bin/foo.par
INFO: Elapsed time: 1.103s, Critical Path: 0.00s
$ bazel-bin/foo.par
Hello, World!

$

However, it appears that the executable is picking up the data files from the local filesystem, not the par file. Switching directories and trying to run the par file from its location produces an error:

$ cd ..
$ foo/bazel-bin/foo.par
Traceback (most recent call last):
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/runpy.py", line 162, in _run_module_as_main
    "__main__", fname, loader, pkg_name)
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/runpy.py", line 72, in _run_code
    exec code in run_globals
  File "foo/bazel-bin/foo.par/__main__.py", line 6, in <module>
IOError: [Errno 2] No such file or directory: 'dir/file.txt'
$

The expected behavior would be that the execution of the par file from outside the workspace succeeds just as it does within the workspace.

Discrepancy with py_binary

Hello,

I have a py_binary rule that utilizes pip_import dependencies. I'm not able to get get it to create a par file that will execute.

Could you tell me what I'm doing wrong?

Here's my code: https://github.com/hutchk/bazel_subpar

Imports added to end of sys.path, not the front

See issue 10 for larger context

The stub script created by Bazel adds items to the front of sys.path, while .par files add items to the end of sys.path. The relevant code is in subpar/runtime/support.py line 76. The .par file behavior should be changed to match Bazel.

Fix tests to use proper select() for python version

Looks like //tests:version_test relies on a custom flag to get the Python version. It should use @bazel_tools//tools/python:PY[2|3]. This may cause problems once --incompatible_use_python_toolchains is enabled (#98).

Move doc generation back into main workspace

In other words, undo f9dc571.

According to that commit's description, the justification for having two workspaces was that Bazel did not well support having both Python 2 and 3 in the same build, and Skydoc did not support Python 3. The issue of two versions in the same build should be addressed once the incompatible flags in Bazel 0.23 are flipped on by default.

External Repositories

Looks it does not work when we have external repositories. I am using it woth protobuf repository, and when I run .par file it says it can not import files from external repository.
Here is .par file contents: As can be seen files from python/google/protobuf are in one place, their __init__.py files are in different places.

./protobuf ./protobuf/python ./protobuf/python/google ./protobuf/python/google/protobuf ./protobuf/python/google/protobuf/descriptor_database.py ./protobuf/python/google/protobuf/pyext ./protobuf/python/google/protobuf/pyext/cpp_message.py ./protobuf/python/google/protobuf/text_encoding.py ./protobuf/python/google/protobuf/message.py ./protobuf/python/google/protobuf/service_reflection.py ./protobuf/python/google/protobuf/field_mask_pb2.py ./protobuf/python/google/protobuf/proto_builder.py ./protobuf/python/google/protobuf/type_pb2.py ./protobuf/python/google/protobuf/descriptor_pool.py ./protobuf/python/google/protobuf/api_pb2.py ./protobuf/python/google/protobuf/duration_pb2.py ./protobuf/python/google/protobuf/source_context_pb2.py ./protobuf/python/google/protobuf/text_format.py ./protobuf/python/google/protobuf/empty_pb2.py ./protobuf/python/google/protobuf/descriptor_pb2.py ./protobuf/python/google/protobuf/wrappers_pb2.py ./protobuf/python/google/protobuf/service.py ./protobuf/python/google/protobuf/reflection.py ./protobuf/python/google/protobuf/compiler ./protobuf/python/google/protobuf/compiler/plugin_pb2.py ./protobuf/python/google/protobuf/json_format.py ./protobuf/python/google/protobuf/internal ./protobuf/python/google/protobuf/internal/encoder.py ./protobuf/python/google/protobuf/internal/type_checkers.py ./protobuf/python/google/protobuf/internal/wire_format.py ./protobuf/python/google/protobuf/internal/_parameterized.py ./protobuf/python/google/protobuf/internal/containers.py ./protobuf/python/google/protobuf/internal/well_known_types.py ./protobuf/python/google/protobuf/internal/testing_refleaks.py ./protobuf/python/google/protobuf/internal/python_message.py ./protobuf/python/google/protobuf/internal/api_implementation.py ./protobuf/python/google/protobuf/internal/enum_type_wrapper.py ./protobuf/python/google/protobuf/internal/decoder.py ./protobuf/python/google/protobuf/internal/message_listener.py ./protobuf/python/google/protobuf/timestamp_pb2.py ./protobuf/python/google/protobuf/struct_pb2.py ./protobuf/python/google/protobuf/any_pb2.py ./protobuf/python/google/protobuf/descriptor.py ./protobuf/python/google/protobuf/message_factory.py ./protobuf/python/google/protobuf/symbol_database.py ./__main__.py ./__main__ ./__main__/external ./__main__/external/protobuf ./__main__/external/protobuf/__init__.py ./__main__/external/protobuf/python ./__main__/external/protobuf/python/google ./__main__/external/protobuf/python/google/protobuf ./__main__/external/protobuf/python/google/protobuf/pyext ./__main__/external/protobuf/python/google/protobuf/pyext/__init__.py ./__main__/external/protobuf/python/google/protobuf/__init__.py ./__main__/external/protobuf/python/google/protobuf/__init__.pyc ./__main__/external/protobuf/python/google/protobuf/compiler ./__main__/external/protobuf/python/google/protobuf/compiler/__init__.py ./__main__/external/protobuf/python/google/protobuf/internal ./__main__/external/protobuf/python/google/protobuf/internal/__init__.py ./__main__/external/protobuf/python/google/__init__.py ./__main__/external/protobuf/python/google/__init__.pyc ./__main__/external/protobuf/python/__init__.py ./__main__/external/__init__.py ./__main__/external/six_archive ./__main__/external/six_archive/__init__.py ./__main__/base ./__main__/base/__init__.py ./__main__/base/my_program.py ./subpar ./subpar/runtime ./subpar/runtime/__init__.py ./subpar/runtime/support.py ./subpar/__init__.py ./six_archive ./six_archive/six.py

	def _prepend_workspace(path, ctx):
	"""Given a path, prepend the workspace name as the parent directory"""
	# It feels like there should be an easier, less fragile way.
	if path.startswith('../'):
	# External workspace, for example
	# '../protobuf/python/google/protobuf/any_pb2.py'
	stored_path = path[len('../'):]
	elif path.startswith('external/'):
	# External workspace, for example
	# 'external/protobuf/python/__init__.py'
	stored_path = path[len('external/'):]
	else:
	# Main workspace, for example 'mypackage/main.py'
	stored_path = ctx.workspace_name + '/' + path
	return stored_path