Git Product home page Git Product logo

dbx_build_tools's Introduction

dbx_build_tools

dbx_build_tools is a collection of Bazel rules and associated tooling to build and test applications deployed to Linux servers.

The build rules support hermetic Python binaries. This includes the ability to use packages from PyPI and link with C libraries built with Bazel’s built-in C/C++ rules. Python binaries include a Python interpreter built with Bazel. The BUILD file generator can automatically generate dependencies for Python libraries and binaries, saving you from the drudgery of updating the deps every time you add or remove an import.

We also have Go rules. Our BUILD file generator supports generating rules entirely from Go source files on the filesystem.

We also include the tooling to generate a custom build and runtime environment. This isolates the build environment from system tools to make remote caching and execution more reliable. It also isolates your binaries outputs from the host system, making major OS upgrades much simpler.

Lightning tour

Install Bazel

Follow the installation instructions making sure Bazel is on your path.

Create a new WORKSPACE

$ mkdir ~/dbx_build_tools_guide
$ cd ~/dbx_build_tools_guide

$ cat > .bazelrc
build --experimental_strict_action_env
build --platforms @dbx_build_tools//build_tools/cc:linux-x64-drte-off
build --host_platform @dbx_build_tools//build_tools/cc:linux-x64-drte-off
build --sandbox_fake_username
build --modify_execution_info=TestRunner=+block-network

# Work around https://github.com/bazelbuild/bazel/issues/6293 by setting a
# dummy lcov.
coverage --test_env=LCOV_MERGER=/bin/true

$ cat > WORKSPACE
load("@bazel_tools//tools/build_defs/repo:http.bzl", "http_archive")


# Real use cases should pin a commit and set the sha256.
http_archive(
    name = "dbx_build_tools",
    urls = ["https://github.com/dropbox/dbx_build_tools/archive/master.tar.gz"],
    strip_prefix = "dbx_build_tools-master",
)

load('@dbx_build_tools//build_tools/bazel:external_workspace.bzl', 'drte_deps')

drte_deps()

register_toolchains(
    "@dbx_build_tools//thirdparty/cpython:drte-off-39-toolchain",
)

Install bzl

$ bazel build @dbx_build_tools//build_tools:bzl
$ sudo cp -rL bazel-bin/external/dbx_build_tools/build_tools/bzl{,.runfiles} /usr/bin/

Create an application

$ mkdir -p python/website
$ cat > python/website/hello.py
import random


def say_hello():
    return 'Hello'

$ cat > python/website/main.py
import sys

from wsgiref.simple_server import make_server

from python.website.hello import say_hello


def hello_app(environ, start_response):
    start_response('200 OK', [('Content-type', 'text/html')])

    return [
        '<html><strong>',
        say_hello(),
        '</strong></html>',
    ]


def main(port):
    server = make_server('localhost', port, hello_app)
    server.serve_forever()


if __name__ == '__main__':
    main(int(sys.argv[1]))

$ cat > python/website/BUILD.in
load('@dbx_build_tools//build_tools/services:svc.bzl', 'dbx_service_daemon')


dbx_py_library(
    name = "hello",
    srcs = ["hello.py"],
)

dbx_py_binary(
    name = "website",
    main = 'main.py',
)

dbx_service_daemon(
    name = "website_service",
    owner = "web_site_team",
    exe = ":website",
    args = ['5432'],
    http_health_check = 'http://localhost:5432',
)

$ bzl gen //python/website/...

Run an interactive development environment

$ bzl itest-start //python/website:website_service
$ curl http://localhost:5432
<html><strong>Hello</strong></html>
$ bzl itest-stop //python/website:website_service

Write some tests

$ cat > python/website/unit_test.py
from python.website.hello import say_hello


def test_say_hello():
    assert say_hello() == 'Hello'

$ cat > python/website/itest.py
try:
    from urllib2 import urlopen
except ImportError:
    from urllib.request import urlopen


def test_hello():
    assert b'Hello' in urlopen('http://localhost:5432').read()

$ cat >> python/website/BUILD.in

dbx_py_pytest_test(
    name ='unit_test',
    srcs = ['unit_test.py'],
)

dbx_py_pytest_test(
    name ='itest',
    srcs = ['itest.py'],
    services = [":website_service"],
)

$ bzl gen //python/website/...
$ bazel test //python/website/...
INFO: Elapsed time: 46.494s, Critical Path: 40.22s
INFO: 1618 processes: 1578 linux-sandbox, 40 local.
INFO: Build completed successfully, 1825 total actions
//python/website:itest                                                   PASSED in 1.3s
//python/website:unit_test                                               PASSED in 1.1s
//python/website:website_service_service_test                            PASSED in 0.5s

INFO: Build completed successfully, 1825 total actions

dbx_build_tools's People

Contributors

adanducci avatar armooo avatar ashtuchkin avatar benjaminp avatar derpberx avatar dzbarsky avatar goffrie avatar ilinum avatar jhance avatar jonkaya avatar jsharpe avatar jukkal avatar kylec1 avatar lesniewski avatar maxbelanger avatar mbakkardb avatar michaelleedbx avatar mrwright avatar mypy-build-bot avatar nikhilm avatar nipunn1313 avatar permutohedra avatar raindrift avatar sakawa avatar savetherbtz avatar svalentin avatar tmildorf avatar utsav-dbx avatar veshij avatar vgod-dbx avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar

dbx_build_tools's Issues

No matching toolchains found for types @bazel_tools//tools/cpp:toolchain_type

Hitting below error when trying out the example on mac:

ERROR: While resolving toolchains for target @dbx_build_tools//build_tools/py:check_conflicts: No matching toolchains found for types @bazel_tools//tools/cpp:toolchain_type. Maybe --incompatible_use_cc_configure_from_rules_cc has been flipped and there is no default C++ toolchain added in the WORKSPACE file? See https://github.com/bazelbuild/bazel/issues/10134 for details and migration instructions.

Some googling suggests that this might be related to Bazel remote execution. Is this expected?

FYI I had to revert the python 3.7 removal commit, otherwise getting below error:

no such target '@dbx_build_tools//thirdparty/cpython:drte-off-38-toolchain': target 'drte-off-38-toolchain' not declared in package 'thirdparty/cpython' (did you mean 'drte-off-37-toolchain'?)

Question: Using dbx_service_task to generate data for dbx_services_test

  1. Problem description:
    • We have integration test which requires an input data
    • Input data is result of python binary execution wrapped with dbx_service_task, it executes some actions and produce hash in a file as result for those actions
    • We want that file be accessible in test's context.
    • Currently, dbx_service_task does NOT declare any output so there's nothing to declare as 'data' dependency for test.
  2. Questions:
    • Do you have similar use cases, and how did you handle them?
    • If not, what do you think is the way to pass files created by the service task into context of dbx_service_test target?

Go builds fail on filesystems that require UTF-8 filenames

I was trying out dbx_build_tools for our python project, but ran into an issue again that has been plaguing me for a while. The root cause is an upstream Bazel issue that I described in bazelbuild/bazel#12986.

Basically Go's source tree has one file - a test case for its own support for UTF-8 filenames - that causes Bazel to blow up on filesystems that require filenames to be UTF-8 encoded. Such filesystems include whatever macOS uses these days, as well as ZFS with the utf8only flag set (which happens to be the default on Ubuntu when enabling ZFS support at installation time). I've attached a screenshot of the build failure resulting when I run bazel build @dbx_build_tools//build_tools:bzl-gen.

rules_go added a workaround in bazelbuild/rules_go#2836. It shells out to the system tar command for tar.gz go archives.

image

using dbx_build_tools with Docker

Hello,

I'm grateful for this package which simplifies running and solving python dependencies quite a bit. However, I'd like to know if there is some recommended way to package the python scripts including runfiles into docker. There seems to be a problem mentioned in bazelbuild/rules_pkg#339 where python runfiles are not properly included in pkg_tar even when include_runfiles is specified

Is this still active?

@jhance if this project is no longer going to be updated then it'd be good to see a message on this repo to that effect and for it to be archived?

mypy bazel rules appear to be missing some dependencies

I would like to try out the mypy rule, and modified the example slightly to include a dbx_mypy_test target. However, I got the following error - every rule of type _dbx_mypy_test implicitly depends upon the target '@dbx_build_tools//dropbox/mypy:mypy_test', but this target could not be found because of: no such package '@dbx_build_tools//dropbox/mypy': BUILD file not found in directory 'dropbox/mypy' of external repository @dbx_build_tools

Digging through the starlark code, there appear to be various labels in mypy.bzl that don't actually appear in the codebase (there is no dropbox/mypy directory). Any advice? Let me know if you need explicit steps to repro.
`

openssl download url breaks when new version is released

When openssl releases a new version, the currently latest version gets moved to source/old/ and hence the url in this repo becomes invalid.
If a list of urls were provided then both urls could be included to prevent build breakages.

Error downloading [http://zlib.net/zlib-1.2.11.tar.gz]

Hi,

it seems the current zlib dependency link is dead and thus the

$ bazel build @dbx_build_tools//build_tools:bzl

command, as references in the README.md fails with ↓

ERROR:
/home/user/.cache/bazel/_bazel_user/d5f246ac2500b7dd646c66395fcf22ef/external/org_python_cpython_39/BUILD.bazel:242:11:
@org_python_cpython_39//:builtin_extensions depends on @net_zlib//:zlib in
repository @net_zlib which failed to fetch. no such package '@net_zlib//':
java.io.IOException: Error downloading [http://zlib.net/zlib-1.2.11.tar.gz] to
/home/user/.cache/bazel/_bazel_user/d5f246ac2500b7dd646c66395fcf22ef/external/net_zlib/temp3678545797472261272/zlib-1.2.11.tar.gz:
GET returned 404 Not Found

AFAIS, it's set here ↓

"net_zlib": ["http://zlib.net/zlib-1.2.11.tar.gz"],

it seems the zlib project moved it to https://zlib.net/fossils/zlib-1.2.11.tar.gz

Cheers

A need for more Documentation

At present, the documentation for dbx_build_tools is very minimal: it provides a "lightning tour" - which is no substitute for more detailed documentation that people might need when they get beyond the "hello world" stage.

This is a shame, because dbx_build_tools is the best option for a hermetic bazel python build, which makes it a better candidate than rules_python for anyone who takes hermeticity seriously. Without a more detailed README, it's hard to get to grips with, and I wonder if this means that it attracts a smaller userbase than it might otherwise do.

For example, things I'd like to know as a brand new user:

  • Do I need to use bzl? It seems to be a feature, rather than a requirement - and bringing in a new workflow that will only impact the (relatively small) python component of my project isn't attractive to me.
  • How can I resolve problems when packages cannot be found? (right now I can't get any imports to work with bzl gen, and I don't have anywhere to look - my debug stage so far involves looking through the dbx_build_tools repo, messing around with dbx_py_pypi_piplib in a BUILD file rather than BUILD.in, and generally chasing my tail).
  • I just want to focus on python3, with python2 hitting EOL over a year ago. But register_toolchains seems to need a python2.7 toolchain as well as the python3.8 toolchain that I want - why?

The documentation doesn't need to go into such details, but as things stand, it seems that such a clearly useful and well-thought-out project is being let down by the documentation. Once I get to grips with using the project I'd be happy to help on that front.

Support test_suite for dbx_services_test

Would it be possible to add support for test suites for the 'test' invoked on a dbx_services_test? The use case is that a collection of tests that can run in parallel may share a set of defined daemon or task services that need to be performed. Right now the following error occurs:

Traceback (most recent call last): File "/root/.cache/bazel/_bazel_root/7ebbdddc3c30692c56fb633c56be95c3/external/dbx_build_tools/build_tools/services/svc.bzl", line 481, column 75, in services_bin_impl launcher_args.append("--svc.test-bin={}".format(ctx.executable.bin.short_path)) Error: 'NoneType' value has no field or method 'short_path'

action_env not propagated to pypi build actions

It seems that action_env isn't included in ctx.configuration.default_shell_env as per the issue detailed here bazelbuild/bazel#12049.
This means that we can't propagate e.g. LIBRARY_PATH or LD_LIBRARY_PATH which is needed to support some mpi compiler wrappers e.g. with Intel MPI when compiling mpi4py.

Hermeticity bug: Package setup can find host libraries (leading to import failures)

When trying to install numpy, I found that it failed to import because it depended on libraries that it could not access - CBLAS and LAPACK - resulting in undefined symbol errors (sometimes cblas_dot, sometimes saxpy_, sometimes _gfortran_concat_string, depending on what you have installed, and what you have tried to provide via Bazel dependencies). The numpy wheel comes bundled with the necessary libraries but bypasses their installation if it finds implementations on the host system - allowing users to utilise alternative implementations suited to their needs.

A minimal reproduction in a docker image showed no problem - numpy worked fine - so the same code worked or borked depending on the host - indicative of a hermeticity issue. To verify, I installed openblas to the docker image before the numpy package is installed, and this gives it undefined symbol errors when trying to import numpy.

I've attached a MWE. Inside is a simple project that simply imports numpy, prints hello world, and outputs a simple mean to demonstrate numpy doing something. This is done within a docker environment to isolate the build from the host machine.

There are two dockerfiles: working.Dockerfile and broken.Dockerfile. The only difference between the two is that broken.Dockerfile installs libopenblas-dev before running bazel build.

To reproduce, download dbx_build_tools_bug.tar.gz and:

tar -xzvf dbx_build_tools_bug.tar.gz && cd dbx_build_tools_bug
docker build --network=host -t dbx_docker_broken -f broken.Dockerfile .
docker build --network=host -t dbx_docker_working -f working.Dockerfile .
docker run -it dbx_docker_working
docker run -it dbx_docker_broken

Running the working docker image outputs:

Starting local Bazel server and connecting to it...
INFO: Analyzed target //usage:example (39 packages loaded, 3812 targets configured).
INFO: Found 1 target...
Target //usage:example up-to-date:
  bazel-bin/usage/example
INFO: Elapsed time: 4.154s, Critical Path: 0.16s
INFO: 1 process: 1 internal.
INFO: Build completed successfully, 1 total action
INFO: Build completed successfully, 1 total action
Hello, world!
3.0

And running the 'broken' docker image outputs:

Starting local Bazel server and connecting to it...
INFO: Analyzed target //usage:example (39 packages loaded, 3812 targets configured).
INFO: Found 1 target...
Target //usage:example up-to-date:
  bazel-bin/usage/example
INFO: Elapsed time: 4.288s, Critical Path: 0.28s
INFO: 1 process: 1 internal.
INFO: Build completed successfully, 1 total action
INFO: Build completed successfully, 1 total action
Traceback (most recent call last):
  File "/root/.cache/bazel/_bazel_root/b570b5ccd0454dc9af9f65ab1833764d/execroot/__main__/bazel-out/k8-fastbuild/bin/usage/example.runfiles/__main__/usage/example-wrapper.py", line 42, in <module>
    exec(code, module.__dict__)
  File "/root/.cache/bazel/_bazel_root/b570b5ccd0454dc9af9f65ab1833764d/execroot/__main__/bazel-out/k8-fastbuild/bin/usage/example.runfiles/__main__/usage/example.py", line 1, in <module>
    import numpy
  File "/root/.cache/bazel/_bazel_root/b570b5ccd0454dc9af9f65ab1833764d/execroot/__main__/bazel-out/k8-fastbuild/bin/usage/example.runfiles/__main__/pip/numpy/numpy-cpython-38/lib/numpy/__init__.py", line 148, in <module>
    from . import lib
  File "/root/.cache/bazel/_bazel_root/b570b5ccd0454dc9af9f65ab1833764d/execroot/__main__/bazel-out/k8-fastbuild/bin/usage/example.runfiles/__main__/pip/numpy/numpy-cpython-38/lib/numpy/lib/__init__.py", line 25, in <module>
    from .index_tricks import *
  File "/root/.cache/bazel/_bazel_root/b570b5ccd0454dc9af9f65ab1833764d/execroot/__main__/bazel-out/k8-fastbuild/bin/usage/example.runfiles/__main__/pip/numpy/numpy-cpython-38/lib/numpy/lib/index_tricks.py", line 12, in <module>
    import numpy.matrixlib as matrixlib
  File "/root/.cache/bazel/_bazel_root/b570b5ccd0454dc9af9f65ab1833764d/execroot/__main__/bazel-out/k8-fastbuild/bin/usage/example.runfiles/__main__/pip/numpy/numpy-cpython-38/lib/numpy/matrixlib/__init__.py", line 4, in <module>
    from .defmatrix import *
  File "/root/.cache/bazel/_bazel_root/b570b5ccd0454dc9af9f65ab1833764d/execroot/__main__/bazel-out/k8-fastbuild/bin/usage/example.runfiles/__main__/pip/numpy/numpy-cpython-38/lib/numpy/matrixlib/defmatrix.py", line 11, in <module>
    from numpy.linalg import matrix_power
  File "/root/.cache/bazel/_bazel_root/b570b5ccd0454dc9af9f65ab1833764d/execroot/__main__/bazel-out/k8-fastbuild/bin/usage/example.runfiles/__main__/pip/numpy/numpy-cpython-38/lib/numpy/linalg/__init__.py", line 73, in <module>
    from .linalg import *
  File "/root/.cache/bazel/_bazel_root/b570b5ccd0454dc9af9f65ab1833764d/execroot/__main__/bazel-out/k8-fastbuild/bin/usage/example.runfiles/__main__/pip/numpy/numpy-cpython-38/lib/numpy/linalg/linalg.py", line 33, in <module>
    from numpy.linalg import lapack_lite, _umath_linalg
ImportError: /root/.cache/bazel/_bazel_root/b570b5ccd0454dc9af9f65ab1833764d/execroot/__main__/bazel-out/k8-fastbuild/bin/usage/example.runfiles/__main__/pip/numpy/numpy-cpython-38/lib/numpy/linalg/lapack_lite.cpython-38-x86_64-linux-gnu.so: undefined symbol: _gfortran_concat_string

Building python for aarch64 / ppc64le

I'm interested in fixing up these build rules for the third party dependencies so that they will build on arm64 and powerpc.
Primarily this is failing in the openssl and python builds at the moment. I can use the bazelify.sh script to generate a valid build file for openssl but what is the best approach for creating a build file that will work across all 3 platforms?
I suspect changing the generation to generate .bzl files which contain functions that return a select for the appropriate platform would be the approach here? This way the bazelify.sh script would be used on each platform to generate the appropriate files rather then trying to have a single build file that supports all 3 platforms?

map_each must be a function that accepts a single positional argument

Getting below error when following README.md to setup a project:

ERROR: /.../external/dbx_build_tools/build_tools/BUILD:68:1: in dbx_py_binary rule @dbx_build_tools//build_tools:bzl:
Traceback (most recent call last):
File "/.../external/dbx_build_tools/build_tools/BUILD", line 68
dbx_py_binary(name = 'bzl')
File "/.../external/dbx_build_tools/build_tools/py/py.bzl", line 610, in _dbx_py_binary_impl
dbx_py_binary_base_impl(ctx, <1 more arguments>)
File "/.../external/dbx_build_tools/build_tools/py/py.bzl", line 659, in dbx_py_binary_base_impl
emit_py_binary(ctx, main = main, <10 more arguments>)
File "/.../external/dbx_build_tools/build_tools/py/common.bzl", line 416, in emit_py_binary
conflict_args.add_all(<2 more arguments>)
map_each must be a function that accepts a single positional argument

Env:
Ubuntu 16.04.6 (docker image running on Mac)
Bazel 2.1.1

It seems to be related to this change: e95b030

I switched to its parent commit in http_archive and all worked fine:

http_archive(
name = "dbx_build_tools",
urls = ["https://github.com/dropbox/dbx_build_tools/archive/c599cbe093c813ced631aa7bd6a5b12a69840f1b.tar.gz"],
strip_prefix = "dbx_build_tools-c599cbe093c813ced631aa7bd6a5b12a69840f1b",
sha256 = '9b5ebf16589c9f55a3976def46a0a8d600015b99717421542a2a36086ae4eb06',
)

Tagged versions

Is there a plan to start tagging versions of dbx_build_tools? If not, I'd like to propose that this be considered. If rolling release is a specific project plan, then I wonder if there are ways of getting some of the user experience from a versioned system by way of regularly updating the README.

At the moment the documentation states the following:

Real use cases should pin a commit and set the sha256.

As a way of excusing a code example that does something... bad:

http_archive(
    name = "dbx_build_tools",
    urls = ["https://github.com/dropbox/dbx_build_tools/archive/master.tar.gz"],
    strip_prefix = "dbx_build_tools-master",
)

It seems that this is a direct result of having no tagged versions, so users who just want the bleeding edge code can do the above and then fix it later once they've got comfortable. However, hermeticity isn't an afterthought, and although it is not within the remit of this project to spoon-feed newcomers to Bazel, if the real use case should do something, the documentation probably shouldn't demonstrate the 'bad' way.

Another reason for versioning is that the documentation can become outdated when code changes, and providing the version that the documentation is valid for is a nice way of preventing confusion. What is valid for master one week might be valid for master the next as a procedural restriction (stable APIs are always nice), there may come a time that a change is required that renders existing documentation incorrect. Tying the documentation to versions helps this - if a README recommends downloading version 1.2.3, one can assume that it is a good representation of version 1.2.3.

Finally, users are able to see specific changes between two minor versions, so they can keep track of how those changes impact their use of the project. This is much harder to do in a rolling approach, because there can be commits that make changes, commits that later revert them, and generally reading through vague commit messages to see what has changed can get... messy, from a user perspective.

(There is the issue of not being able to keep the SHA256 correct at exactly version X.Y.Z, so a usual approach is to remove the SHA256, tag a version, then make an additional commit with the now-known SHA256 so that the first thing a user sees when visiting the github page is a valid set of instructions to get to a reproducible result).

Question: Support for macOS?

I attended bazelcon, and saw that you plan to support macOS. What is a rough timeline for that landing?

Currently we are trying to adopt your library, but running into issues on how to enable developers locally once using dbx_build_tools. We've tried using a containerized version of bazel, but are hitting quite a lot of technical blockers.

How do I build a distribution archive (archive/master.tar.gz)?

I'd like to add support for OSX x86-64.

To start, I want to introduce a change and then package that change into the file archive/master.tar.gz. I would assume there is a target, but I haven't been able to find it so far.

How do I build the distribution archive?

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.