bazelbuild / bazel-bench Goto Github PK

View Code? Open in Web Editor NEW

27.0 35.0 12.0 260 KB

Benchmarking tool for bazel

License: Apache License 2.0

Python 94.80% Starlark 5.20%

bazel-bench's People

Stargazers

Watchers

Forkers

meistert qinusty isabella232 tjgq ungtb10d limdor joeleba snazhmudinov zhengwei143 mmorel-35 c-mita vakhidbetrakhmadov

bazel-bench's Issues

Display non-zero exit status in summary

Description of the problem / feature request:

It's easy to miss a non-zero exit status in the long build log. It should also be visible in the results summary in the end.

If not all runs have the same status, you may even consider excluding measurements from non-zero exit code runs.

Feature requests: what underlying problem are you trying to solve with this feature?

Make it easy to detect errors / flakiness.

Flag --incompatible_no_implicit_file_export will break Bazel Bench in a future Bazel release

Incompatible flag --incompatible_no_implicit_file_export will be enabled by default in a future Bazel release [1], thus breaking Bazel Bench.

The flag is documented here: bazelbuild/bazel#10225

Please check the following CI builds for build and test results:

Never heard of incompatible flags before? We have documentation that explains everything.

If you don't want to receive any future issues for Bazel Bench or if you have any questions,
please file an issue in https://github.com/bazelbuild/continuous-integration

Important: Please do NOT modify the issue title since that might break our tools.

[1] The target release hasn't been determined yet. Our tool will update the issue title once the flag flip has been scheduled.

raise CalledProcessError when trying to start benchmark

I have followed the installation instructions, and tried to do some test benchmarking. But it fails with the following error:
https://gist.github.com/snazhmudinov/fe4b9140135eb3d9c1b42bb9b808a51d

Command I'm trying to run:

bazel run :benchmark \
-- \
--project_source=https://github.com/bazelbuild/rules_cc.git \
--data_directory=/tmp/bazel-bench-data \
--verbose \
-- build //:all

I have bazelisk, python/python3, git installed. I have also installed all the 3rd party requirements (according to installation guides).

I'm on macbook air: 1.6 GHz Dual-Core Intel Core i5, MacOS 12.6,

Add config-file support

Description of the problem / feature request:

Config-file interface would allow us more flexibility in our benchmarks: we'd be able to benchmark the same version of bazel, but with different conditions (e.g. flags, depot cl, ...).

Use case: As a Bazel developer, I'd like to benchmark the effect of my new flag on Bazel performance before submitting the change.

Fail the script when any individual run fails

Description of the problem / feature request:

We should fail the whole script if any individual run fails. Measurements from failed runs aren't very meaningful.

Exclude failed runs from the summary's numbers

Description of the problem / feature request:

This is a follow-up of #39. We'd like to exclude failed runs (exit code != 0) from the final numbers displayed in the summary tables.

Feature requests: what underlying problem are you trying to solve with this feature?

Failed runs aren't very meaningful in terms of performance benchmarking.

Collect JSON profile with bazel-bench

Description of the problem / feature request:

We'd like the option to collect the JSON profiles of our bazel runs with bazel-bench.

Feature requests: what underlying problem are you trying to solve with this feature?

To get a more in-depth perspective of the build performance.

[BazelCI] Add platform namespace for fetched bazel binaries

Description of the problem / feature request:

In bazel-bench-nightly, we currently only separate fetched bazel binaries by commits. This becomes a problem when the same VM is used for 2 different containers of 2 different platforms, as the directory where we store the binaries is shared.
This problem can be resolved by further namespacing the binaries by platforms.

Flag --incompatible_no_implicit_file_export will break Bazel Bench in Bazel 1.2.1

Incompatible flag --incompatible_no_implicit_file_export will break Bazel Bench once Bazel 1.2.1 is released.

Please see the following CI builds for more information:

[:ubuntu: 16.04 (OpenJDK 8)](:ubuntu: 16.04 (OpenJDK 8))
[:ubuntu: 18.04 (OpenJDK 11)](:ubuntu: 18.04 (OpenJDK 11))
[:darwin: (OpenJDK 8)](:darwin: (OpenJDK 8))

Questions? Please file an issue in https://github.com/bazelbuild/continuous-integration

Important: Please do NOT modify the issue title since that might break our tools.

Add option to track BEP metrics in benchmarks

Description of the problem / feature request:

The build event protocol exposes a bunch of good metrics about the invocation such as number of packages loaded, targets configured, actions created and executed.
It would be nice to be able to track them with bazel-bench.

Feature requests: what underlying problem are you trying to solve with this feature?

Get insights into project growth.

suggested by @zhengwei143

unit of memory is MB not s

Not all metrics are times, memory is MB.

[BazelCI] Empty commit string

Description of the problem / feature request:

https://buildkite.com/bazel-trusted/bazel-bench-nightly/builds/12
The previous day had no commit, so there shouldn't have been any fetching of binaries at all. Instead, the pipeline attempted to fetch a bazel binary with commit=''. This is potentially due to a bug in parsing git log.

Beautify stdout table

Description of the problem / feature request:

Improve the table alignment in the stdout output.

Collapse List Of Artifacts

Description of the problem / feature request:

The list of artifacts at the end of the report is pretty long. It'd be nice to be able to collapse it.

Missing benchmark

Description of the problem / feature request:

bazel-bench did not run on all specified bazel commits.

Bugs: what's the simplest, easiest way to reproduce this bug? Please provide a minimal example if possible.

Ran the following command:

bazel run :benchmark -- --bazel_commits ffad58728d850b465a070d25b037ea1bced0f91f,4782b8872c3bef8df56f3714ffd005dd3dcc95f7 --bazel_source /usr/local/google/home/twerth/bazel-up --collect_memory  --data_directory /tmp/bazel-g1_2 --project_commits ffad58728d850b465a070d25b037ea1bced0f91f --project_source /usr/local/google/home/twerth/bazel-up --runs 20 --collect_json_profile -- build --compilation_mode=opt //src:bazel

Note that I am using the same local repository as bazel and project source - is that causing any problems?

From the logs:

INFO: Running command line: bazel-out/k8-py2-fastbuild/bin/benchmark --bazel_commits ffad58728d850b465a070d25b037ea1bced0f91f,4782b8872c3bef8df56f3714ffd005dd3dcc95f7 --bazel_source /usr/local/google/home/twerth/bazel-up --collect_memory --data_directory /tmp/bazel-g1_2 --project_commits ffad58728d850b465a070d25b037ea1bced0f91f --project_source /usr/local/google/home/twerth/bazel-up --runs 20 --collect_json_profile -- build '--compilation_mode=opt' //src:bazel
...
I0619 07:56:51.347815 139708403218240 logger.py:33] Preparing bazelbuild/bazel repository.
I0619 07:56:51.347987 139708403218240 logger.py:33] Path /tmp/.bazel-bench/bazel exists. Updating...
I0619 07:56:51.811319 139708403218240 logger.py:33] Preparing /usr/local/google/home/twerth/bazel-up clone.
I0619 07:56:51.811598 139708403218240 logger.py:33] Path /tmp/.bazel-bench/project-clones/cbf1f465934e5495d5024f0a2bd99ea5 exists. Updating...
I0619 07:56:52.261014 139708403218240 logger.py:33] Binary exists at /tmp/.bazel-bench/bazel-bin/ffad58728d850b465a070d25b037ea1bced0f91f/bazel, reusing...
I0619 07:56:52.328670 139708403218240 logger.py:33] Pre-fetching external dependencies & exporting build event json to /tmp/.bazel-bench/out/build_env.json...

So it is already missing in the preparation phase.

failure unless --collect_memory is passed

Traceback (most recent call last):
  File "/home/benjamin/.cache/bazel/_bazel_benjamin/87896217ed058d0cf7ddf603fa041dc4/execroot/__main__/bazel-out/k8-fastbuild/bin/benchmark.runfiles/__main__/benchmark.py", line 695, in <module>
    app.run(main)
  File "/home/benjamin/.cache/bazel/_bazel_benjamin/87896217ed058d0cf7ddf603fa041dc4/execroot/__main__/bazel-out/k8-fastbuild/bin/benchmark.runfiles/third_party_pypi__absl_py_0_9_0/absl/app.py", line 299, in run
    _run_main(main, args)
  File "/home/benjamin/.cache/bazel/_bazel_benjamin/87896217ed058d0cf7ddf603fa041dc4/execroot/__main__/bazel-out/k8-fastbuild/bin/benchmark.runfiles/third_party_pypi__absl_py_0_9_0/absl/app.py", line 250, in _run_main
    sys.exit(main(argv))
  File "/home/benjamin/.cache/bazel/_bazel_benjamin/87896217ed058d0cf7ddf603fa041dc4/execroot/__main__/bazel-out/k8-fastbuild/bin/benchmark.runfiles/__main__/benchmark.py", line 634, in main
    results, args = _run_benchmark(
  File "/home/benjamin/.cache/bazel/_bazel_benjamin/87896217ed058d0cf7ddf603fa041dc4/execroot/__main__/bazel-out/k8-fastbuild/bin/benchmark.runfiles/__main__/benchmark.py", line 322, in _run_benchmark
    _single_run(bazel_bin_path, command, options, targets, startup_options,
  File "/home/benjamin/.cache/bazel/_bazel_benjamin/87896217ed058d0cf7ddf603fa041dc4/execroot/__main__/bazel-out/k8-fastbuild/bin/benchmark.runfiles/__main__/benchmark.py", line 266, in _single_run
    measurements['memory'],
KeyError: 'memory'

[BazelCI] Add support for environment variables

Description of the problem / feature request:

This is specific to the benchmarking pipeline on BazelCI.

Right now we only specify the platform to run benchmarking on. We should be able to replicate CI's tasks' environments, which include not only the platform but also things like ENV vars, build flags, ...

Example:
https://github.com/bazelbuild/bazel/blob/master/.bazelci/postsubmit.yml#L116

Allow grouping of actions by executed/cached in json profile analysis

Description of the problem / feature request:

Currently, the json_profile_merger logic simply takes time spans from the profiles and collates them for analysis.

It would be nice if we could categorise this data based on whether the action was executed locally, executed remotely, cached locally, cached remotely.

Visually it can be quite clear through chrome://tracing whether an action is using the remote cache as it has a span below with "Downloading Outputs", but this isn't considered when collecting this data as the spans are only related via the CPU executed.

Feature requests: what underlying problem are you trying to solve with this feature?

I would like to be able to produce reports on execution time of executed actions from daily CI builds. This is currently not possible as cached actions are grouped in and are quite fast.

Some targets only work on python2

Description of the problem / feature request:

Support for Python 2 is ending in 2020: https://pythonclock.org/. We currently have some targets that only works on python2. These need to be changed to support python3.

Bazel Bench is failing with Bazel@HEAD in CI

https://buildkite.com/bazel/bazel-at-head-plus-downstream/builds/3391#018b3b61-a496-4d55-ab25-f4e5c8481526Ba

Platform: Ubuntu, MacOS

Logs:

FAILED: //utils:bazel_test (Summary)
FAILED: //utils:values_test (Summary)
FAILED: //:benchmark_test (Summary)

Steps:

git clone https://github.com/bazelbuild/bazel-bench.git 
git reset 63b12c00145c30ccf3a230d62d30103654649c80 --hard
export USE_BAZEL_VERSION=2ddacab80af1dca3c04e74809827c47ec4bc8498
bazel test //...

CC Greenteam @fweikert

[Bazel CI] Error 'struct' value has no field or method 'to_json'

CI: https://buildkite.com/bazel/bazel-at-head-plus-downstream/builds/3740#018e5f2f-780e-4e60-b93d-40d592a3750a

Platform: Macos. Windows

Logs:

ERROR: �[0mAn error occurred during the fetch of repository 'third_party':
Error: 'struct' value has no field or method 'to_json'
ERROR: �[0m/Users/buildkite/builds/bk-imacpro-20/bazel-org-repo-root/bazel-bench/WORKSPACE:23:12: fetching pip_repository rule //external:third_party: Traceback (most recent call last):
Error: 'struct' value has no field or method 'to_json'
ERROR: �[0mSkipping '//...': error loading package under directory '': no such package '@@third_party//': 'struct' value has no field or method 'to_json'
ERROR: �[0merror loading package under directory '': no such package '@@third_party//': 'struct' value has no field or method 'to_json'

Culprit:

Steps:

 git clone https://github.com/bazelbuild/bazel-bench
git reset d1c5d8815f1069a5a278023e4c4344ab7c823b33  --hard
export USE_BAZEL_VERSION=b100e4f1631da2d7c78621c040e88b7644985cd8
bazel build //...

CC Greenteam @SalmaSamy

Error in path: Not a regular file: python3_10_x86_64-apple-darwin/WORKSPACE

https://buildkite.com/bazel/bazel-at-head-plus-downstream/builds/3521#018c5bca-c8f4-4692-b6ba-f2ceaaeb2094

Platform : MacOS

Logs:


(03:12:28) ERROR: An error occurred during the fetch of repository 'third_party':
   Traceback (most recent call last):
	File "/private/var/tmp/_bazel_buildkite/4e9cc81b0b6cd04f715b7270e313d439/external/rules_python/python/pip_install/pip_repository.bzl", line 235, column 63, in _pip_repository_impl
		environment = _create_repository_execution_environment(rctx),
	File "/private/var/tmp/_bazel_buildkite/4e9cc81b0b6cd04f715b7270e313d439/external/rules_python/python/pip_install/pip_repository.bzl", line 158, column 37, in _create_repository_execution_environment
		_maybe_set_xcode_location_cflags(rctx, env)
	File "/private/var/tmp/_bazel_buildkite/4e9cc81b0b6cd04f715b7270e313d439/external/rules_python/python/pip_install/pip_repository.bzl", line 83, column 26, in _maybe_set_xcode_location_cflags
		rctx.path(Label("@{}//:WORKSPACE".format(rctx.attr.python_interpreter_target.workspace_name))).dirname,
Error in path: Not a regular file: /private/var/tmp/_bazel_buildkite/4e9cc81b0b6cd04f715b7270e313d439/external/python3_10_x86_64-apple-darwin/WORKSPACE

Steps:

git clone https://github.com/bazelbuild/bazel-bench.git 
git reset 656fdf243022f3cbadf2f7ce161a401212add6e6 --hard
export USE_BAZEL_VERSION=104a711ae1772be62b6034506d51c5fee486b589
bazel build //...

CC Greenteam @comius

Remove `ubuntu1604` from CI builds

Description of the problem / feature request:
We no longer do benchmarking on ubuntu1604 on BazelCI, so it shouldn't be included in this project's CI environments.

Build fails with Python toolchains flag flip

Bazel 0.27 turns on --incompatible_use_python_toolchains. According to a recent downstream CI run with the flag flip, this project fails, probably because the flag flip causes it to use the correct Python version (as requested by the target) instead of whatever the system python version is. You can likely fix this by adding python_version = "PY2" to your Python 2 targets.

Benchmarking incremental builds

Description of the problem / feature request:

Currently bazel-bench isn't able to benchmark incremental builds due to the following reasons:

Does not allow running --warm_up=n runs prior to the benchmark --runs=n
Does not allow toggling --shutdown and --clean to allow us to toggle whether or not to run blaze shutdown / clean (currently runs both by default).
Does not allow patching of code, should be done via specifying a --patch_file.

Feature requests: what underlying problem are you trying to solve with this feature?

Allows users to benchmark the incremental case, by:

Specifying --warm_up runs to warm up the blaze cache.
Do not shutdown or clean in between runs so that we run with a warm blaze cache.
Apply the patch file in-between runs to simulate code changes.

[BazelCI] Create a dashboard-friendly report

Description of the problem / feature request:

The current report requires scrolling and is more tailored for viewing on personal devices. We'd like to have a dashboard-friendly display where no scrolling is required.

Flag --incompatible_load_python_rules_from_bzl will break Bazel Bench in Bazel 1.2.1

Incompatible flag --incompatible_load_python_rules_from_bzl will break Bazel Bench once Bazel 1.2.1 is released.

Please see the following CI builds for more information:

[:ubuntu: 16.04 (OpenJDK 8)](:ubuntu: 16.04 (OpenJDK 8))
[:ubuntu: 18.04 (OpenJDK 11)](:ubuntu: 18.04 (OpenJDK 11))
[:darwin: (OpenJDK 8)](:darwin: (OpenJDK 8))

Questions? Please file an issue in https://github.com/bazelbuild/continuous-integration

Important: Please do NOT modify the issue title since that might break our tools.

Include tooltips on each of the report components

Description of the problem / feature request:

This is to help the users of the reports to understand the report easier. We can add something like https://getbootstrap.com/docs/4.1/components/tooltips/.

[BazelCI] Add option to benchmark every X commit

Description of the problem / feature request:

As there's a timeout limit, it's not practical to run benchmarking at every commit (also we need to maintain the number of runs per commit to achieve statistical significance). We need an option to benchmark every X commit in the commit list.

Add e2e Tests

Description of the problem / feature request:

We should add some e2e tests. Something like what Bazel has https://github.com/bazelbuild/bazel/tree/master/src/test/shell

Feature requests: what underlying problem are you trying to solve with this feature?

To make sure the tool works properly end-to-end.

Add `--sandbox_tmpfs_path=/tmp` to the default bazel arguments on Linux.

Description of the problem / feature request:

Add --sandbox_tmpfs_path=/tmp to the default bazel arguments on Linux.

Building bazel itself (from clean) without this flag breaks in ~15% of the cases.
With the flag, it doesn't fail once in 50 clean builds.

Any other information, logs, or outputs that you want to share?

This is just a workaround until bazelbuild/bazel#3236 is fixed.

Invoking the example from README fails

Description of the problem / feature request:

Trying to run the example benchmark from README, and it gives me the error:

ERROR: /Users/sherozn/.bazel-bench/project-clones/3915c08828952724cf3dcead4c416118/tools/build_rules/java_rules_skylark.bzl:59:39: invalid escape sequence: \.. Use '\\' to insert '\'.
ERROR: /Users/sherozn/.bazel-bench/project-clones/3915c08828952724cf3dcead4c416118/src/BUILD:140:11: error loading package 'src/java_tools/buildjar/java/com/google/devtools/build/buildjar/jarhelper': compilation of module 'tools/build_rules/java_rules_skylark.bzl' failed and referenced by '//src:embedded_tools_jdk_minimal_srcs'
ERROR: Analysis of target '//src:bazel' failed; build aborted:
INFO: Elapsed time: 0.051s
INFO: 0 processes.
FAILED: Build did NOT complete successfully (4 packages loaded, 0 targets configured)
    currently loading: src/java_tools/buildjar/java/com/google/devtools/build/buildjar/jarhelper ... (2 packages)
    Fetching repository @local_config_cc_toolchains; starting
    Fetching repository @local_jdk; starting
Traceback (most recent call last):
  File "/private/var/tmp/_bazel_sherozn/4e4b6bfdaf070146b8f266fb0b3ab2cc/execroot/__main__/bazel-out/darwin_arm64-fastbuild/bin/benchmark.runfiles/__main__/benchmark.py", line 698, in <module>
    app.run(main)
  File "/private/var/tmp/_bazel_sherozn/4e4b6bfdaf070146b8f266fb0b3ab2cc/execroot/__main__/bazel-out/darwin_arm64-fastbuild/bin/benchmark.runfiles/third_party/pypi__absl_py/site-packages/absl/app.py", line 308, in run
    _run_main(main, args)
  File "/private/var/tmp/_bazel_sherozn/4e4b6bfdaf070146b8f266fb0b3ab2cc/execroot/__main__/bazel-out/darwin_arm64-fastbuild/bin/benchmark.runfiles/third_party/pypi__absl_py/site-packages/absl/app.py", line 254, in _run_main
    sys.exit(main(argv))
  File "/private/var/tmp/_bazel_sherozn/4e4b6bfdaf070146b8f266fb0b3ab2cc/execroot/__main__/bazel-out/darwin_arm64-fastbuild/bin/benchmark.runfiles/__main__/benchmark.py", line 621, in main
    bazel_bin_path = _build_bazel_binary(unit['bazel_commit'],
  File "/private/var/tmp/_bazel_sherozn/4e4b6bfdaf070146b8f266fb0b3ab2cc/execroot/__main__/bazel-out/darwin_arm64-fastbuild/bin/benchmark.runfiles/__main__/benchmark.py", line 175, in _build_bazel_binary
    _exec_command(['bazel', 'build', '//src:bazel'], cwd=repo.working_dir)
  File "/private/var/tmp/_bazel_sherozn/4e4b6bfdaf070146b8f266fb0b3ab2cc/execroot/__main__/bazel-out/darwin_arm64-fastbuild/bin/benchmark.runfiles/__main__/benchmark.py", line 61, in _exec_command
    return subprocess.run(
  File "/private/var/tmp/_bazel_sherozn/4e4b6bfdaf070146b8f266fb0b3ab2cc/external/python3_10_aarch64-apple-darwin/lib/python3.10/subprocess.py", line 524, in run
    raise CalledProcessError(retcode, process.args,
subprocess.CalledProcessError: Command '['bazel', 'build', '//src:bazel']' returned non-zero exit status 1.

Command I'm trying to execute from the terminal:

bazel run :benchmark \
-- \
--project_source=https://github.com/bazelbuild/rules_cc.git \
--data_directory=/tmp/bazel-bench-data \
--verbose \
-- build //:all

only one result printed for config file interface

I am using this:

global_options:
  project_commit: 10eddc3d5394d2483b0568c6f88f6954fc380158
  runs: 5
  collect_profile: True
  project_source: /usr/local/google/home/twerth/bazel
  bazel_binary: /usr/bin/bazel-real
units:
  - command: build --jobs=100 //src:bazel
  - command: build --jobs=600 //src:bazel

prints:

RESULTS:
Bazel commit: /usr/bin/bazel-real, Project commit: 10eddc3d5394d2483b0568c6f88f6954fc380158, Project source: /usr/local/google/home/twerth/bazel
  metric          mean                median          stddev      pval   
    wall:  111.753s             111.931s              0.622s             
     cpu:  124.410s             124.550s              3.667s             
  system:   52.578s              52.810s              0.885s             
  memory:   95.600MB             96.000MB             0.490MB

which is only the result of the second run.

Include task-specific shell commands and build flags

Description of the problem / feature request:

For example: https://github.com/bazelbuild/bazel/blob/master/.bazelci/postsubmit.yml#L119

ubuntu1804_javabase9:
    platform: ubuntu1804_nojava
    name: ":java: OpenJDK 9 Javabase"
    shell_commands:
      - sed -i.bak -e 's/^# android_sdk_repository/android_sdk_repository/' -e 's/^#
        android_ndk_repository/android_ndk_repository/' WORKSPACE
      - rm -f WORKSPACE.bak
    build_flags:
      - "--javabase=@openjdk9_linux_archive//:runtime"
...
ubuntu1804_javabase10:
    platform: ubuntu1804_nojava
    name: ":java: OpenJDK 10 Javabase"
    shell_commands:
      - sed -i.bak -e 's/^# android_sdk_repository/android_sdk_repository/' -e 's/^#
        android_ndk_repository/android_ndk_repository/' WORKSPACE
      - rm -f WORKSPACE.bak
    build_flags:
      - "--javabase=@openjdk10_linux_archive//:runtime"

The same ubuntu1804_nojava platform can have many tasks based on it, each with a different set of shell_commands and build_flags.

It's useful to perform benchmarking on each of these different environments.

bazelbuild / bazel-bench Goto Github PK

bazel-bench's People

Stargazers

Watchers

Forkers

bazel-bench's Issues

Recommend Projects

Recommend Topics

Recommend Org