See #306 (comment)

What happened?

❯ bazel build //appinventor/appengine:MakeWar
INFO: Analyzed target //appinventor/appengine:MakeWar (1 packages loaded, 656 targets configured).
INFO: Found 1 target...
ERROR: D:/github/pavi2410/appinventor/appinventor/appengine/BUILD.bazel:179:18: Copying files to directory failed: (Exit 1): copy_to_directory.exe failed: error executing command (from target //appinventor/appengine:MakeWar) external\copy_to_directory_windows_amd64\copy_to_directory.exe bazel-out/x64_windows-fastbuild/bin/appinventor/appengine/MakeWar_config.json
2023/01/15 21:20:08 open bazel-out/x64_windows-fastbuild/bin/appinventor/appengine/MakeWar/extra/ode\soycReport\splitPoints21.xml.gz: The system cannot find the path specified.
Target //appinventor/appengine:MakeWar failed to build
Use --verbose_failures to see the command lines of failed build steps.
INFO: Elapsed time: 0.662s, Critical Path: 0.18s
INFO: 3 processes: 3 internal.
FAILED: Build did NOT complete successfully

I'm using copy_to_directory rule to copy some build files to a different file under the bin directory. The path in the error above is the dest dir. Why does it attempt to open it? Where do I look for the problem?

Version

Development (host) and target OS/architectures: Windows 11 64-bit

Output of bazel --version: bazel 6.0.0

Version of the Aspect rules, or other relevant rules from your
WORKSPACE or MODULE.bazel file:

• 1.19.2 working
• 1.20.0 not working
• 1.21.3 not working

Language(s) and/or frameworks involved:
Java, GWT, AppEngine

How to reproduce

https://github.com/pavi2410/appinventor-sources/blob/0a96526fe572e7e50e6382dcb5ce2da7b55168d6/appinventor/appengine/BUILD.bazel#L184-L203

Any other information?

No response

Fund our work

• Sponsor our open source work by donating a bug bounty

What happened?

I don't know if this is going to happen, I was simply inspecting the generated .bat file, and saw that it uses cd instead of pushd.

It might be that cd is okay, but if for some reason the source location vs the build location are different, then cd won't work - e.g. it won't change the drive, while pushd would.

Somewhere here (but not the only place)

https://github.com/aspect-build/bazel-lib/blob/main/lib/private/write_source_file.bzl#L264

Version

Development (host) and target OS/architectures:

Output of bazel --version:

Version of the Aspect rules, or other relevant rules from your
WORKSPACE or MODULE.bazel file:

Language(s) and/or frameworks involved:

How to reproduce

No response

Any other information?

No response

Fund our work

• Sponsor our open source work by donating a bug bounty

What happened?

When using write_source_files with a generated file, writing to same folder, a diff_test clause introduced in b2955dc fails with

...external/aspect_bazel_lib/lib/private/diff_test.bzl", line 53, column 13, in _diff_test_impl
                fail(msg)
Error in fail: diff_test comparing the same file <generated file test/hello.txt>

I expect it the test to pass.

Here is the minimized repro:

WORKSPACE:

load("@bazel_tools//tools/build_defs/repo:http.bzl", "http_archive")

####
# aspect_bazel_lib
####

http_archive(
    name = "aspect_bazel_lib",
    sha256 = "695d319362b227725e4daa60d863b4d1969b167889902511f1fd3051cea1071f",
    strip_prefix = "bazel-lib-1.16.3",
    url = "https://github.com/aspect-build/bazel-lib/archive/refs/tags/v1.16.3.tar.gz",
)

load("@aspect_bazel_lib//lib:repositories.bzl", "aspect_bazel_lib_dependencies")

aspect_bazel_lib_dependencies()

test/BUILD:

load("@aspect_bazel_lib//lib:write_source_files.bzl", "write_source_files")

FILE = "hello.txt"

genrule(
    name = "file",
    outs = [FILE],
    cmd = """cat << EOF > $@
Hello world.
EOF"""
)

write_source_files(
    name = "test",
    files = {
        FILE: ":file",
    },
)

Version

Development (host) and target OS/architectures: macOS 12.6.1 (Intel)

Output of bazel --version: bazel 5.3.2

Version of the Aspect rules, or other relevant rules from your
WORKSPACE or MODULE.bazel file: v1.16.3

Language(s) and/or frameworks involved: Just bash.

How to reproduce

Given the above minimal repro, run the following: 


bazel run //test
bazel test //...

### Any other information?

_No response_

### Fund our work

- [ ] Sponsor our open source work by donating a [bug bounty](https://opencollective.com/aspect-build/)

Seems upstream has no support for arm64 and a very slow dev/release cycle

Im wondering if the best workaround for this is to provide the ability to compile jq where there is no binary available

Makes write_source_file more useful

Everywhere we describe installation with WORKSPACE, we should add the alternative section how to use with MODULE.bazel

Ported over from silo.

Related meeting notes.

Goal

Ensure that the versions stay in-sync.

Considerations

• Should we add the test to aspect-build/bazel-lib?
  • Seems like a good candidate as it needs to be shared amongst multiple repos.
  • Could be useful for thid-party repos that use bazel-lib.

Tasks

• Implement the test
• Roll out to repos
  • Roll out test to https://github.com/aspect-build/bazel-benchmarks
  • Roll out test to https://github.com/aspect-build/bazel-examples
  • Roll out test to https://github.com/aspect-build/bazel-lib
  • Roll out test to https://github.com/aspect-build/bazel-super-formatter
  • Roll out test to https://github.com/aspect-build/gcc-toolchain
  • Roll out test to https://github.com/aspect-build/rules_esbuild
  • Roll out test to https://github.com/aspect-build/rules_jasmine
  • Roll out test to https://github.com/aspect-build/rules_jest
  • Roll out test to https://github.com/aspect-build/rules_js
  • Roll out test to https://github.com/aspect-build/rules_rollup
  • Roll out test to https://github.com/aspect-build/rules_sol
  • Roll out test to https://github.com/aspect-build/rules_swc
  • Roll out test to https://github.com/aspect-build/rules_terser
  • Roll out test to https://github.com/aspect-build/rules_ts
  • Roll out test to https://github.com/aspect-build/rules_webpack
  • Roll out test to https://github.com/bazel-contrib/publish-to-bcr
  • Roll out test to https://github.com/bazel-contrib/rules_oci

I introduced a few todos in 15ec78e

• write_source_files
• diff_test
• Compile //:gazelle_bin

We should figure out what's the issue with these and enable them on remote-exec.

What is the current behavior?

Here's a simple use case where I would like to copy an external folder from our node_modules:
<npm_pkg_dir>/dist/fonts => <package_name>/material-design-icons-iconfont.

Currently, I seem to be unable to figure out which combinations of props will give me the desired result. Let's say I use the package [email protected] the prefix would need to be hardcoded:

Currently:

copy_to_directory(
    name = "material-design-icons",
    srcs = [
        "//:node_modules/material-design-icons-iconfont/dir",
    ],
    out = "material-design-icons-iconfont",
    replace_prefixes = {
        # copies an empty `node_modules` folder, not sure where this comes from
        "node_modules": "",
        # would expect to use $(location ...) here
        "node_modules/.aspect_rules_js/[email protected]/node_modules/material-design-icons-iconfont/dist/fonts": "",
    },
)

Expected:

copy_to_directory(
    name = "material-design-icons",
    srcs = [
        "//:node_modules/material-design-icons-iconfont/dir",
    ],
    out = "material-design-icons-iconfont",
    replace_prefixes = {
        "$(location //:node_modules/material-design-icons-iconfont/dir)/dist/fonts": "",
    },
)

# or

copy_to_directory(
    name = "material-design-icons",
    srcs = [
        "//:node_modules/material-design-icons-iconfont/dir",
    ],
    out = "material-design-icons-iconfont",
    root_paths = [
        "$(location //:node_modules/material-design-icons-iconfont/dir)",
    ],
    replace_prefixes = {
        "dist/fonts": "",
    },
)

Describe the feature

I would like to avoid having to compute the path to strip or hardcoding the value. Possibly there is a better way to approach this copy operation? I have tried using root_paths as well with no success, either way, it would be great to have a convenient way of performing this operation.

Fund our work

• Sponsor our open source work by donating a feature bounty

See https://bazelbuild.slack.com/archives/CA31HN1T3/p1665178148461539.

A copy_to_directory on 2500 files is taking on the order of 10s. Investigate what's causing the slowness and see if there are any optimizations that can be made.

See #380 for more context.

Possibly related to bazelbuild/bazel#5566 & bazelbuild/bazel#17162.

@gregmagolan has some code at RH for this

https://bazelbuild.slack.com/archives/CA31HN1T3/p1648811785417179?thread_ts=1648734405.972179&cid=CA31HN1T3

Silently ignoring conflicts and not overwriting could be a source of tricky to find bugs. Both ignore and overwrite behaviours should be explicitly declared.

fyi @devversion

What is the current behavior?

Just curious

Describe the feature

What it looks like

Fund our work

• Sponsor our open source work by donating a feature bounty

Thanks for a very neat lib!

I was wondering if it would be possible to add an optional allow_symlink = {True,False} to copy_to_directory similar to how skylib have for copy_file:
https://github.com/bazelbuild/bazel-skylib/blob/main/tests/copy_file/BUILD#L123

Useful for things like producing a package.json file that has the version field set from git tags.

genrule has an (undocumented) stamp attribute that gives the command access to bazel-out/stable-status.txt and bazel-out/volatile-status.txt.

run_binary should have some way to access workspace status values as well, but in a more ergonomic way. That is, don't make the binary itself responsible for finding and parsing the .txt files -- build that into run_binary. Maybe stamp = True puts each workspace status key-value into the environment? Or the stamp value could be a list of workspace status keys to expose as environment variables.

What happened?

We see that at least the following checksum is changed:

https://github.com/aspect-build/bazel-lib/archive/refs/tags/v1.10.0.tar.gz

➜ Downloads shasum -a 256 ./bazel-lib-1.10.0.tar.gz
5d3d968857e6e443cbbd3e97b97f46ab46b9a24853410b34538b8b683b648302 ./bazel-lib-1.10.0.tar.gz

The expected checksum is mentioned here:

https://github.com/aspect-build/bazel-lib/releases/tag/v1.10.0

Version

Development (host) and target OS/architectures:

Output of bazel --version:

Version of the Aspect rules, or other relevant rules from your
WORKSPACE or MODULE.bazel file:

Language(s) and/or frameworks involved:

How to reproduce

No response

Any other information?

No response

Fund our work

• Sponsor our open source work by donating a bug bounty

This issue lists Renovate updates and detected dependencies. Read the Dependency Dashboard docs to learn more.

Rate-Limited

These updates are currently rate-limited. Click on a checkbox below to force their creation now.

• chore(deps): update dependency bazel to v7
• chore(deps): update dependency bazel_features to v1
• 🔐 Create all rate-limited PRs at once 🔐

Open

These updates have all been created already. Click a checkbox below to force a retry/rebase of any.

• fix(deps): update golang.org/x/exp digest to 814bf88
• chore(deps): update cimg/base docker tag to v2023.12
• chore(deps): update dependency bazel to v6.5.0
• chore(deps): update dependency buildifier_prebuilt to v6.4.0
• chore(deps): update dependency gazelle to v0.35.0
• chore(deps): update dependency io_bazel_rules_go to v0.46.0
• chore(deps): update dependency rules_go to v0.46.0
• chore(deps): update actions/cache action to v4
• chore(deps): update actions/github-script action to v7
• chore(deps): update cimg/base docker tag to v2024
• Click on this checkbox to rebase all open PRs at once

Ignored or Blocked

These are blocked by an existing closed PR and will not be recreated unless you click a checkbox below.

• chore(deps): update dependency io_bazel_stardoc to v0.6.2
• chore(deps): update dependency stardoc to v0.6.2

Detected dependencies

• Check this box to trigger a request for Renovate to run again on this repository

What happened?

More context: https://bazelbuild.slack.com/archives/CEZUUKQ6P/p1676010753876439

npm_package() from @aspect_rules_js doesn't include sources from its own workspace when built in an external workspace. Instead, you get an empty directory (more context). It seems that the include_external_repositories filter of copy_to_directory is a little overzealous and automatically includes sources from the __main__ workspace, not the workspace containing the target. I think the correct behavior here is to automatically include any files in the same workspace as the npm_package() target, and I think that also applies to the underlying copy_to_directory_bin_action().

I think the right fix is to just make copy_to_directory_bin_action() add a --workspace_name flag containing the name of the workspace for the current target and pass that into the copy_to_directory tool. Then the tool should always include files within that workspace regardless of include_external_repositories (after applying other filters ofc).

I have a failing test case in dgp1130@33e3ed9, but haven't started on a proper fix yet.

Version

Reproducible in latest version of @aspect_bazel_lib (1.24.2).

How to reproduce

# my_wksp/BUILD.bazel

npm_package(name = "pkg", srcs = ["file.txt"])

Then link that from another workspace:

# my_other_wksp/WORKSPACE.bazel

local_repository(
    name = "my_wksp",
    path = "../my_wksp",
)

Then build:

(cd my_other_wksp/ && bazel build @my_wksp//:pkg)

It will output an empty directory.

Any other information?

My current workaround is to add include_external_repositories = ["my_wksp"] to npm_package(), but it's not ideal and requires that any downstream workspaces refer to this package's workspace as @my_wksp when they should be free to choose any name.

Fund our work

• Sponsor our open source work by donating a bug bounty

What is the current behavior?

In Bazel by default, executable binaries have a built-in args attribute, so bazel run will automatically use those arguments. However, the args are dropped if the binary is invoked directly via bazel-bin/... or in the runfiles of another binary. Quoting from https://bazel.build/reference/be/common-definitions#common-attributes-binaries:

args - List of strings; optional; subject to $(location) and "Make variable" substitution, and Bourne shell tokenization; nonconfigurable

Command line arguments that Bazel will pass to the target when it is executed either by the run command or as a test. These arguments are passed before the ones that are specified on the bazel run or bazel test command line.

NOTE: The arguments are not passed when you run the target outside of Bazel (for example, by manually executing the binary in bazel-bin/).

For some rules (such as js_run_devserver()) executing a binary from runfiles is required and args is effectively useless, since you can't guarantee it will be present. It would be great to have an easy way to ensure hard-coded arguments in the BUILD file are always present, regardless of how the binary is executed.

Describe the feature

I propose a new baked_binary() rule which takes a baked_args attribute and generates a wrapper binary which hard-codes those arguments, with any subsequent arguments appended after it. This means arguments are "baked" into the binary and will be retained regardless of how the binary is executed. This would look like:

load("@aspect_bazel_lib//lib:baked_binary.bzl", "baked_binary")

sh_binary(
    name = "server",
    srcs = ["server.sh"],
    data = [":site"],
    # Can't rely on `args`, because it won't always be there.
    # args = ["path/to/site/"],
)

baked_binary(
    name = "baked_server",
    binary = ":server",
    # Bake the args so they are always present.
    baked_args = ["path/to/site"],
)

# When this runs `./baked_server --port 1234`, `path/to/site` is already included
# and does not need to be repeated.
sh_test(
    name = "test",
    srcs = ["test.sh"],
    data = [":baked_server"],
)

I made my own implementation of this in dgp1130/rules_prerender@c0065dc#diff-2f9d8f02c056b3df3e9ff8b6fb76b84689e08c39773d3661dcfcd482cf13a267, though it isn't great since it requires Bash, doesn't support Windows, and needed some hacks to be compatible with js_run_devserver() (chdir messes up Bash runfiles).

Fund our work

• Sponsor our open source work by donating a feature bounty

Ported over from silo

Add the following along with their tests:

warn() {
  local msg="${1:-}"
  shift 1
  while (("$#")); do
    msg="${msg:-}"$'\n'"${1}"
    shift 1
  done
  echo >&2 "${msg}"
}

# Echos the provided message to stderr and exits with an error (1).
fail() {
  warn "$@"
  exit 1
}

# Print an error message and dump the usage/help for the utility.
# This function expects a get_usage function to be defined.
usage_error() {
  local msg="${1:-}"
  cmd=(fail)
  [[ -z "${msg:-}" ]] || cmd+=("${msg}" "")
  cmd+=("$(get_usage)")
  "${cmd[@]}"
}

show_usage() {
  get_usage
  exit 0
}

What is the current behavior?

At some point the performance of --experimental_remote_merkle_tree_cache regressed, so we excluded it from our bazelrc presets in #390.

More info:
bazelbuild/bazel#13879
bazelbuild/bazel#16054

Describe the feature

Add the presets back once performance regressions are fixed.

Fund our work

• Sponsor our open source work by donating a feature bounty

ERROR: /Users/alexeagle/Projects/rules_js/examples/npm_deps/BUILD.bazel:123:7: in directory_path rule //examples/npm_deps:actual14__entry_point:
Traceback (most recent call last):
	File "/private/var/tmp/_bazel_alexeagle/deecfabe94c39fb88a37966b779aac74/external/aspect_bazel_lib/lib/private/directory_path.bzl", line 17, column 13, in _directory_path
		fail("directory attribute must be created with Bazel declare_directory (TreeArtifact)")

is what you get if your entry point doesn't exist at all. the error message should indicate that possibility.

What happened?

Hi,

This is a follow-up to #258. The analysis-time performance issues were addressed by 350408b (thank you!), but the execution-time performance is still problematic. I wanted to raise this to track that.

The problem is that the copy_to_directory is generally slow and particularly slow on MacOS. The wall time on our build went up after moving from rules_nodejs to rules_js, mainly because of copy_to_directory. It seems that any non-trivial folder (i.e. a library) is going to block for anything from 2s to 10s on this action (and more on MacOS). This seems problematic for rules_js users in general as copy_to_directory is in the critical path of any change in a linked npm_package, which seems like a path that needs to be as fast as possible for the UX around a developer making local changes and ibazel rebuilding those to work.

copy_to_directory generates a bash script which is invoked through ctx.actions.run_shell. The script, in my case looks like:

if [[ ! -e "bazel-out/darwin_arm64-fastbuild/bin/libs/generic/src/lib/user-pref-service/user-pref-service.js" ]]; then echo "file 'bazel-out/darwin_arm64-fastbuild/bin/libs/generic/src/lib/user-pref-service/user-pref-service.js' does not exist"; exit 1; fi
if [[ -f "bazel-out/darwin_arm64-fastbuild/bin/libs/generic/src/lib/user-pref-service/user-pref-service.js" ]]; then
    mkdir -p "bazel-out/darwin_arm64-fastbuild/bin/libs/generic/generic/src/lib/user-pref-service"
    cp -v -n "bazel-out/darwin_arm64-fastbuild/bin/libs/generic/src/lib/user-pref-service/user-pref-service.js" "bazel-out/darwin_arm64-fastbuild/bin/libs/generic/generic/src/lib/user-pref-service/user-pref-service.js" >> "$OUT_CAPTURE" 2>>"$OUT_CAPTURE"
else
    if [[ -d "bazel-out/darwin_arm64-fastbuild/bin/libs/generic/generic/src/lib/user-pref-service/user-pref-service.js" ]]; then
        # When running outside the sandbox, then an earlier copy will create the dst folder
        # with nested read-only folders, so our copy operation will fail to write there.
        # Make sure the output folders are writeable.
        find "bazel-out/darwin_arm64-fastbuild/bin/libs/generic/generic/src/lib/user-pref-service/user-pref-service.js" -type d -print0 | xargs -0 chmod a+w
    fi
    mkdir -p "bazel-out/darwin_arm64-fastbuild/bin/libs/generic/generic/src/lib/user-pref-service/user-pref-service.js"
    cp -v -R -n "bazel-out/darwin_arm64-fastbuild/bin/libs/generic/src/lib/user-pref-service/user-pref-service.js/." "bazel-out/darwin_arm64-fastbuild/bin/libs/generic/generic/src/lib/user-pref-service/user-pref-service.js" >> "$OUT_CAPTURE" 2>>"$OUT_CAPTURE"
fi


if [[ ! -e "bazel-out/darwin_arm64-fastbuild/bin/libs/generic/src/lib/user-service/index.js" ]]; then echo "file 'bazel-out/darwin_arm64-fastbuild/bin/libs/generic/src/lib/user-service/index.js' does not exist"; exit 1; fi
if [[ -f "bazel-out/darwin_arm64-fastbuild/bin/libs/generic/src/lib/user-service/index.js" ]]; then
    mkdir -p "bazel-out/darwin_arm64-fastbuild/bin/libs/generic/generic/src/lib/user-service"
    cp -v -n "bazel-out/darwin_arm64-fastbuild/bin/libs/generic/src/lib/user-service/index.js" "bazel-out/darwin_arm64-fastbuild/bin/libs/generic/generic/src/lib/user-service/index.js" >> "$OUT_CAPTURE" 2>>"$OUT_CAPTURE"
else
    if [[ -d "bazel-out/darwin_arm64-fastbuild/bin/libs/generic/generic/src/lib/user-service/index.js" ]]; then
        # When running outside the sandbox, then an earlier copy will create the dst folder
        # with nested read-only folders, so our copy operation will fail to write there.
        # Make sure the output folders are writeable.
        find "bazel-out/darwin_arm64-fastbuild/bin/libs/generic/generic/src/lib/user-service/index.js" -type d -print0 | xargs -0 chmod a+w
    fi
    mkdir -p "bazel-out/darwin_arm64-fastbuild/bin/libs/generic/generic/src/lib/user-service/index.js"
    cp -v -R -n "bazel-out/darwin_arm64-fastbuild/bin/libs/generic/src/lib/user-service/index.js/." "bazel-out/darwin_arm64-fastbuild/bin/libs/generic/generic/src/lib/user-service/index.js" >> "$OUT_CAPTURE" 2>>"$OUT_CAPTURE"
fi


if [[ ! -e "bazel-out/darwin_arm64-fastbuild/bin/libs/generic/src/lib/user-service/mock-user-service-providers.js" ]]; then echo "file 'bazel-out/darwin_arm64-fastbuild/bin/libs/generic/src/lib/user-service/mock-user-service-providers.js' does not exist"; exit 1; fi
if [[ -f "bazel-out/darwin_arm64-fastbuild/bin/libs/generic/src/lib/user-service/mock-user-service-providers.js" ]]; then
    mkdir -p "bazel-out/darwin_arm64-fastbuild/bin/libs/generic/generic/src/lib/user-service"
    cp -v -n "bazel-out/darwin_arm64-fastbuild/bin/libs/generic/src/lib/user-service/mock-user-service-providers.js" "bazel-out/darwin_arm64-fastbuild/bin/libs/generic/generic/src/lib/user-service/mock-user-service-providers.js" >> "$OUT_CAPTURE" 2>>"$OUT_CAPTURE"
else
    if [[ -d "bazel-out/darwin_arm64-fastbuild/bin/libs/generic/generic/src/lib/user-service/mock-user-service-providers.js" ]]; then
        # When running outside the sandbox, then an earlier copy will create the dst folder
        # with nested read-only folders, so our copy operation will fail to write there.
        # Make sure the output folders are writeable.
        find "bazel-out/darwin_arm64-fastbuild/bin/libs/generic/generic/src/lib/user-service/mock-user-service-providers.js" -type d -print0 | xargs -0 chmod a+w
    fi
    mkdir -p "bazel-out/darwin_arm64-fastbuild/bin/libs/generic/generic/src/lib/user-service/mock-user-service-providers.js"
    cp -v -R -n "bazel-out/darwin_arm64-fastbuild/bin/libs/generic/src/lib/user-service/mock-user-service-providers.js/." "bazel-out/darwin_arm64-fastbuild/bin/libs/generic/generic/src/lib/user-service/mock-user-service-providers.js" >> "$OUT_CAPTURE" 2>>"$OUT_CAPTURE"
fi

You can see how dong all these checks and bash program invocations serially, and each blocking, and spawning a subprocesses, would get very slow for non-trivial directories. I'm fairly confident that if, rather than using bash, copy_to_directory called out to a go or js program that used non-blocking I/O and could do the work in parallel (not to mention not spawning subprocesses), the performance would be on a different level.

Of course, another question is whether any of this logic can be short-circuited and circumvented. It looks like some of the logic is long the lines of "if the src is a directory, do this, if the src is a file, do that", but doesn't bazel know at analysis time whether these inputs are files or directories (at least assuming no TreeArtifact inputs, which itself feels like an edge case to me)?

I'm keen to help with the implementation of this. It feels like the first step is just figuring out what direction(s) to take.

Version

Development (host) and target OS/architectures:

Output of bazel --version:
6.0.0
Version of the Aspect rules, or other relevant rules from your
WORKSPACE or MODULE.bazel file:
1.18.0

Language(s) and/or frameworks involved:

How to reproduce

No response

Any other information?

No response

Fund our work

• Sponsor our open source work by donating a bug bounty

Coming here from this SO answer.

I am building a template/POC repository for my future C++-centred projects:
https://github.com/rh-ogorod/diamond-dojo/tree/main

The current attempt is to use Bazel for building/testing/deployment coordination - locally and at CI services. My main requirements are: automatic compile_commands.json generation - usable with clang-family dev tools; and ruining external build systems for dependency projects when those projects are not supporting Bazel officially.

compile_commands.json solution has been prototyped with customised Bazel rule from GRAIL and my unbox.js script (bazelbuild-rules-compdb - it is used in the above diamond-dojo project).

Running external build systems with Bazel is difficult. So far I found three possible strategies (excluding writing custom rules):

1. Building external deps before running Bazel and then capturing external build artefacts into Bazel via cc_library. This strategy has been the most practical.
2. Using rules_foreign_cc.
3. Using external build genrule as a label in data attribute of cc_library target.

"pre-bazel" building strategy is not perfect as it is not hermetic (depending on external build system) and somewhat defeats the usage of Bazel.
rules_foreign_cc strategy although covers many practical cases, is not sufficiently generic - even some cmake and make projects are not usable (e.g. with internal custom scripts or unconventional dependencies management). Also, when Bazel-managed external tool uses multi-threading with its own memory management, the host system may easily run out of resources.
genrule strategy requires listing of all its artefact files in corresponding cc_library hdrs and srcs attributes that is time-consuming and not practical when experimenting/playing with new libraries.

In the above diamond-dojo repository, boost's b2 is used as an external build system model (I am aware of nelhage Bazel boost rules). Is it possible to use aspect's run_binary to replace genrule when running arbitrary build systems without the need to list all artefact files in corresponding cc_library hdrs and srcs attributes? Or "somehow" couple run_binary output directly with cc_library/cc_binary? Something like:

https://github.com/rh-ogorod/diamond-dojo/blob/340f09f5b36a7ea04e865b5d19bc08b86b1eef46/external/boost/BUILD.bazel#L196 (not working!)

cp: bazel-out/darwin-fastbuild/bin/web/design/esm/util/typeSafeObjectKeys.d.ts: Permission denied

bisected to find d59ca60 is the cause.

Now that we are outside the sandbox, theory is that the first copy creates a read-only folder and then the second copy is unable to write files in that folder.

I'm using copy_file_action in a custom publishing rule and I've got an example of how it's used here:
https://github.com/purkhusid/rules_dotnet/blob/695ffff8b2dd4f4aae30372f0f3e094be52721dc/examples/publish_binary/BUILD#L21

The issue is that 2 targets try to create the exact same copy script on Windows because the inputs and outputs are the exact same files.

The culprit is this line here:

The comment above mentions that it's intentional to not have the label name in the file name but I'm not 100% understanding the reason for it. Would be be open up to a change that adds the label name to the bat file name?

jq itself will handle non json files in raw mode, but the macro doesnt

handling raw files would be particularly helpful when dealing with genquery output as bazel query doesnt support formatting results as json.

one workaround is to name the source files with a .json suffix but this means you have artefacts that are not valid json with json names

assignee: dylan</s pan>

Ported over from silo.

Related meeting notes.

Tasks

• Roll out config to https://github.com/aspect-build/bazel-benchmarks
• Roll out config to https://github.com/aspect-build/bazel-examples
• Roll out config to https://github.com/aspect-build/bazel-lib
• Roll out config to https://github.com/aspect-build/bazel-super-formatter
• Roll out config to https://github.com/aspect-build/docs
• Roll out config to https://github.com/aspect-build/gcc-toolchain
• Roll out config to https://github.com/aspect-build/rules_esbuild
• Roll out config to https://github.com/aspect-build/rules_jasmine
• Roll out config to https://github.com/aspect-build/rules_jest
• Roll out config to https://github.com/aspect-build/rules_js
• Roll out config to https://github.com/aspect-build/rules_rollup
• Roll out config to https://github.com/aspect-build/rules_sol
• Roll out config to https://github.com/aspect-build/rules_swc
• Roll out config to https://github.com/aspect-build/rules_terser
• Roll out config to https://github.com/aspect-build/rules_ts
• Roll out config to https://github.com/aspect-build/rules_webpack
• Roll out config to https://github.com/bazel-contrib/publish-to-bcr
• Roll out config to https://github.com/bazel-contrib/rules_oci

ERROR: /home/alexeagle/Projects/buildbuddy/proto/BUILD:907:17: in _copy_to_bin rule //proto:_grpc_code_ts_proto_pbjs_copy_srcs_to_bin: 
Traceback (most recent call last):
	File "/shared/cache/bazel/user_base/ed549a90e257f6c571ada9350158b663/external/aspect_bazel_lib/lib/private/copy_to_bin.bzl", line 101, column 38, in _impl
		files = copy_files_to_bin_actions(ctx, ctx.files.srcs, is_windows = is_windows)
	File "/shared/cache/bazel/user_base/ed549a90e257f6c571ada9350158b663/external/aspect_bazel_lib/lib/private/copy_to_bin.bzl", line 96, column 36, in copy_files_to_bin_actions
		return [copy_file_to_bin_action(ctx, file, is_windows = is_windows) for file in files]
	File "/shared/cache/bazel/user_base/ed549a90e257f6c571ada9350158b663/external/aspect_bazel_lib/lib/private/copy_to_bin.bzl", line 40, column 13, in copy_file_to_bin_action
		fail(
Error in fail: 
Expected to find file code.proto in //proto, but instead it is in go_googleapis//google/rpc.

To use copy_to_bin, either move code.proto to //proto, or move the copy_to_bin
target to code.proto's package using:
    
    buildozer 'new copy_to_bin code' go_googleapis//google/rpc:__pkg__
    buildozer 'add srcs code.proto' go_googleapis//google/rpc:code
    buildozer 'new_load @aspect_bazel_lib//lib:copy_to_bin.bzl copy_to_bin' go_googleapis//google/rpc:__pkg__

ERROR: /home/alexeagle/Projects/buildbuddy/proto/BUILD:907:17: Analysis of target '//proto:_grpc_code_ts_proto_pbjs_copy_srcs_to_bin' failed

but the def'n was

ts_proto_library(
    name = "grpc_code_ts_proto",
    proto = "@go_googleapis//google/rpc:code_proto",
)

so that buildozer command is'nt helpful

The last official jq release, and the one these rules are currently using, was released in 2018, it sounds like the project is still alive jqlang/jq#2305 but changes aren't happening quickly. There are some upstream issues around providing binaries for arm64 linux jqlang/jq#1655, but so far there aren't any hosted binaries we can easily download for this.

One option to solving this would be to build jq with bazel instead of vendoring binaries. A quick google came up with these BUILD files https://github.com/attilaolah/bazel-tools/blob/master/jq.bazel which I have not tested. Another option would just be to build an arm64 binary from the last release tag and host it somewhere (like on the releases page of this repo), just to have a usable download.

I have the following rule in a package:

ts_project(
    name = "gen_ts",
    srcs = ["gen.ts"],
    extends = "//:tsconfig_base",
    tsconfig = {
        "compilerOptions": {
            "declaration": False,
        },
    },
)

With tsconfig_base being defined in the root package:

ts_config(
    name = "tsconfig_base",
    src = "tsconfig.base.json",
)

Building //gen:gen_ts (bazel-out/darwin-opt-exec-2B5CBBC6/bin/external/npm_typescript/tsc.sh --project gen/tsconfig_meta_gen_ts.json --outDir gen --rootDir gen) fails, because tsconfig_meta_gen_ts.json contains an incorrect path in extends:

{
    "compilerOptions": { },
    "extends":"./",
    "files": ["./gen.ts"]
}

It should be ../tsconfig.base.json instead.

users can write

update_source_files(name = "foo", additional_update_targets = ["//some/arbitrary-target"])

and the error is pretty obscure. We should ensure that the attribute is declared such that only compatible targets can be supplied.

• remove the override_local_config_platform attribute entirely
• remove deprecated & unused is_windows parameter
• remove deprecated exclude_prefix from copy_to_directory #591
• remove the legacy copy_to_directory_action helper
• remove output_dir in expand_variables
• what to do with the "Always register toolchains" bits for WORKSPACE users?
• remove local_config_platform rule (principled fix landed in Bazel 6 so this is no longer needed)

Some of the above items can be located by searching for "TODO(2.0)" in the sources

From bazel-contrib/SIG-rules-authors#11 I learned that it's overkill for us to build a distribution artifact.

The only reasons to bother are:

• users can bazel query @aspect_bazel_lib//... and not trip on our load() statement from stardoc. But users ~never do that.
• the distribution is smaller since docs and tests aren't included. But we're talking dozens of kilobytes which isn't worth the complexity.

All my remaining reasons for it turned out to be wrong. Trying the simplification here.

Proof of what we need to do when generating a workspace snippet:

alexeagle@system76-pc:~/Projects/bazel-lib$ git archive --format=tar --prefix=bazel-lib-0.2.7/ v0.2.7 | gzip > /tmp/tarball.tgz; sha256sum /tmp/tarball.tgz
ce379ba06f861f5cab773862fc8eaeff3a531e744e2076165f4bbef4cb7ac046  /tmp/tarball.tgz

alexeagle@system76-pc:~/Projects/bazel-lib$ curl -L https://github.com/aspect-build/bazel-lib/archive/v0.2.7.tar.gz|sha256sum -
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100   130  100   130    0     0    431      0 --:--:-- --:--:-- --:--:--   430
100 29335    0 29335    0     0  43588      0 --:--:-- --:--:-- --:--:-- 43588
ce379ba06f861f5cab773862fc8eaeff3a531e744e2076165f4bbef4cb7ac046  -

/cc @cgrindel

it probably should, along with visibility and other common attrs