My skylark's pretty rusty after a year on the outside, but if I'm following correctly:

The pom_file rule crawls the targets' dependencies, using the maven_coordinates=[XXX] tag to identify things that came from maven & can/should be included in {generated_bzl_deps}

That works fine for the deps coming from here since workspace_defs.bzl is turning maven artifact strings into java_import_external rules w/ that tag added.

But anything defined as a native maven_jar is missed; it's got that artifactid directly on it, but not as that particular tag. And despite the claim here that all rules support tags, maven_jar doesn't appear to support them, so I can't even just tack it on.

I'm using bazel-deps to generate all the rules, so it looks like I could potentially hack that to stick the tags on the java_library rules it tosses under third_party, but it feels like if I knew what I was doing, it shouldn't be too bad to automagically get that off the underlying maven_jar rule.

javadoc_library will break when the --incompatible_disallow_legacy_java_provider switch flips. (See
bazelbuild/bazel#7598.)

version:
Build label: 0.19.1
Build target: bazel-out/k8-opt/bin/src/main/java/com/google/devtools/build/lib/bazel/BazelServer_deploy.jar
Build time: Mon Nov 12 15:32:30 2018 (1542036750)
Build timestamp: 1542036750
Build timestamp as int: 1542036750

It complains this label no longer exists: @local_jdk//:jdk-default. I think there's a different one we should be using here instead.

I just came across https://github.com/square/bazel_maven_repository. It sounds like it may have some overlap with bazel_common. We should maybe look at that someday.

Bazel itself uses https://github.com/google/jarjar. It's actually distributed with Bazel as @bazel_tools//third_party/java/jarjar:jarjar_import, although you may not want to depend on those labels for bazel-common.

Issue

When jarjar_runner.sh processes a jar that contains multiple files that differ only in the case of their filename, they will be incorrectly flagged as duplicates when running on a case-insensitive filesystem like macOS's default APFS root volume.

BUILD.bazel:

jarjar_library(
        name="test_shaded",
        rules=[],
        jars=["test.jar"]
)

Steps to reproduce:

Download test.zip (just a zip with three empty files: test, TEST, and build-data.properties)

% mv test.zip test.jar
% bazel build :test_shaded
INFO: Analyzed target //:test_shaded (0 packages loaded, 0 targets configured).
INFO: Found 1 target...

ERROR: /Users/*snip*/BUILD.bazel:11:15: Action test_shaded.jar failed: (Exit 1): bash failed: error executing command /bin/bash -c ... (remaining 1 argument(s) skipped)

Use --sandbox_debug to see verbose messages from the sandbox
Error: duplicate files in merged jar: test~
Target //:test_shaded failed to build
Use --verbose_failures to see the command lines of failed build steps.
INFO: Elapsed time: 0.191s, Critical Path: 0.09s
INFO: 2 processes: 2 internal.
FAILED: Build did NOT complete successfully

Possible solutions

One solution I have tested is to modify jarjar_runner.sh to extract the jars to a temporary sparse disk image created with a case-sensitive filesystem mounted to a tmp directory. However, I've found that this approach is "leaky", i.e. if you ctrl-C the bazel build before it finishes, the user will have a volume mounted that they then will have to manually unmount. I tried to solve this by adding a trap command to jarjar_runner.sh, however, it never seemed to get triggered, probably due to some subtlety of being called from process-wrapper.

Another solution would be to replace jarjar_runner.sh with a Python script that uses the stdlib zipfile library to interact with the jar files to avoid interacting with the native FS.

I am trying to use generate Javadoc from auto-generated Java sources (e.g. Java bindings generated from WSDL). My Java source generator produces a tree artifact. However, I cannot provide tree artifacts to javadoc_library, because javadoc_library simply passes srcs directly to the javadoc command-line. As a result, the action fails with the following:

javadoc: error - Illegal package name: "bazel-out/k8-dbg-obj/bin/path/to/my/treeartifact

Error message:

Error: duplicate files in merged jar: org/xerial/snappy/VERSION~

Found the following code in jarjar_runner.sh:

duplicate_files="$(find * -type f -regex '.*~[0-9]*$')"
if [[ -n "${duplicate_files}" ]]; then
  echo "Error: duplicate files in merged jar: ${duplicate_files}"
  exit 1
fi

What is it for?

FYI
The contents of org/xerial/snappy/VERSION:

SNAPPY_VERSION=1.1.8
BITSHUFFLE_VERSION=0.3.2

and the content of org/xerial/snappy/VERSION~:

SNAPPY_VERSION=1.1.7
BITSHUFFLE_VERSION=0.3.2

This issue was automatically created by Allstar.

Security Policy Violation
Project is out of compliance with Binary Artifacts policy: binaries present in source code

Rule Description
Binary Artifacts are an increased security risk in your repository. Binary artifacts cannot be reviewed, allowing the introduction of possibly obsolete or maliciously subverted executables. For more information see the Security Scorecards Documentation for Binary Artifacts.

Remediation Steps
To remediate, remove the generated executable artifacts from the repository.

Artifacts Found

• tools/jarjar/test-library1.jar
• tools/jarjar/test-library2.jar

Additional Information
This policy is drawn from Security Scorecards, which is a tool that scores a project's adherence to security best practices. You may wish to run a Scorecards scan directly on this repository for more details.

Allstar has been installed on all Google managed GitHub orgs. Policies are gradually being rolled out and enforced by the GOSST and OSPO teams. Learn more at http://go/allstar

This issue will auto resolve when the policy is in compliance.

Issue created by Allstar. See https://github.com/ossf/allstar/ for more information. For questions specific to the repository, please contact the owner or maintainer.

So annotations only exports:

"@org_checkerframework_checker_compat_qual//jar",

It should probably also expose:

 "@org_checkerframework_checker_qual//jar",

compat_qual doesn't have what I would expect like Nullable etc.

https://github.com/bazelbuild/continuous-integration/blob/master/buildkite/README.md

Mentioned in #87 (comment)

We could do this for Dagger and Flogger, too.

I'm trying to use Dagger-gRPC which is, as far as I understand, built with dependencies coming from this repository - specifically grpc-netty (dependency defined here).

The issue is, the grpc-netty version pulled in by bazel-common is quite old: it's the 1.2.0, released in early 2017. It pulls in io.netty:netty-codec-http2:4.1.8.Final, which has a bug (described here, fixed in early 2017 too).

Would it be possible to update the gRPC version pulled in by this repo?

https://github.com/googleapis/googleapis

Another widely useful proto dependency.

From bazelbuild/continuous-integration#1006, I get the impression (but haven't confirmed) that forthcoming versions of Bazel are going to require that any registered Android toolchains are present, even if they're not used. Flogger (and probably Inject-Common and perhaps other projects?) don't use Android, so it would be nice for people who build them not to need to download one.

I'm not sure of the best way to do this, but maybe support google_common_workspace_rules(android=False) (or android_version=None if we want to allow projects to not just disable support but also change the version?)? I'm not sure if this would put us on a road to supporting a zillion configuration options.

Would be good to update the version of protobuf in here to the latest:
https://github.com/protocolbuffers/protobuf/releases/tag/v3.6.1

Currently getting compilation errors related to a very old deprecation being fully removed:

The value 'REPOSITORY_NAME' has been removed in favor of 'repository_name()', please use the latter (https://docs.bazel.build/versions/master/skylark/lib/native.html#repository_name). You can temporarily allow the old name by using --incompatible_package_name_is_a_function=false

Bazel version:

Build label: 0.21.0
Build target: bazel-out/k8-opt/bin/src/main/java/com/google/devtools/build/lib/bazel/BazelServer_deploy.jar
Build time: Wed Dec 19 12:58:44 2018 (1545224324)
Build timestamp: 1545224324
Build timestamp as int: 1545224324

https://github.com/grpc/grpc-proto

This would be hugely useful to be used in conjunction with grpc_java

Issue

When jarjar_runner.sh processes a jar that contains a file ending in ~, it fails.

BUILD.bazel:

jarjar_library(
        name="test_shaded",
        rules=[],
        jars=["test2.jar"]
)

Steps to reproduce:

mkdir jar
cd jar
touch test~ build-data.properties
jar cf test2.jar *
bazel build :test_shaded

INFO: Analyzed target //:test_shaded (0 packages loaded, 0 targets configured).
INFO: Found 1 target...

ERROR: /Users/*snip*/BUILD.bazel:11:15: Action test_shaded.jar failed: (Exit 1): bash failed: error executing command /bin/bash -c ... (remaining 1 argument(s) skipped)

Use --sandbox_debug to see verbose messages from the sandbox
Error: duplicate files in merged jar: test~
Target //:test_shaded failed to build
Use --verbose_failures to see the command lines of failed build steps.
INFO: Elapsed time: 0.191s, Critical Path: 0.09s
INFO: 2 processes: 2 internal.
FAILED: Build did NOT complete successfully

Possible solution

It should be simple to use comm to filter out any files from duplicate_files that are present in any of the source jars.

Per bazel-contrib/SIG-rules-authors#11, the checksum of repository code download archives (https://github.com/google/bazel-common/archive/_COMMIT_.zip) are not stable.

In fact, the checksum was changed earlier today as seen in the above issue, though that change was rolled back. Please consider releasing an archive under /archive/refs/tags/$tag, which its SHA256 value is promised to be stable.

This is Google's recommended library for multiple software platforms, would be a good idea to offer it easily to Bazel projects.

https://github.com/google/tink

Could be a worthwhile addition to this since it requires some annotation processor setup as well much like Auto.

I tried to do this:

jarjar_library(
    name = "shaded_" + name + "_deploy",
    rules = ["shade_rules_hadoop"],
)

But got an error:

find: 'META-INF': No such file or directory
rm: cannot remove 'build-data.properties*': No such file or directory
rm: cannot remove 'META-INF/MANIFEST.MF*': No such file or directory
find: '*': No such file or directory

• : no such file or directory
  Syntax error: Error on line 1: [shade_rules_hadoop]
  Exception in thread "main" java.lang.IllegalArgumentException: Error on line 1: [shade_rules_hadoop]

Do you have any example of running jarjar with a rules file?

Best,

The jarjar.bzl script assumes that the value of {java_home} is always relative, (eg. external/local_jdk) and therefore it needs to be used in conjunction with pwd in order to get the full path to Java's location. See the following line taken from jarjar.bzl @ master:

JAVA_HOME=$(pwd)/{java_home} # this is used outside of the root

This works in most cases, however, we've encountered a scenario where it may be an absolute path, such as when you want to specify your own toolchain for Remote Build Execution.

This issue hits us when we're trying to run a test on Google Cloud's Remote Build Execution. The test will fail with the following error:

$ bazel --bazelrc=.bazelrc test //server/test/server/deduplicator:rocks-db-queue-test --config=remote --remote_instance_name=projects/<our-rbe-project-id>/instances/default_instance --project_id=<our-rbe-project-id> --verbose_failures
INFO: Invocation ID: abae541f-e00d-4122-8849-769bffaa40fb
DEBUG: /root/.cache/bazel/_bazel_root/8a4e683c96774f55a46951715144dd49/external/bazel_skylib/lib.bzl:30:1: WARNING: lib.bzl is deprecated and will go away in the future, please directly load the bzl file(s) of the module(s) needed as it is more efficient.
INFO: Analysed target //server/test/server/deduplicator:rocks-db-queue-test (0 packages loaded, 0 targets configured).
INFO: Found 1 test target...
INFO: From SkylarkAction graql/java/graql.jar:
/bin/bash: line 26: /b/f/w//usr/lib/jvm/java-8-openjdk-amd64/bin/jar: No such file or directory
Exception in thread "main" java.io.FileNotFoundException: /tmp/tmp.GeHanqPjjd/combined.jar (No such file or directory)
	at java.util.zip.ZipFile.open(Native Method)
	at java.util.zip.ZipFile.<init>(ZipFile.java:225)
	at java.util.zip.ZipFile.<init>(ZipFile.java:155)
	at java.util.jar.JarFile.<init>(JarFile.java:166)
	at java.util.jar.JarFile.<init>(JarFile.java:130)
	at org.pantsbuild.jarjar.util.StandaloneJarProcessor.run(StandaloneJarProcessor.java:31)
	at org.pantsbuild.jarjar.Main.process(Main.java:95)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.pantsbuild.jarjar.MainUtil.runMain(MainUtil.java:37)
	at org.pantsbuild.jarjar.Main.main(Main.java:50)
ERROR: /grakn/graql/java/BUILD:23:1: output 'graql/java/graql.jar' was not created
ERROR: /grakn/graql/java/BUILD:23:1: not all outputs were created or valid
Target //server/test/server/deduplicator:rocks-db-queue-test failed to build
INFO: Elapsed time: 6.395s, Critical Path: 3.94s
INFO: 17 processes: 17 remote cache hit.
FAILED: Build did NOT complete successfully

FAILED: Build did NOT complete successfully

As you can see, JAVA_HOME is now pointing to an incorrect location /b/f/w//usr/lib/jvm/java-8-openjdk-amd64/bin/jar. It should be pointing to /usr/lib/jvm/java-8-openjdk-amd64/bin/jar instead.

This can be fixed by handling both absolute and relative path scenario, see my follow up PR #44.

As title.

Hello, I'm trying to use this rule in my Workspace as such

http_archive(
    name = "google_bazel_common",
    strip_prefix = "bazel-common-9e3880428c1837db9fb13335ed390b7e33e346a7",
    urls = ["https://github.com/google/bazel-common/archive/9e3880428c1837db9fb13335ed390b7e33e346a7.zip"],
)

load("@google_bazel_common//:workspace_defs.bzl", "google_common_workspace_rules")

google_common_workspace_rules()

I want to set the api_level to 28 and build_tools_version to 28.0.3, I don't see any way to do that currently.

If I add android_sdk_repository after loading google_common_workspace_rules, I get this error:

Error:/Users/changd/dev/xplat/WORKSPACE:87:1: Cannot redefine repository after any load statement in the WORKSPACE file (for repository 'androidsdk')

Can we parameterize google_common_workspace_rules()?

Hello, I work on @bazelbuild and we'd like to highlight this repository on the Bazel 0.12 release newsletter as a new OSS Bazel-related project. Do you mind if we share it publicly? It will be introduced like so:

Tools

• Common functionality for Google's open-source libraries that are built with Bazel. https://github.com/google/bazel-common

In google/elemental2, most Java sources are actually generated by google/jsinterop-generator (jsinterop_generator rules), and I can't seem to generate javadoc for those using e.g. srcs = [":libcore-src.jar"]

Actually, it looks like srcs is only used to compute whether the rule is uptodate or needs to be re-run, and root_packages and exclude_packages are the real inputs to the javadoc command.

Similar to Maven commands like:

• mvn versions:use-latest-versions
• mvn versions:use-latest-releases
• mvn versions:update-properties

And just to spam more keywords, here are some Maven commands that display updates but don't make the changes:

• mvn versions:display-property-updates
• mvn versions:display-dependency-updates

I poked around the web and found some a project for keeping Bazel deps up to date, but it has its own system that might not play well with bazel-common:
https://github.com/johnynek/bazel-deps

I notice that it's mentioned in:
https://github.com/bazelbuild/BUILD_file_generator#third-party-maven-dependencies

According to https://stackoverflow.com/questions/9047949/missing-artifact-com-sun-jdmkjmxtoolsjar1-2-1, there is a dependency problem in 1.2.15 that causes problems like google/flogger#105.

I think I almost filed this issue once before. This time, I'm going to do it for real.

As an example, asm-tree should depend on asm, and and asm-commons should depend on both asm-tree and asm. Without those deps, it's possible to get errors like class file for org.objectweb.asm.tree.MethodNode not found.

The trick is that targets like asm-tree are java_library targets, which export jvm_import_external-generated java_import targets. And while java_import targets can have deps, java_library targets can't have deps unless they have srcs. (runtime_deps would be wrong here; exports is overkill/unhygienic but would at least work.)

Possible solutions:

• Continue to live with it, adding transitive deps explicitly to the targets in bazel-common users' projects as errors crop up.
• Use exports (again, "overkill/unhygienic but would at least work").
• Set srcs to an empty srcjar (either one that we check in, though that would likely require compliance exemptions, or one that we autogenerate at build time). Ideally we'd generate only one such jar and share it across libraries.
• See if jvm_import_external can set deps (maybe?) or if we should be using something else, like maven_install (which sounds more likely to do this for us). [edit: Err, did something named "maven_install" subsequently become part of rules_jvm_external??] (We may have an issue open in one of our projects about whether maven_install could largely replace bazel-common. Compare what gRPC did.) [edit: The thing we currently use is maven_import, our wrapper around java_import_external.]

Need to support java 17

Version 9.5 is needed to handle Java 21. 9.6 is latest.

If a project has more than one javadoc_library rule in it, one may end up sharing the other's contents to some degree, if --spawn_strategy=local is set, such as in the .bazelrc file.

As long as each invocation of the rule is run in series, there will be no obvious difference from viewing the HTML results - the index.html etc will only point to the expected classes. In theory there could be obvious bugs if the rules are run in parallel, but I haven't seen this occur. The bug is obvious though when inspecting the contents of the javadoc jar.

Example BUILD file:

javadoc_library(
  name = "foo",
  srcs = ["Foo.java"],
)
javadoc_library(
  name = "bar",
  srcs = ["Bar.java"],
)

Where a trivial java file exists for each of these two classes. You would expect to find that each jar contains only the class it references, but instead one will contain both. This will happen even if the rules are declared in different BUILD files in different directories, but different invocations of bazel seem to not share contents.

I have also seen that if the tmp/ directory already exists in the root directory of the project, its contents can be merged into the output javadoc jar, but I can't consistently reproduce that.

I believe this is a bug in the javadoc_library rule, it is not correctly declaring the output it creates and keeping it as "hermetic" as it should. Presently, all output is written to a tmp/ directory, which happens to be in the root of the project (regardless of where bazel is invoked from, or where the rule itself is declared):

Instead, this directory should probably be in the same directory as the output. My understanding of bazel is that this idea could look something like:

    tmp = ctx.actions.declare_directory("%s_javadoc" % ctx.attr.name)

    javadoc_command = [
        java_home + "/bin/javadoc",
        "-use",
        "-encoding UTF8",
        "-classpath",
        ":".join([jar.path for jar in classpath]),
        "-notimestamp",
        "-d %s" % tmp.path, # point to the newly created direc
        "-Xdoclint:-missing",
        "-quiet",
    ]

In turn, the later jar command would need to consume this new file:

    jar_command = "%s/bin/jar cf %s -C %s ." % (java_home, ctx.outputs.jar.path, tmp.path) 

and the action declare this directory as an output:

    ctx.actions.run_shell(
        inputs = srcs + classpath + ctx.files._jdk,
        command = "%s && %s" % (" ".join(javadoc_command), jar_command),
        outputs = [ctx.outputs.jar, tmp], # new output declared here
    )

See google/flogger#122, which I believe indicates a problem in bazel-common.

javadoc.bzl builds its list of sourcepaths via find * [stuff]:

https://github.com/google/bazel-common/blob/master/tools/javadoc/javadoc.bzl#L37

The execroot it's sitting in is chock full of symlinks to the concrete workspace, and that find won't follow them, so the javadoc invocation fails complaining javadoc: error - No public or protected classes found to document.

If I hack that to use either a -L or to glob */, it does find them, but it's blindly crawling my whole source tree and gives javadoc for my test classes.

I'm seeing the same behavior trying to build javadoc for flogger, so I'm pretty sure I'm holding it right, but I tossed together a pretty bare bones demo repository to confirm: https://github.com/deadmoose/javadoctest

Turns out there are more bazel deprecations:
protocolbuffers/protobuf#5547

One thing I noticed is the maven jar is 3.6.1 (released july) and 3.6.1.3 (released december) doesn't have a protobuf_java corollary. I asked them to cut one but I assume that would block adoption here?

https://bazel.build/build/bzlmod

Migrate our dependencies to it? Stop maintaining a list of dependencies entirely now that dependencies have gotten easier (in which case bazel-common would be entirely(?) just rules and macros)?