Support private repository requiring authentication with Authorization http header in a more secure manner about rules_jvm_external HOT 31 CLOSED

Can you think of a solution that would work with .bazelrc-local?

We can do something with environment variables.

$ MAVEN_USERNAME=foo MAVEN_PASSWORD=bar bazel build //...

and in the WORKSPACE file:

maven_install(
    # ...
    repositories = [
        "http://{MAVEN_USERNAME}:{MAVEN_PASSWORD}@localhost:8081/artifactory/my-repository",
    ],
)

Just before we invoke Coursier, we call .format() on the URLs.

This approach also works with .bazelrc / command line flags:

$ bazel build //... --action_env=MAVEN_USERNAME=foo --action_env=MAVEN_PASSWORD=bar

# Assuming MAVEN_USERNAME and MAVEN_PASSWORD are set
build --action_env=MAVEN_USERNAME
build --action_env=MAVEN_PASSWORD

Note that the --action_env approach will apply the env var to all your build actions.

What do you think about this approach?

from rules_jvm_external.

Thanks for reporting this issue -- this is one of the areas that we want to tackle as soon as possible.

@dkelmer is looking into a better HTTP downloader mechanism in Bazel that works with more auth approaches, including the ones you've mentioned.

Currently, we rely on Coursier CLI's API, which only supports the URL scheme approach. To work around directly checking in usernames and passwords in your WORKSPACE, I thought of a few approaches:

1. Load strings from a .bzl file that's not checked in.

load("//secrets:maven.bzl", "USERNAME", "PASSWORD")
maven_install(
    # ...
    repositories = [
        "http://%s:%s@localhost:8081/artifactory/my-repository" % (USERNAME, PASSWORD),
    ],
)

2. Load strings from a remote Bazel WORKSPACE using http_archive/git_repository:

git_repository(
    name = "secrets_repository",
    remote = "[email protected]:secrets/secrets.git",
    tag = "...",
)
load("@secrets_repository//secrets:maven.bzl", "USERNAME", "PASSWORD")
maven_install(
    # ...
    repositories = [
        "http://%s:%s@localhost:8081/artifactory/my-repository" % (USERNAME, PASSWORD),
    ],
)

I don't think http_archive supports authentication yet, but the git_repository rule uses system git, so it works with your regular ssh keys.

3. Load strings from another local (or mounted / authenticated) directory using local_repository. This is similar to the previous approach.

local_repository(
    name = "secrets_repository",
    path = "/path/to/secrets/",
)
load("@secrets_repository//secrets:maven.bzl", "USERNAME", "PASSWORD")
maven_install(
    # ...
    repositories = [
        "http://%s:%s@localhost:8081/artifactory/my-repository" % (USERNAME, PASSWORD),
    ],
)

Do any of these approaches work for you?

from rules_jvm_external.

I've tried on macOS with credentials in ~/Library/Preferences/Coursier/credentials.properties with myrepo.auto=true as mentioned in above comment, I got past fetching to

ERROR: An error occurred during the fetch of repository 'maven':
local variable 'primary_artifact_path' is referenced before assignment.

from rules_jvm_external.

@KaoruDev I think that may be due to coursier/coursier#1319

from rules_jvm_external.

I had to add this:

myrepo.auto=true

Now my build seems to be getting closer to working.

from rules_jvm_external.

Thanks for the feedback. I already thought of something like approach 1 as a workaround, i.e. have a Bazel file in .gitignore that will be included in the build. That should work.

Can you think of a solution that would work with .bazelrc-local?

Thanks for the update on the HTTP downloader feature request. Really appreciate it. I believe this will be the key because we really need http header auth for our repo servers.

from rules_jvm_external.

Looks like Coursier is reworking its credential handling upstream, with support for environment variables and reading from a properties file: coursier/coursier#1102

from rules_jvm_external.

Documentation: https://github.com/coursier/coursier/blob/0bccb51d3572f36fa664e825cb48e749c7ba1845/doc/docs/other-credentials.md

from rules_jvm_external.

Would either or both be a required dependency for fixing this one?

• bazelbuild/bazel#7443 (authentication in jvm_maven_import_external/java_import_external)
• bazelbuild/bazel#7469 (authentication for repository_ctx.download)

from rules_jvm_external.

cc @dkelmer

Both jvm_maven_import_external and java_import_external delegates downloading to repository_ctx.download. Currently rules_jvm_external doesn't use repository_ctx.download, instead it delegates downloading to Coursier. To fix this issue, we will have to upstream more authentication avenues to Coursier, or implement bazelbuild/bazel#7469 and switch rules_jvm_external to use that.

from rules_jvm_external.

FWIW, I've created a local little proxy to deal with authenticated Maven repositories. It also supports streaming from the local .m2 Maven repo when an artifact is available, which will avoid server round trips completely.

https://github.com/guw/bazel-maven-proxy

from rules_jvm_external.

I have a Coursier properties file in: /Library/Preferences/Coursier/credentials.properties but still can't properly authenticate to our private repo. I'm confused, is Coursier looking elsewhere for this file?

from rules_jvm_external.

Seems coursier/coursier#1350 fixed the issue, looks like bumping to version RC-3-4 would pull in the fix.

from rules_jvm_external.

I got them to fully publish that version yesterday, it should be possible to bump it now.

from rules_jvm_external.

😭 Sorry doesn't seem to have fix the issue. I've tried:

• via property
• via ENV

from rules_jvm_external.

Mine is being found via ~/.config/coursier/credentials.properties on Linux.

from rules_jvm_external.

Maybe this is a MacOSx problem, I have a config file in ~/Library/Preferences/Coursier/credentials.properties and in ~/.config/coursier/credentials.properties (just in case)

from rules_jvm_external.

Reopening as issue is still reported on macOS.

from rules_jvm_external.

@whs, @puffnfresh provided a fix in #248. Could you please try pulling rules_jvm_external from the latest commit and give it a shot?

from rules_jvm_external.

Seems to be working for me. (with auto=true, I didn't test without auto=true) Thanks!

from rules_jvm_external.

sorry, I'm missing where to put auto=true?

from rules_jvm_external.

In credentials.properties, #80 (comment)

from rules_jvm_external.

Ah ok thanks @whs I thought this was a bazel config which confused me. As a note to my future self, I replaced myrepo in <myrepo>.auto=true to assign the value to the group. @jin feel free to close this out, thanks!

from rules_jvm_external.

Thanks for verifying, folks.

from rules_jvm_external.

@jin sorry I was too quick to declare this fixed and I now believe this is a rules_jvm_external issue.

What changed?

When I confirmed things were working, I just ran bazel run @unpinned_repo//:pin and did not see any errors. I didn't actually declare it a dependency in one of my targets.

Now that I'm bringing in a project that is using a private version, I encountered an Authentication issue when running sync via Intellij. I verified pulling down the artifact using the Coursier CLI works.

from rules_jvm_external.

Ok I was wrong ^ the issue I'm facing is not related to auth but rather the fact that the JAR i'm trying to pull down has additional letters to it. As an example: org.postgresql:postgresql:9.4-1201-jdbc41 whatever mechanism is used to copy over the jar that Coursier pulled down to be used as Bazel's cache is not working if the jar's version has extra data, will open new issue

from rules_jvm_external.

An error occurred during the fetch of reposit

I have a similar error, take a look please:

If the definition of 'toto' was updated, verify that the hashes were also updated.
ERROR: An error occurred during the fetch of repository 'toto':
   Traceback (most recent call last):
        File "/private/var/tmp/_bazel_lyes.abdelfettah/bdbe9cb9aff117eb863b0a1426519f31/external/rules_jvm_external/coursier.bzl", line 783, column 34, in _coursier_fetch_impl
                mirror_urls = [url + "/" + primary_artifact_path for url in repository_urls]
Error: unsupported binary operation: string + NoneType

I'm fighting with this.
There is no documentation at all !
I had first:

• to export this var COURSIER_CREDENTIALS=/Users//Library/ApplicationSupport/Coursier/credentials.properties
• to add the auto=true described

-> But i still get the above error.
What can I do ?

from rules_jvm_external.

@ablyes does your user token or password contain a / character or @ character? I reported this in #414 and there is also #418 and #368. They are all the same issue, I believe.

from rules_jvm_external.

@ablyes does your user token or password contain a / character or @ character? I reported this in #414 and there is also #418 and #368. They are all the same issue, I believe.

The username contains @ :)
I will read the 3 topics, thank you.
Is there a solution right now ?

from rules_jvm_external.

I describe a path to a workaround using a private fork in #414 but my fix was for / not @ case so my private fork workaround wouldn't work for you.

from rules_jvm_external.

Good job !
I pushed a pull request for my problem (@ in login).
But I forked and tagged a version if somebody else want to use my temporary fix.

from rules_jvm_external.