acmcarther / cargo-raze Goto Github PK

Not sure if that is in the modifications you made, but I run into trouble with this line in toolchain.bzl:

["if [[ -v TMPDIR ]]; then mkdir -p $TMPDIR; fi;"] +

With it in, nothing builds on mac os due some shell snafu. Commenting it out makes the build work for me:

$ bazel build //cargo/vendor:pbr
INFO: Analysed target //cargo/vendor:pbr (6 packages loaded).
INFO: Found 1 target...
ERROR: /Users/hrapp/Desktop/Programming/rust/point_cloud_viewer/cargo/vendor/winapi-0.2.8/BUILD:16:1: error executing shell command: 'set -e; if [[ -v TMPDIR ]]; then mkdir -p $TMPDIR; fi; rm -rf bazel-out/darwin-fastbuild/bin/cargo/vendor/winapi-0.2.8/winapi.deps; mkdir bazel-out/darwin-fastbuild/bin/cargo/vendor/winapi-0.2.8/wi...' failed (Exit 2)
/bin/bash: -c: line 0: conditional binary operator expected
/bin/bash: -c: line 0: syntax error near `TMPDIR'
/bin/bash: -c: line 0: `set -e; if [[ -v TMPDIR ]]; then mkdir -p $TMPDIR; fi; rm -rf bazel-out/darwin-fastbuild/bin/cargo/vendor/winapi-0.2.8/winapi.deps; mkdir bazel-out/darwin-fastbuild/bin/cargo/vendor/winapi-0.2.8/winapi.deps'
Target //cargo/vendor/pbr-1.0.0:pbr failed to build
Use --verbose_failures to see the command lines of failed build steps.
INFO: Elapsed time: 0.250s, Critical Path: 0.10s
FAILED: Build did NOT complete successfully

From:
bazelbuild/rules_k8s#62

It may be possible to satisfy a use case like this by synthesizing a crates.bzl file. Heres how it might work:

WORKSPACE

load("//cargo:crates.bzl", "fetch_remote_crates")
fetch_remote_crates()

//cargo:crates.bzl

## GENERATED BY CARGO-RAZE, DO NOT EDIT

def fetch_remote_crates():
  new_http_archive(
    name = "some_crate-0.0.1",
    url = "http://github.com/some-person/some-crate",
    sha256 = "some-sha",
    build_file = "//cargo:BUILD.some_crate-0.0.1",
  )
  new_http_archive(
    name = "some_other_crate-0.0.1",
    url = "http://github.com/some-person/some-crate",
    sha256 = "some-sha",
    build_file = "//cargo:BUILD.some_other_crate-0.0.1",
  )

//cargo:BUILD.some_crate-0.0.1

rust_library(
  name = "some_crate",
  deps = [
    "@some_other_crate-0.0.1//:some_other_crate",
  ],
  crate_features = [
    "stuff",
  ],

The files within //cargo would be generated similar to how they are today: a single execution of cargo-raze will generate the complete list.

With the functionality currently available in the "raze" section of the cargo toml, it is possible to override sources and dependencies, suppress dependencies, conditionally generate build_rs scripts. This functionality would all be available under the no-vendor proposal as well.

I want to be able to pair each PR with an examples generated branch in acmcarther/cargo-raze-examples. That repo already has scripts to pull master and regenerate the bazel examples. It should be extended to accept a commit sha or maybe a local path.

This is a vague issue, created around my experience of overriding stuff to support openssl. Historically, my approach has been to try to execute the build.rs scripts, but often I think it is more appropriate to perform the steps manually. This is because often the upstream dependency (in this case, the openssl C lib) is actually in the mono repo, so the dynamic configuration is unnecessary.

Here was the process for openssl:

1. Pull down openssl tar and unpack it into the monorepo.
2. Build it, either by manually crafting a build rule with the desired settings, or by building it with make. Either way, produce a .a file. My build file was as follows:

package(default_visibility = ["//visibility:public"])

licenses(["notice"])

cc_library(
    name = "crypto",
    srcs = [
        "libcrypto.a",
    ]
)

cc_library(
    name = "ssl",
    srcs = [
        "libssl.a",
    ]
)

3. Remove openssl and openssl's build_script entries in BUILD. Add deps to openssl_sys for crypto and ssl, and update the cfg flags. The full entries are:

rust_library(
    name = "openssl_sys",
    crate_root = "src/lib.rs",
    srcs = glob(["**/*.rs"]),
    deps = [
      "//cargo/vendor/libc-0.2.33:libc",
      "//third_party/openssl/openssl-1.1.0g:crypto",
      "//third_party/openssl/openssl-1.1.0g:ssl",
    ],
    rustc_flags = [
        "--cap-lints allow",
        "--target=x86_64-unknown-linux-gnu",
        "--cfg=ossl110",
        "--cfg=version=110",
    ],
    crate_features = [],
)

and

rust_library(
    name = "openssl",
    crate_root = "src/lib.rs",
    srcs = glob(["**/*.rs"]),
    deps = [
      "//cargo/vendor/bitflags-0.9.1:bitflags",
      "//cargo/vendor/foreign-types-0.2.0:foreign_types",
      "//cargo/vendor/lazy_static-0.2.11:lazy_static",
      "//cargo/vendor/libc-0.2.33:libc",
      "//cargo/vendor/openssl-sys-0.9.21:openssl_sys",
    ],
    rustc_flags = [
        "--cap-lints allow",
        "--target=x86_64-unknown-linux-gnu",
        "--cfg=ossl110",
    ],
    crate_features = [],
)

I'm thinking perhaps including customizations in the Cargo.toml might be a good step. Here is a sample schema

[raze.crates.normalized_crate_name.0.9.2.override]
use_build_rs = false
include_examples = false
include_tests = false
additional_deps = [
  "//third_party/openssl/openssl-1.1.0g:crypto",
  "//third_party/openssl/openssl-1.1.0g:ssl",
]
additional_flags = [
  "--cfg=ossl110",
]

Transitive dependencies need to be affectable in this way -- I'll observe no difference between them and explicit deps.

I'd like to be able to bring in bindgen and use it in genrules. There isn't an automagic way to do that for now.

As an aside, bindgen in particular has an idiosyncratic issue in that it tries to name the binary target the same thing as the library target.

Now that I've figured out how to make openssl compile (gory details in my demo repo: https://github.com/acmcarther/compile_openssl), we should be able to produce a WORKSPACE file, and run cargo-raze on this project.

That would allow us to promote the running of cargo-raze into a Bazel build rule, which would eliminate the need to run it manually.

This actually also unlocks running cargo-raze for cargo-vendor as well, which would facilitate performing vendoring in a Bazel build rule as well -- ultimately narrowing down the process of using cargo raze to just including a toml.

Right now after the process, you will have your workspace_path = "//cargo" directory populated with:

• Cargo.toml, your source for the third_party packages you want to pull in.
• Cargo.lock, the generated log file.
• vendor directory, containing output of vendor -x and raze.

Building now looks like this: bazel build //cargo/vendor:log. I would prefer if it were //cargo:log. Possible?

I have a situation with the sdl2 crate where I really need to replace sdl2-sys with my own bindgen'd crate. It is possible to add a new crate to serve the purpose, but it isn't possible to remove the old one.

It would be great if there was a straightforward way to add support for multiple architectures at once. A bit like how the Bazel --ios_multi_cpus works, but perhaps just allow an array of targets in Cargo.toml, which generates arch specific rules.

I ran into a couple of issues putting together https://github.com/mfarrugi/cargo-raze-example-stdx/blob/master/cargo/Cargo.toml, though it ultimately worked. The issues are generally mild, but inform on some of the edge case chicanery of build.rs scripts.

1. mime_guess.2.0.0-alpha.3 build script is not correct, since it is trying to generate sources in the source directory. (Work around is running build twice, or committing the generated code)

2. Unicase output dirs appear to compete w/ each other. (Workaround is skipping the build scripts)

3. reqwest requires CARGO_PKG_NAME and CARGO_PKG_VERSION variables when building -- this is more of an issue for rules_rust. (Workaround here was patching the usage, since there was only one).

This is probably not a bug, just a case of you're holding it wrong, but building cgmath with the generated build files does not work. An issue with generated files from a build script, I presume. I have no idea how to debug and fix this though - could you add some documentation somewhere?

ERROR: /Users/hrapp/Desktop/Programming/rust/point_cloud_viewer/cargo/vendor/cgmath-0.16.0/BUILD:17:1: error executing shell command: 'set -e; rm -rf bazel-out/darwin-fastbuild/bin/cargo/vendor/cgmath-0.16.0/cgmath.deps; mkdir bazel-out/darwin-fastbuild/bin/cargo/vendor/cgmath-0.16.0/cgmath.deps
 ln -sf ../../approx-0.1.1/libappro...' failed (Exit 101)
error: environment variable `OUT_DIR` not defined
   --> cargo/vendor/cgmath-0.16.0/src/macros.rs:480:18
    |
480 | include!(concat!(env!("OUT_DIR"), "/swizzle_operator_macro.rs"));
    |                  ^^^^^^^^^^^^^^^

error: couldn't read "cargo/vendor/cgmath-0.16.0/src/0/swizzle_operator_macro.rs": No such file or directory (os error 2)
   --> cargo/vendor/cgmath-0.16.0/src/macros.rs:480:1
    |
480 | include!(concat!(env!("OUT_DIR"), "/swizzle_operator_macro.rs"));
    | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

INFO: Elapsed time: 3.149s, Critical Path: 2.99s
FAILED: Build did NOT complete successfully

@lilianmoraru on #6 noted that usage of genrule's cmd flag isn't very cross platform. Instead, we should pass a complete binary to genrule.

The relevant file is
build_script.template

Maybe a py_binary would be better?

Bazel actually enforces a license for BUILD files in //third_party.

ERROR: /Users/hrapp/Desktop/Programming/rust/point_cloud_viewer/third_party/rust/vendor/cfg-if-0.1.2/BUILD:16:1: third-party rule '//third_party/rust/vendor/cfg-if-0.1.2:cfg_if' lacks a license declaration with one of the following types: notice, reciprocal, permissive, restricted, unencumbered, by_exception_only

TL;DR: Need to extend Cargo.lock with dev dependencies -- currently examples and tests may or may not work.

It turns out that Cargo does not lock dev dependencies for declared dependencies, presumably because they are not required to compile the root crate. This is fine for the typical cargo case, but it prevents users from building examples or tests of linked crates.

It would be nice to be able to generaterust_test and rust_binary` for linked crates, but to do that, we'll need these dev dependencies.

More specifically -- cargo::ops::resolve_ws_precisely does not include dev deps. Fixing this (with custom code) is necessary, but not sufficient, to enable specifying these dependencies, because cargo-vendor needs to vendor them as well.

Unsure if i need to make changes to rules_rust.

An example dependency that needs this is specs-0.10.0 (via shred-derive).

https://github.com/acmcarther/cargo-raze/blob/master/src/settings.rs#L89
^ "//blah" is NOT the format for this argument. The format is actually {dep_name}-{dep_version}, similar to how cargo would represent a dependency.

Related: sfackler/rust-openssl#987

Because cargo-raze depends on an old version of cargo (v0.20.0, current version is v0.30.0), which in turn depends on an old version of openssl, cargo-raze fails to build on systems with OpenSSL v1.1.1 installed because the openssl crate fails to detect the installed OpenSSL version.

I've tried fixing this locally by updating the dependency on Cargo, but it looks like there are some breaking API changes that make that a more complicated process than I expected. Someone more familiar with the Cargo API might have a better shot at this.

In case it's related, here's some system info:
OS: Arch Linux (Linux v4.18.12)
OpenSSL version: 1.1.1 (11 Sep 2018)
Version of the Cargo binary I used to build cargo-raze: 1.29.0

Here's a sample of the build output after updating the version of cargo:

[haliax] ~/i/c/impl →  cargo build
   Compiling cargo-raze v0.1.1 (file:///home/alaroldai/install/cargo-raze/impl)                                                                                                                    
error[E0425]: cannot find function `call_main_without_stdin` in module `cargo`
  --> src/main.rs:91:23
   |
91 |   let result = cargo::call_main_without_stdin(real_main, &cargo_config, USAGE, &args, false);
   |                       ^^^^^^^^^^^^^^^^^^^^^^^ not found in `cargo`

error[E0061]: this function takes 7 parameters but 5 parameters were supplied
  --> src/main.rs:99:21
   |
99 |   try!(cargo_config.configure(
   |                     ^^^^^^^^^ expected 7 parameters

error[E0599]: no method named `into_package_id_specs` found for type `cargo::ops::Packages` in the current scope
   --> src/metadata.rs:241:31
    |
241 |     let specs = Packages::All.into_package_id_specs(&ws)?;
    |                               ^^^^^^^^^^^^^^^^^^^^^
    |
    = help: did you mean `to_package_id_specs`?

error[E0277]: the trait bound `&str: std::error::Error` is not satisfied
   --> src/metadata.rs:252:16
    |
252 |         .ok_or(CargoError::from("root crate should be in cargo resolve"))
    |                ^^^^^^^^^^^^^^^^ the trait `std::error::Error` is not implemented for `&str`
    |
    = note: required because of the requirements on the impl of `failure::Fail` for `&str`
    = note: required because of the requirements on the impl of `std::convert::From<&str>` for `cargo::Error`
    = note: required by `std::convert::From::from`

// alaroldai@: stripped a number of other in calling CargoError::from with a String argument

error: aborting due to 31 previous errors

Some errors occurred: E0061, E0277, E0308, E0425, E0599.
For more information about an error, try `rustc --explain E0061`.
error: Could not compile `cargo-raze`.

To learn more, run the command again with --verbose.

The latest version is effectively unpublishable due to a weird quirk in cargo dependency resolution (which makes me happy that I only need to use it for this exact crate).

Here's the situation. I upgraded to a beta version of the tera crate to fix the formatting of the generated files. This crate depends on a crate, pest. Under standard conditions, this builds fine. However, when trying to publish, cargo promotes the version of pest from -beta.17 to -rc1. Unfortunately, this is a breaking change, and it causes tera to break.

Tera has a fix in Keats/tera@2f9cf11#diff-80398c5faae3c069e4e6aa2ed11b28c0, but it needs a named publish before I can depend on it.

@acmcarther Sorry, this is not really an issue, but I did not know how to reach you otherwise.

My company wants to explore rust, but one of our requirements is that we build everything in bazel. We have existing rust code, but had little success building it in bazel, mostly due to the many third_party dependencies that rust requires.

Can you give me a bit of a status on bazel + rust? What is working? what is not working? who is investing?

Also, who is maintaining the https://github.com/bazelbuild/rules_rust repo? It seems pretty dead at this point in time.

I would like to reference the vendored libs as @cargo//:libc, but cargo raze @// is rejected.

Is there another prefix or setup that would work?

I was planning to build something similar and see you have already done a good chunk of the work!

Any plans to make the output tool and format pluggable? I think having clearly defined interfaces could also help the project's structure.

Would you take PRs to support such a thing? If not and you prefer to just focus on Bazel for now, no worries--I will work in my own repo and maybe we can merge projects in the future.