grailbio / rules_r Goto Github PK

I'm building custom images containing R libs (among other things) using GoogleContainerTools/distroless/package_manager. I have worked though collecting and adding R deps to packages.csv so all of my R world dependencies are fine.

But those R libraries have system library dependencies. When you load the R libraries in R code, they look for the system libraries they use, and throw at the first one they can't find.

I am not a native R person, so please forgive me if this is a stupid question: is there any way to find the native deps of R libraries? Iterating over this is really time consuming at scale.

Any help appreciated.

Having a symlinked, read-only reference to the source files can be problematic for some configure scripts, e.g., for the XML or Rgraphviz packages (they overwrite source files, they create temporary files within the source tree, ...). One quick fix is to make R CMD install work, for those packages, on a full, writable recursive copy of the source files.

I propose a new copy_srcs bool attribute to r_pkg to enable that. This would reuse the machinery of REPRODUCIBLE_BUILD and copy sources with cp -LpR ... instead of cp -a (-L dereferences symlinks).

Currently, r_pkg leans pretty heavily on R to build *.cpp code, which is pretty slow.

What if, instead, C++ code was built with Bazel (which is fast), and then passed to r_pkg through cc_dep, already prebuilt? I've hacked up something like this and it seems to work fine, and it's much faster. I had to use an empty *.cpp file to get r_pkg to link the *.so, and I had to put libRcpp.so into a location where it could easily be found, but other than that this required no changes to the C++ code. The resulting package seems to work OK under r_unit_test. Not sure what r_pkg_test would think of it, but it'll probably be fine as well, since it works OK with other cc_deps and this is no different.

So basically the suggestion here is to modify the rule such that it compiles all C++ source with cc_library (and lines up the *.so's and headers for that to work, so symlinking hack is not necessary), and then feeds the result into build.sh much like it would feed cc_deps.

See RcppCore/RcppArmadillo#331 for discussion.

Note that covr does use the += assignment operator in the personal Makevars file to set the flags.

It would be possible to get code coverage for r_pkg as follows:

• The covr R package could be used for a mix of offline and online instrumentation: the package loader script would be modified offline to instrument the package when it is loaded. Some covr internals have to be used here, but thanks to rules_r it is possible to pin down a compatible version.
• A small function in R converts the covr line-by-line format to an LCOV coverage report.
• The LCOV coverage reports are collected by Bazel into bazel-testlogs when testing with 'bazel coverage'.

It is possible to wrap an r_binary invocation in a similar way.

The LCOV reports can then be consumed in many ways, for instance using the genhtml command-line utility.

Code coverage can be transitive if --instrumentation_filter is properly set, and covr integrates nicely with native code, either generated by rules_r or handled through cc_deps.

I have a WORKSPACE which works on my dev machine but breaks on my team's jenkins box.

My dev laptop is Fedora, and the jenkins box is ubuntu. I don't think that should make a difference but it's the one difference in the environment I can think of.

I had to apt-get install r-base because prior to this there was no R on the jenkins box.

Any help appreciated.

WORKSPACE snippet:

http_archive(
    name = "com_grail_rules_r",
    strip_prefix = "rules_r-master",
    urls = ["https://github.com/grailbio/rules_r/archive/master.tar.gz"],
)

load("@com_grail_rules_r//R:dependencies.bzl", "r_register_toolchains", "r_rules_dependencies")

r_rules_dependencies()

r_register_toolchains()

load("@com_grail_rules_r//R:repositories.bzl", "r_repository_list", "r_repository")

r_repository(
    name = "R_data_table",
    sha256 = "f5b2b7d44ef5d8cb3505b4e6b4c4539e7a2132dffc5516da6f717fa51ebe9d3b",
    strip_prefix = "data.table-c0052964694a4c618ab182aa474f924d40576d94",
    urls = [
        "https://github.com/Rdatatable/data.table/archive/c0052964694a4c618ab182aa474f924d40576d94.tar.gz",
    ],
)

r_repository_list(
    name = "r_repositories_bzl",
    package_list = "//sample_r:packages.csv",
    remote_repos = {
        "CRAB": "https://cloud.r-project.org",
    },
)

load("@r_repositories_bzl//:r_repositories.bzl", "r_repositories")

r_repositories()

load("@com_grail_rules_r//R:dependencies.bzl", "r_coverage_dependencies")

r_coverage_dependencies()

bash log:

$ bazel build //sample_r:sample
INFO: Writing tracer profile to '/home/jenkinsadmin/.cache/bazel/_bazel_jenkinsadmin/b33a4b3c4e6f0c61d58e78df9100fe07/command.profile.gz'
INFO: SHA256 (https://github.com/grailbio/rules_r/archive/master.tar.gz) = bacdb1d81e81db7a95d9e364e9eea73c9cfea43f0901c115bc7b8d3f5eec1ae2
DEBUG: Rule 'com_grail_rules_r' indicated that a canonical reproducible form can be obtained by modifying arguments sha256 = "bacdb1d81e81db7a95d9e364e9eea73c9cfea43f0901c115bc7b8d3f5eec1ae2"
DEBUG: Call stack for the definition of repository 'com_grail_rules_r' which is a http_archive (rule definition at /home/jenkinsadmin/.cache/bazel/_bazel_jenkinsadmin/b33a4b3c4e6f0c61d58e78df9100fe07/external/bazel_tools/tools/build_defs/repo/http.bzl:292:16):
 - /home/jenkinsadmin/repo/mob/WORKSPACE:391:1
INFO: Call stack for the definition of repository 'r_repositories_bzl' which is a r_repository_list (rule definition at /home/jenkinsadmin/.cache/bazel/_bazel_jenkinsadmin/b33a4b3c4e6f0c61d58e78df9100fe07/external/com_grail_rules_r/R/repositories.bzl:139:21):
 - /home/jenkinsadmin/repo/mob/WORKSPACE:414:1
ERROR: An error occurred during the fetch of repository 'r_repositories_bzl':
   Failed to generate bzl file: 

Error in available.packages(repos = repos, type = type) : 
  unused argument (repos = repos)
Calls: generateWorkspaceMacro ... mergeWithRemote -> as.data.frame -> available.packages
Execution halted
ERROR: no such package '@r_repositories_bzl//': Failed to generate bzl file: 

Error in available.packages(repos = repos, type = type) : 
  unused argument (repos = repos)
Calls: generateWorkspaceMacro ... mergeWithRemote -> as.data.frame -> available.packages
Execution halted
ERROR: no such package '@r_repositories_bzl//': Failed to generate bzl file: 

Error in available.packages(repos = repos, type = type) : 
  unused argument (repos = repos)
Calls: generateWorkspaceMacro ... mergeWithRemote -> as.data.frame -> available.packages
Execution halted
INFO: Elapsed time: 1.323s
INFO: 0 processes.
FAILED: Build did NOT complete successfully (0 packages loaded)

I think it'd be cool if r_unit_test could define data dependencies so that Bazel could download and symlink testdata for unit tests in an atomic, pipelined, and checksummed fashion. Currently the only deps for r_unit_test can be other packages. While that might be appropriate for r_pkg_test, seems like for r_unit_test that's too restrictive.

In particular, this could be useful during development if the authors are not planning to release the package, or if package will be modified before release to be able to do a package level test instead. Indeed, the package could dynamically choose either Bazel managed or pre-installed data source depending on how it's being tested.

This repository can then be overridden in the WORKSPACE file, and all built R packages can use these files.

With a default value of '.', this attribute can specify the path in which the packages are installed inside the tarball. Running the rule will also copy the packages from this path into the library directory.

To reproduce:

cd tests
bazel --output_user_root=/tmp/a\\b sync

...
ERROR: An error occurred during the fetch of repository 'r_repositories_bzl':
   Failed to generate bzl file: 

Error in file(filename, "r", encoding = encoding) : 
  cannot open the connection
Calls: source -> file
In addition: Warning message:
In file(filename, "r", encoding = encoding) :
  cannot open file '/private/tmp//fbcd335144c4da9c79038f9ce88e9f6d/external/com_grail_rules_r/scripts/razel.R': No such file or directory
Execution halted

To separately package the system dependency tools that R packages expect. Useful for R library docker images.

We rely on a package's configure script to locate the system dependency. We should be able to specify some system requirements through Bazel and provide configure flags to use the Bazel built requirement.

We have been using a covr dependency with some of our own modifications. Would be nice to get these changes upstreamed and then depend on a CRAN release directly.

The PR is r-lib/covr#468.

The test target should get this information from the package provider, and make the Makevars, headers and library archives available as runfiles. The paths, etc. should also change to represent short paths.

We currently don't test that our rules result in a reproducible build on execution on the same machine. These tests have been performed manually before.

Currently, the libraries to link are aggregated as such:

c_libs_flags = l.path for d in cc_deps for l in d.cc.libs 

As a consequence, in the following case, the linker complains that some symbols in B are not found, because the dependencies are not given to the linker in topological order:

cc_library A transitively depends on cc_library B
cc_deps = [":B", ":A"]
Command: g++ ... -lB -lA

The solution is to aggregate the d.cc.libs using the depset mechanism instead of simply iterating through the list (I suppose for d.cc.libs the order is "topological").

Reproducible mode has been the default at GRAIL for ~8 months and has been working without any issues. We should write more unit tests for it and make it the default and only mode.

I am building an R package via r_pkg that depends on a cc_library A. Library A depends on two system libraries B and C that I've imported as external repositories via new_local_repository and then created a rule for these using cc_library.
Library A is built fine but my R package is having problems linking against A, and what's more it seems to try and include B and C in the linking statement.
The error I see depends on whether I'm running sandboxed or not. If sandboxed, I see:

cp: cannot create regular file 'external/R_flowCore/flowCore/src/libhdf5.so': Read-only file system
cp: cannot create regular file 'external/R_flowCore/flowCore/src/libarmadillo.so': Read-only file system

those are the system dependencies B and C.
If I don't run sandboxed I see:

g++: error: libhdf5.so: No such file or directory
g++: error: libarmadillo.so: No such file or directory

I don't fullly understand what directories bazel is doing its work in, so I'm having trouble understanding why I see these particular errors, especially the latter non-sandboxed one. Poking around I do see an R package source directory where these .so files have been copied into, but it's doesn't contain the compiled .o files for the rest of the cpp code, so presumably the build directory is elsewhere.

I've looked at rules_r/R/scripts/build.sh and there's a lot of complexity there that I'm having difficulty unravelling.

Finally,thinking this may be related to the bundles configure script, but I see this same behavior irrespective of whether I use the package's bundled configure file or if I remove the configure script.

I'd like to understand:

• in what directory are the g++ build commands invoked? I know R usually creates a temporary directory, but I'm having trouble getting insight into what the bash script is actually doing. Is there a mechanism by which I can start to trace the inner workings here?
• Alternately is there a higher level explanation of the build rule available somewhere.
• Alternately, @siddharthab if you are amenable perhaps I could get a bit of your time on a call or synchronous chat to dive a bit deeper?
  Thanks in advance for all your help.

Have the ability for users to specify a metadata dictionary with values allowing substitutions from the volatile and stable status files. Have a boolean attribute stamp to allow users to specify if stable status file should be a dependency.

The packages built have stamped information about the built timestamp, the source directory and the library directory for the installation. This is especially bothersome with docker images as different layers are created with each build.

The build timestamp can be fixed to an empty string with the --built_timestamp flag to R CMD INSTALL. For the rest, we need to build and install in a constant directory, which means fixing a /tmp path for a package, and acquiring a lock on that path so that builds in other workspaces do not interfere with this build.

Is there a deployment story native to bazel? Or is the out of scope for this?

Let's test that we can support ARM processors, specifically on the new AWS instance types. We may have to wait for R 4.1.0 for full support.

For macOS, CRAN recommendation is to use Rosetta for R versions less than 4.1.0.

r_pkg can take a list of strings as a "roclets" attribute to run roxygen on package source code before building it. This will automate a manual step that developers do frequently before running a build or test.

They are not specified as runfiles for r_unit_test, and the PATH variable is not modified. r_pkg_test seems to have them set up correctly.

We currently use new_http_archive for external R packages, and explicitly list CRAN and BioC repo URLs for the package. A new repository rule for R remote repositories should help.

r_pkg_test builds the source archive of the package independently of the r_pkg rule.

This means all the package build configuration and dependencies have to be specified twice.

https://cran.r-project.org/doc/manuals/r-release/R-exts.html#Data-in-packages

A workaround right now is to use post_install files, but this can be determined automatically and the files added to the output files set.

When the list of dependencies gets really long, R CMD INSTALL no longer sees the installed dependencies when building a package.
I can confirm this by shortening R_LIBS_DEPS, or by installing the packages that aren't found into the R/site-library, then R CMD INSTALL sees them properly.
Two questions:

1. is there a known work-around?
2. Is it possible to have rules_r install to a "global" R_LIBS directory rather than package-specific R_LIBS directories that then get joined into a long ungainly string?
   I can clarify this issue further if needed.
   This is on ubuntu.

We should test that our cache state is invalidated whenever the system R is updated, or some other related toolchain like CC, etc. is updated.

In build.sh, we use 200 as the constant for the file descriptor that we lock.

This is most likely causing unwanted waits for the lock to be released when multiple packages are being built simultaneously.

This is just a hunch though that needs to be validated.

Technically, r_pkg is supposed to collect *.so's and bundle them into the package, which is what it does at the preliminary build step. However, this doesn't really work because R CMD build throws out those *.so's, not recognizing them as a valid part of the package, so by the time build.sh needs to run R CMD install they're no longer there.

It would seem that, instead, they should be copied after build archive is expanded. I'm not sure if install won't throw them out, but cursory search in install.R seems to suggest that it shouldn't.

Noticed that dependencies aren't being downloaded from the bazel cache, but rather are redownloaded every time.

Maybe not the workspace macro, but the BUILD file generation can be done entirely in Go.

We assume a brew install prefix of /usr/local in Makevars.darwin for finding gfortran.

Provide a default Makevars file target that is dynamically generated and uses the right brew prefix for the user.

We rely on system provided R and are not controlling the version of R being used, or the packages installed at .Library search path.

A toolchain to have better control will make the build more hermetic.

I think it was not available in the past, but now it is possible to run travis on OSX for Open Source projects.

Just got bit trying to set stamp & metadata on an r_package, but all it passes through is the srcs & deps.

I'm not sure if there's any particular reason not to just let the r_package take **kwargs & directly pass it along to its underlying r_pkg ?

Though honestly, looking at it, I probably would've written r_package purely in terms of kwargs. Then it's a 100% transparent drop-in replacement, plus you don't need to worry at all about updating the convenience wrapper when the real thing adds new options.

E.g.

def r_package(name, **kwargs):
    r_pkg(
        name = name,
        **kwargs,
    )
    r_library(
        name = "library",
        pkgs = [":%s" % name],
        tags = ["manual"],
    )

I wanted to use my R code to call a python code that is also built with bazel. Currently, I have to build them as two separate targets (a py_binary and a r_binary). But is there a way that I can put the py_binary as a deps for the r_binary, so that when I build the r_binary, it will automatically build the py_binary?

Hello,
It's looks like r_pkg rule is failing when WORKSPACE file is in the root of the package. Simple R package repo with such issue: https://github.com/knightdave/simpleR .

rules_r version: 0.5.9.tar.gz

❯ bazel --version
bazel 3.5.0
❯ bazel build --verbose_failures //...               
Starting local Bazel server and connecting to it...
INFO: Invocation ID: 164d6861-16b7-4791-92fc-7e51dbb3f51b
INFO: Analyzed target //:simpleR (18 packages loaded, 115 targets configured).
INFO: Found 1 target...
ERROR: /hidden/simpleR/BUILD.bazel:18:6: Building R package simpleR failed (Exit 1): build.sh failed: error executing command 
  (cd /hidden/.cache/bazel/_bazel_rycerzd/ff5bd6ed62b50ccee0dfeb7260c1c601/sandbox/linux-sandbox/4/execroot/__main__ && \
  exec env - \
    BAZEL_R_DEBUG=false \
    BAZEL_R_VERBOSE=false \
    BUILD_ARGS=''\''--no-build-vignettes'\'' '\''--no-manual'\''' \
    BUILD_TOOLS_EXPORT_CMD='export PATH' \
    CONFIG_OVERRIDE='' \
    C_CPP_FLAGS='' \
    C_LIBS_FLAGS='' \
    C_SO_FILES='' \
    EXPORT_ENV_VARS_CMD='' \
    FLOCK_PATH=bazel-out/host/bin/external/com_grail_rules_r/R/scripts/flock \
    INSTALL_ARGS='' \
    INSTRUMENTED=false \
    INSTRUMENT_SCRIPT=external/com_grail_rules_r/R/scripts/instrument.R \
    INST_FILES_MAP='' \
    METADATA_MAP='' \
    PKG_BIN_ARCHIVE=bazel-out/k8-fastbuild/bin/simpleR.bin.tar.gz \
    PKG_LIB_PATH=bazel-out/k8-fastbuild/bin/lib \
    PKG_NAME=simpleR \
    PKG_SRC_ARCHIVE=bazel-out/k8-fastbuild/bin/simpleR.tar.gz \
    PKG_SRC_DIR='' \
    R='/usr/bin/R --slave --no-restore --no-init-file --no-save --no-site-file --no-environ' \
    REQUIRED_VERSION='' \
    ROCLETS='' \
    RSCRIPT='/usr/bin/Rscript --no-init-file --no-save --no-site-file --no-environ' \
    R_LIBS_DEPS='' \
    R_LIBS_ROCLETS='' \
    R_MAKEVARS_SITE='' \
    R_MAKEVARS_USER='' \
    STATUS_FILES=bazel-out/volatile-status.txt \
  external/com_grail_rules_r/R/scripts/build.sh)
Execution platform: @local_config_platform//:host

Use --sandbox_debug to see verbose messages from the sandbox build.sh failed: error executing command 
  (cd /hidden/.cache/bazel/_bazel_rycerzd/ff5bd6ed62b50ccee0dfeb7260c1c601/sandbox/linux-sandbox/4/execroot/__main__ && \
  exec env - \
    BAZEL_R_DEBUG=false \
    BAZEL_R_VERBOSE=false \
    BUILD_ARGS=''\''--no-build-vignettes'\'' '\''--no-manual'\''' \
    BUILD_TOOLS_EXPORT_CMD='export PATH' \
    CONFIG_OVERRIDE='' \
    C_CPP_FLAGS='' \
    C_LIBS_FLAGS='' \
    C_SO_FILES='' \
    EXPORT_ENV_VARS_CMD='' \
    FLOCK_PATH=bazel-out/host/bin/external/com_grail_rules_r/R/scripts/flock \
    INSTALL_ARGS='' \
    INSTRUMENTED=false \
    INSTRUMENT_SCRIPT=external/com_grail_rules_r/R/scripts/instrument.R \
    INST_FILES_MAP='' \
    METADATA_MAP='' \
    PKG_BIN_ARCHIVE=bazel-out/k8-fastbuild/bin/simpleR.bin.tar.gz \
    PKG_LIB_PATH=bazel-out/k8-fastbuild/bin/lib \
    PKG_NAME=simpleR \
    PKG_SRC_ARCHIVE=bazel-out/k8-fastbuild/bin/simpleR.tar.gz \
    PKG_SRC_DIR='' \
    R='/usr/bin/R --slave --no-restore --no-init-file --no-save --no-site-file --no-environ' \
    REQUIRED_VERSION='' \
    ROCLETS='' \
    RSCRIPT='/usr/bin/Rscript --no-init-file --no-save --no-site-file --no-environ' \
    R_LIBS_DEPS='' \
    R_LIBS_ROCLETS='' \
    R_MAKEVARS_SITE='' \
    R_MAKEVARS_USER='' \
    STATUS_FILES=bazel-out/volatile-status.txt \
  external/com_grail_rules_r/R/scripts/build.sh)
Execution platform: @local_config_platform//:host

Use --sandbox_debug to see verbose messages from the sandbox
cp: cannot stat '/DESCRIPTION': No such file or directory
Target //:simpleR failed to build
INFO: Elapsed time: 8.045s, Critical Path: 0.27s
INFO: 3 processes: 3 remote cache hit.
FAILED: Build did NOT complete successfully

The same package is working when I'll put it to some subfolder.

See bazelbuild/bazel#4570 for progress on the API.

It is not quite ready yet, but this issue will track work on using the API to solve our long standing problem of obtaining PIC.

here is part of my BUILD file

PKG_NAME = "flowWorkspace"
PKG_CHECK_DEPS = [
    "@R_knitr//knitr:knitr",
    "@R_ggcyto//:ggcyto",
    
]
r_pkg(
    name = PKG_NAME,
    srcs = glob(
        ["**"],
        exclude = [
            "BUILD",
            "src/Makevars",
        ],
    ),
    cc_deps = [
        ":flowWorkspace_hdrs",
    ],
    config_override = ":empty",
    makevars = ":flowworkspace_makevars",
    pkg_name = "flowWorkspace",
    deps = [
        "@R_Biobase//Biobase",
        "@R_DelayedArray//DelayedArray",
        "@R_RBGL//RBGL",
        "@R_Rcpp//Rcpp",
        "@R_RcppParallel//RcppParallel",
         ],
)

r_library(
    name="library",
    pkgs = [PKG_NAME]
)

r_pkg_test(
    name = "check",
    timeout = "short",
    pkg = PKG_NAME,
    check_args = ["--no-codoc", "--no-manual", "--no-build-vignettes"],
    suggested_deps = PKG_CHECK_DEPS,
)

I have encountered a test error during R CMD check (invoked by bazel run packages/flowWorkspace:check )

I'd like to reproduce and troubleshoot the test error interactively at regular R console with devtools:test()
I know all the r packages and its deps can be copied to system R library path by running the r_library target (i.e. bazel run packages/flowWorkspace:library), but it will miss the packages defined in PKG_CHECK_DEPS.
I can't paste these to deps list of r_pkg rule since that will fail the bazel build due to some circular dependencies from suggests deps of r package.

I wonder if there is a convenient way to copy the packages listed in PKG_CHECK_DEPS (i.e. bazel-out/k8-fastbuild/bin/packages/flowWorkspace/check.runfiles/ I guess) to system R path instead of manually doing it.

Rule analogous to sh_binary but for the Rscript interpreter. Will have attributes srcs, data and r_pkg_deps.

Creating a tar copies all dependencies in the tar. This can be an expensive operation if done as part of an automated build of all library targets.

The current recommendation is to use the tag manual to skip building library targets.

Using the fact that in sandboxed builds, the runfiles are symlinks, we can let the runfiles be the package files, and on running the library target, copy the library directories into the destination, following symlinks.

The implementation of these rules expect a minimum of R 3.3. We should check for it on bootstrap and message the user accordingly.

Question about having an r_pkg depend on a cc_library through cc_deps.
cc_library has a includes argument that tells downstream dependencies how to search for and use its header files (according to my understanding).
But it seems that this is not passed on R CMD BUILD via r_pkg?
For example I am only seeing the includes related to pkg_deps not cc_deps.
The consequence for my build is that I can't compile against a pure C++ library dependency.
Normally for pure R, I would pass stuff in via configure_args I think, but I don't know the sandbox path ahead of time.

Here's my c++ dependency:compiles fine

cc_library(
    name = "cytolib",
    srcs = glob([
        "src/*.cpp"
    ]),
    hdrs = glob([
        "include/cytolib/*.hpp",
        "include/cytolib/*.h",
    ]),
    copts = [
        "-I.",
        "-std=c++17",
        "-fopenmp",
    ],
    linkopts = ["-lhdf5_serial"],
    include_prefix="cytolib",
    includes=["."],
    strip_include_prefix="include/cytolib",
    deps = [
        "@boost_hdrs//:boost",
        "@hdf5_hdrs//:hdf5",
        "@hdf5_lib//:libhdf5",
    ],
)

and here's my R package that depends on it.

r_pkg(
    name = "flowCore",
    srcs = glob(["**"], exclude=["BUILD"]),
    pkg_name = "flowCore",
    deps =  [
    '@R_Biobase//Biobase:Biobase',
    '@R_BiocGenerics//BiocGenerics:BiocGenerics',
    '@R_Rcpp//Rcpp:Rcpp',
    '@R_matrixStats//matrixStats:matrixStats',
    '@R_RcppArmadillo//RcppArmadillo:RcppArmadillo',
    '@R_S4Vectors//S4Vectors:S4Vectors',
    ],
    cc_deps = ["@cytolib_repo//packages/cytolib:cytolib"],
)

The compilation command I see is:

g++ -std=gnu++11 -I'/home/gfinak/.cache/bazel/_bazel_gfinak/838b2a13fcb4260193034d31efdec1fb/sandbox/linux-sandbox/238/execroot/ozette_mono_repo/bazel-out/k8-fastbuild/bin/packages/cytolib/cytolib' -I"/usr/local/lib/R/include" -DNDEBUG -DROUT -pthread -I/usr/local/include -I/usr/local/cytolib/include -I/usr/local/include/tiledb -DBOOST_NO_AUTO_PTR  -I'/home/gfinak/.cache/bazel/_bazel_gfinak/838b2a13fcb4260193034d31efdec1fb/sandbox/linux-sandbox/238/execroot/ozette_mono_repo/bazel-out/k8-fastbuild/bin/external/R_Rcpp/Rcpp/lib/Rcpp/include' -I'/home/gfinak/.cache/bazel/_bazel_gfinak/838b2a13fcb4260193034d31efdec1fb/sandbox/linux-sandbox/238/execroot/ozette_mono_repo/bazel-out/k8-fastbuild/bin/external/R_RcppArmadillo/RcppArmadillo/lib/RcppArmadillo/include' -I/usr/local/include -Wno-builtin-macro-redefined -D__DATE__="redacted" -D__TIMESTAMP__="redacted" -D__TIME__="redacted" -fdebug-prefix-map="/home/gfinak/.cache/bazel/_bazel_gfinak/838b2a13fcb4260193034d31efdec1fb/sandbox/linux-sandbox/238/execroot/ozette_mono_repo/="   -fpic  -g -O2  -c RcppExports.cpp -o RcppExports.o

with the error:

pairVectorRcppWrap.h:10:10: fatal error: cytolib/compensation.hpp: No such file or directory
   10 | #include <cytolib/compensation.hpp>

So it's not finding the header files for cytolib, since they don't seem to be passed on.

And what I would like / expect to see is:

g++ -std=gnu++11 -I'/home/gfinak/.cache/bazel/_bazel_gfinak/838b2a13fcb4260193034d31efdec1fb/sandbox/linux-sandbox/238/execroot/ozette_mono_repo/bazel-out/k8-fastbuild/bin/packages/cytolib/_virtual_includes/cytolib' -I"/usr/local/lib/R/include" -DNDEBUG -DROUT -pthread -I/usr/local/include -I/usr/local/cytolib/include -I/usr/local/include/tiledb -DBOOST_NO_AUTO_PTR  -I'/home/gfinak/.cache/bazel/_bazel_gfinak/838b2a13fcb4260193034d31efdec1fb/sandbox/linux-sandbox/238/execroot/ozette_mono_repo/bazel-out/k8-fastbuild/bin/external/R_Rcpp/Rcpp/lib/Rcpp/include' -I'/home/gfinak/.cache/bazel/_bazel_gfinak/838b2a13fcb4260193034d31efdec1fb/sandbox/linux-sandbox/238/execroot/ozette_mono_repo/bazel-out/k8-fastbuild/bin/external/R_RcppArmadillo/RcppArmadillo/lib/RcppArmadillo/include' -I/usr/local/include -Wno-builtin-macro-redefined -D__DATE__="redacted" -D__TIMESTAMP__="redacted" -D__TIME__="redacted" -fdebug-prefix-map="/home/gfinak/.cache/bazel/_bazel_gfinak/838b2a13fcb4260193034d31efdec1fb/sandbox/linux-sandbox/238/execroot/ozette_mono_repo/="   -fpic  -g -O2  -c RcppExports.cpp -o RcppExports.o

With the difference being in the first -I flag:

-I'/home/gfinak/.cache/bazel/_bazel_gfinak/838b2a13fcb4260193034d31efdec1fb/sandbox/linux-sandbox/238/execroot/ozette_mono_repo/bazel-out/k8-fastbuild/bin/packages/cytolib/_virtual_includes/cytolib'

Would love some insights if anyone has some.

This rule can be like an http_archive, and generate a default BUILD file for the package.

Modalities to obtain the R package can be, either exclusive or in order of preference,

1. local path
2. URLs (with optional sha256)
3. package name and version; the default repositories can be configured as a .Rprofile file through another WORKSPACE rule.

Running r_library targets should clean up the destination packages before copying new files there. Otherwise, previously installed files from inst/, etc. will continue to exist in the new install.

r_pkg_test rules do not seem to work correctly when the target r_pkg has a different name than pkg_name.

Given the following BUILD file for a package:

load("@com_grail_rules_r//R:defs.bzl", "r_pkg", "r_pkg_test")

r_pkg(
    name = "dummy",
    pkg_name = "<pkgname>",
    srcs = glob(
        ["**"],
        exclude = [
            "BUILD",
        ],
    )
)

r_pkg_test(
    name = "check",
    pkg = ":dummy",
)

I see the following when trying to run the check target (lightly edited for privacy):

bazel run //<pkgname>:check
Loading:
Loading: 0 packages loaded
Analyzing: target //<pkgname>:check (1 packages loaded, 0 targets configured)
INFO: Analyzed target //<pkgname>:check (27 packages loaded, 375 targets configured).
INFO: Found 1 target...
bazel: Entering directory `<snip>/execroot/<pkgname>/'
[0 / 2] [Prepa] BazelWorkspaceStatusAction stable-status.txt
bazel: Leaving directory `<snip>/execroot/<pkgname>/'
Target //<pkgname>:check up-to-date:
  bazel-bin/check
INFO: Elapsed time: 1.024s, Critical Path: 0.66s
INFO: 10 processes: 7 internal, 3 linux-sandbox.
INFO: Build completed successfully, 10 total actions
INFO: Running command line: external/bazel_tools/tools/test/test-setup.sh ./check
INFO: Build completed successfully, 10 total actions
exec ${PAGER:-/usr/bin/less} "$0" || exit 1
Executing tests from //:check
-----------------------------------------------------------------------------
* using log directory ‘<snip>/execroot/<pkgname>/bazel-out/k8-fastbuild/bin/check.runfiles/<pkgname>/dummy.Rcheck’
* using R version 3.6.3 (2020-02-29)
* using platform: x86_64-pc-linux-gnu (64-bit)
* using session charset: UTF-8
* checking package directory ... ERROR
package directory `<snip>/execroot/<pkgname>/bazel-out/k8-fastbuild/bin/check.runfiles/<pkgname>/dummy.Rcheck/00_pkg_src/dummy’ does not exist
* DONE

Status: 1 ERROR
See
  ‘<snip>/execroot/<pkgname>/bazel-out/k8-fastbuild/bin/check.runfiles/<pkgname>/dummy.Rcheck/00check.log’
for details.

This is almost certainly because R CMD check expects the tarball name to match the package name, but when using pkg_name this will not be the case.

It seems a package will only be properly created from files that are under an 'R' subdirectory.

This seems to go against the notion of Bazel; is there no way to enable a more flexible source structure?

The package layout and the symlink structure currently does not find package deps provided through the layers attribute. See image_test.sh.