Git Product home page Git Product logo

rapids-build-backend's Introduction

RAPIDS PEP517 build backend

rapids-build-backend is an adapter around PEP517 builders that provides support for key RAPIDS requirements. It currently support scikit-build-core and setuptools as the wrapped builder. The package's primary purpose is to automate the various bits of preprocessing that are typically done to RAPIDS package metadata prior to publishing packages. This includes the following notable changes:

  • Running rapids-dependency-file-generator to get the dependencies for the CUDA version and architecture.
  • Modifying the package name to include a CUDA suffix (e.g. "rmm" -> "rmm-cu11")
  • Updating the git commit embedded in the importable package.

Since some of these modifications are only desirable in certain scenarios (wheel vs conda builds vs editable installs), all of these functions are customizable via the project's configuration in pyproject.toml. In cases where more dynamic customization is sensible, suitable environment variables and config_settings are supported during builds of distributions.

Supported configuration

Any option without a default is required.

Option Definition Type Default Supports dynamic modification
build-backend The wrapped build backend (e.g. setuptools.build_meta) string N
commit-files List of files in which to write the git commit hash list[str] ["<project_name>/GIT_COMMIT"] N
dependencies-file The path to the dependencies.yaml file to use string "dependencies.yaml" Y
disable-cuda If true, CUDA version in build environment is ignored when setting package name and dependencies bool false Y
matrix-entry A ;-separated list of =-delimited key/value pairs string "" Y
requires List of build requirements (in addition to build-system.requires) list[str] [] N

Outstanding questions

  • How should we split up build requirements between build-system and tool.rapids-build-backend? In theory any dependency that doesn't need suffixing could also go into build-system.requires. I think it's easier to teach that all dependencies other than rapids-build-backend itself should to into tool.rapids-build-backend, but I don't know how others feel.

Rejected ideas

  • We could also include the rewrite of VERSION that we use for RAPIDS builds, but this is really more specific to our release process than the general process of building our wheels. I don't think someone building a wheel locally should see the same version as what we produce via CI. If we really wanted we could pull dunamai as a dependency and write a different version here, though.

rapids-build-backend's People

Contributors

vyasr avatar kylefromnvidia avatar bdice avatar jameslamb avatar raydouglass avatar

Stargazers

 avatar

Watchers

Mike Sarahan avatar Mike McCarty avatar  avatar  avatar  avatar

rapids-build-backend's Issues

Don't generate git commit file in editable installs

#3 manages git commits in a repo by generating a commit file. This file probably should not be generated for editable installs. It's inaccurate because the state of the code in an editable install is not static. A more minor issue is that the file itself is not technically editable; the only way it gets updates is by rerunning the build command, and there's no safe way to symlink the file at install time due to the nature of wheel building going through zip files.

Add long-form docs on configuration options

Description

The table in the README listing configuration options is useful to see what's available at a glance. It's also a great way to get an idea of what types of concerns rapids-build-backend is meant to address.

But the table format is really limiting... putting more than a few words describing a given configuration option would reduce the visual usefulness of the table fast (since it would grow the how table vertically).

At some point, I think it'd be useful to write dedicated documentation with long-form descriptions covering the following for each configuration option:

  • what are the valid values?
  • when might you want to override the default?
  • how do the options relate to each other?

Benefits of this work

Improves the likelihood that someone building with this backend could be productive on their own.

Acceptance Criteria

  • long form docs exist for all the config options

Approach

I am thinking something like a docs/config.rst, with content similar to this:

commit-file
'''''''''''''''''''

``rapids-build-backend`` automatically records the current git commit for the source tree it's building from, and writes that to a text file bundled into the wheel.

By default, that will be in a file `{[project].name}/GIT_COMMIT`. For example, given a `pyproject.toml` like this:

[project]
name = "sparkly-unicorn"

rapids-build-backend will write out that commit sha to a location ``./sparkly_unicorn/GIT_COMMIT``, relative to wherever that ``pyproject.toml`` file is.

Provide some other relative path here to override that behavior and tell it to write to a custom location.

I think it'd be fine for that to just live in a file in this repo. This issue could be considered "done", in my opinion, without setting up any docs rendering or hosting.

Some good references for this:

Notes

N/A

Add debugging messages and configurable verbosity

While working through rapidsai/rmm#1502, I've found myself struggling to debug problems of the form "rapids-build-backend isn't doing what I want, but also isn't raising any warnings or errors".

It'd be helpful to be able to turn on debug-level logging and see exactly what the library is doing.

For example, I'd like to be able to use logs to inspect what's happening in here:

for file_key, file_config in parsed_config.files.items():
if (
rapids_dependency_file_generator.Output.PYPROJECT
not in file_config.output
):
continue
pyproject_dir = os.path.join(
os.path.dirname(config.dependencies_file),
file_config.pyproject_dir,
)
if not (
os.path.exists(pyproject_dir)
and os.path.samefile(pyproject_dir, ".")
):
continue
matrix = _parse_matrix(config.matrix_entry) or dict(file_config.matrix)
if cuda_version is not None:
matrix["cuda"] = [f"{cuda_version[0]}.{cuda_version[1]}"]
rapids_dependency_file_generator.make_dependency_files(
parsed_config=parsed_config,
file_keys=[file_key],
output={rapids_dependency_file_generator.Output.PYPROJECT},
matrix=matrix,
prepend_channels=[],
to_stdout=False,
)

References

Some related conversations:

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.