warp-build / warp Goto Github PK

We want to keep tricorders/toolchains pre-built for the architectures we support. We can do this by writing a pipeline that installs warp using our GitHub Action (see #38) and then executes the following warp pack commands:

warp pack https://rules.warp.build/toolchains/openssl --upload
warp pack https://rules.warp.build/toolchains/erlang --upload
warp pack https://rules.warp.build/toolchains/elixir --upload
warp pack https://rules.warp.build/toolchains/git --upload
warp pack https://rules.warp.build/toolchains/cmake --upload
warp pack https://rules.warp.build/toolchains/rebar3 --upload
warp pack tricorders/beam/mix.exs --upload
warp pack tricorders/rust/Cargo.toml --upload

This pipeline should run when main changes, and it will generate a commit with the new hashes for all the packaged things.

To be able to format all of your code using the formatters you've already configured (or the default ones if you haven't!)

In short, this command doesn't work:

warp build tricorders/rust/Cargo.toml

Because the Rust tricorder currently won't emit signatures for Cargo.toml files. A cargo_binary or cargo_library rule would suffice to make the tricorder build itself.

We can improve on the granularity of the caching here later.

This would fix the edge cases of #47 while also letting us test the rest of the system on any change to rules/manifests.

The main change to do here is make calls like RuleStore::get and PublicStore::try_fetch to return early after reading a file from disk. Namely, that'd be ./rules/{rules,toolchains}/** and ./store/**/manifest.json.

; warp build ./Makefile

This skips the Makefile since we don't have a tricorder for it yet, but it never finishes since its not pulled out of the queue correctly, and so the WorkerPool still believes there's work to do.

Found an example where MIX_ARCHIVE and MIX_HOME were sometimes being flipped around, resulting in an alternating hash for the Elixir toolchain.

At the moment we have a few places where Rule names are being handled as bare strings, which can be the short-name of the rule (like mix_release) or the full name (like https://rules.warp.build/rules/mix_release.js).

I think we should just use URLs for this and normalize as soon as we get a string from a Tricorder.

For testing purposes, the /warp/rules directory can't be trusted to contain the latest rules. We already use the --force-redownload flag to force it to download manifests for published artifacts. If we make this flag also apply to rules, and redownload them on every run, we can make sure to have the latest published rules at the time of executing the code.

Even better would be to have a flag to read rules from the current local code, but this would be a first step.

Currently we are missing a few things to get Warp to build with itself. We'll use this issue to keep track of those things.

• Features:
  • #26
  • #30
  • #37
  • #47
  • #48
• CI/CD:
  • #38
  • #32
  • #35
  • #29
  • #39
• Ecosystem Support:
  • #40

Some toolchains do not have a clear version that are readable from configuration files. One example of this is protoc, that usually is expected to be installed globally via brew or apt.

To do this, we should add support in the Warpfile to define toolchain versions:

{
  "toolchains": {
    "protobuf": "3.21.12"
  }
}

Hi team, this is a checklist of stuff that we should have in place before announcing the first release. Would love to hear your thoughts and if anything is missing here.

• CD:
  • CI passes for Linux and macOS
  • New releases are created on tags and published to Github Releases
• Docs:
  • README.md includes basic information about the project (see https://github.com/denoland/deno/blob/main/README.md)
  • docs.warp.build is deployed from main
  • Installation page shows how to install warp from sources
  • Concepts page explains our core concepts (Target, Signature, Tricorder, Phases, etc)
  • CONTRIBUTING.md includes instructions for
    • How to set up the project locally
    • How to contribute (PR etiquette, commit messages, etc)

Feel free to pick up any of this work, just make an issue for it and go bananas 🚀

cc/ @capitanu @diogomqbm @Enkronan

Just that, we already have the repo. Just gotta make it happen.

It should pull the latest version of the current architecture from the Github Releases API, as its done here: https://github.com/ostera/setup-protoc/blob/master/src/installer.ts#L187-L218

Since we have a complete graph of everything that's being built, we can provide a command that will list all the licenses that are being used.

Since we know how often we are accessing certain artifacts in the local store, we can keep track of the last access to allow for routine cleanups of the store.

This will help us keep our store size down to rather stable numbers.

Since we have the ./store folder in the same repo now, we can make warp pack update hashes of the manifests automatically. This will help us not forget to update them, but also aid in continuously delivering warp-packable things (like the Tricorders).

The gist of this is that after calling warp pack <target> we have access to all the build results, and also the Package that came out of the warp.pack method. This Package includes an internal manifest that includes everything we need to write to disk.

The only thing we need to make sure is that we are not modifying any of the hashes of other architectures.

So if you're running warp pack on Mac M1, that should only modify the aarch64-apple-darwin hashes.

I wanted to download the latest version of Git in the Public Store without having to built it locally if it wasn't available for my architecture. This command would do just that: download if possible, else just say we have to build it.

$ warp fetch https://rules.warp.build/toolchains/elixir
  Downloading https://rules.warp.build/toolchains/elixir
  Finished ...

$ warp fetch https://rules.warp.build/toolchains/git
  The target "https://rules.warp.build/toolchains/git" has never been built before.
  To build it run: warp build https://rules.warp.build/toolchains/git

When running the command below, we see all the right things are being handled BUT the counts of the different events are off.

We have built 5 things, and we have cached 5 things, but there is no mention of the 11 tests that were ran and cached.

; warp test --public-store-metadata-url http://localhost:9000 --force-redownload ./test/verl_SUITE.erl
     Started http://localhost:9000/tricorder/beam/manifest.json
    Readying http://localhost:9000/tricorder/beam/manifest.json
       Ready http://localhost:9000/tricorder/beam/manifest.json
   Cache-hit https://rules.warp.build/toolchains/openssl  
   Cache-hit https://rules.warp.build/toolchains/erlang  
   Cache-hit src/verl_parser.erl  
   Cache-hit src/verl.erl  
        PASS ./test/verl_SUITE.erl parse_test (CACHED)
        PASS ./test/verl_SUITE.erl compare_test (CACHED)
        PASS ./test/verl_SUITE.erl between_test (CACHED)
        PASS ./test/verl_SUITE.erl compile_requirement_test (CACHED)
        PASS ./test/verl_SUITE.erl ./test/verl_SUITE.erl (CACHED)
        PASS ./test/verl_SUITE.erl lte_test (CACHED)
        PASS ./test/verl_SUITE.erl lt_test (CACHED)
        PASS ./test/verl_SUITE.erl gt_test (CACHED)
        PASS ./test/verl_SUITE.erl is_match_test (CACHED)
        PASS ./test/verl_SUITE.erl parse_requirement_test (CACHED)
        PASS ./test/verl_SUITE.erl eq_test (CACHED)
        PASS ./test/verl_SUITE.erl gte_test (CACHED)
    Finished ./test/verl_SUITE.erl in 865ms (5 built, 5 cached)

What we'd expect to see would be:

    Finished ./test/verl_SUITE.erl in 865ms (11 passed, 0 errors, 5 built, 16 cached)

And also, one specific line where we built a library (the test/verl_SUITE.erl library) reads:

        PASS ./test/verl_SUITE.erl ./test/verl_SUITE.erl (CACHED)

where it should read:

   Cache-hit ./test/verl_SUITE.erl

The normpath crate has fixed a bunch of issues with Path manipulation that we'll need to be safe when shipping Windows binaries.

• We can run warp on a new repo and we'll automatically create a Warpfile for it. No complaints.
  • If we're in a subfolder, we'll go up until we find a .git folder that we believe is the root of the repo.
• We can call warp build @all and it'll just queue everything in the workspace.
• warp build @all honors .gitignore
• When running warp build @all, if we find unknown extensions we should just ignore them
• When the network is unreliable, we retry a few times before giving up

Right now on macOS this can lead to multiple volumes. A simple check for /warp's existance should be enough to skip this step.

Right now some of the glob expansion that Warp does, doesn't always follow the ignored files, so sometimes we pick up files that don't end up as part of the official sources.

This happened while pairing with @diogomqbm, and we found out that there was a folder in the BEAM tricorder that had just been ignored, but locally it was being picked up by warp and building.

Edge case, but good to consider this more generally.

Right now, you can only run warp and give a target from the root of the project. This needs to be extended to accept relative paths from the current directory.

We should probably use the invocation dir and the workspace root to figure out the difference and then construct a relative path to the root of the workspace.