I tried to build my nixfiles repo using nix-fast-build and got the following:

$ nix run github:Mic92/nix-fast-build
INFO:nix_fast_build:run nix-eval-jobs --gc-roots-dir /tmp/tmp213llfvo --force-recurse --max-memory-size 4096 --workers 8 --flake '.#checks'
warning: unknown setting 'allowed-users'
warning: unknown setting 'trusted-users'
error: cannot find flake 'flake:nixpkgs' in the flake registries
  building x86_64-linux.lint/nixpkgs-fmt
  building x86_64-linux.lint/deadnix
  building x86_64-linux.lint/statix
  building x86_64-linux.lint/gitleaks
  building x86_64-linux.lint/editorconfig-checker
  building x86_64-linux.lint/tofu-fmt
/nix/store/x8cz3lv4nn3yqnipw1fbsbql0mcp4pij-lint.gitleaks
/nix/store/5yjz8ilhs0azg6axv4w725k6m8jhc61p-lint.editorconfig-checker
/nix/store/0ak8ssa3ddzq4xs5fmziwwkddq7p7ll2-lint.tofu-fmt
/nix/store/xjs2xajl9m4hlkw4hqp0xn4a4dnrnp79-lint.statix
/nix/store/10wfqg315wjqzwywnbs7ispnbmj9mv9s-lint.nixpkgs-fmt
/nix/store/f7gzlqvv6bw47kmk9qj54y9yiylagzp5-lint.deadnix
ERROR:nix_fast_build:nix-output-monitor exited with 1

The error seems weird. I do indeed not have nixpkgs in my flake registry (I instead have a pinned version of nixpkgs under a different name). I think this error is coming from here:

Since nixpkgs in the flake registry is, by default, not pinned to any version I think nix-fast-build should not try to use anything from there and instead use the nix-output-monitor that is already in its closure.

Is this a bug or am I missing some reason for this?

Hi,

I'm using cachix, and I've been blindly doing ls result* | cachix push {cache} and just now realizing that the way nix-fast-build works seems to be clobbering result as it goes.

So, I've been getting lucky that my slowest host to build often includes most things.

It would be nice if I can easily access the things built by nix-fast-build after the fact. Maybe if it took the attr name from the eval and then passed it as --out-link to nix-build ?

I'm just curious why the derivation picks up Nix from nixpkgs here:

As opposed to just relying on the nix from $PATH.

It means that every time you want to run nix-fast-build, even if you already have Nix installed, you always need to fetch the entire nix closure from cache, or build it if it was overlayed/uncached.

Is there a strong dependency on the specific version of Nix the tool uses? I was thinking of overriding the build on my end to remove that bit.

If you construct a simple flake like this:

{
  inputs = {
    nixpkgs.url = "github:NixOS/nixpkgs/nixos-unstable";
    nix-fast-build.url = "github:Mic92/nix-fast-build";
    nix-fast-build.inputs.nixpkgs.follows = "nixpkgs";
  };

  outputs = { self, nixpkgs, nix-fast-build }: 
    let
      forAllSystems = nixpkgs.lib.genAttrs [
        "aarch64-darwin"
        "aarch64-linux"
        "x86_64-darwin"
        "x86_64-linux"
      ];
    in
  {
    devShells = forAllSystems (localSystem:
      let
        pkgs = import nixpkgs { inherit localSystem; };
      in
      {
        default = pkgs.mkShell {
          nativeBuildInputs = [
            nix-fast-build.packages.${localSystem}.nix-fast-build
          ];
        };
      }
    );
  };
}

You end up with python3 in your shell, which is unexpected:

$ which -a python3
/usr/bin/python3

$ nix develop --command which -a python3
/nix/store/rycxjkclx801wrhwrgllak0302xzjdvx-python3-3.11.4/bin/python3
/usr/bin/python3

This doesn't happen with other Python packages, for example:

$ nix-shell -p pre-commit --run "which -a python3"
/usr/bin/python3

After switching to nix-fast-build I've noticed much more unexplainable CI failures. Oftentimes they look like random network issues, but recently I've spotted some really odd ones.

From our CI:

post-build-hook: warning: unknown experimental feature 'fetch-tree'
running post-build-hook '/home/runner/work/_temp/cachix-daemon-KNU4uw/post-build-hook.sh'...
post-build-hook: warning: unknown experimental feature 'fetch-tree'
copying path '/nix/store/vbh4481bzdv7pnqq8m3d0jwz27c0qx0x-clang-wrapper-16.0.6' from 'https://cache.nixos.org/'...
building '/nix/store/pg3ilwm16il5b1cbr4dx4cv0a2fmvjr9-llvm-config.drv'...
running post-build-hook '/home/runner/work/_temp/cachix-daemon-KNU4uw/post-build-hook.sh'...
post-build-hook: warning: unknown experimental feature 'fetch-tree'

It's unclear to me what could be causing it. cachix-action? nix-fast-build?

When building an attributeset of derivations the result links are not stored individually. One randomly wins over the others. it would be good to either number them (result-1, result-2, ..) or have them follow the attribute names that are being built (result-a, result-foo, ..).

I have a CI job setup to auto-update-rebase my nixpkgs/home-manager forks, update my other flake inputs, and build. I haven't received any job failures recently.

Turns out, I had nixos-hardware set to an old fork and got bit by the rocm packages change. However, my CI jobs are all still green.

It seems that nix-fast-build did spit out an error, but must have exitted 0 at the end, since the script kept going and GHA didn't complain.

Is it possible to change this behavior, either by default or with a flag?

Hi,

It's looking like this could drastically reduce my current "wrapper" script of sorts, but one of the things my wrapper does is do local evaluations, push the derivations to a remote, and then SSHs and does the build+cachix call on the remote machine.

This avoids the issue (that I think is still an issue?) of Nix's behavior of building/pulling derivations one at a time to/from a remote.

Does nix-fast-build do this too? Or is it using the built-in remote builder functionality of nix?

nixci is built on devour-flake and seem similar to what you are trying to accomplish here.

I thought you may find it interesting, like to compare it, or potentially find interesting ideas from it.

It would be nice to support the --reference-lock-file functionality of the regular nix CLI for even better drop-in replacement, though I suppose nix-community/nix-eval-jobs#210 would need to land first

I can't figure out how to do it.

Is it possible using --option?

I'm looking at removing almost all of my "nix wrapper" script in favor of nix-fast-build.

One other thing that my wrapper script does is re-specify substituters and trusted-public-keys and ensure that the remote build is performed with them.

1. Does nix-fast-build do this? (If so, I can send a PR noting it in the README in the remote builder section)
2. If not, does it make sense to add such a feature?
3. (I wonder if nix has or should have a feature to print the net of global and user nix conf and use that?)

This tool works great for me in ci, but when running locally, I get the following error:

 error: opening lock file '/nix/store/ca0cw8qskng78k96g99ihmh3mmir4gzl-source.lock': Permission denied
error:

Not sure where to start debug this. I can give more details on my env if needed.

After I "fixed" my flake to make sure there are no evaluations that wouldn't be possible to build on supported system I still get:

> nix run github:Mic92/nix-fast-build -- --skip-cached --no-nom --systems "x86_64-linux"

              (stack trace truncated; use '--show-trace' to show the full trace)

              error: a 'x86_64-darwin' with features {} is required to build '/nix/store/09par5vs5qdw9ylw0ckf6wg3xi94gdg1-android-googletv-license.drv', but I am a 'x86_64-linux' with features {be
nchmark, big-parallel, kvm, nixos-test}

which suggest, that even on x86_64-linux, there are some reachable things somewhere that actually don't compile on it? Possibly they are coming from nixpkgs-android which is an intput I don't control.

I don't really hit these issues with nix flake check or nix build, and it's unclear to me is some problem of my flake,nixpkgs-android or how nix-fast-build does things.

One way or another, I'd rather disable evaluation of other systems and move on.

      run: >
          nix run github:Mic92/nix-fast-build
          -- --skip-cached --no-nom
          --flake
          ".#checks.aarch64-linux"

https://github.com/SomeoneSerge/llama.cpp/blob/998c9b2cf6c77f365b50b6645b41ff81eaedd450/.github/workflows/nix-ci.yml#L102-L106

...run on an x86_64-linux (ubuntu-latest) host, configured with extra-platforms = aarch64-linux

...results in the successful exit but 0 actual builds happening. The same command works fine if nix.conf instead contains system = aarch64-linux and extra-platforms: x86_64-linux

Encountered in ggerganov/llama.cpp#4709

Hi,

In addition to the hack I did for #35, I want to be able to make gcroots for the drvPaths, as nix-eval-jobs support. This is for CI on a self-hosted GHA runner box that is quickly filling up with disk space. I'd like to turn on auto garbage collection, but also want to be able to stash some gcroots for drvPaths and outPaths.

This leads to a thought of adding more fast to nix-fast-build to passthrough to nix-eval-jobs. But it seems it might be more elegant to have a mode where the user can pipe nix-eval-jobs themselves into nix-fast-build, so they can find tune eval/build parameters separately, as they see fit.

Any thoughts?