evmar / n2 Goto Github PK
View Code? Open in Web Editor NEWn2 ("into"), a ninja compatible build system
License: Apache License 2.0
n2 ("into"), a ninja compatible build system
License: Apache License 2.0
This is mostly a "what if?" discussion, not really an issue, so feel free to close!
One thing that always bugs me about build systems is how they often bottom out at shell, which introduces implicit dependency on the environment: /bin/sh
might behave differently on different machines, and it doesn't exist on windows. There's also performance angle here --- there's a whole extra process between the compiler and the build system, and, for mostly no-op builds, I imagine it might add some overhead?
That's why I was a bit surprised to learn that n2 uses sh:
Line 85 in 599e162
I would have expected something like deno_task_shell:
https://github.com/denoland/deno_task_shell
That is, implementing minimal shell in n2 itself, to make sure that behavior is hermetic, cross platform, and doesn't require an extra process (of course, the users can themselves type sh -c
in the task definition if they need a shell).
I am wondering if there are some deep reasons to use the shell here which I might be missing? Is it perhaps "n2 wants to treat command as a string, without any interpretation, that works on windows and needs sh on Unix?"
Hi,
Another weird one (sorry!). After #78 was fixed I was able to build .cpp files from the LLVM repo on Windows, but I run into problems with ASM files, here is the error:
read lib\Support\BLAKE3\CMakeFiles\LLVMSupportBlake3.dir\blake3_avx2_x86-64_windows_msvc.asm.obj.d: The system cannot find the file specified. (os error 2)
Here is the command invocation:
C:\PROGRA~1\MIB055~1\2022\PROFES~1\VC\Tools\MSVC\1436~1.325\bin\Hostx64\x64\ml64.exe -DGTEST_HAS_RTTI=0 -DUNICODE -D_CRT_NONSTDC_NO_DEPRECATE -D_CRT_NONSTDC_NO_WARNINGS -D_CRT_SECURE_NO_DEPRECATE -D_CRT_SECURE_NO_WARNINGS -D_HAS_EXCEPTIONS=0 -D_SCL_SECURE_NO_DEPRECATE -D_SCL_SECURE_NO_WARNINGS -D_UNICODE -D__STDC_CONSTANT_MACROS -D__STDC_FORMAT_MACROS -D__STDC_LIMIT_MACROS -IC:\code\llvm\llvm-project\out\debug\lib\Support\BLAKE3 -IC:\code\llvm\llvm-project\llvm\lib\Support\BLAKE3 -IC:\code\llvm\llvm-project\out\debug\include -IC:\code\llvm\llvm-project\llvm\include -c -Fo lib\Support\BLAKE3\CMakeFiles\LLVMSupportBlake3.dir\blake3_avx2_x86-64_windows_msvc.asm.obj C:\code\llvm\llvm-project\llvm\lib\Support\BLAKE3\blake3_avx2_x86-64_windows_msvc.asm
failed: Building ASM_MASM object lib\Support\BLAKE3\CMakeFiles\LLVMSupportBlake3.dir\blake3_avx2_x86-64_windows_msvc.asm.obj
I am guessing that n2 tries to find the dependency file for the .asm file, but ml64 doesn't output (maybe no assemblers do?). Hopefully an easy fix.
I got the following when I tried to use n2 via a buildkite CI runner on macOS:
thread 'main' panicked at 'byte index 18446744073709551612 is out of bounds of 0s build:configure
', src/progress.rs:185:40
I presume it's getting a max columns of 0 and wrapping around. Forcing a minimum of 4 should fix the crash, but it might also be good to fall back on the default of 80 if it's 0.
Necessary for Windows builds using cl or clang-cl.
Ninja does a periodic "expire old data from the database" step. This bug is a reminder that n2 needs the same.
Hello,
Seeing the recent improvements, I wanted to update my clone of n2
, to run cargo install --path .
on it.
I have issues updating my clone due to the git-lfs
dependencies for the tests:
$ git pull --rebase
remote: Enumerating objects: 133, done.
remote: Counting objects: 100% (102/102), done.
remote: Compressing objects: 100% (46/46), done.
remote: Total 133 (delta 65), reused 82 (delta 52), pack-reused 31
Receiving objects: 100% (133/133), 54.01 KiB | 6.00 MiB/s, done.
Resolving deltas: 100% (74/74), completed with 19 local objects.
From https://github.com/evmar/n2
4bae87a..cc45ae4 main -> origin/main
* [new branch] bufwrite -> origin/bufwrite
Updating 4bae87a..cc45ae4
git-lfs filter-process: git-lfs: command not found
fatal: the remote end hung up unexpectedly
It's not easy for me to add git-lfs to my git install (which is custom built) and I don't plan on developing n2
for now (so I doubt I would make use of the snapshot.zip).
I will look for a way to get the update without git complaining but it's not immediately obvious to me.
I wanted to empty the tests/snapshot/.gitattributes
but I cannot do it before the rebase, and doing the rebase aborts in the middle because of the file.
I created this issue to let you know of this potential issue, but I understand it may be working as intended.
If you think it's worth supporting and have ideas on how to make the git-lfs dependency optional, I'm interested.
Ninja's console
pool has a few different behaviors:
Currently:
I think the implementation options here are either to do what Ninja does, which means giving direct console access and suppressing n2 status updates...
...or maybe try something fancy involving running the process with a pseudo tty (to make it believe 1.iii) and still rendering its progress live.
I think to do that we'd need to implement some terminal emulation behavior, but I expect there are two vague categories of tools here: the ones that just do some simple color printing and line overprinting, or the ones that go wild with complex screen updates. I think we could plausibly implement enough terminal emulation to handle the former and then bail to just letting the command run solo if we encounter any unexpected terminal escape codes. It's plausible to me that no commands actually depend on the latter. (Writing terminal emulators is a hobby of mine, maybe I am just too eager here...)
Forked from discussion in #68.
Distilled it to:
var = 123
rule custom
command = $cmd $var
build out: custom
cmd = echo $var hello
This runs echo hello 123
when it should run echo 123 hello 123
n2: error: build.ninja:29705: input source/2019/05/reykjavÃ-k.md missing
I assume this is related to this bit in the development doc.
@tru wrote:
Every time n2 tries to execute a subprocess, it exits with CreateProcessA: The parameter is incorrect. this happens already when running cmake, and also when trying to build a small build.ninja without involving cmake.
I have tested a small build.ninja that invokes clang-cl and it worked for me, so I am not sure what's different here.
I checked that in here: https://github.com/evmar/n2/tree/main/tests/manual/clang-cl
@tru could you try running n2 on that dir and let me know if it works?
Some other possibilities:
Next I'm eyeballing n2 code vs similar ninja code
In various places like #80 we relax the rules to accommodate some questionable Ninja behaviors.
We currently key some of these off a "ninja compat" flag, but I wonder if it would be better to make some sort of explicit mode flag that is extra strict.
I think there are basically two categories of users of n2:
This is very similar to the question of whether you want to enable -Werror
on a project, where you have the same two categories of people.
It would be great to have some sort of option like a build variable that could suppress the aggregated output in the success case, as while I love the extra insight during build, I'd miss the clean output if I had to decide on only one of the two.
The current n2 behavior is
The use case here is there are commands that display a lot of progress output (e.g. a test runner), but that progress output is ultimately not useful to show specifically in the case where the task succeeds. Another example of such a command is if a task is itself executing another build system (e.g. Cargo) that only displays build progress -- if the task's build succeeds we don't need to display anything.
We could do something like "only print a full log if the task fails", but also it seems plausible there are cases where you do want to see the full output. For example the tail of the output of LLVM's check-lld
task looks like
-- Testing: 2732 tests, 8 workers --
Testing: 0.. 10.. 20.. 30.. 40.. 50.. 60.. 70.. 80.. 90..
Testing Time: 9.99s
Unsupported: 2352
Passed : 380
Where I imagine it's good to display the "testing time" line.
So perhaps this is something to make configurable on a per-task basis?
I tried to build a moderately sized project with a high number of files copied and compiled by the build. I don't know how this buffer is used:
struct WriteBuf {
buf: [u8; 8192],
len: usize,
}
but it led to panicing halfway during the build. Bumping it to twice the size fixed the panics. Happy to gather more information, but the source tree I was building is unfortunately not public.
n2: error: parse error: unexpected character '@'
build.ninja:11795: ...s/[email protected] build/Resour...
@
is a valid path character on common modern filesystems, and ninja accepts it as well.
I'm guessing the bug is here, but I have no idea how these constants are determined so I can't file a PR:
fn is_path_char(c: u8) -> bool {
let lookup: [u64; 4] = [0x3fff80000000000, 0x7fffffe87fffffe, 0x0, 0x0];
(lookup[(c >> 6) as usize] & ((1 as u64) << (c & 63))) != 0
}
Thank you for your work on both ninja and this project - the simplicity and speed is great.
I considerably prefer the progress output of n2, so have been looking at moving our build over to it. The first issue I ran into was #39, and that was an easy fix (though resolving the variables at build.ninja creation time increased the build.ninja size from about 1.6M to 1.9MB).
The other issue took considerably longer to debug - I was seeing some rules being rebuilt on subsequent invocations of n2, and had no idea why. I couldn't resort to using ninja -d explain, as the issue didn't happen there, and n2 has no native explain tool. I ended up hacking together a solution that dumps the mtime of files on each run and compares them to make it clear what root change is triggering a build step to be rebuilt, and finally figured out that it wasn't the mtimes that were changing, but the order of the input and output files in build.ninja. Our build generator was writing the cmdlines consistently, but the inputs/outputs were being accumulated into hashmaps before being written out, leading to non-deterministic output.
Once I knew what the problem was, it was easy to do the sorting in the build generator, but I thought I would mention it in case you feel it's worth solving on n2's end for consistency with ninja, or at least documenting as a migration gotcha.
rule touch
command = touch $out
rule copy
command = cp $in $out
build out/a: copy ${my_dep}
my_dep = out/b
build out/b: touch
default out/a
This build file works in ninja, but not n2. Android has this case. I've only found one occurance that was trivial to remove so far, but I haven't gotten a full build working yet. If possible, maybe it's better not to fix it if it allows faster parsing.
Tried main
this morning and got this following error
error[E0499]: cannot borrow `*progress` as mutable more than once at a time
--> src/run.rs:57:21
|
30 | progress,
| -------- first mutable borrow occurs here
...
52 | work = work::Work::new(
| ---- first borrow might be used here, when `work` is dropped and runs the destructor for type `Work<'_>`
...
57 | progress,
| ^^^^^^^^ second mutable borrow occurs here
When building android, we get this panic:
thread 'main' panicked at src/db.rs:70:17:
oversized WriteBuf
stack backtrace:
0: std::panicking::begin_panic
at /rustc/a28077b28a02b92985b3a3faecf92813155f1ea1/library/std/src/panicking.rs:638:12
1: n2::db::Writer::write_build
2: n2::work::Work::record_finished
at /ssd/n2/src/work.rs:500:9
3: n2::work::Work::run
at /ssd/n2/src/work.rs:771:21
4: n2::run::build::{{closure}}
at /ssd/n2/src/run.rs:83:45
5: n2::trace::scope
at /ssd/n2/src/trace.rs:116:21
6: n2::run::build
at /ssd/n2/src/run.rs:83:17
7: n2::run::run_impl
at /ssd/n2/src/run.rs:216:11
8: n2::run::run
at /ssd/n2/src/run.rs:238:15
9: n2::main
at /ssd/n2/src/main.rs:2:27
10: core::ops::function::FnOnce::call_once
at /rustc/a28077b28a02b92985b3a3faecf92813155f1ea1/library/core/src/ops/function.rs:250:5
note: Some details are omitted, run with `RUST_BACKTRACE=full` for a verbose backtrace.
Android splits its build into 2 ninja invocations, a "bootstrap" invocation, and a main invocation. The bootstrap invocation generates the main build.ninja file, and it has a depfile that discovers all the Android.bp files. The depfile has 11842 entries, so it causes the WriteBuf to overflow.
I see there's a comment on WriteBuf about how we could use a BufWriter
instead, but it might be slightly less efficient. I'm thinking we should probably benchmark if that's the case. BufWriter
does have a with_capacity
constructor, so if we used the same buffer size I'm not sure how it would be any different from WriteBuf
.
In llvm there are a lot of build targets like check-lld
which run a large suite of lit tests. It's useful to see the progress of these commands since they can be very long running, and if something fails in the middle you might want to ctrl-c out of the build and triage that specific failure rather than wait for them all to be done.
We mark each build
with its line number for use in error messages, but computing the line number requires keeping track of each newline we encounter.
Some experiments:
build
to store the filename (error messages look like foo.ninja:123: some error
), but that appeared to be ~no impact on perf -- I imagine it's just a trivial increment of a non-atomic refcount for each build statementA trick I learned from LLVM is to instead annotate each build
with its file byte offset when parsing, which is faster to gather. Then if we encounter an error we can spend the time to re-parse the file for newlines to find the offset. Parsing is slow but it's like milliseconds slow, it's fine to do in an error message codepath.
Unfortunately doing this would mean we need to either keep around the build file text at runtime, or re-read the file when generating an error message.
If we kept the file text at runtime:
If we re-read the file when generating error messages:
Thanks a lot for reviewing so swiftly. I installed the most recent version of n2 and it now gives a similar warning to ninja, however it still fails to build (with a new error this time).
I can create a new issue with the new error, if you want
~/Coding/rr $ n2 --version
n2: error: Unrecognized option: 'version'
rm -rf build && mkdir build && cd build && cmake -G Ninja ../ && cd ..
-- The C compiler identification is GNU 7.5.0
-- The CXX compiler identification is GNU 7.5.0
-- The ASM compiler identification is GNU
-- Found assembler: /usr/bin/cc
-- Check for working C compiler: /usr/bin/cc
-- Check for working C compiler: /usr/bin/cc -- works
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Detecting C compile features
-- Detecting C compile features - done
-- Check for working CXX compiler: /usr/bin/c++
-- Check for working CXX compiler: /usr/bin/c++ -- works
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Detecting CXX compile features
-- Detecting CXX compile features - done
-- Performing Test SUPPORTS_CXX14
-- Performing Test SUPPORTS_CXX14 - Success
-- Found PkgConfig: /usr/bin/pkg-config (found version "0.29.1")
-- Checking for module 'capnp'
-- Found capnp, version 0.6.1
-- Checking for module 'zlib'
-- Found zlib, version 1.2.11
-- Found PythonInterp: /usr/bin/python3 (found suitable version "3.6.12", minimum required is "3")
You have called ADD_LIBRARY for library rrpage without any source files. This typically indicates a problem with your CMakeLists.txt file
You have called ADD_LIBRARY for library rrpreload without any source files. This typically indicates a problem with your CMakeLists.txt file
You have called ADD_LIBRARY for library rraudit without any source files. This typically indicates a problem with your CMakeLists.txt file
You have called ADD_LIBRARY for library rrpage_32 without any source files. This typically indicates a problem with your CMakeLists.txt file
You have called ADD_LIBRARY for library rrpreload_32 without any source files. This typically indicates a problem with your CMakeLists.txt file
You have called ADD_LIBRARY for library rraudit_32 without any source files. This typically indicates a problem with your CMakeLists.txt file
-- Configuring done
-- Generating done
-- Build files have been written to: /home/petr_tik/Coding/rr/build
~/Coding/rr $ RUST_BACKTRACE=full n2 -v -C build/
n2: warn: build.ninja:44074: "../src/preload/preload_interface.h" is repeated in output list
n2: warn: build.ninja:44074: "../src/preload/raw_syscall.S" is repeated in output list
n2: warn: build.ninja:44074: "../src/preload/rrcalls.h" is repeated in output list
thread 'main' panicked at 'expected no file state for ../src/preload/preload_interface.h', src/work.rs:575:17
stack backtrace:
0: 0x561288ee083a - std::backtrace_rs::backtrace::libunwind::trace::h91c465e73bf6c785
at /rustc/9d1b2106e23b1abd32fce1f17267604a5102f57a/library/std/src/../../backtrace/src/backtrace/libunwind.rs:93:5
1: 0x561288ee083a - std::backtrace_rs::backtrace::trace_unsynchronized::hae9da36f5d58b5f3
at /rustc/9d1b2106e23b1abd32fce1f17267604a5102f57a/library/std/src/../../backtrace/src/backtrace/mod.rs:66:5
2: 0x561288ee083a - std::sys_common::backtrace::_print_fmt::h7f499fa126a7effb
at /rustc/9d1b2106e23b1abd32fce1f17267604a5102f57a/library/std/src/sys_common/backtrace.rs:67:5
3: 0x561288ee083a - <std::sys_common::backtrace::_print::DisplayBacktrace as core::fmt::Display>::fmt::h3e2b509ce2ce6007
at /rustc/9d1b2106e23b1abd32fce1f17267604a5102f57a/library/std/src/sys_common/backtrace.rs:46:22
4: 0x561288e8101c - core::fmt::write::h753c7571fa063ecb
at /rustc/9d1b2106e23b1abd32fce1f17267604a5102f57a/library/core/src/fmt/mod.rs:1168:17
5: 0x561288eb6eef - std::io::Write::write_fmt::h2815c0519c99ba09
at /rustc/9d1b2106e23b1abd32fce1f17267604a5102f57a/library/std/src/io/mod.rs:1660:15
6: 0x561288ee18d2 - std::sys_common::backtrace::_print::h64941a6fc8b0ed9b
at /rustc/9d1b2106e23b1abd32fce1f17267604a5102f57a/library/std/src/sys_common/backtrace.rs:49:5
7: 0x561288ee18d2 - std::sys_common::backtrace::print::hcf25e43e1a9b0766
at /rustc/9d1b2106e23b1abd32fce1f17267604a5102f57a/library/std/src/sys_common/backtrace.rs:36:9
8: 0x561288ee18d2 - std::panicking::default_hook::{{closure}}::h78d3e6cf97fc623d
at /rustc/9d1b2106e23b1abd32fce1f17267604a5102f57a/library/std/src/panicking.rs:211:50
9: 0x561288ee26be - std::panicking::default_hook::hda898f8d3ad1a5ae
at /rustc/9d1b2106e23b1abd32fce1f17267604a5102f57a/library/std/src/panicking.rs:228:9
10: 0x561288ee26be - std::panicking::rust_panic_with_hook::h1a5ea2d6c23051aa
at /rustc/9d1b2106e23b1abd32fce1f17267604a5102f57a/library/std/src/panicking.rs:606:17
11: 0x561288ee21a8 - std::panicking::begin_panic_handler::{{closure}}::h07f549390938b73f
at /rustc/9d1b2106e23b1abd32fce1f17267604a5102f57a/library/std/src/panicking.rs:502:13
12: 0x561288ee2126 - std::sys_common::backtrace::__rust_end_short_backtrace::h5ec3758a92cfb00d
at /rustc/9d1b2106e23b1abd32fce1f17267604a5102f57a/library/std/src/sys_common/backtrace.rs:139:18
13: 0x561288ee20e2 - rust_begin_unwind
at /rustc/9d1b2106e23b1abd32fce1f17267604a5102f57a/library/std/src/panicking.rs:498:5
14: 0x561288e79380 - core::panicking::panic_fmt::h3a79a6a99affe1d5
at /rustc/9d1b2106e23b1abd32fce1f17267604a5102f57a/library/core/src/panicking.rs:116:14
15: 0x561288e9b606 - n2::work::Work::check_build_files_missing::hf6446c2d294e5e8f
16: 0x561288e9b606 - n2::work::Work::check_build_dirty::he52af68592e746be
at /home/petr_tik/.cargo/git/checkouts/n2-e6f499fa596a8a82/df0a27b/src/work.rs:590:28
17: 0x561288e9b606 - n2::work::Work::run_without_cleanup::h19d1906faad45644
at /home/petr_tik/.cargo/git/checkouts/n2-e6f499fa596a8a82/df0a27b/src/work.rs:671:21
18: 0x561288e953d5 - n2::work::Work::run::h249428d76b2080e1
at /home/petr_tik/.cargo/git/checkouts/n2-e6f499fa596a8a82/df0a27b/src/work.rs:741:22
19: 0x561288e953d5 - n2::run::build::{{closure}}::h2daab0ddc1bd4e0c
at /home/petr_tik/.cargo/git/checkouts/n2-e6f499fa596a8a82/df0a27b/src/run.rs:49:47
20: 0x561288e953d5 - n2::trace::scope::h2bf8a771a111bd3b
at /home/petr_tik/.cargo/git/checkouts/n2-e6f499fa596a8a82/df0a27b/src/trace.rs:116:21
21: 0x561288e953d5 - n2::run::build::h6dc9ded841a12ecc
at /home/petr_tik/.cargo/git/checkouts/n2-e6f499fa596a8a82/df0a27b/src/run.rs:49:19
22: 0x561288e91d71 - n2::run::run_impl::h0bb597f9f5680885
at /home/petr_tik/.cargo/git/checkouts/n2-e6f499fa596a8a82/df0a27b/src/run.rs:209:22
23: 0x561288e7e3d0 - n2::run::run::hd21211266b0e6a0b
at /home/petr_tik/.cargo/git/checkouts/n2-e6f499fa596a8a82/df0a27b/src/run.rs:234:15
24: 0x561288e7e3d0 - n2::main::h4c022cd59e4f0615
at /home/petr_tik/.cargo/git/checkouts/n2-e6f499fa596a8a82/df0a27b/src/main.rs:4:27
25: 0x561288e7e6f3 - core::ops::function::FnOnce::call_once::h25e67c454c9b4a24
at /rustc/9d1b2106e23b1abd32fce1f17267604a5102f57a/library/core/src/ops/function.rs:227:5
26: 0x561288e7e6f3 - std::sys_common::backtrace::__rust_begin_short_backtrace::he1b68df33ad7ec41
at /rustc/9d1b2106e23b1abd32fce1f17267604a5102f57a/library/std/src/sys_common/backtrace.rs:123:18
27: 0x561288e7ead9 - main
28: 0x7f30d8b8cc87 - __libc_start_main
at /build/glibc-uZu3wS/glibc-2.27/csu/../csu/libc-start.c:310
29: 0x561288e7e2da - _start
30: 0x0 - <unknown>
~/Coding/rr $ ninja -C build/
ninja: Entering directory `build/'
ninja: warning: multiple rules generate ../src/preload/preload_interface.h. builds involving this target will not be correct; continuing anyway [-w dupbuild=warn]
ninja: warning: multiple rules generate ../src/preload/raw_syscall.S. builds involving this target will not be correct; continuing anyway [-w dupbuild=warn]
ninja: warning: multiple rules generate ../src/preload/rrcalls.h. builds involving this target will not be correct; continuing anyway [-w dupbuild=warn]
[1/2405] Linking C executable bin/shared_map_32
[2/2405] Linking C executable bin/strict_priorities_32
[3/2405] Linking C executable bin/segfault_32
ninja: build stopped: interrupted by user.
Originally posted by @petr-tik in #44 (comment)
Bisecting, it looks like 6b4fb1a broke this. The failure is:
n2: error: parse error: expected '\n', got '.'
CMakeFiles/rules.ninja:119: rule CXX_COMPILER__obj.2eLLVMTableGenCom...
So it looks like between the mess that is llvm and #52 (which the breaking commit mentions it fixes) it's going to be a more complicated fix?
When I configure using CMake in Ninja mode, it first tests, if it is [...] able to compile a simple test program
. This Ninja implementation currently fails this test (at least for me on Windows). #36 and #35 already fixed a few Windows specific Problems for me. However this problem remains. Since I'm not that familiar with Ninja syntax yet, I hope you see what the issue is here...
> cmake -G Ninja -B build-msvc-debug
-- The C compiler identification is MSVC 19.31.31105.0
-- The CXX compiler identification is MSVC 19.31.31105.0
-- Check for working C compiler: C:/Program Files/Microsoft Visual Studio/2022/Community/VC/Tools/MSVC/14.31.31103/bin/Hostx64/x64/cl.exe
-- Check for working C compiler: C:/Program Files/Microsoft Visual Studio/2022/Community/VC/Tools/MSVC/14.31.31103/bin/Hostx64/x64/cl.exe - broken
CMake Error at C:/Users/cleme/AppData/Local/Programs/Python/Python310/Lib/site-packages/cmake/data/share/cmake-3.22/Modules/CMakeTestCCompiler.cmake:69 (message):
The C compiler
"C:/Program Files/Microsoft Visual Studio/2022/Community/VC/Tools/MSVC/14.31.31103/bin/Hostx64/x64/cl.exe"
is not able to compile a simple test program.
It fails with the following output:
Change Dir: C:/Users/cleme/dev/cpp/cmake/build-msvc-debug/CMakeFiles/CMakeTmp
Run Build Command(s):C:/PROGRA~1/Ninja/ninja.exe cmTC_b84ad && n2: error: parse error: unexpected whitespace
CMakeFiles\rules.ninja:25: command = C:\PROGRA~1\MICROS~2\2022\CO...
^
The rules.ninja file:
# CMAKE generated file: DO NOT EDIT!
# Generated by \"Ninja\" Generator, CMake Version 3.22
# This file contains all the rules used to get the outputs files
# built from the input files.
# It is included in the main 'build.ninja'.
# =============================================================================
# Project: CMAKE_TRY_COMPILE
# Configurations: Debug
# =============================================================================
# =============================================================================
#############################################
# localized /showIncludes string
msvc_deps_prefix = Note: including file:
#############################################
# Rule for compiling C files.
rule C_COMPILER__cmTC_6741b_Debug
deps = msvc
command = C:\\PROGRA~1\\MICROS~2\\2022\\COMMUN~1\\VC\\Tools\\MSVC\\1431~1.311\\bin\\Hostx64\\x64\\cl.exe /nologo $DEFINES $INCLUDES $FLAGS /showIncludes /Fo$out /Fd$TARGET_COMPILE_PDB /FS -c $in
description = Building C object $out
#############################################
# Rule for linking C executable.
rule C_EXECUTABLE_LINKER__cmTC_6741b_Debug
command = cmd.exe /C \"$PRE_LINK && C:\\Users\\cleme\\AppData\\Local\\Programs\\Python\\Python310\\Lib\\site-packages\\cmake\\data\\bin\\cmake.exe -E vs_link_exe --intdir=$OBJECT_DIR --rc=C:\\PROGRA~2\\WI3CF2~1\\10\\bin\\100220~1.0\\x64\\rc.exe --mt=C:\\PROGRA~2\\WI3CF2~1\\10\\bin\\100220~1.0\\x64\\mt.exe --manifests $MANIFESTS -- C:\\PROGRA~1\\MICROS~2\\2022\\COMMUN~1\\VC\\Tools\\MSVC\\1431~1.311\\bin\\Hostx64\\x64\\link.exe /nologo $in /out:$TARGET_FILE /implib:$TARGET_IMPLIB /pdb:$TARGET_PDB /version:0.0 $LINK_FLAGS $LINK_PATH $LINK_LIBRARIES && $POST_BUILD\"
description = Linking C executable $TARGET_FILE
restat = $RESTAT
#############################################
# Rule for running custom commands.
rule CUSTOM_COMMAND
command = $COMMAND
description = $DESC
#############################################
# Rule for cleaning all built files.
rule CLEAN
command = C:\\PROGRA~1\\Ninja\\ninja.exe $FILE_ARG -t clean $TARGETS
description = Cleaning all built files...
#############################################
# Rule for printing all primary targets available.
rule HELP
command = C:\\PROGRA~1\\Ninja\\ninja.exe -t targets
description = All primary targets available:
Evidence here:
https://github.com/evmar/n2/actions/runs/2086305718
Means that the Windows checks for #28 are also failing (as main is broken)
[Consolidating some comments from #107 and #108]
Currently we read input files into a local buffer. I had thought this was better than mmap because:
However, it turns out that you can just mmap a private page at the end of the file to write the nul, and @Colecf says "Judging based on n2's tracing output, just reading the file into memory takes over twice as long as the android fork's entire ninja invocation".
ninja
in the PATHcmake -GNinja ../llvm
ninja
-> cmake being re-builtThis might be something in the LLVM CMake scripts of course, but I wanted to report it anyway since using it for LLVM would give some good benefits.
# sudo apt install gcc-mingw-w64-x86-64-posix
rustup target add x86_64-pc-windows-gnu
cargo build --release --target=x86_64-pc-windows-gnu
Yields
Compiling jemallocator v0.3.2
Running `rustc --crate-name jemallocator /home/tamas/.cargo/registry/src/github.com-1ecc6299db9ec823/jemallocator-0.3.2/src/lib.rs --error-format=json --json=diagnostic-rendered-ansi,artifacts,future-incompat --crate-type lib --emit=dep-info,metadata,link -C embed-bitcode=no -C debuginfo=2 --cfg 'feature="background_threads_runtime_support"' --cfg 'feature="default"' -C metadata=c1aa0409bbfdcf4d -C extra-filename=-c1aa0409bbfdcf4d --out-dir /home/tamas/src/n2/target/x86_64-pc-windows-gnu/debug/deps --target x86_64-pc-windows-gnu -L dependency=/home/tamas/src/n2/target/x86_64-pc-windows-gnu/debug/deps -L dependency=/home/tamas/src/n2/target/debug/deps --extern jemalloc_sys=/home/tamas/src/n2/target/x86_64-pc-windows-gnu/debug/deps/libjemalloc_sys-f9a3d890e565352c.rmeta --extern libc=/home/tamas/src/n2/target/x86_64-pc-windows-gnu/debug/deps/liblibc-d6282ed627c45757.rmeta --cap-lints allow -L native=/home/tamas/src/n2/target/x86_64-pc-windows-gnu/debug/build/jemalloc-sys-caac9a3b2c7387ed/out/build/lib`
error: could not find native static library `jemalloc`, perhaps an -L flag is missing?
error: could not compile `jemalloc-sys` due to previous error
It looks like this is an issue with the jemallocator crate. The crate does seem to build all the way but perhaps the linker path is not set correctly when linking to the C library.
I don't really know if this is easily solvable on the n2 side (apart from switching to a different allocator).
I have a Ryzen 3950x with 16 cores and 32 threads. Just running n2
with no -j
arg selected 8 workers. Ninja selects 34 on the same system.
I plan to package n2 for Gentoo Linux to provide support for all build systems that can build Anki.
Is there any plan to tag a release (candidate) any time soon? That would simplify versioning the ebuild.
Lines 11 to 12 in fe6c2d5
There are crates that implement this kind of string. For example, bstr
Running n2 on a project that currently uses ninja, I get:
n2: error: no path specified and no default
I think ninja
just builds all targets if it doesn't see a default?
rule touch
command = touch ${out}
build a: touch
build b: touch
$ ninja -v
[1/2] touch a
[2/2] touch b
$ n2
n2: error: no path specified and no default
$ readlink /usr/lib/n2/bin/ninja
/usr/bin/n2
$ /usr/lib/n2/bin/ninja
n2: error: no path specified and no default
Command::spawn()
in the rust stdlib unconditionally calls anon_pipe
here: https://github.com/rust-lang/rust/blob/master/library/std/src/sys/unix/process/process_unix.rs#L62
anon_pipe
on Linux calls pipe2
to set CLOEXEC
on the pipe atomically: https://github.com/rust-lang/rust/blob/521734787ecf80ff12df7ca5998f7ec0b3b7b2c9/library/std/src/sys/unix/pipe.rs#L18
But macOS has no pipe2
, so here the stdlib instead calls pipe()
followed by set_cloexec
: https://github.com/rust-lang/rust/blob/521734787ecf80ff12df7ca5998f7ec0b3b7b2c9/library/std/src/sys/unix/pipe.rs#L35
This means there's a window where the pipe is created but cloexec isn't set on the pipe's FDs yet. If a different thread forks in that window, the pipe's fds get leaked.
(Not that it matters, but set_cloexec
is here https://github.com/rust-lang/rust/blob/master/library/std/src/sys/unix/fd.rs#L204 . On macOS and a few other OSs, it calls ioctl
with FIOCLEX
, which is just 1 syscall. On Linux and a few others, it calls fcntl
with F_GETFD
/F_SETFD
which is 2 syscalls, but we don't need to call set_cloexec
at all in this path on Linux.)
This is likely why n2 -j250
runs out of FDs on macOS, while ninja -j250
works fine.
Possible fixes:
Don't fork on background threads. Create subprocess on main thread, and then hand the created subprocess to background thread for waiting on it and to do deps processing
Don't use threads per subprocesses at all, use a select loop model instead (but still use threads for deps processing)
Don't use the Command
object on non-Windows either (we already don't on Windows), and add mutexes around fork
and pipe/set_cloexec
Spawning subprocesses takes some time (we made it async in chromium at some point for that reason we looked at making it async in ninja for that reason), so moving it to the main thread would add some amount of work on to the critical path of n2.
I noticed that the implementation for StackStack could lead to undefined behavior if any of the values have a drop implementation. I think you need to wrap the types in ManuallyDrop to prevent the drop impls from being called on uninitialized values, or use one of the array-backed vec implementations, like https://docs.rs/arrayvec/latest/arrayvec/struct.ArrayVec.html, which implements this pattern for you.
Hi again! Now that I have it building it seems like something is broken on windows, I configured llvm with:
cmake -GNinja -DCMAKE_TOOLCHAIN_FILE=LLVMToolchain.cmake -DCMAKE_BUILD_TYPE=Debug -DLLVM_ENABLE_PROJECTS="clang;lld" -DLLVM_TARGETS_TO_BUILD=Native ..\llvm
with n2
installed as ninja
in the path.
When running just ninja
it complains about the all
target and if I run ninja lld
it will print the includes on the terminal and then hang and not progress. So it seems like parsing the include files doesn't work correctly?
I am using clang-cl to build llvm.
Example:
rule touch
command = touch $out
build foo: $
touch
Output:
$ ninja
[1/1] touch foo
$ n2
n2: error: parse error: failed to scan ident
build.ninja:3: build foo: $
^
Would be nice to fix, ninja_syntax.py really likes to use wrapped newlines.
Especially when my PC is under some load. A race condition maybe?
---- e2e::validations::build_starts_before_validation_finishes stdout ----
thread 'e2e::validations::build_starts_before_validation_finishes' panicked at tests/e2e/validations.rs:55:5:
assertion failed: space.read("out").is_ok()
stack backtrace:
0: rust_begin_unwind
at /rustc/0e09125c6c3c2fd70d7de961bcf0e51575235fad/library/std/src/panicking.rs:645:5
1: core::panicking::panic_fmt
at /rustc/0e09125c6c3c2fd70d7de961bcf0e51575235fad/library/core/src/panicking.rs:72:14
2: core::panicking::panic
at /rustc/0e09125c6c3c2fd70d7de961bcf0e51575235fad/library/core/src/panicking.rs:144:5
3: e2e_test::e2e::validations::build_starts_before_validation_finishes
at ./tests/e2e/validations.rs:55:5
4: e2e_test::e2e::validations::build_starts_before_validation_finishes::{{closure}}
at ./tests/e2e/validations.rs:24:49
5: core::ops::function::FnOnce::call_once
at /rustc/0e09125c6c3c2fd70d7de961bcf0e51575235fad/library/core/src/ops/function.rs:250:5
6: core::ops::function::FnOnce::call_once
at /rustc/0e09125c6c3c2fd70d7de961bcf0e51575235fad/library/core/src/ops/function.rs:250:5
note: Some details are omitted, run with `RUST_BACKTRACE=full` for a verbose backtrace.
Running x86_64 Arch Linux 6.6.10 on AMD Ryzen 5 4600H.
Rust 1.76.0-beta.1 (1a2666ddd 2023-12-17).
Can also reproduce on stable 1.75.0 (1d8b05cdd 2023-11-20).
Hi @evmar, would you welcome CI PRs to:
Or you are fine without these things, which is also fine? Happy to send any PRs that are welcomed
On Windows paths can contain either /
or \
. It's important for n2 to recognize all references to a file are the same, even if they're written in different ways. I think the failure modes that can come up are like
foo\bar.h
/showIncludes
) it's written foo/bar.h
It doesn't work to just always canonicalize paths to unix-style because a foo\bar.h
in a build line might forward in to a command line to a tool that doesn't understand foo/bar.h
. I believe because of this Ninja goes to effort to keep track of where the slashes were in a path.
Here's an idea I had: what if we preserved the slashes found in the input, but then when looking a path string up we allowed back and forward slashes to both match the same File
? Basically tweak this around here, at the hash lookup time?
@sgraham do you remember this stuff?
Hey, thanks for the awesome project.
A friend/coworker of mine contributed recently and I wanted to try n2 on a decent-sized cpp project -
https://github.com/rr-debugger/rr/
Here is my repro for the bug in the issue title
~/Coding/rr $ cargo install --git https://github.com/evmar/n2
Updating git repository `https://github.com/evmar/n2`
Installing n2 v0.1.0 (https://github.com/evmar/n2#5754141e)
Updating crates.io index
Downloaded libc v0.2.122
Downloaded 1 crate (577.4 KB) in 0.48s
Compiling fs_extra v1.2.0
Compiling libc v0.2.122
Compiling cc v1.0.73
Compiling anyhow v1.0.56
Compiling unicode-width v0.1.9
Compiling lazy_static v1.4.0
Compiling getopts v0.2.21
Compiling jemalloc-sys v0.3.2
Compiling jemallocator v0.3.2
Compiling n2 v0.1.0 (/home/petr_tik/.cargo/git/checkouts/n2-e6f499fa596a8a82/5754141)
Finished release [optimized + debuginfo] target(s) in 29.26s
Installing /home/petr_tik/.cargo/bin/n2
Installed package `n2 v0.1.0 (https://github.com/evmar/n2#5754141e)` (executable `n2`)
Cd (or git clone from git clone https://github.com/rr-debugger/rr.git) and follow the build instructions from
https://github.com/rr-debugger/rr/wiki/Building-And-Installing#project-building
~/Coding/ $ git clone https://github.com/rr-debugger/rr.git && cd rr
~/Coding/rr $ git log --oneline -n1
4aca8b4d (HEAD, upstream/master, upstream/HEAD) Forward SIGTERM to detached tasks immediately
~/Coding/rr $ rm -rf build/
~/Coding/rr $ # follow this https://github.com/rr-debugger/rr/wiki/Building-And-Installing#project-building but call the build dir "build"
~/Coding/rr $ mkdir build && cd build && cmake -G Ninja ../
-- The C compiler identification is GNU 7.5.0
-- The CXX compiler identification is GNU 7.5.0
-- The ASM compiler identification is GNU
-- Found assembler: /usr/bin/cc
-- Check for working C compiler: /usr/bin/cc
-- Check for working C compiler: /usr/bin/cc -- works
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Detecting C compile features
-- Detecting C compile features - done
-- Check for working CXX compiler: /usr/bin/c++
-- Check for working CXX compiler: /usr/bin/c++ -- works
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Detecting CXX compile features
-- Detecting CXX compile features - done
-- Performing Test SUPPORTS_CXX14
-- Performing Test SUPPORTS_CXX14 - Success
-- Found PkgConfig: /usr/bin/pkg-config (found version "0.29.1")
-- Checking for module 'capnp'
-- Found capnp, version 0.6.1
-- Checking for module 'zlib'
-- Found zlib, version 1.2.11
-- Found PythonInterp: /usr/bin/python3 (found suitable version "3.6.12", minimum required is "3")
You have called ADD_LIBRARY for library rrpage without any source files. This typically indicates a problem with your CMakeLists.txt file
You have called ADD_LIBRARY for library rrpreload without any source files. This typically indicates a problem with your CMakeLists.txt file
You have called ADD_LIBRARY for library rraudit without any source files. This typically indicates a problem with your CMakeLists.txt file
You have called ADD_LIBRARY for library rrpage_32 without any source files. This typically indicates a problem with your CMakeLists.txt file
You have called ADD_LIBRARY for library rrpreload_32 without any source files. This typically indicates a problem with your CMakeLists.txt file
You have called ADD_LIBRARY for library rraudit_32 without any source files. This typically indicates a problem with your CMakeLists.txt file
-- Configuring done
-- Generating done
-- Build files have been written to: /home/petr_tik/Coding/rr/build
~/Coding/rr/build $ cd ../
Check the number of lines in the build.ninja file and start a build with n2 until you get an error
~/Coding/rr $ wc -l build/build.ninja
44084 build/build.ninja
~/Coding/rr $ n2 -v -C build
thread 'main' panicked at 'index out of bounds: the len is 4978 but the index is 4978', src/densemap.rs:25:10
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
~/Coding/rr $ RUST_BACKTRACE=full n2 -v -C build
thread 'main' panicked at 'index out of bounds: the len is 4978 but the index is 4978', src/densemap.rs:25:10
stack backtrace:
0: 0x5615872ed76a - std::backtrace_rs::backtrace::libunwind::trace::h91c465e73bf6c785
at /rustc/9d1b2106e23b1abd32fce1f17267604a5102f57a/library/std/src/../../backtrace/src/backtrace/libunwind.rs:93:5
1: 0x5615872ed76a - std::backtrace_rs::backtrace::trace_unsynchronized::hae9da36f5d58b5f3
at /rustc/9d1b2106e23b1abd32fce1f17267604a5102f57a/library/std/src/../../backtrace/src/backtrace/mod.rs:66:5
2: 0x5615872ed76a - std::sys_common::backtrace::_print_fmt::h7f499fa126a7effb
at /rustc/9d1b2106e23b1abd32fce1f17267604a5102f57a/library/std/src/sys_common/backtrace.rs:67:5
3: 0x5615872ed76a - <std::sys_common::backtrace::_print::DisplayBacktrace as core::fmt::Display>::fmt::h3e2b509ce2ce6007
at /rustc/9d1b2106e23b1abd32fce1f17267604a5102f57a/library/std/src/sys_common/backtrace.rs:46:22
4: 0x56158728dfcc - core::fmt::write::h753c7571fa063ecb
at /rustc/9d1b2106e23b1abd32fce1f17267604a5102f57a/library/core/src/fmt/mod.rs:1168:17
5: 0x5615872c3e1f - std::io::Write::write_fmt::h2815c0519c99ba09
at /rustc/9d1b2106e23b1abd32fce1f17267604a5102f57a/library/std/src/io/mod.rs:1660:15
6: 0x5615872ee802 - std::sys_common::backtrace::_print::h64941a6fc8b0ed9b
at /rustc/9d1b2106e23b1abd32fce1f17267604a5102f57a/library/std/src/sys_common/backtrace.rs:49:5
7: 0x5615872ee802 - std::sys_common::backtrace::print::hcf25e43e1a9b0766
at /rustc/9d1b2106e23b1abd32fce1f17267604a5102f57a/library/std/src/sys_common/backtrace.rs:36:9
8: 0x5615872ee802 - std::panicking::default_hook::{{closure}}::h78d3e6cf97fc623d
at /rustc/9d1b2106e23b1abd32fce1f17267604a5102f57a/library/std/src/panicking.rs:211:50
9: 0x5615872ef5ee - std::panicking::default_hook::hda898f8d3ad1a5ae
at /rustc/9d1b2106e23b1abd32fce1f17267604a5102f57a/library/std/src/panicking.rs:228:9
10: 0x5615872ef5ee - std::panicking::rust_panic_with_hook::h1a5ea2d6c23051aa
at /rustc/9d1b2106e23b1abd32fce1f17267604a5102f57a/library/std/src/panicking.rs:606:17
11: 0x5615872ef0d8 - std::panicking::begin_panic_handler::{{closure}}::h07f549390938b73f
at /rustc/9d1b2106e23b1abd32fce1f17267604a5102f57a/library/std/src/panicking.rs:502:13
12: 0x5615872ef056 - std::sys_common::backtrace::__rust_end_short_backtrace::h5ec3758a92cfb00d
at /rustc/9d1b2106e23b1abd32fce1f17267604a5102f57a/library/std/src/sys_common/backtrace.rs:139:18
13: 0x5615872ef012 - rust_begin_unwind
at /rustc/9d1b2106e23b1abd32fce1f17267604a5102f57a/library/std/src/panicking.rs:498:5
14: 0x561587286330 - core::panicking::panic_fmt::h3a79a6a99affe1d5
at /rustc/9d1b2106e23b1abd32fce1f17267604a5102f57a/library/core/src/panicking.rs:116:14
15: 0x5615872862f1 - core::panicking::panic_bounds_check::h449d4ff4d992b84f
at /rustc/9d1b2106e23b1abd32fce1f17267604a5102f57a/library/core/src/panicking.rs:84:5
16: 0x5615872b7b8d - <usize as core::slice::index::SliceIndex<[T]>>::index::h4157e3ec92048f2b
at /rustc/9d1b2106e23b1abd32fce1f17267604a5102f57a/library/core/src/slice/index.rs:189:10
17: 0x5615872b7b8d - core::slice::index::<impl core::ops::index::Index<I> for [T]>::index::h67da73f717a04c6c
at /rustc/9d1b2106e23b1abd32fce1f17267604a5102f57a/library/core/src/slice/index.rs:15:9
18: 0x5615872b7b8d - <alloc::vec::Vec<T,A> as core::ops::index::Index<I>>::index::h4d5671bbb1ca6714
at /rustc/9d1b2106e23b1abd32fce1f17267604a5102f57a/library/alloc/src/vec/mod.rs:2520:9
19: 0x5615872b7b8d - n2::densemap::DenseMap<K,V>::get::h3cfc2bb3a9c6d350
at /home/petr_tik/.cargo/git/checkouts/n2-e6f499fa596a8a82/5754141/src/densemap.rs:25:10
20: 0x5615872b7b8d - n2::graph::Graph::add_build::h1964147cca1ce967
at /home/petr_tik/.cargo/git/checkouts/n2-e6f499fa596a8a82/5754141/src/graph.rs:256:25
21: 0x5615872b7b8d - n2::load::Loader::add_build::h08ca299dbe63baac
at /home/petr_tik/.cargo/git/checkouts/n2-e6f499fa596a8a82/5754141/src/load.rs:137:9
22: 0x5615872b7b8d - n2::load::Loader::parse::h163c7ecc8e2b7ce5
at /home/petr_tik/.cargo/git/checkouts/n2-e6f499fa596a8a82/5754141/src/load.rs:171:44
23: 0x5615872b7b8d - n2::load::Loader::read_file::hc4531311fe85d5d9
at /home/petr_tik/.cargo/git/checkouts/n2-e6f499fa596a8a82/5754141/src/load.rs:146:9
24: 0x5615872a39fe - n2::trace::scope::hcd7c61c7107b5186
at /home/petr_tik/.cargo/git/checkouts/n2-e6f499fa596a8a82/5754141/src/trace.rs:116:21
25: 0x5615872a39fe - n2::load::read::h697ad8d9a113c44c
at /home/petr_tik/.cargo/git/checkouts/n2-e6f499fa596a8a82/5754141/src/load.rs:193:5
26: 0x5615872a39fe - n2::run::build::{{closure}}::hb2a3de0e31f57edb
at /home/petr_tik/.cargo/git/checkouts/n2-e6f499fa596a8a82/5754141/src/run.rs:33:51
27: 0x5615872a1bf0 - n2::trace::scope::hae3aefe50621511a
at /home/petr_tik/.cargo/git/checkouts/n2-e6f499fa596a8a82/5754141/src/trace.rs:116:21
28: 0x5615872a1bf0 - n2::run::build::h6dc9ded841a12ecc
at /home/petr_tik/.cargo/git/checkouts/n2-e6f499fa596a8a82/5754141/src/run.rs:33:21
29: 0x56158729ed21 - n2::run::run_impl::h0bb597f9f5680885
at /home/petr_tik/.cargo/git/checkouts/n2-e6f499fa596a8a82/5754141/src/run.rs:209:22
30: 0x56158728b380 - n2::run::run::hd21211266b0e6a0b
at /home/petr_tik/.cargo/git/checkouts/n2-e6f499fa596a8a82/5754141/src/run.rs:234:15
31: 0x56158728b380 - n2::main::h4c022cd59e4f0615
at /home/petr_tik/.cargo/git/checkouts/n2-e6f499fa596a8a82/5754141/src/main.rs:4:27
32: 0x56158728b6a3 - core::ops::function::FnOnce::call_once::h25e67c454c9b4a24
at /rustc/9d1b2106e23b1abd32fce1f17267604a5102f57a/library/core/src/ops/function.rs:227:5
33: 0x56158728b6a3 - std::sys_common::backtrace::__rust_begin_short_backtrace::he1b68df33ad7ec41
at /rustc/9d1b2106e23b1abd32fce1f17267604a5102f57a/library/std/src/sys_common/backtrace.rs:123:18
34: 0x56158728ba89 - main
35: 0x7efddc4f1c87 - __libc_start_main
at /build/glibc-uZu3wS/glibc-2.27/csu/../csu/libc-start.c:310
36: 0x56158728b28a - _start
37: 0x0 - <unknown>
~/Coding/rr $ ninja -C build
ninja: Entering directory `build'
ninja: warning: multiple rules generate ../src/preload/preload_interface.h. builds involving this target will not be correct; continuing anyway [-w dupbuild=warn]
ninja: warning: multiple rules generate ../src/preload/raw_syscall.S. builds involving this target will not be correct; continuing anyway [-w dupbuild=warn]
ninja: warning: multiple rules generate ../src/preload/rrcalls.h. builds involving this target will not be correct; continuing anyway [-w dupbuild=warn]
[1/2533] Building C object CMakeFiles/rr_exec_stub_32.dir/32/exec_stub.c.o
[2/2533] Building C object CMakeFiles/starvation_multithreaded.dir/src/chaos-test/starvation_multithreaded.c.o
[3/2533] Building C object CMakeFiles/brotli.dir/third-party/brotli/common/transform.c.o
[4/2533] Building C object CMakeFiles/core_count.dir/src/chaos-test/core_count.c.o
[5/2533] Building C object CMakeFiles/brotli.dir/third-party/brotli/dec/bit_reader.c.o
[6/2533] Building C object CMakeFiles/mmap_bits.dir/src/chaos-test/mmap_bits.c.o
[7/2533] Building C object CMakeFiles/starvation_singlethreaded.dir/src/chaos-test/starvation_singlethreaded.c.o
[8/2533] Linking C executable bin/rr_exec_stub_32
[9/2533] Building C object CMakeFiles/brotli.dir/third-party/brotli/dec/state.c.o
# Ctrl-C the process here
Hope this helps! Let me know if you need anything else.
ninja-build/ninja#1578
ninja-build/ninja#977
This is a proposed (and accepted but eternally unmerged) feature of mainline's subninja
that translates all of the paths of the subninja to a root directory.
The way ninja does relative path resolution right now with subninja is a bit aggravating and makes it impossible to support chaining build systems in a generator-agnostic manner without severe performance hits.
Example (with Ninja) - https://lwn.net/Articles/706404/
rule cc
command = gcc -c -o $out $in -MMD -MF $out.d
depfile = $out.d
deps = gcc
In my case, it produces an error:
clang: error: argument to '-MF' is missing (expected 1 value)
Android's ninja fork added a feature somewhat recently where long-running actions (using information from a prior build) would be prioritized more highly than quicker actions. This got rid of some periods of time where the build was bottlenecked on a few actions, and reduced our build times by ~15%. N2 would need the same feature for android to adopt it.
In addition, we also pass a manually-defined file that maps from an output file to a priority int to use on clean builds when ninja doesn't know how long actions take yet.
https://android-review.googlesource.com/c/platform/external/ninja/+/2483775
https://android-review.googlesource.com/c/platform/external/ninja/+/2597009
As a developer working on large Cpp codebases,
I would like my build system to give me a detailed profile of the build
so that I can experiment with speeding it up
https://github.com/ginolatorilla/ninja-log-analyser
https://github.com/nico/ninjatracing
analyze-profile
https://bazel.build/docs/user-manual#profile
https://blog.rust-lang.org/2022/04/07/Rust-1.60.0.html#cargo---timings
Seems like chrome::tracing is the de-facto industry standard for (build) profiles, so probably best to use that.
Ninja has -n
dry run mode. However it's not clear what we should do with respect to things like rsp files.
@jamesr wrote:
The way that I've usually used "-n" is to ask the tool what steps it thinks it should run given the currently available information. In that use case I would consider the rspfile's contents as part of the tool's description of the step that it wants to run and would like access to that information. Another way to factor it would be to have the dry run simply go through the steps it would like to run and have a different query to ask for a fuller description of a particular step or set of steps, including rspfile file contents.
In LLVM, building lldb with a build directory generated with:
cmake ../llvm -G Ninja -DCMAKE_BUILD_TYPE=RelWithDebInfo -DLLVM_ENABLE_PROJECTS='lldb;clang' -DLLDB_ENABLE_PYTHON=true -DLLDB_ENABLE_SWIG=true -DLLDB_INCLUDE_TESTS=true -DLLVM_ENABLE_RUNTIMES=libcxx
Running
ninja lldb
eventually fails (after building ~4k files) with:
n2: error: build.ninja:95047: input tools/lldb/CMakeFiles/lldb-python missing
This failure doesn't happen with ninja
In Ninja, the below refers to the file foo.3
.
build foo.$bar: baz ...
bar = 3
In n2 this doesn't. n2 aggressively expands variables while parsing, which is to say that in the above it never even generates intermediate data like the parse of foo.$bar
, but rather expands $bar
right as it sees it and then interns the path, and even reuses that buffer to parse/canonicalize the next path.
This is fixable but I'm kinda hesitant. To implement the Ninja behavior we'd instead need to build up an array of all the paths and then expand them once we've parsed through the variables. (Edit: another possibly costlier idea, we could skip forward to the variables, parse them, then go back and parse the paths again.)
The fact that builds seem to currently work suggests that maybe existing projects don't depend on this?
Android has ninja files that total ~3GB in size. Android's fork of ninja has a multithreaded parser that is able to parse the ninja files in just under a second, but n2 takes over 14 seconds to parse the same files. All these numbers are for AOSP, the internal branch is roughly 50% larger.
I tried to get numbers for how fast the parsing was before 909ac60, but n2 fails to parse without that commit:
n2: error: parse error: expected '\n', got ' '
out/soong/build.sdk_phone64_x86_64.ninja:807688: ...ang-tidy.sh ${ccCmd} $
Android's multithreaded implementation can be seen here. (unfortunately it's not indexed on cs.android.com)
In #94 Evan expressed interest in only multithreading individual subninja files, but not chunks of a single file. This would be more work for android, as we'd have to change soong and kati to split up their ninja files, but may be possible, I need to look into it.
In addition, Android mmap's the file instead of reading it into memory, which probably helps read times. I'm not sure if this would become less effective if many smaller files had to be mmap'd / read individually.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.