just-buildsystem / justbuild Goto Github PK

just, a generic build tool

License: Apache License 2.0

Python 3.42% Shell 10.29% C++ 86.26% Just 0.02%

justbuild's Introduction

Justbuild

justbuild is a generic build system supporting multi-repository builds. A peculiarity of the tool is the separation between global names and physical location on the one hand, and logical paths used for actions and installation on the other hand (sometimes referred to as "staging"). The language-specific information to translate high-level concepts (libraries, binaries) into individual compile action is taken from user-defined rules described by functional expressions.

Getting Started

The most simple way to build the just binary from scratch is python3 ./bin/bootstrap.py. For more details see the installation guide.
Tutorial

Documentation

justbuild's People

Contributors

Stargazers

Watchers

justbuild's Issues

Boostrap/Package fails if /usr/bin/protoc is a symlink

I'm trying to package justbuild for ArchLinux, linking against native shared libraries, provided by ArchLinux packages as far as possible. Thus, I use the bootstrap.py script with the environment variables PACKAGE=YES LOCALBASE=/usr.

Another problem with this is that the prebuild /usr/bin/protoc executable, provided by the ArchLinux package protobuf, is a symbolic link to a more specifically named exectuable file (currently /usr/bin/protoc-3.21.12.0). So, even with my workaround from #12, this causes the following problem, as in this case the required file itself is a symlink:

Running ['/tmp/tmpqz1lqkg3/bootstrap-just', 'analyse', '-C', '/tmp/tmpqz1lqkg3/repo-conf.json', '-D', '{"COMPILER_FAMILY": "clang", "CC": "/usr/bin/clang", "CXX": "/usr/bin/clang++", "AR": "/usr/bin/ar", "ARCH": "x86_64"}', '--dump-graph', '/tmp/tmpqz1lqkg3/graph.json', '--dump-artifacts-to-build', '/tmp/tmpqz1lqkg3/to_build.json', '', ''] in '/tmp/tmpqz1lqkg3/src'
INFO: Requested target is [["@","just","",""],{"AR":"/usr/bin/ar","ARCH":"x86_64","CC":"/usr/bin/clang","COMPILER_FAMILY":"clang","CXX":"/usr/bin/clang++"}]
ERROR: unsupported type for dir entry /usr/include/./pngconf.h
ERROR: unsupported type for dir entry /usr/include/./pybind11
ERROR: unsupported type for dir entry /usr/include/./ncurses.h
[...]
ERROR: unsupported type for dir entry /usr/bin/pgmoil
ERROR: While processing targets:
       While evaluating export target ["@","just","",""]:
       While evaluating install target ["@","just","","installed just"]:
       While evaluating configure target ["@","just","","just"]:
       While evaluating export target ["@","just","","exported-just"]:
       While analysing ["@","rules-just","CC","binary"] target ["@","just","src/buildtool/main","just"]:
       While analysing ["@","rules-just","CC","library"] target ["@","just","src/buildtool/execution_api/local","garbage_collector"]:
       While analysing ["@","rules-just","CC","library"] target ["@","just","src/buildtool/common","bazel_types"]:
       While analysing ["@","rules-just","CC/proto","service library"] target ["#","a0d338d0250537ba57c641b3c933e548bdd24d9b","1181df6b8bbe306c91444bdcd12cda0310104333"]:
       While analysing ["@","rules-protobuf","data","staged"] target ["@","protobuf","","protoc"]:
       While analysing target ["@","protobuf","bin","protoc"] as implicit source target:
       Cannot determine source file "protoc" in directory "bin" of repository "protobuf"

To fix this, I patched the etc/import.prebuild/TARGETS.protobuf file to search for bin/protoc-3.21.12.0. instead of bin/protoc. However, this feels more like a workaround than a fix, since I would have to adjust the patch in my packaging recipe whenever the protobuf version is updated.

is not exist a rule like genrules, that I can wrapper some project I don't to change it

like this:
genrule( name = "foo", srcs = [], outs = ["foo.h"], cmd = "./$(location create_foo.pl) > \"$@\"", tools = ["create_foo.pl"], )

Bootstrap failure with system absl

Filing this issue because it's a regression from 1.1.4 and I can't debug it further atm.

Building justbuild 1.2.0 with NON_LOCAL_DEPS not containing com_google_absl now fails:

INFO: Requested target is [["@","just","",""],{"ADD_CFLAGS":["-Wno-error"],"ADD_CXXFLAGS":["-Wno-error"],"AR":"/usr/bin/ar","ARCH":"x86_64","CC":"cc","CFLAGS":["-fstack-clash-protection","-D_FORTIFY_SOURCE=2","-mtune=generic","-O2","-pipe","-ffile-prefix-map=/builddir/justbuild-1.2.0=."],"COMPILER_FAMILY":"gnu","CXX":"g++","CXXFLAGS":["-fstack-clash-protection","-D_FORTIFY_SOURCE=2","-mtune=generic","-O2","-pipe","-ffile-prefix-map=/builddir/justbuild-1.2.0=."],"ENV":{"PKG_CONFIG_PATH":"/usr/lib/pkgconfig:/usr/share/pkgconfig"},"LDFLAGS":["-Wl,-z,relro","-Wl,-z,now","-Wl,--as-needed"],"OS":"linux","SOURCE_DATE_EPOCH":1693061128.0,"VERSION_EXTRA_SUFFIX":"-void"}]
ERROR: While processing targets:
       While evaluating export target ["@","just","",""]:
       While analysing ["@","rules-just","CC","install-with-deps"] target ["@","just","","installed just"]:
       While evaluating configure target ["@","just","","just"]:
       While evaluating export target ["@","just","","exported-just"]:
       While analysing ["@","rules-just","CC","binary"] target ["@","just","src/buildtool/main","just"]:
       While analysing ["@","rules-just","CC","library"] target ["@","just","src/buildtool/storage","storage"]:
       While analysing ["@","rules-just","CC","library"] target ["@","just","src/buildtool/common","bazel_types"]:
       While analysing ["@","rules-just","CC/proto","service library"] target ["#","a0d338d0250537ba57c641b3c933e548bdd24d9b","9476599004d9f377bcc71f081af6c0ca4bac0d85"]:
       While analysing ["@","rules-just","CC/proto","library"] target ["#","a0d338d0250537ba57c641b3c933e548bdd24d9b","2c640c9332e66c97a48882cd3211387624e193e9"]:
       While analysing ["@","rules-just","CC/proto","defaults"] target ["@","rules-just","CC/proto","defaults"]:
       While evaluating export target ["@","protobuf","","libprotobuf"]:
       While analysing ["@","rules-protobuf","CC","library"] target ["@","protobuf","src/google/protobuf","libprotobuf"]:
       While searching targets description for ["@","com_google_absl","absl/strings","internal"]:
       JSON file absl/strings/TARGETS.absl does not exist.

and more errors for the other absl/* directories.

System absl is version 20230802.0. This works fine with justbuild 1.1.4.

(I wouldn't mind vendoring absl, but then it fails on x86_64-musl due to some legacy off64_t usage...)

Building against latest libgit2 produces corrupted JustBuild

While preparing for the 1.1.0 release of JustBuild, we were building and testing against libgit2 with the (back-then up-to-date) version 1.5.2. However, libgit2 recently released version 1.6.4, which seemed to introduce an issue:

Although JustBuild compiles just fine, accessing a Git ODB results in lookup errors. We are unsure what exactly is causing this issue, but our current investigations strongly indicate that it might be related to a bug in libgit2.

This issue only occurs when building against libgit2 >1.5.2. If your system ships with an affected version, make sure to build JustBuild with the bundled version of libgit2 that is defined in etc/repos.json.

Missing fmt::formatter for type_safe_arithmetic with {fmt} 10

While working on my Arch Linux AUR package of justbuild, I encountered a new compile error during the first bootstrapping phase when building justbuild 1.1.4. It's most probably caused by Arch Linux' {fmt} package updated to version 10.0.0:

In file included from /home/.../build/src/src/buildtool/main/describe.cpp:15:
In file included from /home/.../build/src/src/buildtool/main/describe.hpp:18:
In file included from /home/.../build/src/src/buildtool/build_engine/base_maps/entity_name.hpp:24:
In file included from /home/.../build/src/src/buildtool/build_engine/base_maps/entity_name_data.hpp:25:
In file included from /home/.../build/src/src/buildtool/build_engine/expression/expression_ptr.hpp:24:
In file included from /home/.../build/src/src/buildtool/build_engine/expression/function_map.hpp:21:
In file included from /home/.../build/src/src/buildtool/build_engine/expression/linked_map.hpp:26:
/usr/include/fmt/core.h:2561:10: error: call to deleted constructor of 'formatter<mapped_type, char_type>' (aka 'formatter<type_safe_arithmetic<PortTag>, char>')
  return formatter<mapped_type, char_type>().parse(ctx);
         ^
/usr/include/fmt/core.h:2620:23: note: in instantiation of function template specialization 'fmt::detail::parse_format_specs<type_safe_arithmetic<PortTag>, fmt::detail::compile_parse_context<char>>' requested here
        parse_funcs_{&parse_format_specs<Args, parse_context_type>...},
                      ^
/usr/include/fmt/core.h:2769:47: note: in instantiation of member function 'fmt::detail::format_string_checker<char, std::basic_string<char>, type_safe_arithmetic<PortTag>>::format_string_checker' requested here
      detail::parse_format_string<true>(str_, checker(s));
                                              ^
/home/.../build/src/src/buildtool/storage/config.hpp:149:32: note: in instantiation of function template specialization 'fmt::basic_format_string<char, std::basic_string<char> &, type_safe_arithmetic<PortTag> &>::basic_format_string<char[6], 0>' requested here
                               "{}:{}", address->host, address->port)}
                               ^
/usr/include/fmt/core.h:792:3: note: 'formatter' has been explicitly marked deleted here
  formatter() = delete;
  ^
2 errors generated.
/usr/include/fmt/core.h:1690:3: error: static assertion failed due to requirement 'formattable': Cannot format an argument. To make type T formattable provide a formatter<T> specialization: https://fmt.dev/latest/api.html#udt
  static_assert(
  ^
/usr/include/fmt/core.h:1711:10: note: in instantiation of function template specialization 'fmt::detail::make_value<fmt::basic_format_context<fmt::appender, char>, type_safe_arithmetic<PortTag> &>' requested here
  return make_value<Context>(val);
         ^
/usr/include/fmt/core.h:1825:23: note: in instantiation of function template specialization 'fmt::detail::make_arg<true, fmt::basic_format_context<fmt::appender, char>, fmt::detail::type::custom_type, type_safe_arithmetic<PortTag> &, 0>' requested here
        data_{detail::make_arg<
                      ^
/usr/include/fmt/core.h:1844:10: note: in instantiation of function template specialization 'fmt::format_arg_store<fmt::basic_format_context<fmt::appender, char>, std::basic_string<char>, type_safe_arithmetic<PortTag>>::format_arg_store<std::basic_string<char> &, type_safe_arithmetic<PortTag> &>' requested here
  return {FMT_FORWARD(args)...};
         ^
/usr/include/fmt/core.h:2817:28: note: in instantiation of function template specialization 'fmt::make_format_args<fmt::basic_format_context<fmt::appender, char>, std::basic_string<char> &, type_safe_arithmetic<PortTag> &>' requested here
  return vformat(fmt, fmt::make_format_args(args...));
                           ^
/home/.../build/src/src/buildtool/storage/config.hpp:148:48: note: in instantiation of function template specialization 'fmt::format<std::basic_string<char> &, type_safe_arithmetic<PortTag> &>' requested here
                 address ? nlohmann::json{fmt::format(
                                               ^

I have to admit that my C++ template magic skills are currently not sufficiently advanced to provide a fix with the correct template specialization for the generic type_safe_arithmetic_tag<> type.

Bootstrapping against prebuild local dependencies fails when /usr/bin or /usr/include contain symlinks

After working around the other issues, this resulted in the following problem:

Running ['/tmp/tmpxaow1ajw/bootstrap-just', 'analyse', '-C', '/tmp/tmpxaow1ajw/repo-conf.json', '-D', '{"COMPILER_FAMILY": "clang", "CC": "/usr/bin/clang", "CXX": "/usr/bin/clang++", "AR": "/usr/bin/ar", "ARCH": "x86_64"}', '--dump-graph', '/tmp/tmpxaow1ajw/graph.json', '--dump-artifacts-to-build', '/tmp/tmpxaow1ajw/to_build.json', '', ''] in '/tmp/tmpxaow1ajw/src'
INFO: Requested target is [["@","just","",""],{"AR":"/usr/bin/ar","ARCH":"x86_64","CC":"/usr/bin/clang","COMPILER_FAMILY":"clang","CXX":"/usr/bin/clang++"}]
ERROR: unsupported type for dir entry /usr/include/./pngconf.h
ERROR: While processing targets:
       While evaluating export target ["@","just","",""]:
       While evaluating install target ["@","just","","installed just"]:
       While evaluating configure target ["@","just","","just"]:
       While evaluating export target ["@","just","","exported-just"]:
       While analysing ["@","rules-just","CC","binary"] target ["@","just","src/buildtool/main","just"]:
       While analysing ["@","rules-just","CC","library"] target ["@","just","src/buildtool/common","config"]:
       While analysing ["@","rules-just","CC","library"] target ["@","just","src/buildtool/file_system","git_cas"]:
       While evaluating configure target ["@","just","","libgit2"]:
       While analysing ["@","rules-git2","CC","library"] target ["@","com_github_libgit2_libgit2","","git2"]:
       While analysing target ["@","com_github_libgit2_libgit2","","git2.h"] as implicit source target:
       Cannot determine source file "git2.h" in directory "" of repository "com_github_libgit2_libgit2"
ERROR: unsupported type for dir entry /usr/bin/ardour7-new_session
ERROR: While processing targets:
       While evaluating export target ["@","just","",""]:
       While evaluating install target ["@","just","","installed just"]:
       While evaluating configure target ["@","just","","just"]:
       While evaluating export target ["@","just","","exported-just"]:
       While analysing ["@","rules-just","CC","binary"] target ["@","just","src/buildtool/main","just"]:
       While analysing ["@","rules-just","CC","library"] target ["@","just","src/buildtool/execution_api/local","garbage_collector"]:
       While analysing ["@","rules-just","CC","library"] target ["@","just","src/buildtool/common","bazel_types"]:
       While analysing ["@","rules-just","CC/proto","service library"] target ["#","a0d338d0250537ba57c641b3c933e548bdd24d9b","1181df6b8bbe306c91444bdcd12cda0310104333"]:
       While evaluating install target ["@","com_github_grpc_grpc","src/compiler","grpc_cpp_plugin"]:
       While analysing target ["@","com_github_grpc_grpc","","bin/grpc_cpp_plugin"] as implicit source target:
       Cannot determine source file "grpc_cpp_plugin" in directory "bin" of repository "com_github_grpc_grpc"
ERROR: unsupported type for dir entry /usr/bin/ardour7-new_session
ERROR: While processing targets:
       While evaluating export target ["@","just","",""]:
       While evaluating install target ["@","just","","installed just"]:
       While evaluating configure target ["@","just","","just"]:
       While evaluating export target ["@","just","","exported-just"]:
       While analysing ["@","rules-just","CC","binary"] target ["@","just","src/buildtool/main","just"]:
       While analysing ["@","rules-just","CC","library"] target ["@","just","src/buildtool/execution_api/local","garbage_collector"]:
       While analysing ["@","rules-just","CC","library"] target ["@","just","src/buildtool/common","bazel_types"]:
       While analysing ["@","rules-just","CC/proto","service library"] target ["#","a0d338d0250537ba57c641b3c933e548bdd24d9b","1181df6b8bbe306c91444bdcd12cda0310104333"]:
       While analysing ["@","rules-protobuf","data","staged"] target ["@","protobuf","","protoc"]:
       While analysing target ["@","protobuf","bin","protoc-3.21.12.0"] as implicit source target:
       Cannot determine source file "protoc-3.21.12.0" in directory "bin" of repository "protobuf"

The mentioned dir entries with unsupported types (/usr/include/./pngconf.h, /usr/bin/ardour7-new_session) are symbolic links to other files within the same directory.

I was able to work around this issue by patching justbuild to ignore symlinks instead of aborting directory scanning: mhthies@4b283e9

Bootstrap fails on i686

Perhaps this is out of scope for you, but also should be easy to fix:

A bootstrap on 32-bit Intel fails with:

Unpacking 'ssl' from '/builddir/justbuild-1.1.1/just-work/fetch/6195bf8242156c9a2fa75702eee058f91b86a88b.tar.gz'
Running ['sh', '-c', "SYS=`uname -s | tr 'A-Z' 'a-z'` && ARCH=`uname -m` && cc '-g' -I . -I src/include -c *.c src/crypto/*.c  src/crypto/*/*.c $SYS-$ARCH/crypto/fipsmodule/*.S && /usr/bin/ar cqs libcrypto.a *.o"] in '/builddir/justbuild-1.1.1/just-work/deps/ssl/boringssl-6195bf8242156c9a2fa75702eee058f91b86a88b'
cc1: fatal error: linux-i686/crypto/fipsmodule/*.S: No such file or directory
compilation terminated.

The correct directory name is linux-x86/crypto/fipsmodule.

Compiled `just-mr` does not support `ssh://` git protocol

There are two common ways to specify a URL of a git repository accessible via ssh

[email protected]:foo/bar
ssh://[email protected]/foo/bar

with the latter having the advantage that the python just-mr script recognizes this as a remote repository and does not assume a git repository on the local file system. This form, however, is not recognized by the compiled just-mr (saying error code 12: unsupported URL protocol) and I believe it should be.

On a side note, the error message in this case does not mention the URL it failed to parse; showing might improve the error message.

Bootstrapping fails with clang 15.0.7 due to warning "array-parameter" in BoringSSL

I'm trying to bootstrap-build justbuild on ArchLinux using CLang 15.0.7. Unfortunately, the compilation fails due to a warning in BoringSSL, which is built with -Werror (even when configuring "COMPILER_FAMILY": "unknown"):

Running ['/usr/bin/clang', '-O2', '-DNDEBUG', '-std=gnu17', '-Wa,--noexecstack', '-D_XOPEN_SOURCE=700', '-Wall', '-Werror', '-Wformat=2', '-Wsign-compare', '-Wmissing-field-initializers', '-Wwrite-strings', '-Wshadow', '-fno-common', '-I', 'work', '-isystem', 'include', '-c', 'work/src/crypto/fipsmodule/bcm.c', '-o', 'work/src/crypto/fipsmodule/bcm.o'] with env {'PATH': '/bin:/sbin:/usr/bin:/usr/sbin'} for action 'dee9161ccc9d5793778e365f064834ab836dcc47'
In file included from work/src/crypto/fipsmodule/bcm.c:38:
work/src/crypto/fipsmodule/bn/asm/x86_64-gcc.c:427:51: error: argument 'a' of type 'const uint64_t[8]' (aka 'const unsigned long[8]') with mismatched bound [-Werror,-Warray-parameter]
void bn_sqr_comba8(BN_ULONG r[16], const BN_ULONG a[8]) {
                                                  ^
work/src/crypto/fipsmodule/bn/asm/../internal.h:292:51: note: previously declared as 'const uint64_t[4]' (aka 'const unsigned long[4]') here
void bn_sqr_comba8(BN_ULONG r[16], const BN_ULONG a[4]);
                                                  ^

I'm using the following command line:

env JUST_BUILD_CONF='{"COMPILER_FAMILY": "unknown", "CC": "/usr/bin/clang", "CXX": "/usr/bin/clang++", "AR": "/usr/bin/ar"}' python3 ./bin/bootstrap.py .

Output of clang --version:

clang version 15.0.7
Target: x86_64-pc-linux-gnu
Thread model: posix
InstalledDir: /usr/bin

As a workaround, I have removed the -Werror from the posix_copts array in etc/defaults/CC/TARGETS.boringssl.

Boostrap/Package: How to adapt for distribution-specific shared library names?

I'm trying to package justbuild for ArchLinux, linking against native shared libraries, provided by ArchLinux packages, as far as possible. After working around the other issues (#9, #10, #11, #12), I encountered the problem that the prebuild shared libraries, provided by the official ArchLinux packages, have completely different names than expected by the justbuild build. So, the linker arguments need to be adjusted accordingly. In addition, the "well-known" proto files, provided with the protobuf compiler, are located in /usr/include/google/protobuf instead of /usr/proto/google/protobuf on ArchLinux.

I patched the TARGET files in etc/import.prebuild to adjust for these differences. However, this resulted in a rather large patch and left me with the question, whether this is the expected way of solving this. Am I supposed to include these changes as patch file with my packaging recipe or is there another way?

native just-mr fails to link with protobuf 23.3

While updating the Arch Linux package recipe, I ran into an error while building the native just-mr executable against abseil-cpp 20230125.3 and protobuf 23.3:

ERROR: expected file at src/other_tools/just_mr/just-mr
ERROR (LocalExecution): could not collect output file src/other_tools/just_mr/just-mr
INFO (action:7d7f8c02c7d20e5b389311304e98b02f9a747591):
     Stderr of command: ["/usr/bin/clang++","-o","src/other_tools/just_mr/just-mr","-target","x86_64-linux-gnu","-O2","-DNDEBUG","-std=c++20","-Wall","-Wextra","-Wpedantic","-Wsign-conversion","-Werror","-pedantic-errors","-Wno-error","@protobuf.cflags","@grpc++.cflags","@fmt.cflags","@nlohmann_json.cflags","@gsl.cflags","@fmt.cflags","@gsl.cflags","@gsl.cflags","@CLI11.cflags","@protobuf.cflags","@grpc++.cflags","@fmt.cflags","@nlohmann_json.cflags","@gsl.cflags","@protobuf.cflags","@grpc++.cflags","@fmt.cflags","@gsl.cflags","@nlohmann_json.cflags","-pthread","@gsl.cflags","@fmt.cflags","-pthread","@protobuf.cflags","@grpc++.cflags","@fmt.cflags","@nlohmann_json.cflags","@gsl.cflags","@protobuf.cflags","@grpc++.cflags","@fmt.cflags","@nlohmann_json.cflags","@gsl.cflags","@nlohmann_json.cflags","@fmt.cflags","@gsl.cflags","src/other_tools/just_mr/main.o","-Wl,-z,stack-size=8388608","src/buildtool/main/libversion.a","@CLI11.ldflags","src/other_tools/ops_maps/librepo_fetch_map.a","src/other_tools/ops_maps/libgit_update_map.a","src/other_tools/repo_map/librepos_to_setup_map.a","src/other_tools/root_maps/libcommit_git_map.a","src/other_tools/root_maps/libcontent_git_map.a","src/other_tools/utils/libarchive_ops.a","@libarchive.ldflags","src/other_tools/root_maps/libdistdir_git_map.a","src/other_tools/root_maps/libfpath_git_map.a","src/other_tools/root_maps/libtree_id_git_map.a","src/other_tools/ops_maps/libcontent_cas_map.a","src/other_tools/utils/libcurl_easy_handle.a","src/other_tools/ops_maps/libimport_to_git_map.a","src/other_tools/ops_maps/libcritical_git_op_map.a","src/other_tools/git_operations/libgit_operations.a","src/buildtool/multithreading/libtask_system.a","src/other_tools/git_operations/libgit_repo_remote.a","src/other_tools/git_operations/libgit_config_settings.a","src/other_tools/utils/libcurl_url_handle.a","src/other_tools/utils/libcurl_context.a","@libcurl.ldflags","src/other_tools/just_mr/libutils.a","src/utils/cpp/libtmp_dir.a","src/buildtool/execution_api/local/liblocal.a","src/buildtool/execution_api/bazel_msg/libblob_tree.a","src/buildtool/common/libconfig.a","src/buildtool/file_system/libgit_tree.a","src/buildtool/storage/libstorage.a","src/buildtool/build_engine/analysed_target/libtarget.a","src/buildtool/build_engine/analysed_target/libgraph_information.a","src/utils/cpp/libfile_locking.a","src/buildtool/execution_api/bazel_msg/libbazel_msg_factory.a","src/buildtool/execution_api/bazel_msg/libdirectory_tree.a","src/buildtool/file_system/libgit_repo.a","src/buildtool/file_system/libgit_cas.a","src/buildtool/file_system/libgit_context.a","src/buildtool/file_system/libgit_utils.a","libgit2.a","libutil_os_unix.a","libutil_hash_openssl.a","libgit2_pcre.a","@zlib.ldflags","libgit2_http_parser.a","@libssl.ldflags","src/buildtool/execution_engine/dag/libdag.a","src/buildtool/system/libsystem.a","src/buildtool/build_engine/expression/libexpression.a","libremote_execution_proto.a","libsemver_proto.a","libgoogle_longrunning_operations_proto.a","libgoogle_api_annotations_proto.a","libgoogle_api_http_proto.a","libgoogle_api_client_proto.a","libgoogle_rpc_status_proto.a","@protobuf.ldflags","@grpc++.ldflags","src/buildtool/crypto/libhasher.a","@libcrypto.ldflags","src/other_tools/just_mr/progress_reporting/libprogress_reporter.a","src/buildtool/progress_reporting/libbase_progress_reporter.a","@nlohmann_json.ldflags","-pthread","@fmt.ldflags","@gsl.ldflags"]
     /usr/bin/ld: libremote_execution_proto.a(remote_execution.pb.o): undefined reference to symbol '_ZN4absl12lts_2023012512log_internal10LogMessage11OstreamViewC1ERNS2_14LogMessageDataE'
     /usr/bin/ld: /usr/bin/../lib64/gcc/x86_64-pc-linux-gnu/13.1.1/../../../../lib64/libabsl_log_internal_message.so.2301.0.0: error adding symbols: DSO missing from command line
     clang-15: error: linker command failed with exit code 1 (use -v to see invocation)

It looks to me like libabsl_log_internal_message is somehow missing in the "@protobuf.ldflags" which seems to be required by the generated libremote_execution_proto.

... using

clang 15.0.7
GNU ld (GNU Binutils) 2.40.0
abseil-cpp 20230125.3
protobuf 23.3

Bootstrapping against preinstalled dependencies fails with latest libgit2

Bootstrapping will fail due to libgit2's treebuilder producing corrupted tree objects, causing errors when composing an action directory. We are unsure what exactly is causing this issue, but our current investigations strongly indicate that it might be related to a bug in libgit2.

This issue only occurs when building against libgit2 >1.5.2. If you want to bootstrap against preinstalled dependencies on a system that ships with an affected libgit2 version, make sure to bootstrap with NON_LOCAL_DEPS='["com_github_libgit2_libgit2"]'. In this way, the bundled libgit2 version 1.5.2 will be used.

Building justbuild against fmt 9.1.x fails due to consteval issue

I'm trying to package justbuild for ArchLinux, linking against native shared libraries, provided by ArchLinux packages, as far as possible. The fmt library is currently provided in version 9.1.0 in the ArchLinux package repositories.

Building justbuild against this library version, using the bootstrap.py with PACKAGE=YES and Clang 15.0.7 fails with the following error message:

Running ['/usr/bin/clang++', '-std=c++20', '-DBOOTSTRAP_BUILD_TOOL', '-I', '/tmp/tmps83ihjbv/src', '-I', '/tmp/tmps83ihjbv/dep_includes', '-I', '/usr/include', '-c', '/tmp/tmps83ihjbv/src/src/utils/cpp/file_locking.cpp', '-o', '/tmp/tmps83ihjbv/src/src/utils/cpp/file_locking.o'] in '/tmp/tmps83ihjbv/src'
In file included from /tmp/tmps83ihjbv/src/src/utils/cpp/file_locking.cpp:19:
In file included from /tmp/tmps83ihjbv/src/src/buildtool/file_system/file_system_manager.hpp:32:
/tmp/tmps83ihjbv/src/src/buildtool/logging/logger.hpp:130:37: error: call to consteval function 'fmt::basic_format_string<char, std::basic_string<char>, std::basic_string<char>, const char *>::basic_format_string<std::basic_string<char>, 0>' is not a constant expression
            auto fmsg = fmt::format(msg, std::forward<T_Args>(args)...);
                                    ^
/tmp/tmps83ihjbv/src/src/buildtool/logging/logger.hpp:93:13: note: in instantiation of function template specialization 'Logger::FormatAndForward<std::basic_string<char>, std::basic_string<char>, const char *>' requested here
            FormatAndForward(nullptr,
            ^
/tmp/tmps83ihjbv/src/src/buildtool/file_system/file_system_manager.hpp:87:21: note: in instantiation of function template specialization 'Logger::Log<std::basic_string<char>, std::basic_string<char>, const char *>' requested here
            Logger::Log(LogLevel::Error,
                    ^
/tmp/tmps83ihjbv/src/src/buildtool/logging/logger.hpp:130:37: note: function parameter 'msg' with unknown value cannot be used in a constant expression
            auto fmsg = fmt::format(msg, std::forward<T_Args>(args)...);
                                    ^
/tmp/tmps83ihjbv/src/src/buildtool/logging/logger.hpp:120:53: note: declared here
                                 std::string const& msg,
                                                    ^

I was able to fix this error, based on this StackOverflow answer: https://stackoverflow.com/a/68675384/10315508 :

I replaced fmt::format() with fmt::vformat() and std::forward<T_Args>() with fmt::make_format_args() in Logger::FormatAndForward(), as shown in this commit. I have not checked whether this is compatible with fmt 7.x and other compilers. Thus, I'll leave it here as a proposal but not create a PR.

Helps about how to boostrap build of justbuild

I used the following command to try to build it.

env PACKAGE=YES LOCALBASE=/usr \
    NON_LOCAL_DEPS='["google_apis","bazel_remote_apis"]' \
    python3 ./bin/bootstrap.py . build

see the error in build log: https://github.com/topazus/justbuild/actions/runs/7335489509/job/19973471274

Boostrap: Allow selecting which of the dependencies are searched in local root

However, some of the dependencies are not available from official ArchLinux packages yet:

gsl-lite is only available from an Arch User Repository (AUR) package and only in a more (too) recent version
google_apis and bazel_remote_apis are not packaged at all by now to my knowledge

Of course, I could create AUR packages for the two Google API repositories and work around the gsl-lite version mismatch. But, as all of these are quite small "compile-time" dependencies, I would like to let the bootstrap script fetch them, while still using Arch's pre-build shared libraries for all other dependencies. Thus, I'd like to be able to select which of the dependencies are downloaded as normal instead of being are searched in the local root directory, e.g. using an environment variable.

I already implemented this feature as a minimal patch for the bootstrap.py script, which simply skips some of the repositories in config_to_local() based on an environment variable (see this commit). This was sufficient in my case, but actually only works for header-only and protobuf dependencies, because the "import targets" repository is switched to "etc/import.prebuild" globally. Thus, it's not possible to use the build target definitions for selected dependencies this way.

Building justbuild fails with GCC 13.2 due to `partial specialization of ‘struct fmt::v10::formatter<std::__cxx11::basic_string<...>>`

I tried to change my Arch Linux package recipe to use GCC instead of Clang and ran into another {fmt}-related issue:

In file included from /usr/include/fmt/ostream.h:20,
                 from /home/michael/Documents/Freizeit/Programme/pkgbuild/justbuild/src/build/src/src/utils/cpp/json.hpp:25,
                 from /home/michael/Documents/Freizeit/Programme/pkgbuild/justbuild/src/build/src/src/buildtool/common/artifact_description.hpp:27,
                 from /home/michael/Documents/Freizeit/Programme/pkgbuild/justbuild/src/build/src/src/buildtool/build_engine/expression/target_result.hpp:23,
                 from /home/michael/Documents/Freizeit/Programme/pkgbuild/justbuild/src/build/src/src/buildtool/build_engine/expression/target_result.cpp:15:
/usr/include/fmt/format.h:4076:1: error: partial specialization of ‘struct fmt::v10::formatter<std::__cxx11::basic_string<_CharT>, Char>’ after instantiation of ‘struct fmt::v10::formatter<std::__cxx11::basic_string<char>, char, void>’ [-fpermissive]
 4076 | FMT_FORMAT_AS(std::basic_string<Char>, basic_string_view<Char>);
      | ^~~~~~~~~~~~~

(and many similar occurances; all of them caused by artifact_description.hpp being included somehow.)

This might be introduced by my patch (#42) and sounds similar to this issue: fmtlib/fmt#2769

I was able to fully fix the problem by replacing the #include "fmt/core.h" with #include "fmt/format.h" in logger.hpp.

Tested with fmt 10.1.0.

Building justbuild against fmt 9.1.x fails due to missing fallback formatter

I'm trying to package justbuild for ArchLinux, linking against native shared libraries, provided by ArchLinux packages, as far as possible. In doing so, I came across yet another problem when building against fmt version 9.1.0 (in addition to #10):

Running ['/usr/bin/clang++', '-std=c++20', '-DBOOTSTRAP_BUILD_TOOL', '-I', '/tmp/tmp1i8ffxxt/src', '-I', '/tmp/tmp1i8ffxxt/dep_includes', '-I', '/usr/include', '-c', '/tmp/tmp1i8ffxxt/src/src/buildtool/main/analyse.cpp', '-o', '/tmp/tmp1i8ffxxt/src/src/buildtool/main/analyse.o'] in '/tmp/tmp1i8ffxxt/src'
In file included from /tmp/tmp1i8ffxxt/src/src/buildtool/main/analyse.cpp:15:
In file included from /tmp/tmp1i8ffxxt/src/src/buildtool/main/analyse.hpp:18:
In file included from /tmp/tmp1i8ffxxt/src/src/buildtool/build_engine/analysed_target/analysed_target.hpp:24:
In file included from /tmp/tmp1i8ffxxt/src/src/buildtool/build_engine/analysed_target/target_graph_information.hpp:21:
In file included from /tmp/tmp1i8ffxxt/src/src/buildtool/build_engine/target_map/configured_target.hpp:20:
/usr/include/fmt/core.h:2743:12: error: call to deleted constructor of 'conditional_t<has_formatter<mapped_type, context>::value, formatter<mapped_type, char_type>, fallback_formatter<stripped_type, char_type>>' (aka 'fmt::detail::fallback_formatter<LogLevel>')
  auto f = conditional_t<has_formatter<mapped_type, context>::value,
           ^
/usr/include/fmt/core.h:2954:23: note: in instantiation of function template specialization 'fmt::detail::parse_format_specs<LogLevel, fmt::detail::compile_parse_context<char>>' requested here
        parse_funcs_{&parse_format_specs<Args, parse_context_type>...},
                      ^
/usr/include/fmt/core.h:3159:47: note: in instantiation of member function 'fmt::detail::format_string_checker<char, fmt::detail::error_handler, LogLevel, LogLevel, LogLevel>::format_string_checker' requested here
      detail::parse_format_string<true>(str_, checker(s, {}));
                                              ^
/tmp/tmp1i8ffxxt/src/src/buildtool/common/cli.hpp:177:24: note: in instantiation of function template specialization 'fmt::basic_format_string<char, const LogLevel &, const LogLevel &, const LogLevel &>::basic_format_string<char[70], 0>' requested here
           fmt::format("Log limit (higher is more verbose) in interval [{},{}] "
                       ^
/usr/include/fmt/core.h:1124:3: note: 'fallback_formatter' has been explicitly marked deleted here
  fallback_formatter() = delete;
  ^
/usr/include/fmt/core.h:1756:3: error: static assertion failed due to requirement 'formattable': Cannot format an argument. To make type T formattable provide a formatter<T> specialization: https://fmt.dev/latest/api.html#udt
  static_assert(
  ^
/usr/include/fmt/core.h:1777:10: note: in instantiation of function template specialization 'fmt::detail::make_value<fmt::basic_format_context<fmt::appender, char>, const LogLevel &>' requested here
  return make_value<Context>(val);
         ^
/usr/include/fmt/core.h:1899:23: note: in instantiation of function template specialization 'fmt::detail::make_arg<true, fmt::basic_format_context<fmt::appender, char>, fmt::detail::type::custom_type, const LogLevel &, 0>' requested here
        data_{detail::make_arg<
                      ^
/usr/include/fmt/core.h:1918:10: note: in instantiation of function template specialization 'fmt::format_arg_store<fmt::basic_format_context<fmt::appender, char>, LogLevel, LogLevel, LogLevel>::format_arg_store<const LogLevel &, const LogLevel &, const LogLevel &>' requested here
  return {FMT_FORWARD(args)...};
         ^
/usr/include/fmt/core.h:3206:28: note: in instantiation of function template specialization 'fmt::make_format_args<fmt::basic_format_context<fmt::appender, char>, const LogLevel &, const LogLevel &, const LogLevel &>' requested here
  return vformat(fmt, fmt::make_format_args(args...));
                           ^
/tmp/tmp1i8ffxxt/src/src/buildtool/common/cli.hpp:177:17: note: in instantiation of function template specialization 'fmt::format<const LogLevel &, const LogLevel &, const LogLevel &>' requested here
           fmt::format("Log limit (higher is more verbose) in interval [{},{}] "
                ^
2 errors generated.

As far as I understand, this points to a missing fmt::formatter implementation for LogLevel. So I added one, based on LogLevelToString() and the fmt::formatter<std::string>. This way, I was able to fix the issue. Here is the full code change: mhthies@548b352

However, I'm not familiar with fmt, so I'm not sure if this is the correct fix. I also didn't test this change against fmt 7.x and other compilers.

Compile errors with fmt 10.1.0

Hi,

both HEAD and tag v1.2.0 do not build with fmt version 10.1.0. I stumbled over this due to Arch bumping its package, but this also occurs if I bump the version of fmt in etc/repos.json. It builds with 10.0.0, so this is not a major issue right now, but you will probably need to address this before updating fmt the next time.

The relevant error message is added below.

In file included from /tmp/tmpgtv8ewdx/src/src/buildtool/main/main.cpp:24:
In file included from /tmp/tmpgtv8ewdx/src/src/buildtool/build_engine/base_maps/entity_name.hpp:24:
In file included from /tmp/tmpgtv8ewdx/src/src/buildtool/build_engine/base_maps/entity_name_data.hpp:25:
In file included from /tmp/tmpgtv8ewdx/src/src/buildtool/build_engine/expression/expression_ptr.hpp:24:
In file included from /tmp/tmpgtv8ewdx/src/src/buildtool/build_engine/expression/function_map.hpp:21:
In file included from /tmp/tmpgtv8ewdx/src/src/buildtool/build_engine/expression/linked_map.hpp:26:
/tmp/tmp3ltxf4ic/dep_includes/fmt/core.h:1577:63: error: implicit instantiation of undefined template
      'fmt::detail::type_is_unformattable_for<const nlohmann::basic_json<>, char>'
    type_is_unformattable_for<T, typename Context::char_type> _;
                                                              ^
/tmp/tmp3ltxf4ic/dep_includes/fmt/core.h:1810:23: note: in instantiation of function template specialization
      'fmt::detail::make_arg<true, fmt::basic_format_context<fmt::appender, char>, const nlohmann::basic_json<>, 0>' requested here
        data_{detail::make_arg<is_packed, Context>(args)...} {
                      ^
/tmp/tmp3ltxf4ic/dep_includes/fmt/core.h:1828:10: note: in instantiation of function template specialization
      'fmt::format_arg_store<fmt::basic_format_context<fmt::appender, char>, nlohmann::basic_json<>,
      nlohmann::basic_json<>>::format_arg_store<const nlohmann::basic_json<>, const nlohmann::basic_json<>>' requested here
  return {args...};
         ^
/tmp/tmp3ltxf4ic/src/src/buildtool/logging/logger.hpp:130:48: note: in instantiation of function template specialization
      'fmt::make_format_args<fmt::basic_format_context<fmt::appender, char>, const nlohmann::basic_json<>, const nlohmann::basic_json<>>'
      requested here
            auto fmsg = fmt::vformat(msg, fmt::make_format_args(args...));
                                               ^
/tmp/tmp3ltxf4ic/src/src/buildtool/logging/logger.hpp:93:13: note: in instantiation of function template specialization
      'Logger::FormatAndForward<const nlohmann::basic_json<> &, const nlohmann::basic_json<> &>' requested here
            FormatAndForward(nullptr,
            ^
/tmp/tmp3ltxf4ic/src/src/buildtool/main/main.cpp:475:17: note: in instantiation of function template specialization
      'Logger::Log<const nlohmann::basic_json<> &, const nlohmann::basic_json<> &>' requested here
        Logger::Log(LogLevel::Error,
                ^
/tmp/tmp3ltxf4ic/dep_includes/fmt/core.h:1555:45: note: template is declared here
template <typename T, typename Char> struct type_is_unformattable_for;
                                            ^
/tmp/tmp3ltxf4ic/dep_includes/fmt/core.h:1580:3: error: static assertion failed due to requirement 'formattable': Cannot
      format an argument. To make type T formattable provide a formatter<T> specialization: https://fmt.dev/latest/api.html#udt
  static_assert(
  ^
/tmp/tmp3ltxf4ic/dep_includes/fmt/core.h:1577:63: error: implicit instantiation of undefined template
      'fmt::detail::type_is_unformattable_for<nlohmann::basic_json<>, char>'
    type_is_unformattable_for<T, typename Context::char_type> _;
                                                              ^
/tmp/tmp3ltxf4ic/dep_includes/fmt/core.h:1810:23: note: in instantiation of function template specialization
      'fmt::detail::make_arg<true, fmt::basic_format_context<fmt::appender, char>, nlohmann::basic_json<>, 0>' requested here
        data_{detail::make_arg<is_packed, Context>(args)...} {
                      ^
/tmp/tmp3ltxf4ic/dep_includes/fmt/core.h:1828:10: note: in instantiation of function template specialization
      'fmt::format_arg_store<fmt::basic_format_context<fmt::appender, char>, nlohmann::basic_json<>, std::basic_string<char>,
      std::basic_string<char>>::format_arg_store<nlohmann::basic_json<>, const std::basic_string<char>, const std::basic_string<char>>'
      requested here
  return {args...};
         ^
/tmp/tmp3ltxf4ic/src/src/buildtool/logging/logger.hpp:130:48: note: in instantiation of function template specialization
      'fmt::make_format_args<fmt::basic_format_context<fmt::appender, char>, nlohmann::basic_json<>, const std::basic_string<char>, const
      std::basic_string<char>>' requested here
            auto fmsg = fmt::vformat(msg, fmt::make_format_args(args...));
                                               ^
/tmp/tmp3ltxf4ic/src/src/buildtool/logging/logger.hpp:93:13: note: in instantiation of function template specialization
      'Logger::FormatAndForward<nlohmann::basic_json<> &, const std::basic_string<char> &, const std::basic_string<char> &>' requested here
            FormatAndForward(nullptr,
            ^
/tmp/tmp3ltxf4ic/src/src/buildtool/main/main.cpp:654:29: note: in instantiation of function template specialization
      'Logger::Log<nlohmann::basic_json<> &, const std::basic_string<char> &, const std::basic_string<char> &>' requested here
                    Logger::Log(LogLevel::Error,
                            ^
/tmp/tmp3ltxf4ic/dep_includes/fmt/core.h:1555:45: note: template is declared here
template <typename T, typename Char> struct type_is_unformattable_for;
                                            ^
/tmp/tmp3ltxf4ic/dep_includes/fmt/core.h:1580:3: error: static assertion failed due to requirement 'formattable': Cannot
      format an argument. To make type T formattable provide a formatter<T> specialization: https://fmt.dev/latest/api.html#udt
  static_assert(
  ^
4 errors generated.

Distribute alternatives to .org files

It would be convenient to distribute the documentation/help files in other formats than .org. For one, github does not render them perfectly (in particular: ~~"foo"~~ ) and it is a typical use case to read them on git. Further, in an installation, it would be good to have them as man pages. The AUR package currently does this by requiring emacs as a make dependency, which is a bit excessive for those of us who do not use emacs.

I am unsure what the correct way to handle this, is. Having them checked-in in three formats is probably not that great. Maybe there is a format that can be converted to all formats by a program more lightweight than emacs.

Broken bootstrapping due to missing include in 1.1.3

Bootrapping justbuild 1.1.3 for packaging on Arch Linux fails with the following error during the first stage:

Running ['/usr/bin/clang++', '-target', 'x86_64-linux-gnu', '-O2', '-DNDEBUG', '-std=c++20', '-Wall', '-Wextra', '-Wpedantic', '-Wsign-conversion', '-Werror', '-pedantic-errors', '-Wno-error', '@protobuf.cflags', '@grpc++.cflags', '@nlohmann_json.cflags', '@fmt.cflags', '@gsl.cflags', '-I', 'work', '-isystem', 'include', '-c', 'work/src/buildtool/build_engine/expression/expression.cpp', '-o', 'work/src/buildtool/build_engine/expression/expression.o'] with env {'PATH': '/bin:/sbin:/usr/bin:/usr/sbin', 'PKG_CONFIG_PATH': '/home/...:/usr/lib/pkgconfig:/usr/share/pkgconfig'} for action '27ef66d6dbe78cb9905a728a05af3baee700acaf'
In file included from work/src/buildtool/execution_api/execution_service/operation_cache.cpp:15:
work/src/buildtool/execution_api/execution_service/operation_cache.hpp:66:14: error: no type named 'unique_lock' in namespace 'std'
        std::unique_lock lock{mutex_};
        ~~~~~^
work/src/buildtool/execution_api/execution_service/operation_cache.cpp:38:14: error: no type named 'unique_lock' in namespace 'std'
        std::unique_lock ulock{mutex_};
        ~~~~~^

Built with clang 15.0.7.

The error seems to be caused by a missing include of <mutex> in src/buildtool/execution_api/execution_service/operation_cache.hpp. It can be fixed with the following patch:

diff --git a/src/buildtool/execution_api/execution_service/operation_cache.hpp b/src/buildtool/execution_api/execution_service/operation_cache.hpp
index 97b137f..25ca49a 100644
--- a/src/buildtool/execution_api/execution_service/operation_cache.hpp
+++ b/src/buildtool/execution_api/execution_service/operation_cache.hpp
@@ -16,6 +16,7 @@
 #define OPERATION_CACHE_HPP
 
 #include <atomic>
+#include <mutex>
 #include <optional>
 #include <shared_mutex>
 #include <string>