Compilation is S L O W about scylladb HOT 38 OPEN

Actually, we have reached the same conclusion, and Scylla 1.5 (due in about a month) will by written in Python 3. Stay tuned!

from scylladb.

Just an idea: how about using pre-compiled headers or C++20's modules in order to not compile some parts of the code over and over again?

@tchaikov is working exactly on that (C++20 modules).

from scylladb.

Just an idea: how about using pre-compiled headers or C++20's modules in order to not compile some parts of the code over and over again?

@ptrsmrn The comment right above yours says:

Issue #12962 proposes that build speed can be improved by using the new C++ "modules" feature instead of old-style header files.

We were eager to use C++20 modules since their inception. We were doing first experiments 2 years ago. But the bottleneck is toolchain support for modules.

For example, it took until clang 15 (released last september) until we could compile a modules program that isn't miscompiled and segfaults on entry. Only clang 17 (not released yet) will have support for module dependency discovery (p1689r5). Cmake support was added in the last months. And so on.

AFAIK the state of modules in Scylla is that @tchaikov converted Seastar to a module (this is hidden behind an ifdef), and that work has been already merged to Seastar for quite some time. I'm not sure what are the next steps. Maybe we are waiting for clang 17 to be released?

from scylladb.

I'm using the following configuration for building Urchin:

[penberg@nero urchin]$ rm -rf build/release/
[penberg@nero urchin]$ ./configure.py --mode release --disable-xen --with seastar
[penberg@nero urchin]$ time ninja-build 
[86/86] LINK build/release/seastar

real    1m38.427s
user    10m22.715s
sys 0m32.587s

It's slow but bearable.

from scylladb.

On Tue, Mar 24, 2015 at 3:27 PM, Pekka Enberg [email protected]
wrote:

I'm using the following configuration for building Urchin:

[penberg@nero urchin]$ rm -rf build/release/
[penberg@nero urchin]$ ./configure.py --mode release --disable-xen --with seastar

Not compiling the tests indeed saves more than half the time. I also
suggested this as an option.

But isn't it sad that compiling just the tests - just 3,000 lines of code -
takes 5 minutes?
I'm sure there's something we can do about this, if we wanted to.

[penberg@nero urchin]$ time ninja-build
[86/86] LINK build/release/seastar

real 1m38.427s

On my machine, 2 cores + 2 hyperthreads, the same compilation you did
(without the tests) takes 3:51.
Maybe I need to ask for a better machine ;-)

from scylladb.

We certainly can do something about it and should do that. Are there any tools to "profile compilation" to detect which compilation units are slowing it down the most?

from scylladb.

2015-03-24 14:43 GMT+01:00 nyh [email protected]:

On Tue, Mar 24, 2015 at 3:27 PM, Pekka Enberg [email protected]
wrote:

I'm using the following configuration for building Urchin:

[penberg@nero urchin]$ rm -rf build/release/
[penberg@nero urchin]$ ./configure.py --mode release --disable-xen
--with seastar

Not compiling the tests indeed saves more than half the time. I also
suggested this as an option.

But isn't it sad that compiling just the tests - just 3,000 lines of code -
takes 5 minutes?
I'm sure there's something we can do about this, if we wanted to.

[penberg@nero urchin]$ time ninja-build
[86/86] LINK build/release/seastar

real 1m38.427s

On my machine, 2 cores + 2 hyperthreads, the same compilation you did
(without the tests) takes 3:51.
Maybe I need to ask for a better machine ;-)

That's not bad for a full recompile. For me the maximum compile time was
almost half an hour (probably full recompile), avg is 1.5 min (depends on
how many files are recompiled)

$ cat ~/.ninja-stats/stats | cut -d"," -f2 | series-stat.py
avg = 97.160396
stdev = 153.099398
min = 0.000000
max = 1622.000000
sum = 49066.000000
samples = 505

from scylladb.

On Tue, Mar 24, 2015 at 3:50 PM, Tomasz Grabiec [email protected]
wrote:

That's not bad for a full recompile.

You are suffering from http://en.wikipedia.org/wiki/Stockholm_syndrome :-)

9 minutes for a recompile is bad. 1.5 for an average compile is even
worse (it means too much needs to be recompiled on every small change).
This is a relatively small project...

It's even worse when you consider that more than half of this time is
compiling just 3,000 lines of code of tests.
What if, god forbid, we continue to write more tests?

I'm not saying there's something trivial we can do to fix this, but at
least let's admit that it's not a good situation.

Nadav Har'El
[email protected]

from scylladb.

2015-03-24 15:00 GMT+01:00 nyh [email protected]:

On Tue, Mar 24, 2015 at 3:50 PM, Tomasz Grabiec [email protected]
wrote:

That's not bad for a full recompile.

You are suffering from http://en.wikipedia.org/wiki/Stockholm_syndrome :-)

I was of course being sarcastic, the compile times are horrible.

9 minutes for a recompile is bad. 1.5 for an average compile is even
worse (it means too much needs to be recompiled on every small change).
This is a relatively small project...

It's even worse when you consider that more than half of this time is
compiling just 3,000 lines of code of tests.
What if, god forbid, we continue to write more tests?

I'm not saying there's something trivial we can do to fix this, but at
least let's admit that it's not a good situation.

Nadav Har'El
[email protected]

Reply to this email directly or view it on GitHub
#1 (comment)
.

from scylladb.

One of the things that mosts ruins my programming speed on Urchin is this:

Say I need to modify sstable.cc and sstable.hh with a large new feature. This causes 100 other files need to be compiled. Ok, fine. But what drives me crazy is that sstable.cc itself gets compiled at a random time - on average after compiling 50 other files which takes a L O N G time - and if I have several errors I often need to repeat this many times.

Even if we don't fix any of the other issues we talked about (smaller header files, less include-the-world, more pimpl, etc.), it would have really helped if Ninja simply found a way to compile sstable.cc first - and tell me my new error immediately.

There's actually a request for this on the ninja bug tracker: ninja-build/ninja#60

from scylladb.

Here's a little trick to do something similar to ninja#60, but manually:

If I see that build of build/debug/sstables/sstables.o is what failed last time, then I manually run

ninja build/debug/tests/urchin/sstables/sstables.o

next time, until the compilation resolve, and then I revert to the usual "ninja".

from scylladb.

I'm closing since we're not actively intend to change anything for the time being

from scylladb.

On Sun, Aug 30, 2015 at 10:31 AM, Dor Laor [email protected] wrote:

I'm closing since we're not actively intend to change anything for the
time being

Not being active on fixing this problem doesn't mean it's solved ;-)

from scylladb.

There is no point to have plenty of open issues if we don't intend to fix
them.
It's just clatters more immediate problems we need to handle

On Sun, Aug 30, 2015 at 11:09 AM, Avi Kivity [email protected]
wrote:

Reopened #1 #1.

—
Reply to this email directly or view it on GitHub
#1 (comment).

from scylladb.

There is no point in opening issues if they are closed in order to remove clutter.

Issues should be prioritized, not closed.

from scylladb.

From reading this thread I can conclude that the afternoon I just invested trying to get scylladb to build (plus the rest of the evening it seems I have to wait for the build to complete) has been wasted. Excuse me for saying this, but people who cannot make a C++ build go faster than this should really stick to using python. I have never in my life seen a slower build, and believe me I have seen some pretty crappy ones in my time.

from scylladb.

@jacobgorm Thanks for the advice, we truly appreciate it!

As for build times, they're slow, but not unbearably slow…

[penberg@nero scylla]$ ./configure.py --mode=release --with=scylla
[penberg@nero scylla]$ time ninja-build -j8
[271/271] LINK build/release/scylla

real    14m0.755s
user    102m8.942s
sys 4m7.185s

from scylladb.

...that's not to say that our build doesn't suck -- it does. There's tons of over eager header dependency issues to tackle, for example.

from scylladb.

It looks like some autogenerated (antlr?) header files is getting sucked in almost everywhere. Never before have I seen gcc consume >2GiB RAM per file it compiles. I have another C++ build (closed source) that I am working on (a 3d engine that a colleague wrote) that has almost exactly the same # LOC and also builds with ninja. Yet it takes less than 10 seconds for a full build, and less than .1 second for an empty build, on the same machine where scylladb took more than an hour. The difference is probably that it uses no STL headers, and that recursive includes are banned, so I would say that in comparison the scylladb build is indeed unbearably slow. For the sake of your project (that I have now given up on using for a customer POC, I will revert to plain cassandra for the time being) I urge you to fix this (probably trivial) build bug, before it wears down morale and scares away more potential users or customers. Nothing kills productivity like a slow build does.

from scylladb.

The problem is not excessive headers or STL. It is heavy use of templates and is directly derived from how the system works. If you think it is trivial, please post a patch.

from scylladb.

@avikivity is correct that templates are a big part of the problem, but it's not the entire problem: Another big problem for developers (rather than users who compile a single time), is header file spaghetti (most of the code is in headers, and the headers are very interdependent) so changing one source file causes almost the entire project to be recompiled.

from scylladb.

.idl.hh are fake dependencies, they won't cause 221 files to be rebuilt (and aren't included by 221 files), just re-evaluated. Try touch idl/* && ninja build/release/scylla; the compiler won't be invoked.

Note sure who the 194 includers of api/* are.

from scylladb.

I've deleted the comment, because the build wasn't complete to begin with. Once I have a complete build again to start from I will repost with real numbers (but it might take a while to get there.)

from scylladb.

Updated comment:
Using this script:

for i in `git ls-files| egrep '\.hh$'`
do
    touch -r $i ref
    touch $i
    (
        dirty=`ninja  build/release/scylla  -n -d explain 2>&1 |grep dirty | wc -l`
        echo $dirty $i
    )

    touch -r ref $i
done

This is the list of top offenders (via sort -r -n):
206 hashing.hh
206 bytes.hh
198 log.hh
195 utils/allocation_strategy.hh
195 gc_clock.hh
194 utils/UUID.hh
194 utils/serialization.hh
194 utils/managed_bytes.hh
194 db_clock.hh
193 types.hh
193 to_string.hh
193 tombstone.hh
193 timestamp.hh
193 exceptions/exceptions.hh
193 db/write_type.hh
193 db/consistency_level_type.hh
193 cql_serialization_format.hh
193 atomic_cell.hh
183 unimplemented.hh
183 cql3/column_specification.hh
182 schema.hh
182 keys.hh
182 json.hh
182 database_fwd.hh
182 compress.hh
182 compound.hh
182 compound_compat.hh
182 compaction_strategy.hh
182 caching_options.hh
180 range.hh
...

In terms of of headers file triggering rebuilds.

from scylladb.

The list looks fairly sane. Maybe compaction_strategy.hh can be replaced by forward declarations, but most of these headers are tiny, and are very stable.

from scylladb.

I am sorry, but I completely agree with @jacobgorm . I have built bigger projects than this, but I cant compile scylla from source. My computer has 6 Gb of RAM and it's not enough for compilation. Please dont tell me its sane or bearable. Its not.

from scylladb.

I found an interesting project, https://github.com/aras-p/ClangBuildAnalyzer, which lets you analyze the output of clang's "-ftime-trace" to find the most expensive files, header files, templates, functions and a lot more.
I attach the full report for the "dev" build mode on my laptop, report.txt
and give some examples from it:

**** Time summary:
Compilation (848 times):
  Parsing (frontend):        10455.2 s
  Codegen & opts (backend):   4066.0 s

**** Files that took longest to parse (compiler frontend):
 89473 ms: build/dev/message/messaging_service.o
 85188 ms: build/dev/main.o
 71943 ms: build/dev/service/storage_service.o
 62647 ms: build/dev/service/storage_proxy.o
 62082 ms: build/dev/db/system_keyspace.o
 57676 ms: build/dev/api/storage_service.o
 57078 ms: build/dev/table.o
 56864 ms: build/dev/sstables/sstables.o
 55186 ms: build/dev/db/schema_tables.o

**** Files that took longest to codegen (compiler backend):
129226 ms: build/dev/message/messaging_service.o
110786 ms: build/dev/sstables/sstables.o
 98445 ms: build/dev/service/storage_proxy.o
 93259 ms: build/dev/service/storage_service.o
 92827 ms: build/dev/main.o
 71921 ms: build/dev/db/schema_tables.o
 71636 ms: build/dev/table.o

These list is not very helpful, because it doesn't help us figure out why these specific source files are so slow. However, it might (?) be worthwhile to split the longest-compiling ones to improve parallelism.

**** Templates that took longest to instantiate:
138239 ms: fmt::formatter<fmt::basic_string_view<char>>::format<fmt::basic_form... (351 times, avg 393 ms)
 85267 ms: utils::loading_cache<std::pair<seastar::basic_sstring<signed char, u... (118 times, avg 722 ms)
 84533 ms: seastar::throw_with_backtrace<marshal_exception, seastar::basic_sstr... (312 times, avg 270 ms)
 84404 ms: seastar::make_backtraced_exception_ptr<marshal_exception, seastar::b... (312 times, avg 270 ms)

I don't know what we can do about those. Maybe it makes sense to have external instantiation (only instantiating a specific version of a template) for some of these. Note that 138 seconds is almost 1% of the entire build time (around 15,000 seconds) - just for compiling this specific template 351 times :-(

**** Template sets that took longest to instantiate:
644482 ms: std::unique_ptr<$> (23125 times, avg 27 ms)
608400 ms: std::__and_<$> (370713 times, avg 1 ms)
522728 ms: std::__uniq_ptr_data<$> (23125 times, avg 22 ms)
516971 ms: std::__uniq_ptr_impl<$> (23125 times, avg 22 ms)
442828 ms: std::__or_<$> (352542 times, avg 1 ms)
361947 ms: fmt::format_to<$> (13824 times, avg 26 ms)
331887 ms: std::tuple<$> (33022 times, avg 10 ms)
318621 ms: seastar::format<$> (10721 times, avg 29 ms)
316114 ms: std::optional<$> (19862 times, avg 15 ms)

These are huge numbers (644 seconds for compiling std::unique_ptr 23,125 times), but other than crying about the design of C++, I'm not sure what we can do about it.

**** Functions that took longest to compile:
 10869 ms: cql3_parser::CqlParser::unaliasedSelector() (build/dev/gen/cql3/CqlParser.cpp)
  8101 ms: main::$_2::operator()() const::'lambda0'()::operator()() const (main.cc)
  6122 ms: database::setup_metrics() (database.cc)
  5854 ms: cql3::query_processor::query_processor(service::storage_proxy&, data... (cql3/query_processor.cc)
  5311 ms: sstables::init_metrics()::$_74::operator()() const (sstables/sstables.cc)
  5067 ms: cql3::functions::add_agg_functions(std::unordered_multimap<cql3::fun... (cql3/functions/aggregate_fcts.cc)
  4036 ms: alternator::stats::stats() (alternator/stats.cc)

Maybe we can take a look at some of these functions to figure out why they are so slow. For example, stats() is just 60 lines of code (!!!) but uses a lot of obviously super-unoptimized boost options tricks :-(

**** Function sets that took longest to compile / optimize:
 37473 ms: fmt::v7::detail::int_writer<$>::on_num() (1356 times, avg 27 ms)
 17437 ms: seastar::continuation<$>::run_and_dispose() (1978 times, avg 8 ms)
 13365 ms: seastar::future<$> seastar::future<$>::then_impl<$>(seastar::noncopy... (1471 times, avg 9 ms)
 12203 ms: seastar::noncopyable_function<$>::direct_vtable_for<$>::call(seastar... (964 times, avg 12 ms)
 10217 ms: seastar::internal::repeat_until_value_state<seastar::data_source_imp... (359 times, avg 28 ms)
 10209 ms: char const* fmt::v7::detail::parse_align<$>(char const*, char const*... (238 times, avg 42 ms)

I don't know what we can do about these.

** Expensive headers:

This is probably the most interesting section. For example

1232665 ms: seastar/include/seastar/core/smp.hh (included 329 times, avg 3746 ms), included via:
  storage_proxy.o storage_proxy.hh sharded.hh  (11982 ms)
  schema_registry.o sharded.hh  (9354 ms)
  connection_notifier.o connection_notifier.hh query_context.hh sharded.hh  (9206 ms)
  storage_service.o storage_service.hh sharded.hh  (9066 ms)
  data_listeners.o data_listeners.hh distributed.hh sharded.hh  (9061 ms)
  stream_manager.o distributed.hh sharded.hh  (8783 ms)
  query_processor.o query_processor.hh sharded.hh  (8636 ms)
  keyspace_utils.o keyspace_utils.hh sharded.hh  (8596 ms)
  event.o event.hh inet_address.hh ip.hh arp.hh net.hh  (7286 ms)
  untyped_result_set.o untyped_result_set.hh sharded.hh  (7096 ms)
  large_data_handler.o query_context.hh sharded.hh  (6743 ms)
  ...

According to this, almost 8% of the entire build time is spent on compiling seastar/core/smp.hh included 329 times. It's important to understand: 1. If we need to include it so much, 2. Why it is so slow (almost 4 seconds) for each include and if it can be made faster, 3. Maybe parts of it can be moved to a different header file to speed up the typical inclusions of smp.hh.

1086901 ms: types.hh (included 312 times, avg 3483 ms), included via:
  error_injection_fcts.o error_injection_fcts.hh native_scalar_function.hh native_function.hh abstract_function.hh function.hh  (13927 ms)
  type_json.o type_json.hh  (13269 ms)
  selector.o selector.hh assignment_testable.hh column_specification.hh  (13126 ms)
  user_function.o user_function.hh abstract_function.hh function.hh  (12823 ms)
  functions.o functions.hh function.hh  (12210 ms)
  mutation_utils.o mutation_utils.hh  (12179 ms)
  type_parser.o type_parser.hh  (12110 ms)
  values.o values.hh  (12067 ms)
  column_specification.o column_specification.hh  (11222 ms)
  castas_fcts.o castas_fcts.hh function.hh  (10970 ms)
  user_types.o user_types.hh column_specification.hh  (10927 ms)
  ...

Another 7% of the build time from including "types.hh" 312 times.

591047 ms: database.hh (included 74 times, avg 7987 ms), included via:
  view_update_generator.o view_update_generator.hh  (21102 ms)
  cql_type_parser.o  (20704 ms)
  table.o  (20328 ms)
  validation.o  (18638 ms)
  database.o  (17775 ms)
  view.o  (17062 ms)
  legacy_schema_migrator.o  (16223 ms)
  streams.o  (16097 ms)
  snapshot-ctl.o  (15231 ms)
  main.o  (14970 ms)
  sstables-format-selector.o  (14135 ms)
  ...

I think @avikivity has already been working on reducing the includes of database.hh, but it's still included 74 times - and each one is very slow (8 seconds)

562433 ms: gms/inet_address.hh (included 291 times, avg 1932 ms), included via:
  event.o event.hh  (13498 ms)
  session_info.o session_info.hh  (12070 ms)
  gossiper.o  (12039 ms)
  storage_service.o storage_service.hh i_endpoint_state_change_subscriber.hh  (11846 ms)
  progress_info.o progress_info.hh  (11834 ms)

That is another header file - a rather silly one at that (how many source files need to handle IP addresses? Why does handling IP addresses take 2 seconds to compile?) included 291 times and single-handedly responsible for 3.7% of the build time. Heck, even seastar/core/future takes less of the build time (only slightly less, though).

from scylladb.

I think @avikivity has already been working on reducing the includes of database.hh, but it's still included 74 times - and each one is very slow (8 seconds)

562433 ms: gms/inet_address.hh (included 291 times, avg 1932 ms), included via:
event.o event.hh (13498 ms)
session_info.o session_info.hh (12070 ms)
gossiper.o (12039 ms)
storage_service.o storage_service.hh i_endpoint_state_change_subscriber.hh (11846 ms)
progress_info.o progress_info.hh (11834 ms)

I wonder how much compilation time was improved after we stopped including database.hh in compaction subsystem.

That is another header file - a rather silly one at that (how many source files need to handle IP addresses? Why does handling IP addresses take 2 seconds to compile?) included 291 times and single-handedly responsible for 3.7% of the build time. Heck, even seastar/core/future takes less of the build time (only slightly less, though).

from scylladb.

I'm now focusing on one header file <gms/inet_address.hh>, and the story is very interesting.
It is included directly or indirectly by almost every source file (291 files) and this can't easily change because we use IP addresses as keys for a lot of things in Scylla. But the problem is not that we include it a lot but the fact it takes a whopping 2 seconds in dev build mode for each of those includes. That's 600 seconds - 10 minutes of CPU time time - just for this silly header file defining an IP address. This can't make sense.

I'm now in the process of preparing Scylla and (unfortunately) Seastar patches to try to fix this situation. We should eventually do this for all "expensive" header files that ClangBuildAnalyzer reports.

from scylladb.

@nyh, when I have time I am planning to try the following, clang has this tool called iwyu (include what you use). It helps you to
only include the header files you actually use. It removes (or at least tell you about) anything that can be "solved" by forward
declarations. If my memory serves me right it even put a comment near every include what it is used for. This can help with
deciding if some include files needs to be broken down.

from scylladb.

perhaps we should split database.hh, with table moved into table.hh. there are a few relevant files which include table, but not database, follow: ['alternator/auth.cc', 'alternator/executor.cc', 'api/storage_proxy.cc', 'auth/service.cc', 'connection_notifier.cc', 'db/batchlog_manager.cc', 'db/consistency_level.cc', 'db/cql_type_parser.cc', 'dht/range_streamer.cc', 'gms/failure_detector.cc', 'gms/gossiper.cc', 'index/secondary_index_manager.cc', 'redis/mutation_utils.cc', 'redis/options.cc', 'service/paxos/paxos_state.cc', 'sstables/sstable_directory.cc', 'sstables/sstable_set.cc', 'sstables/sstables.cc', 'sstables_loader.cc', 'streaming/stream_session.cc', 'streaming/stream_transfer_task.cc']

from scylladb.

By the way, whoever wants to work on this issue, please note that there are two directions we can work on, both of them equally important:

1. Speeding up a full build. The invention of "dev mode" was a good step, and my patches from today (#9875) also went in this direction.
2. Speeding up partial builds during development by ensuring that most of the time when you modify one file, only a handful of other files need to be recompiled - not the entire project. The worst thing for this goal is to have most of your code in header files included by many other files - so when you need modify some header code, half the world needs to be recompiled.

Some things, like templates and template functions for everything, cause problems for both goals. Some solutions - like type erasure (std::function et al.), pimpl, etc. - are helpful for both. Some of these solutions hurt (a bit) the resulting code's performance, so one of the nice things that Avi did in the past is to enable some of these compile-time-saving solutions just on "dev" build mode, and still do the slow but optimal thing in release mode. But in many non-critical parts of the code, the less optimal version is good enough for all build modes.

from scylladb.

perhaps we should split database.hh, with table moved into table.hh. there are a few relevant files which include table, but not database, follow: ['alternator/auth.cc', 'alternator/executor.cc', 'api/storage_proxy.cc', 'auth/service.cc', 'connection_notifier.cc', 'db/batchlog_manager.cc', 'db/consistency_level.cc', 'db/cql_type_parser.cc', 'dht/range_streamer.cc', 'gms/failure_detector.cc', 'gms/gossiper.cc', 'index/secondary_index_manager.cc', 'redis/mutation_utils.cc', 'redis/options.cc', 'service/paxos/paxos_state.cc', 'sstables/sstable_directory.cc', 'sstables/sstable_set.cc', 'sstables/sstables.cc', 'sstables_loader.cc', 'streaming/stream_session.cc', 'streaming/stream_transfer_task.cc']

Many of these have nothing to do with the replica and should use data_dictionary instead.

from scylladb.

I know this is a completely different project, in a different language, but it's still inspiring to read Ingo Molnar post from last month on his project spanning more than a year and 2297 (!) patches to make the Linux kernel compile 78% faster by completely reworking the header files. He notes that the first few - or even first 1500 (!) patches - give only small compile-speed improvements, after you're done with all the cleanup and reorganization, suddenly you can get huge compilation speed improvements from the next patches.

https://lwn.net/ml/linux-kernel/[email protected]/

from scylladb.

Issue #12962 proposes that build speed can be improved by using the new C++ "modules" feature instead of old-style header files.

from scylladb.

Just an idea: how about using pre-compiled headers or C++20's modules in order to not compile some parts of the code over and over again?

from scylladb.

We were eager to use C++20 modules since their inception. We were doing first experiments 2 years ago. But the bottleneck is toolchain support for modules.

Issue #12962 was created specifically to track this effort.
@tchaikov would be nice to update that issue with more details on the current state and what it is that is holding us back today.

from scylladb.

We were eager to use C++20 modules since their inception. We were doing first experiments 2 years ago. But the bottleneck is toolchain support for modules.

Issue #12962 was created specifically to track this effort. @tchaikov would be nice to update that issue with more details on the current state and what it is that is holding us back today.

sure. i updated #12962

from scylladb.