Sharding causes keyspace to disappear about scylladb HOT 7 CLOSED

It's still open despite Tomek's 31d28c5

from scylladb.

@avikivity, seems like a sstable sharding issue?

from scylladb.

An sstable should be loaded on all shards on startup.

A compaction could have deleted it, but I doubt we have it running now.

Maybe a problem in range queries.

from scylladb.

It's an initialization race.

All shards initialize in parallel in init_from_data_directory(), which includes attaching sstables to the system keyspace column_familie:s and then querying of schema tables in system keyspace. But all this happens in parallel on all shards. So it may happen that some shard is already trying to query schema tables before another shard has finished attaching sstables it's responsible for. As a result some rows in schema tables may be missing. As a result, some shards may be missing some keyspaces / tables.

We should wait until system schema is fully populated on all shards before we proceed to querying schema tables.

from scylladb.

On 07/27/2015 08:14 PM, Tomasz Grabiec wrote:

It's an initialization race.

All shards initialize in parallel in |init_from_data_directory()|,
which includes attaching sstables to the system keyspace
column_familie:s and then querying of schema tables in system
keyspace. But all this happens in parallel on all shards. So it may
happen that some shard is already trying to query schema tables before
another shard has finished attaching sstables it's responsible for. As
a result some rows in schema tables may be missing. As a result, some
shards may be missing some keyspaces / tables.

We should wait until system schema is fully populated on all shards
before we proceed to querying schema tables.

—
Reply to this email directly or view it on GitHub
#47 (comment).

I can pick this up.

from scylladb.

On 07/27/2015 08:16 PM, Avi Kivity wrote:

On 07/27/2015 08:14 PM, Tomasz Grabiec wrote:

It's an initialization race.

All shards initialize in parallel in |init_from_data_directory()|,
which includes attaching sstables to the system keyspace
column_familie:s and then querying of schema tables in system
keyspace. But all this happens in parallel on all shards. So it may
happen that some shard is already trying to query schema tables
before another shard has finished attaching sstables it's responsible
for. As a result some rows in schema tables may be missing. As a
result, some shards may be missing some keyspaces / tables.

We should wait until system schema is fully populated on all shards
before we proceed to querying schema tables.

—
Reply to this email directly or view it on GitHub
#47 (comment).

I can pick this up.

Unless, of course, if you already have a handle on this can can easily
fix it.

from scylladb.

Fixed by b3b0d51.

from scylladb.