Allow using AOF (the collection of commands) instead of RDB for full synchronization about valkey HOT 5 OPEN

@hpatro your question is very good. Perhaps I didn't express it clearly at the beginning, but I can explain it by answering your question.

The AOF file used during full synchronization does not rely on the appendonly config. It simply means that we originally used RDB files as snapshots, but now we have replaced it with a new AOF format snapshot. We also need to fork the child process, which converts all data into a set of write commands in AOF format. This way, the migration tool receives a set of commands during the full synchronization phase, eliminating the need to parse RDB encoding and achieving version independence.

from valkey.

Hi @PingXie , I mean when the migration tools parsing RDB, they have two choices:

1. for every key, send the value's whole payload via RESTORE command, thus it doesn't need to decode the different RDB format for each data type (hash/set/list etc.), but the payload may exceeds proto-max-bulk-len.
2. decode the RDB format of each different type of key value and convert it into commands. This way, although not limited by proto-max-bulk-len, the workload is significant.

from valkey.

@soloestoy I've not looked much into AOF, so my question might be very naive.

Wouldn't a user need to pay certain amount of performance penalty (would vary with appendfsync config level) to keep AOF enabled and store the collection of commands whereas with the RDB methodology, libraries needs to stay on top of the updates in the RDB versions but there isn't performance penalty for regular command execution.

from valkey.

@soloestoy, I think it is a great idea to use a no-preamble AOF and I will definitely consider using it for the atomic slot migration work (#23).

such as the restore command's parameters not exceeding 500MB (due to proto-max-bulk-len limitation)

Curious. Do we use the RESP protocol on full sync today? Going through https://github.com/valkey-io/valkey/blob/unstable/src/rdb.c#L3031, my impression is that we don't. If so, are we still bound by proto-max-bulk-len?

from valkey.

Got it. This is not about the full sync but the migration tool. That makes sense. I think another benefit of using no-preamble AOF in either the full sync or slot migration is that we can easily achieve the non-blocking behavior.

from valkey.