Memory issue of kvrocks2redis detected by ASan/TSan about kvrocks HOT 12 OPEN

@jihuayu Sure thing. Log attached.
logs_21881009491.zip

from kvrocks.

@AntiTopQuark Not yet, thank you!

from kvrocks.

@AntiTopQuark Thanks for your great investigation. Can we relax the check condition? like check it N times with a fixed interval and return once it's passed. cc @PragmaTwice What do you think?

from kvrocks.

Thank you @Zakelly !
The action log will be deleted by GitHub after some days. Can you download the log file and upload?

from kvrocks.

IMHO we should spend some time on investigating this issue and find a solution before next release. cc @git-hulk

from kvrocks.

@PragmaTwice I may take some time to investigate this after resolving #2253, and it'd be great if other guys would like to dive deep into this issue.

from kvrocks.

@PragmaTwice I may take some time to investigate this after resolving #2253, and it'd be great if other guys would like to dive deep into this issue.

I think this issue is related to #1988 which makes kvrocks2redis crash.
It is important to keep kvrocks2redis usable, without crashing.

from kvrocks.

@PragmaTwice I may take some time to investigate this after resolving #2253, and it'd be great if other guys would like to dive deep into this issue.

@git-hulk
Hi, have you started studying this core? If not I kind of want to try it.

from kvrocks.

@Zakelly Hi, I've investigated this issue and successfully reproduced it on my computer.

I suspect it's not a memory issue, but rather that compiling with ASAN and TSAN significantly impacts the performance of kvrocks2redis. This leads to slower data synchronization in kvrocks2redis.

And a simple sleep(0.02)(url) in the check_consistency.py test script might not give Redis enough time to sync the corresponding results, resulting in execution errors.

As shown in the diagram below, although the test failed, the corresponding keys and values can still be found in Redis afterward.

Or we can try to make the test script sleep longer?

from kvrocks.

@Zakelly Hi, I've investigated this issue and successfully reproduced it on my computer.

I suspect it's not a memory issue, but rather that compiling with ASAN and TSAN significantly impacts the performance of kvrocks2redis. This leads to slower data synchronization in kvrocks2redis.

And a simple sleep(0.02)(url) in the check_consistency.py test script might not give Redis enough time to sync the corresponding results, resulting in execution errors.

As shown in the diagram below, although the test failed, the corresponding keys and values can still be found in Redis afterward.

Or we can try to make the test script sleep longer?

cc @git-hulk

from kvrocks.

Yeah we can adjust 0.02 to see if it works.

from kvrocks.

Happy to see the great investigation! Thanks @AntiTopQuark

from kvrocks.