[NEW] Introduce automated cross version and cross fork testing infrastructure about valkey HOT 10 OPEN

It would be great if anyone wants to add such tests, because we're a bit overloaded at this point. It doesn't have to be in TCL (maybe Python instead?) and it doesn't even have to be in this repo, if we're testing different forks. We can make a separate interop testing repo. It is about

• cluster bus (nodes of different versions in the same cluster)
• replication (primary/replica of different versions)
• RDB and AOF files (read and write by different versions)
• Anything else?

from valkey.

@valkey-io/core-team WDYT ?

from valkey.

i am willing to spend time. if anyone guide me some starting point.

from valkey.

We can make a separate interop testing repo. It is about

I think it should be this repo, I don't think we want to start introducing tests elsewhere for now.

from valkey.

I'm fine either way, but what id'd like to see is some prototyping to get something running. Use any language, any client lib, but be ready to discard it later.

from valkey.

There are potential benefits of having a completely separate repo for this. It would need to check out various versions of valkey anyway (so it can't just run out of the checked out repo). It could also test valkey against various forks and versions. We can have test a cluster with mixed nodes of KeyDB, Dragonfly, Redict, Redis, Valkey, various versions. We can run redis testsuites on the valkey binary. Etc.

from valkey.

@madolson @zuiderkwast @pragnesh @mattsta Hi everyone! Is anyone already working on this?
We currently have a Python test script that provides the same functionality as the TCL tests, but written in Python. It has been working well for the past two years and has significantly improved the efficiency of writing and debugging tests compared to TCL.
Here is a relatively old version: https://pypi.org/project/pybbt/
If you're interested, I'm willing to spend 1-2 weeks adapting it for Valkey and writing a few test examples.

from valkey.

@suxb201 how is the test setup/teardown done? How long does a similar tcl test takes in python?

from valkey.

@hpatro

1. To create an instance, use a clear statement.
2. Instances made inside an @subcase are safely destroyed when the @subcase ends.
3. Instances created within an @case will be destroyed when the @case concludes.
4. The @subcase is concurrent, which makes testing efficient.
5. There is only one @case per file, and the tests are composed of multiple files.
6. Logs, AOF and RDB files will be arranged in a temporary directory, making it easier to debug.

Here's a example:

from testsuite import *


# Master can replicate command longer than client-query-buffer-limit on replica
@subcase()
def replication_query_buffer_limit():
    t0 = Tair()  # create a process
    t1 = Tair()  # create another process
    t1.wait_slaveof(t0)  # call 'slaveof' and then wait for the replication connection to be set up
    t0.do("config", "set", "client-query-buffer-limit", 2000000)
    t1.do("config", "set", "client-query-buffer-limit", 1048576)  # 1024*1024 = 1mb

    value = "x" * 1100000
    ASSERT_TRUE(t0.do("set", "k", value))  # 2000000 > 1100000 > 1048576
    t0.wait_consistent()
    ASSERT_EQ(t1.do("get", "k"), value.encode())
    ASSERT_EQ(t1.digest(), t0.digest())


@case(tags=["flag0", "flag1", "flag2"])
def main():
    replication_query_buffer_limit()


if __name__ == '__main__':
    main()

from valkey.

It seems to me that porting the tests to a different language is a different topic. I think it can be discussed here: #94.

Personally, I think it would be beneficial to create a separate interoperability-testing repo where scripts and CI jobs can

• check out multiple versions of Valkey, Redis, clients and other tools
• test mixed clusters, e.g. test #52
• test replication between different versions
• create dumps (rdb and aof) and load them it in other versions, newer and older

@bjosv

from valkey.