roc-streaming / rt-tests Goto Github PK

• Build using gcc and clang.
• Build debug and release.
• Run tests (currently they are built but not executed). Later, when we'll add long tests, we'll need to exclude them from CI or run in some short mode, but currently we have only a single test and it's short.

We have created this repo for various real-time integration tests for public API. See #1 for details.

Among other things, it would be great to add integration tests for our command-line tools. Currently they are not covered with tests, and sometimes a breakage remains unnoticeable for a while.

This issue is for creating the very first test for them, to start with something simple. The test should generate a WAV file, run roc-send to send it to network, and run roc-recv to receive it from network and write to a local file. Then the test should analyze the received file and compare it with the sent one.

Again, see #1 for details on this comparison. The files wont be absolutely identical of course.

To run tools, we could use a library like subprocess. Good libraries for WAV files are dr_wav and libsndfile.

Related docs:

• https://roc-streaming.org/toolkit/docs/running/command_line_tools.html
• https://roc-streaming.org/toolkit/docs/manuals/roc_send.html
• https://roc-streaming.org/toolkit/docs/manuals/roc_recv.html

Task

We have created this repo for various real-time integration tests for public API. See #1 for details.

This issue is for adding API benchmarks.

Currently we have two time-sensitive operations in API: roc_sender_write() and roc_receiver_read(). It would be useful to cover them with benchmarks and periodically check them for regressions. This is especially important for weaker CPUs like ARM.

The performance heavily depends on sender or receiver configuration, so we should check different combinations of all important parameters: resampler profile, FEC scheme, frame, packet, and FEC block sizes, packet interleaving.

Benchmarks should be done with automatic timing disabled (see API) because otherwise we will measure performance of sleep.

Benchmarks can print a table with results (measured performance for different combinations of parameters) to stdout.

We likely should employ Google Benchmark library. Then we'll also need to add it to CMake. Google Test is already present there.

Info

rt-tests repo currently has a skeleton of a test using public API:
https://github.com/roc-streaming/rt-tests/blob/1212a60b0357c851282be3ec1cb1887e652ab22e/tests/test_service_quality.cpp

Public API is documented here: https://roc-streaming.org/toolkit/docs/api.html

FEC is documented here: https://roc-streaming.org/toolkit/docs/internals/fec.html

Another integration test using public API can be found here:
https://github.com/roc-streaming/roc-toolkit/blob/master/src/tests/public_api/test_sender_receiver.cpp

Allow cross-compiling project similar to how it's done here:

• https://github.com/roc-streaming/roc-pulse#cross-compilation
• https://github.com/roc-streaming/roc-pulse#prebuilt-toolchains

In particular:

• Copy setup_toolchain.cmake script that sets all needed cmake vars and supports TOOLCHAIN_PREFIX flag for simple cases: https://github.com/roc-streaming/roc-pulse/blob/main/cmake/setup_toolchain.cmake

• Copy/adjust corresponding sections to README linked above, including examples of compiling for arm32 and arm64 using docker.

• In Makefile, use "build/$(uname -m)" instead of just "build" as build directory. In README examples for cross-compilation, use "build/<toolchain_name>". See here: https://github.com/gavv/signal-estimator/blob/main/Makefile

• In CMakeLists.txt, use "bin/${CMAKE_C_COMPILER_TARGET}" instead of just "bin" to install binaries. See here: https://github.com/gavv/signal-estimator/blob/c09f02382a0afc1edffee8d8eb2f873c44110c34/CMakeLists.txt#L289

• Ensure that at least cross-compiling to arm32 and arm64 using rocstreaming/toolchain-aarch64-linux-gnu and rocstreaming/toolchain-arm-linux-gnueabihf works. If needed, fix building of googletest and roc.

• Add arm32 and arm64 to github actions.

Similar to #1, which will measure "service quality" (in other words, number of losses/drops), we want to add an automated test that will measure latency.

I have a project that implements latency measurement: https://github.com/gavv/signal-estimator

It is tool, not a library, but it has an issue for implementing a library with C API: gavv/signal-estimator#33

We can implement that issue, and then implement a test in rt-tests, which will:

• ask signal-estimator library to produce output frames
• pass these frames to roc sender
• in another thread, retrieve frames from roc receiver
• pass retrieved frames to signal-estimator
• ask signal-estimator for measured latency and report it

• Download and build roc-toolkit instead of using system-wide installation. Allow to specify branch/ref
• Use ExternalProject_Add(), build dependencies before other targets, serialize dependencies
• Cleanup warnings options
• Add _USE_MATH_DEFINES
• Enable sanitizers and glibc assertions in debug build

Task

We have created this repo for various real-time integration tests for public API. See #1 for details.

This issue is for adding a stress test to that repo.

The test should randomly create senders and receivers, perform some exchange, and destroy them. This should be done repeatedly from multiple threads.

We can implement this using a simple scheme. Start several threads. Each thread runs a loop: randomly choose some config parameters, create sender (or receiver), randomly choose to which currently running receiver to connect (for sender only), write (or read) samples during random time, destroy sender (or receiver), repeat the loop.

During the test, this algorithm should hit different combinations

• sender without receiver
• one sender attached to one receiver
• multiple senders attached to one receiver

Config parameters that should vary (randomly): FEC scheme, resampler profile, frame / packet / FEC block sizes, FEC redundancy level, packet interleaving, latency parameters. Also, we can bind receiver to: one port (bare RTP), two source & repair ports (reed-solomon or ldpc), or any other number of ports (e.g. multiple source & repair pairs for different or identical FEC schemes). Also we can have a separate context for each sender or receiver, or share context between a random number of senders and receivers.

The test should also perform some minimal check of the received data. E.g. we can send a monotonic sequence (1 2 3 4 ....) and check that we still receiver a monotonic sequence, probably with some amount of loss (lost samples will be zeros).

The user should be able to configure how long the test runs and how much threads to start.

We want to test quite a lot of combinations, but we can implement them incrementally: first implement and merge some minimal version, than iteratively add more randomness.

Info

rt-tests repo currently has a skeleton of a test using public API:
https://github.com/roc-streaming/rt-tests/blob/1212a60b0357c851282be3ec1cb1887e652ab22e/tests/test_service_quality.cpp

Public API is documented here: https://roc-streaming.org/toolkit/docs/api.html

FEC is documented here: https://roc-streaming.org/toolkit/docs/internals/fec.html

Another integration test using public API can be found here:
https://github.com/roc-streaming/roc-toolkit/blob/master/src/tests/public_api/test_sender_receiver.cpp

Intro

Currently we have unit tests for internal components (src/tests) and a few simple integration tests for public API (src/tests/public_api).

All these tests are non-real-time, i.e. they don't expect any action, like delivering a packet, to take any specific amount of time. Actually they mostly are not tied to time at all. It has both advantages and disadvantages:

• Tests can run in any conditions. We run them on CI, which workers are often slow, and on valgrind, which also randomly slows down the test execution.

• On the other hand, such tests can't check how well Roc behaves in real-time conditions. E.g. what is the service quality and the latency, are there underruns and overruns, etc.

To fill this gap, we have created this repo, rt-tests. It will contain various real-time integration tests for public API.

This repo will contain non-functional tests, i.e. tests which result is not just OK/failed, but instead it may be some metric (e.g. 90% loss) or even a plot (e.g. latency trend). We will periodically run these tests on real hardware and analyze results for regressions. In future we will also run this tests on real networks.

Unlike the main repo, rt-tests uses CMake, C++17, and Google Test.

Task

Currently rt-tests contains a skeleton of ServiceQuality test.

https://github.com/roc-streaming/rt-tests/blob/1212a60b0357c851282be3ec1cb1887e652ab22e/tests/test_service_quality.cpp

This issue is for completing that test.

The ServiceQuality test should start two threads. The first thread should write some predefined sequence of samples to the sender. The second should read samples from receiver and check the sequence.

There should be several sub-tests.

First we should check that the sample sequence is the same on sender and receiver (except possible leading zeros) when there is no packet loss.

Then we should introduce some amount of packet loss. It can be done by implementing a simple UDP-level proxy, like we do in our current integration test (but without using internal roc modules). Then check what is the service quality on different combinations of: FEC scheme, FEC block size, FEC redundancy level, and network loss level. We should test three FEC schemes: none, Reed-Solomon, LDPC-staircase.

The service quality can be measured as the percent of dropped samples, i.e. samples that are equal to zero instead of the expected value (on receiver). 100% service quality on 5% packet loss means that all 5% of lost packets were restored using FEC. 95% service quality on 10% of packet loss means that we restored 5% of packets, but was not able to restore another 5% of packets.

We can print a table with results (measured service quality values for different combinations of parameters) to stdout.

In future we will extend this test to run over real network to measure actual service quality on different network types.

Checks

This issue provides an example of how you can produce samples on sender and check them on receiver: roc-streaming/roc-go#101

That algorithm was implemented in this PR: roc-streaming/roc-go#111

These issue and PR are from roc-go repo (Go bindings for roc-toolkit), where we have a test that has some similarities with the one described here.

Info

See "Workflow" section in README: https://github.com/roc-streaming/rt-tests#workflow

Public API is documented here: https://roc-streaming.org/toolkit/docs/api.html

FEC is documented here: https://roc-streaming.org/toolkit/docs/internals/fec.html

Our current non-real-time integration test can be found here:
https://github.com/roc-streaming/roc-toolkit/blob/master/src/tests/public_api/test_sender_receiver.cpp

Greetings @gavv, I am new to open source project development. The issue is that whenever I try to build this project, and call make. it gives me
command failed: 2
make
This happens in the file rt-tests/build/googletest-prefix/src/googletest-stamp/googletest-build-Debug.cmake:49 (message):

I have attached two log files if that can be of any help. Thanks in advance.
googletest-build-out.log
googletest-build-err.log