roc-streaming / rt-tests Goto Github PK
View Code? Open in Web Editor NEWReal-time tests for Roc Toolkit.
License: Mozilla Public License 2.0
Real-time tests for Roc Toolkit.
License: Mozilla Public License 2.0
Currently we have unit tests for internal components (src/tests
) and a few simple integration tests for public API (src/tests/public_api
).
All these tests are non-real-time, i.e. they don't expect any action, like delivering a packet, to take any specific amount of time. Actually they mostly are not tied to time at all. It has both advantages and disadvantages:
Tests can run in any conditions. We run them on CI, which workers are often slow, and on valgrind, which also randomly slows down the test execution.
On the other hand, such tests can't check how well Roc behaves in real-time conditions. E.g. what is the service quality and the latency, are there underruns and overruns, etc.
To fill this gap, we have created this repo, rt-tests. It will contain various real-time integration tests for public API.
This repo will contain non-functional tests, i.e. tests which result is not just OK/failed, but instead it may be some metric (e.g. 90% loss) or even a plot (e.g. latency trend). We will periodically run these tests on real hardware and analyze results for regressions. In future we will also run this tests on real networks.
Unlike the main repo, rt-tests uses CMake, C++17, and Google Test.
Currently rt-tests contains a skeleton of ServiceQuality test.
This issue is for completing that test.
The ServiceQuality test should start two threads. The first thread should write some predefined sequence of samples to the sender. The second should read samples from receiver and check the sequence.
There should be several sub-tests.
First we should check that the sample sequence is the same on sender and receiver (except possible leading zeros) when there is no packet loss.
Then we should introduce some amount of packet loss. It can be done by implementing a simple UDP-level proxy, like we do in our current integration test (but without using internal roc modules). Then check what is the service quality on different combinations of: FEC scheme, FEC block size, FEC redundancy level, and network loss level. We should test three FEC schemes: none, Reed-Solomon, LDPC-staircase.
The service quality can be measured as the percent of dropped samples, i.e. samples that are equal to zero instead of the expected value (on receiver). 100% service quality on 5% packet loss means that all 5% of lost packets were restored using FEC. 95% service quality on 10% of packet loss means that we restored 5% of packets, but was not able to restore another 5% of packets.
We can print a table with results (measured service quality values for different combinations of parameters) to stdout.
In future we will extend this test to run over real network to measure actual service quality on different network types.
This issue provides an example of how you can produce samples on sender and check them on receiver: roc-streaming/roc-go#101
That algorithm was implemented in this PR: roc-streaming/roc-go#111
These issue and PR are from roc-go repo (Go bindings for roc-toolkit), where we have a test that has some similarities with the one described here.
See "Workflow" section in README: https://github.com/roc-streaming/rt-tests#workflow
Public API is documented here: https://roc-streaming.org/toolkit/docs/api.html
FEC is documented here: https://roc-streaming.org/toolkit/docs/internals/fec.html
Our current non-real-time integration test can be found here:
https://github.com/roc-streaming/roc-toolkit/blob/master/src/tests/public_api/test_sender_receiver.cpp
We have created this repo for various real-time integration tests for public API. See #1 for details.
This issue is for adding a stress test to that repo.
The test should randomly create senders and receivers, perform some exchange, and destroy them. This should be done repeatedly from multiple threads.
We can implement this using a simple scheme. Start several threads. Each thread runs a loop: randomly choose some config parameters, create sender (or receiver), randomly choose to which currently running receiver to connect (for sender only), write (or read) samples during random time, destroy sender (or receiver), repeat the loop.
During the test, this algorithm should hit different combinations
Config parameters that should vary (randomly): FEC scheme, resampler profile, frame / packet / FEC block sizes, FEC redundancy level, packet interleaving, latency parameters. Also, we can bind receiver to: one port (bare RTP), two source & repair ports (reed-solomon or ldpc), or any other number of ports (e.g. multiple source & repair pairs for different or identical FEC schemes). Also we can have a separate context for each sender or receiver, or share context between a random number of senders and receivers.
The test should also perform some minimal check of the received data. E.g. we can send a monotonic sequence (1 2 3 4 ....) and check that we still receiver a monotonic sequence, probably with some amount of loss (lost samples will be zeros).
The user should be able to configure how long the test runs and how much threads to start.
We want to test quite a lot of combinations, but we can implement them incrementally: first implement and merge some minimal version, than iteratively add more randomness.
rt-tests repo currently has a skeleton of a test using public API:
https://github.com/roc-streaming/rt-tests/blob/1212a60b0357c851282be3ec1cb1887e652ab22e/tests/test_service_quality.cpp
Public API is documented here: https://roc-streaming.org/toolkit/docs/api.html
FEC is documented here: https://roc-streaming.org/toolkit/docs/internals/fec.html
Another integration test using public API can be found here:
https://github.com/roc-streaming/roc-toolkit/blob/master/src/tests/public_api/test_sender_receiver.cpp
We have created this repo for various real-time integration tests for public API. See #1 for details.
This issue is for adding API benchmarks.
Currently we have two time-sensitive operations in API: roc_sender_write() and roc_receiver_read(). It would be useful to cover them with benchmarks and periodically check them for regressions. This is especially important for weaker CPUs like ARM.
The performance heavily depends on sender or receiver configuration, so we should check different combinations of all important parameters: resampler profile, FEC scheme, frame, packet, and FEC block sizes, packet interleaving.
Benchmarks should be done with automatic timing disabled (see API) because otherwise we will measure performance of sleep.
Benchmarks can print a table with results (measured performance for different combinations of parameters) to stdout.
We likely should employ Google Benchmark library. Then we'll also need to add it to CMake. Google Test is already present there.
rt-tests repo currently has a skeleton of a test using public API:
https://github.com/roc-streaming/rt-tests/blob/1212a60b0357c851282be3ec1cb1887e652ab22e/tests/test_service_quality.cpp
Public API is documented here: https://roc-streaming.org/toolkit/docs/api.html
FEC is documented here: https://roc-streaming.org/toolkit/docs/internals/fec.html
Another integration test using public API can be found here:
https://github.com/roc-streaming/roc-toolkit/blob/master/src/tests/public_api/test_sender_receiver.cpp
Greetings @gavv, I am new to open source project development. The issue is that whenever I try to build this project, and call make
. it gives me
command failed: 2
make
This happens in the file rt-tests/build/googletest-prefix/src/googletest-stamp/googletest-build-Debug.cmake:49 (message):
I have attached two log files if that can be of any help. Thanks in advance.
googletest-build-out.log
googletest-build-err.log
ExternalProject_Add()
, build dependencies before other targets, serialize dependencies_USE_MATH_DEFINES
Similar to #1, which will measure "service quality" (in other words, number of losses/drops), we want to add an automated test that will measure latency.
I have a project that implements latency measurement: https://github.com/gavv/signal-estimator
It is tool, not a library, but it has an issue for implementing a library with C API: gavv/signal-estimator#33
We can implement that issue, and then implement a test in rt-tests, which will:
Allow cross-compiling project similar to how it's done here:
In particular:
Copy setup_toolchain.cmake script that sets all needed cmake vars and supports TOOLCHAIN_PREFIX flag for simple cases: https://github.com/roc-streaming/roc-pulse/blob/main/cmake/setup_toolchain.cmake
Copy/adjust corresponding sections to README linked above, including examples of compiling for arm32 and arm64 using docker.
In Makefile, use "build/$(uname -m)" instead of just "build" as build directory. In README examples for cross-compilation, use "build/<toolchain_name>". See here: https://github.com/gavv/signal-estimator/blob/main/Makefile
In CMakeLists.txt, use "bin/${CMAKE_C_COMPILER_TARGET}" instead of just "bin" to install binaries. See here: https://github.com/gavv/signal-estimator/blob/c09f02382a0afc1edffee8d8eb2f873c44110c34/CMakeLists.txt#L289
Ensure that at least cross-compiling to arm32 and arm64 using rocstreaming/toolchain-aarch64-linux-gnu and rocstreaming/toolchain-arm-linux-gnueabihf works. If needed, fix building of googletest and roc.
Add arm32 and arm64 to github actions.
We have created this repo for various real-time integration tests for public API. See #1 for details.
Among other things, it would be great to add integration tests for our command-line tools. Currently they are not covered with tests, and sometimes a breakage remains unnoticeable for a while.
This issue is for creating the very first test for them, to start with something simple. The test should generate a WAV file, run roc-send to send it to network, and run roc-recv to receive it from network and write to a local file. Then the test should analyze the received file and compare it with the sent one.
Again, see #1 for details on this comparison. The files wont be absolutely identical of course.
To run tools, we could use a library like subprocess. Good libraries for WAV files are dr_wav and libsndfile.
Related docs:
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.