end to end latency over 50ms about aes67-linux-daemon HOT 51 CLOSED

good news on this issue.
I have finally found the root cause of the latency issue reported and documented by @anbraten in the document he shared.
The issue is actually related to the driver constrains applied to the ALSA buffer periods.
If you look at the audio chain used for the RAVENNA device (both playback and capture, option -vv with alsaloop) you will see that despite the latency setting applied by the application the ALSA buffer used is always >= 1536 frames leading to the latency we have. I am working on a fix.

from aes67-linux-daemon.

ok, it looks good then. I see the end to end latency you have is similar to mine now.
We cannot do better in my opinion as now we are using a PCM buffer that is 64 (tic_frame_size_at_1fs) frames * 2 periods = 128 frames. A smaller buffer would lead to under-runs.
I look forward receiving the result of the test with the oscilloscope.

from aes67-linux-daemon.

FYI: I tried reducing the latency for a project as well and best I got was ~50ms with my setup using this driver.
https://www.creative-technologies.de/setup-and-evaluation-of-an-media-ip-real-time-system/

from aes67-linux-daemon.

Thanks , interesting - not sure what they have done (if anything) to the aes67 stuff to make it lower latency.

I can hack together simple TX/RX command line stuff test stuff and send raw stereo PCM audio over UDP point to point with
very small latency over my setup so I know the hardware, network etc is fine.

from aes67-linux-daemon.

I also tried using a gstreamer command on the CM4 that is the player and sending to both the local audio output and plughw:RAVENNA at the same time. This works but the latency here is 1.2s, so maybe thats the problem?

So you have 1.2s latency when you play on the local audio card too ?
If yes, can you try to play using aplay with the option -v and send me the output ?
Also, in general, how do you measure the latency ?

from aes67-linux-daemon.

The expected audio latency when running the default configuration on bare metal (no containers/VMs} with a 1ms Source is about 1.5ms. See picture from Dante Controller included in #17

from aes67-linux-daemon.

playing on the local PI audio card is fine.
I measure large latencies with a stopwatch :)

from aes67-linux-daemon.

ok I'm trying with another PI and building AES67 from scratch. However, the driver fails to build now:
sudo insmod 3rdparty/ravenna-alsa-lkm/driver/MergingRavennaALSA.ko doesn't work because of build error
from build.sh:
./build.sh Cloning into 'ravenna-alsa-lkm'... remote: Enumerating objects: 843, done. remote: Counting objects: 100% (843/843), done. remote: Compressing objects: 100% (450/450), done. remote: Total 843 (delta 400), reused 807 (delta 368), pack-reused 0 Receiving objects: 100% (843/843), 5.25 MiB | 1.02 MiB/s, done. Resolving deltas: 100% (400/400), done. make -C /lib/modules/5.10.60-v7l+/build/ M=/home/pi/aes67-linux-daemon/3rdparty/ravenna-alsa-lkm/driver modules make[1]: *** /lib/modules/5.10.60-v7l+/build/: No such file or directory. Stop. make: *** [Makefile:14: modules] Error 2 .

from aes67-linux-daemon.

It seems the playback issue is the buffering for alsa plughw:RAVENNA.

I tried another setup with a modern desktop PC with Lubuntu 64 bit and still got the same delay when it was a player.
Using a command line like this to set a buffer seems to improve things quite a bit:

aplay testfile48k24bit.wav -B 8000 -D plughw:RAVENNA

from aes67-linux-daemon.

ok, I see: when you measure the latency you include the time spent in the ALSA playback buffer and this depends on the size of such buffer. Please consider that the shorter the buffer the higher the chance of under-run errors on the ALSA playback device.
I usually measure the network latency only and this starts when the RTP timestamp is added to the packet in the Source till the packet gets received and processed by the Sink (both the Source and the Sink are clocked by the same PTP clock).

The driver architecture was meant to minimize the network latency: the whole process of RTP packet creation, timestamping, scheduling and delivery to the network interface is fully implemented in the kernel and scheduled by a kernel high resolution timer whose period is set by the tic_frame_size_at_1fs daemon parameter.

from aes67-linux-daemon.

any update on this ? What end-to-end latency could you reach and how do you measure it ? Thanks

from aes67-linux-daemon.

Did a quick test with some tone bursts and an oscilloscope. send sound to local DAC at the same time as sending to AES67 alsa device. Using small buffer sizes I could get down to around 20ms but as you mention small buffer sizes cause under runs. Have other issues somewhere with slow sync drift maybe related to PTP so trying to solve those first. Unfortunately run out of time for a while on this. Hope to revisit it in a week or so.

from aes67-linux-daemon.

OK thanks for the info.
I have never made these measurements.
You can try to change the player priority in case you have other applications running on the PI to decrease the frequency of the under-runs.
Please let me know how it goes.

from aes67-linux-daemon.

any news on this ?

from aes67-linux-daemon.

It looks like it might be more like 50-100mS for best reliability on a normal Linux system. Thats just about usable for audio playout.
Real-time OS and jack audio server running real-time might improve things a bit.

from aes67-linux-daemon.

I have improved latency a little bit on the receiver by running a jack server to handle playback on this device. This gives a latency of around 70mS. However it seems this driver does not work with jack as an input device. (also doesn't work as out put device) Also the latency slowly increases - after half an hour it crept up by 30mS.

Using this results in an input device error:
jackd -d alsa -P hw:sndrpihifiberry -C RAVENNA -r 48000 -p 256 -n2

jackdmp 1.9.12 JACK server starting in realtime mode with priority 10 self-connect-mode is "Don't restrict self connect requests" ALSA lib control.c:1375:(snd_ctl_open_noupdate) Invalid CTL RAVENNA control open "RAVENNA" (No such file or directory) audio_reservation_init Acquire audio card Audio0 creating alsa driver ... hw:sndrpihifiberry|RAVENNA|256|2|48000|0|0|nomon|swmeter|-|32bit ALSA lib pcm.c:2565:(snd_pcm_open_noupdate) Unknown PCM RAVENNA ALSA: Cannot open PCM device alsa_pcm for capture. Falling back to playback-only mode configuring for 48000Hz, period = 256 frames (5.3 ms), buffer = 2 periods ALSA: final selected sample format for playback: 32bit integer little-endian ALSA: use 2 periods for playback ^CJack main caught signal 2 ALSA lib control.c:1375:(snd_ctl_open_noupdate) Invalid CTL RAVENNA control open "RAVENNA" (No such file or directory) Released audio card Audio0 audio_reservation_finish

from aes67-linux-daemon.

what about trying with (corrections in bold):
jackd -d alsa -P hw:sndrpihifiberry -C hw:RAVENNA -r 48000 -p 64 -n 2
Also please make sure that you have tic_frame_size_at_1fs set to 64 in daemon.conf

from aes67-linux-daemon.

Regarding the drift I don't know how jack handles the audio clock drift between the capture and the playback devices.
The RAVENNA device uses the PTP clock as master clock while the other audio device uses another so a drift is not avoidable.
alsaloop is able to handle it using different strategies selectable via the -S | --sync= option.

from aes67-linux-daemon.

thanks for the suggestions. On my receiver the jack server runs and I can play a test sound through the output:
gst-launch-1.0 -vv audiotestsrc ! jackaudiosink

I can also test without jack and use alsa which works as before (but not what I need):
arecord -t raw -f S24_LE -c 2 -r 48000 -D plughw:RAVENNA | aplay -t raw -f S24_LE -c 2 -r 48000 -D default

However I can't loop jack like this - no sound! - it is patching in jack though.
gst-launch-1.0 -vv jackaudiosrc ! jackaudiosink

I recorded the stream as a wav file:
gst-launch-1.0 j-vv ackaudiosrc ! queue ! audioconvert ! wavenc ! filesink location=aa.wav
However the file has a size and the record showed no errors, but it is silent.

AES67-daemon all ok no errors and it clearly works with plain alsa.

Any clues? I really need it working this way full-duplex with jack, then I can test properly.

Re:drift - understood I can see this might happen. Maybe there is a way to get the aes67-ptp clock into gstreamer?
Gstreamer does have a clockselect option and I did try it but not sure if it did anything.
https://gstreamer.freedesktop.org/documentation/debugutilsbad/clockselect.html?gi-language=c#clockselect-page

from aes67-linux-daemon.

apparently jackd doesn't do any clock drift correction so my suggestion is to not use it for this use case.
The best solution in my opinion is to use alsaloop.
If you are using the setup described in:
https://www.creative-technologies.de/setup-and-evaluation-of-an-media-ip-real-time-system/
you can try to run alsaloop with the following parameters (latency should be set to 480frames =10ms):
sudo nice -n -10 alsaloop -c 2 -r 48000 -f S16_LE -P hw:sndrpihifiberry -C hw:RAVENNA -S 5 -v

from aes67-linux-daemon.

Thanks looks like that is a possible solution

from aes67-linux-daemon.

Excellent - look forward to testing the next version.

from aes67-linux-daemon.

if you checkout the end_to_end_latency branch you can run a test to measure the real end to end latency introduced by RAVENNA playback and capture devices.
For example:

./run_latency_test.sh S24_3LE 48000 2 10

To run a 10 seconds latency test.
With the current driver I get:

End to end latency: 56 ms

A patch to the driver to fix this will follow soon.

from aes67-linux-daemon.

I tried the alsaloop playback method which works, but there is a drift over a few minutes that slowly increases then an under-run then the process repeats as show below:

`sudo nice -n -10 alsaloop -c 2 -r 48000 -f S24_LE -P hw:sndrpihifiberry -C hw:RAVENNA -S 5 -v
Scheduler set to Round Robin with priority 99
Hardware PCM card 0 'snd_rpi_hifiberry_dac' device 0 subdevice 0
Its setup is:
stream : PLAYBACK
access : RW_INTERLEAVED
format : S24_LE
subformat : STD
channels : 2
rate : 48000
exact rate : 48000 (48000/1)
msbits : 32
buffer_size : 1920
period_size : 240
period_time : 5000
tstamp_mode : NONE
tstamp_type : MONOTONIC
period_step : 1
avail_min : 1560
period_event : 0
start_threshold : 2147483647
stop_threshold : 1920
silence_threshold: 0
silence_size : 0
boundary : 2013265920
appl_ptr : 0
hw_ptr : 0
Hardware PCM card 1 'Merging RAVENNA' device 0 subdevice 0
Its setup is:
stream : CAPTURE
access : RW_INTERLEAVED
format : S24_LE
subformat : STD
channels : 2
rate : 48000
exact rate : 48000 (48000/1)
msbits : 32
buffer_size : 1536
period_size : 64
period_time : 1333
tstamp_mode : NONE
tstamp_type : MONOTONIC
period_step : 1
avail_min : 120
period_event : 0
start_threshold : 2147483647
stop_threshold : 1536
silence_threshold: 0
silence_size : 0
boundary : 1610612736
appl_ptr : 0
hw_ptr : 0
Latency 480 frames, 10000.000us, 10.000000ms (100.0000Hz)
playback hw:sndrpihifiberry/capture hw:RAVENNA sync type: SIMPLE

New pitch for playback hw:sndrpihifiberry/capture hw:RAVENNA: 1.00000000 (min/max samples = 0/0)
New pitch for playback hw:sndrpihifiberry/capture hw:RAVENNA: 0.99998958 (min/max samples = -18/0)
New pitch for playback hw:sndrpihifiberry/capture hw:RAVENNA: 0.99997917 (min/max samples = -30/0)
New pitch for playback hw:sndrpihifiberry/capture hw:RAVENNA: 0.99996875 (min/max samples = -36/0)
New pitch for playback hw:sndrpihifiberry/capture hw:RAVENNA: 0.99995833 (min/max samples = -46/0)
New pitch for playback hw:sndrpihifiberry/capture hw:RAVENNA: 0.99994792 (min/max samples = -55/0)
New pitch for playback hw:sndrpihifiberry/capture hw:RAVENNA: 0.99993750 (min/max samples = -63/0)
New pitch for playback hw:sndrpihifiberry/capture hw:RAVENNA: 0.99992708 (min/max samples = -82/0)
New pitch for playback hw:sndrpihifiberry/capture hw:RAVENNA: 0.99991667 (min/max samples = -104/0)
New pitch for playback hw:sndrpihifiberry/capture hw:RAVENNA: 0.99991667 (min/max samples = -104/0)
New pitch for playback hw:sndrpihifiberry/capture hw:RAVENNA: 0.99990625 (min/max samples = -117/0)
New pitch for playback hw:sndrpihifiberry/capture hw:RAVENNA: 0.99990625 (min/max samples = -117/0)
New pitch for playback hw:sndrpihifiberry/capture hw:RAVENNA: 0.99989583 (min/max samples = -145/0)
New pitch for playback hw:sndrpihifiberry/capture hw:RAVENNA: 0.99989583 (min/max samples = -145/0)
New pitch for playback hw:sndrpihifiberry/capture hw:RAVENNA: 0.99988542 (min/max samples = -148/0)
New pitch for playback hw:sndrpihifiberry/capture hw:RAVENNA: 0.99987500 (min/max samples = -154/0)
New pitch for playback hw:sndrpihifiberry/capture hw:RAVENNA: 0.99986458 (min/max samples = -175/0)
New pitch for playback hw:sndrpihifiberry/capture hw:RAVENNA: 0.99985937 (min/max samples = -175/0)
New pitch for playback hw:sndrpihifiberry/capture hw:RAVENNA: 0.99984896 (min/max samples = -178/0)
New pitch for playback hw:sndrpihifiberry/capture hw:RAVENNA: 0.99983854 (min/max samples = -196/0)
New pitch for playback hw:sndrpihifiberry/capture hw:RAVENNA: 0.99983854 (min/max samples = -196/0)
New pitch for playback hw:sndrpihifiberry/capture hw:RAVENNA: 0.99982812 (min/max samples = -215/0)
New pitch for playback hw:sndrpihifiberry/capture hw:RAVENNA: 0.99982812 (min/max samples = -215/0)
New pitch for playback hw:sndrpihifiberry/capture hw:RAVENNA: 0.99981771 (min/max samples = -215/0)
New pitch for playback hw:sndrpihifiberry/capture hw:RAVENNA: 0.99980729 (min/max samples = -233/0)
New pitch for playback hw:sndrpihifiberry/capture hw:RAVENNA: 0.99980729 (min/max samples = -233/0)
New pitch for playback hw:sndrpihifiberry/capture hw:RAVENNA: 0.99979687 (min/max samples = -249/0)
New pitch for playback hw:sndrpihifiberry/capture hw:RAVENNA: 0.99978646 (min/max samples = -252/0)
New pitch for playback hw:sndrpihifiberry/capture hw:RAVENNA: 0.99977604 (min/max samples = -262/0)
New pitch for playback hw:sndrpihifiberry/capture hw:RAVENNA: 0.99976562 (min/max samples = -264/0)
New pitch for playback hw:sndrpihifiberry/capture hw:RAVENNA: 0.99976562 (min/max samples = -264/0)
New pitch for playback hw:sndrpihifiberry/capture hw:RAVENNA: 0.99975521 (min/max samples = -279/0)
New pitch for playback hw:sndrpihifiberry/capture hw:RAVENNA: 0.99974479 (min/max samples = -291/0)
New pitch for playback hw:sndrpihifiberry/capture hw:RAVENNA: 0.99973437 (min/max samples = -302/0)
New pitch for playback hw:sndrpihifiberry/capture hw:RAVENNA: 0.99972396 (min/max samples = -319/0)
New pitch for playback hw:sndrpihifiberry/capture hw:RAVENNA: 0.99972396 (min/max samples = -319/0)
New pitch for playback hw:sndrpihifiberry/capture hw:RAVENNA: 0.99972396 (min/max samples = -319/0)
New pitch for playback hw:sndrpihifiberry/capture hw:RAVENNA: 0.99971354 (min/max samples = -345/0)
New pitch for playback hw:sndrpihifiberry/capture hw:RAVENNA: 0.99970312 (min/max samples = -353/0)
New pitch for playback hw:sndrpihifiberry/capture hw:RAVENNA: 0.99969271 (min/max samples = -370/0)
underrun for playback hw:sndrpihifiberry
New pitch for playback hw:sndrpihifiberry/capture hw:RAVENNA: 0.99968229 (min/max samples = -142/0)
New pitch for playback hw:sndrpihifiberry/capture hw:RAVENNA: 0.99967187 (min/max samples = -151/0)`

from aes67-linux-daemon.

I will be releasing a patch for the problem with the high latency.
The clock drift is another problem and the solution should come from alsaloop.
Can you try to play with the different sync option ?

from aes67-linux-daemon.

The different sync options appear to make no difference. The resampler option does not run at all.

from aes67-linux-daemon.

I have just pushed a change to the driver repo to allow the usage of smaller ALSA buffer periods.
The latency test now reports an end to end latency of 7ms on my ARM board:

./run_latency_test.sh S24_3LE 48000 2 10
Compiling tools ...
g++    -c -o check.o check.cc
g++   check.o   -o check
g++    -c -o createtest.o createtest.cc
g++   createtest.o   -o createtest
g++    -c -o latency.o latency.cc
g++ latency.o -o latency  -lpthread -lasound
Creating configuration files ..
net.ipv4.igmp_max_memberships = 66
Starting PTP master ...
Starting AES67 daemon ...
Waiting for PTP slave to sync ...
Running 10 secs latency test
Scheduler set to Round Robin with priority 99...
Playback device is hw:RAVENNA
Capture device is hw:RAVENNA
Parameters are 48000Hz, S24_3LE, 2 channels, non-blocking mode
Poll mode: no
Loop limit is 480000 frames, minimum latency = 128, maximum latency = 128
Hardware PCM card 1 'Merging RAVENNA' device 0 subdevice 0
Its setup is:
  stream       : PLAYBACK
  access       : RW_INTERLEAVED
  format       : S24_3LE
  subformat    : STD
  channels     : 2
  rate         : 48000
  exact rate   : 48000 (48000/1)
  msbits       : 24
  buffer_size  : 128
  period_size  : 64
  period_time  : 1333
  tstamp_mode  : NONE
  tstamp_type  : MONOTONIC
  period_step  : 1
  avail_min    : 64
  period_event : 0
  start_threshold  : 2147483647
  stop_threshold   : 128
  silence_threshold: 0
  silence_size : 0
  boundary     : 4611686018427387904
  appl_ptr     : 0
  hw_ptr       : 0
Hardware PCM card 1 'Merging RAVENNA' device 0 subdevice 0
Its setup is:
  stream       : CAPTURE
  access       : RW_INTERLEAVED
  format       : S24_3LE
  subformat    : STD
  channels     : 2
  rate         : 48000
  exact rate   : 48000 (48000/1)
  msbits       : 24
  buffer_size  : 128
  period_size  : 64
  period_time  : 1333
  tstamp_mode  : NONE
  tstamp_type  : MONOTONIC
  period_step  : 1
  avail_min    : 64
  period_event : 0
  start_threshold  : 2147483647
  stop_threshold   : 128
  silence_threshold: 0
  silence_size : 0
  boundary     : 4611686018427387904
  appl_ptr     : 0
  hw_ptr       : 0
Trying latency 128 frames, 2666.667us, 2.666667ms (375.0000Hz)
Success
Playback:
*** frames = 480128 ***
  state       : RUNNING
  trigger_time: 21131.675386
  tstamp      : 0.000000
  delay       : 128
  avail       : 0
  avail_max   : 64
Capture:
*** frames = 480000 ***
  state       : RUNNING
  trigger_time: 21131.675390
  tstamp      : 0.000000
  delay       : 0
  avail       : 0
  avail_max   : 64
Maximum read: 64 frames
Maximum read latency: 1333.333us, 1.333333ms (750.0000Hz)
Playback time = 21131.675386, Record time = 21131.675390, diff = -4
End to end latency: 7 ms
Terminating processes ...
daemon exiting with code: 0

This happens because we can now program the driver to use a smaller number of periods. Indeed the RAVENNA buffer size is now set 128 frames.
(Please note that a similar test fails on my mini PC with x86 and I have to use -m 256 -M 256 options when running the latency tool).

@steeley
@anbraten
can you please repeat the measurements with the oscilloscope to double check this ?

My suggestion is to specify a latency > 4ms when using alsaloop (-t 4000) and to use a buffer size > 512 frames when using aplay ( --buffer-size 512).

Thanks !

from aes67-linux-daemon.

The different sync options appear to make no difference. The resampler option does not run at all.

This requires some investigations on alsaloop

from aes67-linux-daemon.

To test with the new driver checkout the end_to_end_latency branch cleanup the repo using the cleanup.sh and rebuild using build.sh

from aes67-linux-daemon.

thanks - will look at this next week.

from aes67-linux-daemon.

Hi -- very interested in this work on lower latency. I just finished compiling the 5.16-rc3 kernel for rpi4B, with the PREEMPT_RT patch and the fully preemptible kernel enabled. I was able to compile the end_to_end_latency branch daemon and driver against this. The platform test is running fine.

However, the latency test yields an error about setting Round Robin, and I'm not sure the results are valid. Is there something I can do to troubleshoot this?

• Neil.

pi@raspberrypi:~/aes67-linux-daemon$ ./run_latency_test.sh S24_3LE 48000 2 10
Compiling tools ...
make: Nothing to be done for 'all'.
Creating configuration files ..
net.ipv4.igmp_max_memberships = 66
Starting PTP master ...
Starting AES67 daemon ...
Waiting for PTP slave to sync ...
Running 10 secs latency test
!!!Scheduler set to Round Robin with priority 99 FAILED!!!
Playback device is hw:RAVENNA
Capture device is hw:RAVENNA
Parameters are 48000Hz, S24_3LE, 2 channels, non-blocking mode
Poll mode: no
Loop limit is 480000 frames, minimum latency = 128, maximum latency = 128
Hardware PCM card 3 'Merging RAVENNA' device 0 subdevice 0
Its setup is:
stream : PLAYBACK
access : RW_INTERLEAVED
format : S24_3LE
subformat : STD
channels : 2
rate : 48000
exact rate : 48000 (48000/1)
msbits : 24
buffer_size : 128
period_size : 64
period_time : 1333
tstamp_mode : NONE
tstamp_type : MONOTONIC
period_step : 1
avail_min : 64
period_event : 0
start_threshold : 2147483647
stop_threshold : 128
silence_threshold: 0
silence_size : 0
boundary : 1073741824
appl_ptr : 0
hw_ptr : 0
Hardware PCM card 3 'Merging RAVENNA' device 0 subdevice 0
Its setup is:
stream : CAPTURE
access : RW_INTERLEAVED
format : S24_3LE
subformat : STD
channels : 2
rate : 48000
exact rate : 48000 (48000/1)
msbits : 24
buffer_size : 128
period_size : 64
period_time : 1333
tstamp_mode : NONE
tstamp_type : MONOTONIC
period_step : 1
avail_min : 64
period_event : 0
start_threshold : 2147483647
stop_threshold : 128
silence_threshold: 0
silence_size : 0
boundary : 1073741824
appl_ptr : 0
hw_ptr : 0
Trying latency 128 frames, 2666.667us, 2.666667ms (375.0000Hz)
Success
Playback:
*** frames = 480128 ***
state : RUNNING
trigger_time: 31573.036136
tstamp : 0.000000
delay : 128
avail : 0
avail_max : 64
Capture:
*** frames = 480000 ***
state : RUNNING
trigger_time: 31573.036137
tstamp : 0.000000
delay : 0
avail : 0
avail_max : 64
Maximum read: 64 frames
Maximum read latency: 1333.333us, 1.333333ms (750.0000Hz)
Playback time = 31573.36136, Record time = 31573.36137, diff = -1
End to end latency: 0 ms
Terminating processes ...
daemon exiting with code: 0

from aes67-linux-daemon.

Not sure about the result. I would try to test with the real setup and check with the oscilloscope.

from aes67-linux-daemon.

Pretty sure it's flawed, perhaps due to the problem setting the Round Robin scheduler?

from aes67-linux-daemon.

No idea at the moment. The fact that you see the End to end latency print it means some samples were captured but I never had a 0 ms as result.
The test is based on a quick hack of an existing application and it can contain bugs. Maybe something related to the way I am reading the system clock.

from aes67-linux-daemon.

solved the Round Robin error -- I ran the test script using sudo.

Still getting 0 ms latency, though, so something isn't quite right. I'll try a different method.

from aes67-linux-daemon.

I did a few changes to the latency application.
Can you please re-try your test after pulling the branch ?

from aes67-linux-daemon.

BTW the patch to the driver is working since I see in your logs that the RAVENNA buffer size is set to 128 frame:
buffer_size : 128
whereas it was 1536 before.

from aes67-linux-daemon.

Had a quick run of the new latency test, results below:

run_latency_test.sh S24_3LE 48000 2 10

Starting PTP master ... Starting AES67 daemon ... Waiting for PTP slave to sync ... Running 10 secs latency test Scheduler set to Round Robin with priority 99... Playback device is hw:RAVENNA Capture device is hw:RAVENNA Parameters are 48000Hz, S24_3LE, 2 channels, non-blocking mode Poll mode: no Loop limit is 480000 frames, minimum latency = 128, maximum latency = 128 Hardware PCM card 1 'Merging RAVENNA' device 0 subdevice 0 Its setup is: stream : PLAYBACK access : RW_INTERLEAVED format : S24_3LE subformat : STD channels : 2 rate : 48000 exact rate : 48000 (48000/1) msbits : 24 buffer_size : 128 period_size : 64 period_time : 1333 tstamp_mode : NONE tstamp_type : MONOTONIC period_step : 1 avail_min : 64 period_event : 0 start_threshold : 2147483647 stop_threshold : 128 silence_threshold: 0 silence_size : 0 boundary : 1073741824 appl_ptr : 0 hw_ptr : 0 Hardware PCM card 1 'Merging RAVENNA' device 0 subdevice 0 Its setup is: stream : CAPTURE access : RW_INTERLEAVED format : S24_3LE subformat : STD channels : 2 rate : 48000 exact rate : 48000 (48000/1) msbits : 24 buffer_size : 128 period_size : 64 period_time : 1333 tstamp_mode : NONE tstamp_type : MONOTONIC period_step : 1 avail_min : 64 period_event : 0 start_threshold : 2147483647 stop_threshold : 128 silence_threshold: 0 silence_size : 0 boundary : 1073741824 appl_ptr : 0 hw_ptr : 0 Trying latency 128 frames, 2666.667us, 2.666667ms (375.0000Hz) Failure Playback: *** frames = 45824 *** state : XRUN trigger_time: 547.242262 tstamp : 0.000000 delay : 0 avail : 64 avail_max : 64 Capture: *** frames = 45696 *** state : XRUN trigger_time: 547.242263 tstamp : 0.000000 delay : 0 avail : 128 avail_max : 128 Maximum read: 64 frames Maximum read latency: 1333.333us, 1.333333ms (750.0000Hz) End to end latency: 7.999 msecs Terminating processes ... daemon exiting with code: 0

from aes67-linux-daemon.

With the new latency test, I no longer need to run via sudo. Running on the stock Raspberry Pi OS kernel for now.

I got the essentially the same results as steeley -- end-to-end latency was 7.997 msecs -- but just like in his results, I get a line stating "Failure" after "Trying latency 128 frames..."

Also, both Playback and Capture show a state of XRUN after around 45000 frames.

Repeating the test up to -M 1024 -m 1024, I still get failures at the same point. The end to end latency reads 63.993 secs

What kind of ARM boards are you guys using? I'm on a 4GB RPi4B -- perhaps the board Andrea is using has much better latency?

-N.

from aes67-linux-daemon.

I'm using a RPI cm4 compute module running at 1.8Ghz.

My audio test now seems instant, and looks like latency is well below 10mS .When I hit 'play' the sound replays
pretty much immediately on the receiver.

Play:
aplay test2448.wav -D plughw:RAVENNA -B 4000 -v

Listen1:
sudo nice -n -10 alsaloop -c 2 -r 48000 -f S24_LE -P hw:sndrpihifiberry -C hw:RAVENNA -S 5 -v

shows lots of pitch messages, then a small glitch and underrun after a few minutes like this:

New pitch for playback hw:sndrpihifiberry/capture hw:RAVENNA: 0.99966667 (min/max samples = -375/12)
underrun for playback hw:sndrpihifiberry
New pitch for playback hw:sndrpihifiberry/capture hw:RAVENNA: 0.99965625 (min/max samples = -141/0)
New pitch for playback hw:sndrpihifiberry/capture hw:RAVENNA: 0.99964583 (min/max samples = -148/0)

It seems these new pitch messages occur even when I'm not sending any audio.

Play2, using gstreamer as player, clocking from input source and resampling to output:
gst-launch-1.0 -vv alsasrc device=plughw:RAVENNA ! audioresample quality=10 ! audio/x-raw, rate=48000 ! alsasink

I think the resampling keeps the sync/drift issue to. a minimum, but ideally I'd rather not use it.
Don't see/hear any dropouts as far as I can tell. Also gstreamer reports the source stream as s16le when it is definitely s24le but sounds ok.

It would be good if it was possible to make the master AES67 device also the PTP clock master.
I think this is normally disabled (so your daemon/driver won't find the local PTP source.)

from aes67-linux-daemon.

Steeley --

If you run ptp4l on the master AES67 device on both the ethernet and loopback interfaces, the daemon can lock to it. I had this problem, too, before.

from aes67-linux-daemon.

What kind of ARM boards are you guys using? I'm on a 4GB RPi4B -- perhaps the board Andrea is using has much better latency?

I am using a Nano Pi Neo 2 running Ubuntu 20.04.1 LTS
I am using the generic kernel provided with this distro:
Linux nanopineo2 5.8.8-sunxi64 #trunk SMP Sun Sep 13 01:50:54 PDT 2020 aarch64 aarch64 aarch64 GNU/Linux
On the top of this I took care to stop the pulseaudio daemon and uninstall pulseaudio package.

from aes67-linux-daemon.

sudo nice -n -10 alsaloop -c 2 -r 48000 -f S24_LE -P hw:sndrpihifiberry -C hw:RAVENNA -S 5 -v

You can try to use option -t 4000 when running alsaloop to reduce the latency.

from aes67-linux-daemon.

What kind of ARM boards are you guys using? I'm on a 4GB RPi4B -- perhaps the board Andrea is using has much better latency?

I am using a Nano Pi Neo 2 running Ubuntu 20.04.1 LTS I am using the generic kernel provided with this distro: Linux nanopineo2 5.8.8-sunxi64 #trunk SMP Sun Sep 13 01:50:54 PDT 2020 aarch64 aarch64 aarch64 GNU/Linux On the top of this I took care to stop the pulseaudio daemon and uninstall pulseaudio package.

Interesting. You're running arm64...I wonder if running the standard Raspberry Pi OS 32-bit is hurting mine and Steeley's performance. Note the substantial improvement in latency sysbench cpu tests using a 64-bit system: https://forums.raspberrypi.com/viewtopic.php?t=243567

I know it's not exactly the same thing, but the difference is quite dramatic. Will test this a bit and get back to you.

from aes67-linux-daemon.

Interesting. You're running arm64...I wonder if running the standard Raspberry Pi OS 32-bit is hurting mine and Steeley's performance. Note the substantial improvement in latency sysbench cpu tests using a 64-bit system: https://forums.raspberrypi.com/viewtopic.php?t=243567

This is confirmed my side too.
If I repeat the latency test on a BeagleBone Black with Linux arm 4.14.108-ti kernel I achieve a working configuration starting from 3072 frames on (-m 3072)
Probably also the number of CPU core can make a difference in this regard.

from aes67-linux-daemon.

Working on this. Built a 64-bit image, as identically as possible to the 32-bit image I was using. Same hardware. sysbench shows cpu latency less than 1/20th of the 32-bit platform.

The Daemon platform test runs well, tested up to 4 channels for 5 min at 24bit/48kHz.

The latency test only runs at 2 channels of 24/48 if I set the buffer to >=3072 frames, and I get the following errors no matter what the testing parameters are:

playback device does not support 2 periods per buffer
capture device does not support 2 periods per buffer

Full report below.

pi@raspberrypi:~/aes67-linux-daemon $ ./run_latency_test.sh S24_3LE 48000 2 10
Compiling tools ...
make: Nothing to be done for 'all'.
Creating configuration files ..
net.ipv4.igmp_max_memberships = 66
Starting PTP master ...
Starting AES67 daemon ...
Waiting for PTP slave to sync ...
Running 10 secs latency test
Scheduler set to Round Robin with priority 99...
Playback device is hw:RAVENNA
Capture device is hw:RAVENNA
Parameters are 48000Hz, S24_3LE, 2 channels, non-blocking mode
Poll mode: no
Loop limit is 480000 frames, minimum latency = 3072, maximum latency = 3072
playback device does not support 2 periods per buffer
capture device does not support 2 periods per buffer
Hardware PCM card 3 'Merging RAVENNA' device 0 subdevice 0
Its setup is:
stream : PLAYBACK
access : RW_INTERLEAVED
format : S24_3LE
subformat : STD
channels : 2
rate : 48000
exact rate : 48000 (48000/1)
msbits : 24
buffer_size : 3072
period_size : 64
period_time : 1333
tstamp_mode : NONE
tstamp_type : MONOTONIC
period_step : 1
avail_min : 64
period_event : 0
start_threshold : 2147483647
stop_threshold : 3072
silence_threshold: 0
silence_size : 0
boundary : 6917529027641081856
appl_ptr : 0
hw_ptr : 0
Hardware PCM card 3 'Merging RAVENNA' device 0 subdevice 0
Its setup is:
stream : CAPTURE
access : RW_INTERLEAVED
format : S24_3LE
subformat : STD
channels : 2
rate : 48000
exact rate : 48000 (48000/1)
msbits : 24
buffer_size : 3072
period_size : 64
period_time : 1333
tstamp_mode : NONE
tstamp_type : MONOTONIC
period_step : 1
avail_min : 64
period_event : 0
start_threshold : 2147483647
stop_threshold : 3072
silence_threshold: 0
silence_size : 0
boundary : 6917529027641081856
appl_ptr : 0
hw_ptr : 0
Trying latency 3072 frames, 64000.000us, 64.000000ms (15.6250Hz)
Success
Playback:
*** frames = 483072 ***
state : RUNNING
trigger_time: 1794.563367
tstamp : 0.000000
delay : 3072
avail : 0
avail_max : 2496
Capture:
*** frames = 480000 ***
state : RUNNING
trigger_time: 1794.563370
tstamp : 0.000000
delay : 0
avail : 0
avail_max : 2432
Maximum read: 1536 frames
Maximum read latency: 32000.000us, 32.000000ms (31.2500Hz)
Playback time = 1794.563367, Record time = 1794.563370, diff = -3
End to end latency: 157.694 msecs
Terminating processes ...
daemon exiting with code: 0

from aes67-linux-daemon.

This execution is OK You can ignore the errors.

from aes67-linux-daemon.

Steeley --

If you run ptp4l on the master AES67 device on both the ethernet and loopback interfaces, the daemon can lock to it. I had this problem, too, before.

Interesting - I've used ptp4l before for a few things - didn't notice it can handle 2 interfaces at the same time.
whats your command line for this?

thanks

from aes67-linux-daemon.

I believe that you can use -i eth0 -i lo at the command line (the -i switch can be used multiple times), but I just set up a quick ptp4l.conf file, which saves me some repetitive work:

[eth0]
delay_mechanism E2E

[lo]
delay_mechanism E2E

[global]
domainNumber 0
time_stamping software
dscp_event 46
dscp_general 34

from aes67-linux-daemon.

are ok that makes sense, thanks

from aes67-linux-daemon.

changes to the Ravenna driver plus latency tests and documentation merged to main branches.

from aes67-linux-daemon.