Strange Opus audio bug affecting at least 2 of our users. about mumble HOT 68 CLOSED

Hmm can't add multiple files. Attached User1 dxdiag and audio sample.

Some system info from the murmur server.
OS: Arch Linux x86_64
Hostname: salmac
Uptime: 49 days, 6:35
Kernel: 3.6.10-1-ARCH
Shell: /bin/bash
Packages: 326
Window Manager: unknown
RAM: 2466 MB / 7973 MB (30%)
CPU: AMD FX(tm)-4100 Quad-Core Processor
Boot: 31M / 464M (7%) (ext4)
Root: 72G / 440G (18%) (ext4)

The following attachments were added on the original comment:

• https://sourceforge.net/p/mumble/bugs/_discuss/thread/c56f11bc/ada4/attachment/User1.DxDiag%26Flac.7z

from mumble.

IRC log for clarification.

[10:26] <Ben_Mumble_com> RED-404: what is the problem? "Ear shattering pain" for the listeners?
[10:27] <RED-404> yes check the sample file in User1.DxDiag&Flac.7z
[10:28] <RED-404> its only with opus and every listener in the chan will hear it
[10:29] <Ben_Mumble_com> I see
[10:31] <DireFog> Sheesh that sounds like Metallica
[10:32] <Ben_Mumble_com> "using opus, certain users appear to send sound that is "ear shattering pain" at random times: Is that description of the problem?
[10:33] <RED-404> we'll all users I think the problem is on the broadcasters' side
[10:34] <DireFog> does everything they say sound like that, or it it just for a while when they start talking?
[10:35] <RED-404> one thing we were going to check is if it had something to do with cool & quiet. The CPU jumping frequencies from ~800 MHz to 2.4GHz
[10:36] <DireFog> that shouldn't be a problem unless people are on really broken WinXP systems
[10:36] <RED-404> and its not constant but its frequent
[10:38] <Ben_Mumble_com> in the "ear shattering pain" do you hear voice at all?
[10:39] <DireFog> so they get normal after a while without restarting the client?
[10:40] <RED-404> I have a 6 min audio clip of him and he only does it one time and yes I hear voice sometime it lasts for 4 to 6 words and its still understandable
[10:40] <RED-404> god I can't spell need sleep
[10:41] <RED-404> and no it never normalizes or fixes itself it can be as frequent as every third word or as infrequent as not happening for 2hr
[10:41] <DireFog> could you check if they have the maximum gain for AGC set really high? It could be an effect of extreme clipping hitting the CODEC
[10:42] <DireFog> hum then it's rather something different
[10:42] <Ben_Mumble_com> RED-404: I would submit this conversation with your bug report for further clarity
[10:42] <RED-404> normally the gain is set between default and 1

from mumble.

Wish you would have included the issue description in your initial report. I spent forever trying to find if a bug report was opened for this.

We recently started using opus on our server and ever since then messages that my friend transmits have a small chance to cause ear shattering pain. Maybe once every 100 times he talks it does this, but when it happens it's like a bomb went off and my ears ring. It is seriously THAT LOUD which in my opinion makes this a very serious bug (potential for hearing damage). It's to the point where I cannot use Mumble with him anymore unless I don't use the headset, because it causes physical pain to my ears. If there is another user in the channel they hear it too. You can barely understand what they were trying to say in their transmitted message because it seems like the gain is set to +10000dB. Then the next thing they say will be at normal, human-safe volume.

We are all using the latest snapshot. He is using windows 8 with his laptop's integrated Conexant audio with the latest drivers available.

This never happened when it was using the old celt codec. Not even once.

from mumble.

I can also confirm that we've had this issue on our Mumble server as well. Are there any updates on this issue?

from mumble.

Unfortunately not at this point. We haven't been able to willfully reproduce this issue up to now so if anyone can come up with reliable steps to do so that would be incredibly helpful.

from mumble.

I'd like to confirm I am also seeing this issue with both my audio output on Linux and two other users I know on Windows. Some users do it some don't no idea how to reproduce.

from mumble.

I had a friend switch from push to talk to voice activation and now he seems to have the problem too maybe this will help with reproducing the error. Settings are:

Mumble version: 1.2.4
Echo: Mixed
Voice Activity: Amplitude
Voice Hold: 0.50s

I highly think it's to do with using voice activation.

from mumble.

Can't produce this issue at will, but at least two of the users on my server have this issue as well--the first second or so of activity sounds like it's being put through an amp set as high as it goes (distortion, incredibly loud). What can I get you guys to help diagnose this issue?

from mumble.

Our server seems to be experiencing this issue as well. It can happen in the middle of someone speaking. It happens to people on push to talk or voice activation. We cannot reproduce it at will and have no idea what factors might cause it. When it happens, everyone hearing the person will hear the sound. Afterward, if the person had 0 lost/late packets before, they still will afterwards. Our only option has been to set Opus threshold to 100% and keep a non-opus-supporting client connecting, forcing CELT.

from mumble.

A recorded audio file with this issue can be found here: http://files.natenom.com/public/mumble/bugs/baeh/01.wav
Maybe helpful.

from mumble.

I have the same problem affecting everyone in my server. It occurs mostly when people first start talking. It's being hosted on an Amazon EC2 instance running Ubuntu 12.04. The server is using the default config, but the noise it occurs with both opusthreshold=100 and opusthreshold=0. Mumble Server Information lists the CELT 0.11.0/Opus codec with 72 kbit/s max.

from mumble.

I recorded 2 instances of this for some more examples (mixed into 1 flac). This is on the Opus codec.
https://druska.s3.amazonaws.com/2+mumble+loud+screeches.flac

from mumble.

When talking with Natenom today he said I had the issue (Opus audio artifact at beginning of talking) quite often.
After I disabled noise suppression it was gone.
After a bit I enabled noise suppression again (-30dB) and the issue occured again, but with lower intensity (I was told).

from mumble.

Thanks for the report, Kissaki. I'll test that as well.

from mumble.

I'm running a mumble server on Ubuntu 14.04. We experience this issue randomly. It seems to follow certain users more then others.

from mumble.

Also experiencing this issue. Only change I made on the server is limiting the bandwidth to 60kb/s. Once in a while someone talks and there is ear shattering pain from the volume.

from mumble.

I too have been having this issues, has anyone found a work around? I've tried having my players reduce their quality settings down, but it doesn't seem to help even at 10kb/s.

from mumble.

The best workaround is to disable Opus. It sucks, but at least your ears will stop bleeding.

from mumble.

How do you disable Opus, I can't seem to find any reference for it?

from mumble.

@Atrixium http://wiki.mumble.info/wiki/Murmur.ini#opusthreshold
If you set this value over 100, it can never be reached, thus the server will always use CELT.

I've been using this workaround since Opus was first added into beta (which is years now).

from mumble.

@jzelinskie
Thanks, this works for me. Hopefully that bug in Opus will get fixed soon.

from mumble.

Really glad to see there at least is a workaround. After the last bug-induced screech on mumble, my ear is actually hurting.

from mumble.

Has a root cause been identified? Are there any planned fixes? I'm glad to see there's a workaround, but not so glad to see we have to lose voice quality for it.

from mumble.

Is there anything we can do to narrow down the cause of this? I'd be willing to run a debug build of Mumble (and have the users on my server "causing" the problem do the same) to try and determine what causes this.

from mumble.

This issue has been plaguing my server for a while now, and it seems to be getting worse. It follows me the most- other users occasionally transmit a pop or quick beep at the front of a transmission, but apparently my client is sending out long screeches that sound just like the attached audio samples that have been posted earlier in this thread. Found this thread, tried disabling noise suppression, no luck. We're all running push to talk, and since I rent the server through mumble.com, their UI won't let me set the opus threshold over 100. (Can anyone think of another workaround in the meantime?)

If this helps at all, long ago before this screeching problem started but after the 1.2.4 release, some of our users found out how to intermittently induce a strange audio artifact. By disconnecting and reconnecting to the server while holding the push to talk key and speaking right as the connection is made, sometimes the transmission would get a strange burst of reverberation, making the transmitted voice sound like the fictional "Geth" character Legion from the Mass Effect series, or like shouting into a PVC tube- some of our users found it quite amusing but it was never a 100% certain occurrence. Potentially unrelated, but I figured it bears mentioning.

from mumble.

can confirm it is screech-tastic. tested with mumble-server hosted on both debian and windows
Started off only affecting certain users but now everyone is having a screech. Trying the workaround, hope it works

from mumble.

Our server is having similar issues, we actually stopped running Mumble for a while because of it. We decided to give the 1.3.0 snapshot a try today and we encountered the screeching bug a few hours into use. Not sure how we can help, but we'd love to go back to OPUS as soon as it's fixed. In the mean time, we've switched back to CELT.

from mumble.

Didn't want to post anything until I had time enough time to test this, but I've found a non-CELT workaround. It's been working for a few weeks now, no screeches where they had previously occurred almost hourly.

It's something to do with the compression quality settings. I set mine to the highest quality available (96.0kb/s) and the lowest audio per packet (10 ms) - this greatly increases the bandwidth needed, but apparently fixes the screech. Noise suppression is off as well.

from mumble.

I can give this a try. Is there some way to force these settings on all clients? Or will I have to instruct all incoming users to change their settings?

from mumble.

We have good experience with just disabling the "Max. amplification" in "Audio Input". I produced the bug very often and since the change it didin't occur anymore.

from mumble.

I've been on the receiving end of this issue for a while. I had been suggesting, on a hunch, that max amplification be set no higher than 2 (where I have it set). It seemed to be doing the trick, but then as I was directing someone with the screech to lower it, he said it was already set to 2, so I asked him to set it to 1 and it seems to be working fine after about an hour, then I proceeded to look more into the issue and ended up here.

Most of the people I talk with are on Windows 7 or 10, while I'm on Linux Mint 17.2 and a couple friends dual-boot Windows with Mint or Ubuntu. All clients have OPUS support. I've confirmed that two of the people with the issue are on Windows 10, one with 1.2.10 and USB mic, one with 1.2.9 and audio jack mic. There were a few others affected in recent months, but I didn't get the details. I'll pay more attention when it inevitably happens in the future.

from mumble.

This should be a high priority bug as it impacts the core functionality of the software. I can say that mumble with screeching is not very usable.

from mumble.

+1

This is very annoying. Scares the crap out of me if noone has said anything in a while, and this kicks in...

from mumble.

We seriously need to work on reproduction steps here. "Me too"s don't help anyone; there's a subscribe button to the right if you just want to be notified of developments.

Important observations:

• The problem occurs for only a fraction of a second, at the very beginning of a transmission
• Disabling amplification seems to prevent the issue from occurring

The most obvious possibility is that the Opus encoder somehow makes Mumble's internal amplification "lag" in terms of setting an appropriate gain. If so, a few ms of near-silence followed by relatively loud voice, with just the right timing, may possibly be a reliable trigger.

A previous post mentioned that setting audio per packet to 10ms solved the issue for them. I observed that myself and another user not exhibiting the issue already had it set to 10ms. The problem user was set to 30 instead. We pulled him down to 10 as a test. Maybe the amplifier is only considering a fixed length of audio, instead of the whole packet? Then I wonder why it would only be Opus and not other codecs as well...

from mumble.

We would like it if people could try out the latest snapshot available on mumble.info.

Version 1.3.0883g2a31708~snapshot.

This snapshot includes the commit 23fa9b3, which we believe could be related to this issue.

Thank you.

from mumble.

For what it's worth no one has complained about me glitching their eardrums since this update. One of the other main culprits (who, now I think about it, has a really poor quality connection and I may have suggested he increase his audio-per-packet to compensate) hasn't been around much, but I'll encourage him to use the snapshots and see what happens

from mumble.

FWIW, the aforementioned commit has now been cherry-picked into 1.2.12, which was released just now.

We hope the bug was fixed by the commit, but we are still not sure.

Thanks.

from mumble.

So we just tried out the latest release.. soon I lowered my audio per packet to the lowest setting which is 20ms if I remember correctly my buddy screamed 'OUCH, it's not fixed'...

Server info
Version: 1.2.12
OS: Archlinux (freshly updated)

Edit: It was set to 10ms which is the lowest...

from mumble.

Yeah, I'm the one who screamed 'OUCH, it's not fixed'...
Dazoe was running client version 1.2.12
I was running client version 1.3.0895gfe81316~snapshot.
We both had APP set to 10 ms and Echo Cancellation enabled
Murmur had never been installed on this server and its configuration was basically still default. Nothing that didn't need to be changed was changed.

Server info
OS: Arch Linux x86_64
Hostname: eligos
Kernel Release: 4.2.5-1-ARCH
Uptime: 40 days, 2:16
WM: None
DE: None
Packages: 375
RAM: 2281 MB / 7988 MB
Processor Type: Intel(R) Xeon(R) CPU L5420 @ 2.50GHz
Root: 98G / 472G (20%) (ext4)
Murmur Version: 1.2.12

dxdiag for my system. "Probably of no help"
Red-DxDiag.txt

Edit: Just wanted to point out some of what has and hasn't changed since this ticket was opened.
Dazoe's hardware, network, OS, ISP and state he lives in has all changed.
My hardware, network and OS have completely changed.
The server has completely different hardware, is in a different datacenter in a different state, under a different company.

Things that are the same.
The server OS is Arch linux 64
I am on the same ISP which is a WISP "Wireless Internet Service Provider" but I am connected to a different tower that is closer. The tower AP is a RouterBOARD RB411AH http://routerboard.com/RB411AH . The only real detrimental effect of being on a WISP is sometimes higher jitter "ping deviation" which may exaggerate the issue. Granted, this may just be a red herring.

from mumble.

Interesting. The issue we fixed - afaik - shouldn't occur at 10ms pp latency. Maybe this is something else. Was really hoping the whole issue would be solved by the fix :(

from mumble.

With the most recent update the extremely loud screech has came back. When you released the update that was supposed to fix the issue it was completely gone. I haven't heard it for weeks until we all updated to the most recent update.

from mumble.

Setting APP to 10ms proved to be a functional workaround for us (and was what we wanted to use anyway) which makes me think there's two issues here, and you fixed one of them. But I haven't exactly looked at the code or anything.

from mumble.

People are saying I made those kinds of noises the other night. But I think the incidence of these has been drastically reduced, so I think it's highly likely a source of this has been fixed

from mumble.

Still not fixed, so far i've only heard it from users with "unstable internet", basicly when they get a lagspike you will hear this "horror scream bug"

from mumble.

@druni, I'm not sure if internet connection plays a role. My server and I
have very stable internet connections. I'm very picky when it comes to
packet loss and ping jitter. I almost always have 2 ping plotters running
watching my connection to the server as well as another server in a
different region.
Me and another guy (with the same ISP and I) are "causing" this we don't
really have anything in common except for using similar AMD processors
(FX-8xxx)
A third person with very poor connection (Rural Wireless) never "causes"
the problem.
So far we haven't been able to come up with a work around. changing the
audio per packet to 10ms seems to lower the frequency of the problem but it
doesn't stop it.
We have since moved away from mumble sadly. But every once in a while we
install a server and give it a go. In the years that mumble has had this
bug it seems to have gotten worse. I remember when it first started it only
happened something like 2 or 3 times a day. (We spend a lot time in VOIP
apps)
The last time we tried it the issue came up right away.

On Sun, Feb 7, 2016 at 1:32 PM, druni [email protected] wrote:

Still not fixed, so far i've only heard it from users with "unstable
internet", basicly when they get a lagspike you will hear this "horror
scream bug"

—
Reply to this email directly or view it on GitHub
#957 (comment).

-Dave

from mumble.

I still hear it

from mumble.

@DGMurdockIII Was this with people using new-ish clients only? The problem will be at the sender-side, so even if you have updated, everyone else will need to, too. (1.2.9 and 1.3.0 snapshots should have the fix for one of these issues, at least.)

from mumble.

My friend had sound artefacts on beginning to transmit audio relatively often. Since he upgraded/changed his PC they no longer occur. He had quite and old client for a long time (1.2.4), but IIRC I asked him to update some time ago - but I'm not sure if he did. Anyway, with the new PC, as he downloaded Mumble, he has a newer version and no more artefacts.

from mumble.

I'm not sure the issue is sender side. I have a custom mumble client/bot. It uses libopus1.2a to decode the packets, which are then piped into ffmpeg to encode them. Source. Server is running a modified ~1.5 year old 1.3 beta on CentOS7x64. Sending client was 1.3.0~19xx an Win10-1607x64 and I believe the receiving clients that heard the screech were the same version/OS. The bot is on the same machine as the server.

from mumble.

Actually I only use libopus1.2a my development environment. On the server it actually uses opus 1.0.2-6.el7.

from mumble.

I have more data, now in the form of opus data. You can find the opus here. The bad noises supposedly happen at about 43 seconds in, but as you can tell this is not recorded. The recorder does not modify the opus data in any way, however it does discard metadata, such as sequence number and position.

from mumble.

I wrote a quick client with gumble to replay the opus data. It didn't trigger the bug. This narrows the problem down to something with metadata or timing (probably). I doubt its a timing bug because it always (at least for my users) triggers for everybody at once, and I doubt a timing issue could be propagated so reliably over our varying connections.

from mumble.

@abextm My comment about it being sender-side was because we fixed a bug a while ago (and backported it to 1.2.x), which could cause a similar screech: 23fa9b3

So if some people on the server were on a version that does not contain the fix, those people could produce a noise that would be heard by everyone else on the server. (That is, one person with an old client could ruin it for everyone else, because the noise is produced in AudioInput.)

from mumble.

@abextm Thanks for working on this, BTW!

from mumble.

BTW, was the speaker in the recording using VAD, Continuous, or PTT?

from mumble.

Excluding the bot, everyone was on Mumble 1.3.0~2389~gdde8173, using Voice activity.

from mumble.

Nice if this gets fixed Voice activity might be actually OK to use then if set up right

from mumble.

About a month ago I enabled Opus again. We had screeches, but one of my visitors was using an old version of Mumble, I can't recount but I think it was around 1.2.10. I told him to upgrade to 1.2.19 and since then we haven't encountered the screeching bug.

Everyone is using Voice Activity and the (Murmur) server version is 1.2.19 on Alpine Linux. It's fixed for us, so thank you!

from mumble.

We have updated Opus to version 1.2.1.

Could you check if the issue still appears with the latest snapshot (1.3.02458g176c041~snapshot)?

from mumble.

I have deleted these two comments because they are failed delivery mails sent from Junk Email Filter, because avatias0's email address isn't available anymore and he is subscribed to this issue.

Every time somebody posts a comment (such as this one), a new one from avatias0 appears. Sadly there is nothing we can do.

Let's hope this issue is solved, otherwise I propose to move the discussion to a new issue.

Update: No new comments after mine, maybe the user disabled the service.

from mumble.

I'm assuming this is not an issue anymore.

from mumble.

Nope still happens, and I still can't repro it in any way

from mumble.

@abextm Does it still happen?

from mumble.

Very rarely, but it does still happen

from mumble.

Is there at least a user running an old version of Mumble?

from mumble.

Server is on a slightly modified f2cbebd

There are clients connected that use old versions of gumble, however these clients never have sent audio that causes the bug, and if they receive audio where other clients would hear the bug they are able to decode it without the bug. source
When the bug does occur all of the normal mumble clients hear it simultaneously

Here are some versions that have been connected to the server recently. The bug only happens about once a month anymore, so I can't provide any more specifics on which clients are connected.

• 1.3.0~2729~g2126495~snapshot
• 1.3.0~2586~g894ade2~snapshot

from mumble.

Is this still a thing?

from mumble.

This issue has been marked as stale, because our request for more information has thus far not been fulfilled.

If no further action occurs, this issue will be closed within 7 days.

from mumble.