worker is broken about tooltool HOT 7 CLOSED

As best I can tell, this started with the deploy on November 11th, and the only code change that I see is da1e386,
which doesn't look remotely related.

I also looked in the cloudops repo, and the only notable thing was a change on Nov 7th that added some config variables (https://github.com/mozilla-services/cloudops-infra/commit/3ee99e07ffd0dfcb183cab4f813c18ea4ecbd40d#diff-47e2aa3945d7ca442e2be075b2d60730, for those that have access). However, there was a deploy on Nov 9th that didn't have this issue, so that seems unlikely to be related.

@oremj - is it possible to tell me if any secrets or other config not in cloudops-infra changed between the deploys on Nov 9th and Nov 11th?

My other theory, which seems less likely now, is that it was caused by one of the packages we took when tooltool migrated out of services. Most notably:

• aioamqp 0.12.0 -> 0.14.0
• amqp 2.4.2 -> 2.5.2

...however, I would've expected to start seeing errors earlier than November 11th if this was the cause.

from tooltool.

There was a tooltool secret change on November 11th via a request from @garbas on slack.

from tooltool.

This surfaces on the rabbitmq side in logs like

=ERROR REPORT==== 11-Dec-2019::22:03:25 ===
closing AMQP connection <0.4867.4015> (35.233.150.32:40258 -> 10.51.2.39:5671):
{handshake_error,starting,0,
                 {amqp_error,access_refused,
                             "PLAIN login refused: user 'releng-tooltool-staging' - invalid credentials",
                             'connection.start_ok'}}

from tooltool.

This surfaces on the rabbitmq side in logs like

=ERROR REPORT==== 11-Dec-2019::22:03:25 ===
closing AMQP connection <0.4867.4015> (35.233.150.32:40258 -> 10.51.2.39:5671):
{handshake_error,starting,0,
                 {amqp_error,access_refused,
                             "PLAIN login refused: user 'releng-tooltool-staging' - invalid credentials",
                             'connection.start_ok'}}

Thank you, that's very helpful! So...I guess we need to find the right credentials then, heh.

from tooltool.

We fixed the credentials issue but we're still seeing the original traceback on the client. @sciurus has confirmed that we're no longer seeing credential issues on the server, and I do see successful logins on the client as well. So, there's a secondary issue to look into on the client, too.

from tooltool.

Sentry looks quiet too \o/

from tooltool.

And now that I'm looking a few hours later, there's no more errors on the client. I'm guessing the ones that I saw shortly after the deployment were the old pod before it shut down. Calling this fixed. Thanks folks!

from tooltool.