Git Product home page Git Product logo

Comments (4)

michaelweiser avatar michaelweiser commented on August 22, 2024

Successfully reproduced on a devel machine with a single, default MariaDB instance and 50 Peekaboos pounding in on it with the same sample:

2020-07-22 10:57:10,727 - peekaboo.server - (Request-23) - DEBUG - New connection incoming.
2020-07-22 10:57:10,761 - peekaboo.server - (Request-23) - INFO - Got run_analysis request for /home/michael/PeekabooAV/foo.py
2020-07-22 10:57:10,761 - peekaboo.sample - (Request-23) - DEBUG - meta_info_request = scan-file
2020-07-22 10:57:10,761 - peekaboo.sample - (Request-23) - DEBUG - meta_info_full_name = /home/michael/PeekabooAV/foo.py
2020-07-22 10:57:10,764 - sqlalchemy.engine.base.Engine - (Request-23) - INFO - BEGIN (implicit)
2020-07-22 10:57:10,792 - sqlalchemy.engine.base.Engine - (Request-23) - INFO - INSERT INTO in_flight_samples_v7 (sha256sum, instance_id, start_tim
e) VALUES (%s, %s, %s)
2020-07-22 10:57:10,794 - sqlalchemy.engine.base.Engine - (Request-23) - INFO - ('fedb5b37f2f7d921c58119c82be428d134c34ba606727002d44bbb0d1ddfbc05'
, 10, datetime.datetime(2020, 7, 22, 10, 57, 10, 763630))
2020-07-22 10:57:10,820 - sqlalchemy.engine.base.Engine - (Request-23) - INFO - ROLLBACK
2020-07-22 10:57:10,822 - peekaboo.queuing - (Request-23) - ERROR - Unable to mark sample as in flight: (MySQLdb._exceptions.OperationalError) (121
3, 'Deadlock found when trying to get lock; try restarting transaction')
[SQL: INSERT INTO in_flight_samples_v7 (sha256sum, instance_id, start_time) VALUES (%s, %s, %s)]
[parameters: ('fedb5b37f2f7d921c58119c82be428d134c34ba606727002d44bbb0d1ddfbc05', 10, datetime.datetime(2020, 7, 22, 10, 57, 10, 763630))]
(Background on this error at: http://sqlalche.me/e/e3q8

So, apparently, two INSERTs can deadlock with each other, which is exactly what we thought to avoid by using an "atomic" operation such as an INSERT.
Resources on Deadlock detection:

InnoDB diagnostic output:

MariaDB [peekaboo]> SHOW ENGINE INNODB STATUS;
------------------------
LATEST DETECTED DEADLOCK
------------------------
2020-07-22 10:57:10 7fb758b04700
*** (1) TRANSACTION:
TRANSACTION 15432303, ACTIVE 0 sec inserting
mysql tables in use 1, locked 1
LOCK WAIT 3 lock struct(s), heap size 1184, 2 row lock(s)
MySQL thread id 235, OS thread handle 0x7fb758942700, query id 4776 localhost peekaboo update
INSERT INTO in_flight_samples_v7 (sha256sum, instance_id, start_time) VALUES ('fedb5b37f2f7d921c58119c82be428d134c34ba606727002d44bbb0d1ddfbc05', 8
, '2020-07-22 10:57:10.764760')
*** (1) WAITING FOR THIS LOCK TO BE GRANTED:
RECORD LOCKS space id 593 page no 3 n bits 72 index `PRIMARY` of table `peekaboo`.`in_flight_samples_v7` trx table locks 1 total table locks 2  trx
 id 15432303 lock_mode X locks rec but not gap waiting lock hold time 0 wait time before grant 0
*** (2) TRANSACTION:
TRANSACTION 15432304, ACTIVE 0 sec inserting
mysql tables in use 1, locked 1
3 lock struct(s), heap size 1184, 2 row lock(s)
MySQL thread id 229, OS thread handle 0x7fb758b04700, query id 4777 localhost peekaboo update
INSERT INTO in_flight_samples_v7 (sha256sum, instance_id, start_time) VALUES ('fedb5b37f2f7d921c58119c82be428d134c34ba606727002d44bbb0d1ddfbc05', 1
0, '2020-07-22 10:57:10.763630')
*** (2) HOLDS THE LOCK(S):
RECORD LOCKS space id 593 page no 3 n bits 72 index `PRIMARY` of table `peekaboo`.`in_flight_samples_v7` trx table locks 1 total table locks 2  trx
 id 15432304 lock mode S locks rec but not gap lock hold time 0 wait time before grant 0
*** (2) WAITING FOR THIS LOCK TO BE GRANTED:
RECORD LOCKS space id 593 page no 3 n bits 72 index `PRIMARY` of table `peekaboo`.`in_flight_samples_v7` trx table locks 1 total table locks 2  trx
 id 15432304 lock_mode X locks rec but not gap waiting lock hold time 0 wait time before grant 0
*** WE ROLL BACK TRANSACTION (2)

My reading of this is that both are waiting for an exclusive lock on the primary key but neither is holding it. In my world one of them should get the lock and the other should continue waiting for it. After the first is finished inserting, the other should get the lock and fail inserting because the entry is already there. Isn't that the whole idea of transactions?

from peekabooav.

michaelweiser avatar michaelweiser commented on August 22, 2024

https://bugs.mysql.com/bug.php?id=39657 leads to https://bugs.mysql.com/bug.php?id=21356 which seems to be what we have here: T2 acquired an S lock and is waiting for that to upgrade it to an X lock. So it's deadlocking with itself. :/

from peekabooav.

michaelweiser avatar michaelweiser commented on August 22, 2024

.. or rather T2 acquired S, T1 wants X and waits for T2 to free S and T2 wants X and waits for T1 to get and free X -> deadlock. :/

from peekabooav.

michaelweiser avatar michaelweiser commented on August 22, 2024

https://stackoverflow.com/questions/41015813/avoiding-mysql-deadlock-when-upgrading-shared-to-exclusive-lock/41174997,
https://docs.oracle.com/cd/E17952_01/mysql-5.7-en/innodb-locks-set.html: Apparently INSERT is riddled with ways it can deadlock. So we go the retry route.

from peekabooav.

Related Issues (20)

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.