Comments (4)
Successfully reproduced on a devel machine with a single, default MariaDB instance and 50 Peekaboos pounding in on it with the same sample:
2020-07-22 10:57:10,727 - peekaboo.server - (Request-23) - DEBUG - New connection incoming.
2020-07-22 10:57:10,761 - peekaboo.server - (Request-23) - INFO - Got run_analysis request for /home/michael/PeekabooAV/foo.py
2020-07-22 10:57:10,761 - peekaboo.sample - (Request-23) - DEBUG - meta_info_request = scan-file
2020-07-22 10:57:10,761 - peekaboo.sample - (Request-23) - DEBUG - meta_info_full_name = /home/michael/PeekabooAV/foo.py
2020-07-22 10:57:10,764 - sqlalchemy.engine.base.Engine - (Request-23) - INFO - BEGIN (implicit)
2020-07-22 10:57:10,792 - sqlalchemy.engine.base.Engine - (Request-23) - INFO - INSERT INTO in_flight_samples_v7 (sha256sum, instance_id, start_tim
e) VALUES (%s, %s, %s)
2020-07-22 10:57:10,794 - sqlalchemy.engine.base.Engine - (Request-23) - INFO - ('fedb5b37f2f7d921c58119c82be428d134c34ba606727002d44bbb0d1ddfbc05'
, 10, datetime.datetime(2020, 7, 22, 10, 57, 10, 763630))
2020-07-22 10:57:10,820 - sqlalchemy.engine.base.Engine - (Request-23) - INFO - ROLLBACK
2020-07-22 10:57:10,822 - peekaboo.queuing - (Request-23) - ERROR - Unable to mark sample as in flight: (MySQLdb._exceptions.OperationalError) (121
3, 'Deadlock found when trying to get lock; try restarting transaction')
[SQL: INSERT INTO in_flight_samples_v7 (sha256sum, instance_id, start_time) VALUES (%s, %s, %s)]
[parameters: ('fedb5b37f2f7d921c58119c82be428d134c34ba606727002d44bbb0d1ddfbc05', 10, datetime.datetime(2020, 7, 22, 10, 57, 10, 763630))]
(Background on this error at: http://sqlalche.me/e/e3q8
So, apparently, two INSERTs can deadlock with each other, which is exactly what we thought to avoid by using an "atomic" operation such as an INSERT.
Resources on Deadlock detection:
- https://dev.mysql.com/doc/refman/5.7/en/innodb-deadlock-detection.html
- https://dev.mysql.com/doc/refman/5.6/en/innodb-standard-monitor.html
InnoDB diagnostic output:
MariaDB [peekaboo]> SHOW ENGINE INNODB STATUS;
------------------------
LATEST DETECTED DEADLOCK
------------------------
2020-07-22 10:57:10 7fb758b04700
*** (1) TRANSACTION:
TRANSACTION 15432303, ACTIVE 0 sec inserting
mysql tables in use 1, locked 1
LOCK WAIT 3 lock struct(s), heap size 1184, 2 row lock(s)
MySQL thread id 235, OS thread handle 0x7fb758942700, query id 4776 localhost peekaboo update
INSERT INTO in_flight_samples_v7 (sha256sum, instance_id, start_time) VALUES ('fedb5b37f2f7d921c58119c82be428d134c34ba606727002d44bbb0d1ddfbc05', 8
, '2020-07-22 10:57:10.764760')
*** (1) WAITING FOR THIS LOCK TO BE GRANTED:
RECORD LOCKS space id 593 page no 3 n bits 72 index `PRIMARY` of table `peekaboo`.`in_flight_samples_v7` trx table locks 1 total table locks 2 trx
id 15432303 lock_mode X locks rec but not gap waiting lock hold time 0 wait time before grant 0
*** (2) TRANSACTION:
TRANSACTION 15432304, ACTIVE 0 sec inserting
mysql tables in use 1, locked 1
3 lock struct(s), heap size 1184, 2 row lock(s)
MySQL thread id 229, OS thread handle 0x7fb758b04700, query id 4777 localhost peekaboo update
INSERT INTO in_flight_samples_v7 (sha256sum, instance_id, start_time) VALUES ('fedb5b37f2f7d921c58119c82be428d134c34ba606727002d44bbb0d1ddfbc05', 1
0, '2020-07-22 10:57:10.763630')
*** (2) HOLDS THE LOCK(S):
RECORD LOCKS space id 593 page no 3 n bits 72 index `PRIMARY` of table `peekaboo`.`in_flight_samples_v7` trx table locks 1 total table locks 2 trx
id 15432304 lock mode S locks rec but not gap lock hold time 0 wait time before grant 0
*** (2) WAITING FOR THIS LOCK TO BE GRANTED:
RECORD LOCKS space id 593 page no 3 n bits 72 index `PRIMARY` of table `peekaboo`.`in_flight_samples_v7` trx table locks 1 total table locks 2 trx
id 15432304 lock_mode X locks rec but not gap waiting lock hold time 0 wait time before grant 0
*** WE ROLL BACK TRANSACTION (2)
My reading of this is that both are waiting for an exclusive lock on the primary key but neither is holding it. In my world one of them should get the lock and the other should continue waiting for it. After the first is finished inserting, the other should get the lock and fail inserting because the entry is already there. Isn't that the whole idea of transactions?
from peekabooav.
https://bugs.mysql.com/bug.php?id=39657 leads to https://bugs.mysql.com/bug.php?id=21356 which seems to be what we have here: T2 acquired an S lock and is waiting for that to upgrade it to an X lock. So it's deadlocking with itself. :/
from peekabooav.
.. or rather T2 acquired S, T1 wants X and waits for T2 to free S and T2 wants X and waits for T1 to get and free X -> deadlock. :/
from peekabooav.
https://stackoverflow.com/questions/41015813/avoiding-mysql-deadlock-when-upgrading-shared-to-exclusive-lock/41174997,
https://docs.oracle.com/cd/E17952_01/mysql-5.7-en/innodb-locks-set.html: Apparently INSERT is riddled with ways it can deadlock. So we go the retry route.
from peekabooav.
Related Issues (20)
- Server hits recv limit HOT 1
- Check file extension extraction for consistency / usefulness in conjunction with cuckoo being sensitive to spaces in filenames HOT 1
- Extract IOC out of Cuckoo report
- Another peepdf traceback in cuckoo HOT 1
- Consider raising minimum version of dependency on python magic to 0.4.17
- Reconsider database transaction locking
- Reconsider aggressive database connection pool recycling
- Validation error with dummy filereport and expression referencing type_as_text
- Add retries for additional database connectivity scenarios
- Handle URI-parameters-like notation in declared filenames
- Security vulnerability: Regex matching in ruleset HOT 3
- python3.10 incompatibility of colorclass affecting us via oletools HOT 4
- expressions cannot express empty set
- Support TLS on REST API
- PID file can contain our own pid and confuse us
- pyparsing 3 compatibility HOT 3
- urllib3.util.retry.Retry DeprecationWarning for 'method_whitelist'
- Early shutdown not working due to switch to asyncio signal handler HOT 1
- Dynamically learn available Cortex analysers and their versions
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from peekabooav.