SQLAlchemy related traceback about legion HOT 10 CLOSED

Hi,

Same problem scannig 150ips, on Kali 2020.4 fullupdated, either using Legion(kali version 0.3.6) or version from git 0.3.7. Even importing a scan done with legion fails.

(python3:84709): Gtk-WARNING **: 16:23:25.008: Attempting to store changes into `/root/.local/share/recently-used.xbel', but failed: Failed to create file “/root/.local/share/recently-used.xbel.0FHOQ0”: Too many open files
Traceback (most recent call last):
File "/usr/lib/python3/dist-packages/sqlalchemy/engine/base.py", line 2339, in _wrap_pool_connect
File "/usr/lib/python3/dist-packages/sqlalchemy/pool/base.py", line 364, in connect
File "/usr/lib/python3/dist-packages/sqlalchemy/pool/base.py", line 778, in _checkout
File "/usr/lib/python3/dist-packages/sqlalchemy/pool/base.py", line 495, in checkout
File "/usr/lib/python3/dist-packages/sqlalchemy/pool/impl.py", line 239, in _do_get
File "/usr/lib/python3/dist-packages/sqlalchemy/pool/base.py", line 309, in _create_connection
File "/usr/lib/python3/dist-packages/sqlalchemy/pool/base.py", line 440, in init
File "/usr/lib/python3/dist-packages/sqlalchemy/pool/base.py", line 661, in _connect
File "/usr/lib/python3/dist-packages/sqlalchemy/util/langhelpers.py", line 68, in exit
File "/usr/lib/python3/dist-packages/sqlalchemy/util/compat.py", line 178, in raise
File "/usr/lib/python3/dist-packages/sqlalchemy/pool/base.py", line 656, in __connect
File "/usr/lib/python3/dist-packages/sqlalchemy/engine/strategies.py", line 114, in connect
File "/usr/lib/python3/dist-packages/sqlalchemy/engine/default.py", line 493, in connect
sqlite3.OperationalError: unable to open database file

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
File "/opt/legion/ui/view.py", line 1321, in updateProcessesTableView
File "/opt/legion/controller/controller.py", line 579, in getProcessesFromDB
File "/opt/legion/db/repositories/ProcessRepository.py", line 51, in getProcesses
File "/usr/lib/python3/dist-packages/sqlalchemy/engine/base.py", line 2237, in execute
File "/usr/lib/python3/dist-packages/sqlalchemy/engine/base.py", line 2305, in _contextual_connect
File "/usr/lib/python3/dist-packages/sqlalchemy/engine/base.py", line 2342, in _wrap_pool_connect
File "/usr/lib/python3/dist-packages/sqlalchemy/engine/base.py", line 1584, in handle_dbapi_exception_noconnection
File "/usr/lib/python3/dist-packages/sqlalchemy/util/compat.py", line 178, in raise
File "/usr/lib/python3/dist-packages/sqlalchemy/engine/base.py", line 2339, in _wrap_pool_connect
File "/usr/lib/python3/dist-packages/sqlalchemy/pool/base.py", line 364, in connect
File "/usr/lib/python3/dist-packages/sqlalchemy/pool/base.py", line 778, in _checkout
File "/usr/lib/python3/dist-packages/sqlalchemy/pool/base.py", line 495, in checkout
File "/usr/lib/python3/dist-packages/sqlalchemy/pool/impl.py", line 239, in _do_get
File "/usr/lib/python3/dist-packages/sqlalchemy/pool/base.py", line 309, in _create_connection
File "/usr/lib/python3/dist-packages/sqlalchemy/pool/base.py", line 440, in init
File "/usr/lib/python3/dist-packages/sqlalchemy/pool/base.py", line 661, in _connect
File "/usr/lib/python3/dist-packages/sqlalchemy/util/langhelpers.py", line 68, in exit
File "/usr/lib/python3/dist-packages/sqlalchemy/util/compat.py", line 178, in raise
File "/usr/lib/python3/dist-packages/sqlalchemy/pool/base.py", line 656, in __connect
File "/usr/lib/python3/dist-packages/sqlalchemy/engine/strategies.py", line 114, in connect
File "/usr/lib/python3/dist-packages/sqlalchemy/engine/default.py", line 493, in connect
sqlalchemy.exc.OperationalError: (sqlite3.OperationalError) unable to open database file
(Background on this error at: http://sqlalche.me/e/13/e3q8)
./startLegion.sh: line 38: 84709 Aborted /usr/bin/env python3 legion.py

from legion.

@re8el Can you check your limits? That "Too many open files" is a strong indication the ulimits are being reached for the user.

from legion.

Hi @sscottgvit ,

how can i check that?

from legion.

Hi @sscottgvit

more details...

opt/legion master !36 > pwd root@k20 18:19:03
/opt/legion
/opt/legion master !36 > ulimit -i root@k20 18:19:05
39624
/opt/legion master !36 > ulimit -n root@k20 18:19:40
1024
/opt/legion master !36 > ulimit -u root@k20 18:20:31
39624
/opt/legion master !36 > ./startLegion.sh root@k20 18:20:39
Strap yourself in, we're starting Legion...
Python 3 bin is python (/usr/bin/python)
Pip 3 bin is pip ()
Detected Kali ?

| | | / | / _ | \ | |
| | | || | _ | | | | | | |
| || |_| || || | || | |\ |
||_|_/|_| _|

{"time": "2020-09-09 18:20:53,495", "name": "Creating temporary project at application start...", "level": "INFO", "data": {"logger_name": "legion-startup"}, "context": {"module": "legion", "filename": "legion.py", "line": 106}}
{"time": "2020-09-09 18:20:53,528", "name": "Wordlist was created/opened: ./tmp/legion-ga8vv450-tool-output/legion-usernames.txt", "level": "INFO", "data": {"logger_name": "legion"}, "context": {"module": "auxiliary", "filename": "auxiliary.py", "line": 115}}
{"time": "2020-09-09 18:20:53,529", "name": "Wordlist was created/opened: ./tmp/legion-ga8vv450-tool-output/legion-passwords.txt", "level": "INFO", "data": {"logger_name": "legion"}, "context": {"module": "auxiliary", "filename": "auxiliary.py",

"INFO", "data": {"logger_name": "legion"}, "context": {"module": "auxiliary", "filename": "auxiliary.py", "line": 115}}
{"time": "2020-09-09 18:21:05,821", "name": "Failed to sort", "level": "ERROR", "data": {"logger_name": "legion"}, "context": {"module": "processmodels", "filename": "processmodels.py", "line": 136}}
0
0
0
0
Traceback (most recent call last):
File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 2338, in _wrap_pool_connect
File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/pool/base.py", line 364, in connect
File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/pool/base.py", line 778, in _checkout
File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/pool/base.py", line 495, in checkout
File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/pool/impl.py", line 239, in _do_get
File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/pool/base.py", line 309, in _create_connection
File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/pool/base.py", line 440, in init
File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/pool/base.py", line 661, in _connect
File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/langhelpers.py", line 68, in exit
File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/compat.py", line 182, in raise
File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/pool/base.py", line 656, in __connect
File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/strategies.py", line 114, in connect
File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/default.py", line 493, in connect
sqlite3.OperationalError: unable to open database file

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
File "/opt/legion/ui/view.py", line 1321, in updateProcessesTableView
File "/opt/legion/controller/controller.py", line 579, in getProcessesFromDB
File "/opt/legion/db/repositories/ProcessRepository.py", line 51, in getProcesses
File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 2236, in execute
File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 2304, in _contextual_connect
File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 2341, in _wrap_pool_connect
File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1583, in handle_dbapi_exception_noconnection
File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/compat.py", line 182, in raise
File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 2338, in _wrap_pool_connect
File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/pool/base.py", line 364, in connect
File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/pool/base.py", line 778, in _checkout
File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/pool/base.py", line 495, in checkout
File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/pool/impl.py", line 239, in _do_get
File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/pool/base.py", line 309, in _create_connection
File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/pool/base.py", line 440, in init
File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/pool/base.py", line 661, in _connect
File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/langhelpers.py", line 68, in exit
File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/compat.py", line 182, in raise
File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/pool/base.py", line 656, in __connect
File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/strategies.py", line 114, in connect
File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/default.py", line 493, in connect
sqlalchemy.exc.OperationalError: (sqlite3.OperationalError) unable to open database file
(Background on this error at: http://sqlalche.me/e/13/e3q8)
./startLegion.sh: line 38: 85474 Aborted /usr/bin/env python3 legion.py

from legion.

Another run...

@sscottgvit

/opt/legion master !36 > ./startLegion.sh root@k20 18:35:10
Strap yourself in, we're starting Legion...
Python 3 bin is python (/usr/bin/python)
Pip 3 bin is pip ()
Detected Kali ?

| | | / | / _ | \ | |
| | | || | _ | | | | | | |
| || |_| || || | || | |\ |
||_|_/|_| _|

{"time": "2020-09-09 18:36:15,131", "name": "Creating temporary project at application start...", "level": "INFO", "data": {"logger_name": "legion-startup"}, "context": {"module": "legion", "filename": "legion.py", "line": 106}}
{"time": "2020-09-09 18:36:15,159", "name": "Wordlist was created/opened: ./tmp/legion-_ajwrxra-tool-output/legion-usernames.txt", "level": "INFO", "data": {"logger_name": "legion"}, "context": {"module": "auxiliary", "filename": "auxiliary.py", "line": 115}}
{"time": "2020-09-09 18:36:15,160", "name": "Wordlist was created/opened: ./tmp/legion-_ajwrxra-tool-output/legion-passwords.txt", "level": "INFO", "data": {"logger_name": "legion"}, "context": {"module": "auxiliary", "filename": "auxiliary.py", "line": 115}}
{"time": "2020-09-09 18:36:15,328", "name": "Loading settings file..", "level": "INFO", "data": {"logger_name": "legion"}, "context": {"module": "settings", "filename": "settings.py", "line": 35}}
legion.py:120: DeprecationWarning: an integer is required (got type float). Implicit conversion to integers using int is deprecated, and may be removed in a future version of Python.
MainWindow.move(x - MainWindow.geometry().width() / 2, y - MainWindow.geometry().height() / 2)
{"time": "2020-09-09 18:36:15,456", "name": "Legion started successfully.", "level": "INFO", "data": {"logger_name": "legion-startup"}, "context": {"module": "legion", "filename": "legion.py", "line": 125}}
{"time": "2020-09-09 18:36:27,096", "name": "Settings have NOT been changed.", "level": "INFO", "data": {"logger_name": "legion"}, "context": {"module": "controller", "filename": "controller.py", "line": 144}}
{"time": "2020-09-09 18:36:28,031", "name": "Closing project /opt/legion/tmp/legion-8xtejg39.legion...", "level": "INFO", "data": {"logger_name": "legion"}, "context": {"module": "ProjectManager", "filename": "ProjectManager.py", "line": 81}}
{"time": "2020-09-09 18:36:28,031", "name": "Removing temporary files and folders...", "level": "INFO", "data": {"logger_name": "legion"}, "context": {"module": "ProjectManager", "filename": "ProjectManager.py", "line": 91}}
{"time": "2020-09-09 18:36:28,031", "name": "Removing running folder at close...", "level": "INFO", "data": {"logger_name": "legion"}, "context": {"module": "ProjectManager", "filename": "ProjectManager.py", "line": 95}}
{"time": "2020-09-09 18:36:28,036", "name": "Opening existing project: ...2.legion...", "level": "INFO", "data": {"logger_name": "legion"}, "context": {"module": "ProjectManager", "filename": "ProjectManager.py", "line": 65}}
{"time": "2020-09-09 18:36:28,042", "name": "Wordlist was created/opened: .../legion-usernames.txt", "level": "INFO", "data": {"logger_name": "legion"}, "context": {"module": "auxiliary", "filename": "auxiliary.py", "line": 115}}
{"time": "2020-09-09 18:36:28,042", "name": "Wordlist was created/opened: .../legion-passwords.txt", "level": "INFO", "data": {"logger_name": "legion"}, "context": {"module": "auxiliary", "filename": "auxiliary.py", "line": 115}}
{"time": "2020-09-09 18:36:28,102", "name": "Failed to sort", "level": "ERROR", "data": {"logger_name": "legion"}, "context": {"module": "processmodels", "filename": "processmodels.py", "line": 136}}
0
0
0
0

(python3:86149): Gtk-WARNING **: 18:36:29.341: Attempting to store changes into '/root/.local/share/recently-used.xbel', but failed: Failed to create file “/root/.local/share/recently-used.xbel.PE5SQ0”: Too many open files
Traceback (most recent call last):
File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 2338, in _wrap_pool_connect
File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/pool/base.py", line 364, in connect
File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/pool/base.py", line 778, in _checkout
File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/pool/base.py", line 495, in checkout
File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/pool/impl.py", line 239, in _do_get
File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/pool/base.py", line 309, in _create_connection
File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/pool/base.py", line 440, in init
File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/pool/base.py", line 661, in _connect
File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/langhelpers.py", line 68, in exit
File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/compat.py", line 182, in raise
File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/pool/base.py", line 656, in __connect
File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/strategies.py", line 114, in connect
File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/default.py", line 493, in connect
sqlite3.OperationalError: unable to open database file

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
File "/opt/legion/ui/view.py", line 1321, in updateProcessesTableView
File "/opt/legion/controller/controller.py", line 579, in getProcessesFromDB
File "/opt/legion/db/repositories/ProcessRepository.py", line 51, in getProcesses
File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 2236, in execute
File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 2304, in _contextual_connect
File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 2341, in _wrap_pool_connect
File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1583, in handle_dbapi_exception_noconnection
File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/compat.py", line 182, in raise
File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 2338, in _wrap_pool_connect
File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/pool/base.py", line 364, in connect
File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/pool/base.py", line 778, in _checkout
File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/pool/base.py", line 495, in checkout
File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/pool/impl.py", line 239, in _do_get
File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/pool/base.py", line 309, in _create_connection
File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/pool/base.py", line 440, in init
File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/pool/base.py", line 661, in _connect
File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/langhelpers.py", line 68, in exit
File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/compat.py", line 182, in raise
File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/pool/base.py", line 656, in __connect
File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/strategies.py", line 114, in connect
File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/default.py", line 493, in connect
sqlalchemy.exc.OperationalError: (sqlite3.OperationalError) unable to open database file
(Background on this error at: http://sqlalche.me/e/13/e3q8)
./startLegion.sh: line 38: 86149 Aborted /usr/bin/env python3 legion.py
/opt/legion master !36 >

from legion.

@re8el I'm seeing 'Too many open files'. I don't recognize your console line. What shell are you running? What user are you running as?

Also can you run each of the following and let me know the output?:
cat /proc/sys/fs/file-max | awk ' { print $1 } '
cat /proc/sys/fs/file-nr | awk ' { print $1 } '
lsof | wc -l
ulimit -Sn
ulimit -Hn

from legion.

Hi @sscottgvit

Im using costum ZSH with root on Kali 2020.4 fullupdated

~ > cat /proc/sys/fs/file-max | awk ' { print $1 } ' root@k20 19:13:24
9223372036854775807
~ > cat /proc/sys/fs/file-nr | awk ' { print $1 } ' root@k20 19:13:31
6272
~ > lsof | wc -l root@k20 19:13:38
43471
~ > ulimit -Sn root@k20 19:13:45
1024
~ > root@k20 19:13:56
~ > ulimit -Hn INT root@k20 19:13:56
1048576
~ >

from legion.

@re8el You're definitely hitting limits. Try updating your user limits to something a bit higher. https://tecadmin.net/increase-open-files-limit-ubuntu/

from legion.

@sscottgvit ulimits increased problem solved! thks!

from legion.

@re8el No problem. Have a good one.

from legion.