monetdbsolutions / monetdb-stethoscope Goto Github PK

Update the read me file and make sure the description is correct.

When a derived field is mentioned in the -i list, then it should also be interpreted as the activation of the corresponding transformer.
For example,
pystethoscope -i pc, stmt sf10
is equivalent to
pystethoscope -i pc, stmt sf10 -t statement

It should perform resource cleanup (output non empty buffers, close files, etc) and gracefully shut stethoscope down.

Clearly separate the use from development by the addition of the simple example.
Warn for possible --upgrade and --user issues

Connecting the stethoscope to a running database to monitor query process illustrates a lot of parsing errors.

[mk@baksteen14 repository]$ mserver5 --version
MonetDB 5 server 11.41.0 (hg id: f7bea632f54b) (64-bit, 128-bit integers)
This is an unreleased version
Copyright (c) 1993 - July 2008 CWI
Copyright (c) August 2008 - 2021 MonetDB B.V., all rights reserved
Visit https://www.monetdb.org/ for further information
Found 251.9GiB available memory, 32 available cpu cores
Libraries:
libpcre: 8.44 2020-02-12
openssl: OpenSSL 1.1.1k FIPS 25 Mar 2021
libxml2: 2.9.10
Compiled by: mk@baksteen14 (x86_64-pc-linux-gnu)
Compilation: /usr/bin/cc -Werror -Wall -Wextra -Werror-implicit-function-declaration -Wpointer-arith -Wundef -Wformat=2 -Wformat-overflow=1 -Wno-format-truncation -Wno-format-nonliteral -Wno-cast-function-type -Winit-self -Winvalid-pch -Wmissing-declarations -Wmissing-format-attribute -Wmissing-prototypes -Wno-missing-field-initializers -Wold-style-definition -Wpacked -Wunknown-pragmas -Wvariadic-macros -Wstack-protector -fstack-protector-all -Wpacked-bitfield-compat -Wsync-nand -Wjump-misses-init -Wmissing-include-dirs -Wlogical-op -Wduplicated-cond -Wduplicated-branches -Wrestrict -Wnested-externs -Wmissing-noreturn -Wuninitialized -Wno-char-subscripts -Wunreachable-code -DNDEBUG=1
Linking : /usr/bin/ld

stethoscope -d demo -i clk state usec stmt -t statement
The workload is SN queries.

Example:
2021-06-28 21:42:52,287 - monetdb_stethoscope.parsing - ERROR - Parsing failed for “arg”:181,“var”:“X_370”,“type”:“str”,“const”:1,“value”:“\”total_estimate\“”,“eol”:461},{“arg”:182,“var”:“X_372”,“type”:“str”,“const”:1,“value”:“\”u_defect\“”,“eol”:461},{“arg”:183,“var”:“X_373”,“type”:“str”,“const”:1,“value”:“\”u_manual_test_count\“”,“eol”:461},{“arg”:184,“var”:“X_374”,“type”:“str”,“const”:1,“value”:“\”calendar_duration\“”,“eol”:461},{“arg”:185,“var”:“X_375”,“type”:“str”,“const”:1,“value”:“\”end_date_derived_from\“”,“eol”:461},{“arg”:186,“var”:“X_376”,“type”:“str”,“const”:1,“value”:“\”close_notes\“”,“eol”:461},{“arg”:187,“var”:“X_377”,“type”:“str”,“const”:1,“value”:“\”contact_type\“”,“eol”:461},{“arg”:188,“var”:“X_379”,“type”:“str”,“const”:1,“value”:“\”target_merge\“”,“eol”:461},{“arg”:189,“var”:“X_381”,“type”:“str”,“const”:1,“value”:“\”activity_due\“”,“eol”:461},{“arg”:190,“var”:“X_383”,“type”:“str”,“const”:1,“value”:“\”cost\“”,“eol”:461},{“arg”:191,“var”:“X_385”,“type”:“str”,“const”:1,“value”:“\”comments\“”,“eol”:461},{“arg”:192,“var”:“X_386”,“type”:“str”,“const”:1,“value”:“\”u_health\“”,“eol”:461},{“arg”:193,“var”:“X_387”,“type”:“str”,“const”:1,“value”:“\”due_date\“”,“eol”:461},{“arg”:194,“var”:“X_388”,“type”:“str”,“const”:1,“value”:“\”sys_mod_count\“”,“eol”:461},{“arg”:195,“var”:“X_389”,“type”:“str”,“const”:1,“value”:“\”x_snc_sdlc_process_docs\“”,“eol”:461},{“arg”:196,“var”:“X_390”,“type”:“str”,“const”:1,“value”:“\”time_zone\“”,“eol”:461},{“arg”:197,“var”:“X_392”,“type”:“str”,“const”:1,“value”:“\”opex_cost\“”,“eol”:461},{“arg”:198,“var”:“X_393”,“type”:“str”,“const”:1,“value”:“\”u_target_version\“”,“eol”:461},{“arg”:199,“var”:“X_394”,“type”:“str”,“const”:1,“value”:“\”milestone\“”,“eol”:461},{“arg”:200,“var”:“X_395”,“type”:“str”,“const”:1,“value”:“\”x_snc_sdlc_survey_needs_toi\“”,“eol”:461},{“arg”:201,“var”:“X_397”,“type”:“str”,“const”:1,“value”:“\”u_scrum_release\“”,“eol”:461},{“arg”:202,“var”:“X_398”,“type”:“str”,“const”:1,“value”:“\”u_hi_sysid\“”,“eol”:461},{“arg”:203,“var”:“X_399”,“type”:“str”,“const”:1,“value”:“\”u_target_merge_slot\“”,“eol”:461}]} (Extra data: line 1 column 6 (char 5))

After the first query is rendered the prerequisite transformer fails to render. The internal symbol table needs to be emptied at every event with pc=0 and state="start".

A missing feature with respect to the previous implementation of stethoscope is the gathering of heartbeats.

Martin reported this on Slack:

the statement transformer does not properly generate the MAL statement. example:
querylog.define(X_1=0@0:void,"select 1;":str,"default_pipe':str,6:int);
should be rendered as
X_1=0@0:void := querylog.define("select 1;":str,"default_pipe":str,6:int);

The -i/-e fields should check for validity of the field names upon start, or the error message should be issued only once

poetry run pystethoscope --formatter=line -i pc,usec,state,stmt sf10

Connected to the database: sf10
Key stmt not found in the JSON object
[0, start, 0]
Key stmt not found in the JSON object
[1, start, 0]

The --transformer statement requires you to also speficy -i stmt
This should be automatic and not confuse the user.

(mk) mk@toke::~> pystethoscope -d sf10 -i clk state thread pc -t statement
Connected to the database: sf10
{"clk": 1623697264566205, "thread": 6, "pc": 0, "state": "start"}

(mk) mk@toke::~> pystethoscope -d sf10 -i clk state thread pc stmt -t statement
Connected to the database: sf10
{"clk": 1623697339696741, "thread": 6, "pc": 0, "state": "start", "stmt": "function user.main();"}

The transformer could evolve to few standard profile settings, predefined or taken from a local file

The logger should be configurable and the defaults should be sane

Failed operating on {
  "version": "11.37.2 (hg id: 402f401ed9f8)",
  "user": 0,
  "clk": 1589895541376280,
  "mclk": 5763105814,
  "thread": 4,
  "program": "user.s4_0",
  "pc": 10,
  "tag": 23,
  "module": "generator",
  "function": "projection",
  "session": "faf1e78f-d8dd-4af6-bcb7-8b94b30edcc7",
  "state": "start",
  "usec": 0,
  "args": [
    {
      "ret": 0,
      "var": "X_22",
      "type": "bat[:hge]",
      "bid": 0,
      "count": 0,
      "size": 0,
      "eol": 14,
      "used": 1,
      "fixed": 1,
      "udf": 0
    },
    {
      "arg": 1,
      "var": "X_19",
      "type": "bat[:oid]",
      "persistence": "transient",
      "sorted": 1,
      "revsorted": 0,
      "nonil": 1,
      "nil": 0,
      "key": 0,
      "file": "tmp_360",
      "bid": 240,
      "count": 9801,
      "size": 78408,
      "eol": 10,
      "used": 1,
      "fixed": 1,
      "udf": 0
    },
    {
      "arg": 2,
      "var": "X_13",
      "type": "bat[:hge]",
      "persistence": "transient",
      "sorted": 1,
      "revsorted": 0,
      "nonil": 1,
      "nil": 0,
      "key": 1,
      "file": "tmp_566",
      "bid": 374,
      "count": 99,
      "size": 1584,
      "eol": 12,
      "used": 1,
      "fixed": 1,
      "udf": 1
    }
  ]
} ('stmt')

and

Failed operating on {
  "version": "11.37.2 (hg id: 402f401ed9f8)",
  "user": 0,
  "clk": 1589897887370379,
  "mclk": 8109099913,
  "thread": 9,
  "program": "user.s4_0",
  "pc": 9,
  "tag": 45,
  "module": "algebra",
  "function": "crossproduct",
  "session": "faf1e78f-d8dd-4af6-bcb7-8b94b30edcc7",
  "state": "done",
  "usec": 184,
  "args": [
    {
      "ret": 0,
      "var": "X_19",
      "type": "bat[:oid]",
      "persistence": "transient",
      "sorted": 1,
      "revsorted": 0,
      "nonil": 1,
      "nil": 0,
      "key": 0,
      "file": "tmp_30",
      "bid": 24,
      "count": 9801,
      "size": 78408,
      "eol": 10,
      "used": 1,
      "fixed": 1,
      "udf": 0
    },
    {
      "ret": 1,
      "var": "X_20",
      "type": "bat[:oid]",
      "persistence": "transient",
      "sorted": 0,
      "revsorted": 0,
      "nonil": 1,
      "nil": 0,
      "key": 0,
      "file": "tmp_324",
      "bid": 212,
      "count": 9801,
      "size": 78408,
      "eol": 11,
      "used": 1,
      "fixed": 1,
      "udf": 0
    },
    {
      "arg": 2,
      "var": "X_13",
      "type": "bat[:hge]",
      "persistence": "transient",
      "sorted": 1,
      "revsorted": 0,
      "nonil": 1,
      "nil": 0,
      "key": 1,
      "file": "tmp_612",
      "bid": 394,
      "count": 99,
      "size": 1584,
      "eol": 12,
      "used": 1,
      "fixed": 1,
      "udf": 1
    },
    {
      "arg": 3,
      "var": "X_13",
      "type": "bat[:hge]",
      "persistence": "transient",
      "sorted": 1,
      "revsorted": 0,
      "nonil": 1,
      "nil": 0,
      "key": 1,
      "file": "tmp_612",
      "bid": 394,
      "count": 99,
      "size": 1584,
      "eol": 12,
      "used": 1,
      "fixed": 1,
      "udf": 1
    },
    {
      "arg": 4,
      "var": "X_21",
      "type": "bit",
      "const": 1,
      "value": "false",
      "eol": 9,
      "used": 1,
      "fixed": 1,
      "udf": 0
    }
  ]
} ('var_name')

implement a switch that displays a version message

Currently there is no way to connect to a database with non-default credentials

On Ubuntu 18, "poetry install" failed with:

$ poetry install
Traceback (most recent call last):
File "/usr/lib/command-not-found", line 28, in
from CommandNotFound import CommandNotFound
File "/usr/lib/python3/dist-packages/CommandNotFound/CommandNotFound.py", line 19, in
from CommandNotFound.db.db import SqliteDatabase
File "/usr/lib/python3/dist-packages/CommandNotFound/db/db.py", line 5, in
import apt_pkg
ModuleNotFoundError: No module named 'apt_pkg'
Error in sys.excepthook:
Traceback (most recent call last):
File "/usr/lib/python3/dist-packages/apport_python_hook.py", line 63, in apport_excepthook
from apport.fileutils import likely_packaged, get_recent_crashes
File "/usr/lib/python3/dist-packages/apport/init.py", line 5, in
from apport.report import Report
File "/usr/lib/python3/dist-packages/apport/report.py", line 30, in
import apport.fileutils
File "/usr/lib/python3/dist-packages/apport/fileutils.py", line 23, in
from apport.packaging_impl import impl as packaging
File "/usr/lib/python3/dist-packages/apport/packaging_impl.py", line 24, in
import apt
File "/usr/lib/python3/dist-packages/apt/init.py", line 23, in
import apt_pkg
ModuleNotFoundError: No module named 'apt_pkg'

Original exception was:
Traceback (most recent call last):
File "/usr/lib/command-not-found", line 28, in
from CommandNotFound import CommandNotFound
File "/usr/lib/python3/dist-packages/CommandNotFound/CommandNotFound.py", line 19, in
from CommandNotFound.db.db import SqliteDatabase
File "/usr/lib/python3/dist-packages/CommandNotFound/db/db.py", line 5, in
import apt_pkg
ModuleNotFoundError: No module named 'apt_pkg'

We need to set up an action so that when a tag is pushed, it will prepare packages and upload them to PyPI.

mk@toke::~/monetdb-pystethoscope> pystethoscope -d demo -i “pc statement”
Connected to the database: demo
Key pc statement not found in the JSON object
{}
Key pc statement not found in the JSON object
{}
Key pc statement not found in the JSON object

This message should only be given once, or terminate the process

This also affects github actions.

Pystethoscope will produce a potential large trace. On a production system it may easy amount to 100MB/min of JSON data.
Since inspection is typically an 'afterthought', we need to limit the amount of data for post-analysis.

Proposal: add the following options:
--logs, to designate a directory where to keep the JSON log files
--interval, to designate the minutes between switching JSON files
--retention, to designate how long the log is kept

The interval preferably is a divisor of the hour.
The logfiles are named as DBNAME_interval-date-time.json.gz

As an intermediate step, this functionality can also be cast in a small StethoscopePool.py program

(mk) mk@toke::~/monetdb-pystethoscope> pystethoscope -d demo -t
Connected to the database: demo
{"pc": 0}

Produce a warning when the option key is given but no choice is made
For all keys, -i -t -x....

Double check the examples

More specifically, it may receive 'null\n', which is invalid JSON and probably represents a missing event.

Not all values in MAL programs are sensitive and are actually needed to understand the behaviour of a program.

• thetaselect(), only low/high value should be masked
• mat.packIncrement(), no masking at all
• join() flags should be kept

other operators should be checked

poetry run stethoscope.py -F json
poetry run stethoscope.py -F json sf10
poetry run pystethoscope -t identity -F json sf10

Only the last one makes a proper connection.
The former 2 trigger errors:
Using python3 (3.7.7)

[OSError]
[Errno 13] Permission denied