illinois-ceesd / logpyle Goto Github PK

View Code? Open in Web Editor NEW

5.0 5.0 5.0 550 KB

logpyle: Time Series Logging (not just) for Simulation

Home Page: https://github.com/illinois-ceesd/logpyle/

License: Other

Python 37.04% Shell 0.03% HTML 62.35% CSS 0.27% JavaScript 0.31%

logging python

logpyle's People

Contributors

Stargazers

Watchers

Forkers

inducer mtcam majosm thomasgibson edgesftw

logpyle's Issues

Logging report bogus initial values

When using logging as status reporter, it report bogus values at step 0.

step = 0, sim time: 0.000000e+00 s
------- P (min, max) (Pa) = (0.000000000e+00, 0.000000000e+00)
------- T (min, max) (K)  = (      0,       0)
------- step walltime: 24.2316 s, log walltime:      0 s

EventCounter.transfer calls nonexistent method

EventCounter transfer calls pop on its counter argument. The counter is an any, however, the method pop does not exist for any class in logpyle.

MacOS 3.9 test fails

In #27, @matthiasdiener says:

I'll merge this even with the Python3.9-macos test failing. Let's hope they release a binary matplotlib package for this config.

I'm not a fan. I like green checkmarks. Could you add a shell snippet to skip that test for now?

Remove from pytools and datapyle

Please create MRs to remove these bits from their source packages, pointing here instead.

add functionality to restart logging

Newly initialized log managers always start at time 0, step 0.

Add functionality to set these parameters so logpyle can be used when restarting a simulation.

Drop reading/writing support for older schema versions

Note that reading such files can still be accomplished via #99.

This would affect:

get_logging
get_warnings
capture_warnings

Add support for user-specified watch formatting

Adding watches like this:

logmgr.add_watches(["step.max", "t_sim.max", "t_step.max",
                         "min_pressure", "max_pressure",
                         "min_temperature", "max_temperature",
                         "norm_momentum1"])`

Generates output that looks like this:

step.max=9 | t_sim.max=4.5e-07 | t_step.max=29.4292 | min_temperature=250.426 | max_temperature=405.511 | min_pressure=100 | max_pressure=105281

Allow formatting of the output data (i.e. float vs scientific), the names of the variables, and overall layout of the data. Perhaps something like this, for example:

step.max=9 | t_sim.max=4.5e-07 | t_step.max=29.4292
min_temperature=250 | max_temperature=406
min_pressure=1.00e2 | max_pressure=1.05e5

or better yet

Step 9  
Simulation time (s) 4.5e-07 | step wall time (s) 29.4292
T (K) (250, 406)
P (Pa) (1.00e2, 1.05e5)

Optional constructor parameter on MultiLogQuantity "units" is not truly optional.

While initialization does not cause a crash, calling "get_expr_dataset" on the corresponding LogManager will crash while parsing the empty string in the quantity's units.
Specifically, "pytools.lex.ParseError: unexpected end of input at end of input"
This doesn't seem to be an issue with the regular log quantity due to units being assigned to None as opposed to being cast to an empty string as in the case of MultiLogQuantity.
Surprisingly, the other optional quantity, "descriptions" does not seem to have this issue.

Synchronize the requirements from setup.py and requirements.txt?

Should we synchronize the requirements from setup.py and requirements.txt?

Originally posted by @matthiasdiener in #58 (comment)

Difference between LogQuantity and PostLogQuantity

Side note: It would be great to document the difference between LogQuantity and PostLogQuantity, but I couldn't think of anything useful.

Originally posted by @matthiasdiener in #58 (comment)

add a way to upgrade databases

Do we have a way to upgrade between schema versions? If not, this may be easier than maintaining code checking for older schemas.

Originally posted by @inducer in #94 (review)

Logging capture runs afoul of thread safety

Exception in thread Thread-1991 (_log_start_if_long):
Traceback (most recent call last):
  File "/home/andreas/pack/mambaforge/envs/ceesd/lib/python3.11/threading.py", line 1038, in _bootstrap_inner
    self.run()
  File "/home/andreas/pack/mambaforge/envs/ceesd/lib/python3.11/threading.py", line 975, in run
    self._target(*self._args, **self._kwargs)
  File "/home/andreas/src/pytools/pytools/__init__.py", line 2617, in _log_start_if_long
    self.logger.log(
  File "/home/andreas/pack/mambaforge/envs/ceesd/lib/python3.11/logging/__init__.py", line 1559, in log
    self._log(level, msg, args, **kwargs)
  File "/home/andreas/pack/mambaforge/envs/ceesd/lib/python3.11/logging/__init__.py", line 1634, in _log
    self.handle(record)
  File "/home/andreas/pack/mambaforge/envs/ceesd/lib/python3.11/logging/__init__.py", line 1644, in handle
    self.callHandlers(record)
  File "/home/andreas/pack/mambaforge/envs/ceesd/lib/python3.11/logging/__init__.py", line 1706, in callHandlers
    hdlr.handle(record)
  File "/home/andreas/pack/mambaforge/envs/ceesd/lib/python3.11/logging/__init__.py", line 978, in handle
    self.emit(record)
  File "/home/andreas/src/logpyle/logpyle/__init__.py", line 662, in emit
    self.mgr.db_conn.execute(
sqlite3.ProgrammingError: SQLite objects created in a thread can only be used in that same thread. The object was created in thread id 139805741225792 and this is thread id 139798635280064.

Support per-rank constants in runalyzer{,-gather}

Currently, if constants differ between ranks, one of the values gets chosen at random. I'd prefer if we had a constants table in the gathered database that preserves per-rank data.

'is_parallel' missing from constants

$ bin/runalyzer-gather summary.dat ../mirgecom/mylog.dat
Trouble with file '../mirgecom/mylog.dat'
Traceback (most recent call last):
  File "bin/runalyzer-gather", line 416, in <module>
    main()
  File "bin/runalyzer-gather", line 394, in main
    features, dbname_to_run_id = scan(fg, infiles)
  File "bin/runalyzer-gather", line 142, in scan
    logmgr = LogManager(dbname, "r")
  File "/Users/mdiener/Work/emirge/logpyle/logpyle/__init__.py", line 503, in __init__
    self._load()
  File "/Users/mdiener/Work/emirge/logpyle/logpyle/__init__.py", line 555, in _load
    self.is_parallel = self.constants["is_parallel"]
KeyError: 'is_parallel'

Remove init from documentation

See https://logpyle.readthedocs.io/en/latest

(e.g.):

Log multiple quantities per time step

@inducer Can we use this

logpyle/logpyle/__init__.py

Lines 131 to 135 in 99c2bd2

 class MultiLogQuantity(object): 

 """A source of multiple loggable scalars.""" 

 sort_weight = 0

Drop single-run mode of runalyzer-gather?

single-run mode leads to a lot of code duplication in runalyzer-gather.

As far as I can see, the only advantage of single-run mode is that the resulting file is slightly smaller due to the missing run_id column in each table.

Also, some functionality of runalyzer does not work with the single-run mode (e.g., the runprops command).

.plot with >2 parameters does not work

$ runalyzer -m summary.dat
Runalyzer running on Python 3.8.5 | packaged by conda-forge | (default, Sep 16 2020, 17:43:11)
[Clang 10.0.1 ]
Copyright (c) Andreas Kloeckner 2008
Run .help to see help for 'magic' commands
>>> .quantities
id | name    | unit | description                 | rank_aggregator
---+---------+------+-----------------------------+-----------------
7  | dt      | s    | Simulation Timestep         | None
9  | fifteen | None | None                        | None
5  | step    | 1    | Timesteps                   | None
2  | t_2step | s    | Step-to-step duration       | None
3  | t_cpu   | s    | Wall time                   | None
4  | t_log   | s    | Time spent updating the log | None
6  | t_sim   | s    | Simulation Time             | None
1  | t_step  | s    | Time step duration          | None
8  | t_vis   | s    | Time spent visualizing      | None
>>> .plot select $step, $t_step, $t_cpu
Traceback (most recent call last):
  File "/Users/mdiener/Work/emirge/logpyle/logpyle/runalyzer.py", line 305, in push
    self.execute_magic(cmdline)
  File "/Users/mdiener/Work/emirge/logpyle/logpyle/runalyzer.py", line 373, in execute_magic
    self.db.plot_cursor(cursor, columnnames)
  File "/Users/mdiener/Work/emirge/logpyle/logpyle/runalyzer.py", line 120, in plot_cursor
    do_plot(x, y, rest)
  File "/Users/mdiener/Work/emirge/logpyle/logpyle/runalyzer.py", line 115, in do_plot
    plot(x, y, hold=True, *args, **my_kwargs)
  File "/Users/mdiener/miniforge3/envs/ceesd/lib/python3.8/site-packages/matplotlib/pyplot.py", line 2840, in plot
    return gca().plot(
  File "/Users/mdiener/miniforge3/envs/ceesd/lib/python3.8/site-packages/matplotlib/axes/_axes.py", line 1743, in plot
    lines = [*self._get_lines(*args, data=data, **kwargs)]
  File "/Users/mdiener/miniforge3/envs/ceesd/lib/python3.8/site-packages/matplotlib/axes/_base.py", line 273, in __call__
    yield from self._plot_args(this, kwargs)
  File "/Users/mdiener/miniforge3/envs/ceesd/lib/python3.8/site-packages/matplotlib/axes/_base.py", line 418, in _plot_args
    return [func(x[:, j % ncx], y[:, j % ncy], kw, kwargs)
  File "/Users/mdiener/miniforge3/envs/ceesd/lib/python3.8/site-packages/matplotlib/axes/_base.py", line 418, in <listcomp>
    return [func(x[:, j % ncx], y[:, j % ncy], kw, kwargs)
  File "/Users/mdiener/miniforge3/envs/ceesd/lib/python3.8/site-packages/matplotlib/axes/_base.py", line 312, in _makeline
    seg = mlines.Line2D(x, y, **kw)
  File "/Users/mdiener/miniforge3/envs/ceesd/lib/python3.8/site-packages/matplotlib/lines.py", line 390, in __init__
    self.update(kwargs)
  File "/Users/mdiener/miniforge3/envs/ceesd/lib/python3.8/site-packages/matplotlib/artist.py", line 996, in update
    raise AttributeError(f"{type(self).__name__!r} object "
AttributeError: 'Line2D' object has no property 'hold'

Capture environment variables as a constant

runalyzer-gather does not work with multiple input files

$ runalyzer-gather -m summary.dat sod1d.dat*
Scanning...          [########################################] ETA ?
Traceback (most recent call last):
  File "/Users/mdiener/miniforge3/envs/ceesd/bin/runalyzer-gather", line 7, in <module>
    exec(compile(f.read(), __file__, 'exec'))
  File "/Users/mdiener/Work/emirge/logpyle/bin/runalyzer-gather", line 416, in <module>
    main()
  File "/Users/mdiener/Work/emirge/logpyle/bin/runalyzer-gather", line 396, in main
    fmap = make_name_map(options.feature_map)
  File "/Users/mdiener/Work/emirge/logpyle/bin/runalyzer-gather", line 188, in make_name_map
    assert match is not None
AssertionError

Input files (rename to *.dat):

Pypi needs description and github link

Script doesn't accept magic commands

$ cat script
.help

$ runalyzer summary.dat  script
Traceback (most recent call last):
  File "/Users/mdiener/miniforge3/envs/ceesd/bin/runalyzer", line 7, in <module>
    exec(compile(f.read(), __file__, 'exec'))
  File "/Users/mdiener/Work/emirge/logpyle/bin/runalyzer", line 37, in <module>
    main()
  File "/Users/mdiener/Work/emirge/logpyle/bin/runalyzer", line 25, in main
    exec(compile(open(args[1]).read(), args[1], 'exec'),
  File "script", line 1
    .help
    ^
SyntaxError: invalid syntax

Asymmetry in errors for double enable/disable for packages "warnings" and "logging"

For the package warnings, the function capture_warnings will report a runtime error if it attempts to set the warning state to its current state.
The function capture_logging will only report a warning under the same conditions if the current state is enabled. No warnings nor errors are reported if a double disable occurs for this function.

Copyright not correct

logpyle/LICENSE

Line 3 in a1e1d8c

Flake8 and the examples run should be separate jobs

LogManager.get_joint_dataset crashes when empty

Calling get_joint_dataset before any timesteps have added data to the expressions results in a runtime error when accessing zipped_dubs[2].

allow updating quantities through LogManager class, not LogQuantity

Allow calling this via

logmgr.set_quantity(name, value)

so that you don't have to lug around the LogQuantity?

Originally posted by @inducer in illinois-ceesd/mirgecom#391 (comment)

Create conda-forge package

https://conda-forge.org/docs/maintainer/adding_pkgs.html

Create a release?

Currently, logpyle has essentially the same code as the most recent versions of pytools.log and datapyle, just changing the references to these packages.

It also has docs and CI.
@inducer

"Cannot operate on a closed database" error

Logpyle has broken backward compatibility recently:

 Traceback (most recent call last):
  File "<frozen runpy>", line 198, in _run_module_as_main
  File "<frozen runpy>", line 88, in _run_code
  File "/home/runner/work/grudge/grudge/.miniforge3/envs/testing/lib/python3.11/site-packages/mpi4py/__main__.py", line 7, in <module>
    main()
  File "/home/runner/work/grudge/grudge/.miniforge3/envs/testing/lib/python3.11/site-packages/mpi4py/run.py", line 198, in main
    run_command_line(args)
  File "/home/runner/work/grudge/grudge/.miniforge3/envs/testing/lib/python3.11/site-packages/mpi4py/run.py", line 47, in run_command_line
    run_path(sys.argv[0], run_name='__main__')
  File "<frozen runpy>", line 291, in run_path
  File "<frozen runpy>", line 98, in _run_module_code
  File "<frozen runpy>", line 88, in _run_code
  File "/home/runner/work/grudge/grudge/test/test_mpi_communication.py", line 322, in <module>
    run_test_with_mpi_inner()
  File "/home/runner/work/grudge/grudge/test/test_mpi_communication.py", line 96, in run_test_with_mpi_inner
    f(actx, *args)
  File "/home/runner/work/grudge/grudge/test/test_mpi_communication.py", line 315, in _test_mpi_wave_op_entrypoint
    logger.info("Rank %d exiting", comm.rank)
  File "/home/runner/work/grudge/grudge/.miniforge3/envs/testing/lib/python3.11/logging/__init__.py", line 1489, in info
    self._log(INFO, msg, args, **kwargs)
  File "/home/runner/work/grudge/grudge/.miniforge3/envs/testing/lib/python3.11/logging/__init__.py", line 1634, in _log
    self.handle(record)
  File "/home/runner/work/grudge/grudge/.miniforge3/envs/testing/lib/python3.11/logging/__init__.py", line 1644, in handle
    self.callHandlers(record)
  File "/home/runner/work/grudge/grudge/.miniforge3/envs/testing/lib/python3.11/logging/__init__.py", line 1706, in callHandlers
    hdlr.handle(record)
  File "/home/runner/work/grudge/grudge/.miniforge3/envs/testing/lib/python3.11/logging/__init__.py", line 978, in handle
    self.emit(record)
  File "/home/runner/work/grudge/grudge/.miniforge3/envs/testing/lib/python3.11/site-packages/logpyle/__init__.py", line 662, in emit
    self.mgr.db_conn.execute(
sqlite3.ProgrammingError: Cannot operate on a closed databas

For example: https://github.com/inducer/grudge/actions/runs/4928343880/jobs/8806583617

LogManager.write_datafile appends first data entry to title.

Before writing the data points to the file, no newline character is inserted.
Ex. writing step vs value might write the following to a file:

# step vs. value20.0	51.0
1.0	15.0
2.0	67.0
3.0	3.0
4.0	0.0

	class MultiLogQuantity(object):
	"""A source of multiple loggable scalars."""

	sort_weight = 0