illinois-ceesd / logpyle Goto Github PK
View Code? Open in Web Editor NEWlogpyle: Time Series Logging (not just) for Simulation
Home Page: https://github.com/illinois-ceesd/logpyle/
License: Other
logpyle: Time Series Logging (not just) for Simulation
Home Page: https://github.com/illinois-ceesd/logpyle/
License: Other
When using logging as status reporter, it report bogus values at step 0.
step = 0, sim time: 0.000000e+00 s
------- P (min, max) (Pa) = (0.000000000e+00, 0.000000000e+00)
------- T (min, max) (K) = ( 0, 0)
------- step walltime: 24.2316 s, log walltime: 0 s
EventCounter transfer calls pop on its counter argument. The counter is an any, however, the method pop does not exist for any class in logpyle.
In #27, @matthiasdiener says:
I'll merge this even with the Python3.9-macos test failing. Let's hope they release a binary matplotlib package for this config.
I'm not a fan. I like green checkmarks. Could you add a shell snippet to skip that test for now?
Please create MRs to remove these bits from their source packages, pointing here instead.
Newly initialized log managers always start at time 0, step 0.
Add functionality to set these parameters so logpyle can be used when restarting a simulation.
Note that reading such files can still be accomplished via #99.
This would affect:
Adding watches like this:
logmgr.add_watches(["step.max", "t_sim.max", "t_step.max",
"min_pressure", "max_pressure",
"min_temperature", "max_temperature",
"norm_momentum1"])`
Generates output that looks like this:
step.max=9 | t_sim.max=4.5e-07 | t_step.max=29.4292 | min_temperature=250.426 | max_temperature=405.511 | min_pressure=100 | max_pressure=105281
Allow formatting of the output data (i.e. float vs scientific), the names of the variables, and overall layout of the data. Perhaps something like this, for example:
step.max=9 | t_sim.max=4.5e-07 | t_step.max=29.4292
min_temperature=250 | max_temperature=406
min_pressure=1.00e2 | max_pressure=1.05e5
or better yet
Step 9
Simulation time (s) 4.5e-07 | step wall time (s) 29.4292
T (K) (250, 406)
P (Pa) (1.00e2, 1.05e5)
While initialization does not cause a crash, calling "get_expr_dataset" on the corresponding LogManager will crash while parsing the empty string in the quantity's units.
Specifically, "pytools.lex.ParseError: unexpected end of input at end of input"
This doesn't seem to be an issue with the regular log quantity due to units being assigned to None as opposed to being cast to an empty string as in the case of MultiLogQuantity.
Surprisingly, the other optional quantity, "descriptions" does not seem to have this issue.
Should we synchronize the requirements from setup.py and requirements.txt?
Originally posted by @matthiasdiener in #58 (comment)
Side note: It would be great to document the difference between LogQuantity and PostLogQuantity, but I couldn't think of anything useful.
Originally posted by @matthiasdiener in #58 (comment)
Do we have a way to upgrade between schema versions? If not, this may be easier than maintaining code checking for older schemas.
Originally posted by @inducer in #94 (review)
Exception in thread Thread-1991 (_log_start_if_long):
Traceback (most recent call last):
File "/home/andreas/pack/mambaforge/envs/ceesd/lib/python3.11/threading.py", line 1038, in _bootstrap_inner
self.run()
File "/home/andreas/pack/mambaforge/envs/ceesd/lib/python3.11/threading.py", line 975, in run
self._target(*self._args, **self._kwargs)
File "/home/andreas/src/pytools/pytools/__init__.py", line 2617, in _log_start_if_long
self.logger.log(
File "/home/andreas/pack/mambaforge/envs/ceesd/lib/python3.11/logging/__init__.py", line 1559, in log
self._log(level, msg, args, **kwargs)
File "/home/andreas/pack/mambaforge/envs/ceesd/lib/python3.11/logging/__init__.py", line 1634, in _log
self.handle(record)
File "/home/andreas/pack/mambaforge/envs/ceesd/lib/python3.11/logging/__init__.py", line 1644, in handle
self.callHandlers(record)
File "/home/andreas/pack/mambaforge/envs/ceesd/lib/python3.11/logging/__init__.py", line 1706, in callHandlers
hdlr.handle(record)
File "/home/andreas/pack/mambaforge/envs/ceesd/lib/python3.11/logging/__init__.py", line 978, in handle
self.emit(record)
File "/home/andreas/src/logpyle/logpyle/__init__.py", line 662, in emit
self.mgr.db_conn.execute(
sqlite3.ProgrammingError: SQLite objects created in a thread can only be used in that same thread. The object was created in thread id 139805741225792 and this is thread id 139798635280064.
Currently, if constants differ between ranks, one of the values gets chosen at random. I'd prefer if we had a constants table in the gathered database that preserves per-rank data.
$ bin/runalyzer-gather summary.dat ../mirgecom/mylog.dat
Trouble with file '../mirgecom/mylog.dat'
Traceback (most recent call last):
File "bin/runalyzer-gather", line 416, in <module>
main()
File "bin/runalyzer-gather", line 394, in main
features, dbname_to_run_id = scan(fg, infiles)
File "bin/runalyzer-gather", line 142, in scan
logmgr = LogManager(dbname, "r")
File "/Users/mdiener/Work/emirge/logpyle/logpyle/__init__.py", line 503, in __init__
self._load()
File "/Users/mdiener/Work/emirge/logpyle/logpyle/__init__.py", line 555, in _load
self.is_parallel = self.constants["is_parallel"]
KeyError: 'is_parallel'
single-run mode leads to a lot of code duplication in runalyzer-gather.
As far as I can see, the only advantage of single-run mode is that the resulting file is slightly smaller due to the missing run_id column in each table.
Also, some functionality of runalyzer does not work with the single-run mode (e.g., the runprops
command).
$ runalyzer -m summary.dat
Runalyzer running on Python 3.8.5 | packaged by conda-forge | (default, Sep 16 2020, 17:43:11)
[Clang 10.0.1 ]
Copyright (c) Andreas Kloeckner 2008
Run .help to see help for 'magic' commands
>>> .quantities
id | name | unit | description | rank_aggregator
---+---------+------+-----------------------------+-----------------
7 | dt | s | Simulation Timestep | None
9 | fifteen | None | None | None
5 | step | 1 | Timesteps | None
2 | t_2step | s | Step-to-step duration | None
3 | t_cpu | s | Wall time | None
4 | t_log | s | Time spent updating the log | None
6 | t_sim | s | Simulation Time | None
1 | t_step | s | Time step duration | None
8 | t_vis | s | Time spent visualizing | None
>>> .plot select $step, $t_step, $t_cpu
Traceback (most recent call last):
File "/Users/mdiener/Work/emirge/logpyle/logpyle/runalyzer.py", line 305, in push
self.execute_magic(cmdline)
File "/Users/mdiener/Work/emirge/logpyle/logpyle/runalyzer.py", line 373, in execute_magic
self.db.plot_cursor(cursor, columnnames)
File "/Users/mdiener/Work/emirge/logpyle/logpyle/runalyzer.py", line 120, in plot_cursor
do_plot(x, y, rest)
File "/Users/mdiener/Work/emirge/logpyle/logpyle/runalyzer.py", line 115, in do_plot
plot(x, y, hold=True, *args, **my_kwargs)
File "/Users/mdiener/miniforge3/envs/ceesd/lib/python3.8/site-packages/matplotlib/pyplot.py", line 2840, in plot
return gca().plot(
File "/Users/mdiener/miniforge3/envs/ceesd/lib/python3.8/site-packages/matplotlib/axes/_axes.py", line 1743, in plot
lines = [*self._get_lines(*args, data=data, **kwargs)]
File "/Users/mdiener/miniforge3/envs/ceesd/lib/python3.8/site-packages/matplotlib/axes/_base.py", line 273, in __call__
yield from self._plot_args(this, kwargs)
File "/Users/mdiener/miniforge3/envs/ceesd/lib/python3.8/site-packages/matplotlib/axes/_base.py", line 418, in _plot_args
return [func(x[:, j % ncx], y[:, j % ncy], kw, kwargs)
File "/Users/mdiener/miniforge3/envs/ceesd/lib/python3.8/site-packages/matplotlib/axes/_base.py", line 418, in <listcomp>
return [func(x[:, j % ncx], y[:, j % ncy], kw, kwargs)
File "/Users/mdiener/miniforge3/envs/ceesd/lib/python3.8/site-packages/matplotlib/axes/_base.py", line 312, in _makeline
seg = mlines.Line2D(x, y, **kw)
File "/Users/mdiener/miniforge3/envs/ceesd/lib/python3.8/site-packages/matplotlib/lines.py", line 390, in __init__
self.update(kwargs)
File "/Users/mdiener/miniforge3/envs/ceesd/lib/python3.8/site-packages/matplotlib/artist.py", line 996, in update
raise AttributeError(f"{type(self).__name__!r} object "
AttributeError: 'Line2D' object has no property 'hold'
$ runalyzer-gather -m summary.dat sod1d.dat*
Scanning... [########################################] ETA ?
Traceback (most recent call last):
File "/Users/mdiener/miniforge3/envs/ceesd/bin/runalyzer-gather", line 7, in <module>
exec(compile(f.read(), __file__, 'exec'))
File "/Users/mdiener/Work/emirge/logpyle/bin/runalyzer-gather", line 416, in <module>
main()
File "/Users/mdiener/Work/emirge/logpyle/bin/runalyzer-gather", line 396, in main
fmap = make_name_map(options.feature_map)
File "/Users/mdiener/Work/emirge/logpyle/bin/runalyzer-gather", line 188, in make_name_map
assert match is not None
AssertionError
Input files (rename to *.dat):
$ cat script
.help
$ runalyzer summary.dat script
Traceback (most recent call last):
File "/Users/mdiener/miniforge3/envs/ceesd/bin/runalyzer", line 7, in <module>
exec(compile(f.read(), __file__, 'exec'))
File "/Users/mdiener/Work/emirge/logpyle/bin/runalyzer", line 37, in <module>
main()
File "/Users/mdiener/Work/emirge/logpyle/bin/runalyzer", line 25, in main
exec(compile(open(args[1]).read(), args[1], 'exec'),
File "script", line 1
.help
^
SyntaxError: invalid syntax
For the package warnings, the function capture_warnings will report a runtime error if it attempts to set the warning state to its current state.
The function capture_logging will only report a warning under the same conditions if the current state is enabled. No warnings nor errors are reported if a double disable occurs for this function.
Line 3 in a1e1d8c
Calling get_joint_dataset before any timesteps have added data to the expressions results in a runtime error when accessing zipped_dubs[2].
Allow calling this via
logmgr.set_quantity(name, value)
so that you don't have to lug around the LogQuantity
?
Originally posted by @inducer in illinois-ceesd/mirgecom#391 (comment)
Currently, logpyle has essentially the same code as the most recent versions of pytools.log and datapyle, just changing the references to these packages.
It also has docs and CI.
@inducer
Logpyle has broken backward compatibility recently:
Traceback (most recent call last):
File "<frozen runpy>", line 198, in _run_module_as_main
File "<frozen runpy>", line 88, in _run_code
File "/home/runner/work/grudge/grudge/.miniforge3/envs/testing/lib/python3.11/site-packages/mpi4py/__main__.py", line 7, in <module>
main()
File "/home/runner/work/grudge/grudge/.miniforge3/envs/testing/lib/python3.11/site-packages/mpi4py/run.py", line 198, in main
run_command_line(args)
File "/home/runner/work/grudge/grudge/.miniforge3/envs/testing/lib/python3.11/site-packages/mpi4py/run.py", line 47, in run_command_line
run_path(sys.argv[0], run_name='__main__')
File "<frozen runpy>", line 291, in run_path
File "<frozen runpy>", line 98, in _run_module_code
File "<frozen runpy>", line 88, in _run_code
File "/home/runner/work/grudge/grudge/test/test_mpi_communication.py", line 322, in <module>
run_test_with_mpi_inner()
File "/home/runner/work/grudge/grudge/test/test_mpi_communication.py", line 96, in run_test_with_mpi_inner
f(actx, *args)
File "/home/runner/work/grudge/grudge/test/test_mpi_communication.py", line 315, in _test_mpi_wave_op_entrypoint
logger.info("Rank %d exiting", comm.rank)
File "/home/runner/work/grudge/grudge/.miniforge3/envs/testing/lib/python3.11/logging/__init__.py", line 1489, in info
self._log(INFO, msg, args, **kwargs)
File "/home/runner/work/grudge/grudge/.miniforge3/envs/testing/lib/python3.11/logging/__init__.py", line 1634, in _log
self.handle(record)
File "/home/runner/work/grudge/grudge/.miniforge3/envs/testing/lib/python3.11/logging/__init__.py", line 1644, in handle
self.callHandlers(record)
File "/home/runner/work/grudge/grudge/.miniforge3/envs/testing/lib/python3.11/logging/__init__.py", line 1706, in callHandlers
hdlr.handle(record)
File "/home/runner/work/grudge/grudge/.miniforge3/envs/testing/lib/python3.11/logging/__init__.py", line 978, in handle
self.emit(record)
File "/home/runner/work/grudge/grudge/.miniforge3/envs/testing/lib/python3.11/site-packages/logpyle/__init__.py", line 662, in emit
self.mgr.db_conn.execute(
sqlite3.ProgrammingError: Cannot operate on a closed databas
For example: https://github.com/inducer/grudge/actions/runs/4928343880/jobs/8806583617
Before writing the data points to the file, no newline character is inserted.
Ex. writing step vs value might write the following to a file:
# step vs. value20.0 51.0
1.0 15.0
2.0 67.0
3.0 3.0
4.0 0.0
Currently, add_quantities()
is restricted to floats.
In response to illinois-ceesd/mirgecom#96 (comment):
It'd be super nice if we could auto-label plot axes with their corresponding expressions.
runalyzer is much more useful with the mangle mode, should it be the default?
The only potential challenge I see is in case a quantity name starts with $
, but that seems relatively unlikely.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.