Currently, some of our Javascript is writing in-place CSS styling for some HTML elements. Sometimes this is due to desired conditional handling of styling, but other times, it's done as a convenience for a specific problem.

Overall, it'd be better to use classes/ids for these types of issues. Also, allowing an extension to specify an extra CSS style sheet (or sheets) and providing utilities to adjust the livvkit element html classes and ids would cover most use cases.

Previously, LIVVkit has boot-strapped the numpy install because "numpy likes to build from source and it's slow and prone to failure."

This was done through:

• easy_install via e01aaf9
• pip via fa5035f

and then removed in 98dee78 because we expect all installs to be done via pip (or conda) now, because pip is now the (python) recommended way to install packages for all python versions.

This will need to be tested on our supported HPC machines because they typically use legacy python setups and can be troublesome.

Any exception raised by extensions writers should use a custom LEXError to report the exception to make error handling easier.

To better align with Github's recommended community standards, we should:

• adopt a code of conduct
• add a CONTRIBUTING file
• provide Issue templates
• provide pull request templates

See: https://github.com/LIVVkit/LIVVkit/community

Some information for the CONTRIBUTING file:

• https://gist.github.com/PurpleBooth/b24679402957c63ec426
• https://mozillascience.github.io/working-open-workshop/contributing/

These updates will enable the same analysis to CESM as in Evans et al 2018, but targeting E3SM.

LIVVkit provides a livvkit.util.functions.merge_dicts function to combine multiple dictionaries (e.g., for combining json files passed to the -e option on the CLI).

This function can be dropped for generalized dictionary unpacking via PEP 448, which is available from python 3.5 on, once python 2 support has been dropped.

With the development of Atlas, it looks like LIVVkit extensions may defy neat classification into validation or numeric as we anticipated. Adding a -e/--extension option, and possibly a more generalized extensions component, would enhance the livv interface.

Jupyter notebooks can render custom html if a class implements the _repr_html method. If the element definitions were refactored as objects they could include _repr_html methods which would just call the relevant javascript functions inside a script tag.

If this was done I think the TeX helper module could be merged into the main element subclasses as well.

@jhkennedy I just randomly had this idea after learning about _repr_html and figured I'd throw it out there. Feel free to close if you don't think it's a good/feasible addition.

From @arbennett on July 29, 2016 19:22

Copied from original issue: ACME-Climate/LIVV#78

When LIVVkit's performance component can't find enough data, it doesn't always fail gracefully and sometimes produces errors like:

 -----------------------------------------------------------------
   Beginning performance test suite 
 -----------------------------------------------------------------

Process Process-14:
Traceback (most recent call last):
  File ".../lib/python2.7/multiprocessing/process.py", line 258, in _bootstrap
    self.run()
  File ".../lib/python2.7/multiprocessing/process.py", line 114, in run
    self._target(*self._args, **self._kwargs)
  File ".../lib/python2.7/site-packages/livvkit/components/performance.py", line 89, in _run_suite
    os.path.join(plot_dir, case + "_weak_scaling_efficiency.png")
  File ".../lib/python2.7/site-packages/livvkit/components/performance.py", line 387, in weak_scaling_efficiency_plot
    case_data['means'] = (means[0] / means) * 100
IndexError: index 0 is out of bounds for axis 0 with size 0

These kind of errors should be handled.

With the refactor of LIVVkit elements to classes, the LIVVkit output should become a class containing all the output elements, and have methods to write the output for the different output representations (HTML, JSON, LaTeX, etc.).

Also, this would allow for multiple outputs to be combined (see #49) easily as long as the JSON data is preserved -- every output should also include the JOSN data.

As of f6495d9 LIVVkit is able to import packages that may look like:

EXT_DIR
    -- ext.py
    -- SUB_PACKAGE
        -- __init__.py
        -- f1.py
        -- f2.py

but importlib won't import SUB_PACKAGE from within ext.py without adding EXT_DIR to the system path.

This could lead to clashes for same-named packages and clutters the system path.

The current workaround is to add EXT_DIR to the path until the all the needed imports are complete, and then removes them from the system path.

There should be a better way to do this.

There are some inconsistencies and missing information in the wiki that should be cleaned up pre-release.

Note: I'm working on these -- using this issue to keep track, and solicit input as needed.

[ ] Discussion of verification and validation in general:

• Terminology is a little mixed up -- need to go decide what we mean by code verification, code validation, performance validation, and scientific validation
• Code Validation section refers to what things done in our verification code. This is inconsistent.

[ ] User workflow

• Basic workflow for users -- perform an initial verification of install from benchmark data

[ ] Devloper workflow

• perform code verification, code validation, and performance validation across commits (pull requests)

[ ] Scientist workflow

• perform scientific validation, possibly in conjunction with performance validation (This is easy, as we don't currently have this capability)

[ ] include a development road map which describes our developmental goals.

The ability to fire up an http server to aid in viewing of the output website was recently added. However, this requires a test to be run and therefore can't be fired up to look at previously generated websites.

Although firing up an http server is a single command line command with python, and documented in the LIVVkit FAQs, it would make sense for this option to be available for previously generated websites.

ISMs and ESMs may use a variety of calendars for time-referencing model input and output data. For example, ESMF can be used in ACME/CESM to provide these calendars:

• 360 day
• Gregorian
• Julian
• Julian Day
• etc.

The datetime module in the python standard library performs calculations natively in the Gregorian calendar and provides an ISO calendar, but doesn't provide other calendars.

Extensions modules can fail to be imported because:

1. Module could not be found
2. Problems within the module like bad syntax or missing dependencies

LIVVkit assumes 1 on failure to import, but 2 is also very likely to happen as extensions may have additional dependencies.

Need to account for 2 in in LIVVkit.

From @jhkennedy on May 26, 2016 19:25

CISM allows repeated sections with identical names in its configurations files:

[CF output]
variables = thk usurf topg bheatflx acab uvel vvel wvel temp beta iarea iareag iareaf ivol rho_ice artm flwa    
name = GIS.8km.Const.4Albany.2D.out.nc
frequency = 5.0

[CF output]
variables = iarea iareag iareaf ivol rho_ice    
name = GIS.8km.Const.4Albany.SC.out.nc
frequency = 1.0

These repeated [CF output] section each setup an individual data 'pipe' [ish] which creates it's own output file.

ConfigParser will clobber these when loading the file, and only one section will be output.

Note: the output may even be a mix match of values from the multiple section as these sections are not ordered either.

We will need to move to a parser which can handle non-unique section names.

@arbennett pointed out we may be able to make a custom dictionary type which ConfigParser can use to provide this functionality:

https://stackoverflow.com/questions/9876059/parsing-configure-file-with-same-section-name-in-python

It would be preferable to create a solution which will allow ConfigParser to read as well as write CISM config files. That way a solution is available for writing python run scripts which take in and modify a base configuration file before running (i.e., all the CISM tests).

For asthetic reasons, it would also be nice to keep the sections order when reading and writing the files.

The custom dict would need to:

• Allow ConfigParser to read and store repeated sections
• Allow ConfigParser to write files with repeated sections
• Maintain the section order from read to write

Copied from original issue: ACME-Climate/LIVV#59

From @arbennett on July 29, 2016 19:49

Other possibilities are now and past, or current and reference. In any case, benchmark as a term carries a whole bunch of baggage with it that isn't appropriate for some of our comparisons. Similarly, test is used way too frequently for a bunch of things so something else would be better.

Copied from original issue: ACME-Climate/LIVV#80

Python 3.6+ natively uses pathlib, an object-oriented approach to paths and filenames. This is recommended as it replaces most of the "cumbersome" code from os.path and glob.glob (which use "dumb" string manipulation).

While the module pathlib2 is available lower python versions, there isn't guaranteed support in associated packages (numpy, scipy, etc) as there is in Python 3+ versions.

Therefore, this should happen in tandem with dropping python 2 support.

From @arbennett on July 29, 2016 18:29

It would be nice if we could provide Latex output alongside the web output. A straightforward way to do this would be to have a new class LatexHelper that would have functions to translate the results of the ElementHelper class's functions from Python dictionaries into the relevant Latex code.

Copied from original issue: ACME-Climate/LIVV#77

From @arbennett on February 18, 2016 6:6

Previously when handling data we had hardcoded ways of formatting. For example, with bit for bit processing there were two main variables with different shapes:

• Velocity - (layer, time, x, y)
• Thickness - (layer, x, y)

When extracting this data from a NetCDF file it just becomes a Numpy array with those shapes. To have smarter handling of this will make it easier to do plotting without resorting to breaking out different variables explicitely. This also seems to be a good situation to put into a model-specific bundle so that intercomparison tests will be possible when we have regridding infrastructure.

Copied from original issue: ACME-Climate/LIVV#48

It would be nice to be able to take multiple output websites, and combine them into a single portable website.

There should be a safe method, where nothing is overwritten if the combining sites have the same output pages, and an merge method that overwrites repeated output with some selected priority (newest? user specified?).

Also, some decisions will need to be made about how the summary/index page is handled...

From @jhkennedy on April 19, 2017 17:2

Currently, the displayed description for a test comes from the display item in the tests config file. For tests with many cases, this provides a nice way to describe each case.

However, for tests that do not have many cases, or will always have the same description, it would make sense to pull the displayed description from the __doc__ string in the test module (prevent duplication of text, making it easier to maintain).

This could be implemented by using the display item from the config file, and falling back to the __doc__ string if no display item exists.

Copied from original issue: ACME-Climate/LIVV#85

LIVVkit's method to import extension modules doesn't allow for extension "packages" which have multiple *.py files (likely in a directory structure). For example, an extension may look like:

    EXT_DIR
       -- ext.json
       -- ext.py
       -- SUB_PACKAGE
            -- __init__.py
            -- f1.py
            -- f2.py

Currently, only ext.py can be imported.

One solution is to add EXT_DIR to sys.path, but this may lead to unintentional package conflicts (e.g., same SUB_PACKAGE names).

Python 2 End Of Life has been declared; it will die in 2020.

We should not support python 2 after PyCon 2020.

Le roi est mort, vive le roi!

The json_tricks API changed between versions 3.8 and 3.11 and depreciated the json_tricks.np.dumps module in favor of always being numpy compatible and only using json_tricks.nonp.dumps when that feature isn't wanted.

See: http://json-tricks.readthedocs.io/en/latest/

We should:

• Update LIVVkit to use the new API
• Pin the json_tricks version b/c they like to break backwards compatibility

Sometimes the LIVVkit 2.1 anaconda distribution doesn't actually install the livv script.

From @jhkennedy on October 7, 2016 20:31

For example, in the bit-for-bit summary table, if some tests aren't bit-for-bit, the whole test row gets colored red, but the config files are typically matching so those should remain black.

Copied from original issue: ACME-Climate/LIVV#82

From @arbennett on July 29, 2016 19:43

For example on numerics pages for ISMIP-HOM it would be nice to have, in addition to the plots. the error percentages.

Copied from original issue: ACME-Climate/LIVV#79

From @arbennett on July 26, 2016 19:42

Copied from original issue: ACME-Climate/LIVV#72

I could not find an answer on the googles, so testing to find out.

From @jhkennedy on February 19, 2016 17:54

An bug was discovered in CISM ( ACME-Climate/cism-piscees#50 ) which resulted in the error message

newchild: child tstep can't be a parent of itself

being printed to stderr, but the test otherwise ran successfully.

LIVV doesn't currently check the stderr output from the tests (`*.config.oe) for any known errors.

We could either:

• parse the *.config.oe for any known errors

or

• split *.config.oe into individual files for stdout and stderr and then check to see if there is anything in the stderr file.

LIVV should then report these errors on a test's page.

Copied from original issue: ACME-Climate/LIVV#49

From @jhkennedy on March 31, 2017 16:1

Chrome, on both my linux and mac, will not display the Javascript elements of the output webpages when viewed on the local file system. However, Firefox and Safari will display everything correctly (have not tested IE).

To reproduce, download this output website (cism206v200.tar.gz), untar, and see if you can view the elements on cism206v200/index.html with Chrome and Firefox.

Interestingly, once the output site is made accessible on the internet, Chrome does display everything correctly. See here.

Copied from original issue: ACME-Climate/LIVV#83

There should be a guideline for how additional dependencies needed by extension modules are handled.

Currently, there is a js method to render a table with both vertical and horizontal headers (V-H Table), but there is no associated LIVVkit element.