Notes from ENSO RF exchanges to define metrics collections

2 levels of doc:
Level A: provided via output of package (short science provenance) in dict{}

1. dict for doc:

• Metric collection
  • name and description
  • parent MC further info URL
• per metric (no info on model and actual values)
  • metric long name and description
  • method to compute the metric
  • units
  • time frequency
  • reference observations
  • time period for reference observations
  • child metric further info URL

2. dict for actual calculation:

• diagnostic per reference observation
• diagnostic per model
  • time period for model
• metric values

[ more thinking needed for organisation ]

Level B: more comprehensive via a URL in short science provenance

• parent MC URL: https://github.com/eguil/ENSO_metrics/wiki/<MetricCollection>
• child metric URL: https://github.com/eguil/ENSO_metrics/wiki/<Metric>

I am thinking of moving the PMP driver as a part of PMP, which will help efficiently maintain the PMP driver. Any suggestion welcome.

Possibilities:

• re-order column in portrait plot
• "dive down" showing a bar plot for the metric selected

in matplotlib, such as:

Hi,

First of all thanks for developing these excellent metrics! I'm very interested in calculating the ocean-driven SSTs, and I noticed that Figures 1 and 2 are missing from https://github.com/CLIVAR-PRP/ENSO_metrics/wiki/EnsodSstOce

Would it be possible for you to include the figures for reference?

Thanks,

Paloma

Check x-axis sorting part and fix it. Discussion will be needed for whether to archive plotting code in this repo or not. The plotting code is only available on my local side at the moment.

ENSO_proc metrics, EnsoFbSshSst & EnsoFbTauxSsh, those needs SSH (zos for model) from AVISO have not been calculated.

I think below two commit should resolve the issue. @yyplanton what do you think?
995398a, a189894

I am re-running the code to verify if those changes solve the issue.

Current version of the code works in the Python 2 environment but not considering the Python 3 environment. Most recent version of cdms has moved toward Python 3 env.

Major differences are:

1. usage of print

• Py 2: print 'abc'
• Py 3: print('abc')

We can import from __future__ import print_function at the beginning of codes which use print statement and revise print in Py 3 style, then the code will work in both Py 2 and 3 env.

2. integer divided by integer

• Py 2: int/int = int
• Py 3: int/int = float

We need to make sure the code does what we want.

Cannot find data/basin_generic_1x1deg.nc:

OSpath__split
                         (path) /export_backup/lee1043/anaconda2/envs/pmp_nightly_20180830/lib/python2.7/site-packages/EnsoMetrics
                         (file) EnsoUvcdatToolsLib.pyc
                         (basin) /export_backup/lee1043/anaconda2/envs/pmp_nightly_20180830/lib/python2.7/site-packages/EnsoMetrics/../d
ata/basin_generic_1x1deg.nc
CDMS system error: No such file or directory
CDMS I/O error: Opening file /export_backup/lee1043/anaconda2/envs/pmp_nightly_20180830/lib/python2.7/site-packages/data/basin_generic_1x1de
g.nc
failed for  ACCESS1-0
Cannot open file /export_backup/lee1043/anaconda2/envs/pmp_nightly_20180830/lib/python2.7/site-packages/data/basin_generic_1x1deg.nc (Variab
le not found)

The issue can be temporarily solved by following in EnsoUvcdatToolsLib.py, but it is definitely not a permanent solution.

    # open file
    this_dir, this_filename = OSpath__split(__file__)
    if debug is True:
        dict_debug = {'line1': '(path) ' + str(this_dir), 'line2': '(file) ' + str(this_filename),
                      'line3': '(basin) ' + str(OSpath__join(this_dir, '../data/basin_generic_1x1deg.nc'))}
        EnsoErrorsWarnings.DebugMode('\033[93m', 'OSpath__split', 20, **dict_debug)
   ##ff = CDMS2open(OSpath__join(this_dir, '../data/basin_generic_1x1deg.nc'))  ## TEMPORARILY REPLACED BY BELOW LINE
    ff = CDMS2open('/export_backup/lee1043/git/ENSO_metrics_20190719/ENSO_metrics/data/basin_generic_1x1deg.nc')  ## TEMPORARY FIX

The this_dir look for conda installation dir even when code is only installed by python setup.py install not using conda.

@doutriaux1 any suggestion please?

• Obtain ENSO CLIVAR group agreement of science contents of MC perf
• for this:
  • apply to all CMIP5 models
  • compute correlation (and other diags, e.g. convergence of metrics) of current set
  • organise telco
  • agree on contents

ENSO metrics notes 26/7/2017
Discussion:

• design modular system for ENSO metrics that allows separation of concern and governance by appropriate experts
• build on tests done at PCMDI on Bellenger et al. paper by Ji Woo and Peter

Ideas:

• list of CLIVAR ENSO approved command lines to run enso_bellenger_compute.py (https://github.com/PCMDI/pcmdi_metrics/tree/issue_321_enso_bellenger_2017/src/python/devel/enso_bellenger)
• develop driver_enso_metrics.py from enso_bellenger_compute.py API (Ji Woo to advise, Yann to test in ENSO metrics environment)
• further develop enso_metrics_lib.py on github from programs by Ji Woo et al., further documenting the science - Yann and interface with ENSO CLIVAR
• interface with CDAT/PMP libraries by Ji Woo and Peter
• interface between CLIVAR ENSO science and metrics lib by Yann and Eric
• provide reference json/graphics/table for past CMIPs : Ji Woo and Peter

Actions, timeline (and who):

• telco late August to firm up work plan and timeline (Yann to organise)
• produce first draft of work plan early Sept. (Eric to lead)
• review by CLIVAR RF mid-Sept. (Andrew and Eric to lead)
• implement first version by late Sept. (Yann and Ji Woo to lead)
• present at Busan CLIVAR ENSO RF mid Oct (Yann and Eric)
• review science contents and make final public release Nov 2017

When ENSO CMIP6 paper is submitted, create a version with Tag 1.0

Yann and Jiwoo to propose list of observations (and date range) for each metrics

As a part of exploring multiple realizations, @gleckler1 and I am interested to analyze CESM-LE.

branch created: cesm_le

Beyond metrics (0D), how to add diagnostics (1-3D)

Goal is to run the ENSO metrics package using ESMValTool. This will require some changes (likely on either sides). Steps to get there:

• Kim et al explore roadmap of interface in ESMValTool dev
• Yann creates new branch 'test-ESMValTool' with one metric
• Exchange with Javier on testing this new branch

@yyplanton As a part of practicing more intense use of this issue board for code development, I suggest remove adamsregrid from code here because it is not included or supported by CDAT anymore.

@doutriaux1 @gleckler1 please correct me if I am wrong.

Use a MC to show different options of portrait plots:

• relative diff, RMSE, ratio
• shading vs. numbers
• etc...

Dear Eric and Yann,

Thanks for your email. Just to clarify: we were not asking for a detailed documentation but rather some guidelines on (a) software installation and (b) how to actually run the tool. Maybe Yann can help with this and send us a brief instruction?

We have now looked at the code in some more detail. One difficulty is that pre-processing, configuration settings, and diagnostics are spread across different routines. This makes an implementation of the ENSO metrics package into the ESMValTool difficult at this stage.

However, the restructuring of the ENSO package that you are planning will certainly be very helpful. In order to make the implementation into the ESMValTool as easy and as consistent with our standard work flow we would recommend splitting the ENSO metrics package code into three parts (if possible):

(1) Pre-processing of input data (including checks, unit conversion, etc.)
(2) Configuration settings (this would form the basis for a wrapper for the ESMValTool)
(3) Diagnostics (actual calculations)

As mentioned, we do not intend to include UVCDAT in the ESMValTool. Hopefully this part can be put into a separate pre-processing part, so when the ENSO package is run via the ESMValTool, the pre-processing could be done by the ESMValTool package instead.

We could then write an ESMValTool namelist and a wrapper that passes all information needed (“configuration”) from the ESMValTool to the ENSO package and then calls the diagnostic part (i.e. the actual calculation of the metrics).

This will ensure that the ENSO package remains a standalone package and new versions of it can easily be integrated into the ESMValTool. We would include a certain version of the ENSO package into the ESMValTool repository alongside with a wrapper and once you have a new release in the ENSO GitHub repository, we could easily update to it.

For the telecon next week, it is not completely clear to us what should be discussed. The next steps seem to be mostly of technical nature but if this is helpful we could explain in some more details why we think the above is important.

Best wishes,

Veronika, Axel, Mattia and Sabrina

Interpret obs requirements from collection description and get from obs4MIPs. Ensure standards and quality control are followed (land/sea, E/W and N/S orgasnisation of data, etc...)

Open issue for reminder and placeholder, after attending the clivar meeting.

API output:

• name
• metric(s) value, diff with obs
• unit(s)
• reference obs
• method
• reference
• range of graphic display ("red-blue" for "worse-better" in portrait plots)
• number of years/month in calculation

As the lifespan of CDAT is coming to an end, LLNL has developed xCDAT, which offers major CDAT capabilities using the xarray framework.

The CDAT library is now in maintenance-only mode, with plans for deprecation and cease of support around the end of calendar year 2023. Until this time, the dependencies for specific CDAT packages (cdms2, cdat_info, cdutil, cdtime, genutil, libcdms) will be monitored to ensure they build and install in Conda environments. CDAT currently supports Python versions 3.7, 3.8, 3.9, and 3.10. Unfortunately, feature requests and bug fixes will no longer be addressed.

With that being considered, PMP is in active to migrate to xCDAT, and such conversion is also needed for this ENSO metrics package.

To see internal variability and range of spread (and compare the spread to inter-model spread).

Maybe CESM large ensemble is another thing to keep in list, as discussed in the meeting brought up by Samantha Stevenson.

Issue opened for reminder and placeholder, after attending the clivar meeting.

• how and when to create a branch
• how and when to merge

Do we use absolute or relative (to std) SST criteria to build EN / LN composites ?
What anomalies are then displayed : absolute or relative ? And if relative, relative to what (local, global (basin) or e.g. nino3 region ?)

Note for future improvement

Current status: Code in the branch test_MC3 is running for all MCs and for CMIP5 and CMIP6.

Below is how long does code take to get diagnostic results nc files:

• Enso_perf: ~ 2 hr / model (thus, e.g., CMIP5: 2*45 ~ 90 hrs)
• Enso_proc, Enso_tel: ~ 1hr / model (thus, e.g., CMIP5 ~ 45 hrs)

Plan can be..

• Merge test_MC3 to master and tag v1.0 to secure current version of the working code before further change
• Audit the code to find reduce I/O and others
• Implement parallel computation
• Migrate to newer machine but it needs code to be revised for py3

Identify science/IT API fromexisting community tools (ESMValTool, CLIMAF, PMP,...):

• input command line options to drive_enso_metrics.py
• standard output files (including data and documentation)

for ENSO telectonnection:

• surface pressure (ps) or sea level pressure (psl)
• what observations / reanalyses ?

From Peter:
As we’ve discussed we have already incorporated a first set of enso metrics into the PMP,
and while there is cleaning up to do they are at: https://github.com/PCMDI/pcmdi_metrics/tree/issue_321_enso_bellenger/src/python/devel/enso_bellenger
We plan to further develop these in collaboration with you with emphasis on targeting the CMIP6 data structure.

I can see how you might want to have a separate git site to emphasize priorities at IPSL, which e.g., might insights that are not yet ready for “routine” ENSO metrics.

Our challenge will be to coordinate these two priorities (and github sites) as much a possible, so it will be great for us to start working together on next steps.

I don’t expect it to be as straightforward as
"these routines would just have arguments (e.g. input files, output metric) and that a wrapper would be needed to include it either in CLIMAF or in the PMP”

because we’ll want to leverage the built-in capabilities of various software tools. While numpy takes us a long way, as a common building block, for the enso metrics in the PMP we’ll certainly want to leverage
the time model functions available via cdms, which greatly simplify handling the data, and the genutil statistics that have been heavily vetted and used for the PMP framework of metrics.

Why don’t you give us one example of a new diagnostic/metric that you’d like to implement and we can work on it from both ends to help us identify how to move forward?

Provenance properties

1. Documentation: whys and hows of the metric and the collection

2. Math definition of the metric

3. Frequency (DA, MO,…)

4. Observations (as many as possible), including which period to use

5. Reference to show the robustness of metric

6. Minimum number of realisations  (for metric to make sense)

7. Minimum lenght of simulation needed (for metric to make sense)

Associated graphic properties:

8. « Dive down » diag (map of RMS,…)

9. Normalisation wrt sharing single color bar in a MME env.

@doutriaux1 I am testing with branch of this code (branch: test_MC3) but its installation using python setup.py install complains for __init.py__ (see bottom of below). In this env I am using python 2.7.15, because the code has not been fully converted for py3 yet. Any idea why I am getting this error?

(pmp_nightly_20180830) -bash-4.1$ git checkout test_MC3
Branch test_MC3 set up to track remote branch test_MC3 from origin.
Switched to a new branch 'test_MC3'

(pmp_nightly_20180830) -bash-4.1$ python setup.py install
running install
running build
running build_py
creating build
creating build/lib
creating build/lib/EnsoMetrics
copying lib/EnsoToolsLib.py -> build/lib/EnsoMetrics
copying lib/version.py -> build/lib/EnsoMetrics
copying lib/KeyArgLib.py -> build/lib/EnsoMetrics
copying lib/EnsoMetricsLib.py -> build/lib/EnsoMetrics
copying lib/EnsoComputeMetricsLib.py -> build/lib/EnsoMetrics
copying lib/__init__.py -> build/lib/EnsoMetrics
copying lib/EnsoUvcdatToolsLib.py -> build/lib/EnsoMetrics
copying lib/EnsoErrorsWarnings.py -> build/lib/EnsoMetrics
copying lib/EnsoCollectionsLib.py -> build/lib/EnsoMetrics
running build_scripts
creating build/scripts-2.7
copying scripts/my_Param.py -> build/scripts-2.7
copying scripts/my_Param_test.py -> build/scripts-2.7
copying scripts/EnsoMetricsGraph.py -> build/scripts-2.7
copying and adjusting scripts/PMPdriver_EnsoMetrics.py -> build/scripts-2.7
copying scripts/driverCCollection.py -> build/scripts-2.7
copying scripts/pmpParser.py -> build/scripts-2.7
copying scripts/driveGraphs.py -> build/scripts-2.7
copying scripts/driverCCollection_testMC3.py -> build/scripts-2.7
warning: build_scripts: scripts/__init__.py is an empty file (skipping)

copying scripts/driveMetrics.py -> build/scripts-2.7
copying scripts/driver_CCollection_testcmip6.py -> build/scripts-2.7
copying scripts/driverCCollectionPcmdiTest.py -> build/scripts-2.7
changing mode of build/scripts-2.7/my_Param.py from 750 to 755
changing mode of build/scripts-2.7/my_Param_test.py from 750 to 755
changing mode of build/scripts-2.7/EnsoMetricsGraph.py from 750 to 755
changing mode of build/scripts-2.7/PMPdriver_EnsoMetrics.py from 640 to 755
changing mode of build/scripts-2.7/driverCCollection.py from 750 to 755
changing mode of build/scripts-2.7/pmpParser.py from 750 to 755
changing mode of build/scripts-2.7/driveGraphs.py from 750 to 755
changing mode of build/scripts-2.7/driverCCollection_testMC3.py from 750 to 755
error: [Errno 2] No such file or directory: 'build/scripts-2.7/__init__.py'

@lee1043 @yyplanton @eguil When possible, YP creates branch with modified JSON structure and JL modifies PMP driver with branch. PG and JL will then test with portrait and parallel coordinates plots. When all ok both branches get merged to master.