noaa-gfdl / fre-cli Goto Github PK

• fre pp checkout-template
• fre pp configure -x $XML

/home/fms/local/opt/fre-commands/canopy/bin/frepp

268 # Checkout postprocessing template and configure
269 if [[ ! -d $srcdir ]]; then
270 echo "Workflow source directory '$srcdir' does not exist, so will now try to checkout template"
271
272 # checkout
273 mkdir -p $HOME/cylc-src
274 cd $HOME/cylc-src
275 # try to reduce checkout size with depth=1
276 #git clone --depth=1 --recursive [email protected]:fre2/workflows/postprocessing.git $name
277 git clone --depth=1 --recursive https://gitlab.gfdl.noaa.gov/fre2/workflows/postprocessing.git $name
278
279 # xml converter
280 cd $srcdir
281 if [[ $dual == true ]]; then
282 bin/fre-bronx-to-canopy.py -x $xml -p $platform -t $target -e $expname -v --dual
283 else
284 bin/fre-bronx-to-canopy.py -x $xml -p $platform -t $target -e $expname -v
285 fi
286 fi

fre cmor run works (or almost works), but when it was click-ified we removed the ability to CMORmixer.py by itself. For power users/developers, let's allow CMORmixer.py to be usable by itself.

e.g. this works:

fre cmor run -d /archive/lwh/ESM4/TOAR_O3RF/ESM4_TOAR_2014climo_2050ozone_v2/gfdl.ncrc5-intel22-prod-openmp/pp/atmos_cmip/ts/monthly/5yr -l GFDL-ESM4_amip_CMOR-Amon.lst -r CMIP6_Amon.json -p CMOR_input_ESM4_pdClim-2050ssp370-radO3.json -o /local2/home/may3

and by adding back the click options in this PR, this works identically (after activating a suitable python):

python fre-cli/fre/cmor/CMORmixer.py -d /archive/lwh/ESM4/TOAR_O3RF/ESM4_TOAR_2014climo_2050ozone_v2/gfdl.ncrc5-intel22-prod-openmp/pp/atmos_cmip/ts/monthly/5yr -l GFDL-ESM4_amip_CMOR-Amon.lst -r CMIP6_Amon.json -p CMOR_input_ESM4_pdClim-2050ssp370-radO3.json -o /local2/home/may3

--overwrite option for checkoutSscript.py that allows for removing and re-checking-out over a preexisting experiment. Cleans up prior directories via cylc clean if needed, gives a warning message with a pause when overwriting experiments, and provides messages about the process.

Do we want another level of lazy_loading in fre-cli so that tool groups don't need relative imports? As of now, an example that can demonstrate this is fre make and fre pp. Basically, this will also organize existing scripts into more specific subdirectories so that the subcommand scripts aren't just scattered around within a tool group's directory. Subdirectories for template files and tests can also be created.

The current tree structure:

pp
├── checkoutScript.py
├── configure_script_xml.py
├── configure_script_yaml.py
├── edits-template.yaml
├── frepp.py
├── __init__.py
├── install.py
├── ppyaml_test
├── __pycache__
├── run.py
├── schema.json
├── status.py
├── validate.py
└── wrapperscript
make
├── createCheckout.py
├── createCompile.py
├── createDocker.py
├── createMakefile.py
├── fremake.py
├── gfdlfremake
├── __init__.py
├── __pycache__
├── README-fremake.md
└── runFremake.py

The envisioned tree structure:

pp
├── frepp.py
├── frepp_scripts
├── __init__.py
├── schemas
├── tests
└── README.md
make
├── fremake.py
├── fremake_scripts
├── __init__.py
├── tests
└── README.md

With a structure like this, the imports within the tool files (i.e. fremake.py) would have to include relative imports for each subcommand like:
from .fremake_scripts.createCheckout import checkout_create

However, the tool group like make can itself become a lazy_group as well, with the capability to add its commands individually like in fre.py. However, this would take away the Click decorators in the file, and those would have to then be in the individual script files (this would mean that @click.pass_context won't need to be used, but scripts would need more rewriting).

https://github.com/NOAA-GFDL/fre-cli/blob/main/fre/frecatalog/frecatalog.py#L8

it says get_intake_gfdl , the file in catalogbuilder is gen_intake_gfdl.py. Make this consistent.

The shebang could use an update to use whatever python is in user environment.

So, use #!/usr/bin/env python instead of /usr/bin/python.

help needs some help.

Usage: frecatalog.py [OPTIONS] COMMAND [ARGS]...

Options:
--help Show this message and exit.

Commands:
buildcatalog - Execute fre catalog build

More FRE-CLI documentation is necessary as it is developed.

We could have a main README for what the FRE-CLI is, what it does, and goals/future mission statements and things we want to integrate here. (Background, vision, etc.).

We can also have a list of tools currently integrated into the CLI network and a "Guide" section that would point to a more detailed google doc (started by Bennett already). This doc can provide clearer user steps (on how to use tools) and developer steps (how to integrate a tool into the FRE-CLI).

fre pp convert currently writes rose-suite.conf, app/regrid-xy/rose-app.conf, and app/remap-pp-components/rose-app.conf.

We want it instead to write rose-suite.conf and pp.yaml.

There's some existing tooling in the fre/canopy "frepp" wrapper script that would really prefer to be here:

https://gitlab.gfdl.noaa.gov/fre2/system-settings/-/blob/main/bin/frepp

Functionality there that could be standalone fre-cli tools (hopefully..):

1. fre pp checkout-template NAME

Checkout pp template, from https://gitlab.gfdl.noaa.gov/fre2/workflows/postprocessing.git, to ~/cylc-src/NAME. If the destination already exists, do nothing.

2. fre pp configure -x XML NAME

Analogous to Dana's "fre pp configure -y pp.yaml" but will use a Bronx XML.

3. fre pp play -t YYYY NAME

If ~/cylc-run/NAME does not exist, then "cylc install NAME" first.

If NAME is already running ("cylc scan" could show this), then trigger the postprocessing at that YYYY time, with "cylc trigger NAME YYYY".

If it's not running, then start it first ("cylc play NAME"), then trigger the pp at the YYYY time.

Identical to "fre pp bronx" except uses pp.yaml instead of XML.

If you try, the frelist will fail. But then when you paste the failed frelist into your terminal, the frelist then works.

The problem is that fre pp configure-xml is somehow running the frelist from a different directory so the XML cannot be located.

The fix then, would be to either run the frelist from the current directory, or convert the relative path XML to absolute path before running the frelist.

The pp yaml includes a metadata section, with experiment, platform, and target. However, these shouldn't be set in the yaml as they can change when the user runs the command.

• create click options for experiment, platform, and yaml
• further clean up code and yaml if needed

fre pp checkout needs an optional branch argument for the clone of the fre2/workflows/postproc repo for testing and developer use.

The tests for fre2/workflows/postproc are supposed to be end-to-end tests that rely on the fre-cli infrastructure in a sort of bronx-like way (i.e. fre make, fre pp, etc.) However, there's a hard-coded clone of the main branch in fre pp checkout that the subsequent fre pp steps reference - if we use it as-is for the fre2/../testing, we're only going to be able to test the main branch, which sort of defeats the purpose of having a main branch in the first place.

Notifies users which directory is cloned into when checking out

Folder/File: frepp/frepp.py
from fre.frepp.configure import * works, but from fre.frepp.checkout import * doesn't work; had to specify import checkoutTemplate; so I changed the way imports work

Previously:
context.forward(yamlInfo)
referred to
from fre.frepp.configure import * at the top of the file

Now:
context.forward(fre.repp.configure.yamlInfo)
which refers to
import fre.frepp.configure at the top of the file

Also realized that arguments/options are needed in fre.py commands, but don't need to be passed into the function itself

https://github.com/aradhakrishnanGFDL/CatalogBuilder/tree/main

Create a README that would include a list of data catalog tools developed/integrated.

• includes what each tool does and how to use them.
• since data catalog tools are packaged, this will provide good base instructions for how to integrate other packaged tools into the FRE-CLI in the future

Constantly ran into modulenotfounderror no module named fre-cli. Even when using package named fre_cli with no dash, same error message would appear despite conda list showing fre_cli installed.

fms_yaml_tools
How to incorporate these tools into the CLI and leave as independent apps?
fre yaml diag convert diag_table_atmos
Maybe instead: fre yaml diag-convert [diag_table_atmos option]
fre yaml diag convert diag_table_land
fre yaml diag combine diag_table_atmos.yaml diag_table_land.yaml diag_table.yaml
fre yaml field convert field_table

for each 3 types: converting from table, combining with each other, validating
Why YAMLs? the why dictates the need for extra tooling
How efficiently can a user figure out/be told by YAML processing tools what history files will be written out, what variables are set in the YAML, etc? Are there any other properties that would be an explicit gain in functionality?

Want to incorporate use of classes and better importing habits.

https://click.palletsprojects.com/en/8.1.x/complex/

In particular, the frelist's take a very long time and give the appearance of the command being stuck, so it would be better to print the frelist's as they go to show the user that it's merely slow, not frozen.

We can create a README that would include a list of postprocessing tools developed. This will include what each tool does and how to use them.

fre pp configure -y edits.yaml, which uses a function from /fre-cli/fre/frepp/config.py, is unable to run successfully unless the user is inside of the /fre-cli/fre/frepp directory, where schema.json is located.

Line 23-24 in config.py:
23 with open("schema.json") as s:
24 schema = json.load(s)

This can be fixed by adding a path like with open("/home/Bennett.Chang/Coding/fre-cli/fre/frepp/schema.json") as s:, but this will not work with users who aren't developers. Will they have their own clone of the fre-cli directory? Or will we have to locate their schema.json elsewhere (which can be tricky because everyone has their own paths)?

Workflow script uses $ANACONDA_USERNAME and $ANACONDA_PASSWORD GitHub secrets; ideally this should be the GFDL Channel's login info, but Chris or my login info may be stored for use for the time being.

In regards to when the action is run, it is set right now to be called on PR (closed, opened, etc.).

However, there is an issue regarding versions. If the current version is not being changed (i.e. 0.1.0 to 0.1.1), the current existing package in the Anaconda cloud gets removed and nothing is uploaded.

Additionally, the end goal is to move the deployment out of this repo itself, and into the GFDL conda-recipes repo...

Create a README that would include a list of fre make tools developed/integrated.

• includes what each tool does and how to use them.
• fre make checkout
• fre make compile
• etc.

#60

• clobber warning
• 'package ignored' for non-python files

fre pp configure currently writes rose-suite.conf, app/regrid-xy/rose-app.conf, and app/remap-pp-components/rose-app.conf.

We want it instead to write rose-suite.conf and save itself to ~/cylc-src/NAME/pp.yaml

where NAME is

Currently, fre pp configure -y pp.yaml accepts a yaml file and writes 3 files in the current directory:

• rose-suite.conf
• app/regrid-xy/rose-app.conf
• app/remap-pp-components/rose-app.conf

The current pp.yaml looks like this (https://github.com/NOAA-GFDL/fre-cli/blob/main/fre/frepp/edits-template.yaml) and has an associated schema (https://github.com/NOAA-GFDL/fre-cli/blob/main/fre/frepp/schema.json).

Chris has reworked that pp.yaml structure to be more Bronx XML-like. Still up for debate / tweaks, though.

refineDiags:
  - script: "/path/to/abs_path/script"
  - script: "path/to/script/in/repo"

preAnalysis:
  - script: "/path/to/preanalysis/script"

settings:
  history_segment:
  start: 1980
  stop:
  site: "ppan"
  fre_analyis_home:
  pp_chunk_a:
  pp_chunk_b:
  grid_spec: "/path/to/tarfile"

switches:
  clean_work: true
  do_mdtf: false
  do_statics: true
  do_timeavgs: true
  do_refinediag: false
  do_atmos_plevel_masking: true
  do_preanalysis: false

directories:
  history: 
  pp:
  ptmp:
  history_refined:
  analysis:

metadata:
  experiment:
  platform:
  target:

components:
  - type: "atmos_cmip"
    source: atmos_month_cmip
    sourceGrid: atmos-cubedsphere
    xyInterp: 288,180
    interpMethod: conserve_order2
    timeSeries:
      - freq: 3hr
        source: atmos_8xdaily_cmip
        chunkLength: 1yr
      - freq: daily
        source: atmos_daily_cmip
        chunkLength: 1yr
        variables: huss tasmin tasmax tas pr psl sfcWind hurs uas vas zg500 ua10 ua850 va850 ua200
  - type: aerosol_cmip
    xyInterp: 288,180
    source: aerosol_month_cmip
    sourceGrid: atmos-cubedsphere
    interpMethod: conserve_order1
    timeSeries:
      - freq: "monthly"
        chunkLength: 1yr
      - freq: monthly
        chunkLength: 1yr
        source: atmos_month_refined

So we'd like fre pp configure to use this reworked pp.yaml instead.

Related points

1. The schema.json also needs corresponding updates
2. Currently, we're still using the regrid-xy and remap-pp-components rose-app config files, so these files should still be written. But soon, the regrid-xy and remap-pp-components tasks will use the pp.yaml directly.
3. fre pp configure currently assumes the user has checked out a suitable cylc template directory and is in that directory. We might be able to remove that assumption if the Cylc templates are stored in a predictable location, and the name could be passed in. e.g. fre pp configure -y pp.yaml NAME corresponding to the template location ~/cylc-src/NAME. But what if that location doesn't exist?

The following deployment scripts for GitHub actions to automatically conda build and upload to Anaconda Cloud are to be tested to see if one successfully works with what we want to achieve:

Anaconda Token is: NO-eb2ffcba-0f96-4b70-9268-34a0b464e504

https://github.com/uibcdf/action-build-and-upload-conda-packages/blob/main/README.md#other-tools-like-this-one

When using fre pp configure the path to the .xml file only works if it is an absolute path, not if it is a local path.

Had to revert changes from renaming subdirectories like frepp into pp due to fatal error that was happening;

• AttributeError: 'Group' object has no attribute

I will examine this in greater detail but for now the names are reverted for functionality's sake. It was caused due to the names conflicting with the @click.group names within fre.py

As for subtool function names, there is a small annoyance (but good practice for following Pythonic conventions), which is that in the main subtool file of a subtool directory, the subtool subcommand function must have a different name than its invoker in fre.py.

For example:

• fre/frepp/frepp.py: 'configureXML' is renamed to 'convert' in fre.pyto avoid a recursion error

Wrapper tool for traditional Bronx XML usage

https://gitlab.gfdl.noaa.gov/fre2/system-settings/-/blob/main/bin/frepp?ref_type=heads

Do we want to have a custom order of FRE tools? Click defaults --help to show the commands and groups in alphabetical order

• install issues, fixed with conda config --append channels noaa-gfdl
• new tool implementations not being reflected in workflow runs and not recognizing new subtool modules, fixed with removing git_url: specifier in meta.yaml
• conda installation being killed for unknown reason, reference
  • potential fix idea suggest: adding more RAM, this, or from my experimenting, upgrading /app/conda/miniconda within workstation systems to be a higher version (versions 23+ are able to install fre-cli faster and without issues)