escomp / cesm Goto Github PK
View Code? Open in Web Editor NEWThe Community Earth System Model
Home Page: http://www.cesm.ucar.edu/
License: Other
The Community Earth System Model
Home Page: http://www.cesm.ucar.edu/
License: Other
By default cesm is pointing to https urls for git repositories. This is fine for reading public repositories, but requires the user to input their github password many times for private repos.
Document the recommended use and setup of ssh keys for accessing private forks.
For cesm2.1.1-rc.02 which updates clm fsurdat files the REFCASES for B compsets need to be updated.
Expect cases to run out of the box, however, they fail in trying to initialize CLM
These tests fail:
SMS_Ln1.f09_f09_mg17.FHIST_BGC.cheyenne_intel.cam-outfrq9s.20190314_115153_xlpcd1
SMS_Ln1.f09_g17.B1850.cheyenne_intel.allactive-defaultio.20190314_115153_xlpcd1
SMS_Ln1.f09_g17.B1PCTcmip6.cheyenne_intel.allactive-defaultio_min.20190314_115153_xlpcd1
For example for SMS_Ln1.f09_f09_mg17.FHIST_BGC.cheyenne_intel.cam-outfrq9s
(GETFIL): attempting to find local file
b.e20.BHIST.f09_g17.20thC.297_01_v2.clm2.r.1979-01-01-00000.nc
(GETFIL): using b.e20.BHIST.f09_g17.20thC.297_01_v2.clm2.r.1979-01-01-00000.nc
in current working directory
Reading restart dataset
check_dim ERROR: mismatch of input dimension 62034 with expected value
62100 for variable landunit
Did you mean to set use_init_interp = .true. in user_nl_clm?
(Setting use_init_interp = .true. is needed when doing a
transient run using an initial conditions file from a non-transient run,
or a non-transient run using an initial conditions file from a transient run,
or when running a resolution or configuration that differs from the initial conditions.)
ERROR:
ERROR in /gpfs/fs1/scratch/erik/cesm2.1.1-rc.02/components/clm/src/main/ncdio_p
io.F90.in at line 368
A seemingly unintuitive aspect of CESM/cime is that, once you have created a case from a particular directory, it is very important not to change the code checked out in that directory until you are completely done with your case. Is that documented somewhere? I don't see it in the quick start guide, and that seems like a good place to include a warning about this. Maybe we can have a section at the end like "Important points to bear in mind"?
Going from a git or svn checkout where the screen is inundated with every file checked out, to manage_externals where it silently sits there for minutes can be disconcerting as you are not sure if the command is making progress or not. A simple "starting to check out XXX", "completed check out of XXX" for each of the components would be useful so the user can make sure progress is actually being made. I ended up opening a new window and seeing that something was happening just to make sure progress was being made and it wasn't simply hung.
There is an important bug fix there.
@fischer-ncar - while not necessary, it will keep things more consistent if Externals.cfg is updated at some point to use the new, uppercase repository names in the repo URLs.
The relevant changes I made yesterday are:
In CESM2.1.0, some refcase qualifiers (e.g., for RUN_REFCASE
) were introduced which are possibly too specific about the compset match. While these items are currently working, they will not work with a proposed CIME modification (see ESMCI/cime#3055).
Problem: entries assume no components after WW3 (or any WAV component).
Expect: more general patterns
cesm2.1.0
no
I think we can change these (24) entries from:
compset="1850_CAM60_CLM50%BGC-CROP.*_CICE.*_POP2%ECO.*_MOSART_CISM2%NOEVOLVE_WW3_BGC%BDRD"
to
compset="1850_CAM60_CLM50%BGC-CROP.*_CICE.*_POP2%ECO.*_MOSART_CISM2%NOEVOLVE_WW3.*_BGC%BDRD"
I ran a test with this substitution with and without my proposed CIME change (which added _SESP
after WW3
). I think making this simple change will allow us to then update CIME to a version with the new compset scheme.
Thoughts? @klindsay28? @jedwards4b? @billsacks? @mvertens?
The script fails with
Checking status of externals: clm, mosart, ww3, cime, cice, pop, cism, rtm, cam,
Checking out externals: clm, mosart, ww3, ERROR:root:Command '[u'svn', u'checkout', u'--quiet', u'https://svn-ccsm-models.cgd.ucar.edu/ww3/release_tags/ww3_cesm2_0_rel_01', u'/home/mboisson/.local/easybuild/software/2017/Core/cesm/2.0.0/components/ww3']' returned non-zero exit status 1
ERROR:root:Failed with output:
svn: E170013: Unable to connect to a repository at URL 'https://svn-ccsm-models.cgd.ucar.edu/ww3/release_tags/ww3_cesm2_0_rel_01'
svn: E230001: Server SSL certificate verification failed: issuer is not trusted
ERROR: In directory
/home/mboisson/.local/easybuild/software/2017/Core/cesm/2.0.0
Process did not run successfully; returned status 1:
svn checkout --quiet https://svn-ccsm-models.cgd.ucar.edu/ww3/release_tags/ww3_cesm2_0_rel_01 /home/mboisson/.local/easybuild/software/2017/Core/cesm/2.0.0/components/ww3
See above for output from failed command.
Please check the log file manage_externals.log for more details.
ERROR: Failed with output:
svn: E170013: Unable to connect to a repository at URL 'https://svn-ccsm-models.cgd.ucar.edu/ww3/release_tags/ww3_cesm2_0_rel_01'
svn: E230001: Server SSL certificate verification failed: issuer is not trusted
ERROR: In directory
/home/mboisson/.local/easybuild/software/2017/Core/cesm/2.0.0
Process did not run successfully; returned status 1:
svn checkout --quiet https://svn-ccsm-models.cgd.ucar.edu/ww3/release_tags/ww3_cesm2_0_rel_01 /home/mboisson/.local/easybuild/software/2017/Core/cesm/2.0.0/components/ww3
See above for output from failed command.
Please check the log file manage_externals.log for more details.```
The current J1850G compset is set to:
1850_DATM%CRU_CLM50%BGC-CROP_CICE_POP2_MOSART_CISM2%EVOLVE_SWAV
But it should be:
1850_DATM%CRU_CLM50%BGC-CROP_CICE_POP2%ECO_MOSART_CISM2%EVOLVE_WW3_BGC%BDRD
Also, running a quick ERS test on the J1850G compset with the resolution used in the glacier spin-up runs results in these errors:
Model datm missing file file1 = '/glade/p/cgd/tss/CTSM_datm_forcing_data/atm_forcing.datm7.cruncep_qianFill.0.5d.V4.c130305/Precip6Hrly/clmforc.cruncep.V4.c2011.0.5d.Prec.1901-01.nc'
Cannot download file since it lives outside of the input_data_root '/glade/p/cesmdata/cseg/inputdata'
Model datm missing file file2 = '/glade/p/cgd/tss/CTSM_datm_forcing_data/atm_forcing.datm7.cruncep_qianFill.0.5d.V4.c130305/Precip6Hrly/clmforc.cruncep.V4.c2011.0.5d.Prec.1901-02.nc'
Cannot download file since it lives outside of the input_data_root '/glade/p/cesmdata/cseg/inputdata'
Model datm missing file file3 = '/glade/p/cgd/tss/CTSM_datm_forcing_data/atm_forcing.datm7.cruncep_qianFill.0.5d.V4.c130305/Precip6Hrly/clmforc.cruncep.V4.c2011.0.5d.Prec.1901-03.nc'
Cannot download file since it lives outside of the input_data_root '/glade/p/cesmdata/cseg/inputdata'
This error goes on 240 times. Also seems to be some errors about missing '/glade/p/cgd/tss/CTSM_datm_forcing_data/atm_forcing.datm7.cruncep_qianFill.0.5d.V4.c130305/Solar6Hrly/clmforc.cruncep.V4.c2011.0.5d.Solr.1917-07.nc' and possibly others. So, maybe the forcing data needs to be updated for this compset as well?
As @fischer-ncar and I noticed, the allactive/default testmods directory just contains a bunch of empty files. We should remove this testmods directory to avoid confusion. Then we'll need to (1) change tests that use allactive/default to instead have no testmod, and (2) remove references to this from include_user_mods files in other allactive testmods.
@fischer-ncar - #84 started with the point of doing this, but then I realized that it would change test names and so break some baseline comparisons, so I thought it would be best to leave this to you to do at a convenient time.
nanr noticed that B1850 points to the wrong REFCASE
I set up the experiment database for the standard B1850 case, setting:
RUN_REFCASE = b.e21.B1850.f09_g17.CMIP6-piControl.001,but my case is coming out of the box with this as my refcase: RUN_REFCASE=b.e20.B1850.f09_g17.pi_control.all.299_merge_v2
Similarly the BW1850 compset is pointing to an older REFCASE as well. All of the other compsets point to b.e21 REFCASES
REFCASE should be b.e21.B1850.f09_g17.CMIP6-piControl.001
but is: b.e20.B1850.f09_g17.pi_control.all.299_merge_v2
SMS_Ln1.f09_g17.B1850.cheyenne_intel.allactive-defaultio> ./xmlquery RUN_REFCASE,RUN_REFDATE
Results in group run_begin_stop_restart
RUN_REFCASE: b.e20.B1850.f09_g17.pi_control.all.299_merge_v3
RUN_REFDATE: 0134-01-01
And for BWHIST:
scripts/SMS_Ln1.f09_g17.BW1850.cheyenne_intel.allactive-defaultio.20190322_143108_50jbax> ./xmlquery RUN_REFCASE,RUN_REFDATE
Results in group run_begin_stop_restart
RUN_REFCASE: b.e20.B1850.f09_g17.pi_control.all.299_merge_v3waccm
RUN_REFDATE: 0134-01-01
### What is the changeset ID of the code, and the machine you are using: cesm2.1.1-exp13
We need to have a tag with updated CTSM to run from 2100-2300 for the SSP5 scenarios (SSP5-3.4, and SSP5-8.5). This will only be running with WACCM.
This will be on the CESM2.1. release series.
The CTSM tag needed for this is documented in this issue:
ESCOMP/CTSM#944
It likely will also need a CAM tag.
[yaman@drybiscuit C0]$ ./case.build
Building case in directory /home/yaman/Testsite/Climate_Model/my_cesm_sandbox/cime/scripts/C0
sharedlib_only is False
model_only is False
Generating component namelists as part of build
Creating component namelists
Calling /home/yaman/Testsite/Climate_Model/my_cesm_sandbox/components/cam//cime_config/buildnml
...calling cam buildcpp to set build time options
CAM namelist copy: file1 /home/yaman/Testsite/Climate_Model/my_cesm_sandbox/cime/scripts/C0/Buildconf/camconf/atm_in file2 /home/yaman/Testsite/Climate_Model/Cases/C0/run/atm_in
Calling /home/yaman/Testsite/Climate_Model/my_cesm_sandbox/components/clm//cime_config/buildnml
Calling /home/yaman/Testsite/Climate_Model/my_cesm_sandbox/components/cice//cime_config/buildnml
...calling cice buildcpp to set build time options
Calling /home/yaman/Testsite/Climate_Model/my_cesm_sandbox/components/pop//cime_config/buildnml
...calling pop buildcpp to set build time options
ERROR: Command /home/yaman/Testsite/Climate_Model/my_cesm_sandbox/components/pop/bld/build-namelist failed rc=2
out=
err=env: ‘/home/yaman/Testsite/Climate_Model/my_cesm_sandbox/components/pop/input_templates/ocn.base.tavg.csh’: No such file or directory
ERROR ocn.base.tavg.csh: env CASEBUILD=/home/yaman/Testsite/Climate_Model/my_cesm_sandbox/cime/scripts/C0/Buildconf OCN_GRID=gx1v7 /home/yaman/Testsite/Climate_Model/my_cesm_sandbox/components/pop/input_templates/ocn.base.tavg.csh failed: 32512
I have given the quick start documentation a quick read. Overall, it's very nice - thank you @bertinia for all of your work on this.
I noticed a few minor issues:
Directory structure of the source: I don't think the source (on master) should have subdirectories for the different versions. A given version of master will only contain a single version of the documentation. e.g., the cesm2.0 release branch will just contain documentation applicable to cesm2.0, the cesm2.1 release branch just documentation for cesm2.1, etc. It's just the build for which we want to have separate directories for the separate versions on a single branch.
Formatting of code blocks: by default, sphinx renders code blocks in python, I believe. This leads to syntax highlighting that is not usually appropriate. You can specify the language (including shell) prior to the code block, with something like .. code-block:: shell
rather than just ::
For CESM2 software requirements:
subversion client version (1.8 or greater)
to subversion client (version 1.8 or greater)
Under "CESM2 component sets", remove this sentence: "An exception to this rule is the use of “G” as a second letter to indicate use of the active glc model, CISM." -- this is no longer true
Would it be possible to store the CESM.cfg file components in alphabetical order? This
would help with the testdb display when comparing the component info stored in the database
with what is in the CESM.cfg file. Here's what is displayed from the database:
[cam]
tag = trunk_tags/cam5_4_143/components/cam
protocol = svn
repo_url = https://svn-ccsm-models.cgd.ucar.edu/cam1
local_path = components/cam
required = True
[cice]
tag = cice5_20170927
protocol = svn
repo_url = https://svn-ccsm-models.cgd.ucar.edu/cice/trunk_tags/cice5_20170927
local_path = components/cice
required = True
[cime]
tag = cime5.4.0-alpha.11
protocol = git
repo_url = https://github.com/ESMCI/cime/releases/tag/cime5.4.0-alpha.11
local_path = cime
required = True
[cime_config]
tag = cime_config0.1.0.alpha.05
protocol = git
repo_url = https://github.com/CESM-Development/cime_config/releases/tag/cime_config0.1.0.alpha.05
local_path = cime_config
required = True
[cism]
tag = trunk_tags/cism2_1_40
protocol = svn
repo_url = https://svn-ccsm-models.cgd.ucar.edu/glc
local_path = components/cism
required = True
[clm]
tag =
protocol = svn
repo_url = https://svn-ccsm-models.cgd.ucar.edu/clm2
local_path = components/clm
required = True
[mosart]
tag = branch_tags/mosart1_0_26_cimeupdates_tags/n01
protocol = svn
repo_url = https://svn-ccsm-models.cgd.ucar.edu/mosart
local_path = components/mosart
required = True
[pop]
tag = cesm_pop_2_1_20171117
protocol = svn
repo_url = https://svn-ccsm-models.cgd.ucar.edu/pop2/trunk_tags/cesm_pop_2_1_20171117
local_path = components/pop
required = True
[rtm]
tag = branch_tags/rtm1_0_62_cimeupdate_tags/n01
protocol = svn
repo_url = https://svn-ccsm-models.cgd.ucar.edu/rivrtm
local_path = components/rtm
required = True
[ww3]
tag = trunk_tags/ww3_170731
protocol = svn
repo_url = https://svn-ccsm-models.cgd.ucar.edu/ww3
local_path = components/ww3
required = True
[externals_description]
schema_version = 1.0.0
External components which maintain standalone capability need to maintain the checkout_externals version. Typically for CAM's case we take a previously tested CESM tag and duplicate the externals used in it. This is now being extended to needing to also maintain the version of checkout_externals. We want to identify and use the version of checkout_externals that was used in a specific CESM tag. Being able to check this easily, becomes more important as time goes on and checkout_externals is only updated sporadically - we need an easy way to determine that it has been updated.
An ideal solution would be for the plans page to have a location somewhere on each tag's page which indicates the checkout_externals which was used. Perhaps this could be populated (automatically?) when checkout_externals is brought into the CESM tag. Currently, there is no good way to determine the checkout_externals tag other than cloning cesm and mining the log.
N/A
N/A
N/A
N/A
@bjandre suggested that I bring @mvertens , @fischer-ncar , @bertinia and possibly @jedwards4b into this discussion.
Currently, when new tags are created on the NCAR SVN server, an automated email is sent to
a couple of different email lists; cgd-cseg for alpha tags and cesm-dev mailman account for
beta and release tags. The automated email is sent from a perl script that runs as a nightly cron job on the NCAR SVN server.
How do we want to handle tag update notifications from this github repo? Should we encourage users and developers to setup a personal github account and manage their notifications /watches or do we need a script that sends notifications similar to what we currently have for SVN?
While the help message highlights that checkout_externals.py must be run in the root of the source tree, the error message indicates a different error. The error message is about "not being able to find the model description file". It should probably also suggest: "Is this command being run from the root of the source tree?"
An important piece of information that we request via the forums is the model version, from running git describe
and manage_externals/checkout_externals --status --verbose
. But I have been finding that users run these from the wrong place or run them incorrectly (e.g., running checkout_externals --status
and then separately checkout_externals --verbose
.
I'd like to make it easier for users to provide the correct information (and easier for us to ensure that users have provided the correct information) by adding a simple python script in the top level of CESM called something like describe_version
. This script (which would not accept any arguments, other than --help
) would:
git describe
manage_externals/checkout_externals --status --verbose
I'd like to add this to both master and the cesm2.1 release branches.
@jedwards4b @mvertens does this make sense to you? Let me know if you have any thoughts on the details.
The 2101-01-01 restart is missing for
so the BWSSP126extcmip6 compset fails.
Expect compset test to run and PASS, it dies at case creation.
./create_test SMS_Ld1.f09_g17.BWSSP126extcmip6.cheyenne_intel.allactive-default
Errors were:
Building test for SMS in directory /glade/scratch/erik/cesm2.1.4-rc.01/cime/scripts/SMS_Ld1.f09_g17.BWSSP126extcmip6.cheyenne_intel.allactive-default.20200406_134811_in66vc
Refcase not found in /glade/p/cesmdata/cseg/inputdata/cesm2_init/b.e21.BWSSP126cmip6.f09_g17.CMIP6-SSP1-2.6-WACCM.001/2101-01-01, will attempt to download from inputdata
Client protocol gftp not enabled
Model refcase missing file refdir = '/glade/p/cesmdata/cseg/inputdata/cesm2_init/b.e21.BWSSP126cmip6.f09_g17.CMIP6-SSP1-2.6-WACCM.001/2101-01-01/'
wget failed with output: and errput --2020-04-06 13:48:18-- ftp://ftp.cgd.ucar.edu/cesm/inputdata/cesm2_init/b.e21.BWSSP126cmip6.f09_g17.CMIP6-SSP1-2.6-WACCM.001/2101-01-01/
=> ‘.listing’
Resolving ftp.cgd.ucar.edu (ftp.cgd.ucar.edu)... 128.117.23.220
Connecting to ftp.cgd.ucar.edu (ftp.cgd.ucar.edu)|128.117.23.220|:21... connected.
Logging in as anonymous ... Logged in!
==> SYST ... done. ==> PWD ... done.
==> TYPE I ... done. ==> CWD (1) /cesm/inputdata/cesm2_init/b.e21.BWSSP126cmip6.f09_g17.CMIP6-SSP1-2.6-WACCM.001/2101-01-01 ...
No such directory ‘cesm/inputdata/cesm2_init/b.e21.BWSSP126cmip6.f09_g17.CMIP6-SSP1-2.6-WACCM.001/2101-01-01’.
Model refcase missing file refdir = '/glade/p/cesmdata/cseg/inputdata/cesm2_init/b.e21.BWSSP126cmip6.f09_g17.CMIP6-SSP1-2.6-WACCM.001/2101-01-01/'
Model refcase missing file refdir = '/glade/p/cesmdata/cseg/inputdata/cesm2_init/b.e21.BWSSP126cmip6.f09_g17.CMIP6-SSP1-2.6-WACCM.001/2101-01-01/'
svn export failed with output: and errput svn: E170000: URL 'https://svn-ccsm-inputdata.cgd.ucar.edu/trunk/inputdata/cesm2_init/b.e21.BWSSP126cmip6.f09_g17.CMIP6-SSP1-2.6-WACCM.001/2101-01-01' doesn't exist
Client protocol None not enabled
Exception during build:
ERROR: Could not download refcase from any serve
In the README, I read that some components in CESM aren't Python 3 compliant.
In less than one month, Python 2 will reach it's end of life.
That means that even if security bugs appear, that that will not be solved.
Also, many numerical packages like Numpy and Scipy will no longer be maintained.
It is probably a good time to upgrade the Python 2 only code 👍
Remove the CMIP5 scenario compsets and tests and only allow the CMIP6 ones.
The CMIP5 RCP's are available as compsets and tests. We should remove them as we add the new CMIP6 ones.
Here are the current CMIP5 rcp's available:
[erik@hobart cime_config]$ grep RCP config_compsets.xml
<!-- CMIP5 RCP8.5 compsets -->
<alias>BRCP85L45BGCR</alias>
<lname>RCP8_CAM60_CLM45%BGC_CICE_POP2_RTM_SGLC_SWAV</lname>
<alias>BRCP85C5L45BGC</alias>
<lname>RCP8_CAM50_CLM45%BGC_CICE_POP2_MOSART_SGLC_SWAV</lname>
<value compset="RCP[2468]_">2005-01-01</value>
[erik@hobart cime_config]$ grep RCP test
testlist_allactive.xml testmods_dirs/
[erik@hobart cime_config]$ grep RCP testlist_allactive.xml
<test name="IRT" grid="f09_g17" compset="BRCP85L45BGCR" testmods="allactive/defaultio">
<test name="ERS_Ld7" grid="f09_g17" compset="BRCP85L45BGCR" testmods="allactive/defaultio">
Is the b1850 usermods directory still needed? It contains https://github.com/ESCOMP/cesm/blob/master/cime_config/usermods_dirs/b1850/user_nl_cpl
It was added in CESM-Development/cime_config#44, but I can't find notes about why it was needed then.
@jedwards4b @mvertens any idea?
If it isn't needed any more, I was thinking we should remove it to keep the usermods directories limited to important things like the cmip6 usermods.
RUN_TYPE and GET_REFCASE are wrong fro the SSP extensions
RUN_TYPE is startup but should be hybrid
and GET_REF_CASE is FALSE, but should be TRUE
Setup a case for: BWSSP585extcmip6, BWSSP534osextcmip6, or BWSSP126extcmip6
cesm2.1.4-rc.01
on cheyenne, but machine doesn't matter.
cesm2_2_beta04 points to 471ac52 for CMEPS (in Externals_cime.cfg). At least some cases built with this can't even create the case.
Note that CTSM is using the CMEPS hash: 253f612acae07b2b1dc73c84f1bb30b8e1b86ddd which is functional for the following case.
on cheyenne
qcmd -- ./create_test SMS_D_Ld5_Vnuopc.f10_f10_musgs.I2000Clm50BgcCropGs.cheyenne_intel.clm-default -r .
.
.
.
Pes setting: tasks is {'NTASKS_ATM': -2, 'NTASKS_ICE': -2, 'NTASKS_CPL': -2, 'NTASKS_LND': -2, 'NTASKS_WAV': -2, 'NTASKS_ROF': -2, 'NTASKS_OCN': -2, 'NTASKS_GLC': -2}
Pes setting: threads is {'NTHRDS_ICE': 1, 'NTHRDS_ATM': 1, 'NTHRDS_ROF': 1, 'NTHRDS_LND': 1, 'NTHRDS_WAV': 1, 'NTHRDS_OCN': 1, 'NTHRDS_CPL': 1, 'NTHRDS_GLC': 1}
Pes setting: rootpe is {'ROOTPE_OCN': 0, 'ROOTPE_LND': 0, 'ROOTPE_ATM': 0, 'ROOTPE_ICE': 0, 'ROOTPE_WAV': 0, 'ROOTPE_CPL': 0, 'ROOTPE_ROF': 0, 'ROOTPE_GLC': 0}
Pes setting: pstrid is {}
Pes other settings: {}
Pes comments: none
Compset is: 2000_DATM%GSWP3v1_CLM50%BGC-CROP_SICE_SOCN_MOSART_SGLC_SWAV_SESP
Grid is: a%10x15_l%10x15_oi%null_r%r05_g%null_w%null_z%null_m%usgs
Components in compset are: ['datm', 'clm', 'sice', 'socn', 'mosart', 'sglc', 'swav', 'sesp', 'drv', 'dart']
No charge_account info available, using value from PROJECT
cesm model version found: ctsm1.0.dev091-9-g1f5afc67
Batch_system_type is pbs
ERROR: Looks like job groups have already been created
git status
On branch pio2_cleanup
Your branch and 'mydev/pio2_cleanup' have diverged,
and have 62 and 2 different commits each, respectively.
(use "git pull" to merge the remote branch into yours)
Changes not staged for commit:
(use "git add <file>..." to update what will be committed)
(use "git checkout -- <file>..." to discard changes in working directory)
modified: src/main/ncdio_pio.F90.in
no changes added to commit (use "git add" and/or "git commit -a")
Current behavior is:
Processing externals description file : Externals.cfg
Processing externals description file : Externals_CLM.cfg
Processing externals description file : Externals_POP.cfg
Processing externals description file : Externals_CISM.cfg
Checking status of externals: clm,
ERROR: DEV_ERROR: regex to detect tracking branch failed.
If I changes the RE_TRACKING to be RE_TRACKING = re.compile(r'[(.+)]')
then I get what I think that I should expect:
Processing externals description file : Externals.cfg
Processing externals description file : Externals_CLM.cfg
Processing externals description file : Externals_POP.cfg
Processing externals description file : Externals_CISM.cfg
Checking status of externals: clm, fates, ptclm, mosart, ww3, cime, cice, pop, cvmix, marbl, cism, source_cism, rtm, cam,
sM ./cime
modified sandbox, cesmdev/master: ahead 136 --> cime5.4.0-alpha.23
M ./components/cam
modified sandbox, on cam1/trunk_tags/cam5_4_166/components/cam
s ./components/cice
clean sandbox, origin/master --> cice5_20180123
./components/cism
clean sandbox, on cism2_1_46
./components/cism/glimmer-cism
clean sandbox, on test_coupling_move_up_v2_n01
sM ./components/clm
modified sandbox, mydev/pio2_cleanup: ahead 62, behind 2 --> clm5.0.dev001
./components/clm/src/fates
clean sandbox, on fates_s1.4.1_a3.0.0_rev3
s ./components/clm/tools/PTCLM
clean sandbox, PTCLM2_171216c --> PTCLM2_180214
./components/mosart
clean sandbox, on mosart1_0_30
./components/pop
clean sandbox, on pop2/trunk_tags/cesm_pop_2_1_20180205
./components/pop/externals/CVMix
clean sandbox, on v0.72-conv_bugfix.002
./components/pop/externals/MARBL
clean sandbox, on MARBL/trunk_tags/MARBL0_22_0
./components/rtm
clean sandbox, on rtm1_0_65
./components/ww3
clean sandbox, on ww3/trunk_tags/ww3_180115
At some point, we should migrate https://wiki.ucar.edu/display/ccsm/Fortran+Compiler+Bug+List to the wiki here. We don't necessarily need to keep all of it: we could just keep the still-relevant pieces. I tried unsuccessfully to convert this automatically to markdown, using pandoc and other tools I found. The conversion of the big tables never went well. We might just need to recreate the portions that are still relevant.
As decided in #20 , we should fold https://github.com/cesm-development/cime_config into this repository.
In the top-level README: We should add a section equivalent to 2.3 "Modifying a component" but for a component in git, since it's important to point out the subtleties of remotes.
I am waiting to do this until ESMCI/manage_externals#4 is resolved, because I'm not able to do this using my preferred workflow with that issue in place.
Please ignore this issue - testing connection to slack
The Downloading the CESM2 Code page includes
clone instruction with a generic branch name "release2.0". The idea is to have
the release2.0 branch point to the most current, officially supported code base.
For example, the initial release2.0 branch would point to CESM2.0.0 tag; when
CESM2.0.1 tag is available, the release2.0 branch would be updated in the repo BUT
the documentation could remain the same.
This will require git describe
instructions be added for how to check the code version in
a users cloned sandbox.
The J1850G is pointing to the older CRUNCEP dataset rather than the v7 version that is now being used. We've removed the V4 version.
J1850G points to the older CRUNCEPV4 version rather than the v7 version
Setup the following test:
SMS_Ld5.f09_g17_gl4.J1850G.cheyenne_gnu.allactive-cism-test_coupling
Here's the fix that needs to be done:
diff --git a/cime_config/config_compsets.xml b/cime_config/config_compsets.xml
index d8a5b70..156b0e0 100644
--- a/cime_config/config_compsets.xml
+++ b/cime_config/config_compsets.xml
@@ -145,7 +145,7 @@
<compset>
<alias>J1850G</alias>
- <lname>1850_DATM%CRU_CLM50%BGC-CROP_CICE_POP2_MOSART_CISM2%EVOLVE_SWAV</lname>
+ <lname>1850_DATM%CRUv7_CLM50%BGC-CROP_CICE_POP2_MOSART_CISM2%EVOLVE_SWAV</lname>
</compset>
<entries
```>
### What is the changeset ID of the code, and the machine you are using: cesm2_2_alpha04g
### have you modified the code? If so, it must be committed and available for testing: No
### Screen output or log file showing the error message and context:
check input data shows data is missing.
> ./check_input_data
> .
> .
> .
> Model datm missing file file239 = '/glade/p/cgd/tss/CTSM_datm_forcing_data/atm_forcing.datm7.cruncep_qianFill.0.5d.V4.c130305/TPHWL6Hrly/clmforc.cruncep.V4.c2011.0.5d.TPQWL.1920-11.nc'
> Model datm missing file file240 = '/glade/p/cgd/tss/CTSM_datm_forcing_data/atm_forcing.datm7.cruncep_qianFill.0.5d.V4.c130305/TPHWL6Hrly/clmforc.cruncep.V4.c2011.0.5d.TPQWL.1920-12.nc'
Technically we could install the V4 data, but we are using the V7 data and want to be using the latest data rather than older versions. The older version takes a TByte of data and we don't want to waste the disk space.
CMIP5 future scenario names which are currently in compset longnames are based on the four standard RCP's: 8.5, 6.0, 4.5, 2.6. The CMIP6 future scenarios are SSP (Shared Socioeconomic Pathway) RCP (Representative concentration Pathway) combinations, usually expressed as SSP#-#.# for the SSP number, and RCP concentration.
The only recognized future scenario's are RCP[2468]. We need something like SSP[12345][123468], which is a minimum of five digits for it rather than the usual four.
The entire list is:
SSP5-RCP8.5
SSP1-RCP2.6
SSP3-RCP7.0
SSP5-RCP3.4
SSP2-RCP4.5
SSP1-RCP1.9
SSP4-RCP3.4
SSP4-RCP6.0
I feel like some of us discussed this before, but I can't remember the outcome: Would it make sense to fold https://github.com/cesm-development/cime_config into this new cesm repository? My understanding of that cime_config repository is that it serves the same role for a full cesm checkout that individual components' cime_config serve for those individual components. In particular: my impression is that https://github.com/cesm-development/cime_config is only needed for all-active cases, so there shouldn't be a need for (e.g.) CTSM or CAM standalone checkouts to include it.
The advantage of folding that in is that we'd have one less repository for which we need to manage issues, etc.
@mvertens @jedwards4b @fischer-ncar or others: Any thoughts on this?
@cecilehannay ran into trouble trying to checkout CESM using the instructions when using yellowstone. It turns out that the out-of-the-box git version on yellowstone is 1.7.1, but there are issues trying to fetch cime with this old git version. This is not related to manage_externals: even doing a manual git clone / git fetch leads to this issue:
First, when cloning:
$ git clone https://github.com/CESM-Development/cime
Initialized empty Git repository in /glade/p/work/sacks/temporary/cime/.git/
remote: Counting objects: 87692, done.
remote: Total 87692 (delta 0), reused 0 (delta 0), pack-reused 87692
Receiving objects: 100% (87692/87692), 33.18 MiB | 18.91 MiB/s, done.
Resolving deltas: 100% (64250/64250), done.
error: refs/tags/cime0_1_0 does not point to a valid object!
error: refs/tags/cime0.3.2 does not point to a valid object!
error: refs/tags/cime0.3.1 does not point to a valid object!
error: refs/tags/cime0.2.3 does not point to a valid object!
error: refs/tags/cime0.2.2 does not point to a valid object!
error: refs/tags/cime0.2.1 does not point to a valid object!
error: refs/tags/cime0.2.0 does not point to a valid object!
Then, when fetching:
$ git fetch origin
remote: Counting objects: 14, done.
remote: Total 14 (delta 1), reused 1 (delta 1), pack-reused 13
Unpacking objects: 100% (14/14), done.
error: unable to find 5fef7ed2dc848bda2aa3c106a77ba0a008ca858f
fatal: object 5fef7ed2dc848bda2aa3c106a77ba0a008ca858f not found
The error in fetching leads checkout_externals to abort.
yellowstone will be going away before this git-based version sees wide use. But should we document somewhere that a newer git version is needed? Version 1.8.3.1 (on hobart) works for me, so it appears that we need version 1.8 or later. Where should this documentation go?
I'm not sure if there are requirements for svn, or if any reasonable svn version should work.
I feel that the ">" at the start of shell commands in the quick start instructions is more troublesome than helpful, because it means you can't copy and paste commands. At least one other person (I think it was Cecile Hannay, but I may be wrong) felt the same way. This is a low-priority cleanup fix, but would be nice to do if others agree.
@bertinia are you attached to these ">" characters?
When I try to build a CAM6 from the released CESM2_0,
or update the components using
$> ./manage_externals/checkout_externals it fails when dealing with CLM
(https://github.com/ESCOMP/ctsm:release-clm5.0.01)
with the message:
Checking status of externals: clm, ERROR:root:
subprocess call to 'git' has exceeded timeout limit of 300 seconds.
Timout errors typically occur when svn or git requires
authentication to access a private repository. ...
When running case.build, I can add --skip-provenance-check
and it builds. But I'm not aware of a way to do a similar thing
when running checkout_externals. And it seems like such a
kluge shouldn't be necessary.
The directory where this is happening is
/gpfs/fs1/work/raeder/Models/cesm2_0,
which has a few untracked files, but nothing added or committed.
I had a clone of the cesm repo, with all externals checked out via manage_externals
(I'm not sure what version of cesm I was at originally - sorry). Then I updated to the latest version of master (2c75ae5). When I reran ./manage_externals/checkout_externals
, I got errors like this:
Checking out externals: clm, mosart, ww3, cime, cice, pop, ERROR:root:Command '[u'svn', u'switch', u'--quiet', u'https://svn-ccsm-models.cgd.ucar.edu/pop2/release_tags/pop2_cesm2_0_rel_n02']' returned non-zero exit status 1
ERROR:root:Failed with output:
svn: E195012: Path '.' does not share common version control ancestry with the requested switch location. Use --ignore-ancestry to disable this check.
svn: E195012: 'https://svn-ccsm-models.cgd.ucar.edu/pop2/release_tags/pop2_cesm2_0_rel_n02' shares no common ancestry with '/Users/sacks/cesm_code/cesm/components/pop'
ERROR: In directory
/Users/sacks/cesm_code/cesm/components/pop
Process did not run successfully; returned status 1:
svn switch --quiet https://svn-ccsm-models.cgd.ucar.edu/pop2/release_tags/pop2_cesm2_0_rel_n02
See above for output from failed command.
Please check the log file manage_externals.log for more details.
ERROR: Failed with output:
svn: E195012: Path '.' does not share common version control ancestry with the requested switch location. Use --ignore-ancestry to disable this check.
svn: E195012: 'https://svn-ccsm-models.cgd.ucar.edu/pop2/release_tags/pop2_cesm2_0_rel_n02' shares no common ancestry with '/Users/sacks/cesm_code/cesm/components/pop'
ERROR: In directory
/Users/sacks/cesm_code/cesm/components/pop
Process did not run successfully; returned status 1:
svn switch --quiet https://svn-ccsm-models.cgd.ucar.edu/pop2/release_tags/pop2_cesm2_0_rel_n02
See above for output from failed command.
Please check the log file manage_externals.log for more details.
I worked around this by doing rm -rf components/pop
then rerunning ./manage_externals/checkout_externals
. This gave a similar error for cam, which I again worked around with rm -rf components/cam
. Then it worked.
We need to add a Code of Conduct (COC) to the CESM project. Having a COC is a github best practice. And it's important for NSF funding. A COC helps maintain diversity and inclusion in a project as well as making the project as a whole more productive.
The COC needs to go both on master and cesm2.1 release since both of those branches continue to be maintained going forward.
http://www.cesm.ucar.edu/models/cesm2/naming_conventions.html#modelOutputFilenames
says that
a new metadata field called time_period_freq is now written to every history file header with one of the corresponding values:
second_N... N ∈ {1..3599} - data are written every N seconds
...
But this is not true for the cpl.hi. files, which seem to contain
instantaneous values.
It would also be helpful to have either a description of the meaning
of the standard characters that follow .h, (a,i,0-9,...) or pointers to that information,
like
http://www.cesm.ucar.edu/models/cesm2/component_settings/cam_nml.html:fincl1
for the first CAM history file.
Similarly, it would be helpful to update the driver namelist documentation
http://www.cesm.ucar.edu/models/cesm2/component_settings/drv_nml.html:history_option
to note that the output will appear in a .hi. file.
Similarly for the cpl.ha* files.
I wouldn't mind writing some of that documentation,
but I can't find the source code for pages like these.
They are not in the CESM2_0 release.
Thanks,
Kevin
New initial conditions for B1850 and BHIST
The current initial conditions REFCASES for B1850 and BHIST are from the LENS cases from quite a while ago.
The REFCASES that need to be used
https://github.com/NOAA-GFDL/FMS/
Needed for MOM ocean and FV3 ATM
(Moved from ESMCI/cime#2349)
I find the tag naming convention I started using in this repo, e.g. cesm2.0.beta07
, is hard to read. I'd like to suggest moving forward we use:
cesm-2.0.0-beta07
We currently have a google doc with some testing documentation:
https://docs.google.com/document/d/1oDNFPzw8bFbuyY129vX6-RJG-C9ZNUs-O1x2uVjSO4M/edit
CESM's old README file (from when CESM was in svn) pointed to this with the following text:
The most current process for updating and testing CESM versions is documented in a shared
NCAR/UCAR google document at:
https://docs.google.com/a/ucar.edu/document/d/1oDNFPzw8bFbuyY129vX6-RJG-C9ZNUs-O1x2uVjSO4M/edit?usp=sharing
I haven't looked through this documentation very carefully. It's quite possible that some of it is out-dated at this point. But some of it may be worth maintaining.
We should consider moving relevant parts to either the Wiki or other documentation within this repository.
As part of an NSC proposal to generate CAM6 ensemble
re-analysis forcing for all of the other CESM2 components,
I need to figure out which variables should be written
by the CAM assimilation into cpl history files.
This will be a 10-20 year series, saving every 3 hours,
so I should pare the list to the minimum, but complete, set.
I need to run some tests before writing the proposal,
which is due in mid-September, so I need to figure this
out soon, in order to calculate the data volume.
My understanding of the DATM mode in CESM2 is that
each component might want a distinct set of variables,
parcelled out into 1 or more streams.
The variables in the streams might be derived from
variables in the cpl history files. We won't calculate
those, but want to provide the input variables.
The 2 degree, CAM4, forcing files we've been using
for CLM, POP, and CICE have a minimal set of variables
a2x_Faxa_{rain,snow,lw,sw}*
a2x_Sa_{dens,pbot,pslv,ptem,shum,tbot,topo,u,v,z}
At the other end of the spectrum,
@mvertens recommended saving instantaneous fields,
which I interpreted to mean
./xmlchange HIST_OPTION='nhours'
./xmlchange HIST_N=3
which generate a cpl.hi file every 3 hours
with ~450 variables:
a2x_*
i2x_*
l2x_*
o2x_*
r2x_*
x2a_*
x2i_*
x2l_*
x2r_*
x2oacc_*
xaoa_*
xaoo_*
dom[ailor]*
frac[aior]*
I'm guessing and hoping that I don't need all of those in CESM2,
but I have questions about, for example, running an ocean
(+CICE?) only case, which should(?) be forced by river run-off,
which comes from CLM, which is not represented by just a2x fields.
Would I need to keep the l2x and r2x fields?
Only some of the variables?
@ekluzek says that for forcing CLM I just want
histaux_a2x3h = .true.,
which "turns on coupler history stream for 3-hour average atm
to coupler fields" according to the cesm1.2 web site.
This generated a cpl.ha2x3h file, with the variables
in the CAM4 set, above, but with 3 more variables;
a2x_Sa_{co2,topo}*
But I don't know if this is sufficient in CESM2
for all the other components.
And are averaged fields desirable for other components?
Erik also pointed me to a recent CLM spinup case,
which used files in
/glade/p/cesm/bgcwg_dev/forcing/b.e20.B1850.f09_g17.pi_control.all.297
They have the same variables as the histaux_a2x3h = .true. case,
plus 14 a2x_Faxa_{bc,dst,oc}* variables.
I haven't figured out how the additional variables were added,
or whether they're actually needed.
I dug up a POP case and see some correspondence between the
variables in its stream files and those in the CLM stream files,
but I can't tell whether all the POP variables can be derived
from the variables needed by CLM.
I haven't looked into the river, land ice, and wave models.
I hope that I can define a sufficient set of variables without
doing that.
Thanks for any (more) guidance on any of this.
Kevin
@fischer-ncar can you please update manage_externals for master and all release branches? @mnlevy1981 just made this change ESMCI/manage_externals#118 which we want moving forward.
an external python program, the current structure of checkout.py does not lend itself well to this desire.
I periodically get questions about how to point to a branch of a component (say, CAM, CTSM or CISM) within a CESM checkout (most recently from @Ivanderkelen). We have some documentation in the CESM README file, but I'm not sure that what we have is actually the method that makes the most sense. In particular, there are at least two problems with telling people to point to their branch in Externals.cfg:
Support for branches with manage_externals still leaves a lot to be desired (see ESMCI/manage_externals#34)
Additional problems are caused if you change an Externals.cfg file in the middle of the tree (rather than the top-level file). For example, if you want to point to a different version of FATES (an external of CTSM), you might be tempted to modify the file components/clm/Externals_CLM.cfg
. However, if you do that, when you rerun manage_externals from the top level (from the root of the CESM clone), you will get an error because the components/clm
external has a modified file. (@ekluzek has suggested allowing checkout_externals to proceed if there are modifications in these Externals files. This may be worth considering, although it might be hard to implement and the implications should be thought through carefully.)
So I'm inclined to recommend that people checkout their branch using regular git commands. For example, if you want to checkout your branch of CTSM in a CESM checkout, I would recommend getting CESM as normal and doing an initial run of manage_externals/checkout_externals
. Then do:
cd components/clm
git remote add ...
git fetch ...
git checkout ...
However, this isn't completely straightforward if your branch is of a component that has its own sub-externals (e.g., you have a branch of CTSM, which has a FATES external). In this case, I would probably still recommend using the above procedure to get your branch, but then getting any sub-externals by running the following from within components/clm
(NOT from the CESM top level):
./manage_externals/checkout_externals clm
I'd like to hear some thoughts from others. I don't feel in a rush to update this documentation, so I think it's best if we take some time to gather thoughts until we feel pretty happy with a recommendation for users.
@gold2718 @cacraigucar @nusbaume @ekluzek @Katetc @mnlevy1981 @alperaltuntas @jedwards4b @mvertens @rsdunlapiv @uturuncoglu
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.