noresmhub / noresm Goto Github PK
View Code? Open in Web Editor NEWNorwegian Earth System Model and Documentation
Home Page: https://noresm-docs.readthedocs.io/en/latest/
License: Other
Norwegian Earth System Model and Documentation
Home Page: https://noresm-docs.readthedocs.io/en/latest/
License: Other
@DirkOlivie @monsieuralok @oyvindseland @matsbn @j34ni @adagj
When the latest issues (tagged next release) are worked up and solved, please initiate testing of the NorESM2.0.2 and report under this issue, which tests are done and what the results was.
Packaging the environment in a container would make installation of NorESM much easier and independent of local machine settings. It requires however root privileges on a given machine.
Initial tests are made to install NorESM via conda.
Coming soon: documentation about how to install the latest conda CESM environment and use it to run NorESM on a Virtual Machine or laptop/desktop to start with (eventually there will be a conda NorESM)
More general: regarding containers, Sigma2 has been asked for some time to install something like Singularity or Sarus (which are perfectly secure for use on HPC, and are actually already used elsewhere) and that was never done, so Sigma2 is aware that there is a demand, but it is clearly not given any priority.
The default drydep_method seems to be:
drydep_method = 'xactive_atm'
in the new release, while in the noresm-dev version (branch featureCESM2.1.0-OsloDevelopment) on metno it has
drydep_method = 'xactive_lnd'
This results in substantially lower SO2 concentrations (amongst other things) and I thought that @Kirkevag and @DirkOlivie told me in a discussion on the issues there, that 'xactive_lnd' was preferrable.
So I guess my question is: is this intentional? Should I use 'xactive_lnd' or 'xactive_atm'?
Both cases are set up with compset NFHISTnorpddmsbcsdyn.
We need to agree on a input data file naming scheme. The easiest is to put version information in the file name just as CESM is doing it.
additional info from Dirk:
Some NorESM data (in this case the directory) have a version date in their directory name :
Location : inputdata/noresm-only/atm/cam/camoslo/AeroTab_8jun17 :
Filenames : aerocomk0.out, aerocomk1.out, aerocomk2.out, aerocomk3.out, ...
Adding emission files significantly affects the reproducibility of the simulations, even if they only contain zeros (whether these zeros are defined as float or as double precision).
This is very likely linked to the problem that occurred in the summer 2019 (while carrying on the CMIP6 runs) although at that time it made the model crash more/less randomly, which is not the case here.
It would be nice to add contributing guidelines in the NorESM repository to explain (and encourage) external contributions.
Currently this is only available for the documentation (https://github.com/NorESMhub/NorESM/blob/master/doc/README.md).
Having a CONTRIBUTING.md file is the main repository seems to be a good practice.
Hi @matsbn @ChunchengGuo @j34ni
I see that the OMIP case examples in the documentation includes the '--run-unsupported' flag. Do we really need this? I tested these examples on FRAM without the '--run-unsupported' flag, and I don't see any problems with my runs.
We should include these, since we have done CMIP6 simulations using these compsets. I am currently doing a test of emission driven historical and will issue pull requests to NorESM and CAM repositories when this test is successful.
Based on feedback from the INES annual meeting, I have been looking into the options for making templates for issues and pull requests.
It is possible to have different templates for different types of issues, e.g. bug fixes, feature requests, improve documentation, etc., and include a default issue text, preset labels and preset assigners. I have tested it out in one of my own repos. This can be nice to have if there are many issues being created by people who do not have write access to the repo, and/or is unsure about what info to include with the issue. The downside is that it creates an intermediate step whenever someone wants to create an issue, but this shouldn't really be a big obstacle. Some people may find it annoying to have these preset options, so I think it is good to also include a blank "general issue" option.
I'm not sure how useful this option really is, as we do not have much (any?) issues created from outside the development team at the moment. It's also a question if we should have preset assigners, and if so, who this should be.
Øyvind mentioned another type of "template", that sounded to me like a step-by-step instruction on how to work with the issue and pull request system in gitHub ( @oyvindseland , please correct me if this is wrong ). I created something like that for the NorESM user workshop that we had last week, the slides are in my google drive
https://docs.google.com/presentation/d/16JL15wrb56wfMETrxWPKNP8tDsK5TVsFUM-lpboKXmc/edit?usp=sharing
(a PDF-version is also available form the NorESM user wokshop repo
https://github.com/NorESMhub/INES_workshop_2020/blob/gh-pages/files/Torsvik_gitcollaborate.pdf )
If there is an interest, I could perhaps reformat this for the NorESM documentation (or maybe the link to the NorESM user workshop is enough?).
Modify config_compsets.xml in CAM-Nor and in CIME : add the science_supported grids to the compsets used for CMIP6.
This avoids having to use the flag --run-unsupported when using building a case (./create_newcase) and gives an indication of which compsets have been tested and run for a longer time (and possibly contributed to CMIP6 simulations).
Hi all,
@matsbn @j34ni @jgriesfeller
in casefolder/Buildconf/micom.input_data_list, there are still several blom files which are not listed. E.g. ocean_regions.nc, inicon.nc. I think @j34ni may have the complete list of missing files. It is very difficult to get an overview of the missing files in BLOM if those files are not listed because you need to run the model several times, wait for it to crash and hope the error message is readable. I'm not sure if these "missing" files will be downloaded automatically if they are not listed. Then we need to add a description at least.
micom.input_data_list:
grid_file = /cluster/shared/noresm/inputdata/ocn/micom/tnx1v4/20170601/grid.nc
ncep_landmask_file = /cluster/shared/noresm/inputdata/ocn/micom/tnx1v4/20170601/land.sfc.gauss.nc
ncep_height_file = /cluster/shared/noresm/inputdata/ocn/micom/tnx1v4/20170601/hgt.sfc.nc
ice_clim_file = /cluster/shared/noresm/inputdata/ocn/micom/tnx1v4/20170601/icec_1968-1996.uf
sst_clim_file = /cluster/shared/noresm/inputdata/ocn/micom/tnx1v4/20170601/skt_1968-1996.uf
tidal_dissipation_file = /cluster/shared/noresm/inputdata/ocn/micom/tnx1v4/20170601/tidal_dissipation.nc
meridional_transport_index_file = /cluster/shared/noresm/inputdata/ocn/micom/tnx1v4/20170601/mertraindex.dat
meridional_transport_basin_file = /cluster/shared/noresm/inputdata/ocn/micom/tnx1v4/20170601/mertraoceans.dat
section_index_file = /cluster/shared/noresm/inputdata/ocn/micom/tnx1v4/20170601/secindex.dat
dust_file = /cluster/shared/noresm/inputdata/ocn/micom/tnx1v4/20170601/INPDUST_mhw.nc
river_file = /cluster/shared/noresm/inputdata/ocn/micom/tnx1v4/20170601/river_nutrients_GNEWS2000.nc
n_deposition_file = /cluster/shared/noresm/inputdata/ocn/micom/tnx1v4/20170601/ndep_tnx1v4_185001-201412.nc
Ada
It's very hard to identify needed files if the code does not tell what it is doing.
In an ideal world you print out all filenames read or written with full path to make them easy to find. If you are annoyed by it, put a verbose flag somewhere.
For KeyClim WP6, simulations will be done with around 10-different model versions. All these model versions will be based on the release-noresm2.0.2, but include changes (often in one aspect of the model - chemistry or PBL parameterisation or ...).
For visibility and intercomparibility, it would be good that all these versions exist as branches on github.com/NorESMhub in the respective repositories (e.g, NorESM, CAM, ...) I assume. The few developers that work on one modification can than exchange via those branches their progress, and developers that work on other modifications can at least see what others do. These branches then also define the exact version used for the different simulations in WP6.
Is this a reasonable way to do this?
But then all the individual developers implied in this process should need write access in the respective NorESMhub (NorESMhub/CAM, or NorESMhub/NorESM, or ...) repositories to start a new branch, and to manage the pull requests.
Is that how it should be?
In Externals.cfg change
[blom]
...
local_path = components/micom
...
to
[blom]
...
local_path = components/blom
...
There is now a new version of the ocean component called BLOM instead of MICOM (see https://github.com/NorESMhub/BLOM/releases/tag/blom0_0_001).
Is there any plan to integrate these changes in the master branch of NorESM?
Is anyone already working (or willing to work) on it at the moment?
@annefou We should link to some short explanations how to fork and pull request in the contribution section. Update a fork? add a file from another branch, etc.
(locations are not decided of.http://ns2345k.web.sigma2.no/NorESMInputdata/ just serves as a proxy)
Organise as follows:
./NorESM2.v0.01
./NorESM2.v0.02
./NorESM2.stable
./papers
./papers/GMD.20101231
./NorESM2.current
Dear all,
@DirkOlivie @lisesg @oyvindseland @tto061 @Kirkevag
We will release a new version of NorESM2 the 15th of June. I hope the documentation can be (rather) done by then.
1) We are missing a model description of CAM6-Nor:
https://noresm-docs.readthedocs.io/en/noresm2/model-description/atm_model.html
I think the best way is to make a list of the NorESM specific additions and most importantly add a description. I think the users should get a brief overview without needing to read all the referenced papers. Who is willing to do that? I don't know CAM6-Nor well enough. The easiest way is if you modify the file directly: https://noresm-docs.readthedocs.io/en/noresm2/model-description/atm_model.html
You can also send me an email and I can copy the information into the document.
Please see the BLOM description for an example:https://noresm-docs.readthedocs.io/en/noresm2/model-description/ocn_model.html
2) We also need to add a description of the initial conditions in CAM6-Nor. Where does the model start from? Is it a cold start? What about the stratosphere? The file to modify is here: https://noresm-docs.readthedocs.io/en/noresm2/model-description/atm_model.html
3) Code modifications. What is the best procedure to make code modifications with the new NorESMhub setup? A description can be added at the bottom of this page:
https://noresm-docs.readthedocs.io/en/noresm2/model-description/atm_model.htm
Let me know if you have questions!
Best regards,
Ada Gjermundsen
Hi,
@DirkOlivie
In README.case the name of the branch used to be included in the INFORMATION ABOUT YOUR GIT VERSION CONTROL SYSTEM section, but now only an * appear after the git branch: regardless of which branch is used. The commit nr is given, so it is possible to get the information but it is much easier to find out what is used if the branch name is listed. Is that possible to fix?
Ex:
Best regards,
Ada
finally managed to run a NorESM test again on Fram, but got an error message about missing inputfiles:
CaseDocs/ocn_in: INIDIC = '/cluster/shared/noresm/inputdata/ocn/blom/inicon/glodapv2_Ct_preind_OMIPinit_20171107.nc'
CaseDocs/ocn_in: INIALK = '/cluster/shared/noresm/inputdata/ocn/blom/inicon/glodapv2_At_OMIPinit_20171107.nc'
Those exist on /work/shared but not when I define my own input root.
On the other hand. When I ran the model 3 weeks ago the model simulations worked without these files, they were not even listed but without extra aerosol output
When I ran the model with standard input root these files were listed also three weeks ago so it looks like building ocn_in just copies the entire directory not individual files? And now the model actually ask for these files?
Or are they only needed for increased aerosol output in the atmosphere strange as that may sound?
Hi!
Most I have discussed with run into several SVN-related errors when launching the model. All need to change required=True to required=False for pop2 and ww3 in Externals.cfg.
POP2 and WW3 are not needed in NorESM2 so I'm thinking it is better if those are set to false by default.
Is it possible to update Externals.cfg so required=False for pop2 and ww3 are default?
After create_newcase and given that the set-up is supported this line is reported back
"This is a CESM scientifically supported compset at this resolution."
However it is a NorESM specific compsets.
Should we let it stand as is. Add "or NorESM" into the wording?.
Ideally we should have our own category, although likely not a priority now.
The mutual dependent pull requests #63, NorESMhub/cime#5, NorESMhub/CAM#2, NorESMhub/CESM_CICE5#2, NorESMhub/BLOM#7 have been issued to complete the transition from MICOM to BLOM, except for documentation.
I have verified bit-identical results and correct restart for N1850frc2 compset with f19_tn14 resolution and NOIIAOC compset with tn14 resolution on Fram. Although output and restart files from BLOM are now are named *.blom.* compared to *.micom.* before, the updated NorESM is able to branch from old experiments with *.micom.* type restart files.
If these pull requests are accepted, new tags should be created for the updated repositories and Externals.cfg updated accordingly.
Hi,
I tested out the newest version of NorESM2 (branch origin/noresm2) on nebula and an old problem which was fixed reappears.
For the re-submission, the model crashes due to this error:
ERROR:
GETFIL: FAILED to get ./NF2000climo_f19_f19_mg17_20200601.mosart.rh0.0026-01-01
-00000.ncC³^HC
³^HC³^HC
³^HC³^HC
³^HC³^HC
³^HC³^HC
³^HC³^HC
³^HC³^HC
³^HC³^HC
³^HC³^HC
³^HC³^HC
³^HC³^HC
³^HC³^HC
³^HC³^HC
³^HC³^HC
³^HC³^HC
³^HC³^HC
³^HC³^HC
³^HC³^HC
³^HC³^HC
³^HC³^HC
³^HC³^HC
³^HC³^HC
³^HC³^HC
³^HC`³^HC
which is a nebula specific error due to how strings are handled. That was fixed in the old noresm-dev repository by:
in components/mosart/src/riverroute/RtmRestFile.F90
changed Line 428 from
long_name=trim(lname), units=trim(uname), fill_value=spval)
to
long_name=trim(lname), units=trim(uname))
but reappeared now using the new NorESM2hub version.
Best regards,
Ada
The Configure Externals.cfg is basically a placeholder with a link to the CESM readme page. It would be useful to expand on this section with a list of available options.
Hi,
I tried to git clone from NorESMhub on Vilje, and got the following error -
git clone https://github.com/NorESMhub/NorESM.git noresm_test
Cloning into 'noresm_test'...
error: error:1407742E:SSL routines:SSL23_GET_SERVER_HELLO:tlsv1 alert protocol version while accessing https://github.com/NorESMhub/NorESM.git/info/refs?service=git-upload-pack
fatal: HTTP request failed
It works well on Fram and my local desktop. Does anyone have any clue on this?
To be able to reproduce a simulation we need to archive the case folder together with the data. Aleksi suggested some tools Mats has developed. the input-group suggest to edit the env_archive.xml somehow.
@annefou In the current trial setup for NorESM, cime_config and its sub-directories have been copied in manually. But shouldn't that be better a fork too? In the future, I would think that we also want to be able to update cime_config when updates in CESM appear.
This maybe implies that the NorESM repository is a fork of CESM ...
I think the README.md file should include a link to the www.noresm.org web page.
In order to identify files needed for the input data repository a user needs to know which files are actually needed. In some cases this is not the case right now since the file name is not printed out before the code tries to open a file. The user is then just left with an error message that some file could not be opened, but he does not know what file it was.
Detailed info will follow.
The BLOM input data directory on Vilje is not the same as on Fram, therefore the case reports missing files after submitting.
The case also failed to download the input files from noresm.org, e.g. -
Checking server https://www.noresm.org/inputdata with protocol wget
ERROR: Could not connect to repo '{0}'
This is most likely either a proxy, or network issue .
One can modify the user_nl_blom file to resolve this, but is there any plan to update the inputdata structure on Vilje? or any other recommendations/tips? @matsbn @JorgSchwinger
I noticed that the current Externals.cfg still points to cam_cesm2_1_rel_05-Nor_v1.0.0 which do not contain changes for the trailing slash, for example (and hence does not work for external users)
More generally how is this kind of updates managed (or how will it be managed in the future)?
How are NorESM users informed of the latest version they have to use?
Should we not point in Externals.cfg to (development) branches rather than releases so we would always have the latest updates?
I do not pretend to know what is best, but I believe that now a decision has to be made
NIRD needs to be the central place for input data. ALL input data needs to be there.
Details to follow.
I am updating the documentation about ESMValTool, mainly the part about NIRD Toolkit.
The access to the CMIP5&6, happyEVA and DCPP data should be explained in one section:
via ESGF
via nird account and read access to NS9034K
via raw output access, with knowledge of the cases used for the CMIP experiments
The link in the newbies guid
https://github.com/NorESMhub/NorESM/blame/noresm2/doc/configurations/newbie-guide.rst#L13
is not working in readthedocs
https://noresm-docs.readthedocs.io/en/noresm2/configurations/newbie-guide.html
I guess this is because of the relative path, but I didn't find an easy fix yet. Does anyone have any ideas?
Hi,
@DirkOlivie @matsbn
Kine and I get this error when trying to checkout manage_externals with the newest version. This is the case for both tag and branch.
Checking status of externals: cam, cice, cime, cism, clm, mosart, pop, blom, rtm, ww3,
Checking out externals: cam, cice, cime, ERROR:root:In repo "cime": tag "cime5.6.10_cesm2_1_rel_06-Nor_v1.0.0" is both a branch and a tag. git may checkout the branch instead of the tag depending on your version of git.
ERROR: In repo "cime": tag "cime5.6.10_cesm2_1_rel_06-Nor_v1.0.0" is both a branch and a tag. git may checkout the branch instead of the tag depending on your version of git.
Potentially linked to issue #55
This issue attempts to pull together various problems experienced so far with NorESM2 integrations:
Proposed tests (all to be run without "airbag" mods):
Dirk will also set up an explicit "zero forcing" volcanic case for Jean Iaquinta hoping that this will be useful for his current experiment.
On many or all pages on https://noresm-docs.readthedocs.io/en/latest/
there is written:
© Copyright 2018, Anne Fouilloux Revision 1229b03.
and no rights information.
I suppose the copyright information is automatically generated from GitHub's upstream commits, but it is incorrect (for at least some pages).
In addition, a CC-BY-4.0 or CC-BY-SA-4.0 licence is an idea.
For this realease, we should take care of the following:
The README.md file in the master branch of NorESM and CAM-Nor should shortly explain the role of the different branches.
This will help potential contributors to easily update the documentation; either from the command line or online.
Students should be able to modify the documentation so the guide should be simple and understandable by anyone even without advanced git/github skills.
@JorgSchwinger and I are working to restructure the input data for BLOM and iHAMOCC (https://github.com/NorESMhub/BLOM/tree/feature-blom_inputdata). The MICOM input data structure has essentially been fixed since 2009 and we have avoided restructuring before because it would have led to a messy and confusing structure due to the many model versions we are maintaining. It is a rare opportunity to do this restructure when transitioning from MICOM to BLOM and it should be completed before announcing a NorESM2 release to a wider user base, otherwise I fear we are stuck with the current structure for another decade. We have started this work on Fram, but Fram has experienced much unannounced downtime lately so we are delayed. We will update this issue with progress on the restructure.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.