I received the following error:

• folder=/projects/NS9034K/CMIP6/CMIP/NCC/NorCPM1/piControl
• cd /projects/NS9034K/CMIP6/CMIP/NCC/NorCPM1/piControl
• reals=($(basename $(find -maxdepth 1 -type d -name 'r*' -print)))
  +++ find -maxdepth 1 -type d -name 'r*' -print
  ++ basename ./r2i1p1f1 ./r1i1p1f1 ./r3i1p1f1
  basename: extra operand ‘./r3i1p1f1’
  Try 'basename --help' for more information.

There is something wrong in the line "reals = ...". The basename command typically only takes one main argument.

The Following fields are now supported:

Fields in daily output (micom.hbgcd)

'chlos       ','srfphyc*122*12/60/1000       '
'phycos      ','srfphyc       '

Sample cmorized fields

under /scratch/yanchun/cmorout/NorESM2-LM/historical/v20190815/Oday

chlos_Oday_NorESM2-LM_historical_r1i1p1f1_gn_20100101-20100228.nc
phycos_Oday_NorESM2-LM_historical_r1i1p1f1_gn_20100101-20100228.nc

commit:

c3c8210

The first two PAMIP experiments pdSST-pdSIC and pdSST-futArcSIC are finished and ready to be CMOR-ized. Both experiments consist of 100 members. I have not been able to run a single experiment with all 100 members because it is very CPU demanding, so each experiment consists of four cases with 25 members in each case.

PAMIP experiment pdSST-pdSIC:
Members 1-25: /projects/NS9560K/noresm/cases/NFHISTnorpddmsbc_PAMIP_1p1_pdSST-pdSIC_mem1-25_f19_mg17_20190819
Members 26-50:
/projects/NS9560K/noresm/cases/NFHISTnorpddmsbc_PAMIP_1p1_pdSST-pdSIC_mem26-50_f19_mg17_20190819
Members 51-75: /projects/NS9560K/noresm/cases/NFHISTnorpddmsbc_PAMIP_1p1_pdSST-pdSIC_mem51-75_f19_mg17_20190819
Members 76-100:
/projects/NS9560K/noresm/cases/NFHISTnorpddmsbc_PAMIP_1p1_pdSST-pdSIC_mem75-100_f19_mg17_20190819
PAMIP experimet pdSST-futArcSIC:
Members 1-25: /projects/NS9560K/noresm/cases/NFHISTnorpddmsbc_PAMIP_1p6_pdSST-futArcSIC_mem1-25_f19_mg17_20190819
Members 26-50: /projects/NS9560K/noresm/cases/NFHISTnorpddmsbc_PAMIP_1p6_pdSST-futArcSIC_mem26-50_f19_mg17_20190819
Members 51-75: /projects/NS9560K/noresm/cases/NFHISTnorpddmsbc_PAMIP_1p6_pdSST-futArcSIC_mem51-75_f19_mg17_20190819
Members 76-100: /projects/NS9560K/noresm/cases/NFHISTnorpddmsbc_PAMIP_1p6_pdSST-futArcSIC_mem76-100_f19_mg17_20190819
Note that the model forces the ensemble member numbering in the file names to be 0001 - 0025 in all cases, so for instance member 1 in case NFHISTnorpddmsbc_PAMIP_1p6_pdSST-futArcSIC_mem26-50_f19_mg17_20190819 is really member 26 etc. So unfortunately, the information on which ensemble members resides in which case can only be found in the case name itself (note the substrings "mem1-25", mem26-50" etc). Let me know if this is not well explained.

The experiments are 14 months long starting from 1. April 2000 and ending 31. May 2001. The first two months should be considered spin-up and must not be included in the CMOR-ized files. This means that the first day in the CMOR-ized files should be 1. June 2000 and the last day should be 31. May 2001 (a full year).

I have gotten a few requests from people who are writing up papers on the PAMIP data, and specific variables they are interested in include monthly-mean temperature, zonal wind, sea level pressure, and geopotential height, is if possible it would be nice if these are included in the initial CMOR-ized output. I have also had requests for the Eliassen-Palm fluxes, but I guess we still have some work to do to get those.

(L. Graff)

I 'source_id' registered NorESM1-F already at -

WCRP-CMIP/CMIP6_CVs#786

And you could see it from the cmip6 cv here (search NorESM) -

https://wcrp-cmip.github.io/CMIP6_CVs/docs/CMIP6_source_id.html

Could someone help with updating the local CV? @IngoBethke may I 'poke' you? thanks

in line 338-367 of file source/m_modelsatm.F

Add makefile for noresm2cmor3 compilation

Including the following activities:
/tos-project1/NS9034K/CMIP6/AerChemMIP
/tos-project1/NS9034K/CMIP6/CMIP
/tos-project1/NS9034K/CMIP6/DAMIP
/tos-project1/NS9034K/CMIP6/RFMIP

under which the following experiments are included:
1pctCO2
abrupt-4xCO2
hist-GHG
historical
hist-piAer
hist-piNTCF
pdSST-futArcSIC
pdSST-pdSIC
piClim-4xCO2
piClim-aer
piClim-BC
piClim-control
piClim-ghg
piClim-lu
piClim-OC
piClim-SO2
piControl

And there are v20190815, v20190909 and v20190917 versions in these experiments.

source_type is currently defined in mod*.nml and set to default value AOGCM. However, for e.g. ocean-only and atmosphere-only simulation the value should be set to e.g. AGCM and OGCM, respectively.

It could be an idea to use the experiment specific required_model_components information (the first entry if multiple entries exist) from CMIP6_CV.json to automatically set source_type in order to avoid QC errors.

Furthermore, the value could be overridden as function "modeling_real" which is variable specific the value could be overriden (e.g. modeling_realm=ocnBchem -> source_type:=BGC), given that the corresponding source type exists in either "required_model_components" or "additional_allowed_model_components".

Including the following activities:

• /tos-project1/NS9034K/CMIP6/AerChemMIP
• /tos-project1/NS9034K/CMIP6/CMIP
• /tos-project1/NS9034K/CMIP6/DAMIP
• /tos-project1/NS9034K/CMIP6/RFMIP

under which the following experiments are included:

• 1pctCO2
• abrupt-4xCO2
• hist-GHG
• historical
• hist-piAer
• hist-piNTCF
• pdSST-futArcSIC
• pdSST-pdSIC
• piClim-4xCO2
• piClim-aer
• piClim-BC
• piClim-control
• piClim-ghg
• piClim-lu
• piClim-OC
• piClim-SO2
• piControl

And there are date versions in these experiments:

• v20190815
• v20190909
• v20190917

Mandatory information:

Full path to the case(s) of the experiment on NIRD
/projects/NS2345K/noresm/cases/N1850_f19_tn14_20190730esm

experiment_id
esm-piControl

CASENAME(s) and years to be CMORized
N1850_f19_tn14_20190730esm, 1851-2100

Optional information

(additional information, if the experiment is branched/hybrid restart from previous parent experiment; you may find more information relevant to the experiment_id here: https://github.com/NorwegianClimateCentre/noresm2cmor/blob/master/tables/CMIP6_CV.json)

parent_experiment_id
piControl
'esm-piControl-spinup'

parent_experiment_rip
e.g., r1i1p1

parent_time_units
'days since 1851-01-01'

branch_method
'Hybrid-restart from year 1851-01-01 of piControl',

other information
(provide other information that might be useful)
Prognostic atmospheric CO2 simulation

in produced v20190815
updated va_*.nml; commit: 10da1af
volcello is reproduced and update in v20190815.

model_id in cmip5 is replaced with source_id in CMIP6
in current implementation, model_id is kept in mod_*.nml, but not used. And source_id is replaced with source.

in mod_*.nml

 2  model_id      = 'NorESM2-LM',
  3  source        = 'NorESM2-LM (2017): ....'

in m_namelist.F

963       CALL merge_strarr(slenmax,smax,source,source1,linebreaks)
1583       CALL json%add(p,'source_id',TRIM(source1))
1584       CALL json%add(p,'source',TRIM(source1))

maybe it is a typo?

Change
CALL json%add(p,'source_id',TRIM(source1))
to
CALL json%add(p,'source_id',TRIM(model_id))
If this is not intended?

Would you please help with cmorising the following omip1/omip2 experiments? They are stored on NIRD at -

/tos-project1/NS2345K/noresm/cases/NOIIAOC20TR_T62_tn14_20190628/
/tos-project1/NS2345K/noresm/cases/NOIIAJRAOC20TR_TL319_tn14_20190710/

(c.guo)

from time to time, the noresm2cmor crashes and reports error open json file. In the log:

 
^[[2;31;47mC Traceback:
! In function: cmor_open_inpathFile
! called from: cmor_dataset_json
! ^[[0m
 
^[[1;31;47m!!!!!!!!!!!!!!!!!!!!!!!!!
!
! Error: Please validate JSON File!
! USE: http://jsonlint.com/
! Syntax Error in file: namelist_NFHISTnorpddmsbc_PAMIP_1p6_pdSST-futArcSIC_mem1-25_f19_mg17_20190819_ra.json
!  {
 
!
!!!!!!!!!!!!!!!!!!!!!!!!!^[[0m
 
forrtl: error (78): process killed (SIGTERM)
Image              PC                Routine            Line        Source
noresm2cmor3       000000000055229A  for__signal_handl     Unknown  Unknown
libpthread-2.17.s  00002B67FC4245D0  Unknown               Unknown  Unknown
libc-2.17.so       00002B67FC6674D7  kill                  Unknown  Unknown
noresm2cmor3       00000000004E6774  Unknown               Unknown  Unknown
noresm2cmor3       00000000004E40D9  Unknown               Unknown  Unknown
noresm2cmor3       00000000004DDEF7  Unknown               Unknown  Unknown
noresm2cmor3       000000000046ABF9  Unknown               Unknown  Unknown
noresm2cmor3       00000000004661BE  Unknown               Unknown  Unknown
noresm2cmor3       00000000004BB55F  Unknown               Unknown  Unknown
noresm2cmor3       000000000040DE6E  Unknown               Unknown  Unknown
libc-2.17.so       00002B67FC6533D5  __libc_start_main     Unknown  Unknown
noresm2cmor3       000000000040DD69  Unknown               Unknown  Unknown

Usually, if restart the program again, it succeed without error.

This might directly related to the CMOR library.

Maybe no fix is need, but just report if others also met this before.

CMORized experiments to be published to ESGF (the 1st request) #32

Including the following activities:
/tos-project1/NS9034K/CMIP6/PAMIP

under which the following experiments are included:
pdSST-futArcSIC

And there is v20190920 version in these experiments.

just wondering if /projects/NS9034K/CMIP6/cmorout is still used in cmor-ization or other scripts or whether it can be removed

I tidied up the /projects/NS9034K/CMIP6 folder structure so that the regular users only see what they need to see. The changes require modification of the files mentioned in the subject. The changes are following

I renamed /projects/NS9034K/CMIP6/cmorout to /projects/NS9034K/CMIP6/.cmorout to hide this folder (also hid the “test” folder).

I temporarily placed a symlink “cmorout” that points to “.cmorout" in order to avoid disruption of CMOR-ization and ESGF download. But if cmor-izing new data please use .cmorout. Eventually I will delete the symlink “cmorout”.

I’ve update all existing symlinks to use .cmorout. I didn’t try to modify your create_CMIP6_links.sh because that one looked outdated to me. Most importantly, if creating new symlinks please use .cmorout instead of cmorout.

Next, I’ve added a “.” at the beginning of the file names of all ripf.sha256sum and ripf.map files to hide these files. When generating new sha256sum-files please use the name format .ripf.sha256sum instead of ripf.sha256sum.

I also deleted all .sha256sum files which I assumed were temporary files. It would be great if these could be deleted automatically in the future.

The following fields are now supported:

Fields in yearly output (micom.hbgcy)

'bfe           ','detoclvl          '
'co3nat        ','natco3lvl         '
'co3satarag    ','co3lvl,omegaalvl  '
'co3satcalc    ','co3lvl,omegaclvl  '
'fgco2         ','co2fxd-co2fxu     '
'fgco2nat      ','natco2fx          '
'phnat         ','natphlvl          '
'cfc11         ','cfc11lvl          ','mol m-3                      ',
'cfc12         ','cfc12lvl          ','mol m-3                      ',
'sf6           ','sf6lvl            ','mol m-3        ',

Sample cmorized fields

under /scratch/yanchun/cmorout/NorESM2-LM/historical/v20190815/Oyr

bfe_Oyr_NorESM2-LM_historical_r1i1p1f1_gr_2010-2010.nc
bsi_Oyr_NorESM2-LM_historical_r1i1p1f1_gn_2010-2010.nc
bsi_Oyr_NorESM2-LM_historical_r1i1p1f1_gr_2010-2010.nc
calc_Oyr_NorESM2-LM_historical_r1i1p1f1_gn_2010-2010.nc
calc_Oyr_NorESM2-LM_historical_r1i1p1f1_gr_2010-2010.nc
cfc11_Oyr_NorESM2-LM_historical_r1i1p1f1_gr_2010-2010.nc
cfc12_Oyr_NorESM2-LM_historical_r1i1p1f1_gr_2010-2010.nc
co3nat_Oyr_NorESM2-LM_historical_r1i1p1f1_gr_2010-2010.nc
co3_Oyr_NorESM2-LM_historical_r1i1p1f1_gn_2010-2010.nc
co3_Oyr_NorESM2-LM_historical_r1i1p1f1_gr_2010-2010.nc
co3satarag_Oyr_NorESM2-LM_historical_r1i1p1f1_gr_2010-2010.nc
co3satcalc_Oyr_NorESM2-LM_historical_r1i1p1f1_gr_2010-2010.nc
detoc_Oyr_NorESM2-LM_historical_r1i1p1f1_gn_2010-2010.nc
detoc_Oyr_NorESM2-LM_historical_r1i1p1f1_gr_2010-2010.nc
dfe_Oyr_NorESM2-LM_historical_r1i1p1f1_gn_2010-2010.nc
dfe_Oyr_NorESM2-LM_historical_r1i1p1f1_gr_2010-2010.nc
dissicnat_Oyr_NorESM2-LM_historical_r1i1p1f1_gn_2010-2010.nc
dissicnat_Oyr_NorESM2-LM_historical_r1i1p1f1_gr_2010-2010.nc
dissic_Oyr_NorESM2-LM_historical_r1i1p1f1_gn_2010-2010.nc
dissic_Oyr_NorESM2-LM_historical_r1i1p1f1_gr_2010-2010.nc
dissoc_Oyr_NorESM2-LM_historical_r1i1p1f1_gn_2010-2010.nc
dissoc_Oyr_NorESM2-LM_historical_r1i1p1f1_gr_2010-2010.nc
fgco2nat_Oyr_NorESM2-LM_historical_r1i1p1f1_gn_2010-2010.nc
fgco2_Oyr_NorESM2-LM_historical_r1i1p1f1_gn_2010-2010.nc
no3_Oyr_NorESM2-LM_historical_r1i1p1f1_gn_2010-2010.nc
no3_Oyr_NorESM2-LM_historical_r1i1p1f1_gr_2010-2010.nc
o2_Oyr_NorESM2-LM_historical_r1i1p1f1_gn_2010-2010.nc
o2_Oyr_NorESM2-LM_historical_r1i1p1f1_gr_2010-2010.nc
phnat_Oyr_NorESM2-LM_historical_r1i1p1f1_gr_2010-2010.nc
ph_Oyr_NorESM2-LM_historical_r1i1p1f1_gn_2010-2010.nc
ph_Oyr_NorESM2-LM_historical_r1i1p1f1_gr_2010-2010.nc
phyc_Oyr_NorESM2-LM_historical_r1i1p1f1_gn_2010-2010.nc
phyc_Oyr_NorESM2-LM_historical_r1i1p1f1_gr_2010-2010.nc
po4_Oyr_NorESM2-LM_historical_r1i1p1f1_gn_2010-2010.nc
po4_Oyr_NorESM2-LM_historical_r1i1p1f1_gr_2010-2010.nc
pp_Oyr_NorESM2-LM_historical_r1i1p1f1_gn_2010-2010.nc
pp_Oyr_NorESM2-LM_historical_r1i1p1f1_gr_2010-2010.nc
sf6_Oyr_NorESM2-LM_historical_r1i1p1f1_gr_2010-2010.nc
si_Oyr_NorESM2-LM_historical_r1i1p1f1_gn_2010-2010.nc
si_Oyr_NorESM2-LM_historical_r1i1p1f1_gr_2010-2010.nc
talknat_Oyr_NorESM2-LM_historical_r1i1p1f1_gn_2010-2010.nc
talknat_Oyr_NorESM2-LM_historical_r1i1p1f1_gr_2010-2010.nc
talk_Oyr_NorESM2-LM_historical_r1i1p1f1_gn_2010-2010.nc
talk_Oyr_NorESM2-LM_historical_r1i1p1f1_gr_2010-2010.nc
zooc_Oyr_NorESM2-LM_historical_r1i1p1f1_gn_2010-2010.nc
zooc_Oyr_NorESM2-LM_historical_r1i1p1f1_gr_2010-2010.nc

commit:

3ec327b

Mandatory information:

Full path to the case(s) of the experiment on NIRD
/projects/NS4659K/chuncheng/cases/

experiment_id
lig127k

CASENAME(s) and years to be CMORized
NBF1850OC_f19_tn11_01_127ka, 1501-1700

Optional information

none

parent_experiment_id
no parent

parent_experiment_rip
r1i1p1

parent_time_units
no parent

branch_method
none

other information
Q: For the PI run with NorESM1-F, we have 1000 years spin-up + another 1000 years PIcontrol. This lig127k run was branched off from the PI spin-up (not PI control) at year 1000. So I guess in this case lig127k has no parent, and the branch method should be none, rather than 'Hybrid-restart from year xxxx of piControl'?

The first RFMIP-SpAer experiment piClim-SpAer-aer is finalized with output stored on nird at

/projects/NS9560K/noresm/cases/piClim-SpAer2014

The MIP leader request CMORized data as quickly as possible (orignal deadline was end of September), CMORIzed, but not necessarily put on ESGF. Could this be given some priority?
Although the upcoming historical counterpart is more important (Tier 1), this grants us contribution in an important manuscript for IPCC AR6. This is separate from the main/standard RFMIP study/manuscript.

Since the aerosol optics and cloud droplet number concentrations (CDNC) are s mix between CAM6-Oslo (for 1850 aerosol emissions) and prescribed optical properties and CDNC (anthropogenic contribution after 1850 is prescribed based on the MACv2-SP code from Stevens et al.), there is some additional specialized output (like instantaneous rdiative forcing) which is calculated only for this model set-up (run with define SPAERO), which I cannot even find listed in the CMIP6 data request google sheet. But I suggest to run CMOR as normal, and handle these specialized output separately, when we have more information about what and on what form(at) exactly this should be submitted. If someone knows anything about the details of this (Yanchun?). please let me (Alf) know.

The json-namelists for individual fields are not all written when using noresm2cmor3_mpi. This causes cmor-ization to halt if the namelists do not already exist.

Mandatory information:

Full path to the case(s) of the experiment on NIRD
/projects/NS9560K/noresm/cases/NHIST_f19_tn14_20190909esm

experiment_id
esm-hist

CASENAME(s) and years to be CMORized
NHIST_f19_tn14_20190909esm, 1850-2014

Optional information

(additional information, if the experiment is branched/hybrid restart from previous parent experiment; you may find more information relevant to the experiment_id here: https://github.com/NorwegianClimateCentre/noresm2cmor/blob/master/tables/CMIP6_CV.json)

parent_experiment_id
esm-piControl

parent_experiment_rip
e.g., r1i1p1

parent_time_units
'days since 1851-01-01'

branch_method
'Hybrid-restart from year 1851-01-01 of esm-piControl',

other information
(provide other information that might be useful)
Prognostic atmospheric CO2 simulation

(A. Gjermundsen)
the cmorized output in /projects/NS9034K/CMIP6/cmorout/NorESM2-LM/ and I have a question regarding the load output. In both historical and piControl the output is missing January. All the files are from February to December. It is not a name error, because the files are shorter compared to others. E.g.:
ls /projects/NS9034K/CMIP6/cmorout/NorESM2-LM/historical/v20190815/loaddust_Emon_NorESM2-LM_historical_r1i1p1f1_gn_185002-185912.nc

Is there a reason or just a bug?

if compare with e.g. CESM2 the output is monthly (it is the same for all the other models I have looked at):

(base) [adagj@login-nird-3 piControl]$ ncdump -h loaddust_Emon_CESM2_piControl_r1i1p1f1_gn_000101-009912.nc 
netcdf loaddust_Emon_CESM2_piControl_r1i1p1f1_gn_000101-009912 {
dimensions:
lat = 192 ;
nbnd = 2 ;
time = UNLIMITED ; // (1188 currently)
lon = 288 ;
variables:
double lat(lat) ;
lat:axis = "Y" ;
lat:bounds = "lat_bnds" ;
lat:standard_name = "latitude" ;
lat:title = "Latitude" ;
lat:type = "double" ;
lat:units = "degrees_north" ;
lat:valid_max = 90. ;
lat:valid_min = -90. ;
float lat_bnds(lat, nbnd) ;
lat_bnds:units = "degrees_north" ;
float loaddust(time, lat, lon) ;
loaddust:_FillValue = 1.e+20f ;
loaddust:cell_measures = "area: areacella" ;
loaddust:cell_methods = "area: time: mean" ;
loaddust:comment = "The total dry mass of dust aerosol particles per unit area." ;
loaddust:coordinates = "time lat lon" ;
loaddust:deflate = "0" ;
loaddust:deflate_level = "0" ;
loaddust:description = "The total dry mass of dust aerosol particles per unit area." ;
loaddust:frequency = "mon" ;
loaddust:id = "loaddust" ;
loaddust:long_name = "Load of Dust" ;
loaddust:mipTable = "Emon" ;
loaddust:missing_value = 1.e+20 ;
loaddust:out_name = "loaddust" ;
loaddust:prov = "PMIP [PMIP-aero-02]" ;
loaddust:realm = "atmos" ;
loaddust:shuffle = "0" ;
loaddust:standard_name = "atmosphere_mass_content_of_dust_dry_aerosol_particles" ;
loaddust:time = "time" ;
loaddust:time_label = "time-mean" ;
loaddust:time_title = "Temporal mean" ;
loaddust:title = "Load of Dust" ;
loaddust:type = "real" ;
loaddust:units = "kg m-2" ;
loaddust:variable_id = "loaddust" ;
double lon(lon) ;
lon:axis = "X" ;
lon:bounds = "lon_bnds" ;
lon:standard_name = "longitude" ;
lon:title = "Longitude" ;
lon:type = "double" ;
lon:units = "degrees_east" ;
lon:valid_max = 360. ;
lon:valid_min = 0. ;
float lon_bnds(lon, nbnd) ;
lon_bnds:units = "degrees_east" ;
double time(time) ;
time:axis = "T" ;
time:bounds = "time_bnds" ;
time:standard_name = "time" ;
time:title = "time" ;
time:type = "double" ;
time:units = "days since 0001-01-01 00:00:00" ;
time:calendar = "noleap" ;
double time_bnds(time, nbnd) ;
time_bnds:calendar = "noleap" ;
time_bnds:units = "days since 0001-01-01 00:00:00" ;

Mandatory information:

Full path to the case(s) of the experiment on NIRD
/projects/NS2345K/noresm/cases

experiment_id
piControl

CASENAME(s) and years to be CMORized
NBF1850OC_f19_tn11_02, 1501-1700

Optional information

none

parent_experiment_id
no parent

parent_experiment_rip
r1i1p1

parent_time_units
no parent

branch_method
none

other information
none

The Following fields are now supported:

Fields in 3hr-averaged output (cam.h3cam.h4,clm.h2)

'mrro      ','QRUNOFF
'mrsos     ','SOILWATER_10CM
'tslsi     ','TSA

Fields in 3hr-instantly output (cam.h4cam.h5, clm.h3):

all not available

The Following fields are not available in the model output:

'uas       ','UAS
'vas       ','VAS
'clt       ','CLDTOT
'hfls      ','LHFLX
'hfss      ','SHFLX
'pr        ','PRECT
'prc       ','PRECC
'prsn      ','PRECSC+PRECSL
'rlds      ','FLDS
'rldscs    ','FLDSC
'rlus      ','FLDS+FLNS
'rsds      ','FSDS
'rsdscs    ','FSDSC
'rsdsdiff  ','SOLLD+SOLSD
'rsus      ','FSDS-FSNS
'rsuscs    ','FSDSC-FSNSC

Sample cmorized fields

under /scratch/yanchun/cmorout/NorESM2-LM/historical/v20190815/3hr

mrro_3hr_NorESM2-LM_historical_r1i1p1f1_gn_201001010130-201001312230.nc
mrsos_3hr_NorESM2-LM_historical_r1i1p1f1_gn_201001010130-201001312230.nc
tslsi_3hr_NorESM2-LM_historical_r1i1p1f1_gn_201001010130-201001312230.nc

Commit

ca5b932
47a304d

UPDATED: 29 Oct 2019, corrected file tags, so that availability of cmorized files are also changed.

Fixed bug that caused program to stop when input files for certain frequencies were not present (the frequencies are skipped now) in commit NorwegianClimateCentre@ebe4eec

(by Jan Griesfeller)

I just downloaded some AOD data from ESGF (var name od550aer) of some models including NorESM and noted a slight difference in the naming scheme. The files are also on NIRD in the /nird/projects/NS9252K/CMIP6/raw/ directory.

It seems we use different naming scheme:
NorESM:
od550aer_AERmon_historical_NorESM2-LM_r1i1p1f1_gn_185001-185912.nc
all other models:
od550aer_AERmon_CESM2-WACCM_historical_r1i1p1f1_gn_185001-201412.nc

The difference is that the other models put the experiment_id
after the model name while we have it the other way round.

I just wonder if we did something wrong or not, but since I am supposed to write a converter to the aerocom naming scheme, this difference causes some additional work.

Any opinions on that?

A script is needed that creates QC reports for groups of cmorized files (per experiment and version) and places these reports together with the files e.g. /projects/NS9034K/CMIP6/.cmorout/NorESM2-LM/historical/v20190810.QCreport

PrePARE INSTALLATION NOTES:

Installation on NIRD has been done with
export PATH=/opt/anaconda3/bin:${PATH}
conda create -c conda-forge cmor --prefix /projects/NS9560K/cmor/PrePARE

PrePARE USAGE:

Before use, do
export PATH=/opt/anaconda3/bin:${PATH}
source activate /projects/NS9560K/cmor/PrePARE
export CMIP6_CMOR_TABLES=/projects/NS9560K/cmor/noresm2cmor/tables

These steps are done automatically if using the script /projects/NS9560K/cmor/PrePARE/bin/PrePARE_wrapper
instead of calling directly PrePARE.

To use PrePARE, call the command and specify as argument the path to a cmor-sized NetCDF file, e.g.
PrePARE /projects/NS9034K/CMIP6/.cmorout/NorCPM1/piControl/v20190914/clt_Amon_NorCPM1_piControl_r1i1p1f1_gn_000101-010012.nc
PrePARE /projects/NS9034K/CMIP6/.cmorout/NorESM2-LM/historical/v20190810/wap_Emon_historical_NorESM2-LM_r1i1p1f1_gn_185002-185912.nc
or a directory that contains the NetCDF files, e.g.
PrePARE /projects/NS9034K/CMIP6/.cmorout/NorESM2-LM/historical/v20190810/

Experiment(s) ready to be published to ESGF:

data path
/projects/NS9034K/CMIP6/RFMIP/NCC/NorESM2-LM/piClim-spAer-aer/r1i1p1f1

activity_id:

• RFMIP

experiment_id

• piClim-spAer-aer

Versions

• v20190920

other information

• PrePARE quality-check, passed
• sha256sum, calculated

Including the following activities:

• /tos-project1/NS9034K/CMIP6/PAMIP

under which the following experiments are included:

• pdSST-futArcSIC
• pdSST-pdSST

And there is date version in these experiments:

• v20190920

MPI-parallel cmor-ization works as of commit NorwegianClimateCentre@ebe4eec

Example for usage:
export I_MPI_WAIT_MODE=1
mpirun -n 10 ./noresm2cmor3_mpi sys.nml mod.nml exp.nml var.nml

Bechmark results for (1 yr model output, 167 files, 3.4 GB):
serial - 16m20s
parallel 10 tasks - 2m22 (~7 times faster than serial)

test code in noresm2cmor/devel/get_timecoord

CALL SYSTEM('ls -l /tmp/') in m_namelist
is this still useful?
otherwise, it dumps out the file list very often in the log file.

When running PrePARE report error for all the cmorized files of PAMIP experiments, e.g, pdSST-pdSIC, pdSST-futArcSIC

$ PrePARE_wrapper /tos-project1/NS9034K/CMIP6/.cmorout/NorESM2-LM/pdSST-pdSIC/v20190920/zg_Amon_NorESM2-LM_pdSST-pdSIC_r98i1p1f1_gn_200006-200105.nc
list index out of range
└──> :: SKIPPED    :: /tos-project1/NS9034K/CMIP6/.cmorout/NorESM2-LM/pdSST-pdSIC/v20190920/zg_Amon_NorESM2-LM_pdSST-pdSIC_r98i1p1f1_gn_200006-200105.nc

Number of files scanned: 1
Number of file with error(s): 1

Any ideas what is wrong?

In OMIP experiment,
LAT/LON data look right.

$ ncdump -v TLAT /projects/NS9560K/noresm/cases/N1850_f19_tn14_20190621/ice/hist/N1850_f19_tn14_20190621.cice.h.1600-01.nc
netcdf N1850_f19_tn14_20190621.cice.h.1600-01 {
dimensions:
        d2 = 2 ;
        ni = 360 ;
        nj = 384 ;
        nc = 5 ;
        nkice = 8 ;
        nksnow = 3 ;
        nkbio = 5 ;
        time = UNLIMITED ; // (1 currently)
        nvertices = 4 ;
variables:
        float time(time) ;
                time:long_name = "model time" ;
                time:units = "days since 0001-01-01 00:00:00" ;
                time:calendar = "noleap" ;
                time:bounds = "time_bounds" ;
        float time_bounds(time, d2) ;
                time_bounds:long_name = "boundaries for time-averaging interval" ;
                time_bounds:units = "days since 0001-01-01 00:00:00" ;
        float TLON(nj, ni) ;
                TLON:long_name = "T grid center longitude" ;
                TLON:units = "degrees_east" ;
                TLON:missing_value = 1.e+30f ;
                TLON:_FillValue = 1.e+30f ;
                TLON:bounds = "lont_bounds" ;
        float TLAT(nj, ni) ;
                TLAT:long_name = "T grid center latitude" ;
                TLAT:units = "degrees_north" ;
                TLAT:missing_value = 1.e+30f ;
                TLAT:_FillValue = 1.e+30f ;
                TLAT:bounds = "latt_bounds" ;
...
...
    76.66061, 76.33442, 76.00912, 75.68472, 75.36127, 75.0388, 74.71732,
    74.39686, 74.07745, 73.75912, 73.44192, 73.12584, 72.81094, 72.49724,
    72.18477, _, _, _, _, _, _, _, _, _, _, _, _, _, _, _, _, _, _, _, _, _,
    _, _, _, _, _, _, _, _, _, _, _, _, _, _, _ ;
}

But in cmorized output, for example,

$ncdump -v latitude /scratch/yanchun/cmorout/NorESM2-LM/omip1/v20190920/siage_SImon_NorESM2-LM_omip1_r1i1p1f1_gn_170001-170012.nc
netcdf siage_SImon_NorESM2-LM_omip1_r1i1p1f1_gn_170001-170012 {
dimensions:
        time = UNLIMITED ; // (12 currently)
        j = 384 ;
        i = 360 ;
        bnds = 2 ;
        vertices = 4 ;
variables:
        double time(time) ;
                time:bounds = "time_bnds" ;
                time:units = "days since 0001-01-01 00:00:00" ;
                time:calendar = "noleap" ;
                time:axis = "T" ;
                time:long_name = "time" ;
                time:standard_name = "time" ;
        double time_bnds(time, bnds) ;
        int j(j) ;
                j:units = "1" ;
                j:long_name = "cell index along second dimension" ;
        int i(i) ;
                i:units = "1" ;
                i:long_name = "cell index along first dimension" ;
        double latitude(j, i) ;
                latitude:standard_name = "latitude" ;
                latitude:long_name = "latitude" ;
                latitude:units = "degrees_north" ;
                latitude:missing_value = 1.e+20 ;
                latitude:_FillValue = 1.e+20 ;
                latitude:bounds = "vertices_latitude" ;
        double longitude(j, i) ;
                longitude:standard_name = "longitude" ;
                longitude:long_name = "longitude" ;
                longitude:units = "degrees_east" ;
                longitude:missing_value = 1.e+20 ;
                longitude:_FillValue = 1.e+20 ;
                longitude:bounds = "vertices_longitude" ;
...
    67.3815460205078, 67.0956344604492, 66.8116760253906, 66.5297241210938,
    66.2498397827148, 65.9720764160156, 1.00000001504747e+30,
    1.00000001504747e+30, 1.00000001504747e+30, 1.00000001504747e+30,
    1.00000001504747e+30, 1.00000001504747e+30, 1.00000001504747e+30,
    1.00000001504747e+30, 1.00000001504747e+30, 1.00000001504747e+30,
    1.00000001504747e+30, 1.00000001504747e+30, 1.00000001504747e+30,
    1.00000001504747e+30, 1.00000001504747e+30 ;
}

The _FillValue/missing value are not recognised correctly.

I've been trying to look more into how to get some the Eliassen-Palm (EP) variables that are needed for the CMIP6 data request, that is, epfy (northward component of the EP flux), epfz (upward component of the EP flux), utendepfd (tendency of eastward wind due to EP flux divergence), vtem (transformed Eulerian mean northward wind component), and wtem (transformed Eulerian mean upward wind component).

These variables are priority 1 for the PAMIP, DynVarMip, and DAMIP simulations, and I have reason to believe that they will be key variables in the analysis of the PAMIP simulations. I've been in contact with a few different people to learn how they are computing the variables at NCAR, and they do it through two steps:
by setting the CAM namelist variable do_circulation_diags to .true., the CESM2 will use ctem.F90 to compute the meridional and vertical heat flux (VTHzm and WTHzm), the meridional and vertical flux og zonal momentum (UVzm and UWzm), and the zonal-mean zonal, meridional, and vertical wind, and the potential temperature (Uzm, Vzm, Wzm, and THzm)
then when the simulation is finished UTHzm, WTHzm, UVzm, UWzm, Uzm, Vzm, Wzm, and THzm are used to compute epfy, epfz, utendepfd, vtem, and wtem as a part of the post-processing.
I have tested that step 1 works for NorESM2, and I am able to output the necessary variables, so it seems OK.

Adam Philips at NCAR sent me the following links to the python script they use for the second step:
https://github.com/NCAR/PyConform/blob/devel/source/pyconform/modules/idl.py
https://github.com/NCAR/PyConform/blob/devel/source/pyconform/modules/dynvarmipdiags.py
https://github.com/NCAR/PyConform/blob/devel/source/pyconform/modules/dynvarmipfunctions.py

Do you think it would be possible to include the second step as a part of the CMOR-ization process of the NorESM output?
(by L.Graff)

update: 20200928
commit:
93d775f
76c7c6e

When an experiment has no parent, the branch_time_in_parent should be omitted (or set to 0.0D0).

CALL json%add(p,'branch_time_in_parent',branch_time_in_parent)

write the value of branch_time_in_parent in *.json file as 0.0E0

However, in CMOR library Src/cmor_CV.c

            if (cmor_has_cur_dataset_attribute(BRANCH_TIME_IN_PARENT) == 0) {
                cmor_get_cur_dataset_attribute(BRANCH_TIME_IN_PARENT, szValue);
                if (strcmp(szValue, "0.0") != 0) {
                    snprintf(msg, CMOR_MAX_STRING,
                             "Your input attribute %s %s \n! "
                             "has been replaced with 0.0 \n! ",
                             BRANCH_TIME_IN_PARENT, szValue);
                    cmor_set_cur_dataset_attribute_internal
                      (BRANCH_TIME_IN_PARENT, "0.0", 1);

                    cmor_handle_error(msg, CMOR_WARNING);
                }
            }

It is compared is the value of the string branch_time_in_parent read from the json file equals "0.0" instead of 0.0E0.

As a result, noresm2cmor tool (more accurate CMOR library), will report a warning.

@IngoBethke Could you change the format of the branch_time_in_parent when it is written to json file? That is to say, change from e.g, ES6.1E1 to F3.1?

I don't find where to change this in the json_*.F90 files.