Comments (18)
adagj
commented on August 10, 2024
1
Hi @YanchunHe
I would expect more output for r2i1p1f1, r3i1p1f1, r4i1p1f1 becouse they are run with extensive aerosol output. But I would expect the same amount of output for all years and all 3. And the same for the remaining r5 - r10.
I will have a look and let you know if I find something.
Best,
Ada
from noresm2cmor.
YanchunHe
commented on August 10, 2024
1
Thanks @YanchunHe for checking. That's not good at all... I'll do some more digging, maybe I need to rerun some parts
Yes, would be nice to check first if all years have the same number of output files.
from noresm2cmor.
YanchunHe
commented on August 10, 2024
I made some corrections to the experiment configuration above.
This applies to issue #328, #329, #330, #331, #332, #333, #334, #334
from noresm2cmor.
YanchunHe
commented on August 10, 2024
Hi @adagj
I see the number of cmorized datasets (variables) are slightly different from different realisations and periods. Is that expected?
For example,
- 529 different datasets during 1850-1959 (or 1949), and 757 datasets during 1960-2014 for r2i1p1f1.
- 529 datasets for all the periods of r3i1p1f1.
- 440 datasets for all (not finished yet) periods of r5i1p1f1.
- 486 datasets for some other realisations.
from noresm2cmor.
YanchunHe
commented on August 10, 2024
Seems like there are something unusual with time axies the yearly hamocc output. So that, for example:
cd /projects/NS9034K/CMIP6/.cmorout/NorESM2-LM/esm-hist/v20230616
$ ls -1 bfe_Oyr*r2i1p1f1*.nc
bfe_Oyr_NorESM2-LM_esm-hist_r2i1p1f1_gr_1861-1861.nc
bfe_Oyr_NorESM2-LM_esm-hist_r2i1p1f1_gr_1874-1874.nc
bfe_Oyr_NorESM2-LM_esm-hist_r2i1p1f1_gr_1899-1899.nc
bfe_Oyr_NorESM2-LM_esm-hist_r2i1p1f1_gr_1910-1910.nc
bfe_Oyr_NorESM2-LM_esm-hist_r2i1p1f1_gr_1923-1923.nc
bfe_Oyr_NorESM2-LM_esm-hist_r2i1p1f1_gr_1946-1946.nc
bfe_Oyr_NorESM2-LM_esm-hist_r2i1p1f1_gr_1951-1957.nc
bfe_Oyr_NorESM2-LM_esm-hist_r2i1p1f1_gr_1960-1969.nc
bfe_Oyr_NorESM2-LM_esm-hist_r2i1p1f1_gr_1971-1979.nc
bfe_Oyr_NorESM2-LM_esm-hist_r2i1p1f1_gr_1980-1989.nc
bfe_Oyr_NorESM2-LM_esm-hist_r2i1p1f1_gr_1990-1999.nc
bfe_Oyr_NorESM2-LM_esm-hist_r2i1p1f1_gr_2000-2009.nc
bfe_Oyr_NorESM2-LM_esm-hist_r2i1p1f1_gr_2010-2014.nc
Those from the ~1960/1970 of Oyr datasets do not include the 10 years record, as expected.
I have not checked the raw output though.
from noresm2cmor.
adagj
commented on August 10, 2024
@YanchunHe I'm not sure which files you use for the cmorization, but seems like all the files are in the case folder, but maybe something happen while archiving... can you try to rerun the cmorization just the first 30 years? Will that work you think?
from noresm2cmor.
YanchunHe
commented on August 10, 2024
For example:
yanchun@ipcc:/projects/NS10013K/noresm/cases/NHIST_1901_f19_tn14_20230201esm/ocn/hist
$ ls *hbgcy.*
NHIST_1901_f19_tn14_20230201esm.blom.hbgcy.1861.nc
NHIST_1901_f19_tn14_20230201esm.blom.hbgcy.1874.nc
NHIST_1901_f19_tn14_20230201esm.blom.hbgcy.1899.nc
NHIST_1901_f19_tn14_20230201esm.blom.hbgcy.1910.nc
NHIST_1901_f19_tn14_20230201esm.blom.hbgcy.1923.nc
NHIST_1901_f19_tn14_20230201esm.blom.hbgcy.1946.nc
NHIST_1901_f19_tn14_20230201esm.blom.hbgcy.1951.nc
NHIST_1901_f19_tn14_20230201esm.blom.hbgcy.1952.nc
NHIST_1901_f19_tn14_20230201esm.blom.hbgcy.1953.nc
NHIST_1901_f19_tn14_20230201esm.blom.hbgcy.1957.nc
NHIST_1901_f19_tn14_20230201esm.blom.hbgcy.1960.nc
NHIST_1901_f19_tn14_20230201esm.blom.hbgcy.1963.nc
NHIST_1901_f19_tn14_20230201esm.blom.hbgcy.1967.nc
NHIST_1901_f19_tn14_20230201esm.blom.hbgcy.1969.nc
NHIST_1901_f19_tn14_20230201esm.blom.hbgcy.1971.nc
NHIST_1901_f19_tn14_20230201esm.blom.hbgcy.1972.nc
NHIST_1901_f19_tn14_20230201esm.blom.hbgcy.1975.nc
NHIST_1901_f19_tn14_20230201esm.blom.hbgcy.1976.nc
NHIST_1901_f19_tn14_20230201esm.blom.hbgcy.1977.nc
NHIST_1901_f19_tn14_20230201esm.blom.hbgcy.1978.nc
NHIST_1901_f19_tn14_20230201esm.blom.hbgcy.1979.nc
...
Seems many years are missing.
Could you check other 'realisations' too?
from noresm2cmor.
adagj
commented on August 10, 2024
Thanks @YanchunHe for checking. That's not good at all... I'll do some more digging, maybe I need to rerun some parts
from noresm2cmor.
adagj
commented on August 10, 2024
I made some corrections to the experiment configuration above.
This applies to issue #328, #329, #330, #331, #332, #333, #334, #334
Hi,
N1850_f19_tn14_20190730esm is stored in two places; NS2345K (years 1850-2200) and NS9560K (years 2200-2310)
from noresm2cmor.
adagj
commented on August 10, 2024
@YanchunHe
I think it is easier to have all the information in one issue, so I started a new one here: #348
from noresm2cmor.
YanchunHe
commented on August 10, 2024
CMORized, and ready to be published to ESGF.
There are more files/datasets for years from 1980-2014.
529 datasets for every 10 years before 1980, and 575 afterwards.
data path
- /projects/NS9034K/CMIP6/.cmorout/NorESM2-LM/esm-hist
- /projects/NS9034K/CMIP6/CMIP/NCC/NorESM2-LM/esm-hist
version
- v20230616
sha256sum
/projects/NS9034K/CMIP6/CMIP/NCC/NorESM2-LM/esm-hist
- .r2i1p1f1.sha256sum_v20230616
from noresm2cmor.
monsieuralok
commented on August 10, 2024
@YanchunHe are r2 to r10 realisation having different number of datasets? as sha256sum files differs a lot quite in size as .r4i1p1f1_20231003.map 3158102 bytes while .r5i1p1f1_20231003.map 2625378 bytes.
from noresm2cmor.
YanchunHe
commented on August 10, 2024
corrected: Here are the corrected list of number of files per realisation:
variant: r2i1p1f1
Ofx, fx, etc 10
yyyy1 yyyy2 nf
1850 1859 529
1860 1869 529
1870 1879 529
1880 1889 529
1890 1899 529
1900 1909 529
1910 1919 529
1920 1929 529
1930 1939 529
1940 1949 529
1950 1959 529
1960 1969 529
1970 1979 529
1980 1989 575
1990 1999 575
2000 2009 575
2010 2014 575
Total: 9187
Total r2i1p1f1: 9187
Shasum lines: 9187
variant: r3i1p1f1
Ofx, fx, etc 10
yyyy1 yyyy2 nf
1850 1859 529
1860 1869 529
1870 1879 529
1880 1889 529
1890 1899 529
1900 1909 529
1910 1919 529
1920 1929 529
1930 1939 529
1940 1949 529
1950 1959 529
1960 1969 529
1970 1979 529
1980 1989 529
1990 1999 529
2000 2009 529
2010 2014 529
Total: 9003
Total r3i1p1f1: 9003
Shasum lines: 9003
variant: r4i1p1f1
Ofx, fx, etc 10
yyyy1 yyyy2 nf
1850 1859 529
1860 1869 529
1870 1879 529
1880 1889 529
1890 1899 529
1900 1909 529
1910 1919 529
1920 1929 529
1930 1939 529
1940 1949 529
1950 1959 529
1960 1969 529
1970 1979 529
1980 1989 529
1990 1999 529
2000 2009 529
2010 2014 529
Total: 9003
Total r4i1p1f1: 9003
Shasum lines: 9003
variant: r5i1p1f1
Ofx, fx, etc 10
yyyy1 yyyy2 nf
1850 1859 440
1860 1869 440
1870 1879 440
1880 1889 440
1890 1899 440
1900 1909 440
1910 1919 440
1920 1929 440
1930 1939 440
1940 1949 440
1950 1959 440
1960 1969 440
1970 1979 440
1980 1989 440
1990 1999 440
2000 2009 440
2010 2014 440
Total: 7490
Total r5i1p1f1: 7490
Shasum lines: 7490
variant: r6i1p1f1
Ofx, fx, etc 10
yyyy1 yyyy2 nf
1850 1859 486
1860 1869 486
1870 1879 486
1880 1889 486
1890 1899 486
1900 1909 486
1910 1919 486
1920 1929 486
1930 1939 486
1940 1949 486
1950 1959 486
1960 1969 486
1970 1979 486
1980 1989 486
1990 1999 486
2000 2009 486
2010 2014 486
Total: 8272
Total r6i1p1f1: 8272
Shasum lines: 8272
variant: r7i1p1f1
Ofx, fx, etc 10
yyyy1 yyyy2 nf
1850 1859 486
1860 1869 486
1870 1879 486
1880 1889 486
1890 1899 486
1900 1909 486
1910 1919 486
1920 1929 486
1930 1939 486
1940 1949 486
1950 1959 486
1960 1969 486
1970 1979 486
1980 1989 486
1990 1999 486
2000 2009 486
2010 2014 486
Total: 8272
Total r7i1p1f1: 8272
Shasum lines: 8272
variant: r8i1p1f1
Ofx, fx, etc 10
yyyy1 yyyy2 nf
1850 1859 486
1860 1869 486
1870 1879 486
1880 1889 486
1890 1899 486
1900 1909 486
1910 1919 486
1920 1929 486
1930 1939 486
1940 1949 486
1950 1959 486
1960 1969 486
1970 1979 486
1980 1989 486
1990 1999 486
2000 2009 486
2010 2014 486
Total: 8272
Total r8i1p1f1: 8272
Shasum lines: 8272
variant: r9i1p1f1
Ofx, fx, etc 10
yyyy1 yyyy2 nf
1850 1859 486
1860 1869 486
1870 1879 486
1880 1889 486
1890 1899 486
1900 1909 486
1910 1919 486
1920 1929 486
1930 1939 486
1940 1949 486
1950 1959 486
1960 1969 486
1970 1979 486
1980 1989 486
1990 1999 486
2000 2009 486
2010 2014 486
Total: 8272
Total r9i1p1f1: 8272
Shasum lines: 8272
variant: r10i1p1f1
Ofx, fx, etc 10
yyyy1 yyyy2 nf
1850 1859 486
1860 1869 486
1870 1879 486
1880 1889 486
1890 1899 486
1900 1909 486
1910 1919 486
1920 1929 486
1930 1939 486
1940 1949 486
1950 1959 486
1960 1969 486
1970 1979 486
1980 1989 486
1990 1999 486
2000 2009 486
2010 2014 486
Total: 8272
Total r10i1p1f1: 8272
Shasum lines: 8272
from noresm2cmor.
monsieuralok
commented on August 10, 2024
@YanchunHe I am mentioning number of lines from nird:
.r2i1p1f1.sha256sum_v20230616 - 9187
.r3i1p1f1.sha256sum_v20230616 - 9003
.r4i1p1f1.sha256sum_v20230616 - 9003
.r5i1p1f1.sha256sum_v20230616 - 7490
.r6i1p1f1.sha256sum_v20230616 - 8272
.r7i1p1f1.sha256sum_v20230616 - 8272
.r8i1p1f1.sha256sum_v20230616 - 8272
.r9i1p1f1.sha256sum_v20230616 - 8272
.r10i1p1f1.sha256sum_v20230616 - 8272
which are different what you mentioned.
from noresm2cmor.
YanchunHe
commented on August 10, 2024
OK, I will check.
from noresm2cmor.
YanchunHe
commented on August 10, 2024
I past the wrong number of files.. just in the above thread.
I will update soon.
from noresm2cmor.
YanchunHe
commented on August 10, 2024
@YanchunHe I am mentioning number of lines from nird:
.r2i1p1f1.sha256sum_v20230616 - 9187 .r3i1p1f1.sha256sum_v20230616 - 9003 .r4i1p1f1.sha256sum_v20230616 - 9003 .r5i1p1f1.sha256sum_v20230616 - 7490 .r6i1p1f1.sha256sum_v20230616 - 8272 .r7i1p1f1.sha256sum_v20230616 - 8272 .r8i1p1f1.sha256sum_v20230616 - 8272 .r9i1p1f1.sha256sum_v20230616 - 8272 .r10i1p1f1.sha256sum_v20230616 - 8272
which are different what you mentioned.
So, these are right numbers.
I don't see anything wrong.
Please clarify me if you have more in doubt.
Thanks:-)
from noresm2cmor.
monsieuralok
commented on August 10, 2024
@YanchunHe published
from noresm2cmor.
Related Issues (20)
- [CMIP6 CMOR-ization & ESGF-publication] NorESM2-LM - esm-ssp245 HOT 4
- [CMIP6 CMOR-ization & ESGF-publication] NorESM2-LM - esm-hist HOT 3
- [CMIP6 CMOR-ization & ESGF-publication] NorESM2-LM - esm-ssp534-over HOT 5
- [CMIP6 CMOR-ization & ESGF-publication] NorESM2-LM - hist-aer HOT 1
- [CMORization request] esm-ssp119 HOT 3
- [CMORization request] esm-ssp245 HOT 1
- [CMIP6 CMOR-ization & ESGF-publication] NorESM2-LM - esm-ssp585-ocn-alk HOT 2
- [CMIP6 CMOR-ization & ESGF-publication] NorESM2-LM new physics,piClim-Control HOT 2
- CMIP6 CMOR-ization & ESGF-publication] NorESM2-LM new physics piClim-aer HOT 8
- [CMIP6 CMOR-ization & ESGF-publication] NorESM2-LM new physics histSST HOT 2
- [CMIP6 CMOR-ization & ESGF-publication] NorESM2-LM new physics histSST-piAer HOT 10
- [CMIP6 CMOR-ization] NorESM2-LM new physics ssp245SST-piAer HOT 1
- [CMIP6 CMOR-ization] NorESM2-LM new physics ssp245SST HOT 1
- [CMORization request] NorESM2-LM LRTAP-base HOT 1
- [CMORization request] NorESM2-LM LRTAP-mtfr HOT 1
- [CMORization request] NorESM2-LM LRTAP-low HOT 1
- Documentation for how to build noresm2cmor contains an empty link HOT 4
- [ERROR] NorESM 1pct co2mass data incorrect HOT 1
- [CMIP6 CMOR-ization & ESGF-publication] NorESM2-LM - esm-ssp585-ocn-alk (ensemble members 2 and 3) HOT 6
- [ERROR] co2mass in emission driven runs HOT 3
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from noresm2cmor.