Append mode about cmor3_documentation HOT 10 CLOSED

I am looking at file_suffix code and hopefully, I can get that to work soon. I don't remember changing that code, so it must be a something I introduced.

I am not aware of the problem with append-mode. I will try to create a test to reproduce this. If you can provide some of your code using the cmor.write() function that will help me investigate it.

Thanks!

from cmor3_documentation.

After registering all variables and axes I am looping over all time steps to write data of each variable however the error also occurs when not looping.
cmor_write(vars[i].cmor_varID, vars[i].data, vars[i].datatype, chunk_files[i], 1, &time_val, time_bndsp, NULL);
The chunk_files[i] parameter is gone in CMOR3. If a zfactor is needed for this variable, I call cmor_write a second time for that time step:
if ( vars[i].zfactor_id > 0 ) cmor_write(vars[i].zfactor_id, vars[ps_index].data, vars[ps_index].datatype, chunk_files[i], 1, &time_val, time_bndsp, &vars[i].cmor_varID);

This works well for all variables (for now) but not in append mode for variables with zfactors. In append mode, the first time_val of the variable which is to append on the chunk is lost in the output - either an Error occurs if the time_val does not fit in the time series or a warning occurs.
Thanks for the quick answer.

from cmor3_documentation.

I will get to it this week, and let you know by Friday. Thanks for the code....

from cmor3_documentation.

Hi Denis,
may I draw your attention to this issue again? I still struggle with the APPEND_MODE. In cmor.c we have

        ierr = mkstemp(outname);
        unlink(outname);

and

/* -------------------------------------------------------------------- */
/*      Decides NetCDF mode                                             */
/* -------------------------------------------------------------------- */
        ncid = cmor_validateFilename(outname, var_id);

In validateFilename it is:

        fperr = fopen(outname, "r");
        if (fperr == NULL) {

/* -------------------------------------------------------------------- */
/*      ok it does not exists... we will open as new                    */
/* -------------------------------------------------------------------- */
            ierr = nc_create(outname, NC_CLOBBER | cmode, &ncid);

        } else {                /*ok it was there already */

But how can it be there already if mkstemp is used? And how can I use NetCDF APPEND_MODE then? How can I pass the chunk to append on?
Best regards,
Fabi

from cmor3_documentation.

It seems that this functionality was stripped when converting to 3.0.0. I will work on it today and will create a nightly version for you to try.

from cmor3_documentation.

I added back this functionality, you can give it a try using the following conda file.

https://anaconda.org/PCMDI/cmor/files?version=2017.09.22.3.2.8&channel=nightly

You can install the latest version using this command.

conda install -c conda-forge -c pcmdi/label/nightly cmor==2017.09.22.3.2.8

I also created a test for it.

https://github.com/PCMDI/cmor/blob/master/Test/test_python_appending.py

from cmor3_documentation.

@wachsylon did you give it a try? Thanks!

from cmor3_documentation.

Thanks for the enhancement! Sorry for the late answer.
It worked in my tests, also for hybrid zaxis.

from cmor3_documentation.

@dnadeau4 It looks like @wachsylon was satisfied by these changes. Should we close this issue.

In the API documentation, https://cmor.llnl.gov/mydoc_cmor3_api/, there is this sentence for the file_suffix parameter of cmor_write.

CMOR2 will be smart enough to figure out if a suffix was used when creating that file.

Should that be CMOR3 instead?

from cmor3_documentation.

@mauzey1 you can close it. Charles removed me from the PCMDI group due to the private repository authentication.

from cmor3_documentation.