Unpacked values from scaled/offset dataset are all NaN. about netcdf-java HOT 6 CLOSED

Actually

NetcdfFile njfile = NetcdfFiles.open (filePathStr);
NetccdfDataset njdataset = NetcdfDatasets.enhance (njfile, ENHANCEMENT, null);

where ENHANCEMENT is the default enhancement plus allowing for an "incomplete" coordinate system.

If you're wondering why I acquire the dataset in two steps, it's actually because there's also a step where I acquire an unenhanced version in order to see the original CDL description.

from netcdf-java.

@haileyajohnson, I thought about trying to hack up a test of modifying ConvertMissing to allow for specifying a more flexible relative difference in its fuzzy-equals test. But I took a second look at how many parameters there already are in the constructor and said, "Nah."

from netcdf-java.

@rschmunk just to clarify, the place where Panoply is getting incorrect data is from a call to NetcdfDatasets.openDataset(filename) with enhanced=all?

from netcdf-java.

@haileyajohnson, I tracked this down, and it has to do with the changes made to the ConvertMissing filter on January 12. I'm not sure which commit in particular was the coup de grace, as there were about 6 of them that day.

TL;DR - the unpacked data values from the sample dataset are all much smaller than Misc.defaultMaxRelativeDiffFloat.

Long version...

As noted above, the scale and offset values in the sample dataset are really small. Tiny even. The scale factor is of order 1E-12 and the offset of order 1E-7, and after unpacking, the output data results are of order 1E-8.

The dataset also specifies that the _FillValue is the same as the missing_value. Unpacked, the fill/missing value is ~ 0 (actually, also of order 1E-12).

The problem then is that ConvertMissing.isMissing receives an unpacked data value, checks if it is less than Misc.defaultMaxRelativeDiffFloat away from the unpacked fill value. Since that default difference is of order 1E-5, then of course the method is returning true. Every value in the array is being deemed invalid and the convert method sends back NaN for the entire array.

So it seems like a smaller difference value needs to be used in the test when dealing with such tiny scale factors and offsets.

from netcdf-java.

Thanks for tracking that down! We did change the fuzzyequals comparison to use a relative diff instead of absolute diff, and that's what broke this case - I'm planning on reverting that change. But another issue I think is that we need to do the missing and fill value comparisons with unpacked data, so that the fuzyequals implementation doesn't matter. I should have a fix up shortly. Thanks for all your help!

from netcdf-java.

I downloaded a fresh snapshot and built it into Panoply. It plots the sample data as expected, so it looks like #1315 was what was needed.

from netcdf-java.