icesat-2hackweek / cloudmask Goto Github PK

The function read_atl06_fromfile in utils_atl06 access an h5 file and returns the data frames. However, some of the h5 files do not have all the features we request included, leading to the following problem:

I am still not sure if this is a problem of the data itself, in icepyx, or in the way I retrieve ATL06 data. This may also be related to this another issue, since I am not specifying the name of the granules I want to access.

For now, I will add a try command in read_atl06_fromfile to avoid this problem, but probably this is no the best solution.

In krigMaskNorth.ipynb, after converting LL to a list, this seems to be an empty list and that is the source of the error.

The name of some features in the ATL06 dictionary have been changed in order to have a more descriptive name or sometimes a shorter name. This is in utils_atl06.py. For example:

Should the names of the variables in the dataframes coincide with the ones in the ATL06 data dictonary?

The current assimilation of ICESat-2 with VIIRS is done via the Ball Tree method. The function 'associate' in 'utils_viirs.py' just calls the method Ball Tree on Sklearn:

In the particular case where (at least) one of the datasets consists on a grid of points (eg, satellite images) the assimilation can be done in a more efficient way, exploiting the structure of the grid.

This algorithm probably is useful in more general application and the method could be an important contribution for other teams to.

The following two functions can be used to compute a time object given the seconds in GPS, that is, the total seconds since 1980-06-01 00:00. The first function uses the package astropy and the second datetime.

However, we can see that they result in different times from the same timestamp (in this case, 0 seconds should match with 1980-06-01 00:00):

The haversine distance from sklearn we use to compute the distance between in 'CloudMask.ipynb' different grids takes pairs of the form [latitude, longitude]. However, in the code the grids are build as lists of the form [longitude, latitude].

Also, when we check the filter of spatial distmatch < tolerance, this tolerance is in the wrong units.

Parallelize the for loops in VIIRS_Bash.ipynb and in the assimilation between climate and ICESat-2 data.

As it was already reported in one of the issues in icepyx, we received the following error after we try to select the variables in ATL06 we want to retrieve:

In utils_atl06 we found the code used to retrieve the ATL06 data and create the associated Pandas data frames. If now we comment the lines where we select the name of the variables in the ATL06 product we want to sublet, then we don't received that error anymore:

Furthermore, it seems that without the subletting, we retrieve all the requiered information in the ATL06 product. However, we would like to ask for specific variables, in order no to download features that we are not going to use.

In the last version of Icepyx, the way to obtain the avail granules has changed. Now, we have to do

import icepyx as ipx

region_a = ipx.Query(short_name, spatial_extent, date_range)

See an example here. We need to change the function that retrieves the ATL data to this and also the notebooks that import icepyx.