http://geoscienceaustralia.github.io/digitalearthau/query/Loading%20Data.html

The link right at the bottom of the page takes to a "does not exist page":

http://datacube-core.readthedocs.io/en/stable/dev/generate/datacube.Datacube.load.html

Tags that are also mentioned in the DEA Glossary link to the glossary page, not notebooks.

To reproduce:

1. Visit Tags index
2. Click a tag like "DEA" which is also included in the Glossary

You are directed to the Glossary page rather than the corresponding Jupyter notebook

Please uncomment the zonal_timseries function once the rasterstats package has been added to the DEA-prod module

In the first cell the path is:

sys.path.append(os.path.expanduser('~/dea-notebooks/10_Scripts'))

which doesn't work.

I have used:

sys.path.append('../10_Scripts/')

which does work.

Need to change data percentage datatype or make array of same size as others else geotiff writer won't work without dropping it

The new Landsat notebooks come with some extra local images that are currently located at dea-notebooks/DEA_datasets. I can see that some of these "GettingStarted..." notebooks might need local files like this, but I think we might want to organise them a bit better. Perhaps they could all be stored in a folder in that directory? Something like:

dea-notebooks/DEA_datasets/images

@asivapra @CEKrause

Moving files from 02_DEA_datasets/Files to Supplementary_Data/Images

Moving files from 09_Workflows/ mv *.png ../Supplementary_Data/Images/

http://geoscienceaustralia.github.io/digitalearthau/notebooks/DEA_datasets/Introduction_to_Landsat.html

The abbreviation "SR" is introduced in the NBAR-T section, but it is not defined (presumably 'surface reflectance'). SR can be defined on the first line of the document.

In the first notebook input cell (In [1]:), it says "Import dea-notebook functions. This assumes you are running this notebook from inside the dea-notebooks directory. If not, replace the relative ../Scripts link with a path to the dea-notebooks/Scripts directory.

It is not clear how to find the dea-notebooks/Scripts directory. I just tried it with a notebook started using the DEA desktop icon and it still didn't work. So an explanation of starting a notebook from a subdirectory inside a cloned version of dea-notebooks on GitHub is needed.

In the section "Creating a true-colour image using three bands", it is somewhat confusing to talk about making a true colour plot and then having a flag "contrast_enhance=True" that results in a "colour-enhanced image". It's not clear which one is the true colour image (with flag True or False?). Maybe clarifying this will help the user understand what's happening better?

For tags, you need to select 'Cell Type --> Raw NBConvert
Before you can select Raw NBConvert Format --> reST on the dropdown box that appears on the top right hand side of the cell.

Will do a pull request but this issue is for anyone else that runs into it before I have time to fix/ if anyone gets to it first that's also good :)
Bex

In the first cell, the path to the scripts folder is incorrect.

WOFS notebook - check that algorithm is based off our surface reflectance product (vs USGS L2), is actually valid, and can be used, or add a caveat that it is pending the newWOFS product and is probably not the best version

Further to discussions today about making sure we reference any direct quotes from previously published material:

I've added a section in dea-notebooks documentation with a link to the GA referencing guide.

We need to please caption and reference any images used that have been previously published elsewhere.

For document references,
I've decided to include an internal and and external link for the first reference to any externally published content, followed by internal links for further references. These internal links go to a references cell at the end of the notebook, where references are formatted according to the GA style guide (now linked to in the dea notebooks getting started guide, or at http://www.ga.gov.au/copyright/how-to-cite-geoscience-australia-source-of-information )

Thoughts on this are requested - maybe there is a better way? I know it's hard to use internal links on github, but the external links work really well.

This is necessary if you have notebooks that consistently use directly copied and pasted information from elsewhere. Please don't use anything copied-and-pasted without enclosing it in quotes and referencing, as per usual scientific publication standards, particularly as these GA- published notebooks are within GeoscienceAustralia's repository and visible to external collaborators.

If you are able to, you can paraphrase external sources but please include a reference if you do so.

Please can we check any published notebooks for these issues.

Regards,
Bex

In the first cell the path is:

sys.path.append(os.path.expanduser('~/dea-notebooks/10_Scripts'))

which doesn't work.

I have used:

sys.path.append('../10_Scripts/')

which does work.

It looks like plots are not appearing when generated with % pylab notebook
E.g. http://geoscienceaustralia.github.io/digitalearthau/notebooks/products/GettingStartedWithSentinel2.html#See-what-Sentinel-2-products-are-currently-available

Using %matplotlib inline will work, and we could investigate other methods, such as %matplotlib notebook

If you set the output CRS to WGS84 when passing a query to the load_clear functions within DEADatahandling an error occurs in the calculation of data_perc as the x and y dimensions are returned as longitude and latitude and this breaks the line:
data_perc = good_quality.sum(dim=['x', 'y']) / (good_quality.shape[1] * good_quality.shape[2])

A change to make this dimension title agnostic should solve the problem.
eg. data_perc = good_quality.sum(axis=1).sum(axis=1) / (good_quality.shape[1] * good_quality.shape[2])

Something to implement after we return from break.

The function :

DEAPlotting.animated_timeseries

requires the dimensions of the input xarray are called x and y when calculating the width ratio:

width_ratio = float(ds.sizes['x']) / float(ds.sizes['y'])

Not essential but would be good to allow for longitude / latitude, lon(g) / lat. More broadly this is an issue I find semi-regularly with functions and libraries related to spatial data so good to keep an eye out for.
@robbibt

pr #210 adds doctesting, but in order to use this we need examples to test.
add examples to the functions and check that this works

Use the following code from this notebook to update the Temporal analysis notebook / Scripts in order to add a function for multi- timestep polygon drill/masking

GWDEA_working/Development_Notebooks/TCW_WOfS_FC_stacked_plot_polygon_drill.ipynb#

import rasterio.mask
import rasterio.features

data = ls578_ds
mask = rasterio.features.geometry_mask([geom.to_crs(data.geobox.crs)for geoms in [geom]],
                                           out_shape=data.geobox.shape,
                                           transform=data.geobox.affine,
                                           all_touched=False,
                                           invert=False)
#for some reason xarray is not playing nicely with our old masking function
mask_xr = xr.DataArray(mask, dims = ('y','x'))
dataMasked = data.where(mask_xr==False)

@BexDunn
Need to reduce the headings levels to at least "##" for all except the first one. (it is not rendering properly on the website with the higher level headings).

On the folder listing page, any top level heading is pulled in, making it hard to distinguish between notebooks.
E.g. for Products: http://geoscienceaustralia.github.io/digitalearthau/notebooks/products/README.html
There are 2 notebooks:

• Getting started with Sentinel 2 data
• View Landsat 8 imagery for a chosen time period

but many headings are pulled in. If we can add another # to the other headings, they can still be used in the sidebar to navigate within the notebook:
http://geoscienceaustralia.github.io/digitalearthau/notebooks/products/GettingStartedWithSentinel2.html#See-what-Sentinel-2-products-are-currently-available

The case_studies notebook directory has no readme description like the other three directories, so it is currently unclear what kind of notebooks/code belong here

https://github.com/GeoscienceAustralia/dea-notebooks/tree/master/case_studies

http://geoscienceaustralia.github.io/digitalearthau/connect/account.html

"Getting Help" is in a drop-down under 'NCI Account Registration"

-can we please move it up to the same level? it's hidden as a default.

it makes it hard to find the Slack sign-up link.
http://geoscienceaustralia.github.io/digitalearthau/connect/account.html#getting-help

The GettingStartedWithLandsat5-7-8.ipynb notebook currently includes "Note that the NBAR-T data inserts a missing value (-999) into the dataset to denote a terrain shadow. This should be converted to a NaN value before using this data to avoid treating it as a valid value."

However, NBAR also includes -999 values for nodata. This should be covered both in the dataset description and in the actual code examples.

In introduction_to_Fractional_cover notebook.

Scripts folders have been moved - we need to update our paths for loading dea_notebooks from Scripts to 10_Scripts.

For exploring data in e.g. ENVI

Specify bounding box and get back a geotiff with all the bands for that location.

@CEKrause @BexDunn I've had some issues with the bands_of_interest parameter in load_nbarx: any time I try to specify a set of bands, the function crashes out with a key error KeyError: 'bands_of_interest'. It works fine when no bands are specified.

I think this is because the function wants bands_of_interest given to it as a **kwarg, which only works when expanding dictionaries. I was thinking I might edit the function to define bands_of_interest as a simple variable like this:

load_nbarx(dc, sensor, query, bands_of_interest, product = 'nbart', filter_cloud = True)

...but this will definitely break people's code if the current function works for them. Would anyone strongly object if I made this change? (just opening this up for discussion because it's probably the single most-used function in the repository at this stage)

This should be able to be done without breaking code (i.e. change the backend from Matplotlib .imshow to Xarray .imshow)

"nbar_clean = xr.concat(sensor_clean.values(),dim='time')" returns the following error:

ValueError: The truth value of an array with more than one element is ambiguous. Use a.any() or a.all()

It is possible due to the use of a different version of xarray module.

If bands_of_interest are specified, then pixel_quality needs to be included, or else it will throw an error. Need to add a check that if bands are specified and filter_cloud = True, pixel quality is also grabbed.

Currently some of the example notebooks (e.g. "GettingStartedWithLandsat5-6-7") use older versions of the DEA functions that have been hard-coded into the notebook.

Instead, we should use the existing DEA module functions.

Looks like we need to change the values to uint8 to get the geotiff to work

One of the key issues discussed at the dea-notebooks meeting last week was the lack of a list of all functions in Algorithms.

I've created a pull request (#57) with a possible way to have some basic documentation of the growing list of functions in Algorithms: simply update the readme file with a list and basic descriptions of the functions. This could be an easy intermediate step until we work out a more automated function documentation.

I'm not sure if this will affect the existing automated .rst import, so this might be something to discuss.

(@CEKrause @andrewdhicks @BexDunn )

The two plotting functions in DEAPlotting produce inconsistent outputs: three_band_image produces a matplotlib "plot" and "figure" objects wrapped in a tuple, while three_band_image_subplots simply plots the image directly without creating plot/figure objects.

For consistency/ease of use, it would be great if these both functioned in the same way.

https://github.com/GeoscienceAustralia/dea-notebooks/blob/master/algorithms/DEAPlotting.py

Currently contour_extract uses indices to match attribute data to each contour before writing to file. However, if one contour is missing, the next index will be (incorrectly) assigned to the next contour which does exist.

Need to test that this is indeed the case; flagging as an issue for now so I don't forget.

https://github.com/GeoscienceAustralia/dea-notebooks/blob/master/10_Scripts/SpatialTools.py

Hey friends,
Thoughts on creating a subfolder/an alternate folder for scripts vs algorithms? Because we have plenty of scripts to load, and they aren't necessarily notebook examples of using algorithms?
Bex

To do list : update to get bigger images

http://geoscienceaustralia.github.io/digitalearthau/connect/account.html

Hyperlink for "Join our Slack Community" currently points to https://opendatacube.signup.team/

Should point to: https://opendatacube.slack.com ?

For example in first cell of Fractional Cover

Import external functions from dea-notebooks
sys.path.append('../10_Scripts/')
import DEAPlotting, DEADataHandling

code assumes clone of repo but not clear in instructions. may need update to remind user before first cell

Robbi,
In the "DEAPlotting.three_band_image_subplots", could you consider changing the following line?

img_toshow = rawimg / 5000
to
img_toshow = (rawimg / 5000).clip(0, 1)

It will avoid warnings like below.

"Clipping input data to the valid range for imshow with RGB data ([0..1] for floats or [0..255] for integers)."

Composite generation notebooks need paths to nbviewer pages updated