clcr / pyeo Goto Github PK

Composites created in some conditions lead to a large blank space in the image; this pads image size to ~12gb.
I think this is something to do with incorrect projections. Am adding a reprojection step to compositing to overcome.

https://forum.step.esa.int/t/new-release-of-sen2cor-2-8-is-available/15481
Automatic checking and movement of sen2cor fails; need present system timestamp to correctly build new sen2cor filepath

Error message:

tail COL_cartagena_S2_rolling_CD.e2668115
/data/clcr/shared/miniconda3/envs/eoenv/lib/python3.6/site-packages/sklearn/base.py:311: UserWarning: Trying to unpickle estimator GridSearchCV from version 0.19.1 when using version 0.19.2. This might lead to breaking code or invalid results. Use at your own risk.
UserWarning)
2019-03-28 17:24:07,878: INFO: Created classification image file: /data/clcr/shared/colombia/amazon/cartagena/output/categories/class_composite_T18NWF_20171230T152631_20180104T152629.tif
2019-03-28 17:24:07,878: WARNING: Model has no n_classes_ attribute (known issue with GridSearch)
Traceback (most recent call last):
File "/data/clcr/vmhl1/pyeo/pyeo/apps/change_detection/rolling_composite_s2_change_detection.py", line 175, in
skip_existing=True, apply_mask=True)
File "/data/clcr/vmhl1/pyeo/pyeo/core.py", line 1683, in classify_image
prob_out_image = create_matching_dataset(image, prob_out_path, bands=model.n_classes_, datatype=gdal.GDT_Float32)
AttributeError: 'GridSearchCV' object has no attribute 'n_classes_'

Temporary fix:

prob_path in classify image function set to 'None'

When there are too many training points (or training classes). This is on Desktop not on ALICE

In the aggregation subroutine, the timestamp of the SAFE directory is NOT a unique identifier of a Sentinel-2 scene. This is why in my processing run, out of 68 scenes only 30 merged tiff files are created - because some have different SAFE names but the same timestamp.

out_path = os.path.join(out_dir, get_sen_2_image_timestamp(safe_dir))+".tif"

The code needs to retain a unique identifier for each aggregated tif file with the full SAFE name.

Function based on numpy to reclassify raster values

Doesn't need money, still fast

E.g. incorporate: https://www.gdal.org/gdal_merge.html

A small script that carries out a dialog with the user to fill out a config file.

E.g. summing up list of rasters

It would be good to add validation for the configuration file, with straight-forward error messages for any required values

Need to add an option to the .ini file and a relevent processing chain step to reproject merged images to a chosen projection

If a query .json contains a feature with geometry:null before the feature containing the query geometry,
the ESA query returns 0.

Pre 2016 files are failing due to a missing prefix for filename

Right now, needs "EPSG:somenum" as input, but should also be able to take an int for projection number

Was wondering, if it is worth adding a function for users who refined their searching results through the interactive graphical user interface? So they can refine the search results in the user interface and download exactly what they want?

This might be quite handy when (1) we only need to update certain images (2) a lot of visualisation of the search results is needed, e.g. small AOI in a quite cloudy areas

Build reprojection step into processing chain
Related to #23

Proposal: build NDVI to prevent comparisons between dry and wet seasons

sen2cor 2.5 > have changed its naming format (again), so the processed L2A.safe is named with a timestamp, which indicates when L1C data was processed. It's unclear when this timestamp is generated inside of sen2cor so this lead to the failure of assessing and moving the L2A folder.

Potential ways of fixing it:
add L2 dir as output dir in "atmospheric_correction/raster_manipulation) and point sen2cor to write to the L2_dir directly.

or try a using a id without the timestamp to identify the L2A image?

At present, user has to open images, mangle to same extent and then apply the mask bit-by-bit. Need to build a simpler function.

Datatype of input raster images is in UInt16

Does anything speak against using the pandas library to read in csv files?

2019-03-29 16:55:57,720: INFO: PROCESSING START
Traceback (most recent call last):
File "/home/ubuntu/pyeo/pyeo/tests/real_data_tests.py", line 433, in
masked_img_list = test_create_masked_image()
File "/home/ubuntu/pyeo/pyeo/tests/real_data_tests.py", line 419, in test_create_masked_image
masked_img = pyeo.create_masked_image(img, test_mask, test_out_path)
File "/home/ubuntu/pyeo/pyeo/core.py", line 1659, in create_masked_image
out_view[mask_view] = image_view
ValueError: shape mismatch: value array of shape (11023,10902) could not be broadcast to indexing result of shape (3675,3633,10902)

Process finished with exit code 1

Need a function to select certain images to be built into composite. Current model builds every image in the current folder.

function goes into here: subprocessing/composite_directory.py

Improvement needed in: pyeo.core compostie_directory (line 1121, now searching for .tif)

A test suite is needed for the function free_space.py so we can build a better file cleanup routine.

Need to fix sen2Get to take whole config file, not just download location

Need to find the right asset to get all the info.

https://developers.planet.com/docs/quickstart/downloading-imagery/#next-guide-best-practices-when-working-with-large-aois

PYEO currently downloads already existing images which are in the composite/L1 directory in a *.SAFE format

eg. if composite_20180201 is built with image from 20180201 and 20180131. When the composite is updated to composite_20180202, then the generated date layer has only two values: 20180202 and 20180201, the date 20180131 from the previous composite will be lost.

We need to selectively reprocess data to filter out certain cloud masks

Enhance flexibility by allowing the user to switch image stacking order which is needed because the models could be trained using different image stacking order

Some integration with QGIS to build automatic maps

According to GlobWetlandAfrica/GWA_models_and_workflows#35, changing the standard path in the Copernicus api url from /dhus to /apihub can avoid HTTP401 errors

fmask is filling the directory a given images comes from with temporary files that need to be removed after processing

At the moment the download function only check for already existing ZIP files (sentinelsat) and not for existing SAFE files. Hence it will download the ZIP files again.

Directory structure does not match powerpoint

Sen2Cor suppresses some error messages, e.g. no write permission in cfg subfolder in SEN2COR_HOME. Would be better to check reason for the exception and log it.

I recieved another Memory Error but this time at a different stage and instead of a GDAL VirtualMemory error it is a numpy MemeoryError. Here the details:

in_masked = in_masked = get_masked_array(in_raster, mask_paths[i])
File "/data/clcr/vmhl1/pyeo/pyeo/core.py", line 1140, in get_masked_array
mask_array = project_array(mask_array, raster_array.shape[0], 0)
File "/data/clcr/vmhl1/pyeo/pyeo/core.py", line 1147, in project_array
array_in = np.repeat(array_in, depth, axis)
File "/data/clcr/shared/miniconda3/envs/eoenv/lib/python3.6/site-packages/numpy/core/fromnumeric.py", line 445, in repeat
return _wrapfunc(a, 'repeat', repeats, axis=axis)
File "/data/clcr/shared/miniconda3/envs/eoenv/lib/python3.6/site-packages/numpy/core/fromnumeric.py", line 51, in _wrapfunc
return getattr(obj, method)(*args, **kwds)
MemoryError

Am I correct in assumeing that it is another RAM issue and it should be fine to increase the required vmem?

I encountered this error twice:
1st error message:
File "/data/clcr/vmhl1/pyeo/pyeo/apps/change_detection/rolling_composite_s2_change_detection.py", line 121, in
pyeo.composite_directory(composite_merged_dir, composite_dir)
File "/data/clcr/vmhl1/pyeo/pyeo/core.py", line 1089, in composite_directory
composite_images_with_mask(sorted_image_paths, composite_out_path, format)
File "/data/clcr/vmhl1/pyeo/pyeo/core.py", line 1028, in composite_images_with_mask
np.copyto(output_view, in_masked, where=np.logical_not(in_masked.mask))
ValueError: could not broadcast input array from shape (4,11039,11039) into shape (4,11038,11039)
/data/clcr/vmhl1/pyeo/bin/rolling_s2_change_detection: line 5: 12208 Segmentation fault python "$PYEO"/pyeo/apps/change_detection/rolling_composite_s2_change_detection.py "$@"

2nd error message:
File "/data/clcr/vmhl1/pyeo/pyeo/apps/change_detection/rolling_composite_s2_change_detection.py", line 121, in
pyeo.composite_directory(composite_merged_dir, composite_dir)
File "/data/clcr/vmhl1/pyeo/pyeo/core.py", line 1089, in composite_directory
composite_images_with_mask(sorted_image_paths, composite_out_path, format)
File "/data/clcr/vmhl1/pyeo/pyeo/core.py", line 1028, in composite_images_with_mask
np.copyto(output_view, in_masked, where=np.logical_not(in_masked.mask))
ValueError: could not broadcast input array from shape (4,11020,11020) into shape (4,11019,11020)
ERROR 1: CPLVirtualMemManagerThread: no mapping found
/data/clcr/vmhl1/pyeo/bin/rolling_s2_change_detection: line 5: 10594 Segmentation fault python "$PYEO"/pyeo/apps/change_detection/rolling_composite_s2_change_detection.py "$@"

Here an additional error message occurs (CPLVirtualMemManagerThread: no mapping found) which might caused by my qsub settings.

Thanks

Hi John, found out the reason for the projection error - the bug we fixed got committed to the Terrain brunch instead of the main one!
in the main brunch line 771, we are missing the srs_id argument
write_geometry(intersection, intersection_path, srs_id = srs_id)