clcr / pyeo Goto Github PK
View Code? Open in Web Editor NEWPython Earth Observation processing chain
License: GNU General Public License v3.0
Python Earth Observation processing chain
License: GNU General Public License v3.0
Composites created in some conditions lead to a large blank space in the image; this pads image size to ~12gb.
I think this is something to do with incorrect projections. Am adding a reprojection step to compositing to overcome.
https://forum.step.esa.int/t/new-release-of-sen2cor-2-8-is-available/15481
Automatic checking and movement of sen2cor fails; need present system timestamp to correctly build new sen2cor filepath
Error message:
tail COL_cartagena_S2_rolling_CD.e2668115
/data/clcr/shared/miniconda3/envs/eoenv/lib/python3.6/site-packages/sklearn/base.py:311: UserWarning: Trying to unpickle estimator GridSearchCV from version 0.19.1 when using version 0.19.2. This might lead to breaking code or invalid results. Use at your own risk.
UserWarning)
2019-03-28 17:24:07,878: INFO: Created classification image file: /data/clcr/shared/colombia/amazon/cartagena/output/categories/class_composite_T18NWF_20171230T152631_20180104T152629.tif
2019-03-28 17:24:07,878: WARNING: Model has no n_classes_ attribute (known issue with GridSearch)
Traceback (most recent call last):
File "/data/clcr/vmhl1/pyeo/pyeo/apps/change_detection/rolling_composite_s2_change_detection.py", line 175, in
skip_existing=True, apply_mask=True)
File "/data/clcr/vmhl1/pyeo/pyeo/core.py", line 1683, in classify_image
prob_out_image = create_matching_dataset(image, prob_out_path, bands=model.n_classes_, datatype=gdal.GDT_Float32)
AttributeError: 'GridSearchCV' object has no attribute 'n_classes_'
Temporary fix:
prob_path in classify image function set to 'None'
When there are too many training points (or training classes). This is on Desktop not on ALICE
In the aggregation subroutine, the timestamp of the SAFE directory is NOT a unique identifier of a Sentinel-2 scene. This is why in my processing run, out of 68 scenes only 30 merged tiff files are created - because some have different SAFE names but the same timestamp.
out_path = os.path.join(out_dir, get_sen_2_image_timestamp(safe_dir))+".tif"
The code needs to retain a unique identifier for each aggregated tif file with the full SAFE name.
Function based on numpy to reclassify raster values
Doesn't need money, still fast
E.g. incorporate: https://www.gdal.org/gdal_merge.html
A small script that carries out a dialog with the user to fill out a config file.
E.g. summing up list of rasters
It would be good to add validation for the configuration file, with straight-forward error messages for any required values
Need to add an option to the .ini file and a relevent processing chain step to reproject merged images to a chosen projection
If a query .json contains a feature with geometry:null before the feature containing the query geometry,
the ESA query returns 0.
Pre 2016 files are failing due to a missing prefix for filename
Right now, needs "EPSG:somenum" as input, but should also be able to take an int for projection number
Was wondering, if it is worth adding a function for users who refined their searching results through the interactive graphical user interface? So they can refine the search results in the user interface and download exactly what they want?
This might be quite handy when (1) we only need to update certain images (2) a lot of visualisation of the search results is needed, e.g. small AOI in a quite cloudy areas
Build reprojection step into processing chain
Related to #23
Proposal: build NDVI to prevent comparisons between dry and wet seasons
sen2cor 2.5 > have changed its naming format (again), so the processed L2A.safe is named with a timestamp, which indicates when L1C data was processed. It's unclear when this timestamp is generated inside of sen2cor so this lead to the failure of assessing and moving the L2A folder.
Potential ways of fixing it:
add L2 dir as output dir in "atmospheric_correction/raster_manipulation) and point sen2cor to write to the L2_dir directly.
or try a using a id without the timestamp to identify the L2A image?
At present, user has to open images, mangle to same extent and then apply the mask bit-by-bit. Need to build a simpler function.
Datatype of input raster images is in UInt16
Does anything speak against using the pandas library to read in csv files?
2019-03-29 16:55:57,720: INFO: PROCESSING START
Traceback (most recent call last):
File "/home/ubuntu/pyeo/pyeo/tests/real_data_tests.py", line 433, in
masked_img_list = test_create_masked_image()
File "/home/ubuntu/pyeo/pyeo/tests/real_data_tests.py", line 419, in test_create_masked_image
masked_img = pyeo.create_masked_image(img, test_mask, test_out_path)
File "/home/ubuntu/pyeo/pyeo/core.py", line 1659, in create_masked_image
out_view[mask_view] = image_view
ValueError: shape mismatch: value array of shape (11023,10902) could not be broadcast to indexing result of shape (3675,3633,10902)
Process finished with exit code 1
Need a function to select certain images to be built into composite. Current model builds every image in the current folder.
function goes into here: subprocessing/composite_directory.py
Improvement needed in: pyeo.core compostie_directory (line 1121, now searching for .tif)
A test suite is needed for the function free_space.py so we can build a better file cleanup routine.
Need to fix sen2Get to take whole config file, not just download location
Need to find the right asset to get all the info.
PYEO currently downloads already existing images which are in the composite/L1 directory in a *.SAFE format
eg. if composite_20180201 is built with image from 20180201 and 20180131. When the composite is updated to composite_20180202, then the generated date layer has only two values: 20180202 and 20180201, the date 20180131 from the previous composite will be lost.
We need to selectively reprocess data to filter out certain cloud masks
Enhance flexibility by allowing the user to switch image stacking order which is needed because the models could be trained using different image stacking order
Some integration with QGIS to build automatic maps
According to GlobWetlandAfrica/GWA_models_and_workflows#35, changing the standard path in the Copernicus api url from /dhus
to /apihub
can avoid HTTP401 errors
fmask is filling the directory a given images comes from with temporary files that need to be removed after processing
At the moment the download function only check for already existing ZIP files (sentinelsat) and not for existing SAFE files. Hence it will download the ZIP files again.
Directory structure does not match powerpoint
Sen2Cor suppresses some error messages, e.g. no write permission in cfg subfolder in SEN2COR_HOME. Would be better to check reason for the exception and log it.
I recieved another Memory Error but this time at a different stage and instead of a GDAL VirtualMemory error it is a numpy MemeoryError. Here the details:
in_masked = in_masked = get_masked_array(in_raster, mask_paths[i])
File "/data/clcr/vmhl1/pyeo/pyeo/core.py", line 1140, in get_masked_array
mask_array = project_array(mask_array, raster_array.shape[0], 0)
File "/data/clcr/vmhl1/pyeo/pyeo/core.py", line 1147, in project_array
array_in = np.repeat(array_in, depth, axis)
File "/data/clcr/shared/miniconda3/envs/eoenv/lib/python3.6/site-packages/numpy/core/fromnumeric.py", line 445, in repeat
return _wrapfunc(a, 'repeat', repeats, axis=axis)
File "/data/clcr/shared/miniconda3/envs/eoenv/lib/python3.6/site-packages/numpy/core/fromnumeric.py", line 51, in _wrapfunc
return getattr(obj, method)(*args, **kwds)
MemoryError
Am I correct in assumeing that it is another RAM issue and it should be fine to increase the required vmem?
I encountered this error twice:
1st error message:
File "/data/clcr/vmhl1/pyeo/pyeo/apps/change_detection/rolling_composite_s2_change_detection.py", line 121, in
pyeo.composite_directory(composite_merged_dir, composite_dir)
File "/data/clcr/vmhl1/pyeo/pyeo/core.py", line 1089, in composite_directory
composite_images_with_mask(sorted_image_paths, composite_out_path, format)
File "/data/clcr/vmhl1/pyeo/pyeo/core.py", line 1028, in composite_images_with_mask
np.copyto(output_view, in_masked, where=np.logical_not(in_masked.mask))
ValueError: could not broadcast input array from shape (4,11039,11039) into shape (4,11038,11039)
/data/clcr/vmhl1/pyeo/bin/rolling_s2_change_detection: line 5: 12208 Segmentation fault python "$PYEO"/pyeo/apps/change_detection/rolling_composite_s2_change_detection.py "$@"
2nd error message:
File "/data/clcr/vmhl1/pyeo/pyeo/apps/change_detection/rolling_composite_s2_change_detection.py", line 121, in
pyeo.composite_directory(composite_merged_dir, composite_dir)
File "/data/clcr/vmhl1/pyeo/pyeo/core.py", line 1089, in composite_directory
composite_images_with_mask(sorted_image_paths, composite_out_path, format)
File "/data/clcr/vmhl1/pyeo/pyeo/core.py", line 1028, in composite_images_with_mask
np.copyto(output_view, in_masked, where=np.logical_not(in_masked.mask))
ValueError: could not broadcast input array from shape (4,11020,11020) into shape (4,11019,11020)
ERROR 1: CPLVirtualMemManagerThread: no mapping found
/data/clcr/vmhl1/pyeo/bin/rolling_s2_change_detection: line 5: 10594 Segmentation fault python "$PYEO"/pyeo/apps/change_detection/rolling_composite_s2_change_detection.py "$@"
Here an additional error message occurs (CPLVirtualMemManagerThread: no mapping found) which might caused by my qsub settings.
Thanks
Hi John, found out the reason for the projection error - the bug we fixed got committed to the Terrain brunch instead of the main one!
in the main brunch line 771, we are missing the srs_id argument
write_geometry(intersection, intersection_path, srs_id = srs_id)
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.