ssciwr / afwizard Goto Github PK

The required shared libraries are not found. This is most likely due to the minimal set of installed packages within the container. Solution: Find the correct set of packages and install them in Dockerfile.

Could e.g. be shown by clicking a help icon or when hovering over the option.

It works in Jupyter and the raw ds.show_hillshade() works in Sphinx, but the app is missing the hillshade visualization.

Should read the whole data set when omitted as the C++ executable does.

Currently, the visualization used in the filter tuning app cannot be configured, but really should be. This should probably be realized by a separate visualization schema which can directly be plugged into WidgetForm.

This was mentioned by Maria Shinoto as a method that proved useful in the past.

Users should be able to have an eye on memory usage.

Possible values are either WKT or EPSG:xxxx{x}, but not EPSG:xxxxx / bla. The latter can automatically be normalized.

After consultation with Lukas, we should replace this with ogr.geometryFromGeoJSON -> pyproj -> ogr.export_to_wkt.

Paths to notebooks are outdated. Probably also the API doc needs updating.

This sets up the two Docker images used on the SSC Jupyter instance.

• on a per-dataset level!

This adds the PDAL backend to adaptivefiltering. Necessary substeps:

• Inherit PDALFilter class from Filter and implement pipeline execution for that class
• Write schema with all ground point filtering algorithms from PDAL

It contains some outlier data points that destroy the scaling. This might be related to your work on reference systems @GwydionJon

The satellite map is not useful for very fine-grained control. Instead, hillshade and slopemap based visualization should be plugged into ipyleaflet. This should probably wait for refactorings that come out of #70

Basic planning of the OOP design for the project. Documentation is found in the project management repository.

Subissues:

• Remove bounding box
• Better camera angle
• Skip visualization if #points above threshold

This removes the weirdnesses of ipympl and at the same time gives us nice new visualization options, see http://pdal.io/stages/writers.gdal.html Also, we can remove the richdem dependency if we wanted to.

When showing maps, one might go for the usual EPSG:4326 in the beginning, but different systems may be helpful if we get into small areas which show large deviation.

Maybe, that Michael can judge whether he needed such functionality for his filtering, and @lrg-bhoefle should be able to comment on this as well. -- I will check for problems with EPSG:4326 when I start with Jupyter notebooks and data sets from South Japan.

See e.g. here: https://adaptivefiltering.readthedocs.io/en/latest/#Demo-of-the-Adaptive-Ground-Point-Filtering-Library: Each code line is preceded with a line that only states the numbered input prompt.

This does not happen in the nbsphinx demo docs and I have no clue yet why we have them.

This is most likely due to inconsistent usage of stdout and stderr within OPALS.

I tried to visualize the slope map of the thingstaette data set, but I get the following error (regardless of resolution).

---------------------------------------------------------------------------
RuntimeError                              Traceback (most recent call last)
<ipython-input-25-f9116e2e4536> in <module>
----> 1 ds.show_slope(resolution=2.0, classification=adaptivefiltering.asprs["ground"])

/opt/conda/lib/python3.9/site-packages/adaptivefiltering/dataset.py in show_slope(self, resolution, classification)
    128 
    129         dataset = PDALInMemoryDataSet.convert(self)
--> 130         return dataset.show_slope(resolution=resolution)
    131 
    132     def save(self, filename, compress=False, overwrite=False):

/opt/conda/lib/python3.9/site-packages/adaptivefiltering/pdal.py in show_slope(self, resolution, classification)
    254 
    255         with tempfile.NamedTemporaryFile() as tmp_file:
--> 256             self.save_mesh(
    257                 str(tmp_file.name),
    258                 resolution=resolution,

/opt/conda/lib/python3.9/site-packages/adaptivefiltering/pdal.py in save_mesh(self, filename, resolution, classification)
    180         if os.path.splitext(filename)[1] == ".tif":
    181             filename = os.path.splitext(filename)[0]
--> 182         execute_pdal_pipeline(
    183             dataset=self,
    184             config=[

/opt/conda/lib/python3.9/site-packages/adaptivefiltering/pdal.py in execute_pdal_pipeline(dataset, config)
     60     # Define and execute the pipeline
     61     pipeline = pdal.Pipeline(json.dumps(config), arrays=arrays)
---> 62     _ = pipeline.execute()
     63 
     64     # We are currently only handling situations with one output array

/opt/conda/lib/python3.9/site-packages/pdal/pipeline.py in execute(self)
     38 
     39     def execute(self):
---> 40         return self.p.execute()
     41 
     42     def validate(self):

libpdalpython.pyx in pdal.libpdalpython.PyPipeline.execute()

RuntimeError: writers.gdal: Grid width out of range.

This makes scrolling the Jupyter notebook/the Sphinx doc quite annoying. Is this possible @GwydionJon ?

Steps:

• Replacement of occurences in the code base
• RTD Integration
• Codecov integration

numpy.bincount can be used to compute this.

Looks horrible: https://adaptivefiltering.readthedocs.io/en/latest/#adaptivefiltering.DataSet.save_mesh

Potential fix: sphinx-doc/sphinx#8771

As e.g. seen here: https://github.com/ssciwr/mobile-phone-network-analysis/blob/main/.github/workflows/ci.yml#L27

The two angles should be passable to the show_hillshade method.

When using the pkgebk_all.las dataset one can not zoom in all the way without ipyleaflet switching from a satellite image to an error messag. This may be happening in different remote locations as well.

Array controls are to the right instead of below again. Also, they do not work. I fixed this in the past, but the fixes are not in main - not sure what happened, properly some wrong merge conflict resolution.

CodeCov struggles with the renaming of the project and does not show the new project. Hopefully, after some days they will rebuild their caches and find the new repository. After that, we should add the new CODECOV_TOKEN as a secret and the CI should run again.

This includes writing a schema for the relevant subset of http://pdal.io/stages/writers.gdal.html

Otherwise, the user always needs to manually select ground points which is annoying.

Michael would love to see an additional backend based on lastools lasground[_new] tools. If this integrates well with our current design, we could add it.

This is a list of parameters that were omitted for technical reasons:

• classbits argument of filters.smrf: There was an upstream bug about the default value (PDAL/PDAL#3068) - the documented default does not work for our current PDAL version. Skipping for now.
• The ignore parameter of filters.{smrf,pmf,csf}: No good default value can be provided which creates a problem if constructing from widget data. I guess that support needs to be implemented as a new feature of WidgetForm that allows to drop empty strings.

Currently, a filter is defined by its configuration value and conforms to a fixed schema. However, it would be beneficial to also be able to define "meta filters" or "filter templates" that allow variability for certain configuration parameters by e.g. specifyng a range for a continuous parameter or a selection for a discrete parameter. Any variable configuration values in such filter template would be fixed to a certain value by the end user at a later stage (at which point it would again be a filter in the current sense). We should investigate how such functionality can be integrated into the current design without sacrificing the simplicity of the current approach.

We do this trickery to work around the fact that RTD does not support Git LFS: https://github.com/ssciwr/adaptivefiltering/blob/main/doc/source/conf.py#L67

Unfortunately this broke last week, but it also did for others: readthedocs/readthedocs.org#8288
The logs suggest that the LFS checkout is successful, but the changes are overriden later on.

In order to have the map visualization and our dataset align well, we need to make sure to use the correct SRS when creating GeoTiffs. This can most likely be done using the override_srs option of writers.gdal or alternatively using GDAL's warp functionality. @GwydionJon

laspy released a non-backwards compatible version 2. We should keep an eye on the Anaconda package being released to switch to the new API soon.

Currently, we are dealing with projection-related issues by unconditionally reprojecting input data. This approach turned out to be more invasive than expected (see e.g. #67). Instead, it would fit much more into the general philosophy to have reprojection be an explicit, user controlled transformation:

dataset = adaptivefiltering.reproject(dataset, referencesystem="EPSG:4326")

Those parts of adaptivefiltering that rely on a certain projection could then throw an actionable error if a projection is required. After implementing #71, the default map of ipyleaflet could also be changed to something that is always present like hillshade. @GwydionJon Your input on this is appreciated.

Halves memory requirement while maintaining accuracy.

Use PDAL to generate DTM and visualize it with a show_dtm method on the data set. Resolution should be a user parameter.

The following aspects still need to be done to finalize #12

• Remove weird prompt lines
• Adapt section headers in Jupyter notebooks to fit into the Sphinx style
• Make sure that the menu properly recognizes
• Check RTD build
• Mark cells for exclusion in nbsphinx where necessary

The latter is achieved with

"metadata": {
    "nbsphinx": "hidden"
   }

We are fitting widgets into layout containers a lot by specifying width='100%' to make them fit exactly. However, many of these still produce a horizontal scrollbar. I consider this a bug of ipywidgets which should be researched and potentially reported upstream.

The GitHub quota are quite restrictive. For this reason, the repository currently cannot be pulled.

There is no intermediate caching of the filtering results which results in redundant running of the filtering pipeline, although it was unchanged.