Comments (11)
@philss @cigrainger After much research and cooperation from community members, steps are beginning to be taken to resolve this issue. Initially I opened this discussion, pola-rs/polars#1830 , but finally Kyle Barron shed light on how to incorporate geospatial analysis into polars, here is a new library that promises if we support it and that we can integrate it into explorer in the future. It is based on georust ecosystem libraries and algorithms are perfectly integrated
https://github.com/kylebarron/geopolars
from explorer.
I know this is not related to the feature request, but I want to add this just as reference for people that want to work with Geospatial data in the Elixir ecosystem today: if you are using Livebook, you can work with GeoJSON and TopoJSON (a subset of GeoJSON) through VegaLite.
The only problem is that you may need to convert the data from other formats to these formats.
I made a small study using those tools here: philss/brazil-in-notebooks#1 👉 Run in Livebook
from explorer.
Excellent work!,
I am trying to integrate some functionalities of the georust ecosystem into an elixir NIF library in order to be used with spatial data, especially related to the gdal library that would allow working with a great variety of geographic data, both raster and vector.
Maybe you don't like the idea of using Rust very much, but there are more advances in this field than in elixir, in addition to the performance in some operations that demand calculation power
from explorer.
Oh very interesting! This is far outside my wheelhouse but absolutely seems like a good idea. My understanding is that it might be worthwhile to implement the Simple Features Standard. https://en.wikipedia.org/wiki/Simple_Features.
This is where the reliance on a Rust library wholesale becomes quite hairy. I think it would need to be approached way upstream with Polars, with respect to maybe the GeoRust ecosystem.
I'll give this some thought.
from explorer.
I have been investigating the georust ecosystem a bit and it seems to me a good option, to carry geopandas you need management of projections, reading of different raster and vector formats and conversions between them, management of geometries and all those functionalities exist within their libraries, It would only be necessary to integrate them into the Polars or maybe create a packages in elixir as a first step.
from explorer.
@roger120981 thank you!
I am trying to integrate some functionalities of the georust ecosystem into an elixir NIF library in order to be used with spatial data, especially related to the gdal library that would allow working with a great variety of geographic data, both raster and vector.
Cool! Please share with us when you have something.
Maybe you don't like the idea of using Rust very much, but there are more advances in this field than in elixir, in addition to the performance in some operations that demand calculation power
I prefer to use Elixir when possible, but I agree that a NIF in Rust for this is a good idea. Specially if they have a complete set of tools for this field. Although we have some tools in Elixir like Geo that can be used for some tasks without problems.
Just be aware of some limitations of NIFs. Even if they are written in Rust, you can have some drawbacks.
from explorer.
@philss Thanks for the observation.
from explorer.
Just to update everyone, we are still several months away from this. There is work happening on bringing geo formats to Arrow, then these need to be surfaced in Polars. Only then we can surface them to Explorer.
from explorer.
There is currently a project well advanced in this direction, https://github.com/geoarrow/geoarrow-rs, lead by @kylebarron, was used as core for this project,, https://developmentseed.org/lonboard, and in the future, it must be the core of this project, https://github.com/geopolars/geopolars
from explorer.
This project serves as a complement to visualize millions of data in web environments https://github.com/geoarrow/deck.gl-layers
from explorer.
@josevalim It would be advisable to look as it has bindings for Python and Javascript, and it would be a good starting point to integrate it as a Nif library in the future.
from explorer.
Related Issues (20)
- Bug: Rounding Error in Tests HOT 1
- exposing the `fold` expressions from Polars HOT 7
- :nif_panicked "Chunk require all its arrays to have an equal number of rows" HOT 1
- Sorting an empty DataFrame results in a runtime Polars error HOT 1
- Performance of `DataFrame.new/2` on dataframes containing list columns HOT 7
- `Series.filter` should work inside `DataFrame.summarise` HOT 5
- Large memory usage when using `Explorer.Dataframe.concat_columns` on 30k (small) data frames. Memory leak? HOT 4
- [Not Issue] - Are the plans to use duckdb as an alternative backend? HOT 2
- Support streaming: true on collect HOT 1
- Select from different dataframe inside mutate HOT 1
- Config to make table view default like in Polars itself HOT 1
- Duckdb select statement only works on non first line in the cell HOT 4
- Explorer.DataFrame.from_query/4 with :snowflake adapter returns dtype error on any numeric field HOT 3
- Orders of names after CSV loading seems buggy since 9.0 HOT 5
- Join on columns of type `:list` HOT 2
- Discrepancy between typespec for Series.cast/2 (parameter dtype) and implementation HOT 1
- LazyFrame not being able to cast dtypes HOT 1
- DataFrame.load_csv!/2 seems to fail on certain options HOT 2
- Mathematical operations are newline dependent in `summarise` macro HOT 2
- Support dplyr-like `keep` argument for `mutate` functions HOT 1
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from explorer.