Comments (4)
@DahnJ indeed, good catch. I opened an issue on the dask/partd side about this: dask/partd#52
Alternative for now is to specify set_index(.., shuffle="tasks")
, so it doesn't use the default "disk"
.
from dask-geopandas.
Interestingly, this is not the case for all the partitions:
>>> [type(partition.compute()) for partition in ddf2.partitions]
[pandas.core.frame.DataFrame,
geopandas.geodataframe.GeoDataFrame,
pandas.core.frame.DataFrame,
pandas.core.frame.DataFrame]
The change seems to happen in rearrange_by_column
from dask-geopandas.
Ah, interesting. Then it might also be a bug in GeoPandas (if some operation on a GeoDataFrame results in a pandas DataFrame where it could have preserved the GeoDataFrame type)
from dask-geopandas.
I believe the problem is in the fact that Dask uses partd.pandas
to (de)serialize the dataframe:
import geopandas as gpd
from partd.pandas import serialize, deserialize
df = gpd.read_file(gpd.datasets.get_path('naturalearth_lowres'))
type(deserialize(serialize(df))) # pandas.core.frame.DataFrame
The reason we sometimes get GeoDataFrame is that dask.shuffle.collect
returns the meta
object for empty partitions and the meta is a GeoDataFrame
.
from dask-geopandas.
Related Issues (20)
- dtype('O') not supported since geopandas 0.13.0
- Spatial_shuffle() can result in ArrowTypeError when using pyarrow 12 HOT 5
- FeatureError from filegdbtable.cpp when reading file HOT 2
- Drop distributed as a required dependency? HOT 2
- Question regarding parallelism over many seperate GeoSeries HOT 2
- dask geopandas to parquet does not seem to persist spatial paritions HOT 1
- Can someone answer why the number and x columns of '201105. shp' in the output of this code also become 0? HOT 1
- msgpack - ValueError: 2369781118 exceeds max_bin_len(2147483647 HOT 1
- Remove dask anti-pattern example on README and docs HOT 1
- DeprecationWarning: underlying geometries through the `.data` attribute is deprecated HOT 1
- Error when reading geoparquet file HOT 3
- Support latest dask.dataframe with query planning (dask-expr) HOT 3
- ddf._meta_nonempty doesnt instantiate correctly when calling `from_dask_dataframe` HOT 1
- BUG: `to_parquet()` failing with `dask=2024.4.1` HOT 2
- Uninformative AttributeError for aggregation methods
- AttributeError: 'DataFrame' object has no attribute 'within' HOT 9
- Can `GeoDataFrame.crs` set `None`? HOT 5
- spatial_shuffle fails when loading from a shapefile HOT 9
- `ValueError: 'left_df' should be GeoDataFrame, got <class 'tuple'>` using sjoin after `spatial_shuffle` HOT 7
- dissolve does not accept multiple columns to groupBy
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from dask-geopandas.