Comments (15)
mmann1123
commented on September 27, 2024
1
@jgrss I talked to Issac this can be closed.
from geowombat.
mmann1123
commented on September 27, 2024
For the moment use
src.gw.to_raster('output.tif',
n_workers=4)
@jgrss any thoughts on what's happening with save?
from geowombat.
jgrss
commented on September 27, 2024
Hi @isaacWpark, the gw.save function uses dask and rasterio. The latter requires that tiled rasters be stored in block sizes in multiples of 16. For example, the smallest block/chunk size you can use is 16. Otherwise, you can use 32, 64, 128, 256, etc.
Your data have chunks that are not in multiples of 16. See chunksize from your printout.
<xarray.DataArray (band: 1, y: 1209, x: 941)>
dask.array<open_rasterio-857b09ff5b845667d53050c504460853,
shape=(1, 1209, 941),
dtype=int16,
chunksize=(1, 32, 250),
chunktype=numpy.ndarray>You can rechunk your DataArray by DataArray.chunk(chunksize). I think you should be able to do:
src.chunk({'band': -1, 'y': 512, 'x': 512}).gw.save(
"../temp/precip_20200601_30m.tif",
overwrite=True
) Of course, change the chunk sizes (row and column chunks above are 512) to your preference.
Let us know how that goes.
from geowombat.
mmann1123
commented on September 27, 2024
@isaacWpark is this resolved?
from geowombat.
mmann1123
commented on September 27, 2024
I am running into this issue with save I tried rechunk to no avail @jgrss
with gw.open(a_quarter, mosaic=True,overlap='max',nodata=missing_data ) as src:
src.chunk({'band': -1, 'y': 512, 'x': 512}).gw.save( filename=f'./mosaic/S2_SR_{quarter}_south.tif',nodata=nan,num_workers=6,compress='lzw')---------------------------------------------------------------------------
RasterBlockError Traceback (most recent call last)
File [/home/mmann1123/Documents/github/WB-spatial-features/create_mosaics.py:14](https://file+.vscode-resource.vscode-cdn.net/home/mmann1123/Documents/github/WB-spatial-features/create_mosaics.py:14)
[11](https://file+.vscode-resource.vscode-cdn.net/home/mmann1123/Documents/github/WB-spatial-features/create_mosaics.py:11) print('files:' , a_quarter)
[12](https://file+.vscode-resource.vscode-cdn.net/home/mmann1123/Documents/github/WB-spatial-features/create_mosaics.py:12) with gw.open(a_quarter, mosaic=True,overlap='max',nodata=missing_data,chunksize=16*5) as src:
---> [14](https://file+.vscode-resource.vscode-cdn.net/home/mmann1123/Documents/github/WB-spatial-features/create_mosaics.py:14) src.chunk({'band': -1, 'y': 512, 'x': 512}).gw.save( filename=f'[./mosaic/S2_SR_](https://file+.vscode-resource.vscode-cdn.net/home/mmann1123/Documents/github/WB-spatial-features/mosaic/S2_SR_){quarter}_south.tif',nodata=nan,num_workers=6,compress='lzw')
[17](https://file+.vscode-resource.vscode-cdn.net/home/mmann1123/Documents/github/WB-spatial-features/create_mosaics.py:17) # # %% interpolate missing doesn't work with multi band images
[18](https://file+.vscode-resource.vscode-cdn.net/home/mmann1123/Documents/github/WB-spatial-features/create_mosaics.py:18) # from xr_fresh.interpolate_series import interpolate_nan
[19](https://file+.vscode-resource.vscode-cdn.net/home/mmann1123/Documents/github/WB-spatial-features/create_mosaics.py:19) # from numpy import nan
(...)
[47](https://file+.vscode-resource.vscode-cdn.net/home/mmann1123/Documents/github/WB-spatial-features/create_mosaics.py:47) # kwargs={"BIGTIFF": "YES"},
[48](https://file+.vscode-resource.vscode-cdn.net/home/mmann1123/Documents/github/WB-spatial-features/create_mosaics.py:48) # )
File [~/miniconda3/envs/crop_pred/lib/python3.9/site-packages/geowombat/core/geoxarray.py:786](https://file+.vscode-resource.vscode-cdn.net/home/mmann1123/Documents/github/WB-spatial-features/~/miniconda3/envs/crop_pred/lib/python3.9/site-packages/geowombat/core/geoxarray.py:786), in GeoWombatAccessor.save(self, filename, mode, nodata, overwrite, client, compute, tags, compress, compression, num_workers, log_progress, tqdm_kwargs)
[779](https://file+.vscode-resource.vscode-cdn.net/home/mmann1123/Documents/github/WB-spatial-features/~/miniconda3/envs/crop_pred/lib/python3.9/site-packages/geowombat/core/geoxarray.py:779) warnings.warn(
[780](https://file+.vscode-resource.vscode-cdn.net/home/mmann1123/Documents/github/WB-spatial-features/~/miniconda3/envs/crop_pred/lib/python3.9/site-packages/geowombat/core/geoxarray.py:780) "The argument 'compression' will be deprecated in >=2.2.0. Use 'compress'.",
[781](https://file+.vscode-resource.vscode-cdn.net/home/mmann1123/Documents/github/WB-spatial-features/~/miniconda3/envs/crop_pred/lib/python3.9/site-packages/geowombat/core/geoxarray.py:781) DeprecationWarning,
[782](https://file+.vscode-resource.vscode-cdn.net/home/mmann1123/Documents/github/WB-spatial-features/~/miniconda3/envs/crop_pred/lib/python3.9/site-packages/geowombat/core/geoxarray.py:782) stacklevel=2,
[783](https://file+.vscode-resource.vscode-cdn.net/home/mmann1123/Documents/github/WB-spatial-features/~/miniconda3/envs/crop_pred/lib/python3.9/site-packages/geowombat/core/geoxarray.py:783) )
[784](https://file+.vscode-resource.vscode-cdn.net/home/mmann1123/Documents/github/WB-spatial-features/~/miniconda3/envs/crop_pred/lib/python3.9/site-packages/geowombat/core/geoxarray.py:784) compress = compression
--> [786](https://file+.vscode-resource.vscode-cdn.net/home/mmann1123/Documents/github/WB-spatial-features/~/miniconda3/envs/crop_pred/lib/python3.9/site-packages/geowombat/core/geoxarray.py:786) return save(
[787](https://file+.vscode-resource.vscode-cdn.net/home/mmann1123/Documents/github/WB-spatial-features/~/miniconda3/envs/crop_pred/lib/python3.9/site-packages/geowombat/core/geoxarray.py:787) self._obj,
[788](https://file+.vscode-resource.vscode-cdn.net/home/mmann1123/Documents/github/WB-spatial-features/~/miniconda3/envs/crop_pred/lib/python3.9/site-packages/geowombat/core/geoxarray.py:788) filename=filename,
[789](https://file+.vscode-resource.vscode-cdn.net/home/mmann1123/Documents/github/WB-spatial-features/~/miniconda3/envs/crop_pred/lib/python3.9/site-packages/geowombat/core/geoxarray.py:789) mode=mode,
[790](https://file+.vscode-resource.vscode-cdn.net/home/mmann1123/Documents/github/WB-spatial-features/~/miniconda3/envs/crop_pred/lib/python3.9/site-packages/geowombat/core/geoxarray.py:790) nodata=nodata,
...
[331](https://file+.vscode-resource.vscode-cdn.net/home/mmann1123/Documents/github/WB-spatial-features/~/miniconda3/envs/crop_pred/lib/python3.9/site-packages/rasterio/__init__.py:331) )
File rasterio/_io.pyx:1420, in rasterio._io.DatasetWriterBase.__init__()
RasterBlockError: The height and width of dataset blocks must be multiples of 16
for reference I am using Q1 data for the south here:
https://www.dropbox.com/scl/fo/py3ez4i7obk1j4id6u42c/h?rlkey=usuklp6tdi0mhs9997twwr187&dl=0
from geowombat.
mmann1123
commented on September 27, 2024
@jgrss adding
with gw.config.update(blockxsize=512, blockysize=512):doesn't seem to resolve it either
from geowombat.
mmann1123
commented on September 27, 2024
Huh... interesting. If I drop nodata from gw.open it works...
from geowombat.
jgrss
commented on September 27, 2024
Thanks for raising. There are probably changes to the array chunk size happening in the 'no data' setting, after you have set the chunks at the user level. I'll look into it.
from geowombat.
jgrss
commented on September 27, 2024
The sub-folders in https://www.dropbox.com/scl/fo/py3ez4i7obk1j4id6u42c/h?rlkey=usuklp6tdi0mhs9997twwr187&dl=0 are empty.
from geowombat.
mmann1123
commented on September 27, 2024
Ok sorry about that. Here's a stand-alone script and a link to data I wont delete until this is resolved.
Access data here
from glob import glob
import geowombat as gw
import os
from numpy import nan
os.chdir("~/Dropbox/share_geowombat")
os.makedirs("mosaic", exist_ok=True)
a_quarter = './*.tif'
with gw.open(a_quarter, mosaic=True,overlap='max', nodata= nan) as src:
gw.save(src, filename=f'./mosaic/S2_SR_1_south.tif',
nodata=nan,
overwrite=True,
num_workers=6,
compress='lzw')from geowombat.
barguzin
commented on September 27, 2024
Running into the same issue when trying to save the geotiff with GeoWombat. Coded the same pipeline in rioxarray and geowombat (read - clip - save). The code fails on gw.save and gw.to_raster with the following corresponding error messages:
- ValueError: Source shape (1, 228) is inconsistent with given indexes 1 (gw.to_raster)
- RasterBlockError: The height and width of dataset blocks must be multiples of 16 (gw.save)
Using 'gw.config.update(blockxsize=512, blockysize=512)' did not have any effect. The rioxarray code completes without error.
from geowombat.
jgrss
commented on September 27, 2024
Thanks both -- in @mmann1123's case, the issue stemmed from the 'band' dimension dropping from single-band arrays. @barguzin did you happen to be using single-band rasters?
I think I have a fix in #304. @mmann1123 this appeared to mosaic your data correctly, but could you confirm using branch jgrss/issue237_nodata?
from geowombat.
mmann1123
commented on September 27, 2024
Sure I'll try to take a look soon. Might be able to tonight, but traveling tomorrow.
Thinking we should add a test to catch issues.
Could we just add a save line to existing open, mosaic etc tests? Or should they be separate?
from geowombat.
barguzin
commented on September 27, 2024
@jgrss yes, this was a single-band raster.
from geowombat.
mmann1123
commented on September 27, 2024
We should have this resolved soon
…On Fri, Apr 19, 2024 at 10:26 PM barguzin ***@***.***> wrote:
@jgrss <https://github.com/jgrss> yes, this was a single-band raster.
—
Reply to this email directly, view it on GitHub
<#237 (comment)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/ABHR6VBV6EUO2JUYY65XKY3Y6HG6XAVCNFSM6AAAAAAUHAPIGKVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDANRXGUYTQMRUG4>
.
You are receiving this because you were mentioned.Message ID:
***@***.***>
from geowombat.
Related Issues (20)
- Failed to Install Geowombat on pip and conda HOT 12
- Requirement differences between package & conda-forge HOT 6
- Coordinating efforts on potential new geospatial Xarray accessor HOT 2
- Regression capability in geowombat HOT 3
- ARM Installation Issues HOT 9
- Apply interpolation HOT 4
- Stac import HOT 2
- series.apply big tiff / bad block
- EVI equation is wrong HOT 2
- Support for pyproj 3.4.1? HOT 8
- Slow extract HOT 2
- mosaic bounds HOT 4
- fit_predict
- Stack & Mosaic Advice? HOT 6
- writing with Bigtiff HOT 2
- [] in open trigger stacking
- polygon_to_array returns shorter y dimension than source rasters HOT 26
- band names HOT 1
- Unable to import geowombat HOT 1
- Mosaic Fails with nan nodata HOT 1
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from geowombat.