ParallelTask window_id actually chunk_id? about geowombat HOT 6 CLOSED

Ok great. Can we close this?

from geowombat.

A a simple solution we could add a chunk id in the generator as follows:

class ParallelTask

def map(self, func, *args):
    ....
    for cidx, wchunk in enumerate(range(0, self.n_windows, self.n_chunks)):

                data_gen = ((self.data[slice_], cidx, widx, *args) for widx, slice_ in enumerate(window_slice))

although I am admittedly a bit confused about chunk vs window, especially as this relates to pt.n_chunks pt.n_windows

from geowombat.

Found a better option (see PR), the following will ensure that the window_id increments from 0 the n_windows without repeat.

class ParallelTask

def map(self, func, *args):
    ....
    for cidx, wchunk in enumerate(range(0, self.n_windows, self.n_chunks)):

                data_gen = ((self.data[slice_], cidx*self.n_chunks + widx, *args) for widx, slice_ in enumerate(window_slice))

from geowombat.

Nice catch. I should have used batches instead of chunks. You can think of them as mini batches of windows passed to the parallel executor. I added it a while back to limit memory overhead.

I think we just need the following, right?

for wchunk in range(0, self.n_windows, self.n_chunks):

    data_gen = ((self.data[slice_], widx+wchunk, *args) for widx, slice_ in enumerate(window_slice))

from geowombat.

from geowombat.

@mmann1123 The window id was also incorrect in to_raster. Both cidx*self.n_chunks + widx and widx+wchunk produced the same result. The fix was pushed in 911145ecc92bad22f06d0326cadf99d46ec893f2.

from geowombat.