nozzle / throttler Goto Github PK

Will the BatchedThrottler function allow grouping of jobs in batches and calling a function between those batches?

For example if I have 10 files, and want to process them in groups of 4 concurrently but call an aggregation function after files 4, 8, and 10.

I want to run many func() error goroutines and stop throttling as soon as any returns an error.

Do you see any issues with exporting maxWorkers?

I'd to adjust the number of workers after t.Throttle() based on some metrics.

Hi,

is it possible to have another function like this one New(2, len(urls)) without providing the number of tasks in total?

Thanks.

panic: makechan: size out of range goroutine 4159 on vendor/github.com/nozzle/throttler/throttler.go:47

The current test cases while providing full code coverage don't fully check that the number of workers is throttled at max workers or that it waits for all jobs to finish before exiting Throttle(). These have all been checked and verified in production, but it'd be nice to have formal test case proof.

WARNING: DATA RACE
Read at 0x00c42054c3e0 by goroutine 42:
code.uber.internal/driver/fleet_finance/vendor/github.com/nozzle/throttler.(*Throttler).Throttle()
/mnt/jenkins/workspace/test-fleet_finance/.tmp/.goroot/src/code.uber.internal/driver/fleet_finance/vendor/github.com/nozzle/throttler/throttler.go:84 +0x1e3

Previous write at 0x00c42054c3e0 by goroutine 43:
code.uber.internal/driver/fleet_finance/vendor/github.com/nozzle/throttler.(*Throttler).Done()
/mnt/jenkins/workspace/test-fleet_finance/.tmp/.goroot/src/code.uber.internal/driver/fleet_finance/vendor/github.com/nozzle/throttler/throttler.go:95 +0x1c8

func (t *Throttler) Done(err error) {
t.doneChan <- struct{}{}
if err != nil {
t.errsMutex.Lock()
t.errs = append(t.errs, err)
t.errorCount++
t.errsMutex.Unlock()
}
}

Should t.doneChan <- struct{}{} after if err != nil?