go-playground / pool Goto Github PK
View Code? Open in Web Editor NEW:speedboat: a limited consumer goroutine or unlimited goroutine pool for easier goroutine handling and cancellation
License: MIT License
:speedboat: a limited consumer goroutine or unlimited goroutine pool for easier goroutine handling and cancellation
License: MIT License
Hey @joeybloggs, could you please take a look at this and let me know what you think -
https://github.com/sudo-suhas/bulk-marshal? I don't have enough expertise to decipher the relatively poor performance of this library compared to goroutines with channels and wait groups. Would very much like to know your take on this.
I think it's better to make a timeout operation When use batch work
Came a http request, how to open a pool to deal with. Can you give me an example? In the process, suddenly came an http request, how to insert into the queue.
"Batch Work" example from readme
go get
go build
.\main.go:12: undefined: pool.NewLimited
.\main.go:44: cannot use func literal (type func(pool.WorkUnit) (interface {}, error)) as type pool.WorkFunc in return argument
.\main.go:50: wu.IsCancelled undefined (type pool.WorkUnit has no field or method IsCancelled)
In the Batch Work example,The email.Value will be nil when called batch.Cancel() that will cause Panic as a result of wrong type desertation
for email := range batch.Results() {
if err := email.Error(); err != nil {
// handle error
// maybe call batch.Cancel()
}
// use return value
// The email.Value will be nil when called batch.Cancel() that will cause Panic as a result of wrong type desertation
fmt.Println(email.Value().(bool))
}
It might be good to write like this
for email := range batch.Results() {
if err := email.Error(); err != nil {
// handle error
// maybe call batch.Cancel()
}else{
// use return value
fmt.Println(email.Value().(bool))
}
Hey @joeybloggs, thanks for this package. The abstractions make it dead simple to start and manage worker pools.
How do you recommend I use the pool instance? Should I create and reuse it(by making it package scope) if I am only ever going to create and use a batch inside a function? Here's the code:
package util
import (
"runtime"
jsoniter "github.com/json-iterator/go"
"github.com/pkg/errors"
log "github.com/sirupsen/logrus"
"gopkg.in/go-playground/pool.v3"
)
// jsoniter.ConfigFastest marshals the float with 6 digits precision (lossy),
// which is significantly faster. It also does not escape HTML.
var json = jsoniter.ConfigFastest
// Unmarshal uses jsoniter for efficiently unmarshalling the byte stream into
// the struct pointer.
func Unmarshal(bs []byte, v interface{}) error {
// See https://github.com/json-iterator/go/blob/master/example_test.go#L69-L88
iter := json.BorrowIterator(bs)
defer json.ReturnIterator(iter)
iter.ReadVal(v)
if iter.Error != nil {
log.WithError(iter.Error).
Error("Got error while trying to unmarshal value into given struct")
return errors.Wrap(iter.Error, "util: unmarshal using jsoniter.ConfigFastest failed")
}
return nil
}
func unmarshalWorker(bs []byte, val interface{}) pool.WorkFunc {
return func(wu pool.WorkUnit) (interface{}, error) {
if wu.IsCancelled() {
// return values not used
return nil, nil
}
return nil, Unmarshal(bs, val)
}
}
// BulkUnmarshal uses worker pool
func BulkUnmarshal(bytesSlice [][]byte, vals []interface{}) error {
if len(bytesSlice) != len(vals) {
return errors.New(
"util: bulk unmarshal failed: length of bytes slice did not match targets",
)
}
p := pool.NewLimited(uint(runtime.NumCPU() * 2))
defer p.Close()
batch := p.Batch()
go func() {
for idx, bs := range bytesSlice {
batch.Queue(unmarshalWorker(bs, vals[idx]))
}
// DO NOT FORGET THIS OR GOROUTINES WILL DEADLOCK
// if calling Cancel() it calles QueueComplete() internally
batch.QueueComplete()
}()
for res := range batch.Results() {
if err := res.Error(); err != nil {
batch.Cancel()
return errors.Wrap(err, "util: bulk unmarshal failed")
}
}
return nil
}
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.