Comments (6)
OK, I understand the problem completely now.
First, here is a workaround for your use case - increase maximum parallelism to 3 :
(defn fetch-missing-items [cache-response-flow]
(m/ap (let [[hit? flow] (m/?> 3 (m/group-by #(contains? % :result) cache-response-flow))]
(m/?> (if hit?
flow
(->> flow
(m/eduction (map :key) (partition-all 100))
fetch-item-details
(m/eduction cat (partition-all 25))
store-item-details
(m/eduction cat)))))))
Now, the issue. Currently, when group-by
's input crashes, the error is redirected on output. When the error is consumed, the active group consumers are terminated along with the main process. This ordering of events is problematic when the consumer backpressure prevents transfers (in this case, ?>
when max parallelism has been reached), because then the group consumers are kept alive and never terminate, which stalls the rest of the pipeline indefinitely.
Proposed fix : when group-by
's input crashes, propagate the error on output and terminate all active group consumers immediately (i.e don't wait for the main consumer to transfer the error).
from missionary.
Thanks for the explanation and workaround - I've hammered the loop for the last 30 minutes, and it's holding up perfectly.
from missionary.
Fixed in b.28
from missionary.
Repro project
https://github.com/lowecg/missionary-deadlock
from missionary.
Thank you for reporting this !
I've managed to reduce it further.
(m/? (m/reduce conj
(m/ap (m/?> (val (m/?> 2 (->> (m/seed (range))
(m/eduction (map (fn [x] (assert (< x 2)) x)))
(m/group-by identity))))))))
This doesn't terminate, it should fail immediately. This is a bug in either group-by
or ap
, related to input failure.
from missionary.
You're very welcome, Leo. Thank you for looking into this so quickly.
Fantastic news that you're able to isolate the issue.
from missionary.
Related Issues (20)
- Doc strings with empty or malformed code blocks
- Consider fairness for `sem`, `dfv`, `mbx`, `rdv`
- discourage unbounded queues - `mbx`, `blk`
- ClojureDart and Flutter HOT 2
- sci support HOT 5
- Recommended basic task usage in production
- Deref in task causing a silent Interrupted Exception HOT 2
- behavior of `ap` with huge parallelism HOT 3
- Lazify `ap` HOT 2
- Add type metadata to flow and task functions like `ap`, `sp`, `via` etc?
- `sem` with non-positive initial permit count ?
- `ap` bug HOT 2
- Cancel in example for basic walkthrough doesn't work HOT 3
- watch can discard latest value HOT 3
- Hello task example prints only Hello with Clojure repl HOT 4
- generalize `signal` to any semigroup HOT 1
- `observe` - on JVM, consider parking the caller thread instead of throwing when pipeline is busy HOT 2
- make `ap` switch consistent with `cp` HOT 1
- `stream` double subscriptions - document correct pattern + investigate bug
- cycles in continuous time - distill the essence of problem, find correct patterns and provide helpers
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from missionary.