Comments (15)
from rayon.
You could put the map under a mutex
ah, thanks!
from rayon.
@flying-sheep I personally would not do this because of performance issues.
from rayon.
Collecting is likely very fast, mutez maybe not, but it depends on how many items and the relative cost of inserting them vs creating them. You could also achieve the async pattern by making a port and having the parallel iterator send msgs to the port (and starting a thread too). I actually do intend to add some kind of combinators or helpers like this but am not sure just how they should look.
On Feb 18, 2016 2:47 PM, Lilian Anatolie Moraru [email protected] wrote:@flying-sheep I personally would not do this because of performance issues.
—Reply to this email directly or view it on GitHub.
from rayon.
ok, then i’ll reopen this.
i don’t really like the idea of having to collect things just to go over them and building a map from them 😐
from rayon.
I think the right solution is to just pass the buck to a ConcurrentHashMap
. I don't know of any good implementations as of now, though.
from rayon.
I've got a somewhat related problem:
I'm using Rayon to count duplicate files within a directory. To keep a progress bar updated, I would like to send the current progress of the operation to a different thread that displays the progress bar (since updating the progress bar from within the thread that does the work is very slow). I've reduced the example to the following code:
https://gist.github.com/GyrosOfWar/059b70688904c6b5b433
The issue I'm having is that I can't share the Sender in the for_each closure, since it's not Sync
. The usual way would be to clone the Sender, but in this case, it doesn't seem to help. I'm pretty sure the problem is just me not understanding Rust's concurrency quite right, but I can't figure this out and Google is not helping (since cloning the Sender doesn't help)
from rayon.
@GyrosOfWar interesting. So this is actually just a bit tricky. (As an aside, I wonder if there is a good reason that the channel is not Sync
, I should check.) What you have to do is to clone the sender once for each directory, so something like:
let senders: Vec<_> = nums.iter().map(|_| sender.clone()).collect();
nums.into_par_iter().zip(senders).for_each(move |i, sender| {
let d = rx.clone();
d.send(i).unwrap();
});
but this seems sub-optimal. If nothing else, it might be nice to have a kind of "repeater" iterator for cloning things that are not Send only as needed. In other words, it would work something like
nums.into_par_iter().zip(repeat(sender)).for_each(|(num, sender_ref)| ...)
where sender_ref
has type &Sender
. The idea would be that it takes ownership of the sender and clones it for each thread split, but then lends it out for sequential processing (so as to do the minimal number of clones possible). That ought to be fairly easy to whip-up.
Of course the hard part is to come up with a good name: it seems like there are a number of similar "adapters" one might want for distinct ownership semantics (e.g., one that clones eagerly, since cloning is cheap, etc)
from rayon.
Channels lazily become Sync internally when cloned, as an optimization for the case when they don't need to be Sync. There was a rejected RFC recently about adding a Sync channel function. The problem as seen by the libs team with exposing a Sender that is Sync as its own type is that all existing code would be incompatible with it, as well as being utterly confusing with the existing inaptly-namer SyncSender that means a completely different kind of synchronous.
from rayon.
.zip(repeat(sender))
Hmm, zip
requires an IndexedParallelIterator
, but repeat
doesn't know the length. It could lie and say usize::MAX
, but I think it might be better to have zip+repeat
be its own special method, kind of like enumerate
is nearly zip+range
.
from rayon.
Of course, it'd probably be easier to just not use the channel, and instead use one of the queue's available from crossbeam, e.g. https://github.com/aturon/crossbeam/blob/master/src/sync/ms_queue.rs.
from rayon.
Thanks for the help! I'm using that lock-free queue plus a scoped thread from the crossbeam crate now and it seems to work well. Haven't tested it a lot yet, though. For anyone who's interested, the code is here (specifically, the find_duplicates
method in src/dupfinder.rs
)
from rayon.
@GyrosOfWar I guess you mean this code? It seems a bit surprising though because I think that the spawned thread will be joined by the end of-scope, and hence I don't think that thread is really doing anything.
If however you move the into_par_iter
code up inside the scope
closure, that would probably work.
FWIW this WIP API https://github.com/nikomatsakis/rayon/pull/71 would also help, but obviously that's not landed yet.
from rayon.
PR #131 adds for_each_atomic
, which basically fulfills the original feature request.
from rayon.
Sorry, I should have referred to https://github.com/nikomatsakis/rayon/pull/124, which specifically adds ability to collect into a hashmap.
from rayon.
Related Issues (20)
- Suggestion for stack size when calling par_sort_unstable HOT 5
- Tag missing for release 1.10.0 HOT 1
- Unable to parallelize properly using `par_iter` or `par_bridge` HOT 8
- Documentation for `ParallelIterator::fold` and/or `ParallelIterator::reduce` seam to be incorrect HOT 5
- Slowdown when using deeply nested vector HOT 6
- Potential `len` method ambiguity in `src/range.rs` HOT 5
- Potential to fix memory ordering for value HOT 2
- a broadcast that only gets executed on idle threads
- `in_place_scope` documentation is confusing/unclear HOT 14
- use_current_thread and the global thread pool HOT 2
- Why does ThreadPool block until a operation is finished? HOT 1
- A way to hook before/after job execution (and/or a way to see a number of tasks in the processing queue)
- Function dependencies with Rayon without blocking the thread / Rayon synchronization primitives?
- Add synchronization guarantees for `ParallelIterator::for_each*` HOT 1
- Par bridge with optional buffering IO handling HOT 2
- Thread pool without work stealing HOT 6
- Question about return types from .map() HOT 4
- Terribly inefficient design and possible solution HOT 5
- [Discussion] Is it possible to impl `IntoIterator` for `ParallelIterator`? HOT 3
- Yield locks HOT 1
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from rayon.