dicl / veloxmr Goto Github PK

@zwigul can we use unique_ptr instead of shared_ptr in here?
https://github.com/DICL/EclipseMR/blob/2336cf253dbbb7f46ba1a0ca7ed3f78fa40bea77/src/mapreduce/nodes/peermr.cc#L209

From now on EclipseMR takes changes from EclipseCACHE, we have to fix the history.

This is the structure of intermediate data at reducer side.

Intermediate data
= (idata of key range 1, idata of key range 2, ..., idata of key range N)

idata of key range k
= (iblock 1, iblock 2, ..., iblock M)

Currently, for any idata of a specific key range, key and values are sorted within each iblock, but not across the entire iblocks.
As a result, a reducer will get a key with "partially sorted" list of values.

• It is required to sort values across the iblocks externally (the original eclipseMR is also missing this feature).

I evaluated of the performance of IWriter itself without network parts.

Test Method

This test mimics the procedure that messages::KeyValueShuffle instances arrives at TaskExecutor::write_key_value function and the arrived key-values are written into the object of IWriter class as follows:

1. Generate 256 MB of key and values. The generation solely takes around 30 seconds.
2. On every key-value generation, call IWriter::add_key_value function with the generated key-value.

We've been tested VeloxMR with kmeans using 10 GB data on 40 nodes, where the amount of shuffled data on each node is approximately 256 MB, so this test is performed with 256 MB key-values.

Parameters

• fine-lock (enable/disable): Fine-grained locking (#45).
• std::move (enable/disable): To avoid data duplication when we insert key-values to iblock.
• iblock_size: The experiments are done with 4 MB, 8 MB, 16 MB, 32 MB and 64 MB of iblock size.

Result

I ran each test 8 times and take the average.

• IWriter::add_key_value function is called over 10 million times, and it takes no more than 2 minutes in total (regarding the key-value generation takes around 30 seconds).
• At least up to 64 MB, the performance improves as increasing iblock_size.
• As iblock_size increases, the performance gap among different optimization schemes (fine-lock and std::move) decreases.
• It seems that std::move rather degrades the performance.

This release should be fully compatible with Eclipse DFS 1.0.* series

• : save to disk intermediate data classes
• : DataSet class
• : Map functionability
• : Reduce functionability
• : How to copy the dynamic library of the application to the remote servers, dfs maybe?