escribapetrus / mapreduce Goto Github PK

As it is implemented, concurrent writes to the same files can cause race conditions and loss of data.

We have to implement safe concurrency for file IO, with something like locks or isolated file io processes.

Many things are repeated in mapper and in reducer. We can refactor, for example:

• create a data_item type: {binary(), binary()}
• create a single signature for worker functions map and reduce: (data_item()) -> data_item(), list(data_item())
• create a mapreduce worker behavior that abstracts identical code in mapper and reducer.