coflow / varys Goto Github PK

Varys control plane already uses netty through Akka.

Consider doing the same for dataplane, specially for overloaded cases. Points of entry: varys.framework.DataServer and varys.framework.client.VarysClient

Set varys.master.numInstances > 1 and run the BroadcastService example with a large file (e.g., 32MB) should reproduce the bug. I suspect it to be a race condition somewhere.

varys.master.numInstances determines the number of MasterActor, which should've helped with processing too many messages. Setting it to 1 isn't really that bad because most of the time is spent in (de)serialization and not in actual action.

It's really a tedious work to revise all the 'send' and 'receive' calls in the yarn framework, is there any ongoing work on this? how did the experiments in the varys paper conduct? i just want to have a look at the flow characteristics of yarn and spark.

Thanks
-Hui