shuffle write is too slow about firestorm HOT 4 CLOSED

In our production environment, the size of shuffle is usually hundreds of GBs. For very small shuffles, Firestorm will be slower than native shuffles. But there seems to be room for optimization.

from firestorm.

@packageman with firestorm, there are some RPC calls to send data, as you mentioned, requirePreallocation etc. Your case is the same as we expected, with small shuffle data, Firestorm will be a little slower than origin shuffle.
For reference, Firestorm is target to :

1. Support Spark on K8s things
2. Improve job's performance/stability with heavy shuffle workload, eg, shuffle data = TB, partition num = 50000+

from firestorm.

Support Spark on K8s things

Currently, we are running Spark on k8s.

with small shuffle data, Firestorm will be a little slower than origin shuffle

It looks not only a little slower but also a lot slower(2mins vs 5.6mins), nearly 2 times. I understand that grpc call will take some extra time, but the total time is more than double that of the native shuffle when working with small shuffle data, which is not consistent with your description.

with native shuffle:

with Firestorm:

from firestorm.

@packageman with your case, 1W+task total, the delay should be depend on task number of parallels, if every task is slower about 0.5s with Firstorm, then

1. 1W parallels task should be slower 0.5s total
2. 1000 parallels task should be slower 5s total
3. 100 parallels task should be slower 50s total
   It's the behave we known, Firstorm is not target to improve performance with any situation. Platform can decide if use Firestorm with all Spark job, or with specific job.
   From your case, input data for every task is less than 1MB, shuffle write data is less than 1kb. Optimize job to have more data for every task will be better.

from firestorm.