fault tolerance about firestorm HOT 4 CLOSED

1. Spark client can be configured to send shuffle data to multiple shuffle servers, and spark task won't fail if one shuffle server fail. But it will cost more storage space depend on configuration. We'll add documents to describe the configuration later.
2. Yes, shuffle data will be written to different drives according to configuration, eg, rss.storage.basePath /data1/rssdata,/data2/rssdata, /data1/rssdata and /data2/rssdata are mounted with different disk drivers

from firestorm.

@colinmjj what if I'm trying to deploy a code change, or a config change, and need to restart the process to pick up this change? Mostly curious if the shuffle data stored will persist and be picked up again. really appreciate the quick response

from firestorm.

@cpd85 The shuffle data exists if spark application is alive according to heartbeat, and data will be removed if there has no heartbeat.

from firestorm.

@colinmjj what if I'm trying to deploy a code change, or a config change, and need to restart the process to pick up this change? Mostly curious if the shuffle data stored will persist and be picked up again. really appreciate the quick response

Our project has moved to Apache Incubator, Our project's name changed from Firestorm to Uniffle. there is a similar issue apache/incubator-uniffle#263, if you have interest, you can see it.

I'll close this issue. thanks.

from firestorm.