Comments (18)
spark.rss.storage.type
is client's configuration.rss.storage.type
is server's configuration.
from firestorm.
Both parameters affect the storage mode of ShuffleServer? Then,which one has a higher priority?
from firestorm.
Both parameters affect the storage mode of ShuffleServer? Then,which one has a higher priority?
Only rss.storage.type
affects.
from firestorm.
What is spark.rss.storage.type
used for?
from firestorm.
What is
spark.rss.storage.type
used for?
client
from firestorm.
Does the client also can store data to a local disk or HDFS?
from firestorm.
Does the client also can store data to a local disk or HDFS?
Client need to know where to read data.
from firestorm.
So these two parameters need be set to the same?
from firestorm.
So these two parameters need be set to the same?
Yes
from firestorm.
ok,thanks
from firestorm.
I have a problem,when I set storageType = HDFS, Error: Blocks read Inconsistent: Expected 61 Blocks, actual 0 Blocks
The version I'm using is 0.2.0
from firestorm.
The configuration items are as follows
rss.rpc.server.port 19969
rss.jetty.http.port 19968
rss.storage.basePath /rss-0.2.0/shuffle-data1,/rss-0.2.0/shuffle-data2
rss.storage.type HDFS
rss.coordinator.quorum xxx:19999,xxx:19999,xxx:19999
rss.server.buffer.capacity 4gb
rss.server.buffer.spill.threshold 2gb
rss.server.partition.buffer.size 15mb
rss.server.read.buffer.capacity 2gb
rss.server.flush.thread.alive 5
rss.server.flush.threadPool.size 10
rss.server.health.check.enable false
#multistorage config
rss.server.multistorage.enable true
rss.server.uploader.enable true
rss.server.uploader.base.path hdfs:/firestorm/shuffle-data
rss.server.uploader.thread.number 3
rss.server.disk.capacity 5115506975
Spark client
spark.shuffle.manager org.apache.spark.shuffle.RssShuffleManager
spark.rss.coordinator.quorum xxx:19999,xxx:19999,xxx:19999
spark.rss.storage.type HDFS
from firestorm.
We don't recommend the storageType HDFS, you would better to use MEMORY_LOCALE.
from firestorm.
Storage HDFS will be removed soon.
from firestorm.
Is there some problems with HDFS storage mode, so you want to remove it?
from firestorm.
In terms of the results of my tests, only when I set storage type = LOCALFILE_HDFS that the shuffle data will be written to HDFS.
But I also find, in this case,if I delete local data, the same exception 'Blocks read Inconsistent' is reported, so i want to figure out if the function of reading shuffle data from HDFS is not sound enough so far.
Thanks for your time!
from firestorm.
Is there some problems with HDFS storage mode, so you want to remove it?
HDFS mode have poor performance. HDFS can work correctly.
from firestorm.
Is there some problems with HDFS storage mode, so you want to remove it?
HDFS mode have poor performance. HDFS can work correctly.
Yes, you're right. I've got some configurations set incorrectly.
from firestorm.
Related Issues (20)
- Whether multiple disks are supported for local storage? HOT 4
- duplicate servlets map in Coordinator Server
- 使用firestorm-0.4.0 运行spark3.1.1官方的JavaWordCount报如下错误,并且在yarn-client模式下driver端进程一直不退出 HOT 10
- yarn-client模式下driver端进程一直不退出 HOT 9
- In local mode, why directory should be deleted first? HOT 1
- [QUESTION] 依赖Hadoop环境? HOT 3
- [QUESTION] Executor在shuffle write/read 过程中是否落本地盘? HOT 2
- [Feature Request]Add a web UI in Coordinated Server to show the detailed server/job/metrics information HOT 1
- hardcoded relative paths HOT 6
- Whether local multiple replicas are supported? HOT 2
- Compared to the native spark, the shuffle write data of firestorm is always smaller HOT 2
- Unexpected crc value for blockId[474989042101783], expected:1518107711, actual:3331113690 HOT 5
- Shuffle read does not read all data completely? HOT 31
- Support shuffle data replica? HOT 5
- Coordinator HA problem HOT 6
- fault tolerance HOT 4
- Clear buffered data when acquiring memory failed and then retry
- To support more tasks with Firestorm
- how to enter into uniffle wechat or dingtalk?
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from firestorm.