Git Product home page Git Product logo

Comments (10)

FMX avatar FMX commented on September 22, 2024

HA 模式下,启动是不需要 rss://master-ip:9097 的。因为配置文件里面已经写了所有的地址。
@CCweixiao

from incubator-celeborn.

CCweixiao avatar CCweixiao commented on September 22, 2024

Spark作业中呢,地址spark.rss.master.address好像这里也只能配置一个

from incubator-celeborn.

CCweixiao avatar CCweixiao commented on September 22, 2024

HA 模式下,启动是不需要 rss://master-ip:9097 的。因为配置文件里面已经写了所有的地址。 @CCweixiao

嗯嗯 已经尝试,可以啦

from incubator-celeborn.

FMX avatar FMX commented on September 22, 2024

Spark作业中呢,地址spark.rss.master.address好像这里也只能配置一个

这里会考虑变成支持配置多个节点的形式,只能配置一个确实存在节点宕机后不能使用的问题。

from incubator-celeborn.

waitinfuture avatar waitinfuture commented on September 22, 2024

HA 模式下,启动是不需要 rss://master-ip:9097 的。因为配置文件里面已经写了所有的地址。 @CCweixiao

这里确实可以改进, 开个issue吧 @FMX

from incubator-celeborn.

FMX avatar FMX commented on September 22, 2024

#26

from incubator-celeborn.

CCweixiao avatar CCweixiao commented on September 22, 2024

#26

收到 感谢感谢

from incubator-celeborn.

wangshengjie123 avatar wangshengjie123 commented on September 22, 2024

@CCweixiao hello,我提交了个pr #43 ,初步解决了这个问题,可以试试。

其实,我们最开始的的想法是:
1、引导大家区别配置Ha和non-ha的配置(虽然修复了客户端可以指定rss.master.address多个节点)
2、启动时,尽量使用rss-default.conf来解决问题

例如ha场景下:
客户端配置:
spark.rss.ha.master.hosts master1,master2,master3
spark.rss.master.port 9097
服务端配置:
rss.ha.master.hosts master1,master2,master3
rss.master.port 9097

non-ha场景下:
客户端配置:
spark.rss.master.address master:9097
服务端配置:
rss.master.address master:9097

这样我们启动master时就不用使用 sh start-master.sh rss://master1:9097[,rss://master2:9097]

from incubator-celeborn.

CCweixiao avatar CCweixiao commented on September 22, 2024

感谢您的回复

目前我们master的配置就是这样的,
rss.ha.master.hosts master1,master2,master3
spark.rss.master.port 9097
启动master时没有指定master的地址,
只是,客户端之前只能指定一个master的IP和端口

from incubator-celeborn.

wangshengjie123 avatar wangshengjie123 commented on September 22, 2024

感谢您的回复

目前我们master的配置就是这样的, rss.ha.master.hosts master1,master2,master3 spark.rss.master.port 9097 启动master时没有指定master的地址, 只是,客户端之前只能指定一个master的IP和端口

不好意思,回复有些迟,之前的服务端配置有些错误,spark.rss.master.port 9097,spark前缀印象中可以去掉。
咱们客户端也可以配置多个host,只是目前只能统一一个port,配置加个spark.前缀就好

from incubator-celeborn.

Related Issues (20)

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.