Comments (9)
We shutdown the executorService in the methodRssShuffleManager.stop
. The stop is not called in your situation. In my opinion the method must be called. Could you ensure RssShuffleManager.stop
whether is be called or not.
from firestorm.
We shutdown the executorService in the method
RssShuffleManager.stop
. The stop is not called in your situation. In my opinion the method must be called. Could you ensureRssShuffleManager.stop
whether is be called or not.
RssShuffleManager.stop 方法没有被调用,这个方法应该是在 SparkSession.stop 方法中调用的,但是在我的程序里面没有显式的调用 SparkContext.stop 方法,所以也就没有调用 RssShuffleManager.stop 方法
from firestorm.
We shutdown the executorService in the method
RssShuffleManager.stop
. The stop is not called in your situation. In my opinion the method must be called. Could you ensureRssShuffleManager.stop
whether is be called or not.RssShuffleManager.stop 方法没有被调用,这个方法应该是在 SparkSession.stop 方法中调用的,但是在我的程序里面没有显式的调用 SparkContext.stop 方法,所以也就没有调用 RssShuffleManager.stop 方法
我觉得更好的方式应该是在程序里调用stop方法,而不是把这个线程设置为守护线程。调用stop逻辑也能让Spark更好的执行一些清理逻辑。
from firestorm.
We shutdown the executorService in the method
RssShuffleManager.stop
. The stop is not called in your situation. In my opinion the method must be called. Could you ensureRssShuffleManager.stop
whether is be called or not.RssShuffleManager.stop 方法没有被调用,这个方法应该是在 SparkSession.stop 方法中调用的,但是在我的程序里面没有显式的调用 SparkContext.stop 方法,所以也就没有调用 RssShuffleManager.stop 方法
我觉得更好的方式应该是在程序里调用stop方法,而不是把这个线程设置为守护线程。调用stop逻辑也能让Spark更好的执行一些清理逻辑。
是这样的,SparkContext 创建的时候是注册了 Shutdown Hook的,正常情况下 即使不调用stop方法,在任务执行完后也会由 ShutdownHookManager 来调用 stop 方法的。
但是由于这个线程没有设置成 守护线程,导致 jvm 误判程序还没有结束,因此就一直不会调用ShutdownHook,所以就会出现driver进程一直不退出的情况。
from firestorm.
这样话这个改成守护进程很合理,如果方便的话你可以提一个PR 修复这个问题。如果不方便的,我这边可以做一个PR修复这个问题。
from firestorm.
这样话这个改成守护进程很合理,如果方便的话你可以提一个PR 修复这个问题。如果不方便的,我这边可以做一个PR修复这个问题。
还是你们辛苦搞一下吧,我这边对提PR的规范不是很熟悉
from firestorm.
这样话这个改成守护进程很合理,如果方便的话你可以提一个PR 修复这个问题。如果不方便的,我这边可以做一个PR修复这个问题。
还是你们辛苦搞一下吧,我这边对提PR的规范不是很熟悉
OK,我这边修一下,其实这边有PR模版,照着填一下就可以。
from firestorm.
类似这种,介绍这个pr修改了什么,为什么需要这个修改,用户是否需要修改配置才能使用这个pr,以及怎么测试的?一般我们会是通过单测、手工测试等多种方式。
What changes were proposed in this pull request?
Heartbeat ExecutorService become daemon executor service.
Why are the changes needed?
If user don't call stop, the driver's shutdownHook won't called. The driver will hang.
Does this PR introduce any user-facing change?
No
How was this patch tested?
Manual test
from firestorm.
这样话这个改成守护进程很合理,如果方便的话你可以提一个PR 修复这个问题。如果不方便的,我这边可以做一个PR修复这个问题。
还是你们辛苦搞一下吧,我这边对提PR的规范不是很熟悉
OK,我这边修一下,其实这边有PR模版,照着填一下就可以。
了解了,多谢
from firestorm.
Related Issues (20)
- Whether multiple disks are supported for local storage? HOT 4
- duplicate servlets map in Coordinator Server
- 使用firestorm-0.4.0 运行spark3.1.1官方的JavaWordCount报如下错误,并且在yarn-client模式下driver端进程一直不退出 HOT 10
- What‘s the difference between `spark.rss.storage.type` and `rss.storage.type`? HOT 18
- In local mode, why directory should be deleted first? HOT 1
- [QUESTION] 依赖Hadoop环境? HOT 3
- [QUESTION] Executor在shuffle write/read 过程中是否落本地盘? HOT 2
- [Feature Request]Add a web UI in Coordinated Server to show the detailed server/job/metrics information HOT 1
- hardcoded relative paths HOT 6
- Whether local multiple replicas are supported? HOT 2
- Compared to the native spark, the shuffle write data of firestorm is always smaller HOT 2
- Unexpected crc value for blockId[474989042101783], expected:1518107711, actual:3331113690 HOT 5
- Shuffle read does not read all data completely? HOT 31
- Support shuffle data replica? HOT 5
- Coordinator HA problem HOT 6
- fault tolerance HOT 4
- Clear buffered data when acquiring memory failed and then retry
- To support more tasks with Firestorm
- how to enter into uniffle wechat or dingtalk?
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from firestorm.