Seeking Solutions for Spark Context Stopping Without Engine Termination #6992
-
Health Check Mechanism in Spark EngineIs there a health check mechanism between the Spark engine and the Spark context? I often encounter an issue where the Spark context stops, but the engine process does not terminate. I have reviewed issue 1800, but it only addresses the problem in cases of Out of Memory (OOM) errors. This does not fully resolve the issue I am facing. Currently, I am using version 1.7.3. Has the higher version of Kyuubi completely resolved this issue? Alternatively, is there a specific mechanism or configuration that can help avoid this problem? After examining the latest code, it seems there is no obvious health check mechanism implemented. If anyone has insights or solutions, I would greatly appreciate your input! |
Beta Was this translation helpful? Give feedback.
Replies: 2 comments 1 reply
-
|
had any exception message? |
Beta Was this translation helpful? Give feedback.
-
|
Non-daemon threads indeed block the Spark driver from exiting, so far no mechanism to ensure |
Beta Was this translation helpful? Give feedback.
-
|
Right now I’m just giving the engine enough resources — around 8 to 16GB of memory. Ever since I did that, it’s been running pretty smoothly. I’ll think about a cleaner solution later. |
Beta Was this translation helpful? Give feedback.
Non-daemon threads indeed block the Spark driver from exiting, so far no mechanism to ensure
SparkContextlives, it sounds good to have one.