org.apache.spark.SparkException: Error sending message [message = RemoveExecutor(0)]

Google Groups | hart jo | 4 months ago
  1. 0

    Sparkling water executor error

    Google Groups | 4 months ago | hart jo
    org.apache.spark.SparkException: Error sending message [message = RemoveExecutor(0)]
  2. 0

    Talend Open Integration Solution | 6 months ago | lei ju
    org.apache.spark.SparkException: Error sending message [message = StopAllReceivers]
  3. 0

    [SPARK-6980] [CORE] Akka timeout exceptions indicate which conf controls them (RPC Layer) by BryanCutler · Pull Request #6205 · apache/spark · GitHub

    github.com | 3 months ago
    org.apache.spark.SparkException: Error sending message [message = (BlockManagerHeartbeat(BlockManagerId(driver, localhost, 51109)),600 seconds)]
  4. Speed up your debug routine!

    Automated exception search integrated into your IDE

  5. 0

    While testing spark jobs on VM we noticed that the spark job logs a lot of heartbeat retries messages in master log. Here is the stacktrace: Spark program ran fine though. {code} 2016-04-29 05:04:05,963 - WARN [driver-heartbeater:o.a.s.Logging$class@91] - Error sending message [message = Heartbeat(driver,[L scala.Tuple2;@6d61f2ad,BlockManagerId(driver, localhost, 49484))] in 3 attempts org.apache.spark.SparkException: Could not find HeartbeatReceiver or it has been stopped. at org.apache.spark.rpc.netty.Dispatcher.postMessage(Dispatcher.scala:161) ~[co.cask.cdap.spark-assembly-1.6.1.jar:na] at org.apache.spark.rpc.netty.Dispatcher.postLocalMessage(Dispatcher.scala:126) ~[co.cask.cdap.spark-assembly-1.6.1.jar:na ] at org.apache.spark.rpc.netty.NettyRpcEnv.ask(NettyRpcEnv.scala:227) ~[co.cask.cdap.spark-assembly-1.6.1.jar:na] at org.apache.spark.rpc.netty.NettyRpcEndpointRef.ask(NettyRpcEnv.scala:511) ~[co.cask.cdap.spark-assembly-1.6.1.jar:na] at org.apache.spark.rpc.RpcEndpointRef.askWithRetry(RpcEndpointRef.scala:100) ~[co.cask.cdap.spark-assembly-1.6.1.jar:na] at org.apache.spark.executor.Executor.org$apache$spark$executor$Executor$$reportHeartBeat(Executor.scala:449) [co.cask.cda p.spark-assembly-1.6.1.jar:na] at org.apache.spark.executor.Executor$$anon$1$$anonfun$run$1.apply$mcV$sp(Executor.scala:470) [co.cask.cdap.spark-assembly -1.6.1.jar:na] at org.apache.spark.executor.Executor$$anon$1$$anonfun$run$1.apply(Executor.scala:470) [co.cask.cdap.spark-assembly-1.6.1. jar:na] at org.apache.spark.executor.Executor$$anon$1$$anonfun$run$1.apply(Executor.scala:470) [co.cask.cdap.spark-assembly-1.6.1. jar:na] at org.apache.spark.util.Utils$.logUncaughtExceptions(Utils.scala:1765) [co.cask.cdap.spark-assembly-1.6.1.jar:na] at org.apache.spark.executor.Executor$$anon$1.run(Executor.scala:470) [co.cask.cdap.spark-assembly-1.6.1.jar:na] at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [na:1.7.0_75] at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:304) [na:1.7.0_75] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178) [ na:1.7.0_75] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) [na:1.7. 0_75] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [na:1.7.0_75] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [na:1.7.0_75] Caused by: org.apache.spark.SparkException: Could not find HeartbeatReceiver or it has been stopped. at org.apache.spark.rpc.netty.Dispatcher.postMessage(Dispatcher.scala:161) ~[co.cask.cdap.spark-assembly-1.6.1.jar:na] at org.apache.spark.rpc.netty.Dispatcher.postLocalMessage(Dispatcher.scala:126) ~[co.cask.cdap.spark-assembly-1.6.1.jar:na] at org.apache.spark.rpc.netty.NettyRpcEnv.ask(NettyRpcEnv.scala:227) ~[co.cask.cdap.spark-assembly-1.6.1.jar:na] at org.apache.spark.rpc.netty.NettyRpcEndpointRef.ask(NettyRpcEnv.scala:511) ~[co.cask.cdap.spark-assembly-1.6.1.jar:na] at org.apache.spark.rpc.RpcEndpointRef.askWithRetry(RpcEndpointRef.scala:100) ~[co.cask.cdap.spark-assembly-1.6.1.jar:na] ... 13 common frames omitted {code}

    Cask Community Issue Tracker | 7 months ago | Rohit Sinha
    org.apache.spark.SparkException: Could not find HeartbeatReceiver or it has been stopped.
  6. 0

    Apache Spark User List - GC overhead limit exceeded

    nabble.com | 3 months ago
    org.apache.spark.SparkException: Could not find HeartbeatReceiver or it has been stopped.

  1. Nikolay Rybak 1 times, last 1 month ago
  2. tyson925 1 times, last 7 months ago
1 unregistered visitors
Not finding the right solution?
Take a tour to get the most out of Samebug.

Tired of useless tips?

Automated exception search integrated into your IDE

Root Cause Analysis

  1. org.apache.spark.SparkException

    Error sending message [message = RemoveExecutor(0)]

    at org.apache.spark.rpc.RpcEndpointRef.askWithRetry()
  2. org.apache.spark
    RpcEndpointRef.askWithRetry
    1. org.apache.spark.rpc.RpcEndpointRef.askWithRetry(RpcEndpointRef.scala:118)
    2. org.apache.spark.rpc.RpcEndpointRef.askWithRetry(RpcEndpointRef.scala:77)
    2 frames
  3. Spark
    CoarseGrainedSchedulerBackend$DriverEndpoint$$anonfun$receiveAndReply$1.applyOrElse
    1. org.apache.spark.storage.BlockManagerMaster.tell(BlockManagerMaster.scala:225)
    2. org.apache.spark.storage.BlockManagerMaster.removeExecutor(BlockManagerMaster.scala:40)
    3. org.apache.spark.scheduler.cluster.CoarseGrainedSchedulerBackend$DriverEndpoint.removeExecutor(CoarseGrainedSchedulerBackend.scala:276)
    4. org.apache.spark.scheduler.cluster.CoarseGrainedSchedulerBackend$DriverEndpoint$$anonfun$receiveAndReply$1.applyOrElse(CoarseGrainedSchedulerBackend.scala:186)
    4 frames
  4. org.apache.spark
    Dispatcher$MessageLoop.run
    1. org.apache.spark.rpc.netty.Inbox$$anonfun$process$1.apply$mcV$sp(Inbox.scala:104)
    2. org.apache.spark.rpc.netty.Inbox.safelyCall(Inbox.scala:204)
    3. org.apache.spark.rpc.netty.Inbox.process(Inbox.scala:100)
    4. org.apache.spark.rpc.netty.Dispatcher$MessageLoop.run(Dispatcher.scala:215)
    4 frames
  5. Java RT
    ThreadPoolExecutor.runWorker
    1. java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
    1 frame