org.apache.spark.SparkException: Error notifying standalone scheduler's driver endpoint

GitHub | joerwin | 5 months ago
  1. 0

    Null pointer getting count from avro file in mesos distributed

    GitHub | 5 months ago | joerwin
    org.apache.spark.SparkException: Error notifying standalone scheduler's driver endpoint
  2. 0

    While testing spark jobs on VM we noticed that the spark job logs a lot of heartbeat retries messages in master log. Here is the stacktrace: Spark program ran fine though. {code} 2016-04-29 05:04:05,963 - WARN [driver-heartbeater:o.a.s.Logging$class@91] - Error sending message [message = Heartbeat(driver,[L scala.Tuple2;@6d61f2ad,BlockManagerId(driver, localhost, 49484))] in 3 attempts org.apache.spark.SparkException: Could not find HeartbeatReceiver or it has been stopped. at org.apache.spark.rpc.netty.Dispatcher.postMessage(Dispatcher.scala:161) ~[co.cask.cdap.spark-assembly-1.6.1.jar:na] at org.apache.spark.rpc.netty.Dispatcher.postLocalMessage(Dispatcher.scala:126) ~[co.cask.cdap.spark-assembly-1.6.1.jar:na ] at org.apache.spark.rpc.netty.NettyRpcEnv.ask(NettyRpcEnv.scala:227) ~[co.cask.cdap.spark-assembly-1.6.1.jar:na] at org.apache.spark.rpc.netty.NettyRpcEndpointRef.ask(NettyRpcEnv.scala:511) ~[co.cask.cdap.spark-assembly-1.6.1.jar:na] at org.apache.spark.rpc.RpcEndpointRef.askWithRetry(RpcEndpointRef.scala:100) ~[co.cask.cdap.spark-assembly-1.6.1.jar:na] at org.apache.spark.executor.Executor.org$apache$spark$executor$Executor$$reportHeartBeat(Executor.scala:449) [co.cask.cda p.spark-assembly-1.6.1.jar:na] at org.apache.spark.executor.Executor$$anon$1$$anonfun$run$1.apply$mcV$sp(Executor.scala:470) [co.cask.cdap.spark-assembly -1.6.1.jar:na] at org.apache.spark.executor.Executor$$anon$1$$anonfun$run$1.apply(Executor.scala:470) [co.cask.cdap.spark-assembly-1.6.1. jar:na] at org.apache.spark.executor.Executor$$anon$1$$anonfun$run$1.apply(Executor.scala:470) [co.cask.cdap.spark-assembly-1.6.1. jar:na] at org.apache.spark.util.Utils$.logUncaughtExceptions(Utils.scala:1765) [co.cask.cdap.spark-assembly-1.6.1.jar:na] at org.apache.spark.executor.Executor$$anon$1.run(Executor.scala:470) [co.cask.cdap.spark-assembly-1.6.1.jar:na] at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [na:1.7.0_75] at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:304) [na:1.7.0_75] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178) [ na:1.7.0_75] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) [na:1.7. 0_75] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [na:1.7.0_75] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [na:1.7.0_75] Caused by: org.apache.spark.SparkException: Could not find HeartbeatReceiver or it has been stopped. at org.apache.spark.rpc.netty.Dispatcher.postMessage(Dispatcher.scala:161) ~[co.cask.cdap.spark-assembly-1.6.1.jar:na] at org.apache.spark.rpc.netty.Dispatcher.postLocalMessage(Dispatcher.scala:126) ~[co.cask.cdap.spark-assembly-1.6.1.jar:na] at org.apache.spark.rpc.netty.NettyRpcEnv.ask(NettyRpcEnv.scala:227) ~[co.cask.cdap.spark-assembly-1.6.1.jar:na] at org.apache.spark.rpc.netty.NettyRpcEndpointRef.ask(NettyRpcEnv.scala:511) ~[co.cask.cdap.spark-assembly-1.6.1.jar:na] at org.apache.spark.rpc.RpcEndpointRef.askWithRetry(RpcEndpointRef.scala:100) ~[co.cask.cdap.spark-assembly-1.6.1.jar:na] ... 13 common frames omitted {code}

    Cask Community Issue Tracker | 7 months ago | Rohit Sinha
    org.apache.spark.SparkException: Could not find HeartbeatReceiver or it has been stopped.
  3. 0

    Apache Spark User List - GC overhead limit exceeded

    nabble.com | 3 months ago
    org.apache.spark.SparkException: Could not find HeartbeatReceiver or it has been stopped.
  4. Speed up your debug routine!

    Automated exception search integrated into your IDE

  5. 0

    Talend Open Integration Solution | 6 months ago | lei ju
    org.apache.spark.SparkException: Error sending message [message = StopAllReceivers]
  6. 0

    Sparkling water executor error

    Google Groups | 4 months ago | hart jo
    org.apache.spark.SparkException: Error sending message [message = RemoveExecutor(0)]

  1. Nikolay Rybak 1 times, last 1 month ago
  2. tyson925 1 times, last 7 months ago
1 unregistered visitors
Not finding the right solution?
Take a tour to get the most out of Samebug.

Tired of useless tips?

Automated exception search integrated into your IDE

Root Cause Analysis

  1. org.apache.spark.SparkException

    Could not find CoarseGrainedScheduler or it has been stopped.

    at org.apache.spark.rpc.netty.Dispatcher.postMessage()
  2. org.apache.spark
    RpcEndpointRef.askWithRetry
    1. org.apache.spark.rpc.netty.Dispatcher.postMessage(Dispatcher.scala:162)
    2. org.apache.spark.rpc.netty.Dispatcher.postLocalMessage(Dispatcher.scala:127)
    3. org.apache.spark.rpc.netty.NettyRpcEnv.ask(NettyRpcEnv.scala:225)
    4. org.apache.spark.rpc.netty.NettyRpcEndpointRef.ask(NettyRpcEnv.scala:508)
    5. org.apache.spark.rpc.RpcEndpointRef.askWithRetry(RpcEndpointRef.scala:101)
    6. org.apache.spark.rpc.RpcEndpointRef.askWithRetry(RpcEndpointRef.scala:78)
    6 frames
  3. Spark
    CoarseMesosSchedulerBackend.statusUpdate
    1. org.apache.spark.scheduler.cluster.CoarseGrainedSchedulerBackend.removeExecutor(CoarseGrainedSchedulerBackend.scala:414)
    2. org.apache.spark.scheduler.cluster.mesos.CoarseMesosSchedulerBackend.executorTerminated(CoarseMesosSchedulerBackend.scala:553)
    3. org.apache.spark.scheduler.cluster.mesos.CoarseMesosSchedulerBackend.statusUpdate(CoarseMesosSchedulerBackend.scala:494)
    3 frames