org.apache.spark.SparkException: Job aborted due to stage failure: Task 0 in stage 0.0 failed 4 times, most recent failure: Lost task 0.3 in stage 0.0 (TID 3, localhost): ExecutorLostFailure (executor 1 lost) Driver stacktrace:

DataStax JIRA | Foo Lim | 2 years ago
  1. 0

    I just tried Spark 1.3.1 with Spark C* connector 1.2.0, which I was told should work. Contacted someone at datastax & they said it was a bug in Scala code & I should file a Jira here. *build.sbt* {noformat} name := """spark_test_cass""" version := "1.0" scalaVersion := "2.10.5" libraryDependencies += "org.apache.spark" %% "spark-core" % "1.3.1" % "provided" libraryDependencies += "com.datastax.spark" %% "spark-cassandra-connector" % "1.2.0" {noformat} *code*: {code:java} import org.apache.spark.SparkContext import org.apache.spark.SparkContext._ import org.apache.spark.SparkConf import com.datastax.spark.connector._ object TestCass extends App { val conf = new SparkConf(true).set("spark.cassandra.connection.host", "127.0.0.1") val sc = new SparkContext("spark://127.0.0.1:7077", "test", conf) val rdd = sc.cassandraTable("test_ks", "fingerprint") println(rdd.count) println(rdd.first) } {code} *build (using addSbtPlugin("com.eed3si9n" % "sbt-assembly" % "0.13.0")):* {noformat} sbt assembly {noformat} *submission:* {noformat} spark-1.3.1-bin-hadoop2.4$ bin/spark-submit --class TestCass --master spark://127.0.0.1:7077 /somepath/target/scala-2.10/spark_test_cass-assembly-1.0.jar {noformat} *errors* {noformat} Using Spark's default log4j profile: org/apache/spark/log4j-defaults.properties 15/05/11 12:11:04 INFO SparkContext: Running Spark version 1.3.1 15/05/11 12:11:04 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable 15/05/11 12:11:04 INFO SecurityManager: Changing view acls to: foolim 15/05/11 12:11:04 INFO SecurityManager: Changing modify acls to: foolim 15/05/11 12:11:04 INFO SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(foolim); users with modify permissions: Set(foolim) 15/05/11 12:11:05 INFO Slf4jLogger: Slf4jLogger started 15/05/11 12:11:05 INFO Remoting: Starting remoting 15/05/11 12:11:05 INFO Remoting: Remoting started; listening on addresses :[akka.tcp://sparkDriver@localhost:58307] 15/05/11 12:11:05 INFO Utils: Successfully started service 'sparkDriver' on port 58307. 15/05/11 12:11:05 INFO SparkEnv: Registering MapOutputTracker 15/05/11 12:11:05 INFO SparkEnv: Registering BlockManagerMaster 15/05/11 12:11:05 INFO DiskBlockManager: Created local directory at /var/folders/r7/8m3s7jbd3k77323lgm1ms78r0000gp/T/spark-654f9d35-2b62-4104-9653-75683649ef5c/blockmgr-4e7304b3-961f-4c21-bcfb-5c64c1489ad9 15/05/11 12:11:05 INFO MemoryStore: MemoryStore started with capacity 1060.0 MB 15/05/11 12:11:05 INFO HttpFileServer: HTTP File server directory is /var/folders/r7/8m3s7jbd3k77323lgm1ms78r0000gp/T/spark-2491c9e2-0554-485b-8913-9cdc6c3417c7/httpd-8f503a25-aed7-4a2f-b9f6-571efd042b77 15/05/11 12:11:05 INFO HttpServer: Starting HTTP Server 15/05/11 12:11:05 INFO Server: jetty-8.y.z-SNAPSHOT 15/05/11 12:11:05 INFO AbstractConnector: Started SocketConnector@0.0.0.0:58308 15/05/11 12:11:05 INFO Utils: Successfully started service 'HTTP file server' on port 58308. 15/05/11 12:11:05 INFO SparkEnv: Registering OutputCommitCoordinator 15/05/11 12:11:05 INFO Server: jetty-8.y.z-SNAPSHOT 15/05/11 12:11:05 INFO AbstractConnector: Started SelectChannelConnector@0.0.0.0:4040 15/05/11 12:11:05 INFO Utils: Successfully started service 'SparkUI' on port 4040. 15/05/11 12:11:05 INFO SparkUI: Started SparkUI at http://localhost:4040 15/05/11 12:11:05 INFO SparkContext: Added JAR file:/somepath/target/scala-2.10/spark_test_cass-assembly-1.0.jar at http://127.0.0.1:58308/jars/spark_test_cass-assembly-1.0.jar with timestamp 1431371465617 15/05/11 12:11:05 INFO AppClient$ClientActor: Connecting to master akka.tcp://sparkMaster@127.0.0.1:7077/user/Master... 15/05/11 12:11:05 INFO SparkDeploySchedulerBackend: Connected to Spark cluster with app ID app-20150511121105-0000 15/05/11 12:11:05 INFO AppClient$ClientActor: Executor added: app-20150511121105-0000/0 on worker-20150511120335-localhost-58215 (localhost:58215) with 8 cores 15/05/11 12:11:05 INFO SparkDeploySchedulerBackend: Granted executor ID app-20150511121105-0000/0 on hostPort localhost:58215 with 8 cores, 512.0 MB RAM 15/05/11 12:11:05 INFO AppClient$ClientActor: Executor updated: app-20150511121105-0000/0 is now RUNNING 15/05/11 12:11:05 INFO AppClient$ClientActor: Executor updated: app-20150511121105-0000/0 is now LOADING 15/05/11 12:11:05 INFO NettyBlockTransferService: Server created on 58310 15/05/11 12:11:05 INFO BlockManagerMaster: Trying to register BlockManager 15/05/11 12:11:05 INFO BlockManagerMasterActor: Registering block manager localhost:58310 with 1060.0 MB RAM, BlockManagerId(<driver>, localhost, 58310) 15/05/11 12:11:05 INFO BlockManagerMaster: Registered BlockManager 15/05/11 12:11:06 INFO SparkDeploySchedulerBackend: SchedulerBackend is ready for scheduling beginning after reached minRegisteredResourcesRatio: 0.0 15/05/11 12:11:06 INFO Cluster: New Cassandra host /127.0.0.1:9042 added 15/05/11 12:11:06 INFO CassandraConnector: Connected to Cassandra cluster: Test Cluster 15/05/11 12:11:06 INFO CassandraConnector: Disconnected from Cassandra cluster: Test Cluster 15/05/11 12:11:07 INFO SparkContext: Starting job: reduce at CassandraTableScanRDD.scala:245 15/05/11 12:11:07 INFO DAGScheduler: Got job 0 (reduce at CassandraTableScanRDD.scala:245) with 1 output partitions (allowLocal=false) 15/05/11 12:11:07 INFO DAGScheduler: Final stage: Stage 0(reduce at CassandraTableScanRDD.scala:245) 15/05/11 12:11:07 INFO DAGScheduler: Parents of final stage: List() 15/05/11 12:11:07 INFO DAGScheduler: Missing parents: List() 15/05/11 12:11:07 INFO DAGScheduler: Submitting Stage 0 (CassandraTableScanRDD[1] at RDD at CassandraRDD.scala:15), which has no missing parents 15/05/11 12:11:07 INFO MemoryStore: ensureFreeSpace(5400) called with curMem=0, maxMem=1111511531 15/05/11 12:11:07 INFO MemoryStore: Block broadcast_0 stored as values in memory (estimated size 5.3 KB, free 1060.0 MB) 15/05/11 12:11:07 INFO MemoryStore: ensureFreeSpace(3646) called with curMem=5400, maxMem=1111511531 15/05/11 12:11:07 INFO MemoryStore: Block broadcast_0_piece0 stored as bytes in memory (estimated size 3.6 KB, free 1060.0 MB) 15/05/11 12:11:07 INFO BlockManagerInfo: Added broadcast_0_piece0 in memory on localhost:58310 (size: 3.6 KB, free: 1060.0 MB) 15/05/11 12:11:07 INFO BlockManagerMaster: Updated info of block broadcast_0_piece0 15/05/11 12:11:07 INFO SparkContext: Created broadcast 0 from broadcast at DAGScheduler.scala:839 15/05/11 12:11:07 INFO DAGScheduler: Submitting 1 missing tasks from Stage 0 (CassandraTableScanRDD[1] at RDD at CassandraRDD.scala:15) 15/05/11 12:11:07 INFO TaskSchedulerImpl: Adding task set 0.0 with 1 tasks 15/05/11 12:11:08 INFO SparkDeploySchedulerBackend: Registered executor: Actor[akka.tcp://sparkExecutor@localhost:58332/user/Executor#-355877641] with ID 0 15/05/11 12:11:08 INFO TaskSetManager: Starting task 0.0 in stage 0.0 (TID 0, localhost, NODE_LOCAL, 28986 bytes) 15/05/11 12:11:08 INFO BlockManagerMasterActor: Registering block manager localhost:58335 with 265.1 MB RAM, BlockManagerId(0, localhost, 58335) 15/05/11 12:11:08 INFO BlockManagerInfo: Added broadcast_0_piece0 in memory on localhost:58335 (size: 3.6 KB, free: 265.1 MB) 15/05/11 12:11:09 WARN TaskSetManager: Lost task 0.0 in stage 0.0 (TID 0, localhost): java.lang.NoSuchMethodError: org.apache.spark.executor.TaskMetrics.inputMetrics_$eq(Lscala/Option;)V at com.datastax.spark.connector.metrics.InputMetricsUpdater$.apply(InputMetricsUpdater.scala:61) at com.datastax.spark.connector.rdd.CassandraTableScanRDD.compute(CassandraTableScanRDD.scala:196) at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:277) at org.apache.spark.rdd.RDD.iterator(RDD.scala:244) at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:61) at org.apache.spark.scheduler.Task.run(Task.scala:64) at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:203) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) 15/05/11 12:11:09 INFO TaskSetManager: Starting task 0.1 in stage 0.0 (TID 1, localhost, NODE_LOCAL, 28986 bytes) 15/05/11 12:11:09 ERROR TaskSchedulerImpl: Lost executor 0 on localhost: remote Akka client disassociated 15/05/11 12:11:09 WARN ReliableDeliverySupervisor: Association with remote system [akka.tcp://sparkExecutor@localhost:58332] has failed, address is now gated for [5000] ms. Reason is: [Disassociated]. 15/05/11 12:11:09 INFO TaskSetManager: Re-queueing tasks for 0 from TaskSet 0.0 15/05/11 12:11:09 WARN TaskSetManager: Lost task 0.1 in stage 0.0 (TID 1, localhost): ExecutorLostFailure (executor 0 lost) 15/05/11 12:11:09 INFO DAGScheduler: Executor lost: 0 (epoch 0) 15/05/11 12:11:09 INFO BlockManagerMasterActor: Trying to remove executor 0 from BlockManagerMaster. 15/05/11 12:11:09 INFO BlockManagerMasterActor: Removing block manager BlockManagerId(0, localhost, 58335) 15/05/11 12:11:09 INFO BlockManagerMaster: Removed 0 successfully in removeExecutor 15/05/11 12:11:09 INFO AppClient$ClientActor: Executor updated: app-20150511121105-0000/0 is now EXITED (Command exited with code 50) 15/05/11 12:11:09 INFO SparkDeploySchedulerBackend: Executor app-20150511121105-0000/0 removed: Command exited with code 50 15/05/11 12:11:09 ERROR SparkDeploySchedulerBackend: Asked to remove non-existent executor 0 15/05/11 12:11:09 INFO AppClient$ClientActor: Executor added: app-20150511121105-0000/1 on worker-20150511120335-localhost-58215 (localhost:58215) with 8 cores 15/05/11 12:11:09 INFO SparkDeploySchedulerBackend: Granted executor ID app-20150511121105-0000/1 on hostPort localhost:58215 with 8 cores, 512.0 MB RAM 15/05/11 12:11:09 INFO AppClient$ClientActor: Executor updated: app-20150511121105-0000/1 is now RUNNING 15/05/11 12:11:09 INFO AppClient$ClientActor: Executor updated: app-20150511121105-0000/1 is now LOADING 15/05/11 12:11:11 INFO SparkDeploySchedulerBackend: Registered executor: Actor[akka.tcp://sparkExecutor@localhost:58342/user/Executor#-1061285171] with ID 1 15/05/11 12:11:11 INFO TaskSetManager: Starting task 0.2 in stage 0.0 (TID 2, localhost, NODE_LOCAL, 28986 bytes) 15/05/11 12:11:11 INFO BlockManagerMasterActor: Registering block manager localhost:58345 with 265.1 MB RAM, BlockManagerId(1, localhost, 58345) 15/05/11 12:11:11 INFO BlockManagerInfo: Added broadcast_0_piece0 in memory on localhost:58345 (size: 3.6 KB, free: 265.1 MB) 15/05/11 12:11:12 INFO TaskSetManager: Lost task 0.2 in stage 0.0 (TID 2) on executor localhost: java.lang.NoSuchMethodError (org.apache.spark.executor.TaskMetrics.inputMetrics_$eq(Lscala/Option;)V) [duplicate 1] 15/05/11 12:11:12 INFO TaskSetManager: Starting task 0.3 in stage 0.0 (TID 3, localhost, NODE_LOCAL, 28986 bytes) 15/05/11 12:11:12 ERROR TaskSchedulerImpl: Lost executor 1 on localhost: remote Akka client disassociated 15/05/11 12:11:12 INFO TaskSetManager: Re-queueing tasks for 1 from TaskSet 0.0 15/05/11 12:11:12 WARN ReliableDeliverySupervisor: Association with remote system [akka.tcp://sparkExecutor@localhost:58342] has failed, address is now gated for [5000] ms. Reason is: [Disassociated]. 15/05/11 12:11:12 WARN TaskSetManager: Lost task 0.3 in stage 0.0 (TID 3, localhost): ExecutorLostFailure (executor 1 lost) 15/05/11 12:11:12 ERROR TaskSetManager: Task 0 in stage 0.0 failed 4 times; aborting job 15/05/11 12:11:12 INFO TaskSchedulerImpl: Removed TaskSet 0.0, whose tasks have all completed, from pool 15/05/11 12:11:12 INFO TaskSchedulerImpl: Cancelling stage 0 15/05/11 12:11:12 INFO DAGScheduler: Stage 0 (reduce at CassandraTableScanRDD.scala:245) failed in 5.202 s 15/05/11 12:11:12 INFO AppClient$ClientActor: Executor updated: app-20150511121105-0000/1 is now EXITED (Command exited with code 50) 15/05/11 12:11:12 INFO SparkDeploySchedulerBackend: Executor app-20150511121105-0000/1 removed: Command exited with code 50 15/05/11 12:11:12 ERROR SparkDeploySchedulerBackend: Asked to remove non-existent executor 1 15/05/11 12:11:12 INFO AppClient$ClientActor: Executor added: app-20150511121105-0000/2 on worker-20150511120335-localhost-58215 (localhost:58215) with 8 cores 15/05/11 12:11:12 INFO SparkDeploySchedulerBackend: Granted executor ID app-20150511121105-0000/2 on hostPort localhost:58215 with 8 cores, 512.0 MB RAM 15/05/11 12:11:12 INFO DAGScheduler: Job 0 failed: reduce at CassandraTableScanRDD.scala:245, took 5.381312 s Exception in thread "main" org.apache.spark.SparkException: Job aborted due to stage failure: Task 0 in stage 0.0 failed 4 times, most recent failure: Lost task 0.3 in stage 0.0 (TID 3, localhost): ExecutorLostFailure (executor 1 lost) Driver stacktrace: at org.apache.spark.scheduler.DAGScheduler.org$apache$spark$scheduler$DAGScheduler$$failJobAndIndependentStages(DAGScheduler.scala:1204) at org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$1.apply(DAGScheduler.scala:1193) at org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$1.apply(DAGScheduler.scala:1192) at scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:59) at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:47) at org.apache.spark.scheduler.DAGScheduler.abortStage(DAGScheduler.scala:1192) at org.apache.spark.scheduler.DAGScheduler$$anonfun$handleTaskSetFailed$1.apply(DAGScheduler.scala:693) at org.apache.spark.scheduler.DAGScheduler$$anonfun$handleTaskSetFailed$1.apply(DAGScheduler.scala:693) at scala.Option.foreach(Option.scala:236) at org.apache.spark.scheduler.DAGScheduler.handleTaskSetFailed(DAGScheduler.scala:693) at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:1393) at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:1354) at org.apache.spark.util.EventLoop$$anon$1.run(EventLoop.scala:48) 15/05/11 12:11:12 INFO AppClient$ClientActor: Executor updated: app-20150511121105-0000/2 is now RUNNING 15/05/11 12:11:12 INFO AppClient$ClientActor: Executor updated: app-20150511121105-0000/2 is now LOADING 15/05/11 12:11:12 INFO DAGScheduler: Executor lost: 1 (epoch 1) 15/05/11 12:11:12 INFO BlockManagerMasterActor: Trying to remove executor 1 from BlockManagerMaster. 15/05/11 12:11:12 INFO BlockManagerMasterActor: Removing block manager BlockManagerId(1, localhost, 58345) 15/05/11 12:11:12 INFO BlockManagerMaster: Removed 1 successfully in removeExecutor {noformat}

    DataStax JIRA | 2 years ago | Foo Lim
    org.apache.spark.SparkException: Job aborted due to stage failure: Task 0 in stage 0.0 failed 4 times, most recent failure: Lost task 0.3 in stage 0.0 (TID 3, localhost): ExecutorLostFailure (executor 1 lost) Driver stacktrace:
  2. 0

    I just tried Spark 1.3.1 with Spark C* connector 1.2.0, which I was told should work. Contacted someone at datastax & they said it was a bug in Scala code & I should file a Jira here. *build.sbt* {noformat} name := """spark_test_cass""" version := "1.0" scalaVersion := "2.10.5" libraryDependencies += "org.apache.spark" %% "spark-core" % "1.3.1" % "provided" libraryDependencies += "com.datastax.spark" %% "spark-cassandra-connector" % "1.2.0" {noformat} *code*: {code:java} import org.apache.spark.SparkContext import org.apache.spark.SparkContext._ import org.apache.spark.SparkConf import com.datastax.spark.connector._ object TestCass extends App { val conf = new SparkConf(true).set("spark.cassandra.connection.host", "127.0.0.1") val sc = new SparkContext("spark://127.0.0.1:7077", "test", conf) val rdd = sc.cassandraTable("test_ks", "fingerprint") println(rdd.count) println(rdd.first) } {code} *build (using addSbtPlugin("com.eed3si9n" % "sbt-assembly" % "0.13.0")):* {noformat} sbt assembly {noformat} *submission:* {noformat} spark-1.3.1-bin-hadoop2.4$ bin/spark-submit --class TestCass --master spark://127.0.0.1:7077 /somepath/target/scala-2.10/spark_test_cass-assembly-1.0.jar {noformat} *errors* {noformat} Using Spark's default log4j profile: org/apache/spark/log4j-defaults.properties 15/05/11 12:11:04 INFO SparkContext: Running Spark version 1.3.1 15/05/11 12:11:04 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable 15/05/11 12:11:04 INFO SecurityManager: Changing view acls to: foolim 15/05/11 12:11:04 INFO SecurityManager: Changing modify acls to: foolim 15/05/11 12:11:04 INFO SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(foolim); users with modify permissions: Set(foolim) 15/05/11 12:11:05 INFO Slf4jLogger: Slf4jLogger started 15/05/11 12:11:05 INFO Remoting: Starting remoting 15/05/11 12:11:05 INFO Remoting: Remoting started; listening on addresses :[akka.tcp://sparkDriver@localhost:58307] 15/05/11 12:11:05 INFO Utils: Successfully started service 'sparkDriver' on port 58307. 15/05/11 12:11:05 INFO SparkEnv: Registering MapOutputTracker 15/05/11 12:11:05 INFO SparkEnv: Registering BlockManagerMaster 15/05/11 12:11:05 INFO DiskBlockManager: Created local directory at /var/folders/r7/8m3s7jbd3k77323lgm1ms78r0000gp/T/spark-654f9d35-2b62-4104-9653-75683649ef5c/blockmgr-4e7304b3-961f-4c21-bcfb-5c64c1489ad9 15/05/11 12:11:05 INFO MemoryStore: MemoryStore started with capacity 1060.0 MB 15/05/11 12:11:05 INFO HttpFileServer: HTTP File server directory is /var/folders/r7/8m3s7jbd3k77323lgm1ms78r0000gp/T/spark-2491c9e2-0554-485b-8913-9cdc6c3417c7/httpd-8f503a25-aed7-4a2f-b9f6-571efd042b77 15/05/11 12:11:05 INFO HttpServer: Starting HTTP Server 15/05/11 12:11:05 INFO Server: jetty-8.y.z-SNAPSHOT 15/05/11 12:11:05 INFO AbstractConnector: Started SocketConnector@0.0.0.0:58308 15/05/11 12:11:05 INFO Utils: Successfully started service 'HTTP file server' on port 58308. 15/05/11 12:11:05 INFO SparkEnv: Registering OutputCommitCoordinator 15/05/11 12:11:05 INFO Server: jetty-8.y.z-SNAPSHOT 15/05/11 12:11:05 INFO AbstractConnector: Started SelectChannelConnector@0.0.0.0:4040 15/05/11 12:11:05 INFO Utils: Successfully started service 'SparkUI' on port 4040. 15/05/11 12:11:05 INFO SparkUI: Started SparkUI at http://localhost:4040 15/05/11 12:11:05 INFO SparkContext: Added JAR file:/somepath/target/scala-2.10/spark_test_cass-assembly-1.0.jar at http://127.0.0.1:58308/jars/spark_test_cass-assembly-1.0.jar with timestamp 1431371465617 15/05/11 12:11:05 INFO AppClient$ClientActor: Connecting to master akka.tcp://sparkMaster@127.0.0.1:7077/user/Master... 15/05/11 12:11:05 INFO SparkDeploySchedulerBackend: Connected to Spark cluster with app ID app-20150511121105-0000 15/05/11 12:11:05 INFO AppClient$ClientActor: Executor added: app-20150511121105-0000/0 on worker-20150511120335-localhost-58215 (localhost:58215) with 8 cores 15/05/11 12:11:05 INFO SparkDeploySchedulerBackend: Granted executor ID app-20150511121105-0000/0 on hostPort localhost:58215 with 8 cores, 512.0 MB RAM 15/05/11 12:11:05 INFO AppClient$ClientActor: Executor updated: app-20150511121105-0000/0 is now RUNNING 15/05/11 12:11:05 INFO AppClient$ClientActor: Executor updated: app-20150511121105-0000/0 is now LOADING 15/05/11 12:11:05 INFO NettyBlockTransferService: Server created on 58310 15/05/11 12:11:05 INFO BlockManagerMaster: Trying to register BlockManager 15/05/11 12:11:05 INFO BlockManagerMasterActor: Registering block manager localhost:58310 with 1060.0 MB RAM, BlockManagerId(<driver>, localhost, 58310) 15/05/11 12:11:05 INFO BlockManagerMaster: Registered BlockManager 15/05/11 12:11:06 INFO SparkDeploySchedulerBackend: SchedulerBackend is ready for scheduling beginning after reached minRegisteredResourcesRatio: 0.0 15/05/11 12:11:06 INFO Cluster: New Cassandra host /127.0.0.1:9042 added 15/05/11 12:11:06 INFO CassandraConnector: Connected to Cassandra cluster: Test Cluster 15/05/11 12:11:06 INFO CassandraConnector: Disconnected from Cassandra cluster: Test Cluster 15/05/11 12:11:07 INFO SparkContext: Starting job: reduce at CassandraTableScanRDD.scala:245 15/05/11 12:11:07 INFO DAGScheduler: Got job 0 (reduce at CassandraTableScanRDD.scala:245) with 1 output partitions (allowLocal=false) 15/05/11 12:11:07 INFO DAGScheduler: Final stage: Stage 0(reduce at CassandraTableScanRDD.scala:245) 15/05/11 12:11:07 INFO DAGScheduler: Parents of final stage: List() 15/05/11 12:11:07 INFO DAGScheduler: Missing parents: List() 15/05/11 12:11:07 INFO DAGScheduler: Submitting Stage 0 (CassandraTableScanRDD[1] at RDD at CassandraRDD.scala:15), which has no missing parents 15/05/11 12:11:07 INFO MemoryStore: ensureFreeSpace(5400) called with curMem=0, maxMem=1111511531 15/05/11 12:11:07 INFO MemoryStore: Block broadcast_0 stored as values in memory (estimated size 5.3 KB, free 1060.0 MB) 15/05/11 12:11:07 INFO MemoryStore: ensureFreeSpace(3646) called with curMem=5400, maxMem=1111511531 15/05/11 12:11:07 INFO MemoryStore: Block broadcast_0_piece0 stored as bytes in memory (estimated size 3.6 KB, free 1060.0 MB) 15/05/11 12:11:07 INFO BlockManagerInfo: Added broadcast_0_piece0 in memory on localhost:58310 (size: 3.6 KB, free: 1060.0 MB) 15/05/11 12:11:07 INFO BlockManagerMaster: Updated info of block broadcast_0_piece0 15/05/11 12:11:07 INFO SparkContext: Created broadcast 0 from broadcast at DAGScheduler.scala:839 15/05/11 12:11:07 INFO DAGScheduler: Submitting 1 missing tasks from Stage 0 (CassandraTableScanRDD[1] at RDD at CassandraRDD.scala:15) 15/05/11 12:11:07 INFO TaskSchedulerImpl: Adding task set 0.0 with 1 tasks 15/05/11 12:11:08 INFO SparkDeploySchedulerBackend: Registered executor: Actor[akka.tcp://sparkExecutor@localhost:58332/user/Executor#-355877641] with ID 0 15/05/11 12:11:08 INFO TaskSetManager: Starting task 0.0 in stage 0.0 (TID 0, localhost, NODE_LOCAL, 28986 bytes) 15/05/11 12:11:08 INFO BlockManagerMasterActor: Registering block manager localhost:58335 with 265.1 MB RAM, BlockManagerId(0, localhost, 58335) 15/05/11 12:11:08 INFO BlockManagerInfo: Added broadcast_0_piece0 in memory on localhost:58335 (size: 3.6 KB, free: 265.1 MB) 15/05/11 12:11:09 WARN TaskSetManager: Lost task 0.0 in stage 0.0 (TID 0, localhost): java.lang.NoSuchMethodError: org.apache.spark.executor.TaskMetrics.inputMetrics_$eq(Lscala/Option;)V at com.datastax.spark.connector.metrics.InputMetricsUpdater$.apply(InputMetricsUpdater.scala:61) at com.datastax.spark.connector.rdd.CassandraTableScanRDD.compute(CassandraTableScanRDD.scala:196) at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:277) at org.apache.spark.rdd.RDD.iterator(RDD.scala:244) at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:61) at org.apache.spark.scheduler.Task.run(Task.scala:64) at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:203) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) 15/05/11 12:11:09 INFO TaskSetManager: Starting task 0.1 in stage 0.0 (TID 1, localhost, NODE_LOCAL, 28986 bytes) 15/05/11 12:11:09 ERROR TaskSchedulerImpl: Lost executor 0 on localhost: remote Akka client disassociated 15/05/11 12:11:09 WARN ReliableDeliverySupervisor: Association with remote system [akka.tcp://sparkExecutor@localhost:58332] has failed, address is now gated for [5000] ms. Reason is: [Disassociated]. 15/05/11 12:11:09 INFO TaskSetManager: Re-queueing tasks for 0 from TaskSet 0.0 15/05/11 12:11:09 WARN TaskSetManager: Lost task 0.1 in stage 0.0 (TID 1, localhost): ExecutorLostFailure (executor 0 lost) 15/05/11 12:11:09 INFO DAGScheduler: Executor lost: 0 (epoch 0) 15/05/11 12:11:09 INFO BlockManagerMasterActor: Trying to remove executor 0 from BlockManagerMaster. 15/05/11 12:11:09 INFO BlockManagerMasterActor: Removing block manager BlockManagerId(0, localhost, 58335) 15/05/11 12:11:09 INFO BlockManagerMaster: Removed 0 successfully in removeExecutor 15/05/11 12:11:09 INFO AppClient$ClientActor: Executor updated: app-20150511121105-0000/0 is now EXITED (Command exited with code 50) 15/05/11 12:11:09 INFO SparkDeploySchedulerBackend: Executor app-20150511121105-0000/0 removed: Command exited with code 50 15/05/11 12:11:09 ERROR SparkDeploySchedulerBackend: Asked to remove non-existent executor 0 15/05/11 12:11:09 INFO AppClient$ClientActor: Executor added: app-20150511121105-0000/1 on worker-20150511120335-localhost-58215 (localhost:58215) with 8 cores 15/05/11 12:11:09 INFO SparkDeploySchedulerBackend: Granted executor ID app-20150511121105-0000/1 on hostPort localhost:58215 with 8 cores, 512.0 MB RAM 15/05/11 12:11:09 INFO AppClient$ClientActor: Executor updated: app-20150511121105-0000/1 is now RUNNING 15/05/11 12:11:09 INFO AppClient$ClientActor: Executor updated: app-20150511121105-0000/1 is now LOADING 15/05/11 12:11:11 INFO SparkDeploySchedulerBackend: Registered executor: Actor[akka.tcp://sparkExecutor@localhost:58342/user/Executor#-1061285171] with ID 1 15/05/11 12:11:11 INFO TaskSetManager: Starting task 0.2 in stage 0.0 (TID 2, localhost, NODE_LOCAL, 28986 bytes) 15/05/11 12:11:11 INFO BlockManagerMasterActor: Registering block manager localhost:58345 with 265.1 MB RAM, BlockManagerId(1, localhost, 58345) 15/05/11 12:11:11 INFO BlockManagerInfo: Added broadcast_0_piece0 in memory on localhost:58345 (size: 3.6 KB, free: 265.1 MB) 15/05/11 12:11:12 INFO TaskSetManager: Lost task 0.2 in stage 0.0 (TID 2) on executor localhost: java.lang.NoSuchMethodError (org.apache.spark.executor.TaskMetrics.inputMetrics_$eq(Lscala/Option;)V) [duplicate 1] 15/05/11 12:11:12 INFO TaskSetManager: Starting task 0.3 in stage 0.0 (TID 3, localhost, NODE_LOCAL, 28986 bytes) 15/05/11 12:11:12 ERROR TaskSchedulerImpl: Lost executor 1 on localhost: remote Akka client disassociated 15/05/11 12:11:12 INFO TaskSetManager: Re-queueing tasks for 1 from TaskSet 0.0 15/05/11 12:11:12 WARN ReliableDeliverySupervisor: Association with remote system [akka.tcp://sparkExecutor@localhost:58342] has failed, address is now gated for [5000] ms. Reason is: [Disassociated]. 15/05/11 12:11:12 WARN TaskSetManager: Lost task 0.3 in stage 0.0 (TID 3, localhost): ExecutorLostFailure (executor 1 lost) 15/05/11 12:11:12 ERROR TaskSetManager: Task 0 in stage 0.0 failed 4 times; aborting job 15/05/11 12:11:12 INFO TaskSchedulerImpl: Removed TaskSet 0.0, whose tasks have all completed, from pool 15/05/11 12:11:12 INFO TaskSchedulerImpl: Cancelling stage 0 15/05/11 12:11:12 INFO DAGScheduler: Stage 0 (reduce at CassandraTableScanRDD.scala:245) failed in 5.202 s 15/05/11 12:11:12 INFO AppClient$ClientActor: Executor updated: app-20150511121105-0000/1 is now EXITED (Command exited with code 50) 15/05/11 12:11:12 INFO SparkDeploySchedulerBackend: Executor app-20150511121105-0000/1 removed: Command exited with code 50 15/05/11 12:11:12 ERROR SparkDeploySchedulerBackend: Asked to remove non-existent executor 1 15/05/11 12:11:12 INFO AppClient$ClientActor: Executor added: app-20150511121105-0000/2 on worker-20150511120335-localhost-58215 (localhost:58215) with 8 cores 15/05/11 12:11:12 INFO SparkDeploySchedulerBackend: Granted executor ID app-20150511121105-0000/2 on hostPort localhost:58215 with 8 cores, 512.0 MB RAM 15/05/11 12:11:12 INFO DAGScheduler: Job 0 failed: reduce at CassandraTableScanRDD.scala:245, took 5.381312 s Exception in thread "main" org.apache.spark.SparkException: Job aborted due to stage failure: Task 0 in stage 0.0 failed 4 times, most recent failure: Lost task 0.3 in stage 0.0 (TID 3, localhost): ExecutorLostFailure (executor 1 lost) Driver stacktrace: at org.apache.spark.scheduler.DAGScheduler.org$apache$spark$scheduler$DAGScheduler$$failJobAndIndependentStages(DAGScheduler.scala:1204) at org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$1.apply(DAGScheduler.scala:1193) at org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$1.apply(DAGScheduler.scala:1192) at scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:59) at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:47) at org.apache.spark.scheduler.DAGScheduler.abortStage(DAGScheduler.scala:1192) at org.apache.spark.scheduler.DAGScheduler$$anonfun$handleTaskSetFailed$1.apply(DAGScheduler.scala:693) at org.apache.spark.scheduler.DAGScheduler$$anonfun$handleTaskSetFailed$1.apply(DAGScheduler.scala:693) at scala.Option.foreach(Option.scala:236) at org.apache.spark.scheduler.DAGScheduler.handleTaskSetFailed(DAGScheduler.scala:693) at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:1393) at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:1354) at org.apache.spark.util.EventLoop$$anon$1.run(EventLoop.scala:48) 15/05/11 12:11:12 INFO AppClient$ClientActor: Executor updated: app-20150511121105-0000/2 is now RUNNING 15/05/11 12:11:12 INFO AppClient$ClientActor: Executor updated: app-20150511121105-0000/2 is now LOADING 15/05/11 12:11:12 INFO DAGScheduler: Executor lost: 1 (epoch 1) 15/05/11 12:11:12 INFO BlockManagerMasterActor: Trying to remove executor 1 from BlockManagerMaster. 15/05/11 12:11:12 INFO BlockManagerMasterActor: Removing block manager BlockManagerId(1, localhost, 58345) 15/05/11 12:11:12 INFO BlockManagerMaster: Removed 1 successfully in removeExecutor {noformat}

    DataStax JIRA | 2 years ago | Foo Lim
    org.apache.spark.SparkException: Job aborted due to stage failure: Task 0 in stage 0.0 failed 4 times, most recent failure: Lost task 0.3 in stage 0.0 (TID 3, localhost): ExecutorLostFailure (executor 1 lost) Driver stacktrace:
  3. 0

    Apache Spark: Master removed our application: Failed when using saveAsTextFile on large RDD

    Stack Overflow | 2 years ago
    org.apache.spark.SparkException: Job aborted due to stage failure: Master removed our application: FAILED
  4. Speed up your debug routine!

    Automated exception search integrated into your IDE

  5. 0

    ArrayIndexOutOfBoundsException with Spark, Spark-Avro and Google Analytics Data

    Stack Overflow | 2 years ago | theouteredge
    org.apache.spark.SparkException: Job aborted due to stage failure: Task 12 in stage 2.0 failed 1 times, most recent failure: Lost task 12.0 in stage 2.0 (TID 27, localhost): java.lang.ArrayIndexOutOfBoundsException Driver stacktrace:
  6. 0

    Spark on Mesos

    mail-archive.com | 1 year ago
    org.apache.spark.SparkException: Job aborted due to stage failure: Task 5 in stage 0.0 failed 4 times, most recent failure: Lost task 5.3 in stage 0.0 (TID 23, 10.253.1.117): ExecutorLostFailure (executor 20150424-104711-1375862026-5050-20113-S1 lost) Driver stacktrace:

  1. tyson925 1 times, last 7 months ago
  2. Nikolay Rybak 1 times, last 1 month ago
  3. tyson925 2 times, last 2 months ago
  4. meneal 1 times, last 4 months ago
20 unregistered visitors
Not finding the right solution?
Take a tour to get the most out of Samebug.

Tired of useless tips?

Automated exception search integrated into your IDE

Root Cause Analysis

  1. org.apache.spark.SparkException

    Job aborted due to stage failure: Task 0 in stage 0.0 failed 4 times, most recent failure: Lost task 0.3 in stage 0.0 (TID 3, localhost): ExecutorLostFailure (executor 1 lost) Driver stacktrace:

    at org.apache.spark.scheduler.DAGScheduler.org$apache$spark$scheduler$DAGScheduler$$failJobAndIndependentStages()
  2. Spark
    DAGScheduler$$anonfun$abortStage$1.apply
    1. org.apache.spark.scheduler.DAGScheduler.org$apache$spark$scheduler$DAGScheduler$$failJobAndIndependentStages(DAGScheduler.scala:1204)
    2. org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$1.apply(DAGScheduler.scala:1193)
    3. org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$1.apply(DAGScheduler.scala:1192)
    3 frames
  3. Scala
    ArrayBuffer.foreach
    1. scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:59)
    2. scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:47)
    2 frames
  4. Spark
    DAGScheduler$$anonfun$handleTaskSetFailed$1.apply
    1. org.apache.spark.scheduler.DAGScheduler.abortStage(DAGScheduler.scala:1192)
    2. org.apache.spark.scheduler.DAGScheduler$$anonfun$handleTaskSetFailed$1.apply(DAGScheduler.scala:693)
    3. org.apache.spark.scheduler.DAGScheduler$$anonfun$handleTaskSetFailed$1.apply(DAGScheduler.scala:693)
    3 frames
  5. Scala
    Option.foreach
    1. scala.Option.foreach(Option.scala:236)
    1 frame
  6. Spark
    EventLoop$$anon$1.run
    1. org.apache.spark.scheduler.DAGScheduler.handleTaskSetFailed(DAGScheduler.scala:693)
    2. org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:1393)
    3. org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:1354)
    4. org.apache.spark.util.EventLoop$$anon$1.run(EventLoop.scala:48)
    4 frames