org.apache.spark.SparkException

There are no available Samebug tips for this exception. Do you have an idea how to solve this issue? A short tip would help users who saw this issue last week.

  • Spark using map in cluster mode
    via Stack Overflow by Saygın Doğu
    ,
  • We have a clone method in {{ColumnType}} (https://github.com/apache/spark/blob/v1.5.0-rc3/sql/core/src/main/scala/org/apache/spark/sql/columnar/ColumnType.scala#L103). Seems we need to override it for String (https://github.com/apache/spark/blob/v1.5.0-rc3/sql/core/src/main/scala/org/apache/spark/sql/columnar/ColumnType.scala#L314) because we are dealing with UTF8String. {code} val df = ctx.range(1, 30000).selectExpr("id % 500 as id").rdd.map(id => Tuple1(s"str_$id")).toDF("i") val cached = df.cache() cached.count() [info] - SPARK-10422: String column in InMemoryColumnarCache needs to override clone method *** FAILED *** (9 seconds, 152 milliseconds) [info] org.apache.spark.SparkException: Job aborted due to stage failure: Task 1 in stage 0.0 failed 1 times, most recent failure: Lost task 1.0 in stage 0.0 (TID 1, localhost): java.util.NoSuchElementException: key not found: str_[0] [info] at scala.collection.MapLike$class.default(MapLike.scala:228) [info] at scala.collection.AbstractMap.default(Map.scala:58) [info] at scala.collection.mutable.HashMap.apply(HashMap.scala:64) [info] at org.apache.spark.sql.columnar.compression.DictionaryEncoding$Encoder.compress(compressionSchemes.scala:258) [info] at org.apache.spark.sql.columnar.compression.CompressibleColumnBuilder$class.build(CompressibleColumnBuilder.scala:110) [info] at org.apache.spark.sql.columnar.NativeColumnBuilder.build(ColumnBuilder.scala:87) [info] at org.apache.spark.sql.columnar.InMemoryRelation$$anonfun$3$$anon$1$$anonfun$next$2.apply(InMemoryColumnarTableScan.scala:152) [info] at org.apache.spark.sql.columnar.InMemoryRelation$$anonfun$3$$anon$1$$anonfun$next$2.apply(InMemoryColumnarTableScan.scala:152) [info] at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:244) [info] at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:244) [info] at scala.collection.IndexedSeqOptimized$class.foreach(IndexedSeqOptimized.scala:33) [info] at scala.collection.mutable.ArrayOps$ofRef.foreach(ArrayOps.scala:108) [info] at scala.collection.TraversableLike$class.map(TraversableLike.scala:244) [info] at scala.collection.mutable.ArrayOps$ofRef.map(ArrayOps.scala:108) [info] at org.apache.spark.sql.columnar.InMemoryRelation$$anonfun$3$$anon$1.next(InMemoryColumnarTableScan.scala:152) [info] at org.apache.spark.sql.columnar.InMemoryRelation$$anonfun$3$$anon$1.next(InMemoryColumnarTableScan.scala:120) {code}
    via by Yin Huai,
    • org.apache.spark.SparkException: Job aborted due to stage failure: Task 0 in stage 1.0 failed 4 times, most recent failure: Lost task 0.3 in stage 1.0 (TID 4, 192.168.1.5): java.util.NoSuchElementException: key not found: node at scala.collection.MapLike$class.default(MapLike.scala:228) at scala.collection.AbstractMap.default(Map.scala:58) at scala.collection.MapLike$class.apply(MapLike.scala:141) at scala.collection.AbstractMap.apply(Map.scala:58) at org.elasticsearch.spark.sql.RowValueReader$class.addToBuffer(RowValueReader.scala:32) at org.elasticsearch.spark.sql.ScalaRowValueReader.addToBuffer(ScalaRowValueReader.scala:9) at org.elasticsearch.spark.sql.ScalaRowValueReader.addToMap(ScalaRowValueReader.scala:16) at org.elasticsearch.hadoop.serialization.ScrollReader.map(ScrollReader.java:596) at org.elasticsearch.hadoop.serialization.ScrollReader.read(ScrollReader.java:519) at org.elasticsearch.hadoop.serialization.ScrollReader.list(ScrollReader.java:560) at org.elasticsearch.hadoop.serialization.ScrollReader.read(ScrollReader.java:522) at org.elasticsearch.hadoop.serialization.ScrollReader.map(ScrollReader.java:596) at org.elasticsearch.hadoop.serialization.ScrollReader.read(ScrollReader.java:519) at org.elasticsearch.hadoop.serialization.ScrollReader.readHitAsMap(ScrollReader.java:339) at org.elasticsearch.hadoop.serialization.ScrollReader.readHit(ScrollReader.java:290) at org.elasticsearch.hadoop.serialization.ScrollReader.read(ScrollReader.java:186) at org.elasticsearch.hadoop.serialization.ScrollReader.read(ScrollReader.java:165) at org.elasticsearch.hadoop.rest.RestRepository.scroll(RestRepository.java:403) at org.elasticsearch.hadoop.rest.ScrollQuery.hasNext(ScrollQuery.java:76) at org.elasticsearch.spark.rdd.AbstractEsRDDIterator.hasNext(AbstractEsRDDIterator.scala:46) at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:327) at scala.collection.Iterator$class.foreach(Iterator.scala:727) at scala.collection.AbstractIterator.foreach(Iterator.scala:1157) at scala.collection.generic.Growable$class.$plus$plus$eq(Growable.scala:48) at scala.collection.mutable.ArrayBuffer.$plus$plus$eq(ArrayBuffer.scala:103) at scala.collection.mutable.ArrayBuffer.$plus$plus$eq(ArrayBuffer.scala:47) at scala.collection.TraversableOnce$class.to(TraversableOnce.scala:273) at scala.collection.AbstractIterator.to(Iterator.scala:1157) at scala.collection.TraversableOnce$class.toBuffer(TraversableOnce.scala:265) at scala.collection.AbstractIterator.toBuffer(Iterator.scala:1157) at scala.collection.TraversableOnce$class.toArray(TraversableOnce.scala:252) at scala.collection.AbstractIterator.toArray(Iterator.scala:1157) at org.apache.spark.rdd.RDD$$anonfun$17.apply(RDD.scala:797) at org.apache.spark.rdd.RDD$$anonfun$17.apply(RDD.scala:797) at org.apache.spark.SparkContext$$anonfun$runJob$4.apply(SparkContext.scala:1353) at org.apache.spark.SparkContext$$anonfun$runJob$4.apply(SparkContext.scala:1353) at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:61) at org.apache.spark.scheduler.Task.run(Task.scala:56) at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:200) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745)
    No Bugmate found.