java.lang.OutOfMemoryError: Direct buffer memory

Google Groups | Jaeki Hong | 2 years ago
tip
Do you know that we can give you better hits? Get more relevant results from Samebug’s stack trace search.
  1. 0

    Executor lost failure when running avocado-submit ?

    Google Groups | 2 years ago | Jaeki Hong
    java.lang.OutOfMemoryError: Direct buffer memory

    Root Cause Analysis

    1. java.lang.OutOfMemoryError

      Direct buffer memory

      at org.apache.spark.shuffle.hash.BlockStoreShuffleFetcher$.org$apache$spark$shuffle$hash$BlockStoreShuffleFetcher$$unpackBlock$1()
    2. Spark
      BlockStoreShuffleFetcher$$anonfun$3.apply
      1. org.apache.spark.shuffle.hash.BlockStoreShuffleFetcher$.org$apache$spark$shuffle$hash$BlockStoreShuffleFetcher$$unpackBlock$1(BlockStoreShuffleFetcher.scala:67)
      2. org.apache.spark.shuffle.hash.BlockStoreShuffleFetcher$$anonfun$3.apply(BlockStoreShuffleFetcher.scala:83)
      3. org.apache.spark.shuffle.hash.BlockStoreShuffleFetcher$$anonfun$3.apply(BlockStoreShuffleFetcher.scala:83)
      3 frames
    3. Scala
      Iterator$$anon$13.hasNext
      1. scala.collection.Iterator$$anon$13.hasNext(Iterator.scala:371)
      1 frame
    4. Spark
      InterruptibleIterator.hasNext
      1. org.apache.spark.util.CompletionIterator.hasNext(CompletionIterator.scala:32)
      2. org.apache.spark.InterruptibleIterator.hasNext(InterruptibleIterator.scala:39)
      2 frames
    5. Scala
      Iterator$$anon$11.hasNext
      1. scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:327)
      1 frame
    6. Spark
      ResultTask.runTask
      1. org.apache.spark.util.collection.ExternalSorter.insertAll(ExternalSorter.scala:217)
      2. org.apache.spark.shuffle.hash.HashShuffleReader.read(HashShuffleReader.scala:61)
      3. org.apache.spark.rdd.ShuffledRDD.compute(ShuffledRDD.scala:92)
      4. org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:280)
      5. org.apache.spark.rdd.RDD.iterator(RDD.scala:247)
      6. org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:35)
      7. org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:280)
      8. org.apache.spark.rdd.RDD.iterator(RDD.scala:247)
      9. org.apache.spark.rdd.FlatMappedRDD.compute(FlatMappedRDD.scala:33)
      10. org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:280)
      11. org.apache.spark.rdd.RDD.iterator(RDD.scala:247)
      12. org.apache.spark.rdd.FlatMappedRDD.compute(FlatMappedRDD.scala:33)
      13. org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:280)
      14. org.apache.spark.rdd.RDD.iterator(RDD.scala:247)
      15. org.apache.spark.rdd.MappedRDD.compute(MappedRDD.scala:31)
      16. org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:280)
      17. org.apache.spark.rdd.RDD.iterator(RDD.scala:247)
      18. org.apache.spark.rdd.MappedRDD.compute(MappedRDD.scala:31)
      19. org.apache.spark.rdd.InstrumentedRDD.compute(InstrumentedRDD.scala:40)
      20. org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:280)
      21. org.apache.spark.rdd.RDD.iterator(RDD.scala:247)
      22. org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:61)
      22 frames