java.lang.OutOfMemoryError: GC overhead limit exceeded

Stack Overflow | user2459075 | 3 months ago
  1. 0

    I'm having a "GC overhead limit exceeded" on Spark 1.5.2 (reproductible every ~20 hours)

    Stack Overflow | 3 months ago | user2459075
    java.lang.OutOfMemoryError: GC overhead limit exceeded
  2. 0

    OOM error

    GitHub | 7 months ago | jpdna
    java.lang.OutOfMemoryError: Java heap space
  3. 0

    Re: Spark Application Master on Yarn client mode - Virtual memory limit

    spark-user | 10 months ago | Olivier Girardot
    java.lang.OutOfMemoryError: Java heap space
  4. Speed up your debug routine!

    Automated exception search integrated into your IDE

  5. 0

    Re: Spark Application Master on Yarn client mode - Virtual memory limit

    spark-user | 10 months ago | Olivier Girardot
    java.lang.OutOfMemoryError: Java heap space
  6. 0

    Re: Spark Application Master on Yarn client mode - Virtual memory limit

    spark-user | 10 months ago | Olivier Girardot
    java.lang.OutOfMemoryError: Java heap space

    1 unregistered visitors
    Not finding the right solution?
    Take a tour to get the most out of Samebug.

    Tired of useless tips?

    Automated exception search integrated into your IDE

    Root Cause Analysis

    1. java.lang.OutOfMemoryError

      GC overhead limit exceeded

      at java.util.IdentityHashMap.resize()
    2. Java RT
      IdentityHashMap.put
      1. java.util.IdentityHashMap.resize(IdentityHashMap.java:471)
      2. java.util.IdentityHashMap.put(IdentityHashMap.java:440)
      2 frames
    3. Spark
      SizeEstimator$$anonfun$visitSingleObject$1.apply
      1. org.apache.spark.util.SizeEstimator$SearchState.enqueue(SizeEstimator.scala:159)
      2. org.apache.spark.util.SizeEstimator$$anonfun$visitSingleObject$1.apply(SizeEstimator.scala:203)
      3. org.apache.spark.util.SizeEstimator$$anonfun$visitSingleObject$1.apply(SizeEstimator.scala:202)
      3 frames
    4. Scala
      List.foreach
      1. scala.collection.immutable.List.foreach(List.scala:318)
      1 frame
    5. Spark
      RDD.iterator
      1. org.apache.spark.util.SizeEstimator$.visitSingleObject(SizeEstimator.scala:202)
      2. org.apache.spark.util.SizeEstimator$.org$apache$spark$util$SizeEstimator$$estimate(SizeEstimator.scala:186)
      3. org.apache.spark.util.SizeEstimator$.estimate(SizeEstimator.scala:54)
      4. org.apache.spark.util.collection.SizeTracker$class.takeSample(SizeTracker.scala:78)
      5. org.apache.spark.util.collection.SizeTracker$class.afterUpdate(SizeTracker.scala:70)
      6. org.apache.spark.util.collection.SizeTrackingVector.$plus$eq(SizeTrackingVector.scala:31)
      7. org.apache.spark.storage.MemoryStore.unrollSafely(MemoryStore.scala:278)
      8. org.apache.spark.CacheManager.putInBlockManager(CacheManager.scala:171)
      9. org.apache.spark.CacheManager.getOrCompute(CacheManager.scala:78)
      10. org.apache.spark.rdd.RDD.iterator(RDD.scala:262)
      11. org.apache.spark.rdd.UnionRDD.compute(UnionRDD.scala:87)
      12. org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:300)
      13. org.apache.spark.rdd.RDD.iterator(RDD.scala:264)
      14. org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)
      15. org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:300)
      16. org.apache.spark.CacheManager.getOrCompute(CacheManager.scala:69)
      17. org.apache.spark.rdd.RDD.iterator(RDD.scala:262)
      18. org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)
      19. org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:300)
      20. org.apache.spark.rdd.RDD.iterator(RDD.scala:264)
      21. org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)
      22. org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:300)
      23. org.apache.spark.rdd.RDD.iterator(RDD.scala:264)
      24. org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)
      25. org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:300)
      26. org.apache.spark.rdd.RDD.iterator(RDD.scala:264)
      26 frames