java.lang.RuntimeException: Can't acquire 1049600 bytes memory to build hash relation, got 74332 bytes

Stack Overflow | Idris Hanafi | 7 months ago
tip
Click on the to mark the solution that helps you, Samebug will learn from it.
As a community member, you’ll be rewarded for you help.
  1. 0

    Increase Java Memory on Spark for Building Large Hash Relations

    Stack Overflow | 7 months ago | Idris Hanafi
    java.lang.RuntimeException: Can't acquire 1049600 bytes memory to build hash relation, got 74332 bytes

    Root Cause Analysis

    1. org.apache.spark.SparkException

      Can't acquire 1049600 bytes memory to build hash relation, got 74332 bytes

      at org.apache.spark.sql.execution.joins.LongToUnsafeRowMap.ensureAcquireMemory()
    2. Spark Project SQL
      LongToUnsafeRowMap.init
      1. org.apache.spark.sql.execution.joins.LongToUnsafeRowMap.ensureAcquireMemory(HashedRelation.scala:414)
      2. org.apache.spark.sql.execution.joins.LongToUnsafeRowMap.init(HashedRelation.scala:424)
      2 frames