java.lang.OutOfMemoryError: Requested array size exceeds VM limit

Cloudera Open Source | Silvius Rus | 1 year ago
  1. 0

    We've seen a catalogd crash triggered by loading the metadata for a table with about 20K partitions and 77 columns that has incremental stats. It looks like the serialized message is over 2GB which is the Java max array size. Ideally we should catch this exception and fail the query that needs this table's metadata with an appropriate message. {code} I1107 06:47:56.641507 30252 jni-util.cc:177] java.lang.OutOfMemoryError: Requested array size exceeds VM limit at java.util.Arrays.copyOf(Arrays.java:2271) at java.io.ByteArrayOutputStream.grow(ByteArrayOutputStream.java:113) at java.io.ByteArrayOutputStream.ensureCapacity(ByteArrayOutputStream.java:93) at java.io.ByteArrayOutputStream.write(ByteArrayOutputStream.java:140) at org.apache.thrift.transport.TIOStreamTransport.write(TIOStreamTransport.java:145) at org.apache.thrift.protocol.TBinaryProtocol.writeString(TBinaryProtocol.java:187) at com.cloudera.impala.thrift.THdfsPartition$THdfsPartitionStandardScheme.write(THdfsPartition.java:1831) at com.cloudera.impala.thrift.THdfsPartition$THdfsPartitionStandardScheme.write(THdfsPartition.java:1543) at com.cloudera.impala.thrift.THdfsPartition.write(THdfsPartition.java:1389) at com.cloudera.impala.thrift.THdfsTable$THdfsTableStandardScheme.write(THdfsTable.java:1123) at com.cloudera.impala.thrift.THdfsTable$THdfsTableStandardScheme.write(THdfsTable.java:969) at com.cloudera.impala.thrift.THdfsTable.write(THdfsTable.java:848) at com.cloudera.impala.thrift.TTable$TTableStandardScheme.write(TTable.java:1628) at com.cloudera.impala.thrift.TTable$TTableStandardScheme.write(TTable.java:1395) at com.cloudera.impala.thrift.TTable.write(TTable.java:1209) at com.cloudera.impala.thrift.TCatalogObject$TCatalogObjectStandardScheme.write(TCatalogObject.java:1241) at com.cloudera.impala.thrift.TCatalogObject$TCatalogObjectStandardScheme.write(TCatalogObject.java:1098) at com.cloudera.impala.thrift.TCatalogObject.write(TCatalogObject.java:938) at com.cloudera.impala.thrift.TGetAllCatalogObjectsResponse$TGetAllCatalogObjectsResponseStandardScheme.write(TGetAllCatalogObjectsResponse.java:487) at com.cloudera.impala.thrift.TGetAllCatalogObjectsResponse$TGetAllCatalogObjectsResponseStandardScheme.write(TGetAllCatalogObjectsResponse.java:421) at com.cloudera.impala.thrift.TGetAllCatalogObjectsResponse.write(TGetAllCatalogObjectsResponse.java:365) at org.apache.thrift.TSerializer.serialize(TSerializer.java:79) at com.cloudera.impala.service.JniCatalog.getCatalogObjects(JniCatalog.java:110) {code} You can identify this issue by looking at the metastore database. Here is how to see the size of the incremental stats for table id 12345. The table with this value of 624 MB of incremental stats led to the catalogd crash shown above. {code} postgres=# select pg_size_pretty(sum(length("PARTITION_PARAMS"."PARAM_KEY") + length("PARTITION_PARAMS"."PARAM_VALUE"))) from "PARTITIONS", "PARTITION_PARAMS" where "PARTITIONS"."TBL_ID"=12345 and "PARTITIONS"."PART_ID" = "PARTITION_PARAMS"."PART_ID" and "PARTITION_PARAMS"."PARAM_KEY" LIKE 'impala_intermediate%'; pg_size_pretty ---------------- 624 MB (1 row) {code}

    Cloudera Open Source | 1 year ago | Silvius Rus
    java.lang.OutOfMemoryError: Requested array size exceeds VM limit
  2. 0

    ByteArrayOutputStream is really, really slow sometimes in JDK6 : Paper Trail

    the-paper-trail.org | 4 months ago
    java.lang.OutOfMemoryError: Requested array size exceeds VM limit
  3. 0

    Spark kmeans got OutOfMemoryError when using Vector sparse API

    Stack Overflow | 2 months ago | Jack
    java.lang.OutOfMemoryError: Requested array size exceeds VM limit
  4. Speed up your debug routine!

    Automated exception search integrated into your IDE

  5. 0

    Java: why is there a difference in memory usage between for- and while-loops?

    Stack Overflow | 2 years ago | Chthonic Project
    java.lang.OutOfMemoryError: Requested array size exceeds VM limit
  6. 0

    OutOfMemoryError while writing a large map to a file

    avro-user | 10 months ago | David Kincaid
    java.lang.OutOfMemoryError: Requested array size exceeds VM limit

    8 unregistered visitors
    Not finding the right solution?
    Take a tour to get the most out of Samebug.

    Tired of useless tips?

    Automated exception search integrated into your IDE

    Root Cause Analysis

    1. java.lang.OutOfMemoryError

      Requested array size exceeds VM limit

      at java.util.Arrays.copyOf()
    2. Java RT
      ByteArrayOutputStream.write
      1. java.util.Arrays.copyOf(Arrays.java:2271)
      2. java.io.ByteArrayOutputStream.grow(ByteArrayOutputStream.java:113)
      3. java.io.ByteArrayOutputStream.ensureCapacity(ByteArrayOutputStream.java:93)
      4. java.io.ByteArrayOutputStream.write(ByteArrayOutputStream.java:140)
      4 frames
    3. Apache Thrift
      TBinaryProtocol.writeString
      1. org.apache.thrift.transport.TIOStreamTransport.write(TIOStreamTransport.java:145)
      2. org.apache.thrift.protocol.TBinaryProtocol.writeString(TBinaryProtocol.java:187)
      2 frames
    4. com.cloudera.impala
      TGetAllCatalogObjectsResponse.write
      1. com.cloudera.impala.thrift.THdfsPartition$THdfsPartitionStandardScheme.write(THdfsPartition.java:1831)
      2. com.cloudera.impala.thrift.THdfsPartition$THdfsPartitionStandardScheme.write(THdfsPartition.java:1543)
      3. com.cloudera.impala.thrift.THdfsPartition.write(THdfsPartition.java:1389)
      4. com.cloudera.impala.thrift.THdfsTable$THdfsTableStandardScheme.write(THdfsTable.java:1123)
      5. com.cloudera.impala.thrift.THdfsTable$THdfsTableStandardScheme.write(THdfsTable.java:969)
      6. com.cloudera.impala.thrift.THdfsTable.write(THdfsTable.java:848)
      7. com.cloudera.impala.thrift.TTable$TTableStandardScheme.write(TTable.java:1628)
      8. com.cloudera.impala.thrift.TTable$TTableStandardScheme.write(TTable.java:1395)
      9. com.cloudera.impala.thrift.TTable.write(TTable.java:1209)
      10. com.cloudera.impala.thrift.TCatalogObject$TCatalogObjectStandardScheme.write(TCatalogObject.java:1241)
      11. com.cloudera.impala.thrift.TCatalogObject$TCatalogObjectStandardScheme.write(TCatalogObject.java:1098)
      12. com.cloudera.impala.thrift.TCatalogObject.write(TCatalogObject.java:938)
      13. com.cloudera.impala.thrift.TGetAllCatalogObjectsResponse$TGetAllCatalogObjectsResponseStandardScheme.write(TGetAllCatalogObjectsResponse.java:487)
      14. com.cloudera.impala.thrift.TGetAllCatalogObjectsResponse$TGetAllCatalogObjectsResponseStandardScheme.write(TGetAllCatalogObjectsResponse.java:421)
      15. com.cloudera.impala.thrift.TGetAllCatalogObjectsResponse.write(TGetAllCatalogObjectsResponse.java:365)
      15 frames
    5. Apache Thrift
      TSerializer.serialize
      1. org.apache.thrift.TSerializer.serialize(TSerializer.java:79)
      1 frame
    6. com.cloudera.impala
      JniCatalog.getCatalogObjects
      1. com.cloudera.impala.service.JniCatalog.getCatalogObjects(JniCatalog.java:110)
      1 frame