org.apache.hadoop.mapred.YarnChild: Error running child : java.lang.OutOfMemoryError: Requested array size exceeds VM limit

Stack Overflow | user2738809 | 4 months ago
  1. 0

    Apache Pig: java.lang.OutOfMemoryError: Requested array size exceeds VM limit

    Stack Overflow | 4 months ago | user2738809
    org.apache.hadoop.mapred.YarnChild: Error running child : java.lang.OutOfMemoryError: Requested array size exceeds VM limit
  2. 0

    When trying to build any cubes in Kylin 1.5.2, I get Java heap space errors on step 2 (Extract Fact Table Distinct Columns). I experience the same heap space error on my own cube, as well as using the sample cube data (from {{sample.sh}}). I'm testing on a fairly stock HDP-2.2.9.0-3393 installation. I've tried tweaking the memory settings, but even with the map memory tuned to allow 8GB, I still get the same heap space error even for the small sample data (where 8GB seems like it should be more than plenty). If I try the same build against the sample data on a fresh 1.5.1 installation, the cube builds as expected, but with a fresh 1.5.2 installation, I get these heap space errors on any build attempts. Here's the backtrace from the failed map job: {code} 2016-05-29 01:51:57,910 WARN [main] org.apache.hadoop.metrics2.impl.MetricsConfig: Cannot locate configuration: tried hadoop-metrics2-maptask.properties,hadoop-metrics2.properties 2016-05-29 01:51:57,977 INFO [main] org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Scheduled snapshot period at 10 second(s). 2016-05-29 01:51:57,977 INFO [main] org.apache.hadoop.metrics2.impl.MetricsSystemImpl: MapTask metrics system started 2016-05-29 01:51:57,984 INFO [main] org.apache.hadoop.mapred.YarnChild: Executing with tokens: 2016-05-29 01:51:58,017 INFO [main] org.apache.hadoop.mapred.YarnChild: Kind: mapreduce.job, Service: job_1464481246919_0079, Ident: (org.apache.hadoop.mapreduce.security.token.JobTokenIdentifier@548e6d58) 2016-05-29 01:51:58,045 INFO [main] org.apache.hadoop.mapred.YarnChild: Sleeping for 0ms before retrying again. Got null now. 2016-05-29 01:51:58,249 INFO [main] org.apache.hadoop.mapred.YarnChild: mapreduce.cluster.local.dir for child: /hadoop/yarn/local/usercache/api-umbrella/appcache/application_1464481246919_0079 2016-05-29 01:51:58,443 INFO [main] org.apache.hadoop.conf.Configuration.deprecation: session.id is deprecated. Instead, use dfs.metrics.session-id 2016-05-29 01:51:58,792 INFO [main] org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter: File Output Committer Algorithm version is 1 2016-05-29 01:51:58,801 INFO [main] org.apache.hadoop.mapred.Task: Using ResourceCalculatorProcessTree : [ ] 2016-05-29 01:51:59,078 INFO [main] org.apache.hadoop.mapred.MapTask: Processing split: org.apache.hive.hcatalog.mapreduce.HCatSplit@12dae582 2016-05-29 01:51:59,218 ERROR [main] org.apache.hadoop.mapred.YarnChild: Error running child : java.lang.OutOfMemoryError: Java heap space at org.apache.hadoop.mapred.MapTask$MapOutputBuffer.init(MapTask.java:983) at org.apache.hadoop.mapred.MapTask.createSortingCollector(MapTask.java:401) at org.apache.hadoop.mapred.MapTask.access$100(MapTask.java:81) at org.apache.hadoop.mapred.MapTask$NewOutputCollector.<init>(MapTask.java:695) at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:767) at org.apache.hadoop.mapred.MapTask.run(MapTask.java:341) at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:167) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1671) at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:162) {code} Any ideas? Let me know if I can supply any other details. Thanks!

    Apache's JIRA Issue Tracker | 6 months ago | Nick Muerdter
    org.apache.hadoop.mapred.YarnChild: Error running child : java.lang.OutOfMemoryError: Java heap space
  3. 0

    hive error: java heap size | Optimal heap size for hive / php web Tools linux commands

    chinabtp.com | 1 year ago
    org.apache.hadoop.mapred.YarnChild: Error running child : java.lang.OutOfMemoryError: Java heap space
  4. Speed up your debug routine!

    Automated exception search integrated into your IDE

  5. 0

    Out of heap error when creating Index in Apache Hive

    Stack Overflow | 2 years ago | Magnus Eriksson
    org.apache.hadoop.mapred.YarnChild: Error running child : java.lang.OutOfMemoryError: Java heap space
  6. 0

    Hadoop Error: Java heap space when using big dataset

    Stack Overflow | 2 years ago
    org.apache.hadoop.mapred.YarnChild: Error running child : java.lang.OutOfMemoryError: Java heap space

    1 unregistered visitors
    Not finding the right solution?
    Take a tour to get the most out of Samebug.

    Tired of useless tips?

    Automated exception search integrated into your IDE

    Root Cause Analysis

    1. org.apache.hadoop.mapred.YarnChild

      Error running child : java.lang.OutOfMemoryError: Requested array size exceeds VM limit

      at java.util.Arrays.copyOf()
    2. Java RT
      DataOutputStream.writeUTF
      1. java.util.Arrays.copyOf(Arrays.java:2271)
      2. java.io.ByteArrayOutputStream.grow(ByteArrayOutputStream.java:113)
      3. java.io.ByteArrayOutputStream.ensureCapacity(ByteArrayOutputStream.java:93)
      4. java.io.ByteArrayOutputStream.write(ByteArrayOutputStream.java:140)
      5. java.io.DataOutputStream.write(DataOutputStream.java:107)
      6. java.io.DataOutputStream.writeUTF(DataOutputStream.java:401)
      7. java.io.DataOutputStream.writeUTF(DataOutputStream.java:323)
      7 frames
    3. org.apache.pig
      PigNullableWritable.write
      1. org.apache.pig.data.utils.SedesHelper.writeChararray(SedesHelper.java:66)
      2. org.apache.pig.data.BinInterSedes.writeDatum(BinInterSedes.java:580)
      3. org.apache.pig.data.BinInterSedes.writeDatum(BinInterSedes.java:462)
      4. org.apache.pig.data.utils.SedesHelper.writeGenericTuple(SedesHelper.java:135)
      5. org.apache.pig.data.BinInterSedes.writeTuple(BinInterSedes.java:650)
      6. org.apache.pig.data.BinInterSedes.writeBag(BinInterSedes.java:641)
      7. org.apache.pig.data.BinInterSedes.writeDatum(BinInterSedes.java:474)
      8. org.apache.pig.data.BinInterSedes.writeDatum(BinInterSedes.java:462)
      9. org.apache.pig.data.utils.SedesHelper.writeGenericTuple(SedesHelper.java:135)
      10. org.apache.pig.data.BinInterSedes.writeTuple(BinInterSedes.java:650)
      11. org.apache.pig.data.BinInterSedes.writeDatum(BinInterSedes.java:470)
      12. org.apache.pig.data.BinSedesTuple.write(BinSedesTuple.java:40)
      13. org.apache.pig.impl.io.PigNullableWritable.write(PigNullableWritable.java:139)
      13 frames
    4. Hadoop
      WritableSerialization$WritableSerializer.serialize
      1. org.apache.hadoop.io.serializer.WritableSerialization$WritableSerializer.serialize(WritableSerialization.java:98)
      2. org.apache.hadoop.io.serializer.WritableSerialization$WritableSerializer.serialize(WritableSerialization.java:82)
      2 frames
    5. Hadoop
      WrappedMapper$Context.write
      1. org.apache.hadoop.mapred.IFile$Writer.append(IFile.java:198)
      2. org.apache.hadoop.mapred.MapTask$MapOutputBuffer.spillSingleRecord(MapTask.java:1696)
      3. org.apache.hadoop.mapred.MapTask$MapOutputBuffer.collect(MapTask.java:1180)
      4. org.apache.hadoop.mapred.MapTask$NewOutputCollector.write(MapTask.java:712)
      5. org.apache.hadoop.mapreduce.task.TaskInputOutputContextImpl.write(TaskInputOutputContextImpl.java:89)
      6. org.apache.hadoop.mapreduce.lib.map.WrappedMapper$Context.write(WrappedMapper.java:112)
      6 frames
    6. org.apache.pig
      PigGenericMapBase.map
      1. org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigGenericMapReduce$Map.collect(PigGenericMapReduce.java:135)
      2. org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigGenericMapBase.runPipeline(PigGenericMapBase.java:281)
      3. org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigGenericMapBase.map(PigGenericMapBase.java:274)
      4. org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigGenericMapBase.map(PigGenericMapBase.java:64)
      4 frames