org.apache.hadoop.mapred.YarnChild: Exception running child : java.lang.IllegalArgumentException: Value not exists!

Apache's JIRA Issue Tracker | Richard Calaba | 5 months ago
  1. 0

    Getting exception in Step 4 - Build Dimension Dictionary: java.lang.IllegalArgumentException: Value not exists! at org.apache.kylin.dimension.Dictionary.getIdFromValueBytes(Dictionary.java:160) at org.apache.kylin.dict.TrieDictionary.getIdFromValueImpl(TrieDictionary.java:158) at org.apache.kylin.dimension.Dictionary.getIdFromValue(Dictionary.java:96) at org.apache.kylin.dimension.Dictionary.getIdFromValue(Dictionary.java:76) at org.apache.kylin.dict.lookup.SnapshotTable.takeSnapshot(SnapshotTable.java:96) at org.apache.kylin.dict.lookup.SnapshotManager.buildSnapshot(SnapshotManager.java:106) at org.apache.kylin.cube.CubeManager.buildSnapshotTable(CubeManager.java:215) at org.apache.kylin.cube.cli.DictionaryGeneratorCLI.processSegment(DictionaryGeneratorCLI.java:59) at org.apache.kylin.cube.cli.DictionaryGeneratorCLI.processSegment(DictionaryGeneratorCLI.java:42) at org.apache.kylin.engine.mr.steps.CreateDictionaryJob.run(CreateDictionaryJob.java:56) at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70) at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:84) at org.apache.kylin.engine.mr.common.HadoopShellExecutable.doWork(HadoopShellExecutable.java:60) at org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:114) at org.apache.kylin.job.execution.DefaultChainedExecutable.doWork(DefaultChainedExecutable.java:50) at org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:114) at org.apache.kylin.job.impl.threadpool.DefaultScheduler$JobRunner.run(DefaultScheduler.java:124) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:745) result code:2 The code which generates the exception is: org.apache.kylin.dimension.Dictionary.java: /** * A lower level API, return ID integer from raw value bytes. In case of not found * <p> * - if roundingFlag=0, throw IllegalArgumentException; <br> * - if roundingFlag<0, the closest smaller ID integer if exist; <br> * - if roundingFlag>0, the closest bigger ID integer if exist. <br> * <p> * Bypassing the cache layer, this could be significantly slower than getIdFromValue(T value). * * @throws IllegalArgumentException * if value is not found in dictionary and rounding is off; * or if rounding cannot find a smaller or bigger ID */ final public int getIdFromValueBytes(byte[] value, int offset, int len, int roundingFlag) throws IllegalArgumentException { if (isNullByteForm(value, offset, len)) return nullId(); else { int id = getIdFromValueBytesImpl(value, offset, len, roundingFlag); if (id < 0) throw new IllegalArgumentException("Value not exists!"); return id; } } ========================================================== The Cube is big - fact 110 mio rows, the largest dimension (customer) has 10 mio rows. I have increased the JVM -Xmx to 16gb and set the kylin.table.snapshot.max_mb=2048 in kylin.properties to make sure the Cube build doesn't fail (previously we were getting exception complaining about the 300MB limit for Dimension dictionary size (req. approx 700MB)). ========================================================== Before that we were getting exception complaining about the Dictionary encoding problem - "Too high cardinality is not suitable for dictionary -- cardinality: 10873977" - this we resolved by changing the affected dimension/row key Encoding from "dict" to "int; length=8" on the Advanced Settings of the Cube. ========================================================== We have 2 high-cardinality fields (one from fact table and one from the big dimension (customer - see above). We need to use in distinc_count measure for our calculations. I wonder if this exception Value not found! is somewhat related ??? Those count_distinct measures are defined one with return type "bitmap" (exact precission - only for Int columns) and 2nd with return type "hllc16" (error rate <= 1.22 %) ========================================================== I am looking for any clues to debug the cause of this error and way how to circumwent this ...

    Apache's JIRA Issue Tracker | 5 months ago | Richard Calaba
    org.apache.hadoop.mapred.YarnChild: Exception running child : java.lang.IllegalArgumentException: Value not exists!
  2. 0

    This is a minor issue per se. I had a typo in my script specifying a negative number of reducers for the SleepJob. It results in the exception that is far from the root cause, and appeared as a serious issue with the map-side sort. {noformat} 2014-06-17 21:42:48,072 INFO [main] org.apache.hadoop.mapred.MapTask: Ignoring exception during close for org.apache.hadoop.mapred.MapTask$NewOutputCollector@972141f java.lang.NullPointerException at org.apache.hadoop.mapred.MapTask$MapOutputBuffer.flush(MapTask.java:1447) at org.apache.hadoop.mapred.MapTask$NewOutputCollector.close(MapTask.java:700) at org.apache.hadoop.mapred.MapTask.closeQuietly(MapTask.java:1990) at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:774) at org.apache.hadoop.mapred.MapTask.run(MapTask.java:340) at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:173) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:415) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1626) at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:168) 2014-06-17 21:42:48,075 WARN [main] org.apache.hadoop.mapred.YarnChild: Exception running child : java.lang.IllegalArgumentException at java.nio.ByteBuffer.allocate(ByteBuffer.java:330) at org.apache.hadoop.mapred.SpillRecord.<init>(SpillRecord.java:51) at org.apache.hadoop.mapred.MapTask$MapOutputBuffer.mergeParts(MapTask.java:1824) at org.apache.hadoop.mapred.MapTask$MapOutputBuffer.flush(MapTask.java:1484) at org.apache.hadoop.mapred.MapTask$NewOutputCollector.close(MapTask.java:700) at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:770) at org.apache.hadoop.mapred.MapTask.run(MapTask.java:340) at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:173) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:415) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1626) at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:168) {noformat}

    Apache's JIRA Issue Tracker | 2 years ago | Gera Shegalov
    org.apache.hadoop.mapred.YarnChild: Exception running child : java.lang.IllegalArgumentException
  3. 0

    [HBase-issues] [jira] [Created] (HBASE-9744) Set io.serializations correctly in HFileOutputFormat.configureIncrementalLoad - Grokbase

    grokbase.com | 1 year ago
    org.apache.hadoop.mapred.YarnChild: Exception running child : java.lang.NullPointerException
  4. Speed up your debug routine!

    Automated exception search integrated into your IDE

  5. 0

    [Hive-dev] Hive and MR2 - Grokbase

    grokbase.com | 1 year ago
    org.apache.hadoop.mapred.YarnChild: Exception running child : java.lang.NullPointerException
  6. 0

    How do we diagnose a problem within a hunk MR job - Question | Splunk Answers

    splunk.com | 9 months ago
    org.apache.hadoop.mapred.YarnChild: Exception running child : java.lang.NullPointerException

    Not finding the right solution?
    Take a tour to get the most out of Samebug.

    Tired of useless tips?

    Automated exception search integrated into your IDE

    Root Cause Analysis

    1. org.apache.hadoop.mapred.YarnChild

      Exception running child : java.lang.IllegalArgumentException: Value not exists!

      at org.apache.kylin.common.util.Dictionary.getIdFromValueBytes()
    2. org.apache.kylin
      MergeCuboidMapper.map
      1. org.apache.kylin.common.util.Dictionary.getIdFromValueBytes(Dictionary.java:162)
      2. org.apache.kylin.common.util.Dictionary.getIdFromValueBytes(Dictionary.java:140)
      3. org.apache.kylin.engine.mr.steps.MergeCuboidMapper.map(MergeCuboidMapper.java:207)
      4. org.apache.kylin.engine.mr.steps.MergeCuboidMapper.map(MergeCuboidMapper.java:63)
      4 frames
    3. Hadoop
      YarnChild$2.run
      1. org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:145)
      2. org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:764)
      3. org.apache.hadoop.mapred.MapTask.run(MapTask.java:340)
      4. org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:167)
      4 frames
    4. Java RT
      Subject.doAs
      1. java.security.AccessController.doPrivileged(Native Method)
      2. javax.security.auth.Subject.doAs(Subject.java:415)
      2 frames
    5. Hadoop
      UserGroupInformation.doAs
      1. org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657)
      1 frame
    6. Hadoop
      YarnChild.main
      1. org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:162)
      1 frame