java.lang.IllegalArgumentException: Value not exists!

Apache's JIRA Issue Tracker | Richard Calaba | 7 months ago
  1. 0

    Getting exception in Step 4 - Build Dimension Dictionary: java.lang.IllegalArgumentException: Value not exists! at org.apache.kylin.dimension.Dictionary.getIdFromValueBytes( at org.apache.kylin.dict.TrieDictionary.getIdFromValueImpl( at org.apache.kylin.dimension.Dictionary.getIdFromValue( at org.apache.kylin.dimension.Dictionary.getIdFromValue( at org.apache.kylin.dict.lookup.SnapshotTable.takeSnapshot( at org.apache.kylin.dict.lookup.SnapshotManager.buildSnapshot( at org.apache.kylin.cube.CubeManager.buildSnapshotTable( at org.apache.kylin.cube.cli.DictionaryGeneratorCLI.processSegment( at org.apache.kylin.cube.cli.DictionaryGeneratorCLI.processSegment( at at at at at org.apache.kylin.job.execution.AbstractExecutable.execute( at org.apache.kylin.job.execution.DefaultChainedExecutable.doWork( at org.apache.kylin.job.execution.AbstractExecutable.execute( at org.apache.kylin.job.impl.threadpool.DefaultScheduler$ at java.util.concurrent.ThreadPoolExecutor.runWorker( at java.util.concurrent.ThreadPoolExecutor$ at result code:2 The code which generates the exception is: /** * A lower level API, return ID integer from raw value bytes. In case of not found * <p> * - if roundingFlag=0, throw IllegalArgumentException; <br> * - if roundingFlag<0, the closest smaller ID integer if exist; <br> * - if roundingFlag>0, the closest bigger ID integer if exist. <br> * <p> * Bypassing the cache layer, this could be significantly slower than getIdFromValue(T value). * * @throws IllegalArgumentException * if value is not found in dictionary and rounding is off; * or if rounding cannot find a smaller or bigger ID */ final public int getIdFromValueBytes(byte[] value, int offset, int len, int roundingFlag) throws IllegalArgumentException { if (isNullByteForm(value, offset, len)) return nullId(); else { int id = getIdFromValueBytesImpl(value, offset, len, roundingFlag); if (id < 0) throw new IllegalArgumentException("Value not exists!"); return id; } } ========================================================== The Cube is big - fact 110 mio rows, the largest dimension (customer) has 10 mio rows. I have increased the JVM -Xmx to 16gb and set the kylin.table.snapshot.max_mb=2048 in to make sure the Cube build doesn't fail (previously we were getting exception complaining about the 300MB limit for Dimension dictionary size (req. approx 700MB)). ========================================================== Before that we were getting exception complaining about the Dictionary encoding problem - "Too high cardinality is not suitable for dictionary -- cardinality: 10873977" - this we resolved by changing the affected dimension/row key Encoding from "dict" to "int; length=8" on the Advanced Settings of the Cube. ========================================================== We have 2 high-cardinality fields (one from fact table and one from the big dimension (customer - see above). We need to use in distinc_count measure for our calculations. I wonder if this exception Value not found! is somewhat related ??? Those count_distinct measures are defined one with return type "bitmap" (exact precission - only for Int columns) and 2nd with return type "hllc16" (error rate <= 1.22 %) ========================================================== I am looking for any clues to debug the cause of this error and way how to circumwent this ...

    Apache's JIRA Issue Tracker | 7 months ago | Richard Calaba
    java.lang.IllegalArgumentException: Value not exists!
  2. 0
    samebug tip
    I was missing a partitioning column because I did not specify the "basePath" option on read
  3. 0
    samebug tip
    Index settings can't be on the nodes via elasticsearch.yml. You need to set them at your index, or using index templates to avoid inconsistencies.
  4. Speed up your debug routine!

    Automated exception search integrated into your IDE

    Not finding the right solution?
    Take a tour to get the most out of Samebug.

    Tired of useless tips?

    Automated exception search integrated into your IDE

    Root Cause Analysis

    1. java.lang.IllegalArgumentException

      Value not exists!

      at org.apache.kylin.dimension.Dictionary.getIdFromValueBytes()
    2. org.apache.kylin
      1. org.apache.kylin.dimension.Dictionary.getIdFromValueBytes(
      2. org.apache.kylin.dict.TrieDictionary.getIdFromValueImpl(
      3. org.apache.kylin.dimension.Dictionary.getIdFromValue(
      4. org.apache.kylin.dimension.Dictionary.getIdFromValue(
      4 frames