java.lang.RuntimeException: Failed to create dictionary on DEFAULT.XXXXXXXXXXXXXXX_URL

Apache's JIRA Issue Tracker | Richard Calaba | 10 months ago
Do you know that we can give you better hits? Get more relevant results from Samebug’s stack trace search.
  1. 0

    Getting exception in Step 4 - Build Dimension Dictionary: java.lang.IllegalArgumentException: Value not exists! at org.apache.kylin.dimension.Dictionary.getIdFromValueBytes( at org.apache.kylin.dict.TrieDictionary.getIdFromValueImpl( at org.apache.kylin.dimension.Dictionary.getIdFromValue( at org.apache.kylin.dimension.Dictionary.getIdFromValue( at org.apache.kylin.dict.lookup.SnapshotTable.takeSnapshot( at org.apache.kylin.dict.lookup.SnapshotManager.buildSnapshot( at org.apache.kylin.cube.CubeManager.buildSnapshotTable( at org.apache.kylin.cube.cli.DictionaryGeneratorCLI.processSegment( at org.apache.kylin.cube.cli.DictionaryGeneratorCLI.processSegment( at at at at at org.apache.kylin.job.execution.AbstractExecutable.execute( at org.apache.kylin.job.execution.DefaultChainedExecutable.doWork( at org.apache.kylin.job.execution.AbstractExecutable.execute( at org.apache.kylin.job.impl.threadpool.DefaultScheduler$ at java.util.concurrent.ThreadPoolExecutor.runWorker( at java.util.concurrent.ThreadPoolExecutor$ at result code:2 The code which generates the exception is: /** * A lower level API, return ID integer from raw value bytes. In case of not found * <p> * - if roundingFlag=0, throw IllegalArgumentException; <br> * - if roundingFlag<0, the closest smaller ID integer if exist; <br> * - if roundingFlag>0, the closest bigger ID integer if exist. <br> * <p> * Bypassing the cache layer, this could be significantly slower than getIdFromValue(T value). * * @throws IllegalArgumentException * if value is not found in dictionary and rounding is off; * or if rounding cannot find a smaller or bigger ID */ final public int getIdFromValueBytes(byte[] value, int offset, int len, int roundingFlag) throws IllegalArgumentException { if (isNullByteForm(value, offset, len)) return nullId(); else { int id = getIdFromValueBytesImpl(value, offset, len, roundingFlag); if (id < 0) throw new IllegalArgumentException("Value not exists!"); return id; } } ========================================================== The Cube is big - fact 110 mio rows, the largest dimension (customer) has 10 mio rows. I have increased the JVM -Xmx to 16gb and set the kylin.table.snapshot.max_mb=2048 in to make sure the Cube build doesn't fail (previously we were getting exception complaining about the 300MB limit for Dimension dictionary size (req. approx 700MB)). ========================================================== Before that we were getting exception complaining about the Dictionary encoding problem - "Too high cardinality is not suitable for dictionary -- cardinality: 10873977" - this we resolved by changing the affected dimension/row key Encoding from "dict" to "int; length=8" on the Advanced Settings of the Cube. ========================================================== We have 2 high-cardinality fields (one from fact table and one from the big dimension (customer - see above). We need to use in distinc_count measure for our calculations. I wonder if this exception Value not found! is somewhat related ??? Those count_distinct measures are defined one with return type "bitmap" (exact precission - only for Int columns) and 2nd with return type "hllc16" (error rate <= 1.22 %) ========================================================== I am looking for any clues to debug the cause of this error and way how to circumwent this ...

    Apache's JIRA Issue Tracker | 10 months ago | Richard Calaba
    java.lang.RuntimeException: Failed to create dictionary on DEFAULT.XXXXXXXXXXXXXXX_URL

    Root Cause Analysis

    1. java.lang.NegativeArraySizeException

      No message provided

      at org.apache.kylin.dict.TrieDictionaryBuilder.buildTrieBytes()
    2. org.apache.kylin
      1. org.apache.kylin.dict.TrieDictionaryBuilder.buildTrieBytes(
      3. org.apache.kylin.dict.DictionaryGenerator$
      4. org.apache.kylin.dict.DictionaryGenerator.buildDictionary(
      5. org.apache.kylin.dict.DictionaryGenerator.buildDictionary(
      6. org.apache.kylin.dict.DictionaryManager.buildDictionary(
      6 frames