java.lang.IllegalArgumentException: Comparison method violates its general contract!

spark-issues | Sital Kedia (JIRA) | 1 year ago
tip
Do you know that we can give you better hits? Get more relevant results from Samebug’s stack trace search.
  1. 0

    [jira] [Created] (SPARK-13850) TimSort Comparison method violates its general contract

    spark-issues | 1 year ago | Sital Kedia (JIRA)
    java.lang.IllegalArgumentException: Comparison method violates its general contract!
  2. 0

    Running an SQL query that has a sub query and multiple left joins fails when there is a CLUSTER BY (which implies a sortBy). This gives the following stack trace; {code} Job aborted due to stage failure: Task 118 in stage 4.0 failed 4 times, most recent failure: Lost task 118.3 in stage 4.0 (TID 18392, node142): java.lang.IllegalArgumentException: Comparison method violates its general contract! at org.apache.spark.util.collection.TimSort$SortState.mergeHi(TimSort.java:900) at org.apache.spark.util.collection.TimSort$SortState.mergeAt(TimSort.java:509) at org.apache.spark.util.collection.TimSort$SortState.mergeCollapse(TimSort.java:435) at org.apache.spark.util.collection.TimSort$SortState.access$200(TimSort.java:307) at org.apache.spark.util.collection.TimSort.sort(TimSort.java:135) at org.apache.spark.util.collection.Sorter.sort(Sorter.scala:37) at org.apache.spark.util.collection.PartitionedPairBuffer.partitionedDestructiveSortedIterator(PartitionedPairBuffer.scala:70) at org.apache.spark.util.collection.ExternalSorter.partitionedIterator(ExternalSorter.scala:690) at org.apache.spark.util.collection.ExternalSorter.iterator(ExternalSorter.scala:708) at org.apache.spark.sql.execution.ExternalSort$$anonfun$doExecute$6$$anonfun$apply$7.apply(basicOperators.scala:222) at org.apache.spark.sql.execution.ExternalSort$$anonfun$doExecute$6$$anonfun$apply$7.apply(basicOperators.scala:218) at org.apache.spark.rdd.RDD$$anonfun$mapPartitions$1$$anonfun$apply$17.apply(RDD.scala:686) at org.apache.spark.rdd.RDD$$anonfun$mapPartitions$1$$anonfun$apply$17.apply(RDD.scala:686) at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:35) at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:277) at org.apache.spark.rdd.RDD.iterator(RDD.scala:244) at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:35) at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:277) at org.apache.spark.rdd.RDD.iterator(RDD.scala:244) at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:35) at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:277) at org.apache.spark.rdd.RDD.iterator(RDD.scala:244) at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:35) at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:277) at org.apache.spark.rdd.RDD.iterator(RDD.scala:244) at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:63) at org.apache.spark.scheduler.Task.run(Task.scala:70) at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:213) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:745) Driver stacktrace: {code} The query looks like; {code} val df = sqlContext.sql("""SELECT CID |, PW_END_DATE |, PROD_NBR_KEY |, SUM(CASE WHEN SUBST_IDX = 1 THEN L13W_SALE END) AS SUB1_L13W_SALE |FROM |(SELECT BASE.CID |, BASE.PW_END_DATE |, BASE.PROD_NBR_KEY |, SUBN.SUBST_IDX |, CASE WHEN IDX.PW_END_DATE BETWEEN DATE_SUB(BASE.PW_END_DATE, 13*7 - 1) AND BASE.PW_END_DATE THEN IDX.TOT_AMT_INCLD_GST END AS L13W_SALE |FROM TESTBASE BASE |LEFT JOIN TABLX SUBN |ON BASE.PROD_NBR_KEY = SUBN.PROD_NBR_KEY AND SUBN.SUBST_IDX <= 3 |LEFT JOIN TABLEF IDX |ON BASE.CRN = IDX.CRN |AND SUBN.CROSS_PROD_NBR = IDX.PROD_NBR_KEY |) SUBSPREM | GROUP BY CRN, PW_END_DATE, PROD_NBR_KEY""".stripMargin) {code}

    Apache's JIRA Issue Tracker | 2 years ago | Nathan McCarthy
    java.lang.IllegalArgumentException: Comparison method violates its general contract!
  3. 0

    When we need to sort rows based on partitioning columns within a task, timsort somehow failed. I do not know what is the cause. The only thing I know right now is that the partitioning columns are three string columns. {code} org.apache.spark.SparkException: Task failed while writing rows. at org.apache.spark.sql.execution.datasources.DynamicPartitionWriterContainer.writeRows(WriterContainer.scala:396) at org.apache.spark.sql.execution.datasources.InsertIntoHadoopFsRelation$$anonfun$run$1$$anonfun$apply$mcV$sp$3.apply(InsertIntoHadoopFsRelation.scala:150) at org.apache.spark.sql.execution.datasources.InsertIntoHadoopFsRelation$$anonfun$run$1$$anonfun$apply$mcV$sp$3.apply(InsertIntoHadoopFsRelation.scala:150) at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:66) at org.apache.spark.scheduler.Task.run(Task.scala:88) at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:214) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: java.lang.IllegalArgumentException: Comparison method violates its general contract! at org.apache.spark.util.collection.TimSort$SortState.mergeLo(TimSort.java:794) at org.apache.spark.util.collection.TimSort$SortState.mergeAt(TimSort.java:525) at org.apache.spark.util.collection.TimSort$SortState.mergeCollapse(TimSort.java:453) at org.apache.spark.util.collection.TimSort$SortState.access$200(TimSort.java:325) at org.apache.spark.util.collection.TimSort.sort(TimSort.java:153) at org.apache.spark.util.collection.Sorter.sort(Sorter.scala:37) at org.apache.spark.util.collection.unsafe.sort.UnsafeInMemorySorter.getSortedIterator(UnsafeInMemorySorter.java:231) at org.apache.spark.util.collection.unsafe.sort.UnsafeExternalSorter.getSortedIterator(UnsafeExternalSorter.java:404) at org.apache.spark.sql.execution.UnsafeKVExternalSorter.sortedIterator(UnsafeKVExternalSorter.java:159) at org.apache.spark.sql.execution.datasources.DynamicPartitionWriterContainer.writeRows(WriterContainer.scala:365) ... 8 more {code}

    Apache's JIRA Issue Tracker | 1 year ago | Yin Huai
    org.apache.spark.SparkException: Task failed while writing rows.
  4. Speed up your debug routine!

    Automated exception search integrated into your IDE

  5. 0

    Universal Image Loader : IllegalArgumentException when using FileNameGenerator with extension

    Stack Overflow | 2 years ago
    java.lang.IllegalArgumentException: keys must match regex [a-z0-9_-] {1,64}: "1828294.jpg" at com.nostra13.universalimageloader.cache.disc.impl.ext.DiskLruCache.valida teKey(DiskLruCache.java:697) at com.nostra13.universalimageloader.cache.disc.impl.ext.DiskLruCache.get(Di skLruCache.java:414) at com.nostra13.universalimageloader.cache.disc.impl.ext.LruDiscCache.get(Lr uDiscCache.java:133) at com.nostra13.universalimageloader.core.ImageLoaderEngine$1.run(ImageLoade rEngine.java:72) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java :1112) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.jav a:587)
  6. 0

    [elasticsearch] elasticsearch couchdb-river startup issues - Grokbase

    grokbase.com | 9 months ago
    java.lang.IllegalArgumentException: URI can't be null. at sun.net.spi.DefaultProxySelector.select(DefaultProxySelector.java:141) at sun.net.www.protocol.http.HttpURLConnection.plainConnect(HttpURLConnection.java:925) at sun.net.www.protocol.http.HttpURLConnection.connect(HttpURLConnection.java:849) at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1299) at org.elasticsearch.river.couchdb.CouchdbRiver$Slurper.run(CouchdbRiver.java:468)

    Not finding the right solution?
    Take a tour to get the most out of Samebug.

    Tired of useless tips?

    Automated exception search integrated into your IDE

    Root Cause Analysis

    1. java.lang.IllegalArgumentException

      Comparison method violates its general contract!

      at org.apache.spark.util.collection.TimSort$SortState.mergeLo()
    2. Spark
      Sorter.sort
      1. org.apache.spark.util.collection.TimSort$SortState.mergeLo(TimSort.java:794)
      2. org.apache.spark.util.collection.TimSort$SortState.mergeAt(TimSort.java:525)
      3. org.apache.spark.util.collection.TimSort$SortState.mergeCollapse(TimSort.java:453)
      4. org.apache.spark.util.collection.TimSort$SortState.access$200(TimSort.java:325)
      5. org.apache.spark.util.collection.TimSort.sort(TimSort.java:153)
      6. org.apache.spark.util.collection.Sorter.sort(Sorter.scala:37)
      6 frames
    3. org.apache.spark
      UnsafeExternalSorter.insertRecord
      1. org.apache.spark.util.collection.unsafe.sort.UnsafeInMemorySorter.getSortedIterator(UnsafeInMemorySorter.java:228)
      2. org.apache.spark.util.collection.unsafe.sort.UnsafeExternalSorter.spill(UnsafeExternalSorter.java:186)
      3. org.apache.spark.memory.TaskMemoryManager.acquireExecutionMemory(TaskMemoryManager.java:175)
      4. org.apache.spark.memory.TaskMemoryManager.allocatePage(TaskMemoryManager.java:249)
      5. org.apache.spark.memory.MemoryConsumer.allocatePage(MemoryConsumer.java:112)
      6. org.apache.spark.util.collection.unsafe.sort.UnsafeExternalSorter.acquireNewPageIfNecessary(UnsafeExternalSorter.java:318)
      7. org.apache.spark.util.collection.unsafe.sort.UnsafeExternalSorter.insertRecord(UnsafeExternalSorter.java:333)
      7 frames
    4. Spark Project SQL
      Sort$$anonfun$1.apply
      1. org.apache.spark.sql.execution.UnsafeExternalRowSorter.insertRow(UnsafeExternalRowSorter.java:91)
      2. org.apache.spark.sql.execution.UnsafeExternalRowSorter.sort(UnsafeExternalRowSorter.java:168)
      3. org.apache.spark.sql.execution.Sort$$anonfun$1.apply(Sort.scala:90)
      4. org.apache.spark.sql.execution.Sort$$anonfun$1.apply(Sort.scala:64)
      4 frames
    5. Spark
      Executor$TaskRunner.run
      1. org.apache.spark.rdd.RDD$$anonfun$mapPartitionsInternal$1$$anonfun$apply$21.apply(RDD.scala:728)
      2. org.apache.spark.rdd.RDD$$anonfun$mapPartitionsInternal$1$$anonfun$apply$21.apply(RDD.scala:728)
      3. org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)
      4. org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:306)
      5. org.apache.spark.rdd.RDD.iterator(RDD.scala:270)
      6. org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)
      7. org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:306)
      8. org.apache.spark.rdd.RDD.iterator(RDD.scala:270)
      9. org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)
      10. org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:306)
      11. org.apache.spark.rdd.RDD.iterator(RDD.scala:270)
      12. org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)
      13. org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:306)
      14. org.apache.spark.rdd.RDD.iterator(RDD.scala:270)
      15. org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)
      16. org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:306)
      17. org.apache.spark.rdd.RDD.iterator(RDD.scala:270)
      18. org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:66)
      19. org.apache.spark.scheduler.Task.run(Task.scala:89)
      20. org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:214)
      20 frames
    6. Java RT
      Thread.run
      1. java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
      2. java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
      3. java.lang.Thread.run(Thread.java:745)
      3 frames