java.lang.IllegalArgumentException: Comparison method violates its general contract!

github.com | 5 months ago
tip
Click on the to mark the solution that helps you, Samebug will learn from it.
As a community member, you’ll be rewarded for you help.
  1. 0

    [SPARK-3032] Potential bug when running sort-based shuffle with sorting using TimSort - ASF JIRA

    apache.org | 1 year ago
    java.lang.IllegalArgumentException: Comparison method violates its general contract!
  2. 0

    eco-release-metadata/RELEASENOTES.1.2.0.md at master · aw-was-here/eco-release-metadata · GitHub

    github.com | 5 months ago
    java.lang.IllegalArgumentException: Comparison method violates its general contract!

    Root Cause Analysis

    1. java.lang.IllegalArgumentException

      Comparison method violates its general contract!

      at org.apache.spark.util.collection.Sorter$SortState.mergeLo()
    2. Spark
      Executor$TaskRunner.run
      1. org.apache.spark.util.collection.Sorter$SortState.mergeLo(Sorter.java:755)
      2. org.apache.spark.util.collection.Sorter$SortState.mergeAt(Sorter.java:493)
      3. org.apache.spark.util.collection.Sorter$SortState.mergeCollapse(Sorter.java:420)
      4. org.apache.spark.util.collection.Sorter$SortState.access$200(Sorter.java:294)
      5. org.apache.spark.util.collection.Sorter.sort(Sorter.java:128)
      6. org.apache.spark.util.collection.SizeTrackingPairBuffer.destructiveSortedIterator(SizeTrackingPairBuffer.scala:83)
      7. org.apache.spark.util.collection.ExternalSorter.spillToMergeableFile(ExternalSorter.scala:323)
      8. org.apache.spark.util.collection.ExternalSorter.spill(ExternalSorter.scala:271)
      9. org.apache.spark.util.collection.ExternalSorter.maybeSpill(ExternalSorter.scala:249)
      10. org.apache.spark.util.collection.ExternalSorter.insertAll(ExternalSorter.scala:220)
      11. org.apache.spark.shuffle.sort.SortShuffleWriter.write(SortShuffleWriter.scala:85)
      12. org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:68)
      13. org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:41)
      14. org.apache.spark.scheduler.Task.run(Task.scala:54)
      15. org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:199)
      15 frames
    3. Java RT
      Thread.run
      1. java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
      2. java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
      3. java.lang.Thread.run(Thread.java:722)
      3 frames