java.lang.IllegalArgumentException: Required executor memory (1500+8704 MB) is above the max threshold (8192 MB) of this cluster! Please check the values of 'yarn.scheduler.maximum-allocation-mb' and/or 'yarn.nodemanager.resource.memory-mb'.

GitHub | Asmaa-Ali | 9 months ago
tip
Do you know that we can give you better hits? Get more relevant results from Samebug’s stack trace search.
  1. 0

    Problem running SparkBWA

    GitHub | 9 months ago | Asmaa-Ali
    java.lang.IllegalArgumentException: Required executor memory (1500+8704 MB) is above the max threshold (8192 MB) of this cluster! Please check the values of 'yarn.scheduler.maximum-allocation-mb' and/or 'yarn.nodemanager.resource.memory-mb'.

    Root Cause Analysis

    1. java.lang.IllegalArgumentException

      Required executor memory (1500+8704 MB) is above the max threshold (8192 MB) of this cluster! Please check the values of 'yarn.scheduler.maximum-allocation-mb' and/or 'yarn.nodemanager.resource.memory-mb'.

      at org.apache.spark.deploy.yarn.Client.verifyClusterResources()
    2. Spark Project YARN Stable API
      Client.submitApplication
      1. org.apache.spark.deploy.yarn.Client.verifyClusterResources(Client.scala:283)
      2. org.apache.spark.deploy.yarn.Client.submitApplication(Client.scala:139)
      2 frames
    3. Spark
      JavaSparkContext.<init>
      1. org.apache.spark.scheduler.cluster.YarnClientSchedulerBackend.start(YarnClientSchedulerBackend.scala:57)
      2. org.apache.spark.scheduler.TaskSchedulerImpl.start(TaskSchedulerImpl.scala:144)
      3. org.apache.spark.SparkContext.<init>(SparkContext.scala:530)
      4. org.apache.spark.api.java.JavaSparkContext.<init>(JavaSparkContext.scala:59)
      4 frames
    4. Unknown
      SparkBWA.main
      1. BwaInterpreter.initInterpreter(BwaInterpreter.java:123)
      2. BwaInterpreter.<init>(BwaInterpreter.java:94)
      3. SparkBWA.main(SparkBWA.java:25)
      3 frames
    5. Java RT
      Method.invoke
      1. sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
      2. sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
      3. sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
      4. java.lang.reflect.Method.invoke(Method.java:498)
      4 frames
    6. Spark
      SparkSubmit.main
      1. org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:731)
      2. org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:181)
      3. org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:206)
      4. org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:121)
      5. org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
      5 frames