java.lang.IllegalArgumentException: Required executor memory (1500+8704 MB) is above the max threshold (8192 MB) of this cluster! Please check the values of 'yarn.scheduler.maximum-allocation-mb' and/or 'yarn.nodemanager.resource.memory-mb'.

GitHub | Asmaa-Ali | 11 months ago
tip
Your exception is missing from the Samebug knowledge base.
Here are the best solutions we found on the Internet.
Click on the to mark the helpful solution and get rewards for you help.
  1. 0

    Problem running SparkBWA

    GitHub | 11 months ago | Asmaa-Ali
    java.lang.IllegalArgumentException: Required executor memory (1500+8704 MB) is above the max threshold (8192 MB) of this cluster! Please check the values of 'yarn.scheduler.maximum-allocation-mb' and/or 'yarn.nodemanager.resource.memory-mb'.

    Root Cause Analysis

    1. java.lang.IllegalArgumentException

      Required executor memory (1500+8704 MB) is above the max threshold (8192 MB) of this cluster! Please check the values of 'yarn.scheduler.maximum-allocation-mb' and/or 'yarn.nodemanager.resource.memory-mb'.

      at org.apache.spark.deploy.yarn.Client.verifyClusterResources()
    2. Spark Project YARN Stable API
      Client.submitApplication
      1. org.apache.spark.deploy.yarn.Client.verifyClusterResources(Client.scala:283)
      2. org.apache.spark.deploy.yarn.Client.submitApplication(Client.scala:139)
      2 frames
    3. Spark
      JavaSparkContext.<init>
      1. org.apache.spark.scheduler.cluster.YarnClientSchedulerBackend.start(YarnClientSchedulerBackend.scala:57)
      2. org.apache.spark.scheduler.TaskSchedulerImpl.start(TaskSchedulerImpl.scala:144)
      3. org.apache.spark.SparkContext.<init>(SparkContext.scala:530)
      4. org.apache.spark.api.java.JavaSparkContext.<init>(JavaSparkContext.scala:59)
      4 frames
    4. Unknown
      SparkBWA.main
      1. BwaInterpreter.initInterpreter(BwaInterpreter.java:123)
      2. BwaInterpreter.<init>(BwaInterpreter.java:94)
      3. SparkBWA.main(SparkBWA.java:25)
      3 frames
    5. Java RT
      Method.invoke
      1. sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
      2. sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
      3. sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
      4. java.lang.reflect.Method.invoke(Method.java:498)
      4 frames
    6. Spark
      SparkSubmit.main
      1. org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:731)
      2. org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:181)
      3. org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:206)
      4. org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:121)
      5. org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
      5 frames