org.apache.spark.SparkException: YarnSparkHadoopUtil is not available in non-YARN mode!

Stack Overflow | aamirr | 2 months ago
tip
Do you find the tips below useful? Click on the to mark them and say thanks to rafael . Or join the community to write better ones.
  1. 0

    Branch 1.5 by somideshmukh · Pull Request #9464 · apache/spark · GitHub

    github.com | 1 year ago
    org.apache.spark.SparkException: YarnSparkHadoopUtil is not available in non-YARN mode!
  2. 0
    samebug tip
    Change the queue name to hadoop
  3. 0

    Unable to run pyspark in yarn-client mode (pyspark standalone is working though)

    Stack Overflow | 2 months ago | aamirr
    org.apache.spark.SparkException: YarnSparkHadoopUtil is not available in non-YARN mode!
  4. Speed up your debug routine!

    Automated exception search integrated into your IDE

  5. 0

    Apache spark job failed to deploy on Yarn

    Stack Overflow | 2 years ago | Adetiloye Philip Kehinde
    org.apache.spark.SparkException: Error asking standalone scheduler to shut down executors at org.apache.spark.scheduler.cluster.CoarseGrainedSchedulerBackend.stopExecutors(C oarseGrainedSchedulerBackend.scala:261) at org.apache.spark.scheduler.cluster.CoarseGrainedSchedulerBackend.stop(CoarseGrai nedSchedulerBackend.scala:2 66) at org.apache.spark.scheduler.cluster.YarnClientSchedulerBackend.stop(YarnClientSch edulerBackend.scala:158) at org.apache.spark.scheduler.TaskSchedulerImpl.stop(TaskSchedulerImpl.scala:416)

    Not finding the right solution?
    Take a tour to get the most out of Samebug.

    Tired of useless tips?

    Automated exception search integrated into your IDE

    Root Cause Analysis

    1. org.apache.spark.SparkException

      YarnSparkHadoopUtil is not available in non-YARN mode!

      at org.apache.spark.deploy.yarn.YarnSparkHadoopUtil$.get()
    2. Spark Project YARN Stable API
      YarnSparkHadoopUtil$.get
      1. org.apache.spark.deploy.yarn.YarnSparkHadoopUtil$.get(YarnSparkHadoopUtil.scala:352)
      1 frame
    3. Spark
      JavaSparkContext.<init>
      1. org.apache.spark.scheduler.cluster.YarnClientSchedulerBackend.stop(YarnClientSchedulerBackend.scala:152)
      2. org.apache.spark.scheduler.TaskSchedulerImpl.stop(TaskSchedulerImpl.scala:455)
      3. org.apache.spark.scheduler.DAGScheduler.stop(DAGScheduler.scala:1605)
      4. org.apache.spark.SparkContext$$anonfun$stop$8.apply$mcV$sp(SparkContext.scala:1798)
      5. org.apache.spark.util.Utils$.tryLogNonFatalError(Utils.scala:1287)
      6. org.apache.spark.SparkContext.stop(SparkContext.scala:1797)
      7. org.apache.spark.SparkContext.<init>(SparkContext.scala:565)
      8. org.apache.spark.api.java.JavaSparkContext.<init>(JavaSparkContext.scala:58)
      8 frames
    4. Java RT
      Constructor.newInstance
      1. sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
      2. sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
      3. sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
      4. java.lang.reflect.Constructor.newInstance(Constructor.java:423)
      4 frames
    5. Py4J
      GatewayConnection.run
      1. py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:240)
      2. py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:357)
      3. py4j.Gateway.invoke(Gateway.java:236)
      4. py4j.commands.ConstructorCommand.invokeConstructor(ConstructorCommand.java:80)
      5. py4j.commands.ConstructorCommand.execute(ConstructorCommand.java:69)
      6. py4j.GatewayConnection.run(GatewayConnection.java:214)
      6 frames
    6. Java RT
      Thread.run
      1. java.lang.Thread.run(Thread.java:745)
      1 frame