org.apache.spark.SparkException: Job aborted due to stage failure: Task 0 in stage 3.0 failed 1 times, most recent failure: Lost task 0.0 in stage 3.0 (TID 3, localhost): java.lang.UnsupportedOperationException: Cannot evaluate expression: PythonUDF#<lambda>(input[2, StringType])

Stack Overflow | user5147250 | 8 months ago
tip
Do you know that we can give you better hits? Get more relevant results from Samebug’s stack trace search.
  1. 0

    Spark-Submit python file on cluster

    Stack Overflow | 8 months ago | user5147250
    org.apache.spark.SparkException: Job aborted due to stage failure: Task 0 in stage 3.0 failed 1 times, most recent failure: Lost task 0.0 in stage 3.0 (TID 3, localhost): java.lang.UnsupportedOperationException: Cannot evaluate expression: PythonUDF#<lambda>(input[2, StringType])

    Root Cause Analysis

    1. org.apache.spark.SparkException

      Job aborted due to stage failure: Task 0 in stage 3.0 failed 1 times, most recent failure: Lost task 0.0 in stage 3.0 (TID 3, localhost): java.lang.UnsupportedOperationException: Cannot evaluate expression: PythonUDF#<lambda>(input[2, StringType])

      at org.apache.spark.sql.catalyst.expressions.Unevaluable$class.eval()
    2. Spark Project Catalyst
      Unevaluable$class.eval
      1. org.apache.spark.sql.catalyst.expressions.Unevaluable$class.eval(Expression.scala:188)
      1 frame
    3. Spark Project SQL
      PythonUDF.eval
      1. org.apache.spark.sql.execution.PythonUDF.eval(python.scala:44)
      1 frame
    4. Spark Project Catalyst
      InterpretedMutableProjection.apply
      1. org.apache.spark.sql.catalyst.expressions.InterpretedMutableProjection.apply(Projection.scala:82)
      2. org.apache.spark.sql.catalyst.expressions.InterpretedMutableProjection.apply(Projection.scala:61)
      2 frames
    5. Spark Project SQL
      BatchPythonEvaluation$$anonfun$doExecute$1$$anonfun$10$$anonfun$11.apply
      1. org.apache.spark.sql.execution.BatchPythonEvaluation$$anonfun$doExecute$1$$anonfun$10$$anonfun$11.apply(python.scala:379)
      2. org.apache.spark.sql.execution.BatchPythonEvaluation$$anonfun$doExecute$1$$anonfun$10$$anonfun$11.apply(python.scala:377)
      2 frames
    6. Scala
      AbstractTraversable.map
      1. scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:244)
      2. scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:244)
      3. scala.collection.immutable.List.foreach(List.scala:318)
      4. scala.collection.TraversableLike$class.map(TraversableLike.scala:244)
      5. scala.collection.AbstractTraversable.map(Traversable.scala:105)
      5 frames
    7. Spark Project SQL
      BatchPythonEvaluation$$anonfun$doExecute$1$$anonfun$10.apply
      1. org.apache.spark.sql.execution.BatchPythonEvaluation$$anonfun$doExecute$1$$anonfun$10.apply(python.scala:377)
      2. org.apache.spark.sql.execution.BatchPythonEvaluation$$anonfun$doExecute$1$$anonfun$10.apply(python.scala:376)
      2 frames
    8. Scala
      AbstractIterator.foreach
      1. scala.collection.Iterator$$anon$11.next(Iterator.scala:328)
      2. scala.collection.Iterator$class.foreach(Iterator.scala:727)
      3. scala.collection.AbstractIterator.foreach(Iterator.scala:1157)
      3 frames
    9. Spark
      PythonRunner$WriterThread.run
      1. org.apache.spark.api.python.PythonRDD$.writeIteratorToStream(PythonRDD.scala:452)
      2. org.apache.spark.api.python.PythonRunner$WriterThread$$anonfun$run$3.apply(PythonRDD.scala:280)
      3. org.apache.spark.util.Utils$.logUncaughtExceptions(Utils.scala:1699)
      4. org.apache.spark.api.python.PythonRunner$WriterThread.run(PythonRDD.scala:239)
      4 frames