java.lang.UnsupportedOperationException: Cannot evaluate expression: PythonUDF#<lambda>(input[2, StringType])

Stack Overflow | user5147250 | 8 months ago
tip
Click on the to mark the solution that helps you, Samebug will learn from it.
As a community member, you’ll be rewarded for you help.
  1. 0

    Spark-Submit python file on cluster

    Stack Overflow | 8 months ago | user5147250
    org.apache.spark.api.python.PythonException: Traceback (most recent call last): File "/ephemeral/usr/hdp/2.3.4.33-1/spark/python/lib/pyspark.zip/pyspark/worker.py", line 98, in main command = pickleSer._read_with_length(infile) File "/ephemeral/usr/hdp/2.3.4.33-1/spark/python/lib/pyspark.zip/pyspark/serializers.py", line 156, in _read_with_length length = read_int(stream) File "/ephemeral/usr/hdp/2.3.4.33-1/spark/python/lib/pyspark.zip/pyspark/serializers.py", line 545, in read_int raise EOFError EOFError

    Root Cause Analysis

    1. java.lang.UnsupportedOperationException

      Cannot evaluate expression: PythonUDF#<lambda>(input[2, StringType])

      at org.apache.spark.sql.catalyst.expressions.Unevaluable$class.eval()
    2. Spark Project Catalyst
      Unevaluable$class.eval
      1. org.apache.spark.sql.catalyst.expressions.Unevaluable$class.eval(Expression.scala:188)
      1 frame
    3. Spark Project SQL
      PythonUDF.eval
      1. org.apache.spark.sql.execution.PythonUDF.eval(python.scala:44)
      1 frame
    4. Spark Project Catalyst
      InterpretedMutableProjection.apply
      1. org.apache.spark.sql.catalyst.expressions.InterpretedMutableProjection.apply(Projection.scala:82)
      2. org.apache.spark.sql.catalyst.expressions.InterpretedMutableProjection.apply(Projection.scala:61)
      2 frames
    5. Spark Project SQL
      BatchPythonEvaluation$$anonfun$doExecute$1$$anonfun$10$$anonfun$11.apply
      1. org.apache.spark.sql.execution.BatchPythonEvaluation$$anonfun$doExecute$1$$anonfun$10$$anonfun$11.apply(python.scala:379)
      2. org.apache.spark.sql.execution.BatchPythonEvaluation$$anonfun$doExecute$1$$anonfun$10$$anonfun$11.apply(python.scala:377)
      2 frames
    6. Scala
      AbstractTraversable.map
      1. scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:244)
      2. scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:244)
      3. scala.collection.immutable.List.foreach(List.scala:318)
      4. scala.collection.TraversableLike$class.map(TraversableLike.scala:244)
      5. scala.collection.AbstractTraversable.map(Traversable.scala:105)
      5 frames
    7. Spark Project SQL
      BatchPythonEvaluation$$anonfun$doExecute$1$$anonfun$10.apply
      1. org.apache.spark.sql.execution.BatchPythonEvaluation$$anonfun$doExecute$1$$anonfun$10.apply(python.scala:377)
      2. org.apache.spark.sql.execution.BatchPythonEvaluation$$anonfun$doExecute$1$$anonfun$10.apply(python.scala:376)
      2 frames
    8. Scala
      AbstractIterator.foreach
      1. scala.collection.Iterator$$anon$11.next(Iterator.scala:328)
      2. scala.collection.Iterator$class.foreach(Iterator.scala:727)
      3. scala.collection.AbstractIterator.foreach(Iterator.scala:1157)
      3 frames
    9. Spark
      PythonRunner$WriterThread.run
      1. org.apache.spark.api.python.PythonRDD$.writeIteratorToStream(PythonRDD.scala:452)
      2. org.apache.spark.api.python.PythonRunner$WriterThread$$anonfun$run$3.apply(PythonRDD.scala:280)
      3. org.apache.spark.util.Utils$.logUncaughtExceptions(Utils.scala:1699)
      4. org.apache.spark.api.python.PythonRunner$WriterThread.run(PythonRDD.scala:239)
      4 frames