org.apache.spark.SparkException: Task not serializable

GitHub | AndreasKostler | 4 months ago
  1. 0

    GitHub comment 622#237200173

    GitHub | 4 months ago | AndreasKostler
    org.apache.spark.SparkException: Task not serializable
  2. 0

    DefaultCSVFormat, CSVFormat and TSVFormat are not serializable

    GitHub | 2 years ago | dgomesbr
    org.apache.spark.SparkException: Task not serializable
  3. 0

    Spark cluster computing framework

    gmane.org | 1 year ago
    org.apache.spark.SparkException: Task not serializable at org.apache.spark.util.ClosureCleaner$.ensureSerializable(ClosureCleaner.scala:166)
  4. Speed up your debug routine!

    Automated exception search integrated into your IDE

  5. 0

    Parsing date time information from CSV in Zeppelin and Spark

    Stack Overflow | 1 year ago | Jinho Yoo
    org.apache.spark.SparkException: Task not serializable
  6. 0

    Databricks Apache Spark 1.4: Task not serializable (Scala)

    Stack Overflow | 1 year ago | user3399275
    org.apache.spark.SparkException: Task not serializable

  1. Nikolay Rybak 14 times, last 3 weeks ago
2 unregistered visitors
Not finding the right solution?
Take a tour to get the most out of Samebug.

Tired of useless tips?

Automated exception search integrated into your IDE

Root Cause Analysis

  1. java.io.NotSerializableException

    shapeless.package$$anon$2 Serialization stack: - object not serializable (class: shapeless.package$$anon$2, value: shapeless.package$$anon$2@166b21ea) - field (class: commbank.grimlock.spark.ReducibleMatrix$$anonfun$melt$1, name: ev3$19, type: interface shapeless.package$$eq$colon$bang$eq) - object (class commbank.grimlock.spark.ReducibleMatrix$$anonfun$melt$1, <function1>)

    at org.apache.spark.serializer.SerializationDebugger$.improveException()
  2. Spark
    RDD.map
    1. org.apache.spark.serializer.SerializationDebugger$.improveException(SerializationDebugger.scala:40)
    2. org.apache.spark.serializer.JavaSerializationStream.writeObject(JavaSerializer.scala:47)
    3. org.apache.spark.serializer.JavaSerializerInstance.serialize(JavaSerializer.scala:84)
    4. org.apache.spark.util.ClosureCleaner$.ensureSerializable(ClosureCleaner.scala:301)
    5. org.apache.spark.util.ClosureCleaner$.org$apache$spark$util$ClosureCleaner$$clean(ClosureCleaner.scala:294)
    6. org.apache.spark.util.ClosureCleaner$.clean(ClosureCleaner.scala:122)
    7. org.apache.spark.SparkContext.clean(SparkContext.scala:2021)
    8. org.apache.spark.rdd.RDD$$anonfun$map$1.apply(RDD.scala:314)
    9. org.apache.spark.rdd.RDD$$anonfun$map$1.apply(RDD.scala:313)
    10. org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:147)
    11. org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:108)
    12. org.apache.spark.rdd.RDD.withScope(RDD.scala:306)
    13. org.apache.spark.rdd.RDD.map(RDD.scala:313)
    13 frames
  3. commbank.grimlock.spark
    Conditional.main
    1. commbank.grimlock.spark.ReducibleMatrix$class.melt(Matrix.scala:749)
    2. commbank.grimlock.spark.Matrix5D.melt(Matrix.scala:1438)
    3. commbank.grimlock.spark.examples.Conditional$.main(Conditional.scala:65)
    4. commbank.grimlock.spark.examples.Conditional.main(Conditional.scala)
    4 frames
  4. Java RT
    Method.invoke
    1. sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    2. sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
    3. sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    4. java.lang.reflect.Method.invoke(Method.java:606)
    4 frames
  5. Spark
    SparkSubmit.main
    1. org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:672)
    2. org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:180)
    3. org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:205)
    4. org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:120)
    5. org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
    5 frames