org.apache.spark.SparkException

Task not serializable

Samebug tips0

There are no available Samebug tips for this exception. If you know how to solve this issue, help other users by writing a short tip.

Don't give up yet. Paste your full stack trace to get a solution.

Solutions on the web1754

  • Task not serializable org.apache.spark.SparkException: Task not serializable
  • via Stack Overflow by Steven.Prgm
    , 8 months ago
    Task not serializable org.apache.spark.SparkException: Task not serializable
  • via GitHub by xubo245
    , 3 months ago
    Task not serializable
  • Stack trace

    • org.apache.spark.SparkException: Task not serializable at org.apache.spark.util.ClosureCleaner$.ensureSerializable(ClosureCleaner.scala:304) at org.apache.spark.util.ClosureCleaner$.org$apache$spark$util$ClosureCleaner$$clean(ClosureCleaner.scala:294) at org.apache.spark.util.ClosureCleaner$.clean(ClosureCleaner.scala:122) at org.apache.spark.SparkContext.clean(SparkContext.scala:2021) at org.apache.spark.rdd.RDD$$anonfun$map$1.apply(RDD.scala:314) at org.apache.spark.rdd.RDD$$anonfun$map$1.apply(RDD.scala:313) at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:147) at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:108) at org.apache.spark.rdd.RDD.withScope(RDD.scala:306) at org.apache.spark.rdd.RDD.map(RDD.scala:313) at commbank.grimlock.spark.ReducibleMatrix$class.melt(Matrix.scala:749) at commbank.grimlock.spark.Matrix5D.melt(Matrix.scala:1438) at commbank.grimlock.spark.examples.Conditional$.main(Conditional.scala:65) at commbank.grimlock.spark.examples.Conditional.main(Conditional.scala) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:672) at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:180) at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:205) at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:120) at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala) Caused by: java.io.NotSerializableException: shapeless.package$$anon$2 Serialization stack: - object not serializable (class: shapeless.package$$anon$2, value: shapeless.package$$anon$2@166b21ea) - field (class: commbank.grimlock.spark.ReducibleMatrix$$anonfun$melt$1, name: ev3$19, type: interface shapeless.package$$eq$colon$bang$eq) - object (class commbank.grimlock.spark.ReducibleMatrix$$anonfun$melt$1, <function1>) at org.apache.spark.serializer.SerializationDebugger$.improveException(SerializationDebugger.scala:40) at org.apache.spark.serializer.JavaSerializationStream.writeObject(JavaSerializer.scala:47) at org.apache.spark.serializer.JavaSerializerInstance.serialize(JavaSerializer.scala:84) at org.apache.spark.util.ClosureCleaner$.ensureSerializable(ClosureCleaner.scala:301) ... 22 more

    Write tip

    You have a different solution? A short tip here would help you and many other users who saw this issue last week.

    Users with the same issue

    15 times, 5 months ago
    Unknown user
    Once, 1 year ago
    Once, 1 month ago
    Once, 1 month ago
    14 times, 4 months ago
    7 more bugmates