org.apache.spark.SparkException

Task not serializable


Solutions on the web1796

Solution icon of stackoverflow
Task not serializable org.apache.spark.SparkException: Task not serializable

Solution icon of stackoverflow
via Stack Overflow by Steven.Prgm
, 9 months ago
Task not serializable org.apache.spark.SparkException: Task not serializable

Solution icon of github
via GitHub by sgomezg
, 1 year ago
Task not serializable

Solution icon of github
via GitHub by xubo245
, 4 months ago
Task not serializable

Solution icon of github
via GitHub by jbloom22
, 11 months ago
Task not serializable

Solution icon of github
via GitHub by pdendek
, 1 year ago
Task not serializable

Solution icon of github
Task not serializable

Solution icon of github
via GitHub by gad2103
, 9 months ago
Task not serializable

Solution icon of github
Task not serializable

Solution icon of googlegroups
via Google Groups by 周梦想, 2 weeks ago
Task not serializable

Stack trace

  • org.apache.spark.SparkException: Task not serializable at org.apache.spark.util.ClosureCleaner$.ensureSerializable(ClosureCleaner.scala:304) at org.apache.spark.util.ClosureCleaner$.org$apache$spark$util$ClosureCleaner$$clean(ClosureCleaner.scala:294) at org.apache.spark.util.ClosureCleaner$.clean(ClosureCleaner.scala:122) at org.apache.spark.SparkContext.clean(SparkContext.scala:2055) at org.apache.spark.rdd.RDD$$anonfun$flatMap$1.apply(RDD.scala:333) at org.apache.spark.rdd.RDD$$anonfun$flatMap$1.apply(RDD.scala:332) at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:150) at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:111) at org.apache.spark.rdd.RDD.withScope(RDD.scala:316) at org.apache.spark.rdd.RDD.flatMap(RDD.scala:332) at geotrellis.spark.merge.RDDLayoutMergeMethods.merge(RddLayoutMergeMethods.scala:26) at treecanopy.Ingest$$anonfun$main$1.apply(Main.scala:77) at treecanopy.Ingest$$anonfun$main$1.apply(Main.scala:74) at scala.collection.immutable.Stream.foreach(Stream.scala:547) at treecanopy.Ingest$.main(Main.scala:74) at treecanopy.Ingest.main(Main.scala) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at org.apache.spark.deploy.yarn.ApplicationMaster$$anon$2.run(ApplicationMaster.scala:542) Caused by: java.io.NotSerializableException: geotrellis.spark.package$ Serialization stack: - object not serializable (class: geotrellis.spark.package$, value: geotrellis.spark.package$@1c2e0107) - field (class: geotrellis.spark.merge.Implicits$withRDDLayoutMergeMethods, name: $outer, type: interface geotrellis.spark.merge.Implicits) - object (class geotrellis.spark.merge.Implicits$withRDDLayoutMergeMethods, geotrellis.spark.merge.Implicits$withRDDLayoutMergeMethods@37acb901) - field (class: geotrellis.spark.merge.RDDLayoutMergeMethods$$anonfun$1, name: $outer, type: class geotrellis.spark.merge.RDDLayoutMergeMethods) - object (class geotrellis.spark.merge.RDDLayoutMergeMethods$$anonfun$1, <function1>) at org.apache.spark.serializer.SerializationDebugger$.improveException(SerializationDebugger.scala:40) at org.apache.spark.serializer.JavaSerializationStream.writeObject(JavaSerializer.scala:47) at org.apache.spark.serializer.JavaSerializerInstance.serialize(JavaSerializer.scala:101) at org.apache.spark.util.ClosureCleaner$.ensureSerializable(ClosureCleaner.scala:301) ... 20 more

Write tip

You have a different solution? A short tip here would help you and many other users who saw this issue last week.

Users with the same issue

Once, 2 months ago
Once, 2 months ago
14 times, 5 months ago
15 times, 6 months ago
30 times, 7 months ago
7 more bugmates