org.apache.spark.SparkException: Task not serializable

search-hadoop.com | 8 months ago
tip
Click on the to mark the solution that helps you, Samebug will learn from it.
As a community member, you’ll be rewarded for you help.
  1. 0

    Spark, mail # user - SPARK Exception thrown in awaitResult - 2016-07-28, 09:45

    search-hadoop.com | 8 months ago
    org.apache.spark.SparkException: Task not serializable
  2. 0

    Spark UDF how to convert Map to column

    Stack Overflow | 5 months ago | nir
    org.apache.spark.SparkException: Task not serializable
  3. Speed up your debug routine!

    Automated exception search integrated into your IDE

  4. 0

    spark structured streaming (java): task not serializable

    Stack Overflow | 2 weeks ago | user2221654
    org.apache.spark.SparkException: Task not serializable
  5. 0

    GitHub comment 394#196891797

    GitHub | 1 year ago | timodonnell
    org.apache.spark.SparkException: Task not serializable
Not finding the right solution?
Take a tour to get the most out of Samebug.

Tired of useless tips?

Automated exception search integrated into your IDE

Root Cause Analysis

  1. org.apache.spark.SparkException

    Task not serializable

    at org.apache.spark.util.ClosureCleaner$.ensureSerializable()
  2. Spark
    RDD.mapPartitionsWithIndex
    1. org.apache.spark.util.ClosureCleaner$.ensureSerializable(ClosureCleaner.scala:298)
    2. org.apache.spark.util.ClosureCleaner$.org$apache$spark$util$ClosureCleaner$$clean(ClosureCleaner.scala:288)
    3. org.apache.spark.util.ClosureCleaner$.clean(ClosureCleaner.scala:108)
    4. org.apache.spark.SparkContext.clean(SparkContext.scala:2037)
    5. org.apache.spark.rdd.RDD$$anonfun$mapPartitionsWithIndex$1.apply(RDD.scala:798)
    6. org.apache.spark.rdd.RDD$$anonfun$mapPartitionsWithIndex$1.apply(RDD.scala:797)
    7. org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:151)
    8. org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:112)
    9. org.apache.spark.rdd.RDD.withScope(RDD.scala:358)
    10. org.apache.spark.rdd.RDD.mapPartitionsWithIndex(RDD.scala:797)
    10 frames
  3. Spark Project SQL
    SparkPlan$$anonfun$executeQuery$1.apply
    1. org.apache.spark.sql.execution.WholeStageCodegenExec.doExecute(WholeStageCodegenExec.scala:364)
    2. org.apache.spark.sql.execution.SparkPlan$$anonfun$execute$1.apply(SparkPlan.scala:115)
    3. org.apache.spark.sql.execution.SparkPlan$$anonfun$execute$1.apply(SparkPlan.scala:115)
    4. org.apache.spark.sql.execution.SparkPlan$$anonfun$executeQuery$1.apply(SparkPlan.scala:136)
    4 frames
  4. Spark
    RDDOperationScope$.withScope
    1. org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:151)
    1 frame
  5. Spark Project SQL
    Dataset.take
    1. org.apache.spark.sql.execution.SparkPlan.executeQuery(SparkPlan.scala:133)
    2. org.apache.spark.sql.execution.SparkPlan.execute(SparkPlan.scala:114)
    3. org.apache.spark.sql.execution.TakeOrderedAndProjectExec.executeCollect(limit.scala:128)
    4. org.apache.spark.sql.Dataset$$anonfun$org$apache$spark$sql$Dataset$$execute$1$1.apply(Dataset.scala:2183)
    5. org.apache.spark.sql.execution.SQLExecution$.withNewExecutionId(SQLExecution.scala:57)
    6. org.apache.spark.sql.Dataset.withNewExecutionId(Dataset.scala:2532)
    7. org.apache.spark.sql.Dataset.org$apache$spark$sql$Dataset$$execute$1(Dataset.scala:2182)
    8. org.apache.spark.sql.Dataset.org$apache$spark$sql$Dataset$$collect(Dataset.scala:2189)
    9. org.apache.spark.sql.Dataset$$anonfun$head$1.apply(Dataset.scala:1925)
    10. org.apache.spark.sql.Dataset$$anonfun$head$1.apply(Dataset.scala:1924)
    11. org.apache.spark.sql.Dataset.withTypedCallback(Dataset.scala:2562)
    12. org.apache.spark.sql.Dataset.head(Dataset.scala:1924)
    13. org.apache.spark.sql.Dataset.take(Dataset.scala:2139)
    13 frames
  6. org.mksmart.amaretto
    DatasetPerHourVerOneTest.testBuildDatasetNew
    1. org.mksmart.amaretto.ml.DatasetPerHourVerOneTest.testBuildDatasetNew(DatasetPerHourVerOneTest.java:202)
    1 frame
  7. Java RT
    Method.invoke
    1. sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    2. sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    3. sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    4. java.lang.reflect.Method.invoke(Method.java:497)
    4 frames
  8. JUnit
    ParentRunner.run
    1. org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
    2. org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
    3. org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
    4. org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
    5. org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
    6. org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
    7. org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325)
    8. org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78)
    9. org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57)
    10. org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
    11. org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
    12. org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
    13. org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
    14. org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
    15. org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
    16. org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
    17. org.junit.runners.ParentRunner.run(ParentRunner.java:363)
    17 frames
  9. JUnit4 Provider
    JUnit4Provider.invoke
    1. org.apache.maven.surefire.junit4.JUnit4TestSet.execute(JUnit4TestSet.java:53)
    2. org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:123)
    3. org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:104)
    3 frames
  10. Java RT
    NativeMethodAccessorImpl.invoke0
    1. sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    1 frame