org.apache.spark.SparkException: Task not serializable

search-hadoop.com | 9 months ago
tip
Your exception is missing from the Samebug knowledge base.
Here are the best solutions we found on the Internet.
Click on the to mark the helpful solution and get rewards for you help.
  1. 0

    Spark, mail # user - SPARK Exception thrown in awaitResult - 2016-07-28, 09:45

    search-hadoop.com | 9 months ago
    org.apache.spark.SparkException: Task not serializable
  2. 0

    Spark UDF how to convert Map to column

    Stack Overflow | 6 months ago | nir
    org.apache.spark.SparkException: Task not serializable
  3. Speed up your debug routine!

    Automated exception search integrated into your IDE

  4. 0

    spark structured streaming (java): task not serializable

    Stack Overflow | 1 month ago | user2221654
    org.apache.spark.SparkException: Task not serializable
  5. 0

    GitHub comment 394#196891797

    GitHub | 1 year ago | timodonnell
    org.apache.spark.SparkException: Task not serializable
Not finding the right solution?
Take a tour to get the most out of Samebug.

Tired of useless tips?

Automated exception search integrated into your IDE

Root Cause Analysis

  1. org.apache.spark.SparkException

    Task not serializable

    at org.apache.spark.util.ClosureCleaner$.ensureSerializable()
  2. Spark
    RDD.mapPartitionsWithIndex
    1. org.apache.spark.util.ClosureCleaner$.ensureSerializable(ClosureCleaner.scala:298)
    2. org.apache.spark.util.ClosureCleaner$.org$apache$spark$util$ClosureCleaner$$clean(ClosureCleaner.scala:288)
    3. org.apache.spark.util.ClosureCleaner$.clean(ClosureCleaner.scala:108)
    4. org.apache.spark.SparkContext.clean(SparkContext.scala:2037)
    5. org.apache.spark.rdd.RDD$$anonfun$mapPartitionsWithIndex$1.apply(RDD.scala:798)
    6. org.apache.spark.rdd.RDD$$anonfun$mapPartitionsWithIndex$1.apply(RDD.scala:797)
    7. org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:151)
    8. org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:112)
    9. org.apache.spark.rdd.RDD.withScope(RDD.scala:358)
    10. org.apache.spark.rdd.RDD.mapPartitionsWithIndex(RDD.scala:797)
    10 frames
  3. Spark Project SQL
    SparkPlan$$anonfun$executeQuery$1.apply
    1. org.apache.spark.sql.execution.WholeStageCodegenExec.doExecute(WholeStageCodegenExec.scala:364)
    2. org.apache.spark.sql.execution.SparkPlan$$anonfun$execute$1.apply(SparkPlan.scala:115)
    3. org.apache.spark.sql.execution.SparkPlan$$anonfun$execute$1.apply(SparkPlan.scala:115)
    4. org.apache.spark.sql.execution.SparkPlan$$anonfun$executeQuery$1.apply(SparkPlan.scala:136)
    4 frames
  4. Spark
    RDDOperationScope$.withScope
    1. org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:151)
    1 frame
  5. Spark Project SQL
    Dataset.take
    1. org.apache.spark.sql.execution.SparkPlan.executeQuery(SparkPlan.scala:133)
    2. org.apache.spark.sql.execution.SparkPlan.execute(SparkPlan.scala:114)
    3. org.apache.spark.sql.execution.TakeOrderedAndProjectExec.executeCollect(limit.scala:128)
    4. org.apache.spark.sql.Dataset$$anonfun$org$apache$spark$sql$Dataset$$execute$1$1.apply(Dataset.scala:2183)
    5. org.apache.spark.sql.execution.SQLExecution$.withNewExecutionId(SQLExecution.scala:57)
    6. org.apache.spark.sql.Dataset.withNewExecutionId(Dataset.scala:2532)
    7. org.apache.spark.sql.Dataset.org$apache$spark$sql$Dataset$$execute$1(Dataset.scala:2182)
    8. org.apache.spark.sql.Dataset.org$apache$spark$sql$Dataset$$collect(Dataset.scala:2189)
    9. org.apache.spark.sql.Dataset$$anonfun$head$1.apply(Dataset.scala:1925)
    10. org.apache.spark.sql.Dataset$$anonfun$head$1.apply(Dataset.scala:1924)
    11. org.apache.spark.sql.Dataset.withTypedCallback(Dataset.scala:2562)
    12. org.apache.spark.sql.Dataset.head(Dataset.scala:1924)
    13. org.apache.spark.sql.Dataset.take(Dataset.scala:2139)
    13 frames
  6. org.mksmart.amaretto
    DatasetPerHourVerOneTest.testBuildDatasetNew
    1. org.mksmart.amaretto.ml.DatasetPerHourVerOneTest.testBuildDatasetNew(DatasetPerHourVerOneTest.java:202)
    1 frame
  7. Java RT
    Method.invoke
    1. sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    2. sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    3. sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    4. java.lang.reflect.Method.invoke(Method.java:497)
    4 frames
  8. JUnit
    ParentRunner.run
    1. org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
    2. org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
    3. org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
    4. org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
    5. org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
    6. org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
    7. org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325)
    8. org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78)
    9. org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57)
    10. org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
    11. org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
    12. org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
    13. org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
    14. org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
    15. org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
    16. org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
    17. org.junit.runners.ParentRunner.run(ParentRunner.java:363)
    17 frames
  9. JUnit4 Provider
    JUnit4Provider.invoke
    1. org.apache.maven.surefire.junit4.JUnit4TestSet.execute(JUnit4TestSet.java:53)
    2. org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:123)
    3. org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:104)
    3 frames
  10. Java RT
    NativeMethodAccessorImpl.invoke0
    1. sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    1 frame