org.apache.spark.SparkException: Task not serializable

GitHub | jbloom22 | 1 year ago
tip
Your exception is missing from the Samebug knowledge base.
Here are the best solutions we found on the Internet.
Click on the to mark the helpful solution and get rewards for you help.
  1. 0

    GitHub comment 68#155304880

    GitHub | 1 year ago | jbloom22
    org.apache.spark.SparkException: Task not serializable
  2. 0

    How to pass spark broadcast and accumulator variables to map and reduce functions

    Stack Overflow | 2 years ago | Sohaib
    org.apache.spark.SparkException: Task not serializable
  3. Speed up your debug routine!

    Automated exception search integrated into your IDE

  4. 0

    Why does using custom case class in Spark shell lead to serialization error?

    Stack Overflow | 1 year ago | drecute
    java.lang.reflect.InvocationTargetException
  5. 0

    Yarn/ hdfs attempting to use kerberos when security is disabled

    Stack Overflow | 1 year ago | javadba
    java.lang.IllegalArgumentException: Can't get Kerberos realm

  1. r1chjames 1 times, last 3 months ago
  2. bmacedo 8 times, last 3 months ago
  3. aldrinleal 1 times, last 4 months ago
  4. ajinkya_w 96 times, last 4 months ago
  5. davidvanlaatum 176 times, last 5 months ago
16 more registered users
79 unregistered visitors
Not finding the right solution?
Take a tour to get the most out of Samebug.

Tired of useless tips?

Automated exception search integrated into your IDE

Root Cause Analysis

  1. java.lang.ArrayIndexOutOfBoundsException

    1

    at java.io.ObjectStreamClass$FieldReflector.getObjFieldValues()
  2. Java RT
    Method.invoke
    1. java.io.ObjectStreamClass$FieldReflector.getObjFieldValues(ObjectStreamClass.java:2050)
    2. java.io.ObjectStreamClass.getObjFieldValues(ObjectStreamClass.java:1252)
    3. sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    4. sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    5. sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    6. java.lang.reflect.Method.invoke(Method.java:497)
    6 frames
  3. Spark
    RDD.map
    1. org.apache.spark.serializer.SerializationDebugger$ObjectStreamClassMethods$.getObjFieldValues$extension(SerializationDebugger.scala:240)
    2. org.apache.spark.serializer.SerializationDebugger$SerializationDebugger.visitSerializable(SerializationDebugger.scala:150)
    3. org.apache.spark.serializer.SerializationDebugger$SerializationDebugger.visit(SerializationDebugger.scala:99)
    4. org.apache.spark.serializer.SerializationDebugger$SerializationDebugger.visitSerializable(SerializationDebugger.scala:158)
    5. org.apache.spark.serializer.SerializationDebugger$SerializationDebugger.visit(SerializationDebugger.scala:99)
    6. org.apache.spark.serializer.SerializationDebugger$SerializationDebugger.visitSerializable(SerializationDebugger.scala:158)
    7. org.apache.spark.serializer.SerializationDebugger$SerializationDebugger.visit(SerializationDebugger.scala:99)
    8. org.apache.spark.serializer.SerializationDebugger$.find(SerializationDebugger.scala:58)
    9. org.apache.spark.serializer.SerializationDebugger$.improveException(SerializationDebugger.scala:39)
    10. org.apache.spark.serializer.JavaSerializationStream.writeObject(JavaSerializer.scala:47)
    11. org.apache.spark.serializer.JavaSerializerInstance.serialize(JavaSerializer.scala:80)
    12. org.apache.spark.util.ClosureCleaner$.ensureSerializable(ClosureCleaner.scala:164)
    13. org.apache.spark.util.ClosureCleaner$.clean(ClosureCleaner.scala:158)
    14. org.apache.spark.SparkContext.clean(SparkContext.scala:1622)
    15. org.apache.spark.rdd.RDD.map(RDD.scala:286)
    15 frames
  4. org.broadinstitute.hail
    MendelErrorsSuite.test
    1. org.broadinstitute.hail.methods.MendelErrors.writeMendel(MendelErrors.scala:143)
    2. org.broadinstitute.hail.methods.MendelErrorsSuite.test(MendelErrorsSuite.scala:50)
    2 frames
  5. Java RT
    Method.invoke
    1. sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    2. sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    3. sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    4. java.lang.reflect.Method.invoke(Method.java:497)
    4 frames
  6. TestNG
    RemoteTestNGStarter.main
    1. org.testng.internal.MethodInvocationHelper.invokeMethod(MethodInvocationHelper.java:85)
    2. org.testng.internal.Invoker.invokeMethod(Invoker.java:696)
    3. org.testng.internal.Invoker.invokeTestMethod(Invoker.java:882)
    4. org.testng.internal.Invoker.invokeTestMethods(Invoker.java:1189)
    5. org.testng.internal.TestMethodWorker.invokeTestMethods(TestMethodWorker.java:124)
    6. org.testng.internal.TestMethodWorker.run(TestMethodWorker.java:108)
    7. org.testng.TestRunner.privateRun(TestRunner.java:767)
    8. org.testng.TestRunner.run(TestRunner.java:617)
    9. org.testng.SuiteRunner.runTest(SuiteRunner.java:348)
    10. org.testng.SuiteRunner.runSequentially(SuiteRunner.java:343)
    11. org.testng.SuiteRunner.privateRun(SuiteRunner.java:305)
    12. org.testng.SuiteRunner.run(SuiteRunner.java:254)
    13. org.testng.SuiteRunnerWorker.runSuite(SuiteRunnerWorker.java:52)
    14. org.testng.SuiteRunnerWorker.run(SuiteRunnerWorker.java:86)
    15. org.testng.TestNG.runSuitesSequentially(TestNG.java:1224)
    16. org.testng.TestNG.runSuitesLocally(TestNG.java:1149)
    17. org.testng.TestNG.run(TestNG.java:1057)
    18. org.testng.IDEARemoteTestNG.run(IDEARemoteTestNG.java:72)
    19. org.testng.RemoteTestNGStarter.main(RemoteTestNGStarter.java:122)
    19 frames
  7. Java RT
    Method.invoke
    1. sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    2. sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    3. sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    4. java.lang.reflect.Method.invoke(Method.java:497)
    4 frames
  8. IDEA
    AppMain.main
    1. com.intellij.rt.execution.application.AppMain.main(AppMain.java:144)
    1 frame