org.apache.spark.scheduler.TaskSetManager: Lost task 0.0 in stage 6.0 (TID 71, cluster-1-w-0.c.justeat-datalake.internal): java.lang.NoSuchMethodError: org.apache.avro.generic.GenericData.createDatumWriter(Lorg/apache/avro/Schema;)Lorg/apache/avro/io/DatumWriter;

GitHub | samelamin | 9 months ago
tip
Do you know that we can give you better hits? Get more relevant results from Samebug’s stack trace search.
  1. 0

    Failing to save dataframe to bigquery

    GitHub | 9 months ago | samelamin
    org.apache.spark.scheduler.TaskSetManager: Lost task 0.0 in stage 6.0 (TID 71, cluster-1-w-0.c.justeat-datalake.internal): java.lang.NoSuchMethodError: org.apache.avro.generic.GenericData.createDatumWriter(Lorg/apache/avro/Schema;)Lorg/apache/avro/io/DatumWriter;

    Root Cause Analysis

    1. org.apache.spark.scheduler.TaskSetManager

      Lost task 0.0 in stage 6.0 (TID 71, cluster-1-w-0.c.justeat-datalake.internal): java.lang.NoSuchMethodError: org.apache.avro.generic.GenericData.createDatumWriter(Lorg/apache/avro/Schema;)Lorg/apache/avro/io/DatumWriter;

      at org.apache.avro.mapreduce.AvroKeyRecordWriter.<init>()
    2. Apache Avro Mapred API
      AvroKeyOutputFormat.getRecordWriter
      1. org.apache.avro.mapreduce.AvroKeyRecordWriter.<init>(AvroKeyRecordWriter.java:55)
      2. org.apache.avro.mapreduce.AvroKeyOutputFormat$RecordWriterFactory.create(AvroKeyOutputFormat.java:79)
      3. org.apache.avro.mapreduce.AvroKeyOutputFormat.getRecordWriter(AvroKeyOutputFormat.java:105)
      3 frames
    3. com.databricks.spark
      AvroOutputWriterFactory.newInstance
      1. com.databricks.spark.avro.AvroOutputWriter.<init>(AvroOutputWriter.scala:82)
      2. com.databricks.spark.avro.AvroOutputWriterFactory.newInstance(AvroOutputWriterFactory.scala:31)
      2 frames
    4. org.apache.spark
      InsertIntoHadoopFsRelation$$anonfun$run$1$$anonfun$apply$mcV$sp$3.apply
      1. org.apache.spark.sql.execution.datasources.BaseWriterContainer.newOutputWriter(WriterContainer.scala:129)
      2. org.apache.spark.sql.execution.datasources.DefaultWriterContainer.writeRows(WriterContainer.scala:255)
      3. org.apache.spark.sql.execution.datasources.InsertIntoHadoopFsRelation$$anonfun$run$1$$anonfun$apply$mcV$sp$3.apply(InsertIntoHadoopFsRelation.scala:148)
      4. org.apache.spark.sql.execution.datasources.InsertIntoHadoopFsRelation$$anonfun$run$1$$anonfun$apply$mcV$sp$3.apply(InsertIntoHadoopFsRelation.scala:148)
      4 frames
    5. Spark
      Executor$TaskRunner.run
      1. org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:66)
      2. org.apache.spark.scheduler.Task.run(Task.scala:89)
      3. org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:227)
      3 frames
    6. Java RT
      Thread.run
      1. java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
      2. java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
      3. java.lang.Thread.run(Thread.java:745)
      3 frames