scheduler.TaskSetManager: Lost task 0.0 in stage 1.0 (TID 3, ip-172-31-32-183.us-west-2.compute.internal): java.lang.NoClassDefFoundError: org/apache/kafka/common/message/KafkaLZ4BlockOutputStream

spark-user | Ted Yu | 1 year ago
tip
Click on the to mark the solution that helps you, Samebug will learn from it.
As a community member, you’ll be rewarded for you help.
  1. 0

    Re: Spark Streaming: java.lang.NoClassDefFoundError: org/apache/kafka/common/message/KafkaLZ4BlockOutputStream

    spark-user | 1 year ago | Ted Yu
    scheduler.TaskSetManager: Lost task 0.0 in stage 1.0 (TID 3, ip-172-31-32-183.us-west-2.compute.internal): java.lang.NoClassDefFoundError: org/apache/kafka/common/message/KafkaLZ4BlockOutputStream

    Root Cause Analysis

    1. scheduler.TaskSetManager

      Lost task 0.0 in stage 1.0 (TID 3, ip-172-31-32-183.us-west-2.compute.internal): java.lang.NoClassDefFoundError: org/apache/kafka/common/message/KafkaLZ4BlockOutputStream

      at kafka.message.ByteBufferMessageSet$.decompress()
    2. Apache Kafka
      IteratorTemplate.hasNext
      1. kafka.message.ByteBufferMessageSet$.decompress(ByteBufferMessageSet.scala:65)
      2. kafka.message.ByteBufferMessageSet$$anon$1.makeNextOuter(ByteBufferMessageSet.scala:179)
      3. kafka.message.ByteBufferMessageSet$$anon$1.makeNext(ByteBufferMessageSet.scala:192)
      4. kafka.message.ByteBufferMessageSet$$anon$1.makeNext(ByteBufferMessageSet.scala:146)
      5. kafka.utils.IteratorTemplate.maybeComputeNext(IteratorTemplate.scala:66)
      6. kafka.utils.IteratorTemplate.hasNext(IteratorTemplate.scala:58)
      6 frames
    3. Scala
      Iterator$$anon$19.hasNext
      1. scala.collection.Iterator$$anon$1.hasNext(Iterator.scala:847)
      2. scala.collection.Iterator$$anon$19.hasNext(Iterator.scala:615)
      2 frames
    4. Spark Project External Kafka
      KafkaRDD$KafkaRDDIterator.getNext
      1. org.apache.spark.streaming.kafka.KafkaRDD$KafkaRDDIterator.getNext(KafkaRDD.scala:160)
      1 frame
    5. Spark
      NextIterator.hasNext
      1. org.apache.spark.util.NextIterator.hasNext(NextIterator.scala:71)
      1 frame
    6. Scala
      Iterator$$anon$11.hasNext
      1. scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:327)
      2. scala.collection.Iterator$$anon$14.hasNext(Iterator.scala:388)
      3. scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:327)
      4. scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:327)
      4 frames
    7. Spark
      Executor$TaskRunner.run
      1. org.apache.spark.util.collection.ExternalSorter.insertAll(ExternalSorter.scala:199)
      2. org.apache.spark.shuffle.sort.SortShuffleWriter.write(SortShuffleWriter.scala:56)
      3. org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:70)
      4. org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:41)
      5. org.apache.spark.scheduler.Task.run(Task.scala:70)
      6. org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:213)
      6 frames
    8. Java RT
      Thread.run
      1. java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
      2. java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
      3. java.lang.Thread.run(Thread.java:745)
      3 frames