java.lang.ClassCastException: org.apache.hadoop.io.NullWritable cannot be cast to org.apache.hadoop.io.LongWritable

Apache's JIRA Issue Tracker | Sushanth Sowmyan | 2 years ago
tip
Do you know that we can give you better hits? Get more relevant results from Samebug’s stack trace search.
  1. 0

    Attempting to write to a HCatalog defined table backed by the AvroSerde fails with the following stacktrace: {code} java.lang.ClassCastException: org.apache.hadoop.io.NullWritable cannot be cast to org.apache.hadoop.io.LongWritable at org.apache.hadoop.hive.ql.io.avro.AvroContainerOutputFormat$1.write(AvroContainerOutputFormat.java:84) at org.apache.hcatalog.mapreduce.FileRecordWriterContainer.write(FileRecordWriterContainer.java:253) at org.apache.hcatalog.mapreduce.FileRecordWriterContainer.write(FileRecordWriterContainer.java:53) at org.apache.hcatalog.pig.HCatBaseStorer.putNext(HCatBaseStorer.java:242) at org.apache.hcatalog.pig.HCatStorer.putNext(HCatStorer.java:52) at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigOutputFormat$PigRecordWriter.write(PigOutputFormat.java:139) at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigOutputFormat$PigRecordWriter.write(PigOutputFormat.java:98) at org.apache.hadoop.mapred.MapTask$NewDirectOutputCollector.write(MapTask.java:559) at org.apache.hadoop.mapreduce.task.TaskInputOutputContextImpl.write(TaskInputOutputContextImpl.java:85) {code} The proximal cause of this failure is that the AvroContainerOutputFormat's signature mandates a LongWritable key and HCat's FileRecordWriterContainer forces a NullWritable. I'm not sure of a general fix, other than redefining HiveOutputFormat to mandate a WritableComparable. It looks like accepting WritableComparable is what's done in the other Hive OutputFormats, and there's no reason AvroContainerOutputFormat couldn't also be changed, since it's ignoring the key. That way fixing things so FileRecordWriterContainer can always use NullWritable could get spun into a different issue? The underlying cause for failure to write to AvroSerde tables is that AvroContainerOutputFormat doesn't meaningfully implement getRecordWriter, so fixing the above will just push the failure into the placeholder RecordWriter.

    Apache's JIRA Issue Tracker | 2 years ago | Sushanth Sowmyan
    java.lang.ClassCastException: org.apache.hadoop.io.NullWritable cannot be cast to org.apache.hadoop.io.LongWritable
  2. 0

    Writing to Avro backed HCatalog table using mapreduce

    Stack Overflow | 2 years ago
    java.lang.ClassCastException: org.apache.hadoop.io.NullWritable cannot be cast to org.apache.hadoop.io.LongWritable

    Root Cause Analysis

    1. java.lang.ClassCastException

      org.apache.hadoop.io.NullWritable cannot be cast to org.apache.hadoop.io.LongWritable

      at org.apache.hadoop.hive.ql.io.avro.AvroContainerOutputFormat$1.write()
    2. Hive Query Language
      AvroContainerOutputFormat$1.write
      1. org.apache.hadoop.hive.ql.io.avro.AvroContainerOutputFormat$1.write(AvroContainerOutputFormat.java:84)
      1 frame
    3. org.apache.hcatalog
      HCatStorer.putNext
      1. org.apache.hcatalog.mapreduce.FileRecordWriterContainer.write(FileRecordWriterContainer.java:253)
      2. org.apache.hcatalog.mapreduce.FileRecordWriterContainer.write(FileRecordWriterContainer.java:53)
      3. org.apache.hcatalog.pig.HCatBaseStorer.putNext(HCatBaseStorer.java:242)
      4. org.apache.hcatalog.pig.HCatStorer.putNext(HCatStorer.java:52)
      4 frames
    4. org.apache.pig
      PigOutputFormat$PigRecordWriter.write
      1. org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigOutputFormat$PigRecordWriter.write(PigOutputFormat.java:139)
      2. org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigOutputFormat$PigRecordWriter.write(PigOutputFormat.java:98)
      2 frames
    5. Hadoop
      TaskInputOutputContextImpl.write
      1. org.apache.hadoop.mapred.MapTask$NewDirectOutputCollector.write(MapTask.java:559)
      2. org.apache.hadoop.mapreduce.task.TaskInputOutputContextImpl.write(TaskInputOutputContextImpl.java:85)
      2 frames