java.io.IOException

Could not read footer: java.lang.NoClassDefFoundError: parquet/org/codehaus/jackson/JsonGenerationException at >>>> parquet.hadoop.ParquetFileReader.readAllFootersInParallel(ParquetFileReader.java:189) at >>>> parquet.hadoop.ParquetFileReader.readAllFootersInParallelUsingSummaryFiles(ParquetFileReader.java:145) at >>>> parquet.hadoop.ParquetInputFormat.getFooters(ParquetInputFormat.java:354) at >>>> parquet.hadoop.ParquetInputFormat.getFooters(ParquetInputFormat.java:339) at >>>> parquet.hadoop.ParquetInputFormat.getSplits(ParquetInputFormat.java:246) at >>>> org.apache.spark.rdd.NewHadoopRDD.getPartitions(NewHadoopRDD.scala:85) at >>>> org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:207) at >>>> org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:205)

Samebug tips0

There are no available Samebug tips for this exception. If you know how to solve this issue, help other users by writing a short tip.

Don't give up yet. Paste your full stack trace to get a solution.

Solutions on the web15237

  • via Google Groups by Uri Laserson, 10 months ago
    >>>>> parquet.hadoop.ParquetInputFormat.getSplits(ParquetInputFormat.java:246) at >>>>> org.apache.spark.rdd.NewHadoopRDD.getPartitions(NewHadoopRDD.scala:85) at >>>>> org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:207) at >>>>> org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:205)
  • via GitHub by skoppar
    , 1 year ago
    Could not read footer: java.lang.RuntimeException: file:/Users/skoppar/workspace/pyspark-beacon/stream/allproto.log is not a Parquet file. expected magic number at tail [80, 65, 82, 49] but found [55, 73, 67, 10] at
  • via Google Groups by Dragisa Krsmanovic, 1 year ago
    Could not read footer: java.io.IOException: Could not read footer for file FileStatus{path=alluxio://dev17-spark-master01:19998/diva/eventProductRefs.parquet/part-r-00000-a5df7b1b-e9e4-4663-bdbb-e3bf68a32f0c.gz.parquet; isDirectory=false; length
  • Stack trace

    • java.io.IOException: Could not read footer: java.lang.NoClassDefFoundError: parquet/org/codehaus/jackson/JsonGenerationException at >>>> parquet.hadoop.ParquetFileReader.readAllFootersInParallel(ParquetFileReader.java:189) at >>>> parquet.hadoop.ParquetFileReader.readAllFootersInParallelUsingSummaryFiles(ParquetFileReader.java:145) at >>>> parquet.hadoop.ParquetInputFormat.getFooters(ParquetInputFormat.java:354) at >>>> parquet.hadoop.ParquetInputFormat.getFooters(ParquetInputFormat.java:339) at >>>> parquet.hadoop.ParquetInputFormat.getSplits(ParquetInputFormat.java:246) at >>>> org.apache.spark.rdd.NewHadoopRDD.getPartitions(NewHadoopRDD.scala:85) at >>>> org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:207) at >>>> org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:205) at scala.Option.getOrElse(Option.scala:120) at org.apache.spark.rdd.RDD.partitions(RDD.scala:205) at org.apache.spark.SparkContext.runJob(SparkContext.scala:863) at org.apache.spark.rdd.RDD.collect(RDD.scala:602) at $iwC$$iwC$$iwC$$iwC.<init>(<console>:20) at $iwC$$iwC$$iwC.<init>(<console>:25) at $iwC$$iwC.<init>(<console>:27) at $iwC.<init>(<console>:29)

    Write tip

    You have a different solution? A short tip here would help you and many other users who saw this issue last week.

    Users with the same issue

    You’re the first here who have seen this exception.